amorfo77/json
1
0
Fork 0
mirror of https://github.com/nlohmann/json.git synced 2026-07-16 23:44:54 +00:00
Code Issues Packages Projects Releases Wiki Activity

deploy: 7c9208bfb3

Browse Source
This commit is contained in:
nlohmann
2026-07-08 18:19:46 +00:00
parent 9a231845c2
commit 279d757031
758 changed files with 56489 additions and 503 deletions
Download Patch File Download Diff File Expand all files Collapse all files
features/binary_formats/bjdata.md
+206
View File
@@ -0,0 +1,206 @@
# BJData
The [BJData format](https://neurojson.org) was derived from and improved upon
[Universal Binary JSON(UBJSON)](https://ubjson.org) specification (Draft 12). Specifically, it introduces an optimized
array container for efficient storage of N-dimensional packed arrays (**ND-arrays**); it also adds 5 new type markers -
`[u] - uint16`, `[m] - uint32`, `[M] - uint64`, `[h] - float16` and `[B] - byte` - to unambiguously map common binary
numeric types; furthermore, it uses little-endian (LE) to store all numerics instead of big-endian (BE) as in UBJSON to
avoid unnecessary conversions on commonly available platforms.
Compared to other binary JSON-like formats such as MessagePack and CBOR, both BJData and UBJSON demonstrate a rare
combination of being both binary and **quasi-human-readable**. This is because all semantic elements in BJData and
UBJSON, including the data-type markers and name/string types, are directly human-readable. Data stored in the
BJData/UBJSON format is not only compact in size, fast to read/write, but also can be directly searched or read using
simple processing.
!!! abstract "References"
- [BJData Specification](https://neurojson.org/bjdata/draft2)
## Serialization
The library uses the following mapping from JSON values types to BJData types according to the BJData specification:
| JSON value type | value/range | BJData type | marker |
|-----------------|-------------------------------------------|----------------|--------|
| null | `null` | null | `Z` |
| boolean | `true` | true | `T` |
| boolean | `false` | false | `F` |
| number_integer | -9223372036854775808..-2147483649 | int64 | `L` |
| number_integer | -2147483648..-32769 | int32 | `l` |
| number_integer | -32768..-129 | int16 | `I` |
| number_integer | -128..127 | int8 | `i` |
| number_integer | 128..255 | uint8 | `U` |
| number_integer | 256..32767 | int16 | `I` |
| number_integer | 32768..65535 | uint16 | `u` |
| number_integer | 65536..2147483647 | int32 | `l` |
| number_integer | 2147483648..4294967295 | uint32 | `m` |
| number_integer | 4294967296..9223372036854775807 | int64 | `L` |
| number_integer | 9223372036854775808..18446744073709551615 | uint64 | `M` |
| number_unsigned | 0..127 | int8 | `i` |
| number_unsigned | 128..255 | uint8 | `U` |
| number_unsigned | 256..32767 | int16 | `I` |
| number_unsigned | 32768..65535 | uint16 | `u` |
| number_unsigned | 65536..2147483647 | int32 | `l` |
| number_unsigned | 2147483648..4294967295 | uint32 | `m` |
| number_unsigned | 4294967296..9223372036854775807 | int64 | `L` |
| number_unsigned | 9223372036854775808..18446744073709551615 | uint64 | `M` |
| number_float | *any value* | float64 | `D` |
| string | *with shortest length indicator* | string | `S` |
| array | *see notes on optimized format/ND-array* | array | `[` |
| object | *see notes on optimized format* | map | `{` |
| binary | *see notes on binary values* | array | `[$B` |
!!! success "Complete mapping"
The mapping is **complete** in the sense that any JSON value type can be converted to a BJData value.
Any BJData output created by `to_bjdata` can be successfully parsed by `from_bjdata`.
!!! warning "Size constraints"
The following values can **not** be converted to a BJData value:
- strings with more than 18446744073709551615 bytes, i.e., $2^{64}-1$ bytes (theoretical)
!!! info "Unused BJData markers"
The following markers are not used in the conversion:
- `Z`: no-op values are not created.
- `C`: single-byte strings are serialized with `S` markers.
!!! info "NaN/infinity handling"
If NaN or Infinity are stored inside a JSON number, they are serialized properly. This behavior differs from the
`dump()` function which serializes NaN or Infinity to `#!json null`.
!!! info "Endianness"
A breaking difference between BJData and UBJSON is the endianness of numerical values. In BJData, all numerical data
types (integers `UiuImlML` and floating-point values `hdD`) are stored in the little-endian (LE) byte order as
opposed to big-endian as used by UBJSON. Adopting LE to store numeric records avoids unnecessary byte swapping on
most modern computers where LE is used as the default byte order.
!!! info "Optimized formats"
Optimized formats for containers are supported via two parameters of
[`to_bjdata`](../../api/basic_json/to_bjdata.md):
- Parameter `use_size` adds size information to the beginning of a container and removes the closing marker.
- Parameter `use_type` further checks whether all elements of a container have the same type and adds the type
marker to the beginning of the container. The `use_type` parameter must only be used together with
`use_size = true`.
Note that `use_size = true` alone may result in larger representations - the benefit of this parameter is that the
receiving side is immediately informed of the number of elements in the container.
!!! info "ND-array optimized format"
BJData extends UBJSON's optimized array **size** marker to support ND-arrays of uniform numerical data types
(referred to as *packed arrays*). For example, the 2-D `uint8` integer array `[[1,2],[3,4],[5,6]]`, stored as nested
optimized array in UBJSON `[ [$U#i2 1 2 [$U#i2 3 4 [$U#i2 5 6 ]`, can be further compressed in BJData to
`[$U#[$i#i2 2 3 1 2 3 4 5 6` or `[$U#[i2 i3] 1 2 3 4 5 6`.
To maintain type and size information, ND-arrays are converted to JSON objects following the **annotated array
format** (defined in the [JData specification (Draft 3)][JDataAAFmt]), when parsed using
[`from_bjdata`](../../api/basic_json/from_bjdata.md). For example, the above 2-D `uint8` array can be parsed and
accessed as
```json
{
"_ArrayType_": "uint8",
"_ArraySize_": [2,3],
"_ArrayData_": [1,2,3,4,5,6]
}
```
Likewise, when a JSON object in the above form is serialized using
[`to_bjdata`](../../api/basic_json/to_bjdata.md), it is automatically converted into a compact BJData ND-array. The
only exception is, that when the 1-dimensional vector stored in `"_ArraySize_"` contains a single integer or two
integers with one being 1, a regular 1-D optimized array is generated.
The current version of this library does not yet support automatic detection of and conversion from a nested JSON
array input to a BJData ND-array.
[JDataAAFmt]: https://github.com/NeuroJSON/jdata/blob/master/JData_specification.md#annotated-storage-of-n-d-arrays
!!! info "Restrictions in optimized data types for arrays and objects"
Due to diminished space saving, hampered readability, and increased security risks, in BJData, the allowed data
types following the `$` marker in an optimized array and object container are restricted to
**non-zero-fixed-length** data types. Therefore, the valid optimized type markers can only be one of
`UiuImlMLhdDCB`. This also means other variable (`[{SH`) or zero-length types (`TFN`) can not be used in an
optimized array or object in BJData.
!!! info "Binary values"
BJData provides a dedicated `B` marker (defined in the [BJData specification (Draft 3)][BJDataBinArr]) that is used
in optimized arrays to designate binary data. This means that, unlike UBJSON, binary data can be both serialized and
deserialized.
To preserve compatibility with BJData Draft 2, the Draft 3 optimized binary array must be explicitly enabled using
the `version` parameter of [`to_bjdata`](../../api/basic_json/to_bjdata.md).
In Draft2 mode (default), if the JSON data contains the binary type, the value stored as a list of integers, as
suggested by the BJData documentation. In particular, this means that the serialization and the deserialization of
JSON containing binary values into BJData and back will result in a different JSON object.
[BJDataBinArr]: https://github.com/NeuroJSON/bjdata/blob/master/Binary_JData_Specification.md#optimized-binary-array
??? example
```cpp
--8<-- "examples/to_bjdata.cpp"
```
Output:
```c
--8<-- "examples/to_bjdata.output"
```
## Deserialization
The library maps BJData types to JSON value types as follows:
| BJData type | JSON value type | marker |
|-------------|------------------------------------------|----------|
| no-op | *no value, next value is read* | `N` |
| null | `null` | `Z` |
| false | `false` | `F` |
| true | `true` | `T` |
| float16 | number_float | `h` |
| float32 | number_float | `d` |
| float64 | number_float | `D` |
| uint8 | number_unsigned | `U` |
| int8 | number_integer | `i` |
| uint16 | number_unsigned | `u` |
| int16 | number_integer | `I` |
| uint32 | number_unsigned | `m` |
| int32 | number_integer | `l` |
| uint64 | number_unsigned | `M` |
| int64 | number_integer | `L` |
| byte | number_unsigned | `B` |
| string | string | `S` |
| char | string | `C` |
| array | array (optimized values are supported) | `[` |
| ND-array | object (in JData annotated array format) | `[$.#[.` |
| object | object (optimized values are supported) | `{` |
| binary | binary (strongly-typed byte array) | `[$B` |
!!! success "Complete mapping"
The mapping is **complete** in the sense that any BJData value can be converted to a JSON value.
??? example
```cpp
--8<-- "examples/from_bjdata.cpp"
```
Output:
```json
--8<-- "examples/from_bjdata.output"
```
features/binary_formats/bjdata/index.html
+1 -1
View File
File diff suppressed because one or more lines are too long
features/binary_formats/bjdata/index.md
+254
View File
@@ -0,0 +1,254 @@
# BJData
The [BJData format](https://neurojson.org) was derived from and improved upon [Universal Binary JSON(UBJSON)](https://ubjson.org) specification (Draft 12). Specifically, it introduces an optimized array container for efficient storage of N-dimensional packed arrays (**ND-arrays**); it also adds 5 new type markers - `[u] - uint16`, `[m] - uint32`, `[M] - uint64`, `[h] - float16` and `[B] - byte` - to unambiguously map common binary numeric types; furthermore, it uses little-endian (LE) to store all numerics instead of big-endian (BE) as in UBJSON to avoid unnecessary conversions on commonly available platforms.
Compared to other binary JSON-like formats such as MessagePack and CBOR, both BJData and UBJSON demonstrate a rare combination of being both binary and **quasi-human-readable**. This is because all semantic elements in BJData and UBJSON, including the data-type markers and name/string types, are directly human-readable. Data stored in the BJData/UBJSON format is not only compact in size, fast to read/write, but also can be directly searched or read using simple processing.
References
- [BJData Specification](https://neurojson.org/bjdata/draft2)
## Serialization
The library uses the following mapping from JSON values types to BJData types according to the BJData specification:
| JSON value type | value/range | BJData type | marker |
| --------------- | ----------------------------------------- | ----------- | ------ |
| null | `null` | null | `Z` |
| boolean | `true` | true | `T` |
| boolean | `false` | false | `F` |
| number_integer | -9223372036854775808..-2147483649 | int64 | `L` |
| number_integer | -2147483648..-32769 | int32 | `l` |
| number_integer | -32768..-129 | int16 | `I` |
| number_integer | -128..127 | int8 | `i` |
| number_integer | 128..255 | uint8 | `U` |
| number_integer | 256..32767 | int16 | `I` |
| number_integer | 32768..65535 | uint16 | `u` |
| number_integer | 65536..2147483647 | int32 | `l` |
| number_integer | 2147483648..4294967295 | uint32 | `m` |
| number_integer | 4294967296..9223372036854775807 | int64 | `L` |
| number_integer | 9223372036854775808..18446744073709551615 | uint64 | `M` |
| number_unsigned | 0..127 | int8 | `i` |
| number_unsigned | 128..255 | uint8 | `U` |
| number_unsigned | 256..32767 | int16 | `I` |
| number_unsigned | 32768..65535 | uint16 | `u` |
| number_unsigned | 65536..2147483647 | int32 | `l` |
| number_unsigned | 2147483648..4294967295 | uint32 | `m` |
| number_unsigned | 4294967296..9223372036854775807 | int64 | `L` |
| number_unsigned | 9223372036854775808..18446744073709551615 | uint64 | `M` |
| number_float | *any value* | float64 | `D` |
| string | *with shortest length indicator* | string | `S` |
| array | *see notes on optimized format/ND-array* | array | `[` |
| object | *see notes on optimized format* | map | `{` |
| binary | *see notes on binary values* | array | `[$B` |
Complete mapping
The mapping is **complete** in the sense that any JSON value type can be converted to a BJData value.
Any BJData output created by `to_bjdata` can be successfully parsed by `from_bjdata`.
Size constraints
The following values can **not** be converted to a BJData value:
- strings with more than 18446744073709551615 bytes, i.e., 2^{64}-1 bytes (theoretical)
Unused BJData markers
The following markers are not used in the conversion:
- `Z`: no-op values are not created.
- `C`: single-byte strings are serialized with `S` markers.
NaN/infinity handling
If NaN or Infinity are stored inside a JSON number, they are serialized properly. This behavior differs from the `dump()` function which serializes NaN or Infinity to `null`.
Endianness
A breaking difference between BJData and UBJSON is the endianness of numerical values. In BJData, all numerical data types (integers `UiuImlML` and floating-point values `hdD`) are stored in the little-endian (LE) byte order as opposed to big-endian as used by UBJSON. Adopting LE to store numeric records avoids unnecessary byte swapping on most modern computers where LE is used as the default byte order.
Optimized formats
Optimized formats for containers are supported via two parameters of [`to_bjdata`](https://json.nlohmann.me/api/basic_json/to_bjdata/index.md):
- Parameter `use_size` adds size information to the beginning of a container and removes the closing marker.
- Parameter `use_type` further checks whether all elements of a container have the same type and adds the type marker to the beginning of the container. The `use_type` parameter must only be used together with `use_size = true`.
Note that `use_size = true` alone may result in larger representations - the benefit of this parameter is that the receiving side is immediately informed of the number of elements in the container.
ND-array optimized format
BJData extends UBJSON's optimized array **size** marker to support ND-arrays of uniform numerical data types (referred to as *packed arrays*). For example, the 2-D `uint8` integer array `[[1,2],[3,4],[5,6]]`, stored as nested optimized array in UBJSON `[ [$U#i2 1 2 [$U#i2 3 4 [$U#i2 5 6 ]`, can be further compressed in BJData to `[$U#[$i#i2 2 3 1 2 3 4 5 6` or `[$U#[i2 i3] 1 2 3 4 5 6`.
To maintain type and size information, ND-arrays are converted to JSON objects following the **annotated array format** (defined in the [JData specification (Draft 3)](https://github.com/NeuroJSON/jdata/blob/master/JData_specification.md#annotated-storage-of-n-d-arrays)), when parsed using [`from_bjdata`](https://json.nlohmann.me/api/basic_json/from_bjdata/index.md). For example, the above 2-D `uint8` array can be parsed and accessed as
```
{
"_ArrayType_": "uint8",
"_ArraySize_": [2,3],
"_ArrayData_": [1,2,3,4,5,6]
}
```
Likewise, when a JSON object in the above form is serialized using [`to_bjdata`](https://json.nlohmann.me/api/basic_json/to_bjdata/index.md), it is automatically converted into a compact BJData ND-array. The only exception is, that when the 1-dimensional vector stored in `"_ArraySize_"` contains a single integer or two integers with one being 1, a regular 1-D optimized array is generated.
The current version of this library does not yet support automatic detection of and conversion from a nested JSON array input to a BJData ND-array.
Restrictions in optimized data types for arrays and objects
Due to diminished space saving, hampered readability, and increased security risks, in BJData, the allowed data types following the `$` marker in an optimized array and object container are restricted to **non-zero-fixed-length** data types. Therefore, the valid optimized type markers can only be one of `UiuImlMLhdDCB`. This also means other variable (`[{SH`) or zero-length types (`TFN`) can not be used in an optimized array or object in BJData.
Binary values
BJData provides a dedicated `B` marker (defined in the [BJData specification (Draft 3)](https://github.com/NeuroJSON/bjdata/blob/master/Binary_JData_Specification.md#optimized-binary-array)) that is used in optimized arrays to designate binary data. This means that, unlike UBJSON, binary data can be both serialized and deserialized.
To preserve compatibility with BJData Draft 2, the Draft 3 optimized binary array must be explicitly enabled using the `version` parameter of [`to_bjdata`](https://json.nlohmann.me/api/basic_json/to_bjdata/index.md).
In Draft2 mode (default), if the JSON data contains the binary type, the value stored as a list of integers, as suggested by the BJData documentation. In particular, this means that the serialization and the deserialization of JSON containing binary values into BJData and back will result in a different JSON object.
Example
```
#include <iostream>
#include <iomanip>
#include <nlohmann/json.hpp>
using json = nlohmann::json;
using namespace nlohmann::literals;
// function to print BJData's diagnostic format
void print_byte(uint8_t byte)
{
if (32 < byte and byte < 128)
{
std::cout << (char)byte;
}
else
{
std::cout << (int)byte;
}
}
int main()
{
// create a JSON value
json j = R"({"compact": true, "schema": false})"_json;
// serialize it to BJData
std::vector<std::uint8_t> v = json::to_bjdata(j);
// print the vector content
for (auto& byte : v)
{
print_byte(byte);
}
std::cout << std::endl;
// create an array of numbers
json array = {1, 2, 3, 4, 5, 6, 7, 8};
// serialize it to BJData using default representation
std::vector<std::uint8_t> v_array = json::to_bjdata(array);
// serialize it to BJData using size optimization
std::vector<std::uint8_t> v_array_size = json::to_bjdata(array, true);
// serialize it to BJData using type optimization
std::vector<std::uint8_t> v_array_size_and_type = json::to_bjdata(array, true, true);
// print the vector contents
for (auto& byte : v_array)
{
print_byte(byte);
}
std::cout << std::endl;
for (auto& byte : v_array_size)
{
print_byte(byte);
}
std::cout << std::endl;
for (auto& byte : v_array_size_and_type)
{
print_byte(byte);
}
std::cout << std::endl;
}
```
Output:
```
{i7compactTi6schemaF}
[i1i2i3i4i5i6i7i8]
[#i8i1i2i3i4i5i6i7i8
[$i#i812345678
```
## Deserialization
The library maps BJData types to JSON value types as follows:
| BJData type | JSON value type | marker |
| ----------- | ---------------------------------------- | -------- |
| no-op | *no value, next value is read* | `N` |
| null | `null` | `Z` |
| false | `false` | `F` |
| true | `true` | `T` |
| float16 | number_float | `h` |
| float32 | number_float | `d` |
| float64 | number_float | `D` |
| uint8 | number_unsigned | `U` |
| int8 | number_integer | `i` |
| uint16 | number_unsigned | `u` |
| int16 | number_integer | `I` |
| uint32 | number_unsigned | `m` |
| int32 | number_integer | `l` |
| uint64 | number_unsigned | `M` |
| int64 | number_integer | `L` |
| byte | number_unsigned | `B` |
| string | string | `S` |
| char | string | `C` |
| array | array (optimized values are supported) | `[` |
| ND-array | object (in JData annotated array format) | `[$.#[.` |
| object | object (optimized values are supported) | `{` |
| binary | binary (strongly-typed byte array) | `[$B` |
Complete mapping
The mapping is **complete** in the sense that any BJData value can be converted to a JSON value.
Example
```
#include <iostream>
#include <iomanip>
#include <nlohmann/json.hpp>
using json = nlohmann::json;
int main()
{
// create byte vector
std::vector<std::uint8_t> v = {0x7B, 0x69, 0x07, 0x63, 0x6F, 0x6D, 0x70, 0x61,
0x63, 0x74, 0x54, 0x69, 0x06, 0x73, 0x63, 0x68,
0x65, 0x6D, 0x61, 0x69, 0x00, 0x7D
};
// deserialize it with BJData
json j = json::from_bjdata(v);
// print the deserialized JSON value
std::cout << std::setw(2) << j << std::endl;
}
```
Output:
```
{
"compact": true,
"schema": 0
}
```
features/binary_formats/bson.md
+98
View File
@@ -0,0 +1,98 @@
# BSON
BSON, short for Binary JSON, is a binary-encoded serialization of JSON-like documents. Like JSON, BSON supports the
embedding of documents and arrays within other documents and arrays. BSON also contains extensions that allow
representation of data types that are not part of the JSON spec. For example, BSON has a Date type and a BinData type.
!!! abstract "References"
- [BSON Website](http://bsonspec.org) - the main source on BSON
- [BSON Specification](http://bsonspec.org/spec.html) - the specification
## Serialization
The library uses the following mapping from JSON values types to BSON types:
| JSON value type | value/range | BSON type | marker |
|-----------------|-------------------------------------------|-----------|--------|
| null | `null` | null | 0x0A |
| boolean | `true`, `false` | boolean | 0x08 |
| number_integer | -9223372036854775808..-2147483649 | int64 | 0x12 |
| number_integer | -2147483648..2147483647 | int32 | 0x10 |
| number_integer | 2147483648..9223372036854775807 | int64 | 0x12 |
| number_unsigned | 0..2147483647 | int32 | 0x10 |
| number_unsigned | 2147483648..9223372036854775807 | int64 | 0x12 |
| number_unsigned | 9223372036854775808..18446744073709551615 | uint64 | 0x11 |
| number_float | *any value* | double | 0x01 |
| string | *any value* | string | 0x02 |
| array | *any value* | document | 0x04 |
| object | *any value* | document | 0x03 |
| binary | *any value* | binary | 0x05 |
!!! warning "Incomplete mapping"
The mapping is **incomplete**, since only JSON-objects (and things contained therein) can be serialized to BSON.
Also, keys may not contain U+0000, since they are serialized a zero-terminated c-strings.
??? example
```cpp
--8<-- "examples/to_bson.cpp"
```
Output:
```c
--8<-- "examples/to_bson.output"
```
## Deserialization
The library maps BSON record types to JSON value types as follows:
| BSON type | BSON marker byte | JSON value type |
|--------------------------|------------------|-----------------|
| double | 0x01 | number_float |
| string | 0x02 | string |
| document | 0x03 | object |
| array | 0x04 | array |
| binary | 0x05 | binary |
| undefined | 0x06 | *unsupported* |
| ObjectId | 0x07 | *unsupported* |
| boolean | 0x08 | boolean |
| UTC Date-Time | 0x09 | *unsupported* |
| null | 0x0A | null |
| Regular Expr. | 0x0B | *unsupported* |
| DB Pointer | 0x0C | *unsupported* |
| JavaScript Code | 0x0D | *unsupported* |
| Symbol | 0x0E | *unsupported* |
| JavaScript Code w/ scope | 0x0F | *unsupported* |
| int32 | 0x10 | number_integer |
| uint64(Timestamp) | 0x11 | number_unsigned |
| int64 | 0x12 | number_integer |
| 128-bit decimal float | 0x13 | *unsupported* |
| Max Key | 0x7F | *unsupported* |
| Min Key | 0xFF | *unsupported* |
!!! warning "Incomplete mapping"
The mapping is **incomplete**. The unsupported mappings are indicated in the table above.
!!! note "Handling of BSON type 0x11"
BSON type 0x11 is used to represent uint64 numbers. This library treats these values purely as uint64 numbers
and does not parse them into date-related formats.
??? example
```cpp
--8<-- "examples/from_bson.cpp"
```
Output:
```json
--8<-- "examples/from_bson.output"
```
features/binary_formats/bson/index.html
+1 -1
View File
File diff suppressed because one or more lines are too long
features/binary_formats/bson/index.md
+136
View File
@@ -0,0 +1,136 @@
# BSON
BSON, short for Binary JSON, is a binary-encoded serialization of JSON-like documents. Like JSON, BSON supports the embedding of documents and arrays within other documents and arrays. BSON also contains extensions that allow representation of data types that are not part of the JSON spec. For example, BSON has a Date type and a BinData type.
References
- [BSON Website](http://bsonspec.org) - the main source on BSON
- [BSON Specification](http://bsonspec.org/spec.html) - the specification
## Serialization
The library uses the following mapping from JSON values types to BSON types:
| JSON value type | value/range | BSON type | marker |
| --------------- | ----------------------------------------- | --------- | ------ |
| null | `null` | null | 0x0A |
| boolean | `true`, `false` | boolean | 0x08 |
| number_integer | -9223372036854775808..-2147483649 | int64 | 0x12 |
| number_integer | -2147483648..2147483647 | int32 | 0x10 |
| number_integer | 2147483648..9223372036854775807 | int64 | 0x12 |
| number_unsigned | 0..2147483647 | int32 | 0x10 |
| number_unsigned | 2147483648..9223372036854775807 | int64 | 0x12 |
| number_unsigned | 9223372036854775808..18446744073709551615 | uint64 | 0x11 |
| number_float | *any value* | double | 0x01 |
| string | *any value* | string | 0x02 |
| array | *any value* | document | 0x04 |
| object | *any value* | document | 0x03 |
| binary | *any value* | binary | 0x05 |
Incomplete mapping
The mapping is **incomplete**, since only JSON-objects (and things contained therein) can be serialized to BSON. Also, keys may not contain U+0000, since they are serialized a zero-terminated c-strings.
Example
```
#include <iostream>
#include <iomanip>
#include <nlohmann/json.hpp>
using json = nlohmann::json;
using namespace nlohmann::literals;
int main()
{
// create a JSON value
json j = R"({"compact": true, "schema": 0})"_json;
// serialize it to BSON
std::vector<std::uint8_t> v = json::to_bson(j);
// print the vector content
for (auto& byte : v)
{
std::cout << "0x" << std::hex << std::setw(2) << std::setfill('0') << (int)byte << " ";
}
std::cout << std::endl;
}
```
Output:
```
0x1b 0x00 0x00 0x00 0x08 0x63 0x6f 0x6d 0x70 0x61 0x63 0x74 0x00 0x01 0x10 0x73 0x63 0x68 0x65 0x6d 0x61 0x00 0x00 0x00 0x00 0x00 0x00
```
## Deserialization
The library maps BSON record types to JSON value types as follows:
| BSON type | BSON marker byte | JSON value type |
| ------------------------ | ---------------- | --------------- |
| double | 0x01 | number_float |
| string | 0x02 | string |
| document | 0x03 | object |
| array | 0x04 | array |
| binary | 0x05 | binary |
| undefined | 0x06 | *unsupported* |
| ObjectId | 0x07 | *unsupported* |
| boolean | 0x08 | boolean |
| UTC Date-Time | 0x09 | *unsupported* |
| null | 0x0A | null |
| Regular Expr. | 0x0B | *unsupported* |
| DB Pointer | 0x0C | *unsupported* |
| JavaScript Code | 0x0D | *unsupported* |
| Symbol | 0x0E | *unsupported* |
| JavaScript Code w/ scope | 0x0F | *unsupported* |
| int32 | 0x10 | number_integer |
| uint64(Timestamp) | 0x11 | number_unsigned |
| int64 | 0x12 | number_integer |
| 128-bit decimal float | 0x13 | *unsupported* |
| Max Key | 0x7F | *unsupported* |
| Min Key | 0xFF | *unsupported* |
Incomplete mapping
The mapping is **incomplete**. The unsupported mappings are indicated in the table above.
Handling of BSON type 0x11
BSON type 0x11 is used to represent uint64 numbers. This library treats these values purely as uint64 numbers and does not parse them into date-related formats.
Example
```
#include <iostream>
#include <iomanip>
#include <nlohmann/json.hpp>
using json = nlohmann::json;
int main()
{
// create byte vector
std::vector<std::uint8_t> v = {0x1b, 0x00, 0x00, 0x00, 0x08, 0x63, 0x6f, 0x6d,
0x70, 0x61, 0x63, 0x74, 0x00, 0x01, 0x10, 0x73,
0x63, 0x68, 0x65, 0x6d, 0x61, 0x00, 0x00, 0x00,
0x00, 0x00, 0x00
};
// deserialize it with BSON
json j = json::from_bson(v);
// print the deserialized JSON value
std::cout << std::setw(2) << j << std::endl;
}
```
Output:
```
{
"compact": true,
"schema": 0
}
```
features/binary_formats/cbor.md
+181
View File
@@ -0,0 +1,181 @@
# CBOR
The Concise Binary Object Representation (CBOR) is a data format whose design goals include the possibility of
extremely small code sizes, fairly small message size, and extensibility without the need for version negotiation.
!!! abstract "References"
- [CBOR Website](http://cbor.io) - the main source on CBOR
- [CBOR Playground](http://cbor.me) - an interactive webpage to translate between JSON and CBOR
- [RFC 7049](https://tools.ietf.org/html/rfc7049) - the CBOR specification
## Serialization
The library uses the following mapping from JSON values types to CBOR types according to the CBOR specification
([RFC 7049](https://www.rfc-editor.org/rfc/rfc7049.html)):
| JSON value type | value/range | CBOR type | first byte |
|-----------------|--------------------------------------------|-----------------------------------|------------|
| null | `null` | Null | 0xF6 |
| boolean | `true` | True | 0xF5 |
| boolean | `false` | False | 0xF4 |
| number_integer | -9223372036854775808..-2147483649 | Negative integer (8 bytes follow) | 0x3B |
| number_integer | -2147483648..-32769 | Negative integer (4 bytes follow) | 0x3A |
| number_integer | -32768..-129 | Negative integer (2 bytes follow) | 0x39 |
| number_integer | -128..-25 | Negative integer (1 byte follow) | 0x38 |
| number_integer | -24..-1 | Negative integer | 0x20..0x37 |
| number_integer | 0..23 | Integer | 0x00..0x17 |
| number_integer | 24..255 | Unsigned integer (1 byte follow) | 0x18 |
| number_integer | 256..65535 | Unsigned integer (2 bytes follow) | 0x19 |
| number_integer | 65536..4294967295 | Unsigned integer (4 bytes follow) | 0x1A |
| number_integer | 4294967296..18446744073709551615 | Unsigned integer (8 bytes follow) | 0x1B |
| number_unsigned | 0..23 | Integer | 0x00..0x17 |
| number_unsigned | 24..255 | Unsigned integer (1 byte follow) | 0x18 |
| number_unsigned | 256..65535 | Unsigned integer (2 bytes follow) | 0x19 |
| number_unsigned | 65536..4294967295 | Unsigned integer (4 bytes follow) | 0x1A |
| number_unsigned | 4294967296..18446744073709551615 | Unsigned integer (8 bytes follow) | 0x1B |
| number_float | *any value representable by a float* | Single-Precision Float | 0xFA |
| number_float | *any value NOT representable by a float* | Double-Precision Float | 0xFB |
| string | *length*: 0..23 | UTF-8 string | 0x60..0x77 |
| string | *length*: 24..255 | UTF-8 string (1 byte follow) | 0x78 |
| string | *length*: 256..65535 | UTF-8 string (2 bytes follow) | 0x79 |
| string | *length*: 65536..4294967295 | UTF-8 string (4 bytes follow) | 0x7A |
| string | *length*: 4294967296..18446744073709551615 | UTF-8 string (8 bytes follow) | 0x7B |
| array | *size*: 0..23 | array | 0x80..0x97 |
| array | *size*: 24..255 | array (1 byte follow) | 0x98 |
| array | *size*: 256..65535 | array (2 bytes follow) | 0x99 |
| array | *size*: 65536..4294967295 | array (4 bytes follow) | 0x9A |
| array | *size*: 4294967296..18446744073709551615 | array (8 bytes follow) | 0x9B |
| object | *size*: 0..23 | map | 0xA0..0xB7 |
| object | *size*: 24..255 | map (1 byte follow) | 0xB8 |
| object | *size*: 256..65535 | map (2 bytes follow) | 0xB9 |
| object | *size*: 65536..4294967295 | map (4 bytes follow) | 0xBA |
| object | *size*: 4294967296..18446744073709551615 | map (8 bytes follow) | 0xBB |
| binary | *size*: 0..23 | byte string | 0x40..0x57 |
| binary | *size*: 24..255 | byte string (1 byte follow) | 0x58 |
| binary | *size*: 256..65535 | byte string (2 bytes follow) | 0x59 |
| binary | *size*: 65536..4294967295 | byte string (4 bytes follow) | 0x5A |
| binary | *size*: 4294967296..18446744073709551615 | byte string (8 bytes follow) | 0x5B |
Binary values with subtype are mapped to tagged values (0xD8..0xDB) depending on the subtype, followed by a byte string,
see "binary" cells in the table above.
!!! success "Complete mapping"
The mapping is **complete** in the sense that any JSON value type can be converted to a CBOR value.
!!! info "NaN/infinity handling"
If NaN or Infinity are stored inside a JSON number, they are serialized properly. This behavior differs from the normal JSON serialization which serializes NaN or Infinity to `null`.
!!! info "Unused CBOR types"
The following CBOR types are not used in the conversion:
- UTF-8 strings terminated by "break" (0x7F)
- arrays terminated by "break" (0x9F)
- maps terminated by "break" (0xBF)
- byte strings terminated by "break" (0x5F)
- date/time (0xC0..0xC1)
- bignum (0xC2..0xC3)
- decimal fraction (0xC4)
- bigfloat (0xC5)
- expected conversions (0xD5..0xD7)
- simple values (0xE0..0xF3, 0xF8)
- undefined (0xF7)
- half-precision floats (0xF9)
- break (0xFF)
!!! info "Tagged items"
Binary subtypes will be serialized as tagged items. See [binary values](../binary_values.md#cbor) for an example.
??? example
```cpp
--8<-- "examples/to_cbor.cpp"
```
Output:
```c
--8<-- "examples/to_cbor.output"
```
## Deserialization
The library maps CBOR types to JSON value types as follows:
| CBOR type | JSON value type | first byte |
|------------------------|-----------------|------------|
| Integer | number_unsigned | 0x00..0x17 |
| Unsigned integer | number_unsigned | 0x18 |
| Unsigned integer | number_unsigned | 0x19 |
| Unsigned integer | number_unsigned | 0x1A |
| Unsigned integer | number_unsigned | 0x1B |
| Negative integer | number_integer | 0x20..0x37 |
| Negative integer | number_integer | 0x38 |
| Negative integer | number_integer | 0x39 |
| Negative integer | number_integer | 0x3A |
| Negative integer | number_integer | 0x3B |
| Byte string | binary | 0x40..0x57 |
| Byte string | binary | 0x58 |
| Byte string | binary | 0x59 |
| Byte string | binary | 0x5A |
| Byte string | binary | 0x5B |
| UTF-8 string | string | 0x60..0x77 |
| UTF-8 string | string | 0x78 |
| UTF-8 string | string | 0x79 |
| UTF-8 string | string | 0x7A |
| UTF-8 string | string | 0x7B |
| UTF-8 string | string | 0x7F |
| array | array | 0x80..0x97 |
| array | array | 0x98 |
| array | array | 0x99 |
| array | array | 0x9A |
| array | array | 0x9B |
| array | array | 0x9F |
| map | object | 0xA0..0xB7 |
| map | object | 0xB8 |
| map | object | 0xB9 |
| map | object | 0xBA |
| map | object | 0xBB |
| map | object | 0xBF |
| False | `false` | 0xF4 |
| True | `true` | 0xF5 |
| Null | `null` | 0xF6 |
| Half-Precision Float | number_float | 0xF9 |
| Single-Precision Float | number_float | 0xFA |
| Double-Precision Float | number_float | 0xFB |
!!! warning "Incomplete mapping"
The mapping is **incomplete** in the sense that not all CBOR types can be converted to a JSON value. The following CBOR types are not supported and will yield parse errors:
- date/time (0xC0..0xC1)
- bignum (0xC2..0xC3)
- decimal fraction (0xC4)
- bigfloat (0xC5)
- expected conversions (0xD5..0xD7)
- simple values (0xE0..0xF3, 0xF8)
- undefined (0xF7)
!!! warning "Object keys"
CBOR allows map keys of any type, whereas JSON only allows strings as keys in object values. Therefore, CBOR maps with keys other than UTF-8 strings are rejected.
!!! warning "Tagged items"
Tagged items will throw a parse error by default. They can be ignored by passing `cbor_tag_handler_t::ignore` to function `from_cbor`. They can be stored by passing `cbor_tag_handler_t::store` to function `from_cbor`.
??? example
```cpp
--8<-- "examples/from_cbor.cpp"
```
Output:
```json
--8<-- "examples/from_cbor.output"
```
features/binary_formats/cbor/index.html
+1 -1
View File
File diff suppressed because one or more lines are too long
features/binary_formats/cbor/index.md
+221
View File
@@ -0,0 +1,221 @@
# CBOR
The Concise Binary Object Representation (CBOR) is a data format whose design goals include the possibility of extremely small code sizes, fairly small message size, and extensibility without the need for version negotiation.
References
- [CBOR Website](http://cbor.io) - the main source on CBOR
- [CBOR Playground](http://cbor.me) - an interactive webpage to translate between JSON and CBOR
- [RFC 7049](https://tools.ietf.org/html/rfc7049) - the CBOR specification
## Serialization
The library uses the following mapping from JSON values types to CBOR types according to the CBOR specification ([RFC 7049](https://www.rfc-editor.org/rfc/rfc7049.html)):
| JSON value type | value/range | CBOR type | first byte |
| --------------- | ------------------------------------------ | --------------------------------- | ---------- |
| null | `null` | Null | 0xF6 |
| boolean | `true` | True | 0xF5 |
| boolean | `false` | False | 0xF4 |
| number_integer | -9223372036854775808..-2147483649 | Negative integer (8 bytes follow) | 0x3B |
| number_integer | -2147483648..-32769 | Negative integer (4 bytes follow) | 0x3A |
| number_integer | -32768..-129 | Negative integer (2 bytes follow) | 0x39 |
| number_integer | -128..-25 | Negative integer (1 byte follow) | 0x38 |
| number_integer | -24..-1 | Negative integer | 0x20..0x37 |
| number_integer | 0..23 | Integer | 0x00..0x17 |
| number_integer | 24..255 | Unsigned integer (1 byte follow) | 0x18 |
| number_integer | 256..65535 | Unsigned integer (2 bytes follow) | 0x19 |
| number_integer | 65536..4294967295 | Unsigned integer (4 bytes follow) | 0x1A |
| number_integer | 4294967296..18446744073709551615 | Unsigned integer (8 bytes follow) | 0x1B |
| number_unsigned | 0..23 | Integer | 0x00..0x17 |
| number_unsigned | 24..255 | Unsigned integer (1 byte follow) | 0x18 |
| number_unsigned | 256..65535 | Unsigned integer (2 bytes follow) | 0x19 |
| number_unsigned | 65536..4294967295 | Unsigned integer (4 bytes follow) | 0x1A |
| number_unsigned | 4294967296..18446744073709551615 | Unsigned integer (8 bytes follow) | 0x1B |
| number_float | *any value representable by a float* | Single-Precision Float | 0xFA |
| number_float | *any value NOT representable by a float* | Double-Precision Float | 0xFB |
| string | *length*: 0..23 | UTF-8 string | 0x60..0x77 |
| string | *length*: 24..255 | UTF-8 string (1 byte follow) | 0x78 |
| string | *length*: 256..65535 | UTF-8 string (2 bytes follow) | 0x79 |
| string | *length*: 65536..4294967295 | UTF-8 string (4 bytes follow) | 0x7A |
| string | *length*: 4294967296..18446744073709551615 | UTF-8 string (8 bytes follow) | 0x7B |
| array | *size*: 0..23 | array | 0x80..0x97 |
| array | *size*: 24..255 | array (1 byte follow) | 0x98 |
| array | *size*: 256..65535 | array (2 bytes follow) | 0x99 |
| array | *size*: 65536..4294967295 | array (4 bytes follow) | 0x9A |
| array | *size*: 4294967296..18446744073709551615 | array (8 bytes follow) | 0x9B |
| object | *size*: 0..23 | map | 0xA0..0xB7 |
| object | *size*: 24..255 | map (1 byte follow) | 0xB8 |
| object | *size*: 256..65535 | map (2 bytes follow) | 0xB9 |
| object | *size*: 65536..4294967295 | map (4 bytes follow) | 0xBA |
| object | *size*: 4294967296..18446744073709551615 | map (8 bytes follow) | 0xBB |
| binary | *size*: 0..23 | byte string | 0x40..0x57 |
| binary | *size*: 24..255 | byte string (1 byte follow) | 0x58 |
| binary | *size*: 256..65535 | byte string (2 bytes follow) | 0x59 |
| binary | *size*: 65536..4294967295 | byte string (4 bytes follow) | 0x5A |
| binary | *size*: 4294967296..18446744073709551615 | byte string (8 bytes follow) | 0x5B |
Binary values with subtype are mapped to tagged values (0xD8..0xDB) depending on the subtype, followed by a byte string, see "binary" cells in the table above.
Complete mapping
The mapping is **complete** in the sense that any JSON value type can be converted to a CBOR value.
NaN/infinity handling
If NaN or Infinity are stored inside a JSON number, they are serialized properly. This behavior differs from the normal JSON serialization which serializes NaN or Infinity to `null`.
Unused CBOR types
The following CBOR types are not used in the conversion:
- UTF-8 strings terminated by "break" (0x7F)
- arrays terminated by "break" (0x9F)
- maps terminated by "break" (0xBF)
- byte strings terminated by "break" (0x5F)
- date/time (0xC0..0xC1)
- bignum (0xC2..0xC3)
- decimal fraction (0xC4)
- bigfloat (0xC5)
- expected conversions (0xD5..0xD7)
- simple values (0xE0..0xF3, 0xF8)
- undefined (0xF7)
- half-precision floats (0xF9)
- break (0xFF)
Tagged items
Binary subtypes will be serialized as tagged items. See [binary values](https://json.nlohmann.me/features/binary_values/#cbor) for an example.
Example
```
#include <iostream>
#include <iomanip>
#include <nlohmann/json.hpp>
using json = nlohmann::json;
using namespace nlohmann::literals;
int main()
{
// create a JSON value
json j = R"({"compact": true, "schema": 0})"_json;
// serialize it to CBOR
std::vector<std::uint8_t> v = json::to_cbor(j);
// print the vector content
for (auto& byte : v)
{
std::cout << "0x" << std::hex << std::setw(2) << std::setfill('0') << (int)byte << " ";
}
std::cout << std::endl;
}
```
Output:
```
0xa2 0x67 0x63 0x6f 0x6d 0x70 0x61 0x63 0x74 0xf5 0x66 0x73 0x63 0x68 0x65 0x6d 0x61 0x00
```
## Deserialization
The library maps CBOR types to JSON value types as follows:
| CBOR type | JSON value type | first byte |
| ---------------------- | --------------- | ---------- |
| Integer | number_unsigned | 0x00..0x17 |
| Unsigned integer | number_unsigned | 0x18 |
| Unsigned integer | number_unsigned | 0x19 |
| Unsigned integer | number_unsigned | 0x1A |
| Unsigned integer | number_unsigned | 0x1B |
| Negative integer | number_integer | 0x20..0x37 |
| Negative integer | number_integer | 0x38 |
| Negative integer | number_integer | 0x39 |
| Negative integer | number_integer | 0x3A |
| Negative integer | number_integer | 0x3B |
| Byte string | binary | 0x40..0x57 |
| Byte string | binary | 0x58 |
| Byte string | binary | 0x59 |
| Byte string | binary | 0x5A |
| Byte string | binary | 0x5B |
| UTF-8 string | string | 0x60..0x77 |
| UTF-8 string | string | 0x78 |
| UTF-8 string | string | 0x79 |
| UTF-8 string | string | 0x7A |
| UTF-8 string | string | 0x7B |
| UTF-8 string | string | 0x7F |
| array | array | 0x80..0x97 |
| array | array | 0x98 |
| array | array | 0x99 |
| array | array | 0x9A |
| array | array | 0x9B |
| array | array | 0x9F |
| map | object | 0xA0..0xB7 |
| map | object | 0xB8 |
| map | object | 0xB9 |
| map | object | 0xBA |
| map | object | 0xBB |
| map | object | 0xBF |
| False | `false` | 0xF4 |
| True | `true` | 0xF5 |
| Null | `null` | 0xF6 |
| Half-Precision Float | number_float | 0xF9 |
| Single-Precision Float | number_float | 0xFA |
| Double-Precision Float | number_float | 0xFB |
Incomplete mapping
The mapping is **incomplete** in the sense that not all CBOR types can be converted to a JSON value. The following CBOR types are not supported and will yield parse errors:
- date/time (0xC0..0xC1)
- bignum (0xC2..0xC3)
- decimal fraction (0xC4)
- bigfloat (0xC5)
- expected conversions (0xD5..0xD7)
- simple values (0xE0..0xF3, 0xF8)
- undefined (0xF7)
Object keys
CBOR allows map keys of any type, whereas JSON only allows strings as keys in object values. Therefore, CBOR maps with keys other than UTF-8 strings are rejected.
Tagged items
Tagged items will throw a parse error by default. They can be ignored by passing `cbor_tag_handler_t::ignore` to function `from_cbor`. They can be stored by passing `cbor_tag_handler_t::store` to function `from_cbor`.
Example
```
#include <iostream>
#include <iomanip>
#include <nlohmann/json.hpp>
using json = nlohmann::json;
int main()
{
// create byte vector
std::vector<std::uint8_t> v = {0xa2, 0x67, 0x63, 0x6f, 0x6d, 0x70, 0x61, 0x63,
0x74, 0xf5, 0x66, 0x73, 0x63, 0x68, 0x65, 0x6d,
0x61, 0x00
};
// deserialize it with CBOR
json j = json::from_cbor(v);
// print the deserialized JSON value
std::cout << std::setw(2) << j << std::endl;
}
```
Output:
```
{
"compact": true,
"schema": 0
}
```
features/binary_formats/index.html
+1 -1
View File
File diff suppressed because one or more lines are too long
features/binary_formats/index.md
+51
View File
@@ -0,0 +1,51 @@
# Binary Formats
Though JSON is a ubiquitous data format, it is not a very compact format suitable for data exchange, for instance, over a network. Hence, the library supports
- [BJData](https://json.nlohmann.me/features/binary_formats/bjdata/index.md) (Binary JData),
- [BSON](https://json.nlohmann.me/features/binary_formats/bson/index.md) (Binary JSON),
- [CBOR](https://json.nlohmann.me/features/binary_formats/cbor/index.md) (Concise Binary Object Representation),
- [MessagePack](https://json.nlohmann.me/features/binary_formats/messagepack/index.md), and
- [UBJSON](https://json.nlohmann.me/features/binary_formats/ubjson/index.md) (Universal Binary JSON)
to efficiently encode JSON values to byte vectors and to decode such vectors.
## Comparison
### Completeness
| Format | Serialization | Deserialization |
| ----------- | --------------------------------------------- | -------------------------------------------- |
| BJData | complete | complete |
| BSON | incomplete: top-level value must be an object | incomplete, but all JSON types are supported |
| CBOR | complete | incomplete, but all JSON types are supported |
| MessagePack | complete | complete |
| UBJSON | complete | complete |
### Binary values
| Format | Binary values | Binary subtypes |
| ----------- | ------------- | --------------- |
| BJData | not supported | not supported |
| BSON | supported | supported |
| CBOR | supported | supported |
| MessagePack | supported | supported |
| UBJSON | not supported | not supported |
See [binary values](https://json.nlohmann.me/features/binary_values/index.md) for more information.
### Sizes
| Format | canada.json | twitter.json | citm_catalog.json | jeopardy.json |
| ------------------ | ----------- | ------------ | ----------------- | ------------- |
| BJData | 53.2 % | 91.1 % | 78.1 % | 96.6 % |
| BJData (size) | 58.6 % | 92.1 % | 86.7 % | 97.4 % |
| BJData (size+type) | 58.6 % | 92.1 % | 86.5 % | 97.4 % |
| BSON | 85.8 % | 95.2 % | 95.8 % | 106.7 % |
| CBOR | 50.5 % | 86.3 % | 68.4 % | 88.0 % |
| MessagePack | 50.5 % | 86.0 % | 68.5 % | 87.9 % |
| UBJSON | 53.2 % | 91.3 % | 78.2 % | 96.6 % |
| UBJSON (size) | 58.6 % | 92.3 % | 86.8 % | 97.4 % |
| UBJSON (size+type) | 55.9 % | 92.3 % | 85.0 % | 95.0 % |
Sizes compared to minified JSON value.
features/binary_formats/messagepack.md
+143
View File
@@ -0,0 +1,143 @@
# MessagePack
MessagePack is an efficient binary serialization format. It lets you exchange data among multiple languages like JSON.
But it's faster and smaller. Small integers are encoded into a single byte, and typical short strings require only one
extra byte in addition to the strings themselves.
!!! abstract "References"
- [MessagePack website](https://msgpack.org)
- [MessagePack specification](https://github.com/msgpack/msgpack/blob/master/spec.md)
## Serialization
The library uses the following mapping from JSON values types to MessagePack types according to the MessagePack
specification:
| JSON value type | value/range | MessagePack type | first byte |
|-----------------|------------------------------------------|------------------|------------|
| null | `null` | nil | 0xC0 |
| boolean | `true` | true | 0xC3 |
| boolean | `false` | false | 0xC2 |
| number_integer | -9223372036854775808..-2147483649 | int64 | 0xD3 |
| number_integer | -2147483648..-32769 | int32 | 0xD2 |
| number_integer | -32768..-129 | int16 | 0xD1 |
| number_integer | -128..-33 | int8 | 0xD0 |
| number_integer | -32..-1 | negative fixint | 0xE0..0xFF |
| number_integer | 0..127 | positive fixint | 0x00..0x7F |
| number_integer | 128..255 | uint 8 | 0xCC |
| number_integer | 256..65535 | uint 16 | 0xCD |
| number_integer | 65536..4294967295 | uint 32 | 0xCE |
| number_integer | 4294967296..18446744073709551615 | uint 64 | 0xCF |
| number_unsigned | 0..127 | positive fixint | 0x00..0x7F |
| number_unsigned | 128..255 | uint 8 | 0xCC |
| number_unsigned | 256..65535 | uint 16 | 0xCD |
| number_unsigned | 65536..4294967295 | uint 32 | 0xCE |
| number_unsigned | 4294967296..18446744073709551615 | uint 64 | 0xCF |
| number_float | *any value representable by a float* | float 32 | 0xCA |
| number_float | *any value NOT representable by a float* | float 64 | 0xCB |
| string | *length*: 0..31 | fixstr | 0xA0..0xBF |
| string | *length*: 32..255 | str 8 | 0xD9 |
| string | *length*: 256..65535 | str 16 | 0xDA |
| string | *length*: 65536..4294967295 | str 32 | 0xDB |
| array | *size*: 0..15 | fixarray | 0x90..0x9F |
| array | *size*: 16..65535 | array 16 | 0xDC |
| array | *size*: 65536..4294967295 | array 32 | 0xDD |
| object | *size*: 0..15 | fix map | 0x80..0x8F |
| object | *size*: 16..65535 | map 16 | 0xDE |
| object | *size*: 65536..4294967295 | map 32 | 0xDF |
| binary | *size*: 0..255 | bin 8 | 0xC4 |
| binary | *size*: 256..65535 | bin 16 | 0xC5 |
| binary | *size*: 65536..4294967295 | bin 32 | 0xC6 |
!!! success "Complete mapping"
The mapping is **complete** in the sense that any JSON value type can be converted to a MessagePack value.
Any MessagePack output created by `to_msgpack` can be successfully parsed by `from_msgpack`.
!!! warning "Size constraints"
The following values can **not** be converted to a MessagePack value:
- strings with more than 4294967295 bytes
- byte strings with more than 4294967295 bytes
- arrays with more than 4294967295 elements
- objects with more than 4294967295 elements
!!! info "NaN/infinity handling"
If NaN or Infinity are stored inside a JSON number, they are serialized properly in contrast to the
[dump](../../api/basic_json/dump.md) function which serializes NaN or Infinity to `null`.
??? example
```cpp
--8<-- "examples/to_msgpack.cpp"
```
Output:
```c
--8<-- "examples/to_msgpack.output"
```
## Deserialization
The library maps MessagePack types to JSON value types as follows:
| MessagePack type | JSON value type | first byte |
|------------------|-----------------|------------|
| positive fixint | number_unsigned | 0x00..0x7F |
| fixmap | object | 0x80..0x8F |
| fixarray | array | 0x90..0x9F |
| fixstr | string | 0xA0..0xBF |
| nil | `null` | 0xC0 |
| false | `false` | 0xC2 |
| true | `true` | 0xC3 |
| float 32 | number_float | 0xCA |
| float 64 | number_float | 0xCB |
| uint 8 | number_unsigned | 0xCC |
| uint 16 | number_unsigned | 0xCD |
| uint 32 | number_unsigned | 0xCE |
| uint 64 | number_unsigned | 0xCF |
| int 8 | number_integer | 0xD0 |
| int 16 | number_integer | 0xD1 |
| int 32 | number_integer | 0xD2 |
| int 64 | number_integer | 0xD3 |
| str 8 | string | 0xD9 |
| str 16 | string | 0xDA |
| str 32 | string | 0xDB |
| array 16 | array | 0xDC |
| array 32 | array | 0xDD |
| map 16 | object | 0xDE |
| map 32 | object | 0xDF |
| bin 8 | binary | 0xC4 |
| bin 16 | binary | 0xC5 |
| bin 32 | binary | 0xC6 |
| ext 8 | binary | 0xC7 |
| ext 16 | binary | 0xC8 |
| ext 32 | binary | 0xC9 |
| fixext 1 | binary | 0xD4 |
| fixext 2 | binary | 0xD5 |
| fixext 4 | binary | 0xD6 |
| fixext 8 | binary | 0xD7 |
| fixext 16 | binary | 0xD8 |
| negative fixint | number_integer | 0xE0-0xFF |
!!! info
Any MessagePack output created by `to_msgpack` can be successfully parsed by `from_msgpack`.
??? example
```cpp
--8<-- "examples/from_msgpack.cpp"
```
Output:
```json
--8<-- "examples/from_msgpack.output"
```
features/binary_formats/messagepack/index.html
+1 -1
View File
File diff suppressed because one or more lines are too long
features/binary_formats/messagepack/index.md
+181
View File
@@ -0,0 +1,181 @@
# MessagePack
MessagePack is an efficient binary serialization format. It lets you exchange data among multiple languages like JSON. But it's faster and smaller. Small integers are encoded into a single byte, and typical short strings require only one extra byte in addition to the strings themselves.
References
- [MessagePack website](https://msgpack.org)
- [MessagePack specification](https://github.com/msgpack/msgpack/blob/master/spec.md)
## Serialization
The library uses the following mapping from JSON values types to MessagePack types according to the MessagePack specification:
| JSON value type | value/range | MessagePack type | first byte |
| --------------- | ---------------------------------------- | ---------------- | ---------- |
| null | `null` | nil | 0xC0 |
| boolean | `true` | true | 0xC3 |
| boolean | `false` | false | 0xC2 |
| number_integer | -9223372036854775808..-2147483649 | int64 | 0xD3 |
| number_integer | -2147483648..-32769 | int32 | 0xD2 |
| number_integer | -32768..-129 | int16 | 0xD1 |
| number_integer | -128..-33 | int8 | 0xD0 |
| number_integer | -32..-1 | negative fixint | 0xE0..0xFF |
| number_integer | 0..127 | positive fixint | 0x00..0x7F |
| number_integer | 128..255 | uint 8 | 0xCC |
| number_integer | 256..65535 | uint 16 | 0xCD |
| number_integer | 65536..4294967295 | uint 32 | 0xCE |
| number_integer | 4294967296..18446744073709551615 | uint 64 | 0xCF |
| number_unsigned | 0..127 | positive fixint | 0x00..0x7F |
| number_unsigned | 128..255 | uint 8 | 0xCC |
| number_unsigned | 256..65535 | uint 16 | 0xCD |
| number_unsigned | 65536..4294967295 | uint 32 | 0xCE |
| number_unsigned | 4294967296..18446744073709551615 | uint 64 | 0xCF |
| number_float | *any value representable by a float* | float 32 | 0xCA |
| number_float | *any value NOT representable by a float* | float 64 | 0xCB |
| string | *length*: 0..31 | fixstr | 0xA0..0xBF |
| string | *length*: 32..255 | str 8 | 0xD9 |
| string | *length*: 256..65535 | str 16 | 0xDA |
| string | *length*: 65536..4294967295 | str 32 | 0xDB |
| array | *size*: 0..15 | fixarray | 0x90..0x9F |
| array | *size*: 16..65535 | array 16 | 0xDC |
| array | *size*: 65536..4294967295 | array 32 | 0xDD |
| object | *size*: 0..15 | fix map | 0x80..0x8F |
| object | *size*: 16..65535 | map 16 | 0xDE |
| object | *size*: 65536..4294967295 | map 32 | 0xDF |
| binary | *size*: 0..255 | bin 8 | 0xC4 |
| binary | *size*: 256..65535 | bin 16 | 0xC5 |
| binary | *size*: 65536..4294967295 | bin 32 | 0xC6 |
Complete mapping
The mapping is **complete** in the sense that any JSON value type can be converted to a MessagePack value.
Any MessagePack output created by `to_msgpack` can be successfully parsed by `from_msgpack`.
Size constraints
The following values can **not** be converted to a MessagePack value:
- strings with more than 4294967295 bytes
- byte strings with more than 4294967295 bytes
- arrays with more than 4294967295 elements
- objects with more than 4294967295 elements
NaN/infinity handling
If NaN or Infinity are stored inside a JSON number, they are serialized properly in contrast to the [dump](https://json.nlohmann.me/api/basic_json/dump/index.md) function which serializes NaN or Infinity to `null`.
Example
```
#include <iostream>
#include <iomanip>
#include <nlohmann/json.hpp>
using json = nlohmann::json;
using namespace nlohmann::literals;
int main()
{
// create a JSON value
json j = R"({"compact": true, "schema": 0})"_json;
// serialize it to MessagePack
std::vector<std::uint8_t> v = json::to_msgpack(j);
// print the vector content
for (auto& byte : v)
{
std::cout << "0x" << std::hex << std::setw(2) << std::setfill('0') << (int)byte << " ";
}
std::cout << std::endl;
}
```
Output:
```
0x82 0xa7 0x63 0x6f 0x6d 0x70 0x61 0x63 0x74 0xc3 0xa6 0x73 0x63 0x68 0x65 0x6d 0x61 0x00
```
## Deserialization
The library maps MessagePack types to JSON value types as follows:
| MessagePack type | JSON value type | first byte |
| ---------------- | --------------- | ---------- |
| positive fixint | number_unsigned | 0x00..0x7F |
| fixmap | object | 0x80..0x8F |
| fixarray | array | 0x90..0x9F |
| fixstr | string | 0xA0..0xBF |
| nil | `null` | 0xC0 |
| false | `false` | 0xC2 |
| true | `true` | 0xC3 |
| float 32 | number_float | 0xCA |
| float 64 | number_float | 0xCB |
| uint 8 | number_unsigned | 0xCC |
| uint 16 | number_unsigned | 0xCD |
| uint 32 | number_unsigned | 0xCE |
| uint 64 | number_unsigned | 0xCF |
| int 8 | number_integer | 0xD0 |
| int 16 | number_integer | 0xD1 |
| int 32 | number_integer | 0xD2 |
| int 64 | number_integer | 0xD3 |
| str 8 | string | 0xD9 |
| str 16 | string | 0xDA |
| str 32 | string | 0xDB |
| array 16 | array | 0xDC |
| array 32 | array | 0xDD |
| map 16 | object | 0xDE |
| map 32 | object | 0xDF |
| bin 8 | binary | 0xC4 |
| bin 16 | binary | 0xC5 |
| bin 32 | binary | 0xC6 |
| ext 8 | binary | 0xC7 |
| ext 16 | binary | 0xC8 |
| ext 32 | binary | 0xC9 |
| fixext 1 | binary | 0xD4 |
| fixext 2 | binary | 0xD5 |
| fixext 4 | binary | 0xD6 |
| fixext 8 | binary | 0xD7 |
| fixext 16 | binary | 0xD8 |
| negative fixint | number_integer | 0xE0-0xFF |
Info
Any MessagePack output created by `to_msgpack` can be successfully parsed by `from_msgpack`.
Example
```
#include <iostream>
#include <iomanip>
#include <nlohmann/json.hpp>
using json = nlohmann::json;
int main()
{
// create byte vector
std::vector<std::uint8_t> v = {0x82, 0xa7, 0x63, 0x6f, 0x6d, 0x70, 0x61, 0x63,
0x74, 0xc3, 0xa6, 0x73, 0x63, 0x68, 0x65, 0x6d,
0x61, 0x00
};
// deserialize it with MessagePack
json j = json::from_msgpack(v);
// print the deserialized JSON value
std::cout << std::setw(2) << j << std::endl;
}
```
Output:
```
{
"compact": true,
"schema": 0
}
```
features/binary_formats/ubjson.md
+126
View File
@@ -0,0 +1,126 @@
# UBJSON
Universal Binary JSON (UBJSON) is a binary form directly imitating JSON, but requiring fewer bytes of data. It aims to
achieve the generality of JSON, combined with being much easier to process than JSON.
!!! abstract "References"
- [UBJSON Website](http://ubjson.org)
## Serialization
The library uses the following mapping from JSON values types to UBJSON types according to the UBJSON specification:
| JSON value type | value/range | UBJSON type | marker |
|-----------------|-----------------------------------|----------------|--------|
| null | `null` | null | `Z` |
| boolean | `true` | true | `T` |
| boolean | `false` | false | `F` |
| number_integer | -9223372036854775808..-2147483649 | int64 | `L` |
| number_integer | -2147483648..-32769 | int32 | `l` |
| number_integer | -32768..-129 | int16 | `I` |
| number_integer | -128..127 | int8 | `i` |
| number_integer | 128..255 | uint8 | `U` |
| number_integer | 256..32767 | int16 | `I` |
| number_integer | 32768..2147483647 | int32 | `l` |
| number_integer | 2147483648..9223372036854775807 | int64 | `L` |
| number_unsigned | 0..127 | int8 | `i` |
| number_unsigned | 128..255 | uint8 | `U` |
| number_unsigned | 256..32767 | int16 | `I` |
| number_unsigned | 32768..2147483647 | int32 | `l` |
| number_unsigned | 2147483648..9223372036854775807 | int64 | `L` |
| number_unsigned | 2147483649..18446744073709551615 | high-precision | `H` |
| number_float | *any value* | float64 | `D` |
| string | *with shortest length indicator* | string | `S` |
| array | *see notes on optimized format* | array | `[` |
| object | *see notes on optimized format* | map | `{` |
!!! success "Complete mapping"
The mapping is **complete** in the sense that any JSON value type can be converted to a UBJSON value.
Any UBJSON output created by `to_ubjson` can be successfully parsed by `from_ubjson`.
!!! warning "Size constraints"
The following values can **not** be converted to a UBJSON value:
- strings with more than 9223372036854775807 bytes (theoretical)
!!! info "Unused UBJSON markers"
The following markers are not used in the conversion:
- `Z`: no-op values are not created.
- `C`: single-byte strings are serialized with `S` markers.
!!! info "NaN/infinity handling"
If NaN or Infinity are stored inside a JSON number, they are serialized properly. This behavior differs from the
`dump()` function which serializes NaN or Infinity to `null`.
!!! info "Optimized formats"
The optimized formats for containers are supported: Parameter `use_size` adds size information to the beginning of a
container and removes the closing marker. Parameter `use_type` further checks whether all elements of a container
have the same type and adds the type marker to the beginning of the container. The `use_type` parameter must only be
used together with `use_size = true`.
Note that `use_size = true` alone may result in larger representations - the benefit of this parameter is that the
receiving side is immediately informed on the number of elements of the container.
!!! info "Binary values"
If the JSON data contains the binary type, the value stored is a list of integers, as suggested by the UBJSON
documentation. In particular, this means that serialization and the deserialization of a JSON containing binary
values into UBJSON and back will result in a different JSON object.
??? example
```cpp
--8<-- "examples/to_ubjson.cpp"
```
Output:
```c
--8<-- "examples/to_ubjson.output"
```
## Deserialization
The library maps UBJSON types to JSON value types as follows:
| UBJSON type | JSON value type | marker |
|-------------|-----------------------------------------|--------|
| no-op | *no value, next value is read* | `N` |
| null | `null` | `Z` |
| false | `false` | `F` |
| true | `true` | `T` |
| float32 | number_float | `d` |
| float64 | number_float | `D` |
| uint8 | number_unsigned | `U` |
| int8 | number_integer | `i` |
| int16 | number_integer | `I` |
| int32 | number_integer | `l` |
| int64 | number_integer | `L` |
| string | string | `S` |
| char | string | `C` |
| array | array (optimized values are supported) | `[` |
| object | object (optimized values are supported) | `{` |
!!! success "Complete mapping"
The mapping is **complete** in the sense that any UBJSON value can be converted to a JSON value.
??? example
```cpp
--8<-- "examples/from_ubjson.cpp"
```
Output:
```json
--8<-- "examples/from_ubjson.output"
```
features/binary_formats/ubjson/index.html
+1 -1
View File
File diff suppressed because one or more lines are too long
features/binary_formats/ubjson/index.md
+206
View File
@@ -0,0 +1,206 @@
# UBJSON
Universal Binary JSON (UBJSON) is a binary form directly imitating JSON, but requiring fewer bytes of data. It aims to achieve the generality of JSON, combined with being much easier to process than JSON.
References
- [UBJSON Website](http://ubjson.org)
## Serialization
The library uses the following mapping from JSON values types to UBJSON types according to the UBJSON specification:
| JSON value type | value/range | UBJSON type | marker |
| --------------- | --------------------------------- | -------------- | ------ |
| null | `null` | null | `Z` |
| boolean | `true` | true | `T` |
| boolean | `false` | false | `F` |
| number_integer | -9223372036854775808..-2147483649 | int64 | `L` |
| number_integer | -2147483648..-32769 | int32 | `l` |
| number_integer | -32768..-129 | int16 | `I` |
| number_integer | -128..127 | int8 | `i` |
| number_integer | 128..255 | uint8 | `U` |
| number_integer | 256..32767 | int16 | `I` |
| number_integer | 32768..2147483647 | int32 | `l` |
| number_integer | 2147483648..9223372036854775807 | int64 | `L` |
| number_unsigned | 0..127 | int8 | `i` |
| number_unsigned | 128..255 | uint8 | `U` |
| number_unsigned | 256..32767 | int16 | `I` |
| number_unsigned | 32768..2147483647 | int32 | `l` |
| number_unsigned | 2147483648..9223372036854775807 | int64 | `L` |
| number_unsigned | 2147483649..18446744073709551615 | high-precision | `H` |
| number_float | *any value* | float64 | `D` |
| string | *with shortest length indicator* | string | `S` |
| array | *see notes on optimized format* | array | `[` |
| object | *see notes on optimized format* | map | `{` |
Complete mapping
The mapping is **complete** in the sense that any JSON value type can be converted to a UBJSON value.
Any UBJSON output created by `to_ubjson` can be successfully parsed by `from_ubjson`.
Size constraints
The following values can **not** be converted to a UBJSON value:
- strings with more than 9223372036854775807 bytes (theoretical)
Unused UBJSON markers
The following markers are not used in the conversion:
- `Z`: no-op values are not created.
- `C`: single-byte strings are serialized with `S` markers.
NaN/infinity handling
If NaN or Infinity are stored inside a JSON number, they are serialized properly. This behavior differs from the `dump()` function which serializes NaN or Infinity to `null`.
Optimized formats
The optimized formats for containers are supported: Parameter `use_size` adds size information to the beginning of a container and removes the closing marker. Parameter `use_type` further checks whether all elements of a container have the same type and adds the type marker to the beginning of the container. The `use_type` parameter must only be used together with `use_size = true`.
Note that `use_size = true` alone may result in larger representations - the benefit of this parameter is that the receiving side is immediately informed on the number of elements of the container.
Binary values
If the JSON data contains the binary type, the value stored is a list of integers, as suggested by the UBJSON documentation. In particular, this means that serialization and the deserialization of a JSON containing binary values into UBJSON and back will result in a different JSON object.
Example
```
#include <iostream>
#include <iomanip>
#include <nlohmann/json.hpp>
using json = nlohmann::json;
using namespace nlohmann::literals;
// function to print UBJSON's diagnostic format
void print_byte(uint8_t byte)
{
if (32 < byte and byte < 128)
{
std::cout << (char)byte;
}
else
{
std::cout << (int)byte;
}
}
int main()
{
// create a JSON value
json j = R"({"compact": true, "schema": false})"_json;
// serialize it to UBJSON
std::vector<std::uint8_t> v = json::to_ubjson(j);
// print the vector content
for (auto& byte : v)
{
print_byte(byte);
}
std::cout << std::endl;
// create an array of numbers
json array = {1, 2, 3, 4, 5, 6, 7, 8};
// serialize it to UBJSON using default representation
std::vector<std::uint8_t> v_array = json::to_ubjson(array);
// serialize it to UBJSON using size optimization
std::vector<std::uint8_t> v_array_size = json::to_ubjson(array, true);
// serialize it to UBJSON using type optimization
std::vector<std::uint8_t> v_array_size_and_type = json::to_ubjson(array, true, true);
// print the vector contents
for (auto& byte : v_array)
{
print_byte(byte);
}
std::cout << std::endl;
for (auto& byte : v_array_size)
{
print_byte(byte);
}
std::cout << std::endl;
for (auto& byte : v_array_size_and_type)
{
print_byte(byte);
}
std::cout << std::endl;
}
```
Output:
```
{i7compactTi6schemaF}
[i1i2i3i4i5i6i7i8]
[#i8i1i2i3i4i5i6i7i8
[$i#i812345678
```
## Deserialization
The library maps UBJSON types to JSON value types as follows:
| UBJSON type | JSON value type | marker |
| ----------- | --------------------------------------- | ------ |
| no-op | *no value, next value is read* | `N` |
| null | `null` | `Z` |
| false | `false` | `F` |
| true | `true` | `T` |
| float32 | number_float | `d` |
| float64 | number_float | `D` |
| uint8 | number_unsigned | `U` |
| int8 | number_integer | `i` |
| int16 | number_integer | `I` |
| int32 | number_integer | `l` |
| int64 | number_integer | `L` |
| string | string | `S` |
| char | string | `C` |
| array | array (optimized values are supported) | `[` |
| object | object (optimized values are supported) | `{` |
Complete mapping
The mapping is **complete** in the sense that any UBJSON value can be converted to a JSON value.
Example
```
#include <iostream>
#include <iomanip>
#include <nlohmann/json.hpp>
using json = nlohmann::json;
int main()
{
// create byte vector
std::vector<std::uint8_t> v = {0x7B, 0x69, 0x07, 0x63, 0x6F, 0x6D, 0x70, 0x61,
0x63, 0x74, 0x54, 0x69, 0x06, 0x73, 0x63, 0x68,
0x65, 0x6D, 0x61, 0x69, 0x00, 0x7D
};
// deserialize it with UBJSON
json j = json::from_ubjson(v);
// print the deserialized JSON value
std::cout << std::setw(2) << j << std::endl;
}
```
Output:
```
{
"compact": true,
"schema": 0
}
```