Bound what the DOM parsers reserve for arrays that announce a size (#6082)

* Bound the total number of valueless UBJSON elements per input

An optimized UBJSON array of type Z, T or F has no payload, so its
declared count is not bounded by the input. The limit of 1048576 applied
to each array on its own, so 9 bytes (`[$Z#l` and a count) produced
1048576 values and every further 9 bytes another 1048576: 362 bytes
allocated 661 MB in a DOM, and a SAX handler received 1M events per 9
bytes.

The reader now adds the counts of all such arrays of one input and
rejects the input with out_of_range.408 when the sum exceeds 1048576
plus 4 for every byte read so far. The base is the per-array limit, so
an array that was accepted on its own before is still accepted at the
start of an input. These arrays add about 80 bytes of memory per input
byte (measured 77; one `Z` per byte needs 32), plus the base; the memory
of a document as a whole is not bounded by this. The check runs once per
valueless array, before its start event, with plain size_t arithmetic
that cannot overflow for a 32-bit size_t. Containers with a payload type
and objects (every member needs a key) are bounded by the input already
and are unchanged. BJData rejects these types.

to_ubjson counts the elements of the typed arrays it writes in one call
and writes an array in the unoptimized form (`[#n Z Z ...`) once it would
bring the total above 1048576, which a reader accepts at any input size.
The check is made only for a typed array of a valueless type, so BJData
and other arrays do not pay for it. Everything to_ubjson writes in
version 3.13.0 can therefore be read back.

Two consequences:
- to_ubjson output changes for values with more than 1048576 elements in
  arrays of type Z, T or F: such an array is written at one byte per
  element, e.g. 270000 rows of 100 nulls take 26.8 MB instead of 1.35 MB.
- Output of version 3.12.0 and earlier (typed arrays without such a
  total) is rejected by from_ubjson when the arrays hold more than
  1048576 elements plus 4 per byte of input. 3.12.0 writes a row of up to 255 nulls in 5 bytes (6 above
  255), so a table is rejected from 1075 rows of 1000 nulls or 209723
  rows of 25 (accepted: 1000 rows of 1000, 209000 rows of 25); rows of
  20 or fewer are never rejected.

Measured on the test data: no file of json_test_data (converted with
to_ubjson in all three modes, 846 round trips) holds an array of this
kind with more than 0.17 values per byte. All of them are read back.

Reproducer (K arrays of 1048576 nulls), Apple clang -O2:
  K=40 (362 bytes): before 41943040 values, 664 MB; after rejected, 33 MB
  K=2 (20 bytes):   before 2097152 values, 53 MB;   after rejected, 33 MB

The new tests count events with a SAX handler where they can: building
and comparing million-element values made the first version take 176 s
under Valgrind, the current one about 22 s.

Behaviour change: an input that holds more than 1048576 elements in
arrays of type Z, T or F at more than 4 per byte is now rejected.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Report an empty array for UBJSON `[$N#<n>` instead of n elements

A no-op (N) is not a value, so an optimized array of type N holds no
element whatever count it declares. The reader skipped the elements but
still passed the declared count to the SAX parser: a handler was told
start_array(n) and no element followed, and the DOM parser reserved
min(n, 16384) values (256 KB) for an array that stays empty. Nine bytes
(`[$N#l 0x7FFFFFFF`) therefore committed 256 KB: 2000 of them in one
input, 18 KB, committed 512 MB of heap (capacity 163,840,000 values for
the 10000 arrays of the original probe: 2.6 GB for 70 KB of input).
Untouched pages keep the resident size small, but the commit charge and
a 32-bit address space are consumed.

The reader now announces 0 elements for an optimized array of type N.
The loop that skips the elements is unchanged (it never ran). A count
that does not fit into std::size_t is still rejected by the count
reader; a count that is too large for the array type of the DOM (before:
out_of_range.408 "excessive array size" from the DOM handler) is
accepted now, because the array is empty.

Behaviour change: sax_parse reports start_array(0) for `[$N#n` (before:
n); from_ubjson no longer throws out_of_range.408 for an N-typed array
whose count exceeds the maximum size of the array type.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Bound what the DOM parsers reserve for arrays that announce a size

Cause: #5476 made the DOM parsers reserve storage (up to 16384 elements)
for a counted array of CBOR, MessagePack, UBJSON and BJData when it
opens, before any input backs the count. The arrays that are open at the
same time keep their storage until the parse ends, so N nested headers
that announce 16384 elements and are never completed hold
N * 16384 * sizeof(json): 100,000 nested CBOR headers `99 40 00`
(300,000 bytes) peaked at 1.65 GiB, 3.12.0 at 12 MiB; the same for
MessagePack `DC 40 00`, UBJSON `[#I 16384` and BJData.

Fix: json_sax_dom_error_state, the base of the three DOM parsers, keeps a
credit of elements that may still be reserved: 65,536 at the start plus
2 for every byte the reader has read. An array that announces no more
than the credit left is reserved as before (one comparison and one
subtraction per counted array); a larger one is reserved what is left
and grows as its elements arrive. binary_reader tells a DOM parser (a
handler whose pointer converts to json_sax_dom_error_state*; a handler
that derives from one privately or protectedly, or any other SAX handler,
is not told anything and reserves as in the base of this change) where to
find its count of characters read for the duration of sax_parse(). An
RAII guard puts back the previous binding on every exit, also by
exception, so the parser never keeps a pointer to a reader that is gone
(unbound, only the per-array cap of 16384 applies); the public SAX
interface is unchanged, and start_array() still reports the announced number. A
bound per array by the remaining input would not help, since the same
bytes would back every open level; the bound has to be over all arrays.

The elements of an array take at least one byte of input each, so the
credit is more than honest input needs: none of the 19 inputs measured
(canada, citm_catalog, twitter, mesh, update-center, random, numbers,
animation, ik_rig, instruments, geo_response, jeopardy and 7 synthetic
ones, each as CBOR, MessagePack, UBJSON and BJData, with and without
size/type optimization) was given less than it announced. A value that
nests five or more arrays of 16384 elements, each with an array as its
first element, is the exception: the later ones grow as before.

Measurements (clang -O2, macOS arm64, peak memory footprint of
`/usr/bin/time -l`, reproducer: n nested headers):

              n = 20,000                n = 100,000
              develop   fix   3.12.0   develop    fix   3.12.0
  CBOR        338.7    5.9    8.2      1687.7    23.3   11.8   MiB
  MessagePack 338.9    6.0    7.3      1687.9    23.8   11.9   MiB
  UBJSON      339.0    7.6    8.0      1687.9    30.1   11.6   MiB
  BJData      339.0    7.3    7.9      1688.0    30.8   11.6   MiB

That is at most (65,536 + 2 per input byte) values: 1 MiB plus 32 bytes
per input byte with 16-byte values (a value is 24 bytes with
JSON_DIAGNOSTICS and up to 40 with JSON_DIAGNOSTIC_POSITIONS), as
documented in untrusted_input.md. Lower fixed caps do not get there: a
cap of 64 elements still peaks at 113 MiB (n = 100,000) and costs 4.7 %
on arrays of 300 elements (mat300.cbor) and 5.0 % on arrays of 10,000
(arrays_of_big.cbor).

Reader instructions retired (min of 7, 108 rows: 18 inputs in the 6 forms
CBOR, MessagePack, UBJSON and BJData plain and optimized; jeopardy, 46 MB,
is within 0.1 %), compared with the base of this change: -0.19 % to +0.37 % in the formats that announce
sizes (canada.cbor 81.9 M -> 82.1 M, many_small.cbor 333.5 M -> 334.2 M);
the benefit of #5476 is kept (without any reserve canada.cbor takes
116.8 M, +42.6 %, many_small.cbor +71.5 %, mat300.cbor +81.6 %). Rows of
plain UBJSON and BJData, whose arrays announce no size, move by -2.7 % to
+2.6 % without any code of theirs changing: the base alternates
between two instruction counts per process (452 M and 463 M for
many_pairs.ubjson, equal in cycles), and a struct with three unused
members added to the same class reproduces the 474 M mode.

Tests (CBOR, MessagePack, UBJSON, BJData): 200 nested headers that
announce 16384 elements peak at 52,433,600 bytes of DOM allocations on
the base and below 128 Ki values with the fix (scaled by sizeof(value),
so that it holds with the diagnostic macros) (counting allocator, no
process memory measured); an array that completes is still reserved in
one allocation; CBOR also checks that a DOM parser is not bound to a finished
parse (ASan: stack-use-after-scope without the guard), that wrappers
derived publicly (also with a parse inside the parse), privately and
protectedly compile and are bound or not as described, that an array that got less than it
announced decodes and that a SAX handler is told the announced size.
The new tests take 4.6 s under Valgrind at -O0 (gcc:12, all four files).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Clarify the reservation note for announced array counts

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Report an empty array for UBJSON [$N#<n>] instead of n elements (#6079)

* Report an empty array for UBJSON `[$N#<n>` instead of n elements

A no-op (N) is not a value, so an optimized array of type N holds no
element whatever count it declares. The reader skipped the elements but
still passed the declared count to the SAX parser: a handler was told
start_array(n) and no element followed, and the DOM parser reserved
min(n, 16384) values (256 KB) for an array that stays empty. Nine bytes
(`[$N#l 0x7FFFFFFF`) therefore committed 256 KB: 2000 of them in one
input, 18 KB, committed 512 MB of heap (capacity 163,840,000 values for
the 10000 arrays of the original probe: 2.6 GB for 70 KB of input).
Untouched pages keep the resident size small, but the commit charge and
a 32-bit address space are consumed.

The reader now announces 0 elements for an optimized array of type N.
The loop that skips the elements is unchanged (it never ran). A count
that does not fit into std::size_t is still rejected by the count
reader; a count that is too large for the array type of the DOM (before:
out_of_range.408 "excessive array size" from the DOM handler) is
accepted now, because the array is empty.

Behaviour change: sax_parse reports start_array(0) for `[$N#n` (before:
n); from_ubjson no longer throws out_of_range.408 for an N-typed array
whose count exceeds the maximum size of the array type.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Clarify the reservation note for announced array counts

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Add JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE to set the UBJSON whole-input budget (#6083)

* Add JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE to set the UBJSON whole-input budget

The whole-input budget for the elements of optimized UBJSON arrays of
type Z, T and F (1048576 plus 4 per byte read) rejects files written by
3.12.0 that have long arrays of these types, such as 1100 rows of 1000
nulls with a `$Z` annotation, and there was no way to read them. The
per-byte rate is now the macro JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE.

Semantics:
- default 4, so the default build is unchanged (the generated code of
  from_ubjson is identical at -O2 up to source line numbers);
- the value must be a positive integer: 0, negative, non-integral, bool,
  character and wider-than-uintmax_t values fail to compile with a
  static_assert that names the macro;
- a value that does not fit into std::size_t is the largest size_t, so
  one spelling works for 32- and 64-bit targets;
- the limit base + rate * bytes_read saturates at the largest size_t
  instead of wrapping around, for any rate and input size; a rate of
  131072 (2^20 / 8) or more cannot be exceeded by any input (the
  shortest header of an array of more than 32767 elements is 8 bytes),
  which switches the whole-input budget off;
- the limit of one array (1048576) and the base (1048576) stay fixed
  constants, so the writer fallback still follows the base only and a
  reader with any rate reads everything the writer produces;
- the macro is not part of the ABI tag and must have the same value in
  every translation unit; it is documented that way.

The memory the arrays can take in a DOM is about 20 bytes per rate and
byte of input (measured, macOS, containers of 9 * rate elements:
25 bytes per input byte at rate 1, 81 at 4, 312 at 16, 1226 at 64, 5493
at 256) plus about 33 MB for the base. A 3.12.0 row of N values has a
header of b bytes (5 up to 255, 6 up to 32767, 8 above), so a table is
read completely if N <= rate * b, and otherwise up to
(1048576 + h * rate, h = outer header: 6, 7 or 9 bytes) / (N - rate * b) rows; 1100 rows of 1000 nulls need
a rate of 8. The new page of the macro documents the formulas, the
measurements, and two examples that are run by the documentation checks.

The tests of the budget move to tests/src/unit-ubjson_valueless.cpp, which
derives every boundary from the macro and is built with the rates 1,
1000000 and 0xFFFFFFFFFFFFFFFF in addition to the default (the last two
check that nothing is refused and that the limit does not wrap around);
they fail with the unchanged header for rate 1, 1000000 and the maximum.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Refuse char8_t for JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE and clarify its documentation

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Use fseek instead of rewind in the UBJSON FILE* test

clang-tidy (bugprone-unsafe-functions) rejects rewind() because it has no
error detection. Check the result of fseek() instead.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Charge an announced array only what is reserved for it

An array that announced more than reserve_cap() elements but no more than
the credit left was charged the announced count, while reserve_array()
reserved at most reserve_cap() of them. An outer array of 20000 elements
therefore left a third nested array of 16384 with 12796 instead of the
full reservation. reserve_count() now caps the count first and charges
that, and the DOM parsers reserve it through reserve_array_capped(),
which does not cap again, so the fast path costs the same as before
(instructions retired within +-0.06 % on canada, citm_catalog and twitter
in CBOR, MessagePack and UBJSON).

Also re-wrap the comment of valueless_elements_base.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
This commit is contained in:
Niels Lohmann authored and GitHub committed 2026-11-15 17:02:05 +01:00
1 parent 73ec296320
commit 253e4c3926
28 files changed
+2068 -295

No files matched your search

@@ -91,8 +91,9 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va
- Throws [out_of_range.408](../../home/exceptions.md#jsonexceptionout_of_range408) if the count of an optimized
container exceeds the capacity of the container (or of `std::size_t`; an array of type `N` is empty, so only the
latter applies to it), if an optimized array of `Z`, `T`, or `F` has more than 1,048,576 elements, or if the
optimized arrays of `Z`, `T`, and `F` in the input together have more than 1,048,576 elements plus 4 for every byte
read so far
optimized arrays of `Z`, `T`, and `F` in the input together have more than 1,048,576 elements plus
[`JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE`](../macros/json_ubjson_valueless_elements_per_byte.md) (4 by default) for
every byte read so far
## Complexity
@@ -121,6 +122,8 @@ Linear in the size of the input.
- [Parsing untrusted input](../../features/parsing/untrusted_input.md) - limits and safe handling of hostile data
- [Binary values](../../features/binary_values.md) - how binary values and subtypes are handled
- [`to_ubjson`](to_ubjson.md) - create a UBJSON serialization of a JSON value
- [`JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE`](../macros/json_ubjson_valueless_elements_per_byte.md) -
set how many elements of arrays of `null`, `true`, or `false` the input may announce per byte
- [`from_cbor`](from_cbor.md) - create a JSON value from an input in CBOR format
- [`from_msgpack`](from_msgpack.md) - create a JSON value from an input in MessagePack format
- [`from_bson`](from_bson.md) - create a JSON value from an input in BSON format
@@ -136,7 +139,10 @@ Linear in the size of the input.
- Added `error_handler` parameter in version 3.13.0.
- Report `start_array(0)` to a SAX handler for an optimized array of type `N` (`[$N#n`), whatever its count, in version
3.13.0.
- Limited the total number of elements of optimized arrays of `Z`, `T`, and `F` in one input in version 3.13.0.
- Limited the total number of elements of optimized arrays of `Z`, `T`, and `F` in one input in version 3.13.0; the
number of elements per byte can be set with
[`JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE`](../macros/json_ubjson_valueless_elements_per_byte.md), for example to read
files written by version 3.12.0 and earlier that exceed the default.
!!! warning "Deprecation"
+5 -1
View File
@@ -105,6 +105,8 @@ Linear in the size of the JSON value `j`.
- [Binary formats](../../features/binary_formats/index.md) - overview and comparison of the supported binary formats
- [Binary values](../../features/binary_values.md) - how binary values and subtypes are handled
- [`from_ubjson`](from_ubjson.md) - create a JSON value from an input in UBJSON format
- [`JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE`](../macros/json_ubjson_valueless_elements_per_byte.md) -
set how many elements of arrays of `null`, `true`, or `false` a reader accepts per byte of input
- [`to_cbor`](to_cbor.md) - create a CBOR serialization of a JSON value
- [`to_msgpack`](to_msgpack.md) - create a MessagePack serialization of a JSON value
- [`to_bson`](to_bson.md) - create a BSON serialization of a JSON value
@@ -119,6 +121,8 @@ Linear in the size of the JSON value `j`.
[`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled) throws `type_error.316`.
- Writes arrays of `Z`, `T`, or `F` without the `$` type annotation (`[#n Z Z ...`) once their elements in one call
add up to more than 1,048,576, in version 3.13.0, so that `from_ubjson` can read the output back. Such output is
larger: 270,000 rows of 100 `null` values take 26.8 MB instead of 1.35 MB.
larger: 270,000 rows of 100 `null` values take 26.8 MB instead of 1.35 MB. The limit does not depend on
[`JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE`](../macros/json_ubjson_valueless_elements_per_byte.md): a reader accepts at
least 1,048,576 elements whatever it is set to, so the writer follows that base only.
- Throws `type_error.321` for a discarded value since version 3.13.0; previously, a discarded value nested in an
array or object was silently skipped, producing invalid UBJSON.
+2
View File
@@ -26,6 +26,8 @@ header. See also the [macro overview page](../../features/macros.md).
UBJSON, BJData, and BSON writers
- [**JSON_STRICT_NUL_HANDLING**](json_strict_nul_handling.md) - opt in to rejecting a NUL byte in the input instead of
treating it as end of input
- [**JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE**](json_ubjson_valueless_elements_per_byte.md) - set how many elements of
UBJSON arrays of `null`, `true`, or `false` an input may announce per byte it has
## Language support
@@ -0,0 +1,175 @@
# JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE
```cpp
#define JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE /* value */
```
[`from_ubjson`](../basic_json/from_ubjson.md) and [`sax_parse`](../basic_json/sax_parse.md) refuse an input in which the
optimized arrays of type `Z` (null), `T` (true), and `F` (false) together have more than 1,048,576 elements plus
`JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE` for every byte read so far, with
[`out_of_range.408`](../../home/exceptions.md#jsonexceptionout_of_range408). The macro sets that number. It is the
elements per byte that the input is allowed to announce without paying for them: an element of such an array is a
single marker that is implied by the array's `$Z#n` header, so a few bytes can announce a million values (see
[Parsing Untrusted Input](../../features/parsing/untrusted_input.md)).
The macro only matters for input that was not written by this library 3.13.0 or later:
[`to_ubjson`](../basic_json/to_ubjson.md) never writes more than the base of 1,048,576 elements in typed arrays of
these types, and a reader accepts at least that many whatever the macro is. Its main use is reading files written by
version 3.12.0 and earlier, which did not limit them (see
[below](#reading-files-written-by-version-3120-and-earlier)).
## Default definition
The default value is `4`.
```cpp
#define JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE 4
```
## Notes
!!! info "Allowed values"
The value must be a **positive integer** (a value of `0` would not allow any element beyond the base, and a negative
or fractional rate has no meaning). Any other value is a compile-time error with a message that names the macro:
```
error: static assertion failed: JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE must be a positive integer
error: static assertion failed: JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE must be an integer (not bool or a character type)
error: static assertion failed: JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE must not be wider than std::uintmax_t
```
The first is reported for `0` and negative values, the second for values such as `4.5`, `1e3`, `true`, or a
character of any character type (`'a'`, `L'a'`, `u8'a'`, ...), and the third for an integer type that is wider
than `#!cpp std::uintmax_t` (for example `unsigned __int128`). A value that is larger than the largest `#!cpp
std::size_t` is treated as that largest value, so that a project can use one spelling, for example
`0xFFFFFFFFFFFFFFFFu`, for 32-bit and 64-bit targets.
!!! info "Large values: the limit is switched off"
The limit is the base plus the rate times the bytes read. It saturates at the largest `#!cpp std::size_t` instead of
overflowing, for any rate and any input size (also with a 32-bit `#!cpp std::size_t`), so a large value never makes
the limit smaller. A rate of **at least 131,072** (`1048576 / 8`) effectively switches the limit for the whole
input off: the shortest header of an array of more than 32,767 elements takes 8 bytes (`$Z#l` and four bytes of the
count, inside an optimized array of arrays; 9 bytes, `[$Z#l`, at the top level), and a shorter header (`$Z#I` and
two bytes) can announce at most 32,767. No array can then announce more than the rate times the bytes of its own
header. With a rate of 131,071 that is not so: rows of 1,048,576 elements in 8 bytes each gain 8 elements per row,
and a table of 278,527 such rows (2.2 MB) is the shortest input that is refused.
!!! info "What the macro does not change"
- The limit of **one array** stays at 1,048,576 elements
([`out_of_range.408`](../../home/exceptions.md#jsonexceptionout_of_range408), "excessive array size"), whatever
the macro is. A value that switches the whole-input limit off does not allow a single array of 1,048,577
elements.
- The **base** of 1,048,576 elements is a fixed constant, the same as the limit of one array. An array that was
accepted on its own before the whole-input limit existed is therefore still accepted at the start of an input.
- The **writer** is not affected. [`to_ubjson`](../basic_json/to_ubjson.md) follows the base only: it writes a typed
array of `Z`, `T`, or `F` only while the arrays written so far and this one stay within 1,048,576 elements, and
writes the elements one by one otherwise. That is within the limit of every reader, because the limit is never
below the base.
- **Objects** and arrays of other types are not counted: every member of an object has a key, and every element of
another type takes at least a byte of input.
- [`from_bjdata`](../basic_json/from_bjdata.md) is not affected: BJData has no optimized arrays of these types.
!!! info "Memory"
A value takes 16 bytes in a DOM, so every element that the limit lets pass costs memory. Beyond the first 1,048,576
elements (about 33 MB: 16 bytes per element, and a vector that grows by reallocation needs the old and the new
storage at once), a malicious input of `n` bytes can take **about `20 * K * n` bytes** of memory in a DOM, where `K`
is the value of the macro. The measurements below read inputs that are at the limit, with containers of `9 * K`
elements (`[$Z#l` and a count, 9 bytes each), with `from_ubjson` on macOS, arm64, compiled with `-O2`. The numbers
on other systems differ by the allocator; the 16 bytes per element are the lower bound.
| `K` | bytes of memory per byte of input | memory for 1 MB of input |
|------:|----------------------------------:|-------------------------:|
| 1 | 25 | 25 MB |
| 4 | 81 | 81 MB |
| 16 | 312 | 312 MB |
| 64 | 1,226 | 1.2 GB |
| 256 | 5,493 | 5.5 GB |
A SAX handler receives the same number of events, and no memory beyond what it keeps. Choose the smallest value that
accepts your files, not a large one, if the input comes from an untrusted source: a value of 131,072 or more
switches the limit of the whole input off, so that every 8 bytes of input can announce another 1,048,576 values,
the limit of one array.
!!! warning "Define it before the include, in every translation unit"
This macro must be defined **before** including `<nlohmann/json.hpp>`. Defining it after the include has no effect.
The value is compiled into the member functions of the binary reader, which are inline, but it is **not**
encoded in the [namespace](../../features/namespace.md) of the library (there is no ABI tag for it). It must
have the same value in every translation unit of a program (for example by defining it on the command line or in
a common header); translation units with different values violate the One Definition Rule, which is undefined
behavior (no diagnostic is required). In practice, which value a read uses depends on which functions the compiler
inlined and which out-of-line copy of each remaining function the linker kept, so a function compiled with one
value can call a function compiled with the other, and a single parse can mix both values.
### Reading files written by version 3.12.0 and earlier
Version 3.12.0 wrote an array of `Z`, `T`, or `F` of any length in the optimized form, and a table of equal rows as an
optimized array of optimized arrays: a header of `[$[#` and the number of rows (`i` or `U` and 1 byte, `I` and 2 bytes, or
`l` and 4 bytes: `h` = 6, 7, or 9 bytes in all for up to 255, 32,767, or more rows), then for every row `$Z#` and its count:
5 bytes for up to 255 elements (`U`), 6 bytes for up to 32,767 (`I`), and 8 bytes above (`l`). Every row announces `N`
elements for the `b` bytes of its header, so the limit grows by `K * b` and the number of elements by `N` per row.
- A table is never refused if its rows have at most `K * b` elements each: at most 5 `K` for rows of up to 255
elements (20 at the default of `K = 4`), at most 6 `K` for rows of 256 to 32,767 (rows of 1,000 need `K >= 167`),
and at most 8 `K` above.
- Otherwise the first `(1,048,576 + h * K) / (N - K * b)` rows are accepted, rounded down (`h` as above; 7 for the
numbers here). For the default, a table of rows of 1,000 `null` values is read up to row 1,074 (the input of the first
example below has 1,100); for `K = 1`, up to row 1,054; for `K = 16`, up to row 1,160.
- In general, the smallest value that accepts an input is the largest of
`(the elements of the arrays so far - 1,048,576) / (the bytes read when the header of the last of them has been
read)` over all arrays, rounded up. For 1,100 rows of 1,000 elements that is
`(1,100,000 - 1,048,576) / 6,607 = 7.8`, so `K = 8`.
To read such a file, define the macro, before the include (or with `-DJSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE=8`), to
the smallest value that accepts the file, and see the table above for what that value allows an untrusted input to
cost. A file can be converted once: read it with the larger value and write it again with `to_ubjson`, which writes no
typed array that a reader with the default value would refuse, so that the result does not need the macro.
## Examples
??? example "Example: default limit"
The table of 1,100 rows of 1,000 `null` values (6,607 bytes) that version 3.12.0 wrote announces 1,100,000 values,
which is more than 1,048,576 plus 4 for every byte read.
```cpp
--8<-- "examples/ubjson_valueless_elements_per_byte.cpp"
```
Output:
```
--8<-- "examples/ubjson_valueless_elements_per_byte.output"
```
??? example "Example: raised limit"
With 16 elements per byte the same table is read. The limit of one array still applies.
```cpp
--8<-- "examples/ubjson_valueless_elements_per_byte_raised.cpp"
```
Output:
```
--8<-- "examples/ubjson_valueless_elements_per_byte_raised.output"
```
## See also
- [`from_ubjson`](../basic_json/from_ubjson.md) - create a JSON value from a UBJSON serialization
- [`to_ubjson`](../basic_json/to_ubjson.md) - create a UBJSON serialization of a JSON value
- [`sax_parse`](../basic_json/sax_parse.md) - generate SAX events
- [UBJSON](../../features/binary_formats/ubjson.md) - the format and its limits in the library
- [Parsing Untrusted Input](../../features/parsing/untrusted_input.md) - limits and checks for input you do not control
- [`out_of_range.408`](../../home/exceptions.md#jsonexceptionout_of_range408) - the exception that reports the limit
## Version history
- Added in version 3.13.0.
+2 -2
View File
@@ -50,8 +50,8 @@ flowchart LR
enabled explicitly. Invalid UTF-8 is rejected.
- **Errors are reported, not ignored.** Malformed input results in a [`parse_error`](../home/exceptions.md#parse-errors)
with the byte position of the error. Binary readers do not trust announced sizes: strings and binary values grow
only as bytes are actually read, arrays reserve at most a fixed number of elements up front, and sizes that no
container can hold are rejected.
only as bytes are actually read, arrays reserve at most a fixed number of elements up front (and all of them
together no more than the input read so far can back), and sizes that no container can hold are rejected.
- **Memory is owned by values.** Each `basic_json` value owns its content, and there is no manual memory management in
user code. The destructor recurses only over a bounded number of nesting levels and then continues without the call
stack, so destroying a deeply nested value does not exhaust the stack.
@@ -0,0 +1,34 @@
#include <cstdint>
#include <iostream>
#include <vector>
#include <nlohmann/json.hpp>
using json = nlohmann::json;
// the bytes version 3.12.0 wrote for a table of 1100 rows of 1000 null values
// with to_ubjson(table, true, true): `[$[#I` and the number of rows, then
// `$Z#I` and the number of elements for every row
std::vector<std::uint8_t> table_written_by_3_12_0()
{
std::vector<std::uint8_t> bytes = {'[', '$', '[', '#', 'I', 1100 >> 8, 1100 & 0xFF};
for (int row = 0; row < 1100; ++row)
{
bytes.insert(bytes.end(), {'$', 'Z', '#', 'I', 1000 >> 8, 1000 & 0xFF});
}
return bytes;
}
int main()
{
try
{
// 6607 bytes announce 1,100,000 values, more than 1,048,576 plus 4 for
// every byte read (the limit is exceeded in row 1075)
json table = json::from_ubjson(table_written_by_3_12_0());
std::cout << table.size() << " rows\n";
}
catch (const json::out_of_range& e)
{
std::cout << e.what() << '\n';
}
}
@@ -0,0 +1 @@
[json.exception.out_of_range.408] syntax error while parsing UBJSON size: excessive number of elements without payload
@@ -0,0 +1,42 @@
#include <cstdint>
#include <iostream>
#include <vector>
// accept 16 values per byte of input instead of 4; define the macro before the
// first inclusion of the header, and to the same value in every translation unit
#define JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE 16
#include <nlohmann/json.hpp>
using json = nlohmann::json;
// the bytes version 3.12.0 wrote for a table of 1100 rows of 1000 null values
// with to_ubjson(table, true, true): `[$[#I` and the number of rows, then
// `$Z#I` and the number of elements for every row
std::vector<std::uint8_t> table_written_by_3_12_0()
{
std::vector<std::uint8_t> bytes = {'[', '$', '[', '#', 'I', 1100 >> 8, 1100 & 0xFF};
for (int row = 0; row < 1100; ++row)
{
bytes.insert(bytes.end(), {'$', 'Z', '#', 'I', 1000 >> 8, 1000 & 0xFF});
}
return bytes;
}
int main()
{
// 6607 bytes announce 1,100,000 values, which is below 1,048,576 plus 16
// for every byte read
json table = json::from_ubjson(table_written_by_3_12_0());
std::cout << table.size() << " rows of " << table[0].size() << " values\n";
// a single container is still limited to 1,048,576 elements
const std::vector<std::uint8_t> too_long = {'[', '$', 'Z', '#', 'l', 0x00, 0x10, 0x00, 0x01};
try
{
json j = json::from_ubjson(too_long);
}
catch (const json::out_of_range& e)
{
std::cout << e.what() << '\n';
}
}
@@ -0,0 +1,2 @@
1100 rows of 1000 values
[json.exception.out_of_range.408] syntax error while parsing UBJSON size: excessive array size
@@ -119,17 +119,27 @@ The library uses the following mapping from JSON values types to UBJSON types ac
receiving side is immediately informed on the number of elements of the container.
An array whose type marker is `Z` (null), `T` (true) or `F` (false) stores no payload at all, because the marker
already is the value. Its declared count is therefore the only thing that decides how much memory the receiving side
allocates, and a handful of bytes can describe billions of elements. `from_ubjson` rejects such an array with
already is the value. Its declared count is therefore the only thing that decides how much memory the receiving
side allocates, and a handful of bytes can describe billions of elements. `from_ubjson` rejects such an array with
[`out_of_range.408`](../../home/exceptions.md#jsonexceptionout_of_range408) when the count exceeds 1,048,576
(`1 << 20`), or when the counts of all such arrays in the input add up to more than 1,048,576 plus 4 for every byte
read so far. `to_ubjson` therefore writes an array of these types in the counted but untyped form when it is longer
than that, or when it would bring the total of the typed arrays written so far above 1,048,576: the `$` type marker
is dropped, the `#` count is kept, and every element is written with its own marker (`[#l 1048577 Z Z ...`), so
everything it produces can be read back. Only input from another encoder, or written by version 3.12.0 and earlier of
this library, can exceed the limit: for example, 1,100 rows of 1,000 `null` values or 210,000 rows of 25, each with
a `$Z` annotation. Version 3.12.0 writes such a row in 5 bytes (6 above 255 elements), so rows of 20 or fewer such
values are never rejected. Objects and untyped counted arrays are not subject to this limit.
(`1 << 20`), or when the counts of all such arrays in the input add up to more than 1,048,576 plus _K_ for every byte
read so far, where _K_ is the value of the macro
[`JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE`](../../api/macros/json_ubjson_valueless_elements_per_byte.md) (4 by
default). `to_ubjson` therefore writes an array of these types in the counted but untyped form when it is longer
than that, or when it would bring the total of the typed arrays written so far above 1,048,576: the `$` type
marker is dropped, the `#` count is kept, and every element is written with its own marker
(`[#l 1048577 Z Z ...`), so everything it produces can be read back. Only input from another encoder, or written by version 3.12.0
and earlier of this library, can exceed the limit: for example, 1,100 rows of 1,000 `null` values or 210,000 rows
of 25, each with a `$Z` annotation. Version 3.12.0 writes a row in 5 bytes (6 above 255 elements, 8 above 32,767),
which raises the limit by 5 _K_ (6 _K_, 8 _K_), so rows of up to 5 _K_ values (20 at the default) are never
rejected. To read such a file, define the macro to a larger value (`8` is enough for the 1,100 rows of 1,000
`null` values; memory for the arrays then rises to about 20 _K_ bytes per byte of input in a DOM); the macro's
page explains how to choose a value and what it costs. The limit of one array of 1,048,576 elements does not
depend on the macro. Objects and untyped counted arrays are not subject to this limit.
An optimized array of type `N` (no-op) is empty whatever its count is, because a no-op is not a value. Its count is
not passed on: `sax_parse` reports `start_array(0)` (versions before 3.13.0 reported the declared count), and the DOM
parser reserves nothing for it.
An optimized array of type `N` (no-op) is empty whatever its count is, because a no-op is not a value. Its count is
not passed on: `sax_parse` reports `start_array(0)` (versions before 3.13.0 reported the declared count), and the DOM
@@ -189,7 +199,9 @@ The library maps UBJSON types to JSON value types as follows:
Every well-formed UBJSON value can be converted to a JSON value, with these exceptions: a high-precision number
outside the range of `number_float_t` ([`out_of_range.406`](../../home/exceptions.md#jsonexceptionout_of_range406)),
optimized arrays of `Z`, `T`, or `F` with more elements than the limit allows (1,048,576 plus 4 per byte read),
optimized arrays of `Z`, `T`, or `F` with more elements than the limit allows (1,048,576 plus
[`JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE`](../../api/macros/json_ubjson_valueless_elements_per_byte.md), 4 by default,
per byte read),
and a count that exceeds the capacity of the container
([`out_of_range.408`](../../home/exceptions.md#jsonexceptionout_of_range408)). High-precision numbers that do not
fit an integer type are stored as floating-point numbers and may lose precision. Ill-formed input is rejected with a
+10
View File
@@ -261,6 +261,16 @@ This macro overrides `#!cpp try` calls inside the library.
See [full documentation of `JSON_TRY_USER`](../api/macros/json_throw_user.md).
## `JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE`
Sets how many elements of optimized UBJSON arrays of `Z` (null), `T` (true), and `F` (false) a reader accepts for
every byte of input it has read, on top of a base of 1,048,576 elements. These arrays have no payload, so a few bytes
can announce a million values. The value must be a positive integer; the default is `4`, and a value of 131,072 or
more switches the limit of the whole input off. The limit of 1,048,576 elements for a single array always applies. The
macro is not part of the [ABI tag](namespace.md): use the same value in every translation unit.
See [full documentation of `JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE`](../api/macros/json_ubjson_valueless_elements_per_byte.md).
## `JSON_USE_IMPLICIT_CONVERSIONS`
When defined to `0`, implicit conversions are switched off. By default, implicit conversions are switched on.
@@ -107,11 +107,21 @@ For the binary formats, an announced size is never trusted outright:
actually consumed, rather than allocating the announced length up front — a truncated input runs out of bytes
(reported as a parse error) instead of triggering an oversized allocation.
- When an array announces its number of elements and the array container supports `reserve()` (as `#!cpp std::vector`,
the default, does), the library reserves storage for at most 16384 of them upfront, regardless of how large the
announced count is; further elements still grow the container normally as they are read. A truncated input whose
count is larger than its remaining bytes therefore reserves at most that much before the parse error is reported.
Every array element takes at least one byte of input, except in the [UBJSON](../binary_formats/ubjson.md) arrays of
type `Z`, `T` or `F` described below.
the default, does), the DOM parser reserves storage for them before the elements are read, but for at most 16384
elements per array, regardless of how large the announced count is; further elements still grow the container
normally as they are read. The arrays that are open at the same time hold their storage at the same time, and an
announcement is not backed by input when it is read, so the elements reserved for all arrays of one input together
may not exceed 65,536 plus 2 for every byte of input read so far; an array that announces more gets what is left
of that and grows as its elements arrive. An element takes at least one byte of input (except in the
[UBJSON](../binary_formats/ubjson.md) arrays of type `Z`, `T` or `F` described below), so the elements of the
arrays that were read use up what was reserved for them, and only arrays that are nested deeply, with an array as
their first element, can announce more than is left. For a truncated or hostile input, the storage that
announcements can hold is therefore at most (65,536 + 2 per byte of input) values, that is about 1 MiB plus 32 bytes
per byte of input with 16-byte values (a value is larger with `JSON_DIAGNOSTICS` or `JSON_DIAGNOSTIC_POSITIONS`, 24 to
40 bytes, and so is the bound). For example, 100,000 nested CBOR array headers `99 40 00` (300,000 bytes) that
announce 16384 elements each peaked at 23 MiB, against 1.65 GiB if every array reserved what it announced and 12 MiB
with version 3.12.0, which reserved nothing.
A [SAX](../../api/json_sax/index.md) handler is not affected: it is told the announced number of elements.
- An announced array or object size that exceeds what the target container could ever hold (its `max_size()`) is
rejected immediately as [`out_of_range.408`](../../home/exceptions.md#jsonexceptionout_of_range408), without
attempting to allocate anything.
@@ -119,11 +129,17 @@ For the binary formats, an announced size is never trusted outright:
`start_array(0)` and the DOM parser reserves nothing.
- A UBJSON array of type `Z`, `T` or `F` has no payload, so its announced count is not bounded by the input. One array
may announce at most 1,048,576 elements, and all such arrays of one input together may contain at most 1,048,576
elements plus 4 for every byte read so far (checked when each array's header has been read); beyond that the input is
rejected as [`out_of_range.408`](../../home/exceptions.md#jsonexceptionout_of_range408). For a DOM that bounds the memory these
arrays can need to about 80 bytes per byte of input plus about 33 MB for the first 1,048,576 elements; a SAX handler
receives at most that many events. Every other optimized array element, and every UBJSON object member (it has a
key), takes at least one byte of input.
elements plus _K_ for every byte read so far (checked when each array's header has been read), where _K_ is the
value of the macro
[`JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE`](../../api/macros/json_ubjson_valueless_elements_per_byte.md), 4 by
default; beyond that the input is rejected as
[`out_of_range.408`](../../home/exceptions.md#jsonexceptionout_of_range408). For a DOM that bounds the memory these
arrays can need to about 20 _K_ bytes per byte of input (81 at the default, measured; 312 for _K_ = 16) plus about
33 MB for the first 1,048,576 elements; a SAX handler receives at most that many events. Every other optimized array
element, and every UBJSON object member (it has a key), takes at least one byte of input. Leave the macro at the
default for input you do not control; raise it only to read trusted files that version 3.12.0 or earlier wrote with
long arrays of these types, and see its page for how a value is chosen and what it costs. The limit of one array
does not depend on the macro.
## Strings
+7 -3
View File
@@ -1054,13 +1054,17 @@ The exception is also thrown for [UBJSON](../features/binary_formats/ubjson.md)
its marker alone (`Z`, `T` or `F`). Such an array has no payload, so its count alone decides how much memory is
allocated, and a handful of bytes would otherwise describe billions of values. It is thrown if the declared count of one
array exceeds 1,048,576 (`1 << 20`), and if the counts of all such arrays read so far in one input add up to more than
1,048,576 plus 4 for every byte of the input read when the header of the last array has been read.
1,048,576 plus 4 for every byte of the input read when the header of the last array has been read. The 4 is the
default of the macro
[`JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE`](../api/macros/json_ubjson_valueless_elements_per_byte.md), which raises it
(for the limit of the whole input; the limit of one array stays).
[`to_ubjson`](../api/basic_json/to_ubjson.md) writes an array of these types without the `$` type annotation (the `#`
count and one marker per element are kept) when it is longer, or when it would bring the total of the typed arrays
written so far above 1,048,576, so everything it produces can be read back. Only input written by another encoder, or
by version 3.12.0 and earlier of this library, can exceed the limit: for example, 1,100 rows of 1,000 `null` values or
210,000 rows of 25, each with a `$Z` annotation. Version 3.12.0 writes such a row in 5 bytes (6 above 255 elements),
so rows of 20 or fewer such values are never rejected.
210,000 rows of 25, each with a `$Z` annotation. Version 3.12.0 writes such a row in 5 bytes (6 above 255 elements, 8
above 32,767), so rows of 20 or fewer such values are never rejected at the default; define the macro to a larger value
to read them.
!!! failure "Example messages"
+1
View File
@@ -459,6 +459,7 @@ nav:
- 'JSON_SKIP_UNSUPPORTED_COMPILER_CHECK': api/macros/json_skip_unsupported_compiler_check.md
- 'JSON_STRICT_BINARY_UTF8': api/macros/json_strict_binary_utf8.md
- 'JSON_STRICT_NUL_HANDLING': api/macros/json_strict_nul_handling.md
- 'JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE': api/macros/json_ubjson_valueless_elements_per_byte.md
- 'JSON_USE_GLOBAL_UDLS': api/macros/json_use_global_udls.md
- 'JSON_USE_IMPLICIT_CONVERSIONS': api/macros/json_use_implicit_conversions.md
- 'JSON_USE_LEGACY_DISCARDED_VALUE_COMPARISON': api/macros/json_use_legacy_discarded_value_comparison.md