Bound the total number of valueless UBJSON elements per input

An optimized UBJSON array of type Z, T or F has no payload, so its
declared count is not bounded by the input. The limit of 1048576 applied
to each array on its own, so 9 bytes (`[$Z#l` and a count) produced
1048576 values and every further 9 bytes another 1048576: 362 bytes
allocated 661 MB in a DOM, and a SAX handler received 1M events per 9
bytes.

The reader now adds the counts of all such arrays of one input and
rejects the input with out_of_range.408 when the sum exceeds 1048576
plus 4 for every byte read so far. The base is the per-array limit, so
an array that was accepted on its own before is still accepted at the
start of an input. These arrays add about 80 bytes of memory per input
byte (measured 77; one `Z` per byte needs 32), plus the base; the memory
of a document as a whole is not bounded by this. The check runs once per
valueless array, before its start event, with plain size_t arithmetic
that cannot overflow for a 32-bit size_t. Containers with a payload type
and objects (every member needs a key) are bounded by the input already
and are unchanged. BJData rejects these types.

to_ubjson counts the elements of the typed arrays it writes in one call
and writes an array in the unoptimized form (`[#n Z Z ...`) once it would
bring the total above 1048576, which a reader accepts at any input size.
The check is made only for a typed array of a valueless type, so BJData
and other arrays do not pay for it. Everything to_ubjson writes in
version 3.13.0 can therefore be read back.

Two consequences:
- to_ubjson output changes for values with more than 1048576 elements in
  arrays of type Z, T or F: such an array is written at one byte per
  element, e.g. 270000 rows of 100 nulls take 26.8 MB instead of 1.35 MB.
- Output of version 3.12.0 and earlier (typed arrays without such a
  total) is rejected by from_ubjson when the arrays hold more than
  1048576 elements plus 4 per byte of input. 3.12.0 writes a row of up to 255 nulls in 5 bytes (6 above
  255), so a table is rejected from 1075 rows of 1000 nulls or 209723
  rows of 25 (accepted: 1000 rows of 1000, 209000 rows of 25); rows of
  20 or fewer are never rejected.

Measured on the test data: no file of json_test_data (converted with
to_ubjson in all three modes, 846 round trips) holds an array of this
kind with more than 0.17 values per byte. All of them are read back.

Reproducer (K arrays of 1048576 nulls), Apple clang -O2:
  K=40 (362 bytes): before 41943040 values, 664 MB; after rejected, 33 MB
  K=2 (20 bytes):   before 2097152 values, 53 MB;   after rejected, 33 MB

The new tests count events with a SAX handler where they can: building
and comparing million-element values made the first version take 176 s
under Valgrind, the current one about 22 s.

Behaviour change: an input that holds more than 1048576 elements in
arrays of type Z, T or F at more than 4 per byte is now rejected.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

docs/mkdocs/docs/api/basic_json/from_ubjson.md

@@ -89,8 +89,9 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va
 - Throws [out_of_range.406](../../home/exceptions.md#jsonexceptionout_of_range406) if a high-precision number, or a
   float64 when `number_float_t` is `float`, does not fit into `number_float_t`
 - Throws [out_of_range.408](../../home/exceptions.md#jsonexceptionout_of_range408) if the count of an optimized
-  container exceeds the capacity of the container (or of `std::size_t`), or if an optimized array of `Z`, `T`, or `F` has
-  more than 1,048,576 elements
+  container exceeds the capacity of the container (or of `std::size_t`), if an optimized array of `Z`, `T`, or `F` has
+  more than 1,048,576 elements, or if the optimized arrays of `Z`, `T`, and `F` in the input together have more than
+  1,048,576 elements plus 4 for every byte read so far
 
 ## Complexity
 
@@ -132,6 +133,7 @@ Linear in the size of the input.
 - Extended container support (1) to include types with lvalue-only ADL `begin`/`end` (matching `std::begin`/`std::end` semantics) in version 3.13.0.
 - Extended overload (2) to accept heterogeneous iterator+sentinel pairs (C++20 ranges support) in version 3.13.0.
 - Added `error_handler` parameter in version 3.13.0.
+- Limited the total number of elements of optimized arrays of `Z`, `T`, and `F` in one input in version 3.13.0.
 
 !!! warning "Deprecation"
 

docs/mkdocs/docs/api/basic_json/to_ubjson.md

@@ -117,5 +117,8 @@ Linear in the size of the JSON value `j`.
 - Added `error_handler` parameter in version 3.13.0. Its default, `keep`, writes the bytes of a string or object key
   that is not valid UTF-8 unchanged, as before; `strict` (the default if
   [`JSON_STRICT_BINARY_UTF8`](../macros/json_strict_binary_utf8.md) is enabled) throws `type_error.316`.
+- Writes arrays of `Z`, `T`, or `F` without the `$` type annotation (`[#n Z Z ...`) once their elements in one call
+  add up to more than 1,048,576, in version 3.13.0, so that `from_ubjson` can read the output back. Such output is
+  larger: 270,000 rows of 100 `null` values take 26.8 MB instead of 1.35 MB.
 - Throws `type_error.321` for a discarded value since version 3.13.0; previously, a discarded value nested in an
   array or object was silently skipped, producing invalid UBJSON.

docs/mkdocs/docs/features/binary_formats/ubjson.md

@@ -122,9 +122,14 @@ The library uses the following mapping from JSON values types to UBJSON types ac
     already is the value. Its declared count is therefore the only thing that decides how much memory the receiving side
     allocates, and a handful of bytes can describe billions of elements. `from_ubjson` rejects such an array with
     [`out_of_range.408`](../../home/exceptions.md#jsonexceptionout_of_range408) when the count exceeds 1,048,576
-    (`1 << 20`), and `to_ubjson` writes longer arrays of these types in the counted but untyped form: the `$` type marker is
-    dropped, the `#` count is kept, and every element is written with its own marker (`[#l 1048577 Z Z ...`), so any value
-    it produces can be read back. Objects and untyped counted arrays are not subject to this limit.
+    (`1 << 20`), or when the counts of all such arrays in the input add up to more than 1,048,576 plus 4 for every byte
+    read so far. `to_ubjson` therefore writes an array of these types in the counted but untyped form when it is longer
+    than that, or when it would bring the total of the typed arrays written so far above 1,048,576: the `$` type marker
+    is dropped, the `#` count is kept, and every element is written with its own marker (`[#l 1048577 Z Z ...`), so
+    everything it produces can be read back. Only input from another encoder, or written by version 3.12.0 and earlier of
+    this library, can exceed the limit: for example, 1,100 rows of 1,000 `null` values or 210,000 rows of 25, each with
+    a `$Z` annotation. Version 3.12.0 writes such a row in 5 bytes (6 above 255 elements), so rows of 20 or fewer such
+    values are never rejected. Objects and untyped counted arrays are not subject to this limit.
 
 !!! info "Binary values"
 
@@ -180,10 +185,11 @@ The library maps UBJSON types to JSON value types as follows:
 
     Every well-formed UBJSON value can be converted to a JSON value, with these exceptions: a high-precision number
     outside the range of `number_float_t` ([`out_of_range.406`](../../home/exceptions.md#jsonexceptionout_of_range406)),
-    an optimized array of `Z`, `T`, or `F` with more than 1,048,576 elements, and a count that exceeds the capacity of the
-    container ([`out_of_range.408`](../../home/exceptions.md#jsonexceptionout_of_range408)). High-precision numbers that
-    do not fit an integer type are stored as floating-point numbers and may lose precision. Ill-formed input is rejected
-    with a parse error.
+    optimized arrays of `Z`, `T`, or `F` with more elements than the limit allows (1,048,576 plus 4 per byte read),
+    and a count that exceeds the capacity of the container
+    ([`out_of_range.408`](../../home/exceptions.md#jsonexceptionout_of_range408)). High-precision numbers that do not
+    fit an integer type are stored as floating-point numbers and may lose precision. Ill-formed input is rejected with a
+    parse error.
 
 !!! info "Duplicate object keys"
 

docs/mkdocs/docs/features/parsing/untrusted_input.md

@@ -76,6 +76,13 @@ For the binary formats, an announced size is never trusted outright:
 - An announced array or object size that exceeds what the target container could ever hold (its `max_size()`) is
   rejected immediately as [`out_of_range.408`](../../home/exceptions.md#jsonexceptionout_of_range408), without
   attempting to allocate anything.
+- A [UBJSON](../binary_formats/ubjson.md) array of type `Z`, `T` or `F` has no payload, so its announced count is not
+  bounded by the input. One array may announce at most 1,048,576 elements, and all such arrays of one input together
+  may contain at most 1,048,576 elements plus 4 for every byte read so far (checked when each array's header has been
+  read); beyond that the input is rejected as
+  [`out_of_range.408`](../../home/exceptions.md#jsonexceptionout_of_range408). For a DOM that bounds the memory these
+  arrays can need to about 80 bytes per byte of input plus about 33 MB for the first 1,048,576 elements; a SAX handler
+  receives at most that many events. A UBJSON object member always costs a key, so it needs input too.
 
 ## Strings
 

docs/mkdocs/docs/home/exceptions.md

@@ -1050,11 +1050,17 @@ The size of an array or object in a [binary format](../features/binary_formats/i
 the size following `#` for [UBJSON](../features/binary_formats/ubjson.md)/[BJData](../features/binary_formats/bjdata.md),
 or the encoded length for [CBOR](../features/binary_formats/cbor.md).
 
-The exception is also thrown for a [UBJSON](../features/binary_formats/ubjson.md) array of a type that is encoded by its
-marker alone (`Z`, `T` or `F`) whose declared count exceeds 1,048,576 (`1 << 20`). Such an array has no payload, so its
-count alone decides how much memory is allocated, and a handful of bytes would otherwise describe billions of values.
-[`to_ubjson`](../api/basic_json/to_ubjson.md) writes longer arrays of these types without the `$` type annotation (the `#`
-count and one marker per element are kept), so any value it produces can still be read back.
+The exception is also thrown for [UBJSON](../features/binary_formats/ubjson.md) arrays of a type that is encoded by
+its marker alone (`Z`, `T` or `F`). Such an array has no payload, so its count alone decides how much memory is
+allocated, and a handful of bytes would otherwise describe billions of values. It is thrown if the declared count of one
+array exceeds 1,048,576 (`1 << 20`), and if the counts of all such arrays read so far in one input add up to more than
+1,048,576 plus 4 for every byte of the input read when the header of the last array has been read.
+[`to_ubjson`](../api/basic_json/to_ubjson.md) writes an array of these types without the `$` type annotation (the `#`
+count and one marker per element are kept) when it is longer, or when it would bring the total of the typed arrays
+written so far above 1,048,576, so everything it produces can be read back. Only input written by another encoder, or
+by version 3.12.0 and earlier of this library, can exceed the limit: for example, 1,100 rows of 1,000 `null` values or
+210,000 rows of 25, each with a `$Z` annotation. Version 3.12.0 writes such a row in 5 bytes (6 above 255 elements),
+so rows of 20 or fewer such values are never rejected.
 
 !!! failure "Example messages"
 
@@ -1070,6 +1076,9 @@ count and one marker per element are kept), so any value it produces can still b
     ```
     [json.exception.out_of_range.408] syntax error while parsing UBJSON size: excessive array size
     ```
+    ```
+    [json.exception.out_of_range.408] syntax error while parsing UBJSON size: excessive number of elements without payload
+    ```
 
 ### json.exception.out_of_range.409