mirror of
https://github.com/nlohmann/json.git
synced 2026-11-12 07:00:46 +00:00
Report an empty array for UBJSON [$N#<n> instead of n elements
A no-op (N) is not a value, so an optimized array of type N holds no element whatever count it declares. The reader skipped the elements but still passed the declared count to the SAX parser: a handler was told start_array(n) and no element followed, and the DOM parser reserved min(n, 16384) values (256 KB) for an array that stays empty. Nine bytes (`[$N#l 0x7FFFFFFF`) therefore committed 256 KB: 2000 of them in one input, 18 KB, committed 512 MB of heap (capacity 163,840,000 values for the 10000 arrays of the original probe: 2.6 GB for 70 KB of input). Untouched pages keep the resident size small, but the commit charge and a 32-bit address space are consumed. The reader now announces 0 elements for an optimized array of type N. The loop that skips the elements is unchanged (it never ran). A count that does not fit into std::size_t is still rejected by the count reader; a count that is too large for the array type of the DOM (before: out_of_range.408 "excessive array size" from the DOM handler) is accepted now, because the array is empty. Behaviour change: sax_parse reports start_array(0) for `[$N#n` (before: n); from_ubjson no longer throws out_of_range.408 for an N-typed array whose count exceeds the maximum size of the array type. Signed-off-by: Niels Lohmann <mail@nlohmann.me>
This commit is contained in:
6 files changed
+116
-26
No files matched your search
@@ -89,9 +89,10 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va
|
||||
- Throws [out_of_range.406](../../home/exceptions.md#jsonexceptionout_of_range406) if a high-precision number, or a
|
||||
float64 when `number_float_t` is `float`, does not fit into `number_float_t`
|
||||
- Throws [out_of_range.408](../../home/exceptions.md#jsonexceptionout_of_range408) if the count of an optimized
|
||||
container exceeds the capacity of the container (or of `std::size_t`), if an optimized array of `Z`, `T`, or `F` has
|
||||
more than 1,048,576 elements, or if the optimized arrays of `Z`, `T`, and `F` in the input together have more than
|
||||
1,048,576 elements plus 4 for every byte read so far
|
||||
container exceeds the capacity of the container (or of `std::size_t`; an array of type `N` is empty, so only the
|
||||
latter applies to it), if an optimized array of `Z`, `T`, or `F` has more than 1,048,576 elements, or if the
|
||||
optimized arrays of `Z`, `T`, and `F` in the input together have more than 1,048,576 elements plus 4 for every byte
|
||||
read so far
|
||||
|
||||
## Complexity
|
||||
|
||||
@@ -133,6 +134,8 @@ Linear in the size of the input.
|
||||
- Extended container support (1) to include types with lvalue-only ADL `begin`/`end` (matching `std::begin`/`std::end` semantics) in version 3.13.0.
|
||||
- Extended overload (2) to accept heterogeneous iterator+sentinel pairs (C++20 ranges support) in version 3.13.0.
|
||||
- Added `error_handler` parameter in version 3.13.0.
|
||||
- Report `start_array(0)` to a SAX handler for an optimized array of type `N` (`[$N#n`), whatever its count, in version
|
||||
3.13.0.
|
||||
- Limited the total number of elements of optimized arrays of `Z`, `T`, and `F` in one input in version 3.13.0.
|
||||
|
||||
!!! warning "Deprecation"
|
||||
|
||||
@@ -131,6 +131,10 @@ The library uses the following mapping from JSON values types to UBJSON types ac
|
||||
a `$Z` annotation. Version 3.12.0 writes such a row in 5 bytes (6 above 255 elements), so rows of 20 or fewer such
|
||||
values are never rejected. Objects and untyped counted arrays are not subject to this limit.
|
||||
|
||||
An optimized array of type `N` (no-op) is empty whatever its count is, because a no-op is not a value. Its count is
|
||||
not passed on: `sax_parse` reports `start_array(0)` (versions before 3.13.0 reported the declared count), and the DOM
|
||||
parser reserves nothing for it.
|
||||
|
||||
!!! info "Binary values"
|
||||
|
||||
If the JSON data contains the binary type, the value stored is a list of integers, as suggested by the UBJSON
|
||||
|
||||
@@ -72,17 +72,22 @@ For the binary formats, an announced size is never trusted outright:
|
||||
(reported as a parse error) instead of triggering an oversized allocation.
|
||||
- When an array announces its number of elements and the array container supports `reserve()` (as `#!cpp std::vector`,
|
||||
the default, does), the library reserves storage for at most 16384 of them upfront, regardless of how large the
|
||||
announced count is; further elements still grow the container normally as they are read.
|
||||
announced count is; further elements still grow the container normally as they are read. A count that no
|
||||
input backs, as in a truncated input, reserves at most that much until the parse error is reported. A parse that
|
||||
completes keeps storage for such a count only where the elements take no input at all, which the two
|
||||
[UBJSON](../binary_formats/ubjson.md) cases below cover.
|
||||
- An announced array or object size that exceeds what the target container could ever hold (its `max_size()`) is
|
||||
rejected immediately as [`out_of_range.408`](../../home/exceptions.md#jsonexceptionout_of_range408), without
|
||||
attempting to allocate anything.
|
||||
- A [UBJSON](../binary_formats/ubjson.md) array of type `Z`, `T` or `F` has no payload, so its announced count is not
|
||||
bounded by the input. One array may announce at most 1,048,576 elements, and all such arrays of one input together
|
||||
may contain at most 1,048,576 elements plus 4 for every byte read so far (checked when each array's header has been
|
||||
read); beyond that the input is rejected as
|
||||
[`out_of_range.408`](../../home/exceptions.md#jsonexceptionout_of_range408). For a DOM that bounds the memory these
|
||||
- A [UBJSON](../binary_formats/ubjson.md) array of the no-op type (`[$N#<n>`) is empty whatever count it declares: a
|
||||
SAX handler is told `start_array(0)` and the DOM parser reserves nothing.
|
||||
- A UBJSON array of type `Z`, `T` or `F` has no payload, so its announced count is not bounded by the input. One array
|
||||
may announce at most 1,048,576 elements, and all such arrays of one input together may contain at most 1,048,576
|
||||
elements plus 4 for every byte read so far (checked when each array's header has been read); beyond that the input is
|
||||
rejected as [`out_of_range.408`](../../home/exceptions.md#jsonexceptionout_of_range408). For a DOM that bounds the memory these
|
||||
arrays can need to about 80 bytes per byte of input plus about 33 MB for the first 1,048,576 elements; a SAX handler
|
||||
receives at most that many events. A UBJSON object member always costs a key, so it needs input too.
|
||||
receives at most that many events. Every other optimized array element, and every UBJSON object member (it has a
|
||||
key), takes at least one byte of input.
|
||||
|
||||
## Strings
|
||||
|
||||
|
||||
Reference in new issue
Block a user