* Fix develop CI: GCC 4.8, Clang 3.4 and clang-tidy errors
- unit-json_patch.cpp: move multi-line raw strings out of CHECK(), which
GCC 4.8 cannot parse inside a macro argument
- unit-recursion_depth_limit.cpp: value-initialize the const std::hash
(Clang 3.4 rejects default initialization of a const object)
- json_view: silence modernize-use-transparent-functors (std::less<> is
C++14), bugprone-suspicious-stringview-data-usage (the length comes from
the node) and the deliberate use-after-clear() / owning-memory hits in
unit-json_view_builder.cpp; use a raw string literal there
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix clang-tidy errors in the std::u8string tests
Test the lvalue path with the const original instead of a mutable copy
and the rvalue path with a temporary (misc-const-correctness), and mark
the intentional char8_t array with NOLINT (avoid-c-arrays).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Read a std::u8string from a JSON string and store it as one in every configuration
In C++20, json(std::u8string) was stored as a JSON string only when
std::filesystem::path support was available (the to_json overload sat
inside the JSON_HAS_FILESYSTEM guard), and there was no matching
from_json, so get<std::u8string>() threw type_error.302 ("type must be
array, but is string"). Without std::filesystem, the same expression
still produced an array of code units (the 3.12.0 behavior).
Move the to_json overload out of the filesystem guard so that a
std::u8string is a string in every configuration, build the value
directly from the code units (this also works for a custom string_t
such as the tests' alt_string), and add a from_json overload that reads
a JSON string. The array of numbers written by 3.12.0 is still accepted;
other types throw type_error.302 ("type must be string or array").
std::u8string_view and char8_t arrays are unchanged.
The tests now run in C++20/23 (the file mentions JSON_HAS_CPP_20/23),
also without std::filesystem (test-conversions2_no_filesystem), and with
ordered_json and a custom string type. The new behavior is documented.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fill the std::u8string with resize and indexing instead of push_back
The push_back loop is not vectorized: reading a 1 kB string took about
2.2 us against 0.39 us with resize and direct element access (clang -O2).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Write the std::u8string from_json template header in the compact form
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Ignore libstdc++ char to char8_t sign change in the clang sanitizer job
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The test for json_view documents over an alternative string type called alt_editable_document::load() and save(). Both exist only on little-endian targets, so the test suite did not compile on big-endian targets such as s390x. Guard the two image lines with the condition the other image tests use (unit-json_view_image.cpp, unit-json_view_positions.cpp).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Add JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE to set the UBJSON whole-input budget
The whole-input budget for the elements of optimized UBJSON arrays of
type Z, T and F (1048576 plus 4 per byte read) rejects files written by
3.12.0 that have long arrays of these types, such as 1100 rows of 1000
nulls with a `$Z` annotation, and there was no way to read them. The
per-byte rate is now the macro JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE.
Semantics:
- default 4, so the default build is unchanged (the generated code of
from_ubjson is identical at -O2 up to source line numbers);
- the value must be a positive integer: 0, negative, non-integral, bool,
character and wider-than-uintmax_t values fail to compile with a
static_assert that names the macro;
- a value that does not fit into std::size_t is the largest size_t, so
one spelling works for 32- and 64-bit targets;
- the limit base + rate * bytes_read saturates at the largest size_t
instead of wrapping around, for any rate and input size; a rate of
131072 (2^20 / 8) or more cannot be exceeded by any input (the
shortest header of an array of more than 32767 elements is 8 bytes),
which switches the whole-input budget off;
- the limit of one array (1048576) and the base (1048576) stay fixed
constants, so the writer fallback still follows the base only and a
reader with any rate reads everything the writer produces;
- the macro is not part of the ABI tag and must have the same value in
every translation unit; it is documented that way.
The memory the arrays can take in a DOM is about 20 bytes per rate and
byte of input (measured, macOS, containers of 9 * rate elements:
25 bytes per input byte at rate 1, 81 at 4, 312 at 16, 1226 at 64, 5493
at 256) plus about 33 MB for the base. A 3.12.0 row of N values has a
header of b bytes (5 up to 255, 6 up to 32767, 8 above), so a table is
read completely if N <= rate * b, and otherwise up to
(1048576 + h * rate, h = outer header: 6, 7 or 9 bytes) / (N - rate * b) rows; 1100 rows of 1000 nulls need
a rate of 8. The new page of the macro documents the formulas, the
measurements, and two examples that are run by the documentation checks.
The tests of the budget move to tests/src/unit-ubjson_valueless.cpp, which
derives every boundary from the macro and is built with the rates 1,
1000000 and 0xFFFFFFFFFFFFFFFF in addition to the default (the last two
check that nothing is refused and that the limit does not wrap around);
they fail with the unchanged header for rate 1, 1000000 and the maximum.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Refuse char8_t for JSON_UBJSON_VALUELESS_ELEMENTS_PER_BYTE and clarify its documentation
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Report an empty array for UBJSON `[$N#<n>` instead of n elements
A no-op (N) is not a value, so an optimized array of type N holds no
element whatever count it declares. The reader skipped the elements but
still passed the declared count to the SAX parser: a handler was told
start_array(n) and no element followed, and the DOM parser reserved
min(n, 16384) values (256 KB) for an array that stays empty. Nine bytes
(`[$N#l 0x7FFFFFFF`) therefore committed 256 KB: 2000 of them in one
input, 18 KB, committed 512 MB of heap (capacity 163,840,000 values for
the 10000 arrays of the original probe: 2.6 GB for 70 KB of input).
Untouched pages keep the resident size small, but the commit charge and
a 32-bit address space are consumed.
The reader now announces 0 elements for an optimized array of type N.
The loop that skips the elements is unchanged (it never ran). A count
that does not fit into std::size_t is still rejected by the count
reader; a count that is too large for the array type of the DOM (before:
out_of_range.408 "excessive array size" from the DOM handler) is
accepted now, because the array is empty.
Behaviour change: sax_parse reports start_array(0) for `[$N#n` (before:
n); from_ubjson no longer throws out_of_range.408 for an N-typed array
whose count exceeds the maximum size of the array type.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Clarify the reservation note for announced array counts
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Two views compared equal when the tokens of two floats were equal, and
the same node was equal to itself. An edit writes a NaN as the token
"nan", so a view of it (or of an array holding it) was equal to itself
and to another NaN, while json(NaN) == json(NaN) and view == json are
false. The token shortcut now excludes the edited "nan" token (found by
the missing digit layout), dispatched on the document mode by a tag: a
read-only document cannot hold a NaN, and its walk is unchanged
(instruction count and cycles on canada, citm and twitter within
noise). For editable documents the check costs about 2 % instructions
on canada (cycles within noise). The same-node shortcut is kept for
read-only documents and for editable ones that were never edited; a
document with edits is walked, so comparing it with itself is linear
instead of constant.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
image_check::full accepted a float node whose token is an integer literal
that fits 64 bits (e.g. 10446744073709551616 changed from 18446744073709551616).
The parser stores an integer node for such a token, so the loaded
document differed from what parsing the same text gives. check_float_ranges
now requires an integer token of a float node not to fit uint64_t (or
int64_t when negative).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Document that read() must not be given a view of the document itself
basic_json_document::read() releases the previous text, the decoded
strings and the edits before it parses its input. An input that points
into them (source(), get_string(), number_token(), or a string of an
edited document) is read after its memory was cleared or freed.
A guard that copies such input costs about 1 ns per read() (5 % of a
read of a 7-byte input), which this API exists to avoid, so state the
precondition in read.md, lifetime.md and the pages of the views that
hand out such strings instead, and show how to copy the string first.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Assert in debug builds that read() is not given its own text
read() releases the document's text, decoded strings, owned image and
edits before it parses, so an input that points into them is read after
release. Without NDEBUG, read() now asserts that a borrowed input, a
copied contiguous input and (before C++17) a string_ref input are not
held by the document: its owned text, its decoded strings, an owned
image, and the current buffer of edited strings. Release builds keep
the documented precondition without a check, which would cost 2-4.5 %
per read() of a small message.
JSON_ASSERT is undefined at the end of json.hpp, so the view uses
assert() through a new internal NLOHMANN_VIEW_ASSERT.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Cause: #5476 made the DOM parsers reserve storage (up to 16384 elements)
for a counted array of CBOR, MessagePack, UBJSON and BJData when it
opens, before any input backs the count. The arrays that are open at the
same time keep their storage until the parse ends, so N nested headers
that announce 16384 elements and are never completed hold
N * 16384 * sizeof(json): 100,000 nested CBOR headers `99 40 00`
(300,000 bytes) peaked at 1.65 GiB, 3.12.0 at 12 MiB; the same for
MessagePack `DC 40 00`, UBJSON `[#I 16384` and BJData.
Fix: json_sax_dom_error_state, the base of the three DOM parsers, keeps a
credit of elements that may still be reserved: 65,536 at the start plus
2 for every byte the reader has read. An array that announces no more
than the credit left is reserved as before (one comparison and one
subtraction per counted array); a larger one is reserved what is left
and grows as its elements arrive. binary_reader tells a DOM parser (a
handler whose pointer converts to json_sax_dom_error_state*; a handler
that derives from one privately or protectedly, or any other SAX handler,
is not told anything and reserves as in the base of this change) where to
find its count of characters read for the duration of sax_parse(). An
RAII guard puts back the previous binding on every exit, also by
exception, so the parser never keeps a pointer to a reader that is gone
(unbound, only the per-array cap of 16384 applies); the public SAX
interface is unchanged, and start_array() still reports the announced number. A
bound per array by the remaining input would not help, since the same
bytes would back every open level; the bound has to be over all arrays.
The elements of an array take at least one byte of input each, so the
credit is more than honest input needs: none of the 19 inputs measured
(canada, citm_catalog, twitter, mesh, update-center, random, numbers,
animation, ik_rig, instruments, geo_response, jeopardy and 7 synthetic
ones, each as CBOR, MessagePack, UBJSON and BJData, with and without
size/type optimization) was given less than it announced. A value that
nests five or more arrays of 16384 elements, each with an array as its
first element, is the exception: the later ones grow as before.
Measurements (clang -O2, macOS arm64, peak memory footprint of
`/usr/bin/time -l`, reproducer: n nested headers):
n = 20,000 n = 100,000
develop fix 3.12.0 develop fix 3.12.0
CBOR 338.7 5.9 8.2 1687.7 23.3 11.8 MiB
MessagePack 338.9 6.0 7.3 1687.9 23.8 11.9 MiB
UBJSON 339.0 7.6 8.0 1687.9 30.1 11.6 MiB
BJData 339.0 7.3 7.9 1688.0 30.8 11.6 MiB
That is at most (65,536 + 2 per input byte) values: 1 MiB plus 32 bytes
per input byte with 16-byte values (a value is 24 bytes with
JSON_DIAGNOSTICS and up to 40 with JSON_DIAGNOSTIC_POSITIONS), as
documented in untrusted_input.md. Lower fixed caps do not get there: a
cap of 64 elements still peaks at 113 MiB (n = 100,000) and costs 4.7 %
on arrays of 300 elements (mat300.cbor) and 5.0 % on arrays of 10,000
(arrays_of_big.cbor).
Reader instructions retired (min of 7, 108 rows: 18 inputs in the 6 forms
CBOR, MessagePack, UBJSON and BJData plain and optimized; jeopardy, 46 MB,
is within 0.1 %), compared with the base of this change: -0.19 % to +0.37 % in the formats that announce
sizes (canada.cbor 81.9 M -> 82.1 M, many_small.cbor 333.5 M -> 334.2 M);
the benefit of #5476 is kept (without any reserve canada.cbor takes
116.8 M, +42.6 %, many_small.cbor +71.5 %, mat300.cbor +81.6 %). Rows of
plain UBJSON and BJData, whose arrays announce no size, move by -2.7 % to
+2.6 % without any code of theirs changing: the base alternates
between two instruction counts per process (452 M and 463 M for
many_pairs.ubjson, equal in cycles), and a struct with three unused
members added to the same class reproduces the 474 M mode.
Tests (CBOR, MessagePack, UBJSON, BJData): 200 nested headers that
announce 16384 elements peak at 52,433,600 bytes of DOM allocations on
the base and below 128 Ki values with the fix (scaled by sizeof(value),
so that it holds with the diagnostic macros) (counting allocator, no
process memory measured); an array that completes is still reserved in
one allocation; CBOR also checks that a DOM parser is not bound to a finished
parse (ASan: stack-use-after-scope without the guard), that wrappers
derived publicly (also with a parse inside the parse), privately and
protectedly compile and are bound or not as described, that an array that got less than it
announced decodes and that a SAX handler is told the announced size.
The new tests take 4.6 s under Valgrind at -O0 (gcc:12, all four files).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
A no-op (N) is not a value, so an optimized array of type N holds no
element whatever count it declares. The reader skipped the elements but
still passed the declared count to the SAX parser: a handler was told
start_array(n) and no element followed, and the DOM parser reserved
min(n, 16384) values (256 KB) for an array that stays empty. Nine bytes
(`[$N#l 0x7FFFFFFF`) therefore committed 256 KB: 2000 of them in one
input, 18 KB, committed 512 MB of heap (capacity 163,840,000 values for
the 10000 arrays of the original probe: 2.6 GB for 70 KB of input).
Untouched pages keep the resident size small, but the commit charge and
a 32-bit address space are consumed.
The reader now announces 0 elements for an optimized array of type N.
The loop that skips the elements is unchanged (it never ran). A count
that does not fit into std::size_t is still rejected by the count
reader; a count that is too large for the array type of the DOM (before:
out_of_range.408 "excessive array size" from the DOM handler) is
accepted now, because the array is empty.
Behaviour change: sax_parse reports start_array(0) for `[$N#n` (before:
n); from_ubjson no longer throws out_of_range.408 for an N-typed array
whose count exceeds the maximum size of the array type.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
An optimized UBJSON array of type Z, T or F has no payload, so its
declared count is not bounded by the input. The limit of 1048576 applied
to each array on its own, so 9 bytes (`[$Z#l` and a count) produced
1048576 values and every further 9 bytes another 1048576: 362 bytes
allocated 661 MB in a DOM, and a SAX handler received 1M events per 9
bytes.
The reader now adds the counts of all such arrays of one input and
rejects the input with out_of_range.408 when the sum exceeds 1048576
plus 4 for every byte read so far. The base is the per-array limit, so
an array that was accepted on its own before is still accepted at the
start of an input. These arrays add about 80 bytes of memory per input
byte (measured 77; one `Z` per byte needs 32), plus the base; the memory
of a document as a whole is not bounded by this. The check runs once per
valueless array, before its start event, with plain size_t arithmetic
that cannot overflow for a 32-bit size_t. Containers with a payload type
and objects (every member needs a key) are bounded by the input already
and are unchanged. BJData rejects these types.
to_ubjson counts the elements of the typed arrays it writes in one call
and writes an array in the unoptimized form (`[#n Z Z ...`) once it would
bring the total above 1048576, which a reader accepts at any input size.
The check is made only for a typed array of a valueless type, so BJData
and other arrays do not pay for it. Everything to_ubjson writes in
version 3.13.0 can therefore be read back.
Two consequences:
- to_ubjson output changes for values with more than 1048576 elements in
arrays of type Z, T or F: such an array is written at one byte per
element, e.g. 270000 rows of 100 nulls take 26.8 MB instead of 1.35 MB.
- Output of version 3.12.0 and earlier (typed arrays without such a
total) is rejected by from_ubjson when the arrays hold more than
1048576 elements plus 4 per byte of input. 3.12.0 writes a row of up to 255 nulls in 5 bytes (6 above
255), so a table is rejected from 1075 rows of 1000 nulls or 209723
rows of 25 (accepted: 1000 rows of 1000, 209000 rows of 25); rows of
20 or fewer are never rejected.
Measured on the test data: no file of json_test_data (converted with
to_ubjson in all three modes, 846 round trips) holds an array of this
kind with more than 0.17 values per byte. All of them are read back.
Reproducer (K arrays of 1048576 nulls), Apple clang -O2:
K=40 (362 bytes): before 41943040 values, 664 MB; after rejected, 33 MB
K=2 (20 bytes): before 2097152 values, 53 MB; after rejected, 33 MB
The new tests count events with a SAX handler where they can: building
and comparing million-element values made the first version take 176 s
under Valgrind, the current one about 22 s.
Behaviour change: an input that holds more than 1048576 elements in
arrays of type Z, T or F at more than 4 per byte is now rejected.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Read iterator pairs of std::string and std::vector as pointer ranges
Before C++20, a pair of iterators of a contiguous container cannot be told
from any other random-access iterator, so json::parse(s.begin(), s.end())
on a std::string or std::vector<char> was read one character at a time,
without the bulk scans of strings, numbers and indentation that the
container itself gets (the container overload of input_adapter() already
maps it to a pointer range). The iterators of std::string and of a
std::vector of 1-byte integral elements are now matched exactly and mapped
to the pointer range they cover; an empty range is not dereferenced. The
character type stays the same, so the results and error positions do not
change.
parse(first, last) of a std::string gets 1.7-2.0x as fast in C++17 on an
Apple M1 Max, accept(first, last) 2.7-3.1x, and from_cbor(first, last) of a
std::vector 8-23% faster.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix CI: clang-tidy, unused result, Infer false positive in tests
- unit-deserialization.cpp: raw string literals, and assign the result
of parse() in CHECK_THROWS_AS (-Wunused-result)
- unit-bson.cpp: mark Infer's NULLPTR_DEREFERENCE on from_bson() with an
iterator pair as a false positive
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix implicit char conversion in iterator-pair test
The test built std::vector<unsigned char> and <signed char> from a char
range, which -fsanitize=integer reports as an implicit conversion.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Tests: sweep_allocator::construct() accepts pointers to const
Recent libc++ (emsdk latest) probes construct(const T*, const T&) in an
unevaluated context while checking for trivial copies; the static_cast to
void* in the noexcept specification then fails to compile.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Tests: guard sweep_allocator::construct() against const types instead of const_cast
The const_cast added in the previous commit is rejected by
cppcoreguidelines-pro-type-const-cast (clang-tidy). Make construct() not
viable for const U instead, which is what libc++ probes for.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix CI: astyle and clang-tidy for sweep_allocator::construct()
astyle 3.4.13 reformats the enable_if<!...> default template argument
(Check amalgamation). Delete the const overload instead, which makes the
libc++ probe for const objects fail in the SFINAE context just the same
and needs no <type_traits>.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Iterator ranges: astyle-safe trait members; reserve in the test helper
The byte check and the container match are static constexpr members
instead of integral_constant arguments and a default template argument
with comparisons, which astyle spaced out.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
dump() batches float tokens of 16 or 17 digits with at least 8 fraction
digits. A damaged image loaded with image_check::bounds can record a
token with no digit before the point and 17 after it, which no JSON
number has. Its q = -17 is outside the range of locate_token(), which
then read fives[17] past its 17 entries; on x86-64 the resulting digits
had 18 digits and failed the assertion of write_decimal() (OSS-Fuzz
issue 583898810, json_view_image_fuzzer).
The batch now takes 8 to 16 fraction digits, which every valid token of
16 or 17 digits has, so it batches the same tokens of valid documents.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Parse JSON in one block of memory with a fused pointer-based parser
The parser and the lexer exchange one token at a time, and the lexer keeps
its state (the current character, a pending unget, where the token starts)
in members, which every character pays for: json::accept() of twitter.json
took 2.8x as long as json_document::accept(). detail::contiguous_parser
reads text in one block of memory through a local pointer, with the
grammar, the whitespace and the tokens in one loop (after yyjson's reader),
and drives a SAX handler: parse()'s DOM handler, accept()'s handler, or the
user's handler of sax_parse(). It is used with the default options (no
callback, no comments, no trailing commas; sax_parse() with strict = true).
Errors are reported by the library parser, so the exception, its message and
its position stay the same. contiguous_parser stops at the token that the
library parser would reject, before reporting it, so the events up to the
error are the same; then:
- parse() runs the library parser with a handler that only reports the
error (json_sax_error_reporter), with the lexer of parse(), which is
instantiated anyway;
- sax_parse() runs it with a handler that forwards only parse_error() to
the user's handler (json_sax_error_forwarder), so no event arrives twice;
- accept() returns the verdict.
A handler that returns false stops the parse as before. An error at the end
of a large input costs parse() about 1.2x of what it cost before.
On an Apple M1 Max: accept() 1.6-2.0x as fast, parse() 10-16% faster,
ordered_json 11-16%, sax_parse() with a user's handler 1.36-1.46x, small
messages 12-22% (parse()) and 2.1-2.4x (accept()). On x86-64 (GCC 15,
Clang 21): accept() 1.6-1.9x, parse() 5-17%, ordered_json 12-19%,
sax_parse() 1.32-1.57x. A translation unit that parses a string compiles 3%
(-O0) and 9% (-O2) slower.
unit-contiguous_parser.cpp compares the contiguous path with the byte path
(std::list iterators) on corner cases, generated documents, their
truncations and mutations, deep nesting, and narrow number types: the same
values with the same number types and float bits, exception messages,
accept() verdicts and sax_parse() events.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix CI: unused parameter without exceptions, cpplint, BUILD.bazel
- json_sax_error_reporter::parse_error: mark the exception unused, as
JSON_THROW does not use it with JSON_NOEXCEPTION
- unit-contiguous_parser.cpp: compare the exceptions of both paths only
when exceptions are enabled (invalid input aborted the test otherwise)
- silence cpplint's readability/braces false positives on the if
statements inside the parser's macros
- add contiguous_parser.hpp to BUILD.bazel
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* accept(): skip digits without computing their value
accept() needs the value of a number only for a float whose exponent may
reach the largest value. skip_digits() finds the end of the digits without
the multiply-adds of accumulate_digits(), and once an 8-byte word fails the
digit test, the remaining loop needs no bounds check: a non-digit is in
that word. The value is computed afterwards for the rare floats that need
it.
accept() of canada.json: -16 % on Apple M1, -22 % with GCC 15 and -21 %
with Clang 21 on x86-64; files with many numbers -9 to -15 %.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Tests: mark parser callback in unit-contiguous_parser noexcept
Fixes the -Werror=noexcept failure with GCC 26 (ci_test_standards_gcc (26)).
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Tests: sweep_allocator::construct() accepts pointers to const
Recent libc++ (emsdk latest) probes construct(const T*, const T&) in an
unevaluated context while checking for trivial copies; the static_cast to
void* in the noexcept specification then fails to compile.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz
(cherry picked from commit 794bf70823c12a2a1cb2a8a95f34dfb6357746a8)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Tests: guard sweep_allocator::construct() against const types instead of const_cast
The const_cast added in the previous commit is rejected by
cppcoreguidelines-pro-type-const-cast (clang-tidy). Make construct() not
viable for const U instead, which is what libc++ probes for.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz
(cherry picked from commit de099fa607001b710862fd1bbecef3609c54cbc4)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix CI: astyle and clang-tidy for sweep_allocator::construct()
astyle 3.4.13 reformats the enable_if<!...> default template argument
(Check amalgamation). Delete the const overload instead, which makes the
libc++ probe for const objects fail in the SFINAE context just the same
and needs no <type_traits>.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Contiguous parser: review fixes (wording, macros, diagnostic positions)
- "default parser" instead of "library parser" in comments and tests
- the three comments in json.hpp say the same thing
- current() replaces the repeated `p != end ? *p : 0`
- the backslashes of each macro are aligned
- the docs say that JSON_DIAGNOSTIC_POSITIONS turns the fast path off
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
* Scan the buffer of a FILE in bulk with glibc and the Apple C library
Parsing from a FILE* still read one character at a time through
getc_unlocked(), while a std::istream has its get area scanned in bulk.
file_input_adapter now offers the lexer the same kind of window into the
buffer of the FILE: it reads the fields that the C library's own
getc_unlocked() macro reads (glibc: _IO_read_ptr and _IO_read_end; Apple:
_p and _r) and consumes exactly the characters getc_unlocked() would
return, so the file is left where it was left before. The part of the
current token in a buffer that is about to be refilled is kept for error
messages, as for streams. The window needs the file locked for the whole
parse, so JSON_DISABLE_UNLOCKED_FILE_IO and MemorySanitizer builds turn it
off; uClibc, which defines __GLIBC__ but has other fields, does not get it.
parse() from a FILE* gets 2.0-2.6x as fast on an Apple M1 Max and 1.7-2.2x
on x86-64 (glibc, GCC 15 and Clang 21), accept() 2.9-4.4x; that is about as
fast as parsing from a std::ifstream. The new tests read with FILE buffers
of 1 to 70 bytes and unbuffered files, after ungetc(), with SAX positions
and with binary strings across the buffer's edges.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Tests: no setvbuf() buffer below 2 bytes on Windows
The Microsoft C runtime rejects a _IOFBF buffer size below 2 as an
invalid parameter and ends the process (0xc0000409), which made
test-file_input fail on all MSVC and clang-cl jobs.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix CI: astyle formatting in unit-file_input.cpp
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Tests: limit the multi-character ungetc() test to glibc and Apple
The Windows C runtime only supports one pushed-back character, which
broke the Windows CI jobs.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Tests: sweep_allocator::construct() accepts pointers to const
Recent libc++ (emsdk latest) probes construct(const T*, const T&) in an
unevaluated context while checking for trivial copies; the static_cast to
void* in the noexcept specification then fails to compile.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz
(cherry picked from commit 794bf70823c12a2a1cb2a8a95f34dfb6357746a8)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Tests: guard sweep_allocator::construct() against const types instead of const_cast
The const_cast added in the previous commit is rejected by
cppcoreguidelines-pro-type-const-cast (clang-tidy). Make construct() not
viable for const U instead, which is what libc++ probes for.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz
(cherry picked from commit de099fa607001b710862fd1bbecef3609c54cbc4)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix CI: astyle and clang-tidy for sweep_allocator::construct()
astyle 3.4.13 reformats the enable_if<!...> default template argument
(Check amalgamation). Delete the const overload instead, which makes the
libc++ probe for const objects fail in the SFINAE context just the same
and needs no <type_traits>.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
* Tests: make sweep_allocator::construct() not viable for const types
The newest libc++ (emsdk latest, used by ci_emscripten) probes
construct(const T*, const T&) in an unevaluated context while checking
whether copies are trivial. For the test allocator, instantiating the
noexcept specification then fails to compile, because static_cast<void*>
cannot drop the const qualifier. Constrain construct() to non-const types,
which makes the probe fail in the SFINAE context instead of with a hard
error.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix CI: astyle and clang-tidy for sweep_allocator::construct()
astyle 3.4.13 reformats the enable_if<!...> default template argument
(Check amalgamation). Delete the const overload instead, which makes the
libc++ probe for const objects fail in the SFINAE context just the same
and needs no <type_traits>.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
* Binary reader and writer: remove checks that invariants make dead
Branch coverage of the test suite and of 24k mutated inputs showed
these conditions as never taken; each is implied by a check made
earlier on the same path:
- check_bson_document_size: `document_size < 0` (the size was rejected
below 5 before it was stored)
- get_ubjson_ndarray_size: `second != 'N'` ('N' is rejected as a type
marker before `second` can be set)
- bjd_type_name() == nullptr for an ndarray dtype: every marker
is_valid_optimized_type() accepts for BJData has a name (now an assert)
- the early is_bjd_excluded_optimized_type() check in
get_ubjson_size_type(): is_valid_optimized_type() rejects the same
eight markers with the same error
- `input_format == bjdata` next to `is_ndarray`, bit 8 of the type, or a
'B'/'u'/'m'/'M' marker: these only arise for BJData
- `first != npos` next to bit 8: the bit is only set after a count
- MessagePack writer: the `<= max` halves of the negative int8/16/32
ladder (the ladder is in the `else` of `n >= 0`)
No change in behaviour: a build that aborts on each removed side passes
the test suite and the fuzz corpora, and a 1.6M-input differential run
of the reader against the original headers found no difference.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Remove is_bjd_excluded_optimized_type(), which has no caller left; fix two comments
is_valid_optimized_type() rejects the markers BJData excludes, so the reader
no longer needs its own list; the test now checks is_valid_optimized_type()
of a BJData reader against the writer's list. The BSON size comment now says
that open_bson_document() rejects a size below 5.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Make the BJData reader in the lookup test const (misc-const-correctness)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
output_vector_sink grew the vector by resizing it to its whole new
capacity. resize() zeroes the characters it adds, so every doubling
zeroed megabytes of a fresh allocation at once: that faults in all its
pages and evicts them from the cache before the writer reaches them,
and the writer's stores then miss again. The sink now reserves the same
capacity as before but resizes the vector only to 64 KiB
(vector_sink_window) ahead of the position, which is still cached when
the writer's stores come.
to_cbor(), to_msgpack(), to_ubjson(), to_bjdata() and to_bon8() of an
array of one million numbers into a fresh vector: 4-26 % faster on x86
(Skylake laptop with Apple clang 13, Haswell VPS with GCC 16 and
Clang 23), 1-2.5 % on Apple M1 Max; other documents unchanged (within
0.4 % in reruns). The capacity and growth of the vector are as before.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* UBJSON/BJData reader: reject a container count equal to SIZE_MAX
The reader marks a container whose count is not known by npos, which
is SIZE_MAX. A count of exactly that value was stored unchanged and
then read as "no count", so the BJData input
[ # M ff ff ff ff ff ff ff ff i 01 ]
parsed as [1] instead of throwing out_of_range.408 as a count of
2^64-2 does. Where size_t is 32 bits, the same holds for 'm' and 'L'
counts of 2^32-1, in UBJSON too.
The counts of 'm', 'M' and the signed types now go through
store_ubjson_count(), which rejects npos with "integer value overflow"
(the message a count beyond size_t already gets), as the CBOR reader
does in get_cbor_container_size(). An ND-array dimension of 2^64-1 gets
the same message now, where it was "excessive ndarray size caused
overflow" (the product check) on 64-bit platforms.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* MessagePack reader: reject an array/map 32 length equal to SIZE_MAX
On a platform with a 32-bit std::size_t, an array 32 or map 32 length of
0xFFFFFFFF equals unknown_size(), so start_array()/start_object() were
told the size was unknown and skipped the max_size() check. Route both
through the CBOR length check, renamed get_container_size(), which
already rejects that value.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix the build of develop: tests from #5997 and #5991
- unit-msgpack-cpp17: assign the result of json::parse in
CHECK_THROWS_WITH_AS (GCC -Werror=unused-result)
- unit-json_view: failing_allocator::allocate is [[noreturn]]
(clang -Wmissing-noreturn)
- unit-class_lexer: build the quoted string once
(clang-tidy performance-inefficient-string-concatenation)
- unit-allocator: sweep_allocator provides pointer, reference and size
typedefs, which GCC 4.8's std::map takes from the allocator
- unit-allocator: skip the "sax_parse into an existing value" sweep with
iterator debugging, like the parser callback sweep: MSVC's std::vector
default constructor is noexcept but allocates its debug proxy through
the allocator, so a failing allocation in create<array_t>() terminates
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix the remaining CI failures of the tests from #5997 and #5991
- unit-custom-object-key-type: drop the redundant member initializer and
return a braced init list (clang-tidy); direct-initialize the converted
value, as copy-initialization makes GCC 4.8 instantiate the copy
constructor, which needs a range constructor strict_object lacks
- unit-stream-bulk-scan: pick the claimed get area size by tag dispatch
instead of a constant condition (MSVC C4127)
- unit-allocator: skip the parser callback and sax_parse sweeps with
Clang 3.4 and 3.5, which leak memory when an element constructor throws
inside a libstdc++ std::map copy or std::vector reallocation
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix the CI failures that the first failing TU of each job hid
- unit-stream-bulk-scan: compute the claimed size without a condition;
the tag-dispatched template was unused with a 64-bit std::size_t
(-Wunused-template)
- unit-json_view: give failing_allocator the typedefs, rebind and
members that GCC 4.8's std::map takes from the allocator directly
- clang-tidy hits in the tests from #5997: raw string literals
(unit-json_view), redundant c_str() and a NOLINT for &text[0]
(unit-locale-cpp), reserve() before a loop (unit-ordered_json), and a
NOLINT for a return that a braced list would turn into an array
(unit-regression1)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Skip the 3 GiB get-area test with a 32-bit std::size_t in the preprocessor
The claimed size was a constant 0 on Win32, so `claimed == 0 || ...` was a
constant condition and MSVC reported C4127. Guard the section with
`#if SIZE_MAX > UINT32_MAX` instead, as unit-msgpack-custom-types.cpp does.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Silence modernize-use-transparent-functors for failing_map in unit-json_view
std::less<> needs C++14, and the test is also built as C++11.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
json_view number(): #6006 folded the check of the digit after a minus
sign into `else if (!negative || ...)`, which MSVC 19.44 reports as
C4127 (conditional expression is constant) and /WX turns into an
error in every MSVC and clang-cl job. The check now sits in the
`if (negative)` block, right after the sign is skipped; the error and
its position are unchanged.
unit-locale-cpp.cpp: the locale_switching_token added in #5997 held a
`const char*`, so GCC's -Weffc++ asked for a copy constructor and copy
assignment. The locale name is now a std::string.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
enter_cbor_array and enter_msgpack_array called the out-of-line scalar
run for every non-empty array. When the array's element is an object or
an array, as on every level of [[...]] or the single ring of a GeoJSON
polygon, the call reads nothing; it made 1,000 arrays nested in each
other cost 4 % more instructions (GCC 13) and up to 8 % more time (Apple
clang) than without the run. #5975 had fixed that with an inline
pre-check, which was later dropped after measuring canada only.
The run is now entered only for arrays of two or more elements: the same
comparison as before, so no other row changes. Arrays of one scalar,
which the run read faster, now take the general path; they are rare in
documents (one in citm_catalog, none in twitter or canada). The new
benchmark FromBinaryShape/singles measures them, next to
FromBinaryShape/nested.
Instructions per repetition, new/old, GCC 13 callgrind: nested arrays
0.973 (CBOR) and 0.971 (MessagePack), arrays of one scalar 1.037 and
1.021, every other row within 0.15 %.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Create the DOM parser's member value before placing it (#5985)
json_sax_dom_parser::set_member() constructed the new value in place over
the null member. basic_json(value_t) sets the type before the json_value
allocates, so a std::bad_alloc left a member typed object/array with a
null pointer, and the destructor crashed instead of propagating the
exception. Construct the value first and move it over the member (the
move constructor is noexcept).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix CI for the parse() bad_alloc sweep test (#5985)
- sweep_allocator: provide max_size(), construct() and destroy(), which
GCC 4.8's std::map calls on the allocator directly.
- Declare the non-throwing test lambdas noexcept (-Werror=noexcept on
recent GCC).
- Give the sax_parse lambda an explicit return type; Clang 3.4 crashes on
the deduced one in C++11 mode.
- Skip the parser callback sweep with iterator debugging (MSVC debug
builds, clang-cl), where containers allocate debug proxies inside
noexcept move constructors and terminate instead of throwing, as in the
other allocator tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N5NasAgbDGG28MqXjQvLqp
Signed-off-by: Claude <noreply@anthropic.com>
* Construct a scalar member in place; only a throwing value goes through a temporary
Creating the value first and moving it over the member is needed only when
creating it can throw (a string, a binary value or a container). A scalar
or null is again constructed in place, without the extra move and destroy
per member (and, with JSON_DIAGNOSTICS, without the extra set_parents()).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Spell the nothrow trait without && and fix the construct_member comments
The dispatch in set_member tests std::is_nothrow_constructible, not
whether the value is a scalar. Say so in the comments, and drop the &&
from the trait argument (equivalent for a forwarding reference) so that
astyle no longer inserts spaces around the template brackets.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Say "for example" in the construct_member comments
The overloads are selected by std::is_nothrow_constructible, not by the
kind of value: a custom type that cannot throw takes the first one.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Signed-off-by: Claude <noreply@anthropic.com>
Co-authored-by: Claude <noreply@anthropic.com>
* CI: measure line coverage without lcov's unexecuted-block adjustment
Drop geninfo_unexecuted_blocks=1 from the lcov capture of ci_test_coverage:
it reset every line of an inlined or templated function to 0 as soon as one
copy had an unexecuted block, which understated the line coverage by about
three points.
Add -fvar-tracking-assignments to both coverage builds. At -O0, GCC inlines
always_inline functions before it places the coverage counters, so the call
sites of such helpers (put_literal() and others) got no line record and
showed as 0 in an -O3 translation unit that never reaches them. With the
flag, the call sites carry counts in every translation unit. The generated
code is identical.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Remove unreachable code found by the coverage analysis
Remove code that no input can reach, and mark the lines that cannot be
covered with LCOV_EXCL_LINE:
- BJData writer: the dry_run parameter of the ND-array element writers was
always true; replace them by range checks.
- BON8 writer: call throw_integer_out_of_range() directly instead of
write_bon8_value() for values that always throw.
- binary_writer: integer_bits() is only called for integers of up to 64 bits.
- serializer, view serializer, lexer: replace guards that the callers make
unreachable by assertions or by the simpler code (surrogate pairs under
ensure_ascii, next_unget, '-' at the end of the buffer).
- json_view: drop unreachable branches in scalar(), set(), check_utf8_scan(),
append_input_impl() and unescape().
- Mark exception-cleanup braces, 2 GiB limits, big-endian and 32-bit code, and
fall-through case labels with LCOV_EXCL_LINE.
No behavior changes for any input that reaches these functions.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Tests: cover the remaining lines of the library headers
Add tests for the lines of include/nlohmann that were not covered: the
binary readers and writers, the lexer and locale paths, the json_view
modules, the custom number, key and array types, json_pointer, json_patch,
diff and the iterators.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix test failures in the coverage tests
- unit-custom-object-key-type: remove the unused make_strict_key(strict_key)
overload, the unused range constructor and default constructor, and guard
data() with JSON_DIAGNOSTICS (unused-function/-member-function errors with
GCC and clang -Weverything).
- unit-json_view_compare: move the two new sections to the new
unit-json_view_compare_values.cpp, as the object file got too large for the
MinGW assembler again, and use a std::map with std::greater instead of
std::unordered_map, which does not accept the incomplete basic_json
(clang 3.9, icpc).
- unit-json_view_dump: rename the variables that hid outer ones (MSVC C4456).
- unit-stream-bulk-scan: restore the real size of the oversized get area
before the adapter is destroyed (JSON_PRECISE_STREAM_POSITION reads the
character at the read position), and skip the test where the streambuf
cannot hold a get area larger than an int (MSVC).
- unit-regression1: with JSON_NO_THREAD_LOCAL, copies are built member by
member and keep correct timestamps, so check the corresponding outcome.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N5NasAgbDGG28MqXjQvLqp
Signed-off-by: Claude <noreply@anthropic.com>
* Return std::size_t from the hash helper in the ordered_json test
MSVC (Win32) rejected the implicit conversion of the unsigned long long
product to the 32-bit std::size_t (C4244 as an error). Also note when
JSON_NO_THREAD_LOCAL is set in the regression test.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N5NasAgbDGG28MqXjQvLqp
Signed-off-by: Claude <noreply@anthropic.com>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Signed-off-by: Claude <noreply@anthropic.com>
Co-authored-by: Claude <noreply@anthropic.com>
* Keep the byte range in exceptions about the root value (#5987)
With JSON_DIAGNOSTICS and JSON_DIAGNOSTIC_POSITIONS both set, exception::diagnostics()
and the json_view diagnostics() returned early for a value without a path (the
root), dropping "(bytes a-b)". Build the "(path) " prefix only when there are
tokens and always append the byte range.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Guard the null value of a value-initialized iterator; drop the unneeded found flag
With the root's byte range, diagnostics() reaches get_byte_positions() also
with JSON_DIAGNOSTICS when there is no value, as for a value-initialized
iterator (invalid_iterator.212). get_byte_positions() now returns no range
for a null value, the same change as #6003, so either can be merged first.
In the view, a value that value_path() does not find leaves no tokens, so the
found flag added nothing to the tokens.empty() test.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* json_view: get<T>() of a boolean throws for number_integer_t, number_unsigned_t and number_float_t like basic_json (#5988)
basic_json::get<T>() of a boolean throws type_error.302 for exactly
number_integer_t, number_unsigned_t and number_float_t (dedicated from_json()
overloads) and gives 0/1 for every other arithmetic type. The view returned
0/1 for all of them, contradicting its documentation. The view now throws
the same exception (same message, with path/bytes decoration) for these three
types. basic_json is unchanged; its behavior is documented and pinned by a test.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* tests: skip throwing json_view bool get<T>() combinations without exceptions (#5988)
check_boolean_get_parity() called get<T>() for number_integer_t,
number_unsigned_t and number_float_t in its JSON_NOEXCEPTION branch, but a
boolean throws type_error.302 for exactly these types, so JSON_THROW aborted
test-json_view_cpp11/cpp17 in ci_test_noexceptions. Skip these combinations
there; the throwing checks stay in the exceptions-enabled build.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N5NasAgbDGG28MqXjQvLqp
Signed-off-by: Claude <noreply@anthropic.com>
* Expect the byte range of the root in the json_view boolean get<T>() test
With JSON_DIAGNOSTIC_POSITIONS, the type_error.302 message starts with
"(bytes 0-4)"; build the expected text with json_view_test::message(),
as the other tests of the root value do.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N5NasAgbDGG28MqXjQvLqp
Signed-off-by: Claude <noreply@anthropic.com>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Signed-off-by: Claude <noreply@anthropic.com>
Co-authored-by: Claude <noreply@anthropic.com>
* Make basic_json::insert() provide its documented strong exception guarantee (#5986)
A failed allocation while the new elements were copied left the value
changed, and with JSON_DIAGNOSTICS stale parent pointers behind. The
extra work now happens almost only on the failure path:
- Array insert(pos, cnt, val) and insert(pos, first, last) append the
copies at the end and rotate them into place. If a copy throws, the
appended elements are erased again (noexcept) and the parents are set.
Moving a basic_json cannot throw, so reallocation is safe. This is no
slower than vector::insert (faster for numbers), needs no temporary
array, and keeps the elements that libstdc++ lost on reallocation.
- Array insert(pos, val) copies a non-scalar value first and inserts it
by move, except at the end, where vector::insert is strong. Scalars
keep the direct path.
- The object range insert(first, last) into an empty object uses the
container's own range insertion and clears the object on an exception.
Into a non-empty object it inserts member by member and logs one bit
per source member (a new member is detected by the size change; no
allocation for the first 256 members); on an exception the members
added are erased by walking the source range. No assumption that
key_type is string_t.
- The parents are set when a failed insertion is undone.
The regression test fails every allocation of every insert() overload in
turn, for json and ordered_json, and checks the value and its parents.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix clang-tidy and -Wnrvo findings in insert rollback helpers
Declare the rollback helpers non-movable, replace the nested conditional
and C array in object_insert_rollback, silence the intentional copy in the
allocator test, and avoid the missed NRVO in the test factory lambda.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N5NasAgbDGG28MqXjQvLqp
Signed-off-by: Claude <noreply@anthropic.com>
* insert(): strong guarantee for single elements, basic for the rest
Rework of the previous two commits, following the discussion in #5996:
the standard containers give the strong guarantee only for a single
element (std::vector: at the end, or if the exception does not come from
copying T; std::map: single-element insert), and copying a basic_json
allocates. Rebuilding a rollback for the count, range and object-range
overloads on top of array_t/object_t made the code verbose and slower.
- insert(pos, const basic_json&): at end() forward as before (the
standard covers it); elsewhere copy first and insert by move, as
libstdc++ does internally (libc++ shifts the tail before copying).
- insert_iterator() and object insert(first, last) set the parents in a
scope guard, so a failed insertion leaves a valid value with
JSON_DIAGNOSTICS too (the next assert_invariant() aborted before).
- Docs: strong guarantee for (1), basic guarantee for (2)-(5); the
argument checks throw before anything is changed.
- Test: allocation-failure sweep checks the strong guarantee for (1) and
validity, type and parents for the other overloads.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* insert(): document the exception safety of overloads 2 to 5 once
The four basic-guarantee statements were identical up to array/object.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* insert(): simplify the wording of the basic guarantee
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Signed-off-by: Claude <noreply@anthropic.com>
Co-authored-by: Claude <noreply@anthropic.com>
* json_view: dump() without StringType::capacity() (#5994)
output_buffer::grow() compared the output size with m_out.capacity(),
which the StringType requirements do not include, so dump() did not
compile with the tests' alt_string. Keep what was last reserved in the
buffer itself and use it as a lower bound of the capacity instead.
Add json_view tests with alt_string (read-only and editable documents,
dump with several growth steps, save/load), and document reserve(n) and
resize(n) as the StringType requirements of basic_json_view::dump.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Use a raw string literal in the alt-string json_view test
clang-tidy (modernize-raw-string-literal) rejects the escaped literal.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N5NasAgbDGG28MqXjQvLqp
Signed-off-by: Claude <noreply@anthropic.com>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Signed-off-by: Claude <noreply@anthropic.com>
Co-authored-by: Claude <noreply@anthropic.com>
* Do not construct the base class when destroying a container (#5995)
json_value::destroy_container() and destroy_walk() used default-
constructed basic_json values as stand-ins, which construct a
CustomBaseClass that belongs to no value. A base class whose default
constructor allocates made the noexcept destructor allocate again, and
terminate if that allocation failed, which #5762 set out to prevent.
Use data (type and value, no base class) as the stand-in instead, and
pass the children's m_data to the walk helpers.
Add a test with a counting base class: destroying a value constructs no
base object and destroys exactly one per value, also for values nested
deeper than the recursion depth limit.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Release the test value with = nullptr instead of reset()
clang-tidy (readability-ambiguous-smartptr-reset-call) rejects j.reset()
because the pointee inherits a reset() from counting_base.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N5NasAgbDGG28MqXjQvLqp
Signed-off-by: Claude <noreply@anthropic.com>
* Use std::swap for the container stand-in in destroy_container
The unqualified call after `using std::swap` would let a user-provided
swap(json_value&, json_value&) intercept the destruction path. Call
std::swap directly.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Signed-off-by: Claude <noreply@anthropic.com>
Co-authored-by: Claude <noreply@anthropic.com>
* JSON_DIAGNOSTIC_POSITIONS: comparing a value-initialized iterator throws instead of crashing
With JSON_DIAGNOSTIC_POSITIONS, comparing a value-initialized iterator
(which has no container) with another iterator built the
invalid_iterator.212 message from the byte positions of the iterator's
value, which is a null pointer: UBSan reports "member call on null
pointer" in exception::get_byte_positions(), then the process crashes.
Every other configuration throws invalid_iterator.212.
get_byte_positions() now returns no positions for a null value, as the
JSON_DIAGNOSTICS path already does.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Declare the value-initialized iterator in the test const
clang-tidy (misc-const-correctness) flags it, as it is only compared.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N5NasAgbDGG28MqXjQvLqp
Signed-off-by: Claude <noreply@anthropic.com>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Signed-off-by: Claude <noreply@anthropic.com>
Co-authored-by: Claude <noreply@anthropic.com>
readability-math-missing-parentheses in unit-json_view_builder.cpp (#5974)
and performance-inefficient-string-concatenation in
unit-json_view_positions.cpp.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* UBJSON/BJData readers: read optimized arrays of numbers in one loop
An array with a type marker for a fixed-size number ($U, $i, $I, $l, $L,
$d, $D, and BJData's $u, $m, $M), including the data of a BJData ndarray,
gives its elements no markers. When the input is contiguous, read the
elements that are in the input in one loop - load, byte swap for the
format's byte order, emit - instead of one trip through
parse_ubjson_internal and get_ubjson_value per element. Elements the
input ends before, other types, and non-contiguous inputs are read as
before.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* UBJSON/BJData tests: assign the parsed value instead of casting it to void
GCC reports -Werror=unused-result for static_cast<void>(json::from_ubjson(...))
and json::from_bjdata(...): a cast to void does not silence warn_unused_result
on GCC. Assign to a json, as the other tests do.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* MessagePack reader: read runs of scalar array elements in one loop
As for CBOR: when an array opens and the input is contiguous, read the
integers, floats, booleans, and nils that follow in a tight loop instead
of going through parse_msgpack_internal and parse_msgpack_value for every
element. The loop stops at any other kind of value and when fewer than 9
bytes are left; those values are read by the general path as before.
Non-contiguous inputs are not affected.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* MessagePack reader: enter the scalar run only when a scalar follows
As for CBOR: enter_msgpack_array tests the first byte and the remaining length
inline and calls the out-of-line run only for a scalar, so arrays of maps or
strings no longer pay for a call that reads nothing; the run's loop condition
moves to the end (do-while).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* MessagePack reader: mark the initial accepted in the scalar run as a false-positive dead store
ci_infer reports DEAD_STORE for 'bool accepted = true;' in the MessagePack
scalar run, as in the CBOR one, because every case assigns accepted before it
is read.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* MessagePack tests: assign the parsed value instead of casting it to void
GCC (ci_test_gcc) reports -Werror=unused-result for
static_cast<void>(json::from_msgpack(...)): a cast to void does not silence
warn_unused_result on GCC. Assign to a json, as the other tests in the file do.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* MessagePack reader: drop the scalar pre-check before the run
enter_msgpack_array now calls the run directly, and parse_msgpack_scalar_run
tests the remaining count and length at the top of a while loop instead of
relying on msgpack_scalar_follows(); its own switch already leaves on any
value that is not a scalar.
Instructions per repetition, new/old: SAX canada 0.946 (clang) and 0.973
(GCC 13), DOM read canada 0.991 / 0.992; no row more than 0.3 % higher.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* MessagePack reader: say why the initial accepted in the scalar run stays
As for CBOR: Infer's DEAD_STORE report is right, not a false positive. The
initializer is kept for clang-tidy's cppcoreguidelines-init-variables.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* MessagePack reader: give the scalar run the bulk tag instead of a wrapper
As for CBOR: parse_msgpack_scalar_run takes the bulk-scan tag itself and is
the function kept out of line, so parse_msgpack_array_start goes.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* MessagePack reader: start the scalar run's accepted at false
As for CBOR: Infer ignores dead stores of 0, so the suppression goes.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* MessagePack reader: let the scalar run find its array itself
As for CBOR: parse_msgpack_scalar_run reads container_stack.back() in its
bulk version, so the caller no longer evaluates it for input that is not
contiguous.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* CBOR reader: read runs of scalar array elements in one loop
When a definite-length CBOR array opens and the input is contiguous, read
the integers, floats, booleans, and nulls that follow in a tight loop
instead of going through parse_cbor_internal's container logic and
parse_cbor_value's dispatch for every element. The loop stops at any other
kind of value and when fewer than 9 bytes are left; those values are read
by the general path exactly as before. Non-contiguous inputs are not
affected (tag dispatch on the bulk-scan capability).
Also pass a member empty string as the number source to number_float
instead of constructing a std::string for every float.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* CBOR reader: address review of the scalar-run reader
- parse_cbor_scalar_run: set the size explicitly in every case that
reads a value, instead of relying on a default that every other case
overwrites.
- Remove cbor_scalar_follows(): parse_cbor_scalar_run already returns on
the same conditions, and without the pre-check SAX canada needs another
8.5 % fewer instructions with GCC 13 (0.915 against the previous
version), with no row more than 0.5 % worse (Mac and GCC).
- Restore the comment on why parse_cbor_internal copies the container
frame.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* CBOR reader: enter the scalar run only when a scalar follows
An array of containers or strings called the out-of-line run start once per
array only to leave it at once; on nested data (arrays of objects) that call
made reading slower than before the run reader. enter_cbor_array now tests the
first byte and the remaining length inline and calls the run only for a
scalar, so the run's loop condition moves to the end (do-while).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* CBOR reader: mark the initial accepted in the scalar run as a false-positive dead store
ci_infer reports DEAD_STORE for 'bool accepted = true;' in
parse_cbor_scalar_run because every case assigns accepted before it is read.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* CBOR reader: drop the scalar pre-check before the run
enter_cbor_array calls the run directly again, and parse_cbor_scalar_run
tests the remaining count and length at the top of a while loop; its own
switch already leaves on any value that is not a scalar. The MessagePack
reader is built the same way.
Instructions per repetition, new/old: clang SAX canada 0.983, read canada
0.993; GCC 13 read canada 0.997, the other rows at most 1.004.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* CBOR reader: say why the initial accepted in the scalar run stays
Infer's DEAD_STORE report is right, not a false positive: every case that
reaches the read assigns accepted. The initializer is kept for clang-tidy's
cppcoreguidelines-init-variables.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* CBOR reader: give the scalar run the bulk tag instead of a wrapper
parse_cbor_scalar_run takes the bulk-scan tag itself and is the function that
is kept out of line, so the one-line parse_cbor_array_start wrapper goes, as
the UBJSON typed run already does.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* CBOR reader: start the scalar run's accepted at false
Infer ignores dead stores of 0 (--liveness-ignored-constant defaults to "0"),
and false is 0 to it, so the suppression and its comment go.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* CBOR reader: let the scalar run find its array itself
parse_cbor_scalar_run reads container_stack.back() in its bulk version, so
the caller no longer evaluates it for input that is not contiguous.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Niels Lohmann
