The 4 GiB limit and the fallback for an input that parse() accepts but
the view rejects (a bug) are excluded from the coverage; shrink_to_fit()
of an empty document is tested.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
- the input dispatch takes byte ranges by const reference and reads the
size once (which also settles a finding of the static analyzer); input
adapters are taken by value
- the classification of inputs keeps its nested conditional operators, a
constant expression of C++11 (NOLINT)
- the test's C arrays, fixed seed, and escaped literals are marked, as in
the other tests
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The public classes of the zero-copy view (#5295), in the new header
<nlohmann/json_view.hpp>:
- basic_json_document<BasicJsonType>: parse (borrowing contiguous byte
inputs, owning rvalue strings, streams, and other inputs), parse_copy,
accept, read, root, is_discarded, source, owns_source, node_count,
memory_usage, shrink_to_fit
- basic_json_view<BasicJsonType>: type and the is_* queries, size, empty,
materialize (the value parse() would produce, built by the same SAX
handler), source_offset
- the aliases json_document, json_view, ordered_json_document, and
ordered_json_view
A parse error throws the exception basic_json::parse would throw for the
same input: the library parser is run on the failing input, so messages,
positions, and exception ids are the same. Inputs of 4 GiB or more are
rejected with out_of_range.416.
The single header single_include/nlohmann/json_view.hpp keeps including
json.hpp; make amalgamate, check-amalgamation, include.zip, and release
handle it.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
GCC -Werror=useless-cast (ci_test_gcc on Linux x86-64) rejected
static_cast<std::size_t>(guess + (guess / 4) + 64): the sum is a
std::uint64_t prvalue, the same type as std::size_t there, while the cast
is needed where std::size_t is 32 bits wide. Cast a named variable
instead, which GCC does not report. The build stopped at an earlier error
before, so the previous CI run did not show this one.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
- msvc (Win32, /W4 /WX) reported C4127 (conditional expression is
constant) for `TrailingCommas && cur() == ']'` and the like when the
option is off. Route the template arguments through a static enabled()
function, as json.hpp's nesting_depth_exhausted() does.
- ci_test_single_header compiled unit-json_view_builder.cpp against
single_include/, which does not contain the internal
nlohmann/detail/view headers. Build that test only with the multiple
headers.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Since whole blocks of eight digits are read directly, parse_upto8() only
gets fewer than eight digits; its eight-digit case was dead code.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
A block of eight digits of a number token lies inside the input (the
digits were counted while scanning, or are recorded in the digit
layout), so parse_upto19() reads it without the bounds check of the
last, partial block. Traversing canada.json: -11% instructions, -6%
cycles.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
json::parse ends the input at a NUL only between values (where it does
at all); inside a string, a NUL is a control character that must be
escaped. The view reported it as a missing closing quote. (Only the
error code differed: the exception comes from the library parser.)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
- tables as std::array; the frames of the first 64 levels stay a C array
(not initialized on purpose, NOLINT)
- \u escapes are decoded with the library's hex_codepoint() instead of a
second table
- the parse failure is private, with an accessor; the special member
functions of the builder are all declared
- no nested conditional operators; explicit parentheses; a repeated
branch body merged; auto for casts
- the test's C arrays, fixed seed, and escaped literals are marked, as in
the other tests
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
With JSON_NOEXCEPTION, NLOHMANN_VIEW_THROW(e) was std::abort() alone, so
the parameters of the functions that build the exceptions were unused, a
warning that the builds with -Werror turn into an error. The exception is
now evaluated before std::abort(); the program ends anyway.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The builder parses JSON text in one pass into the node index: strings and
numbers stay in the source (escaped strings are decoded into an arena),
integers are converted while their digits are in cache, and floats keep
their digit layout for a later conversion. It accepts exactly what
json::parse accepts, for every combination of comments and trailing commas,
with and without a terminating NUL, and with JSON_STRICT_NUL_HANDLING.
Parse state lives in a local cursor whose address never escapes, so that it
stays in registers; out-of-line helpers (errors, regrowth, escapes,
comments) are members of the builder and get the positions they need. The
value dispatch is expanded once for array elements and once for member
values. Literals are compared with memcmp and words read in a fixed byte
order, so nothing depends on the platform's byte order. Error messages come
with the public classes.
Tests (unit-json_view_builder.cpp): accept/reject and values against
json::parse for handwritten, generated, and damaged documents under all
option combinations, from std::string and from exact-size buffers (no read
past the input under AddressSanitizer), deep nesting up to 100,000 levels,
NUL/BOM/whitespace cases, and the test-suite files.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Internal parts of the zero-copy view (#5295), under detail/view and not
included by json.hpp, so that users of json.hpp compile nothing of it:
- macro_scope.hpp/macro_unscope.hpp: the few macros the view needs, under
its own prefix (json.hpp undefines its own at its end); the throw macro
honors JSON_NOEXCEPTION and JSON_THROW_USER like JSON_THROW
- string_ref.hpp: std::string_view from C++17 on, else a small stand-in
- node.hpp: the 16-byte node of the index; its kinds are value_t values
(checked by a static_assert)
- document_data.hpp: the storage of a parsed document (node array, decode
arena, owned input)
- scan.hpp: string and digit scanning with unrolled checks at fixed offsets
(after yyjson) and the library's SWAR and UTF-8 checks, independent of the
byte order
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
json.hpp undefines JSON_STRICT_NUL_HANDLING and
JSON_USE_LEGACY_DISCARDED_VALUE_COMPARISON at its end. Code that builds on
the library after it, such as the planned json_view.hpp, reads them from
detail::abi_config instead. The constants live in the ABI namespace, which
already encodes both settings, so they always match the basic_json in use.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
get_codepoint() read the four hex digits of a \u escape with four calls
to get(), each classified by a chain of range comparisons. For contiguous
input, get_codepoint_bulk() now decodes them with one lookup per byte
(hex_codepoint() in string_scan.hpp, after yyjson's read_hex_u16): a
256-entry table maps a byte to its value, or 0xFF for anything else, and
an invalid digit shows in the OR of the four values. It then skips the
four bytes and updates the position counters as four get() calls would.
If a digit is invalid or fewer than four bytes are left, it changes
nothing and the existing loop runs, so errors are reported with the same
message and position as before.
json::parse, best of 5 runs in separate processes (M1 Max): the escaped
twitter.json (every non-ASCII character as \u) -13.6%, all other files
within 0.3%.
Tests compare the contiguous and the streaming path (value or exception
message) for valid escapes, surrogate pairs, truncated and invalid digits
at every position, and 3,000 seeded random escapes.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
find_string_special() and find_ascii_copyable_run() test eight bytes at a
time, but located the stopping byte inside a word with a byte loop. The
lowest flagged byte of the SWAR tests is always a true hit (the borrows of
the subtractions can only flag bytes above one), so its index is now the
trailing-zero count of the mask; words are read in little-endian order on
every platform, so this does not depend on the byte order.
scalar_string_bulk_run() validates a run of multi-byte UTF-8 sequences one
after another instead of searching for the next special byte in between,
which helps text in non-Latin scripts.
The kernels serve the lexer's contiguous fast path, the serializer, and the
binary formats. New tests compare all three with byte-by-byte reference
scans on 100,000 generated buffers at three alignments; the portable
fallback of count_trailing_zeros() was checked against the builtin.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The strtold fallback, which is left only for long double formats that
are not binary64 (x87, binary128), substituted the first byte of the
locale's decimal point for '.'. Under a locale whose decimal point is
longer than one byte, such as fa_IR.UTF-8 or ar_EG.UTF-8 (U+066B),
strtold stopped there and the value was truncated at the decimal point.
A longer decimal point is now put into a copy of the token.
The test "locale with a multi-byte decimal point" now compares the long
double values with those of the "C" locale; with x87 long doubles it
failed before.
Fixes#5660.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
float, double, and long double where it is IEEE-754 binary64 (MSVC, Apple
arm64) are now converted by the library itself, correctly rounded and
independent of the locale and of the C and C++ libraries:
- The token is split into sign, significand w (at most 19 digits), and
decimal exponent q, using the positions of the decimal point and the
exponent that the scanners already recorded, so no character is
classified again.
- Clinger's fast path where w and 10^|q| are exact.
- Eisel-Lemire otherwise, now templated for binary32 and binary64.
- For tokens with more than 19 digits whose w and w + 1 round differently,
an exact big-integer comparison with the midpoint between the two
candidates (the digit comparison of fast_float, simplified).
This replaces the separate token walks of Clinger's fast path and of
Eisel-Lemire, the significant-digit gate that avoided the former, and, for
float and double, std::from_chars and the locale-aware strtod. std::from_chars
and strtold remain only for other long double formats (x87, binary128,
double-double) and for types that are not IEEE-754. Values are bit-identical
to before wherever the previous conversion was correctly rounded; tokens
converted in a locale with a multi-byte decimal point are now also exact.
Overflow still gives out_of_range.406, underflow a signed zero.
convert_float() is the entry point for other parsers of JSON text: it
converts like the lexer, without allocation for binary32/binary64.
Tests: exact-bit tests for double and float (ties, subnormal and overflow
boundaries, huge exponents, more digits than any midpoint), Eisel-Lemire for
binary32, the round trips of 200,000 doubles and 100,000 floats without
declines, 508 generated hard cases with the expected bits of both formats
(float_hard_cases.hpp) through the converter and both scanners, and
JSON-level overflow/underflow checks for double and float. The locale tests
now check the values in a locale with a multi-byte decimal point.
Docs: the statements that parsing uses strtod/strtof/strtold; the fast_float
credit now names the digit comparison.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
contains(const json_pointer&) and operator/=(std::size_t) (and hence
operator/(std::size_t)) used string_t operations that the StringType
template parameter documentation explicitly does not require:
comparing string_t with a const char* literal, c_str(), and
constructibility from std::string. This made both functions fail to
compile for a conforming custom StringType, even though the
documentation's own reference StringType satisfies the requirements.
Fix contains() to compare individual chars ('0'..'9') instead of
comparing string_t with const char* literals, and to call data()
(documented to be null-terminated) instead of c_str(). Fix
operator/=(std::size_t) to build the array-index token via the
existing detail::to_string<StringType> helper (ADL int_to_string() or
assignment from std::to_string()) instead of via std::to_string()
directly, matching how diff(), items(), and std::hash already convert
a std::size_t to a StringType.
Add regression tests to tests/src/unit-alt-string.cpp: contains() for
present/missing keys and indices, "-", a leading zero, and a
non-numeric token on an array, plus json_pointer::operator/(std::size_t).
Fixes#5666.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Add JSON_NO_UDLS to leave out the user-defined string literals
The bodies of operator""_json and operator""_json_pointer call the
parser, so every translation unit including the library instantiates it,
even if it never parses anything. Defining JSON_NO_UDLS leaves the
literals out entirely, which saves 15-35% compile time for such
translation units (#5294). Nothing changes if the macro is not defined.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Mention JSON_NO_UDLS in the list of exported module symbols
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Move the user-defined string literals to <nlohmann/json_literals.hpp>
Following the review in #5294, the literals now live in their own header
instead of being removed entirely: <nlohmann/json.hpp> includes it at the
end unless JSON_NO_AUTOMATIC_UDLS (renamed from JSON_NO_UDLS) is defined,
so a project can opt out globally and include the header only where the
literals are used.
The header only uses public and standard macros, because the library's
internal macros are undefined at the end of json.hpp and the amalgamation
inlines macro_scope.hpp only once. For the same reason, the library no
longer defines and undefines JSON_USE_GLOBAL_UDLS, so a user's definition
is still visible to the header. The single-header copy is identical to
the multi-header one, as it only includes <nlohmann/json.hpp>. The module
always exports the literals.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix CI: include cycle, GCC 4.8 literal operator spacing, and global UDLs off in the JSON_NO_AUTOMATIC_UDLS test
- Suppress clang-tidy misc-header-include-cycle on the intentional mutual
include of json.hpp and json_literals.hpp.
- Use operator"" _json with a space for GCC 4.8 in the test's detection
aliases, as the header does.
- Only test the global literal operators when JSON_USE_GLOBAL_UDLS is on.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Declare the literal operators through a local macro
The GCC 4.8 spacing condition was repeated for both operator definitions
and the global using-declarations. NLOHMANN_JSON_LITERAL_OPERATOR(suffix)
now selects operator""##suffix or operator"" suffix in one place and is
undefined at the end of json_literals.hpp.
Suggested by gregmarr in review.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
When a parser callback rejects an object's or array's start event,
json_sax_dom_callback_parser kept calling it for everything inside
that container anyway: nested keys, values, and the start/end events
of containers below it. This contradicts parser_callback_t's own
documentation, which promises that discarding a container at its
start event also hides its content from the callback.
The same code path also kept a full copy of every key inside such a
discarded container in key_stack until the whole parse finished,
because the early return for values that are not stored skipped the
matching pop. Filtering out a large subtree is the main reason to use
a callback, so this made peak memory during the parse scale with the
size of the very subtree the callback was trying to skip.
Fix start_object(), start_array(), and key() so that a container
whose own start event was discarded, or that is nested inside one, is
never handed to the callback, and no longer pushes onto the key
stacks. A container whose start event was accepted but whose key was
rejected still gets its content reported, as documented ("the
callback is still called for the associated value, but its return
value has no further effect"); only its own bookkeeping is skipped
since it will not be stored.
Fixes#5643.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
to_bson() rejected a binary value's subtype above 255 (out_of_range.415)
in write_bson_binary(), which only has the binary_t, not the basic_json
value that holds it, so the exception was created with no JSON_DIAGNOSTICS
context even though the equivalent to_msgpack() check names the value's
path. The check also ran after the document size, all preceding elements,
and this element's header and length had already reached the output
adapter, so a caller-provided std::vector or std::string ended up holding
a truncated document.
calc_bson_sizes() already walks every value before anything is written,
to size embedded documents and arrays and to reject invalid keys
(out_of_range.409) up front. The subtype check now runs there instead,
in calc_bson_binary_size(), which is given the basic_json value so the
exception can use it as context. The now-redundant check in
write_bson_binary() is removed, since calc_bson_sizes() always throws
first if any binary value in the document has an oversized subtype.
Fixes#5675.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
With JSON_STRICT_NUL_HANDLING defined to 1, parsing a wide, UTF-16,
UTF-32, or (C++20) UTF-8 string literal (e.g. json::parse(L"[1]"))
failed with parse_error.101 at the terminating NUL of the literal,
and accept() returned false. The array overload of input_adapter()
only dropped the compiler-added trailing '\0' for arrays of char,
so for wchar_t, char16_t, char32_t, and char8_t arrays that
terminator was passed to the parser as data, which the macro then
rejected.
Broaden the trimming to every character type that a string literal
can use (char, wchar_t, char16_t, char32_t, and, since C++20,
char8_t). Arrays of any other element type (unsigned char,
std::uint8_t, ...), as used for CBOR/MessagePack, are unaffected: a
trailing zero byte there is still read as genuine data.
Fixes#5658.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Parsing from a std::istream crashed in two unusual but valid stream
states, both in input_stream_adapter:
- With eofbit in the stream's exceptions() mask, get_character() sets
eofbit via is->clear(), which throws std::ios_base::failure. While
that exception unwinds, ~input_stream_adapter() called clear() again
to reset eofbit, which is still set and still in the exception mask,
so it throws a second time out of the (implicitly noexcept)
destructor and std::terminate() is called. The destructor now only
calls clear() if a bit other than eofbit remains set, so the first
exception can propagate normally.
- For an std::istream without a stream buffer (rdbuf() == nullptr,
e.g. std::istream(nullptr)), the constructor stored the null
pointer without checking it, and get_character() dereferenced it.
input_adapter(std::istream&) now throws parse_error.101 for such a
stream, the same as it already does for a null FILE* or char*.
Added regression tests to unit-deserialization.cpp and, for the
JSON_PRECISE_STREAM_POSITION variant of get_character(), to
unit-precise-stream-position.cpp; both crashed before this fix.
Documented the two exceptions in parse.md and operator_gtgt.md.
Fixes#5646.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
from_json built its out_of_range.410 message with "..." + j.dump(). If the
unmatched value is (or contains) a string with invalid UTF-8, that dump()
itself throws type_error.316, so the caller got type_error.316 instead of
the documented out_of_range.410; such strings can reach get<Enum>()
unvalidated, e.g. from from_cbor()/from_msgpack(). With a custom string_t,
j.dump() returns that type, and "const char*" + string_t does not compile
unless the type happens to provide operator+, so the macro failed to
compile for such types.
Build the message with detail::concat(), which appends any type exposing
data()/size() and always yields a std::string, and dump with
error_handler_t::replace so building the message itself cannot throw.
Added regression tests: an invalid-UTF-8 case in the existing strict-enum
test in unit-conversions.cpp, and a strict-enum use with alt_string (the
custom string_t from unit-alt-string.cpp) to cover the compile failure.
Fixes#5667.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix to_msgpack() reading the inactive number union member
basic_json stores number_integer and number_unsigned in a union, and
number_unsigned_t only has to be at least as wide as number_integer_t
(with the default types, both are 64-bit and have the same
representation). When number_integer_t is narrower, write_msgpack()
read the wrong union member in two places:
- The number_unsigned case wrote number_integer's bits instead of
number_unsigned's, silently writing the wrong value whenever it
did not fit in number_integer_t.
- The number_integer case (non-negative branch) picked the encoded
width by comparing number_unsigned's bits, which is undefined
behavior, though the value written was still number_integer's, so
at worst a too-wide encoding was chosen.
Read the active member in both cases, like the other binary writers
(CBOR, UBJSON, BJData, BSON, BON8) already do.
Fixes#5644.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Cast number_integer to number_unsigned_t only once in to_msgpack()
Addresses review comment by @gregmarr.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Let to_json(std::optional<T>) propagate exceptions from T's to_json
The overload was marked noexcept even though its body assigns *opt to
the JSON value, which calls T's to_json (or allocates for std::string,
std::vector, or json). Any exception from there -- a user-defined
to_json reporting an error, or std::bad_alloc -- called std::terminate()
instead of propagating. The noexcept also made basic_json's converting
constructor noexcept(true) for std::optional<T>, so json j = opt; could
not report the error either.
Fixes#5642.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix CI: make to_json(std::optional<T>) conditionally noexcept
GCC's -Wnoexcept (an error in ci_test_gcc) fired at to_json_fn's
noexcept(noexcept(to_json(j, val))): after dropping the unconditional
noexcept, to_json(std::optional<int>) had no exception specification
although GCC could prove its body cannot throw. It also made
json(std::optional<int>) lose its noexcept.
Declare the overload noexcept exactly when assigning the contained
value to the JSON value is (std::is_nothrow_assignable<BasicJsonType&,
const T&>), which is what the body does. std::optional<int> is noexcept
again; a T whose to_json may throw still propagates the exception.
Static assertions in the test check both cases.
The test's throwing to_json triggered -Wmissing-prototypes and
-Wmissing-noreturn (clang) and -Wmissing-declarations and
-Wsuggest-attribute=noreturn (GCC). Move the type and its to_json into
an anonymous namespace, mark the function [[noreturn]], and compile
them only without JSON_NOEXCEPTION, like the test that uses them.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
value(KeyType&&, default) is constrained on is_comparable_with_object_key,
which passes KeyType as a reference. is_comparable's dispatch on
is_json_pointer_of<A, B> only matches a json_pointer as a plain type or a
plain reference, so a const-qualified reference (as produced when KeyType
is deduced from a json_pointer argument) fell through to
is_comparable_no_json_pointer, which instantiates the deprecated
json_pointer/string comparison operators. This made ordered_json's
transparent comparator (and any transparent comparator on a custom string
type) warn under -Wdeprecated-declarations when calling
value(json_pointer, default), even though no such comparison is ever
performed. at() was already fixed for this in #5289, which does not use
is_comparable_with_object_key.
Strip references and cv-qualifiers with uncvref_t before the
is_json_pointer_of dispatch, so any reference-to-json_pointer is
recognized regardless of qualifiers.
Fixes#5664.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
When converting a JSON array to std::map or std::unordered_map with a
non-string key, each element must itself be a [key, value] array. If an
element is not an array, from_json() threw type_error 302 with the outer
array's value (&j) as the exception context, so with JSON_DIAGNOSTICS
enabled the message pointed at the whole array instead of the offending
element (e.g. "(/outer/m)" instead of "(/outer/m/2)"), even though the
message text already described the element's type.
Both from_json() overloads now pass the element (&p) as the context, so
the reported JSON Pointer matches the type named in the message, the
same way std::vector<std::vector<T>> and similar conversions already do.
Fixes#5668.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
get_arithmetic_value() rejects boolean_t, so the default from_json()
for enums failed to compile for an enum whose underlying type is
bool (e.g. enum class Flag : bool { off, on }), even though the
matching to_json() serializes such enums as an unsigned number.
Read the underlying value through number_unsigned_t in that case,
matching what to_json() writes, then cast back to the underlying
type before constructing the enum.
Fixes#5671.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Copying and comparing stopped their descent at
basic_json::nesting_depth_limit(), while serializing, hashing and
merging used detail::recursion_depth_limit(). Both were 128, but
nothing kept them equal. The thread-local count now tests against
detail::recursion_depth_limit() as well, and a static_assert keeps
the limit small enough for the byte that holds the count.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Do not throw in contains() for an empty array reference token
json_pointer::contains() rejected malformed array indices, but an empty
reference token (e.g. "/a/" where "a" is an array, or "/" on an array)
passed every check and reached array_index(), which throws
out_of_range.404. contains() must not throw (cf. #5395), so it now
returns false for an empty token. at() still throws out_of_range.404.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix CI: bind j_nested_const by reference in the json_pointer test
clang-tidy (ci_clang_tidy) flagged the new test with
performance-unnecessary-copy-initialization: the local copy
j_nested_const of j_nested is never modified. Bind it as a const
reference instead; it still exercises the const overloads of at() and
contains().
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Convert floats with Eisel-Lemire when std::from_chars is unavailable
Float tokens that Clinger's fast path cannot convert (e.g. the 17-digit
coordinates of canada.json) went to strtod unless std::from_chars was
available. It is not used in C++11/14, and not with libc++, which does not
define __cpp_lib_to_chars. The Eisel-Lemire algorithm (after fast_float's
compute_float) now converts them with integer arithmetic, correctly rounded
for any token with at most 19 significant digits. Longer tokens are
truncated; the result is used if w and w + 1 round alike, else strtod
decides as before. Overflow still yields infinity (out_of_range.406).
The table of powers of five (fast_float's) lives in pow5_table.hpp; a unit
test recomputes every entry with big-integer arithmetic. Further tests:
known values generated with Python (whose float() is correctly rounded),
200,000 round trips through to_chars, and the 128-bit multiplication and
leading-zero count against big-integer references (both with and without a
128-bit type). Checked against strtod on 6.5 million tokens, among them
60,000 exact halfway cases: no difference.
json::parse on canada.json: -8.6% (C++11), -7.6% (C++17, Apple clang);
other files unchanged. Compile time of a TU including json.hpp: +0.7%.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix CI: unused parse result and find() == npos in the Eisel-Lemire tests
GCC (-Werror=unused-result) rejected CHECK_THROWS_WITH_AS(json::parse(...))
because parse() is [[nodiscard]]; assign the result to a dummy json as the
other tests do. clang-tidy flagged longer.find('.') == npos with
abseil-string-find-str-contains; store the position in a variable first.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
lexer::convert_number() converted float tokens with std::from_chars (when
available), Clinger's fast path, and the locale-aware strtod fallback, all
as lexer members. They are now free functions in number_parse.hpp:
- convert_float_fast(): std::from_chars, then Clinger's fast path, skipped
when the mantissa has too many significant digits
- convert_float_locale_aware(): strtof/strtod/strtold with the decimal point
of the current locale, retried when the locale changed (#5198)
so that other code converting JSON number tokens gets the same values. No
change in behavior; the lexer no longer includes <clocale> and <cstdlib>.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Look up the locale decimal point at conversion time, not lexer construction
The lexer read localeconv()->decimal_point once in its constructor and wrote
that character into token_buffer in place of '.'. The strtod fallback then
used the locale current at conversion time, so an LC_NUMERIC change in
between (parser callback, SAX handler, another thread) truncated the value
in release builds and fired the endptr assertion in debug builds.
token_buffer now always holds '.'. Only the strtof/strtod/strtold fallback
depends on the locale: it looks up the decimal point right before the call,
restores '.' afterwards, and repeats the conversion if the locale changed in
between. As a side effect, std::from_chars and Clinger's fast path now also
apply under locales whose decimal point is not '.'.
Fixes#5198
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Stop the strtod retry loop when the decimal point is unchanged
convert_float_locale_aware() repeated the conversion until strtod
consumed the whole token, assuming an early stop can only mean a locale
change. Under a locale whose decimal point is not a single character
(e.g. the two-byte U+066B of ar_EG.UTF-8, ar_SA.UTF-8, or fa_IR.UTF-8,
all available on macOS), the in-place substitution can never succeed,
so parsing any float that reaches the strtod fallback (for example
3.14159265358979323846 at C++11) hung forever. Before this branch, the
same input was truncated.
Retry only if the decimal point changed since the previous attempt;
otherwise keep the value strtod parsed so far, as before. Add a test
that parses such numbers under a multi-byte decimal point locale; it
hangs without this change.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix -Weffc++ errors in the #5198 locale test
GCC's -Weffc++ (an error in ci_test_gcc and ci_test_standards_gcc)
rejected LocaleSwitchingSax: it has a pointer data member but does not
declare its copy operations, and its vectors are not initialized in the
member initializer list. Store the locale name as a std::string and give
the vectors brace initializers, like SaxEventLogger in
unit-deserialization.cpp.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Name the key type when rejecting non-string CBOR/MessagePack map keys
CBOR and MessagePack allow map keys of any type, but JSON object keys
are always strings, so such maps are rejected. The error so far was the
one for a malformed string (e.g. "expected length specification
(0xA0-0xBF, 0xD9-0xDB); last byte: 0xC0" for a nil key), which does not
tell the user what went wrong. Report the type of the key instead:
syntax error while parsing MessagePack object key: only string keys
are supported, but found nil; last byte: 0xC0
The exception id (parse_error.113) and type are unchanged. Malformed
string keys and a missing key keep their previous messages. Document
the restriction on the CBOR and MessagePack pages.
Refs #2766, #3381
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Point the MessagePack key note to the spec's profile section
The note linked to "Serialization: type to format conversion", which says nothing about key types. Restricting map keys to strings is only mentioned in the "Profile" section (under "Future discussion") as an example of a JSON-compatible profile, so link there and describe it as such instead of as a permission.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Add BON8 support
Add to_bon8/from_bon8 and input_format_t::bon8 for BON8, a binary format
that uses the byte values that cannot begin a UTF-8 character as type
markers, so strings need no length prefix. It is the most compact of the
supported binary formats on the benchmark files.
The reader is non-recursive like the other binary readers. A string ends
at the first byte that cannot continue it, so the reader hands the one or
two bytes it reads past a string back to the value that follows. The
writer produces the canonical representation of the specification, except
for NFC normalization; its output is identical to that of the reference
implementation (HikoGUI) on all files of the test data.
The round-trip tests need the .bon8 files of json_test_data 3.2.0.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address review comments
- Reuse detail::validate_one_utf8 to check strings in to_bon8; the error
now names the first byte of the invalid sequence.
- Document that to_bon8 leaves bytes in the output adapter on an
exception, and that string_open is only an output of write_bon8_marker.
- Explain why the pushback buffer of the BON8 reader cannot overflow.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Select the BON8 float prefix by type
get_bon8_float_prefix only depends on the type of its argument, so make
the type a template parameter instead of passing an unused value.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Rename a test variable that Flawfinder mistakes for read()
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix the BON8 CI failures
- compare the float in write_bon8_float with number_float_t constants,
so GCC does not warn about a float-to-double conversion
- mark check_bon8_utf8's context as used when exceptions are disabled
- choose the compact float prefix in a helper rather than with nested
conditional operators (clang-tidy)
- use auto for the cast in the BON8 integer reader (clang-tidy)
- write the int32 minimum test values as long long literals (MSVC C4146)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Amalgamate
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Read BON8 strings in bulk from contiguous input
- copy the valid UTF-8 of a string in one step when the input is
contiguous (twitter.json is read in 1.68 instead of 2.52 ms,
jeopardy.json in 196 instead of 297 ms, close to CBOR and MessagePack)
- share the new valid_utf8_prefix() with the writer's UTF-8 check, which
now skips ASCII 8 bytes at a time
- let the fuzzer check that contiguous and stream input give the same
value or error, and test both paths in the unit tests
- clarify that a second 0xFF after a string is an empty string
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Link the BON8 functions from the other binary format pages
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Name the bulk scan flag after the input, not BON8
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Read BSON keys in bulk from contiguous input
BSON keys (and array indices) are C-style strings, which were read byte
by byte. For contiguous input they are now read up to their \x00-byte in
one step, using the same bulk_scan flag as BON8 strings: twitter.json is
read in 1.46 instead of 2.01 ms, citm_catalog.json in 2.93 instead of
3.33 ms, jeopardy.json in 182 instead of 207 ms. canada.json, whose keys
are almost all one-digit array indices, takes 2 % longer.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix the BON8 CI failures of the bulk-read tests
- skip the contiguous-versus-stream tests of BON8 strings and BSON keys
when exceptions are disabled: they catch the parse errors of invalid
input, and without exceptions the library aborts instead
- use static_cast for the int64 test value (google-readability-casting)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Move the explicit basic_json instantiation into its own test file
Linking test-regression3_cpp20 with clang and MinGW failed with
"relocation truncated to fit: IMAGE_REL_AMD64_REL32 against `.rdata'",
as test-regression2 did before #5511. The explicit instantiation of
basic_json<> for #4825 compiles every member function, including the
BON8 reader and writer, into that object, and it was already close to
the limit (2,226,104 bytes on develop, 2,234,960 with BON8; clang -O1,
C++20).
Give the instantiation a file of its own: unit-regression3 is now
1,594,736 bytes and unit-explicit_instantiation 1,095,064. The new file
mentions JSON_HAS_CPP_17 and JSON_HAS_CPP_20 so it keeps being built
for the C++17 standard the regression was about.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Convert the bytes of the BON8 test strings explicitly
The str() helper constructed a std::string from a byte range, which
converts each unsigned char implicitly; -fsanitize=integer reports that
for bytes of 0x80 and above (ci_test_clang_sanitizer).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
When using cbor_tag_handler_t::store, tags 0xD8-0xDB previously assumed
that the tagged item was a byte string, unconditionally attempting to
parse binary data and failing on valid CBOR documents containing tags
applied to integers, strings, arrays, or objects (such as self-describe
tag 55799).
Check whether the tagged data item is a byte string (0x40-0x5B or 0x5F).
If it is a byte string, store the subtype on the binary value as before.
Otherwise, iteratively process the tagged value in the driver loop using
item_read so that chained tags do not consume native stack space.
Part of #5316.
Signed-off-by: ReturnKartikey <kartikeynegi2000.work@gmail.com>
* Throw instead of writing MessagePack lengths beyond UINT32_MAX
MessagePack stores the length of a string, binary value, array, or
object in at most 32 bits. For a larger value, to_msgpack wrote no length
at all, so the output could not be read back. It now throws
out_of_range.412, which BSON already uses for its 32-bit length fields.
The check lives in one function, so each length is written by an
if/else chain that ends in a plain else, without a condition that can
never be false. It is tested with string and binary types that report a
size beyond UINT32_MAX without allocating it, like the BSON tests do.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix the CI failures of the MessagePack length check
- mark to_msgpack_length's value as used when exceptions are disabled
(-Wunused-parameter, misc-unused-parameters)
- put "Exception safety" before "Exceptions" in to_msgpack.md, as the
documentation style check requires
- create the test's string value from its type: constructing it from a
beyond_uint32_string_t considers the std::filesystem::path conversion,
which libstdc++ 10 reports as ambiguous for a class derived from
std::string (clang 13)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Skip the MessagePack string length test for clang with libstdc++ 10
C++17 builds consider the std::filesystem::path conversion for the
string type, and with clang and libstdc++ 10 that conversion is
ambiguous for a class derived from std::string. Creating the value from
its type did not avoid it, since any basic_json with that string type
instantiates the check. The binary and ext cases are still tested there.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Keep the MessagePack string test type and its alias in one block
astyle indented the alias oddly when it had an #ifdef of its own after
the binary alias; declare it right after the string type, in the same
block.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Coverage reported conditions in the binary writer that can never be
false, and marked the code behind them with LCOV_EXCL. Remove them
instead of excluding them:
- CBOR writes the length of a string, binary value, array, or object
exactly like an unsigned integer, only with another major type. One
function, write_cbor_head(), now writes both, so the integer tests
cover every width and the four excluded 64-bit length branches are
gone.
- A last `else if` whose condition holds for every remaining value
(an unsigned value at most UINT64_MAX, a signed one in the range of
int64_t) is now a plain `else`.
- Whether a signed integer fits into an int64 for UBJSON and BJData is
decided by its type at compile time. Only an integer type wider than
64 bits gets a range check and the high-precision fallback.
- The private get_impl(boolean_t*) was never called.
The UBJSON type prefix 'H' of an optimized container of unsigned
integers beyond the range of int64 was reachable although excluded; it
is tested now.
The output is unchanged.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Write BSON in linear time, without recursing per nesting level
to_bson() had two problems with nested values:
- It recursed once per nesting level, so a value nested deeply enough -
100,000 levels on an 8 MiB stack - exhausted the call stack and
terminated the process, although parse() accepts such values without
complaint.
- BSON prefixes every document and array with its length. The writer
computed that length by walking the entire value below it, again for
every nested document it wrote, which made serializing O(size x depth).
A 200-level document took 30 ms instead of 1.
Both passes are now iterative, and each length is computed exactly once:
- calc_bson_sizes() computes the length of every document and array in
one pass, each from the lengths of its entries, into a table ordered
the way they are written.
- write_bson_document() then writes the document, taking each length from
the table.
Everything observable is unchanged, as a differential test against
develop confirms byte for byte:
- The same bytes are written.
- A key containing U+0000 still throws out_of_range.409 for the same
first key, with the same diagnostics path, before anything is written.
- A document too large for BSON still throws out_of_range.412 before
anything is written.
- A binary subtype above 255 still throws out_of_range.415 after the
same partial output.
Only the enclosing objects and arrays are kept on a stack, so a flat
document allocates nothing for it. Measured against develop (clang -O3,
median of 201 runs): flat objects unchanged, flat arrays 37% faster (the
array length was computed twice), a nested 3,000-object document 2x
faster, a 200-level document 33x faster.
to_bson.md documented the quadratic complexity since #5334; it is linear
again.
Fixes#5392 for BSON, and #5308.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Do not require a default-constructible string_t in the BSON writer
GCC 4.9 and MSVC rejected the test's huge_string_t, which has no default
constructor; develop never default-constructed string_t here either.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Let the BSON index-name helper only fill its output parameter
It returned a reference to the string it filled, so callers held a second
name for index_name. Addresses review feedback.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Make JSON_STRICT_NUL_HANDLING part of the ABI tag
JSON_STRICT_NUL_HANDLING (#5534) changes the bodies of inline functions:
the lexer's handling of '\0' and input_adapter() for char arrays. So
translation units compiled with and without it define the same functions
differently, an ODR violation - the case the ABI tag exists for, as with
JSON_BRACE_INIT_COPY_SEMANTICS (_bics). It now appends _snul to the inline
namespace. The macro is new in 3.13.0, so no existing namespace changes.
Its default moves to abi_macros.hpp, and it is only #undef'd without
JSON_TEST_KEEP_MACROS, as for the other ABI macros. The ABI config tests,
the namespace docs, the macro's docs and the Natvis file cover the new tag.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Amalgamate
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* 🐛 fix BSON conformance issue
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* 🐛 fix BSON conformance issue
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* 🐛 reject ill-formed UTF-8 in CBOR/MessagePack/BSON text strings at decode time (#5531)
from_cbor()/from_msgpack()/from_bson() copied the raw bytes of a decoded
text string into the resulting json value without any UTF-8 validation,
even though RFC 8949 §3.1 (CBOR) and the MessagePack/BSON specifications
all require text strings to be valid UTF-8. Malformed input only failed
later, if the value was dump()'d, with a type_error.316 - so the
allow_exceptions=false pattern used specifically to get a discarded
sentinel instead of an exception did not discard this category of
malformed input, unlike every other kind of malformed binary input this
library rejects at decode time (see #5529).
Fix this at the single choke point shared by BSON/CBOR/MessagePack/UBJSON
string reads, binary_reader::get_string(): validate the bytes with the
UTF-8 DFA right after they are read, and report failures the same way as
every other binary_reader error (parse_error.113), so allow_exceptions
and strict discarding behave consistently. get_binary()/binary blob reads
are untouched and still accept arbitrary bytes, since only text strings
are required to be UTF-8.
There were two independent implementations of a UTF-8 validator: the
lexer's streaming scanner, and the serializer's Hoehrmann DFA used by
dump_escaped_impl(). Rather than write a third, the serializer's decode()
function, its utf8d table and the UTF8_ACCEPT/UTF8_REJECT constants are
extracted into detail/string_utils.hpp (a low-level header already
included before both detail/input/ and detail/output/), alongside a new
is_valid_utf8() helper built on the same decode() step. serializer.hpp's
dump_escaped_impl() now calls the shared decode(), so there is exactly
one UTF-8 validator in the codebase; dump()'s exact type_error.316
messages and byte-index reporting are unchanged (see the added
regression-guard test in unit-serialization.cpp).
Claude-Session: https://claude.ai/code/session_01N4RQ1Ahan5YAGbnAQGjZTY
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* ⚡ validate only newly read bytes of binary-format strings
get_string() validated the whole result after each call, but get_bytes()
appends to it and CBOR indefinite-length strings collect all chunks in
the same result, so every chunk re-validated everything read before it.
An input of many small chunks took quadratic time (80000 one-byte chunks,
160 KB of input, took about 7 seconds). Only the newly read bytes are
validated now, which also matches RFC 8949's requirement that every
chunk is valid UTF-8 on its own.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Support zero-member types in NLOHMANN_DEFINE_TYPE_* macros (#4041)
NLOHMANN_DEFINE_TYPE_INTRUSIVE(Type) and its 11 sibling macros produced
broken code for types with no members to serialize. Invoking a variadic
macro so __VA_ARGS__ is empty is only standard-conforming since C++20,
so a plain __VA_OPT__ fix (as tried in #5142) breaks every pre-C++20
build under -pedantic. Instead, make all 12 macros purely variadic and
dispatch on argument count using a sentinel-padded extension of the
existing NLOHMANN_JSON_GET_MACRO idiom, giving full C++11-C++26 support
with no feature-test gate.
Verified against real GCC 16 and Clang at -std=c++11/14/17/20 with
-pedantic -Werror -Wvariadic-macros: zero regressions in the existing
unit-udt_macro.cpp suite plus 12 new zero-member test cases.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix CI failures in zero-member NLOHMANN_DEFINE_TYPE_* macros
Three issues surfaced on PR #5272's real CI that weren't caught by
local testing against a narrower flag set:
- GCC -Werror=noexcept: the four truly-empty from_json bodies (plain
INTRUSIVE/NON_INTRUSIVE, with and without _WITH_DEFAULT) provably
never throw but weren't declared noexcept; mark them noexcept
explicitly. to_json and the derived-type from_json overloads are
left alone since they genuinely can throw (object assignment /
delegating to the base class's from_json).
- clang-tidy bugprone-macro-parentheses: false positive on the same
8 zero-member bodies (Type/BaseType used purely as declarator
types); suppressed with NOLINTNEXTLINE comments in the same style
already used elsewhere in this file (see NLOHMANN_JSON_SERIALIZE_ENUM).
- MSVC's traditional preprocessor doesn't fully expand
NLOHMANN_JSON_CAT(prefix, NLOHMANN_JSON_TYPE_TAG(...))(...) in one
pass, which broke a pre-existing one-member usage in
unit-regression2.cpp with syntax errors. Wrap all 12 public
dispatcher macros in an extra outer NLOHMANN_JSON_EXPAND(...),
matching the pattern NLOHMANN_JSON_PASTE already uses for the same
MSVC quirk.
Re-verified against real GCC 16 and Clang at -std=c++11/14/17/20 with
-pedantic -Werror -Wvariadic-macros -Wnoexcept, including the exact
files that failed in CI (unit-udt_macro.cpp, unit-regression2.cpp),
against both the modular headers and the re-amalgamated single header.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix clang-tidy misc-const-correctness in unit-udt_macro.cpp
The four zero-member ONLY_SERIALIZE test objects are only ever read
(via to_json), never mutated, so mark them const per clang-tidy.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix derived-type macro dispatch capping members at 62 instead of 63
NLOHMANN_JSON_GET_MACRO resolves 64 positional arguments, with NAME at
position 65. NLOHMANN_JSON_TYPE_TAG dispatches on Type plus the member
list, so it resolves correctly up to the 63 members NLOHMANN_JSON_PASTE
supports. NLOHMANN_JSON_DERIVED_TYPE_TAG dispatched on the two-token
Type,BaseType prefix plus the member list, running out one slot early:
at 63 members, position 65 landed on the last member name instead of a
sentinel and NLOHMANN_JSON_CAT built an undefined identifier such as
NLOHMANN_JSON_DEFINE_DERIVED_TYPE_INTRUSIVE_m63, with the compiler
reporting "unknown type name 'm1'" once per member and nothing pointing
at an argument-count limit.
That silently reduced all six NLOHMANN_DEFINE_DERIVED_TYPE_* macros from
63 members to 62, contradicting the "up to 63 members" contract in
docs/mkdocs/docs/api/macros/nlohmann_define_derived_type.md.
Drop the leading Type and defer to NLOHMANN_JSON_TYPE_TAG so the tag is
computed from BaseType plus the member list, which fits the available
slots. The zero-own-member derived bodies are therefore selected by tag
1 rather than 2, and the sentinel table for the derived tag is no longer
needed.
Add a regression test at the documented maximum for both the plain and
the derived macros; it fails to compile against the previous dispatch.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Name the zero-member macro bodies by intent, not argument count
The dispatch tag was the literal token 1 or N, pasted onto a macro prefix
to select the zero-member or member-carrying body. For the derived-type
macros that reads wrong: their tag is computed after dropping the leading
Type, so the zero-member body was named _1 while taking two parameters
(Type, BaseType).
Emit EMPTY and MEMBERS instead. The mechanism is unchanged -- the tag is
still a token pasted onto the prefix by NLOHMANN_JSON_CAT -- but the body
names now say what they are rather than encoding an argument count that
only lines up for half of the macros.
Collapse the four duplicated zero-member bodies while here: with no
members there is nothing to default, so each _WITH_DEFAULT_EMPTY body was
a byte-for-byte copy of its plain counterpart. They are now one-line
aliases, leaving a single definition of what an empty object serializes
to per intrusive/non-intrusive and base/derived combination.
No functional change: for both zero-member and member-carrying types the
preprocessed to_json/from_json output is token-for-token identical, and
the arity limits are unchanged (63 members, base and derived).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Document zero-member support in the macro API reference
docs/mkdocs/docs/features/arbitrary_types.md already gained a note, but
the three api/macros pages are where the parameter contract is actually
specified and they still described member as a non-empty list.
State that the list may be empty on each page, and add a note showing
what the zero-member case generates: an empty JSON object for the plain
macros, and base-type-only serialization for the derived ones. Both notes
record that the WITH_NAMES variants do not support this.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Keep user macros named EMPTY or MEMBERS out of the member-count dispatch
The dispatch produced the bare token EMPTY or MEMBERS and pasted it onto
the macro prefix afterwards. In between, the token was rescanned, so a
user macro with either name replaced it: with `#define MEMBERS x` in
scope, even NLOHMANN_DEFINE_TYPE_INTRUSIVE(A, member) -- which compiled
before -- expanded to garbage, and `#define EMPTY` broke the zero-member
form.
Paste the suffix onto the prefix directly in the GET_MACRO slot table
instead. Operands of ## are not macro-expanded, so the selected body name
is formed before any user macro can interfere. NLOHMANN_JSON_TYPE_TAG and
NLOHMANN_JSON_DERIVED_TYPE_TAG become NLOHMANN_JSON_TYPE_BODY and
NLOHMANN_JSON_DERIVED_TYPE_BODY, taking the prefix as their first
argument; NLOHMANN_JSON_CAT is no longer needed. The body macro names are
unchanged, and so is the generated code.
Add a regression test that defines EMPTY and MEMBERS around plain and
derived types, with and without members.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Test for EMPTY and MEMBERS so -Wunused-macros accepts them
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: qualify the operator>> stream positioning guarantee
operator>>'s notes state that it leaves the stream positioned right
after the parsed value, so that concatenated JSON values can be read
back to back. That does not hold when the value is a number: a number
is only terminated by the character that follows it, and the lexer's
unget() is simulated (it rewinds only the lexer's own bookkeeping),
so that character stays consumed from the stream.
Document the actual behaviour: the guarantee holds for all value types
except numbers, which must be followed by whitespace. Also qualify the
cross-reference on the JSON Lines page, which repeated the unqualified
claim.
Documentation only; the behaviour itself is tracked in #5340.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* fix: restore the character that terminates a number (#5340)
operator>> is documented to leave the stream positioned right after the
parsed value, so that concatenated JSON values can be read back to back.
That did not hold for numbers: a number is only terminated by the
character following it, and lexer::scan_number() reads that character
and calls unget() -- which is simulated and rewinds only the lexer's own
bookkeeping. input_stream_adapter consumes via sbumpc() with no matching
sungetc(), so the terminating character stayed consumed and the next
extraction started one byte too late ('1true' left the stream at 'rue').
Propagating unget() to the adapter directly does not work: next_unget
makes the following get() replay the cached character, so the terminator
would be delivered twice. Instead, restore the still-pending character
once at the end of a non-strict parse, where the input is handed back to
the caller:
- input_stream_adapter gains unget_character() (sungetc()) and advertises
it via supports_unget, detected the same way as supports_seek.
- lexer::restore_pending_unget() turns a pending simulated unget of a
real (non-EOF) character into a real one and clears next_unget so the
character is not also replayed. It is a no-op for adapters that cannot
unget, and reports failure when sungetc() fails, in which case the
input is left as it was before.
- parser calls it on the three non-strict paths, i.e. for operator>> and
sax_parse(strict = false).
Strict parse()/accept() are unaffected: they require the input to end
after the value, so the character is consumed by the end-of-input check
anyway. Parse error messages and reported positions are unchanged.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* tests: fix CI failures in the #5340 test helpers
Four CI failures, all in the new test code:
- GCC (-Werror=useless-cast): drop the `json(...)` wrapper around
`json::parse(...)`, which already returns a `json`.
- GCC (-Werror=unused-result): assign the discarded `json::parse()`
result to a dummy, the idiom used elsewhere in the test suite, and
catch `json::parse_error&` for consistency.
- clang-tidy (google-default-arguments): remove the default argument
from the `pbackfail()` override; `sungetc()` supplies the base
declaration's default.
- MSVC (bad allocation): `no_putback_streambuf::underflow()` set a
one-character get area without advancing `m_pos`, so an implementation
whose `istream::get` peeks before it bumps re-read the same character
forever. Keep no get area at all: `underflow()` peeks, `uflow()`
consumes, and `sungetc()` still always lands in `pbackfail()`, which
is what the test needs.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* fix: leave the character that terminates a number in the input
Read the character following a number without consuming it, instead of
consuming it and putting it back. input_stream_adapter now peeks with
sgetc() and only steps over the character when the next one is requested
or when the adapter is destroyed, so releasing it cannot fail - no
putback position is required from the streambuf.
Suggested by gregmarr in #5344.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: match the version history wording to the peek-based fix
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: drop the whitespace-separator caveat from the parsing pages
The caveat added in #5343 describes the behavior this branch fixes: a
number no longer consumes the character that terminates it, so
concatenated values need no separator.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* refactor: split the strict and non-strict paths in parser
Folding the release_lookahead() call into the existing strict check left
the "in strict mode" comment on an else-if branch, and made the strict
condition in sax_parse() redundant with the branch it followed.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Put the stream position fix behind JSON_PRECISE_STREAM_POSITION
Leaving the character that terminates a number in the stream is observable:
reading "1,2,3" with repeated operator>> works today only because the comma
after each number is swallowed, and std::getline after a number skips the
line break. Both break with the fix, so make it opt-in for 3.x, as suggested
by @gregmarr in the review.
- JSON_PRECISE_STREAM_POSITION (default 0) selects the peek-based
input_stream_adapter. Without it, the adapter is the consuming one from
develop and has no supports_lookahead, so lexer::release_lookahead() and
the parser's calls to it compile to nothing.
- The macro changes input_stream_adapter's layout and member functions, so
it gets the ABI tag _psp, after _bics. The ABI config tests, the natvis
generator, and nlohmann_json.natvis (regenerated) know the tag.
- The tests for the fix move to unit-precise-stream-position.cpp, which
defines the macro itself and runs in every build, and gain the two cases
above. unit-deserialization.cpp pins the default behavior instead.
- The docs describe the default behavior again and point to the new macro
page; version history says "added in 3.13.0, planned default in 4.0.0".
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Hash deeply nested values without recursing per nesting level
std::hash<basic_json> hashed an array or object by hashing each element,
which called detail::hash again once per nesting level. A value nested
deeply enough - 50,000 levels of objects on an 8 MiB stack - exhausted
the call stack and terminated the process. parse() accepts such values
without complaint, since the parser is iterative, and a parsed value is
hashed wherever it is used as a key in an unordered container.
Bound the descent the same way dump() does: detail::hash takes the
nesting level, and once hash_depth_limit() (128) levels have been entered,
hash_iteratively() hashes what is left on an explicit stack. It combines
the seeds in exactly the same order, so hash values are unchanged. A value
nested less deeply than the bound is hashed by the same code as before,
without allocating, and is as fast as before.
Tests check that every depth up to twice the bound hashes exactly like
the recursive definition of the hash, and that values nested 100,000
levels deep hash without crashing.
Fixes#5545 for std::hash.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Declare hash_frame's constructor noexcept
GCC's -Wnoexcept (an error in CI) flags the emplace_back() into the
hash stack under C++26: the constructor cannot throw, since cbegin() is
noexcept, but it did not say so. dump_frame's constructor is noexcept
for the same reason.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Share one recursion depth limit, and copy the hash frame out of the stack
dump() and hash() each defined their own limit on how many nesting levels
they recurse into, and the operations still to come would have added more,
free to diverge over time. They now all use detail::recursion_depth_limit(),
in a header of its own; serializer::dump_depth_limit() and
hash_depth_limit() are gone.
hash_iteratively() now copies the frame it works on out of the stack and
changes the frame only through stack.back(), so nothing can refer into
the stack after entering an element has grown it.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Parenthesize multiplications in the hash test for clang-tidy
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Test JSON_BRACE_INIT_COPY_SEMANTICS for real, and fix one-element tuples under it
The opt-in JSON_BRACE_INIT_COPY_SEMANTICS was never exercised by CI:
- Its only test, in unit-regression3.cpp, was guarded by
`#if defined(JSON_BRACE_INIT_COPY_SEMANTICS)` after the #include. The
header #undefs the macro unconditionally in macro_unscope.hpp, so the
guard was always false and the test compiled to nothing, whatever -D
flag was passed.
- The ci_test_brace_init_copy_semantics target that passes the flag was
not named by any workflow.
Move the test into its own translation unit that defines the macro before
including the header, as unit-diagnostics.cpp does for JSON_DIAGNOSTICS.
It now runs in every CI job and for every standard. Remove the unused
target: it ran the whole suite with the macro, and that suite deliberately
relies on default brace-init semantics in about 90 places
(e.g. `json({1})` meaning `[1]`), so it could never pass.
Running the whole suite with the macro did find one library bug:
to_json for std::tuple builds `j = { std::get<Idx>(t)... }`, so with copy
semantics a one-element tuple became its element. `json(std::tuple<int>{5})`
was `5` instead of `[5]`, and `get<std::tuple<int>>()` threw type_error.302
on the result. Under the macro, a one-element tuple now builds exactly what
the default deduction builds. Without the macro nothing changes.
The new tests also pin that the library's other conversions produce the
same values with and without the macro. The macro page now says that the
macro affects every single-element list (`json j = {1}` is `1`), and that
all translation units must agree on it, since it has no ABI tag.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Make JSON_BRACE_INIT_COPY_SEMANTICS part of the ABI tag
The macro changes the body of the initializer-list constructor and adds a
to_json_tuple_impl overload, both with the same mangled names in either
mode, so mixing translation units silently picked one definition. Encode
it in the inline namespace as `_bics`, as JSON_USE_LEGACY_DISCARDED_VALUE_COMPARISON
does with `_ldvcmp`. The macro is new in the unreleased 3.13.0, so no
existing namespace name changes.
- Move the macro's default into abi_macros.hpp so json_fwd.hpp computes
the same namespace, and keep it defined under JSON_TEST_KEEP_MACROS.
- Check the tag in the ABI config tests and in the unit test.
- List `_bics` (and the missing `_dp`) in the namespace docs and in the
natvis generator; regenerate nlohmann_json.natvis.
- Replace the "define it consistently" warning with an ABI note.
Suggested by @gregmarr in the review of #5544.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix the cppcheck, clang-tidy and legacy-comparison CI failures
- to_json_tuple_impl() moved the element in both branches of a ternary;
only one runs, but cppcheck reported accessMoved. Use if/else.
- The ABI tag test looked for "json_abi_bics", which misses when another
tag comes first, as in json_abi_ldvcmp_bics; look for "_bics".
- readability-qualified-auto in the items() test.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
write_ubjson() built a std::vector of the eight markers BJData forbids as
the type of an optimized container - one heap allocation plus a linear
search for every array and object it wrote with use_type, even for plain
UBJSON output, where the list isn't consulted. The list was also spelled
out twice. A constexpr helper, is_bjdata_excluded_type_marker(), replaces
both.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Niels Lohmann
Kartikey Negi
Dadi Reddy Sai Praneeth Reddy