The benchmarks fetched Google Benchmark's main branch, so two builds on
different days could measure with different library code, and CMake 3.30
and later warn that the single-argument FetchContent_Populate() is
deprecated. Fetch the 1.9.5 release archive, verified by its SHA-256,
with FetchContent_MakeAvailable() instead. That needs CMake 3.14; Google
Benchmark itself already needed 3.13.
Its -Werror is switched off, so a newer compiler's new warnings cannot
break the pinned release, and its install rules are no longer added.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The target has configured ${PROJECT_SOURCE_DIR}/benchmarks since it was
added in #2561, but the benchmarks live in tests/benchmarks.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The benchmark project hasn't configured since #4793: download_test_data.cmake
compiles cmake/detect_libcpp_version.cpp relative to CMAKE_SOURCE_DIR, which
is tests/benchmarks when that is the top-level project, so try_run fails and
so does `make run_benchmarks`. The path is now relative to the module itself,
which is the same file for the main build.
The Dump benchmark discarded dump()'s result, which is [[nodiscard]] by now;
it warned, and left the optimizer free to shorten the loop. The result is
now kept with benchmark::DoNotOptimize.
A new cache variable, JSON_BENCHMARK_INCLUDE_DIR, names the directory holding
the nlohmann/json.hpp to benchmark (single_include by default, as before),
so the same benchmarks can be built against two versions and compared.
tests/benchmarks/README.md documents what is measured, how to build and run
the benchmarks, how to read the output, and how to compare two versions with
Google Benchmark's compare.py; it recommends doing so by hand before a
release rather than in CI.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: document that a NUL byte in the input is treated as end of input
A NUL byte anywhere in the input - trailing, or embedded ahead of more
otherwise well-formed JSON - is currently treated the same as genuine
end of input, so parsing silently stops there instead of raising the
parse_error.101 any other unexpected byte triggers. This mirrors the
NUL-terminated-C-string convention already used when no explicit input
length is given (json::parse(const char*) already stops at strlen()),
just applied uniformly rather than only when a length is genuinely
unavailable.
This behavior predates this change and is not being altered here -
changing it would be an observable, backwards-incompatible behavior
change for any caller that (knowingly or not) depends on it, which is
not something to do silently in a patch. Documenting the current,
verified behavior as a new FAQ entry instead, so it's an intentional
and discoverable part of the contract rather than a surprise.
Fixes#5530.
Signed-off-by: Niels Lohmann <niels.lohmann@gmail.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N4RQ1Ahan5YAGbnAQGjZTY
* Add JSON_STRICT_NUL_HANDLING opt-in macro for issue #5530
A NUL byte anywhere in the input is currently treated the same as real
end of input, rather than raising parse_error.101 like any other
unexpected byte (documented in the previous commit's FAQ entry). A full
unconditional fix was tried in PR #5532 but rejected as too risky to
ship by default: any caller could depend on the current behavior, even
unknowingly (e.g. a zero-padded buffer). On PR #5534, gregmarr proposed
a compile-time opt-in flag instead, and the maintainer agreed, wanting
it available now and defaulting to the corrected behavior in 4.0.0.
This mirrors the existing JSON_BRACE_INIT_COPY_SEMANTICS precedent as
closely as sensible:
- JSON_STRICT_NUL_HANDLING defaults to 0 (off); the three lexer sites
that treat '\0' as EOF/comment-terminator are gated with
`#if !JSON_STRICT_NUL_HANDLING` so the default-off behavior is
byte-for-byte identical to today's.
- input_adapters.hpp's `T (&array)[N]` overload additionally trims a
single trailing '\0' from a `char` array (e.g. a string literal like
`json::parse("123")`) when the macro is on, so that case keeps
working; every other element type (unsigned char, std::uint8_t, ...)
always keeps its full extent. This intentionally does *not* reuse the
existing strlen()-based pointer overload via SFINAE-excluding `char`
from the array overload, as originally sketched for this change: that
approach is ambiguous against the newer generic container overload
added since PR #5532, and even where it compiles, strlen()-scanning a
`char` array that is not NUL-terminated within its bounds reads past
the end of the array (confirmed with AddressSanitizer). Trimming only
a single trailing byte, without scanning, avoids both problems.
- Documented via docs/mkdocs/docs/api/macros/json_strict_nul_handling.md,
linked from the macros index/nav/features page, the FAQ entry, and
the parse/accept/operator>> reference pages.
- Tested in unit-class_parser.cpp and unit-deserialization.cpp, default
state unguarded and opt-in state guarded. Since the library itself
#undefs the macro at the end of json.hpp (as JSON_BRACE_INIT_COPY_SEMANTICS
already does), a plain `#if defined(JSON_STRICT_NUL_HANDLING)` guard
after the include never actually triggers; the tests instead capture
the command-line value into a test-local macro before including the
header. A few pre-existing fixtures elsewhere (std::array<uint8_t, N>
sized one larger than their literal, relying on value-initialization
to silently add a trailing zero byte) needed the same one-byte
adjustment to keep passing under the opt-in behavior.
Unlike the precedent, this adds a proper `JSON_StrictNulHandling` CMake
option (rather than a raw -DCMAKE_CXX_FLAGS injection) and wires its
ci_test_strict_nul_handling target into the ci_cmake_options job matrix
in .github/workflows/ubuntu.yml, so the opt-in build is actually
exercised in CI -- closing the one gap in the precedent's own CI setup
(ci_test_brace_init_copy_semantics is defined but never referenced by
any workflow, so it has never actually run).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Clarify where JSON_STRICT_NUL_HANDLING does not reject NUL bytes
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <niels.lohmann@gmail.com>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Add 60 contributors whose work was not yet credited and update seven
links that pointed to renamed or reassigned GitHub accounts.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Devirtualize binary_writer via a value-type output sink
to_cbor/to_msgpack/to_ubjson/to_bjdata/to_bson wrote every byte through
output_adapter_t, a shared_ptr<output_adapter_protocol> whose
write_character/write_characters are virtual. Unlike the lexer (templated
on a concrete InputAdapterType), the binary writer never got that
treatment, so binary output paid a vtable lookup per byte and a
make_shared per call.
Template binary_writer on an OutputSinkType and give it two concrete,
non-virtual sinks:
- output_vector_sink: appends straight into a std::vector (push_back /
insert), used by the vector-returning to_* convenience functions. No
vtable, no shared_ptr; the writes inline.
- output_adapter_sink: forwards to a type-erased output_adapter_t, so the
existing to_*(j, output_adapter) overloads (streams, strings, custom
adapters) keep working exactly as before -- one virtual call each,
unchanged.
binary_writer keeps a convenience constructor taking output_adapter_t
(building the default output_adapter_sink), so the adapter overloads are
untouched; only the convenience functions switch to the vector sink. The
friend declaration and the basic_json binary_writer alias gain the new
(defaulted) template parameter.
Output is byte-for-byte identical: verified across ~3000 randomized
values plus curated edge cases (all scalar widths, strings with invalid
UTF-8, binary, nested arrays/objects) for CBOR, MessagePack, UBJSON (both
size/type settings), BJData, and BSON, plus the output_adapter path, in
C++11/17/20. Warning-clean under clang -Weverything and the gcc pedantic
set; clang-tidy clean on the changed headers; make check-amalgamation
clean.
Throughput (g++ -O3, vs develop): scalar-dense binary output such as
integer arrays ~1.4x; many small to_cbor calls ~1.04x (DOM traversal
bound); string/blob-heavy output unchanged (already bulk-bound). No
workload regressed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix CI failures from binary_writer output-sink change
Four CI jobs failed on the initial commit; all are addressed here without
changing any output (binary encodings remain byte-for-byte identical to
develop across the differential corpus):
1. ci_test_gcc / cuda (-Werror=duplicated-branches): for number_float_t ==
float, static_cast<float>(n) is the identity, so write_compact_float's
two branches are intentionally identical. Once the concrete vector sink
is inlined, GCC constant-folds and diagnoses this (the type-erased path
hid it behind a non-inlined virtual call). Silence -Wduplicated-branches
for GCC (clang has no such warning) alongside the existing -Wfloat-equal
pragma.
2. ci_static_analysis_clang (UBSan nonnull-attribute): binary_writer passes
a null pointer with length 0 for empty strings/binary. output_vector_sink
/ output_adapter_sink declared write_characters JSON_HEDLEY_NON_NULL, so
the sanitizer flagged the (harmless) zero-length call once the sink was
called directly rather than through the attribute-free virtual base. Drop
the attribute from both sinks, matching the pre-existing behavior.
3. ci_cpplint (build/include_what_you_use): output_adapter_sink uses
std::move; add #include <utility>.
4. ci_cuda_example (nvcc 11.8): NVCC's front end rejects the default
template argument on the binary_writer alias template. Revert the alias
to its original single-parameter form (relying on binary_writer's own
defaulted OutputSinkType) and spell out the full type in the vector-sink
convenience functions.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Encode big-endian numbers with a byte swap instead of std::reverse
write_number() reordered multi-byte numbers for the big-endian formats
(CBOR/MessagePack/UBJSON) with std::reverse over the byte array. GCC
lowered only some sizes to a bswap; clang kept a scalar byte shuffle
(0 bswap instructions in the CBOR number path). Replace the reverse with
size-dispatched __builtin_bswap16/32/64 helpers (portable shift fallback
for other compilers; std::reverse retained for exotic sizes such as a
long double number_float_t).
Codegen: the CBOR number path now emits bswap on both compilers
(gcc 2 -> 16, clang 0 -> 4). Output is byte-for-byte identical to the
previous implementation across the binary differential corpus.
Throughput (isolated vs the std::reverse version, best of 9):
CBOR int64 array gcc +7% clang +10%
CBOR uint16 array gcc +27% clang flat
Modest but consistent on number-dense encodings; negligible on
string/blob-heavy output, as expected.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Reserve output capacity up front for binary serialization
The vector-returning to_cbor/to_msgpack/to_ubjson/to_bjdata/to_bson grew
the output buffer purely by geometric reallocation. Reserving an estimate
up front avoids the early reallocations, which is the dominant per-byte
cost for array/object-heavy output.
The estimate (binary_reserve_hint) is deliberately conservative and safe
against untrusted input: it consults only the top-level element count
(O(1), no walk of the DOM), guards the multiplication against overflow,
and clamps the result to a fixed 1 MiB ceiling, so a large or hostile DOM
can never force an oversized allocation here. The buffer still grows
geometrically past the hint, so an underestimate only costs a few later
reallocations; scalars/strings/binary are written in one shot and get no
hint. Reserving capacity does not change the bytes produced.
Throughput (g++/clang -O3, vs the previous commit):
cbor int array +10% / +13%
cbor object array +20% / +38%
Output is byte-for-byte identical to develop across the binary
differential corpus.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address review findings on the binary writer output sinks
- binary_reserve_hint(): the 4-bytes-per-element estimate over-reserved by up
to 4x for arrays of small scalars (CBOR encodes 0..23 in one byte), and the
returned vector kept that capacity. Make the hint a strict lower bound on the
encoded size instead, which also removes the 1 MiB clamp whose branch no test
could reach (the largest container in the suite has 65793 elements).
- Guard the -Wduplicated-branches pragma with __GNUC__ >= 7. The warning does
not exist before GCC 7, so naming it made GCC 4.8/4.9/5/6 - which the CI
matrix still builds - warn under -Wpragmas on every including translation
unit, breaking downstream -Werror builds.
- Constrain the adapter constructor of binary_writer with the enable_if its
documentation already claimed, so a writer over some other sink type is no
longer advertised as constructible from an output adapter.
- Let output_vector_adapter wrap output_vector_sink rather than duplicating the
append logic, so the type-erased and templated paths share one implementation.
- Collapse the three copies of the memcpy/byte_swap/memcpy dance into a single
byte_swap_buffer() helper, and add the MSVC _byteswap_* intrinsics so MSVC no
longer falls back to the scalar shuffle this change exists to eliminate.
- Add a vector_writer() helper for the five vector-returning to_* overloads
instead of spelling out the writer type at each call site, and drop a dead
default member initializer on output_adapter_sink.
- New tests: the vector sink and the adapter sink must produce identical bytes
for every format (the two to_* overloads no longer delegate to each other and
could otherwise drift), and binary_reserve_hint() must never exceed the size
actually written.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Route the -Wduplicated-branches pragma through Hedley
Match #5485, which moved the binary writer's hand-rolled diagnostic
pragmas onto JSON_HEDLEY_PRAGMA (merged into develop while this branch
was open). The devirtualization's -Wduplicated-branches suppression in
write_compact_float was the one raw '#pragma GCC diagnostic' left; it
now uses JSON_HEDLEY_PRAGMA like the adjacent -Wfloat-equal line, still
guarded to GCC >= 7 and non-clang (the warning exists only there).
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* Reject MessagePack/BSON binary subtypes that don't fit their wire format
Both formats store byte_container_with_subtype's subtype (a uint64_t)
in a single byte. The writers cast to std::int8_t/std::uint8_t without
a range check, so subtypes above 255 were silently truncated modulo
256 instead of raising an error. Throw out_of_range.413 instead when
the subtype exceeds the representable range of 0-255.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Move the new binary-subtype regression test out of unit-regression2.cpp
unit-regression2.cpp is already at the edge of what the MinGW linker
can relocate; adding this test's ~26 lines tips test-regression2_cpp20
(clang, Windows) over into "relocation truncated to fit:
IMAGE_REL_AMD64_REL32 against `.rdata'" (see 8ce64b9c1 / b82717c8a for
the same failure mode). Split the test along format lines instead:
MessagePack assertions move to unit-msgpack.cpp, BSON assertions to
unit-bson.cpp. The CBOR round-trip guard is dropped as redundant --
unit-cbor.cpp's "Tagged values" section already round-trips subtypes
up to 8589934590, far past the 70000 checked here.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The array-range insert() overload checked that pos fits the current
value and that first/last share the same owning value, but never
verified that value is itself an array. Passing iterators from an
object, a primitive, or null handed value-initialized (singular)
std::vector iterators straight to array_t::insert(), which is
undefined behavior. Add the missing is_array() check, mirroring the
equivalent check already present in the object-range insert()
overload.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Honor allow_exceptions=false for excessive array/object size (out_of_range.408)
The SAX DOM parsers' start_object()/start_array() threw out_of_range.408
directly via JSON_THROW when a binary format (CBOR/UBJSON/BJData) declared
a container size exceeding max_size(), bypassing the allow_exceptions flag
that every other malformed-input error path in these classes honors via
parse_error(). This meant that json::from_cbor(data, true, false) etc.
could still throw (or abort under JSON_NOEXCEPTION) instead of returning a
discarded value, contrary to the allow_exceptions=false contract.
Route all four call sites (two in json_sax_dom_parser, two in
json_sax_dom_callback_parser) through parse_error() instead, matching the
existing error-handling pattern used elsewhere in this file. Behavior is
unchanged when allow_exceptions is true (the default); the exception
message and type are identical.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Drop a non-portable exact exception message check in the 408 test
The allow_exceptions=false regression test checked the exact message
text produced when allow_exceptions=true (the default). On platforms
where std::size_t is 32-bit (e.g. mingw x86, MSVC Win32 builds), a
declared CBOR length of 2^63 is intercepted earlier, by
get_cbor_container_size()'s own (pre-existing, already correct)
length-narrowing check, with different wording than this fix's
start_array()/start_object() size check -- same error code, same
"still throws when allow_exceptions=true" guarantee, different text.
CHECK_THROWS_AS already verifies the behavior this test cares about
(still throws json::out_of_range, unchanged); drop the exact-message
assertion since it isn't portable across size_t widths and doesn't
add coverage of this fix specifically.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix -Werror=unused-result on json::from_cbor() in the 408 regression test
from_cbor() is [[nodiscard]]; CHECK_THROWS_AS() otherwise discards its
result, which GCC flags under -Werror. Assign to a throwaway json, as
the rest of the suite already does for from_cbor()/from_msgpack().
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Restore a duplicate key's prior value when the callback rejects its new value
json_sax_dom_callback_parser::key() unconditionally overwrote the object
slot for a key with a `discarded` placeholder as soon as the key was
accepted by the parser callback. For a duplicate key (legal JSON), this
destroyed the pre-existing value from an earlier occurrence of the same
key before the new value was even parsed. If the new value was then
rejected by the callback, remove_discarded_value() erased the member
entirely instead of leaving the original value in place, contradicting
the documented behavior that a discarded value behaves as if it was
never read.
Add a small stash of (slot pointer, previous value) pairs so that when
key() overwrites an existing member with the discarded placeholder, the
previous value can be restored later if the corresponding value (scalar,
object, or array) is rejected, instead of being erased. The stash entry
is dropped without restoring once the new value is definitively
accepted (in handle_value() for scalars, end_object()/end_array() for
containers), so a duplicate key whose new value is accepted still keeps
the last value as before. Non-duplicate keys are unaffected: rejecting
their value still removes the member entirely, since there is nothing
to restore.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Mark parser-callback test lambdas noexcept to fix GCC -Wnoexcept -Werror
GCC's libstdc++ std::function move assignment evaluates a noexcept
check that invokes a wrapped callable in an unevaluated context; a
non-noexcept parser_callback_t lambda then trips -Wnoexcept ("noexcept-
expression evaluates to 'false'"), which CI's ci_test_gcc job builds
with -Werror. The pre-existing parser_callback_t test lambdas in this
file already work around this by declaring themselves noexcept; apply
the same fix to the three added lambdas that didn't.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Make diff() account for member order in ordered_json objects
diff() compared source/target objects purely by key set, ignoring
relative member order. For ordered_json (insertion-ordered, vector-
backed object_t), two objects that differ only in member order are
unequal via operator==, but diff() never emitted any patch operation
to fix the order, so source.patch(diff(source, target)) == target
could fail to hold.
Fix by detecting when common keys appear in a different relative
order in source vs. target (or when a new key would need to land
somewhere other than the end), and in that case removing and
re-adding the affected keys in target's order, which relies on
patch()'s "add" op appending new keys at the end of an ordered_map.
For plain json (std::map-backed, always key-sorted iteration) this
is a no-op and the original minimal per-key diff path is unchanged.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Avoid redundant lookups in diff()'s object-order tracking
The previous fix for ordered_json member order re-derived common-key
order and suffix information with extra target.find()/source.find()
calls layered on top of the pre-existing removed/added-key passes,
instead of reusing those same passes. This roughly tripled the number
of map lookups per diff() call for every object, including plain
`json`, where the reordering path is never taken.
Piggyback the order tracking (and the "add" op construction for new
keys) onto the two passes the algorithm already needs to detect
removed/added keys, and walk the fast path's recursion in lockstep
with the precomputed common-key list instead of re-querying `target`.
This restores diff() to its pre-existing lookup count; benchmarked at
n=1000 keys, ordered_json::diff() was roughly 2x slower than baseline
before this change and is back within noise of baseline after it.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Preserve diff()'s original op ordering and fix a slow-path deletion gap
Splitting removed-key detection and common-key recursion into separate
passes (for the earlier lookup-count fix) changed the emitted patch's
op order: all "remove" ops now came before all recursive per-key diffs,
instead of interleaved in source's iteration order as the original
implementation did. This broke docs/mkdocs/docs/examples/diff.output's
exact-match CI check (ci_test_examples) even though the patch was still
semantically correct.
Defer "remove" emission into the same walk that does the recursive
diffs, so common keys and deleted keys are interleaved in source order
again, matching historical output.
While restructuring that walk, the reordering ("slow path") branch was
only emitting "remove" for keys common to both objects, never for keys
present in source but genuinely absent from target -- a key deleted
alongside an actual reorder would silently survive the patch. Fixed by
removing every source key in the slow path (both deleted and common
keys need removing there; common keys are then re-added in target's
order). Verified with a targeted reorder+deletion case and a fresh
20,000-case round-trip fuzz run (0 failures).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The characterization test added in #5482 asserted that basic_json::swap()
does NOT exchange start_position/end_position, based on a misreading of
the code cited for #5420. In fact swap() (json.hpp, around line 3637)
does swap start_position/end_position along with the value, consistent
with copy-assignment. The test's assumption was backwards, so it failed
on every CI job across every branch/PR since the commit landed. Correct
the assertions to match the actual (and correct) behavior: positions are
exchanged together with values.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Fall back to plain-object encoding when _ArrayType_ is not a string
write_bjdata_ndarray() looked up _ArrayType_ by calling get<string_t>()
directly, which throws type_error.302 when the annotation is not a
string (e.g. a number, null, boolean, array, or object). Per the
documented BJData ndarray contract, an object only qualifies for the
compact ndarray encoding if _ArrayType_ names a known type; anything
else must fall back to plain-object encoding, the same way an unknown
type-name string already does.
Add an is_string() check before the get<string_t>() call so a
non-string _ArrayType_ takes the existing "unrecognized type name"
fallback path instead of throwing.
Fixes#5398.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Relax the BJData fuzzer's round-trip check from byte-exact to value-exact
Fixing #5398 lets to_bjdata() proceed past the object it used to reject,
which exposed a pre-existing, unrelated round-trip quirk to the fuzzer:
a binary_t value serialized through the non-optimized ("$U#"-less)
array encoding is parsed back as a plain array of numbers, since
from_bjdata() has no way to tell "array of uint8 numbers" apart from
"array of bytes" without that optimized header. Re-serializing that
plain array then goes through the generic smallest-type writer, which
- unrelated to this PR, and long predating it - prefers the 'i' (int8)
marker over 'U' (uint8) for values that fit both, so the re-encoded
bytes can differ from the original even though both decode to the same
value.
This is not introduced by the #5398 fix; the same divergence reproduces
from a bare json::binary_t value with no _ArrayType_ annotation
involved at all, on the commit immediately preceding it. A general fix
would mean changing the shared UBJSON/BJData smallest-type selection
that hundreds of existing tests pin to 'i' for small positive
integers, which is out of scope and too risky for this PR.
Update fuzzer-parse_bjdata.cpp's round-trip assertions to check that
re-serializing is value-stable (from_bjdata(to_bjdata(j)) == j) rather
than byte-exact, matching the guarantee BJData actually provides, and
add a regression test in unit-bjdata.cpp using the exact OSS-Fuzz input
that documents the behavior.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Compare dump()s instead of json values in the BJData fuzzer's round-trip check
The value-stability assertion added to fix the earlier OSS-Fuzz crash
(json::from_bjdata(to_bjdata(j2)) == j2) itself broke on a NaN payload:
IEEE 754 NaN is never equal to itself, so operator== reports two
structurally-identical trees containing a non-finite double as
different -- not a round-trip bug, just NaN's ordinary
non-reflexivity. dump() serializes any non-finite double the same
deterministic way (as JSON null, since JSON cannot represent NaN or
Infinity), so comparing dumps is stable under exactly the values that
break operator==.
Verified against both the original OSS-Fuzz crash input and the new
one (0x68 0x68 0x7c, which decodes to a NaN), plus a local 2.5M-case
random-input sweep with no failures.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix to_bjdata() emitting the Draft-3-only 'B' marker in default Draft-2 mode
_ArrayType_ = "byte" mapped unconditionally to the BJData type marker
'B', regardless of the requested bjdata_version. 'B' is defined only by
BJData Draft 3; with the default version (draft2), this produced a
stream that is invalid for Draft 2 and, unlike every other
_ArrayType_, round-tripped back as a binary value instead of the
original annotated object.
Only accept "byte" / emit 'B' when bjdata_version selects Draft 3.
Under Draft 2, fall back to the same plain-object encoding used
elsewhere in this function for other invalid-annotation cases, so the
value round-trips correctly.
Fixes#5404.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Future-proof the Draft-3-only 'B' marker gate
@gregmarr pointed out that dtype == 'B' && bjdata_version != draft3
only future-proofs by accident, since bjdata_version_t currently has
exactly two values. Compare with < instead, so a later draft that
keeps the 'B' marker valid does not need this gate revisited.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix dead ill-formed-fourth-byte UTF-8 test sections (byte3/byte4 typo)
The "ill-formed: wrong fourth byte" SECTIONs in unit-unicode3.cpp,
unit-unicode4.cpp, and unit-unicode5.cpp guarded their loop with a check
on byte3 instead of byte4. Since the enclosing loop already restricts
byte3 to its valid range, the guard was always true and the section's
"continue" fired unconditionally, so check_utf8string()/check_utf8dump()
were never actually invoked for a malformed fourth byte.
Fixing the guard naively (byte3 -> byte4) would also have swept the full
byte2 x byte3 combinatorics for every byte4 value, adding millions of
redundant iterations: the lexer validates continuation bytes strictly in
sequence with early exit (see next_byte_in_range() in lexer.hpp), so once
byte2/byte3 are within their valid range, the byte4 outcome does not
depend on which valid byte2/byte3 values were chosen. Instead, byte2 and
byte3 are now held to a small hedge of representative valid prefixes
(range corners plus a midpoint) while byte4 is still swept exhaustively
over its full 0x00-0xFF range, since that is the actual property under
test. Also fixed the garbled "skip fourth second byte" comment in
unit-unicode3.cpp.
Verified offline: before the fix, the "wrong fourth byte" subcase
executes 0 assertions in all three files (proving it was dead code);
after the fix, it executes 11520 (unicode3), 34560 (unicode4), and 11520
(unicode5) assertions, and a deliberately reintroduced bug in the
lexer's byte4 range check causes it to fail (proving it is now
meaningful). Total per-file assertion counts grow by the same small
amounts, not by millions, and all other sections in these files still
pass unchanged.
Fixes#5416
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Keep full byte2 x byte3 combinatorics in the wrong-fourth-byte sections
The maintainer wants exhaustive coverage of every byte combination here
rather than the representative-prefix reduction, matching the style of
the sibling "wrong second/third byte" sections in the same files.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Add missing diagnostic-positions test coverage (lifetime, input adapters, SAX)
Building on the merged unit-class_parser.cpp from #5417, add
characterization tests (regression protection for existing behavior, not a
behavior change) for JSON_DIAGNOSTIC_POSITIONS:
- value lifetime: copy ctor copies positions recursively, move ctor resets
the moved-from value to npos, and mutating a parsed document (operator[],
push_back, erase) leaves the parent's stale span and siblings' positions
untouched while new values get npos.
- input adapters: wide-string input positions count transcoded UTF-8 bytes
(not wide characters), BOM-prefixed input's start_pos() reflects the
skipped 3-byte BOM, istringstream/ifstream/iterator-pair inputs report
consistent (non-npos) positions, and binary formats (CBOR, MessagePack,
UBJSON, BSON) always report npos.
- a user-constructed json_sax_dom_parser with no lexer (as used when driving
json::sax_parse() directly) reports npos for every value, since it has no
m_lexer_ref to source positions from.
While characterizing swap(), found that basic_json::swap() (and the friend
swap() that forwards to it) does not swap start_position/end_position,
unlike copy-assignment's operator=(basic_json), which does as part of its
copy-and-swap implementation. This looks like a real inconsistency/bug, but
per the scope of this test-only change it is only pinned (not fixed) here;
see the comment at the "swap() does NOT exchange positions" section.
Fixes#5420
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix MSVC source-encoding portability in the wide-string position test
Use é escapes instead of a literal UTF-8-encoded 'é' inside the L""
literal, so the wide string's content does not depend on the compiler's
assumed source character set (MSVC without /utf-8 decodes raw non-ASCII
source bytes using the system code page rather than as UTF-8, which was
producing a wstring of unexpected length/content and failing the
ws.size()/end_pos() assertions on Windows CI).
Also reworded a comment that unintentionally embedded the literal
substring "TODO check", which clang-tidy's google-readability-todo check
flags regardless of quoting context.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* test: cover JSON_NO_IO, JSON_THROW/TRY/CATCH_USER, JSON_SKIP_LIBRARY_VERSION_CHECK, and JSON_DisableEnumSerialization in CI (#5423)
These four supported configuration macros were never actually compiled
anywhere in the test matrix:
- JSON_NO_IO and the JSON_THROW_USER/JSON_TRY_USER/JSON_CATCH_USER trio
are exercised together in a new tests/src/unit-no_io_and_user_exceptions.cpp,
which is automatically picked up by the existing unit-*.cpp test glob and
thus built across the whole standard test matrix.
- JSON_SKIP_LIBRARY_VERSION_CHECK is exercised by a new, dedicated
tests/src/skip_library_version_check.cpp, compiled directly by the new
ci_test_skiplibraryversioncheck target in cmake/ci.cmake: the scenario it
simulates (mixing two differently-versioned inclusions of the library)
unavoidably triggers the compiler's own "macro redefined" warning, which
would fail under the library's own -Weverything/-Werror unit test matrix
for a reason unrelated to the macro under test.
- JSON_DisableEnumSerialization already had #if-guarded tests in several
unit-*.cpp files (from #4384), but no CMake target ever actually set the
JSON_DisableEnumSerialization CMake option, so that guarded code was never
compiled. Add ci_test_disableenumserialization, mirroring the existing
ci_test_noimplicitconversions/ci_test_noglobaludls targets. Building the
full test suite with this option on surfaced one real, narrow gap: get<T>()
on std::vector<std::byte> (used by unit-regression2.cpp's custom BinaryType
tests) relies on std::byte being handled via enum serialization, so add the
same #if-guard convention to the two affected SECTIONs there.
Both new CI targets are added to the ci_cmake_options matrix in
.github/workflows/ubuntu.yml, alongside the existing ci_test_* targets.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* test: cover multi-digit widths and bare alignment in std::formatter<json> (#5423)
Every existing std::formatter spec with a width used a single digit (e.g.
"{:2}"), so the width-parsing loop's accumulation of a second/third digit was
never exercised; add multi-digit width cases. Likewise, every existing spec
with an alignment character also had an explicit fill character, so the
bare-alignment branch (e.g. "{:<}", with no fill) was never exercised; add
cases asserting it keeps the default space indent character.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* test: add coverage for patch_inplace() (#5423)
patch_inplace() had no unit test at all. Add a happy-path case mirroring an
existing patch() example, and -- more importantly -- pin its distinguishing
contract versus patch(): when a multi-operation JSON Patch fails partway
through, patch_inplace() (which mutates the document directly, operation by
operation) leaves whatever operations already succeeded applied, whereas
patch() (which applies the patch to an internal copy that is discarded on
exception) leaves the original completely untouched either way. Verified
empirically against the current implementation before writing the assertions.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* test: fix duplicate TEST_CASE name in unit-no-mem-leak-on-adl-serialize.cpp (#5423)
Two distinct TEST_CASEs were both named "check_for_mem_leak_on_adl_to_json-2".
doctest allows duplicate names, so both still ran, but it makes
--test-case=<name> filtering and reporting ambiguous. Rename the second one
to "-3", continuing the existing "-1"/"-2" sequence.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* test: add direct coverage for the std::u8string to_json overload (#5423)
The ADL to_json overload for std::basic_string<char8_t, ...> was only ever
reached indirectly, via std::filesystem::path::u8string(). Add a test that
constructs a json value directly from a std::u8string, gated the same way as
the overload itself (include/nlohmann/detail/conversions/to_json.hpp): behind
both the std::filesystem::path feature guard and __cpp_lib_char8_t, since the
overload only exists when both are satisfied.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* test: verify move semantics of byte_container_with_subtype's rvalue constructors (#5423)
The two rvalue-reference constructors were never distinguished from their
const-lvalue-reference twins by any test. Add a "move semantics" section that
constructs from an rvalue std::vector, checks the resulting container keeps
the exact same buffer address as the source (a stronger check than just
observing the source ended up empty, since a copy-then-clear could do that
too), and confirms the source vector was left empty.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Guard patch_inplace() partial-application test against JSON_NOEXCEPTION
The "distinguishing contract vs patch(): partial application on
failure" test relies on doc.patch_inplace(patch) actually throwing so
the partially-applied state can be observed right after the throw
point. Under ci_test_noexceptions, JSON_THROW() calls std::abort()
instead of throwing, and doctest's --no-throw test filter (which that
CI job passes) makes CHECK_THROWS_AS() a no-op that never even
evaluates its expression -- so patch_inplace() is never called and the
follow-up assertions fail against the untouched original document.
Guard the whole SECTION with #if !defined(JSON_NOEXCEPTION), following
the same convention already used elsewhere in the test suite (e.g.
unit-class_parser.cpp) for exception-dependent tests.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix MSVC C2220 in the std::u8string conversion test
MSVC's C5321 ("nonstandard extension used: encoding '\xNN' as a
multi-byte utf-8 character") is promoted to a hard error by our MSVC CI
configs. It fires because the test composed a non-ASCII UTF-8 sequence
inside a u8"" literal using raw \x byte escapes; MSVC treats that as
nonstandard and suggests using \u universal-character-names instead,
which every compiler agrees on and which compiles down to the exact
same encoded bytes.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Guard the JSON_THROW_USER test against JSON_NOEXCEPTION and GCC's -Wunused-result
Two independent CI configurations failed to build/run this new test:
- ci_test_noexceptions runs the whole suite with -DJSON_NOEXCEPTION and
doctest's "--no-throw" filter, which compiles CHECK_THROWS_AS() down
to a no-op that never even invokes the guarded expression. Since this
test's whole point is to observe json_throw_user_call_count after
json::parse()/at() actually throw, it can't be meaningfully run under
that filter (our JSON_THROW_USER override still throws real
exceptions regardless of JSON_NOEXCEPTION, but the assertion never
gets a chance to run). Guard the TEST_CASE with
#if !defined(JSON_NOEXCEPTION), mirroring the existing precedent in
unit-json_patch.cpp.
- ci_test_gcc and ci_test_standards_gcc(11) failed with
-Werror=unused-result on the discarded json::parse() return value.
json::parse() is marked warn_unused_result, and unlike a real
[[nodiscard]] attribute, GCC does not consider that satisfied by
doctest's (void)-cast around the expression in C++11 mode. Assign the
result to a discarded local instead, matching the established
`json _ = json::parse(...)` idiom already used throughout
unit-class_parser.cpp.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Suppress a clang-tidy false positive on an intentional defensive copy
performance-unnecessary-copy-initialization suggests copy_for_patch
could be a reference since it's never modified -- but the copy is the
point: it guards against a hypothetical regression where patch()
mutates its receiver, which a reference could never catch (the
follow-up assertion would just compare `original` to itself).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix clang-tidy findings in the JSON_NO_IO/JSON_THROW_USER test
- bugprone-macro-parentheses: wrap the JSON_THROW_USER macro argument in
parentheses at the throw site.
- modernize-raw-string-literal: switch two escaped JSON string literals to
raw string literals.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Add test coverage for ordered_json/alt_json across binary formats and patch/diff/flatten APIs
Closes a test-coverage gap from #5421: ordered_json (and the alt_string-based
basic_json specialization from unit-alt-string.cpp) were never round-tripped
through the binary formats (CBOR/MessagePack/UBJSON/BSON/BJData), nor through
flatten()/unflatten(), diff()/patch()/patch_inplace(), or merge_patch(). Also
adds a std::formatter<ordered_json> spot-check, mirroring the precedent set
by the format_as() ADL-deduction test.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Pass alt_string's std::string constructor argument by value (clang-tidy modernize-pass-by-value)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
#5285's indentation regression test didn't exercise json::binary,
which serializes as an object but always writes its byte array
compactly (dump_byte()). #5186 had covered this case before it was
closed as superseded; port just that coverage here.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: document the implicit requirements on basic_json's template parameters
The requirements that basic_json places on its eleven template parameters
were only implied by how the library uses the resulting object_t, array_t,
string_t, etc. Consumers had to discover them by trial and error.
Add "Template Parameter Requirements" collecting them, split into what is
always required and what is only required when a particular part of the API
is instantiated. Notable findings that were previously undocumented:
- ObjectType must provide a key_compare member type (actual_object_comparator
names object_t::key_compare in both arms of a std::conditional), and its
third template parameter is used as a comparator, so std::unordered_map
cannot be used without a wrapper.
- ArrayType must provide capacity() -- push_back(), emplace_back(),
operator+=(), and operator[](size_type) call it unconditionally -- and
needs random-access iterators, so std::deque and std::list do not work.
- StringType needs contiguous, null-terminated data(), a one-byte value_type,
and either assignability from std::to_string or an ADL int_to_string().
- NumberFloatType must be float, double, or long double for parsing and
serialization; the integer types must satisfy std::is_integral.
- AllocatorType must be stateless, support incomplete types, and use plain
pointers.
- BooleanType and the number types are union members and must be trivial.
Link the new page from the basic_json overview, the types feature page, and
the individual type alias pages, and correct the container examples given for
ObjectType (std::unordered_map) and ArrayType (std::list), which do not work.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018hxZxz8svM54c6ATEvXp5E
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix object_comparator_t for object types without key_compare
detail::actual_object_comparator selected between object_t::key_compare and
default_object_comparator_t with std::conditional. Both type arguments of
std::conditional are named eagerly, so object_t::key_compare had to exist
regardless of the condition, and the has_key_compare guard added in 3.11.0
never took effect: any ObjectType without a key_compare member type failed to
compile while instantiating basic_json itself.
Use detected_or_t instead, which resolves through a SFINAE partial
specialization and only names object_t::key_compare when it exists. The
selected type is unchanged for every object type that compiled before, so
object_comparator_t -- a public member type -- keeps its meaning and ABI.
has_key_compare had no other users and is removed.
Add a regression test using an adapter around std::unordered_map, which has no
key_compare; it fails to compile without this change.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018hxZxz8svM54c6ATEvXp5E
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: list the types that are known to work for each template parameter
Follow up on the template parameter requirements page: state, for every
template parameter, which concrete types work and where they stop working.
Each entry was verified by compiling and running a common workload (DOM
access, dump, parse, CBOR/MessagePack round-trip, flatten, hash) against that
instantiation.
Findings worth calling out:
- ObjectType no longer needs a key_compare member type, so the std::unordered_map
adapter only has to restore the template argument order. A hash-ordered
ObjectType works everywhere except unflatten(), which reconstructs an array
only when it meets the reference token 0 before the other indices.
- ArrayType: std::deque works when wrapped to add capacity(); std::list does not.
- StringType: std::pmr::string and std::basic_string with a custom allocator
compile for the DOM, dump, and parse, but not for the binary readers, flatten,
or diff, because the library assigns std::string values to string_t and
int_to_string cannot be overloaded for a type in namespace std.
- NumberFloatType: long double works for dump and parse but not for the binary
formats, which have no encoding for it.
- BinaryType: std::vector<std::byte> supports assignment, get, and the binary
formats, but neither dump nor std::hash<basic_json>.
Also record the object_comparator_t fix in its version history.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018hxZxz8svM54c6ATEvXp5E
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix unflatten and binary dumping for non-default configurations
unflatten() decided between array and object by looking at the first reference
token it happened to see for a node: it started an array only when that token
was 0. With a sorted object type the token 0 always arrives first, so the
result was correct by accident; with an object type whose iteration order is
unspecified, {"/c/2":3,"/c/1":2,"/c/0":1} unflattened to an object with the
keys "0", "1", and "2" instead of an array.
Collect the pointer prefixes that have a reference token 0 among their children
before building the result, and let get_and_create() consult that set. The
outcome is now independent of the iteration order and matches, for every input,
what a sorted object type produced before: a value is restored as an array if
and only if one of its keys is 0. Iterating the flattened object in a different
order would have been simpler, but it would have changed the key order of the
result for insertion-ordered object types.
The serializer, std::hash, and the UBJSON writer converted the elements of a
binary value to an integer implicitly, which does not compile for a BinaryType
whose value type is std::byte, and which made dump() write the bytes of a
signed value type as negative numbers. Convert to std::uint8_t explicitly in
all three places, so every byte type dumps as 0..255. The default
std::vector<std::uint8_t> configuration is unaffected.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018hxZxz8svM54c6ATEvXp5E
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: note which Abseil containers can be used as template arguments
Checked against Abseil release 20250127.0 with the same workload as the other
entries on the page (DOM access, dump, parse, CBOR/MessagePack/UBJSON
round-trip, flatten, hash), with and without JSON_DIAGNOSTICS.
absl::flat_hash_map and absl::node_hash_map work as ObjectType through an
adapter that restores the template argument order and makes erase(iterator)
return the following iterator, which Abseil's returns as void. The page now
carries that adapter, and notes that absl::flat_hash_map does not keep
references to the mapped values valid across insertions while
absl::node_hash_map does. Both have a capacity() member, so JSON_DIAGNOSTICS
already refreshes the parent pointers conservatively for them.
absl::btree_map and absl::InlinedVector cannot be used at all: object_t and
array_t are formed while basic_json is still incomplete, and both inspect
their value type at class scope. std::map and std::vector are required by the
standard to tolerate this, third-party containers generally are not, so the
page states the constraint on its own rather than only per container.
absl::InlinedVector does work as BinaryType, where it is instantiated with a
complete type. absl::FixedArray and absl::Cord are not usable.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018hxZxz8svM54c6ATEvXp5E
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Relax the ArrayType and ObjectType requirements
Two requirements forced users of otherwise suitable containers to write a
wrapper, and neither was load-bearing.
array_t::capacity() was read in push_back(), emplace_back(), operator+=(), and
operator[](size_type), but set_parent() only looks at the value under
JSON_DIAGNOSTICS; without diagnostics it was computed and discarded. Read it
through array_capacity(), which reports unknown_size() when diagnostics are off
or when the array type has no capacity() at all, and treat an unknown capacity
as "the elements may have moved" so the parent pointers are refreshed
conservatively. std::deque now works as ArrayType, in both builds, and
capacity() is no longer named at all in a default build. Since the capacity is
now only meaningful for array insertions, it moves out of set_parent() into
set_parent_after_array_insert().
basic_json::erase(iterator) assigned the object's erase() return value, which
requires the container to return the following iterator. Abseil's hash maps
return void to avoid computing a successor the caller may not need. Detect that
and compute the successor before erasing; containers that return an iterator,
including the vector-backed ordered_map where a precomputed successor would be
wrong, keep the existing path.
Together these leave an Abseil hash map needing only an alias that restores the
template argument order, and no adapter at all for std::deque.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018hxZxz8svM54c6ATEvXp5E
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Do not require string_t to be convertible from std::string
Three places built a std::string and handed it to something expecting a
string_t: the UBJSON high-precision number reader, which every binary reader
instantiates, and the BSON writer's array element size calculation and write.
That silently required string_t to be implicitly convertible from std::string,
which std::string itself and types with a string_view conversion satisfy, but
many string types do not.
Construct the string_t explicitly from the data and size, which the
requirements already cover. This makes boost::container::string, eastl::string,
std::pmr::string, and std::basic_string with a custom allocator work as
StringType, none of which could previously be used with any binary format.
Add binary format coverage to the alt_string test, which had none, including a
UBJSON high-precision number -- the case that goes through the reader path.
BSON stays uncovered there: it additionally needs string_t::find(value_type),
which alt_string does not provide.
Also record which containers from Boost, Abseil, and EASTL work for each
template parameter, and correct two claims: std::pmr::string is usable after
this change, and tsl::ordered_map is not usable at all, because its iterators
expose the mapped value as const while basic_json modifies it in place.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: record compatibility for the common header-only hash maps
ankerl::unordered_dense (map and segmented_map), phmap (flat_hash_map and
node_hash_map), and robin_hood::unordered_flat_map all work as ObjectType
through the same adapter as Abseil's and Boost's hash maps, which only has to
restore the template argument order.
phmap::btree_map and robin_hood::unordered_node_map do not: like the other
btree containers they require a complete value type.
Note that none of these hash maps defines key_compare, so every one of them
depends on object_comparator_t falling back to default_object_comparator_t.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: record Folly and the remaining vector replacements
Folly works, with the caveat that its headers need C++20: folly::fbstring as
StringType, folly::fbvector and folly::small_vector as ArrayType,
folly::fbvector<std::uint8_t> as BinaryType, and folly::F14NodeMap as
ObjectType through the usual argument-order adapter. folly::F14FastMap is the
exception and requires a complete value type.
For ArrayType, boost::container::devector, boost::container::static_vector
(within its fixed capacity), and std::pmr::vector work as well.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: cover fifo_map, gtl, folly::sorted_vector_map, and Qt
nlohmann::fifo_map works through the adapter that has always been documented
for it, and preserves the insertion order. Restore its mention in the object
order page, which was dropped together with the tsl::ordered_map one: unlike
ordered_map it keeps a lookup index, so it is the insertion-ordered option
without the quadratic cost.
gtl::flat_hash_map and folly::sorted_vector_map work as well, the latter
through an alias that drops the allocator, whose value type it disagrees on.
gtl::btree_map does not, for the same reason as the other btree containers.
None of the Qt containers can be used, each for its own reason: QMap has no
value_type, QHash iterators yield the mapped value rather than a pair, QList
has no max_size(), QByteArray spells empty() as isEmpty(), and QString is
UTF-16.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: qualify the std::pmr::string support claim
Listing std::pmr::string as fully supported was an overclaim: it was only ever
checked with the default memory resource, which is not what PMR is for.
basic_json cannot be given an allocator or a memory resource, so a pmr string
inside a value always allocates from std::pmr::get_default_resource(), and
assigning an arena-backed string into a value silently drops its resource,
because polymorphic_allocator does not propagate on copy construction. Passing
polymorphic_allocator as AllocatorType does not compile either. Only the
process-global set_default_resource() redirects these allocations.
Say so, and separate the row from std::basic_string with a custom stateless
allocator, which is unaffected.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: remove a duplicated StringType compatibility section
The StringType section carried two 'Compatible types' tables and two copies of
the reference-implementation tip. The second table was a stale copy from before
the binary format string fixes and still listed std::pmr::string and
std::basic_string with a custom allocator as unusable, contradicting the
corrected table a few lines above it, and it dragged along the old explanation
that blamed int_to_string.
Drop the stale copy and put the surviving table before the notes, so the
'see below' in the std::pmr::string row points forwards.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: correct the template parameter requirements after independent verification
Every claim on the page was re-checked by compiling and running it, including
the rows that say a type cannot be used, which were checked to fail for the
documented reason and not merely to fail. Twenty-four claims were wrong.
The most consequential: the incomplete-type constraint applies to ObjectType
only. object_t is instantiated inside the class definition, because it is
probed for key_compare; array_t is only named there and is not instantiated
until basic_json is complete. So eastl::vector, QList and QVector are not
excluded by incomplete types at all -- they simply have no max_size() -- and
absl::InlinedVector is excluded for a subtler reason of its own.
Further corrections: ObjectType does not need erase(key), which has a fallback,
but does need at(key) for UBJSON output; only == and < are used, or == and <=>
under C++20, not all six; the documented adapter does not fit ankerl or
robin_hood. ArrayType needs no initializer-list insert, and value_type, the
(count, value) constructor and swappability are per-function, not always.
BinaryType needs a range insert for CBOR indefinite-length byte strings and
does not need push_back. StringType needs append(const StringType&)
unconditionally, and does not need operator!= or operator== against const
char*; empty(), resize(n) and reserve(n) are per-subsystem; int_to_string is
needed by diff, items and std::hash rather than by JSON Pointer or flatten.
BooleanType must be implicitly convertible from bool, and JSONSerializer's
second parameter need not carry a default.
std::pmr::string was wrong in the other direction this time: a moved-in string
does keep its memory resource, and later growth allocates from it. Only copies
land on the default resource.
Five requirement violations are not caught at compile time rather than the two
the page claimed; they are now listed together up front. Split every
compatibility table into what works and what does not, as the reasons in the
second half are the useful part.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Reduce the string_t and array_t members the library requires
Several members were required only because of how the library happened to
be written, not because the functionality needs them. Dropping them widens
the set of usable string and array types, and one of them was also a
performance problem.
string_t:
- c_str() is gone. Every call site already knew the length and passed it
along, so data() is enough. The one place that did not, the diagnostics
path in exceptions.hpp, now builds the token from data() and size(),
which also stops it from truncating keys that contain a null byte.
- back() is gone; the serializer indexes the last character instead.
- find(str, pos), replace(), and substr() are gone. escape() and
unescape() rebuilt the string with one replace() per escaped character,
which moves the tail every time: escaping a string of n characters that
all need escaping cost O(n^2). Both now scan with find_first_of() -- a
member the pointer parser already required -- and append whole runs, so
the common case is one search and one copy. Escaping 64000 tildes drops
from 717 ms to 20 ms; a string with nothing to escape gets faster too
(8.4 ms to 5.8 ms), because the scan is still a single memchr per pass.
json_pointer::split() takes its reference tokens with the
(const char*, size_type) constructor rather than substr().
- json_pointer::to_string() accumulates with concat<string_t> instead of
letting concat default to std::string and converting afterwards, so
streaming a json_pointer no longer requires string_t to be assignable
from a std::string.
array_t:
- at(size_type) is gone. basic_json::at(size_type) checked the index by
calling array_t::at() and translating std::out_of_range, which also
required the array type to throw that exact exception. It now compares
against size() and uses operator[]. The thrown exception, its message,
and the behaviour under JSON_NOEXCEPTION are unchanged.
The BSON writer wrote the terminating null byte out of the string's own
buffer (size() + 1). It now writes the byte itself, so string_t::data()
need not be null-terminated for to_bson().
The tests pin the reduced API: alt_string loses the five dropped members
and gains coverage of the escaping paths, and a std::vector whose at() is
hidden is used as an ArrayType.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: record the reduced string_t and array_t requirements
Drop c_str(), back(), find(str, pos), replace(), and substr() from the
StringType requirements and at(size_type) from the ArrayType ones, and
note the string assignment the JSON pointer code performs. Streaming a
json_pointer no longer needs assignability from a std::string.
Add the non-null-terminated data() to the list of violations that are not
diagnosed at compile time -- it was described in the StringType section
but missing from the summary at the top -- and correct the QString row,
which no longer fails for the c_str() it lacks.
JSON_CATCH_USER no longer wraps a catch of std::out_of_range: the last one
went away with array_t::at(). Describe what the library actually catches.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Use character literals for the signed BinaryType test
MSVC rejects char(0xFF) with C4310 (cast truncates constant value),
which the Windows workflow treats as an error. The character literals
carry the same byte values without a narrowing cast.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Do not instantiate a hash map with an incomplete basic_json in the tests
object_t is probed for key_compare inside the definition of basic_json, so
it is instantiated while basic_json is still incomplete. Whether a hash map
survives that depends on the standard library: libstdc++ 9 needs the size of
the mapped type to instantiate std::unordered_map's node type and rejects
the adapter, which broke the GCC 9 builds.
The test now derives its no-key_compare object type from std::map -- which
does cope -- and shadows the inherited key_compare member type with an
entity that is not a type, so the library's probe finds none, exactly as for
a hash map. The unflatten() order-independence checks in unit-json_pointer
already cover the behaviour that the unordered object type was there for.
The limitation is documented for std::unordered_map.
Also address two Clang-Tidy findings the earlier commits introduced:
erase_from_object() declares its iterator with auto, and at(size_type) checks
the type first and then falls through to the return instead of throwing from
an else branch.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Keep diagnostic key paths null-terminated
Building the token from data() and size() kept an embedded null byte in the
key, and since what() hands out a C string, that truncated the whole message
rather than just the key: to_bson() on a key containing U+0000 reported
"[json.exception.out_of_range.409] (/en" instead of the full explanation.
This broke test-bson under JSON_DIAGNOSTICS.
Constructing from data() alone stops at the first null byte, which is what
c_str() did before, so the message is unchanged -- without requiring
string_t to provide c_str().
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Do not parse the value in the array-at() test
JSON_DIAGNOSTIC_POSITIONS adds the byte range of the value to the exception
message, which a parsed value has and an in-memory one does not, so the two
message checks failed in that configuration. Build the array in memory
instead of parsing it; the test is about at(size_type) not needing
array_t::at(), and the byte range is beside the point.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Move the custom BinaryType tests into their own translation unit
The two sections added to unit-regression2.cpp brought a third full
basic_json instantiation into a translation unit that was already large.
With Clang on MinGW that pushed the object over the reach of a 32-bit
relocation and test-regression2_cpp20.exe failed to link:
relocation truncated to fit: IMAGE_REL_AMD64_REL32 against `.rdata'
unit-regression2.cpp is restored to exactly what it was before, and the
coverage moves to unit-custom-binary-type.cpp, next to the object and array
type tests it belongs with. The signed value type is now also covered in
C++11, where std::byte is not available.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Do not require the container iterators to be nothrow move constructible
iter_impl declared its defaulted move operations noexcept. The exception
specification a defaulted function gets implicitly follows from its members,
here internal_iterator, which holds the object and array iterators. libstdc++
gives std::deque's iterator a user-provided copy constructor without noexcept
before version 11, so the implicit specification is noexcept(false) and does
not match the declared one. That deletes the function -- and with g++ 4.8,
which predates CWG 1778, it is an error outright:
error: function 'iter_impl<basic_json<std::map, std::deque> >::iter_impl(
iter_impl&&)' defaulted on its first declaration with an
exception-specification that differs from the implicit declaration
So std::deque, which this branch documents as a usable array type, could not
be used with an older standard library. Leaving the specification to be
computed cannot mismatch; iteration_proxy_value already spells out the same
condition next door.
The default configuration is unaffected: json::iterator, json::const_iterator
and ordered_json::iterator stay nothrow move constructible and move
assignable, which the test now checks so it cannot regress unnoticed.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address two Clang-Tidy findings the custom container tests exposed
Both come from instantiating basic_json with containers other than the
default ones, and neither shows up with the Clang-Tidy version available
outside CI:
- insert(const_iterator, basic_json&&) forwards its by-value iterator to
the const-reference overload. performance-unnecessary-value-param asks
for the copy to be a move; it only fires for an iterator that is not
trivially copyable, as std::deque's is not. The NOLINT on the function
does not cover it, because the finding is reported where the parameter
is used rather than where it is declared. Move it, which is what the
check asks for and is a (very small) improvement in its own right.
- cppcoreguidelines-use-enum-class rejects the unnamed enum that shadowed
the inherited key_compare member type. An enum class would not do, since
it declares a type of that name and the probe would find it again; a
member function declaration hides the name just as well.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Assert the iterators' exception specification relative to the container
The test pinned that nlohmann::json's iterators stay nothrow movable after
iter_impl's defaulted move operations lost their declared noexcept. That is
not a property of the library, though: the exception specification is now
computed from the container iterators, so it holds only for standard library
implementations whose iterators are themselves nothrow movable.
MSVC's checked iterators before VS2017 are not -- _Iterator_base12 registers
the iterator with the container's debug proxy in a copy constructor that
carries no noexcept -- so the assertions fail on a Visual Studio 2015 debug
build, which is the one debug configuration in the AppVeyor matrix and has no
counterpart in the GitHub Actions matrix.
Assert what the change actually guarantees instead: the iterators are nothrow
movable exactly when the object and array iterators they are built from are.
That still pins the default configuration against a silent regression, and it
is true whatever the standard library provides.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Detect a void-returning erase() through a named trait
erase_from_object() distinguished its two overloads with a decltype of a
member call written inline in a default template argument. Every other
detection in the library goes through the detector machinery in detected.hpp
instead -- has_erase_with_key_type is the same question about the same member
function -- and the inline form is the one shape older compilers are least
reliable about.
Express it the same way: detect_erase_with_iterator plus is_detected_exact,
both of which the library already relies on elsewhere. No behaviour changes.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Give the custom container types only the constructors the library uses
The three container types in the new tests inherited every constructor of
their base with using Base::Base. That asks for more than the test needs: the
library builds an object or an array by default construction, by copy or move,
and -- when converting between two basic_json types or from an initializer
list -- from an iterator range. Declaring those directly makes the requirement
visible in the test, and keeps object types out of a corner where a compiler
has to declare std::map's whole constructor set for a derived class while
basic_json is still incomplete.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Temporarily disable the new custom container tests
AppVeyor is the only CI that builds MSVC 2015 and 2017, and it has now
rejected three heads of this branch. Its build log is not reachable from
where this is being worked on, so the verdict is a single bit and the cause
has to be narrowed down by bisection.
Everything else stays: the library changes, the reduced alt_string, and the
unflatten() tests. If AppVeyor passes with these three translation units
disabled, the cause is one of the six basic_json instantiations they add; if
it fails, it is in the library. Either way this commit is reverted.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Guard the disabled tests with a macro rather than #if 0
Clang-Tidy's readability-avoid-unconditional-preprocessor-if rejects a literal
#if 0. Use a macro that is never defined instead, which the check does not
look at. Still temporary, and reverted together with the previous commit.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Re-enable the array and binary container tests
AppVeyor passed with all three new translation units disabled, so the library
changes, the reduced alt_string, and the unflatten() tests are fine on MSVC
2015 and 2017; the cause is one of the six basic_json instantiations the new
tests add.
Bring back two of the three. If AppVeyor passes again, the cause is in
unit-custom-object-type.cpp, which is the one still disabled; if it fails, it
is in one of these two and needs one more split. Still temporary.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Diagnose two silently violated template parameter requirements
Both were on the list of requirements that are not caught at compile time and
corrupt values rather than failing, and both are a plain size comparison:
- A BinaryType whose value_type is wider than one byte, which the readers and
writers reinterpret as raw bytes anyway.
- A NumberUnsignedType too narrow to hold the absolute value of every
NumberIntegerType value, which makes basic_json(INT64_MIN).dump() yield -0
for std::int64_t with std::uint32_t.
Neither static_assert rejects a configuration that worked before: both only
fire where the result was already wrong. Also add the two comments the review
asked for, in write_bson_string() and calc_bson_array_size(), matching the
ones their counterparts already carry.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Align the template parameter tables and record what is now diagnosed
Every table in the page is reformatted so each column is exactly as wide as
its widest cell, which is what the review asked for in a dozen places: the
separator rows that ran two dashes long, the stray spaces, and the columns
padded well past their content.
The row listing six containers that require a complete mapped type is split
in two so that one cell no longer sets the width of the whole table.
Content changes: NumberUnsignedType is described as any unsigned integer type
at least as wide as NumberIntegerType rather than any unsigned integer type;
the two requirements that are now static_asserts move out of the list of
violations that are not caught at compile time; and the two places that
require a non-const operator[] say why data() will not do (std::string has no
non-const data() before C++17).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Bisect the other way: only the object container tests
The previous head touched only docs/, which AppVeyor's only_commits filter
skips, so it produced no build and no status at all -- the pull request looked
green without ever having been built on MSVC 2015 or 2017.
Swap the guards instead of repeating that step: unit-custom-object-type.cpp is
enabled and the array and binary translation units are disabled. AppVeyor
already passed with all three disabled, so a failure here pins the cause on
no_key_compare_json or void_erase_json, and a pass pins it on the array or
binary file. Still temporary.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Split the two object types apart
AppVeyor failed with only unit-custom-object-type.cpp enabled and passed with
all three new translation units disabled, so the cause is one of the two
object types in this file and not the array or binary ones.
Guard out void_erase_map and leave no_key_compare_map, which separates the two
constructs under suspicion: shadowing the inherited key_compare member type
with an entity that is not a type, and hiding the inherited erase with a
void-returning overload. A failure here points at the first, a pass at the
second. Still temporary.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Build the no-key_compare object type by composition, not inheritance
The "object type without key_compare" test failed on AppVeyor's MSVC
2017 jobs (/std:c++17): its no_key_compare_map derived publicly from
std::map and shadowed the inherited key_compare type with a same-named
member function, relying on ordinary member hiding to make key_compare
unreachable as a type for the library's detection trait. MSVC 2017
does not honor that hiding for a typename-qualified lookup performed
from outside the class and still resolves key_compare to the base's
comparator type, so object_comparator_t incorrectly picked it up
instead of falling back to default_object_comparator_t.
Wrapping a std::map by composition instead removes the base class
entirely, so there is no key_compare to find under any lookup rule,
on any compiler. Also drops the now-unneeded JSON_BISECT_CUSTOM_CONTAINER_TESTS
guard left over from narrowing this down: the void_erase_map test in
the same file was never the cause and is re-enabled unconditionally.
Verified locally with clang++ and g++ under C++17 and C++20.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Re-enable the array and binary custom-container tests
unit-custom-array-type.cpp and unit-custom-binary-type.cpp were still
guarded behind JSON_BISECT_CUSTOM_CONTAINER_TESTS from bisecting the
AppVeyor failure fixed in 7c39f3227, which was unrelated to either
file. The macro was never defined, so none of these tests actually ran
in CI. Verified locally with clang++ and g++ under C++17 and C++20
before removing the guards.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix indentation of custom_object_type per astyle
The one-line function bodies in the composition-based no_key_compare_map
(7c39f3227) do not match the project's Allman brace style, which the
ci_test_amalgamation job enforces with astyle. Reformatted with the
pinned astyle 3.4.13; no functional change.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: add compiled reference implementations for the container template parameters
Each of ObjectType, ArrayType, StringType, and BinaryType now links to a
minimal, self-contained header (docs/mkdocs/docs/examples/custom_*_type.hpp)
that wraps the corresponding standard container by composition and satisfies
every "Always required" member listed on that page. Unlike the prose
requirement lists, these are real code: each header has a companion .cpp that
instantiates a basic_json specialization with it and is compiled and run by
the existing ci_test_examples check (docs/Makefile's check_output_portable),
so the reference implementations cannot silently drift from what the library
actually requires. The .output files were generated with that same target.
StringType's existing pointer to tests/src/unit-alt-string.cpp's alt_string
is kept alongside the new header as a more thorough, battle-tested example.
Verified locally: astyle (pinned 3.4.13, project .astylerc) on the new files;
clang++/g++ under C++11/17/20 for each example against the amalgamated
header; `make check_output_portable` in docs/; `mkdocs build --strict` and
scripts/check_structure.py for the page itself.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Declare no_key_compare_map's accessors noexcept
The GCC C++20 job builds with -Wnoexcept and -Werror, and the standard
library takes noexcept(c.begin()) and noexcept(c.end()) in ranges_base.h and
range_access.h. Forwarding to std::map without repeating its noexcept made
those expressions false, which the warning reports as an error:
error: noexcept-expression evaluates to 'false' because of a call to
no_key_compare_map<...>::begin() [-Werror=noexcept]
note: but ... does not throw; perhaps it should be declared 'noexcept'
Give the accessors the exception specification of what they forward to.
std::map declares begin, end, cbegin, cend, empty, size, max_size, and clear
noexcept, so the wrapper does too. swap is left alone: std::map's is only
conditionally noexcept, and nothing asks for it.
void_erase_map is unaffected because it still derives from std::map and
inherits accessors that already carry the specification.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018hxZxz8svM54c6ATEvXp5E
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Write out what the defaulted constructor of no_key_compare_map implied
ci_test_gcc builds with -Weffc++, which asks for data to be initialized in a
member initialization list; a defaulted default constructor does not do that:
error: 'no_key_compare_map<...>::data' should be initialized in the member
initialization list [-Werror=effc++]
Writing the constructor out satisfies that but drops the exception
specification the defaulted one carried, which -Wnoexcept then objects to
where the standard library takes noexcept(construct(...)). Declare it the way
the defaulted constructor was: noexcept when the wrapped map's default
constructor is.
This is the cost of composition -- inheritance carried std::map's exception
specifications and initialization for free, and forwarding by hand has to
restate them.
Checked with the repository's own GCC warning set from cmake/gcc_flags.cmake,
all 346 flags, at C++11, C++17 and C++20: no diagnostics for this file, nor
for the two custom container translation units that were disabled while the
MSVC failure was narrowed down and are built again now.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018hxZxz8svM54c6ATEvXp5E
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Mark no_key_compare_map::swap noexcept
Clang-Tidy rejects a swap that is not:
error: swap functions should be marked noexcept
[cppcoreguidelines-noexcept-swap,performance-noexcept-swap]
It was left unmarked on the grounds that std::map::swap is only
conditionally noexcept, so an unconditional promise would be wrong for a
comparator or allocator that can throw while swapping. Both concerns are met
by taking the specification from the wrapped map rather than asserting one:
noexcept(noexcept(data.swap(other.data))). Clang-Tidy accepts that, and no
NOLINT is needed.
Last in the series of specifications that inheritance used to supply and
composition has to write out by hand.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018hxZxz8svM54c6ATEvXp5E
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Both overloads swapped the underlying container storage but never called
set_parents(), leaving elements moved into *this with stale m_parent
pointers (typically nullptr from the free-standing array_t/object_t).
This produced wrong JSON Pointer paths in diagnostic messages and could
trip assert_invariant() on subsequent copies. Mirrors the fix already
applied in swap(reference other).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
repology.org refuses requests coming from GitHub Actions runners. The
Material privacy plugin logs the failed download as a warning, but then
raises FileNotFoundError when it reads back the cache entry it never
wrote, so ci_test_build_documentation aborts the whole documentation
build. develop and every open pull request are currently red because of
this.
The badges are fine for readers -- repology only rejects non-browser
clients -- so exclude them from the plugin's asset self-hosting and keep
them as external references. All other external assets (fonts, shields,
CDN files) are still downloaded and self-hosted as before.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
basic_json::swap() (and the friend swap() that forwards to it) only
exchanged m_data.m_type/m_data.m_value, leaving start_position/end_position
untouched under JSON_DIAGNOSTIC_POSITIONS. This is inconsistent with
copy-assignment's operator=(basic_json), which swaps positions as part of
its copy-and-swap implementation, so after swap(a, b) each value ended up
with the other value's content but its own original position.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Reserve capped array capacity for definite-length binary arrays
CBOR, MessagePack, and the optimized [$type#count UBJSON/BJData form all
pass an exact element count to sax->start_array(len), but
json_sax_dom_parser::start_array() (and the callback variant) only used
len for an overflow check against max_size() and never reserved the
underlying vector, so each element triggered a reallocation cascade via
emplace_back().
Reserve upfront, but cap the reservation at 16384 elements: max_size()
for a std::vector is far larger than any realistic input, so an
unbounded reserve(len) would let a crafted/truncated header (e.g. CBOR
0x9A + a huge uint32 count with no data) trigger a multi-gigabyte
allocation attempt instead of the normal graceful parse_error. With the
cap, a hostile length still fails fast with the existing parse_error,
while realistic arrays get a single up-front allocation.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Make the huge-claimed-length DoS regression tests portable across size_t widths
On a platform where size_t is narrower than 64 bits (e.g. 32-bit mingw/msvc
x86), the previously-hardcoded huge test lengths either collide with that
platform's unknown_size() sentinel (CBOR/MessagePack, both using exactly
SIZE_MAX) or exceed the platform's smaller vector<json>::max_size()
(UBJSON/BJData's 0x7FFFFFFF), so the header is now rejected outright
(out_of_range.408) instead of being accepted and only found short of data
(parse_error.110). Both are safe, bounded rejections of the hostile input;
the property under test -- no attempt to allocate space for billions of
elements -- holds either way. Accept both outcomes instead of pinning the
64-bit-only exact result.
Also fixed an unrelated clang-tidy finding (google-readability-casting) on
the functional-style std::size_t(...) casts in the neighboring "arrays of
various sizes" section.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix remaining CI failures in the huge-claimed-length DoS regression tests
- Apply the same google-readability-casting fix (std::size_t{N} instead
of std::size_t(N)) to the "arrays of various sizes" section in
unit-msgpack.cpp, unit-ubjson.cpp, and unit-bjdata.cpp; only
unit-cbor.cpp had been fixed previously, since clang-tidy's build
didn't get far enough to report the other three in the same pass.
- json_sax_dom_parser::start_array()'s max_size() check calls JSON_THROW
directly rather than going through sax->parse_error(), so unlike the
scanner's own "not enough data" parse_error it is not gated by
allow_exceptions=false. On a platform where a header's claimed count
exceeds max_size() (e.g. 32-bit, for UBJSON/BJData's 0x7FFFFFFF test
value), from_ubjson/from_bjdata(input, true, false) can therefore still
throw instead of returning a discarded value. Make that assertion
tolerant of either outcome, same as the main exception-catching check
above it.
- Guard all four "a huge claimed length..." SECTIONs with
#if !defined(JSON_NOEXCEPTION), matching this test suite's existing
convention for exception-dependent tests: under JSON_NOEXCEPTION,
JSON_THROW never produces a catchable C++ exception at all (it aborts
the process), so a section that relies on try/catch to distinguish
between two acceptable outcomes cannot be expressed under that build
configuration regardless of platform.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Use (std::min)(len, reserve_cap) instead of a ternary in start_array()
Addresses review feedback from @gregmarr on PR #5476.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Discard dump()'s [[nodiscard]] return value in an exception-only check
CHECK_THROWS_WITH_AS(j.dump(), ...) called dump() only to trigger and
catch the exception, but never used the return value. dump() is
warn_unused_result, so GCC's pedantic build (-Werror --all-warnings)
rejected it as -Werror=unused-result, breaking ci_test_gcc. Wrapped in
utils::ignore_return_value(), matching every other such call in this
file.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Mark container_frame top as const in CBOR/UBJSON readers
clang-tidy's misc-const-correctness flagged these on PR #5520's CI: the
BSON sibling copy was already const, but these two were left mutable
even though only container_stack.back().remaining is ever written.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The benchmark suite covered parsing JSON text, dumping, and serializing to
CBOR, but only one binary read: FromMsgpack. Nothing measured from_cbor,
from_ubjson, from_bjdata or from_bson, so a change to binary_reader.hpp had
no baseline to be compared against.
Add read benchmarks for every format, in the two shapes that matter: from a
contiguous buffer, which is what most callers pass, and from a FILE*, which
is what FromMsgpack already measures and which compiles to different code.
FromMsgpack itself is left untouched so its numbers stay comparable across
releases. The input is derived at setup time by serializing a parsed test
file, because the test data repository ships JSON only.
The test files are wide and shallow, but the readers' cost is per container,
so add three value shapes they do not cover -- deeply nested containers, many
sibling containers, and one flat array of numbers -- plus an indefinite-length
CBOR string, a form the writer never emits and which therefore has to be
assembled by hand. UBJSON and BJData are also captured in their size- and
type-annotated form, which the readers handle in a separate code path.
BSON requires an object at the top level, so it cannot reuse the array-rooted
test files; it is captured on the object-rooted ones, and the shapes are
wrapped in an object so every format measures the same value. The setup marks
the benchmark as skipped rather than letting the exception escape if that
requirement is ever violated.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Read BSON documents without recursing per nesting level
An embedded document (record type 0x03) or array (0x04) was read by calling
back into the document reader, which read its element list, which called the
element reader again for the next embedded one. The native call stack
therefore grew with the nesting depth of the input, and about seven bytes buy
a level, so a document of a few hundred kilobytes crashes the process
(#5104). This is the last of the four binary formats to still do that.
Apply the same shape as the other three: open_bson_document() reads the size
prefix and opens the document, parse_bson_element_internal() calls it for both
record types instead of recursing, and parse_bson_internal() loops over the
element list of whichever document is innermost, closing it when its
terminator is reached and resuming the one below.
check_bson_document_size() is unchanged, and so is when it runs: a document is
still measured from the byte before its size prefix to the byte after its
terminator, and still reported before the end event. The frame carries those
two values, which is what a per-document check needs once the reads are
interleaved rather than nested. Nothing else about the element reader changes.
unit-bson passes unchanged. Round trips through to_bson of nested objects,
arrays, arrays of objects and mixed nesting are identical to the previous
commit, as are the errors for a truncated document, an unsupported record
type, a negative size and a size that does not match, including their byte
offsets. A 30,000-level document built by to_bson is now read to completion
where it used to crash.
Note for sequencing: #5185 changes parse_bson_internal(), the element list and
the array reader, which are the functions this commit restructures. It should
land first; this commit then keeps its checks and moves them onto the loop.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Make parse_bson_internal's end-of-document top a copy, not a reference
Same issue as the CBOR and UBJSON/BJData readers: top aliased
container_stack.back() and was read (top.is_object) right after
container_stack.pop_back() ended its lifetime. A copy stays valid
regardless of what happens to the stack; nothing here mutates the live
entry, so no field needs to go through container_stack.back() directly.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Read UBJSON and BJData containers without recursing per nesting level
get_ubjson_array() and get_ubjson_object() read their elements by calling
back into the value reader, which called them again for a nested container,
so the native call stack grew with the nesting depth of the input. '[' alone
opens a container, so half a million of them crashes the process before the
input runs out (#5104). The optimized forms reach the same path through a
size or type annotation, and in plain UBJSON '[' and '{' are permitted as the
type of an optimized container, so "[$[#i\x01" repeated nests just as deeply
at six bytes a level.
Both readers now only open their container, and parse_ubjson_internal() loops:
it closes the containers that have ended, claims the next element of the
innermost one, reads its key when it is an object, and works out the marker
of the value to read next. That last part is where the formats differ, and
the loop follows what the four element loops used to do:
- a sized, typed container gives its elements no marker of their own
- a sized, untyped container reads one for each element
- a container that ends at a marker has the byte already, from the test
against ']' or '}'; for an object it is the first byte of the key
The ND-array wrapper and the 'B' binary shortcut stay as they are. Both read
a complete value rather than opening a container, and their elements are
always scalars: BJData does not permit '[' or '{' as an optimized type, which
is also why only plain UBJSON needed the type-marker case above.
A container of no-ops keeps its behaviour of holding no elements while still
announcing its declared size to the SAX parser, by opening it and then
setting its count to zero.
unit-ubjson and unit-bjdata pass unchanged, 1.39 million assertions between
them, and a behaviour comparison against the previous commit over every
container form -- sized, unsized, typed, untyped, empty, no-op, ND-array,
binary, and the forms nested inside one another -- gives identical values,
error codes, messages and byte offsets. 500,000 levels of each vector now
report a parse error instead of crashing, and a well-formed 100,000-level
value is read to completion.
The driver costs about 3 % on parsing 60,000 small objects and one array of a
million integers, for the reason given in the previous commit; reading the
frame once per element rather than per branch halved what it cost before.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Make the UBJSON/BJData advance loop's top a copy, not a reference
Same issue as the CBOR reader: top aliased container_stack.back() and
was read (top.is_object) right after container_stack.pop_back() ended
its lifetime. A copy stays valid regardless of what happens to the
stack; the one place that mutates the live entry (--top.remaining) now
goes through container_stack.back() directly.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Read CBOR containers and tags without recursing per nesting level
get_cbor_array() and get_cbor_object() read their elements by calling back
into the value reader, which called them again for a nested container, and a
tag was handled by reading the tagged value the same way. All three cost
native stack, and all three cost a single byte to encode: 0x9F opens an
indefinite-length array, 0x81 a one-element array, and 0xC2 is a tag. Half a
million of any of them crashes the process before the input runs out (#5104).
Apply the shape the MessagePack reader already uses: the open containers live
on the heap stack, parse_cbor_value() reads a single value and only opens a
container rather than reading it to its end, and parse_cbor_internal() loops,
resuming the innermost container after each element.
Two things are specific to CBOR. An indefinite-length container ends at a
break marker rather than at a count, and testing for that marker consumes a
byte which is the first byte of the next element when it is not one; the
frame's count is npos for those, and the driver tracks whether the next value
starts at a fresh byte. And a tag is not a value of its own: instead of
reading the tagged value by recursing, the value reader reports that a tag was
read and the driver reads on, so a chain of tags costs no stack at all.
The switch that decodes a value is unchanged apart from the twelve container
cases and the two tag sites. Verified against the previous commit over
definite and indefinite arrays and maps, all four counted forms, empty
containers, nesting of the forms inside each other, truncated inputs, and all
three tag handlers: identical values, error codes, messages and byte offsets.
500,000 levels of each of the three vectors now report parse_error.110 instead
of crashing, and a well-formed 200,000-level value is read to completion.
On performance: the driver does per element what a counted loop used to do
per container, and CBOR pays for it more than MessagePack because the value
reader also has to be told whether to fetch a byte. Parsing 60,000 small
objects and one array of a million integers is 3 to 4 % slower than the
recursive reader, measured over five alternating runs. Against develop the
same two inputs are about 44 % faster, because the entry point no longer
copies the value it parsed; the earlier commit in this series is what pays
for that.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Make parse_cbor_internal's top a copy so it survives pop_back()
top aliased container_stack.back(), and was still read (top.is_object)
right after container_stack.pop_back() destroyed the element it aliased.
Nothing currently reorders those two lines, but the comment claiming the
reference's lifetime was already fine only accounted for reallocation
from a push, not this. A trivially-copyable container_frame makes top a
copy instead, so reads of it stay valid regardless of what happens to the
stack; the one place that mutates the live entry now does so through
container_stack.back() directly rather than through top.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
get_msgpack_array() and get_msgpack_object() read their elements by calling
back into parse_msgpack_internal(), which calls them again for a nested
container. The native call stack therefore grew with the nesting depth of the
input, and each level costs only one byte to encode: 0x91 is a one-element
array, so a few hundred thousand of them crash the process before any of the
input is rejected (#5104).
Keep the open containers on a heap stack instead, the way
parser::sax_parse_internal() has always done for JSON text. A frame records
how many elements are left and whether to close with end_object() or
end_array(); parse_msgpack_value() reads a single value and, for a container,
only opens it; and parse_msgpack_internal() loops, resuming the innermost
container after each element and closing it when its count runs out. Whether
the value that was begun is complete is answered by the stack being empty, so
no separate bookkeeping is needed.
The switch that decodes a value is untouched apart from the six container
cases, which now call enter_container() rather than a reader that loops. That
keeps this diff to the control flow and leaves the decoding of every other
type byte-identical.
enter_container() is the only place a binary reader emits start_object() or
start_array(), so a check that rejects a container can be added there once and
is guaranteed to run before the start event. The frame type and the stack are
shared, ready for the other three formats.
Verified against develop over empty, nested, counted (array 16/32, map 16/32)
and truncated inputs: identical values, error codes, messages and byte
offsets. 300,000 levels now report parse_error.110 instead of crashing, and a
well-formed 300,000-level value is read to completion through the SAX
interface, where develop crashes.
Reading such a value into a basic_json needs the return-by-move change as
well, without which the recursive copy constructor overflows on the way out;
that is the parent commit, and the test for the value path covers the two
together. Timing is unchanged: parsing 60,000 small objects and one array of
a million integers is within run-to-run noise of develop either way.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Linking test-regression2 with clang and MinGW fails with
relocation truncated to fit: IMAGE_REL_AMD64_REL32 against `.rdata'
once the translation unit grows past a certain size: the code can no longer
reach the read-only data it references within the range of a 32-bit
relocation. The file is one of the largest in the test suite and had been
sitting just under that limit, so an unrelated change elsewhere in the
library is enough to tip it over. It is already the second such file --
unit-regression1.cpp was split for size before -- and windows.yml already
carries a workaround for the same limit hitting the debug sections of this
same target, where -g0 was enough because that relocation was against
`.debug_line'. This one is against `.rdata', which no compiler flag avoids.
Move the second half of the regression tests, and the helper types only they
use, into unit-regression3.cpp. The sections are independent -- every
statement in "regression tests 2" was already inside a SECTION -- so they
move unchanged, and the counts confirm nothing was lost: 168 assertions
before the split, 50 plus 118 after.
The result is that both files are comfortably smaller than the one that used
to link, measured with clang at -O1 for C++20:
read-only data text object
before 58,233 1,287,764 3,158,120
unit-regression2.cpp 48,161 1,012,988 2,522,296
unit-regression3.cpp 41,710 772,704 1,878,880
No CMake change is needed: tests/CMakeLists.txt globs src/unit-*.cpp, so the
new file is picked up and built for every standard like its siblings.
CONTRIBUTING.md pointed contributors at unit-regression2.cpp for new bug
tests; it now points at the smaller file and says why the two exist, so the
split does not quietly undo itself.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Bound UBJSON optimized arrays of a valueless type
An element of type 'Z' (null), 'T' (true) or 'F' (false) is encoded by its
type marker alone, so an optimized UBJSON array of one of those has no
payload: reading an element consumes no input at all. Its declared count is
therefore the only thing that decides how much is allocated, and nothing
bounded it. "[$Z#l" and a four-byte count is nine bytes of input describing
two billion values; #2793 reports 35 GB and 150 seconds from ten bytes, and
OSS-Fuzz has an out-of-memory and a timeout report for the same shape.
Every other type costs at least one byte per element, so the end of the input
bounds it. 'N' (no-op) is already skipped rather than stored. Objects are not
affected either: each element is preceded by its key, which costs bytes. And
BJData already refuses these markers as an optimized type, so this is a plain
UBJSON matter.
Reject a count above 1,048,576 elements for those three types with
out_of_range.408, the code this reader already uses for a declared size it
will not honour. The check runs before the SAX start event, so no container
is opened and then abandoned.
Rejecting on the read side alone would break the guarantee that anything
to_ubjson() writes can be read back, and would trip the round-trip assertion
in fuzzer-parse_ubjson.cpp. So the writer falls back to the unoptimized
encoding, one byte per element, for arrays of these types above the same
limit. Its decision depends only on the array's size, which is identical for
a value and for anything parsed back from it, so the round trip is stable.
No existing test changes: the largest such count in the test suite is 65,793.
The excessive-size test that already used this shape still passes, now
rejected a little earlier than by the max_size() check it used to reach.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Note the 1,048,576 valueless-array limit as (1 << 20) in the docs
Addresses review feedback from @gregmarr on PR #5504: spell out the
binary/hex form next to the decimal count so it reads as the round
power-of-two it is, matching how include/nlohmann/detail/input/binary_reader.hpp
defines max_valueless_container_size. Applied in both docs/exceptions.md
and ubjson.md, as requested.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
get_ubjson_size_type() takes an inside_ndarray parameter saying whether it is
being called for an ndarray's dimension vector, where another ndarray is not
allowed. It then seeded the flag it passes down to get_ubjson_size_value()
with `false` rather than with that parameter, and only consulted
inside_ndarray afterwards, on the '$' branch.
So on the '#' branch nothing stopped the descent: every "#[" pair of an input
like "[" followed by "#[#[#[..." opened another dimension vector, several
native stack frames deeper each time, and the recursion was only reported on
the way back out. 100,000 pairs crash the process. This is #5104 again, in a
path that has nothing to do with containers.
Seed the flag with inside_ndarray, which is what get_ubjson_size_value()
documents it wants: "for input, `true` means already inside an ndarray vector
or ndarray dimension is not allowed". The nested '[' is then refused where it
is read, so the length of the chain no longer matters.
Both post-checks gain `&& !inside_ndarray`, because an ndarray was found
*here* only if the flag flipped -- get_ubjson_size_value() only ever returns
`true` when its initial value was `false`, as its documentation says. With
that, the "ndarray can not be recursive" branch is unreachable: a recursive
ndarray is now caught one level earlier, and reported as "ndarray dimensional
vector is not allowed" like every other nested dimension vector.
Three existing expectations move accordingly (vR2, vR4, vR6). All three now
fail earlier, and all three now report the same error that vR1, vR5 and vH
already reported for the same shape, which is the more consistent outcome.
Everything else is unchanged: valid 1D and 2D ndarrays, optimized containers
and plain arrays produce identical results, and unit-ubjson is untouched.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
get_cbor_string() and get_cbor_binary() handled the indefinite-length forms
(0x7F and 0x5F) by calling themselves once per chunk. Each chunk therefore
cost a native stack frame, and since a chunk may itself be an indefinite-
length string, an input of repeated 0x7F bytes reached one frame per input
byte: 200,000 of them crash the process with SIGSEGV before a single byte is
rejected. This is the same defect as #5104, in a path the container-level
work does not touch.
Count the open levels instead of recursing through them. That is enough here
because every chunk is appended to the same result -- get_bytes() writes at
result.size() -- so there is no per-level state to keep. The temporary chunk
string and its copy into the result go away with the recursion.
The definite-length cases move to get_cbor_string_chunk() and
get_cbor_binary_chunk() unchanged, including their error messages, which
still name 0x7F and 0x5F because those are handled one level up.
Behaviour is unchanged. Comparing against develop over the interesting byte
sequences -- empty, single-chunk, nested, over-closed and truncated forms,
both strings and byte arrays, and an indefinite-length map key -- produces
identical values, error codes, messages and byte offsets. The 200,000-level
input now reports parse_error.110 at byte 200001 instead of crashing.
Note that nesting these is not valid CBOR: RFC 8949, Section 3.2.3 forbids
it. This does not change that either way -- it has always been accepted, and
rejecting it is a separate decision (#5317, #5325). Should it be rejected
later, that is now one condition on the level counter rather than a change to
the control flow.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Move the scanned string into the value instead of copying it
The SAX interface documents that the string handed to json_sax::string() may
be moved from, and the DOM handlers already move the one handed to binary().
string() did not, so every string value was copy-constructed out of the
lexer's token buffer, which then kept the buffer alive at its high-water mark
until the next token overwrote it.
Moving hands that buffer to the new value instead. The allocation count is
unchanged - the value needed one either way - but the copy is gone.
jeopardy 247.3 ms -> 240.9 ms (-2.6%)
citm_catalog 4.61 ms -> 4.48 ms (-2.8%)
40k 30-char strings 7.80 ms -> 7.64 ms (-2.1%)
Note this deliberately does not extend to the object key. Moving the key
hands the lexer's buffer - sized for the largest token seen so far - to a key
that is usually short, so the next value has to grow a fresh buffer. Measured,
that costs 11.9% on a document of many small keys with longer values.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Spell out the move rationale at every handle_value(std::move) site
The comment explaining why the value is moved sat only on
json_sax_dom_parser::string(), and the callback parser's string() pointed
at it with "see json_sax_dom_parser::string()". That reference cannot be
searched for - the function is declared as `bool string(string_t& val)`
inside the class, so the qualified name appears nowhere - and the two
binary() overloads, which have always moved, carried no explanation at all.
Put the same comment on all four sites and name json_sax, which is
greppable, instead of a member that is not.
Comment-only; no generated code changes.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
When a parser callback rejects a value, the placeholder stored for it has to
be removed from its parent again. remove_discarded_value() found it by
scanning the parent from the beginning, so filtering a container cost one
scan per rejected member - quadratic in the number of members of a single
container.
A rejected value can only ever be the one most recently added to its parent:
the last element of an array, or the placeholder key() stored under the
current key in an object. Record that key alongside the existing
key_keep_stack, and for a container record it again alongside ref_stack so
end_object()/end_array() can find it in the parent. Removal is then O(1) for
an array and O(log n) for an object, and finding nothing there means nothing
was stored, so there is nothing to remove.
The key for a container is read before handle_value() may consume it, so it
is also correct when the callback rejects the container at its start event
and it never reaches its parent at all.
Discarding half the members of one object, before -> after:
members value rejected container rejected at start
16 000 392 ms -> 3.7 ms 803 ms -> 8.2 ms
64 000 6238 ms -> 14.6 ms 12651 ms -> 32.2 ms
128 000 25339 ms -> 30.6 ms
Results are unchanged: 48 000 randomized documents parsed under 12 different
filtering callbacks - covering duplicate keys, empty keys, rejected keys and
containers rejected at both their start and end events - produce byte
identical output before and after.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Every input in the benchmark corpus is minified or only lightly spaced, so
none of them exercise the lexer's whitespace handling. Real-world JSON is
frequently indented - configuration files, pretty-printed API responses,
anything kept under version control - where the insignificant whitespace can
outweigh the data itself.
Add a ParseIndented family that re-serializes each document with an
indentation and parses that. The content is identical to the matching
ParseString row, so the pair isolates the cost of the whitespace alone.
Measured here, parsing the indented form costs 16-34% more than the minified
form of the same document, which nothing in the suite currently reports.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Stop dump() from heap-allocating its output adapter per call
The serializer held its output sink as output_adapter_t<char>
(a std::shared_ptr<output_adapter_protocol<char>>), which dump() and
operator<< built via make_shared -- one heap allocation per call for a
sink that only wraps a reference to the caller's string or stream.
Hold the sink as a non-owning output_adapter_protocol<char>* instead and
construct the concrete adapter on the stack at the call site. The write
path (o->write_characters) is unchanged, so output is byte-for-byte
identical; a compact dump() of a small object drops from 2 heap
allocations to 1 (only the returned string remains), ~3% faster.
Completes the per-call allocation cleanup on this branch, which already
removed the indent_string buffer (both were reported in #5413).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1oJ2ggRHS37zeVe94QTA1
Signed-off-by: Claude <noreply@anthropic.com>
* Take the output adapter by reference at the serializer ctor
Per review: the serializer still holds the adapter as a non-owning
pointer, but the constructor now takes output_adapter_protocol<char>&
and takes its address internally, so every call site passes a
reference. A reference cannot be null and reads as a borrow, which
makes the lifetime contract harder to get wrong than handing over a
raw pointer. The stored member and the write path are unchanged.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Claude <noreply@anthropic.com>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Co-authored-by: Claude <noreply@anthropic.com>
* Add SWAR bulk fast path to string serialization (dump_escaped)
When ensure_ascii is false, dump_escaped previously ran every byte of
every string and object key through the UTF-8 DFA decoder, even for the
common case of ordinary text with nothing to escape. This mirrors the
per-byte cost the parser had before the contiguous fast paths.
At a character boundary, bulk-copy the longest run of bytes that need no
escaping using string_bulk_run() - the same SWAR scanner and UTF-8 bulk
validator the lexer's contiguous path uses - and only fall back to the
byte-at-a-time DFA loop for the first byte that needs individual handling
(a quote, backslash, control character, or ill-formed/truncated UTF-8).
Because every "hard" or invalid byte is still processed by the unchanged
byte path, escaping output and error handling (including strict-mode
error 316 position and message) are byte-identical to before.
The ensure_ascii=true path is unchanged: it must escape non-ASCII and
0x7F, which string_bulk_run does not stop on, so a separate predicate
would be needed for it.
Verified byte-for-byte identical dump output against the pre-change
implementation across ~20k randomized byte strings plus curated edge
cases (all escapes, control chars, valid multibyte, surrogates,
overlong, truncated sequences) for both ensure_ascii settings and all
three error handlers, in C++11/17/20 at -O2/-O3.
Throughput (g++ -O3, ensure_ascii=false, vs pre-change):
long ASCII strings 4.2x
twitter-like objects 2.3x
dense CJK 1.4x (further headroom with JSON_USE_SIMDUTF)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Buffer serializer output and add ensure_ascii string fast path
Two further serialization speedups on top of the ensure_ascii=false bulk
copy, both reusing the SWAR primitives in detail/input/string_scan.hpp.
1. Internal write buffer (devirtualization). Every structural character
('{', '"', ',', ...) previously went straight to the output adapter
through a virtual call. Route all writes through put_char/put_chars
into a 1 KiB buffer that flushes in bulk; the public dump() flushes
once the top-level value is done (the recursive worker is split out as
dump_internal). Runs larger than the buffer are written straight
through, so large payloads are not copied twice. This is the dominant
cost for object/array-heavy values.
2. ensure_ascii fast path. dump_escaped previously ran the UTF-8 DFA over
every byte when escaping non-ASCII. Add find_ascii_copyable_run() (a
SWAR scan stopping at '"', '\\', < 0x20, 0x7F, and >= 0x80) so runs of
printable ASCII are bulk-copied, with the byte path handling each
escape/non-ASCII byte exactly as before.
Behavior is unchanged: dump output is byte-for-byte identical to the
previous implementation across ~20k randomized byte strings plus curated
edge cases (all escapes, control chars, 0x7F, valid multibyte,
surrogates, overlong, truncated), for object/array/pretty output, both
ensure_ascii settings, and all three error handlers, in C++11/17/20 at
-O2/-O3. New unit tests cover the buffer flush boundaries, the escape and
0x7F handling, multibyte under both settings, and invalid-UTF-8 handling.
Throughput (g++ -O3, vs the ensure_ascii=false-only baseline):
long ASCII, ensure_ascii=0 4.2x
long ASCII, ensure_ascii=1 4.1x
twitter-like objects 2.7x
dense CJK 1.8x
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Flush serializer buffer in dump_escaped unit test
test-convenience failed (macOS finished first; the failure is
platform-independent) because check_escaped() calls the internal
serializer::dump_escaped() directly and then reads the output stream.
Since dump_escaped() now writes into the serializer's internal write
buffer, the bytes were still buffered and the stream was empty.
Expose flush() under JSON_PRIVATE_UNLESS_TESTED (same visibility as
dump_escaped) and flush in check_escaped() before inspecting the output.
Per-string flushing inside dump_escaped() was rejected on purpose: it
would defeat the buffering that makes object/array-heavy dumps faster.
Library behavior is unchanged (flush()'s body is identical; only its
access label moved).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Avoid deep recursion in serialization write-buffer test
The "many small structural writes exceed the write buffer" subcase built
a 1100-deep nested array and dumped it to force >1024 consecutive
single-character writes through put_char (exercising the write buffer's
flush-when-full branch). dump() recurses per nesting level, so on MSVC
debug builds (smaller default stack, larger frames) this overflowed the
stack and crashed test-serialization; Linux/macOS have enough headroom to
hide it.
Replace the nesting with a flat array of 500 empty strings. Each element
emits '"', '"', ',' via put_char, so the dump is a long run of
single-character writes (1501 bytes > the 1024-byte buffer) at nesting
depth two, hitting the same flush branch without deep recursion. Library
code is unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Split the write-buffer helpers and write indentation directly
Follow-up to @gregmarr's review: put_chars() was doing four unrelated jobs, so
give the two that can be made safe their own entry points.
- put_literal(): takes the literal by reference and deduces the length from the
array bound, so the 27 hand-counted lengths at the call sites can no longer
drift from the literals they describe. A literal is checked at compile time to
fit the buffer, so this path needs no write-through branch.
- put_buffer(): takes the fixed-size buffer itself rather than a bare pointer,
so the length can be checked against the buffer's own bound.
- put_indent(): memsets the indentation into the write buffer, filling and
flushing it as needed. This removes indent_string entirely, and with it both
bugs of #5186: the indentation string was grown by doubling, which is not
enough when indent_step more than doubles it (a heap over-read - dump(2000)
read 2000 bytes out of a 1024-byte string), and the grown part was filled with
a space instead of the configured indent_char. next_indent() keeps that PR's
assertion against the unsigned indentation accumulation wrapping on deep
nesting.
put_chars() keeps the two cases that are genuinely a pointer and a count: the
run-length copies out of the string being escaped, and to_chars() output.
Tests cover an indent_step wider than the write buffer, a non-space indentation
character past the old growth point, and nesting whose accumulated indentation
spans several buffer-fulls. All three fail against develop.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fill the indentation buffer once instead of once per flush
@gregmarr's point on the fill-and-flush loop: flushing does not disturb what
the write buffer holds, so an indentation spanning several buffer-fulls only
has to be written into the buffer once and can then be handed to the adapter
as many times as needed. The loop re-filled it every time, doing work it
already knew was there.
put_indent() now fills the room left in the buffer, and if anything remains,
flushes, fills the buffer once, and re-flushes that same content. It also
returns early for a zero-width indentation, which is what the closing brace of
every outermost value asks for.
Measured over a dump(), counting memset calls and bytes inside put_indent:
indent before after
4 1 call / 4 B 1 call / 4 B
2000 2 calls / 2000 B 2 calls / 2046 B
100000 98 calls / 100000 B 2 calls / 2046 B
The wide case is now constant work rather than proportional to the indentation
width; ordinary widths are unchanged. Tests extended to cover several whole
buffer-fulls and an exact multiple of the buffer size.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Tighten the write-buffer helpers after review
More of @gregmarr's review on the put_* split:
- Reattach the put_chars() doc comment, which the new helpers had been
inserted in front of, leaving it describing put_indent().
- Compute the literal length once in put_literal() instead of spelling N - 1
at each use.
- Add put_string(str, start, end), which keeps the pointer arithmetic and the
bounds assertions inside the function instead of at the call site. With
dump_float()'s to_chars() output moved onto put_buffer() as well, put_chars()
now has no callers outside put_string()/put_buffer(): nothing passes a bare
pointer and a count any more.
- Carry the indentation as std::size_t rather than unsigned int. It is a size,
it is compared and combined with buffer sizes throughout, and the casts in
put_indent() disappear. next_indent() keeps its assertion, which is far
harder to trip on a 64-bit size_t but still reachable where that is 32 bits.
No output change: pretty and compact dumps, binary values included, are
byte-identical to develop.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Silence avoid-c-arrays on put_literal's array reference
clang-tidy flags the reference-to-array parameter under
cppcoreguidelines/hicpp/modernize-avoid-c-arrays, and the CI treats warnings as
errors. Binding to the array is the whole point here - it is what lets the
length be deduced from the literal instead of hand-written at the call site - so
suppress it the same way from_json(), to_json() and get_to() already suppress it
for their own T (&arr)[N] parameters.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Bound the descent of dump()
Serializing a container serializes its elements, so dump() descended into one
call per nesting level. A value nested deeply enough exhausted the call stack
and terminated the process with a segmentation fault - no exception, nothing
the caller could catch. Parsing such a value works, as the parser is
iterative, and so does destroying one, as #1436 made destruction iterative.
Bound how far the descent goes rather than take the call stack away from it.
The first 128 levels are written by exactly the code that always wrote them,
and only below that does dump_iteratively write out what is left, keeping the
containers it has entered on an explicit stack. Serializing can therefore no
longer exhaust the stack, however deeply a value is nested, while a value
nested less deeply than the bound pays only for one comparison per container.
Writing every value that way instead measured between 2% and 20% slower - 20%
on object-heavy documents - which is why the descent is kept for all but the
values that cannot afford it. The bound costs nothing measurable: between
-1.4% and +1.2% across compact and pretty output of number, integer, string,
object-heavy, wide-object and deeply nested documents.
The output is unchanged for every value. Both ways of writing a container
emit the separator in front of every element but the first, rather than
after every element but the last, which puts exactly one between each pair
and none at the end.
This fixes#5387 for dump(). The copy constructor is fixed in #5389.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fold ensure_ascii into the escaper and write bytes without dump_integer
Two hot spots that the write buffer and the bulk scanner left behind.
dump_escaped took ensure_ascii as a runtime flag and tested it inside the
loop, once per character run, although it cannot change while a string is
written. It is now a template parameter, dispatched once per string, which
folds the choice of scanner and lets each of the two be inlined into a loop
of its own. This is the hottest loop in the serializer: it runs over every
string and every object key.
A binary value's bytes went through dump_integer, which counts digits and
does 64-bit arithmetic for a number that is always in [0, 255]. dump_byte
writes the three digits it takes at most straight into the write buffer
instead. Any byte type that is not a plain unsigned byte is still left to
dump_integer, whose representation of it may differ.
Measured against the previous commit (medians of 9 interleaved runs, clang
-O3): binary values -33.8%, dense CJK with ensure_ascii -20.6%, key-heavy
objects -17.8%, deeply nested pretty output -17.9%, dense CJK without
ensure_ascii -11.8%, object-heavy documents -9.3% compact and -9.5% pretty,
a small value dumped in a loop -21.4%, wide objects -2.3%. Arrays of plain
ASCII strings measured 3.5% to 4.2% slower, the one shape that loses; number
and integer arrays are unchanged.
Also tried and dropped: leaving the write and string buffers uninitialized
rather than zeroing 1.5 KB per dump() call. It is worth -30% on small values,
but two nearly identical string workloads moved 18% apart in opposite
directions, so the measurements did not support it.
The output is unchanged for every value: the differential now also covers
every one of the 256 byte values, alone and together, in both binary layouts.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Write a byte without walking a pointer over the buffer
clang-tidy's misc-const-correctness reads the pointer dump_byte advanced over
the write buffer as one whose pointee could be const. Index the buffer
instead, which says the same thing without a raw pointer at all.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Parenthesize the reserve arithmetic in the deep-nesting test
clang-tidy's readability-math-missing-parentheses wants the multiplication
spelled out in reserve(6 * depth + 1), and CI treats its warnings as errors.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Do not scan for a copyable run that cannot exist
Under ensure_ascii, dump_escaped() calls find_ascii_copyable_run() at every
character boundary. When the text is dense non-ASCII - CJK, where every byte
is >= 0x80 - the scanner stops on its first byte and returns zero, so its SWAR
block runs once per character and buys nothing, on top of the escaping that
still has to happen afterwards.
A run can only be non-empty when the first byte is one the scanner may copy,
so test that single byte before calling it. Runs that do exist are found
exactly as before, so the bulk-copy win is unchanged; only the calls that were
always going to return zero are skipped.
Output is unchanged: the dump digest over canada/citm/twitter, in compact,
pretty and ensure_ascii form, matches develop byte for byte.
dump(ensure_ascii=true) develop before after
CJK text 3.54ms 4.25ms 3.36ms
CJK, no ASCII at all 3.09ms 4.02ms 3.02ms
Latin-1-ish text 4.39ms 3.04ms 2.93ms
plain ASCII 3.92ms 0.80ms 0.79ms
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address review of the write-buffer helpers
Three points from @gregmarr's review:
put_chars() is gone. It was the only entry point taking a bare pointer and
a count, and it existed only so put_string() and put_buffer() had something
to delegate to. Its body now lives in put_string(), and put_buffer() is
put_string(buffer, 0, length) - std::array already carries data() and
size(), so it satisfies the same interface a string does. Nothing appends
characters without a bound any more.
dump_escaped()'s documentation block was duplicated. The dispatcher was
inserted between the original comment and the function it described, and
the comment was copied rather than split. The worker now has its own short
comment saying why ensure_ascii is a template parameter.
The local in dump_byte() is deliberate, and is now documented as such:
writing through write_buffer[] is a char write, which may alias any object,
so with write_buffer_pos updated in place the compiler must reload and
store it around every digit. Measured on a dump of a 4 MiB binary value,
18.0 ms without the local against 7.4 ms with it.
Output is unchanged: byte-identical dumps across 77 files in compact,
pretty, ensure_ascii, pretty+ascii, indent 600 and tab-indent form.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address review: drop unneeded backslash-escapes and duplicate scan loop
'"' does not need escaping in a char literal, unlike in a string literal.
find_ascii_copyable_run() also duplicated the byte-at-a-time search that
already exists as the loop's own scalar tail; break into it instead of
re-deriving the offset in a second, near-identical loop.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Move pretty_print, ensure_ascii and indent_step into the serializer
None of these change over the life of a serializer, unlike current_indent
and depth, which do change on every recursive call. They are now captured
once in the constructor - matching indent_char and error_handler - instead
of being threaded through dump(), dump_internal(), dump_iteratively(),
dump_value() and dump_escaped() on every call.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Stop the serializer from holding onto std::localeconv()'s pointer
loc was only ever read twice, immediately, to seed thousands_sep and
decimal_point; nothing else in the class used it. A local in the
constructor body serves the same purpose without keeping the pointer
around for the serializer's lifetime.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Keep thousands_sep/decimal_point const via a small locale_chars struct
const members can't be assigned in a constructor body, so seeding them
from std::localeconv() meant either dropping const or holding onto the
lconv* for longer than needed. A sub-object computes both from the
pointer in its own constructor and is itself initialized in serializer's
mem-initializer-list, so the two chars stay const, std::localeconv() is
still called exactly once, and nothing outlives the constructor.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
tests/src/unit-class_parser_diagnostic_positions.cpp and
tests/src/unit-diagnostic-positions-only.cpp were maintained as near-copies
of unit-class_parser.cpp and unit-diagnostic-positions.cpp respectively, and
had drifted: trailing-comma handling, the #5342 filter-array/filter-value
sections, and the cross-input-adapter diagnostics test were never ported to
the positions-enabled copy.
Fold the position-specific assertions into the base files, guarded by
file a second time with the relevant macro set via CMake COMPILE_DEFINITIONS
(mirroring the existing test-comparison_legacy pattern) instead of
maintaining a separate source file. This removes the duplication and, as a
side effect, closes the coverage gaps above since the full test file now
compiles under JSON_DIAGNOSTIC_POSITIONS=1 as well.
Fixes#5417
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
write_bjdata_ndarray() validated that each _ArrayData_ element matched
the number kind (integer vs. float) named by _ArrayType_, but not its
range. An element that did not fit the target C++ type (e.g. 256 for
"uint8") was silently wrapped by the static_cast used to write it, or,
for "single", silently overflowed to infinity.
Range-check each element against the type named by _ArrayType_ before
writing it, reusing the existing fallback path that already encodes
the annotated object as a plain object for other invalid-annotation
cases in this function.
Fixes#5403.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Speed up number parsing in the lexer (fast paths from the fast_float/simdjson world)
The number scanner converted its already-validated digit buffer with
std::strtoull/std::strtoll/std::strtod. Those pull in locale and errno
machinery and dominate number-heavy parsing (strtod runs at ~6 M/s).
Replace them with dedicated parsers over the validated buffer:
- parse_integer_unsigned / parse_integer_signed: accumulate digits with
overflow detection, falling back to the float path on overflow exactly
as the strtoull/strtoll round-trip check did. Overflow behavior is
unchanged for narrower or wider custom number types.
- parse_float_fast: Clinger's exact fast path for `double` (<=19 significant
digits, |exp10| <= 22, significand < 2^53), where significand * 10^exp is
exact under IEEE round-to-nearest. This is the same fast path used by
fast_float/simdjson. It is bit-identical to strtod on this subset and
declines (falling back to strtod) otherwise. Only `double` uses it; float
and long double keep std::strtof/std::strtold via a templated overload.
Measured on representative data (g++ 13, -O3):
- integers: DOM parse +11%, SAX +25-34%
- floats: DOM parse +37%, SAX +70% (clang: float DOM ~1.9x)
No dependencies added; header-only and C++11-clean. Existing parser,
lexer, conversion and deserialization unit tests pass unchanged; a
3M-value random-double fuzz matches strtod bit-for-bit.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Add SWAR bulk string scanning for contiguous input (simdjson-style)
scan_string() read the input one character at a time through the input
adapter and classified every byte with a large switch. For contiguous
byte buffers we can instead scan 8 bytes at a time with a SWAR word test
that finds the first byte needing individual handling (the closing quote,
an escape, a control character, or a non-ASCII UTF-8 byte) and bulk-append
the ordinary run in one go.
- input adapters expose supports_bulk_scan / bulk_data / bulk_remaining /
bulk_skip for provably-contiguous, same-type, 1-byte iterator ranges
(raw pointers in every standard; std::string/std::vector/std::array and
friends additionally in C++20 via std::contiguous_iterator).
- the lexer gains a bulk_scan capability (gated on lazy_token_string so
bypassing the per-character capture cannot lose error diagnostics) and a
scan_string_bulk() fast path; streaming/wide/user adapters are unchanged
and keep the byte-at-a-time scanner.
The run contains no newline (all bytes < 0x20 are treated as special), so
position bookkeeping stays exact, and error tokens are still reconstructed
lazily from the consumed byte range. The SWAR special-byte test is pure
uint64_t arithmetic - no intrinsics, no runtime dispatch, C++11-clean.
Measured on representative data, pointer input, g++ 13 -O3
(string values discarded by accept() see the largest gains):
long ASCII strings: DOM +4.5x, SAX +14x, accept +17x (to ~2 GB/s)
short strings: DOM +15%, SAX +62%, accept +85%
escape-heavy: DOM +31%, SAX +26%, accept +28%
Same-input parity verified: 200k randomized documents (escapes, multibyte
UTF-8, surrogate pairs) accept/parse identically via the contiguous SWAR
path and the streaming byte path; unit lexer/parser/diagnostic-position/
deserialization/conversions suites pass unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Validate UTF-8 in the bulk string scanner (portable ~2-3x on non-ASCII text)
The SWAR bulk string path stopped at the first non-ASCII byte and handed
every multibyte character to the byte-at-a-time scanner, whose per-byte
get()/next_byte_in_range()/add() machinery runs at roughly half the speed
of validating straight from the buffer. As a result, dense non-ASCII text
(CJK, emoji, accented Latin) parsed ~10-15x slower than ASCII.
Fold well-formed UTF-8 into the bulk run: scan_string_bulk() now, on a
non-ASCII lead byte, validates one sequence with validate_one_utf8() -
which mirrors scan_string()'s per-byte switch ranges exactly (rejecting
overlong forms, surrogates, and out-of-range code points) - and appends it
in place, continuing until the closing quote, an escape, a control byte,
or an ill-formed sequence. All error handling still defers to the byte
path, so error messages and positions are byte-for-byte unchanged.
Because only well-formed content is fast-pathed and every rejection falls
through to the existing scanner, behavior is identical; the win is purely
throughput. Measured on pointer input (accept, string values discarded):
content g++ 13 clang 18
dense CJK 277 -> 648 ~605 MB/s (~2.3x)
dense emoji 299 -> 857 ~702 MB/s (~2.6-2.9x)
mixed 90% ASCII 246 -> 331 ~334 MB/s (~1.35x)
pure ASCII unchanged (~3.2 / 4.1 GB/s)
Verified: 2,000,000 randomized documents built from arbitrary bytes
(overlong, surrogate, truncated, out-of-range sequences) accept/reject and
parse identically via the contiguous path and the streaming byte path;
lexer/parser/diagnostic-position/deserialization/conversions suites pass
unchanged. Pure C++11, no intrinsics.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Route contiguous byte containers through the pointer adapter (fast paths in C++11)
json::parse(std::string) - the most common entry point - did not benefit
from the contiguous fast paths (bulk string scanning, UTF-8 bulk
validation, memcpy for binary formats) in C++11..17: std::string::iterator
is a library wrapper, not a raw pointer, and pre-C++20 there is no portable
way to prove it contiguous, so supports_bulk_scan was false. Only raw
pointers, string literals, and C-arrays (and, in C++20, anything modelling
std::contiguous_iterator) took the fast path.
Detect contiguous single-byte containers (std::string, std::vector<char>,
std::vector<std::uint8_t>, std::string_view, ...) via is_contiguous_byte_
container and route them through an iterator_input_adapter built from
data()/data()+size(). The generic iterator-based container overload is
constrained to exclude these, so the two overloads are disjoint and there
is no ambiguity (a plain competing overload loses to the greedy
forwarding-reference container overload on reference binding, and a factory
partial-specialization is ambiguous - both were tried and rejected).
The pointer keeps the container's own element type, so char_type - and
therefore all parsing behavior - is byte-for-byte identical to the iterator
path (const char* for std::string, const std::uint8_t* for
std::vector<std::uint8_t>); only the raw pointer additionally turns on the
fast paths. Lifetimes are unchanged: the container outlives the adapter for
the full parse expression, exactly as the iterators it replaces did.
Measured, C++11, json::parse/accept(std::string), g++ 13:
long ASCII strings: accept 201 -> 3200 MB/s (~16x), parse 174 -> 1444
dense CJK: accept 263 -> 697 MB/s (~2.6x)
short strings: accept 163 -> 243 MB/s (~1.5x)
Verified: char_type preserved for std::string (char) and
std::vector<std::uint8_t> (uint8_t); CBOR/MsgPack round-trips from
std::vector<std::uint8_t> unchanged; 1,000,000 randomized documents accept
and parse identically via std::string and via std::istream;
deserialization/user-defined-input/parser/lexer/conversions/diagnostic-
position suites pass (20,480 assertions); warning-clean on g++ and clang in
C++11/17/20.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Add optional simdutf backend for bulk UTF-8 validation (JSON_USE_SIMDUTF)
The bulk string scanner validates UTF-8 straight from a contiguous buffer.
The scalar validator caps at ~0.3-0.7 GB/s on non-ASCII text; a SIMD
validator reaches several GB/s. Rather than hand-rolling SIMD UTF-8
validation (easy to get subtly wrong - a from-scratch SSE attempt rejected
valid CJK), wire in the vetted simdutf library behind an opt-in switch.
simdutf is not header-only (it ships simdutf.cpp and uses runtime CPU
dispatch), so it is not vendored: defining JSON_USE_SIMDUTF includes
<simdutf.h> and routes the bulk validator through simdutf::validate_utf8;
the project supplies and links simdutf. Undefined (the default), nothing
external is included and the portable C++11 scalar path is used, so the
library stays header-only and its baseline behavior is unchanged.
Design keeps behavior identical either way:
- scan_string_bulk() now finds the run up to the next quote/escape/control
byte (non-ASCII allowed) and validates it in one shot; on the rare
validation failure it recomputes the exact valid prefix with the scalar
helper, so ill-formed input still falls through to the byte path and is
reported at the same position with the same message.
- the per-sequence scalar path is factored into scalar_string_bulk_run()
and is the default backend; the refactor is behavior-preserving and does
not change scalar throughput.
Verified: default and JSON_USE_SIMDUTF builds accept/reject/parse
identically across 2,000,000 arbitrary-byte documents and 1,000,000
mixed-escape/UTF-8 documents (differential fuzz vs the streaming byte
path); lexer/parser/diagnostic-position/deserialization suites pass under
both configurations (20,188 assertions with the backend enabled);
warning-clean on g++ and clang, C++11 and C++20, both configurations.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Add a contiguous fast path for scanning numbers
scan_number() reads a number one character at a time through the input
adapter (get()) and appends each byte to token_buffer (add()) before
converting. For contiguous input, the per-character get()/add() overhead
dominates: it is roughly two thirds of the time spent on number-heavy
parsing, far more than the value conversion itself.
Add scan_number_bulk_contiguous(), which parses the whole number token
straight from the input buffer: it validates and classifies the extent
with the same grammar as scan_number()'s state machine, materializes
token_buffer in one copy (substituting the locale decimal point exactly as
scan_number() does), advances the adapter, and reuses the shared
convert_number() tail. On anything it does not recognize as a well-formed
number it makes no state change and returns token_type::uninitialized, so
the caller falls back to scan_number(), which then produces the exact
diagnostic. Errors and their positions are therefore unchanged.
The conversion tail is factored out of scan_number() into convert_number()
so both scanners share it; the fast path is selected by tag dispatch on the
existing bulk_scan capability, so streaming/wide/user adapters are
unaffected.
Measured on pointer input, g++ 13 -O3:
- integers: parse +65%, accept +98%
- floats: parse +39%, accept +70%
Verified: 2,000,000 randomized number documents (including overflow-range
integers, long digit strings and %.17g doubles) parse identically via the
contiguous path and the streaming byte path, matching value, type and
round-trip text; the locale suite and existing parser/lexer/conversions/
deserialization tests pass; a new "lexer number fast path" test checks
contiguous-vs-streaming parity, token classification, and that malformed
numbers are rejected identically. Pure C++11, no intrinsics.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Move byte-level scan/parse helpers out of lexer.hpp
lexer.hpp had grown by ~600 lines of byte-level helpers that have no
dependency on the lexer's template parameters and clutter the state
machine. Move them, unchanged, into two focused headers as free functions
in namespace detail:
- number_parse.hpp: parse_integer_unsigned/parse_integer_signed (now
templated on the number type) and parse_float_fast (Clinger's exact
double fast path, with the decimal point passed as an argument instead of
read from a lexer member).
- string_scan.hpp: the SWAR string helpers (is_string_special,
swar_string_special, find_string_special, validate_one_utf8,
scalar_string_bulk_run) and the backend-dispatched string_bulk_run,
including the optional simdutf include and find_string_delimiter.
lexer.hpp now includes these and calls the free functions; the methods that
touch lexer state (scan_string, scan_number, scan_string_bulk,
scan_number_bulk_contiguous, convert_number) stay put. This is a pure code
move with no behavior change: lexer.hpp drops from 2357 to 1934 lines, the
now-unused <cstdint>/<cstring>/<limits> includes are removed, and the
free-function form makes the SWAR helpers reusable elsewhere (e.g. the
serializer's string escaping).
Verified: default and JSON_USE_SIMDUTF builds compile; 2,000,000 number and
2,000,000 arbitrary-byte-string differential-fuzz documents parse
identically to before; lexer/parser/conversions/deserialization/locale/
diagnostic-position suites pass (20,576 assertions); warning-clean on g++
and clang in C++11/17/20; the amalgamation regenerates and passes
check-amalgamation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix clang-tidy findings and document JSON_USE_SIMDUTF in the nav
- number_parse.hpp: use std::array for the powers-of-ten table
(avoid-c-arrays) and `auto` for the cast-initialized result
(modernize-use-auto), matching the codebase style (cf. the serializer's
utf8d table). Indexing casts keep the -Wsign-conversion build clean.
- add JSON_USE_SIMDUTF to the mkdocs navigation so the macro page is
reachable.
No behavior change; clang-tidy is clean on the new headers and the
amalgamation is regenerated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix number fast path for custom string types without assign()
The contiguous number fast path materialized token_buffer with
token_buffer.assign(data, len), but string_t is only required to provide
the minimal interface the rest of the lexer uses (push_back, append,
clear, operator[], ...). Custom string types such as the test's alt_string
do not implement assign(), so scan_number_bulk_contiguous() failed to
compile for them (unit-alt-string), breaking the gcc/clang standards and
old-compiler CI jobs.
reset() already clears token_buffer, so fill it with append() - which
alt_string and std::string both provide and which the string fast path
already relies on - instead of assign().
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Satisfy clang-tidy: parenthesize math and drop unused forwarding reference
The CI clang-tidy (newer than the locally available version) reported two
additional checks on the new code:
- readability-math-missing-parentheses: parenthesize the (a * b) + c digit
accumulations in number_parse.hpp.
- cppcoreguidelines-missing-std-forward: the contiguous-byte-container
input_adapter overload took a forwarding reference but only reads
data()/size() and never forwards it. It is already disjoint from the
generic container overload via SFINAE, so a plain const& is correct and
clearer (and keeps the container alive for the whole parse just as before).
No behavior change; char_type and routing are unchanged (std::string and
std::vector<std::uint8_t> still take the pointer adapter with char/uint8_t
char_type), CBOR/MsgPack round-trips and the 2M number fuzz still pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Use std::from_chars (Eisel-Lemire) for float conversion when available
The Clinger fast path is exact only for the "easy" subset (<=19 significant
digits, |exp10| <= 22); high-precision and scientific floats fall through to
strtod, where the failed Clinger attempt actually makes parsing a net loss.
std::from_chars implements the Eisel-Lemire algorithm in modern standard
libraries: locale-independent, correctly rounded, and fast over the whole
value range.
convert_number() now tries parse_float_from_chars() first (guarded by
__cpp_lib_to_chars, so C++11 and libc++-without-float-support keep the
Clinger + strtod path unchanged), then Clinger, then strtof. from_chars is
used only when it consumes the entire token; a partial parse means a non-'.'
locale decimal point, and an under-/overflow (result_out_of_range) also
declines - in both cases the existing strtod fallback supplies the exact
value and the well-defined +/-inf/0 the parser expects, side-stepping the
P4168 divergence between implementations. float and long double now get the
fast path too (Clinger was double-only).
Measured, C++17, g++ 13 -O3, json::parse/accept:
- canada-style floats: ~unchanged (Clinger already covered them)
- high-precision (17 digits): parse 2.1x, accept 2.5x
- scientific (17 digits + exp): parse 3.6x, accept 4.1x
Verified: C++11 (Clinger/strtod) and C++17 (from_chars) parse every value -
including subnormals, boundary values, and 1e9999/1e-9999 over-/underflow -
to bit-identical results; 2M number-fuzz clean; conversions/deserialization/
locale/number-fast-path suites pass in both C++11 and C++17; clang-tidy
clean; warning-clean on g++ and clang in C++11/17/20.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Guard from_chars use on JSON_HAS_CPP_17, not just __cpp_lib_to_chars
libstdc++ 15 defines __cpp_lib_to_chars even in C++14 mode (via
bits/version.h pulled in by other headers), but <charconv> is only included
under JSON_HAS_CPP_17. That made parse_float_from_chars() reference
std::from_chars without the header in C++14 builds, breaking gcc-latest,
icpx, and the offline-testdata jobs.
Gate the use on JSON_HAS_CPP_17 && __cpp_lib_to_chars so it matches the
include condition exactly; C++11/14 always take the scalar fallback.
Verified by forcing __cpp_lib_to_chars in a C++14 build: the guard
suppresses std::from_chars and it compiles. C++17 behavior is unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Disable Clinger float fast path under extended FP precision (x87)
The contiguous number fast path uses a Clinger-style exact algorithm
(significand * 10^scale in double arithmetic), which is only correctly
rounded when double operations are evaluated in true 53-bit precision.
On the x87 FPU used by 32-bit x86 (FLT_EVAL_METHOD == 2) the single
multiply/divide is computed in 80-bit and then double-rounded to double,
so a small fraction of values land 1 ULP off.
This surfaced as test-cbor_cpp11 and test-msgpack_cpp11 failing on the
mingw (x86) job for regression/floats.json: the C++17 builds pass because
they take the correctly-rounded std::from_chars path, while C++11 falls
back to parse_float_fast(). A 5M-sample check over shortest round-trip
decimals reproduces it: 0 divergences with 53-bit doubles, ~1 in 25 000
with 80-bit intermediates; declining to std::strtod fixes all of them.
Guard parse_float_fast() on FLT_EVAL_METHOD so it declines whenever the
platform evaluates doubles in extended precision, letting the caller use
the correctly-rounded std::from_chars / std::strtod path instead. On
mainstream x86-64/ARM64 (FLT_EVAL_METHOD == 0) the fast path is unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Guard <charconv> include with __has_include for GCC 7
GCC 7 sets __cplusplus to the C++17 value under -std=gnu++1z, so
JSON_HAS_CPP_17 is defined, but its libstdc++ ships no <charconv> header
(added in GCC 8; floating-point from_chars in GCC 11). The unconditional
"#if defined(JSON_HAS_CPP_17) #include <charconv>" therefore failed to
compile there: "fatal error: charconv: No such file or directory" in the
ci_test_compilers_gcc (7) job.
Wrap the include in __has_include(<charconv>), mirroring the library's
existing handling of <version> and <filesystem> in macro_scope.hpp. When
the header is absent, __cpp_lib_to_chars stays undefined and
parse_float_from_chars() takes its scalar fallback, so the from_chars use
site (already gated on __cpp_lib_to_chars) is never reached. GCC 8-10,
which have <charconv> but no floating-point from_chars, are unaffected:
they include the header but still take the fallback. GCC 11+ is unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Restore the column when ungetting a newline
The contiguous number fast path never reads the character that terminates
a number token, while scan_number() reads it and then ungets it. When that
character is a newline, get() has already cleared chars_read_current_line,
and unget() could only restore lines_read - leaving the column at 0. The
two paths therefore reported different columns for the same document:
json::parse("[01\n]") -> line 1, column 3
json::parse(stringstream) -> line 1, column 0
Remember the column the newline was read at so unget() can restore it.
Both paths now report the position the offending token actually starts at,
which also fixes the pre-existing column-0 artifact for streaming input.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Extend the bulk scan fast paths to sized sentinels
supports_bulk_scan required IteratorType and SentinelType to be the same
type, which excluded std::counted_iterator paired with std::default_sentinel_t
- the combination #5268 had already enabled for the memcpy fast path. Such
input fell back to the byte-at-a-time scanner even though it is contiguous
and its remaining length is computable in O(1).
Factor the "distance is computable in O(1)" test into sentinel_is_sized and
use it for iterator_is_contiguous, supports_seek, and supports_bulk_scan
alike, and share the std::ranges::distance/std::distance dispatch through a
remaining_count() helper.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Document JSON_USE_SIMDUTF on the macro overview page
The macro was only listed in the API macro index; add it to the supported
macros overview alongside the other JSON_USE_* macros, and note that it
selects between two definitions of the same inline function and so must be
defined identically in every translation unit.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Amalgamate source code
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Cover the counted-iterator bulk scan paths
Sized sentinels newly reach the bulk string/number scanners and the
seek-based token reconstruction, so exercise both:
- diagnostics that quote the offending token, which are rebuilt from the
consumed input via copy_consumed_range()
- inputs whose count ends before the underlying buffer does, including a
closing quote that exists only behind the count, a cut inside an 8-byte
SWAR stride, and a cut inside a UTF-8 sequence
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix the SentinelType example on iterator_input_adapter
The comment offered "a C++20 sentinel or counted_iterator" as examples of a
SentinelType, but std::counted_iterator is the IteratorType - the sentinel it
pairs with is std::default_sentinel_t. #5268 corrected the same wording in the
API documentation and left the code comment behind.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Do not discard the parse result in the error position check
json::parse is declared warn_unused_result, and CHECK_THROWS_WITH_AS
evaluates its expression as a discarded statement, so the assertion broke
the -Werror builds (GCC -Werror=unused-result, MSVC C4834 under /WX).
Compare against the helper that already captures the message instead.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Lock the two number grammars together with a parity test
The JSON number grammar is encoded twice: as the scan_number() state machine
and as the contiguous fast path. The fast path declining on anything it does
not recognize keeps most divergence harmless, but if it ever accepted
something the state machine rejects the result would be a silent correctness
bug, and the existing test only pinned a hand-written list of numbers.
Enumerate every string of length 1..4 over "01.eE+-" (2800 tokens) and
require both paths to agree on the parsed value and on the exact error
message. Verified to fail if the fast path's grammar is perturbed.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Skip token_buffer for integers on the contiguous path
An integer token does not need token_buffer: the number_integer and
number_unsigned SAX callbacks take only the value, and the overflow
diagnostic rebuilds the text from the input via get_token_string(). Convert
straight from the input buffer and materialize the token only for the
floating-point tail, which still needs a NUL-terminated buffer for strtod.
JSON_DIAGNOSTIC_POSITIONS derives a number's start position from
get_string().size(), so the copy is kept when that is enabled.
The integer dispatch is factored into convert_integer() and shared with
convert_number(), so both scanners keep using one implementation.
Integer-heavy input, 400k values, -O3:
parse accept
gcc 16 +14% +23%
clang +15% +22%
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Amalgamate source code
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Cover the bulk string and UTF-8 scanners
These paths had no dedicated tests and rested on differential fuzzing only.
Add three sections, all comparing the contiguous scanner against the
byte-at-a-time one on the parsed value and on the exact error message:
- every string of length 1..3 over an alphabet of ordinary ASCII, both
specials, a control byte, escape characters, UTF-8 lead and continuation
bytes, and a byte that is never valid - each at offset 0 and offset 9, so
the bulk scanner sees them with and without a run behind them
- every kind of run-ending byte at each offset across two 8-byte SWAR words,
so multibyte sequences also straddle the word boundary
- the boundaries of every range validate_one_utf8() recognizes: shortest and
longest encodings, overlongs, both ends of the surrogate block, U+10FFFF
and just past it, and truncated sequences
Verified to fail if the bulk validator accepts surrogates, and if the SWAR
word test stops detecting control characters.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Format the new string fast path test with astyle
The pinned astyle expands a braced-init-list used as a range-for range onto
several lines; hoist the two offsets into a named vector instead, which reads
better and leaves nothing for astyle to reformat.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix shadowed locals and guard the exception-dependent tests
Two problems in the tests added for the bulk scanners, both found by CI:
- the inner `const json j` in the counted-iterator diagnostics shadowed the
one declared at test-case scope, which -Wshadow rejects on GCC and clang
and C4456 rejects on MSVC under /WX; rename them
- the new parity checks parse deliberately invalid input, which calls
std::abort() rather than throwing when JSON_NOEXCEPTION is defined, so
they would have crashed the no-exception build; guard them the way the
other tests do
json::accept() does not abort, so the UTF-8 range assertions stay compiled
without exceptions and keep covering validate_one_utf8() there.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Gate the C++20 iterator classification on JSON_HAS_RANGES
Making supports_bulk_scan depend on iterator_is_contiguous meant the trait
is now instantiated for every adapter, not only when get_elements() is
called. On standard libraries with an incomplete <ranges> that is fatal:
libstdc++ 10 evaluates std::contiguous_iterator<std::counted_iterator<T*>>
by calling std::to_address, which needs an operator-> its counted_iterator
does not have, so satisfaction checking is a hard error rather than false.
Reported by clang 14 + libstdc++ 10.
JSON_HAS_RANGES already encodes exactly this ("libstdc++ < 11 has incomplete
C++20 ranges", #4440), so require it for the C++20 branch. Affected
toolchains fall back to the pointer-only test and the byte-at-a-time
scanner, which parses identically, just without the bulk fast paths.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Amalgamate source code
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Satisfy clang-tidy in the new bulk scanner tests
- give the helper lambdas an explicit std::string return type and return
braced initializer lists (modernize-return-braced-init-list)
- replace the C-style array of test cases with a std::vector
(modernize-avoid-c-arrays)
- silence pro-type-member-init on the two brace-initialized aggregates;
default member initializers would stop them being aggregates in C++11
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Avoid escaped literals in the counted-iterator diagnostics list
clang-tidy reads "[\"\\ud834\"]" as a literal better written raw, and the
two literals written next to each other in "[\"a\x01""b\"]" as a missing
comma. The concatenation was there to stop the hex escape swallowing the
following character; build those documents from explicit bytes instead and
use raw strings elsewhere. The byte sequences are unchanged.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Reattach convert_number's documentation
@gregmarr spotted that convert_integer() was inserted between convert_number()
and its doc block, leaving convert_integer() with two stacked blocks and
convert_number() with none. Comment only; no code change.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address review comments on the simdutf backend
string_bulk_run() had the same `return scalar_string_bulk_run(...)` in both
arms of the `#if`. The simdutf arm already falls through when validation
fails, so a single return after the `#endif` says the same thing.
The JSON_USE_SIMDUTF example showed `#include <simdutf.h>`, which
string_scan.hpp already does under the same guard; users only have to put the
header on the include path and link the library, not include it themselves.
Set the version history entry to 3.13.0, matching the other macro pages
documenting unreleased features.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Pin the number error position against a non-newline terminator
The comment claimed the reported column is the one the offending token starts
at. It is the column reached after the token's last character - which is the
actual point of the unget() change: a number terminated by a newline now
reports what the same number terminated by a space always did.
Assert that equality directly, and add a multi-character token where the start
and end columns differ, so the invariant cannot be read off a single-character
example.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Do not repeat the integer conversion that just failed
scan_number_bulk_contiguous() converts an integer token straight from the input
buffer. When the value does not fit, it materializes token_buffer and calls
convert_number(), which tried the very same integer conversion again before
falling back to floating point.
Recording the outcome in number_type skips the second attempt. The resulting
token type and value are unchanged: convert_number() reached the float tail
either way.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Require a container's value_type to match what data() points at
is_contiguous_byte_container accepted any type with a data() returning a
pointer to a single-byte integral plus a size(). That is duck typing: the two
members say nothing about size() counting the units data() points at.
A type where it does not - fixed-size records, say - was routed to the
pointer-based adapter and parsed as [data(), data() + size()) bytes, silently
truncating input the iterator-based adapter had read in full:
struct record_buffer {
using value_type = std::array<char, 4>;
std::string bytes;
const char* data() const; // raw bytes
std::size_t size() const; // in records
const char* begin() const; const char* end() const;
};
json::parse(record_buffer{"[1,2,3,4,5]"}); // parse error at column 3
Requiring the container's own value_type to be that same element type ties the
two together. Every contiguous standard container satisfies it, so std::string,
std::vector<char>, std::array<char, N> and std::string_view keep the fast path;
anything else falls back to the iterator-based adapter, which is always correct.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Only compile the simdutf backend from C++17 on
simdutf.h rejects anything below C++17 with an #error, so defining
JSON_USE_SIMDUTF in a C++11 or C++14 translation unit did not fail with a
message about simdutf being unavailable - it failed to compile at all, taking
the library's C++11 support with it. Nothing caught this because no build ever
compiled that path.
Gate the include and both uses on JSON_HAS_CPP_17, the same way number_parse.hpp
gates std::from_chars. Below C++17 the macro now has no effect and the scalar
validator runs; it accepts and rejects exactly the same input, so the macro is
safe to set project-wide even when some translation units use an older standard.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Test the simdutf backend in CI
JSON_USE_SIMDUTF was documented and shipped but never compiled by anything in
the repository, so nothing held the backend to the behavior the docs promise.
Add JSON_TestSimdutf (OFF by default), which fetches simdutf and defines
JSON_USE_SIMDUTF for every test target, and a ci_test_simdutf target that runs
the whole suite in that configuration. Because simdutf needs C++17, the suite is
built at C++11 as well, so one job covers both the scalar fallback with the
macro defined and simdutf itself.
The dependency hangs off test_main, whose usage requirements every test target
inherits. The library target and the installed CMake package are deliberately
untouched: making nlohmann_json link simdutf would put a find_dependency() in
the exported package, which is a separate decision.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Warn when JSON_TestSimdutf cannot reach the simdutf backend
simdutf needs C++17: without it the dependency does not even compile, and with
a C++17 compiler but no C++17-or-later standard under test it builds and then
goes unused. Either way the option silently did nothing useful, or broke the
configure step outright.
Resolve the tested standards first, then check them: when none of them can
reach simdutf, skip the dependency and say so, naming which of the two reasons
applies and how to fix it. The tests then run against the scalar validator,
which is what would have happened anyway.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Use record_buffer::data() so clang does not flag it unneeded
The record_buffer test type declares data() and size() so the
is_contiguous_byte_container trait can see both and still reject the
type on its value_type. data() was never called, so clang's
-Wunneeded-member-function (under -Weverything -Werror) failed the
C++20 build. Assert that data() points at the underlying bytes: it
ODR-uses the member and documents the property the type is meant to
demonstrate.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Skip the float fast path when it cannot succeed
parse_float_fast() (Clinger) needs a significand below 2^53, so it always
declines once the mantissa has 17 or more significant digits. convert_number()
called it unconditionally, so those numbers were walked an extra time before
strtod had to run anyway. On streaming input, where scanning is byte-at-a-time
and there is no compensating win, that made canada.json about 6% slower than
develop.
Derive the significant-digit count from token_buffer indices - the digits are
not scanned again - and skip the call when it is guaranteed to decline. Both
scanners pass the offset where the mantissa ends; the count only has to be
corrected for a leading "0", which the JSON grammar admits nowhere else. The
integer path returns before the check, so integer-heavy input is unaffected.
Values are unchanged: this only avoids an attempt that would have failed.
Verified bit-exact against develop over every number in canada.json,
floats.json, signed_ints.json, unsigned_ints.json, small_signed_ints.json,
citm_catalog.json and twitter.json, for both the contiguous and the streaming
scanner.
parse, streaming develop before after
canada.json 19.4ms 20.5ms 19.3ms
floats.json 135.9ms 131.8ms 128.0ms
parse, contiguous develop before after
canada.json 15.5ms 12.9ms 11.7ms
floats.json 98.6ms 69.8ms 66.7ms
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The binary entry points end with
return res ? result : basic_json(value_t::discarded);
The condition operator's second operand is an lvalue, so this is not a case
where the return value can be elided or implicitly moved from: every
successful from_cbor(), from_msgpack(), from_ubjson(), from_bjdata() and
from_bson() call deep-copies the value it just parsed, and then destroys the
original.
The copy is not cheap, and it is not incidental: basic_json's copy
constructor walks the whole value. Parsing a 2 MB CBOR document with 60,000
objects, median of 25 runs, clang 17 -O3:
from_cbor 26.99 ms -> 14.65 ms
from_msgpack 26.82 ms -> 14.82 ms
Moving instead of copying is the entire change; the parsed value is not used
again after the return expression is evaluated.
There is a second reason to prefer the move. The copy constructor recurses
once per nesting level, so the copy is also a stack-overflow path on the
return side, on a value the reader has already accepted. That is currently
masked because the readers themselves recurse and overflow first (#5104), but
it has to be fixed for making them iterative to have any effect.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Reserve capacity in from_json() object conversion when supported
The object-to-container from_json() overload filled the target
container one element at a time without reserving capacity, even
when the target type supports reserve() (e.g. std::unordered_map)
and the number of elements is already known. This caused unnecessary
rehashing while parsing large objects into such containers.
Add a reserve-detecting overload (from_json_object_impl), mirroring
the priority_tag-based SFINAE technique already used by the array
conversion path (from_json_array_impl), so that reserve(size()) is
called up front when available and the loop falls back unchanged
otherwise (e.g. for std::map).
Fixes#5406
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Update doc example outputs for new object-conversion iteration order
Reserving capacity in from_json()'s object-conversion path before
inserting elements changes libstdc++'s std::unordered_map bucket
layout, which changes the iteration order used by
get__ValueType_const.cpp, get_to.cpp and operator__ValueType.cpp to
print the elements of a converted std::unordered_map<std::string,
json>. Verified against a clean develop checkout (built with the
same GCC/libstdc++ used in CI) that the old order was produced
without this PR's change and the new order is produced with it, and
that the three affected examples now match their updated expected
output byte-for-byte.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Factor out a reserve-dispatch helper instead of duplicating the object from_json loop
Addresses review feedback from @gregmarr on PR #5472: the emplace loop no
longer needs to exist twice for the reserve/no-reserve cases. A small
from_json_object_reserve() overload pair (SFINAE-dispatched on whether
reserve() exists, mirroring the priority_tag technique used elsewhere)
either calls reserve() or is a no-op; from_json_object_impl() calls it once.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Inline from_json_object_impl into from_json now that it is called only once
Addresses review feedback from @gregmarr on PR #5472: with the reserve
loop de-duplicated, from_json_object_impl no longer needs to be a
separate function that from_json immediately delegates to.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
contains(const json_pointer&) is documented to never throw, but a purely
numeric reference token that is syntactically a valid array index yet
numerically too large to be represented (exceeding size_type's max, or
exceeding ULLONG_MAX and causing strtoull() to set errno to ERANGE) made
it fall through to array_index(), which throws out_of_range.410/404.
Pre-check the token's magnitude the same way array_index() does, but
return false instead of throwing, mirroring how the surrounding code
already rejects other malformed tokens (leading zero, non-digit
characters, "-") without throwing. operator[]/at() are untouched and
keep throwing for these inputs.
Fixes#5395
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Extract C++17-only content from unit-items.cpp into its own test file
unit-items.cpp is a 1433-line file that was being compiled twice per
CI configuration (once for C++11, once for C++17) purely because it
contained a single, small JSON_HAS_CPP_17-gated SECTION ("structured
bindings", 14 lines). Move that SECTION into a new, dedicated file
(tests/src/unit-items-cpp17.cpp) so only that tiny file needs a
second build; unit-items.cpp itself now builds/tests only once. No
tests/CMakeLists.txt changes are needed since the existing
file(GLOB ... src/unit-*.cpp) plus json_test_add_test_for() already
auto-register and standard-gate any new unit-*.cpp file based on
whether it textually contains JSON_HAS_CPP_<N> (the same mechanism
already used for the existing unit-iterators3.cpp file, which follows
the identical pattern).
Verified with plain clang++ under -std=c++11/14/17/20 and via a local
CMake configure+build that:
- unit-items.cpp now only produces a test-items_cpp11 target (the
former test-items_cpp17 target is gone) and its assertion/test-case
counts are unchanged (2 test cases / 222 assertions) for every
standard.
- The new unit-items-cpp17.cpp produces test-items-cpp17_cpp11 (an
intentionally empty translation unit under C++11 that reports 0
tests, 0 assertions, SUCCESS) and test-items-cpp17_cpp17 (1 test
case / 1 assertion, identical to what "structured bindings" ran
as before it was moved).
Separately, unit-regression1.cpp (1530 lines) was also being built
twice per CI configuration because it contained the substring
JSON_HAS_CPP_17 -- but on inspection this was dead code: an orphaned
"#ifdef JSON_HAS_CPP_17 / #include <variant> / #endif" left over from
when the actual std::variant-based regression test (issue #1292) was
relocated to unit-regression2.cpp. Nothing in unit-regression1.cpp
uses <variant>, so there is no SECTION/TEST_CASE to preserve here;
the dead include is simply removed. This was verified by grepping the
file for any other use of "variant" (none) and confirming issue #1292
is still covered by unit-regression2.cpp. Compiled and ran under
-std=c++11/14/17/20 and via CMake: unit-regression1.cpp now only
produces a test-regression1_cpp11 target (test-regression1_cpp17 is
gone) with an unchanged test-case count (3) under every standard.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix astyle indentation of #include inside #ifdef in unit-items-cpp17.cpp
This repo's astyle style keeps preprocessor directives at column 0
even inside #ifdef blocks. The new tests/src/unit-items-cpp17.cpp
had its #include <map>/#include <string> indented, which made the
'check' CI job's amalgamation/formatting diff non-empty and failed
the aggregate check.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Add test coverage for documented lenient BSON input handling
Issue #5333 documented three intentionally-lenient behaviors of the BSON
reader (any non-zero byte accepted as a boolean `true`, BSON array element
keys not validated against the required decimal sequence, and the payload
of binary subtype 0x02 "old binary" returned as-is including its inner
length prefix), but none of them was pinned by a test, so a future change
could silently regress the documented behavior.
Also add coverage for the out_of_range.412 length-overflow check
(shared by binary, string, and (sub-)document BSON length fields) for
the string and document cases; only the binary case was previously
tested.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix 32-bit overflow in huge_string_t BSON length-overflow tests
huge_string_t doubles as basic_json's StringType, so it is used not only
for the JSON string value under test but also for object keys (e.g. "s",
"nested"). Making size() unconditionally lie about being huge therefore
inflated the keys' reported sizes as well, pushing the running totals
computed while walking the BSON document (calc_bson_object_size and
friends in binary_writer.hpp) past what a 32-bit std::size_t can hold.
On 64-bit platforms this happens to still produce a working (if
needlessly large) result, but on 32-bit platforms (e.g. the mingw x86 CI
job) the size_t arithmetic silently wraps around: for the "document" test
this merely surfaces the wrong number in the exception message, but for
the "string" test the wrapped total happens to fall back under
INT32_MAX, so the intended out_of_range.412 guard is skipped entirely and
the code goes on to actually write ~2 GiB worth of characters from the
key's real, tiny buffer - which is what raised the reported
"vector::_M_range_insert" exception instead of a controlled 412.
Make the fake-huge size opt-in via huge_string_t::as_huge() and only
apply it to the string value under test, leaving keys at their real
(small) size. This keeps every intermediate size well within 32-bit
size_t range on any platform, matching how huge_binary_t already avoids
the same trap (it is only ever used as the BSON value type, never as a
key). Expected out_of_range.412 messages are updated accordingly.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Broaden JSON_HEDLEY_WARN_UNUSED_RESULT coverage to pure query functions
Add JSON_HEDLEY_WARN_UNUSED_RESULT to the unambiguous, const,
side-effect-free observer functions whose return value is the entire
purpose of the call:
- dump()
- type(), type_name()
- all is_* predicates (is_primitive, is_structured, is_null,
is_boolean, is_number, is_number_integer, is_number_unsigned,
is_number_float, is_object, is_array, is_string, is_binary,
is_discarded)
- empty(), size(), max_size()
- count(...) (both overloads) and contains(...) (all overloads,
including the deprecated json_pointer<BasicJsonType> overload)
This mirrors the direction the standard library has taken with
[[nodiscard]] on the analogous std::vector/std::map members, and
catches real bugs such as `j.empty();` (meant `j.clear();`) or
`j.contains(k);` with the result thrown away.
Deliberately out of scope (left for a separate, later policy
decision, per the issue): at(), value(), get*(), flatten(),
unflatten(), patch(), merge_patch(), begin()/end(), comparison
operators, erase(), and emplace().
Compiling the full test suite (tests/src/unit-*.cpp) with
-Wunused-result -Werror uncovered one real hit: a regression test in
unit-regression2.cpp called dump() purely to check it does not throw,
discarding the result. Fixed by explicitly casting to void, since the
call is intentionally result-less there.
Fixes#5410
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix discarded nodiscard results across the test suite for GCC's warn_unused_result
A plain (void) cast on a call expression suppresses the C++17 [[nodiscard]]
warning but not GCC's warning for functions annotated via the GNU
__attribute__((warn_unused_result)) form -- which is what
JSON_HEDLEY_WARN_UNUSED_RESULT expands to on GCC. Several existing tests
that call a newly-annotated function (dump(), empty()) purely to check
that it throws/does not throw, discarding the result via (void), newly
warned (and failed -Werror builds) once the annotation was broadened.
Route those discards through a small ignore_return_value() helper
instead, which actually consumes the value and suppresses the warning
on both attribute forms.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Use utils::ignore_return_value() for the issue #1445 dump() discard too
Addresses review feedback from @gregmarr on PR #5477: this call site was
still using the older "capture in a variable, then (void) it" pattern
from before this PR introduced utils::ignore_return_value(), instead of
the helper now used at every other discarded-nodiscard-result call site
this PR touches.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
accept() is a pure query whose only effect is the returned bool; both
parse() overloads and the deprecated accept(span_input_adapter&&, ...)
overload already carry JSON_HEDLEY_WARN_UNUSED_RESULT, but the two
current, recommended accept() overloads were missing it. Add the
annotation to match, so discarding accept()'s result now warns under
-Wunused-result / [[nodiscard]], as it already does for parse().
Fixes#5407
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Reject JSON Patch move when from is a proper prefix of path
RFC 6902 (section 4.4) forbids "from" from being a proper prefix of
"path" for a "move" operation: "a location cannot be moved into one
of its children." "move" is implemented as remove-then-add with no
check for this. For object targets, the subsequent "add" happened to
throw as a side effect of resolving through the now-removed parent,
but for array targets, removing the "from" element shifts subsequent
indices, so "path" silently re-resolves to a different element and
the operation "succeeds" with a silently corrupted document.
Add a check, before performing the remove/add, for whether "from" is
a proper prefix of "path" at the reference-token level. This compares
json_pointer's already-unescaped reference_tokens vectors (basic_json
is a friend of json_pointer) rather than the raw pointer strings, so
that tokens containing escaped '/' or '~' characters are compared
correctly, and a token that merely looks like a string prefix (e.g.
"/ab" vs "/abc/x") is not mistaken for a pointer-token prefix. When
"from" is a proper prefix of "path", throw out_of_range.414.
Fixes#5397.
Stacked on top of the fix for #5396 (branch
issue-5396-patch-remove-primitive-parent), since both touch the same
patch_inplace move/remove handling in include/nlohmann/json.hpp.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Add root-pointer and array-append-token edge case tests for the move prefix check
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Replace std::equal with an explicitly-bounded loop in the move prefix check
The three-iterator std::equal(first1, last1, first2) form has no
explicit end iterator for the second range, which a static analyzer
(Flawfinder, CWE-126) flags as a potential over-read even though the
preceding size comparison already guarantees the second range is long
enough. Rather than argue the point, make the bound visible in the code
itself via an explicit loop -- every access to ptr.reference_tokens is
now guarded by the same index the loop condition bounds against
from_size.
(The C++14 four-iterator std::equal(first1, last1, first2, last2) form
was tried first as a more minimal fix, but this codebase targets C++11
and that overload is not safely usable under -std=c++11 with all
supported standard library implementations.)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Extract the move prefix check into a named helper lambda
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Account for JSON_DIAGNOSTIC_POSITIONS in the move-prefix-check error messages
out_of_range::create() includes a "(bytes X-Y)" position annotation when
JSON_DIAGNOSTIC_POSITIONS is enabled, which the ci_test_diagnostic_positions
CI job builds the whole suite with. The five new out_of_range.414
assertions only checked the annotation-free message. Confirmed
JSON_DIAGNOSTICS produces the same (annotation-free) message as the
default build for this particular throw site (its path-based annotation
is empty at the root, where &result always points here), so only two
message variants are needed, not three.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
RFC 6902 (section 4.2) requires the target location of a "remove"
operation to exist. operation_remove handled parent.is_object() and
parent.is_array(), but had no final else branch: when the resolved
parent was a primitive value or null, neither branch matched and the
operation silently did nothing instead of failing.
Add the missing else branch, throwing out_of_range.413 with wording
that matches the existing out_of_range.411 thrown by the analogous
"add" case (operation_add) for the same kind of invalid parent.
Fixes#5396.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Route hand-rolled diagnostic pragmas through Hedley
Several places in the library hand-roll compiler diagnostic suppression
with raw `#pragma`/`#ifdef __GNUC__`/`#ifdef __clang__` guards instead of
using the Hedley primitives already bundled and used elsewhere
(JSON_HEDLEY_DIAGNOSTIC_PUSH/POP, JSON_HEDLEY_PRAGMA, ...). Converted six
of the seven listed push/pop pairs to use those primitives instead of
raw `#pragma GCC diagnostic`/`#pragma clang diagnostic` text:
- include/nlohmann/json.hpp (~3770, ~3863): -Wfloat-equal
- include/nlohmann/detail/conversions/to_chars.hpp (~1078): -Wfloat-equal
- include/nlohmann/detail/output/binary_writer.hpp (~1844): -Wfloat-equal
- include/nlohmann/detail/iterators/iteration_proxy.hpp (~211): -Wmismatched-tags
- include/nlohmann/detail/exceptions.hpp (~36): -Wweak-vtables
iteration_proxy.hpp did not previously include macro_scope.hpp itself
(it only compiled because some other header included earlier in
json.hpp happened to pull macro_scope.hpp in first); it now includes it
directly like the other detail headers that use Hedley macros, so it is
self-contained.
Each push/pop pair now uses JSON_HEDLEY_DIAGNOSTIC_PUSH/POP
unconditionally (a no-op on compilers that don't need it) and wraps the
actual `#pragma ... diagnostic ignored` text in JSON_HEDLEY_PRAGMA so it
goes through Hedley's _Pragma()-based emission instead of a raw #pragma
line, while keeping the original `#ifdef __GNUC__` / `#if
defined(__clang__)` guard around the ignored-pragma itself.
Deviation from the issue's suggested transformation: the issue's example
replaces the `#ifdef __GNUC__` guard with `#if
JSON_HEDLEY_HAS_WARNING("-Wfloat-equal")`. JSON_HEDLEY_HAS_WARNING is
implemented purely via Clang's `__has_warning` builtin and evaluates to
0 on real GCC (`#define JSON_HEDLEY_HAS_WARNING(warning) (0)` when
`__has_warning` is not defined), so adopting it verbatim would silently
stop suppressing -Wfloat-equal on GCC -- a real regression, not just a
style change. The existing `#ifdef __GNUC__` / `#if defined(__clang__)`
guards were kept for the ignored-pragma to stay behavior-preserving, and
only the push/pop/pragma-emission mechanism was routed through Hedley.
Two of the seven locations from the issue (the -Wignored-attributes
push at the very top of json.hpp and its matching pop after
`#include <nlohmann/detail/macro_unscope.hpp>`) were intentionally left
unconverted:
- The push, at the very top of json.hpp, runs before
`detail/macro_scope.hpp` (and therefore hedley.hpp) has been included
anywhere in the translation unit, so JSON_HEDLEY_DIAGNOSTIC_PUSH is not
yet defined at that point.
- The pop runs after `macro_unscope.hpp`, which -- via hedley_undef.hpp
-- has already #undef'd every JSON_HEDLEY_* macro (by design, see
#5408) precisely so they don't leak to users, so JSON_HEDLEY_DIAGNOSTIC_POP
is no longer defined by the time the pop is reached either.
Making this one pair work would require either hoisting the ~2000
line vendored hedley.hpp to the very top of the amalgamated single
header (a much bigger structural change to single_include than a pure
mechanism swap) or special-casing this one pop ahead of the general
macro cleanup. Both are riskier than the mechanical, behavior-preserving
change requested, so this pair was left as-is.
## Validation
- Compiled include/nlohmann/json.hpp and single_include/nlohmann/json.hpp
with `-Wall -Wextra -Wfloat-equal -Wmismatched-tags -Wweak-vtables`
(clang, which self-identifies as __GNUC__ too): no warnings, same as
before the change.
- Compiled and ran tests/src/unit-to_chars.cpp, unit-conversions.cpp,
unit-iterators1.cpp, unit-iterators2.cpp, and unit-class_parser.cpp
against the fixed include/: all pass.
- Compiled unit-msgpack.cpp, unit-bjdata.cpp, and unit-ubjson.cpp (which
exercise binary_writer.hpp's write_compact_float extensively): all
compile cleanly; the vast majority of assertions pass (the only
failures are pre-existing environment issues unrelated to this change
-- missing generated test-data files, not code correctness).
- Ran `make amalgamate`; the single_include diff is limited to exactly
the lines touched in include/, with no unrelated reordering.
- No real (non-Apple) GCC was available in this environment to test
directly; the `_Pragma("GCC diagnostic ...")` text emitted by
JSON_HEDLEY_PRAGMA is byte-identical to the prior `#pragma GCC
diagnostic ...` text, and the `#ifdef __GNUC__` guard is unchanged, so
GCC's behavior is expected to be identical. CI covers the GCC matrix.
This PR is stacked on top of #5475 (issue-5408-hedley-undef-leak) since
both touch the same files; only the last commit here is new.
Fixes#5409.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Guard JSON_HEDLEY_DIAGNOSTIC_PUSH/POP with the same compiler check as the pragma they bracket
Addresses review feedback from @gregmarr on PR #5485: the push/pop calls
were unconditional, so compilers other than the one the ignored-pragma
targets (e.g. MSVC, or GCC where the pair only applies under __clang__)
now did a needless push/pop with nothing suppressed in between. Move the
existing #ifdef __GNUC__ / #if defined(__clang__) guard to also cover the
push/pop, restoring the original zero-overhead behavior on other compilers
while still emitting the pragma itself through Hedley.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Undefine the four JSON_HEDLEY_* macros that leak after including json.hpp
include/nlohmann/detail/macro_unscope.hpp includes hedley_undef.hpp to
#undef every JSON_HEDLEY_* macro so none of them leak into the including
translation unit. Four macros were missing from that list and therefore
stayed defined after #include <nlohmann/json.hpp>:
- JSON_HEDLEY_PRAGMA
- JSON_HEDLEY_PREDICT_TRUE
- JSON_HEDLEY_PREDICT_FALSE
- JSON_HEDLEY_CLANG_HAS_DECLSPEC_ATTRIBUTE
hedley_undef.hpp is generated (via `make update_hedley`) by grepping
hedley.hpp for its own internal `#undef JSON_HEDLEY_X` redefinition
guards. JSON_HEDLEY_PRAGMA/PREDICT_TRUE/PREDICT_FALSE have no such guard
in upstream Hedley, so they were never picked up. The guard for
JSON_HEDLEY_CLANG_HAS_DECLSPEC_ATTRIBUTE also has an upstream typo
(`JSON_HEDLEY_CLANG_HAS_DECLSPEC_DECLSPEC_ATTRIBUTE`), so hedley_undef.hpp
was undefining the wrong (never-defined) name.
Fixes:
- include/nlohmann/thirdparty/hedley/hedley_undef.hpp: corrected the
DECLSPEC_ATTRIBUTE typo and added the three missing #undef lines,
keeping the file's alphabetical ordering.
- Makefile (update_hedley target): changed hedley_undef.hpp generation to
extract macro names directly from every `#define JSON_HEDLEY_...` in
hedley.hpp instead of from existing `#undef` guards, so a future
`make update_hedley` run undefines every macro Hedley actually defines,
even ones without a pre-existing redefinition guard. This was not run
in this PR (it would also pull in an unrelated upstream Hedley sync);
hedley_undef.hpp was hand-patched instead and single_include was
regenerated with `make amalgamate`.
- tests/src/unit-no-macro-leak.cpp: new regression test (picked up
automatically by tests/CMakeLists.txt's existing unit-*.cpp glob) that
includes json.hpp and then #ifdef/#error-checks every JSON_HEDLEY_*
macro name, so any future leak of any of the 151 vendored macros fails
the build, not just the four fixed here.
Fixes#5408.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Derive the JSON_HEDLEY_* leak-check test from hedley.hpp at build time
tests/src/unit-no-macro-leak.cpp previously hardcoded a static list of
~151 #ifdef/#error checks, one per JSON_HEDLEY_* macro name known at the
time it was written. That list would silently go stale the next time
`make update_hedley` pulls in a vendor update that adds, removes, or
renames a macro, since nothing would force it to be regenerated.
Add cmake/scripts/gen_hedley_undef_check.cmake, which derives the full
list of JSON_HEDLEY_* macro names directly from
include/nlohmann/thirdparty/hedley/hedley.hpp:
- tests/CMakeLists.txt uses it (MODE=checks) to (re)generate
hedley_undef_checks.inc at configure and build time, and wires the
generating custom target as a dependency of the test-no-macro-leak_cpp*
targets so it can never build against a stale copy. unit-no-macro-leak.cpp
now just #include-s the generated file inside its TEST_CASE instead of
carrying the checks itself.
- The Makefile's `update_hedley` target now delegates hedley_undef.hpp
generation to the same script (MODE=undef, new `update_hedley_undef`
target), so the vendored header, the generated #undef list, and the
generated test checks are all derived from the same extraction logic and
cannot drift apart.
This mirrors the approach taken independently in #5415 for the same
issue (#5408), credited there to a self-regenerating mechanism that
"can never drift again" -- ported into this branch instead of the
static list originally proposed here.
Verified with a local CMake configure + build + ctest, both against
include/ (JSON_MultipleHeaders=ON) and against the amalgamated
single_include/nlohmann/json.hpp (JSON_MultipleHeaders=OFF), and by
temporarily deleting a #undef line from hedley_undef.hpp to confirm the
generated test actually fails on a real leak.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix REUSE compliance failure in gen_hedley_undef_check.cmake
The generated file's embedded banner contains the literal text
'SPDX-License-Identifier: MIT' as part of the *content* being written
to hedley_undef.hpp, not as this .cmake script's own REUSE header (it
is already covered by the blanket 'Files: *' rule in .reuse/dep5).
The reuse tool matched that embedded line as an SPDX tag for the
script itself and failed to parse the trailing 'MIT\n")' as a valid
SPDX License Expression, breaking ci_reuse_compliance. Wrap the
embedded banner in REUSE-IgnoreStart/REUSE-IgnoreEnd comments, as
recommended by the tool's own diagnostic output.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Speed up whitespace skipping in the lexer
lexer::skip_whitespace() called get() for every whitespace byte, and
get() checks the (almost always false, once past the first character)
next_unget flag on every call. skip_whitespace() now reads its first
character with get() (needed to honor a pending unget() left over from
finishing the previous token, e.g. scan_number() always ungets the
character that terminated the number) and every further whitespace
character with a new get_ignoring_pending_unget() variant that skips
that branch, since nothing in the loop calls unget().
This is a narrower fix than the full contiguous-buffer bulk-skip
suggested in the issue (scan a run of whitespace directly in the
adapter's buffer and update position counters once per run). That
approach depends on bulk-scan adapter infrastructure
(supports_bulk_scan/bulk_data()/bulk_skip()) introduced by the open,
unmerged parser-performance PR #5283, which this change intentionally
does not depend on or replicate. Building new bulk-scan adapter
infrastructure from scratch was judged out of scope/riskier than
warranted here, so this change is limited to the safe, always-correct
improvement of removing redundant per-character bookkeeping from the
existing byte-at-a-time loop; full bulk-skipping is left as future
work once #5283 (or equivalent adapter support) lands.
Line/column/byte-offset bookkeeping is untouched and verified
bit-for-bit identical before and after this change, including for
pretty-printed (dump(4)) input with embedded newlines.
Fixes#5412
Stacked on top of the PR for #5411 (branch
issue-5411-lexer-skip-conversion).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix codegen regression in skip_whitespace() from #5490
Benchmarking found the get()/get_ignoring_pending_unget() split in
skip_whitespace() made long whitespace runs (e.g. indentation in
pretty-printed JSON) 1.75x-3.2x SLOWER instead of faster, reproducible
with both Apple Clang and GCC.
Root cause: rewriting the loop from a plain do-while into an initial
get() followed by a while-loop defeated the compiler's ability to keep
the input adapter's read/end pointers in registers across iterations;
both compilers instead reloaded them from memory on every character.
The function split itself was not the problem (it still fully
inlines); the loop's control-flow shape was.
The fix keeps the same two-function structure but restores a
do-while shape (guarded by an if for the "first char not whitespace"
case), which lets both compilers hoist the pointers back into
registers, matching or beating pre-#5490 performance.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Share the position-counter bump between get() and get_ignoring_pending_unget()
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Extract current_is_whitespace() to deduplicate skip_whitespace()'s two whitespace checks
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Use a raw string literal for the multi-line error-position test input
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix clang-tidy raw-string-literal finding and guard a new test against JSON_NOEXCEPTION
The issue #5412 whitespace-skipping test added a check_error() helper that
relies on catching json::parse_error to verify the exception message; under
JSON_NOEXCEPTION, JSON_THROW aborts instead of throwing, which crashed
ci_test_noexceptions (and cascaded into the other ci_cmake_options jobs).
Guard the whole section with #if !defined(JSON_NOEXCEPTION), matching the
existing pattern used by sibling tests in this file.
Also switch one escaped string literal to a raw string literal to satisfy
clang-tidy's modernize-raw-string-literal check.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Skip integer conversion in accept()/SAX validation when the value is unused
lexer::scan_number() always converted every numeric token with
strtoull()/strtoll() before returning, even though accept() (and any
consumer using json_sax_acceptor) immediately discards the converted
value. For value_unsigned/value_integer tokens whose digit count
already guarantees the value fits into 64 bits, the conversion cannot
change the accept/reject decision (such tokens are always finite and
unconditionally accepted), so scan_number() can skip strtoull()/
strtoll() entirely in that case when the caller signals it does not
need the value. Numbers with more digits keep using the exact,
unmodified conversion path, so overflow reclassification to
value_float (and the finiteness check on it) is unaffected.
parse() and value_float handling are completely unchanged.
Fixes#5411
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Document why the digit-count fast path is safe regardless of number_unsigned_t/number_integer_t width
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Avoid temporary-string concatenation flagged by clang-tidy in the differential test
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
@@ -1421,7 +1421,7 @@ I deeply appreciate the help of the following people.
6. [Joshua C. Randall](https://github.com/jrandall) fixed a bug in the floating-point serialization.
7. [Aaron Burghardt](https://github.com/aburgh) implemented code to parse streams incrementally. Furthermore, he greatly improved the parser class by allowing the definition of a filter function to discard undesired elements while parsing.
8. [Daniel Kopeček](https://github.com/dkopecek) fixed a bug in the compilation with GCC 5.0.
9. [Florian Weber](https://github.com/Florianjw) fixed a bug in and improved the performance of the comparison operators.
9. [Fiona Johanna Weber](https://github.com/Fiona-J-W) fixed a bug in and improved the performance of the comparison operators.
10. [Eric Cornelius](https://github.com/EricMCornelius) pointed out a bug in the handling with NaN and infinity values. He also improved the performance of the string escaping.
11. [易思龙](https://github.com/likebeta) implemented a conversion from anonymous enums.
12. [kepkin](https://github.com/kepkin) patiently pushed forward the support for Microsoft Visual Studio.
@@ -1523,14 +1523,14 @@ I deeply appreciate the help of the following people.
108. [Kevin Tonon](https://github.com/ktonon) overworked the C++11 compiler checks in CMake.
109. [Axel Huebl](https://github.com/ax3l) simplified a CMake check and added support for the [Spack package manager](https://spack.io).
110. [Carlos O'Ryan](https://github.com/coryan) fixed a typo.
111. [James Upjohn](https://github.com/jammehcow) fixed a version number in the compilers section.
111. [James Upjohn](https://github.com/jupjohn) fixed a version number in the compilers section.
112. [Chuck Atkins](https://github.com/chuckatkins) adjusted the CMake files to the CMake packaging guidelines and provided documentation for the CMake integration.
113. [Jan Schöppach](https://github.com/dns13) fixed a typo.
114. [martin-mfg](https://github.com/martin-mfg) fixed a typo.
115. [Matthias Möller](https://github.com/TinyTinni) removed the dependency from `std::stringstream`.
116. [agrianius](https://github.com/agrianius) added code to use alternative string implementations.
117. [Daniel599](https://github.com/Daniel599) allowed to use more algorithms with the `items()` function.
118. [Julius Rakow](https://github.com/jrakow) fixed the Meson include directory and fixed the links to [cppreference.com](https://cppreference.com).
118. [Julius Rakow](https://github.com/juliusrakow) fixed the Meson include directory and fixed the links to [cppreference.com](https://cppreference.com).
119. [Sonu Lohani](https://github.com/sonulohani) fixed the compilation with MSVC 2015 in debug mode.
120. [grembo](https://github.com/grembo) fixed the test suite and re-enabled several test cases.
121. [Hyeon Kim](https://github.com/simnalamburt) introduced the macro `JSON_INTERNAL_CATCH` to control the exception handling inside the library.
@@ -1581,7 +1581,7 @@ I deeply appreciate the help of the following people.
166. [Mark Beckwith](https://github.com/wythe) fixed a typo.
167. [yann-morin-1998](https://github.com/yann-morin-1998) helped to reduce the CMake requirement to version 3.1.
168. [Konstantin Podsvirov](https://github.com/podsvirov) maintains a package for the MSYS2 software distro.
169. [remyabel](https://github.com/remyabel) added GNUInstallDirs to the CMake files.
169. [remyabel](https://github.com/remyabel2) added GNUInstallDirs to the CMake files.
170. [Taylor Howard](https://github.com/taylorhoward92) fixed a unit test.
171. [Gabe Ron](https://github.com/Macr0Nerd) implemented the `to_string` method.
172. [Watal M. Iwasaki](https://github.com/heavywatal) fixed a Clang warning.
@@ -1608,7 +1608,7 @@ I deeply appreciate the help of the following people.
193. [Hubert Chathi](https://github.com/uhoreg) made CMake's version config file architecture-independent.
194. [OmnipotentEntity](https://github.com/OmnipotentEntity) implemented the binary values for CBOR, MessagePack, BSON, and UBJSON.
195. [ArtemSarmini](https://github.com/ArtemSarmini) fixed a compilation issue with GCC 10 and fixed a leak.
196. [Evgenii Sopov](https://github.com/sea-kg) integrated the library to the wsjcpp package manager.
196. [Evgenii Sopov](https://github.com/sea5kg) integrated the library to the wsjcpp package manager.
197. [Sergey Linev](https://github.com/linev) fixed a compiler warning.
198. [Miguel Magalhães](https://github.com/magamig) fixed the year in the copyright.
199. [Gareth Sylvester-Bradley](https://github.com/garethsb-sony) fixed a compilation issue with MSVC.
@@ -1702,7 +1702,7 @@ I deeply appreciate the help of the following people.
287. [NN](https://github.com/NN---) added the Visual Studio output directory to `.gitignore`.
288. [Romain Reignier](https://github.com/romainreignier) improved the performance of the vector output adapter.
289. [Mike](https://github.com/Mike-Leo-Smith) fixed the `std::iterator_traits`.
291. [vakokako](https://github.com/vakokako) fixed tests when compiling with C++20.
292. [Alexander “weej” Jones](https://github.com/alexweej) fixed an example in the README.
293. [Eli Schwartz](https://github.com/eli-schwartz) added more files to the `include.zip` archive.
@@ -1727,7 +1727,7 @@ I deeply appreciate the help of the following people.
312. [Gareth Sylvester-Bradley](https://github.com/garethsb) added `operator/=` and `operator/` to construct JSON pointers.
313. [Michael Macnair](https://github.com/mykter) added support for afl-fuzz testing.
314. [Berkus Decker](https://github.com/berkus) fixed a typo in the README.
315. [Illia Polishchuk](https://github.com/effolkronium) improved the CMake testing.
315. [Illia Polishchuk](https://github.com/ilqvya) improved the CMake testing.
316. [Ikko Ashimine](https://github.com/eltociear) fixed a typo.
317. [Raphael Grimm](https://github.com/barcode) added the possibility to define a custom base class.
318. [tocic](https://github.com/tocic) fixed typos in the documentation.
@@ -1797,6 +1797,66 @@ I deeply appreciate the help of the following people.
382. [bitFiedler](https://github.com/bitFiedler) made GDB pretty printer work with Python 3.8.
383. [Gianfranco Costamagna](https://github.com/LocutusOfBorg) fixed a compiler warning.
384. [risa2000](https://github.com/risa2000) made `std::filesystem::path` conversion to/from UTF-8 encoded string explicit.
385. [AM](https://github.com/maqnouch) fixed typos in the README.
386. [dmenendez-gruposantander](https://github.com/dmenendez-gruposantander) fixed typos in the comments of the examples.
387. [Mihai Stan](https://github.com/mstan-xx) fixed comparisons against the literal `0`.
388. [Matt Gumbel](https://github.com/intelmatt) fixed some `-Weffc++` warnings.
389. [vimpunk](https://github.com/vimpunk) moved a lambda out of an unevaluated context to support older compilers.
390. [Chris Harris](https://github.com/cjh1) fixed the compilation with GCC 4.8.
391. [Palmer Dabbelt](https://github.com/palmer-dabbelt) generated and installed a pkg-config file.
392. [Gus Pozuelo](https://github.com/ap-viavi) made `ordered_map` compatible with GCC 5.5, Clang 3.6, and Xcode 9.
393. [AK](https://github.com/Lioncky) fixed an MSVC build error caused by the `min`/`max` macros from `windows.h`.
394. [Sergiu Deitsch](https://github.com/sergiud) provided a fallback for missing `char8_t` support.
395. [Xiaochuan Ye](https://github.com/XueSongTap) fixed `from_msgpack` for `std::byte` input by specializing `std::char_traits`.
396. [Ville Vesilehto](https://github.com/thevilledev) fixed an overflow in the BJData size calculation and rejected overflowing negative integers in CBOR.
397. [NmPassTHFan](https://github.com/nmpassthf) replaced the deprecated `std::is_trivial` for C++26.
398. [Chris Ever](https://github.com/chirsz-ever) added the `ignore_trailing_commas` parser option.
399. [Kuan-Fu Wu](https://github.com/kfwu1999) fixed the example code for `json_pointer` initialization.
400. [David Kilzer](https://github.com/ddkilzer) added a missing header to the input adapters.
401. [Miko](https://github.com/mikomikotaishi) added proper C++20 module support, simplified the module API, and fixed missing exports.
402. [hitgirl](https://github.com/hitgil) fixed the CMake configuration when cross-compiling.
403. [Devon Thomas](https://github.com/ThomaDevOSU) mentioned the Artistic Style formatting in the contribution guidelines.
404. [Erik Hu](https://github.com/Erikhu1) made Coveralls upload errors non-fatal in the CI.
406. [DmitriBogdanov](https://github.com/DmitriBogdanov) fixed broken package manager links in the documentation.
407. [Bander](https://github.com/banderzhm) improved the MSVC compatibility of the C++ modules.
408. [Andy Choi](https://github.com/ccpong) removed an unnecessary `template` keyword before `get` in the README and the documentation.
409. [SamareshSingh](https://github.com/ssam18) fixed single-element brace initialization to copy/move instead of wrapping in an array, fixed the `WITH_DEFAULT` macros for `ordered_map`, and handled moved events in `serve_header.py`.
410. [Aditya](https://github.com/Lumowhisp) improved the documentation of the documentation generation.
411. [cheese1](https://github.com/cheese1) clarified the README.
412. [KhloodElhossiny](https://github.com/khloodelhossiny) enabled `std::string_view` keys in `operator[]`.
413. [Charles Cabergs](https://github.com/cacharle) fixed a `-Wtautological-constant-out-of-range-compare` warning.
414. [EALePain](https://github.com/EALePain) made the `std::tuple` conversion work with reference types such as `std::tie`.
415. [koala_oishi](https://github.com/chibi-dogs) fixed grammatical wording in the README.
416. [riccardoori11](https://github.com/riccardoori11) fixed a typo in the documentation.
417. [Swastik Bose](https://github.com/VasuBhakt) fixed the parent pointers after `update()` with `JSON_DIAGNOSTICS` and fixed the Doxygen autolinking of requirements.
418. [trdesilva](https://github.com/trdesilva) added `front`, `pop_front`, and `push_front` to `json_pointer`.
419. [Akhilesh Arora](https://github.com/akhilesharora) fixed an incomplete-type error with `ordered_json`.
420. [Hariom Phulre](https://github.com/hariomphulre) fixed the C++20 modules compilation with GCC.
422. [George Sedov](https://github.com/radistmorse) added the `NLOHMANN_DEFINE_TYPE_*_WITH_NAMES` macros.
423. [Caillin Nugent](https://github.com/nugentcaillin) added the `NLOHMANN_JSON_SERIALIZE_ENUM_STRICT` macro.
424. [Cosmin D.](https://github.com/drcosmin) fixed `std::filesystem::path` conversions and added an MSVC workaround for `std::unique_ptr`.
425. [Paul Dreik](https://github.com/pauldreik) fixed a test relying on implementation-specific behavior.
426. [Daniel Falk](https://github.com/daniel-falk) added missing copyright notices to the SBOM.
427. [Federico Sfriso](https://github.com/federicosfriso05-dotcom) added support for constructing JSON values from C++20 range views.
428. [Luke Banicevic](https://github.com/banaboi) fixed corrupt BSON output for lengths exceeding `INT32_MAX`, cleaned up the BSON writer, and improved the documentation.
429. [Patrick Armstrong](https://github.com/Patrick10199) updated the CBOR references and the half-precision float assertions.
430. [Yash Bavadiya](https://github.com/xevrion) added checks to all BSON reads.
431. [hum4nBeing](https://github.com/hum4nBeing) fixed the overflow handling of high-precision numbers in UBJSON.
432. [tomatotomata](https://github.com/tomatotomata) added checks for reading CBOR tagged subtypes.
433. [YingqiDuan](https://github.com/YingqiDuan) documented the BSON interoperability.
434. [KBS](https://github.com/youdie006) documented the standards compliance and the strictness of `parse()` and `operator>>`.
435. [Petr Bělohlávek](https://github.com/petrbel) added Clang 21 and 22 to the CI.
436. [Dmitry Rantovov](https://github.com/darkdi) fixed the placement of a CBOR documentation block.
437. [ljcjclljc](https://github.com/ljcjclljc) fixed the comparison of large unsigned integers with signed integers.
438. [Sahil Kamate](https://github.com/sahilkamate03) fixed the handling of CBOR tags 0-5 and 21-23.
439. [Krishnanand G](https://github.com/Krishnanand-G) made the UBJSON writer reject `use_type` without `use_size`.
440. [whn](https://github.com/Whning0513) documented the lenient BSON input handling and corrected the complexity of `to_bson`.
441. [elix3r](https://github.com/22elix3r) fixed `update()` with `merge_objects` when merging a primitive into an object.
442. [Avionic Harshit](https://github.com/avionicharshit-byte) made `diff()` linear when an array shrinks.
443. [Qatadaha Bin Matloob](https://github.com/qatcod) fixed comparisons between integers and floats and fixed unparsable BJData output.
444. [Wu Shuwen](https://github.com/dajiaohuang) removed an unused include.
Thanks a lot for helping out! Please [let me know](mailto:mail@nlohmann.me) if I forgot someone.
@@ -34,6 +34,10 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va
("add", "remove", "move")
- Throws [`out_of_range.411`](../../home/exceptions.md#jsonexceptionout_of_range411) if an "add" operation's target
location has a parent that is neither an object nor an array.
- Throws [`out_of_range.413`](../../home/exceptions.md#jsonexceptionout_of_range413) if a "remove" operation's target
location has a parent that is neither an object nor an array.
- Throws [`out_of_range.414`](../../home/exceptions.md#jsonexceptionout_of_range414) if a "move" operation's "from"
location is a proper prefix of its "path" location.
- Throws [`other_error.501`](../../home/exceptions.md#jsonexceptionother_error501) if "test" operation was
unsuccessful.
@@ -75,3 +79,7 @@ is thrown. In any case, the original value is not changed: the patch is applied
- Added in version 2.0.0.
- Added [`out_of_range.411`](../../home/exceptions.md#jsonexceptionout_of_range411) and stopped relying on an internal assertion when an "add" operation's
target location has a non-object/non-array parent in version 3.13.0.
- Added [`out_of_range.413`](../../home/exceptions.md#jsonexceptionout_of_range413) and stopped silently ignoring a "remove" operation whose target
location has a non-object/non-array parent in version 3.13.0.
- Added [`out_of_range.414`](../../home/exceptions.md#jsonexceptionout_of_range414) and rejected a "move" operation whose "from" location is a proper
prefix of its "path" location instead of silently producing a corrupted result in version 3.13.0.
@@ -30,6 +30,10 @@ No guarantees, value may be corrupted by an unsuccessful patch operation.
("add", "remove", "move")
- Throws [`out_of_range.411`](../../home/exceptions.md#jsonexceptionout_of_range411) if an "add" operation's target
location has a parent that is neither an object nor an array.
- Throws [`out_of_range.413`](../../home/exceptions.md#jsonexceptionout_of_range413) if a "remove" operation's target
location has a parent that is neither an object nor an array.
- Throws [`out_of_range.414`](../../home/exceptions.md#jsonexceptionout_of_range414) if a "move" operation's "from"
location is a proper prefix of its "path" location.
- Throws [`other_error.501`](../../home/exceptions.md#jsonexceptionother_error501) if "test" operation was
unsuccessful.
@@ -72,3 +76,7 @@ function throws an exception.
- Added in version 3.11.0.
- Added [`out_of_range.411`](../../home/exceptions.md#jsonexceptionout_of_range411) and stopped relying on an internal assertion when an "add" operation's
target location has a non-object/non-array parent in version 3.13.0.
- Added [`out_of_range.413`](../../home/exceptions.md#jsonexceptionout_of_range413) and stopped silently ignoring a "remove" operation whose target
location has a non-object/non-array parent in version 3.13.0.
- Added [`out_of_range.414`](../../home/exceptions.md#jsonexceptionout_of_range414) and rejected a "move" operation whose "from" location is a proper
prefix of its "path" location instead of silently producing a corrupted result in version 3.13.0.
@@ -51,7 +51,11 @@ If you do want to preserve the **insertion order**, you can use the type [`nlohm
--8<-- "examples/ordered_json.output"
```
Alternatively, you can use a more sophisticated ordered map like [`tsl::ordered_map`](https://github.com/Tessil/ordered-map) ([integration](https://github.com/nlohmann/json/issues/546#issuecomment-304447518)) or [`nlohmann::fifo_map`](https://github.com/nlohmann/fifo_map) ([integration](https://github.com/nlohmann/json/issues/485#issuecomment-333652309)).
Alternatively, [`nlohmann::fifo_map`](https://github.com/nlohmann/fifo_map) also preserves the insertion order and, unlike [`ordered_map`](../api/ordered_map.md), keeps a lookup index, so it does not have the quadratic cost described below. It is used through a small adapter ([integration](https://github.com/nlohmann/json/issues/485#issuecomment-333652309)).
If the order does not matter and you only want faster lookup, `boost::unordered_flat_map`, `absl::flat_hash_map`, `absl::node_hash_map`, and several other hash maps work through an adapter that restores the template argument order `basic_json` expects; see [Template Parameter Requirements](types/template_parameters.md#objecttype). Note these are *unordered*, not insertion-ordered.
[`tsl::ordered_map`](https://github.com/Tessil/ordered-map) cannot be used: its iterators expose the mapped value as `const`, while `basic_json` needs to modify it in place.
The [`ordered_map`](../api/ordered_map.md) behind `nlohmann::ordered_json` is deliberately minimal and has no lookup
index, so every key access is a linear scan and building an object of `n` keys costs O(n²). This is unnoticeable at
| [`StringType`](#stringtype) | `std::string` | `std::string`-like types over `char` |
| [`BooleanType`](#booleantype) | `bool` | none worth using |
| [`NumberIntegerType`](#numberintegertype-and-numberunsignedtype) | `std::int64_t` | any signed integer type |
| [`NumberUnsignedType`](#numberintegertype-and-numberunsignedtype) | `std::uint64_t` | any unsigned integer type at least as wide as `NumberIntegerType` |
| [`diff`](../../api/basic_json/diff.md), [`items`](../../api/basic_json/items.md), [`std::hash`](../../api/basic_json/std_hash.md) | conversion of a `#!cpp std::size_t` to `StringType`: either assignability from the result of `#!cpp std::to_string`, or an ADL overload `#!cpp void int_to_string(StringType&, std::size_t)` |
| [`std::hash<basic_json>`](../../api/basic_json/std_hash.md) | additionally a specialization of `#!cpp std::hash<StringType>` |
| [`to_bson`](../../api/basic_json/to_bson.md) | `find(value_type)` and `npos` |
| [`parse`](../../api/basic_json/parse.md) from a `string_t` | the input adapters must accept it; otherwise pass a character range |
| `folly::fbstring` | requires C++20, see the note above |
| `eastl::string` | needs a user-supplied `#!cpp std::hash` and an ADL `int_to_string` (it is not assignable from a `#!cpp std::string`); [`parse`](../../api/basic_json/parse.md) does not accept it directly -- pass a character range or a `#!cpp std::string` |
| a custom string class in a user-defined namespace | if the requirements above are met |
| `#!cpp std::int64_t` / `#!cpp std::uint64_t` (default) | full |
| `#!cpp std::int32_t` / `#!cpp std::uint32_t`, `#!cpp long long` / `#!cpp unsigned long long` | full; narrower types change which literals the parser can represent |
| any other pair of standard signed/unsigned integer types | full |
| class types, enumerations | not usable; `#!cpp std::is_integral` must hold |
| `#!cpp bool`, or a type already used for another member of the union | not usable; `#!cpp std::is_integral<bool>` is in fact `#!cpp true`, but the `get_impl_ptr` overloads for `boolean_t`, `number_integer_t`, `number_unsigned_t` and `number_float_t` would collide |
## `NumberFloatType`
`number_float_t` is stored **directly** inside `basic_json`'s union.
### Always required
- Trivially default-constructible, trivially copyable, and trivially destructible (union member).
- `#!cpp std::numeric_limits` must be specialized; `max_digits10` is used to size the conversion.
- `#!cpp std::isfinite` must be applicable to the type.
### Required for parsing and serialization
`NumberFloatType` must be one of `#!cpp float`, `#!cpp double`, or `#!cpp long double`:
- The [parser](../parsing/index.md) converts number literals with `#!cpp std::strtof`, `#!cpp std::strtod`, or
`#!cpp std::strtold`; the library provides overloads for exactly these three types.
- [`dump`](../../api/basic_json/dump.md) falls back to `#!cpp std::snprintf` with the `%g` and `%Lg` conversion
specifiers, for which the library likewise provides only `#!cpp double` and `#!cpp long double` overloads
(`#!cpp float` is promoted to `#!cpp double`).
If `#!cpp std::numeric_limits<NumberFloatType>` describes an IEEE 754 binary32 or binary64 number, `dump` uses the
Grisu2 algorithm, which produces the shortest representation that round-trips. Otherwise the `snprintf` fallback with
`max_digits10` digits is used.
### Required for the binary formats
`NumberFloatType` must be `#!cpp float` or `#!cpp double`. The writers for
[CBOR, MessagePack, UBJSON, BJData, and BSON](../binary_formats/index.md) map a floating-point value onto an IEEE 754
binary32 or binary64 field and have no encoding for `#!cpp long double`.
| `QByteArray` | no `empty()` (it spells that `isEmpty()`); its `insert` takes an index rather than an iterator; and it converts to `string_t`, which makes `to_json` ambiguous between a string and a binary value |
| `#!cpp std::string` | `binary_t::container_type` and `string_t` would be the same type, so the two [`swap`](../../api/basic_json/swap.md) overloads collide and `basic_json` cannot be instantiated at all |
| `#!cpp std::deque<std::uint8_t>` | storage is not contiguous, so there is no `data()` |
| containers whose `value_type` is wider than one byte | see above -- accepted by the compiler, wrong at runtime |
## `CustomBaseClass`
`CustomBaseClass` is an extension point: unless it is `#!cpp void` (the default, which selects the empty
`nlohmann::json_default_base`), `basic_json` publicly derives from it.
### Always required
- A non-`final`, default-constructible class type.
- `basic_json` is copy-/move-constructible and copy-/move-assignable only if `CustomBaseClass` is.
### Notes
`basic_json` is documented to be a
[StandardLayoutType](https://en.cppreference.com/w/cpp/named_req/StandardLayoutType). Because `basic_json` has
non-static data members of its own, a `CustomBaseClass` with non-static data members forfeits this guarantee.
Note the namespace of `CustomBaseClass` becomes an associated namespace of `basic_json` for the purpose of
argument-dependent lookup.
See [`json_base_class_t`](../../api/basic_json/json_base_class_t.md) for an example.
A JSON Patch `remove` operation cannot be applied because the target location's parent is neither an object nor an array. Per [RFC 6902](https://datatracker.ietf.org/doc/html/rfc6902), a `remove` target must reference a member of an existing object or an element of an existing array; a primitive value (string, number, boolean, etc.) or `null` has no members or elements to remove.
!!! failure "Example message"
```
cannot remove value: the JSON Patch 'remove' target's parent is of type number, but must be an object or array
```
!!! note
This exception was added in version 3.13.0. Before that, this situation was silently ignored (the `remove` operation had no effect).
### json.exception.out_of_range.414
A JSON Patch `move` operation's `"from"` location is a proper prefix of its `"path"` location. Per [RFC 6902](https://datatracker.ietf.org/doc/html/rfc6902) (section 4.4), a location cannot be moved into one of its own children.
!!! failure "Example message"
```
cannot move value: 'from' path '/0' is a proper prefix of 'path' '/0/0'
```
!!! note
This exception was added in version 3.13.0. Before that, this situation could succeed with a corrupted result: for an array target, removing the "from" element before the "add" step shifted subsequent indices, so "path" silently re-resolved to a different element than intended.
### json.exception.out_of_range.415
MessagePack's ext type and BSON's binary subtype are each stored in a single byte. This exception is thrown when serializing a
@@ -90,6 +90,54 @@ The library supports **Unicode input** as follows:
In most cases, the parser is right to complain, because the input is not UTF-8 encoded. This is especially true for Microsoft Windows, where Latin-1 or ISO 8859-1 is often the standard encoding.
### NUL bytes in the input
!!! question "Questions"
- Why does `json::parse()` silently ignore part of my input?
- Why does a `std::string`/buffer with extra data after the JSON text parse without error, while a similar-looking string with extra text does not?
A `'\0'` (NUL) byte anywhere in the input is treated the same as the real end of the input, rather than as an ordinary (and, outside of a string, invalid) byte. Everything from that byte onward is silently ignored, without a parse error — including further, otherwise well-formed JSON:
```cpp
json::parse(std::string("123") + '\0'); // == 123, no error
json::parse(std::string("123") + '\0' + "true"); // == 123, the "true" is silently ignored too
```
This is different from any other unexpected trailing byte, which *does* raise [`parse_error.101`](../home/exceptions.md#jsonexceptionparse_error101):
```cpp
json::parse("123x"); // throws parse_error.101: unexpected additional data
```
This falls out of the same convention used when no explicit input length is given at all: `json::parse(const char*)` already stops at the first NUL byte via `strlen()`, since a bare pointer has no length of its own. The library applies that same NUL-terminated-C-string convention uniformly, rather than only when a length is genuinely unavailable — so a `std::string`, iterator range, or container whose content happens to include a NUL byte is affected the same way a raw `const char*` would be.
If your input may contain a trailing or embedded NUL that is **not** meant to signal the end of the JSON text — for instance, a fixed-size, zero-padded buffer — trim it yourself before calling `parse()`, since the library will otherwise silently stop there instead of raising an error:
```cpp
s.resize(s.find('\0')); // drop everything from the first NUL onward, if any
json::parse(s);
```
**Opt-in strict handling (since version 3.13.0)**
Manually trimming every input is easy to forget. If you define [`JSON_STRICT_NUL_HANDLING`](../api/macros/json_strict_nul_handling.md) to `1` before including the library, a `'\0'` byte is instead rejected like any other unexpected byte and raises `parse_error.101`, instead of being treated as end of input:
This macro defaults to `0` (disabled, preserving the behavior described above) to avoid breaking existing code that may depend on it, even unknowingly; it is planned to become the default in version 4.0.0. See [its documentation](../api/macros/json_strict_nul_handling.md) for details, including how it also affects `char` arrays such as string literals.
Note that this is unrelated to an *unescaped* NUL byte occurring **inside** a quoted JSON string, which is a different, already-invalid case and is correctly rejected either way:
```cpp
json::parse(std::string("\"") + '\0' + "\""); // throws parse_error.101: control character U+0000 (NUL) must be escaped to \u0000
@@ -198,6 +198,11 @@ Use the non-amalgamated version of the library. This option is `ON` by default.
Treat the library headers like system headers (i.e., adding `SYSTEM` to the [`target_include_directories`](https://cmake.org/cmake/help/latest/command/target_include_directories.html) call) to check for this library by tools like Clang-Tidy. This option is `OFF` by default.
### `JSON_StrictNulHandling`
Reject a `'\0'` (NUL) byte in the input instead of treating it as end of input, by defining the macro
[`JSON_STRICT_NUL_HANDLING`](../api/macros/json_strict_nul_handling.md). This option is `OFF` by default.
### `JSON_Valgrind`
Execute the test suite with [Valgrind](https://valgrind.org). This option is `OFF` by default. Depends on `JSON_BuildTests`.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Niels Lohmann
dependabot[bot]
Wu Shuwen