Under ensure_ascii, dump_escaped() calls find_ascii_copyable_run() at every
character boundary. When the text is dense non-ASCII - CJK, where every byte
is >= 0x80 - the scanner stops on its first byte and returns zero, so its SWAR
block runs once per character and buys nothing, on top of the escaping that
still has to happen afterwards.
A run can only be non-empty when the first byte is one the scanner may copy,
so test that single byte before calling it. Runs that do exist are found
exactly as before, so the bulk-copy win is unchanged; only the calls that were
always going to return zero are skipped.
Output is unchanged: the dump digest over canada/citm/twitter, in compact,
pretty and ensure_ascii form, matches develop byte for byte.
dump(ensure_ascii=true) develop before after
CJK text 3.54ms 4.25ms 3.36ms
CJK, no ASCII at all 3.09ms 4.02ms 3.02ms
Latin-1-ish text 4.39ms 3.04ms 2.93ms
plain ASCII 3.92ms 0.80ms 0.79ms
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
clang-tidy's readability-math-missing-parentheses wants the multiplication
spelled out in reserve(6 * depth + 1), and CI treats its warnings as errors.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
clang-tidy's misc-const-correctness reads the pointer dump_byte advanced over
the write buffer as one whose pointee could be const. Index the buffer
instead, which says the same thing without a raw pointer at all.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Two hot spots that the write buffer and the bulk scanner left behind.
dump_escaped took ensure_ascii as a runtime flag and tested it inside the
loop, once per character run, although it cannot change while a string is
written. It is now a template parameter, dispatched once per string, which
folds the choice of scanner and lets each of the two be inlined into a loop
of its own. This is the hottest loop in the serializer: it runs over every
string and every object key.
A binary value's bytes went through dump_integer, which counts digits and
does 64-bit arithmetic for a number that is always in [0, 255]. dump_byte
writes the three digits it takes at most straight into the write buffer
instead. Any byte type that is not a plain unsigned byte is still left to
dump_integer, whose representation of it may differ.
Measured against the previous commit (medians of 9 interleaved runs, clang
-O3): binary values -33.8%, dense CJK with ensure_ascii -20.6%, key-heavy
objects -17.8%, deeply nested pretty output -17.9%, dense CJK without
ensure_ascii -11.8%, object-heavy documents -9.3% compact and -9.5% pretty,
a small value dumped in a loop -21.4%, wide objects -2.3%. Arrays of plain
ASCII strings measured 3.5% to 4.2% slower, the one shape that loses; number
and integer arrays are unchanged.
Also tried and dropped: leaving the write and string buffers uninitialized
rather than zeroing 1.5 KB per dump() call. It is worth -30% on small values,
but two nearly identical string workloads moved 18% apart in opposite
directions, so the measurements did not support it.
The output is unchanged for every value: the differential now also covers
every one of the 256 byte values, alone and together, in both binary layouts.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Serializing a container serializes its elements, so dump() descended into one
call per nesting level. A value nested deeply enough exhausted the call stack
and terminated the process with a segmentation fault - no exception, nothing
the caller could catch. Parsing such a value works, as the parser is
iterative, and so does destroying one, as #1436 made destruction iterative.
Bound how far the descent goes rather than take the call stack away from it.
The first 128 levels are written by exactly the code that always wrote them,
and only below that does dump_iteratively write out what is left, keeping the
containers it has entered on an explicit stack. Serializing can therefore no
longer exhaust the stack, however deeply a value is nested, while a value
nested less deeply than the bound pays only for one comparison per container.
Writing every value that way instead measured between 2% and 20% slower - 20%
on object-heavy documents - which is why the descent is kept for all but the
values that cannot afford it. The bound costs nothing measurable: between
-1.4% and +1.2% across compact and pretty output of number, integer, string,
object-heavy, wide-object and deeply nested documents.
The output is unchanged for every value. Both ways of writing a container
emit the separator in front of every element but the first, rather than
after every element but the last, which puts exactly one between each pair
and none at the end.
This fixes#5387 for dump(). The copy constructor is fixed in #5389.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
clang-tidy flags the reference-to-array parameter under
cppcoreguidelines/hicpp/modernize-avoid-c-arrays, and the CI treats warnings as
errors. Binding to the array is the whole point here - it is what lets the
length be deduced from the literal instead of hand-written at the call site - so
suppress it the same way from_json(), to_json() and get_to() already suppress it
for their own T (&arr)[N] parameters.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
More of @gregmarr's review on the put_* split:
- Reattach the put_chars() doc comment, which the new helpers had been
inserted in front of, leaving it describing put_indent().
- Compute the literal length once in put_literal() instead of spelling N - 1
at each use.
- Add put_string(str, start, end), which keeps the pointer arithmetic and the
bounds assertions inside the function instead of at the call site. With
dump_float()'s to_chars() output moved onto put_buffer() as well, put_chars()
now has no callers outside put_string()/put_buffer(): nothing passes a bare
pointer and a count any more.
- Carry the indentation as std::size_t rather than unsigned int. It is a size,
it is compared and combined with buffer sizes throughout, and the casts in
put_indent() disappear. next_indent() keeps its assertion, which is far
harder to trip on a 64-bit size_t but still reachable where that is 32 bits.
No output change: pretty and compact dumps, binary values included, are
byte-identical to develop.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
@gregmarr's point on the fill-and-flush loop: flushing does not disturb what
the write buffer holds, so an indentation spanning several buffer-fulls only
has to be written into the buffer once and can then be handed to the adapter
as many times as needed. The loop re-filled it every time, doing work it
already knew was there.
put_indent() now fills the room left in the buffer, and if anything remains,
flushes, fills the buffer once, and re-flushes that same content. It also
returns early for a zero-width indentation, which is what the closing brace of
every outermost value asks for.
Measured over a dump(), counting memset calls and bytes inside put_indent:
indent before after
4 1 call / 4 B 1 call / 4 B
2000 2 calls / 2000 B 2 calls / 2046 B
100000 98 calls / 100000 B 2 calls / 2046 B
The wide case is now constant work rather than proportional to the indentation
width; ordinary widths are unchanged. Tests extended to cover several whole
buffer-fulls and an exact multiple of the buffer size.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Follow-up to @gregmarr's review: put_chars() was doing four unrelated jobs, so
give the two that can be made safe their own entry points.
- put_literal(): takes the literal by reference and deduces the length from the
array bound, so the 27 hand-counted lengths at the call sites can no longer
drift from the literals they describe. A literal is checked at compile time to
fit the buffer, so this path needs no write-through branch.
- put_buffer(): takes the fixed-size buffer itself rather than a bare pointer,
so the length can be checked against the buffer's own bound.
- put_indent(): memsets the indentation into the write buffer, filling and
flushing it as needed. This removes indent_string entirely, and with it both
bugs of #5186: the indentation string was grown by doubling, which is not
enough when indent_step more than doubles it (a heap over-read - dump(2000)
read 2000 bytes out of a 1024-byte string), and the grown part was filled with
a space instead of the configured indent_char. next_indent() keeps that PR's
assertion against the unsigned indentation accumulation wrapping on deep
nesting.
put_chars() keeps the two cases that are genuinely a pointer and a count: the
run-length copies out of the string being escaped, and to_chars() output.
Tests cover an indent_step wider than the write buffer, a non-space indentation
character past the old growth point, and nesting whose accumulated indentation
spans several buffer-fulls. All three fail against develop.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The "many small structural writes exceed the write buffer" subcase built
a 1100-deep nested array and dumped it to force >1024 consecutive
single-character writes through put_char (exercising the write buffer's
flush-when-full branch). dump() recurses per nesting level, so on MSVC
debug builds (smaller default stack, larger frames) this overflowed the
stack and crashed test-serialization; Linux/macOS have enough headroom to
hide it.
Replace the nesting with a flat array of 500 empty strings. Each element
emits '"', '"', ',' via put_char, so the dump is a long run of
single-character writes (1501 bytes > the 1024-byte buffer) at nesting
depth two, hitting the same flush branch without deep recursion. Library
code is unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
test-convenience failed (macOS finished first; the failure is
platform-independent) because check_escaped() calls the internal
serializer::dump_escaped() directly and then reads the output stream.
Since dump_escaped() now writes into the serializer's internal write
buffer, the bytes were still buffered and the stream was empty.
Expose flush() under JSON_PRIVATE_UNLESS_TESTED (same visibility as
dump_escaped) and flush in check_escaped() before inspecting the output.
Per-string flushing inside dump_escaped() was rejected on purpose: it
would defeat the buffering that makes object/array-heavy dumps faster.
Library behavior is unchanged (flush()'s body is identical; only its
access label moved).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Two further serialization speedups on top of the ensure_ascii=false bulk
copy, both reusing the SWAR primitives in detail/input/string_scan.hpp.
1. Internal write buffer (devirtualization). Every structural character
('{', '"', ',', ...) previously went straight to the output adapter
through a virtual call. Route all writes through put_char/put_chars
into a 1 KiB buffer that flushes in bulk; the public dump() flushes
once the top-level value is done (the recursive worker is split out as
dump_internal). Runs larger than the buffer are written straight
through, so large payloads are not copied twice. This is the dominant
cost for object/array-heavy values.
2. ensure_ascii fast path. dump_escaped previously ran the UTF-8 DFA over
every byte when escaping non-ASCII. Add find_ascii_copyable_run() (a
SWAR scan stopping at '"', '\\', < 0x20, 0x7F, and >= 0x80) so runs of
printable ASCII are bulk-copied, with the byte path handling each
escape/non-ASCII byte exactly as before.
Behavior is unchanged: dump output is byte-for-byte identical to the
previous implementation across ~20k randomized byte strings plus curated
edge cases (all escapes, control chars, 0x7F, valid multibyte,
surrogates, overlong, truncated), for object/array/pretty output, both
ensure_ascii settings, and all three error handlers, in C++11/17/20 at
-O2/-O3. New unit tests cover the buffer flush boundaries, the escape and
0x7F handling, multibyte under both settings, and invalid-UTF-8 handling.
Throughput (g++ -O3, vs the ensure_ascii=false-only baseline):
long ASCII, ensure_ascii=0 4.2x
long ASCII, ensure_ascii=1 4.1x
twitter-like objects 2.7x
dense CJK 1.8x
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
When ensure_ascii is false, dump_escaped previously ran every byte of
every string and object key through the UTF-8 DFA decoder, even for the
common case of ordinary text with nothing to escape. This mirrors the
per-byte cost the parser had before the contiguous fast paths.
At a character boundary, bulk-copy the longest run of bytes that need no
escaping using string_bulk_run() - the same SWAR scanner and UTF-8 bulk
validator the lexer's contiguous path uses - and only fall back to the
byte-at-a-time DFA loop for the first byte that needs individual handling
(a quote, backslash, control character, or ill-formed/truncated UTF-8).
Because every "hard" or invalid byte is still processed by the unchanged
byte path, escaping output and error handling (including strict-mode
error 316 position and message) are byte-identical to before.
The ensure_ascii=true path is unchanged: it must escape non-ASCII and
0x7F, which string_bulk_run does not stop on, so a separate predicate
would be needed for it.
Verified byte-for-byte identical dump output against the pre-change
implementation across ~20k randomized byte strings plus curated edge
cases (all escapes, control chars, valid multibyte, surrogates,
overlong, truncated sequences) for both ensure_ascii settings and all
three error handlers, in C++11/17/20 at -O2/-O3.
Throughput (g++ -O3, ensure_ascii=false, vs pre-change):
long ASCII strings 4.2x
twitter-like objects 2.3x
dense CJK 1.4x (further headroom with JSON_USE_SIMDUTF)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Resolves the number-parsing conflict between the discard_number_values
fast path (an early return for accept()-only calls that skips value
conversion for short digit runs) and the contiguous number scanner's
convert_number()/convert_integer() split: the discard check now runs
as the first step of convert_number(), so both the byte-at-a-time
scanner and the bulk contiguous path benefit from it.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Reserve capacity in from_json() object conversion when supported
The object-to-container from_json() overload filled the target
container one element at a time without reserving capacity, even
when the target type supports reserve() (e.g. std::unordered_map)
and the number of elements is already known. This caused unnecessary
rehashing while parsing large objects into such containers.
Add a reserve-detecting overload (from_json_object_impl), mirroring
the priority_tag-based SFINAE technique already used by the array
conversion path (from_json_array_impl), so that reserve(size()) is
called up front when available and the loop falls back unchanged
otherwise (e.g. for std::map).
Fixes#5406
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Update doc example outputs for new object-conversion iteration order
Reserving capacity in from_json()'s object-conversion path before
inserting elements changes libstdc++'s std::unordered_map bucket
layout, which changes the iteration order used by
get__ValueType_const.cpp, get_to.cpp and operator__ValueType.cpp to
print the elements of a converted std::unordered_map<std::string,
json>. Verified against a clean develop checkout (built with the
same GCC/libstdc++ used in CI) that the old order was produced
without this PR's change and the new order is produced with it, and
that the three affected examples now match their updated expected
output byte-for-byte.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Factor out a reserve-dispatch helper instead of duplicating the object from_json loop
Addresses review feedback from @gregmarr on PR #5472: the emplace loop no
longer needs to exist twice for the reserve/no-reserve cases. A small
from_json_object_reserve() overload pair (SFINAE-dispatched on whether
reserve() exists, mirroring the priority_tag technique used elsewhere)
either calls reserve() or is a no-op; from_json_object_impl() calls it once.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Inline from_json_object_impl into from_json now that it is called only once
Addresses review feedback from @gregmarr on PR #5472: with the reserve
loop de-duplicated, from_json_object_impl no longer needs to be a
separate function that from_json immediately delegates to.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
contains(const json_pointer&) is documented to never throw, but a purely
numeric reference token that is syntactically a valid array index yet
numerically too large to be represented (exceeding size_type's max, or
exceeding ULLONG_MAX and causing strtoull() to set errno to ERANGE) made
it fall through to array_index(), which throws out_of_range.410/404.
Pre-check the token's magnitude the same way array_index() does, but
return false instead of throwing, mirroring how the surrounding code
already rejects other malformed tokens (leading zero, non-digit
characters, "-") without throwing. operator[]/at() are untouched and
keep throwing for these inputs.
Fixes#5395
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Extract C++17-only content from unit-items.cpp into its own test file
unit-items.cpp is a 1433-line file that was being compiled twice per
CI configuration (once for C++11, once for C++17) purely because it
contained a single, small JSON_HAS_CPP_17-gated SECTION ("structured
bindings", 14 lines). Move that SECTION into a new, dedicated file
(tests/src/unit-items-cpp17.cpp) so only that tiny file needs a
second build; unit-items.cpp itself now builds/tests only once. No
tests/CMakeLists.txt changes are needed since the existing
file(GLOB ... src/unit-*.cpp) plus json_test_add_test_for() already
auto-register and standard-gate any new unit-*.cpp file based on
whether it textually contains JSON_HAS_CPP_<N> (the same mechanism
already used for the existing unit-iterators3.cpp file, which follows
the identical pattern).
Verified with plain clang++ under -std=c++11/14/17/20 and via a local
CMake configure+build that:
- unit-items.cpp now only produces a test-items_cpp11 target (the
former test-items_cpp17 target is gone) and its assertion/test-case
counts are unchanged (2 test cases / 222 assertions) for every
standard.
- The new unit-items-cpp17.cpp produces test-items-cpp17_cpp11 (an
intentionally empty translation unit under C++11 that reports 0
tests, 0 assertions, SUCCESS) and test-items-cpp17_cpp17 (1 test
case / 1 assertion, identical to what "structured bindings" ran
as before it was moved).
Separately, unit-regression1.cpp (1530 lines) was also being built
twice per CI configuration because it contained the substring
JSON_HAS_CPP_17 -- but on inspection this was dead code: an orphaned
"#ifdef JSON_HAS_CPP_17 / #include <variant> / #endif" left over from
when the actual std::variant-based regression test (issue #1292) was
relocated to unit-regression2.cpp. Nothing in unit-regression1.cpp
uses <variant>, so there is no SECTION/TEST_CASE to preserve here;
the dead include is simply removed. This was verified by grepping the
file for any other use of "variant" (none) and confirming issue #1292
is still covered by unit-regression2.cpp. Compiled and ran under
-std=c++11/14/17/20 and via CMake: unit-regression1.cpp now only
produces a test-regression1_cpp11 target (test-regression1_cpp17 is
gone) with an unchanged test-case count (3) under every standard.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix astyle indentation of #include inside #ifdef in unit-items-cpp17.cpp
This repo's astyle style keeps preprocessor directives at column 0
even inside #ifdef blocks. The new tests/src/unit-items-cpp17.cpp
had its #include <map>/#include <string> indented, which made the
'check' CI job's amalgamation/formatting diff non-empty and failed
the aggregate check.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Add test coverage for documented lenient BSON input handling
Issue #5333 documented three intentionally-lenient behaviors of the BSON
reader (any non-zero byte accepted as a boolean `true`, BSON array element
keys not validated against the required decimal sequence, and the payload
of binary subtype 0x02 "old binary" returned as-is including its inner
length prefix), but none of them was pinned by a test, so a future change
could silently regress the documented behavior.
Also add coverage for the out_of_range.412 length-overflow check
(shared by binary, string, and (sub-)document BSON length fields) for
the string and document cases; only the binary case was previously
tested.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix 32-bit overflow in huge_string_t BSON length-overflow tests
huge_string_t doubles as basic_json's StringType, so it is used not only
for the JSON string value under test but also for object keys (e.g. "s",
"nested"). Making size() unconditionally lie about being huge therefore
inflated the keys' reported sizes as well, pushing the running totals
computed while walking the BSON document (calc_bson_object_size and
friends in binary_writer.hpp) past what a 32-bit std::size_t can hold.
On 64-bit platforms this happens to still produce a working (if
needlessly large) result, but on 32-bit platforms (e.g. the mingw x86 CI
job) the size_t arithmetic silently wraps around: for the "document" test
this merely surfaces the wrong number in the exception message, but for
the "string" test the wrapped total happens to fall back under
INT32_MAX, so the intended out_of_range.412 guard is skipped entirely and
the code goes on to actually write ~2 GiB worth of characters from the
key's real, tiny buffer - which is what raised the reported
"vector::_M_range_insert" exception instead of a controlled 412.
Make the fake-huge size opt-in via huge_string_t::as_huge() and only
apply it to the string value under test, leaving keys at their real
(small) size. This keeps every intermediate size well within 32-bit
size_t range on any platform, matching how huge_binary_t already avoids
the same trap (it is only ever used as the BSON value type, never as a
key). Expected out_of_range.412 messages are updated accordingly.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Broaden JSON_HEDLEY_WARN_UNUSED_RESULT coverage to pure query functions
Add JSON_HEDLEY_WARN_UNUSED_RESULT to the unambiguous, const,
side-effect-free observer functions whose return value is the entire
purpose of the call:
- dump()
- type(), type_name()
- all is_* predicates (is_primitive, is_structured, is_null,
is_boolean, is_number, is_number_integer, is_number_unsigned,
is_number_float, is_object, is_array, is_string, is_binary,
is_discarded)
- empty(), size(), max_size()
- count(...) (both overloads) and contains(...) (all overloads,
including the deprecated json_pointer<BasicJsonType> overload)
This mirrors the direction the standard library has taken with
[[nodiscard]] on the analogous std::vector/std::map members, and
catches real bugs such as `j.empty();` (meant `j.clear();`) or
`j.contains(k);` with the result thrown away.
Deliberately out of scope (left for a separate, later policy
decision, per the issue): at(), value(), get*(), flatten(),
unflatten(), patch(), merge_patch(), begin()/end(), comparison
operators, erase(), and emplace().
Compiling the full test suite (tests/src/unit-*.cpp) with
-Wunused-result -Werror uncovered one real hit: a regression test in
unit-regression2.cpp called dump() purely to check it does not throw,
discarding the result. Fixed by explicitly casting to void, since the
call is intentionally result-less there.
Fixes#5410
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix discarded nodiscard results across the test suite for GCC's warn_unused_result
A plain (void) cast on a call expression suppresses the C++17 [[nodiscard]]
warning but not GCC's warning for functions annotated via the GNU
__attribute__((warn_unused_result)) form -- which is what
JSON_HEDLEY_WARN_UNUSED_RESULT expands to on GCC. Several existing tests
that call a newly-annotated function (dump(), empty()) purely to check
that it throws/does not throw, discarding the result via (void), newly
warned (and failed -Werror builds) once the annotation was broadened.
Route those discards through a small ignore_return_value() helper
instead, which actually consumes the value and suppresses the warning
on both attribute forms.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Use utils::ignore_return_value() for the issue #1445 dump() discard too
Addresses review feedback from @gregmarr on PR #5477: this call site was
still using the older "capture in a variable, then (void) it" pattern
from before this PR introduced utils::ignore_return_value(), instead of
the helper now used at every other discarded-nodiscard-result call site
this PR touches.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
accept() is a pure query whose only effect is the returned bool; both
parse() overloads and the deprecated accept(span_input_adapter&&, ...)
overload already carry JSON_HEDLEY_WARN_UNUSED_RESULT, but the two
current, recommended accept() overloads were missing it. Add the
annotation to match, so discarding accept()'s result now warns under
-Wunused-result / [[nodiscard]], as it already does for parse().
Fixes#5407
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Reject JSON Patch move when from is a proper prefix of path
RFC 6902 (section 4.4) forbids "from" from being a proper prefix of
"path" for a "move" operation: "a location cannot be moved into one
of its children." "move" is implemented as remove-then-add with no
check for this. For object targets, the subsequent "add" happened to
throw as a side effect of resolving through the now-removed parent,
but for array targets, removing the "from" element shifts subsequent
indices, so "path" silently re-resolves to a different element and
the operation "succeeds" with a silently corrupted document.
Add a check, before performing the remove/add, for whether "from" is
a proper prefix of "path" at the reference-token level. This compares
json_pointer's already-unescaped reference_tokens vectors (basic_json
is a friend of json_pointer) rather than the raw pointer strings, so
that tokens containing escaped '/' or '~' characters are compared
correctly, and a token that merely looks like a string prefix (e.g.
"/ab" vs "/abc/x") is not mistaken for a pointer-token prefix. When
"from" is a proper prefix of "path", throw out_of_range.414.
Fixes#5397.
Stacked on top of the fix for #5396 (branch
issue-5396-patch-remove-primitive-parent), since both touch the same
patch_inplace move/remove handling in include/nlohmann/json.hpp.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Add root-pointer and array-append-token edge case tests for the move prefix check
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Replace std::equal with an explicitly-bounded loop in the move prefix check
The three-iterator std::equal(first1, last1, first2) form has no
explicit end iterator for the second range, which a static analyzer
(Flawfinder, CWE-126) flags as a potential over-read even though the
preceding size comparison already guarantees the second range is long
enough. Rather than argue the point, make the bound visible in the code
itself via an explicit loop -- every access to ptr.reference_tokens is
now guarded by the same index the loop condition bounds against
from_size.
(The C++14 four-iterator std::equal(first1, last1, first2, last2) form
was tried first as a more minimal fix, but this codebase targets C++11
and that overload is not safely usable under -std=c++11 with all
supported standard library implementations.)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Extract the move prefix check into a named helper lambda
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Account for JSON_DIAGNOSTIC_POSITIONS in the move-prefix-check error messages
out_of_range::create() includes a "(bytes X-Y)" position annotation when
JSON_DIAGNOSTIC_POSITIONS is enabled, which the ci_test_diagnostic_positions
CI job builds the whole suite with. The five new out_of_range.414
assertions only checked the annotation-free message. Confirmed
JSON_DIAGNOSTICS produces the same (annotation-free) message as the
default build for this particular throw site (its path-based annotation
is empty at the root, where &result always points here), so only two
message variants are needed, not three.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
RFC 6902 (section 4.2) requires the target location of a "remove"
operation to exist. operation_remove handled parent.is_object() and
parent.is_array(), but had no final else branch: when the resolved
parent was a primitive value or null, neither branch matched and the
operation silently did nothing instead of failing.
Add the missing else branch, throwing out_of_range.413 with wording
that matches the existing out_of_range.411 thrown by the analogous
"add" case (operation_add) for the same kind of invalid parent.
Fixes#5396.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Route hand-rolled diagnostic pragmas through Hedley
Several places in the library hand-roll compiler diagnostic suppression
with raw `#pragma`/`#ifdef __GNUC__`/`#ifdef __clang__` guards instead of
using the Hedley primitives already bundled and used elsewhere
(JSON_HEDLEY_DIAGNOSTIC_PUSH/POP, JSON_HEDLEY_PRAGMA, ...). Converted six
of the seven listed push/pop pairs to use those primitives instead of
raw `#pragma GCC diagnostic`/`#pragma clang diagnostic` text:
- include/nlohmann/json.hpp (~3770, ~3863): -Wfloat-equal
- include/nlohmann/detail/conversions/to_chars.hpp (~1078): -Wfloat-equal
- include/nlohmann/detail/output/binary_writer.hpp (~1844): -Wfloat-equal
- include/nlohmann/detail/iterators/iteration_proxy.hpp (~211): -Wmismatched-tags
- include/nlohmann/detail/exceptions.hpp (~36): -Wweak-vtables
iteration_proxy.hpp did not previously include macro_scope.hpp itself
(it only compiled because some other header included earlier in
json.hpp happened to pull macro_scope.hpp in first); it now includes it
directly like the other detail headers that use Hedley macros, so it is
self-contained.
Each push/pop pair now uses JSON_HEDLEY_DIAGNOSTIC_PUSH/POP
unconditionally (a no-op on compilers that don't need it) and wraps the
actual `#pragma ... diagnostic ignored` text in JSON_HEDLEY_PRAGMA so it
goes through Hedley's _Pragma()-based emission instead of a raw #pragma
line, while keeping the original `#ifdef __GNUC__` / `#if
defined(__clang__)` guard around the ignored-pragma itself.
Deviation from the issue's suggested transformation: the issue's example
replaces the `#ifdef __GNUC__` guard with `#if
JSON_HEDLEY_HAS_WARNING("-Wfloat-equal")`. JSON_HEDLEY_HAS_WARNING is
implemented purely via Clang's `__has_warning` builtin and evaluates to
0 on real GCC (`#define JSON_HEDLEY_HAS_WARNING(warning) (0)` when
`__has_warning` is not defined), so adopting it verbatim would silently
stop suppressing -Wfloat-equal on GCC -- a real regression, not just a
style change. The existing `#ifdef __GNUC__` / `#if defined(__clang__)`
guards were kept for the ignored-pragma to stay behavior-preserving, and
only the push/pop/pragma-emission mechanism was routed through Hedley.
Two of the seven locations from the issue (the -Wignored-attributes
push at the very top of json.hpp and its matching pop after
`#include <nlohmann/detail/macro_unscope.hpp>`) were intentionally left
unconverted:
- The push, at the very top of json.hpp, runs before
`detail/macro_scope.hpp` (and therefore hedley.hpp) has been included
anywhere in the translation unit, so JSON_HEDLEY_DIAGNOSTIC_PUSH is not
yet defined at that point.
- The pop runs after `macro_unscope.hpp`, which -- via hedley_undef.hpp
-- has already #undef'd every JSON_HEDLEY_* macro (by design, see
#5408) precisely so they don't leak to users, so JSON_HEDLEY_DIAGNOSTIC_POP
is no longer defined by the time the pop is reached either.
Making this one pair work would require either hoisting the ~2000
line vendored hedley.hpp to the very top of the amalgamated single
header (a much bigger structural change to single_include than a pure
mechanism swap) or special-casing this one pop ahead of the general
macro cleanup. Both are riskier than the mechanical, behavior-preserving
change requested, so this pair was left as-is.
## Validation
- Compiled include/nlohmann/json.hpp and single_include/nlohmann/json.hpp
with `-Wall -Wextra -Wfloat-equal -Wmismatched-tags -Wweak-vtables`
(clang, which self-identifies as __GNUC__ too): no warnings, same as
before the change.
- Compiled and ran tests/src/unit-to_chars.cpp, unit-conversions.cpp,
unit-iterators1.cpp, unit-iterators2.cpp, and unit-class_parser.cpp
against the fixed include/: all pass.
- Compiled unit-msgpack.cpp, unit-bjdata.cpp, and unit-ubjson.cpp (which
exercise binary_writer.hpp's write_compact_float extensively): all
compile cleanly; the vast majority of assertions pass (the only
failures are pre-existing environment issues unrelated to this change
-- missing generated test-data files, not code correctness).
- Ran `make amalgamate`; the single_include diff is limited to exactly
the lines touched in include/, with no unrelated reordering.
- No real (non-Apple) GCC was available in this environment to test
directly; the `_Pragma("GCC diagnostic ...")` text emitted by
JSON_HEDLEY_PRAGMA is byte-identical to the prior `#pragma GCC
diagnostic ...` text, and the `#ifdef __GNUC__` guard is unchanged, so
GCC's behavior is expected to be identical. CI covers the GCC matrix.
This PR is stacked on top of #5475 (issue-5408-hedley-undef-leak) since
both touch the same files; only the last commit here is new.
Fixes#5409.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Guard JSON_HEDLEY_DIAGNOSTIC_PUSH/POP with the same compiler check as the pragma they bracket
Addresses review feedback from @gregmarr on PR #5485: the push/pop calls
were unconditional, so compilers other than the one the ignored-pragma
targets (e.g. MSVC, or GCC where the pair only applies under __clang__)
now did a needless push/pop with nothing suppressed in between. Move the
existing #ifdef __GNUC__ / #if defined(__clang__) guard to also cover the
push/pop, restoring the original zero-overhead behavior on other compilers
while still emitting the pragma itself through Hedley.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Undefine the four JSON_HEDLEY_* macros that leak after including json.hpp
include/nlohmann/detail/macro_unscope.hpp includes hedley_undef.hpp to
#undef every JSON_HEDLEY_* macro so none of them leak into the including
translation unit. Four macros were missing from that list and therefore
stayed defined after #include <nlohmann/json.hpp>:
- JSON_HEDLEY_PRAGMA
- JSON_HEDLEY_PREDICT_TRUE
- JSON_HEDLEY_PREDICT_FALSE
- JSON_HEDLEY_CLANG_HAS_DECLSPEC_ATTRIBUTE
hedley_undef.hpp is generated (via `make update_hedley`) by grepping
hedley.hpp for its own internal `#undef JSON_HEDLEY_X` redefinition
guards. JSON_HEDLEY_PRAGMA/PREDICT_TRUE/PREDICT_FALSE have no such guard
in upstream Hedley, so they were never picked up. The guard for
JSON_HEDLEY_CLANG_HAS_DECLSPEC_ATTRIBUTE also has an upstream typo
(`JSON_HEDLEY_CLANG_HAS_DECLSPEC_DECLSPEC_ATTRIBUTE`), so hedley_undef.hpp
was undefining the wrong (never-defined) name.
Fixes:
- include/nlohmann/thirdparty/hedley/hedley_undef.hpp: corrected the
DECLSPEC_ATTRIBUTE typo and added the three missing #undef lines,
keeping the file's alphabetical ordering.
- Makefile (update_hedley target): changed hedley_undef.hpp generation to
extract macro names directly from every `#define JSON_HEDLEY_...` in
hedley.hpp instead of from existing `#undef` guards, so a future
`make update_hedley` run undefines every macro Hedley actually defines,
even ones without a pre-existing redefinition guard. This was not run
in this PR (it would also pull in an unrelated upstream Hedley sync);
hedley_undef.hpp was hand-patched instead and single_include was
regenerated with `make amalgamate`.
- tests/src/unit-no-macro-leak.cpp: new regression test (picked up
automatically by tests/CMakeLists.txt's existing unit-*.cpp glob) that
includes json.hpp and then #ifdef/#error-checks every JSON_HEDLEY_*
macro name, so any future leak of any of the 151 vendored macros fails
the build, not just the four fixed here.
Fixes#5408.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Derive the JSON_HEDLEY_* leak-check test from hedley.hpp at build time
tests/src/unit-no-macro-leak.cpp previously hardcoded a static list of
~151 #ifdef/#error checks, one per JSON_HEDLEY_* macro name known at the
time it was written. That list would silently go stale the next time
`make update_hedley` pulls in a vendor update that adds, removes, or
renames a macro, since nothing would force it to be regenerated.
Add cmake/scripts/gen_hedley_undef_check.cmake, which derives the full
list of JSON_HEDLEY_* macro names directly from
include/nlohmann/thirdparty/hedley/hedley.hpp:
- tests/CMakeLists.txt uses it (MODE=checks) to (re)generate
hedley_undef_checks.inc at configure and build time, and wires the
generating custom target as a dependency of the test-no-macro-leak_cpp*
targets so it can never build against a stale copy. unit-no-macro-leak.cpp
now just #include-s the generated file inside its TEST_CASE instead of
carrying the checks itself.
- The Makefile's `update_hedley` target now delegates hedley_undef.hpp
generation to the same script (MODE=undef, new `update_hedley_undef`
target), so the vendored header, the generated #undef list, and the
generated test checks are all derived from the same extraction logic and
cannot drift apart.
This mirrors the approach taken independently in #5415 for the same
issue (#5408), credited there to a self-regenerating mechanism that
"can never drift again" -- ported into this branch instead of the
static list originally proposed here.
Verified with a local CMake configure + build + ctest, both against
include/ (JSON_MultipleHeaders=ON) and against the amalgamated
single_include/nlohmann/json.hpp (JSON_MultipleHeaders=OFF), and by
temporarily deleting a #undef line from hedley_undef.hpp to confirm the
generated test actually fails on a real leak.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix REUSE compliance failure in gen_hedley_undef_check.cmake
The generated file's embedded banner contains the literal text
'SPDX-License-Identifier: MIT' as part of the *content* being written
to hedley_undef.hpp, not as this .cmake script's own REUSE header (it
is already covered by the blanket 'Files: *' rule in .reuse/dep5).
The reuse tool matched that embedded line as an SPDX tag for the
script itself and failed to parse the trailing 'MIT\n")' as a valid
SPDX License Expression, breaking ci_reuse_compliance. Wrap the
embedded banner in REUSE-IgnoreStart/REUSE-IgnoreEnd comments, as
recommended by the tool's own diagnostic output.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Speed up whitespace skipping in the lexer
lexer::skip_whitespace() called get() for every whitespace byte, and
get() checks the (almost always false, once past the first character)
next_unget flag on every call. skip_whitespace() now reads its first
character with get() (needed to honor a pending unget() left over from
finishing the previous token, e.g. scan_number() always ungets the
character that terminated the number) and every further whitespace
character with a new get_ignoring_pending_unget() variant that skips
that branch, since nothing in the loop calls unget().
This is a narrower fix than the full contiguous-buffer bulk-skip
suggested in the issue (scan a run of whitespace directly in the
adapter's buffer and update position counters once per run). That
approach depends on bulk-scan adapter infrastructure
(supports_bulk_scan/bulk_data()/bulk_skip()) introduced by the open,
unmerged parser-performance PR #5283, which this change intentionally
does not depend on or replicate. Building new bulk-scan adapter
infrastructure from scratch was judged out of scope/riskier than
warranted here, so this change is limited to the safe, always-correct
improvement of removing redundant per-character bookkeeping from the
existing byte-at-a-time loop; full bulk-skipping is left as future
work once #5283 (or equivalent adapter support) lands.
Line/column/byte-offset bookkeeping is untouched and verified
bit-for-bit identical before and after this change, including for
pretty-printed (dump(4)) input with embedded newlines.
Fixes#5412
Stacked on top of the PR for #5411 (branch
issue-5411-lexer-skip-conversion).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix codegen regression in skip_whitespace() from #5490
Benchmarking found the get()/get_ignoring_pending_unget() split in
skip_whitespace() made long whitespace runs (e.g. indentation in
pretty-printed JSON) 1.75x-3.2x SLOWER instead of faster, reproducible
with both Apple Clang and GCC.
Root cause: rewriting the loop from a plain do-while into an initial
get() followed by a while-loop defeated the compiler's ability to keep
the input adapter's read/end pointers in registers across iterations;
both compilers instead reloaded them from memory on every character.
The function split itself was not the problem (it still fully
inlines); the loop's control-flow shape was.
The fix keeps the same two-function structure but restores a
do-while shape (guarded by an if for the "first char not whitespace"
case), which lets both compilers hoist the pointers back into
registers, matching or beating pre-#5490 performance.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Share the position-counter bump between get() and get_ignoring_pending_unget()
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Extract current_is_whitespace() to deduplicate skip_whitespace()'s two whitespace checks
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Use a raw string literal for the multi-line error-position test input
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix clang-tidy raw-string-literal finding and guard a new test against JSON_NOEXCEPTION
The issue #5412 whitespace-skipping test added a check_error() helper that
relies on catching json::parse_error to verify the exception message; under
JSON_NOEXCEPTION, JSON_THROW aborts instead of throwing, which crashed
ci_test_noexceptions (and cascaded into the other ci_cmake_options jobs).
Guard the whole section with #if !defined(JSON_NOEXCEPTION), matching the
existing pattern used by sibling tests in this file.
Also switch one escaped string literal to a raw string literal to satisfy
clang-tidy's modernize-raw-string-literal check.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Skip integer conversion in accept()/SAX validation when the value is unused
lexer::scan_number() always converted every numeric token with
strtoull()/strtoll() before returning, even though accept() (and any
consumer using json_sax_acceptor) immediately discards the converted
value. For value_unsigned/value_integer tokens whose digit count
already guarantees the value fits into 64 bits, the conversion cannot
change the accept/reject decision (such tokens are always finite and
unconditionally accepted), so scan_number() can skip strtoull()/
strtoll() entirely in that case when the caller signals it does not
need the value. Numbers with more digits keep using the exact,
unmodified conversion path, so overflow reclassification to
value_float (and the finiteness check on it) is unaffected.
parse() and value_float handling are completely unchanged.
Fixes#5411
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Document why the digit-count fast path is safe regardless of number_unsigned_t/number_integer_t width
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Avoid temporary-string concatenation flagged by clang-tidy in the differential test
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix to_bjdata() emitting unparsable output when _ArraySize_ is not an array
write_bjdata_ndarray() never checked that _ArraySize_ is an array. The shape
is written verbatim as the header length, so a null shape emitted 'Z' and an
object shape emitted '{' after the '#', neither of which from_bjdata()
accepts, and the round-trip guarantee in the BJData docs was broken.
Both slipped through the existing validation: for null, empty() is true so
the element count starts at 0 and the per-dimension loop never runs, and for
an object the loop walks its values, which can satisfy the non-negative
integer check. When _ArrayData_ then matched that count, the writer took the
ndarray path.
Require the shape to be an array, so anything else falls back to a plain
object encoding that round-trips, as the fallback rule in the docs already
specifies.
Signed-off-by: qatcod <79017227+qatcod@users.noreply.github.com>
* Document that _ArraySize_ must be an array in the ndarray requirements
The list at bjdata.md is the exhaustive set of conditions for the ndarray
encoding, but it only implied this one through 'every entry of'.
Signed-off-by: qatcod <79017227+qatcod@users.noreply.github.com>
---------
Signed-off-by: qatcod <79017227+qatcod@users.noreply.github.com>
* docs: add 89 customers and sort all sections
Extend the customers page from 136 to 225 entries. New entries were
found by searching vendor open-source notices and by inspecting
dependency manifests in public repositories.
Every added entry was verified to link either to a page that credits
the library, or to an open source repository where its use is directly
visible (a vendored copy, a build manifest, or an include). Candidates
whose only evidence was a transitive dependency (via ICU or
KDDockWidgets), packaging metadata, or an unused vendored file were
dropped rather than listed.
Also sort every section alphabetically ignoring case, and keep the
Peregrine lunar lander first under "Space Exploration".
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: add 41 customers and rebuild the overview image
The list was assembled by searching for names one at a time, which is why it
had plateaued. These additions come from enumerating instead: a public code
search returning every repository that references the library, and the reverse
build-dependencies of the Debian and Ubuntu source indices.
Every addition cites a first-party call site rather than a repository root,
and every evidence URL was checked to resolve. Several prominent candidates
were rejected on exactly that test:
- Node.js reaches the library only through deps/icu-small, the inherited
attribution ICU carries into everything that ships it.
- Visual Studio Code's only hits are copies of the library's own headers
used as sample C++ in the Copilot extension's test fixtures.
- OSS-Fuzz fuzzes the library rather than calling it; projects/json/ is the
library's own OSS-Fuzz integration.
- libuv matched only its AUTHORS file, which lists a contributor by name.
- simdjson and LLVM matched only benchmarks and a mangled-symbol test.
Two entries were already present under a different name and are merged rather
than duplicated: PrestoDB, and DB Browser for SQLite under its repository name.
GitHub CodeQL's citation moves from the repository root to shared/cpp/Diagnostics.h,
which includes the header directly.
The image is rebuilt with 236 logos over 17 rows. It is smaller than the one it
replaces, 1.02 MB against 1.35 MB, despite carrying 100 more marks.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
parse_float_fast() (Clinger) needs a significand below 2^53, so it always
declines once the mantissa has 17 or more significant digits. convert_number()
called it unconditionally, so those numbers were walked an extra time before
strtod had to run anyway. On streaming input, where scanning is byte-at-a-time
and there is no compensating win, that made canada.json about 6% slower than
develop.
Derive the significant-digit count from token_buffer indices - the digits are
not scanned again - and skip the call when it is guaranteed to decline. Both
scanners pass the offset where the mantissa ends; the count only has to be
corrected for a leading "0", which the JSON grammar admits nowhere else. The
integer path returns before the check, so integer-heavy input is unaffected.
Values are unchanged: this only avoids an attempt that would have failed.
Verified bit-exact against develop over every number in canada.json,
floats.json, signed_ints.json, unsigned_ints.json, small_signed_ints.json,
citm_catalog.json and twitter.json, for both the contiguous and the streaming
scanner.
parse, streaming develop before after
canada.json 19.4ms 20.5ms 19.3ms
floats.json 135.9ms 131.8ms 128.0ms
parse, contiguous develop before after
canada.json 15.5ms 12.9ms 11.7ms
floats.json 98.6ms 69.8ms 66.7ms
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The record_buffer test type declares data() and size() so the
is_contiguous_byte_container trait can see both and still reject the
type on its value_type. data() was never called, so clang's
-Wunneeded-member-function (under -Weverything -Werror) failed the
C++20 build. Assert that data() points at the underlying bytes: it
ODR-uses the member and documents the property the type is meant to
demonstrate.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
simdutf needs C++17: without it the dependency does not even compile, and with
a C++17 compiler but no C++17-or-later standard under test it builds and then
goes unused. Either way the option silently did nothing useful, or broke the
configure step outright.
Resolve the tested standards first, then check them: when none of them can
reach simdutf, skip the dependency and say so, naming which of the two reasons
applies and how to fix it. The tests then run against the scalar validator,
which is what would have happened anyway.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
JSON_USE_SIMDUTF was documented and shipped but never compiled by anything in
the repository, so nothing held the backend to the behavior the docs promise.
Add JSON_TestSimdutf (OFF by default), which fetches simdutf and defines
JSON_USE_SIMDUTF for every test target, and a ci_test_simdutf target that runs
the whole suite in that configuration. Because simdutf needs C++17, the suite is
built at C++11 as well, so one job covers both the scalar fallback with the
macro defined and simdutf itself.
The dependency hangs off test_main, whose usage requirements every test target
inherits. The library target and the installed CMake package are deliberately
untouched: making nlohmann_json link simdutf would put a find_dependency() in
the exported package, which is a separate decision.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
simdutf.h rejects anything below C++17 with an #error, so defining
JSON_USE_SIMDUTF in a C++11 or C++14 translation unit did not fail with a
message about simdutf being unavailable - it failed to compile at all, taking
the library's C++11 support with it. Nothing caught this because no build ever
compiled that path.
Gate the include and both uses on JSON_HAS_CPP_17, the same way number_parse.hpp
gates std::from_chars. Below C++17 the macro now has no effect and the scalar
validator runs; it accepts and rejects exactly the same input, so the macro is
safe to set project-wide even when some translation units use an older standard.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
is_contiguous_byte_container accepted any type with a data() returning a
pointer to a single-byte integral plus a size(). That is duck typing: the two
members say nothing about size() counting the units data() points at.
A type where it does not - fixed-size records, say - was routed to the
pointer-based adapter and parsed as [data(), data() + size()) bytes, silently
truncating input the iterator-based adapter had read in full:
struct record_buffer {
using value_type = std::array<char, 4>;
std::string bytes;
const char* data() const; // raw bytes
std::size_t size() const; // in records
const char* begin() const; const char* end() const;
};
json::parse(record_buffer{"[1,2,3,4,5]"}); // parse error at column 3
Requiring the container's own value_type to be that same element type ties the
two together. Every contiguous standard container satisfies it, so std::string,
std::vector<char>, std::array<char, N> and std::string_view keep the fast path;
anything else falls back to the iterator-based adapter, which is always correct.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
scan_number_bulk_contiguous() converts an integer token straight from the input
buffer. When the value does not fit, it materializes token_buffer and calls
convert_number(), which tried the very same integer conversion again before
falling back to floating point.
Recording the outcome in number_type skips the second attempt. The resulting
token type and value are unchanged: convert_number() reached the float tail
either way.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The comment claimed the reported column is the one the offending token starts
at. It is the column reached after the token's last character - which is the
actual point of the unget() change: a number terminated by a newline now
reports what the same number terminated by a space always did.
Assert that equality directly, and add a multi-character token where the start
and end columns differ, so the invariant cannot be read off a single-character
example.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
string_bulk_run() had the same `return scalar_string_bulk_run(...)` in both
arms of the `#if`. The simdutf arm already falls through when validation
fails, so a single return after the `#endif` says the same thing.
The JSON_USE_SIMDUTF example showed `#include <simdutf.h>`, which
string_scan.hpp already does under the same guard; users only have to put the
header on the include path and link the library, not include it themselves.
Set the version history entry to 3.13.0, matching the other macro pages
documenting unreleased features.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
@gregmarr spotted that convert_integer() was inserted between convert_number()
and its doc block, leaving convert_integer() with two stacked blocks and
convert_number() with none. Comment only; no code change.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
clang-tidy reads "[\"\\ud834\"]" as a literal better written raw, and the
two literals written next to each other in "[\"a\x01""b\"]" as a missing
comma. The concatenation was there to stop the hex escape swallowing the
following character; build those documents from explicit bytes instead and
use raw strings elsewhere. The byte sequences are unchanged.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
- give the helper lambdas an explicit std::string return type and return
braced initializer lists (modernize-return-braced-init-list)
- replace the C-style array of test cases with a std::vector
(modernize-avoid-c-arrays)
- silence pro-type-member-init on the two brace-initialized aggregates;
default member initializers would stop them being aggregates in C++11
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Making supports_bulk_scan depend on iterator_is_contiguous meant the trait
is now instantiated for every adapter, not only when get_elements() is
called. On standard libraries with an incomplete <ranges> that is fatal:
libstdc++ 10 evaluates std::contiguous_iterator<std::counted_iterator<T*>>
by calling std::to_address, which needs an operator-> its counted_iterator
does not have, so satisfaction checking is a hard error rather than false.
Reported by clang 14 + libstdc++ 10.
JSON_HAS_RANGES already encodes exactly this ("libstdc++ < 11 has incomplete
C++20 ranges", #4440), so require it for the C++20 branch. Affected
toolchains fall back to the pointer-only test and the byte-at-a-time
scanner, which parses identically, just without the bulk fast paths.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Two problems in the tests added for the bulk scanners, both found by CI:
- the inner `const json j` in the counted-iterator diagnostics shadowed the
one declared at test-case scope, which -Wshadow rejects on GCC and clang
and C4456 rejects on MSVC under /WX; rename them
- the new parity checks parse deliberately invalid input, which calls
std::abort() rather than throwing when JSON_NOEXCEPTION is defined, so
they would have crashed the no-exception build; guard them the way the
other tests do
json::accept() does not abort, so the UTF-8 range assertions stay compiled
without exceptions and keep covering validate_one_utf8() there.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The pinned astyle expands a braced-init-list used as a range-for range onto
several lines; hoist the two offsets into a named vector instead, which reads
better and leaves nothing for astyle to reformat.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
These paths had no dedicated tests and rested on differential fuzzing only.
Add three sections, all comparing the contiguous scanner against the
byte-at-a-time one on the parsed value and on the exact error message:
- every string of length 1..3 over an alphabet of ordinary ASCII, both
specials, a control byte, escape characters, UTF-8 lead and continuation
bytes, and a byte that is never valid - each at offset 0 and offset 9, so
the bulk scanner sees them with and without a run behind them
- every kind of run-ending byte at each offset across two 8-byte SWAR words,
so multibyte sequences also straddle the word boundary
- the boundaries of every range validate_one_utf8() recognizes: shortest and
longest encodings, overlongs, both ends of the surrogate block, U+10FFFF
and just past it, and truncated sequences
Verified to fail if the bulk validator accepts surrogates, and if the SWAR
word test stops detecting control characters.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
An integer token does not need token_buffer: the number_integer and
number_unsigned SAX callbacks take only the value, and the overflow
diagnostic rebuilds the text from the input via get_token_string(). Convert
straight from the input buffer and materialize the token only for the
floating-point tail, which still needs a NUL-terminated buffer for strtod.
JSON_DIAGNOSTIC_POSITIONS derives a number's start position from
get_string().size(), so the copy is kept when that is enabled.
The integer dispatch is factored into convert_integer() and shared with
convert_number(), so both scanners keep using one implementation.
Integer-heavy input, 400k values, -O3:
parse accept
gcc 16 +14% +23%
clang +15% +22%
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The JSON number grammar is encoded twice: as the scan_number() state machine
and as the contiguous fast path. The fast path declining on anything it does
not recognize keeps most divergence harmless, but if it ever accepted
something the state machine rejects the result would be a silent correctness
bug, and the existing test only pinned a hand-written list of numbers.
Enumerate every string of length 1..4 over "01.eE+-" (2800 tokens) and
require both paths to agree on the parsed value and on the exact error
message. Verified to fail if the fast path's grammar is perturbed.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
json::parse is declared warn_unused_result, and CHECK_THROWS_WITH_AS
evaluates its expression as a discarded statement, so the assertion broke
the -Werror builds (GCC -Werror=unused-result, MSVC C4834 under /WX).
Compare against the helper that already captures the message instead.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Niels Lohmann
dependabot[bot]
Qatadaha Bin Matloob