Under ensure_ascii, dump_escaped() calls find_ascii_copyable_run() at every
character boundary. When the text is dense non-ASCII - CJK, where every byte
is >= 0x80 - the scanner stops on its first byte and returns zero, so its SWAR
block runs once per character and buys nothing, on top of the escaping that
still has to happen afterwards.
A run can only be non-empty when the first byte is one the scanner may copy,
so test that single byte before calling it. Runs that do exist are found
exactly as before, so the bulk-copy win is unchanged; only the calls that were
always going to return zero are skipped.
Output is unchanged: the dump digest over canada/citm/twitter, in compact,
pretty and ensure_ascii form, matches develop byte for byte.
dump(ensure_ascii=true) develop before after
CJK text 3.54ms 4.25ms 3.36ms
CJK, no ASCII at all 3.09ms 4.02ms 3.02ms
Latin-1-ish text 4.39ms 3.04ms 2.93ms
plain ASCII 3.92ms 0.80ms 0.79ms
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
clang-tidy's misc-const-correctness reads the pointer dump_byte advanced over
the write buffer as one whose pointee could be const. Index the buffer
instead, which says the same thing without a raw pointer at all.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Two hot spots that the write buffer and the bulk scanner left behind.
dump_escaped took ensure_ascii as a runtime flag and tested it inside the
loop, once per character run, although it cannot change while a string is
written. It is now a template parameter, dispatched once per string, which
folds the choice of scanner and lets each of the two be inlined into a loop
of its own. This is the hottest loop in the serializer: it runs over every
string and every object key.
A binary value's bytes went through dump_integer, which counts digits and
does 64-bit arithmetic for a number that is always in [0, 255]. dump_byte
writes the three digits it takes at most straight into the write buffer
instead. Any byte type that is not a plain unsigned byte is still left to
dump_integer, whose representation of it may differ.
Measured against the previous commit (medians of 9 interleaved runs, clang
-O3): binary values -33.8%, dense CJK with ensure_ascii -20.6%, key-heavy
objects -17.8%, deeply nested pretty output -17.9%, dense CJK without
ensure_ascii -11.8%, object-heavy documents -9.3% compact and -9.5% pretty,
a small value dumped in a loop -21.4%, wide objects -2.3%. Arrays of plain
ASCII strings measured 3.5% to 4.2% slower, the one shape that loses; number
and integer arrays are unchanged.
Also tried and dropped: leaving the write and string buffers uninitialized
rather than zeroing 1.5 KB per dump() call. It is worth -30% on small values,
but two nearly identical string workloads moved 18% apart in opposite
directions, so the measurements did not support it.
The output is unchanged for every value: the differential now also covers
every one of the 256 byte values, alone and together, in both binary layouts.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Serializing a container serializes its elements, so dump() descended into one
call per nesting level. A value nested deeply enough exhausted the call stack
and terminated the process with a segmentation fault - no exception, nothing
the caller could catch. Parsing such a value works, as the parser is
iterative, and so does destroying one, as #1436 made destruction iterative.
Bound how far the descent goes rather than take the call stack away from it.
The first 128 levels are written by exactly the code that always wrote them,
and only below that does dump_iteratively write out what is left, keeping the
containers it has entered on an explicit stack. Serializing can therefore no
longer exhaust the stack, however deeply a value is nested, while a value
nested less deeply than the bound pays only for one comparison per container.
Writing every value that way instead measured between 2% and 20% slower - 20%
on object-heavy documents - which is why the descent is kept for all but the
values that cannot afford it. The bound costs nothing measurable: between
-1.4% and +1.2% across compact and pretty output of number, integer, string,
object-heavy, wide-object and deeply nested documents.
The output is unchanged for every value. Both ways of writing a container
emit the separator in front of every element but the first, rather than
after every element but the last, which puts exactly one between each pair
and none at the end.
This fixes#5387 for dump(). The copy constructor is fixed in #5389.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
clang-tidy flags the reference-to-array parameter under
cppcoreguidelines/hicpp/modernize-avoid-c-arrays, and the CI treats warnings as
errors. Binding to the array is the whole point here - it is what lets the
length be deduced from the literal instead of hand-written at the call site - so
suppress it the same way from_json(), to_json() and get_to() already suppress it
for their own T (&arr)[N] parameters.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
More of @gregmarr's review on the put_* split:
- Reattach the put_chars() doc comment, which the new helpers had been
inserted in front of, leaving it describing put_indent().
- Compute the literal length once in put_literal() instead of spelling N - 1
at each use.
- Add put_string(str, start, end), which keeps the pointer arithmetic and the
bounds assertions inside the function instead of at the call site. With
dump_float()'s to_chars() output moved onto put_buffer() as well, put_chars()
now has no callers outside put_string()/put_buffer(): nothing passes a bare
pointer and a count any more.
- Carry the indentation as std::size_t rather than unsigned int. It is a size,
it is compared and combined with buffer sizes throughout, and the casts in
put_indent() disappear. next_indent() keeps its assertion, which is far
harder to trip on a 64-bit size_t but still reachable where that is 32 bits.
No output change: pretty and compact dumps, binary values included, are
byte-identical to develop.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
@gregmarr's point on the fill-and-flush loop: flushing does not disturb what
the write buffer holds, so an indentation spanning several buffer-fulls only
has to be written into the buffer once and can then be handed to the adapter
as many times as needed. The loop re-filled it every time, doing work it
already knew was there.
put_indent() now fills the room left in the buffer, and if anything remains,
flushes, fills the buffer once, and re-flushes that same content. It also
returns early for a zero-width indentation, which is what the closing brace of
every outermost value asks for.
Measured over a dump(), counting memset calls and bytes inside put_indent:
indent before after
4 1 call / 4 B 1 call / 4 B
2000 2 calls / 2000 B 2 calls / 2046 B
100000 98 calls / 100000 B 2 calls / 2046 B
The wide case is now constant work rather than proportional to the indentation
width; ordinary widths are unchanged. Tests extended to cover several whole
buffer-fulls and an exact multiple of the buffer size.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Follow-up to @gregmarr's review: put_chars() was doing four unrelated jobs, so
give the two that can be made safe their own entry points.
- put_literal(): takes the literal by reference and deduces the length from the
array bound, so the 27 hand-counted lengths at the call sites can no longer
drift from the literals they describe. A literal is checked at compile time to
fit the buffer, so this path needs no write-through branch.
- put_buffer(): takes the fixed-size buffer itself rather than a bare pointer,
so the length can be checked against the buffer's own bound.
- put_indent(): memsets the indentation into the write buffer, filling and
flushing it as needed. This removes indent_string entirely, and with it both
bugs of #5186: the indentation string was grown by doubling, which is not
enough when indent_step more than doubles it (a heap over-read - dump(2000)
read 2000 bytes out of a 1024-byte string), and the grown part was filled with
a space instead of the configured indent_char. next_indent() keeps that PR's
assertion against the unsigned indentation accumulation wrapping on deep
nesting.
put_chars() keeps the two cases that are genuinely a pointer and a count: the
run-length copies out of the string being escaped, and to_chars() output.
Tests cover an indent_step wider than the write buffer, a non-space indentation
character past the old growth point, and nesting whose accumulated indentation
spans several buffer-fulls. All three fail against develop.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
test-convenience failed (macOS finished first; the failure is
platform-independent) because check_escaped() calls the internal
serializer::dump_escaped() directly and then reads the output stream.
Since dump_escaped() now writes into the serializer's internal write
buffer, the bytes were still buffered and the stream was empty.
Expose flush() under JSON_PRIVATE_UNLESS_TESTED (same visibility as
dump_escaped) and flush in check_escaped() before inspecting the output.
Per-string flushing inside dump_escaped() was rejected on purpose: it
would defeat the buffering that makes object/array-heavy dumps faster.
Library behavior is unchanged (flush()'s body is identical; only its
access label moved).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Two further serialization speedups on top of the ensure_ascii=false bulk
copy, both reusing the SWAR primitives in detail/input/string_scan.hpp.
1. Internal write buffer (devirtualization). Every structural character
('{', '"', ',', ...) previously went straight to the output adapter
through a virtual call. Route all writes through put_char/put_chars
into a 1 KiB buffer that flushes in bulk; the public dump() flushes
once the top-level value is done (the recursive worker is split out as
dump_internal). Runs larger than the buffer are written straight
through, so large payloads are not copied twice. This is the dominant
cost for object/array-heavy values.
2. ensure_ascii fast path. dump_escaped previously ran the UTF-8 DFA over
every byte when escaping non-ASCII. Add find_ascii_copyable_run() (a
SWAR scan stopping at '"', '\\', < 0x20, 0x7F, and >= 0x80) so runs of
printable ASCII are bulk-copied, with the byte path handling each
escape/non-ASCII byte exactly as before.
Behavior is unchanged: dump output is byte-for-byte identical to the
previous implementation across ~20k randomized byte strings plus curated
edge cases (all escapes, control chars, 0x7F, valid multibyte,
surrogates, overlong, truncated), for object/array/pretty output, both
ensure_ascii settings, and all three error handlers, in C++11/17/20 at
-O2/-O3. New unit tests cover the buffer flush boundaries, the escape and
0x7F handling, multibyte under both settings, and invalid-UTF-8 handling.
Throughput (g++ -O3, vs the ensure_ascii=false-only baseline):
long ASCII, ensure_ascii=0 4.2x
long ASCII, ensure_ascii=1 4.1x
twitter-like objects 2.7x
dense CJK 1.8x
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
When ensure_ascii is false, dump_escaped previously ran every byte of
every string and object key through the UTF-8 DFA decoder, even for the
common case of ordinary text with nothing to escape. This mirrors the
per-byte cost the parser had before the contiguous fast paths.
At a character boundary, bulk-copy the longest run of bytes that need no
escaping using string_bulk_run() - the same SWAR scanner and UTF-8 bulk
validator the lexer's contiguous path uses - and only fall back to the
byte-at-a-time DFA loop for the first byte that needs individual handling
(a quote, backslash, control character, or ill-formed/truncated UTF-8).
Because every "hard" or invalid byte is still processed by the unchanged
byte path, escaping output and error handling (including strict-mode
error 316 position and message) are byte-identical to before.
The ensure_ascii=true path is unchanged: it must escape non-ASCII and
0x7F, which string_bulk_run does not stop on, so a separate predicate
would be needed for it.
Verified byte-for-byte identical dump output against the pre-change
implementation across ~20k randomized byte strings plus curated edge
cases (all escapes, control chars, valid multibyte, surrogates,
overlong, truncated sequences) for both ensure_ascii settings and all
three error handlers, in C++11/17/20 at -O2/-O3.
Throughput (g++ -O3, ensure_ascii=false, vs pre-change):
long ASCII strings 4.2x
twitter-like objects 2.3x
dense CJK 1.4x (further headroom with JSON_USE_SIMDUTF)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Resolves the number-parsing conflict between the discard_number_values
fast path (an early return for accept()-only calls that skips value
conversion for short digit runs) and the contiguous number scanner's
convert_number()/convert_integer() split: the discard check now runs
as the first step of convert_number(), so both the byte-at-a-time
scanner and the bulk contiguous path benefit from it.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Reserve capacity in from_json() object conversion when supported
The object-to-container from_json() overload filled the target
container one element at a time without reserving capacity, even
when the target type supports reserve() (e.g. std::unordered_map)
and the number of elements is already known. This caused unnecessary
rehashing while parsing large objects into such containers.
Add a reserve-detecting overload (from_json_object_impl), mirroring
the priority_tag-based SFINAE technique already used by the array
conversion path (from_json_array_impl), so that reserve(size()) is
called up front when available and the loop falls back unchanged
otherwise (e.g. for std::map).
Fixes#5406
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Update doc example outputs for new object-conversion iteration order
Reserving capacity in from_json()'s object-conversion path before
inserting elements changes libstdc++'s std::unordered_map bucket
layout, which changes the iteration order used by
get__ValueType_const.cpp, get_to.cpp and operator__ValueType.cpp to
print the elements of a converted std::unordered_map<std::string,
json>. Verified against a clean develop checkout (built with the
same GCC/libstdc++ used in CI) that the old order was produced
without this PR's change and the new order is produced with it, and
that the three affected examples now match their updated expected
output byte-for-byte.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Factor out a reserve-dispatch helper instead of duplicating the object from_json loop
Addresses review feedback from @gregmarr on PR #5472: the emplace loop no
longer needs to exist twice for the reserve/no-reserve cases. A small
from_json_object_reserve() overload pair (SFINAE-dispatched on whether
reserve() exists, mirroring the priority_tag technique used elsewhere)
either calls reserve() or is a no-op; from_json_object_impl() calls it once.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Inline from_json_object_impl into from_json now that it is called only once
Addresses review feedback from @gregmarr on PR #5472: with the reserve
loop de-duplicated, from_json_object_impl no longer needs to be a
separate function that from_json immediately delegates to.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
contains(const json_pointer&) is documented to never throw, but a purely
numeric reference token that is syntactically a valid array index yet
numerically too large to be represented (exceeding size_type's max, or
exceeding ULLONG_MAX and causing strtoull() to set errno to ERANGE) made
it fall through to array_index(), which throws out_of_range.410/404.
Pre-check the token's magnitude the same way array_index() does, but
return false instead of throwing, mirroring how the surrounding code
already rejects other malformed tokens (leading zero, non-digit
characters, "-") without throwing. operator[]/at() are untouched and
keep throwing for these inputs.
Fixes#5395
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Broaden JSON_HEDLEY_WARN_UNUSED_RESULT coverage to pure query functions
Add JSON_HEDLEY_WARN_UNUSED_RESULT to the unambiguous, const,
side-effect-free observer functions whose return value is the entire
purpose of the call:
- dump()
- type(), type_name()
- all is_* predicates (is_primitive, is_structured, is_null,
is_boolean, is_number, is_number_integer, is_number_unsigned,
is_number_float, is_object, is_array, is_string, is_binary,
is_discarded)
- empty(), size(), max_size()
- count(...) (both overloads) and contains(...) (all overloads,
including the deprecated json_pointer<BasicJsonType> overload)
This mirrors the direction the standard library has taken with
[[nodiscard]] on the analogous std::vector/std::map members, and
catches real bugs such as `j.empty();` (meant `j.clear();`) or
`j.contains(k);` with the result thrown away.
Deliberately out of scope (left for a separate, later policy
decision, per the issue): at(), value(), get*(), flatten(),
unflatten(), patch(), merge_patch(), begin()/end(), comparison
operators, erase(), and emplace().
Compiling the full test suite (tests/src/unit-*.cpp) with
-Wunused-result -Werror uncovered one real hit: a regression test in
unit-regression2.cpp called dump() purely to check it does not throw,
discarding the result. Fixed by explicitly casting to void, since the
call is intentionally result-less there.
Fixes#5410
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix discarded nodiscard results across the test suite for GCC's warn_unused_result
A plain (void) cast on a call expression suppresses the C++17 [[nodiscard]]
warning but not GCC's warning for functions annotated via the GNU
__attribute__((warn_unused_result)) form -- which is what
JSON_HEDLEY_WARN_UNUSED_RESULT expands to on GCC. Several existing tests
that call a newly-annotated function (dump(), empty()) purely to check
that it throws/does not throw, discarding the result via (void), newly
warned (and failed -Werror builds) once the annotation was broadened.
Route those discards through a small ignore_return_value() helper
instead, which actually consumes the value and suppresses the warning
on both attribute forms.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Use utils::ignore_return_value() for the issue #1445 dump() discard too
Addresses review feedback from @gregmarr on PR #5477: this call site was
still using the older "capture in a variable, then (void) it" pattern
from before this PR introduced utils::ignore_return_value(), instead of
the helper now used at every other discarded-nodiscard-result call site
this PR touches.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
accept() is a pure query whose only effect is the returned bool; both
parse() overloads and the deprecated accept(span_input_adapter&&, ...)
overload already carry JSON_HEDLEY_WARN_UNUSED_RESULT, but the two
current, recommended accept() overloads were missing it. Add the
annotation to match, so discarding accept()'s result now warns under
-Wunused-result / [[nodiscard]], as it already does for parse().
Fixes#5407
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Reject JSON Patch move when from is a proper prefix of path
RFC 6902 (section 4.4) forbids "from" from being a proper prefix of
"path" for a "move" operation: "a location cannot be moved into one
of its children." "move" is implemented as remove-then-add with no
check for this. For object targets, the subsequent "add" happened to
throw as a side effect of resolving through the now-removed parent,
but for array targets, removing the "from" element shifts subsequent
indices, so "path" silently re-resolves to a different element and
the operation "succeeds" with a silently corrupted document.
Add a check, before performing the remove/add, for whether "from" is
a proper prefix of "path" at the reference-token level. This compares
json_pointer's already-unescaped reference_tokens vectors (basic_json
is a friend of json_pointer) rather than the raw pointer strings, so
that tokens containing escaped '/' or '~' characters are compared
correctly, and a token that merely looks like a string prefix (e.g.
"/ab" vs "/abc/x") is not mistaken for a pointer-token prefix. When
"from" is a proper prefix of "path", throw out_of_range.414.
Fixes#5397.
Stacked on top of the fix for #5396 (branch
issue-5396-patch-remove-primitive-parent), since both touch the same
patch_inplace move/remove handling in include/nlohmann/json.hpp.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Add root-pointer and array-append-token edge case tests for the move prefix check
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Replace std::equal with an explicitly-bounded loop in the move prefix check
The three-iterator std::equal(first1, last1, first2) form has no
explicit end iterator for the second range, which a static analyzer
(Flawfinder, CWE-126) flags as a potential over-read even though the
preceding size comparison already guarantees the second range is long
enough. Rather than argue the point, make the bound visible in the code
itself via an explicit loop -- every access to ptr.reference_tokens is
now guarded by the same index the loop condition bounds against
from_size.
(The C++14 four-iterator std::equal(first1, last1, first2, last2) form
was tried first as a more minimal fix, but this codebase targets C++11
and that overload is not safely usable under -std=c++11 with all
supported standard library implementations.)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Extract the move prefix check into a named helper lambda
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Account for JSON_DIAGNOSTIC_POSITIONS in the move-prefix-check error messages
out_of_range::create() includes a "(bytes X-Y)" position annotation when
JSON_DIAGNOSTIC_POSITIONS is enabled, which the ci_test_diagnostic_positions
CI job builds the whole suite with. The five new out_of_range.414
assertions only checked the annotation-free message. Confirmed
JSON_DIAGNOSTICS produces the same (annotation-free) message as the
default build for this particular throw site (its path-based annotation
is empty at the root, where &result always points here), so only two
message variants are needed, not three.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
RFC 6902 (section 4.2) requires the target location of a "remove"
operation to exist. operation_remove handled parent.is_object() and
parent.is_array(), but had no final else branch: when the resolved
parent was a primitive value or null, neither branch matched and the
operation silently did nothing instead of failing.
Add the missing else branch, throwing out_of_range.413 with wording
that matches the existing out_of_range.411 thrown by the analogous
"add" case (operation_add) for the same kind of invalid parent.
Fixes#5396.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Route hand-rolled diagnostic pragmas through Hedley
Several places in the library hand-roll compiler diagnostic suppression
with raw `#pragma`/`#ifdef __GNUC__`/`#ifdef __clang__` guards instead of
using the Hedley primitives already bundled and used elsewhere
(JSON_HEDLEY_DIAGNOSTIC_PUSH/POP, JSON_HEDLEY_PRAGMA, ...). Converted six
of the seven listed push/pop pairs to use those primitives instead of
raw `#pragma GCC diagnostic`/`#pragma clang diagnostic` text:
- include/nlohmann/json.hpp (~3770, ~3863): -Wfloat-equal
- include/nlohmann/detail/conversions/to_chars.hpp (~1078): -Wfloat-equal
- include/nlohmann/detail/output/binary_writer.hpp (~1844): -Wfloat-equal
- include/nlohmann/detail/iterators/iteration_proxy.hpp (~211): -Wmismatched-tags
- include/nlohmann/detail/exceptions.hpp (~36): -Wweak-vtables
iteration_proxy.hpp did not previously include macro_scope.hpp itself
(it only compiled because some other header included earlier in
json.hpp happened to pull macro_scope.hpp in first); it now includes it
directly like the other detail headers that use Hedley macros, so it is
self-contained.
Each push/pop pair now uses JSON_HEDLEY_DIAGNOSTIC_PUSH/POP
unconditionally (a no-op on compilers that don't need it) and wraps the
actual `#pragma ... diagnostic ignored` text in JSON_HEDLEY_PRAGMA so it
goes through Hedley's _Pragma()-based emission instead of a raw #pragma
line, while keeping the original `#ifdef __GNUC__` / `#if
defined(__clang__)` guard around the ignored-pragma itself.
Deviation from the issue's suggested transformation: the issue's example
replaces the `#ifdef __GNUC__` guard with `#if
JSON_HEDLEY_HAS_WARNING("-Wfloat-equal")`. JSON_HEDLEY_HAS_WARNING is
implemented purely via Clang's `__has_warning` builtin and evaluates to
0 on real GCC (`#define JSON_HEDLEY_HAS_WARNING(warning) (0)` when
`__has_warning` is not defined), so adopting it verbatim would silently
stop suppressing -Wfloat-equal on GCC -- a real regression, not just a
style change. The existing `#ifdef __GNUC__` / `#if defined(__clang__)`
guards were kept for the ignored-pragma to stay behavior-preserving, and
only the push/pop/pragma-emission mechanism was routed through Hedley.
Two of the seven locations from the issue (the -Wignored-attributes
push at the very top of json.hpp and its matching pop after
`#include <nlohmann/detail/macro_unscope.hpp>`) were intentionally left
unconverted:
- The push, at the very top of json.hpp, runs before
`detail/macro_scope.hpp` (and therefore hedley.hpp) has been included
anywhere in the translation unit, so JSON_HEDLEY_DIAGNOSTIC_PUSH is not
yet defined at that point.
- The pop runs after `macro_unscope.hpp`, which -- via hedley_undef.hpp
-- has already #undef'd every JSON_HEDLEY_* macro (by design, see
#5408) precisely so they don't leak to users, so JSON_HEDLEY_DIAGNOSTIC_POP
is no longer defined by the time the pop is reached either.
Making this one pair work would require either hoisting the ~2000
line vendored hedley.hpp to the very top of the amalgamated single
header (a much bigger structural change to single_include than a pure
mechanism swap) or special-casing this one pop ahead of the general
macro cleanup. Both are riskier than the mechanical, behavior-preserving
change requested, so this pair was left as-is.
## Validation
- Compiled include/nlohmann/json.hpp and single_include/nlohmann/json.hpp
with `-Wall -Wextra -Wfloat-equal -Wmismatched-tags -Wweak-vtables`
(clang, which self-identifies as __GNUC__ too): no warnings, same as
before the change.
- Compiled and ran tests/src/unit-to_chars.cpp, unit-conversions.cpp,
unit-iterators1.cpp, unit-iterators2.cpp, and unit-class_parser.cpp
against the fixed include/: all pass.
- Compiled unit-msgpack.cpp, unit-bjdata.cpp, and unit-ubjson.cpp (which
exercise binary_writer.hpp's write_compact_float extensively): all
compile cleanly; the vast majority of assertions pass (the only
failures are pre-existing environment issues unrelated to this change
-- missing generated test-data files, not code correctness).
- Ran `make amalgamate`; the single_include diff is limited to exactly
the lines touched in include/, with no unrelated reordering.
- No real (non-Apple) GCC was available in this environment to test
directly; the `_Pragma("GCC diagnostic ...")` text emitted by
JSON_HEDLEY_PRAGMA is byte-identical to the prior `#pragma GCC
diagnostic ...` text, and the `#ifdef __GNUC__` guard is unchanged, so
GCC's behavior is expected to be identical. CI covers the GCC matrix.
This PR is stacked on top of #5475 (issue-5408-hedley-undef-leak) since
both touch the same files; only the last commit here is new.
Fixes#5409.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Guard JSON_HEDLEY_DIAGNOSTIC_PUSH/POP with the same compiler check as the pragma they bracket
Addresses review feedback from @gregmarr on PR #5485: the push/pop calls
were unconditional, so compilers other than the one the ignored-pragma
targets (e.g. MSVC, or GCC where the pair only applies under __clang__)
now did a needless push/pop with nothing suppressed in between. Move the
existing #ifdef __GNUC__ / #if defined(__clang__) guard to also cover the
push/pop, restoring the original zero-overhead behavior on other compilers
while still emitting the pragma itself through Hedley.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Undefine the four JSON_HEDLEY_* macros that leak after including json.hpp
include/nlohmann/detail/macro_unscope.hpp includes hedley_undef.hpp to
#undef every JSON_HEDLEY_* macro so none of them leak into the including
translation unit. Four macros were missing from that list and therefore
stayed defined after #include <nlohmann/json.hpp>:
- JSON_HEDLEY_PRAGMA
- JSON_HEDLEY_PREDICT_TRUE
- JSON_HEDLEY_PREDICT_FALSE
- JSON_HEDLEY_CLANG_HAS_DECLSPEC_ATTRIBUTE
hedley_undef.hpp is generated (via `make update_hedley`) by grepping
hedley.hpp for its own internal `#undef JSON_HEDLEY_X` redefinition
guards. JSON_HEDLEY_PRAGMA/PREDICT_TRUE/PREDICT_FALSE have no such guard
in upstream Hedley, so they were never picked up. The guard for
JSON_HEDLEY_CLANG_HAS_DECLSPEC_ATTRIBUTE also has an upstream typo
(`JSON_HEDLEY_CLANG_HAS_DECLSPEC_DECLSPEC_ATTRIBUTE`), so hedley_undef.hpp
was undefining the wrong (never-defined) name.
Fixes:
- include/nlohmann/thirdparty/hedley/hedley_undef.hpp: corrected the
DECLSPEC_ATTRIBUTE typo and added the three missing #undef lines,
keeping the file's alphabetical ordering.
- Makefile (update_hedley target): changed hedley_undef.hpp generation to
extract macro names directly from every `#define JSON_HEDLEY_...` in
hedley.hpp instead of from existing `#undef` guards, so a future
`make update_hedley` run undefines every macro Hedley actually defines,
even ones without a pre-existing redefinition guard. This was not run
in this PR (it would also pull in an unrelated upstream Hedley sync);
hedley_undef.hpp was hand-patched instead and single_include was
regenerated with `make amalgamate`.
- tests/src/unit-no-macro-leak.cpp: new regression test (picked up
automatically by tests/CMakeLists.txt's existing unit-*.cpp glob) that
includes json.hpp and then #ifdef/#error-checks every JSON_HEDLEY_*
macro name, so any future leak of any of the 151 vendored macros fails
the build, not just the four fixed here.
Fixes#5408.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Derive the JSON_HEDLEY_* leak-check test from hedley.hpp at build time
tests/src/unit-no-macro-leak.cpp previously hardcoded a static list of
~151 #ifdef/#error checks, one per JSON_HEDLEY_* macro name known at the
time it was written. That list would silently go stale the next time
`make update_hedley` pulls in a vendor update that adds, removes, or
renames a macro, since nothing would force it to be regenerated.
Add cmake/scripts/gen_hedley_undef_check.cmake, which derives the full
list of JSON_HEDLEY_* macro names directly from
include/nlohmann/thirdparty/hedley/hedley.hpp:
- tests/CMakeLists.txt uses it (MODE=checks) to (re)generate
hedley_undef_checks.inc at configure and build time, and wires the
generating custom target as a dependency of the test-no-macro-leak_cpp*
targets so it can never build against a stale copy. unit-no-macro-leak.cpp
now just #include-s the generated file inside its TEST_CASE instead of
carrying the checks itself.
- The Makefile's `update_hedley` target now delegates hedley_undef.hpp
generation to the same script (MODE=undef, new `update_hedley_undef`
target), so the vendored header, the generated #undef list, and the
generated test checks are all derived from the same extraction logic and
cannot drift apart.
This mirrors the approach taken independently in #5415 for the same
issue (#5408), credited there to a self-regenerating mechanism that
"can never drift again" -- ported into this branch instead of the
static list originally proposed here.
Verified with a local CMake configure + build + ctest, both against
include/ (JSON_MultipleHeaders=ON) and against the amalgamated
single_include/nlohmann/json.hpp (JSON_MultipleHeaders=OFF), and by
temporarily deleting a #undef line from hedley_undef.hpp to confirm the
generated test actually fails on a real leak.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix REUSE compliance failure in gen_hedley_undef_check.cmake
The generated file's embedded banner contains the literal text
'SPDX-License-Identifier: MIT' as part of the *content* being written
to hedley_undef.hpp, not as this .cmake script's own REUSE header (it
is already covered by the blanket 'Files: *' rule in .reuse/dep5).
The reuse tool matched that embedded line as an SPDX tag for the
script itself and failed to parse the trailing 'MIT\n")' as a valid
SPDX License Expression, breaking ci_reuse_compliance. Wrap the
embedded banner in REUSE-IgnoreStart/REUSE-IgnoreEnd comments, as
recommended by the tool's own diagnostic output.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Speed up whitespace skipping in the lexer
lexer::skip_whitespace() called get() for every whitespace byte, and
get() checks the (almost always false, once past the first character)
next_unget flag on every call. skip_whitespace() now reads its first
character with get() (needed to honor a pending unget() left over from
finishing the previous token, e.g. scan_number() always ungets the
character that terminated the number) and every further whitespace
character with a new get_ignoring_pending_unget() variant that skips
that branch, since nothing in the loop calls unget().
This is a narrower fix than the full contiguous-buffer bulk-skip
suggested in the issue (scan a run of whitespace directly in the
adapter's buffer and update position counters once per run). That
approach depends on bulk-scan adapter infrastructure
(supports_bulk_scan/bulk_data()/bulk_skip()) introduced by the open,
unmerged parser-performance PR #5283, which this change intentionally
does not depend on or replicate. Building new bulk-scan adapter
infrastructure from scratch was judged out of scope/riskier than
warranted here, so this change is limited to the safe, always-correct
improvement of removing redundant per-character bookkeeping from the
existing byte-at-a-time loop; full bulk-skipping is left as future
work once #5283 (or equivalent adapter support) lands.
Line/column/byte-offset bookkeeping is untouched and verified
bit-for-bit identical before and after this change, including for
pretty-printed (dump(4)) input with embedded newlines.
Fixes#5412
Stacked on top of the PR for #5411 (branch
issue-5411-lexer-skip-conversion).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix codegen regression in skip_whitespace() from #5490
Benchmarking found the get()/get_ignoring_pending_unget() split in
skip_whitespace() made long whitespace runs (e.g. indentation in
pretty-printed JSON) 1.75x-3.2x SLOWER instead of faster, reproducible
with both Apple Clang and GCC.
Root cause: rewriting the loop from a plain do-while into an initial
get() followed by a while-loop defeated the compiler's ability to keep
the input adapter's read/end pointers in registers across iterations;
both compilers instead reloaded them from memory on every character.
The function split itself was not the problem (it still fully
inlines); the loop's control-flow shape was.
The fix keeps the same two-function structure but restores a
do-while shape (guarded by an if for the "first char not whitespace"
case), which lets both compilers hoist the pointers back into
registers, matching or beating pre-#5490 performance.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Share the position-counter bump between get() and get_ignoring_pending_unget()
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Extract current_is_whitespace() to deduplicate skip_whitespace()'s two whitespace checks
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Use a raw string literal for the multi-line error-position test input
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix clang-tidy raw-string-literal finding and guard a new test against JSON_NOEXCEPTION
The issue #5412 whitespace-skipping test added a check_error() helper that
relies on catching json::parse_error to verify the exception message; under
JSON_NOEXCEPTION, JSON_THROW aborts instead of throwing, which crashed
ci_test_noexceptions (and cascaded into the other ci_cmake_options jobs).
Guard the whole section with #if !defined(JSON_NOEXCEPTION), matching the
existing pattern used by sibling tests in this file.
Also switch one escaped string literal to a raw string literal to satisfy
clang-tidy's modernize-raw-string-literal check.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Skip integer conversion in accept()/SAX validation when the value is unused
lexer::scan_number() always converted every numeric token with
strtoull()/strtoll() before returning, even though accept() (and any
consumer using json_sax_acceptor) immediately discards the converted
value. For value_unsigned/value_integer tokens whose digit count
already guarantees the value fits into 64 bits, the conversion cannot
change the accept/reject decision (such tokens are always finite and
unconditionally accepted), so scan_number() can skip strtoull()/
strtoll() entirely in that case when the caller signals it does not
need the value. Numbers with more digits keep using the exact,
unmodified conversion path, so overflow reclassification to
value_float (and the finiteness check on it) is unaffected.
parse() and value_float handling are completely unchanged.
Fixes#5411
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Document why the digit-count fast path is safe regardless of number_unsigned_t/number_integer_t width
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Avoid temporary-string concatenation flagged by clang-tidy in the differential test
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix to_bjdata() emitting unparsable output when _ArraySize_ is not an array
write_bjdata_ndarray() never checked that _ArraySize_ is an array. The shape
is written verbatim as the header length, so a null shape emitted 'Z' and an
object shape emitted '{' after the '#', neither of which from_bjdata()
accepts, and the round-trip guarantee in the BJData docs was broken.
Both slipped through the existing validation: for null, empty() is true so
the element count starts at 0 and the per-dimension loop never runs, and for
an object the loop walks its values, which can satisfy the non-negative
integer check. When _ArrayData_ then matched that count, the writer took the
ndarray path.
Require the shape to be an array, so anything else falls back to a plain
object encoding that round-trips, as the fallback rule in the docs already
specifies.
Signed-off-by: qatcod <79017227+qatcod@users.noreply.github.com>
* Document that _ArraySize_ must be an array in the ndarray requirements
The list at bjdata.md is the exhaustive set of conditions for the ndarray
encoding, but it only implied this one through 'every entry of'.
Signed-off-by: qatcod <79017227+qatcod@users.noreply.github.com>
---------
Signed-off-by: qatcod <79017227+qatcod@users.noreply.github.com>
parse_float_fast() (Clinger) needs a significand below 2^53, so it always
declines once the mantissa has 17 or more significant digits. convert_number()
called it unconditionally, so those numbers were walked an extra time before
strtod had to run anyway. On streaming input, where scanning is byte-at-a-time
and there is no compensating win, that made canada.json about 6% slower than
develop.
Derive the significant-digit count from token_buffer indices - the digits are
not scanned again - and skip the call when it is guaranteed to decline. Both
scanners pass the offset where the mantissa ends; the count only has to be
corrected for a leading "0", which the JSON grammar admits nowhere else. The
integer path returns before the check, so integer-heavy input is unaffected.
Values are unchanged: this only avoids an attempt that would have failed.
Verified bit-exact against develop over every number in canada.json,
floats.json, signed_ints.json, unsigned_ints.json, small_signed_ints.json,
citm_catalog.json and twitter.json, for both the contiguous and the streaming
scanner.
parse, streaming develop before after
canada.json 19.4ms 20.5ms 19.3ms
floats.json 135.9ms 131.8ms 128.0ms
parse, contiguous develop before after
canada.json 15.5ms 12.9ms 11.7ms
floats.json 98.6ms 69.8ms 66.7ms
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
simdutf.h rejects anything below C++17 with an #error, so defining
JSON_USE_SIMDUTF in a C++11 or C++14 translation unit did not fail with a
message about simdutf being unavailable - it failed to compile at all, taking
the library's C++11 support with it. Nothing caught this because no build ever
compiled that path.
Gate the include and both uses on JSON_HAS_CPP_17, the same way number_parse.hpp
gates std::from_chars. Below C++17 the macro now has no effect and the scalar
validator runs; it accepts and rejects exactly the same input, so the macro is
safe to set project-wide even when some translation units use an older standard.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
is_contiguous_byte_container accepted any type with a data() returning a
pointer to a single-byte integral plus a size(). That is duck typing: the two
members say nothing about size() counting the units data() points at.
A type where it does not - fixed-size records, say - was routed to the
pointer-based adapter and parsed as [data(), data() + size()) bytes, silently
truncating input the iterator-based adapter had read in full:
struct record_buffer {
using value_type = std::array<char, 4>;
std::string bytes;
const char* data() const; // raw bytes
std::size_t size() const; // in records
const char* begin() const; const char* end() const;
};
json::parse(record_buffer{"[1,2,3,4,5]"}); // parse error at column 3
Requiring the container's own value_type to be that same element type ties the
two together. Every contiguous standard container satisfies it, so std::string,
std::vector<char>, std::array<char, N> and std::string_view keep the fast path;
anything else falls back to the iterator-based adapter, which is always correct.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
scan_number_bulk_contiguous() converts an integer token straight from the input
buffer. When the value does not fit, it materializes token_buffer and calls
convert_number(), which tried the very same integer conversion again before
falling back to floating point.
Recording the outcome in number_type skips the second attempt. The resulting
token type and value are unchanged: convert_number() reached the float tail
either way.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
string_bulk_run() had the same `return scalar_string_bulk_run(...)` in both
arms of the `#if`. The simdutf arm already falls through when validation
fails, so a single return after the `#endif` says the same thing.
The JSON_USE_SIMDUTF example showed `#include <simdutf.h>`, which
string_scan.hpp already does under the same guard; users only have to put the
header on the include path and link the library, not include it themselves.
Set the version history entry to 3.13.0, matching the other macro pages
documenting unreleased features.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
@gregmarr spotted that convert_integer() was inserted between convert_number()
and its doc block, leaving convert_integer() with two stacked blocks and
convert_number() with none. Comment only; no code change.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The comment offered "a C++20 sentinel or counted_iterator" as examples of a
SentinelType, but std::counted_iterator is the IteratorType - the sentinel it
pairs with is std::default_sentinel_t. #5268 corrected the same wording in the
API documentation and left the code comment behind.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
GCC 7 sets __cplusplus to the C++17 value under -std=gnu++1z, so
JSON_HAS_CPP_17 is defined, but its libstdc++ ships no <charconv> header
(added in GCC 8; floating-point from_chars in GCC 11). The unconditional
"#if defined(JSON_HAS_CPP_17) #include <charconv>" therefore failed to
compile there: "fatal error: charconv: No such file or directory" in the
ci_test_compilers_gcc (7) job.
Wrap the include in __has_include(<charconv>), mirroring the library's
existing handling of <version> and <filesystem> in macro_scope.hpp. When
the header is absent, __cpp_lib_to_chars stays undefined and
parse_float_from_chars() takes its scalar fallback, so the from_chars use
site (already gated on __cpp_lib_to_chars) is never reached. GCC 8-10,
which have <charconv> but no floating-point from_chars, are unaffected:
they include the header but still take the fallback. GCC 11+ is unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The contiguous number fast path uses a Clinger-style exact algorithm
(significand * 10^scale in double arithmetic), which is only correctly
rounded when double operations are evaluated in true 53-bit precision.
On the x87 FPU used by 32-bit x86 (FLT_EVAL_METHOD == 2) the single
multiply/divide is computed in 80-bit and then double-rounded to double,
so a small fraction of values land 1 ULP off.
This surfaced as test-cbor_cpp11 and test-msgpack_cpp11 failing on the
mingw (x86) job for regression/floats.json: the C++17 builds pass because
they take the correctly-rounded std::from_chars path, while C++11 falls
back to parse_float_fast(). A 5M-sample check over shortest round-trip
decimals reproduces it: 0 divergences with 53-bit doubles, ~1 in 25 000
with 80-bit intermediates; declining to std::strtod fixes all of them.
Guard parse_float_fast() on FLT_EVAL_METHOD so it declines whenever the
platform evaluates doubles in extended precision, letting the caller use
the correctly-rounded std::from_chars / std::strtod path instead. On
mainstream x86-64/ARM64 (FLT_EVAL_METHOD == 0) the fast path is unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
libstdc++ 15 defines __cpp_lib_to_chars even in C++14 mode (via
bits/version.h pulled in by other headers), but <charconv> is only included
under JSON_HAS_CPP_17. That made parse_float_from_chars() reference
std::from_chars without the header in C++14 builds, breaking gcc-latest,
icpx, and the offline-testdata jobs.
Gate the use on JSON_HAS_CPP_17 && __cpp_lib_to_chars so it matches the
include condition exactly; C++11/14 always take the scalar fallback.
Verified by forcing __cpp_lib_to_chars in a C++14 build: the guard
suppresses std::from_chars and it compiles. C++17 behavior is unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The Clinger fast path is exact only for the "easy" subset (<=19 significant
digits, |exp10| <= 22); high-precision and scientific floats fall through to
strtod, where the failed Clinger attempt actually makes parsing a net loss.
std::from_chars implements the Eisel-Lemire algorithm in modern standard
libraries: locale-independent, correctly rounded, and fast over the whole
value range.
convert_number() now tries parse_float_from_chars() first (guarded by
__cpp_lib_to_chars, so C++11 and libc++-without-float-support keep the
Clinger + strtod path unchanged), then Clinger, then strtof. from_chars is
used only when it consumes the entire token; a partial parse means a non-'.'
locale decimal point, and an under-/overflow (result_out_of_range) also
declines - in both cases the existing strtod fallback supplies the exact
value and the well-defined +/-inf/0 the parser expects, side-stepping the
P4168 divergence between implementations. float and long double now get the
fast path too (Clinger was double-only).
Measured, C++17, g++ 13 -O3, json::parse/accept:
- canada-style floats: ~unchanged (Clinger already covered them)
- high-precision (17 digits): parse 2.1x, accept 2.5x
- scientific (17 digits + exp): parse 3.6x, accept 4.1x
Verified: C++11 (Clinger/strtod) and C++17 (from_chars) parse every value -
including subnormals, boundary values, and 1e9999/1e-9999 over-/underflow -
to bit-identical results; 2M number-fuzz clean; conversions/deserialization/
locale/number-fast-path suites pass in both C++11 and C++17; clang-tidy
clean; warning-clean on g++ and clang in C++11/17/20.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The CI clang-tidy (newer than the locally available version) reported two
additional checks on the new code:
- readability-math-missing-parentheses: parenthesize the (a * b) + c digit
accumulations in number_parse.hpp.
- cppcoreguidelines-missing-std-forward: the contiguous-byte-container
input_adapter overload took a forwarding reference but only reads
data()/size() and never forwards it. It is already disjoint from the
generic container overload via SFINAE, so a plain const& is correct and
clearer (and keeps the container alive for the whole parse just as before).
No behavior change; char_type and routing are unchanged (std::string and
std::vector<std::uint8_t> still take the pointer adapter with char/uint8_t
char_type), CBOR/MsgPack round-trips and the 2M number fuzz still pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The contiguous number fast path materialized token_buffer with
token_buffer.assign(data, len), but string_t is only required to provide
the minimal interface the rest of the lexer uses (push_back, append,
clear, operator[], ...). Custom string types such as the test's alt_string
do not implement assign(), so scan_number_bulk_contiguous() failed to
compile for them (unit-alt-string), breaking the gcc/clang standards and
old-compiler CI jobs.
reset() already clears token_buffer, so fill it with append() - which
alt_string and std::string both provide and which the string fast path
already relies on - instead of assign().
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
- number_parse.hpp: use std::array for the powers-of-ten table
(avoid-c-arrays) and `auto` for the cast-initialized result
(modernize-use-auto), matching the codebase style (cf. the serializer's
utf8d table). Indexing casts keep the -Wsign-conversion build clean.
- add JSON_USE_SIMDUTF to the mkdocs navigation so the macro page is
reachable.
No behavior change; clang-tidy is clean on the new headers and the
amalgamation is regenerated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
lexer.hpp had grown by ~600 lines of byte-level helpers that have no
dependency on the lexer's template parameters and clutter the state
machine. Move them, unchanged, into two focused headers as free functions
in namespace detail:
- number_parse.hpp: parse_integer_unsigned/parse_integer_signed (now
templated on the number type) and parse_float_fast (Clinger's exact
double fast path, with the decimal point passed as an argument instead of
read from a lexer member).
- string_scan.hpp: the SWAR string helpers (is_string_special,
swar_string_special, find_string_special, validate_one_utf8,
scalar_string_bulk_run) and the backend-dispatched string_bulk_run,
including the optional simdutf include and find_string_delimiter.
lexer.hpp now includes these and calls the free functions; the methods that
touch lexer state (scan_string, scan_number, scan_string_bulk,
scan_number_bulk_contiguous, convert_number) stay put. This is a pure code
move with no behavior change: lexer.hpp drops from 2357 to 1934 lines, the
now-unused <cstdint>/<cstring>/<limits> includes are removed, and the
free-function form makes the SWAR helpers reusable elsewhere (e.g. the
serializer's string escaping).
Verified: default and JSON_USE_SIMDUTF builds compile; 2,000,000 number and
2,000,000 arbitrary-byte-string differential-fuzz documents parse
identically to before; lexer/parser/conversions/deserialization/locale/
diagnostic-position suites pass (20,576 assertions); warning-clean on g++
and clang in C++11/17/20; the amalgamation regenerates and passes
check-amalgamation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
scan_number() reads a number one character at a time through the input
adapter (get()) and appends each byte to token_buffer (add()) before
converting. For contiguous input, the per-character get()/add() overhead
dominates: it is roughly two thirds of the time spent on number-heavy
parsing, far more than the value conversion itself.
Add scan_number_bulk_contiguous(), which parses the whole number token
straight from the input buffer: it validates and classifies the extent
with the same grammar as scan_number()'s state machine, materializes
token_buffer in one copy (substituting the locale decimal point exactly as
scan_number() does), advances the adapter, and reuses the shared
convert_number() tail. On anything it does not recognize as a well-formed
number it makes no state change and returns token_type::uninitialized, so
the caller falls back to scan_number(), which then produces the exact
diagnostic. Errors and their positions are therefore unchanged.
The conversion tail is factored out of scan_number() into convert_number()
so both scanners share it; the fast path is selected by tag dispatch on the
existing bulk_scan capability, so streaming/wide/user adapters are
unaffected.
Measured on pointer input, g++ 13 -O3:
- integers: parse +65%, accept +98%
- floats: parse +39%, accept +70%
Verified: 2,000,000 randomized number documents (including overflow-range
integers, long digit strings and %.17g doubles) parse identically via the
contiguous path and the streaming byte path, matching value, type and
round-trip text; the locale suite and existing parser/lexer/conversions/
deserialization tests pass; a new "lexer number fast path" test checks
contiguous-vs-streaming parity, token classification, and that malformed
numbers are rejected identically. Pure C++11, no intrinsics.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The bulk string scanner validates UTF-8 straight from a contiguous buffer.
The scalar validator caps at ~0.3-0.7 GB/s on non-ASCII text; a SIMD
validator reaches several GB/s. Rather than hand-rolling SIMD UTF-8
validation (easy to get subtly wrong - a from-scratch SSE attempt rejected
valid CJK), wire in the vetted simdutf library behind an opt-in switch.
simdutf is not header-only (it ships simdutf.cpp and uses runtime CPU
dispatch), so it is not vendored: defining JSON_USE_SIMDUTF includes
<simdutf.h> and routes the bulk validator through simdutf::validate_utf8;
the project supplies and links simdutf. Undefined (the default), nothing
external is included and the portable C++11 scalar path is used, so the
library stays header-only and its baseline behavior is unchanged.
Design keeps behavior identical either way:
- scan_string_bulk() now finds the run up to the next quote/escape/control
byte (non-ASCII allowed) and validates it in one shot; on the rare
validation failure it recomputes the exact valid prefix with the scalar
helper, so ill-formed input still falls through to the byte path and is
reported at the same position with the same message.
- the per-sequence scalar path is factored into scalar_string_bulk_run()
and is the default backend; the refactor is behavior-preserving and does
not change scalar throughput.
Verified: default and JSON_USE_SIMDUTF builds accept/reject/parse
identically across 2,000,000 arbitrary-byte documents and 1,000,000
mixed-escape/UTF-8 documents (differential fuzz vs the streaming byte
path); lexer/parser/diagnostic-position/deserialization suites pass under
both configurations (20,188 assertions with the backend enabled);
warning-clean on g++ and clang, C++11 and C++20, both configurations.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
json::parse(std::string) - the most common entry point - did not benefit
from the contiguous fast paths (bulk string scanning, UTF-8 bulk
validation, memcpy for binary formats) in C++11..17: std::string::iterator
is a library wrapper, not a raw pointer, and pre-C++20 there is no portable
way to prove it contiguous, so supports_bulk_scan was false. Only raw
pointers, string literals, and C-arrays (and, in C++20, anything modelling
std::contiguous_iterator) took the fast path.
Detect contiguous single-byte containers (std::string, std::vector<char>,
std::vector<std::uint8_t>, std::string_view, ...) via is_contiguous_byte_
container and route them through an iterator_input_adapter built from
data()/data()+size(). The generic iterator-based container overload is
constrained to exclude these, so the two overloads are disjoint and there
is no ambiguity (a plain competing overload loses to the greedy
forwarding-reference container overload on reference binding, and a factory
partial-specialization is ambiguous - both were tried and rejected).
The pointer keeps the container's own element type, so char_type - and
therefore all parsing behavior - is byte-for-byte identical to the iterator
path (const char* for std::string, const std::uint8_t* for
std::vector<std::uint8_t>); only the raw pointer additionally turns on the
fast paths. Lifetimes are unchanged: the container outlives the adapter for
the full parse expression, exactly as the iterators it replaces did.
Measured, C++11, json::parse/accept(std::string), g++ 13:
long ASCII strings: accept 201 -> 3200 MB/s (~16x), parse 174 -> 1444
dense CJK: accept 263 -> 697 MB/s (~2.6x)
short strings: accept 163 -> 243 MB/s (~1.5x)
Verified: char_type preserved for std::string (char) and
std::vector<std::uint8_t> (uint8_t); CBOR/MsgPack round-trips from
std::vector<std::uint8_t> unchanged; 1,000,000 randomized documents accept
and parse identically via std::string and via std::istream;
deserialization/user-defined-input/parser/lexer/conversions/diagnostic-
position suites pass (20,480 assertions); warning-clean on g++ and clang in
C++11/17/20.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The SWAR bulk string path stopped at the first non-ASCII byte and handed
every multibyte character to the byte-at-a-time scanner, whose per-byte
get()/next_byte_in_range()/add() machinery runs at roughly half the speed
of validating straight from the buffer. As a result, dense non-ASCII text
(CJK, emoji, accented Latin) parsed ~10-15x slower than ASCII.
Fold well-formed UTF-8 into the bulk run: scan_string_bulk() now, on a
non-ASCII lead byte, validates one sequence with validate_one_utf8() -
which mirrors scan_string()'s per-byte switch ranges exactly (rejecting
overlong forms, surrogates, and out-of-range code points) - and appends it
in place, continuing until the closing quote, an escape, a control byte,
or an ill-formed sequence. All error handling still defers to the byte
path, so error messages and positions are byte-for-byte unchanged.
Because only well-formed content is fast-pathed and every rejection falls
through to the existing scanner, behavior is identical; the win is purely
throughput. Measured on pointer input (accept, string values discarded):
content g++ 13 clang 18
dense CJK 277 -> 648 ~605 MB/s (~2.3x)
dense emoji 299 -> 857 ~702 MB/s (~2.6-2.9x)
mixed 90% ASCII 246 -> 331 ~334 MB/s (~1.35x)
pure ASCII unchanged (~3.2 / 4.1 GB/s)
Verified: 2,000,000 randomized documents built from arbitrary bytes
(overlong, surrogate, truncated, out-of-range sequences) accept/reject and
parse identically via the contiguous path and the streaming byte path;
lexer/parser/diagnostic-position/deserialization/conversions suites pass
unchanged. Pure C++11, no intrinsics.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
scan_string() read the input one character at a time through the input
adapter and classified every byte with a large switch. For contiguous
byte buffers we can instead scan 8 bytes at a time with a SWAR word test
that finds the first byte needing individual handling (the closing quote,
an escape, a control character, or a non-ASCII UTF-8 byte) and bulk-append
the ordinary run in one go.
- input adapters expose supports_bulk_scan / bulk_data / bulk_remaining /
bulk_skip for provably-contiguous, same-type, 1-byte iterator ranges
(raw pointers in every standard; std::string/std::vector/std::array and
friends additionally in C++20 via std::contiguous_iterator).
- the lexer gains a bulk_scan capability (gated on lazy_token_string so
bypassing the per-character capture cannot lose error diagnostics) and a
scan_string_bulk() fast path; streaming/wide/user adapters are unchanged
and keep the byte-at-a-time scanner.
The run contains no newline (all bytes < 0x20 are treated as special), so
position bookkeeping stays exact, and error tokens are still reconstructed
lazily from the consumed byte range. The SWAR special-byte test is pure
uint64_t arithmetic - no intrinsics, no runtime dispatch, C++11-clean.
Measured on representative data, pointer input, g++ 13 -O3
(string values discarded by accept() see the largest gains):
long ASCII strings: DOM +4.5x, SAX +14x, accept +17x (to ~2 GB/s)
short strings: DOM +15%, SAX +62%, accept +85%
escape-heavy: DOM +31%, SAX +26%, accept +28%
Same-input parity verified: 200k randomized documents (escapes, multibyte
UTF-8, surrogate pairs) accept/parse identically via the contiguous SWAR
path and the streaming byte path; unit lexer/parser/diagnostic-position/
deserialization/conversions suites pass unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The number scanner converted its already-validated digit buffer with
std::strtoull/std::strtoll/std::strtod. Those pull in locale and errno
machinery and dominate number-heavy parsing (strtod runs at ~6 M/s).
Replace them with dedicated parsers over the validated buffer:
- parse_integer_unsigned / parse_integer_signed: accumulate digits with
overflow detection, falling back to the float path on overflow exactly
as the strtoull/strtoll round-trip check did. Overflow behavior is
unchanged for narrower or wider custom number types.
- parse_float_fast: Clinger's exact fast path for `double` (<=19 significant
digits, |exp10| <= 22, significand < 2^53), where significand * 10^exp is
exact under IEEE round-to-nearest. This is the same fast path used by
fast_float/simdjson. It is bit-identical to strtod on this subset and
declines (falling back to strtod) otherwise. Only `double` uses it; float
and long double keep std::strtof/std::strtold via a templated overload.
Measured on representative data (g++ 13, -O3):
- integers: DOM parse +11%, SAX +25-34%
- floats: DOM parse +37%, SAX +70% (clang: float DOM ~1.9x)
No dependencies added; header-only and C++11-clean. Existing parser,
lexer, conversion and deserialization unit tests pass unchanged; a
3M-value random-double fuzz matches strtod bit-for-bit.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The mixed number arms of JSON_IMPLEMENT_OPERATOR cast the integer to
number_float_t before comparing. Past the float's mantissa that cast is lossy:
2^63-2 and 2^63-1 both round to 2^63, so each compares equal to that float
while differing from each other. Equality is therefore intransitive and the
ordering is not a strict weak ordering, which makes std::sort over such values,
or using them as keys in std::set or std::map, undefined behavior.
Compare the two exactly instead. The integer's range is a power of two the
float represents exactly, so a float outside it is ordered by magnitude alone;
inside it, truncating the float is exact, and the integer parts and then any
fractional part decide. The helper hands back a pair whose comparison with the
original operator reproduces that ordering, which keeps every operator's return
type as it was, including partial_ordering for the spaceship.
A NaN operand is returned in both members, so NaN stays false for the
relational operators and unordered for <=>. Values a float represents exactly
still compare equal, so json(1) == json(1.0) is unchanged.
Signed-off-by: qatcod <79017227+qatcod@users.noreply.github.com>
When merge_objects is true, recurse only if the existing value is an
object. Otherwise overwrite, matching the documented "all other values
are overwritten as usual" behavior.
Fixes#5402
Signed-off-by: elix3r <157088510+22elix3r@users.noreply.github.com>
Niels Lohmann
Qatadaha Bin Matloob
Avionic Harshit
elix3r