Commit Graph
928 Commits
Author SHA1 Message Date
Niels Lohmann cf0c72cd95 Merge branch 'claude/implement-5891-deff16' into claude/from-json-arith-array
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-24 07:20:56 +02:00
Niels Lohmann 6ee6351361 Address review: throw helpers take input_format_t, simpler type checks
- throw_on_discarded() and throw_integer_out_of_range() take an
  input_format_t and call input_format_name() themselves
- has_no_children(), set_parents() and max_size() check array, then
  object, instead of calling is_primitive() first
- parse_impl() chains like accept_impl(); sax_parse_impl() uses if/else
- std::array for the control character messages; negation<> in
  write_object_key(); clarify the is_primitive() comment

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-24 07:16:52 +02:00
Niels Lohmann 80c5fe8a5a Merge branch 'claude/implement-5891-deff16' into claude/from-json-arith-array
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-23 22:16:44 +02:00
Niels Lohmann f8df3df021 Fix CI failures: constexpr, NRVO, cppcheck, IWYU (#5891)
- json_sax_dom_error_state::is_errored() cannot be constexpr: the class
  is not a literal type (GCC 4.8, GCC -Wpedantic, Clang 3.5).
- to_binary_vector: destroy the vector writer before returning, as the
  output_vector_sink sets the size of the vector in its destructor.
  Without NRVO (MSVC Debug in C++14 mode), the vector was moved out
  before it was resized.
- parser::sax_parse: return false instead of result after the early exit
  (cppcheck identicalConditionAfterEarlyExit).
- json.hpp: export detail/exceptions.hpp for IWYU, so json_view.hpp can
  name exception_id without including a detail header.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-23 22:16:26 +02:00
Niels Lohmann 21bc758b89 from_json: number-first dispatch for arithmetic types, loop for std::array (#5891)
from_json(ArithmeticType&) now reads numbers through get_number_value() and
handles a boolean on a cold path, which puts the three number types first in
the per-element type dispatch.

get<std::array<T, N>>() always took the identity_tag overload, a pack
expansion of N calls j.at(i).get<T>() that clang unrolls into straight-line
code (about 28,000 instructions for N = 1000), so its speed depended on
inlining decisions in that body. For trivial element types, which cannot tell
default construction plus assignment from construction in place, it now fills
a std::array in a loop. That loop, shared with get_to(), checks the size once
and reads the inner array directly; a short array still converts the elements
it has and then throws the out_of_range error at() would throw.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-23 16:00:12 +02:00
Niels Lohmann fc347b5779 1.5: keep get_bon8_string's own UTF-8 ranges (#5891)
Taking the second-byte range from utf8_shape() made BON8 decoding 2-3% slower
(FromBinaryShape/scalars, FromBinaryBuffer, alignment-pinned builds), so the byte
path keeps its hand-written ranges. The exhaustive validator test still checks
that it accepts the same sequences as the other validators.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-23 12:18:56 +02:00
Niels Lohmann fce5a54722 2.5.7: force-inline detail::parse_array_index (#5891)
Moving parse_array_index out of json_pointer stopped clang from inlining it
into the pointer resolution (JsonPointerAt/citm_catalog +17%,
JsonPointerAt/twitter +8%); JSON_HEDLEY_ALWAYS_INLINE restores the baseline.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-23 12:07:53 +02:00
Niels Lohmann 96b5c82496 2.4.5: keep the inlined number cases in from_json(ArithmeticType&) (#5891)
Calling get_number_value() from from_json(ArithmeticType&) made
GetArray<std::array<int, 1000>> 2.4x slower (it changes the inlining of the
std::array conversion), so that overload keeps its own switch, with a comment.
get_arithmetic_value() still uses the shared helper.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-23 12:00:39 +02:00
Niels Lohmann 7506a1b0bf 2.5.7: share parse_array_index between json_pointer and json_view (#5891)
Move json_pointer::parse_array_index() and its status enum to namespace detail
(json_pointer keeps a forwarding member), and let the json_view pointer
resolution use it instead of its own copy of the RFC 6901 index grammar. Both
accepted the same tokens with the same outcome. Also update the comment on
detail::decode(), which listed four of the five UTF-8 validators and now points
to the test that keeps them in sync.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-23 10:33:52 +02:00
Niels Lohmann ae59fbb5a4 2.3.8: finish_input helper for the strict-EOF block in the parser (#5891)
sax_parse keeps its if (result) guard and return value. Also adds the
clang-tidy NOLINT for the 2.3.6 message table.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-23 10:30:35 +02:00
Niels Lohmann bd7878f899 2.3.6: table of control-character messages in the lexer (#5891)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-23 10:30:34 +02:00
Niels Lohmann 0092111f4d 2.3.5: json_sax_dom_error_state base for the DOM SAX parsers (#5891)
Holds errored, allow_exceptions, parse_error() and is_errored(). The scalar
handlers stay in each class (parse hot path).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-23 10:30:34 +02:00
Niels Lohmann c533a107a1 2.3.7, 2.3.9: read_object_key and leave_container_and_check for CBOR/MessagePack/BON8 (#5891)
UBJSON keys (different getter signature) and its container tail stay as is.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-23 10:30:33 +02:00
Niels Lohmann d556800512 2.3.3, 2.3.4, 2.3.10: unsupported_key_type, empty_input_message, input_format_name in the readers (#5891)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-23 10:30:32 +02:00
Niels Lohmann 0721d3c389 2.3.2: get_string_with_length for the plain-order string sites (#5891)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-23 10:30:31 +02:00
Niels Lohmann daf104b549 2.3.1: get_cbor_sized_bytes for the CBOR chunk getters (#5891)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-23 10:30:30 +02:00
Niels Lohmann 643acf23e2 1.5: use utf8_shape in get_bon8_string, add exhaustive UTF-8 validator test (#5891)
get_bon8_string takes the valid range of the second byte from utf8_shape()
(0xF5..0xF7 stay errors, unget behavior unchanged). validate_one_utf8 keeps its
if-chain (hot bulk path); the new test checks decode, validate_one_utf8,
utf8_shape, json::accept and the BON8 reader against each other for every lead
and second byte.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-23 10:30:29 +02:00
Niels Lohmann df98274f6c 1.2/1.3/1.5/2.5.1/2.5.2/2.5.5: share helpers in the view code (#5891)
- detail::exception_id at every numeric id; throw_* helpers take it
- hex_byte, hex4_lower, utf8_shape replace local hex/table code
- throw_array_index_out_of_range/_past_end, throw_key_not_found
- has_native_float_format replaces the hand-written IEEE check
No change of messages, ids or output.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-23 10:29:27 +02:00
Niels Lohmann f503b67ac7 1.7: json_pointer: flatten walk drops the primitive case list (#5891)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-23 10:29:26 +02:00
Niels Lohmann 727911bc80 2.4.5/2.4.6: from_json: share get_number_value and reserve_if_possible (#5891)
The three number cases of get_arithmetic_value and the arithmetic
from_json share one helper (boolean stays accepted only by the latter);
the array and object reserve overload pairs become one.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-23 10:29:24 +02:00
Niels Lohmann 210a398ce8 1.7/2.4.9: iter_impl, iteration_proxy: drop primitive case lists, share compare preamble (#5891)
Object and array stay the first branches; everything else is the
primitive case. operator== and operator< share the different-container
check.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-23 10:29:23 +02:00
Niels Lohmann f266ce92e5 2.4.2/2.4.3/1.7: json_pointer: share slash and escape checks, drop primitive case lists (#5891)
The leading-slash and invalid '~' escape checks are shared helpers; the
reference-token walks test object and array first and treat everything
else as primitive.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-23 10:29:22 +02:00
Niels Lohmann 92e2e84722 2.4.1/1.7: exceptions: share what_impl, flatten diagnostics switch (#5891)
The four create() bodies build their message through one protected
static helper; the diagnostics parent walk handles array/object and
ignores all other types.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-23 10:29:21 +02:00
Niels Lohmann 8262acca0e serializer: constexpr ascii table, hex4_lower, dedupe helpers (#5891)
1.1: build ascii_copyable() at compile time from is_ascii_copyable and
test find_ascii_copyable_run against it for every byte. 1.2/2.5.4: use
detail::hex4_lower in both \uXXXX writers. 2.1.6: factor the repeated
flush/emit blocks in dump_escaped_impl into always-inline helpers.
2.1.7: comment why the escape copy does not use put_buffer.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-23 10:29:13 +02:00
Niels Lohmann 4b3fcd0740 2.3.10: writer-side format names from input_format_name (#5891)
The CBOR/MessagePack/UBJSON/BSON/BJData/BON8 literals in throw_on_discarded and
throw_integer_out_of_range calls now come from detail::input_format_name; the
texts are identical, and UBJSON vs BJData still follows use_bjdata.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-23 10:29:12 +02:00
Niels Lohmann 35930392d8 2.1.5: shared object-key writer for CBOR and MessagePack (#5891)
write_cbor_key and write_msgpack_key were the same SFINAE pair apart from the
string/scalar writers; both now delegate to write_object_key<Walk>. UBJSON key
and to_format_length stay as they are.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-23 10:29:11 +02:00
Niels Lohmann f8d6aa95eb 2.1.4: write_bytes helper for the byte-string write sites (#5891)
The 13 oa.write_characters(reinterpret_cast<const CharType*>(x.data()), n)
sites now call write_bytes(const void*, size_t); copy_characters sites and the
buffer-based writes are untouched.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-23 10:29:10 +02:00
Niels Lohmann 48bd677617 2.1.3, 2.1.8: one UBJSON container start helper, shared frame base (#5891)
write_ubjson_start_array/_object are now write_ubjson_start_container<IsArray>,
keeping the array-only begin()+1 scan and excessive_valueless rule.
ubjson_frame derives from binary_container_frame instead of duplicating it.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-23 10:29:09 +02:00
Niels Lohmann 379bf8eddb 2.1.2: one iterative walker for CBOR and MessagePack (#5891)
write_cbor_iterative and write_msgpack_iterative (and their *_value_or_push
helpers) were identical but for the head, scalar, and key writers. They are now
one template over a small policy struct, so everything inlines. BON8 keeps its
own walker (open string run, closing marker), noted in a comment.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-23 10:29:09 +02:00
Niels Lohmann c9c9b90f1f 2.1.1: one MessagePack container prefix helper (#5891)
Array and object prefixes differed only in the fix marker and the 16/32-bit
markers; both now delegate to write_msgpack_container_prefix. Output is
byte-identical.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-23 10:29:08 +02:00
Niels Lohmann 058d6fd23d 1.4: BSON writers share one write-pass UTF-8 sanitizer (#5891)
write_bson_entry_header and write_bson_string re-derived the error-handler
policy differently. Both now use sanitize_utf8_for_bson_write, which treats
strict like keep (the size pass already validated) and sanitizes replace/ignore
into the bytes the size pass measured. No context plumbing is needed.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-23 10:29:07 +02:00
Niels Lohmann 25e7edad54 Add shared detail helpers for the #5891 cleanup
Add detail::is_primitive(value_t), detail::hex4_lower(), detail::utf8_shape(),
detail::input_format_name(), detail::empty_input_message(), the exception ids
116, 319, 320 and 416 to detail::exception_id, and make
detail::is_ascii_copyable() constexpr. Later commits switch the duplicated
code over to them.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-23 10:10:45 +02:00
Niels Lohmann 0b924f5375 Split and print JSON pointers in one pass (#5887)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-23 06:53:15 +02:00
Niels Lohmann 6fa14d8f95 Write binary formats through reserved sink space and number runs (#5885)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-23 06:52:58 +02:00
Niels Lohmann dec0e9b749 Write strings, scalars and indentation in dump() with fewer calls (#5883)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-23 06:52:41 +02:00
Niels Lohmann 86d5c56c54 Speed up flatten(), unflatten(), diff() and patch() further (#5886)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-23 06:48:12 +02:00
Niels Lohmann aa0b371f7c Faster std::istream input: inline get-area path, binary reads, unbuffered streambufs (#5884)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-23 06:48:00 +02:00
Niels Lohmann 9855db81f3 Silence macro-to-enum for the endianness macros (#5881)
#5876 added JSON_LITTLE_ENDIAN_TARGET and JSON_BIG_ENDIAN_TARGET to
macro_scope.hpp. They must stay macros because they select code in #if,
so suppress the clang-tidy check as simd_scan.hpp already does.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-22 08:40:05 +02:00
Niels Lohmann 9bf531436f Speed up flatten(), unflatten(), diff() and patch() (#5861)
* Flatten into a std::map by inserting next to the previous entry

flatten() added every path with operator[], so each insertion into the
std::map of the result searched the tree from the root and compared long
shared path prefixes, which took half the time on canada.json. Paths come
out in nearly sorted order, so they are now inserted with emplace_hint
next to the entry added last, for object types that have it. ordered_map
keeps its append without lookup, and other object types keep
operator[].

Path segments are also appended in place, without a temporary string
for each escaped key or array index.

flatten() of canada.json, citm_catalog.json and twitter.json is 1.4x to
1.9x as fast; ordered_json is unchanged (1.05x). The output is
unchanged.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Unflatten without splitting keys or walking each path from the root

unflatten() split every key into a vector of reference tokens, stored the
pointer prefixes in a tree to decide which values become arrays, and then
looked up every token of every key from the root again.

Keys are now checked by a scan that raises the same parse_error.107/108 as
split(), skipping the part a key shares with the previous one, which was
checked already. The prefixes that denote arrays are kept in a small hash
set of key slices; the array/object decision is unchanged. The walk keeps
the values along the previous key's path, so only the part of a key that
differs from the previous one is looked up. Keys of a flattened object are
usually sorted, so that part is short.

unflatten() of canada.json, citm_catalog.json and twitter.json is 1.9x to
2.3x as fast; the output and the exceptions are unchanged.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Apply plain JSON Patch operations without building a json_pointer

patch() built a json_pointer for every operation, which splits the path
into a vector of strings, and for "add" and "remove" a second one for
the parent. That took 20% to 45% of the time not spent copying the
document.

"add", "remove", "replace" and "test" with a path that has no escapes and
denotes existing values (the parent, for "add"), as diff() creates them,
are now applied by resolving the path in place, reusing one string for
the reference tokens. Every other operation, and every operation that
would fail, takes the existing code, so the results, the exceptions and
the state left after a failing operation are unchanged.

patch() of the diffs of canada.json, citm_catalog.json and twitter.json
is 1.02x to 1.10x as fast; the rest is the copy of the document.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Build diff operations member by member

diff() created every operation from an initializer list, so each
{"op", "replace"} pair became a temporary array with two heap strings
before it was turned into an object member; an operation took about 25
allocations. The operations are now built as objects whose members are
assigned one by one. Paths of members and elements are also built in
place, without a temporary string for each escaped key or array index.

diff() of canada.json, citm_catalog.json and twitter.json, with every
eighth scalar changed, is 1.3x to 1.6x as fast; the output is unchanged.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Diff objects with the same keys in lockstep and keep one path buffer

diff() looked up every key of the source object in the target object,
and built a new string for the path of every value it descended into.
Objects of the same size usually have the same keys in the same order,
so they are now walked together as long as that holds, without a
lookup; objects that differ fall back to the lookup from the first key
that does not match. The path is one buffer that is extended before
descending and shortened again afterwards. Objects whose members are
equal but in a different order (possible with ordered_json) still get
no operations.

Together with the previous commit, diff() of canada.json,
citm_catalog.json and twitter.json is 1.5x to 2.0x as fast; ordered_json
1.65x to 1.85x. The output is unchanged.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Use std::array for the index buffer in append_index()

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Declare the partner iterator in the loop initializer in diff_object_keys()

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Fix GCC and clang-tidy warnings in the JSON pointer speedups

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-21 21:27:33 +02:00
Niels Lohmann 3e031b06a5 Use standard library helpers where available, with C++11 fallbacks (#5876)
* Use standard library helpers where available, with C++11 fallbacks

Add detail helpers that call the standard library function when the
library provides it (checked with the __cpp_lib_* feature-test macros, so
no new ABI-tag macros) and a C++11 implementation otherwise:

- detail::clamp (std::clamp): by value, so that static constexpr members
  need no out-of-line definition before C++17; used in json_view's dump()
  buffer sizing, where step() can now be a static constexpr member.
- detail::bit_cast (std::bit_cast): replaces dtoa_impl::reinterpret_bits
  and the memcpy bit copies in the number parser, to_chars, the json_view
  serializer, and binary_writer::to_char_type.
- detail::byteswap (std::byteswap): one implementation instead of the
  binary writer's own byte_swap() overloads and the bswap chain in
  to_chars' store_msb_first().
- count_leading_zeros/count_trailing_zeros: two 32-bit _BitScanReverse/
  _BitScanForward calls for MSVC x86/ARM32 instead of the loop, and
  std::countl_zero/countr_zero for compilers without builtins.
- conjunction, disjunction, negation, bool_constant, and uncvref_t map to
  their std counterparts in C++17/C++20.

Builtins and intrinsics stay ahead of the <bit> functions, as MSVC's
std::countl_zero checks for LZCNT at run time unless AVX2 is enabled.

The generated code is unchanged with Clang (C++11/20/23, arm64 and
x86-64) and GCC 16 for parsing, dumping, and the binary formats. The
binary reader keeps its own byte_swap() and the writer its memcpy calls:
the smaller code from detail::byteswap/std::bit_cast there made GCC 16
inline get_number()/write_number() differently, which slowed reading by
10-18 % and writing by about 1 %.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Name the target byte order once (JSON_LITTLE_ENDIAN_TARGET, JSON_BIG_ENDIAN_TARGET)

to_chars.hpp and bit_ops.hpp repeated the long __BYTE_ORDER__/_MSC_VER
conditions; macro_scope.hpp now defines them once (undefined again in
macro_unscope.hpp).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Explain why the serializer growth bound is not a clamp

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-21 20:14:59 +02:00
Niels Lohmann 7c26e99ac5 Copy ASCII strings in the view's dump with ensure_ascii (#5879)
With ensure_ascii, the view's dump wrote every string through the escaping
path, which decodes UTF-8 byte by byte, even strings of the source that are
plain ASCII. A string of the source without escape sequences has no quote,
backslash, or control character, so ensure_ascii leaves it unchanged
exactly when it has no byte >= 0x7F. The dump now checks that eight bytes
at a time (one add and one or per word) and copies such strings like the
dump without ensure_ascii.

The compact walk (dump_compact) now also serves ensure_ascii, through a
separate out-of-line instantiation, so the walk without ensure_ascii keeps
its speed.

Parsing is unchanged. A flag set by the parser would let the dump skip the
check and was about twice as fast for compact output, but it costs every
parse of string-heavy input 1-2% with GCC.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-21 20:01:38 +02:00
Niels Lohmann 6aefd8c662 Copy short printable-ASCII strings without starting the scanner (#5878)
dump() now checks strings of up to 32 bytes against a table of the bytes
that can be written verbatim, four bytes per step, and copies them in one
go if all of them can. That is cheaper than setting up the vector scanner,
whose setup costs more than such a string takes to scan; SQLite's
jsonAppendString() writes strings with the same kind of table loop. Most
object keys and many values are this short (citm_catalog averages 8 bytes
per string, jeopardy 14).

GCC 16 and 13 stopped inlining put_indent() and put_literal() into the
pretty-printing loops once dump_escaped() had grown, which made indented
output up to 13% slower; both are now always inlined.

The new test puts every ASCII character and a 2-byte character at every
position of strings of 1 to 40 bytes and compares the output with the
reference implementation of the existing UTF-8 tests.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-21 20:01:28 +02:00
Niels Lohmann 52fecc1fab Move object keys into the value when parsing (#5855)
json_sax_dom_parser::key() copied the key from the lexer's buffer into the
object. The lexer starts every token with an empty buffer (reset()), so the
key can be moved instead, which saves an allocation and a copy for every key
that does not fit into the string's inline buffer. json_sax_dom_parser
already moved string values the same way. ordered_map has no rvalue
operator[], so ordered_json still copies keys.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-21 20:00:51 +02:00
Niels Lohmann fa37b952df Match literals without a stack array and keep parser states in bytes (#5854)
* Match literals without a stack array and keep parser states in bytes

scan_after_whitespace() built the expected text of true, false, and null in
local std::array objects. With -fstack-protector-strong (the default of
Ubuntu's GCC, among others), a local array gives the function a stack canary
check, which then ran on every token. scan_literal() now takes the text
after the first character as a string literal, with its length as a template
parameter; for contiguous input it compares the rest at once and falls back
to the byte loop on a mismatch, so errors are reported at the same
character. It is out of line, which keeps the token dispatch small.

The parser kept the nesting of arrays and objects in a std::vector<bool>,
whose push_back() GCC calls out of line; it is a std::vector<std::uint8_t>
now.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Skip the literal fast path test without exceptions

The test case compares the error messages of both lexer paths, so it does
not compile with JSON_NOEXCEPTION (the caught exception does not exist).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Allow the string literal parameter of scan_literal() for clang-tidy

The parameter takes the literal by array reference so that its length is
a template parameter; this is the same exemption the C array overloads of
to_json() and from_json() have.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-21 20:00:51 +02:00
Niels Lohmann e9970066c5 Scan strings with SSE2, SSSE3, or NEON (#5853)
* Scan strings with SSE2, SSSE3, or NEON

The string fast path found the end of a run of plain characters 8 bytes at
a time (SWAR) and validated non-ASCII text one sequence at a time, calling
validate_one_utf8() out of line for each.

The vector code of json_view's scanner, in the new shared header
detail/input/simd_scan.hpp, now does both 16 bytes at a time: SSE2 finds
the end of a run on x86-64, and the UTF-8 check of Keiser and Lemire runs
with NEON on AArch64 and with SSSE3 on x86-64. SSSE3 is not part of x86-64,
so the check is compiled for it with a function attribute and used when the
CPU has it (cached in a constant-initialized atomic), else the scalar check;
code compiled with -mssse3 skips the CPU check. The first word of a run is
still checked with SWAR (the first four with NEON, where a vector mask
costs more), as most keys and short values end there, and the scalar
helpers are always inlined. On anything else, the scalar code is unchanged.

The new macro JSON_NO_SIMD selects the portable code everywhere, including
the SSE2/NEON digit formatting of the floating-point writer. The lexer tests
now also run with JSON_NO_SIMD and, on x86-64, with -mssse3, and a new test
places every byte that ends a run at each offset across several 16-byte
blocks, after ASCII and after CJK text.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Keep the string scan from slowing down short strings and dump() on Arm

Three benchmarks were 7-9% slower than develop on Apple M1 (Apple Clang,
-O3 -flto): Accept/jeopardy, Dump/citm_catalog, and Dump/jeopardy.

- dump(): the serializer passes a string without its closing quote, so
  plain_run_end() ran out of input and called the out-of-line NEON loop
  for every string, with fewer than 16 bytes left. The loop is now only
  called for 16 bytes or more.
- accept(): no string of jeopardy.json reaches the vector UTF-8 check,
  yet compiling it into the lexer's scan function made it 5% slower: the
  function runs once per token and now saved more registers and kept a
  stack slot. lexer_string_bulk_run() checks the first four words inline
  and continues out of line, and the serializer keeps string_bulk_run()
  inline, which is faster there.
- The byte that ends a run is read from the scanned buffer instead of
  through get(), which Apple Clang merged with the other get() calls that
  load the read position again right after it was stored (3%).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Define string_bulk_run_from() only where lexer_string_bulk_run() calls it

With NEON and simdutf in C++17, lexer_string_bulk_run() takes the
string_bulk_run() path, so the out-of-line helper was unused.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Do not dispatch the SSSE3 string scan with GCC's C++20 modules

GCC does not keep the target("ssse3") attribute on functions it reads
back from a module interface, so the module build (ci_module_cpp20,
gcc:latest) rejected the intrinsics in vector_utf8_run() with "needs isa
option -mssse3". With __cpp_modules, GCC now uses the portable UTF-8 check
like other compilers without the target attribute; the SSE2 scan stays,
and -mssse3 or more still compiles the SSSE3 check directly. Clang is not
affected.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Include <cpuid.h> only once when json.hpp and json_view.hpp are both used

The SSSE3 dispatch of the string scan and the one of json_view each
include <cpuid.h>. Before GCC 11 the header has no include guard, so a
translation unit that includes both headers fails with GCC 10:
"redefinition of 'unsigned int __get_cpuid_max(unsigned int, unsigned int*)'".
Guard both includes with NLOHMANN_JSON_CPUID_INCLUDED.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Add an ABI tag for JSON_NO_SIMD

JSON_NO_SIMD selects between two definitions of the internal scanning and
formatting functions. Like JSON_VIEW_NO_SIMD (_vns), it now appends _ns to
the inline namespace, so translation units built with and without it can
be linked into one program without ODR violations.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-21 20:00:50 +02:00
Niels Lohmann 5e13f834c1 Copy the text of float tokens only when a SAX handler asks for it (#5852)
After the digits are converted while scanning, copying the token text into
token_buffer (an out-of-line std::string::append) was about 10% of parsing
canada.json. The only reader is the string argument of
json_sax::number_float(), and the library's own handlers
(json_sax_dom_parser, json_sax_dom_callback_parser, json_sax_acceptor)
ignore it.

For contiguous input, the fast path now keeps a pointer to the token in the
input, and get_string() copies it on first use. The parser passes an empty
string to the handlers marked by the new internal trait
sax_ignores_number_text instead of calling get_string(), so json::parse()
and json::accept() never copy float text. User handlers, and
JSON_DIAGNOSTIC_POSITIONS (which reads get_string().size()), still see the
exact text. Keys and strings, which never have pending text, use the
unchecked get_string_token(). Input read through a window into a stream
(std::istream, FILE*) copies the text right away, because the next read may
refill the window.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-21 12:26:46 +02:00
Niels Lohmann e13e976801 Accumulate number digits while scanning them (#5851)
* Accumulate number digits while scanning them

The contiguous number fast path validated a token byte by byte, copied it
into token_buffer, and then converted it in out-of-line functions that
scanned the digits a second time: parse_float_native() called
parse_float_significand(), which returned its result through memory and
called std::any_of (out of line with GCC) twice per number, and
parse_integer_signed() divided by ten on every digit for its overflow check.

scan_number_bulk_contiguous() now accumulates the significand while it
validates the digits, eight at a time where at least eight bytes remain
(is_eight_digits() and parse_eight_digits(), after fast_float). Floats with
at most 19 digits go straight to decimal_to_float(); integers with at most
18 digits are stored without an overflow check (a narrower number type
still rejects values that do not round-trip). Longer tokens take the
existing path. has_nonzero_digit() is a plain loop and
parse_float_significand() is always inlined, which helps that path.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Skip the digit accumulation checks without exceptions

The section compares error messages from both lexer paths and parses
out-of-range numbers, so it does not compile (the caught exception does not
exist) and would abort with JSON_NOEXCEPTION.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Share the significant-digit loop of the integer part and the fraction

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-21 12:26:46 +02:00
Niels Lohmann bd5227c2ec Fix silent wrong results for custom number types (#5805)
* Fix silent wrong results for custom number types

Squashed onto develop cd6b428ef (CI fixes from #5860). Original commits:

- Fix silent wrong results for custom number types
- Fix CI: build the out_of_range.407 message from string_t and fix a link
- Fix CI: diagnostics message, clang-tidy, and MinGW section limit
- Format the out_of_range.407 number without dump()
- Move the std::deque array test out of unit-regression3.cpp
- Fix clang-tidy and clang 3.4/3.5 failures in the custom number type changes
- Fix CI: cast the 128-bit unsigned value explicitly in the integer fast path

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Name the range checks of integer_fits_max/min

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-21 08:36:31 +00:00
Niels Lohmann 0cbe95ffa2 Speed up writing CBOR and MessagePack, above all integer arrays (#5824)
* Speed up writing CBOR and MessagePack, above all integer arrays

Squashed onto develop cd6b428ef (CI fixes from #5860). Original commits:

- Append to the output vector through a position-tracking sink
- Write CBOR integers without branching on their sign or size
- Write MessagePack integers without branching on their sign or size
- Use std::array for the integer size class table
- Fix clang-tidy and GCC warnings in the integer array test
- Regenerate the amalgamation

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Convert the streamed bytes explicitly in the binary writer sink tests

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-21 10:23:48 +02:00
Niels Lohmann 08d7fbb0a1 Read FILE* input with std::fgetc under MemorySanitizer (#5874)
With glibc and optimization, <stdio.h> turns getc_unlocked into an extern
inline function that reads the stdio buffer (fp->_IO_read_ptr) directly.
glibc, which is not instrumented, fills that buffer, so MemorySanitizer
treats the bytes as uninitialized and reports a false positive when the
parsers read from a FILE* (e.g., the fuzz harness FILE* leg, OSS-Fuzz
575915982). std::fgetc is an opaque call whose result MemorySanitizer treats
as initialized.

Select JSON_UNLOCKED_FILE_IO 0 when compiled with MemorySanitizer. This is a
regression from #5828 and not part of any release.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-21 10:22:50 +02:00