amorfo77/json - json - Gitea: Git with a cup of tea

mirror of https://github.com/nlohmann/json.git synced 2026-09-11 18:57:58 +00:00

Author	SHA1	Message	Date
Niels LohmannandClaude Opus 4.8	18293c7db1	Encode big-endian numbers with a byte swap instead of std::reverse write_number() reordered multi-byte numbers for the big-endian formats (CBOR/MessagePack/UBJSON) with std::reverse over the byte array. GCC lowered only some sizes to a bswap; clang kept a scalar byte shuffle (0 bswap instructions in the CBOR number path). Replace the reverse with size-dispatched __builtin_bswap16/32/64 helpers (portable shift fallback for other compilers; std::reverse retained for exotic sizes such as a long double number_float_t). Codegen: the CBOR number path now emits bswap on both compilers (gcc 2 -> 16, clang 0 -> 4). Output is byte-for-byte identical to the previous implementation across the binary differential corpus. Throughput (isolated vs the std::reverse version, best of 9): CBOR int64 array gcc +7% clang +10% CBOR uint16 array gcc +27% clang flat Modest but consistent on number-dense encodings; negligible on string/blob-heavy output, as expected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-09-11 18:10:38 +02:00
Niels LohmannandClaude Opus 4.8	c7f3ef97da	Fix CI failures from binary_writer output-sink change Four CI jobs failed on the initial commit; all are addressed here without changing any output (binary encodings remain byte-for-byte identical to develop across the differential corpus): 1. ci_test_gcc / cuda (-Werror=duplicated-branches): for number_float_t == float, static_cast<float>(n) is the identity, so write_compact_float's two branches are intentionally identical. Once the concrete vector sink is inlined, GCC constant-folds and diagnoses this (the type-erased path hid it behind a non-inlined virtual call). Silence -Wduplicated-branches for GCC (clang has no such warning) alongside the existing -Wfloat-equal pragma. 2. ci_static_analysis_clang (UBSan nonnull-attribute): binary_writer passes a null pointer with length 0 for empty strings/binary. output_vector_sink / output_adapter_sink declared write_characters JSON_HEDLEY_NON_NULL, so the sanitizer flagged the (harmless) zero-length call once the sink was called directly rather than through the attribute-free virtual base. Drop the attribute from both sinks, matching the pre-existing behavior. 3. ci_cpplint (build/include_what_you_use): output_adapter_sink uses std::move; add #include <utility>. 4. ci_cuda_example (nvcc 11.8): NVCC's front end rejects the default template argument on the binary_writer alias template. Revert the alias to its original single-parameter form (relying on binary_writer's own defaulted OutputSinkType) and spell out the full type in the vector-sink convenience functions. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-09-11 18:10:37 +02:00
Niels LohmannandClaude Opus 4.8	cb95492718	Devirtualize binary_writer via a value-type output sink to_cbor/to_msgpack/to_ubjson/to_bjdata/to_bson wrote every byte through output_adapter_t, a shared_ptr<output_adapter_protocol> whose write_character/write_characters are virtual. Unlike the lexer (templated on a concrete InputAdapterType), the binary writer never got that treatment, so binary output paid a vtable lookup per byte and a make_shared per call. Template binary_writer on an OutputSinkType and give it two concrete, non-virtual sinks: - output_vector_sink: appends straight into a std::vector (push_back / insert), used by the vector-returning to_* convenience functions. No vtable, no shared_ptr; the writes inline. - output_adapter_sink: forwards to a type-erased output_adapter_t, so the existing to_*(j, output_adapter) overloads (streams, strings, custom adapters) keep working exactly as before -- one virtual call each, unchanged. binary_writer keeps a convenience constructor taking output_adapter_t (building the default output_adapter_sink), so the adapter overloads are untouched; only the convenience functions switch to the vector sink. The friend declaration and the basic_json binary_writer alias gain the new (defaulted) template parameter. Output is byte-for-byte identical: verified across ~3000 randomized values plus curated edge cases (all scalar widths, strings with invalid UTF-8, binary, nested arrays/objects) for CBOR, MessagePack, UBJSON (both size/type settings), BJData, and BSON, plus the output_adapter path, in C++11/17/20. Warning-clean under clang -Weverything and the gcc pedantic set; clang-tidy clean on the changed headers; make check-amalgamation clean. Throughput (g++ -O3, vs develop): scalar-dense binary output such as integer arrays ~1.4x; many small to_cbor calls ~1.04x (DOM traversal bound); string/blob-heavy output unchanged (already bulk-bound). No workload regressed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-09-11 18:10:36 +02:00
Niels Lohmann	aa391dc0a5	Fix two develop CI regressions: dump() nodiscard warning and binary-reader const-correctness (#5520 ) * Discard dump()'s [[nodiscard]] return value in an exception-only check CHECK_THROWS_WITH_AS(j.dump(), ...) called dump() only to trigger and catch the exception, but never used the return value. dump() is warn_unused_result, so GCC's pedantic build (-Werror --all-warnings) rejected it as -Werror=unused-result, breaking ci_test_gcc. Wrapped in utils::ignore_return_value(), matching every other such call in this file. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Mark container_frame top as const in CBOR/UBJSON readers clang-tidy's misc-const-correctness flagged these on PR #5520's CI: the BSON sibling copy was already const, but these two were left mutable even though only container_stack.back().remaining is ever written. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-09-11 17:55:46 +02:00
Niels Lohmann	0452641c18	Read BSON documents without recursing per nesting level (#5508 ) * Read BSON documents without recursing per nesting level An embedded document (record type 0x03) or array (0x04) was read by calling back into the document reader, which read its element list, which called the element reader again for the next embedded one. The native call stack therefore grew with the nesting depth of the input, and about seven bytes buy a level, so a document of a few hundred kilobytes crashes the process (#5104). This is the last of the four binary formats to still do that. Apply the same shape as the other three: open_bson_document() reads the size prefix and opens the document, parse_bson_element_internal() calls it for both record types instead of recursing, and parse_bson_internal() loops over the element list of whichever document is innermost, closing it when its terminator is reached and resuming the one below. check_bson_document_size() is unchanged, and so is when it runs: a document is still measured from the byte before its size prefix to the byte after its terminator, and still reported before the end event. The frame carries those two values, which is what a per-document check needs once the reads are interleaved rather than nested. Nothing else about the element reader changes. unit-bson passes unchanged. Round trips through to_bson of nested objects, arrays, arrays of objects and mixed nesting are identical to the previous commit, as are the errors for a truncated document, an unsupported record type, a negative size and a size that does not match, including their byte offsets. A 30,000-level document built by to_bson is now read to completion where it used to crash. Note for sequencing: #5185 changes parse_bson_internal(), the element list and the array reader, which are the functions this commit restructures. It should land first; this commit then keeps its checks and moves them onto the loop. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Make parse_bson_internal's end-of-document top a copy, not a reference Same issue as the CBOR and UBJSON/BJData readers: top aliased container_stack.back() and was read (top.is_object) right after container_stack.pop_back() ended its lifetime. A copy stays valid regardless of what happens to the stack; nothing here mutates the live entry, so no field needs to go through container_stack.back() directly. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-09-11 08:34:45 +02:00
Niels Lohmann	a14619b354	Read UBJSON and BJData containers without recursing per nesting level (#5507 ) * Read UBJSON and BJData containers without recursing per nesting level get_ubjson_array() and get_ubjson_object() read their elements by calling back into the value reader, which called them again for a nested container, so the native call stack grew with the nesting depth of the input. '[' alone opens a container, so half a million of them crashes the process before the input runs out (#5104). The optimized forms reach the same path through a size or type annotation, and in plain UBJSON '[' and '{' are permitted as the type of an optimized container, so "[$[#i\x01" repeated nests just as deeply at six bytes a level. Both readers now only open their container, and parse_ubjson_internal() loops: it closes the containers that have ended, claims the next element of the innermost one, reads its key when it is an object, and works out the marker of the value to read next. That last part is where the formats differ, and the loop follows what the four element loops used to do: - a sized, typed container gives its elements no marker of their own - a sized, untyped container reads one for each element - a container that ends at a marker has the byte already, from the test against ']' or '}'; for an object it is the first byte of the key The ND-array wrapper and the 'B' binary shortcut stay as they are. Both read a complete value rather than opening a container, and their elements are always scalars: BJData does not permit '[' or '{' as an optimized type, which is also why only plain UBJSON needed the type-marker case above. A container of no-ops keeps its behaviour of holding no elements while still announcing its declared size to the SAX parser, by opening it and then setting its count to zero. unit-ubjson and unit-bjdata pass unchanged, 1.39 million assertions between them, and a behaviour comparison against the previous commit over every container form -- sized, unsized, typed, untyped, empty, no-op, ND-array, binary, and the forms nested inside one another -- gives identical values, error codes, messages and byte offsets. 500,000 levels of each vector now report a parse error instead of crashing, and a well-formed 100,000-level value is read to completion. The driver costs about 3 % on parsing 60,000 small objects and one array of a million integers, for the reason given in the previous commit; reading the frame once per element rather than per branch halved what it cost before. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Make the UBJSON/BJData advance loop's top a copy, not a reference Same issue as the CBOR reader: top aliased container_stack.back() and was read (top.is_object) right after container_stack.pop_back() ended its lifetime. A copy stays valid regardless of what happens to the stack; the one place that mutates the live entry (--top.remaining) now goes through container_stack.back() directly. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-09-11 08:34:44 +02:00
Niels Lohmann	91ab3e81f5	Read CBOR containers and tags without recursing per nesting level (#5506 ) * Read CBOR containers and tags without recursing per nesting level get_cbor_array() and get_cbor_object() read their elements by calling back into the value reader, which called them again for a nested container, and a tag was handled by reading the tagged value the same way. All three cost native stack, and all three cost a single byte to encode: 0x9F opens an indefinite-length array, 0x81 a one-element array, and 0xC2 is a tag. Half a million of any of them crashes the process before the input runs out (#5104). Apply the shape the MessagePack reader already uses: the open containers live on the heap stack, parse_cbor_value() reads a single value and only opens a container rather than reading it to its end, and parse_cbor_internal() loops, resuming the innermost container after each element. Two things are specific to CBOR. An indefinite-length container ends at a break marker rather than at a count, and testing for that marker consumes a byte which is the first byte of the next element when it is not one; the frame's count is npos for those, and the driver tracks whether the next value starts at a fresh byte. And a tag is not a value of its own: instead of reading the tagged value by recursing, the value reader reports that a tag was read and the driver reads on, so a chain of tags costs no stack at all. The switch that decodes a value is unchanged apart from the twelve container cases and the two tag sites. Verified against the previous commit over definite and indefinite arrays and maps, all four counted forms, empty containers, nesting of the forms inside each other, truncated inputs, and all three tag handlers: identical values, error codes, messages and byte offsets. 500,000 levels of each of the three vectors now report parse_error.110 instead of crashing, and a well-formed 200,000-level value is read to completion. On performance: the driver does per element what a counted loop used to do per container, and CBOR pays for it more than MessagePack because the value reader also has to be told whether to fetch a byte. Parsing 60,000 small objects and one array of a million integers is 3 to 4 % slower than the recursive reader, measured over five alternating runs. Against develop the same two inputs are about 44 % faster, because the entry point no longer copies the value it parsed; the earlier commit in this series is what pays for that. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Make parse_cbor_internal's top a copy so it survives pop_back() top aliased container_stack.back(), and was still read (top.is_object) right after container_stack.pop_back() destroyed the element it aliased. Nothing currently reorders those two lines, but the comment claiming the reference's lifetime was already fine only accounted for reallocation from a push, not this. A trivially-copyable container_frame makes top a copy instead, so reads of it stay valid regardless of what happens to the stack; the one place that mutates the live entry now does so through container_stack.back() directly rather than through top. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-09-11 08:34:44 +02:00
Niels Lohmann	0dd8ca9023	Read MessagePack containers without recursing per nesting level (#5505 ) get_msgpack_array() and get_msgpack_object() read their elements by calling back into parse_msgpack_internal(), which calls them again for a nested container. The native call stack therefore grew with the nesting depth of the input, and each level costs only one byte to encode: 0x91 is a one-element array, so a few hundred thousand of them crash the process before any of the input is rejected (#5104). Keep the open containers on a heap stack instead, the way parser::sax_parse_internal() has always done for JSON text. A frame records how many elements are left and whether to close with end_object() or end_array(); parse_msgpack_value() reads a single value and, for a container, only opens it; and parse_msgpack_internal() loops, resuming the innermost container after each element and closing it when its count runs out. Whether the value that was begun is complete is answered by the stack being empty, so no separate bookkeeping is needed. The switch that decodes a value is untouched apart from the six container cases, which now call enter_container() rather than a reader that loops. That keeps this diff to the control flow and leaves the decoding of every other type byte-identical. enter_container() is the only place a binary reader emits start_object() or start_array(), so a check that rejects a container can be added there once and is guaranteed to run before the start event. The frame type and the stack are shared, ready for the other three formats. Verified against develop over empty, nested, counted (array 16/32, map 16/32) and truncated inputs: identical values, error codes, messages and byte offsets. 300,000 levels now report parse_error.110 instead of crashing, and a well-formed 300,000-level value is read to completion through the SAX interface, where develop crashes. Reading such a value into a basic_json needs the return-by-move change as well, without which the recursive copy constructor overflows on the way out; that is the parent commit, and the test for the value path covers the two together. Timing is unchanged: parsing 60,000 small objects and one array of a million integers is within run-to-run noise of develop either way. Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-09-11 08:34:43 +02:00
Niels Lohmann	44f8ec30e9	Bound UBJSON optimized arrays of a valueless type (#5504 ) * Bound UBJSON optimized arrays of a valueless type An element of type 'Z' (null), 'T' (true) or 'F' (false) is encoded by its type marker alone, so an optimized UBJSON array of one of those has no payload: reading an element consumes no input at all. Its declared count is therefore the only thing that decides how much is allocated, and nothing bounded it. "[$Z#l" and a four-byte count is nine bytes of input describing two billion values; #2793 reports 35 GB and 150 seconds from ten bytes, and OSS-Fuzz has an out-of-memory and a timeout report for the same shape. Every other type costs at least one byte per element, so the end of the input bounds it. 'N' (no-op) is already skipped rather than stored. Objects are not affected either: each element is preceded by its key, which costs bytes. And BJData already refuses these markers as an optimized type, so this is a plain UBJSON matter. Reject a count above 1,048,576 elements for those three types with out_of_range.408, the code this reader already uses for a declared size it will not honour. The check runs before the SAX start event, so no container is opened and then abandoned. Rejecting on the read side alone would break the guarantee that anything to_ubjson() writes can be read back, and would trip the round-trip assertion in fuzzer-parse_ubjson.cpp. So the writer falls back to the unoptimized encoding, one byte per element, for arrays of these types above the same limit. Its decision depends only on the array's size, which is identical for a value and for anything parsed back from it, so the round trip is stable. No existing test changes: the largest such count in the test suite is 65,793. The excessive-size test that already used this shape still passes, now rejected a little earlier than by the max_size() check it used to reach. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Note the 1,048,576 valueless-array limit as (1 << 20) in the docs Addresses review feedback from @gregmarr on PR #5504: spell out the binary/hex form next to the decimal count so it reads as the round power-of-two it is, matching how include/nlohmann/detail/input/binary_reader.hpp defines max_valueless_container_size. Applied in both docs/exceptions.md and ubjson.md, as requested. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-09-11 08:34:42 +02:00
Niels Lohmann	4bb2de8cc5	Reject a nested BJData ndarray dimension vector where it is read (#5503 ) get_ubjson_size_type() takes an inside_ndarray parameter saying whether it is being called for an ndarray's dimension vector, where another ndarray is not allowed. It then seeded the flag it passes down to get_ubjson_size_value() with `false` rather than with that parameter, and only consulted inside_ndarray afterwards, on the '$' branch. So on the '#' branch nothing stopped the descent: every "#[" pair of an input like "[" followed by "#[#[#[..." opened another dimension vector, several native stack frames deeper each time, and the recursion was only reported on the way back out. 100,000 pairs crash the process. This is #5104 again, in a path that has nothing to do with containers. Seed the flag with inside_ndarray, which is what get_ubjson_size_value() documents it wants: "for input, `true` means already inside an ndarray vector or ndarray dimension is not allowed". The nested '[' is then refused where it is read, so the length of the chain no longer matters. Both post-checks gain `&& !inside_ndarray`, because an ndarray was found here only if the flag flipped -- get_ubjson_size_value() only ever returns `true` when its initial value was `false`, as its documentation says. With that, the "ndarray can not be recursive" branch is unreachable: a recursive ndarray is now caught one level earlier, and reported as "ndarray dimensional vector is not allowed" like every other nested dimension vector. Three existing expectations move accordingly (vR2, vR4, vR6). All three now fail earlier, and all three now report the same error that vR1, vR5 and vH already reported for the same shape, which is the more consistent outcome. Everything else is unchanged: valid 1D and 2D ndarrays, optimized containers and plain arrays produce identical results, and unit-ubjson is untouched. Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-09-11 08:34:41 +02:00
Niels Lohmann	dd50f0eb16	Stop CBOR indefinite-length strings from recursing per chunk (#5502 ) get_cbor_string() and get_cbor_binary() handled the indefinite-length forms (0x7F and 0x5F) by calling themselves once per chunk. Each chunk therefore cost a native stack frame, and since a chunk may itself be an indefinite- length string, an input of repeated 0x7F bytes reached one frame per input byte: 200,000 of them crash the process with SIGSEGV before a single byte is rejected. This is the same defect as #5104, in a path the container-level work does not touch. Count the open levels instead of recursing through them. That is enough here because every chunk is appended to the same result -- get_bytes() writes at result.size() -- so there is no per-level state to keep. The temporary chunk string and its copy into the result go away with the recursion. The definite-length cases move to get_cbor_string_chunk() and get_cbor_binary_chunk() unchanged, including their error messages, which still name 0x7F and 0x5F because those are handled one level up. Behaviour is unchanged. Comparing against develop over the interesting byte sequences -- empty, single-chunk, nested, over-closed and truncated forms, both strings and byte arrays, and an indefinite-length map key -- produces identical values, error codes, messages and byte offsets. The 200,000-level input now reports parse_error.110 at byte 200001 instead of crashing. Note that nesting these is not valid CBOR: RFC 8949, Section 3.2.3 forbids it. This does not change that either way -- it has always been accepted, and rejecting it is a separate decision (#5317, #5325). Should it be rejected later, that is now one condition on the level counter rather than a change to the control flow. Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-09-11 08:34:41 +02:00
Niels Lohmann	fae364db24	Move the scanned string into the value instead of copying it (#5458 ) * Move the scanned string into the value instead of copying it The SAX interface documents that the string handed to json_sax::string() may be moved from, and the DOM handlers already move the one handed to binary(). string() did not, so every string value was copy-constructed out of the lexer's token buffer, which then kept the buffer alive at its high-water mark until the next token overwrote it. Moving hands that buffer to the new value instead. The allocation count is unchanged - the value needed one either way - but the copy is gone. jeopardy 247.3 ms -> 240.9 ms (-2.6%) citm_catalog 4.61 ms -> 4.48 ms (-2.8%) 40k 30-char strings 7.80 ms -> 7.64 ms (-2.1%) Note this deliberately does not extend to the object key. Moving the key hands the lexer's buffer - sized for the largest token seen so far - to a key that is usually short, so the next value has to grow a fresh buffer. Measured, that costs 11.9% on a document of many small keys with longer values. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Spell out the move rationale at every handle_value(std::move) site The comment explaining why the value is moved sat only on json_sax_dom_parser::string(), and the callback parser's string() pointed at it with "see json_sax_dom_parser::string()". That reference cannot be searched for - the function is declared as `bool string(string_t& val)` inside the class, so the qualified name appears nowhere - and the two binary() overloads, which have always moved, carried no explanation at all. Put the same comment on all four sites and name json_sax, which is greppable, instead of a member that is not. Comment-only; no generated code changes. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-09-11 08:22:51 +02:00
Niels Lohmann	1dc1d09fc6	Do not search a container for the value the callback rejected (#5457 ) When a parser callback rejects a value, the placeholder stored for it has to be removed from its parent again. remove_discarded_value() found it by scanning the parent from the beginning, so filtering a container cost one scan per rejected member - quadratic in the number of members of a single container. A rejected value can only ever be the one most recently added to its parent: the last element of an array, or the placeholder key() stored under the current key in an object. Record that key alongside the existing key_keep_stack, and for a container record it again alongside ref_stack so end_object()/end_array() can find it in the parent. Removal is then O(1) for an array and O(log n) for an object, and finding nothing there means nothing was stored, so there is nothing to remove. The key for a container is read before handle_value() may consume it, so it is also correct when the callback rejects the container at its start event and it never reaches its parent at all. Discarding half the members of one object, before -> after: members value rejected container rejected at start 16 000 392 ms -> 3.7 ms 803 ms -> 8.2 ms 64 000 6238 ms -> 14.6 ms 12651 ms -> 32.2 ms 128 000 25339 ms -> 30.6 ms Results are unchanged: 48 000 randomized documents parsed under 12 different filtering callbacks - covering duplicate keys, empty keys, rejected keys and containers rejected at both their start and end events - produce byte identical output before and after. Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-09-11 08:22:50 +02:00
Niels LohmannandClaude	da42a627dc	Stop dump() from heap-allocating its output adapter per call (#5449 ) * Stop dump() from heap-allocating its output adapter per call The serializer held its output sink as output_adapter_t<char> (a std::shared_ptr<output_adapter_protocol<char>>), which dump() and operator<< built via make_shared -- one heap allocation per call for a sink that only wraps a reference to the caller's string or stream. Hold the sink as a non-owning output_adapter_protocol<char>* instead and construct the concrete adapter on the stack at the call site. The write path (o->write_characters) is unchanged, so output is byte-for-byte identical; a compact dump() of a small object drops from 2 heap allocations to 1 (only the returned string remains), ~3% faster. Completes the per-call allocation cleanup on this branch, which already removed the indent_string buffer (both were reported in #5413). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L1oJ2ggRHS37zeVe94QTA1 Signed-off-by: Claude <noreply@anthropic.com> * Take the output adapter by reference at the serializer ctor Per review: the serializer still holds the adapter as a non-owning pointer, but the constructor now takes output_adapter_protocol<char>& and takes its address internally, so every call site passes a reference. A reference cannot be null and reads as a borrow, which makes the lifetime contract harder to get wrong than handing over a raw pointer. The stored member and the write path are unchanged. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Claude <noreply@anthropic.com> Signed-off-by: Niels Lohmann <mail@nlohmann.me> Co-authored-by: Claude <noreply@anthropic.com>	2026-09-11 08:22:50 +02:00
Niels LohmannandClaude Opus 4.8	ff80ed3295	Speed up dump(), and keep it from overflowing the stack (#5285 ) * Add SWAR bulk fast path to string serialization (dump_escaped) When ensure_ascii is false, dump_escaped previously ran every byte of every string and object key through the UTF-8 DFA decoder, even for the common case of ordinary text with nothing to escape. This mirrors the per-byte cost the parser had before the contiguous fast paths. At a character boundary, bulk-copy the longest run of bytes that need no escaping using string_bulk_run() - the same SWAR scanner and UTF-8 bulk validator the lexer's contiguous path uses - and only fall back to the byte-at-a-time DFA loop for the first byte that needs individual handling (a quote, backslash, control character, or ill-formed/truncated UTF-8). Because every "hard" or invalid byte is still processed by the unchanged byte path, escaping output and error handling (including strict-mode error 316 position and message) are byte-identical to before. The ensure_ascii=true path is unchanged: it must escape non-ASCII and 0x7F, which string_bulk_run does not stop on, so a separate predicate would be needed for it. Verified byte-for-byte identical dump output against the pre-change implementation across ~20k randomized byte strings plus curated edge cases (all escapes, control chars, valid multibyte, surrogates, overlong, truncated sequences) for both ensure_ascii settings and all three error handlers, in C++11/17/20 at -O2/-O3. Throughput (g++ -O3, ensure_ascii=false, vs pre-change): long ASCII strings 4.2x twitter-like objects 2.3x dense CJK 1.4x (further headroom with JSON_USE_SIMDUTF) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Buffer serializer output and add ensure_ascii string fast path Two further serialization speedups on top of the ensure_ascii=false bulk copy, both reusing the SWAR primitives in detail/input/string_scan.hpp. 1. Internal write buffer (devirtualization). Every structural character ('{', '"', ',', ...) previously went straight to the output adapter through a virtual call. Route all writes through put_char/put_chars into a 1 KiB buffer that flushes in bulk; the public dump() flushes once the top-level value is done (the recursive worker is split out as dump_internal). Runs larger than the buffer are written straight through, so large payloads are not copied twice. This is the dominant cost for object/array-heavy values. 2. ensure_ascii fast path. dump_escaped previously ran the UTF-8 DFA over every byte when escaping non-ASCII. Add find_ascii_copyable_run() (a SWAR scan stopping at '"', '\\', < 0x20, 0x7F, and >= 0x80) so runs of printable ASCII are bulk-copied, with the byte path handling each escape/non-ASCII byte exactly as before. Behavior is unchanged: dump output is byte-for-byte identical to the previous implementation across ~20k randomized byte strings plus curated edge cases (all escapes, control chars, 0x7F, valid multibyte, surrogates, overlong, truncated), for object/array/pretty output, both ensure_ascii settings, and all three error handlers, in C++11/17/20 at -O2/-O3. New unit tests cover the buffer flush boundaries, the escape and 0x7F handling, multibyte under both settings, and invalid-UTF-8 handling. Throughput (g++ -O3, vs the ensure_ascii=false-only baseline): long ASCII, ensure_ascii=0 4.2x long ASCII, ensure_ascii=1 4.1x twitter-like objects 2.7x dense CJK 1.8x Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Flush serializer buffer in dump_escaped unit test test-convenience failed (macOS finished first; the failure is platform-independent) because check_escaped() calls the internal serializer::dump_escaped() directly and then reads the output stream. Since dump_escaped() now writes into the serializer's internal write buffer, the bytes were still buffered and the stream was empty. Expose flush() under JSON_PRIVATE_UNLESS_TESTED (same visibility as dump_escaped) and flush in check_escaped() before inspecting the output. Per-string flushing inside dump_escaped() was rejected on purpose: it would defeat the buffering that makes object/array-heavy dumps faster. Library behavior is unchanged (flush()'s body is identical; only its access label moved). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Avoid deep recursion in serialization write-buffer test The "many small structural writes exceed the write buffer" subcase built a 1100-deep nested array and dumped it to force >1024 consecutive single-character writes through put_char (exercising the write buffer's flush-when-full branch). dump() recurses per nesting level, so on MSVC debug builds (smaller default stack, larger frames) this overflowed the stack and crashed test-serialization; Linux/macOS have enough headroom to hide it. Replace the nesting with a flat array of 500 empty strings. Each element emits '"', '"', ',' via put_char, so the dump is a long run of single-character writes (1501 bytes > the 1024-byte buffer) at nesting depth two, hitting the same flush branch without deep recursion. Library code is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Split the write-buffer helpers and write indentation directly Follow-up to @gregmarr's review: put_chars() was doing four unrelated jobs, so give the two that can be made safe their own entry points. - put_literal(): takes the literal by reference and deduces the length from the array bound, so the 27 hand-counted lengths at the call sites can no longer drift from the literals they describe. A literal is checked at compile time to fit the buffer, so this path needs no write-through branch. - put_buffer(): takes the fixed-size buffer itself rather than a bare pointer, so the length can be checked against the buffer's own bound. - put_indent(): memsets the indentation into the write buffer, filling and flushing it as needed. This removes indent_string entirely, and with it both bugs of #5186: the indentation string was grown by doubling, which is not enough when indent_step more than doubles it (a heap over-read - dump(2000) read 2000 bytes out of a 1024-byte string), and the grown part was filled with a space instead of the configured indent_char. next_indent() keeps that PR's assertion against the unsigned indentation accumulation wrapping on deep nesting. put_chars() keeps the two cases that are genuinely a pointer and a count: the run-length copies out of the string being escaped, and to_chars() output. Tests cover an indent_step wider than the write buffer, a non-space indentation character past the old growth point, and nesting whose accumulated indentation spans several buffer-fulls. All three fail against develop. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fill the indentation buffer once instead of once per flush @gregmarr's point on the fill-and-flush loop: flushing does not disturb what the write buffer holds, so an indentation spanning several buffer-fulls only has to be written into the buffer once and can then be handed to the adapter as many times as needed. The loop re-filled it every time, doing work it already knew was there. put_indent() now fills the room left in the buffer, and if anything remains, flushes, fills the buffer once, and re-flushes that same content. It also returns early for a zero-width indentation, which is what the closing brace of every outermost value asks for. Measured over a dump(), counting memset calls and bytes inside put_indent: indent before after 4 1 call / 4 B 1 call / 4 B 2000 2 calls / 2000 B 2 calls / 2046 B 100000 98 calls / 100000 B 2 calls / 2046 B The wide case is now constant work rather than proportional to the indentation width; ordinary widths are unchanged. Tests extended to cover several whole buffer-fulls and an exact multiple of the buffer size. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Tighten the write-buffer helpers after review More of @gregmarr's review on the put_* split: - Reattach the put_chars() doc comment, which the new helpers had been inserted in front of, leaving it describing put_indent(). - Compute the literal length once in put_literal() instead of spelling N - 1 at each use. - Add put_string(str, start, end), which keeps the pointer arithmetic and the bounds assertions inside the function instead of at the call site. With dump_float()'s to_chars() output moved onto put_buffer() as well, put_chars() now has no callers outside put_string()/put_buffer(): nothing passes a bare pointer and a count any more. - Carry the indentation as std::size_t rather than unsigned int. It is a size, it is compared and combined with buffer sizes throughout, and the casts in put_indent() disappear. next_indent() keeps its assertion, which is far harder to trip on a 64-bit size_t but still reachable where that is 32 bits. No output change: pretty and compact dumps, binary values included, are byte-identical to develop. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Silence avoid-c-arrays on put_literal's array reference clang-tidy flags the reference-to-array parameter under cppcoreguidelines/hicpp/modernize-avoid-c-arrays, and the CI treats warnings as errors. Binding to the array is the whole point here - it is what lets the length be deduced from the literal instead of hand-written at the call site - so suppress it the same way from_json(), to_json() and get_to() already suppress it for their own T (&arr)[N] parameters. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Bound the descent of dump() Serializing a container serializes its elements, so dump() descended into one call per nesting level. A value nested deeply enough exhausted the call stack and terminated the process with a segmentation fault - no exception, nothing the caller could catch. Parsing such a value works, as the parser is iterative, and so does destroying one, as #1436 made destruction iterative. Bound how far the descent goes rather than take the call stack away from it. The first 128 levels are written by exactly the code that always wrote them, and only below that does dump_iteratively write out what is left, keeping the containers it has entered on an explicit stack. Serializing can therefore no longer exhaust the stack, however deeply a value is nested, while a value nested less deeply than the bound pays only for one comparison per container. Writing every value that way instead measured between 2% and 20% slower - 20% on object-heavy documents - which is why the descent is kept for all but the values that cannot afford it. The bound costs nothing measurable: between -1.4% and +1.2% across compact and pretty output of number, integer, string, object-heavy, wide-object and deeply nested documents. The output is unchanged for every value. Both ways of writing a container emit the separator in front of every element but the first, rather than after every element but the last, which puts exactly one between each pair and none at the end. This fixes #5387 for dump(). The copy constructor is fixed in #5389. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fold ensure_ascii into the escaper and write bytes without dump_integer Two hot spots that the write buffer and the bulk scanner left behind. dump_escaped took ensure_ascii as a runtime flag and tested it inside the loop, once per character run, although it cannot change while a string is written. It is now a template parameter, dispatched once per string, which folds the choice of scanner and lets each of the two be inlined into a loop of its own. This is the hottest loop in the serializer: it runs over every string and every object key. A binary value's bytes went through dump_integer, which counts digits and does 64-bit arithmetic for a number that is always in [0, 255]. dump_byte writes the three digits it takes at most straight into the write buffer instead. Any byte type that is not a plain unsigned byte is still left to dump_integer, whose representation of it may differ. Measured against the previous commit (medians of 9 interleaved runs, clang -O3): binary values -33.8%, dense CJK with ensure_ascii -20.6%, key-heavy objects -17.8%, deeply nested pretty output -17.9%, dense CJK without ensure_ascii -11.8%, object-heavy documents -9.3% compact and -9.5% pretty, a small value dumped in a loop -21.4%, wide objects -2.3%. Arrays of plain ASCII strings measured 3.5% to 4.2% slower, the one shape that loses; number and integer arrays are unchanged. Also tried and dropped: leaving the write and string buffers uninitialized rather than zeroing 1.5 KB per dump() call. It is worth -30% on small values, but two nearly identical string workloads moved 18% apart in opposite directions, so the measurements did not support it. The output is unchanged for every value: the differential now also covers every one of the 256 byte values, alone and together, in both binary layouts. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Write a byte without walking a pointer over the buffer clang-tidy's misc-const-correctness reads the pointer dump_byte advanced over the write buffer as one whose pointee could be const. Index the buffer instead, which says the same thing without a raw pointer at all. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Parenthesize the reserve arithmetic in the deep-nesting test clang-tidy's readability-math-missing-parentheses wants the multiplication spelled out in reserve(6 * depth + 1), and CI treats its warnings as errors. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Do not scan for a copyable run that cannot exist Under ensure_ascii, dump_escaped() calls find_ascii_copyable_run() at every character boundary. When the text is dense non-ASCII - CJK, where every byte is >= 0x80 - the scanner stops on its first byte and returns zero, so its SWAR block runs once per character and buys nothing, on top of the escaping that still has to happen afterwards. A run can only be non-empty when the first byte is one the scanner may copy, so test that single byte before calling it. Runs that do exist are found exactly as before, so the bulk-copy win is unchanged; only the calls that were always going to return zero are skipped. Output is unchanged: the dump digest over canada/citm/twitter, in compact, pretty and ensure_ascii form, matches develop byte for byte. dump(ensure_ascii=true) develop before after CJK text 3.54ms 4.25ms 3.36ms CJK, no ASCII at all 3.09ms 4.02ms 3.02ms Latin-1-ish text 4.39ms 3.04ms 2.93ms plain ASCII 3.92ms 0.80ms 0.79ms Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address review of the write-buffer helpers Three points from @gregmarr's review: put_chars() is gone. It was the only entry point taking a bare pointer and a count, and it existed only so put_string() and put_buffer() had something to delegate to. Its body now lives in put_string(), and put_buffer() is put_string(buffer, 0, length) - std::array already carries data() and size(), so it satisfies the same interface a string does. Nothing appends characters without a bound any more. dump_escaped()'s documentation block was duplicated. The dispatcher was inserted between the original comment and the function it described, and the comment was copied rather than split. The worker now has its own short comment saying why ensure_ascii is a template parameter. The local in dump_byte() is deliberate, and is now documented as such: writing through write_buffer[] is a char write, which may alias any object, so with write_buffer_pos updated in place the compiler must reload and store it around every digit. Measured on a dump of a 4 MiB binary value, 18.0 ms without the local against 7.4 ms with it. Output is unchanged: byte-identical dumps across 77 files in compact, pretty, ensure_ascii, pretty+ascii, indent 600 and tab-indent form. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address review: drop unneeded backslash-escapes and duplicate scan loop '"' does not need escaping in a char literal, unlike in a string literal. find_ascii_copyable_run() also duplicated the byte-at-a-time search that already exists as the loop's own scalar tail; break into it instead of re-deriving the offset in a second, near-identical loop. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Move pretty_print, ensure_ascii and indent_step into the serializer None of these change over the life of a serializer, unlike current_indent and depth, which do change on every recursive call. They are now captured once in the constructor - matching indent_char and error_handler - instead of being threaded through dump(), dump_internal(), dump_iteratively(), dump_value() and dump_escaped() on every call. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Stop the serializer from holding onto std::localeconv()'s pointer loc was only ever read twice, immediately, to seed thousands_sep and decimal_point; nothing else in the class used it. A local in the constructor body serves the same purpose without keeping the pointer around for the serializer's lifetime. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Keep thousands_sep/decimal_point const via a small locale_chars struct const members can't be assigned in a constructor body, so seeding them from std::localeconv() meant either dropping const or holding onto the lconv* for longer than needed. A sub-object computes both from the pointer in its own constructor and is itself initialized in serializer's mem-initializer-list, so the two chars stay const, std::localeconv() is still called exactly once, and nothing outlives the constructor. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>	2026-09-11 08:22:49 +02:00
Niels Lohmann	bfb07786cd	Fix to_bjdata() silently truncating out-of-range _ArrayData_ elements (#5473 ) write_bjdata_ndarray() validated that each _ArrayData_ element matched the number kind (integer vs. float) named by _ArrayType_, but not its range. An element that did not fit the target C++ type (e.g. 256 for "uint8") was silently wrapped by the static_cast used to write it, or, for "single", silently overflowed to infinity. Range-check each element against the type named by _ArrayType_ before writing it, reusing the existing fallback path that already encodes the annotated object as a plain object for other invalid-annotation cases in this function. Fixes #5403. Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-09-10 17:59:28 +02:00
Niels LohmannandClaude Opus 4.8	4a93aa4e2f	Speed up parsing of contiguous input (numbers, strings, UTF-8) (#5283 ) * Speed up number parsing in the lexer (fast paths from the fast_float/simdjson world) The number scanner converted its already-validated digit buffer with std::strtoull/std::strtoll/std::strtod. Those pull in locale and errno machinery and dominate number-heavy parsing (strtod runs at ~6 M/s). Replace them with dedicated parsers over the validated buffer: - parse_integer_unsigned / parse_integer_signed: accumulate digits with overflow detection, falling back to the float path on overflow exactly as the strtoull/strtoll round-trip check did. Overflow behavior is unchanged for narrower or wider custom number types. - parse_float_fast: Clinger's exact fast path for `double` (<=19 significant digits, \|exp10\| <= 22, significand < 2^53), where significand * 10^exp is exact under IEEE round-to-nearest. This is the same fast path used by fast_float/simdjson. It is bit-identical to strtod on this subset and declines (falling back to strtod) otherwise. Only `double` uses it; float and long double keep std::strtof/std::strtold via a templated overload. Measured on representative data (g++ 13, -O3): - integers: DOM parse +11%, SAX +25-34% - floats: DOM parse +37%, SAX +70% (clang: float DOM ~1.9x) No dependencies added; header-only and C++11-clean. Existing parser, lexer, conversion and deserialization unit tests pass unchanged; a 3M-value random-double fuzz matches strtod bit-for-bit. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Add SWAR bulk string scanning for contiguous input (simdjson-style) scan_string() read the input one character at a time through the input adapter and classified every byte with a large switch. For contiguous byte buffers we can instead scan 8 bytes at a time with a SWAR word test that finds the first byte needing individual handling (the closing quote, an escape, a control character, or a non-ASCII UTF-8 byte) and bulk-append the ordinary run in one go. - input adapters expose supports_bulk_scan / bulk_data / bulk_remaining / bulk_skip for provably-contiguous, same-type, 1-byte iterator ranges (raw pointers in every standard; std::string/std::vector/std::array and friends additionally in C++20 via std::contiguous_iterator). - the lexer gains a bulk_scan capability (gated on lazy_token_string so bypassing the per-character capture cannot lose error diagnostics) and a scan_string_bulk() fast path; streaming/wide/user adapters are unchanged and keep the byte-at-a-time scanner. The run contains no newline (all bytes < 0x20 are treated as special), so position bookkeeping stays exact, and error tokens are still reconstructed lazily from the consumed byte range. The SWAR special-byte test is pure uint64_t arithmetic - no intrinsics, no runtime dispatch, C++11-clean. Measured on representative data, pointer input, g++ 13 -O3 (string values discarded by accept() see the largest gains): long ASCII strings: DOM +4.5x, SAX +14x, accept +17x (to ~2 GB/s) short strings: DOM +15%, SAX +62%, accept +85% escape-heavy: DOM +31%, SAX +26%, accept +28% Same-input parity verified: 200k randomized documents (escapes, multibyte UTF-8, surrogate pairs) accept/parse identically via the contiguous SWAR path and the streaming byte path; unit lexer/parser/diagnostic-position/ deserialization/conversions suites pass unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Validate UTF-8 in the bulk string scanner (portable ~2-3x on non-ASCII text) The SWAR bulk string path stopped at the first non-ASCII byte and handed every multibyte character to the byte-at-a-time scanner, whose per-byte get()/next_byte_in_range()/add() machinery runs at roughly half the speed of validating straight from the buffer. As a result, dense non-ASCII text (CJK, emoji, accented Latin) parsed ~10-15x slower than ASCII. Fold well-formed UTF-8 into the bulk run: scan_string_bulk() now, on a non-ASCII lead byte, validates one sequence with validate_one_utf8() - which mirrors scan_string()'s per-byte switch ranges exactly (rejecting overlong forms, surrogates, and out-of-range code points) - and appends it in place, continuing until the closing quote, an escape, a control byte, or an ill-formed sequence. All error handling still defers to the byte path, so error messages and positions are byte-for-byte unchanged. Because only well-formed content is fast-pathed and every rejection falls through to the existing scanner, behavior is identical; the win is purely throughput. Measured on pointer input (accept, string values discarded): content g++ 13 clang 18 dense CJK 277 -> 648 ~605 MB/s (~2.3x) dense emoji 299 -> 857 ~702 MB/s (~2.6-2.9x) mixed 90% ASCII 246 -> 331 ~334 MB/s (~1.35x) pure ASCII unchanged (~3.2 / 4.1 GB/s) Verified: 2,000,000 randomized documents built from arbitrary bytes (overlong, surrogate, truncated, out-of-range sequences) accept/reject and parse identically via the contiguous path and the streaming byte path; lexer/parser/diagnostic-position/deserialization/conversions suites pass unchanged. Pure C++11, no intrinsics. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Route contiguous byte containers through the pointer adapter (fast paths in C++11) json::parse(std::string) - the most common entry point - did not benefit from the contiguous fast paths (bulk string scanning, UTF-8 bulk validation, memcpy for binary formats) in C++11..17: std::string::iterator is a library wrapper, not a raw pointer, and pre-C++20 there is no portable way to prove it contiguous, so supports_bulk_scan was false. Only raw pointers, string literals, and C-arrays (and, in C++20, anything modelling std::contiguous_iterator) took the fast path. Detect contiguous single-byte containers (std::string, std::vector<char>, std::vector<std::uint8_t>, std::string_view, ...) via is_contiguous_byte_ container and route them through an iterator_input_adapter built from data()/data()+size(). The generic iterator-based container overload is constrained to exclude these, so the two overloads are disjoint and there is no ambiguity (a plain competing overload loses to the greedy forwarding-reference container overload on reference binding, and a factory partial-specialization is ambiguous - both were tried and rejected). The pointer keeps the container's own element type, so char_type - and therefore all parsing behavior - is byte-for-byte identical to the iterator path (const char* for std::string, const std::uint8_t* for std::vector<std::uint8_t>); only the raw pointer additionally turns on the fast paths. Lifetimes are unchanged: the container outlives the adapter for the full parse expression, exactly as the iterators it replaces did. Measured, C++11, json::parse/accept(std::string), g++ 13: long ASCII strings: accept 201 -> 3200 MB/s (~16x), parse 174 -> 1444 dense CJK: accept 263 -> 697 MB/s (~2.6x) short strings: accept 163 -> 243 MB/s (~1.5x) Verified: char_type preserved for std::string (char) and std::vector<std::uint8_t> (uint8_t); CBOR/MsgPack round-trips from std::vector<std::uint8_t> unchanged; 1,000,000 randomized documents accept and parse identically via std::string and via std::istream; deserialization/user-defined-input/parser/lexer/conversions/diagnostic- position suites pass (20,480 assertions); warning-clean on g++ and clang in C++11/17/20. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Add optional simdutf backend for bulk UTF-8 validation (JSON_USE_SIMDUTF) The bulk string scanner validates UTF-8 straight from a contiguous buffer. The scalar validator caps at ~0.3-0.7 GB/s on non-ASCII text; a SIMD validator reaches several GB/s. Rather than hand-rolling SIMD UTF-8 validation (easy to get subtly wrong - a from-scratch SSE attempt rejected valid CJK), wire in the vetted simdutf library behind an opt-in switch. simdutf is not header-only (it ships simdutf.cpp and uses runtime CPU dispatch), so it is not vendored: defining JSON_USE_SIMDUTF includes <simdutf.h> and routes the bulk validator through simdutf::validate_utf8; the project supplies and links simdutf. Undefined (the default), nothing external is included and the portable C++11 scalar path is used, so the library stays header-only and its baseline behavior is unchanged. Design keeps behavior identical either way: - scan_string_bulk() now finds the run up to the next quote/escape/control byte (non-ASCII allowed) and validates it in one shot; on the rare validation failure it recomputes the exact valid prefix with the scalar helper, so ill-formed input still falls through to the byte path and is reported at the same position with the same message. - the per-sequence scalar path is factored into scalar_string_bulk_run() and is the default backend; the refactor is behavior-preserving and does not change scalar throughput. Verified: default and JSON_USE_SIMDUTF builds accept/reject/parse identically across 2,000,000 arbitrary-byte documents and 1,000,000 mixed-escape/UTF-8 documents (differential fuzz vs the streaming byte path); lexer/parser/diagnostic-position/deserialization suites pass under both configurations (20,188 assertions with the backend enabled); warning-clean on g++ and clang, C++11 and C++20, both configurations. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Add a contiguous fast path for scanning numbers scan_number() reads a number one character at a time through the input adapter (get()) and appends each byte to token_buffer (add()) before converting. For contiguous input, the per-character get()/add() overhead dominates: it is roughly two thirds of the time spent on number-heavy parsing, far more than the value conversion itself. Add scan_number_bulk_contiguous(), which parses the whole number token straight from the input buffer: it validates and classifies the extent with the same grammar as scan_number()'s state machine, materializes token_buffer in one copy (substituting the locale decimal point exactly as scan_number() does), advances the adapter, and reuses the shared convert_number() tail. On anything it does not recognize as a well-formed number it makes no state change and returns token_type::uninitialized, so the caller falls back to scan_number(), which then produces the exact diagnostic. Errors and their positions are therefore unchanged. The conversion tail is factored out of scan_number() into convert_number() so both scanners share it; the fast path is selected by tag dispatch on the existing bulk_scan capability, so streaming/wide/user adapters are unaffected. Measured on pointer input, g++ 13 -O3: - integers: parse +65%, accept +98% - floats: parse +39%, accept +70% Verified: 2,000,000 randomized number documents (including overflow-range integers, long digit strings and %.17g doubles) parse identically via the contiguous path and the streaming byte path, matching value, type and round-trip text; the locale suite and existing parser/lexer/conversions/ deserialization tests pass; a new "lexer number fast path" test checks contiguous-vs-streaming parity, token classification, and that malformed numbers are rejected identically. Pure C++11, no intrinsics. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Move byte-level scan/parse helpers out of lexer.hpp lexer.hpp had grown by ~600 lines of byte-level helpers that have no dependency on the lexer's template parameters and clutter the state machine. Move them, unchanged, into two focused headers as free functions in namespace detail: - number_parse.hpp: parse_integer_unsigned/parse_integer_signed (now templated on the number type) and parse_float_fast (Clinger's exact double fast path, with the decimal point passed as an argument instead of read from a lexer member). - string_scan.hpp: the SWAR string helpers (is_string_special, swar_string_special, find_string_special, validate_one_utf8, scalar_string_bulk_run) and the backend-dispatched string_bulk_run, including the optional simdutf include and find_string_delimiter. lexer.hpp now includes these and calls the free functions; the methods that touch lexer state (scan_string, scan_number, scan_string_bulk, scan_number_bulk_contiguous, convert_number) stay put. This is a pure code move with no behavior change: lexer.hpp drops from 2357 to 1934 lines, the now-unused <cstdint>/<cstring>/<limits> includes are removed, and the free-function form makes the SWAR helpers reusable elsewhere (e.g. the serializer's string escaping). Verified: default and JSON_USE_SIMDUTF builds compile; 2,000,000 number and 2,000,000 arbitrary-byte-string differential-fuzz documents parse identically to before; lexer/parser/conversions/deserialization/locale/ diagnostic-position suites pass (20,576 assertions); warning-clean on g++ and clang in C++11/17/20; the amalgamation regenerates and passes check-amalgamation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix clang-tidy findings and document JSON_USE_SIMDUTF in the nav - number_parse.hpp: use std::array for the powers-of-ten table (avoid-c-arrays) and `auto` for the cast-initialized result (modernize-use-auto), matching the codebase style (cf. the serializer's utf8d table). Indexing casts keep the -Wsign-conversion build clean. - add JSON_USE_SIMDUTF to the mkdocs navigation so the macro page is reachable. No behavior change; clang-tidy is clean on the new headers and the amalgamation is regenerated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix number fast path for custom string types without assign() The contiguous number fast path materialized token_buffer with token_buffer.assign(data, len), but string_t is only required to provide the minimal interface the rest of the lexer uses (push_back, append, clear, operator[], ...). Custom string types such as the test's alt_string do not implement assign(), so scan_number_bulk_contiguous() failed to compile for them (unit-alt-string), breaking the gcc/clang standards and old-compiler CI jobs. reset() already clears token_buffer, so fill it with append() - which alt_string and std::string both provide and which the string fast path already relies on - instead of assign(). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Satisfy clang-tidy: parenthesize math and drop unused forwarding reference The CI clang-tidy (newer than the locally available version) reported two additional checks on the new code: - readability-math-missing-parentheses: parenthesize the (a * b) + c digit accumulations in number_parse.hpp. - cppcoreguidelines-missing-std-forward: the contiguous-byte-container input_adapter overload took a forwarding reference but only reads data()/size() and never forwards it. It is already disjoint from the generic container overload via SFINAE, so a plain const& is correct and clearer (and keeps the container alive for the whole parse just as before). No behavior change; char_type and routing are unchanged (std::string and std::vector<std::uint8_t> still take the pointer adapter with char/uint8_t char_type), CBOR/MsgPack round-trips and the 2M number fuzz still pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Use std::from_chars (Eisel-Lemire) for float conversion when available The Clinger fast path is exact only for the "easy" subset (<=19 significant digits, \|exp10\| <= 22); high-precision and scientific floats fall through to strtod, where the failed Clinger attempt actually makes parsing a net loss. std::from_chars implements the Eisel-Lemire algorithm in modern standard libraries: locale-independent, correctly rounded, and fast over the whole value range. convert_number() now tries parse_float_from_chars() first (guarded by __cpp_lib_to_chars, so C++11 and libc++-without-float-support keep the Clinger + strtod path unchanged), then Clinger, then strtof. from_chars is used only when it consumes the entire token; a partial parse means a non-'.' locale decimal point, and an under-/overflow (result_out_of_range) also declines - in both cases the existing strtod fallback supplies the exact value and the well-defined +/-inf/0 the parser expects, side-stepping the P4168 divergence between implementations. float and long double now get the fast path too (Clinger was double-only). Measured, C++17, g++ 13 -O3, json::parse/accept: - canada-style floats: ~unchanged (Clinger already covered them) - high-precision (17 digits): parse 2.1x, accept 2.5x - scientific (17 digits + exp): parse 3.6x, accept 4.1x Verified: C++11 (Clinger/strtod) and C++17 (from_chars) parse every value - including subnormals, boundary values, and 1e9999/1e-9999 over-/underflow - to bit-identical results; 2M number-fuzz clean; conversions/deserialization/ locale/number-fast-path suites pass in both C++11 and C++17; clang-tidy clean; warning-clean on g++ and clang in C++11/17/20. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Guard from_chars use on JSON_HAS_CPP_17, not just __cpp_lib_to_chars libstdc++ 15 defines __cpp_lib_to_chars even in C++14 mode (via bits/version.h pulled in by other headers), but <charconv> is only included under JSON_HAS_CPP_17. That made parse_float_from_chars() reference std::from_chars without the header in C++14 builds, breaking gcc-latest, icpx, and the offline-testdata jobs. Gate the use on JSON_HAS_CPP_17 && __cpp_lib_to_chars so it matches the include condition exactly; C++11/14 always take the scalar fallback. Verified by forcing __cpp_lib_to_chars in a C++14 build: the guard suppresses std::from_chars and it compiles. C++17 behavior is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Disable Clinger float fast path under extended FP precision (x87) The contiguous number fast path uses a Clinger-style exact algorithm (significand * 10^scale in double arithmetic), which is only correctly rounded when double operations are evaluated in true 53-bit precision. On the x87 FPU used by 32-bit x86 (FLT_EVAL_METHOD == 2) the single multiply/divide is computed in 80-bit and then double-rounded to double, so a small fraction of values land 1 ULP off. This surfaced as test-cbor_cpp11 and test-msgpack_cpp11 failing on the mingw (x86) job for regression/floats.json: the C++17 builds pass because they take the correctly-rounded std::from_chars path, while C++11 falls back to parse_float_fast(). A 5M-sample check over shortest round-trip decimals reproduces it: 0 divergences with 53-bit doubles, ~1 in 25 000 with 80-bit intermediates; declining to std::strtod fixes all of them. Guard parse_float_fast() on FLT_EVAL_METHOD so it declines whenever the platform evaluates doubles in extended precision, letting the caller use the correctly-rounded std::from_chars / std::strtod path instead. On mainstream x86-64/ARM64 (FLT_EVAL_METHOD == 0) the fast path is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Guard <charconv> include with __has_include for GCC 7 GCC 7 sets __cplusplus to the C++17 value under -std=gnu++1z, so JSON_HAS_CPP_17 is defined, but its libstdc++ ships no <charconv> header (added in GCC 8; floating-point from_chars in GCC 11). The unconditional "#if defined(JSON_HAS_CPP_17) #include <charconv>" therefore failed to compile there: "fatal error: charconv: No such file or directory" in the ci_test_compilers_gcc (7) job. Wrap the include in __has_include(<charconv>), mirroring the library's existing handling of <version> and <filesystem> in macro_scope.hpp. When the header is absent, __cpp_lib_to_chars stays undefined and parse_float_from_chars() takes its scalar fallback, so the from_chars use site (already gated on __cpp_lib_to_chars) is never reached. GCC 8-10, which have <charconv> but no floating-point from_chars, are unaffected: they include the header but still take the fallback. GCC 11+ is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Restore the column when ungetting a newline The contiguous number fast path never reads the character that terminates a number token, while scan_number() reads it and then ungets it. When that character is a newline, get() has already cleared chars_read_current_line, and unget() could only restore lines_read - leaving the column at 0. The two paths therefore reported different columns for the same document: json::parse("[01\n]") -> line 1, column 3 json::parse(stringstream) -> line 1, column 0 Remember the column the newline was read at so unget() can restore it. Both paths now report the position the offending token actually starts at, which also fixes the pre-existing column-0 artifact for streaming input. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Extend the bulk scan fast paths to sized sentinels supports_bulk_scan required IteratorType and SentinelType to be the same type, which excluded std::counted_iterator paired with std::default_sentinel_t - the combination #5268 had already enabled for the memcpy fast path. Such input fell back to the byte-at-a-time scanner even though it is contiguous and its remaining length is computable in O(1). Factor the "distance is computable in O(1)" test into sentinel_is_sized and use it for iterator_is_contiguous, supports_seek, and supports_bulk_scan alike, and share the std::ranges::distance/std::distance dispatch through a remaining_count() helper. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Document JSON_USE_SIMDUTF on the macro overview page The macro was only listed in the API macro index; add it to the supported macros overview alongside the other JSON_USE_* macros, and note that it selects between two definitions of the same inline function and so must be defined identically in every translation unit. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Amalgamate source code Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Cover the counted-iterator bulk scan paths Sized sentinels newly reach the bulk string/number scanners and the seek-based token reconstruction, so exercise both: - diagnostics that quote the offending token, which are rebuilt from the consumed input via copy_consumed_range() - inputs whose count ends before the underlying buffer does, including a closing quote that exists only behind the count, a cut inside an 8-byte SWAR stride, and a cut inside a UTF-8 sequence Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix the SentinelType example on iterator_input_adapter The comment offered "a C++20 sentinel or counted_iterator" as examples of a SentinelType, but std::counted_iterator is the IteratorType - the sentinel it pairs with is std::default_sentinel_t. #5268 corrected the same wording in the API documentation and left the code comment behind. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Do not discard the parse result in the error position check json::parse is declared warn_unused_result, and CHECK_THROWS_WITH_AS evaluates its expression as a discarded statement, so the assertion broke the -Werror builds (GCC -Werror=unused-result, MSVC C4834 under /WX). Compare against the helper that already captures the message instead. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Lock the two number grammars together with a parity test The JSON number grammar is encoded twice: as the scan_number() state machine and as the contiguous fast path. The fast path declining on anything it does not recognize keeps most divergence harmless, but if it ever accepted something the state machine rejects the result would be a silent correctness bug, and the existing test only pinned a hand-written list of numbers. Enumerate every string of length 1..4 over "01.eE+-" (2800 tokens) and require both paths to agree on the parsed value and on the exact error message. Verified to fail if the fast path's grammar is perturbed. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Skip token_buffer for integers on the contiguous path An integer token does not need token_buffer: the number_integer and number_unsigned SAX callbacks take only the value, and the overflow diagnostic rebuilds the text from the input via get_token_string(). Convert straight from the input buffer and materialize the token only for the floating-point tail, which still needs a NUL-terminated buffer for strtod. JSON_DIAGNOSTIC_POSITIONS derives a number's start position from get_string().size(), so the copy is kept when that is enabled. The integer dispatch is factored into convert_integer() and shared with convert_number(), so both scanners keep using one implementation. Integer-heavy input, 400k values, -O3: parse accept gcc 16 +14% +23% clang +15% +22% Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Amalgamate source code Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Cover the bulk string and UTF-8 scanners These paths had no dedicated tests and rested on differential fuzzing only. Add three sections, all comparing the contiguous scanner against the byte-at-a-time one on the parsed value and on the exact error message: - every string of length 1..3 over an alphabet of ordinary ASCII, both specials, a control byte, escape characters, UTF-8 lead and continuation bytes, and a byte that is never valid - each at offset 0 and offset 9, so the bulk scanner sees them with and without a run behind them - every kind of run-ending byte at each offset across two 8-byte SWAR words, so multibyte sequences also straddle the word boundary - the boundaries of every range validate_one_utf8() recognizes: shortest and longest encodings, overlongs, both ends of the surrogate block, U+10FFFF and just past it, and truncated sequences Verified to fail if the bulk validator accepts surrogates, and if the SWAR word test stops detecting control characters. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Format the new string fast path test with astyle The pinned astyle expands a braced-init-list used as a range-for range onto several lines; hoist the two offsets into a named vector instead, which reads better and leaves nothing for astyle to reformat. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix shadowed locals and guard the exception-dependent tests Two problems in the tests added for the bulk scanners, both found by CI: - the inner `const json j` in the counted-iterator diagnostics shadowed the one declared at test-case scope, which -Wshadow rejects on GCC and clang and C4456 rejects on MSVC under /WX; rename them - the new parity checks parse deliberately invalid input, which calls std::abort() rather than throwing when JSON_NOEXCEPTION is defined, so they would have crashed the no-exception build; guard them the way the other tests do json::accept() does not abort, so the UTF-8 range assertions stay compiled without exceptions and keep covering validate_one_utf8() there. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Gate the C++20 iterator classification on JSON_HAS_RANGES Making supports_bulk_scan depend on iterator_is_contiguous meant the trait is now instantiated for every adapter, not only when get_elements() is called. On standard libraries with an incomplete <ranges> that is fatal: libstdc++ 10 evaluates std::contiguous_iterator<std::counted_iterator<T>> by calling std::to_address, which needs an operator-> its counted_iterator does not have, so satisfaction checking is a hard error rather than false. Reported by clang 14 + libstdc++ 10. JSON_HAS_RANGES already encodes exactly this ("libstdc++ < 11 has incomplete C++20 ranges", #4440), so require it for the C++20 branch. Affected toolchains fall back to the pointer-only test and the byte-at-a-time scanner, which parses identically, just without the bulk fast paths. Signed-off-by: Niels Lohmann <mail@nlohmann.me> Amalgamate source code Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Satisfy clang-tidy in the new bulk scanner tests - give the helper lambdas an explicit std::string return type and return braced initializer lists (modernize-return-braced-init-list) - replace the C-style array of test cases with a std::vector (modernize-avoid-c-arrays) - silence pro-type-member-init on the two brace-initialized aggregates; default member initializers would stop them being aggregates in C++11 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Avoid escaped literals in the counted-iterator diagnostics list clang-tidy reads "[\"\\ud834\"]" as a literal better written raw, and the two literals written next to each other in "[\"a\x01""b\"]" as a missing comma. The concatenation was there to stop the hex escape swallowing the following character; build those documents from explicit bytes instead and use raw strings elsewhere. The byte sequences are unchanged. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Reattach convert_number's documentation @gregmarr spotted that convert_integer() was inserted between convert_number() and its doc block, leaving convert_integer() with two stacked blocks and convert_number() with none. Comment only; no code change. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address review comments on the simdutf backend string_bulk_run() had the same `return scalar_string_bulk_run(...)` in both arms of the `#if`. The simdutf arm already falls through when validation fails, so a single return after the `#endif` says the same thing. The JSON_USE_SIMDUTF example showed `#include <simdutf.h>`, which string_scan.hpp already does under the same guard; users only have to put the header on the include path and link the library, not include it themselves. Set the version history entry to 3.13.0, matching the other macro pages documenting unreleased features. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Pin the number error position against a non-newline terminator The comment claimed the reported column is the one the offending token starts at. It is the column reached after the token's last character - which is the actual point of the unget() change: a number terminated by a newline now reports what the same number terminated by a space always did. Assert that equality directly, and add a multi-character token where the start and end columns differ, so the invariant cannot be read off a single-character example. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Do not repeat the integer conversion that just failed scan_number_bulk_contiguous() converts an integer token straight from the input buffer. When the value does not fit, it materializes token_buffer and calls convert_number(), which tried the very same integer conversion again before falling back to floating point. Recording the outcome in number_type skips the second attempt. The resulting token type and value are unchanged: convert_number() reached the float tail either way. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Require a container's value_type to match what data() points at is_contiguous_byte_container accepted any type with a data() returning a pointer to a single-byte integral plus a size(). That is duck typing: the two members say nothing about size() counting the units data() points at. A type where it does not - fixed-size records, say - was routed to the pointer-based adapter and parsed as [data(), data() + size()) bytes, silently truncating input the iterator-based adapter had read in full: struct record_buffer { using value_type = std::array<char, 4>; std::string bytes; const char* data() const; // raw bytes std::size_t size() const; // in records const char* begin() const; const char* end() const; }; json::parse(record_buffer{"[1,2,3,4,5]"}); // parse error at column 3 Requiring the container's own value_type to be that same element type ties the two together. Every contiguous standard container satisfies it, so std::string, std::vector<char>, std::array<char, N> and std::string_view keep the fast path; anything else falls back to the iterator-based adapter, which is always correct. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Only compile the simdutf backend from C++17 on simdutf.h rejects anything below C++17 with an #error, so defining JSON_USE_SIMDUTF in a C++11 or C++14 translation unit did not fail with a message about simdutf being unavailable - it failed to compile at all, taking the library's C++11 support with it. Nothing caught this because no build ever compiled that path. Gate the include and both uses on JSON_HAS_CPP_17, the same way number_parse.hpp gates std::from_chars. Below C++17 the macro now has no effect and the scalar validator runs; it accepts and rejects exactly the same input, so the macro is safe to set project-wide even when some translation units use an older standard. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Test the simdutf backend in CI JSON_USE_SIMDUTF was documented and shipped but never compiled by anything in the repository, so nothing held the backend to the behavior the docs promise. Add JSON_TestSimdutf (OFF by default), which fetches simdutf and defines JSON_USE_SIMDUTF for every test target, and a ci_test_simdutf target that runs the whole suite in that configuration. Because simdutf needs C++17, the suite is built at C++11 as well, so one job covers both the scalar fallback with the macro defined and simdutf itself. The dependency hangs off test_main, whose usage requirements every test target inherits. The library target and the installed CMake package are deliberately untouched: making nlohmann_json link simdutf would put a find_dependency() in the exported package, which is a separate decision. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Warn when JSON_TestSimdutf cannot reach the simdutf backend simdutf needs C++17: without it the dependency does not even compile, and with a C++17 compiler but no C++17-or-later standard under test it builds and then goes unused. Either way the option silently did nothing useful, or broke the configure step outright. Resolve the tested standards first, then check them: when none of them can reach simdutf, skip the dependency and say so, naming which of the two reasons applies and how to fix it. The tests then run against the scalar validator, which is what would have happened anyway. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Use record_buffer::data() so clang does not flag it unneeded The record_buffer test type declares data() and size() so the is_contiguous_byte_container trait can see both and still reject the type on its value_type. data() was never called, so clang's -Wunneeded-member-function (under -Weverything -Werror) failed the C++20 build. Assert that data() points at the underlying bytes: it ODR-uses the member and documents the property the type is meant to demonstrate. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Skip the float fast path when it cannot succeed parse_float_fast() (Clinger) needs a significand below 2^53, so it always declines once the mantissa has 17 or more significant digits. convert_number() called it unconditionally, so those numbers were walked an extra time before strtod had to run anyway. On streaming input, where scanning is byte-at-a-time and there is no compensating win, that made canada.json about 6% slower than develop. Derive the significant-digit count from token_buffer indices - the digits are not scanned again - and skip the call when it is guaranteed to decline. Both scanners pass the offset where the mantissa ends; the count only has to be corrected for a leading "0", which the JSON grammar admits nowhere else. The integer path returns before the check, so integer-heavy input is unaffected. Values are unchanged: this only avoids an attempt that would have failed. Verified bit-exact against develop over every number in canada.json, floats.json, signed_ints.json, unsigned_ints.json, small_signed_ints.json, citm_catalog.json and twitter.json, for both the contiguous and the streaming scanner. parse, streaming develop before after canada.json 19.4ms 20.5ms 19.3ms floats.json 135.9ms 131.8ms 128.0ms parse, contiguous develop before after canada.json 15.5ms 12.9ms 11.7ms floats.json 98.6ms 69.8ms 66.7ms Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>	2026-09-10 17:20:48 +02:00
Niels Lohmann	d69fb8654d	Return the parsed value by move from from_cbor() and friends (#5501 ) The binary entry points end with return res ? result : basic_json(value_t::discarded); The condition operator's second operand is an lvalue, so this is not a case where the return value can be elided or implicitly moved from: every successful from_cbor(), from_msgpack(), from_ubjson(), from_bjdata() and from_bson() call deep-copies the value it just parsed, and then destroys the original. The copy is not cheap, and it is not incidental: basic_json's copy constructor walks the whole value. Parsing a 2 MB CBOR document with 60,000 objects, median of 25 runs, clang 17 -O3: from_cbor 26.99 ms -> 14.65 ms from_msgpack 26.82 ms -> 14.82 ms Moving instead of copying is the entire change; the parsed value is not used again after the return expression is evaluated. There is a second reason to prefer the move. The copy constructor recurses once per nesting level, so the copy is also a stack-overflow path on the return side, on a value the reader has already accepted. That is currently masked because the readers themselves recurse and overflow first (#5104), but it has to be fixed for making them iterative to have any effect. Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-09-10 17:15:34 +02:00
Wu Shuwen	3e2de32226	Remove unused iomanip include (#5516 ) Signed-off-by: dajiaohuang <mikewushuwen@outlook.com>	2026-09-10 08:27:05 +02:00
Niels Lohmann	b17c272f46	Reserve capacity in from_json() object conversion when the target container supports it (#5472 ) * Reserve capacity in from_json() object conversion when supported The object-to-container from_json() overload filled the target container one element at a time without reserving capacity, even when the target type supports reserve() (e.g. std::unordered_map) and the number of elements is already known. This caused unnecessary rehashing while parsing large objects into such containers. Add a reserve-detecting overload (from_json_object_impl), mirroring the priority_tag-based SFINAE technique already used by the array conversion path (from_json_array_impl), so that reserve(size()) is called up front when available and the loop falls back unchanged otherwise (e.g. for std::map). Fixes #5406 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Update doc example outputs for new object-conversion iteration order Reserving capacity in from_json()'s object-conversion path before inserting elements changes libstdc++'s std::unordered_map bucket layout, which changes the iteration order used by get__ValueType_const.cpp, get_to.cpp and operator__ValueType.cpp to print the elements of a converted std::unordered_map<std::string, json>. Verified against a clean develop checkout (built with the same GCC/libstdc++ used in CI) that the old order was produced without this PR's change and the new order is produced with it, and that the three affected examples now match their updated expected output byte-for-byte. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Factor out a reserve-dispatch helper instead of duplicating the object from_json loop Addresses review feedback from @gregmarr on PR #5472: the emplace loop no longer needs to exist twice for the reserve/no-reserve cases. A small from_json_object_reserve() overload pair (SFINAE-dispatched on whether reserve() exists, mirroring the priority_tag technique used elsewhere) either calls reserve() or is a no-op; from_json_object_impl() calls it once. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Inline from_json_object_impl into from_json now that it is called only once Addresses review feedback from @gregmarr on PR #5472: with the reserve loop de-duplicated, from_json_object_impl no longer needs to be a separate function that from_json immediately delegates to. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-09-09 09:54:12 +02:00
Niels Lohmann	8e26b9d4b3	Make contains(json_pointer) return false instead of throwing on unrepresentable array-index tokens (#5495 ) contains(const json_pointer&) is documented to never throw, but a purely numeric reference token that is syntactically a valid array index yet numerically too large to be represented (exceeding size_type's max, or exceeding ULLONG_MAX and causing strtoull() to set errno to ERANGE) made it fall through to array_index(), which throws out_of_range.410/404. Pre-check the token's magnitude the same way array_index() does, but return false instead of throwing, mirroring how the surrounding code already rejects other malformed tokens (leading zero, non-digit characters, "-") without throwing. operator[]/at() are untouched and keep throwing for these inputs. Fixes #5395 Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-09-09 09:53:25 +02:00
Niels Lohmann	c9f1d2d646	Broaden JSON_HEDLEY_WARN_UNUSED_RESULT coverage to pure query functions (#5477 ) * Broaden JSON_HEDLEY_WARN_UNUSED_RESULT coverage to pure query functions Add JSON_HEDLEY_WARN_UNUSED_RESULT to the unambiguous, const, side-effect-free observer functions whose return value is the entire purpose of the call: - dump() - type(), type_name() - all is_* predicates (is_primitive, is_structured, is_null, is_boolean, is_number, is_number_integer, is_number_unsigned, is_number_float, is_object, is_array, is_string, is_binary, is_discarded) - empty(), size(), max_size() - count(...) (both overloads) and contains(...) (all overloads, including the deprecated json_pointer<BasicJsonType> overload) This mirrors the direction the standard library has taken with [[nodiscard]] on the analogous std::vector/std::map members, and catches real bugs such as `j.empty();` (meant `j.clear();`) or `j.contains(k);` with the result thrown away. Deliberately out of scope (left for a separate, later policy decision, per the issue): at(), value(), get(), flatten(), unflatten(), patch(), merge_patch(), begin()/end(), comparison operators, erase(), and emplace(). Compiling the full test suite (tests/src/unit-.cpp) with -Wunused-result -Werror uncovered one real hit: a regression test in unit-regression2.cpp called dump() purely to check it does not throw, discarding the result. Fixed by explicitly casting to void, since the call is intentionally result-less there. Fixes #5410 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix discarded nodiscard results across the test suite for GCC's warn_unused_result A plain (void) cast on a call expression suppresses the C++17 [[nodiscard]] warning but not GCC's warning for functions annotated via the GNU __attribute__((warn_unused_result)) form -- which is what JSON_HEDLEY_WARN_UNUSED_RESULT expands to on GCC. Several existing tests that call a newly-annotated function (dump(), empty()) purely to check that it throws/does not throw, discarding the result via (void), newly warned (and failed -Werror builds) once the annotation was broadened. Route those discards through a small ignore_return_value() helper instead, which actually consumes the value and suppresses the warning on both attribute forms. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Use utils::ignore_return_value() for the issue #1445 dump() discard too Addresses review feedback from @gregmarr on PR #5477: this call site was still using the older "capture in a variable, then (void) it" pattern from before this PR introduced utils::ignore_return_value(), instead of the helper now used at every other discarded-nodiscard-result call site this PR touches. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-09-09 09:50:39 +02:00
Niels Lohmann	a316738cfa	Add JSON_HEDLEY_WARN_UNUSED_RESULT to the current accept() overloads (#5471 ) accept() is a pure query whose only effect is the returned bool; both parse() overloads and the deprecated accept(span_input_adapter&&, ...) overload already carry JSON_HEDLEY_WARN_UNUSED_RESULT, but the two current, recommended accept() overloads were missing it. Add the annotation to match, so discarding accept()'s result now warns under -Wunused-result / [[nodiscard]], as it already does for parse(). Fixes #5407 Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-09-09 09:50:39 +02:00
Niels Lohmann	8c38e270b5	Reject JSON Patch move when from is a proper prefix of path (#5497 ) * Reject JSON Patch move when from is a proper prefix of path RFC 6902 (section 4.4) forbids "from" from being a proper prefix of "path" for a "move" operation: "a location cannot be moved into one of its children." "move" is implemented as remove-then-add with no check for this. For object targets, the subsequent "add" happened to throw as a side effect of resolving through the now-removed parent, but for array targets, removing the "from" element shifts subsequent indices, so "path" silently re-resolves to a different element and the operation "succeeds" with a silently corrupted document. Add a check, before performing the remove/add, for whether "from" is a proper prefix of "path" at the reference-token level. This compares json_pointer's already-unescaped reference_tokens vectors (basic_json is a friend of json_pointer) rather than the raw pointer strings, so that tokens containing escaped '/' or '~' characters are compared correctly, and a token that merely looks like a string prefix (e.g. "/ab" vs "/abc/x") is not mistaken for a pointer-token prefix. When "from" is a proper prefix of "path", throw out_of_range.414. Fixes #5397. Stacked on top of the fix for #5396 (branch issue-5396-patch-remove-primitive-parent), since both touch the same patch_inplace move/remove handling in include/nlohmann/json.hpp. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Add root-pointer and array-append-token edge case tests for the move prefix check Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Replace std::equal with an explicitly-bounded loop in the move prefix check The three-iterator std::equal(first1, last1, first2) form has no explicit end iterator for the second range, which a static analyzer (Flawfinder, CWE-126) flags as a potential over-read even though the preceding size comparison already guarantees the second range is long enough. Rather than argue the point, make the bound visible in the code itself via an explicit loop -- every access to ptr.reference_tokens is now guarded by the same index the loop condition bounds against from_size. (The C++14 four-iterator std::equal(first1, last1, first2, last2) form was tried first as a more minimal fix, but this codebase targets C++11 and that overload is not safely usable under -std=c++11 with all supported standard library implementations.) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Extract the move prefix check into a named helper lambda Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Account for JSON_DIAGNOSTIC_POSITIONS in the move-prefix-check error messages out_of_range::create() includes a "(bytes X-Y)" position annotation when JSON_DIAGNOSTIC_POSITIONS is enabled, which the ci_test_diagnostic_positions CI job builds the whole suite with. The five new out_of_range.414 assertions only checked the annotation-free message. Confirmed JSON_DIAGNOSTICS produces the same (annotation-free) message as the default build for this particular throw site (its path-based annotation is empty at the root, where &result always points here), so only two message variants are needed, not three. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-09-09 09:49:12 +02:00
Niels Lohmann	c9477ccf91	Throw when JSON Patch remove's target path resolves through a primitive or null parent (#5496 ) RFC 6902 (section 4.2) requires the target location of a "remove" operation to exist. operation_remove handled parent.is_object() and parent.is_array(), but had no final else branch: when the resolved parent was a primitive value or null, neither branch matched and the operation silently did nothing instead of failing. Add the missing else branch, throwing out_of_range.413 with wording that matches the existing out_of_range.411 thrown by the analogous "add" case (operation_add) for the same kind of invalid parent. Fixes #5396. Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-09-09 09:49:12 +02:00
Niels Lohmann	d6660cf718	Route hand-rolled diagnostic pragmas through Hedley (#5485 ) * Route hand-rolled diagnostic pragmas through Hedley Several places in the library hand-roll compiler diagnostic suppression with raw `#pragma`/`#ifdef __GNUC__`/`#ifdef __clang__` guards instead of using the Hedley primitives already bundled and used elsewhere (JSON_HEDLEY_DIAGNOSTIC_PUSH/POP, JSON_HEDLEY_PRAGMA, ...). Converted six of the seven listed push/pop pairs to use those primitives instead of raw `#pragma GCC diagnostic`/`#pragma clang diagnostic` text: - include/nlohmann/json.hpp (~3770, ~3863): -Wfloat-equal - include/nlohmann/detail/conversions/to_chars.hpp (~1078): -Wfloat-equal - include/nlohmann/detail/output/binary_writer.hpp (~1844): -Wfloat-equal - include/nlohmann/detail/iterators/iteration_proxy.hpp (~211): -Wmismatched-tags - include/nlohmann/detail/exceptions.hpp (~36): -Wweak-vtables iteration_proxy.hpp did not previously include macro_scope.hpp itself (it only compiled because some other header included earlier in json.hpp happened to pull macro_scope.hpp in first); it now includes it directly like the other detail headers that use Hedley macros, so it is self-contained. Each push/pop pair now uses JSON_HEDLEY_DIAGNOSTIC_PUSH/POP unconditionally (a no-op on compilers that don't need it) and wraps the actual `#pragma ... diagnostic ignored` text in JSON_HEDLEY_PRAGMA so it goes through Hedley's _Pragma()-based emission instead of a raw #pragma line, while keeping the original `#ifdef __GNUC__` / `#if defined(__clang__)` guard around the ignored-pragma itself. Deviation from the issue's suggested transformation: the issue's example replaces the `#ifdef __GNUC__` guard with `#if JSON_HEDLEY_HAS_WARNING("-Wfloat-equal")`. JSON_HEDLEY_HAS_WARNING is implemented purely via Clang's `__has_warning` builtin and evaluates to 0 on real GCC (`#define JSON_HEDLEY_HAS_WARNING(warning) (0)` when `__has_warning` is not defined), so adopting it verbatim would silently stop suppressing -Wfloat-equal on GCC -- a real regression, not just a style change. The existing `#ifdef __GNUC__` / `#if defined(__clang__)` guards were kept for the ignored-pragma to stay behavior-preserving, and only the push/pop/pragma-emission mechanism was routed through Hedley. Two of the seven locations from the issue (the -Wignored-attributes push at the very top of json.hpp and its matching pop after `#include <nlohmann/detail/macro_unscope.hpp>`) were intentionally left unconverted: - The push, at the very top of json.hpp, runs before `detail/macro_scope.hpp` (and therefore hedley.hpp) has been included anywhere in the translation unit, so JSON_HEDLEY_DIAGNOSTIC_PUSH is not yet defined at that point. - The pop runs after `macro_unscope.hpp`, which -- via hedley_undef.hpp -- has already #undef'd every JSON_HEDLEY_* macro (by design, see #5408) precisely so they don't leak to users, so JSON_HEDLEY_DIAGNOSTIC_POP is no longer defined by the time the pop is reached either. Making this one pair work would require either hoisting the ~2000 line vendored hedley.hpp to the very top of the amalgamated single header (a much bigger structural change to single_include than a pure mechanism swap) or special-casing this one pop ahead of the general macro cleanup. Both are riskier than the mechanical, behavior-preserving change requested, so this pair was left as-is. ## Validation - Compiled include/nlohmann/json.hpp and single_include/nlohmann/json.hpp with `-Wall -Wextra -Wfloat-equal -Wmismatched-tags -Wweak-vtables` (clang, which self-identifies as __GNUC__ too): no warnings, same as before the change. - Compiled and ran tests/src/unit-to_chars.cpp, unit-conversions.cpp, unit-iterators1.cpp, unit-iterators2.cpp, and unit-class_parser.cpp against the fixed include/: all pass. - Compiled unit-msgpack.cpp, unit-bjdata.cpp, and unit-ubjson.cpp (which exercise binary_writer.hpp's write_compact_float extensively): all compile cleanly; the vast majority of assertions pass (the only failures are pre-existing environment issues unrelated to this change -- missing generated test-data files, not code correctness). - Ran `make amalgamate`; the single_include diff is limited to exactly the lines touched in include/, with no unrelated reordering. - No real (non-Apple) GCC was available in this environment to test directly; the `_Pragma("GCC diagnostic ...")` text emitted by JSON_HEDLEY_PRAGMA is byte-identical to the prior `#pragma GCC diagnostic ...` text, and the `#ifdef __GNUC__` guard is unchanged, so GCC's behavior is expected to be identical. CI covers the GCC matrix. This PR is stacked on top of #5475 (issue-5408-hedley-undef-leak) since both touch the same files; only the last commit here is new. Fixes #5409. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Guard JSON_HEDLEY_DIAGNOSTIC_PUSH/POP with the same compiler check as the pragma they bracket Addresses review feedback from @gregmarr on PR #5485: the push/pop calls were unconditional, so compilers other than the one the ignored-pragma targets (e.g. MSVC, or GCC where the pair only applies under __clang__) now did a needless push/pop with nothing suppressed in between. Move the existing #ifdef __GNUC__ / #if defined(__clang__) guard to also cover the push/pop, restoring the original zero-overhead behavior on other compilers while still emitting the pragma itself through Hedley. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-09-09 09:48:18 +02:00
Niels Lohmann	c05347dd54	Undefine the four JSON_HEDLEY_* macros that leak after including json.hpp (#5475 ) * Undefine the four JSON_HEDLEY_* macros that leak after including json.hpp include/nlohmann/detail/macro_unscope.hpp includes hedley_undef.hpp to #undef every JSON_HEDLEY_* macro so none of them leak into the including translation unit. Four macros were missing from that list and therefore stayed defined after #include <nlohmann/json.hpp>: - JSON_HEDLEY_PRAGMA - JSON_HEDLEY_PREDICT_TRUE - JSON_HEDLEY_PREDICT_FALSE - JSON_HEDLEY_CLANG_HAS_DECLSPEC_ATTRIBUTE hedley_undef.hpp is generated (via `make update_hedley`) by grepping hedley.hpp for its own internal `#undef JSON_HEDLEY_X` redefinition guards. JSON_HEDLEY_PRAGMA/PREDICT_TRUE/PREDICT_FALSE have no such guard in upstream Hedley, so they were never picked up. The guard for JSON_HEDLEY_CLANG_HAS_DECLSPEC_ATTRIBUTE also has an upstream typo (`JSON_HEDLEY_CLANG_HAS_DECLSPEC_DECLSPEC_ATTRIBUTE`), so hedley_undef.hpp was undefining the wrong (never-defined) name. Fixes: - include/nlohmann/thirdparty/hedley/hedley_undef.hpp: corrected the DECLSPEC_ATTRIBUTE typo and added the three missing #undef lines, keeping the file's alphabetical ordering. - Makefile (update_hedley target): changed hedley_undef.hpp generation to extract macro names directly from every `#define JSON_HEDLEY_...` in hedley.hpp instead of from existing `#undef` guards, so a future `make update_hedley` run undefines every macro Hedley actually defines, even ones without a pre-existing redefinition guard. This was not run in this PR (it would also pull in an unrelated upstream Hedley sync); hedley_undef.hpp was hand-patched instead and single_include was regenerated with `make amalgamate`. - tests/src/unit-no-macro-leak.cpp: new regression test (picked up automatically by tests/CMakeLists.txt's existing unit-.cpp glob) that includes json.hpp and then #ifdef/#error-checks every JSON_HEDLEY_ macro name, so any future leak of any of the 151 vendored macros fails the build, not just the four fixed here. Fixes #5408. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Derive the JSON_HEDLEY_* leak-check test from hedley.hpp at build time tests/src/unit-no-macro-leak.cpp previously hardcoded a static list of ~151 #ifdef/#error checks, one per JSON_HEDLEY_* macro name known at the time it was written. That list would silently go stale the next time `make update_hedley` pulls in a vendor update that adds, removes, or renames a macro, since nothing would force it to be regenerated. Add cmake/scripts/gen_hedley_undef_check.cmake, which derives the full list of JSON_HEDLEY_* macro names directly from include/nlohmann/thirdparty/hedley/hedley.hpp: - tests/CMakeLists.txt uses it (MODE=checks) to (re)generate hedley_undef_checks.inc at configure and build time, and wires the generating custom target as a dependency of the test-no-macro-leak_cpp* targets so it can never build against a stale copy. unit-no-macro-leak.cpp now just #include-s the generated file inside its TEST_CASE instead of carrying the checks itself. - The Makefile's `update_hedley` target now delegates hedley_undef.hpp generation to the same script (MODE=undef, new `update_hedley_undef` target), so the vendored header, the generated #undef list, and the generated test checks are all derived from the same extraction logic and cannot drift apart. This mirrors the approach taken independently in #5415 for the same issue (#5408), credited there to a self-regenerating mechanism that "can never drift again" -- ported into this branch instead of the static list originally proposed here. Verified with a local CMake configure + build + ctest, both against include/ (JSON_MultipleHeaders=ON) and against the amalgamated single_include/nlohmann/json.hpp (JSON_MultipleHeaders=OFF), and by temporarily deleting a #undef line from hedley_undef.hpp to confirm the generated test actually fails on a real leak. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix REUSE compliance failure in gen_hedley_undef_check.cmake The generated file's embedded banner contains the literal text 'SPDX-License-Identifier: MIT' as part of the content being written to hedley_undef.hpp, not as this .cmake script's own REUSE header (it is already covered by the blanket 'Files: *' rule in .reuse/dep5). The reuse tool matched that embedded line as an SPDX tag for the script itself and failed to parse the trailing 'MIT\n")' as a valid SPDX License Expression, breaking ci_reuse_compliance. Wrap the embedded banner in REUSE-IgnoreStart/REUSE-IgnoreEnd comments, as recommended by the tool's own diagnostic output. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-09-09 09:48:17 +02:00
Niels Lohmann	6d86cc0f6b	Speed up whitespace skipping in the lexer (#5490 ) * Speed up whitespace skipping in the lexer lexer::skip_whitespace() called get() for every whitespace byte, and get() checks the (almost always false, once past the first character) next_unget flag on every call. skip_whitespace() now reads its first character with get() (needed to honor a pending unget() left over from finishing the previous token, e.g. scan_number() always ungets the character that terminated the number) and every further whitespace character with a new get_ignoring_pending_unget() variant that skips that branch, since nothing in the loop calls unget(). This is a narrower fix than the full contiguous-buffer bulk-skip suggested in the issue (scan a run of whitespace directly in the adapter's buffer and update position counters once per run). That approach depends on bulk-scan adapter infrastructure (supports_bulk_scan/bulk_data()/bulk_skip()) introduced by the open, unmerged parser-performance PR #5283, which this change intentionally does not depend on or replicate. Building new bulk-scan adapter infrastructure from scratch was judged out of scope/riskier than warranted here, so this change is limited to the safe, always-correct improvement of removing redundant per-character bookkeeping from the existing byte-at-a-time loop; full bulk-skipping is left as future work once #5283 (or equivalent adapter support) lands. Line/column/byte-offset bookkeeping is untouched and verified bit-for-bit identical before and after this change, including for pretty-printed (dump(4)) input with embedded newlines. Fixes #5412 Stacked on top of the PR for #5411 (branch issue-5411-lexer-skip-conversion). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix codegen regression in skip_whitespace() from #5490 Benchmarking found the get()/get_ignoring_pending_unget() split in skip_whitespace() made long whitespace runs (e.g. indentation in pretty-printed JSON) 1.75x-3.2x SLOWER instead of faster, reproducible with both Apple Clang and GCC. Root cause: rewriting the loop from a plain do-while into an initial get() followed by a while-loop defeated the compiler's ability to keep the input adapter's read/end pointers in registers across iterations; both compilers instead reloaded them from memory on every character. The function split itself was not the problem (it still fully inlines); the loop's control-flow shape was. The fix keeps the same two-function structure but restores a do-while shape (guarded by an if for the "first char not whitespace" case), which lets both compilers hoist the pointers back into registers, matching or beating pre-#5490 performance. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Share the position-counter bump between get() and get_ignoring_pending_unget() Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Extract current_is_whitespace() to deduplicate skip_whitespace()'s two whitespace checks Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Use a raw string literal for the multi-line error-position test input Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix clang-tidy raw-string-literal finding and guard a new test against JSON_NOEXCEPTION The issue #5412 whitespace-skipping test added a check_error() helper that relies on catching json::parse_error to verify the exception message; under JSON_NOEXCEPTION, JSON_THROW aborts instead of throwing, which crashed ci_test_noexceptions (and cascaded into the other ci_cmake_options jobs). Guard the whole section with #if !defined(JSON_NOEXCEPTION), matching the existing pattern used by sibling tests in this file. Also switch one escaped string literal to a raw string literal to satisfy clang-tidy's modernize-raw-string-literal check. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-09-09 09:46:26 +02:00
Niels Lohmann	faa35cc647	Skip integer conversion in accept()/SAX validation when the value is unused (#5484 ) * Skip integer conversion in accept()/SAX validation when the value is unused lexer::scan_number() always converted every numeric token with strtoull()/strtoll() before returning, even though accept() (and any consumer using json_sax_acceptor) immediately discards the converted value. For value_unsigned/value_integer tokens whose digit count already guarantees the value fits into 64 bits, the conversion cannot change the accept/reject decision (such tokens are always finite and unconditionally accepted), so scan_number() can skip strtoull()/ strtoll() entirely in that case when the caller signals it does not need the value. Numbers with more digits keep using the exact, unmodified conversion path, so overflow reclassification to value_float (and the finiteness check on it) is unaffected. parse() and value_float handling are completely unchanged. Fixes #5411 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Document why the digit-count fast path is safe regardless of number_unsigned_t/number_integer_t width Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Avoid temporary-string concatenation flagged by clang-tidy in the differential test Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-09-09 09:46:26 +02:00
Qatadaha Bin Matloob	09b6b6b5ba	Fix to_bjdata() emitting unparsable output when _ArraySize_ is not an array (#5455 ) * Fix to_bjdata() emitting unparsable output when _ArraySize_ is not an array write_bjdata_ndarray() never checked that _ArraySize_ is an array. The shape is written verbatim as the header length, so a null shape emitted 'Z' and an object shape emitted '{' after the '#', neither of which from_bjdata() accepts, and the round-trip guarantee in the BJData docs was broken. Both slipped through the existing validation: for null, empty() is true so the element count starts at 0 and the per-dimension loop never runs, and for an object the loop walks its values, which can satisfy the non-negative integer check. When _ArrayData_ then matched that count, the writer took the ndarray path. Require the shape to be an array, so anything else falls back to a plain object encoding that round-trips, as the fallback rule in the docs already specifies. Signed-off-by: qatcod <79017227+qatcod@users.noreply.github.com> * Document that _ArraySize_ must be an array in the ndarray requirements The list at bjdata.md is the exhaustive set of conditions for the ndarray encoding, but it only implied this one through 'every entry of'. Signed-off-by: qatcod <79017227+qatcod@users.noreply.github.com> --------- Signed-off-by: qatcod <79017227+qatcod@users.noreply.github.com>	2026-09-05 20:30:22 +02:00
Avionic Harshit	3c875683a5	Make diff() linear when an array shrinks (#5461 ) Signed-off-by: avionicharshit-byte <harshitavionic@gmail.com>	2026-09-05 09:52:53 +02:00
Qatadaha Bin Matloob	137a40b9aa	Compare integers with floats exactly instead of widening the integer (#5459 ) The mixed number arms of JSON_IMPLEMENT_OPERATOR cast the integer to number_float_t before comparing. Past the float's mantissa that cast is lossy: 2^63-2 and 2^63-1 both round to 2^63, so each compares equal to that float while differing from each other. Equality is therefore intransitive and the ordering is not a strict weak ordering, which makes std::sort over such values, or using them as keys in std::set or std::map, undefined behavior. Compare the two exactly instead. The integer's range is a power of two the float represents exactly, so a float outside it is ordered by magnitude alone; inside it, truncating the float is exact, and the integer parts and then any fractional part decide. The helper hands back a pair whose comparison with the original operator reproduces that ordering, which keeps every operator's return type as it was, including partial_ordering for the spaceship. A NaN operand is returned in both members, so NaN stays false for the relational operators and unordered for <=>. Values a float represents exactly still compare equal, so json(1) == json(1.0) is unchanged. Signed-off-by: qatcod <79017227+qatcod@users.noreply.github.com>	2026-09-05 09:43:07 +02:00
elix3r	35705d79d8	Fix update(merge_objects=true) throwing on primitive-to-object merge (#5414 ) When merge_objects is true, recurse only if the existing value is an object. Otherwise overwrite, matching the documented "all other values are overwritten as usual" behavior. Fixes #5402 Signed-off-by: elix3r <157088510+22elix3r@users.noreply.github.com>	2026-08-28 13:38:28 +01:00
Krishnanand G	2f025f401e	Throw other_error.502 when UBJSON use_type is set without use_size (#5380 ) * Throw other_error.502 when UBJSON use_type is set without use_size Fixes #5321 Signed-off-by: Krishnanand G <118352827+Krishnanand-G@users.noreply.github.com> * Scope UBJSON use_type check to container branches and expand tests Signed-off-by: Krishnanand G <118352827+Krishnanand-G@users.noreply.github.com> * Re-amalgamate single_include/json.hpp The previous commit updated the split headers but the amalgamated file didn't go back through astyle before I committed it, so CI's amalgamation check caught formatting drift in json_fwd.hpp and a few noexcept clauses in basic_json, plus one doc example. None of it touches the UBJSON logic. Applied the patch CI generated to bring single_include back in sync. Signed-off-by: Krishnanand G <118352827+Krishnanand-G@users.noreply.github.com> --------- Signed-off-by: Krishnanand G <118352827+Krishnanand-G@users.noreply.github.com>	2026-08-25 07:16:10 +01:00
Sahil_Kamate	b5378e8deb	Fix CBOR tag handlers not recognizing tags 0-5 and 21-23 (#5331 ) * Fix CBOR tag handlers not recognizing tags 0-5 and 21-23 The tagged-item switch in binary_reader::parse_cbor_internal() only handled head bytes 0xC6-0xD4 and 0xD8-0xDB. Bytes 0xC0-0xC5 (tags 0-5: date/time, epoch, bignum, decimal, bigfloat) and 0xD5-0xD7 (tags 21-23: base64url, base64, base16 conversion hints) fell through to the default case and were reported as invalid bytes, even under cbor_tag_handler_t::ignore and ::store, despite being valid CBOR major-type-6 tags per RFC 8949. Add the missing case labels so the full 0xC0-0xDB range is handled uniformly. Extend the "Tagged values" test in unit-cbor.cpp to cover 0xC0-0xD7, and update the CBOR docs to state the corrected tag range. Fixes #5315 Signed-off-by: sahilkamate03 <45514385+sahilkamate03@users.noreply.github.com> * Fix stale CBOR tag docs and add store-mode binary-payload test The "Incomplete mapping" warning still listed tags 0-5 (date/time, bignum, decimal fraction, bigfloat) and 21-23 (expected conversions) as unsupported, even though they now parse correctly under cbor_tag_handler_t::ignore/store, same as 0xC6..0xD4/0xD8..0xDB. Remove those five bullets and cross-reference the "Tagged items" warning below, matching the equivalent docs fix landed independently in PR #5367. Also add a cbor_tag_handler_t::store test that wraps a binary payload (not just a string) for every byte in 0xC0..0xD7, confirming these tags are unwrapped the same way as 0xC6..0xD4 rather than mistaken for the 0xD8..0xDB binary-subtype marker syntax, per review feedback on #5331. Signed-off-by: sahilkamate03 <45514385+sahilkamate03@users.noreply.github.com> --------- Signed-off-by: sahilkamate03 <45514385+sahilkamate03@users.noreply.github.com>	2026-08-19 20:19:47 +02:00
ljcjclljc	6285225fd0	Fix integer comparison bug (#5211 ) * Fix integer comparison bug Signed-off-by: ljccjlljc <939159710@qq.com> * commit Signed-off-by: ljccjlljc <939159710@qq.com> * Remove generated CI artifacts and update amalgamation Signed-off-by: ljccjlljc <939159710@qq.com> * Silence cpplint braces warning in comparison macro Signed-off-by: ljccjlljc <939159710@qq.com> * Update amalgamation after cpplint fix Signed-off-by: ljccjlljc <939159710@qq.com> * Add mixed signed and unsigned comparison regression test Signed-off-by: ljccjlljc <939159710@qq.com> * Clarify mixed signed and unsigned comparison handling Signed-off-by: ljccjlljc <939159710@qq.com> * Expand mixed signed and unsigned comparison tests Signed-off-by: ljccjlljc <939159710@qq.com> --------- Signed-off-by: ljccjlljc <939159710@qq.com>	2026-08-12 09:21:19 +02:00
Dmitry	1c136a66c4	Move the CBOR doc block to the function it describes (#5363 ) The block documenting get_char and tag_handler sat above get_cbor_negative_integer(), which takes neither, so Doxygen attached it there and parse_cbor_internal() was left undocumented. Comment placement only. Signed-off-by: Dmitry <45711841+darkdi@users.noreply.github.com>	2026-08-06 08:30:15 +02:00
Niels Lohmann	bacdabd176	Fix start_pos() for strings containing escape sequences (#5361 ) The diagnostic position of a string value was derived by subtracting the parsed value's length from the end position. Escape sequences make the source token longer than the value it parses to, so the reported start position landed inside the string, one byte off per escape sequence: input: {"a":"\n\n\n\n\n\n"} start_pos() == 11, so the reported range covered n\n\n\n" instead of the documented "\n\n\n\n\n\n" This contradicts the documented behavior of start_pos(), which is the position of the opening quote, and it also corrupted the "(bytes N-M)" part of JSON_DIAGNOSTICS exception messages. Strings with multi-byte UTF-8 but no escapes were unaffected, which is why this went unnoticed. Record the offset of the token in the lexer when it starts scanning and use that, instead of reconstructing it from the parsed value. Booleans, null and numbers already reported correct positions and are unchanged. The new lexer member and accessor are compiled only when JSON_DIAGNOSTIC_POSITIONS is enabled, which is already part of the ABI tag, so the default build is unaffected. Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-08-05 16:01:23 +02:00
Niels Lohmann	d5647e6a3b	Resolve the TODO(niels) in get_ubjson_string (#5355 ) The comment asked whether the no-op marker 'N' may be ignored when a string is read. It may not: at that point the next byte must be a string length type specification, and 'N' is not one. No-ops at positions where a value may start are already consumed by the callers through get_ignore_noop(), so nothing is lost by not skipping them here. Replace the TODO with a comment stating that, and add regression tests pinning both directions: a no-op is accepted at top level (also repeated), before and after an array element, and before an object key, between key and value, and before the closing brace of an object of unknown size; it is rejected where a length type specification is expected, i.e. after the 'S' marker of a string value and as the key length of an object of known size. Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-08-05 14:47:40 +02:00
Niels Lohmann	9a091d2b82	Do not write BJData ndarrays whose size overflows std::size_t (#5362 ) * Do not write BJData ndarrays whose size overflows std::size_t write_bjdata_ndarray() multiplied the _ArraySize_ dimensions into a std::size_t without checking for overflow. A product that wraps around to a value that happens to match the size of _ArrayData_ passed the length check, and the writer emitted an ndarray header announcing an element count that cannot be represented: {"_ArrayType_":"uint8","_ArraySize_":[9223372036854775808,2],"_ArrayData_":[]} was encoded as 5b 24 55 23 5b 4d 00 00 00 00 00 00 00 80 69 02 5d, an ndarray of 2^64 elements followed by no data. Reading that back throws out_of_range.408 ("excessive ndarray size caused overflow"), so to_bjdata produced output that from_bjdata rejects. This is reachable by parsing untrusted JSON and re-encoding it as BJData. Mirror the overflow check the binary reader already performs, and also reject a single dimension that does not fit into std::size_t, which the previous cast silently truncated where std::size_t is narrower than 64 bits. Such objects now fall back to a plain object encoding, which is what the surrounding type and length validation already does for annotations it cannot represent, and they round-trip unchanged. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Document when to_bjdata converts a JData annotation to an ND-array The BJData page described the 1-D vector case as the only situation in which an object carrying _ArrayType_/_ArraySize_/_ArrayData_ is not written as a compact ND-array. The writer has always had several other fallbacks -- an unknown _ArrayType_, a dimension that is not a non-negative integer, an _ArrayData_ whose length does not match the product of the dimensions, and elements that are not numbers of the annotated kind -- all of which cause the value to be serialized as a regular JSON object instead. Spell out the conditions, including the size-overflow check added in the preceding commit, so the documented behavior matches the implementation. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-08-05 13:44:15 +02:00
Angadi56	dca9d49a33	reject out-of-range code points in UTF-32 wide-string input (#5348 ) * reject out-of-range code points in UTF-32 wide-string input Signed-off-by: Angadi Yashaswini <angadi@digiscrypt.com> * remove useless cast to char_traits<char>::int_type Signed-off-by: Angadi Yashaswini <angadi@digiscrypt.com> --------- Signed-off-by: Angadi Yashaswini <angadi@digiscrypt.com>	2026-08-05 13:43:36 +02:00
Angadi56	68f0722a19	remove discarded array from parent object in end_array (#5342 ) * remove discarded array from parent object in end_array Signed-off-by: Angadi Yashaswini <angadi@digiscrypt.com> * remove discarded scalar value from parent object in handle_value Signed-off-by: Angadi Yashaswini <angadi@digiscrypt.com> --------- Signed-off-by: Angadi Yashaswini <angadi@digiscrypt.com>	2026-08-03 19:13:31 +02:00
Angadi56	5f121d8c50	avoid sign extension in char_traits<signed char>::to_int_type (#5336 ) * avoid sign extension in char_traits<signed char>::to_int_type Signed-off-by: Angadi Yashaswini <angadi@digiscrypt.com> * spell out-of-range signed char constants as negative values (MSVC C4309) Signed-off-by: Angadi Yashaswini <angadi@digiscrypt.com> --------- Signed-off-by: Angadi Yashaswini <angadi@digiscrypt.com>	2026-08-03 08:35:53 +02:00
Niels Lohmann	585929bff9	Fix Clang deprecation warning for json_pointer operator== with ordered_json (#5289 ) is_comparable used a flat && chain to both exclude json_pointer/string comparisons (added for #4621) and check whether Compare(A, B) is well-formed. Naming std::is_constructible<decltype(...)> as a later operand of that chain still causes the decltype to be substituted regardless of the first operand's value, since the operands aren't lazily deferred like std::conjunction would defer them. That instantiates the transparent std::equal_to<>::operator() used by ordered_json, whose noexcept-specifier evaluates the deprecated json_pointer/string operator==, which Clang (unlike GCC in this case) warns about even though the result is discarded. Split is_comparable so the Compare(A, B) checks live in a separate helper that is only referenced from the specialization selected when is_json_pointer_of is false, so the decltype is never written when A/B are a json_pointer/string pair, regardless of compiler. Signed-off-by: Niels Lohmann <niels.lohmann@gmail.com>	2026-08-03 08:20:41 +02:00
tomatotomata	2222d386c9	fix: check CBOR tagged subtype reads (#5339 )	2026-07-31 22:06:55 +02:00
Angadi56	d94cbd99dc	reject CBOR array/map length equal to the indefinite-length marker (#5274 ) * reject CBOR array/map length equal to the indefinite-length marker Signed-off-by: Angadi Yashaswini <angadi@digiscrypt.com> * reject CBOR lengths that do not fit in std::size_t via value_in_range_of Signed-off-by: Angadi Yashaswini <angadi@digiscrypt.com> --------- Signed-off-by: Angadi Yashaswini <angadi@digiscrypt.com>	2026-07-31 18:04:30 +02:00
hum4nBeingandAJ369ninja	bc48951128	Fix UBJSON high-precision floating-point overflow handling (#5323 ) This adds a std::isfinite check to the UBJSON floating-point parsing path, throwing out_of_range.406 on overflow. This makes the UBJSON parser's behavior consistent with the normal JSON parser. Fixes #5322. Signed-off-by: AJ369ninja <abhishek.j@iitg.ac.in> Co-authored-by: AJ369ninja <abhishek.j@iitg.ac.in>	2026-07-31 18:04:13 +02:00
Angadi56	fd72ecfc8c	validate ndarray element types in write_bjdata_ndarray (#5301 ) * validate ndarray element types in write_bjdata_ndarray Signed-off-by: Angadi Yashaswini <angadi@digiscrypt.com> * read ndarray elements through get<> instead of a fixed union member _ArrayType_ names the wire type, not how the value is stored: parsing keeps a non-negative integer as number_unsigned while the C++ API keeps an int literal as number_integer. Selecting the union member from the type marker therefore reads the inactive alternative for one of the two, so read through get<> instead, which dispatches on the active member. Also reject a negative _ArraySize_ entry, which is not a usable dimension, and cover the parse-built path in the tests. Signed-off-by: Angadi Yashaswini <angadi@digiscrypt.com> --------- Signed-off-by: Angadi Yashaswini <angadi@digiscrypt.com>	2026-07-31 08:52:27 +02:00
Yash Bavadiya	dd24e2dffd	check all BSON reads and add an EOF check for booleans (#5332 ) Signed-off-by: Yash Bavadiya <krbavadiya11@gmail.com>	2026-07-30 23:44:31 +02:00
Patrick10199	868506dcc0	Fix CBOR half-float assertion bounds (#5335 ) Signed-off-by: Patrick Armstrong <patrick@erpassistant.ai>	2026-07-30 23:38:13 +02:00

1 2 3 4 5 ...