1135 Commits

Include renames

Author	SHA1	Message	Date
Niels Lohmann	bae6216491	Merge develop into remove-dead-checks/binary-formats ... Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-11-02 18:27:09 +01:00
Niels Lohmann	6d5581c4e8	CBOR/MessagePack readers: leave one-element arrays to the general path (#6008) ... enter_cbor_array and enter_msgpack_array called the out-of-line scalar run for every non-empty array. When the array's element is an object or an array, as on every level of [[...]] or the single ring of a GeoJSON polygon, the call reads nothing; it made 1,000 arrays nested in each other cost 4 % more instructions (GCC 13) and up to 8 % more time (Apple clang) than without the run. #5975 had fixed that with an inline pre-check, which was later dropped after measuring canada only. The run is now entered only for arrays of two or more elements: the same comparison as before, so no other row changes. Arrays of one scalar, which the run read faster, now take the general path; they are rare in documents (one in citm_catalog, none in twitter or canada). The new benchmark FromBinaryShape/singles measures them, next to FromBinaryShape/nested. Instructions per repetition, new/old, GCC 13 callgrind: nested arrays 0.973 (CBOR) and 0.971 (MessagePack), arrays of one scalar 1.037 and 1.021, every other row within 0.15 %. Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-11-02 18:25:44 +01:00
Niels LohmannandClaude	590b622d48	parse(): propagate std::bad_alloc instead of crashing on a half-constructed member (#5991) ... * Create the DOM parser's member value before placing it (#5985) json_sax_dom_parser::set_member() constructed the new value in place over the null member. basic_json(value_t) sets the type before the json_value allocates, so a std::bad_alloc left a member typed object/array with a null pointer, and the destructor crashed instead of propagating the exception. Construct the value first and move it over the member (the move constructor is noexcept). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix CI for the parse() bad_alloc sweep test (#5985) - sweep_allocator: provide max_size(), construct() and destroy(), which GCC 4.8's std::map calls on the allocator directly. - Declare the non-throwing test lambdas noexcept (-Werror=noexcept on recent GCC). - Give the sax_parse lambda an explicit return type; Clang 3.4 crashes on the deduced one in C++11 mode. - Skip the parser callback sweep with iterator debugging (MSVC debug builds, clang-cl), where containers allocate debug proxies inside noexcept move constructors and terminate instead of throwing, as in the other allocator tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N5NasAgbDGG28MqXjQvLqp Signed-off-by: Claude <noreply@anthropic.com> * Construct a scalar member in place; only a throwing value goes through a temporary Creating the value first and moving it over the member is needed only when creating it can throw (a string, a binary value or a container). A scalar or null is again constructed in place, without the extra move and destroy per member (and, with JSON_DIAGNOSTICS, without the extra set_parents()). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Spell the nothrow trait without && and fix the construct_member comments The dispatch in set_member tests std::is_nothrow_constructible, not whether the value is a scalar. Say so in the comments, and drop the && from the trait argument (equivalent for a forwarding reference) so that astyle no longer inserts spaces around the template brackets. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Say "for example" in the construct_member comments The overloads are selected by std::is_nothrow_constructible, not by the kind of value: a custom type that cannot throw takes the first one. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> Signed-off-by: Claude <noreply@anthropic.com> Co-authored-by: Claude <noreply@anthropic.com>	2026-11-02 18:25:30 +01:00
Niels Lohmann	100f676b3e	Restore the two commits of #6005 that the stack merge dropped (fixes the build of develop) (#6012) ... * basic_json, iterators, serializer: remove checks that invariants make dead Branch coverage of the test suite and of 24k mutated inputs showed these conditions as never taken; each is implied by the code around it: - copy_object_level(): `src_it != src_object.cend()`; dst was built from exactly src_object.size() unique keys and src_it advances at most once per member - the general comparison operators (equal_general, less_general, three_way_general): the five same-type scalar cases, which the inline fast path of every operator handles before calling them (now an assert); and the NaN half of compares_unordered() in the mixed-type branch, reached only after all six number pairs returned, where only a discarded operand is unordered - serializer, ensure_ascii: the reset of `bytes` under error_handler_t::keep and the surrogate-pair escape in the UTF8_ACCEPT case; every well-formed multi-byte character is escaped by write_u_escapes() before decode() sees it, so decode() only ever accepts DEL here, a single byte of which nothing was buffered (now asserts) - iter_impl(pointer): the if/else chain assigned value-initialized iterators to the value-initialized m_it No change in behaviour: a build that aborts on each removed side passes the test suite, the fuzz corpora and 23.2M dumps of all 1-, 2- and boundary 4-byte strings in both ensure_ascii modes. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * iter_impl: keep the type-dependent initialization in the constructor The check is not needed for correctness (m_it is value-initialized), but without it GCC 13 generates slower iteration loops: a recursive walk over a parsed document takes 0.5-2.7 % longer (callgrind: +1.9 % instructions on twitter.json, +0.95 % on canada.json). Clang generates identical code either way. See the measurements in #6005. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-11-02 18:23:44 +01:00
Niels Lohmann	596028c460	Drop the null_result parameter JSON_IMPLEMENT_OPERATOR no longer uses (#6005) ... Without the same-type scalar cases, the macro body has no use for it, and JSON_IMPLEMENT_SCALAR_CASES is now used only by JSON_IMPLEMENT_SCALAR_FAST. Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-11-02 17:54:39 +01:00
Niels Lohmann	5c4f585ac2	Lexer, parser, stream adapter, SAX: remove checks that invariants make dead (#6004) ... Branch coverage of the test suite and of 24k mutated inputs showed these conditions as never taken; each is implied by the code around it: - lexer: `next_unget` in get_codepoint_bulk() and scan_literal_bulk(). unget() is only called at the end of scan_number(), in scan_comment() and in skip_bom(); every token scan starts with a plain get() in skip_whitespace() that consumes a pending unget, and nothing between that get() and the bulk helpers ungets (now asserts, as in get_after_string_run()) - lexer: `number_type == value_integer` in convert_integer(): both callers exclude value_float and the type has three values - parser: `!context.empty()` in exception_message(): all callers pass a literal - input_stream_adapter: `refill` in get_character_slow(): the function is only called after get_character() found the get area exhausted, and the variable recomputed that comparison - json_sax_dom_callback_parser: `ref_stack.back()` in key() (the null case returned a few lines earlier, and the callback in between cannot reach ref_stack), `is_structured()` in end_object() (non-null entries are only pushed for objects and arrays), `!key_stack.empty()` in current_key() (a kept object pushed its key before any value) No change in behaviour: a build that aborts on each removed side passes the test suite, the fuzz corpora and 40k random documents through every input adapter in four macro configurations. Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-11-02 17:54:39 +01:00
Niels LohmannandClaude Opus 5.5	1d69ef9ab4	Raise line coverage to 99.9 %: fix the measurement, remove unreachable code, add tests (#5997) ... * CI: measure line coverage without lcov's unexecuted-block adjustment Drop geninfo_unexecuted_blocks=1 from the lcov capture of ci_test_coverage: it reset every line of an inlined or templated function to 0 as soon as one copy had an unexecuted block, which understated the line coverage by about three points. Add -fvar-tracking-assignments to both coverage builds. At -O0, GCC inlines always_inline functions before it places the coverage counters, so the call sites of such helpers (put_literal() and others) got no line record and showed as 0 in an -O3 translation unit that never reaches them. With the flag, the call sites carry counts in every translation unit. The generated code is identical. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Remove unreachable code found by the coverage analysis Remove code that no input can reach, and mark the lines that cannot be covered with LCOV_EXCL_LINE: - BJData writer: the dry_run parameter of the ND-array element writers was always true; replace them by range checks. - BON8 writer: call throw_integer_out_of_range() directly instead of write_bon8_value() for values that always throw. - binary_writer: integer_bits() is only called for integers of up to 64 bits. - serializer, view serializer, lexer: replace guards that the callers make unreachable by assertions or by the simpler code (surrogate pairs under ensure_ascii, next_unget, '-' at the end of the buffer). - json_view: drop unreachable branches in scalar(), set(), check_utf8_scan(), append_input_impl() and unescape(). - Mark exception-cleanup braces, 2 GiB limits, big-endian and 32-bit code, and fall-through case labels with LCOV_EXCL_LINE. No behavior changes for any input that reaches these functions. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Tests: cover the remaining lines of the library headers Add tests for the lines of include/nlohmann that were not covered: the binary readers and writers, the lexer and locale paths, the json_view modules, the custom number, key and array types, json_pointer, json_patch, diff and the iterators. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix test failures in the coverage tests - unit-custom-object-key-type: remove the unused make_strict_key(strict_key) overload, the unused range constructor and default constructor, and guard data() with JSON_DIAGNOSTICS (unused-function/-member-function errors with GCC and clang -Weverything). - unit-json_view_compare: move the two new sections to the new unit-json_view_compare_values.cpp, as the object file got too large for the MinGW assembler again, and use a std::map with std::greater instead of std::unordered_map, which does not accept the incomplete basic_json (clang 3.9, icpc). - unit-json_view_dump: rename the variables that hid outer ones (MSVC C4456). - unit-stream-bulk-scan: restore the real size of the oversized get area before the adapter is destroyed (JSON_PRECISE_STREAM_POSITION reads the character at the read position), and skip the test where the streambuf cannot hold a get area larger than an int (MSVC). - unit-regression1: with JSON_NO_THREAD_LOCAL, copies are built member by member and keep correct timestamps, so check the corresponding outcome. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N5NasAgbDGG28MqXjQvLqp Signed-off-by: Claude <noreply@anthropic.com> * Return std::size_t from the hash helper in the ordered_json test MSVC (Win32) rejected the implicit conversion of the unsigned long long product to the 32-bit std::size_t (C4244 as an error). Also note when JSON_NO_THREAD_LOCAL is set in the regression test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N5NasAgbDGG28MqXjQvLqp Signed-off-by: Claude <noreply@anthropic.com> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> Signed-off-by: Claude <noreply@anthropic.com> Co-authored-by: Claude <noreply@anthropic.com>	2026-11-02 17:54:38 +01:00
Niels Lohmann	ff0bdfdef3	Keep the byte range in exceptions about the root value (#5990) ... * Keep the byte range in exceptions about the root value (#5987) With JSON_DIAGNOSTICS and JSON_DIAGNOSTIC_POSITIONS both set, exception::diagnostics() and the json_view diagnostics() returned early for a value without a path (the root), dropping "(bytes a-b)". Build the "(path) " prefix only when there are tokens and always append the byte range. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Guard the null value of a value-initialized iterator; drop the unneeded found flag With the root's byte range, diagnostics() reaches get_byte_positions() also with JSON_DIAGNOSTICS when there is no value, as for a value-initialized iterator (invalid_iterator.212). get_byte_positions() now returns no range for a null value, the same change as #6003, so either can be merged first. In the view, a value that value_path() does not find leaves no tokens, so the found flag added nothing to the tokens.empty() test. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-11-02 17:50:40 +01:00
Niels LohmannandClaude	494a74c242	insert(): strong guarantee for single elements, basic guarantee for the rest (#5996) ... * Make basic_json::insert() provide its documented strong exception guarantee (#5986) A failed allocation while the new elements were copied left the value changed, and with JSON_DIAGNOSTICS stale parent pointers behind. The extra work now happens almost only on the failure path: - Array insert(pos, cnt, val) and insert(pos, first, last) append the copies at the end and rotate them into place. If a copy throws, the appended elements are erased again (noexcept) and the parents are set. Moving a basic_json cannot throw, so reallocation is safe. This is no slower than vector::insert (faster for numbers), needs no temporary array, and keeps the elements that libstdc++ lost on reallocation. - Array insert(pos, val) copies a non-scalar value first and inserts it by move, except at the end, where vector::insert is strong. Scalars keep the direct path. - The object range insert(first, last) into an empty object uses the container's own range insertion and clears the object on an exception. Into a non-empty object it inserts member by member and logs one bit per source member (a new member is detected by the size change; no allocation for the first 256 members); on an exception the members added are erased by walking the source range. No assumption that key_type is string_t. - The parents are set when a failed insertion is undone. The regression test fails every allocation of every insert() overload in turn, for json and ordered_json, and checks the value and its parents. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix clang-tidy and -Wnrvo findings in insert rollback helpers Declare the rollback helpers non-movable, replace the nested conditional and C array in object_insert_rollback, silence the intentional copy in the allocator test, and avoid the missed NRVO in the test factory lambda. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N5NasAgbDGG28MqXjQvLqp Signed-off-by: Claude <noreply@anthropic.com> * insert(): strong guarantee for single elements, basic for the rest Rework of the previous two commits, following the discussion in #5996: the standard containers give the strong guarantee only for a single element (std::vector: at the end, or if the exception does not come from copying T; std::map: single-element insert), and copying a basic_json allocates. Rebuilding a rollback for the count, range and object-range overloads on top of array_t/object_t made the code verbose and slower. - insert(pos, const basic_json&): at end() forward as before (the standard covers it); elsewhere copy first and insert by move, as libstdc++ does internally (libc++ shifts the tail before copying). - insert_iterator() and object insert(first, last) set the parents in a scope guard, so a failed insertion leaves a valid value with JSON_DIAGNOSTICS too (the next assert_invariant() aborted before). - Docs: strong guarantee for (1), basic guarantee for (2)-(5); the argument checks throw before anything is changed. - Test: allocation-failure sweep checks the strong guarantee for (1) and validity, type and parents for the other overloads. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * insert(): document the exception safety of overloads 2 to 5 once The four basic-guarantee statements were identical up to array/object. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * insert(): simplify the wording of the basic guarantee Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> Signed-off-by: Claude <noreply@anthropic.com> Co-authored-by: Claude <noreply@anthropic.com>	2026-11-02 17:48:29 +01:00
Niels LohmannandClaude	66e2eeeec0	Do not construct the base class when destroying a container (#6000) ... * Do not construct the base class when destroying a container (#5995) json_value::destroy_container() and destroy_walk() used default- constructed basic_json values as stand-ins, which construct a CustomBaseClass that belongs to no value. A base class whose default constructor allocates made the noexcept destructor allocate again, and terminate if that allocation failed, which #5762 set out to prevent. Use data (type and value, no base class) as the stand-in instead, and pass the children's m_data to the walk helpers. Add a test with a counting base class: destroying a value constructs no base object and destroys exactly one per value, also for values nested deeper than the recursion depth limit. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Release the test value with = nullptr instead of reset() clang-tidy (readability-ambiguous-smartptr-reset-call) rejects j.reset() because the pointee inherits a reset() from counting_base. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N5NasAgbDGG28MqXjQvLqp Signed-off-by: Claude <noreply@anthropic.com> * Use std::swap for the container stand-in in destroy_container The unqualified call after `using std::swap` would let a user-provided swap(json_value&, json_value&) intercept the destruction path. Call std::swap directly. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> Signed-off-by: Claude <noreply@anthropic.com> Co-authored-by: Claude <noreply@anthropic.com>	2026-11-02 17:48:03 +01:00
Niels LohmannandClaude Opus 5.5	03d408aafa	JSON_DIAGNOSTIC_POSITIONS: comparing a value-initialized iterator throws instead of crashing (#6003) ... * JSON_DIAGNOSTIC_POSITIONS: comparing a value-initialized iterator throws instead of crashing With JSON_DIAGNOSTIC_POSITIONS, comparing a value-initialized iterator (which has no container) with another iterator built the invalid_iterator.212 message from the byte positions of the iterator's value, which is a null pointer: UBSan reports "member call on null pointer" in exception::get_byte_positions(), then the process crashes. Every other configuration throws invalid_iterator.212. get_byte_positions() now returns no positions for a null value, as the JSON_DIAGNOSTICS path already does. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Declare the value-initialized iterator in the test const clang-tidy (misc-const-correctness) flags it, as it is only compared. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N5NasAgbDGG28MqXjQvLqp Signed-off-by: Claude <noreply@anthropic.com> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> Signed-off-by: Claude <noreply@anthropic.com> Co-authored-by: Claude <noreply@anthropic.com>	2026-11-02 17:47:48 +01:00
Niels LohmannandClaude Opus 5.5	76dbe8f9c9	Fix the red CI on develop: Visual Studio 2017 build of the BSON writer, Valgrind timeouts (#6009) ... * CI: give the Valgrind tests a timeout of one hour Under Valgrind, test-json_view_builder_cpp11 and test-ordered_json_cpp11 took 1435 s and 1390 s on develop (ddc34f5), close to ctest's default timeout of 1500 s. On the next run (b2674a4) the whole job was about 10 % slower and both timed out, failing ci_test_valgrind on develop and on every PR based on it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N5NasAgbDGG28MqXjQvLqp Signed-off-by: Claude <noreply@anthropic.com> * BSON writer: drop the SFINAE guard that Visual Studio 2017 cannot deduce write_bson_document_single_pass() was a member template enabled by is_detected<sink_written_t, SinkType>. MSVC 19.16 (Visual Studio 2017) fails to deduce it through the class-scope alias template (C2672, C2783) when it compiles to_bson() and to_bon8(), so every Visual Studio 2017 job on AppVeyor fails since #5979. The guard is not needed: as a non-template member of the class template, the function is only instantiated when called, and its only caller is the write_bson_document() overload for sinks that have written(), which already calls oa.written() and oa.rewind() itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N5NasAgbDGG28MqXjQvLqp Signed-off-by: Claude <noreply@anthropic.com> --------- Signed-off-by: Claude <noreply@anthropic.com> Co-authored-by: Claude <noreply@anthropic.com>	2026-11-02 17:47:21 +01:00
Niels Lohmann	7503394a30	Remove is_bjd_excluded_optimized_type(), which has no caller left; fix two comments ... is_valid_optimized_type() rejects the markers BJData excludes, so the reader no longer needs its own list; the test now checks is_valid_optimized_type() of a BJData reader against the writer's list. The BSON size comment now says that open_bson_document() rejects a size below 5. Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-11-02 09:40:51 +01:00
Niels Lohmann	caabb97337	Binary reader and writer: remove checks that invariants make dead ... Branch coverage of the test suite and of 24k mutated inputs showed these conditions as never taken; each is implied by a check made earlier on the same path: - check_bson_document_size: `document_size < 0` (the size was rejected below 5 before it was stored) - get_ubjson_ndarray_size: `second != 'N'` ('N' is rejected as a type marker before `second` can be set) - bjd_type_name() == nullptr for an ndarray dtype: every marker is_valid_optimized_type() accepts for BJData has a name (now an assert) - the early is_bjd_excluded_optimized_type() check in get_ubjson_size_type(): is_valid_optimized_type() rejects the same eight markers with the same error - `input_format == bjdata` next to `is_ndarray`, bit 8 of the type, or a 'B'/'u'/'m'/'M' marker: these only arise for BJData - `first != npos` next to bit 8: the bit is only set after a count - MessagePack writer: the `<= max` halves of the negative int8/16/32 ladder (the ladder is in the `else` of `n >= 0`) No change in behaviour: a build that aborts on each removed side passes the test suite and the fuzz corpora, and a 1.6M-input differential run of the reader against the original headers found no difference. Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-11-02 06:58:13 +01:00
Niels Lohmann	b2674a4a2f	BSON reader: check array element names in place in strict mode (#5984)	2026-11-01 21:03:35 +01:00
Niels Lohmann	ddc34f546a	json_view: exceptions name the path and the bytes of the value (#5974) ... Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-11-01 17:25:26 +01:00
Niels Lohmann	f022e3c59c	Binary readers and writers: faster UTF-8 checks of strings (#5982) ... Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-11-01 17:23:27 +01:00
Niels Lohmann	9742edc294	BSON writer: write to_bson(j) in one pass, with sizes written afterwards (#5979) ... Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-11-01 17:23:12 +01:00
Niels Lohmann	517444b504	accept() checks strings without copying them (#5978) ... Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-11-01 17:22:58 +01:00
Niels Lohmann	fb515ddc34	UBJSON/BJData readers: read optimized arrays of numbers in one loop (#5981) ... * UBJSON/BJData readers: read optimized arrays of numbers in one loop An array with a type marker for a fixed-size number ($U, $i, $I, $l, $L, $d, $D, and BJData's $u, $m, $M), including the data of a BJData ndarray, gives its elements no markers. When the input is contiguous, read the elements that are in the input in one loop - load, byte swap for the format's byte order, emit - instead of one trip through parse_ubjson_internal and get_ubjson_value per element. Elements the input ends before, other types, and non-contiguous inputs are read as before. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * UBJSON/BJData tests: assign the parsed value instead of casting it to void GCC reports -Werror=unused-result for static_cast<void>(json::from_ubjson(...)) and json::from_bjdata(...): a cast to void does not silence warn_unused_result on GCC. Assign to a json, as the other tests do. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-11-01 17:22:33 +01:00
Niels Lohmann	edf8d17601	MessagePack reader: read runs of scalar array elements in one loop (#5980) ... * MessagePack reader: read runs of scalar array elements in one loop As for CBOR: when an array opens and the input is contiguous, read the integers, floats, booleans, and nils that follow in a tight loop instead of going through parse_msgpack_internal and parse_msgpack_value for every element. The loop stops at any other kind of value and when fewer than 9 bytes are left; those values are read by the general path as before. Non-contiguous inputs are not affected. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * MessagePack reader: enter the scalar run only when a scalar follows As for CBOR: enter_msgpack_array tests the first byte and the remaining length inline and calls the out-of-line run only for a scalar, so arrays of maps or strings no longer pay for a call that reads nothing; the run's loop condition moves to the end (do-while). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * MessagePack reader: mark the initial accepted in the scalar run as a false-positive dead store ci_infer reports DEAD_STORE for 'bool accepted = true;' in the MessagePack scalar run, as in the CBOR one, because every case assigns accepted before it is read. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * MessagePack tests: assign the parsed value instead of casting it to void GCC (ci_test_gcc) reports -Werror=unused-result for static_cast<void>(json::from_msgpack(...)): a cast to void does not silence warn_unused_result on GCC. Assign to a json, as the other tests in the file do. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * MessagePack reader: drop the scalar pre-check before the run enter_msgpack_array now calls the run directly, and parse_msgpack_scalar_run tests the remaining count and length at the top of a while loop instead of relying on msgpack_scalar_follows(); its own switch already leaves on any value that is not a scalar. Instructions per repetition, new/old: SAX canada 0.946 (clang) and 0.973 (GCC 13), DOM read canada 0.991 / 0.992; no row more than 0.3 % higher. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * MessagePack reader: say why the initial accepted in the scalar run stays As for CBOR: Infer's DEAD_STORE report is right, not a false positive. The initializer is kept for clang-tidy's cppcoreguidelines-init-variables. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * MessagePack reader: give the scalar run the bulk tag instead of a wrapper As for CBOR: parse_msgpack_scalar_run takes the bulk-scan tag itself and is the function kept out of line, so parse_msgpack_array_start goes. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * MessagePack reader: start the scalar run's accepted at false As for CBOR: Infer ignores dead stores of 0, so the suppression goes. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * MessagePack reader: let the scalar run find its array itself As for CBOR: parse_msgpack_scalar_run reads container_stack.back() in its bulk version, so the caller no longer evaluates it for input that is not contiguous. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-11-01 17:22:32 +01:00
Niels Lohmann	aecbef3949	Binary readers: swap the bytes of floats with a single byteswap (#5976) ... * Binary readers: swap the bytes of floats with a single byteswap byte_swap reversed float and double values byte by byte. GCC does not merge that loop into one bswap, and C++23's std::byteswap accepts only integers. Overloads for float and double now copy the bits into a uint32_t/uint64_t, call detail::byteswap, and copy them back. This applies to every binary reader that uses get_number. In the CBOR reader, the single- and double-precision float cases call a non-inlined helper. Without it, GCC stopped inlining the stream adapter's get_character() into parse_cbor_internal, which made reading from a std::istream about 0.5 % slower. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * CBOR reader: inline get_cbor_float() and always inline get() instead The never-inlined get_cbor_float() costs a call for every float read from input without bulk access: reading canada.json from CBOR through a FILE* became 5.6 % slower on Apple clang in the benchmark suite. The helper existed only because GCC 13 stopped inlining the stream adapter's get_character() into parse_cbor_internal. Always inlining binary_reader::get() fixes that as well. callgrind, all binary formats on buffer, std::istringstream and FILE* input (canada and citm_catalog): GCC 13 0.986-1.003, GCC 16 0.994-1.006, Clang 23 identical. Apple clang suite: FromBinaryFile/cbor/canada 0.939, geomean of the 53 binary reader rows 0.998. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-11-01 17:22:32 +01:00
Niels Lohmann	6c3ab966d3	CBOR reader: read runs of scalar array elements in one loop (#5975) ... * CBOR reader: read runs of scalar array elements in one loop When a definite-length CBOR array opens and the input is contiguous, read the integers, floats, booleans, and nulls that follow in a tight loop instead of going through parse_cbor_internal's container logic and parse_cbor_value's dispatch for every element. The loop stops at any other kind of value and when fewer than 9 bytes are left; those values are read by the general path exactly as before. Non-contiguous inputs are not affected (tag dispatch on the bulk-scan capability). Also pass a member empty string as the number source to number_float instead of constructing a std::string for every float. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * CBOR reader: address review of the scalar-run reader - parse_cbor_scalar_run: set the size explicitly in every case that reads a value, instead of relying on a default that every other case overwrites. - Remove cbor_scalar_follows(): parse_cbor_scalar_run already returns on the same conditions, and without the pre-check SAX canada needs another 8.5 % fewer instructions with GCC 13 (0.915 against the previous version), with no row more than 0.5 % worse (Mac and GCC). - Restore the comment on why parse_cbor_internal copies the container frame. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * CBOR reader: enter the scalar run only when a scalar follows An array of containers or strings called the out-of-line run start once per array only to leave it at once; on nested data (arrays of objects) that call made reading slower than before the run reader. enter_cbor_array now tests the first byte and the remaining length inline and calls the run only for a scalar, so the run's loop condition moves to the end (do-while). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * CBOR reader: mark the initial accepted in the scalar run as a false-positive dead store ci_infer reports DEAD_STORE for 'bool accepted = true;' in parse_cbor_scalar_run because every case assigns accepted before it is read. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * CBOR reader: drop the scalar pre-check before the run enter_cbor_array calls the run directly again, and parse_cbor_scalar_run tests the remaining count and length at the top of a while loop; its own switch already leaves on any value that is not a scalar. The MessagePack reader is built the same way. Instructions per repetition, new/old: clang SAX canada 0.983, read canada 0.993; GCC 13 read canada 0.997, the other rows at most 1.004. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * CBOR reader: say why the initial accepted in the scalar run stays Infer's DEAD_STORE report is right, not a false positive: every case that reaches the read assigns accepted. The initializer is kept for clang-tidy's cppcoreguidelines-init-variables. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * CBOR reader: give the scalar run the bulk tag instead of a wrapper parse_cbor_scalar_run takes the bulk-scan tag itself and is the function that is kept out of line, so the one-line parse_cbor_array_start wrapper goes, as the UBJSON typed run already does. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * CBOR reader: start the scalar run's accepted at false Infer ignores dead stores of 0 (--liveness-ignored-constant defaults to "0"), and false is 0 to it, so the suppression and its comment go. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * CBOR reader: let the scalar run find its array itself parse_cbor_scalar_run reads container_stack.back() in its bulk version, so the caller no longer evaluates it for input that is not contiguous. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-11-01 17:22:31 +01:00
Niels Lohmann	76001c6b4c	BSON reader: find C-string terminators 8 bytes at a time, skip array element names (#5968) ... Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-11-01 17:20:54 +01:00
Niels Lohmann	9c349b1d7c	Write integers of 17 to 20 digits eight digits at a time in dump() (#5964) ... #5634 made dump() write integers below 10^16 eight digits at a time; larger ones still took the two-digits-per-division loop and count_digits(). Random 64-bit integers almost all have 17 to 20 digits (99.9 % of signed_ints.json and unsigned_ints.json), so Dump/signed_ints and Dump/unsigned_ints gained only 1.4x over 3.12.0, against 2.3x for small integers. A 64-bit magnitude is now written as its first one to four digits and two groups of eight. Wider magnitudes (__int128 and class types, #5805) keep the loop: number types of at most 64 bits are selected by type, so no comparison is reported as always true; for others the value is compared. The sign handling of #1447/#5815 (remove_sign() into number_magnitude_t) is unchanged. Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-10-31 13:54:45 +01:00
Niels Lohmann	0115bb47b3	Name the source and license of adapted code in the file comments (#5965) ... The files with code adapted from fast_float, Zmij and simdjson carried the authors' SPDX copyright lines, but their file comments did not say what was taken or under which license: number_parse.hpp, to_chars.hpp, simd_scan.hpp and view/simd.hpp now do, as zmij.hpp and the yyjson-based view files already did. README and the license page now also name fast_float's eight-digit parsing and the lexer's use of simdjson's UTF-8 check. Comments only. Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-10-30 15:17:24 +01:00
Niels Lohmann	54f6b3ef06	json_view: source positions in materialize() with JSON_DIAGNOSTIC_POSITIONS (#5945) ... * json_view: source positions in materialize() with JSON_DIAGNOSTIC_POSITIONS With JSON_DIAGNOSTIC_POSITIONS, materialize() now gives every value the start_pos()/end_pos() that basic_json::parse() would give it, so exceptions about materialized values name the same bytes (#5864). - json_sax_dom_parser takes the lexer type as a defaulted template parameter; materialize() passes position_source, which reports the positions it derives from the index. diagnostic_positions treats npos as "no position" and keeps ranges all-or-nothing. - Strings with escapes keep the offset of their opening quote plus one in `next` (only with the macro); materialize() scans from there to the closing quote. The end of an array or object is found by scanning from the end of its last element over whitespace, comments and a trailing comma. - In editable documents, values written by edits, arrays and objects whose elements changed, and containers whose last element has no position get none. Images keep the flag is_new, and load() checks that `next` of a string is within the text. Nothing changes on the parse path without the macro. The view's own exceptions still carry no byte range, so the tests that compare them with those of materialized values skip it under the macro, as they skip the path under JSON_DIAGNOSTICS. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Indent comment in unit-json_view_from_json.cpp Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address review: clearer position code and docs - position_source gets clear(); set_scalar() returns early for new values, switches on node_kind, and uses no local copies; the one position_source per materialize() call is documented - no npos + 1 for containers without a position; a comment on why the end of a container is reported as its closing bracket - less duplication under #if in materialize() and the builder - diagnostic_positions sets npos explicitly - docs: positions of basic_json values after changes, positions in images depend on how the document was parsed, when next is set, and the cost of positions while parsing Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Share the comment scanners of the view builder with materialize() skip_line_comment() and find_block_comment_end() in scan.hpp are used by the builder's comment() and by position_source::container_end(), which had its own copy. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address review: if/else for container positions, clearer next field - the opening bracket of a container sets positions with an if/else instead of a ternary, and the frame takes its start from positions - the closing bracket of a container is found in nested if/else; the parent's last_end is taken from positions, so it is npos whenever no range is set - architecture.md: the next field reads "or 0 if it was not; all other values: 0" instead of "else 0; otherwise 0" Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix GCC unused-result and clang-tidy raw-string findings in unit-json_view_positions.cpp Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-10-28 20:24:49 +01:00
Niels Lohmann	091c10d4a5	Fix self-aliasing in json_pointer::operator/= and patch_inplace (#5936) ... * Fix self-aliasing in json_pointer::operator/= and patch_inplace json_pointer::operator/=(const json_pointer&) passed iterators into its own token vector to insert() when a pointer was appended to itself (p /= p), which is undefined behavior; libstdc++ produced "/a/b//" for "/a/b". The pointer now appends a copy in that case. basic_json::patch_inplace iterated the patch while changing *this, so j.patch_inplace(j) read freed memory. It now applies a copy of the patch when the patch is the value itself. A patch stored inside the value remains undefined behavior and is documented as such; copying the patch unconditionally would double the time of a large patch. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix JSON_DIAGNOSTICS test crash and clang-tidy hit in patch_inplace self-aliasing The spare-capacity test reallocated non-empty elements through get_ref(), which leaves stale parent pointers under JSON_DIAGNOSTICS; reserve while the array is empty instead. Suppress performance-unnecessary-copy-initialization for the copy that is needed because *this is modified. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Detect a patch stored inside the patched value with JSON_DIAGNOSTICS patch_inplace applied a copy only when the patch was the value itself. With JSON_DIAGNOSTICS the parent pointers make it cheap to also detect a patch nested inside the value, so that case is handled the same way. Without diagnostics it stays documented undefined behavior. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Silence misc-const-correctness for patch_aliases_this Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix clang-tidy findings in the operator/= self-append test Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-10-28 20:23:55 +01:00
Niels Lohmann	78a0687147	Strong exception guarantee for operator[](json_pointer) (#5940) ... * Keep the value unchanged when operator[] with a JSON pointer throws The non-const operator[](json_pointer) is documented with strong exception safety, but get_unchecked created intermediate values while walking the pointer and kept them when a later reference token failed, e.g. json j; j["/a/99999999999999999999999"_json_pointer] threw 404 and left {"a":[]}; padding an existing array or appending with "-" behaved alike, and so did std::length_error/std::bad_alloc from a huge index. get_unchecked now only walks existing values up to the last reference token. The last token's key insertion or array growth is strongly exception safe by itself. Any other change goes through get_unchecked_create, which records the one change it makes to an existing value (null converted, key inserted, array grown) in a RAII guard and reverts it if a later step throws; everything else it creates lies below that change. The guard needs no try/catch, so JSON_NOEXCEPTION builds are unaffected. Successful calls create the same values as before. Docs: state the guarantee in the exceptions list and version history of operator[], fix the complexity of the pointer overloads of operator[] and at, and warn that padding arrays up to an untrusted index allocates memory proportional to the index (operator[].md, untrusted_input.md). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Make the strongly exception-safe pointer operator[] faster than before The previous commit made get_unchecked strongly exception safe but slower with GCC 16 when it creates values (up to +6 % instructions for appending with "-"). This restructures it so that no benchmark row is slower than before the strong guarantee, with GCC 16, Clang 22 and Apple Clang. - get_unchecked walks existing values in one loop that inserts nothing. The last reference token is resolved in that walk: "-" appends with emplace_back(), an index grows the array with operator[], and a missing key is inserted with emplace_hint() at the position the walk's lower_bound() found, so the key is searched once. - With std::less<string_t> or std::less<> as comparator, the walk checks the key at the lower bound for equality instead of equivalence, which is the same for these comparators and cheaper. - get_unchecked_create (not inlined) handles all other changes. Its guard swaps a null value with a fresh null value instead of copying it and reverts the change in a non-inlined function, so a successful call pays one compare for it. As every null value converted afterwards is a fresh one, it is converted in place instead of assigning a new array or object. - Keys are added to new objects with emplace(), which also replaces the basic_json::operator[] call for ordered_json. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix rollback of pointer operator[] for unusual object and string types Follow-up to the strong exception guarantee of the non-const operator[](json_pointer): - get_unchecked_create assumed that emplace() always inserts the key that the walk did not find. For object types whose lookup and emplace() disagree (e.g. a copied fifo_map), nothing was inserted, the rollback erased the existing member, and the walk continued with a value of the wrong type (assertion, type_error.305, or a crash). It now checks the result of emplace(), arms the rollback only if a key was inserted, and otherwise continues the walk with the member found, as basic_json's operator[] does: such pointers throw out_of_range.404 and leave the value unchanged, as before the strong guarantee. - The rollback erased the inserted key by key, which converts string_t to object_t::key_type and may allocate inside a noexcept function (std::terminate on bad_alloc with custom key types). It now erases by the iterator emplace() returned, and shrinks a grown array with erase() at the end instead of resize(). The documentation states the remaining requirement for custom template parameters. - Creating values no longer calls string_t::begin(), so the non-const operator[](json_pointer) compiles with string types that lack it (as the alternative string type of the tests). - Correct the @throw lists of get_unchecked and get_unchecked_create. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address review comments on operator[](json_pointer) rollback - undo() shrinks the array with resize() instead of erase() - comments say what "will be" done in get_unchecked_create - tests: avoid a clang-tidy performance-unnecessary-copy-initialization hit (unit-alt-string) and the word "access" in section names, which Flawfinder flags as race/access Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Let the pointer rollback guard call undo() unconditionally The check in the destructor and JSON_HEDLEY_NEVER_INLINE on undo() made no measurable difference: instruction counts of j[ptr] are identical with GCC 13 -O3 and within 0.4 % with clang, timings are within noise. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-10-27 17:15:55 +01:00
Niels Lohmann	9e67bfa115	json_view: the library's std conversions read views directly (#5923) ... * json_view: the library's std conversions read views directly The library's from_json() overloads (containers, maps, std::array, C arrays, pairs, tuples, optional, valarray, forward_list, enums, binary, paths, and scalars) accept views: their traits use the view's basic_json type, and the few accesses that differ (get_ptr) go through small helpers. from_json() of a view now finds them; only the probe still looks up by ADL only. get<T>() of a view thus converts these types without materialize(), with the same values and exceptions; from an object, a repeated key keeps the last value, as parse() does. The view keeps its direct paths for vectors of the types it converts directly (faster than from_json()) and for containers of views; its other vector and map paths are replaced by from_json(). For basic_json, std::valarray and std::forward_list are read in one pass (40 % and 3.5 % fewer instructions); all other conversions are unchanged. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * json_view: enable_if_not_view_t instead of '!' in the from_json() templates astyle spaces out the template argument list when it contains a negation; the alias spells the condition with negation<> and leaves the lines alone. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * json_view: compact the template parameter lists astyle had spaced out Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-10-26 20:13:31 +01:00
Niels Lohmann	e5fe4ebdd2	json_view: from_json() of the macros reads views directly (#5922) ... get<T>() and get_to() of a view call T's from_json() with the view itself when that from_json() accepts views but not an unrelated probe type, and fall back to materialize() otherwise. The from_json() functions of the NLOHMANN_DEFINE_TYPE_*, NLOHMANN_DEFINE_DERIVED_TYPE_* and NLOHMANN_JSON_SERIALIZE_ENUM(_STRICT) macros accept basic_json values and views; a derived type whose base reads only json reads the base part through materialize(). The library's own from_json() overloads are not called with views: from_json() of a view (and of the probe) only looks up functions by ADL. New public trait nlohmann::is_json_view<T>, and the member type basic_json_type of basic_json_view. With duplicate keys, the members of such a T are those the view's lookups find (the first ones). bench_decode.cpp measures decoding into structs (4-6x faster than materialize().get<T>() on twitter, citm_catalog, and synthetic records). Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-10-26 20:13:31 +01:00
Niels Lohmann	5a2819c940	json_view: one SIMD macro set with the library (#5921) ... The view's vector code follows JSON_NO_SIMD, like the lexer and the float writer, and skips the CPU check of SSSE3 when the code is compiled for SSSE3 anyway (__SSSE3__), like the lexer. JSON_VIEW_NO_SIMD and JSON_VIEW_USE_SSSE3 and their ABI tags _vns and _vssse3 are removed; they were never released. JSON_NO_SIMD keeps its tag _ns. Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-10-26 20:13:31 +01:00
Niels Lohmann	4cdfa6d0f4	json_view: no save() and load() on big-endian targets (#5920) ... The image format is little-endian only. Instead of throwing type_error.320 at runtime, save_image(), load_image() and the members save() and load() do not exist on a big-endian target, so a call fails to compile, like set() on a read-only document. type_error.320 remains for saving a discarded document. Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-10-26 20:13:30 +01:00
Niels Lohmann	60107eda3f	Read BJData ND-arrays of float16, restrict ND-array char to 0..127, correct BJData docs (#5931) ... * Read ND-arrays of half floats and enforce the char limit in ND-arrays BJData N-dimensional arrays with element type 'h' (float16) failed to read with a misleading parse_error.112, because the reader's dtype name table lacked 'h'. It now maps to "half", the JData name. The writer still does not pack "half"; such an object is written as a plain object and reads back equal. BJData limits 'C' (char) to 0..127 in Draft 2 and Draft 3, and the 1-D reader enforces this, but ND-arrays with _ArrayType_ "char" wrote values up to 255 as 'C' elements and read them back without a check. The writer now only packs values in 0..127 (others fall back to a plain object that round-trips), and the reader rejects 'C' elements above 127 with parse_error.113 like the 1-D reader. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: correct and complete the BJData documentation Verify the BJData chapter against the library and the Draft 2/3/4 specifications and fix what did not match: - bjdata.md: correct the list of unused markers (N, not Z), the ND-array example (3x2, not 2x3), the conditions under which an object is packed as an ND-array (exactly three keys, row vectors, float-only elements for single/double, char 0..127, byte only in Draft 3), the binary table row and notes (which flag/draft combinations write [$B), the "$" type restriction (Draft 3, Z is excluded too), the H marker in the deserialization table, and the claim of a complete deserialization mapping. Document the supported drafts, that Draft 4 (SoA containers, extension type E) and column-major ND-arrays are rejected, ND-arrays of half floats, duplicate object keys, and link the Draft 3 specifications instead of master. - Add an API page for basic_json::bjdata_version_t with an example, and link it from the nav, the API index and to_bjdata. - Update binary_formats/index.md and binary_values.md, which still claimed that BJData has no binary type, and fix a wrong byte comment there. - Document parse_error.115 and the out_of_range.408 cases for from_bjdata, the precise other_error.502 condition for to_bjdata, and reword the note on truncated unsigned integers in exceptions.md (only ND-array elements were truncated before 3.13.0). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Regenerate single_include and apply astyle Result of make amalgamate (including make pretty) for the preceding changes. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address review of the BJData documentation The $ type restriction exists since Draft 2 (Draft 3 added B), $ requires a count, ND char above 127 is rejected on read, and the new API page is listed in alphabetical order in the nav. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: format tables, use the Oxford comma Reformat the tables changed by this PR (BJData serialization and deserialization mappings, completeness and binary-value tables) so all rows are aligned, and add the Oxford comma to lists of three or more items in the added text. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-10-26 18:25:46 +01:00
Niels Lohmann	dc768a7d5d	Fix BSON binary subtype 0x02 and check the declared document size earlier (#5930) ... * Write and read the inner length of BSON binary subtype 0x02 The BSON specification defines the payload of the "binary (old)" subtype 0x02 as an int32 followed by that many bytes. to_bson wrote the plain payload instead, so decoders such as pymongo and libbson-based readers rejected the bytes, and from_bson returned the inner length as part of the payload (documented as lenient behavior in gh-5333). The writer now emits the outer length (payload + 4), the subtype, the inner int32 holding the payload size, and the payload. The reader requires an outer length of at least 4 and an inner length equal to the outer length minus 4 (the cases the bson-corpus lists as decode errors), reports parse error 112 otherwise, and returns only the payload. Both directions change together so that the round trip keeps working. The single-reservation fast path for scalar entries declines subtype 0x02 because it has no room for the inner length. This is a deliberate exception to the opt-in rule for behavior changes: the previous output was invalid BSON. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Check the declared BSON document size before reading its content The BSON reader compared a document's declared size with the number of bytes read only when it reached the terminator. The declared size never bounded the reads, so a SAX handler saw the keys and values of a document whose size was too small before parse error 112 was reported. The container frame now stores the position where the document ends according to its declared size. open_bson_document rejects a size below 5 (size prefix plus terminator) and an embedded document that does not end before the terminator of its enclosing document; the element loop rejects an element whose type byte leaves no room for the name terminator and the document terminator. The terminator check stays and still catches a declared size that is too large. No input that was accepted before is rejected now; documents with a wrong size are reported earlier, with fewer SAX events, a different byte position and, in three cases, a different message. The error id (parse_error.112) is unchanged and strict=false still does not skip the check. The tests that asserted the old messages and positions are updated, with the outer sizes of two inputs corrected so that they still exercise the embedded document. New tests cover the SAX events, a size below 5, and an embedded document that fits exactly. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Document BSON subtype 0x02, document size checks, and lenient reads Describe the inner length of binary subtype 0x02 that to_bson now writes and from_bson checks and removes. Add the document size checks (parse_error.112, not affected by strict=false) to bson.md and from_bson.md, and extend the list of lenient reads with the UUID and MD5 payload lengths (not checked), duplicate keys (last value wins), embedded NUL bytes in strings, and the ignored trailing bytes with strict=false. State that the members of an object are written in iteration order. Correct the statement that element names are never validated on read: they are checked and sanitized like string values by the error handler. Add the BSON message to the examples of out_of_range.415 and type_error.321, the subtype limit of 255 to the binary values page, CBOR to the formats that use byte_container_with_subtype, and fix a typo in the incomplete mapping box. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Regenerate single_include and apply astyle Result of make amalgamate (including make pretty) for the preceding changes. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address review of the BSON documentation Describe exactly when a too small document size is detected and note that subtype 2 data written by earlier versions is not readable. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: use the Oxford comma, drop discarded-value remarks Add the serial comma in a list added by this PR and remove the BSON example of type_error.321 (with the type_error.317 remark) so the docs do not mention discarded values for BSON; restore the original singular heading of that section. No tables were changed by this PR. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix -Weffc++ in the BSON recorder test and a clang-tidy use-auto hit Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix clang-tidy readability-redundant-member-init in BSON test Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-10-26 18:25:46 +01:00
Niels Lohmann	0c3570d44a	Validate the UBJSON/BJData container type marker for count 0; fix UBJSON docs (#5929) ... * Validate the optimized-container type marker for count 0 in UBJSON and BJData The '$' type marker of an optimized container was only checked when an element was read, so containers with count 0 accepted any marker: `[$x#i0`, `{$x#i0` and `[$E#i0` parsed as empty containers although a non-empty container with the same marker is rejected and the UBJSON specification requires a legal marker after '$'. get_ubjson_size_type() now checks the marker right after '$' against the allowed set (UBJSON: Z N T F i U I l L d D H C S [ {; BJData: B U i u I m l M L h d D C, with the existing exclusion of T F Z N S H [ {) and reports parse_error.112 "marker ... is not a permitted optimized array type". Input that was accepted and is now rejected: an optimized container with count 0 and an invalid '$' type marker. Valid input is unaffected. Two existing tests that used an invalid marker incidentally ("[$#" and a negative-count case) were adapted. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Correct and complete the UBJSON documentation Fix statements that disagree with the behavior (checked with probes): - `Z` is null; the unused marker is `N`. - `$` requires `#`; use_type with use_size=false throws other_error.502 only for non-empty arrays, objects and binary values. - NaN and infinity are written as IEEE 754 float64 and read back unchanged; the UBJSON specification encodes infinity as null. - `H` is documented (semantics, parse_error.115, out_of_range.406); the "complete mapping" claim is qualified. - use_type "same type" means same marker. - Long Z/T/F arrays lose only the `$` annotation; the `#` count and the element markers are kept. - from_ubjson lists the 112, 113, 115, 406 and 408 causes that occur for UBJSON, without the BJData-only ND-array wording. - Duplicate object keys keep the last value; the `$` type marker is validated even for count 0. - Smaller fixes: float32 row, "map" -> "object", https link, exception 110 strict case, to_ubjson history of discarded values. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Regenerate single_include and apply astyle Result of make amalgamate (including make pretty) for the preceding changes. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address review of the UBJSON documentation Clarify that empty containers carry no $ annotation of their own and note that a counted N object is not supported. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: format tables, use the Oxford comma, drop discarded-value remarks Reformat the two UBJSON mapping tables that this PR changed (uniform column widths, padded cells), use the serial comma in lists of three or more in the added text, and restore the original type_error.321 bullet in to_ubjson.md. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix narrowing of the optimized-container count in BJData and UBJSON tests Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-10-26 18:25:45 +01:00
Niels Lohmann	d5da84249f	Skip CBOR tags in front of map keys, validate indefinite text chunks, fix CBOR docs (#5927) ... * Skip CBOR tags in front of map keys and validate indefinite text chunks A tag in front of a text string map key was rejected with parse_error.113 even with cbor_tag_handler_t::ignore or store, although tags in front of values are handled. The tags are now skipped iteratively before the key is read; the handler error still rejects them. The chunks of an indefinite-length text string were concatenated before the single UTF-8 check, so a code point split over two chunks, or an invalid chunk, was accepted. RFC 8949, Section 3.2.3 requires every chunk to be valid on its own. With error_handler_t::strict, replace and ignore each chunk is now checked; keep is unchanged. The chunks are appended with append(ptr, len), so StringType needs nothing new. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: correct and extend the CBOR documentation Fix the negative-integer rows (they used MessagePack boundaries), the number_integer range of the 0x1B row, and the half-precision entry in the list of unused types; correct the NaN history (0xF9 since 3.8.0) and the tagged-item wording (only the tag directly in front of a byte string is stored, and only in the 0xD8..0xDB form). Document that to_cbor is not RFC 8949 deterministic, that duplicate keys are accepted (last wins), tags in front of keys, per-chunk UTF-8 validation, the exceptions parse_error.113, out_of_range.406/408 of from_cbor, the pointer and C array input traps, the partially written output on exceptions, and the store/ignore results in the binary values example. Drop the claim that the CBOR reader needs StringType::append(const StringType&) and BinaryType::insert(pos, first, last). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Regenerate single_include and apply astyle Result of make amalgamate (including make pretty) for the preceding changes. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address review of the CBOR changes Cite RFC 8949 Section 5.3.1 for duplicate keys, describe the out_of_range.408 condition precisely, rewrap a long line, and update the stale get_string comment (per-chunk UTF-8 check). single_include regenerated. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: drop discarded-value remarks Remove the mention that a discarded value cannot be serialized to CBOR (type_error.321) from the exception-safety and mapping sections. The integer table and all added prose already follow the table-format and Oxford-comma rules. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-10-26 18:25:44 +01:00
Niels Lohmann	81d66291c5	Create objects for pointer tokens that cannot be array indices (#5358) ... * create objects for tokens that cannot be array indices (#5357) When operator[](json_pointer) traverses a level that does not exist yet, the null value is turned into an array or an object depending on the reference token. The check only tested whether all characters are digits, so tokens that can never be a valid array index selected an array and then failed: - "01" (and any other token with a leading '0') threw parse_error.106 - the empty token threw out_of_range.404 Both tokens are valid object keys, and both work when the level already exists as an object, so creating the level changed the outcome. Test the token against the RFC 6901, Sect. 4 grammar for array indices instead, so that such tokens create an object. This only affects pointers that threw before. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Clarify that the token 0 is a valid array index in operator[] docs The note described valid array indices as "a nonempty sequence of digits without a leading 0", which read literally excludes the token "0" itself, contradicting both the code and the /foo/0/0/0 example in the next sentence. Spell out the RFC 6901 grammar instead. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-10-26 17:13:48 +01:00
Niels LohmannandClaude Sonnet 5.5	d6431a9104	Throw out_of_range.411 for JSON Patch add into a null parent (#5917) (#5918) ... Claude-Session: https://claude.ai/code/session_01MKJf48vnGtTE25KscmjfaW Signed-off-by: Niels Lohmann <niels.lohmann@gmail.com> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>	2026-10-26 17:11:25 +01:00
Niels Lohmann	2a56afc714	Fix VS 2015 build of the DOM parser (#5882) (#5916) ... - Name std::hash only once the object type has emplace_new_key: VS 2015 static-asserts on std::hash of a type without one (alt_string) instead of reporting a substitution failure (C2338). - Replace the std::copy calls on frames by loops: VS 2015 reports raw pointers handed to std::copy as C4996, an error under /WX. Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-10-25 12:08:58 +01:00
Niels Lohmann	3eae5680d6	from_json: number-first dispatch for arithmetic types, loop for std::array (#5902) ... * from_json: number-first dispatch for arithmetic types, loop for std::array (#5891) from_json(ArithmeticType&) now reads numbers through get_number_value() and handles a boolean on a cold path, which puts the three number types first in the per-element type dispatch. get<std::array<T, N>>() always took the identity_tag overload, a pack expansion of N calls j.at(i).get<T>() that clang unrolls into straight-line code (about 28,000 instructions for N = 1000), so its speed depended on inlining decisions in that body. For trivial element types, which cannot tell default construction plus assignment from construction in place, it now fills a std::array in a loop. That loop, shared with get_to(), checks the size once and reads the inner array directly; a short array still converts the elements it has and then throws the out_of_range error at() would throw. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Skip the get_to result check without exceptions Without exceptions, the get_to calls in CHECK_THROWS_WITH_AS do not run, so the array keeps its initial values (ci_test_noexceptions). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Make is_trivial_element a struct so astyle does not indent it Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Expect the element path in the std::array type-error tests with JSON_DIAGNOSTICS Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Shorten from_json_inplace_array so astyle aligns its parameters The opening parenthesis of the previous name sat in column 41, beyond astyle's default --max-continuation-indent of 40, so the continuation lines fell back to a plain double indent. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Use the compact template<...> style for the new from_json overloads Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-10-25 06:17:51 +01:00
Niels Lohmann	fe5e401870	Consolidate duplicated detail helpers and bodies (#5891) (#5901) ... * Add shared detail helpers for the #5891 cleanup Add detail::is_primitive(value_t), detail::hex4_lower(), detail::utf8_shape(), detail::input_format_name(), detail::empty_input_message(), the exception ids 116, 319, 320 and 416 to detail::exception_id, and make detail::is_ascii_copyable() constexpr. Later commits switch the duplicated code over to them. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 1.4: BSON writers share one write-pass UTF-8 sanitizer (#5891) write_bson_entry_header and write_bson_string re-derived the error-handler policy differently. Both now use sanitize_utf8_for_bson_write, which treats strict like keep (the size pass already validated) and sanitizes replace/ignore into the bytes the size pass measured. No context plumbing is needed. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.1.1: one MessagePack container prefix helper (#5891) Array and object prefixes differed only in the fix marker and the 16/32-bit markers; both now delegate to write_msgpack_container_prefix. Output is byte-identical. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.1.2: one iterative walker for CBOR and MessagePack (#5891) write_cbor_iterative and write_msgpack_iterative (and their *_value_or_push helpers) were identical but for the head, scalar, and key writers. They are now one template over a small policy struct, so everything inlines. BON8 keeps its own walker (open string run, closing marker), noted in a comment. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.1.3, 2.1.8: one UBJSON container start helper, shared frame base (#5891) write_ubjson_start_array/_object are now write_ubjson_start_container<IsArray>, keeping the array-only begin()+1 scan and excessive_valueless rule. ubjson_frame derives from binary_container_frame instead of duplicating it. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.1.4: write_bytes helper for the byte-string write sites (#5891) The 13 oa.write_characters(reinterpret_cast<const CharType*>(x.data()), n) sites now call write_bytes(const void*, size_t); copy_characters sites and the buffer-based writes are untouched. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.1.5: shared object-key writer for CBOR and MessagePack (#5891) write_cbor_key and write_msgpack_key were the same SFINAE pair apart from the string/scalar writers; both now delegate to write_object_key<Walk>. UBJSON key and to_format_length stay as they are. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.3.10: writer-side format names from input_format_name (#5891) The CBOR/MessagePack/UBJSON/BSON/BJData/BON8 literals in throw_on_discarded and throw_integer_out_of_range calls now come from detail::input_format_name; the texts are identical, and UBJSON vs BJData still follows use_bjdata. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * serializer: constexpr ascii table, hex4_lower, dedupe helpers (#5891) 1.1: build ascii_copyable() at compile time from is_ascii_copyable and test find_ascii_copyable_run against it for every byte. 1.2/2.5.4: use detail::hex4_lower in both \uXXXX writers. 2.1.6: factor the repeated flush/emit blocks in dump_escaped_impl into always-inline helpers. 2.1.7: comment why the escape copy does not use put_buffer. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 1.6: use string_t-aware to_string/concat in meta() (#5891) Replaces the four std::to_string sites; rendered text unchanged. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 1.7: use detail::is_primitive in json.hpp switches (#5891) has_no_children, set_parents, max_size and the two diff walkers. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.2.1: factor insert()/iterator throw helpers (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.2.2: share swap() overloads via swap_impl (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.2.4: factor primitive reset and range check in erase()/range ctor (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.2.6: share push_back array bodies via push_back_array_impl (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.2.7: route parse/accept/sax_parse through *_impl helpers (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.2.8: share the vector-returning to_* bodies via to_binary_vector (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.2.10: merge destroy_string/destroy_binary into destroy_pointer (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.4.1/1.7: exceptions: share what_impl, flatten diagnostics switch (#5891) The four create() bodies build their message through one protected static helper; the diagnostics parent walk handles array/object and ignores all other types. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.4.2/2.4.3/1.7: json_pointer: share slash and escape checks, drop primitive case lists (#5891) The leading-slash and invalid '~' escape checks are shared helpers; the reference-token walks test object and array first and treat everything else as primitive. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 1.7/2.4.9: iter_impl, iteration_proxy: drop primitive case lists, share compare preamble (#5891) Object and array stay the first branches; everything else is the primitive case. operator== and operator< share the different-container check. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.4.5/2.4.6: from_json: share get_number_value and reserve_if_possible (#5891) The three number cases of get_arithmetic_value and the arithmetic from_json share one helper (boolean stays accepted only by the latter); the array and object reserve overload pairs become one. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.4.8: ordered_map: share erase_impl between the key erase overloads (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 1.7: json_pointer: flatten walk drops the primitive case list (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 1.2/1.3/1.5/2.5.1/2.5.2/2.5.5: share helpers in the view code (#5891) - detail::exception_id at every numeric id; throw_* helpers take it - hex_byte, hex4_lower, utf8_shape replace local hex/table code - throw_array_index_out_of_range/_past_end, throw_key_not_found - has_native_float_format replaces the hand-written IEEE check No change of messages, ids or output. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 1.5: use utf8_shape in get_bon8_string, add exhaustive UTF-8 validator test (#5891) get_bon8_string takes the valid range of the second byte from utf8_shape() (0xF5..0xF7 stay errors, unget behavior unchanged). validate_one_utf8 keeps its if-chain (hot bulk path); the new test checks decode, validate_one_utf8, utf8_shape, json::accept and the BON8 reader against each other for every lead and second byte. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.3.1: get_cbor_sized_bytes for the CBOR chunk getters (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.3.2: get_string_with_length for the plain-order string sites (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.3.3, 2.3.4, 2.3.10: unsupported_key_type, empty_input_message, input_format_name in the readers (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.3.7, 2.3.9: read_object_key and leave_container_and_check for CBOR/MessagePack/BON8 (#5891) UBJSON keys (different getter signature) and its container tail stay as is. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.3.5: json_sax_dom_error_state base for the DOM SAX parsers (#5891) Holds errored, allow_exceptions, parse_error() and is_errored(). The scalar handlers stay in each class (parse hot path). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.3.6: table of control-character messages in the lexer (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.3.8: finish_input helper for the strict-EOF block in the parser (#5891) sax_parse keeps its if (result) guard and return value. Also adds the clang-tidy NOLINT for the 2.3.6 message table. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.5.7: share parse_array_index between json_pointer and json_view (#5891) Move json_pointer::parse_array_index() and its status enum to namespace detail (json_pointer keeps a forwarding member), and let the json_view pointer resolution use it instead of its own copy of the RFC 6901 index grammar. Both accepted the same tokens with the same outcome. Also update the comment on detail::decode(), which listed four of the five UTF-8 validators and now points to the test that keeps them in sync. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Amalgamate (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.4.5: keep the inlined number cases in from_json(ArithmeticType&) (#5891) Calling get_number_value() from from_json(ArithmeticType&) made GetArray<std::array<int, 1000>> 2.4x slower (it changes the inlining of the std::array conversion), so that overload keeps its own switch, with a comment. get_arithmetic_value() still uses the shared helper. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.5.7: force-inline detail::parse_array_index (#5891) Moving parse_array_index out of json_pointer stopped clang from inlining it into the pointer resolution (JsonPointerAt/citm_catalog +17%, JsonPointerAt/twitter +8%); JSON_HEDLEY_ALWAYS_INLINE restores the baseline. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 1.5: keep get_bon8_string's own UTF-8 ranges (#5891) Taking the second-byte range from utf8_shape() made BON8 decoding 2-3% slower (FromBinaryShape/scalars, FromBinaryBuffer, alignment-pinned builds), so the byte path keeps its hand-written ranges. The exhaustive validator test still checks that it accepts the same sequences as the other validators. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Amalgamate (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix CI failures: constexpr, NRVO, cppcheck, IWYU (#5891) - json_sax_dom_error_state::is_errored() cannot be constexpr: the class is not a literal type (GCC 4.8, GCC -Wpedantic, Clang 3.5). - to_binary_vector: destroy the vector writer before returning, as the output_vector_sink sets the size of the vector in its destructor. Without NRVO (MSVC Debug in C++14 mode), the vector was moved out before it was resized. - parser::sax_parse: return false instead of result after the early exit (cppcheck identicalConditionAfterEarlyExit). - json.hpp: export detail/exceptions.hpp for IWYU, so json_view.hpp can name exception_id without including a detail header. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address review: throw helpers take input_format_t, simpler type checks - throw_on_discarded() and throw_integer_out_of_range() take an input_format_t and call input_format_name() themselves - has_no_children(), set_parents() and max_size() check array, then object, instead of calling is_primitive() first - parse_impl() chains like accept_impl(); sax_parse_impl() uses if/else - std::array for the control character messages; negation<> in write_object_key(); clarify the is_primitive() comment Signed-off-by: Niels Lohmann <mail@nlohmann.me> * json_view: use exception_id in the value() overloads Signed-off-by: Niels Lohmann <mail@nlohmann.me> * json_sax: assert the invariant of place_container for the static analyzer Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-10-25 06:17:50 +01:00
Niels Lohmann	9c50bdd140	Make the DOM parser cheaper per container and per member (#5882) ... * Reserve the parser's nesting stack once The stack of open arrays and objects was a std::vector<std::uint8_t> that allocated at the first push_back and then reallocated at 1, 2, 4, 8 bytes, so a document nested four levels deep made four allocations before it parsed a value. A document that starts with a container now reserves 32 levels in one allocation; deeper documents grow as before. The loop that reads the tokens is the code it was, so large documents are not affected (a stack object with its own storage, and a reserve at every container, both cost the token loop instructions or cycles). Fixed cost of parsing a 124-byte object (macOS instructions): accept -7.6 %, DOM -2.5 %; no change for a single number. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Build arrays of the DOM parser in one exact-size allocation Reading an array of unknown length appended its elements to a std::vector one at a time, so a small array reallocated (and moved its elements) at sizes 1, 2, 4, ... The DOM SAX parser now collects the elements of arrays whose length is unknown on one stack shared by all open arrays and moves them into an array of exactly the right size when the closing bracket arrives. An array that grows past 16 elements switches to appending to its own vector, so large arrays pay no second copy. Binary formats, which announce the length, and builds with JSON_DIAGNOSTICS or JSON_DIAGNOSTIC_POSITIONS (whose values need their final place from the start) keep storing into the array directly. The stacks of open containers are reserved on first use instead of growing from one element. The binary readers get a DOM parser of their own, json_sax_dom_binary_parser: it is the plain parser that stores every value directly in its container (the code the readers used before). Their containers announce their size, so the bookkeeping of the collecting parser (frame entries, the destination of a finished container) was pure overhead for them: with the collecting parser FromBinaryShape nested was 3-12 % and containers up to 5 % more instructions (GCC) than before. With the plain parser Linux Ir of FromBinaryShape and FromBinaryBuffer is within 0.8 % of develop (Clang: identical). canada.json has 56 000 two-element arrays that took three allocations each and now take two. DOM parse Ir on Linux against develop (GCC / Clang): canada -7.0 % / -7.7 %, citm -0.8 % / -1.6 %, 124-byte object -5.4 % / -5.5 %; accept and SAX are untouched. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Reuse one value for the key events of the callback parser parse() with a callback built a new string value for every key it passed to the callback (a heap allocation and release per key), and copied the key a second time into the object. The callback parser now keeps one string value for key events, which it assigns, and moves the key into the object. Callback parse, Linux Ir (GCC / Clang): twitter -13.3 % / -11.7 %, citm -9.5 % / -7.6 %. (Earlier macOS measurement: twitter -19.5 % instructions.) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Remember where the DOM parser stores the next value The DOM SAX parser looked up the innermost open container for every value: the end of its frame stack, then the object or array pointer in the frame. Compilers have to reload all of that after each value is stored, since a store of a value may alias the frame. The parser now keeps the innermost container's kind and, for an array that stores directly, the array itself in members, updated when a container opens or closes. Flat arrays of numbers are the case that mattered: parsing floats.json (1 000 000 doubles) into a DOM took 2 % more cycles than before the previous commit on macOS and now takes 0.4 % fewer than the code before it (signed_ints.json -0.8 %). Linux Ir is unchanged (canada -6.1 % GCC / -6.9 % Clang, citm -0.3 % / -0.8 %, twitter +0.2 % / 0.0 %). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Look up the keys of big ordered_json objects in a hash table ordered_map keeps its elements in a vector and searches it linearly for every key it adds, so parsing an object with n keys into ordered_json took O(n^2) key comparisons: 33 billion instructions for one object with 30 000 keys, where json needs 245 million. From 64 keys on, the DOM parser now keeps a hash table of the keys of the open object (hashes and positions only; the elements stay in the vector) and finds duplicate keys there. The result is the same, including the position of a key that occurs twice. Smaller objects, std::map based objects and key types without std::hash are untouched. ordered_json parse, macOS instructions: object with 1 000 keys 64.8 M -> 14.6 M, 10 000 keys 5 104 M -> 38.7 M, 30 000 keys 33 358 M -> 89 M. Linux Ir of an object with 2 000 keys: 43.0 M -> 1.9 M (GCC), 37.9 M -> 1.8 M (Clang). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Trim the work the DOM parser does per container and per member Opening and closing an array or object cost the DOM parser more than before the exact-size array change, which showed on documents of many small objects (jeopardy.json: +1.4 % Ir with GCC). This takes it back: - the stack of open containers keeps its first eight entries in the parser object instead of in a vector, so it needs no capacity check beyond the push itself and no allocation for typical documents; - a container is created in the place it ends up in, instead of as a temporary that is then moved there; - a new object member, which is null, is constructed over the null instead of being swapped with it (only with the default base class of basic_json); - the key index that big ordered_json objects use is only a part of the stack entries of objects that can have one; - an array that outgrows the stack reserves twice what it has so far, as a vector that grew by itself would have. DOM parse Ir of the series up to here against develop, Linux GCC / Clang: canada -6.6 % / -7.3 %, citm +1.2 % / +0.2 %, jeopardy +0.4 % / -0.1 %, 124-byte object -6.5 % / -6.2 %. (twitter measures +6.8 % / +6.6 % at this point in steady state, almost all of it in glibc's malloc/consolidate for the new allocation sizes; with the last commit of the series it is -0.1 % / 0.0 %.) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Test parsing containers of the sizes where the DOM parser changes storage The DOM parser collects the elements of an array on a stack (up to 16) and the members of an ordered_json object (up to 32) before it moves them into the container, and uses a hash table for keys from 64 members on. Parse arrays and objects of the sizes around these limits, nested into each other, with repeated keys, and as random trees that are built next to their text, and compare json and ordered_json against the value built through the API. The tests pass on the code from before these changes as well. The random trees are what finds the mistakes that are easy to make here: while the collecting change was being written, a collected array that outgrew the stack at the moment its parent had just reached the limit left its elements in the parent, and only these tests (142 failed assertions) showed it. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Do not convert floats that cannot overflow in accept() accept() only has to find out whether a float token is finite: json_sax_acceptor ignores the value, and the parser rejects only a float that overflows. A float whose digits fit into 19 digits and whose decimal exponent is at most max_exponent10 - 19 is below 10^max_exponent10 and cannot overflow, so the lexer, told by accept() that the value is not needed, skips the conversion for those. A token that may overflow, with more digits, or in another float format is converted as before, so 1e309 and its neighbours are rejected as they were (the new test compares accept() with parse() on tokens around the limit). json::accept(): canada.json Ir -8.6 % (GCC) / -10.9 % (Clang), 124-byte object -3.2 %; cycles on the M1 (release benchmarks, LTO): canada -7.8 %, floats.json -9.0 %. The check tests the accept() flag first, so the parser that needs the value pays one load and branch: SaxParse canada +0.35 % Ir (GCC and Clang), DOM canada +0.4 %. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Collect the members of ordered_json objects on a stack until they are known An ordered_json object keeps its members in a vector and appends every new member to it, so reading an object of unknown size reallocated the vector at 1, 2, 4, 8, ... members, and ordered_map copies all keys on every reallocation. The DOM SAX parser now collects the members of such objects on one stack shared by all open objects, as it does the elements of arrays, and moves them into a vector of exactly the right size when the closing brace arrives. An object that gets past 32 members switches to appending to its own vector (and, from 64, to the key index of the earlier commit). The members are the same, in the same order, and a key that occurs twice keeps the place of its first occurrence. std::map based objects, objects whose size the input announces (binary formats), and builds with JSON_DIAGNOSTICS or JSON_DIAGNOSTIC_POSITIONS are untouched. What only ordered objects need (the collected members, the key indexes) is a member of the parser only for them, and the code that stores a value tries the array that takes its elements directly first. An array that outgrows the stack now reserves 32 elements, the capacity a vector that grew by itself has at that point; with the larger reserve before, parsing floats.json (1 000 000 numbers) was 2 % slower on Apple M1 than before this series. ordered_json parse, Linux Ir against develop (GCC / Clang): citm -11.6 % / -11.7 %, twitter -8.4 % / -8.4 %, jeopardy (12 000 objects of 7 keys) -10.5 % / -8.0 %, canada -3.9 % / -5.2 %, 124-byte object -20.9 % / -16.5 %. macOS time (M1, earlier base): citm -19 %, twitter -13 %, jeopardy -13 %, canada -14 %, 124-byte object -26 %. json parse: steady-state twitter -0.1 % (see the previous commits), citm -1.2 %. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Rename the key index's match callback so Flawfinder doesn't read it as std::equal Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Bind the parser test's strings by reference in the range-for loop GCC and Clang 14 (-Wrange-loop-construct) reported the copy. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Collect container values only in basic_json's own parse functions A json_sax_dom_parser that a user constructs (as in the sax_no_exception example) must hold the values read before a parse error, as on develop: its value is used after the error. The values of containers of unknown size are now collected on a stack only with the new template parameter Collect (default false), which json::parse() and the json_view materialization set; without it, every value is stored in place as it arrives. A test checks the values kept after an error for json and ordered_json. Also fixes the clang-tidy, cpplint and Infer findings: the frame stack keeps its first frames in a std::array (left uninitialized) and the others in a std::unique_ptr, auto for create<>(), NOLINT for member initializers -Weffc++ needs, and an @infer-ignore for the nullptr that place_container() does not use. json::parse() and ordered_json::parse() instructions are unchanged (Apple clang; twitter, citm, canada, jeopardy: -0.03 % to +0.25 %). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Clarify the DOM parser's collection stacks and rename reenter_enclosing_container() Address review comments: explain why a closed container erases only the top of the shared stack, set the payload before the type, rename the helper that makes the enclosing container current again, and declare the parser's state stack right before reserving it. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Export json_sax_dom_parser from the C++20 module for MSVC MSVC needs the detail classes that basic_json befriends to be visible in the module (#3970, as json_sax_dom_callback_parser); with its new template parameter, json_sax_dom_parser failed in ci_module_cpp20 (C2039, not a member of nlohmann::detail). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Keep the overflow-limit numbers of the parser test in a std::vector clang-tidy (cppcoreguidelines-avoid-c-arrays) rejected the C array. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix the clang-tidy findings of the ordered_json parser tests The container sizes are a std::array, the expected text uses raw string literals, the generator's arithmetic is in parentheses, auto for the result of make<>(), and single characters are appended as characters. clang-tidy now reports nothing on the lines the branch adds to unit-ordered_json.cpp and unit-class_parser.cpp (apart from checks CI's clang-tidy does not have). Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-10-24 07:41:21 +02:00
Niels Lohmann	0b924f5375	Split and print JSON pointers in one pass (#5887) ... Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-10-23 06:53:15 +02:00
Niels Lohmann	6fa14d8f95	Write binary formats through reserved sink space and number runs (#5885) ... Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-10-23 06:52:58 +02:00
Niels Lohmann	dec0e9b749	Write strings, scalars and indentation in dump() with fewer calls (#5883) ... Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-10-23 06:52:41 +02:00
Niels Lohmann	86d5c56c54	Speed up flatten(), unflatten(), diff() and patch() further (#5886) ... Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-10-23 06:48:12 +02:00
Niels Lohmann	aa0b371f7c	Faster std::istream input: inline get-area path, binary reads, unbuffered streambufs (#5884) ... Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-10-23 06:48:00 +02:00
Niels Lohmann	9855db81f3	Silence macro-to-enum for the endianness macros (#5881) ... #5876 added JSON_LITTLE_ENDIAN_TARGET and JSON_BIG_ENDIAN_TARGET to macro_scope.hpp. They must stay macros because they select code in #if, so suppress the clang-tidy check as simd_scan.hpp already does. Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-10-22 08:40:05 +02:00
Niels Lohmann	9bf531436f	Speed up flatten(), unflatten(), diff() and patch() (#5861) ... * Flatten into a std::map by inserting next to the previous entry flatten() added every path with operator[], so each insertion into the std::map of the result searched the tree from the root and compared long shared path prefixes, which took half the time on canada.json. Paths come out in nearly sorted order, so they are now inserted with emplace_hint next to the entry added last, for object types that have it. ordered_map keeps its append without lookup, and other object types keep operator[]. Path segments are also appended in place, without a temporary string for each escaped key or array index. flatten() of canada.json, citm_catalog.json and twitter.json is 1.4x to 1.9x as fast; ordered_json is unchanged (1.05x). The output is unchanged. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Unflatten without splitting keys or walking each path from the root unflatten() split every key into a vector of reference tokens, stored the pointer prefixes in a tree to decide which values become arrays, and then looked up every token of every key from the root again. Keys are now checked by a scan that raises the same parse_error.107/108 as split(), skipping the part a key shares with the previous one, which was checked already. The prefixes that denote arrays are kept in a small hash set of key slices; the array/object decision is unchanged. The walk keeps the values along the previous key's path, so only the part of a key that differs from the previous one is looked up. Keys of a flattened object are usually sorted, so that part is short. unflatten() of canada.json, citm_catalog.json and twitter.json is 1.9x to 2.3x as fast; the output and the exceptions are unchanged. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Apply plain JSON Patch operations without building a json_pointer patch() built a json_pointer for every operation, which splits the path into a vector of strings, and for "add" and "remove" a second one for the parent. That took 20% to 45% of the time not spent copying the document. "add", "remove", "replace" and "test" with a path that has no escapes and denotes existing values (the parent, for "add"), as diff() creates them, are now applied by resolving the path in place, reusing one string for the reference tokens. Every other operation, and every operation that would fail, takes the existing code, so the results, the exceptions and the state left after a failing operation are unchanged. patch() of the diffs of canada.json, citm_catalog.json and twitter.json is 1.02x to 1.10x as fast; the rest is the copy of the document. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Build diff operations member by member diff() created every operation from an initializer list, so each {"op", "replace"} pair became a temporary array with two heap strings before it was turned into an object member; an operation took about 25 allocations. The operations are now built as objects whose members are assigned one by one. Paths of members and elements are also built in place, without a temporary string for each escaped key or array index. diff() of canada.json, citm_catalog.json and twitter.json, with every eighth scalar changed, is 1.3x to 1.6x as fast; the output is unchanged. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Diff objects with the same keys in lockstep and keep one path buffer diff() looked up every key of the source object in the target object, and built a new string for the path of every value it descended into. Objects of the same size usually have the same keys in the same order, so they are now walked together as long as that holds, without a lookup; objects that differ fall back to the lookup from the first key that does not match. The path is one buffer that is extended before descending and shortened again afterwards. Objects whose members are equal but in a different order (possible with ordered_json) still get no operations. Together with the previous commit, diff() of canada.json, citm_catalog.json and twitter.json is 1.5x to 2.0x as fast; ordered_json 1.65x to 1.85x. The output is unchanged. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Use std::array for the index buffer in append_index() Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Declare the partner iterator in the loop initializer in diff_object_keys() Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix GCC and clang-tidy warnings in the JSON pointer speedups Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-10-21 21:27:33 +02:00