mirror of
https://github.com/nlohmann/json.git
synced 2026-10-03 13:10:33 +00:00
7e8e8e219bef03eb71fbe0b843ec90cd3280c88d
813
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Niels Lohmann
|
7e8e8e219b |
Merge remote-tracking branch 'origin/develop' into claude/fix-issue-3989-db7e45
Signed-off-by: Niels Lohmann <mail@nlohmann.me> # Conflicts: # include/nlohmann/detail/input/binary_reader.hpp # include/nlohmann/json.hpp # single_include/nlohmann/json.hpp |
||
|
Niels Lohmann
|
c261578431 |
Deduplicate binary reader/writer helpers and fix stale comments (#5730)
* Fix stale and missing comments in binary_writer The doc block of write_number() ended up above the byte_swap() helpers added in #5286, about 80 lines from the function. It was also a plain comment that Doxygen skips, said "write a number to output input", and left BON8 out of the big-endian formats. Move it back onto write_number() as a /*! block and fix the text. write_bson() documented "@pre j.type() == value_t::object", but it throws type_error.317 for every other type, and to_bson() relies on that. Document the exception instead. Explain why the CBOR binary subtype is always written with a 0xD8..0xDB head and never in the one-byte tag form: binary_reader with cbor_tag_handler_t::store only keeps those heads as a subtype, so switching to write_cbor_head() would break round trips for subtypes 0..23. Also fix the grammar of the to_char_type comment. Comments only; no change in behavior, API or ABI. Part of #5710 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Merge the duplicated UBJSON/BJData integer marker ladders write_number_with_ubjson_prefix() (unsigned and signed overloads) and ubjson_prefix() (number_integer and number_unsigned cases) each picked the UBJSON/BJData integer marker (i, U, I, u, l, m, L, M, H) with their own independent if/else ladder, and the values beyond 64 bits were handled by a second, tag-dispatched pair of ladders. An optimized container announces the marker of its first element via ubjson_prefix() and then writes every element through write_number_with_ubjson_prefix(), so the two had to be kept in lockstep by hand across four call sites. Replace all of that with one ubjson_integer_prefix() built on value_in_range_of<T>, and one write_ubjson_integer_payload() that writes the value (or, for 'H', the decimal digits) for a given marker. write_number_with_ubjson_prefix() and ubjson_prefix() keep their signatures and now just call these two helpers. Behavior, the public API and the ABI are unchanged. Verified with a new regression test covering scalars and $-optimized arrays/objects at every int8/uint8/int16/uint16/int32/uint32/int64/uint64 boundary for to_ubjson/to_bjdata (both use_size/use_type settings), and by diffing to_ubjson/to_bjdata output before and after over the json_test_data corpus (bit-identical). Part of #5710 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Remove dead get_char parameters in binary_reader The non-recursive rewrite of the binary readers (#5505, #5506, #5507) left parse_cbor_internal()'s and parse_ubjson_internal()'s get_char parameters dead: parse_cbor_internal() has one caller and it always passes true, and parse_ubjson_internal() has one caller and it always uses the true default. Both parameters, and the @param docs describing the "reuse the last character" mode they used to select, no longer correspond to anything. Drop both parameters, initialise fetch/prefix unconditionally, and update the two call sites in sax_parse(). parse_cbor_value()'s and get_ubjson_string()'s own get_char parameters are unrelated and are left alone; both still have a false caller. Also delete a stray `@return whether a valid MessagePack value was passed to the SAX parser` doxygen block that sits directly above parse_msgpack_value()'s real doc comment, a leftover of the same rewrite. Behavior, the public API and the ABI are unchanged; these are private members of detail::binary_reader. Verified by compiling with -Wunused-parameter and running unit-cbor, unit-ubjson, unit-bjdata and unit-msgpack (offline, against the stubbed test_data.hpp). Part of #5711 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Share the IEEE half-precision decoder between CBOR and BJData binary_reader had two ~45-line copies of the IEEE 754 half-precision decoder: CBOR's case 0xF9 and BJData's case 'h'. Once formatting is normalised, the two blocks were identical except for the byte order used to assemble the 16-bit half (CBOR is big endian, BJData is little endian). Any future change to half-float decoding had to be made and kept in sync in both places. Add one get_half_float(format, little_endian) helper that does the two get()/unexpect_eof() reads, assembles the half in the requested byte order, decodes it per RFC 8949 Appendix D, and calls sax->number_float. Both cases now just call it with their byte order; the BJData case keeps its bjdata-only guard. Behavior, the public API and the ABI are unchanged. Verified with a scratch probe comparing the old and new decoders bit-for-bit (NaN by isnan()) over all 65536 wire byte pairs, in both formats, and by running unit-cbor and unit-bjdata (offline, against the stubbed test_data.hpp). Part of #5711 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Deduplicate the MessagePack unsigned-integer writer ladder The number_integer (non-negative branch) and number_unsigned cases in write_msgpack() each held their own copy of the fixint/uint8/16/32/64 ladder, kept in lockstep only by a comment ("we used the code from the value_t::number_unsigned case here"). Both copies mixed union members: the signed copy compared number_unsigned but wrote number_integer, and vice versa. Extract write_msgpack_unsigned(std::uint64_t), mirroring how write_cbor_head() already avoids the same duplication for CBOR, and call it from both cases. Each case now reads only its own active union member. Output bytes are unchanged for the default 64-bit number types. #5710 item 3 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Unify float marker selection and fix the long double compile error Four formats picked between a float32 and float64 marker through four different helper styles: dummy-argument overloads for CBOR and MessagePack, an std::is_same template for BON8, and a runtime if-chain on input_format_t for write_compact_float(). With number_float_t set to long double, to_cbor, to_msgpack and to_ubjson failed inside the library with "call to 'get_cbor_float_prefix' is ambiguous", while to_bson kept working because write_bson_double() takes a plain double. Change write_compact_float() to take the two marker bytes directly (each of its three callers already knows them at compile time) instead of an input_format_t it only forwarded, and delete the now-unused get_cbor_float_prefix(), get_msgpack_float_prefix(), get_bon8_float_prefix() and get_compact_float_prefix() helpers. Turn the two get_ubjson_float_prefix() overloads into one template. Both write_compact_float() and get_ubjson_float_prefix() now report an unsupported number_float_t with a static_assert naming the requirement, rather than an ambiguous-overload error; the assert lives in the function body, not the class scope, so to_bson with long double is unaffected. Verified with a probe basic_json<..., long double>: to_bson still compiles and round-trips, while to_cbor/to_msgpack/to_ubjson now fail to compile with the new static_assert message. This changes the text of an existing compile error for users with an unsupported number_float_t (documented as a public-API-visible change in #5710). #5710 item 1 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Deduplicate the BJData ndarray writer's dtype dispatch and drop <map> write_bjdata_ndarray() built a 12-entry std::map<string_t, CharType> on every call just to translate the _ArrayType_ name to a dtype marker (the only reason binary_writer.hpp included <map>), then mapped dtype to C++ type twice more: once as a switch for the range-check pass and once as a separate if/else chain for the write pass, with nothing checking that the two agreed. The caller also ran three at() lookups, and the callee called value.at(key) about ten more times for the same three members. Replace the map with bjdata_ndarray_type_marker(), a plain string comparison chain (a C++11 constexpr function cannot contain a switch, so this mirrors binary_reader's own static table style). Replace the switch/if-chain pair with one write_bjdata_ndarray_elements() that switches on dtype once and calls a per-type helper - write_bjdata_ndarray_element<T>() for the eight integer dtypes and write_bjdata_ndarray_float_element() for 'd' - with a dry_run flag selecting the range check or the actual write, so the two passes can no longer disagree on the type. _ArrayType_, _ArraySize_ and _ArrayData_ are now looked up once into references, and the four header marker bytes ('[', '$', '#') are written through to_char_type() like the rest of the UBJSON/BJData writer. The 'd' (single-precision) rule is left exactly as before, since #5707 is expected to change it separately. Verified byte-for-byte identical output before/after for every dtype (including the Draft 2/Draft 3 'byte' fallback and the use_count/ use_type combinations) via a standalone probe, plus round-tripping through from_bjdata(). Overlaps #5707, which is expected to touch the 'd' dtype case, and #5518, which is expected to move the write_bjdata_ndarray() call site. #5710 item 4 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Assert that write_bson_document() consumes every calc_bson_sizes() entry calc_bson_sizes() and write_bson_document() are a hand-synchronized pair of passes over the same object/array tree, introduced by #5553: the size pass appends to nested_sizes in visiting order, and the write pass consumes the table by position with nested_sizes[next_size++]. Nothing checked that the write pass consumed the whole table. If a future change touched only one of the two passes - for example to skip or reject an entry - every later size prefix in the document would be silently wrong. Add JSON_ASSERT(next_size == nested_sizes.size()) where write_bson_document() returns, so such a future drift between the two passes is caught immediately (JSON_ASSERT expands to nothing in release builds using assert(), and the fuzzers/tests already build with it enabled). The two passes agree today, so this changes nothing observable; it only guards against the risk described in #5710 item 5. Extracting a shared stepper for the two passes (the second half of the proposed change) is left for a follow-up: it only saves ~30 lines and the issue asks for it only if the result reads clearly, which needs more room to get right than a mechanical cleanup pass allows. #5710 item 5 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Make the BJData lookup tables static functions instead of members binary_reader held bjd_optimized_type_markers and bjd_types_map as non-static const members (12 string_t objects for the type-name table), built and destroyed on every from_cbor/from_msgpack/from_bson/ from_ubjson/from_bon8/from_bjdata call even though only from_bjdata ever reads them. They also needed the #define/decltype/#undef workaround from #3637 and two NOLINTNEXTLINE suppressions, and binary_writer already carries the same two lists in another form (is_bjdata_excluded_type_marker() and a local std::map in write_bjdata_ndarray(), the latter removed by the item-4 commit), so the excluded-marker lists could drift apart. Replace bjd_optimized_type_markers with static constexpr is_bjd_excluded_optimized_type(char_int_type), using the same ||-chain as binary_writer's is_bjdata_excluded_type_marker(). Replace bjd_types_map with a non-constexpr static bjd_type_name(char_int_type) switch returning nullptr for an unknown marker (a C++11 constexpr function cannot contain a switch). Delete both JSON_BINARY_READER_MAKE_* macros, the bjd_type pair alias, the NOLINTNEXTLINE suppressions, detail::make_array() (no longer used anywhere), and the now-unused <algorithm> and <array> includes. Update the two call sites (the ND-array excluded-type check and the _ArrayType_ lookup) accordingly, and replace unit-bjdata.cpp's "LUT arrays are sorted" section, which only checked the two tables' internal ordering, with a check of all 12 type names and all 8 excluded markers against both new functions. #5711 item 1 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Read CBOR's 1/2/4/8-byte argument through one helper parse_cbor_internal() hand-wrote the same "read a 1/2/4/8-byte big-endian unsigned integer" ladder four times over: - twice for tag numbers 0xD8-0xDB, once in the tag_handler::ignore branch and once, nearly identically, in the ::store branch (~90 lines to read one integer); - twice more for container lengths, once for array heads 0x98-0x9B and once for map heads 0xB8-0xBB, where the 1/2-byte forms called enter_array()/enter_object() directly and the 4/8-byte forms additionally went through get_cbor_container_size(). Add get_cbor_argument(std::uint64_t&), reading the width selected by current & 0x1F via the same get_number() calls as before (so EOF is reported exactly as before), and route all four sites through it: - 0xD8-0xDB now read the argument once per branch instead of switching on `current` a second time; behavior split cleanly from embedded tags 0xC0-0xD7 (tag value in the head, no argument to read), which is now its own case block that no longer has to fall into the ::store switch's "default" case to reach the same tag_pending = true; return true; outcome. - 0x98-0x9B and 0xB8-0xBB collapse into one case block each, always going through get_cbor_container_size() (harmless for 1/2-byte lengths, which already always fit). Verified byte-for-byte identical behavior before/after with a standalone probe covering embedded and multi-byte tags under all three tag_handler_t settings, a tag over a byte string (subtype path), truncated tag/length arguments of every width, and array/map lengths of every width, including the out_of_range.408 "excessive size" case: same exceptions, same messages, same chars_read, same successful results. Left the string/byte-string length ladders in get_cbor_string()/ get_cbor_binary() untouched, as noted in #5711 item 2, since #5325 is expected to touch them separately. Overlaps #5601 (adds a branch right above the embedded-tag case) and #5607 (touches the integer cases 0x18-0x1B, which share this ladder's shape in separate hunks). #5711 item 2 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Add leave_container() to match enter_container() Every container is opened through enter_container(), whose docs promise that a check placed there runs before every start event. The close side had no equivalent: the same "container_stack.pop_back(); dispatch to end_object() or end_array()" sequence was written out separately in BSON, CBOR, MessagePack, UBJSON/BJData and BON8, each copying the pattern of keeping an is_object flag around the pop_back() that would otherwise invalidate a reference to it. A check needed on close would have had to be added in five places, and a sixth copy could go unnoticed. Add leave_container() next to enter_container(), doing the same pop-then-dispatch, and replace the five sites with it. Each site keeps its own surrounding logic (BSON's check_bson_document_size() call before popping, MessagePack's is_object copy used again below, UBJSON/BJData's remaining-container handling after popping, BON8's top used again below); only the repeated pop/dispatch line pair is now shared. Verified all six binary-format unit suites and unit-regression2's deep-nesting tests (dependent count/reuse count and the bjdata ndarray depth cases) still pass, compiled with -Wall -Wextra and ASan/UBSan. Overlaps #5601, which is expected to add a sixth close site in its own skip loop; that site can route through leave_container() too once it lands. #5711 item 4 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Stop passing the input format to sax_parse() when the reader already has it binary_reader's constructor stores the format in the input_format member, and sax_parse(format, sax_, strict, tag_handler) took the same value again purely to dispatch on it. Every in-tree caller passed the same value both times (all 16 from_cbor/from_msgpack/from_ubjson/ from_bjdata/from_bon8/from_bson call sites in json.hpp, and the three public basic_json::sax_parse() overloads), so nothing was broken today, but a caller of the detail class directly (only reachable via JSON_PRIVATE_UNLESS_TESTED, as unit-bjdata.cpp already does) could pass a mismatched pair - say bjdata to the constructor and ubjson to sax_parse - and dispatch on one format while applying the other format's rules; the default-constructed input_format_t::json reader would additionally hit JSON_ASSERT(false) in exception_message() on its first error. Add sax_parse(json_sax_t*, bool, cbor_tag_handler_t) forwarding to the existing overload with the stored input_format, and switch every caller to it: the 16 from_*() sites (keeping their `// cppcheck-suppress[accessMoved]` comments) and the three basic_json::sax_parse() overloads, all of which already had the format available from their own `format` parameter. The four-argument overload is kept for anyone still calling it, now with JSON_ASSERT(format == input_format) so a mismatch fails immediately in a debug build (assert-enabled binaries, including the fuzzers and test suite) instead of misbehaving; verified with a probe that constructs a reader for one format and calls the explicit overload with another, which aborts on that assertion as expected. Removing or asserting against the constructor's input_format_t::json default, which would affect direct detail users, is left as a separate decision per #5711 item 5. Overlaps #5601, which is expected to add an AllowRecovery template parameter to sax_parse() and touch these same call sites in json.hpp. #5711 item 5 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Deduplicate UBJSON/BJData signed-count handling, drop dead ndarray checks get_ubjson_size_value()'s 'i'/'I'/'l'/'L' cases each read a differently sized signed integer and then repeated the same "reject negative with error 113" check; only 'L' additionally checked value_in_range_of for the out_of_range.408 case. Any change to that error path had to be made four times. Add get_ubjson_signed_count<SignedType>(std::size_t&), doing the read, the negative check and the range check once, and route all four markers through it. The range check is a no-op for 'i'/'I'/'l' (their values always fit std::size_t) and only live for 'L' on a 32-bit std::size_t target, matching today's behavior exactly. In the ndarray dimension-product loop, the preceding loop already returns early on any zero dimension and result starts at 1, so `i > 0` in the pre-multiplication overflow check was always true, and `result == 0` in the post-multiplication check could not be reached either: two positive factors whose product does not overflow (as the pre-check already guarantees) cannot be zero. Drop the dead `i > 0 &&` and narrow the post-check to `result == npos`, the one case the pre-check cannot rule out (an exact, non-overflowing match with the sentinel reserved for unknown-size containers), with a comment explaining why. Verified byte-for-byte identical behavior before/after with a standalone probe covering negative counts for every marker, a matching positive count, and ndarray inputs, plus the full unit-ubjson and unit-bjdata suites (same assertion counts as before this change). Overlaps #5601 (rewrites the four parse_error calls and the overflow checks touched here) and #5607/#5707 (touch neighboring lines in the same functions). #5711 item 6 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Drop redundant format parameter and dummy float argument (review) binary_reader::sax_parse(format, ...) only ever had to equal the format given to the constructor, which it asserted. With every caller already on the format-less overload, remove the four-argument overload and dispatch on the stored input_format directly. binary_reader is a detail class, so this is not a public API change. get_ubjson_float_prefix() took a value only to deduce its type; make the type an explicit template argument instead. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
35802e78d6 |
Remove dead Makefile targets; document macro_builder; tidy serve_header (#5735)
* Remove dead doctest help entry and pretty_format target from Makefile The top-level Makefile still carried three leftovers: - The help text listed a "doctest" target that was removed in #4560, so "make doctest" fails with "No rule to make target". The example check now runs as "make check_output -C docs". - "pretty_format" ran clang-format on all sources, but .clang-format was deleted in #4573, so the target reformatted everything in the default LLVM style, against the Artistic Style formatting that "make pretty" applies and CI enforces. - "clean" removed benchmarks/files/numbers/*.json, a directory that no longer exists since the benchmarks moved to tests/benchmarks (#3462). Only maintainer tooling changes; the library is not affected. Part of #5717 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Document tools/macro_builder and tidy up serve_header.py tools/macro_builder generates the NLOHMANN_JSON_EXPAND, NLOHMANN_JSON_GET_MACRO and NLOHMANN_JSON_PASTE* macros in macro_scope.hpp, but nothing referred to it. Add a README that explains what it generates, how to run it and where the output goes, and which dependent tables (NLOHMANN_JSON_DOUBLE_PASTE, NLOHMANN_JSON_TYPE_BODY) are maintained by hand. Point to it from a comment above NLOHMANN_JSON_EXPAND. The generator itself is unchanged; following the README reproduces the header byte for byte. In serve_header.py, drop the LGTM suppression (LGTM.com shut down in 2022), replace the """.""" placeholder docstrings with real ones, and import socket and ssl at module level. DualStackServer.server_bind uses socket, which was only imported under __main__; when the module was imported instead, the NameError was swallowed and IPV6_V6ONLY was not cleared. The header change is a comment only; behavior, API and ABI are unchanged. Part of #5717 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Hash every release_files artifact, not a hardcoded subset The `release` target signed and copied json_fwd.hpp into release_files alongside json.hpp, but the shasum line that writes hashes.txt only listed json.hpp, include.zip and json.tar.xz. Users could not verify the published json_fwd.hpp against hashes.txt. Hash every file in release_files except the .asc signatures instead of naming files by hand, so a newly shipped header (such as the json_literals.hpp that #5610 adds to this target) cannot be missed again. Only affects the generated hashes.txt release artifact; the library itself is unaffected. Overlaps #5610, which touches the same lines to add json_literals.hpp to the release target. #5717 item 1 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Remove the broken fuzz_testing* Makefile targets fuzz_testing and fuzz_testing_{bon8,bson,cbor,msgpack,ubjson} seeded fuzz-testing/testcases from tests/data, which was removed in |
||
|
Niels Lohmann
|
3b6ae43c53 |
Add JSON_DISABLE_TUPLE_REFERENCE_CONVERSION to fix std::tuple conversions (#5598)
* Add JSON_DISABLE_TUPLE_REFERENCE_CONVERSION to fix std::tuple conversions basic_json can be constructed from std::tuple<json&>, which it turns into a one-element array. Because of this, std::tuple picks its converting constructor that converts the whole source tuple instead of the element-wise one. As a result, std::tuple<const json&> built from std::forward_as_tuple(j) binds to a temporary (a compile error with libc++, a dangling reference with other standard libraries), and std::tuple<json> built the same way holds [j] instead of a copy of j. The new opt-in macro JSON_DISABLE_TUPLE_REFERENCE_CONVERSION (CMake option JSON_DisableTupleReferenceConversion) removes the conversion from a one-element tuple holding a reference to the same basic_json type, so std::tuple converts element-wise. It is off by default, so existing behavior is unchanged. It does not change any function body and therefore is not part of the ABI tag. Fixes #2226 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Convert one-element tuples to arrays on every compiler to_json for std::tuple assigns a braced list, j = { std::get<Idx>(t)... }. With a single element that is itself a basic_json, Apple clang 15 and 16 treat j = {x} as a copy of x, so std::tuple<json>{true} became true instead of [true]. The macOS jobs (Xcode 15.1, 16.1) failed the new checks in unit-disable-tuple-reference-conversion and unit-regression2. The one-element overload that already handles JSON_BRACE_INIT_COPY_SEMANTICS builds the array (or object, for a [string, value] element) explicitly, the same way the initializer-list constructor does. Use it unconditionally. The output is unchanged on compilers that already wrapped the element. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Skip json reference tuple tests on clang < 4 and GCC < 5 ci_test_compilers_gcc_old (4.8) and ci_test_compilers_clang (3.4) could not compile the new tuple tests. Creating a std::tuple of basic_json references, e.g. std::forward_as_tuple(j), makes these compilers instantiate basic_json's conversion operator for libstdc++'s internal tuple bases, which fails hard. This happens with and without JSON_DISABLE_TUPLE_REFERENCE_CONVERSION, so it is a limitation of these compilers, not of the new option. Tested with the CI images: clang 3.4 to 3.9 and GCC 4.8 and 4.9 fail, clang 4, 5, and 6 and GCC 5 and 6 compile all cases. Skip only the checks that create such tuples; the is_constructible checks still run. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
fdcc569eee |
Use only documented StringType members in json_pointer (#5692)
contains(const json_pointer&) and operator/=(std::size_t) (and hence
operator/(std::size_t)) used string_t operations that the StringType
template parameter documentation explicitly does not require:
comparing string_t with a const char* literal, c_str(), and
constructibility from std::string. This made both functions fail to
compile for a conforming custom StringType, even though the
documentation's own reference StringType satisfies the requirements.
Fix contains() to compare individual chars ('0'..'9') instead of
comparing string_t with const char* literals, and to call data()
(documented to be null-terminated) instead of c_str(). Fix
operator/=(std::size_t) to build the array-index token via the
existing detail::to_string<StringType> helper (ADL int_to_string() or
assignment from std::to_string()) instead of via std::to_string()
directly, matching how diff(), items(), and std::hash already convert
a std::size_t to a StringType.
Add regression tests to tests/src/unit-alt-string.cpp: contains() for
present/missing keys and indices, "-", a leading zero, and a
non-numeric token on an array, plus json_pointer::operator/(std::size_t).
Fixes #5666.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
|
||
|
Niels Lohmann
|
5f1727cef2 |
Merge remote-tracking branch 'origin/develop' into claude/fix-issue-3989-db7e45
Signed-off-by: Niels Lohmann <mail@nlohmann.me> # Conflicts: # tests/src/unit-alt-string.cpp |
||
|
Niels Lohmann
|
6fc0d501f3 |
Move the user-defined string literals to <nlohmann/json_literals.hpp> and add JSON_NO_AUTOMATIC_UDLS (#5610)
* Add JSON_NO_UDLS to leave out the user-defined string literals The bodies of operator""_json and operator""_json_pointer call the parser, so every translation unit including the library instantiates it, even if it never parses anything. Defining JSON_NO_UDLS leaves the literals out entirely, which saves 15-35% compile time for such translation units (#5294). Nothing changes if the macro is not defined. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Mention JSON_NO_UDLS in the list of exported module symbols Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Move the user-defined string literals to <nlohmann/json_literals.hpp> Following the review in #5294, the literals now live in their own header instead of being removed entirely: <nlohmann/json.hpp> includes it at the end unless JSON_NO_AUTOMATIC_UDLS (renamed from JSON_NO_UDLS) is defined, so a project can opt out globally and include the header only where the literals are used. The header only uses public and standard macros, because the library's internal macros are undefined at the end of json.hpp and the amalgamation inlines macro_scope.hpp only once. For the same reason, the library no longer defines and undefines JSON_USE_GLOBAL_UDLS, so a user's definition is still visible to the header. The single-header copy is identical to the multi-header one, as it only includes <nlohmann/json.hpp>. The module always exports the literals. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix CI: include cycle, GCC 4.8 literal operator spacing, and global UDLs off in the JSON_NO_AUTOMATIC_UDLS test - Suppress clang-tidy misc-header-include-cycle on the intentional mutual include of json.hpp and json_literals.hpp. - Use operator"" _json with a space for GCC 4.8 in the test's detection aliases, as the header does. - Only test the global literal operators when JSON_USE_GLOBAL_UDLS is on. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Declare the literal operators through a local macro The GCC 4.8 spacing condition was repeated for both operator definitions and the global using-declarations. NLOHMANN_JSON_LITERAL_OPERATOR(suffix) now selects operator""##suffix or operator"" suffix in one place and is undefined at the end of json_literals.hpp. Suggested by gregmarr in review. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
7fd6895788 |
Hide a discarded container's content from the parser callback (#5706)
When a parser callback rejects an object's or array's start event,
json_sax_dom_callback_parser kept calling it for everything inside
that container anyway: nested keys, values, and the start/end events
of containers below it. This contradicts parser_callback_t's own
documentation, which promises that discarding a container at its
start event also hides its content from the callback.
The same code path also kept a full copy of every key inside such a
discarded container in key_stack until the whole parse finished,
because the early return for values that are not stored skipped the
matching pop. Filtering out a large subtree is the main reason to use
a callback, so this made peak memory during the parse scale with the
size of the very subtree the callback was trying to skip.
Fix start_object(), start_array(), and key() so that a container
whose own start event was discarded, or that is nested inside one, is
never handed to the callback, and no longer pushes onto the key
stacks. A container whose start event was accepted but whose key was
rejected still gets its content reported, as documented ("the
callback is still called for the associated value, but its return
value has no further effect"); only its own bookkeeping is skipped
since it will not be stored.
Fixes #5643.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
|
||
|
Niels Lohmann
|
5bd766aa50 |
Move to_bson's binary subtype check into calc_bson_sizes (#5703)
to_bson() rejected a binary value's subtype above 255 (out_of_range.415) in write_bson_binary(), which only has the binary_t, not the basic_json value that holds it, so the exception was created with no JSON_DIAGNOSTICS context even though the equivalent to_msgpack() check names the value's path. The check also ran after the document size, all preceding elements, and this element's header and length had already reached the output adapter, so a caller-provided std::vector or std::string ended up holding a truncated document. calc_bson_sizes() already walks every value before anything is written, to size embedded documents and arrays and to reject invalid keys (out_of_range.409) up front. The subtype check now runs there instead, in calc_bson_binary_size(), which is given the basic_json value so the exception can use it as context. The now-redundant check in write_bson_binary() is removed, since calc_bson_sizes() always throws first if any binary value in the document has an oversized subtype. Fixes #5675. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
4bb1b14b06 |
Trim the compiler-appended NUL from wide/UTF string literals too (#5702)
With JSON_STRICT_NUL_HANDLING defined to 1, parsing a wide, UTF-16, UTF-32, or (C++20) UTF-8 string literal (e.g. json::parse(L"[1]")) failed with parse_error.101 at the terminating NUL of the literal, and accept() returned false. The array overload of input_adapter() only dropped the compiler-added trailing '\0' for arrays of char, so for wchar_t, char16_t, char32_t, and char8_t arrays that terminator was passed to the parser as data, which the macro then rejected. Broaden the trimming to every character type that a string literal can use (char, wchar_t, char16_t, char32_t, and, since C++20, char8_t). Arrays of any other element type (unsigned char, std::uint8_t, ...), as used for CBOR/MessagePack, are unaffected: a trailing zero byte there is still read as genuine data. Fixes #5658. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
 Niels LohmannandClaude Sonnet 5
|
bfe0f32d71 |
Fix std::terminate and null pointer access in input_stream_adapter (#5699)
Parsing from a std::istream crashed in two unusual but valid stream states, both in input_stream_adapter: - With eofbit in the stream's exceptions() mask, get_character() sets eofbit via is->clear(), which throws std::ios_base::failure. While that exception unwinds, ~input_stream_adapter() called clear() again to reset eofbit, which is still set and still in the exception mask, so it throws a second time out of the (implicitly noexcept) destructor and std::terminate() is called. The destructor now only calls clear() if a bit other than eofbit remains set, so the first exception can propagate normally. - For an std::istream without a stream buffer (rdbuf() == nullptr, e.g. std::istream(nullptr)), the constructor stored the null pointer without checking it, and get_character() dereferenced it. input_adapter(std::istream&) now throws parse_error.101 for such a stream, the same as it already does for a null FILE* or char*. Added regression tests to unit-deserialization.cpp and, for the JSON_PRECISE_STREAM_POSITION variant of get_character(), to unit-precise-stream-position.cpp; both crashed before this fix. Documented the two exceptions in parse.md and operator_gtgt.md. Fixes #5646. Signed-off-by: Niels Lohmann <mail@nlohmann.me> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
Niels Lohmann
|
44a88d85be |
Fix NLOHMANN_JSON_SERIALIZE_ENUM_STRICT's from_json message (#5698)
from_json built its out_of_range.410 message with "..." + j.dump(). If the unmatched value is (or contains) a string with invalid UTF-8, that dump() itself throws type_error.316, so the caller got type_error.316 instead of the documented out_of_range.410; such strings can reach get<Enum>() unvalidated, e.g. from from_cbor()/from_msgpack(). With a custom string_t, j.dump() returns that type, and "const char*" + string_t does not compile unless the type happens to provide operator+, so the macro failed to compile for such types. Build the message with detail::concat(), which appends any type exposing data()/size() and always yields a std::string, and dump with error_handler_t::replace so building the message itself cannot throw. Added regression tests: an invalid-UTF-8 case in the existing strict-enum test in unit-conversions.cpp, and a strict-enum use with alt_string (the custom string_t from unit-alt-string.cpp) to cover the compile failure. Fixes #5667. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
bfea6f36d3 |
Fix to_msgpack() reading the inactive number union member (#5694)
* Fix to_msgpack() reading the inactive number union member basic_json stores number_integer and number_unsigned in a union, and number_unsigned_t only has to be at least as wide as number_integer_t (with the default types, both are 64-bit and have the same representation). When number_integer_t is narrower, write_msgpack() read the wrong union member in two places: - The number_unsigned case wrote number_integer's bits instead of number_unsigned's, silently writing the wrong value whenever it did not fit in number_integer_t. - The number_integer case (non-negative branch) picked the encoded width by comparing number_unsigned's bits, which is undefined behavior, though the value written was still number_integer's, so at worst a too-wide encoding was chosen. Read the active member in both cases, like the other binary writers (CBOR, UBJSON, BJData, BSON, BON8) already do. Fixes #5644. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Cast number_integer to number_unsigned_t only once in to_msgpack() Addresses review comment by @gregmarr. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
42f89e3130 |
Let to_json(std::optional<T>) propagate exceptions from T's to_json (#5684)
* Let to_json(std::optional<T>) propagate exceptions from T's to_json The overload was marked noexcept even though its body assigns *opt to the JSON value, which calls T's to_json (or allocates for std::string, std::vector, or json). Any exception from there -- a user-defined to_json reporting an error, or std::bad_alloc -- called std::terminate() instead of propagating. The noexcept also made basic_json's converting constructor noexcept(true) for std::optional<T>, so json j = opt; could not report the error either. Fixes #5642. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix CI: make to_json(std::optional<T>) conditionally noexcept GCC's -Wnoexcept (an error in ci_test_gcc) fired at to_json_fn's noexcept(noexcept(to_json(j, val))): after dropping the unconditional noexcept, to_json(std::optional<int>) had no exception specification although GCC could prove its body cannot throw. It also made json(std::optional<int>) lose its noexcept. Declare the overload noexcept exactly when assigning the contained value to the JSON value is (std::is_nothrow_assignable<BasicJsonType&, const T&>), which is what the body does. std::optional<int> is noexcept again; a T whose to_json may throw still propagates the exception. Static assertions in the test check both cases. The test's throwing to_json triggered -Wmissing-prototypes and -Wmissing-noreturn (clang) and -Wmissing-declarations and -Wsuggest-attribute=noreturn (GCC). Move the type and its to_json into an anonymous namespace, mark the function [[noreturn]], and compile them only without JSON_NOEXCEPTION, like the test that uses them. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
6d7845d207 |
Fix deprecated json_pointer/string operator== warning in value() (#5683)
value(KeyType&&, default) is constrained on is_comparable_with_object_key, which passes KeyType as a reference. is_comparable's dispatch on is_json_pointer_of<A, B> only matches a json_pointer as a plain type or a plain reference, so a const-qualified reference (as produced when KeyType is deduced from a json_pointer argument) fell through to is_comparable_no_json_pointer, which instantiates the deprecated json_pointer/string comparison operators. This made ordered_json's transparent comparator (and any transparent comparator on a custom string type) warn under -Wdeprecated-declarations when calling value(json_pointer, default), even though no such comparison is ever performed. at() was already fixed for this in #5289, which does not use is_comparable_with_object_key. Strip references and cv-qualifiers with uncvref_t before the is_json_pointer_of dispatch, so any reference-to-json_pointer is recognized regardless of qualifiers. Fixes #5664. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
678fd3017b |
Fix element path for map/unordered_map JSON_DIAGNOSTICS errors (#5681)
When converting a JSON array to std::map or std::unordered_map with a non-string key, each element must itself be a [key, value] array. If an element is not an array, from_json() threw type_error 302 with the outer array's value (&j) as the exception context, so with JSON_DIAGNOSTICS enabled the message pointed at the whole array instead of the offending element (e.g. "(/outer/m)" instead of "(/outer/m/2)"), even though the message text already described the element's type. Both from_json() overloads now pass the element (&p) as the context, so the reported JSON Pointer matches the type named in the message, the same way std::vector<std::vector<T>> and similar conversions already do. Fixes #5668. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
68beba727c |
Fix from_json() for enums with underlying type bool (#5679)
get_arithmetic_value() rejects boolean_t, so the default from_json()
for enums failed to compile for an enum whose underlying type is
bool (e.g. enum class Flag : bool { off, on }), even though the
matching to_json() serializes such enums as an unsigned number.
Read the underlying value through number_unsigned_t in that case,
matching what to_json() writes, then cast back to the underlying
type before constructing the enum.
Fixes #5671.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
|
||
|
Niels Lohmann
|
0050d0f7a4 |
Share one nesting depth limit between all bounded descents (#5637)
Copying and comparing stopped their descent at basic_json::nesting_depth_limit(), while serializing, hashing and merging used detail::recursion_depth_limit(). Both were 128, but nothing kept them equal. The thread-local count now tests against detail::recursion_depth_limit() as well, and a static_assert keeps the limit small enough for the byte that holds the count. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
6a8a7735ed |
Do not throw in contains() for an empty array reference token (#5614)
* Do not throw in contains() for an empty array reference token json_pointer::contains() rejected malformed array indices, but an empty reference token (e.g. "/a/" where "a" is an array, or "/" on an array) passed every check and reached array_index(), which throws out_of_range.404. contains() must not throw (cf. #5395), so it now returns false for an empty token. at() still throws out_of_range.404. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix CI: bind j_nested_const by reference in the json_pointer test clang-tidy (ci_clang_tidy) flagged the new test with performance-unnecessary-copy-initialization: the local copy j_nested_const of j_nested is never modified. Bind it as a const reference instead; it still exercises the const overloads of at() and contains(). Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
d268eaa693 |
Convert floats with Eisel-Lemire when std::from_chars is unavailable (#5617)
* Convert floats with Eisel-Lemire when std::from_chars is unavailable Float tokens that Clinger's fast path cannot convert (e.g. the 17-digit coordinates of canada.json) went to strtod unless std::from_chars was available. It is not used in C++11/14, and not with libc++, which does not define __cpp_lib_to_chars. The Eisel-Lemire algorithm (after fast_float's compute_float) now converts them with integer arithmetic, correctly rounded for any token with at most 19 significant digits. Longer tokens are truncated; the result is used if w and w + 1 round alike, else strtod decides as before. Overflow still yields infinity (out_of_range.406). The table of powers of five (fast_float's) lives in pow5_table.hpp; a unit test recomputes every entry with big-integer arithmetic. Further tests: known values generated with Python (whose float() is correctly rounded), 200,000 round trips through to_chars, and the 128-bit multiplication and leading-zero count against big-integer references (both with and without a 128-bit type). Checked against strtod on 6.5 million tokens, among them 60,000 exact halfway cases: no difference. json::parse on canada.json: -8.6% (C++11), -7.6% (C++17, Apple clang); other files unchanged. Compile time of a TU including json.hpp: +0.7%. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix CI: unused parse result and find() == npos in the Eisel-Lemire tests GCC (-Werror=unused-result) rejected CHECK_THROWS_WITH_AS(json::parse(...)) because parse() is [[nodiscard]]; assign the result to a dummy json as the other tests do. clang-tidy flagged longer.find('.') == npos with abseil-string-find-str-contains; store the position in a variable first. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
5b9ffae8bb |
Move the float conversion chain out of the lexer (#5616)
lexer::convert_number() converted float tokens with std::from_chars (when available), Clinger's fast path, and the locale-aware strtod fallback, all as lexer members. They are now free functions in number_parse.hpp: - convert_float_fast(): std::from_chars, then Clinger's fast path, skipped when the mantissa has too many significant digits - convert_float_locale_aware(): strtof/strtod/strtold with the decimal point of the current locale, retried when the locale changed (#5198) so that other code converting JSON number tokens gets the same values. No change in behavior; the lexer no longer includes <clocale> and <cstdlib>. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
792853d725 |
Fix CI: clang-tidy and JSON_NOEXCEPTION in the #3989 changes
ci_clang_tidy flagged nested conditional operators in the BON8 skip code and in remove_incomplete_utf8_sequence() (readability-avoid-nested-conditional-operator), and two branches with the same body in recover_string() (bugprone-branch-clone). Use if chains instead of the nested conditionals and merge the two branches into one condition; the short-circuit order is unchanged. ci_test_noexceptions aborted in the #3989 regression test: it compares the first recovered error with the message of the exception that from_cbor() and friends throw, and under JSON_NOEXCEPTION that call aborts instead of throwing. Guard binary_error_message() and its uses with #if !defined(JSON_NOEXCEPTION), as other tests in the suite do. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
3e683e9c04 |
Merge branch 'develop' into claude/fix-issue-3989-db7e45
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
d1d84ed9af |
Fix Flawfinder: format the BSON element type without snprintf
Moving the report of an unsupported BSON element type into skip_unsupported_bson_element() moved its snprintf() call, which Flawfinder then reported as a new CWE-134 finding. The two hexadecimal digits are now computed directly; the message is unchanged. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
fc03b9912e |
Look up the locale decimal point at conversion time, not lexer construction (#5597)
* Look up the locale decimal point at conversion time, not lexer construction The lexer read localeconv()->decimal_point once in its constructor and wrote that character into token_buffer in place of '.'. The strtod fallback then used the locale current at conversion time, so an LC_NUMERIC change in between (parser callback, SAX handler, another thread) truncated the value in release builds and fired the endptr assertion in debug builds. token_buffer now always holds '.'. Only the strtof/strtod/strtold fallback depends on the locale: it looks up the decimal point right before the call, restores '.' afterwards, and repeats the conversion if the locale changed in between. As a side effect, std::from_chars and Clinger's fast path now also apply under locales whose decimal point is not '.'. Fixes #5198 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Stop the strtod retry loop when the decimal point is unchanged convert_float_locale_aware() repeated the conversion until strtod consumed the whole token, assuming an early stop can only mean a locale change. Under a locale whose decimal point is not a single character (e.g. the two-byte U+066B of ar_EG.UTF-8, ar_SA.UTF-8, or fa_IR.UTF-8, all available on macOS), the in-place substitution can never succeed, so parsing any float that reaches the strtod fallback (for example 3.14159265358979323846 at C++11) hung forever. Before this branch, the same input was truncated. Retry only if the decimal point changed since the previous attempt; otherwise keep the value strtod parsed so far, as before. Add a test that parses such numbers under a multi-byte decimal point locale; it hangs without this change. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix -Weffc++ errors in the #5198 locale test GCC's -Weffc++ (an error in ci_test_gcc and ci_test_standards_gcc) rejected LocaleSwitchingSax: it has a pointer data member but does not declare its copy operations, and its vectors are not initialized in the member initializer list. Store the locale name as a std::string and give the vectors brace initializers, like SaxEventLogger in unit-deserialization.cpp. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
de8529f99b |
Merge remote-tracking branch 'origin/develop' into claude/fix-issue-3989-db7e45
Signed-off-by: Niels Lohmann <mail@nlohmann.me> # Conflicts: # include/nlohmann/detail/input/binary_reader.hpp # single_include/nlohmann/json.hpp |
||
|
Niels Lohmann
|
9e1a09eec0 |
Name the key type when rejecting non-string CBOR/MessagePack map keys (#5594)
* Name the key type when rejecting non-string CBOR/MessagePack map keys CBOR and MessagePack allow map keys of any type, but JSON object keys are always strings, so such maps are rejected. The error so far was the one for a malformed string (e.g. "expected length specification (0xA0-0xBF, 0xD9-0xDB); last byte: 0xC0" for a nil key), which does not tell the user what went wrong. Report the type of the key instead: syntax error while parsing MessagePack object key: only string keys are supported, but found nil; last byte: 0xC0 The exception id (parse_error.113) and type are unchanged. Malformed string keys and a missing key keep their previous messages. Document the restriction on the CBOR and MessagePack pages. Refs #2766, #3381 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Point the MessagePack key note to the spec's profile section The note linked to "Serialization: type to format conversion", which says nothing about key types. Restricting map keys to strings is only mentioned in the "Profile" section (under "Future discussion") as an example of a JSON-compatible profile, so link there and describe it as such instead of as a permission. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
677794f076 |
Fix CI: recover numbers with the string operations every string_t has
recover_number() used back(), pop_back(), front(), and find_first_of(const char*), which the minimal alt_string of unit-alt-string.cpp does not provide. Since the binary readers recover UBJSON/BJData high-precision numbers with it, from_ubjson() instantiated it, too, and the test no longer compiled. It now uses only size(), operator[], and resize(), and unit-alt-string.cpp recovers from errors in JSON text, UBJSON, and CBOR. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
437a95cfdb |
Repair complete items in binary formats when parse_error() returns true (#3989)
When the SAX parser asks to recover, the binary readers now repair an item whose end is known and read on after it, as RFC 8949, Section 5.3 describes for CBOR: - CBOR: tags are ignored, and simple values other than false, true, and null become null (RFC 8949, Section 6.1); a negative integer below the range of number_integer_t becomes the nearest floating-point number. - Strings that are not valid UTF-8 get U+FFFD for each ill-formed sequence, as in JSON text; so does a UBJSON/BJData char above 0x7F. - UBJSON/BJData high-precision numbers keep their longest valid beginning (via the lexer's recover_token()), or become infinity. - Members whose key is not a string are skipped (CBOR, MessagePack, BON8), like members without a key in JSON text. - BSON elements of types the library does not read (ObjectId, datetime, decimal128, ...) become null; a string without its terminator and a document whose size does not match are kept. Where the end of an item is unknown, reading stops as before, except that BSON skips to the end of the document, whose size it knows. The value read before such an error is now completed by the reader from its container stack, as the JSON parser does, instead of by a proxy SAX parser, which is removed. Like the parser, binary_reader gets an AllowRecovery template parameter, so that from_*() compile without the new code. Tests: a table of repairs, numbers out of range, errors that stop, and a sweep over changed and removed bytes of eight encodings that checks balanced events and that the first error is the one from_*() reports. All fuzzers now run a recovering checker; the binary ones also check that it reports an error exactly when from_*() fails. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
9adb510a0d |
Recover from parse errors when parse_error() returns true (#3989)
The return value of json_sax::parse_error() was documented both as "must return false" and as "whether the parsing should continue", and the code just passed it on. For JSON text, parsing stopped anyway, but sax_parse() could report success for invalid input. The binary readers read on after the error, looping forever on a CBOR indefinite-length array without its end. Now false stops parsing, and true recovers from the error: - JSON text is repaired with the smallest local edit (insert a missing ',' or ':', remove a stray token, keep the readable part of a broken string or number, null for a value that cannot be read, close the innermost container at a wrong closing bracket and all of them at the end of the input), and parsing continues. The SAX events stay balanced, every key is followed by exactly one value, and each token is reported at most once. - The binary formats cannot resynchronize, so they stop, but complete the value read so far. sax_parse() returns false after any error. parse(), accept(), and the from_*() functions never recover and compile to the same code as before. Supersedes #4522. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
1e101ecac1 |
Add BON8 support (#2998)
* Add BON8 support Add to_bon8/from_bon8 and input_format_t::bon8 for BON8, a binary format that uses the byte values that cannot begin a UTF-8 character as type markers, so strings need no length prefix. It is the most compact of the supported binary formats on the benchmark files. The reader is non-recursive like the other binary readers. A string ends at the first byte that cannot continue it, so the reader hands the one or two bytes it reads past a string back to the value that follows. The writer produces the canonical representation of the specification, except for NFC normalization; its output is identical to that of the reference implementation (HikoGUI) on all files of the test data. The round-trip tests need the .bon8 files of json_test_data 3.2.0. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address review comments - Reuse detail::validate_one_utf8 to check strings in to_bon8; the error now names the first byte of the invalid sequence. - Document that to_bon8 leaves bytes in the output adapter on an exception, and that string_open is only an output of write_bon8_marker. - Explain why the pushback buffer of the BON8 reader cannot overflow. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Select the BON8 float prefix by type get_bon8_float_prefix only depends on the type of its argument, so make the type a template parameter instead of passing an unused value. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Rename a test variable that Flawfinder mistakes for read() Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix the BON8 CI failures - compare the float in write_bon8_float with number_float_t constants, so GCC does not warn about a float-to-double conversion - mark check_bon8_utf8's context as used when exceptions are disabled - choose the compact float prefix in a helper rather than with nested conditional operators (clang-tidy) - use auto for the cast in the BON8 integer reader (clang-tidy) - write the int32 minimum test values as long long literals (MSVC C4146) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Amalgamate Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Read BON8 strings in bulk from contiguous input - copy the valid UTF-8 of a string in one step when the input is contiguous (twitter.json is read in 1.68 instead of 2.52 ms, jeopardy.json in 196 instead of 297 ms, close to CBOR and MessagePack) - share the new valid_utf8_prefix() with the writer's UTF-8 check, which now skips ASCII 8 bytes at a time - let the fuzzer check that contiguous and stream input give the same value or error, and test both paths in the unit tests - clarify that a second 0xFF after a string is an empty string Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Link the BON8 functions from the other binary format pages Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Name the bulk scan flag after the input, not BON8 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Read BSON keys in bulk from contiguous input BSON keys (and array indices) are C-style strings, which were read byte by byte. For contiguous input they are now read up to their \x00-byte in one step, using the same bulk_scan flag as BON8 strings: twitter.json is read in 1.46 instead of 2.01 ms, citm_catalog.json in 2.93 instead of 3.33 ms, jeopardy.json in 182 instead of 207 ms. canada.json, whose keys are almost all one-digit array indices, takes 2 % longer. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix the BON8 CI failures of the bulk-read tests - skip the contiguous-versus-stream tests of BON8 strings and BSON keys when exceptions are disabled: they catch the parse errors of invalid input, and without exceptions the library aborts instead - use static_cast for the int64 test value (google-readability-casting) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Move the explicit basic_json instantiation into its own test file Linking test-regression3_cpp20 with clang and MinGW failed with "relocation truncated to fit: IMAGE_REL_AMD64_REL32 against `.rdata'", as test-regression2 did before #5511. The explicit instantiation of basic_json<> for #4825 compiles every member function, including the BON8 reader and writer, into that object, and it was already close to the limit (2,226,104 bytes on develop, 2,234,960 with BON8; clang -O1, C++20). Give the instantiation a file of its own: unit-regression3 is now 1,594,736 bytes and unit-explicit_instantiation 1,095,064. The new file mentions JSON_HAS_CPP_17 and JSON_HAS_CPP_20 so it keeps being built for the C++17 standard the regression was about. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Convert the bytes of the BON8 test strings explicitly The str() helper constructed a std::string from a byte range, which converts each unsigned char implicitly; -fsanitize=integer reports that for bytes of 0x80 and above (ci_test_clang_sanitizer). Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
 Kartikey Negi
|
6bd106893a |
Fix CBOR tag handling in cbor_tag_handler_t::store for non-binary items (#5559)
When using cbor_tag_handler_t::store, tags 0xD8-0xDB previously assumed that the tagged item was a byte string, unconditionally attempting to parse binary data and failing on valid CBOR documents containing tags applied to integers, strings, arrays, or objects (such as self-describe tag 55799). Check whether the tagged data item is a byte string (0x40-0x5B or 0x5F). If it is a byte string, store the subtype on the binary value as before. Otherwise, iteratively process the tagged value in the driver loop using item_read so that chained tags do not consume native stack space. Part of #5316. Signed-off-by: ReturnKartikey <kartikeynegi2000.work@gmail.com> |
||
|
Niels Lohmann
|
f7972970a4 |
Throw instead of writing MessagePack lengths beyond UINT32_MAX (#5584)
* Throw instead of writing MessagePack lengths beyond UINT32_MAX MessagePack stores the length of a string, binary value, array, or object in at most 32 bits. For a larger value, to_msgpack wrote no length at all, so the output could not be read back. It now throws out_of_range.412, which BSON already uses for its 32-bit length fields. The check lives in one function, so each length is written by an if/else chain that ends in a plain else, without a condition that can never be false. It is tested with string and binary types that report a size beyond UINT32_MAX without allocating it, like the BSON tests do. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix the CI failures of the MessagePack length check - mark to_msgpack_length's value as used when exceptions are disabled (-Wunused-parameter, misc-unused-parameters) - put "Exception safety" before "Exceptions" in to_msgpack.md, as the documentation style check requires - create the test's string value from its type: constructing it from a beyond_uint32_string_t considers the std::filesystem::path conversion, which libstdc++ 10 reports as ambiguous for a class derived from std::string (clang 13) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Skip the MessagePack string length test for clang with libstdc++ 10 C++17 builds consider the std::filesystem::path conversion for the string type, and with clang and libstdc++ 10 that conversion is ambiguous for a class derived from std::string. Creating the value from its type did not avoid it, since any basic_json with that string type instantiates the check. The binary and ext cases are still tested there. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Keep the MessagePack string test type and its alias in one block astyle indented the alias oddly when it had an #ifdef of its own after the binary alias; declare it right after the string type, in the same block. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
4fa95d9810 |
Remove unreachable branches from the binary writer (#5583)
Coverage reported conditions in the binary writer that can never be false, and marked the code behind them with LCOV_EXCL. Remove them instead of excluding them: - CBOR writes the length of a string, binary value, array, or object exactly like an unsigned integer, only with another major type. One function, write_cbor_head(), now writes both, so the integer tests cover every width and the four excluded 64-bit length branches are gone. - A last `else if` whose condition holds for every remaining value (an unsigned value at most UINT64_MAX, a signed one in the range of int64_t) is now a plain `else`. - Whether a signed integer fits into an int64 for UBJSON and BJData is decided by its type at compile time. Only an integer type wider than 64 bits gets a range check and the high-precision fallback. - The private get_impl(boolean_t*) was never called. The UBJSON type prefix 'H' of an optimized container of unsigned integers beyond the range of int64 was reachable although excluded; it is tested now. The output is unchanged. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
 Dadi Reddy Sai Praneeth Reddy
|
fe4a544c7e |
handled when size exceed uint32 (#5515)
* handled when size exceed uint32 Signed-off-by: dsp0redy <saipraneethreddy.dadireddy@gmail.com> * addressed review comments Signed-off-by: dsp0redy <saipraneethreddy.dadireddy@gmail.com> * updated unit test Signed-off-by: dsp0redy <saipraneethreddy.dadireddy@gmail.com> * added amalgamation patch Signed-off-by: dsp0redy <saipraneethreddy.dadireddy@gmail.com> --------- Signed-off-by: dsp0redy <saipraneethreddy.dadireddy@gmail.com> |
||
|
Niels Lohmann
|
95e9a5931c |
Write BSON in linear time, without recursing per nesting level (#5553)
* Write BSON in linear time, without recursing per nesting level to_bson() had two problems with nested values: - It recursed once per nesting level, so a value nested deeply enough - 100,000 levels on an 8 MiB stack - exhausted the call stack and terminated the process, although parse() accepts such values without complaint. - BSON prefixes every document and array with its length. The writer computed that length by walking the entire value below it, again for every nested document it wrote, which made serializing O(size x depth). A 200-level document took 30 ms instead of 1. Both passes are now iterative, and each length is computed exactly once: - calc_bson_sizes() computes the length of every document and array in one pass, each from the lengths of its entries, into a table ordered the way they are written. - write_bson_document() then writes the document, taking each length from the table. Everything observable is unchanged, as a differential test against develop confirms byte for byte: - The same bytes are written. - A key containing U+0000 still throws out_of_range.409 for the same first key, with the same diagnostics path, before anything is written. - A document too large for BSON still throws out_of_range.412 before anything is written. - A binary subtype above 255 still throws out_of_range.415 after the same partial output. Only the enclosing objects and arrays are kept on a stack, so a flat document allocates nothing for it. Measured against develop (clang -O3, median of 201 runs): flat objects unchanged, flat arrays 37% faster (the array length was computed twice), a nested 3,000-object document 2x faster, a 200-level document 33x faster. to_bson.md documented the quadratic complexity since #5334; it is linear again. Fixes #5392 for BSON, and #5308. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Do not require a default-constructible string_t in the BSON writer GCC 4.9 and MSVC rejected the test's huge_string_t, which has no default constructor; develop never default-constructed string_t here either. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Let the BSON index-name helper only fill its output parameter It returned a reference to the string it filled, so callers held a second name for index_name. Addresses review feedback. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
1e44262091 |
Make JSON_STRICT_NUL_HANDLING part of the ABI tag (#5560)
* Make JSON_STRICT_NUL_HANDLING part of the ABI tag JSON_STRICT_NUL_HANDLING (#5534) changes the bodies of inline functions: the lexer's handling of '\0' and input_adapter() for char arrays. So translation units compiled with and without it define the same functions differently, an ODR violation - the case the ABI tag exists for, as with JSON_BRACE_INIT_COPY_SEMANTICS (_bics). It now appends _snul to the inline namespace. The macro is new in 3.13.0, so no existing namespace changes. Its default moves to abi_macros.hpp, and it is only #undef'd without JSON_TEST_KEEP_MACROS, as for the other ABI macros. The ABI config tests, the namespace docs, the macro's docs and the Natvis file cover the new tag. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Amalgamate Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels LohmannandClaude Sonnet 5
|
d19f7f5dce |
Fix BSON conformance issue (#5185)
* 🐛 fix BSON conformance issue Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🐛 fix BSON conformance issue Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🐛 reject ill-formed UTF-8 in CBOR/MessagePack/BSON text strings at decode time (#5531) from_cbor()/from_msgpack()/from_bson() copied the raw bytes of a decoded text string into the resulting json value without any UTF-8 validation, even though RFC 8949 §3.1 (CBOR) and the MessagePack/BSON specifications all require text strings to be valid UTF-8. Malformed input only failed later, if the value was dump()'d, with a type_error.316 - so the allow_exceptions=false pattern used specifically to get a discarded sentinel instead of an exception did not discard this category of malformed input, unlike every other kind of malformed binary input this library rejects at decode time (see #5529). Fix this at the single choke point shared by BSON/CBOR/MessagePack/UBJSON string reads, binary_reader::get_string(): validate the bytes with the UTF-8 DFA right after they are read, and report failures the same way as every other binary_reader error (parse_error.113), so allow_exceptions and strict discarding behave consistently. get_binary()/binary blob reads are untouched and still accept arbitrary bytes, since only text strings are required to be UTF-8. There were two independent implementations of a UTF-8 validator: the lexer's streaming scanner, and the serializer's Hoehrmann DFA used by dump_escaped_impl(). Rather than write a third, the serializer's decode() function, its utf8d table and the UTF8_ACCEPT/UTF8_REJECT constants are extracted into detail/string_utils.hpp (a low-level header already included before both detail/input/ and detail/output/), alongside a new is_valid_utf8() helper built on the same decode() step. serializer.hpp's dump_escaped_impl() now calls the shared decode(), so there is exactly one UTF-8 validator in the codebase; dump()'s exact type_error.316 messages and byte-index reporting are unchanged (see the added regression-guard test in unit-serialization.cpp). Claude-Session: https://claude.ai/code/session_01N4RQ1Ahan5YAGbnAQGjZTY Signed-off-by: Niels Lohmann <mail@nlohmann.me> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> * ⚡ validate only newly read bytes of binary-format strings get_string() validated the whole result after each call, but get_bytes() appends to it and CBOR indefinite-length strings collect all chunks in the same result, so every chunk re-validated everything read before it. An input of many small chunks took quadratic time (80000 one-byte chunks, 160 KB of input, took about 7 seconds). Only the newly read bytes are validated now, which also matches RFC 8949's requirement that every chunk is valid UTF-8 on its own. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
Niels Lohmann
|
632a5812a8 |
Support zero-member types in NLOHMANN_DEFINE_TYPE_* macros (#4041) (#5272)
* Support zero-member types in NLOHMANN_DEFINE_TYPE_* macros (#4041) NLOHMANN_DEFINE_TYPE_INTRUSIVE(Type) and its 11 sibling macros produced broken code for types with no members to serialize. Invoking a variadic macro so __VA_ARGS__ is empty is only standard-conforming since C++20, so a plain __VA_OPT__ fix (as tried in #5142) breaks every pre-C++20 build under -pedantic. Instead, make all 12 macros purely variadic and dispatch on argument count using a sentinel-padded extension of the existing NLOHMANN_JSON_GET_MACRO idiom, giving full C++11-C++26 support with no feature-test gate. Verified against real GCC 16 and Clang at -std=c++11/14/17/20 with -pedantic -Werror -Wvariadic-macros: zero regressions in the existing unit-udt_macro.cpp suite plus 12 new zero-member test cases. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix CI failures in zero-member NLOHMANN_DEFINE_TYPE_* macros Three issues surfaced on PR #5272's real CI that weren't caught by local testing against a narrower flag set: - GCC -Werror=noexcept: the four truly-empty from_json bodies (plain INTRUSIVE/NON_INTRUSIVE, with and without _WITH_DEFAULT) provably never throw but weren't declared noexcept; mark them noexcept explicitly. to_json and the derived-type from_json overloads are left alone since they genuinely can throw (object assignment / delegating to the base class's from_json). - clang-tidy bugprone-macro-parentheses: false positive on the same 8 zero-member bodies (Type/BaseType used purely as declarator types); suppressed with NOLINTNEXTLINE comments in the same style already used elsewhere in this file (see NLOHMANN_JSON_SERIALIZE_ENUM). - MSVC's traditional preprocessor doesn't fully expand NLOHMANN_JSON_CAT(prefix, NLOHMANN_JSON_TYPE_TAG(...))(...) in one pass, which broke a pre-existing one-member usage in unit-regression2.cpp with syntax errors. Wrap all 12 public dispatcher macros in an extra outer NLOHMANN_JSON_EXPAND(...), matching the pattern NLOHMANN_JSON_PASTE already uses for the same MSVC quirk. Re-verified against real GCC 16 and Clang at -std=c++11/14/17/20 with -pedantic -Werror -Wvariadic-macros -Wnoexcept, including the exact files that failed in CI (unit-udt_macro.cpp, unit-regression2.cpp), against both the modular headers and the re-amalgamated single header. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix clang-tidy misc-const-correctness in unit-udt_macro.cpp The four zero-member ONLY_SERIALIZE test objects are only ever read (via to_json), never mutated, so mark them const per clang-tidy. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix derived-type macro dispatch capping members at 62 instead of 63 NLOHMANN_JSON_GET_MACRO resolves 64 positional arguments, with NAME at position 65. NLOHMANN_JSON_TYPE_TAG dispatches on Type plus the member list, so it resolves correctly up to the 63 members NLOHMANN_JSON_PASTE supports. NLOHMANN_JSON_DERIVED_TYPE_TAG dispatched on the two-token Type,BaseType prefix plus the member list, running out one slot early: at 63 members, position 65 landed on the last member name instead of a sentinel and NLOHMANN_JSON_CAT built an undefined identifier such as NLOHMANN_JSON_DEFINE_DERIVED_TYPE_INTRUSIVE_m63, with the compiler reporting "unknown type name 'm1'" once per member and nothing pointing at an argument-count limit. That silently reduced all six NLOHMANN_DEFINE_DERIVED_TYPE_* macros from 63 members to 62, contradicting the "up to 63 members" contract in docs/mkdocs/docs/api/macros/nlohmann_define_derived_type.md. Drop the leading Type and defer to NLOHMANN_JSON_TYPE_TAG so the tag is computed from BaseType plus the member list, which fits the available slots. The zero-own-member derived bodies are therefore selected by tag 1 rather than 2, and the sentinel table for the derived tag is no longer needed. Add a regression test at the documented maximum for both the plain and the derived macros; it fails to compile against the previous dispatch. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Name the zero-member macro bodies by intent, not argument count The dispatch tag was the literal token 1 or N, pasted onto a macro prefix to select the zero-member or member-carrying body. For the derived-type macros that reads wrong: their tag is computed after dropping the leading Type, so the zero-member body was named _1 while taking two parameters (Type, BaseType). Emit EMPTY and MEMBERS instead. The mechanism is unchanged -- the tag is still a token pasted onto the prefix by NLOHMANN_JSON_CAT -- but the body names now say what they are rather than encoding an argument count that only lines up for half of the macros. Collapse the four duplicated zero-member bodies while here: with no members there is nothing to default, so each _WITH_DEFAULT_EMPTY body was a byte-for-byte copy of its plain counterpart. They are now one-line aliases, leaving a single definition of what an empty object serializes to per intrusive/non-intrusive and base/derived combination. No functional change: for both zero-member and member-carrying types the preprocessed to_json/from_json output is token-for-token identical, and the arity limits are unchanged (63 members, base and derived). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Document zero-member support in the macro API reference docs/mkdocs/docs/features/arbitrary_types.md already gained a note, but the three api/macros pages are where the parameter contract is actually specified and they still described member as a non-empty list. State that the list may be empty on each page, and add a note showing what the zero-member case generates: an empty JSON object for the plain macros, and base-type-only serialization for the derived ones. Both notes record that the WITH_NAMES variants do not support this. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Keep user macros named EMPTY or MEMBERS out of the member-count dispatch The dispatch produced the bare token EMPTY or MEMBERS and pasted it onto the macro prefix afterwards. In between, the token was rescanned, so a user macro with either name replaced it: with `#define MEMBERS x` in scope, even NLOHMANN_DEFINE_TYPE_INTRUSIVE(A, member) -- which compiled before -- expanded to garbage, and `#define EMPTY` broke the zero-member form. Paste the suffix onto the prefix directly in the GET_MACRO slot table instead. Operands of ## are not macro-expanded, so the selected body name is formed before any user macro can interfere. NLOHMANN_JSON_TYPE_TAG and NLOHMANN_JSON_DERIVED_TYPE_TAG become NLOHMANN_JSON_TYPE_BODY and NLOHMANN_JSON_DERIVED_TYPE_BODY, taking the prefix as their first argument; NLOHMANN_JSON_CAT is no longer needed. The body macro names are unchanged, and so is the generated code. Add a regression test that defines EMPTY and MEMBERS around plain and derived types, with and without members. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Test for EMPTY and MEMBERS so -Wunused-macros accepts them Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
abbe52d6de |
Add JSON_PRECISE_STREAM_POSITION to leave the character that terminates a number in the stream (#5344)
* docs: qualify the operator>> stream positioning guarantee operator>>'s notes state that it leaves the stream positioned right after the parsed value, so that concatenated JSON values can be read back to back. That does not hold when the value is a number: a number is only terminated by the character that follows it, and the lexer's unget() is simulated (it rewinds only the lexer's own bookkeeping), so that character stays consumed from the stream. Document the actual behaviour: the guarantee holds for all value types except numbers, which must be followed by whitespace. Also qualify the cross-reference on the JSON Lines page, which repeated the unqualified claim. Documentation only; the behaviour itself is tracked in #5340. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * fix: restore the character that terminates a number (#5340) operator>> is documented to leave the stream positioned right after the parsed value, so that concatenated JSON values can be read back to back. That did not hold for numbers: a number is only terminated by the character following it, and lexer::scan_number() reads that character and calls unget() -- which is simulated and rewinds only the lexer's own bookkeeping. input_stream_adapter consumes via sbumpc() with no matching sungetc(), so the terminating character stayed consumed and the next extraction started one byte too late ('1true' left the stream at 'rue'). Propagating unget() to the adapter directly does not work: next_unget makes the following get() replay the cached character, so the terminator would be delivered twice. Instead, restore the still-pending character once at the end of a non-strict parse, where the input is handed back to the caller: - input_stream_adapter gains unget_character() (sungetc()) and advertises it via supports_unget, detected the same way as supports_seek. - lexer::restore_pending_unget() turns a pending simulated unget of a real (non-EOF) character into a real one and clears next_unget so the character is not also replayed. It is a no-op for adapters that cannot unget, and reports failure when sungetc() fails, in which case the input is left as it was before. - parser calls it on the three non-strict paths, i.e. for operator>> and sax_parse(strict = false). Strict parse()/accept() are unaffected: they require the input to end after the value, so the character is consumed by the end-of-input check anyway. Parse error messages and reported positions are unchanged. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * tests: fix CI failures in the #5340 test helpers Four CI failures, all in the new test code: - GCC (-Werror=useless-cast): drop the `json(...)` wrapper around `json::parse(...)`, which already returns a `json`. - GCC (-Werror=unused-result): assign the discarded `json::parse()` result to a dummy, the idiom used elsewhere in the test suite, and catch `json::parse_error&` for consistency. - clang-tidy (google-default-arguments): remove the default argument from the `pbackfail()` override; `sungetc()` supplies the base declaration's default. - MSVC (bad allocation): `no_putback_streambuf::underflow()` set a one-character get area without advancing `m_pos`, so an implementation whose `istream::get` peeks before it bumps re-read the same character forever. Keep no get area at all: `underflow()` peeks, `uflow()` consumes, and `sungetc()` still always lands in `pbackfail()`, which is what the test needs. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * fix: leave the character that terminates a number in the input Read the character following a number without consuming it, instead of consuming it and putting it back. input_stream_adapter now peeks with sgetc() and only steps over the character when the next one is requested or when the adapter is destroyed, so releasing it cannot fail - no putback position is required from the streambuf. Suggested by gregmarr in #5344. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: match the version history wording to the peek-based fix Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: drop the whitespace-separator caveat from the parsing pages The caveat added in #5343 describes the behavior this branch fixes: a number no longer consumes the character that terminates it, so concatenated values need no separator. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * refactor: split the strict and non-strict paths in parser Folding the release_lookahead() call into the existing strict check left the "in strict mode" comment on an else-if branch, and made the strict condition in sax_parse() redundant with the branch it followed. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Put the stream position fix behind JSON_PRECISE_STREAM_POSITION Leaving the character that terminates a number in the stream is observable: reading "1,2,3" with repeated operator>> works today only because the comma after each number is swallowed, and std::getline after a number skips the line break. Both break with the fix, so make it opt-in for 3.x, as suggested by @gregmarr in the review. - JSON_PRECISE_STREAM_POSITION (default 0) selects the peek-based input_stream_adapter. Without it, the adapter is the consuming one from develop and has no supports_lookahead, so lexer::release_lookahead() and the parser's calls to it compile to nothing. - The macro changes input_stream_adapter's layout and member functions, so it gets the ABI tag _psp, after _bics. The ABI config tests, the natvis generator, and nlohmann_json.natvis (regenerated) know the tag. - The tests for the fix move to unit-precise-stream-position.cpp, which defines the macro itself and runs in every build, and gain the two cases above. unit-deserialization.cpp pins the default behavior instead. - The docs describe the default behavior again and point to the new macro page; version history says "added in 3.13.0, planned default in 4.0.0". Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
7c90ec2323 |
Hash deeply nested values without recursing per nesting level (#5546)
* Hash deeply nested values without recursing per nesting level std::hash<basic_json> hashed an array or object by hashing each element, which called detail::hash again once per nesting level. A value nested deeply enough - 50,000 levels of objects on an 8 MiB stack - exhausted the call stack and terminated the process. parse() accepts such values without complaint, since the parser is iterative, and a parsed value is hashed wherever it is used as a key in an unordered container. Bound the descent the same way dump() does: detail::hash takes the nesting level, and once hash_depth_limit() (128) levels have been entered, hash_iteratively() hashes what is left on an explicit stack. It combines the seeds in exactly the same order, so hash values are unchanged. A value nested less deeply than the bound is hashed by the same code as before, without allocating, and is as fast as before. Tests check that every depth up to twice the bound hashes exactly like the recursive definition of the hash, and that values nested 100,000 levels deep hash without crashing. Fixes #5545 for std::hash. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Declare hash_frame's constructor noexcept GCC's -Wnoexcept (an error in CI) flags the emplace_back() into the hash stack under C++26: the constructor cannot throw, since cbegin() is noexcept, but it did not say so. dump_frame's constructor is noexcept for the same reason. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Share one recursion depth limit, and copy the hash frame out of the stack dump() and hash() each defined their own limit on how many nesting levels they recurse into, and the operations still to come would have added more, free to diverge over time. They now all use detail::recursion_depth_limit(), in a header of its own; serializer::dump_depth_limit() and hash_depth_limit() are gone. hash_iteratively() now copies the frame it works on out of the stack and changes the frame only through stack.back(), so nothing can refer into the stack after entering an element has grown it. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Parenthesize multiplications in the hash test for clang-tidy Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
2e91641de2 |
Test JSON_BRACE_INIT_COPY_SEMANTICS for real, and fix one-element tuples under it (#5544)
* Test JSON_BRACE_INIT_COPY_SEMANTICS for real, and fix one-element tuples under it
The opt-in JSON_BRACE_INIT_COPY_SEMANTICS was never exercised by CI:
- Its only test, in unit-regression3.cpp, was guarded by
`#if defined(JSON_BRACE_INIT_COPY_SEMANTICS)` after the #include. The
header #undefs the macro unconditionally in macro_unscope.hpp, so the
guard was always false and the test compiled to nothing, whatever -D
flag was passed.
- The ci_test_brace_init_copy_semantics target that passes the flag was
not named by any workflow.
Move the test into its own translation unit that defines the macro before
including the header, as unit-diagnostics.cpp does for JSON_DIAGNOSTICS.
It now runs in every CI job and for every standard. Remove the unused
target: it ran the whole suite with the macro, and that suite deliberately
relies on default brace-init semantics in about 90 places
(e.g. `json({1})` meaning `[1]`), so it could never pass.
Running the whole suite with the macro did find one library bug:
to_json for std::tuple builds `j = { std::get<Idx>(t)... }`, so with copy
semantics a one-element tuple became its element. `json(std::tuple<int>{5})`
was `5` instead of `[5]`, and `get<std::tuple<int>>()` threw type_error.302
on the result. Under the macro, a one-element tuple now builds exactly what
the default deduction builds. Without the macro nothing changes.
The new tests also pin that the library's other conversions produce the
same values with and without the macro. The macro page now says that the
macro affects every single-element list (`json j = {1}` is `1`), and that
all translation units must agree on it, since it has no ABI tag.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Make JSON_BRACE_INIT_COPY_SEMANTICS part of the ABI tag
The macro changes the body of the initializer-list constructor and adds a
to_json_tuple_impl overload, both with the same mangled names in either
mode, so mixing translation units silently picked one definition. Encode
it in the inline namespace as `_bics`, as JSON_USE_LEGACY_DISCARDED_VALUE_COMPARISON
does with `_ldvcmp`. The macro is new in the unreleased 3.13.0, so no
existing namespace name changes.
- Move the macro's default into abi_macros.hpp so json_fwd.hpp computes
the same namespace, and keep it defined under JSON_TEST_KEEP_MACROS.
- Check the tag in the ABI config tests and in the unit test.
- List `_bics` (and the missing `_dp`) in the namespace docs and in the
natvis generator; regenerate nlohmann_json.natvis.
- Replace the "define it consistently" warning with an ABI note.
Suggested by @gregmarr in the review of #5544.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix the cppcheck, clang-tidy and legacy-comparison CI failures
- to_json_tuple_impl() moved the element in both branches of a ternary;
only one runs, but cppcheck reported accessMoved. Use if/else.
- The ABI tag test looked for "json_abi_bics", which misses when another
tag comes first, as in json_abi_ldvcmp_bics; look for "_bics".
- readability-qualified-auto in the items() test.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
|
||
|
Niels Lohmann
|
8699de3064 |
Stop allocating the BJData excluded-marker list per container (#5555)
write_ubjson() built a std::vector of the eight markers BJData forbids as the type of an optimized container - one heap allocation plus a linear search for every array and object it wrote with use_type, even for plain UBJSON output, where the list isn't consulted. The list was also spelled out twice. A constexpr helper, is_bjdata_excluded_type_marker(), replaces both. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
918da64657 |
Keep BJData ndarray annotations that would not survive a round trip as objects (#5542)
write_bjdata_ndarray() encoded a JData-annotated object as a BJData
ND-array whenever its dimensions' product matched _ArrayData_.size(),
which lost information in two ways:
- _ArrayData_ was never required to be an array. null has size 0, any
other scalar has size 1, and iterating an object visits its values, so
e.g. {"_ArraySize_":[1],"_ArrayData_":5} was written as the array [5],
and an object _ArrayData_ came back as an array.
- The reader only restores an annotated object from an ND-array with at
least two non-zero dimensions that is not a 1xN row vector; an empty,
1-D, row-vector, or zero-sized shape is read back as a plain array. The
writer nonetheless emitted ND-array headers for these shapes, so the
annotation was silently dropped.
OSS-Fuzz issue 563659413 hit this in parse_bjdata_fuzzer: an empty binary
_ArraySize_ is written as a plain object and read back as an empty array,
after which {"_ArrayType_":"int16","_ArraySize_":[],"_ArrayData_":null}
was encoded as the ND-array header "[$I#[]" and re-read as [], failing the
harness's value-stability check.
Such objects now fall back to a plain object encoding, which round-trips.
Genuine ND-arrays (two or more positive dimensions, not a 1xN row vector)
are encoded exactly as before. Existing fallback tests that used 1-D
shapes are moved to 2-D shapes so they keep exercising the check they
were written for, and the BJData documentation is updated.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
|
||
|
Niels LohmannandClaude Sonnet 5
|
56b3ee566c |
Fix stack overflow when copying a deeply nested value (#5387) (#5389)
* Bound the descent of the copy constructor basic_json's copy constructor copied objects and arrays by handing the container to its own copy constructor, which copy-constructs every element and so reaches this constructor again, once per nesting level. A value nested deeply enough exhausted the call stack and terminated the process with a segmentation fault - no exception, nothing the caller could catch. Parsing such a value works, as the parser is iterative, and so does destroying one, as #1436 made destruction iterative. Bound how far the copy descends rather than take the call stack away from it. The first levels are copied exactly as they were - the containers copy their own elements, which is by far the fastest way to fill them - and only once the copy has descended 128 levels is the value below it finished without the call stack, through an explicit worklist. Copying can therefore no longer exhaust the stack, however deeply a value is nested, while a value nested less deeply than the bound - all but a vanishing minority - is copied by the very same code as before and pays only for one counter. That counter lives in thread_local storage, as one shared between threads would be raced. JSON_NO_THREAD_LOCAL switches it off for toolchains without thread_local; copying then goes through the worklist right away, which yields the same values but is measurably slower. The deferred values are completed before the copy they belong to returns, so a value copied while another copy is going on - by a custom base class, say - is unaffected by the copy it is nested in. operator= takes its argument by value, so copy assignment is fixed as well. Copying is as fast as it was, within measurement noise (medians of 9 interleaved runs, clang -O3): -1.3% for an array of strings, +0.0% for a flat object, +0.1% for a flat array of numbers, +0.3% for nested arrays, +0.6% for nested objects and +1.2% for a twitter-like document. Copying a three-key object costs about ten nanoseconds more, the counter. Deferring every level instead, rather than only those below the bound, measured between 3% and 9% slower depending on the shape of the value. This fixes #5387 for the copy constructor. dump() is still recursive. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Test the copy constructor's iterative path in CI The copy constructor descends into 128 levels before it finishes a value without the call stack, so the iterative path is otherwise only reached by the few tests that nest deeper than that. JSON_NO_THREAD_LOCAL switches the descent off, which sends every value down that path. Running the whole test suite that way covers it with every object type, string type, allocator, and base class the suite already exercises. The new ci_test_no_thread_local target does that; the macro had no build coverage at all before. Copying a nested value also has to carry over what the element-wise copy constructor would have copied: the parents that JSON_DIAGNOSTICS relies on, and the positions that JSON_DIAGNOSTIC_POSITIONS reports. Both are now checked on either side of the descent bound, for objects and arrays. Neither was tested before, and dropping either one makes the new tests fail. Also quantify what JSON_NO_THREAD_LOCAL costs a copy instead of calling it "measurably slower". Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Split the regression tests so that they keep linking Linking test-regression2 fails with "relocation truncated to fit: IMAGE_REL_AMD64_REL32 against `.rdata'" once its object grows past what the MinGW linker copes with, and the copy constructor's helpers push it over: the object grows by 6.3%, from 4,654,128 to 4,944,920 bytes at -O0, and develop links at the smaller of the two. Building the tests optimized shrinks the object enough to link, but the binaries clang 11.0.1 and clang 18.1.8 then produce crash before doctest prints its first line - 39 of 102 tests on clang 18 - so the objects have to become smaller rather than denser. Moving the test cases that follow "regression tests 2" into a file of their own brings that object to 4,687,888 bytes, which is 0.7% above the size that links today rather than 6.3%. Both files still build for C++11, C++17 and C++20, and run the same 9 test cases and 135 assertions as before, now spread over two binaries. New regression tests belong in unit-regression3.cpp from here on, which is what CONTRIBUTING.md now says. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Do not use thread_local storage with Clang targeting MinGW Every test that copies a value segfaults there - 42 of 105 on clang 11.0.1, 39 of 102 on clang 18.1.8 - while the same tests pass with GCC targeting MinGW, with Clang targeting MSVC, and with every other toolchain the library is tested on. The counter that bounds the copy constructor's descent is the library's first use of thread_local, so that job had never exercised it before. JSON_NO_THREAD_LOCAL already covers toolchains without thread_local storage, and copying yields the same values with it, only more slowly. Define it for this one automatically. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Balance the warning suppression the split separated unit-regression2.cpp opens a DOCTEST_CLANG_SUPPRESS_WARNING_PUSH block at the top and closed it at the very bottom, which the split moved into unit-regression3.cpp: one file was left with a push and no pop, the other with a pop and no push, which clang reports as an error. Give each file the pair it needs. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Check both shapes without a C-style array clang-tidy rejects the array the two shapes were iterated over (cppcoreguidelines-avoid-c-arrays). The array only existed because astyle reformats a range-for over a braced initializer list into something unreadable; naming the two cases avoids both. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Split the regression tests far enough to leave room The first split left unit-regression2.cpp 0.7% below the size develop links at, which the comparison change in the follow-up immediately used up: the MinGW linker fails on test-regression2_cpp20 again, naming copy_shallow and to_partial_ordering among the relocations it cannot fit. Move the sections from "issue #2067" on, and the helper types they use, so that the file stops being the one that decides whether the tests can be linked at all. At -O0 and C++20, unit-regression2.cpp is now 2,964,944 bytes against develop's 4,708,248, and 3,070,568 bytes with the follow-up applied - roughly a third smaller either way, rather than a fraction of a percent larger. The 135 assertions are the same ones as before, now spread over three test cases in two files. Also silence the clang-tidy findings the deep-nesting tests draw: the copies they make are what is being tested, and the reserve() computation gets its parentheses. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Move the #4804 alias to the file that uses it The split left the json_4804 alias behind in unit-regression2.cpp while the test case that uses it went to unit-regression3.cpp, which does not build for C++17 and C++20 as a result. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Include <span> where the split moved its only use The #2546 test case guards itself with __has_include(<span>), but the include itself sat in unit-regression2.cpp's preamble and stayed behind, so the section compiled without a declaration wherever the guard passed - which nvhpc reported and libc++ builds do not, as they skip the section altogether. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Keep the descent bookkeeping in one place Copying carried a depth count, a depth limit and a guard of its own, and the comparison in the follow-up added a second set beside them. Neither operation needs its own: they are never nested inside one another by the library - copying a value does not compare one, and comparing two values does not copy them - and where user code nests them anyway, sharing the count only ends a descent sooner than it had to. So there is now one nesting_depth(), one nesting_depth_limit() and one nesting_depth_guard, which the follow-up uses instead of adding its own. Inverting the test in copy_structured leaves the too-deep case and the no-thread-local case as the same code. The guard takes the count rather than looking it up, because the caller has looked it up already to test it against the limit, and reaching thread-local storage twice on the path that is taken almost every time is worth avoiding. The switch that copies the value of anything that is not an object or an array was written twice - once in the copy constructor, once in copy_shallow - so that adding a value_t meant editing both, and missing one would have been silent. It is copy_leaf_value now, and inlined: both callers have already sorted the containers out, and folding that test into the switch is what keeps a value made mostly of numbers copying as fast as it did. Copying canada.json, citm_catalog.json and twitter.json is within 0.6% of what it was before, measured as a paired ratio over 18 interleaved rounds against a run-to-run spread of 0.3%. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Check that an abandoned copy can still be destroyed Copying a value without the call stack builds the copy from the top down, and every value whose own copy has not been made yet stays a null value until it is. That is what lets a copy be abandoned half-built: the destructor finds nothing but complete values and null ones. Nothing tested it. Failing an allocation part-way through a copy of a deeply nested value does, with the allocator the file already has for exactly this kind of test. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Name the test's locals so Flawfinder stops matching them The code scanning job reports CWE-362 - "check when opening files" - for a test that opens no files: Flawfinder matched a local variable called open. Rename it and its partner. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Keep the descent guard's bookkeeping self-contained nesting_depth_limit() and nesting_depth_guard were only used inside the JSON_NO_THREAD_LOCAL-guarded branch of copy_structured(), but were defined unconditionally. Move them inside the #ifndef, and have the guard look up the depth and test it against the limit itself (via okay()) instead of making the caller do it - the caller no longer needs to touch nesting_depth() at all. Also shrink the thread-local counter to std::uint8_t, matching what its own doc comment already argued. Addresses gregmarr's review comments on #5389. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Make nesting_depth_guard usable regardless of JSON_NO_THREAD_LOCAL nesting_depth_limit() and nesting_depth() stay behind #ifndef JSON_NO_THREAD_LOCAL, since a descent cannot be bounded without a per-thread count. But the guard itself now always exists, becoming a no-op that is never okay() under that macro - the same way the bound is already reached on every call without one. copy_structured() no longer needs to know which case it is in. This is what lets #5390 reuse the guard for comparison, which cannot test JSON_NO_THREAD_LOCAL where the macro-based operators use it: the guard now carries that distinction itself instead of requiring every caller to. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Silence VS2015's C4503 for the custom-base-class test The deep-copy support added for #5387 lengthened the mangled name of std::allocator_traits<...>::construct for the test's map type past VS2015's limit, which /WX turns into a build failure even though the name is only used for (now-truncated) debug info. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Remove dead unused-parameter casts from copy_metadata() @gregmarr asked whether the static_cast<void> pair in the JSON_DIAGNOSTIC_POSITIONS-off branch was needed for an empty json_base_class_t. It isn't: src and dst are already referenced unconditionally by the base-class copy above, so no -Wunused-parameter warning fires either way (checked with -Wall -Wextra -Wunused-parameter, JSON_DIAGNOSTIC_POSITIONS 0 and 1). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix CI: build custom array types without a fill constructor, re-amalgamate copy_array_level() built the destination array with the fill constructor array_t(count, value), which is not part of the array container interface the library otherwise assumes (e.g. custom ArrayTypes that only provide a default and an iterator-pair constructor, as covered by unit-custom-array-type.cpp). Default- construct the array and resize() it instead, matching how the rest of the codebase already grows array_t. Also re-run the amalgamation, which had fallen out of sync with include/nlohmann/json.hpp. Signed-off-by: Niels Lohmann <mail@nlohmann.me> Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
Niels LohmannandClaude Sonnet 5
|
ed513715a8 |
Document that a NUL byte in the input is treated as end of input (#5534)
* docs: document that a NUL byte in the input is treated as end of input A NUL byte anywhere in the input - trailing, or embedded ahead of more otherwise well-formed JSON - is currently treated the same as genuine end of input, so parsing silently stops there instead of raising the parse_error.101 any other unexpected byte triggers. This mirrors the NUL-terminated-C-string convention already used when no explicit input length is given (json::parse(const char*) already stops at strlen()), just applied uniformly rather than only when a length is genuinely unavailable. This behavior predates this change and is not being altered here - changing it would be an observable, backwards-incompatible behavior change for any caller that (knowingly or not) depends on it, which is not something to do silently in a patch. Documenting the current, verified behavior as a new FAQ entry instead, so it's an intentional and discoverable part of the contract rather than a surprise. Fixes #5530. Signed-off-by: Niels Lohmann <niels.lohmann@gmail.com> Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N4RQ1Ahan5YAGbnAQGjZTY * Add JSON_STRICT_NUL_HANDLING opt-in macro for issue #5530 A NUL byte anywhere in the input is currently treated the same as real end of input, rather than raising parse_error.101 like any other unexpected byte (documented in the previous commit's FAQ entry). A full unconditional fix was tried in PR #5532 but rejected as too risky to ship by default: any caller could depend on the current behavior, even unknowingly (e.g. a zero-padded buffer). On PR #5534, gregmarr proposed a compile-time opt-in flag instead, and the maintainer agreed, wanting it available now and defaulting to the corrected behavior in 4.0.0. This mirrors the existing JSON_BRACE_INIT_COPY_SEMANTICS precedent as closely as sensible: - JSON_STRICT_NUL_HANDLING defaults to 0 (off); the three lexer sites that treat '\0' as EOF/comment-terminator are gated with `#if !JSON_STRICT_NUL_HANDLING` so the default-off behavior is byte-for-byte identical to today's. - input_adapters.hpp's `T (&array)[N]` overload additionally trims a single trailing '\0' from a `char` array (e.g. a string literal like `json::parse("123")`) when the macro is on, so that case keeps working; every other element type (unsigned char, std::uint8_t, ...) always keeps its full extent. This intentionally does *not* reuse the existing strlen()-based pointer overload via SFINAE-excluding `char` from the array overload, as originally sketched for this change: that approach is ambiguous against the newer generic container overload added since PR #5532, and even where it compiles, strlen()-scanning a `char` array that is not NUL-terminated within its bounds reads past the end of the array (confirmed with AddressSanitizer). Trimming only a single trailing byte, without scanning, avoids both problems. - Documented via docs/mkdocs/docs/api/macros/json_strict_nul_handling.md, linked from the macros index/nav/features page, the FAQ entry, and the parse/accept/operator>> reference pages. - Tested in unit-class_parser.cpp and unit-deserialization.cpp, default state unguarded and opt-in state guarded. Since the library itself #undefs the macro at the end of json.hpp (as JSON_BRACE_INIT_COPY_SEMANTICS already does), a plain `#if defined(JSON_STRICT_NUL_HANDLING)` guard after the include never actually triggers; the tests instead capture the command-line value into a test-local macro before including the header. A few pre-existing fixtures elsewhere (std::array<uint8_t, N> sized one larger than their literal, relying on value-initialization to silently add a trailing zero byte) needed the same one-byte adjustment to keep passing under the opt-in behavior. Unlike the precedent, this adds a proper `JSON_StrictNulHandling` CMake option (rather than a raw -DCMAKE_CXX_FLAGS injection) and wires its ci_test_strict_nul_handling target into the ci_cmake_options job matrix in .github/workflows/ubuntu.yml, so the opt-in build is actually exercised in CI -- closing the one gap in the precedent's own CI setup (ci_test_brace_init_copy_semantics is defined but never referenced by any workflow, so it has never actually run). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Clarify where JSON_STRICT_NUL_HANDLING does not reject NUL bytes Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <niels.lohmann@gmail.com> Signed-off-by: Niels Lohmann <mail@nlohmann.me> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
Niels LohmannandClaude
|
1054b2097e |
Speed up binary writing: value-type output sink + byte-swap number encoding (#5286)
* Devirtualize binary_writer via a value-type output sink to_cbor/to_msgpack/to_ubjson/to_bjdata/to_bson wrote every byte through output_adapter_t, a shared_ptr<output_adapter_protocol> whose write_character/write_characters are virtual. Unlike the lexer (templated on a concrete InputAdapterType), the binary writer never got that treatment, so binary output paid a vtable lookup per byte and a make_shared per call. Template binary_writer on an OutputSinkType and give it two concrete, non-virtual sinks: - output_vector_sink: appends straight into a std::vector (push_back / insert), used by the vector-returning to_* convenience functions. No vtable, no shared_ptr; the writes inline. - output_adapter_sink: forwards to a type-erased output_adapter_t, so the existing to_*(j, output_adapter) overloads (streams, strings, custom adapters) keep working exactly as before -- one virtual call each, unchanged. binary_writer keeps a convenience constructor taking output_adapter_t (building the default output_adapter_sink), so the adapter overloads are untouched; only the convenience functions switch to the vector sink. The friend declaration and the basic_json binary_writer alias gain the new (defaulted) template parameter. Output is byte-for-byte identical: verified across ~3000 randomized values plus curated edge cases (all scalar widths, strings with invalid UTF-8, binary, nested arrays/objects) for CBOR, MessagePack, UBJSON (both size/type settings), BJData, and BSON, plus the output_adapter path, in C++11/17/20. Warning-clean under clang -Weverything and the gcc pedantic set; clang-tidy clean on the changed headers; make check-amalgamation clean. Throughput (g++ -O3, vs develop): scalar-dense binary output such as integer arrays ~1.4x; many small to_cbor calls ~1.04x (DOM traversal bound); string/blob-heavy output unchanged (already bulk-bound). No workload regressed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix CI failures from binary_writer output-sink change Four CI jobs failed on the initial commit; all are addressed here without changing any output (binary encodings remain byte-for-byte identical to develop across the differential corpus): 1. ci_test_gcc / cuda (-Werror=duplicated-branches): for number_float_t == float, static_cast<float>(n) is the identity, so write_compact_float's two branches are intentionally identical. Once the concrete vector sink is inlined, GCC constant-folds and diagnoses this (the type-erased path hid it behind a non-inlined virtual call). Silence -Wduplicated-branches for GCC (clang has no such warning) alongside the existing -Wfloat-equal pragma. 2. ci_static_analysis_clang (UBSan nonnull-attribute): binary_writer passes a null pointer with length 0 for empty strings/binary. output_vector_sink / output_adapter_sink declared write_characters JSON_HEDLEY_NON_NULL, so the sanitizer flagged the (harmless) zero-length call once the sink was called directly rather than through the attribute-free virtual base. Drop the attribute from both sinks, matching the pre-existing behavior. 3. ci_cpplint (build/include_what_you_use): output_adapter_sink uses std::move; add #include <utility>. 4. ci_cuda_example (nvcc 11.8): NVCC's front end rejects the default template argument on the binary_writer alias template. Revert the alias to its original single-parameter form (relying on binary_writer's own defaulted OutputSinkType) and spell out the full type in the vector-sink convenience functions. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Encode big-endian numbers with a byte swap instead of std::reverse write_number() reordered multi-byte numbers for the big-endian formats (CBOR/MessagePack/UBJSON) with std::reverse over the byte array. GCC lowered only some sizes to a bswap; clang kept a scalar byte shuffle (0 bswap instructions in the CBOR number path). Replace the reverse with size-dispatched __builtin_bswap16/32/64 helpers (portable shift fallback for other compilers; std::reverse retained for exotic sizes such as a long double number_float_t). Codegen: the CBOR number path now emits bswap on both compilers (gcc 2 -> 16, clang 0 -> 4). Output is byte-for-byte identical to the previous implementation across the binary differential corpus. Throughput (isolated vs the std::reverse version, best of 9): CBOR int64 array gcc +7% clang +10% CBOR uint16 array gcc +27% clang flat Modest but consistent on number-dense encodings; negligible on string/blob-heavy output, as expected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Reserve output capacity up front for binary serialization The vector-returning to_cbor/to_msgpack/to_ubjson/to_bjdata/to_bson grew the output buffer purely by geometric reallocation. Reserving an estimate up front avoids the early reallocations, which is the dominant per-byte cost for array/object-heavy output. The estimate (binary_reserve_hint) is deliberately conservative and safe against untrusted input: it consults only the top-level element count (O(1), no walk of the DOM), guards the multiplication against overflow, and clamps the result to a fixed 1 MiB ceiling, so a large or hostile DOM can never force an oversized allocation here. The buffer still grows geometrically past the hint, so an underestimate only costs a few later reallocations; scalars/strings/binary are written in one shot and get no hint. Reserving capacity does not change the bytes produced. Throughput (g++/clang -O3, vs the previous commit): cbor int array +10% / +13% cbor object array +20% / +38% Output is byte-for-byte identical to develop across the binary differential corpus. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address review findings on the binary writer output sinks - binary_reserve_hint(): the 4-bytes-per-element estimate over-reserved by up to 4x for arrays of small scalars (CBOR encodes 0..23 in one byte), and the returned vector kept that capacity. Make the hint a strict lower bound on the encoded size instead, which also removes the 1 MiB clamp whose branch no test could reach (the largest container in the suite has 65793 elements). - Guard the -Wduplicated-branches pragma with __GNUC__ >= 7. The warning does not exist before GCC 7, so naming it made GCC 4.8/4.9/5/6 - which the CI matrix still builds - warn under -Wpragmas on every including translation unit, breaking downstream -Werror builds. - Constrain the adapter constructor of binary_writer with the enable_if its documentation already claimed, so a writer over some other sink type is no longer advertised as constructible from an output adapter. - Let output_vector_adapter wrap output_vector_sink rather than duplicating the append logic, so the type-erased and templated paths share one implementation. - Collapse the three copies of the memcpy/byte_swap/memcpy dance into a single byte_swap_buffer() helper, and add the MSVC _byteswap_* intrinsics so MSVC no longer falls back to the scalar shuffle this change exists to eliminate. - Add a vector_writer() helper for the five vector-returning to_* overloads instead of spelling out the writer type at each call site, and drop a dead default member initializer on output_adapter_sink. - New tests: the vector sink and the adapter sink must produce identical bytes for every format (the two to_* overloads no longer delegate to each other and could otherwise drift), and binary_reserve_hint() must never exceed the size actually written. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Route the -Wduplicated-branches pragma through Hedley Match #5485, which moved the binary writer's hand-rolled diagnostic pragmas onto JSON_HEDLEY_PRAGMA (merged into develop while this branch was open). The devirtualization's -Wduplicated-branches suppression in write_compact_float was the one raw '#pragma GCC diagnostic' left; it now uses JSON_HEDLEY_PRAGMA like the adjacent -Wfloat-equal line, still guarded to GCC >= 7 and non-clang (the warning exists only there). Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Niels Lohmann
|
0b20b7e622 |
Reject MessagePack/BSON binary subtypes that don't fit their wire format (#5469)
* Reject MessagePack/BSON binary subtypes that don't fit their wire format Both formats store byte_container_with_subtype's subtype (a uint64_t) in a single byte. The writers cast to std::int8_t/std::uint8_t without a range check, so subtypes above 255 were silently truncated modulo 256 instead of raising an error. Throw out_of_range.413 instead when the subtype exceeds the representable range of 0-255. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Move the new binary-subtype regression test out of unit-regression2.cpp unit-regression2.cpp is already at the edge of what the MinGW linker can relocate; adding this test's ~26 lines tips test-regression2_cpp20 (clang, Windows) over into "relocation truncated to fit: IMAGE_REL_AMD64_REL32 against `.rdata'" (see |
||
|
Niels Lohmann
|
a2b19d6158 |
Honor allow_exceptions=false for excessive array/object size (out_of_range.408) (#5467)
* Honor allow_exceptions=false for excessive array/object size (out_of_range.408) The SAX DOM parsers' start_object()/start_array() threw out_of_range.408 directly via JSON_THROW when a binary format (CBOR/UBJSON/BJData) declared a container size exceeding max_size(), bypassing the allow_exceptions flag that every other malformed-input error path in these classes honors via parse_error(). This meant that json::from_cbor(data, true, false) etc. could still throw (or abort under JSON_NOEXCEPTION) instead of returning a discarded value, contrary to the allow_exceptions=false contract. Route all four call sites (two in json_sax_dom_parser, two in json_sax_dom_callback_parser) through parse_error() instead, matching the existing error-handling pattern used elsewhere in this file. Behavior is unchanged when allow_exceptions is true (the default); the exception message and type are identical. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Drop a non-portable exact exception message check in the 408 test The allow_exceptions=false regression test checked the exact message text produced when allow_exceptions=true (the default). On platforms where std::size_t is 32-bit (e.g. mingw x86, MSVC Win32 builds), a declared CBOR length of 2^63 is intercepted earlier, by get_cbor_container_size()'s own (pre-existing, already correct) length-narrowing check, with different wording than this fix's start_array()/start_object() size check -- same error code, same "still throws when allow_exceptions=true" guarantee, different text. CHECK_THROWS_AS already verifies the behavior this test cares about (still throws json::out_of_range, unchanged); drop the exact-message assertion since it isn't portable across size_t widths and doesn't add coverage of this fix specifically. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix -Werror=unused-result on json::from_cbor() in the 408 regression test from_cbor() is [[nodiscard]]; CHECK_THROWS_AS() otherwise discards its result, which GCC flags under -Werror. Assign to a throwaway json, as the rest of the suite already does for from_cbor()/from_msgpack(). Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
e485441123 |
Restore a duplicate key's prior value when the callback rejects its new value (#5466)
* Restore a duplicate key's prior value when the callback rejects its new value json_sax_dom_callback_parser::key() unconditionally overwrote the object slot for a key with a `discarded` placeholder as soon as the key was accepted by the parser callback. For a duplicate key (legal JSON), this destroyed the pre-existing value from an earlier occurrence of the same key before the new value was even parsed. If the new value was then rejected by the callback, remove_discarded_value() erased the member entirely instead of leaving the original value in place, contradicting the documented behavior that a discarded value behaves as if it was never read. Add a small stash of (slot pointer, previous value) pairs so that when key() overwrites an existing member with the discarded placeholder, the previous value can be restored later if the corresponding value (scalar, object, or array) is rejected, instead of being erased. The stash entry is dropped without restoring once the new value is definitively accepted (in handle_value() for scalars, end_object()/end_array() for containers), so a duplicate key whose new value is accepted still keeps the last value as before. Non-duplicate keys are unaffected: rejecting their value still removes the member entirely, since there is nothing to restore. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Mark parser-callback test lambdas noexcept to fix GCC -Wnoexcept -Werror GCC's libstdc++ std::function move assignment evaluates a noexcept check that invokes a wrapped callable in an unevaluated context; a non-noexcept parser_callback_t lambda then trips -Wnoexcept ("noexcept- expression evaluates to 'false'"), which CI's ci_test_gcc job builds with -Werror. The pre-existing parser_callback_t test lambdas in this file already work around this by declaring themselves noexcept; apply the same fix to the three added lambdas that didn't. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |