mirror of
https://github.com/nlohmann/json.git
synced 2026-10-01 12:10:32 +00:00
develop
228
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Niels Lohmann
|
e5a89d671f |
Fix lint debt: enum-macro NOLINTs, doctest as SYSTEM, no-op analyzer (#5737)
* Drop stale LCOV_EXCL_LINE from the json_pointer out_of_range.410 throw The comment said the size_type overflow check in array_index() is only triggered on special platforms like 32-bit, and the throw was excluded from coverage. On 64-bit platforms the check is true for SIZE_MAX itself, and unit-json_pointer.cpp has asserted that case four times since #5395, so the line is executed in the coverage job. Reword the comment and remove the exclusion marker so the coverage report notices if the tests stop reaching it. Part of #5725 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Name all three C-array check aliases in the enum-macro NOLINTs NLOHMANN_JSON_SERIALIZE_ENUM(_STRICT) suppressed the c-array warning under modernize-avoid-c-arrays only, but clang-tidy emits the same diagnostic under the aliases cppcoreguidelines-avoid-c-arrays and hicpp-avoid-c-arrays too. Any user running those checks got a false positive at every macro expansion, and our own tests needed a local NOLINT at each call site to work around it. Name all three aliases in the four macro comments instead, and drop the now-redundant c-array names from the five test call-site NOLINTs. Comment-only change; behavior, the public API, and the ABI do not change. Ran make amalgamate. Part of #5725 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Include doctest as a SYSTEM directory instead of disabling warnings for all tests test_main added -Wno-deprecated and -Wno-float-equal as PUBLIC compile options for every non-MSVC compiler, so they were applied to every translation unit, library headers included, and silenced the CI warnings meant to check the library's own -Wfloat-equal pragmas. The only code that actually needed the suppression was the vendored doctest.h, which was included as a normal (non-SYSTEM) directory. Include thirdparty/doctest as SYSTEM for test_main, matching what tests/abi/CMakeLists.txt already does, and drop the two suppressions from both targets. Verified locally that unit-comparison, unit-conversions and unit-constructor1 compile clean with -Werror -Weverything and doctest as -isystem, and that CMake still configures with JSON_BuildTests=ON. Part of #5725 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Remove the no-op ci_clang_analyze target ci_clang_analyze configured the build with the real compiler and only then wrapped ninja with scan-build. scan-build intercepts compiles by overriding CC/CXX, but build.ninja already had the compiler path baked in from the configure step, so every run bypassed the analyzer: CI logs show "No bugs found" after a normal build, never an analysis. The job also used Debian's frozen clang-tools-14 rather than the image's own clang, and CLANG_ANALYZER_CHECKS still named three valist.* checkers that current clang merged into security.VAList. ci_clang_tidy already runs every clang-analyzer-* check (via .clang-tidy's "Checks: '*'") with warnings as errors, so nothing is lost by removing the dead job. Delete ci_clang_analyze, CLANG_ANALYZER_CHECKS and the SCAN_BUILD_TOOL lookup from cmake/ci.cmake, drop it from the ubuntu.yml ci_static_analysis_clang matrix, and drop the now-unused clang-tools apt package (iwyu stays for ci_single_binaries). Reword quality_assurance.md and assurance_case.md, which described the dead job as a working control, to say the Clang Static Analyzer checks run through clang-tidy. Verified that `cmake -DJSON_CI=ON` still configures cleanly and that ci_clang_analyze no longer appears in the generated build or in any CMake/workflow file. Part of #5725 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Re-enable portability-template-virtual-member-function; remove redundant forwards .clang-tidy disabled three checks "to get the CI going" (#4489, 2024-11-13): portability-template-virtual-member-function, bugprone-use-after-move and its alias hicpp-invalid-access-moved. portability-template-virtual-member-function only flagged output_stream_adapter::write_character/write_characters; annotate both with NOLINT and re-enable the check. bugprone-use-after-move flagged several double forwards that have no effect at runtime: - from_json.hpp calls std::forward<BasicJsonType>(j).at(Idx) inside pack expansions; at() has no ref-qualified overloads and always returns an lvalue reference, so the forward is a no-op. Replace with plain j.at(Idx) in all four places. - the move constructor forwards the whole object to its base class and then reads other's members. That is item 9 of #5724 (together with its cppcheck suppressions) and is left to that change. - input_adapters.hpp forwards the container twice on purpose, so the begin/end iterator types match adapter_type; annotate with NOLINT and a comment instead of changing behavior. The check still flags the move constructor (see above) and two sites in at(KeyType&&) (both overloads, json.hpp, in the throw's string_t(std::forward<KeyType>(key)) after find(std::forward<KeyType>(key))). Open PR #5689 rewrites that hunk, so bugprone-use-after-move (and hicpp-invalid-access-moved) stay disabled for now, with a comment explaining why; re-enable them once #5689 and the #5724 move-constructor change have landed. Also resolve the portability-avoid-pragma-once TODO: single_include never has #pragma once (amalgamate.py strips it) and every supported compiler accepts it in include/, so keep it disabled with an explanatory comment instead of a TODO. Fix the stale "json.hpp, around line 1265" comment in unit-class_parser.cpp, which now points at the move constructor's actual line. Behavior, the public API and the ABI do not change. Verified with clang-tidy 22.1.8 that portability-template-virtual-member-function now reports nothing, that bugprone-use-after-move/ hicpp-invalid-access-moved report only the known at(KeyType&&) and move-constructor sites, and that unit-custom-base-class, unit-constructor1, unit-conversions, unit-element_access2, unit-class_parser and unit-diagnostic-positions (JSON_DIAGNOSTIC_POSITIONS=1) compile under ASan/UBSan and pass with the same assertion counts as before. Ran make amalgamate. Part of #5725 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix stale and malformed NOLINT comments json_sax.hpp named "-warnings-as-errors" in the NOLINT list on the two JSON_ASSERT(false) lines; that is the suffix clang-tidy appends to a diagnostic tag under WarningsAsErrors, not a check name, and every other JSON_ASSERT(false) omits it. unit-capacity.cpp carried 30 "// NOLINT(misc-const-correctness)" comments on "json j = ...;" declarations that are all used with non-const members afterwards, so the check has nothing to report there. unit-constructor2.cpp used a blanket "// NOLINT: access after move is OK here" on a use-after-move that hides every check on the line; naming bugprone-use-after-move and hicpp-invalid-access-moved keeps the intent once those checks are re-enabled (#5724). Signed-off-by: Niels Lohmann <mail@nlohmann.me> #5725 item 10 * Remove stale .clang-tidy entries -google-runtime-references disabled a check that neither clang-tidy 22.1.8 nor 23.1.2 lists under --list-checks -checks='*'; it was removed upstream. The commented-out HeaderFilterRegex line has been unused since the active HeaderFilterRegex was introduced in #2561 (2021). Signed-off-by: Niels Lohmann <mail@nlohmann.me> #5725 item 11 * Remove the GCC C++20 -Wignored-attributes pragma in json.hpp The pragma (added in #5164) claimed to work around the C++ modules redefinition errors of #5103, but #5103 is about hard errors (e.g. "redefinition of std::__is_constant_evaluated()", conflicting std::integral_constant) that ignoring a warning cannot suppress; they are traced to GCC PR 124430 and reproduce with <map> or <string> instead of json.hpp too. A GCC 16.2 -std=gnu++20 -fmodules build following #5103's repro steps still fails with the pragma in place, and a build of all test TUs with GCC_CXXFLAGS (which enable -Wignored-attributes) and the pragma removed produces no such warning. The block only hid a warning class from GCC C++20 users while suggesting #5103 was handled. Overlaps #5610, whose hunks touch the closing half of this pragma to insert the json_literals.hpp include. Signed-off-by: Niels Lohmann <mail@nlohmann.me> #5725 item 9 * Fix stale doxygen comments hidden by the -Wdocumentation pragma macro_scope.hpp ignores -Wdocumentation and -Wdocumentation-unknown-command for the whole library, which also hides genuine documentation mistakes: - detail::unescape() documented "@return unescaped string" but returns void and unescapes its argument in place; reworded to "@param[in,out] s string to unescape in place" and dropped the bogus @return. - basic_json::get()'s copy-conversion overload wrote "converted to @tparam ValueType" inside @return, which Doxygen and Clang parse as a second, malformed @tparam; changed to "@a ValueType", matching the two other get() overloads a few lines above that already use it. This narrows the gap the -Wdocumentation pragma needs to cover; fully replacing the Doxygen-only commands it also hides (item 2c) is left for after #5267. Signed-off-by: Niels Lohmann <mail@nlohmann.me> #5725 item 2 * Fix -Wextra-semi-stmt at its actual source, not assert() clang_flags.cmake blamed the global -Wno-extra-semi-stmt on assert(), but assert() expands to an expression under glibc and libc++ and does not trigger this warning. unit-assert_macro.cpp overrides JSON_ASSERT with "{if (!(x)) ++assert_counter; }", a bare block followed by a semicolon at every JSON_ASSERT(...) call site in the library; that was the actual source of 151 of the 208 -Wextra-semi-stmt sites found in a Clang 22 -Weverything sweep of the test suite with the flag removed. Switched to the standard do/while(false) macro idiom, which does not expand to a statement-plus-semicolon, and corrected the comment to name the remaining source instead: vendored Doctest's CAPTURE(x) shim, which already ends in a semicolon. Verified with clang++ -Wextra-semi-stmt (plus the file's other CI ignores) that unit-assert_macro.cpp now compiles without any -Wextra-semi-stmt diagnostic. Signed-off-by: Niels Lohmann <mail@nlohmann.me> #5725 item 8 (step 1 of 2; step 2 covers the CAPTURE() call sites) * Drop the redundant semicolon from CAPTURE() call sites; remove -Wno-extra-semi-stmt doctest_compatibility.h defines CAPTURE(x) as DOCTEST_CAPTURE(x); (with a trailing semicolon baked into the macro), specifically so call sites do not need to add one themselves; most of the ~267 call sites already follow that convention. The remaining 64 call sites across 20 files wrote "CAPTURE(x);" anyway, turning into a statement plus an empty statement and triggering -Wextra-semi-stmt. Dropped the redundant semicolon at each of those sites. With item 6 having already made vendored Doctest a SYSTEM include, and this the last known source of -Wextra-semi-stmt findings, removed the flag from clang_flags.cmake entirely. Verified with clang++ -Wextra-semi-stmt (plus the file's other CI ignores) that all 20 touched files, plus a file with no CAPTURE() use (unit-json_pointer.cpp), compile without any -Wextra-semi-stmt diagnostic. Signed-off-by: Niels Lohmann <mail@nlohmann.me> #5725 item 8 (step 2 of 2) * Switch ci_static_analysis_clang off the frozen LLVM 22 dev image ubuntu.yml pinned the clang-tidy/clang-tidy-sanitizer/single-binaries job to silkeh/clang:dev, a tag last pushed 2026-02-18 that reports "clang version 22.0.0 (...+20251015...)", a pre-release snapshot from before the LLVM 22 release; the maintainer now updates dev-unstable, 22, and latest instead. Switched to silkeh/clang:22, matching the other clang jobs on :latest. Verified with clang-tidy 22.1.8 (the image's actual version) against this repository's .clang-tidy and library headers what the release image newly reports compared to :dev: - readability-redundant-typename fires at ~250 sites across the _cpp20-relevant conversion/to_chars headers; the library targets C++11 and keeps the typenames, so the check is disabled in .clang-tidy, matching how the file already handles checks that don't fit a C++11 codebase. - misc-anonymous-namespace-in-header fires on the two anonymous namespaces in from_json.hpp and to_json.hpp; added the alias to their existing NOLINT (cert-dcl59-cpp, fuchsia-header-anon-namespaces, google-build-namespaces). - bugprone-std-namespace-modification fires on every addition to namespace std: the std::hash, std::formatter and std::swap overloads in json.hpp, and the std::tuple_size/std::tuple_element specializations in iteration_proxy.hpp (this last file is not named in #5725's item 5, found by actually running clang-tidy 22.1.8 against the current tree). All six are legal, deliberate additions to namespace std (explicit/partial specializations of std types, or the pre-C++20 std::swap overload); annotated each with the check name next to its existing cert-dcl58-cpp NOLINT. - modernize-avoid-c-style-cast reported nothing new. Also added clang++-22/21, clang-tidy-22/21, g++-16 and gcov-16 to the find_program search lists in ci.cmake so a local "maximal warnings" configure prefers the current toolchain version over an older one on PATH. #5725 item 5 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Regenerate cmake/gcc_flags.cmake for GCC 16.2.0 GCC_CXXFLAGS was generated for GCC 15.1.0, but ci_test_gcc and ci_test_gcc_cxx{11..26} now run in gcc:latest, currently GCC 16.2.0, so the "maximal warnings" job was missing warnings introduced since 15.1.0 while carrying entries GCC 16 treats as duplicates or no-ops. Regenerated with https://github.com/nlohmann/gcc_flags (patched locally to not crash on an option whose "-x c++ <opt> -" probe fails before it reads stdin, e.g. -Wabi=; the tool otherwise raises BrokenPipeError instead of recording the option as an error) run against g++ 16.2.0 in the official gcc:16 Docker image, keeping the documented -Wno-* exclusions and the same alphabetical placement scheme as before. Also added three GCC 16 warnings the generator cannot discover on its own because it only probes value ranges/lists it finds in the -Q option name itself, not in the enum choices --help=warnings documents separately: - -Wbidi-chars=any, -Wleading-whitespace=spaces: manually verified these compile cleanly with g++ 16.2.0. - -Wstrict-flex-arrays: deliberately NOT added, unlike the other two. Without -fstrict-flex-arrays (which the library does not enable, as it would change codegen for flexible array members), GCC prints "'-Wstrict-flex-arrays' is ignored when '-fstrict-flex-arrays' is not present" on every translation unit, and under our -Werror that note itself aborts the build. This differs from the harmless no-op warnings already kept in the file (-Whsa, -Wsynth, -Wunreachable-code, -Wunsafe-loop-optimizations), which emit nothing; #5725 item 7 named -Wstrict-flex-arrays as one of the flags GCC 16 adds, but did not anticipate this failure mode. Verified: compiled the library header and a representative set of test translation units (including ones touched by items 1, 3, 8, 9, 10 of this issue) with the regenerated GCC_CXXFLAGS plus -Werror under g++ 16.2.0 at -std=c++11 through -std=c++26, with zero warnings; ran the full local test suite (129/129 passing, unrelated to this compiler) as a regression check. CI must still confirm the actual ci_test_gcc / ci_test_standards_gcc targets end to end, since this was verified with direct g++ invocations rather than through the CMake/ CXXFLAGS environment-variable plumbing in ci.cmake. #5725 item 7 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Avoid std::basic_string<CharType> for non-character output_adapter CharType output_adapter<CharType, StringType> defaulted StringType to std::basic_string<CharType>, and (with JSON_NO_IO undefined) always declared a std::basic_ostream<CharType>&-taking constructor. For CharType with no non-deprecated std::char_traits specialization (only std::uint8_t is ever used this way, by the binary writers), simply naming either type - as an unused default template argument, or as an unused, never-called constructor's parameter type - instantiates std::char_traits<CharType> merely to name it, which some standard libraries mark deprecated: with the library-wide -Wdocumentation pragma (item 2's other half, left for a later commit) temporarily removed, an Apple clang 21 / libc++ TU calling json::to_cbor(j, vec) with std::vector<std::uint8_t>& got one -Wdeprecated-declarations warning per binary writer at the old output_adapters.hpp:193. Replaced the eager std::basic_string<CharType> / std::basic_ostream <CharType> defaults with a bool-tagged partial specialization (not std::conditional, which requires naming both branches' types up front regardless of which is selected, reproducing the same warning) that only ever names std::basic_string<CharType> / std::basic_ostream <CharType> when CharType is actually one of char, wchar_t, char16_t, char32_t, or (with __cpp_lib_char8_t) char8_t. For any other CharType, output_adapter's StringType and ostream-constructor parameter fall back to two distinct empty placeholder types, kept distinct so the two constructor overloads do not collide into a single redeclaration. Public API / behavior: passing a std::basic_string<std::uint8_t>& or std::basic_ostream<std::uint8_t>& directly to a binary writer's output_adapter now fails to compile instead of compiling with a deprecation warning; this was neither documented nor tested. All documented uses (std::vector<CharType>, std::basic_ostream<CharType> and StringType for character CharType) are unaffected. Verified with Apple clang 21 / libc++, with the two -Wdocumentation* "ignored" pragma lines in macro_scope.hpp temporarily removed and -std=c++11/c++20 plus the project's -Weverything flag set: calling to_cbor/to_msgpack/to_ubjson/to_bjdata/to_bson/to_bon8 on a std::vector<std::uint8_t> now produces no char_traits<unsigned char> (or any other) deprecation warning, while the char-based string- and ostream-adapter paths, and a to_cbor/from_cbor round trip, still compile and run correctly; also verified with GCC 16.2.0. Ran the full local test suite, including the binary-format unit tests (unit-cbor, unit-msgpack, unit-ubjson, unit-bjdata, unit-bson, unit-bon8, unit-binary_writer_sinks, unit-binary_formats, unit-custom-binary-type): 129/129 passing. #5725 item 2 (step a) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Remove the library-wide -Wdocumentation pragma; fix what it hid macro_scope.hpp / macro_unscope.hpp pushed and popped a Clang diagnostic region over the entire library that ignored -Wdocumentation and -Wdocumentation-unknown-command. Removed both pragmas and fixed every finding a full -Wdocumentation (which implies -Wdocumentation-unknown-command and -Wdocumentation-deprecated-sync) build reports, so the library now compiles clean under Clang's documentation checks without a blanket suppression. Overlaps #5267, which is still open and edits a nearby doc block (json.hpp's get()/get_impl() @return, already fixed in the item 2 step (b) commit of this branch); this commit does not touch that block again. Unknown Doxygen alias commands (Doxyfile removed in #3071, so these were never rendered by anything) rewritten as plain prose, keeping the same information: - @requirement REQ-JSON-01 / REQ-JSON-02 (iter_impl.hpp, json_reverse_iterator.hpp): now "This class satisfies the following concept requirements (REQ-JSON-0N):". - @liveexample{prose,example-id} (three sites in json.hpp): kept the prose, dropped the command wrapper and the trailing example-id (docs/mkdocs/docs/examples/*.cpp still exist and are used directly by the rendered docs, not through this in-header alias) and unescaped the "\," commas that were only needed for the old alias's comma-separated argument syntax. - @complexity X (json.hpp x4, json_pointer.hpp x2, serializer.hpp x1): now "Complexity: X". Backslash sequences Clang's comment lexer tried to parse as commands, escaped to render as literal backslashes: - lexer.hpp get_codepoint(): two `\u` occurrences. - binary_reader.hpp get_bson_cstr() / get_bson_cstr_bulk(): two `\x00` occurrences. - serializer.hpp: three `\uXXXX` occurrences (constructor @param, append_codepoint_to_string_buffer() @brief, and the ensure_ascii member comment). One finding remained after all of the above: Clang reports "declaration is marked with '@deprecated' command but does not have a deprecation attribute" on the deprecated sax_parse(span_input_adapter&&, ...) overload, even though JSON_HEDLEY_DEPRECATED_FOR does expand to __attribute__((deprecated(...))) for Clang. Several isolated reproductions of this exact declaration shape - doc comment, template<>, two stacked __attribute__ macros, an overload set sharing the name - did not reproduce the warning, so this looks like a Clang comment/declaration-association quirk specific to this overload inside the much larger basic_json class template, not an actual documentation defect. Rather than keep the pragma library-wide for one Clang false positive, added a tightly scoped -Wdocumentation-deprecated-sync push/pop around just that overload. Verified with Apple clang 21 and the project's actual -Weverything flag set (cmake/clang_flags.cmake) on the full header at -std=c++11 and -std=c++20: zero -Wdocumentation* diagnostics. Also compiled clean with GCC 16.2.0 (the pragmas are already __clang__-gated, so this only confirms no unrelated breakage). Ran make check-amalgamation and the full local test suite: 129/129 passing. #5725 item 2 (step c) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Take the JSON value by const reference in the array and tuple from_json paths Review feedback on #5737 (gregmarr): once the no-op std::forward calls are gone, the forwarding references have no purpose. from_json_fn passes the value as const BasicJsonType&, so these functions were only ever instantiated with a const lvalue anyway. The std::array, std::pair and std::tuple overloads of from_json and their helpers now take const BasicJsonType& and pass j on unchanged. Because the deduced BasicJsonType is now the plain type, tuple_type and the static_assert name const BasicJsonType& explicitly, so the reference checks are unchanged: get<std::tuple<const std::string&>>() still works, and get<std::tuple<std::string&>>() still fails the same static_assert. from_json_tuple_get_impl keeps its forwarding reference, since tuple_type calls it through std::declval. Behavior, the public API and the ABI do not change. unit-conversions, unit-constructor1, unit-udt, unit-udt_macro, unit-regression1/2/3, unit-deserialization, unit-noexcept, unit-items, unit-allocator, unit-custom-object-type, unit-ordered_json2 and unit-brace-init-copy-semantics pass at C++11, C++17 and C++20 with unchanged assertion counts. Ran make amalgamate. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
3926fcaac3 |
Deduplicate serializer dump code; fix stale includes, docs, and lint (#5729)
* Share scalar serialization between dump_internal and dump_value dump_value()'s cases for string, binary, boolean, number_integer, number_unsigned, number_float, discarded and null were a byte-for-byte copy of dump_internal()'s (added together in #5285 for the iterative fallback path). Any future change to scalar output had to be made in both places, or the recursive and depth-limited paths would silently start producing different bytes. Extract the shared cases into a private dump_scalar() and have both dump_internal() and dump_value() call it. Output is unchanged: dump(), dump(4), dump(-1,' ',true) and the replace/ignore error_handler_t variants are byte-identical over the json_test_data corpus before and after, and dump() throughput on a scalar-heavy document is unaffected. Part of #5709 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Drop serializer.hpp's dependency on binary_writer.hpp The only use of binary_writer in serializer.hpp was binary_writer<BasicJsonType, char>::to_char_type() to write the U+FFFD replacement character's three bytes. With CharType=char this is an identity conversion, so the include of binary_writer.hpp (and transitively binary_reader.hpp) pulled in a large, unrelated header for a no-op call. Write the three bytes directly instead. serializer.hpp compiles standalone with -Wall -Wextra -Werror, with and without -funsigned-char, and unit-serialization's error_handler_t::replace cases (with and without ensure_ascii) still pass. Moving binary_writer's to_char_type/to_msgpack_length to its private section is left as an optional follow-up. Part of #5709 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix stale #include lines in the output headers output_adapters.hpp included <algorithm> and <iterator> for std::copy and std::back_inserter, which have not been used there since #3569 (2022). serializer.hpp included <algorithm> for std::reverse (also unused), <cmath> for labs/isnan/signbit (only std::isfinite is used) and <utility> for std::move (nothing from <utility> is used there), while using std::next without including <iterator> at all, relying on getting it transitively through output_adapters.hpp's own stale <iterator>. Drop the unused includes, add <iterator> for std::next, and correct the remaining include comments. Both headers still compile standalone with -Wall -Wextra -Werror. Part of #5709 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Remove JSON_HEDLEY_NON_NULL(2) from write_characters() overrides output_vector_adapter, output_stream_adapter and output_string_adapter declared their write_characters(const CharType*, std::size_t) override JSON_HEDLEY_NON_NULL(2), but binary_writer legitimately calls it with a null pointer and length 0 for an empty string or binary value; the type-erased call path only stayed silent under UBSan because the static callee at those call sites is the unattributed virtual base. A nonnull attribute on a definition lets GCC and Clang assume the parameter is non-null inside the function body even when the call is virtual, so this was latent undefined behavior, not just style. Drop the attribute from the three overrides and document the (nullptr, 0) contract on output_adapter_protocol::write_characters. unit-cbor, unit-msgpack, unit-bson and unit-bon8 (which all exercise empty binary/string payloads through the stream and vector/string adapters) pass under -fsanitize=address,undefined,nonnull-attribute. Part of #5709 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Update stale serializer doc comments to match the current implementation dump_internal()'s doc block still described the pre-#5285/#5449 implementation: an escape_string() function that does not exist (the function is dump_escaped), integer conversion "implicitly via operator<<" (dump_integer actually uses a digit-pair lookup table), and floating-point conversion via "%g" (IEEE-754 types go through to_chars, others through snprintf). dump_value()'s comment said elements are pushed for dump_internal to walk, but it is dump_iteratively() that walks the stack. dump_escaped(), dump_integer() and dump_float() each said they write "to output stream @a o", which has not been true since the writer moved to write_buffer. Doc-only change; no behavior, API or ABI impact. Part of #5709 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Merge duplicate byte-to-hex helper and drop stale '| 0' promotions serializer::hex_bytes() and binary_writer::hex_byte() had identical bodies. Keep one, detail::hex_byte() in string_utils.hpp, and use it from both. Also drop the `| 0` at the two serializer call sites (hex_bytes(byte | 0) and hex_bytes(s.back() | 0)): #3088 ( |
||
|
Niels Lohmann
|
c261578431 |
Deduplicate binary reader/writer helpers and fix stale comments (#5730)
* Fix stale and missing comments in binary_writer The doc block of write_number() ended up above the byte_swap() helpers added in #5286, about 80 lines from the function. It was also a plain comment that Doxygen skips, said "write a number to output input", and left BON8 out of the big-endian formats. Move it back onto write_number() as a /*! block and fix the text. write_bson() documented "@pre j.type() == value_t::object", but it throws type_error.317 for every other type, and to_bson() relies on that. Document the exception instead. Explain why the CBOR binary subtype is always written with a 0xD8..0xDB head and never in the one-byte tag form: binary_reader with cbor_tag_handler_t::store only keeps those heads as a subtype, so switching to write_cbor_head() would break round trips for subtypes 0..23. Also fix the grammar of the to_char_type comment. Comments only; no change in behavior, API or ABI. Part of #5710 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Merge the duplicated UBJSON/BJData integer marker ladders write_number_with_ubjson_prefix() (unsigned and signed overloads) and ubjson_prefix() (number_integer and number_unsigned cases) each picked the UBJSON/BJData integer marker (i, U, I, u, l, m, L, M, H) with their own independent if/else ladder, and the values beyond 64 bits were handled by a second, tag-dispatched pair of ladders. An optimized container announces the marker of its first element via ubjson_prefix() and then writes every element through write_number_with_ubjson_prefix(), so the two had to be kept in lockstep by hand across four call sites. Replace all of that with one ubjson_integer_prefix() built on value_in_range_of<T>, and one write_ubjson_integer_payload() that writes the value (or, for 'H', the decimal digits) for a given marker. write_number_with_ubjson_prefix() and ubjson_prefix() keep their signatures and now just call these two helpers. Behavior, the public API and the ABI are unchanged. Verified with a new regression test covering scalars and $-optimized arrays/objects at every int8/uint8/int16/uint16/int32/uint32/int64/uint64 boundary for to_ubjson/to_bjdata (both use_size/use_type settings), and by diffing to_ubjson/to_bjdata output before and after over the json_test_data corpus (bit-identical). Part of #5710 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Remove dead get_char parameters in binary_reader The non-recursive rewrite of the binary readers (#5505, #5506, #5507) left parse_cbor_internal()'s and parse_ubjson_internal()'s get_char parameters dead: parse_cbor_internal() has one caller and it always passes true, and parse_ubjson_internal() has one caller and it always uses the true default. Both parameters, and the @param docs describing the "reuse the last character" mode they used to select, no longer correspond to anything. Drop both parameters, initialise fetch/prefix unconditionally, and update the two call sites in sax_parse(). parse_cbor_value()'s and get_ubjson_string()'s own get_char parameters are unrelated and are left alone; both still have a false caller. Also delete a stray `@return whether a valid MessagePack value was passed to the SAX parser` doxygen block that sits directly above parse_msgpack_value()'s real doc comment, a leftover of the same rewrite. Behavior, the public API and the ABI are unchanged; these are private members of detail::binary_reader. Verified by compiling with -Wunused-parameter and running unit-cbor, unit-ubjson, unit-bjdata and unit-msgpack (offline, against the stubbed test_data.hpp). Part of #5711 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Share the IEEE half-precision decoder between CBOR and BJData binary_reader had two ~45-line copies of the IEEE 754 half-precision decoder: CBOR's case 0xF9 and BJData's case 'h'. Once formatting is normalised, the two blocks were identical except for the byte order used to assemble the 16-bit half (CBOR is big endian, BJData is little endian). Any future change to half-float decoding had to be made and kept in sync in both places. Add one get_half_float(format, little_endian) helper that does the two get()/unexpect_eof() reads, assembles the half in the requested byte order, decodes it per RFC 8949 Appendix D, and calls sax->number_float. Both cases now just call it with their byte order; the BJData case keeps its bjdata-only guard. Behavior, the public API and the ABI are unchanged. Verified with a scratch probe comparing the old and new decoders bit-for-bit (NaN by isnan()) over all 65536 wire byte pairs, in both formats, and by running unit-cbor and unit-bjdata (offline, against the stubbed test_data.hpp). Part of #5711 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Deduplicate the MessagePack unsigned-integer writer ladder The number_integer (non-negative branch) and number_unsigned cases in write_msgpack() each held their own copy of the fixint/uint8/16/32/64 ladder, kept in lockstep only by a comment ("we used the code from the value_t::number_unsigned case here"). Both copies mixed union members: the signed copy compared number_unsigned but wrote number_integer, and vice versa. Extract write_msgpack_unsigned(std::uint64_t), mirroring how write_cbor_head() already avoids the same duplication for CBOR, and call it from both cases. Each case now reads only its own active union member. Output bytes are unchanged for the default 64-bit number types. #5710 item 3 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Unify float marker selection and fix the long double compile error Four formats picked between a float32 and float64 marker through four different helper styles: dummy-argument overloads for CBOR and MessagePack, an std::is_same template for BON8, and a runtime if-chain on input_format_t for write_compact_float(). With number_float_t set to long double, to_cbor, to_msgpack and to_ubjson failed inside the library with "call to 'get_cbor_float_prefix' is ambiguous", while to_bson kept working because write_bson_double() takes a plain double. Change write_compact_float() to take the two marker bytes directly (each of its three callers already knows them at compile time) instead of an input_format_t it only forwarded, and delete the now-unused get_cbor_float_prefix(), get_msgpack_float_prefix(), get_bon8_float_prefix() and get_compact_float_prefix() helpers. Turn the two get_ubjson_float_prefix() overloads into one template. Both write_compact_float() and get_ubjson_float_prefix() now report an unsupported number_float_t with a static_assert naming the requirement, rather than an ambiguous-overload error; the assert lives in the function body, not the class scope, so to_bson with long double is unaffected. Verified with a probe basic_json<..., long double>: to_bson still compiles and round-trips, while to_cbor/to_msgpack/to_ubjson now fail to compile with the new static_assert message. This changes the text of an existing compile error for users with an unsupported number_float_t (documented as a public-API-visible change in #5710). #5710 item 1 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Deduplicate the BJData ndarray writer's dtype dispatch and drop <map> write_bjdata_ndarray() built a 12-entry std::map<string_t, CharType> on every call just to translate the _ArrayType_ name to a dtype marker (the only reason binary_writer.hpp included <map>), then mapped dtype to C++ type twice more: once as a switch for the range-check pass and once as a separate if/else chain for the write pass, with nothing checking that the two agreed. The caller also ran three at() lookups, and the callee called value.at(key) about ten more times for the same three members. Replace the map with bjdata_ndarray_type_marker(), a plain string comparison chain (a C++11 constexpr function cannot contain a switch, so this mirrors binary_reader's own static table style). Replace the switch/if-chain pair with one write_bjdata_ndarray_elements() that switches on dtype once and calls a per-type helper - write_bjdata_ndarray_element<T>() for the eight integer dtypes and write_bjdata_ndarray_float_element() for 'd' - with a dry_run flag selecting the range check or the actual write, so the two passes can no longer disagree on the type. _ArrayType_, _ArraySize_ and _ArrayData_ are now looked up once into references, and the four header marker bytes ('[', '$', '#') are written through to_char_type() like the rest of the UBJSON/BJData writer. The 'd' (single-precision) rule is left exactly as before, since #5707 is expected to change it separately. Verified byte-for-byte identical output before/after for every dtype (including the Draft 2/Draft 3 'byte' fallback and the use_count/ use_type combinations) via a standalone probe, plus round-tripping through from_bjdata(). Overlaps #5707, which is expected to touch the 'd' dtype case, and #5518, which is expected to move the write_bjdata_ndarray() call site. #5710 item 4 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Assert that write_bson_document() consumes every calc_bson_sizes() entry calc_bson_sizes() and write_bson_document() are a hand-synchronized pair of passes over the same object/array tree, introduced by #5553: the size pass appends to nested_sizes in visiting order, and the write pass consumes the table by position with nested_sizes[next_size++]. Nothing checked that the write pass consumed the whole table. If a future change touched only one of the two passes - for example to skip or reject an entry - every later size prefix in the document would be silently wrong. Add JSON_ASSERT(next_size == nested_sizes.size()) where write_bson_document() returns, so such a future drift between the two passes is caught immediately (JSON_ASSERT expands to nothing in release builds using assert(), and the fuzzers/tests already build with it enabled). The two passes agree today, so this changes nothing observable; it only guards against the risk described in #5710 item 5. Extracting a shared stepper for the two passes (the second half of the proposed change) is left for a follow-up: it only saves ~30 lines and the issue asks for it only if the result reads clearly, which needs more room to get right than a mechanical cleanup pass allows. #5710 item 5 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Make the BJData lookup tables static functions instead of members binary_reader held bjd_optimized_type_markers and bjd_types_map as non-static const members (12 string_t objects for the type-name table), built and destroyed on every from_cbor/from_msgpack/from_bson/ from_ubjson/from_bon8/from_bjdata call even though only from_bjdata ever reads them. They also needed the #define/decltype/#undef workaround from #3637 and two NOLINTNEXTLINE suppressions, and binary_writer already carries the same two lists in another form (is_bjdata_excluded_type_marker() and a local std::map in write_bjdata_ndarray(), the latter removed by the item-4 commit), so the excluded-marker lists could drift apart. Replace bjd_optimized_type_markers with static constexpr is_bjd_excluded_optimized_type(char_int_type), using the same ||-chain as binary_writer's is_bjdata_excluded_type_marker(). Replace bjd_types_map with a non-constexpr static bjd_type_name(char_int_type) switch returning nullptr for an unknown marker (a C++11 constexpr function cannot contain a switch). Delete both JSON_BINARY_READER_MAKE_* macros, the bjd_type pair alias, the NOLINTNEXTLINE suppressions, detail::make_array() (no longer used anywhere), and the now-unused <algorithm> and <array> includes. Update the two call sites (the ND-array excluded-type check and the _ArrayType_ lookup) accordingly, and replace unit-bjdata.cpp's "LUT arrays are sorted" section, which only checked the two tables' internal ordering, with a check of all 12 type names and all 8 excluded markers against both new functions. #5711 item 1 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Read CBOR's 1/2/4/8-byte argument through one helper parse_cbor_internal() hand-wrote the same "read a 1/2/4/8-byte big-endian unsigned integer" ladder four times over: - twice for tag numbers 0xD8-0xDB, once in the tag_handler::ignore branch and once, nearly identically, in the ::store branch (~90 lines to read one integer); - twice more for container lengths, once for array heads 0x98-0x9B and once for map heads 0xB8-0xBB, where the 1/2-byte forms called enter_array()/enter_object() directly and the 4/8-byte forms additionally went through get_cbor_container_size(). Add get_cbor_argument(std::uint64_t&), reading the width selected by current & 0x1F via the same get_number() calls as before (so EOF is reported exactly as before), and route all four sites through it: - 0xD8-0xDB now read the argument once per branch instead of switching on `current` a second time; behavior split cleanly from embedded tags 0xC0-0xD7 (tag value in the head, no argument to read), which is now its own case block that no longer has to fall into the ::store switch's "default" case to reach the same tag_pending = true; return true; outcome. - 0x98-0x9B and 0xB8-0xBB collapse into one case block each, always going through get_cbor_container_size() (harmless for 1/2-byte lengths, which already always fit). Verified byte-for-byte identical behavior before/after with a standalone probe covering embedded and multi-byte tags under all three tag_handler_t settings, a tag over a byte string (subtype path), truncated tag/length arguments of every width, and array/map lengths of every width, including the out_of_range.408 "excessive size" case: same exceptions, same messages, same chars_read, same successful results. Left the string/byte-string length ladders in get_cbor_string()/ get_cbor_binary() untouched, as noted in #5711 item 2, since #5325 is expected to touch them separately. Overlaps #5601 (adds a branch right above the embedded-tag case) and #5607 (touches the integer cases 0x18-0x1B, which share this ladder's shape in separate hunks). #5711 item 2 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Add leave_container() to match enter_container() Every container is opened through enter_container(), whose docs promise that a check placed there runs before every start event. The close side had no equivalent: the same "container_stack.pop_back(); dispatch to end_object() or end_array()" sequence was written out separately in BSON, CBOR, MessagePack, UBJSON/BJData and BON8, each copying the pattern of keeping an is_object flag around the pop_back() that would otherwise invalidate a reference to it. A check needed on close would have had to be added in five places, and a sixth copy could go unnoticed. Add leave_container() next to enter_container(), doing the same pop-then-dispatch, and replace the five sites with it. Each site keeps its own surrounding logic (BSON's check_bson_document_size() call before popping, MessagePack's is_object copy used again below, UBJSON/BJData's remaining-container handling after popping, BON8's top used again below); only the repeated pop/dispatch line pair is now shared. Verified all six binary-format unit suites and unit-regression2's deep-nesting tests (dependent count/reuse count and the bjdata ndarray depth cases) still pass, compiled with -Wall -Wextra and ASan/UBSan. Overlaps #5601, which is expected to add a sixth close site in its own skip loop; that site can route through leave_container() too once it lands. #5711 item 4 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Stop passing the input format to sax_parse() when the reader already has it binary_reader's constructor stores the format in the input_format member, and sax_parse(format, sax_, strict, tag_handler) took the same value again purely to dispatch on it. Every in-tree caller passed the same value both times (all 16 from_cbor/from_msgpack/from_ubjson/ from_bjdata/from_bon8/from_bson call sites in json.hpp, and the three public basic_json::sax_parse() overloads), so nothing was broken today, but a caller of the detail class directly (only reachable via JSON_PRIVATE_UNLESS_TESTED, as unit-bjdata.cpp already does) could pass a mismatched pair - say bjdata to the constructor and ubjson to sax_parse - and dispatch on one format while applying the other format's rules; the default-constructed input_format_t::json reader would additionally hit JSON_ASSERT(false) in exception_message() on its first error. Add sax_parse(json_sax_t*, bool, cbor_tag_handler_t) forwarding to the existing overload with the stored input_format, and switch every caller to it: the 16 from_*() sites (keeping their `// cppcheck-suppress[accessMoved]` comments) and the three basic_json::sax_parse() overloads, all of which already had the format available from their own `format` parameter. The four-argument overload is kept for anyone still calling it, now with JSON_ASSERT(format == input_format) so a mismatch fails immediately in a debug build (assert-enabled binaries, including the fuzzers and test suite) instead of misbehaving; verified with a probe that constructs a reader for one format and calls the explicit overload with another, which aborts on that assertion as expected. Removing or asserting against the constructor's input_format_t::json default, which would affect direct detail users, is left as a separate decision per #5711 item 5. Overlaps #5601, which is expected to add an AllowRecovery template parameter to sax_parse() and touch these same call sites in json.hpp. #5711 item 5 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Deduplicate UBJSON/BJData signed-count handling, drop dead ndarray checks get_ubjson_size_value()'s 'i'/'I'/'l'/'L' cases each read a differently sized signed integer and then repeated the same "reject negative with error 113" check; only 'L' additionally checked value_in_range_of for the out_of_range.408 case. Any change to that error path had to be made four times. Add get_ubjson_signed_count<SignedType>(std::size_t&), doing the read, the negative check and the range check once, and route all four markers through it. The range check is a no-op for 'i'/'I'/'l' (their values always fit std::size_t) and only live for 'L' on a 32-bit std::size_t target, matching today's behavior exactly. In the ndarray dimension-product loop, the preceding loop already returns early on any zero dimension and result starts at 1, so `i > 0` in the pre-multiplication overflow check was always true, and `result == 0` in the post-multiplication check could not be reached either: two positive factors whose product does not overflow (as the pre-check already guarantees) cannot be zero. Drop the dead `i > 0 &&` and narrow the post-check to `result == npos`, the one case the pre-check cannot rule out (an exact, non-overflowing match with the sentinel reserved for unknown-size containers), with a comment explaining why. Verified byte-for-byte identical behavior before/after with a standalone probe covering negative counts for every marker, a matching positive count, and ndarray inputs, plus the full unit-ubjson and unit-bjdata suites (same assertion counts as before this change). Overlaps #5601 (rewrites the four parse_error calls and the overflow checks touched here) and #5607/#5707 (touch neighboring lines in the same functions). #5711 item 6 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Drop redundant format parameter and dummy float argument (review) binary_reader::sax_parse(format, ...) only ever had to equal the format given to the constructor, which it asserted. With every caller already on the format-less overload, remove the four-argument overload and dispatch on the stored input_format directly. binary_reader is a detail class, so this is not a public API change. get_ubjson_float_prefix() took a value only to deduce its type; make the type an explicit template argument instead. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
5bd766aa50 |
Move to_bson's binary subtype check into calc_bson_sizes (#5703)
to_bson() rejected a binary value's subtype above 255 (out_of_range.415) in write_bson_binary(), which only has the binary_t, not the basic_json value that holds it, so the exception was created with no JSON_DIAGNOSTICS context even though the equivalent to_msgpack() check names the value's path. The check also ran after the document size, all preceding elements, and this element's header and length had already reached the output adapter, so a caller-provided std::vector or std::string ended up holding a truncated document. calc_bson_sizes() already walks every value before anything is written, to size embedded documents and arrays and to reject invalid keys (out_of_range.409) up front. The subtype check now runs there instead, in calc_bson_binary_size(), which is given the basic_json value so the exception can use it as context. The now-redundant check in write_bson_binary() is removed, since calc_bson_sizes() always throws first if any binary value in the document has an oversized subtype. Fixes #5675. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
bfea6f36d3 |
Fix to_msgpack() reading the inactive number union member (#5694)
* Fix to_msgpack() reading the inactive number union member basic_json stores number_integer and number_unsigned in a union, and number_unsigned_t only has to be at least as wide as number_integer_t (with the default types, both are 64-bit and have the same representation). When number_integer_t is narrower, write_msgpack() read the wrong union member in two places: - The number_unsigned case wrote number_integer's bits instead of number_unsigned's, silently writing the wrong value whenever it did not fit in number_integer_t. - The number_integer case (non-negative branch) picked the encoded width by comparing number_unsigned's bits, which is undefined behavior, though the value written was still number_integer's, so at worst a too-wide encoding was chosen. Read the active member in both cases, like the other binary writers (CBOR, UBJSON, BJData, BSON, BON8) already do. Fixes #5644. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Cast number_integer to number_unsigned_t only once in to_msgpack() Addresses review comment by @gregmarr. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
1e101ecac1 |
Add BON8 support (#2998)
* Add BON8 support Add to_bon8/from_bon8 and input_format_t::bon8 for BON8, a binary format that uses the byte values that cannot begin a UTF-8 character as type markers, so strings need no length prefix. It is the most compact of the supported binary formats on the benchmark files. The reader is non-recursive like the other binary readers. A string ends at the first byte that cannot continue it, so the reader hands the one or two bytes it reads past a string back to the value that follows. The writer produces the canonical representation of the specification, except for NFC normalization; its output is identical to that of the reference implementation (HikoGUI) on all files of the test data. The round-trip tests need the .bon8 files of json_test_data 3.2.0. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address review comments - Reuse detail::validate_one_utf8 to check strings in to_bon8; the error now names the first byte of the invalid sequence. - Document that to_bon8 leaves bytes in the output adapter on an exception, and that string_open is only an output of write_bon8_marker. - Explain why the pushback buffer of the BON8 reader cannot overflow. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Select the BON8 float prefix by type get_bon8_float_prefix only depends on the type of its argument, so make the type a template parameter instead of passing an unused value. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Rename a test variable that Flawfinder mistakes for read() Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix the BON8 CI failures - compare the float in write_bon8_float with number_float_t constants, so GCC does not warn about a float-to-double conversion - mark check_bon8_utf8's context as used when exceptions are disabled - choose the compact float prefix in a helper rather than with nested conditional operators (clang-tidy) - use auto for the cast in the BON8 integer reader (clang-tidy) - write the int32 minimum test values as long long literals (MSVC C4146) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Amalgamate Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Read BON8 strings in bulk from contiguous input - copy the valid UTF-8 of a string in one step when the input is contiguous (twitter.json is read in 1.68 instead of 2.52 ms, jeopardy.json in 196 instead of 297 ms, close to CBOR and MessagePack) - share the new valid_utf8_prefix() with the writer's UTF-8 check, which now skips ASCII 8 bytes at a time - let the fuzzer check that contiguous and stream input give the same value or error, and test both paths in the unit tests - clarify that a second 0xFF after a string is an empty string Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Link the BON8 functions from the other binary format pages Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Name the bulk scan flag after the input, not BON8 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Read BSON keys in bulk from contiguous input BSON keys (and array indices) are C-style strings, which were read byte by byte. For contiguous input they are now read up to their \x00-byte in one step, using the same bulk_scan flag as BON8 strings: twitter.json is read in 1.46 instead of 2.01 ms, citm_catalog.json in 2.93 instead of 3.33 ms, jeopardy.json in 182 instead of 207 ms. canada.json, whose keys are almost all one-digit array indices, takes 2 % longer. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix the BON8 CI failures of the bulk-read tests - skip the contiguous-versus-stream tests of BON8 strings and BSON keys when exceptions are disabled: they catch the parse errors of invalid input, and without exceptions the library aborts instead - use static_cast for the int64 test value (google-readability-casting) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Move the explicit basic_json instantiation into its own test file Linking test-regression3_cpp20 with clang and MinGW failed with "relocation truncated to fit: IMAGE_REL_AMD64_REL32 against `.rdata'", as test-regression2 did before #5511. The explicit instantiation of basic_json<> for #4825 compiles every member function, including the BON8 reader and writer, into that object, and it was already close to the limit (2,226,104 bytes on develop, 2,234,960 with BON8; clang -O1, C++20). Give the instantiation a file of its own: unit-regression3 is now 1,594,736 bytes and unit-explicit_instantiation 1,095,064. The new file mentions JSON_HAS_CPP_17 and JSON_HAS_CPP_20 so it keeps being built for the C++17 standard the regression was about. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Convert the bytes of the BON8 test strings explicitly The str() helper constructed a std::string from a byte range, which converts each unsigned char implicitly; -fsanitize=integer reports that for bytes of 0x80 and above (ci_test_clang_sanitizer). Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
f7972970a4 |
Throw instead of writing MessagePack lengths beyond UINT32_MAX (#5584)
* Throw instead of writing MessagePack lengths beyond UINT32_MAX MessagePack stores the length of a string, binary value, array, or object in at most 32 bits. For a larger value, to_msgpack wrote no length at all, so the output could not be read back. It now throws out_of_range.412, which BSON already uses for its 32-bit length fields. The check lives in one function, so each length is written by an if/else chain that ends in a plain else, without a condition that can never be false. It is tested with string and binary types that report a size beyond UINT32_MAX without allocating it, like the BSON tests do. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix the CI failures of the MessagePack length check - mark to_msgpack_length's value as used when exceptions are disabled (-Wunused-parameter, misc-unused-parameters) - put "Exception safety" before "Exceptions" in to_msgpack.md, as the documentation style check requires - create the test's string value from its type: constructing it from a beyond_uint32_string_t considers the std::filesystem::path conversion, which libstdc++ 10 reports as ambiguous for a class derived from std::string (clang 13) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Skip the MessagePack string length test for clang with libstdc++ 10 C++17 builds consider the std::filesystem::path conversion for the string type, and with clang and libstdc++ 10 that conversion is ambiguous for a class derived from std::string. Creating the value from its type did not avoid it, since any basic_json with that string type instantiates the check. The binary and ext cases are still tested there. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Keep the MessagePack string test type and its alias in one block astyle indented the alias oddly when it had an #ifdef of its own after the binary alias; declare it right after the string type, in the same block. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
4fa95d9810 |
Remove unreachable branches from the binary writer (#5583)
Coverage reported conditions in the binary writer that can never be false, and marked the code behind them with LCOV_EXCL. Remove them instead of excluding them: - CBOR writes the length of a string, binary value, array, or object exactly like an unsigned integer, only with another major type. One function, write_cbor_head(), now writes both, so the integer tests cover every width and the four excluded 64-bit length branches are gone. - A last `else if` whose condition holds for every remaining value (an unsigned value at most UINT64_MAX, a signed one in the range of int64_t) is now a plain `else`. - Whether a signed integer fits into an int64 for UBJSON and BJData is decided by its type at compile time. Only an integer type wider than 64 bits gets a range check and the high-precision fallback. - The private get_impl(boolean_t*) was never called. The UBJSON type prefix 'H' of an optimized container of unsigned integers beyond the range of int64 was reachable although excluded; it is tested now. The output is unchanged. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
 Dadi Reddy Sai Praneeth Reddy
|
fe4a544c7e |
handled when size exceed uint32 (#5515)
* handled when size exceed uint32 Signed-off-by: dsp0redy <saipraneethreddy.dadireddy@gmail.com> * addressed review comments Signed-off-by: dsp0redy <saipraneethreddy.dadireddy@gmail.com> * updated unit test Signed-off-by: dsp0redy <saipraneethreddy.dadireddy@gmail.com> * added amalgamation patch Signed-off-by: dsp0redy <saipraneethreddy.dadireddy@gmail.com> --------- Signed-off-by: dsp0redy <saipraneethreddy.dadireddy@gmail.com> |
||
|
Niels Lohmann
|
95e9a5931c |
Write BSON in linear time, without recursing per nesting level (#5553)
* Write BSON in linear time, without recursing per nesting level to_bson() had two problems with nested values: - It recursed once per nesting level, so a value nested deeply enough - 100,000 levels on an 8 MiB stack - exhausted the call stack and terminated the process, although parse() accepts such values without complaint. - BSON prefixes every document and array with its length. The writer computed that length by walking the entire value below it, again for every nested document it wrote, which made serializing O(size x depth). A 200-level document took 30 ms instead of 1. Both passes are now iterative, and each length is computed exactly once: - calc_bson_sizes() computes the length of every document and array in one pass, each from the lengths of its entries, into a table ordered the way they are written. - write_bson_document() then writes the document, taking each length from the table. Everything observable is unchanged, as a differential test against develop confirms byte for byte: - The same bytes are written. - A key containing U+0000 still throws out_of_range.409 for the same first key, with the same diagnostics path, before anything is written. - A document too large for BSON still throws out_of_range.412 before anything is written. - A binary subtype above 255 still throws out_of_range.415 after the same partial output. Only the enclosing objects and arrays are kept on a stack, so a flat document allocates nothing for it. Measured against develop (clang -O3, median of 201 runs): flat objects unchanged, flat arrays 37% faster (the array length was computed twice), a nested 3,000-object document 2x faster, a 200-level document 33x faster. to_bson.md documented the quadratic complexity since #5334; it is linear again. Fixes #5392 for BSON, and #5308. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Do not require a default-constructible string_t in the BSON writer GCC 4.9 and MSVC rejected the test's huge_string_t, which has no default constructor; develop never default-constructed string_t here either. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Let the BSON index-name helper only fill its output parameter It returned a reference to the string it filled, so callers held a second name for index_name. Addresses review feedback. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
 Niels LohmannandClaude Sonnet 5
|
d19f7f5dce |
Fix BSON conformance issue (#5185)
* 🐛 fix BSON conformance issue Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🐛 fix BSON conformance issue Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🐛 reject ill-formed UTF-8 in CBOR/MessagePack/BSON text strings at decode time (#5531) from_cbor()/from_msgpack()/from_bson() copied the raw bytes of a decoded text string into the resulting json value without any UTF-8 validation, even though RFC 8949 §3.1 (CBOR) and the MessagePack/BSON specifications all require text strings to be valid UTF-8. Malformed input only failed later, if the value was dump()'d, with a type_error.316 - so the allow_exceptions=false pattern used specifically to get a discarded sentinel instead of an exception did not discard this category of malformed input, unlike every other kind of malformed binary input this library rejects at decode time (see #5529). Fix this at the single choke point shared by BSON/CBOR/MessagePack/UBJSON string reads, binary_reader::get_string(): validate the bytes with the UTF-8 DFA right after they are read, and report failures the same way as every other binary_reader error (parse_error.113), so allow_exceptions and strict discarding behave consistently. get_binary()/binary blob reads are untouched and still accept arbitrary bytes, since only text strings are required to be UTF-8. There were two independent implementations of a UTF-8 validator: the lexer's streaming scanner, and the serializer's Hoehrmann DFA used by dump_escaped_impl(). Rather than write a third, the serializer's decode() function, its utf8d table and the UTF8_ACCEPT/UTF8_REJECT constants are extracted into detail/string_utils.hpp (a low-level header already included before both detail/input/ and detail/output/), alongside a new is_valid_utf8() helper built on the same decode() step. serializer.hpp's dump_escaped_impl() now calls the shared decode(), so there is exactly one UTF-8 validator in the codebase; dump()'s exact type_error.316 messages and byte-index reporting are unchanged (see the added regression-guard test in unit-serialization.cpp). Claude-Session: https://claude.ai/code/session_01N4RQ1Ahan5YAGbnAQGjZTY Signed-off-by: Niels Lohmann <mail@nlohmann.me> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> * ⚡ validate only newly read bytes of binary-format strings get_string() validated the whole result after each call, but get_bytes() appends to it and CBOR indefinite-length strings collect all chunks in the same result, so every chunk re-validated everything read before it. An input of many small chunks took quadratic time (80000 one-byte chunks, 160 KB of input, took about 7 seconds). Only the newly read bytes are validated now, which also matches RFC 8949's requirement that every chunk is valid UTF-8 on its own. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
Niels Lohmann
|
7c90ec2323 |
Hash deeply nested values without recursing per nesting level (#5546)
* Hash deeply nested values without recursing per nesting level std::hash<basic_json> hashed an array or object by hashing each element, which called detail::hash again once per nesting level. A value nested deeply enough - 50,000 levels of objects on an 8 MiB stack - exhausted the call stack and terminated the process. parse() accepts such values without complaint, since the parser is iterative, and a parsed value is hashed wherever it is used as a key in an unordered container. Bound the descent the same way dump() does: detail::hash takes the nesting level, and once hash_depth_limit() (128) levels have been entered, hash_iteratively() hashes what is left on an explicit stack. It combines the seeds in exactly the same order, so hash values are unchanged. A value nested less deeply than the bound is hashed by the same code as before, without allocating, and is as fast as before. Tests check that every depth up to twice the bound hashes exactly like the recursive definition of the hash, and that values nested 100,000 levels deep hash without crashing. Fixes #5545 for std::hash. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Declare hash_frame's constructor noexcept GCC's -Wnoexcept (an error in CI) flags the emplace_back() into the hash stack under C++26: the constructor cannot throw, since cbegin() is noexcept, but it did not say so. dump_frame's constructor is noexcept for the same reason. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Share one recursion depth limit, and copy the hash frame out of the stack dump() and hash() each defined their own limit on how many nesting levels they recurse into, and the operations still to come would have added more, free to diverge over time. They now all use detail::recursion_depth_limit(), in a header of its own; serializer::dump_depth_limit() and hash_depth_limit() are gone. hash_iteratively() now copies the frame it works on out of the stack and changes the frame only through stack.back(), so nothing can refer into the stack after entering an element has grown it. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Parenthesize multiplications in the hash test for clang-tidy Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
8699de3064 |
Stop allocating the BJData excluded-marker list per container (#5555)
write_ubjson() built a std::vector of the eight markers BJData forbids as the type of an optimized container - one heap allocation plus a linear search for every array and object it wrote with use_type, even for plain UBJSON output, where the list isn't consulted. The list was also spelled out twice. A constexpr helper, is_bjdata_excluded_type_marker(), replaces both. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
918da64657 |
Keep BJData ndarray annotations that would not survive a round trip as objects (#5542)
write_bjdata_ndarray() encoded a JData-annotated object as a BJData
ND-array whenever its dimensions' product matched _ArrayData_.size(),
which lost information in two ways:
- _ArrayData_ was never required to be an array. null has size 0, any
other scalar has size 1, and iterating an object visits its values, so
e.g. {"_ArraySize_":[1],"_ArrayData_":5} was written as the array [5],
and an object _ArrayData_ came back as an array.
- The reader only restores an annotated object from an ND-array with at
least two non-zero dimensions that is not a 1xN row vector; an empty,
1-D, row-vector, or zero-sized shape is read back as a plain array. The
writer nonetheless emitted ND-array headers for these shapes, so the
annotation was silently dropped.
OSS-Fuzz issue 563659413 hit this in parse_bjdata_fuzzer: an empty binary
_ArraySize_ is written as a plain object and read back as an empty array,
after which {"_ArrayType_":"int16","_ArraySize_":[],"_ArrayData_":null}
was encoded as the ND-array header "[$I#[]" and re-read as [], failing the
harness's value-stability check.
Such objects now fall back to a plain object encoding, which round-trips.
Genuine ND-arrays (two or more positive dimensions, not a 1xN row vector)
are encoded exactly as before. Existing fallback tests that used 1-D
shapes are moved to 2-D shapes so they keep exercising the check they
were written for, and the BJData documentation is updated.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
|
||
|
Niels LohmannandClaude
|
1054b2097e |
Speed up binary writing: value-type output sink + byte-swap number encoding (#5286)
* Devirtualize binary_writer via a value-type output sink to_cbor/to_msgpack/to_ubjson/to_bjdata/to_bson wrote every byte through output_adapter_t, a shared_ptr<output_adapter_protocol> whose write_character/write_characters are virtual. Unlike the lexer (templated on a concrete InputAdapterType), the binary writer never got that treatment, so binary output paid a vtable lookup per byte and a make_shared per call. Template binary_writer on an OutputSinkType and give it two concrete, non-virtual sinks: - output_vector_sink: appends straight into a std::vector (push_back / insert), used by the vector-returning to_* convenience functions. No vtable, no shared_ptr; the writes inline. - output_adapter_sink: forwards to a type-erased output_adapter_t, so the existing to_*(j, output_adapter) overloads (streams, strings, custom adapters) keep working exactly as before -- one virtual call each, unchanged. binary_writer keeps a convenience constructor taking output_adapter_t (building the default output_adapter_sink), so the adapter overloads are untouched; only the convenience functions switch to the vector sink. The friend declaration and the basic_json binary_writer alias gain the new (defaulted) template parameter. Output is byte-for-byte identical: verified across ~3000 randomized values plus curated edge cases (all scalar widths, strings with invalid UTF-8, binary, nested arrays/objects) for CBOR, MessagePack, UBJSON (both size/type settings), BJData, and BSON, plus the output_adapter path, in C++11/17/20. Warning-clean under clang -Weverything and the gcc pedantic set; clang-tidy clean on the changed headers; make check-amalgamation clean. Throughput (g++ -O3, vs develop): scalar-dense binary output such as integer arrays ~1.4x; many small to_cbor calls ~1.04x (DOM traversal bound); string/blob-heavy output unchanged (already bulk-bound). No workload regressed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix CI failures from binary_writer output-sink change Four CI jobs failed on the initial commit; all are addressed here without changing any output (binary encodings remain byte-for-byte identical to develop across the differential corpus): 1. ci_test_gcc / cuda (-Werror=duplicated-branches): for number_float_t == float, static_cast<float>(n) is the identity, so write_compact_float's two branches are intentionally identical. Once the concrete vector sink is inlined, GCC constant-folds and diagnoses this (the type-erased path hid it behind a non-inlined virtual call). Silence -Wduplicated-branches for GCC (clang has no such warning) alongside the existing -Wfloat-equal pragma. 2. ci_static_analysis_clang (UBSan nonnull-attribute): binary_writer passes a null pointer with length 0 for empty strings/binary. output_vector_sink / output_adapter_sink declared write_characters JSON_HEDLEY_NON_NULL, so the sanitizer flagged the (harmless) zero-length call once the sink was called directly rather than through the attribute-free virtual base. Drop the attribute from both sinks, matching the pre-existing behavior. 3. ci_cpplint (build/include_what_you_use): output_adapter_sink uses std::move; add #include <utility>. 4. ci_cuda_example (nvcc 11.8): NVCC's front end rejects the default template argument on the binary_writer alias template. Revert the alias to its original single-parameter form (relying on binary_writer's own defaulted OutputSinkType) and spell out the full type in the vector-sink convenience functions. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Encode big-endian numbers with a byte swap instead of std::reverse write_number() reordered multi-byte numbers for the big-endian formats (CBOR/MessagePack/UBJSON) with std::reverse over the byte array. GCC lowered only some sizes to a bswap; clang kept a scalar byte shuffle (0 bswap instructions in the CBOR number path). Replace the reverse with size-dispatched __builtin_bswap16/32/64 helpers (portable shift fallback for other compilers; std::reverse retained for exotic sizes such as a long double number_float_t). Codegen: the CBOR number path now emits bswap on both compilers (gcc 2 -> 16, clang 0 -> 4). Output is byte-for-byte identical to the previous implementation across the binary differential corpus. Throughput (isolated vs the std::reverse version, best of 9): CBOR int64 array gcc +7% clang +10% CBOR uint16 array gcc +27% clang flat Modest but consistent on number-dense encodings; negligible on string/blob-heavy output, as expected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Reserve output capacity up front for binary serialization The vector-returning to_cbor/to_msgpack/to_ubjson/to_bjdata/to_bson grew the output buffer purely by geometric reallocation. Reserving an estimate up front avoids the early reallocations, which is the dominant per-byte cost for array/object-heavy output. The estimate (binary_reserve_hint) is deliberately conservative and safe against untrusted input: it consults only the top-level element count (O(1), no walk of the DOM), guards the multiplication against overflow, and clamps the result to a fixed 1 MiB ceiling, so a large or hostile DOM can never force an oversized allocation here. The buffer still grows geometrically past the hint, so an underestimate only costs a few later reallocations; scalars/strings/binary are written in one shot and get no hint. Reserving capacity does not change the bytes produced. Throughput (g++/clang -O3, vs the previous commit): cbor int array +10% / +13% cbor object array +20% / +38% Output is byte-for-byte identical to develop across the binary differential corpus. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address review findings on the binary writer output sinks - binary_reserve_hint(): the 4-bytes-per-element estimate over-reserved by up to 4x for arrays of small scalars (CBOR encodes 0..23 in one byte), and the returned vector kept that capacity. Make the hint a strict lower bound on the encoded size instead, which also removes the 1 MiB clamp whose branch no test could reach (the largest container in the suite has 65793 elements). - Guard the -Wduplicated-branches pragma with __GNUC__ >= 7. The warning does not exist before GCC 7, so naming it made GCC 4.8/4.9/5/6 - which the CI matrix still builds - warn under -Wpragmas on every including translation unit, breaking downstream -Werror builds. - Constrain the adapter constructor of binary_writer with the enable_if its documentation already claimed, so a writer over some other sink type is no longer advertised as constructible from an output adapter. - Let output_vector_adapter wrap output_vector_sink rather than duplicating the append logic, so the type-erased and templated paths share one implementation. - Collapse the three copies of the memcpy/byte_swap/memcpy dance into a single byte_swap_buffer() helper, and add the MSVC _byteswap_* intrinsics so MSVC no longer falls back to the scalar shuffle this change exists to eliminate. - Add a vector_writer() helper for the five vector-returning to_* overloads instead of spelling out the writer type at each call site, and drop a dead default member initializer on output_adapter_sink. - New tests: the vector sink and the adapter sink must produce identical bytes for every format (the two to_* overloads no longer delegate to each other and could otherwise drift), and binary_reserve_hint() must never exceed the size actually written. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Route the -Wduplicated-branches pragma through Hedley Match #5485, which moved the binary writer's hand-rolled diagnostic pragmas onto JSON_HEDLEY_PRAGMA (merged into develop while this branch was open). The devirtualization's -Wduplicated-branches suppression in write_compact_float was the one raw '#pragma GCC diagnostic' left; it now uses JSON_HEDLEY_PRAGMA like the adjacent -Wfloat-equal line, still guarded to GCC >= 7 and non-clang (the warning exists only there). Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Niels Lohmann
|
0b20b7e622 |
Reject MessagePack/BSON binary subtypes that don't fit their wire format (#5469)
* Reject MessagePack/BSON binary subtypes that don't fit their wire format Both formats store byte_container_with_subtype's subtype (a uint64_t) in a single byte. The writers cast to std::int8_t/std::uint8_t without a range check, so subtypes above 255 were silently truncated modulo 256 instead of raising an error. Throw out_of_range.413 instead when the subtype exceeds the representable range of 0-255. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Move the new binary-subtype regression test out of unit-regression2.cpp unit-regression2.cpp is already at the edge of what the MinGW linker can relocate; adding this test's ~26 lines tips test-regression2_cpp20 (clang, Windows) over into "relocation truncated to fit: IMAGE_REL_AMD64_REL32 against `.rdata'" (see |
||
|
Niels Lohmann
|
c41152e620 |
Fall back to plain-object encoding when to_bjdata()'s _ArrayType_ annotation is not a string (#5494)
* Fall back to plain-object encoding when _ArrayType_ is not a string write_bjdata_ndarray() looked up _ArrayType_ by calling get<string_t>() directly, which throws type_error.302 when the annotation is not a string (e.g. a number, null, boolean, array, or object). Per the documented BJData ndarray contract, an object only qualifies for the compact ndarray encoding if _ArrayType_ names a known type; anything else must fall back to plain-object encoding, the same way an unknown type-name string already does. Add an is_string() check before the get<string_t>() call so a non-string _ArrayType_ takes the existing "unrecognized type name" fallback path instead of throwing. Fixes #5398. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Relax the BJData fuzzer's round-trip check from byte-exact to value-exact Fixing #5398 lets to_bjdata() proceed past the object it used to reject, which exposed a pre-existing, unrelated round-trip quirk to the fuzzer: a binary_t value serialized through the non-optimized ("$U#"-less) array encoding is parsed back as a plain array of numbers, since from_bjdata() has no way to tell "array of uint8 numbers" apart from "array of bytes" without that optimized header. Re-serializing that plain array then goes through the generic smallest-type writer, which - unrelated to this PR, and long predating it - prefers the 'i' (int8) marker over 'U' (uint8) for values that fit both, so the re-encoded bytes can differ from the original even though both decode to the same value. This is not introduced by the #5398 fix; the same divergence reproduces from a bare json::binary_t value with no _ArrayType_ annotation involved at all, on the commit immediately preceding it. A general fix would mean changing the shared UBJSON/BJData smallest-type selection that hundreds of existing tests pin to 'i' for small positive integers, which is out of scope and too risky for this PR. Update fuzzer-parse_bjdata.cpp's round-trip assertions to check that re-serializing is value-stable (from_bjdata(to_bjdata(j)) == j) rather than byte-exact, matching the guarantee BJData actually provides, and add a regression test in unit-bjdata.cpp using the exact OSS-Fuzz input that documents the behavior. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Compare dump()s instead of json values in the BJData fuzzer's round-trip check The value-stability assertion added to fix the earlier OSS-Fuzz crash (json::from_bjdata(to_bjdata(j2)) == j2) itself broke on a NaN payload: IEEE 754 NaN is never equal to itself, so operator== reports two structurally-identical trees containing a non-finite double as different -- not a round-trip bug, just NaN's ordinary non-reflexivity. dump() serializes any non-finite double the same deterministic way (as JSON null, since JSON cannot represent NaN or Infinity), so comparing dumps is stable under exactly the values that break operator==. Verified against both the original OSS-Fuzz crash input and the new one (0x68 0x68 0x7c, which decodes to a NaN), plus a local 2.5M-case random-input sweep with no failures. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
502e9d66f6 |
Fix to_bjdata() emitting the Draft-3-only 'B' marker in default Draft-2 mode (#5479)
* Fix to_bjdata() emitting the Draft-3-only 'B' marker in default Draft-2 mode _ArrayType_ = "byte" mapped unconditionally to the BJData type marker 'B', regardless of the requested bjdata_version. 'B' is defined only by BJData Draft 3; with the default version (draft2), this produced a stream that is invalid for Draft 2 and, unlike every other _ArrayType_, round-tripped back as a binary value instead of the original annotated object. Only accept "byte" / emit 'B' when bjdata_version selects Draft 3. Under Draft 2, fall back to the same plain-object encoding used elsewhere in this function for other invalid-annotation cases, so the value round-trips correctly. Fixes #5404. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Future-proof the Draft-3-only 'B' marker gate @gregmarr pointed out that dtype == 'B' && bjdata_version != draft3 only future-proofs by accident, since bjdata_version_t currently has exactly two values. Compare with < instead, so a later draft that keeps the 'B' marker valid does not need this gate revisited. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels LohmannandClaude Opus 5
|
c72f37a40d |
Support custom object/array types and improve template parameter handling (#5443)
* docs: document the implicit requirements on basic_json's template parameters The requirements that basic_json places on its eleven template parameters were only implied by how the library uses the resulting object_t, array_t, string_t, etc. Consumers had to discover them by trial and error. Add "Template Parameter Requirements" collecting them, split into what is always required and what is only required when a particular part of the API is instantiated. Notable findings that were previously undocumented: - ObjectType must provide a key_compare member type (actual_object_comparator names object_t::key_compare in both arms of a std::conditional), and its third template parameter is used as a comparator, so std::unordered_map cannot be used without a wrapper. - ArrayType must provide capacity() -- push_back(), emplace_back(), operator+=(), and operator[](size_type) call it unconditionally -- and needs random-access iterators, so std::deque and std::list do not work. - StringType needs contiguous, null-terminated data(), a one-byte value_type, and either assignability from std::to_string or an ADL int_to_string(). - NumberFloatType must be float, double, or long double for parsing and serialization; the integer types must satisfy std::is_integral. - AllocatorType must be stateless, support incomplete types, and use plain pointers. - BooleanType and the number types are union members and must be trivial. Link the new page from the basic_json overview, the types feature page, and the individual type alias pages, and correct the container examples given for ObjectType (std::unordered_map) and ArrayType (std::list), which do not work. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018hxZxz8svM54c6ATEvXp5E Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix object_comparator_t for object types without key_compare detail::actual_object_comparator selected between object_t::key_compare and default_object_comparator_t with std::conditional. Both type arguments of std::conditional are named eagerly, so object_t::key_compare had to exist regardless of the condition, and the has_key_compare guard added in 3.11.0 never took effect: any ObjectType without a key_compare member type failed to compile while instantiating basic_json itself. Use detected_or_t instead, which resolves through a SFINAE partial specialization and only names object_t::key_compare when it exists. The selected type is unchanged for every object type that compiled before, so object_comparator_t -- a public member type -- keeps its meaning and ABI. has_key_compare had no other users and is removed. Add a regression test using an adapter around std::unordered_map, which has no key_compare; it fails to compile without this change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018hxZxz8svM54c6ATEvXp5E Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: list the types that are known to work for each template parameter Follow up on the template parameter requirements page: state, for every template parameter, which concrete types work and where they stop working. Each entry was verified by compiling and running a common workload (DOM access, dump, parse, CBOR/MessagePack round-trip, flatten, hash) against that instantiation. Findings worth calling out: - ObjectType no longer needs a key_compare member type, so the std::unordered_map adapter only has to restore the template argument order. A hash-ordered ObjectType works everywhere except unflatten(), which reconstructs an array only when it meets the reference token 0 before the other indices. - ArrayType: std::deque works when wrapped to add capacity(); std::list does not. - StringType: std::pmr::string and std::basic_string with a custom allocator compile for the DOM, dump, and parse, but not for the binary readers, flatten, or diff, because the library assigns std::string values to string_t and int_to_string cannot be overloaded for a type in namespace std. - NumberFloatType: long double works for dump and parse but not for the binary formats, which have no encoding for it. - BinaryType: std::vector<std::byte> supports assignment, get, and the binary formats, but neither dump nor std::hash<basic_json>. Also record the object_comparator_t fix in its version history. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018hxZxz8svM54c6ATEvXp5E Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix unflatten and binary dumping for non-default configurations unflatten() decided between array and object by looking at the first reference token it happened to see for a node: it started an array only when that token was 0. With a sorted object type the token 0 always arrives first, so the result was correct by accident; with an object type whose iteration order is unspecified, {"/c/2":3,"/c/1":2,"/c/0":1} unflattened to an object with the keys "0", "1", and "2" instead of an array. Collect the pointer prefixes that have a reference token 0 among their children before building the result, and let get_and_create() consult that set. The outcome is now independent of the iteration order and matches, for every input, what a sorted object type produced before: a value is restored as an array if and only if one of its keys is 0. Iterating the flattened object in a different order would have been simpler, but it would have changed the key order of the result for insertion-ordered object types. The serializer, std::hash, and the UBJSON writer converted the elements of a binary value to an integer implicitly, which does not compile for a BinaryType whose value type is std::byte, and which made dump() write the bytes of a signed value type as negative numbers. Convert to std::uint8_t explicitly in all three places, so every byte type dumps as 0..255. The default std::vector<std::uint8_t> configuration is unaffected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018hxZxz8svM54c6ATEvXp5E Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: note which Abseil containers can be used as template arguments Checked against Abseil release 20250127.0 with the same workload as the other entries on the page (DOM access, dump, parse, CBOR/MessagePack/UBJSON round-trip, flatten, hash), with and without JSON_DIAGNOSTICS. absl::flat_hash_map and absl::node_hash_map work as ObjectType through an adapter that restores the template argument order and makes erase(iterator) return the following iterator, which Abseil's returns as void. The page now carries that adapter, and notes that absl::flat_hash_map does not keep references to the mapped values valid across insertions while absl::node_hash_map does. Both have a capacity() member, so JSON_DIAGNOSTICS already refreshes the parent pointers conservatively for them. absl::btree_map and absl::InlinedVector cannot be used at all: object_t and array_t are formed while basic_json is still incomplete, and both inspect their value type at class scope. std::map and std::vector are required by the standard to tolerate this, third-party containers generally are not, so the page states the constraint on its own rather than only per container. absl::InlinedVector does work as BinaryType, where it is instantiated with a complete type. absl::FixedArray and absl::Cord are not usable. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018hxZxz8svM54c6ATEvXp5E Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Relax the ArrayType and ObjectType requirements Two requirements forced users of otherwise suitable containers to write a wrapper, and neither was load-bearing. array_t::capacity() was read in push_back(), emplace_back(), operator+=(), and operator[](size_type), but set_parent() only looks at the value under JSON_DIAGNOSTICS; without diagnostics it was computed and discarded. Read it through array_capacity(), which reports unknown_size() when diagnostics are off or when the array type has no capacity() at all, and treat an unknown capacity as "the elements may have moved" so the parent pointers are refreshed conservatively. std::deque now works as ArrayType, in both builds, and capacity() is no longer named at all in a default build. Since the capacity is now only meaningful for array insertions, it moves out of set_parent() into set_parent_after_array_insert(). basic_json::erase(iterator) assigned the object's erase() return value, which requires the container to return the following iterator. Abseil's hash maps return void to avoid computing a successor the caller may not need. Detect that and compute the successor before erasing; containers that return an iterator, including the vector-backed ordered_map where a precomputed successor would be wrong, keep the existing path. Together these leave an Abseil hash map needing only an alias that restores the template argument order, and no adapter at all for std::deque. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018hxZxz8svM54c6ATEvXp5E Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Do not require string_t to be convertible from std::string Three places built a std::string and handed it to something expecting a string_t: the UBJSON high-precision number reader, which every binary reader instantiates, and the BSON writer's array element size calculation and write. That silently required string_t to be implicitly convertible from std::string, which std::string itself and types with a string_view conversion satisfy, but many string types do not. Construct the string_t explicitly from the data and size, which the requirements already cover. This makes boost::container::string, eastl::string, std::pmr::string, and std::basic_string with a custom allocator work as StringType, none of which could previously be used with any binary format. Add binary format coverage to the alt_string test, which had none, including a UBJSON high-precision number -- the case that goes through the reader path. BSON stays uncovered there: it additionally needs string_t::find(value_type), which alt_string does not provide. Also record which containers from Boost, Abseil, and EASTL work for each template parameter, and correct two claims: std::pmr::string is usable after this change, and tsl::ordered_map is not usable at all, because its iterators expose the mapped value as const while basic_json modifies it in place. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: record compatibility for the common header-only hash maps ankerl::unordered_dense (map and segmented_map), phmap (flat_hash_map and node_hash_map), and robin_hood::unordered_flat_map all work as ObjectType through the same adapter as Abseil's and Boost's hash maps, which only has to restore the template argument order. phmap::btree_map and robin_hood::unordered_node_map do not: like the other btree containers they require a complete value type. Note that none of these hash maps defines key_compare, so every one of them depends on object_comparator_t falling back to default_object_comparator_t. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: record Folly and the remaining vector replacements Folly works, with the caveat that its headers need C++20: folly::fbstring as StringType, folly::fbvector and folly::small_vector as ArrayType, folly::fbvector<std::uint8_t> as BinaryType, and folly::F14NodeMap as ObjectType through the usual argument-order adapter. folly::F14FastMap is the exception and requires a complete value type. For ArrayType, boost::container::devector, boost::container::static_vector (within its fixed capacity), and std::pmr::vector work as well. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: cover fifo_map, gtl, folly::sorted_vector_map, and Qt nlohmann::fifo_map works through the adapter that has always been documented for it, and preserves the insertion order. Restore its mention in the object order page, which was dropped together with the tsl::ordered_map one: unlike ordered_map it keeps a lookup index, so it is the insertion-ordered option without the quadratic cost. gtl::flat_hash_map and folly::sorted_vector_map work as well, the latter through an alias that drops the allocator, whose value type it disagrees on. gtl::btree_map does not, for the same reason as the other btree containers. None of the Qt containers can be used, each for its own reason: QMap has no value_type, QHash iterators yield the mapped value rather than a pair, QList has no max_size(), QByteArray spells empty() as isEmpty(), and QString is UTF-16. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: qualify the std::pmr::string support claim Listing std::pmr::string as fully supported was an overclaim: it was only ever checked with the default memory resource, which is not what PMR is for. basic_json cannot be given an allocator or a memory resource, so a pmr string inside a value always allocates from std::pmr::get_default_resource(), and assigning an arena-backed string into a value silently drops its resource, because polymorphic_allocator does not propagate on copy construction. Passing polymorphic_allocator as AllocatorType does not compile either. Only the process-global set_default_resource() redirects these allocations. Say so, and separate the row from std::basic_string with a custom stateless allocator, which is unaffected. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: remove a duplicated StringType compatibility section The StringType section carried two 'Compatible types' tables and two copies of the reference-implementation tip. The second table was a stale copy from before the binary format string fixes and still listed std::pmr::string and std::basic_string with a custom allocator as unusable, contradicting the corrected table a few lines above it, and it dragged along the old explanation that blamed int_to_string. Drop the stale copy and put the surviving table before the notes, so the 'see below' in the std::pmr::string row points forwards. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: correct the template parameter requirements after independent verification Every claim on the page was re-checked by compiling and running it, including the rows that say a type cannot be used, which were checked to fail for the documented reason and not merely to fail. Twenty-four claims were wrong. The most consequential: the incomplete-type constraint applies to ObjectType only. object_t is instantiated inside the class definition, because it is probed for key_compare; array_t is only named there and is not instantiated until basic_json is complete. So eastl::vector, QList and QVector are not excluded by incomplete types at all -- they simply have no max_size() -- and absl::InlinedVector is excluded for a subtler reason of its own. Further corrections: ObjectType does not need erase(key), which has a fallback, but does need at(key) for UBJSON output; only == and < are used, or == and <=> under C++20, not all six; the documented adapter does not fit ankerl or robin_hood. ArrayType needs no initializer-list insert, and value_type, the (count, value) constructor and swappability are per-function, not always. BinaryType needs a range insert for CBOR indefinite-length byte strings and does not need push_back. StringType needs append(const StringType&) unconditionally, and does not need operator!= or operator== against const char*; empty(), resize(n) and reserve(n) are per-subsystem; int_to_string is needed by diff, items and std::hash rather than by JSON Pointer or flatten. BooleanType must be implicitly convertible from bool, and JSONSerializer's second parameter need not carry a default. std::pmr::string was wrong in the other direction this time: a moved-in string does keep its memory resource, and later growth allocates from it. Only copies land on the default resource. Five requirement violations are not caught at compile time rather than the two the page claimed; they are now listed together up front. Split every compatibility table into what works and what does not, as the reasons in the second half are the useful part. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Reduce the string_t and array_t members the library requires Several members were required only because of how the library happened to be written, not because the functionality needs them. Dropping them widens the set of usable string and array types, and one of them was also a performance problem. string_t: - c_str() is gone. Every call site already knew the length and passed it along, so data() is enough. The one place that did not, the diagnostics path in exceptions.hpp, now builds the token from data() and size(), which also stops it from truncating keys that contain a null byte. - back() is gone; the serializer indexes the last character instead. - find(str, pos), replace(), and substr() are gone. escape() and unescape() rebuilt the string with one replace() per escaped character, which moves the tail every time: escaping a string of n characters that all need escaping cost O(n^2). Both now scan with find_first_of() -- a member the pointer parser already required -- and append whole runs, so the common case is one search and one copy. Escaping 64000 tildes drops from 717 ms to 20 ms; a string with nothing to escape gets faster too (8.4 ms to 5.8 ms), because the scan is still a single memchr per pass. json_pointer::split() takes its reference tokens with the (const char*, size_type) constructor rather than substr(). - json_pointer::to_string() accumulates with concat<string_t> instead of letting concat default to std::string and converting afterwards, so streaming a json_pointer no longer requires string_t to be assignable from a std::string. array_t: - at(size_type) is gone. basic_json::at(size_type) checked the index by calling array_t::at() and translating std::out_of_range, which also required the array type to throw that exact exception. It now compares against size() and uses operator[]. The thrown exception, its message, and the behaviour under JSON_NOEXCEPTION are unchanged. The BSON writer wrote the terminating null byte out of the string's own buffer (size() + 1). It now writes the byte itself, so string_t::data() need not be null-terminated for to_bson(). The tests pin the reduced API: alt_string loses the five dropped members and gains coverage of the escaping paths, and a std::vector whose at() is hidden is used as an ArrayType. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: record the reduced string_t and array_t requirements Drop c_str(), back(), find(str, pos), replace(), and substr() from the StringType requirements and at(size_type) from the ArrayType ones, and note the string assignment the JSON pointer code performs. Streaming a json_pointer no longer needs assignability from a std::string. Add the non-null-terminated data() to the list of violations that are not diagnosed at compile time -- it was described in the StringType section but missing from the summary at the top -- and correct the QString row, which no longer fails for the c_str() it lacks. JSON_CATCH_USER no longer wraps a catch of std::out_of_range: the last one went away with array_t::at(). Describe what the library actually catches. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Use character literals for the signed BinaryType test MSVC rejects char(0xFF) with C4310 (cast truncates constant value), which the Windows workflow treats as an error. The character literals carry the same byte values without a narrowing cast. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Do not instantiate a hash map with an incomplete basic_json in the tests object_t is probed for key_compare inside the definition of basic_json, so it is instantiated while basic_json is still incomplete. Whether a hash map survives that depends on the standard library: libstdc++ 9 needs the size of the mapped type to instantiate std::unordered_map's node type and rejects the adapter, which broke the GCC 9 builds. The test now derives its no-key_compare object type from std::map -- which does cope -- and shadows the inherited key_compare member type with an entity that is not a type, so the library's probe finds none, exactly as for a hash map. The unflatten() order-independence checks in unit-json_pointer already cover the behaviour that the unordered object type was there for. The limitation is documented for std::unordered_map. Also address two Clang-Tidy findings the earlier commits introduced: erase_from_object() declares its iterator with auto, and at(size_type) checks the type first and then falls through to the return instead of throwing from an else branch. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Keep diagnostic key paths null-terminated Building the token from data() and size() kept an embedded null byte in the key, and since what() hands out a C string, that truncated the whole message rather than just the key: to_bson() on a key containing U+0000 reported "[json.exception.out_of_range.409] (/en" instead of the full explanation. This broke test-bson under JSON_DIAGNOSTICS. Constructing from data() alone stops at the first null byte, which is what c_str() did before, so the message is unchanged -- without requiring string_t to provide c_str(). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Do not parse the value in the array-at() test JSON_DIAGNOSTIC_POSITIONS adds the byte range of the value to the exception message, which a parsed value has and an in-memory one does not, so the two message checks failed in that configuration. Build the array in memory instead of parsing it; the test is about at(size_type) not needing array_t::at(), and the byte range is beside the point. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Move the custom BinaryType tests into their own translation unit The two sections added to unit-regression2.cpp brought a third full basic_json instantiation into a translation unit that was already large. With Clang on MinGW that pushed the object over the reach of a 32-bit relocation and test-regression2_cpp20.exe failed to link: relocation truncated to fit: IMAGE_REL_AMD64_REL32 against `.rdata' unit-regression2.cpp is restored to exactly what it was before, and the coverage moves to unit-custom-binary-type.cpp, next to the object and array type tests it belongs with. The signed value type is now also covered in C++11, where std::byte is not available. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Do not require the container iterators to be nothrow move constructible iter_impl declared its defaulted move operations noexcept. The exception specification a defaulted function gets implicitly follows from its members, here internal_iterator, which holds the object and array iterators. libstdc++ gives std::deque's iterator a user-provided copy constructor without noexcept before version 11, so the implicit specification is noexcept(false) and does not match the declared one. That deletes the function -- and with g++ 4.8, which predates CWG 1778, it is an error outright: error: function 'iter_impl<basic_json<std::map, std::deque> >::iter_impl( iter_impl&&)' defaulted on its first declaration with an exception-specification that differs from the implicit declaration So std::deque, which this branch documents as a usable array type, could not be used with an older standard library. Leaving the specification to be computed cannot mismatch; iteration_proxy_value already spells out the same condition next door. The default configuration is unaffected: json::iterator, json::const_iterator and ordered_json::iterator stay nothrow move constructible and move assignable, which the test now checks so it cannot regress unnoticed. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address two Clang-Tidy findings the custom container tests exposed Both come from instantiating basic_json with containers other than the default ones, and neither shows up with the Clang-Tidy version available outside CI: - insert(const_iterator, basic_json&&) forwards its by-value iterator to the const-reference overload. performance-unnecessary-value-param asks for the copy to be a move; it only fires for an iterator that is not trivially copyable, as std::deque's is not. The NOLINT on the function does not cover it, because the finding is reported where the parameter is used rather than where it is declared. Move it, which is what the check asks for and is a (very small) improvement in its own right. - cppcoreguidelines-use-enum-class rejects the unnamed enum that shadowed the inherited key_compare member type. An enum class would not do, since it declares a type of that name and the probe would find it again; a member function declaration hides the name just as well. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Assert the iterators' exception specification relative to the container The test pinned that nlohmann::json's iterators stay nothrow movable after iter_impl's defaulted move operations lost their declared noexcept. That is not a property of the library, though: the exception specification is now computed from the container iterators, so it holds only for standard library implementations whose iterators are themselves nothrow movable. MSVC's checked iterators before VS2017 are not -- _Iterator_base12 registers the iterator with the container's debug proxy in a copy constructor that carries no noexcept -- so the assertions fail on a Visual Studio 2015 debug build, which is the one debug configuration in the AppVeyor matrix and has no counterpart in the GitHub Actions matrix. Assert what the change actually guarantees instead: the iterators are nothrow movable exactly when the object and array iterators they are built from are. That still pins the default configuration against a silent regression, and it is true whatever the standard library provides. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Detect a void-returning erase() through a named trait erase_from_object() distinguished its two overloads with a decltype of a member call written inline in a default template argument. Every other detection in the library goes through the detector machinery in detected.hpp instead -- has_erase_with_key_type is the same question about the same member function -- and the inline form is the one shape older compilers are least reliable about. Express it the same way: detect_erase_with_iterator plus is_detected_exact, both of which the library already relies on elsewhere. No behaviour changes. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Give the custom container types only the constructors the library uses The three container types in the new tests inherited every constructor of their base with using Base::Base. That asks for more than the test needs: the library builds an object or an array by default construction, by copy or move, and -- when converting between two basic_json types or from an initializer list -- from an iterator range. Declaring those directly makes the requirement visible in the test, and keeps object types out of a corner where a compiler has to declare std::map's whole constructor set for a derived class while basic_json is still incomplete. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Temporarily disable the new custom container tests AppVeyor is the only CI that builds MSVC 2015 and 2017, and it has now rejected three heads of this branch. Its build log is not reachable from where this is being worked on, so the verdict is a single bit and the cause has to be narrowed down by bisection. Everything else stays: the library changes, the reduced alt_string, and the unflatten() tests. If AppVeyor passes with these three translation units disabled, the cause is one of the six basic_json instantiations they add; if it fails, it is in the library. Either way this commit is reverted. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Guard the disabled tests with a macro rather than #if 0 Clang-Tidy's readability-avoid-unconditional-preprocessor-if rejects a literal #if 0. Use a macro that is never defined instead, which the check does not look at. Still temporary, and reverted together with the previous commit. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Re-enable the array and binary container tests AppVeyor passed with all three new translation units disabled, so the library changes, the reduced alt_string, and the unflatten() tests are fine on MSVC 2015 and 2017; the cause is one of the six basic_json instantiations the new tests add. Bring back two of the three. If AppVeyor passes again, the cause is in unit-custom-object-type.cpp, which is the one still disabled; if it fails, it is in one of these two and needs one more split. Still temporary. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Diagnose two silently violated template parameter requirements Both were on the list of requirements that are not caught at compile time and corrupt values rather than failing, and both are a plain size comparison: - A BinaryType whose value_type is wider than one byte, which the readers and writers reinterpret as raw bytes anyway. - A NumberUnsignedType too narrow to hold the absolute value of every NumberIntegerType value, which makes basic_json(INT64_MIN).dump() yield -0 for std::int64_t with std::uint32_t. Neither static_assert rejects a configuration that worked before: both only fire where the result was already wrong. Also add the two comments the review asked for, in write_bson_string() and calc_bson_array_size(), matching the ones their counterparts already carry. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Align the template parameter tables and record what is now diagnosed Every table in the page is reformatted so each column is exactly as wide as its widest cell, which is what the review asked for in a dozen places: the separator rows that ran two dashes long, the stray spaces, and the columns padded well past their content. The row listing six containers that require a complete mapped type is split in two so that one cell no longer sets the width of the whole table. Content changes: NumberUnsignedType is described as any unsigned integer type at least as wide as NumberIntegerType rather than any unsigned integer type; the two requirements that are now static_asserts move out of the list of violations that are not caught at compile time; and the two places that require a non-const operator[] say why data() will not do (std::string has no non-const data() before C++17). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Bisect the other way: only the object container tests The previous head touched only docs/, which AppVeyor's only_commits filter skips, so it produced no build and no status at all -- the pull request looked green without ever having been built on MSVC 2015 or 2017. Swap the guards instead of repeating that step: unit-custom-object-type.cpp is enabled and the array and binary translation units are disabled. AppVeyor already passed with all three disabled, so a failure here pins the cause on no_key_compare_json or void_erase_json, and a pass pins it on the array or binary file. Still temporary. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Split the two object types apart AppVeyor failed with only unit-custom-object-type.cpp enabled and passed with all three new translation units disabled, so the cause is one of the two object types in this file and not the array or binary ones. Guard out void_erase_map and leave no_key_compare_map, which separates the two constructs under suspicion: shadowing the inherited key_compare member type with an entity that is not a type, and hiding the inherited erase with a void-returning overload. A failure here points at the first, a pass at the second. Still temporary. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Build the no-key_compare object type by composition, not inheritance The "object type without key_compare" test failed on AppVeyor's MSVC 2017 jobs (/std:c++17): its no_key_compare_map derived publicly from std::map and shadowed the inherited key_compare type with a same-named member function, relying on ordinary member hiding to make key_compare unreachable as a type for the library's detection trait. MSVC 2017 does not honor that hiding for a typename-qualified lookup performed from outside the class and still resolves key_compare to the base's comparator type, so object_comparator_t incorrectly picked it up instead of falling back to default_object_comparator_t. Wrapping a std::map by composition instead removes the base class entirely, so there is no key_compare to find under any lookup rule, on any compiler. Also drops the now-unneeded JSON_BISECT_CUSTOM_CONTAINER_TESTS guard left over from narrowing this down: the void_erase_map test in the same file was never the cause and is re-enabled unconditionally. Verified locally with clang++ and g++ under C++17 and C++20. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Re-enable the array and binary custom-container tests unit-custom-array-type.cpp and unit-custom-binary-type.cpp were still guarded behind JSON_BISECT_CUSTOM_CONTAINER_TESTS from bisecting the AppVeyor failure fixed in |
||
|
Niels Lohmann
|
44f8ec30e9 |
Bound UBJSON optimized arrays of a valueless type (#5504)
* Bound UBJSON optimized arrays of a valueless type An element of type 'Z' (null), 'T' (true) or 'F' (false) is encoded by its type marker alone, so an optimized UBJSON array of one of those has no payload: reading an element consumes no input at all. Its declared count is therefore the only thing that decides how much is allocated, and nothing bounded it. "[$Z#l" and a four-byte count is nine bytes of input describing two billion values; #2793 reports 35 GB and 150 seconds from ten bytes, and OSS-Fuzz has an out-of-memory and a timeout report for the same shape. Every other type costs at least one byte per element, so the end of the input bounds it. 'N' (no-op) is already skipped rather than stored. Objects are not affected either: each element is preceded by its key, which costs bytes. And BJData already refuses these markers as an optimized type, so this is a plain UBJSON matter. Reject a count above 1,048,576 elements for those three types with out_of_range.408, the code this reader already uses for a declared size it will not honour. The check runs before the SAX start event, so no container is opened and then abandoned. Rejecting on the read side alone would break the guarantee that anything to_ubjson() writes can be read back, and would trip the round-trip assertion in fuzzer-parse_ubjson.cpp. So the writer falls back to the unoptimized encoding, one byte per element, for arrays of these types above the same limit. Its decision depends only on the array's size, which is identical for a value and for anything parsed back from it, so the round trip is stable. No existing test changes: the largest such count in the test suite is 65,793. The excessive-size test that already used this shape still passes, now rejected a little earlier than by the max_size() check it used to reach. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Note the 1,048,576 valueless-array limit as (1 << 20) in the docs Addresses review feedback from @gregmarr on PR #5504: spell out the binary/hex form next to the decimal count so it reads as the round power-of-two it is, matching how include/nlohmann/detail/input/binary_reader.hpp defines max_valueless_container_size. Applied in both docs/exceptions.md and ubjson.md, as requested. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels LohmannandClaude
|
da42a627dc |
Stop dump() from heap-allocating its output adapter per call (#5449)
* Stop dump() from heap-allocating its output adapter per call The serializer held its output sink as output_adapter_t<char> (a std::shared_ptr<output_adapter_protocol<char>>), which dump() and operator<< built via make_shared -- one heap allocation per call for a sink that only wraps a reference to the caller's string or stream. Hold the sink as a non-owning output_adapter_protocol<char>* instead and construct the concrete adapter on the stack at the call site. The write path (o->write_characters) is unchanged, so output is byte-for-byte identical; a compact dump() of a small object drops from 2 heap allocations to 1 (only the returned string remains), ~3% faster. Completes the per-call allocation cleanup on this branch, which already removed the indent_string buffer (both were reported in #5413). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01L1oJ2ggRHS37zeVe94QTA1 Signed-off-by: Claude <noreply@anthropic.com> * Take the output adapter by reference at the serializer ctor Per review: the serializer still holds the adapter as a non-owning pointer, but the constructor now takes output_adapter_protocol<char>& and takes its address internally, so every call site passes a reference. A reference cannot be null and reads as a borrow, which makes the lifetime contract harder to get wrong than handing over a raw pointer. The stored member and the write path are unchanged. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Claude <noreply@anthropic.com> Signed-off-by: Niels Lohmann <mail@nlohmann.me> Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Niels LohmannandClaude Opus 4.8
|
ff80ed3295 |
Speed up dump(), and keep it from overflowing the stack (#5285)
* Add SWAR bulk fast path to string serialization (dump_escaped) When ensure_ascii is false, dump_escaped previously ran every byte of every string and object key through the UTF-8 DFA decoder, even for the common case of ordinary text with nothing to escape. This mirrors the per-byte cost the parser had before the contiguous fast paths. At a character boundary, bulk-copy the longest run of bytes that need no escaping using string_bulk_run() - the same SWAR scanner and UTF-8 bulk validator the lexer's contiguous path uses - and only fall back to the byte-at-a-time DFA loop for the first byte that needs individual handling (a quote, backslash, control character, or ill-formed/truncated UTF-8). Because every "hard" or invalid byte is still processed by the unchanged byte path, escaping output and error handling (including strict-mode error 316 position and message) are byte-identical to before. The ensure_ascii=true path is unchanged: it must escape non-ASCII and 0x7F, which string_bulk_run does not stop on, so a separate predicate would be needed for it. Verified byte-for-byte identical dump output against the pre-change implementation across ~20k randomized byte strings plus curated edge cases (all escapes, control chars, valid multibyte, surrogates, overlong, truncated sequences) for both ensure_ascii settings and all three error handlers, in C++11/17/20 at -O2/-O3. Throughput (g++ -O3, ensure_ascii=false, vs pre-change): long ASCII strings 4.2x twitter-like objects 2.3x dense CJK 1.4x (further headroom with JSON_USE_SIMDUTF) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Buffer serializer output and add ensure_ascii string fast path Two further serialization speedups on top of the ensure_ascii=false bulk copy, both reusing the SWAR primitives in detail/input/string_scan.hpp. 1. Internal write buffer (devirtualization). Every structural character ('{', '"', ',', ...) previously went straight to the output adapter through a virtual call. Route all writes through put_char/put_chars into a 1 KiB buffer that flushes in bulk; the public dump() flushes once the top-level value is done (the recursive worker is split out as dump_internal). Runs larger than the buffer are written straight through, so large payloads are not copied twice. This is the dominant cost for object/array-heavy values. 2. ensure_ascii fast path. dump_escaped previously ran the UTF-8 DFA over every byte when escaping non-ASCII. Add find_ascii_copyable_run() (a SWAR scan stopping at '"', '\\', < 0x20, 0x7F, and >= 0x80) so runs of printable ASCII are bulk-copied, with the byte path handling each escape/non-ASCII byte exactly as before. Behavior is unchanged: dump output is byte-for-byte identical to the previous implementation across ~20k randomized byte strings plus curated edge cases (all escapes, control chars, 0x7F, valid multibyte, surrogates, overlong, truncated), for object/array/pretty output, both ensure_ascii settings, and all three error handlers, in C++11/17/20 at -O2/-O3. New unit tests cover the buffer flush boundaries, the escape and 0x7F handling, multibyte under both settings, and invalid-UTF-8 handling. Throughput (g++ -O3, vs the ensure_ascii=false-only baseline): long ASCII, ensure_ascii=0 4.2x long ASCII, ensure_ascii=1 4.1x twitter-like objects 2.7x dense CJK 1.8x Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Flush serializer buffer in dump_escaped unit test test-convenience failed (macOS finished first; the failure is platform-independent) because check_escaped() calls the internal serializer::dump_escaped() directly and then reads the output stream. Since dump_escaped() now writes into the serializer's internal write buffer, the bytes were still buffered and the stream was empty. Expose flush() under JSON_PRIVATE_UNLESS_TESTED (same visibility as dump_escaped) and flush in check_escaped() before inspecting the output. Per-string flushing inside dump_escaped() was rejected on purpose: it would defeat the buffering that makes object/array-heavy dumps faster. Library behavior is unchanged (flush()'s body is identical; only its access label moved). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Avoid deep recursion in serialization write-buffer test The "many small structural writes exceed the write buffer" subcase built a 1100-deep nested array and dumped it to force >1024 consecutive single-character writes through put_char (exercising the write buffer's flush-when-full branch). dump() recurses per nesting level, so on MSVC debug builds (smaller default stack, larger frames) this overflowed the stack and crashed test-serialization; Linux/macOS have enough headroom to hide it. Replace the nesting with a flat array of 500 empty strings. Each element emits '"', '"', ',' via put_char, so the dump is a long run of single-character writes (1501 bytes > the 1024-byte buffer) at nesting depth two, hitting the same flush branch without deep recursion. Library code is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Split the write-buffer helpers and write indentation directly Follow-up to @gregmarr's review: put_chars() was doing four unrelated jobs, so give the two that can be made safe their own entry points. - put_literal(): takes the literal by reference and deduces the length from the array bound, so the 27 hand-counted lengths at the call sites can no longer drift from the literals they describe. A literal is checked at compile time to fit the buffer, so this path needs no write-through branch. - put_buffer(): takes the fixed-size buffer itself rather than a bare pointer, so the length can be checked against the buffer's own bound. - put_indent(): memsets the indentation into the write buffer, filling and flushing it as needed. This removes indent_string entirely, and with it both bugs of #5186: the indentation string was grown by doubling, which is not enough when indent_step more than doubles it (a heap over-read - dump(2000) read 2000 bytes out of a 1024-byte string), and the grown part was filled with a space instead of the configured indent_char. next_indent() keeps that PR's assertion against the unsigned indentation accumulation wrapping on deep nesting. put_chars() keeps the two cases that are genuinely a pointer and a count: the run-length copies out of the string being escaped, and to_chars() output. Tests cover an indent_step wider than the write buffer, a non-space indentation character past the old growth point, and nesting whose accumulated indentation spans several buffer-fulls. All three fail against develop. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fill the indentation buffer once instead of once per flush @gregmarr's point on the fill-and-flush loop: flushing does not disturb what the write buffer holds, so an indentation spanning several buffer-fulls only has to be written into the buffer once and can then be handed to the adapter as many times as needed. The loop re-filled it every time, doing work it already knew was there. put_indent() now fills the room left in the buffer, and if anything remains, flushes, fills the buffer once, and re-flushes that same content. It also returns early for a zero-width indentation, which is what the closing brace of every outermost value asks for. Measured over a dump(), counting memset calls and bytes inside put_indent: indent before after 4 1 call / 4 B 1 call / 4 B 2000 2 calls / 2000 B 2 calls / 2046 B 100000 98 calls / 100000 B 2 calls / 2046 B The wide case is now constant work rather than proportional to the indentation width; ordinary widths are unchanged. Tests extended to cover several whole buffer-fulls and an exact multiple of the buffer size. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Tighten the write-buffer helpers after review More of @gregmarr's review on the put_* split: - Reattach the put_chars() doc comment, which the new helpers had been inserted in front of, leaving it describing put_indent(). - Compute the literal length once in put_literal() instead of spelling N - 1 at each use. - Add put_string(str, start, end), which keeps the pointer arithmetic and the bounds assertions inside the function instead of at the call site. With dump_float()'s to_chars() output moved onto put_buffer() as well, put_chars() now has no callers outside put_string()/put_buffer(): nothing passes a bare pointer and a count any more. - Carry the indentation as std::size_t rather than unsigned int. It is a size, it is compared and combined with buffer sizes throughout, and the casts in put_indent() disappear. next_indent() keeps its assertion, which is far harder to trip on a 64-bit size_t but still reachable where that is 32 bits. No output change: pretty and compact dumps, binary values included, are byte-identical to develop. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Silence avoid-c-arrays on put_literal's array reference clang-tidy flags the reference-to-array parameter under cppcoreguidelines/hicpp/modernize-avoid-c-arrays, and the CI treats warnings as errors. Binding to the array is the whole point here - it is what lets the length be deduced from the literal instead of hand-written at the call site - so suppress it the same way from_json(), to_json() and get_to() already suppress it for their own T (&arr)[N] parameters. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Bound the descent of dump() Serializing a container serializes its elements, so dump() descended into one call per nesting level. A value nested deeply enough exhausted the call stack and terminated the process with a segmentation fault - no exception, nothing the caller could catch. Parsing such a value works, as the parser is iterative, and so does destroying one, as #1436 made destruction iterative. Bound how far the descent goes rather than take the call stack away from it. The first 128 levels are written by exactly the code that always wrote them, and only below that does dump_iteratively write out what is left, keeping the containers it has entered on an explicit stack. Serializing can therefore no longer exhaust the stack, however deeply a value is nested, while a value nested less deeply than the bound pays only for one comparison per container. Writing every value that way instead measured between 2% and 20% slower - 20% on object-heavy documents - which is why the descent is kept for all but the values that cannot afford it. The bound costs nothing measurable: between -1.4% and +1.2% across compact and pretty output of number, integer, string, object-heavy, wide-object and deeply nested documents. The output is unchanged for every value. Both ways of writing a container emit the separator in front of every element but the first, rather than after every element but the last, which puts exactly one between each pair and none at the end. This fixes #5387 for dump(). The copy constructor is fixed in #5389. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fold ensure_ascii into the escaper and write bytes without dump_integer Two hot spots that the write buffer and the bulk scanner left behind. dump_escaped took ensure_ascii as a runtime flag and tested it inside the loop, once per character run, although it cannot change while a string is written. It is now a template parameter, dispatched once per string, which folds the choice of scanner and lets each of the two be inlined into a loop of its own. This is the hottest loop in the serializer: it runs over every string and every object key. A binary value's bytes went through dump_integer, which counts digits and does 64-bit arithmetic for a number that is always in [0, 255]. dump_byte writes the three digits it takes at most straight into the write buffer instead. Any byte type that is not a plain unsigned byte is still left to dump_integer, whose representation of it may differ. Measured against the previous commit (medians of 9 interleaved runs, clang -O3): binary values -33.8%, dense CJK with ensure_ascii -20.6%, key-heavy objects -17.8%, deeply nested pretty output -17.9%, dense CJK without ensure_ascii -11.8%, object-heavy documents -9.3% compact and -9.5% pretty, a small value dumped in a loop -21.4%, wide objects -2.3%. Arrays of plain ASCII strings measured 3.5% to 4.2% slower, the one shape that loses; number and integer arrays are unchanged. Also tried and dropped: leaving the write and string buffers uninitialized rather than zeroing 1.5 KB per dump() call. It is worth -30% on small values, but two nearly identical string workloads moved 18% apart in opposite directions, so the measurements did not support it. The output is unchanged for every value: the differential now also covers every one of the 256 byte values, alone and together, in both binary layouts. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Write a byte without walking a pointer over the buffer clang-tidy's misc-const-correctness reads the pointer dump_byte advanced over the write buffer as one whose pointee could be const. Index the buffer instead, which says the same thing without a raw pointer at all. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Parenthesize the reserve arithmetic in the deep-nesting test clang-tidy's readability-math-missing-parentheses wants the multiplication spelled out in reserve(6 * depth + 1), and CI treats its warnings as errors. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Do not scan for a copyable run that cannot exist Under ensure_ascii, dump_escaped() calls find_ascii_copyable_run() at every character boundary. When the text is dense non-ASCII - CJK, where every byte is >= 0x80 - the scanner stops on its first byte and returns zero, so its SWAR block runs once per character and buys nothing, on top of the escaping that still has to happen afterwards. A run can only be non-empty when the first byte is one the scanner may copy, so test that single byte before calling it. Runs that do exist are found exactly as before, so the bulk-copy win is unchanged; only the calls that were always going to return zero are skipped. Output is unchanged: the dump digest over canada/citm/twitter, in compact, pretty and ensure_ascii form, matches develop byte for byte. dump(ensure_ascii=true) develop before after CJK text 3.54ms 4.25ms 3.36ms CJK, no ASCII at all 3.09ms 4.02ms 3.02ms Latin-1-ish text 4.39ms 3.04ms 2.93ms plain ASCII 3.92ms 0.80ms 0.79ms Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address review of the write-buffer helpers Three points from @gregmarr's review: put_chars() is gone. It was the only entry point taking a bare pointer and a count, and it existed only so put_string() and put_buffer() had something to delegate to. Its body now lives in put_string(), and put_buffer() is put_string(buffer, 0, length) - std::array already carries data() and size(), so it satisfies the same interface a string does. Nothing appends characters without a bound any more. dump_escaped()'s documentation block was duplicated. The dispatcher was inserted between the original comment and the function it described, and the comment was copied rather than split. The worker now has its own short comment saying why ensure_ascii is a template parameter. The local in dump_byte() is deliberate, and is now documented as such: writing through write_buffer[] is a char write, which may alias any object, so with write_buffer_pos updated in place the compiler must reload and store it around every digit. Measured on a dump of a 4 MiB binary value, 18.0 ms without the local against 7.4 ms with it. Output is unchanged: byte-identical dumps across 77 files in compact, pretty, ensure_ascii, pretty+ascii, indent 600 and tab-indent form. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address review: drop unneeded backslash-escapes and duplicate scan loop '"' does not need escaping in a char literal, unlike in a string literal. find_ascii_copyable_run() also duplicated the byte-at-a-time search that already exists as the loop's own scalar tail; break into it instead of re-deriving the offset in a second, near-identical loop. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Move pretty_print, ensure_ascii and indent_step into the serializer None of these change over the life of a serializer, unlike current_indent and depth, which do change on every recursive call. They are now captured once in the constructor - matching indent_char and error_handler - instead of being threaded through dump(), dump_internal(), dump_iteratively(), dump_value() and dump_escaped() on every call. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Stop the serializer from holding onto std::localeconv()'s pointer loc was only ever read twice, immediately, to seed thousands_sep and decimal_point; nothing else in the class used it. A local in the constructor body serves the same purpose without keeping the pointer around for the serializer's lifetime. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Keep thousands_sep/decimal_point const via a small locale_chars struct const members can't be assigned in a constructor body, so seeding them from std::localeconv() meant either dropping const or holding onto the lconv* for longer than needed. A sub-object computes both from the pointer in its own constructor and is itself initialized in serializer's mem-initializer-list, so the two chars stay const, std::localeconv() is still called exactly once, and nothing outlives the constructor. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Niels Lohmann
|
bfb07786cd |
Fix to_bjdata() silently truncating out-of-range _ArrayData_ elements (#5473)
write_bjdata_ndarray() validated that each _ArrayData_ element matched the number kind (integer vs. float) named by _ArrayType_, but not its range. An element that did not fit the target C++ type (e.g. 256 for "uint8") was silently wrapped by the static_cast used to write it, or, for "single", silently overflowed to infinity. Range-check each element against the type named by _ArrayType_ before writing it, reusing the existing fallback path that already encodes the annotated object as a plain object for other invalid-annotation cases in this function. Fixes #5403. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
 Wu Shuwen
|
3e2de32226 |
Remove unused iomanip include (#5516)
Signed-off-by: dajiaohuang <mikewushuwen@outlook.com> |
||
|
Niels Lohmann
|
d6660cf718 |
Route hand-rolled diagnostic pragmas through Hedley (#5485)
* Route hand-rolled diagnostic pragmas through Hedley
Several places in the library hand-roll compiler diagnostic suppression
with raw `#pragma`/`#ifdef __GNUC__`/`#ifdef __clang__` guards instead of
using the Hedley primitives already bundled and used elsewhere
(JSON_HEDLEY_DIAGNOSTIC_PUSH/POP, JSON_HEDLEY_PRAGMA, ...). Converted six
of the seven listed push/pop pairs to use those primitives instead of
raw `#pragma GCC diagnostic`/`#pragma clang diagnostic` text:
- include/nlohmann/json.hpp (~3770, ~3863): -Wfloat-equal
- include/nlohmann/detail/conversions/to_chars.hpp (~1078): -Wfloat-equal
- include/nlohmann/detail/output/binary_writer.hpp (~1844): -Wfloat-equal
- include/nlohmann/detail/iterators/iteration_proxy.hpp (~211): -Wmismatched-tags
- include/nlohmann/detail/exceptions.hpp (~36): -Wweak-vtables
iteration_proxy.hpp did not previously include macro_scope.hpp itself
(it only compiled because some other header included earlier in
json.hpp happened to pull macro_scope.hpp in first); it now includes it
directly like the other detail headers that use Hedley macros, so it is
self-contained.
Each push/pop pair now uses JSON_HEDLEY_DIAGNOSTIC_PUSH/POP
unconditionally (a no-op on compilers that don't need it) and wraps the
actual `#pragma ... diagnostic ignored` text in JSON_HEDLEY_PRAGMA so it
goes through Hedley's _Pragma()-based emission instead of a raw #pragma
line, while keeping the original `#ifdef __GNUC__` / `#if
defined(__clang__)` guard around the ignored-pragma itself.
Deviation from the issue's suggested transformation: the issue's example
replaces the `#ifdef __GNUC__` guard with `#if
JSON_HEDLEY_HAS_WARNING("-Wfloat-equal")`. JSON_HEDLEY_HAS_WARNING is
implemented purely via Clang's `__has_warning` builtin and evaluates to
0 on real GCC (`#define JSON_HEDLEY_HAS_WARNING(warning) (0)` when
`__has_warning` is not defined), so adopting it verbatim would silently
stop suppressing -Wfloat-equal on GCC -- a real regression, not just a
style change. The existing `#ifdef __GNUC__` / `#if defined(__clang__)`
guards were kept for the ignored-pragma to stay behavior-preserving, and
only the push/pop/pragma-emission mechanism was routed through Hedley.
Two of the seven locations from the issue (the -Wignored-attributes
push at the very top of json.hpp and its matching pop after
`#include <nlohmann/detail/macro_unscope.hpp>`) were intentionally left
unconverted:
- The push, at the very top of json.hpp, runs before
`detail/macro_scope.hpp` (and therefore hedley.hpp) has been included
anywhere in the translation unit, so JSON_HEDLEY_DIAGNOSTIC_PUSH is not
yet defined at that point.
- The pop runs after `macro_unscope.hpp`, which -- via hedley_undef.hpp
-- has already #undef'd every JSON_HEDLEY_* macro (by design, see
#5408) precisely so they don't leak to users, so JSON_HEDLEY_DIAGNOSTIC_POP
is no longer defined by the time the pop is reached either.
Making this one pair work would require either hoisting the ~2000
line vendored hedley.hpp to the very top of the amalgamated single
header (a much bigger structural change to single_include than a pure
mechanism swap) or special-casing this one pop ahead of the general
macro cleanup. Both are riskier than the mechanical, behavior-preserving
change requested, so this pair was left as-is.
## Validation
- Compiled include/nlohmann/json.hpp and single_include/nlohmann/json.hpp
with `-Wall -Wextra -Wfloat-equal -Wmismatched-tags -Wweak-vtables`
(clang, which self-identifies as __GNUC__ too): no warnings, same as
before the change.
- Compiled and ran tests/src/unit-to_chars.cpp, unit-conversions.cpp,
unit-iterators1.cpp, unit-iterators2.cpp, and unit-class_parser.cpp
against the fixed include/: all pass.
- Compiled unit-msgpack.cpp, unit-bjdata.cpp, and unit-ubjson.cpp (which
exercise binary_writer.hpp's write_compact_float extensively): all
compile cleanly; the vast majority of assertions pass (the only
failures are pre-existing environment issues unrelated to this change
-- missing generated test-data files, not code correctness).
- Ran `make amalgamate`; the single_include diff is limited to exactly
the lines touched in include/, with no unrelated reordering.
- No real (non-Apple) GCC was available in this environment to test
directly; the `_Pragma("GCC diagnostic ...")` text emitted by
JSON_HEDLEY_PRAGMA is byte-identical to the prior `#pragma GCC
diagnostic ...` text, and the `#ifdef __GNUC__` guard is unchanged, so
GCC's behavior is expected to be identical. CI covers the GCC matrix.
This PR is stacked on top of #5475 (issue-5408-hedley-undef-leak) since
both touch the same files; only the last commit here is new.
Fixes #5409.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Guard JSON_HEDLEY_DIAGNOSTIC_PUSH/POP with the same compiler check as the pragma they bracket
Addresses review feedback from @gregmarr on PR #5485: the push/pop calls
were unconditional, so compilers other than the one the ignored-pragma
targets (e.g. MSVC, or GCC where the pair only applies under __clang__)
now did a needless push/pop with nothing suppressed in between. Move the
existing #ifdef __GNUC__ / #if defined(__clang__) guard to also cover the
push/pop, restoring the original zero-overhead behavior on other compilers
while still emitting the pragma itself through Hedley.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
|
||
 Qatadaha Bin Matloob
|
09b6b6b5ba |
Fix to_bjdata() emitting unparsable output when _ArraySize_ is not an array (#5455)
* Fix to_bjdata() emitting unparsable output when _ArraySize_ is not an array
write_bjdata_ndarray() never checked that _ArraySize_ is an array. The shape
is written verbatim as the header length, so a null shape emitted 'Z' and an
object shape emitted '{' after the '#', neither of which from_bjdata()
accepts, and the round-trip guarantee in the BJData docs was broken.
Both slipped through the existing validation: for null, empty() is true so
the element count starts at 0 and the per-dimension loop never runs, and for
an object the loop walks its values, which can satisfy the non-negative
integer check. When _ArrayData_ then matched that count, the writer took the
ndarray path.
Require the shape to be an array, so anything else falls back to a plain
object encoding that round-trips, as the fallback rule in the docs already
specifies.
Signed-off-by: qatcod <79017227+qatcod@users.noreply.github.com>
* Document that _ArraySize_ must be an array in the ndarray requirements
The list at bjdata.md is the exhaustive set of conditions for the ndarray
encoding, but it only implied this one through 'every entry of'.
Signed-off-by: qatcod <79017227+qatcod@users.noreply.github.com>
---------
Signed-off-by: qatcod <79017227+qatcod@users.noreply.github.com>
|
||
 Krishnanand G
|
2f025f401e |
Throw other_error.502 when UBJSON use_type is set without use_size (#5380)
* Throw other_error.502 when UBJSON use_type is set without use_size Fixes #5321 Signed-off-by: Krishnanand G <118352827+Krishnanand-G@users.noreply.github.com> * Scope UBJSON use_type check to container branches and expand tests Signed-off-by: Krishnanand G <118352827+Krishnanand-G@users.noreply.github.com> * Re-amalgamate single_include/json.hpp The previous commit updated the split headers but the amalgamated file didn't go back through astyle before I committed it, so CI's amalgamation check caught formatting drift in json_fwd.hpp and a few noexcept clauses in basic_json, plus one doc example. None of it touches the UBJSON logic. Applied the patch CI generated to bring single_include back in sync. Signed-off-by: Krishnanand G <118352827+Krishnanand-G@users.noreply.github.com> --------- Signed-off-by: Krishnanand G <118352827+Krishnanand-G@users.noreply.github.com> |
||
|
Niels Lohmann
|
9a091d2b82 |
Do not write BJData ndarrays whose size overflows std::size_t (#5362)
* Do not write BJData ndarrays whose size overflows std::size_t
write_bjdata_ndarray() multiplied the _ArraySize_ dimensions into a
std::size_t without checking for overflow. A product that wraps around
to a value that happens to match the size of _ArrayData_ passed the
length check, and the writer emitted an ndarray header announcing an
element count that cannot be represented:
{"_ArrayType_":"uint8","_ArraySize_":[9223372036854775808,2],"_ArrayData_":[]}
was encoded as 5b 24 55 23 5b 4d 00 00 00 00 00 00 00 80 69 02 5d, an
ndarray of 2^64 elements followed by no data. Reading that back throws
out_of_range.408 ("excessive ndarray size caused overflow"), so to_bjdata
produced output that from_bjdata rejects. This is reachable by parsing
untrusted JSON and re-encoding it as BJData.
Mirror the overflow check the binary reader already performs, and also
reject a single dimension that does not fit into std::size_t, which the
previous cast silently truncated where std::size_t is narrower than 64
bits. Such objects now fall back to a plain object encoding, which is
what the surrounding type and length validation already does for
annotations it cannot represent, and they round-trip unchanged.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Document when to_bjdata converts a JData annotation to an ND-array
The BJData page described the 1-D vector case as the only situation in
which an object carrying _ArrayType_/_ArraySize_/_ArrayData_ is not
written as a compact ND-array. The writer has always had several other
fallbacks -- an unknown _ArrayType_, a dimension that is not a
non-negative integer, an _ArrayData_ whose length does not match the
product of the dimensions, and elements that are not numbers of the
annotated kind -- all of which cause the value to be serialized as a
regular JSON object instead.
Spell out the conditions, including the size-overflow check added in the
preceding commit, so the documented behavior matches the implementation.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
|
||
 Angadi56
|
fd72ecfc8c |
validate ndarray element types in write_bjdata_ndarray (#5301)
* validate ndarray element types in write_bjdata_ndarray Signed-off-by: Angadi Yashaswini <angadi@digiscrypt.com> * read ndarray elements through get<> instead of a fixed union member _ArrayType_ names the wire type, not how the value is stored: parsing keeps a non-negative integer as number_unsigned while the C++ API keeps an int literal as number_integer. Selecting the union member from the type marker therefore reads the inactive alternative for one of the two, so read through get<> instead, which dispatches on the active member. Also reject a negative _ArraySize_ entry, which is not a usable dimension, and cover the parse-built path in the tests. Signed-off-by: Angadi Yashaswini <angadi@digiscrypt.com> --------- Signed-off-by: Angadi Yashaswini <angadi@digiscrypt.com> |
||
 Luke Banicevic
|
2e23687092 | to_bson() silently emits corrupt documents when a length exceeds INT32_MAX (#5314) | ||
|
Luke Banicevic
|
8ec98e2c9e | adding cleanups to bson writer (#5313) | ||
|
Niels Lohmann
|
366f3d26e5 |
Replace snprintf with a branch-free writer for \uXXXX escapes (#5235)
* Replace snprintf with a branch-free writer for \uXXXX escapes dump_escaped called std::snprintf(..., "\u%04x", ...) once per escaped code point in the string serialization hot path. snprintf re-parses the format string and pulls in locale/printf machinery on every call, which is far heavier than the fixed 6-/12-byte output warrants. This is hot for any string containing control characters, and for all non-ASCII text when ensure_ascii is set. Replace it with write_u_escape, a small helper that writes the escape directly into string_buffer via a nibble-to-hex lookup table, mirroring the existing hand-rolled dump_integer fast path in the same file. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix clang-tidy avoid-c-arrays warning in write_u_escape Use a const char* rather than a char[] lookup table, matching the existing hex_bytes helper in the same file. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * ♻️ adjust write_u_escape signature Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
31dd15b258 |
Fix ambiguous static_cast (#5221)
* 🐛 fix ambiguous static_cast Signed-off-by: Niels Lohmann <mail@nlohmann.me> * ✅ add regression test Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🐛 fix warning Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🚨 fix warning Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
 Kirill Lokotkov
|
b1bb9fce0c | Fix for printing long doubles bug in dump_float (#3929) | ||
|
Niels Lohmann
|
5a05627b1f | 🚨 fix warning (#5169) | ||
 Charles Cabergs
|
5ed07097fa |
Fix -Wtautological-constant-out-of-range-compare in serializer (#5050)
Signed-off-by: Charles Cabergs <me@cacharle.xyz> |
||
|
Niels Lohmann
|
515d994acb |
📄 adjust year (#5044)
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
54be9b04f0 | 📄 update REUSE (#4960) | ||
|
Niels Lohmann
|
b19f058465 |
Encode infinity and NaN as float for MsgPack and CBOR (#4802)
* 🚸 encode infinity and NaN as float for MsgPack and CBOR Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🚨 suppress warnings Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
 NmPassTHFan
|
46e7cd3dc2 | Replace deprecated std::is_trivial in C++26 (#4775) | ||
|
Niels Lohmann
|
9110918cf8 |
Fix typos (#4748)
* ✏️ fix typos Signed-off-by: Niels Lohmann <mail@nlohmann.me> * ✏️ address review comments Signed-off-by: Niels Lohmann <mail@nlohmann.me> * ✏️ address review comments Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
88c92e605c |
Fix compilation failure and warnings with NVHPC (#4744)
* 🚨 fix warnings Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🚨 fix warnings Signed-off-by: Niels Lohmann <mail@nlohmann.me> * ⚗️ enable ranges support Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🔥 remove ci_nvhpc job Signed-off-by: Niels Lohmann <mail@nlohmann.me> * ⚗️ enable ranges support Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🔥 remove ci_nvhpc job Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🚨 fix warning Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
4cca3b9cb2 |
Fix warning and add emscripten CI step (#4738)
* 🚨 fix warning Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 👷 add emscripten Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 👷 add emscripten Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 👷 add emscripten Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 👷 add emscripten Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 👷 add emscripten Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 👷 add emscripten Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 👷 add emscripten Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 👷 add emscripten Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 👷 add emscripten Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 📝 add compiler to list Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
1705bfe914 |
🔖 set version to 3.12.0 (#4727)
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
f06604fce0 |
Bump the copyright years (#4606)
* 📄 bump the copyright years Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 📄 bump the copyright years Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 📄 bump the copyright years Signed-off-by: Niels Lohmann <niels.lohmann@gmail.com> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> Signed-off-by: Niels Lohmann <niels.lohmann@gmail.com> |
||
 Michael Valladolid
|
2d42229f4d |
Support BSON uint64 de/serialization (#4590)
* Support BSON uint64 de/serialization Signed-off-by: Michael Valladolid <mikevalladolid@gmail.com> * Treat 0x11 as uint64 and not timestamp specific Signed-off-by: Michael Valladolid <mikevalladolid@gmail.com> --------- Signed-off-by: Michael Valladolid <mikevalladolid@gmail.com> |
||
 Nebojša Cvetković
|
48e7b4c23b | BJData Fixes (#4588) | ||
|
Nebojša Cvetković
|
2e50d5b2f3 | BJData optimized binary array type (#4513) | ||
|
Niels Lohmann
|
1b9a9d1f21 |
Update licenses (#4521)
* 📄 update licenses * 📄 update licenses |
||
|
Niels Lohmann
|
1825117e63 |
Another desperate try to fix the CI (#4489)
* 🚨 fix warning * 💚 update actions * 🚨 fix warning * 🚨 fix warning * 🚨 fix warning * 💚 update actions * 💚 update actions * 🚨 fix warning * 🚨 fix warning * 💚 update actions * 🚨 fix warning * 💚 update actions * 💚 update actions * 💚 update actions * 🚨 fix warning * 🚨 fix warning * 🚨 fix warning * 🚨 fix warning * 💚 update actions * 💚 update actions * 🚨 fix warning * 💚 update actions * 💚 update actions * 💚 update actions * 💚 update actions * 💚 update actions |