mirror of
https://github.com/nlohmann/json.git
synced 2026-10-02 20:50:32 +00:00
e5a89d671f3627dbe2dac0dd481c01ea037c21e2
53
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
e5a89d671f |
Fix lint debt: enum-macro NOLINTs, doctest as SYSTEM, no-op analyzer (#5737)
* Drop stale LCOV_EXCL_LINE from the json_pointer out_of_range.410 throw The comment said the size_type overflow check in array_index() is only triggered on special platforms like 32-bit, and the throw was excluded from coverage. On 64-bit platforms the check is true for SIZE_MAX itself, and unit-json_pointer.cpp has asserted that case four times since #5395, so the line is executed in the coverage job. Reword the comment and remove the exclusion marker so the coverage report notices if the tests stop reaching it. Part of #5725 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Name all three C-array check aliases in the enum-macro NOLINTs NLOHMANN_JSON_SERIALIZE_ENUM(_STRICT) suppressed the c-array warning under modernize-avoid-c-arrays only, but clang-tidy emits the same diagnostic under the aliases cppcoreguidelines-avoid-c-arrays and hicpp-avoid-c-arrays too. Any user running those checks got a false positive at every macro expansion, and our own tests needed a local NOLINT at each call site to work around it. Name all three aliases in the four macro comments instead, and drop the now-redundant c-array names from the five test call-site NOLINTs. Comment-only change; behavior, the public API, and the ABI do not change. Ran make amalgamate. Part of #5725 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Include doctest as a SYSTEM directory instead of disabling warnings for all tests test_main added -Wno-deprecated and -Wno-float-equal as PUBLIC compile options for every non-MSVC compiler, so they were applied to every translation unit, library headers included, and silenced the CI warnings meant to check the library's own -Wfloat-equal pragmas. The only code that actually needed the suppression was the vendored doctest.h, which was included as a normal (non-SYSTEM) directory. Include thirdparty/doctest as SYSTEM for test_main, matching what tests/abi/CMakeLists.txt already does, and drop the two suppressions from both targets. Verified locally that unit-comparison, unit-conversions and unit-constructor1 compile clean with -Werror -Weverything and doctest as -isystem, and that CMake still configures with JSON_BuildTests=ON. Part of #5725 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Remove the no-op ci_clang_analyze target ci_clang_analyze configured the build with the real compiler and only then wrapped ninja with scan-build. scan-build intercepts compiles by overriding CC/CXX, but build.ninja already had the compiler path baked in from the configure step, so every run bypassed the analyzer: CI logs show "No bugs found" after a normal build, never an analysis. The job also used Debian's frozen clang-tools-14 rather than the image's own clang, and CLANG_ANALYZER_CHECKS still named three valist.* checkers that current clang merged into security.VAList. ci_clang_tidy already runs every clang-analyzer-* check (via .clang-tidy's "Checks: '*'") with warnings as errors, so nothing is lost by removing the dead job. Delete ci_clang_analyze, CLANG_ANALYZER_CHECKS and the SCAN_BUILD_TOOL lookup from cmake/ci.cmake, drop it from the ubuntu.yml ci_static_analysis_clang matrix, and drop the now-unused clang-tools apt package (iwyu stays for ci_single_binaries). Reword quality_assurance.md and assurance_case.md, which described the dead job as a working control, to say the Clang Static Analyzer checks run through clang-tidy. Verified that `cmake -DJSON_CI=ON` still configures cleanly and that ci_clang_analyze no longer appears in the generated build or in any CMake/workflow file. Part of #5725 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Re-enable portability-template-virtual-member-function; remove redundant forwards .clang-tidy disabled three checks "to get the CI going" (#4489, 2024-11-13): portability-template-virtual-member-function, bugprone-use-after-move and its alias hicpp-invalid-access-moved. portability-template-virtual-member-function only flagged output_stream_adapter::write_character/write_characters; annotate both with NOLINT and re-enable the check. bugprone-use-after-move flagged several double forwards that have no effect at runtime: - from_json.hpp calls std::forward<BasicJsonType>(j).at(Idx) inside pack expansions; at() has no ref-qualified overloads and always returns an lvalue reference, so the forward is a no-op. Replace with plain j.at(Idx) in all four places. - the move constructor forwards the whole object to its base class and then reads other's members. That is item 9 of #5724 (together with its cppcheck suppressions) and is left to that change. - input_adapters.hpp forwards the container twice on purpose, so the begin/end iterator types match adapter_type; annotate with NOLINT and a comment instead of changing behavior. The check still flags the move constructor (see above) and two sites in at(KeyType&&) (both overloads, json.hpp, in the throw's string_t(std::forward<KeyType>(key)) after find(std::forward<KeyType>(key))). Open PR #5689 rewrites that hunk, so bugprone-use-after-move (and hicpp-invalid-access-moved) stay disabled for now, with a comment explaining why; re-enable them once #5689 and the #5724 move-constructor change have landed. Also resolve the portability-avoid-pragma-once TODO: single_include never has #pragma once (amalgamate.py strips it) and every supported compiler accepts it in include/, so keep it disabled with an explanatory comment instead of a TODO. Fix the stale "json.hpp, around line 1265" comment in unit-class_parser.cpp, which now points at the move constructor's actual line. Behavior, the public API and the ABI do not change. Verified with clang-tidy 22.1.8 that portability-template-virtual-member-function now reports nothing, that bugprone-use-after-move/ hicpp-invalid-access-moved report only the known at(KeyType&&) and move-constructor sites, and that unit-custom-base-class, unit-constructor1, unit-conversions, unit-element_access2, unit-class_parser and unit-diagnostic-positions (JSON_DIAGNOSTIC_POSITIONS=1) compile under ASan/UBSan and pass with the same assertion counts as before. Ran make amalgamate. Part of #5725 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix stale and malformed NOLINT comments json_sax.hpp named "-warnings-as-errors" in the NOLINT list on the two JSON_ASSERT(false) lines; that is the suffix clang-tidy appends to a diagnostic tag under WarningsAsErrors, not a check name, and every other JSON_ASSERT(false) omits it. unit-capacity.cpp carried 30 "// NOLINT(misc-const-correctness)" comments on "json j = ...;" declarations that are all used with non-const members afterwards, so the check has nothing to report there. unit-constructor2.cpp used a blanket "// NOLINT: access after move is OK here" on a use-after-move that hides every check on the line; naming bugprone-use-after-move and hicpp-invalid-access-moved keeps the intent once those checks are re-enabled (#5724). Signed-off-by: Niels Lohmann <mail@nlohmann.me> #5725 item 10 * Remove stale .clang-tidy entries -google-runtime-references disabled a check that neither clang-tidy 22.1.8 nor 23.1.2 lists under --list-checks -checks='*'; it was removed upstream. The commented-out HeaderFilterRegex line has been unused since the active HeaderFilterRegex was introduced in #2561 (2021). Signed-off-by: Niels Lohmann <mail@nlohmann.me> #5725 item 11 * Remove the GCC C++20 -Wignored-attributes pragma in json.hpp The pragma (added in #5164) claimed to work around the C++ modules redefinition errors of #5103, but #5103 is about hard errors (e.g. "redefinition of std::__is_constant_evaluated()", conflicting std::integral_constant) that ignoring a warning cannot suppress; they are traced to GCC PR 124430 and reproduce with <map> or <string> instead of json.hpp too. A GCC 16.2 -std=gnu++20 -fmodules build following #5103's repro steps still fails with the pragma in place, and a build of all test TUs with GCC_CXXFLAGS (which enable -Wignored-attributes) and the pragma removed produces no such warning. The block only hid a warning class from GCC C++20 users while suggesting #5103 was handled. Overlaps #5610, whose hunks touch the closing half of this pragma to insert the json_literals.hpp include. Signed-off-by: Niels Lohmann <mail@nlohmann.me> #5725 item 9 * Fix stale doxygen comments hidden by the -Wdocumentation pragma macro_scope.hpp ignores -Wdocumentation and -Wdocumentation-unknown-command for the whole library, which also hides genuine documentation mistakes: - detail::unescape() documented "@return unescaped string" but returns void and unescapes its argument in place; reworded to "@param[in,out] s string to unescape in place" and dropped the bogus @return. - basic_json::get()'s copy-conversion overload wrote "converted to @tparam ValueType" inside @return, which Doxygen and Clang parse as a second, malformed @tparam; changed to "@a ValueType", matching the two other get() overloads a few lines above that already use it. This narrows the gap the -Wdocumentation pragma needs to cover; fully replacing the Doxygen-only commands it also hides (item 2c) is left for after #5267. Signed-off-by: Niels Lohmann <mail@nlohmann.me> #5725 item 2 * Fix -Wextra-semi-stmt at its actual source, not assert() clang_flags.cmake blamed the global -Wno-extra-semi-stmt on assert(), but assert() expands to an expression under glibc and libc++ and does not trigger this warning. unit-assert_macro.cpp overrides JSON_ASSERT with "{if (!(x)) ++assert_counter; }", a bare block followed by a semicolon at every JSON_ASSERT(...) call site in the library; that was the actual source of 151 of the 208 -Wextra-semi-stmt sites found in a Clang 22 -Weverything sweep of the test suite with the flag removed. Switched to the standard do/while(false) macro idiom, which does not expand to a statement-plus-semicolon, and corrected the comment to name the remaining source instead: vendored Doctest's CAPTURE(x) shim, which already ends in a semicolon. Verified with clang++ -Wextra-semi-stmt (plus the file's other CI ignores) that unit-assert_macro.cpp now compiles without any -Wextra-semi-stmt diagnostic. Signed-off-by: Niels Lohmann <mail@nlohmann.me> #5725 item 8 (step 1 of 2; step 2 covers the CAPTURE() call sites) * Drop the redundant semicolon from CAPTURE() call sites; remove -Wno-extra-semi-stmt doctest_compatibility.h defines CAPTURE(x) as DOCTEST_CAPTURE(x); (with a trailing semicolon baked into the macro), specifically so call sites do not need to add one themselves; most of the ~267 call sites already follow that convention. The remaining 64 call sites across 20 files wrote "CAPTURE(x);" anyway, turning into a statement plus an empty statement and triggering -Wextra-semi-stmt. Dropped the redundant semicolon at each of those sites. With item 6 having already made vendored Doctest a SYSTEM include, and this the last known source of -Wextra-semi-stmt findings, removed the flag from clang_flags.cmake entirely. Verified with clang++ -Wextra-semi-stmt (plus the file's other CI ignores) that all 20 touched files, plus a file with no CAPTURE() use (unit-json_pointer.cpp), compile without any -Wextra-semi-stmt diagnostic. Signed-off-by: Niels Lohmann <mail@nlohmann.me> #5725 item 8 (step 2 of 2) * Switch ci_static_analysis_clang off the frozen LLVM 22 dev image ubuntu.yml pinned the clang-tidy/clang-tidy-sanitizer/single-binaries job to silkeh/clang:dev, a tag last pushed 2026-02-18 that reports "clang version 22.0.0 (...+20251015...)", a pre-release snapshot from before the LLVM 22 release; the maintainer now updates dev-unstable, 22, and latest instead. Switched to silkeh/clang:22, matching the other clang jobs on :latest. Verified with clang-tidy 22.1.8 (the image's actual version) against this repository's .clang-tidy and library headers what the release image newly reports compared to :dev: - readability-redundant-typename fires at ~250 sites across the _cpp20-relevant conversion/to_chars headers; the library targets C++11 and keeps the typenames, so the check is disabled in .clang-tidy, matching how the file already handles checks that don't fit a C++11 codebase. - misc-anonymous-namespace-in-header fires on the two anonymous namespaces in from_json.hpp and to_json.hpp; added the alias to their existing NOLINT (cert-dcl59-cpp, fuchsia-header-anon-namespaces, google-build-namespaces). - bugprone-std-namespace-modification fires on every addition to namespace std: the std::hash, std::formatter and std::swap overloads in json.hpp, and the std::tuple_size/std::tuple_element specializations in iteration_proxy.hpp (this last file is not named in #5725's item 5, found by actually running clang-tidy 22.1.8 against the current tree). All six are legal, deliberate additions to namespace std (explicit/partial specializations of std types, or the pre-C++20 std::swap overload); annotated each with the check name next to its existing cert-dcl58-cpp NOLINT. - modernize-avoid-c-style-cast reported nothing new. Also added clang++-22/21, clang-tidy-22/21, g++-16 and gcov-16 to the find_program search lists in ci.cmake so a local "maximal warnings" configure prefers the current toolchain version over an older one on PATH. #5725 item 5 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Regenerate cmake/gcc_flags.cmake for GCC 16.2.0 GCC_CXXFLAGS was generated for GCC 15.1.0, but ci_test_gcc and ci_test_gcc_cxx{11..26} now run in gcc:latest, currently GCC 16.2.0, so the "maximal warnings" job was missing warnings introduced since 15.1.0 while carrying entries GCC 16 treats as duplicates or no-ops. Regenerated with https://github.com/nlohmann/gcc_flags (patched locally to not crash on an option whose "-x c++ <opt> -" probe fails before it reads stdin, e.g. -Wabi=; the tool otherwise raises BrokenPipeError instead of recording the option as an error) run against g++ 16.2.0 in the official gcc:16 Docker image, keeping the documented -Wno-* exclusions and the same alphabetical placement scheme as before. Also added three GCC 16 warnings the generator cannot discover on its own because it only probes value ranges/lists it finds in the -Q option name itself, not in the enum choices --help=warnings documents separately: - -Wbidi-chars=any, -Wleading-whitespace=spaces: manually verified these compile cleanly with g++ 16.2.0. - -Wstrict-flex-arrays: deliberately NOT added, unlike the other two. Without -fstrict-flex-arrays (which the library does not enable, as it would change codegen for flexible array members), GCC prints "'-Wstrict-flex-arrays' is ignored when '-fstrict-flex-arrays' is not present" on every translation unit, and under our -Werror that note itself aborts the build. This differs from the harmless no-op warnings already kept in the file (-Whsa, -Wsynth, -Wunreachable-code, -Wunsafe-loop-optimizations), which emit nothing; #5725 item 7 named -Wstrict-flex-arrays as one of the flags GCC 16 adds, but did not anticipate this failure mode. Verified: compiled the library header and a representative set of test translation units (including ones touched by items 1, 3, 8, 9, 10 of this issue) with the regenerated GCC_CXXFLAGS plus -Werror under g++ 16.2.0 at -std=c++11 through -std=c++26, with zero warnings; ran the full local test suite (129/129 passing, unrelated to this compiler) as a regression check. CI must still confirm the actual ci_test_gcc / ci_test_standards_gcc targets end to end, since this was verified with direct g++ invocations rather than through the CMake/ CXXFLAGS environment-variable plumbing in ci.cmake. #5725 item 7 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Avoid std::basic_string<CharType> for non-character output_adapter CharType output_adapter<CharType, StringType> defaulted StringType to std::basic_string<CharType>, and (with JSON_NO_IO undefined) always declared a std::basic_ostream<CharType>&-taking constructor. For CharType with no non-deprecated std::char_traits specialization (only std::uint8_t is ever used this way, by the binary writers), simply naming either type - as an unused default template argument, or as an unused, never-called constructor's parameter type - instantiates std::char_traits<CharType> merely to name it, which some standard libraries mark deprecated: with the library-wide -Wdocumentation pragma (item 2's other half, left for a later commit) temporarily removed, an Apple clang 21 / libc++ TU calling json::to_cbor(j, vec) with std::vector<std::uint8_t>& got one -Wdeprecated-declarations warning per binary writer at the old output_adapters.hpp:193. Replaced the eager std::basic_string<CharType> / std::basic_ostream <CharType> defaults with a bool-tagged partial specialization (not std::conditional, which requires naming both branches' types up front regardless of which is selected, reproducing the same warning) that only ever names std::basic_string<CharType> / std::basic_ostream <CharType> when CharType is actually one of char, wchar_t, char16_t, char32_t, or (with __cpp_lib_char8_t) char8_t. For any other CharType, output_adapter's StringType and ostream-constructor parameter fall back to two distinct empty placeholder types, kept distinct so the two constructor overloads do not collide into a single redeclaration. Public API / behavior: passing a std::basic_string<std::uint8_t>& or std::basic_ostream<std::uint8_t>& directly to a binary writer's output_adapter now fails to compile instead of compiling with a deprecation warning; this was neither documented nor tested. All documented uses (std::vector<CharType>, std::basic_ostream<CharType> and StringType for character CharType) are unaffected. Verified with Apple clang 21 / libc++, with the two -Wdocumentation* "ignored" pragma lines in macro_scope.hpp temporarily removed and -std=c++11/c++20 plus the project's -Weverything flag set: calling to_cbor/to_msgpack/to_ubjson/to_bjdata/to_bson/to_bon8 on a std::vector<std::uint8_t> now produces no char_traits<unsigned char> (or any other) deprecation warning, while the char-based string- and ostream-adapter paths, and a to_cbor/from_cbor round trip, still compile and run correctly; also verified with GCC 16.2.0. Ran the full local test suite, including the binary-format unit tests (unit-cbor, unit-msgpack, unit-ubjson, unit-bjdata, unit-bson, unit-bon8, unit-binary_writer_sinks, unit-binary_formats, unit-custom-binary-type): 129/129 passing. #5725 item 2 (step a) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Remove the library-wide -Wdocumentation pragma; fix what it hid macro_scope.hpp / macro_unscope.hpp pushed and popped a Clang diagnostic region over the entire library that ignored -Wdocumentation and -Wdocumentation-unknown-command. Removed both pragmas and fixed every finding a full -Wdocumentation (which implies -Wdocumentation-unknown-command and -Wdocumentation-deprecated-sync) build reports, so the library now compiles clean under Clang's documentation checks without a blanket suppression. Overlaps #5267, which is still open and edits a nearby doc block (json.hpp's get()/get_impl() @return, already fixed in the item 2 step (b) commit of this branch); this commit does not touch that block again. Unknown Doxygen alias commands (Doxyfile removed in #3071, so these were never rendered by anything) rewritten as plain prose, keeping the same information: - @requirement REQ-JSON-01 / REQ-JSON-02 (iter_impl.hpp, json_reverse_iterator.hpp): now "This class satisfies the following concept requirements (REQ-JSON-0N):". - @liveexample{prose,example-id} (three sites in json.hpp): kept the prose, dropped the command wrapper and the trailing example-id (docs/mkdocs/docs/examples/*.cpp still exist and are used directly by the rendered docs, not through this in-header alias) and unescaped the "\," commas that were only needed for the old alias's comma-separated argument syntax. - @complexity X (json.hpp x4, json_pointer.hpp x2, serializer.hpp x1): now "Complexity: X". Backslash sequences Clang's comment lexer tried to parse as commands, escaped to render as literal backslashes: - lexer.hpp get_codepoint(): two `\u` occurrences. - binary_reader.hpp get_bson_cstr() / get_bson_cstr_bulk(): two `\x00` occurrences. - serializer.hpp: three `\uXXXX` occurrences (constructor @param, append_codepoint_to_string_buffer() @brief, and the ensure_ascii member comment). One finding remained after all of the above: Clang reports "declaration is marked with '@deprecated' command but does not have a deprecation attribute" on the deprecated sax_parse(span_input_adapter&&, ...) overload, even though JSON_HEDLEY_DEPRECATED_FOR does expand to __attribute__((deprecated(...))) for Clang. Several isolated reproductions of this exact declaration shape - doc comment, template<>, two stacked __attribute__ macros, an overload set sharing the name - did not reproduce the warning, so this looks like a Clang comment/declaration-association quirk specific to this overload inside the much larger basic_json class template, not an actual documentation defect. Rather than keep the pragma library-wide for one Clang false positive, added a tightly scoped -Wdocumentation-deprecated-sync push/pop around just that overload. Verified with Apple clang 21 and the project's actual -Weverything flag set (cmake/clang_flags.cmake) on the full header at -std=c++11 and -std=c++20: zero -Wdocumentation* diagnostics. Also compiled clean with GCC 16.2.0 (the pragmas are already __clang__-gated, so this only confirms no unrelated breakage). Ran make check-amalgamation and the full local test suite: 129/129 passing. #5725 item 2 (step c) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Take the JSON value by const reference in the array and tuple from_json paths Review feedback on #5737 (gregmarr): once the no-op std::forward calls are gone, the forwarding references have no purpose. from_json_fn passes the value as const BasicJsonType&, so these functions were only ever instantiated with a const lvalue anyway. The std::array, std::pair and std::tuple overloads of from_json and their helpers now take const BasicJsonType& and pass j on unchanged. Because the deduced BasicJsonType is now the plain type, tuple_type and the static_assert name const BasicJsonType& explicitly, so the reference checks are unchanged: get<std::tuple<const std::string&>>() still works, and get<std::tuple<std::string&>>() still fails the same static_assert. from_json_tuple_get_impl keeps its forwarding reference, since tuple_type calls it through std::declval. Behavior, the public API and the ABI do not change. unit-conversions, unit-constructor1, unit-udt, unit-udt_macro, unit-regression1/2/3, unit-deserialization, unit-noexcept, unit-items, unit-allocator, unit-custom-object-type, unit-ordered_json2 and unit-brace-init-copy-semantics pass at C++11, C++17 and C++20 with unchanged assertion counts. Ran make amalgamate. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
|
b54ed188e6 |
Remove test debt: dead guards, discarded results, and unreferenced files (#5732)
* Run the README test case in JSON_FastTests jobs The "README" test case was marked doctest::skip() when the tests moved from Catch to doctest in 2019, where it replaced Catch's hidden tag. It is not slow (17 assertions, about 0.00 s), but cmake/test.cmake only passes --no-skip when JSON_FastTests is off, so the per-compiler ci_test_*_cxxNN matrix, macOS, Windows Release/ARM, icpc, icpx and nvhpc compiled the README examples without running them. Drop the skip decorator so every job runs the case. Test-only change. Part of #5713 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Remove stale clang ranges guards in unit-iterators2.cpp The "algorithms" and "views" sections were guarded by clang/libstdc++ checks written for a clang 15 (04/2022) bug. The first guard's condition contradicts its own comment: it skips clang+libc++ and keeps clang+libstdc++. Both sections already sit inside `#if JSON_HAS_RANGES`, which macro_scope.hpp excludes for the toolchains these guards targeted, so the inner guards never let the sections run on the platforms they meant to protect and are redundant on the rest. Verified locally with Apple clang 21/libc++ and clang 16.0.6/libstdc++ 12 (Docker): both pass all 1355 assertions with the guards removed. Part of #5713 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix copy-pasted CBOR half-float checks; enable stale encode checks In the RFC 8949 Appendix A test case, the decode checks for 5.960464477539063e-8 (0xf9 0x00 0x01) and 0.00006103515625 (0xf9 0x04 0x00) were copy-pasted from the neighboring -4.0 example, so those two half-float byte sequences were never actually decoded and checked, and -4.0 was checked three times instead. The two float32 encode checks for 100000.0 and 3.4028234663852886e+38 were commented out before the writer supported emitting float32 and are now verified to match byte for byte, so they are enabled. The remaining commented-out half-precision to_cbor checks are collapsed into a single explanatory comment, since the writer never emits half-precision floats. Part of #5713 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Assert on the result of STL container conversions in tests The "object-like STL containers" and "array-like STL containers" sections converted json values into std::map, unordered_map, multimap, unordered_multimap, list, forward_list, array, valarray, vector, deque, set and unordered_set and discarded the result, so these ~60 conversions only proved that the code compiles and does not throw; a conversion that dropped or reordered elements would still pass. Bind each result and compare it against the expected container. Also fix a copy-paste slip in the deque section (`j2.get<std::deque<double>>()` instead of j3, so j3's doubles were never converted to a deque), and remove the dead `// CHECK(m5["one"] == "eins")` comments that referred to a variable that did not exist by asserting the equivalent through the bound result. Part of #5713 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Deduplicate SaxCountdown and other test helpers across formats SaxCountdown was copied byte-for-byte into six binary-format test files (unit-cbor.cpp, unit-msgpack.cpp, unit-ubjson.cpp, unit-bjdata.cpp, unit-bon8.cpp, unit-bson.cpp), about 370 redundant lines. Move it into tests/src/sax_countdown.hpp (namespace utils, alongside test_utils.hpp and round_trip_corpus.hpp) and include it from all six. trait_test_arg and the "value_in_range_of trait" TEST_CASE_TEMPLATE_DEFINE were duplicated between unit-32bit.cpp and unit-bjdata.cpp; the trait is a detail/meta trait, not specific to either file. Move it into tests/src/value_in_range_of_test.hpp; unit-32bit.cpp keeps its own include, since JSON_32bitTest=ONLY builds only that file. Each file keeps its own TEST_CASE_TEMPLATE_INVOKE list. sax_no_exception and the "issue #2824" section were duplicated in unit-regression2.cpp and unit-disabled_exceptions.cpp. Drop the copy from unit-regression2.cpp; unit-disabled_exceptions.cpp already covers the no-exceptions case that #2824 was about, and ci_test_noexceptions reruns it. No behavior change. Verified by building and running unit-cbor, unit-msgpack, unit-ubjson, unit-bjdata, unit-bon8, unit-bson, unit-32bit, unit-regression2 and unit-disabled_exceptions against include/ (clang++ -std=c++11, ASan/UBSan where applicable); assertion counts are unchanged from before the refactor. Part of #5714 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Remove the unreferenced vendored libFuzzer tests/thirdparty/Fuzzer (155 files, ~776 KB of vendored Apache-2.0 LLVM code from the 2016 OSS-Fuzz import) is not referenced by any CMakeLists, Makefile or workflow: the fuzz drivers link against -fsanitize=fuzzer or the repo's own tests/src/fuzzer-driver_afl.cpp. Its vendored README only points at llvm.org's own libFuzzer docs. Being dead code, it also adds noise to the flawfinder code-scanning workflow, which scans the whole tree. Remove the directory and its .reuse/dep5 entry. Part of #5714 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Remove unreferenced 2016 benchmark and fuzz reports tests/reports (1.6 MB) holds AFL status pages and plots from 2016-08-29 and 2016-10-02, and a nativejson-benchmark snapshot from 2016 with links to rawgit.com, which shut down in 2019. Nothing references this directory: no doc, README section, script or workflow points at it, and it describes a ten-years-old, pre-2.0 snapshot of the library. Part of #5714 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Run the CBOR, MessagePack, BSON and BON8 round-trip invariants in CI tests/src/round_trip_corpus.hpp exists so that the byte-stability invariant the fuzzer drivers check also runs on a fixed corpus in CI, instead of only at OSS-Fuzz. So far only the UBJSON and BJData drivers had a matching unit test; the CBOR, MessagePack, BSON and BON8 drivers assert the same invariant (assert(to_X(j2) == vec)) but nothing ran it outside OSS-Fuzz. Add "<FORMAT> round-trip invariants" test cases to unit-cbor.cpp, unit-msgpack.cpp, unit-bson.cpp and unit-bon8.cpp, modeled on the UBJSON case: seed j1 from the corpus (skipping values that do not survive the format's own round trip, as the fuzzer drivers only ever see values from_X() actually produced), then require from_X(to_X(j1)) not to throw and check to_X(j2) == to_X(j1). BSON only serializes objects, so non-object corpus values are skipped. Update the comments in round_trip_corpus.hpp and tests/fuzzing.md to name all six formats. The stream-versus-contiguous check in the BON8 driver is left out, as #5601 reworks it. A local probe confirms no violations on the current corpus (CBOR 3849 checked, MessagePack 3909, BSON 2958, BON8 3841 - matching the counts already recorded for this probe in the issue). Closes #5714 item 1. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix fuzzer driver step lists to match the checks the code performs The header comment of six of the seven binary-format fuzzer drivers listed an invariant the code does not check: CBOR, MessagePack, BSON and BON8 said "assert(j1 == j2)", but the code checks byte stability, assert(to_X(j2) == vec). UBJSON and BJData still described the old "assert(j1 == j2/j3/j4)" byte-exact check from before PR #5494 replaced it with a use_size/use_type-aware round trip (UBJSON) and a value-stability check (BJData); BJData's added paragraph already explained the new check, but the step list above it did not. Also remove a dead branch in fuzzer-parse_bson.cpp: from_bson() is called with allow_exceptions = true, so it throws instead of returning a discarded value, and the "if (j1.is_discarded()) return 0;" guard could never trigger. Drop the unused <iostream> include from all seven drivers and <sstream> from all but fuzzer-parse_bon8.cpp, which is the only one that uses std::istringstream. Overlaps #5601, which edits all seven drivers in the same hunks. Closes #5714 item 4. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Silence the CMP0169 deprecation in cmake_fetch_content, fix stale guards tests/cmake_fetch_content/project calls the single-argument FetchContent_Populate(json) after FetchContent_Declare(), which CMake 3.30 deprecated as CMP0169. Since the project declares cmake_minimum_required(VERSION 3.11...3.14), the policy stays unset, so every configure with a current CMake prints the deprecation warning. The test is kept on purpose: it is the only coverage of the FetchContent_Populate + add_subdirectory pattern for CMake 3.11-3.13 users, which the docs still describe as supported. Explicitly set CMP0169 to OLD, with a comment explaining why. Also fix two stale version guards: - tests/cmake_fetch_content/CMakeLists.txt guarded the test with VERSION_GREATER "3.11.0", which is dead now that tests/CMakeLists.txt requires CMake 3.13. - tests/cmake_fetch_content2/CMakeLists.txt guarded with VERSION_GREATER "3.14.0", which skips exactly 3.14.0, the first version with FetchContent_MakeAvailable. Change it to VERSION_GREATER_EQUAL "3.14". Verified locally: `ctest -R cmake_fetch_content` passes with CMake 4.1, and the CMP0169 deprecation warning that appeared before this change is gone. Closes #5714 item 5. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Make the CMake integration-test wrappers consistent The six tests/cmake_* integration-test wrappers had drifted: - Only cmake_import and cmake_import_minver forwarded -A "${CMAKE_GENERATOR_PLATFORM}" to the inner configure, and none forwarded -T "${CMAKE_GENERATOR_TOOLSET}". The Windows workflow configures the outer build with -A Win32 -T ClangCL, so without forwarding, the inner projects of cmake_add_subdirectory, cmake_fetch_content, cmake_fetch_content2 and cmake_target_include_directories built with the generator defaults instead of matching the outer build's platform and toolset. Forward both consistently from all six wrappers. - cmake_fetch_content and cmake_fetch_content2 passed -Dnlohmann_json_source to their inner projects, which never read it (CMake warns "manually-specified variables were not used"); the inner projects fetch their own copy of the library instead. Drop it. - tests/CMakeLists.txt set JSON_FORCED_GLOBAL_COMPILE_OPTIONS from the matching environment variable but never read the cache variable again; the lines right below it read $ENV{JSON_FORCED_GLOBAL_COMPILE_OPTIONS} directly, like the LINK_OPTIONS counterpart already does. Remove the dead set(). This changes which platform and toolset the Win32 and ClangCL CI jobs build the four newly-forwarding wrappers' inner projects with, which may surface new failures there; CI has to confirm those jobs. Verified locally with Ninja (empty -A ""/-T "" is accepted): all 12 cmake_* tests still pass, and the inner fetch_content configures no longer warn about the unused variable. Closes #5714 item 6. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Turn the #972 fifo_map regression test into a real test The #972 regression test in unit-regression1.cpp only built a my_json array from a string literal (the original crash) and had no CHECK, so the fifo_map object type it exists to demonstrate was never exercised. Meanwhile the docs recommend fifo_map for keeping object keys in insertion order (object_order.md, template_parameters.md), and nothing tested that recommendation. Extend the section: after the original array assignment, parse an object with my_json::parse() (not via the "..."_json UDL, which returns a plain nlohmann::json and would exercise the cross-basic_json conversion constructor instead of the parser's own key insertion - and, as tried locally, does not keep fifo order for this stateful comparator) and check that dump() keeps insertion order, and that it survives erase() and inserting a new key. Also narrow thirdparty/fifo_map off the include path of every other test-* target: it was a PUBLIC include directory of test_main, even though unit-regression1.cpp is its only user. Add a small fifo_map_include INTERFACE library with that include directory and attach it to test-regression1 only via json_test_set_test_options(). Verified locally (test-regression1_cpp11, default build and -fsanitize=address,undefined): the new checks pass; `git grep fifo_map tests` still only finds unit-regression1.cpp and the vendored header. Closes #5714 item 7. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Document the vendored doctest.h patch; fix stale doctest_compatibility.h comments tests/thirdparty/doctest/doctest.h is doctest 2.4.12, imported in #4771. Two weeks later, #4801 hand-edited translateActiveException() to declare "String res;" inside the translator loop instead of before it, so a translator that does not match does not leave a previous translator's result in "res" for the next iteration to see. Nothing recorded this, so re-vendoring doctest.h from upstream would silently drop the fix. Add a comment at the patched site naming the version, the PR and the reason, so a future re-vendor knows to re-apply it. Also fix two stale comments in doctest_compatibility.h: - The DOCTEST_THREAD_LOCAL comment referenced Xcode 6/7, which is no longer supported; reword it to explain why the define must stay regardless (it keeps doctest's own thread_local usage out of the way of the same Clang/MinGW crash that JSON_NO_THREAD_LOCAL works around in the library, see ci_test_no_thread_local). - The <iosfwd> include's comment justified it with tests that define "private" as "public"; no test under tests/src does that any more (removed by #2352). Reword the comment instead of dropping the include, since confirming it is safe to drop needs the full CI matrix including MSVC 2015+. Verified locally that tests/src/unit-readme.cpp still builds and passes 17/17 with these headers. Closes #5714 item 9. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Stop compiling unit-wstring.cpp out entirely on classic ICC tests/src/unit-wstring.cpp wrapped the whole file in #ifndef __INTEL_COMPILER, with the comment "ICPC errors out on multibyte character sequences in source files". The ci_icpc job (intel/oneapi-hpckit:2023.2.1) still exists, so that job ran none of the wstring/u16string/u32string input adapter tests, including the malformed-input checks #5704 (open) extends. Only 9 lines contained non-ASCII bytes: the three *_is_utf16()/ *_is_utf32() probe functions, and three std::wstring/u16string/ u32string literals plus their narrow-string dump() expectations. Rewrite all of them with \u/\U escapes in the wide/u16/u32 literals and \x escapes (split into separate string-literal tokens so a following byte is never read as part of the same hex escape, e.g. "\xE1\x83\x85" "a") in the narrow ones. Remove the #ifndef __INTEL_COMPILER/#endif guard along with it. The *_is_utf16()/*_is_utf32() probes compared a raw multibyte literal against an escape-based one to detect a compiler that misreads the source file's encoding; with no raw literals left to misread, the comparison is now tautological, so drop the probes and the "if" guards around each SECTION's body instead of leaving them in as dead checks. The same non-ASCII-in-source-and-in-a-narrow-comparison pattern existed once more in unit-deserialization.cpp's "Using _json with char8_t literals #4945" test: a raw emoji character in a u8R"(...)" literal, guarded by a check_utf8() that returned false for ICC (same reason) and for Windows without the active UTF-8 code page. Rewrite the literal with a \U escape and compare it against a \x-escaped expectation instead of a second raw literal, and drop check_utf8() and the now-unused <windows.h> include along with the guard. Verified locally (clang, -std=c++11 and -std=c++20, -fsanitize=address,undefined, and a plain build): test-wstring keeps 18/18 assertions and unit-deserialization keeps 466/466 (c++11) and 477/477 (c++20) assertions, matching this branch before the change exactly - no coverage was gained or lost, only the source-encoding dependency was removed. ci_icpc has to confirm classic ICC actually builds and passes test-wstring now; if it does not, that is a real finding, not a reason to restore the guard. Overlaps #5704 (open), which edits unit-wstring.cpp inside the previously-guarded region (an include near the top, checks in the invalid-string sections, and a new section at the end). Closes #5713 item 5. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
|
c261578431 |
Deduplicate binary reader/writer helpers and fix stale comments (#5730)
* Fix stale and missing comments in binary_writer The doc block of write_number() ended up above the byte_swap() helpers added in #5286, about 80 lines from the function. It was also a plain comment that Doxygen skips, said "write a number to output input", and left BON8 out of the big-endian formats. Move it back onto write_number() as a /*! block and fix the text. write_bson() documented "@pre j.type() == value_t::object", but it throws type_error.317 for every other type, and to_bson() relies on that. Document the exception instead. Explain why the CBOR binary subtype is always written with a 0xD8..0xDB head and never in the one-byte tag form: binary_reader with cbor_tag_handler_t::store only keeps those heads as a subtype, so switching to write_cbor_head() would break round trips for subtypes 0..23. Also fix the grammar of the to_char_type comment. Comments only; no change in behavior, API or ABI. Part of #5710 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Merge the duplicated UBJSON/BJData integer marker ladders write_number_with_ubjson_prefix() (unsigned and signed overloads) and ubjson_prefix() (number_integer and number_unsigned cases) each picked the UBJSON/BJData integer marker (i, U, I, u, l, m, L, M, H) with their own independent if/else ladder, and the values beyond 64 bits were handled by a second, tag-dispatched pair of ladders. An optimized container announces the marker of its first element via ubjson_prefix() and then writes every element through write_number_with_ubjson_prefix(), so the two had to be kept in lockstep by hand across four call sites. Replace all of that with one ubjson_integer_prefix() built on value_in_range_of<T>, and one write_ubjson_integer_payload() that writes the value (or, for 'H', the decimal digits) for a given marker. write_number_with_ubjson_prefix() and ubjson_prefix() keep their signatures and now just call these two helpers. Behavior, the public API and the ABI are unchanged. Verified with a new regression test covering scalars and $-optimized arrays/objects at every int8/uint8/int16/uint16/int32/uint32/int64/uint64 boundary for to_ubjson/to_bjdata (both use_size/use_type settings), and by diffing to_ubjson/to_bjdata output before and after over the json_test_data corpus (bit-identical). Part of #5710 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Remove dead get_char parameters in binary_reader The non-recursive rewrite of the binary readers (#5505, #5506, #5507) left parse_cbor_internal()'s and parse_ubjson_internal()'s get_char parameters dead: parse_cbor_internal() has one caller and it always passes true, and parse_ubjson_internal() has one caller and it always uses the true default. Both parameters, and the @param docs describing the "reuse the last character" mode they used to select, no longer correspond to anything. Drop both parameters, initialise fetch/prefix unconditionally, and update the two call sites in sax_parse(). parse_cbor_value()'s and get_ubjson_string()'s own get_char parameters are unrelated and are left alone; both still have a false caller. Also delete a stray `@return whether a valid MessagePack value was passed to the SAX parser` doxygen block that sits directly above parse_msgpack_value()'s real doc comment, a leftover of the same rewrite. Behavior, the public API and the ABI are unchanged; these are private members of detail::binary_reader. Verified by compiling with -Wunused-parameter and running unit-cbor, unit-ubjson, unit-bjdata and unit-msgpack (offline, against the stubbed test_data.hpp). Part of #5711 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Share the IEEE half-precision decoder between CBOR and BJData binary_reader had two ~45-line copies of the IEEE 754 half-precision decoder: CBOR's case 0xF9 and BJData's case 'h'. Once formatting is normalised, the two blocks were identical except for the byte order used to assemble the 16-bit half (CBOR is big endian, BJData is little endian). Any future change to half-float decoding had to be made and kept in sync in both places. Add one get_half_float(format, little_endian) helper that does the two get()/unexpect_eof() reads, assembles the half in the requested byte order, decodes it per RFC 8949 Appendix D, and calls sax->number_float. Both cases now just call it with their byte order; the BJData case keeps its bjdata-only guard. Behavior, the public API and the ABI are unchanged. Verified with a scratch probe comparing the old and new decoders bit-for-bit (NaN by isnan()) over all 65536 wire byte pairs, in both formats, and by running unit-cbor and unit-bjdata (offline, against the stubbed test_data.hpp). Part of #5711 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Deduplicate the MessagePack unsigned-integer writer ladder The number_integer (non-negative branch) and number_unsigned cases in write_msgpack() each held their own copy of the fixint/uint8/16/32/64 ladder, kept in lockstep only by a comment ("we used the code from the value_t::number_unsigned case here"). Both copies mixed union members: the signed copy compared number_unsigned but wrote number_integer, and vice versa. Extract write_msgpack_unsigned(std::uint64_t), mirroring how write_cbor_head() already avoids the same duplication for CBOR, and call it from both cases. Each case now reads only its own active union member. Output bytes are unchanged for the default 64-bit number types. #5710 item 3 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Unify float marker selection and fix the long double compile error Four formats picked between a float32 and float64 marker through four different helper styles: dummy-argument overloads for CBOR and MessagePack, an std::is_same template for BON8, and a runtime if-chain on input_format_t for write_compact_float(). With number_float_t set to long double, to_cbor, to_msgpack and to_ubjson failed inside the library with "call to 'get_cbor_float_prefix' is ambiguous", while to_bson kept working because write_bson_double() takes a plain double. Change write_compact_float() to take the two marker bytes directly (each of its three callers already knows them at compile time) instead of an input_format_t it only forwarded, and delete the now-unused get_cbor_float_prefix(), get_msgpack_float_prefix(), get_bon8_float_prefix() and get_compact_float_prefix() helpers. Turn the two get_ubjson_float_prefix() overloads into one template. Both write_compact_float() and get_ubjson_float_prefix() now report an unsupported number_float_t with a static_assert naming the requirement, rather than an ambiguous-overload error; the assert lives in the function body, not the class scope, so to_bson with long double is unaffected. Verified with a probe basic_json<..., long double>: to_bson still compiles and round-trips, while to_cbor/to_msgpack/to_ubjson now fail to compile with the new static_assert message. This changes the text of an existing compile error for users with an unsupported number_float_t (documented as a public-API-visible change in #5710). #5710 item 1 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Deduplicate the BJData ndarray writer's dtype dispatch and drop <map> write_bjdata_ndarray() built a 12-entry std::map<string_t, CharType> on every call just to translate the _ArrayType_ name to a dtype marker (the only reason binary_writer.hpp included <map>), then mapped dtype to C++ type twice more: once as a switch for the range-check pass and once as a separate if/else chain for the write pass, with nothing checking that the two agreed. The caller also ran three at() lookups, and the callee called value.at(key) about ten more times for the same three members. Replace the map with bjdata_ndarray_type_marker(), a plain string comparison chain (a C++11 constexpr function cannot contain a switch, so this mirrors binary_reader's own static table style). Replace the switch/if-chain pair with one write_bjdata_ndarray_elements() that switches on dtype once and calls a per-type helper - write_bjdata_ndarray_element<T>() for the eight integer dtypes and write_bjdata_ndarray_float_element() for 'd' - with a dry_run flag selecting the range check or the actual write, so the two passes can no longer disagree on the type. _ArrayType_, _ArraySize_ and _ArrayData_ are now looked up once into references, and the four header marker bytes ('[', '$', '#') are written through to_char_type() like the rest of the UBJSON/BJData writer. The 'd' (single-precision) rule is left exactly as before, since #5707 is expected to change it separately. Verified byte-for-byte identical output before/after for every dtype (including the Draft 2/Draft 3 'byte' fallback and the use_count/ use_type combinations) via a standalone probe, plus round-tripping through from_bjdata(). Overlaps #5707, which is expected to touch the 'd' dtype case, and #5518, which is expected to move the write_bjdata_ndarray() call site. #5710 item 4 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Assert that write_bson_document() consumes every calc_bson_sizes() entry calc_bson_sizes() and write_bson_document() are a hand-synchronized pair of passes over the same object/array tree, introduced by #5553: the size pass appends to nested_sizes in visiting order, and the write pass consumes the table by position with nested_sizes[next_size++]. Nothing checked that the write pass consumed the whole table. If a future change touched only one of the two passes - for example to skip or reject an entry - every later size prefix in the document would be silently wrong. Add JSON_ASSERT(next_size == nested_sizes.size()) where write_bson_document() returns, so such a future drift between the two passes is caught immediately (JSON_ASSERT expands to nothing in release builds using assert(), and the fuzzers/tests already build with it enabled). The two passes agree today, so this changes nothing observable; it only guards against the risk described in #5710 item 5. Extracting a shared stepper for the two passes (the second half of the proposed change) is left for a follow-up: it only saves ~30 lines and the issue asks for it only if the result reads clearly, which needs more room to get right than a mechanical cleanup pass allows. #5710 item 5 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Make the BJData lookup tables static functions instead of members binary_reader held bjd_optimized_type_markers and bjd_types_map as non-static const members (12 string_t objects for the type-name table), built and destroyed on every from_cbor/from_msgpack/from_bson/ from_ubjson/from_bon8/from_bjdata call even though only from_bjdata ever reads them. They also needed the #define/decltype/#undef workaround from #3637 and two NOLINTNEXTLINE suppressions, and binary_writer already carries the same two lists in another form (is_bjdata_excluded_type_marker() and a local std::map in write_bjdata_ndarray(), the latter removed by the item-4 commit), so the excluded-marker lists could drift apart. Replace bjd_optimized_type_markers with static constexpr is_bjd_excluded_optimized_type(char_int_type), using the same ||-chain as binary_writer's is_bjdata_excluded_type_marker(). Replace bjd_types_map with a non-constexpr static bjd_type_name(char_int_type) switch returning nullptr for an unknown marker (a C++11 constexpr function cannot contain a switch). Delete both JSON_BINARY_READER_MAKE_* macros, the bjd_type pair alias, the NOLINTNEXTLINE suppressions, detail::make_array() (no longer used anywhere), and the now-unused <algorithm> and <array> includes. Update the two call sites (the ND-array excluded-type check and the _ArrayType_ lookup) accordingly, and replace unit-bjdata.cpp's "LUT arrays are sorted" section, which only checked the two tables' internal ordering, with a check of all 12 type names and all 8 excluded markers against both new functions. #5711 item 1 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Read CBOR's 1/2/4/8-byte argument through one helper parse_cbor_internal() hand-wrote the same "read a 1/2/4/8-byte big-endian unsigned integer" ladder four times over: - twice for tag numbers 0xD8-0xDB, once in the tag_handler::ignore branch and once, nearly identically, in the ::store branch (~90 lines to read one integer); - twice more for container lengths, once for array heads 0x98-0x9B and once for map heads 0xB8-0xBB, where the 1/2-byte forms called enter_array()/enter_object() directly and the 4/8-byte forms additionally went through get_cbor_container_size(). Add get_cbor_argument(std::uint64_t&), reading the width selected by current & 0x1F via the same get_number() calls as before (so EOF is reported exactly as before), and route all four sites through it: - 0xD8-0xDB now read the argument once per branch instead of switching on `current` a second time; behavior split cleanly from embedded tags 0xC0-0xD7 (tag value in the head, no argument to read), which is now its own case block that no longer has to fall into the ::store switch's "default" case to reach the same tag_pending = true; return true; outcome. - 0x98-0x9B and 0xB8-0xBB collapse into one case block each, always going through get_cbor_container_size() (harmless for 1/2-byte lengths, which already always fit). Verified byte-for-byte identical behavior before/after with a standalone probe covering embedded and multi-byte tags under all three tag_handler_t settings, a tag over a byte string (subtype path), truncated tag/length arguments of every width, and array/map lengths of every width, including the out_of_range.408 "excessive size" case: same exceptions, same messages, same chars_read, same successful results. Left the string/byte-string length ladders in get_cbor_string()/ get_cbor_binary() untouched, as noted in #5711 item 2, since #5325 is expected to touch them separately. Overlaps #5601 (adds a branch right above the embedded-tag case) and #5607 (touches the integer cases 0x18-0x1B, which share this ladder's shape in separate hunks). #5711 item 2 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Add leave_container() to match enter_container() Every container is opened through enter_container(), whose docs promise that a check placed there runs before every start event. The close side had no equivalent: the same "container_stack.pop_back(); dispatch to end_object() or end_array()" sequence was written out separately in BSON, CBOR, MessagePack, UBJSON/BJData and BON8, each copying the pattern of keeping an is_object flag around the pop_back() that would otherwise invalidate a reference to it. A check needed on close would have had to be added in five places, and a sixth copy could go unnoticed. Add leave_container() next to enter_container(), doing the same pop-then-dispatch, and replace the five sites with it. Each site keeps its own surrounding logic (BSON's check_bson_document_size() call before popping, MessagePack's is_object copy used again below, UBJSON/BJData's remaining-container handling after popping, BON8's top used again below); only the repeated pop/dispatch line pair is now shared. Verified all six binary-format unit suites and unit-regression2's deep-nesting tests (dependent count/reuse count and the bjdata ndarray depth cases) still pass, compiled with -Wall -Wextra and ASan/UBSan. Overlaps #5601, which is expected to add a sixth close site in its own skip loop; that site can route through leave_container() too once it lands. #5711 item 4 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Stop passing the input format to sax_parse() when the reader already has it binary_reader's constructor stores the format in the input_format member, and sax_parse(format, sax_, strict, tag_handler) took the same value again purely to dispatch on it. Every in-tree caller passed the same value both times (all 16 from_cbor/from_msgpack/from_ubjson/ from_bjdata/from_bon8/from_bson call sites in json.hpp, and the three public basic_json::sax_parse() overloads), so nothing was broken today, but a caller of the detail class directly (only reachable via JSON_PRIVATE_UNLESS_TESTED, as unit-bjdata.cpp already does) could pass a mismatched pair - say bjdata to the constructor and ubjson to sax_parse - and dispatch on one format while applying the other format's rules; the default-constructed input_format_t::json reader would additionally hit JSON_ASSERT(false) in exception_message() on its first error. Add sax_parse(json_sax_t*, bool, cbor_tag_handler_t) forwarding to the existing overload with the stored input_format, and switch every caller to it: the 16 from_*() sites (keeping their `// cppcheck-suppress[accessMoved]` comments) and the three basic_json::sax_parse() overloads, all of which already had the format available from their own `format` parameter. The four-argument overload is kept for anyone still calling it, now with JSON_ASSERT(format == input_format) so a mismatch fails immediately in a debug build (assert-enabled binaries, including the fuzzers and test suite) instead of misbehaving; verified with a probe that constructs a reader for one format and calls the explicit overload with another, which aborts on that assertion as expected. Removing or asserting against the constructor's input_format_t::json default, which would affect direct detail users, is left as a separate decision per #5711 item 5. Overlaps #5601, which is expected to add an AllowRecovery template parameter to sax_parse() and touch these same call sites in json.hpp. #5711 item 5 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Deduplicate UBJSON/BJData signed-count handling, drop dead ndarray checks get_ubjson_size_value()'s 'i'/'I'/'l'/'L' cases each read a differently sized signed integer and then repeated the same "reject negative with error 113" check; only 'L' additionally checked value_in_range_of for the out_of_range.408 case. Any change to that error path had to be made four times. Add get_ubjson_signed_count<SignedType>(std::size_t&), doing the read, the negative check and the range check once, and route all four markers through it. The range check is a no-op for 'i'/'I'/'l' (their values always fit std::size_t) and only live for 'L' on a 32-bit std::size_t target, matching today's behavior exactly. In the ndarray dimension-product loop, the preceding loop already returns early on any zero dimension and result starts at 1, so `i > 0` in the pre-multiplication overflow check was always true, and `result == 0` in the post-multiplication check could not be reached either: two positive factors whose product does not overflow (as the pre-check already guarantees) cannot be zero. Drop the dead `i > 0 &&` and narrow the post-check to `result == npos`, the one case the pre-check cannot rule out (an exact, non-overflowing match with the sentinel reserved for unknown-size containers), with a comment explaining why. Verified byte-for-byte identical behavior before/after with a standalone probe covering negative counts for every marker, a matching positive count, and ndarray inputs, plus the full unit-ubjson and unit-bjdata suites (same assertion counts as before this change). Overlaps #5601 (rewrites the four parse_error calls and the overflow checks touched here) and #5607/#5707 (touch neighboring lines in the same functions). #5711 item 6 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Drop redundant format parameter and dummy float argument (review) binary_reader::sax_parse(format, ...) only ever had to equal the format given to the constructor, which it asserted. With every caller already on the format-less overload, remove the four-argument overload and dispatch on the stored input_format directly. binary_reader is a detail class, so this is not a public API change. get_ubjson_float_prefix() took a value only to deduce its type; make the type an explicit template argument instead. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
|
98e00d22e5 |
Cut test suite runtime in binary roundtrips and integer sweeps (#5519)
* Cut test suite runtime in binary roundtrips and integer sweeps The Linux CI jobs pass --no-skip, so skip() does not help there. Parse each corpus file once in the binary roundtrip loops instead of four times. Sample the 16-bit integer ranges with stride 7 (still hits every low byte) and always keep the endpoints. Also drop the 5M-node parse test to 500k, which still covers the non-recursive destructor, and move jeopardy.json into its own skipped test so the cheaper binary-format size checks actually run. See #5418. Signed-off-by: ayush-singh-0601 <singhayush062006@gmail.com> * Drop useless int32_t casts in the sampled integer loops ci_test_gcc compiles with -Werror=useless-cast. On that compiler int32_t is int, so static_cast<int32_t> of the loop bound is an error. The bounds are already int, and the sampled values do not change. Signed-off-by: ayush-singh-0601 <singhayush062006@gmail.com> * Revert unit-binary_formats.cpp to develop and fix comment Revert tests/src/unit-binary_formats.cpp to its develop state. The test-case split made valgrind jobs slower instead of faster, because the cheaper corpus files (canada/twitter/citm/sample) now ran under valgrind where they never did before. Fix the next_integer_sample comment: the function has no 'first' parameter, so describe what the function actually does. Signed-off-by: ayush-singh-0601 <singhayush062006@gmail.com> --------- Signed-off-by: ayush-singh-0601 <singhayush062006@gmail.com> |
||
|
|
85f8b21e1c |
Add tests for uncovered code paths (#5581)
* Add tests for uncovered code paths Cover code the test suite did not reach, found from the Coveralls report of develop and a local coverage run of HEAD: - dump() of every kind of value below the bound of the recursive descent (pretty-printed objects, binary values, discarded values, scalars), and flushes of the escape and write buffers mid-string and mid-binary - the iterative comparison: objects with different keys, containers that are a prefix of each other, and elements that cannot be ordered, each both at the top level and below the nesting bound - SAX handlers that stop at any event, including the end of a nested container, in the BSON, CBOR, MessagePack, UBJSON and BJData readers - from_bson/cbor/msgpack/ubjson/bjdata returning a discarded value through the iterator and pointer overloads - JSON Patch, diff, merge_patch and update(..., true) on ordered_json - smaller gaps: get_allocator(), to_ubjson/to_bjdata into a string, value() with an unresolvable JSON pointer, integer/float comparison below the integer range and with negative fractions, conversion to a custom binary type, std::formatter::parse on a spec without '}', unescape() of a lone '~', and the callback parser's start_array() Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Cover more paths that were thought unreachable - parse_float_fast() declining malformed or inexact input, called directly since the lexer only passes well-formed numbers to it - a UTF-16 high surrogate followed by a unit above the low surrogates - self-assignment of a const_iterator - a truncated CBOR string read through non-contiguous iterators - serializing a long double under the de_DE locale, which undoes the locale's decimal point and thousands separator - values read from a binary format carrying no diagnostic positions, with and without a parser callback Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix the CI failures of the new coverage tests - declare the self-assignment reference const (misc-const-correctness) - expect the (/path) prefix that JSON_DIAGNOSTICS adds to the messages of the failing ordered_json patch operations Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Expect the byte range JSON_DIAGNOSTIC_POSITIONS adds to the patch errors Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Build the expected dump of the nested-object test with += clang-tidy (performance-inefficient-string-concatenation) reported the chain of operator+ calls that assembled the expected indented output. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Compare the BJData and UBJSON test outputs byte by byte Building a std::string from the byte vector converts each byte implicitly, which -fsanitize=integer reports for bytes of 0x80 and above (ci_test_clang_sanitizer). Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
|
cc472af13f |
Check the fuzzers' UBJSON/BJData round-trip invariants in the unit tests (#5569)
* Check the fuzzers' UBJSON/BJData round-trip invariants in the unit tests The strongest correctness checks for the UBJSON and BJData writers lived only in the OSS-Fuzz drivers: anything from_ubjson()/from_bjdata() returns must serialize with every option combination, parse back, and re-serialize stably. Those checks only run at OSS-Fuzz, so regressions surfaced days later as external reports - the same BJData assert pair was reported five times over three years, and #5494's harness change was followed by OSS-Fuzz 563659413 within a day. Add "UBJSON round-trip invariants" and "BJData round-trip invariants" test cases that run the drivers' checks on a fixed, deterministic corpus (tests/src/round_trip_corpus.hpp): integer and float boundaries, non-finite numbers, strings, binary values, optimized containers, deep nesting, the JData annotated-array matrix, and seeded random containers. They also check two properties the drivers do not: the first round trip preserves the value, and re-serializing reproduces the exact bytes. For BJData both exclude values containing a binary value, which is read back as an array of integers unless it was written as a Draft 3 optimized binary array; this carve-out is now documented in bjdata.md. Run against the headers before #5542, the BJData test fails, including on the shape from OSS-Fuzz 563659413. Also document how OSS-Fuzz reports are handled (reference them as "OSS-Fuzz: <id>", turn the reproducer into a unit test, keep drivers and unit tests in sync) in tests/fuzzing.md, and link it from the PR template and the quality assurance page. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Add the OSS-Fuzz reproducers for 474400817 and 474480402 as unit tests Following the convention added to tests/fuzzing.md, the reproducers of the two BJData fuzzer asserts tracked since January are now unit tests: - 474400817 (assert(false)): an empty object _ArraySize_ was written as the ND-array header length, which from_bjdata() could not read back. Fixed by #5455. - 474480402 (to_bjdata(j2, false, false) == vec2): a one-byte Draft 3 binary array is written in Draft 2 mode as a uint8 array and then re-serialized with the int8 marker. This is the documented exception to byte stability, not a library bug; OSS-Fuzz closed it after #5494 relaxed the harness to value stability. The test pins the exact bytes so the exception stays deliberate. The 563659413 reproducer is already a unit test (#5542). A comment also ties the existing UBJSON excessive-count test to the timeout OSS-Fuzz reported for that shape (testcase 6347769435193344). OSS-Fuzz: 474400817 OSS-Fuzz: 474480402 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix GCC -Weffc++ and -Wuseless-cast warnings in the round-trip corpus Initialize the atoms in the member initialization list, and drop the cast of the generator's result, which already is std::size_t on 64-bit Linux. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
|
918da64657 |
Keep BJData ndarray annotations that would not survive a round trip as objects (#5542)
write_bjdata_ndarray() encoded a JData-annotated object as a BJData
ND-array whenever its dimensions' product matched _ArrayData_.size(),
which lost information in two ways:
- _ArrayData_ was never required to be an array. null has size 0, any
other scalar has size 1, and iterating an object visits its values, so
e.g. {"_ArraySize_":[1],"_ArrayData_":5} was written as the array [5],
and an object _ArrayData_ came back as an array.
- The reader only restores an annotated object from an ND-array with at
least two non-zero dimensions that is not a 1xN row vector; an empty,
1-D, row-vector, or zero-sized shape is read back as a plain array. The
writer nonetheless emitted ND-array headers for these shapes, so the
annotation was silently dropped.
OSS-Fuzz issue 563659413 hit this in parse_bjdata_fuzzer: an empty binary
_ArraySize_ is written as a plain object and read back as an empty array,
after which {"_ArrayType_":"int16","_ArraySize_":[],"_ArrayData_":null}
was encoded as the ND-array header "[$I#[]" and re-read as [], failing the
harness's value-stability check.
Such objects now fall back to a plain object encoding, which round-trips.
Genuine ND-arrays (two or more positive dimensions, not a 1xN row vector)
are encoded exactly as before. Existing fallback tests that used 1-D
shapes are moved to 2-D shapes so they keep exercising the check they
were written for, and the BJData documentation is updated.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
|
||
|
|
c41152e620 |
Fall back to plain-object encoding when to_bjdata()'s _ArrayType_ annotation is not a string (#5494)
* Fall back to plain-object encoding when _ArrayType_ is not a string write_bjdata_ndarray() looked up _ArrayType_ by calling get<string_t>() directly, which throws type_error.302 when the annotation is not a string (e.g. a number, null, boolean, array, or object). Per the documented BJData ndarray contract, an object only qualifies for the compact ndarray encoding if _ArrayType_ names a known type; anything else must fall back to plain-object encoding, the same way an unknown type-name string already does. Add an is_string() check before the get<string_t>() call so a non-string _ArrayType_ takes the existing "unrecognized type name" fallback path instead of throwing. Fixes #5398. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Relax the BJData fuzzer's round-trip check from byte-exact to value-exact Fixing #5398 lets to_bjdata() proceed past the object it used to reject, which exposed a pre-existing, unrelated round-trip quirk to the fuzzer: a binary_t value serialized through the non-optimized ("$U#"-less) array encoding is parsed back as a plain array of numbers, since from_bjdata() has no way to tell "array of uint8 numbers" apart from "array of bytes" without that optimized header. Re-serializing that plain array then goes through the generic smallest-type writer, which - unrelated to this PR, and long predating it - prefers the 'i' (int8) marker over 'U' (uint8) for values that fit both, so the re-encoded bytes can differ from the original even though both decode to the same value. This is not introduced by the #5398 fix; the same divergence reproduces from a bare json::binary_t value with no _ArrayType_ annotation involved at all, on the commit immediately preceding it. A general fix would mean changing the shared UBJSON/BJData smallest-type selection that hundreds of existing tests pin to 'i' for small positive integers, which is out of scope and too risky for this PR. Update fuzzer-parse_bjdata.cpp's round-trip assertions to check that re-serializing is value-stable (from_bjdata(to_bjdata(j)) == j) rather than byte-exact, matching the guarantee BJData actually provides, and add a regression test in unit-bjdata.cpp using the exact OSS-Fuzz input that documents the behavior. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Compare dump()s instead of json values in the BJData fuzzer's round-trip check The value-stability assertion added to fix the earlier OSS-Fuzz crash (json::from_bjdata(to_bjdata(j2)) == j2) itself broke on a NaN payload: IEEE 754 NaN is never equal to itself, so operator== reports two structurally-identical trees containing a non-finite double as different -- not a round-trip bug, just NaN's ordinary non-reflexivity. dump() serializes any non-finite double the same deterministic way (as JSON null, since JSON cannot represent NaN or Infinity), so comparing dumps is stable under exactly the values that break operator==. Verified against both the original OSS-Fuzz crash input and the new one (0x68 0x68 0x7c, which decodes to a NaN), plus a local 2.5M-case random-input sweep with no failures. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
|
502e9d66f6 |
Fix to_bjdata() emitting the Draft-3-only 'B' marker in default Draft-2 mode (#5479)
* Fix to_bjdata() emitting the Draft-3-only 'B' marker in default Draft-2 mode _ArrayType_ = "byte" mapped unconditionally to the BJData type marker 'B', regardless of the requested bjdata_version. 'B' is defined only by BJData Draft 3; with the default version (draft2), this produced a stream that is invalid for Draft 2 and, unlike every other _ArrayType_, round-tripped back as a binary value instead of the original annotated object. Only accept "byte" / emit 'B' when bjdata_version selects Draft 3. Under Draft 2, fall back to the same plain-object encoding used elsewhere in this function for other invalid-annotation cases, so the value round-trips correctly. Fixes #5404. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Future-proof the Draft-3-only 'B' marker gate @gregmarr pointed out that dtype == 'B' && bjdata_version != draft3 only future-proofs by accident, since bjdata_version_t currently has exactly two values. Compare with < instead, so a later draft that keeps the 'B' marker valid does not need this gate revisited. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
|
a50c2537eb |
Reserve capped array capacity in json_sax_dom_parser::start_array() for definite-length binary arrays (#5476)
* Reserve capped array capacity for definite-length binary arrays CBOR, MessagePack, and the optimized [$type#count UBJSON/BJData form all pass an exact element count to sax->start_array(len), but json_sax_dom_parser::start_array() (and the callback variant) only used len for an overflow check against max_size() and never reserved the underlying vector, so each element triggered a reallocation cascade via emplace_back(). Reserve upfront, but cap the reservation at 16384 elements: max_size() for a std::vector is far larger than any realistic input, so an unbounded reserve(len) would let a crafted/truncated header (e.g. CBOR 0x9A + a huge uint32 count with no data) trigger a multi-gigabyte allocation attempt instead of the normal graceful parse_error. With the cap, a hostile length still fails fast with the existing parse_error, while realistic arrays get a single up-front allocation. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Make the huge-claimed-length DoS regression tests portable across size_t widths On a platform where size_t is narrower than 64 bits (e.g. 32-bit mingw/msvc x86), the previously-hardcoded huge test lengths either collide with that platform's unknown_size() sentinel (CBOR/MessagePack, both using exactly SIZE_MAX) or exceed the platform's smaller vector<json>::max_size() (UBJSON/BJData's 0x7FFFFFFF), so the header is now rejected outright (out_of_range.408) instead of being accepted and only found short of data (parse_error.110). Both are safe, bounded rejections of the hostile input; the property under test -- no attempt to allocate space for billions of elements -- holds either way. Accept both outcomes instead of pinning the 64-bit-only exact result. Also fixed an unrelated clang-tidy finding (google-readability-casting) on the functional-style std::size_t(...) casts in the neighboring "arrays of various sizes" section. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix remaining CI failures in the huge-claimed-length DoS regression tests - Apply the same google-readability-casting fix (std::size_t{N} instead of std::size_t(N)) to the "arrays of various sizes" section in unit-msgpack.cpp, unit-ubjson.cpp, and unit-bjdata.cpp; only unit-cbor.cpp had been fixed previously, since clang-tidy's build didn't get far enough to report the other three in the same pass. - json_sax_dom_parser::start_array()'s max_size() check calls JSON_THROW directly rather than going through sax->parse_error(), so unlike the scanner's own "not enough data" parse_error it is not gated by allow_exceptions=false. On a platform where a header's claimed count exceeds max_size() (e.g. 32-bit, for UBJSON/BJData's 0x7FFFFFFF test value), from_ubjson/from_bjdata(input, true, false) can therefore still throw instead of returning a discarded value. Make that assertion tolerant of either outcome, same as the main exception-catching check above it. - Guard all four "a huge claimed length..." SECTIONs with #if !defined(JSON_NOEXCEPTION), matching this test suite's existing convention for exception-dependent tests: under JSON_NOEXCEPTION, JSON_THROW never produces a catchable C++ exception at all (it aborts the process), so a section that relies on try/catch to distinguish between two acceptable outcomes cannot be expressed under that build configuration regardless of platform. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Use (std::min)(len, reserve_cap) instead of a ternary in start_array() Addresses review feedback from @gregmarr on PR #5476. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
|
4bb2de8cc5 |
Reject a nested BJData ndarray dimension vector where it is read (#5503)
get_ubjson_size_type() takes an inside_ndarray parameter saying whether it is being called for an ndarray's dimension vector, where another ndarray is not allowed. It then seeded the flag it passes down to get_ubjson_size_value() with `false` rather than with that parameter, and only consulted inside_ndarray afterwards, on the '$' branch. So on the '#' branch nothing stopped the descent: every "#[" pair of an input like "[" followed by "#[#[#[..." opened another dimension vector, several native stack frames deeper each time, and the recursion was only reported on the way back out. 100,000 pairs crash the process. This is #5104 again, in a path that has nothing to do with containers. Seed the flag with inside_ndarray, which is what get_ubjson_size_value() documents it wants: "for input, `true` means already inside an ndarray vector or ndarray dimension is not allowed". The nested '[' is then refused where it is read, so the length of the chain no longer matters. Both post-checks gain `&& !inside_ndarray`, because an ndarray was found *here* only if the flag flipped -- get_ubjson_size_value() only ever returns `true` when its initial value was `false`, as its documentation says. With that, the "ndarray can not be recursive" branch is unreachable: a recursive ndarray is now caught one level earlier, and reported as "ndarray dimensional vector is not allowed" like every other nested dimension vector. Three existing expectations move accordingly (vR2, vR4, vR6). All three now fail earlier, and all three now report the same error that vR1, vR5 and vH already reported for the same shape, which is the more consistent outcome. Everything else is unchanged: valid 1D and 2D ndarrays, optimized containers and plain arrays produce identical results, and unit-ubjson is untouched. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
|
bfb07786cd |
Fix to_bjdata() silently truncating out-of-range _ArrayData_ elements (#5473)
write_bjdata_ndarray() validated that each _ArrayData_ element matched the number kind (integer vs. float) named by _ArrayType_, but not its range. An element that did not fit the target C++ type (e.g. 256 for "uint8") was silently wrapped by the static_cast used to write it, or, for "single", silently overflowed to infinity. Range-check each element against the type named by _ArrayType_ before writing it, reusing the existing fallback path that already encodes the annotated object as a plain object for other invalid-annotation cases in this function. Fixes #5403. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
|
09b6b6b5ba |
Fix to_bjdata() emitting unparsable output when _ArraySize_ is not an array (#5455)
* Fix to_bjdata() emitting unparsable output when _ArraySize_ is not an array
write_bjdata_ndarray() never checked that _ArraySize_ is an array. The shape
is written verbatim as the header length, so a null shape emitted 'Z' and an
object shape emitted '{' after the '#', neither of which from_bjdata()
accepts, and the round-trip guarantee in the BJData docs was broken.
Both slipped through the existing validation: for null, empty() is true so
the element count starts at 0 and the per-dimension loop never runs, and for
an object the loop walks its values, which can satisfy the non-negative
integer check. When _ArrayData_ then matched that count, the writer took the
ndarray path.
Require the shape to be an array, so anything else falls back to a plain
object encoding that round-trips, as the fallback rule in the docs already
specifies.
Signed-off-by: qatcod <79017227+qatcod@users.noreply.github.com>
* Document that _ArraySize_ must be an array in the ndarray requirements
The list at bjdata.md is the exhaustive set of conditions for the ndarray
encoding, but it only implied this one through 'every entry of'.
Signed-off-by: qatcod <79017227+qatcod@users.noreply.github.com>
---------
Signed-off-by: qatcod <79017227+qatcod@users.noreply.github.com>
|
||
|
|
2f025f401e |
Throw other_error.502 when UBJSON use_type is set without use_size (#5380)
* Throw other_error.502 when UBJSON use_type is set without use_size Fixes #5321 Signed-off-by: Krishnanand G <118352827+Krishnanand-G@users.noreply.github.com> * Scope UBJSON use_type check to container branches and expand tests Signed-off-by: Krishnanand G <118352827+Krishnanand-G@users.noreply.github.com> * Re-amalgamate single_include/json.hpp The previous commit updated the split headers but the amalgamated file didn't go back through astyle before I committed it, so CI's amalgamation check caught formatting drift in json_fwd.hpp and a few noexcept clauses in basic_json, plus one doc example. None of it touches the UBJSON logic. Applied the patch CI generated to bring single_include back in sync. Signed-off-by: Krishnanand G <118352827+Krishnanand-G@users.noreply.github.com> --------- Signed-off-by: Krishnanand G <118352827+Krishnanand-G@users.noreply.github.com> |
||
|
|
9a091d2b82 |
Do not write BJData ndarrays whose size overflows std::size_t (#5362)
* Do not write BJData ndarrays whose size overflows std::size_t
write_bjdata_ndarray() multiplied the _ArraySize_ dimensions into a
std::size_t without checking for overflow. A product that wraps around
to a value that happens to match the size of _ArrayData_ passed the
length check, and the writer emitted an ndarray header announcing an
element count that cannot be represented:
{"_ArrayType_":"uint8","_ArraySize_":[9223372036854775808,2],"_ArrayData_":[]}
was encoded as 5b 24 55 23 5b 4d 00 00 00 00 00 00 00 80 69 02 5d, an
ndarray of 2^64 elements followed by no data. Reading that back throws
out_of_range.408 ("excessive ndarray size caused overflow"), so to_bjdata
produced output that from_bjdata rejects. This is reachable by parsing
untrusted JSON and re-encoding it as BJData.
Mirror the overflow check the binary reader already performs, and also
reject a single dimension that does not fit into std::size_t, which the
previous cast silently truncated where std::size_t is narrower than 64
bits. Such objects now fall back to a plain object encoding, which is
what the surrounding type and length validation already does for
annotations it cannot represent, and they round-trip unchanged.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Document when to_bjdata converts a JData annotation to an ND-array
The BJData page described the 1-D vector case as the only situation in
which an object carrying _ArrayType_/_ArraySize_/_ArrayData_ is not
written as a compact ND-array. The writer has always had several other
fallbacks -- an unknown _ArrayType_, a dimension that is not a
non-negative integer, an _ArrayData_ whose length does not match the
product of the dimensions, and elements that are not numbers of the
annotated kind -- all of which cause the value to be serialized as a
regular JSON object instead.
Spell out the conditions, including the size-overflow check added in the
preceding commit, so the documented behavior matches the implementation.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
|
||
|
|
bc48951128 |
Fix UBJSON high-precision floating-point overflow handling (#5323)
This adds a std::isfinite check to the UBJSON floating-point parsing path, throwing out_of_range.406 on overflow. This makes the UBJSON parser's behavior consistent with the normal JSON parser. Fixes #5322. Signed-off-by: AJ369ninja <abhishek.j@iitg.ac.in> Co-authored-by: AJ369ninja <abhishek.j@iitg.ac.in> |
||
|
|
fd72ecfc8c |
validate ndarray element types in write_bjdata_ndarray (#5301)
* validate ndarray element types in write_bjdata_ndarray Signed-off-by: Angadi Yashaswini <angadi@digiscrypt.com> * read ndarray elements through get<> instead of a fixed union member _ArrayType_ names the wire type, not how the value is stored: parsing keeps a non-negative integer as number_unsigned while the C++ API keeps an int literal as number_integer. Selecting the union member from the type marker therefore reads the inactive alternative for one of the two, so read through get<> instead, which dispatches on the active member. Also reject a negative _ArraySize_ entry, which is not a usable dimension, and cover the parse-built path in the tests. Signed-off-by: Angadi Yashaswini <angadi@digiscrypt.com> --------- Signed-off-by: Angadi Yashaswini <angadi@digiscrypt.com> |
||
|
|
3565f40229 | reject negative UBJSON/BJData string length (#5284) | ||
|
|
ca76c37650 |
Add iterator+sentinel tests and docs for binary deserializers (#5265)
* Add iterator+sentinel tests and docs for binary deserializers This commit extends the C++20 ranges support (iterator+sentinel pairs) to the binary format deserializers from_cbor, from_msgpack, from_ubjson, from_bjdata, and from_bson, matching what was already done for parse(), accept(), and sax_parse(). Changes: - Add istreambuf_sentinel helper to test_utils.hpp for EOF detection in tests - Add 5 new test cases that read binary files directly via std::istreambuf_iterator<char> + sentinel, without pre-buffering - Update documentation for all 5 from_* functions to document overload (3) with SentinelType parameter - All tests pass; verified against existing test suite data - Fix potential buffer over-read warning in heterogeneous iterator test Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Merge iterator+sentinel overloads and fix ambiguity/CI issues Address PR review feedback and CI failures: - Merge the separate same-type and sentinel-type iterator overloads of parse(), accept(), sax_parse(), and the five from_* binary deserializers into a single overload with SentinelType defaulted to IteratorType, as suggested in review. Applied the same simplification to the detail::input_adapter() free functions. - Fix a latent ambiguity: some compilers (e.g. GCC 4.8) unreliably SFINAE the operator!= detection for std::nullptr_t against container/string types, making calls like parse(s, nullptr, ...) ambiguous with the compatible-input overload. can_compare_ne now explicitly excludes std::nullptr_t as a SentinelType. - Use a named enable_if_t template parameter instead of an unnamed function parameter for the SFINAE guard, fixing a clang-tidy hicpp-named-parameter/readability-named-parameter failure. - Update parse.md, accept.md, sax_parse.md, and the five from_*.md pages to document the merged overload instead of separate (2)/(3) overloads, also fixing an over-160-char line that broke the documentation style_check CI job. - Rework the BSON iterator+sentinel test to parse a BSON file already present in the test suite instead of writing/deleting a temp file. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix -Wunneeded-internal-declaration for CustomSentinel in test CustomSentinel lives in an anonymous namespace (internal linkage), and the library's parse loop only ever evaluates the iterator-first direction (it != last), so the reversed-order friend operator!= was never referenced. Clang's -Weverything flags such unused internal declarations as an error. Drop the unused overload; the used direction is enough to satisfy can_compare_ne's either-order detection. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix clang-tidy hicpp-named-parameter and misc-const-correctness - Drop the unused reversed-order operator!= overload from utils::istreambuf_sentinel (only iterator != sentinel is ever evaluated) and name the remaining friend's sentinel parameter, fixing hicpp-named-parameter/readability-named-parameter. - Mark the istreambuf_iterator first/last helper variable const in the five binary-format sentinel tests, fixing misc-const-correctness. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix clang-tidy misc-const-correctness in heterogeneous sentinel test json_str is only read via .data()/.size() and never reassigned, so clang-tidy correctly flags it as const-able. Verified against the exact CI job (silkeh/clang:dev, ci_clang_tidy target) by running clang-tidy directly on this file plus the five binary-format sentinel tests touched by prior commits; all are now clean. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
|
515d994acb |
📄 adjust year (#5044)
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
|
54be9b04f0 | 📄 update REUSE (#4960) | ||
|
|
2566828ed5 |
Fix CI (#4954)
* 🚨 suppress readability-redundant-parentheses Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🚨 suppress google-runtime-float Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🚨 suppress bugprone-throwing-static-initialization Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🚨 suppress bugprone-throwing-static-initialization Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🚨 suppress bugprone-throwing-static-initialization Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🚨 suppress bugprone-throwing-static-initialization Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🚨 suppress bugprone-throwing-static-initialization Signed-off-by: Niels Lohmann <mail@nlohmann.me> * ⚗️ fix Clang build Signed-off-by: Niels Lohmann <mail@nlohmann.me> * ⚗️ fix Clang build Signed-off-by: Niels Lohmann <mail@nlohmann.me> * ⚗️ fix Clang build Signed-off-by: Niels Lohmann <mail@nlohmann.me> * ⚗️ fix Clang build Signed-off-by: Niels Lohmann <mail@nlohmann.me> * ⚗️ fix Clang build Signed-off-by: Niels Lohmann <mail@nlohmann.me> * ⚗️ fix Clang build Signed-off-by: Niels Lohmann <mail@nlohmann.me> * ⚗️ fix Clang build Signed-off-by: Niels Lohmann <mail@nlohmann.me> * ⚗️ fix Clang build Signed-off-by: Niels Lohmann <mail@nlohmann.me> * ⚗️ fix Clang build Signed-off-by: Niels Lohmann <mail@nlohmann.me> * ⚗️ fix Clang build Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🗑️ do not use macos-13 image Signed-off-by: Niels Lohmann <mail@nlohmann.me> * ⚗️ fix Clang build Signed-off-by: Niels Lohmann <mail@nlohmann.me> * ⚗️ fix Clang build Signed-off-by: Niels Lohmann <mail@nlohmann.me> * ⚗️ use more Xcode versions Signed-off-by: Niels Lohmann <mail@nlohmann.me> * ⚗️ fix Clang build Signed-off-by: Niels Lohmann <mail@nlohmann.me> * ⚗️ fix Clang build Signed-off-by: Niels Lohmann <mail@nlohmann.me> * ⚗️ fix Clang build Signed-off-by: Niels Lohmann <mail@nlohmann.me> * ⚗️ fix Clang build Signed-off-by: Niels Lohmann <mail@nlohmann.me> * ⚗️ fix Clang build Signed-off-by: Niels Lohmann <mail@nlohmann.me> * ⚗️ fix Clang build Signed-off-by: Niels Lohmann <mail@nlohmann.me> * ⚗️ fix Clang build Signed-off-by: Niels Lohmann <mail@nlohmann.me> * ⚗️ fix Clang build Signed-off-by: Niels Lohmann <mail@nlohmann.me> * ⚗️ fix Clang build Signed-off-by: Niels Lohmann <mail@nlohmann.me> * ⚗️ fix Clang build Signed-off-by: Niels Lohmann <mail@nlohmann.me> * ⚗️ fix Clang build Signed-off-by: Niels Lohmann <mail@nlohmann.me> * ⚗️ fix Clang build Signed-off-by: Niels Lohmann <mail@nlohmann.me> * ⚗️ fix Clang build Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🔨 clean up Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
|
e02de2f971 |
Update to Doctest 2.4.12 (#4771)
* ⬆️ Doctest 2.4.12 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🚨 fix warnings Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🚨 fix warnings Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🚨 fix warnings Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🚨 fix warnings Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🚨 fix warnings Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🚨 fix warnings Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🚨 fix warnings Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🚨 fix warnings Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🚨 fix warnings Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🚨 fix warnings Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🚨 fix warnings Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🚨 fix warnings Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🚨 fix warnings Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🚨 fix warnings Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🚨 fix warnings Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
|
dff2b4756c |
fix: BJData size calculation overflow (#4765)
Adds pre-multiplication overflow detection to catch cases where dimension products would exceed size_t max. The previous check only detected when overflow resulted in exactly 0 or SIZE_MAX, missing other cases. Retains the original post-multiplication check for backward compatibility. Adds tests verifying overflow detection with dimensions (2^32+1)×(2^32), which previously overflowed silently to 2^32. This prevents custom SAX handlers from receiving incorrect array sizes that could lead to buffer overflows. Signed-off-by: Ville Vesilehto <ville@vesilehto.fi> |
||
|
|
1705bfe914 |
🔖 set version to 3.12.0 (#4727)
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
|
f06604fce0 |
Bump the copyright years (#4606)
* 📄 bump the copyright years Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 📄 bump the copyright years Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 📄 bump the copyright years Signed-off-by: Niels Lohmann <niels.lohmann@gmail.com> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> Signed-off-by: Niels Lohmann <niels.lohmann@gmail.com> |
||
|
|
2e50d5b2f3 | BJData optimized binary array type (#4513) | ||
|
|
6057b31df7 |
Overwork astyle call (#4573)
* 🔧 overwork astyle call * 🔧 overwork astyle call * 🔧 overwork astyle call * 🔧 overwork astyle call * 🔧 overwork astyle call * 🔧 overwork astyle call * 🔧 overwork astyle call * 🔧 overwork astyle call * 🔧 overwork astyle call * 🔧 overwork astyle call * 🔧 overwork astyle call * Use ubuntu-latest image to run Valgrind (#4575) * 🔧 use Clang image to run valgrind * 🔧 use Clang image to run valgrind * 🔧 use Clang image to run valgrind * 🔧 use Ubuntu image to run valgrind * Use Clang image to run iwyu (#4574) * 🔧 use Clang image to run iwyu * 🔧 use Clang image to run iwyu * 🔧 overwork astyle call * 🔧 overwork astyle call * 🔧 overwork astyle call * 🔧 overwork astyle call * 🔧 overwork astyle call * 🔧 overwork astyle call * 🔧 overwork astyle call * 🔧 overwork astyle call * 🔧 overwork astyle call * 🔧 overwork astyle call * 🔧 overwork astyle call * 🎨 format code * 🔨 clean up |
||
|
|
1b9a9d1f21 |
Update licenses (#4521)
* 📄 update licenses * 📄 update licenses |
||
|
|
1825117e63 |
Another desperate try to fix the CI (#4489)
* 🚨 fix warning * 💚 update actions * 🚨 fix warning * 🚨 fix warning * 🚨 fix warning * 💚 update actions * 💚 update actions * 🚨 fix warning * 🚨 fix warning * 💚 update actions * 🚨 fix warning * 💚 update actions * 💚 update actions * 💚 update actions * 🚨 fix warning * 🚨 fix warning * 🚨 fix warning * 🚨 fix warning * 💚 update actions * 💚 update actions * 🚨 fix warning * 💚 update actions * 💚 update actions * 💚 update actions * 💚 update actions * 💚 update actions |
||
|
|
9cca280a4d | JSON for Modern C++ 3.11.3 (#4222) | ||
|
|
f56c6e2e30 | Update documentation for the next release (#4216) | ||
|
|
1ce29fa22f | Fix CI (#4160) | ||
|
|
836b7beca4 | Fix CI, again (#4083) | ||
|
|
4b2c8ce6bc | Fix some typos for n-dimensional arrays (#3767) | ||
|
|
58bd97e2b1 |
Add clang-tools to required tools for ci_static_analysis_clang (#3724)
* 💚 add clang-tools to required tools for ci_static_analysis_clang * 🚨 update Clang-Tidy warning selection * 🚨 fix Clang-Tidy warnings * 🚨 fix Clang-Tidy warnings * 🚨 fix Clang-Tidy warnings * 🚨 fix Clang-Tidy warnings * 🚨 fix Clang-Tidy warnings * 🚨 fix Clang-Tidy warnings * 🚨 fix Clang-Tidy warnings * 🚨 fix Clang-Tidy warnings * 🚨 fix Clang-Tidy warnings * 🚨 fix Clang-Tidy warnings * 🚨 fix Clang-Tidy warnings (#3738) * ⏪ revert fix * ⏪ revert fix * 🚨 fix Clang-Tidy warnings (#3739) Co-authored-by: Florian Albrechtskirchinger <falbrechtskirchinger@gmail.com> |
||
|
|
9d69186291 | 🔖 set version to 3.11.2 | ||
|
|
32242022f7 |
Minor BJData fixes (#3637)
* Replace vector/map LUTs in binary_reader with arrays * Replace string_t::npos in binary_reader |
||
|
|
f2020da0dd | 🔖 set version to 3.11.1 | ||
|
|
ce0e13ccea | 🔖 set version to 3.11.0 | ||
|
|
feef0eb595 |
Add error message if test suite cannot be found (#3585)
* 🚸 add error message if test suite cannot be found Fixes #3584 |
||
|
|
527da54dcb |
Use REUSE framework (#3546)
* 📄 add licenses * 👷 add REUSE compliance check * 📝 add badge for REUSE Co-authored-by: Florian Albrechtskirchinger <falbrechtskirchinger@gmail.com> |
||
|
|
13730235f2 |
BJData dimension length can not be string_t::npos, fix #3541 (#3543)
* BJData dimension length can not be string_t::npos, fix #3541 * handle error messages on 32bit machine * add explanation to why size can not be string_t::npos * add test cases to 32bit unit test Co-authored-by: Florian Albrechtskirchinger <falbrechtskirchinger@gmail.com> |
||
|
|
e80945da2c |
CI: Enable 32bit unit test (3) (#3532)
* Enable JSON_MultipleHeaders by default * CI: Add single-header build * CI: Enable 32bit unit test * Fix "-Wuseless-cast" warnings * Remove coverage exclusion * Fix 32bit test case |
||
|
|
48a102c2c5 |
Fix ndarray dimension signedness, fix ndarray length overflow (2); add 32bit unit test (#3523)
* Fix ndarray dimension signness, fix ndarray length overflow, close #3519 * detect size overflow in ubjson and bjdata * force reformatting * Fix MSVC compiler warning * Add value_in_range_of trait * Use value_in_range_of trait * Correct 408 parse_errors to out_of_range * Add 32bit unit test The test can be enabled by setting JSON_32bitTest=ON. * Exclude unreachable lines from coverage Certain lines are unreachable in 64bit builds. Co-authored-by: Qianqian Fang <fangqq@gmail.com> |
||
|
|
046927cc29 |
Fix nlohmann/json#3513, explain is_ndarray flag (#3514)
* Fix nlohmann/json#3513, explain is_ndarray flag * add test for ndarray size following H |
||
|
|
ede6667858 |
Prevent ndarray size vector from recursive use, fix nlohmann/json#3503 (#3505)
* Prevent ndarray size vector from recursive use, fix nlohmann/json#3503 * fix ci error * complete coverage * add missing coverage * fix style issue in added test |
||
|
|
41226d0a03 | prevent ndarray dimension vector from recusive array, nlohmann/json#3500 (#3502) | ||
|
|
93c9e0c703 |
Discard optimized containers with negative counts in UBJSON/BJData (#3491,#3492,#3490) (#3500)
* Discard optimized containers with negative counts in UBJSON/BJData (#3491,#3492,#3490) * fix msvc error * update unit tests for negative sized containers * use a loop to test 0 ndarray dimension * throw an error when count is negative, merge CHECK_THROW_AS and _WITH with _WITH_AS |
||
|
|
6a7392058e |
Complete documentation for 3.11.0 (#3464)
* 👥 update contributor and sponsor list * 🚧 document BJData format * 🚧 document BJData format * 📝 clarified documentation of [json.exception.parse_error.112] * ✏️ adjust titles * 📝 add more examples * 🚨 adjust warnings for index.md files * 📝 add more examples * 🔥 remove example for deprecated code * 📝 add missing enum entry * 📝 overwork table for binary formats * ✅ add test to create table for binary formats * 📝 fix wording in example * 📝 add more examples * Update iterators.md (#3481) * ✨ add check for overloads to linter #3455 * 👥 update contributor list * 📝 add more examples * 📝 fix documentation * 📝 add more examples * 🎨 fix indentation * 🔥 remove example for destructor * 📝 overwork documentation * Updated BJData documentation, #3464 (#3493) * update bjdata.md for #3464 * Minor edit * Fix URL typo * Add info on demoting ND array to a 1-D optimized array when singleton dimension Co-authored-by: Chaoqi Zhang <prncoprs@163.com> Co-authored-by: Qianqian Fang <fangqq@gmail.com> |