mirror of
https://github.com/nlohmann/json.git
synced 2026-09-12 03:07:57 +00:00
aa391dc0a56f8409e2e7aca6e7c9a9d766d44ce8
62
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Niels Lohmann
|
44f8ec30e9 |
Bound UBJSON optimized arrays of a valueless type (#5504)
* Bound UBJSON optimized arrays of a valueless type An element of type 'Z' (null), 'T' (true) or 'F' (false) is encoded by its type marker alone, so an optimized UBJSON array of one of those has no payload: reading an element consumes no input at all. Its declared count is therefore the only thing that decides how much is allocated, and nothing bounded it. "[$Z#l" and a four-byte count is nine bytes of input describing two billion values; #2793 reports 35 GB and 150 seconds from ten bytes, and OSS-Fuzz has an out-of-memory and a timeout report for the same shape. Every other type costs at least one byte per element, so the end of the input bounds it. 'N' (no-op) is already skipped rather than stored. Objects are not affected either: each element is preceded by its key, which costs bytes. And BJData already refuses these markers as an optimized type, so this is a plain UBJSON matter. Reject a count above 1,048,576 elements for those three types with out_of_range.408, the code this reader already uses for a declared size it will not honour. The check runs before the SAX start event, so no container is opened and then abandoned. Rejecting on the read side alone would break the guarantee that anything to_ubjson() writes can be read back, and would trip the round-trip assertion in fuzzer-parse_ubjson.cpp. So the writer falls back to the unoptimized encoding, one byte per element, for arrays of these types above the same limit. Its decision depends only on the array's size, which is identical for a value and for anything parsed back from it, so the round trip is stable. No existing test changes: the largest such count in the test suite is 65,793. The excessive-size test that already used this shape still passes, now rejected a little earlier than by the max_size() check it used to reach. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Note the 1,048,576 valueless-array limit as (1 << 20) in the docs Addresses review feedback from @gregmarr on PR #5504: spell out the binary/hex form next to the decimal count so it reads as the round power-of-two it is, matching how include/nlohmann/detail/input/binary_reader.hpp defines max_valueless_container_size. Applied in both docs/exceptions.md and ubjson.md, as requested. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
 Niels LohmannandClaude Opus 4.8
|
4a93aa4e2f |
Speed up parsing of contiguous input (numbers, strings, UTF-8) (#5283)
* Speed up number parsing in the lexer (fast paths from the fast_float/simdjson world) The number scanner converted its already-validated digit buffer with std::strtoull/std::strtoll/std::strtod. Those pull in locale and errno machinery and dominate number-heavy parsing (strtod runs at ~6 M/s). Replace them with dedicated parsers over the validated buffer: - parse_integer_unsigned / parse_integer_signed: accumulate digits with overflow detection, falling back to the float path on overflow exactly as the strtoull/strtoll round-trip check did. Overflow behavior is unchanged for narrower or wider custom number types. - parse_float_fast: Clinger's exact fast path for `double` (<=19 significant digits, |exp10| <= 22, significand < 2^53), where significand * 10^exp is exact under IEEE round-to-nearest. This is the same fast path used by fast_float/simdjson. It is bit-identical to strtod on this subset and declines (falling back to strtod) otherwise. Only `double` uses it; float and long double keep std::strtof/std::strtold via a templated overload. Measured on representative data (g++ 13, -O3): - integers: DOM parse +11%, SAX +25-34% - floats: DOM parse +37%, SAX +70% (clang: float DOM ~1.9x) No dependencies added; header-only and C++11-clean. Existing parser, lexer, conversion and deserialization unit tests pass unchanged; a 3M-value random-double fuzz matches strtod bit-for-bit. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Add SWAR bulk string scanning for contiguous input (simdjson-style) scan_string() read the input one character at a time through the input adapter and classified every byte with a large switch. For contiguous byte buffers we can instead scan 8 bytes at a time with a SWAR word test that finds the first byte needing individual handling (the closing quote, an escape, a control character, or a non-ASCII UTF-8 byte) and bulk-append the ordinary run in one go. - input adapters expose supports_bulk_scan / bulk_data / bulk_remaining / bulk_skip for provably-contiguous, same-type, 1-byte iterator ranges (raw pointers in every standard; std::string/std::vector/std::array and friends additionally in C++20 via std::contiguous_iterator). - the lexer gains a bulk_scan capability (gated on lazy_token_string so bypassing the per-character capture cannot lose error diagnostics) and a scan_string_bulk() fast path; streaming/wide/user adapters are unchanged and keep the byte-at-a-time scanner. The run contains no newline (all bytes < 0x20 are treated as special), so position bookkeeping stays exact, and error tokens are still reconstructed lazily from the consumed byte range. The SWAR special-byte test is pure uint64_t arithmetic - no intrinsics, no runtime dispatch, C++11-clean. Measured on representative data, pointer input, g++ 13 -O3 (string values discarded by accept() see the largest gains): long ASCII strings: DOM +4.5x, SAX +14x, accept +17x (to ~2 GB/s) short strings: DOM +15%, SAX +62%, accept +85% escape-heavy: DOM +31%, SAX +26%, accept +28% Same-input parity verified: 200k randomized documents (escapes, multibyte UTF-8, surrogate pairs) accept/parse identically via the contiguous SWAR path and the streaming byte path; unit lexer/parser/diagnostic-position/ deserialization/conversions suites pass unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Validate UTF-8 in the bulk string scanner (portable ~2-3x on non-ASCII text) The SWAR bulk string path stopped at the first non-ASCII byte and handed every multibyte character to the byte-at-a-time scanner, whose per-byte get()/next_byte_in_range()/add() machinery runs at roughly half the speed of validating straight from the buffer. As a result, dense non-ASCII text (CJK, emoji, accented Latin) parsed ~10-15x slower than ASCII. Fold well-formed UTF-8 into the bulk run: scan_string_bulk() now, on a non-ASCII lead byte, validates one sequence with validate_one_utf8() - which mirrors scan_string()'s per-byte switch ranges exactly (rejecting overlong forms, surrogates, and out-of-range code points) - and appends it in place, continuing until the closing quote, an escape, a control byte, or an ill-formed sequence. All error handling still defers to the byte path, so error messages and positions are byte-for-byte unchanged. Because only well-formed content is fast-pathed and every rejection falls through to the existing scanner, behavior is identical; the win is purely throughput. Measured on pointer input (accept, string values discarded): content g++ 13 clang 18 dense CJK 277 -> 648 ~605 MB/s (~2.3x) dense emoji 299 -> 857 ~702 MB/s (~2.6-2.9x) mixed 90% ASCII 246 -> 331 ~334 MB/s (~1.35x) pure ASCII unchanged (~3.2 / 4.1 GB/s) Verified: 2,000,000 randomized documents built from arbitrary bytes (overlong, surrogate, truncated, out-of-range sequences) accept/reject and parse identically via the contiguous path and the streaming byte path; lexer/parser/diagnostic-position/deserialization/conversions suites pass unchanged. Pure C++11, no intrinsics. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Route contiguous byte containers through the pointer adapter (fast paths in C++11) json::parse(std::string) - the most common entry point - did not benefit from the contiguous fast paths (bulk string scanning, UTF-8 bulk validation, memcpy for binary formats) in C++11..17: std::string::iterator is a library wrapper, not a raw pointer, and pre-C++20 there is no portable way to prove it contiguous, so supports_bulk_scan was false. Only raw pointers, string literals, and C-arrays (and, in C++20, anything modelling std::contiguous_iterator) took the fast path. Detect contiguous single-byte containers (std::string, std::vector<char>, std::vector<std::uint8_t>, std::string_view, ...) via is_contiguous_byte_ container and route them through an iterator_input_adapter built from data()/data()+size(). The generic iterator-based container overload is constrained to exclude these, so the two overloads are disjoint and there is no ambiguity (a plain competing overload loses to the greedy forwarding-reference container overload on reference binding, and a factory partial-specialization is ambiguous - both were tried and rejected). The pointer keeps the container's own element type, so char_type - and therefore all parsing behavior - is byte-for-byte identical to the iterator path (const char* for std::string, const std::uint8_t* for std::vector<std::uint8_t>); only the raw pointer additionally turns on the fast paths. Lifetimes are unchanged: the container outlives the adapter for the full parse expression, exactly as the iterators it replaces did. Measured, C++11, json::parse/accept(std::string), g++ 13: long ASCII strings: accept 201 -> 3200 MB/s (~16x), parse 174 -> 1444 dense CJK: accept 263 -> 697 MB/s (~2.6x) short strings: accept 163 -> 243 MB/s (~1.5x) Verified: char_type preserved for std::string (char) and std::vector<std::uint8_t> (uint8_t); CBOR/MsgPack round-trips from std::vector<std::uint8_t> unchanged; 1,000,000 randomized documents accept and parse identically via std::string and via std::istream; deserialization/user-defined-input/parser/lexer/conversions/diagnostic- position suites pass (20,480 assertions); warning-clean on g++ and clang in C++11/17/20. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Add optional simdutf backend for bulk UTF-8 validation (JSON_USE_SIMDUTF) The bulk string scanner validates UTF-8 straight from a contiguous buffer. The scalar validator caps at ~0.3-0.7 GB/s on non-ASCII text; a SIMD validator reaches several GB/s. Rather than hand-rolling SIMD UTF-8 validation (easy to get subtly wrong - a from-scratch SSE attempt rejected valid CJK), wire in the vetted simdutf library behind an opt-in switch. simdutf is not header-only (it ships simdutf.cpp and uses runtime CPU dispatch), so it is not vendored: defining JSON_USE_SIMDUTF includes <simdutf.h> and routes the bulk validator through simdutf::validate_utf8; the project supplies and links simdutf. Undefined (the default), nothing external is included and the portable C++11 scalar path is used, so the library stays header-only and its baseline behavior is unchanged. Design keeps behavior identical either way: - scan_string_bulk() now finds the run up to the next quote/escape/control byte (non-ASCII allowed) and validates it in one shot; on the rare validation failure it recomputes the exact valid prefix with the scalar helper, so ill-formed input still falls through to the byte path and is reported at the same position with the same message. - the per-sequence scalar path is factored into scalar_string_bulk_run() and is the default backend; the refactor is behavior-preserving and does not change scalar throughput. Verified: default and JSON_USE_SIMDUTF builds accept/reject/parse identically across 2,000,000 arbitrary-byte documents and 1,000,000 mixed-escape/UTF-8 documents (differential fuzz vs the streaming byte path); lexer/parser/diagnostic-position/deserialization suites pass under both configurations (20,188 assertions with the backend enabled); warning-clean on g++ and clang, C++11 and C++20, both configurations. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Add a contiguous fast path for scanning numbers scan_number() reads a number one character at a time through the input adapter (get()) and appends each byte to token_buffer (add()) before converting. For contiguous input, the per-character get()/add() overhead dominates: it is roughly two thirds of the time spent on number-heavy parsing, far more than the value conversion itself. Add scan_number_bulk_contiguous(), which parses the whole number token straight from the input buffer: it validates and classifies the extent with the same grammar as scan_number()'s state machine, materializes token_buffer in one copy (substituting the locale decimal point exactly as scan_number() does), advances the adapter, and reuses the shared convert_number() tail. On anything it does not recognize as a well-formed number it makes no state change and returns token_type::uninitialized, so the caller falls back to scan_number(), which then produces the exact diagnostic. Errors and their positions are therefore unchanged. The conversion tail is factored out of scan_number() into convert_number() so both scanners share it; the fast path is selected by tag dispatch on the existing bulk_scan capability, so streaming/wide/user adapters are unaffected. Measured on pointer input, g++ 13 -O3: - integers: parse +65%, accept +98% - floats: parse +39%, accept +70% Verified: 2,000,000 randomized number documents (including overflow-range integers, long digit strings and %.17g doubles) parse identically via the contiguous path and the streaming byte path, matching value, type and round-trip text; the locale suite and existing parser/lexer/conversions/ deserialization tests pass; a new "lexer number fast path" test checks contiguous-vs-streaming parity, token classification, and that malformed numbers are rejected identically. Pure C++11, no intrinsics. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Move byte-level scan/parse helpers out of lexer.hpp lexer.hpp had grown by ~600 lines of byte-level helpers that have no dependency on the lexer's template parameters and clutter the state machine. Move them, unchanged, into two focused headers as free functions in namespace detail: - number_parse.hpp: parse_integer_unsigned/parse_integer_signed (now templated on the number type) and parse_float_fast (Clinger's exact double fast path, with the decimal point passed as an argument instead of read from a lexer member). - string_scan.hpp: the SWAR string helpers (is_string_special, swar_string_special, find_string_special, validate_one_utf8, scalar_string_bulk_run) and the backend-dispatched string_bulk_run, including the optional simdutf include and find_string_delimiter. lexer.hpp now includes these and calls the free functions; the methods that touch lexer state (scan_string, scan_number, scan_string_bulk, scan_number_bulk_contiguous, convert_number) stay put. This is a pure code move with no behavior change: lexer.hpp drops from 2357 to 1934 lines, the now-unused <cstdint>/<cstring>/<limits> includes are removed, and the free-function form makes the SWAR helpers reusable elsewhere (e.g. the serializer's string escaping). Verified: default and JSON_USE_SIMDUTF builds compile; 2,000,000 number and 2,000,000 arbitrary-byte-string differential-fuzz documents parse identically to before; lexer/parser/conversions/deserialization/locale/ diagnostic-position suites pass (20,576 assertions); warning-clean on g++ and clang in C++11/17/20; the amalgamation regenerates and passes check-amalgamation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix clang-tidy findings and document JSON_USE_SIMDUTF in the nav - number_parse.hpp: use std::array for the powers-of-ten table (avoid-c-arrays) and `auto` for the cast-initialized result (modernize-use-auto), matching the codebase style (cf. the serializer's utf8d table). Indexing casts keep the -Wsign-conversion build clean. - add JSON_USE_SIMDUTF to the mkdocs navigation so the macro page is reachable. No behavior change; clang-tidy is clean on the new headers and the amalgamation is regenerated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix number fast path for custom string types without assign() The contiguous number fast path materialized token_buffer with token_buffer.assign(data, len), but string_t is only required to provide the minimal interface the rest of the lexer uses (push_back, append, clear, operator[], ...). Custom string types such as the test's alt_string do not implement assign(), so scan_number_bulk_contiguous() failed to compile for them (unit-alt-string), breaking the gcc/clang standards and old-compiler CI jobs. reset() already clears token_buffer, so fill it with append() - which alt_string and std::string both provide and which the string fast path already relies on - instead of assign(). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Satisfy clang-tidy: parenthesize math and drop unused forwarding reference The CI clang-tidy (newer than the locally available version) reported two additional checks on the new code: - readability-math-missing-parentheses: parenthesize the (a * b) + c digit accumulations in number_parse.hpp. - cppcoreguidelines-missing-std-forward: the contiguous-byte-container input_adapter overload took a forwarding reference but only reads data()/size() and never forwards it. It is already disjoint from the generic container overload via SFINAE, so a plain const& is correct and clearer (and keeps the container alive for the whole parse just as before). No behavior change; char_type and routing are unchanged (std::string and std::vector<std::uint8_t> still take the pointer adapter with char/uint8_t char_type), CBOR/MsgPack round-trips and the 2M number fuzz still pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Use std::from_chars (Eisel-Lemire) for float conversion when available The Clinger fast path is exact only for the "easy" subset (<=19 significant digits, |exp10| <= 22); high-precision and scientific floats fall through to strtod, where the failed Clinger attempt actually makes parsing a net loss. std::from_chars implements the Eisel-Lemire algorithm in modern standard libraries: locale-independent, correctly rounded, and fast over the whole value range. convert_number() now tries parse_float_from_chars() first (guarded by __cpp_lib_to_chars, so C++11 and libc++-without-float-support keep the Clinger + strtod path unchanged), then Clinger, then strtof. from_chars is used only when it consumes the entire token; a partial parse means a non-'.' locale decimal point, and an under-/overflow (result_out_of_range) also declines - in both cases the existing strtod fallback supplies the exact value and the well-defined +/-inf/0 the parser expects, side-stepping the P4168 divergence between implementations. float and long double now get the fast path too (Clinger was double-only). Measured, C++17, g++ 13 -O3, json::parse/accept: - canada-style floats: ~unchanged (Clinger already covered them) - high-precision (17 digits): parse 2.1x, accept 2.5x - scientific (17 digits + exp): parse 3.6x, accept 4.1x Verified: C++11 (Clinger/strtod) and C++17 (from_chars) parse every value - including subnormals, boundary values, and 1e9999/1e-9999 over-/underflow - to bit-identical results; 2M number-fuzz clean; conversions/deserialization/ locale/number-fast-path suites pass in both C++11 and C++17; clang-tidy clean; warning-clean on g++ and clang in C++11/17/20. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Guard from_chars use on JSON_HAS_CPP_17, not just __cpp_lib_to_chars libstdc++ 15 defines __cpp_lib_to_chars even in C++14 mode (via bits/version.h pulled in by other headers), but <charconv> is only included under JSON_HAS_CPP_17. That made parse_float_from_chars() reference std::from_chars without the header in C++14 builds, breaking gcc-latest, icpx, and the offline-testdata jobs. Gate the use on JSON_HAS_CPP_17 && __cpp_lib_to_chars so it matches the include condition exactly; C++11/14 always take the scalar fallback. Verified by forcing __cpp_lib_to_chars in a C++14 build: the guard suppresses std::from_chars and it compiles. C++17 behavior is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Disable Clinger float fast path under extended FP precision (x87) The contiguous number fast path uses a Clinger-style exact algorithm (significand * 10^scale in double arithmetic), which is only correctly rounded when double operations are evaluated in true 53-bit precision. On the x87 FPU used by 32-bit x86 (FLT_EVAL_METHOD == 2) the single multiply/divide is computed in 80-bit and then double-rounded to double, so a small fraction of values land 1 ULP off. This surfaced as test-cbor_cpp11 and test-msgpack_cpp11 failing on the mingw (x86) job for regression/floats.json: the C++17 builds pass because they take the correctly-rounded std::from_chars path, while C++11 falls back to parse_float_fast(). A 5M-sample check over shortest round-trip decimals reproduces it: 0 divergences with 53-bit doubles, ~1 in 25 000 with 80-bit intermediates; declining to std::strtod fixes all of them. Guard parse_float_fast() on FLT_EVAL_METHOD so it declines whenever the platform evaluates doubles in extended precision, letting the caller use the correctly-rounded std::from_chars / std::strtod path instead. On mainstream x86-64/ARM64 (FLT_EVAL_METHOD == 0) the fast path is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Guard <charconv> include with __has_include for GCC 7 GCC 7 sets __cplusplus to the C++17 value under -std=gnu++1z, so JSON_HAS_CPP_17 is defined, but its libstdc++ ships no <charconv> header (added in GCC 8; floating-point from_chars in GCC 11). The unconditional "#if defined(JSON_HAS_CPP_17) #include <charconv>" therefore failed to compile there: "fatal error: charconv: No such file or directory" in the ci_test_compilers_gcc (7) job. Wrap the include in __has_include(<charconv>), mirroring the library's existing handling of <version> and <filesystem> in macro_scope.hpp. When the header is absent, __cpp_lib_to_chars stays undefined and parse_float_from_chars() takes its scalar fallback, so the from_chars use site (already gated on __cpp_lib_to_chars) is never reached. GCC 8-10, which have <charconv> but no floating-point from_chars, are unaffected: they include the header but still take the fallback. GCC 11+ is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Restore the column when ungetting a newline The contiguous number fast path never reads the character that terminates a number token, while scan_number() reads it and then ungets it. When that character is a newline, get() has already cleared chars_read_current_line, and unget() could only restore lines_read - leaving the column at 0. The two paths therefore reported different columns for the same document: json::parse("[01\n]") -> line 1, column 3 json::parse(stringstream) -> line 1, column 0 Remember the column the newline was read at so unget() can restore it. Both paths now report the position the offending token actually starts at, which also fixes the pre-existing column-0 artifact for streaming input. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Extend the bulk scan fast paths to sized sentinels supports_bulk_scan required IteratorType and SentinelType to be the same type, which excluded std::counted_iterator paired with std::default_sentinel_t - the combination #5268 had already enabled for the memcpy fast path. Such input fell back to the byte-at-a-time scanner even though it is contiguous and its remaining length is computable in O(1). Factor the "distance is computable in O(1)" test into sentinel_is_sized and use it for iterator_is_contiguous, supports_seek, and supports_bulk_scan alike, and share the std::ranges::distance/std::distance dispatch through a remaining_count() helper. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Document JSON_USE_SIMDUTF on the macro overview page The macro was only listed in the API macro index; add it to the supported macros overview alongside the other JSON_USE_* macros, and note that it selects between two definitions of the same inline function and so must be defined identically in every translation unit. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Amalgamate source code Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Cover the counted-iterator bulk scan paths Sized sentinels newly reach the bulk string/number scanners and the seek-based token reconstruction, so exercise both: - diagnostics that quote the offending token, which are rebuilt from the consumed input via copy_consumed_range() - inputs whose count ends before the underlying buffer does, including a closing quote that exists only behind the count, a cut inside an 8-byte SWAR stride, and a cut inside a UTF-8 sequence Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix the SentinelType example on iterator_input_adapter The comment offered "a C++20 sentinel or counted_iterator" as examples of a SentinelType, but std::counted_iterator is the IteratorType - the sentinel it pairs with is std::default_sentinel_t. #5268 corrected the same wording in the API documentation and left the code comment behind. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Do not discard the parse result in the error position check json::parse is declared warn_unused_result, and CHECK_THROWS_WITH_AS evaluates its expression as a discarded statement, so the assertion broke the -Werror builds (GCC -Werror=unused-result, MSVC C4834 under /WX). Compare against the helper that already captures the message instead. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Lock the two number grammars together with a parity test The JSON number grammar is encoded twice: as the scan_number() state machine and as the contiguous fast path. The fast path declining on anything it does not recognize keeps most divergence harmless, but if it ever accepted something the state machine rejects the result would be a silent correctness bug, and the existing test only pinned a hand-written list of numbers. Enumerate every string of length 1..4 over "01.eE+-" (2800 tokens) and require both paths to agree on the parsed value and on the exact error message. Verified to fail if the fast path's grammar is perturbed. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Skip token_buffer for integers on the contiguous path An integer token does not need token_buffer: the number_integer and number_unsigned SAX callbacks take only the value, and the overflow diagnostic rebuilds the text from the input via get_token_string(). Convert straight from the input buffer and materialize the token only for the floating-point tail, which still needs a NUL-terminated buffer for strtod. JSON_DIAGNOSTIC_POSITIONS derives a number's start position from get_string().size(), so the copy is kept when that is enabled. The integer dispatch is factored into convert_integer() and shared with convert_number(), so both scanners keep using one implementation. Integer-heavy input, 400k values, -O3: parse accept gcc 16 +14% +23% clang +15% +22% Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Amalgamate source code Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Cover the bulk string and UTF-8 scanners These paths had no dedicated tests and rested on differential fuzzing only. Add three sections, all comparing the contiguous scanner against the byte-at-a-time one on the parsed value and on the exact error message: - every string of length 1..3 over an alphabet of ordinary ASCII, both specials, a control byte, escape characters, UTF-8 lead and continuation bytes, and a byte that is never valid - each at offset 0 and offset 9, so the bulk scanner sees them with and without a run behind them - every kind of run-ending byte at each offset across two 8-byte SWAR words, so multibyte sequences also straddle the word boundary - the boundaries of every range validate_one_utf8() recognizes: shortest and longest encodings, overlongs, both ends of the surrogate block, U+10FFFF and just past it, and truncated sequences Verified to fail if the bulk validator accepts surrogates, and if the SWAR word test stops detecting control characters. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Format the new string fast path test with astyle The pinned astyle expands a braced-init-list used as a range-for range onto several lines; hoist the two offsets into a named vector instead, which reads better and leaves nothing for astyle to reformat. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix shadowed locals and guard the exception-dependent tests Two problems in the tests added for the bulk scanners, both found by CI: - the inner `const json j` in the counted-iterator diagnostics shadowed the one declared at test-case scope, which -Wshadow rejects on GCC and clang and C4456 rejects on MSVC under /WX; rename them - the new parity checks parse deliberately invalid input, which calls std::abort() rather than throwing when JSON_NOEXCEPTION is defined, so they would have crashed the no-exception build; guard them the way the other tests do json::accept() does not abort, so the UTF-8 range assertions stay compiled without exceptions and keep covering validate_one_utf8() there. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Gate the C++20 iterator classification on JSON_HAS_RANGES Making supports_bulk_scan depend on iterator_is_contiguous meant the trait is now instantiated for every adapter, not only when get_elements() is called. On standard libraries with an incomplete <ranges> that is fatal: libstdc++ 10 evaluates std::contiguous_iterator<std::counted_iterator<T*>> by calling std::to_address, which needs an operator-> its counted_iterator does not have, so satisfaction checking is a hard error rather than false. Reported by clang 14 + libstdc++ 10. JSON_HAS_RANGES already encodes exactly this ("libstdc++ < 11 has incomplete C++20 ranges", #4440), so require it for the C++20 branch. Affected toolchains fall back to the pointer-only test and the byte-at-a-time scanner, which parses identically, just without the bulk fast paths. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Amalgamate source code Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Satisfy clang-tidy in the new bulk scanner tests - give the helper lambdas an explicit std::string return type and return braced initializer lists (modernize-return-braced-init-list) - replace the C-style array of test cases with a std::vector (modernize-avoid-c-arrays) - silence pro-type-member-init on the two brace-initialized aggregates; default member initializers would stop them being aggregates in C++11 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Avoid escaped literals in the counted-iterator diagnostics list clang-tidy reads "[\"\\ud834\"]" as a literal better written raw, and the two literals written next to each other in "[\"a\x01""b\"]" as a missing comma. The concatenation was there to stop the hex escape swallowing the following character; build those documents from explicit bytes instead and use raw strings elsewhere. The byte sequences are unchanged. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Reattach convert_number's documentation @gregmarr spotted that convert_integer() was inserted between convert_number() and its doc block, leaving convert_integer() with two stacked blocks and convert_number() with none. Comment only; no code change. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address review comments on the simdutf backend string_bulk_run() had the same `return scalar_string_bulk_run(...)` in both arms of the `#if`. The simdutf arm already falls through when validation fails, so a single return after the `#endif` says the same thing. The JSON_USE_SIMDUTF example showed `#include <simdutf.h>`, which string_scan.hpp already does under the same guard; users only have to put the header on the include path and link the library, not include it themselves. Set the version history entry to 3.13.0, matching the other macro pages documenting unreleased features. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Pin the number error position against a non-newline terminator The comment claimed the reported column is the one the offending token starts at. It is the column reached after the token's last character - which is the actual point of the unget() change: a number terminated by a newline now reports what the same number terminated by a space always did. Assert that equality directly, and add a multi-character token where the start and end columns differ, so the invariant cannot be read off a single-character example. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Do not repeat the integer conversion that just failed scan_number_bulk_contiguous() converts an integer token straight from the input buffer. When the value does not fit, it materializes token_buffer and calls convert_number(), which tried the very same integer conversion again before falling back to floating point. Recording the outcome in number_type skips the second attempt. The resulting token type and value are unchanged: convert_number() reached the float tail either way. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Require a container's value_type to match what data() points at is_contiguous_byte_container accepted any type with a data() returning a pointer to a single-byte integral plus a size(). That is duck typing: the two members say nothing about size() counting the units data() points at. A type where it does not - fixed-size records, say - was routed to the pointer-based adapter and parsed as [data(), data() + size()) bytes, silently truncating input the iterator-based adapter had read in full: struct record_buffer { using value_type = std::array<char, 4>; std::string bytes; const char* data() const; // raw bytes std::size_t size() const; // in records const char* begin() const; const char* end() const; }; json::parse(record_buffer{"[1,2,3,4,5]"}); // parse error at column 3 Requiring the container's own value_type to be that same element type ties the two together. Every contiguous standard container satisfies it, so std::string, std::vector<char>, std::array<char, N> and std::string_view keep the fast path; anything else falls back to the iterator-based adapter, which is always correct. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Only compile the simdutf backend from C++17 on simdutf.h rejects anything below C++17 with an #error, so defining JSON_USE_SIMDUTF in a C++11 or C++14 translation unit did not fail with a message about simdutf being unavailable - it failed to compile at all, taking the library's C++11 support with it. Nothing caught this because no build ever compiled that path. Gate the include and both uses on JSON_HAS_CPP_17, the same way number_parse.hpp gates std::from_chars. Below C++17 the macro now has no effect and the scalar validator runs; it accepts and rejects exactly the same input, so the macro is safe to set project-wide even when some translation units use an older standard. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Test the simdutf backend in CI JSON_USE_SIMDUTF was documented and shipped but never compiled by anything in the repository, so nothing held the backend to the behavior the docs promise. Add JSON_TestSimdutf (OFF by default), which fetches simdutf and defines JSON_USE_SIMDUTF for every test target, and a ci_test_simdutf target that runs the whole suite in that configuration. Because simdutf needs C++17, the suite is built at C++11 as well, so one job covers both the scalar fallback with the macro defined and simdutf itself. The dependency hangs off test_main, whose usage requirements every test target inherits. The library target and the installed CMake package are deliberately untouched: making nlohmann_json link simdutf would put a find_dependency() in the exported package, which is a separate decision. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Warn when JSON_TestSimdutf cannot reach the simdutf backend simdutf needs C++17: without it the dependency does not even compile, and with a C++17 compiler but no C++17-or-later standard under test it builds and then goes unused. Either way the option silently did nothing useful, or broke the configure step outright. Resolve the tested standards first, then check them: when none of them can reach simdutf, skip the dependency and say so, naming which of the two reasons applies and how to fix it. The tests then run against the scalar validator, which is what would have happened anyway. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Use record_buffer::data() so clang does not flag it unneeded The record_buffer test type declares data() and size() so the is_contiguous_byte_container trait can see both and still reject the type on its value_type. data() was never called, so clang's -Wunneeded-member-function (under -Weverything -Werror) failed the C++20 build. Assert that data() points at the underlying bytes: it ODR-uses the member and documents the property the type is meant to demonstrate. Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Skip the float fast path when it cannot succeed parse_float_fast() (Clinger) needs a significand below 2^53, so it always declines once the mantissa has 17 or more significant digits. convert_number() called it unconditionally, so those numbers were walked an extra time before strtod had to run anyway. On streaming input, where scanning is byte-at-a-time and there is no compensating win, that made canada.json about 6% slower than develop. Derive the significant-digit count from token_buffer indices - the digits are not scanned again - and skip the call when it is guaranteed to decline. Both scanners pass the offset where the mantissa ends; the count only has to be corrected for a leading "0", which the JSON grammar admits nowhere else. The integer path returns before the check, so integer-heavy input is unaffected. Values are unchanged: this only avoids an attempt that would have failed. Verified bit-exact against develop over every number in canada.json, floats.json, signed_ints.json, unsigned_ints.json, small_signed_ints.json, citm_catalog.json and twitter.json, for both the contiguous and the streaming scanner. parse, streaming develop before after canada.json 19.4ms 20.5ms 19.3ms floats.json 135.9ms 131.8ms 128.0ms parse, contiguous develop before after canada.json 15.5ms 12.9ms 11.7ms floats.json 98.6ms 69.8ms 66.7ms Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
 Qatadaha Bin Matloob
|
09b6b6b5ba |
Fix to_bjdata() emitting unparsable output when _ArraySize_ is not an array (#5455)
* Fix to_bjdata() emitting unparsable output when _ArraySize_ is not an array
write_bjdata_ndarray() never checked that _ArraySize_ is an array. The shape
is written verbatim as the header length, so a null shape emitted 'Z' and an
object shape emitted '{' after the '#', neither of which from_bjdata()
accepts, and the round-trip guarantee in the BJData docs was broken.
Both slipped through the existing validation: for null, empty() is true so
the element count starts at 0 and the per-dimension loop never runs, and for
an object the loop walks its values, which can satisfy the non-negative
integer check. When _ArrayData_ then matched that count, the writer took the
ndarray path.
Require the shape to be an array, so anything else falls back to a plain
object encoding that round-trips, as the fallback rule in the docs already
specifies.
Signed-off-by: qatcod <79017227+qatcod@users.noreply.github.com>
* Document that _ArraySize_ must be an array in the ndarray requirements
The list at bjdata.md is the exhaustive set of conditions for the ndarray
encoding, but it only implied this one through 'every entry of'.
Signed-off-by: qatcod <79017227+qatcod@users.noreply.github.com>
---------
Signed-off-by: qatcod <79017227+qatcod@users.noreply.github.com>
|
||
|
Niels LohmannandClaude
|
3fa93dac65 |
docs: document std::pair/std::tuple serializing as an object for string-keyed pairs (#5442)
A std::pair or std::tuple whose every element is itself a two-element array with a string first element (e.g. std::pair<std::string, int>) serializes to a JSON object instead of a JSON array, because to_json builds the value with a brace initializer and the initializer-list object-detection rule fires. The resulting object cannot be read back into the original type and collapses duplicate keys. Document this quirk in the conversions guide, together with the unaffected cases and the idiom to force an array. Claude-Session: https://claude.ai/code/session_016cwQq8WQRFzQcGQtbJTtJg Signed-off-by: Claude <noreply@anthropic.com> Co-authored-by: Claude <noreply@anthropic.com> |
||
 whn
|
01853ed6bc |
docs: document lenient BSON input handling (#5333)
Signed-off-by: whn <142425816+Whning0513@users.noreply.github.com> |
||
 Sahil_Kamate
|
b5378e8deb |
Fix CBOR tag handlers not recognizing tags 0-5 and 21-23 (#5331)
* Fix CBOR tag handlers not recognizing tags 0-5 and 21-23 The tagged-item switch in binary_reader::parse_cbor_internal() only handled head bytes 0xC6-0xD4 and 0xD8-0xDB. Bytes 0xC0-0xC5 (tags 0-5: date/time, epoch, bignum, decimal, bigfloat) and 0xD5-0xD7 (tags 21-23: base64url, base64, base16 conversion hints) fell through to the default case and were reported as invalid bytes, even under cbor_tag_handler_t::ignore and ::store, despite being valid CBOR major-type-6 tags per RFC 8949. Add the missing case labels so the full 0xC0-0xDB range is handled uniformly. Extend the "Tagged values" test in unit-cbor.cpp to cover 0xC0-0xD7, and update the CBOR docs to state the corrected tag range. Fixes #5315 Signed-off-by: sahilkamate03 <45514385+sahilkamate03@users.noreply.github.com> * Fix stale CBOR tag docs and add store-mode binary-payload test The "Incomplete mapping" warning still listed tags 0-5 (date/time, bignum, decimal fraction, bigfloat) and 21-23 (expected conversions) as unsupported, even though they now parse correctly under cbor_tag_handler_t::ignore/store, same as 0xC6..0xD4/0xD8..0xDB. Remove those five bullets and cross-reference the "Tagged items" warning below, matching the equivalent docs fix landed independently in PR #5367. Also add a cbor_tag_handler_t::store test that wraps a binary payload (not just a string) for every byte in 0xC0..0xD7, confirming these tags are unwrapped the same way as 0xC6..0xD4 rather than mistaken for the 0xD8..0xDB binary-subtype marker syntax, per review feedback on #5331. Signed-off-by: sahilkamate03 <45514385+sahilkamate03@users.noreply.github.com> --------- Signed-off-by: sahilkamate03 <45514385+sahilkamate03@users.noreply.github.com> |
||
|
Niels Lohmann
|
9a091d2b82 |
Do not write BJData ndarrays whose size overflows std::size_t (#5362)
* Do not write BJData ndarrays whose size overflows std::size_t
write_bjdata_ndarray() multiplied the _ArraySize_ dimensions into a
std::size_t without checking for overflow. A product that wraps around
to a value that happens to match the size of _ArrayData_ passed the
length check, and the writer emitted an ndarray header announcing an
element count that cannot be represented:
{"_ArrayType_":"uint8","_ArraySize_":[9223372036854775808,2],"_ArrayData_":[]}
was encoded as 5b 24 55 23 5b 4d 00 00 00 00 00 00 00 80 69 02 5d, an
ndarray of 2^64 elements followed by no data. Reading that back throws
out_of_range.408 ("excessive ndarray size caused overflow"), so to_bjdata
produced output that from_bjdata rejects. This is reachable by parsing
untrusted JSON and re-encoding it as BJData.
Mirror the overflow check the binary reader already performs, and also
reject a single dimension that does not fit into std::size_t, which the
previous cast silently truncated where std::size_t is narrower than 64
bits. Such objects now fall back to a plain object encoding, which is
what the surrounding type and length validation already does for
annotations it cannot represent, and they round-trip unchanged.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Document when to_bjdata converts a JData annotation to an ND-array
The BJData page described the 1-D vector case as the only situation in
which an object carrying _ArrayType_/_ArraySize_/_ArrayData_ is not
written as a compact ND-array. The writer has always had several other
fallbacks -- an unknown _ArrayType_, a dimension that is not a
non-negative integer, an _ArrayData_ whose length does not match the
product of the dimensions, and elements that are not numbers of the
annotated kind -- all of which cause the value to be serialized as a
regular JSON object instead.
Spell out the conditions, including the size-overflow check added in the
preceding commit, so the documented behavior matches the implementation.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
|
||
|
Niels Lohmann
|
ad94fb01cc |
docs: document size-mismatch behavior of fixed-size conversions (#5352)
Conversions whose element count is fixed by the destination C++ type -- `std::pair`, `std::tuple`, `std::array<T, N>`, C arrays, and `std::map`/`std::unordered_map` with a non-string key -- read exactly the elements they need via `at` and never compare the JSON array's size to that number. Excess elements are silently discarded, while a shortfall throws `out_of_range.401` rather than a `type_error`. Neither direction was documented in `conversions.md`, `get.md`, or `from_json.md`. The existing warning covered only `std::array` and stated that a too-short JSON array leaves the remaining elements default-constructed with no exception thrown; that is not what happens. Generalize it to all fixed-size destinations and correct the shortfall direction. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
c2e1cc50e0 |
docs: document the complexity of ordered_map operations (#5353)
* docs: document the complexity of ordered_map operations ordered_map stores its elements in a std::vector in insertion order and has no lookup index, so emplace, operator[], at, find, count, erase, and insert are all linear scans. The documentation stated no complexity for any operation, neither in ordered_map.md nor in ordered_json.md. Add a per-operation complexity table and note the consequence: building or parsing an ordered_json object of n keys is O(n^2). Measured with -O2 -DNDEBUG for parsing a flat object of n keys, ordered_json is 5x slower than json at n=2000 and 54x slower at n=16000, with the timings quadrupling per doubling of n. Cross-reference the table from ordered_json.md and from the object order page, which recommends ordered_json without mentioning the cost. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: move the Complexity section after Member functions scripts/check_structure.py enforces a fixed section order for pages under docs/mkdocs/docs/api, in which Complexity comes after Member functions. The section had been placed right after Iterator invalidation, which made ci_test_build_documentation fail with structure/section_order. No content change beyond the move; the table columns are realigned to the narrower content. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
173f2a7407 |
Documentation review: exceptions from binary-format hardening, and tuple reference types (#5359)
* 📝 Document exceptions newly thrown by the binary-format hardening A round of binary-format input validation (#5274, #5284, #5287, #5332) added new failure modes without updating exceptions.md, and left two descriptions factually narrower than the code: - parse_error.110 said "CBOR or MessagePack"; BSON and UBJSON also throw it. Generalized, and added the BSON EOF example (#5332). - parse_error.112: added the BSON document-size mismatch example (#5287). - parse_error.113 said "while parsing a map key", but its own existing UBJSON char example already contradicted that. Broadened to cover invalid length specifications, and added the negative-string-length example (#5284). - out_of_range.408 said "of an UBJSON array or object"; CBOR now throws it too (#5274). Generalized and added both CBOR examples. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 📝 Correct which types may be referenced in a tuple extraction The note added in #5271 said a referenced type must be one the library stores "or an arithmetic type it can convert to/from". The parenthetical is wrong: is_compatible_reference_type requires an exact match against the stored types, so std::tuple<int&> is rejected by static_assert even though int converts fine as a value. Only the value case is permissive. Spell out the eight admissible types, give the int& counter-example, and separate the reference restriction from by-value conversion. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
  KBSandmanon
|
78821cd9c2 |
docs: document standards compliance and parse() vs operator>> strictness (#5326)
* docs: document RFC 8259 / JSONTestSuite compliance and parse() vs operator>> strictness The compliance story lived only in tests/src/unit-testsuites.cpp, so drive-by comparisons kept claiming the library "does not fully pass JSONTestSuite". Make it discoverable: - README: add a "Standards compliance" note stating that both nst JSONTestSuite revisions run in CI, that all mandatory y_/n_ cases pass through the strict parse() entry point, and listing the deliberate implementation-defined i_ choices (unbounded nesting, silent BOM stripping, noncharacters forwarded, strict rejection of invalid UTF-8 and lone surrogates, out_of_range.406 on numeric overflow). - features/parsing: add a "Strictness and trailing data" section documenting that parse() is strict and rejects trailing data while operator>> follows relaxed iostream semantics (parses one value and leaves the stream positioned after it) -- the single place a naive test yields a "non-compliant" result. Documentation only; no parser behavior change. Closes #5290. Signed-off-by: manon <youdie006@users.noreply.github.com> * docs: correct test-data vendoring and parse()/operator>> claims per review - README: the JSONTestSuite data is downloaded from nlohmann/json_test_data at configure time, not vendored/committed; say so. - README: only the updated suite runs y_ and n_ cases through strict parse(); the original suite's y_ cases go through operator>>. Narrow the claim. - parsing/index.md and operator_gtgt.md: note that operator>> consumes a number's terminating byte, so concatenated numbers must be whitespace-separated (1 2 works, 1true does not); structural and literal values are unaffected. Signed-off-by: manon <youdie006@users.noreply.github.com> --------- Signed-off-by: manon <youdie006@users.noreply.github.com> Co-authored-by: manon <youdie006@users.noreply.github.com> |
||
|
Niels Lohmann
|
31ba5208c8 |
docs: qualify the operator>> stream positioning guarantee (#5343)
operator>>'s notes state that it leaves the stream positioned right after the parsed value, so that concatenated JSON values can be read back to back. That does not hold when the value is a number: a number is only terminated by the character that follows it, and the lexer's unget() is simulated (it rewinds only the lexer's own bookkeeping), so that character stays consumed from the stream. Document the actual behaviour: the guarantee holds for all value types except numbers, which must be followed by whitespace. Also qualify the cross-reference on the JSON Lines page, which repeated the unqualified claim. Documentation only; the behaviour itself is tracked in #5340. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
 YingqiDuan
|
de8a099ba5 |
Document BSON interoperability and subtype-less binary round trips (#5330)
- warn about BSON marker 0x11 interoperability in both directions - explain subtype-less binary normalization to subtype 0x00 - add a round-trip test for binary values without a subtype Signed-off-by: YingqiDuan <141370165+YingqiDuan@users.noreply.github.com> |
||
 Patrick10199
|
868506dcc0 |
Fix CBOR half-float assertion bounds (#5335)
Signed-off-by: Patrick Armstrong <patrick@erpassistant.ai> |
||
|
Niels Lohmann
|
b2b47c69b1 |
📝 Document std::pair/std::tuple conversion and C++20 range-view construction (#5271)
Both had zero documentation anywhere in docs/mkdocs/. The tuple/pair gap was first spotted in the very first git-log audit pass but never turned into an actionable todo, so it persisted uncaught across four subsequent passes. - Document basic positional std::pair/std::tuple <-> json array conversion, plus #5016's reference-extraction capability (get<std::tuple<T&, T&>>() returning references into the stored array elements). - Document #5205's new json-from-C++20-range-view constructor (e.g. nums | std::views::filter(...)). Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
e9c3985f0a |
Fix documentation gaps found in a full GitHub Discussions review (#5264)
* 📝 Fix documentation gaps found in a full GitHub Discussions review Reviewed all 1008 GitHub Discussions (2020-2026) for recurring questions that better or more visible documentation would have avoided. Adds/expands documentation for ~26 distinct gaps, including: - New "Debugging" page collecting natvis, GDB pretty printer, LLDB status, and JSON_DIAGNOSTICS pointers (previously scattered/undiscoverable) - Thread-safety and schema-validation FAQ entries - StringType's char-based requirement (no wstring/u16string/u32string) - Brace-initialization-yields-arrays warning directly on the constructor reference page (previously only in the FAQ, missed by users reading the constructor docs) - std::any exclusion from get<T>(), with a manual-dispatch example - Non-string-keyed std::map serializing as an array of pairs - ordered_json compatibility with NLOHMANN_DEFINE_TYPE_* macros (already worked, was undocumented) - std::array truncation on size-mismatched conversion (no exception) - static_cast vs. get<std::optional<T>>() divergence - Recipe for omitting a std::optional field instead of emitting null - No built-in nesting-depth limit during parsing + a callback-based workaround recipe - Recipe for streaming a large homogeneous array via parser callbacks - operator>> stream-position semantics for concatenated JSON values - JSON Pointer array-vs-object creation rule for non-existing paths - CMake target name (nlohmann_json_modules) needed to link C++20 modules - ESP-IDF/PlatformIO: no official package, link to a community fork - get(key, default) as the Python dict.get() equivalent - reserve() recipe for pre-allocating array capacity - JSONC as an alias for the existing ignore_comments/ignore_trailing_commas combination (distinct from the unsupported JSON5) - items() dereferenced-element type: decltype() idiom + detail-namespace stability caveat - Various macro/type-conversion limitations (MSGPACK_DEFINE_ARRAY equivalent, char-array round-tripping, ADL serializer macro gap) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🎨 fix format Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
631e667fe5 |
Document a duplicate-object-key rejection recipe (#5259)
* 📝 Document a duplicate-object-key rejection recipe RFC 8259 leaves handling of duplicate object keys to the implementation; this library silently keeps only the last value for a repeated key. Discussion #5085 asked for an opt-in rejection mode. Decision: don't change library behavior, but document the existing parser-callback workaround instead. Adds a "Recipe: rejecting duplicate object keys" section to parser_callbacks.md, adapted from a community-contributed workaround. Fixed an off-by-one bug in the original snippet: object_start reports the depth of the object's parent, while key events inside that object report depth+1, so indexing the per-depth key set with the same depth in both places caused an out-of-bounds access on nested objects. Verified the published snippet compiles and behaves correctly for flat duplicates, nested duplicates, sibling objects sharing key names, and arrays of objects. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Cross-link the duplicate-key recipe with the existing object_t behavior docs object_t.md and features/types/index.md already document that duplicate object keys resolve to an unspecified value (RFC 8259 leaves this to the implementation). The new recipe's intro overstated this as a guaranteed "last value wins" rule, which isn't true in general -- parsing text keeps the last value, but constructing from an initializer list keeps the first. Reworded the recipe to point at object_t's "unspecified" behavior instead of asserting a specific rule, and added cross-links from both existing pages to the new recipe. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Turn the duplicate-key recipe into a standalone, compiled example Replace the inline code fence in the "rejecting duplicate object keys" recipe with a proper docs/mkdocs/docs/examples/*.cpp + .output pair, included via --8<-- like every other example on the site. The .output file was generated by running it through the project's actual example build (docs/Makefile: single_include, -std=c++11, -DJSON_USE_GLOBAL_UDLS=0) and cross-checked with `make check_output`, and the source passes the pinned astyle 3.4.13 formatting unchanged. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels LohmannandClaude Code
|
d0de6a9111 |
Document std::optional<T> direct construction limitation (#5247)
* Document std::optional<T> direct-init/copy-init limitation with null
Add regression test pinning current behavior (CHECK_THROWS_AS) in the null
section of unit-conversions.cpp with detailed comment explaining the C++
language-level cause (std::optional's own converting constructor wins
overload resolution over basic_json::operator T()).
Add a warning callout in conversions.md documenting that direct construction/
assignment of std::optional<T> from JSON null throws type_error 302, with a
clear workaround (use get<std::optional<T>>() or get_to() instead, which
correctly produce std::nullopt).
This is a limitation at the language level: there is no SFINAE path to
distinguish "called from inside std::optional's own constructor" from "direct
call", so fixing it would require breaking changes to operator ValueType().
A permanent fix belongs in the 4.0 type-strictness redesign (#3453).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Co-Authored-By: Claude Code <noreply@anthropic.com>
* Fix issue reference in std::optional test comment
Update the comment in the null section test to reference #5246 instead of
placeholder #XXXX, clarifying where the direct-init/copy-init limitation is tracked.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Use CHECK_THROWS_AS_WITH for std::optional test assertions
Update the regression tests to use CHECK_THROWS_AS_WITH instead of
CHECK_THROWS_AS to verify both the exception type and the error message.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix CI: use CHECK_THROWS_WITH_AS, the macro that actually exists
CHECK_THROWS_AS_WITH is not a doctest macro; the correct one used throughout
this test suite is CHECK_THROWS_WITH_AS(expr, message, exception_type&), with
the message before the type and the type as a reference. The previous commit
didn't catch this because it only compiled the file standalone with default
settings; this TEST_CASE only compiles under
`#if !JSON_USE_IMPLICIT_CONVERSIONS`, which is why ci_test_noimplicitconversions
was the job that failed. Verified by building and running the test in that
exact configuration (JSON_USE_IMPLICIT_CONVERSIONS=0): 14/14 assertions pass.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Run std::optional test under default implicit-conversions build too
TEST_CASE("std::optional") was guarded by #if !JSON_USE_IMPLICIT_CONVERSIONS,
so it only ever compiled in the non-default build with implicit conversions
disabled. This traces back to commit
|
||
|
Niels Lohmann
|
521a084827 |
Documentation review (#5257)
* 📝 Fix documentation gaps for 3.13.0 release (todos 138-142) - Todo 138: Add "Known issues" section to modules.md with compiler-specific troubleshooting (GCC redefinition, MSVC symbol export). Add pointer note to quality_assurance.md. - Todo 139: Document CBOR/MessagePack half-precision float encoding for NaN/Infinity (0xF9/0xCA with exact byte sequences). Explain pre-3.13.0 double-precision bug mechanism without issue citations. - Todo 140: Document CBOR negative-integer-overflow rejection (parse_error.112) for magnitudes exceeding int64_t range (already implemented in rev 1). - Todo 141: Update version history in value.md and operator[].md with behavior-change details, removing issue citations per citation policy (prose is self-contained). - Todo 142: Global sed replace of 3.12.x → 3.13.0 placeholder across all 20 documentation files. Revision 2 incorporates feedback to reduce changelog-like issue citations. Only citations that add unique troubleshooting value are retained (#5103 for GCC workaround, #3970 for MSVC symbol export). "Known issues" section follows PR #5252's visual pattern (info admonition with bold-bullet format). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 📝 Document integer type selection, type_name() invalid value, and std::optional get() fix - number_handling.md: clarify that positive/negative integers select unsigned/signed storage based on the leading minus sign (todo 143). - type_name.md: document the new "invalid" return value for corrupted JSON values (todo 145). - get.md: note that get<std::optional<T>>() was unreachable in every configuration prior to 3.13.0 due to an internal macro-guard bug, unrelated to JSON_USE_IMPLICIT_CONVERSIONS's actual effect (todo 144). Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
c034480c22 |
📝 add more docs (#5231)
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
ca49ab6123 |
Extend value to arrays when using JSON pointers (#5223)
* ✨ extend value to arrays when using JSON pointers Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 💚 avoid exceptions Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 💚 avoid exceptions Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
272411c5e6 |
Overwork project infrastructure (#5218)
* 📝 overwork project infrastructure Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🐛 fix GCC16 issue Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🐛 fix GCC16 issue Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🐛 only build module for GCC Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🐛 fix build Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 📝 fix documentation Closes #5012: fix the error_handler_t::ignore wording Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 📝 fix documentation Closes #4354: fix "Custom data source" example Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
c5b2b26fdc | 📝 fix docs (#5217) | ||
|
Niels Lohmann
|
a5f8e230ac |
Document number conversion (#5208)
* 📝 document number conversion Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 👷 fix CI Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
 Miko
|
e054d4df94 |
docs: Fix missing newline necessary in docs website (#5190)
* Fix missing newline necessary in docs website Signed-off-by: Miko <110693261+mikomikotaishi@users.noreply.github.com> * Remove std export mention, as it no longer applies Signed-off-by: Miko <110693261+mikomikotaishi@users.noreply.github.com> --------- Signed-off-by: Miko <110693261+mikomikotaishi@users.noreply.github.com> |
||
  Caillin NugentandNiels Lohmann
|
58cfecf7f7 |
Serialize enum (#5151)
* Added NLOHNMANN_JSON_SERIALIZE_ENUM_STRICT - duplicate of NLOHMANN_JSON_SERIALIZE_ENUM Signed-off-by: Caillin Nugent <caillinn@student.unimelb.edu.au> * Added failing tests for NLOHMANN_JSON_SERIALIZE_ENUM_STRICT Signed-off-by: Caillin Nugent <caillinn@student.unimelb.edu.au> * modified NLOHMANN_JSON_SERIALIZE_STRICT to throw Signed-off-by: Caillin Nugent <caillinn@student.unimelb.edu.au> * added documentation and changed readme to include NLOHMANN_JSON_SERIALIZE_ENUM_STRICT Signed-off-by: Caillin Nugent <caillinn@student.unimelb.edu.au> * ran amalgamate Signed-off-by: Caillin Nugent <caillinn@student.unimelb.edu.au> * docs(macros): add page for JSON_SERIALIZE_ENUM_STRICT - added page to nav - added links to new page where appropriate Signed-off-by: Caillin Nugent <caillinn@student.unimelb.edu.au> * refactor(macros): make JSON_SERIALIZE_ENUM_STRICT use JSON_THROW - added templated wrapper function to fix scope error in calling JSON_THROW Signed-off-by: Caillin Nugent <caillinn@student.unimelb.edu.au> * refactor(macros): make NLOHMANN_SERIALIZE_ENUM_STRICT use error code 410 - added error code 410 to docs Signed-off-by: Caillin Nugent <caillinn@student.unimelb.edu.au> * tests(macros): add test for to_json with enum value not mentioned in mapping for NLOHMANN_JSON_SERIALIZE_ENUM_STRICT Signed-off-by: Caillin Nugent <caillinn@student.unimelb.edu.au> * Apply suggestions from code review Co-authored-by: Niels Lohmann <niels.lohmann@gmail.com> Signed-off-by: Caillin Nugent <nugentcaillin@gmail.com> * fix(macro): prevent compilation error with -Werror and -Wunused-parameter with NLOHMANN_JSON_SERIALIZE_ENUM_STRICT - casted exception to void to avoid warning Signed-off-by: Caillin Nugent <caillinn@student.unimelb.edu.au> * fix(docs): add link to NLOHMANN_SERIALIZE_ENUM_STRICT docs to exception page Signed-off-by: Caillin Nugent <caillinn@student.unimelb.edu.au> * docs(macros): add example of exception throwing for NLOHMANN_JSON_SERIALIZE_ENUM_STRICT Signed-off-by: Caillin Nugent <caillinn@student.unimelb.edu.au> * refactor(macros): add more in-depth error message to NLOHMANN_JSON_SERIALIZE_ENUM_STRICT - changed error message to follow style of nlohmann/json#4989 - made description of throw wrapper more general - updated tests and example of exceptions Signed-off-by: Caillin Nugent <caillinn@student.unimelb.edu.au> --------- Signed-off-by: Caillin Nugent <caillinn@student.unimelb.edu.au> Signed-off-by: Caillin Nugent <nugentcaillin@gmail.com> Co-authored-by: Niels Lohmann <niels.lohmann@gmail.com> |
||
 George SedovandNiels Lohmann
|
cba5dc0ed8 |
New macros for the named JSON convertor generation (#4563)
* Add new macros for named conversions * Unit tests for the named conversion macros * Update the docs to include the new macros * Fix the documentation for the macros the correct maximum number of member variables is 63 * Fix CI tests * update the named macros * move the example files * update the explicit macros expansion * update documentation * fix documentation hiccups * astyle changes * add static analysis exceptions * change md header to explicit html to fit the length * Small corrections to docs Co-authored-by: Niels Lohmann <niels.lohmann@gmail.com> Signed-off-by: George Sedov <radist.morse@gmail.com> --------- Signed-off-by: George Sedov <radist.morse@gmail.com> Co-authored-by: Niels Lohmann <niels.lohmann@gmail.com> |
||
|
Miko
|
a64fc21243 |
Fix missing exports from json.cppm (#5137)
* Fix missing exports from `json.cppm` Signed-off-by: Miko <110693261+mikomikotaishi@users.noreply.github.com> * Update documentation to describe what symbols are exported Signed-off-by: Miko <110693261+mikomikotaishi@users.noreply.github.com> * Add mention of `std` symbols Signed-off-by: Miko <110693261+mikomikotaishi@users.noreply.github.com> * Add `json_literals` inline namespace to `using` statement Signed-off-by: Miko <110693261+mikomikotaishi@users.noreply.github.com> * Remove internals (`nlohmann::detail::*`) from module Signed-off-by: Miko <110693261+mikomikotaishi@users.noreply.github.com> * Omit internals (`nlohmann::detail::*`) from modules documentation Signed-off-by: Miko <110693261+mikomikotaishi@users.noreply.github.com> * Restore `nlohmann::detail` symbols with mention of the MSVC bug Signed-off-by: Miko <110693261+mikomikotaishi@users.noreply.github.com> --------- Signed-off-by: Miko <110693261+mikomikotaishi@users.noreply.github.com> |
||
 Andy Choi
|
cda9c1e3bd |
Use get instead of template get in REAMD.md and docs in non-template context (#4846)
Signed-off-by: Andy Choi <ccpong516@gmail.com> |
||
 co63oc
|
44bee1b138 |
chore: fix typos (#4921)
Signed-off-by: co63oc <co63oc@users.noreply.github.com> |
||
|
Miko
|
a3211ab3ca |
Simplify module API (#4840)
* Simplify module API Signed-off-by: Miko <110693261+mikomikotaishi@users.noreply.github.com> * Forgot missing newline Signed-off-by: Miko <110693261+mikomikotaishi@users.noreply.github.com> --------- Signed-off-by: Miko <110693261+mikomikotaishi@users.noreply.github.com> |
||
|
Miko
|
b7f741165b | Add proper C++20 module support (#4799) | ||
 Kuan-Fu, Wu
|
7421ac31a7 |
doc: Fix JSON Pointer example to use direct initialization (#4468)
Changed the example code in the documentation from copy initialization to direct initialization for `json::json_pointer`. This prevents compilation errors caused by the constructor being explicit. |
||
 chirszandNiels Lohmann
|
4b17f90f65 |
Add ignore_trailing_commas option (#4609)
Added examples and modified the corresponding documents and unit tests. Signed-off-by: chirsz-ever <chirsz-ever@outlook.com> Co-authored-by: Niels Lohmann <niels.lohmann@gmail.com> |
||
|
Niels Lohmann
|
9110918cf8 |
Fix typos (#4748)
* ✏️ fix typos Signed-off-by: Niels Lohmann <mail@nlohmann.me> * ✏️ address review comments Signed-off-by: Niels Lohmann <mail@nlohmann.me> * ✏️ address review comments Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
4424a0fcc1 |
📝 update documentation (#4723)
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
 Robert Chisholm
|
34665ae648 |
Correct typo in sax_interface.md (#4679)
binary -> binary_t Signed-off-by: Robert Chisholm <robert.chisholm@sheffield.ac.uk> |
||
|
Niels Lohmann
|
8fb5d6f92d |
Some documentation updates (#4636)
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
e90c860d53 |
Add note on derived return type for value function (#4628)
* 📝 add note on derived return type Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 📝 add note on derived return type Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
0f9e6ae098 |
Fix broken links (#4605)
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
 kimci86
|
8a882f32ed |
Generate template functions with NLOHMANN_DEFINE_TYPE macros (#4597)
* Support any basic_json type in NLOHMANN_DEFINE_TYPE_* macros Signed-off-by: kimci86 <kimci86@hotmail.fr> * Test NLOHMANN_DEFINE_TYPE_* macros also support unordered_json Signed-off-by: kimci86 <kimci86@hotmail.fr> * Simplify test about NLOHMANN_DEFINE_TYPE_ with many arguments Signed-off-by: kimci86 <kimci86@hotmail.fr> * Remove extra scope in macros tests Signed-off-by: kimci86 <kimci86@hotmail.fr> * Remove unused test class in macros tests Signed-off-by: kimci86 <kimci86@hotmail.fr> * Update documentation about NLOHMANN_DEFINE_TYPE_* macros Signed-off-by: kimci86 <kimci86@hotmail.fr> * Fix NLOHMANN_JSON_SERIALIZE_ENUM documentation Signed-off-by: kimci86 <kimci86@hotmail.fr> * Mark some variables const in macros tests, fixes clang-tidy Signed-off-by: kimci86 <kimci86@hotmail.fr> * Workaround clang 3.5 issue with const object initialization Signed-off-by: kimci86 <kimci86@hotmail.fr> * Update highlighted lines in NLOHMANN_DEFINE_TYPE_* macros examples Signed-off-by: kimci86 <kimci86@hotmail.fr> * Fix swapped macros in documentation Signed-off-by: kimci86 <kimci86@hotmail.fr> * Remove extra backslashes at the end of macros Signed-off-by: kimci86 <kimci86@hotmail.fr> * Require basic_json type in NLOHMANN_DEFINE_TYPE_* generated functions Signed-off-by: kimci86 <kimci86@hotmail.fr> * Fix typos in macros documentation Signed-off-by: kimci86 <kimci86@hotmail.fr> --------- Signed-off-by: kimci86 <kimci86@hotmail.fr> |
||
|
Niels Lohmann
|
26cfec34be | Clean up and document project files (#4560) | ||
 Michael Valladolid
|
2d42229f4d |
Support BSON uint64 de/serialization (#4590)
* Support BSON uint64 de/serialization Signed-off-by: Michael Valladolid <mikevalladolid@gmail.com> * Treat 0x11 as uint64 and not timestamp specific Signed-off-by: Michael Valladolid <mikevalladolid@gmail.com> --------- Signed-off-by: Michael Valladolid <mikevalladolid@gmail.com> |
||
 Nebojša Cvetković
|
2e50d5b2f3 | BJData optimized binary array type (#4513) | ||
 Einars Netlis-Galejs
|
af4ad7915c | Add ONLY_SERIALIZE for NLOHMANN_DEFINE_DERIVED_TYPE_* macros (#4562) | ||
|
Niels Lohmann
|
549c79ba7e | Overwork documentation (#4516) | ||
 Fallen_Breath
|
8c391e04fe |
Docs: Fix wrong code usage in the Value access section of json_pointer.md (#4255)
|
||
 Alex Prabhat Bara
|
c883fb0f17 | Fix for incorrect function name in documentation example (#4342) | ||
 Vyacheslav Zhdanovskiy
|
360ce457f4 | Add serialization-only user defined type macros (#3816) | ||
|
Niels Lohmann
|
0261bc04d3 | Fix CI (again) (#4196) |