Serializing a container serializes its elements, so dump() descended into one
call per nesting level. A value nested deeply enough exhausted the call stack
and terminated the process with a segmentation fault - no exception, nothing
the caller could catch. Parsing such a value works, as the parser is
iterative, and so does destroying one, as #1436 made destruction iterative.
Bound how far the descent goes rather than take the call stack away from it.
The first 128 levels are written by exactly the code that always wrote them,
and only below that does dump_iteratively write out what is left, keeping the
containers it has entered on an explicit stack. Serializing can therefore no
longer exhaust the stack, however deeply a value is nested, while a value
nested less deeply than the bound pays only for one comparison per container.
Writing every value that way instead measured between 2% and 20% slower - 20%
on object-heavy documents - which is why the descent is kept for all but the
values that cannot afford it. The bound costs nothing measurable: between
-1.4% and +1.2% across compact and pretty output of number, integer, string,
object-heavy, wide-object and deeply nested documents.
The output is unchanged for every value. Both ways of writing a container
emit the separator in front of every element but the first, rather than
after every element but the last, which puts exactly one between each pair
and none at the end.
This fixes#5387 for dump(). The copy constructor is fixed in #5389.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
@gregmarr's point on the fill-and-flush loop: flushing does not disturb what
the write buffer holds, so an indentation spanning several buffer-fulls only
has to be written into the buffer once and can then be handed to the adapter
as many times as needed. The loop re-filled it every time, doing work it
already knew was there.
put_indent() now fills the room left in the buffer, and if anything remains,
flushes, fills the buffer once, and re-flushes that same content. It also
returns early for a zero-width indentation, which is what the closing brace of
every outermost value asks for.
Measured over a dump(), counting memset calls and bytes inside put_indent:
indent before after
4 1 call / 4 B 1 call / 4 B
2000 2 calls / 2000 B 2 calls / 2046 B
100000 98 calls / 100000 B 2 calls / 2046 B
The wide case is now constant work rather than proportional to the indentation
width; ordinary widths are unchanged. Tests extended to cover several whole
buffer-fulls and an exact multiple of the buffer size.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Follow-up to @gregmarr's review: put_chars() was doing four unrelated jobs, so
give the two that can be made safe their own entry points.
- put_literal(): takes the literal by reference and deduces the length from the
array bound, so the 27 hand-counted lengths at the call sites can no longer
drift from the literals they describe. A literal is checked at compile time to
fit the buffer, so this path needs no write-through branch.
- put_buffer(): takes the fixed-size buffer itself rather than a bare pointer,
so the length can be checked against the buffer's own bound.
- put_indent(): memsets the indentation into the write buffer, filling and
flushing it as needed. This removes indent_string entirely, and with it both
bugs of #5186: the indentation string was grown by doubling, which is not
enough when indent_step more than doubles it (a heap over-read - dump(2000)
read 2000 bytes out of a 1024-byte string), and the grown part was filled with
a space instead of the configured indent_char. next_indent() keeps that PR's
assertion against the unsigned indentation accumulation wrapping on deep
nesting.
put_chars() keeps the two cases that are genuinely a pointer and a count: the
run-length copies out of the string being escaped, and to_chars() output.
Tests cover an indent_step wider than the write buffer, a non-space indentation
character past the old growth point, and nesting whose accumulated indentation
spans several buffer-fulls. All three fail against develop.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The "many small structural writes exceed the write buffer" subcase built
a 1100-deep nested array and dumped it to force >1024 consecutive
single-character writes through put_char (exercising the write buffer's
flush-when-full branch). dump() recurses per nesting level, so on MSVC
debug builds (smaller default stack, larger frames) this overflowed the
stack and crashed test-serialization; Linux/macOS have enough headroom to
hide it.
Replace the nesting with a flat array of 500 empty strings. Each element
emits '"', '"', ',' via put_char, so the dump is a long run of
single-character writes (1501 bytes > the 1024-byte buffer) at nesting
depth two, hitting the same flush branch without deep recursion. Library
code is unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
test-convenience failed (macOS finished first; the failure is
platform-independent) because check_escaped() calls the internal
serializer::dump_escaped() directly and then reads the output stream.
Since dump_escaped() now writes into the serializer's internal write
buffer, the bytes were still buffered and the stream was empty.
Expose flush() under JSON_PRIVATE_UNLESS_TESTED (same visibility as
dump_escaped) and flush in check_escaped() before inspecting the output.
Per-string flushing inside dump_escaped() was rejected on purpose: it
would defeat the buffering that makes object/array-heavy dumps faster.
Library behavior is unchanged (flush()'s body is identical; only its
access label moved).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Two further serialization speedups on top of the ensure_ascii=false bulk
copy, both reusing the SWAR primitives in detail/input/string_scan.hpp.
1. Internal write buffer (devirtualization). Every structural character
('{', '"', ',', ...) previously went straight to the output adapter
through a virtual call. Route all writes through put_char/put_chars
into a 1 KiB buffer that flushes in bulk; the public dump() flushes
once the top-level value is done (the recursive worker is split out as
dump_internal). Runs larger than the buffer are written straight
through, so large payloads are not copied twice. This is the dominant
cost for object/array-heavy values.
2. ensure_ascii fast path. dump_escaped previously ran the UTF-8 DFA over
every byte when escaping non-ASCII. Add find_ascii_copyable_run() (a
SWAR scan stopping at '"', '\\', < 0x20, 0x7F, and >= 0x80) so runs of
printable ASCII are bulk-copied, with the byte path handling each
escape/non-ASCII byte exactly as before.
Behavior is unchanged: dump output is byte-for-byte identical to the
previous implementation across ~20k randomized byte strings plus curated
edge cases (all escapes, control chars, 0x7F, valid multibyte,
surrogates, overlong, truncated), for object/array/pretty output, both
ensure_ascii settings, and all three error handlers, in C++11/17/20 at
-O2/-O3. New unit tests cover the buffer flush boundaries, the escape and
0x7F handling, multibyte under both settings, and invalid-UTF-8 handling.
Throughput (g++ -O3, vs the ensure_ascii=false-only baseline):
long ASCII, ensure_ascii=0 4.2x
long ASCII, ensure_ascii=1 4.1x
twitter-like objects 2.7x
dense CJK 1.8x
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The record_buffer test type declares data() and size() so the
is_contiguous_byte_container trait can see both and still reject the
type on its value_type. data() was never called, so clang's
-Wunneeded-member-function (under -Weverything -Werror) failed the
C++20 build. Assert that data() points at the underlying bytes: it
ODR-uses the member and documents the property the type is meant to
demonstrate.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
simdutf needs C++17: without it the dependency does not even compile, and with
a C++17 compiler but no C++17-or-later standard under test it builds and then
goes unused. Either way the option silently did nothing useful, or broke the
configure step outright.
Resolve the tested standards first, then check them: when none of them can
reach simdutf, skip the dependency and say so, naming which of the two reasons
applies and how to fix it. The tests then run against the scalar validator,
which is what would have happened anyway.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
JSON_USE_SIMDUTF was documented and shipped but never compiled by anything in
the repository, so nothing held the backend to the behavior the docs promise.
Add JSON_TestSimdutf (OFF by default), which fetches simdutf and defines
JSON_USE_SIMDUTF for every test target, and a ci_test_simdutf target that runs
the whole suite in that configuration. Because simdutf needs C++17, the suite is
built at C++11 as well, so one job covers both the scalar fallback with the
macro defined and simdutf itself.
The dependency hangs off test_main, whose usage requirements every test target
inherits. The library target and the installed CMake package are deliberately
untouched: making nlohmann_json link simdutf would put a find_dependency() in
the exported package, which is a separate decision.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
is_contiguous_byte_container accepted any type with a data() returning a
pointer to a single-byte integral plus a size(). That is duck typing: the two
members say nothing about size() counting the units data() points at.
A type where it does not - fixed-size records, say - was routed to the
pointer-based adapter and parsed as [data(), data() + size()) bytes, silently
truncating input the iterator-based adapter had read in full:
struct record_buffer {
using value_type = std::array<char, 4>;
std::string bytes;
const char* data() const; // raw bytes
std::size_t size() const; // in records
const char* begin() const; const char* end() const;
};
json::parse(record_buffer{"[1,2,3,4,5]"}); // parse error at column 3
Requiring the container's own value_type to be that same element type ties the
two together. Every contiguous standard container satisfies it, so std::string,
std::vector<char>, std::array<char, N> and std::string_view keep the fast path;
anything else falls back to the iterator-based adapter, which is always correct.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The comment claimed the reported column is the one the offending token starts
at. It is the column reached after the token's last character - which is the
actual point of the unget() change: a number terminated by a newline now
reports what the same number terminated by a space always did.
Assert that equality directly, and add a multi-character token where the start
and end columns differ, so the invariant cannot be read off a single-character
example.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
clang-tidy reads "[\"\\ud834\"]" as a literal better written raw, and the
two literals written next to each other in "[\"a\x01""b\"]" as a missing
comma. The concatenation was there to stop the hex escape swallowing the
following character; build those documents from explicit bytes instead and
use raw strings elsewhere. The byte sequences are unchanged.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
- give the helper lambdas an explicit std::string return type and return
braced initializer lists (modernize-return-braced-init-list)
- replace the C-style array of test cases with a std::vector
(modernize-avoid-c-arrays)
- silence pro-type-member-init on the two brace-initialized aggregates;
default member initializers would stop them being aggregates in C++11
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Making supports_bulk_scan depend on iterator_is_contiguous meant the trait
is now instantiated for every adapter, not only when get_elements() is
called. On standard libraries with an incomplete <ranges> that is fatal:
libstdc++ 10 evaluates std::contiguous_iterator<std::counted_iterator<T*>>
by calling std::to_address, which needs an operator-> its counted_iterator
does not have, so satisfaction checking is a hard error rather than false.
Reported by clang 14 + libstdc++ 10.
JSON_HAS_RANGES already encodes exactly this ("libstdc++ < 11 has incomplete
C++20 ranges", #4440), so require it for the C++20 branch. Affected
toolchains fall back to the pointer-only test and the byte-at-a-time
scanner, which parses identically, just without the bulk fast paths.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Two problems in the tests added for the bulk scanners, both found by CI:
- the inner `const json j` in the counted-iterator diagnostics shadowed the
one declared at test-case scope, which -Wshadow rejects on GCC and clang
and C4456 rejects on MSVC under /WX; rename them
- the new parity checks parse deliberately invalid input, which calls
std::abort() rather than throwing when JSON_NOEXCEPTION is defined, so
they would have crashed the no-exception build; guard them the way the
other tests do
json::accept() does not abort, so the UTF-8 range assertions stay compiled
without exceptions and keep covering validate_one_utf8() there.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The pinned astyle expands a braced-init-list used as a range-for range onto
several lines; hoist the two offsets into a named vector instead, which reads
better and leaves nothing for astyle to reformat.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
These paths had no dedicated tests and rested on differential fuzzing only.
Add three sections, all comparing the contiguous scanner against the
byte-at-a-time one on the parsed value and on the exact error message:
- every string of length 1..3 over an alphabet of ordinary ASCII, both
specials, a control byte, escape characters, UTF-8 lead and continuation
bytes, and a byte that is never valid - each at offset 0 and offset 9, so
the bulk scanner sees them with and without a run behind them
- every kind of run-ending byte at each offset across two 8-byte SWAR words,
so multibyte sequences also straddle the word boundary
- the boundaries of every range validate_one_utf8() recognizes: shortest and
longest encodings, overlongs, both ends of the surrogate block, U+10FFFF
and just past it, and truncated sequences
Verified to fail if the bulk validator accepts surrogates, and if the SWAR
word test stops detecting control characters.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The JSON number grammar is encoded twice: as the scan_number() state machine
and as the contiguous fast path. The fast path declining on anything it does
not recognize keeps most divergence harmless, but if it ever accepted
something the state machine rejects the result would be a silent correctness
bug, and the existing test only pinned a hand-written list of numbers.
Enumerate every string of length 1..4 over "01.eE+-" (2800 tokens) and
require both paths to agree on the parsed value and on the exact error
message. Verified to fail if the fast path's grammar is perturbed.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
json::parse is declared warn_unused_result, and CHECK_THROWS_WITH_AS
evaluates its expression as a discarded statement, so the assertion broke
the -Werror builds (GCC -Werror=unused-result, MSVC C4834 under /WX).
Compare against the helper that already captures the message instead.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Sized sentinels newly reach the bulk string/number scanners and the
seek-based token reconstruction, so exercise both:
- diagnostics that quote the offending token, which are rebuilt from the
consumed input via copy_consumed_range()
- inputs whose count ends before the underlying buffer does, including a
closing quote that exists only behind the count, a cut inside an 8-byte
SWAR stride, and a cut inside a UTF-8 sequence
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
supports_bulk_scan required IteratorType and SentinelType to be the same
type, which excluded std::counted_iterator paired with std::default_sentinel_t
- the combination #5268 had already enabled for the memcpy fast path. Such
input fell back to the byte-at-a-time scanner even though it is contiguous
and its remaining length is computable in O(1).
Factor the "distance is computable in O(1)" test into sentinel_is_sized and
use it for iterator_is_contiguous, supports_seek, and supports_bulk_scan
alike, and share the std::ranges::distance/std::distance dispatch through a
remaining_count() helper.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The contiguous number fast path never reads the character that terminates
a number token, while scan_number() reads it and then ungets it. When that
character is a newline, get() has already cleared chars_read_current_line,
and unget() could only restore lines_read - leaving the column at 0. The
two paths therefore reported different columns for the same document:
json::parse("[01\n]") -> line 1, column 3
json::parse(stringstream) -> line 1, column 0
Remember the column the newline was read at so unget() can restore it.
Both paths now report the position the offending token actually starts at,
which also fixes the pre-existing column-0 artifact for streaming input.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The Clinger fast path is exact only for the "easy" subset (<=19 significant
digits, |exp10| <= 22); high-precision and scientific floats fall through to
strtod, where the failed Clinger attempt actually makes parsing a net loss.
std::from_chars implements the Eisel-Lemire algorithm in modern standard
libraries: locale-independent, correctly rounded, and fast over the whole
value range.
convert_number() now tries parse_float_from_chars() first (guarded by
__cpp_lib_to_chars, so C++11 and libc++-without-float-support keep the
Clinger + strtod path unchanged), then Clinger, then strtof. from_chars is
used only when it consumes the entire token; a partial parse means a non-'.'
locale decimal point, and an under-/overflow (result_out_of_range) also
declines - in both cases the existing strtod fallback supplies the exact
value and the well-defined +/-inf/0 the parser expects, side-stepping the
P4168 divergence between implementations. float and long double now get the
fast path too (Clinger was double-only).
Measured, C++17, g++ 13 -O3, json::parse/accept:
- canada-style floats: ~unchanged (Clinger already covered them)
- high-precision (17 digits): parse 2.1x, accept 2.5x
- scientific (17 digits + exp): parse 3.6x, accept 4.1x
Verified: C++11 (Clinger/strtod) and C++17 (from_chars) parse every value -
including subnormals, boundary values, and 1e9999/1e-9999 over-/underflow -
to bit-identical results; 2M number-fuzz clean; conversions/deserialization/
locale/number-fast-path suites pass in both C++11 and C++17; clang-tidy
clean; warning-clean on g++ and clang in C++11/17/20.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
scan_number() reads a number one character at a time through the input
adapter (get()) and appends each byte to token_buffer (add()) before
converting. For contiguous input, the per-character get()/add() overhead
dominates: it is roughly two thirds of the time spent on number-heavy
parsing, far more than the value conversion itself.
Add scan_number_bulk_contiguous(), which parses the whole number token
straight from the input buffer: it validates and classifies the extent
with the same grammar as scan_number()'s state machine, materializes
token_buffer in one copy (substituting the locale decimal point exactly as
scan_number() does), advances the adapter, and reuses the shared
convert_number() tail. On anything it does not recognize as a well-formed
number it makes no state change and returns token_type::uninitialized, so
the caller falls back to scan_number(), which then produces the exact
diagnostic. Errors and their positions are therefore unchanged.
The conversion tail is factored out of scan_number() into convert_number()
so both scanners share it; the fast path is selected by tag dispatch on the
existing bulk_scan capability, so streaming/wide/user adapters are
unaffected.
Measured on pointer input, g++ 13 -O3:
- integers: parse +65%, accept +98%
- floats: parse +39%, accept +70%
Verified: 2,000,000 randomized number documents (including overflow-range
integers, long digit strings and %.17g doubles) parse identically via the
contiguous path and the streaming byte path, matching value, type and
round-trip text; the locale suite and existing parser/lexer/conversions/
deserialization tests pass; a new "lexer number fast path" test checks
contiguous-vs-streaming parity, token classification, and that malformed
numbers are rejected identically. Pure C++11, no intrinsics.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
When merge_objects is true, recurse only if the existing value is an
object. Otherwise overwrite, matching the documented "all other values
are overwritten as usual" behavior.
Fixes#5402
Signed-off-by: elix3r <157088510+22elix3r@users.noreply.github.com>
* Throw other_error.502 when UBJSON use_type is set without use_size
Fixes#5321
Signed-off-by: Krishnanand G <118352827+Krishnanand-G@users.noreply.github.com>
* Scope UBJSON use_type check to container branches and expand tests
Signed-off-by: Krishnanand G <118352827+Krishnanand-G@users.noreply.github.com>
* Re-amalgamate single_include/json.hpp
The previous commit updated the split headers but the amalgamated
file didn't go back through astyle before I committed it, so CI's
amalgamation check caught formatting drift in json_fwd.hpp and a
few noexcept clauses in basic_json, plus one doc example. None of
it touches the UBJSON logic. Applied the patch CI generated to
bring single_include back in sync.
Signed-off-by: Krishnanand G <118352827+Krishnanand-G@users.noreply.github.com>
---------
Signed-off-by: Krishnanand G <118352827+Krishnanand-G@users.noreply.github.com>
* Fix CBOR tag handlers not recognizing tags 0-5 and 21-23
The tagged-item switch in binary_reader::parse_cbor_internal() only handled
head bytes 0xC6-0xD4 and 0xD8-0xDB. Bytes 0xC0-0xC5 (tags 0-5: date/time,
epoch, bignum, decimal, bigfloat) and 0xD5-0xD7 (tags 21-23: base64url,
base64, base16 conversion hints) fell through to the default case and were
reported as invalid bytes, even under cbor_tag_handler_t::ignore and ::store,
despite being valid CBOR major-type-6 tags per RFC 8949.
Add the missing case labels so the full 0xC0-0xDB range is handled
uniformly. Extend the "Tagged values" test in unit-cbor.cpp to cover
0xC0-0xD7, and update the CBOR docs to state the corrected tag range.
Fixes#5315
Signed-off-by: sahilkamate03 <45514385+sahilkamate03@users.noreply.github.com>
* Fix stale CBOR tag docs and add store-mode binary-payload test
The "Incomplete mapping" warning still listed tags 0-5 (date/time,
bignum, decimal fraction, bigfloat) and 21-23 (expected conversions)
as unsupported, even though they now parse correctly under
cbor_tag_handler_t::ignore/store, same as 0xC6..0xD4/0xD8..0xDB.
Remove those five bullets and cross-reference the "Tagged items"
warning below, matching the equivalent docs fix landed independently
in PR #5367.
Also add a cbor_tag_handler_t::store test that wraps a binary
payload (not just a string) for every byte in 0xC0..0xD7, confirming
these tags are unwrapped the same way as 0xC6..0xD4 rather than
mistaken for the 0xD8..0xDB binary-subtype marker syntax, per review
feedback on #5331.
Signed-off-by: sahilkamate03 <45514385+sahilkamate03@users.noreply.github.com>
---------
Signed-off-by: sahilkamate03 <45514385+sahilkamate03@users.noreply.github.com>
The diagnostic position of a string value was derived by subtracting the
parsed value's length from the end position. Escape sequences make the
source token longer than the value it parses to, so the reported start
position landed inside the string, one byte off per escape sequence:
input: {"a":"\n\n\n\n\n\n"}
start_pos() == 11, so the reported range covered n\n\n\n"
instead of the documented "\n\n\n\n\n\n"
This contradicts the documented behavior of start_pos(), which is the
position of the opening quote, and it also corrupted the "(bytes N-M)"
part of JSON_DIAGNOSTICS exception messages. Strings with multi-byte
UTF-8 but no escapes were unaffected, which is why this went unnoticed.
Record the offset of the token in the lexer when it starts scanning and
use that, instead of reconstructing it from the parsed value. Booleans,
null and numbers already reported correct positions and are unchanged.
The new lexer member and accessor are compiled only when
JSON_DIAGNOSTIC_POSITIONS is enabled, which is already part of the ABI
tag, so the default build is unaffected.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The comment asked whether the no-op marker 'N' may be ignored when a
string is read. It may not: at that point the next byte must be a string
length type specification, and 'N' is not one. No-ops at positions where
a value may start are already consumed by the callers through
get_ignore_noop(), so nothing is lost by not skipping them here.
Replace the TODO with a comment stating that, and add regression tests
pinning both directions: a no-op is accepted at top level (also
repeated), before and after an array element, and before an object key,
between key and value, and before the closing brace of an object of
unknown size; it is rejected where a length type specification is
expected, i.e. after the 'S' marker of a string value and as the key
length of an object of known size.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Do not write BJData ndarrays whose size overflows std::size_t
write_bjdata_ndarray() multiplied the _ArraySize_ dimensions into a
std::size_t without checking for overflow. A product that wraps around
to a value that happens to match the size of _ArrayData_ passed the
length check, and the writer emitted an ndarray header announcing an
element count that cannot be represented:
{"_ArrayType_":"uint8","_ArraySize_":[9223372036854775808,2],"_ArrayData_":[]}
was encoded as 5b 24 55 23 5b 4d 00 00 00 00 00 00 00 80 69 02 5d, an
ndarray of 2^64 elements followed by no data. Reading that back throws
out_of_range.408 ("excessive ndarray size caused overflow"), so to_bjdata
produced output that from_bjdata rejects. This is reachable by parsing
untrusted JSON and re-encoding it as BJData.
Mirror the overflow check the binary reader already performs, and also
reject a single dimension that does not fit into std::size_t, which the
previous cast silently truncated where std::size_t is narrower than 64
bits. Such objects now fall back to a plain object encoding, which is
what the surrounding type and length validation already does for
annotations it cannot represent, and they round-trip unchanged.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Document when to_bjdata converts a JData annotation to an ND-array
The BJData page described the 1-D vector case as the only situation in
which an object carrying _ArrayType_/_ArraySize_/_ArrayData_ is not
written as a compact ND-array. The writer has always had several other
fallbacks -- an unknown _ArrayType_, a dimension that is not a
non-negative integer, an _ArrayData_ whose length does not match the
product of the dimensions, and elements that are not numbers of the
annotated kind -- all of which cause the value to be serialized as a
regular JSON object instead.
Spell out the conditions, including the size-overflow check added in the
preceding commit, so the documented behavior matches the implementation.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* reject CBOR array/map length equal to the indefinite-length marker
Signed-off-by: Angadi Yashaswini <angadi@digiscrypt.com>
* reject CBOR lengths that do not fit in std::size_t via value_in_range_of
Signed-off-by: Angadi Yashaswini <angadi@digiscrypt.com>
---------
Signed-off-by: Angadi Yashaswini <angadi@digiscrypt.com>
This adds a std::isfinite check to the UBJSON floating-point parsing path, throwing out_of_range.406 on overflow. This makes the UBJSON parser's behavior consistent with the normal JSON parser. Fixes#5322.
Signed-off-by: AJ369ninja <abhishek.j@iitg.ac.in>
Co-authored-by: AJ369ninja <abhishek.j@iitg.ac.in>
* validate ndarray element types in write_bjdata_ndarray
Signed-off-by: Angadi Yashaswini <angadi@digiscrypt.com>
* read ndarray elements through get<> instead of a fixed union member
_ArrayType_ names the wire type, not how the value is stored: parsing
keeps a non-negative integer as number_unsigned while the C++ API keeps
an int literal as number_integer. Selecting the union member from the
type marker therefore reads the inactive alternative for one of the two,
so read through get<> instead, which dispatches on the active member.
Also reject a negative _ArraySize_ entry, which is not a usable
dimension, and cover the parse-built path in the tests.
Signed-off-by: Angadi Yashaswini <angadi@digiscrypt.com>
---------
Signed-off-by: Angadi Yashaswini <angadi@digiscrypt.com>
- warn about BSON marker 0x11 interoperability in both directions
- explain subtype-less binary normalization to subtype 0x00
- add a round-trip test for binary values without a subtype
Signed-off-by: YingqiDuan <141370165+YingqiDuan@users.noreply.github.com>
* Add iterator+sentinel tests and docs for binary deserializers
This commit extends the C++20 ranges support (iterator+sentinel pairs) to the
binary format deserializers from_cbor, from_msgpack, from_ubjson, from_bjdata,
and from_bson, matching what was already done for parse(), accept(), and
sax_parse().
Changes:
- Add istreambuf_sentinel helper to test_utils.hpp for EOF detection in tests
- Add 5 new test cases that read binary files directly via
std::istreambuf_iterator<char> + sentinel, without pre-buffering
- Update documentation for all 5 from_* functions to document overload (3)
with SentinelType parameter
- All tests pass; verified against existing test suite data
- Fix potential buffer over-read warning in heterogeneous iterator test
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Merge iterator+sentinel overloads and fix ambiguity/CI issues
Address PR review feedback and CI failures:
- Merge the separate same-type and sentinel-type iterator overloads of
parse(), accept(), sax_parse(), and the five from_* binary deserializers
into a single overload with SentinelType defaulted to IteratorType,
as suggested in review. Applied the same simplification to the
detail::input_adapter() free functions.
- Fix a latent ambiguity: some compilers (e.g. GCC 4.8) unreliably SFINAE
the operator!= detection for std::nullptr_t against container/string
types, making calls like parse(s, nullptr, ...) ambiguous with the
compatible-input overload. can_compare_ne now explicitly excludes
std::nullptr_t as a SentinelType.
- Use a named enable_if_t template parameter instead of an unnamed
function parameter for the SFINAE guard, fixing a clang-tidy
hicpp-named-parameter/readability-named-parameter failure.
- Update parse.md, accept.md, sax_parse.md, and the five from_*.md pages
to document the merged overload instead of separate (2)/(3) overloads,
also fixing an over-160-char line that broke the documentation
style_check CI job.
- Rework the BSON iterator+sentinel test to parse a BSON file already
present in the test suite instead of writing/deleting a temp file.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix -Wunneeded-internal-declaration for CustomSentinel in test
CustomSentinel lives in an anonymous namespace (internal linkage), and
the library's parse loop only ever evaluates the iterator-first
direction (it != last), so the reversed-order friend operator!= was
never referenced. Clang's -Weverything flags such unused internal
declarations as an error. Drop the unused overload; the used direction
is enough to satisfy can_compare_ne's either-order detection.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix clang-tidy hicpp-named-parameter and misc-const-correctness
- Drop the unused reversed-order operator!= overload from
utils::istreambuf_sentinel (only iterator != sentinel is ever
evaluated) and name the remaining friend's sentinel parameter, fixing
hicpp-named-parameter/readability-named-parameter.
- Mark the istreambuf_iterator first/last helper variable const in the
five binary-format sentinel tests, fixing misc-const-correctness.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix clang-tidy misc-const-correctness in heterogeneous sentinel test
json_str is only read via .data()/.size() and never reassigned, so
clang-tidy correctly flags it as const-able. Verified against the exact
CI job (silkeh/clang:dev, ci_clang_tidy target) by running clang-tidy
directly on this file plus the five binary-format sentinel tests
touched by prior commits; all are now clean.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix discussion #4209: custom BinaryType direct assignment and extraction
When a custom BinaryType is configured (other than the default std::vector<uint8_t>),
users can now:
1. Assign values of that type directly to create binary values (not arrays)
2. Extract binary values back to that type with get<>()
3. Extract arrays to that type (for backward compatibility)
Implementation:
- Add is_compatible_binary_type trait to centralize SFINAE condition
- Update to_json to accept custom BinaryType values directly
- Update from_json to handle both binary and array inputs for custom BinaryType
- Add #include <vector> with IWYU comment to from_json.hpp
- Add comprehensive tests for assignment and array extraction
- Update binary_t documentation with example
This is purely additive and invisible to the default nlohmann::json alias, which
continues to treat std::vector<uint8_t> as arrays.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix CI: missing include and const-correctness
- Add #include <vector> to type_traits.hpp for the new
is_compatible_binary_type trait's std::vector<std::uint8_t> reference
(caught by cpplint's include-what-you-use check)
- Mark test-local json variables const where never reassigned
(caught by clang-tidy's misc-const-correctness check)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Niels Lohmann
elix3r
Krishnanand G
Sahil_Kamate
ljcjclljc
Angadi56
tomatotomata
hum4nBeingandAJ369ninja
YingqiDuan
Yash Bavadiya
Patrick10199
Luke Banicevic
Federico Sfriso