get_cbor_array() and get_cbor_object() read their elements by calling back
into the value reader, which called them again for a nested container, and a
tag was handled by reading the tagged value the same way. All three cost
native stack, and all three cost a single byte to encode: 0x9F opens an
indefinite-length array, 0x81 a one-element array, and 0xC2 is a tag. Half a
million of any of them crashes the process before the input runs out (#5104).
Apply the shape the MessagePack reader already uses: the open containers live
on the heap stack, parse_cbor_value() reads a single value and only opens a
container rather than reading it to its end, and parse_cbor_internal() loops,
resuming the innermost container after each element.
Two things are specific to CBOR. An indefinite-length container ends at a
break marker rather than at a count, and testing for that marker consumes a
byte which is the first byte of the next element when it is not one; the
frame's count is npos for those, and the driver tracks whether the next value
starts at a fresh byte. And a tag is not a value of its own: instead of
reading the tagged value by recursing, the value reader reports that a tag was
read and the driver reads on, so a chain of tags costs no stack at all.
The switch that decodes a value is unchanged apart from the twelve container
cases and the two tag sites. Verified against the previous commit over
definite and indefinite arrays and maps, all four counted forms, empty
containers, nesting of the forms inside each other, truncated inputs, and all
three tag handlers: identical values, error codes, messages and byte offsets.
500,000 levels of each of the three vectors now report parse_error.110 instead
of crashing, and a well-formed 200,000-level value is read to completion.
On performance: the driver does per element what a counted loop used to do
per container, and CBOR pays for it more than MessagePack because the value
reader also has to be told whether to fetch a byte. Parsing 60,000 small
objects and one array of a million integers is 3 to 4 % slower than the
recursive reader, measured over five alternating runs. Against develop the
same two inputs are about 44 % faster, because the entry point no longer
copies the value it parsed; the earlier commit in this series is what pays
for that.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
get_cbor_string() and get_cbor_binary() handled the indefinite-length forms
(0x7F and 0x5F) by calling themselves once per chunk. Each chunk therefore
cost a native stack frame, and since a chunk may itself be an indefinite-
length string, an input of repeated 0x7F bytes reached one frame per input
byte: 200,000 of them crash the process with SIGSEGV before a single byte is
rejected. This is the same defect as #5104, in a path the container-level
work does not touch.
Count the open levels instead of recursing through them. That is enough here
because every chunk is appended to the same result -- get_bytes() writes at
result.size() -- so there is no per-level state to keep. The temporary chunk
string and its copy into the result go away with the recursion.
The definite-length cases move to get_cbor_string_chunk() and
get_cbor_binary_chunk() unchanged, including their error messages, which
still name 0x7F and 0x5F because those are handled one level up.
Behaviour is unchanged. Comparing against develop over the interesting byte
sequences -- empty, single-chunk, nested, over-closed and truncated forms,
both strings and byte arrays, and an indefinite-length map key -- produces
identical values, error codes, messages and byte offsets. The 200,000-level
input now reports parse_error.110 at byte 200001 instead of crashing.
Note that nesting these is not valid CBOR: RFC 8949, Section 3.2.3 forbids
it. This does not change that either way -- it has always been accepted, and
rejecting it is a separate decision (#5317, #5325). Should it be rejected
later, that is now one condition on the level counter rather than a change to
the control flow.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix CBOR tag handlers not recognizing tags 0-5 and 21-23
The tagged-item switch in binary_reader::parse_cbor_internal() only handled
head bytes 0xC6-0xD4 and 0xD8-0xDB. Bytes 0xC0-0xC5 (tags 0-5: date/time,
epoch, bignum, decimal, bigfloat) and 0xD5-0xD7 (tags 21-23: base64url,
base64, base16 conversion hints) fell through to the default case and were
reported as invalid bytes, even under cbor_tag_handler_t::ignore and ::store,
despite being valid CBOR major-type-6 tags per RFC 8949.
Add the missing case labels so the full 0xC0-0xDB range is handled
uniformly. Extend the "Tagged values" test in unit-cbor.cpp to cover
0xC0-0xD7, and update the CBOR docs to state the corrected tag range.
Fixes#5315
Signed-off-by: sahilkamate03 <45514385+sahilkamate03@users.noreply.github.com>
* Fix stale CBOR tag docs and add store-mode binary-payload test
The "Incomplete mapping" warning still listed tags 0-5 (date/time,
bignum, decimal fraction, bigfloat) and 21-23 (expected conversions)
as unsupported, even though they now parse correctly under
cbor_tag_handler_t::ignore/store, same as 0xC6..0xD4/0xD8..0xDB.
Remove those five bullets and cross-reference the "Tagged items"
warning below, matching the equivalent docs fix landed independently
in PR #5367.
Also add a cbor_tag_handler_t::store test that wraps a binary
payload (not just a string) for every byte in 0xC0..0xD7, confirming
these tags are unwrapped the same way as 0xC6..0xD4 rather than
mistaken for the 0xD8..0xDB binary-subtype marker syntax, per review
feedback on #5331.
Signed-off-by: sahilkamate03 <45514385+sahilkamate03@users.noreply.github.com>
---------
Signed-off-by: sahilkamate03 <45514385+sahilkamate03@users.noreply.github.com>
* reject CBOR array/map length equal to the indefinite-length marker
Signed-off-by: Angadi Yashaswini <angadi@digiscrypt.com>
* reject CBOR lengths that do not fit in std::size_t via value_in_range_of
Signed-off-by: Angadi Yashaswini <angadi@digiscrypt.com>
---------
Signed-off-by: Angadi Yashaswini <angadi@digiscrypt.com>
* Add iterator+sentinel tests and docs for binary deserializers
This commit extends the C++20 ranges support (iterator+sentinel pairs) to the
binary format deserializers from_cbor, from_msgpack, from_ubjson, from_bjdata,
and from_bson, matching what was already done for parse(), accept(), and
sax_parse().
Changes:
- Add istreambuf_sentinel helper to test_utils.hpp for EOF detection in tests
- Add 5 new test cases that read binary files directly via
std::istreambuf_iterator<char> + sentinel, without pre-buffering
- Update documentation for all 5 from_* functions to document overload (3)
with SentinelType parameter
- All tests pass; verified against existing test suite data
- Fix potential buffer over-read warning in heterogeneous iterator test
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Merge iterator+sentinel overloads and fix ambiguity/CI issues
Address PR review feedback and CI failures:
- Merge the separate same-type and sentinel-type iterator overloads of
parse(), accept(), sax_parse(), and the five from_* binary deserializers
into a single overload with SentinelType defaulted to IteratorType,
as suggested in review. Applied the same simplification to the
detail::input_adapter() free functions.
- Fix a latent ambiguity: some compilers (e.g. GCC 4.8) unreliably SFINAE
the operator!= detection for std::nullptr_t against container/string
types, making calls like parse(s, nullptr, ...) ambiguous with the
compatible-input overload. can_compare_ne now explicitly excludes
std::nullptr_t as a SentinelType.
- Use a named enable_if_t template parameter instead of an unnamed
function parameter for the SFINAE guard, fixing a clang-tidy
hicpp-named-parameter/readability-named-parameter failure.
- Update parse.md, accept.md, sax_parse.md, and the five from_*.md pages
to document the merged overload instead of separate (2)/(3) overloads,
also fixing an over-160-char line that broke the documentation
style_check CI job.
- Rework the BSON iterator+sentinel test to parse a BSON file already
present in the test suite instead of writing/deleting a temp file.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix -Wunneeded-internal-declaration for CustomSentinel in test
CustomSentinel lives in an anonymous namespace (internal linkage), and
the library's parse loop only ever evaluates the iterator-first
direction (it != last), so the reversed-order friend operator!= was
never referenced. Clang's -Weverything flags such unused internal
declarations as an error. Drop the unused overload; the used direction
is enough to satisfy can_compare_ne's either-order detection.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix clang-tidy hicpp-named-parameter and misc-const-correctness
- Drop the unused reversed-order operator!= overload from
utils::istreambuf_sentinel (only iterator != sentinel is ever
evaluated) and name the remaining friend's sentinel parameter, fixing
hicpp-named-parameter/readability-named-parameter.
- Mark the istreambuf_iterator first/last helper variable const in the
five binary-format sentinel tests, fixing misc-const-correctness.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix clang-tidy misc-const-correctness in heterogeneous sentinel test
json_str is only read via .data()/.size() and never reassigned, so
clang-tidy correctly flags it as const-able. Verified against the exact
CI job (silkeh/clang:dev, ci_clang_tidy target) by running clang-tidy
directly on this file plus the five binary-format sentinel tests
touched by prior commits; all are now clean.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* fix(cbor): reject negative ints overflowing int64
CBOR encodes negative integers as "-1 - n" where n is uint64_t. When
n > INT64_MAX, casting to int64_t caused undefined behavior and silent
data corruption. Large negative values were incorrectly parsed as
positive integers (e.g., -9223372036854775809 became 9223372036854775807).
Add bounds check for to reject values that exceed int64_t
representable range, returning parse_error instead of silently
corrupting data.
Added regression test cases to verify.
Signed-off-by: Ville Vesilehto <ville@vesilehto.fi>
* chore: clarify tests
Add test for "n=0" case (result=-1) to cover the smallest magnitude
boundary. Update comments to explain CBOR 0x3B encoding and why
"result=0" is not possible. Clarify that n is an unsigned integer
in the formula "result = -1 - n" to help understanding the tests.
Signed-off-by: Ville Vesilehto <ville@vesilehto.fi>
* fix(cbor): extend overflow checks for other types
Extend negative integer overflow detection to all CBOR negative
integer cases (0x38, 0x39, 0x3A) for consistency with the existing
0x3B check.
Signed-off-by: Ville Vesilehto <ville@vesilehto.fi>
---------
Signed-off-by: Ville Vesilehto <ville@vesilehto.fi>
* Add implementation to retrieve start and end positions of json during parse
* Add more unit tests and add start/stop parsing for arrays
* Add raw value for all types
* Add more tests and fix compiler warning
* Amalgamate
* Fix CLang GCC warnings
* Fix error in build
* Style using astyle 3.1
* Fix whitespace changes
* revert
* more whitespace reverts
* Address PR comments
* Fix failing issues
* More whitespace reverts
* Address remaining PR comments
* Address comments
* Switch to using custom base class instead of default basic_json
* Adding a basic using for a json using the new base class. Also address PR comments and fix CI failures
* Address decltype comments
* Diagnostic positions macro (#4)
Co-authored-by: Sush Shringarputale <sushring@linux.microsoft.com>
* Fix missed include deletion
* Add docs and address other PR comments (#5)
* Add docs and address other PR comments
---------
Co-authored-by: Sush Shringarputale <sushring@linux.microsoft.com>
* Address new PR comments and fix CI tests for documentation
* Update documentation based on feedback (#6)
---------
Co-authored-by: Sush Shringarputale <sushring@linux.microsoft.com>
* Address std::size_t and other comments
* Fix new CI issues
* Fix lcov
* Improve lcov case with update to handle_diagnostic_positions call for discarded values
* Fix indentation of LCOV_EXCL_STOP comments
* fix amalgamation astyle issue
---------
Co-authored-by: Sush Shringarputale <sushring@linux.microsoft.com>
Niels Lohmann
Sahil_Kamate
Angadi56
Patrick10199
Ville Vesilehto
Sushrut ShringarputaleandSush Shringarputale
Florian Albrechtskirchinger