With error_handler_t::strict (and replace/ignore), every string a binary
reader reads is checked for valid UTF-8, and every string a binary writer
writes. The readers used the byte-at-a-time decoder; the writers skipped
ASCII eight bytes at a time.
- New valid_utf8_string() decides strings of up to 16 ASCII bytes, most
keys and many values, with one or two word tests, and checks longer
strings 16 bytes at a time with the NEON/SSSE3 UTF-8 check the lexer
uses (vector_utf8_valid_blocks: the same lookup, but quotes and control
characters are ordinary characters). Other targets and JSON_NO_SIMD use
valid_utf8_prefix() as before.
- The readers check strings with it (for one-byte character types) in an
out-of-line function, so that reading without a check does not pay for
its code.
- The writers' check_utf8() uses it, and the CBOR and MessagePack writers
write a checked string of up to max_text_in_reservation characters in
one reservation, as they write unchecked ones.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Test that the contiguous fast paths never read outside the input
The bulk string scanner, the SWAR searches, the UTF-8 prefix validator,
the 8-digit number loop, the 8-byte indentation skip, and the BSON/BON8
bulk string readers load several bytes at a time from contiguous input.
ASan misses over-reads into the slack of the same heap allocation, and
fuzzers rarely end an input exactly at a page boundary.
The new test copies each input directly in front of, and directly
behind, an inaccessible page (mmap/mprotect on POSIX,
VirtualAlloc/VirtualProtect on Windows), so reading even one byte
outside the input crashes. It covers the string-scan functions directly
and json::parse, accept, sax_parse (with and without comments) and all
binary readers through the public API, with inputs ending at every byte
of strings, escapes, UTF-8 sequences, numbers, literals, indentation
runs, and binary documents. Each result must match parsing the same
input from a std::string.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Rename the reader parameter to avoid a flawfinder false positive
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Make the guard page test work without exceptions and under UBSan
Under JSON_NOEXCEPTION, the error paths abort instead of throwing, so the
outcomes are now computed with allow_exceptions=false there. Copying the
input with std::copy from char to unsigned char tripped Clang's implicit
conversion sanitizer; std::memcpy avoids the conversion.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Test parse_float_native() at the guard page
parse_float_eisel_lemire() was removed by #5738, so the test no longer
compiled. The lexer converts floats from its token buffer, so the JSON
section never places a float token at the page boundary; the section now
calls parse_float_native() for float and double on every complete number
token directly and compares the result bit for bit with converting the
same token from a std::string.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix clang-tidy findings in the guard-page tests
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Convert explicitly between bytes and characters in the guard-page tests
The clang sanitizer job (-fsanitize=integer, x86-64 where char is signed)
reported the implicit char -> unsigned char conversions of std::copy() into
the guarded buffer and of the std::string built from the binary inputs.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Niels Lohmann