amorfo77/json
1
0 Fork 0
mirror of https://github.com/nlohmann/json.git synced 2026-10-23 14:28:54 +00:00
Code Issues Packages Projects Releases Wiki Activity

1.5: keep get_bon8_string's own UTF-8 ranges (#5891)

Browse Source 
Taking the second-byte range from utf8_shape() made BON8 decoding 2-3% slower
(FromBinaryShape/scalars, FromBinaryBuffer, alignment-pinned builds), so the byte
path keeps its hand-written ranges. The exhaustive validator test still checks
that it accepts the same sequences as the other validators.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
This commit is contained in:
Niels Lohmann committed 2026-10-23 12:18:56 +02:00
1 parent fce5a54722
commit fc347b5779
2 files changed
+19 -7
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

include/nlohmann/detail/input/binary_reader.hpp
+17 -5
View File
@@ -3810,11 +3810,23 @@ class binary_reader
// the valid range of the second byte excludes overlong forms,
// surrogates, and code points above U+10FFFF
// (RFC 3629, section 4); length 0 flags the lead bytes
// 0xF5..0xF7, which BON8 can read but UTF-8 cannot encode
const auto shape = utf8_shape(static_cast<std::uint8_t>(byte));
const int continuation_bytes = (byte <= 0xDF) ? 1 : ((byte <= 0xEF) ? 2 : 3);
const bool valid_second = shape.length != 0 && second >= shape.second_lo && second <= shape.second_hi;
// (RFC 3629, section 4)
int continuation_bytes = 0;
bool valid_second = true;
if (byte <= 0xDF)
{
continuation_bytes = 1;
}
else if (byte <= 0xEF)
{
continuation_bytes = 2;
valid_second = (byte != 0xE0 || second >= 0xA0) && (byte != 0xED || second <= 0x9F);
}
else
{
continuation_bytes = 3;
valid_second = byte <= 0xF4 && (byte != 0xF0 || second >= 0x90) && (byte != 0xF4 || second <= 0x8F);
}
if (JSON_HEDLEY_UNLIKELY(!valid_second))
{
include/nlohmann/detail/string_utils.hpp
+2 -2
View File
@@ -210,8 +210,8 @@ places, which differ in speed, diagnostics, and how they read the input:
bulk string scan, the bulk path of the BON8 reader, and the BON8 writer.
They must accept exactly what the lexer's switch accepts.
- the byte path of binary_reader::get_bon8_string(): BON8 input without bulk
access, and the bytes the bulk path leaves to it. It takes the valid range
of the second byte from utf8_shape().
access, and the bytes the bulk path leaves to it. It keeps its own ranges:
taking them from utf8_shape() made BON8 decoding 2-3% slower (#5891).
- check_utf8() in detail/view/edit.hpp: strings stored in an editable
json_document, with the index of the offending byte. It also uses
utf8_shape().