mirror of
https://github.com/nlohmann/json.git
synced 2026-10-31 09:55:42 +00:00
json_view: decode chains of unicode escapes in a loop
After the first \uXXXX escape of a string, slow_string() now decodes the escapes that follow it directly, while the next six bytes are \u and four hex digits that are not a surrogate, instead of going round its main loop for each of them (that is how Python's json.dumps writes non-ASCII text by default). A surrogate, a bad digit, or fewer than six bytes leave the position at the backslash: the main loop handles them as before and reports the same error at the same position. twitterescaped.json (simdjson-data, 549 KB, 33,036 escapes), json_document::parse(), A/B in one binary, 6 code layouts, time after / before: - Apple M1 Max, Apple clang 21: 0.897 - x86-64 Haswell, GCC 16.2: 0.907 - x86-64 Haswell, Clang 23: 0.910 The other files of the benchmark set (twitter, citm_catalog, canada, gsoc-2018, apache_builds, instruments, tree-pretty, github_events, random, update-center, semanticscholar-corpus, twitter_timeline) stay within the noise of each machine. New test: chains of every length from 1 to 40 of escapes that give one, two, and three bytes of UTF-8, followed by the end of the string, text, escapes, surrogates, bad digits, and cut in every place, in keys and in values, against json::parse(). Signed-off-by: Niels Lohmann <mail@nlohmann.me>
This commit is contained in:
3 files changed
+83
No files matched your search
Generated
+15
@@ -1585,6 +1585,21 @@ class builder
|
||||
}
|
||||
arena_ensure(w, w_end, 4);
|
||||
w = put_utf8(w, cp);
|
||||
// A chain of unicode escapes ("\u3042\u3044\u3046", as Python's json.dumps writes non-ASCII text) is
|
||||
// decoded here, without going round the loop above for each of them. Anything else (a surrogate, a
|
||||
// bad digit, fewer than six bytes) leaves p at the backslash: the loop above handles it, and reports
|
||||
// the same error at the same position as it would without this loop.
|
||||
while (end - p >= 6 && p[0] == '\\' && p[1] == 'u')
|
||||
{
|
||||
const int next = hex_codepoint(p + 2);
|
||||
if (next < 0 || (next & 0xF800) == 0xD800)
|
||||
{
|
||||
break;
|
||||
}
|
||||
p += 6;
|
||||
arena_ensure(w, w_end, 4);
|
||||
w = put_utf8(w, next);
|
||||
}
|
||||
}
|
||||
else if (NLOHMANN_VIEW_LIKELY(d < 128 && simple_escape[d] != 0))
|
||||
{
|
||||
|
||||
Reference in new issue
Block a user