mirror of
https://github.com/nlohmann/json.git
synced 2026-10-22 14:08:13 +00:00
Niels Lohmann
272d07b48b
With ensure_ascii, a string of the source that has a byte >= 0x7F was written by a generic path: put() per piece and a validate-and-decode step for every character beyond ASCII. It now goes through a writer with the cursor in a local variable. The part of the string that has nothing to escape is found 16 bytes at a time (SSE2 or NEON; the existing 8-byte scan without vector support) and copied; strings that are ASCII throughout are still decided by the 8-byte check of dump() with ensure_ascii (a 16-byte scan for them was 2-5 % slower for short strings). The two-byte sequences (lead C2..DF) and the three-byte sequences whose continuation bytes may have any value (lead E1..EF except ED) are decoded with a few compares; E0, ED, four-byte sequences, and damaged input go through validate_one_utf8() as before, so damaged images still throw type_error.316. dump(-1, ' ', true) instructions per call, macOS arm64 (develop -> now); the compact dump without ensure_ascii and dumps of ASCII text are unchanged: twitter 5.42M -> 3.57M citm 3.28M -> 3.24M jeopardy(4k) 5.78M -> 5.65M CJK text, raw 66.5M -> 35.3M escaped \u 32.7M -> 17.0M Latin text with accents 11.5M -> 9.4M ASCII text 4.40M -> 4.40M Linux instructions (GCC 13 / Clang 18), dump(-1, ' ', true): twitter -9 % / -33 % CJK text -14 % / -47 % Latin text -8 % / -18 % citm -0.2 % / -0.4 % ASCII text -0.1 % / -0.3 % The compact dump without ensure_ascii is unchanged (canada 0.0 %). The unit test covers every character of the Basic Multilingual Plane, and a character that needs an escape at each position of strings of 0..40 bytes. Signed-off-by: Niels Lohmann <mail@nlohmann.me>