Two further serialization speedups on top of the ensure_ascii=false bulk
copy, both reusing the SWAR primitives in detail/input/string_scan.hpp.
1. Internal write buffer (devirtualization). Every structural character
('{', '"', ',', ...) previously went straight to the output adapter
through a virtual call. Route all writes through put_char/put_chars
into a 1 KiB buffer that flushes in bulk; the public dump() flushes
once the top-level value is done (the recursive worker is split out as
dump_internal). Runs larger than the buffer are written straight
through, so large payloads are not copied twice. This is the dominant
cost for object/array-heavy values.
2. ensure_ascii fast path. dump_escaped previously ran the UTF-8 DFA over
every byte when escaping non-ASCII. Add find_ascii_copyable_run() (a
SWAR scan stopping at '"', '\\', < 0x20, 0x7F, and >= 0x80) so runs of
printable ASCII are bulk-copied, with the byte path handling each
escape/non-ASCII byte exactly as before.
Behavior is unchanged: dump output is byte-for-byte identical to the
previous implementation across ~20k randomized byte strings plus curated
edge cases (all escapes, control chars, 0x7F, valid multibyte,
surrogates, overlong, truncated), for object/array/pretty output, both
ensure_ascii settings, and all three error handlers, in C++11/17/20 at
-O2/-O3. New unit tests cover the buffer flush boundaries, the escape and
0x7F handling, multibyte under both settings, and invalid-UTF-8 handling.
Throughput (g++ -O3, vs the ensure_ascii=false-only baseline):
long ASCII, ensure_ascii=0 4.2x
long ASCII, ensure_ascii=1 4.1x
twitter-like objects 2.7x
dense CJK 1.8x
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
simdutf.h rejects anything below C++17 with an #error, so defining
JSON_USE_SIMDUTF in a C++11 or C++14 translation unit did not fail with a
message about simdutf being unavailable - it failed to compile at all, taking
the library's C++11 support with it. Nothing caught this because no build ever
compiled that path.
Gate the include and both uses on JSON_HAS_CPP_17, the same way number_parse.hpp
gates std::from_chars. Below C++17 the macro now has no effect and the scalar
validator runs; it accepts and rejects exactly the same input, so the macro is
safe to set project-wide even when some translation units use an older standard.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
string_bulk_run() had the same `return scalar_string_bulk_run(...)` in both
arms of the `#if`. The simdutf arm already falls through when validation
fails, so a single return after the `#endif` says the same thing.
The JSON_USE_SIMDUTF example showed `#include <simdutf.h>`, which
string_scan.hpp already does under the same guard; users only have to put the
header on the include path and link the library, not include it themselves.
Set the version history entry to 3.13.0, matching the other macro pages
documenting unreleased features.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
lexer.hpp had grown by ~600 lines of byte-level helpers that have no
dependency on the lexer's template parameters and clutter the state
machine. Move them, unchanged, into two focused headers as free functions
in namespace detail:
- number_parse.hpp: parse_integer_unsigned/parse_integer_signed (now
templated on the number type) and parse_float_fast (Clinger's exact
double fast path, with the decimal point passed as an argument instead of
read from a lexer member).
- string_scan.hpp: the SWAR string helpers (is_string_special,
swar_string_special, find_string_special, validate_one_utf8,
scalar_string_bulk_run) and the backend-dispatched string_bulk_run,
including the optional simdutf include and find_string_delimiter.
lexer.hpp now includes these and calls the free functions; the methods that
touch lexer state (scan_string, scan_number, scan_string_bulk,
scan_number_bulk_contiguous, convert_number) stay put. This is a pure code
move with no behavior change: lexer.hpp drops from 2357 to 1934 lines, the
now-unused <cstdint>/<cstring>/<limits> includes are removed, and the
free-function form makes the SWAR helpers reusable elsewhere (e.g. the
serializer's string escaping).
Verified: default and JSON_USE_SIMDUTF builds compile; 2,000,000 number and
2,000,000 arbitrary-byte-string differential-fuzz documents parse
identically to before; lexer/parser/conversions/deserialization/locale/
diagnostic-position suites pass (20,576 assertions); warning-clean on g++
and clang in C++11/17/20; the amalgamation regenerates and passes
check-amalgamation.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Niels LohmannandClaude Opus 4.8