Niels LohmannandClaude Opus 4.8
9a86ad483c
The bulk string scanner validates UTF-8 straight from a contiguous buffer. The scalar validator caps at ~0.3-0.7 GB/s on non-ASCII text; a SIMD validator reaches several GB/s. Rather than hand-rolling SIMD UTF-8 validation (easy to get subtly wrong - a from-scratch SSE attempt rejected valid CJK), wire in the vetted simdutf library behind an opt-in switch. simdutf is not header-only (it ships simdutf.cpp and uses runtime CPU dispatch), so it is not vendored: defining JSON_USE_SIMDUTF includes <simdutf.h> and routes the bulk validator through simdutf::validate_utf8; the project supplies and links simdutf. Undefined (the default), nothing external is included and the portable C++11 scalar path is used, so the library stays header-only and its baseline behavior is unchanged. Design keeps behavior identical either way: - scan_string_bulk() now finds the run up to the next quote/escape/control byte (non-ASCII allowed) and validates it in one shot; on the rare validation failure it recomputes the exact valid prefix with the scalar helper, so ill-formed input still falls through to the byte path and is reported at the same position with the same message. - the per-sequence scalar path is factored into scalar_string_bulk_run() and is the default backend; the refactor is behavior-preserving and does not change scalar throughput. Verified: default and JSON_USE_SIMDUTF builds accept/reject/parse identically across 2,000,000 arbitrary-byte documents and 1,000,000 mixed-escape/UTF-8 documents (differential fuzz vs the streaming byte path); lexer/parser/diagnostic-position/deserialization suites pass under both configurations (20,188 assertions with the backend enabled); warning-clean on g++ and clang, C++11 and C++20, both configurations. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA Signed-off-by: Niels Lohmann <mail@nlohmann.me>
1.8 KiB
JSON_USE_SIMDUTF
#define JSON_USE_SIMDUTF
When defined, the parser validates the UTF-8 content of JSON strings that come from a contiguous byte input
(std::string, std::vector<char>/<std::uint8_t>, string literals, const char* ranges, …) using the
simdutf library instead of the built-in scalar validator. On text with many
non-ASCII characters (e.g. CJK or emoji) this can validate several times faster.
This is an opt-in external dependency. The library itself remains header-only and its behavior is unchanged: the
same input is accepted or rejected either way, and every parse error is reported at the same position with the same
message (simdutf is only used to fast-path valid runs; anything it flags falls back to the scalar path so the exact
diagnostic is preserved). Streaming inputs (files, std::istream, wide strings, user-defined adapters) always use the
scalar path.
When JSON_USE_SIMDUTF is defined you must make the simdutf.h header available on the include path and link the
simdutf library. When it is not defined, no simdutf header is included and there is no dependency.
Default definition
By default, #!cpp JSON_USE_SIMDUTF is not defined and the portable C++11 scalar validator is used.
#undef JSON_USE_SIMDUTF
Examples
??? example
The code below enables the simdutf backend for UTF-8 validation.
```cpp
#define JSON_USE_SIMDUTF 1
#include <simdutf.h>
#include <nlohmann/json.hpp>
...
```
The project must also link against simdutf, e.g. with CMake:
```cmake
target_compile_definitions(your_target PRIVATE JSON_USE_SIMDUTF)
target_link_libraries(your_target PRIVATE simdutf::simdutf)
```
Version history
- Added in version 3.12.1.