Niels Lohmann
c6742f9761
The string fast path found the end of a run of plain characters 8 bytes at a time (SWAR) and validated non-ASCII text one sequence at a time, calling validate_one_utf8() out of line for each. The vector code of json_view's scanner, in the new shared header detail/input/simd_scan.hpp, now does both 16 bytes at a time: SSE2 finds the end of a run on x86-64, and the UTF-8 check of Keiser and Lemire runs with NEON on AArch64 and with SSSE3 on x86-64. SSSE3 is not part of x86-64, so the check is compiled for it with a function attribute and used when the CPU has it (cached in a constant-initialized atomic), else the scalar check; code compiled with -mssse3 skips the CPU check. The first word of a run is still checked with SWAR (the first four with NEON, where a vector mask costs more), as most keys and short values end there, and the scalar helpers are always inlined. On anything else, the scalar code is unchanged. The new macro JSON_NO_SIMD selects the portable code everywhere, including the SSE2/NEON digit formatting of the floating-point writer. The lexer tests now also run with JSON_NO_SIMD and, on x86-64, with -mssse3, and a new test places every byte that ends a run at each offset across several 16-byte blocks, after ASCII and after CJK text. Signed-off-by: Niels Lohmann <mail@nlohmann.me>
3.4 KiB
JSON_USE_SIMDUTF
#define JSON_USE_SIMDUTF
When defined, the parser validates the UTF-8 content of JSON strings that come from a contiguous byte input
(std::string, std::vector<char>/<std::uint8_t>, string literals, const char* ranges, …) using the
simdutf library instead of the built-in validator (which uses SSSE3 or NEON where
available, see JSON_NO_SIMD). On text with many non-ASCII characters (e.g. CJK or emoji) this can
validate faster, depending on the CPU.
This is an opt-in external dependency. The library itself remains header-only and its behavior is unchanged: the
same input is accepted or rejected either way, and every parse error is reported at the same position with the same
message (simdutf is only used to fast-path valid runs; anything it flags falls back to the scalar path so the exact
diagnostic is preserved). A std::istream uses simdutf for the characters its stream buffer holds (see
JSON_DISABLE_STREAM_BULK_SCAN); other streaming inputs (FILE*, wide strings,
user-defined adapters) always use the scalar path.
When JSON_USE_SIMDUTF is defined you must make the simdutf.h header available on the include path and link the
simdutf library. When it is not defined, no simdutf header is included and there is no dependency.
!!! note "Requires C++17"
simdutf requires C++17 and its header rejects older standards with an `#!cpp #error`. The backend is therefore only
compiled in from C++17 on. In C++11 and C++14 the macro has no effect and the scalar validator is used, which
accepts and rejects exactly the same input -- only throughput differs. Setting the macro project-wide is therefore
safe even when some translation units are built with an older standard.
!!! warning "Define consistently"
The macro selects between two definitions of the same inline validation function. It must therefore be defined
identically for **every** translation unit that includes the library; mixing translation units that define it with
ones that do not is an ODR violation. Prefer setting it as a compile definition on the target rather than with
`#!cpp #define` in individual source files.
Default definition
By default, #!cpp JSON_USE_SIMDUTF is not defined and the portable C++11 scalar validator is used.
#undef JSON_USE_SIMDUTF
Examples
??? example
The code below enables the simdutf backend for UTF-8 validation.
```cpp
#define JSON_USE_SIMDUTF 1
#include <nlohmann/json.hpp>
...
```
The project must also link against simdutf, e.g. with CMake:
```cmake
target_compile_definitions(your_target PRIVATE JSON_USE_SIMDUTF)
target_link_libraries(your_target PRIVATE simdutf::simdutf)
```
!!! hint "Testing this configuration"
The unit tests can be built against the simdutf backend with the CMake option
[`JSON_TestSimdutf`](../../integration/cmake.md#json_testsimdutf) (`OFF` by default), which fetches simdutf and
defines `JSON_USE_SIMDUTF` for every test target. The `ci_test_simdutf` target runs the whole test suite in that
configuration.
See also
- :simple-cmake: JSON_TestSimdutf - CMake option to build the unit tests against the simdutf backend
Version history
- Added in version 3.13.0.