The SWAR bulk string path stopped at the first non-ASCII byte and handed
every multibyte character to the byte-at-a-time scanner, whose per-byte
get()/next_byte_in_range()/add() machinery runs at roughly half the speed
of validating straight from the buffer. As a result, dense non-ASCII text
(CJK, emoji, accented Latin) parsed ~10-15x slower than ASCII.
Fold well-formed UTF-8 into the bulk run: scan_string_bulk() now, on a
non-ASCII lead byte, validates one sequence with validate_one_utf8() -
which mirrors scan_string()'s per-byte switch ranges exactly (rejecting
overlong forms, surrogates, and out-of-range code points) - and appends it
in place, continuing until the closing quote, an escape, a control byte,
or an ill-formed sequence. All error handling still defers to the byte
path, so error messages and positions are byte-for-byte unchanged.
Because only well-formed content is fast-pathed and every rejection falls
through to the existing scanner, behavior is identical; the win is purely
throughput. Measured on pointer input (accept, string values discarded):
content g++ 13 clang 18
dense CJK 277 -> 648 ~605 MB/s (~2.3x)
dense emoji 299 -> 857 ~702 MB/s (~2.6-2.9x)
mixed 90% ASCII 246 -> 331 ~334 MB/s (~1.35x)
pure ASCII unchanged (~3.2 / 4.1 GB/s)
Verified: 2,000,000 randomized documents built from arbitrary bytes
(overlong, surrogate, truncated, out-of-range sequences) accept/reject and
parse identically via the contiguous path and the streaming byte path;
lexer/parser/diagnostic-position/deserialization/conversions suites pass
unchanged. Pure C++11, no intrinsics.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
scan_string() read the input one character at a time through the input
adapter and classified every byte with a large switch. For contiguous
byte buffers we can instead scan 8 bytes at a time with a SWAR word test
that finds the first byte needing individual handling (the closing quote,
an escape, a control character, or a non-ASCII UTF-8 byte) and bulk-append
the ordinary run in one go.
- input adapters expose supports_bulk_scan / bulk_data / bulk_remaining /
bulk_skip for provably-contiguous, same-type, 1-byte iterator ranges
(raw pointers in every standard; std::string/std::vector/std::array and
friends additionally in C++20 via std::contiguous_iterator).
- the lexer gains a bulk_scan capability (gated on lazy_token_string so
bypassing the per-character capture cannot lose error diagnostics) and a
scan_string_bulk() fast path; streaming/wide/user adapters are unchanged
and keep the byte-at-a-time scanner.
The run contains no newline (all bytes < 0x20 are treated as special), so
position bookkeeping stays exact, and error tokens are still reconstructed
lazily from the consumed byte range. The SWAR special-byte test is pure
uint64_t arithmetic - no intrinsics, no runtime dispatch, C++11-clean.
Measured on representative data, pointer input, g++ 13 -O3
(string values discarded by accept() see the largest gains):
long ASCII strings: DOM +4.5x, SAX +14x, accept +17x (to ~2 GB/s)
short strings: DOM +15%, SAX +62%, accept +85%
escape-heavy: DOM +31%, SAX +26%, accept +28%
Same-input parity verified: 200k randomized documents (escapes, multibyte
UTF-8, surrogate pairs) accept/parse identically via the contiguous SWAR
path and the streaming byte path; unit lexer/parser/diagnostic-position/
deserialization/conversions suites pass unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The number scanner converted its already-validated digit buffer with
std::strtoull/std::strtoll/std::strtod. Those pull in locale and errno
machinery and dominate number-heavy parsing (strtod runs at ~6 M/s).
Replace them with dedicated parsers over the validated buffer:
- parse_integer_unsigned / parse_integer_signed: accumulate digits with
overflow detection, falling back to the float path on overflow exactly
as the strtoull/strtoll round-trip check did. Overflow behavior is
unchanged for narrower or wider custom number types.
- parse_float_fast: Clinger's exact fast path for `double` (<=19 significant
digits, |exp10| <= 22, significand < 2^53), where significand * 10^exp is
exact under IEEE round-to-nearest. This is the same fast path used by
fast_float/simdjson. It is bit-identical to strtod on this subset and
declines (falling back to strtod) otherwise. Only `double` uses it; float
and long double keep std::strtof/std::strtold via a templated overload.
Measured on representative data (g++ 13, -O3):
- integers: DOM parse +11%, SAX +25-34%
- floats: DOM parse +37%, SAX +70% (clang: float DOM ~1.9x)
No dependencies added; header-only and C++11-clean. Existing parser,
lexer, conversion and deserialization unit tests pass unchanged; a
3M-value random-double fuzz matches strtod bit-for-bit.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
`lexer::get()` copied every scanned character into `token_string` on the
whole successful-parse hot path, yet that buffer is consumed only by
`get_token_string()` when rendering the "last read" fragment of a parse
error. On well-formed input the per-byte copy (plus the `unget()` pop)
is pure overhead that is always discarded.
For seekable input adapters - random-access, single-byte iterators such
as those backing `std::string`, `const char*`, and `std::vector<char>` -
the offending token is now reconstructed on demand from the input when
an error is reported, using a saved start offset, and the eager copy is
skipped. Streaming adapters (file, istream, wide-string, and user-defined
adapters) keep the eager copy; the strategy is chosen at compile time via
`input_adapter_supports_seek`, so adapters without the capability are
unaffected.
Error messages are byte-for-byte identical across all adapters, verified
by a new parity regression test. Microbenchmark (4 MB mixed JSON, parsed
from a std::string): ~149 -> ~160 MB/s, about +8%.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix: integer parsed as float when EINTR set in errno
* chore: make amalgamate
* chore: make pretty
---------
Co-authored-by: Stuart Gorman <Stuart.Gorman@kallipr.com>
* Add versioned inline namespace
Add a versioned inline namespace to prevent ABI issues when linking code
using multiple library versions.
* Add namespace macros
* Encode ABI information in inline namespace
Add _diag suffix to inline namespace if JSON_DIAGNOSTICS is enabled, and
_ldvcmp suffix if JSON_USE_LEGACY_DISCARDED_VALUE_COMPARISON is enabled.
* Move ABI-affecting macros into abi_macros.hpp
* Move std_fs namespace definition into std_fs.hpp
* Remove std_fs namespace from unit test
* Format more files in tests directory
* Add unit tests
* Update documentation
* Fix GDB pretty printer
* fixup! Add namespace macros
* Derive ABI prefix from NLOHMANN_JSON_VERSION_*
Niels LohmannandClaude Opus 4.8
Colby HaskellandGitHub
Florian AlbrechtskirchingerandGitHub
Niels LohmannandGitHub
Tobias Kux