mirror of
https://github.com/nlohmann/json.git
synced 2026-08-18 23:23:21 +00:00
Add SWAR bulk string scanning for contiguous input (simdjson-style)
scan_string() read the input one character at a time through the input adapter and classified every byte with a large switch. For contiguous byte buffers we can instead scan 8 bytes at a time with a SWAR word test that finds the first byte needing individual handling (the closing quote, an escape, a control character, or a non-ASCII UTF-8 byte) and bulk-append the ordinary run in one go. - input adapters expose supports_bulk_scan / bulk_data / bulk_remaining / bulk_skip for provably-contiguous, same-type, 1-byte iterator ranges (raw pointers in every standard; std::string/std::vector/std::array and friends additionally in C++20 via std::contiguous_iterator). - the lexer gains a bulk_scan capability (gated on lazy_token_string so bypassing the per-character capture cannot lose error diagnostics) and a scan_string_bulk() fast path; streaming/wide/user adapters are unchanged and keep the byte-at-a-time scanner. The run contains no newline (all bytes < 0x20 are treated as special), so position bookkeeping stays exact, and error tokens are still reconstructed lazily from the consumed byte range. The SWAR special-byte test is pure uint64_t arithmetic - no intrinsics, no runtime dispatch, C++11-clean. Measured on representative data, pointer input, g++ 13 -O3 (string values discarded by accept() see the largest gains): long ASCII strings: DOM +4.5x, SAX +14x, accept +17x (to ~2 GB/s) short strings: DOM +15%, SAX +62%, accept +85% escape-heavy: DOM +31%, SAX +26%, accept +28% Same-input parity verified: 200k randomized documents (escapes, multibyte UTF-8, surrogate pairs) accept/parse identically via the contiguous SWAR path and the streaming byte path; unit lexer/parser/diagnostic-position/ deserialization/conversions suites pass unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
This commit is contained in:
@@ -231,6 +231,33 @@ class iterator_input_adapter
|
||||
std::is_same<IteratorType, SentinelType>::value && std::is_pointer<IteratorType>::value;
|
||||
#endif
|
||||
|
||||
public:
|
||||
// Whether the remaining input is a single contiguous block of 1-byte
|
||||
// elements that the lexer can inspect directly (used for the SWAR string
|
||||
// fast path). Restricted to same-type iterator/sentinel pairs so that plain
|
||||
// std::distance/std::advance are well-defined in all standards.
|
||||
static constexpr bool supports_bulk_scan =
|
||||
iterator_is_contiguous && std::is_same<IteratorType, SentinelType>::value && sizeof(char_type) == 1;
|
||||
|
||||
// Pointer to the next unread element; only valid when bulk_remaining() > 0.
|
||||
const char_type* bulk_data() const
|
||||
{
|
||||
return &*current;
|
||||
}
|
||||
|
||||
// Number of unread elements available as one contiguous block.
|
||||
std::size_t bulk_remaining() const
|
||||
{
|
||||
return static_cast<std::size_t>(std::distance(current, end));
|
||||
}
|
||||
|
||||
// Consume @a n elements previously inspected via bulk_data().
|
||||
void bulk_skip(std::size_t n)
|
||||
{
|
||||
std::advance(current, static_cast<typename std::iterator_traits<IteratorType>::difference_type>(n));
|
||||
}
|
||||
|
||||
private:
|
||||
// contiguous fast path: bulk copy the remaining range with std::memcpy
|
||||
template<class T>
|
||||
std::size_t get_elements_impl(T* dest, std::size_t count, std::true_type /*contiguous*/)
|
||||
|
||||
Reference in New Issue
Block a user