Derive the lexer position lazily for seekable input

The lexer counted the read position on every character: two increments
and a newline check per get(), all on members kept in memory. For
seekable input (pointers, strings, other random-access byte ranges) the
position can be derived instead: chars_read_total is the adapter's read
count (including reads at EOF) minus a pending unget, and line and
column are recomputed from the consumed input only when get_position()
is called, i.e. on parse errors. JSON_DIAGNOSTIC_POSITIONS uses the new
O(1) get_chars_read_total(). Stream, file, wide-string and user-defined
adapters keep the eager counters.

Parsing pretty-printed input gets 8-12% faster and accept() 12-27%
(canada, citm_catalog, twitter; clang 21 arm64 and GCC 13 x86-64).
Error messages, byte offsets, SAX positions and diagnostic positions are
unchanged (differential comparison against develop on 548k inputs).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
This commit is contained in:
Niels Lohmann committed 2026-10-13 08:52:49 +02:00
1 parent 40a62947d3
commit 3dbed2a0cb
7 files changed
+328 -57

No files matched your search

+5 -3
View File
@@ -370,7 +370,7 @@ class parser
}
case token_type::end_of_input:
{
if (JSON_HEDLEY_UNLIKELY(m_lexer.get_position().chars_read_total == 1))
if (JSON_HEDLEY_UNLIKELY(m_lexer.get_chars_read_total() == 1))
{
return syntax_error(*sax, "attempting to parse an empty input; check that your input string or stream contains the expected JSON");
}
@@ -520,8 +520,10 @@ class parser
{
// MSVC 2015 reports C4100 (unreferenced parameter) when SAX::parse_error is static
static_cast<void>(sax);
return sax.parse_error(m_lexer.get_position(), m_lexer.get_token_string(),
parse_error::create(exception_id::syntax_error, m_lexer.get_position(), message, nullptr));
// computing the position is O(input read so far) for seekable input
const position_t position = m_lexer.get_position();
return sax.parse_error(position, m_lexer.get_token_string(),
parse_error::create(exception_id::syntax_error, position, message, nullptr));
}
std::string exception_message(const token_type expected, const std::string& context)