Derive the lexer position lazily for seekable input (#5809)

* Derive the lexer position lazily for seekable input

The lexer counted the read position on every character: two increments
and a newline check per get(), all on members kept in memory. For
seekable input (pointers, strings, other random-access byte ranges) the
position can be derived instead: chars_read_total is the adapter's read
count (including reads at EOF) minus a pending unget, and line and
column are recomputed from the consumed input only when get_position()
is called, i.e. on parse errors. JSON_DIAGNOSTIC_POSITIONS uses the new
O(1) get_chars_read_total(). Stream, file, wide-string and user-defined
adapters keep the eager counters.

Parsing pretty-printed input gets 8-12% faster and accept() 12-27%
(canada, citm_catalog, twitter; clang 21 arm64 and GCC 13 x86-64).
Error messages, byte offsets, SAX positions and diagnostic positions are
unchanged (differential comparison against develop on 548k inputs).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Move the std::vector<std::byte> binary type tests out of unit-regression3

The #4804/#4209 tests instantiated a second basic_json specialization
(with_binary_t<std::vector<std::byte>>), including its parser and CBOR
reader. With clang 18, the C++20 object of unit-regression3.cpp had
32656 COFF sections on develop; the lazy position helpers pushed it to
32975, past what the MinGW linker of the Windows clang jobs handles
("relocation truncated to fit: IMAGE_REL_AMD64_REL32 against .rdata").
unit-custom-binary-type.cpp already instantiates the same type, so the
tests move there; unit-regression3.cpp drops to 27616 sections.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Report the parse errors of the position test via SAX

The test caught json::parse_error, so it aborted in the
ci_test_noexceptions build, where JSON_NOEXCEPTION turns the throw into
std::abort(). sax_parse() hands the error position and the exception
object to parse_error() without throwing, so the test now records them
there and runs in every configuration.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
This commit is contained in:
Niels Lohmann authored and GitHub committed 2026-10-13 17:16:52 +02:00
1 parent 40a62947d3
commit f23830b65e
9 files changed
+380 -105

No files matched your search

@@ -8,12 +8,12 @@
#pragma once
#include <algorithm> // min
#include <algorithm> // count, find, min
#include <array> // array
#include <cstddef> // size_t
#include <cstdint> // uint32_t
#include <cstring> // strlen
#include <iterator> // begin, end, iterator_traits, random_access_iterator_tag, distance, next
#include <iterator> // begin, end, iterator_traits, random_access_iterator_tag, distance, next, reverse_iterator
#include <streambuf> // streambuf
#include <string> // string, char_traits
#include <type_traits> // enable_if, is_base_of, is_pointer, is_integral, remove_pointer
@@ -297,6 +297,7 @@ class iterator_input_adapter
return result;
}
++eof_reads;
return char_traits<char_type>::eof();
}
@@ -306,6 +307,26 @@ class iterator_input_adapter
return static_cast<std::size_t>(std::distance(begin, current));
}
// number of get_character() calls so far, including those that returned
// EOF; the lexer derives its position from it (see lexer::lazy_position)
std::size_t get_read_count() const
{
return get_consumed_count() + eof_reads;
}
// number of newlines in the consumed range [0, last_index); @a line_start
// is set to the index after the last of them (0 if there is none); only
// valid when supports_seek
std::size_t count_consumed_lines(std::size_t last_index, std::size_t& line_start) const
{
const auto to = std::next(begin, static_cast<typename std::iterator_traits<IteratorType>::difference_type>(last_index));
const auto newline = char_traits<char_type>::to_char_type('\n');
using reverse_it = std::reverse_iterator<IteratorType>;
const auto last_newline = std::find(reverse_it(to), reverse_it(begin), newline);
line_start = static_cast<std::size_t>(std::distance(begin, last_newline.base()));
return static_cast<std::size_t>(std::count(begin, last_newline.base(), newline));
}
// append the already-consumed characters in the half-open range
// [first_index, last_index) to @a out; only valid when supports_seek
template<typename ContainerType>
@@ -422,6 +443,8 @@ class iterator_input_adapter
IteratorType begin;
IteratorType current;
SentinelType end;
/// number of get_character() calls that returned EOF
std::size_t eof_reads = 0;
template<typename BaseInputAdapter, size_t T>
friend struct wide_string_input_helper;