amorfo77/json
1
0 Fork 0
mirror of https://github.com/nlohmann/json.git synced 2026-10-17 19:48:12 +00:00
Code Issues Packages Projects Releases Wiki Activity

Count lines incrementally when a recovering parse reports many errors

Browse Source 
With seekable input, the lexer derives line and column on demand (#5809)
by counting the newlines in all input consumed so far. A SAX parser that
recovers from errors asks for the position once per error, so input with
an error at every character took quadratic time: "{" followed by 100000
commas took 85 s at -O0, and the parser tests timed out under Valgrind
(1500 s, was 69 s on develop).

get_position() now remembers what it counted and only scans the input
consumed since the previous call; count_consumed_lines() takes the start
of the range. The 100000 commas now take 0.17 s, unit-class_parser 0.9 s
instead of 86 s. Over 69284 error messages from random multi-line inputs,
the messages are identical to those before the change.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
This commit is contained in:
Niels Lohmann committed 2026-10-17 20:10:10 +02:00
1 parent fb729028ba
commit 24de5b2f5c
3 files changed
+78 -26
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

include/nlohmann/detail/input/input_adapters.hpp
+13 -8
View File
@@ -314,17 +314,22 @@ class iterator_input_adapter
return get_consumed_count() + eof_reads;
}
// number of newlines in the consumed range [0, last_index); @a line_start
// is set to the index after the last of them (0 if there is none); only
// valid when supports_seek
std::size_t count_consumed_lines(std::size_t last_index, std::size_t& line_start) const
// number of newlines in the consumed range [first_index, last_index); if
// there is one, @a line_start is set to the index after the last of them,
// otherwise it is left unchanged; only valid when supports_seek
std::size_t count_consumed_lines(std::size_t first_index, std::size_t last_index, std::size_t& line_start) const
{
const auto to = std::next(begin, static_cast<typename std::iterator_traits<IteratorType>::difference_type>(last_index));
using difference_type = typename std::iterator_traits<IteratorType>::difference_type;
const auto from = std::next(begin, static_cast<difference_type>(first_index));
const auto to = std::next(from, static_cast<difference_type>(last_index - first_index));
const auto newline = char_traits<char_type>::to_char_type('\n');
using reverse_it = std::reverse_iterator<IteratorType>;
const auto last_newline = std::find(reverse_it(to), reverse_it(begin), newline);
line_start = static_cast<std::size_t>(std::distance(begin, last_newline.base()));
return static_cast<std::size_t>(std::count(begin, last_newline.base(), newline));
const auto last_newline = std::find(reverse_it(to), reverse_it(from), newline);
if (last_newline.base() != from)
{
line_start = static_cast<std::size_t>(std::distance(begin, last_newline.base()));
}
return static_cast<std::size_t>(std::count(from, last_newline.base(), newline));
}
// append the already-consumed characters in the half-open range