Niels Lohmann eed1587000 Reconstruct lexer diagnostics lazily for seekable input (#120) (#5234) ...

`lexer::get()` copied every scanned character into `token_string` on the
whole successful-parse hot path, yet that buffer is consumed only by
`get_token_string()` when rendering the "last read" fragment of a parse
error. On well-formed input the per-byte copy (plus the `unget()` pop)
is pure overhead that is always discarded.

For seekable input adapters - random-access, single-byte iterators such
as those backing `std::string`, `const char*`, and `std::vector<char>` -
the offending token is now reconstructed on demand from the input when
an error is reported, using a saved start offset, and the eager copy is
skipped. Streaming adapters (file, istream, wide-string, and user-defined
adapters) keep the eager copy; the strategy is chosen at compile time via
`input_adapter_supports_seek`, so adapters without the capability are
unaffected.

Error messages are byte-for-byte identical across all adapters, verified
by a new parity regression test. Microbenchmark (4 MB mixed JSON, parsed
from a std::string): ~149 -> ~160 MB/s, about +8%.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

2026-07-05 10:42:48 +02:00

..

nlohmann

Reconstruct lexer diagnostics lazily for seekable input (#120) (#5234)

2026-07-05 10:42:48 +02:00