mirror of
https://github.com/nlohmann/json.git
synced 2026-11-13 15:35:23 +00:00
Niels Lohmann
71baba09fe
For "last read" in parse errors, the lexer keeps the text of the token it reads. The token start moves only at strings and numbers, so everything after a string or number belongs to it until the next one starts. Input that cannot be read again copies that text while parsing (the istream and FILE* adapters in their carry buffer, the lexer in token_string for iterators), so valid input of 1 followed by 100,000,000 newlines peaked at about 225 MB, and a long string was kept a second time. Replace the carry buffer and token_string by token_text_buffer, which keeps the first 4096 and at least the last 4112 characters of a token and drops the characters in between where it would otherwise grow, so appending stays as cheap as before. get_token_string() shows a token of more than 8192 characters as its first and last 4096 characters around "...", the same for every kind of input; contiguous input copies only these parts. The example input now peaks at 1.8 MB. Instruction counts of parsing from std::string, std::istream and FILE* are unchanged (clang 21 and GCC 15); parsing from std::list<char> needs 1 to 17 % fewer. Fixes #6120. Signed-off-by: Niels Lohmann <mail@nlohmann.me>