* Parse JSON in one block of memory with a fused pointer-based parser The parser and the lexer exchange one token at a time, and the lexer keeps its state (the current character, a pending unget, where the token starts) in members, which every character pays for: json::accept() of twitter.json took 2.8x as long as json_document::accept(). detail::contiguous_parser reads text in one block of memory through a local pointer, with the grammar, the whitespace and the tokens in one loop (after yyjson's reader), and drives a SAX handler: parse()'s DOM handler, accept()'s handler, or the user's handler of sax_parse(). It is used with the default options (no callback, no comments, no trailing commas; sax_parse() with strict = true). Errors are reported by the library parser, so the exception, its message and its position stay the same. contiguous_parser stops at the token that the library parser would reject, before reporting it, so the events up to the error are the same; then: - parse() runs the library parser with a handler that only reports the error (json_sax_error_reporter), with the lexer of parse(), which is instantiated anyway; - sax_parse() runs it with a handler that forwards only parse_error() to the user's handler (json_sax_error_forwarder), so no event arrives twice; - accept() returns the verdict. A handler that returns false stops the parse as before. An error at the end of a large input costs parse() about 1.2x of what it cost before. On an Apple M1 Max: accept() 1.6-2.0x as fast, parse() 10-16% faster, ordered_json 11-16%, sax_parse() with a user's handler 1.36-1.46x, small messages 12-22% (parse()) and 2.1-2.4x (accept()). On x86-64 (GCC 15, Clang 21): accept() 1.6-1.9x, parse() 5-17%, ordered_json 12-19%, sax_parse() 1.32-1.57x. A translation unit that parses a string compiles 3% (-O0) and 9% (-O2) slower. unit-contiguous_parser.cpp compares the contiguous path with the byte path (std::list iterators) on corner cases, generated documents, their truncations and mutations, deep nesting, and narrow number types: the same values with the same number types and float bits, exception messages, accept() verdicts and sax_parse() events. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix CI: unused parameter without exceptions, cpplint, BUILD.bazel - json_sax_error_reporter::parse_error: mark the exception unused, as JSON_THROW does not use it with JSON_NOEXCEPTION - unit-contiguous_parser.cpp: compare the exceptions of both paths only when exceptions are enabled (invalid input aborted the test otherwise) - silence cpplint's readability/braces false positives on the if statements inside the parser's macros - add contiguous_parser.hpp to BUILD.bazel Signed-off-by: Niels Lohmann <mail@nlohmann.me> * accept(): skip digits without computing their value accept() needs the value of a number only for a float whose exponent may reach the largest value. skip_digits() finds the end of the digits without the multiply-adds of accumulate_digits(), and once an 8-byte word fails the digit test, the remaining loop needs no bounds check: a non-digit is in that word. The value is computed afterwards for the rare floats that need it. accept() of canada.json: -16 % on Apple M1, -22 % with GCC 15 and -21 % with Clang 21 on x86-64; files with many numbers -9 to -15 %. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Tests: mark parser callback in unit-contiguous_parser noexcept Fixes the -Werror=noexcept failure with GCC 26 (ci_test_standards_gcc (26)). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Tests: sweep_allocator::construct() accepts pointers to const Recent libc++ (emsdk latest) probes construct(const T*, const T&) in an unevaluated context while checking for trivial copies; the static_cast to void* in the noexcept specification then fails to compile. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz (cherry picked from commit 794bf70823c12a2a1cb2a8a95f34dfb6357746a8) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Tests: guard sweep_allocator::construct() against const types instead of const_cast The const_cast added in the previous commit is rejected by cppcoreguidelines-pro-type-const-cast (clang-tidy). Make construct() not viable for const U instead, which is what libc++ probes for. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz (cherry picked from commit de099fa607001b710862fd1bbecef3609c54cbc4) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix CI: astyle and clang-tidy for sweep_allocator::construct() astyle 3.4.13 reformats the enable_if<!...> default template argument (Check amalgamation). Delete the const overload instead, which makes the libc++ probe for const objects fail in the SFINAE context just the same and needs no <type_traits>. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Contiguous parser: review fixes (wording, macros, diagnostic positions) - "default parser" instead of "library parser" in comments and tests - the three comments in json.hpp say the same thing - current() replaces the repeated `p != end ? *p : 0` - the backslashes of each macro are aligned - the docs say that JSON_DIAGNOSTIC_POSITIONS turns the fast path off Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
5.7 KiB
JSON_DIAGNOSTIC_POSITIONS
#define JSON_DIAGNOSTIC_POSITIONS /* value */
This macro enables position diagnostics for generated JSON objects.
When enabled, two new member functions start_pos() and
end_pos() are added to basic_json values. If the value was
created by calling theparse function, then these functions allow querying the byte positions
of the value in the input it was parsed from. In case the value was constructed by other means, std::string::npos is
returned.
start_pos() returns the position of the first character of a given value in the original
JSON string, while end_pos() returns the position of the character following the last
character. For objects and arrays, the first and last characters correspond to the opening or closing braces/brackets,
respectively. For primitive values, the first and last character represents the opening and closing quotes (strings) or
the first and last character of the field's numerical or predefined value (true, false, null), respectively.
| JSON type | return value start_pos() |
return value end_pos() |
|---|---|---|
| object | position of the opening { |
position after the closing } |
| array | position of the opening [ |
position after the closing ] |
| string | position of the opening " |
position after the closing " |
| number | position of the first character | position after the last character |
| boolean | position of t for true and f for false |
position after e |
| null | position of n |
position after l |
Given the above, end_pos()-start_pos() for a JSON
value provides the length of the parsed JSON string for that value, including the opening or closing braces, brackets,
or quotes.
Note that enabling this macro increases the size of every JSON value by two std::size_t fields and adds slight runtime
overhead to parsing, copying JSON value objects, and the generation of error messages for exceptions. It also causes
these values to be reported in those error messages.
In addition, parse(), accept(), and sax_parse() no longer use the
faster path for input in one block of memory, so
parsing a string or a buffer is slower than without the macro.
If JSON_DIAGNOSTICS is also set, the message contains the path and the byte range, such as
(/1) (bytes 4-8). An exception about the root value has no path and only the range, such as (bytes 0-4).
Default definition
The default value is 0 (position diagnostics are switched off).
#define JSON_DIAGNOSTIC_POSITIONS 0
When the macro is not defined, the library will define it to its default value.
Notes
!!! note "CMake option"
Diagnostic positions can also be controlled with the CMake option
[`JSON_Diagnostic_Positions`](../../integration/cmake.md#json_diagnostic_positions) (`OFF` by default)
which defines `JSON_DIAGNOSTIC_POSITIONS` accordingly.
!!! note "Availability"
Diagnostic positions are only available if the value was created by the [`parse`](../basic_json/parse.md) function.
The [`sax_parse`](../basic_json/sax_parse.md) function or all other means to create a JSON value **do not** set the
diagnostic positions and [`start_pos()`](../basic_json/start_pos.md) and [`end_pos()`](../basic_json/end_pos.md)
will only return `std::string::npos` for these values.
!!! warning "Invalidation"
The returned positions are only valid as long as the JSON value is not changed. The positions are *not* updated
when the JSON value is changed.
Examples
??? example "Example: retrieving positions"
```cpp
--8<-- "examples/diagnostic_positions.cpp"
```
Output:
```
--8<-- "examples/diagnostic_positions.output"
```
The output shows the start/end positions of all the objects and fields in the JSON string.
??? example "Example: using only diagnostic positions in exceptions"
```cpp
--8<-- "examples/diagnostic_positions_exception.cpp"
```
Output:
```
--8<-- "examples/diagnostic_positions_exception.output"
```
The output shows the exception with start/end positions only.
??? example "Example: using extended diagnostics with positions enabled in exceptions"
```cpp
--8<-- "examples/diagnostics_extended_positions.cpp"
```
Output:
```
--8<-- "examples/diagnostics_extended_positions.output"
```
The output shows the exception with diagnostic path info and start/end positions.
See also
- :simple-cmake: JSON_Diagnostic_Positions - CMake option to control the macro
JSON_DIAGNOSTICS- macro to control extended diagnosticsmaterialize()- values of a JSON view, with the same positions- Parsing and Exceptions - handling parse errors and locating them in the input
Version history
- Added in version 3.12.0.