Parsing from a std::istream read one character at a time through the streambuf, so none of the parser's fast paths applied: it was 1.8 to 2.5 times slower than reading the file into a string and parsing that. input_stream_adapter now exposes the streambuf's get area (gptr() to egptr()) to the lexer as a window. The protected get-area members are reached through member pointers formed in a class derived from std::streambuf, which is ordinary, standard-conforming member access. The lexer scans the window with the same bulk paths as a string, and asks the streambuf for the next character as before when the window is used up. Nothing is read ahead, copied, or put back, so the stream is left exactly where it was left before, with and without JSON_PRECISE_STREAM_POSITION, and a streambuf without a get area is still read one character at a time. The adapter also keeps the current token for error messages as a pointer into the get area, and copies it out only before the get area is refilled, instead of the lexer copying every character. On an Apple M1 Max (Apple Clang, -O3), parsing from an std::ifstream is 1.6 to 2.1 times as fast (twitter.json 4.08 -> 2.30 ms, citm_catalog.json 8.75 -> 4.08 ms), and reading 100,000 small values with operator>> 1.4 (objects) to 4.0 (integers) times as fast. With GCC 13 and libstdc++ on x86-64 Linux, std::ifstream is 1.5 to 2.2 times as fast. Parsing from a stream is now within 11 to 18 % of reading the file into a string first. JSON_DISABLE_STREAM_BULK_SCAN restores reading one character at a time, for streambufs that do not follow the usual get-area contract. The new tests compare every observable result (values, exception messages, stream state, remaining characters) for get areas of 1 to 4096 bytes against a streambuf without a get area, and run again with the macro and with JSON_PRECISE_STREAM_POSITION. Signed-off-by: Niels Lohmann <mail@nlohmann.me>
11 KiB
nlohmann::basic_json::parse
// (1)
template<typename InputType>
static basic_json parse(InputType&& i,
const parser_callback_t cb = nullptr,
const bool allow_exceptions = true,
const bool ignore_comments = false,
const bool ignore_trailing_commas = false);
// (2)
template<typename IteratorType, typename SentinelType = IteratorType>
static basic_json parse(IteratorType first, SentinelType last,
const parser_callback_t cb = nullptr,
const bool allow_exceptions = true,
const bool ignore_comments = false,
const bool ignore_trailing_commas = false);
-
Deserialize from a compatible input.
-
Deserialize from a pair of character iterators, or an iterator and a sentinel of a different type (C++20 ranges support)
The
value_typeof the iterator must be an integral type with size of 1, 2, or 4 bytes, which will be interpreted respectively as UTF-8, UTF-16, and UTF-32. IfSentinelTypediffers fromIteratorType, it must be comparable to the iterator type withoperator!=.
Template parameters
InputType- A compatible input, for instance:
- an
std::istreamobject - a
FILEpointer (throws if null) - a C-style array of characters
- a pointer to a null-terminated string of single byte characters (throws if null)
- a
std::string - a container
objfor whichbegin(obj)andend(obj)produce a valid pair of iterators (as found via ADL or member functions, with semantics compatible tostd::beginandstd::end)
- an
IteratorType- a compatible iterator type, for instance.
- a pair of
std::string::iteratororstd::vector<std::uint8_t>::iterator - a pair of pointers such as
ptrandptr + len
- a pair of
SentinelType- defaults to
IteratorType; may be a different type comparable toIteratorTypeviaoperator!=, for instance.- a custom sentinel type for C++20 ranges
std::default_sentinel_t, whenIteratorTypeisstd::counted_iterator
Parameters
i(in)- Input to parse from.
cb(in)- a parser callback function of type
parser_callback_twhich is used to control the deserialization by filtering unwanted values (optional) allow_exceptions(in)- whether to throw exceptions in case of a parse error (optional,
#!cpp trueby default) ignore_comments(in)- whether comments should be ignored and treated like whitespace (
#!cpp true) or yield a parse error (#!cpp false); (optional,#!cpp falseby default) ignore_trailing_commas(in)- whether trailing commas in arrays or objects should be ignored and treated like whitespace (
#!cpp true) or yield a parse error (#!cpp false); (optional,#!cpp falseby default) first(in)- iterator to the start of a character range
last(in)- iterator to the end of a character range, or a sentinel value that compares equal to the end iterator with
operator!=
Return value
Deserialized JSON value; in case of a parse error and allow_exceptions set to #!cpp false, the return value will be
value_t::discarded. The latter can be checked with is_discarded.
Exception safety
Strong guarantee: if an exception is thrown, there are no changes in the JSON value.
Exceptions
- Throws
parse_error.101in case of an unexpected token, or empty input like a nullFILE*orchar*pointer, or anstd::istreamwithout a stream buffer (#!cpp i.rdbuf() == nullptr, for instance#!cpp std::istream(nullptr)). - If reading from an
std::istreamreaches the end of the input andeofbitis part of the stream'sexceptions()mask, thestd::ios_base::failurethrown by the stream itself propagates instead of aparse_error, the same as it would for the standard library's own extraction operators.
Complexity
Linear in the length of the input. The parser is a predictive LL(1) parser. The complexity can be higher if the parser
callback function cb or reading from (1) the input i or (2) the iterator range [first, last] has a
super-linear complexity.
Notes
A UTF-8 byte order mark is silently ignored.
Invalid Unicode escapes and unpaired surrogates in the input are reported as
parse_error.101 with a detailed message.
By default, a '\0' (NUL) byte anywhere in the input is treated as end of input, rather than as an ordinary (and,
outside of a string, invalid) byte; see the FAQ entry for details and the
JSON_STRICT_NUL_HANDLING macro to opt into rejecting it instead.
!!! tip "Performance"
Contiguous inputs (strings, string views, vectors, arrays, pointer ranges) are the fastest to parse. A
`#!cpp std::istream` comes close: the parser scans the characters its stream buffer holds in bulk, without reading
ahead, so the stream is left exactly where it would be after reading one character at a time. See
[input sources and performance](../../features/parsing/index.md#input-sources-and-performance) and
[`JSON_DISABLE_STREAM_BULK_SCAN`](../macros/json_disable_stream_bulk_scan.md).
Examples
??? example "Example: (1) parse from a character array"
The example below demonstrates the `parse()` function reading from an array.
```cpp
--8<-- "examples/parse__array__parser_callback_t.cpp"
```
Output:
```json
--8<-- "examples/parse__array__parser_callback_t.output"
```
??? example "Example: (1) parse from a string"
The example below demonstrates the `parse()` function with and without callback function.
```cpp
--8<-- "examples/parse__string__parser_callback_t.cpp"
```
Output:
```json
--8<-- "examples/parse__string__parser_callback_t.output"
```
??? example "Example: (1) parse from an input stream"
The example below demonstrates the `parse()` function with and without callback function.
```cpp
--8<-- "examples/parse__istream__parser_callback_t.cpp"
```
Output:
```json
--8<-- "examples/parse__istream__parser_callback_t.output"
```
??? example "Example: (1) parse from a contiguous container"
The example below demonstrates the `parse()` function reading from a contiguous container.
```cpp
--8<-- "examples/parse__contiguouscontainer__parser_callback_t.cpp"
```
Output:
```json
--8<-- "examples/parse__contiguouscontainer__parser_callback_t.output"
```
??? example "Example: (2) parse from a non-null-terminated string"
The example below demonstrates the `parse()` function reading from a string that is not null-terminated.
```cpp
--8<-- "examples/parse__pointers.cpp"
```
Output:
```json
--8<-- "examples/parse__pointers.output"
```
??? example "Example: (2) parse from an iterator pair"
The example below demonstrates the `parse()` function reading from an iterator pair.
```cpp
--8<-- "examples/parse__iterator_pair.cpp"
```
Output:
```json
--8<-- "examples/parse__iterator_pair.output"
```
??? example "Example: effect of allow_exceptions parameter"
The example below demonstrates the effect of the `allow_exceptions` parameter in the `parse()` function.
```cpp
--8<-- "examples/parse__allow_exceptions.cpp"
```
Output:
```json
--8<-- "examples/parse__allow_exceptions.output"
```
??? example "Example: effect of ignore_comments parameter"
The example below demonstrates the effect of the `ignore_comments` parameter in the `parse()` function.
```cpp
--8<-- "examples/comments.cpp"
```
Output:
```
--8<-- "examples/comments.output"
```
??? example "Example: effect of ignore_trailing_commas parameter"
The example below demonstrates the effect of the `ignore_trailing_commas` parameter in the `parse()` function.
```cpp
--8<-- "examples/trailing_commas.cpp"
```
Output:
```
--8<-- "examples/trailing_commas.output"
```
See also
- accept - check if the input is valid JSON
- sax_parse - parse input using the SAX interface
- operator>> - deserialize from stream
JSON_STRICT_NUL_HANDLING- opt in to rejecting a NUL byte in the input instead of treating it as end of inputJSON_DISABLE_STREAM_BULK_SCAN- read astd::istreamone character at a time
Version history
- Added in version 1.0.0.
- Overload for contiguous containers (1) added in version 2.0.3.
- Ignoring comments via
ignore_commentsadded in version 3.9.0. - Changed runtime assertion in case of
FILE*null pointers to exception in version 3.12.0. - Added
ignore_trailing_commasin version 3.13.0. - Extended container support (1) to include types with lvalue-only ADL
begin/end(matchingstd::begin/std::endsemantics) in version 3.13.0. - Extended overload (2) to accept heterogeneous iterator+sentinel pairs (C++20 ranges support) in version 3.13.0.
JSON_STRICT_NUL_HANDLINGadded in version 3.13.0 to optionally reject a NUL byte in the input instead of treating it as end of input; planned to become the default in version 4.0.0.- Extended empty-input detection to also cover an
std::istreamwithout a stream buffer, and fixed a crash (std::terminate) when parsing from anstd::istreamwitheofbitin its exception mask, in version 3.13.0. - Parsing from an
std::istreamscans the characters in its stream buffer in bulk in version 3.13.0 (JSON_DISABLE_STREAM_BULK_SCANrestores reading one character at a time).
!!! warning "Deprecation"
Overload (2) replaces calls to `parse` with a pair of iterators as their first parameter which has been
deprecated in version 3.8.0. This overload will be removed in version 4.0.0. Please replace all calls like
`#!cpp parse({ptr, ptr+len}, ...);` with `#!cpp parse(ptr, ptr+len, ...);`.
You should be warned by your compiler with a `-Wdeprecated-declarations` warning if you are using a deprecated
function.
See the [migration guide](../../integration/migration_guide.md#parsing) for how to update existing code.