amorfo77/json
1
0 Fork 0
mirror of https://github.com/nlohmann/json.git synced 2026-10-24 23:08:56 +00:00
Code Issues Packages Projects Releases Wiki Activity

Compare commits

merge into: amorfo77/json:claude/test-compilation-speed-993125
amorfo77/json:develop amorfo77/json:perf/view-compare amorfo77/json:gh-pages amorfo77/json:claude/nlohmann-json-docs-b89a59 amorfo77/json:claude/from-json-arith-array amorfo77/json:claude/implement-5891-deff16 amorfo77/json:docs/cross-reference-json-view-3-13-0 amorfo77/json:claude/fix-issue-3989-db7e45 amorfo77/json:pin-ci-dependencies amorfo77/json:claude/test-compilation-speed-993125 amorfo77/json:claude/todo-191-plan-508110 amorfo77/json:fix/json_pointer_create_object_5357 amorfo77/json:master
amorfo77/json:v3.12.0 amorfo77/json:v3.11.3 amorfo77/json:v3.11.2 amorfo77/json:v3.11.1 amorfo77/json:v3.11.0 amorfo77/json:v3.10.5 amorfo77/json:v3.10.4 amorfo77/json:v3.10.3 amorfo77/json:v3.10.2 amorfo77/json:v3.10.1 amorfo77/json:v3.10.0 amorfo77/json:v3.9.1 amorfo77/json:v3.9.0 amorfo77/json:v3.8.0 amorfo77/json:v3.7.3 amorfo77/json:v3.7.2 amorfo77/json:v3.7.1 amorfo77/json:v3.7.0 amorfo77/json:v3.6.1 amorfo77/json:3.6.1 amorfo77/json:v3.6.0 amorfo77/json:v3.5.0 amorfo77/json:v3.4.0 amorfo77/json:v3.3.0 amorfo77/json:3.3.0 amorfo77/json:v3.2.0 amorfo77/json:3.2.0 amorfo77/json:v3.1.2 amorfo77/json:3.1.2 amorfo77/json:v3.1.1 amorfo77/json:v3.1.0 amorfo77/json:3.1.0 amorfo77/json:v3.0.1 amorfo77/json:3.0.1 amorfo77/json:v3.0.0 amorfo77/json:3.0.0 amorfo77/json:v2.1.1 amorfo77/json:2.1.1 amorfo77/json:v2.1.0 amorfo77/json:2.1.0 amorfo77/json:v2.0.10 amorfo77/json:2.0.10 amorfo77/json:v2.0.9 amorfo77/json:2.0.9 amorfo77/json:v2.0.8 amorfo77/json:2.0.8 amorfo77/json:v2.0.7 amorfo77/json:v2.0.6 amorfo77/json:v2.0.5 amorfo77/json:v2.0.4 amorfo77/json:v2.0.3 amorfo77/json:v2.0.2 amorfo77/json:v2.0.1 amorfo77/json:v2.0.0 amorfo77/json:v1.1.0 amorfo77/json:v1.0.0 amorfo77/json:v1.0.0-rc1
..
pull from: amorfo77/json:claude/nlohmann-json-docs-b89a59
amorfo77/json:perf/view-compare amorfo77/json:gh-pages amorfo77/json:claude/nlohmann-json-docs-b89a59 amorfo77/json:claude/from-json-arith-array amorfo77/json:claude/implement-5891-deff16 amorfo77/json:docs/cross-reference-json-view-3-13-0 amorfo77/json:develop amorfo77/json:claude/fix-issue-3989-db7e45 amorfo77/json:pin-ci-dependencies amorfo77/json:claude/test-compilation-speed-993125 amorfo77/json:claude/todo-191-plan-508110 amorfo77/json:fix/json_pointer_create_object_5357 amorfo77/json:master
amorfo77/json:v3.12.0 amorfo77/json:v3.11.3 amorfo77/json:v3.11.2 amorfo77/json:v3.11.1 amorfo77/json:v3.11.0 amorfo77/json:v3.10.5 amorfo77/json:v3.10.4 amorfo77/json:v3.10.3 amorfo77/json:v3.10.2 amorfo77/json:v3.10.1 amorfo77/json:v3.10.0 amorfo77/json:v3.9.1 amorfo77/json:v3.9.0 amorfo77/json:v3.8.0 amorfo77/json:v3.7.3 amorfo77/json:v3.7.2 amorfo77/json:v3.7.1 amorfo77/json:v3.7.0 amorfo77/json:v3.6.1 amorfo77/json:3.6.1 amorfo77/json:v3.6.0 amorfo77/json:v3.5.0 amorfo77/json:v3.4.0 amorfo77/json:v3.3.0 amorfo77/json:3.3.0 amorfo77/json:v3.2.0 amorfo77/json:3.2.0 amorfo77/json:v3.1.2 amorfo77/json:3.1.2 amorfo77/json:v3.1.1 amorfo77/json:v3.1.0 amorfo77/json:3.1.0 amorfo77/json:v3.0.1 amorfo77/json:3.0.1 amorfo77/json:v3.0.0 amorfo77/json:3.0.0 amorfo77/json:v2.1.1 amorfo77/json:2.1.1 amorfo77/json:v2.1.0 amorfo77/json:2.1.0 amorfo77/json:v2.0.10 amorfo77/json:2.0.10 amorfo77/json:v2.0.9 amorfo77/json:2.0.9 amorfo77/json:v2.0.8 amorfo77/json:2.0.8 amorfo77/json:v2.0.7 amorfo77/json:v2.0.6 amorfo77/json:v2.0.5 amorfo77/json:v2.0.4 amorfo77/json:v2.0.3 amorfo77/json:v2.0.2 amorfo77/json:v2.0.1 amorfo77/json:v2.0.0 amorfo77/json:v1.1.0 amorfo77/json:v1.0.0 amorfo77/json:v1.0.0-rc1
33 Commits
claude/test-compilation-speed-993125 .. claude/nlohmann-json-docs-b89a59
Author SHA1 Message Date
Niels Lohmann 7b2a1b2de5 Docs: fill documentation gaps found in user questions 
Add pages on class-hierarchy/smart-pointer/variant/chrono recipes, working
without exceptions, and large, streamed and partial input. Extend the FAQ
(23 entries, Known bugs first), parse error documentation (empty-input
checklist, line/column, diagnostic positions), element access, conversions,
creating values, ADL troubleshooting, memory, and build/distribution notes.
Add 14 examples with generated output. Correct the statement that parse errors
cannot be recovered from with -fno-exceptions.

The topics come from recurring questions on Stack Overflow, Reddit, Hacker
News, YouTube, and the project's GitHub discussions and issues.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-10-24 19:13:49 +02:00
Niels Lohmann 01139dc192 Silence an Infer false positive in json_view materialize (#5905) 
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-10-24 17:10:31 +02:00
Niels Lohmann 9c50bdd140 Make the DOM parser cheaper per container and per member (#5882) 
* Reserve the parser's nesting stack once

The stack of open arrays and objects was a std::vector<std::uint8_t> that
allocated at the first push_back and then reallocated at 1, 2, 4, 8 bytes,
so a document nested four levels deep made four allocations before it
parsed a value. A document that starts with a container now reserves 32
levels in one allocation; deeper documents grow as before. The loop that
reads the tokens is the code it was, so large documents are not affected
(a stack object with its own storage, and a reserve at every container,
both cost the token loop instructions or cycles).

Fixed cost of parsing a 124-byte object (macOS instructions): accept
-7.6 %, DOM -2.5 %; no change for a single number.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Build arrays of the DOM parser in one exact-size allocation

Reading an array of unknown length appended its elements to a std::vector
one at a time, so a small array reallocated (and moved its elements) at
sizes 1, 2, 4, ... The DOM SAX parser now collects the elements of arrays
whose length is unknown on one stack shared by all open arrays and moves
them into an array of exactly the right size when the closing bracket
arrives. An array that grows past 16 elements switches to appending to its
own vector, so large arrays pay no second copy. Binary formats, which
announce the length, and builds with JSON_DIAGNOSTICS or
JSON_DIAGNOSTIC_POSITIONS (whose values need their final place from the
start) keep storing into the array directly.

The stacks of open containers are reserved on first use instead of growing
from one element.

The binary readers get a DOM parser of their own, json_sax_dom_binary_parser:
it is the plain parser that stores every value directly in its container
(the code the readers used before). Their containers announce their size, so
the bookkeeping of the collecting parser (frame entries, the destination of a
finished container) was pure overhead for them: with the collecting parser
FromBinaryShape nested was 3-12 % and containers up to 5 % more instructions
(GCC) than before. With the plain parser Linux Ir of FromBinaryShape and
FromBinaryBuffer is within 0.8 % of develop (Clang: identical).

canada.json has 56 000 two-element arrays that took three allocations each
and now take two. DOM parse Ir on Linux against develop (GCC / Clang): canada
-7.0 % / -7.7 %, citm -0.8 % / -1.6 %, 124-byte object -5.4 % / -5.5 %;
accept and SAX are untouched.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Reuse one value for the key events of the callback parser

parse() with a callback built a new string value for every key it passed to
the callback (a heap allocation and release per key), and copied the key a
second time into the object. The callback parser now keeps one string value
for key events, which it assigns, and moves the key into the object.

Callback parse, Linux Ir (GCC / Clang): twitter -13.3 % / -11.7 %, citm
-9.5 % / -7.6 %. (Earlier macOS measurement: twitter -19.5 % instructions.)

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Remember where the DOM parser stores the next value

The DOM SAX parser looked up the innermost open container for every value:
the end of its frame stack, then the object or array pointer in the frame.
Compilers have to reload all of that after each value is stored, since a
store of a value may alias the frame. The parser now keeps the innermost
container's kind and, for an array that stores directly, the array itself
in members, updated when a container opens or closes.

Flat arrays of numbers are the case that mattered: parsing floats.json
(1 000 000 doubles) into a DOM took 2 % more cycles than before the
previous commit on macOS and now takes 0.4 % fewer than the code before it
(signed_ints.json -0.8 %). Linux Ir is unchanged (canada -6.1 % GCC /
-6.9 % Clang, citm -0.3 % / -0.8 %, twitter +0.2 % / 0.0 %).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Look up the keys of big ordered_json objects in a hash table

ordered_map keeps its elements in a vector and searches it linearly for
every key it adds, so parsing an object with n keys into ordered_json took
O(n^2) key comparisons: 33 billion instructions for one object with 30 000
keys, where json needs 245 million. From 64 keys on, the DOM parser now
keeps a hash table of the keys of the open object (hashes and positions
only; the elements stay in the vector) and finds duplicate keys there. The
result is the same, including the position of a key that occurs twice.
Smaller objects, std::map based objects and key types without std::hash
are untouched.

ordered_json parse, macOS instructions: object with 1 000 keys 64.8 M ->
14.6 M, 10 000 keys 5 104 M -> 38.7 M, 30 000 keys 33 358 M -> 89 M.
Linux Ir of an object with 2 000 keys: 43.0 M -> 1.9 M (GCC), 37.9 M ->
1.8 M (Clang).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Trim the work the DOM parser does per container and per member

Opening and closing an array or object cost the DOM parser more than before
the exact-size array change, which showed on documents of many small
objects (jeopardy.json: +1.4 % Ir with GCC). This takes it back:

- the stack of open containers keeps its first eight entries in the parser
  object instead of in a vector, so it needs no capacity check beyond the
  push itself and no allocation for typical documents;
- a container is created in the place it ends up in, instead of as a
  temporary that is then moved there;
- a new object member, which is null, is constructed over the null instead
  of being swapped with it (only with the default base class of basic_json);
- the key index that big ordered_json objects use is only a part of the
  stack entries of objects that can have one;
- an array that outgrows the stack reserves twice what it has so far, as a
  vector that grew by itself would have.

DOM parse Ir of the series up to here against develop, Linux GCC / Clang:
canada -6.6 % / -7.3 %, citm +1.2 % / +0.2 %, jeopardy +0.4 % / -0.1 %,
124-byte object -6.5 % / -6.2 %. (twitter measures +6.8 % / +6.6 % at this
point in steady state, almost all of it in glibc's malloc/consolidate for the
new allocation sizes; with the last commit of the series it is -0.1 % / 0.0 %.)

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Test parsing containers of the sizes where the DOM parser changes storage

The DOM parser collects the elements of an array on a stack (up to 16) and
the members of an ordered_json object (up to 32) before it moves them into
the container, and uses a hash table for keys from 64 members on. Parse
arrays and objects of the sizes around these limits, nested into each other,
with repeated keys, and as random trees that are built next to their text,
and compare json and ordered_json against the value built through the API.

The tests pass on the code from before these changes as well. The random
trees are what finds the mistakes that are easy to make here: while the
collecting change was being written, a collected array that outgrew the
stack at the moment its parent had just reached the limit left its elements
in the parent, and only these tests (142 failed assertions) showed it.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Do not convert floats that cannot overflow in accept()

accept() only has to find out whether a float token is finite: json_sax_acceptor
ignores the value, and the parser rejects only a float that overflows. A
float whose digits fit into 19 digits and whose decimal exponent is at most
max_exponent10 - 19 is below 10^max_exponent10 and cannot overflow, so the
lexer, told by accept() that the value is not needed, skips the conversion
for those. A token that may overflow, with more digits, or in another float
format is converted as before, so 1e309 and its neighbours are rejected as
they were (the new test compares accept() with parse() on tokens around the
limit).

json::accept(): canada.json Ir -8.6 % (GCC) / -10.9 % (Clang), 124-byte
object -3.2 %; cycles on the M1 (release benchmarks, LTO): canada -7.8 %,
floats.json -9.0 %. The check tests the accept() flag first, so the parser
that needs the value pays one load and branch: SaxParse canada +0.35 % Ir
(GCC and Clang), DOM canada +0.4 %.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Collect the members of ordered_json objects on a stack until they are known

An ordered_json object keeps its members in a vector and appends every new
member to it, so reading an object of unknown size reallocated the vector at
1, 2, 4, 8, ... members, and ordered_map copies all keys on every
reallocation. The DOM SAX parser now collects the members of such objects on
one stack shared by all open objects, as it does the elements of arrays, and
moves them into a vector of exactly the right size when the closing brace
arrives. An object that gets past 32 members switches to appending to its
own vector (and, from 64, to the key index of the earlier commit). The
members are the same, in the same order, and a key that occurs twice keeps
the place of its first occurrence. std::map based objects, objects whose
size the input announces (binary formats), and builds with JSON_DIAGNOSTICS
or JSON_DIAGNOSTIC_POSITIONS are untouched.

What only ordered objects need (the collected members, the key indexes) is
a member of the parser only for them, and the code that stores a value tries
the array that takes its elements directly first. An array that outgrows
the stack now reserves 32 elements, the capacity a vector that grew by
itself has at that point; with the larger reserve before, parsing
floats.json (1 000 000 numbers) was 2 % slower on Apple M1 than before this
series.

ordered_json parse, Linux Ir against develop (GCC / Clang): citm -11.6 % /
-11.7 %, twitter -8.4 % / -8.4 %, jeopardy (12 000 objects of 7 keys)
-10.5 % / -8.0 %, canada -3.9 % / -5.2 %, 124-byte object -20.9 % / -16.5 %.
macOS time (M1, earlier base): citm -19 %, twitter -13 %, jeopardy -13 %,
canada -14 %, 124-byte object -26 %. json parse: steady-state twitter -0.1 %
(see the previous commits), citm -1.2 %.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Rename the key index's match callback so Flawfinder doesn't read it as std::equal

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Bind the parser test's strings by reference in the range-for loop

GCC and Clang 14 (-Wrange-loop-construct) reported the copy.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Collect container values only in basic_json's own parse functions

A json_sax_dom_parser that a user constructs (as in the sax_no_exception
example) must hold the values read before a parse error, as on develop:
its value is used after the error. The values of containers of unknown
size are now collected on a stack only with the new template parameter
Collect (default false), which json::parse() and the json_view
materialization set; without it, every value is stored in place as it
arrives. A test checks the values kept after an error for json and
ordered_json.

Also fixes the clang-tidy, cpplint and Infer findings: the frame stack
keeps its first frames in a std::array (left uninitialized) and the
others in a std::unique_ptr, auto for create<>(), NOLINT for member
initializers -Weffc++ needs, and an @infer-ignore for the nullptr that
place_container() does not use.

json::parse() and ordered_json::parse() instructions are unchanged
(Apple clang; twitter, citm, canada, jeopardy: -0.03 % to +0.25 %).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Clarify the DOM parser's collection stacks and rename reenter_enclosing_container()

Address review comments: explain why a closed container erases only the top
of the shared stack, set the payload before the type, rename the helper that
makes the enclosing container current again, and declare the parser's state
stack right before reserving it.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Export json_sax_dom_parser from the C++20 module for MSVC

MSVC needs the detail classes that basic_json befriends to be visible in
the module (#3970, as json_sax_dom_callback_parser); with its new
template parameter, json_sax_dom_parser failed in ci_module_cpp20
(C2039, not a member of nlohmann::detail).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Keep the overflow-limit numbers of the parser test in a std::vector

clang-tidy (cppcoreguidelines-avoid-c-arrays) rejected the C array.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Fix the clang-tidy findings of the ordered_json parser tests

The container sizes are a std::array, the expected text uses raw string
literals, the generator's arithmetic is in parentheses, auto for the
result of make<>(), and single characters are appended as characters.
clang-tidy now reports nothing on the lines the branch adds to
unit-ordered_json.cpp and unit-class_parser.cpp (apart from checks CI's
clang-tidy does not have).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-10-24 07:41:21 +02:00
Niels Lohmann dd7686a269 json_view: faster reading, indexing and C-stream parsing (#5888) 
* Read C streams into a json_document in blocks

json_document::parse(FILE*) (and read(), accept(), parse_copy()) went through the
library's file input adapter and appended the text character by character
with fgetc. Read it with fread in blocks instead, like the std::istream
overload already does.

Parsing citm_catalog.json (1.7 MB) from a FILE*:
  Apple M1: 80.6 M -> 8.4 M instructions, 4.8 ms -> 0.55 ms (parse of the same text from a std::string: 8.7 M)
  Linux x86-64 Ir (whole run), GCC 13 / Clang 18: 47.0 M -> 13.7 M / 45.8 M -> 14.2 M
The parsed documents and the error cases are identical (checked over the test corpus
and 30 mutants of each file). A null FILE* is handled as before (by the library's adapter).

Belongs to the PR "json_view: read-side speedups".

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Index arrays without subtrees directly

operator[](idx), at(idx) and back() skipped the elements of an array one
after the other, whatever they were. In an array none of whose elements is
an array or object (numbers, strings, booleans, null) the elements are
adjacent nodes, which the array node's own sizes tell: its subtree has
exactly one node more than it has elements. Such an element is now the
node idx after the array node; any other array is walked as before.

Reading r[i] for every i of an array of 3000 integers:
  Apple M1: 28.8 M instructions, 3.1 ms -> 0.1 M instructions, 4 us
  Linux x86-64 Ir (whole run), GCC 13 / Clang 18: 84.1 M -> 3.1 M / 75.0 M -> 3.0 M
Arrays that contain arrays are unchanged.

Belongs to the PR "json_view: read-side speedups".

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Skip the bounds checks of number and string tokens in NUL-terminated input

For a NUL-terminated input (std::string, C string, string literal), the
parser reads the terminator instead of comparing p with the end, via
cursor::cur(). Number and string scanning still tested `p != e` before each
look at the next byte (after the integer digits for '.', for 'e', after
the exponent sign and digits, and after a string for its closing quote),
although the terminator can never be one of those characters. They now use
cur(), which is the plain load for NUL-terminated input and the bounds check
for ptr+len input, as before.

Parsing, Apple M1 instructions per parse: twitter 3.126 M -> 3.089 M (-1.2 %),
citm_catalog 8.70 M -> 8.58 M (-1.3 %), jeopardy 306.1 M -> 302.5 M (-1.2 %), signed
integers 281 M -> 266 M (-5.4 %), canada and floats -0.1 % / -0.9 %.
Linux x86-64 Ir, GCC 13 / Clang 18: twitter -0.5 % / -0.0 %, signed integers +0.2 % / -0.7 %.

Belongs to the PR "json_view: read-side speedups".

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Make the index text of items() on arrays lazy

basic_json_view::items() on an array formatted every element's index with
std::to_string when the item was made, whether or not the loop asks for the
key. The item now keeps the position and makes the text on the first key()
call. Objects are not affected.

Iterating over the items of an array of 3000 numbers and reading only the values:
  Apple M1: 25.5 us -> 4.6 us (0.45 M -> ~0 net instructions)
  Linux x86-64 Ir (whole run), GCC 13 / Clang 18: 14.9 M -> 4.1 M / 15.3 M -> 4.6 M
Reading key() costs what it did (24.5 us).

Belongs to the PR "json_view: read-side speedups".

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Make the default of value() with a C string only if the member is missing

basic_json_view::value(key, "text") and value(pointer, "text") built a
std::string from the default first and passed it on, so a call allocated
for the default even when the member exists, and copied it again when it
does not. They now look the member up and make the result directly: from the
member if there is one, else from the C string. Behavior, including the
exceptions, is the same.

Two calls (member found, member missing) with a 40 character default, Apple M1:
  1.6 K -> 0.7 K instructions
  Linux x86-64 Ir (500 rounds), GCC 13 / Clang 18: 2.46 M -> 2.22 M / 2.40 M -> 2.15 M

Belongs to the PR "json_view: read-side speedups".

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Materialize scalars directly and keep the stack of small subtrees local

basic_json_view::materialize() (and get<BasicJsonType>()) always set up the
SAX handler that parse() uses and a std::vector for the open containers,
sized from the document's deepest nesting. For a scalar, which is most of what
is materialized one value at a time, neither is needed: the value is made
directly (the same constructors the handler ends up calling). For arrays and
objects the stack of open containers is now a local array for the first 16
levels, and the vector is only used for deeper documents.

Materializing five scalars of a small document, Apple M1: 3.6 K -> 0.9 K instructions
  Linux x86-64 Ir (500 rounds), GCC 13 / Clang 18: 2.96 M -> 2.37 M / 2.90 M -> 2.29 M

Belongs to the PR "json_view: read-side speedups".

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Copy the text between escapes while scanning it

A string that contains an escape is decoded by builder::slow_string(). After
each escape it scanned the next run of plain bytes (8 or 16 byte-by-byte
steps, then a vector loop), and then copied that run to the decode arena in a
second pass. The run is now copied 16 bytes at a time while it is scanned
(one vector load, a store of the same 16 bytes, and the mask of the bytes
that end the run), and the write position lives in a local variable, not in
the builder, since a store of a character may alias the builder's members and
made the compiler reload it after every escape. A run that ends at a quote
needs no scan, and non-ASCII text goes through the UTF-8 check as before;
the portable code is unchanged.

Parsing, Apple M1 instructions: jeopardy.json 304.0 M -> 293.8 M (-3.4 %), 3000 strings with
escapes 1.72 M -> 1.54 M (-10 %), twitter -0.6 %.
Linux x86-64 Ir, GCC 13 / Clang 18: escaped strings -7.6 % / -11.4 %, twitter 0.0 % / 0.0 %.
Documents and error messages are identical on 900 k random strings of escapes, UTF-8 and control
characters (also under ASan), and on mutants of the test corpus.

Belongs to the PR "json_view: read-side speedups".

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Hash the last bytes of a key of a large object without a loop

key_hash() mixed a key eight bytes at a time and then packed the remaining
0-7 bytes one by one, in a loop. The last word of a key of eight or more
bytes is now read as the last eight bytes of the key (overlapping the word
before), and a key of 2-7 bytes as two overlapping words of four or two
bytes. A product only moves bits upwards, and the slot is taken from the low
bits: the high bits of the last word are folded down before the
multiplication, since keys that differ only in their last characters ("user_17",
"user_18") would otherwise share slots. The hash is not stored, so its values
may change.

Looking up every key of an object of 1000 random keys, Apple M1 instructions per 1000 lookups:
  keys of 5 bytes 175 K -> 93 K, hex strings of 8 bytes 177 K -> 168 K, keys of 12 bytes 196 K -> 186 K
Linux x86-64 Ir (whole run), GCC 13 / Clang 18: 5-byte keys -4.7 % / -8.8 %, 12-byte keys -3.3 % / -7.4 %.
Building the tables while parsing such objects is cheaper as well (5-byte keys: 13.3 us -> 10.1 us).

Belongs to the PR "json_view: read-side speedups".

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Compare short keys of a large object without a call of memcmp

find_indexed() compared the key of a member the table points to with
memcmp, a library call for any length. Up to 16 bytes, the bytes are now
compared as two overlapping words (the first and the last 8, 4, or 2 bytes),
as find_member() already does for the objects without a table; longer keys
still use memcmp.

Looking up every key of an object of 1000 random keys, Apple M1 instructions per 1000 lookups
(with the previous commit): keys of 5 bytes 93 K -> 72 K, hex strings of 8 bytes 168 K -> 148 K,
keys of 12 bytes 186 K -> 170 K, keys of 24 bytes 198 K -> 193 K (unchanged, memcmp).
Linux x86-64 Ir (whole run), GCC 13 / Clang 18: 5-byte keys -7.2 % / -7.3 %, 12-byte keys -5.9 % / -6.6 %.

Belongs to the PR "json_view: read-side speedups".

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Read an unedited editable document like a read-only one

Views of a json_editable_document followed edits at every step: the
lookup, the element access, back(), dump() and materialize() were built
for nodes that can be moved or linked, though no edit changes the index until
the first one (every edit creates the edit state before it touches a node).
Until then these now take the read-only code: one check of the document's
edit state per call, not per member or per node. After the first edit
nothing changes. Views that were made earlier keep working, as the check is
made when they are used.

Unedited editable document, Apple M1: dump() of twitter.json 1.40 M -> 1.22 M instructions (-12.6 %);
lookups unchanged. Linux x86-64 Ir, GCC 13 / Clang 18: dump -1.7 % / -1.0 %, lookups -0.1 % / 0.0 %.
Walking with iterators is unchanged, and so is dump() after an edit.

Belongs to the PR "json_view: read-side speedups".

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Format the numbered test keys of json_view's index with a stream

The keys were made with snprintf() and a format from a list, which
Flawfinder reports as an error (CWE-134, a format string that is not a
literal). A stream now formats them; the 30000 keys of the test are the
same as before.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Put the key number of the json_view index test in parentheses

astyle formatted i * step after << as a pointer declaration (i* step),
which failed the amalgamation check.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Fix the compiler and lint findings of json_view's read side and tests

- find() and contains() dispatch on Editable for the member lookup; for
  documents that cannot be edited both branches were the same call
  (GCC -Wduplicated-branches)
- json_view.hpp includes <cstdio> for the FILE overload (cpplint)
- the arena check macro of the builder is on one line (cpplint read it
  as an if body without braces)
- tests: no nested conditional (also MSVC C4018), no useless cast,
  parentheses around mixed operators, raw string literals, += instead
  of temporary concatenations, and NOLINT for the fixed seeds and a
  find() != npos, as elsewhere in the tests

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Grow the decode arena through a function and store its position back on every return

slow_string() kept its write position in locals and grew the arena through
the macro NLOHMANN_VIEW_ENSURE. The macro is now the always-inline member
arena_ensure() (a lambda is not inlined at every call site by clang, which
then keeps the position in memory: +11 % instructions on escape-heavy
strings), and a guard stores the position back into m_arena_write on every
return instead of by hand on the successful one. Instruction counts are
unchanged (clang) or 1 % lower (GCC).

Also word the complexity of back() as that of at().

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-10-24 07:41:06 +02:00
Niels Lohmann e9c4fe3cdf json_view: indented and ensure_ascii dumps through the fast writer (#5889) 
* Write the indented output of json_view's dump() with the fast writer

The compact writer of dump() (cursor in a register, fixed-size copies of
tokens) served dump(-1) only; every dump(indent), and everything with
ensure_ascii that was indented, went through a generic walk that calls put()
for each token and put_repeated() (a memset call) for each line break. The
fast writer now writes indented output, too, with or without ensure_ascii, and
the generic walk is gone. A line break is a copy of a 64-byte ruler (a line
break and 63 indentation characters) when the line is at most 64 bytes, and a
store plus memset beyond that. The depth is counted only in the indented
instantiations (counting it in the compact one cost an editable document 5 %
with Clang). The indented walks are out of line (dump_pretty()): inlined
into dump() next to the compact ones, they made compact dumps of integer
arrays 7 % slower.

A decoded string (one with escape sequences) of a damaged image loaded with
image_check::bounds can contain invalid UTF-8. As before, the compact dump
writes such bytes unchanged, and the indented dump throws type_error.316; its
runs are found by the lexer's validating scan (a scalar validation of every
sequence made dump(2) of decoded CJK text twice as slow as before).

dump(2) instructions per call, macOS arm64 (before -> after):

               twitter         citm            canada          jeopardy(4k)
  json         3.75M -> 1.85M  8.28M -> 3.93M  67.1M -> 56.0M  9.56M -> 4.81M
  editable     3.82M -> 1.84M  8.48M -> 4.02M  68.1M -> 56.2M  9.66M -> 4.75M

Linux instructions (GCC 13 / Clang 18), dump(2) and dump(2, ' ', true):

  twitter      -25 % / -30 %      -11 % / -11 %
  citm         -29 % / -28 %      -24 % / -24 %
  jeopardy(4k) -25 % / -32 %      -20 % / -24 %
  CJK text with \u escapes  -6 % / -11 %   -2 % / -2 %

The compact dump is unchanged (within +0.4 % on Linux, within noise on the Mac).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Scan the strings of json_view's dump() with ensure_ascii 16 bytes at a time

With ensure_ascii, a string of the source that has a byte >= 0x7F was written
by a generic path: put() per piece and a validate-and-decode step for every
character beyond ASCII. It now goes through a writer with the cursor in a
local variable. The part of the string that has nothing to escape is found
16 bytes at a time (SSE2 or NEON; the existing 8-byte scan without vector
support) and copied; strings that are ASCII throughout are still decided by
the 8-byte check of dump() with ensure_ascii (a 16-byte scan for them was
2-5 % slower for short strings). The two-byte sequences (lead C2..DF) and
the three-byte sequences whose continuation bytes may have any value (lead
E1..EF except ED) are decoded with a few compares; E0, ED, four-byte
sequences, and damaged input go through validate_one_utf8() as before, so
damaged images still throw type_error.316.

dump(-1, ' ', true) instructions per call, macOS arm64 (develop -> now);
the compact dump without ensure_ascii and dumps of ASCII text are unchanged:

  twitter 5.42M -> 3.57M    citm 3.28M -> 3.24M     jeopardy(4k) 5.78M -> 5.65M
  CJK text, raw 66.5M -> 35.3M    escaped \u 32.7M -> 17.0M
  Latin text with accents 11.5M -> 9.4M    ASCII text 4.40M -> 4.40M

Linux instructions (GCC 13 / Clang 18), dump(-1, ' ', true):

  twitter -9 % / -33 %    CJK text -14 % / -47 %    Latin text -8 % / -18 %
  citm -0.2 % / -0.4 %    ASCII text -0.1 % / -0.3 %

The compact dump without ensure_ascii is unchanged (canada 0.0 %).

The unit test covers every character of the Basic Multilingual Plane, and a
character that needs an escape at each position of strings of 0..40 bytes.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Find the end of the plain run of a decoded string 16 bytes at a time with SSE2

The strings of a document that had escape sequences are written in runs
without a quote, backslash, or control character; the compact dump found the
end of a run 8 bytes at a time with a SWAR test. With SSE2 (x86-64) it now
compares 16 bytes per step. NEON keeps the SWAR code: there the mask of a
step costs more than it saves on the short runs that are typical here.

Instructions of dump(), Linux (GCC 13 / Clang 18):

  twitter.json                  1.99M -> 1.88M (-5.2 %) / 1.97M -> 1.92M (-3.0 %)
  CJK text with \u escapes      5.98M -> 4.52M (-24.5 %) / 5.30M -> 4.25M (-19.9 %)
  ASCII text with a few escapes 4.28M -> 4.09M (-4.4 %) / 4.24M -> 4.11M (-3.2 %)
  jeopardy (4,000 questions)    5.08M -> 4.78M (-5.8 %) / 4.97M -> 4.80M (-3.4 %)

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Rename depth_counter's open() and close() to enter() and leave()

Flawfinder took them for the file functions open() and close().

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Fix the clang-tidy findings in json_view's dump and its tests

- the SSE2 loop of the string scan is within NOLINTBEGIN/END
  (portability-simd-intrinsics), as in to_chars.hpp
- depth_counter<false>'s leave() and get() keep the interface of
  depth_counter<true> instead of being static
- write_ascii() compares the room left as a size_t variable
- the test's texts are a std::array, and a JSON fragment with escapes
  is a raw string literal

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Move dump_ascii() and dump_pretty() next to dump()

Address a review comment: the two dispatchers that dump() calls now follow
it directly.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-10-24 07:38:17 +02:00
dependabot[bot] 1e9222f955 Bump dompurify from 3.4.16 to 3.4.17 in /docs/mkdocs/scripts/mermaid (#5904) 
Bumps [dompurify](https://github.com/cure53/DOMPurify) from 3.4.16 to 3.4.17.
- [Release notes](https://github.com/cure53/DOMPurify/releases)
- [Commits](https://github.com/cure53/DOMPurify/compare/3.4.16...3.4.17)

---
updated-dependencies:
- dependency-name: dompurify
  dependency-version: 3.4.17
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-10-24 07:37:42 +02:00
Niels Lohmann ffcd392be0 Fix Semgrep findings in Python tooling (#5899) 
- generate_docset.py: use SHA-256 instead of SHA-1 for the cache file
  name digest of downloaded images.
- generate_docset.py: suppress dynamic-urllib-use-detected; the URL
  scheme is restricted to http(s) right before the call.
- generate_natvis.py: suppress direct-use-of-jinja2; the template
  renders a local XML file with autoescape enabled.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-10-23 07:11:17 +02:00
dependabot[bot] e28506dd86 Bump jsdom from 30.1.2 to 30.1.3 in /docs/mkdocs/scripts/mermaid (#5898) 
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-10-23 06:54:23 +02:00
dependabot[bot] 03a0595c1e Bump actions/upload-artifact from 7.0.1 to 7.0.2 (#5897) 
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-10-23 06:54:07 +02:00
dependabot[bot] 4e72af9d54 Bump step-security/harden-runner from 2.21.1 to 2.22.1 (#5896) 
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-10-23 06:53:50 +02:00
dependabot[bot] 0f5d07310c Bump the codeql-action group with 4 updates (#5895) 
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-10-23 06:53:32 +02:00
Niels Lohmann 0b924f5375 Split and print JSON pointers in one pass (#5887) 
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-10-23 06:53:15 +02:00
Niels Lohmann 6fa14d8f95 Write binary formats through reserved sink space and number runs (#5885) 
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-10-23 06:52:58 +02:00
Niels Lohmann dec0e9b749 Write strings, scalars and indentation in dump() with fewer calls (#5883) 
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-10-23 06:52:41 +02:00
Niels Lohmann 86d5c56c54 Speed up flatten(), unflatten(), diff() and patch() further (#5886) 
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-10-23 06:48:12 +02:00
Niels Lohmann aa0b371f7c Faster std::istream input: inline get-area path, binary reads, unbuffered streambufs (#5884) 
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-10-23 06:48:00 +02:00
Niels Lohmann ace1030107 Fix develop CI: bit_cast ADL ambiguity and artifact-less comment runs (#5894) 
- unit-stl_helpers.cpp: with a std::array argument, an unqualified
  bit_cast<...>(bytes) also finds std::bit_cast via ADL in C++20 and
  MSVC 2019 rejects the call as ambiguous (AppVeyor). Qualify the call.
- comment_check_amalgamation.yml: skip, rather than crash, when the
  triggering run has no "pr" artifact (stale queued runs that complete
  as failed without ever running a job).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-10-22 20:22:53 +02:00
Niels Lohmann 2dc1e54856 Compare json_view with yyjson, simdjson, and Boost.JSON (#5627) 
* Compare json_view with yyjson, simdjson, and Boost.JSON

tests/benchmarks/json_view/ holds a comparison with other
libraries, answering the question users ask when they pick one.
It is not built by CMake and not run by CI.

- bench_view.cpp: parse, traverse, select and dump of twitter,
  citm_catalog, canada, jeopardy, a tweet and an RPC request,
  against json_view, yyjson, simdjson, Boost.JSON and json::parse,
  all agreeing and run interleaved.
- bench_corpus.cpp: parse, traverse and dump of any file list.
- bench_edit.cpp: parses, edits through each library's own API,
  and serializes; all outputs must describe the same value.
- compare.py: builds both programs with system or pinned,
  SHA-256-checked downloads, and writes results with the commit,
  CPU, OS, compiler and library versions.
- README.md and a workflow that runs on demand or via a PR label.

Fairness fixes folded in: each engine runs once untimed before
each timed round, so the next one no longer pays for the
previous one's cleanup; dump is also compared with source numbers
against yyjson's raw-number mode; a "simdjson DOM (fresh)" column
and a reused-document column for json_view were added;
compare.py's paths are absolute; downloads use a .part file and a
failed checksum removes the archive.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Make the downloads of json_view's compare.py robust

Hash the archives in chunks instead of reading them into memory, download
with a timeout, extract into a temporary directory that is renamed into
place only after success (a half-extracted directory was trusted forever),
and split CXX and CC into arguments so that values like 'ccache g++' work.
Note at the pins that a SHA-256 mismatch of a GitHub tag archive means
that GitHub regenerated it.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Add a timeout and a concurrency group to the json_view benchmarks

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Check the CSV file and the round count in the json_view benchmarks

fopen() of the CSV output was not checked, so an unwritable directory made
fprintf() write to a null FILE*; the round count is parsed with strtol and
clamped instead of atoi.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-10-22 08:40:33 +02:00
Niels Lohmann 9855db81f3 Silence macro-to-enum for the endianness macros (#5881) 
#5876 added JSON_LITTLE_ENDIAN_TARGET and JSON_BIG_ENDIAN_TARGET to
macro_scope.hpp. They must stay macros because they select code in #if,
so suppress the clang-tidy check as simd_scan.hpp already does.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-10-22 08:40:05 +02:00
Niels Lohmann 533cbfee51 Let generate_natvis.py cover custom ABI tag combinations (#5880) 
Since #5877, the checked-in nlohmann_json.natvis only covers namespaces
with up to two ABI tags. The generator gets options to produce a file
for other configurations:

- --abi-tags TAGS adds the namespaces of one configuration (repeatable)
- --all generates every combination of ABI tags
- --list-tags prints the tags with the macros that enable them
- --output FILE writes somewhere else than the repository root

`make natvis NATVIS_ARGS="..."` passes the options through. The default
output is unchanged. The options are documented in the generator's
README, the debugging page, and FILES.md, and checked in CI.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-10-21 22:57:54 +02:00
Niels Lohmann 9bf531436f Speed up flatten(), unflatten(), diff() and patch() (#5861) 
* Flatten into a std::map by inserting next to the previous entry

flatten() added every path with operator[], so each insertion into the
std::map of the result searched the tree from the root and compared long
shared path prefixes, which took half the time on canada.json. Paths come
out in nearly sorted order, so they are now inserted with emplace_hint
next to the entry added last, for object types that have it. ordered_map
keeps its append without lookup, and other object types keep
operator[].

Path segments are also appended in place, without a temporary string
for each escaped key or array index.

flatten() of canada.json, citm_catalog.json and twitter.json is 1.4x to
1.9x as fast; ordered_json is unchanged (1.05x). The output is
unchanged.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Unflatten without splitting keys or walking each path from the root

unflatten() split every key into a vector of reference tokens, stored the
pointer prefixes in a tree to decide which values become arrays, and then
looked up every token of every key from the root again.

Keys are now checked by a scan that raises the same parse_error.107/108 as
split(), skipping the part a key shares with the previous one, which was
checked already. The prefixes that denote arrays are kept in a small hash
set of key slices; the array/object decision is unchanged. The walk keeps
the values along the previous key's path, so only the part of a key that
differs from the previous one is looked up. Keys of a flattened object are
usually sorted, so that part is short.

unflatten() of canada.json, citm_catalog.json and twitter.json is 1.9x to
2.3x as fast; the output and the exceptions are unchanged.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Apply plain JSON Patch operations without building a json_pointer

patch() built a json_pointer for every operation, which splits the path
into a vector of strings, and for "add" and "remove" a second one for
the parent. That took 20% to 45% of the time not spent copying the
document.

"add", "remove", "replace" and "test" with a path that has no escapes and
denotes existing values (the parent, for "add"), as diff() creates them,
are now applied by resolving the path in place, reusing one string for
the reference tokens. Every other operation, and every operation that
would fail, takes the existing code, so the results, the exceptions and
the state left after a failing operation are unchanged.

patch() of the diffs of canada.json, citm_catalog.json and twitter.json
is 1.02x to 1.10x as fast; the rest is the copy of the document.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Build diff operations member by member

diff() created every operation from an initializer list, so each
{"op", "replace"} pair became a temporary array with two heap strings
before it was turned into an object member; an operation took about 25
allocations. The operations are now built as objects whose members are
assigned one by one. Paths of members and elements are also built in
place, without a temporary string for each escaped key or array index.

diff() of canada.json, citm_catalog.json and twitter.json, with every
eighth scalar changed, is 1.3x to 1.6x as fast; the output is unchanged.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Diff objects with the same keys in lockstep and keep one path buffer

diff() looked up every key of the source object in the target object,
and built a new string for the path of every value it descended into.
Objects of the same size usually have the same keys in the same order,
so they are now walked together as long as that holds, without a
lookup; objects that differ fall back to the lookup from the first key
that does not match. The path is one buffer that is extended before
descending and shortened again afterwards. Objects whose members are
equal but in a different order (possible with ordered_json) still get
no operations.

Together with the previous commit, diff() of canada.json,
citm_catalog.json and twitter.json is 1.5x to 2.0x as fast; ordered_json
1.65x to 1.85x. The output is unchanged.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Use std::array for the index buffer in append_index()

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Declare the partner iterator in the loop initializer in diff_object_keys()

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Fix GCC and clang-tidy warnings in the JSON pointer speedups

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-10-21 21:27:33 +02:00
Niels Lohmann 3e031b06a5 Use standard library helpers where available, with C++11 fallbacks (#5876) 
* Use standard library helpers where available, with C++11 fallbacks

Add detail helpers that call the standard library function when the
library provides it (checked with the __cpp_lib_* feature-test macros, so
no new ABI-tag macros) and a C++11 implementation otherwise:

- detail::clamp (std::clamp): by value, so that static constexpr members
  need no out-of-line definition before C++17; used in json_view's dump()
  buffer sizing, where step() can now be a static constexpr member.
- detail::bit_cast (std::bit_cast): replaces dtoa_impl::reinterpret_bits
  and the memcpy bit copies in the number parser, to_chars, the json_view
  serializer, and binary_writer::to_char_type.
- detail::byteswap (std::byteswap): one implementation instead of the
  binary writer's own byte_swap() overloads and the bswap chain in
  to_chars' store_msb_first().
- count_leading_zeros/count_trailing_zeros: two 32-bit _BitScanReverse/
  _BitScanForward calls for MSVC x86/ARM32 instead of the loop, and
  std::countl_zero/countr_zero for compilers without builtins.
- conjunction, disjunction, negation, bool_constant, and uncvref_t map to
  their std counterparts in C++17/C++20.

Builtins and intrinsics stay ahead of the <bit> functions, as MSVC's
std::countl_zero checks for LZCNT at run time unless AVX2 is enabled.

The generated code is unchanged with Clang (C++11/20/23, arm64 and
x86-64) and GCC 16 for parsing, dumping, and the binary formats. The
binary reader keeps its own byte_swap() and the writer its memcpy calls:
the smaller code from detail::byteswap/std::bit_cast there made GCC 16
inline get_number()/write_number() differently, which slowed reading by
10-18 % and writing by about 1 %.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Name the target byte order once (JSON_LITTLE_ENDIAN_TARGET, JSON_BIG_ENDIAN_TARGET)

to_chars.hpp and bit_ops.hpp repeated the long __BYTE_ORDER__/_MSC_VER
conditions; macro_scope.hpp now defines them once (undefined again in
macro_unscope.hpp).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Explain why the serializer growth bound is not a clamp

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-10-21 20:14:59 +02:00
Niels Lohmann 7c26e99ac5 Copy ASCII strings in the view's dump with ensure_ascii (#5879) 
With ensure_ascii, the view's dump wrote every string through the escaping
path, which decodes UTF-8 byte by byte, even strings of the source that are
plain ASCII. A string of the source without escape sequences has no quote,
backslash, or control character, so ensure_ascii leaves it unchanged
exactly when it has no byte >= 0x7F. The dump now checks that eight bytes
at a time (one add and one or per word) and copies such strings like the
dump without ensure_ascii.

The compact walk (dump_compact) now also serves ensure_ascii, through a
separate out-of-line instantiation, so the walk without ensure_ascii keeps
its speed.

Parsing is unchanged. A flag set by the parser would let the dump skip the
check and was about twice as fast for compact output, but it costs every
parse of string-heavy input 1-2% with GCC.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-10-21 20:01:38 +02:00
Niels Lohmann 6aefd8c662 Copy short printable-ASCII strings without starting the scanner (#5878) 
dump() now checks strings of up to 32 bytes against a table of the bytes
that can be written verbatim, four bytes per step, and copies them in one
go if all of them can. That is cheaper than setting up the vector scanner,
whose setup costs more than such a string takes to scan; SQLite's
jsonAppendString() writes strings with the same kind of table loop. Most
object keys and many values are this short (citm_catalog averages 8 bytes
per string, jeopardy 14).

GCC 16 and 13 stopped inlining put_indent() and put_literal() into the
pretty-printing loops once dump_escaped() had grown, which made indented
output up to 13% slower; both are now always inlined.

The new test puts every ASCII character and a 2-byte character at every
position of strings of 1 to 40 bytes and compares the output with the
reference implementation of the existing UTF-8 tests.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-10-21 20:01:28 +02:00
Niels Lohmann 52fecc1fab Move object keys into the value when parsing (#5855) 
json_sax_dom_parser::key() copied the key from the lexer's buffer into the
object. The lexer starts every token with an empty buffer (reset()), so the
key can be moved instead, which saves an allocation and a copy for every key
that does not fit into the string's inline buffer. json_sax_dom_parser
already moved string values the same way. ordered_map has no rvalue
operator[], so ordered_json still copies keys.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-10-21 20:00:51 +02:00
Niels Lohmann fa37b952df Match literals without a stack array and keep parser states in bytes (#5854) 
* Match literals without a stack array and keep parser states in bytes

scan_after_whitespace() built the expected text of true, false, and null in
local std::array objects. With -fstack-protector-strong (the default of
Ubuntu's GCC, among others), a local array gives the function a stack canary
check, which then ran on every token. scan_literal() now takes the text
after the first character as a string literal, with its length as a template
parameter; for contiguous input it compares the rest at once and falls back
to the byte loop on a mismatch, so errors are reported at the same
character. It is out of line, which keeps the token dispatch small.

The parser kept the nesting of arrays and objects in a std::vector<bool>,
whose push_back() GCC calls out of line; it is a std::vector<std::uint8_t>
now.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Skip the literal fast path test without exceptions

The test case compares the error messages of both lexer paths, so it does
not compile with JSON_NOEXCEPTION (the caught exception does not exist).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Allow the string literal parameter of scan_literal() for clang-tidy

The parameter takes the literal by array reference so that its length is
a template parameter; this is the same exemption the C array overloads of
to_json() and from_json() have.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-10-21 20:00:51 +02:00
Niels Lohmann e9970066c5 Scan strings with SSE2, SSSE3, or NEON (#5853) 
* Scan strings with SSE2, SSSE3, or NEON

The string fast path found the end of a run of plain characters 8 bytes at
a time (SWAR) and validated non-ASCII text one sequence at a time, calling
validate_one_utf8() out of line for each.

The vector code of json_view's scanner, in the new shared header
detail/input/simd_scan.hpp, now does both 16 bytes at a time: SSE2 finds
the end of a run on x86-64, and the UTF-8 check of Keiser and Lemire runs
with NEON on AArch64 and with SSSE3 on x86-64. SSSE3 is not part of x86-64,
so the check is compiled for it with a function attribute and used when the
CPU has it (cached in a constant-initialized atomic), else the scalar check;
code compiled with -mssse3 skips the CPU check. The first word of a run is
still checked with SWAR (the first four with NEON, where a vector mask
costs more), as most keys and short values end there, and the scalar
helpers are always inlined. On anything else, the scalar code is unchanged.

The new macro JSON_NO_SIMD selects the portable code everywhere, including
the SSE2/NEON digit formatting of the floating-point writer. The lexer tests
now also run with JSON_NO_SIMD and, on x86-64, with -mssse3, and a new test
places every byte that ends a run at each offset across several 16-byte
blocks, after ASCII and after CJK text.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Keep the string scan from slowing down short strings and dump() on Arm

Three benchmarks were 7-9% slower than develop on Apple M1 (Apple Clang,
-O3 -flto): Accept/jeopardy, Dump/citm_catalog, and Dump/jeopardy.

- dump(): the serializer passes a string without its closing quote, so
  plain_run_end() ran out of input and called the out-of-line NEON loop
  for every string, with fewer than 16 bytes left. The loop is now only
  called for 16 bytes or more.
- accept(): no string of jeopardy.json reaches the vector UTF-8 check,
  yet compiling it into the lexer's scan function made it 5% slower: the
  function runs once per token and now saved more registers and kept a
  stack slot. lexer_string_bulk_run() checks the first four words inline
  and continues out of line, and the serializer keeps string_bulk_run()
  inline, which is faster there.
- The byte that ends a run is read from the scanned buffer instead of
  through get(), which Apple Clang merged with the other get() calls that
  load the read position again right after it was stored (3%).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Define string_bulk_run_from() only where lexer_string_bulk_run() calls it

With NEON and simdutf in C++17, lexer_string_bulk_run() takes the
string_bulk_run() path, so the out-of-line helper was unused.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Do not dispatch the SSSE3 string scan with GCC's C++20 modules

GCC does not keep the target("ssse3") attribute on functions it reads
back from a module interface, so the module build (ci_module_cpp20,
gcc:latest) rejected the intrinsics in vector_utf8_run() with "needs isa
option -mssse3". With __cpp_modules, GCC now uses the portable UTF-8 check
like other compilers without the target attribute; the SSE2 scan stays,
and -mssse3 or more still compiles the SSSE3 check directly. Clang is not
affected.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Include <cpuid.h> only once when json.hpp and json_view.hpp are both used

The SSSE3 dispatch of the string scan and the one of json_view each
include <cpuid.h>. Before GCC 11 the header has no include guard, so a
translation unit that includes both headers fails with GCC 10:
"redefinition of 'unsigned int __get_cpuid_max(unsigned int, unsigned int*)'".
Guard both includes with NLOHMANN_JSON_CPUID_INCLUDED.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Add an ABI tag for JSON_NO_SIMD

JSON_NO_SIMD selects between two definitions of the internal scanning and
formatting functions. Like JSON_VIEW_NO_SIMD (_vns), it now appends _ns to
the inline namespace, so translation units built with and without it can
be linked into one program without ODR violations.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-10-21 20:00:50 +02:00
Niels LohmannandKartikey 3f8677d028 Limit Natvis namespace variants to two ABI tags (#5877) 
The generator wrote a namespace entry for every combination of ABI tags,
so nlohmann_json.natvis doubled with each new tag. With the 12 tags on
develop it had reached 57.9 MB, and one more tag would push it past
GitHub's 100 MB file limit. It now covers the default namespace, every
single tag, and every pair of tags (versioned and unversioned), which is
158 entries / 1 MB with 12 tags.

Taken over from #5868. Refs #5863.

Signed-off-by: Kartikey <kg704925@gmail.com>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Co-authored-by: Kartikey <kg704925@gmail.com>
 2026-10-21 18:26:07 +02:00
Niels Lohmann ef3d350639 Split the Visual Studio 2019 AppVeyor job into two shards (#5872) 
The Visual Studio 2019 job (Release, x86, JSON_ImplicitConversions=OFF)
builds the tests for all standards and takes 49 to 57 minutes when it
passes, close to AppVeyor's 60-minute limit per job. In build 54917118
(#5867), build 54917312 (#5620), and build 54930878 (#5851) it was
stopped at the limit while still compiling. Split it with JSON_TestShard
like the Visual Studio 2017 jobs.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-10-21 16:25:04 +02:00
Niels Lohmann 5e13f834c1 Copy the text of float tokens only when a SAX handler asks for it (#5852) 
After the digits are converted while scanning, copying the token text into
token_buffer (an out-of-line std::string::append) was about 10% of parsing
canada.json. The only reader is the string argument of
json_sax::number_float(), and the library's own handlers
(json_sax_dom_parser, json_sax_dom_callback_parser, json_sax_acceptor)
ignore it.

For contiguous input, the fast path now keeps a pointer to the token in the
input, and get_string() copies it on first use. The parser passes an empty
string to the handlers marked by the new internal trait
sax_ignores_number_text instead of calling get_string(), so json::parse()
and json::accept() never copy float text. User handlers, and
JSON_DIAGNOSTIC_POSITIONS (which reads get_string().size()), still see the
exact text. Keys and strings, which never have pending text, use the
unchecked get_string_token(). Input read through a window into a stream
(std::istream, FILE*) copies the text right away, because the next read may
refill the window.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-10-21 12:26:46 +02:00
Niels Lohmann e13e976801 Accumulate number digits while scanning them (#5851) 
* Accumulate number digits while scanning them

The contiguous number fast path validated a token byte by byte, copied it
into token_buffer, and then converted it in out-of-line functions that
scanned the digits a second time: parse_float_native() called
parse_float_significand(), which returned its result through memory and
called std::any_of (out of line with GCC) twice per number, and
parse_integer_signed() divided by ten on every digit for its overflow check.

scan_number_bulk_contiguous() now accumulates the significand while it
validates the digits, eight at a time where at least eight bytes remain
(is_eight_digits() and parse_eight_digits(), after fast_float). Floats with
at most 19 digits go straight to decimal_to_float(); integers with at most
18 digits are stored without an overflow check (a narrower number type
still rejects values that do not round-trip). Longer tokens take the
existing path. has_nonzero_digit() is a plain loop and
parse_float_significand() is always inlined, which helps that path.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Skip the digit accumulation checks without exceptions

The section compares error messages from both lexer paths and parses
out-of-range numbers, so it does not compile (the caught exception does not
exist) and would abort with JSON_NOEXCEPTION.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Share the significant-digit loop of the integer part and the fraction

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-10-21 12:26:46 +02:00
Niels Lohmann 898e6ee81d Add benchmarks for SAX parsing, value operations, JSON Pointer, and conversions (#5845) 
Squashed onto develop cd6b428ef (CI fixes from #5860). Original commits:

- Add benchmarks for SAX parsing, value operations, JSON Pointer, and conversions
- Time copies together with their destruction in the benchmarks
- Time destroying the json in ToJsonStrings

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-10-21 12:25:42 +02:00
Niels Lohmann ef1cdb59ac Update performance docs for fast std::istream and FILE* parsing (#5875) 
performance.md still claimed streams have no fast path (outdated since #5828).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-10-21 12:25:10 +02:00
177 changed files with 24909 additions and 601341 deletions
Expand all files Collapse all files

No files matched your search

.github/CONTRIBUTING.md
+3 -10
View File
@@ -108,16 +108,9 @@ The tests are located in [`tests/src/unit-*.cpp`](https://github.com/nlohmann/js
are structured along the features of the library or the nature of the tests. Usually, it should be clear from the
context which existing file needs to be extended, and only very few cases require creating new test files.
When fixing a bug, edit `unit-regression3.cpp` and add a section referencing the fixed issue. Tests that need C++17 or
C++20 go into `unit-regression3-cpp17.cpp` or `unit-regression3-cpp20.cpp` instead, so the large file is not rebuilt
for every C++ standard. `unit-regression2.cpp` holds the older tests; the two files exist because a single one grew
large enough for the MinGW linker to fail relocating it, so please keep adding to the smaller file rather than growing
the larger one.
To keep compile times down, several test files are compiled together as one translation unit. This sets a few rules
for test files, such as giving file-scope helpers file-specific names. See the README in the
[`tests`](https://github.com/nlohmann/json/tree/develop/tests) folder for how the tests are built and what to keep in
mind when adding them.
When fixing a bug, edit `unit-regression3.cpp` and add a section referencing the fixed issue.
`unit-regression2.cpp` holds the older tests; the two files exist because a single one grew large enough for the
MinGW linker to fail relocating it, so please keep adding to the smaller file rather than growing the larger one.
#### Exceptions
.github/external_ci/appveyor.yml
+9 -2
View File
@@ -18,7 +18,8 @@ environment:
matrix:
# The Visual Studio 2017 jobs compile everything with /std:c++17, so they
# only build the C++17 variant of each test, split into two jobs each to
# stay below AppVeyor's 60-minute limit per job.
# stay below AppVeyor's 60-minute limit per job. The Visual Studio 2019
# job builds the tests for all standards and is split for the same reason.
- APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2015
configuration: Debug
platform: x86
@@ -50,7 +51,13 @@ environment:
configuration: Release
platform: x86
CXX_FLAGS: "/W4 /WX"
CMAKE_OPTIONS: "-DJSON_ImplicitConversions=OFF"
CMAKE_OPTIONS: "-DJSON_ImplicitConversions=OFF -DJSON_TestShard=0/2"
GENERATOR: Visual Studio 16 2019
- APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2019
configuration: Release
platform: x86
CXX_FLAGS: "/W4 /WX"
CMAKE_OPTIONS: "-DJSON_ImplicitConversions=OFF -DJSON_TestShard=1/2"
GENERATOR: Visual Studio 16 2019
- APPVEYOR_BUILD_WORKER_IMAGE: Visual Studio 2015
.github/labeler.yml
+1
View File
@@ -64,6 +64,7 @@ labels:
- "tests/src/unit-json_view.*"
- "tests/src/json_view_test_helpers\\.hpp"
- "tests/src/fuzzer-(parse_json_view|json_view_image)\\.cpp"
- "tests/benchmarks/json_view/.*"
- "tools/amalgamate/config_json_view\\.json"
- "docs/mkdocs/docs/features/json_view\\.md"
- "docs/mkdocs/docs/api/basic_json_(document|view)/.*"
.github/workflows/check_amalgamation.yml
+22
View File
@@ -154,6 +154,28 @@ jobs:
# PR content in a job without credentials.
python3 tools/generate_natvis/generate_natvis.py $MAIN_DIR
- name: Check the nlohmann_json.natvis generator options
run: |
cd $MAIN_DIR
# The options for configurations outside the checked-in file write
# to a temporary file, so they do not affect the diff below.
GENERATOR=tools/generate_natvis/generate_natvis.py
python3 $GENERATOR --list-tags
TAG_COUNT=$(python3 $GENERATOR --list-tags | wc -l)
python3 $GENERATOR --abi-tags diag,_dp,_vns --abi-tags _diag --output $RUNNER_TEMP/abi_tags.natvis
grep -q '<!-- Namespace nlohmann::json_abi_diag_dp_vns -->' $RUNNER_TEMP/abi_tags.natvis
grep -q '<!-- Namespace nlohmann::json_abi_diag_dp_vns_v[0-9_]* -->' $RUNNER_TEMP/abi_tags.natvis
test $(grep -c '<!-- Namespace ' $RUNNER_TEMP/abi_tags.natvis) -eq $(( $(grep -c '<!-- Namespace ' nlohmann_json.natvis) + 2 ))
python3 $GENERATOR --all --output $RUNNER_TEMP/all.natvis
test $(grep -c '<!-- Namespace ' $RUNNER_TEMP/all.natvis) -eq $(( (1 << TAG_COUNT) * 2 + 1 ))
if python3 $GENERATOR --abi-tags _unknown --output $RUNNER_TEMP/unknown.natvis; then
echo "generate_natvis.py accepted an unknown ABI tag"
exit 1
fi
- name: Build patch and check for differences
id: diff
run: |
.github/workflows/codeql-analysis.yml
+3 -3
View File
@@ -38,14 +38,14 @@ jobs:
# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@24c54180a607b1449ed407dd24f251e4e9147c8d # v4.38.3
uses: github/codeql-action/init@6f2ac81f5cd81fe351a197ba0b00d91d03cd2aee # v4.39.0
with:
languages: c-cpp
# Autobuild attempts to build any compiled languages (C/C++, C#, or Java).
# If this step fails, then you should remove it and run the build manually (see below)
- name: Autobuild
uses: github/codeql-action/autobuild@24c54180a607b1449ed407dd24f251e4e9147c8d # v4.38.3
uses: github/codeql-action/autobuild@6f2ac81f5cd81fe351a197ba0b00d91d03cd2aee # v4.39.0
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@24c54180a607b1449ed407dd24f251e4e9147c8d # v4.38.3
uses: github/codeql-action/analyze@6f2ac81f5cd81fe351a197ba0b00d91d03cd2aee # v4.39.0
.github/workflows/comment_check_amalgamation.yml
+11 -1
View File
@@ -37,6 +37,14 @@ jobs:
var matchArtifact = artifacts.data.artifacts.filter((artifact) => {
return artifact.name == "pr"
})[0];
// a run that failed before its jobs started (e.g. a stale queued
// run cancelled long after its push) has no artifact to report on
if (!matchArtifact) {
core.notice('Run ${{ github.event.workflow_run.id }} has no "pr" artifact; nothing to comment on.');
core.setOutput('found', 'false');
return;
}
core.setOutput('found', 'true');
var download = await github.rest.actions.downloadArtifact({
owner: context.repo.owner,
repo: context.repo.repo,
@@ -51,9 +59,11 @@ jobs:
# Extract the untrusted PR artifact into a dedicated empty directory and
# read only the two expected files by fixed path afterwards. This avoids a
# malicious archive overwriting workspace files or escaping via ../ paths.
- run: unzip -o pr.zip -d ./pr_artifact
- if: steps.download.outputs.found == 'true'
run: unzip -o pr.zip -d ./pr_artifact
- name: 'Comment on PR'
if: steps.download.outputs.found == 'true'
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
.github/workflows/flawfinder.yml
+1 -1
View File
@@ -47,6 +47,6 @@ jobs:
output: 'flawfinder_results.sarif'
- name: Upload analysis results to GitHub Security tab
uses: github/codeql-action/upload-sarif@24c54180a607b1449ed407dd24f251e4e9147c8d # v4.38.3
uses: github/codeql-action/upload-sarif@6f2ac81f5cd81fe351a197ba0b00d91d03cd2aee # v4.39.0
with:
sarif_file: ${{github.workspace}}/flawfinder_results.sarif
.github/workflows/json_view_benchmarks.yml
+84
View File
@@ -0,0 +1,84 @@
name: "json_view benchmarks"
# On demand only: runs the comparison of json_view with yyjson, simdjson, and
# Boost.JSON (tests/benchmarks/json_view/compare.py) on GitHub-hosted runners,
# for numbers from x86-64 and AArch64 Linux. It runs when started by hand, or
# when a pull request gets the label "benchmark" (on both architectures, with
# GCC and the default settings). Shared runners are noisy: the results show
# where json_view stands, but published numbers need a quiet machine (see
# tests/benchmarks/json_view/README.md).
on:
pull_request:
types: [labeled]
workflow_dispatch:
inputs:
runner:
description: "Runner image"
type: choice
options:
- ubuntu-24.04
- ubuntu-24.04-arm
default: ubuntu-24.04
compiler:
description: "Compiler"
type: choice
options:
- g++
- clang++
default: g++
native:
description: "Compile for the runner's CPU (-march=native)"
type: boolean
default: false
rounds:
description: "Rounds of bench_view"
type: number
default: 30
permissions:
contents: read
concurrency:
# (the label is part of the group: adding an unrelated label must not cancel a running benchmark)
group: json-view-benchmarks-${{ github.workflow }}-${{ github.ref }}-${{ github.event.label.name }}
cancel-in-progress: true
jobs:
compare:
if: github.event_name == 'workflow_dispatch' || github.event.label.name == 'benchmark'
strategy:
matrix:
runner: ${{ fromJSON(github.event_name == 'workflow_dispatch' && format('["{0}"]', inputs.runner) || '["ubuntu-24.04", "ubuntu-24.04-arm"]') }}
runs-on: ${{ matrix.runner }}
timeout-minutes: 60
steps:
- name: Harden Runner
uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1
with:
egress-policy: audit
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Download test data
run: |
cmake -S . -B build -DJSON_BuildTests=On
cmake --build build --target download_test_data
- name: Run the comparison
env:
CXX: ${{ inputs.compiler || 'g++' }}
CC: ${{ inputs.compiler == 'clang++' && 'clang' || 'gcc' }}
ROUNDS: ${{ inputs.rounds || 30 }}
NATIVE: ${{ inputs.native && '--native' || '' }}
run: python3 tests/benchmarks/json_view/compare.py --data build/test_files --download --rounds "$ROUNDS" $NATIVE
- name: Summary
run: cat tests/benchmarks/json_view/results/*.md >> "$GITHUB_STEP_SUMMARY"
- uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
with:
name: json_view-benchmarks-${{ matrix.runner }}-${{ inputs.compiler || 'g++' }}
path: tests/benchmarks/json_view/results/
.github/workflows/scorecards.yml
+1 -1
View File
@@ -80,6 +80,6 @@ jobs:
# Upload the results to GitHub's code scanning dashboard.
- name: "Upload to code-scanning"
uses: github/codeql-action/upload-sarif@24c54180a607b1449ed407dd24f251e4e9147c8d # v4.38.3
uses: github/codeql-action/upload-sarif@6f2ac81f5cd81fe351a197ba0b00d91d03cd2aee # v4.39.0
with:
sarif_file: results.sarif
.github/workflows/semgrep.yml
+1 -1
View File
@@ -65,7 +65,7 @@ jobs:
# Upload SARIF file generated in previous step
- name: Upload SARIF file
uses: github/codeql-action/upload-sarif@24c54180a607b1449ed407dd24f251e4e9147c8d # v4.38.3
uses: github/codeql-action/upload-sarif@6f2ac81f5cd81fe351a197ba0b00d91d03cd2aee # v4.39.0
with:
sarif_file: semgrep.sarif
if: always()
BUILD.bazel
+1
View File
@@ -37,6 +37,7 @@ cc_library(
"include/nlohmann/detail/input/parser.hpp",
"include/nlohmann/detail/input/position_t.hpp",
"include/nlohmann/detail/input/pow5_table.hpp",
"include/nlohmann/detail/input/simd_scan.hpp",
"include/nlohmann/detail/input/string_scan.hpp",
"include/nlohmann/detail/iterators/internal_iterator.hpp",
"include/nlohmann/detail/iterators/iter_impl.hpp",
FILES.md
+27
View File
@@ -6,6 +6,7 @@ This file describes the source for supporting files; that is, files that are not
- [GitHub](#github)
- [REUSE](#reuse)
- [Package Managers](#package-managers)
- [Debugging](#debugging)
## Continuous Integration
@@ -289,3 +290,29 @@ The module definition for [Bazel](https://bazel.build)'s [Bzlmod](https://bazel.
Further documentation:
- [Bazel modules](https://bazel.build/external/module)
## Debugging
### `nlohmann_json.natvis`
The [Natvis](https://learn.microsoft.com/en-us/visualstudio/debugger/create-custom-views-of-native-objects) file that
gives `json` values a friendly view in the Visual Studio debugger. The file is generated by
`tools/generate_natvis/generate_natvis.py` from the template `tools/generate_natvis/nlohmann_json.natvis.j2` and the ABI
tags and version in `include/nlohmann/detail/abi_macros.hpp`; change the script or the template rather than editing
the file by hand. The file can be updated by calling
```shell
make natvis
```
The "Check amalgamation" workflow fails if the file is out of date.
The file covers the namespace variants with up to two ABI tags. A Natvis file for other configurations can be
generated with the `--abi-tags` or `--all` options, for example
`make natvis NATVIS_ARGS="--abi-tags _diag,_dp,_vns --output my_nlohmann_json.natvis"`; see
[`tools/generate_natvis`](tools/generate_natvis/README.md). With MSVC, CMake adds the file to the `nlohmann_json`
target and installs it.
Further documentation:
- [Create custom views of C++ objects in the debugger using the Natvis framework](https://learn.microsoft.com/en-us/visualstudio/debugger/create-custom-views-of-native-objects)
Makefile
+4 -2
View File
@@ -46,7 +46,7 @@ all:
@echo "clean - remove built files"
@echo "(fuzzers) - see tests/fuzzing.md, or run 'make -C tests fuzzers' to build the AFL fuzzers"
@echo "macro_builder_check - check that macro_scope.hpp matches tools/macro_builder's output"
@echo "natvis - regenerate nlohmann_json.natvis from the current ABI tags and version"
@echo "natvis - regenerate nlohmann_json.natvis from the current ABI tags and version (pass generator options via NATVIS_ARGS, see tools/generate_natvis/README.md)"
@echo "pretty - beautify code with Artistic Style"
@echo "pvs_studio - run the PVS-Studio static analyzer and open the report"
@echo "release - create the release files (include.zip, json.tar.xz, single headers, signatures, hashes)"
@@ -120,13 +120,15 @@ $(AMALGAMATED_VIEW_FILE): $(SRCS)
tools/amalgamate/amalgamate.py -c tools/amalgamate/config_json_view.json -s . --verbose=yes
NATVIS_VENV=tools/generate_natvis/venv
# extra options for generate_natvis.py, e.g. NATVIS_ARGS="--abi-tags _diag,_dp,_vns --output my.natvis"
NATVIS_ARGS=
# regenerate nlohmann_json.natvis from the ABI tags and version in include/nlohmann/detail/abi_macros.hpp
# (the dependencies are installed into a virtual environment first)
natvis:
@test -d $(NATVIS_VENV) || python3 -mvenv $(NATVIS_VENV)
@$(NATVIS_VENV)/bin/python3 -c 'import jinja2' 2>/dev/null || $(NATVIS_VENV)/bin/pip3 install --quiet -r tools/generate_natvis/requirements.txt
$(NATVIS_VENV)/bin/python3 tools/generate_natvis/generate_natvis.py .
$(NATVIS_VENV)/bin/python3 tools/generate_natvis/generate_natvis.py $(NATVIS_ARGS) .
# regenerate the two tools/macro_builder blocks of $(MACRO_SCOPE_HPP) (see its README.md) and diff against the
# checked-in header; phony, because it never writes $(MACRO_SCOPE_HPP) itself
cmake/ci.cmake
+3 -3
View File
@@ -890,11 +890,11 @@ add_custom_target(ci_icpx
# The following tests are excluded as they trigger known nvc++ 25.5 defects (not
# library bugs); see https://github.com/nlohmann/json for tracking. Only the
# affected language-standard variants are excluded so coverage is otherwise kept:
# - test-comparison-cpp20_cpp20, test-comparison_legacy-cpp20_cpp20
# - test-comparison_cpp20, test-comparison_legacy_cpp20
# miscompiles cross-type/<=> comparison (e.g. `-17 <= null`)
# - test-constructor1_cpp11
# std::initializer_list lifetime bug -> SIGSEGV
# - test-deserialization-cpp20_cpp20
# - test-deserialization_cpp20
# mangles the UTF-8 u8"" string literal in the char8_t (C++20) section
add_custom_target(ci_nvhpc
COMMAND ${CMAKE_COMMAND}
@@ -906,7 +906,7 @@ add_custom_target(ci_nvhpc
COMMAND ${CMAKE_COMMAND} --build ${PROJECT_BINARY_DIR}/build_nvhpc
# the pipes are escaped so the surrounding shell passes them to ctest verbatim
# instead of treating them as shell pipe operators
COMMAND cd ${PROJECT_BINARY_DIR}/build_nvhpc && ${CMAKE_CTEST_COMMAND} --parallel ${N} --exclude-regex "test-comparison-cpp20_cpp20\\|test-comparison_legacy-cpp20_cpp20\\|test-constructor1_cpp11\\|test-deserialization-cpp20_cpp20" --output-on-failure
COMMAND cd ${PROJECT_BINARY_DIR}/build_nvhpc && ${CMAKE_CTEST_COMMAND} --parallel ${N} --exclude-regex "test-comparison_cpp20\\|test-comparison_legacy_cpp20\\|test-constructor1_cpp11\\|test-deserialization_cpp20" --output-on-failure
COMMENT "Compile and test with NVIDIA HPC SDK (nvc++)"
)
cmake/test.cmake
+39 -366
View File
@@ -109,55 +109,6 @@ function(_json_test_apply_test_properties test_target properties_target)
endif()
endfunction()
# for internal use by _json_test_add_test() and _json_test_add_unity_batch():
# registers the CTest test <test_name>_cpp<cxx_standard> (plus its Valgrind
# variant), which runs the executable target <test_target> with the arguments
# in ARGN, and applies the test properties of the test- and standard-specific
# interface targets
function(_json_test_register_test test_name test_target cxx_standard)
set(ctest_name ${test_name}_cpp${cxx_standard})
if (JSON_FastTests)
add_test(NAME ${ctest_name}
COMMAND ${test_target} ${DOCTEST_TEST_FILTER} ${ARGN}
WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}
)
else()
add_test(NAME ${ctest_name}
COMMAND ${test_target} ${DOCTEST_TEST_FILTER} ${ARGN} --no-skip
WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}
)
endif()
set_tests_properties(${ctest_name} PROPERTIES LABELS "all" FIXTURES_REQUIRED TEST_DATA)
# apply standard-specific test properties
if(TARGET _json_test_interface__cpp_${cxx_standard})
_json_test_apply_test_properties(${ctest_name} _json_test_interface__cpp_${cxx_standard})
endif()
# apply test-specific test properties
if(TARGET _json_test_interface_${test_name})
_json_test_apply_test_properties(${ctest_name} _json_test_interface_${test_name})
endif()
# apply test- and standard-specific test properties
if(TARGET _json_test_interface_${test_name}_cpp_${cxx_standard})
_json_test_apply_test_properties(${ctest_name}
_json_test_interface_${test_name}_cpp_${cxx_standard}
)
endif()
if(JSON_Valgrind)
add_test(NAME ${ctest_name}_valgrind
COMMAND ${memcheck_command} $<TARGET_FILE:${test_target}> ${DOCTEST_TEST_FILTER} ${ARGN}
WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}
)
set_tests_properties(${ctest_name}_valgrind PROPERTIES
LABELS "valgrind" FIXTURES_REQUIRED TEST_DATA
)
endif()
endfunction()
# for internal use by json_test_add_test_for()
function(_json_test_add_test test_name file main cxx_standard)
set(test_target ${test_name}_cpp${cxx_standard})
@@ -192,7 +143,45 @@ function(_json_test_add_test test_name file main cxx_standard)
)
endif()
_json_test_register_test(${test_name} ${test_target} ${cxx_standard})
if (JSON_FastTests)
add_test(NAME ${test_target}
COMMAND ${test_target} ${DOCTEST_TEST_FILTER}
WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}
)
else()
add_test(NAME ${test_target}
COMMAND ${test_target} ${DOCTEST_TEST_FILTER} --no-skip
WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}
)
endif()
set_tests_properties(${test_target} PROPERTIES LABELS "all" FIXTURES_REQUIRED TEST_DATA)
# apply standard-specific test properties
if(TARGET _json_test_interface__cpp_${cxx_standard})
_json_test_apply_test_properties(${test_target} _json_test_interface__cpp_${cxx_standard})
endif()
# apply test-specific test properties
if(TARGET _json_test_interface_${test_name})
_json_test_apply_test_properties(${test_target} _json_test_interface_${test_name})
endif()
# apply test- and standard-specific test properties
if(TARGET _json_test_interface_${test_name}_cpp_${cxx_standard})
_json_test_apply_test_properties(${test_target}
_json_test_interface_${test_name}_cpp_${cxx_standard}
)
endif()
if(JSON_Valgrind)
add_test(NAME ${test_target}_valgrind
COMMAND ${memcheck_command} $<TARGET_FILE:${test_target}> ${DOCTEST_TEST_FILTER}
WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}
)
set_tests_properties(${test_target}_valgrind PROPERTIES
LABELS "valgrind" FIXTURES_REQUIRED TEST_DATA
)
endif()
endfunction()
#############################################################################
@@ -211,14 +200,6 @@ endfunction()
# Use NAME <name> to override the filename-derived test name.
# Use FORCE to create the test regardless of the file containing
# JSON_HAS_CPP_<version_number>.
#
# Tests that depend on the C++ standard (e.g., because they use the macros
# JSON_HAS_FILESYSTEM, JSON_HAS_RANGES, or JSON_HAS_THREE_WAY_COMPARISON)
# should not make the whole of a large unit-foo.cpp be rebuilt for every
# standard. Put them into a separate file unit-foo-cpp<NN>.cpp (see, e.g.,
# unit-items-cpp17.cpp) which wraps its content in #ifdef JSON_HAS_CPP_<NN>.
# Then, unit-foo.cpp itself contains no JSON_HAS_CPP_<NN> and is only built for
# C++11.
# Test targets are linked against <main>.
# CXX_STANDARDS defaults to "11".
#############################################################################
@@ -262,314 +243,6 @@ function(json_test_add_test_for file)
endforeach()
endfunction()
# for internal use by json_test_add_unity_tests(): sets <result> to whether the
# (absolute) <file> is built for <cxx_standard>; same rule as in
# json_test_add_test_for(): C++11 always, others only if the file contains
# JSON_HAS_CPP_<cxx_standard> or <force> is set
function(_json_test_unity_applies file cxx_standard force result)
set(${result} TRUE PARENT_SCOPE)
if(NOT ("${cxx_standard}" STREQUAL 11 OR force))
file(READ ${file} file_content)
string(FIND "${file_content}" JSON_HAS_CPP_${cxx_standard} has_cpp_found)
if(${has_cpp_found} EQUAL -1)
set(${result} FALSE PARENT_SCOPE)
endif()
endif()
endfunction()
# for internal use by json_test_add_unity_tests(): creates the executable
# test-unity-<batch_name>_cpp<cxx_standard> from the (absolute) source files
# in ARGN, which are #include-d by a generated source file, and registers one
# CTest test per source file that runs only the test cases of that file; if
# <private> is true, the generated file defines JSON_TESTS_PRIVATE first
function(_json_test_add_unity_batch batch_name cxx_standard main private)
set(batch_target test-unity-${batch_name}_cpp${cxx_standard})
set(batch_source ${PROJECT_BINARY_DIR}/tests/unity/${batch_target}.cpp)
set(batch_content "// generated by cmake/test.cmake; do not edit\n")
if(private)
string(APPEND batch_content "// at least one file of this batch needs access to private members of the library\n")
string(APPEND batch_content "#define JSON_TESTS_PRIVATE\n")
# the files define the macro again; mark this definition as used, as
# -Wunused-macros reports an unused definition when it is redefined
string(APPEND batch_content "#ifdef JSON_TESTS_PRIVATE\n#endif\n")
endif()
foreach(file ${ARGN})
string(APPEND batch_content "#include \"${file}\" // NOLINT(bugprone-suspicious-include)\n")
endforeach()
# only touch the generated file if it changed to keep incremental builds incremental
set(old_content "")
if(EXISTS ${batch_source})
file(READ ${batch_source} old_content)
endif()
if(NOT "${old_content}" STREQUAL "${batch_content}")
file(WRITE ${batch_source} "${batch_content}")
endif()
add_executable(${batch_target} ${batch_source})
target_link_libraries(${batch_target} PRIVATE ${main})
set_target_properties(${batch_target} PROPERTIES
CXX_STANDARD ${cxx_standard}
CXX_STANDARD_REQUIRED ON
)
if(TARGET _json_test_interface__cpp_${cxx_standard})
target_link_libraries(${batch_target} PRIVATE _json_test_interface__cpp_${cxx_standard})
endif()
# rebuild the batch when one of its files changes, and show the files in IDEs;
# files that are also built standalone (VARIANT_FILES) are left out, as
# HEADER_FILE_ONLY is a per-file property and would also affect those targets
set_source_files_properties(${batch_source} PROPERTIES OBJECT_DEPENDS "${ARGN}")
foreach(file ${ARGN})
if(NOT file IN_LIST _json_test_unity_variant_files)
set_source_files_properties(${file} PROPERTIES HEADER_FILE_ONLY ON)
target_sources(${batch_target} PRIVATE ${file})
endif()
endforeach()
foreach(file ${ARGN})
get_filename_component(file_basename ${file} NAME_WE)
string(REGEX REPLACE "unit-(.+)" "test-\\1" test_name ${file_basename})
# run only the test cases defined in this file (and in the shared
# make_test_data_available.hpp if the file uses it)
file(READ ${file} file_content)
set(source_filter "--source-file=*${file_basename}.cpp")
string(FIND "${file_content}" make_test_data_available.hpp uses_test_data)
if(NOT ${uses_test_data} EQUAL -1)
string(APPEND source_filter ",*make_test_data_available.hpp")
endif()
_json_test_register_test(${test_name} ${batch_target} ${cxx_standard} "${source_filter}")
endforeach()
endfunction()
#############################################################################
# json_test_add_unity_tests(
# FILES <files>...
# MAIN <main>
# [CXX_STANDARDS <version_number>...] [FORCE]
# [BATCH_SIZE <size>]
# [GROUPS <group>...]
# [VARIANT_FILES <files>...])
#
# Like calling json_test_add_test_for(<file> MAIN <main> ...) for each of the
# <files>, but compiles several files together to speed up the build: for each
# C++ standard, the files are split into batches of <size> files (default: 8)
# and each batch is built as a single executable
#
# test-unity-<pool><index>_cpp<version_number>
#
# whose generated source file #include-s the files of the batch (so they share
# the template instantiations of the library). The tests are still named
# test-foo_cpp<version_number>, one per file, but run the batch executable with
# a doctest filter that selects the test cases of that file only.
#
# Files are only batched with files that agree on the macros defined before
# the library is included: files that define at most JSON_TESTS_PRIVATE (or
# macros derived from global compile definitions) form the pools "plain" and
# "private". All other files are added with json_test_add_test_for() as usual,
# as are files with test-specific build settings (see
# json_test_set_test_options()) and the files listed in the explicit
# exclusion list below.
# Each <group> names a list variable json_test_unity_group_<group> of test file
# stems (file names without "unit-" and ".cpp"). The batchable files of a group
# are compiled together (regardless of BATCH_SIZE) as one executable
#
# test-unity-<group>_cpp<version_number>
#
# so that related tests, which instantiate the same templates, share one
# translation unit. A group may mix the pools "plain" and "private"; if any of
# its files needs JSON_TESTS_PRIVATE, the whole group is built with it. Files
# that cannot be batched stay standalone even if they are listed in a group.
# Files in no group are batched by BATCH_SIZE as described above.
# <files> in VARIANT_FILES are also built standalone with other settings, so
# they are not marked as header-only sources of the batch.
#############################################################################
function(json_test_add_unity_tests)
cmake_parse_arguments(args "FORCE" "MAIN;BATCH_SIZE" "FILES;CXX_STANDARDS;VARIANT_FILES;GROUPS" ${ARGN})
if("${args_MAIN}" STREQUAL "")
message(FATAL_ERROR "Required argument MAIN <main> missing.")
endif()
if("${args_BATCH_SIZE}" STREQUAL "")
set(args_BATCH_SIZE 8)
endif()
if("${args_CXX_STANDARDS}" STREQUAL "")
set(args_CXX_STANDARDS 11)
endif()
if(args_FORCE)
set(force FORCE)
else()
set(force "")
endif()
set(_json_test_unity_variant_files "")
foreach(file ${args_VARIANT_FILES})
get_filename_component(file ${file} ABSOLUTE)
list(APPEND _json_test_unity_variant_files ${file})
endforeach()
# files that must not be merged into a batch: unit-32bit.cpp is only built
# for 32bit targets, and unit-no-macro-leak.cpp checks that including the
# library defines no unprefixed macro, which any other file would disturb
set(standalone_files unit-32bit.cpp unit-no-macro-leak.cpp)
set(harmless_macros "^(DOCTEST_.*|SKIP_TESTS_FOR_.*|JSON_TEST_DEPRECATED_FUNCTIONS_DELETED|JSON_TEST_STRICT_NUL_HANDLING_ENABLED|JSON_TEST_STRINGIZE)$")
# classify the files
set(plain_files "")
set(private_files "")
set(standalone_abs_files "")
foreach(file ${args_FILES})
get_filename_component(file_name ${file} NAME)
get_filename_component(file_basename ${file} NAME_WE)
string(REGEX REPLACE "unit-(.+)" "test-\\1" test_name ${file_basename})
set(batchable TRUE)
if(file_name IN_LIST standalone_files)
set(batchable FALSE)
endif()
if(TARGET _json_test_interface_${test_name})
set(batchable FALSE)
endif()
foreach(cxx_standard ${args_CXX_STANDARDS})
if(TARGET _json_test_interface_${test_name}_cpp_${cxx_standard})
set(batchable FALSE)
endif()
endforeach()
# files that suppress GCC's -Wnoexcept around their include of the
# library: the suppression has no effect once another file of the
# batch included the library first
if(batchable)
file(READ ${file} file_content)
string(FIND "${file_content}" "SUPPRESS_WARNING(\"-Wnoexcept\")" suppresses_noexcept)
if(NOT ${suppresses_noexcept} EQUAL -1)
set(batchable FALSE)
endif()
endif()
set(pool plain)
if(batchable)
# collect the macros (un)defined before the library is included
file(READ ${file} file_content)
string(FIND "${file_content}" "#include <nlohmann/" include_position)
if(NOT ${include_position} EQUAL -1)
string(SUBSTRING "${file_content}" 0 ${include_position} file_content)
endif()
string(REGEX MATCHALL "(^|\n)[ \t]*#[ \t]*(define|undef)[ \t]+[A-Za-z_0-9]+" directives "${file_content}")
foreach(directive ${directives})
string(REGEX REPLACE "^.*[ \t]([A-Za-z_0-9]+)$" "\\1" macro "${directive}")
if(macro MATCHES "${harmless_macros}")
continue()
elseif("${macro}" STREQUAL JSON_TESTS_PRIVATE)
set(pool private)
else()
set(batchable FALSE)
endif()
endforeach()
endif()
get_filename_component(file_abs ${file} ABSOLUTE)
if(NOT batchable)
list(APPEND standalone_abs_files ${file_abs})
json_test_add_test_for(${file} MAIN ${args_MAIN} CXX_STANDARDS ${args_CXX_STANDARDS} ${force})
continue()
endif()
get_filename_component(file ${file} ABSOLUTE)
list(APPEND ${pool}_files ${file})
endforeach()
# resolve the explicit groups: group_<name>_files are the (absolute) files
# of the group, group_<name>_private tells whether one of them is private
set(grouped_files "")
foreach(group ${args_GROUPS})
if(NOT DEFINED json_test_unity_group_${group})
message(FATAL_ERROR "Unity test group '${group}' is not defined (json_test_unity_group_${group}).")
endif()
set(group_${group}_files "")
set(group_${group}_private FALSE)
foreach(stem ${json_test_unity_group_${group}})
# check against the source directory, because FILES may be filtered (JSON_TestShard)
get_filename_component(file ${CMAKE_CURRENT_SOURCE_DIR}/src/unit-${stem}.cpp ABSOLUTE)
if(NOT EXISTS ${file})
message(FATAL_ERROR "Unity test group '${group}' lists '${stem}', but ${file} does not exist.")
endif()
if(file IN_LIST grouped_files)
message(FATAL_ERROR "Unity test file unit-${stem}.cpp is listed in more than one group (second: '${group}').")
endif()
list(APPEND grouped_files ${file})
if(file IN_LIST standalone_abs_files)
message(STATUS "Unity test group '${group}': unit-${stem}.cpp cannot be batched and stays standalone")
elseif(file IN_LIST plain_files)
list(APPEND group_${group}_files ${file})
list(REMOVE_ITEM plain_files ${file})
elseif(file IN_LIST private_files)
list(APPEND group_${group}_files ${file})
list(REMOVE_ITEM private_files ${file})
set(group_${group}_private TRUE)
endif()
endforeach()
endforeach()
foreach(cxx_standard ${args_CXX_STANDARDS})
if(NOT compiler_supports_cpp_${cxx_standard})
continue()
endif()
# explicit groups: one batch per group
foreach(group ${args_GROUPS})
set(batch_files "")
foreach(file ${group_${group}_files})
_json_test_unity_applies(${file} ${cxx_standard} "${force}" applies)
if(applies)
list(APPEND batch_files ${file})
endif()
endforeach()
if(batch_files)
_json_test_add_unity_batch(${group} ${cxx_standard} ${args_MAIN} ${group_${group}_private} ${batch_files})
endif()
endforeach()
# remaining files: batches of BATCH_SIZE files per pool
foreach(pool plain private)
set(is_private FALSE)
if(pool STREQUAL private)
set(is_private TRUE)
endif()
set(batch_files "")
set(batch_count 0)
set(batch_index 0)
foreach(file ${${pool}_files})
_json_test_unity_applies(${file} ${cxx_standard} "${force}" applies)
if(NOT applies)
continue()
endif()
list(APPEND batch_files ${file})
math(EXPR batch_count "${batch_count} + 1")
if(batch_count EQUAL args_BATCH_SIZE)
_json_test_add_unity_batch(${pool}${batch_index} ${cxx_standard} ${args_MAIN} ${is_private} ${batch_files})
set(batch_files "")
set(batch_count 0)
math(EXPR batch_index "${batch_index} + 1")
endif()
endforeach()
if(batch_files)
_json_test_add_unity_batch(${pool}${batch_index} ${cxx_standard} ${args_MAIN} ${is_private} ${batch_files})
endif()
endforeach()
endforeach()
endfunction()
#############################################################################
# json_test_should_build_32bit_test(
# <build_32bit_var> <build_32bit_only_var> <input>)
docs/docset/generate_docset.py
+2 -1
View File
@@ -341,13 +341,14 @@ def download(url, docs) -> str:
raise ValueError(f'not an http(s) URL: {url}')
req = urllib.request.Request(u.geturl(), headers={'User-Agent': USER_AGENT})
# (the scheme is checked above)
# nosemgrep: python.lang.security.audit.dynamic-urllib-use-detected.dynamic-urllib-use-detected
with urllib.request.urlopen(req, timeout=20) as r: # nosec B310
data = r.read()
ctype = r.headers.get_content_type()
path = urllib.parse.unquote(u.path).lstrip('/')
ext = os.path.splitext(path)[1]
if u.query or not ext or path.endswith('/'):
digest = hashlib.sha1(url.encode(), usedforsecurity=False).hexdigest()[:12]
digest = hashlib.sha256(url.encode()).hexdigest()[:12]
path = os.path.join(os.path.dirname(path), digest + CONTENT_TYPE_EXT.get(ctype, ext or '.bin'))
rel = os.path.normpath(os.path.join('assets', 'external', u.hostname, path))
out = os.path.join(docs, rel)
docs/mkdocs/docs/api/basic_json_view/at.md
+2 -1
View File
@@ -80,7 +80,8 @@ None of these exceptions carry a [`JSON_DIAGNOSTICS`](../macros/json_diagnostics
Objects with 128 or more members get a hash index of their keys while parsing, so that a lookup in them takes
constant time on average.
2. Linear in `idx`: elements are skipped one at a time from the first one, since they are not a fixed size in the
index (unlike `BasicJsonType`'s array, which is random-access).
index (unlike `BasicJsonType`'s array, which is random-access). Constant if no element of the array is an array
or object, since the elements are then adjacent in the index.
3. Linear in the number of reference tokens of `ptr` and, for each token, in the number of members of the object at
that level (as 1.) or the index into the array (as 2.).
docs/mkdocs/docs/api/basic_json_view/back.md
+2 -1
View File
@@ -28,7 +28,8 @@ This exception does not carry a [`JSON_DIAGNOSTICS`](../macros/json_diagnostics.
Linear in the size of the array or object: unlike [`front()`](front.md), which only ever looks at the first
element, `back()` has to walk every element to find where they end, since elements are not a fixed size in the
index.
index. Constant if no element of the array is an array or object, since the elements are then adjacent
in the index.
## Notes
docs/mkdocs/docs/api/basic_json_view/operator[].md
+2 -1
View File
@@ -86,7 +86,8 @@ None of these exceptions carry a [`JSON_DIAGNOSTICS`](../macros/json_diagnostics
Objects with 128 or more members get a hash index of their keys while parsing, so that a lookup in them takes
constant time on average.
2. Linear in `idx`: elements are skipped one at a time from the first one, since they are not a fixed size in the
index (unlike `BasicJsonType`'s array, which is random-access).
index (unlike `BasicJsonType`'s array, which is random-access). Constant if no element of the array is an array
or object, since the elements are then adjacent in the index.
3. Linear in the number of reference tokens of `ptr` and, for each token, in the number of members of the object at
that level (as 1.) or the index into the array (as 2.).
docs/mkdocs/docs/api/macros/index.md
+1
View File
@@ -38,6 +38,7 @@ header. See also the [macro overview page](../../features/macros.md).
- [**JSON_HAS_THREE_WAY_COMPARISON**](json_has_three_way_comparison.md) - control 3-way comparison support
- [**JSON_NO_AUTOMATIC_UDLS**](json_no_automatic_udls.md) - do not include the user-defined string literals (UDLs) automatically
- [**JSON_NO_IO**](json_no_io.md) - switch off functions relying on certain C++ I/O headers
- [**JSON_NO_SIMD**](json_no_simd.md) - use only portable code instead of vector instructions (SSE2, SSSE3, NEON)
- [**JSON_NO_THREAD_LOCAL**](json_no_thread_local.md) - switch off the use of `thread_local` storage
- [**JSON_SKIP_UNSUPPORTED_COMPILER_CHECK**](json_skip_unsupported_compiler_check.md) - do not warn about unsupported compilers
- [**JSON_USE_GLOBAL_UDLS**](json_use_global_udls.md) - place user-defined string literals (UDLs) into the global namespace
docs/mkdocs/docs/api/macros/json_no_simd.md
+83
View File
@@ -0,0 +1,83 @@
# JSON_NO_SIMD
```cpp
#define JSON_NO_SIMD
```
When defined, the library uses only portable C++ code where it would otherwise use vector instructions: when scanning
strings during parsing, and when writing floating-point numbers during serialization.
The macro is an escape hatch, for example for toolchains that cannot compile the vector code, for sanitizers or
analyzers that do not understand it, or to compare the speed of both variants. It does not change any result: values,
exceptions, and error positions are the same either way. Only the speed differs.
## Default definition
By default, `#!cpp JSON_NO_SIMD` is not defined, and vector instructions are used where the target has them.
```cpp
#undef JSON_NO_SIMD
```
## Notes
!!! note "Which instructions are used"
- **x86-64:** SSE2, which every x86-64 CPU has, finds the end of runs of plain string characters 16 bytes at a time
and writes the digits of floating-point numbers. The UTF-8 check of non-ASCII text needs SSSE3, which is not part
of x86-64: it is compiled for SSSE3 with a function attribute and used when the CPU has SSSE3 (every x86-64 CPU
since about 2011), else the portable check. Code compiled for SSSE3 anyway (e.g., with `-mssse3` or `-march=native`)
skips this check of the CPU. 32-bit x86 uses SSE2 for strings when the compiler targets it.
- **AArch64:** NEON, which every 64-bit Arm CPU has, checks UTF-8 text and long runs of plain string characters 16
bytes at a time, and writes the digits of floating-point numbers. With MSVC, only the floating-point digits use
NEON.
- **Other targets:** portable code only.
Strings are only scanned this way for contiguous input and for the characters a `#!cpp std::istream` holds in its
stream buffer (see [`JSON_DISABLE_STREAM_BULK_SCAN`](json_disable_stream_bulk_scan.md)); other inputs are read one
character at a time.
!!! note "Interaction with `JSON_USE_SIMDUTF`"
With [`JSON_USE_SIMDUTF`](json_use_simdutf.md), runs of string characters are first validated by simdutf; the code
described here is used for what simdutf does not accept, so that errors are reported at the same position.
`JSON_NO_SIMD` does not affect simdutf.
!!! note "ABI compatibility"
The macro selects between two definitions of the library's internal scanning and formatting functions. Whether it
is defined is encoded in the [namespace](../../features/namespace.md) (tag `_ns`), resulting in distinct symbol
names. Translation units compiled with and without it can therefore be linked into the same program without One
Definition Rule (ODR) violations, but they cannot exchange instances of library types.
## Examples
??? example "Example: switching off vector instructions"
The program parses and serializes the same values with and without the macro:
```cpp
#define JSON_NO_SIMD
#include <iostream>
#include <nlohmann/json.hpp>
using json = nlohmann::json;
int main()
{
const json j = json::parse(R"({"name": "日本語 text", "pi": 3.141592653589793})");
std::cout << j.dump() << '\n'; // {"name":"日本語 text","pi":3.141592653589793}
}
```
## See also
- [**parse**](../basic_json/parse.md) - deserialize from a compatible input
- [**dump**](../basic_json/dump.md) - serialization
- [**JSON_USE_SIMDUTF**](json_use_simdutf.md) - use the simdutf library to accelerate UTF-8 validation
- [**JSON_DISABLE_STREAM_BULK_SCAN**](json_disable_stream_bulk_scan.md) - read a `std::istream` one character at a time
- [Parsing: input sources and performance](../../features/parsing/index.md#input-sources-and-performance)
## Version history
- Added in version 3.13.0.
docs/mkdocs/docs/api/macros/json_use_simdutf.md
+3 -2
View File
@@ -6,8 +6,9 @@
When defined, the parser validates the UTF-8 content of JSON strings that come from a **contiguous byte input**
(`std::string`, `std::vector<char>`/`<std::uint8_t>`, string literals, `const char*` ranges, …) using the
[simdutf](https://github.com/simdutf/simdutf) library instead of the built-in scalar validator. On text with many
non-ASCII characters (e.g. CJK or emoji) this can validate several times faster.
[simdutf](https://github.com/simdutf/simdutf) library instead of the built-in validator (which uses SSSE3 or NEON where
available, see [`JSON_NO_SIMD`](json_no_simd.md)). On text with many non-ASCII characters (e.g. CJK or emoji) this can
validate faster, depending on the CPU.
This is an **opt-in external dependency**. The library itself remains header-only and its behavior is unchanged: the
same input is accepted or rejected either way, and every parse error is reported at the same position with the same
docs/mkdocs/docs/api/macros/nlohmann_json_namespace.md
+2 -2
View File
@@ -39,6 +39,6 @@ the library.
## Version history
- Added in version 3.11.0. Changed inline namespace name in version 3.11.2. Added the ABI tag `_dp` in
version 3.12.0, and the ABI tags `_bics`, `_psp`, `_snul`, `_sbu8`, `_ekmo`, `_dsbs`, `_dufio`, `_vns`, and
`_vssse3` in version 3.13.0; see
version 3.12.0, and the ABI tags `_bics`, `_psp`, `_snul`, `_sbu8`, `_ekmo`, `_dsbs`, `_dufio`, `_ns`, `_vns`,
and `_vssse3` in version 3.13.0; see
[`nlohmann` Namespace](../../features/namespace.md#structure).
docs/mkdocs/docs/examples/no_exceptions_access.cpp
+42
View File
@@ -0,0 +1,42 @@
#include <iostream>
#include <nlohmann/json.hpp>
using json = nlohmann::json;
int main()
{
json j = json::parse(R"({"name": "Mary", "age": 42})");
// look up the key, then test the type: neither step throws
for (const char* key : {"name", "age", "city"})
{
const auto it = j.find(key);
if (it == j.end())
{
std::cout << key << ": missing\n";
}
else if (it->is_string())
{
std::cout << key << ": string \"" << it->get_ref<const std::string&>() << "\"\n";
}
else
{
std::cout << key << ": not a string, but " << it->type_name() << '\n';
}
}
// get_ptr returns a null pointer if the value has another type
const std::string* name = j["name"].get_ptr<const std::string*>();
const std::string* age = j["age"].get_ptr<const std::string*>();
std::cout << "get_ptr<string*>: name = " << (name != nullptr) << ", age = " << (age != nullptr) << '\n';
// value() is NOT exception-free: it throws if the key exists with a different type
try
{
std::cout << j.value("name", 0) << '\n';
}
catch (const json::type_error& e)
{
std::cout << e.what() << '\n';
}
}
docs/mkdocs/docs/examples/no_exceptions_access.output
+5
View File
@@ -0,0 +1,5 @@
name: string "Mary"
age: not a string, but number
city: missing
get_ptr<string*>: name = 1, age = 0
[json.exception.type_error.302] type must be number, but is string
docs/mkdocs/docs/examples/no_exceptions_convert.c++17.cpp
+46
View File
@@ -0,0 +1,46 @@
#include <iostream>
#include <optional>
#include <string>
#include <nlohmann/json.hpp>
using json = nlohmann::json;
struct person
{
std::string name;
int age;
};
NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE(person, name, age)
// Compile this function with exceptions enabled and call it from code that does not use them.
std::optional<person> to_person(const json& j)
{
try
{
return j.get<person>();
}
catch (const json::exception&)
{
return std::nullopt;
}
}
int main()
{
const json good = {{"name", "Mary"}, {"age", 42}};
const json bad = {{"name", "Mary"}, {"age", "forty-two"}};
const json incomplete = {{"name", "Mary"}};
for (const json& j : {good, bad, incomplete})
{
if (const auto p = to_person(j))
{
std::cout << "person: " << p->name << ", " << p->age << '\n';
}
else
{
std::cout << "not a person: " << j << '\n';
}
}
}
docs/mkdocs/docs/examples/no_exceptions_convert.c++17.output
+3
View File
@@ -0,0 +1,3 @@
person: Mary, 42
not a person: {"age":"forty-two","name":"Mary"}
not a person: {"name":"Mary"}
docs/mkdocs/docs/examples/no_exceptions_parse.cpp
+28
View File
@@ -0,0 +1,28 @@
#include <cstdint>
#include <iostream>
#include <vector>
#include <nlohmann/json.hpp>
using json = nlohmann::json;
// None of the calls below throws, so this example also works with -fno-exceptions.
int main()
{
const std::string invalid = R"({"name": "Mary", "age": )";
// parse() with allow_exceptions = false returns a discarded value
json j = json::parse(invalid, nullptr, false);
std::cout << std::boolalpha << "parse: discarded = " << j.is_discarded() << '\n';
// accept() only tells whether the input is valid JSON
std::cout << "accept: " << json::accept(invalid) << '\n';
// the binary formats have the same parameter
const std::vector<std::uint8_t> truncated_cbor = {0x83, 0x01};
json c = json::from_cbor(truncated_cbor, true, false);
std::cout << "from_cbor: discarded = " << c.is_discarded() << '\n';
// valid input is parsed as usual
json valid = json::parse(R"({"name": "Mary"})", nullptr, false);
std::cout << "valid input: " << valid << '\n';
}
docs/mkdocs/docs/examples/no_exceptions_parse.output
+4
View File
@@ -0,0 +1,4 @@
parse: discarded = true
accept: false
from_cbor: discarded = true
valid input: {"name":"Mary"}
docs/mkdocs/docs/examples/parse_error_position.cpp
+42
View File
@@ -0,0 +1,42 @@
#include <iostream>
#include <utility>
#include <nlohmann/json.hpp>
using json = nlohmann::json;
// line and column (both starting at 1) of a byte index in a text
std::pair<std::size_t, std::size_t> line_column(const std::string& input, std::size_t byte)
{
std::size_t line = 1;
std::size_t column = 1;
for (std::size_t i = 0; i + 1 < byte && i < input.size(); ++i)
{
if (input[i] == '\n')
{
++line;
column = 1;
}
else
{
++column;
}
}
return {line, column};
}
int main()
{
const std::string input = "{\n \"a\": 1,\n \"b\": ]\n}";
try
{
const json j = json::parse(input);
std::cout << j << '\n';
}
catch (const json::parse_error& e)
{
const auto position = line_column(input, e.byte);
std::cout << e.what() << '\n';
std::cout << "byte " << e.byte << " = line " << position.first << ", column " << position.second << '\n';
}
}
docs/mkdocs/docs/examples/parse_error_position.output
+2
View File
@@ -0,0 +1,2 @@
[json.exception.parse_error.101] parse error at line 3, column 8: syntax error while parsing value - unexpected ']'; expected '[', '{', or a literal
byte 20 = line 3, column 8
docs/mkdocs/docs/examples/parse_error_position_diagnostic.cpp
+46
View File
@@ -0,0 +1,46 @@
#define JSON_DIAGNOSTIC_POSITIONS 1
#include <iostream>
#include <utility>
#include <nlohmann/json.hpp>
using json = nlohmann::json;
// line and column (both starting at 1) of a byte index in a text
std::pair<std::size_t, std::size_t> line_column(const std::string& input, std::size_t byte)
{
std::size_t line = 1;
std::size_t column = 1;
for (std::size_t i = 0; i + 1 < byte && i < input.size(); ++i)
{
if (input[i] == '\n')
{
++line;
column = 1;
}
else
{
++column;
}
}
return {line, column};
}
int main()
{
const std::string input = "{\n \"name\": \"Mary\",\n \"age\": \"forty-two\"\n}";
const json j = json::parse(input); // the input is valid JSON ...
try
{
const int age = j["age"]; // ... but the value has the wrong type
std::cout << age << '\n';
}
catch (const json::exception& e)
{
std::cout << e.what() << '\n';
// start_pos() is a 0-based index into the input; the helper expects a 1-based byte index
const auto position = line_column(input, j["age"].start_pos() + 1);
std::cout << "the value starts at line " << position.first << ", column " << position.second << '\n';
}
}
docs/mkdocs/docs/examples/parse_error_position_diagnostic.output
+2
View File
@@ -0,0 +1,2 @@
[json.exception.type_error.302] (bytes 29-40) type must be number, but is string
the value starts at line 3, column 10
docs/mkdocs/docs/examples/partial_input_buffering.cpp
+46
View File
@@ -0,0 +1,46 @@
#include <iostream>
#include <nlohmann/json.hpp>
using json = nlohmann::json;
std::string buffer;
// call for every chunk; returns true and sets `value` once the buffer holds one complete JSON value
bool push(const std::string& chunk, json& value)
{
buffer += chunk;
try
{
value = json::parse(buffer);
buffer.clear();
return true;
}
catch (const json::parse_error& e)
{
if (e.byte == buffer.size() + 1)
{
return false; // the input ends prematurely: wait for more data
}
throw; // a real syntax error
}
}
int main()
{
json value;
for (const std::string chunk : {"{\"na", "me\": \"Ma", "ry\", \"l", "ist\": [1, 2", ", 3]}"})
{
std::cout << "chunk " << chunk << " -> " << (push(chunk, value) ? "complete" : "incomplete") << '\n';
}
std::cout << value << '\n';
// a syntax error in the middle is not mistaken for missing data
try
{
push("[1, ]", value);
}
catch (const json::parse_error& e)
{
std::cout << e.what() << '\n';
}
}
docs/mkdocs/docs/examples/partial_input_buffering.output
+7
View File
@@ -0,0 +1,7 @@
chunk {"na -> incomplete
chunk me": "Ma -> incomplete
chunk ry", "l -> incomplete
chunk ist": [1, 2 -> incomplete
chunk , 3]} -> complete
{"list":[1,2,3],"name":"Mary"}
[json.exception.parse_error.101] parse error at line 1, column 5: syntax error while parsing value - unexpected ']'; expected '[', '{', or a literal
docs/mkdocs/docs/examples/partial_input_cbor.cpp
+23
View File
@@ -0,0 +1,23 @@
#include <iostream>
#include <sstream>
#include <nlohmann/json.hpp>
using json = nlohmann::json;
int main()
{
// a CBOR value followed by other data
const std::vector<std::uint8_t> cbor = json::to_cbor(json{1, 2, 3});
std::string data(cbor.begin(), cbor.end());
data += "TRAILING";
std::istringstream input(data);
json j = json::from_cbor(input, false); // strict = false: stop after the first value
std::cout << "value: " << j << '\n';
std::cout << "bytes used: " << input.tellg() << '\n';
std::string rest;
input >> rest;
std::cout << "remaining input: " << rest << '\n';
}
docs/mkdocs/docs/examples/partial_input_cbor.output
+3
View File
@@ -0,0 +1,3 @@
value: [1,2,3]
bytes used: 4
remaining input: TRAILING
docs/mkdocs/docs/examples/partial_input_concatenated.cpp
+18
View File
@@ -0,0 +1,18 @@
#include <iostream>
#include <sstream>
#include <nlohmann/json.hpp>
using json = nlohmann::json;
int main()
{
std::istringstream input(R"({"a":1} [2,3] 4 "s")");
// check for the end of the stream before each read, because a read at the end throws
json j;
while (input >> std::ws && input.peek() != std::char_traits<char>::eof())
{
input >> j;
std::cout << j << '\n';
}
}
docs/mkdocs/docs/examples/partial_input_concatenated.output
+4
View File
@@ -0,0 +1,4 @@
{"a":1}
[2,3]
4
"s"
docs/mkdocs/docs/examples/partial_input_discard_records.cpp
+27
View File
@@ -0,0 +1,27 @@
#include <iostream>
#include <sstream>
#include <nlohmann/json.hpp>
using json = nlohmann::json;
void handle(const json& record)
{
std::cout << "handling " << record << '\n';
}
int main()
{
std::istringstream file(R"([{"id": 1}, {"id": 2}, {"id": 3}])");
json result = json::parse(file, [](int depth, json::parse_event_t event, json & record)
{
if (depth == 1 && event == json::parse_event_t::object_end)
{
handle(record);
return false; // discard the record
}
return true;
});
std::cout << "result: " << result << '\n';
}
docs/mkdocs/docs/examples/partial_input_discard_records.output
+4
View File
@@ -0,0 +1,4 @@
handling {"id":1}
handling {"id":2}
handling {"id":3}
result: []
docs/mkdocs/docs/examples/partial_input_sax_stop.cpp
+81
View File
@@ -0,0 +1,81 @@
#include <iostream>
#include <nlohmann/json.hpp>
using json = nlohmann::json;
struct find_id : nlohmann::json_sax<json>
{
bool next_is_id = false;
json::number_unsigned_t id = 0;
bool key(std::string& name) override
{
next_is_id = (name == "id");
return true;
}
bool number_unsigned(json::number_unsigned_t value) override
{
if (next_is_id)
{
id = value;
return false; // found: stop parsing
}
return true;
}
bool null() override
{
return true;
}
bool boolean(bool /*val*/) override
{
return true;
}
bool number_integer(json::number_integer_t /*val*/) override
{
return true;
}
bool number_float(json::number_float_t /*val*/, const std::string& /*s*/) override
{
return true;
}
bool string(std::string& /*val*/) override
{
return true;
}
bool binary(json::binary_t& /*val*/) override
{
return true;
}
bool start_object(std::size_t /*elements*/) override
{
return true;
}
bool end_object() override
{
return true;
}
bool start_array(std::size_t /*elements*/) override
{
return true;
}
bool end_array() override
{
return true;
}
bool parse_error(std::size_t /*position*/, const std::string& /*last_token*/, const json::exception& /*ex*/) override
{
return false;
}
};
int main()
{
find_id handler;
const bool completed = json::sax_parse(R"({"x": [1, 2, 3], "id": 42, "rest": [4, 5, 6]})", &handler);
std::cout << std::boolalpha;
std::cout << "parse completed: " << completed << '\n';
std::cout << "id: " << handler.id << '\n';
}
docs/mkdocs/docs/examples/partial_input_sax_stop.output
+2
View File
@@ -0,0 +1,2 @@
parse completed: false
id: 42
docs/mkdocs/docs/examples/type_recipes_chrono.cpp
+56
View File
@@ -0,0 +1,56 @@
#include <chrono>
#include <iostream>
#include <nlohmann/json.hpp>
using json = nlohmann::json;
namespace nlohmann
{
template <typename Rep, typename Period>
struct adl_serializer<std::chrono::duration<Rep, Period>>
{
static void to_json(json& j, const std::chrono::duration<Rep, Period>& d)
{
j = d.count();
}
static void from_json(const json& j, std::chrono::duration<Rep, Period>& d)
{
d = std::chrono::duration<Rep, Period>(j.get<Rep>());
}
};
template <typename Duration>
struct adl_serializer<std::chrono::time_point<std::chrono::system_clock, Duration>>
{
static void to_json(json& j, const std::chrono::time_point<std::chrono::system_clock, Duration>& t)
{
j = t.time_since_epoch();
}
static void from_json(const json& j, std::chrono::time_point<std::chrono::system_clock, Duration>& t)
{
t = std::chrono::time_point<std::chrono::system_clock, Duration>(j.get<Duration>());
}
};
} // namespace nlohmann
int main()
{
using namespace std::chrono;
std::cout << std::boolalpha;
// the unit is not part of the JSON value
const json timeout = milliseconds(1500);
std::cout << "duration: " << timeout << '\n';
std::cout << "as milliseconds: " << timeout.get<milliseconds>().count() << '\n';
// a system_clock time point is the time since the Unix epoch
const time_point<system_clock, milliseconds> t(milliseconds(1700000000123));
const json stamp = t;
std::cout << "time point: " << stamp << '\n';
const auto back = stamp.get<time_point<system_clock, milliseconds>>();
std::cout << "round trip: " << (back == t) << '\n';
}
docs/mkdocs/docs/examples/type_recipes_chrono.output
+4
View File
@@ -0,0 +1,4 @@
duration: 1500
as milliseconds: 1500
time point: 1700000000123
round trip: true
docs/mkdocs/docs/examples/type_recipes_class_hierarchy.c++14.cpp
+99
View File
@@ -0,0 +1,99 @@
#include <iostream>
#include <memory>
#include <stdexcept>
#include <vector>
#include <nlohmann/json.hpp>
using json = nlohmann::json;
struct shape
{
virtual ~shape() = default;
virtual json serialize() const = 0;
};
struct circle : shape
{
double radius;
explicit circle(double r) : radius(r) {}
json serialize() const override
{
return {{"type", "circle"}, {"radius", radius}};
}
};
struct rectangle : shape
{
double width;
double height;
rectangle(double w, double h) : width(w), height(h) {}
json serialize() const override
{
return {{"type", "rectangle"}, {"width", width}, {"height", height}};
}
};
namespace nlohmann
{
template <>
struct adl_serializer<std::unique_ptr<shape>>
{
static void to_json(json& j, const std::unique_ptr<shape>& p)
{
j = p ? p->serialize() : json(nullptr);
}
static void from_json(const json& j, std::unique_ptr<shape>& p)
{
if (j.is_null())
{
p.reset();
return;
}
const auto& type = j.at("type").get_ref<const std::string&>();
if (type == "circle")
{
p = std::make_unique<circle>(j.at("radius").get<double>());
}
else if (type == "rectangle")
{
p = std::make_unique<rectangle>(j.at("width").get<double>(), j.at("height").get<double>());
}
else
{
throw std::invalid_argument("unknown shape type: " + type);
}
}
};
} // namespace nlohmann
int main()
{
std::vector<std::unique_ptr<shape>> shapes;
shapes.push_back(std::make_unique<circle>(1.5));
shapes.push_back(std::make_unique<rectangle>(2, 3));
const json j = shapes;
std::cout << j << '\n';
// reading creates objects of the derived types
const auto back = j.get<std::vector<std::unique_ptr<shape>>>();
std::cout << std::boolalpha;
std::cout << "first is a circle: " << (dynamic_cast<circle*>(back[0].get()) != nullptr) << '\n';
std::cout << "second is a rectangle: " << (dynamic_cast<rectangle*>(back[1].get()) != nullptr) << '\n';
// an unknown type tag is reported
try
{
json{{"type", "star"}}.get<std::unique_ptr<shape>>();
}
catch (const std::invalid_argument& e)
{
std::cout << e.what() << '\n';
}
}
docs/mkdocs/docs/examples/type_recipes_class_hierarchy.c++14.output
+4
View File
@@ -0,0 +1,4 @@
[{"radius":1.5,"type":"circle"},{"height":3.0,"type":"rectangle","width":2.0}]
first is a circle: true
second is a rectangle: true
unknown shape type: star
docs/mkdocs/docs/examples/type_recipes_smart_pointers.c++14.cpp
+90
View File
@@ -0,0 +1,90 @@
#include <iostream>
#include <memory>
#include <vector>
#include <nlohmann/json.hpp>
using json = nlohmann::json;
namespace nlohmann
{
template <typename T>
struct adl_serializer<std::shared_ptr<T>>
{
static void to_json(json& j, const std::shared_ptr<T>& p)
{
if (p)
{
j = *p;
}
else
{
j = nullptr;
}
}
static void from_json(const json& j, std::shared_ptr<T>& p)
{
if (j.is_null())
{
p = nullptr;
}
else
{
p = std::make_shared<T>(j.get<typename std::remove_const<T>::type>());
}
}
};
template <typename T>
struct adl_serializer<std::unique_ptr<T>>
{
static void to_json(json& j, const std::unique_ptr<T>& p)
{
if (p)
{
j = *p;
}
else
{
j = nullptr;
}
}
static void from_json(const json& j, std::unique_ptr<T>& p)
{
if (j.is_null())
{
p = nullptr;
}
else
{
p = std::make_unique<T>(j.get<T>());
}
}
};
} // namespace nlohmann
struct point
{
int x;
int y;
};
NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE(point, x, y)
int main()
{
// an empty pointer is written as null
const std::vector<std::shared_ptr<point>> points = {std::make_shared<point>(point{1, 2}), nullptr};
const json j = points;
std::cout << j << '\n';
// reading allocates new objects
const auto back = j.get<std::vector<std::shared_ptr<point>>>();
std::cout << back[0]->x << ", " << back[0]->y << ", " << (back[1] == nullptr) << '\n';
// pointers to const and unique_ptr work as well
const auto const_ptr = j[0].get<std::shared_ptr<const point>>();
const auto unique = j[0].get<std::unique_ptr<point>>();
std::cout << const_ptr->y << ", " << unique->x << '\n';
}
docs/mkdocs/docs/examples/type_recipes_smart_pointers.c++14.output
+3
View File
@@ -0,0 +1,3 @@
[{"x":1,"y":2},null]
1, 2, 1
2, 1
docs/mkdocs/docs/examples/type_recipes_variant.c++17.cpp
+63
View File
@@ -0,0 +1,63 @@
#include <iostream>
#include <stdexcept>
#include <variant>
#include <vector>
#include <nlohmann/json.hpp>
using json = nlohmann::json;
namespace geo
{
struct circ
{
double radius;
};
struct rect
{
double width;
double height;
};
NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE(circ, radius)
NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE(rect, width, height)
using figure = std::variant<circ, rect>;
void to_json(json& j, const figure& f)
{
std::visit([&j](const auto & alternative)
{
j = alternative;
}, f);
j["type"] = std::holds_alternative<circ>(f) ? "circ" : "rect";
}
void from_json(const json& j, figure& f)
{
const auto& type = j.at("type").get_ref<const std::string&>();
if (type == "circ")
{
f = j.get<circ>();
}
else if (type == "rect")
{
f = j.get<rect>();
}
else
{
throw std::invalid_argument("unknown figure type: " + type);
}
}
} // namespace geo
int main()
{
const std::vector<geo::figure> figures = {geo::circ{2}, geo::rect{3, 4}};
const json j = figures;
std::cout << j << '\n';
const auto back = j.get<std::vector<geo::figure>>();
std::cout << "alternatives read back: " << back[0].index() << ", " << back[1].index() << '\n';
}
docs/mkdocs/docs/examples/type_recipes_variant.c++17.output
+2
View File
@@ -0,0 +1,2 @@
[{"radius":2.0,"type":"circ"},{"height":4.0,"type":"rect","width":3.0}]
alternatives read back: 0, 1
docs/mkdocs/docs/features/arbitrary_types.md
+27
View File
@@ -409,6 +409,32 @@ namespace nlohmann {
}
```
## Why isn't my `to_json` or `from_json` found?
The library calls `to_json` and `from_json` without qualification, so the compiler finds them by
[argument-dependent lookup](https://en.cppreference.com/w/cpp/language/adl) in the namespace of the type, or in the
namespaces of its template arguments. The usual reasons for a function that is "not found" are:
- **The type is an alias of a standard type.** After `#!cpp using Hash = std::array<std::uint8_t, 6>;` the type
*is* `std::array<std::uint8_t, 6>`, which lives in namespace `std`. A `to_json(json&, const Hash&)` in your namespace
is never selected. Either give the type its own identity, for example `#!cpp struct Hash : std::array<std::uint8_t, 6> {};`
(the function is then found in `Hash`'s namespace), or specialize
[`adl_serializer<Hash>`](#how-do-i-convert-third-party-types).
- **The function is in the wrong namespace.** It must be in the namespace of the type (a global type needs a global
function), not in the namespace of the code that calls the conversion. Specializations of `adl_serializer` belong in
namespace `nlohmann`.
- **It is declared too late.** A conversion can only use what has been declared when it is instantiated. If
`#!cpp json j = std::vector<third_party::string>{...};` appears before the `adl_serializer` specialization, the
compiler reports an error such as "explicit specialization of `adl_serializer<...>` after instantiation". Put the
specialization in a small header that every file includes **before** it uses the type with `json`.
- **It is a template.** For `template <class T> struct box`, write `template <class T> void to_json(json&, const box<T>&)`,
not an overload for one `T`.
- **It worked by accident.** Some versions found `to_json` overloads in `nlohmann::detail` through argument-dependent
lookup of the base class of `basic_json`. Do not rely on this; it is not part of the interface.
For smart pointers, class hierarchies, `std::variant`, and `std::chrono`, see
[Recipes for common types](type_recipes.md).
## Why can't I convert to/from `std::any`?
`std::any` is intentionally excluded from `get<T>()`/generic conversion support, so `get<std::any>()` and
@@ -505,6 +531,7 @@ struct bad_serializer
## See also
- [Recipes for common types](type_recipes.md) - smart pointers, class hierarchies, `std::variant`, and `std::chrono`
- [Converting values](conversions.md) - the general overview of `get`/`get_to` and implicit conversions
- [Specializing enum conversion](enum_conversion.md) - map enums to JSON strings instead of integers
- [Supported macros](macros.md) - reference for `NLOHMANN_DEFINE_TYPE_*` and related macros
docs/mkdocs/docs/features/assertions.md
+3 -1
View File
@@ -7,7 +7,9 @@ detect bugs in client code.
## Switch off runtime assertions
Runtime assertions can be switched off by defining the preprocessor macro `NDEBUG` (see the
[documentation of assert](https://en.cppreference.com/w/cpp/error/assert)) which is the default for release builds.
[documentation of assert](https://en.cppreference.com/w/cpp/error/assert)) which is the default for release builds. Without `NDEBUG`, the assertions are compiled in and contain the name of the source
file (`__FILE__`); see [notes for builds and distribution](../integration/index.md#notes-for-builds-and-distribution)
if that path must not end up in your binaries.
## Change assertion behavior
docs/mkdocs/docs/features/conversions.md
+23
View File
@@ -112,6 +112,29 @@ Implicit conversions are convenient but can be surprising (for example, in overl
be disabled by defining [`JSON_USE_IMPLICIT_CONVERSIONS`](../api/macros/json_use_implicit_conversions.md) to `#!cpp 0`,
which forces the explicit `get` form and can catch unintended conversions at compile time.
!!! warning "`auto` copies the JSON value; it does not convert it"
The type of `#!cpp j["name"]` is `json`, not the C++ type of the value inside. `#!cpp auto` therefore deduces
`json`:
```cpp
json j = {{"name", "Mary"}, {"age", 42}};
auto name = j["name"]; // json: a copy of the value; prints as "Mary" (with quotes)
std::string s1 = j["name"]; // std::string: implicit conversion
auto s2 = j["name"].get<std::string>(); // std::string: explicit conversion
auto& age = j["age"]; // json&: a reference, no copy
age = 43; // changes j
```
This is also why `#!cpp std::cout << j["name"]` prints the quotes (see
[getting a string without quotes](#getting-values-out)), and why a `json` value cannot be passed to `printf`: the
argument is not an `int` or a `char*`, and a compiler either rejects passing it through `...` (Clang does) or
compiles it and prints garbage. Convert first: `#!cpp std::printf("%d", j["age"].get<int>());`.
Prefer `get<T>()` over `auto` when you want the C++ value, and `auto&` or `const auto&` when you want to avoid a
copy of a large value.
!!! warning "Conversions do not range-check numbers"
Just like C++ itself, the `get` family performs numeric conversions without range checks — retrieving a
docs/mkdocs/docs/features/creating_values.md
+19
View File
@@ -59,6 +59,25 @@ with a string as the first element is treated as an object, everything else as a
[FAQ](../home/faq.md#brace-initialization-yields-arrays) for details and the opt-in
[`JSON_BRACE_INIT_COPY_SEMANTICS`](../api/macros/json_brace_init_copy_semantics.md) macro.
### What does this produce?
The following table shows how some common spellings are interpreted. When in doubt, use `#!cpp json::array()`,
`#!cpp json::object()`, or the `#!cpp json(value)` form with parentheses.
| code | result | note |
|-------------------------------------------------|------------------------------|----------------------------------------------|
| `#!cpp json j = {};` / `#!cpp json j{};` | `#!json null` | use `json::object()` or `json::array()` |
| `#!cpp json j = json::object();` | `#!json {}` | |
| `#!cpp json j = json::array();` | `#!json []` | |
| `#!cpp json j(true);` | `#!json true` | parentheses create the value itself |
| `#!cpp json j = {true};` | `#!json [true]` | braces create a one-element array (`#!cpp json j{true};` depends on the compiler, see the [FAQ](../home/faq.md#brace-initialization-yields-arrays)) |
| `#!cpp json j = {{"a", 1}};` | `#!json {"a":1}` | all elements are `[string, value]` pairs: object |
| `#!cpp json j = {{"a", 1}, {"b"}};` | `#!json [["a",1],["b"]]` | one element is not a pair: array |
| `#!cpp json j = {{"a", 1}, 2};` | `#!json [["a",1],2]` | a plain value among pairs: array |
| `#!cpp json j = {{"a", 1, 2}};` | `#!json [["a",1,2]]` | a three-element list is not a pair: array |
| `#!cpp json j = "{\"a\":1}";` | `#!json "{\"a\":1}"` | a string **value**; it is not parsed |
| `#!cpp json j = json::parse("{\"a\":1}");` | `#!json {"a":1}` | parse text with `parse` or `_json` |
## Building incrementally
A value can also be built up piece by piece. Accessing a non-existing object key or array index with
docs/mkdocs/docs/features/element_access/index.md
+39
View File
@@ -21,3 +21,42 @@ flowchart TD
A --> F{"just testing first?"}
F -->|"yes"| G["contains() / find()"]
```
## Does this key exist?
Use [`contains`](../../api/basic_json/contains.md) to test for a key; it does not modify the value and does not throw:
```cpp
json j = {{"name", "Mary"}, {"address", {{"city", "Berlin"}}}};
j.contains("name"); // true
j.contains("age"); // false
j.contains("city"); // false: contains() only looks at the top level
j["address"].contains("city"); // true
j.contains(json::json_pointer("/address/city")); // true: a JSON pointer tests a nested path in one call
```
If the value is needed afterwards, [`find`](../../api/basic_json/find.md) tests and accesses with a single lookup:
```cpp
if (auto it = j.find("name"); it != j.end())
{
std::cout << *it << '\n';
}
```
`contains` also returns `true` for a key whose value is `null`; use `j.contains("k") && !j["k"].is_null()` if a
`null` should count as missing. To read a value with a fallback instead, see [`value`](default_value.md).
!!! warning "`operator[]` inserts missing keys"
Like `#!cpp std::map::operator[]`, the non-const [`operator[]`](unchecked_access.md) adds a `null` value if the key
is missing. Testing a key with it therefore *changes the object*:
```cpp
json j = {{"foo", 1}};
bool absent = j["bar"].is_null(); // true, but j is now {"bar": null, "foo": 1}
```
Use `contains`/`find` to test, and [`at`](checked_access.md) to access a value that must exist (it throws an
[`out_of_range`](../../home/exceptions.md#out-of-range) exception otherwise).
docs/mkdocs/docs/features/macros.md
+7
View File
@@ -174,6 +174,13 @@ security reasons (e.g., Intel Software Guard Extensions (SGX)).
See [full documentation of `JSON_NO_IO`](../api/macros/json_no_io.md).
## `JSON_NO_SIMD`
When defined, the library uses only portable code instead of vector instructions (SSE2, SSSE3, NEON) for scanning
strings and writing floating-point numbers. Results are the same either way. It is not defined by default.
See [full documentation of `JSON_NO_SIMD`](../api/macros/json_no_simd.md).
## `JSON_NO_THREAD_LOCAL`
When defined, the library does not use `#!cpp thread_local` storage. Copying a value and comparing two values then
docs/mkdocs/docs/features/namespace.md
+3 -1
View File
@@ -27,6 +27,7 @@ The complete default namespace name is derived as follows:
`_dsbs`.
- [`JSON_DISABLE_UNLOCKED_FILE_IO`](../api/macros/json_disable_unlocked_file_io.md) defined non-zero appends
`_dufio`.
- [`JSON_NO_SIMD`](../api/macros/json_no_simd.md) defined appends `_ns`.
- [`JSON_VIEW_NO_SIMD`](../api/macros/json_view_no_simd.md) defined appends `_vns`.
- [`JSON_VIEW_USE_SSSE3`](../api/macros/json_view_use_ssse3.md) defined appends `_vssse3`.
- The inline namespace ends with the suffix `_v` followed by the 3 components of the version number separated by
@@ -114,6 +115,7 @@ follows:
([`JSON_STRICT_BINARY_UTF8`](../api/macros/json_strict_binary_utf8.md)), `_ekmo`
([`JSON_USE_OBJECTS_FOR_ENUM_KEYED_MAPS`](../api/macros/json_use_objects_for_enum_keyed_maps.md)), `_dsbs`
([`JSON_DISABLE_STREAM_BULK_SCAN`](../api/macros/json_disable_stream_bulk_scan.md)), `_dufio`
([`JSON_DISABLE_UNLOCKED_FILE_IO`](../api/macros/json_disable_unlocked_file_io.md)), `_vns`
([`JSON_DISABLE_UNLOCKED_FILE_IO`](../api/macros/json_disable_unlocked_file_io.md)), `_ns`
([`JSON_NO_SIMD`](../api/macros/json_no_simd.md)), `_vns`
([`JSON_VIEW_NO_SIMD`](../api/macros/json_view_no_simd.md)), and `_vssse3`
([`JSON_VIEW_USE_SSSE3`](../api/macros/json_view_use_ssse3.md)) in version 3.13.0.
docs/mkdocs/docs/features/no_exceptions.md
+113
View File
@@ -0,0 +1,113 @@
# Working Without Exceptions
By default, the library reports errors by throwing exceptions (see [Exceptions](../home/exceptions.md)). This page
collects what to use when exceptions are unwanted or unavailable, for example in embedded systems, game engines, or
code compiled with `-fno-exceptions`.
## Two different questions
1. **Can my code avoid *catching* exceptions?** Yes. For parsing and for accessing values there are functions that
report failure through a return value. They work in a normal build and need no `#!cpp try`/`#!cpp catch`.
2. **Can the library be *compiled* without exceptions?** Yes, with `-fno-exceptions` or by defining
[`JSON_NOEXCEPTION`](../api/macros/json_noexception.md). Both have the same effect: every `#!cpp throw` in the
library becomes a call to `std::abort()`. The functions listed below never reach a `#!cpp throw` for the failures
they report, so they keep working in this configuration. Anything else that fails aborts the program.
## Parsing
| function | on invalid input |
|---------------------------------------------------------------------------------------------|-----------------------------------------------------|
| `#!cpp json::parse(input, nullptr, false)` | returns a *discarded* value; test with `is_discarded()` |
| [`accept`](../api/basic_json/accept.md) | returns `#!cpp false` |
| `#!cpp json::from_cbor(input, true, false)` (and the other `from_*` functions) | returns a *discarded* value |
| [`sax_parse`](../api/basic_json/sax_parse.md) with your own handler | returns `#!cpp false` after calling `parse_error` |
??? example "Example: parse and validate without exceptions"
```cpp
--8<-- "examples/no_exceptions_parse.cpp"
```
Output:
```
--8<-- "examples/no_exceptions_parse.output"
```
None of the calls in this example throws, so it also compiles and runs with `-fno-exceptions`.
The discarded value carries no message. If you need the position or the reason, use the SAX interface; see
[Parsing and exceptions](parsing/parse_exceptions.md).
## Accessing values
The access functions are not equally safe. Only some of them never throw:
| expression | if the key is missing or the type is wrong |
|-------------------------------------|-------------------------------------------------------------------------------|
| `#!cpp j.contains("k")` | returns `#!cpp false` for a missing key; never throws |
| `#!cpp j.find("k")` | returns `end()` for a missing key; never throws |
| `#!cpp j.is_string()`, ... | the `is_*` functions test the type; never throw |
| `#!cpp j.get_ptr<const std::string*>()` | returns `#!cpp nullptr` for a different type; never throws |
| `#!cpp j.at("k")` | throws `out_of_range.403` for a missing key |
| `#!cpp j.get<int>()` | throws `type_error.302` for a value that is not a number |
| `#!cpp j.value("k", 0)` | returns the default for a missing key, but **throws `type_error.302` if `k` exists with another type** |
| `#!cpp j["k"]` (non-const) | inserts `null` for a missing key |
| `#!cpp j["k"]` (const) | undefined behavior for a missing key |
So reading a value that may be absent or of the wrong type without exceptions takes two steps: look up, then test the
type.
??? example "Example: look up a key and test its type"
```cpp
--8<-- "examples/no_exceptions_access.cpp"
```
Output:
```
--8<-- "examples/no_exceptions_access.output"
```
The last lines of the example show why `value` is not on the safe list: it returns the default only for a *missing*
key.
For numbers, test with `is_number()` (or `is_number_integer()`, `is_number_unsigned()`, `is_number_float()`) before
calling `get<int>()`, and see [number handling](types/number_handling.md) for range and precision. Whether a value
fits into the target type is not checked by `get`.
## Converting to your own types
`from_json` functions (including the ones generated by the
[`NLOHMANN_DEFINE_TYPE_*` macros](arbitrary_types.md#simplify-your-life-with-macros)) report problems by throwing.
There is no variant that returns an error code. Without exceptions you have two options:
- Validate the JSON value yourself, as shown above, and fill your struct field by field.
- Convert in a function that is compiled **with** exceptions enabled and catches them at the boundary, returning
`std::optional` or an error code to the rest of your code:
??? example "Example: convert to a struct and report failure as std::optional"
```cpp
--8<-- "examples/no_exceptions_convert.c++17.cpp"
```
Output:
```
--8<-- "examples/no_exceptions_convert.c++17.output"
```
## Overriding what happens instead of an exception
With exceptions off, you can replace the `abort()` by your own code, for example to log the message first. See
[Switch off exceptions](../home/exceptions.md#switch-off-exceptions) and
[`JSON_THROW_USER`](../api/macros/json_throw_user.md). The handler must not return normally.
## See also
- [Exceptions](../home/exceptions.md) - all exceptions and their ids
- [Parsing and exceptions](parsing/parse_exceptions.md) - the parsing alternatives in detail
- [Parsing untrusted input](parsing/untrusted_input.md)
- [`JSON_NOEXCEPTION`](../api/macros/json_noexception.md)
docs/mkdocs/docs/features/parsing/index.md
+17 -2
View File
@@ -33,7 +33,9 @@ json data = json::parse(f);
```
The input must be encoded in UTF-8; other encodings are not supported. A single input may contain only one JSON value.
Inputs consisting of multiple values separated by newlines are handled by the [JSON Lines](json_lines.md) format.
Inputs consisting of multiple values separated by newlines are handled by the [JSON Lines](json_lines.md) format; input
that arrives in chunks or contains several values is covered in
[Large, streamed, and partial input](large_and_partial_input.md).
By default, the library rejects comments and trailing commas. Both can be enabled with parameters of the `parse`
function — see [comments](../comments.md) and [trailing commas](../trailing_commas.md).
@@ -79,9 +81,13 @@ follow the usual `std::streambuf` contract, and
[`JSON_DISABLE_UNLOCKED_FILE_IO`](../../api/macros/json_disable_unlocked_file_io.md) for programs in which another
thread must use the same `#!cpp FILE` object while it is parsed.
Where the target has them, strings are scanned with vector instructions: SSE2 on x86-64 (and SSSE3 for UTF-8 text, when
the CPU has it) and NEON on AArch64. [`JSON_NO_SIMD`](../../api/macros/json_no_simd.md) selects portable code instead,
again without changing any result.
## SAX vs. DOM parsing
The library offers two parsing models:
The library offers two parsing models, and a third, read-only alternative:
- **DOM parsing** (the default): the complete input is read and stored as an in-memory `basic_json` value that can be
traversed and modified freely. This is what [`parse`](../../api/basic_json/parse.md) does, and it is the right choice
@@ -91,6 +97,13 @@ The library offers two parsing models:
inputs or when you only need to extract parts of the input. See the [SAX interface](sax_interface.md) for details and
[`sax_parse`](../../api/basic_json/sax_parse.md) for the API.
- **Document and views** (`#!cpp <nlohmann/json_view.hpp>`, added in version 3.13.0): [`json_document::parse`](../../api/basic_json_document/parse.md)
reads the same JSON grammar but builds a flat, read-only index of the values instead of a `basic_json` tree. Strings and
floating-point numbers are read from the text on demand, and only the parts you
[`materialize()`](../../api/basic_json_view/materialize.md) become `basic_json` values. This avoids most of the work when you
need only a few values from a large or frequently parsed text; see [Zero-copy JSON views](../json_view.md). It is read-only, so use
`parse` when you need to modify the value.
You can influence a DOM parse without switching to the SAX interface by passing a
[parser callback](parser_callbacks.md), which is called during parsing and can, for example, discard parts of the input.
@@ -107,7 +120,9 @@ options.
- [`accept`](../../api/basic_json/accept.md) - check if the input is valid JSON
- [`sax_parse`](../../api/basic_json/sax_parse.md) - generate SAX events
- [JSON Lines](json_lines.md) - parse newline-delimited JSON
- [Large, streamed, and partial input](large_and_partial_input.md) - chunked, concatenated, and very large input
- [parser callbacks](parser_callbacks.md) - influence the parsing by a callback function
- [SAX interface](sax_interface.md) - implement a custom SAX handler
- [Zero-copy JSON views](../json_view.md) - parse into a read-only document and read it through views
- [parsing and exceptions](parse_exceptions.md) - control error handling
- [parsing untrusted input](untrusted_input.md) - what to consider when parsing input from untrusted sources
docs/mkdocs/docs/features/parsing/large_and_partial_input.md
+154
View File
@@ -0,0 +1,154 @@
# Large, Streamed, and Partial Input
[`parse`](../../api/basic_json/parse.md) reads exactly one complete JSON value and builds it in memory. It cannot be
paused and resumed, and it does not hand back a half-parsed value when the input ends early. This page shows what to do
when the input does not fit that model: it arrives in pieces, it contains several values, or it is too large to hold
as a `json` value.
```mermaid
flowchart TD
A["my input is..."] --> B{"several JSON values in one stream?"}
B -->|"yes"| C["read them one at a time"]
A --> D{"arriving in chunks (socket, pipe)?"}
D -->|"yes"| E["frame the messages, or buffer until complete"]
A --> F{"too large to keep in memory?"}
F -->|"yes"| G["discard parts with a callback, or use SAX"]
A --> H{"only a small part is needed?"}
H -->|"yes"| I["stop early with SAX, or read through a view"]
```
## Several values in one stream
For newline-delimited input, read line by line and parse each line, as described in [JSON Lines](json_lines.md). For
values that are simply concatenated, [`operator>>`](../../api/operator_gtgt.md) reads one value and leaves the stream
positioned after it. Check for the end of the stream before each read, because a read at the end throws:
??? example "Example: read concatenated values"
```cpp
--8<-- "examples/partial_input_concatenated.cpp"
```
Output:
```
--8<-- "examples/partial_input_concatenated.output"
```
The same works for binary formats. [`from_cbor`](../../api/basic_json/from_cbor.md) and its relatives accept
`#!cpp strict = false` to stop after the first value and leave the rest of the stream untouched; `tellg()` then reports
how many bytes the value took:
??? example "Example: read one CBOR value and find out how many bytes it used"
```cpp
--8<-- "examples/partial_input_cbor.cpp"
```
Output:
```
--8<-- "examples/partial_input_cbor.output"
```
## Input arriving in chunks
A socket, pipe, or HTTP body hands over data in pieces of arbitrary size, and a piece can end in the middle of a
number, a string, or an object. The parser does not keep state between calls, so there are two ways to deal with this.
**Frame the messages.** The most robust approach is a protocol that says where a value ends: one value per line
([JSON Lines](json_lines.md)), a length prefix, or a delimiter. Collect the bytes of one message and parse them when
the frame is complete. Prefer this whenever you control the protocol.
**Buffer until the value is complete.** If the data has no framing, append every chunk to a buffer and try to parse
it. A parse error at the very end of the buffer means that the input ended prematurely, while an error anywhere else is
a real syntax error. The member `byte` of the [`parse_error`](../../home/exceptions.md#jsonexceptionparse_error101)
exception is one past the buffer size in the first case:
??? example "Example: buffer chunks until the value is complete"
```cpp
--8<-- "examples/partial_input_buffering.cpp"
```
Output:
```
--8<-- "examples/partial_input_buffering.output"
```
The example feeds a document in five chunks. The first four are reported as incomplete, and a syntax error is reported as
such, not as missing data.
!!! warning "Limits of buffering"
- The buffer must contain exactly one value. Data that belongs to the next value is a syntax error. Use framing if
values follow each other without a separator.
- A number at the end of the buffer is a complete value, even if more digits are still to come (`12` may be the
start of `123`). Buffering cannot detect this; use framing for top-level numbers.
- Every attempt parses the buffer from the beginning, so the total work grows quadratically with the size of the
value. Limit the size of the buffer for input you do not trust (see [untrusted input](untrusted_input.md)).
## Input too large for memory
The input does not have to become one huge `json` value. The two tools below process it piece by piece.
### Handle and discard elements with a callback
A [parser callback](parser_callbacks.md) is called for every finished value. If it returns `#!cpp false`, the value is
discarded and its memory is released at once. For a file that consists of one huge array (or one huge object) of
records, handle each record at depth 1 and discard it:
??? example "Example: handle and discard each record of a large array"
```cpp
--8<-- "examples/partial_input_discard_records.cpp"
```
Output:
```
--8<-- "examples/partial_input_discard_records.output"
```
Only one record is in memory at any time. The same recipe, with more detail, is in the
[parser callbacks](parser_callbacks.md#recipe-streaming-a-large-homogeneous-array) page. To skip entire members you do not
need, discard them when their key is read; see the [example](parser_callbacks.md) for `parse_event_t::key`.
### Extract a few values and stop with SAX
With the [SAX interface](sax_interface.md), no value is built at all, and **returning `#!cpp false` from any handler
function stops the parser**. [`sax_parse`](../../api/basic_json/sax_parse.md) then returns `#!cpp false`, and the rest
of the input is not read. The example finds the first `id` in a document without parsing the remainder:
??? example "Example: stop parsing as soon as a value is found"
```cpp
--8<-- "examples/partial_input_sax_stop.cpp"
```
Output:
```
--8<-- "examples/partial_input_sax_stop.output"
```
Note that `sax_parse` returns `#!cpp false` both when a handler stopped the parse and when the input is invalid, so
record the reason in the handler (here: `id`) if you need to tell the two apart.
### Read parts of a large document with a view
If the whole text is available and you only need to read some of it, a [`json_document`](../json_view.md) from
`<nlohmann/json_view.hpp>` (added in version 3.13.0) avoids the `basic_json` tree altogether: it keeps the text, builds a flat index of the values,
and gives read-only [views](../../api/basic_json_view/index.md) of the parts you look at. Only what you
[`materialize()`](../../api/basic_json_view/materialize.md) becomes a `basic_json` value. See
[Zero-copy JSON views](../json_view.md) for details and for the lifetime rules.
## See also
- [Parsing](index.md) - overview of the parsing functions
- [JSON Lines](json_lines.md) - newline-delimited JSON
- [Parser callbacks](parser_callbacks.md) - discard values while parsing
- [SAX interface](sax_interface.md) - event-based parsing
- [Zero-copy JSON views](../json_view.md) - read parts of a large document without building a tree
- [Untrusted input](untrusted_input.md) - limits for input you do not control
docs/mkdocs/docs/features/parsing/parse_exceptions.md
+62
View File
@@ -21,6 +21,10 @@ catch (json::parse_error& ex)
In case exceptions are undesired or not supported by the environment, there are different ways to proceed:
!!! tip
[Working without exceptions](../no_exceptions.md) also covers accessing values and converting to your own types.
## Switch off exceptions
The [`parse()`](../../api/basic_json/parse.md) function accepts a `#!cpp bool` parameter `allow_exceptions` which
@@ -81,3 +85,61 @@ The return value indicates whether the parsing should continue, so the function
```
--8<-- "examples/sax_no_exception.output"
```
## Where did the error occur?
The message of a [`parse_error`](../../home/exceptions.md#jsonexceptionparse_error101) for JSON text contains the line
and column of the error:
```
[json.exception.parse_error.101] parse error at line 3, column 8: syntax error while parsing value - unexpected ']'; expected '[', '{', or a literal
```
The exception object itself only stores the **byte index** of the last read character in the member `byte` (the first
byte of the input has index 1). The same index is passed to `parse_error` in the [SAX interface](sax_interface.md). If
you need the line and column as numbers, for example to show them in an editor, compute them from the input:
??? example "Example: compute line and column of a parse error"
```cpp
--8<-- "examples/parse_error_position.cpp"
```
Output:
```
--8<-- "examples/parse_error_position.output"
```
Columns count bytes, so a line with multi-byte UTF-8 characters before the error may differ from the column your editor
shows.
### Where is the value that caused a later error?
A `parse_error` is raised while the text is read. Many mistakes show up later: the text is valid JSON, but a value has
the wrong type when you convert it (`type_error.302`), or a key is missing (`out_of_range.403`). The exception then says
what is wrong, but not where in the input the value was. Define
[`JSON_DIAGNOSTIC_POSITIONS`](../../api/macros/json_diagnostic_positions.md) before including the library to add the
byte range of the offending value to such messages (since version 3.12.0), and to query the range of any value with
[`start_pos()`](../../api/basic_json/start_pos.md) and [`end_pos()`](../../api/basic_json/end_pos.md):
??? example "Example: locate a value with diagnostic positions"
```cpp
--8<-- "examples/parse_error_position_diagnostic.cpp"
```
Output:
```
--8<-- "examples/parse_error_position_diagnostic.output"
```
Positions are only available for values created by [`parse`](../../api/basic_json/parse.md), they are not updated when a
value is changed, and the macro makes every value 16 bytes larger on 64-bit platforms, so use it where the cost is
acceptable, for example for configuration files. Note that `start_pos()` is a 0-based index into the input, whereas
`byte` of a `parse_error` starts at 1.
For binary formats such as CBOR, there are no lines; `byte` is the position in the byte sequence. To find out how many
bytes a successfully parsed value used, see
[Several values in one stream](large_and_partial_input.md#several-values-in-one-stream).
docs/mkdocs/docs/features/parsing/untrusted_input.md
+5 -3
View File
@@ -26,9 +26,11 @@ several alternatives; see [Parsing and exceptions](parse_exceptions.md) for the
[example](parse_exceptions.md#user-defined-sax-interface) that overrides it to print instead of throw.
If exceptions are unavailable entirely (`-fno-exceptions`, or [`JSON_NOEXCEPTION`](../../api/macros/json_noexception.md)
defined), every `#!cpp throw` in the library becomes a call to `std::abort()` -- there is no way to recover from a
parse error of untrusted input in that configuration; see
[Switch off exceptions](../../home/exceptions.md#switch-off-exceptions) for the details and for overriding this with
defined), every `#!cpp throw` in the library becomes a call to `std::abort()`. The alternatives above do not throw, so
`parse()` with `allow_exceptions` set to `#!cpp false` (and likewise `accept()` and the `from_*` functions with
`allow_exceptions` set to `#!cpp false`) can still reject invalid input in that configuration; a plain `parse()` call
aborts on invalid input. See [Working without exceptions](../no_exceptions.md) and
[Switch off exceptions](../../home/exceptions.md#switch-off-exceptions) for the details and for overriding the abort with
`JSON_THROW_USER`.
## Nesting depth
docs/mkdocs/docs/features/performance.md
+52 -20
View File
@@ -20,32 +20,39 @@ below apply on every supported standard. This adapter exposes two optimizations
- it can reconstruct already-consumed input on demand for error messages, instead of copying every character as it is
read, and
- the lexer can scan ordinary string characters directly out of the buffer, several bytes at a time, rather than one
character (and one function call) at a time.
- the lexer can scan ordinary string characters, numbers, and indentation directly out of the buffer, several bytes at
a time, rather than one character (and one function call) at a time.
A `#!cpp std::istream` (including `#!cpp std::ifstream`) or `#!cpp FILE*`, by contrast, is read through
`input_stream_adapter` or `file_input_adapter`, which read one character (or one block, for binary formats) at a time
and expose neither optimization -- the lexer falls back to the same byte-at-a-time path it uses for any
non-contiguous, general-purpose iterator range. An iterator pair over non-contiguous but random-access storage (e.g.
`#!cpp std::deque<char>::iterator`) gets the first optimization but not the second, since the byte-scanning fast path
additionally requires contiguous storage.
A `#!cpp std::istream` (including `#!cpp std::ifstream`) gets the same bulk scanning: `input_stream_adapter` scans the
characters its stream buffer already holds (the *get area*) and asks the stream buffer for more only when they are used
up. Nothing is read ahead, so the stream is left exactly where it would be after reading one character at a time, and
[`operator>>`](../api/operator_gtgt.md) can still read several values from one stream. A stream buffer without a get
area is read one character at a time. A `#!cpp FILE*` is still read one character at a time, but `file_input_adapter`
locks the file once per parse instead of once per character. See
[input sources and performance](parsing/index.md#input-sources-and-performance) for the full table. According to the
description of [#5828](https://github.com/nlohmann/json/pull/5828), parsing from a stream is within about 10 to 20 % of
parsing an equivalent `#!cpp std::string`.
Practically: if the JSON text is already in memory, or small enough to read into memory, prefer passing a
`#!cpp std::string`, a `#!cpp std::vector<char>`, or a pointer range to `parse` over a `#!cpp std::istream`. For a
file, that means reading it into a string first and then parsing the string, rather than passing a
`#!cpp std::ifstream` directly to `parse` -- the latter never benefits from either optimization:
Inputs that are still read one character at a time: iterator ranges that are not contiguous (e.g.
`#!cpp std::list<char>` or `#!cpp std::istreambuf_iterator<char>`), wide strings, and user-defined adapters. An
iterator pair over non-contiguous but random-access storage (e.g. `#!cpp std::deque<char>::iterator`) gets the first
optimization but not the second, since the byte-scanning fast path requires contiguous storage.
Practically: a file can be passed directly to `parse`, which also avoids the memory for a copy of the file. Reading it
into a string first is no longer necessary for speed:
```cpp
// gets the contiguous fast paths
// fast: the stream buffer's get area is scanned in bulk
std::ifstream f("example.json");
std::string contents((std::istreambuf_iterator<char>(f)), std::istreambuf_iterator<char>());
json j = json::parse(contents);
// does not: input_stream_adapter has no fast path
std::ifstream f2("example.json");
json j2 = json::parse(f2);
json j = json::parse(f);
```
Two macros switch the stream optimizations off without changing any result:
[`JSON_DISABLE_STREAM_BULK_SCAN`](../api/macros/json_disable_stream_bulk_scan.md) for stream buffers that do not follow
the usual `#!cpp std::streambuf` contract, and
[`JSON_DISABLE_UNLOCKED_FILE_IO`](../api/macros/json_disable_unlocked_file_io.md) for programs in which another thread
must use the same `#!cpp FILE` object while it is parsed.
For contiguous input with many non-ASCII characters, [`JSON_USE_SIMDUTF`](../api/macros/json_use_simdutf.md) can
additionally speed up UTF-8 validation by using the [simdutf](https://github.com/simdutf/simdutf) library instead of
the built-in scalar validator; streaming inputs (files, `#!cpp std::istream`, wide strings, user-defined adapters)
@@ -115,7 +122,9 @@ an adapter, since their template argument order does not match what `basic_json`
[linear in its size](../api/basic_json/basic_json.md#complexity) for the copy constructor but
[constant](../api/basic_json/basic_json.md#complexity) for the move constructor. The same applies to assigning a
large `#!cpp std::string`, `#!cpp std::vector`, or other container into a value: pass it as `#!cpp std::move(x)`
rather than `x` whenever `x` is no longer needed afterwards.
rather than `x` whenever `x` is no longer needed afterwards. Since version 3.13.0, converting an rvalue container
(`#!cpp json j = std::move(vec);`) moves the elements instead of copying them; the source is then left in a valid but
unspecified state.
- **Access without copying.** [`get<T>()`](../api/basic_json/get.md) returns a copy of the stored value converted to
`T`. When a reference or pointer to the value already stored inside the `basic_json` is enough,
[`get_ref()`](../api/basic_json/get_ref.md) and [`get_ptr()`](../api/basic_json/get_ptr.md) access it directly:
@@ -144,6 +153,29 @@ an adapter, since their template argument order does not match what `basic_json`
["reserving array capacity"](element_access/unchecked_access.md#performance-reserving-array-capacity) for the
one-line recipe.
## Memory
- **A value is small, its content is on the heap.** A `basic_json` is 16 bytes on a 64-bit platform (a type tag and a
union). Strings, arrays, objects, and binary values are allocated on the heap and owned by the value. Destroying the
value frees everything it owns.
- **Copies are deep.** After `#!cpp json b = a;`, changing `b` does not change `a`, and both own their memory. Copying a
large value therefore doubles its memory; see [Avoiding copies](#avoiding-copies). `#!cpp auto x = j["key"];` is a
copy, `#!cpp auto& x = j["key"];` is not.
- **A moved-from value is `null`.** After `#!cpp json c = std::move(a);` the value `a` is `#!json null` and owns nothing.
- **Reading does not require a tree.** If you only read the data, parse it into a [`json_document`](json_view.md)
(`#!cpp <nlohmann/json_view.hpp>`, added in version 3.13.0) instead of a `basic_json`. It stores 16 bytes per value, does not copy floating-point
numbers or strings without escape sequences out of the text, and creates no `basic_json` value except for the parts
you [`materialize()`](../api/basic_json_view/materialize.md). A document that borrows the text needs the text to
outlive it; see [Zero-copy JSON views](json_view.md).
- **Containers keep their capacity.** `clear()` and `erase()` on an array remove the elements, but the underlying
`#!cpp std::vector` keeps its allocated capacity, like `#!cpp std::vector::clear` does. To give the memory back,
assign a new value (`#!cpp j = json::array();`) or destroy the value.
- **Freed memory does not always go back to the operating system.** The library returns memory to the allocator
(`#!cpp std::allocator` by default, or the [one you configured](types/template_parameters.md)). Whether the process
size shrinks then depends on the allocator and the platform, as for any C++ program. A process that parses a large
document and drops it is not leaking if its resident size stays high; run it under a tool such as
[Valgrind](https://valgrind.org) or the sanitizers to tell a leak from retained memory.
## Serialization
[`dump()`](../api/basic_json/dump.md) with the default `#!cpp indent = -1` selects "the most compact representation"
docs/mkdocs/docs/features/type_recipes.md
+122
View File
@@ -0,0 +1,122 @@
# Recipes for Common Types
The library converts the standard containers, strings, numbers, tuples, `std::optional`, and `std::filesystem::path`
(the last two since version 3.12.0, C++17) out of the box, and your own types once you write conversion functions or use
the macros (see [Arbitrary type conversions](arbitrary_types.md)). It deliberately does **not** convert smart pointers,
`std::variant`, `std::chrono` types, or class hierarchies, because there is more than one reasonable JSON
representation for each of them. This page shows a representation for each that you can copy and adapt.
All recipes use a specialization of [`adl_serializer`](../api/adl_serializer/index.md), which you may provide for types
you do not own, or free `to_json`/`from_json` functions in the namespace of the type. Place them where they are
visible before any code that converts the type; see
[Why isn't my `to_json` found?](arbitrary_types.md#why-isnt-my-to_json-or-from_json-found).
## Smart pointers
A `std::shared_ptr<T>` or `std::unique_ptr<T>` is written as the value it points to, or as `#!json null` for an empty
pointer, and read back by allocating a new `T`:
??? example "Example: serialize shared_ptr and unique_ptr"
```cpp
--8<-- "examples/type_recipes_smart_pointers.c++14.cpp"
```
Output:
```
--8<-- "examples/type_recipes_smart_pointers.c++14.output"
```
With this, containers of smart pointers, pointers to `const`, and members of these types in your own structs convert as
usual. A pointer to a base class cannot be filled this way, because `T` would have
to be created as a derived type; see the next section.
Reading creates a new object. Two `shared_ptr`s that pointed to the same object are written twice and read back as two
separate objects.
## Class hierarchies
!!! tip "Derived classes whose type is known at compile time"
If you only need to serialize the data members of a derived class, and the code always knows the concrete type, you do
not need a type tag. The [`NLOHMANN_DEFINE_DERIVED_TYPE_*` macros](arbitrary_types.md#simplify-your-life-with-macros)
(since version 3.12.0) add the members of the derived class to those of its base class. This section is for the case
where the concrete type is only known at run time, such as a `std::unique_ptr<shape>` that holds a circle or a
rectangle.
JSON has no notion of types, so a value in a hierarchy needs a **type tag** that says which derived class to create.
Write the tag in the object, and select the class from it when reading:
??? example "Example: serialize a class hierarchy with a type tag"
```cpp
--8<-- "examples/type_recipes_class_hierarchy.c++14.cpp"
```
Output:
```
--8<-- "examples/type_recipes_class_hierarchy.c++14.output"
```
A `#!cpp std::vector<std::unique_ptr<shape>>` converts to an array of tagged objects and back, creating a `circle`
and a `rectangle`. The same works for `std::shared_ptr<shape>`. Remember the virtual destructor: without it, deleting a
derived object through the base pointer is undefined behavior.
## `std::variant`
A variant has the same problem: the alternative must be recorded. If the alternatives are your own types, write a type
tag next to their members, exactly as for a hierarchy:
??? example "Example: serialize a std::variant with a type tag"
```cpp
--8<-- "examples/type_recipes_variant.c++17.cpp"
```
Output:
```
--8<-- "examples/type_recipes_variant.c++17.output"
```
In the example, the functions are found
because `geo` is the namespace of the template arguments of `#!cpp std::variant<circ, rect>`; for a variant of only
standard types (for example `#!cpp std::variant<int, std::string>`) there is no such namespace, and you need an
`adl_serializer` specialization or a wrapper type of your own.
If the alternatives are JSON-native (numbers, strings, ...), a variant such as `#!cpp std::variant<int, std::string>`
can instead be written as the plain value, and read by testing the JSON type (`is_number_integer()`, `is_string()`)
to choose the alternative.
## `std::chrono`
A duration is written as its count and read back with the same representation and period. The **unit is not part of
the JSON value**, so document it, for example in the member name (`timeout_ms`):
??? example "Example: serialize durations and system_clock time points"
```cpp
--8<-- "examples/type_recipes_chrono.cpp"
```
Output:
```
--8<-- "examples/type_recipes_chrono.output"
```
A `system_clock::time_point` with millisecond precision is written as the number of milliseconds since the Unix epoch.
The time point serializer is limited to `system_clock`,
because the epoch of other clocks (for example `steady_clock`) has no meaning outside the running program.
If you prefer a text representation such as ISO 8601 (`2023-11-14T22:13:20Z`), write the string with
`#!cpp std::format("{:%FT%TZ}", t)` and read it with `std::chrono::parse` (both C++20), or with a date library, in the
same two functions.
## See also
- [Arbitrary type conversions](arbitrary_types.md) - the conversion mechanism and macros
- [Why isn't my `to_json` found?](arbitrary_types.md#why-isnt-my-to_json-or-from_json-found)
- [Converting values](conversions.md) - `get`, implicit conversions, and `std::optional`
docs/mkdocs/docs/home/architecture.md
+6 -1
View File
@@ -64,7 +64,7 @@ below are relative to `include/nlohmann`.
|-----------|----------|
| Value type enumeration | [`detail/value_t.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/value_t.hpp) |
| Input adapters | [`detail/input/input_adapters.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/input_adapters.hpp) |
| Lexer | [`detail/input/lexer.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/lexer.hpp), [`detail/input/number_parse.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/number_parse.hpp), [`detail/input/string_scan.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/string_scan.hpp) |
| Lexer | [`detail/input/lexer.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/lexer.hpp), [`detail/input/number_parse.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/number_parse.hpp), [`detail/input/string_scan.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/string_scan.hpp), [`detail/input/simd_scan.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/simd_scan.hpp) |
| Parser | [`detail/input/parser.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/parser.hpp) |
| SAX interface and DOM builders | [`detail/input/json_sax.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/json_sax.hpp) |
| Binary format readers | [`detail/input/binary_reader.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/binary_reader.hpp) |
@@ -339,6 +339,11 @@ The `serializer` (used by [`dump`](../api/basic_json/dump.md) and [`operator<<`]
- `output_stream_adapter` writes to a `std::ostream`.
- `output_string_adapter` appends to a string.
The `binary_writer` does not call an adapter for every byte. The `to_*` functions that return a `std::vector` write
into the vector through a sink (`output_vector_sink`) that is not virtual. The functions that take an output adapter
write through a sink (`output_adapter_sink`) that collects the output in a buffer of one kibibyte and passes it to the
adapter in blocks, so that the adapter is called (and a stream locked) once per block instead of once per value.
## Value conversion
Values are converted from and to other types with the `JSONSerializer` template parameter. The default,
docs/mkdocs/docs/home/debugging.md
+23
View File
@@ -10,6 +10,29 @@ at its root, a [Natvis](https://learn.microsoft.com/en-us/visualstudio/debugger/
file that gives `json`/`ordered_json` values a friendly, key/value debugger view instead of showing raw internal
fields, when debugging with the MSVC debug engine (`cppvsdbg`) in Visual Studio or VS Code.
The checked-in visualizers cover namespace variants with zero, one, or two ABI tags. Configurations with three or more
ABI tags are not covered and may show raw internal fields instead. To generate a Natvis file for such a configuration:
1. Check out the release tag of the library version you use.
2. List the ABI tags and the macros that enable them:
```shell
make natvis NATVIS_ARGS="--list-tags"
```
3. Generate a Natvis file with the tags of your configuration, for example for `JSON_DIAGNOSTICS`,
`JSON_DIAGNOSTIC_POSITIONS`, and `JSON_VIEW_NO_SIMD`:
```shell
make natvis NATVIS_ARGS="--abi-tags _diag,_dp,_vns --output my_nlohmann_json.natvis"
```
4. Use the generated file instead of `nlohmann_json.natvis`.
`--abi-tags` can be repeated for several configurations, and `--all` generates every combination (about 60 MB for 12
ABI tags, doubling with each further tag). See
[`tools/generate_natvis`](https://github.com/nlohmann/json/tree/develop/tools/generate_natvis) for all options.
Debug engines that wrap LLDB instead of the MSVC debug engine (for example, `codelldb` in VS Code) only have
partial/experimental Natvis support, and commonly fall back to showing raw internal fields even with the
`.natvis` file present. Switching to `cppvsdbg` where available, or checking your debug extension's own Natvis
docs/mkdocs/docs/home/exceptions.md
+30
View File
@@ -39,6 +39,8 @@ classDiagram
Exceptions are used widely within the library. They can, however, be switched off with either using the compiler flag `-fno-exceptions` or by defining the symbol [`JSON_NOEXCEPTION`](../api/macros/json_noexception.md). In this case, exceptions are replaced by `abort()` calls. You can further control this behavior by defining `JSON_THROW_USER` (overriding `#!cpp throw`), `JSON_TRY_USER` (overriding `#!cpp try`), and `JSON_CATCH_USER` (overriding `#!cpp catch`).
See [Working without exceptions](../features/no_exceptions.md) for the functions that report errors without throwing.
Note that [`JSON_THROW_USER`](../api/macros/json_throw_user.md) should leave the current scope (e.g., by throwing or aborting), as continuing after it may yield undefined behavior.
??? example "Example: switch off exceptions and log errors before aborting"
@@ -192,6 +194,34 @@ This error indicates a syntax error while deserializing a JSON text. The error m
- Make sure the input is correctly read. Try to write the input to standard output to check if, for instance, the input file was successfully opened.
- Paste the input to a JSON validator like <https://jsonlint.com> or a tool like [jq](https://jqlang.github.io/jq/).
??? info "Parsing a file fails at line 1, column 1 (or \"unexpected end of input\")"
The messages `attempting to parse an empty input` (since version 3.11.3) and `unexpected end of input` (also the message of
earlier versions for an empty input) at the very start of the input mean
that the parser did not see a single character. The library only reads from the stream it is given: it does not
open files and cannot tell a missing file from an empty one. The usual causes are:
- **The file was not opened.** A relative path is resolved against the *working directory of the running program*,
not against the source file or the project. IDEs and build tools often start the program in a build directory
(for example `build/` or `cmake-build-debug/`), so `"example.json"` may need to be `"../example.json"`. Use an
absolute path to rule this out.
- **The file is empty**, or a previous run truncated it (for example when it was opened for writing).
- **The stream was already read.** A stream whose content has been consumed (by printing it with
`std::cout << file.rdbuf()`, or by an earlier `accept`/`parse`/`>>`) is at its end. Parse it once, or rewind it
with `seekg(0)` after clearing the state with `clear()`.
Check the stream before parsing it:
```cpp
std::ifstream f("example.json");
if (!f.is_open())
{
std::cerr << "cannot open example.json\n";
return 1;
}
json data = json::parse(f);
```
### json.exception.parse_error.102
JSON uses the `\uxxxx` format to describe Unicode characters. Code points above 0xFFFF are split into two `\uxxxx` entries ("surrogate pairs"). This error indicates that the surrogate pair is incomplete or contains an invalid code point.
docs/mkdocs/docs/home/faq.md
+234 -1
View File
@@ -58,6 +58,208 @@ json j{obj}; // -> {"key":"value"} (copy, not array)
Without the macro (default behavior), `json j{obj}` creates `[{"key":"value"}]`. This opt-in macro fixes issue #5074 while preserving backwards compatibility for existing code.
## Getting started
### Which file do I need?
!!! question
Do I have to clone the repository or build the library?
No. The library is header-only: the single file [`json.hpp`](https://github.com/nlohmann/json/releases) from the
releases page is all you need, including the `_json` literals. Put it in a directory `nlohmann` on your include path.
Companion headers (`json_fwd.hpp`, `json_view.hpp`, `json_literals.hpp`) are only needed if you use them. See
[Integration](../integration/index.md#header-only).
### Using CMake or a package manager
!!! question
How do I add the library with CMake, Conan, vcpkg, Homebrew, or another tool?
See [CMake](../integration/cmake.md) for `find_package`, `add_subdirectory`, and `FetchContent`, and
[Package managers](../integration/package_managers.md) for the list of supported package managers.
### The IDE cannot open the header
!!! question
The compiler finds `nlohmann/json.hpp`, but my IDE reports "cannot open source file". Why?
The compiler's include path is set correctly, but the IDE's is not. Add the same directory to the include paths of the
IDE (or of its language server); the build itself is not affected.
### Compile times
!!! question
Including the library slows down my build. What can I do?
Include [`json_fwd.hpp`](../integration/compile_times.md#include-json_fwdhpp-in-headers) in headers and `json.hpp` only
in the source files that need it, instantiate `basic_json` once, and consider precompiled headers or
[C++20 modules](../features/modules.md). See [Compile times](../integration/compile_times.md).
## Parsing
### Parsing a file fails with "unexpected end of input"
!!! question
`json::parse(file)` throws `parse_error.101` at line 1, column 1 for a file that is valid JSON. Why?
The parser did not see any character: the file was most likely not opened. A relative path is resolved against the
working directory of the running program, which is often the build directory of your IDE, not the source directory.
Check `is_open()` and try an absolute path. Other causes are an empty file and a stream that was already read. See
[`parse_error.101`](exceptions.md#jsonexceptionparse_error101).
### Input that arrives in chunks
!!! question
My data comes from a socket in pieces. How do I parse it?
The parser cannot be resumed. Frame the messages (one value per line, a length prefix) and parse each complete frame, or
buffer the chunks until the value is complete; a `parse_error` at the end of the buffer means the input ended
prematurely. See [Large, streamed, and partial input](../features/parsing/large_and_partial_input.md#input-arriving-in-chunks).
### Several values in one stream
!!! question
How do I read concatenated JSON values or one value per line?
Read newline-delimited input line by line ([JSON Lines](../features/parsing/json_lines.md)). For concatenated values, use
`operator>>` and check for the end of the stream before each read, as shown in
[Several values in one stream](../features/parsing/large_and_partial_input.md#several-values-in-one-stream).
### Files that are too large for memory
!!! question
How do I process a huge JSON file?
Handle and discard the records of a large array with a [parser callback](../features/parsing/parser_callbacks.md), stop
early with the [SAX interface](../features/parsing/sax_interface.md), or read parts of the text through a
[`json_document`](../features/json_view.md) (added in version 3.13.0). See
[Input too large for memory](../features/parsing/large_and_partial_input.md#input-too-large-for-memory).
### Position of a parse error
!!! question
How do I get the line and column of a parse error?
The message contains them. The exception stores the byte index in `byte`, from which you can compute both; see
[Where did the error occur?](../features/parsing/parse_exceptions.md#where-did-the-error-occur). To locate a value that
causes a *later* error, such as a `type_error`, use
[`JSON_DIAGNOSTIC_POSITIONS`](../features/parsing/parse_exceptions.md#where-is-the-value-that-caused-a-later-error).
### Comments and trailing commas
!!! question
Can the parser read JSON with comments or trailing commas?
Yes, when asked to: see [Comments](../features/comments.md) and [Trailing commas](../features/trailing_commas.md). Both
are rejected by default.
## Accessing and creating values
### Reading a value into a variable
!!! question
How do I get a C++ value out of a `json` value?
Use [`get<T>()`](../api/basic_json/get.md), for example `#!cpp int age = j["age"].get<int>();`, or an implicit conversion.
`#!cpp auto x = j["age"];` is **not** a conversion: it copies the `json` value. See
[Converting values](../features/conversions.md).
### Strings are printed with quotes
!!! question
`std::cout << j["name"]` prints `"Mary"` with quotes. How do I get the string?
The value is a `json`, and printing it produces JSON text. Convert it first: `#!cpp j["name"].get<std::string>()`. See
[Getting a string without quotes](../features/conversions.md#getting-values-out).
### Testing whether a key exists
!!! question
How do I check if a key exists, without changing the object?
Use `contains` or `find`. Do not test with the non-const `operator[]`: like `std::map::operator[]`, it inserts a `null`
value for a missing key. See [Does this key exist?](../features/element_access/index.md#does-this-key-exist).
### Reading a value with a fallback
!!! question
How do I read a value that may be missing?
Use [`value`](../features/element_access/default_value.md), which returns a default for a missing key. It still throws if
the key exists with a different type; see [Working without exceptions](../features/no_exceptions.md).
### Iterating and nested values
!!! question
How do I loop over an array or object, or reach a nested value?
See [Iterators](../features/iterators.md) and [`items()`](../api/basic_json/items.md) for loops, and
[JSON Pointer](../features/json_pointer.md) to address or test a nested value by path.
### Unexpected arrays when creating values
!!! question
Why does `json j{x}` or `{{"a", 1}, {"b"}}` produce an array instead of what I expected?
Braces create arrays and objects based on their content, and single-element braces create a one-element array. See
[Brace initialization yields arrays](#brace-initialization-yields-arrays) and the table in
[Creating values](../features/creating_values.md#what-does-this-produce).
### Keeping the order of object keys
!!! question
The keys of my object come out sorted. How do I keep the insertion order?
Use `ordered_json`, and call `ordered_json::parse` when reading. See [Object order](../features/object_order.md).
## Converting your own types
### Structs, enums, and optional fields
!!! question
How do I convert my own types to and from JSON?
Write `to_json`/`from_json` functions or use the `NLOHMANN_DEFINE_TYPE_*` macros; see
[Arbitrary type conversions](../features/arbitrary_types.md). Enums can be mapped to strings
([Enum conversion](../features/enum_conversion.md)), and an empty `std::optional` can be omitted
([Omitting a field](../features/conversions.md#omitting-a-field-when-serializing-stdoptional)).
### Smart pointers, variants, chrono, and class hierarchies
!!! question
Why can't I convert `std::shared_ptr`, `std::variant`, `std::chrono` types, or a pointer to a base class?
There is more than one sensible JSON representation for them, so the library leaves the choice to you.
[Recipes for common types](../features/type_recipes.md) shows one for each.
### My `to_json` is not found
!!! question
I wrote `to_json`/`from_json`, but the compiler does not find them. Why?
Typical causes are a type alias of a standard type, a function in the wrong namespace, or a declaration that comes too
late. See [Why isn't my `to_json` found?](../features/arbitrary_types.md#why-isnt-my-to_json-or-from_json-found).
## Limitations
### Relaxed parsing
@@ -202,6 +404,36 @@ Not directly, but the companion project [json-schema-validator](https://github.c
builds JSON Schema (draft 7; draft 4 in its older, now-superseded 1.x releases) validation on top of this library
and is a common recommendation for this use case.
### Memory use
!!! question
Does the library leak memory? My process stays large after I free a `json` value.
Destroying a value frees everything it owns. Freed memory is returned to the allocator, which does not necessarily give
it back to the operating system, so a high resident size alone is not a leak. Copies are deep, and `clear()` keeps the
capacity of an array. See [Memory](../features/performance.md#memory), and use Valgrind or the sanitizers to tell a leak
from retained memory.
### Reproducible builds
!!! question
The header path ends up in my binary. How do I get path-independent builds?
The runtime assertions embed `__FILE__`. Define `NDEBUG` (the default for release builds) or map the path prefix; see
[Notes for builds and distribution](../integration/index.md#notes-for-builds-and-distribution).
### Slow debug builds
!!! question
Parsing a large file is very slow in my debug build. Can I compile the library with optimizations?
The library is header-only, so there is no separate source file. Keep the code that uses `json` in its own source file and
compile that file with optimizations; see
[Notes for builds and distribution](../integration/index.md#notes-for-builds-and-distribution).
## Exceptions
### Parsing without exceptions
@@ -210,7 +442,8 @@ and is a common recommendation for this use case.
Is it possible to indicate a parse error without throwing an exception?
Yes, see [Parsing and exceptions](../features/parsing/parse_exceptions.md).
Yes, see [Parsing and exceptions](../features/parsing/parse_exceptions.md) and
[Working without exceptions](../features/no_exceptions.md).
### Key name in exceptions
docs/mkdocs/docs/index.md
+4
View File
@@ -29,6 +29,10 @@ int main()
j["name"] = "Niels";
j["list"] = {1, 0, 2};
// read a value into a C++ variable and use it
double pi = j["pi"].get<double>();
std::cout << "2 * pi = " << 2 * pi << '\n';
// serialize with an indent of 4 spaces
std::cout << j.dump(4) << '\n';
}
docs/mkdocs/docs/integration/cmake.md
-11
View File
@@ -235,17 +235,6 @@ Build the unit tests against the [simdutf](https://github.com/simdutf/simdutf) U
its version is set by the cache variable `JSON_SIMDUTF_VERSION`. This option is `OFF` by default. Depends on
`JSON_BuildTests`.
### `JSON_TestUnityBuild`
Build the unit tests in batches of several test files per executable to speed up compilation: the files of a batch are
compiled as one translation unit, so the template instantiations of the library are shared. Every test file still gets
its own CTest test (e.g., `test-foo_cpp11`), which runs only the test cases of that file from the shared executable.
This option is `ON` by default (and `OFF` with MinGW). Set it to `OFF` to get one executable per test file, e.g., when
debugging a single file. The number of files per executable is set by the cache variable `JSON_TestUnityBatchSize`
(default: `8`). Related test files (e.g., all binary formats) are grouped explicitly in `tests/CMakeLists.txt`; each
group is compiled as one executable per C++ standard regardless of the batch size, and only the remaining files are
batched by size. Depends on `JSON_BuildTests`.
### `JSON_Valgrind`
Execute the test suite with [Valgrind](https://valgrind.org). This option is `OFF` by default. Depends on `JSON_BuildTests`.
docs/mkdocs/docs/integration/index.md
+33
View File
@@ -43,6 +43,21 @@ using json = nlohmann::json;
to the files you want to process JSON and set the necessary switches to enable C++11 (e.g., `-std=c++11` for GCC and
Clang).
!!! tip "Which file do I download?"
For a project without a package manager, the library itself, including the `_json` literals, is **one file**:
[`json.hpp`](https://github.com/nlohmann/json/releases/latest/download/json.hpp) from the
[releases page](https://github.com/nlohmann/json/releases). You do not need to clone the repository, build
anything, or link a library. Put the file in a directory named `nlohmann` that is on your include path (for
example `third_party/nlohmann/json.hpp` with `-I third_party`), so that `#include <nlohmann/json.hpp>` works.
Download the companion headers listed below only if you use them: `json_view.hpp` for the
[zero-copy views](../features/json_view.md), `json_fwd.hpp` for forward declarations, and `json_literals.hpp` if
you define `JSON_NO_AUTOMATIC_UDLS`. Keep them in the same `nlohmann` directory.
Do not copy only parts of the repository's `include/` directory: the headers in it include each other and must be
kept together. If your IDE reports "cannot open source file" while the compiler succeeds, the compiler's include
path is set correctly but the IDE's is not; add the same directory to the IDE's include paths.
You can further use file
[`single_include/nlohmann/json_fwd.hpp`](https://github.com/nlohmann/json/blob/develop/single_include/nlohmann/json_fwd.hpp)
for forward declarations (see [Compile times](compile_times.md)), and file
@@ -54,3 +69,21 @@ For the read-only, non-owning [`basic_json_document`](../api/basic_json_document
types, additionally include
[`single_include/nlohmann/json_view.hpp`](https://github.com/nlohmann/json/blob/develop/single_include/nlohmann/json_view.hpp);
see [Zero-copy JSON views](../features/json_view.md).
## Notes for builds and distribution
**Reproducible and path-independent builds.** The runtime assertions use `assert`, which embeds the name of the source
file (`__FILE__`) in the binary. Release builds do not contain them, because they define `NDEBUG`. If you compile with
assertions enabled and need binaries that do not depend on the build directory, either define `NDEBUG` or map the
prefix of the path with the compiler option `-ffile-prefix-map=<old>=<new>` (GCC 8 and later, Clang 10 and later). See
[Runtime assertions](../features/assertions.md).
**There is no separate source file to compile.** The library is header-only, so there is no `.cpp` that could be
built with different options than the code that includes it. It is also not available as a precompiled library. To
compile the library with optimizations in an otherwise unoptimized build (for example to parse large files in a debug
build, which is slow because of the unoptimized code and the checks of the debug standard library), keep the code that
uses `json` in a separate source file and set the options for that file only. [Compile times](compile_times.md)
shows how to [instantiate `basic_json` once](compile_times.md#instantiate-basic_json-once).
**Debugger visualization.** The Natvis file works with the Visual Studio debugger, including the `cppvsdbg` engine in
Visual Studio Code, but not with debuggers that wrap LLDB, such as `codelldb`; see [Debugging](../home/debugging.md).
docs/mkdocs/mkdocs.yml
+4
View File
@@ -60,6 +60,7 @@ nav:
- Features:
- features/index.md
- features/arbitrary_types.md
- features/type_recipes.md
- Binary Formats:
- features/binary_formats/index.md
- features/binary_formats/bjdata.md
@@ -84,11 +85,13 @@ nav:
- features/modifying_values.md
- features/modules.md
- 'nlohmann Namespace': features/namespace.md
- features/no_exceptions.md
- features/object_order.md
- features/performance.md
- Parsing:
- features/parsing/index.md
- features/parsing/json_lines.md
- features/parsing/large_and_partial_input.md
- features/parsing/parse_exceptions.md
- features/parsing/parser_callbacks.md
- features/parsing/sax_interface.md
@@ -383,6 +386,7 @@ nav:
- 'JSON_NO_AUTOMATIC_UDLS': api/macros/json_no_automatic_udls.md
- 'JSON_NOEXCEPTION': api/macros/json_noexception.md
- 'JSON_NO_IO': api/macros/json_no_io.md
- 'JSON_NO_SIMD': api/macros/json_no_simd.md
- 'JSON_NO_THREAD_LOCAL': api/macros/json_no_thread_local.md
- 'JSON_PRECISE_STREAM_POSITION': api/macros/json_precise_stream_position.md
- 'JSON_SKIP_LIBRARY_VERSION_CHECK': api/macros/json_skip_library_version_check.md
docs/mkdocs/scripts/mermaid/package-lock.json
Generated
+15 -15
View File
@@ -6,7 +6,7 @@
"": {
"name": "check-mermaid",
"dependencies": {
"jsdom": "30.1.2",
"jsdom": "30.1.3",
"mermaid": "11.17.2"
}
},
@@ -40,9 +40,9 @@
}
},
"node_modules/@asamuzakjp/dom-selector": {
"version": "9.2.4",
"resolved": "https://registry.npmjs.org/@asamuzakjp/dom-selector/-/dom-selector-9.2.4.tgz",
"integrity": "sha512-YnVzxDxLaqL0t7Q4wfcgHZjg55Wm6y5EaO7Bhu3q9J7wq/8iw3PhsnWiOQhYekx4VexnbkgWpTT6n+QIV7kczw==",
"version": "10.0.3",
"resolved": "https://registry.npmjs.org/@asamuzakjp/dom-selector/-/dom-selector-10.0.3.tgz",
"integrity": "sha512-3+777rEPsOOjoy5/sRmIzHnoRDaLpm3XWY2dmWn/N7f/XwDG0bcpnFaKL3X9FYJUbNwLmpcJC1bUYexJRxsFlw==",
"license": "MIT",
"dependencies": {
"bidi-js": "^1.1.0",
@@ -1105,9 +1105,9 @@
}
},
"node_modules/dompurify": {
"version": "3.4.16",
"resolved": "https://registry.npmjs.org/dompurify/-/dompurify-3.4.16.tgz",
"integrity": "sha512-sqo+pNp3qRhCIpbgRi1y8Tgk27Bo2Ry7w0dC1NBeNTdZChWjz9Xb/KOoZbRP/R6pQZ80Qw8YhXw13hWWBbMRnQ==",
"version": "3.4.17",
"resolved": "https://registry.npmjs.org/dompurify/-/dompurify-3.4.17.tgz",
"integrity": "sha512-B/pSlYVHQ58kqISD/yDjuJ1yfzI34m1gF4lAod2bwSRhiN4K5Cl3H8Af7hqB2PmTL911tdEudnnXn5FnhRoz0w==",
"license": "(MPL-2.0 OR Apache-2.0)",
"optionalDependencies": {
"@types/trusted-types": "^2.0.7"
@@ -1202,13 +1202,13 @@
"license": "MIT"
},
"node_modules/jsdom": {
"version": "30.1.2",
"resolved": "https://registry.npmjs.org/jsdom/-/jsdom-30.1.2.tgz",
"integrity": "sha512-0FFE/jE1rppmVfUrJUgxqXjcwolZYIGtAgj1pTussMhNZxIxi7W/PvfWaMNroGi2B36X1IKHbexpZ9DhkoOPiQ==",
"version": "30.1.3",
"resolved": "https://registry.npmjs.org/jsdom/-/jsdom-30.1.3.tgz",
"integrity": "sha512-hQMUPvRgN7KxsfIytDQji7bimuJeRQa6Qo5+iuiZQk5CafeRxLIA10/cG5YCBFNpD9RO3oqIiN4CO1Y44kxqXg==",
"license": "MIT",
"dependencies": {
"@asamuzakjp/css-color": "^7.1.3",
"@asamuzakjp/dom-selector": "^9.2.4",
"@asamuzakjp/dom-selector": "^10.0.3",
"@bramus/specificity": "^2.4.2",
"@csstools/css-syntax-patches-for-csstree": "^1.1.15",
"@exodus/bytes": "^1.16.0",
@@ -1221,7 +1221,7 @@
"parse5": "^8.0.1",
"saxes": "^6.0.0",
"tough-cookie": "^6.0.2",
"undici": "^8.11.2",
"undici": "^8.12.1",
"w3c-xmlserializer": "^6.0.0",
"webidl-conversions": "^8.0.1",
"whatwg-mimetype": "^5.0.0",
@@ -1521,9 +1521,9 @@
}
},
"node_modules/undici": {
"version": "8.11.2",
"resolved": "https://registry.npmjs.org/undici/-/undici-8.11.2.tgz",
"integrity": "sha512-u4UB2/IrKdU6lFxumHmmo1a3fCQO5tzQllRorfoRS63txhrB7xTpSn1PftwC4qEHkOaqP95fCWW4lJzwErwzhQ==",
"version": "8.12.2",
"resolved": "https://registry.npmjs.org/undici/-/undici-8.12.2.tgz",
"integrity": "sha512-XtuDWVdqTLr3NAxm2ly9/hU6kUsJOgGh1RASzMBbsw6aWCOaQzHdW+QQDfdERjfJ2V5vL5pRPa4RetO9UfivDg==",
"license": "MIT",
"engines": {
"node": ">=22.19.0"
docs/mkdocs/scripts/mermaid/package.json
+1 -1
View File
@@ -4,7 +4,7 @@
"description": "Validate the Mermaid diagrams of the documentation (see check_mermaid.mjs)",
"type": "module",
"dependencies": {
"jsdom": "30.1.2",
"jsdom": "30.1.3",
"mermaid": "11.17.2"
}
}
include/nlohmann/detail/abi_macros.hpp
+10 -3
View File
@@ -122,6 +122,12 @@
#define NLOHMANN_JSON_ABI_TAG_DISABLE_UNLOCKED_FILE_IO
#endif
#ifdef JSON_NO_SIMD
#define NLOHMANN_JSON_ABI_TAG_NO_SIMD _ns
#else
#define NLOHMANN_JSON_ABI_TAG_NO_SIMD
#endif
#ifdef JSON_VIEW_NO_SIMD
#define NLOHMANN_JSON_ABI_TAG_VIEW_NO_SIMD _vns
#else
@@ -139,9 +145,9 @@
#endif
// Construct the namespace ABI tags component
#define NLOHMANN_JSON_ABI_TAGS_CONCAT_EX(a, b, c, d, e, f, g, h, i, j, k, l) json_abi ## a ## b ## c ## d ## e ## f ## g ## h ## i ## j ## k ## l
#define NLOHMANN_JSON_ABI_TAGS_CONCAT(a, b, c, d, e, f, g, h, i, j, k, l) \
NLOHMANN_JSON_ABI_TAGS_CONCAT_EX(a, b, c, d, e, f, g, h, i, j, k, l)
#define NLOHMANN_JSON_ABI_TAGS_CONCAT_EX(a, b, c, d, e, f, g, h, i, j, k, l, m) json_abi ## a ## b ## c ## d ## e ## f ## g ## h ## i ## j ## k ## l ## m
#define NLOHMANN_JSON_ABI_TAGS_CONCAT(a, b, c, d, e, f, g, h, i, j, k, l, m) \
NLOHMANN_JSON_ABI_TAGS_CONCAT_EX(a, b, c, d, e, f, g, h, i, j, k, l, m)
#define NLOHMANN_JSON_ABI_TAGS \
NLOHMANN_JSON_ABI_TAGS_CONCAT( \
@@ -155,6 +161,7 @@
NLOHMANN_JSON_ABI_TAG_OBJECTS_FOR_ENUM_KEYED_MAPS, \
NLOHMANN_JSON_ABI_TAG_DISABLE_STREAM_BULK_SCAN, \
NLOHMANN_JSON_ABI_TAG_DISABLE_UNLOCKED_FILE_IO, \
NLOHMANN_JSON_ABI_TAG_NO_SIMD, \
NLOHMANN_JSON_ABI_TAG_VIEW_NO_SIMD, \
NLOHMANN_JSON_ABI_TAG_VIEW_USE_SSSE3)
include/nlohmann/detail/bit_ops.hpp
+98 -4
View File
@@ -14,11 +14,24 @@
#include <intrin0.h> // __umulh, _umul128, _BitScanForward64, _BitScanReverse64
#endif
#ifdef _MSC_VER
#include <cstdlib> // _byteswap_ushort, _byteswap_ulong, _byteswap_uint64
#endif
#if defined(_MSC_VER) && !defined(__clang__) && (defined(_M_IX86) || defined(_M_ARM))
#include <intrin.h> // _BitScanForward, _BitScanReverse
#endif
#include <nlohmann/detail/macro_scope.hpp> // JSON_HEDLEY_ALWAYS_INLINE, NLOHMANN_JSON_NAMESPACE_BEGIN
#if defined(__cpp_lib_bit_cast) || defined(__cpp_lib_bitops) || defined(__cpp_lib_byteswap)
#include <bit> // bit_cast, byteswap, countl_zero, countr_zero
#endif
// Portable bit-level helpers for the number and string scanners. They use
// compiler builtins or platform-specific intrinsics where available and plain
// C++ otherwise, so they work regardless of byte order.
// compiler builtins or platform-specific intrinsics where available, the
// C++20 <bit> functions next, and plain C++ otherwise, so they work regardless
// of byte order. (The builtins and intrinsics come first: MSVC's
// std::countl_zero checks for LZCNT at run time unless AVX2 is enabled.)
NLOHMANN_JSON_NAMESPACE_BEGIN
namespace detail
@@ -33,6 +46,16 @@ inline int count_leading_zeros(std::uint64_t x) noexcept
unsigned long index = 0; // NOLINT(runtime/int): the type _BitScan*64 takes
_BitScanReverse64(&index, x);
return 63 - static_cast<int>(index);
#elif defined(_MSC_VER) && (defined(_M_IX86) || defined(_M_ARM))
unsigned long index = 0; // NOLINT(runtime/int): the type _BitScan* takes
if (_BitScanReverse(&index, static_cast<unsigned long>(x >> 32u))) // NOLINT(runtime/int)
{
return 31 - static_cast<int>(index);
}
_BitScanReverse(&index, static_cast<unsigned long>(x)); // NOLINT(runtime/int)
return 63 - static_cast<int>(index);
#elif defined(__cpp_lib_bitops)
return std::countl_zero(x);
#else
int n = 0;
for (int shift = 32; shift != 0; shift >>= 1)
@@ -56,6 +79,16 @@ inline int count_trailing_zeros(std::uint64_t x) noexcept
unsigned long index = 0; // NOLINT(runtime/int): the type _BitScan*64 takes
_BitScanForward64(&index, x);
return static_cast<int>(index);
#elif defined(_MSC_VER) && (defined(_M_IX86) || defined(_M_ARM))
unsigned long index = 0; // NOLINT(runtime/int): the type _BitScan* takes
if (_BitScanForward(&index, static_cast<unsigned long>(x))) // NOLINT(runtime/int)
{
return static_cast<int>(index);
}
_BitScanForward(&index, static_cast<unsigned long>(x >> 32u)); // NOLINT(runtime/int)
return 32 + static_cast<int>(index);
#elif defined(__cpp_lib_bitops)
return std::countr_zero(x);
#else
int n = 0;
for (int shift = 32; shift != 0; shift >>= 1)
@@ -70,6 +103,67 @@ inline int count_trailing_zeros(std::uint64_t x) noexcept
#endif
}
/// the object representation of @a from as a @a To (std::bit_cast in C++20)
template<typename To, typename From>
JSON_HEDLEY_ALWAYS_INLINE To bit_cast(const From& from) noexcept
{
static_assert(sizeof(To) == sizeof(From), "bit_cast needs types of the same size");
#if defined(__cpp_lib_bit_cast)
return std::bit_cast<To>(from);
#else
To to {};
std::memcpy(&to, &from, sizeof(To));
return to;
#endif
}
/// x with its bytes in reverse order (std::byteswap in C++23); compilers lower
/// it to a single bswap/rev/movbe
inline std::uint16_t byteswap(std::uint16_t x) noexcept
{
#if defined(__GNUC__) || defined(__clang__)
return __builtin_bswap16(x);
#elif defined(_MSC_VER)
return _byteswap_ushort(x);
#elif defined(__cpp_lib_byteswap)
return std::byteswap(x);
#else
return static_cast<std::uint16_t>((x >> 8u) | (x << 8u));
#endif
}
/// x with its bytes in reverse order (std::byteswap in C++23)
inline std::uint32_t byteswap(std::uint32_t x) noexcept
{
#if defined(__GNUC__) || defined(__clang__)
return __builtin_bswap32(x);
#elif defined(_MSC_VER)
return _byteswap_ulong(x);
#elif defined(__cpp_lib_byteswap)
return std::byteswap(x);
#else
return ((x & 0x000000FFu) << 24u) | ((x & 0x0000FF00u) << 8u)
| ((x & 0x00FF0000u) >> 8u) | ((x & 0xFF000000u) >> 24u);
#endif
}
/// x with its bytes in reverse order (std::byteswap in C++23)
inline std::uint64_t byteswap(std::uint64_t x) noexcept
{
#if defined(__GNUC__) || defined(__clang__)
return __builtin_bswap64(x);
#elif defined(_MSC_VER)
return _byteswap_uint64(x);
#elif defined(__cpp_lib_byteswap)
return std::byteswap(x);
#else
x = ((x & 0x00000000FFFFFFFFull) << 32u) | ((x & 0xFFFFFFFF00000000ull) >> 32u);
x = ((x & 0x0000FFFF0000FFFFull) << 16u) | ((x & 0xFFFF0000FFFF0000ull) >> 16u);
x = ((x & 0x00FF00FF00FF00FFull) << 8u) | ((x & 0xFF00FF00FF00FF00ull) >> 8u);
return x;
#endif
}
/// the 128-bit product of two 64-bit numbers
struct uint128_parts
{
@@ -107,8 +201,8 @@ inline uint128_parts full_multiplication(std::uint64_t a, std::uint64_t b) noexc
/// targets; always inlined, as GCC otherwise calls it in the number loops)
JSON_HEDLEY_ALWAYS_INLINE std::uint64_t read_eight_bytes(const unsigned char* b) noexcept
{
#if defined(_MSC_VER) || defined(__x86_64__) || defined(__i386__) || (defined(__BYTE_ORDER__) && defined(__ORDER_LITTLE_ENDIAN__) && __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__)
// the byte order already matches (all MSVC targets are little-endian)
#if JSON_LITTLE_ENDIAN_TARGET
// the byte order already matches
std::uint64_t result = 0;
std::memcpy(&result, b, sizeof(result));
return result;
include/nlohmann/detail/conversions/to_chars.hpp
+9 -25
View File
@@ -18,17 +18,14 @@
#include <limits> // numeric_limits
#include <type_traits> // conditional
#ifdef _MSC_VER
#include <cstdlib> // _byteswap_uint64
#endif
// SSE2 (every x86-64 CPU) and NEON (every 64-bit Arm CPU) convert the 16
// digits of a double at once
#if defined(__x86_64__) || (defined(_M_X64) && !defined(_M_ARM64EC))
// digits of a double at once (unless JSON_NO_SIMD selects the portable code)
#if !defined(JSON_NO_SIMD) && (defined(__x86_64__) || (defined(_M_X64) && !defined(_M_ARM64EC)))
#include <emmintrin.h>
#define JSON_DTOA_SSE2 1
#define JSON_DTOA_NEON 0
#elif (defined(__aarch64__) || defined(_M_ARM64)) && !defined(_M_ARM64EC) && !defined(__ARM_BIG_ENDIAN)
#elif !defined(JSON_NO_SIMD) && (defined(__aarch64__) || defined(_M_ARM64)) && !defined(_M_ARM64EC) && !defined(__ARM_BIG_ENDIAN)
#include <arm_neon.h>
#define JSON_DTOA_SSE2 0
#define JSON_DTOA_NEON 1
@@ -37,6 +34,7 @@
#define JSON_DTOA_NEON 0
#endif
#include <nlohmann/detail/bit_ops.hpp>
#include <nlohmann/detail/conversions/zmij.hpp>
#include <nlohmann/detail/macro_scope.hpp>
@@ -66,16 +64,6 @@ For a detailed description of the algorithm see:
namespace dtoa_impl
{
template<typename Target, typename Source>
Target reinterpret_bits(const Source source)
{
static_assert(sizeof(Target) == sizeof(Source), "size mismatch");
Target target;
std::memcpy(&target, &source, sizeof(Source));
return target;
}
struct diyfp // f * 2^e
{
static constexpr int kPrecision = 64; // = q
@@ -230,7 +218,7 @@ boundaries compute_boundaries(FloatType value)
using bits_type = typename std::conditional<kPrecision == 24, std::uint32_t, std::uint64_t >::type;
const auto bits = static_cast<std::uint64_t>(reinterpret_bits<bits_type>(value));
const auto bits = static_cast<std::uint64_t>(bit_cast<bits_type>(value));
const std::uint64_t E = bits >> (kPrecision - 1);
const std::uint64_t F = bits & (kHiddenBit - 1);
@@ -1084,13 +1072,10 @@ inline std::uint64_t eight_digit_bytes(std::uint64_t abcdefgh) noexcept
/// the byte stores once this is inlined)
inline void store_msb_first(char* p, std::uint64_t v) noexcept
{
#if defined(__BYTE_ORDER__) && defined(__ORDER_LITTLE_ENDIAN__) && __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
v = __builtin_bswap64(v);
#if JSON_LITTLE_ENDIAN_TARGET
v = byteswap(v);
std::memcpy(p, &v, sizeof(v));
#elif defined(__BYTE_ORDER__) && defined(__ORDER_BIG_ENDIAN__) && __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__
std::memcpy(p, &v, sizeof(v));
#elif defined(_MSC_VER) // (little-endian on all its targets)
v = _byteswap_uint64(v);
#elif JSON_BIG_ENDIAN_TARGET
std::memcpy(p, &v, sizeof(v));
#else
for (unsigned i = 0; i < 8; ++i)
@@ -1462,8 +1447,7 @@ char* write_positive_zmij(char* first, const char* last, FloatType value)
{
static_assert(is_binary64<FloatType>::value,
"internal error: the conversion of Zmij needs IEEE 754 binary64 numbers");
std::uint64_t bits = 0;
std::memcpy(&bits, &value, sizeof(bits));
const auto bits = bit_cast<std::uint64_t>(value);
const zmij::shortest_decimal d = zmij::to_shortest(bits);
if (JSON_HEDLEY_LIKELY(last - first >= 41))
{
include/nlohmann/detail/input/binary_reader.hpp
+1 -1
View File
@@ -86,7 +86,7 @@ JSON_INLINE_VARIABLE constexpr std::size_t max_valueless_container_size = 1 << 2
/*!
@brief deserialization of BJData, BON8, BSON, CBOR, MessagePack, and UBJSON values
*/
template<typename BasicJsonType, typename InputAdapterType, typename SAX = json_sax_dom_parser<BasicJsonType, InputAdapterType>>
template<typename BasicJsonType, typename InputAdapterType, typename SAX = json_sax_dom_binary_parser<BasicJsonType, InputAdapterType>>
class binary_reader
{
using number_integer_t = typename BasicJsonType::number_integer_t;
include/nlohmann/detail/input/input_adapters.hpp
+191 -32
View File
@@ -8,17 +8,18 @@
#pragma once
#include <algorithm> // count, find, min
#include <algorithm> // count, find, max, min
#include <array> // array
#include <cstddef> // size_t
#include <cstdint> // uint32_t
#include <cstring> // strlen
#include <iterator> // begin, end, iterator_traits, random_access_iterator_tag, distance, next, reverse_iterator
#include <limits> // numeric_limits
#include <memory> // unique_ptr
#include <streambuf> // streambuf
#include <string> // string, char_traits
#include <type_traits> // enable_if, is_base_of, is_pointer, is_integral, remove_pointer
#include <utility> // pair, declval
#include <utility> // declval, move, pair
#include <vector> // vector
#ifndef JSON_NO_IO
@@ -72,6 +73,82 @@ enum class input_format_t { json, cbor, msgpack, ubjson, bson, bjdata, bon8 };
#ifndef JSON_NO_IO
/*!
A growable array of characters for the part of a token that an input adapter
copied out of its window: unlike std::vector<char>::push_back(), appending a
character takes no reference to a local, and is a few instructions when there
is room.
*/
class token_carry_buffer
{
public:
token_carry_buffer() = default;
token_carry_buffer(const token_carry_buffer&) = delete;
token_carry_buffer& operator=(const token_carry_buffer&) = delete;
token_carry_buffer(token_carry_buffer&& rhs) noexcept
: m_data(std::move(rhs.m_data)), m_end(rhs.m_end), m_capacity_end(rhs.m_capacity_end)
{
rhs.m_end = rhs.m_capacity_end = nullptr;
}
token_carry_buffer& operator=(token_carry_buffer&&) = delete;
~token_carry_buffer() = default;
void clear() noexcept
{
m_end = m_data.get();
}
void push_back(char c)
{
if (JSON_HEDLEY_UNLIKELY(m_end == m_capacity_end))
{
grow(1);
}
*m_end++ = c;
}
void append(const char* first, const char* last)
{
const auto n = static_cast<std::size_t>(last - first);
const auto room = static_cast<std::size_t>(m_capacity_end - m_end);
if (n > room)
{
grow(n);
}
if (n != 0)
{
std::memcpy(m_end, first, n);
}
m_end += n;
}
/// replace the contents of @a out
void copy_to(std::vector<char>& out) const
{
out.assign(m_data.get(), m_end);
}
private:
/// make room for @a extra more characters
JSON_HEDLEY_NEVER_INLINE void grow(std::size_t extra)
{
const auto size = static_cast<std::size_t>(m_end - m_data.get());
const auto capacity = static_cast<std::size_t>(m_capacity_end - m_data.get());
const std::size_t new_capacity = (std::max)(capacity < 32 ? static_cast<std::size_t>(32) : 2 * capacity, size + extra);
std::unique_ptr<char[]> data(new char[new_capacity]); // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
if (size != 0)
{
std::memcpy(data.get(), m_data.get(), size);
}
m_data = std::move(data);
m_end = m_data.get() + size;
m_capacity_end = m_data.get() + new_capacity;
}
std::unique_ptr<char[]> m_data{}; // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays,readability-redundant-member-init)
char* m_end = nullptr;
char* m_capacity_end = nullptr;
};
/*!
Input adapter for stdio file access. This adapter read only 1 byte and do not use any
buffer. This adapter is a very low level adapter.
*/
@@ -209,6 +286,7 @@ class input_stream_adapter
, token_segment(rhs.token_segment)
, current_in_get_area(rhs.current_in_get_area)
, tracking_token(rhs.tracking_token)
, no_get_area(rhs.no_get_area)
#endif
{
rhs.is = nullptr;
@@ -295,37 +373,20 @@ class input_stream_adapter
std::char_traits<char>::int_type get_character()
{
#if !JSON_DISABLE_STREAM_BULK_SCAN
// sbumpc() refills the get area if it is exhausted
const bool refill = streambuf_access::get_ptr(sb) == streambuf_access::get_end(sb);
if (JSON_HEDLEY_UNLIKELY(refill))
{
save_token_segment();
}
#endif
auto res = sb->sbumpc();
// set eof manually, as we don't use the istream interface.
if (JSON_HEDLEY_UNLIKELY(res == std::char_traits<char>::eof()))
{
#if !JSON_DISABLE_STREAM_BULK_SCAN
current_in_get_area = false;
#endif
is->clear(is->rdstate() | std::ios::eofbit);
}
#if !JSON_DISABLE_STREAM_BULK_SCAN
else if (JSON_HEDLEY_UNLIKELY(refill))
{
// the consumed character is at gptr() - 1, unless the streambuf
// has no get area and returned it from uflow()
const char* const next = streambuf_access::get_ptr(sb);
after_refill(res, next != nullptr && next != streambuf_access::get_begin(sb));
}
else
// the character is in the get area: step over it, as sbumpc() does
const char* const next = streambuf_access::get_ptr(sb);
if (JSON_HEDLEY_LIKELY(next != streambuf_access::get_end(sb)))
{
current_in_get_area = true;
streambuf_access::bump(sb, 1);
return std::char_traits<char>::to_int_type(*next);
}
if (no_get_area)
{
return get_character_no_get_area();
}
#endif
return res;
return get_character_slow();
}
#endif
@@ -334,6 +395,18 @@ class input_stream_adapter
{
#if JSON_PRECISE_STREAM_POSITION
commit_lookahead();
#endif
#if !JSON_DISABLE_STREAM_BULK_SCAN
// the elements are in the get area: copy them, as sgetn() does, but
// without the virtual call
const char* const first = streambuf_access::get_ptr(sb);
const auto available = static_cast<std::size_t>(streambuf_access::get_end(sb) - first);
if (JSON_HEDLEY_LIKELY(count * sizeof(T) <= available && count * sizeof(T) <= static_cast<std::size_t>((std::numeric_limits<int>::max)())))
{
std::memcpy(dest, first, count * sizeof(T));
streambuf_access::bump(sb, static_cast<int>(count * sizeof(T)));
return count * sizeof(T);
}
#endif
auto res = static_cast<std::size_t>(sb->sgetn(reinterpret_cast<char*>(dest), static_cast<std::streamsize>(count * sizeof(T))));
if (JSON_HEDLEY_UNLIKELY(res < count * sizeof(T)))
@@ -421,7 +494,7 @@ class input_stream_adapter
// last one if @a drop_last.
void copy_token(std::vector<char>& out, bool drop_last) const
{
out = token_carry;
token_carry.copy_to(out);
if (token_segment != nullptr)
{
out.insert(out.end(), token_segment, current_end());
@@ -434,6 +507,47 @@ class input_stream_adapter
#endif
private:
#if !JSON_PRECISE_STREAM_POSITION
// the get area is exhausted, or there is none
#if !JSON_DISABLE_STREAM_BULK_SCAN
JSON_HEDLEY_NEVER_INLINE
#endif
std::char_traits<char>::int_type get_character_slow()
{
#if !JSON_DISABLE_STREAM_BULK_SCAN
// sbumpc() refills the get area if it is exhausted
const bool refill = streambuf_access::get_ptr(sb) == streambuf_access::get_end(sb);
if (JSON_HEDLEY_UNLIKELY(refill))
{
save_token_segment();
}
#endif
auto res = sb->sbumpc();
// set eof manually, as we don't use the istream interface.
if (JSON_HEDLEY_UNLIKELY(res == std::char_traits<char>::eof()))
{
#if !JSON_DISABLE_STREAM_BULK_SCAN
current_in_get_area = false;
#endif
is->clear(is->rdstate() | std::ios::eofbit);
}
#if !JSON_DISABLE_STREAM_BULK_SCAN
else if (JSON_HEDLEY_UNLIKELY(refill))
{
// the consumed character is at gptr() - 1, unless the streambuf
// has no get area and returned it from uflow()
const char* const next = streambuf_access::get_ptr(sb);
after_refill(res, next != nullptr && next != streambuf_access::get_begin(sb));
}
else
{
current_in_get_area = true;
}
#endif
return res;
}
#endif
#if !JSON_DISABLE_STREAM_BULK_SCAN
// Reads the protected get-area pointers of any std::streambuf. Naming the
// members through a derived class makes forming the member pointers
@@ -457,6 +571,11 @@ class input_stream_adapter
{
(s->*(&streambuf_access::gbump))(n);
}
// sbumpc() for an exhausted get area
static std::streambuf::int_type next_character(std::streambuf* s)
{
return (s->*(&streambuf_access::uflow))();
}
};
// one past the character last returned by get_character(), if that is in
@@ -477,16 +596,53 @@ class input_stream_adapter
{
if (token_segment != nullptr)
{
token_carry.insert(token_carry.end(), token_segment, streambuf_access::get_end(sb));
token_carry.append(token_segment, streambuf_access::get_end(sb));
token_segment = nullptr;
}
}
#if !JSON_PRECISE_STREAM_POSITION
// The streambuf has no get area (e.g., std::cin synchronized with stdio):
// every character comes from uflow(), and the token is kept by value.
JSON_HEDLEY_NEVER_INLINE std::char_traits<char>::int_type get_character_no_get_area()
{
// the get area is exhausted, so this is what sbumpc() does
const auto res = streambuf_access::next_character(sb);
if (JSON_HEDLEY_UNLIKELY(res == std::char_traits<char>::eof() || streambuf_access::get_end(sb) != nullptr))
{
return get_character_no_get_area_other(res);
}
// current_in_get_area is false: only the first character of a get
// area, or the end of the input, is read here
if (tracking_token)
{
token_carry.push_back(std::char_traits<char>::to_char_type(res));
}
return res;
}
// @a res came from uflow() of a streambuf without a get area: it is the
// end of the input, or the streambuf has a get area now
JSON_HEDLEY_NEVER_INLINE std::char_traits<char>::int_type get_character_no_get_area_other(std::char_traits<char>::int_type res)
{
if (res == std::char_traits<char>::eof())
{
current_in_get_area = false;
is->clear(is->rdstate() | std::ios::eofbit);
return res;
}
// the consumed character is at gptr() - 1
after_refill(res, streambuf_access::get_ptr(sb) != streambuf_access::get_begin(sb));
return res;
}
#endif
// character @a c was just read into a new get area (@a in_get_area) or
// returned without one
void after_refill(std::char_traits<char>::int_type c, bool in_get_area)
{
current_in_get_area = in_get_area;
no_get_area = !in_get_area;
if (!tracking_token)
{
return;
@@ -525,13 +681,16 @@ class input_stream_adapter
#endif
#if !JSON_DISABLE_STREAM_BULK_SCAN
/// the part of the current token that was in earlier get areas
std::vector<char> token_carry{}; // NOLINT(readability-redundant-member-init)
token_carry_buffer token_carry{}; // NOLINT(readability-redundant-member-init)
/// start of the current token's part in the get area, or nullptr
const char* token_segment = nullptr;
/// whether the character last returned by get_character() is in the get area
bool current_in_get_area = false;
/// whether a token is tracked at all; the binary readers never start one
bool tracking_token = false;
/// whether the streambuf had no get area when it was last asked for a
/// character
bool no_get_area = false;
#endif
};
#endif // JSON_NO_IO
include/nlohmann/detail/input/json_sax.hpp
+1062 -13
View File
File diff suppressed because it is too large. Load diff
include/nlohmann/detail/input/lexer.hpp
+193 -53
View File
@@ -12,7 +12,9 @@
#include <cstddef> // size_t
#include <cstdint> // uint32_t
#include <cstdio> // snprintf
#include <cstring> // memcmp
#include <initializer_list> // initializer_list
#include <limits> // numeric_limits
#include <string> // char_traits, string
#include <utility> // move
#include <vector> // vector
@@ -417,30 +419,43 @@ class lexer : public lexer_base<BasicJsonType>
}
/// contiguous input: bulk-append the run of ordinary characters and complete
/// well-formed UTF-8 sequences starting at the current read position, leaving
/// the first byte that needs individual handling (the closing quote, an
/// escape, a control character, or an ill-formed UTF-8 byte) for get()
void scan_string_bulk(std::true_type /*bulk*/)
/// well-formed UTF-8 sequences starting at the current read position, then
/// read the first byte that needs individual handling (the closing quote, an
/// escape, a control character, or an ill-formed UTF-8 byte) like get().
/// That byte is taken from the buffer that was just scanned. Through get(),
/// Apple Clang merged this read with the other get() calls of the scan
/// function, which load the read position again right after it was stored
/// (3% of json::accept() on jeopardy.json on Apple M1).
char_int_type get_after_string_run(std::true_type /*bulk*/)
{
// a pending unget must be consumed through the normal path first
if (next_unget)
{
return;
return get();
}
const std::size_t remaining = ia.bulk_remaining();
if (remaining == 0)
{
return;
return get();
}
const auto* const data = reinterpret_cast<const unsigned char*>(ia.bulk_data());
const auto* const raw = ia.bulk_data();
const auto* const data = reinterpret_cast<const unsigned char*>(raw);
const std::size_t pos = string_bulk_run(data, remaining);
if (pos == 0)
const std::size_t pos = lexer_string_bulk_run(data, remaining);
if (pos != 0)
{
return;
token_buffer.append(reinterpret_cast<const typename string_t::value_type*>(data), pos);
bulk_consume(reinterpret_cast<const char*>(data), pos);
}
token_buffer.append(reinterpret_cast<const typename string_t::value_type*>(data), pos);
bulk_consume(reinterpret_cast<const char*>(data), pos);
if (pos == remaining)
{
return get(); // end of the contiguous input
}
// what get() does, with the byte at hand
advance_position();
ia.bulk_skip(1);
current = char_traits<char_type>::to_int_type(raw[pos]);
return track_after_read();
}
/// consume @a n characters at @a data (the adapter's bulk_data()) and do
@@ -473,7 +488,10 @@ class lexer : public lexer_base<BasicJsonType>
}
/// streaming input: no bulk fast path
void scan_string_bulk(std::false_type /*bulk*/) const noexcept {}
char_int_type get_after_string_run(std::false_type /*bulk*/)
{
return get();
}
/*!
@brief scan a string literal
@@ -501,11 +519,8 @@ class lexer : public lexer_base<BasicJsonType>
while (true)
{
// bulk-consume ordinary characters from contiguous input, then
// handle the next special byte through the switch below
scan_string_bulk(std::integral_constant<bool, bulk_scan> {});
// get the next character
switch (get())
// get the next character, the one that ends the run
switch (get_after_string_run(std::integral_constant<bool, bulk_scan>()))
{
// end of file while parsing the string
case char_traits<char_type>::eof():
@@ -1625,9 +1640,12 @@ scan_number_done:
const std::size_t avail = rem + 1;
// validate + classify the number extent (mirrors scan_number()'s grammar)
// and accumulate the mantissa digits on the way, so that tokens with up
// to 19 digits need no second pass for their value
std::size_t i = 0;
std::size_t dot_index = std::string::npos;
token_type number_type = token_type::value_unsigned;
std::uint64_t w = 0;
if (data[0] == '-')
{
number_type = token_type::value_integer;
@@ -1637,22 +1655,21 @@ scan_number_done:
return token_type::uninitialized;
}
}
const std::size_t integer_begin = i;
if (data[i] == '0')
{
++i;
}
else if (data[i] >= '1' && data[i] <= '9')
{
++i;
while (i < avail && data[i] >= '0' && data[i] <= '9')
{
++i;
}
i = accumulate_digits(data, i, avail, w);
}
else
{
return token_type::uninitialized;
}
std::size_t digit_count = i - integer_begin;
std::size_t fraction_digits = 0;
if (i < avail && data[i] == '.')
{
number_type = token_type::value_float;
@@ -1662,29 +1679,39 @@ scan_number_done:
{
return token_type::uninitialized;
}
while (i < avail && data[i] >= '0' && data[i] <= '9')
{
++i;
}
const std::size_t fraction_begin = i;
i = accumulate_digits(data, i, avail, w);
fraction_digits = i - fraction_begin;
digit_count += fraction_digits;
}
// the mantissa ends here, whether or not an exponent part follows
const std::size_t mantissa_end = i;
std::int64_t exponent = 0;
if (i < avail && (data[i] == 'e' || data[i] == 'E'))
{
number_type = token_type::value_float;
++i;
bool negative_exponent = false;
if (i < avail && (data[i] == '+' || data[i] == '-'))
{
negative_exponent = data[i] == '-';
++i;
}
if (i >= avail || !(data[i] >= '0' && data[i] <= '9'))
{
return token_type::uninitialized;
}
// saturated far beyond every range, see parse_float_exponent()
constexpr std::int64_t saturation = 100000000000000000; // 10^17
while (i < avail && data[i] >= '0' && data[i] <= '9')
{
if (exponent < saturation)
{
exponent = (exponent * 10) + (data[i] - '0');
}
++i;
}
exponent = negative_exponent ? -exponent : exponent;
}
const std::size_t len = i;
if (number_may_continue(len == avail, std::integral_constant<bool, bulk_window> {}))
@@ -1705,7 +1732,10 @@ scan_number_done:
#if !JSON_DIAGNOSTIC_POSITIONS
if (number_type != token_type::value_float)
{
const token_type integer_result = convert_integer(number_type, data, data + len);
// up to 18 digits always fit into std::uint64_t and std::int64_t
const token_type integer_result = (digit_count <= 18)
? store_integer(number_type, w)
: convert_integer(number_type, data, data + len);
if (JSON_HEDLEY_LIKELY(integer_result != token_type::uninitialized))
{
bulk_consume(data + 1, len - 1);
@@ -1719,15 +1749,42 @@ scan_number_done:
}
#endif
// materialize the token exactly as scan_number() would. reset() already
// cleared token_buffer, so append() fills it (assign() is avoided
// because custom string_t types need not provide it)
token_buffer.append(reinterpret_cast<const typename string_t::value_type*>(data), len);
decimal_point_position = dot_index;
bulk_consume(data + 1, len - 1);
read_number_terminator(std::integral_constant<bool, bulk_window> {});
// a float whose digits all fit into w converts from the accumulated
// value; longer ones (and other float formats) take the general path
if (number_type == token_type::value_float && digit_count <= 19)
{
float_significand s;
s.w = w;
s.exponent = exponent - static_cast<std::int64_t>(fraction_digits);
s.negative = data[0] == '-';
if (convert_accumulated_float(s, std::integral_constant<bool, has_native_float_format<number_float_t>::value> {}))
{
// the token text is only copied into token_buffer when
// get_string() asks for it; the library's own SAX handlers
// ignore it. A window into a stream may be refilled by the
// next read, so there the text is copied right away.
if (bulk_window)
{
token_buffer.append(reinterpret_cast<const typename string_t::value_type*>(data), len);
}
else
{
pending_number_text = data;
pending_number_length = len;
}
return token_type::value_float;
}
}
// materialize the token exactly as scan_number() would. reset() already
// cleared token_buffer, so append() fills it (assign() is avoided
// because custom string_t types need not provide it)
token_buffer.append(reinterpret_cast<const typename string_t::value_type*>(data), len);
return convert_number(number_type, mantissa_end);
}
@@ -1756,6 +1813,44 @@ scan_number_done:
/// the input is not handed back, so the terminator can stay unread
void read_number_terminator(std::false_type /*window*/) const noexcept {}
/// binary32/binary64: the value of a float token from its accumulated
/// significand (see scan_number_bulk_contiguous())
bool convert_accumulated_float(const float_significand& s, std::true_type /*native*/) noexcept
{
// accept() only needs to know that the value is finite (see the parser):
// w < 10^19, so a value below 10^max_exponent10 cannot overflow, and a
// value that underflows is a finite number as well. Only a token that
// may overflow has to be converted to find out.
if (JSON_HEDLEY_UNLIKELY(discard_number_values && s.exponent + 19 <= std::numeric_limits<number_float_t>::max_exponent10))
{
value_float = 0;
return true;
}
value_float = decimal_to_float<number_float_t>(s);
return true;
}
/// other formats (e.g. x87 long double): not instantiated for them, as
/// decimal_to_float() only supports binary32/binary64
static bool convert_accumulated_float(const float_significand& /*s*/, std::false_type /*native*/) noexcept
{
return false;
}
/// the value of an integer token with at most 18 digits, accumulated in @a w
token_type store_integer(token_type number_type, std::uint64_t w) noexcept
{
if (number_type == token_type::value_unsigned)
{
value_unsigned = static_cast<number_unsigned_t>(w);
// reject values that do not round-trip into a narrower type
return (static_cast<std::uint64_t>(value_unsigned) == w) ? token_type::value_unsigned : token_type::uninitialized;
}
const std::int64_t x = -static_cast<std::int64_t>(w);
value_integer = static_cast<number_integer_t>(x);
return (static_cast<std::int64_t>(value_integer) == x) ? token_type::value_integer : token_type::uninitialized;
}
/// contiguous input: try the number fast path, else the byte-path scanner
token_type scan_number_dispatch(std::true_type /*bulk*/)
{
@@ -1770,18 +1865,23 @@ scan_number_done:
}
/*!
@param[in] literal_text the literal text to expect
@param[in] length the length of the passed literal text
@param[in] return_type the token type to return on success
@param[in] rest the literal text after its first character, which
is current (its length is a template parameter, so
that the comparison of the contiguous fast path is
a fixed-size load, not a call of memcmp)
@param[in] return_type the token type to return on success
*/
JSON_HEDLEY_NON_NULL(2)
token_type scan_literal(const char_type* literal_text, const std::size_t length,
token_type return_type)
template<std::size_t N>
JSON_HEDLEY_NEVER_INLINE token_type scan_literal(const char (&rest)[N], token_type return_type) // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
{
JSON_ASSERT(char_traits<char_type>::to_char_type(current) == literal_text[0]);
for (std::size_t i = 1; i < length; ++i)
constexpr std::size_t length = N - 1;
if (scan_literal_bulk(rest, length, std::integral_constant<bool, bulk_scan> {}))
{
if (JSON_HEDLEY_UNLIKELY(char_traits<char_type>::to_char_type(get()) != literal_text[i]))
return return_type;
}
for (std::size_t i = 0; i < length; ++i)
{
if (JSON_HEDLEY_UNLIKELY(char_traits<char_type>::to_char_type(get()) != static_cast<char_type>(rest[i])))
{
error_message = "invalid literal";
return token_type::parse_error;
@@ -1790,6 +1890,27 @@ scan_number_done:
return return_type;
}
/// contiguous input: compare the rest of a literal at once; false (and
/// nothing consumed) if it does not match, so that the byte path reports
/// the error at the right character
JSON_HEDLEY_ALWAYS_INLINE bool scan_literal_bulk(const char* rest, const std::size_t length, std::true_type /*bulk*/)
{
if (next_unget || ia.bulk_remaining() < length
|| std::memcmp(ia.bulk_data(), rest, length) != 0)
{
return false;
}
bulk_consume(reinterpret_cast<const char*>(ia.bulk_data()), length);
current = static_cast<char_int_type>(static_cast<unsigned char>(rest[length - 1]));
return true;
}
/// streaming input: no bulk fast path
bool scan_literal_bulk(const char* /*rest*/, const std::size_t /*length*/, std::false_type /*bulk*/) const noexcept
{
return false;
}
/////////////////////
// input management
/////////////////////
@@ -1798,6 +1919,7 @@ scan_number_done:
void reset() noexcept
{
token_buffer.clear();
pending_number_text = nullptr;
decimal_point_position = std::string::npos;
#if JSON_DIAGNOSTIC_POSITIONS
@@ -2057,10 +2179,30 @@ scan_number_done:
/// return current string value
string_t& get_string()
{
if (pending_number_text != nullptr)
{
token_buffer.append(reinterpret_cast<const typename string_t::value_type*>(pending_number_text), pending_number_length);
pending_number_text = nullptr;
}
// a number token holds '.' regardless of the locale (#4084)
return token_buffer;
}
/// the value of a string token (key or string), which never has pending
/// number text, so get_string()'s check is not needed
string_t& get_string_token() noexcept
{
JSON_ASSERT(pending_number_text == nullptr);
return token_buffer;
}
/// the text of a float token for a SAX handler that ignores it (the
/// library's own handlers): empty, without copying the token
const string_t& get_ignored_number_text() const noexcept
{
return token_buffer;
}
/////////////////////
// diagnostics
/////////////////////
@@ -2394,21 +2536,14 @@ scan_number_done:
return token_type::value_separator;
// literals
// (the text after the first character; no local arrays, which
// would give this function a stack canary on every token)
case 't':
{
std::array<char_type, 4> true_literal = {{static_cast<char_type>('t'), static_cast<char_type>('r'), static_cast<char_type>('u'), static_cast<char_type>('e')}};
return scan_literal(true_literal.data(), true_literal.size(), token_type::literal_true);
}
return scan_literal("rue", token_type::literal_true);
case 'f':
{
std::array<char_type, 5> false_literal = {{static_cast<char_type>('f'), static_cast<char_type>('a'), static_cast<char_type>('l'), static_cast<char_type>('s'), static_cast<char_type>('e')}};
return scan_literal(false_literal.data(), false_literal.size(), token_type::literal_false);
}
return scan_literal("alse", token_type::literal_false);
case 'n':
{
std::array<char_type, 4> null_literal = {{static_cast<char_type>('n'), static_cast<char_type>('u'), static_cast<char_type>('l'), static_cast<char_type>('l')}};
return scan_literal(null_literal.data(), null_literal.size(), token_type::literal_null);
}
return scan_literal("ull", token_type::literal_null);
// string
case '\"':
@@ -2456,6 +2591,11 @@ scan_number_done:
/// whether the next get() call should just return current
bool next_unget = false;
/// the text of the current float token in the input, not yet copied into
/// token_buffer (see get_string()), or nullptr
const char* pending_number_text = nullptr;
std::size_t pending_number_length = 0;
/// the start position of the current token
position_t position {};
include/nlohmann/detail/input/number_parse.hpp
+74 -30
View File
@@ -16,7 +16,6 @@
#include <cstddef> // size_t
#include <cstdint> // int64_t, uint64_t
#include <cstdlib> // strtof, strtod, strtold
#include <cstring> // memcpy
#include <limits> // numeric_limits
#include <string> // string
#include <type_traits> // conditional, integral_constant, is_signed, true_type, false_type
@@ -275,13 +274,77 @@ JSON_HEDLEY_ALWAYS_INLINE std::uint32_t parse_eight_digits(std::uint64_t v) noex
return static_cast<std::uint32_t>(((v & 0x0000FFFF0000FFFFu) * 42949672960001u) >> 32u);
}
/// whether [first, last) contains a digit other than '0'
inline bool has_nonzero_digit(const char* first, const char* last) noexcept
/// whether the eight bytes in @a v (see read_eight_bytes()) are all ASCII
/// digits (after fast_float)
JSON_HEDLEY_ALWAYS_INLINE bool is_eight_digits(std::uint64_t v) noexcept
{
return std::any_of(first, last, [](char c) noexcept
return ((v & 0xF0F0F0F0F0F0F0F0u) | (((v + 0x0606060606060606u) & 0xF0F0F0F0F0F0F0F0u) >> 4u)) == 0x3333333333333333u;
}
/*!
@brief skip the ASCII digits starting at data[i] and accumulate their value
Eight digits are taken at a time while at least eight bytes remain. @a w wraps
around after 19 digits; the caller only uses it for shorter digit runs.
@return the index of the first byte at or after @a i that is not a digit
*/
JSON_HEDLEY_ALWAYS_INLINE std::size_t accumulate_digits(const char* data, std::size_t i, std::size_t avail, std::uint64_t& w) noexcept
{
while (avail - i >= 8)
{
return c != '0';
});
const std::uint64_t v = read_eight_bytes(data + i);
if (!is_eight_digits(v))
{
break;
}
w = (w * 100000000u) + parse_eight_digits(v);
i += 8;
}
while (i < avail && data[i] >= '0' && data[i] <= '9')
{
w = (w * 10u) + static_cast<std::uint64_t>(data[i] - '0');
++i;
}
return i;
}
/// whether [first, last) contains a digit other than '0'; a plain loop, as
/// GCC calls std::any_of out of line even for the usual empty range
JSON_HEDLEY_ALWAYS_INLINE bool has_nonzero_digit(const char* first, const char* last) noexcept
{
for (; first != last; ++first)
{
if (*first != '0')
{
return true;
}
}
return false;
}
/*!
@brief accumulate the validated digits [p, last) into @a w, as long as
@a remaining digits still fit into it
Eight digits are taken at a time while at least eight remain, and @a remaining
is decreased by the number of digits taken.
@return a pointer past the last digit taken
*/
JSON_HEDLEY_ALWAYS_INLINE const char* accumulate_significant_digits(const char* p, const char* last, int& remaining, std::uint64_t& w) noexcept
{
while (remaining >= 8 && last - p >= 8)
{
w = (w * 100000000u) + parse_eight_digits(read_eight_bytes(p));
p += 8;
remaining -= 8;
}
for (; remaining > 0 && p != last; ++p, --remaining)
{
w = (w * 10u) + static_cast<std::uint64_t>(*p - '0');
}
return p;
}
/// the value of the validated exponent digits [+-]?[0-9]+ in [first, last),
@@ -327,7 +390,7 @@ truncated is set.
std::string::npos if there is none
@param[in] mantissa_end index of the 'e'/'E', or the token length
*/
inline float_significand parse_float_significand(const char* first, const char* last,
JSON_HEDLEY_ALWAYS_INLINE float_significand parse_float_significand(const char* first, const char* last,
std::size_t decimal_point_position, std::size_t mantissa_end) noexcept
{
float_significand s;
@@ -342,16 +405,7 @@ inline float_significand parse_float_significand(const char* first, const char*
int remaining = 19; // digits that still fit into w
if (*p != '0') // the integer part is "0" or [1-9][0-9]*
{
while (remaining >= 8 && integer_last - p >= 8)
{
w = (w * 100000000u) + parse_eight_digits(read_eight_bytes(p));
p += 8;
remaining -= 8;
}
for (; remaining > 0 && p != integer_last; ++p, --remaining)
{
w = (w * 10u) + static_cast<std::uint64_t>(*p - '0');
}
p = accumulate_significant_digits(p, integer_last, remaining, w);
s.exponent = integer_last - p;
s.truncated = has_nonzero_digit(p, integer_last);
}
@@ -370,16 +424,7 @@ inline float_significand parse_float_significand(const char* first, const char*
s.exponent -= p - zeros;
}
const char* const digits = p;
while (remaining >= 8 && mantissa_last - p >= 8)
{
w = (w * 100000000u) + parse_eight_digits(read_eight_bytes(p));
p += 8;
remaining -= 8;
}
for (; remaining > 0 && p != mantissa_last; ++p, --remaining)
{
w = (w * 10u) + static_cast<std::uint64_t>(*p - '0');
}
p = accumulate_significant_digits(p, mantissa_last, remaining, w);
s.exponent -= p - digits;
s.truncated = s.truncated || has_nonzero_digit(p, mantissa_last);
}
@@ -712,15 +757,14 @@ std::uint64_t digit_comparison(const char* first, const char* last, std::uint64_
inline void float_from_bits(std::uint64_t bits, double& value) noexcept
{
static_assert(sizeof(double) == sizeof(std::uint64_t), "double must have 64 bits");
std::memcpy(&value, &bits, sizeof(value));
value = bit_cast<double>(bits);
}
/// the float with the IEEE-754 bits @a bits (the lower 32)
inline void float_from_bits(std::uint64_t bits, float& value) noexcept
{
static_assert(sizeof(float) == sizeof(std::uint32_t), "float must have 32 bits");
const auto bits32 = static_cast<std::uint32_t>(bits);
std::memcpy(&value, &bits32, sizeof(bits32));
value = bit_cast<float>(static_cast<std::uint32_t>(bits));
}
/// the powers of ten that are exact in binary64 (up to 10^22)
include/nlohmann/detail/input/parser.hpp
+50 -11
View File
@@ -51,6 +51,20 @@ template<typename BasicJsonType>
using parser_callback_t =
std::function<bool(int /*depth*/, parse_event_t /*event*/, BasicJsonType& /*parsed*/)>;
/// whether a SAX handler ignores the text of float tokens (the library's own
/// handlers do), so that the parser need not materialize it
template<typename SAX>
struct sax_ignores_number_text : std::false_type {};
template<typename BasicJsonType, typename InputAdapterType, bool Collect>
struct sax_ignores_number_text<json_sax_dom_parser<BasicJsonType, InputAdapterType, Collect>> : std::true_type {};
template<typename BasicJsonType, typename InputAdapterType>
struct sax_ignores_number_text<json_sax_dom_callback_parser<BasicJsonType, InputAdapterType>> : std::true_type {};
template<typename BasicJsonType>
struct sax_ignores_number_text<json_sax_acceptor<BasicJsonType>> : std::true_type {};
/*!
@brief syntax analysis
@@ -117,7 +131,7 @@ class parser
}
else
{
json_sax_dom_parser<BasicJsonType, InputAdapterType> sdp(result, allow_exceptions, &m_lexer);
json_sax_dom_parser<BasicJsonType, InputAdapterType, true> sdp(result, allow_exceptions, &m_lexer);
// in case of an error, return a discarded value
if (!parse_dom(sdp, strict))
@@ -171,6 +185,23 @@ class parser
}
private:
/// the text of the current float token for @a sax (see sax_ignores_number_text)
template<typename SAX>
const string_t& number_float_text(SAX* /*sax*/)
{
return number_float_text(std::integral_constant<bool, sax_ignores_number_text<SAX>::value> {});
}
const string_t& number_float_text(std::true_type /*ignored*/) const noexcept
{
return m_lexer.get_ignored_number_text();
}
string_t& number_float_text(std::false_type /*ignored*/)
{
return m_lexer.get_string();
}
/*!
@brief run a DOM SAX parser to completion and position the lexer
@@ -212,12 +243,20 @@ class parser
JSON_HEDLEY_NON_NULL(2)
bool sax_parse_internal(SAX* sax)
{
// stack to remember the hierarchy of structured values we are parsing
// true = array; false = object
std::vector<bool> states;
// value to avoid a goto (see comment where set to true)
bool skip_to_state_evaluation = false;
// stack to remember the hierarchy of structured values we are parsing
// true = array; false = object (bytes, not std::vector<bool>, whose
// push_back() is a call with GCC)
std::vector<std::uint8_t> states;
// a document that starts with a container pushes onto the stack at
// once: allocate it in one step instead of growing it from 1 byte
if (last_token == token_type::begin_object || last_token == token_type::begin_array)
{
states.reserve(32);
}
while (true)
{
if (!skip_to_state_evaluation)
@@ -247,7 +286,7 @@ class parser
{
return syntax_error(*sax, exception_message(token_type::value_string, "object key"));
}
if (JSON_HEDLEY_UNLIKELY(!sax->key(m_lexer.get_string())))
if (JSON_HEDLEY_UNLIKELY(!sax->key(m_lexer.get_string_token())))
{
return false;
}
@@ -259,7 +298,7 @@ class parser
}
// remember we are now inside an object
states.push_back(false);
states.push_back(0);
// parse values
get_token();
@@ -284,7 +323,7 @@ class parser
}
// remember we are now inside an array
states.push_back(true);
states.push_back(1);
// parse values (no need to call get_token)
continue;
@@ -301,7 +340,7 @@ class parser
out_of_range::create(exception_id::number_overflow, concat("number overflow parsing '", m_lexer.get_token_string(), '\''), nullptr));
}
if (JSON_HEDLEY_UNLIKELY(!sax->number_float(res, m_lexer.get_string())))
if (JSON_HEDLEY_UNLIKELY(!sax->number_float(res, number_float_text(sax))))
{
return false;
}
@@ -347,7 +386,7 @@ class parser
case token_type::value_string:
{
if (JSON_HEDLEY_UNLIKELY(!sax->string(m_lexer.get_string())))
if (JSON_HEDLEY_UNLIKELY(!sax->string(m_lexer.get_string_token())))
{
return false;
}
@@ -401,7 +440,7 @@ class parser
return true;
}
if (states.back()) // array
if (states.back() != 0) // array
{
// comma -> next value
// or end of array (ignore_trailing_commas = true)
@@ -456,7 +495,7 @@ class parser
return syntax_error(*sax, exception_message(token_type::value_string, "object key"));
}
if (JSON_HEDLEY_UNLIKELY(!sax->key(m_lexer.get_string())))
if (JSON_HEDLEY_UNLIKELY(!sax->key(m_lexer.get_string_token())))
{
return false;
}
include/nlohmann/detail/input/simd_scan.hpp
+321
View File
@@ -0,0 +1,321 @@
// __ _____ _____ _____
// __| | __| | | | JSON for Modern C++
// | | |__ | | | | | | version 3.12.0
// |_____|_____|_____|_|___| https://github.com/nlohmann/json
//
// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
// SPDX-FileCopyrightText: 2018-2025 The simdjson authors <https://github.com/simdjson/simdjson>
// SPDX-License-Identifier: MIT
#pragma once
#include <array> // array
#include <atomic> // atomic
#include <cstddef> // size_t
#include <cstdint> // uint8_t, uint64_t
#include <nlohmann/detail/bit_ops.hpp>
#include <nlohmann/detail/macro_scope.hpp>
// Vector code for runs of string bytes, used by the lexer for contiguous input
// (see string_bulk_run()). NEON (AArch64) and SSE2 (x86-64) belong to the
// baseline instruction sets and are used by default. The vector UTF-8 check
// needs NEON or SSSE3. SSSE3 is not part of x86-64, and the code must not
// depend on the flags of a translation unit (two translation units with
// different flags would have different definitions of the same inline
// functions): the check is compiled for SSSE3 with a function attribute and
// used where the CPU has SSSE3 (all x86-64 CPUs since about 2011), else the
// portable check. Code compiled for SSSE3 anyway (-mssse3 or more) skips the
// CPU check. JSON_NO_SIMD selects the portable code everywhere.
// NLOHMANN_JSON_NO_TARGET_ATTRIBUTE (for compilers that ignore the target
// attribute and then reject the SSSE3 intrinsics) keeps the check portable
// without disabling the SSE2 code.
#if !defined(JSON_NO_SIMD) && defined(__aarch64__) && (defined(__GNUC__) || defined(__clang__)) && defined(__BYTE_ORDER__) && defined(__ORDER_LITTLE_ENDIAN__) && __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
#include <arm_neon.h>
#define JSON_SCAN_NEON 1
#else
#define JSON_SCAN_NEON 0
#endif
// (x86 only: other targets can define __SSE2__ as well, e.g., WebAssembly with -msse2, but have no <cpuid.h>)
#if !defined(JSON_NO_SIMD) && !JSON_SCAN_NEON && (defined(__x86_64__) || defined(__i386__) || defined(_M_X64) || defined(_M_IX86)) && !defined(_M_ARM64EC) && (defined(__SSE2__) || defined(_M_X64) || (defined(_M_IX86_FP) && _M_IX86_FP >= 2))
#include <emmintrin.h>
#define JSON_SCAN_SSE2 1
#else
#define JSON_SCAN_SSE2 0
#endif
#if JSON_SCAN_SSE2 && defined(__SSSE3__)
#include <tmmintrin.h>
#define JSON_SCAN_SSSE3 1 // NOLINT(cppcoreguidelines-macro-to-enum,modernize-macro-to-enum)
#else
#define JSON_SCAN_SSSE3 0 // NOLINT(cppcoreguidelines-macro-to-enum,modernize-macro-to-enum)
#endif
#if JSON_SCAN_SSE2 && !JSON_SCAN_SSSE3 && !defined(NLOHMANN_JSON_NO_TARGET_ATTRIBUTE) && ((defined(__clang__) && __clang_major__ >= 4) || (defined(__GNUC__) && !defined(__clang__) && (__GNUC__ > 4 || (__GNUC__ == 4 && __GNUC_MINOR__ >= 9)) && !defined(__cpp_modules)))
// (GCC before 4.9 has no SSSE3 intrinsics without -mssse3; GCC with C++20
// modules drops the target attribute from functions it reads from a module
// interface and then rejects the intrinsics: "needs isa option -mssse3")
// <cpuid.h> of GCC before 11 has no include guard, and json.hpp and
// json_view.hpp both include it
#ifndef NLOHMANN_JSON_CPUID_INCLUDED
#define NLOHMANN_JSON_CPUID_INCLUDED
#include <cpuid.h>
#endif
#include <tmmintrin.h>
#define JSON_SCAN_SSSE3_DISPATCH 1 // NOLINT(cppcoreguidelines-macro-to-enum,modernize-macro-to-enum)
#define JSON_SCAN_SSSE3_TARGET __attribute__((target("ssse3")))
#elif JSON_SCAN_SSE2 && !JSON_SCAN_SSSE3 && defined(_MSC_VER)
// (MSVC compiles intrinsics of any instruction set)
#include <intrin.h>
#include <tmmintrin.h>
#define JSON_SCAN_SSSE3_DISPATCH 1 // NOLINT(cppcoreguidelines-macro-to-enum,modernize-macro-to-enum)
#define JSON_SCAN_SSSE3_TARGET
#else
#define JSON_SCAN_SSSE3_DISPATCH 0 // NOLINT(cppcoreguidelines-macro-to-enum,modernize-macro-to-enum)
#define JSON_SCAN_SSSE3_TARGET
#endif
#define JSON_SCAN_VECTOR (JSON_SCAN_NEON || JSON_SCAN_SSE2)
#define JSON_SCAN_VECTOR_UTF8 (JSON_SCAN_NEON || JSON_SCAN_SSSE3 || JSON_SCAN_SSSE3_DISPATCH)
NLOHMANN_JSON_NAMESPACE_BEGIN
namespace detail
{
#if JSON_SCAN_VECTOR
/*!
@brief the first byte of a string run that is a quote, a backslash, a control
character, or not ASCII, 16 bytes per step
Stops at such a byte, or where fewer than 16 bytes are left (the caller tells
the two apart). A signed compare with 0x20 finds control characters and
non-ASCII bytes at once.
*/
JSON_HEDLEY_ALWAYS_INLINE const unsigned char* vector_plain_run(const unsigned char* p, const unsigned char* e) noexcept
{
while (e - p >= 16)
{
#if JSON_SCAN_NEON
const uint8x16_t in = vld1q_u8(p);
const uint8x16_t special = vorrq_u8(vorrq_u8(vceqq_u8(in, vdupq_n_u8('"')), vceqq_u8(in, vdupq_n_u8('\\'))),
vcltq_s8(vreinterpretq_s8_u8(in), vdupq_n_s8(0x20)));
// one nibble per byte (the usual NEON replacement of x86's movemask, see
// D. Kutenin, "Porting x86 vector bitmask optimizations to Arm NEON", 2022)
const std::uint64_t bits = vget_lane_u64(vreinterpret_u64_u8(vshrn_n_u16(vreinterpretq_u16_u8(special), 4)), 0);
if (bits != 0)
{
return p + (count_trailing_zeros(bits) >> 2u);
}
#else
const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(p))); // NOLINT(bugprone-casting-through-void)
const __m128i special = _mm_or_si128(_mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('"')), _mm_cmpeq_epi8(in, _mm_set1_epi8('\\'))),
_mm_cmplt_epi8(in, _mm_set1_epi8(0x20)));
const auto bits = static_cast<std::uint64_t>(static_cast<unsigned>(_mm_movemask_epi8(special)));
if (bits != 0)
{
return p + count_trailing_zeros(bits);
}
#endif
p += 16;
}
return p;
}
#endif
#if JSON_SCAN_VECTOR
/// vector_plain_run() for runs that are already longer than a few words:
/// inline with SSE2, out of line with NEON, where a larger string loop slows
/// down the rest of the lexer
#if JSON_SCAN_NEON
JSON_HEDLEY_NEVER_INLINE inline
#else
JSON_HEDLEY_ALWAYS_INLINE
#endif
const unsigned char* vector_plain_run_long(const unsigned char* p, const unsigned char* e) noexcept
{
return vector_plain_run(p, e);
}
#endif
#if JSON_SCAN_SSSE3_DISPATCH
/// whether the CPU has SSSE3 (CPUID leaf 1, ECX bit 9)
inline bool cpu_ssse3() noexcept
{
#if defined(_MSC_VER) && !defined(__clang__)
std::array<int, 4> regs {{}};
__cpuid(regs.data(), 1);
return (static_cast<unsigned>(regs[2]) & (1u << 9u)) != 0;
#else
unsigned eax = 0;
unsigned ebx = 0;
unsigned ecx = 0;
unsigned edx = 0;
return __get_cpuid(1, &eax, &ebx, &ecx, &edx) != 0 && (ecx & (1u << 9u)) != 0;
#endif
}
/// whether the CPU has SSSE3, asked once: the answer is kept in an atomic
/// that is initialized at compile time, so that neither a guard of a local
/// static nor a global constructor is needed (threads that ask at the same
/// time all store the same answer)
JSON_HEDLEY_ALWAYS_INLINE bool cpu_has_ssse3() noexcept
{
static std::atomic<int> known{0}; // 0: not asked yet, 1: no, 2: yes
int state = known.load(std::memory_order_relaxed);
if (JSON_HEDLEY_UNLIKELY(state == 0))
{
state = cpu_ssse3() ? 2 : 1;
known.store(state, std::memory_order_relaxed);
}
return state == 2;
}
#endif
#if JSON_SCAN_VECTOR_UTF8
/// Tables of the UTF-8 check of J. Keiser and D. Lemire, "Validating UTF-8 In
/// Less Than One Instruction Per Byte" (2021), as in simdjson ("lookup4"): each
/// maps a nibble (high and low nibble of the previous byte, high nibble of the
/// current byte) to the errors it allows; a byte pair is ill-formed if all
/// three have an error bit in common.
template<typename Dummy = void>
struct utf8_lookup4
{
static constexpr std::uint8_t too_short = 1u << 0u, too_long = 1u << 1u, overlong_3 = 1u << 2u, too_large = 1u << 3u;
static constexpr std::uint8_t surrogate = 1u << 4u, overlong_2 = 1u << 5u, too_large_1000 = 1u << 6u, overlong_4 = 1u << 6u;
static constexpr std::uint8_t two_conts = 1u << 7u, carry = too_short | too_long | two_conts;
static const std::array<std::uint8_t, 16> byte_1_high;
static const std::array<std::uint8_t, 16> byte_1_low;
static const std::array<std::uint8_t, 16> byte_2_high;
};
template<typename Dummy>
const std::array<std::uint8_t, 16> utf8_lookup4<Dummy>::byte_1_high =
{
{
too_long, too_long, too_long, too_long, too_long, too_long, too_long, too_long,
two_conts, two_conts, two_conts, two_conts,
too_short | overlong_2, too_short, too_short | overlong_3 | surrogate, too_short | too_large | too_large_1000 | overlong_4
}
};
template<typename Dummy>
const std::array<std::uint8_t, 16> utf8_lookup4<Dummy>::byte_1_low =
{
{
carry | overlong_3 | overlong_2 | overlong_4, carry | overlong_2, carry, carry,
carry | too_large, carry | too_large | too_large_1000, carry | too_large | too_large_1000, carry | too_large | too_large_1000,
carry | too_large | too_large_1000, carry | too_large | too_large_1000, carry | too_large | too_large_1000, carry | too_large | too_large_1000,
carry | too_large | too_large_1000, carry | too_large | too_large_1000 | surrogate, carry | too_large | too_large_1000, carry | too_large | too_large_1000
}
};
template<typename Dummy>
const std::array<std::uint8_t, 16> utf8_lookup4<Dummy>::byte_2_high =
{
{
too_short, too_short, too_short, too_short, too_short, too_short, too_short, too_short,
static_cast<std::uint8_t>(too_long | overlong_2 | two_conts | overlong_3 | too_large_1000 | overlong_4),
static_cast<std::uint8_t>(too_long | overlong_2 | two_conts | overlong_3 | too_large),
static_cast<std::uint8_t>(too_long | overlong_2 | two_conts | surrogate | too_large),
static_cast<std::uint8_t>(too_long | overlong_2 | two_conts | surrogate | too_large),
too_short, too_short, too_short, too_short
}
};
/*!
@brief the run of plain string bytes and well-formed UTF-8 at p (a character
boundary), 16 bytes per step
The first quote, backslash, or control character is found with vector
compares, and the UTF-8 check of J. Keiser and D. Lemire covers the bytes up
to it. Returns true with @a block at that byte; returns false with @a block at
the start of the 16 bytes where the check found ill-formed UTF-8 or where
fewer than 16 bytes are left, for the caller to finish one sequence at a time.
Out of line, so that no constants of the check occupy registers in the
caller's loop. On x86-64, it is compiled for SSSE3 (see cpu_has_ssse3()).
*/
JSON_SCAN_SSSE3_TARGET JSON_HEDLEY_NEVER_INLINE inline bool vector_utf8_run(const unsigned char* p, const unsigned char* e, const unsigned char*& block_out) noexcept
{
using lookup = utf8_lookup4<>;
const unsigned char* block = p;
#if JSON_SCAN_NEON
const uint8x16_t t1h = vld1q_u8(lookup::byte_1_high.data());
const uint8x16_t t1l = vld1q_u8(lookup::byte_1_low.data());
const uint8x16_t t2h = vld1q_u8(lookup::byte_2_high.data());
uint8x16_t prev = vdupq_n_u8(0);
while (e - block >= 16)
{
const uint8x16_t in = vld1q_u8(block);
const uint8x16_t special = vorrq_u8(vorrq_u8(vceqq_u8(in, vdupq_n_u8('"')), vceqq_u8(in, vdupq_n_u8('\\'))), vcltq_u8(in, vdupq_n_u8(0x20)));
const uint8x16_t prev1 = vextq_u8(prev, in, 15);
const uint8x16_t sc = vandq_u8(vandq_u8(vqtbl1q_u8(t1h, vshrq_n_u8(prev1, 4)), vqtbl1q_u8(t1l, vandq_u8(prev1, vdupq_n_u8(0x0F)))), vqtbl1q_u8(t2h, vshrq_n_u8(in, 4)));
const uint8x16_t must23 = vorrq_u8(vqsubq_u8(vextq_u8(prev, in, 14), vdupq_n_u8(0xE0 - 0x80)), vqsubq_u8(vextq_u8(prev, in, 13), vdupq_n_u8(0xF0 - 0x80)));
const uint8x16_t err = veorq_u8(vandq_u8(must23, vdupq_n_u8(0x80)), sc);
const std::uint64_t special_bits = vget_lane_u64(vreinterpret_u64_u8(vshrn_n_u16(vreinterpretq_u16_u8(special), 4)), 0);
const std::uint64_t err_bits = vget_lane_u64(vreinterpret_u64_u8(vshrn_n_u16(vreinterpretq_u16_u8(vtstq_u8(err, err)), 4)), 0);
if (special_bits != 0)
{
// errors up to the special byte count (an incomplete sequence
// before a quote shows at the quote); the bytes after it do not
const unsigned k = static_cast<unsigned>(count_trailing_zeros(special_bits)) >> 2u;
const std::uint64_t upto = k == 15 ? ~std::uint64_t{0} :
(std::uint64_t{1} << (4u * (k + 1u))) - 1u;
if ((err_bits & upto) == 0)
{
block_out = block + k;
return true;
}
break;
}
if (err_bits != 0)
{
break;
}
prev = in;
block += 16;
}
#else
// the same with SSSE3 (pshufb for the table lookups; nibbles from 16-bit
// shifts, as there are no byte shifts)
const __m128i t1h = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(lookup::byte_1_high.data()))); // NOLINT(bugprone-casting-through-void)
const __m128i t1l = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(lookup::byte_1_low.data()))); // NOLINT(bugprone-casting-through-void)
const __m128i t2h = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(lookup::byte_2_high.data()))); // NOLINT(bugprone-casting-through-void)
const __m128i nibble = _mm_set1_epi8(0x0F);
const __m128i zero = _mm_setzero_si128();
__m128i prev = zero;
while (e - block >= 16)
{
const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(block))); // NOLINT(bugprone-casting-through-void)
const __m128i special = _mm_or_si128(_mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('"')), _mm_cmpeq_epi8(in, _mm_set1_epi8('\\'))),
_mm_cmpeq_epi8(_mm_subs_epu8(in, _mm_set1_epi8(0x1F)), zero)); // in < 0x20
const __m128i prev1 = _mm_alignr_epi8(in, prev, 15);
const __m128i sc = _mm_and_si128(_mm_and_si128(_mm_shuffle_epi8(t1h, _mm_and_si128(_mm_srli_epi16(prev1, 4), nibble)),
_mm_shuffle_epi8(t1l, _mm_and_si128(prev1, nibble))),
_mm_shuffle_epi8(t2h, _mm_and_si128(_mm_srli_epi16(in, 4), nibble)));
const __m128i must23 = _mm_or_si128(_mm_subs_epu8(_mm_alignr_epi8(in, prev, 14), _mm_set1_epi8(0xE0 - 0x80)),
_mm_subs_epu8(_mm_alignr_epi8(in, prev, 13), _mm_set1_epi8(0xF0 - 0x80)));
const __m128i err = _mm_xor_si128(_mm_and_si128(must23, _mm_set1_epi8(static_cast<char>(-128))), sc);
const auto special_bits = static_cast<unsigned>(_mm_movemask_epi8(special));
const auto err_bits = ~static_cast<unsigned>(_mm_movemask_epi8(_mm_cmpeq_epi8(err, zero))) & 0xFFFFu;
if (special_bits != 0)
{
const auto k = static_cast<unsigned>(count_trailing_zeros(static_cast<std::uint64_t>(special_bits)));
if ((err_bits & ((2u << k) - 1u)) == 0)
{
block_out = block + k;
return true;
}
break;
}
if (err_bits != 0)
{
break;
}
prev = in;
block += 16;
}
#endif
block_out = block;
return false;
}
#endif
} // namespace detail
NLOHMANN_JSON_NAMESPACE_END
include/nlohmann/detail/input/string_scan.hpp
+161 -16
View File
@@ -14,6 +14,7 @@
#include <cstring> // memcpy
#include <nlohmann/detail/bit_ops.hpp>
#include <nlohmann/detail/input/simd_scan.hpp>
#include <nlohmann/detail/macro_scope.hpp>
// Optional SIMD backend for bulk UTF-8 validation. This is an opt-in external
@@ -34,7 +35,7 @@
// This file contains the byte-level string-scanning helpers used by the lexer's
// contiguous fast path. They operate purely on raw bytes (no dependency on the
// lexer's template parameters) so they are free functions, keeping the lexer
// itself focused on the state machine; see lexer::scan_string_bulk().
// itself focused on the state machine; see lexer::get_after_string_run().
NLOHMANN_JSON_NAMESPACE_BEGIN
namespace detail
@@ -66,7 +67,7 @@ inline std::uint64_t swar_string_special(std::uint64_t v) noexcept
// return the index of the first is_string_special() byte in [data, data+n), or
// n if every byte is ordinary; scans 8 bytes at a time
inline std::size_t find_string_special(const unsigned char* data, std::size_t n) noexcept
JSON_HEDLEY_ALWAYS_INLINE std::size_t find_string_special(const unsigned char* data, std::size_t n) noexcept
{
std::size_t i = 0;
for (; i + 8 <= n; i += 8)
@@ -235,35 +236,143 @@ inline std::size_t valid_utf8_prefix(const unsigned char* data, std::size_t n) n
return pos;
}
// Scalar (C++11) computation of the bulk run length: the number of leading
// bytes in [data, data+n) that are ordinary ASCII or complete well-formed UTF-8
#if JSON_SCAN_VECTOR_UTF8
/// the end of scan_string_vector from block, where the vector loop stopped
/// (ill-formed UTF-8, or fewer than 16 bytes left): one byte or sequence at a
/// time, from the start of a sequence that crosses into the block
inline const unsigned char* scan_string_finish(const unsigned char* p, const unsigned char* block, const unsigned char* e) noexcept
{
for (int i = 1; i <= 3 && block - i >= p; ++i)
{
const unsigned char c = block[-i];
if (c < 0x80)
{
break;
}
if (c >= 0xC0)
{
const int len = 2 + static_cast<int>(c >= 0xE0) + static_cast<int>(c >= 0xF0);
if (len > i)
{
block -= i;
}
break;
}
}
for (p = block; p != e;)
{
if (*p < 0x80)
{
if (is_string_special(*p))
{
return p;
}
++p;
continue;
}
const std::size_t n = validate_one_utf8(p, static_cast<std::size_t>(e - p));
if (n == 0)
{
return p;
}
p += n;
}
return p;
}
/// the rest of a string from p (a character boundary), with the vector UTF-8
/// check and the scalar check for the end of the block where it stopped
inline const unsigned char* scan_string_vector(const unsigned char* p, const unsigned char* e) noexcept
{
const unsigned char* block = p;
if (vector_utf8_run(p, e, block))
{
return block;
}
return scan_string_finish(p, block, e);
}
#endif
#if JSON_SCAN_VECTOR
/// the end of the run of plain string characters at @a p: the first quote,
/// backslash, control character, or non-ASCII byte, or @a e. The first words
/// are checked with SWAR (one; four with NEON, where the vector mask costs
/// more), as most keys and short values end there; longer runs continue 16
/// bytes at a time.
JSON_HEDLEY_ALWAYS_INLINE const unsigned char* plain_run_end(const unsigned char* p, const unsigned char* e) noexcept
{
constexpr int prefix_words = JSON_SCAN_NEON ? 4 : 1;
for (int words = prefix_words; words > 0 && e - p >= 8; --words)
{
const std::uint64_t special = swar_string_special(read_eight_bytes(p));
if (special != 0)
{
return p + (static_cast<std::size_t>(count_trailing_zeros(special)) / 8);
}
p += 8;
}
// the vector loop only for 16 bytes or more: the serializer passes a
// string without its closing quote, so most of its runs end at e, and a
// call of the out-of-line NEON loop for a shorter rest made dump() up to
// 10% slower
if (e - p >= 16)
{
p = vector_plain_run_long(p, e);
if (e - p >= 16)
{
return p; // the vector loop stopped at a special byte
}
}
return p + find_string_special(p, static_cast<std::size_t>(e - p));
}
#endif
// Computation of the bulk run length: the number of leading bytes in
// [data, data+n) that are ordinary ASCII or complete well-formed UTF-8
// sequences, stopping before the first byte that needs individual handling (the
// closing quote, an escape, a control character, or an ill-formed/truncated
// sequence). ASCII is skipped 8 bytes at a time.
inline std::size_t scalar_string_bulk_run(const unsigned char* data, std::size_t n) noexcept
// sequence). ASCII is skipped 16 bytes at a time with SSE2 and 8 bytes at a
// time otherwise; runs of non-ASCII text use the vector UTF-8 check where
// available.
JSON_HEDLEY_ALWAYS_INLINE std::size_t scalar_string_bulk_run(const unsigned char* data, std::size_t n) noexcept
{
std::size_t pos = 0;
while (pos < n)
const unsigned char* const e = data + n;
const unsigned char* p = data;
while (p != e)
{
pos += find_string_special(data + pos, n - pos);
if (pos >= n || data[pos] < 0x80u)
#if JSON_SCAN_VECTOR
p = plain_run_end(p, e);
#else
p += find_string_special(p, static_cast<std::size_t>(e - p));
#endif
if (p == e || *p < 0x80u)
{
break; // end of buffer, or a quote/escape/control byte
}
#if JSON_SCAN_VECTOR_UTF8
#if JSON_SCAN_SSSE3_DISPATCH
if (JSON_HEDLEY_LIKELY(cpu_has_ssse3()))
#endif
{
return static_cast<std::size_t>(scan_string_vector(p, e) - data);
}
#endif
#if !JSON_SCAN_VECTOR_UTF8 || JSON_SCAN_SSSE3_DISPATCH
// a run of multi-byte sequences (e.g. CJK text) is validated sequence
// by sequence without searching for the next special byte in between
do
{
const std::size_t seq = validate_one_utf8(data + pos, n - pos);
const std::size_t seq = validate_one_utf8(p, static_cast<std::size_t>(e - p));
if (seq == 0)
{
return pos; // ill-formed or truncated: let the byte path diagnose it
return static_cast<std::size_t>(p - data); // ill-formed or truncated: let the byte path diagnose it
}
pos += seq;
p += seq;
}
while (pos < n && data[pos] >= 0x80u);
while (p != e && *p >= 0x80u);
#endif
}
return pos;
return static_cast<std::size_t>(p - data);
}
#if defined(JSON_USE_SIMDUTF) && defined(JSON_HAS_CPP_17)
@@ -304,7 +413,7 @@ inline std::size_t find_string_delimiter(const unsigned char* data, std::size_t
// next delimiter is validated in one shot by simdutf; on the rare failure the
// scalar helper recomputes the exact valid prefix so the byte path still
// produces the precise diagnostic. Without it, the pure scalar path is used.
inline std::size_t string_bulk_run(const unsigned char* data, std::size_t n) noexcept
JSON_HEDLEY_ALWAYS_INLINE std::size_t string_bulk_run(const unsigned char* data, std::size_t n) noexcept
{
#if defined(JSON_USE_SIMDUTF) && defined(JSON_HAS_CPP_17)
const std::size_t run = find_string_delimiter(data, n);
@@ -316,6 +425,42 @@ inline std::size_t string_bulk_run(const unsigned char* data, std::size_t n) noe
return scalar_string_bulk_run(data, n);
}
#if JSON_SCAN_NEON && !(defined(JSON_USE_SIMDUTF) && defined(JSON_HAS_CPP_17))
/// scalar_string_bulk_run() from @a start on, out of line (see
/// lexer_string_bulk_run())
JSON_HEDLEY_NEVER_INLINE inline std::size_t string_bulk_run_from(const unsigned char* data, std::size_t start, std::size_t n) noexcept
{
return start + scalar_string_bulk_run(data + start, n - start);
}
#endif
// string_bulk_run() for the lexer. With NEON, only the first four words are
// checked inline (most keys and short values end there), and longer runs and
// non-ASCII text continue out of line. The lexer's scan function runs once per
// token, and the vector code inlined into it needs more registers, which every
// token pays for: 5% of json::accept() on jeopardy.json with Apple Clang on
// Apple M1, even though no string of that file reaches the vector code. The
// serializer keeps string_bulk_run() inline, which is faster there.
JSON_HEDLEY_ALWAYS_INLINE std::size_t lexer_string_bulk_run(const unsigned char* data, std::size_t n) noexcept
{
#if JSON_SCAN_NEON && !(defined(JSON_USE_SIMDUTF) && defined(JSON_HAS_CPP_17))
std::size_t i = 0;
for (int words = 4; words > 0 && n - i >= 8; --words)
{
const std::uint64_t special = swar_string_special(read_eight_bytes(data + i));
if (special != 0)
{
const std::size_t pos = i + (static_cast<std::size_t>(count_trailing_zeros(special)) / 8);
return data[pos] < 0x80u ? pos : string_bulk_run_from(data, pos, n);
}
i += 8;
}
return string_bulk_run_from(data, i, n);
#else
return string_bulk_run(data, n);
#endif
}
// Decode the 4 hex digits at [data, data+4) - the digits following a `\u`
// escape - into a codepoint 0x0000..0xFFFF via one table lookup per byte
// (after yyjson's read_hex_u16), or return -1 if any of the 4 bytes is not a
include/nlohmann/detail/json_pointer.hpp
+625 -206
View File
File diff suppressed because it is too large. Load diff
include/nlohmann/detail/macro_scope.hpp
+16
View File
@@ -187,6 +187,22 @@
#endif
#endif
// the byte order of the target where the compiler tells it (std::endian is
// C++20 and cannot select code in #if): JSON_LITTLE_ENDIAN_TARGET or
// JSON_BIG_ENDIAN_TARGET is 1, or both are 0 where it is unknown (code then
// handles the bytes one at a time); MSVC and x86 targets are little-endian
#if defined(_MSC_VER) || defined(__x86_64__) || defined(__i386__) \
|| (defined(__BYTE_ORDER__) && defined(__ORDER_LITTLE_ENDIAN__) && __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__)
#define JSON_LITTLE_ENDIAN_TARGET 1 // NOLINT(cppcoreguidelines-macro-to-enum,modernize-macro-to-enum)
#define JSON_BIG_ENDIAN_TARGET 0 // NOLINT(cppcoreguidelines-macro-to-enum,modernize-macro-to-enum)
#elif defined(__BYTE_ORDER__) && defined(__ORDER_BIG_ENDIAN__) && __BYTE_ORDER__ == __ORDER_BIG_ENDIAN__
#define JSON_LITTLE_ENDIAN_TARGET 0 // NOLINT(cppcoreguidelines-macro-to-enum,modernize-macro-to-enum)
#define JSON_BIG_ENDIAN_TARGET 1 // NOLINT(cppcoreguidelines-macro-to-enum,modernize-macro-to-enum)
#else
#define JSON_LITTLE_ENDIAN_TARGET 0 // NOLINT(cppcoreguidelines-macro-to-enum,modernize-macro-to-enum)
#define JSON_BIG_ENDIAN_TARGET 0 // NOLINT(cppcoreguidelines-macro-to-enum,modernize-macro-to-enum)
#endif
#ifdef JSON_HAS_CPP_17
#define JSON_INLINE_VARIABLE inline
#else
include/nlohmann/detail/macro_unscope.hpp
+11 -2
View File
@@ -19,9 +19,20 @@
#undef NLOHMANN_CAN_CALL_STD_FUNC_IMPL
#undef JSON_INLINE_VARIABLE
#undef JSON_NO_UNIQUE_ADDRESS
#undef JSON_DISABLE_ENUM_SERIALIZATION
#undef JSON_DISABLE_TUPLE_REFERENCE_CONVERSION
#undef JSON_UNLOCKED_FILE_IO
#undef JSON_DTOA_SSE2
#undef JSON_DTOA_NEON
#undef JSON_LITTLE_ENDIAN_TARGET
#undef JSON_BIG_ENDIAN_TARGET
#undef JSON_SCAN_NEON
#undef JSON_SCAN_SSE2
#undef JSON_SCAN_SSSE3
#undef JSON_SCAN_SSSE3_DISPATCH
#undef JSON_SCAN_SSSE3_TARGET
#undef JSON_SCAN_VECTOR
#undef JSON_SCAN_VECTOR_UTF8
#ifndef JSON_TEST_KEEP_MACROS
#undef JSON_CATCH
@@ -46,8 +57,6 @@
#undef JSON_STRICT_BINARY_UTF8
#undef JSON_DELETE_DEPRECATED_FUNCTIONS
#undef JSON_USE_OBJECTS_FOR_ENUM_KEYED_MAPS
#undef JSON_DISABLE_ENUM_SERIALIZATION
#undef JSON_DISABLE_TUPLE_REFERENCE_CONVERSION
#undef JSON_DISABLE_STREAM_BULK_SCAN
#undef JSON_DISABLE_UNLOCKED_FILE_IO
#endif
include/nlohmann/detail/meta/cpp_future.hpp
+26 -3
View File
@@ -10,17 +10,26 @@
#pragma once
#include <cstddef> // size_t
#include <type_traits> // conditional, enable_if, false_type, integral_constant, is_constructible, is_integral, is_same, remove_cv, remove_reference, true_type
#include <type_traits> // conditional, enable_if, false_type, integral_constant, is_arithmetic, is_constructible, is_integral, is_same, remove_cv, remove_reference, true_type
#include <utility> // index_sequence, make_index_sequence, index_sequence_for
#include <nlohmann/detail/macro_scope.hpp>
#ifdef JSON_HAS_CPP_17
#include <algorithm> // clamp
#endif
NLOHMANN_JSON_NAMESPACE_BEGIN
namespace detail
{
template<typename T>
using uncvref_t = typename std::remove_cv<typename std::remove_reference<T>::type>::type;
#ifdef __cpp_lib_remove_cvref
template<typename T>
using uncvref_t = std::remove_cvref_t<T>;
#else
template<typename T>
using uncvref_t = typename std::remove_cv<typename std::remove_reference<T>::type>::type;
#endif
#ifdef JSON_HAS_CPP_14
@@ -160,5 +169,19 @@ struct static_const
constexpr T static_const<T>::value;
#endif
/// @a v bounded to [@a lo, @a hi] (requires !(hi < lo)); like std::clamp,
/// but by value, so that static constexpr members can be passed without an
/// out-of-line definition before C++17
template<typename T, enable_if_t<std::is_arithmetic<T>::value, int> = 0>
constexpr T clamp(T v, T lo, T hi) noexcept
{
#ifdef __cpp_lib_clamp
return std::clamp(v, lo, hi);
#else
// NOLINTNEXTLINE(readability-avoid-nested-conditional-operator): a constant expression of C++11
return v < lo ? lo : (hi < v ? hi : v);
#endif
}
} // namespace detail
NLOHMANN_JSON_NAMESPACE_END
include/nlohmann/detail/meta/type_traits.hpp
+16 -2
View File
@@ -313,6 +313,14 @@ struct char_traits<std::byte> : std::char_traits<char>
// is_ functions //
///////////////////
#ifdef __cpp_lib_logical_traits
using std::conjunction;
using std::disjunction;
using std::negation;
#else
// https://en.cppreference.com/w/cpp/types/conjunction
template<class...> struct conjunction : std::true_type { };
template<class B> struct conjunction<B> : B { };
@@ -330,6 +338,8 @@ struct disjunction<B, Bn...>
// https://en.cppreference.com/w/cpp/types/negation
template<class B> struct negation : std::integral_constant < bool, !B::value > { };
#endif
// Reimplementation of is_constructible and is_default_constructible, due to them being broken for
// std::pair and std::tuple until LWG 2367 fix (see https://cplusplus.github.io/LWG/lwg-defects.html#2367).
// This causes compile errors in e.g., Clang 3.5 or GCC 4.9.
@@ -1060,8 +1070,12 @@ constexpr bool value_in_range_of(T val)
return value_in_range_of_impl1<OfType, T>::test(val);
}
template<bool Value>
using bool_constant = std::integral_constant<bool, Value>;
#ifdef __cpp_lib_bool_constant
using std::bool_constant;
#else
template<bool Value>
using bool_constant = std::integral_constant<bool, Value>;
#endif
// Stands in for number_unsigned_t in overload sets if it is the same type as
// number_integer_t, so that the unsigned overloads do not collide with the
include/nlohmann/detail/output/binary_writer.hpp
+1725 -220
View File
File diff suppressed because it is too large. Load diff
include/nlohmann/detail/output/output_adapters.hpp
+214 -33
View File
@@ -55,6 +55,66 @@ using output_adapter_t = std::shared_ptr<output_adapter_protocol<CharType>>;
/// output_vector_sink also appends writes of at least this size with insert()
constexpr std::size_t max_reserve_size = 256;
/// @brief copy @a length characters, at most 32, without a call of memcpy()
///
/// Short strings are the typical payload of a binary value, and a call of
/// memcpy() costs more than copying them. A copy of 16 to 32 characters is two
/// possibly overlapping 16-byte copies, one of 8 to 15 two 8-byte copies, and
/// so on down to a single character; each is a fixed-size load and store.
/// Nothing outside of [src, src + length) is read or written.
template<typename CharType>
inline void copy_short(CharType* dst, const CharType* src, const std::size_t length, std::true_type /*is_byte*/) noexcept
{
JSON_ASSERT(length <= 32);
if (length >= 16)
{
std::array<char, 16> first; // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init): written before read
std::array<char, 16> last; // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init): written before read
std::memcpy(first.data(), src, 16);
std::memcpy(last.data(), src + (length - 16), 16);
std::memcpy(dst, first.data(), 16);
std::memcpy(dst + (length - 16), last.data(), 16);
}
else if (length >= 8)
{
std::array<char, 8> first; // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init): written before read
std::array<char, 8> last; // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init): written before read
std::memcpy(first.data(), src, 8);
std::memcpy(last.data(), src + (length - 8), 8);
std::memcpy(dst, first.data(), 8);
std::memcpy(dst + (length - 8), last.data(), 8);
}
else if (length >= 4)
{
std::array<char, 4> first; // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init): written before read
std::array<char, 4> last; // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init): written before read
std::memcpy(first.data(), src, 4);
std::memcpy(last.data(), src + (length - 4), 4);
std::memcpy(dst, first.data(), 4);
std::memcpy(dst + (length - 4), last.data(), 4);
}
else if (length != 0)
{
// 1 to 3: the first, the middle, and the last character
const CharType a = src[0];
const CharType b = src[length >> 1];
const CharType c = src[length - 1];
dst[0] = a;
dst[length >> 1] = b;
dst[length - 1] = c;
}
}
/// @brief copy @a length characters, of a character type wider than a byte
template<typename CharType>
inline void copy_short(CharType* dst, const CharType* src, const std::size_t length, std::false_type /*is_byte*/) noexcept
{
std::memcpy(dst, src, length * sizeof(CharType));
}
/// the number of characters an empty vector is grown to by output_vector_sink
constexpr std::size_t min_vector_sink_capacity = 64;
/// @brief non-virtual output sink writing into a std::vector
///
/// This sink is not part of the virtual output_adapter_protocol hierarchy: it is
@@ -78,7 +138,7 @@ class output_vector_sink
{
public:
explicit output_vector_sink(std::vector<CharType, AllocatorType>& vec) noexcept
: v(&vec), position(vec.size())
: v(&vec), cursor(vec.data() + vec.size()), limit(cursor)
{}
output_vector_sink(const output_vector_sink&) = delete;
@@ -86,7 +146,7 @@ class output_vector_sink
output_vector_sink& operator=(output_vector_sink&&) = delete;
output_vector_sink(output_vector_sink&& other) noexcept
: v(other.v), position(other.position)
: v(other.v), cursor(other.cursor), limit(other.limit)
{
other.v = nullptr;
}
@@ -95,7 +155,7 @@ class output_vector_sink
{
if (v != nullptr)
{
v->resize(position);
v->resize(position());
}
}
@@ -104,11 +164,11 @@ class output_vector_sink
JSON_HEDLEY_ALWAYS_INLINE
void write_character(CharType c)
{
if (JSON_HEDLEY_UNLIKELY(position == v->size()))
if (JSON_HEDLEY_UNLIKELY(cursor == limit))
{
grow(1);
}
(*v)[position++] = c;
*cursor++ = c;
}
// no JSON_HEDLEY_NON_NULL here: binary_writer legitimately passes a null
@@ -117,7 +177,7 @@ class output_vector_sink
JSON_HEDLEY_ALWAYS_INLINE
void write_characters(const CharType* s, std::size_t length)
{
if (JSON_HEDLEY_UNLIKELY(v->size() - position < length))
if (JSON_HEDLEY_UNLIKELY(space() < length))
{
if (length >= max_reserve_size)
{
@@ -126,10 +186,15 @@ class output_vector_sink
}
grow(length);
}
if (length != 0)
if (length <= 32)
{
std::memcpy(v->data() + position, s, length * sizeof(CharType));
position += length;
copy_short(cursor, s, length, std::integral_constant<bool, sizeof(CharType) == 1>());
cursor += length;
}
else
{
std::memcpy(cursor, s, length * sizeof(CharType));
cursor += length;
}
}
@@ -144,21 +209,37 @@ class output_vector_sink
CharType* reserve(const std::size_t length)
{
JSON_ASSERT(length <= max_reserve_size);
if (JSON_HEDLEY_UNLIKELY(v->size() - position < length))
if (JSON_HEDLEY_UNLIKELY(space() < length))
{
grow(length);
}
return v->data() + position;
return cursor;
}
/// keep the first @a length characters written to the space from reserve()
JSON_HEDLEY_ALWAYS_INLINE
void commit(const std::size_t length) noexcept
{
position += length;
cursor += length;
}
/// nothing to pass on: the characters are in the vector (see output_adapter_sink)
void flush() noexcept
{}
private:
/// number of characters of *v written so far
std::size_t position() const noexcept
{
return static_cast<std::size_t>(cursor - v->data());
}
/// number of characters that fit the vector behind the write position
std::size_t space() const noexcept
{
return static_cast<std::size_t>(limit - cursor);
}
/// make room for at least @a length more characters after the position;
/// kept out of line so that the write functions stay small enough for
/// GCC to inline them into the writer
@@ -166,12 +247,15 @@ class output_vector_sink
void grow(const std::size_t length)
{
// use the capacity the vector already has, else at least double it;
// never less than 64 characters, so that a small value does not
// never less than min_vector_sink_capacity, so that a small value does not
// reallocate for every few bytes
const std::size_t needed = position + length;
v->resize((std::max)(std::size_t{64}, needed <= v->capacity()
const std::size_t written = position();
const std::size_t needed = written + length;
v->resize((std::max)(min_vector_sink_capacity, needed <= v->capacity()
? v->capacity()
: (std::max)(needed, v->capacity() * 2)));
cursor = v->data() + written;
limit = v->data() + v->size();
}
/// append @a length (at least max_reserve_size) characters that do not fit
@@ -181,20 +265,24 @@ class output_vector_sink
JSON_HEDLEY_NEVER_INLINE
void append(const CharType* s, const std::size_t length)
{
v->resize(position);
if (v->capacity() - position < length)
const std::size_t written = position();
v->resize(written);
if (v->capacity() - written < length)
{
// grow geometrically, with room for the writes that follow, so
// that the next grow() need not reallocate right away
v->reserve((std::max)(v->capacity() * 2, position + length + max_reserve_size));
v->reserve((std::max)(v->capacity() * 2, written + length + max_reserve_size));
}
v->insert(v->end(), s, s + length);
position = v->size();
cursor = v->data() + v->size();
limit = cursor;
}
std::vector<CharType, AllocatorType>* v;
/// number of characters of *v written so far
std::size_t position;
/// the write position: in *v, at the number of characters written so far
CharType* cursor;
/// the end of *v
CharType* limit;
};
/// output adapter for byte vectors
@@ -277,9 +365,17 @@ class output_string_adapter : public output_adapter_protocol<CharType>
///
/// Wraps the polymorphic output_adapter_t so the same binary_writer template can
/// also target arbitrary adapters (output streams, strings, user-provided
/// adapters) via the `output_adapter`-based overloads. Each write still goes
/// through one virtual call, exactly as before; only the concrete sinks above
/// avoid it.
/// adapters) via the `output_adapter`-based overloads.
///
/// A write through the adapter is a virtual call, and for a stream a call with
/// a sentry on top, which costs more than encoding the few bytes of most values.
/// The sink therefore collects what is written in a buffer and passes it on in
/// one piece when it is full, a write is too large for it, or @ref flush is
/// called. Everything written reaches the adapter in the same order and with
/// the same bytes; the destructor passes on what is left, so that a write that
/// throws leaves the adapter with exactly the bytes written before it. Only
/// @ref flush can report an error of the adapter (the destructor must not
/// throw), so it has to be called once the value is written.
template<typename CharType>
class output_adapter_sink
{
@@ -290,36 +386,121 @@ class output_adapter_sink
JSON_ASSERT(oa);
}
output_adapter_sink(const output_adapter_sink&) = delete;
output_adapter_sink& operator=(const output_adapter_sink&) = delete;
output_adapter_sink& operator=(output_adapter_sink&&) = delete;
output_adapter_sink(output_adapter_sink&& other) noexcept
: oa(std::move(other.oa)), used(other.used)
{
std::memcpy(buffer.data(), other.buffer.data(), other.used * sizeof(CharType));
other.used = 0;
}
~output_adapter_sink()
{
// a normal return has flushed already, so this is for a write that threw
if (used != 0)
{
JSON_TRY
{
flush();
}
JSON_CATCH(...)
{
// nothing can be done about an adapter that throws here
}
}
}
JSON_HEDLEY_ALWAYS_INLINE
void write_character(CharType c)
{
oa->write_character(c);
if (JSON_HEDLEY_UNLIKELY(used == buffer_size))
{
flush();
}
buffer[used++] = c;
}
// no JSON_HEDLEY_NON_NULL: forwards (null, 0) for empty payloads, exactly as
// the type-erased path already did before this sink existed
JSON_HEDLEY_ALWAYS_INLINE
void write_characters(const CharType* s, std::size_t length)
{
oa->write_characters(s, length);
if (JSON_HEDLEY_UNLIKELY(buffer_size - used < length))
{
write_large(s, length);
return;
}
if (length <= 32)
{
copy_short(buffer.data() + used, s, length, std::integral_constant<bool, sizeof(CharType) == 1>());
used += length;
}
else
{
std::memcpy(buffer.data() + used, s, length * sizeof(CharType));
used += length;
}
}
/// space for writing up to @a length characters (see
/// output_vector_sink::reserve)
CharType* reserve(const std::size_t length) noexcept
JSON_HEDLEY_ALWAYS_INLINE
CharType* reserve(const std::size_t length)
{
JSON_ASSERT(length <= max_reserve_size);
static_cast<void>(length);
return reserved.data();
if (JSON_HEDLEY_UNLIKELY(buffer_size - used < length))
{
flush();
}
return buffer.data() + used;
}
/// write the first @a length characters written to the space from reserve()
void commit(const std::size_t length)
/// keep the first @a length characters written to the space from reserve()
JSON_HEDLEY_ALWAYS_INLINE
void commit(const std::size_t length) noexcept
{
oa->write_characters(reserved.data(), length);
used += length;
}
/// pass everything written so far on to the adapter
void flush()
{
if (used != 0)
{
// reset first: the characters count as written even if the adapter throws
const std::size_t length = used;
used = 0;
oa->write_characters(buffer.data(), length);
}
}
private:
/// a write that does not fit the space left in the buffer
JSON_HEDLEY_NEVER_INLINE
void write_large(const CharType* s, const std::size_t length)
{
flush();
if (length >= buffer_size)
{
oa->write_characters(s, length);
}
else
{
std::memcpy(buffer.data(), s, length * sizeof(CharType));
used = length;
}
}
/// large enough for reserve() at any time
static constexpr std::size_t buffer_size = 4 * max_reserve_size;
output_adapter_t<CharType> oa;
std::array<CharType, max_reserve_size> reserved{};
/// number of characters of @ref buffer not yet passed on
std::size_t used = 0;
std::array<CharType, buffer_size> buffer{};
};
/// @brief whether std::basic_string<CharType> has a non-deprecated std::char_traits
include/nlohmann/detail/output/serializer.hpp
+702 -228
View File
File diff suppressed because it is too large. Load diff
include/nlohmann/detail/view/builder.hpp
+163 -47
View File
@@ -275,11 +275,34 @@ class builder
0, 0, '\b', 0, 0, 0, '\f', 0, 0, 0, 0, 0, 0, 0, '\n', 0, 0, 0, '\r', 0, '\t', 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
}
};
// The write position is kept in locals: the stores of characters into
// the arena may alias the members of the builder, which would then be
// reloaded after every one. It is stored back where the arena grows,
// and by `write_back` on every return.
const unsigned char* const end = m_end;
const std::size_t start = arena_used();
arena_run(s, static_cast<std::size_t>(p - s));
char* w = m_arena_write;
const char* w_end = m_arena_end;
const arena_write_back write_back(*this, w);
{
const auto n = static_cast<std::size_t>(p - s);
if (n <= 16 && end - s >= 16)
{
arena_ensure(w, w_end, 16);
std::memcpy(w, s, 16);
}
else if (n != 0)
{
// (an empty run leaves an empty arena unallocated, and memcpy()
// must not be passed its null pointer even for zero bytes)
arena_ensure(w, w_end, n);
std::memcpy(w, s, n);
}
w += n;
}
for (;;)
{
if (p == m_end)
if (p == end)
{
return failed(error_code::string_missing_quote, p);
}
@@ -287,7 +310,9 @@ class builder
if (c == '"')
{
++p;
return decoded{p, start, arena_used() - start};
// (from w: `write_back` stores it only after the return value
// is made; the arena is allocated, since an escape was written)
return decoded{p, start, static_cast<std::size_t>(w - m_doc.arena.data()) - start};
}
if (c != '\\')
{
@@ -297,7 +322,7 @@ class builder
return failed(c < 0x20 ? error_code::string_control_character : error_code::string_utf8, p);
}
++p;
if (p == m_end)
if (p == end)
{
return failed(error_code::string_missing_quote, p);
}
@@ -315,7 +340,7 @@ class builder
{
return failed(error_code::string_surrogate_low, p);
}
if (m_end - p < 2 || p[0] != '\\' || p[1] != 'u')
if (end - p < 2 || p[0] != '\\' || p[1] != 'u')
{
return failed(error_code::string_surrogate_high, p);
}
@@ -331,26 +356,106 @@ class builder
}
cp = 0x10000 + ((cp - 0xD800) << 10) + (lo - 0xDC00);
}
arena_ensure(4);
m_arena_write = put_utf8(m_arena_write, cp);
arena_ensure(w, w_end, 4);
w = put_utf8(w, cp);
}
else if (NLOHMANN_VIEW_LIKELY(d < 128 && simple_escape[d] != 0))
{
arena_ensure(1);
*m_arena_write++ = simple_escape[d];
arena_ensure(w, w_end, 1);
*w++ = simple_escape[d];
}
else
{
--p;
return failed(error_code::string_escape, p);
}
if (p != m_end && *p == '\\')
#if NLOHMANN_VIEW_NEON
if (p != end && *p == '\\')
{
continue; // consecutive escapes ("\u00e4\u00f6"): no run in between
}
// The first 16 bytes of the run are checked one by one, in predicted branches, and copied
// as one block: where the runs between escapes are short, the vector step below would put
// its longer dependency chain (load, compare, mask, count, add) on p for each of them,
// and be slower than these steps (the scanner of the keys works the same way)
{
if (end - p >= 16)
{
const std::uint8_t* const plain = string_plain();
std::size_t steps = 16;
#define NLOHMANN_VIEW_STEP(i) if (NLOHMANN_VIEW_LIKELY(plain[p[i]] != 0)) {} else { steps = (i); goto run_found; }
NLOHMANN_VIEW_REPEAT16(NLOHMANN_VIEW_STEP)
#undef NLOHMANN_VIEW_STEP
run_found:
arena_ensure(w, w_end, 16);
std::memcpy(w, p, 16); // (only the first `steps` bytes are kept)
w += steps;
p += steps;
if (steps < 16)
{
if (*p >= 0x80)
{
// non-ASCII: the UTF-8 check, and the rest of the run
const unsigned char* const r = p;
p = scan_string_run(p, end);
const auto n = static_cast<std::size_t>(p - r);
arena_ensure(w, w_end, n);
std::memcpy(w, r, n);
w += n;
}
continue;
}
}
}
#else
if (p != end && (*p == '\\' || *p == '"'))
{
continue; // consecutive escapes ("\u00e4\u00f6"), or the end of the string: no run in between
}
#endif
#if NLOHMANN_VIEW_VECTOR
// the plain bytes up to the next quote, backslash, or control
// character are copied as they are scanned, 16 bytes at a time
for (;;)
{
if (NLOHMANN_VIEW_UNLIKELY(end - p < 16))
{
const unsigned char* const r = p;
p = scan_string_run(p, end);
const auto n = static_cast<std::size_t>(p - r);
arena_ensure(w, w_end, n);
std::memcpy(w, r, n);
w += n;
break;
}
arena_ensure(w, w_end, 16);
const unsigned k = copy_plain_run16(p, w);
w += k;
p += k;
if (k == 16)
{
continue;
}
if (NLOHMANN_VIEW_UNLIKELY(*p >= 0x80))
{
// non-ASCII: the UTF-8 check, and the rest of the run
const unsigned char* const r = p;
p = scan_string_run(p, end);
const auto n = static_cast<std::size_t>(p - r);
arena_ensure(w, w_end, n);
std::memcpy(w, r, n);
w += n;
}
break;
}
#else
const unsigned char* const r = p;
p = scan_string_run(p, m_end);
arena_run(r, static_cast<std::size_t>(p - r));
p = scan_string_run(p, end);
const auto n = static_cast<std::size_t>(p - r);
arena_ensure(w, w_end, n);
std::memcpy(w, r, n);
w += n;
#endif
}
}
@@ -392,19 +497,49 @@ class builder
char* m_arena_write = nullptr;
char* m_arena_end = nullptr;
/// room for n more bytes at w, the write position kept in a local, with
/// w_end the end of the arena. (A lambda instead is not always inlined,
/// and then w would be kept in memory, as m_arena_write is.)
NLOHMANN_VIEW_ALWAYS_INLINE void arena_ensure(char*& w, const char*& w_end, std::size_t n)
{
if (NLOHMANN_VIEW_UNLIKELY(static_cast<std::size_t>(w_end - w) < n))
{
m_arena_write = w;
arena_grow(n);
w = m_arena_write;
w_end = m_arena_end;
}
}
/// stores a write position kept in a local back into m_arena_write when
/// the function that keeps it returns, on every path
class arena_write_back
{
public:
arena_write_back(builder& b, char*& w) noexcept
: m_builder(b)
, m_w(w)
{}
~arena_write_back()
{
m_builder.m_arena_write = m_w;
}
arena_write_back(const arena_write_back&) = delete;
arena_write_back& operator=(const arena_write_back&) = delete;
arena_write_back(arena_write_back&&) = delete;
arena_write_back& operator=(arena_write_back&&) = delete;
private:
builder& m_builder;
char*& m_w;
};
NLOHMANN_VIEW_ALWAYS_INLINE std::size_t arena_used() const noexcept
{
return m_arena_write != nullptr ? static_cast<std::size_t>(m_arena_write - m_doc.arena.data()) : 0;
}
NLOHMANN_VIEW_ALWAYS_INLINE void arena_ensure(std::size_t n)
{
if (NLOHMANN_VIEW_UNLIKELY(static_cast<std::size_t>(m_arena_end - m_arena_write) < n))
{
arena_grow(n);
}
}
NLOHMANN_VIEW_NOINLINE void arena_grow(std::size_t n)
{
const std::size_t used = arena_used();
@@ -413,25 +548,6 @@ class builder
m_arena_end = m_doc.arena.data() + m_doc.arena.size();
}
/// append the run [r, r + n) to the arena; short runs as one fixed-size
/// 16-byte move when both sides have the room (no library call)
NLOHMANN_VIEW_ALWAYS_INLINE void arena_run(const unsigned char* r, std::size_t n)
{
if (n <= 16 && m_end - r >= 16)
{
arena_ensure(16);
std::memcpy(m_arena_write, r, 16);
}
else if (n != 0)
{
// (an empty run leaves an empty arena unallocated, and memcpy()
// must not be passed its null pointer even for zero bytes)
arena_ensure(n);
std::memcpy(m_arena_write, r, n);
}
m_arena_write += n;
}
/// UTF-8 encoding of cp at w (room for 4 bytes); returns the position after
/// it. (A returned pointer stays in a register; writing through a
/// reference to m_arena_write would store and reload it around every byte.)
@@ -954,11 +1070,11 @@ indent_done:
++p;
}
const unsigned char* const int_start = p;
if (p != e && *p == '0')
if (cur() == '0')
{
++p;
}
else if (NLOHMANN_VIEW_LIKELY(p != e && *p >= '1' && *p <= '9'))
else if (NLOHMANN_VIEW_LIKELY(cur() >= '1' && cur() <= '9'))
{
p = skip_digits(p + 1, e);
}
@@ -969,7 +1085,7 @@ indent_done:
const auto int_digits = static_cast<std::size_t>(p - int_start);
std::size_t frac_digits = 0;
bool is_float = false;
if (p != e && *p == '.')
if (cur() == '.')
{
++p;
const unsigned char* const f0 = p;
@@ -982,20 +1098,20 @@ indent_done:
is_float = true;
}
std::int64_t exponent = 0;
if (p != e && (*p | 0x20) == 'e')
if ((cur() | 0x20) == 'e')
{
++p;
bool exp_negative = false;
if (p != e && (*p == '+' || *p == '-'))
if (cur() == '+' || cur() == '-')
{
exp_negative = *p == '-';
++p;
}
if (NLOHMANN_VIEW_UNLIKELY(p == e || !is_digit(*p)))
if (NLOHMANN_VIEW_UNLIKELY(!is_digit(cur())))
{
return fail(error_code::number_after_exponent);
}
while (p != e && is_digit(*p))
while (is_digit(cur()))
{
if (exponent < 100000)
{
@@ -1057,7 +1173,7 @@ indent_done:
++p; // opening quote
const unsigned char* const s = p;
p = scan_string_run<Value>(p, e);
if (NLOHMANN_VIEW_LIKELY(p != e && *p == '"'))
if (NLOHMANN_VIEW_LIKELY(cur() == '"'))
{
push_node(value_t::string, 0, 0, static_cast<std::size_t>(s - b), static_cast<std::uint64_t>(p - s));
++p;
include/nlohmann/detail/view/input.hpp
+34
View File
@@ -10,6 +10,9 @@
#include <algorithm> // max
#include <cstddef> // size_t
#ifndef JSON_NO_IO
#include <cstdio> // FILE, fread
#endif
#include <istream> // istream, streambuf
#include <string> // basic_string, char_traits, string
#include <type_traits> // decay, integral_constant, is_array, is_const, is_integral, is_lvalue_reference, is_pointer, is_same, remove_cv, remove_extent, remove_pointer, remove_reference
@@ -154,6 +157,37 @@ inline void append_stream(std::string& buf, std::istream& is)
is.clear(is.rdstate() | std::ios::eofbit);
}
/// a pointer to a C stream (never with JSON_NO_IO, which leaves out <cstdio>)
template<typename T>
struct is_c_stream
#ifndef JSON_NO_IO
: std::is_same<T, std::FILE*>
#else
: std::false_type
#endif
{};
#ifndef JSON_NO_IO
/// append the rest of a C stream: in blocks with fread, not character by
/// character (the library's file adapter reads with fgetc);
/// f is not null
inline void append_file(std::string& buf, std::FILE* f)
{
for (;;)
{
const std::size_t used = buf.size();
const std::size_t room = (std::max)(buf.capacity() - used, std::size_t{4096});
buf.resize(used + room);
const std::size_t got = std::fread(&buf[used], 1, room, f);
buf.resize(used + got);
if (got < room)
{
break; // end of file, or an error (as fgetc, which returns EOF for both)
}
}
}
#endif
} // namespace view
} // namespace detail
NLOHMANN_JSON_NAMESPACE_END
include/nlohmann/detail/view/iterator.hpp
+10 -8
View File
@@ -147,20 +147,21 @@ class view_item
view_item(const iterator& it, std::size_t index)
: m_it(it)
{
if (!it.is_object_iterator())
{
m_index = std::to_string(index);
}
}
, m_position(index)
{}
/// the member key, or the element index for arrays
/// the member key, or the element index for arrays (as text made on the
/// first call, so that iterating over the values pays nothing for it)
string_view_t key() const
{
if (m_it.is_object_iterator())
{
return m_it.key();
}
if (m_index.empty())
{
m_index = std::to_string(m_position);
}
return string_view_t(m_index.data(), m_index.size());
}
@@ -183,7 +184,8 @@ class view_item
private:
iterator m_it;
std::string m_index{}; // NOLINT(readability-redundant-member-init)
std::size_t m_position; ///< the index of an array element
mutable std::string m_index{}; ///< the text of m_position, once key() was called // NOLINT(readability-redundant-member-init)
};
/// the range returned by basic_json_view::items()
include/nlohmann/detail/view/lookup.hpp
+8
View File
@@ -166,6 +166,10 @@ const node* element_at(const document_data& d, const node* array, std::size_t id
{
return e + idx; // a growable block: one link per element
}
if (array->next == array->len + 1u && (!nav::editable() || (array->flags & node_flags::moved) == 0))
{
return e + idx; // no element has a subtree: the elements are the nodes after the array
}
for (std::size_t i = 0; i < idx; ++i)
{
e = document_data::after(e);
@@ -181,6 +185,10 @@ const node* last_child(const document_data& d, const node* container) noexcept
const std::size_t value_offset = container->kind == static_cast<std::uint8_t>(value_t::object) ? 1 : 0;
const node* const end = navigation<Editable>::end(d, container);
const node* last = navigation<Editable>::first(d, container);
if (value_offset == 0 && container->next == container->len + 1u && (!navigation<Editable>::editable() || (container->flags & node_flags::moved) == 0))
{
return last + (container->len - 1u); // an array without subtrees: the elements are the nodes after it
}
for (const node* c = document_data::after(last + value_offset); c != end; c = document_data::after(c + value_offset))
{
last = c;
include/nlohmann/detail/view/materialize.hpp
+47 -7
View File
@@ -8,7 +8,7 @@
#pragma once
#include <algorithm> // max
#include <array> // array
#include <cstddef> // size_t
#include <cstdint> // int64_t, uint8_t
#include <string> // string
@@ -40,12 +40,13 @@ template<typename BasicJsonType, bool Editable>
BasicJsonType materialize(const document_data& d, const node* n)
{
using string_t = typename BasicJsonType::string_t;
using sax_t = json_sax_dom_parser<BasicJsonType, iterator_input_adapter<const char*>>;
using sax_t = json_sax_dom_parser<BasicJsonType, iterator_input_adapter<const char*>, true>;
using nav = navigation<Editable>;
// the open containers: their next element (or key of the next member),
// their end, and whether they are objects; sized once from the
// document's deepest nesting (for a subtree an upper bound)
// their end, and whether they are objects; for the first 16 levels on the
// stack, else sized once from the document's deepest nesting (for a
// subtree an upper bound)
struct frame
{
const node* pos;
@@ -53,10 +54,43 @@ BasicJsonType materialize(const document_data& d, const node* n)
bool object;
};
// a scalar needs neither the handler nor the stack of open containers
switch (static_cast<value_t>(n->kind))
{
case value_t::string:
return BasicJsonType(string_t(d.str(*n), n->len));
case value_t::number_integer:
return BasicJsonType(static_cast<typename BasicJsonType::number_integer_t>(static_cast<std::int64_t>(integer_bits(*n))));
case value_t::number_unsigned:
return BasicJsonType(static_cast<typename BasicJsonType::number_unsigned_t>(integer_bits(*n)));
case value_t::number_float:
return BasicJsonType(float_value<typename BasicJsonType::number_float_t>(d, *n));
case value_t::boolean:
return BasicJsonType((n->flags & node_flags::is_true) != 0);
case value_t::null:
return BasicJsonType(nullptr);
case value_t::object:
case value_t::array:
case value_t::binary:
case value_t::discarded:
default:
break;
}
BasicJsonType result;
sax_t sax(result, true);
const string_t no_token{};
std::vector<frame> open((std::max)(d.max_depth, std::size_t{1}));
// (shallow subtrees on the stack: a call for a small value then allocates only for the value)
std::array<frame, 16> shallow; // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init): filled as containers open
std::vector<frame> deep{};
frame* open = shallow.data();
std::size_t capacity = shallow.size();
if (d.max_depth > capacity)
{
deep.resize(d.max_depth);
open = deep.data();
capacity = deep.size();
}
std::size_t depth = 0;
for (;;)
{
@@ -76,9 +110,13 @@ BasicJsonType materialize(const document_data& d, const node* n)
{
sax.start_array(n->len);
}
if (NLOHMANN_VIEW_UNLIKELY(depth == open.size()))
if (NLOHMANN_VIEW_UNLIKELY(depth == capacity))
{
open.resize(depth * 2); // (deeper than the recorded nesting, e.g. after edits)
// (deeper than the recorded nesting, e.g. after edits)
deep.assign(open, open + depth);
deep.resize(depth * 2);
open = deep.data();
capacity = deep.size();
}
open[depth++] = frame{nav::first(d, n), nav::end(d, n), object};
break;
@@ -105,6 +143,8 @@ BasicJsonType materialize(const document_data& d, const node* n)
case value_t::binary:
case value_t::discarded:
default:
// false positive: the parser's frame stack points into its own storage (never null)
// @infer-ignore NULLPTR_DEREFERENCE
sax.null();
break;
}
include/nlohmann/detail/view/object_index.hpp
+101 -11
View File
@@ -47,27 +47,96 @@ constexpr std::size_t index_max_displacement = 64;
static_assert(document_data::index_min_members >= 2 * index_max_displacement,
"a table must be much larger than the probe window of a lookup");
/// hash of a key: its bytes, eight at a time, in a fixed byte order
/// hash of a key: its bytes, eight at a time (in a fixed byte order), where the
/// last word may overlap the one before (so that no loop over single bytes is
/// needed); a key of fewer than eight bytes is read as two overlapping words of
/// four or two bytes (in the native byte order: the hash is not stored, and the
/// same on both sides of a lookup)
inline std::uint64_t key_hash(const char* s, std::size_t n) noexcept
{
std::uint64_t h = 0x9E3779B97F4A7C15u * (n + 1);
const auto* p = reinterpret_cast<const unsigned char*>(s); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
while (n >= 8)
{
h = (h ^ read_eight_bytes(p)) * 0xBF58476D1CE4E5B9u;
h ^= h >> 29u;
p += 8;
n -= 8;
}
std::uint64_t w = 0;
for (std::size_t i = 0; i < n; ++i)
if (n >= 8)
{
w |= static_cast<std::uint64_t>(p[i]) << (8u * i);
const unsigned char* const last = p + n - 8;
while (p < last)
{
h = (h ^ read_eight_bytes(p)) * 0xBF58476D1CE4E5B9u;
h ^= h >> 29u;
p += 8;
}
w = read_eight_bytes(last);
}
h = (h ^ w) * 0x94D049BB133111EBu;
else if (n >= 4)
{
std::uint32_t head = 0;
std::uint32_t tail = 0;
std::memcpy(&head, p, 4);
std::memcpy(&tail, p + n - 4, 4);
w = static_cast<std::uint64_t>(head) | (static_cast<std::uint64_t>(tail) << 32u);
}
else if (n >= 2)
{
std::uint16_t head = 0;
std::uint16_t tail = 0;
std::memcpy(&head, p, 2);
std::memcpy(&tail, p + n - 2, 2);
w = static_cast<std::uint64_t>(head) | (static_cast<std::uint64_t>(tail) << 16u);
}
else if (n == 1)
{
w = p[0];
}
// (a product only carries bits upwards, and the slot is taken from the low
// bits: the high bits of the last word are folded down before the multiplication)
h = (h ^ w ^ (w >> 29u)) * 0x94D049BB133111EBu;
return h ^ (h >> 31u);
}
/// whether the n <= 16 bytes at a and b are equal, as two overlapping words
/// (the first and the last 8, 4, or 2 bytes) and without a call
NLOHMANN_VIEW_ALWAYS_INLINE bool same_short_bytes(const unsigned char* a, const unsigned char* b, std::size_t n) noexcept
{
if (n >= 8)
{
std::uint64_t x0 = 0;
std::uint64_t x1 = 0;
std::uint64_t y0 = 0;
std::uint64_t y1 = 0;
std::memcpy(&x0, a, 8);
std::memcpy(&x1, a + n - 8, 8);
std::memcpy(&y0, b, 8);
std::memcpy(&y1, b + n - 8, 8);
return x0 == y0 && x1 == y1;
}
if (n >= 4)
{
std::uint32_t x0 = 0;
std::uint32_t x1 = 0;
std::uint32_t y0 = 0;
std::uint32_t y1 = 0;
std::memcpy(&x0, a, 4);
std::memcpy(&x1, a + n - 4, 4);
std::memcpy(&y0, b, 4);
std::memcpy(&y1, b + n - 4, 4);
return x0 == y0 && x1 == y1;
}
if (n >= 2)
{
std::uint16_t x0 = 0;
std::uint16_t x1 = 0;
std::uint16_t y0 = 0;
std::uint16_t y1 = 0;
std::memcpy(&x0, a, 2);
std::memcpy(&x1, a + n - 2, 2);
std::memcpy(&y0, b, 2);
std::memcpy(&y1, b + n - 2, 2);
return x0 == y0 && x1 == y1;
}
return n == 0 || *a == *b;
}
/// build the table of a large object
inline void build_object_index(document_data& d, node* obj)
{
@@ -132,6 +201,27 @@ inline const node* find_indexed(const document_data& d, const node* obj, const c
const document_data::object_index& ix = d.indexes[obj->extra - 1u];
const std::uint32_t* const table = d.index_slots.data() + ix.start;
std::size_t slot = conditional_static_cast<std::size_t>(key_hash(key, n)) & ix.mask;
const auto* const k_bytes = reinterpret_cast<const unsigned char*>(key); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
// (the loop for longer keys comes first, as the code is laid out; it is not the more common case)
if (NLOHMANN_VIEW_UNLIKELY(n <= 16))
{
// (up to 16 bytes, the bytes are compared as two words, without a call)
for (std::size_t distance = 0; distance <= index_max_displacement; ++distance)
{
const std::uint32_t s = table[slot];
if (s == 0)
{
return nullptr;
}
const node* const k = obj + s;
if (k->len == n && same_short_bytes(reinterpret_cast<const unsigned char*>(d.str(*k)), k_bytes, n)) // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
{
return k;
}
slot = (slot + 1) & ix.mask;
}
return nullptr;
}
for (std::size_t distance = 0; distance <= index_max_displacement; ++distance)
{
const std::uint32_t s = table[slot];
Loaded 100 of 177 files, more files were not shown because too many files have changed in this diff. Show more