Keep ci_test_documentation in repository_checks.yml and apply the
harden-runner 2.22.2 bump from #5962 there.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
- bench_view: add "(editable)" rows for json_editable_document and a
parse+dump workload, so the cost of lazy float conversion shows next to the
parse row (the traverse workload already includes the parse). Check that
all engines' dumps agree where outputs are comparable.
- compare.py: add --aligned, which also builds and runs with aligned code and
reports both tables, so code-layout artifacts become visible. Off by
default; the default output is unchanged. Not run end to end yet.
- README: say which engines are like-for-like peers (yyjson, simdjson DOM) and
which are mutable reference points (Boost.JSON, json::parse, the editable
json_view); explain the lazy float conversion and the eager prototype that
was measured and dropped.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Compare json_view values without a vector per container and a basic_json per number
operator== of views (view/view, view/json, and editable documents) built two
vectors of members for every object it entered, sorted them with stable_sort
for every object, and converted each number to a basic_json value. It now:
- keeps the members of all open objects in one vector (reserved with
geometric growth, so that nested objects do not reallocate), and orders
them by insertion sort for up to 16 members, or not at all when the keys
ascend already;
- compares numbers of one type directly, and builds a basic_json only for
numbers of different types;
- compares short texts (keys, numbers, most strings) with two overlapping
loads (bytes_equal(), in node.hpp);
- first walks both documents in lockstep, in the order of their text, which
finds most pairs of equal values without the work of the comparison. The
walk only enters an object if the comparison would visit its members in the
same order (the object type keeps the order, the keys ascend, or all values
are scalars), so a difference is never found later than before, and it
records the containers it found different, so that the comparison does not
walk them again;
- rejects values of different types and arrays of different sizes before any
of this, and, for an object type that keeps the order, objects whose first
(up to four) members have different keys, which ends a comparison that
differs near the start without preparing the members.
Results are unchanged: numbers compare by value across types, duplicate keys
resolve as parse() resolves them, and objects compare in document order for
ordered_json and by key otherwise.
Instructions of one comparison of a document with an equal copy, macOS arm64
(develop -> now), json_document / ordered_json_document / editable:
twitter 16.1M -> 7.0M 17.7M -> 1.5M 16.6M -> 7.3M
citm 45.0M -> 3.4M 47.0M -> 3.2M 45.7M -> 4.3M
canada 111.4M -> 13.7M 111.4M -> 13.7M 113.3M -> 15.0M
jeopardy 26.2M -> 3.7M 27.3M -> 3.7M 26.6M -> 12.1M
and of a view with a basic_json value:
twitter 11.4M -> 5.9M 11.2M -> 7.3M 11.6M -> 5.9M
citm 33.9M -> 9.7M 32.6M -> 9.1M 34.6M -> 9.8M
canada 92.9M -> 32.1M 92.9M -> 32.1M 93.9M -> 32.8M
and Linux instruction counts (GCC 13 / Clang 18) of the same comparisons:
document with an equal copy json_document ordered_json_document editable
twitter -49 % / -52 % -89 % / -89 % -48 % / -50 %
citm -90 % / -89 % -91 % / -90 % -88 % / -87 %
jeopardy (4,000 questions) -85 % / -85 % -86 % / -86 % -46 % / -49 %
view with a basic_json value
twitter -41 % / -45 % -28 % / -35 % -45 % / -44 %
citm -63 % / -65 % -63 % / -66 % -65 % / -64 %
jeopardy -39 % / -43 % -14 % / -24 % -44 % / -43 %
A sweep that places one difference at about 2,100 positions of eight
documents (a token at an even spacing, the first and last tokens, the first,
middle and last member of the root removed, one appended, the document
wrapped) in five kinds of comparison (view/view of read-only and of editable
documents, view/json, json_document, editable/view) finds the same results as
before and no placement slower on the Mac (Clang); on Linux no placement is
slower than 3 % (Clang and GCC).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Check the float tokens of a json_document image without a sort and a scan of every character
load(image_check::full) checked every float token of an image by sorting
the token ranges, scanning the token character by character, and comparing
the digit layout with the node's. Now:
- the ranges are sorted only if they are not sorted already (save() writes
them in the order of the nodes, so they are);
- a token of the shape [-]digits.digits whose digit counts are the ones the
node records (nearly all of them) is accepted by checking its digits eight
at a time, without the general scan;
- the general scan skips runs of digits eight at a time.
Damaged tokens are still rejected as before: the check accepts exactly the
tokens that the general scan accepts.
Instructions of load(image_check::full) (develop -> now):
macOS arm64 canada 44.6M -> 30.9M (-31 %), jeopardy 11.3M -> 10.7M,
citm 10.4M -> 10.0M, twitter 5.06M -> 4.94M
Linux GCC 13 canada -50 %, jeopardy -43 %, twitter -31 %, citm -28 %
Linux Clang 18 canada -42 %, jeopardy -48 %, twitter -35 %, citm -32 %
image_check::bounds is unchanged (within 0.4 %).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Rename names that Flawfinder takes for the file function open()
The comparison cursors' open() is now enter(), and a test's open and
close strings are head and tail.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix the compiler and lint failures of the json_view comparison
- all_digits(): the lambda declares its constant itself; MSVC does not
let a lambda without captures read a constexpr local of the function
- equal(): the frame's "in document order" index is a namespace-scope
constant (document_order); the local constexpr npos in the default
member initializers of the local struct crashed GCC 9 and MinGW GCC
and was not found by MSVC
- keys_differ_early() keeps the first keys in std::arrays (cpplint:
variable-length arrays; clang-tidy: C arrays), and the basic_json
side's first_keys() is not noexcept (key() may throw)
- the numbers of one type are compared in an if chain instead of a
nested conditional operator
- the dump test avoids C arrays and temporary string concatenations
View comparison instructions are unchanged (Apple clang, twitter, citm,
canada: -0.1 % to +0.1 %).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Keep the byte comparison helpers in compare.hpp
Address review comments: bytes_equal() and bytes_less() move from node.hpp
to compare.hpp, their only user; key_less() no longer splits the comment of
resolve_indices() from its function; and the loop counters of bytes_equal()
and all_digits() are scoped to their loops.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Dispatch on the compile-time conditions of the json_view comparison
MSVC (/W4 /WX) reported C4127 (conditional expression is constant) for
the template parameter EA and the object type's ordering in an if
condition; both are now tag-dispatched.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Remove a nested duplicate guard from the json_view image test
clang-tidy reported the inner #if !defined(JSON_NOEXCEPTION) as
redundant (the test case is in the same #if) and a find() != npos, which
gets the NOLINT the other tests use.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Mark Infer's null dereference in materialize() as a false positive
Infer (ci_infer, also red on develop since #5882) reports a null
dereference in json_sax_dom_parser::null() called from materialize(). The
member pointer the parser stores a value of an object in is set by the
key() before it, so it is not null there.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Move the json_view comparison tests into their own file
With the tests this branch adds, the object file of unit-json_view_dump.cpp
got too large for the MinGW x64 assembler ("file too big"). The test case
"json_view comparison" is now unit-json_view_compare.cpp, unchanged and
also built with C++17.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Build the DOM parser with Visual Studio 2015 again
Since #5882, the three VS 2015 jobs of AppVeyor fail on develop:
- the detection of keys that can be indexed instantiated std::hash for
every string type, which VS 2015 rejects with a static_assert for a type
it does not hash (alt_string): std::hash is now looked at only for
objects in a vector
- the frame stack copied its frames with std::copy to a raw pointer, which
VS 2015 reports as unsafe (C4996, an error with /WX): a loop now
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* json_view: no per-object allocations in ordered_json comparisons
For an object type that keeps the order of its members (ordered_json),
resolve_indices() allocated a vector per compared object, and
resolve_members() two. They now use one working vector owned by equal(),
whose capacity is reused, and resolve_indices() needs none when the keys
are unique. resolve_members() sorts small objects by insertion, as the
other paths do.
Instructions per comparison (callgrind, GCC 13 and clang 18, x86-64):
twitter, ordered_json view == ordered_json: 0.939 / 0.941; 20000 small
objects with a duplicate key: 0.741-0.768; all other rows (json,
citm_catalog, canada, view == view) within 0.988-1.003.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* json_view: check_utf8 skips well-formed runs with the parser's string scan
Strings of 32 bytes or more that are written into an editable document
are validated with scan_string_run(), which skips well-formed UTF-8 16
bytes at a time; the byte-wise check runs only from where the scan found
ill-formed UTF-8, to report the same type_error.316. Shorter strings
keep the byte-wise check (inlined with a constant start, so that this
path is the code it was), where the vector scan costs more than it
saves.
set()/push_back() of strings of 32 bytes or more take 0.29-0.76 of the
time (x86-64 GCC, averaged over code layouts) and 0.25-0.80 (AArch64
clang); shorter strings take 1.5-4 % fewer instructions; the json_view
rows of bench_edit execute at most 0.04 % more instructions.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* tests: avoid sign conversion in json_view_edit UTF-8 check
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* json_view: use while in check_utf8_bytewise
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The files with code adapted from fast_float, Zmij and simdjson carried the
authors' SPDX copyright lines, but their file comments did not say what
was taken or under which license: number_parse.hpp, to_chars.hpp,
simd_scan.hpp and view/simd.hpp now do, as zmij.hpp and the yyjson-based
view files already did. README and the license page now also name
fast_float's eight-digit parsing and the lexer's use of simdjson's UTF-8
check. Comments only.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Constructing a std::string from uint8_t iterators converts values >= 0x80
implicitly to char, which ci_test_clang_sanitizer rejects. Convert each
byte with an explicit static_cast.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Document the contract of object keys passed to json_sax::key and key callbacks
Describe what the name passed to json_sax::key and to a parser callback
for parse_event_t::key holds when parsing JSON text: escapes are decoded
to UTF-8, different spellings of a name yield identical strings, the name
is validated before the call (invalid escapes or UTF-8 give a parse error
and no key event), no normalization or case conversion is applied, the
event precedes the ':' check, and duplicate names are reported once per
occurrence. Also note that sax_parse passes binary-format keys unchanged.
See discussion #5955.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Add a test for the object key contract and correct the BON8 note
Pin the guarantees documented for json_sax::key and the key event of
parser callbacks: decoded names, validation before delivery, delivery
before the colon, one event per repeated name, and the binary formats.
BON8 rejects ill-formed UTF-8 in a name even with sax_parse, so the
note no longer lists it with the formats that pass bytes through.
See discussion #5955.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix the GCC build of the object key contract test
GCC 11 in C++11 mode rejects a default argument on a lambda parameter
(-Wc++14-extensions) and ignores a static_cast<void> on the
warn_unused_result of parse(). Pass the input format explicitly and bind
the parse result to a variable.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Add version history for the object key contract
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Split the combined has_save/has_load static_asserts so clang-tidy no
longer sees both sides of && as equivalent (develop ci_clang_tidy failure).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
VS 2015 does not accept std::string::npos in a constant expression (C2131),
and the constant is only used at run time.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* json_view: source positions in materialize() with JSON_DIAGNOSTIC_POSITIONS
With JSON_DIAGNOSTIC_POSITIONS, materialize() now gives every value the
start_pos()/end_pos() that basic_json::parse() would give it, so exceptions
about materialized values name the same bytes (#5864).
- json_sax_dom_parser takes the lexer type as a defaulted template
parameter; materialize() passes position_source, which reports the
positions it derives from the index. diagnostic_positions treats npos as
"no position" and keeps ranges all-or-nothing.
- Strings with escapes keep the offset of their opening quote plus one in
`next` (only with the macro); materialize() scans from there to the closing
quote. The end of an array or object is found by scanning from the end of
its last element over whitespace, comments and a trailing comma.
- In editable documents, values written by edits, arrays and objects whose
elements changed, and containers whose last element has no position get
none. Images keep the flag is_new, and load() checks that `next` of a
string is within the text.
Nothing changes on the parse path without the macro. The view's own
exceptions still carry no byte range, so the tests that compare them with
those of materialized values skip it under the macro, as they skip the path
under JSON_DIAGNOSTICS.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Indent comment in unit-json_view_from_json.cpp
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address review: clearer position code and docs
- position_source gets clear(); set_scalar() returns early for new values,
switches on node_kind, and uses no local copies; the one position_source
per materialize() call is documented
- no npos + 1 for containers without a position; a comment on why the end
of a container is reported as its closing bracket
- less duplication under #if in materialize() and the builder
- diagnostic_positions sets npos explicitly
- docs: positions of basic_json values after changes, positions in images
depend on how the document was parsed, when next is set, and the cost of
positions while parsing
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Share the comment scanners of the view builder with materialize()
skip_line_comment() and find_block_comment_end() in scan.hpp are used by
the builder's comment() and by position_source::container_end(), which
had its own copy.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address review: if/else for container positions, clearer next field
- the opening bracket of a container sets positions with an if/else instead
of a ternary, and the frame takes its start from positions
- the closing bracket of a container is found in nested if/else; the parent's
last_end is taken from positions, so it is npos whenever no range is set
- architecture.md: the next field reads "or 0 if it was not; all other
values: 0" instead of "else 0; otherwise 0"
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix GCC unused-result and clang-tidy raw-string findings in unit-json_view_positions.cpp
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix self-aliasing in json_pointer::operator/= and patch_inplace
json_pointer::operator/=(const json_pointer&) passed iterators into its
own token vector to insert() when a pointer was appended to itself
(p /= p), which is undefined behavior; libstdc++ produced "/a/b//" for
"/a/b". The pointer now appends a copy in that case.
basic_json::patch_inplace iterated the patch while changing *this, so
j.patch_inplace(j) read freed memory. It now applies a copy of the
patch when the patch is the value itself. A patch stored inside the
value remains undefined behavior and is documented as such; copying the
patch unconditionally would double the time of a large patch.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix JSON_DIAGNOSTICS test crash and clang-tidy hit in patch_inplace self-aliasing
The spare-capacity test reallocated non-empty elements through get_ref(), which leaves stale parent pointers under JSON_DIAGNOSTICS; reserve while the array is empty instead. Suppress performance-unnecessary-copy-initialization for the copy that is needed because *this is modified.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Detect a patch stored inside the patched value with JSON_DIAGNOSTICS
patch_inplace applied a copy only when the patch was the value itself. With JSON_DIAGNOSTICS the parent pointers make it cheap to also detect a patch nested inside the value, so that case is handled the same way. Without diagnostics it stays documented undefined behavior.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Silence misc-const-correctness for patch_aliases_this
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix clang-tidy findings in the operator/= self-append test
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
MSVC 2015 treats two function templates that differ only in a decltype
return type as a redefinition (C2995), which breaks all three Visual
Studio 2015 jobs on AppVeyor. Overload on the view's mode in the
parameter type instead; template argument deduction ignores the
read-only-to-editable conversion, so each view still selects the
overload of its own mode.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix clang-tidy findings in unit-json_view_from_json.cpp
Silence misc-use-internal-linkage on the macro-generated ADL functions,
reconcile -Weffc++ with readability-redundant-member-init, use auto for
get<T>() initializers, and use value() on the optional access.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Satisfy bugprone-unchecked-optional-access in unit-json_view_from_json.cpp
clang-tidy 22 also flags value(); check has_value() in the same expression.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The test relies on a copied fifo_map restarting its timestamps. Copying a
basic_json does that only on the recursive path; with JSON_NO_THREAD_LOCAL
the copy is built member by member and keeps correct timestamps, so the
premise was missing and the test failed (ci_test_no_thread_local, MinGW
clang). Copy the object_t explicitly to create the state on every path.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The 19 macOS and 32 Windows jobs ran on every push to every pull request.
They now use the same sets as the Ubuntu jobs:
- PR set (16): four Visual Studio 2022 and two Visual Studio 2026
configurations, ARM64 Debug, Clang 20.1.8 with MinGW, MinGW x86, the
C++20 module test, Xcode 16.0, 16.4, 26.3 and 26.6, C++14 on the
latest Xcode, and the Swift Package Manager check.
- full set (34): every other Visual Studio configuration, Clang
version, clang-cl, Xcode version and C++ standard.
- nightly set (1): 64-bit MinGW, which takes about 68 minutes. A failed
nightly run opens or updates an issue.
No configuration is removed: tools/ci_jobs/ci_jobs.py --compare lists
157 configurations before and after, and all jobs keep their runner,
steps and environment. ci_jobs.py now checks all three workflows and
expands matrices that consist only of include entries correctly.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
All ~100 Ubuntu jobs ran on every push to every pull request. They now
belong to three sets, chosen by each job's if: condition:
- PR set (16 configurations): every push to a pull request and every
push. Together with the checks that always run, these jobs would
have caught at least 92 % of the failing commits of Sep/Oct 2026.
- full set (73 configurations): after the PR set passed, on pushes and
on pull requests with the "🚀 ready to merge" label.
- nightly set (10 configurations): Infer, Valgrind, coverage, icpc,
icpx, nvhpc, CUDA, and ARM64; nightly on develop, on pushes to master
and release/*, and by hand. A failed nightly run opens or updates an
issue.
A matrix whose entries belong to different sets is split into one job
per set; the jobs share their steps through YAML anchors. No
configuration is removed: tools/ci_jobs/ci_jobs.py --compare lists 150
configurations before and after. The same tool checks in CI that every
job belongs to exactly one set, that the full set waits for the whole
PR set, and that the job table in quality_assurance.md is up to date.
The documentation describes the sets in a new section "How CI runs",
and the QA pipeline figure shows the new badges.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
ubuntu.yml, macos.yml and windows.yml run about 155 jobs on every pull
request, including those that change only the documentation. They now
have a pull_request path filter: it starts with '**' and excludes the
documentation, Markdown files, GitHub metadata, other workflows and the
license files, so a new file runs them unless it is excluded on purpose.
Pushes to develop, master and release/* are not filtered.
The documentation jobs and the REUSE check move from ubuntu.yml to the
new repository_checks.yml, which has no path filter and also runs
tools/check_path_filters. That check keeps the three filters identical
and fails when a file the compiler jobs keep mentions an excluded file.
It found two: tools/check_build_options reads the Meson section of
package_managers.md, and the swiftpm job builds the Swift example from
the documentation; both are included in the filter.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
- unit-bson: initialize the SAX recorder's member (-Weffc++)
- unit-json_patch: build the reference document independently instead of
copying it (performance-unnecessary-copy-initialization)
- unit-json_view_builder: use node_kind::link instead of an out-of-range
enum cast (clang-analyzer-optin.core.EnumCastOutOfRange)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Spell out that numbers comparing equal produce no operation even if one is
zero and the other negative zero, in diff.md and features/json_patch.md.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Keep the value unchanged when operator[] with a JSON pointer throws
The non-const operator[](json_pointer) is documented with strong exception
safety, but get_unchecked created intermediate values while walking the
pointer and kept them when a later reference token failed, e.g.
json j; j["/a/99999999999999999999999"_json_pointer] threw 404 and left
{"a":[]}; padding an existing array or appending with "-" behaved alike,
and so did std::length_error/std::bad_alloc from a huge index.
get_unchecked now only walks existing values up to the last reference
token. The last token's key insertion or array growth is strongly
exception safe by itself. Any other change goes through
get_unchecked_create, which records the one change it makes to an existing
value (null converted, key inserted, array grown) in a RAII guard and
reverts it if a later step throws; everything else it creates lies below
that change. The guard needs no try/catch, so JSON_NOEXCEPTION builds are
unaffected. Successful calls create the same values as before.
Docs: state the guarantee in the exceptions list and version history of
operator[], fix the complexity of the pointer overloads of operator[] and
at, and warn that padding arrays up to an untrusted index allocates memory
proportional to the index (operator[].md, untrusted_input.md).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Make the strongly exception-safe pointer operator[] faster than before
The previous commit made get_unchecked strongly exception safe but slower
with GCC 16 when it creates values (up to +6 % instructions for appending
with "-"). This restructures it so that no benchmark row is slower than
before the strong guarantee, with GCC 16, Clang 22 and Apple Clang.
- get_unchecked walks existing values in one loop that inserts nothing.
The last reference token is resolved in that walk: "-" appends with
emplace_back(), an index grows the array with operator[], and a missing
key is inserted with emplace_hint() at the position the walk's
lower_bound() found, so the key is searched once.
- With std::less<string_t> or std::less<> as comparator, the walk checks
the key at the lower bound for equality instead of equivalence, which is
the same for these comparators and cheaper.
- get_unchecked_create (not inlined) handles all other changes. Its guard
swaps a null value with a fresh null value instead of copying it and
reverts the change in a non-inlined function, so a successful call pays
one compare for it. As every null value converted afterwards is a fresh
one, it is converted in place instead of assigning a new array or object.
- Keys are added to new objects with emplace(), which also replaces the
basic_json::operator[] call for ordered_json.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix rollback of pointer operator[] for unusual object and string types
Follow-up to the strong exception guarantee of the non-const
operator[](json_pointer):
- get_unchecked_create assumed that emplace() always inserts the key that
the walk did not find. For object types whose lookup and emplace()
disagree (e.g. a copied fifo_map), nothing was inserted, the rollback
erased the existing member, and the walk continued with a value of the
wrong type (assertion, type_error.305, or a crash). It now checks the
result of emplace(), arms the rollback only if a key was inserted, and
otherwise continues the walk with the member found, as basic_json's
operator[] does: such pointers throw out_of_range.404 and leave the value
unchanged, as before the strong guarantee.
- The rollback erased the inserted key by key, which converts string_t to
object_t::key_type and may allocate inside a noexcept function
(std::terminate on bad_alloc with custom key types). It now erases by the
iterator emplace() returned, and shrinks a grown array with erase() at
the end instead of resize(). The documentation states the remaining
requirement for custom template parameters.
- Creating values no longer calls string_t::begin(), so the non-const
operator[](json_pointer) compiles with string types that lack it (as the
alternative string type of the tests).
- Correct the @throw lists of get_unchecked and get_unchecked_create.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address review comments on operator[](json_pointer) rollback
- undo() shrinks the array with resize() instead of erase()
- comments say what "will be" done in get_unchecked_create
- tests: avoid a clang-tidy performance-unnecessary-copy-initialization
hit (unit-alt-string) and the word "access" in section names, which
Flawfinder flags as race/access
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Let the pointer rollback guard call undo() unconditionally
The check in the destructor and JSON_HEDLEY_NEVER_INLINE on undo() made no
measurable difference: instruction counts of j[ptr] are identical with
GCC 13 -O3 and within 0.4 % with clang, timings are within noise.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: new design for the documentation site
Apply the brand to the Material for MkDocs site: Inter and JetBrains Mono
served by the site (upright and real italic), brand colours, blue header band
with the syntax pattern, wordmark in the header, new home page (hero, cards,
animated feature tour), 404 page, polish for tabs, search, footer, focus
rings, tables and code blocks, and theme features (code copy, previous/next
footer, TOC follow, search suggestions).
Draw the number railroad diagram, the parser callback events and the iterator
illustrations as inline SVGs that follow the light and dark scheme. The
generator scripts are in docs/mkdocs/scripts/figures; the feature slideshow
(json-light.svg, json-dark.svg, replacing json.gif on the home page) is built
by docs/mkdocs/scripts/slideshow. Decision nodes in five flowcharts become
hexagons, which need far less room.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: add figures, table captions and a macro chooser
- Figures with numbered captions: SAX stepper, json_view layout and edit
stepper, exception map, JSON pointer and patch walkers, QA pipeline,
scaling and binary size charts, roadmap lanes, release timeline,
integration decision tree and namespace builder
- Table captions (nj-table-caption) on 9 pages
- Compile times: modules and PCH charts, filesystem section, re-measured
Figure 1, links between features/modules.md and compile_times.md
- Interactive macro chooser for features/arbitrary_types.md
- Link check: exclude URLs that answer the plugin with 400/401/419 or are
dead (customers.md)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: macro chooser generates adl_serializer for non-default-constructible types
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: cross-link concept pages and API pages in See also sections
Add See also links in both directions between the concept pages (features,
home, integration) and the API pages, write API links in code font, give every
entry a comment after a dash, and show the callback event figure on the parser
callbacks page.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: address review comments on the new design
- Home: add a "Latest release" button
- Sponsors: update the list and group it by tier
- Features: list all feature pages on the index page
- Releases: add highlights per release, link the 4.0 roadmap, add the first commit to the timeline
- Exceptions: prefix all example messages with the exception name
- API: no code font for format names in the See also links of the to_/from_ pages
- Merge Patch: show the document before, the patch, and the result
- Replace "--" with typographic dashes, mark external links with an icon, center Mermaid diagrams
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* json_view: pointer tokens that cannot be array indices create objects
editable_document::set() chose an array for a null parent whenever the
token consisted of digits, so "01" threw parse_error.106 and the empty
token threw, while basic_json::operator[](json_pointer) has created an
object for such tokens since #5358. This broke
unit-json_view_edit_values on develop. Use the same RFC 6901 array index
test as basic_json.
Also re-indent the unflatten test that the astyle check flagged.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix the other red jobs on develop
- unit-bson: initialize recorder::events in its declaration (-Weffc++)
- unit-json_view_from_json: suppress -Wexit-time-destructors for the
serialize-enum tables and -Wfloat-equal for same()
- unit-pointer_access: @infer-ignore NULLPTR_DEREFERENCE for the
dereferences after get_ptr() that Infer reports
- json_view.hpp: IWYU no_include for the detail headers that the view
code uses through nlohmann/json.hpp
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix GCC warnings in the json_view from_json test
GCC (-Weffc++, -Wfloat-equal) rejected unit-json_view_from_json.cpp
(ci_test_standards_gcc (20)): initialize the members of shape and
with_names, and suppress -Wfloat-equal for GCC like for clang.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
- Size the buffer of decoded strings on its first growth from the input (a
quarter of it plus 16 bytes, at most 4 KiB) instead of 272 bytes, so that a
document with escapes does not grow it twice, with a zero-filled copy each
time.
- Allocate the node index of documents with an estimate of up to 1024 nodes
(was 256) together with the header, which saves one allocation. When the
estimate is too low (dense numbers), that block stays unused: at most 16 KiB
instead of 4 KiB. shrink_to_fit() cannot release it separately; the
documentation says so.
Fresh parse of a 2.5 KB minified tweet: 993 to 958 ns (-3.5 %), traverse and
select -3 %, with no row consistently slower (Apple M1 Max, 30 rounds, three
runs). Dense numeric documents parse about 9 % faster and keep up to 16 KiB
more memory.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: writing generic code for json_view
A "Writing generic code" section on the json_view page: why a function
taking json_view accepts no other view, the three signatures for
editable, read-only, or either kind of view (passed by value), why the
modes do not convert, is_json_view, and how a from_json() reads views
(constrained to json values and views; the macros and the library's
conversions do it already) with what differs then (duplicate keys,
JSON_DIAGNOSTICS paths).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: name the documents the generic view function accepts
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: views of read-only documents convert to editable views
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* json_view: the library's std conversions read views directly
The library's from_json() overloads (containers, maps, std::array, C
arrays, pairs, tuples, optional, valarray, forward_list, enums, binary,
paths, and scalars) accept views: their traits use the view's
basic_json type, and the few accesses that differ (get_ptr) go through
small helpers. from_json() of a view now finds them; only the probe
still looks up by ADL only. get<T>() of a view thus converts these types
without materialize(), with the same values and exceptions; from an
object, a repeated key keeps the last value, as parse() does.
The view keeps its direct paths for vectors of the types it converts
directly (faster than from_json()) and for containers of views; its
other vector and map paths are replaced by from_json(). For basic_json,
std::valarray and std::forward_list are read in one pass (40 % and 3.5 %
fewer instructions); all other conversions are unchanged.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* json_view: enable_if_not_view_t instead of '!' in the from_json() templates
astyle spaces out the template argument list when it contains a negation;
the alias spells the condition with negation<> and leaves the lines alone.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* json_view: compact the template parameter lists astyle had spaced out
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
get<T>() and get_to() of a view call T's from_json() with the view itself
when that from_json() accepts views but not an unrelated probe type, and
fall back to materialize() otherwise. The from_json() functions of the
NLOHMANN_DEFINE_TYPE_*, NLOHMANN_DEFINE_DERIVED_TYPE_* and
NLOHMANN_JSON_SERIALIZE_ENUM(_STRICT) macros accept basic_json values and
views; a derived type whose base reads only json reads the base part
through materialize(). The library's own from_json() overloads are not
called with views: from_json() of a view (and of the probe) only looks up
functions by ADL.
New public trait nlohmann::is_json_view<T>, and the member type
basic_json_type of basic_json_view. With duplicate keys, the members of
such a T are those the view's lookups find (the first ones).
bench_decode.cpp measures decoding into structs (4-6x faster than
materialize().get<T>() on twitter, citm_catalog, and synthetic records).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The view's vector code follows JSON_NO_SIMD, like the lexer and the float
writer, and skips the CPU check of SSSE3 when the code is compiled for
SSSE3 anyway (__SSSE3__), like the lexer. JSON_VIEW_NO_SIMD and
JSON_VIEW_USE_SSSE3 and their ABI tags _vns and _vssse3 are removed; they
were never released. JSON_NO_SIMD keeps its tag _ns.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The image format is little-endian only. Instead of throwing
type_error.320 at runtime, save_image(), load_image() and the members
save() and load() do not exist on a big-endian target, so a call fails to
compile, like set() on a read-only document. type_error.320 remains for
saving a discarded document.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* json_view: view_mode enum instead of bool Editable, node_kind for node::kind
basic_json_document and basic_json_view take a view_mode (read_only, the
default, or editable) instead of a bool: basic_json_document<json, true>
did not say what true means. The shortcut aliases are unchanged.
The kind of a node is a node_kind (the value_t values plus link, 10 as
before), which removes the static_cast<std::uint8_t>(value_t::...) of
every comparison. The node and image layouts are unchanged.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* json_view: node_kind in the node writers, spell out read_only, comment fixes
The builder and the edit functions take and store a node_kind instead of
casting a value_t; the numbering assertions and the comments now speak of
node_kind, and the link kind notes that it follows value_t::discarded.
The read-only aliases name view_mode::read_only like the editable ones.
view_mode.md says the mode is a compile-time property of the type.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* json_view: views of read-only documents convert to editable views
A read-only document has no edits, so a view of it reads correctly as a
view of an editable document before its first edit. The implicit
conversion lets a function taking a json_editable_view accept the views
of both modes. There is no conversion the other way: a read-only view of
an editable document would read wrong values after the next edit.
The mode comparisons in the cross-mode operator==/!= use an
is_other_mode trait, which astyle keeps compact.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: fix BON8 documentation gaps found in the format review
Verified each claim with small C++ probes before writing it.
- index.md: BON8 serialization is also incomplete for invalid UTF-8.
- bon8.md: document duplicate keys (last wins), the error ids for
malformed input incl. 0xF5..0xF7 + 0x80..0xBF, exact string
termination wording, wide integers and discarded values in the
unsupported list, and title two untitled admonitions.
- exceptions.md: to_bon8 in type_error.316, BON8 in parse_error.112
(with example) and .113, wide integers in out_of_range.407.
- binary_values.md: bytes 0..39 take one byte in BON8.
- from_bon8.md: note on strict UTF-8 and duplicate keys.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address review of the BON8 documentation
Restore the correct statement that marker bytes cannot begin a UTF-8 character (0x80..0xBF occur in valid UTF-8 as continuation bytes), and reword two sentences.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: drop the discarded-value remark from the BON8 page
Discarded values are not mentioned on the individual format pages.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: do not list invalid UTF-8 strings as a BON8 serialization gap
JSON text must be valid UTF-8, and to_bon8 rejects ill-formed strings by design (type_error.316), as documented on the BON8 page. Restore the original cell and realign the table.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Read ND-arrays of half floats and enforce the char limit in ND-arrays
BJData N-dimensional arrays with element type 'h' (float16) failed to
read with a misleading parse_error.112, because the reader's dtype name
table lacked 'h'. It now maps to "half", the JData name. The writer still
does not pack "half"; such an object is written as a plain object and
reads back equal.
BJData limits 'C' (char) to 0..127 in Draft 2 and Draft 3, and the 1-D
reader enforces this, but ND-arrays with _ArrayType_ "char" wrote values
up to 255 as 'C' elements and read them back without a check. The writer
now only packs values in 0..127 (others fall back to a plain object that
round-trips), and the reader rejects 'C' elements above 127 with
parse_error.113 like the 1-D reader.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: correct and complete the BJData documentation
Verify the BJData chapter against the library and the Draft 2/3/4
specifications and fix what did not match:
- bjdata.md: correct the list of unused markers (N, not Z), the ND-array
example (3x2, not 2x3), the conditions under which an object is packed
as an ND-array (exactly three keys, row vectors, float-only elements for
single/double, char 0..127, byte only in Draft 3), the binary table row
and notes (which flag/draft combinations write [$B), the "$" type
restriction (Draft 3, Z is excluded too), the H marker in the
deserialization table, and the claim of a complete deserialization
mapping. Document the supported drafts, that Draft 4 (SoA containers,
extension type E) and column-major ND-arrays are rejected, ND-arrays of
half floats, duplicate object keys, and link the Draft 3 specifications
instead of master.
- Add an API page for basic_json::bjdata_version_t with an example, and
link it from the nav, the API index and to_bjdata.
- Update binary_formats/index.md and binary_values.md, which still claimed
that BJData has no binary type, and fix a wrong byte comment there.
- Document parse_error.115 and the out_of_range.408 cases for from_bjdata,
the precise other_error.502 condition for to_bjdata, and reword the note
on truncated unsigned integers in exceptions.md (only ND-array elements
were truncated before 3.13.0).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Regenerate single_include and apply astyle
Result of make amalgamate (including make pretty) for the preceding changes.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address review of the BJData documentation
The $ type restriction exists since Draft 2 (Draft 3 added B), $ requires a count, ND char above 127 is rejected on read, and the new API page is listed in alphabetical order in the nav.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: format tables, use the Oxford comma
Reformat the tables changed by this PR (BJData serialization and
deserialization mappings, completeness and binary-value tables) so all
rows are aligned, and add the Oxford comma to lists of three or more
items in the added text.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Write and read the inner length of BSON binary subtype 0x02
The BSON specification defines the payload of the "binary (old)" subtype
0x02 as an int32 followed by that many bytes. to_bson wrote the plain
payload instead, so decoders such as pymongo and libbson-based readers
rejected the bytes, and from_bson returned the inner length as part of the
payload (documented as lenient behavior in gh-5333).
The writer now emits the outer length (payload + 4), the subtype, the inner
int32 holding the payload size, and the payload. The reader requires an
outer length of at least 4 and an inner length equal to the outer length
minus 4 (the cases the bson-corpus lists as decode errors), reports parse
error 112 otherwise, and returns only the payload. Both directions change
together so that the round trip keeps working.
The single-reservation fast path for scalar entries declines subtype 0x02
because it has no room for the inner length.
This is a deliberate exception to the opt-in rule for behavior changes:
the previous output was invalid BSON.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Check the declared BSON document size before reading its content
The BSON reader compared a document's declared size with the number of bytes
read only when it reached the terminator. The declared size never bounded
the reads, so a SAX handler saw the keys and values of a document whose size
was too small before parse error 112 was reported.
The container frame now stores the position where the document ends
according to its declared size. open_bson_document rejects a size below 5
(size prefix plus terminator) and an embedded document that does not end
before the terminator of its enclosing document; the element loop rejects an
element whose type byte leaves no room for the name terminator and the
document terminator. The terminator check stays and still catches a declared
size that is too large. No input that was accepted before is rejected now;
documents with a wrong size are reported earlier, with fewer SAX events, a
different byte position and, in three cases, a different message. The error
id (parse_error.112) is unchanged and strict=false still does not skip the
check.
The tests that asserted the old messages and positions are updated, with the
outer sizes of two inputs corrected so that they still exercise the embedded
document. New tests cover the SAX events, a size below 5, and an embedded
document that fits exactly.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Document BSON subtype 0x02, document size checks, and lenient reads
Describe the inner length of binary subtype 0x02 that to_bson now writes and
from_bson checks and removes. Add the document size checks (parse_error.112,
not affected by strict=false) to bson.md and from_bson.md, and extend the
list of lenient reads with the UUID and MD5 payload lengths (not checked),
duplicate keys (last value wins), embedded NUL bytes in strings, and the
ignored trailing bytes with strict=false. State that the members of an object
are written in iteration order.
Correct the statement that element names are never validated on read: they
are checked and sanitized like string values by the error handler. Add the
BSON message to the examples of out_of_range.415 and type_error.321, the
subtype limit of 255 to the binary values page, CBOR to the formats that use
byte_container_with_subtype, and fix a typo in the incomplete mapping box.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Regenerate single_include and apply astyle
Result of make amalgamate (including make pretty) for the preceding changes.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address review of the BSON documentation
Describe exactly when a too small document size is detected and note that subtype 2 data written by earlier versions is not readable.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: use the Oxford comma, drop discarded-value remarks
Add the serial comma in a list added by this PR and remove the BSON
example of type_error.321 (with the type_error.317 remark) so the docs do
not mention discarded values for BSON; restore the original singular
heading of that section. No tables were changed by this PR.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix -Weffc++ in the BSON recorder test and a clang-tidy use-auto hit
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix clang-tidy readability-redundant-member-init in BSON test
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Niels Lohmann
dependabot[bot]