* Fix silent wrong results for custom number types
Squashed onto develop cd6b428ef (CI fixes from #5860). Original commits:
- Fix silent wrong results for custom number types
- Fix CI: build the out_of_range.407 message from string_t and fix a link
- Fix CI: diagnostics message, clang-tidy, and MinGW section limit
- Format the out_of_range.407 number without dump()
- Move the std::deque array test out of unit-regression3.cpp
- Fix clang-tidy and clang 3.4/3.5 failures in the custom number type changes
- Fix CI: cast the 128-bit unsigned value explicitly in the integer fast path
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Name the range checks of integer_fits_max/min
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Speed up writing CBOR and MessagePack, above all integer arrays
Squashed onto develop cd6b428ef (CI fixes from #5860). Original commits:
- Append to the output vector through a position-tracking sink
- Write CBOR integers without branching on their sign or size
- Write MessagePack integers without branching on their sign or size
- Use std::array for the integer size class table
- Fix clang-tidy and GCC warnings in the integer array test
- Regenerate the amalgamation
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Convert the streamed bytes explicitly in the binary writer sink tests
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
With glibc and optimization, <stdio.h> turns getc_unlocked into an extern
inline function that reads the stdio buffer (fp->_IO_read_ptr) directly.
glibc, which is not instrumented, fills that buffer, so MemorySanitizer
treats the bytes as uninitialized and reports a false positive when the
parsers read from a FILE* (e.g., the fuzz harness FILE* leg, OSS-Fuzz
575915982). std::fgetc is an opaque call whose result MemorySanitizer treats
as initialized.
Select JSON_UNLOCKED_FILE_IO 0 when compiled with MemorySanitizer. This is a
regression from #5828 and not part of any release.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Write json_view's dump() without a call or conversion per token
The default dump() (no indentation, no ensure_ascii) gets its own
writer that makes the same walk and produces the same output:
- the write position stays in a local variable instead of a
member, so the compiler keeps it in a register across stores
through aliasing char pointers;
- strings and number tokens are copied with fixed-size 32-byte
moves wherever enough source bytes remain, instead of one
memcpy call per token;
- the innermost open container lives in local variables; a stack
that starts as a local array of 32 entries holds the rest;
- unedited documents are walked through the node array in order,
and integer tokens are read from the source directly.
On top of that, float tokens of at most 15 significant digits are
written straight from their digits via zmij::to_shortest() and
write_shortest(), without converting to a double and back: such
decimals are farther apart than a double's rounding interval, so
the token's digits are the double's shortest digits. Tokens of
16+ digits, or edited values, still go through decimal_to_float().
The view's own NEON write_decimal() is removed in favor of the
shared writer, and the dump output now grows in 64 KiB steps
instead of being resized to its estimate at once.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix MSVC C4127 and Infer finding in the view serializer
Test the template options through enabled() so that conditions combined
with them are not constant, and mark the nav::value() null dereference
as a false positive like materialize.hpp does.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix out-of-bounds read when dumping a float token of an unchecked image
A float node of an image loaded with image_check::bounds can hold a
token whose bytes are not digits, so its value need not have the digit
count recorded in the node. write_double_at() passed that count to
write_short_decimal(), which indexes its table of powers of ten by it:
an assertion failure in debug builds, an out-of-bounds read in release
builds. Use the counted overload only if the value has exactly that many
digits, otherwise count them.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Tidy the view serializer: doxygen block placement, enabled() for a constant
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Use a named document in the dump test: root() of a temporary does not compile
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Test write_short_decimal and powers_of_ten_16 against to_chars
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix clang-tidy 22 findings in unit-to_chars.cpp
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Write doubles from their bits only where double is IEEE 754 binary64
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Split the value and pointer edit tests off unit-json_view_edit.cpp
The MinGW linker of the Windows clang jobs cannot link object files with more
than 32767 sections ("relocation truncated to fit: IMAGE_REL_AMD64_REL32
against `.rdata'"). unit-json_view_edit.cpp has 35409 sections since the fast
dump instantiates more code for editable documents, so its test cases "views
and values", "deeply nested values", "pointers below a null value", and
"strings of other documents are checked" move into
unit-json_view_edit_values.cpp. The helper exception_of_call() that both
files use moves into json_view_test_helpers.hpp.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Move the dump tests of this branch to unit-json_view_dump.cpp
The dump and comparison tests moved to their own file in the dump pull
request; the tests of the fast dump() join them there.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Regenerate the amalgamated headers
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address review: one powers-of-ten table, literal sizes from the type
- dtoa_impl::powers_of_ten() (10^0 to 10^19, one entry per line)
replaces powers_of_ten_16() and the table of int_pow10()
- put_literal() takes the length of a string literal from its type
- small_stack holds the heap vector by reference
- the growth step of the output buffer is written in hex
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Explain why the growth of the dump's stack is static
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Regenerate the amalgamated headers
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Add images of json_documents: save() and load()
An image is a document stored so that loading it needs no
parsing: save() writes the node index, the text and the decoded
strings; the static load() reads an image written by save().
load() takes a pointer and size, a borrowed vector, or an owned
rvalue vector; the nodes are copied so they are aligned and can
be edited, while the text and decoded strings stay in the image.
image_check controls how much load() trusts the input: full
checks structure, bounds, strings and numbers, the parser's own
guarantees; bounds checks structure and bounds only; none skips
all checks, for images from a trusted source.
Layout is little-endian only ("NJVI" header, nodes, text, decoded
strings), following the idea of zero-copy formats such as
FlatBuffers and YaFF; the check follows FlatBuffers' Verifier.
New errors: parse_error.116 for a malformed image or a failed
check, type_error.320 for a discarded document or a big-endian
target.
A dedicated fuzzer and 6,000 seeded corruptions, checked under
ASan/UBSan, found and fixed two gaps: unchecked reserved header
fields, and unbounded null/boolean offsets that could make
dump() throw std::length_error.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix MSVC errors in image test and add image_check doc anchor
Move the raw string literal out of the CHECK macro (MSVC preprocessor),
cast 64-bit header fields to std::size_t (C4244 on 32-bit), and give the
image_check section in load.md a real anchor for the mkdocs strict build.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Check each distinct string and float range once in the full image check
The full check scanned the contents of every string node and every float token over its own range, so many nodes pointing to one large range made load() quadratic. The structural walk now only collects the ranges; after it, each kind is sorted and checked once per distinct range. Ranges of one kind that overlap without being identical are rejected, as save() never writes them (nodes that share a value share the whole range). The cost is linear in the size of the image plus sorting the ranges.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Adapt the image tests to named documents and last-wins lookups
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Test that the targets of links never reach an image
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Test lookups in loaded images with duplicate and colliding keys
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Release the list of large objects once load() has built the indexes
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Test that invalid UTF-8 of a loaded image is not copied into an editable document
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Document that editable documents refuse invalid UTF-8 copied from a loaded image
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Expect first-wins lookups in the image tests
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Avoid useless casts of header fields in the image tests
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix clang-tidy 22 findings in unit-json_view_image.cpp and image.hpp
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix MSVC C4244 and C2017 in the image test
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix the indentation of a loop in the image test
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Make json_view.hpp pass include-what-you-use
ci_single_binaries runs IWYU with --error on every header. For json_view.hpp
it suggested adding <array> (std::array is used), the headers that json.hpp
already provides (abi_config, abi_macros, input_adapters, json_pointer,
cpp_future, string_concat, value_t, json_fwd), and <version> for
std::nullptr_t, and removing <cstddef>.
- include <array>
- keep <cstddef> (nullptr_t, size_t; IWYU attributes them to <version> and <cstring>)
- tell IWYU not to suggest the headers that json.hpp provides: the amalgamated
json_view.hpp only includes json.hpp, so including them here would duplicate
their definitions
- drop edit_storage.hpp, which edit.hpp includes
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Silence a clang-tidy C++17 finding in the image test
bugprone-suspicious-stringview-data-usage flags data() of the source view
under C++17, although the length is passed and checked by the REQUIRE
before it.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Build the image link test without exceptions
load_result() exists only with exceptions; with JSON_NOEXCEPTION the test
loads the image with the check directly (a failed check aborts).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Regenerate the amalgamated headers
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Follow the renames from the review of the view builder
The review of #5620 renamed document_data's tape/tape_size/tape_cap to
nodes/node_count/node_capacity and made the decode arena a
std::vector<char>; adapt the code of this change to it.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Rename the emit lambda in the image writer for Qt compatibility
Qt defines emit as an empty macro, so a Qt header included before
json_view.hpp broke the lambda's declaration (see #5865). The slots
variables in the image tests are renamed as well.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address the review of json_document images
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Regenerate the amalgamated headers
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Add editable json_documents: set, push_back, insert, and erase
basic_json_document gets a second template parameter, Editable
(false by default), plus the aliases json_editable_document,
json_editable_view, ordered_json_editable_document and
ordered_json_editable_view.
Editable documents can change values and structure without
rewriting the source text: set()/push_back() on values, keys,
array indices and JSON pointers; insert() before an array
element; erase() of an object key, array index or JSON pointer.
New values and element sequences go into edit storage that the
document owns and never moves, so views keep referring to their
value across edits and a parsed node never moves. Read-only
documents walk the plain node array and are unaffected.
Strings are checked for UTF-8 on entry, so dump() of an editable
document never throws type_error.316. Binary values cannot be
stored (type_error.319).
A seeded differential test applies random edits to an editable
document and to the equivalent ordered_json and compares both
after every step.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix old clang and MSVC C4127 in editable json documents
Value-initialize the const std::less in find_parent (clang 3.4/3.6 do not
implement DR 253), and test the Editable template argument through a
function to avoid MSVC C4127.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Copy nested values into a document without recursion
Counting and copying the nodes of a view or a basic_json value into an
editable document recursed once per nesting level, so that a deeply
nested value overflowed the stack. The four functions now walk the value
with an explicit stack, as materialize() does.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Do not leave a half-assigned container when set_moved() throws
assign() rewrote the kind, length and extent of a slot before set_moved()
registered its new element sequence. set_moved() can throw
(std::bad_alloc from reserving the bookkeeping vectors) for a slot that
is not moved yet, which left a container without the moved flag that
showed its old children. The bookkeeping is now reserved first
(reserve_moved()), so that nothing after the first write to the slot can
throw. block_of() reserves before it marks anything for the same reason.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Check strings copied from another document for valid UTF-8
An editable document only holds valid UTF-8, but copy_scalar() copied the
strings and keys of a view of another document unchecked. A document of
a weaker check (or a borrowed text that changed after parsing) could
therefore bring ill-formed UTF-8 into it. The copy is checked now, with
the error that dump() reports for the string; copies within the same
document stay unchecked.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Allow the edit arena to grow up to 4 GiB - 1 bytes
append_text() doubled the capacity of the arena and rejected the result
if it exceeded 4 GiB, so that an arena of more than 2 GiB could not grow
even though the 32-bit offsets of nodes address 4 GiB - 1 bytes. The
capacity is now clamped to that limit (text_capacity()), and an append
is rejected only if the bytes themselves do not fit.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Set a null value below a pointer as basic_json does
set(json_pointer, value) turned a null parent into an object for every
last reference token, so "/a/0" and "/a/-" below {"a":null} made
{"a":{"0":1}}, where basic_json's operator[](json_pointer) makes an
array. A null parent now becomes an array for "-" and for digit tokens
(filled with nulls up to the index) and an object otherwise. An invalid
index is reported before the parent changes.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Replace a container by a scalar in constant time once it is linked
Replacing an array or object that spans several nodes by a scalar looks
up its parent from the root (find_parent), which is linear in the size
of the document: setting every element of a 20,000 element array took
more than half a second. The parent only has to switch to links once;
afterward the extent of the replaced value no longer matters. Nodes that
an entry of a moved sequence links to are now marked (node_flags::linked),
and assign() skips the lookup for them.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Update the json_view banner and type_error.319 for editable documents
The banner still described a read-only view, and the documentation of
type_error.319 listed set and push_back only, although insert rejects
binary values as well.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Assign the member that lookups find when set() meets duplicate keys
An editable document reads the last member of a repeated key (as the
read-only document does), but set(object, key, value) assigned the first
one: a view taken from object[key] before the call did not show the new
value. It now assigns the last member's value, keeps the key at the
position of its first occurrence (where materialize() puts it), and drops
the other members.
The seeded differential test now also edits documents whose objects
repeat keys and compares every lookup (operator[], at, find, value,
contains, count, and JSON pointers) with the parsed basic_json value.
A targeted test covers duplicates before and after edits that move the
object, in large objects with an index, in moved arrays, and in values
copied from other documents.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Assign the first member when set() meets duplicate keys again
Lookups find the first member of a repeated key again, so set(object, key,
value) assigns that member (keeping the key where it is) and drops the
others, as documented before, and a view taken from object[key] shows the
new value. This undoes the code and documentation changes of 5cbb8e6d6.
Lookups in edited objects (navigation of editable documents) stop at the
first match, like those of parsed objects.
The tests that commit added expect the first member from lookups now. In the
seeded differential test, the documents with repeated keys repeat them after
the real members; the reference holds the first members, which the edits
address and the lookups are compared with, while materialize() is compared
with what parse() makes of the document's dump.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix MSVC C4127 in the lookups of editable json documents
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix clang-tidy 22 findings in the json_view headers
Silence bugprone-casting-through-void for the SSE loads (a reinterpret_cast
would trip -Wcast-align=strict), use auto for a cast initialiser, add
parentheses to a mixed expression, and name bugprone-std-namespace-modification
in the NOLINTs of the tuple_size/tuple_element specialisations.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
(cherry picked from commit 0effbaa7a2)
* Fix clang-tidy 22 findings in unit-json_view_edit.cpp and edit.hpp
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Keep a raw string with a backslash out of a CHECK (MSVC C2017)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Make json_view.hpp pass include-what-you-use
ci_single_binaries runs IWYU with --error on every header. For json_view.hpp
it suggested adding <array> (std::array is used), and the headers that json.hpp
already provides (abi_config, abi_macros, input_adapters, json_pointer,
cpp_future, string_concat, value_t, json_fwd), plus <version> for
std::nullptr_t, and removing <cstddef> and edit_storage.hpp.
- include <array>
- keep <cstddef> (nullptr_t, size_t; IWYU attributes them to <version> and <cstring>)
- tell IWYU not to suggest the headers that json.hpp provides: the amalgamated
json_view.hpp only includes json.hpp, so including them here would duplicate
their definitions
- drop edit_storage.hpp, which edit.hpp includes
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Keep the view's edit headers clean for clang-tidy in C++17 mode
clang-tidy 22 checks the headers with -std=gnu++17 as well, where it asks
for a transparent comparator and std::make_unique. The regions map uses its
default comparator, and the allocation of the edit state keeps new (the
library is C++11) with a NOLINT.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Regenerate the amalgamated headers
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Follow the renames from the review of the view builder
The review of #5620 renamed document_data's tape/tape_size/tape_cap to
nodes/node_count/node_capacity and made the decode arena a
std::vector<char>; adapt the code of this change to it.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Expect count() to count every member in the edit tests
Since #5622, count() counts every member with the key, so documents
with duplicate keys no longer give 1.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Count matching keys as unsigned in the json_view edit tests
Like in the access test, adding the int result of the conditional to a
std::size_t fails -Werror=sign-conversion with GCC.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address the review of erase() in editable json documents
From gregmarr's review of #5630:
- invalid_iterator.202 is thrown by overloads 1 and 2 only; the page said
"every overload" and then excluded 3 in parentheses.
- The note on links says what moves (the links after the removed element,
as insert() moves them the other way) and that every iterator over the
container is invalidated, not only those after the removed element.
- edit.hpp compares keys with document_data::key_equals().
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Regenerate the amalgamated headers
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address the review of editable json documents
From gregmarr's review of #5630:
- set() with a JSON pointer below a null value checks for "-" before
scanning the token for digits.
- node_flags::storage is defined as escaped | edited, and the comment on
document_data::base says that the fourth slot is unused.
- source_extent() names its root check.
- The integer-index overloads of set(), insert(), and erase() use a named
trait instead of a negated condition, which astyle formats like the
library's other enable_ifs.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Regenerate the amalgamated headers
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address the rest of the review of editable json documents
From gregmarr's review of #5630:
- alloc_nodes() gives a request of half of the next chunk or more a chunk
of its own, so that the rest of the current chunk stays available for
smaller requests instead of being abandoned.
- The comments on edit_state, append_text(), and reserve_moved() say that
views stay valid across every edit (not just the first), that n is 0 for
empty strings, and why both capacities are checked.
- The architecture page names flag bits 3, 4, and 5 (linked) separately
and says that a discarded view has no node; erase.md rewords overload 3
and says why it never throws invalid_iterator.202.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Call alloc_nodes() before reading chunks.back() in the json_view chunk test
The operands of == are unsequenced, and MSVC evaluated
d->edits->chunks.back() before alloc_nodes() added the new chunk, so the
check compared the result with the previous chunk and failed on
msvc-arm64 (Debug and Release).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Scan json_view strings with SIMD and index large objects
Speed up json_view's parser with SIMD scanning and a hash table
for large objects.
Long runs of string bytes are scanned 16 bytes at a time with NEON
(AArch64, GCC and Clang) and SSE2 (x86-64), both baseline
instruction sets. Keys keep 16 table checks before the vector
loop, because their lengths repeat from record to record; string
values get 8, because their lengths vary more. Non-ASCII text is
validated 16 bytes at a time with simdjson's "lookup4" check
(Keiser and Lemire, 2021), with NEON on AArch64 and, on x86-64,
with SSSE3. SSSE3 is not part of baseline x86-64, so the check is
compiled for SSSE3 with a function attribute and used only where
CPUID reports it, which all x86-64 CPUs since about 2011 do; the
answer is cached in a statically initialized atomic, so there is
no guard of a local static and no global constructor. The same
input is accepted either way. JSON_VIEW_NO_SIMD selects the
portable code.
On x86-64, string runs are now checked vector-first: one SSE2
compare from the first byte finds the end of most keys and short
values, instead of a branch per byte for the first 8-16 bytes.
AArch64 keeps the byte-wise steps, where a NEON mask costs more and
the branches predict well. Entering an object or array no longer
stalls: open() stores the parent's frame field by field instead of
building it on the stack and reading it back with wider loads,
which waited for the narrower stores to retire.
Objects with 128 members or more get an open-addressing hash table
built when the object closes, so operator[], at(), find(),
contains(), count(), value(), and JSON pointers take constant time
on average in such objects; of duplicate keys, the first is kept,
as for the linear search. The idea comes from Boost.JSON.
simdjson is credited in simd.hpp's SPDX block, the README, and
license.md.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix GCC module build and MSVC C4127 in the json_view SIMD code
GCC ignores the target attribute in modules, so the SSSE3 dispatch is
disabled for the module interface (the check stays portable, SSE2 is kept).
Make the 8-vs-16 byte unrolling condition in scan_string_run a
preprocessor/template split to avoid a constant condition (C4127).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Remove an accidentally committed GCC module cache
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Bound the probe length of the large-object hash index
The key hash is not seeded, so keys chosen to collide made building the table quadratic (20,000 colliding keys took 470 ms to parse). A key may now sit at most 64 slots from its home slot; if a key would sit further away, the table is dropped and the object is searched linearly. Lookups stop after the same distance.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Release the spare capacity of the hash index in shrink_to_fit
shrink_to_fit() now trims the tables of large objects like the node array and the decoded strings, and the list of large objects is released as soon as the tables are built.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Use the portable string scan on non-x86 targets that define __SSE2__
WebAssembly with -msse2 defines __SSE2__, but has no <cpuid.h> or x86 intrinsics headers.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Document the hash index number in node::extra
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Keep the last of duplicate keys in the object hash index
Lookups in objects with 128 members or more now return the last member of
a repeated key, like the linear search of smaller objects and like
materialize() and basic_json::parse(). build_object_index let the first
occurrence win, so the same text gave different results depending on the
size of the object.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Keep the first of duplicate keys in the object hash index again
Lookups in objects with 128 members or more return the first member of a
repeated key again, like the linear search of smaller objects. This undoes
the code change of a69542046; its test now expects the first member from
lookups (with and without a table) and the last value from materialize()
and basic_json::parse().
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix clang-tidy 22 findings in the json_view headers
Silence bugprone-casting-through-void for the SSE loads (a reinterpret_cast
would trip -Wcast-align=strict), use auto for a cast initialiser, add
parentheses to a mixed expression, and name bugprone-std-namespace-modification
in the NOLINTs of the tuple_size/tuple_element specialisations.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix clang-tidy 22 findings in unit-json_view.cpp and unit-json_view_builder.cpp
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Avoid GCC useless-cast and strict-overflow warnings in unit-json_view.cpp
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Index strings with size_t in the colliding-keys test (MSVC C4244 on Win32)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Make json_view.hpp pass include-what-you-use
ci_single_binaries runs IWYU with --error on every header. For json_view.hpp
it suggested adding <array> (std::array is used), the headers that json.hpp
already provides (abi_config, abi_macros, input_adapters, json_pointer,
cpp_future, string_concat, value_t, json_fwd), and <version> for
std::nullptr_t, and removing <cstddef>.
- include <array>
- keep <cstddef> (nullptr_t, size_t; IWYU attributes them to <version> and <cstring>)
- tell IWYU not to suggest the headers that json.hpp provides: the amalgamated
json_view.hpp only includes json.hpp, so including them here would duplicate
their definitions
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Move the dump, comparison, and large-object tests to unit-json_view_dump.cpp
The dump and comparison tests moved to their own file in the dump pull
request; the hash index tests of this branch join them there.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Regenerate the amalgamated headers
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Follow the renames from the review of the view builder
The review of #5620 renamed document_data's tape/tape_size/tape_cap to
nodes/node_count/node_capacity and made the decode arena a
std::vector<char>; adapt the code of this change to it.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Rename slots in the object index for Qt compatibility
Qt defines slots as a macro, so a Qt header included before
json_view.hpp broke the declarations of the local variable (see #5865).
The test variable is renamed as well.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address the review of the json_view object index and SIMD scan
From gregmarr's review of #5628:
- Docs and comments say that the hash index is one of the object's keys.
- object_index.hpp: conditional_static_cast for the hash, "slot" for the
table position, no flag for a degenerate table, and a static_assert
that the smallest indexed object is larger than the probe window.
- scan.hpp unrolls with NLOHMANN_VIEW_REPEAT_FIRST_8/SECOND_8.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Regenerate the amalgamated headers
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Encode JSON_VIEW_NO_SIMD and JSON_VIEW_USE_SSSE3 in the ABI namespace
Both macros select between definitions of the same inline functions, so
translation units that disagree on them violated the ODR. Like the other
ABI-affecting macros, they now add a tag to the inline namespace (_vns and
_vssse3), and the docs replace the "Define consistently" warning with the
usual ABI compatibility note. From gregmarr's review of #5628.
nlohmann_json.natvis covers twice as many tags now, so it lists four times
as many namespaces (14.7 MB instead of 3.4 MB).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Regenerate the amalgamated headers
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Do not include type_traits.hpp directly in the object index
json.hpp, which object_index.hpp already includes, provides it; included
directly, the json_view amalgamation (which only treats nlohmann/json.hpp
as external) copied the json internals into json_view.hpp a second time,
so the fuzzers, which include both single headers, failed with
redefinitions (make_void, integer_sequence, priority_tag, ...).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Compare keys with key_equals() in the object hash index
As the linear lookups do since the second review round of #5622.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Regenerate the amalgamated headers
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Add dump() and comparisons to json_view
Add basic_json_view::dump() and the comparison operators, and read
floats from the parser's digit layout instead of rescanning the
token.
dump(indent, indent_char, ensure_ascii, number_format) writes a
value the way ordered_json::parse(text).dump() writes it for the
same arguments: members in document order, all of them should a
key occur more than once; strings escaped by the same rules, using
the library's scanning kernels; floats written with the library's
to_chars conversion, so the output equals basic_json's byte for
byte; integers copied from the source, where they are already
canonical, except -0, which parse() reads as 0. There is no
error_handler argument, because the view only holds valid UTF-8.
number_format::source copies numbers exactly as they appear in the
source (e.g. "1.50", "1E2", "-0"), which basic_json cannot provide.
operator<< takes the indentation from the stream width, as for
basic_json. The writer walks iteratively, so nesting depth is
limited by memory only.
operator== and operator!= compare two views, or a view and a
basic_json value in either order, by the rules basic_json's
operator== uses: numbers compare by value across their types,
objects compare by their members with duplicate keys resolved as
parse() resolves them, member order matters only where the object
type keeps one, and discarded views compare as discarded basic_json
values do, including under JSON_USE_LEGACY_DISCARDED_VALUE_COMPARISON.
Nothing is materialized except single scalars.
While parsing, the view now records where the integer digits, the
fraction digits, and the exponent of a float token are, so floats
and doubles with at most 19 digits are read from that layout with
the library's decimal_to_float() instead of rescanning the token.
Both round correctly, so the values are those of parse(). get<double>(),
materialize(), dump(), and the comparisons all use it.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Avoid raw string with escapes inside CHECK macro (MSVC C2017)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Size the dump buffer of a small value by its own extent
The source extent of a value was read from the next node, falling back to the rest of the document when that node held a decoded string. dump() of a small value could thus allocate a buffer as large as the document. Skip a few such nodes, cap the fallback estimate, and shrink a buffer that is much larger than its output.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Hold the documents of the dump and comparison tests in names
root() of a temporary document is deleted: its views would dangle.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Do not shrink the output of dump()
finish() copied the whole output when the buffer was more than twice as
large as the result (citm dump +11%). The tighter source_extent()
estimate already keeps the buffer of a small value small.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Document and test first-wins lookups next to dump() and ==
dump() writes every member and == resolves duplicate keys as parse()
does, whereas lookups find the first member of a duplicate key.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix the clang-tidy 22 findings in json_view and its tests
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Split the dump and comparison tests off unit-json_view.cpp
The MinGW linker of the Windows clang jobs cannot link object files with more
than 32767 sections ("relocation truncated to fit: IMAGE_REL_AMD64_REL32
against `.rdata'"). unit-json_view.cpp reaches that limit as the stack
grows, so its "json_view dump" and "json_view comparison" test cases move
into unit-json_view_dump.cpp. The test generator and has_duplicate_keys()
that both files use move into json_view_test_helpers.hpp.
The new file mentions JSON_HAS_CPP_17, so it is built for C++17 like the file
it was split from.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Make json_view.hpp pass include-what-you-use
ci_single_binaries runs IWYU with --error on every header. For json_view.hpp
it suggested adding <array> (std::array is used), the headers that json.hpp
already provides (abi_config, abi_macros, input_adapters, json_pointer,
cpp_future, string_concat, value_t, json_fwd), and <version> for
std::nullptr_t, and removing <cstddef>.
- include <array>
- keep <cstddef> (nullptr_t, size_t; IWYU attributes them to <version> and <cstring>)
- tell IWYU not to suggest the headers that json.hpp provides: the amalgamated
json_view.hpp only includes json.hpp, so including them here would duplicate
their definitions
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Regenerate the amalgamated headers
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Follow the renames from the review of the view builder
The review of #5620 renamed document_data's tape/tape_size/tape_cap to
nodes/node_count/node_capacity and made the decode arena a
std::vector<char>; adapt the code of this change to it.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Keep the stream includes of json_view.hpp for IWYU
json_view.hpp uses std::istream and std::ostream itself, but IWYU sees
them through the exported json.hpp and asked to remove both includes,
which fails ci_single_binaries (single_json_view).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address the review of json_view dump() and comparisons
From gregmarr's review of #5624:
- The output buffer takes string literals without a length.
- write_float() names the IEEE single/double condition once, as the
library's serializer does.
- equal() checks the types first and compares with !=; the reference to
the top frame is noted as dangling after pop_back().
- dump.md: one paragraph per case in the notes.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Regenerate the amalgamated headers
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Exclude all json_view test files from Infer's reports
The same std::unique_ptr false positive (USE_AFTER_DELETE) also shows for
two json_documents parsed in one lambda in unit-json_view_dump.cpp
(ci_infer). It affects any test that holds several documents, so the
exclusion now covers tests/src/unit-json_view*.cpp; ASan and Valgrind
cover these tests.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Allow the string literal parameter of the view serializer's put() for clang-tidy
The overload takes the literal by array reference so that its length is a
template parameter, like scan_literal() and the C array overloads of
to_json() and from_json().
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Bind the serializer's frame only after the check for its end
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Regenerate the amalgamated headers
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Add element access, iteration, values, and JSON pointers to json_view
Give basic_json_view the read-only access functions of basic_json:
operator[] and at() with keys and indices, front()/back(), find(),
contains(), count(), begin()/end() and cbegin()/cend(), items()
with structured bindings from C++17 on, and type_name().
Exceptions have the ids and messages of the const functions of
basic_json. Where basic_json has undefined behavior the view
answers safely: operator[] with a missing key or an out-of-range
index returns a discarded view, and front()/back() of an empty
container throw invalid_iterator.214. Objects are iterated in
document order, and all members are visited; duplicate-key lookups
find the first member (as yyjson and simdjson do), while parse(),
materialize(), and the map conversions keep the last value, as
parse() does. Keys of up to 16 bytes are compared with two
overlapping loads.
Add value conversions: get<T>()/get_to() for arithmetic types,
bool, nullptr_t, strings (std::basic_string copied,
string_view_t without a copy), BasicJsonType, views, std::vector,
and maps with string keys; get_string() for the string without a
copy; number_token() for the number exactly as written in the
source; value() with keys and JSON pointers; and operator[]/at()/
contains() with JSON pointers. Everything else, including types
with from_json(), goes through materialize() of that subtree.
get<T>() of arithmetic types is inlined down to the conversion, so
reading an integer needs no call.
detail::json_pointer_access exposes a pointer's reference tokens
to code outside basic_json.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Make view lookups last-wins and chained access safe
* Lookups (operator[], at, find, contains, count, value, JSON pointers)
return the last member of a duplicate key, as materialize() and
parse() keep it.
* operator[] on a discarded view returns a discarded view instead of
throwing, so v["a"]["b"] is safe for a missing "a".
* operator[] and at() take any integer type (not only int and size_t),
fixing ambiguous calls with unsigned, long, std::int64_t, ...
* Fix the operator[] documentation, which claimed a discarded view for
a type mismatch where type_error.305 is thrown.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Replace the removed operator bool of basic_json_view in the header and its tests
A view is tested with is_discarded(); the lookups in at(), value() and
contains(json_pointer) and the unit tests no longer rely on the explicit
conversion to bool.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Use named documents in the json_view tests that called root() on a temporary
root() of an rvalue document is deleted, because the views would dangle.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Test discarded views with is_discarded() in the operator[] docs and examples
The explicit conversion to bool of basic_json_view is gone.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Return the first member of duplicate keys from lookups again
Looking up the last member of a duplicate key cannot stop at a match, so
every lookup scanned the whole object (1.6 to 3.4 times slower for small
objects). operator[](key), at, find, value, contains, count, and JSON
pointer resolution return the first member again, as yyjson and simdjson
do; materialize() and get<map>() keep the last value, as parse().
The documentation says so in the feature page and on each lookup page,
and explains how to get the value parse() would give. The integer index
templates and the discarded chaining of operator[] stay.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Do not force-inline the view's functions on MSVC
__forceinline makes MSVC report C4714 for function templates it cannot
inline (get<std::string>), an error under /WX. The forced inlining is
only a performance hint, so use plain inline there.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Make json_view.hpp pass include-what-you-use
Mark the includes that are needed by design (json.hpp, the macro
unscope, <tuple> for the tuple_size specialization), take size_t from
<cstring>, and drop the forward declaration of basic_json_document that
the friend declaration makes redundant.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Avoid GCC useless-cast warnings in the integer index tests
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Make json_view.hpp pass include-what-you-use
ci_single_binaries runs IWYU with --error on every header. For json_view.hpp
it suggested adding <array> (std::array is used), the headers that json.hpp
already provides (abi_config, abi_macros, input_adapters, json_pointer,
cpp_future, string_concat, value_t, json_fwd), and <version> for
std::nullptr_t, and removing <cstddef>.
- include <array>
- keep <cstddef> (nullptr_t, size_t; IWYU attributes them to <version> and <cstring>)
- tell IWYU not to suggest the headers that json.hpp provides: the amalgamated
json_view.hpp only includes json.hpp, so including them here would duplicate
their definitions
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix the clang-tidy 22 findings in json_view and its tests
The same changes as on the dump branch, where they were first made, so
that this branch passes clang-tidy on its own.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Regenerate the amalgamated headers
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Report one-character non-numeric array indices like basic_json in the view
Develop's #5800 made basic_json report a JSON pointer array index such as
"/a" as parse_error.109 instead of out_of_range.404, as longer ones were;
the view's array_index() follows, with "-" and the empty token still
unresolved.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address the review of json_view element access
From gregmarr's review of #5622:
- count() counts every member with the key, so count(key) > 1 detects
duplicate keys; it was contains() as 0/1 before.
- get_to() passes types that get<T>() does not convert directly to
from_json with the existing value, as basic_json::get_to() does, so that
a from_json which sets only the fields present keeps the others.
- array_index() checks the empty token and "-" first; the results are
unchanged.
- Docs: no "if an exception is thrown" where a view never changes
anything; the borrowed source must also stay unmodified; items() on a
temporary view is safe; back()/front() notes shortened; the basic_json
see-also entries say how the view's results can differ.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Regenerate the amalgamated headers
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Count matching keys as unsigned in the json_view access test
The conditional added 1 or 0 as int to a std::size_t, which GCC rejects
with -Werror=sign-conversion (ci_test_standards_gcc).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address the second review round of json_view element access
From gregmarr's review of #5622:
- begin()/end(): "a null value or a discarded view iterates as an empty
range", singular like the primitive case before it.
- The class page's list of function groups has no trailing commas.
- document_data::key_equals() compares a key node with a string, for the
key comparisons that were spelled out in each lookup.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Regenerate the amalgamated headers
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Write the negated enable_if conditions of json_view with negation<>
With a ! or && in the condition, astyle spaces the whole template
header (template < ... >); detail::negation and a named trait keep it
compact, as the library writes such conditions elsewhere. From gregmarr's
review of #5630.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Regenerate the amalgamated headers
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Keep detail::negation out of json_view.hpp's enable_ifs
include-what-you-use asks json_view.hpp to include
detail/meta/type_traits.hpp for it, which the single header must not do
(the amalgamation would duplicate json's internals). The two conditions of
value() go through a trait in value.hpp, like the others.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Regenerate the amalgamated headers
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Add json_document and json_view: node index, parser, and document
Add json_document and json_view, a read-only, zero-copy index of a
JSON text, as the first public slice of the zero-copy view (#5295).
A parse produces a flat array of 16-byte nodes in document order,
one per value and one per object key. Strings stay in the source
text; escaped strings are decoded into an arena. Integers are
converted while their digits are in the cache; floats keep only
their digit layout and are converted on read. Containers store the
size of their subtree, so a reader can step over one in constant
time. A document makes a handful of allocations, however many
values it has.
The parser accepts exactly what json::parse accepts, with every
combination of ignore_comments and ignore_trailing_commas, with and
without a trailing NUL, and under JSON_STRICT_NUL_HANDLING. It is
portable C++11 and does not depend on byte order.
basic_json_document adds parse, parse_copy, accept, read (reuses a
document's memory), root, is_discarded, source, owns_source,
node_count, memory_usage, and shrink_to_fit. basic_json_view adds
type, the is_* queries, operator bool, size, empty, materialize,
and source_offset. A parse error throws the same exception
basic_json::parse would throw for the same input, message and
position included.
detail::abi_config keeps JSON_STRICT_NUL_HANDLING and
JSON_USE_LEGACY_DISCARDED_VALUE_COMPARISON readable after json.hpp
undefines them, in the ABI namespace so they always match the
basic_json in use.
A NUL byte that ends a // comment is the end of the input, as in
parse() since #5696.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix document_data for old Clang (3.4-3.6)
Drop the empty braced NSDMIs of the std::string members: old Clang
rejects the defaulted constructor when it is used by a member
initializer before the end of the class definition.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Exclude libstdc++'s string_view comparison from the sign-change check
basic_string_view::_S_compare stores the difference of two lengths in a
signed difference_type on purpose, which -fsanitize=integer reports for
every comparison with a shorter view (test-json_view_cpp17).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Reject integer lengths in json_document::parse
parse(ptr, len) compiled: len converted to allow_exceptions, and ptr was
read as a C string, past the end of a buffer without a terminating NUL.
Delete the overloads of parse, parse_copy, accept, and read that take an
integer other than bool where the flags are expected.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Copy const rvalue strings and document the accepted inputs
json_document::parse(std::move(const_string)) failed to compile with
"no matching read_kind": only a non-const rvalue std::string can be moved
from. Treat a const rvalue as a copied byte container.
parse() does not accept everything BasicJsonType::parse() does: a FILE*
and pointers to or arrays of wide characters are rejected at compile time.
List the supported inputs instead.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Remove the conversion to bool from json_view
explicit operator bool meant "refers to a value", which silently differs
from what a basic_json converts to. Use !v.is_discarded() instead.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Delete json_document::root() on temporary documents
auto v = json_document::parse(text).root() compiled and left the view
dangling. Delete the overload for rvalue documents, take a named document
in the tests, and document the lifetime rule.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Store node integers by halves on big-endian targets
The non-little-endian path of the node writer wrote the integer's native
word over len and next, so len got the high half there. Compose and split
the value explicitly (len is the low half, next the high half); the
little-endian path stays a plain memcpy.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Check the node array size for overflow
reserve(n) and the growth of the index computed n * sizeof(node) without a
check, which wraps around on 32-bit targets for inputs of about 1 GiB and
allocates a too small array. Throw std::bad_alloc for a count beyond the
address space and clamp the growth step to it.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* State the exact input size limit of json_document
The check rejects inputs of 0xFFFFFFF0 bytes or more, but the exception
message and the documentation said 4 GiB. Name the limit once
(max_input_size), and state 4 GiB minus 16 bytes in the message and the
documentation. Test the limit with a container that only claims the size.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Remove the unused legacy_discarded_value_comparison from abi_config
Nothing reads it: the view does not compare values.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Align the banner comment of json_view.hpp
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fuzz json_document with exact-size buffers and parse options
The fuzzer only parsed a std::string with default options. Also parse an
exact-size byte vector, which has no NUL after its last byte and takes the
bounds-checked path, and derive ignore_comments and ignore_trailing_commas
from the first input byte, comparing against json::parse and json::accept
with the same options.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Give document_data a user-provided constructor
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix clang-tidy findings in unit-json_view_builder.cpp
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Annotate intentional patterns in unit-json_view.cpp for clang-tidy
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Suppress Flawfinder's read() finding on the deleted overload
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Check the steps of the deep-nesting test separately
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Make max_input_size a function (GCC -Wunused-const-variable)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Make json_view.hpp pass include-what-you-use
ci_single_binaries runs IWYU with --error on every header. For json_view.hpp
it suggested adding <array> (std::array is used), the headers that json.hpp
already provides (abi_config, abi_macros, input_adapters, json_pointer,
cpp_future, string_concat, value_t, json_fwd), and <version> for
std::nullptr_t, and removing <cstddef>.
- include <array>
- keep <cstddef> (nullptr_t, size_t; IWYU attributes them to <version> and <cstring>)
- tell IWYU not to suggest the headers that json.hpp provides: the amalgamated
json_view.hpp only includes json.hpp, so including them here would duplicate
their definitions
- export json.hpp, keep macro_unscope.hpp, and drop the unused forward declaration of the document class
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Read the node array base after emit() in the view builder's open()
emit() moves the node array when it grows, and the subtraction read base
in the same expression, so the order was unspecified. MSVC Release builds
without forced inlining read the old base; the container index then
pointed outside the array and close() wrote out of bounds.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Silence a clang-tidy use-after-move finding in the json_view test
const_text is const, so std::move does not move from it; the second call
is part of the const rvalue test.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Regenerate the amalgamated headers
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address the review of the view builder and document
Changes promised in the replies to the review of #5620:
- read() reuses the owned copy: a copied input (an rvalue byte container,
a stream, FILE*, wide characters, an iterator range) is appended to the
cleared owned buffer instead of a fresh std::string being move-assigned
over it. A moved-in std::string still replaces it.
- The error replay runs the library parser with a SAX handler that stores
nothing and throws the reported exception, instead of
BasicJsonType::parse(), which built a DOM up to the error.
- Pointers to and arrays of wide characters, and FILE*, take the input
adapter path (converted to UTF-8 into an owned buffer) instead of
failing a static_assert. Wide pointers are NUL-terminated.
- Streams are read in blocks through rdbuf() into the buffer, instead of
character by character through the input adapter; eofbit is set at the
end, as by the library's stream adapter.
- tape/tape_size/tape_cap are now nodes/node_count/node_capacity.
- The builder's members get descriptive, m_-prefixed names.
- The shallow frame stack is a std::array (in a wrapper with a
user-provided constructor: -Weffc++ wants the member initialized, and
value-initializing the 64 frames costs up to 30 % on small documents).
- The value dispatch macro has no NEXT parameter; the switch is left with
break, and the macro is undefined after its last use.
- literal() and digits_exceed() deduce the length from the string literal;
each literal has one helper; the two `if (!is_float)` blocks are merged.
- arena_ensure() sits right before the writes; the decode arena is a
std::vector<char>.
- The builder records the deepest nesting, and materialize() sizes its
stack from it once (growing it only if a subtree is deeper).
- Comment fixes (hex4, slow_string, the NUL rule, put_utf8, 0 - m, the
arena resize, grow()) and formatting (macro backslashes, digit cases,
one power of ten per line, value_t assertion next to is_container()).
- build_with's template parameters follow the builder's order.
Greg's idea of keeping the source offset in `off` and the arena offset in
`next` for escaped strings was measured and not adopted: string access got
7-18 % slower with a select, and 3.5-18 % slower with a branch-free field
pick (Apple M-series, clang -O3), parsing was unchanged.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Apply the review's wording fixes to the json_view documentation
- Copying is disabled because of its cost, not because of two documents
borrowing the same buffer; the unique_ptr follows from move-only.
- Inputs: streams, FILE*, and wide characters (strings, arrays, pointers)
are all supported and owned; point to parse_copy() for owned copies.
- The error replay only runs for failing input, builds no values, and is
skipped with allow_exceptions = false.
- read() writes copied inputs into the previous owned copy.
- Views can dangle like std::string_view, and why there is no
shared/weak pointer design.
- The 4 GiB limit (16-byte nodes, 16-byte scanner steps), streams being
read to the end first, which macros apply, why materialize() has no
source positions, and wording nits (accept() vs basic_json::accept(),
is_binary, is_discarded, see-also lists, size table, "back into").
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Say that json_view converts integers while parsing
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address the review: null inline nodes, trait spelling, and max() in json_view
- inline_nodes is null when the document has no inline nodes, instead of
pointing past the end of the allocation; shrink_to_fit() no longer
passes it to memcpy() for an empty node array
- is_byte_char and is_wide_char use conjunction/disjunction, which
astyle leaves unspaced
- append_stream() uses (std::max) for the count read
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Include materialize.hpp in the view builder test
The test calls detail::view::materialize(), which only the single header
pulled in; the multiple-header build (JSON_TEST_USING_MULTIPLE_HEADERS)
failed with "'materialize' is not a member of detail::view".
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Rename emit() in the view builder for Qt compatibility
Qt defines emit as an empty macro, so a Qt header included before
json_view.hpp broke the declaration of the builder's emit(), see
issue #5865. It is now called push_node().
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Do not copy an empty run into an unallocated arena
memcpy() must not be passed a null pointer even for zero bytes, which
arena_run() did for an empty run while the arena was still empty
(ci_test_clang_sanitizer: "null pointer passed as argument 1" in
builder.hpp). Only the slow path can see such a run.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Exclude the view builder test from Infer's reports
Infer's std::unique_ptr model confuses the values that the test's build()
helper returns by value and reports USE_AFTER_DELETE for one of them after
the destructor of an unrelated temporary (ci_infer, 12 reports, all in
tests/src/unit-json_view_builder.cpp). AddressSanitizer is clean on the
same tests.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Skip the deleted-overload checks of json_view on MSVC 2015
MSVC 2015 does not treat selecting a deleted function in decltype as a
substitution failure, so is_detected reports the deleted
parse/parse_copy/accept/read(ptr, len) overloads as callable and the
static_asserts in "integer arguments do not compile" fail (AppVeyor
VS2015 jobs). Calling them still fails to compile there. Guard the
negative checks as unit-delete_deprecated_functions.cpp does.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The check ran generate_natvis.py from the develop checkout and took only
the template from the pull request, so a pull request that changes the
generator (such as #5868, which limits the namespaces to two ABI tags)
could never pass: develop's generator rewrote the committed file. The
check now runs the pull request's generator, which loads its own
template, as the macro_builder step already does.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Test that the contiguous fast paths never read outside the input
The bulk string scanner, the SWAR searches, the UTF-8 prefix validator,
the 8-digit number loop, the 8-byte indentation skip, and the BSON/BON8
bulk string readers load several bytes at a time from contiguous input.
ASan misses over-reads into the slack of the same heap allocation, and
fuzzers rarely end an input exactly at a page boundary.
The new test copies each input directly in front of, and directly
behind, an inaccessible page (mmap/mprotect on POSIX,
VirtualAlloc/VirtualProtect on Windows), so reading even one byte
outside the input crashes. It covers the string-scan functions directly
and json::parse, accept, sax_parse (with and without comments) and all
binary readers through the public API, with inputs ending at every byte
of strings, escapes, UTF-8 sequences, numbers, literals, indentation
runs, and binary documents. Each result must match parsing the same
input from a std::string.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Rename the reader parameter to avoid a flawfinder false positive
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Make the guard page test work without exceptions and under UBSan
Under JSON_NOEXCEPTION, the error paths abort instead of throwing, so the
outcomes are now computed with allow_exceptions=false there. Copying the
input with std::copy from char to unsigned char tripped Clang's implicit
conversion sanitizer; std::memcpy avoids the conversion.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Test parse_float_native() at the guard page
parse_float_eisel_lemire() was removed by #5738, so the test no longer
compiled. The lexer converts floats from its token buffer, so the JSON
section never places a float token at the page boundary; the section now
calls parse_float_native() for float and double on every complete number
token directly and compares the result bit for bit with converting the
same token from a std::string.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix clang-tidy findings in the guard-page tests
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Convert explicitly between bytes and characters in the guard-page tests
The clang sanitizer job (-fsanitize=integer, x86-64 where char is signed)
reported the implicit char -> unsigned char conversions of std::copy() into
the guarded buffer and of the std::string built from the binary inputs.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
mkdocs --strict rejects links to pages that only exist as redirects.
Also link operator<< to operator_ltlt.md and operator>> to
operator_gtgt.md instead of the other way round.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Regenerate tools/api_checker/api_surface.json with the CI libclang
(18.1.1 on Linux) and point the new ordered_map links in
features/performance.md to api/ordered_map/index.md.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
- ordered_map.hpp: develop's refactored members, with this branch's
@sa links on every public member
- byte_container_with_subtype operator==/!= and ordered_map docs:
develop's reviewed text (#5638); drop the now unreferenced examples
- basic_json.md: keep both version-history additions
- add the docset entries that develop's style check now requires for the
25 API pages this branch adds
- regenerate tools/api_checker/api_surface.json for develop's API
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Conflicts: none textual. Follow-up fixes for develop's changes:
- docs/mkdocs/docs/api/ordered_map/index.md: develop's new paragraph
(#5609) was written for api/ordered_map.md; fixed its relative
ordered_json.md link for the page's new location, plus two
pre-existing links from an earlier merge (ordered_json.md,
features/object_order.md)
- tools/api_checker/api_surface.json: regenerated with libclang 18.1.1
on Linux (swap() noexcept now includes json_base_class_t; new
integral-key contains/count/find/value overloads)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Codacy runs two docstring checkers with opposite rules for module
docstrings: with the summary on the first line it reported D213, with
the summary on the second line it reports D212. A one-line docstring
satisfies both. Keep the summary as the docstring and move the details
into a comment below it; nothing reads the module docstrings.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The develop merge regenerated api_surface.json on macOS, whose libclang
spells the name of the deprecated conversion operator as
"operator nlohmann::json_pointer::string_t_helper<...>::type". CI's
pinned libclang 18.1.1 on Linux spells it
"operator typename string_t_helper<...>::type", so check_api_docs
reported the file as out of date. This restores CI's spelling, which is
byte-identical to the file CI regenerated.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Put multi-line docstring summaries on their own line, as a single
sentence followed by a blank line (pydocstyle D205, D209, D213, D415).
Annotate the subprocess import and calls with nosec: they only run
fixed argument lists, never through a shell (Bandit B404, B603, B607).
Do the same for the three broad except clauses in extract_api.py,
which deliberately fall through to the next libclang candidate or skip
an unresolvable alias (Bandit B110, B112).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Resolve the conflict in docs/mkdocs/docs/home/architecture.md by taking
develop's rewritten page and pointing its ordered_map links at the moved
api/ordered_map/index.md. Also update the ordered_map.md links develop
added in ordered_json.md, object_order.md, and template_parameters.md.
Regenerate api_surface.json for the new BON8 functions and the changed
comparison operator. Restrict the documented_non_public leak check to
@sa URLs into json.nlohmann.me: develop now uses @sa to link GitHub
issues from private members such as copy_structured, which is not a
documentation leak.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
develop added JSON_HEDLEY_WARN_UNUSED_RESULT to a number of basic_json
members (#5520) and reworked operator<<'s output adapter, so the
committed API surface no longer matched what extract_api.py produces and
the "Check for uncommitted API surface changes" step failed.
This is the patch the workflow itself uploads for that purpose; the
changes are limited to recorded signatures plus the way libclang spells
json_pointer's conversion operator.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
develop gained iterator+sentinel support for accept()/parse()/sax_parse()/
from_cbor()/from_msgpack()/from_ubjson()/from_bjdata()/from_bson() (#5205)
since this branch was created, which check_api_docs.yml's drift check caught
immediately: GitHub's pull_request trigger checks out the PR-vs-base merge
commit, not the PR branch in isolation, so CI was comparing the committed
api_surface.json (generated before that upstream change existed) against a
fresh extraction of a json.hpp that already had it.
This superseded an earlier, incorrect diagnosis of the same symptom (a
JSON_HAS_RANGES cross-environment pin) -- that fix remains in place since it's
still a real, independent determinism improvement for three other
JSON_HAS_RANGES-gated locations in type_traits.hpp, but it wasn't the actual
cause of this particular mismatch: these new overloads turned out to be
unconditional (SFINAE-gated via can_compare_ne, not preprocessor-gated), so
no environment-detection difference was involved here at all.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
extract_api.py's own extraction wasn't deterministic across machines, which
CI's drift check caught immediately: JSON_HAS_RANGES auto-detects via the
standard library's __cpp_lib_ranges feature-test macro, which isn't reliably
gated to C++20 mode by every stdlib -- undefined under -std=c++17 with macOS's
libc++, but defined under the identical flag with the Ubuntu stdlib CI uses,
so parse()/accept()/from_*() extracted different signatures purely depending
on which machine ran the extraction. Pinned to -DJSON_HAS_RANGES=0: the
deterministic and safe choice, since pinning to 1 was tried first and found to
fail to parse on a stdlib without full <ranges> support even when the macro
claims otherwise.
Also found and fixed a second, independent source of the same class of drift:
get_identity_name() used cursor.spelling verbatim for CONVERSION_FUNCTION
cursors, which libclang renders as its own internally-canonicalized form of
the return type rather than what's literally written. Confirmed for
json_pointer::operator string_t() spelling differently on two machines
pinned to the identical libclang==18.1.1 wheel, with the JSON_HAS_RANGES fix
above ruled out as the cause. Now derived from the cursor's own raw source
text instead, immune to libclang's dependent-type resolution differences and
incidentally more readable than the libclang-internal forms it replaces.
Bumped SURFACE_FORMAT_VERSION to 3 and regenerated all 27 history snapshots
and the committed api_surface.json; both fixes are documented in
tools/api_checker/history/README.md's format-history log.
Also fixes diff_api.py's format_version guard, which only compared the two
loaded surfaces against each other and never against SURFACE_FORMAT_VERSION
(what this build actually understands) -- two surfaces on the same,
newer-than-expected format_version would have silently passed the guard.
Remaining fixes are the concretely actionable findings from Codacy's review
of the new tools/api_checker/ files: unused imports/variables, a stray
f-string with no placeholders. Left the docstring-formatting nitpicks
(pydocstyle D2xx/D4xx) and generic subprocess-usage notices alone -- the
former has no established convention elsewhere in this codebase's Python
tooling to conform to, and the latter are inherent to a dev tool that shells
out to git/clang with developer-controlled arguments, not user input.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Adds tools/api_checker/: extract_api.py derives the public API surface directly
from the libclang AST (independent of documentation status), check_docs.py flags
public entries missing @sa links (and @sa on non-public ones), diff_api.py does
an overload-aware breaking/feature diff between two refs, check_macros.py cross-
checks documented macros against #define sites, and snapshot_release.py backfills
immutable per-release surface snapshots into tools/api_checker/history/ (v3.1.0
through v3.12.0) so diff_api.py can compare releases without live extraction.
POLICY.md documents what counts as public API and what stability is guaranteed.
Running this tooling against the current tree found and fixed a real documentation
backlog: ~25 new API doc pages (ordered_map's methods, json_sax's ctor/dtor/
operator=, byte_container_with_subtype's comparison operators, several orphaned
type aliases), each with a compiled and output-verified example, plus missing
@sa comments and stale/incorrect Version History entries on several existing
pages (found by diffing consecutive release pairs and checking whether the
resulting change was actually reflected in the target page's history section).
Also adds docs/home/api_changes.md, a per-release, per-function reference of
public API changes (v3.1.0 through v3.12.0) generated from the history/
snapshots, complementing (not replacing) the existing release notes.
Along the way, found and fixed several extractor bugs by testing against real
release tags rather than trusting the algorithm in isolation -- most notably an
identity-key scheme based on libclang's USR that encoded the enclosing class
template's own arity, and a since-renamed ABI inline-namespace pattern
(json_v3_11_0 vs. today's json_abi_v3_11_2) that neither of two earlier regex
attempts stripped correctly. Both are documented in extract_api.py's docstrings
and tools/api_checker/history/README.md so the failure mode doesn't recur
silently.
.github/workflows/check_api_docs.yml runs extract_api.py + check_docs.py in CI,
advisory-only for now (documented backlog may not be at zero for entities this
PR didn't touch), plus a blocking drift check on the committed
tools/api_checker/api_surface.json.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-07-11 18:59:00 +02:00
174 changed files with 80905 additions and 1277 deletions
:books: If you want to **learn more** about how to use the library, check out the rest of the [**README**](#examples), have a look at [**code examples**](https://github.com/nlohmann/json/tree/develop/docs/mkdocs/docs/examples), or browse through the [**help pages**](https://json.nlohmann.me).
:construction: If you want to understand the **API** better, check out the [**API Reference**](https://json.nlohmann.me/api/basic_json/) or have a look at the [quick reference](#quick-reference) below.
:construction: If you want to understand the **API** better, check out the [**API Reference**](https://json.nlohmann.me/api/basic_json/) or have a look at the [quick reference](#quick-reference) below. The public API surface is derived mechanically and checked for documentation coverage by the tooling in [`tools/api_checker/`](tools/api_checker/), whose [POLICY.md](tools/api_checker/POLICY.md) defines what counts as public API and what stability is guaranteed.
:bug: If you found a **bug**, please check the [**FAQ**](https://json.nlohmann.me/home/faq/) if it is a known issue or the result of a design decision. Please also have a look at the [**issue list**](https://github.com/nlohmann/json/issues) before you [**create a new issue**](https://github.com/nlohmann/json/issues/new/choose). Please provide as much information as possible to help us understand and reproduce your issue.
| `too short` | `image` is `#!cpp nullptr`, or `size` is smaller than the 64-byte header |
| `unknown format` | the header's magic bytes or version do not match, or a reserved header field is not zero |
| `sizes out of range` | the node count, text size, or decoded-string size the header describes does not fit `size`, or the `#!cpp '\0'` after the text or after the decoded strings is missing |
| `the check failed` | `check` is not `#!cpp image_check::none`, and the image fails it -- see [`image_check`](#image_check) |
!!! failure "Example messages"
```
[json.exception.parse_error.116] parse error: invalid json_document image: too short
```
```
[json.exception.parse_error.116] parse error: invalid json_document image: unknown format
```
```
[json.exception.parse_error.116] parse error: invalid json_document image: sizes out of range
```
```
[json.exception.parse_error.116] parse error: invalid json_document image: the check failed
```
## Complexity
Linear in the number of nodes, which are always copied into the document. `#!cpp image_check::none` does not read
the text or the decoded strings. `#!cpp image_check::bounds` reads one byte of the text for each float token (its sign,
to check the recorded digits against the token's length), and nothing else of the text or the decoded strings.
With `#!cpp image_check::full`, linear in the size of the image (the nodes, the text, and the decoded strings), plus
sorting the ranges of the strings and of the float tokens (at most one per node): the contents of each distinct range
are checked once, however many nodes refer to it.
## Notes
### `image_check` { #image_check }
```cpp
using image_check = detail::view::image_check;
enum class image_check
{
full,
bounds,
none
};
```
How thoroughly `load()` validates `image` before trusting it.
`full` (the default)
: Checks everything the parser itself guarantees: structure and bounds; that every string is valid UTF-8 (and, for a
string still in the source text, that it contains no quote, backslash, or control character); and that every number
token is well-formed and matches the value stored for it. Strings and float tokens may share a range only if the
ranges are identical (as nodes that share a value do), and never overlap otherwise. Reading and serializing a
checked image is safe and always produces valid JSON, exactly as for a parsed document. This is the right choice for
an image from anything you do not fully control -- a file, a cache shared with other processes, a peer on the
network.
`bounds`
: Checks structure and bounds only: that every offset and count in the node index stays inside the image. So
`load()`, [`root()`](root.md), element access, and [`materialize()`](../basic_json_view/materialize.md) can never
read outside the image, even on a damaged one. It does *not* guarantee that the image describes valid JSON: a
string that a `full` check would have rejected can make [`dump()`](../basic_json_view/dump.md) write invalid UTF-8
or invalid JSON, or throw [`type_error.316`](../../home/exceptions.md#jsonexceptiontype_error316), and a number can
read back with a value that does not match how it is spelled. An [editable document](../json_editable_document.md)
does not take such a string over: [`set`](set.md), [`push_back`](push_back.md) and [`insert`](insert.md) throw
`type_error.316` when they copy it from a view of the loaded document, and leave the editable document unchanged.
`bounds` skips scanning the text and the decoded strings, so it fits a cache your own process just wrote and reads
straight back, where damage would mean a bug or a hardware fault rather than adversarial input.
`none`
: Checks nothing. Reading a damaged image is undefined behavior, so use it only for an image you trust as much as
your own memory.
**Lifetime.** Overloads 1 and 2 borrow `image`: it must stay alive and byte-for-byte unchanged for as long as the
returned document, and any [view](../basic_json_view/index.md) taken from it, is used -- exactly like a document
[`parse()`](parse.md) borrowed its input for. Overload 3 avoids this by keeping the vector itself; see
[`owns_source`](owns_source.md).
!!! warning "Experimental"
The image format is versioned but not yet stable, and may change in an incompatible way before it is declared
stable; `load()` already rejects an image written by a different format version with `parse_error.116`
("unknown format"). Use images to cache a document within one build of the library, or to hand one to another
process running the *same* build on the *same* (little-endian) machine -- not as a long-term storage format.
## Examples
??? example "Caching a document, ownership, and a rejected image"
The example below saves a parsed document as an image, checks that `load()` reproduces the original
[`dump()`](../basic_json_view/dump.md) without parsing, and shows the difference between
`load(std::move(image))` (owned) and `load(image)` (borrowed). It then damages one byte of the image and shows
`image_check::full` rejecting it with `parse_error.116`, while `image_check::bounds` -- meant for a cache the
process already trusts -- still reads it without going out of bounds.
@@ -320,6 +371,7 @@ the index is described in the [architecture overview](../home/architecture.md#no
| **Mutability** | freely mutable | not applicable (a one-shot event stream) | read-only | [`set`](../api/basic_json_document/set.md)/[`push_back`](../api/basic_json_document/push_back.md)/[`insert`](../api/basic_json_document/insert.md)/[`erase`](../api/basic_json_document/erase.md) edit in place; the source text is never rewritten |
| **What you get** | a full tree you can read, write, and keep as long as you like | a sequence of callbacks; whatever your handler builds from them | a flat index plus, on demand, [`materialize()`](../api/basic_json_view/materialize.md)d `json`/`ordered_json` values for the parts you actually use | the same, plus [`dump()`](../api/basic_json_view/dump.md) of an edited document that keeps the member order and, with [`number_format::source`](../api/basic_json_view/number_format.md), the spelling of every untouched number |
| **Typical use** | general-purpose JSON handling: config, request/response bodies you build or modify, anything you hold onto | validating or projecting a text into your own data structure without ever holding the whole thing as JSON | large or high-volume input where you only need part of it, or need it repeatedly, and can keep the source text (or a copy) alive for as long as the document lives | a document you read, patch a few fields of, and write back -- a configuration file, for instance -- where the rest of it should come back exactly as it was |
| **Caching/reload** | not applicable -- re-parse, or roll your own serialization | not applicable | [`save()`](../api/basic_json_document/save.md)/[`load()`](../api/basic_json_document/load.md): cache the parsed index as an image and reload it without parsing | same, saving the document's current -- possibly edited -- state |
@@ -51,16 +51,16 @@ If you do want to preserve the **insertion order**, you can use the type [`nlohm
--8<-- "examples/ordered_json.output"
```
Alternatively, [`nlohmann::fifo_map`](https://github.com/nlohmann/fifo_map) also preserves the insertion order and, unlike [`ordered_map`](../api/ordered_map.md), keeps a lookup index, so it does not have the quadratic cost described below. It is used through a small adapter ([integration](https://github.com/nlohmann/json/issues/485#issuecomment-333652309)).
Alternatively, [`nlohmann::fifo_map`](https://github.com/nlohmann/fifo_map) also preserves the insertion order and, unlike [`ordered_map`](../api/ordered_map/index.md), keeps a lookup index, so it does not have the quadratic cost described below. It is used through a small adapter ([integration](https://github.com/nlohmann/json/issues/485#issuecomment-333652309)).
If the order does not matter and you only want faster lookup, `boost::unordered_flat_map`, `absl::flat_hash_map`, `absl::node_hash_map`, and several other hash maps work through an adapter that restores the template argument order `basic_json` expects; see [Template Parameter Requirements](types/template_parameters.md#objecttype). Note these are *unordered*, not insertion-ordered.
[`tsl::ordered_map`](https://github.com/Tessil/ordered-map) cannot be used: its iterators expose the mapped value as `const`, while `basic_json` needs to modify it in place.
The [`ordered_map`](../api/ordered_map.md) behind `nlohmann::ordered_json` is deliberately minimal and has no lookup
The [`ordered_map`](../api/ordered_map/index.md) behind `nlohmann::ordered_json` is deliberately minimal and has no lookup
index, so every key access is a linear scan and building an object of `n` keys costs O(n²). This is unnoticeable at
typical object sizes but becomes significant for objects with many thousands of keys; see
[`ordered_map` complexity](../api/ordered_map.md#complexity). The alternatives above keep a lookup index and do not
[`ordered_map` complexity](../api/ordered_map/index.md#complexity). The alternatives above keep a lookup index and do not
- Since version 3.9.0, integer numbers beyond int64 are serialized as high-precision UBJSON numbers.
- Since version 3.12.0, integer numbers beyond int64 are serialized as uint64 BSON numbers.
!!! failure "Example message"
Before version 3.13.0, CBOR, MessagePack, and BSON silently truncated integers wider than 64 bits, and BJData
truncated unsigned integers wider than 64 bits.
!!! failure "Example messages"
```
integer number 18446744073709551615 cannot be represented by BON8 as it does not fit int64
```
```
integer number 1267650600228229401496703205376 cannot be represented by CBOR as it does not fit [-2^64, 2^64-1]
```
### json.exception.out_of_range.408
@@ -1088,7 +1141,9 @@ MessagePack's ext type and BSON's binary subtype are each stored in a single byt
so they do not support an input of 4294967280 bytes (4 GiB minus 16 bytes) or more. The same 32-bit limit applies to an **editable** document's own
storage: [`set`](../api/basic_json_document/set.md) and [`push_back`](../api/basic_json_document/push_back.md) throw
this exception once the strings and number tokens written by edits reach 4 GiB in total, or once more than
4294967295 arrays/objects have had an element set or appended to them.
4294967295 arrays/objects have had an element set or appended to them. The same limit applies to an
[image](../features/json_view.md#images): [`save()`](../api/basic_json_document/save.md) throws it if the node
count, the text, or the decoded strings it would write would individually reach 4 GiB.
!!! failure "Example messages"
@@ -1098,6 +1153,9 @@ this exception once the strings and number tokens written by edits reach 4 GiB i
```
[json.exception.out_of_range.416] edits of 4 GiB or more are not supported by json_document
```
```
[json.exception.out_of_range.416] images of 4 GiB or more are not supported by json_document
```
!!! note
Loaded 100 of 174 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Niels Lohmann
dependabot[bot]
