Holds errored, allow_exceptions, parse_error() and is_errored(). The scalar
handlers stay in each class (parse hot path).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
get_bon8_string takes the valid range of the second byte from utf8_shape()
(0xF5..0xF7 stay errors, unget behavior unchanged). validate_one_utf8 keeps its
if-chain (hot bulk path); the new test checks decode, validate_one_utf8,
utf8_shape, json::accept and the BON8 reader against each other for every lead
and second byte.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
- detail::exception_id at every numeric id; throw_* helpers take it
- hex_byte, hex4_lower, utf8_shape replace local hex/table code
- throw_array_index_out_of_range/_past_end, throw_key_not_found
- has_native_float_format replaces the hand-written IEEE check
No change of messages, ids or output.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The three number cases of get_arithmetic_value and the arithmetic
from_json share one helper (boolean stays accepted only by the latter);
the array and object reserve overload pairs become one.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Object and array stay the first branches; everything else is the
primitive case. operator== and operator< share the different-container
check.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The leading-slash and invalid '~' escape checks are shared helpers; the
reference-token walks test object and array first and treat everything
else as primitive.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The four create() bodies build their message through one protected
static helper; the diagnostics parent walk handles array/object and
ignores all other types.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
1.1: build ascii_copyable() at compile time from is_ascii_copyable and
test find_ascii_copyable_run against it for every byte. 1.2/2.5.4: use
detail::hex4_lower in both \uXXXX writers. 2.1.6: factor the repeated
flush/emit blocks in dump_escaped_impl into always-inline helpers.
2.1.7: comment why the escape copy does not use put_buffer.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The CBOR/MessagePack/UBJSON/BSON/BJData/BON8 literals in throw_on_discarded and
throw_integer_out_of_range calls now come from detail::input_format_name; the
texts are identical, and UBJSON vs BJData still follows use_bjdata.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
write_cbor_key and write_msgpack_key were the same SFINAE pair apart from the
string/scalar writers; both now delegate to write_object_key<Walk>. UBJSON key
and to_format_length stay as they are.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The 13 oa.write_characters(reinterpret_cast<const CharType*>(x.data()), n)
sites now call write_bytes(const void*, size_t); copy_characters sites and the
buffer-based writes are untouched.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
write_ubjson_start_array/_object are now write_ubjson_start_container<IsArray>,
keeping the array-only begin()+1 scan and excessive_valueless rule.
ubjson_frame derives from binary_container_frame instead of duplicating it.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
write_cbor_iterative and write_msgpack_iterative (and their *_value_or_push
helpers) were identical but for the head, scalar, and key writers. They are now
one template over a small policy struct, so everything inlines. BON8 keeps its
own walker (open string run, closing marker), noted in a comment.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Array and object prefixes differed only in the fix marker and the 16/32-bit
markers; both now delegate to write_msgpack_container_prefix. Output is
byte-identical.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
write_bson_entry_header and write_bson_string re-derived the error-handler
policy differently. Both now use sanitize_utf8_for_bson_write, which treats
strict like keep (the size pass already validated) and sanitizes replace/ignore
into the bytes the size pass measured. No context plumbing is needed.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Add detail::is_primitive(value_t), detail::hex4_lower(), detail::utf8_shape(),
detail::input_format_name(), detail::empty_input_message(), the exception ids
116, 319, 320 and 416 to detail::exception_id, and make
detail::is_ascii_copyable() constexpr. Later commits switch the duplicated
code over to them.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
- generate_docset.py: use SHA-256 instead of SHA-1 for the cache file
name digest of downloaded images.
- generate_docset.py: suppress dynamic-urllib-use-detected; the URL
scheme is restricted to http(s) right before the call.
- generate_natvis.py: suppress direct-use-of-jinja2; the template
renders a local XML file with autoescape enabled.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
- unit-stl_helpers.cpp: with a std::array argument, an unqualified
bit_cast<...>(bytes) also finds std::bit_cast via ADL in C++20 and
MSVC 2019 rejects the call as ambiguous (AppVeyor). Qualify the call.
- comment_check_amalgamation.yml: skip, rather than crash, when the
triggering run has no "pr" artifact (stale queued runs that complete
as failed without ever running a job).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Compare json_view with yyjson, simdjson, and Boost.JSON
tests/benchmarks/json_view/ holds a comparison with other
libraries, answering the question users ask when they pick one.
It is not built by CMake and not run by CI.
- bench_view.cpp: parse, traverse, select and dump of twitter,
citm_catalog, canada, jeopardy, a tweet and an RPC request,
against json_view, yyjson, simdjson, Boost.JSON and json::parse,
all agreeing and run interleaved.
- bench_corpus.cpp: parse, traverse and dump of any file list.
- bench_edit.cpp: parses, edits through each library's own API,
and serializes; all outputs must describe the same value.
- compare.py: builds both programs with system or pinned,
SHA-256-checked downloads, and writes results with the commit,
CPU, OS, compiler and library versions.
- README.md and a workflow that runs on demand or via a PR label.
Fairness fixes folded in: each engine runs once untimed before
each timed round, so the next one no longer pays for the
previous one's cleanup; dump is also compared with source numbers
against yyjson's raw-number mode; a "simdjson DOM (fresh)" column
and a reused-document column for json_view were added;
compare.py's paths are absolute; downloads use a .part file and a
failed checksum removes the archive.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Make the downloads of json_view's compare.py robust
Hash the archives in chunks instead of reading them into memory, download
with a timeout, extract into a temporary directory that is renamed into
place only after success (a half-extracted directory was trusted forever),
and split CXX and CC into arguments so that values like 'ccache g++' work.
Note at the pins that a SHA-256 mismatch of a GitHub tag archive means
that GitHub regenerated it.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Add a timeout and a concurrency group to the json_view benchmarks
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Check the CSV file and the round count in the json_view benchmarks
fopen() of the CSV output was not checked, so an unwritable directory made
fprintf() write to a null FILE*; the round count is parsed with strtol and
clamped instead of atoi.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
#5876 added JSON_LITTLE_ENDIAN_TARGET and JSON_BIG_ENDIAN_TARGET to
macro_scope.hpp. They must stay macros because they select code in #if,
so suppress the clang-tidy check as simd_scan.hpp already does.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Since #5877, the checked-in nlohmann_json.natvis only covers namespaces
with up to two ABI tags. The generator gets options to produce a file
for other configurations:
- --abi-tags TAGS adds the namespaces of one configuration (repeatable)
- --all generates every combination of ABI tags
- --list-tags prints the tags with the macros that enable them
- --output FILE writes somewhere else than the repository root
`make natvis NATVIS_ARGS="..."` passes the options through. The default
output is unchanged. The options are documented in the generator's
README, the debugging page, and FILES.md, and checked in CI.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Flatten into a std::map by inserting next to the previous entry
flatten() added every path with operator[], so each insertion into the
std::map of the result searched the tree from the root and compared long
shared path prefixes, which took half the time on canada.json. Paths come
out in nearly sorted order, so they are now inserted with emplace_hint
next to the entry added last, for object types that have it. ordered_map
keeps its append without lookup, and other object types keep
operator[].
Path segments are also appended in place, without a temporary string
for each escaped key or array index.
flatten() of canada.json, citm_catalog.json and twitter.json is 1.4x to
1.9x as fast; ordered_json is unchanged (1.05x). The output is
unchanged.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Unflatten without splitting keys or walking each path from the root
unflatten() split every key into a vector of reference tokens, stored the
pointer prefixes in a tree to decide which values become arrays, and then
looked up every token of every key from the root again.
Keys are now checked by a scan that raises the same parse_error.107/108 as
split(), skipping the part a key shares with the previous one, which was
checked already. The prefixes that denote arrays are kept in a small hash
set of key slices; the array/object decision is unchanged. The walk keeps
the values along the previous key's path, so only the part of a key that
differs from the previous one is looked up. Keys of a flattened object are
usually sorted, so that part is short.
unflatten() of canada.json, citm_catalog.json and twitter.json is 1.9x to
2.3x as fast; the output and the exceptions are unchanged.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Apply plain JSON Patch operations without building a json_pointer
patch() built a json_pointer for every operation, which splits the path
into a vector of strings, and for "add" and "remove" a second one for
the parent. That took 20% to 45% of the time not spent copying the
document.
"add", "remove", "replace" and "test" with a path that has no escapes and
denotes existing values (the parent, for "add"), as diff() creates them,
are now applied by resolving the path in place, reusing one string for
the reference tokens. Every other operation, and every operation that
would fail, takes the existing code, so the results, the exceptions and
the state left after a failing operation are unchanged.
patch() of the diffs of canada.json, citm_catalog.json and twitter.json
is 1.02x to 1.10x as fast; the rest is the copy of the document.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Build diff operations member by member
diff() created every operation from an initializer list, so each
{"op", "replace"} pair became a temporary array with two heap strings
before it was turned into an object member; an operation took about 25
allocations. The operations are now built as objects whose members are
assigned one by one. Paths of members and elements are also built in
place, without a temporary string for each escaped key or array index.
diff() of canada.json, citm_catalog.json and twitter.json, with every
eighth scalar changed, is 1.3x to 1.6x as fast; the output is unchanged.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Diff objects with the same keys in lockstep and keep one path buffer
diff() looked up every key of the source object in the target object,
and built a new string for the path of every value it descended into.
Objects of the same size usually have the same keys in the same order,
so they are now walked together as long as that holds, without a
lookup; objects that differ fall back to the lookup from the first key
that does not match. The path is one buffer that is extended before
descending and shortened again afterwards. Objects whose members are
equal but in a different order (possible with ordered_json) still get
no operations.
Together with the previous commit, diff() of canada.json,
citm_catalog.json and twitter.json is 1.5x to 2.0x as fast; ordered_json
1.65x to 1.85x. The output is unchanged.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Use std::array for the index buffer in append_index()
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Declare the partner iterator in the loop initializer in diff_object_keys()
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix GCC and clang-tidy warnings in the JSON pointer speedups
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Use standard library helpers where available, with C++11 fallbacks
Add detail helpers that call the standard library function when the
library provides it (checked with the __cpp_lib_* feature-test macros, so
no new ABI-tag macros) and a C++11 implementation otherwise:
- detail::clamp (std::clamp): by value, so that static constexpr members
need no out-of-line definition before C++17; used in json_view's dump()
buffer sizing, where step() can now be a static constexpr member.
- detail::bit_cast (std::bit_cast): replaces dtoa_impl::reinterpret_bits
and the memcpy bit copies in the number parser, to_chars, the json_view
serializer, and binary_writer::to_char_type.
- detail::byteswap (std::byteswap): one implementation instead of the
binary writer's own byte_swap() overloads and the bswap chain in
to_chars' store_msb_first().
- count_leading_zeros/count_trailing_zeros: two 32-bit _BitScanReverse/
_BitScanForward calls for MSVC x86/ARM32 instead of the loop, and
std::countl_zero/countr_zero for compilers without builtins.
- conjunction, disjunction, negation, bool_constant, and uncvref_t map to
their std counterparts in C++17/C++20.
Builtins and intrinsics stay ahead of the <bit> functions, as MSVC's
std::countl_zero checks for LZCNT at run time unless AVX2 is enabled.
The generated code is unchanged with Clang (C++11/20/23, arm64 and
x86-64) and GCC 16 for parsing, dumping, and the binary formats. The
binary reader keeps its own byte_swap() and the writer its memcpy calls:
the smaller code from detail::byteswap/std::bit_cast there made GCC 16
inline get_number()/write_number() differently, which slowed reading by
10-18 % and writing by about 1 %.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Name the target byte order once (JSON_LITTLE_ENDIAN_TARGET, JSON_BIG_ENDIAN_TARGET)
to_chars.hpp and bit_ops.hpp repeated the long __BYTE_ORDER__/_MSC_VER
conditions; macro_scope.hpp now defines them once (undefined again in
macro_unscope.hpp).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Explain why the serializer growth bound is not a clamp
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
With ensure_ascii, the view's dump wrote every string through the escaping
path, which decodes UTF-8 byte by byte, even strings of the source that are
plain ASCII. A string of the source without escape sequences has no quote,
backslash, or control character, so ensure_ascii leaves it unchanged
exactly when it has no byte >= 0x7F. The dump now checks that eight bytes
at a time (one add and one or per word) and copies such strings like the
dump without ensure_ascii.
The compact walk (dump_compact) now also serves ensure_ascii, through a
separate out-of-line instantiation, so the walk without ensure_ascii keeps
its speed.
Parsing is unchanged. A flag set by the parser would let the dump skip the
check and was about twice as fast for compact output, but it costs every
parse of string-heavy input 1-2% with GCC.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
dump() now checks strings of up to 32 bytes against a table of the bytes
that can be written verbatim, four bytes per step, and copies them in one
go if all of them can. That is cheaper than setting up the vector scanner,
whose setup costs more than such a string takes to scan; SQLite's
jsonAppendString() writes strings with the same kind of table loop. Most
object keys and many values are this short (citm_catalog averages 8 bytes
per string, jeopardy 14).
GCC 16 and 13 stopped inlining put_indent() and put_literal() into the
pretty-printing loops once dump_escaped() had grown, which made indented
output up to 13% slower; both are now always inlined.
The new test puts every ASCII character and a 2-byte character at every
position of strings of 1 to 40 bytes and compares the output with the
reference implementation of the existing UTF-8 tests.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
json_sax_dom_parser::key() copied the key from the lexer's buffer into the
object. The lexer starts every token with an empty buffer (reset()), so the
key can be moved instead, which saves an allocation and a copy for every key
that does not fit into the string's inline buffer. json_sax_dom_parser
already moved string values the same way. ordered_map has no rvalue
operator[], so ordered_json still copies keys.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Niels Lohmann
dependabot[bot]