JSON text must be valid UTF-8, and to_bon8 rejects ill-formed strings by design (type_error.316), as documented on the BON8 page. Restore the original cell and realign the table.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Restore the correct statement that marker bytes cannot begin a UTF-8 character (0x80..0xBF occur in valid UTF-8 as continuation bytes), and reword two sentences.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Verified each claim with small C++ probes before writing it.
- index.md: BON8 serialization is also incomplete for invalid UTF-8.
- bon8.md: document duplicate keys (last wins), the error ids for
malformed input incl. 0xF5..0xF7 + 0x80..0xBF, exact string
termination wording, wide integers and discarded values in the
unsupported list, and title two untitled admonitions.
- exceptions.md: to_bon8 in type_error.316, BON8 in parse_error.112
(with example) and .113, wide integers in out_of_range.407.
- binary_values.md: bytes 0..39 take one byte in BON8.
- from_bon8.md: note on strict UTF-8 and duplicate keys.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Reformat the tables changed by this PR (BJData serialization and
deserialization mappings, completeness and binary-value tables) so all
rows are aligned, and add the Oxford comma to lists of three or more
items in the added text.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The $ type restriction exists since Draft 2 (Draft 3 added B), $ requires a count, ND char above 127 is rejected on read, and the new API page is listed in alphabetical order in the nav.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Verify the BJData chapter against the library and the Draft 2/3/4
specifications and fix what did not match:
- bjdata.md: correct the list of unused markers (N, not Z), the ND-array
example (3x2, not 2x3), the conditions under which an object is packed
as an ND-array (exactly three keys, row vectors, float-only elements for
single/double, char 0..127, byte only in Draft 3), the binary table row
and notes (which flag/draft combinations write [$B), the "$" type
restriction (Draft 3, Z is excluded too), the H marker in the
deserialization table, and the claim of a complete deserialization
mapping. Document the supported drafts, that Draft 4 (SoA containers,
extension type E) and column-major ND-arrays are rejected, ND-arrays of
half floats, duplicate object keys, and link the Draft 3 specifications
instead of master.
- Add an API page for basic_json::bjdata_version_t with an example, and
link it from the nav, the API index and to_bjdata.
- Update binary_formats/index.md and binary_values.md, which still claimed
that BJData has no binary type, and fix a wrong byte comment there.
- Document parse_error.115 and the out_of_range.408 cases for from_bjdata,
the precise other_error.502 condition for to_bjdata, and reword the note
on truncated unsigned integers in exceptions.md (only ND-array elements
were truncated before 3.13.0).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
BJData N-dimensional arrays with element type 'h' (float16) failed to
read with a misleading parse_error.112, because the reader's dtype name
table lacked 'h'. It now maps to "half", the JData name. The writer still
does not pack "half"; such an object is written as a plain object and
reads back equal.
BJData limits 'C' (char) to 0..127 in Draft 2 and Draft 3, and the 1-D
reader enforces this, but ND-arrays with _ArrayType_ "char" wrote values
up to 255 as 'C' elements and read them back without a check. The writer
now only packs values in 0..127 (others fall back to a plain object that
round-trips), and the reader rejects 'C' elements above 127 with
parse_error.113 like the 1-D reader.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Add the serial comma in a list added by this PR and remove the BSON
example of type_error.321 (with the type_error.317 remark) so the docs do
not mention discarded values for BSON; restore the original singular
heading of that section. No tables were changed by this PR.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Describe exactly when a too small document size is detected and note that subtype 2 data written by earlier versions is not readable.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Describe the inner length of binary subtype 0x02 that to_bson now writes and
from_bson checks and removes. Add the document size checks (parse_error.112,
not affected by strict=false) to bson.md and from_bson.md, and extend the
list of lenient reads with the UUID and MD5 payload lengths (not checked),
duplicate keys (last value wins), embedded NUL bytes in strings, and the
ignored trailing bytes with strict=false. State that the members of an object
are written in iteration order.
Correct the statement that element names are never validated on read: they
are checked and sanitized like string values by the error handler. Add the
BSON message to the examples of out_of_range.415 and type_error.321, the
subtype limit of 255 to the binary values page, CBOR to the formats that use
byte_container_with_subtype, and fix a typo in the incomplete mapping box.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The BSON reader compared a document's declared size with the number of bytes
read only when it reached the terminator. The declared size never bounded
the reads, so a SAX handler saw the keys and values of a document whose size
was too small before parse error 112 was reported.
The container frame now stores the position where the document ends
according to its declared size. open_bson_document rejects a size below 5
(size prefix plus terminator) and an embedded document that does not end
before the terminator of its enclosing document; the element loop rejects an
element whose type byte leaves no room for the name terminator and the
document terminator. The terminator check stays and still catches a declared
size that is too large. No input that was accepted before is rejected now;
documents with a wrong size are reported earlier, with fewer SAX events, a
different byte position and, in three cases, a different message. The error
id (parse_error.112) is unchanged and strict=false still does not skip the
check.
The tests that asserted the old messages and positions are updated, with the
outer sizes of two inputs corrected so that they still exercise the embedded
document. New tests cover the SAX events, a size below 5, and an embedded
document that fits exactly.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The BSON specification defines the payload of the "binary (old)" subtype
0x02 as an int32 followed by that many bytes. to_bson wrote the plain
payload instead, so decoders such as pymongo and libbson-based readers
rejected the bytes, and from_bson returned the inner length as part of the
payload (documented as lenient behavior in gh-5333).
The writer now emits the outer length (payload + 4), the subtype, the inner
int32 holding the payload size, and the payload. The reader requires an
outer length of at least 4 and an inner length equal to the outer length
minus 4 (the cases the bson-corpus lists as decode errors), reports parse
error 112 otherwise, and returns only the payload. Both directions change
together so that the round trip keeps working.
The single-reservation fast path for scalar entries declines subtype 0x02
because it has no room for the inner length.
This is a deliberate exception to the opt-in rule for behavior changes:
the previous output was invalid BSON.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Reformat the two UBJSON mapping tables that this PR changed (uniform column
widths, padded cells), use the serial comma in lists of three or more in the
added text, and restore the original type_error.321 bullet in to_ubjson.md.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Clarify that empty containers carry no $ annotation of their own and note that a counted N object is not supported.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Fix statements that disagree with the behavior (checked with probes):
- `Z` is null; the unused marker is `N`.
- `$` requires `#`; use_type with use_size=false throws other_error.502 only
for non-empty arrays, objects and binary values.
- NaN and infinity are written as IEEE 754 float64 and read back unchanged;
the UBJSON specification encodes infinity as null.
- `H` is documented (semantics, parse_error.115, out_of_range.406); the
"complete mapping" claim is qualified.
- use_type "same type" means same marker.
- Long Z/T/F arrays lose only the `$` annotation; the `#` count and the
element markers are kept.
- from_ubjson lists the 112, 113, 115, 406 and 408 causes that occur for
UBJSON, without the BJData-only ND-array wording.
- Duplicate object keys keep the last value; the `$` type marker is
validated even for count 0.
- Smaller fixes: float32 row, "map" -> "object", https link, exception 110
strict case, to_ubjson history of discarded values.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The '$' type marker of an optimized container was only checked when an
element was read, so containers with count 0 accepted any marker:
`[$x#i0`, `{$x#i0` and `[$E#i0` parsed as empty containers although a
non-empty container with the same marker is rejected and the UBJSON
specification requires a legal marker after '$'.
get_ubjson_size_type() now checks the marker right after '$' against the
allowed set (UBJSON: Z N T F i U I l L d D H C S [ {; BJData: B U i u I m
l M L h d D C, with the existing exclusion of T F Z N S H [ {) and reports
parse_error.112 "marker ... is not a permitted optimized array type".
Input that was accepted and is now rejected: an optimized container with
count 0 and an invalid '$' type marker. Valid input is unaffected. Two
existing tests that used an invalid marker incidentally ("[$#" and a
negative-count case) were adapted.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Name parse_error.113 for a 0xC1 map key, move the duplicate-key sentence out of the ext section, and keep the 0xC1 row in marker order.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Add an "Extension types (ext, fixext)" section to the MessagePack page:
ext/fixext are read as binary, the signed ext type is exposed as an
unsigned subtype 0..255 (-1 reads as 255), Timestamp is not interpreted,
subtypes are always written as ext/fixext, and subtypes above 255 throw
out_of_range.415. Add the 0xC1 row to the deserialization table, note
that duplicate keys keep the last value, and correct the NaN/Infinity
explanation, the "31 bytes / 15 elements" wording, the ill-formed UTF-8
sentence, and the generic 0xC1-key error message.
Add fixext16 and the unsigned subtype mapping to binary_t.md and
binary_values.md, reword "maximum of the MessagePack ext type" in
to_msgpack.md and exceptions.md, and mark MessagePack deserialization
as incomplete (non-string keys) in the binary formats overview.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Remove the mention that a discarded value cannot be serialized to CBOR
(type_error.321) from the exception-safety and mapping sections. The
integer table and all added prose already follow the table-format and
Oxford-comma rules.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Cite RFC 8949 Section 5.3.1 for duplicate keys, describe the out_of_range.408 condition precisely, rewrap a long line, and update the stale get_string comment (per-chunk UTF-8 check). single_include regenerated.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Fix the negative-integer rows (they used MessagePack boundaries), the
number_integer range of the 0x1B row, and the half-precision entry in the
list of unused types; correct the NaN history (0xF9 since 3.8.0) and the
tagged-item wording (only the tag directly in front of a byte string is
stored, and only in the 0xD8..0xDB form). Document that to_cbor is not
RFC 8949 deterministic, that duplicate keys are accepted (last wins),
tags in front of keys, per-chunk UTF-8 validation, the exceptions
parse_error.113, out_of_range.406/408 of from_cbor, the pointer and
C array input traps, the partially written output on exceptions, and the
store/ignore results in the binary values example. Drop the claim that
the CBOR reader needs StringType::append(const StringType&) and
BinaryType::insert(pos, first, last).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
A tag in front of a text string map key was rejected with parse_error.113
even with cbor_tag_handler_t::ignore or store, although tags in front of
values are handled. The tags are now skipped iteratively before the key is
read; the handler error still rejects them.
The chunks of an indefinite-length text string were concatenated before
the single UTF-8 check, so a code point split over two chunks, or an
invalid chunk, was accepted. RFC 8949, Section 3.2.3 requires every chunk
to be valid on its own. With error_handler_t::strict, replace and ignore
each chunk is now checked; keep is unchanged. The chunks are appended with
append(ptr, len), so StringType needs nothing new.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
- Name std::hash only once the object type has emplace_new_key: VS 2015
static-asserts on std::hash of a type without one (alt_string) instead of
reporting a substitution failure (C2338).
- Replace the std::copy calls on frames by loops: VS 2015 reports raw
pointers handed to std::copy as C4996, an error under /WX.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: byte layout sections, glossary, and QA page updates
- Move "Byte layout" of each binary format to its own section before
"Serialization" and describe the idea of the format.
- Glossary: add LLDB and other terms used throughout the documentation.
- quality_assurance: correct Clang Static Analyzer and CMake entries; add
CodeQL, Semgrep, Flawfinder, CIFuzz, OpenSSF Scorecard, REUSE, Meson/SwiftPM,
and reproducibility checks.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: drop unused length colour from CBOR and MessagePack byte layout legends
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Document clang-tidy exclusions and drop unneeded ones
Re-enable every excluded check in a run of clang-tidy 22 over all tests
and remove the seven that never fire (cert-err58-cpp,
clang-analyzer-webkit.NoUncountedMemberChecker,
cppcoreguidelines-virtual-class-destructor, google-readability-function-size,
hicpp-function-size, hicpp-no-assembler, readability-function-size).
Document the reason for every remaining exclusion at the top of the file.
Exclude in advance the checks that only exist in clang-tidy 23 and are
noise here (bugprone-signed-bitwise, *-explicit-constructor,
readability-trailing-comma, modernize-use-string-view), plus
modernize-use-integer-sign-comparison, which needs C++20 std::cmp_less
and flags deliberate casts.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix clang-tidy findings in documentation examples and replace redundant aliases
Apply clang-tidy fixes to the documentation examples:
- performance-avoid-endl: std::endl -> '\n'
- readability-qualified-auto: auto -> auto* / const auto*
- performance-for-range-copy: copy -> const reference
- google-readability-casting: C-style cast -> static_cast
Replace redundant aliases (readability-redundant-qualified-alias) such as
"using json = nlohmann::json;" with "using nlohmann::json;" in the
documentation, README, examples, tests, and benchmarks. Aliases at class
scope and aliases of other types (with_base_class_t etc.) are kept.
No changes to the library or its public API. All changed examples were
compiled and run; their output matches the stored .output files (except
max_size and meta, which depend on the platform and compiler).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Format doc examples with the pinned astyle
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Docs: fill documentation gaps found in user questions
Add pages on class-hierarchy/smart-pointer/variant/chrono recipes, working
without exceptions, and large, streamed and partial input. Extend the FAQ
(23 entries, Known bugs first), parse error documentation (empty-input
checklist, line/column, diagnostic positions), element access, conversions,
creating values, ADL troubleshooting, memory, and build/distribution notes.
Add 14 examples with generated output. Correct the statement that parse errors
cannot be recovered from with -fno-exceptions.
The topics come from recurring questions on Stack Overflow, Reddit, Hacker
News, YouTube, and the project's GitHub discussions and issues.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: declare range-for lists in variables to keep astyle quiet
astyle reformats a braced init list placed directly in a range-for
header; declare the list first so check_amalgamation passes.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: cross-reference json_view and json_document for 3.13.0
Document the new zero-copy document/view API across the docs: add a
"Documents and views" section to the basic_json API hub linking to all
specializations, point parsing/conversion/modifying pages to the view-based
alternatives, add document/view sections to feature pages, update the
landing page to mention json_view.hpp/json_literals.hpp, and cross-link
debugging/exceptions.
Also document wide-string inputs (and ill-formed UTF-16/32 rejection),
add parse/accept examples for wide strings, fix related doc gaps, and
align features/parsing and FAQ with the added support.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: fix article before std::istream in FAQ
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: clarify document inputs, decoding, get<T>() and editing
Address review comments: say which inputs are borrowed and that a wide
string pointer is an extra input, which values are decoded during parsing,
that get<T>() reads only built-in types directly, all forms of set() and
that the document's insert() is array-only, and how to store wide strings.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: integers are converted while parsing a document
Also say "They accept every input" in basic_json_document::parse.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* from_json: number-first dispatch for arithmetic types, loop for std::array (#5891)
from_json(ArithmeticType&) now reads numbers through get_number_value() and
handles a boolean on a cold path, which puts the three number types first in
the per-element type dispatch.
get<std::array<T, N>>() always took the identity_tag overload, a pack
expansion of N calls j.at(i).get<T>() that clang unrolls into straight-line
code (about 28,000 instructions for N = 1000), so its speed depended on
inlining decisions in that body. For trivial element types, which cannot tell
default construction plus assignment from construction in place, it now fills
a std::array in a loop. That loop, shared with get_to(), checks the size once
and reads the inner array directly; a short array still converts the elements
it has and then throws the out_of_range error at() would throw.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Skip the get_to result check without exceptions
Without exceptions, the get_to calls in CHECK_THROWS_WITH_AS do not run,
so the array keeps its initial values (ci_test_noexceptions).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Make is_trivial_element a struct so astyle does not indent it
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Expect the element path in the std::array type-error tests with JSON_DIAGNOSTICS
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Shorten from_json_inplace_array so astyle aligns its parameters
The opening parenthesis of the previous name sat in column 41, beyond
astyle's default --max-continuation-indent of 40, so the continuation
lines fell back to a plain double indent.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Use the compact template<...> style for the new from_json overloads
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Add shared detail helpers for the #5891 cleanup
Add detail::is_primitive(value_t), detail::hex4_lower(), detail::utf8_shape(),
detail::input_format_name(), detail::empty_input_message(), the exception ids
116, 319, 320 and 416 to detail::exception_id, and make
detail::is_ascii_copyable() constexpr. Later commits switch the duplicated
code over to them.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* 1.4: BSON writers share one write-pass UTF-8 sanitizer (#5891)
write_bson_entry_header and write_bson_string re-derived the error-handler
policy differently. Both now use sanitize_utf8_for_bson_write, which treats
strict like keep (the size pass already validated) and sanitizes replace/ignore
into the bytes the size pass measured. No context plumbing is needed.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* 2.1.1: one MessagePack container prefix helper (#5891)
Array and object prefixes differed only in the fix marker and the 16/32-bit
markers; both now delegate to write_msgpack_container_prefix. Output is
byte-identical.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* 2.1.2: one iterative walker for CBOR and MessagePack (#5891)
write_cbor_iterative and write_msgpack_iterative (and their *_value_or_push
helpers) were identical but for the head, scalar, and key writers. They are now
one template over a small policy struct, so everything inlines. BON8 keeps its
own walker (open string run, closing marker), noted in a comment.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* 2.1.3, 2.1.8: one UBJSON container start helper, shared frame base (#5891)
write_ubjson_start_array/_object are now write_ubjson_start_container<IsArray>,
keeping the array-only begin()+1 scan and excessive_valueless rule.
ubjson_frame derives from binary_container_frame instead of duplicating it.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* 2.1.4: write_bytes helper for the byte-string write sites (#5891)
The 13 oa.write_characters(reinterpret_cast<const CharType*>(x.data()), n)
sites now call write_bytes(const void*, size_t); copy_characters sites and the
buffer-based writes are untouched.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* 2.1.5: shared object-key writer for CBOR and MessagePack (#5891)
write_cbor_key and write_msgpack_key were the same SFINAE pair apart from the
string/scalar writers; both now delegate to write_object_key<Walk>. UBJSON key
and to_format_length stay as they are.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* 2.3.10: writer-side format names from input_format_name (#5891)
The CBOR/MessagePack/UBJSON/BSON/BJData/BON8 literals in throw_on_discarded and
throw_integer_out_of_range calls now come from detail::input_format_name; the
texts are identical, and UBJSON vs BJData still follows use_bjdata.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* serializer: constexpr ascii table, hex4_lower, dedupe helpers (#5891)
1.1: build ascii_copyable() at compile time from is_ascii_copyable and
test find_ascii_copyable_run against it for every byte. 1.2/2.5.4: use
detail::hex4_lower in both \uXXXX writers. 2.1.6: factor the repeated
flush/emit blocks in dump_escaped_impl into always-inline helpers.
2.1.7: comment why the escape copy does not use put_buffer.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* 1.6: use string_t-aware to_string/concat in meta() (#5891)
Replaces the four std::to_string sites; rendered text unchanged.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* 1.7: use detail::is_primitive in json.hpp switches (#5891)
has_no_children, set_parents, max_size and the two diff walkers.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* 2.2.1: factor insert()/iterator throw helpers (#5891)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* 2.2.2: share swap() overloads via swap_impl (#5891)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* 2.2.4: factor primitive reset and range check in erase()/range ctor (#5891)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* 2.2.6: share push_back array bodies via push_back_array_impl (#5891)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* 2.2.7: route parse/accept/sax_parse through *_impl helpers (#5891)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* 2.2.8: share the vector-returning to_* bodies via to_binary_vector (#5891)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* 2.2.10: merge destroy_string/destroy_binary into destroy_pointer (#5891)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* 2.4.1/1.7: exceptions: share what_impl, flatten diagnostics switch (#5891)
The four create() bodies build their message through one protected
static helper; the diagnostics parent walk handles array/object and
ignores all other types.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* 2.4.2/2.4.3/1.7: json_pointer: share slash and escape checks, drop primitive case lists (#5891)
The leading-slash and invalid '~' escape checks are shared helpers; the
reference-token walks test object and array first and treat everything
else as primitive.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* 1.7/2.4.9: iter_impl, iteration_proxy: drop primitive case lists, share compare preamble (#5891)
Object and array stay the first branches; everything else is the
primitive case. operator== and operator< share the different-container
check.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* 2.4.5/2.4.6: from_json: share get_number_value and reserve_if_possible (#5891)
The three number cases of get_arithmetic_value and the arithmetic
from_json share one helper (boolean stays accepted only by the latter);
the array and object reserve overload pairs become one.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* 2.4.8: ordered_map: share erase_impl between the key erase overloads (#5891)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* 1.7: json_pointer: flatten walk drops the primitive case list (#5891)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* 1.2/1.3/1.5/2.5.1/2.5.2/2.5.5: share helpers in the view code (#5891)
- detail::exception_id at every numeric id; throw_* helpers take it
- hex_byte, hex4_lower, utf8_shape replace local hex/table code
- throw_array_index_out_of_range/_past_end, throw_key_not_found
- has_native_float_format replaces the hand-written IEEE check
No change of messages, ids or output.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* 1.5: use utf8_shape in get_bon8_string, add exhaustive UTF-8 validator test (#5891)
get_bon8_string takes the valid range of the second byte from utf8_shape()
(0xF5..0xF7 stay errors, unget behavior unchanged). validate_one_utf8 keeps its
if-chain (hot bulk path); the new test checks decode, validate_one_utf8,
utf8_shape, json::accept and the BON8 reader against each other for every lead
and second byte.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* 2.3.1: get_cbor_sized_bytes for the CBOR chunk getters (#5891)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* 2.3.2: get_string_with_length for the plain-order string sites (#5891)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* 2.3.3, 2.3.4, 2.3.10: unsupported_key_type, empty_input_message, input_format_name in the readers (#5891)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* 2.3.7, 2.3.9: read_object_key and leave_container_and_check for CBOR/MessagePack/BON8 (#5891)
UBJSON keys (different getter signature) and its container tail stay as is.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* 2.3.5: json_sax_dom_error_state base for the DOM SAX parsers (#5891)
Holds errored, allow_exceptions, parse_error() and is_errored(). The scalar
handlers stay in each class (parse hot path).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* 2.3.6: table of control-character messages in the lexer (#5891)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* 2.3.8: finish_input helper for the strict-EOF block in the parser (#5891)
sax_parse keeps its if (result) guard and return value. Also adds the
clang-tidy NOLINT for the 2.3.6 message table.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* 2.5.7: share parse_array_index between json_pointer and json_view (#5891)
Move json_pointer::parse_array_index() and its status enum to namespace detail
(json_pointer keeps a forwarding member), and let the json_view pointer
resolution use it instead of its own copy of the RFC 6901 index grammar. Both
accepted the same tokens with the same outcome. Also update the comment on
detail::decode(), which listed four of the five UTF-8 validators and now points
to the test that keeps them in sync.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Amalgamate (#5891)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* 2.4.5: keep the inlined number cases in from_json(ArithmeticType&) (#5891)
Calling get_number_value() from from_json(ArithmeticType&) made
GetArray<std::array<int, 1000>> 2.4x slower (it changes the inlining of the
std::array conversion), so that overload keeps its own switch, with a comment.
get_arithmetic_value() still uses the shared helper.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* 2.5.7: force-inline detail::parse_array_index (#5891)
Moving parse_array_index out of json_pointer stopped clang from inlining it
into the pointer resolution (JsonPointerAt/citm_catalog +17%,
JsonPointerAt/twitter +8%); JSON_HEDLEY_ALWAYS_INLINE restores the baseline.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* 1.5: keep get_bon8_string's own UTF-8 ranges (#5891)
Taking the second-byte range from utf8_shape() made BON8 decoding 2-3% slower
(FromBinaryShape/scalars, FromBinaryBuffer, alignment-pinned builds), so the byte
path keeps its hand-written ranges. The exhaustive validator test still checks
that it accepts the same sequences as the other validators.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Amalgamate (#5891)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix CI failures: constexpr, NRVO, cppcheck, IWYU (#5891)
- json_sax_dom_error_state::is_errored() cannot be constexpr: the class
is not a literal type (GCC 4.8, GCC -Wpedantic, Clang 3.5).
- to_binary_vector: destroy the vector writer before returning, as the
output_vector_sink sets the size of the vector in its destructor.
Without NRVO (MSVC Debug in C++14 mode), the vector was moved out
before it was resized.
- parser::sax_parse: return false instead of result after the early exit
(cppcheck identicalConditionAfterEarlyExit).
- json.hpp: export detail/exceptions.hpp for IWYU, so json_view.hpp can
name exception_id without including a detail header.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address review: throw helpers take input_format_t, simpler type checks
- throw_on_discarded() and throw_integer_out_of_range() take an
input_format_t and call input_format_name() themselves
- has_no_children(), set_parents() and max_size() check array, then
object, instead of calling is_primitive() first
- parse_impl() chains like accept_impl(); sax_parse_impl() uses if/else
- std::array for the control character messages; negation<> in
write_object_key(); clarify the is_primitive() comment
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* json_view: use exception_id in the value() overloads
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* json_sax: assert the invariant of place_container for the static analyzer
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
- Make the iterator range illustrations legible in dark mode (prefers-color-scheme
rules inside the SVGs).
- Add a "Byte layout" section with a color-coded byte table to each binary format
page (BJData, BON8, BSON, CBOR, MessagePack, UBJSON), and highlight the smallest
size per file in the comparison table.
- Add "See also" sections to the overview pages that lacked one.
- Split the release notes into one page per release, grouped by minor version on
the overview page, with an "API changes" section per release generated from the
API pages' version histories (hooks/api_changes.py).
- Fact-check the release notes and version histories against the release tags and
correct wrong or misleading entries (e.g., json_serializer 2.1.0, merge_patch
3.1.0, items, count, operator[], to_bjdata, std::string_view key support).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Reserve the parser's nesting stack once
The stack of open arrays and objects was a std::vector<std::uint8_t> that
allocated at the first push_back and then reallocated at 1, 2, 4, 8 bytes,
so a document nested four levels deep made four allocations before it
parsed a value. A document that starts with a container now reserves 32
levels in one allocation; deeper documents grow as before. The loop that
reads the tokens is the code it was, so large documents are not affected
(a stack object with its own storage, and a reserve at every container,
both cost the token loop instructions or cycles).
Fixed cost of parsing a 124-byte object (macOS instructions): accept
-7.6 %, DOM -2.5 %; no change for a single number.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Build arrays of the DOM parser in one exact-size allocation
Reading an array of unknown length appended its elements to a std::vector
one at a time, so a small array reallocated (and moved its elements) at
sizes 1, 2, 4, ... The DOM SAX parser now collects the elements of arrays
whose length is unknown on one stack shared by all open arrays and moves
them into an array of exactly the right size when the closing bracket
arrives. An array that grows past 16 elements switches to appending to its
own vector, so large arrays pay no second copy. Binary formats, which
announce the length, and builds with JSON_DIAGNOSTICS or
JSON_DIAGNOSTIC_POSITIONS (whose values need their final place from the
start) keep storing into the array directly.
The stacks of open containers are reserved on first use instead of growing
from one element.
The binary readers get a DOM parser of their own, json_sax_dom_binary_parser:
it is the plain parser that stores every value directly in its container
(the code the readers used before). Their containers announce their size, so
the bookkeeping of the collecting parser (frame entries, the destination of a
finished container) was pure overhead for them: with the collecting parser
FromBinaryShape nested was 3-12 % and containers up to 5 % more instructions
(GCC) than before. With the plain parser Linux Ir of FromBinaryShape and
FromBinaryBuffer is within 0.8 % of develop (Clang: identical).
canada.json has 56 000 two-element arrays that took three allocations each
and now take two. DOM parse Ir on Linux against develop (GCC / Clang): canada
-7.0 % / -7.7 %, citm -0.8 % / -1.6 %, 124-byte object -5.4 % / -5.5 %;
accept and SAX are untouched.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Reuse one value for the key events of the callback parser
parse() with a callback built a new string value for every key it passed to
the callback (a heap allocation and release per key), and copied the key a
second time into the object. The callback parser now keeps one string value
for key events, which it assigns, and moves the key into the object.
Callback parse, Linux Ir (GCC / Clang): twitter -13.3 % / -11.7 %, citm
-9.5 % / -7.6 %. (Earlier macOS measurement: twitter -19.5 % instructions.)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Remember where the DOM parser stores the next value
The DOM SAX parser looked up the innermost open container for every value:
the end of its frame stack, then the object or array pointer in the frame.
Compilers have to reload all of that after each value is stored, since a
store of a value may alias the frame. The parser now keeps the innermost
container's kind and, for an array that stores directly, the array itself
in members, updated when a container opens or closes.
Flat arrays of numbers are the case that mattered: parsing floats.json
(1 000 000 doubles) into a DOM took 2 % more cycles than before the
previous commit on macOS and now takes 0.4 % fewer than the code before it
(signed_ints.json -0.8 %). Linux Ir is unchanged (canada -6.1 % GCC /
-6.9 % Clang, citm -0.3 % / -0.8 %, twitter +0.2 % / 0.0 %).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Look up the keys of big ordered_json objects in a hash table
ordered_map keeps its elements in a vector and searches it linearly for
every key it adds, so parsing an object with n keys into ordered_json took
O(n^2) key comparisons: 33 billion instructions for one object with 30 000
keys, where json needs 245 million. From 64 keys on, the DOM parser now
keeps a hash table of the keys of the open object (hashes and positions
only; the elements stay in the vector) and finds duplicate keys there. The
result is the same, including the position of a key that occurs twice.
Smaller objects, std::map based objects and key types without std::hash
are untouched.
ordered_json parse, macOS instructions: object with 1 000 keys 64.8 M ->
14.6 M, 10 000 keys 5 104 M -> 38.7 M, 30 000 keys 33 358 M -> 89 M.
Linux Ir of an object with 2 000 keys: 43.0 M -> 1.9 M (GCC), 37.9 M ->
1.8 M (Clang).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Trim the work the DOM parser does per container and per member
Opening and closing an array or object cost the DOM parser more than before
the exact-size array change, which showed on documents of many small
objects (jeopardy.json: +1.4 % Ir with GCC). This takes it back:
- the stack of open containers keeps its first eight entries in the parser
object instead of in a vector, so it needs no capacity check beyond the
push itself and no allocation for typical documents;
- a container is created in the place it ends up in, instead of as a
temporary that is then moved there;
- a new object member, which is null, is constructed over the null instead
of being swapped with it (only with the default base class of basic_json);
- the key index that big ordered_json objects use is only a part of the
stack entries of objects that can have one;
- an array that outgrows the stack reserves twice what it has so far, as a
vector that grew by itself would have.
DOM parse Ir of the series up to here against develop, Linux GCC / Clang:
canada -6.6 % / -7.3 %, citm +1.2 % / +0.2 %, jeopardy +0.4 % / -0.1 %,
124-byte object -6.5 % / -6.2 %. (twitter measures +6.8 % / +6.6 % at this
point in steady state, almost all of it in glibc's malloc/consolidate for the
new allocation sizes; with the last commit of the series it is -0.1 % / 0.0 %.)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Test parsing containers of the sizes where the DOM parser changes storage
The DOM parser collects the elements of an array on a stack (up to 16) and
the members of an ordered_json object (up to 32) before it moves them into
the container, and uses a hash table for keys from 64 members on. Parse
arrays and objects of the sizes around these limits, nested into each other,
with repeated keys, and as random trees that are built next to their text,
and compare json and ordered_json against the value built through the API.
The tests pass on the code from before these changes as well. The random
trees are what finds the mistakes that are easy to make here: while the
collecting change was being written, a collected array that outgrew the
stack at the moment its parent had just reached the limit left its elements
in the parent, and only these tests (142 failed assertions) showed it.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Do not convert floats that cannot overflow in accept()
accept() only has to find out whether a float token is finite: json_sax_acceptor
ignores the value, and the parser rejects only a float that overflows. A
float whose digits fit into 19 digits and whose decimal exponent is at most
max_exponent10 - 19 is below 10^max_exponent10 and cannot overflow, so the
lexer, told by accept() that the value is not needed, skips the conversion
for those. A token that may overflow, with more digits, or in another float
format is converted as before, so 1e309 and its neighbours are rejected as
they were (the new test compares accept() with parse() on tokens around the
limit).
json::accept(): canada.json Ir -8.6 % (GCC) / -10.9 % (Clang), 124-byte
object -3.2 %; cycles on the M1 (release benchmarks, LTO): canada -7.8 %,
floats.json -9.0 %. The check tests the accept() flag first, so the parser
that needs the value pays one load and branch: SaxParse canada +0.35 % Ir
(GCC and Clang), DOM canada +0.4 %.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Collect the members of ordered_json objects on a stack until they are known
An ordered_json object keeps its members in a vector and appends every new
member to it, so reading an object of unknown size reallocated the vector at
1, 2, 4, 8, ... members, and ordered_map copies all keys on every
reallocation. The DOM SAX parser now collects the members of such objects on
one stack shared by all open objects, as it does the elements of arrays, and
moves them into a vector of exactly the right size when the closing brace
arrives. An object that gets past 32 members switches to appending to its
own vector (and, from 64, to the key index of the earlier commit). The
members are the same, in the same order, and a key that occurs twice keeps
the place of its first occurrence. std::map based objects, objects whose
size the input announces (binary formats), and builds with JSON_DIAGNOSTICS
or JSON_DIAGNOSTIC_POSITIONS are untouched.
What only ordered objects need (the collected members, the key indexes) is
a member of the parser only for them, and the code that stores a value tries
the array that takes its elements directly first. An array that outgrows
the stack now reserves 32 elements, the capacity a vector that grew by
itself has at that point; with the larger reserve before, parsing
floats.json (1 000 000 numbers) was 2 % slower on Apple M1 than before this
series.
ordered_json parse, Linux Ir against develop (GCC / Clang): citm -11.6 % /
-11.7 %, twitter -8.4 % / -8.4 %, jeopardy (12 000 objects of 7 keys)
-10.5 % / -8.0 %, canada -3.9 % / -5.2 %, 124-byte object -20.9 % / -16.5 %.
macOS time (M1, earlier base): citm -19 %, twitter -13 %, jeopardy -13 %,
canada -14 %, 124-byte object -26 %. json parse: steady-state twitter -0.1 %
(see the previous commits), citm -1.2 %.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Rename the key index's match callback so Flawfinder doesn't read it as std::equal
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Bind the parser test's strings by reference in the range-for loop
GCC and Clang 14 (-Wrange-loop-construct) reported the copy.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Collect container values only in basic_json's own parse functions
A json_sax_dom_parser that a user constructs (as in the sax_no_exception
example) must hold the values read before a parse error, as on develop:
its value is used after the error. The values of containers of unknown
size are now collected on a stack only with the new template parameter
Collect (default false), which json::parse() and the json_view
materialization set; without it, every value is stored in place as it
arrives. A test checks the values kept after an error for json and
ordered_json.
Also fixes the clang-tidy, cpplint and Infer findings: the frame stack
keeps its first frames in a std::array (left uninitialized) and the
others in a std::unique_ptr, auto for create<>(), NOLINT for member
initializers -Weffc++ needs, and an @infer-ignore for the nullptr that
place_container() does not use.
json::parse() and ordered_json::parse() instructions are unchanged
(Apple clang; twitter, citm, canada, jeopardy: -0.03 % to +0.25 %).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Clarify the DOM parser's collection stacks and rename reenter_enclosing_container()
Address review comments: explain why a closed container erases only the top
of the shared stack, set the payload before the type, rename the helper that
makes the enclosing container current again, and declare the parser's state
stack right before reserving it.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Export json_sax_dom_parser from the C++20 module for MSVC
MSVC needs the detail classes that basic_json befriends to be visible in
the module (#3970, as json_sax_dom_callback_parser); with its new
template parameter, json_sax_dom_parser failed in ci_module_cpp20
(C2039, not a member of nlohmann::detail).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Keep the overflow-limit numbers of the parser test in a std::vector
clang-tidy (cppcoreguidelines-avoid-c-arrays) rejected the C array.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix the clang-tidy findings of the ordered_json parser tests
The container sizes are a std::array, the expected text uses raw string
literals, the generator's arithmetic is in parentheses, auto for the
result of make<>(), and single characters are appended as characters.
clang-tidy now reports nothing on the lines the branch adds to
unit-ordered_json.cpp and unit-class_parser.cpp (apart from checks CI's
clang-tidy does not have).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Read C streams into a json_document in blocks
json_document::parse(FILE*) (and read(), accept(), parse_copy()) went through the
library's file input adapter and appended the text character by character
with fgetc. Read it with fread in blocks instead, like the std::istream
overload already does.
Parsing citm_catalog.json (1.7 MB) from a FILE*:
Apple M1: 80.6 M -> 8.4 M instructions, 4.8 ms -> 0.55 ms (parse of the same text from a std::string: 8.7 M)
Linux x86-64 Ir (whole run), GCC 13 / Clang 18: 47.0 M -> 13.7 M / 45.8 M -> 14.2 M
The parsed documents and the error cases are identical (checked over the test corpus
and 30 mutants of each file). A null FILE* is handled as before (by the library's adapter).
Belongs to the PR "json_view: read-side speedups".
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Index arrays without subtrees directly
operator[](idx), at(idx) and back() skipped the elements of an array one
after the other, whatever they were. In an array none of whose elements is
an array or object (numbers, strings, booleans, null) the elements are
adjacent nodes, which the array node's own sizes tell: its subtree has
exactly one node more than it has elements. Such an element is now the
node idx after the array node; any other array is walked as before.
Reading r[i] for every i of an array of 3000 integers:
Apple M1: 28.8 M instructions, 3.1 ms -> 0.1 M instructions, 4 us
Linux x86-64 Ir (whole run), GCC 13 / Clang 18: 84.1 M -> 3.1 M / 75.0 M -> 3.0 M
Arrays that contain arrays are unchanged.
Belongs to the PR "json_view: read-side speedups".
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Skip the bounds checks of number and string tokens in NUL-terminated input
For a NUL-terminated input (std::string, C string, string literal), the
parser reads the terminator instead of comparing p with the end, via
cursor::cur(). Number and string scanning still tested `p != e` before each
look at the next byte (after the integer digits for '.', for 'e', after
the exponent sign and digits, and after a string for its closing quote),
although the terminator can never be one of those characters. They now use
cur(), which is the plain load for NUL-terminated input and the bounds check
for ptr+len input, as before.
Parsing, Apple M1 instructions per parse: twitter 3.126 M -> 3.089 M (-1.2 %),
citm_catalog 8.70 M -> 8.58 M (-1.3 %), jeopardy 306.1 M -> 302.5 M (-1.2 %), signed
integers 281 M -> 266 M (-5.4 %), canada and floats -0.1 % / -0.9 %.
Linux x86-64 Ir, GCC 13 / Clang 18: twitter -0.5 % / -0.0 %, signed integers +0.2 % / -0.7 %.
Belongs to the PR "json_view: read-side speedups".
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Make the index text of items() on arrays lazy
basic_json_view::items() on an array formatted every element's index with
std::to_string when the item was made, whether or not the loop asks for the
key. The item now keeps the position and makes the text on the first key()
call. Objects are not affected.
Iterating over the items of an array of 3000 numbers and reading only the values:
Apple M1: 25.5 us -> 4.6 us (0.45 M -> ~0 net instructions)
Linux x86-64 Ir (whole run), GCC 13 / Clang 18: 14.9 M -> 4.1 M / 15.3 M -> 4.6 M
Reading key() costs what it did (24.5 us).
Belongs to the PR "json_view: read-side speedups".
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Make the default of value() with a C string only if the member is missing
basic_json_view::value(key, "text") and value(pointer, "text") built a
std::string from the default first and passed it on, so a call allocated
for the default even when the member exists, and copied it again when it
does not. They now look the member up and make the result directly: from the
member if there is one, else from the C string. Behavior, including the
exceptions, is the same.
Two calls (member found, member missing) with a 40 character default, Apple M1:
1.6 K -> 0.7 K instructions
Linux x86-64 Ir (500 rounds), GCC 13 / Clang 18: 2.46 M -> 2.22 M / 2.40 M -> 2.15 M
Belongs to the PR "json_view: read-side speedups".
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Materialize scalars directly and keep the stack of small subtrees local
basic_json_view::materialize() (and get<BasicJsonType>()) always set up the
SAX handler that parse() uses and a std::vector for the open containers,
sized from the document's deepest nesting. For a scalar, which is most of what
is materialized one value at a time, neither is needed: the value is made
directly (the same constructors the handler ends up calling). For arrays and
objects the stack of open containers is now a local array for the first 16
levels, and the vector is only used for deeper documents.
Materializing five scalars of a small document, Apple M1: 3.6 K -> 0.9 K instructions
Linux x86-64 Ir (500 rounds), GCC 13 / Clang 18: 2.96 M -> 2.37 M / 2.90 M -> 2.29 M
Belongs to the PR "json_view: read-side speedups".
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Copy the text between escapes while scanning it
A string that contains an escape is decoded by builder::slow_string(). After
each escape it scanned the next run of plain bytes (8 or 16 byte-by-byte
steps, then a vector loop), and then copied that run to the decode arena in a
second pass. The run is now copied 16 bytes at a time while it is scanned
(one vector load, a store of the same 16 bytes, and the mask of the bytes
that end the run), and the write position lives in a local variable, not in
the builder, since a store of a character may alias the builder's members and
made the compiler reload it after every escape. A run that ends at a quote
needs no scan, and non-ASCII text goes through the UTF-8 check as before;
the portable code is unchanged.
Parsing, Apple M1 instructions: jeopardy.json 304.0 M -> 293.8 M (-3.4 %), 3000 strings with
escapes 1.72 M -> 1.54 M (-10 %), twitter -0.6 %.
Linux x86-64 Ir, GCC 13 / Clang 18: escaped strings -7.6 % / -11.4 %, twitter 0.0 % / 0.0 %.
Documents and error messages are identical on 900 k random strings of escapes, UTF-8 and control
characters (also under ASan), and on mutants of the test corpus.
Belongs to the PR "json_view: read-side speedups".
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Hash the last bytes of a key of a large object without a loop
key_hash() mixed a key eight bytes at a time and then packed the remaining
0-7 bytes one by one, in a loop. The last word of a key of eight or more
bytes is now read as the last eight bytes of the key (overlapping the word
before), and a key of 2-7 bytes as two overlapping words of four or two
bytes. A product only moves bits upwards, and the slot is taken from the low
bits: the high bits of the last word are folded down before the
multiplication, since keys that differ only in their last characters ("user_17",
"user_18") would otherwise share slots. The hash is not stored, so its values
may change.
Looking up every key of an object of 1000 random keys, Apple M1 instructions per 1000 lookups:
keys of 5 bytes 175 K -> 93 K, hex strings of 8 bytes 177 K -> 168 K, keys of 12 bytes 196 K -> 186 K
Linux x86-64 Ir (whole run), GCC 13 / Clang 18: 5-byte keys -4.7 % / -8.8 %, 12-byte keys -3.3 % / -7.4 %.
Building the tables while parsing such objects is cheaper as well (5-byte keys: 13.3 us -> 10.1 us).
Belongs to the PR "json_view: read-side speedups".
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Compare short keys of a large object without a call of memcmp
find_indexed() compared the key of a member the table points to with
memcmp, a library call for any length. Up to 16 bytes, the bytes are now
compared as two overlapping words (the first and the last 8, 4, or 2 bytes),
as find_member() already does for the objects without a table; longer keys
still use memcmp.
Looking up every key of an object of 1000 random keys, Apple M1 instructions per 1000 lookups
(with the previous commit): keys of 5 bytes 93 K -> 72 K, hex strings of 8 bytes 168 K -> 148 K,
keys of 12 bytes 186 K -> 170 K, keys of 24 bytes 198 K -> 193 K (unchanged, memcmp).
Linux x86-64 Ir (whole run), GCC 13 / Clang 18: 5-byte keys -7.2 % / -7.3 %, 12-byte keys -5.9 % / -6.6 %.
Belongs to the PR "json_view: read-side speedups".
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Read an unedited editable document like a read-only one
Views of a json_editable_document followed edits at every step: the
lookup, the element access, back(), dump() and materialize() were built
for nodes that can be moved or linked, though no edit changes the index until
the first one (every edit creates the edit state before it touches a node).
Until then these now take the read-only code: one check of the document's
edit state per call, not per member or per node. After the first edit
nothing changes. Views that were made earlier keep working, as the check is
made when they are used.
Unedited editable document, Apple M1: dump() of twitter.json 1.40 M -> 1.22 M instructions (-12.6 %);
lookups unchanged. Linux x86-64 Ir, GCC 13 / Clang 18: dump -1.7 % / -1.0 %, lookups -0.1 % / 0.0 %.
Walking with iterators is unchanged, and so is dump() after an edit.
Belongs to the PR "json_view: read-side speedups".
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Format the numbered test keys of json_view's index with a stream
The keys were made with snprintf() and a format from a list, which
Flawfinder reports as an error (CWE-134, a format string that is not a
literal). A stream now formats them; the 30000 keys of the test are the
same as before.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Put the key number of the json_view index test in parentheses
astyle formatted i * step after << as a pointer declaration (i* step),
which failed the amalgamation check.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix the compiler and lint findings of json_view's read side and tests
- find() and contains() dispatch on Editable for the member lookup; for
documents that cannot be edited both branches were the same call
(GCC -Wduplicated-branches)
- json_view.hpp includes <cstdio> for the FILE overload (cpplint)
- the arena check macro of the builder is on one line (cpplint read it
as an if body without braces)
- tests: no nested conditional (also MSVC C4018), no useless cast,
parentheses around mixed operators, raw string literals, += instead
of temporary concatenations, and NOLINT for the fixed seeds and a
find() != npos, as elsewhere in the tests
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Grow the decode arena through a function and store its position back on every return
slow_string() kept its write position in locals and grew the arena through
the macro NLOHMANN_VIEW_ENSURE. The macro is now the always-inline member
arena_ensure() (a lambda is not inlined at every call site by clang, which
then keeps the position in memory: +11 % instructions on escape-heavy
strings), and a guard stores the position back into m_arena_write on every
return instead of by hand on the successful one. Instruction counts are
unchanged (clang) or 1 % lower (GCC).
Also word the complexity of back() as that of at().
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Write the indented output of json_view's dump() with the fast writer
The compact writer of dump() (cursor in a register, fixed-size copies of
tokens) served dump(-1) only; every dump(indent), and everything with
ensure_ascii that was indented, went through a generic walk that calls put()
for each token and put_repeated() (a memset call) for each line break. The
fast writer now writes indented output, too, with or without ensure_ascii, and
the generic walk is gone. A line break is a copy of a 64-byte ruler (a line
break and 63 indentation characters) when the line is at most 64 bytes, and a
store plus memset beyond that. The depth is counted only in the indented
instantiations (counting it in the compact one cost an editable document 5 %
with Clang). The indented walks are out of line (dump_pretty()): inlined
into dump() next to the compact ones, they made compact dumps of integer
arrays 7 % slower.
A decoded string (one with escape sequences) of a damaged image loaded with
image_check::bounds can contain invalid UTF-8. As before, the compact dump
writes such bytes unchanged, and the indented dump throws type_error.316; its
runs are found by the lexer's validating scan (a scalar validation of every
sequence made dump(2) of decoded CJK text twice as slow as before).
dump(2) instructions per call, macOS arm64 (before -> after):
twitter citm canada jeopardy(4k)
json 3.75M -> 1.85M 8.28M -> 3.93M 67.1M -> 56.0M 9.56M -> 4.81M
editable 3.82M -> 1.84M 8.48M -> 4.02M 68.1M -> 56.2M 9.66M -> 4.75M
Linux instructions (GCC 13 / Clang 18), dump(2) and dump(2, ' ', true):
twitter -25 % / -30 % -11 % / -11 %
citm -29 % / -28 % -24 % / -24 %
jeopardy(4k) -25 % / -32 % -20 % / -24 %
CJK text with \u escapes -6 % / -11 % -2 % / -2 %
The compact dump is unchanged (within +0.4 % on Linux, within noise on the Mac).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Scan the strings of json_view's dump() with ensure_ascii 16 bytes at a time
With ensure_ascii, a string of the source that has a byte >= 0x7F was written
by a generic path: put() per piece and a validate-and-decode step for every
character beyond ASCII. It now goes through a writer with the cursor in a
local variable. The part of the string that has nothing to escape is found
16 bytes at a time (SSE2 or NEON; the existing 8-byte scan without vector
support) and copied; strings that are ASCII throughout are still decided by
the 8-byte check of dump() with ensure_ascii (a 16-byte scan for them was
2-5 % slower for short strings). The two-byte sequences (lead C2..DF) and
the three-byte sequences whose continuation bytes may have any value (lead
E1..EF except ED) are decoded with a few compares; E0, ED, four-byte
sequences, and damaged input go through validate_one_utf8() as before, so
damaged images still throw type_error.316.
dump(-1, ' ', true) instructions per call, macOS arm64 (develop -> now);
the compact dump without ensure_ascii and dumps of ASCII text are unchanged:
twitter 5.42M -> 3.57M citm 3.28M -> 3.24M jeopardy(4k) 5.78M -> 5.65M
CJK text, raw 66.5M -> 35.3M escaped \u 32.7M -> 17.0M
Latin text with accents 11.5M -> 9.4M ASCII text 4.40M -> 4.40M
Linux instructions (GCC 13 / Clang 18), dump(-1, ' ', true):
twitter -9 % / -33 % CJK text -14 % / -47 % Latin text -8 % / -18 %
citm -0.2 % / -0.4 % ASCII text -0.1 % / -0.3 %
The compact dump without ensure_ascii is unchanged (canada 0.0 %).
The unit test covers every character of the Basic Multilingual Plane, and a
character that needs an escape at each position of strings of 0..40 bytes.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Find the end of the plain run of a decoded string 16 bytes at a time with SSE2
The strings of a document that had escape sequences are written in runs
without a quote, backslash, or control character; the compact dump found the
end of a run 8 bytes at a time with a SWAR test. With SSE2 (x86-64) it now
compares 16 bytes per step. NEON keeps the SWAR code: there the mask of a
step costs more than it saves on the short runs that are typical here.
Instructions of dump(), Linux (GCC 13 / Clang 18):
twitter.json 1.99M -> 1.88M (-5.2 %) / 1.97M -> 1.92M (-3.0 %)
CJK text with \u escapes 5.98M -> 4.52M (-24.5 %) / 5.30M -> 4.25M (-19.9 %)
ASCII text with a few escapes 4.28M -> 4.09M (-4.4 %) / 4.24M -> 4.11M (-3.2 %)
jeopardy (4,000 questions) 5.08M -> 4.78M (-5.8 %) / 4.97M -> 4.80M (-3.4 %)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Rename depth_counter's open() and close() to enter() and leave()
Flawfinder took them for the file functions open() and close().
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix the clang-tidy findings in json_view's dump and its tests
- the SSE2 loop of the string scan is within NOLINTBEGIN/END
(portability-simd-intrinsics), as in to_chars.hpp
- depth_counter<false>'s leave() and get() keep the interface of
depth_counter<true> instead of being static
- write_ascii() compares the room left as a size_t variable
- the test's texts are a std::array, and a JSON fragment with escapes
is a raw string literal
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Move dump_ascii() and dump_pretty() next to dump()
Address a review comment: the two dispatchers that dump() calls now follow
it directly.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
- generate_docset.py: use SHA-256 instead of SHA-1 for the cache file
name digest of downloaded images.
- generate_docset.py: suppress dynamic-urllib-use-detected; the URL
scheme is restricted to http(s) right before the call.
- generate_natvis.py: suppress direct-use-of-jinja2; the template
renders a local XML file with autoescape enabled.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Niels Lohmann
dependabot[bot]