mirror of
https://github.com/nlohmann/json.git
synced 2026-11-01 18:35:34 +00:00
af60d07e89cd27f8257abff3bdbbe153ee0bfc8a
1125
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Niels Lohmann
|
af60d07e89 |
Merge branch 'perf/msgpack-scalar-runs' into perf/ubjson-typed-arrays
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
d2436351cd |
MessagePack reader: mark the initial accepted in the scalar run as a false-positive dead store
ci_infer reports DEAD_STORE for 'bool accepted = true;' in the MessagePack scalar run, as in the CBOR one, because every case assigns accepted before it is read. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
2a816f9842 |
Merge branch 'perf/binary-reader-byteswap' into perf/msgpack-scalar-runs
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
a0e2cdbab9 |
Merge branch 'perf/cbor-scalar-runs' into perf/binary-reader-byteswap
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
bf81ab564f |
CBOR reader: mark the initial accepted in the scalar run as a false-positive dead store
ci_infer reports DEAD_STORE for 'bool accepted = true;' in parse_cbor_scalar_run because every case assigns accepted before it is read. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
a0b9ca9021 |
Merge branch 'perf/msgpack-scalar-runs' into perf/ubjson-typed-arrays
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
70a4b25cd2 |
MessagePack reader: enter the scalar run only when a scalar follows
As for CBOR: enter_msgpack_array tests the first byte and the remaining length inline and calls the out-of-line run only for a scalar, so arrays of maps or strings no longer pay for a call that reads nothing; the run's loop condition moves to the end (do-while). Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
5c7589dd15 |
Merge branch 'perf/binary-reader-byteswap' into perf/msgpack-scalar-runs
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
6b07a393fb |
Merge branch 'perf/cbor-scalar-runs' into perf/binary-reader-byteswap
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
8bbfb5ec1d |
CBOR reader: enter the scalar run only when a scalar follows
An array of containers or strings called the out-of-line run start once per array only to leave it at once; on nested data (arrays of objects) that call made reading slower than before the run reader. enter_cbor_array now tests the first byte and the remaining length inline and calls the run only for a scalar, so the run's loop condition moves to the end (do-while). Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
7f38373909 |
UBJSON/BJData readers: read optimized arrays of numbers in one loop
An array with a type marker for a fixed-size number ($U, $i, $I, $l, $L, $d, $D, and BJData's $u, $m, $M), including the data of a BJData ndarray, gives its elements no markers. When the input is contiguous, read the elements that are in the input in one loop - load, byte swap for the format's byte order, emit - instead of one trip through parse_ubjson_internal and get_ubjson_value per element. Elements the input ends before, other types, and non-contiguous inputs are read as before. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
e5cd9db7b3 |
MessagePack reader: read runs of scalar array elements in one loop
As for CBOR: when an array opens and the input is contiguous, read the integers, floats, booleans, and nils that follow in a tight loop instead of going through parse_msgpack_internal and parse_msgpack_value for every element. The loop stops at any other kind of value and when fewer than 9 bytes are left; those values are read by the general path as before. Non-contiguous inputs are not affected. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
cc31370a78 |
Binary readers: swap the bytes of floats with a single byteswap
byte_swap reversed float and double values byte by byte. GCC does not merge that loop into one bswap, and C++23's std::byteswap accepts only integers. Overloads for float and double now copy the bits into a uint32_t/uint64_t, call detail::byteswap, and copy them back. This applies to every binary reader that uses get_number. In the CBOR reader, the single- and double-precision float cases call a non-inlined helper. Without it, GCC stopped inlining the stream adapter's get_character() into parse_cbor_internal, which made reading from a std::istream about 0.5 % slower. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
ce7498b54c |
CBOR reader: address review of the scalar-run reader
- parse_cbor_scalar_run: set the size explicitly in every case that reads a value, instead of relying on a default that every other case overwrites. - Remove cbor_scalar_follows(): parse_cbor_scalar_run already returns on the same conditions, and without the pre-check SAX canada needs another 8.5 % fewer instructions with GCC 13 (0.915 against the previous version), with no row more than 0.5 % worse (Mac and GCC). - Restore the comment on why parse_cbor_internal copies the container frame. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
e3e46253ed |
CBOR reader: read runs of scalar array elements in one loop
When a definite-length CBOR array opens and the input is contiguous, read the integers, floats, booleans, and nulls that follow in a tight loop instead of going through parse_cbor_internal's container logic and parse_cbor_value's dispatch for every element. The loop stops at any other kind of value and when fewer than 9 bytes are left; those values are read by the general path exactly as before. Non-contiguous inputs are not affected (tag dispatch on the bulk-scan capability). Also pass a member empty string as the number source to number_float instead of constructing a std::string for every float. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
0115bb47b3 |
Name the source and license of adapted code in the file comments (#5965)
The files with code adapted from fast_float, Zmij and simdjson carried the authors' SPDX copyright lines, but their file comments did not say what was taken or under which license: number_parse.hpp, to_chars.hpp, simd_scan.hpp and view/simd.hpp now do, as zmij.hpp and the yyjson-based view files already did. README and the license page now also name fast_float's eight-digit parsing and the lexer's use of simdjson's UTF-8 check. Comments only. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
54f6b3ef06 |
json_view: source positions in materialize() with JSON_DIAGNOSTIC_POSITIONS (#5945)
* json_view: source positions in materialize() with JSON_DIAGNOSTIC_POSITIONS With JSON_DIAGNOSTIC_POSITIONS, materialize() now gives every value the start_pos()/end_pos() that basic_json::parse() would give it, so exceptions about materialized values name the same bytes (#5864). - json_sax_dom_parser takes the lexer type as a defaulted template parameter; materialize() passes position_source, which reports the positions it derives from the index. diagnostic_positions treats npos as "no position" and keeps ranges all-or-nothing. - Strings with escapes keep the offset of their opening quote plus one in `next` (only with the macro); materialize() scans from there to the closing quote. The end of an array or object is found by scanning from the end of its last element over whitespace, comments and a trailing comma. - In editable documents, values written by edits, arrays and objects whose elements changed, and containers whose last element has no position get none. Images keep the flag is_new, and load() checks that `next` of a string is within the text. Nothing changes on the parse path without the macro. The view's own exceptions still carry no byte range, so the tests that compare them with those of materialized values skip it under the macro, as they skip the path under JSON_DIAGNOSTICS. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Indent comment in unit-json_view_from_json.cpp Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address review: clearer position code and docs - position_source gets clear(); set_scalar() returns early for new values, switches on node_kind, and uses no local copies; the one position_source per materialize() call is documented - no npos + 1 for containers without a position; a comment on why the end of a container is reported as its closing bracket - less duplication under #if in materialize() and the builder - diagnostic_positions sets npos explicitly - docs: positions of basic_json values after changes, positions in images depend on how the document was parsed, when next is set, and the cost of positions while parsing Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Share the comment scanners of the view builder with materialize() skip_line_comment() and find_block_comment_end() in scan.hpp are used by the builder's comment() and by position_source::container_end(), which had its own copy. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address review: if/else for container positions, clearer next field - the opening bracket of a container sets positions with an if/else instead of a ternary, and the frame takes its start from positions - the closing bracket of a container is found in nested if/else; the parent's last_end is taken from positions, so it is npos whenever no range is set - architecture.md: the next field reads "or 0 if it was not; all other values: 0" instead of "else 0; otherwise 0" Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix GCC unused-result and clang-tidy raw-string findings in unit-json_view_positions.cpp Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
091c10d4a5 |
Fix self-aliasing in json_pointer::operator/= and patch_inplace (#5936)
* Fix self-aliasing in json_pointer::operator/= and patch_inplace json_pointer::operator/=(const json_pointer&) passed iterators into its own token vector to insert() when a pointer was appended to itself (p /= p), which is undefined behavior; libstdc++ produced "/a/b//" for "/a/b". The pointer now appends a copy in that case. basic_json::patch_inplace iterated the patch while changing *this, so j.patch_inplace(j) read freed memory. It now applies a copy of the patch when the patch is the value itself. A patch stored inside the value remains undefined behavior and is documented as such; copying the patch unconditionally would double the time of a large patch. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix JSON_DIAGNOSTICS test crash and clang-tidy hit in patch_inplace self-aliasing The spare-capacity test reallocated non-empty elements through get_ref(), which leaves stale parent pointers under JSON_DIAGNOSTICS; reserve while the array is empty instead. Suppress performance-unnecessary-copy-initialization for the copy that is needed because *this is modified. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Detect a patch stored inside the patched value with JSON_DIAGNOSTICS patch_inplace applied a copy only when the patch was the value itself. With JSON_DIAGNOSTICS the parent pointers make it cheap to also detect a patch nested inside the value, so that case is handled the same way. Without diagnostics it stays documented undefined behavior. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Silence misc-const-correctness for patch_aliases_this Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix clang-tidy findings in the operator/= self-append test Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
78a0687147 |
Strong exception guarantee for operator[](json_pointer) (#5940)
* Keep the value unchanged when operator[] with a JSON pointer throws
The non-const operator[](json_pointer) is documented with strong exception
safety, but get_unchecked created intermediate values while walking the
pointer and kept them when a later reference token failed, e.g.
json j; j["/a/99999999999999999999999"_json_pointer] threw 404 and left
{"a":[]}; padding an existing array or appending with "-" behaved alike,
and so did std::length_error/std::bad_alloc from a huge index.
get_unchecked now only walks existing values up to the last reference
token. The last token's key insertion or array growth is strongly
exception safe by itself. Any other change goes through
get_unchecked_create, which records the one change it makes to an existing
value (null converted, key inserted, array grown) in a RAII guard and
reverts it if a later step throws; everything else it creates lies below
that change. The guard needs no try/catch, so JSON_NOEXCEPTION builds are
unaffected. Successful calls create the same values as before.
Docs: state the guarantee in the exceptions list and version history of
operator[], fix the complexity of the pointer overloads of operator[] and
at, and warn that padding arrays up to an untrusted index allocates memory
proportional to the index (operator[].md, untrusted_input.md).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Make the strongly exception-safe pointer operator[] faster than before
The previous commit made get_unchecked strongly exception safe but slower
with GCC 16 when it creates values (up to +6 % instructions for appending
with "-"). This restructures it so that no benchmark row is slower than
before the strong guarantee, with GCC 16, Clang 22 and Apple Clang.
- get_unchecked walks existing values in one loop that inserts nothing.
The last reference token is resolved in that walk: "-" appends with
emplace_back(), an index grows the array with operator[], and a missing
key is inserted with emplace_hint() at the position the walk's
lower_bound() found, so the key is searched once.
- With std::less<string_t> or std::less<> as comparator, the walk checks
the key at the lower bound for equality instead of equivalence, which is
the same for these comparators and cheaper.
- get_unchecked_create (not inlined) handles all other changes. Its guard
swaps a null value with a fresh null value instead of copying it and
reverts the change in a non-inlined function, so a successful call pays
one compare for it. As every null value converted afterwards is a fresh
one, it is converted in place instead of assigning a new array or object.
- Keys are added to new objects with emplace(), which also replaces the
basic_json::operator[] call for ordered_json.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix rollback of pointer operator[] for unusual object and string types
Follow-up to the strong exception guarantee of the non-const
operator[](json_pointer):
- get_unchecked_create assumed that emplace() always inserts the key that
the walk did not find. For object types whose lookup and emplace()
disagree (e.g. a copied fifo_map), nothing was inserted, the rollback
erased the existing member, and the walk continued with a value of the
wrong type (assertion, type_error.305, or a crash). It now checks the
result of emplace(), arms the rollback only if a key was inserted, and
otherwise continues the walk with the member found, as basic_json's
operator[] does: such pointers throw out_of_range.404 and leave the value
unchanged, as before the strong guarantee.
- The rollback erased the inserted key by key, which converts string_t to
object_t::key_type and may allocate inside a noexcept function
(std::terminate on bad_alloc with custom key types). It now erases by the
iterator emplace() returned, and shrinks a grown array with erase() at
the end instead of resize(). The documentation states the remaining
requirement for custom template parameters.
- Creating values no longer calls string_t::begin(), so the non-const
operator[](json_pointer) compiles with string types that lack it (as the
alternative string type of the tests).
- Correct the @throw lists of get_unchecked and get_unchecked_create.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address review comments on operator[](json_pointer) rollback
- undo() shrinks the array with resize() instead of erase()
- comments say what "will be" done in get_unchecked_create
- tests: avoid a clang-tidy performance-unnecessary-copy-initialization
hit (unit-alt-string) and the word "access" in section names, which
Flawfinder flags as race/access
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Let the pointer rollback guard call undo() unconditionally
The check in the destructor and JSON_HEDLEY_NEVER_INLINE on undo() made no
measurable difference: instruction counts of j[ptr] are identical with
GCC 13 -O3 and within 0.4 % with clang, timings are within noise.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
|
||
|
Niels Lohmann
|
9e67bfa115 |
json_view: the library's std conversions read views directly (#5923)
* json_view: the library's std conversions read views directly The library's from_json() overloads (containers, maps, std::array, C arrays, pairs, tuples, optional, valarray, forward_list, enums, binary, paths, and scalars) accept views: their traits use the view's basic_json type, and the few accesses that differ (get_ptr) go through small helpers. from_json() of a view now finds them; only the probe still looks up by ADL only. get<T>() of a view thus converts these types without materialize(), with the same values and exceptions; from an object, a repeated key keeps the last value, as parse() does. The view keeps its direct paths for vectors of the types it converts directly (faster than from_json()) and for containers of views; its other vector and map paths are replaced by from_json(). For basic_json, std::valarray and std::forward_list are read in one pass (40 % and 3.5 % fewer instructions); all other conversions are unchanged. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * json_view: enable_if_not_view_t instead of '!' in the from_json() templates astyle spaces out the template argument list when it contains a negation; the alias spells the condition with negation<> and leaves the lines alone. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * json_view: compact the template parameter lists astyle had spaced out Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
e5fe4ebdd2 |
json_view: from_json() of the macros reads views directly (#5922)
get<T>() and get_to() of a view call T's from_json() with the view itself when that from_json() accepts views but not an unrelated probe type, and fall back to materialize() otherwise. The from_json() functions of the NLOHMANN_DEFINE_TYPE_*, NLOHMANN_DEFINE_DERIVED_TYPE_* and NLOHMANN_JSON_SERIALIZE_ENUM(_STRICT) macros accept basic_json values and views; a derived type whose base reads only json reads the base part through materialize(). The library's own from_json() overloads are not called with views: from_json() of a view (and of the probe) only looks up functions by ADL. New public trait nlohmann::is_json_view<T>, and the member type basic_json_type of basic_json_view. With duplicate keys, the members of such a T are those the view's lookups find (the first ones). bench_decode.cpp measures decoding into structs (4-6x faster than materialize().get<T>() on twitter, citm_catalog, and synthetic records). Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
5a2819c940 |
json_view: one SIMD macro set with the library (#5921)
The view's vector code follows JSON_NO_SIMD, like the lexer and the float writer, and skips the CPU check of SSSE3 when the code is compiled for SSSE3 anyway (__SSSE3__), like the lexer. JSON_VIEW_NO_SIMD and JSON_VIEW_USE_SSSE3 and their ABI tags _vns and _vssse3 are removed; they were never released. JSON_NO_SIMD keeps its tag _ns. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
4cdfa6d0f4 |
json_view: no save() and load() on big-endian targets (#5920)
The image format is little-endian only. Instead of throwing type_error.320 at runtime, save_image(), load_image() and the members save() and load() do not exist on a big-endian target, so a call fails to compile, like set() on a read-only document. type_error.320 remains for saving a discarded document. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
60107eda3f |
Read BJData ND-arrays of float16, restrict ND-array char to 0..127, correct BJData docs (#5931)
* Read ND-arrays of half floats and enforce the char limit in ND-arrays BJData N-dimensional arrays with element type 'h' (float16) failed to read with a misleading parse_error.112, because the reader's dtype name table lacked 'h'. It now maps to "half", the JData name. The writer still does not pack "half"; such an object is written as a plain object and reads back equal. BJData limits 'C' (char) to 0..127 in Draft 2 and Draft 3, and the 1-D reader enforces this, but ND-arrays with _ArrayType_ "char" wrote values up to 255 as 'C' elements and read them back without a check. The writer now only packs values in 0..127 (others fall back to a plain object that round-trips), and the reader rejects 'C' elements above 127 with parse_error.113 like the 1-D reader. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: correct and complete the BJData documentation Verify the BJData chapter against the library and the Draft 2/3/4 specifications and fix what did not match: - bjdata.md: correct the list of unused markers (N, not Z), the ND-array example (3x2, not 2x3), the conditions under which an object is packed as an ND-array (exactly three keys, row vectors, float-only elements for single/double, char 0..127, byte only in Draft 3), the binary table row and notes (which flag/draft combinations write [$B), the "$" type restriction (Draft 3, Z is excluded too), the H marker in the deserialization table, and the claim of a complete deserialization mapping. Document the supported drafts, that Draft 4 (SoA containers, extension type E) and column-major ND-arrays are rejected, ND-arrays of half floats, duplicate object keys, and link the Draft 3 specifications instead of master. - Add an API page for basic_json::bjdata_version_t with an example, and link it from the nav, the API index and to_bjdata. - Update binary_formats/index.md and binary_values.md, which still claimed that BJData has no binary type, and fix a wrong byte comment there. - Document parse_error.115 and the out_of_range.408 cases for from_bjdata, the precise other_error.502 condition for to_bjdata, and reword the note on truncated unsigned integers in exceptions.md (only ND-array elements were truncated before 3.13.0). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Regenerate single_include and apply astyle Result of make amalgamate (including make pretty) for the preceding changes. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address review of the BJData documentation The $ type restriction exists since Draft 2 (Draft 3 added B), $ requires a count, ND char above 127 is rejected on read, and the new API page is listed in alphabetical order in the nav. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: format tables, use the Oxford comma Reformat the tables changed by this PR (BJData serialization and deserialization mappings, completeness and binary-value tables) so all rows are aligned, and add the Oxford comma to lists of three or more items in the added text. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
dc768a7d5d |
Fix BSON binary subtype 0x02 and check the declared document size earlier (#5930)
* Write and read the inner length of BSON binary subtype 0x02 The BSON specification defines the payload of the "binary (old)" subtype 0x02 as an int32 followed by that many bytes. to_bson wrote the plain payload instead, so decoders such as pymongo and libbson-based readers rejected the bytes, and from_bson returned the inner length as part of the payload (documented as lenient behavior in gh-5333). The writer now emits the outer length (payload + 4), the subtype, the inner int32 holding the payload size, and the payload. The reader requires an outer length of at least 4 and an inner length equal to the outer length minus 4 (the cases the bson-corpus lists as decode errors), reports parse error 112 otherwise, and returns only the payload. Both directions change together so that the round trip keeps working. The single-reservation fast path for scalar entries declines subtype 0x02 because it has no room for the inner length. This is a deliberate exception to the opt-in rule for behavior changes: the previous output was invalid BSON. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Check the declared BSON document size before reading its content The BSON reader compared a document's declared size with the number of bytes read only when it reached the terminator. The declared size never bounded the reads, so a SAX handler saw the keys and values of a document whose size was too small before parse error 112 was reported. The container frame now stores the position where the document ends according to its declared size. open_bson_document rejects a size below 5 (size prefix plus terminator) and an embedded document that does not end before the terminator of its enclosing document; the element loop rejects an element whose type byte leaves no room for the name terminator and the document terminator. The terminator check stays and still catches a declared size that is too large. No input that was accepted before is rejected now; documents with a wrong size are reported earlier, with fewer SAX events, a different byte position and, in three cases, a different message. The error id (parse_error.112) is unchanged and strict=false still does not skip the check. The tests that asserted the old messages and positions are updated, with the outer sizes of two inputs corrected so that they still exercise the embedded document. New tests cover the SAX events, a size below 5, and an embedded document that fits exactly. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Document BSON subtype 0x02, document size checks, and lenient reads Describe the inner length of binary subtype 0x02 that to_bson now writes and from_bson checks and removes. Add the document size checks (parse_error.112, not affected by strict=false) to bson.md and from_bson.md, and extend the list of lenient reads with the UUID and MD5 payload lengths (not checked), duplicate keys (last value wins), embedded NUL bytes in strings, and the ignored trailing bytes with strict=false. State that the members of an object are written in iteration order. Correct the statement that element names are never validated on read: they are checked and sanitized like string values by the error handler. Add the BSON message to the examples of out_of_range.415 and type_error.321, the subtype limit of 255 to the binary values page, CBOR to the formats that use byte_container_with_subtype, and fix a typo in the incomplete mapping box. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Regenerate single_include and apply astyle Result of make amalgamate (including make pretty) for the preceding changes. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address review of the BSON documentation Describe exactly when a too small document size is detected and note that subtype 2 data written by earlier versions is not readable. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: use the Oxford comma, drop discarded-value remarks Add the serial comma in a list added by this PR and remove the BSON example of type_error.321 (with the type_error.317 remark) so the docs do not mention discarded values for BSON; restore the original singular heading of that section. No tables were changed by this PR. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix -Weffc++ in the BSON recorder test and a clang-tidy use-auto hit Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix clang-tidy readability-redundant-member-init in BSON test Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
0c3570d44a |
Validate the UBJSON/BJData container type marker for count 0; fix UBJSON docs (#5929)
* Validate the optimized-container type marker for count 0 in UBJSON and BJData
The '$' type marker of an optimized container was only checked when an
element was read, so containers with count 0 accepted any marker:
`[$x#i0`, `{$x#i0` and `[$E#i0` parsed as empty containers although a
non-empty container with the same marker is rejected and the UBJSON
specification requires a legal marker after '$'.
get_ubjson_size_type() now checks the marker right after '$' against the
allowed set (UBJSON: Z N T F i U I l L d D H C S [ {; BJData: B U i u I m
l M L h d D C, with the existing exclusion of T F Z N S H [ {) and reports
parse_error.112 "marker ... is not a permitted optimized array type".
Input that was accepted and is now rejected: an optimized container with
count 0 and an invalid '$' type marker. Valid input is unaffected. Two
existing tests that used an invalid marker incidentally ("[$#" and a
negative-count case) were adapted.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Correct and complete the UBJSON documentation
Fix statements that disagree with the behavior (checked with probes):
- `Z` is null; the unused marker is `N`.
- `$` requires `#`; use_type with use_size=false throws other_error.502 only
for non-empty arrays, objects and binary values.
- NaN and infinity are written as IEEE 754 float64 and read back unchanged;
the UBJSON specification encodes infinity as null.
- `H` is documented (semantics, parse_error.115, out_of_range.406); the
"complete mapping" claim is qualified.
- use_type "same type" means same marker.
- Long Z/T/F arrays lose only the `$` annotation; the `#` count and the
element markers are kept.
- from_ubjson lists the 112, 113, 115, 406 and 408 causes that occur for
UBJSON, without the BJData-only ND-array wording.
- Duplicate object keys keep the last value; the `$` type marker is
validated even for count 0.
- Smaller fixes: float32 row, "map" -> "object", https link, exception 110
strict case, to_ubjson history of discarded values.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Regenerate single_include and apply astyle
Result of make amalgamate (including make pretty) for the preceding changes.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address review of the UBJSON documentation
Clarify that empty containers carry no $ annotation of their own and note that a counted N object is not supported.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: format tables, use the Oxford comma, drop discarded-value remarks
Reformat the two UBJSON mapping tables that this PR changed (uniform column
widths, padded cells), use the serial comma in lists of three or more in the
added text, and restore the original type_error.321 bullet in to_ubjson.md.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix narrowing of the optimized-container count in BJData and UBJSON tests
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
|
||
|
Niels Lohmann
|
d5da84249f |
Skip CBOR tags in front of map keys, validate indefinite text chunks, fix CBOR docs (#5927)
* Skip CBOR tags in front of map keys and validate indefinite text chunks A tag in front of a text string map key was rejected with parse_error.113 even with cbor_tag_handler_t::ignore or store, although tags in front of values are handled. The tags are now skipped iteratively before the key is read; the handler error still rejects them. The chunks of an indefinite-length text string were concatenated before the single UTF-8 check, so a code point split over two chunks, or an invalid chunk, was accepted. RFC 8949, Section 3.2.3 requires every chunk to be valid on its own. With error_handler_t::strict, replace and ignore each chunk is now checked; keep is unchanged. The chunks are appended with append(ptr, len), so StringType needs nothing new. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: correct and extend the CBOR documentation Fix the negative-integer rows (they used MessagePack boundaries), the number_integer range of the 0x1B row, and the half-precision entry in the list of unused types; correct the NaN history (0xF9 since 3.8.0) and the tagged-item wording (only the tag directly in front of a byte string is stored, and only in the 0xD8..0xDB form). Document that to_cbor is not RFC 8949 deterministic, that duplicate keys are accepted (last wins), tags in front of keys, per-chunk UTF-8 validation, the exceptions parse_error.113, out_of_range.406/408 of from_cbor, the pointer and C array input traps, the partially written output on exceptions, and the store/ignore results in the binary values example. Drop the claim that the CBOR reader needs StringType::append(const StringType&) and BinaryType::insert(pos, first, last). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Regenerate single_include and apply astyle Result of make amalgamate (including make pretty) for the preceding changes. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address review of the CBOR changes Cite RFC 8949 Section 5.3.1 for duplicate keys, describe the out_of_range.408 condition precisely, rewrap a long line, and update the stale get_string comment (per-chunk UTF-8 check). single_include regenerated. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: drop discarded-value remarks Remove the mention that a discarded value cannot be serialized to CBOR (type_error.321) from the exception-safety and mapping sections. The integer table and all added prose already follow the table-format and Oxford-comma rules. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
81d66291c5 |
Create objects for pointer tokens that cannot be array indices (#5358)
* create objects for tokens that cannot be array indices (#5357) When operator[](json_pointer) traverses a level that does not exist yet, the null value is turned into an array or an object depending on the reference token. The check only tested whether all characters are digits, so tokens that can never be a valid array index selected an array and then failed: - "01" (and any other token with a leading '0') threw parse_error.106 - the empty token threw out_of_range.404 Both tokens are valid object keys, and both work when the level already exists as an object, so creating the level changed the outcome. Test the token against the RFC 6901, Sect. 4 grammar for array indices instead, so that such tokens create an object. This only affects pointers that threw before. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Clarify that the token 0 is a valid array index in operator[] docs The note described valid array indices as "a nonempty sequence of digits without a leading 0", which read literally excludes the token "0" itself, contradicting both the code and the /foo/0/0/0 example in the next sentence. Spell out the RFC 6901 grammar instead. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
 Niels LohmannandClaude Sonnet 5.5
|
d6431a9104 |
Throw out_of_range.411 for JSON Patch add into a null parent (#5917) (#5918)
Claude-Session: https://claude.ai/code/session_01MKJf48vnGtTE25KscmjfaW Signed-off-by: Niels Lohmann <niels.lohmann@gmail.com> Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com> |
||
|
Niels Lohmann
|
2a56afc714 |
Fix VS 2015 build of the DOM parser (#5882) (#5916)
- Name std::hash only once the object type has emplace_new_key: VS 2015 static-asserts on std::hash of a type without one (alt_string) instead of reporting a substitution failure (C2338). - Replace the std::copy calls on frames by loops: VS 2015 reports raw pointers handed to std::copy as C4996, an error under /WX. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
3eae5680d6 |
from_json: number-first dispatch for arithmetic types, loop for std::array (#5902)
* from_json: number-first dispatch for arithmetic types, loop for std::array (#5891) from_json(ArithmeticType&) now reads numbers through get_number_value() and handles a boolean on a cold path, which puts the three number types first in the per-element type dispatch. get<std::array<T, N>>() always took the identity_tag overload, a pack expansion of N calls j.at(i).get<T>() that clang unrolls into straight-line code (about 28,000 instructions for N = 1000), so its speed depended on inlining decisions in that body. For trivial element types, which cannot tell default construction plus assignment from construction in place, it now fills a std::array in a loop. That loop, shared with get_to(), checks the size once and reads the inner array directly; a short array still converts the elements it has and then throws the out_of_range error at() would throw. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Skip the get_to result check without exceptions Without exceptions, the get_to calls in CHECK_THROWS_WITH_AS do not run, so the array keeps its initial values (ci_test_noexceptions). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Make is_trivial_element a struct so astyle does not indent it Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Expect the element path in the std::array type-error tests with JSON_DIAGNOSTICS Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Shorten from_json_inplace_array so astyle aligns its parameters The opening parenthesis of the previous name sat in column 41, beyond astyle's default --max-continuation-indent of 40, so the continuation lines fell back to a plain double indent. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Use the compact template<...> style for the new from_json overloads Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
fe5e401870 |
Consolidate duplicated detail helpers and bodies (#5891) (#5901)
* Add shared detail helpers for the #5891 cleanup Add detail::is_primitive(value_t), detail::hex4_lower(), detail::utf8_shape(), detail::input_format_name(), detail::empty_input_message(), the exception ids 116, 319, 320 and 416 to detail::exception_id, and make detail::is_ascii_copyable() constexpr. Later commits switch the duplicated code over to them. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 1.4: BSON writers share one write-pass UTF-8 sanitizer (#5891) write_bson_entry_header and write_bson_string re-derived the error-handler policy differently. Both now use sanitize_utf8_for_bson_write, which treats strict like keep (the size pass already validated) and sanitizes replace/ignore into the bytes the size pass measured. No context plumbing is needed. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.1.1: one MessagePack container prefix helper (#5891) Array and object prefixes differed only in the fix marker and the 16/32-bit markers; both now delegate to write_msgpack_container_prefix. Output is byte-identical. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.1.2: one iterative walker for CBOR and MessagePack (#5891) write_cbor_iterative and write_msgpack_iterative (and their *_value_or_push helpers) were identical but for the head, scalar, and key writers. They are now one template over a small policy struct, so everything inlines. BON8 keeps its own walker (open string run, closing marker), noted in a comment. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.1.3, 2.1.8: one UBJSON container start helper, shared frame base (#5891) write_ubjson_start_array/_object are now write_ubjson_start_container<IsArray>, keeping the array-only begin()+1 scan and excessive_valueless rule. ubjson_frame derives from binary_container_frame instead of duplicating it. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.1.4: write_bytes helper for the byte-string write sites (#5891) The 13 oa.write_characters(reinterpret_cast<const CharType*>(x.data()), n) sites now call write_bytes(const void*, size_t); copy_characters sites and the buffer-based writes are untouched. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.1.5: shared object-key writer for CBOR and MessagePack (#5891) write_cbor_key and write_msgpack_key were the same SFINAE pair apart from the string/scalar writers; both now delegate to write_object_key<Walk>. UBJSON key and to_format_length stay as they are. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.3.10: writer-side format names from input_format_name (#5891) The CBOR/MessagePack/UBJSON/BSON/BJData/BON8 literals in throw_on_discarded and throw_integer_out_of_range calls now come from detail::input_format_name; the texts are identical, and UBJSON vs BJData still follows use_bjdata. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * serializer: constexpr ascii table, hex4_lower, dedupe helpers (#5891) 1.1: build ascii_copyable() at compile time from is_ascii_copyable and test find_ascii_copyable_run against it for every byte. 1.2/2.5.4: use detail::hex4_lower in both \uXXXX writers. 2.1.6: factor the repeated flush/emit blocks in dump_escaped_impl into always-inline helpers. 2.1.7: comment why the escape copy does not use put_buffer. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 1.6: use string_t-aware to_string/concat in meta() (#5891) Replaces the four std::to_string sites; rendered text unchanged. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 1.7: use detail::is_primitive in json.hpp switches (#5891) has_no_children, set_parents, max_size and the two diff walkers. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.2.1: factor insert()/iterator throw helpers (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.2.2: share swap() overloads via swap_impl (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.2.4: factor primitive reset and range check in erase()/range ctor (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.2.6: share push_back array bodies via push_back_array_impl (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.2.7: route parse/accept/sax_parse through *_impl helpers (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.2.8: share the vector-returning to_* bodies via to_binary_vector (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.2.10: merge destroy_string/destroy_binary into destroy_pointer (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.4.1/1.7: exceptions: share what_impl, flatten diagnostics switch (#5891) The four create() bodies build their message through one protected static helper; the diagnostics parent walk handles array/object and ignores all other types. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.4.2/2.4.3/1.7: json_pointer: share slash and escape checks, drop primitive case lists (#5891) The leading-slash and invalid '~' escape checks are shared helpers; the reference-token walks test object and array first and treat everything else as primitive. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 1.7/2.4.9: iter_impl, iteration_proxy: drop primitive case lists, share compare preamble (#5891) Object and array stay the first branches; everything else is the primitive case. operator== and operator< share the different-container check. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.4.5/2.4.6: from_json: share get_number_value and reserve_if_possible (#5891) The three number cases of get_arithmetic_value and the arithmetic from_json share one helper (boolean stays accepted only by the latter); the array and object reserve overload pairs become one. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.4.8: ordered_map: share erase_impl between the key erase overloads (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 1.7: json_pointer: flatten walk drops the primitive case list (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 1.2/1.3/1.5/2.5.1/2.5.2/2.5.5: share helpers in the view code (#5891) - detail::exception_id at every numeric id; throw_* helpers take it - hex_byte, hex4_lower, utf8_shape replace local hex/table code - throw_array_index_out_of_range/_past_end, throw_key_not_found - has_native_float_format replaces the hand-written IEEE check No change of messages, ids or output. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 1.5: use utf8_shape in get_bon8_string, add exhaustive UTF-8 validator test (#5891) get_bon8_string takes the valid range of the second byte from utf8_shape() (0xF5..0xF7 stay errors, unget behavior unchanged). validate_one_utf8 keeps its if-chain (hot bulk path); the new test checks decode, validate_one_utf8, utf8_shape, json::accept and the BON8 reader against each other for every lead and second byte. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.3.1: get_cbor_sized_bytes for the CBOR chunk getters (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.3.2: get_string_with_length for the plain-order string sites (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.3.3, 2.3.4, 2.3.10: unsupported_key_type, empty_input_message, input_format_name in the readers (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.3.7, 2.3.9: read_object_key and leave_container_and_check for CBOR/MessagePack/BON8 (#5891) UBJSON keys (different getter signature) and its container tail stay as is. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.3.5: json_sax_dom_error_state base for the DOM SAX parsers (#5891) Holds errored, allow_exceptions, parse_error() and is_errored(). The scalar handlers stay in each class (parse hot path). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.3.6: table of control-character messages in the lexer (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.3.8: finish_input helper for the strict-EOF block in the parser (#5891) sax_parse keeps its if (result) guard and return value. Also adds the clang-tidy NOLINT for the 2.3.6 message table. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.5.7: share parse_array_index between json_pointer and json_view (#5891) Move json_pointer::parse_array_index() and its status enum to namespace detail (json_pointer keeps a forwarding member), and let the json_view pointer resolution use it instead of its own copy of the RFC 6901 index grammar. Both accepted the same tokens with the same outcome. Also update the comment on detail::decode(), which listed four of the five UTF-8 validators and now points to the test that keeps them in sync. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Amalgamate (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.4.5: keep the inlined number cases in from_json(ArithmeticType&) (#5891) Calling get_number_value() from from_json(ArithmeticType&) made GetArray<std::array<int, 1000>> 2.4x slower (it changes the inlining of the std::array conversion), so that overload keeps its own switch, with a comment. get_arithmetic_value() still uses the shared helper. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.5.7: force-inline detail::parse_array_index (#5891) Moving parse_array_index out of json_pointer stopped clang from inlining it into the pointer resolution (JsonPointerAt/citm_catalog +17%, JsonPointerAt/twitter +8%); JSON_HEDLEY_ALWAYS_INLINE restores the baseline. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 1.5: keep get_bon8_string's own UTF-8 ranges (#5891) Taking the second-byte range from utf8_shape() made BON8 decoding 2-3% slower (FromBinaryShape/scalars, FromBinaryBuffer, alignment-pinned builds), so the byte path keeps its hand-written ranges. The exhaustive validator test still checks that it accepts the same sequences as the other validators. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Amalgamate (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix CI failures: constexpr, NRVO, cppcheck, IWYU (#5891) - json_sax_dom_error_state::is_errored() cannot be constexpr: the class is not a literal type (GCC 4.8, GCC -Wpedantic, Clang 3.5). - to_binary_vector: destroy the vector writer before returning, as the output_vector_sink sets the size of the vector in its destructor. Without NRVO (MSVC Debug in C++14 mode), the vector was moved out before it was resized. - parser::sax_parse: return false instead of result after the early exit (cppcheck identicalConditionAfterEarlyExit). - json.hpp: export detail/exceptions.hpp for IWYU, so json_view.hpp can name exception_id without including a detail header. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address review: throw helpers take input_format_t, simpler type checks - throw_on_discarded() and throw_integer_out_of_range() take an input_format_t and call input_format_name() themselves - has_no_children(), set_parents() and max_size() check array, then object, instead of calling is_primitive() first - parse_impl() chains like accept_impl(); sax_parse_impl() uses if/else - std::array for the control character messages; negation<> in write_object_key(); clarify the is_primitive() comment Signed-off-by: Niels Lohmann <mail@nlohmann.me> * json_view: use exception_id in the value() overloads Signed-off-by: Niels Lohmann <mail@nlohmann.me> * json_sax: assert the invariant of place_container for the static analyzer Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
9c50bdd140 |
Make the DOM parser cheaper per container and per member (#5882)
* Reserve the parser's nesting stack once The stack of open arrays and objects was a std::vector<std::uint8_t> that allocated at the first push_back and then reallocated at 1, 2, 4, 8 bytes, so a document nested four levels deep made four allocations before it parsed a value. A document that starts with a container now reserves 32 levels in one allocation; deeper documents grow as before. The loop that reads the tokens is the code it was, so large documents are not affected (a stack object with its own storage, and a reserve at every container, both cost the token loop instructions or cycles). Fixed cost of parsing a 124-byte object (macOS instructions): accept -7.6 %, DOM -2.5 %; no change for a single number. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Build arrays of the DOM parser in one exact-size allocation Reading an array of unknown length appended its elements to a std::vector one at a time, so a small array reallocated (and moved its elements) at sizes 1, 2, 4, ... The DOM SAX parser now collects the elements of arrays whose length is unknown on one stack shared by all open arrays and moves them into an array of exactly the right size when the closing bracket arrives. An array that grows past 16 elements switches to appending to its own vector, so large arrays pay no second copy. Binary formats, which announce the length, and builds with JSON_DIAGNOSTICS or JSON_DIAGNOSTIC_POSITIONS (whose values need their final place from the start) keep storing into the array directly. The stacks of open containers are reserved on first use instead of growing from one element. The binary readers get a DOM parser of their own, json_sax_dom_binary_parser: it is the plain parser that stores every value directly in its container (the code the readers used before). Their containers announce their size, so the bookkeeping of the collecting parser (frame entries, the destination of a finished container) was pure overhead for them: with the collecting parser FromBinaryShape nested was 3-12 % and containers up to 5 % more instructions (GCC) than before. With the plain parser Linux Ir of FromBinaryShape and FromBinaryBuffer is within 0.8 % of develop (Clang: identical). canada.json has 56 000 two-element arrays that took three allocations each and now take two. DOM parse Ir on Linux against develop (GCC / Clang): canada -7.0 % / -7.7 %, citm -0.8 % / -1.6 %, 124-byte object -5.4 % / -5.5 %; accept and SAX are untouched. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Reuse one value for the key events of the callback parser parse() with a callback built a new string value for every key it passed to the callback (a heap allocation and release per key), and copied the key a second time into the object. The callback parser now keeps one string value for key events, which it assigns, and moves the key into the object. Callback parse, Linux Ir (GCC / Clang): twitter -13.3 % / -11.7 %, citm -9.5 % / -7.6 %. (Earlier macOS measurement: twitter -19.5 % instructions.) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Remember where the DOM parser stores the next value The DOM SAX parser looked up the innermost open container for every value: the end of its frame stack, then the object or array pointer in the frame. Compilers have to reload all of that after each value is stored, since a store of a value may alias the frame. The parser now keeps the innermost container's kind and, for an array that stores directly, the array itself in members, updated when a container opens or closes. Flat arrays of numbers are the case that mattered: parsing floats.json (1 000 000 doubles) into a DOM took 2 % more cycles than before the previous commit on macOS and now takes 0.4 % fewer than the code before it (signed_ints.json -0.8 %). Linux Ir is unchanged (canada -6.1 % GCC / -6.9 % Clang, citm -0.3 % / -0.8 %, twitter +0.2 % / 0.0 %). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Look up the keys of big ordered_json objects in a hash table ordered_map keeps its elements in a vector and searches it linearly for every key it adds, so parsing an object with n keys into ordered_json took O(n^2) key comparisons: 33 billion instructions for one object with 30 000 keys, where json needs 245 million. From 64 keys on, the DOM parser now keeps a hash table of the keys of the open object (hashes and positions only; the elements stay in the vector) and finds duplicate keys there. The result is the same, including the position of a key that occurs twice. Smaller objects, std::map based objects and key types without std::hash are untouched. ordered_json parse, macOS instructions: object with 1 000 keys 64.8 M -> 14.6 M, 10 000 keys 5 104 M -> 38.7 M, 30 000 keys 33 358 M -> 89 M. Linux Ir of an object with 2 000 keys: 43.0 M -> 1.9 M (GCC), 37.9 M -> 1.8 M (Clang). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Trim the work the DOM parser does per container and per member Opening and closing an array or object cost the DOM parser more than before the exact-size array change, which showed on documents of many small objects (jeopardy.json: +1.4 % Ir with GCC). This takes it back: - the stack of open containers keeps its first eight entries in the parser object instead of in a vector, so it needs no capacity check beyond the push itself and no allocation for typical documents; - a container is created in the place it ends up in, instead of as a temporary that is then moved there; - a new object member, which is null, is constructed over the null instead of being swapped with it (only with the default base class of basic_json); - the key index that big ordered_json objects use is only a part of the stack entries of objects that can have one; - an array that outgrows the stack reserves twice what it has so far, as a vector that grew by itself would have. DOM parse Ir of the series up to here against develop, Linux GCC / Clang: canada -6.6 % / -7.3 %, citm +1.2 % / +0.2 %, jeopardy +0.4 % / -0.1 %, 124-byte object -6.5 % / -6.2 %. (twitter measures +6.8 % / +6.6 % at this point in steady state, almost all of it in glibc's malloc/consolidate for the new allocation sizes; with the last commit of the series it is -0.1 % / 0.0 %.) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Test parsing containers of the sizes where the DOM parser changes storage The DOM parser collects the elements of an array on a stack (up to 16) and the members of an ordered_json object (up to 32) before it moves them into the container, and uses a hash table for keys from 64 members on. Parse arrays and objects of the sizes around these limits, nested into each other, with repeated keys, and as random trees that are built next to their text, and compare json and ordered_json against the value built through the API. The tests pass on the code from before these changes as well. The random trees are what finds the mistakes that are easy to make here: while the collecting change was being written, a collected array that outgrew the stack at the moment its parent had just reached the limit left its elements in the parent, and only these tests (142 failed assertions) showed it. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Do not convert floats that cannot overflow in accept() accept() only has to find out whether a float token is finite: json_sax_acceptor ignores the value, and the parser rejects only a float that overflows. A float whose digits fit into 19 digits and whose decimal exponent is at most max_exponent10 - 19 is below 10^max_exponent10 and cannot overflow, so the lexer, told by accept() that the value is not needed, skips the conversion for those. A token that may overflow, with more digits, or in another float format is converted as before, so 1e309 and its neighbours are rejected as they were (the new test compares accept() with parse() on tokens around the limit). json::accept(): canada.json Ir -8.6 % (GCC) / -10.9 % (Clang), 124-byte object -3.2 %; cycles on the M1 (release benchmarks, LTO): canada -7.8 %, floats.json -9.0 %. The check tests the accept() flag first, so the parser that needs the value pays one load and branch: SaxParse canada +0.35 % Ir (GCC and Clang), DOM canada +0.4 %. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Collect the members of ordered_json objects on a stack until they are known An ordered_json object keeps its members in a vector and appends every new member to it, so reading an object of unknown size reallocated the vector at 1, 2, 4, 8, ... members, and ordered_map copies all keys on every reallocation. The DOM SAX parser now collects the members of such objects on one stack shared by all open objects, as it does the elements of arrays, and moves them into a vector of exactly the right size when the closing brace arrives. An object that gets past 32 members switches to appending to its own vector (and, from 64, to the key index of the earlier commit). The members are the same, in the same order, and a key that occurs twice keeps the place of its first occurrence. std::map based objects, objects whose size the input announces (binary formats), and builds with JSON_DIAGNOSTICS or JSON_DIAGNOSTIC_POSITIONS are untouched. What only ordered objects need (the collected members, the key indexes) is a member of the parser only for them, and the code that stores a value tries the array that takes its elements directly first. An array that outgrows the stack now reserves 32 elements, the capacity a vector that grew by itself has at that point; with the larger reserve before, parsing floats.json (1 000 000 numbers) was 2 % slower on Apple M1 than before this series. ordered_json parse, Linux Ir against develop (GCC / Clang): citm -11.6 % / -11.7 %, twitter -8.4 % / -8.4 %, jeopardy (12 000 objects of 7 keys) -10.5 % / -8.0 %, canada -3.9 % / -5.2 %, 124-byte object -20.9 % / -16.5 %. macOS time (M1, earlier base): citm -19 %, twitter -13 %, jeopardy -13 %, canada -14 %, 124-byte object -26 %. json parse: steady-state twitter -0.1 % (see the previous commits), citm -1.2 %. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Rename the key index's match callback so Flawfinder doesn't read it as std::equal Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Bind the parser test's strings by reference in the range-for loop GCC and Clang 14 (-Wrange-loop-construct) reported the copy. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Collect container values only in basic_json's own parse functions A json_sax_dom_parser that a user constructs (as in the sax_no_exception example) must hold the values read before a parse error, as on develop: its value is used after the error. The values of containers of unknown size are now collected on a stack only with the new template parameter Collect (default false), which json::parse() and the json_view materialization set; without it, every value is stored in place as it arrives. A test checks the values kept after an error for json and ordered_json. Also fixes the clang-tidy, cpplint and Infer findings: the frame stack keeps its first frames in a std::array (left uninitialized) and the others in a std::unique_ptr, auto for create<>(), NOLINT for member initializers -Weffc++ needs, and an @infer-ignore for the nullptr that place_container() does not use. json::parse() and ordered_json::parse() instructions are unchanged (Apple clang; twitter, citm, canada, jeopardy: -0.03 % to +0.25 %). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Clarify the DOM parser's collection stacks and rename reenter_enclosing_container() Address review comments: explain why a closed container erases only the top of the shared stack, set the payload before the type, rename the helper that makes the enclosing container current again, and declare the parser's state stack right before reserving it. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Export json_sax_dom_parser from the C++20 module for MSVC MSVC needs the detail classes that basic_json befriends to be visible in the module (#3970, as json_sax_dom_callback_parser); with its new template parameter, json_sax_dom_parser failed in ci_module_cpp20 (C2039, not a member of nlohmann::detail). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Keep the overflow-limit numbers of the parser test in a std::vector clang-tidy (cppcoreguidelines-avoid-c-arrays) rejected the C array. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix the clang-tidy findings of the ordered_json parser tests The container sizes are a std::array, the expected text uses raw string literals, the generator's arithmetic is in parentheses, auto for the result of make<>(), and single characters are appended as characters. clang-tidy now reports nothing on the lines the branch adds to unit-ordered_json.cpp and unit-class_parser.cpp (apart from checks CI's clang-tidy does not have). Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
0b924f5375 |
Split and print JSON pointers in one pass (#5887)
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
6fa14d8f95 |
Write binary formats through reserved sink space and number runs (#5885)
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
dec0e9b749 |
Write strings, scalars and indentation in dump() with fewer calls (#5883)
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
86d5c56c54 |
Speed up flatten(), unflatten(), diff() and patch() further (#5886)
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
aa0b371f7c |
Faster std::istream input: inline get-area path, binary reads, unbuffered streambufs (#5884)
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
9855db81f3 |
Silence macro-to-enum for the endianness macros (#5881)
#5876 added JSON_LITTLE_ENDIAN_TARGET and JSON_BIG_ENDIAN_TARGET to macro_scope.hpp. They must stay macros because they select code in #if, so suppress the clang-tidy check as simd_scan.hpp already does. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
9bf531436f |
Speed up flatten(), unflatten(), diff() and patch() (#5861)
* Flatten into a std::map by inserting next to the previous entry flatten() added every path with operator[], so each insertion into the std::map of the result searched the tree from the root and compared long shared path prefixes, which took half the time on canada.json. Paths come out in nearly sorted order, so they are now inserted with emplace_hint next to the entry added last, for object types that have it. ordered_map keeps its append without lookup, and other object types keep operator[]. Path segments are also appended in place, without a temporary string for each escaped key or array index. flatten() of canada.json, citm_catalog.json and twitter.json is 1.4x to 1.9x as fast; ordered_json is unchanged (1.05x). The output is unchanged. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Unflatten without splitting keys or walking each path from the root unflatten() split every key into a vector of reference tokens, stored the pointer prefixes in a tree to decide which values become arrays, and then looked up every token of every key from the root again. Keys are now checked by a scan that raises the same parse_error.107/108 as split(), skipping the part a key shares with the previous one, which was checked already. The prefixes that denote arrays are kept in a small hash set of key slices; the array/object decision is unchanged. The walk keeps the values along the previous key's path, so only the part of a key that differs from the previous one is looked up. Keys of a flattened object are usually sorted, so that part is short. unflatten() of canada.json, citm_catalog.json and twitter.json is 1.9x to 2.3x as fast; the output and the exceptions are unchanged. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Apply plain JSON Patch operations without building a json_pointer patch() built a json_pointer for every operation, which splits the path into a vector of strings, and for "add" and "remove" a second one for the parent. That took 20% to 45% of the time not spent copying the document. "add", "remove", "replace" and "test" with a path that has no escapes and denotes existing values (the parent, for "add"), as diff() creates them, are now applied by resolving the path in place, reusing one string for the reference tokens. Every other operation, and every operation that would fail, takes the existing code, so the results, the exceptions and the state left after a failing operation are unchanged. patch() of the diffs of canada.json, citm_catalog.json and twitter.json is 1.02x to 1.10x as fast; the rest is the copy of the document. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Build diff operations member by member diff() created every operation from an initializer list, so each {"op", "replace"} pair became a temporary array with two heap strings before it was turned into an object member; an operation took about 25 allocations. The operations are now built as objects whose members are assigned one by one. Paths of members and elements are also built in place, without a temporary string for each escaped key or array index. diff() of canada.json, citm_catalog.json and twitter.json, with every eighth scalar changed, is 1.3x to 1.6x as fast; the output is unchanged. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Diff objects with the same keys in lockstep and keep one path buffer diff() looked up every key of the source object in the target object, and built a new string for the path of every value it descended into. Objects of the same size usually have the same keys in the same order, so they are now walked together as long as that holds, without a lookup; objects that differ fall back to the lookup from the first key that does not match. The path is one buffer that is extended before descending and shortened again afterwards. Objects whose members are equal but in a different order (possible with ordered_json) still get no operations. Together with the previous commit, diff() of canada.json, citm_catalog.json and twitter.json is 1.5x to 2.0x as fast; ordered_json 1.65x to 1.85x. The output is unchanged. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Use std::array for the index buffer in append_index() Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Declare the partner iterator in the loop initializer in diff_object_keys() Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix GCC and clang-tidy warnings in the JSON pointer speedups Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
3e031b06a5 |
Use standard library helpers where available, with C++11 fallbacks (#5876)
* Use standard library helpers where available, with C++11 fallbacks Add detail helpers that call the standard library function when the library provides it (checked with the __cpp_lib_* feature-test macros, so no new ABI-tag macros) and a C++11 implementation otherwise: - detail::clamp (std::clamp): by value, so that static constexpr members need no out-of-line definition before C++17; used in json_view's dump() buffer sizing, where step() can now be a static constexpr member. - detail::bit_cast (std::bit_cast): replaces dtoa_impl::reinterpret_bits and the memcpy bit copies in the number parser, to_chars, the json_view serializer, and binary_writer::to_char_type. - detail::byteswap (std::byteswap): one implementation instead of the binary writer's own byte_swap() overloads and the bswap chain in to_chars' store_msb_first(). - count_leading_zeros/count_trailing_zeros: two 32-bit _BitScanReverse/ _BitScanForward calls for MSVC x86/ARM32 instead of the loop, and std::countl_zero/countr_zero for compilers without builtins. - conjunction, disjunction, negation, bool_constant, and uncvref_t map to their std counterparts in C++17/C++20. Builtins and intrinsics stay ahead of the <bit> functions, as MSVC's std::countl_zero checks for LZCNT at run time unless AVX2 is enabled. The generated code is unchanged with Clang (C++11/20/23, arm64 and x86-64) and GCC 16 for parsing, dumping, and the binary formats. The binary reader keeps its own byte_swap() and the writer its memcpy calls: the smaller code from detail::byteswap/std::bit_cast there made GCC 16 inline get_number()/write_number() differently, which slowed reading by 10-18 % and writing by about 1 %. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Name the target byte order once (JSON_LITTLE_ENDIAN_TARGET, JSON_BIG_ENDIAN_TARGET) to_chars.hpp and bit_ops.hpp repeated the long __BYTE_ORDER__/_MSC_VER conditions; macro_scope.hpp now defines them once (undefined again in macro_unscope.hpp). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Explain why the serializer growth bound is not a clamp Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
6aefd8c662 |
Copy short printable-ASCII strings without starting the scanner (#5878)
dump() now checks strings of up to 32 bytes against a table of the bytes that can be written verbatim, four bytes per step, and copies them in one go if all of them can. That is cheaper than setting up the vector scanner, whose setup costs more than such a string takes to scan; SQLite's jsonAppendString() writes strings with the same kind of table loop. Most object keys and many values are this short (citm_catalog averages 8 bytes per string, jeopardy 14). GCC 16 and 13 stopped inlining put_indent() and put_literal() into the pretty-printing loops once dump_escaped() had grown, which made indented output up to 13% slower; both are now always inlined. The new test puts every ASCII character and a 2-byte character at every position of strings of 1 to 40 bytes and compares the output with the reference implementation of the existing UTF-8 tests. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
52fecc1fab |
Move object keys into the value when parsing (#5855)
json_sax_dom_parser::key() copied the key from the lexer's buffer into the object. The lexer starts every token with an empty buffer (reset()), so the key can be moved instead, which saves an allocation and a copy for every key that does not fit into the string's inline buffer. json_sax_dom_parser already moved string values the same way. ordered_map has no rvalue operator[], so ordered_json still copies keys. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
fa37b952df |
Match literals without a stack array and keep parser states in bytes (#5854)
* Match literals without a stack array and keep parser states in bytes scan_after_whitespace() built the expected text of true, false, and null in local std::array objects. With -fstack-protector-strong (the default of Ubuntu's GCC, among others), a local array gives the function a stack canary check, which then ran on every token. scan_literal() now takes the text after the first character as a string literal, with its length as a template parameter; for contiguous input it compares the rest at once and falls back to the byte loop on a mismatch, so errors are reported at the same character. It is out of line, which keeps the token dispatch small. The parser kept the nesting of arrays and objects in a std::vector<bool>, whose push_back() GCC calls out of line; it is a std::vector<std::uint8_t> now. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Skip the literal fast path test without exceptions The test case compares the error messages of both lexer paths, so it does not compile with JSON_NOEXCEPTION (the caught exception does not exist). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Allow the string literal parameter of scan_literal() for clang-tidy The parameter takes the literal by array reference so that its length is a template parameter; this is the same exemption the C array overloads of to_json() and from_json() have. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
e9970066c5 |
Scan strings with SSE2, SSSE3, or NEON (#5853)
* Scan strings with SSE2, SSSE3, or NEON The string fast path found the end of a run of plain characters 8 bytes at a time (SWAR) and validated non-ASCII text one sequence at a time, calling validate_one_utf8() out of line for each. The vector code of json_view's scanner, in the new shared header detail/input/simd_scan.hpp, now does both 16 bytes at a time: SSE2 finds the end of a run on x86-64, and the UTF-8 check of Keiser and Lemire runs with NEON on AArch64 and with SSSE3 on x86-64. SSSE3 is not part of x86-64, so the check is compiled for it with a function attribute and used when the CPU has it (cached in a constant-initialized atomic), else the scalar check; code compiled with -mssse3 skips the CPU check. The first word of a run is still checked with SWAR (the first four with NEON, where a vector mask costs more), as most keys and short values end there, and the scalar helpers are always inlined. On anything else, the scalar code is unchanged. The new macro JSON_NO_SIMD selects the portable code everywhere, including the SSE2/NEON digit formatting of the floating-point writer. The lexer tests now also run with JSON_NO_SIMD and, on x86-64, with -mssse3, and a new test places every byte that ends a run at each offset across several 16-byte blocks, after ASCII and after CJK text. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Keep the string scan from slowing down short strings and dump() on Arm Three benchmarks were 7-9% slower than develop on Apple M1 (Apple Clang, -O3 -flto): Accept/jeopardy, Dump/citm_catalog, and Dump/jeopardy. - dump(): the serializer passes a string without its closing quote, so plain_run_end() ran out of input and called the out-of-line NEON loop for every string, with fewer than 16 bytes left. The loop is now only called for 16 bytes or more. - accept(): no string of jeopardy.json reaches the vector UTF-8 check, yet compiling it into the lexer's scan function made it 5% slower: the function runs once per token and now saved more registers and kept a stack slot. lexer_string_bulk_run() checks the first four words inline and continues out of line, and the serializer keeps string_bulk_run() inline, which is faster there. - The byte that ends a run is read from the scanned buffer instead of through get(), which Apple Clang merged with the other get() calls that load the read position again right after it was stored (3%). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Define string_bulk_run_from() only where lexer_string_bulk_run() calls it With NEON and simdutf in C++17, lexer_string_bulk_run() takes the string_bulk_run() path, so the out-of-line helper was unused. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Do not dispatch the SSSE3 string scan with GCC's C++20 modules GCC does not keep the target("ssse3") attribute on functions it reads back from a module interface, so the module build (ci_module_cpp20, gcc:latest) rejected the intrinsics in vector_utf8_run() with "needs isa option -mssse3". With __cpp_modules, GCC now uses the portable UTF-8 check like other compilers without the target attribute; the SSE2 scan stays, and -mssse3 or more still compiles the SSSE3 check directly. Clang is not affected. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Include <cpuid.h> only once when json.hpp and json_view.hpp are both used The SSSE3 dispatch of the string scan and the one of json_view each include <cpuid.h>. Before GCC 11 the header has no include guard, so a translation unit that includes both headers fails with GCC 10: "redefinition of 'unsigned int __get_cpuid_max(unsigned int, unsigned int*)'". Guard both includes with NLOHMANN_JSON_CPUID_INCLUDED. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Add an ABI tag for JSON_NO_SIMD JSON_NO_SIMD selects between two definitions of the internal scanning and formatting functions. Like JSON_VIEW_NO_SIMD (_vns), it now appends _ns to the inline namespace, so translation units built with and without it can be linked into one program without ODR violations. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
5e13f834c1 |
Copy the text of float tokens only when a SAX handler asks for it (#5852)
After the digits are converted while scanning, copying the token text into token_buffer (an out-of-line std::string::append) was about 10% of parsing canada.json. The only reader is the string argument of json_sax::number_float(), and the library's own handlers (json_sax_dom_parser, json_sax_dom_callback_parser, json_sax_acceptor) ignore it. For contiguous input, the fast path now keeps a pointer to the token in the input, and get_string() copies it on first use. The parser passes an empty string to the handlers marked by the new internal trait sax_ignores_number_text instead of calling get_string(), so json::parse() and json::accept() never copy float text. User handlers, and JSON_DIAGNOSTIC_POSITIONS (which reads get_string().size()), still see the exact text. Keys and strings, which never have pending text, use the unchecked get_string_token(). Input read through a window into a stream (std::istream, FILE*) copies the text right away, because the next read may refill the window. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
e13e976801 |
Accumulate number digits while scanning them (#5851)
* Accumulate number digits while scanning them The contiguous number fast path validated a token byte by byte, copied it into token_buffer, and then converted it in out-of-line functions that scanned the digits a second time: parse_float_native() called parse_float_significand(), which returned its result through memory and called std::any_of (out of line with GCC) twice per number, and parse_integer_signed() divided by ten on every digit for its overflow check. scan_number_bulk_contiguous() now accumulates the significand while it validates the digits, eight at a time where at least eight bytes remain (is_eight_digits() and parse_eight_digits(), after fast_float). Floats with at most 19 digits go straight to decimal_to_float(); integers with at most 18 digits are stored without an overflow check (a narrower number type still rejects values that do not round-trip). Longer tokens take the existing path. has_nonzero_digit() is a plain loop and parse_float_significand() is always inlined, which helps that path. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Skip the digit accumulation checks without exceptions The section compares error messages from both lexer paths and parses out-of-range numbers, so it does not compile (the caught exception does not exist) and would abort with JSON_NOEXCEPTION. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Share the significant-digit loop of the integer part and the fraction Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
bd5227c2ec |
Fix silent wrong results for custom number types (#5805)
* Fix silent wrong results for custom number types
Squashed onto develop
|
||
|
Niels Lohmann
|
0cbe95ffa2 |
Speed up writing CBOR and MessagePack, above all integer arrays (#5824)
* Speed up writing CBOR and MessagePack, above all integer arrays
Squashed onto develop
|
||
|
Niels Lohmann
|
08d7fbb0a1 |
Read FILE* input with std::fgetc under MemorySanitizer (#5874)
With glibc and optimization, <stdio.h> turns getc_unlocked into an extern inline function that reads the stdio buffer (fp->_IO_read_ptr) directly. glibc, which is not instrumented, fills that buffer, so MemorySanitizer treats the bytes as uninitialized and reports a false positive when the parsers read from a FILE* (e.g., the fuzz harness FILE* leg, OSS-Fuzz 575915982). std::fgetc is an opaque call whose result MemorySanitizer treats as initialized. Select JSON_UNLOCKED_FILE_IO 0 when compiled with MemorySanitizer. This is a regression from #5828 and not part of any release. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |