Compare commits

...
Author SHA1 Message Date
Niels Lohmann a20a34b0cc json_view: keep accept()'s string check out of run()
Clang 23 made run() slower on files without escapes (mesh.pretty +15 %) when
the check-only string path was a separate call target. Dispatch inside
slow_string() instead, so the call in run() is the same as before, and skip
the SIMD run scan between escapes that follow each other (twitterescaped
1.11 -> 0.82 against the previous commit).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-31 09:09:46 +01:00
Claude 5607cd181f tests: avoid astyle-mangled build<> call in json_view builder test
astyle pads the template argument list of the build<...>() call with
spaces because the second argument is a negated expression. Name the
constant first (as json_view.hpp does with nul_is_end) so astyle leaves
the call alone.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N5NasBf4ToHzdcoNEPAJhS
2026-10-31 07:21:48 +00:00
Claude 88a3c1bc15 json_view: pass accept()'s validate-only request as a parameter
Address review of #5970:
- build() takes validate_only as an argument; the validate_only members of
  document_data and basic_json_document are removed, so neither grows
- the internal read path takes an on_error mode (discard, raise, validate)
  instead of allow_exceptions; accept() calls it with on_error::validate
- build_with() has no default for Store
- drop the unneeded static_casts in push_node()
- name the NulIsEnd argument so astyle keeps build<...> intact
- explain in the docs and code why comments/trailing commas keep the index

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N5NasF8Toq7uBWUMCrSjXV
2026-10-30 23:09:53 +00:00
Niels Lohmann 9ba25b1b66 json_view: accept() checks strings without decoding them
The builder that accept() runs stores no nodes, but still decoded every string
with escapes into the arena (slow_string(): memcpy of the runs between escapes
and the UTF-8 of every \u escape). It now calls check_string() instead, which
has the same checks and the same errors and error positions (control characters,
ill-formed UTF-8, bad escapes, bad hex digits, lone and unpaired surrogates, a
missing closing quote) and skips the text between them with scan_string_run(),
without writing anything. The builder that stores keeps slow_string().

The flag that passes the request from accept() to build() now sits next to
`discarded` in document_data, where it uses padding: no member of document_data
moves (the first version put it after `nodes`, which moved all members behind it
by eight bytes).

Tests: the new section of the accept test checks runs of text of every length from
0 to 40 between valid and invalid escapes, UTF-8 sequences and control characters,
in values and in keys, cut near the end of the text, against json::accept().
Checked against a checker that accepts a lone low surrogate.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-30 22:34:32 +01:00
Niels Lohmann dea2a4434a json_view: accept() checks the text without writing an index
json_document::accept() ran the builder that writes the node index and
discarded the index. For input without comments and trailing commas it now
runs a second instantiation of the builder, with a template parameter Store
that is false: the grammar, number, string, UTF-8 and nesting checks are the
same code, but no node is written, the container nodes are not patched when
they close, the node array is not reserved, and no list of large objects is
kept. What only builds the arguments of a node (integer conversion, the digit
counts of floats) is not run either. accept() with comments or trailing commas
and parse() are not changed; the builder that stores is the one parse() always
used. A flag of the document passes the request from accept() to build().

accept() on twitter, citm_catalog, canada and the two minified files, A/B in
one binary, 6 code layouts, time after / before (geometric mean of the five;
the other 14 files of the set are faster too, none is slower):

- Apple M1 Max, Apple clang 21: 0.89
- x86-64 Haswell, GCC 16.2: 0.75
- x86-64 Haswell, Clang 23: 0.78

(semanticscholar-corpus.json: 0.52 on both x86 compilers.) parse() runs the same
instructions as before; its timings moved by up to 3 % on a few small files with
unchanged instruction counts (code placement).

Tests: accept() against json::accept() for the same inputs with all four
option combinations and with and without a terminating NUL: 2,000 generated
documents and damaged copies of them, nesting from 63 to 1000 levels (closed and
not), objects with 300 members, a null byte in the text, and float overflow for
a float document. Checked against a deliberately broken validator.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-30 22:18:14 +01:00
Niels Lohmann 755bb16d7a Fix the red CI on develop: remove the duplicate is_editable in basic_json_view (#5966)
#5919 replaced the bool template parameter Editable of basic_json_view by
view_mode Mode and defined is_editable from Mode. #5890 was written before
that and added a second is_editable = Editable to the private section, which
no longer compiles ("duplicate member", "undeclared identifier Editable"). It
broke every job that builds the json_view tests (macOS, MSVC, clang-cl,
clang on Windows, the C++20 module job, CodeQL).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-30 16:48:31 +01:00
Niels Lohmann d20c4dcd89 json_view benchmarks: compare fairly and state what each row means (#5914)
- bench_view: add "(editable)" rows for json_editable_document and a
  parse+dump workload, so the cost of lazy float conversion shows next to the
  parse row (the traverse workload already includes the parse). Check that
  all engines' dumps agree where outputs are comparable.
- compare.py: add --aligned, which also builds and runs with aligned code and
  reports both tables, so code-layout artifacts become visible. Off by
  default; the default output is unchanged. Not run end to end yet.
- README: say which engines are like-for-like peers (yyjson, simdjson DOM) and
  which are mutable reference points (Boost.JSON, json::parse, the editable
  json_view); explain the lazy float conversion and the eager prototype that
  was measured and dropped.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-30 15:20:12 +01:00
Niels Lohmann ea03d601af json_view: comparison without temporaries and a faster image float check (#5890)
* Compare json_view values without a vector per container and a basic_json per number

operator== of views (view/view, view/json, and editable documents) built two
vectors of members for every object it entered, sorted them with stable_sort
for every object, and converted each number to a basic_json value. It now:

- keeps the members of all open objects in one vector (reserved with
  geometric growth, so that nested objects do not reallocate), and orders
  them by insertion sort for up to 16 members, or not at all when the keys
  ascend already;
- compares numbers of one type directly, and builds a basic_json only for
  numbers of different types;
- compares short texts (keys, numbers, most strings) with two overlapping
  loads (bytes_equal(), in node.hpp);
- first walks both documents in lockstep, in the order of their text, which
  finds most pairs of equal values without the work of the comparison. The
  walk only enters an object if the comparison would visit its members in the
  same order (the object type keeps the order, the keys ascend, or all values
  are scalars), so a difference is never found later than before, and it
  records the containers it found different, so that the comparison does not
  walk them again;
- rejects values of different types and arrays of different sizes before any
  of this, and, for an object type that keeps the order, objects whose first
  (up to four) members have different keys, which ends a comparison that
  differs near the start without preparing the members.

Results are unchanged: numbers compare by value across types, duplicate keys
resolve as parse() resolves them, and objects compare in document order for
ordered_json and by key otherwise.

Instructions of one comparison of a document with an equal copy, macOS arm64
(develop -> now), json_document / ordered_json_document / editable:

  twitter    16.1M -> 7.0M   17.7M -> 1.5M    16.6M -> 7.3M
  citm       45.0M -> 3.4M   47.0M -> 3.2M    45.7M -> 4.3M
  canada    111.4M -> 13.7M  111.4M -> 13.7M  113.3M -> 15.0M
  jeopardy   26.2M -> 3.7M   27.3M -> 3.7M    26.6M -> 12.1M

and of a view with a basic_json value:

  twitter    11.4M -> 5.9M   11.2M -> 7.3M    11.6M -> 5.9M
  citm       33.9M -> 9.7M   32.6M -> 9.1M    34.6M -> 9.8M
  canada     92.9M -> 32.1M  92.9M -> 32.1M   93.9M -> 32.8M

and Linux instruction counts (GCC 13 / Clang 18) of the same comparisons:

  document with an equal copy   json_document     ordered_json_document   editable
    twitter                     -49 % / -52 %     -89 % / -89 %           -48 % / -50 %
    citm                        -90 % / -89 %     -91 % / -90 %           -88 % / -87 %
    jeopardy (4,000 questions)  -85 % / -85 %     -86 % / -86 %           -46 % / -49 %
  view with a basic_json value
    twitter                     -41 % / -45 %     -28 % / -35 %           -45 % / -44 %
    citm                        -63 % / -65 %     -63 % / -66 %           -65 % / -64 %
    jeopardy                    -39 % / -43 %     -14 % / -24 %           -44 % / -43 %

A sweep that places one difference at about 2,100 positions of eight
documents (a token at an even spacing, the first and last tokens, the first,
middle and last member of the root removed, one appended, the document
wrapped) in five kinds of comparison (view/view of read-only and of editable
documents, view/json, json_document, editable/view) finds the same results as
before and no placement slower on the Mac (Clang); on Linux no placement is
slower than 3 % (Clang and GCC).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Check the float tokens of a json_document image without a sort and a scan of every character

load(image_check::full) checked every float token of an image by sorting
the token ranges, scanning the token character by character, and comparing
the digit layout with the node's. Now:

- the ranges are sorted only if they are not sorted already (save() writes
  them in the order of the nodes, so they are);
- a token of the shape [-]digits.digits whose digit counts are the ones the
  node records (nearly all of them) is accepted by checking its digits eight
  at a time, without the general scan;
- the general scan skips runs of digits eight at a time.

Damaged tokens are still rejected as before: the check accepts exactly the
tokens that the general scan accepts.

Instructions of load(image_check::full) (develop -> now):

  macOS arm64        canada 44.6M -> 30.9M (-31 %), jeopardy 11.3M -> 10.7M,
                     citm 10.4M -> 10.0M, twitter 5.06M -> 4.94M
  Linux GCC 13       canada -50 %, jeopardy -43 %, twitter -31 %, citm -28 %
  Linux Clang 18     canada -42 %, jeopardy -48 %, twitter -35 %, citm -32 %

image_check::bounds is unchanged (within 0.4 %).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Rename names that Flawfinder takes for the file function open()

The comparison cursors' open() is now enter(), and a test's open and
close strings are head and tail.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Fix the compiler and lint failures of the json_view comparison

- all_digits(): the lambda declares its constant itself; MSVC does not
  let a lambda without captures read a constexpr local of the function
- equal(): the frame's "in document order" index is a namespace-scope
  constant (document_order); the local constexpr npos in the default
  member initializers of the local struct crashed GCC 9 and MinGW GCC
  and was not found by MSVC
- keys_differ_early() keeps the first keys in std::arrays (cpplint:
  variable-length arrays; clang-tidy: C arrays), and the basic_json
  side's first_keys() is not noexcept (key() may throw)
- the numbers of one type are compared in an if chain instead of a
  nested conditional operator
- the dump test avoids C arrays and temporary string concatenations

View comparison instructions are unchanged (Apple clang, twitter, citm,
canada: -0.1 % to +0.1 %).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Keep the byte comparison helpers in compare.hpp

Address review comments: bytes_equal() and bytes_less() move from node.hpp
to compare.hpp, their only user; key_less() no longer splits the comment of
resolve_indices() from its function; and the loop counters of bytes_equal()
and all_digits() are scoped to their loops.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Dispatch on the compile-time conditions of the json_view comparison

MSVC (/W4 /WX) reported C4127 (conditional expression is constant) for
the template parameter EA and the object type's ordering in an if
condition; both are now tag-dispatched.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Remove a nested duplicate guard from the json_view image test

clang-tidy reported the inner #if !defined(JSON_NOEXCEPTION) as
redundant (the test case is in the same #if) and a find() != npos, which
gets the NOLINT the other tests use.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Mark Infer's null dereference in materialize() as a false positive

Infer (ci_infer, also red on develop since #5882) reports a null
dereference in json_sax_dom_parser::null() called from materialize(). The
member pointer the parser stores a value of an object in is set by the
key() before it, so it is not null there.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Move the json_view comparison tests into their own file

With the tests this branch adds, the object file of unit-json_view_dump.cpp
got too large for the MinGW x64 assembler ("file too big"). The test case
"json_view comparison" is now unit-json_view_compare.cpp, unchanged and
also built with C++17.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Build the DOM parser with Visual Studio 2015 again

Since #5882, the three VS 2015 jobs of AppVeyor fail on develop:
- the detection of keys that can be indexed instantiated std::hash for
  every string type, which VS 2015 rejects with a static_assert for a type
  it does not hash (alt_string): std::hash is now looked at only for
  objects in a vector
- the frame stack copied its frames with std::copy to a raw pointer, which
  VS 2015 reports as unsafe (C4996, an error with /WX): a loop now

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* json_view: no per-object allocations in ordered_json comparisons

For an object type that keeps the order of its members (ordered_json),
resolve_indices() allocated a vector per compared object, and
resolve_members() two. They now use one working vector owned by equal(),
whose capacity is reused, and resolve_indices() needs none when the keys
are unique. resolve_members() sorts small objects by insertion, as the
other paths do.

Instructions per comparison (callgrind, GCC 13 and clang 18, x86-64):
twitter, ordered_json view == ordered_json: 0.939 / 0.941; 20000 small
objects with a duplicate key: 0.741-0.768; all other rows (json,
citm_catalog, canada, view == view) within 0.988-1.003.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-30 15:19:42 +01:00
dependabot[bot] e2cb15c479 Bump the codeql-action group across 1 directory with 4 updates (#5961)
Bumps the codeql-action group with 4 updates in the / directory: [github/codeql-action/init](https://github.com/github/codeql-action), [github/codeql-action/autobuild](https://github.com/github/codeql-action), [github/codeql-action/analyze](https://github.com/github/codeql-action) and [github/codeql-action/upload-sarif](https://github.com/github/codeql-action).


Updates `github/codeql-action/init` from 4.39.0 to 4.39.1
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/6f2ac81f5cd81fe351a197ba0b00d91d03cd2aee...9b3254085494b55a0f7c4c2b3d2d52cc712ce370)

Updates `github/codeql-action/autobuild` from 4.39.0 to 4.39.1
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/6f2ac81f5cd81fe351a197ba0b00d91d03cd2aee...9b3254085494b55a0f7c4c2b3d2d52cc712ce370)

Updates `github/codeql-action/analyze` from 4.39.0 to 4.39.1
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/6f2ac81f5cd81fe351a197ba0b00d91d03cd2aee...9b3254085494b55a0f7c4c2b3d2d52cc712ce370)

Updates `github/codeql-action/upload-sarif` from 4.39.0 to 4.39.1
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/6f2ac81f5cd81fe351a197ba0b00d91d03cd2aee...9b3254085494b55a0f7c4c2b3d2d52cc712ce370)

---
updated-dependencies:
- dependency-name: github/codeql-action/analyze
  dependency-version: 4.39.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: codeql-action
- dependency-name: github/codeql-action/autobuild
  dependency-version: 4.39.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: codeql-action
- dependency-name: github/codeql-action/init
  dependency-version: 4.39.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: codeql-action
- dependency-name: github/codeql-action/upload-sarif
  dependency-version: 4.39.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: codeql-action
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-30 15:19:08 +01:00
dependabot[bot] c451fac1bc Bump step-security/harden-runner from 2.22.1 to 2.22.2 (#5962)
Bumps [step-security/harden-runner](https://github.com/step-security/harden-runner) from 2.22.1 to 2.22.2.
- [Release notes](https://github.com/step-security/harden-runner/releases)
- [Commits](https://github.com/step-security/harden-runner/compare/ccd8616d44fd3846e67624a50d5aad6d37bf2d25...dc2769abcc0fcfbadfc1df54081c1608579ef6b2)

---
updated-dependencies:
- dependency-name: step-security/harden-runner
  dependency-version: 2.22.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-30 15:18:56 +01:00
dependabot[bot] 28f71914b5 Bump egor-tensin/setup-mingw (#5963)
Bumps [egor-tensin/setup-mingw](https://github.com/egor-tensin/setup-mingw) from 41b837e47d7f85214629d255b9c4bc3fcbe9fd63 to d54f5ef0ce585bd3fbb1f8a71b5b254f59dbf931.
- [Release notes](https://github.com/egor-tensin/setup-mingw/releases)
- [Changelog](https://github.com/egor-tensin/setup-mingw/blob/master/CHANGELOG.md)
- [Commits](https://github.com/egor-tensin/setup-mingw/compare/41b837e47d7f85214629d255b9c4bc3fcbe9fd63...d54f5ef0ce585bd3fbb1f8a71b5b254f59dbf931)

---
updated-dependencies:
- dependency-name: egor-tensin/setup-mingw
  dependency-version: d54f5ef0ce585bd3fbb1f8a71b5b254f59dbf931
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-10-30 15:18:44 +01:00
Niels Lohmann 0b91f9ddd8 json_view: check_utf8 skips well-formed runs with the parser's string scan (#5926)
* json_view: check_utf8 skips well-formed runs with the parser's string scan

Strings of 32 bytes or more that are written into an editable document
are validated with scan_string_run(), which skips well-formed UTF-8 16
bytes at a time; the byte-wise check runs only from where the scan found
ill-formed UTF-8, to report the same type_error.316. Shorter strings
keep the byte-wise check (inlined with a constant start, so that this
path is the code it was), where the vector scan costs more than it
saves.

set()/push_back() of strings of 32 bytes or more take 0.29-0.76 of the
time (x86-64 GCC, averaged over code layouts) and 0.25-0.80 (AArch64
clang); shorter strings take 1.5-4 % fewer instructions; the json_view
rows of bench_edit execute at most 0.04 % more instructions.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* tests: avoid sign conversion in json_view_edit UTF-8 check

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* json_view: use while in check_utf8_bytewise

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-30 15:17:55 +01:00
Niels Lohmann 0115bb47b3 Name the source and license of adapted code in the file comments (#5965)
The files with code adapted from fast_float, Zmij and simdjson carried the
authors' SPDX copyright lines, but their file comments did not say what
was taken or under which license: number_parse.hpp, to_chars.hpp,
simd_scan.hpp and view/simd.hpp now do, as zmij.hpp and the yyjson-based
view files already did. README and the license page now also name
fast_float's eight-digit parsing and the lexer's use of simdjson's UTF-8
check. Comments only.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-30 15:17:24 +01:00
38 changed files with 3771 additions and 797 deletions

No files matched your search

+1 -1
View File
@@ -28,7 +28,7 @@ jobs:
steps: steps:
- name: Harden Runner - name: Harden Runner
uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1 uses: step-security/harden-runner@dc2769abcc0fcfbadfc1df54081c1608579ef6b2 # v2.22.2
with: with:
egress-policy: audit egress-policy: audit
+2 -2
View File
@@ -22,7 +22,7 @@ jobs:
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- name: Harden Runner - name: Harden Runner
uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1 uses: step-security/harden-runner@dc2769abcc0fcfbadfc1df54081c1608579ef6b2 # v2.22.2
with: with:
egress-policy: audit egress-policy: audit
@@ -46,7 +46,7 @@ jobs:
steps: steps:
- name: Harden Runner - name: Harden Runner
uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1 uses: step-security/harden-runner@dc2769abcc0fcfbadfc1df54081c1608579ef6b2 # v2.22.2
with: with:
egress-policy: audit egress-policy: audit
+1 -1
View File
@@ -20,7 +20,7 @@ jobs:
timeout-minutes: 30 timeout-minutes: 30
steps: steps:
- name: Harden Runner - name: Harden Runner
uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1 uses: step-security/harden-runner@dc2769abcc0fcfbadfc1df54081c1608579ef6b2 # v2.22.2
with: with:
egress-policy: audit egress-policy: audit
+1 -1
View File
@@ -13,7 +13,7 @@ jobs:
runs-on: ubuntu-22.04 runs-on: ubuntu-22.04
steps: steps:
- name: Harden Runner - name: Harden Runner
uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1 uses: step-security/harden-runner@dc2769abcc0fcfbadfc1df54081c1608579ef6b2 # v2.22.2
with: with:
egress-policy: audit egress-policy: audit
+4 -4
View File
@@ -27,7 +27,7 @@ jobs:
steps: steps:
- name: Harden Runner - name: Harden Runner
uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1 uses: step-security/harden-runner@dc2769abcc0fcfbadfc1df54081c1608579ef6b2 # v2.22.2
with: with:
egress-policy: audit egress-policy: audit
@@ -38,14 +38,14 @@ jobs:
# Initializes the CodeQL tools for scanning. # Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL - name: Initialize CodeQL
uses: github/codeql-action/init@6f2ac81f5cd81fe351a197ba0b00d91d03cd2aee # v4.39.0 uses: github/codeql-action/init@9b3254085494b55a0f7c4c2b3d2d52cc712ce370 # v4.39.1
with: with:
languages: c-cpp languages: c-cpp
# Autobuild attempts to build any compiled languages (C/C++, C#, or Java). # Autobuild attempts to build any compiled languages (C/C++, C#, or Java).
# If this step fails, then you should remove it and run the build manually (see below) # If this step fails, then you should remove it and run the build manually (see below)
- name: Autobuild - name: Autobuild
uses: github/codeql-action/autobuild@6f2ac81f5cd81fe351a197ba0b00d91d03cd2aee # v4.39.0 uses: github/codeql-action/autobuild@9b3254085494b55a0f7c4c2b3d2d52cc712ce370 # v4.39.1
- name: Perform CodeQL Analysis - name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@6f2ac81f5cd81fe351a197ba0b00d91d03cd2aee # v4.39.0 uses: github/codeql-action/analyze@9b3254085494b55a0f7c4c2b3d2d52cc712ce370 # v4.39.1
@@ -20,7 +20,7 @@ jobs:
pull-requests: write pull-requests: write
steps: steps:
- name: Harden Runner - name: Harden Runner
uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1 uses: step-security/harden-runner@dc2769abcc0fcfbadfc1df54081c1608579ef6b2 # v2.22.2
with: with:
egress-policy: audit egress-policy: audit
+1 -1
View File
@@ -21,7 +21,7 @@ jobs:
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- name: Harden Runner - name: Harden Runner
uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1 uses: step-security/harden-runner@dc2769abcc0fcfbadfc1df54081c1608579ef6b2 # v2.22.2
with: with:
egress-policy: audit egress-policy: audit
+2 -2
View File
@@ -31,7 +31,7 @@ jobs:
security-events: write security-events: write
steps: steps:
- name: Harden Runner - name: Harden Runner
uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1 uses: step-security/harden-runner@dc2769abcc0fcfbadfc1df54081c1608579ef6b2 # v2.22.2
with: with:
egress-policy: audit egress-policy: audit
@@ -48,6 +48,6 @@ jobs:
output: 'flawfinder_results.sarif' output: 'flawfinder_results.sarif'
- name: Upload analysis results to GitHub Security tab - name: Upload analysis results to GitHub Security tab
uses: github/codeql-action/upload-sarif@6f2ac81f5cd81fe351a197ba0b00d91d03cd2aee # v4.39.0 uses: github/codeql-action/upload-sarif@9b3254085494b55a0f7c4c2b3d2d52cc712ce370 # v4.39.1
with: with:
sarif_file: ${{github.workspace}}/flawfinder_results.sarif sarif_file: ${{github.workspace}}/flawfinder_results.sarif
+1 -1
View File
@@ -54,7 +54,7 @@ jobs:
timeout-minutes: 60 timeout-minutes: 60
steps: steps:
- name: Harden Runner - name: Harden Runner
uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1 uses: step-security/harden-runner@dc2769abcc0fcfbadfc1df54081c1608579ef6b2 # v2.22.2
with: with:
egress-policy: audit egress-policy: audit
+1 -1
View File
@@ -23,7 +23,7 @@ jobs:
steps: steps:
- name: Harden Runner - name: Harden Runner
uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1 uses: step-security/harden-runner@dc2769abcc0fcfbadfc1df54081c1608579ef6b2 # v2.22.2
with: with:
egress-policy: audit egress-policy: audit
+1 -1
View File
@@ -37,7 +37,7 @@ jobs:
runs-on: ubuntu-latest runs-on: ubuntu-latest
steps: steps:
- name: Harden Runner - name: Harden Runner
uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1 uses: step-security/harden-runner@dc2769abcc0fcfbadfc1df54081c1608579ef6b2 # v2.22.2
with: with:
egress-policy: audit egress-policy: audit
+2 -2
View File
@@ -40,7 +40,7 @@ jobs:
steps: steps:
- name: Harden Runner - name: Harden Runner
uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1 uses: step-security/harden-runner@dc2769abcc0fcfbadfc1df54081c1608579ef6b2 # v2.22.2
with: with:
egress-policy: audit egress-policy: audit
@@ -80,6 +80,6 @@ jobs:
# Upload the results to GitHub's code scanning dashboard. # Upload the results to GitHub's code scanning dashboard.
- name: "Upload to code-scanning" - name: "Upload to code-scanning"
uses: github/codeql-action/upload-sarif@6f2ac81f5cd81fe351a197ba0b00d91d03cd2aee # v4.39.0 uses: github/codeql-action/upload-sarif@9b3254085494b55a0f7c4c2b3d2d52cc712ce370 # v4.39.1
with: with:
sarif_file: results.sarif sarif_file: results.sarif
+2 -2
View File
@@ -36,7 +36,7 @@ jobs:
runs-on: ubuntu-26.04 runs-on: ubuntu-26.04
steps: steps:
- name: Harden Runner - name: Harden Runner
uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1 uses: step-security/harden-runner@dc2769abcc0fcfbadfc1df54081c1608579ef6b2 # v2.22.2
with: with:
egress-policy: audit egress-policy: audit
@@ -65,7 +65,7 @@ jobs:
# Upload SARIF file generated in previous step # Upload SARIF file generated in previous step
- name: Upload SARIF file - name: Upload SARIF file
uses: github/codeql-action/upload-sarif@6f2ac81f5cd81fe351a197ba0b00d91d03cd2aee # v4.39.0 uses: github/codeql-action/upload-sarif@9b3254085494b55a0f7c4c2b3d2d52cc712ce370 # v4.39.1
with: with:
sarif_file: semgrep.sarif sarif_file: semgrep.sarif
if: always() if: always()
+1 -1
View File
@@ -16,7 +16,7 @@ jobs:
steps: steps:
- name: Harden Runner - name: Harden Runner
uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1 uses: step-security/harden-runner@dc2769abcc0fcfbadfc1df54081c1608579ef6b2 # v2.22.2
with: with:
egress-policy: audit egress-policy: audit
+5 -5
View File
@@ -80,7 +80,7 @@ jobs:
runs-on: ubuntu-26.04 runs-on: ubuntu-26.04
steps: steps:
- name: Harden Runner - name: Harden Runner
uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1 uses: step-security/harden-runner@dc2769abcc0fcfbadfc1df54081c1608579ef6b2 # v2.22.2
with: with:
egress-policy: audit egress-policy: audit
@@ -112,7 +112,7 @@ jobs:
target: [ci_test_single_header, ci_cppcheck, ci_cpplint, ci_reproducible_tests, ci_non_git_tests, ci_offline_testdata, ci_reuse_compliance, ci_test_valgrind] target: [ci_test_single_header, ci_cppcheck, ci_cpplint, ci_reproducible_tests, ci_non_git_tests, ci_offline_testdata, ci_reuse_compliance, ci_test_valgrind]
steps: steps:
- name: Harden Runner - name: Harden Runner
uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1 uses: step-security/harden-runner@dc2769abcc0fcfbadfc1df54081c1608579ef6b2 # v2.22.2
with: with:
egress-policy: audit egress-policy: audit
@@ -178,7 +178,7 @@ jobs:
runs-on: ubuntu-26.04 runs-on: ubuntu-26.04
steps: steps:
- name: Harden Runner - name: Harden Runner
uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1 uses: step-security/harden-runner@dc2769abcc0fcfbadfc1df54081c1608579ef6b2 # v2.22.2
with: with:
egress-policy: audit egress-policy: audit
@@ -433,7 +433,7 @@ jobs:
runs-on: ubuntu-26.04 runs-on: ubuntu-26.04
steps: steps:
- name: Harden Runner - name: Harden Runner
uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1 uses: step-security/harden-runner@dc2769abcc0fcfbadfc1df54081c1608579ef6b2 # v2.22.2
with: with:
egress-policy: audit egress-policy: audit
@@ -456,7 +456,7 @@ jobs:
target: [ci_test_examples, ci_test_build_documentation, ci_test_documentation_mermaid] target: [ci_test_examples, ci_test_build_documentation, ci_test_documentation_mermaid]
steps: steps:
- name: Harden Runner - name: Harden Runner
uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1 uses: step-security/harden-runner@dc2769abcc0fcfbadfc1df54081c1608579ef6b2 # v2.22.2
with: with:
egress-policy: audit egress-policy: audit
+2 -2
View File
@@ -28,7 +28,7 @@ jobs:
with: with:
persist-credentials: false persist-credentials: false
- name: Set up MinGW - name: Set up MinGW
uses: egor-tensin/setup-mingw@41b837e47d7f85214629d255b9c4bc3fcbe9fd63 # v3.0 uses: egor-tensin/setup-mingw@d54f5ef0ce585bd3fbb1f8a71b5b254f59dbf931 # v3.0
with: with:
platform: ${{ matrix.architecture }} platform: ${{ matrix.architecture }}
version: 12.2.0 # https://github.com/egor-tensin/setup-mingw/issues/14 version: 12.2.0 # https://github.com/egor-tensin/setup-mingw/issues/14
@@ -153,7 +153,7 @@ jobs:
- name: Install Clang - name: Install Clang
run: curl -fsSL -o LLVM${{ matrix.version }}.exe https://github.com/llvm/llvm-project/releases/download/llvmorg-${{ matrix.version }}/LLVM-${{ matrix.version }}-win64.exe ; 7z x LLVM${{ matrix.version }}.exe -y -o"C:/Program Files/LLVM" run: curl -fsSL -o LLVM${{ matrix.version }}.exe https://github.com/llvm/llvm-project/releases/download/llvmorg-${{ matrix.version }}/LLVM-${{ matrix.version }}-win64.exe ; 7z x LLVM${{ matrix.version }}.exe -y -o"C:/Program Files/LLVM"
- name: Set up MinGW - name: Set up MinGW
uses: egor-tensin/setup-mingw@41b837e47d7f85214629d255b9c4bc3fcbe9fd63 # v3.0 uses: egor-tensin/setup-mingw@d54f5ef0ce585bd3fbb1f8a71b5b254f59dbf931 # v3.0
with: with:
platform: x64 platform: x64
version: 12.2.0 # https://github.com/egor-tensin/setup-mingw/issues/14 version: 12.2.0 # https://github.com/egor-tensin/setup-mingw/issues/14
+2 -2
View File
@@ -1408,9 +1408,9 @@ THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR I
- The class contains a port of the shortest double-to-decimal conversion of [Żmij](https://github.com/vitaut/zmij) by Victor Zverovich, including the conversion of the digits to text by Xiang JunBo and the SIMD instruction sequence of Dougall Johnson, which is licensed under the [MIT License](https://opensource.org/licenses/MIT) (see above). Copyright &copy; 2025 [Victor Zverovich](https://github.com/vitaut) - The class contains a port of the shortest double-to-decimal conversion of [Żmij](https://github.com/vitaut/zmij) by Victor Zverovich, including the conversion of the digits to text by Xiang JunBo and the SIMD instruction sequence of Dougall Johnson, which is licensed under the [MIT License](https://opensource.org/licenses/MIT) (see above). Copyright &copy; 2025 [Victor Zverovich](https://github.com/vitaut)
- The class contains a copy of [Hedley](https://nemequ.github.io/hedley/) from Evan Nemerson which is licensed as [CC0-1.0](https://creativecommons.org/publicdomain/zero/1.0/). - The class contains a copy of [Hedley](https://nemequ.github.io/hedley/) from Evan Nemerson which is licensed as [CC0-1.0](https://creativecommons.org/publicdomain/zero/1.0/).
- The class contains parts of [Google Abseil](https://github.com/abseil/abseil-cpp) which is licensed under the [Apache 2.0 License](https://opensource.org/licenses/Apache-2.0). - The class contains parts of [Google Abseil](https://github.com/abseil/abseil-cpp) which is licensed under the [Apache 2.0 License](https://opensource.org/licenses/Apache-2.0).
- The class contains an adapted version of the Eisel-Lemire algorithm, its table of powers of five, and its digit comparison for long numbers from [fast_float](https://github.com/fastfloat/fast_float) by Daniel Lemire and contributors, which is available under the [MIT License](https://opensource.org/licenses/MIT) (used here), the Apache 2.0 License, and the Boost Software License. Copyright &copy; 2021 The fast_float authors - The class contains an adapted version of the Eisel-Lemire algorithm, its table of powers of five, its digit comparison for long numbers, and its parsing of eight digits at a time from [fast_float](https://github.com/fastfloat/fast_float) by Daniel Lemire and contributors, which is available under the [MIT License](https://opensource.org/licenses/MIT) (used here), the Apache 2.0 License, and the Boost Software License. Copyright &copy; 2021 The fast_float authors
- The view's parser (`<nlohmann/json_view.hpp>`) contains techniques and code adapted from [yyjson](https://github.com/ibireme/yyjson) by YaoYuan, which is licensed under the [MIT License](https://opensource.org/licenses/MIT) (see above): table-driven decoding of `\u` escapes and fixed-offset unrolled checks. - The view's parser (`<nlohmann/json_view.hpp>`) contains techniques and code adapted from [yyjson](https://github.com/ibireme/yyjson) by YaoYuan, which is licensed under the [MIT License](https://opensource.org/licenses/MIT) (see above): table-driven decoding of `\u` escapes and fixed-offset unrolled checks.
- The view's parser (`<nlohmann/json_view.hpp>`) validates non-ASCII strings with the vector UTF-8 check of [simdjson](https://github.com/simdjson/simdjson) by Daniel Lemire, Geoff Langdale, John Keiser, and contributors (its "lookup4" algorithm and tables, after J. Keiser and D. Lemire, "Validating UTF-8 In Less Than One Instruction Per Byte", 2021), which is available under the [MIT License](https://opensource.org/licenses/MIT) (used here) and the Apache 2.0 License. Copyright &copy; 2018-2025 The simdjson authors - The class's lexer and the view's parser (`<nlohmann/json_view.hpp>`) validate non-ASCII strings with the vector UTF-8 check of [simdjson](https://github.com/simdjson/simdjson) by Daniel Lemire, Geoff Langdale, John Keiser, and contributors (its "lookup4" algorithm and tables, after J. Keiser and D. Lemire, "Validating UTF-8 In Less Than One Instruction Per Byte", 2021), which is available under the [MIT License](https://opensource.org/licenses/MIT) (used here) and the Apache 2.0 License. Copyright &copy; 2018-2025 The simdjson authors
<img align="right" src="https://git.fsfe.org/reuse/reuse-ci/raw/branch/master/reuse-horizontal.png" alt="REUSE Software"> <img align="right" src="https://git.fsfe.org/reuse/reuse-ci/raw/branch/master/reuse-horizontal.png" alt="REUSE Software">
@@ -13,9 +13,15 @@ function never throws an exception for invalid input, and the returned `#!cpp bo
is returned. is returned.
The result is the same as that of `BasicJsonType::accept()` by design; the difference is how the check is made. This The result is the same as that of `BasicJsonType::accept()` by design; the difference is how the check is made. This
function runs the document's index builder (and discards the index), while `BasicJsonType::accept()` runs the SAX function runs the document's builder, which checks the text as [`parse()`](parse.md) does, while
parser with a handler that ignores every event. Use this one if you go on to parse the same input with `BasicJsonType::accept()` runs the SAX parser with a handler that ignores every event. Use this one if you go on to
[`parse()`](parse.md) anyway, so that both use the same code; either is fine for a plain yes/no answer. parse the same input with [`parse()`](parse.md) anyway, so that both use the same code; either is fine for a plain
yes/no answer.
When both `ignore_comments` and `ignore_trailing_commas` are `#!cpp false` (the default), the builder writes no index.
With either option set, it builds the index as [`parse()`](parse.md) does and discards it: the result is the same, only
the time differs. (Neither option needs the index; the variant that skips it exists only for the default options to
keep the code size down.)
## Template parameters ## Template parameters
+2 -2
View File
@@ -22,8 +22,8 @@ The class contains a port of the shortest double-to-decimal conversion of [Żmij
The class contains a copy of [Hedley](https://nemequ.github.io/hedley/) from Evan Nemerson which is licensed as [CC0-1.0](https://creativecommons.org/publicdomain/zero/1.0/). The class contains a copy of [Hedley](https://nemequ.github.io/hedley/) from Evan Nemerson which is licensed as [CC0-1.0](https://creativecommons.org/publicdomain/zero/1.0/).
The class contains an adapted version of the Eisel-Lemire algorithm, its table of powers of five, and its digit comparison for long numbers from [fast_float](https://github.com/fastfloat/fast_float) by Daniel Lemire and contributors, which is available under the [MIT License](https://opensource.org/licenses/MIT) (used here), the Apache 2.0 License, and the Boost Software License. Copyright &copy; 2021 The fast_float authors The class contains an adapted version of the Eisel-Lemire algorithm, its table of powers of five, its digit comparison for long numbers, and its parsing of eight digits at a time from [fast_float](https://github.com/fastfloat/fast_float) by Daniel Lemire and contributors, which is available under the [MIT License](https://opensource.org/licenses/MIT) (used here), the Apache 2.0 License, and the Boost Software License. Copyright &copy; 2021 The fast_float authors
The view's parser (`<nlohmann/json_view.hpp>`) contains techniques and code adapted from [yyjson](https://github.com/ibireme/yyjson) by YaoYuan, which is licensed under the [MIT License](https://opensource.org/licenses/MIT) (see above): table-driven decoding of `\u` escapes and fixed-offset unrolled checks. The view's parser (`<nlohmann/json_view.hpp>`) contains techniques and code adapted from [yyjson](https://github.com/ibireme/yyjson) by YaoYuan, which is licensed under the [MIT License](https://opensource.org/licenses/MIT) (see above): table-driven decoding of `\u` escapes and fixed-offset unrolled checks.
The view's parser (`<nlohmann/json_view.hpp>`) validates non-ASCII strings with the vector UTF-8 check of [simdjson](https://github.com/simdjson/simdjson) by Daniel Lemire, Geoff Langdale, John Keiser, and contributors (its "lookup4" algorithm and tables, after J. Keiser and D. Lemire, "Validating UTF-8 In Less Than One Instruction Per Byte", 2021), which is available under the [MIT License](https://opensource.org/licenses/MIT) (used here) and the Apache 2.0 License. Copyright &copy; 2018-2025 The simdjson authors The class's lexer and the view's parser (`<nlohmann/json_view.hpp>`) validate non-ASCII strings with the vector UTF-8 check of [simdjson](https://github.com/simdjson/simdjson) by Daniel Lemire, Geoff Langdale, John Keiser, and contributors (its "lookup4" algorithm and tables, after J. Keiser and D. Lemire, "Validating UTF-8 In Less Than One Instruction Per Byte", 2021), which is available under the [MIT License](https://opensource.org/licenses/MIT) (used here) and the Apache 2.0 License. Copyright &copy; 2018-2025 The simdjson authors
@@ -19,6 +19,11 @@
#include <type_traits> // conditional #include <type_traits> // conditional
// The shortest-digit conversion of doubles (see zmij.hpp) and the writing of
// its digits, including the conversion of the digits to text by Xiang JunBo
// and the SIMD instruction sequence by Dougall Johnson, are ported from Zmij
// by Victor Zverovich (https://github.com/vitaut/zmij, MIT license).
//
// SSE2 (every x86-64 CPU) and NEON (every 64-bit Arm CPU) convert the 16 // SSE2 (every x86-64 CPU) and NEON (every 64-bit Arm CPU) convert the 16
// digits of a double at once (unless JSON_NO_SIMD selects the portable code) // digits of a double at once (unless JSON_NO_SIMD selects the portable code)
#if !defined(JSON_NO_SIMD) && (defined(__x86_64__) || (defined(_M_X64) && !defined(_M_ARM64EC))) #if !defined(JSON_NO_SIMD) && (defined(__x86_64__) || (defined(_M_X64) && !defined(_M_ARM64EC)))
@@ -44,6 +44,13 @@
// use std::from_chars or std::strtold. They are free functions so the lexer // use std::from_chars or std::strtold. They are free functions so the lexer
// stays focused on scanning (see lexer::convert_number()) and so that other // stays focused on scanning (see lexer::convert_number()) and so that other
// parsers of JSON text can convert tokens exactly like it does. // parsers of JSON text can convert tokens exactly like it does.
//
// The Eisel-Lemire conversion (eisel_lemire()), the digit comparison for the
// cases it cannot decide, and the eight-digits-at-a-time helpers
// (is_eight_digits(), parse_eight_digits()) are adapted from fast_float
// (https://github.com/fastfloat/fast_float, Daniel Lemire and contributors,
// available under the MIT, Apache 2.0, and Boost Software licenses; used here
// under the MIT license). The table of powers of five is in pow5_table.hpp.
NLOHMANN_JSON_NAMESPACE_BEGIN NLOHMANN_JSON_NAMESPACE_BEGIN
namespace detail namespace detail
@@ -30,6 +30,13 @@
// NLOHMANN_JSON_NO_TARGET_ATTRIBUTE (for compilers that ignore the target // NLOHMANN_JSON_NO_TARGET_ATTRIBUTE (for compilers that ignore the target
// attribute and then reject the SSSE3 intrinsics) keeps the check portable // attribute and then reject the SSSE3 intrinsics) keeps the check portable
// without disabling the SSE2 code. // without disabling the SSE2 code.
//
// The vector UTF-8 check (utf8_lookup4) is adapted from simdjson
// (https://github.com/simdjson/simdjson, Daniel Lemire, Geoff Langdale, John
// Keiser, and contributors, available under the Apache 2.0 and MIT licenses;
// used here under the MIT license): the "lookup4" algorithm and tables of
// J. Keiser and D. Lemire, "Validating UTF-8 In Less Than One Instruction Per
// Byte" (2021).
#if !defined(JSON_NO_SIMD) && defined(__aarch64__) && (defined(__GNUC__) || defined(__clang__)) && defined(__BYTE_ORDER__) && defined(__ORDER_LITTLE_ENDIAN__) && __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__ #if !defined(JSON_NO_SIMD) && defined(__aarch64__) && (defined(__GNUC__) || defined(__clang__)) && defined(__BYTE_ORDER__) && defined(__ORDER_LITTLE_ENDIAN__) && __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
#include <arm_neon.h> #include <arm_neon.h>
#define JSON_SCAN_NEON 1 #define JSON_SCAN_NEON 1
+120 -18
View File
@@ -69,7 +69,8 @@ struct parse_failure
}; };
/// FloatType: the number_float_t of the document, whose overflow parse() rejects /// FloatType: the number_float_t of the document, whose overflow parse() rejects
template<typename FloatType, bool Comments, bool TrailingCommas, bool NulIsEnd, bool Sentinel> /// Store: false for accept(): the grammar, number, string and UTF-8 checks run as for parse(), but no node is written
template<typename FloatType, bool Comments, bool TrailingCommas, bool NulIsEnd, bool Sentinel, bool Store>
class builder class builder
{ {
public: public:
@@ -245,11 +246,87 @@ class builder
return p; return p;
} }
/// the string from p, the first backslash, control character, ill-formed UTF-8 byte, or the end, checked and not
/// decoded (accept() stores nothing, so the arena is not written): the checks, errors and error positions of
/// slow_string(), for everything up to the closing quote
NLOHMANN_VIEW_NOINLINE decoded check_string(const unsigned char* p)
{
const unsigned char* const end = m_end;
for (;;)
{
if (p == end)
{
return failed(error_code::string_missing_quote, p);
}
const unsigned char c = *p;
if (c == '"')
{
++p;
return decoded{p, 0, 0};
}
if (c != '\\')
{
return failed(c < 0x20 ? error_code::string_control_character : error_code::string_utf8, p);
}
++p;
if (p == end)
{
return failed(error_code::string_missing_quote, p);
}
const unsigned char d = *p++;
if (d == 'u')
{
const int cp = hex4(p); // (moves p past the four digits)
if (NLOHMANN_VIEW_UNLIKELY(cp < 0))
{
return failed(error_code::string_unicode_hex, p);
}
if (NLOHMANN_VIEW_UNLIKELY((cp & 0xF800) == 0xD800)) // a surrogate
{
if (cp >= 0xDC00)
{
return failed(error_code::string_surrogate_low, p);
}
if (end - p < 2 || p[0] != '\\' || p[1] != 'u')
{
return failed(error_code::string_surrogate_high, p);
}
p += 2;
const int lo = hex4(p); // (moves p past the four digits)
if (lo < 0)
{
return failed(error_code::string_unicode_hex, p);
}
if (lo < 0xDC00 || lo > 0xDFFF)
{
return failed(error_code::string_surrogate_high, p);
}
}
}
else if (!(d == '"' || d == '\\' || d == '/' || d == 'b' || d == 'f' || d == 'n' || d == 'r' || d == 't'))
{
--p;
return failed(error_code::string_escape, p);
}
if (p != end && *p == '\\')
{
continue; // (escapes in a row: no run of text between them)
}
p = scan_string_run(p, end); // the text up to the next backslash, quote, control character, or ill-formed UTF-8
}
}
/// s points to the start of the string, p to a backslash, a control /// s points to the start of the string, p to a backslash, a control
/// character, an ill-formed UTF-8 byte, or the end: decode the string into /// character, an ill-formed UTF-8 byte, or the end: decode the string into
/// the arena /// the arena
NLOHMANN_VIEW_NOINLINE decoded slow_string(const unsigned char* s, const unsigned char* p) NLOHMANN_VIEW_NOINLINE decoded slow_string(const unsigned char* s, const unsigned char* p)
{ {
if (!enabled(Store))
{
// (accept(): the string is checked, not decoded)
static_cast<void>(s);
return check_string(p);
}
// single-character escapes; 0: invalid (and 'u', handled separately) // single-character escapes; 0: invalid (and 'u', handled separately)
static const std::array<char, 128> simple_escape = static const std::array<char, 128> simple_escape =
{ {
@@ -610,10 +687,13 @@ run_found:
NLOHMANN_VIEW_ALWAYS_INLINE bool run() NLOHMANN_VIEW_ALWAYS_INLINE bool run()
{ {
cold.m_doc.reserve(estimate_nodes(reinterpret_cast<const char*>(b), static_cast<std::size_t>(e - b))); if (enabled(Store))
base = cold.m_doc.nodes; {
out = base; cold.m_doc.reserve(estimate_nodes(reinterpret_cast<const char*>(b), static_cast<std::size_t>(e - b)));
cap = base + cold.m_doc.node_capacity; base = cold.m_doc.nodes;
out = base;
cap = base + cold.m_doc.node_capacity;
}
if (e - p >= 3 && p[0] == 0xEF && p[1] == 0xBB && p[2] == 0xBF) if (e - p >= 3 && p[0] == 0xEF && p[1] == 0xBB && p[2] == 0xBF)
{ {
@@ -797,7 +877,10 @@ close_object:
// an array pays nothing for this) // an array pays nothing for this)
if (NLOHMANN_VIEW_UNLIKELY(cur_count >= document_data::index_min_members)) if (NLOHMANN_VIEW_UNLIKELY(cur_count >= document_data::index_min_members))
{ {
cold.note_large_object(cur_idx); if (enabled(Store))
{
cold.note_large_object(cur_idx);
}
} }
close_container: close_container:
@@ -957,6 +1040,12 @@ indent_done:
/// an integer's value); two stores on little-endian targets /// an integer's value); two stores on little-endian targets
NLOHMANN_VIEW_ALWAYS_INLINE node* push_node(node_kind k, std::uint8_t flags, std::uint16_t extra, std::size_t off, std::uint64_t second) NLOHMANN_VIEW_ALWAYS_INLINE node* push_node(node_kind k, std::uint8_t flags, std::uint16_t extra, std::size_t off, std::uint64_t second)
{ {
if (!enabled(Store))
{
// accept(): nothing is stored, so what computes the arguments is not run either. base and out stay
// null: their difference, which open() takes, is 0.
return out;
}
if (NLOHMANN_VIEW_UNLIKELY(out == cap)) if (NLOHMANN_VIEW_UNLIKELY(out == cap))
{ {
const auto n = static_cast<std::size_t>(out - base); const auto n = static_cast<std::size_t>(out - base);
@@ -1015,9 +1104,12 @@ indent_done:
NLOHMANN_VIEW_ALWAYS_INLINE void close() NLOHMANN_VIEW_ALWAYS_INLINE void close()
{ {
node& n = base[cur_idx]; if (enabled(Store))
n.len = cur_count; {
n.next = static_cast<std::uint32_t>(out - base) - cur_idx; node& n = base[cur_idx];
n.len = cur_count;
n.next = static_cast<std::uint32_t>(out - base) - cur_idx;
}
if (--depth != 0) if (--depth != 0)
{ {
if (NLOHMANN_VIEW_LIKELY(depth <= 64)) if (NLOHMANN_VIEW_LIKELY(depth <= 64))
@@ -1192,34 +1284,44 @@ indent_done:
}; };
/// run the builder with compile-time options /// run the builder with compile-time options
template<typename FloatType, bool Comments, bool TrailingCommas, bool NulIsEnd> template<typename FloatType, bool Comments, bool TrailingCommas, bool NulIsEnd, bool Store>
inline bool build_with(document_data& d, const char* src, std::size_t size, bool sentinel, parse_failure& failure) inline bool build_with(document_data& d, const char* src, std::size_t size, bool sentinel, parse_failure& failure)
{ {
if (sentinel) if (sentinel)
{ {
builder<FloatType, Comments, TrailingCommas, NulIsEnd, true> bld(d, src, size); builder<FloatType, Comments, TrailingCommas, NulIsEnd, true, Store> bld(d, src, size);
const bool ok = bld.run(); const bool ok = bld.run();
failure = bld.failure(); failure = bld.failure();
return ok; return ok;
} }
builder<FloatType, Comments, TrailingCommas, NulIsEnd, false> bld(d, src, size); builder<FloatType, Comments, TrailingCommas, NulIsEnd, false, Store> bld(d, src, size);
const bool ok = bld.run(); const bool ok = bld.run();
failure = bld.failure(); failure = bld.failure();
return ok; return ok;
} }
/// sentinel: src[size] is readable and 0 (e.g. std::string); FloatType: the /// sentinel: src[size] is readable and 0 (e.g. std::string); FloatType: the
/// number_float_t of the document /// number_float_t of the document; validate_only: accept(), only the result is
/// used
template<typename FloatType, bool NulIsEnd> template<typename FloatType, bool NulIsEnd>
inline bool build(document_data& d, const char* src, std::size_t size, bool comments, bool trailing_commas, bool sentinel, parse_failure& failure) inline bool build(document_data& d, const char* src, std::size_t size, bool comments, bool trailing_commas, bool sentinel, bool validate_only,
parse_failure& failure)
{ {
if (validate_only && !comments && !trailing_commas)
{
// accept() of plain JSON: nothing is stored. The other option sets
// could skip the stores just as well; they keep the builder that
// stores only so that there are two more instantiations of the parse
// loop, not eight.
return build_with<FloatType, false, false, NulIsEnd, false>(d, src, size, sentinel, failure);
}
if (comments) if (comments)
{ {
return trailing_commas ? build_with<FloatType, true, true, NulIsEnd>(d, src, size, sentinel, failure) return trailing_commas ? build_with<FloatType, true, true, NulIsEnd, true>(d, src, size, sentinel, failure)
: build_with<FloatType, true, false, NulIsEnd>(d, src, size, sentinel, failure); : build_with<FloatType, true, false, NulIsEnd, true>(d, src, size, sentinel, failure);
} }
return trailing_commas ? build_with<FloatType, false, true, NulIsEnd>(d, src, size, sentinel, failure) return trailing_commas ? build_with<FloatType, false, true, NulIsEnd, true>(d, src, size, sentinel, failure)
: build_with<FloatType, false, false, NulIsEnd>(d, src, size, sentinel, failure); : build_with<FloatType, false, false, NulIsEnd, true>(d, src, size, sentinel, failure);
} }
} // namespace view } // namespace view
File diff suppressed because it is too large. Load diff
+40 -3
View File
@@ -26,6 +26,7 @@
#include <nlohmann/detail/view/lookup.hpp> #include <nlohmann/detail/view/lookup.hpp>
#include <nlohmann/detail/view/macro_scope.hpp> #include <nlohmann/detail/view/macro_scope.hpp>
#include <nlohmann/detail/view/node.hpp> #include <nlohmann/detail/view/node.hpp>
#include <nlohmann/detail/view/scan.hpp>
#include <nlohmann/detail/view/view_mode.hpp> #include <nlohmann/detail/view/view_mode.hpp>
NLOHMANN_JSON_NAMESPACE_BEGIN NLOHMANN_JSON_NAMESPACE_BEGIN
@@ -46,10 +47,9 @@ struct first_true<false, Conditions...> : std::integral_constant < int, 1 + firs
/// (type_error.316 with the same message), so that an editable document /// (type_error.316 with the same message), so that an editable document
/// only holds valid UTF-8: the error is at the first byte that no /// only holds valid UTF-8: the error is at the first byte that no
/// well-formed sequence can continue with (Unicode, Table 3-7). /// well-formed sequence can continue with (Unicode, Table 3-7).
inline void check_utf8(const char* s, std::size_t n) NLOHMANN_VIEW_ALWAYS_INLINE void check_utf8_bytewise(const unsigned char* p, std::size_t n, std::size_t i)
{ {
const auto* const p = reinterpret_cast<const unsigned char*>(s); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast) while (i < n)
for (std::size_t i = 0; i < n;)
{ {
const unsigned char c = p[i]; const unsigned char c = p[i];
if (c < 0x80) if (c < 0x80)
@@ -79,6 +79,43 @@ inline void check_utf8(const char* s, std::size_t n)
} }
} }
/// The long-string path of check_utf8(): skips well-formed runs with the
/// parser's string scan (which stops at a quote, a backslash, or a control
/// character, all legal here, and at ill-formed UTF-8); the byte-wise check
/// runs only from where the scan found ill-formed UTF-8, to report it.
NLOHMANN_VIEW_NOINLINE inline void check_utf8_scan(const unsigned char* p, std::size_t n)
{
const unsigned char* const e = p + n;
const unsigned char* q = p;
for (;;)
{
q = scan_string_run<true>(q, e);
if (q == e)
{
return;
}
if (*q >= 0x80)
{
check_utf8_bytewise(p, n, static_cast<std::size_t>(q - p));
return; // not reached: the scan stopped at ill-formed UTF-8
}
++q; // a quote, backslash, or control character
}
}
inline void check_utf8(const char* s, std::size_t n)
{
const auto* const p = reinterpret_cast<const unsigned char*>(s); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
if (NLOHMANN_VIEW_UNLIKELY(n >= 32))
{
check_utf8_scan(p, n);
return;
}
// short strings: the vector scan costs more than it saves; (the loop is
// inlined with a constant start, so this path is the plain byte-wise check)
check_utf8_bytewise(p, n, 0);
}
/*! /*!
@brief the edits of an editable basic_json_document @brief the edits of an editable basic_json_document
+100 -18
View File
@@ -258,6 +258,81 @@ struct number_token
bool is_float; ///< a fraction or an exponent bool is_float; ///< a fraction or an exponent
}; };
/// advance over ASCII digits, eight bytes at a time
inline const unsigned char* digit_run(const unsigned char* p, const unsigned char* e) noexcept
{
constexpr std::uint64_t ones = 0x0101010101010101ull;
while (e - p >= 8)
{
// (the digits become 0..9; a byte above 9, or with bit 7 set, is no digit. A carry
// out of a byte with bit 7 set only affects the bytes above, after the first one
// that count_trailing_zeros finds.)
const std::uint64_t x = read_eight_bytes(p) ^ (ones * 0x30u);
const std::uint64_t not_digit = ((x + (ones * 0x76u)) | x) & (ones * 0x80u);
if (not_digit != 0)
{
return p + (static_cast<std::size_t>(count_trailing_zeros(not_digit)) / 8);
}
p += 8;
}
while (p != e && is_digit(*p))
{
++p;
}
return p;
}
/// whether the n bytes at p are ASCII digits (n > 0)
inline bool all_digits(const unsigned char* p, std::size_t n) noexcept
{
const auto digits = [](const unsigned char* q)
{
constexpr std::uint64_t ones = 0x0101010101010101ull;
const std::uint64_t x = read_eight_bytes(q) ^ (ones * 0x30u);
return (((x + (ones * 0x76u)) | x) & (ones * 0x80u)) == 0;
};
if (n >= 8)
{
for (std::size_t i = 0; i + 8 < n; i += 8)
{
if (!digits(p + i))
{
return false;
}
}
return digits(p + n - 8); // (overlapping the bytes before: they are digits)
}
for (std::size_t i = 0; i < n; ++i)
{
if (!is_digit(p[i]))
{
return false;
}
}
return true;
}
/// Whether a float token has the shape that the parser records for most of
/// them: [-]digits.digits, with the numbers of digits of the node's layout
/// (extra). Then the token is a number of that layout, and the general scan
/// that follows is not needed.
inline bool plain_float_token(const unsigned char* s, std::size_t len, std::uint16_t extra) noexcept
{
const std::size_t int_digits = extra & 0xFFu;
const std::size_t frac_digits = extra >> 8u;
if (int_digits == 0 || int_digits >= 255 || frac_digits == 0 || frac_digits >= 255 || len == 0)
{
return false;
}
const std::size_t negative = *s == '-' ? 1 : 0;
if (len != negative + int_digits + 1 + frac_digits)
{
return false;
}
const unsigned char* const p = s + negative;
return !(int_digits > 1 && *p == '0') && p[int_digits] == '.' && all_digits(p, int_digits) && all_digits(p + int_digits + 1, frac_digits);
}
/// whether [s, s + len) is a JSON number (the grammar the parser accepts) /// whether [s, s + len) is a JSON number (the grammar the parser accepts)
inline bool scan_number_token(const unsigned char* s, std::size_t len, number_token& t) inline bool scan_number_token(const unsigned char* s, std::size_t len, number_token& t)
{ {
@@ -276,11 +351,7 @@ inline bool scan_number_token(const unsigned char* s, std::size_t len, number_to
} }
else if (*p >= '1' && *p <= '9') else if (*p >= '1' && *p <= '9')
{ {
do p = digit_run(p + 1, e);
{
++p;
}
while (p != e && is_digit(*p));
} }
else else
{ {
@@ -292,10 +363,7 @@ inline bool scan_number_token(const unsigned char* s, std::size_t len, number_to
if (p != e && *p == '.') if (p != e && *p == '.')
{ {
const unsigned char* const f0 = ++p; const unsigned char* const f0 = ++p;
while (p != e && is_digit(*p)) p = digit_run(p, e);
{
++p;
}
if (p == f0) if (p == f0)
{ {
return false; return false;
@@ -406,10 +474,15 @@ struct float_range
template<typename Check> template<typename Check>
bool check_distinct_ranges(std::vector<byte_range>& ranges, Check check) bool check_distinct_ranges(std::vector<byte_range>& ranges, Check check)
{ {
std::sort(ranges.begin(), ranges.end(), [](const byte_range & a, const byte_range & b) // (save() writes the ranges in the order of the nodes, which is sorted already)
const auto before = [](const byte_range & a, const byte_range & b)
{ {
return a.off != b.off ? a.off < b.off : a.len < b.len; return a.off != b.off ? a.off < b.off : a.len < b.len;
}); };
if (!std::is_sorted(ranges.begin(), ranges.end(), before))
{
std::sort(ranges.begin(), ranges.end(), before);
}
std::size_t end = 0; std::size_t end = 0;
for (std::size_t i = 0; i < ranges.size(); ++i) for (std::size_t i = 0; i < ranges.size(); ++i)
{ {
@@ -431,7 +504,8 @@ bool check_distinct_ranges(std::vector<byte_range>& ranges, Check check)
/// record layouts that match it), tokens must not overlap. /// record layouts that match it), tokens must not overlap.
inline bool check_float_ranges(std::vector<float_range>& ranges, const unsigned char* text) inline bool check_float_ranges(std::vector<float_range>& ranges, const unsigned char* text)
{ {
std::sort(ranges.begin(), ranges.end(), [](const float_range & a, const float_range & b) // (save() writes the tokens in the order of the nodes, which is sorted already)
const auto before = [](const float_range & a, const float_range & b)
{ {
if (a.off != b.off) if (a.off != b.off)
{ {
@@ -442,7 +516,11 @@ inline bool check_float_ranges(std::vector<float_range>& ranges, const unsigned
return a.len < b.len; return a.len < b.len;
} }
return a.extra < b.extra; return a.extra < b.extra;
}); };
if (!std::is_sorted(ranges.begin(), ranges.end(), before))
{
std::sort(ranges.begin(), ranges.end(), before);
}
std::size_t end = 0; std::size_t end = 0;
std::size_t i = 0; std::size_t i = 0;
while (i < ranges.size()) while (i < ranges.size())
@@ -452,14 +530,18 @@ inline bool check_float_ranges(std::vector<float_range>& ranges, const unsigned
{ {
return false; return false;
} }
number_token t{};
const unsigned char* const s = text + r.off; const unsigned char* const s = text + r.off;
if (!scan_number_token(s, r.len, t) || !float_token_finite(s, r.len, t)) std::uint16_t layout = r.extra;
if (!plain_float_token(s, r.len, r.extra))
{ {
return false; number_token t{};
if (!scan_number_token(s, r.len, t) || !float_token_finite(s, r.len, t))
{
return false;
}
// the digit layout the parser records (or "many", as compaction writes it)
layout = float_layout(t);
} }
// the digit layout the parser records (or "many", as compaction writes it)
const std::uint16_t layout = float_layout(t);
for (; i < ranges.size() && ranges[i].off == r.off && ranges[i].len == r.len; ++i) for (; i < ranges.size() && ranges[i].off == r.off && ranges[i].len == r.len; ++i)
{ {
if (ranges[i].extra != layout && ranges[i].extra != 0xFFFFu) if (ranges[i].extra != layout && ranges[i].extra != 0xFFFFu)
+7
View File
@@ -29,6 +29,13 @@
// NLOHMANN_VIEW_NO_TARGET_ATTRIBUTE (set by the GCC module interface, where GCC // NLOHMANN_VIEW_NO_TARGET_ATTRIBUTE (set by the GCC module interface, where GCC
// ignores the target attribute and then rejects the SSSE3 intrinsics) keeps the // ignores the target attribute and then rejects the SSSE3 intrinsics) keeps the
// check portable without disabling the SSE2 code. // check portable without disabling the SSE2 code.
//
// The vector UTF-8 check (utf8_lookup4) is adapted from simdjson
// (https://github.com/simdjson/simdjson, Daniel Lemire, Geoff Langdale, John
// Keiser, and contributors, available under the Apache 2.0 and MIT licenses;
// used here under the MIT license): the "lookup4" algorithm and tables of
// J. Keiser and D. Lemire, "Validating UTF-8 In Less Than One Instruction Per
// Byte" (2021).
#if !defined(JSON_NO_SIMD) && defined(__aarch64__) && (defined(__GNUC__) || defined(__clang__)) && NLOHMANN_VIEW_LITTLE_ENDIAN #if !defined(JSON_NO_SIMD) && defined(__aarch64__) && (defined(__GNUC__) || defined(__clang__)) && NLOHMANN_VIEW_LITTLE_ENDIAN
#include <arm_neon.h> #include <arm_neon.h>
#define NLOHMANN_VIEW_NEON 1 #define NLOHMANN_VIEW_NEON 1
+38 -19
View File
@@ -763,6 +763,7 @@ class basic_json_view
template<typename, view_mode> friend class basic_json_document; template<typename, view_mode> friend class basic_json_document;
template<typename, view_mode> friend class basic_json_view; template<typename, view_mode> friend class basic_json_view;
template<typename, typename> friend class detail::view::editor; template<typename, typename> friend class detail::view::editor;
template<typename, typename> friend class detail::view::view_side;
friend iterator; friend iterator;
basic_json_view(const document_data* d, const node* n) noexcept basic_json_view(const document_data* d, const node* n) noexcept
@@ -1059,7 +1060,7 @@ class basic_json_document
const bool ignore_trailing_commas = false) const bool ignore_trailing_commas = false)
{ {
basic_json_document d; basic_json_document d;
d.read_range(first, last, allow_exceptions, ignore_comments, ignore_trailing_commas); d.read_range(first, last, error_mode(allow_exceptions), ignore_comments, ignore_trailing_commas);
return d; return d;
} }
@@ -1074,7 +1075,7 @@ class basic_json_document
basic_json_document d; basic_json_document d;
std::string buf; std::string buf;
append_input(buf, std::forward<InputType>(input)); append_input(buf, std::forward<InputType>(input));
d.build_owned(std::move(buf), allow_exceptions, ignore_comments, ignore_trailing_commas); d.build_owned(std::move(buf), error_mode(allow_exceptions), ignore_comments, ignore_trailing_commas);
return d; return d;
} }
@@ -1087,7 +1088,8 @@ class basic_json_document
static bool accept(InputType&& input, const bool ignore_comments = false, const bool ignore_trailing_commas = false) static bool accept(InputType&& input, const bool ignore_comments = false, const bool ignore_trailing_commas = false)
{ {
basic_json_document d; basic_json_document d;
d.read(std::forward<InputType>(input), false, ignore_comments, ignore_trailing_commas); d.read_kind(std::forward<InputType>(input), on_error::validate, ignore_comments, ignore_trailing_commas,
std::integral_constant<detail::view::input_kind, detail::view::classify_input<InputType>::value> {});
return !d.is_discarded(); return !d.is_discarded();
} }
@@ -1103,7 +1105,7 @@ class basic_json_document
const bool ignore_comments = false, const bool ignore_comments = false,
const bool ignore_trailing_commas = false) const bool ignore_trailing_commas = false)
{ {
read_kind(std::forward<InputType>(input), allow_exceptions, ignore_comments, ignore_trailing_commas, read_kind(std::forward<InputType>(input), error_mode(allow_exceptions), ignore_comments, ignore_trailing_commas,
std::integral_constant<detail::view::input_kind, detail::view::classify_input<InputType>::value> {}); std::integral_constant<detail::view::input_kind, detail::view::classify_input<InputType>::value> {});
} }
@@ -1462,6 +1464,22 @@ class basic_json_document
return idx; return idx;
} }
/// what a failed parse does
enum class on_error : std::uint8_t
{
discard, ///< the document is discarded
raise, ///< parse_error is thrown
validate ///< accept(): the document is discarded, and no node is stored while parsing
};
static on_error error_mode(bool allow_exceptions) noexcept
{
return allow_exceptions ? on_error::raise : on_error::discard;
}
/// NulIsEnd of the builder: a null byte ends the input unless strict_nul_handling
static constexpr bool nul_is_end = !detail::abi_config::strict_nul_handling;
/// create the storage (sized for the input) on first use /// create the storage (sized for the input) on first use
void ensure_data(const char* src, std::size_t size) void ensure_data(const char* src, std::size_t size)
{ {
@@ -1473,7 +1491,7 @@ class basic_json_document
/// parse a buffer the document takes ownership of (it replaces the /// parse a buffer the document takes ownership of (it replaces the
/// previous owned copy) /// previous owned copy)
void build_owned(std::string&& buf, bool allow_exceptions, bool comments, bool trailing_commas) void build_owned(std::string&& buf, on_error allow_exceptions, bool comments, bool trailing_commas)
{ {
ensure_data(buf.data(), buf.size()); ensure_data(buf.data(), buf.size());
m_data->owned = std::move(buf); m_data->owned = std::move(buf);
@@ -1483,7 +1501,7 @@ class basic_json_document
/// parse a copy of the input; the copy is appended to the cleared owned /// parse a copy of the input; the copy is appended to the cleared owned
/// buffer, so that its capacity is reused /// buffer, so that its capacity is reused
template<typename T> template<typename T>
void build_copy(T&& input, bool allow_exceptions, bool comments, bool trailing_commas) void build_copy(T&& input, on_error allow_exceptions, bool comments, bool trailing_commas)
{ {
if (!m_data) if (!m_data)
{ {
@@ -1504,7 +1522,7 @@ class basic_json_document
} }
/// sentinel: src[size] is readable and 0 (std::string, C strings) /// sentinel: src[size] is readable and 0 (std::string, C strings)
void build(const char* src, std::size_t size, bool allow_exceptions, bool comments, bool trailing_commas, bool owned, bool sentinel) void build(const char* src, std::size_t size, on_error allow_exceptions, bool comments, bool trailing_commas, bool owned, bool sentinel)
{ {
ensure_data(src, size); ensure_data(src, size);
document_data& d = *m_data; document_data& d = *m_data;
@@ -1531,7 +1549,8 @@ class basic_json_document
} }
else else
{ {
ok = detail::view::build < typename BasicJsonType::number_float_t, !detail::abi_config::strict_nul_handling > (d, src, size, comments, trailing_commas, sentinel, failure); ok = detail::view::build<typename BasicJsonType::number_float_t, nul_is_end>(d, src, size, comments, trailing_commas, sentinel,
allow_exceptions == on_error::validate, failure);
} }
if (NLOHMANN_VIEW_LIKELY(ok)) if (NLOHMANN_VIEW_LIKELY(ok))
{ {
@@ -1544,7 +1563,7 @@ class basic_json_document
return; return;
} }
std::vector<std::uint32_t>().swap(d.large_objects); std::vector<std::uint32_t>().swap(d.large_objects);
if (allow_exceptions) if (allow_exceptions == on_error::raise)
{ {
detail::view::throw_parse_failure<BasicJsonType>(failure, src, size, comments, trailing_commas); detail::view::throw_parse_failure<BasicJsonType>(failure, src, size, comments, trailing_commas);
} }
@@ -1552,13 +1571,13 @@ class basic_json_document
// --- input dispatch (see detail::view::input_kind) --- // --- input dispatch (see detail::view::input_kind) ---
void read_kind(std::string&& s, bool ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::move_string> /*unused*/) void read_kind(std::string&& s, on_error ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::move_string> /*unused*/)
{ {
build_owned(std::move(s), ae, c, tc); build_owned(std::move(s), ae, c, tc);
} }
template<typename CharT> template<typename CharT>
void read_kind(CharT* s, bool ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::c_string> /*unused*/) void read_kind(CharT* s, on_error ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::c_string> /*unused*/)
{ {
if (s == nullptr) if (s == nullptr)
{ {
@@ -1572,7 +1591,7 @@ class basic_json_document
} }
template<typename Array> template<typename Array>
void read_kind(Array& a, bool ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::char_array> /*unused*/) void read_kind(Array& a, on_error ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::char_array> /*unused*/)
{ {
using CharT = typename std::remove_cv<typename std::remove_extent<Array>::type>::type; using CharT = typename std::remove_cv<typename std::remove_extent<Array>::type>::type;
const std::size_t n = std::extent<Array>::value; const std::size_t n = std::extent<Array>::value;
@@ -1583,7 +1602,7 @@ class basic_json_document
} }
template<typename T> template<typename T>
void read_kind(const T& s, bool ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::borrow_range> /*unused*/) void read_kind(const T& s, on_error ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::borrow_range> /*unused*/)
{ {
// std::basic_string guarantees data()[size()] == 0: use it as sentinel // std::basic_string guarantees data()[size()] == 0: use it as sentinel
const auto size = static_cast<std::size_t>(s.size()); const auto size = static_cast<std::size_t>(s.size());
@@ -1592,13 +1611,13 @@ class basic_json_document
} }
template<typename T> template<typename T>
void read_kind(const T& s, bool ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::copy_range> /*unused*/) void read_kind(const T& s, on_error ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::copy_range> /*unused*/)
{ {
build_copy(s, ae, c, tc); build_copy(s, ae, c, tc);
} }
template<typename CharT> template<typename CharT>
void read_kind(CharT* s, bool ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::wide_c_string> /*unused*/) void read_kind(CharT* s, on_error ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::wide_c_string> /*unused*/)
{ {
if (s == nullptr) if (s == nullptr)
{ {
@@ -1610,13 +1629,13 @@ class basic_json_document
} }
template<typename T> template<typename T>
void read_kind(T&& input, bool ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::adapter> /*unused*/) void read_kind(T&& input, on_error ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::adapter> /*unused*/)
{ {
build_copy(std::forward<T>(input), ae, c, tc); build_copy(std::forward<T>(input), ae, c, tc);
} }
template<typename IteratorType> template<typename IteratorType>
void read_range(IteratorType first, IteratorType last, bool ae, bool c, bool tc) void read_range(IteratorType first, IteratorType last, on_error ae, bool c, bool tc)
{ {
using value_type = typename std::remove_cv<typename std::iterator_traits<IteratorType>::value_type>::type; using value_type = typename std::remove_cv<typename std::iterator_traits<IteratorType>::value_type>::type;
// borrow the range where the library's input adapter scans it as one // borrow the range where the library's input adapter scans it as one
@@ -1627,14 +1646,14 @@ class basic_json_document
} }
template<typename IteratorType> template<typename IteratorType>
void read_range_impl(IteratorType first, IteratorType last, bool ae, bool c, bool tc, std::true_type /*contiguous bytes*/) void read_range_impl(IteratorType first, IteratorType last, on_error ae, bool c, bool tc, std::true_type /*contiguous bytes*/)
{ {
const auto size = static_cast<std::size_t>(std::distance(first, last)); const auto size = static_cast<std::size_t>(std::distance(first, last));
build(size == 0 ? "" : reinterpret_cast<const char*>(&*first), size, ae, c, tc, false, size == 0); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast) build(size == 0 ? "" : reinterpret_cast<const char*>(&*first), size, ae, c, tc, false, size == 0); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
} }
template<typename IteratorType> template<typename IteratorType>
void read_range_impl(IteratorType first, IteratorType last, bool ae, bool c, bool tc, std::false_type /*other*/) void read_range_impl(IteratorType first, IteratorType last, on_error ae, bool c, bool tc, std::false_type /*other*/)
{ {
build_copy(detail::view::make_adapter_input(detail::input_adapter(first, last)), ae, c, tc); build_copy(detail::view::make_adapter_input(detail::input_adapter(first, last)), ae, c, tc);
} }
+19
View File
@@ -10970,6 +10970,13 @@ NLOHMANN_JSON_NAMESPACE_END
// use std::from_chars or std::strtold. They are free functions so the lexer // use std::from_chars or std::strtold. They are free functions so the lexer
// stays focused on scanning (see lexer::convert_number()) and so that other // stays focused on scanning (see lexer::convert_number()) and so that other
// parsers of JSON text can convert tokens exactly like it does. // parsers of JSON text can convert tokens exactly like it does.
//
// The Eisel-Lemire conversion (eisel_lemire()), the digit comparison for the
// cases it cannot decide, and the eight-digits-at-a-time helpers
// (is_eight_digits(), parse_eight_digits()) are adapted from fast_float
// (https://github.com/fastfloat/fast_float, Daniel Lemire and contributors,
// available under the MIT, Apache 2.0, and Boost Software licenses; used here
// under the MIT license). The table of powers of five is in pow5_table.hpp.
NLOHMANN_JSON_NAMESPACE_BEGIN NLOHMANN_JSON_NAMESPACE_BEGIN
namespace detail namespace detail
@@ -12078,6 +12085,13 @@ NLOHMANN_JSON_NAMESPACE_END
// NLOHMANN_JSON_NO_TARGET_ATTRIBUTE (for compilers that ignore the target // NLOHMANN_JSON_NO_TARGET_ATTRIBUTE (for compilers that ignore the target
// attribute and then reject the SSSE3 intrinsics) keeps the check portable // attribute and then reject the SSSE3 intrinsics) keeps the check portable
// without disabling the SSE2 code. // without disabling the SSE2 code.
//
// The vector UTF-8 check (utf8_lookup4) is adapted from simdjson
// (https://github.com/simdjson/simdjson, Daniel Lemire, Geoff Langdale, John
// Keiser, and contributors, available under the Apache 2.0 and MIT licenses;
// used here under the MIT license): the "lookup4" algorithm and tables of
// J. Keiser and D. Lemire, "Validating UTF-8 In Less Than One Instruction Per
// Byte" (2021).
#if !defined(JSON_NO_SIMD) && defined(__aarch64__) && (defined(__GNUC__) || defined(__clang__)) && defined(__BYTE_ORDER__) && defined(__ORDER_LITTLE_ENDIAN__) && __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__ #if !defined(JSON_NO_SIMD) && defined(__aarch64__) && (defined(__GNUC__) || defined(__clang__)) && defined(__BYTE_ORDER__) && defined(__ORDER_LITTLE_ENDIAN__) && __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
#include <arm_neon.h> #include <arm_neon.h>
#define JSON_SCAN_NEON 1 #define JSON_SCAN_NEON 1
@@ -32495,6 +32509,11 @@ NLOHMANN_JSON_NAMESPACE_END
#include <type_traits> // conditional #include <type_traits> // conditional
// The shortest-digit conversion of doubles (see zmij.hpp) and the writing of
// its digits, including the conversion of the digits to text by Xiang JunBo
// and the SIMD instruction sequence by Dougall Johnson, are ported from Zmij
// by Victor Zverovich (https://github.com/vitaut/zmij, MIT license).
//
// SSE2 (every x86-64 CPU) and NEON (every 64-bit Arm CPU) convert the 16 // SSE2 (every x86-64 CPU) and NEON (every 64-bit Arm CPU) convert the 16
// digits of a double at once (unless JSON_NO_SIMD selects the portable code) // digits of a double at once (unless JSON_NO_SIMD selects the portable code)
#if !defined(JSON_NO_SIMD) && (defined(__x86_64__) || (defined(_M_X64) && !defined(_M_ARM64EC))) #if !defined(JSON_NO_SIMD) && (defined(__x86_64__) || (defined(_M_X64) && !defined(_M_ARM64EC)))
+1592 -478
View File
File diff suppressed because it is too large. Load diff
+27 -1
View File
@@ -21,6 +21,10 @@ python3 tests/benchmarks/json_view/compare.py --data <json_test_data directory>
- The other libraries come from the system: pkg-config, or Homebrew (`brew install yyjson simdjson boost`). With - The other libraries come from the system: pkg-config, or Homebrew (`brew install yyjson simdjson boost`). With
`--download`, pinned releases are downloaded instead and checked against their SHA-256. Without Boost headers (or `--download`, pinned releases are downloaded instead and checked against their SHA-256. Without Boost headers (or
with `--no-boost`), the Boost.JSON columns are skipped, and the results say so. with `--no-boost`), the Boost.JSON columns are skipped, and the results say so.
- `--aligned` additionally builds and runs everything with all functions aligned to 64 bytes (`-mllvm
-align-all-functions=6` for Clang, `-falign-functions=64 -falign-loops=64` for GCC), and reports both result
tables as "default build" and "aligned build". Code layout alone can move a result by a few percent; a difference
that appears in only one of the two tables is a layout artifact. Off by default, which doubles the run time.
- `--corpus file...` adds files to the corpus benchmark, e.g. those of - `--corpus file...` adds files to the corpus benchmark, e.g. those of
[simdjson-data](https://github.com/simdjson/simdjson-data) or the [simdjson-data](https://github.com/simdjson/simdjson-data) or the
[yyjson benchmark](https://github.com/ibireme/yyjson_benchmark). [yyjson benchmark](https://github.com/ibireme/yyjson_benchmark).
@@ -39,7 +43,7 @@ where `json_view` stands on another architecture; they are not meant for publica
## What is measured ## What is measured
`bench_view.cpp` runs four workloads on twitter, citm_catalog, canada, jeopardy, a single tweet (`status`), and a `bench_view.cpp` runs these workloads on twitter, citm_catalog, canada, jeopardy, a single tweet (`status`), and a
JSON-RPC request (`rpc`): JSON-RPC request (`rpc`):
| workload | what it does | | workload | what it does |
@@ -48,6 +52,7 @@ JSON-RPC request (`rpc`):
| traverse | parse, then visit every value, convert every number, touch every string and key | | traverse | parse, then visit every value, convert every number, touch every string and key |
| select | parse, then read a few fields per record (e.g. id, user name, and retweet count of each tweet) | | select | parse, then read a few fields per record (e.g. id, user name, and retweet count of each tweet) |
| dump | serialize a parsed document (compact) | | dump | serialize a parsed document (compact) |
| parse+dump | build a fresh document, then serialize it (compact), in one timed call |
`bench_corpus.cpp` runs parse, traverse, and dump on any list of files, so that no library is tuned to a handful of `bench_corpus.cpp` runs parse, traverse, and dump on any list of files, so that no library is tuned to a handful of
documents. Its dump also writes the numbers as they are in the input: `json_view` with `number_format::source`, and documents. Its dump also writes the numbers as they are in the input: `json_view` with `number_format::source`, and
@@ -72,11 +77,32 @@ call follows an untimed call of the same engine: otherwise the engine after `jso
cleaning up the tens of thousands of nodes `json::parse` just freed (with glibc, this made `json_view` look 1.7 times cleaning up the tens of thousands of nodes `json::parse` just freed (with glibc, this made `json_view` look 1.7 times
slower on citm_catalog traverse). slower on citm_catalog traverse).
`json_view` is an index into the source text, not a tree of owning nodes. Its like-for-like peers are yyjson and
simdjson DOM (immutable, indexed, no per-node allocation). Boost.JSON and `json::parse` build mutable, owning DOMs and
are reference points for what that costs, not equals; simdjson On-Demand is lazy and forward-only, so it is the other
way round: it has no random access to pay for. All engines validate UTF-8 and syntax by default.
Number conversion differs: `json_view` converts integers while parsing but converts floating-point numbers lazily,
when they are read or dumped; yyjson (default flags) and simdjson DOM convert every number while parsing. The cost of
float conversion therefore appears in the traverse and dump rows for `json_view`, and in the parse row for the others.
For float-heavy documents (canada), the fair reading is a sum of parse and use: the `traverse` workload already
includes the parse, and `parse+dump` measures a fresh document and its dump in one timed call; the `dump` row alone
leaves the conversion in `json_view` and out of the others. Eager conversion at parse time was prototyped as an opt-in
and measured: canada traverse (with its parse) was 4-7 % faster, but parse+dump 11-30 % slower and parse 36-90 % slower. It was
dropped, so lazy conversion is the only mode.
`json_view` can also be edited: `json_editable_document` has the same read API, and the `(editable)` rows in
`bench_view` measure it on the same workloads. They are the fair counterpart of Boost.JSON and `json::parse`, which are
mutable too; the plain `json_view` rows are the counterpart of yyjson and simdjson DOM, which are not. `bench_edit` adds
the read-modify-write workloads. Read the tables with that in mind: an immutable-only library is not compared with an
editable `json_view` unless it says so.
The engines do not all offer the same features, which the numbers should be read with: The engines do not all offer the same features, which the numbers should be read with:
| engine | document | random access | editable | notes | | engine | document | random access | editable | notes |
|---|---|---|---|---| |---|---|---|---|---|
| `json_view` | immutable index into the text | yes | no | a fresh document per parse; "reused" parses into the same document | | `json_view` | immutable index into the text | yes | no | a fresh document per parse; "reused" parses into the same document |
| `json_view` (editable) | index into the text plus edits | yes | yes, in place | `json_editable_document`; keeps member order and number spellings |
| yyjson | immutable (`yyjson_read`) | yes | via a mutable copy | | | yyjson | immutable (`yyjson_read`) | yes | via a mutable copy | |
| simdjson DOM | immutable, parser reused | yes | no | "fresh" uses a new parser per parse | | simdjson DOM | immutable, parser reused | yes | no | "fresh" uses a new parser per parse |
| simdjson On-Demand | none: forward-only, lazy | no | no | only traverse and select | | simdjson On-Demand | none: forward-only, lazy | no | no | only traverse and select |
+92 -24
View File
@@ -8,8 +8,9 @@
// Same-feature-set benchmark: read-only JSON documents with random access. // Same-feature-set benchmark: read-only JSON documents with random access.
// //
// json_view nlohmann/json_view.hpp (fresh document per parse / reused) // json_view nlohmann/json_view.hpp (fresh document per parse / reused;
// yyjson yyjson_read(): immutable document, random access // "(editable)": json_editable_document, same read API)
// yyjson yyjson_read(): immutable document, random access
// simdjson DOM dom::parser (reused, as recommended; "fresh": a new parser // simdjson DOM dom::parser (reused, as recommended; "fresh": a new parser
// per parse): immutable, random access // per parse): immutable, random access
// references (different feature sets): // references (different feature sets):
@@ -19,7 +20,11 @@
// //
// Workloads: parse (build + free), traverse (visit everything, convert every // Workloads: parse (build + free), traverse (visit everything, convert every
// number, touch every string and key), select (a few fields per document), // number, touch every string and key), select (a few fields per document),
// dump (compact serialization of the parsed document). // dump (compact serialization of the parsed document), and parse+dump (a fresh
// document, then its dump, in one timed call). json_view converts floating-point
// numbers lazily (when read or dumped), yyjson and simdjson DOM while parsing:
// for float-heavy documents, traverse (which includes the parse) and parse+dump
// are the fair reading of the parse and dump rows.
// All engines run interleaved in every round, each timed call after an untimed // All engines run interleaved in every round, each timed call after an untimed
// one of the same engine; the best round is reported. // one of the same engine; the best round is reported.
#include <nlohmann/json_view.hpp> #include <nlohmann/json_view.hpp>
@@ -43,6 +48,8 @@
using nlohmann::json; using nlohmann::json;
using nlohmann::json_document; using nlohmann::json_document;
using nlohmann::json_editable_document;
using nlohmann::json_editable_view;
using nlohmann::json_view; using nlohmann::json_view;
static volatile double g_sink; static volatile double g_sink;
@@ -55,7 +62,8 @@ struct stats
// ---------------- traversal ---------------- // ---------------- traversal ----------------
static void walk(json_view v, stats& st) template<class V>
static void walk_view(V v, stats& st)
{ {
++st.nodes; ++st.nodes;
switch (v.type()) switch (v.type())
@@ -64,32 +72,42 @@ static void walk(json_view v, stats& st)
for (auto it = v.begin(); it != v.end(); ++it) for (auto it = v.begin(); it != v.end(); ++it)
{ {
st.str += it.key().size(); st.str += it.key().size();
walk(*it, st); walk_view<V>(*it, st);
} }
break; break;
case json::value_t::array: case json::value_t::array:
for (const json_view e : v) for (const V e : v)
{ {
walk(e, st); walk_view<V>(e, st);
} }
break; break;
case json::value_t::string: case json::value_t::string:
st.str += v.get_string().size(); st.str += v.get_string().size();
break; break;
case json::value_t::number_integer: case json::value_t::number_integer:
st.num += static_cast<double>(v.get<std::int64_t>()); st.num += static_cast<double>(v.template get<std::int64_t>());
break; break;
case json::value_t::number_unsigned: case json::value_t::number_unsigned:
st.num += static_cast<double>(v.get<std::uint64_t>()); st.num += static_cast<double>(v.template get<std::uint64_t>());
break; break;
case json::value_t::number_float: case json::value_t::number_float:
st.num += v.get<double>(); st.num += v.template get<double>();
break; break;
default: default:
break; break;
} }
} }
static void walk(json_view v, stats& st)
{
walk_view<json_view>(v, st);
}
static void walk(json_editable_view v, stats& st)
{
walk_view<json_editable_view>(v, st);
}
static void walk(const json& j, stats& st) static void walk(const json& j, stats& st)
{ {
++st.nodes; ++st.nodes;
@@ -300,20 +318,21 @@ static void walk(const boost::json::value& v, stats& st)
// status (one tweet): id, user.screen_name, retweet_count // status (one tweet): id, user.screen_name, retweet_count
// rpc: method, params.minuend, id // rpc: method, params.minuend, id
static double pick(const std::string& name, json_view r) template<class V>
static double pick_view(const std::string& name, V r)
{ {
double acc = 0; double acc = 0;
if (name == "twitter" || name == "status") if (name == "twitter" || name == "status")
{ {
auto one = [&](json_view s) auto one = [&](V s)
{ {
acc += static_cast<double>(s["id"].get<std::uint64_t>()); acc += static_cast<double>(s["id"].template get<std::uint64_t>());
acc += static_cast<double>(s["user"]["screen_name"].get_string().size()); acc += static_cast<double>(s["user"]["screen_name"].get_string().size());
acc += static_cast<double>(s["retweet_count"].get<std::int64_t>()); acc += static_cast<double>(s["retweet_count"].template get<std::int64_t>());
}; };
if (name == "twitter") if (name == "twitter")
{ {
for (const json_view s : r["statuses"]) for (const V s : r["statuses"])
{ {
one(s); one(s);
} }
@@ -325,20 +344,20 @@ static double pick(const std::string& name, json_view r)
} }
else if (name == "citm_catalog") else if (name == "citm_catalog")
{ {
for (const json_view p : r["performances"]) for (const V p : r["performances"])
{ {
acc += static_cast<double>(p["id"].get<std::uint64_t>() + p["eventId"].get<std::uint64_t>() + p["seatCategories"].size()); acc += static_cast<double>(p["id"].template get<std::uint64_t>() + p["eventId"].template get<std::uint64_t>() + p["seatCategories"].size());
} }
acc += static_cast<double>(r["events"].size()); acc += static_cast<double>(r["events"].size());
} }
else if (name == "canada") else if (name == "canada")
{ {
const json_view g = r["features"][0]["geometry"]; const V g = r["features"][0]["geometry"];
acc += static_cast<double>(r["type"].get_string().size() + g["type"].get_string().size() + g["coordinates"].size()); acc += static_cast<double>(r["type"].get_string().size() + g["type"].get_string().size() + g["coordinates"].size());
} }
else if (name == "jeopardy") else if (name == "jeopardy")
{ {
for (const json_view q : r) for (const V q : r)
{ {
acc += q["round"].get_string() == "Final Jeopardy!" ? 1 : 0; acc += q["round"].get_string() == "Final Jeopardy!" ? 1 : 0;
acc += static_cast<double>(q["category"].get_string().size()); acc += static_cast<double>(q["category"].get_string().size());
@@ -347,11 +366,21 @@ static double pick(const std::string& name, json_view r)
else if (name == "rpc") else if (name == "rpc")
{ {
acc += static_cast<double>(r["method"].get_string().size()); acc += static_cast<double>(r["method"].get_string().size());
acc += static_cast<double>(r["params"]["minuend"].get<std::int64_t>() + r["id"].get<std::int64_t>()); acc += static_cast<double>(r["params"]["minuend"].template get<std::int64_t>() + r["id"].template get<std::int64_t>());
} }
return acc; return acc;
} }
static double pick(const std::string& name, json_view r)
{
return pick_view<json_view>(name, r);
}
static double pick(const std::string& name, json_editable_view r)
{
return pick_view<json_editable_view>(name, r);
}
static double pick(const std::string& name, const json& r) static double pick(const std::string& name, const json& r)
{ {
double acc = 0; double acc = 0;
@@ -634,15 +663,35 @@ int main(int argc, char** argv)
walk(json::parse(dc.text), a); walk(json::parse(dc.text), a);
auto d = json_document::parse(dc.text); auto d = json_document::parse(dc.text);
walk(d.root(), b); walk(d.root(), b);
auto ed = json_editable_document::parse(dc.text);
stats e;
walk(ed.root(), e);
yyjson_doc* y = yyjson_read(dc.text.data(), dc.text.size(), 0); yyjson_doc* y = yyjson_read(dc.text.data(), dc.text.size(), 0);
walk(yyjson_doc_get_root(y), c); walk(yyjson_doc_get_root(y), c);
simdjson::dom::parser p; simdjson::dom::parser p;
walk(p.parse(dc.text).value(), dd); walk(p.parse(dc.text).value(), dd);
const bool ok = a.nodes == b.nodes && a.nodes == c.nodes && a.nodes == dd.nodes && a.str == b.str && a.str == c.str && a.str == dd.str const bool ok = a.nodes == b.nodes && a.nodes == e.nodes && a.str == e.str && std::fabs(a.num - e.num) <= 1e-9 * std::fabs(a.num) && a.nodes == c.nodes && a.nodes == dd.nodes && a.str == b.str && a.str == c.str && a.str == dd.str
&& std::fabs(a.num - b.num) <= 1e-9 * std::fabs(a.num) && std::fabs(a.num - c.num) <= 1e-9 * std::fabs(a.num); && std::fabs(a.num - b.num) <= 1e-9 * std::fabs(a.num) && std::fabs(a.num - c.num) <= 1e-9 * std::fabs(a.num);
const double pa = pick(dc.name, json::parse(dc.text)), pb = pick(dc.name, d.root()), pc = pick(dc.name, yyjson_doc_get_root(y)), pd = pick(dc.name, p.parse(dc.text).value()); const double pa = pick(dc.name, json::parse(dc.text)), pb = pick(dc.name, d.root()), pe = pick(dc.name, ed.root()), pc = pick(dc.name, yyjson_doc_get_root(y)), pd = pick(dc.name, p.parse(dc.text).value());
std::printf("check %-13s traverse %s select %s\n", dc.name.c_str(), ok ? "OK" : "MISMATCH", (pa == pb && pa == pc && pa == pd) ? "OK" : "MISMATCH"); std::printf("check %-13s traverse %s select %s\n", dc.name.c_str(), ok ? "OK" : "MISMATCH", (pa == pb && pa == pe && pa == pc && pa == pd) ? "OK" : "MISMATCH");
yyjson_doc_free(y); yyjson_doc_free(y);
// the dumps of the engines may format floats differently, so only: every dump is non-empty, and the
// dump of json_view (also editable) and of json::parse parse back to the same number of nodes
const std::string dj = json::parse(dc.text).dump();
const std::string dv = d.root().dump();
const std::string de = ed.root().dump();
std::size_t yn = 0;
yyjson_doc* y2 = yyjson_read(dc.text.data(), dc.text.size(), 0);
char* yo = yyjson_write(y2, 0, &yn);
const bool dumps_ok = !dj.empty() && !dv.empty() && !de.empty() && yo != nullptr && yn > 0 && !simdjson::to_string(p.parse(dc.text).value()).empty();
std::free(yo);
yyjson_doc_free(y2);
stats sj_nodes, sv_nodes, se_nodes;
walk(json::parse(dj), sj_nodes);
walk(json::parse(dv), sv_nodes);
walk(json::parse(de), se_nodes);
std::printf("check %-13s dump %s\n", dc.name.c_str(), (dumps_ok && sj_nodes.nodes == a.nodes && sv_nodes.nodes == a.nodes && se_nodes.nodes == a.nodes) ? "OK" : "MISMATCH");
} }
std::FILE* csv = std::fopen("bench_view.csv", "w"); std::FILE* csv = std::fopen("bench_view.csv", "w");
@@ -669,6 +718,7 @@ int main(int argc, char** argv)
workloads.push_back({"parse", { workloads.push_back({"parse", {
{"json_view", [&] { auto d = json_document::parse(s); g_sink = static_cast<double>(d.node_count()); }}, {"json_view", [&] { auto d = json_document::parse(s); g_sink = static_cast<double>(d.node_count()); }},
{"json_view (reused)", [&] { reused.read(s); g_sink = static_cast<double>(reused.node_count()); }}, {"json_view (reused)", [&] { reused.read(s); g_sink = static_cast<double>(reused.node_count()); }},
{"json_view (editable)", [&] { auto d = json_editable_document::parse(s); g_sink = static_cast<double>(d.root().size()); }},
{"yyjson", [&] { yyjson_doc* d = yyjson_read(s.data(), s.size(), 0); g_sink = static_cast<double>(yyjson_doc_get_val_count(d)); yyjson_doc_free(d); }}, {"yyjson", [&] { yyjson_doc* d = yyjson_read(s.data(), s.size(), 0); g_sink = static_cast<double>(yyjson_doc_get_val_count(d)); yyjson_doc_free(d); }},
{"simdjson DOM", [&] { auto e = sj.parse(ps).value_unsafe(); g_sink = e.is_object(); }}, {"simdjson DOM", [&] { auto e = sj.parse(ps).value_unsafe(); g_sink = e.is_object(); }},
{"simdjson DOM (fresh)", [&] { simdjson::dom::parser p; auto e = p.parse(ps).value_unsafe(); g_sink = e.is_object(); }}, {"simdjson DOM (fresh)", [&] { simdjson::dom::parser p; auto e = p.parse(ps).value_unsafe(); g_sink = e.is_object(); }},
@@ -680,6 +730,7 @@ int main(int argc, char** argv)
workloads.push_back({"traverse", { workloads.push_back({"traverse", {
{"json_view", [&] { auto d = json_document::parse(s); stats st; walk(d.root(), st); g_sink = st.num; }}, {"json_view", [&] { auto d = json_document::parse(s); stats st; walk(d.root(), st); g_sink = st.num; }},
{"json_view (reused)", [&] { reused.read(s); stats st; walk(reused.root(), st); g_sink = st.num; }}, {"json_view (reused)", [&] { reused.read(s); stats st; walk(reused.root(), st); g_sink = st.num; }},
{"json_view (editable)", [&] { auto d = json_editable_document::parse(s); stats st; walk(d.root(), st); g_sink = st.num; }},
{"yyjson", [&] { yyjson_doc* d = yyjson_read(s.data(), s.size(), 0); stats st; walk(yyjson_doc_get_root(d), st); g_sink = st.num; yyjson_doc_free(d); }}, {"yyjson", [&] { yyjson_doc* d = yyjson_read(s.data(), s.size(), 0); stats st; walk(yyjson_doc_get_root(d), st); g_sink = st.num; yyjson_doc_free(d); }},
{"simdjson DOM", [&] { stats st; walk(sj.parse(ps).value_unsafe(), st); g_sink = st.num; }}, {"simdjson DOM", [&] { stats st; walk(sj.parse(ps).value_unsafe(), st); g_sink = st.num; }},
{"simdjson OD", [&] { auto d = od.iterate(ps).value_unsafe(); stats st; walk_od(d.get_value().value_unsafe(), st); g_sink = st.num; }}, {"simdjson OD", [&] { auto d = od.iterate(ps).value_unsafe(); stats st; walk_od(d.get_value().value_unsafe(), st); g_sink = st.num; }},
@@ -691,15 +742,31 @@ int main(int argc, char** argv)
workloads.push_back({"select", { workloads.push_back({"select", {
{"json_view", [&] { auto d = json_document::parse(s); g_sink = pick(name, d.root()); }}, {"json_view", [&] { auto d = json_document::parse(s); g_sink = pick(name, d.root()); }},
{"json_view (reused)", [&] { reused.read(s); g_sink = pick(name, reused.root()); }}, {"json_view (reused)", [&] { reused.read(s); g_sink = pick(name, reused.root()); }},
{"json_view (editable)", [&] { auto d = json_editable_document::parse(s); g_sink = pick(name, d.root()); }},
{"yyjson", [&] { yyjson_doc* d = yyjson_read(s.data(), s.size(), 0); g_sink = pick(name, yyjson_doc_get_root(d)); yyjson_doc_free(d); }}, {"yyjson", [&] { yyjson_doc* d = yyjson_read(s.data(), s.size(), 0); g_sink = pick(name, yyjson_doc_get_root(d)); yyjson_doc_free(d); }},
{"simdjson DOM", [&] { g_sink = pick(name, sj.parse(ps).value_unsafe()); }}, {"simdjson DOM", [&] { g_sink = pick(name, sj.parse(ps).value_unsafe()); }},
{"simdjson OD", [&] { auto d = od.iterate(ps).value_unsafe(); g_sink = pick_od(name, d); }}, {"simdjson OD", [&] { auto d = od.iterate(ps).value_unsafe(); g_sink = pick_od(name, d); }},
{"json::parse", [&] { json j = json::parse(s); g_sink = pick(name, j); }}, {"json::parse", [&] { json j = json::parse(s); g_sink = pick(name, j); }},
}}); }});
// pipeline: a fresh document, then a dump of it, in one timed call
// (simdjson DOM reuses its parser, as in the parse row); the traverse
// workload above is already parse+traverse
workloads.push_back({"parse+dump", {
{"json_view", [&] { auto d = json_document::parse(s); std::string o = d.root().dump(); g_sink = static_cast<double>(o.size()); }},
{"json_view (editable)", [&] { auto d = json_editable_document::parse(s); std::string o = d.root().dump(); g_sink = static_cast<double>(o.size()); }},
{"yyjson", [&] { yyjson_doc* d = yyjson_read(s.data(), s.size(), 0); std::size_t n = 0; char* o = yyjson_write(d, 0, &n); g_sink = static_cast<double>(n); std::free(o); yyjson_doc_free(d); }},
{"simdjson DOM", [&] { std::string o = simdjson::to_string(sj.parse(ps).value_unsafe()); g_sink = static_cast<double>(o.size()); }},
#if JSON_VIEW_BENCH_BOOST
{"Boost.JSON", [&] { boost::json::monotonic_resource mr; auto v = boost::json::parse(s, &mr); std::string o = boost::json::serialize(v); g_sink = static_cast<double>(o.size()); }},
#endif
{"json::parse", [&] { json j = json::parse(s); std::string o = j.dump(); g_sink = static_cast<double>(o.size()); }},
}});
{ {
// serialization of an already parsed document // serialization of an already parsed document
static json_document vd; static json_document vd;
vd.read(s); vd.read(s);
static json_editable_document ed = json_editable_document::parse(s);
ed = json_editable_document::parse(s);
static yyjson_doc* yd = nullptr; static yyjson_doc* yd = nullptr;
if (yd) if (yd)
{ {
@@ -713,6 +780,7 @@ int main(int argc, char** argv)
jd = json::parse(s); jd = json::parse(s);
workloads.push_back({"dump", { workloads.push_back({"dump", {
{"json_view", [&] { std::string o = vd.root().dump(); g_sink = static_cast<double>(o.size()); }}, {"json_view", [&] { std::string o = vd.root().dump(); g_sink = static_cast<double>(o.size()); }},
{"json_view (editable)", [&] { std::string o = ed.root().dump(); g_sink = static_cast<double>(o.size()); }},
{"yyjson", [&] { std::size_t n = 0; char* o = yyjson_write(yd, 0, &n); g_sink = static_cast<double>(n); std::free(o); }}, {"yyjson", [&] { std::size_t n = 0; char* o = yyjson_write(yd, 0, &n); g_sink = static_cast<double>(n); std::free(o); }},
{"simdjson DOM", [&] { std::string o = simdjson::to_string(se); g_sink = static_cast<double>(o.size()); }}, {"simdjson DOM", [&] { std::string o = simdjson::to_string(se); g_sink = static_cast<double>(o.size()); }},
{"json::parse", [&] { std::string o = jd.dump(); g_sink = static_cast<double>(o.size()); }}, {"json::parse", [&] { std::string o = jd.dump(); g_sink = static_cast<double>(o.size()); }},
@@ -740,7 +808,7 @@ int main(int argc, char** argv)
} }
} }
const double ref = best[0]; const double ref = best[0];
std::printf("%-13s %-9s", dc.name.c_str(), wl.first.c_str()); std::printf("%-13s %-14s", dc.name.c_str(), wl.first.c_str());
for (std::size_t k = 0; k < wl.second.size(); ++k) for (std::size_t k = 0; k < wl.second.size(); ++k)
{ {
const double us = best[k] / 1e3; const double us = best[k] / 1e3;
+47 -25
View File
@@ -239,6 +239,8 @@ def main():
ap.add_argument('--download', action='store_true', help='use pinned downloads instead of system libraries') ap.add_argument('--download', action='store_true', help='use pinned downloads instead of system libraries')
ap.add_argument('--no-boost', action='store_true', help='skip Boost.JSON') ap.add_argument('--no-boost', action='store_true', help='skip Boost.JSON')
ap.add_argument('--native', action='store_true', help='compile for this CPU (-march=native / -mcpu=native)') ap.add_argument('--native', action='store_true', help='compile for this CPU (-march=native / -mcpu=native)')
ap.add_argument('--aligned', action='store_true',
help='also build and run with all functions (and loops) aligned, to make code layout artifacts visible')
ap.add_argument('--rounds', type=int, default=30, help='rounds of bench_view (default: 30)') ap.add_argument('--rounds', type=int, default=30, help='rounds of bench_view (default: 30)')
ap.add_argument('--corpus', nargs='*', default=[], help='more files for bench_corpus') ap.add_argument('--corpus', nargs='*', default=[], help='more files for bench_corpus')
ap.add_argument('--build-dir', default=os.path.join(HERE, 'build'), help='where to build (default: build/ next to this script)') ap.add_argument('--build-dir', default=os.path.join(HERE, 'build'), help='where to build (default: build/ next to this script)')
@@ -282,32 +284,46 @@ def main():
+ ['-I' + d for d in lib.include]) + ['-I' + d for d in lib.include])
objects.append(obj) objects.append(obj)
binaries = {} cxx_version = output(cxx + ['--version'])
for bench in ['bench_view', 'bench_corpus', 'bench_edit', 'bench_decode']: is_clang = 'clang' in cxx_version.lower()
exe = os.path.join(args.build_dir, bench) # alignment flags: functions and (gcc) loops on 64-byte boundaries
run(cxx + flags + include + [os.path.join(HERE, bench + '.cpp')] + objects + link + ['-o', exe]) align_flags = ['-mllvm', '-align-all-functions=6'] if is_clang else ['-falign-functions=64', '-falign-loops=64']
binaries[bench] = exe
# the builds: the default one and, with --aligned, one with aligned code
variants = [('default build', args.build_dir, [])]
if args.aligned:
variants.append(('aligned build', os.path.join(args.build_dir, 'aligned'), align_flags))
# run: bench_view on its documents, bench_corpus on those and the given files
corpus = [os.path.join(args.data, f) for f in DEFAULT_CORPUS] + args.corpus corpus = [os.path.join(args.data, f) for f in DEFAULT_CORPUS] + args.corpus
outputs = {} # label -> {benchmark: output}
outputs['bench_view'] = run([binaries['bench_view'], args.data, str(args.rounds)], cwd=args.build_dir, all_outputs = {}
capture_output=True, text=True).stdout for label, build_dir, extra in variants:
outputs['bench_corpus'] = run([binaries['bench_corpus']] + corpus, cwd=args.build_dir, os.makedirs(build_dir, exist_ok=True)
capture_output=True, text=True).stdout binaries = {}
outputs['bench_edit'] = run([binaries['bench_edit'], args.data, str(max(1, args.rounds // 2))], cwd=args.build_dir, for bench in ['bench_view', 'bench_corpus', 'bench_edit', 'bench_decode']:
capture_output=True, text=True).stdout exe = os.path.join(build_dir, bench)
outputs['bench_decode'] = run([binaries['bench_decode'], args.data, str(max(1, args.rounds // 2))], cwd=args.build_dir, run(cxx + flags + extra + include + [os.path.join(HERE, bench + '.cpp')] + objects + link + ['-o', exe])
capture_output=True, text=True).stdout binaries[bench] = exe
for name, text in outputs.items():
print(text) # run: bench_view on its documents, bench_corpus on those and the given files
outputs = {}
outputs['bench_view'] = run([binaries['bench_view'], args.data, str(args.rounds)], cwd=build_dir,
capture_output=True, text=True).stdout
outputs['bench_corpus'] = run([binaries['bench_corpus']] + corpus, cwd=build_dir,
capture_output=True, text=True).stdout
outputs['bench_edit'] = run([binaries['bench_edit'], args.data, str(max(1, args.rounds // 2))], cwd=build_dir,
capture_output=True, text=True).stdout
outputs['bench_decode'] = run([binaries['bench_decode'], args.data, str(max(1, args.rounds // 2))],
cwd=build_dir, capture_output=True, text=True).stdout
for name, text in outputs.items():
print(text)
all_outputs[label] = outputs
# results with their metadata # results with their metadata
now = datetime.datetime.now() now = datetime.datetime.now()
host = re.sub(r'[^A-Za-z0-9-]+', '-', platform.node().split('.')[0]) or 'host' host = re.sub(r'[^A-Za-z0-9-]+', '-', platform.node().split('.')[0]) or 'host'
stem = os.path.join(HERE, 'results', f'{now:%Y-%m-%d}-{host}') stem = os.path.join(HERE, 'results', f'{now:%Y-%m-%d}-{host}')
os.makedirs(os.path.dirname(stem), exist_ok=True) os.makedirs(os.path.dirname(stem), exist_ok=True)
cxx_version = output(cxx + ['--version'])
compiler_version = cxx_version.splitlines()[0] if cxx_version else ' '.join(cxx) compiler_version = cxx_version.splitlines()[0] if cxx_version else ' '.join(cxx)
meta = [ meta = [
('date', f'{now:%Y-%m-%d %H:%M}'), ('date', f'{now:%Y-%m-%d %H:%M}'),
@@ -316,6 +332,7 @@ def main():
('OS', f'{platform.system()} {platform.release()} ({platform.machine()})'), ('OS', f'{platform.system()} {platform.release()} ({platform.machine()})'),
('compiler', compiler_version), ('compiler', compiler_version),
('flags', ' '.join(flags)), ('flags', ' '.join(flags)),
('aligned results', ('included (flags: ' + ' '.join(align_flags) + ')') if args.aligned else 'not included'),
('yyjson', libs['yyjson'].version), ('yyjson', libs['yyjson'].version),
('simdjson', libs['simdjson'].version), ('simdjson', libs['simdjson'].version),
('Boost.JSON', libs['boost'].version if with_boost else 'skipped (not found)'), ('Boost.JSON', libs['boost'].version if with_boost else 'skipped (not found)'),
@@ -328,15 +345,20 @@ def main():
f.write('| | |\n|---|---|\n') f.write('| | |\n|---|---|\n')
for key, value in meta: for key, value in meta:
f.write(f'| {key} | {value} |\n') f.write(f'| {key} | {value} |\n')
for name, text in outputs.items(): for label, outputs in all_outputs.items():
f.write(f'\n## {name}\n\n```\n{text.rstrip()}\n```\n') # (without --aligned there is only the default build, and no extra heading level)
prefix = f'{label}: ' if args.aligned else ''
for name, text in outputs.items():
f.write(f'\n## {prefix}{name}\n\n```\n{text.rstrip()}\n```\n')
with open(stem + '.csv', 'w', encoding='utf-8') as out: with open(stem + '.csv', 'w', encoding='utf-8') as out:
out.write(''.join(f'# {key}: {value}\n' for key, value in meta)) out.write(''.join(f'# {key}: {value}\n' for key, value in meta))
for name in ['bench_view', 'bench_corpus', 'bench_edit', 'bench_decode']: for label, build_dir, _ in variants:
path = os.path.join(args.build_dir, name + '.csv') for name in ['bench_view', 'bench_corpus', 'bench_edit', 'bench_decode']:
if os.path.isfile(path): path = os.path.join(build_dir, name + '.csv')
with open(path, encoding='utf-8') as f: if os.path.isfile(path):
out.write(f'# {name}\n' + f.read()) with open(path, encoding='utf-8') as f:
# (the default build keeps its plain section names)
out.write(f'# {name}' + (f' ({label})' if args.aligned else '') + '\n' + f.read())
print(f'results: {stem}.md, {stem}.csv') print(f'results: {stem}.md, {stem}.csv')
+145 -1
View File
@@ -59,7 +59,8 @@ built build(const std::string& text, bool comments, bool trailing_commas, bool s
r.copy.assign(text.begin(), text.end()); r.copy.assign(text.begin(), text.end());
src = r.copy.data(); src = r.copy.data();
} }
r.ok = nlohmann::detail::view::build < FloatType, !nlohmann::detail::abi_config::strict_nul_handling > (*r.data, src, text.size(), comments, trailing_commas, sentinel, r.failure); constexpr bool nul_is_end = !nlohmann::detail::abi_config::strict_nul_handling;
r.ok = nlohmann::detail::view::build<FloatType, nul_is_end>(*r.data, src, text.size(), comments, trailing_commas, sentinel, false, r.failure);
r.data->src = src; r.data->src = src;
r.data->base[0] = src; r.data->base[0] = src;
r.data->base[1] = r.data->arena.data(); r.data->base[1] = r.data->arena.data();
@@ -172,6 +173,22 @@ void check_string(const std::string& content)
} }
} }
// accept() stores no nodes: it must accept and reject what json::accept does, for all options and for inputs with
// and without a terminating NUL
void check_accept(const std::string& text)
{
CAPTURE(text)
const std::vector<char> bytes(text.begin(), text.end());
for (int options = 0; options < 4; ++options)
{
const bool comments = (options & 1) != 0;
const bool trailing_commas = (options & 2) != 0;
const bool accepted = json::accept(text, comments, trailing_commas);
CHECK(nlohmann::json_document::accept(text, comments, trailing_commas) == accepted);
CHECK(nlohmann::json_document::accept(bytes, comments, trailing_commas) == accepted);
}
}
// a small deterministic generator of documents // a small deterministic generator of documents
struct generator struct generator
{ {
@@ -423,6 +440,133 @@ TEST_CASE("json_view builder")
} }
} }
SECTION("accept() without an index")
{
for (const char* text :
{
"", " ", "\n", "null", "nul", "true", "[", "]", "[]", "{}", "[1,2,3]", "[1,2,3", "[1,2,3,]", "{\"a\":1,}", "{\"a\"}", "{\"a\":}",
"[1] x", "[1][2]", "\xEF\xBB\xBF[1]", "[\"a\\", "[\"\\u12\"]", "[\"\\ud800\"]", "[\"\x01\"]", "[\"\xff\"]", "[01]", "[1.]",
"[.5]", "[-]", "[1e]", "// c\n[1]", "/* c */ {\"a\" /* x */ : 1}", "/* c", "[1] // c", "1e400", "-1e400", "123456789012345678901234567890",
"{\"a\":1,\"a\":2}", "[[[]]]", "\"\\u00e4\\n\\ud83d\\ude00\""
})
{
check_accept(text);
}
// a null byte inside the text, and after it
check_accept(std::string("[1,\0]", 5));
check_accept(std::string("[1]\0", 4));
check_accept(std::string("\0", 1));
// generated documents and damaged copies, as for the builder above
generator g;
for (int i = 0; i < 2000; ++i)
{
std::string text;
g.value(text, 0);
check_accept(text);
std::string damaged = text;
const auto at = static_cast<std::size_t>(g.r(static_cast<int>(damaged.size())));
static const char replacements[] = {'x', '"', '\\', ',', ':', ']', '}', '[', '{', '1', '-', '.', 'e', '\0', '\n', '/'}; // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
damaged[at] = replacements[g.r(16)];
check_accept(damaged);
check_accept(text.substr(0, at));
}
// nesting beyond the 64 levels that are kept on the builder's own stack, closed and not closed
for (const std::size_t depth :
{
63, 64, 65, 66, 200, 1000
})
{
check_accept(std::string(depth, '[') + std::string(depth, ']'));
check_accept(std::string(depth, '[') + std::string(depth - 1, ']'));
std::string objects;
for (std::size_t i = 0; i < depth; ++i)
{
objects += "{\"k\":";
}
objects += "1" + std::string(depth, '}');
check_accept(objects);
check_accept(objects.substr(0, objects.size() - 1));
}
// objects with enough members to get a hash index when they are indexed (and duplicate keys)
std::string large = "{";
for (int i = 0; i < 300; ++i)
{
large += (i == 0 ? "\"k" : ",\"k") + std::to_string(i % 200) + "\":" + std::to_string(i);
}
check_accept(large + "}");
check_accept(large);
check_accept("[" + large + "},{" + large.substr(1) + "}]");
// the overflow check uses the floating-point type of the document
using float_json = nlohmann::basic_json<std::map, std::vector, std::string, bool, std::int64_t, std::uint64_t, float>;
for (const char* text :
{
"1e39", "-1e39", "3.4028235e38", "3.4028236e38", "[1.5e38, 3.5e38]", "{\"a\": 1e-50, \"b\": 1e39}", "340282356779733661637539395458142568448"
})
{
CAPTURE(text)
CHECK(nlohmann::basic_json_document<float_json>::accept(text) == float_json::accept(text));
}
// strings are checked and not decoded: runs of text of every length from 0 to 40 between valid and invalid
// escapes, UTF-8 sequences and control characters, in values and in keys, cut near the end of the text
const std::array<const char*, 9> valid = {{"\\n", "\\\"", "\\\\", "\\/", "\\t", "\\u00e4", "\\u20ac", "\\ud83d\\ude00", "\\u0000"}};
const std::array<const char*, 12> invalid = {{"\\x", "\\ud800", "\\udc00", "\\u12G4", "\\u12", "\\ud83dx", "\\ud83d\\u0041", "\\ud83d\\ud83d", "\\", "\x01", "\x1f", "\x7f"}};
const std::array<const char*, 6> utf8 = {{"\xc3\xa4", "\xe2\x82\xac", "\xf0\x9f\x98\x80", "\xc3", "\xff", "\x80"}};
std::mt19937 rng(2029); // NOLINT(cert-msc32-c,cert-msc51-cpp,bugprone-random-generator-seed): reproducible
for (std::size_t run = 0; run <= 40; ++run)
{
for (int i = 0; i < 30; ++i)
{
std::string content = std::string(rng() % 20, 'a');
const auto pieces = 1 + (rng() % 6);
for (unsigned k = 0; k < pieces; ++k)
{
content += valid[rng() % valid.size()];
content += std::string(k == 0 ? run : rng() % 41, static_cast<char>('b' + (rng() % 20)));
switch (rng() % 8)
{
case 0:
content += utf8[rng() % 3];
break;
case 1:
content += utf8[rng() % utf8.size()];
break;
case 2:
content += invalid[rng() % invalid.size()];
break;
default:
break;
}
}
for (const std::string& text :
{
"[\"" + content + "\"]", "{\"" + content + "\":1}", "[\"" + content, "[\"" + content.substr(0, content.size() - (rng() % 4)) + "\"]"
})
{
check_accept(text);
}
}
}
}
SECTION("escapes in a row")
{
for (const char* content :
{
"\\n", "\\n\\t\\\\", "\\u00e9\\u00e9\\u4e2d", "\\u00e9\\u00e", "\\u00e9\\", "\\n\\q", "\\n\\ud83d\\ude00\\n",
"\\ud83d\\ude00\\ud83d", "\\n\\n\\n abc \\n\\n", "\\n\x01", "\\n\xC3\x28", "\\n\\"
})
{
check_accept(std::string("[\"") + content + "\"]");
check_accept(std::string("[\"") + content);
check_accept(std::string("{\"") + content + "\":1}");
}
}
SECTION("test files") SECTION("test files")
{ {
for (const char* name : for (const char* name :
+364
View File
@@ -0,0 +1,364 @@
// __ _____ _____ _____
// __| | __| | | | JSON for Modern C++ (supporting code)
// | | |__ | | | | | | version 3.12.0
// |_____|_____|_____|_|___| https://github.com/nlohmann/json
//
// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
// SPDX-License-Identifier: MIT
#include "doctest_compatibility.h"
#include <nlohmann/json_view.hpp>
using nlohmann::json;
using nlohmann::ordered_json;
using nlohmann::json_document;
using nlohmann::ordered_json_editable_document;
using nlohmann::json_view;
using nlohmann::ordered_json_document;
#include "json_view_test_helpers.hpp"
using json_view_test::generator;
#include <array>
#include <cstddef>
#include <string>
#include <vector>
// These tests were split off unit-json_view_dump.cpp, whose object file got too
// large for the MinGW assembler ("file too big"). Like that file, this one is
// also built with C++17, as it mentions JSON_HAS_CPP_17.
TEST_CASE("json_view comparison")
{
SECTION("equality of the values parse() produces")
{
generator g;
std::vector<std::string> texts;
for (int i = 0; i < 600; ++i)
{
std::string text;
g.value(text, 0);
texts.push_back(text);
// the same value written differently: sorted keys, canonical numbers
texts.push_back(json::parse(text).dump(1));
}
for (std::size_t i = 0; i + 2 < texts.size(); ++i)
{
for (std::size_t k = i; k < i + 3; ++k)
{
CAPTURE(texts[i])
CAPTURE(texts[k])
const json_document a = json_document::parse(texts[i]);
const json_document b = json_document::parse(texts[k]);
const json ja = json::parse(texts[i]);
const json jb = json::parse(texts[k]);
CHECK((a.root() == b.root()) == (ja == jb));
CHECK((a.root() != b.root()) == (ja != jb));
CHECK((a.root() == jb) == (ja == jb));
CHECK((jb == a.root()) == (ja == jb));
CHECK((a.root() != jb) == (ja != jb));
CHECK((jb != a.root()) == (ja != jb));
// ordered_json compares members in order
const ordered_json_document oa = ordered_json_document::parse(texts[i]);
const ordered_json_document ob = ordered_json_document::parse(texts[k]);
const ordered_json oja = ordered_json::parse(texts[i]);
const ordered_json ojb = ordered_json::parse(texts[k]);
CHECK((oa.root() == ob.root()) == (oja == ojb));
CHECK((oa.root() == ojb) == (oja == ojb));
}
}
}
SECTION("objects whose first keys differ or repeat")
{
const std::array<const char*, 17> texts =
{
{
R"({"a":1,"b":2})", R"({"a":1,"c":2})", R"({"b":1,"a":2})", R"({"a":1,"a":2,"b":3})", R"({"a":2,"b":3})",
R"({"a":1,"b":1,"a":2})", R"({"a":2,"b":1})", R"({"a":1,"a":2})", R"({"b":1})", R"({"a":1,"b":2,"c":3,"d":4,"e":5})",
R"({"a":1,"b":2,"c":3,"d":4,"f":5})", R"({"a":1,"b":2,"c":3,"a":4,"e":5})", R"({"a":4,"b":2,"c":3,"e":5})",
R"({"\u0061":1,"b":2})", R"({})", R"([{"a":1,"b":2}])", R"([{"a":1,"c":2}])"
}
};
for (const char* const x : texts)
{
for (const char* const y : texts)
{
CAPTURE(x)
CAPTURE(y)
const ordered_json_document dx = ordered_json_document::parse(x);
const ordered_json_document dy = ordered_json_document::parse(y);
const ordered_json_editable_document ex = ordered_json_editable_document::parse(x);
const ordered_json jx = ordered_json::parse(x);
const ordered_json jy = ordered_json::parse(y);
CHECK((dx.root() == dy.root()) == (jx == jy));
CHECK((dx.root() == jy) == (jx == jy));
CHECK((ex.root() == dy.root()) == (jx == jy));
}
}
}
SECTION("objects with duplicate keys, small and large")
{
// n members with keys in descending order; with duplicates, some keys
// repeat later (the first after unique ones in key order); in reverse,
// the members are in ascending key order
const auto object = [](int n, bool duplicates, bool reverse, int value)
{
std::string s = "{";
for (int i = 0; i < n; ++i)
{
const int k = reverse ? i : n - 1 - i;
s += (i != 0 ? ",\"k" : "\"k") + std::to_string(100 + k) + "\":" + std::to_string(value + k);
}
if (duplicates)
{
for (int k = n / 2; k < n; k += 3)
{
s += ",\"k" + std::to_string(100 + k) + "\":" + std::to_string(value + k);
}
}
return s + "}";
};
std::vector<std::string> texts;
for (const int n :
{
3, 16, 17, 40
})
{
for (const int flags :
{
0, 1, 2, 3
})
{
texts.push_back(object(n, (flags & 1) != 0, (flags & 2) != 0, 0));
}
texts.push_back(object(n, true, false, 1)); // other values
}
// many objects in one comparison (working memory reused across objects)
std::string array = "[";
std::string array_other = "[";
for (std::size_t i = 0; i < texts.size(); ++i)
{
array += (i != 0 ? "," : "") + texts[i];
array_other += (i != 0 ? "," : "") + texts[texts.size() - 1 - i];
}
texts.push_back(array + "]");
texts.push_back(array_other + "]");
for (const std::string& x : texts)
{
for (const std::string& y : texts)
{
CAPTURE(x)
CAPTURE(y)
const ordered_json_document dx = ordered_json_document::parse(x);
const ordered_json_document dy = ordered_json_document::parse(y);
const ordered_json jx = ordered_json::parse(x);
const ordered_json jy = ordered_json::parse(y);
CHECK((dx.root() == dy.root()) == (jx == jy));
CHECK((dx.root() == jy) == (jx == jy));
const json_document ux = json_document::parse(x);
const json uy = json::parse(y);
CHECK((ux.root() == uy) == (json::parse(x) == uy));
}
}
}
SECTION("numbers, duplicate keys, member order")
{
const auto same = [](const char* x, const char* y)
{
const json_document dx = json_document::parse(x);
const json_document dy = json_document::parse(y);
return dx.root() == dy.root();
};
CHECK(same("1", "1.0"));
CHECK(same("[1, -1, 2.5]", "[1.0, -1.0, 25e-1]"));
CHECK(!same("1", "1.5"));
CHECK(same("18446744073709551615", "18446744073709551615"));
CHECK(same(R"({"a": 1, "a": 2})", R"({"a": 2})"));
CHECK(!same(R"({"a": 1, "a": 2})", R"({"a": 1})"));
CHECK(same(R"({"a": 1, "b": 2})", R"({"b": 2, "a": 1})"));
CHECK(!same(R"({"a": 1})", R"({"a": 1, "b": 2})"));
CHECK(!same("[1, 2]", "[2, 1]"));
CHECK(!same("\"a\"", "\"b\""));
CHECK(same("\"\\u00e9\"", "\"\xc3\xa9\""));
CHECK(!same("null", "false"));
CHECK(!same("[]", "{}"));
const ordered_json_document dup = ordered_json_document::parse(R"({"a": 1, "b": 2, "a": 3})");
const ordered_json_document last = ordered_json_document::parse(R"({"a": 3, "b": 2})");
CHECK(dup.root() == last.root());
// (== compares as parse() resolves duplicates, a lookup finds the first)
CHECK(dup.root()["a"].materialize() == 1);
CHECK(dup.root()["a"] != last.root()["a"]);
const ordered_json_document ab = ordered_json_document::parse(R"({"a": 1, "b": 2})");
const ordered_json_document ba = ordered_json_document::parse(R"({"b": 2, "a": 1})");
CHECK(ab.root() != ba.root());
// discarded values compare as basic_json's do
const json discarded(json::value_t::discarded);
CHECK((json_view() == json_view()) == (discarded == discarded)); // NOLINT(readability-container-size-empty): operator== is tested
CHECK((json_view() == discarded) == (discarded == discarded)); // NOLINT(readability-container-size-empty)
const json_document null_document = json_document::parse("null");
CHECK(!(json_view() == null_document.root())); // NOLINT(readability-container-size-empty)
CHECK(!(null_document.root() == discarded));
}
SECTION("values that are the same text, or the same value written differently, at any depth")
{
// nested arrays and objects around depths where the walk over equal text gives up
for (const int depth :
{
1, 5, 47, 48, 49, 50, 60, 300
})
{
CAPTURE(depth)
std::string head;
std::string tail;
for (int i = 0; i < depth; ++i)
{
head += i % 2 == 0 ? R"({"k":[1,"s",{"x":true},)" : R"([null,2.5,{"y":)";
tail.insert(0, i % 2 == 0 ? "]}" : "}]");
}
// the same text, one number changed, and one number written differently
const auto with_number = [&](const char* number)
{
std::string result = head;
result += number;
result += tail;
return result;
};
const std::string text = with_number("7");
const std::string changed = with_number("8");
const std::string spelled = with_number("7.0");
const json_document a = json_document::parse(text);
const json_document b = json_document::parse(text);
const json_document c = json_document::parse(changed);
const json_document d = json_document::parse(spelled);
CHECK(a.root() == b.root());
CHECK(a.root() != c.root());
CHECK(c.root() != a.root());
CHECK(a.root() == d.root());
CHECK(d.root() == a.root());
CHECK(a.root() == json::parse(text));
CHECK(a.root() != json::parse(changed));
CHECK(json::parse(spelled) == a.root());
const nlohmann::json_editable_document e = nlohmann::json_editable_document::parse(text);
CHECK(a.root() == e.root());
CHECK(e.root() == a.root());
CHECK(c.root() != e.root());
CHECK(e.root() == e.root());
const ordered_json_document oa = ordered_json_document::parse(text);
const ordered_json_document oc = ordered_json_document::parse(changed);
CHECK(oa.root() != oc.root());
CHECK(oa.root() == ordered_json::parse(text));
CHECK(oc.root() != ordered_json::parse(text));
}
// not the same text, but the same value
const auto same = [](const char* x, const char* y)
{
const json_document dx = json_document::parse(x);
const json_document dy = json_document::parse(y);
const bool equal = dx.root() == dy.root();
CHECK(equal == (json::parse(x) == json::parse(y)));
CHECK(equal == (dx.root() == json::parse(y)));
return equal;
};
CHECK(same(R"({"a":1,"a":2})", R"({"a":3,"a":2})"));
CHECK(same(R"({"a":1,"b":[1,{"c":2,"c":3}]})", R"({"b":[1,{"c":9,"c":3}],"a":1})"));
CHECK(!same(R"({"a":1,"a":2})", R"({"a":2,"a":3})"));
CHECK(same("[1.0,-0,0,1e2]", "[1.00,0,-0,100.0]"));
CHECK(same(R"(["\u00e9","\n"])", "[\"\xc3\xa9\",\"\\n\"]"));
CHECK(!same(R"({"a":[1,2,{"b":null}]})", R"({"a":[1,2,{"b":false}]})"));
CHECK(!same(R"({"a":[1,2,{"b":null}]})", R"({"a":[1,2,{"c":null}]})"));
CHECK(!same(R"({"a":[1,2,{"b":null}]})", R"({"a":[1,2,{"b":null,"c":1}]})"));
// edited documents: moved sequences, links, and values of new nodes
nlohmann::json_editable_document edited = nlohmann::json_editable_document::parse(R"({"a":[1,2,3],"b":{"c":"d"},"e":0.5})");
const json_document plain = json_document::parse(R"({"a":[1,2,3,4],"b":{"c":"x","y":null},"e":0.5,"z":[]})");
edited.push_back(edited.root()["a"], 4);
edited.set(edited.root()["b"], "c", "x");
edited.set(edited.root()["b"], "y", nullptr);
edited.set(edited.root(), "z", json::array());
CHECK(edited.root() == plain.root());
CHECK(plain.root() == edited.root());
edited.erase(edited.root()["a"], 0);
CHECK(edited.root() != plain.root());
CHECK(edited.root()["a"] == json::parse("[2,3,4]"));
const json_document tail = json_document::parse("[2,3,4]");
CHECK(edited.root()["a"] == tail.root());
}
SECTION("large documents, with a difference at any place")
{
const auto make = [](std::size_t count, std::size_t changed, bool swap_members)
{
std::string text = R"({"b":1,"list":[)";
for (std::size_t i = 0; i < count; ++i)
{
text += (i != 0 ? "," : "");
const std::string id = std::to_string(i == changed ? i + 1 : i);
if (swap_members && i % 3 == 0)
{
text += R"({"v":[1.5,"s",{"k":)";
text += id;
text += R"(}],"id":)";
text += id;
text += "}";
}
else
{
text += R"({"id":)";
text += id;
text += R"(,"v":[1.5,"s",{"k":)";
text += id;
text += "}]}";
}
}
return text + R"(],"a":{"y":2,"x":3}})";
};
const std::size_t count = 3000;
for (const std::size_t changed :
{
count, std::size_t{0}, std::size_t{1}, count / 2, count - 1
})
{
for (const bool swap :
{
false, true
})
{
CAPTURE(changed)
CAPTURE(swap)
const std::string text_a = make(count, count, false);
const std::string text_b = make(count, changed, swap);
const json_document a = json_document::parse(text_a);
const json_document b = json_document::parse(text_b);
const bool equal = changed == count;
CHECK((a.root() == b.root()) == equal);
CHECK((b.root() == a.root()) == equal);
CHECK((a.root() == json::parse(text_b)) == equal);
CHECK((json::parse(text_b) == a.root()) == equal);
CHECK((a.root()["list"] == b.root()["list"]) == equal);
const ordered_json_document oa = ordered_json_document::parse(text_a);
const ordered_json_document ob = ordered_json_document::parse(text_b);
CHECK((oa.root() == ob.root()) == (equal && !swap));
CHECK((oa.root() == ordered_json::parse(text_b)) == (equal && !swap));
}
}
}
SECTION("deep nesting")
{
const std::string deep = std::string(100000, '[') + std::string(100000, ']');
const json_document a = json_document::parse(deep);
const json_document b = json_document::parse(deep);
CHECK(a.root() == b.root());
CHECK(a.root() == json::parse(deep));
const std::string other = std::string(100000, '[') + "1" + std::string(100000, ']');
const json_document c = json_document::parse(other);
CHECK(a.root() != c.root());
}
}
-95
View File
@@ -432,101 +432,6 @@ TEST_CASE("json_view dump")
} }
} }
TEST_CASE("json_view comparison")
{
SECTION("equality of the values parse() produces")
{
generator g;
std::vector<std::string> texts;
for (int i = 0; i < 600; ++i)
{
std::string text;
g.value(text, 0);
texts.push_back(text);
// the same value written differently: sorted keys, canonical numbers
texts.push_back(json::parse(text).dump(1));
}
for (std::size_t i = 0; i + 2 < texts.size(); ++i)
{
for (std::size_t k = i; k < i + 3; ++k)
{
CAPTURE(texts[i])
CAPTURE(texts[k])
const json_document a = json_document::parse(texts[i]);
const json_document b = json_document::parse(texts[k]);
const json ja = json::parse(texts[i]);
const json jb = json::parse(texts[k]);
CHECK((a.root() == b.root()) == (ja == jb));
CHECK((a.root() != b.root()) == (ja != jb));
CHECK((a.root() == jb) == (ja == jb));
CHECK((jb == a.root()) == (ja == jb));
CHECK((a.root() != jb) == (ja != jb));
CHECK((jb != a.root()) == (ja != jb));
// ordered_json compares members in order
const ordered_json_document oa = ordered_json_document::parse(texts[i]);
const ordered_json_document ob = ordered_json_document::parse(texts[k]);
const ordered_json oja = ordered_json::parse(texts[i]);
const ordered_json ojb = ordered_json::parse(texts[k]);
CHECK((oa.root() == ob.root()) == (oja == ojb));
CHECK((oa.root() == ojb) == (oja == ojb));
}
}
}
SECTION("numbers, duplicate keys, member order")
{
const auto same = [](const char* x, const char* y)
{
const json_document dx = json_document::parse(x);
const json_document dy = json_document::parse(y);
return dx.root() == dy.root();
};
CHECK(same("1", "1.0"));
CHECK(same("[1, -1, 2.5]", "[1.0, -1.0, 25e-1]"));
CHECK(!same("1", "1.5"));
CHECK(same("18446744073709551615", "18446744073709551615"));
CHECK(same(R"({"a": 1, "a": 2})", R"({"a": 2})"));
CHECK(!same(R"({"a": 1, "a": 2})", R"({"a": 1})"));
CHECK(same(R"({"a": 1, "b": 2})", R"({"b": 2, "a": 1})"));
CHECK(!same(R"({"a": 1})", R"({"a": 1, "b": 2})"));
CHECK(!same("[1, 2]", "[2, 1]"));
CHECK(!same("\"a\"", "\"b\""));
CHECK(same("\"\\u00e9\"", "\"\xc3\xa9\""));
CHECK(!same("null", "false"));
CHECK(!same("[]", "{}"));
const ordered_json_document dup = ordered_json_document::parse(R"({"a": 1, "b": 2, "a": 3})");
const ordered_json_document last = ordered_json_document::parse(R"({"a": 3, "b": 2})");
CHECK(dup.root() == last.root());
// (== compares as parse() resolves duplicates, a lookup finds the first)
CHECK(dup.root()["a"].materialize() == 1);
CHECK(dup.root()["a"] != last.root()["a"]);
const ordered_json_document ab = ordered_json_document::parse(R"({"a": 1, "b": 2})");
const ordered_json_document ba = ordered_json_document::parse(R"({"b": 2, "a": 1})");
CHECK(ab.root() != ba.root());
// discarded values compare as basic_json's do
const json discarded(json::value_t::discarded);
CHECK((json_view() == json_view()) == (discarded == discarded)); // NOLINT(readability-container-size-empty): operator== is tested
CHECK((json_view() == discarded) == (discarded == discarded)); // NOLINT(readability-container-size-empty)
const json_document null_document = json_document::parse("null");
CHECK(!(json_view() == null_document.root())); // NOLINT(readability-container-size-empty)
CHECK(!(null_document.root() == discarded));
}
SECTION("deep nesting")
{
const std::string deep = std::string(100000, '[') + std::string(100000, ']');
const json_document a = json_document::parse(deep);
const json_document b = json_document::parse(deep);
CHECK(a.root() == b.root());
CHECK(a.root() == json::parse(deep));
const std::string other = std::string(100000, '[') + "1" + std::string(100000, ']');
const json_document c = json_document::parse(other);
CHECK(a.root() != c.root());
}
}
TEST_CASE("json_view large objects") TEST_CASE("json_view large objects")
{ {
// objects with 128 members or more are looked up with a hash index // objects with 128 members or more are looked up with a hash index
+120
View File
@@ -941,3 +941,123 @@ TEST_CASE("json_view edits: duplicate keys")
CHECK(d.root()["o"]["d"].get<int>() == 0); CHECK(d.root()["o"]["d"].get<int>() == 0);
} }
} }
#if !defined(JSON_NOEXCEPTION)
TEST_CASE("json_view edits: check_utf8 agrees with the byte-wise check")
{
namespace dv = nlohmann::detail::view;
// "ok" or the exception message
const auto run = [](bool fast, const std::string & s) -> std::string
{
try
{
if (fast)
{
dv::check_utf8(s.data(), s.size());
}
else
{
dv::check_utf8_bytewise(reinterpret_cast<const unsigned char*>(s.data()), s.size(), 0);
}
}
catch (const json::type_error& e)
{
return {e.what()};
}
return {"ok"};
};
std::size_t errors = 0;
const auto same = [&](const std::string & s)
{
const std::string a = run(true, s);
const std::string b = run(false, s);
if (a != b)
{
INFO("fast: " << a << ", byte-wise: " << b << ", length " << s.size());
CHECK(a == b);
}
errors += b != "ok" ? 1u : 0u;
};
const std::vector<std::string> pieces =
{
"a", "Z", " ", "\"", "\\", "\n", "\x01", "\x7f", "\xc3\xa4", "\xe4\xb8\xad", "\xf0\x9f\x98\x80",
"\x80", "\xbf", "\xc0\x80", "\xc1\xbf", "\xc2", "\xe0\x80\x80", "\xe0\x9f\xbf", "\xed\xa0\x80", "\xed\xbf\xbf",
"\xf0\x80\x80\x80", "\xf0\x8f\xbf\xbf", "\xf4\x90\x80\x80", "\xf4\x8f\xbf\xbf", "\xf5", "\xf8", "\xff", "\xfe",
"\xe4\xb8", "\xf0\x9f\x98", "\xf0\x9f", "\xf0"
};
// every piece behind prefixes of every length, and followed by more bytes
for (std::size_t prefix = 0; prefix <= 64; ++prefix)
{
for (const auto& piece : pieces)
{
same(std::string(prefix, 'x') + piece);
same(std::string(prefix, 'x') + piece + "yz");
same(std::string(prefix, '"') + piece);
same(std::string(prefix, '\xc3') + piece);
same(std::string(prefix / 2, '\xc3').append(prefix / 2, '\xa4') + piece);
same(std::string(prefix, 'x') + "\xe4\xb8\xad" + piece);
}
}
// an invalid byte at every position of valid strings of every length
for (std::size_t len = 0; len <= 64; ++len)
{
const std::string ascii(len, 'a');
std::string mixed;
while (mixed.size() < len)
{
if (mixed.size() % 7 == 0)
{
mixed += "\xc3\xa4";
}
else if (mixed.size() % 11 == 0)
{
mixed += "\xf0\x9f\x98\x80";
}
else
{
mixed += "a";
}
}
same(ascii);
same(mixed);
for (std::size_t pos = 0; pos < len; ++pos)
{
for (const char bad :
{'\x80', '\xc0', '\xc2', '\xe0', '\xf4', '\xff', '"', '\\', '\x1f'
})
{
std::string s = ascii;
s[pos] = bad;
same(s);
s = mixed;
s[pos] = bad;
same(s);
}
}
}
// random bytes, biased to the interesting ones
std::mt19937 gen(5869); // NOLINT(cert-msc32-c,cert-msc51-cpp,bugprone-random-generator-seed): reproducible
static const std::array<unsigned char, 23> interesting = {{0x00, 0x1f, 0x22, 0x5c, 0x7f, 0x80, 0x8f, 0x90, 0x9f, 0xa0, 0xbf, 0xc0, 0xc2, 0xdf, 0xe0, 0xe1, 0xed, 0xef, 0xf0, 0xf1, 0xf4, 0xf5, 0xff}};
for (int i = 0; i < 100000; ++i)
{
std::string s(gen() % 80, 'a');
for (auto& c : s)
{
const unsigned r = gen() % 8;
if (r == 0)
{
c = static_cast<char>(gen() & 0xff);
}
else if (r == 1)
{
c = static_cast<char>(interesting[gen() % interesting.size()]);
}
}
same(s);
}
CHECK(errors > 1000);
}
#endif
+58
View File
@@ -701,6 +701,64 @@ TEST_CASE("json_view images: ownership")
#if !defined(JSON_NOEXCEPTION) #if !defined(JSON_NOEXCEPTION)
TEST_CASE("json_view images: errors") TEST_CASE("json_view images: errors")
{ {
SECTION("float tokens: a changed byte is accepted if the text is still a number with the digits the node records")
{
for (const char* const token :
{
"-12.375", "0.5", "123456.0", "-0.0", "9999999999999999999.5", "0.000000000000000000000001", "1.25e10", "5E-3"
})
{
CAPTURE(token)
const std::string text = std::string("[") + token + "]";
const json_document d = json_document::parse(text);
const std::vector<std::uint8_t> image = d.save();
const std::size_t at = text_at(image) + 1; // the first byte of the token
const std::uint16_t recorded = node_at(image, 1).extra;
CHECK(loaded_dump(image) == json::parse(text).dump());
for (std::size_t pos = 0; token[pos] != '\0'; ++pos)
{
for (unsigned int byte = 0; byte < 256; ++byte)
{
std::vector<std::uint8_t> changed = image;
changed[at + pos] = static_cast<std::uint8_t>(byte);
std::string t = token;
t[pos] = static_cast<char>(byte);
// the digits of the integer part and of the fraction of the changed token
std::size_t integer = 0;
std::size_t fraction = 0;
std::size_t k = (t[0] == '-') ? 1 : 0;
for (; k < t.size() && t[k] >= '0' && t[k] <= '9'; ++k)
{
++integer;
}
if (k < t.size() && t[k] == '.')
{
for (++k; k < t.size() && t[k] >= '0' && t[k] <= '9'; ++k)
{
++fraction;
}
}
// (as a JSON text, the token may have white space around it)
const bool float_token = json::accept(t) && t.find_first_of(std::string(" \t\r\n\0", 5)) == std::string::npos && (t.find('.') != std::string::npos || t.find_first_of("eE") != std::string::npos) // NOLINT(abseil-string-find-str-contains)
&& ((integer | (fraction << 8U)) == recorded);
CAPTURE(pos)
CAPTURE(byte)
std::string result;
try
{
result = loaded_dump(changed);
}
catch (const json::parse_error& e)
{
CHECK(std::string(e.what()) == check_failed);
result = "rejected";
}
CHECK((result != "rejected") == float_token);
}
}
}
}
SECTION("invalid UTF-8 of a decoded string: the compact dump writes it, the others throw") SECTION("invalid UTF-8 of a decoded string: the compact dump writes it, the others throw")
{ {
const json_document d = json_document::parse(R"({"k\n":"\u00e9x"})"); const json_document d = json_document::parse(R"({"k\n":"\u00e9x"})");