amorfo77/json
1
0 Fork 0
mirror of https://github.com/nlohmann/json.git synced 2026-11-04 03:39:22 +00:00
Code Issues Packages Projects Releases Wiki Activity

Measure each library's fastest modes in the json_view harness

Browse Source 
Address the points of a review from the other libraries' side:

- simdjson 5.0.8 (On-Demand structs, key selectors) and Glaze 9.1.0
- number-skipping modes: yyjson and RapidJSON raw numbers, Boost.JSON
  number_precision::none
- reused parsers and allocators for yyjson, RapidJSON and Boost.JSON
- Boost.JSON parse_into, Glaze generic with double numbers, and the
  same struct types for nlohmann, Glaze, simdjson and serde
- minified copies of twitter.json and citm_catalog.json
- a fourth probe (trailing garbage) and Glaze "(strict)" variants
- a new random variant order in every round, as a variant's time could
  depend on the one before it by up to 15 % on the Mac
- --files for other test files and --conformance for JSONTestSuite,
  with conformance-rs.sh for the Rust program

summarize.py also reports each library's best reused variant.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
This commit is contained in:
Niels Lohmann committed 2026-10-30 15:29:51 +01:00
1 parent 647bb98ba6
commit ff2b8ffdac
5 files changed
+1336 -218
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

tests/benchmarks/json_view/others/README.md
+100 -39
View File
@@ -32,17 +32,22 @@ The edits:
| `canada.json` | `features[0].properties.name` = `"Kanada"`; append `[0.5,0.25]` to the first ring (`geometry.coordinates[0]`) of every feature |
| `jeopardy.json` | for every record: `value` = `null`, erase `air_date`, add `"seen": true` |
The files are `twitter.json`, `citm_catalog.json`, `canada.json` (nativejson-benchmark), `jeopardy.json`, and the first
status of `twitter.json` (written by `json::dump()`, timed in batches of 200).
The files are `twitter.json`, `citm_catalog.json`, `canada.json` (nativejson-benchmark), `jeopardy.json`, the first
status of `twitter.json` (`status`; written by `json::dump()`, timed in batches of 200), and `twitter_min` and
`citm_catalog_min`: `twitter.json` and `citm_catalog.json` with all whitespace outside strings removed, built in memory at
startup by the same small function in both programs (`strip_ws`; the bytes inside strings are untouched). The minified
files use the select and edit of their originals.
### Input contract
The input is a `const std::string` holding the file, with at least 64 bytes of spare capacity (`reserve`), so libraries
that need padding (simdjson) can use it without a copy. A variant that modifies the input (in-situ parsing: yyjson
`YYJSON_READ_INSITU`, RapidJSON `ParseInsitu`, simd-json) copies it inside the timed call; its name says `(copy)`.
Each timed call starts from the text and frees everything it built, except variants named `(reused)`, which keep a
parser or document across calls (simdjson's reused parsers, `json_document::read()`, simd-json's buffers). These are
reported but never chosen as a library's best. In Rust, the input is the file as bytes (`&[u8]`), so every library
Each timed call starts from the text and frees everything it built, except variants named `(reused)` or
`(reused allocator)`, which keep a parser, document or allocator across calls (simdjson's reused parsers,
`json_document::read()`, Boost.JSON's reset `parser` and `basic_parser`, yyjson's dynamic allocator, RapidJSON's
user-buffer allocators, simd-json's buffers). These are reported but never chosen as a library's best;
`summarize.py` reports each library's best reused variant separately. In Rust, the input is the file as bytes (`&[u8]`), so every library
has to validate UTF-8 itself.
### Correctness rules
@@ -53,7 +58,8 @@ Before timing, every variant runs on the file and must produce the right result:
- `sum`: the same number of values, the same number of string and key bytes, and a sum of all numbers that is
bit-identical to `json_document`'s (document order), so every number is converted exactly; variants that visit
objects in another order (`nlohmann json: parse`, simd-json and serde_json values) are compared within 1e-9.
- `select`: the same value as `json_document`.
- `select`: the same value as `json_document`. A variant that does not convert numbers (`(raw numbers)`,
`(numbers: none)`) has to convert the numbers it selects exactly itself.
- `edit`, `minify`, `write`: the output, read with `json::parse`, compares equal (`==`) to the expected document (for
`edit`, the edit applied to a `json` value; floats compare exactly, so every float must be written exactly). In
Rust, `serde_json::Value` (with `float_roundtrip`) is the reference.
@@ -63,17 +69,18 @@ A variant that gives a wrong result on a file, or crashes on it, is printed as
### Validation levels
Except for `write`, every variant is also probed with three malformed inputs derived from each file: the file
truncated by one byte (after trailing whitespace), a stray `,` before the first `}`, and a `0xFF` byte inside a string
value (the first in the second half of the file, or the last one; never a key, so that a variant that accepts the byte
still finds the keys it looks up). The probes run in a child process, so a crash is caught. The result is the
Except for `write`, every variant is also probed with four malformed inputs derived from each file: the file
truncated by one byte (after trailing whitespace), the file followed by ` ]x` (trailing garbage), a stray `,` before
the first `}` (before the last `]` in a file without objects), and a `0xFF` byte inside a string value (the first in
the second half of the file, or the last one; never a key, so that a variant that accepts the byte still finds the
keys it looks up; a file without string values gets no `0xFF` probe). The probes run in a child process, so a crash is caught. The result is the
variant's *validation level*, printed in its cell (`sonic-cpp: Document [no UTF-8] 380.1us`):
| level | meaning |
|---|---|
| `full` | rejects all three probes |
| `no UTF-8` | rejects the truncated input and the stray comma, accepts the `0xFF` byte |
| `partial` | accepts the truncated input or the stray comma, or crashes on any probe |
| `full` | rejects all four probes |
| `no UTF-8` | rejects the structural probes, accepts the `0xFF` byte |
| `partial` | accepts the truncated input, the trailing garbage or the stray comma, or crashes on any probe |
| `n/a` | `write`: there is no input to validate (counts as `full`) |
What a variant accepts is printed as `level: <file> <use case> | <variant> | <level> | <what>`. A variant's level in
@@ -85,6 +92,15 @@ the input is checked with `std::str::from_utf8` first (the `from_utf8 + ...` var
The sum and select reference values are printed as `check ... reference:` lines; `summarize.py` warns if the C++ and
Rust logs disagree.
The four probes do not cover everything a parser can get wrong. `--conformance <dir>` runs every `validate` variant
on every file of [JSONTestSuite](https://github.com/nst/JSONTestSuite)'s `test_parsing` (json_test_data has it as
`nst_json_testsuite2/test_parsing`) and prints `conformance: <file> | <variant> | accepted|rejected|crashed`; a `y_`
file must be accepted, an `n_` file rejected, and an `i_` file may go either way. bench_all runs each call in a child
process (a call that hangs for 60 s counts as crashed); a stack overflow aborts bench_rs, so `conformance-rs.sh`
runs it one file per process and, for a file that aborts, one variant per process (`CONF_VARIANT`). The trailing
garbage probe was added after this check: Glaze reads anything after the value unless `validate_trailing_whitespace`
is set.
### Variants
Level on the Mac in brackets where it is not `full` (`P` = partial, `U` = no UTF-8); every `write` variant is `n/a`.
@@ -93,75 +109,118 @@ Level on the Mac in brackets where it is not `full` (`P` = partial, `U` = no UTF
|---|---|---|---|---|---|---|
| nlohmann json_document | `accept`, `parse`, `read (reused)` | `parse`, `read (reused)` | `parse`, `structs (from_json)`, `read (reused)` | `editable`, `editable (source numbers)` (`json_editable_document`: set/erase, `dump` with `number_format::source`) | `parse + dump`, `parse + dump (source numbers)`, `read + dump (reused)` | `dump`, `dump (source numbers)` |
| nlohmann json | `accept`, `parse` | `parse`, `sax` | `parse`, `structs (from_json)` | `parse`, `ordered_json` | `parse + dump` | `dump` |
| yyjson | `read`, `insitu (copy)` | `read`, `insitu (copy)` | `read`, `insitu (copy)` | `mut copy`, `insitu (copy) + mut copy` (`yyjson_doc_mut_copy`) | `read + write`, `insitu (copy) + write` | `write` |
| simdjson | `DOM`, `DOM (reused)` | `DOM`, `DOM (reused)`, `On-Demand`, `On-Demand (reused)` | `DOM`, `DOM (reused)`, `On-Demand` [P], `On-Demand (reused)` [P] | n/a (no mutable document) | `DOM + to_string`, `DOM (reused) + to_string`, `minify` [P] | `to_string` |
| RapidJSON | `SAX` (null handler), `Document`, `insitu (copy)` | `Document`, `SAX`, `insitu (copy)` | `Document`, `insitu (copy)` | `Document`, `insitu (copy)` | `Document + Writer`, `SAX to Writer`, `SAX to Writer (raw numbers)` | `Writer` |
| yyjson | `read`, `insitu (copy)`, `read (raw numbers)`, `read (reused allocator)` | `read`, `insitu (copy)`, `read (reused allocator)` | `read`, `insitu (copy)`, `read (raw numbers)`, `read (reused allocator)` | `mut copy`, `insitu (copy) + mut copy` (`yyjson_doc_mut_copy`) | `read + write`, `insitu (copy) + write`, `read (raw numbers) + write`, `read + write (reused allocator)` | `write` |
| simdjson | `DOM`, `DOM (reused)` | `DOM`, `DOM (reused)`, `On-Demand`, `On-Demand (reused)` | `DOM`, `DOM (reused)`, `On-Demand` [P], `On-Demand (reused)` [P], `On-Demand structs` [P], `On-Demand (key selectors)` [P] | n/a (no mutable document) | `DOM + to_string`, `DOM (reused) + to_string`, `minify` [P] | `to_string` |
| RapidJSON | `SAX` (null handler), `Document`, `insitu (copy)`, `SAX (raw numbers)`, `Document (reused allocator)` | `Document`, `SAX`, `insitu (copy)`, `Document (reused allocator)` | `Document`, `insitu (copy)`, `Document (raw numbers)`, `Document (reused allocator)` | `Document`, `insitu (copy)` | `Document + Writer`, `SAX to Writer`, `SAX to Writer (raw numbers)`, `Document + Writer (reused allocator)` | `Writer` |
| RapidJSON, `(no encoding check)` | the same [U] | the same [U] | the same [U] | the same [U] | the same [U] | |
| sonic-cpp | `Document` [U] | `Document` [U] | `Document` [U] | `Document` [U] | `Document + Serialize` [U] | `Serialize` |
| Glaze | `validate_json`, `generic` | `generic`, `lazy_json` [P] | `structs` [P], `structs (validate_skipped)`, `generic`, `lazy_json` [P] | `generic` | `generic`, `minify_json` [P] | `generic` |
| Boost.JSON | `SAX` (`basic_parser`), `parse` | `parse`, `SAX` | `parse` | `parse` | `parse + serialize` | `serialize` |
| Glaze | `validate_json`, `generic` [P], `generic (double)` [P], `generic (strict)`, `generic (double, strict)` | `generic` [P], `generic (double)` [P], `generic (strict)`, `generic (double, strict)`, `lazy_json` [P], `lazy_json (string_view)` (x) | `structs` [P], `structs (validate_skipped)`, `generic` [P], `generic (double)` [P], `generic (strict)`, `generic (double, strict)`, `lazy_json` [P], `lazy_json (string_view)` (x) | `generic` [P], `generic (double)` (x), `generic (strict)`, `generic (double, strict)` (x) | `generic` [P], `generic (double)` (x), `generic (strict)`, `generic (double, strict)` (x), `minify_json` [P] | `generic`, `generic (double)` (x), `generic (strict)`, `generic (double, strict)` (x) |
| Boost.JSON | `SAX` (`basic_parser`), `parse`, `SAX (numbers: none)`, `parse (numbers: none)`, `SAX (reused)`, `parser (reused)` | `parse`, `SAX`, `SAX (reused)`, `parser (reused)` | `parse`, `parse (numbers: none)` (x), `parser (reused)`, `parse_into structs` | `parse` | `parse + serialize`, `parser (reused) + serialize` | `serialize` |
| serde_json | `IgnoredAny` [U], `from_utf8 + IgnoredAny`, `Value` | `visitor`, `Value` | `structs` [U], `from_utf8 + structs`, `Value` | `Value` | `Value` | `Value` |
| sonic-rs | `IgnoredAny`, `Value` | `visitor`, `Value` | `structs`, `Value` | `Value` | `Value` | `Value` |
| simd-json | `tape (copy)`, `borrowed (copy)`, `tape (reused)` | `visitor (copy)`, `borrowed (copy)`, `borrowed (reused)` | `structs (copy)`, `borrowed (copy)` | `owned (copy)`, `borrowed (copy)` | `borrowed (copy)` | `borrowed` |
`structs` read into types that declare only the fields `select` uses (Glaze with `error_on_unknown_keys = false`,
and with `validate_skipped` to validate what it skips; serde derive; nlohmann/json's `from_json()` of the
`NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE` macro, from a `json` value or from a `json_document`'s view); `visitor` is a serde `DeserializeSeed` that sums without building a value; `SAX` and `sax` are the
libraries' event interfaces with a summing (or, for `validate`, an empty) handler. RapidJSON parses with
`kParseFullPrecisionFlag` (correctly rounded floats; without it, sums and written floats are wrong) and Boost.JSON with
`number_precision::precise`, as every other library rounds correctly by default.
(x): excluded for a wrong result on some file, and so in the summary for every file: `generic (double)` loses the
64-bit ids of `twitter.json` when it writes them; `lazy_json (string_view)` reads the raw bytes of a string, so escaped
strings have the wrong length (`sum`, and the `"` in 19,077 `jeopardy.json` categories in `select`);
`numbers: none` reads every number as 0.
`structs` read into types that declare only the fields `select` uses and hold the same data: nlohmann/json's
`from_json()` of the `NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE` macro (from a `json` value or from a `json_document`'s view)
reads into `nsel`, where the arrays and objects that are only counted (`seatCategories`, the values of `events`,
`coordinates`) are a `count_only` holding their size. Glaze reads into the same types (reflected; `count_only` has a
reader that skips and counts the elements; `error_on_unknown_keys = false`, and with `validate_skipped` to validate
what it skips), and so does simdjson On-Demand (`get<T>()` with `tag_invoke`, fields in document order,
`count_elements()`/`count_fields()`). serde derive reads into the same fields with a `Count` that counts with
`IgnoredAny` (and a `BTreeMap`, as `std::map`). Boost.JSON's `parse_into` has no type that skips any value, so its
counted members only count what it converts into them: `seatCategories` into a counting sequence of an empty described
struct, `coordinates` into three levels of counting sequences down to `double`, and the `events` values into an empty
described struct, so their members are not counted. `key selectors` are simdjson 5.0's `object::for_each<"k1",
...>`, which walks each object once and matches keys with a compile-time perfect hash. `visitor` is a serde
`DeserializeSeed` that sums without building a value; `SAX` and `sax` are the libraries' event interfaces with a
summing (or, for `validate`, an empty) handler. RapidJSON parses with `kParseFullPrecisionFlag` (correctly rounded
floats; without it, sums and written floats are wrong) and Boost.JSON with `number_precision::precise`, as every other
library rounds correctly by default.
Variants named `(raw numbers)` or `(numbers: none)` use a library's mode that validates numbers without converting
them (yyjson `YYJSON_READ_NUMBER_AS_RAW`, RapidJSON `kParseNumbersAsStringsFlag`, Boost.JSON
`number_precision::none`), as `json_document` converts a number only when it is read. In `select` they convert the
selected numbers themselves (`std::from_chars`); Boost.JSON's `none` leaves no number text in its DOM, so it cannot.
yyjson writes a raw number verbatim, so `read (raw numbers) + write` is exact. sonic-cpp 1.0.2 has no parse flags,
and Glaze's generic DOM has no raw number mode (`validate_json` converts no number anyway). `(reused allocator)`:
yyjson's `yyjson_alc_dyn_new()`, RapidJSON's `MemoryPoolAllocator`s over user buffers for the values and the parse
stack; Boost.JSON's `parser (reused)` builds its value in a `monotonic_resource` over a buffer kept across calls. These
buffers grow after a call that needed more, so the timed calls do not allocate. `generic (double)` is `glz::generic`,
which stores every number as a double. The Glaze `(strict)` variants read with `validate_trailing_whitespace` (else
Glaze accepts anything after the value) and write with `escape_control_characters` (else Glaze writes a string's
control characters, e.g. U+0001, unescaped, which is not JSON; `gsoc-2018.json` has some).
| library | version | notes |
|---|---|---|
| `json_document`, `json` | this checkout | |
| yyjson | 0.13.0 | |
| simdjson | 4.6.11 | |
| simdjson | 5.0.8 | |
| sonic-cpp | 1.0.2 | |
| RapidJSON | master 24b5e7a8b27f (2025-02-05) | |
| Glaze | 9.0.0 | `generic_u64` (owning DOM, member order kept) |
| Glaze | 9.1.0 | `generic_u64` and `generic` (owning DOMs, member order kept) |
| Boost.JSON | 1.92.0 | monotonic resource |
| sonic-rs, simd-json, serde_json | see `rust/Cargo.lock` | serde_json with `float_roundtrip` |
SHA-256 of the downloads:
```
dd7b033c5bf6e4308615bb7834c541291d5b20f64121d9a68222a39414635517 glaze v9.0.0.tar.gz
55f8a675397c505f9bee6c160e1c26b0085868f185d04d7a7727531fdbff4bd4 glaze-v9.1.0.tar.gz
30edf9e47c417f653546c39295fdb66ba5f89fb37eb3780122c6ee66605e5837 sonic-cpp v1.0.2.tar.gz
2d2601a82d2d3b7e143a3c8d43ef616671391034bc46891a9816b79cf2d3e7a8 rapidjson 24b5e7a8b27f42fa16b96fc70aade9106cf7102f.tar.gz
34e0f62a2bc11ab20d601e8ca1cc2b2079503aa45119a19133d89d19b94a0fae yyjson 0.13.0.tar.gz
61d948fc24f0d793829ad658058e7597d064988a89b4607ea02e401a82df98ff simdjson v4.6.11.tar.gz
356cc3a2d6a3d0d45bf5eba813e58ef4f2f6d9cc2a25d8aada2563bc13755789 simdjson-v5.0.8.tar.gz
c4a3b310ddd2472416e091067166b0713be97c63f38c212c484ada022fd296ce boost_1_92_0.tar.gz
```
### Timing
Every variant of a use case runs interleaved in every round, each timed call after an untimed one, and the best
round is reported (the method of `bench_view.cpp`). `jeopardy.json` (50 MB) gets a fifth of the rounds (at least 3,
round is reported (the method of `bench_view.cpp`). The order of the variants changes in every round (a shuffle with a
fixed seed; the first round keeps the listed order): on the Mac, a variant's time can depend on which variant ran
before it, by up to 15 %, and a fixed order would give each variant the same neighbour in every round. `jeopardy.json` (50 MB) gets a fifth of the rounds (at least 3,
at most the given number).
## Building and running
All libraries except yyjson and simdjson are used header-only. With the downloads unpacked into `DL`
(`compare.py --download` fetches yyjson, simdjson and Boost into its build directory):
(`compare.py --download` fetches yyjson and Boost into its build directory, and simdjson 4.6.11, so take simdjson
5.0.8 from its release archive):
```sh
cc -O3 -DNDEBUG -c $DL/yyjson-0.13.0/src/yyjson.c -o yyjson.o
c++ -std=c++17 -O3 -DNDEBUG -c $DL/simdjson-4.6.11/singleheader/simdjson.cpp -o simdjson.o
c++ -std=c++17 -O3 -DNDEBUG -c $DL/simdjson-5.0.8/singleheader/simdjson.cpp -o simdjson.o
c++ -std=c++23 -O3 -DNDEBUG -DRAPIDJSON_NEON \
-I ../../../../include -I $DL/yyjson-0.13.0/src -I $DL/simdjson-4.6.11/singleheader -I $DL/boost_1_92_0 \
-I $DL/glaze/include -I $DL/rapidjson/include -I $DL/sonic/include \
-I ../../../../include -I $DL/yyjson-0.13.0/src -I $DL/simdjson-5.0.8/singleheader -I $DL/boost_1_92_0 \
-I $DL/glaze-9.1.0/include -I $DL/rapidjson/include -I $DL/sonic/include \
bench_all.cpp yyjson.o simdjson.o -o bench_all
./bench_all <json_test_data> 30 > mac-cpp-1.log
cd rust && cargo build --release && ./target/release/bench_rs <json_test_data> 30 > ../mac-rs-1.log
cd rust && cargo build --release --locked && ./target/release/bench_rs <json_test_data> 30 > ../mac-rs-1.log
python3 summarize.py mac-cpp-*.log --rust mac-rs-*.log --json mac.json
```
With GCC 16 and Homebrew Clang on macOS, build the three steps with the same flags plus
`--sysroot=$(xcrun --show-sdk-path)` (`gcc-16`, `g++-16`) or `-isysroot $(xcrun --show-sdk-path)`
(`/opt/homebrew/opt/llvm/bin/clang`, `clang++`). The key selectors and simdjson's `tag_invoke` need C++20, so
`bench_all.cpp` needs `-std=c++20` or newer.
Both programs take `<json_test_data> [rounds] [file] [use case]`: `./bench_all <json_test_data> 30 canada edit` times
only the edit of `canada.json`; `""` stands for all files. The checks cover only the selected files and use cases.
only the edit of `canada.json`; `""` stands for all files; the file names are `twitter`, `citm_catalog`, `canada`,
`jeopardy`, `status`, `twitter_min` and `citm_catalog_min`. The checks cover only the selected files and use cases.
bench_all uses `fork()` for the probes and the crash check (macOS, Linux).
`--files <dir> [rounds] [file] [use case]` runs `validate`, `sum`, `minify` and `write` (`select` and `edit` are
defined only for the files above) on every `*.json` file of a directory, named by their stem; files under 500 KB are
timed in batches of about 500 KB of input. It is meant for files the libraries were not tuned on, such as the
`jsonexamples` of [simdjson-data](https://github.com/simdjson/simdjson-data) without `twitter.json`,
`citm_catalog.json` and `canada.json`. `summarize.py` takes such logs as well.
On x86-64, use `-march=haswell -DRAPIDJSON_SSE42` instead of `-DRAPIDJSON_NEON` (sonic-cpp needs AVX2), pin the
process to one core, and time one file per process. Glaze 9 needs `std::expected`, which Clang 18 and older cannot
use from GCC 13's standard library (Clang 19 and newer can); `-DBENCH_NO_GLAZE=1` builds without it. For Rust, build
@@ -172,9 +231,11 @@ with `RUSTFLAGS="-C target-cpu=haswell"` to match `-march=haswell`.
A log line holds the file, the use case, and one cell per variant: `| <library>: <variant> [<level>] <time>us
(<ratio>x)`, the ratio relative to the first variant of the line. `summarize.py` prints, for every use case and file,
(a) per library the best `full` variant and, if faster, the best variant at any level, with their ratios to the best
`full` `nlohmann json_document` variant (`(reused)` variants are never a library's best), (b) all variants with their
levels and times, and (c) what the variants below `full` accept, and the excluded variants. `--json` writes
`{use case: {file: {library: {"best": {"variant", "us", "level"} or null, "best_any": {...}, "variants": {variant:
{"us", "level"}}, "excluded": {...}}}}}` for the chart scripts. Several logs are combined by taking each cell's best
`full` `nlohmann json_document` variant (`(reused)` and `(reused allocator)` variants are never a library's best; a
`reused:` line below the library shows its best reused variant in the same way), (b) all variants with their levels
and times, and (c) what the variants below `full` accept, and the excluded variants. `--json` writes
`{use case: {file: {library: {"best": {"variant", "us", "level"} or null, "best_any": {...}, "best_reused": {...} or
null, "best_reused_any": {...} or null, "variants": {variant: {"us", "level"}}, "excluded": {...}}}}}` for the chart
scripts (`best_reused`: the best full reused variant; `best_reused_any`: at any level). Several logs are combined by taking each cell's best
time; a variant excluded for one file is left out for every file of that use case.
On a shared machine, differences of a few percent are noise: run twice and compare.
tests/benchmarks/json_view/others/bench_all.cpp
+1036 -124
View File
File diff suppressed because it is too large. Load diff
tests/benchmarks/json_view/others/conformance-rs.sh
Executable
+14
View File
@@ -0,0 +1,14 @@
#!/bin/bash
# the Rust conformance run, one file per process; a file that aborts the process (stack overflow) is rerun one
# variant per process, and a variant that aborts is printed as crashed
B=$1; D=$2
VARS=$($B --conformance $D 1 y_structure_lonely_int.json | sed -n 's/^conformance: [^|]* | \(.*\) | [a-z]*$/\1/p')
for f in $(ls $D | grep '\.json$'); do
out=$($B --conformance $D 1 "$f" 2>/dev/null)
if [ $? = 0 ]; then echo "$out" | grep '^conformance:'; continue; fi
while IFS= read -r v; do
o=$(CONF_VARIANT="$v" $B --conformance $D 1 "$f" 2>/dev/null)
if [ $? = 0 ]; then echo "$o" | grep '^conformance:'; else echo "conformance: $f | $v | crashed"; fi
done <<< "$VARS"
done
echo "done: conformance"
tests/benchmarks/json_view/others/rust/src/main.rs
+152 -39
View File
@@ -250,11 +250,40 @@ pick_value!(pick_serde, serde_json::Value);
pick_value!(pick_sonic, sonic_rs::Value);
pick_value!(pick_simd, simd_json::BorrowedValue<'_>);
// into structs that declare only the fields the selection uses; unknown keys are skipped
// into structs that declare only the fields the selection uses, the same data as nlohmann/json's from_json() structs in
// bench_all.cpp (nsel); unknown keys are skipped; Count counts the elements of an array or the members of an object
// (nlohmann's count_only)
mod sel {
use serde::de::IgnoredAny;
use serde::de::{Deserializer, IgnoredAny, MapAccess, SeqAccess, Visitor};
use serde::Deserialize;
use std::collections::HashMap;
use std::collections::BTreeMap;
pub struct Count(pub usize);
impl<'de> Deserialize<'de> for Count {
fn deserialize<D: Deserializer<'de>>(d: D) -> Result<Count, D::Error> {
struct V;
impl<'de> Visitor<'de> for V {
type Value = Count;
fn expecting(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
f.write_str("an array or an object")
}
fn visit_seq<A: SeqAccess<'de>>(self, mut a: A) -> Result<Count, A::Error> {
let mut n = 0;
while a.next_element::<IgnoredAny>()?.is_some() {
n += 1;
}
Ok(Count(n))
}
fn visit_map<A: MapAccess<'de>>(self, mut a: A) -> Result<Count, A::Error> {
let mut n = 0;
while a.next_entry::<IgnoredAny, IgnoredAny>()?.is_some() {
n += 1;
}
Ok(Count(n))
}
}
d.deserialize_any(V)
}
}
#[derive(Deserialize)]
pub struct User {
pub screen_name: String,
@@ -275,18 +304,18 @@ mod sel {
#[serde(rename = "eventId")]
pub event_id: u64,
#[serde(rename = "seatCategories")]
pub seat_categories: Vec<IgnoredAny>,
pub seat_categories: Count,
}
#[derive(Deserialize)]
pub struct Citm {
pub events: HashMap<String, IgnoredAny>,
pub events: BTreeMap<String, Count>,
pub performances: Vec<Performance>,
}
#[derive(Deserialize)]
pub struct Geometry {
#[serde(rename = "type")]
pub type_: String,
pub coordinates: Vec<IgnoredAny>,
pub coordinates: Count,
}
#[derive(Deserialize)]
pub struct Feature {
@@ -355,14 +384,14 @@ fn pick_structs<D: De>(name: &str, s: &[u8]) -> Option<f64> {
"citm_catalog" => {
let c: sel::Citm = D::de(s)?;
for p in &c.performances {
acc += (p.id + p.event_id + p.seat_categories.len() as u64) as f64;
acc += (p.id + p.event_id + p.seat_categories.0 as u64) as f64;
}
acc += c.events.len() as f64;
}
"canada" => {
let c: sel::Canada = D::de(s)?;
let g = &c.features[0].geometry;
acc += (c.type_.len() + g.type_.len() + g.coordinates.len()) as f64;
acc += (c.type_.len() + g.type_.len() + g.coordinates.0) as f64;
}
"jeopardy" => {
let q: Vec<sel::Question> = D::de(s)?;
@@ -568,6 +597,31 @@ fn padded(s: &[u8]) -> Vec<u8> {
r
}
// the minified inputs (twitter_min, citm_catalog_min): the whitespace outside strings removed, the bytes inside strings
// untouched (the same function as strip_ws in bench_all.cpp)
fn strip_ws(s: &[u8]) -> Vec<u8> {
let mut r = Vec::with_capacity(s.len());
let mut in_string = false;
let mut i = 0;
while i < s.len() {
let ch = s[i];
if in_string {
r.push(ch);
if ch == b'\\' && i + 1 < s.len() {
i += 1;
r.push(s[i]);
} else if ch == b'"' {
in_string = false;
}
} else if ch != b' ' && ch != b'\n' && ch != b'\r' && ch != b'\t' {
r.push(ch);
in_string = ch == b'"';
}
i += 1;
}
r
}
// malformed inputs derived from a valid document, as in bench_all.cpp (0xFF in a string value, not a key)
// (what, text, is the 0xFF probe)
fn malformed(s: &[u8]) -> Vec<(&'static str, Vec<u8>, bool)> {
@@ -612,14 +666,26 @@ fn malformed(s: &[u8]) -> Vec<(&'static str, Vec<u8>, bool)> {
while matches!(t.last(), Some(b' ' | b'\n' | b'\r' | b'\t')) {
t.pop();
}
let mut trailing = t.clone();
trailing.extend_from_slice(b" ]x"); // after the value
t.pop();
let mut out = vec![("truncated input", padded(&t), false)];
let mut out = vec![("truncated input", padded(&t), false), ("trailing garbage", padded(&trailing), false)];
let mut t = s.to_vec();
t.insert(first_close.unwrap(), b',');
out.push(("a stray comma before '}'", padded(&t), false));
let mut t = s.to_vec();
t.insert(mid_quote.or(last_quote).unwrap() + 1, 0xFF);
out.push(("invalid UTF-8 (0xFF in a string)", padded(&t), true));
if let Some(c) = first_close {
t.insert(c, b',');
out.push(("a stray comma before '}'", padded(&t), false));
} else {
// a file without objects (--files): the comma goes before the last ']'
let c = (0..t.len()).rev().find(|&i| t[i] == b']').unwrap();
t.insert(c, b',');
out.push(("a stray comma before ']'", padded(&t), false));
}
// a file without string values (--files) gets no UTF-8 probe
if let Some(q) = mid_quote.or(last_quote) {
let mut t = s.to_vec();
t.insert(q + 1, 0xFF);
out.push(("invalid UTF-8 (0xFF in a string)", padded(&t), true));
}
out
}
@@ -629,34 +695,55 @@ fn same_text(out: &[u8], expected: &serde_json::Value) -> bool {
fn main() {
let args: Vec<String> = std::env::args().collect();
if args.len() < 2 {
eprintln!("usage: {} <json_test_data dir> [rounds] [document] [use case]", args[0]);
// --files <dir>: every *.json file of the directory instead of the files below, without select and edit;
// --conformance <dir>: every validate variant on every *.json file of the directory (JSONTestSuite), no timing
let files = args.len() > 1 && args[1] == "--files";
let conformance = args.len() > 1 && args[1] == "--conformance";
let a = if files || conformance { 2 } else { 1 };
if args.len() <= a {
eprintln!("usage: {0} <json_test_data dir> [rounds] [document] [use case]\n {0} --files <dir> [rounds] [document] [use case]\n {0} --conformance <dir>", args[0]);
std::process::exit(1);
}
let t = format!("{}/", args[1]);
let rounds: usize = args.get(2).map(|r| r.parse().unwrap()).unwrap_or(30).max(1);
let only: String = args.get(3).cloned().unwrap_or_default();
let only_uc: String = args.get(4).cloned().unwrap_or_default();
let mut docs: Vec<(String, Vec<u8>, usize)> = Vec::new();
for f in ["nativejson-benchmark/twitter.json", "nativejson-benchmark/citm_catalog.json", "nativejson-benchmark/canada.json", "jeopardy/jeopardy.json"] {
let name = f.split('/').nth(1).unwrap().trim_end_matches(".json").to_string();
docs.push((name, padded(&std::fs::read(format!("{}{}", t, f)).unwrap()), 1));
let t = format!("{}/", args[a]);
let rounds: usize = args.get(a + 1).map(|r| r.parse().unwrap()).unwrap_or(30).max(1);
let only: String = args.get(a + 2).cloned().unwrap_or_default();
let only_uc: String = if conformance { "validate".to_string() } else { args.get(a + 3).cloned().unwrap_or_default() };
// (file, text, batch, the document whose select and edit apply)
let mut docs: Vec<(String, Vec<u8>, usize, String)> = Vec::new();
if files || conformance {
let mut paths: Vec<std::path::PathBuf> = std::fs::read_dir(&t).unwrap().map(|e| e.unwrap().path()).filter(|p| p.extension().map_or(false, |x| x == "json")).collect();
paths.sort();
for p in paths {
// small files are timed in batches of about 500 KB of input, as status is
let text = padded(&std::fs::read(&p).unwrap());
let name = if conformance { p.file_name() } else { p.file_stem() }.unwrap().to_string_lossy().to_string();
let batch = (500000 / text.len().max(1)).max(1);
docs.push((name.clone(), text, batch, name));
}
} else {
for f in ["nativejson-benchmark/twitter.json", "nativejson-benchmark/citm_catalog.json", "nativejson-benchmark/canada.json", "jeopardy/jeopardy.json"] {
let name = f.split('/').nth(1).unwrap().trim_end_matches(".json").to_string();
docs.push((name.clone(), padded(&std::fs::read(format!("{}{}", t, f)).unwrap()), 1, name));
}
let tw: serde_json::Value = serde_json::from_slice(&docs[0].1).unwrap();
docs.push(("status".to_string(), padded(serde_json::to_string(&tw["statuses"][0]).unwrap().as_bytes()), 200, "status".to_string()));
let (tw_min, citm_min) = (padded(&strip_ws(&docs[0].1)), padded(&strip_ws(&docs[1].1)));
docs.push(("twitter_min".to_string(), tw_min, 1, "twitter".to_string()));
docs.push(("citm_catalog_min".to_string(), citm_min, 1, "citm_catalog".to_string()));
}
let tw: serde_json::Value = serde_json::from_slice(&docs[0].1).unwrap();
docs.push(("status".to_string(), padded(serde_json::to_string(&tw["statuses"][0]).unwrap().as_bytes()), 200));
std::panic::set_hook(Box::new(|_| {})); // a panic is a failed call, reported by the checks
let simd_reused = RefCell::new((simd_json::Buffers::default(), Vec::<u8>::new()));
let mut excluded_total = 0;
for (name, s, batch) in &docs {
if !only.is_empty() && *name != only {
for (file, s, batch, base) in &docs {
if !only.is_empty() && *file != only {
continue;
}
let name = name.as_str();
let (file, name) = (file.as_str(), base.as_str());
let s = s.as_slice();
let bad = malformed(s);
let bad = if conformance { Vec::new() } else { malformed(s) };
for uc in ["validate", "sum", "select", "edit", "minify", "write"] {
if !only_uc.is_empty() && uc != only_uc {
if (!only_uc.is_empty() && uc != only_uc) || (files && (uc == "select" || uc == "edit")) {
continue;
}
// write: each library's document, parsed once outside the timer
@@ -753,11 +840,11 @@ fn main() {
match uc {
"sum" => {
sum_serde(s, &mut ref_st);
println!("check {:<13} {:<9} reference: nodes {} str {} num {:.16e}", name, uc, ref_st.nodes, ref_st.str_, ref_st.num);
println!("check {:<13} {:<9} reference: nodes {} str {} num {:.16e}", file, uc, ref_st.nodes, ref_st.str_, ref_st.num);
}
"select" => {
ref_acc = pick_serde(name, &serde_json::from_slice(s).unwrap());
println!("check {:<13} {:<9} reference: {:.16e}", name, uc, ref_acc);
println!("check {:<13} {:<9} reference: {:.16e}", file, uc, ref_acc);
}
"validate" => {}
_ => {
@@ -767,6 +854,20 @@ fn main() {
}
}
}
if conformance {
// CONF_VARIANT=<name>: only this variant (a stack overflow aborts the process, see conformance.sh)
let only_var = std::env::var("CONF_VARIANT").unwrap_or_default();
for var in all.iter().filter(|v| only_var.is_empty() || v.name == only_var) {
let mut r = Res::new(true);
let verdict = match catch_unwind(AssertUnwindSafe(|| (var.f)(s, &mut r))) {
Ok(true) => "accepted",
Ok(false) => "rejected",
Err(_) => "crashed",
};
println!("conformance: {} | {} | {}", file, var.name, verdict);
}
continue;
}
let n_all = all.len();
let mut vs: Vec<Variant> = Vec::new();
let mut levels: Vec<&str> = Vec::new();
@@ -812,25 +913,37 @@ fn main() {
}
level = if structural { "partial" } else if utf8 { "no UTF-8" } else { "full" };
if !notes.is_empty() {
println!("level: {} {} | {} | {} | {}", name, uc, var.name, level, notes.join("; "));
println!("level: {} {} | {} | {} | {}", file, uc, var.name, level, notes.join("; "));
}
}
vs.push(var);
levels.push(level);
} else {
excluded_total += 1;
println!("excluded: {} {} | {} | {}", name, uc, var.name, why);
println!("excluded: {} {} | {} | {}", file, uc, var.name, why);
}
}
println!("check {:<13} {:<9} {} of {} variants give the right result", name, uc, vs.len(), n_all);
println!("check {:<13} {:<9} {} of {} variants give the right result", file, uc, vs.len(), n_all);
if vs.is_empty() {
continue;
}
let r = if s.len() > 10_000_000 { rounds.min((rounds / 5).max(3)) } else { rounds };
let mut best = vec![f64::MAX; vs.len()];
for _ in 0..r {
for (k, var) in vs.iter().enumerate() {
// a new order of the variants in every round (as in bench_all.cpp): a Fisher-Yates shuffle with xorshift64
let mut order: Vec<usize> = (0..vs.len()).collect();
let mut rng: u64 = 12345;
for i in 0..r {
if i > 0 {
for j in (1..order.len()).rev() {
rng ^= rng << 13;
rng ^= rng >> 7;
rng ^= rng << 17;
order.swap(j, (rng % (j as u64 + 1)) as usize);
}
}
for &k in &order {
let var = &vs[k];
let mut res = Res::new(false);
run(var, s, &mut res); // untimed: the previous variant's leftovers are cleaned up here
sink(res.st.num + res.acc);
@@ -844,7 +957,7 @@ fn main() {
best[k] = best[k].min(ns);
}
}
print!("{:<13} {:<9}", name, uc);
print!("{:<13} {:<9}", file, uc);
for (k, var) in vs.iter().enumerate() {
print!(" | {} [{}] {:.1}us ({:.2}x)", var.name, levels[k], best[k] / 1e3, best[k] / best[0]);
}
tests/benchmarks/json_view/others/summarize.py
+34 -16
View File
@@ -16,13 +16,15 @@ comma and invalid UTF-8), "no UTF-8" (accepts only the invalid UTF-8), "partial"
error, or crashes on any probe), or "n/a" (write: no input to validate; counts as full). A variant's level is the
worst it got for any file. Prints, for every use case and file,
(a) per library the best "full" variant and, if faster, the best variant at any level, with their ratios to the
best "full" nlohmann json_document variant (variants named "(reused)" keep a parser or document across calls and
are never chosen as a library's best),
best "full" nlohmann json_document variant (variants named "(reused)" or "(reused allocator)" keep a parser,
document or allocator across calls and are never chosen as a library's best), and in a "reused:" line the
library's best reused variant in the same way,
(b) all variants with their levels and times,
(c) the variants below "full" with what the probes found, and the variants excluded for wrong results.
--json writes {use case: {file: {library: {"best": {"variant", "us", "level"} or null, "best_any": {...},
"variants": {variant: {"us", "level"}}, "excluded": {variant: reason}}}}} for the chart scripts; variant names there
omit the "<library>: " prefix.
"best_reused": {...} or null, "best_reused_any": {...} or null, "variants": {variant: {"us", "level"}}, "excluded":
{variant: reason}}}}} for the chart scripts ("best_reused" is the best full reused variant, "best_reused_any" the best
reused variant at any level); variant names there omit the "<library>: " prefix.
Several logs are combined by taking each cell's best time. A variant excluded for one file (or in one log) is excluded
for every file of that use case. The reference values that bench_all and bench_rs print for sum and select are
@@ -33,7 +35,7 @@ import re
import sys
USE_CASES = ["validate", "sum", "select", "edit", "minify", "write"]
DOCS = ["twitter", "citm_catalog", "canada", "jeopardy", "status"]
DOCS = ["twitter", "citm_catalog", "canada", "jeopardy", "status", "twitter_min", "citm_catalog_min"]
OURS = "nlohmann json_document"
RANK = {"full": 0, "n/a": 0, "no UTF-8": 1, "partial": 2}
CELL = re.compile(r"^(.+) \[(full|no UTF-8|partial|n/a)\] ([\d.]+)us(?: \(([\d.]+)x\))?$")
@@ -49,7 +51,7 @@ def variant(name):
def is_reused(name):
return "(reused)" in name
return re.search(r"\(reused[ )]", name) is not None
def worse(a, b):
@@ -84,6 +86,8 @@ def parse(path, cells, levels, notes, excluded, refs):
m = CELL.match(c.strip())
if m:
key = (head[0], head[1], m[1])
if head[0] not in DOCS: # bench_all --files: the documents are the files of a directory
DOCS.append(head[0])
cells[key] = min(cells.get(key, float("inf")), float(m[3]))
levels[(head[1], m[1])] = worse(levels.get((head[1], m[1])), m[2])
@@ -145,29 +149,43 @@ def main():
libs.setdefault(library(n), {})[n] = t
entry = result.setdefault(uc, {}).setdefault(doc, {})
rows = []
def pick(vs):
"""the best full variant and the best variant at any level"""
full = {n: t for n, t in vs.items() if is_full(levels[(uc, n)])}
return (min(full.items(), key=lambda x: x[1]) if full else None,
min(vs.items(), key=lambda x: x[1]) if vs else None)
def described(x):
return {"variant": variant(x[0]), "us": x[1], "level": levels[(uc, x[0])]} if x else None
for lib, vs in libs.items():
fresh = {n: t for n, t in vs.items() if not is_reused(n)}
full = {n: t for n, t in fresh.items() if is_full(levels[(uc, n)])}
b = min(full.items(), key=lambda x: x[1]) if full else None
a = min(fresh.items(), key=lambda x: x[1]) if fresh else None
b, a = pick({n: t for n, t in vs.items() if not is_reused(n)})
rb, ra = pick({n: t for n, t in vs.items() if is_reused(n)})
entry[lib] = {
"best": {"variant": variant(b[0]), "us": b[1], "level": levels[(uc, b[0])]} if b else None,
"best_any": {"variant": variant(a[0]), "us": a[1], "level": levels[(uc, a[0])]} if a else None,
"best": described(b),
"best_any": described(a),
"best_reused": described(rb),
"best_reused_any": described(ra),
"variants": {variant(n): {"us": t, "level": levels[(uc, n)]} for n, t in sorted(vs.items(), key=lambda x: x[1])},
}
rows.append((lib, b, a))
rows.append((lib, b, a, rb, ra))
for (u, n), whys in excluded.items():
if u == uc:
lib = entry.setdefault(library(n), {"best": None, "best_any": None, "variants": {}})
lib = entry.setdefault(library(n), {"best": None, "best_any": None, "best_reused": None, "best_reused_any": None, "variants": {}})
lib.setdefault("excluded", {})[variant(n)] = "; ".join(sorted(whys))
ours = next((b[1] for lib, b, a in rows if lib == OURS and b), None)
ours = next((b[1] for lib, b, a, rb, ra in rows if lib == OURS and b), None)
print("%s%s" % (doc, "" if ours else " (no full %s result)" % OURS))
rows.sort(key=lambda r: (r[1] is None, r[1][1] if r[1] else r[2][1] if r[2] else 0))
for lib, b, a in rows:
for lib, b, a, rb, ra in rows:
line = " %-24s %s" % (lib, show(*b, ours, uc) if b else "%-40s %-10s %14s %5s" % ("(none)", "", "", ""))
if a and (not b or a[0] != b[0]):
line += " | faster: %s" % show(*a, ours, uc)
print(line)
if ra:
line = " %-22s %s" % ("reused:", show(*rb, ours, uc) if rb else "%-40s %-10s %14s %5s" % ("(none)", "", "", ""))
if not rb or ra[0] != rb[0]:
line += " | faster: %s" % show(*ra, ours, uc)
print(line)
print("\n# (b) All variants")
for uc in USE_CASES: