From 540eb7c5f27566b86c674503d8830215eb59ea07 Mon Sep 17 00:00:00 2001 From: Niels Lohmann Date: Wed, 28 Oct 2026 13:47:36 +0100 Subject: [PATCH] Compare JSON libraries by use case in the json_view harness bench_all and bench_rs now time use cases (validate, sum, select, edit, minify, write) instead of library functions. Every library may solve a use case in every way it offers; each variant's result is checked, and three malformed inputs give its validation level (full, no UTF-8, partial). summarize.py reports each library's fastest fully validating variant and a faster one at a lower level, as text and as JSON. Signed-off-by: Niels Lohmann --- docs/mkdocs/docs/images/perf_blog_view.svg | 100 - .../docs/images/perf_blog_view_dump.svg | 106 - tests/benchmarks/json_view/others/README.md | 159 +- .../benchmarks/json_view/others/bench_all.cpp | 2290 +++++++++++++---- .../json_view/others/rust/Cargo.lock | 1 + .../json_view/others/rust/Cargo.toml | 1 + .../json_view/others/rust/src/main.rs | 816 +++++- .../benchmarks/json_view/others/summarize.py | 214 +- 8 files changed, 2831 insertions(+), 856 deletions(-) delete mode 100644 docs/mkdocs/docs/images/perf_blog_view.svg delete mode 100644 docs/mkdocs/docs/images/perf_blog_view_dump.svg diff --git a/docs/mkdocs/docs/images/perf_blog_view.svg b/docs/mkdocs/docs/images/perf_blog_view.svg deleted file mode 100644 index e355db7cf..000000000 --- a/docs/mkdocs/docs/images/perf_blog_view.svg +++ /dev/null @@ -1,100 +0,0 @@ - -Parse throughput of nlohmann/json's json_document and json::parse and five other document parsers, fresh documents, Apple M1 Max -Grouped horizontal bars of parse throughput in GB/s, more is better. twitter.json nlohmann/json json_document 3.45 GB/s; twitter.json yyjson 3.26 GB/s; twitter.json simdjson D​OM 3.13 GB/s; twitter.json sonic-rs (Rust) 2.09 GB/s; twitter.json sonic-cpp 1.67 GB/s; twitter.json Boost.J​SON 0.73 GB/s; twitter.json nlohmann/json json::parse 0.47 GB/s; citm_catalog.json nlohmann/json json_document 4.75 GB/s; citm_catalog.json yyjson 3.61 GB/s; citm_catalog.json simdjson D​OM 3.55 GB/s; citm_catalog.json sonic-rs (Rust) 2.60 GB/s; citm_catalog.json sonic-cpp 1.75 GB/s; citm_catalog.json Boost.J​SON 1.16 GB/s; citm_catalog.json nlohmann/json json::parse 0.72 GB/s; canada.json nlohmann/json json_document 2.39 GB/s; canada.json yyjson 1.51 GB/s; canada.json simdjson D​OM 1.07 GB/s; canada.json sonic-rs (Rust) 1.18 GB/s; canada.json sonic-cpp 0.87 GB/s; canada.json Boost.J​SON 0.91 GB/s; canada.json nlohmann/json json::parse 0.38 GB/s; jeopardy.json nlohmann/json json_document 2.45 GB/s; jeopardy.json yyjson 1.83 GB/s; jeopardy.json simdjson D​OM 2.06 GB/s; jeopardy.json sonic-rs (Rust) 1.84 GB/s; jeopardy.json sonic-cpp 1.36 GB/s; jeopardy.json Boost.J​SON 0.62 GB/s; jeopardy.json nlohmann/json json::parse 0.34 GB/s. - -nlohmann/json json_document - -yyjson - -simdjson D​OM - -sonic-rs (Rust) - -sonic-cpp - -Boost.J​SON - -nlohmann/json json::parse - -0.5 - -1 - -1.5 - -2 - -2.5 - -3 - -3.5 - -4 - -4.5 - -5 -GB/s, more is better -twitter.json -nlohmann/json json_document, twitter.json: 3.45 GB/s -3.45 -yyjson, twitter.json: 3.26 GB/s -3.26 -simdjson D​OM, twitter.json: 3.13 GB/s -3.13 -sonic-rs (Rust), twitter.json: 2.09 GB/s -2.09 -sonic-cpp, twitter.json: 1.67 GB/s -1.67 -Boost.J​SON, twitter.json: 0.73 GB/s -0.73 -nlohmann/json json::parse, twitter.json: 0.47 GB/s -0.47 -citm_catalog.json -nlohmann/json json_document, citm_catalog.json: 4.75 GB/s -4.75 -yyjson, citm_catalog.json: 3.61 GB/s -3.61 -simdjson D​OM, citm_catalog.json: 3.55 GB/s -3.55 -sonic-rs (Rust), citm_catalog.json: 2.60 GB/s -2.60 -sonic-cpp, citm_catalog.json: 1.75 GB/s -1.75 -Boost.J​SON, citm_catalog.json: 1.16 GB/s -1.16 -nlohmann/json json::parse, citm_catalog.json: 0.72 GB/s -0.72 -canada.json -nlohmann/json json_document, canada.json: 2.39 GB/s -2.39 -yyjson, canada.json: 1.51 GB/s -1.51 -simdjson D​OM, canada.json: 1.07 GB/s -1.07 -sonic-rs (Rust), canada.json: 1.18 GB/s -1.18 -sonic-cpp, canada.json: 0.87 GB/s -0.87 -Boost.J​SON, canada.json: 0.91 GB/s -0.91 -nlohmann/json json::parse, canada.json: 0.38 GB/s -0.38 -jeopardy.json -nlohmann/json json_document, jeopardy.json: 2.45 GB/s -2.45 -yyjson, jeopardy.json: 1.83 GB/s -1.83 -simdjson D​OM, jeopardy.json: 2.06 GB/s -2.06 -sonic-rs (Rust), jeopardy.json: 1.84 GB/s -1.84 -sonic-cpp, jeopardy.json: 1.36 GB/s -1.36 -Boost.J​SON, jeopardy.json: 0.62 GB/s -0.62 -nlohmann/json json::parse, jeopardy.json: 0.34 GB/s -0.34 - - diff --git a/docs/mkdocs/docs/images/perf_blog_view_dump.svg b/docs/mkdocs/docs/images/perf_blog_view_dump.svg deleted file mode 100644 index 7bb658e13..000000000 --- a/docs/mkdocs/docs/images/perf_blog_view_dump.svg +++ /dev/null @@ -1,106 +0,0 @@ - -Write throughput of nlohmann/json's json_document and json, and five other libraries, each writing a document parsed beforehand, Apple M1 Max -Grouped horizontal bars of write throughput in GB/s, more is better. twitter.json nlohmann/json json_document 6.27 GB/s; twitter.json yyjson 4.48 GB/s; twitter.json simdjson D​OM 1.32 GB/s; twitter.json sonic-rs (Rust) 2.20 GB/s; twitter.json sonic-cpp 4.46 GB/s; twitter.json Boost.J​SON 0.94 GB/s; twitter.json nlohmann/json json::dump 2.15 GB/s; citm_catalog.json nlohmann/json json_document 3.77 GB/s; citm_catalog.json yyjson 3.02 GB/s; citm_catalog.json simdjson D​OM 1.00 GB/s; citm_catalog.json sonic-rs (Rust) 1.13 GB/s; citm_catalog.json sonic-cpp 2.25 GB/s; citm_catalog.json Boost.J​SON 1.00 GB/s; citm_catalog.json nlohmann/json json::dump 1.75 GB/s; canada.json nlohmann/json json_document 1.29 GB/s; canada.json yyjson 1.22 GB/s; canada.json simdjson D​OM 0.31 GB/s; canada.json sonic-rs (Rust) 0.86 GB/s; canada.json sonic-cpp 0.69 GB/s; canada.json Boost.J​SON 0.71 GB/s; canada.json nlohmann/json json::dump 0.93 GB/s; jeopardy.json nlohmann/json json_document 3.56 GB/s; jeopardy.json yyjson 2.61 GB/s; jeopardy.json simdjson D​OM 0.96 GB/s; jeopardy.json sonic-rs (Rust) 1.55 GB/s; jeopardy.json sonic-cpp 2.65 GB/s; jeopardy.json Boost.J​SON 0.83 GB/s; jeopardy.json nlohmann/json json::dump 1.68 GB/s. - -nlohmann/json json_document - -yyjson - -simdjson D​OM - -sonic-rs (Rust) - -sonic-cpp - -Boost.J​SON - -nlohmann/json json::dump - -0.5 - -1 - -1.5 - -2 - -2.5 - -3 - -3.5 - -4 - -4.5 - -5 - -5.5 - -6 - -6.5 -GB/s, more is better -twitter.json -nlohmann/json json_document, twitter.json: 6.27 GB/s -6.27 -yyjson, twitter.json: 4.48 GB/s -4.48 -simdjson D​OM, twitter.json: 1.32 GB/s -1.32 -sonic-rs (Rust), twitter.json: 2.20 GB/s -2.20 -sonic-cpp, twitter.json: 4.46 GB/s -4.46 -Boost.J​SON, twitter.json: 0.94 GB/s -0.94 -nlohmann/json json::dump, twitter.json: 2.15 GB/s -2.15 -citm_catalog.json -nlohmann/json json_document, citm_catalog.json: 3.77 GB/s -3.77 -yyjson, citm_catalog.json: 3.02 GB/s -3.02 -simdjson D​OM, citm_catalog.json: 1.00 GB/s -1.00 -sonic-rs (Rust), citm_catalog.json: 1.13 GB/s -1.13 -sonic-cpp, citm_catalog.json: 2.25 GB/s -2.25 -Boost.J​SON, citm_catalog.json: 1.00 GB/s -1.00 -nlohmann/json json::dump, citm_catalog.json: 1.75 GB/s -1.75 -canada.json -nlohmann/json json_document, canada.json: 1.29 GB/s -1.29 -yyjson, canada.json: 1.22 GB/s -1.22 -simdjson D​OM, canada.json: 0.31 GB/s -0.31 -sonic-rs (Rust), canada.json: 0.86 GB/s -0.86 -sonic-cpp, canada.json: 0.69 GB/s -0.69 -Boost.J​SON, canada.json: 0.71 GB/s -0.71 -nlohmann/json json::dump, canada.json: 0.93 GB/s -0.93 -jeopardy.json -nlohmann/json json_document, jeopardy.json: 3.56 GB/s -3.56 -yyjson, jeopardy.json: 2.61 GB/s -2.61 -simdjson D​OM, jeopardy.json: 0.96 GB/s -0.96 -sonic-rs (Rust), jeopardy.json: 1.55 GB/s -1.55 -sonic-cpp, jeopardy.json: 2.65 GB/s -2.65 -Boost.J​SON, jeopardy.json: 0.83 GB/s -0.83 -nlohmann/json json::dump, jeopardy.json: 1.68 GB/s -1.68 - - diff --git a/tests/benchmarks/json_view/others/README.md b/tests/benchmarks/json_view/others/README.md index 7c69da094..d70ae3678 100644 --- a/tests/benchmarks/json_view/others/README.md +++ b/tests/benchmarks/json_view/others/README.md @@ -7,34 +7,120 @@ built by CMake or run by CI. ## What is measured -The same files and method as `bench_view.cpp`: `twitter.json`, `citm_catalog.json`, `canada.json`, `jeopardy.json` -and one status object of `twitter.json`; every engine runs interleaved in every round, each timed call after an -untimed one, and the best round is reported. +The programs compare **use cases**, not library functions. A library may implement a use case in every reasonable way +(another API, SAX events, in-situ parsing, a struct binding); each such way is a *variant*, named +`: ` (`yyjson: insitu (copy)`, `RapidJSON: SAX`, `nlohmann json_document: accept`). The report shows +each library's fastest correct variant. `nlohmann json` (`basic_json`) and `nlohmann json_document` (with +`json_view`) count as two libraries. -| workload | what it does | +| use case | what it does | |---|---| -| `parse` | build the document and free it | -| `traverse` | parse, then visit every value, convert every number, touch every string and key | -| `select` | parse, then read a few fields of every record (the fields are listed in `bench_view.cpp`) | -| `parse+dump` | parse, then write the document as compact JSON | -| `dump` | write a document parsed beforehand as compact JSON; only the writing is timed | +| `validate` | is this text valid JSON? returns a bool | +| `sum` | visit every value, convert every number, touch every string and key; sum all numbers in document order | +| `select` | read a few fields of every record (the fields of `bench_view.cpp`, e.g. `id`, `user.screen_name` and `retweet_count` of every status) | +| `edit` | change some values of every record and write the document as compact JSON (see below) | +| `minify` | text in, compact text out | +| `write` | write a document parsed beforehand as compact JSON; only the writing is timed | -Before timing, every engine must report the same number of values, the same number of string bytes, and a sum of all -numbers in document order that is bit-identical to `json_document`'s (`json::parse` visits objects in key order and -is compared within 1e-9): every number is converted exactly. For *select*, all engines must return the same value. -For *dump*, every engine's output must read back as the same values with the same sum, so every float is written -exactly. +The edits: -| engine | version | notes | +| file | edit | +|---|---| +| `twitter.json` | for every element of `statuses`: `retweet_count` + 1, erase `entities`, add `"seen": true` | +| status | the same edit on the one status | +| `citm_catalog.json` | for every element of `performances`: `start` + 1, erase `seatMapImage`, add `"seen": true` | +| `canada.json` | `features[0].properties.name` = `"Kanada"`; append `[0.5,0.25]` to the first ring (`geometry.coordinates[0]`) of every feature | +| `jeopardy.json` | for every record: `value` = `null`, erase `air_date`, add `"seen": true` | + +The files are `twitter.json`, `citm_catalog.json`, `canada.json` (nativejson-benchmark), `jeopardy.json`, and the first +status of `twitter.json` (written by `json::dump()`, timed in batches of 200). + +### Input contract + +The input is a `const std::string` holding the file, with at least 64 bytes of spare capacity (`reserve`), so libraries +that need padding (simdjson) can use it without a copy. A variant that modifies the input (in-situ parsing: yyjson +`YYJSON_READ_INSITU`, RapidJSON `ParseInsitu`, simd-json) copies it inside the timed call; its name says `(copy)`. +Each timed call starts from the text and frees everything it built, except variants named `(reused)`, which keep a +parser or document across calls (simdjson's reused parsers, `json_document::read()`, simd-json's buffers). These are +reported but never chosen as a library's best. In Rust, the input is the file as bytes (`&[u8]`), so every library +has to validate UTF-8 itself. + +### Correctness rules + +Before timing, every variant runs on the file and must produce the right result: + +- `validate`: `true`. +- `sum`: the same number of values, the same number of string and key bytes, and a sum of all numbers that is + bit-identical to `json_document`'s (document order), so every number is converted exactly; variants that visit + objects in another order (`nlohmann json: parse`, simd-json and serde_json values) are compared within 1e-9. +- `select`: the same value as `json_document`. +- `edit`, `minify`, `write`: the output, read with `json::parse`, compares equal (`==`) to the expected document (for + `edit`, the edit applied to a `json` value; floats compare exactly, so every float must be written exactly). In + Rust, `serde_json::Value` (with `float_roundtrip`) is the reference. + +A variant that gives a wrong result on a file, or crashes on it, is printed as +`excluded: | | ` and not timed. + +### Validation levels + +Except for `write`, every variant is also probed with three malformed inputs derived from each file: the file +truncated by one byte (after trailing whitespace), a stray `,` before the first `}`, and a `0xFF` byte inside a string +value (the first in the second half of the file, or the last one; never a key, so that a variant that accepts the byte +still finds the keys it looks up). The probes run in a child process, so a crash is caught. The result is the +variant's *validation level*, printed in its cell (`sonic-cpp: Document [no UTF-8] 380.1us`): + +| level | meaning | +|---|---| +| `full` | rejects all three probes | +| `no UTF-8` | rejects the truncated input and the stray comma, accepts the `0xFF` byte | +| `partial` | accepts the truncated input or the stray comma, or crashes on any probe | +| `n/a` | `write`: there is no input to validate (counts as `full`) | + +What a variant accepts is printed as `level: | | | `. A variant's level in +the summary is the worst it got for any file. UTF-8 validation is enabled where a library has a switch (RapidJSON +`kParseValidateEncodingFlag`; the `(no encoding check)` variants leave it off). sonic-cpp 1.0.2 cannot validate +UTF-8 at all; serde_json does not validate UTF-8 in the parts it skips (`IgnoredAny`, unknown struct fields) unless +the input is checked with `std::str::from_utf8` first (the `from_utf8 + ...` variants). + +The sum and select reference values are printed as `check ... reference:` lines; `summarize.py` warns if the C++ and +Rust logs disagree. + +### Variants + +Level on the Mac in brackets where it is not `full` (`P` = partial, `U` = no UTF-8); every `write` variant is `n/a`. + +| library | validate | sum | select | edit | minify | write | +|---|---|---|---|---|---|---| +| nlohmann json_document | `accept`, `parse`, `read (reused)` | `parse`, `read (reused)` | `parse`, `structs (from_json)`, `read (reused)` | `editable`, `editable (source numbers)` (`json_editable_document`: set/erase, `dump` with `number_format::source`) | `parse + dump`, `parse + dump (source numbers)`, `read + dump (reused)` | `dump`, `dump (source numbers)` | +| nlohmann json | `accept`, `parse` | `parse`, `sax` | `parse`, `structs (from_json)` | `parse`, `ordered_json` | `parse + dump` | `dump` | +| yyjson | `read`, `insitu (copy)` | `read`, `insitu (copy)` | `read`, `insitu (copy)` | `mut copy`, `insitu (copy) + mut copy` (`yyjson_doc_mut_copy`) | `read + write`, `insitu (copy) + write` | `write` | +| simdjson | `DOM`, `DOM (reused)` | `DOM`, `DOM (reused)`, `On-Demand`, `On-Demand (reused)` | `DOM`, `DOM (reused)`, `On-Demand` [P], `On-Demand (reused)` [P] | n/a (no mutable document) | `DOM + to_string`, `DOM (reused) + to_string`, `minify` [P] | `to_string` | +| RapidJSON | `SAX` (null handler), `Document`, `insitu (copy)` | `Document`, `SAX`, `insitu (copy)` | `Document`, `insitu (copy)` | `Document`, `insitu (copy)` | `Document + Writer`, `SAX to Writer`, `SAX to Writer (raw numbers)` | `Writer` | +| RapidJSON, `(no encoding check)` | the same [U] | the same [U] | the same [U] | the same [U] | the same [U] | | +| sonic-cpp | `Document` [U] | `Document` [U] | `Document` [U] | `Document` [U] | `Document + Serialize` [U] | `Serialize` | +| Glaze | `validate_json`, `generic` | `generic`, `lazy_json` [P] | `structs` [P], `structs (validate_skipped)`, `generic`, `lazy_json` [P] | `generic` | `generic`, `minify_json` [P] | `generic` | +| Boost.JSON | `SAX` (`basic_parser`), `parse` | `parse`, `SAX` | `parse` | `parse` | `parse + serialize` | `serialize` | +| serde_json | `IgnoredAny` [U], `from_utf8 + IgnoredAny`, `Value` | `visitor`, `Value` | `structs` [U], `from_utf8 + structs`, `Value` | `Value` | `Value` | `Value` | +| sonic-rs | `IgnoredAny`, `Value` | `visitor`, `Value` | `structs`, `Value` | `Value` | `Value` | `Value` | +| simd-json | `tape (copy)`, `borrowed (copy)`, `tape (reused)` | `visitor (copy)`, `borrowed (copy)`, `borrowed (reused)` | `structs (copy)`, `borrowed (copy)` | `owned (copy)`, `borrowed (copy)` | `borrowed (copy)` | `borrowed` | + +`structs` read into types that declare only the fields `select` uses (Glaze with `error_on_unknown_keys = false`, +and with `validate_skipped` to validate what it skips; serde derive; nlohmann/json's `from_json()` of the +`NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE` macro, from a `json` value or from a `json_document`'s view); `visitor` is a serde `DeserializeSeed` that sums without building a value; `SAX` and `sax` are the +libraries' event interfaces with a summing (or, for `validate`, an empty) handler. RapidJSON parses with +`kParseFullPrecisionFlag` (correctly rounded floats; without it, sums and written floats are wrong) and Boost.JSON with +`number_precision::precise`, as every other library rounds correctly by default. + +| library | version | notes | |---|---|---| -| `json_document` | this checkout | `json_view (reused)`: one document whose `read()` reuses its memory | +| `json_document`, `json` | this checkout | | | yyjson | 0.13.0 | | -| simdjson | 4.6.11 | DOM with a reused parser (as recommended) and a fresh one; On-Demand for *traverse* and *select* | +| simdjson | 4.6.11 | | | sonic-cpp | 1.0.2 | | -| RapidJSON | master 24b5e7a8b27f (2025-02-05) | `kParseFullPrecisionFlag` (correctly rounded floats); also default flags and in situ parsing with a copy of the input | -| Glaze | 9.0.0 | `generic_u64` (owning DOM), `lazy_json` (no validation), and for *select* reading into structs that declare only the fields used | -| Boost.JSON | 1.92.0 (Mac), 1.83 (system, x86-64) | monotonic resource | -| sonic-rs, simd-json, serde_json | see `rust/Cargo.lock` | simd-json parses in place: its times include copying the input; serde_json with `float_roundtrip` | +| RapidJSON | master 24b5e7a8b27f (2025-02-05) | | +| Glaze | 9.0.0 | `generic_u64` (owning DOM, member order kept) | +| Boost.JSON | 1.92.0 | monotonic resource | +| sonic-rs, simd-json, serde_json | see `rust/Cargo.lock` | serde_json with `float_roundtrip` | SHA-256 of the downloads: @@ -47,6 +133,12 @@ dd7b033c5bf6e4308615bb7834c541291d5b20f64121d9a68222a39414635517 glaze v9.0.0.t c4a3b310ddd2472416e091067166b0713be97c63f38c212c484ada022fd296ce boost_1_92_0.tar.gz ``` +### Timing + +Every variant of a use case runs interleaved in every round, each timed call after an untimed one, and the best +round is reported (the method of `bench_view.cpp`). `jeopardy.json` (50 MB) gets a fifth of the rounds (at least 3, +at most the given number). + ## Building and running All libraries except yyjson and simdjson are used header-only. With the downloads unpacked into `DL` @@ -63,17 +155,26 @@ c++ -std=c++23 -O3 -DNDEBUG -DRAPIDJSON_NEON \ cd rust && cargo build --release && ./target/release/bench_rs 30 > ../mac-rs-1.log -python3 summarize.py mac-cpp-*.log --rust mac-rs-*.log +python3 summarize.py mac-cpp-*.log --rust mac-rs-*.log --json mac.json ``` +Both programs take ` [rounds] [file] [use case]`: `./bench_all 30 canada edit` times +only the edit of `canada.json`; `""` stands for all files. The checks cover only the selected files and use cases. +bench_all uses `fork()` for the probes and the crash check (macOS, Linux). + On x86-64, use `-march=haswell -DRAPIDJSON_SSE42` instead of `-DRAPIDJSON_NEON` (sonic-cpp needs AVX2), pin the -process to one core, and add the document name as third argument to time one document per process. A fourth -argument times only one workload (`./bench_all 30 "" dump`); `bench_rs` takes the workload as third -argument. Glaze 9 needs -`std::expected`, which Clang 18 cannot use from GCC 13's standard library; `-DBENCH_NO_GLAZE=1` builds without it. +process to one core, and time one file per process. Glaze 9 needs `std::expected`, which Clang 18 and older cannot +use from GCC 13's standard library (Clang 19 and newer can); `-DBENCH_NO_GLAZE=1` builds without it. For Rust, build +with `RUSTFLAGS="-C target-cpu=haswell"` to match `-march=haswell`. ## Reading the results -`summarize.py` prints, for every document and workload, the engines ordered by time and the fastest other engine -relative to `json_document`. Fresh documents are compared with fresh ones; the reused `json_document` is compared -with simdjson's reused parser. On a shared machine, differences of a few percent are noise: run twice and compare. +A log line holds the file, the use case, and one cell per variant: `| : []