mirror of
https://github.com/nlohmann/json.git
synced 2026-11-04 11:49:22 +00:00
Compare JSON libraries by use case in the json_view harness
bench_all and bench_rs now time use cases (validate, sum, select, edit, minify, write) instead of library functions. Every library may solve a use case in every way it offers; each variant's result is checked, and three malformed inputs give its validation level (full, no UTF-8, partial). summarize.py reports each library's fastest fully validating variant and a faster one at a lower level, as text and as JSON. Signed-off-by: Niels Lohmann <mail@nlohmann.me>
This commit is contained in:
8 files changed
+2831
-856
No files matched your search
@@ -7,34 +7,120 @@ built by CMake or run by CI.
|
||||
|
||||
## What is measured
|
||||
|
||||
The same files and method as `bench_view.cpp`: `twitter.json`, `citm_catalog.json`, `canada.json`, `jeopardy.json`
|
||||
and one status object of `twitter.json`; every engine runs interleaved in every round, each timed call after an
|
||||
untimed one, and the best round is reported.
|
||||
The programs compare **use cases**, not library functions. A library may implement a use case in every reasonable way
|
||||
(another API, SAX events, in-situ parsing, a struct binding); each such way is a *variant*, named
|
||||
`<library>: <variant>` (`yyjson: insitu (copy)`, `RapidJSON: SAX`, `nlohmann json_document: accept`). The report shows
|
||||
each library's fastest correct variant. `nlohmann json` (`basic_json`) and `nlohmann json_document` (with
|
||||
`json_view`) count as two libraries.
|
||||
|
||||
| workload | what it does |
|
||||
| use case | what it does |
|
||||
|---|---|
|
||||
| `parse` | build the document and free it |
|
||||
| `traverse` | parse, then visit every value, convert every number, touch every string and key |
|
||||
| `select` | parse, then read a few fields of every record (the fields are listed in `bench_view.cpp`) |
|
||||
| `parse+dump` | parse, then write the document as compact JSON |
|
||||
| `dump` | write a document parsed beforehand as compact JSON; only the writing is timed |
|
||||
| `validate` | is this text valid JSON? returns a bool |
|
||||
| `sum` | visit every value, convert every number, touch every string and key; sum all numbers in document order |
|
||||
| `select` | read a few fields of every record (the fields of `bench_view.cpp`, e.g. `id`, `user.screen_name` and `retweet_count` of every status) |
|
||||
| `edit` | change some values of every record and write the document as compact JSON (see below) |
|
||||
| `minify` | text in, compact text out |
|
||||
| `write` | write a document parsed beforehand as compact JSON; only the writing is timed |
|
||||
|
||||
Before timing, every engine must report the same number of values, the same number of string bytes, and a sum of all
|
||||
numbers in document order that is bit-identical to `json_document`'s (`json::parse` visits objects in key order and
|
||||
is compared within 1e-9): every number is converted exactly. For *select*, all engines must return the same value.
|
||||
For *dump*, every engine's output must read back as the same values with the same sum, so every float is written
|
||||
exactly.
|
||||
The edits:
|
||||
|
||||
| engine | version | notes |
|
||||
| file | edit |
|
||||
|---|---|
|
||||
| `twitter.json` | for every element of `statuses`: `retweet_count` + 1, erase `entities`, add `"seen": true` |
|
||||
| status | the same edit on the one status |
|
||||
| `citm_catalog.json` | for every element of `performances`: `start` + 1, erase `seatMapImage`, add `"seen": true` |
|
||||
| `canada.json` | `features[0].properties.name` = `"Kanada"`; append `[0.5,0.25]` to the first ring (`geometry.coordinates[0]`) of every feature |
|
||||
| `jeopardy.json` | for every record: `value` = `null`, erase `air_date`, add `"seen": true` |
|
||||
|
||||
The files are `twitter.json`, `citm_catalog.json`, `canada.json` (nativejson-benchmark), `jeopardy.json`, and the first
|
||||
status of `twitter.json` (written by `json::dump()`, timed in batches of 200).
|
||||
|
||||
### Input contract
|
||||
|
||||
The input is a `const std::string` holding the file, with at least 64 bytes of spare capacity (`reserve`), so libraries
|
||||
that need padding (simdjson) can use it without a copy. A variant that modifies the input (in-situ parsing: yyjson
|
||||
`YYJSON_READ_INSITU`, RapidJSON `ParseInsitu`, simd-json) copies it inside the timed call; its name says `(copy)`.
|
||||
Each timed call starts from the text and frees everything it built, except variants named `(reused)`, which keep a
|
||||
parser or document across calls (simdjson's reused parsers, `json_document::read()`, simd-json's buffers). These are
|
||||
reported but never chosen as a library's best. In Rust, the input is the file as bytes (`&[u8]`), so every library
|
||||
has to validate UTF-8 itself.
|
||||
|
||||
### Correctness rules
|
||||
|
||||
Before timing, every variant runs on the file and must produce the right result:
|
||||
|
||||
- `validate`: `true`.
|
||||
- `sum`: the same number of values, the same number of string and key bytes, and a sum of all numbers that is
|
||||
bit-identical to `json_document`'s (document order), so every number is converted exactly; variants that visit
|
||||
objects in another order (`nlohmann json: parse`, simd-json and serde_json values) are compared within 1e-9.
|
||||
- `select`: the same value as `json_document`.
|
||||
- `edit`, `minify`, `write`: the output, read with `json::parse`, compares equal (`==`) to the expected document (for
|
||||
`edit`, the edit applied to a `json` value; floats compare exactly, so every float must be written exactly). In
|
||||
Rust, `serde_json::Value` (with `float_roundtrip`) is the reference.
|
||||
|
||||
A variant that gives a wrong result on a file, or crashes on it, is printed as
|
||||
`excluded: <file> <use case> | <variant> | <reason>` and not timed.
|
||||
|
||||
### Validation levels
|
||||
|
||||
Except for `write`, every variant is also probed with three malformed inputs derived from each file: the file
|
||||
truncated by one byte (after trailing whitespace), a stray `,` before the first `}`, and a `0xFF` byte inside a string
|
||||
value (the first in the second half of the file, or the last one; never a key, so that a variant that accepts the byte
|
||||
still finds the keys it looks up). The probes run in a child process, so a crash is caught. The result is the
|
||||
variant's *validation level*, printed in its cell (`sonic-cpp: Document [no UTF-8] 380.1us`):
|
||||
|
||||
| level | meaning |
|
||||
|---|---|
|
||||
| `full` | rejects all three probes |
|
||||
| `no UTF-8` | rejects the truncated input and the stray comma, accepts the `0xFF` byte |
|
||||
| `partial` | accepts the truncated input or the stray comma, or crashes on any probe |
|
||||
| `n/a` | `write`: there is no input to validate (counts as `full`) |
|
||||
|
||||
What a variant accepts is printed as `level: <file> <use case> | <variant> | <level> | <what>`. A variant's level in
|
||||
the summary is the worst it got for any file. UTF-8 validation is enabled where a library has a switch (RapidJSON
|
||||
`kParseValidateEncodingFlag`; the `(no encoding check)` variants leave it off). sonic-cpp 1.0.2 cannot validate
|
||||
UTF-8 at all; serde_json does not validate UTF-8 in the parts it skips (`IgnoredAny`, unknown struct fields) unless
|
||||
the input is checked with `std::str::from_utf8` first (the `from_utf8 + ...` variants).
|
||||
|
||||
The sum and select reference values are printed as `check ... reference:` lines; `summarize.py` warns if the C++ and
|
||||
Rust logs disagree.
|
||||
|
||||
### Variants
|
||||
|
||||
Level on the Mac in brackets where it is not `full` (`P` = partial, `U` = no UTF-8); every `write` variant is `n/a`.
|
||||
|
||||
| library | validate | sum | select | edit | minify | write |
|
||||
|---|---|---|---|---|---|---|
|
||||
| nlohmann json_document | `accept`, `parse`, `read (reused)` | `parse`, `read (reused)` | `parse`, `structs (from_json)`, `read (reused)` | `editable`, `editable (source numbers)` (`json_editable_document`: set/erase, `dump` with `number_format::source`) | `parse + dump`, `parse + dump (source numbers)`, `read + dump (reused)` | `dump`, `dump (source numbers)` |
|
||||
| nlohmann json | `accept`, `parse` | `parse`, `sax` | `parse`, `structs (from_json)` | `parse`, `ordered_json` | `parse + dump` | `dump` |
|
||||
| yyjson | `read`, `insitu (copy)` | `read`, `insitu (copy)` | `read`, `insitu (copy)` | `mut copy`, `insitu (copy) + mut copy` (`yyjson_doc_mut_copy`) | `read + write`, `insitu (copy) + write` | `write` |
|
||||
| simdjson | `DOM`, `DOM (reused)` | `DOM`, `DOM (reused)`, `On-Demand`, `On-Demand (reused)` | `DOM`, `DOM (reused)`, `On-Demand` [P], `On-Demand (reused)` [P] | n/a (no mutable document) | `DOM + to_string`, `DOM (reused) + to_string`, `minify` [P] | `to_string` |
|
||||
| RapidJSON | `SAX` (null handler), `Document`, `insitu (copy)` | `Document`, `SAX`, `insitu (copy)` | `Document`, `insitu (copy)` | `Document`, `insitu (copy)` | `Document + Writer`, `SAX to Writer`, `SAX to Writer (raw numbers)` | `Writer` |
|
||||
| RapidJSON, `(no encoding check)` | the same [U] | the same [U] | the same [U] | the same [U] | the same [U] | |
|
||||
| sonic-cpp | `Document` [U] | `Document` [U] | `Document` [U] | `Document` [U] | `Document + Serialize` [U] | `Serialize` |
|
||||
| Glaze | `validate_json`, `generic` | `generic`, `lazy_json` [P] | `structs` [P], `structs (validate_skipped)`, `generic`, `lazy_json` [P] | `generic` | `generic`, `minify_json` [P] | `generic` |
|
||||
| Boost.JSON | `SAX` (`basic_parser`), `parse` | `parse`, `SAX` | `parse` | `parse` | `parse + serialize` | `serialize` |
|
||||
| serde_json | `IgnoredAny` [U], `from_utf8 + IgnoredAny`, `Value` | `visitor`, `Value` | `structs` [U], `from_utf8 + structs`, `Value` | `Value` | `Value` | `Value` |
|
||||
| sonic-rs | `IgnoredAny`, `Value` | `visitor`, `Value` | `structs`, `Value` | `Value` | `Value` | `Value` |
|
||||
| simd-json | `tape (copy)`, `borrowed (copy)`, `tape (reused)` | `visitor (copy)`, `borrowed (copy)`, `borrowed (reused)` | `structs (copy)`, `borrowed (copy)` | `owned (copy)`, `borrowed (copy)` | `borrowed (copy)` | `borrowed` |
|
||||
|
||||
`structs` read into types that declare only the fields `select` uses (Glaze with `error_on_unknown_keys = false`,
|
||||
and with `validate_skipped` to validate what it skips; serde derive; nlohmann/json's `from_json()` of the
|
||||
`NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE` macro, from a `json` value or from a `json_document`'s view); `visitor` is a serde `DeserializeSeed` that sums without building a value; `SAX` and `sax` are the
|
||||
libraries' event interfaces with a summing (or, for `validate`, an empty) handler. RapidJSON parses with
|
||||
`kParseFullPrecisionFlag` (correctly rounded floats; without it, sums and written floats are wrong) and Boost.JSON with
|
||||
`number_precision::precise`, as every other library rounds correctly by default.
|
||||
|
||||
| library | version | notes |
|
||||
|---|---|---|
|
||||
| `json_document` | this checkout | `json_view (reused)`: one document whose `read()` reuses its memory |
|
||||
| `json_document`, `json` | this checkout | |
|
||||
| yyjson | 0.13.0 | |
|
||||
| simdjson | 4.6.11 | DOM with a reused parser (as recommended) and a fresh one; On-Demand for *traverse* and *select* |
|
||||
| simdjson | 4.6.11 | |
|
||||
| sonic-cpp | 1.0.2 | |
|
||||
| RapidJSON | master 24b5e7a8b27f (2025-02-05) | `kParseFullPrecisionFlag` (correctly rounded floats); also default flags and in situ parsing with a copy of the input |
|
||||
| Glaze | 9.0.0 | `generic_u64` (owning DOM), `lazy_json` (no validation), and for *select* reading into structs that declare only the fields used |
|
||||
| Boost.JSON | 1.92.0 (Mac), 1.83 (system, x86-64) | monotonic resource |
|
||||
| sonic-rs, simd-json, serde_json | see `rust/Cargo.lock` | simd-json parses in place: its times include copying the input; serde_json with `float_roundtrip` |
|
||||
| RapidJSON | master 24b5e7a8b27f (2025-02-05) | |
|
||||
| Glaze | 9.0.0 | `generic_u64` (owning DOM, member order kept) |
|
||||
| Boost.JSON | 1.92.0 | monotonic resource |
|
||||
| sonic-rs, simd-json, serde_json | see `rust/Cargo.lock` | serde_json with `float_roundtrip` |
|
||||
|
||||
SHA-256 of the downloads:
|
||||
|
||||
@@ -47,6 +133,12 @@ dd7b033c5bf6e4308615bb7834c541291d5b20f64121d9a68222a39414635517 glaze v9.0.0.t
|
||||
c4a3b310ddd2472416e091067166b0713be97c63f38c212c484ada022fd296ce boost_1_92_0.tar.gz
|
||||
```
|
||||
|
||||
### Timing
|
||||
|
||||
Every variant of a use case runs interleaved in every round, each timed call after an untimed one, and the best
|
||||
round is reported (the method of `bench_view.cpp`). `jeopardy.json` (50 MB) gets a fifth of the rounds (at least 3,
|
||||
at most the given number).
|
||||
|
||||
## Building and running
|
||||
|
||||
All libraries except yyjson and simdjson are used header-only. With the downloads unpacked into `DL`
|
||||
@@ -63,17 +155,26 @@ c++ -std=c++23 -O3 -DNDEBUG -DRAPIDJSON_NEON \
|
||||
|
||||
cd rust && cargo build --release && ./target/release/bench_rs <json_test_data> 30 > ../mac-rs-1.log
|
||||
|
||||
python3 summarize.py mac-cpp-*.log --rust mac-rs-*.log
|
||||
python3 summarize.py mac-cpp-*.log --rust mac-rs-*.log --json mac.json
|
||||
```
|
||||
|
||||
Both programs take `<json_test_data> [rounds] [file] [use case]`: `./bench_all <json_test_data> 30 canada edit` times
|
||||
only the edit of `canada.json`; `""` stands for all files. The checks cover only the selected files and use cases.
|
||||
bench_all uses `fork()` for the probes and the crash check (macOS, Linux).
|
||||
|
||||
On x86-64, use `-march=haswell -DRAPIDJSON_SSE42` instead of `-DRAPIDJSON_NEON` (sonic-cpp needs AVX2), pin the
|
||||
process to one core, and add the document name as third argument to time one document per process. A fourth
|
||||
argument times only one workload (`./bench_all <json_test_data> 30 "" dump`); `bench_rs` takes the workload as third
|
||||
argument. Glaze 9 needs
|
||||
`std::expected`, which Clang 18 cannot use from GCC 13's standard library; `-DBENCH_NO_GLAZE=1` builds without it.
|
||||
process to one core, and time one file per process. Glaze 9 needs `std::expected`, which Clang 18 and older cannot
|
||||
use from GCC 13's standard library (Clang 19 and newer can); `-DBENCH_NO_GLAZE=1` builds without it. For Rust, build
|
||||
with `RUSTFLAGS="-C target-cpu=haswell"` to match `-march=haswell`.
|
||||
|
||||
## Reading the results
|
||||
|
||||
`summarize.py` prints, for every document and workload, the engines ordered by time and the fastest other engine
|
||||
relative to `json_document`. Fresh documents are compared with fresh ones; the reused `json_document` is compared
|
||||
with simdjson's reused parser. On a shared machine, differences of a few percent are noise: run twice and compare.
|
||||
A log line holds the file, the use case, and one cell per variant: `| <library>: <variant> [<level>] <time>us
|
||||
(<ratio>x)`, the ratio relative to the first variant of the line. `summarize.py` prints, for every use case and file,
|
||||
(a) per library the best `full` variant and, if faster, the best variant at any level, with their ratios to the best
|
||||
`full` `nlohmann json_document` variant (`(reused)` variants are never a library's best), (b) all variants with their
|
||||
levels and times, and (c) what the variants below `full` accept, and the excluded variants. `--json` writes
|
||||
`{use case: {file: {library: {"best": {"variant", "us", "level"} or null, "best_any": {...}, "variants": {variant:
|
||||
{"us", "level"}}, "excluded": {...}}}}}` for the chart scripts. Several logs are combined by taking each cell's best
|
||||
time; a variant excluded for one file is left out for every file of that use case.
|
||||
On a shared machine, differences of a few percent are noise: run twice and compare.
|
||||
File diff suppressed because it is too large.
Load diff
@@ -31,6 +31,7 @@ checksum = "f2032f911046de80f0a198e0901378627c33f59ea0ac00e363d481118bd70a53"
|
||||
name = "bench_rs"
|
||||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"serde",
|
||||
"serde_json",
|
||||
"simd-json",
|
||||
"sonic-rs",
|
||||
|
||||
@@ -6,6 +6,7 @@ edition = "2021"
|
||||
[dependencies]
|
||||
simd-json = "*"
|
||||
sonic-rs = "*"
|
||||
serde = { version = "1", features = ["derive"] }
|
||||
serde_json = { version = "*", features = ["float_roundtrip"] }
|
||||
|
||||
[profile.release]
|
||||
|
||||
@@ -6,11 +6,17 @@
|
||||
// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
|
||||
// SPDX-License-Identifier: MIT
|
||||
|
||||
// Same workloads and method as bench_all.cpp: parse (build + free), traverse (parse, then visit every
|
||||
// value, convert every number, touch every string and key), parse+dump, and dump (write a document parsed
|
||||
// beforehand); every engine interleaved, an untimed call before each timed one, best round reported.
|
||||
// simd-json parses in place, so its timings include copying the input into a mutable buffer.
|
||||
// The Rust counterpart of bench_all.cpp: the same use cases (validate, sum, select, edit, minify, write), files,
|
||||
// checks (right result; validation level from malformed-input probes) and timing method (every variant interleaved in every round, each
|
||||
// timed call after an untimed one, best round); see README.md. The input is the file as bytes (&[u8]); simd-json
|
||||
// parses in place, so its variants copy the input inside the timed call.
|
||||
//
|
||||
// Usage: bench_rs <json_test_data dir> [rounds] [document] [use case]
|
||||
use serde::de::{DeserializeOwned, DeserializeSeed, IgnoredAny, MapAccess, SeqAccess, Visitor};
|
||||
use simd_json::prelude::*;
|
||||
use sonic_rs::{JsonContainerTrait, JsonValueMutTrait, JsonValueTrait};
|
||||
use std::cell::RefCell;
|
||||
use std::panic::{catch_unwind, AssertUnwindSafe};
|
||||
use std::time::Instant;
|
||||
|
||||
#[derive(Default, Debug, Clone, Copy)]
|
||||
@@ -20,6 +26,8 @@ struct Stats {
|
||||
nodes: usize,
|
||||
}
|
||||
|
||||
// ---- sum: walk a DOM, visit every value, convert every number, touch every string and key
|
||||
|
||||
fn walk_simd(v: &simd_json::BorrowedValue, st: &mut Stats) {
|
||||
st.nodes += 1;
|
||||
if let Some(o) = v.as_object() {
|
||||
@@ -43,7 +51,6 @@ fn walk_simd(v: &simd_json::BorrowedValue, st: &mut Stats) {
|
||||
}
|
||||
|
||||
fn walk_sonic(v: &sonic_rs::Value, st: &mut Stats) {
|
||||
use sonic_rs::{JsonContainerTrait, JsonValueTrait};
|
||||
st.nodes += 1;
|
||||
if let Some(o) = v.as_object() {
|
||||
for (k, x) in o.iter() {
|
||||
@@ -95,8 +102,458 @@ fn walk_serde(v: &serde_json::Value, st: &mut Stats) {
|
||||
}
|
||||
}
|
||||
|
||||
fn same(a: &Stats, b: &Stats) -> bool {
|
||||
a.nodes == b.nodes && a.str_ == b.str_ && (a.num - b.num).abs() <= 1e-9 * a.num.abs()
|
||||
// the same visit without a DOM: a serde visitor, driven by any serde deserializer, in document order
|
||||
struct Sum<'a>(&'a mut Stats);
|
||||
struct Key<'a>(&'a mut Stats);
|
||||
|
||||
impl<'de> DeserializeSeed<'de> for Sum<'_> {
|
||||
type Value = ();
|
||||
fn deserialize<D: serde::Deserializer<'de>>(self, d: D) -> Result<(), D::Error> {
|
||||
d.deserialize_any(self)
|
||||
}
|
||||
}
|
||||
|
||||
impl<'de> Visitor<'de> for Sum<'_> {
|
||||
type Value = ();
|
||||
fn expecting(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
|
||||
f.write_str("a JSON value")
|
||||
}
|
||||
fn visit_unit<E: serde::de::Error>(self) -> Result<(), E> {
|
||||
self.0.nodes += 1;
|
||||
Ok(())
|
||||
}
|
||||
fn visit_bool<E: serde::de::Error>(self, _: bool) -> Result<(), E> {
|
||||
self.0.nodes += 1;
|
||||
Ok(())
|
||||
}
|
||||
fn visit_i64<E: serde::de::Error>(self, v: i64) -> Result<(), E> {
|
||||
self.0.nodes += 1;
|
||||
self.0.num += v as f64;
|
||||
Ok(())
|
||||
}
|
||||
fn visit_u64<E: serde::de::Error>(self, v: u64) -> Result<(), E> {
|
||||
self.0.nodes += 1;
|
||||
self.0.num += v as f64;
|
||||
Ok(())
|
||||
}
|
||||
fn visit_f64<E: serde::de::Error>(self, v: f64) -> Result<(), E> {
|
||||
self.0.nodes += 1;
|
||||
self.0.num += v;
|
||||
Ok(())
|
||||
}
|
||||
fn visit_str<E: serde::de::Error>(self, v: &str) -> Result<(), E> {
|
||||
self.0.nodes += 1;
|
||||
self.0.str_ += v.len();
|
||||
Ok(())
|
||||
}
|
||||
fn visit_seq<A: SeqAccess<'de>>(self, mut a: A) -> Result<(), A::Error> {
|
||||
let st = self.0;
|
||||
st.nodes += 1;
|
||||
while a.next_element_seed(Sum(&mut *st))?.is_some() {}
|
||||
Ok(())
|
||||
}
|
||||
fn visit_map<A: MapAccess<'de>>(self, mut a: A) -> Result<(), A::Error> {
|
||||
let st = self.0;
|
||||
st.nodes += 1;
|
||||
while a.next_key_seed(Key(&mut *st))?.is_some() {
|
||||
a.next_value_seed(Sum(&mut *st))?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
impl<'de> DeserializeSeed<'de> for Key<'_> {
|
||||
type Value = ();
|
||||
fn deserialize<D: serde::Deserializer<'de>>(self, d: D) -> Result<(), D::Error> {
|
||||
d.deserialize_str(self)
|
||||
}
|
||||
}
|
||||
|
||||
impl<'de> Visitor<'de> for Key<'_> {
|
||||
type Value = ();
|
||||
fn expecting(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
|
||||
f.write_str("a key")
|
||||
}
|
||||
fn visit_str<E: serde::de::Error>(self, v: &str) -> Result<(), E> {
|
||||
self.0.str_ += v.len();
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
fn sum_serde(s: &[u8], st: &mut Stats) -> bool {
|
||||
let mut de = serde_json::Deserializer::from_slice(s);
|
||||
Sum(st).deserialize(&mut de).is_ok() && de.end().is_ok()
|
||||
}
|
||||
|
||||
fn sum_sonic(s: &[u8], st: &mut Stats) -> bool {
|
||||
let mut de = sonic_rs::Deserializer::from_slice(s);
|
||||
Sum(st).deserialize(&mut de).is_ok() && de.end().is_ok()
|
||||
}
|
||||
|
||||
fn sum_simd(s: &[u8], st: &mut Stats) -> bool {
|
||||
let mut b = s.to_vec();
|
||||
match simd_json::Deserializer::from_slice(&mut b) {
|
||||
Ok(mut de) => Sum(st).deserialize(&mut de).is_ok(),
|
||||
Err(_) => false,
|
||||
}
|
||||
}
|
||||
|
||||
// ---- select: read a few fields of every record (the fields of bench_view.cpp)
|
||||
|
||||
// on a DOM; serde_json's is the reference
|
||||
macro_rules! pick_value {
|
||||
($fname:ident, $t:ty) => {
|
||||
fn $fname(name: &str, r: &$t) -> f64 {
|
||||
fn size(v: &$t) -> usize {
|
||||
if let Some(a) = v.as_array() {
|
||||
a.len()
|
||||
} else {
|
||||
v.as_object().unwrap().len()
|
||||
}
|
||||
}
|
||||
fn status(acc: &mut f64, s: &$t) {
|
||||
*acc += s["id"].as_u64().unwrap() as f64;
|
||||
*acc += s["user"]["screen_name"].as_str().unwrap().len() as f64;
|
||||
*acc += s["retweet_count"].as_i64().unwrap() as f64;
|
||||
}
|
||||
let mut acc = 0.0;
|
||||
match name {
|
||||
"twitter" => {
|
||||
for s in r["statuses"].as_array().unwrap().iter() {
|
||||
status(&mut acc, s);
|
||||
}
|
||||
}
|
||||
"status" => status(&mut acc, r),
|
||||
"citm_catalog" => {
|
||||
for p in r["performances"].as_array().unwrap().iter() {
|
||||
acc += (p["id"].as_u64().unwrap() + p["eventId"].as_u64().unwrap() + size(&p["seatCategories"]) as u64) as f64;
|
||||
}
|
||||
acc += size(&r["events"]) as f64;
|
||||
}
|
||||
"canada" => {
|
||||
let g = &r["features"][0]["geometry"];
|
||||
acc += (r["type"].as_str().unwrap().len() + g["type"].as_str().unwrap().len() + size(&g["coordinates"])) as f64;
|
||||
}
|
||||
"jeopardy" => {
|
||||
for q in r.as_array().unwrap().iter() {
|
||||
acc += if q["round"].as_str().unwrap() == "Final Jeopardy!" { 1.0 } else { 0.0 };
|
||||
acc += q["category"].as_str().unwrap().len() as f64;
|
||||
}
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
acc
|
||||
}
|
||||
};
|
||||
}
|
||||
pick_value!(pick_serde, serde_json::Value);
|
||||
pick_value!(pick_sonic, sonic_rs::Value);
|
||||
pick_value!(pick_simd, simd_json::BorrowedValue<'_>);
|
||||
|
||||
// into structs that declare only the fields the selection uses; unknown keys are skipped
|
||||
mod sel {
|
||||
use serde::de::IgnoredAny;
|
||||
use serde::Deserialize;
|
||||
use std::collections::HashMap;
|
||||
#[derive(Deserialize)]
|
||||
pub struct User {
|
||||
pub screen_name: String,
|
||||
}
|
||||
#[derive(Deserialize)]
|
||||
pub struct Status {
|
||||
pub id: u64,
|
||||
pub user: User,
|
||||
pub retweet_count: i64,
|
||||
}
|
||||
#[derive(Deserialize)]
|
||||
pub struct Twitter {
|
||||
pub statuses: Vec<Status>,
|
||||
}
|
||||
#[derive(Deserialize)]
|
||||
pub struct Performance {
|
||||
pub id: u64,
|
||||
#[serde(rename = "eventId")]
|
||||
pub event_id: u64,
|
||||
#[serde(rename = "seatCategories")]
|
||||
pub seat_categories: Vec<IgnoredAny>,
|
||||
}
|
||||
#[derive(Deserialize)]
|
||||
pub struct Citm {
|
||||
pub events: HashMap<String, IgnoredAny>,
|
||||
pub performances: Vec<Performance>,
|
||||
}
|
||||
#[derive(Deserialize)]
|
||||
pub struct Geometry {
|
||||
#[serde(rename = "type")]
|
||||
pub type_: String,
|
||||
pub coordinates: Vec<IgnoredAny>,
|
||||
}
|
||||
#[derive(Deserialize)]
|
||||
pub struct Feature {
|
||||
pub geometry: Geometry,
|
||||
}
|
||||
#[derive(Deserialize)]
|
||||
pub struct Canada {
|
||||
#[serde(rename = "type")]
|
||||
pub type_: String,
|
||||
pub features: Vec<Feature>,
|
||||
}
|
||||
#[derive(Deserialize)]
|
||||
pub struct Question {
|
||||
pub category: String,
|
||||
pub round: String,
|
||||
}
|
||||
}
|
||||
|
||||
trait De {
|
||||
fn de<T: DeserializeOwned>(s: &[u8]) -> Option<T>;
|
||||
}
|
||||
struct SerdeJson;
|
||||
struct SerdeJsonStr; // checks UTF-8 with std::str::from_utf8 first, then reads from the &str
|
||||
struct SonicRs;
|
||||
struct SimdJson;
|
||||
impl De for SerdeJson {
|
||||
fn de<T: DeserializeOwned>(s: &[u8]) -> Option<T> {
|
||||
serde_json::from_slice(s).ok()
|
||||
}
|
||||
}
|
||||
impl De for SerdeJsonStr {
|
||||
fn de<T: DeserializeOwned>(s: &[u8]) -> Option<T> {
|
||||
serde_json::from_str(std::str::from_utf8(s).ok()?).ok()
|
||||
}
|
||||
}
|
||||
impl De for SonicRs {
|
||||
fn de<T: DeserializeOwned>(s: &[u8]) -> Option<T> {
|
||||
sonic_rs::from_slice(s).ok()
|
||||
}
|
||||
}
|
||||
impl De for SimdJson {
|
||||
fn de<T: DeserializeOwned>(s: &[u8]) -> Option<T> {
|
||||
let mut b = s.to_vec();
|
||||
simd_json::serde::from_slice(&mut b).ok()
|
||||
}
|
||||
}
|
||||
|
||||
fn pick_structs<D: De>(name: &str, s: &[u8]) -> Option<f64> {
|
||||
fn status(acc: &mut f64, x: &sel::Status) {
|
||||
*acc += x.id as f64;
|
||||
*acc += x.user.screen_name.len() as f64;
|
||||
*acc += x.retweet_count as f64;
|
||||
}
|
||||
let mut acc = 0.0;
|
||||
match name {
|
||||
"twitter" => {
|
||||
let t: sel::Twitter = D::de(s)?;
|
||||
for x in &t.statuses {
|
||||
status(&mut acc, x);
|
||||
}
|
||||
}
|
||||
"status" => {
|
||||
let x: sel::Status = D::de(s)?;
|
||||
status(&mut acc, &x);
|
||||
}
|
||||
"citm_catalog" => {
|
||||
let c: sel::Citm = D::de(s)?;
|
||||
for p in &c.performances {
|
||||
acc += (p.id + p.event_id + p.seat_categories.len() as u64) as f64;
|
||||
}
|
||||
acc += c.events.len() as f64;
|
||||
}
|
||||
"canada" => {
|
||||
let c: sel::Canada = D::de(s)?;
|
||||
let g = &c.features[0].geometry;
|
||||
acc += (c.type_.len() + g.type_.len() + g.coordinates.len()) as f64;
|
||||
}
|
||||
"jeopardy" => {
|
||||
let q: Vec<sel::Question> = D::de(s)?;
|
||||
for x in &q {
|
||||
acc += if x.round == "Final Jeopardy!" { 1.0 } else { 0.0 };
|
||||
acc += x.category.len() as f64;
|
||||
}
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
Some(acc)
|
||||
}
|
||||
|
||||
// ---- edit: the same edits as bench_all.cpp, then write compact JSON
|
||||
|
||||
fn edit_serde(name: &str, r: &mut serde_json::Value) {
|
||||
fn status(s: &mut serde_json::Value) {
|
||||
let o = s.as_object_mut().unwrap();
|
||||
let rc = o["retweet_count"].as_i64().unwrap();
|
||||
o.insert("retweet_count".to_string(), (rc + 1).into());
|
||||
o.remove("entities");
|
||||
o.insert("seen".to_string(), true.into());
|
||||
}
|
||||
match name {
|
||||
"twitter" => {
|
||||
for s in r["statuses"].as_array_mut().unwrap() {
|
||||
status(s);
|
||||
}
|
||||
}
|
||||
"status" => status(r),
|
||||
"citm_catalog" => {
|
||||
for p in r["performances"].as_array_mut().unwrap() {
|
||||
let o = p.as_object_mut().unwrap();
|
||||
let start = o["start"].as_i64().unwrap();
|
||||
o.insert("start".to_string(), (start + 1).into());
|
||||
o.remove("seatMapImage");
|
||||
o.insert("seen".to_string(), true.into());
|
||||
}
|
||||
}
|
||||
"canada" => {
|
||||
r["features"][0]["properties"]["name"] = "Kanada".into();
|
||||
for f in r["features"].as_array_mut().unwrap() {
|
||||
f["geometry"]["coordinates"][0].as_array_mut().unwrap().push(serde_json::json!([0.5, 0.25]));
|
||||
}
|
||||
}
|
||||
"jeopardy" => {
|
||||
for q in r.as_array_mut().unwrap() {
|
||||
let o = q.as_object_mut().unwrap();
|
||||
o.insert("value".to_string(), serde_json::Value::Null);
|
||||
o.remove("air_date");
|
||||
o.insert("seen".to_string(), true.into());
|
||||
}
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
|
||||
fn edit_sonic(name: &str, r: &mut sonic_rs::Value) {
|
||||
fn status(s: &mut sonic_rs::Value) {
|
||||
let o = s.as_object_mut().unwrap();
|
||||
let rc = o.get(&"retweet_count").unwrap().as_i64().unwrap();
|
||||
o.insert("retweet_count", rc + 1);
|
||||
o.remove(&"entities");
|
||||
o.insert("seen", true);
|
||||
}
|
||||
match name {
|
||||
"twitter" => {
|
||||
for s in r.get_mut("statuses").unwrap().as_array_mut().unwrap().iter_mut() {
|
||||
status(s);
|
||||
}
|
||||
}
|
||||
"status" => status(r),
|
||||
"citm_catalog" => {
|
||||
for p in r.get_mut("performances").unwrap().as_array_mut().unwrap().iter_mut() {
|
||||
let o = p.as_object_mut().unwrap();
|
||||
let start = o.get(&"start").unwrap().as_i64().unwrap();
|
||||
o.insert("start", start + 1);
|
||||
o.remove(&"seatMapImage");
|
||||
o.insert("seen", true);
|
||||
}
|
||||
}
|
||||
"canada" => {
|
||||
let fs = r.get_mut("features").unwrap().as_array_mut().unwrap();
|
||||
fs[0].get_mut("properties").unwrap().as_object_mut().unwrap().insert("name", "Kanada");
|
||||
for f in fs.iter_mut() {
|
||||
let ring = f.get_mut("geometry").unwrap().get_mut("coordinates").unwrap().get_mut(0).unwrap();
|
||||
ring.as_array_mut().unwrap().push(sonic_rs::json!([0.5, 0.25]));
|
||||
}
|
||||
}
|
||||
"jeopardy" => {
|
||||
for q in r.as_array_mut().unwrap().iter_mut() {
|
||||
let o = q.as_object_mut().unwrap();
|
||||
o.insert("value", sonic_rs::Value::new_null());
|
||||
o.remove(&"air_date");
|
||||
o.insert("seen", true);
|
||||
}
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
|
||||
macro_rules! edit_simd {
|
||||
($fname:ident, $t:ty) => {
|
||||
fn $fname(name: &str, r: &mut $t) {
|
||||
fn status(s: &mut $t) {
|
||||
let o = s.as_object_mut().unwrap();
|
||||
let rc = o.get("retweet_count").unwrap().as_i64().unwrap();
|
||||
o.insert("retweet_count".into(), (rc + 1).into());
|
||||
o.remove("entities");
|
||||
o.insert("seen".into(), true.into());
|
||||
}
|
||||
match name {
|
||||
"twitter" => {
|
||||
for s in r.as_object_mut().unwrap().get_mut("statuses").unwrap().as_array_mut().unwrap().iter_mut() {
|
||||
status(s);
|
||||
}
|
||||
}
|
||||
"status" => status(r),
|
||||
"citm_catalog" => {
|
||||
for p in r.as_object_mut().unwrap().get_mut("performances").unwrap().as_array_mut().unwrap().iter_mut() {
|
||||
let o = p.as_object_mut().unwrap();
|
||||
let start = o.get("start").unwrap().as_i64().unwrap();
|
||||
o.insert("start".into(), (start + 1).into());
|
||||
o.remove("seatMapImage");
|
||||
o.insert("seen".into(), true.into());
|
||||
}
|
||||
}
|
||||
"canada" => {
|
||||
let fs = r.as_object_mut().unwrap().get_mut("features").unwrap().as_array_mut().unwrap();
|
||||
fs[0].as_object_mut().unwrap().get_mut("properties").unwrap().as_object_mut().unwrap().insert("name".into(), "Kanada".into());
|
||||
for f in fs.iter_mut() {
|
||||
let g = f.as_object_mut().unwrap().get_mut("geometry").unwrap().as_object_mut().unwrap();
|
||||
let ring = &mut g.get_mut("coordinates").unwrap().as_array_mut().unwrap()[0];
|
||||
ring.as_array_mut().unwrap().push(vec![<$t>::from(0.5), <$t>::from(0.25)].into());
|
||||
}
|
||||
}
|
||||
"jeopardy" => {
|
||||
for q in r.as_array_mut().unwrap().iter_mut() {
|
||||
let o = q.as_object_mut().unwrap();
|
||||
o.insert("value".into(), <$t>::Static(simd_json::StaticNode::Null));
|
||||
o.remove("air_date");
|
||||
o.insert("seen".into(), true.into());
|
||||
}
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
};
|
||||
}
|
||||
edit_simd!(edit_simd_owned, simd_json::OwnedValue);
|
||||
edit_simd!(edit_simd_borrowed, simd_json::BorrowedValue<'_>);
|
||||
|
||||
// ---- driver
|
||||
|
||||
struct Res {
|
||||
st: Stats,
|
||||
acc: f64,
|
||||
want: bool,
|
||||
out: Vec<u8>,
|
||||
}
|
||||
|
||||
impl Res {
|
||||
fn new(want: bool) -> Res {
|
||||
Res { st: Stats::default(), acc: 0.0, want, out: Vec::new() }
|
||||
}
|
||||
}
|
||||
|
||||
fn emit(r: &mut Res, o: Vec<u8>) -> bool {
|
||||
r.acc = o.len() as f64;
|
||||
if r.want {
|
||||
r.out = o;
|
||||
}
|
||||
true
|
||||
}
|
||||
|
||||
struct Variant<'a> {
|
||||
name: &'static str,
|
||||
f: Box<dyn Fn(&[u8], &mut Res) -> bool + 'a>,
|
||||
any_order: bool, // sum: visits objects in another order than the document's
|
||||
}
|
||||
|
||||
fn v<'a>(name: &'static str, f: impl Fn(&[u8], &mut Res) -> bool + 'a) -> Variant<'a> {
|
||||
Variant { name, f: Box::new(f), any_order: false }
|
||||
}
|
||||
|
||||
fn v_any<'a>(name: &'static str, f: impl Fn(&[u8], &mut Res) -> bool + 'a) -> Variant<'a> {
|
||||
Variant { name, f: Box::new(f), any_order: true }
|
||||
}
|
||||
|
||||
fn run(var: &Variant, s: &[u8], r: &mut Res) -> bool {
|
||||
catch_unwind(AssertUnwindSafe(|| (var.f)(s, r))).unwrap_or(false)
|
||||
}
|
||||
|
||||
static mut SINK: f64 = 0.0;
|
||||
@@ -104,102 +561,295 @@ fn sink(x: f64) {
|
||||
unsafe { std::ptr::write_volatile(std::ptr::addr_of_mut!(SINK), x) }
|
||||
}
|
||||
|
||||
// the input contract of bench_all.cpp: at least 64 bytes of spare capacity
|
||||
fn padded(s: &[u8]) -> Vec<u8> {
|
||||
let mut r = Vec::with_capacity(s.len() + 64);
|
||||
r.extend_from_slice(s);
|
||||
r
|
||||
}
|
||||
|
||||
// malformed inputs derived from a valid document, as in bench_all.cpp (0xFF in a string value, not a key)
|
||||
// (what, text, is the 0xFF probe)
|
||||
fn malformed(s: &[u8]) -> Vec<(&'static str, Vec<u8>, bool)> {
|
||||
let (mut first_close, mut mid_quote, mut last_quote) = (None, None, None);
|
||||
let mut in_string = false;
|
||||
let mut stack: Vec<u8> = Vec::new();
|
||||
let mut prev = 0u8;
|
||||
let mut i = 0;
|
||||
while i < s.len() {
|
||||
let ch = s[i];
|
||||
if in_string {
|
||||
if ch == b'\\' {
|
||||
i += 1;
|
||||
} else if ch == b'"' {
|
||||
in_string = false;
|
||||
}
|
||||
} else if ch == b'"' {
|
||||
in_string = true;
|
||||
let value = prev == b':' || (stack.last() == Some(&b'[') && (prev == b'[' || prev == b','));
|
||||
if value {
|
||||
last_quote = Some(i);
|
||||
if mid_quote.is_none() && i >= s.len() / 2 {
|
||||
mid_quote = Some(i);
|
||||
}
|
||||
}
|
||||
prev = b'"';
|
||||
} else if ch == b'{' || ch == b'[' {
|
||||
stack.push(ch);
|
||||
prev = ch;
|
||||
} else if ch == b'}' || ch == b']' {
|
||||
if ch == b'}' && first_close.is_none() {
|
||||
first_close = Some(i);
|
||||
}
|
||||
stack.pop();
|
||||
prev = ch;
|
||||
} else if ch == b':' || ch == b',' {
|
||||
prev = ch;
|
||||
}
|
||||
i += 1;
|
||||
}
|
||||
let mut t = s.to_vec();
|
||||
while matches!(t.last(), Some(b' ' | b'\n' | b'\r' | b'\t')) {
|
||||
t.pop();
|
||||
}
|
||||
t.pop();
|
||||
let mut out = vec![("truncated input", padded(&t), false)];
|
||||
let mut t = s.to_vec();
|
||||
t.insert(first_close.unwrap(), b',');
|
||||
out.push(("a stray comma before '}'", padded(&t), false));
|
||||
let mut t = s.to_vec();
|
||||
t.insert(mid_quote.or(last_quote).unwrap() + 1, 0xFF);
|
||||
out.push(("invalid UTF-8 (0xFF in a string)", padded(&t), true));
|
||||
out
|
||||
}
|
||||
|
||||
fn same_text(out: &[u8], expected: &serde_json::Value) -> bool {
|
||||
serde_json::from_slice::<serde_json::Value>(out).map(|v| v == *expected).unwrap_or(false)
|
||||
}
|
||||
|
||||
fn main() {
|
||||
let args: Vec<String> = std::env::args().collect();
|
||||
if args.len() < 2 {
|
||||
eprintln!("usage: {} <json_test_data dir> [rounds] [document] [use case]", args[0]);
|
||||
std::process::exit(1);
|
||||
}
|
||||
let t = format!("{}/", args[1]);
|
||||
let rounds: usize = args.get(2).map(|r| r.parse().unwrap()).unwrap_or(30);
|
||||
let only_workload: String = args.get(3).cloned().unwrap_or_default();
|
||||
let mut docs: Vec<(String, String, usize)> = Vec::new();
|
||||
let rounds: usize = args.get(2).map(|r| r.parse().unwrap()).unwrap_or(30).max(1);
|
||||
let only: String = args.get(3).cloned().unwrap_or_default();
|
||||
let only_uc: String = args.get(4).cloned().unwrap_or_default();
|
||||
let mut docs: Vec<(String, Vec<u8>, usize)> = Vec::new();
|
||||
for f in ["nativejson-benchmark/twitter.json", "nativejson-benchmark/citm_catalog.json", "nativejson-benchmark/canada.json", "jeopardy/jeopardy.json"] {
|
||||
let name = f.split('/').nth(1).unwrap().trim_end_matches(".json").to_string();
|
||||
docs.push((name, std::fs::read_to_string(format!("{}{}", t, f)).unwrap(), 1));
|
||||
}
|
||||
let tw: serde_json::Value = serde_json::from_str(&docs[0].1).unwrap();
|
||||
docs.push(("status".to_string(), serde_json::to_string(&tw["statuses"][0]).unwrap(), 200));
|
||||
|
||||
for (name, text, _) in &docs {
|
||||
let mut r = Stats::default();
|
||||
walk_serde(&serde_json::from_str(text).unwrap(), &mut r);
|
||||
let mut a = Stats::default();
|
||||
let mut buf = text.as_bytes().to_vec();
|
||||
walk_simd(&simd_json::to_borrowed_value(&mut buf).unwrap(), &mut a);
|
||||
let mut b = Stats::default();
|
||||
walk_sonic(&sonic_rs::from_str(text).unwrap(), &mut b);
|
||||
// sonic-rs keeps document order: its sum is compared bit for bit by the caller with json_view's
|
||||
println!("check {:<13} simd-json:{} sonic-rs:{} serde_json:OK (nodes {} str {} num {:.17e} sonic-num {:.17e})",
|
||||
name, if same(&r, &a) { "OK" } else { "MISMATCH" }, if same(&r, &b) { "OK" } else { "MISMATCH" }, r.nodes, r.str_, r.num, b.num);
|
||||
}
|
||||
|
||||
for (name, text, _) in &docs {
|
||||
// every writer's output must read back as the same values
|
||||
let mut r = Stats::default();
|
||||
walk_serde(&serde_json::from_str(text).unwrap(), &mut r);
|
||||
let mut buf = text.as_bytes().to_vec();
|
||||
let outs = [
|
||||
("simd-json", simd_json::to_string(&simd_json::to_borrowed_value(&mut buf).unwrap()).unwrap()),
|
||||
("sonic-rs", sonic_rs::to_string(&sonic_rs::from_str::<sonic_rs::Value>(text).unwrap()).unwrap()),
|
||||
("serde_json", serde_json::to_string(&serde_json::from_str::<serde_json::Value>(text).unwrap()).unwrap()),
|
||||
];
|
||||
print!("check {:<13} dump", name);
|
||||
for (n, o) in &outs {
|
||||
let mut st = Stats::default();
|
||||
walk_serde(&serde_json::from_str(o).unwrap(), &mut st);
|
||||
print!(" {}:{}({} bytes)", n, if same(&r, &st) { "OK" } else { "MISMATCH" }, o.len());
|
||||
}
|
||||
println!();
|
||||
docs.push((name, padded(&std::fs::read(format!("{}{}", t, f)).unwrap()), 1));
|
||||
}
|
||||
let tw: serde_json::Value = serde_json::from_slice(&docs[0].1).unwrap();
|
||||
docs.push(("status".to_string(), padded(serde_json::to_string(&tw["statuses"][0]).unwrap().as_bytes()), 200));
|
||||
|
||||
std::panic::set_hook(Box::new(|_| {})); // a panic is a failed call, reported by the checks
|
||||
let simd_reused = RefCell::new((simd_json::Buffers::default(), Vec::<u8>::new()));
|
||||
let mut excluded_total = 0;
|
||||
for (name, s, batch) in &docs {
|
||||
// for dump: each engine's document, parsed once outside the timer
|
||||
let mut dump_buf = s.as_bytes().to_vec();
|
||||
let dump_simd = simd_json::to_borrowed_value(&mut dump_buf).unwrap();
|
||||
let dump_sonic: sonic_rs::Value = sonic_rs::from_str(s).unwrap();
|
||||
let dump_serde: serde_json::Value = serde_json::from_str(s).unwrap();
|
||||
type Eng<'a> = (&'static str, Box<dyn Fn() + 'a>);
|
||||
let workloads: Vec<(&str, Vec<Eng>)> = vec![
|
||||
("parse", vec![
|
||||
("simd-json", Box::new(|| { let mut b = s.as_bytes().to_vec(); let v = simd_json::to_borrowed_value(&mut b).unwrap(); sink(v.is_object() as u8 as f64); })),
|
||||
("sonic-rs", Box::new(|| { let v: sonic_rs::Value = sonic_rs::from_str(s).unwrap(); sink(sonic_rs::JsonValueTrait::is_object(&v) as u8 as f64); })),
|
||||
("serde_json", Box::new(|| { let v: serde_json::Value = serde_json::from_str(s).unwrap(); sink(v.is_object() as u8 as f64); })),
|
||||
]),
|
||||
("traverse", vec![
|
||||
("simd-json", Box::new(|| { let mut b = s.as_bytes().to_vec(); let v = simd_json::to_borrowed_value(&mut b).unwrap(); let mut st = Stats::default(); walk_simd(&v, &mut st); sink(st.num); })),
|
||||
("sonic-rs", Box::new(|| { let v: sonic_rs::Value = sonic_rs::from_str(s).unwrap(); let mut st = Stats::default(); walk_sonic(&v, &mut st); sink(st.num); })),
|
||||
("serde_json", Box::new(|| { let v: serde_json::Value = serde_json::from_str(s).unwrap(); let mut st = Stats::default(); walk_serde(&v, &mut st); sink(st.num); })),
|
||||
]),
|
||||
("parse+dump", vec![
|
||||
("simd-json", Box::new(|| { let mut b = s.as_bytes().to_vec(); let v = simd_json::to_borrowed_value(&mut b).unwrap(); let o = simd_json::to_string(&v).unwrap(); sink(o.len() as f64); })),
|
||||
("sonic-rs", Box::new(|| { let v: sonic_rs::Value = sonic_rs::from_str(s).unwrap(); let o = sonic_rs::to_string(&v).unwrap(); sink(o.len() as f64); })),
|
||||
("serde_json", Box::new(|| { let v: serde_json::Value = serde_json::from_str(s).unwrap(); let o = serde_json::to_string(&v).unwrap(); sink(o.len() as f64); })),
|
||||
]),
|
||||
("dump", vec![
|
||||
("simd-json", Box::new(|| { let o = simd_json::to_string(&dump_simd).unwrap(); sink(o.len() as f64); })),
|
||||
("sonic-rs", Box::new(|| { let o = sonic_rs::to_string(&dump_sonic).unwrap(); sink(o.len() as f64); })),
|
||||
("serde_json", Box::new(|| { let o = serde_json::to_string(&dump_serde).unwrap(); sink(o.len() as f64); })),
|
||||
]),
|
||||
];
|
||||
for (wl, engines) in &workloads {
|
||||
if !only_workload.is_empty() && *wl != only_workload {
|
||||
if !only.is_empty() && *name != only {
|
||||
continue;
|
||||
}
|
||||
let name = name.as_str();
|
||||
let s = s.as_slice();
|
||||
let bad = malformed(s);
|
||||
for uc in ["validate", "sum", "select", "edit", "minify", "write"] {
|
||||
if !only_uc.is_empty() && uc != only_uc {
|
||||
continue;
|
||||
}
|
||||
let r = if s.len() > 10_000_000 { std::cmp::max(3, rounds / 5) } else { rounds };
|
||||
let mut best = vec![f64::MAX; engines.len()];
|
||||
// write: each library's document, parsed once outside the timer
|
||||
let mut wbuf: Vec<u8>;
|
||||
let (w_serde, w_sonic, w_simd) = if uc == "write" {
|
||||
wbuf = s.to_vec();
|
||||
(
|
||||
Some(serde_json::from_slice::<serde_json::Value>(s).unwrap()),
|
||||
Some(sonic_rs::from_slice::<sonic_rs::Value>(s).unwrap()),
|
||||
Some(simd_json::to_borrowed_value(&mut wbuf).unwrap()),
|
||||
)
|
||||
} else {
|
||||
(None, None, None)
|
||||
};
|
||||
let reuse = &simd_reused;
|
||||
let all: Vec<Variant> = match uc {
|
||||
"validate" => vec![
|
||||
v("serde_json: IgnoredAny", |s, _| serde_json::from_slice::<IgnoredAny>(s).is_ok()),
|
||||
v("serde_json: from_utf8 + IgnoredAny", |s, _| std::str::from_utf8(s).ok().map_or(false, |t| serde_json::from_str::<IgnoredAny>(t).is_ok())),
|
||||
v("serde_json: Value", |s, _| serde_json::from_slice::<serde_json::Value>(s).is_ok()),
|
||||
v("sonic-rs: IgnoredAny", |s, _| sonic_rs::from_slice::<IgnoredAny>(s).is_ok()),
|
||||
v("sonic-rs: Value", |s, _| sonic_rs::from_slice::<sonic_rs::Value>(s).is_ok()),
|
||||
v("simd-json: tape (copy)", |s, _| { let mut b = s.to_vec(); simd_json::to_tape(&mut b).is_ok() }),
|
||||
v("simd-json: borrowed (copy)", |s, _| { let mut b = s.to_vec(); let ok = simd_json::to_borrowed_value(&mut b).is_ok(); ok }),
|
||||
v("simd-json: tape (reused)", move |s, _| {
|
||||
let mut g = reuse.borrow_mut();
|
||||
let (ref mut bufs, ref mut b) = *g;
|
||||
b.clear();
|
||||
b.extend_from_slice(s);
|
||||
simd_json::to_tape_with_buffers(b, bufs).is_ok()
|
||||
}),
|
||||
],
|
||||
"sum" => vec![
|
||||
v("serde_json: visitor", |s, r| sum_serde(s, &mut r.st)),
|
||||
v_any("serde_json: Value", |s, r| serde_json::from_slice::<serde_json::Value>(s).map(|x| walk_serde(&x, &mut r.st)).is_ok()),
|
||||
v("sonic-rs: visitor", |s, r| sum_sonic(s, &mut r.st)),
|
||||
v("sonic-rs: Value", |s, r| sonic_rs::from_slice::<sonic_rs::Value>(s).map(|x| walk_sonic(&x, &mut r.st)).is_ok()),
|
||||
v("simd-json: visitor (copy)", |s, r| sum_simd(s, &mut r.st)),
|
||||
v_any("simd-json: borrowed (copy)", |s, r| { let mut b = s.to_vec(); simd_json::to_borrowed_value(&mut b).map(|x| walk_simd(&x, &mut r.st)).is_ok() }),
|
||||
v_any("simd-json: borrowed (reused)", move |s, r| {
|
||||
let mut g = reuse.borrow_mut();
|
||||
let (ref mut bufs, ref mut b) = *g;
|
||||
b.clear();
|
||||
b.extend_from_slice(s);
|
||||
simd_json::to_borrowed_value_with_buffers(b, bufs).map(|x| walk_simd(&x, &mut r.st)).is_ok()
|
||||
}),
|
||||
],
|
||||
"select" => vec![
|
||||
v("serde_json: structs", move |s, r| pick_structs::<SerdeJson>(name, s).map(|a| r.acc = a).is_some()),
|
||||
v("serde_json: from_utf8 + structs", move |s, r| pick_structs::<SerdeJsonStr>(name, s).map(|a| r.acc = a).is_some()),
|
||||
v("serde_json: Value", move |s, r| serde_json::from_slice::<serde_json::Value>(s).map(|x| r.acc = pick_serde(name, &x)).is_ok()),
|
||||
v("sonic-rs: structs", move |s, r| pick_structs::<SonicRs>(name, s).map(|a| r.acc = a).is_some()),
|
||||
v("sonic-rs: Value", move |s, r| sonic_rs::from_slice::<sonic_rs::Value>(s).map(|x| r.acc = pick_sonic(name, &x)).is_ok()),
|
||||
v("simd-json: structs (copy)", move |s, r| pick_structs::<SimdJson>(name, s).map(|a| r.acc = a).is_some()),
|
||||
v("simd-json: borrowed (copy)", move |s, r| { let mut b = s.to_vec(); let ok = simd_json::to_borrowed_value(&mut b).map(|x| r.acc = pick_simd(name, &x)).is_ok(); ok }),
|
||||
],
|
||||
"edit" => vec![
|
||||
v("serde_json: Value", move |s, r| match serde_json::from_slice::<serde_json::Value>(s) {
|
||||
Ok(mut x) => { edit_serde(name, &mut x); emit(r, serde_json::to_vec(&x).unwrap()) }
|
||||
Err(_) => false,
|
||||
}),
|
||||
v("sonic-rs: Value", move |s, r| match sonic_rs::from_slice::<sonic_rs::Value>(s) {
|
||||
Ok(mut x) => { edit_sonic(name, &mut x); emit(r, sonic_rs::to_vec(&x).unwrap()) }
|
||||
Err(_) => false,
|
||||
}),
|
||||
v("simd-json: owned (copy)", move |s, r| { let mut b = s.to_vec(); match simd_json::to_owned_value(&mut b) {
|
||||
Ok(mut x) => { edit_simd_owned(name, &mut x); emit(r, simd_json::to_vec(&x).unwrap()) }
|
||||
Err(_) => false,
|
||||
}}),
|
||||
v("simd-json: borrowed (copy)", move |s, r| { let mut b = s.to_vec(); let ok = match simd_json::to_borrowed_value(&mut b) {
|
||||
Ok(mut x) => { edit_simd_borrowed(name, &mut x); emit(r, simd_json::to_vec(&x).unwrap()) }
|
||||
Err(_) => false,
|
||||
}; ok }),
|
||||
],
|
||||
"minify" => vec![
|
||||
v("serde_json: Value", |s, r| serde_json::from_slice::<serde_json::Value>(s).map(|x| emit(r, serde_json::to_vec(&x).unwrap())).unwrap_or(false)),
|
||||
v("sonic-rs: Value", |s, r| sonic_rs::from_slice::<sonic_rs::Value>(s).map(|x| emit(r, sonic_rs::to_vec(&x).unwrap())).unwrap_or(false)),
|
||||
v("simd-json: borrowed (copy)", |s, r| { let mut b = s.to_vec(); simd_json::to_borrowed_value(&mut b).map(|x| emit(r, simd_json::to_vec(&x).unwrap())).unwrap_or(false) }),
|
||||
],
|
||||
_ => {
|
||||
let (a, b, c) = (w_serde.as_ref().unwrap(), w_sonic.as_ref().unwrap(), w_simd.as_ref().unwrap());
|
||||
vec![
|
||||
v("serde_json: Value", move |_, r| emit(r, serde_json::to_vec(a).unwrap())),
|
||||
v("sonic-rs: Value", move |_, r| emit(r, sonic_rs::to_vec(b).unwrap())),
|
||||
v("simd-json: borrowed", move |_, r| emit(r, simd_json::to_vec(c).unwrap())),
|
||||
]
|
||||
}
|
||||
};
|
||||
|
||||
// the reference result: serde_json's visitor (sum, document order), Value (select, edit, minify, write)
|
||||
let mut ref_st = Stats::default();
|
||||
let mut ref_acc = 0.0;
|
||||
let mut ref_json = serde_json::Value::Null;
|
||||
match uc {
|
||||
"sum" => {
|
||||
sum_serde(s, &mut ref_st);
|
||||
println!("check {:<13} {:<9} reference: nodes {} str {} num {:.16e}", name, uc, ref_st.nodes, ref_st.str_, ref_st.num);
|
||||
}
|
||||
"select" => {
|
||||
ref_acc = pick_serde(name, &serde_json::from_slice(s).unwrap());
|
||||
println!("check {:<13} {:<9} reference: {:.16e}", name, uc, ref_acc);
|
||||
}
|
||||
"validate" => {}
|
||||
_ => {
|
||||
ref_json = serde_json::from_slice(s).unwrap();
|
||||
if uc == "edit" {
|
||||
edit_serde(name, &mut ref_json);
|
||||
}
|
||||
}
|
||||
}
|
||||
let n_all = all.len();
|
||||
let mut vs: Vec<Variant> = Vec::new();
|
||||
let mut levels: Vec<&str> = Vec::new();
|
||||
for var in all {
|
||||
let mut r = Res::new(true);
|
||||
let mut why = String::new();
|
||||
let mut panicked = false;
|
||||
if !catch_unwind(AssertUnwindSafe(|| (var.f)(s, &mut r))).unwrap_or_else(|_| {
|
||||
panicked = true;
|
||||
false
|
||||
}) {
|
||||
why = if panicked { "crashes on the valid input" } else { "rejects the valid input" }.to_string();
|
||||
} else if uc == "sum"
|
||||
&& !(r.st.nodes == ref_st.nodes
|
||||
&& r.st.str_ == ref_st.str_
|
||||
&& if var.any_order { (r.st.num - ref_st.num).abs() <= 1e-9 * ref_st.num.abs() } else { r.st.num == ref_st.num })
|
||||
{
|
||||
why = format!("wrong result (nodes {} str {} num {:.16e})", r.st.nodes, r.st.str_, r.st.num);
|
||||
} else if uc == "select" && r.acc != ref_acc {
|
||||
why = format!("wrong result ({:.16e})", r.acc);
|
||||
} else if (uc == "edit" || uc == "minify" || uc == "write") && !same_text(&r.out, &ref_json) {
|
||||
why = "the output does not read back as the expected document".to_string();
|
||||
}
|
||||
if why.is_empty() {
|
||||
// the validation level from the malformed-input probes, as in bench_all.cpp
|
||||
let mut level = "n/a";
|
||||
if uc != "write" {
|
||||
let (mut structural, mut utf8) = (false, false);
|
||||
let mut notes: Vec<String> = Vec::new();
|
||||
for (what, b, is_utf8) in &bad {
|
||||
let mut rb = Res::new(true);
|
||||
match catch_unwind(AssertUnwindSafe(|| (var.f)(b, &mut rb))) {
|
||||
Ok(false) => continue,
|
||||
Ok(true) => {
|
||||
notes.push(format!("accepts {}", what));
|
||||
if *is_utf8 { utf8 = true } else { structural = true }
|
||||
}
|
||||
Err(_) => {
|
||||
notes.push(format!("crashes on {}", what));
|
||||
structural = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
level = if structural { "partial" } else if utf8 { "no UTF-8" } else { "full" };
|
||||
if !notes.is_empty() {
|
||||
println!("level: {} {} | {} | {} | {}", name, uc, var.name, level, notes.join("; "));
|
||||
}
|
||||
}
|
||||
vs.push(var);
|
||||
levels.push(level);
|
||||
} else {
|
||||
excluded_total += 1;
|
||||
println!("excluded: {} {} | {} | {}", name, uc, var.name, why);
|
||||
}
|
||||
}
|
||||
println!("check {:<13} {:<9} {} of {} variants give the right result", name, uc, vs.len(), n_all);
|
||||
if vs.is_empty() {
|
||||
continue;
|
||||
}
|
||||
|
||||
let r = if s.len() > 10_000_000 { rounds.min((rounds / 5).max(3)) } else { rounds };
|
||||
let mut best = vec![f64::MAX; vs.len()];
|
||||
for _ in 0..r {
|
||||
for (k, (_, f)) in engines.iter().enumerate() {
|
||||
f();
|
||||
for (k, var) in vs.iter().enumerate() {
|
||||
let mut res = Res::new(false);
|
||||
run(var, s, &mut res); // untimed: the previous variant's leftovers are cleaned up here
|
||||
sink(res.st.num + res.acc);
|
||||
let t0 = Instant::now();
|
||||
for _ in 0..*batch {
|
||||
f();
|
||||
let mut rt = Res::new(false);
|
||||
let ok = run(var, s, &mut rt);
|
||||
sink(if ok { rt.st.num + rt.acc } else { -1.0 });
|
||||
}
|
||||
let ns = t0.elapsed().as_nanos() as f64 / *batch as f64;
|
||||
best[k] = best[k].min(ns);
|
||||
}
|
||||
}
|
||||
print!("{:<13} {:<11}", name, wl);
|
||||
for (k, (n, _)) in engines.iter().enumerate() {
|
||||
print!(" {} {:.1}us", n, best[k] / 1e3);
|
||||
print!("{:<13} {:<9}", name, uc);
|
||||
for (k, var) in vs.iter().enumerate() {
|
||||
print!(" | {} [{}] {:.1}us ({:.2}x)", var.name, levels[k], best[k] / 1e3, best[k] / best[0]);
|
||||
}
|
||||
println!();
|
||||
}
|
||||
}
|
||||
println!("done: {} variant(s) excluded", excluded_total);
|
||||
}
|
||||
@@ -7,65 +7,199 @@
|
||||
# SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
|
||||
# SPDX-License-Identifier: MIT
|
||||
|
||||
"""Summarize runs of bench_all and bench_rs: for every document and workload, the engines ordered by time, and the
|
||||
fastest other engine relative to json_document (above 1: json_document is ahead).
|
||||
"""Summarize runs of bench_all and bench_rs by use case.
|
||||
|
||||
Usage: summarize.py LOG... [--rust LOG...]
|
||||
Usage: summarize.py LOG... [--rust LOG...] [--json OUT.json]
|
||||
|
||||
Several logs of the same program are combined by taking each cell's best time. Fresh documents are compared with
|
||||
fresh ones: "json_view (reused)" and "simdjson DOM" (a reused parser) are left out of the "fastest other" column
|
||||
and compared with each other instead.
|
||||
Every variant carries a validation level from the malformed-input probes: "full" (rejects truncated input, a stray
|
||||
comma and invalid UTF-8), "no UTF-8" (accepts only the invalid UTF-8), "partial" (accepts or crashes on a structural
|
||||
error, or crashes on any probe), or "n/a" (write: no input to validate; counts as full). A variant's level is the
|
||||
worst it got for any file. Prints, for every use case and file,
|
||||
(a) per library the best "full" variant and, if faster, the best variant at any level, with their ratios to the
|
||||
best "full" nlohmann json_document variant (variants named "(reused)" keep a parser or document across calls and
|
||||
are never chosen as a library's best),
|
||||
(b) all variants with their levels and times,
|
||||
(c) the variants below "full" with what the probes found, and the variants excluded for wrong results.
|
||||
--json writes {use case: {file: {library: {"best": {"variant", "us", "level"} or null, "best_any": {...},
|
||||
"variants": {variant: {"us", "level"}}, "excluded": {variant: reason}}}}} for the chart scripts; variant names there
|
||||
omit the "<library>: " prefix.
|
||||
|
||||
Several logs are combined by taking each cell's best time. A variant excluded for one file (or in one log) is excluded
|
||||
for every file of that use case. The reference values that bench_all and bench_rs print for sum and select are
|
||||
compared, so both programs are known to have checked against the same numbers.
|
||||
"""
|
||||
import json
|
||||
import re
|
||||
import sys
|
||||
|
||||
REUSED = {"json_view (reused)", "simdjson DOM"}
|
||||
USE_CASES = ["validate", "sum", "select", "edit", "minify", "write"]
|
||||
DOCS = ["twitter", "citm_catalog", "canada", "jeopardy", "status"]
|
||||
WORKLOADS = ["parse", "traverse", "select", "parse+dump", "dump"]
|
||||
OURS = "nlohmann json_document"
|
||||
RANK = {"full": 0, "n/a": 0, "no UTF-8": 1, "partial": 2}
|
||||
CELL = re.compile(r"^(.+) \[(full|no UTF-8|partial|n/a)\] ([\d.]+)us(?: \(([\d.]+)x\))?$")
|
||||
REF = re.compile(r"^check (\S+)\s+(\S+)\s+reference: (.*)$")
|
||||
|
||||
|
||||
def parse(path, suffix=""):
|
||||
cells = {}
|
||||
for line in open(path):
|
||||
if line.startswith(("check", "info", "WARNING")):
|
||||
def library(name):
|
||||
return name.split(": ", 1)[0]
|
||||
|
||||
|
||||
def variant(name):
|
||||
return name.split(": ", 1)[1] if ": " in name else name
|
||||
|
||||
|
||||
def is_reused(name):
|
||||
return "(reused)" in name
|
||||
|
||||
|
||||
def worse(a, b):
|
||||
return a if a is not None and (b is None or RANK[a] >= RANK[b]) else b
|
||||
|
||||
|
||||
def parse(path, cells, levels, notes, excluded, refs):
|
||||
for line in open(path, encoding="utf-8"):
|
||||
line = line.rstrip("\n")
|
||||
if line.startswith(("excluded: ", "level: ")):
|
||||
kind, rest = line.split(": ", 1)
|
||||
fields = rest.split(" | ")
|
||||
doc, uc = fields[0].split()
|
||||
name = fields[1]
|
||||
target = excluded if kind == "excluded" else notes
|
||||
for why in fields[-1].split("; "):
|
||||
target.setdefault((uc, name), {}).setdefault(why, set()).add(doc)
|
||||
if kind == "level":
|
||||
levels[(uc, name)] = worse(levels.get((uc, name)), fields[2])
|
||||
continue
|
||||
m = re.match(r"(\S+)\s+(\S+)\s+(.*)", line)
|
||||
if not m:
|
||||
m = REF.match(line)
|
||||
if m:
|
||||
refs.setdefault((m[1], m[2]), {})[path] = m[3]
|
||||
continue
|
||||
for engine, us in re.findall(r"\s*(.+?) ([\d.]+)us(?: \([\d.]+x\))?", m[3]):
|
||||
cells[(m[1], m[2], engine.strip() + suffix)] = float(us)
|
||||
return cells
|
||||
if line.startswith(("check", "done", "WARNING")) or " | " not in line:
|
||||
continue
|
||||
parts = line.split(" | ")
|
||||
head = parts[0].split()
|
||||
if len(head) != 2 or head[1] not in USE_CASES:
|
||||
continue
|
||||
for c in parts[1:]:
|
||||
m = CELL.match(c.strip())
|
||||
if m:
|
||||
key = (head[0], head[1], m[1])
|
||||
cells[key] = min(cells.get(key, float("inf")), float(m[3]))
|
||||
levels[(head[1], m[1])] = worse(levels.get((head[1], m[1])), m[2])
|
||||
|
||||
|
||||
def best(paths, suffix=""):
|
||||
runs = [parse(p, suffix) for p in paths]
|
||||
keys = set(runs[0]).intersection(*runs[1:])
|
||||
return {k: min(r[k] for r in runs) for k in keys}
|
||||
def check_refs(refs):
|
||||
"""The sum and select references of all logs must agree (bench_rs prints Rust's float formatting)."""
|
||||
def numbers(text):
|
||||
return [float(x) for x in re.findall(r"[-+]?\d[\d.]*(?:e[-+]?\d+)?", text)]
|
||||
ok = True
|
||||
for (doc, uc), by_log in sorted(refs.items()):
|
||||
values = {tuple(numbers(t)) for t in by_log.values()}
|
||||
if len(values) > 1:
|
||||
ok = False
|
||||
print("WARNING: the %s %s references differ: %s" % (doc, uc, "; ".join("%s: %s" % kv for kv in by_log.items())))
|
||||
return ok
|
||||
|
||||
|
||||
def is_full(level):
|
||||
return RANK[level] == 0
|
||||
|
||||
|
||||
def main():
|
||||
args = sys.argv[1:]
|
||||
out_json = None
|
||||
if "--json" in args:
|
||||
i = args.index("--json")
|
||||
out_json = args[i + 1]
|
||||
del args[i:i + 2]
|
||||
cpp = args[:args.index("--rust")] if "--rust" in args else args
|
||||
rust = args[args.index("--rust") + 1:] if "--rust" in args else []
|
||||
cells = best(cpp)
|
||||
if rust:
|
||||
cells.update(best(rust, " (Rust)"))
|
||||
for wl in WORKLOADS:
|
||||
print("\n== " + wl)
|
||||
for doc in DOCS:
|
||||
cell = {e: v for (d, w, e), v in cells.items() if d == doc and w == wl}
|
||||
if "json_view" not in cell:
|
||||
continue
|
||||
jv = cell["json_view"]
|
||||
order = sorted(cell.items(), key=lambda x: x[1])
|
||||
# in dump, every engine writes a document parsed beforehand: nothing is reused
|
||||
fresh = {e: v for e, v in cell.items() if (wl == "dump" or e not in REUSED) and e != "json_view"}
|
||||
other, t = min(fresh.items(), key=lambda x: x[1])
|
||||
line = "%-13s json_view %10.1f us fastest other: %s %.2f" % (doc, jv, other, t / jv)
|
||||
if wl != "dump" and "json_view (reused)" in cell and "simdjson DOM" in cell:
|
||||
line += " | reused: simdjson DOM %.2f" % (cell["simdjson DOM"] / cell["json_view (reused)"])
|
||||
print(line)
|
||||
print(" " + ", ".join("%s %.2f" % (e, v / jv) for e, v in order))
|
||||
if not cpp and not rust:
|
||||
sys.exit(__doc__)
|
||||
cells, levels, notes, excluded, refs = {}, {}, {}, {}, {}
|
||||
for p in cpp + rust:
|
||||
parse(p, cells, levels, notes, excluded, refs)
|
||||
check_refs(refs)
|
||||
result = {}
|
||||
|
||||
def docs_of(uc):
|
||||
return [d for d in DOCS if any(k[0] == d and k[1] == uc for k in cells)]
|
||||
|
||||
def cell_of(doc, uc):
|
||||
return {n: t for (d, u, n), t in cells.items() if d == doc and u == uc and (uc, n) not in excluded}
|
||||
|
||||
def show(n, t, ours, uc):
|
||||
ratio = "%5.2f" % (t / ours) if ours else " -"
|
||||
return "%-40s %-10s %11.1f us %s" % (variant(n), "[%s]" % levels[(uc, n)], t, ratio)
|
||||
|
||||
print("# (a) Per library: the best full variant, and the best at any level if faster (ratio to the best full %s"
|
||||
" variant; below 1: faster)" % OURS)
|
||||
for uc in USE_CASES:
|
||||
if not docs_of(uc):
|
||||
continue
|
||||
print("\n== " + uc)
|
||||
for doc in docs_of(uc):
|
||||
cell = cell_of(doc, uc)
|
||||
libs = {}
|
||||
for n, t in cell.items():
|
||||
libs.setdefault(library(n), {})[n] = t
|
||||
entry = result.setdefault(uc, {}).setdefault(doc, {})
|
||||
rows = []
|
||||
for lib, vs in libs.items():
|
||||
fresh = {n: t for n, t in vs.items() if not is_reused(n)}
|
||||
full = {n: t for n, t in fresh.items() if is_full(levels[(uc, n)])}
|
||||
b = min(full.items(), key=lambda x: x[1]) if full else None
|
||||
a = min(fresh.items(), key=lambda x: x[1]) if fresh else None
|
||||
entry[lib] = {
|
||||
"best": {"variant": variant(b[0]), "us": b[1], "level": levels[(uc, b[0])]} if b else None,
|
||||
"best_any": {"variant": variant(a[0]), "us": a[1], "level": levels[(uc, a[0])]} if a else None,
|
||||
"variants": {variant(n): {"us": t, "level": levels[(uc, n)]} for n, t in sorted(vs.items(), key=lambda x: x[1])},
|
||||
}
|
||||
rows.append((lib, b, a))
|
||||
for (u, n), whys in excluded.items():
|
||||
if u == uc:
|
||||
lib = entry.setdefault(library(n), {"best": None, "best_any": None, "variants": {}})
|
||||
lib.setdefault("excluded", {})[variant(n)] = "; ".join(sorted(whys))
|
||||
ours = next((b[1] for lib, b, a in rows if lib == OURS and b), None)
|
||||
print("%s%s" % (doc, "" if ours else " (no full %s result)" % OURS))
|
||||
rows.sort(key=lambda r: (r[1] is None, r[1][1] if r[1] else r[2][1] if r[2] else 0))
|
||||
for lib, b, a in rows:
|
||||
line = " %-24s %s" % (lib, show(*b, ours, uc) if b else "%-40s %-10s %14s %5s" % ("(none)", "", "", ""))
|
||||
if a and (not b or a[0] != b[0]):
|
||||
line += " | faster: %s" % show(*a, ours, uc)
|
||||
print(line)
|
||||
|
||||
print("\n# (b) All variants")
|
||||
for uc in USE_CASES:
|
||||
if not docs_of(uc):
|
||||
continue
|
||||
print("\n== " + uc)
|
||||
for doc in docs_of(uc):
|
||||
cell = cell_of(doc, uc)
|
||||
ours = min((t for n, t in cell.items() if library(n) == OURS and not is_reused(n) and is_full(levels[(uc, n)])), default=None)
|
||||
print(doc)
|
||||
for n, t in sorted(cell.items(), key=lambda x: x[1]):
|
||||
ratio = "%5.2f" % (t / ours) if ours else " -"
|
||||
print(" %-66s %-10s %11.1f us %s" % (n, "[%s]" % levels[(uc, n)], t, ratio))
|
||||
|
||||
print("\n# (c) Variants below full, and excluded variants")
|
||||
for uc in USE_CASES:
|
||||
rows = sorted((n, whys) for (u, n), whys in notes.items() if u == uc)
|
||||
ex = sorted((n, whys) for (u, n), whys in excluded.items() if u == uc)
|
||||
if not rows and not ex:
|
||||
continue
|
||||
print("\n== " + uc)
|
||||
for n, whys in rows:
|
||||
for why, docs in sorted(whys.items()):
|
||||
print(" %-58s %-10s %s (%s)" % (n, "[%s]" % levels.get((uc, n), "?"), why, ", ".join(d for d in DOCS if d in docs)))
|
||||
for n, whys in ex:
|
||||
for why, docs in sorted(whys.items()):
|
||||
print(" %-58s %-10s %s (%s)" % (n, "excluded", why, ", ".join(d for d in DOCS if d in docs)))
|
||||
|
||||
if out_json:
|
||||
with open(out_json, "w", encoding="utf-8") as f:
|
||||
json.dump(result, f, indent=2)
|
||||
f.write("\n")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
Reference in new issue
Block a user