diff --git a/docs/mkdocs/docs/home/blog/how_3_13_0_got_faster.md b/docs/mkdocs/docs/home/blog/how_3_13_0_got_faster.md index 28cc887bb..47bde3545 100644 --- a/docs/mkdocs/docs/home/blog/how_3_13_0_got_faster.md +++ b/docs/mkdocs/docs/home/blog/how_3_13_0_got_faster.md @@ -615,7 +615,8 @@ every number exactly; RapidJSON guarantees that only with `kParseFullPrecisionFl (with its default flags it happened to produce the same values for these files, and those rows are in the data). The workloads: *parse* (build the document and free it), *traverse* (parse, then visit every value and convert every -number), *select* (parse, then read a few fields of every record) and *parse+dump*. We ran them on the M1 Max with +number), *select* (parse, then read a few fields of every record), *parse+dump*, and *dump* (write a document that +was parsed beforehand; only the writing is timed). We ran them on the M1 Max with Apple clang 21 (and Rust 1.99), and on the x86-64 Haswell server with GCC 13 and Clang 18 (`-march=haswell`, one pinned core). That server has other work running, so differences of a few percent there are noise. Glaze 9 could not be built with Clang 18 on it (Clang 18 can't use GCC 13's `std::expected`), and the Rust engines ran on the Mac @@ -646,6 +647,41 @@ interleaved rounds):

The parse times of the table above as throughput (file size divided by time), more is better. RapidJSON, Glaze, serde_json and simd-json are left out to keep the chart readable; they are in the table.

+Writing, with each document parsed beforehand, so that only the writing of compact JSON into a new string is timed. +Before anything is timed, every output must read back as the same values, with bit-identical numbers, so every +engine writes every float exactly: + +| engine | `twitter.json` | `citm_catalog.json` | `canada.json` | `jeopardy.json` | +| --- | --: | --: | --: | --: | +| nlohmann/json `json_document` | **74.5** | **133** | **1,625** | **14,757** | +| yyjson | 104 | 166 | 1,716 | 20,134 | +| sonic-cpp | 105 | 222 | 3,044 | 19,836 | +| Glaze `generic_u64` | 208 | 469 | 1,700 | 34,367 | +| sonic-rs (Rust) | 213 | 444 | 2,417 | 33,818 | +| nlohmann/json `json::dump` | 217 | 286 | 2,244 | 31,205 | +| serde_json (Rust) | 329 | 487 | 2,236 | 52,629 | +| simdjson DOM | 354 | 498 | 6,816 | 54,833 | +| simd-json (Rust) | 358 | 473 | 2,707 | 47,998 | +| Boost.JSON | 498 | 501 | 2,942 | 63,392 | +| RapidJSON | 592 | 815 | 6,999 | 83,397 | + +

Write times in µs, less is better: compact output of a document parsed beforehand, on an Apple M1 Max, best of 2 × 30 interleaved rounds (2 × 6 for `jeopardy.json`). The fastest engine of each file is in bold. Boost.JSON writes some floats of `canada.json` with more digits, so its output is 9 % larger.

+ +
+--8<-- "images/perf_blog_view_dump.svg" +
+ +

The write times of the table above as throughput (size of the compact output divided by time), more is better, for the engines of the parse chart.

+ +On the Mac, `json_document` writes fastest on every file, 1.05 to 1.40 times ahead of the next engine. Part of that is its +design, not the writer: a string that had no escape sequence in the source has nothing to escape, so it is copied +from the source text without being scanned, and integers are copied as their source text. Floats are formatted +anew with Żmij (theme 3), and since `json_document` keeps floats as text, a float of more than 15 significant digits +is first converted from its digits to a `double`. That is why `canada.json` is the closest race on the Mac (Glaze +takes 1.05 times as long, yyjson 1.06) and is lost on x86-64, where yyjson takes 0.78 (GCC) and 0.93 (Clang) of +`json_document`'s time; with GCC, even `json::dump()`, which has the `double`s already, is faster there (0.92). + + The fastest *other* engine for each workload, as its time divided by nlohmann/json's `json_document`'s, over the four files above: above 1, `json_document` is ahead; below 1, it is behind. @@ -655,6 +691,7 @@ four files above: above 1, `json_document` is ahead; below 1, it is behind. | traverse | 1.04–1.18 (yyjson, simdjson On-Demand) | **0.58–0.86** (simdjson On-Demand) | **0.86**–1.20 (sonic-cpp, simdjson On-Demand, yyjson) | | select | **0.34–0.84** (Glaze structs, simdjson On-Demand) | **0.26–0.65** | **0.33–0.86** | | parse+dump | 1.18–1.35 (yyjson) | 1.20–1.41 (yyjson, sonic-cpp) | 1.16–1.27 (yyjson, sonic-cpp) | +| dump | 1.05–1.40 (yyjson, Glaze, sonic-cpp) | 1.23–1.34 (yyjson) on `twitter`, `citm_catalog`; **0.78** (yyjson) on `canada`; 0.99 (yyjson) on `jeopardy` | 1.28–1.33 (yyjson) on `twitter`, `citm_catalog`; **0.93** (yyjson) on `canada`; **0.95** (sonic-cpp) on `jeopardy` |

Time of the fastest other engine divided by the time of nlohmann/json's `json_document`, as a range over the four files, with the engines that were fastest; below 1, nlohmann/json is behind.

@@ -672,11 +709,17 @@ Apple silicon. On x86-64 it ties with simdjson's DOM on typical documents. Where - **Float-heavy documents, measured fairly.** `json_document` converts a float when you read it; the others convert while parsing. The `canada.json` parse row flatters it; *traverse* is the fair comparison, and there it is 1.04× ahead on the Mac and behind sonic-cpp with Clang on x86-64 (0.88). +- **Writing floats on x86-64.** Writing a document it parsed beforehand, `json_document` is behind yyjson on + `canada.json` (0.78 with GCC, 0.93 with Clang), because it first converts most floats from their text, and ties + on `jeopardy.json` (0.95 to 0.99). - **Small documents.** On a 2.5 KB message, all document parsers take about a microsecond, and yyjson and simdjson's DOM are as fast or a few percent faster. And `json::parse`? Boost.JSON builds a mutable tree as `json` does, so it is the fair comparison. It is 1.6 to 2.4 -times as fast on the Mac, and 2.4 to 4.5 times on the x86-64 server. +times as fast on the Mac, and 2.4 to 4.5 times on the x86-64 server. For writing, the comparison goes the other way: +`json::dump()` writes 1.3 to 2.3 times as fast as Boost.JSON's `serialize()` on the Mac. On the x86-64 server it is +1.1 to 1.6 times as fast on three files, but Boost.JSON writes `jeopardy.json` 4 % (Clang) to 12 % (GCC) faster. On the Mac it is also ahead of RapidJSON, simdjson's DOM and simd-json on every file, and of sonic-rs and +serde_json on all but one file each, where they are within 2 %. The programs are in [`tests/benchmarks/json_view/others`](https://github.com/nlohmann/json/tree/develop/tests/benchmarks/json_view/others), @@ -772,33 +815,39 @@ behaves differently. We did not test Windows for speed at all. ## Thanks -Most of what is in this post was invented by other people. The libraries and papers we took ideas or code from, in -the order they appear above, each with its license in parentheses. The library contains code adapted from four of -them, marked "adapted code", which it uses under the MIT License; the -[README](https://github.com/nlohmann/json#license) has their copyright notices. From the others we took the idea, -not their code. +Most of what is in this post was invented by other people. Each library is listed with its license in +parentheses. + +**Code we adapted.** The library contains code adapted from these four projects. We use it under the MIT License, +and the source files that contain it carry the authors' copyright lines and say what was taken; the +[README](https://github.com/nlohmann/json#license) lists them too. + +- [fast_float](https://github.com/fastfloat/fast_float) (Apache-2.0, MIT or BSL-1.0) by Daniel Lemire and + contributors: the Eisel-Lemire algorithm, its table of powers of five, its digit comparison for long numbers, and + its parsing of eight digits at a time (N. Mushtak and D. Lemire, "Fast number parsing without fallback", 2023). + W. D. Clinger for the fast path, M. Eisel for the algorithm. +- [Żmij](https://github.com/vitaut/zmij) (MIT) by Victor Zverovich, with the digit conversion of Xiang JunBo and the + SIMD sequence and compressed powers of ten of Dougall Johnson: ported to C++11. +- [simdjson](https://github.com/simdjson/simdjson) (Apache-2.0 or MIT) by Daniel Lemire, Geoff Langdale, John Keiser + and contributors: the "lookup4" UTF-8 check and its tables (J. Keiser and D. Lemire, "Validating UTF-8 In Less + Than One Instruction Per Byte", 2021), in `json.hpp`'s lexer and in `json_view.hpp`. +- [yyjson](https://github.com/ibireme/yyjson) (MIT) by YaoYuan: table-driven `\u` decoding and the unrolled checks + at fixed offsets in `json_view.hpp`'s parser. + +**Ideas we took.** From these we took the idea, not their code, in the order they appear above: - [serde_json](https://github.com/serde-rs/json) (MIT or Apache-2.0): deriving positions from the offset only on error. - Sean Eron Anderson's [Bit Twiddling Hacks](https://graphics.stanford.edu/~seander/bithacks.html) (snippets in the public domain): the zero-byte test behind every word-at-a-time scan. - [Glaze](https://github.com/stephenberry/glaze) (MIT): skipping indentation in 8-byte steps (`skip_matching_ws`). -- [simdjson](https://github.com/simdjson/simdjson) (Apache-2.0 or MIT) by Daniel Lemire, Geoff Langdale, John Keiser - and contributors: the "lookup4" UTF-8 check (J. Keiser and D. Lemire, "Validating UTF-8 In Less Than One - Instruction Per Byte", 2021; adapted code), and the tape that showed what a flat index can do. - [SQLite](https://sqlite.org) (public domain): writing short strings with a table loop, as its `jsonAppendString()` does. - [orjson](https://github.com/ijl/orjson) (Apache-2.0 or MIT; its escape table MPL-2.0) and [sonic-rs](https://github.com/cloudwego/sonic-rs) (Apache-2.0), from which orjson adapted it: escape sequences as fixed-size table rows (`QUOTE_TAB`). -- [fast_float](https://github.com/fastfloat/fast_float) (Apache-2.0, MIT or BSL-1.0) by Daniel Lemire and - contributors: the Eisel-Lemire algorithm, its table of powers of five, its digit comparison, and eight digits at a - time (N. Mushtak and D. Lemire, "Fast number parsing without fallback", 2023; adapted code). W. D. Clinger for the - fast path, M. Eisel for the algorithm. -- [Żmij](https://github.com/vitaut/zmij) (MIT) by Victor Zverovich, with the digit conversion of Xiang JunBo and the - SIMD sequence of Dougall Johnson (adapted code: ported to C++11). +- [simdjson](https://github.com/simdjson/simdjson)'s tape and [yyjson](https://github.com/ibireme/yyjson)'s 16-byte + immutable document: a flat index in document order instead of a tree. - [Boost.JSON](https://github.com/boostorg/json) (BSL-1.0): switching to a hash table for large objects. -- [yyjson](https://github.com/ibireme/yyjson) (MIT) by YaoYuan: table-driven `\u` decoding and fixed-offset unrolled - checks (adapted code in `json_view.hpp`), and a 16-byte flat document to measure against. - [RapidJSON](https://github.com/Tencent/rapidjson) (MIT): the measured reason to keep recursion as the default. And the people who sent or shaped pull requests: [@gregmarr](https://github.com/gregmarr) reviewed most of them and diff --git a/docs/mkdocs/docs/images/perf_blog_view.svg b/docs/mkdocs/docs/images/perf_blog_view.svg index 160ca0d94..e355db7cf 100644 --- a/docs/mkdocs/docs/images/perf_blog_view.svg +++ b/docs/mkdocs/docs/images/perf_blog_view.svg @@ -1,6 +1,6 @@ Parse throughput of nlohmann/json's json_document and json::parse and five other document parsers, fresh documents, Apple M1 Max -Grouped horizontal bars in GB/s, more is better. twitter.json nlohmann/json json_document 3.45 GB/s; twitter.json yyjson 3.26 GB/s; twitter.json simdjson D​OM 3.13 GB/s; twitter.json sonic-rs (Rust) 2.09 GB/s; twitter.json sonic-cpp 1.67 GB/s; twitter.json Boost.J​SON 0.73 GB/s; twitter.json nlohmann/json json::parse 0.47 GB/s; citm_catalog.json nlohmann/json json_document 4.75 GB/s; citm_catalog.json yyjson 3.61 GB/s; citm_catalog.json simdjson D​OM 3.55 GB/s; citm_catalog.json sonic-rs (Rust) 2.60 GB/s; citm_catalog.json sonic-cpp 1.75 GB/s; citm_catalog.json Boost.J​SON 1.16 GB/s; citm_catalog.json nlohmann/json json::parse 0.72 GB/s; canada.json nlohmann/json json_document 2.39 GB/s; canada.json yyjson 1.51 GB/s; canada.json simdjson D​OM 1.07 GB/s; canada.json sonic-rs (Rust) 1.18 GB/s; canada.json sonic-cpp 0.87 GB/s; canada.json Boost.J​SON 0.91 GB/s; canada.json nlohmann/json json::parse 0.38 GB/s; jeopardy.json nlohmann/json json_document 2.45 GB/s; jeopardy.json yyjson 1.83 GB/s; jeopardy.json simdjson D​OM 2.06 GB/s; jeopardy.json sonic-rs (Rust) 1.84 GB/s; jeopardy.json sonic-cpp 1.36 GB/s; jeopardy.json Boost.J​SON 0.62 GB/s; jeopardy.json nlohmann/json json::parse 0.34 GB/s. +Grouped horizontal bars of parse throughput in GB/s, more is better. twitter.json nlohmann/json json_document 3.45 GB/s; twitter.json yyjson 3.26 GB/s; twitter.json simdjson D​OM 3.13 GB/s; twitter.json sonic-rs (Rust) 2.09 GB/s; twitter.json sonic-cpp 1.67 GB/s; twitter.json Boost.J​SON 0.73 GB/s; twitter.json nlohmann/json json::parse 0.47 GB/s; citm_catalog.json nlohmann/json json_document 4.75 GB/s; citm_catalog.json yyjson 3.61 GB/s; citm_catalog.json simdjson D​OM 3.55 GB/s; citm_catalog.json sonic-rs (Rust) 2.60 GB/s; citm_catalog.json sonic-cpp 1.75 GB/s; citm_catalog.json Boost.J​SON 1.16 GB/s; citm_catalog.json nlohmann/json json::parse 0.72 GB/s; canada.json nlohmann/json json_document 2.39 GB/s; canada.json yyjson 1.51 GB/s; canada.json simdjson D​OM 1.07 GB/s; canada.json sonic-rs (Rust) 1.18 GB/s; canada.json sonic-cpp 0.87 GB/s; canada.json Boost.J​SON 0.91 GB/s; canada.json nlohmann/json json::parse 0.38 GB/s; jeopardy.json nlohmann/json json_document 2.45 GB/s; jeopardy.json yyjson 1.83 GB/s; jeopardy.json simdjson D​OM 2.06 GB/s; jeopardy.json sonic-rs (Rust) 1.84 GB/s; jeopardy.json sonic-cpp 1.36 GB/s; jeopardy.json Boost.J​SON 0.62 GB/s; jeopardy.json nlohmann/json json::parse 0.34 GB/s. nlohmann/json json_document diff --git a/docs/mkdocs/docs/images/perf_blog_view_dump.svg b/docs/mkdocs/docs/images/perf_blog_view_dump.svg new file mode 100644 index 000000000..7bb658e13 --- /dev/null +++ b/docs/mkdocs/docs/images/perf_blog_view_dump.svg @@ -0,0 +1,106 @@ + +Write throughput of nlohmann/json's json_document and json, and five other libraries, each writing a document parsed beforehand, Apple M1 Max +Grouped horizontal bars of write throughput in GB/s, more is better. twitter.json nlohmann/json json_document 6.27 GB/s; twitter.json yyjson 4.48 GB/s; twitter.json simdjson D​OM 1.32 GB/s; twitter.json sonic-rs (Rust) 2.20 GB/s; twitter.json sonic-cpp 4.46 GB/s; twitter.json Boost.J​SON 0.94 GB/s; twitter.json nlohmann/json json::dump 2.15 GB/s; citm_catalog.json nlohmann/json json_document 3.77 GB/s; citm_catalog.json yyjson 3.02 GB/s; citm_catalog.json simdjson D​OM 1.00 GB/s; citm_catalog.json sonic-rs (Rust) 1.13 GB/s; citm_catalog.json sonic-cpp 2.25 GB/s; citm_catalog.json Boost.J​SON 1.00 GB/s; citm_catalog.json nlohmann/json json::dump 1.75 GB/s; canada.json nlohmann/json json_document 1.29 GB/s; canada.json yyjson 1.22 GB/s; canada.json simdjson D​OM 0.31 GB/s; canada.json sonic-rs (Rust) 0.86 GB/s; canada.json sonic-cpp 0.69 GB/s; canada.json Boost.J​SON 0.71 GB/s; canada.json nlohmann/json json::dump 0.93 GB/s; jeopardy.json nlohmann/json json_document 3.56 GB/s; jeopardy.json yyjson 2.61 GB/s; jeopardy.json simdjson D​OM 0.96 GB/s; jeopardy.json sonic-rs (Rust) 1.55 GB/s; jeopardy.json sonic-cpp 2.65 GB/s; jeopardy.json Boost.J​SON 0.83 GB/s; jeopardy.json nlohmann/json json::dump 1.68 GB/s. + +nlohmann/json json_document + +yyjson + +simdjson D​OM + +sonic-rs (Rust) + +sonic-cpp + +Boost.J​SON + +nlohmann/json json::dump + +0.5 + +1 + +1.5 + +2 + +2.5 + +3 + +3.5 + +4 + +4.5 + +5 + +5.5 + +6 + +6.5 +GB/s, more is better +twitter.json +nlohmann/json json_document, twitter.json: 6.27 GB/s +6.27 +yyjson, twitter.json: 4.48 GB/s +4.48 +simdjson D​OM, twitter.json: 1.32 GB/s +1.32 +sonic-rs (Rust), twitter.json: 2.20 GB/s +2.20 +sonic-cpp, twitter.json: 4.46 GB/s +4.46 +Boost.J​SON, twitter.json: 0.94 GB/s +0.94 +nlohmann/json json::dump, twitter.json: 2.15 GB/s +2.15 +citm_catalog.json +nlohmann/json json_document, citm_catalog.json: 3.77 GB/s +3.77 +yyjson, citm_catalog.json: 3.02 GB/s +3.02 +simdjson D​OM, citm_catalog.json: 1.00 GB/s +1.00 +sonic-rs (Rust), citm_catalog.json: 1.13 GB/s +1.13 +sonic-cpp, citm_catalog.json: 2.25 GB/s +2.25 +Boost.J​SON, citm_catalog.json: 1.00 GB/s +1.00 +nlohmann/json json::dump, citm_catalog.json: 1.75 GB/s +1.75 +canada.json +nlohmann/json json_document, canada.json: 1.29 GB/s +1.29 +yyjson, canada.json: 1.22 GB/s +1.22 +simdjson D​OM, canada.json: 0.31 GB/s +0.31 +sonic-rs (Rust), canada.json: 0.86 GB/s +0.86 +sonic-cpp, canada.json: 0.69 GB/s +0.69 +Boost.J​SON, canada.json: 0.71 GB/s +0.71 +nlohmann/json json::dump, canada.json: 0.93 GB/s +0.93 +jeopardy.json +nlohmann/json json_document, jeopardy.json: 3.56 GB/s +3.56 +yyjson, jeopardy.json: 2.61 GB/s +2.61 +simdjson D​OM, jeopardy.json: 0.96 GB/s +0.96 +sonic-rs (Rust), jeopardy.json: 1.55 GB/s +1.55 +sonic-cpp, jeopardy.json: 2.65 GB/s +2.65 +Boost.J​SON, jeopardy.json: 0.83 GB/s +0.83 +nlohmann/json json::dump, jeopardy.json: 1.68 GB/s +1.68 + + diff --git a/docs/mkdocs/scripts/figures/perf_blog.py b/docs/mkdocs/scripts/figures/perf_blog.py index a8ff34c71..d1476459a 100644 --- a/docs/mkdocs/scripts/figures/perf_blog.py +++ b/docs/mkdocs/scripts/figures/perf_blog.py @@ -8,6 +8,7 @@ Generated files: - perf_blog_swar.svg: the eight-byte string test on one word of input - perf_blog_cliffs.svg: operations that were quadratic or worse, 3.12.0 against 3.13.0 (log scale) - perf_blog_view.svg: parse throughput of json_document and six other document parsers +- perf_blog_view_dump.svg: write throughput of the same engines, each writing a document parsed beforehand The numbers are copied from the tables of the post (the timeline from data/perf_timeline.csv, which `tests/benchmarks/history/history.py summarize` writes); the tables stay on the page as the accessible fallback. Update both together. @@ -319,9 +320,23 @@ VIEW_US = [ # (engine, css class, [µs per file]), fresh documents, from the t ("nlohmann/json json::parse", "nj-charts-c1 nj-charts-dim", [1352, 2386, 5915, 163750]), ] VIEW = [(e, c, [n / (t * 1000) for n, t in zip(VIEW_BYTES, us)]) for e, c, us in VIEW_US] # GB/s +# dump: the bytes of the compact output, and the same engines, from the dump table in "How it compares" +DUMP_BYTES = [466906, 500299, 2090234, 52508728] +DUMP_US = [ + ("nlohmann/json json_document", "nj-charts-c1", [74.5, 132.8, 1625.1, 14757.2]), + ("yyjson", "nj-charts-c2", [104.2, 165.7, 1715.7, 20133.5]), + ("simdjson DOM", "nj-charts-c3", [354.2, 498.2, 6816.1, 54833.4]), + ("sonic-rs (Rust)", "nj-charts-c4", [212.6, 443.9, 2416.8, 33818.5]), + ("sonic-cpp", "nj-charts-c5", [104.6, 222.3, 3044.2, 19836.1]), + ("Boost.JSON", "nj-charts-c6", [497.6, 501.4, 2941.8, 63392.3]), + ("nlohmann/json json::dump", "nj-charts-c1 nj-charts-dim", [216.7, 285.5, 2244.0, 31205.3]), +] +DUMP = [(e, c, [n / (t * 1000) for n, t in zip(DUMP_BYTES, us)]) for e, c, us in DUMP_US] # GB/s -def view(): +def view(VIEW=VIEW, name="perf_blog_view.svg", tid="nj-blog-view", + title="Parse throughput of nlohmann/json's json_document and json::parse and five other document parsers, fresh documents, Apple M1 Max", + what="parse"): W, L, R = 580, 142, 545 hi = max(max(v) for _, _, v in VIEW) hi = math.ceil(hi * 2) / 2 @@ -354,9 +369,8 @@ def view(): b.append('%.2f' % (X(vals[i]) + 4, y + bh - 1.5, vals[i])) desc.append("%s %s %.2f GB/s" % (f, eng, vals[i])) b.append('' % (X(0), top - 4, X(0), axis_y)) - write("perf_blog_view.svg", W, axis_y + 44, "nj-charts--blog", "nj-blog-view", - "Parse throughput of nlohmann/json's json_document and json::parse and five other document parsers, fresh documents, Apple M1 Max", - "Grouped horizontal bars in GB/s, more is better. " + "; ".join(desc) + ".", b) + write(name, W, axis_y + 44, "nj-charts--blog", tid, title, + "Grouped horizontal bars of %s throughput in GB/s, more is better. " % what + "; ".join(desc) + ".", b) if __name__ == "__main__": @@ -369,3 +383,7 @@ if __name__ == "__main__": cliffs() if VIEW: view() + if DUMP: + view(DUMP, "perf_blog_view_dump.svg", "nj-blog-view-dump", + "Write throughput of nlohmann/json's json_document and json, and five other libraries, each writing a " + "document parsed beforehand, Apple M1 Max", "write") diff --git a/tests/benchmarks/history/history.cpp b/tests/benchmarks/history/history.cpp index cd4794a93..f607bf961 100644 --- a/tests/benchmarks/history/history.cpp +++ b/tests/benchmarks/history/history.cpp @@ -12,8 +12,8 @@ // Prints one line per workload: the name and the median of 9 batches in microseconds per call. #include #if __has_include() && !defined(HISTORY_NO_VIEW) -#include -#define HISTORY_VIEW 1 + #include + #define HISTORY_VIEW 1 #endif #include #include @@ -42,16 +42,25 @@ static void bench(const char* name, const std::function& f) for (;;) { auto t0 = clk::now(); - for (std::size_t i = 0; i < n; ++i) sink = f(); + for (std::size_t i = 0; i < n; ++i) + { + sink = f(); + } double dt = std::chrono::duration(clk::now() - t0).count(); - if (dt > 0.02) break; + if (dt > 0.02) + { + break; + } n *= 2; } std::vector v; for (int b = 0; b < 9; ++b) { auto t0 = clk::now(); - for (std::size_t i = 0; i < n; ++i) sink = f(); + for (std::size_t i = 0; i < n; ++i) + { + sink = f(); + } v.push_back(std::chrono::duration(clk::now() - t0).count() / n); } std::sort(v.begin(), v.end()); @@ -62,7 +71,7 @@ int main(int, char** argv) { std::string d = argv[1]; const std::string tw = slurp(d + "/twitter.json"), ci = slurp(d + "/citm_catalog.json"), - ca = slurp(d + "/canada.json"), cii = slurp(d + "/citm_compact.json"); + ca = slurp(d + "/canada.json"), cii = slurp(d + "/citm_compact.json"); const json jtw = json::parse(tw), jci = json::parse(ci), jca = json::parse(ca); const std::vector cbtw = json::to_cbor(jtw); diff --git a/tests/benchmarks/json_view/others/README.md b/tests/benchmarks/json_view/others/README.md index 5f5509158..7c69da094 100644 --- a/tests/benchmarks/json_view/others/README.md +++ b/tests/benchmarks/json_view/others/README.md @@ -17,10 +17,13 @@ untimed one, and the best round is reported. | `traverse` | parse, then visit every value, convert every number, touch every string and key | | `select` | parse, then read a few fields of every record (the fields are listed in `bench_view.cpp`) | | `parse+dump` | parse, then write the document as compact JSON | +| `dump` | write a document parsed beforehand as compact JSON; only the writing is timed | Before timing, every engine must report the same number of values, the same number of string bytes, and a sum of all numbers in document order that is bit-identical to `json_document`'s (`json::parse` visits objects in key order and is compared within 1e-9): every number is converted exactly. For *select*, all engines must return the same value. +For *dump*, every engine's output must read back as the same values with the same sum, so every float is written +exactly. | engine | version | notes | |---|---|---| @@ -64,7 +67,9 @@ python3 summarize.py mac-cpp-*.log --rust mac-rs-*.log ``` On x86-64, use `-march=haswell -DRAPIDJSON_SSE42` instead of `-DRAPIDJSON_NEON` (sonic-cpp needs AVX2), pin the -process to one core, and add the document name as third argument to time one document per process. Glaze 9 needs +process to one core, and add the document name as third argument to time one document per process. A fourth +argument times only one workload (`./bench_all 30 "" dump`); `bench_rs` takes the workload as third +argument. Glaze 9 needs `std::expected`, which Clang 18 cannot use from GCC 13's standard library; `-DBENCH_NO_GLAZE=1` builds without it. ## Reading the results diff --git a/tests/benchmarks/json_view/others/bench_all.cpp b/tests/benchmarks/json_view/others/bench_all.cpp index 68d64ba0f..8e079a36b 100644 --- a/tests/benchmarks/json_view/others/bench_all.cpp +++ b/tests/benchmarks/json_view/others/bench_all.cpp @@ -16,8 +16,8 @@ #include #include #if !BENCH_NO_GLAZE -#include -#include + #include + #include #endif #include #include @@ -62,13 +62,25 @@ static void walk(const json& j, stats& st) } break; case json::value_t::array: - for (const auto& e : j.get_ref()) walk(e, st); + for (const auto& e : j.get_ref()) + { + walk(e, st); + } + break; + case json::value_t::string: + st.str += j.get_ref().size(); + break; + case json::value_t::number_integer: + st.num += static_cast(*j.get_ptr()); + break; + case json::value_t::number_unsigned: + st.num += static_cast(*j.get_ptr()); + break; + case json::value_t::number_float: + st.num += *j.get_ptr(); + break; + default: break; - case json::value_t::string: st.str += j.get_ref().size(); break; - case json::value_t::number_integer: st.num += static_cast(*j.get_ptr()); break; - case json::value_t::number_unsigned: st.num += static_cast(*j.get_ptr()); break; - case json::value_t::number_float: st.num += *j.get_ptr(); break; - default: break; } } @@ -85,13 +97,25 @@ static void walk(json_view v, stats& st) } break; case json::value_t::array: - for (const json_view e : v) walk(e, st); + for (const json_view e : v) + { + walk(e, st); + } + break; + case json::value_t::string: + st.str += v.get_string().size(); + break; + case json::value_t::number_integer: + st.num += static_cast(v.get()); + break; + case json::value_t::number_unsigned: + st.num += static_cast(v.get()); + break; + case json::value_t::number_float: + st.num += v.get(); + break; + default: break; - case json::value_t::string: st.str += v.get_string().size(); break; - case json::value_t::number_integer: st.num += static_cast(v.get()); break; - case json::value_t::number_unsigned: st.num += static_cast(v.get()); break; - case json::value_t::number_float: st.num += v.get(); break; - default: break; } } @@ -104,7 +128,7 @@ static void walk(yyjson_val* v, stats& st) case YYJSON_TYPE_OBJ: { std::size_t idx, max; - yyjson_val *k, *val; + yyjson_val* k, * val; yyjson_obj_foreach(v, idx, max, k, val) { st.str += yyjson_get_len(k); @@ -119,13 +143,25 @@ static void walk(yyjson_val* v, stats& st) yyjson_arr_foreach(v, idx, max, val) walk(val, st); break; } - case YYJSON_TYPE_STR: st.str += yyjson_get_len(v); break; + case YYJSON_TYPE_STR: + st.str += yyjson_get_len(v); + break; case YYJSON_TYPE_NUM: - if (yyjson_is_sint(v)) st.num += static_cast(yyjson_get_sint(v)); - else if (yyjson_is_uint(v)) st.num += static_cast(yyjson_get_uint(v)); - else st.num += yyjson_get_real(v); + if (yyjson_is_sint(v)) + { + st.num += static_cast(yyjson_get_sint(v)); + } + else if (yyjson_is_uint(v)) + { + st.num += static_cast(yyjson_get_uint(v)); + } + else + { + st.num += yyjson_get_real(v); + } + break; + default: break; - default: break; } } @@ -143,13 +179,25 @@ static void walk(simdjson::dom::element e, stats& st) } break; case simdjson::dom::element_type::ARRAY: - for (auto c : simdjson::dom::array(e)) walk(c, st); + for (auto c : simdjson::dom::array(e)) + { + walk(c, st); + } + break; + case simdjson::dom::element_type::STRING: + st.str += std::string_view(e).size(); + break; + case simdjson::dom::element_type::INT64: + st.num += static_cast(int64_t(e)); + break; + case simdjson::dom::element_type::UINT64: + st.num += static_cast(uint64_t(e)); + break; + case simdjson::dom::element_type::DOUBLE: + st.num += double(e); + break; + default: break; - case simdjson::dom::element_type::STRING: st.str += std::string_view(e).size(); break; - case simdjson::dom::element_type::INT64: st.num += static_cast(int64_t(e)); break; - case simdjson::dom::element_type::UINT64: st.num += static_cast(uint64_t(e)); break; - case simdjson::dom::element_type::DOUBLE: st.num += double(e); break; - default: break; } } @@ -166,22 +214,37 @@ static void walk_od(simdjson::ondemand::value v, stats& st) } break; case simdjson::ondemand::json_type::array: - for (auto c : v.get_array()) walk_od(c.value(), st); + for (auto c : v.get_array()) + { + walk_od(c.value(), st); + } + break; + case simdjson::ondemand::json_type::string: + st.str += std::string_view(v.get_string()).size(); break; - case simdjson::ondemand::json_type::string: st.str += std::string_view(v.get_string()).size(); break; case simdjson::ondemand::json_type::number: { simdjson::ondemand::number n = v.get_number(); switch (n.get_number_type()) { - case simdjson::ondemand::number_type::signed_integer: st.num += static_cast(n.get_int64()); break; - case simdjson::ondemand::number_type::unsigned_integer: st.num += static_cast(n.get_uint64()); break; - default: st.num += n.get_double(); break; + case simdjson::ondemand::number_type::signed_integer: + st.num += static_cast(n.get_int64()); + break; + case simdjson::ondemand::number_type::unsigned_integer: + st.num += static_cast(n.get_uint64()); + break; + default: + st.num += n.get_double(); + break; } break; } - case simdjson::ondemand::json_type::boolean: (void)bool(v.get_bool()); break; - default: (void)v.is_null(); break; + case simdjson::ondemand::json_type::boolean: + (void)bool(v.get_bool()); + break; + default: + (void)v.is_null(); + break; } } @@ -199,15 +262,30 @@ static void walk(const rapidjson::Value& v, stats& st) } break; case rapidjson::kArrayType: - for (const auto& e : v.GetArray()) walk(e, st); + for (const auto& e : v.GetArray()) + { + walk(e, st); + } + break; + case rapidjson::kStringType: + st.str += v.GetStringLength(); break; - case rapidjson::kStringType: st.str += v.GetStringLength(); break; case rapidjson::kNumberType: - if (v.IsDouble()) st.num += v.GetDouble(); - else if (v.IsInt64()) st.num += static_cast(v.GetInt64()); - else st.num += static_cast(v.GetUint64()); + if (v.IsDouble()) + { + st.num += v.GetDouble(); + } + else if (v.IsInt64()) + { + st.num += static_cast(v.GetInt64()); + } + else + { + st.num += static_cast(v.GetUint64()); + } + break; + default: break; - default: break; } } @@ -226,7 +304,10 @@ static void walk_sonic(const N& v, stats& st) } else if (v.IsArray()) { - for (auto e = v.Begin(); e != v.End(); ++e) walk_sonic(*e, st); + for (auto e = v.Begin(); e != v.End(); ++e) + { + walk_sonic(*e, st); + } } else if (v.IsString()) { @@ -234,9 +315,18 @@ static void walk_sonic(const N& v, stats& st) } else if (v.IsNumber()) { - if (v.IsDouble()) st.num += v.GetDouble(); - else if (v.IsUint64()) st.num += static_cast(v.GetUint64()); - else st.num += static_cast(v.GetInt64()); + if (v.IsDouble()) + { + st.num += v.GetDouble(); + } + else if (v.IsUint64()) + { + st.num += static_cast(v.GetUint64()); + } + else + { + st.num += static_cast(v.GetInt64()); + } } } @@ -245,7 +335,7 @@ static void walk_sonic(const N& v, stats& st) static void walk(const glz::generic_u64& j, stats& st) { ++st.nodes; - std::visit([&](const auto& x) + std::visit([&](const auto & x) { using T = std::decay_t; if constexpr (std::is_same_v) @@ -258,10 +348,19 @@ static void walk(const glz::generic_u64& j, stats& st) } else if constexpr (std::is_same_v) { - for (const auto& e : x) walk(e, st); + for (const auto& e : x) + { + walk(e, st); + } + } + else if constexpr (std::is_same_v) + { + st.str += x.size(); + } + else if constexpr (std::is_same_v || std::is_same_v || std::is_same_v) + { + st.num += static_cast(x); } - else if constexpr (std::is_same_v) st.str += x.size(); - else if constexpr (std::is_same_v || std::is_same_v || std::is_same_v) st.num += static_cast(x); }, j.data); } @@ -279,7 +378,10 @@ static void walk_lazy(const V& v, stats& st) } else if (v.is_array()) { - for (auto item : v) walk_lazy(item, st); + for (auto item : v) + { + walk_lazy(item, st); + } } else if (v.is_string()) { @@ -288,9 +390,18 @@ static void walk_lazy(const V& v, stats& st) else if (v.is_number()) { const std::string_view raw = v.raw_json(); - if (raw.find_first_of(".eE") != std::string_view::npos) st.num += v.template get().value(); - else if (raw[0] == '-') st.num += static_cast(v.template get().value()); - else st.num += static_cast(v.template get().value()); + if (raw.find_first_of(".eE") != std::string_view::npos) + { + st.num += v.template get().value(); + } + else if (raw[0] == '-') + { + st.num += static_cast(v.template get().value()); + } + else + { + st.num += static_cast(v.template get().value()); + } } } @@ -310,13 +421,25 @@ static void walk(const boost::json::value& v, stats& st) } break; case boost::json::kind::array: - for (const auto& c : v.get_array()) walk(c, st); + for (const auto& c : v.get_array()) + { + walk(c, st); + } + break; + case boost::json::kind::string: + st.str += v.get_string().size(); + break; + case boost::json::kind::int64: + st.num += static_cast(v.get_int64()); + break; + case boost::json::kind::uint64: + st.num += static_cast(v.get_uint64()); + break; + case boost::json::kind::double_: + st.num += v.get_double(); + break; + default: break; - case boost::json::kind::string: st.str += v.get_string().size(); break; - case boost::json::kind::int64: st.num += static_cast(v.get_int64()); break; - case boost::json::kind::uint64: st.num += static_cast(v.get_uint64()); break; - case boost::json::kind::double_: st.num += v.get_double(); break; - default: break; } } @@ -558,15 +681,50 @@ static double pick_od(const std::string& name, simdjson::ondemand::document& d) // Glaze reading into structs that declare only the fields the selection uses; unknown keys are skipped namespace sel { -struct user { std::string screen_name; }; -struct status { std::uint64_t id{}; sel::user user; std::int64_t retweet_count{}; }; -struct twitter { std::vector statuses; }; -struct performance { std::uint64_t id{}; std::uint64_t eventId{}; std::vector seatCategories; }; -struct citm { std::map events; std::vector performances; }; -struct geometry { std::string type; std::vector coordinates; }; -struct feature { sel::geometry geometry; }; -struct canada { std::string type; std::vector features; }; -struct question { std::string category; std::string round; }; +struct user +{ + std::string screen_name; +}; +struct status +{ + std::uint64_t id{}; + sel::user user; + std::int64_t retweet_count{}; +}; +struct twitter +{ + std::vector statuses; +}; +struct performance +{ + std::uint64_t id{}; + std::uint64_t eventId{}; + std::vector seatCategories; +}; +struct citm +{ + std::map events; + std::vector performances; +}; +struct geometry +{ + std::string type; + std::vector coordinates; +}; +struct feature +{ + sel::geometry geometry; +}; +struct canada +{ + std::string type; + std::vector features; +}; +struct question +{ + std::string category; + std::string round; +}; } constexpr glz::opts sel_opts{.error_on_unknown_keys = false}; @@ -576,7 +734,7 @@ static double pick_glaze_structs(const std::string& name, const std::string& s) double acc = 0; if (name == "twitter" || name == "status") { - auto one = [&](const sel::status& x) + auto one = [&](const sel::status & x) { acc += static_cast(x.id); acc += static_cast(x.user.screen_name.size()); @@ -585,33 +743,54 @@ static double pick_glaze_structs(const std::string& name, const std::string& s) if (name == "twitter") { sel::twitter t; - if (glz::read(t, s)) return -1; - for (const auto& x : t.statuses) one(x); + if (glz::read(t, s)) + { + return -1; + } + for (const auto& x : t.statuses) + { + one(x); + } } else { sel::status x; - if (glz::read(x, s)) return -1; + if (glz::read(x, s)) + { + return -1; + } one(x); } } else if (name == "citm_catalog") { sel::citm c; - if (glz::read(c, s)) return -1; - for (const auto& p : c.performances) acc += static_cast(p.id + p.eventId + p.seatCategories.size()); + if (glz::read(c, s)) + { + return -1; + } + for (const auto& p : c.performances) + { + acc += static_cast(p.id + p.eventId + p.seatCategories.size()); + } acc += static_cast(c.events.size()); } else if (name == "canada") { sel::canada c; - if (glz::read(c, s)) return -1; + if (glz::read(c, s)) + { + return -1; + } acc += static_cast(c.type.size() + c.features[0].geometry.type.size() + c.features[0].geometry.coordinates.size()); } else if (name == "jeopardy") { std::vector q; - if (glz::read(q, s)) return -1; + if (glz::read(q, s)) + { + return -1; + } for (const auto& x : q) { acc += x.round == "Final Jeopardy!" ? 1 : 0; @@ -625,7 +804,7 @@ template static double pick_glaze_lazy(const std::string& name, const D& d) { double acc = 0; - auto one = [&](const auto& x) + auto one = [&](const auto & x) { acc += static_cast(x["id"].template get().value()); acc += static_cast(x["user"]["screen_name"].template get().value().size()); @@ -633,7 +812,10 @@ static double pick_glaze_lazy(const std::string& name, const D& d) }; if (name == "twitter") { - for (auto x : d["statuses"]) one(x); + for (auto x : d["statuses"]) + { + one(x); + } } else if (name == "status") { @@ -642,7 +824,9 @@ static double pick_glaze_lazy(const std::string& name, const D& d) else if (name == "citm_catalog") { for (auto p : d["performances"]) + { acc += static_cast(p["id"].template get().value() + p["eventId"].template get().value() + p["seatCategories"].size()); + } acc += static_cast(d["events"].size()); } else if (name == "canada") @@ -687,14 +871,17 @@ int main(int argc, char** argv) { const std::string T = std::string(argv[1]) + "/"; const int rounds = argc > 2 ? std::atoi(argv[2]) : 30; - const std::string only = argc > 3 ? argv[3] : ""; // time only this document (the check covers all) + const std::string only = argc > 3 ? argv[3] : ""; // time only this document (the check covers all); "" for all + const std::string only_workload = argc > 4 ? argv[4] : ""; // time only this workload struct doc { std::string name, text; int batch; }; std::vector docs; - for (const char* f : {"nativejson-benchmark/twitter.json", "nativejson-benchmark/citm_catalog.json", "nativejson-benchmark/canada.json", "jeopardy/jeopardy.json"}) + for (const char* f : + {"nativejson-benchmark/twitter.json", "nativejson-benchmark/citm_catalog.json", "nativejson-benchmark/canada.json", "jeopardy/jeopardy.json" + }) { std::string n = std::string(f).substr(std::string(f).find('/') + 1); docs.push_back({n.substr(0, n.size() - 5), slurp(T + f), 1}); @@ -708,31 +895,97 @@ int main(int argc, char** argv) stats ref; walk(json::parse(dc.text), ref); std::vector> got(9, {"(not built)", stats{}}); - { auto d = json_document::parse(dc.text); walk(d.root(), got[0].second); got[0].first = "json_view"; } - { yyjson_doc* d = yyjson_read(dc.text.data(), dc.text.size(), 0); walk(yyjson_doc_get_root(d), got[1].second); yyjson_doc_free(d); got[1].first = "yyjson"; } - { simdjson::dom::parser p; walk(p.parse(dc.text).value(), got[2].second); got[2].first = "simdjson DOM"; } - { simdjson::ondemand::parser p; simdjson::padded_string ps(dc.text); auto d = p.iterate(ps).value(); walk_od(d.get_value().value(), got[3].second); got[3].first = "simdjson OD"; } - { rapidjson::Document d; d.Parse(dc.text.data(), dc.text.size()); if (d.HasParseError()) std::puts("RapidJSON parse error"); walk(d, got[4].second); got[4].first = "RapidJSON"; } - { rapidjson::Document d; d.Parse(dc.text.data(), dc.text.size()); stats x; walk(d, x); std::printf("info %-13s RapidJSON default flags: sum %s\n", dc.name.c_str(), x.num == got[0].second.num ? "exact" : "differs from the correctly rounded one"); } - { sonic_json::Document d; d.Parse(dc.text.data(), dc.text.size()); if (d.HasParseError()) std::puts("sonic parse error"); walk_sonic(d, got[5].second); got[5].first = "sonic-cpp"; } + { + auto d = json_document::parse(dc.text); + walk(d.root(), got[0].second); + got[0].first = "json_view"; + } + { + yyjson_doc* d = yyjson_read(dc.text.data(), dc.text.size(), 0); + walk(yyjson_doc_get_root(d), got[1].second); + yyjson_doc_free(d); + got[1].first = "yyjson"; + } + { + simdjson::dom::parser p; + walk(p.parse(dc.text).value(), got[2].second); + got[2].first = "simdjson DOM"; + } + { + simdjson::ondemand::parser p; + simdjson::padded_string ps(dc.text); + auto d = p.iterate(ps).value(); + walk_od(d.get_value().value(), got[3].second); + got[3].first = "simdjson OD"; + } + { + rapidjson::Document d; + d.Parse(dc.text.data(), dc.text.size()); + if (d.HasParseError()) + { + std::puts("RapidJSON parse error"); + } + walk(d, got[4].second); + got[4].first = "RapidJSON"; + } + { + rapidjson::Document d; + d.Parse(dc.text.data(), dc.text.size()); + stats x; + walk(d, x); + std::printf("info %-13s RapidJSON default flags: sum %s\n", dc.name.c_str(), x.num == got[0].second.num ? "exact" : "differs from the correctly rounded one"); + } + { + sonic_json::Document d; + d.Parse(dc.text.data(), dc.text.size()); + if (d.HasParseError()) + { + std::puts("sonic parse error"); + } + walk_sonic(d, got[5].second); + got[5].first = "sonic-cpp"; + } #if !BENCH_NO_GLAZE - { glz::generic_u64 j; auto ec = glz::read_json(j, dc.text); if (ec) std::puts("glaze parse error"); walk(j, got[6].second); got[6].first = "Glaze generic"; } + { + glz::generic_u64 j; + auto ec = glz::read_json(j, dc.text); + if (ec) + { + std::puts("glaze parse error"); + } + walk(j, got[6].second); + got[6].first = "Glaze generic"; + } #endif #if !BENCH_NO_GLAZE - { auto r = glz::lazy_json(dc.text); walk_lazy(r->root(), got[7].second); got[7].first = "Glaze lazy"; } + { + auto r = glz::lazy_json(dc.text); + walk_lazy(r->root(), got[7].second); + got[7].first = "Glaze lazy"; + } #endif - { auto v = boost::json::parse(dc.text); walk(v, got[8].second); got[8].first = "Boost.JSON"; } + { + auto v = boost::json::parse(dc.text); + walk(v, got[8].second); + got[8].first = "Boost.JSON"; + } std::printf("check %-13s", dc.name.c_str()); for (const auto& g : got) { - if (g.first == "(not built)") continue; + if (g.first == "(not built)") + { + continue; + } const bool ok = same(ref, g.second); all_ok = all_ok && ok; // every engine but json::parse (std::map order) visits the values in document order, so their sums // must be bit-identical to json_view's if every number was converted to the same double const bool exact = g.second.num == got[0].second.num; std::printf(" %s:%s%s", g.first.c_str(), ok ? "OK" : "MISMATCH", exact ? "" : "(inexact)"); - if (!ok) std::printf("(nodes %zu/%zu str %zu/%zu num %.17g/%.17g)", g.second.nodes, ref.nodes, g.second.str, ref.str, g.second.num, ref.num); + if (!ok) + { + std::printf("(nodes %zu/%zu str %zu/%zu num %.17g/%.17g)", g.second.nodes, ref.nodes, g.second.str, ref.str, g.second.num, ref.num); + } } std::printf("\n"); } @@ -758,78 +1011,186 @@ int main(int argc, char** argv) all_ok = all_ok && ok; std::printf("check %-13s select %s (json_view %.17g yyjson %.17g OD %.17g Glaze structs %.17g lazy %.17g)\n", dc.name.c_str(), ok ? "OK" : "MISMATCH", ref, py, po, pg, pl); } - if (!all_ok) std::printf("WARNING: some engines disagree, see above\n"); + for (const auto& dc : docs) + { + // every writer's output must read back as the same values; all but json::dump (std::map order) keep the + // document order, so their sums must be bit-identical to the input's: no float may change when written + const auto in = json_document::parse(dc.text); + stats ref; + walk(in.root(), ref); + std::vector> out; + out.push_back({"json_view", in.root().dump()}); + { + yyjson_doc* d = yyjson_read(dc.text.data(), dc.text.size(), 0); + std::size_t n = 0; + char* o = yyjson_write(d, 0, &n); + out.push_back({"yyjson", std::string(o, n)}); + std::free(o); + yyjson_doc_free(d); + } + { + simdjson::dom::parser p; + out.push_back({"simdjson DOM", simdjson::to_string(p.parse(dc.text).value())}); + } + { + rapidjson::Document d; + d.Parse(dc.text.data(), dc.text.size()); + rapidjson::StringBuffer sb; + rapidjson::Writer w(sb); + d.Accept(w); + out.push_back({"RapidJSON", std::string(sb.GetString(), sb.GetSize())}); + } + { + sonic_json::Document d; + d.Parse(dc.text.data(), dc.text.size()); + sonic_json::WriteBuffer wb; + d.Serialize(wb); + out.push_back({"sonic-cpp", std::string(wb.ToString(), wb.Size())}); + } +#if !BENCH_NO_GLAZE + { + glz::generic_u64 j; + (void)glz::read_json(j, dc.text); + std::string o; + (void)glz::write_json(j, o); + out.push_back({"Glaze generic", o}); + } +#endif + { + auto v = boost::json::parse(dc.text); + out.push_back({"Boost.JSON", boost::json::serialize(v)}); + } + out.push_back({"json::dump", json::parse(dc.text).dump()}); + std::printf("check %-13s dump", dc.name.c_str()); + for (const auto& o : out) + { + stats st; + bool ok = true; + try + { + const auto back = json_document::parse(o.second); + walk(back.root(), st); + } + catch (...) + { + ok = false; + } + ok = ok && (o.first == "json::dump" ? same(ref, st) : st.nodes == ref.nodes && st.str == ref.str && st.num == ref.num); + all_ok = all_ok && ok; + std::printf(" %s:%s(%zu bytes)", o.first.c_str(), ok ? "OK" : "MISMATCH", o.second.size()); + } + std::printf("\n"); + } + if (!all_ok) + { + std::printf("WARNING: some engines disagree, see above\n"); + } json_document reused; // read() reuses its memory, as simdjson DOM reuses its parser simdjson::dom::parser sj; simdjson::ondemand::parser od; for (const auto& dc : docs) { - if (!only.empty() && dc.name != only) continue; + if (!only.empty() && dc.name != only) + { + continue; + } const std::string& s = dc.text; const simdjson::padded_string ps(s); + // for dump: each engine's document, parsed once outside the timer + const json_document dump_view = json_document::parse(s); + yyjson_doc* const dump_yy = yyjson_read(s.data(), s.size(), 0); + simdjson::dom::parser dump_sj_parser; + const simdjson::dom::element dump_sj = dump_sj_parser.parse(ps).value_unsafe(); + rapidjson::Document dump_rj; + dump_rj.Parse(s.data(), s.size()); + sonic_json::Document dump_sonic; + dump_sonic.Parse(s.data(), s.size()); +#if !BENCH_NO_GLAZE + glz::generic_u64 dump_glaze; + (void)glz::read_json(dump_glaze, s); +#endif + boost::json::monotonic_resource dump_mr; + const boost::json::value dump_boost = boost::json::parse(s, &dump_mr); + const json dump_json = json::parse(s); std::vector>> workloads; workloads.push_back({"parse", { - {"json_view", [&] { auto d = json_document::parse(s); g_sink = static_cast(d.node_count()); }}, - {"json_view (reused)", [&] { reused.read(s); g_sink = static_cast(reused.node_count()); }}, - {"yyjson", [&] { yyjson_doc* d = yyjson_read(s.data(), s.size(), 0); g_sink = static_cast(yyjson_doc_get_val_count(d)); yyjson_doc_free(d); }}, - {"simdjson DOM", [&] { auto e = sj.parse(ps).value_unsafe(); g_sink = e.is_object(); }}, - {"simdjson DOM (fresh)", [&] { simdjson::dom::parser p; auto e = p.parse(ps).value_unsafe(); g_sink = e.is_object(); }}, - {"RapidJSON", [&] { rapidjson::Document d; d.Parse(s.data(), s.size()); g_sink = d.IsObject(); }}, - {"RapidJSON (default flags)", [&] { rapidjson::Document d; d.Parse(s.data(), s.size()); g_sink = d.IsObject(); }}, - {"RapidJSON (in situ, with copy)", [&] { std::string c = s; rapidjson::Document d; d.ParseInsitu(&c[0]); g_sink = d.IsObject(); }}, - {"sonic-cpp", [&] { sonic_json::Document d; d.Parse(s.data(), s.size()); g_sink = d.IsObject(); }}, + {"json_view", [&] { auto d = json_document::parse(s); g_sink = static_cast(d.node_count()); }}, + {"json_view (reused)", [&] { reused.read(s); g_sink = static_cast(reused.node_count()); }}, + {"yyjson", [&] { yyjson_doc* d = yyjson_read(s.data(), s.size(), 0); g_sink = static_cast(yyjson_doc_get_val_count(d)); yyjson_doc_free(d); }}, + {"simdjson DOM", [&] { auto e = sj.parse(ps).value_unsafe(); g_sink = e.is_object(); }}, + {"simdjson DOM (fresh)", [&] { simdjson::dom::parser p; auto e = p.parse(ps).value_unsafe(); g_sink = e.is_object(); }}, + {"RapidJSON", [&] { rapidjson::Document d; d.Parse(s.data(), s.size()); g_sink = d.IsObject(); }}, + {"RapidJSON (default flags)", [&] { rapidjson::Document d; d.Parse(s.data(), s.size()); g_sink = d.IsObject(); }}, + {"RapidJSON (in situ, with copy)", [&] { std::string c = s; rapidjson::Document d; d.ParseInsitu(&c[0]); g_sink = d.IsObject(); }}, + {"sonic-cpp", [&] { sonic_json::Document d; d.Parse(s.data(), s.size()); g_sink = d.IsObject(); }}, #if !BENCH_NO_GLAZE - {"Glaze generic", [&] { glz::generic_u64 j; auto ec = glz::read_json(j, s); g_sink = static_cast(bool(ec)); }}, + {"Glaze generic", [&] { glz::generic_u64 j; auto ec = glz::read_json(j, s); g_sink = static_cast(bool(ec)); }}, #endif - {"Boost.JSON", [&] { boost::json::monotonic_resource mr; auto v = boost::json::parse(s, &mr); g_sink = v.is_object(); }}, - {"json::parse", [&] { json j = json::parse(s); g_sink = static_cast(j.size()); }}, - }}); + {"Boost.JSON", [&] { boost::json::monotonic_resource mr; auto v = boost::json::parse(s, &mr); g_sink = v.is_object(); }}, + {"json::parse", [&] { json j = json::parse(s); g_sink = static_cast(j.size()); }}, + }}); workloads.push_back({"traverse", { - {"json_view", [&] { auto d = json_document::parse(s); stats st; walk(d.root(), st); g_sink = st.num; }}, - {"json_view (reused)", [&] { reused.read(s); stats st; walk(reused.root(), st); g_sink = st.num; }}, - {"yyjson", [&] { yyjson_doc* d = yyjson_read(s.data(), s.size(), 0); stats st; walk(yyjson_doc_get_root(d), st); g_sink = st.num; yyjson_doc_free(d); }}, - {"simdjson DOM", [&] { stats st; walk(sj.parse(ps).value_unsafe(), st); g_sink = st.num; }}, - {"simdjson OD", [&] { auto d = od.iterate(ps).value_unsafe(); stats st; walk_od(d.get_value().value_unsafe(), st); g_sink = st.num; }}, - {"RapidJSON", [&] { rapidjson::Document d; d.Parse(s.data(), s.size()); stats st; walk(d, st); g_sink = st.num; }}, - {"RapidJSON (default flags)", [&] { rapidjson::Document d; d.Parse(s.data(), s.size()); stats st; walk(d, st); g_sink = st.num; }}, - {"sonic-cpp", [&] { sonic_json::Document d; d.Parse(s.data(), s.size()); stats st; walk_sonic(d, st); g_sink = st.num; }}, + {"json_view", [&] { auto d = json_document::parse(s); stats st; walk(d.root(), st); g_sink = st.num; }}, + {"json_view (reused)", [&] { reused.read(s); stats st; walk(reused.root(), st); g_sink = st.num; }}, + {"yyjson", [&] { yyjson_doc* d = yyjson_read(s.data(), s.size(), 0); stats st; walk(yyjson_doc_get_root(d), st); g_sink = st.num; yyjson_doc_free(d); }}, + {"simdjson DOM", [&] { stats st; walk(sj.parse(ps).value_unsafe(), st); g_sink = st.num; }}, + {"simdjson OD", [&] { auto d = od.iterate(ps).value_unsafe(); stats st; walk_od(d.get_value().value_unsafe(), st); g_sink = st.num; }}, + {"RapidJSON", [&] { rapidjson::Document d; d.Parse(s.data(), s.size()); stats st; walk(d, st); g_sink = st.num; }}, + {"RapidJSON (default flags)", [&] { rapidjson::Document d; d.Parse(s.data(), s.size()); stats st; walk(d, st); g_sink = st.num; }}, + {"sonic-cpp", [&] { sonic_json::Document d; d.Parse(s.data(), s.size()); stats st; walk_sonic(d, st); g_sink = st.num; }}, #if !BENCH_NO_GLAZE - {"Glaze generic", [&] { glz::generic_u64 j; auto ec = glz::read_json(j, s); stats st; walk(j, st); g_sink = st.num + static_cast(bool(ec)); }}, + {"Glaze generic", [&] { glz::generic_u64 j; auto ec = glz::read_json(j, s); stats st; walk(j, st); g_sink = st.num + static_cast(bool(ec)); }}, #endif #if !BENCH_NO_GLAZE - {"Glaze lazy", [&] { auto r = glz::lazy_json(s); stats st; walk_lazy(r->root(), st); g_sink = st.num; }}, + {"Glaze lazy", [&] { auto r = glz::lazy_json(s); stats st; walk_lazy(r->root(), st); g_sink = st.num; }}, #endif - {"Boost.JSON", [&] { boost::json::monotonic_resource mr; auto v = boost::json::parse(s, &mr); stats st; walk(v, st); g_sink = st.num; }}, - {"json::parse", [&] { json j = json::parse(s); stats st; walk(j, st); g_sink = st.num; }}, - }}); + {"Boost.JSON", [&] { boost::json::monotonic_resource mr; auto v = boost::json::parse(s, &mr); stats st; walk(v, st); g_sink = st.num; }}, + {"json::parse", [&] { json j = json::parse(s); stats st; walk(j, st); g_sink = st.num; }}, + }}); { const std::string name = dc.name; workloads.push_back({"select", { - {"json_view", [&, name] { auto d = json_document::parse(s); g_sink = pick(name, d.root()); }}, - {"json_view (reused)", [&, name] { reused.read(s); g_sink = pick(name, reused.root()); }}, - {"yyjson", [&, name] { yyjson_doc* d = yyjson_read(s.data(), s.size(), 0); g_sink = pick(name, yyjson_doc_get_root(d)); yyjson_doc_free(d); }}, - {"simdjson OD", [&, name] { auto d = od.iterate(ps).value_unsafe(); g_sink = pick_od(name, d); }}, + {"json_view", [&, name] { auto d = json_document::parse(s); g_sink = pick(name, d.root()); }}, + {"json_view (reused)", [&, name] { reused.read(s); g_sink = pick(name, reused.root()); }}, + {"yyjson", [&, name] { yyjson_doc* d = yyjson_read(s.data(), s.size(), 0); g_sink = pick(name, yyjson_doc_get_root(d)); yyjson_doc_free(d); }}, + {"simdjson OD", [&, name] { auto d = od.iterate(ps).value_unsafe(); g_sink = pick_od(name, d); }}, #if !BENCH_NO_GLAZE - {"Glaze structs", [&, name] { g_sink = pick_glaze_structs(name, s); }}, - {"Glaze lazy", [&, name] { auto r = glz::lazy_json(s); g_sink = pick_glaze_lazy(name, *r); }}, + {"Glaze structs", [&, name] { g_sink = pick_glaze_structs(name, s); }}, + {"Glaze lazy", [&, name] { auto r = glz::lazy_json(s); g_sink = pick_glaze_lazy(name, *r); }}, #endif - }}); + }}); } workloads.push_back({"parse+dump", { - {"json_view", [&] { auto d = json_document::parse(s); std::string o = d.root().dump(); g_sink = static_cast(o.size()); }}, - {"yyjson", [&] { yyjson_doc* d = yyjson_read(s.data(), s.size(), 0); std::size_t n = 0; char* o = yyjson_write(d, 0, &n); g_sink = static_cast(n); std::free(o); yyjson_doc_free(d); }}, - {"simdjson DOM", [&] { std::string o = simdjson::to_string(sj.parse(ps).value_unsafe()); g_sink = static_cast(o.size()); }}, - {"RapidJSON", [&] { rapidjson::Document d; d.Parse(s.data(), s.size()); rapidjson::StringBuffer sb; rapidjson::Writer w(sb); d.Accept(w); g_sink = static_cast(sb.GetSize()); }}, - {"sonic-cpp", [&] { sonic_json::Document d; d.Parse(s.data(), s.size()); sonic_json::WriteBuffer wb; d.Serialize(wb); g_sink = static_cast(wb.Size()); }}, + {"json_view", [&] { auto d = json_document::parse(s); std::string o = d.root().dump(); g_sink = static_cast(o.size()); }}, + {"yyjson", [&] { yyjson_doc* d = yyjson_read(s.data(), s.size(), 0); std::size_t n = 0; char* o = yyjson_write(d, 0, &n); g_sink = static_cast(n); std::free(o); yyjson_doc_free(d); }}, + {"simdjson DOM", [&] { std::string o = simdjson::to_string(sj.parse(ps).value_unsafe()); g_sink = static_cast(o.size()); }}, + {"RapidJSON", [&] { rapidjson::Document d; d.Parse(s.data(), s.size()); rapidjson::StringBuffer sb; rapidjson::Writer w(sb); d.Accept(w); g_sink = static_cast(sb.GetSize()); }}, + {"sonic-cpp", [&] { sonic_json::Document d; d.Parse(s.data(), s.size()); sonic_json::WriteBuffer wb; d.Serialize(wb); g_sink = static_cast(wb.Size()); }}, #if !BENCH_NO_GLAZE - {"Glaze generic", [&] { glz::generic_u64 j; (void)glz::read_json(j, s); std::string o; (void)glz::write_json(j, o); g_sink = static_cast(o.size()); }}, + {"Glaze generic", [&] { glz::generic_u64 j; (void)glz::read_json(j, s); std::string o; (void)glz::write_json(j, o); g_sink = static_cast(o.size()); }}, #endif - {"Boost.JSON", [&] { boost::json::monotonic_resource mr; auto v = boost::json::parse(s, &mr); std::string o = boost::json::serialize(v); g_sink = static_cast(o.size()); }}, - {"json::parse", [&] { json j = json::parse(s); std::string o = j.dump(); g_sink = static_cast(o.size()); }}, - }}); + {"Boost.JSON", [&] { boost::json::monotonic_resource mr; auto v = boost::json::parse(s, &mr); std::string o = boost::json::serialize(v); g_sink = static_cast(o.size()); }}, + {"json::parse", [&] { json j = json::parse(s); std::string o = j.dump(); g_sink = static_cast(o.size()); }}, + }}); + workloads.push_back({"dump", { + {"json_view", [&] { std::string o = dump_view.root().dump(); g_sink = static_cast(o.size()); }}, + {"yyjson", [&] { std::size_t n = 0; char* o = yyjson_write(dump_yy, 0, &n); g_sink = static_cast(n); std::free(o); }}, + {"simdjson DOM", [&] { std::string o = simdjson::to_string(dump_sj); g_sink = static_cast(o.size()); }}, + {"RapidJSON", [&] { rapidjson::StringBuffer sb; rapidjson::Writer w(sb); dump_rj.Accept(w); g_sink = static_cast(sb.GetSize()); }}, + {"sonic-cpp", [&] { sonic_json::WriteBuffer wb; dump_sonic.Serialize(wb); g_sink = static_cast(wb.Size()); }}, +#if !BENCH_NO_GLAZE + {"Glaze generic", [&] { std::string o; (void)glz::write_json(dump_glaze, o); g_sink = static_cast(o.size()); }}, +#endif + {"Boost.JSON", [&] { std::string o = boost::json::serialize(dump_boost); g_sink = static_cast(o.size()); }}, + {"json::dump", [&] { std::string o = dump_json.dump(); g_sink = static_cast(o.size()); }}, + }}); for (auto& wl : workloads) { + if (!only_workload.empty() && wl.first != only_workload) + { + continue; + } std::vector best(wl.second.size(), 1e300); const int r = s.size() > 10000000 ? std::max(3, rounds / 5) : rounds; for (int i = 0; i < r; ++i) @@ -838,15 +1199,21 @@ int main(int argc, char** argv) { wl.second[k].fn(); // untimed: the previous engine's leftovers are cleaned up here const auto t0 = std::chrono::steady_clock::now(); - for (int b = 0; b < dc.batch; ++b) wl.second[k].fn(); + for (int b = 0; b < dc.batch; ++b) + { + wl.second[k].fn(); + } best[k] = std::min(best[k], std::chrono::duration(std::chrono::steady_clock::now() - t0).count() / dc.batch); } } std::printf("%-13s %-11s", dc.name.c_str(), wl.first.c_str()); for (std::size_t k = 0; k < wl.second.size(); ++k) + { std::printf(" %s %.1fus (%.2fx)", wl.second[k].name.c_str(), best[k] / 1e3, best[k] / best[0]); + } std::printf("\n"); std::fflush(stdout); } + yyjson_doc_free(dump_yy); } } diff --git a/tests/benchmarks/json_view/others/rust/src/main.rs b/tests/benchmarks/json_view/others/rust/src/main.rs index 42da1c3cf..87c392a71 100644 --- a/tests/benchmarks/json_view/others/rust/src/main.rs +++ b/tests/benchmarks/json_view/others/rust/src/main.rs @@ -7,8 +7,8 @@ // SPDX-License-Identifier: MIT // Same workloads and method as bench_all.cpp: parse (build + free), traverse (parse, then visit every -// value, convert every number, touch every string and key), parse+dump; every engine interleaved, an -// untimed call before each timed one, best round reported. +// value, convert every number, touch every string and key), parse+dump, and dump (write a document parsed +// beforehand); every engine interleaved, an untimed call before each timed one, best round reported. // simd-json parses in place, so its timings include copying the input into a mutable buffer. use simd_json::prelude::*; use std::time::Instant; @@ -108,6 +108,7 @@ fn main() { let args: Vec = std::env::args().collect(); let t = format!("{}/", args[1]); let rounds: usize = args.get(2).map(|r| r.parse().unwrap()).unwrap_or(30); + let only_workload: String = args.get(3).cloned().unwrap_or_default(); let mut docs: Vec<(String, String, usize)> = Vec::new(); for f in ["nativejson-benchmark/twitter.json", "nativejson-benchmark/citm_catalog.json", "nativejson-benchmark/canada.json", "jeopardy/jeopardy.json"] { let name = f.split('/').nth(1).unwrap().trim_end_matches(".json").to_string(); @@ -129,7 +130,31 @@ fn main() { name, if same(&r, &a) { "OK" } else { "MISMATCH" }, if same(&r, &b) { "OK" } else { "MISMATCH" }, r.nodes, r.str_, r.num, b.num); } + for (name, text, _) in &docs { + // every writer's output must read back as the same values + let mut r = Stats::default(); + walk_serde(&serde_json::from_str(text).unwrap(), &mut r); + let mut buf = text.as_bytes().to_vec(); + let outs = [ + ("simd-json", simd_json::to_string(&simd_json::to_borrowed_value(&mut buf).unwrap()).unwrap()), + ("sonic-rs", sonic_rs::to_string(&sonic_rs::from_str::(text).unwrap()).unwrap()), + ("serde_json", serde_json::to_string(&serde_json::from_str::(text).unwrap()).unwrap()), + ]; + print!("check {:<13} dump", name); + for (n, o) in &outs { + let mut st = Stats::default(); + walk_serde(&serde_json::from_str(o).unwrap(), &mut st); + print!(" {}:{}({} bytes)", n, if same(&r, &st) { "OK" } else { "MISMATCH" }, o.len()); + } + println!(); + } + for (name, s, batch) in &docs { + // for dump: each engine's document, parsed once outside the timer + let mut dump_buf = s.as_bytes().to_vec(); + let dump_simd = simd_json::to_borrowed_value(&mut dump_buf).unwrap(); + let dump_sonic: sonic_rs::Value = sonic_rs::from_str(s).unwrap(); + let dump_serde: serde_json::Value = serde_json::from_str(s).unwrap(); type Eng<'a> = (&'static str, Box); let workloads: Vec<(&str, Vec)> = vec![ ("parse", vec![ @@ -147,8 +172,16 @@ fn main() { ("sonic-rs", Box::new(|| { let v: sonic_rs::Value = sonic_rs::from_str(s).unwrap(); let o = sonic_rs::to_string(&v).unwrap(); sink(o.len() as f64); })), ("serde_json", Box::new(|| { let v: serde_json::Value = serde_json::from_str(s).unwrap(); let o = serde_json::to_string(&v).unwrap(); sink(o.len() as f64); })), ]), + ("dump", vec![ + ("simd-json", Box::new(|| { let o = simd_json::to_string(&dump_simd).unwrap(); sink(o.len() as f64); })), + ("sonic-rs", Box::new(|| { let o = sonic_rs::to_string(&dump_sonic).unwrap(); sink(o.len() as f64); })), + ("serde_json", Box::new(|| { let o = serde_json::to_string(&dump_serde).unwrap(); sink(o.len() as f64); })), + ]), ]; for (wl, engines) in &workloads { + if !only_workload.is_empty() && *wl != only_workload { + continue; + } let r = if s.len() > 10_000_000 { std::cmp::max(3, rounds / 5) } else { rounds }; let mut best = vec![f64::MAX; engines.len()]; for _ in 0..r { diff --git a/tests/benchmarks/json_view/others/summarize.py b/tests/benchmarks/json_view/others/summarize.py index f0c668513..7e81785e9 100755 --- a/tests/benchmarks/json_view/others/summarize.py +++ b/tests/benchmarks/json_view/others/summarize.py @@ -21,7 +21,7 @@ import sys REUSED = {"json_view (reused)", "simdjson DOM"} DOCS = ["twitter", "citm_catalog", "canada", "jeopardy", "status"] -WORKLOADS = ["parse", "traverse", "select", "parse+dump"] +WORKLOADS = ["parse", "traverse", "select", "parse+dump", "dump"] def parse(path, suffix=""): @@ -58,10 +58,11 @@ def main(): continue jv = cell["json_view"] order = sorted(cell.items(), key=lambda x: x[1]) - fresh = {e: v for e, v in cell.items() if e not in REUSED and e != "json_view"} + # in dump, every engine writes a document parsed beforehand: nothing is reused + fresh = {e: v for e, v in cell.items() if (wl == "dump" or e not in REUSED) and e != "json_view"} other, t = min(fresh.items(), key=lambda x: x[1]) line = "%-13s json_view %10.1f us fastest other: %s %.2f" % (doc, jv, other, t / jv) - if "json_view (reused)" in cell and "simdjson DOM" in cell: + if wl != "dump" and "json_view (reused)" in cell and "simdjson DOM" in cell: line += " | reused: simdjson DOM %.2f" % (cell["simdjson DOM"] / cell["json_view (reused)"]) print(line) print(" " + ", ".join("%s %.2f" % (e, v / jv) for e, v in order))