docs: add the blog post "How 3.13.0 got faster"

A post for the documentation (Home > Blog) on how 3.13.0 became faster:
eight themes with the pull requests behind them, the ideas taken from other
libraries, what did not work, how the fast paths are tested, and the
compile-time cost. Every number comes from a run on 2026-10-27 (3.12.0 against
develop plus the open non-draft PRs, Apple M1 Max) or from the linked PR.

- docs/mkdocs/docs/home/blog/how_3_13_0_got_faster.md: the post.
- docs/mkdocs/scripts/figures/perf_blog.py: draws its 12 figures (overview,
  eight timeline panels, the SWAR test, the cliffs, json_view throughput) as
  inline SVGs in the nj-charts style; data/perf_timeline.csv holds the
  timeline.
- tests/benchmarks/history: history.cpp and history.py time one fixed program
  against the header of every commit that changed it, interleaved, and write
  one CSV row per commit with its PR. The timeline of the post comes from it.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
This commit is contained in:
Niels Lohmann committed 2026-10-30 15:29:45 +01:00
1 parent d20c4dcd89
commit ea848cd2e8
21 files changed
+2297

No files matched your search

+3
View File
@@ -138,6 +138,9 @@ the `View*` benchmarks (in `src/benchmarks_view.cpp`) are only built if the dire
Only benchmarks present in both result files are compared, so for older releases, either filter the benchmarks or
build that release's own `tests/benchmarks` against its own header.
To find out *which commit* changed a number, [`history/`](history/README.md) times one fixed program against the
header of every commit that changed it.
## Getting stable numbers
- Build and run both versions on the same machine with the same compiler and Google Benchmark, one right after the
+43
View File
@@ -0,0 +1,43 @@
# Performance history
The benchmarks in [`tests/benchmarks`](../README.md) compare two versions. The program here answers a different
question: *which commit* made something faster or slower. `history.cpp` is one fixed benchmark that uses only API
3.12.0 already has. `history.py` compiles it against the `single_include` of every first-parent commit that changed
`json.hpp` or `json_view.hpp`, runs all of them interleaved, and writes one CSV row per commit with its pull request.
The timeline in the post [How 3.13.0 got faster](https://json.nlohmann.me/home/blog/how_3_13_0_got_faster/) comes from
it. It is not built by CMake or run by CI.
## Workloads
| name | what it does |
|---|---|
| `parse_twitter`, `parse_citm`, `parse_canada` | `json::parse()` of a `std::string` (`citm_catalog.json` as distributed is indented by four spaces) |
| `parse_citm_compact` | the same with a minified copy of `citm_catalog.json` |
| `parse_twitter_ifstream` | `json::parse()` of an `std::ifstream`, opening the file included |
| `accept_twitter` | `json::accept()` |
| `view_parse_*` | `json_document::parse()`; only where the header has `json_view.hpp` |
| `dump_twitter`, `dump_citm`, `dump_canada`, `dump4_citm` | `dump()` and `dump(4)` |
| `from_cbor_twitter`, `to_cbor_twitter` | read and write CBOR |
| `copy_destroy_citm` | copy a parsed document and destroy the copy |
| `equal_citm` | copy a parsed document and compare it with `==` |
Each workload runs in batches of at least 20 ms; the program prints the median of 9 batches in microseconds per call.
## Running it
```sh
mkdir -p ~/history/data
cp <json_test_data>/nativejson-benchmark/{twitter,citm_catalog,canada}.json ~/history/data/
python3 history.py build --repo ../../.. --work ~/history # about 300 builds, a few minutes in parallel
caffeinate -i python3 history.py run --work ~/history --rounds 4 # about 1 h 45 min on an M1 Max (about 4 s per binary and round)
python3 history.py summarize --work ~/history > history.csv
```
`--extra DIR` adds one more header directory, such as an amalgamation of `develop` with open pull requests merged
in. `run` waits whenever the 1-minute load average is 4 or more.
## Reading the results
A single commit can move a workload by a few percent through code layout alone: the functions it did not touch
move to other addresses. Read a change as a step only if it is larger than that and stays. To check one, time the
two commits against each other with more rounds, or compare instruction counts.
+88
View File
@@ -0,0 +1,88 @@
// __ _____ _____ _____
// __| | __| | | | JSON for Modern C++ (supporting code)
// | | |__ | | | | | | version 3.12.0
// |_____|_____|_____|_|___| https://github.com/nlohmann/json
//
// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
// SPDX-License-Identifier: MIT
// One fixed benchmark, compiled against the json.hpp of every commit that changed it (see README.md).
// It uses only API that 3.12.0 has; the json_document rows exist only where the header has json_view.hpp.
// Usage: history <dir with twitter.json, citm_catalog.json, canada.json and citm_compact.json>
// Prints one line per workload: the name and the median of 9 batches in microseconds per call.
#include <nlohmann/json.hpp>
#if __has_include(<nlohmann/json_view.hpp>) && !defined(HISTORY_NO_VIEW)
#include <nlohmann/json_view.hpp>
#define HISTORY_VIEW 1
#endif
#include <algorithm>
#include <chrono>
#include <cstdio>
#include <fstream>
#include <functional>
#include <sstream>
#include <string>
#include <vector>
using nlohmann::json;
static std::string slurp(const std::string& p)
{
std::ifstream f(p, std::ios::binary);
std::stringstream s;
s << f.rdbuf();
return s.str();
}
static volatile std::size_t sink;
static void bench(const char* name, const std::function<std::size_t()>& f)
{
using clk = std::chrono::steady_clock;
sink = f(); // warm up
std::size_t n = 1;
for (;;)
{
auto t0 = clk::now();
for (std::size_t i = 0; i < n; ++i) sink = f();
double dt = std::chrono::duration<double>(clk::now() - t0).count();
if (dt > 0.02) break;
n *= 2;
}
std::vector<double> v;
for (int b = 0; b < 9; ++b)
{
auto t0 = clk::now();
for (std::size_t i = 0; i < n; ++i) sink = f();
v.push_back(std::chrono::duration<double, std::micro>(clk::now() - t0).count() / n);
}
std::sort(v.begin(), v.end());
std::printf("%s %.3f\n", name, v[4]);
}
int main(int, char** argv)
{
std::string d = argv[1];
const std::string tw = slurp(d + "/twitter.json"), ci = slurp(d + "/citm_catalog.json"),
ca = slurp(d + "/canada.json"), cii = slurp(d + "/citm_compact.json");
const json jtw = json::parse(tw), jci = json::parse(ci), jca = json::parse(ca);
const std::vector<std::uint8_t> cbtw = json::to_cbor(jtw);
bench("parse_twitter", [&] { return json::parse(tw).size(); });
bench("parse_citm", [&] { return json::parse(ci).size(); });
bench("parse_canada", [&] { return json::parse(ca).size(); });
bench("parse_citm_compact", [&] { return json::parse(cii).size(); });
bench("parse_twitter_ifstream", [&] { std::ifstream f(d + "/twitter.json"); return json::parse(f).size(); });
bench("accept_twitter", [&] { return static_cast<std::size_t>(json::accept(tw)); });
#ifdef HISTORY_VIEW
bench("view_parse_twitter", [&] { const auto doc = nlohmann::json_document::parse(tw); return doc.root().size(); });
bench("view_parse_citm", [&] { const auto doc = nlohmann::json_document::parse(ci); return doc.root().size(); });
bench("view_parse_canada", [&] { const auto doc = nlohmann::json_document::parse(ca); return doc.root().size(); });
#endif
bench("dump_twitter", [&] { return jtw.dump().size(); });
bench("dump_citm", [&] { return jci.dump().size(); });
bench("dump_canada", [&] { return jca.dump().size(); });
bench("dump4_citm", [&] { return jci.dump(4).size(); });
bench("from_cbor_twitter", [&] { return json::from_cbor(cbtw).size(); });
bench("to_cbor_twitter", [&] { return json::to_cbor(jtw).size(); });
bench("copy_destroy_citm", [&] { json c = jci; return c.size(); });
bench("equal_citm", [&] { json c = jci; return static_cast<std::size_t>(c == jci); });
}
+151
View File
@@ -0,0 +1,151 @@
#!/usr/bin/env python3
# __ _____ _____ _____
# __| | __| | | | JSON for Modern C++ (supporting code)
# | | |__ | | | | | | version 3.12.0
# |_____|_____|_____|_|___| https://github.com/nlohmann/json
#
# SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
# SPDX-License-Identifier: MIT
"""Time one fixed benchmark against the json.hpp of every commit that changed it.
Usage:
history.py build --repo REPO --work DIR [--from v3.12.0] [--to origin/develop] [--extra INCLUDE_DIR]
history.py run --work DIR [--rounds 4]
history.py summarize --work DIR [--repo REPO] > history.csv
build: lists the first-parent commits between --from and --to that changed single_include/nlohmann/json.hpp or
json_view.hpp, extracts single_include/ of each (and of --from itself), and compiles history.cpp against it into
DIR/bin/NNN, NNN being the line of the commit in DIR/commits.txt. --extra adds one more header directory at the end,
for example an amalgamation of develop with open pull requests merged in. It also writes the input files to DIR/data
(the nativejson-benchmark files from the test data, which must be in DIR/data already, plus a minified copy of
citm_catalog.json).
run: runs every binary once per round, in a new random order each round, and only while the 1-minute load average is
below 4. Output: DIR/res/NNN-R.txt.
summarize: the median over the rounds per binary and workload, as CSV with the commit, its date and its pull request.
Only the Python 3 standard library is used; a C++17 compiler (c++) and git are needed.
"""
import argparse
import concurrent.futures
import csv
import json
import os
import random
import re
import statistics
import subprocess
import sys
import time
from pathlib import Path
HERE = Path(__file__).resolve().parent
FLAGS = ["-std=c++17", "-O3", "-DNDEBUG", "-w"]
def git(repo, *args):
return subprocess.run(["git", "-C", str(repo), *args], check=True, capture_output=True, text=True).stdout
def build_one(work, idx, rev, repo):
hdr = work / "hdr" / idx
hdr.mkdir(parents=True, exist_ok=True)
if rev.startswith("dir:"):
subprocess.run(["cp", "-R", rev[4:] + "/nlohmann", str(hdr)], check=True)
else:
archive = subprocess.run(["git", "-C", str(repo), "archive", rev, "single_include/nlohmann"],
check=True, capture_output=True).stdout
subprocess.run(["tar", "-x", "-C", str(hdr), "--strip-components=1"], input=archive, check=True)
out = work / "bin" / idx
cmd = ["c++", *FLAGS, "-I", str(hdr), str(HERE / "history.cpp"), "-o", str(out)]
# json_view.hpp of the first commits that had it may not compile with history.cpp: time json.hpp alone then
if subprocess.run(cmd, capture_output=True).returncode != 0:
if subprocess.run(cmd[:-4] + ["-DHISTORY_NO_VIEW"] + cmd[-4:], capture_output=True).returncode != 0:
return "FAIL %s %s" % (idx, rev)
return None
def build(a):
work, repo = Path(a.work), Path(a.repo)
(work / "bin").mkdir(parents=True, exist_ok=True)
revs = [a.__dict__["from"]] + git(repo, "log", "--first-parent", "--reverse", "--format=%H",
"%s..%s" % (a.__dict__["from"], a.to), "--",
"single_include/nlohmann/json.hpp", "single_include/nlohmann/json_view.hpp").split()
if a.extra:
revs.append("dir:" + str(Path(a.extra).resolve()))
(work / "commits.txt").write_text("\n".join(revs) + "\n")
data = work / "data"
citm = json.loads((data / "citm_catalog.json").read_text(encoding="utf-8"))
(data / "citm_compact.json").write_text(json.dumps(citm, separators=(",", ":"), ensure_ascii=False), encoding="utf-8")
with concurrent.futures.ThreadPoolExecutor(os.cpu_count()) as ex:
for r in ex.map(lambda p: build_one(work, "%03d" % p[0], p[1], repo), enumerate(revs)):
if r:
print(r, file=sys.stderr)
def load():
out = subprocess.run(["sysctl", "-n", "vm.loadavg"], capture_output=True, text=True).stdout
return float(out.split()[1]) if out else os.getloadavg()[0]
def run(a):
work = Path(a.work)
(work / "res").mkdir(exist_ok=True)
bins = sorted(p.name for p in (work / "bin").iterdir())
for r in range(1, a.rounds + 1):
random.shuffle(bins)
for b in bins:
while load() >= 4:
time.sleep(15)
out = subprocess.run([str(work / "bin" / b), str(work / "data")], check=True, capture_output=True, text=True)
(work / "res" / ("%s-%d.txt" % (b, r))).write_text(out.stdout)
print("round %d done" % r, file=sys.stderr)
def summarize(a):
work = Path(a.work)
revs = (work / "commits.txt").read_text().split()
times, names = {}, []
for f in sorted((work / "res").glob("*.txt")):
idx = int(f.name.split("-")[0])
for line in f.read_text().splitlines():
name, us = line.split()
if name not in names:
names.append(name)
times.setdefault((idx, name), []).append(float(us))
w = csv.writer(sys.stdout, lineterminator="\n")
w.writerow(["index", "commit", "date", "pr", "subject"] + names)
for idx, rev in enumerate(revs):
if rev.startswith("dir:"):
sha, date, pr, subject = "candidate", "", "", "develop + open pull requests"
else:
sha, date, subject = git(a.repo, "log", "-1", "--format=%h%x09%cs%x09%s", rev).strip().split("\t")
m = re.search(r"\(#(\d+)\)$", subject)
pr = m.group(1) if m else ""
w.writerow([idx, sha, date, pr, subject] +
["%.3f" % statistics.median(times[(idx, n)]) if (idx, n) in times else "" for n in names])
def main():
p = argparse.ArgumentParser(description=__doc__.splitlines()[0])
sub = p.add_subparsers(dest="cmd", required=True)
b = sub.add_parser("build")
b.add_argument("--repo", required=True)
b.add_argument("--work", required=True)
b.add_argument("--from", default="v3.12.0")
b.add_argument("--to", default="origin/develop")
b.add_argument("--extra")
r = sub.add_parser("run")
r.add_argument("--work", required=True)
r.add_argument("--rounds", type=int, default=4)
s = sub.add_parser("summarize")
s.add_argument("--work", required=True)
s.add_argument("--repo", default=str(HERE.parents[2]))
a = p.parse_args()
{"build": build, "run": run, "summarize": summarize}[a.cmd](a)
if __name__ == "__main__":
main()