mirror of https://github.com/nlohmann/json.git synced 2026-10-03 13:10:33 +00:00

Files

T

Niels Lohmann fbe05f2ed0 Run the CBOR, MessagePack, BSON and BON8 round-trip invariants in CI

tests/src/round_trip_corpus.hpp exists so that the byte-stability
invariant the fuzzer drivers check also runs on a fixed corpus in CI,
instead of only at OSS-Fuzz. So far only the UBJSON and BJData drivers
had a matching unit test; the CBOR, MessagePack, BSON and BON8 drivers
assert the same invariant (assert(to_X(j2) == vec)) but nothing ran it
outside OSS-Fuzz.

Add "<FORMAT> round-trip invariants" test cases to unit-cbor.cpp,
unit-msgpack.cpp, unit-bson.cpp and unit-bon8.cpp, modeled on the
UBJSON case: seed j1 from the corpus (skipping values that do not
survive the format's own round trip, as the fuzzer drivers only ever
see values from_X() actually produced), then require from_X(to_X(j1))
not to throw and check to_X(j2) == to_X(j1). BSON only serializes
objects, so non-object corpus values are skipped. Update the comments
in round_trip_corpus.hpp and tests/fuzzing.md to name all six formats.

The stream-versus-contiguous check in the BON8 driver is left out, as
#5601 reworks it.

A local probe confirms no violations on the current corpus (CBOR 3849
checked, MessagePack 3909, BSON 2958, BON8 3841 - matching the counts
already recorded for this probe in the issue).

Closes #5714 item 1.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

2026-09-30 18:01:35 +02:00

4.9 KiB

Raw Blame History

Fuzz testing

Each parser of the library (JSON, BJData, BON8, BSON, CBOR, MessagePack, and UBJSON) can be fuzz tested. Currently, libFuzzer and afl++ are supported.

Corpus creation

For most effective fuzzing, a corpus should be provided. A corpus is a directory with some simple input files that cover several features of the parser and is hence a good starting point for mutations.

TEST_DATA_VERSION=3.2.0
wget https://github.com/nlohmann/json_test_data/archive/refs/tags/v$TEST_DATA_VERSION.zip
unzip v$TEST_DATA_VERSION.zip
rm v$TEST_DATA_VERSION.zip
for FORMAT in json bjdata bon8 bson cbor msgpack ubjson
do
  rm -fr corpus_$FORMAT
  mkdir corpus_$FORMAT
  find json_test_data-$TEST_DATA_VERSION -size -5k -name "*.$FORMAT" -exec cp "{}" "corpus_$FORMAT" \;
done
rm -fr json_test_data-$TEST_DATA_VERSION

The generated corpus can be used with both libFuzzer and afl++. The remainder of this documentation assumes the corpus directories have been created in the tests directory.

libFuzzer

To use libFuzzer, you need to pass -fsanitize=fuzzer as FUZZER_ENGINE. In the tests directory, call

make fuzzers FUZZER_ENGINE="-fsanitize=fuzzer"

This creates a fuzz tester binary for each parser that supports these command line options.

In case your default compiler is not a Clang compiler that includes libFuzzer (Clang 6.0 or later), you need to set the CXX variable accordingly. Note the compiler provided by Xcode (AppleClang) does not contain libFuzzer. Please install Clang via Homebrew calling brew install llvm and add CXX=$(brew --prefix llvm)/bin/clang to the make call:

make fuzzers FUZZER_ENGINE="-fsanitize=fuzzer" CXX=$(brew --prefix llvm)/bin/clang

Then pass the corpus directory as command-line argument (assuming it is located in tests):

./parse_cbor_fuzzer corpus_cbor

The fuzzer should be able to run indefinitely without crashing. In case of a crash, the tested input is dumped into a file starting with crash-.

afl++

To use afl++, you need to pass -fsanitize=fuzzer as FUZZER_ENGINE. It will be replaced by a libAFLDriver.a to re-use the same code written for libFuzzer with afl++. Furthermore, set afl-clang-fast++ as compiler.

CXX=afl-clang-fast++ make fuzzers FUZZER_ENGINE="-fsanitize=fuzzer"

Then the fuzzer is called like this in the tests directory:

afl-fuzz -i corpus_cbor -o out  -- ./parse_cbor_fuzzer

The fuzzer should be able to run indefinitely without crashing. In case of a crash, the tested input is written to the directory out.

OSS-Fuzz

The library is further fuzz-tested 24/7 by Google's OSS-Fuzz project. It uses the same fuzzers target as above and also relies on the FUZZER_ENGINE variable. See the used build script for more information.

In case the build at OSS-Fuzz fails, an issue will be created automatically.

Handling OSS-Fuzz reports

OSS-Fuzz files the crashes it finds in its own issue tracker, not on GitHub. So that each report can be traced to the change that fixed it, and each fix to the report it answers, fixes follow these conventions:

Reference the OSS-Fuzz issue in the pull request, next to any GitHub issue it closes, as OSS-Fuzz: <id> (for example, OSS-Fuzz: 563659413), and in the commit message. The ID alone does not disclose the crash. If the report was triaged into a GitHub issue, link the OSS-Fuzz issue there too.
Turn the reproducer into a unit test. Download the testcase from the OSS-Fuzz report, reduce it if possible, and add it as a regression test to the unit test of the affected format (e.g., tests/src/unit-bjdata.cpp), with a comment naming the OSS-Fuzz issue. This way the input is checked by every CI run rather than only by OSS-Fuzz, and it stays covered even if OSS-Fuzz later closes the report as not reproducible.
Keep the fuzzer drivers and the unit tests in sync. The round-trip checks of the BJData, BON8, BSON, CBOR, MessagePack and UBJSON drivers are also run on a fixed corpus in the unit tests (see tests/src/round_trip_corpus.hpp and the "round-trip invariants" test cases), so a regression shows up in CI first. When a driver's checks change, change the unit tests with them.
Record in the report whether the bug shipped. OSS-Fuzz asks whether a crash was a short-lived regression or affects a released version; answer it when the fix is merged, as it decides whether the fix needs a release note or a security advisory (see the security policy).

After the fix is merged, OSS-Fuzz re-runs the reproducer on its next build and marks the report as verified and closed. If it does not, the fix is incomplete.

4.9 KiB Raw Blame History