Compare commits

...
6 Commits
Author SHA1 Message Date
Jeremy Nimmer 6c8ea0a6d1 Remove Bazel alwayslink=True (#5376)
This should have no effect for header only libraries as mentioned.

It was previously removed in e509007d but then accidentally added
again in 26cfec34.

Signed-off-by: Jeremy Nimmer <jeremy.nimmer@tri.global>
2026-09-25 08:46:19 +02:00
Niels Lohmann 01b53c8c15 Keep JSON_DIAGNOSTICS parent pointers of ordered_json members after erase() and update() (#5552)
* Keep JSON_DIAGNOSTICS parent pointers of ordered_json members after erase() and update()

ordered_json stores its members in a vector, and two operations moved
members without restoring their parent pointers afterwards:

- ordered_map::erase() re-constructs every member after the erased one in
  place. The basic_json move constructor leaves m_parent at nullptr, and
  none of the object branches of basic_json::erase() (by key, iterator, or
  iterator range) called set_parents(). This also affected merge_patch()
  with a null member and patch() with a remove operation.
- update() only set the parent pointer of the inserted member. Adding a key
  can reallocate the vector, which copies all other members and leaves
  their m_parent at nullptr. The set_parents() call added for #4813 only
  repaired this for the nested object of a merge, not for the target.

The next assert_invariant() on such an object (for instance, when copying
it) aborted, and diagnostic messages lost the path prefix above the moved
member. std::map-based json was not affected, because its nodes do not
move.

Erasing from an ordered_map object now calls set_parents(), and update()
uses set_parent(), which already refreshes all members for vector-based
objects. This makes the #4813 workaround redundant.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Account for JSON_DIAGNOSTIC_POSITIONS in the ordered_json parent-pointer test

The merge_patch() case parses its input, so with JSON_DIAGNOSTIC_POSITIONS
the exception message also carries the byte range of the parsed value.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Silence clang-tidy for the intentional copy in the ordered_json parent-pointer test

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Keep parent pointers when update() merges past its descent bound

The iterative path of update() only set the parent pointer of the member
it inserted, like the recursive one did before. It now uses set_parent()
too, so ordered_json members that move when a nested object grows keep
their parents, and the set_parents() calls that patched this up after
each nested merge are gone.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-09-25 08:29:36 +02:00
Niels Lohmann cc472af13f Check the fuzzers' UBJSON/BJData round-trip invariants in the unit tests (#5569)
* Check the fuzzers' UBJSON/BJData round-trip invariants in the unit tests

The strongest correctness checks for the UBJSON and BJData writers lived
only in the OSS-Fuzz drivers: anything from_ubjson()/from_bjdata()
returns must serialize with every option combination, parse back, and
re-serialize stably. Those checks only run at OSS-Fuzz, so regressions
surfaced days later as external reports - the same BJData assert pair
was reported five times over three years, and #5494's harness change
was followed by OSS-Fuzz 563659413 within a day.

Add "UBJSON round-trip invariants" and "BJData round-trip invariants"
test cases that run the drivers' checks on a fixed, deterministic corpus
(tests/src/round_trip_corpus.hpp): integer and float boundaries,
non-finite numbers, strings, binary values, optimized containers, deep
nesting, the JData annotated-array matrix, and seeded random containers.
They also check two properties the drivers do not: the first round trip
preserves the value, and re-serializing reproduces the exact bytes. For
BJData both exclude values containing a binary value, which is read back
as an array of integers unless it was written as a Draft 3 optimized
binary array; this carve-out is now documented in bjdata.md. Run against
the headers before #5542, the BJData test fails, including on the shape
from OSS-Fuzz 563659413.

Also document how OSS-Fuzz reports are handled (reference them as
"OSS-Fuzz: <id>", turn the reproducer into a unit test, keep drivers and
unit tests in sync) in tests/fuzzing.md, and link it from the PR
template and the quality assurance page.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Add the OSS-Fuzz reproducers for 474400817 and 474480402 as unit tests

Following the convention added to tests/fuzzing.md, the reproducers of
the two BJData fuzzer asserts tracked since January are now unit tests:

- 474400817 (assert(false)): an empty object _ArraySize_ was written as
  the ND-array header length, which from_bjdata() could not read back.
  Fixed by #5455.

- 474480402 (to_bjdata(j2, false, false) == vec2): a one-byte Draft 3
  binary array is written in Draft 2 mode as a uint8 array and then
  re-serialized with the int8 marker. This is the documented exception to
  byte stability, not a library bug; OSS-Fuzz closed it after #5494
  relaxed the harness to value stability. The test pins the exact bytes
  so the exception stays deliberate.

The 563659413 reproducer is already a unit test (#5542). A comment also
ties the existing UBJSON excessive-count test to the timeout OSS-Fuzz
reported for that shape (testcase 6347769435193344).

OSS-Fuzz: 474400817
OSS-Fuzz: 474480402

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Fix GCC -Weffc++ and -Wuseless-cast warnings in the round-trip corpus

Initialize the atoms in the member initialization list, and drop the cast of
the generator's result, which already is std::size_t on 64-bit Linux.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-09-25 08:29:02 +02:00
Niels Lohmann 80bf54a5a2 Add a security assurance case to the documentation (#5572)
Describe the threat model, the trust boundaries, the secure-design
argument, and how common weaknesses are countered, with links to the
quality assurance page as evidence.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-09-25 08:28:34 +02:00
Niels Lohmann aada27405d Add a roadmap page to the documentation (#5571)
Describe what the project will and will not do over the next year,
and point to issue #3453 for the open question of a 4.0 release.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-09-25 08:28:16 +02:00
Niels Lohmann 3901b223e5 Complete the architecture documentation page (#5570)
* Complete the architecture documentation page

Replace the placeholder bullets and TODOs with a description of the
component pipeline (with a diagram), the source layout, the template
parameters, the value storage (now in struct data), the input and
output adapters, and the SAX interface.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Link sources and basic_json, document full input adapter interface

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Align the default column of the template parameter table

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-09-25 08:28:00 +02:00
19 changed files with 868 additions and 68 deletions
+1
View File
@@ -2,6 +2,7 @@
- [ ] The changes are described in detail, both the what and why. - [ ] The changes are described in detail, both the what and why.
- [ ] If applicable, an [existing issue](https://github.com/nlohmann/json/issues) is referenced. - [ ] If applicable, an [existing issue](https://github.com/nlohmann/json/issues) is referenced.
- [ ] If applicable, a fixed [OSS-Fuzz](https://issues.oss-fuzz.com) issue is referenced as `OSS-Fuzz: <id>` (see [fuzz testing](https://github.com/nlohmann/json/blob/develop/tests/fuzzing.md#handling-oss-fuzz-reports)).
- [ ] The [Code coverage](https://coveralls.io/github/nlohmann/json) remained at 100%. A test case for every new line of code. - [ ] The [Code coverage](https://coveralls.io/github/nlohmann/json) remained at 100%. A test case for every new line of code.
- [ ] If applicable, the [documentation](https://json.nlohmann.me) is updated. - [ ] If applicable, the [documentation](https://json.nlohmann.me) is updated.
- [ ] The source code is amalgamated by running `make amalgamate`. - [ ] The source code is amalgamated by running `make amalgamate`.
-1
View File
@@ -71,7 +71,6 @@ cc_library(
], ],
includes = ["include"], includes = ["include"],
visibility = ["//visibility:public"], visibility = ["//visibility:public"],
alwayslink = True,
) )
cc_library( cc_library(
-1
View File
@@ -42,7 +42,6 @@ string(APPEND CONTENT [=[
], ],
includes = ["include"], includes = ["include"],
visibility = ["//visibility:public"], visibility = ["//visibility:public"],
alwayslink = True,
) )
cc_library( cc_library(
@@ -0,0 +1,70 @@
# Assurance case
This page argues why the library meets its security requirements. It describes the threats the library faces, where the
trust boundaries lie, and how the library's design and the [quality assurance](quality_assurance.md) counter these
threats. To report a vulnerability, see the [security policy](security_policy.md).
## Threat model
The library parses, stores, and serializes JSON values in memory. It does not open network connections, does not open
files (it only reads from streams or `std::FILE*` handles that the caller has already opened), does not read environment
variables, and does not implement cryptography or handle credentials.
The primary threat is therefore **untrusted input**: JSON text or binary data (BJData, BSON, CBOR, MessagePack, UBJSON)
that an attacker controls, passed to [`parse`](../api/basic_json/parse.md), [`accept`](../api/basic_json/accept.md),
[`sax_parse`](../api/basic_json/sax_parse.md), or one of the `from_*` functions such as
[`from_cbor`](../api/basic_json/from_cbor.md). Such input may try to
- make the library read or write out of bounds (malformed lengths, truncated input, invalid UTF-8),
- trigger undefined behavior (integer overflow in sizes or numbers, invalid casts),
- exhaust memory (huge announced sizes), or
- exhaust the call stack (deeply nested arrays and objects).
## Trust boundaries
- **Untrusted:** all serialized input read by the parser, the SAX interface, and the binary readers. The library must
handle every possible input by either producing a value or throwing a [`parse_error`](../home/exceptions.md#parse-errors)
(or returning `false` when exceptions are disabled for the call).
- **Trusted:** the C++ code that calls the library. Calling a function with violated preconditions, for instance
accessing an array with [`operator[]`](../api/basic_json/operator%5B%5D.md) out of range, is a programming error and
not a security boundary. Such preconditions are checked with [runtime assertions](../features/assertions.md) in debug
builds; functions such as [`at`](../api/basic_json/at.md) offer checked access with exceptions.
## Secure design
- **Strict parsing.** The parser accepts exactly the JSON grammar of [RFC 8259](https://datatracker.ietf.org/doc/html/rfc8259).
Extensions such as [comments](../features/comments.md) and [trailing commas](../features/trailing_commas.md) must be
enabled explicitly. Invalid UTF-8 is rejected.
- **Errors are reported, not ignored.** Malformed input results in a [`parse_error`](../home/exceptions.md#parse-errors)
with the byte position of the error. Binary readers do not trust announced sizes: strings and binary values grow
only as bytes are actually read, arrays reserve at most a fixed number of elements up front, and sizes that no
container can hold are rejected.
- **Memory is owned by values.** Each `basic_json` value owns its content, and there is no manual memory management in
user code. The destructor does not recurse, so destroying a deeply nested value does not exhaust the stack.
- **Bounded recursion.** The JSON parser and the binary readers keep their state in explicit stacks instead of
recursing per nesting level. Operations that walk a value, such as [`dump`](../api/basic_json/dump.md), copying,
hashing, and [`merge_patch`](../api/basic_json/merge_patch.md), recurse only up to a fixed depth and continue with an
explicit stack below it. Some operations, such as comparison, [`diff`](../api/basic_json/diff.md),
[`flatten`](../api/basic_json/flatten.md), and the binary writers, still recurse once per nesting level; work on them
is in progress. Applications that process untrusted input can limit its nesting depth with a
[parser callback](../features/parsing/parser_callbacks.md).
- **Invariants are checked.** The class invariant (for instance, that the pointer for the stored type is never null) is
checked with runtime assertions throughout the test suite.
## Common weaknesses
The following table maps the relevant classes of the [Common Weakness Enumeration](https://cwe.mitre.org) to the
measures that counter them. The measures are described in detail in [Quality assurance](quality_assurance.md).
| Weakness | Countermeasures |
|---------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------|
| Out-of-bounds read/write ([CWE-125](https://cwe.mitre.org/data/definitions/125.html), [CWE-787](https://cwe.mitre.org/data/definitions/787.html)) | bounds checks on all reads from the input; AddressSanitizer and Valgrind on the test suite; OSS-Fuzz |
| Integer overflow ([CWE-190](https://cwe.mitre.org/data/definitions/190.html)) | UndefinedBehaviorSanitizer with integer overflow detection; Clang-Tidy; Cppcheck |
| Use after free, double free ([CWE-416](https://cwe.mitre.org/data/definitions/416.html), [CWE-415](https://cwe.mitre.org/data/definitions/415.html)) | ownership of all memory by values; AddressSanitizer and Valgrind; Clang Static Analyzer |
| Memory leaks ([CWE-401](https://cwe.mitre.org/data/definitions/401.html)) | Valgrind (Memcheck) on the test suite |
| Uncontrolled recursion ([CWE-674](https://cwe.mitre.org/data/definitions/674.html)) | iterative parser, binary readers, and destructor; bounded recursion in value operations; tests with deeply nested inputs |
| Uncontrolled resource consumption ([CWE-400](https://cwe.mitre.org/data/definitions/400.html)) | allocations based on announced sizes are capped; OSS-Fuzz with memory limits |
| Undefined behavior in general ([CWE-758](https://cwe.mitre.org/data/definitions/758.html)) | UndefinedBehaviorSanitizer; runtime assertions; Clang-Tidy, Cppcheck, Clang Static Analyzer, Infer |
In addition, every line of the library is covered by the unit tests, and all parsers are fuzz-tested around the clock
by [OSS-Fuzz](https://github.com/google/oss-fuzz/tree/master/projects/json).
+2
View File
@@ -5,4 +5,6 @@
- [Contribution Guidelines](contribution_guidelines.md) - guidelines how to contribute to this project - [Contribution Guidelines](contribution_guidelines.md) - guidelines how to contribute to this project
- [Governance](governance.md) - the governance model of this project - [Governance](governance.md) - the governance model of this project
- [Quality Assurance](quality_assurance.md) - how the quality of this project is assured - [Quality Assurance](quality_assurance.md) - how the quality of this project is assured
- [Roadmap](roadmap.md) - what the project will and will not do
- [Security Policy](security_policy.md) - the security policy of the project - [Security Policy](security_policy.md) - the security policy of the project
- [Assurance Case](assurance_case.md) - why the library meets its security requirements
@@ -164,6 +164,9 @@ Note: Some modern features (like C++20 ranges or filesystem support) may be disa
- [x] The parser is tested against extensive correctness suites for JSON compliance. - [x] The parser is tested against extensive correctness suites for JSON compliance.
- [x] In addition, the library is continuously fuzz-tested at [OSS-Fuzz](https://google.github.io/oss-fuzz/) where the - [x] In addition, the library is continuously fuzz-tested at [OSS-Fuzz](https://google.github.io/oss-fuzz/) where the
library is checked against billions of inputs. library is checked against billions of inputs.
- [x] Every crash reported by OSS-Fuzz is fixed together with a unit test that reproduces it, and the fix references
the OSS-Fuzz issue. The round-trip checks of the fuzzer drivers are also part of the unit tests. See the
[fuzz testing documentation](https://github.com/nlohmann/json/blob/develop/tests/fuzzing.md#handling-oss-fuzz-reports).
## Static analysis ## Static analysis
+43
View File
@@ -0,0 +1,43 @@
# Roadmap
This page describes what the project intends to do, and what it does not intend to do, over the next year. Concrete
work items are tracked in the [GitHub milestones](https://github.com/nlohmann/json/milestones) and the
[issue tracker](https://github.com/nlohmann/json/issues).
## What the project will do
- **Keep the C++11 baseline.** The library will continue to compile with every
[supported C++11 compiler](https://github.com/nlohmann/json/blob/develop/README.md#supported-compilers). Features of
later standards are only used when they are guarded by the `JSON_HAS_CPP_*` macros.
- **Stay conformant to JSON.** The parser and serializer follow [RFC 8259](https://datatracker.ietf.org/doc/html/rfc8259).
Extensions such as [comments](../features/comments.md) or [trailing commas](../features/trailing_commas.md) remain
opt-in.
- **Keep the 3.x public API stable.** Releases follow [semantic versioning](https://semver.org). Changes that would
break existing code are only added behind a feature macro, so users can opt in and test their code before a next
major release.
- **Support a broad range of compilers and platforms.** The [CI](quality_assurance.md) keeps testing old and new
versions of GCC, Clang, MSVC, and other compilers on Linux, macOS, and Windows.
- **Keep the quality assurance up.** Every change keeps the test coverage at 100%, passes the static and dynamic
analysis, and is fuzz-tested by OSS-Fuzz, see [Quality assurance](quality_assurance.md).
- **Harden the library against hostile input.** Handling deeply nested values without exhausting the call stack is
ongoing work.
- **Fix bugs and security issues** reported through the issue tracker and the [security policy](security_policy.md).
## What the project will not do
- **Break the public API of version 3.x.** See the
[contribution guidelines](https://github.com/nlohmann/json/blob/develop/.github/CONTRIBUTING.md#break-the-public-api)
for what counts as a breaking change.
- **Require a newer C++ standard than C++11.**
- **Break JSON conformance** or enable non-standard extensions by default.
- **Add dependencies** or require a build step. The library remains header-only, and the single header
`json.hpp` remains a complete distribution.
- **Trade simplicity for speed or memory efficiency.** Performance improvements are welcome, but the library is not
meant to compete with the fastest JSON libraries, see [Design goals](../home/design_goals.md).
## Version 4.0
There is no decision yet on whether or when a version 4.0 with breaking changes will be released. Proposals that need
a major version, for instance stricter type conversions, are collected in issue
[#3453](https://github.com/nlohmann/json/issues/3453). Until then, such changes are only added as opt-in behavior
behind feature macros.
@@ -208,6 +208,16 @@ The library maps BJData types to JSON value types as follows:
The mapping is **complete** in the sense that any BJData value can be converted to a JSON value. The mapping is **complete** in the sense that any BJData value can be converted to a JSON value.
!!! info "Round trips"
A value returned by [`from_bjdata`](../../api/basic_json/from_bjdata.md) can be serialized with
[`to_bjdata`](../../api/basic_json/to_bjdata.md) using any combination of options and parsed back into an equal
value, and serializing that value again with the same options produces the same bytes. The exception is binary
values: they are only written as an optimized binary array (`[$B`) if Draft 3 is enabled and both `use_size` and
`use_type` are set. Otherwise, they are written as arrays of integers and parsed back as such (see the notes on
binary values above), and serializing such an array again may choose different, but equally valid, type markers.
The bytes can then differ, but parsing them again yields the same value.
??? example ??? example
```cpp ```cpp
+177 -35
View File
@@ -1,34 +1,125 @@
# Architecture # Architecture
!!! info This page gives a high-level overview of the library's architecture. It should help new contributors to get an idea of
the used concepts and where to make changes.
This page is still under construction. Its goal is to provide a high-level overview of the library's architecture.
This should help new contributors to get an idea of the used concepts and where to make changes.
## Overview ## Overview
The main structure is class [nlohmann::basic_json](../api/basic_json/index.md). The library is built around a single class template, [`nlohmann::basic_json`](../api/basic_json/index.md). A
`basic_json` value is a node in a tree of JSON values. All other components either create such a tree from an input
(parsing), write a tree to an output (serialization), or give access to it (iterators, JSON Pointer, conversions).
- public API ```mermaid
- container interface flowchart LR
- iterators input[/"input<br>(string, stream,<br>iterator range, file)"/]
ia["input adapter"]
lexer["lexer"]
parser["parser"]
breader["binary_reader"]
sax["SAX interface"]
value[("basic_json<br>value tree")]
serializer["serializer"]
bwriter["binary_writer"]
oa["output adapter"]
output[/"output<br>(string, stream,<br>vector)"/]
## Template specializations input --> ia
ia --> lexer --> parser --> sax
ia --> breader --> sax
sax --> value
value --> serializer --> oa
value --> bwriter --> oa
oa --> output
```
- describe template parameters of `basic_json` - **JSON text** is read by an [input adapter](#input-adapters), tokenized by the lexer, and turned into SAX events by
- [`json`](../api/json.md) the parser.
- [`ordered_json`](../api/ordered_json.md) via [`ordered_map`](../api/ordered_map.md) - **Binary formats** (BJData, BSON, CBOR, MessagePack, UBJSON) are read by an input adapter and turned into the same SAX
events by the `binary_reader`.
- A [SAX consumer](#sax-interface) receives the events. The one used by [`parse`](../api/basic_json/parse.md) builds a
`basic_json` value tree.
- The `serializer` (JSON text) or the `binary_writer` (binary formats) writes a value tree to an
[output adapter](#output-adapters).
## Source layout
The public headers are in [`include/nlohmann`](https://github.com/nlohmann/json/tree/develop/include/nlohmann):
- [`json.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/json.hpp) defines class [`basic_json`](../api/basic_json/index.md).
- [`json_fwd.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/json_fwd.hpp) contains forward declarations.
- [`adl_serializer.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/adl_serializer.hpp), [`byte_container_with_subtype.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/byte_container_with_subtype.hpp), and [`ordered_map.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/ordered_map.hpp) define
[`adl_serializer`](../api/adl_serializer/index.md),
[`byte_container_with_subtype`](../api/byte_container_with_subtype/index.md), and
[`ordered_map`](../api/ordered_map.md).
Everything else lives in [`detail/`](https://github.com/nlohmann/json/tree/develop/include/nlohmann/detail) and namespace `nlohmann::detail`, which is not part of the public API. Paths
below are relative to `include/nlohmann`.
| Component | Location |
|-----------|----------|
| Value type enumeration | [`detail/value_t.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/value_t.hpp) |
| Input adapters | [`detail/input/input_adapters.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/input_adapters.hpp) |
| Lexer | [`detail/input/lexer.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/lexer.hpp), [`detail/input/number_parse.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/number_parse.hpp), [`detail/input/string_scan.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/string_scan.hpp) |
| Parser | [`detail/input/parser.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/parser.hpp) |
| SAX interface and DOM builders | [`detail/input/json_sax.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/json_sax.hpp) |
| Binary format readers | [`detail/input/binary_reader.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/binary_reader.hpp) |
| JSON serializer | [`detail/output/serializer.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/output/serializer.hpp), [`detail/conversions/to_chars.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/conversions/to_chars.hpp) |
| Binary format writers | [`detail/output/binary_writer.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/output/binary_writer.hpp) |
| Output adapters | [`detail/output/output_adapters.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/output/output_adapters.hpp) |
| Iterators | [`detail/iterators/`](https://github.com/nlohmann/json/tree/develop/include/nlohmann/detail/iterators) |
| Conversions from/to arbitrary types | [`detail/conversions/from_json.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/conversions/from_json.hpp), [`detail/conversions/to_json.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/conversions/to_json.hpp) |
| JSON Pointer | [`detail/json_pointer.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/json_pointer.hpp) |
| Exceptions | [`detail/exceptions.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/exceptions.hpp) |
| Type traits and C++ feature backports | [`detail/meta/`](https://github.com/nlohmann/json/tree/develop/include/nlohmann/detail/meta) |
| Macros | [`detail/macro_scope.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/macro_scope.hpp), [`detail/macro_unscope.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/macro_unscope.hpp), [`detail/abi_macros.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/abi_macros.hpp) |
The single-header version [`single_include/nlohmann/json.hpp`](https://github.com/nlohmann/json/blob/develop/single_include/nlohmann/json.hpp)
is generated from these files with `make amalgamate` and must not be edited by hand.
## Template parameters
[`basic_json`](../api/basic_json/index.md) is parameterized by the types it uses to store values and to convert from and to other types:
| Template parameter | Default | Used for |
|----------------------|-----------------------------|-------------------------------------------------------------------|
| `ObjectType` | `std::map` | objects, see [`object_t`](../api/basic_json/object_t.md) |
| `ArrayType` | `std::vector` | arrays, see [`array_t`](../api/basic_json/array_t.md) |
| `StringType` | `std::string` | strings and object keys, see [`string_t`](../api/basic_json/string_t.md) |
| `BooleanType` | `bool` | Booleans, see [`boolean_t`](../api/basic_json/boolean_t.md) |
| `NumberIntegerType` | `std::int64_t` | signed integers, see [`number_integer_t`](../api/basic_json/number_integer_t.md) |
| `NumberUnsignedType` | `std::uint64_t` | unsigned integers, see [`number_unsigned_t`](../api/basic_json/number_unsigned_t.md) |
| `NumberFloatType` | `double` | floating-point numbers, see [`number_float_t`](../api/basic_json/number_float_t.md) |
| `AllocatorType` | `std::allocator` | allocating objects, arrays, strings, and binary values |
| `JSONSerializer` | `adl_serializer` | conversions from/to other types, see [`adl_serializer`](../api/adl_serializer/index.md) |
| `BinaryType` | `std::vector<std::uint8_t>` | binary values, see [`binary_t`](../api/basic_json/binary_t.md) |
| `CustomBaseClass` | `void` | an optional base class, see [`json_base_class_t`](../api/basic_json/json_base_class_t.md) |
The library provides two specializations:
- [`json`](../api/json.md) uses all default template arguments.
- [`ordered_json`](../api/ordered_json.md) uses [`ordered_map`](../api/ordered_map.md) as `ObjectType` to keep the
insertion order of object keys.
The requirements on the template arguments are listed in
[Template Parameter Requirements](../features/types/template_parameters.md).
## Value storage ## Value storage
Values are stored as a tagged union of [value_t](../api/basic_json/value_t.md) and json_value. Each [`basic_json`](../api/basic_json/index.md) value stores its content as a tagged union: an enumeration [`value_t`](../api/basic_json/value_t.md)
names the type of the value, and a union `json_value` holds the value itself. Both are members of the nested struct
`data`, which is the only data member `m_data` of `basic_json`:
```cpp ```cpp
/// the type of the current element struct data
value_t m_type = value_t::null; {
/// the type of the current element
value_t m_type = value_t::null;
/// the value of the current element /// the value of the current element
json_value m_value = {}; json_value m_value = {};
};
data m_data = {};
``` ```
with with
@@ -68,42 +159,83 @@ union json_value {
}; };
``` ```
## Parsing inputs (deserialization) Objects, arrays, strings, and binary values are allocated on the heap with `AllocatorType`, and the union only stores a
pointer to them. This keeps a `basic_json` value small: one pointer-sized union and one byte for the type. The class
maintains the invariant that the pointer matching `m_type` is never null; `assert_invariant()` checks it with
[runtime assertions](../features/assertions.md).
Input is read via **input adapters** that abstract a source with a common interface: ## Input adapters
Input is read via **input adapters** that abstract a source. Every input adapter provides this interface:
```cpp ```cpp
/// read a single character /// the type of the characters in the input
std::char_traits<char>::int_type get_character() noexcept; using char_type = ...;
/// read multiple characters to a destination buffer and /// read a single character; returns std::char_traits<char_type>::eof() at the end of the input
/// returns the number of characters successfully read typename std::char_traits<char_type>::int_type get_character();
/// read up to count * sizeof(T) bytes into dest and return the number of bytes read
/// (used by the binary readers)
template<class T> template<class T>
std::size_t get_elements(T* dest, std::size_t count = 1); std::size_t get_elements(T* dest, std::size_t count = 1);
``` ```
List examples of input adapters. The lexer detects two optional extensions at compile time. Only `iterator_input_adapter` provides them, and only for
random-access input of single-byte characters:
## SAX Interface - `supports_seek`, `get_consumed_count()`, and `copy_consumed_range()` let the lexer reconstruct already consumed input
for error messages instead of copying every character it reads.
- `supports_bulk_scan`, `bulk_data()`, `bulk_remaining()`, and `bulk_skip()` let the lexer scan strings directly in
contiguous memory, several bytes at a time.
TODO The function `input_adapter` picks the right adapter for the argument passed to `parse`, `accept`, `sax_parse`, or the
`from_*` functions:
## Writing outputs (serialization) - `iterator_input_adapter` reads from an iterator range, which also covers strings, containers, and pointers.
- `wide_string_input_adapter` reads from ranges of `wchar_t`, `char16_t`, or `char32_t` and converts them to UTF-8.
It cannot be used for binary formats; its `get_elements()` throws.
- `input_stream_adapter` reads from a `std::istream`.
- `file_input_adapter` reads from a `std::FILE*`.
## SAX interface
The parser does not build values itself. It reports what it reads as events to a [SAX](../features/parsing/sax_interface.md)
consumer, which implements the interface [`json_sax`](../api/json_sax/index.md): `null`, `boolean`, `number_integer`,
`number_unsigned`, `number_float`, `string`, `binary`, `start_object`, `key`, `end_object`, `start_array`, `end_array`,
and `parse_error`.
The library comes with two consumers in `detail/input/json_sax.hpp`:
- `json_sax_dom_parser` builds a [`basic_json`](../api/basic_json/index.md) value tree. [`parse`](../api/basic_json/parse.md) uses it.
- `json_sax_dom_callback_parser` does the same, but calls a [parser callback](../features/parsing/parser_callbacks.md)
for each event, which can skip values. `parse` uses it when a callback is given.
The `binary_reader` emits the same events for binary formats, so [`sax_parse`](../api/basic_json/sax_parse.md) works
with a user-defined consumer for JSON and for all binary formats alike.
## Output adapters
Output is written via **output adapters**: Output is written via **output adapters**:
```cpp ```cpp
template<typename T>
void write_character(CharType c); void write_character(CharType c);
template<typename CharType>
void write_characters(const CharType* s, std::size_t length); void write_characters(const CharType* s, std::size_t length);
``` ```
List examples of output adapters. The `serializer` (used by [`dump`](../api/basic_json/dump.md) and [`operator<<`](../api/operator_ltlt.md)) and the
`binary_writer` (used by the `to_*` functions) write to one of these adapters:
- `output_vector_adapter` appends to a `std::vector`.
- `output_stream_adapter` writes to a `std::ostream`.
- `output_string_adapter` appends to a string.
## Value conversion ## Value conversion
Values are converted from and to other types with the `JSONSerializer` template parameter. The default,
[`adl_serializer`](../api/adl_serializer/index.md), calls the free functions
```cpp ```cpp
template<class T> template<class T>
void to_json(basic_json& j, const T& t); void to_json(basic_json& j, const T& t);
@@ -112,13 +244,23 @@ template<class T>
void from_json(const basic_json& j, T& t); void from_json(const basic_json& j, T& t);
``` ```
found by argument-dependent lookup. The library defines them for standard types in `detail/conversions`; users add them
for their own types, see [Arbitrary Type Conversions](../features/arbitrary_types.md). The
[serialization macros](../features/macros.md) generate these functions.
## Additional features ## Additional features
- JSON Pointers - [JSON Pointer](../features/json_pointer.md) (class `json_pointer`) addresses values inside a tree. It is also the
- Binary formats basis of [JSON Patch](../features/json_patch.md).
- Custom base class - [Binary formats](../features/binary_formats/index.md) are read by `binary_reader` and written by `binary_writer`.
- Conversion macros - A [custom base class](../api/basic_json/json_base_class_t.md) can add members to every [`basic_json`](../api/basic_json/index.md) value.
- [Serialization macros](../features/macros.md) generate `to_json` and `from_json` functions for user-defined types.
## Details namespace ## Details namespace
- C++ feature backports Namespace `nlohmann::detail` contains all implementation details. It is not part of the public API and may change in any
release. Besides the components above, it contains:
- type traits to detect the capabilities of user-defined types (`detail/meta/type_traits.hpp`),
- backports of C++14/17 features to C++11 (`detail/meta/cpp_future.hpp`), and
- helpers such as `string_concat` and `string_escape`.
+2
View File
@@ -317,7 +317,9 @@ nav:
- community/contribution_guidelines.md - community/contribution_guidelines.md
- community/quality_assurance.md - community/quality_assurance.md
- community/governance.md - community/governance.md
- community/roadmap.md
- community/security_policy.md - community/security_policy.md
- community/assurance_case.md
# Extras # Extras
extra: extra:
+33 -15
View File
@@ -1243,6 +1243,27 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
} }
/// @brief restore the parent pointers after erasing from an object
/// ordered_json keeps its members in a vector, and erasing a member
/// re-constructs every member after it in place, which resets their
/// parent pointers
void set_parents_after_object_erase()
{
#if JSON_DIAGNOSTICS
#ifdef JSON_HEDLEY_MSVC_VERSION
#pragma warning(push )
#pragma warning(disable : 4127) // ignore warning to replace if with if constexpr
#endif
if (detail::is_ordered_map<object_t>::value)
{
set_parents();
}
#ifdef JSON_HEDLEY_MSVC_VERSION
#pragma warning( pop )
#endif
#endif
}
public: public:
////////////////////////// //////////////////////////
// JSON parser callback // // JSON parser callback //
@@ -2931,6 +2952,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
case value_t::object: case value_t::object:
{ {
result.m_it.object_iterator = erase_from_object(pos.m_it.object_iterator); result.m_it.object_iterator = erase_from_object(pos.m_it.object_iterator);
set_parents_after_object_erase();
break; break;
} }
@@ -3003,6 +3025,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
{ {
result.m_it.object_iterator = m_data.m_value.object->erase(first.m_it.object_iterator, result.m_it.object_iterator = m_data.m_value.object->erase(first.m_it.object_iterator,
last.m_it.object_iterator); last.m_it.object_iterator);
set_parents_after_object_erase();
break; break;
} }
@@ -3033,7 +3056,9 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
JSON_THROW(type_error::create(307, detail::concat("cannot use erase() with ", type_name()), this)); JSON_THROW(type_error::create(307, detail::concat("cannot use erase() with ", type_name()), this));
} }
return m_data.m_value.object->erase(std::forward<KeyType>(key)); const auto erased = m_data.m_value.object->erase(std::forward<KeyType>(key));
set_parents_after_object_erase();
return erased;
} }
template < typename KeyType, detail::enable_if_t < template < typename KeyType, detail::enable_if_t <
@@ -3050,6 +3075,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
if (it != m_data.m_value.object->end()) if (it != m_data.m_value.object->end())
{ {
m_data.m_value.object->erase(it); m_data.m_value.object->erase(it);
set_parents_after_object_erase();
return 1; return 1;
} }
return 0; return 0;
@@ -3961,16 +3987,12 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
if (it2 != m_data.m_value.object->end() && it2->second.is_object()) if (it2 != m_data.m_value.object->end() && it2->second.is_object())
{ {
it2->second.update_members(it.value().cbegin(), it.value().cend(), true, depth + 1); it2->second.update_members(it.value().cbegin(), it.value().cend(), true, depth + 1);
#if JSON_DIAGNOSTICS
it2->second.set_parents();
#endif
continue; continue;
} }
} }
m_data.m_value.object->operator[](it.key()) = it.value(); // set_parent() also repairs the other members, which ordered_json
#if JSON_DIAGNOSTICS // relocates when adding a key makes its vector grow
m_data.m_value.object->operator[](it.key()).m_parent = this; set_parent(m_data.m_value.object->operator[](it.key()) = it.value());
#endif
} }
} }
@@ -3999,9 +4021,6 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
} }
// a nested object is merged: continue with its parent // a nested object is merged: continue with its parent
#if JSON_DIAGNOSTICS
target->set_parents();
#endif
target = stack.back().target; target = stack.back().target;
first = stack.back().position; first = stack.back().position;
last = stack.back().last; last = stack.back().last;
@@ -4023,10 +4042,9 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
continue; continue;
} }
} }
target->m_data.m_value.object->operator[](first.key()) = first.value(); // set_parent() also repairs the other members, which ordered_json
#if JSON_DIAGNOSTICS // relocates when adding a key makes its vector grow
target->m_data.m_value.object->operator[](first.key()).m_parent = target; target->set_parent(target->m_data.m_value.object->operator[](first.key()) = first.value());
#endif
++first; ++first;
} }
} }
+33 -15
View File
@@ -25701,6 +25701,27 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
} }
/// @brief restore the parent pointers after erasing from an object
/// ordered_json keeps its members in a vector, and erasing a member
/// re-constructs every member after it in place, which resets their
/// parent pointers
void set_parents_after_object_erase()
{
#if JSON_DIAGNOSTICS
#ifdef JSON_HEDLEY_MSVC_VERSION
#pragma warning(push )
#pragma warning(disable : 4127) // ignore warning to replace if with if constexpr
#endif
if (detail::is_ordered_map<object_t>::value)
{
set_parents();
}
#ifdef JSON_HEDLEY_MSVC_VERSION
#pragma warning( pop )
#endif
#endif
}
public: public:
////////////////////////// //////////////////////////
// JSON parser callback // // JSON parser callback //
@@ -27389,6 +27410,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
case value_t::object: case value_t::object:
{ {
result.m_it.object_iterator = erase_from_object(pos.m_it.object_iterator); result.m_it.object_iterator = erase_from_object(pos.m_it.object_iterator);
set_parents_after_object_erase();
break; break;
} }
@@ -27461,6 +27483,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
{ {
result.m_it.object_iterator = m_data.m_value.object->erase(first.m_it.object_iterator, result.m_it.object_iterator = m_data.m_value.object->erase(first.m_it.object_iterator,
last.m_it.object_iterator); last.m_it.object_iterator);
set_parents_after_object_erase();
break; break;
} }
@@ -27491,7 +27514,9 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
JSON_THROW(type_error::create(307, detail::concat("cannot use erase() with ", type_name()), this)); JSON_THROW(type_error::create(307, detail::concat("cannot use erase() with ", type_name()), this));
} }
return m_data.m_value.object->erase(std::forward<KeyType>(key)); const auto erased = m_data.m_value.object->erase(std::forward<KeyType>(key));
set_parents_after_object_erase();
return erased;
} }
template < typename KeyType, detail::enable_if_t < template < typename KeyType, detail::enable_if_t <
@@ -27508,6 +27533,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
if (it != m_data.m_value.object->end()) if (it != m_data.m_value.object->end())
{ {
m_data.m_value.object->erase(it); m_data.m_value.object->erase(it);
set_parents_after_object_erase();
return 1; return 1;
} }
return 0; return 0;
@@ -28419,16 +28445,12 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
if (it2 != m_data.m_value.object->end() && it2->second.is_object()) if (it2 != m_data.m_value.object->end() && it2->second.is_object())
{ {
it2->second.update_members(it.value().cbegin(), it.value().cend(), true, depth + 1); it2->second.update_members(it.value().cbegin(), it.value().cend(), true, depth + 1);
#if JSON_DIAGNOSTICS
it2->second.set_parents();
#endif
continue; continue;
} }
} }
m_data.m_value.object->operator[](it.key()) = it.value(); // set_parent() also repairs the other members, which ordered_json
#if JSON_DIAGNOSTICS // relocates when adding a key makes its vector grow
m_data.m_value.object->operator[](it.key()).m_parent = this; set_parent(m_data.m_value.object->operator[](it.key()) = it.value());
#endif
} }
} }
@@ -28457,9 +28479,6 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
} }
// a nested object is merged: continue with its parent // a nested object is merged: continue with its parent
#if JSON_DIAGNOSTICS
target->set_parents();
#endif
target = stack.back().target; target = stack.back().target;
first = stack.back().position; first = stack.back().position;
last = stack.back().last; last = stack.back().last;
@@ -28481,10 +28500,9 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
continue; continue;
} }
} }
target->m_data.m_value.object->operator[](first.key()) = first.value(); // set_parent() also repairs the other members, which ordered_json
#if JSON_DIAGNOSTICS // relocates when adding a key makes its vector grow
target->m_data.m_value.object->operator[](first.key()).m_parent = target; target->set_parent(target->m_data.m_value.object->operator[](first.key()) = first.value());
#endif
++first; ++first;
} }
} }
+23
View File
@@ -79,3 +79,26 @@ the same `fuzzers` target as above and also relies on the `FUZZER_ENGINE` variab
[build script](https://github.com/google/oss-fuzz/blob/master/projects/json/build.sh) for more information. [build script](https://github.com/google/oss-fuzz/blob/master/projects/json/build.sh) for more information.
In case the build at OSS-Fuzz fails, an issue will be created automatically. In case the build at OSS-Fuzz fails, an issue will be created automatically.
### Handling OSS-Fuzz reports
OSS-Fuzz files the crashes it finds in its own [issue tracker](https://issues.oss-fuzz.com), not on GitHub. So that
each report can be traced to the change that fixed it, and each fix to the report it answers, fixes follow these
conventions:
- **Reference the OSS-Fuzz issue in the pull request**, next to any GitHub issue it closes, as `OSS-Fuzz: <id>` (for
example, `OSS-Fuzz: 563659413`), and in the commit message. The ID alone does not disclose the crash. If the report
was triaged into a GitHub issue, link the OSS-Fuzz issue there too.
- **Turn the reproducer into a unit test.** Download the testcase from the OSS-Fuzz report, reduce it if possible, and
add it as a regression test to the unit test of the affected format (e.g., `tests/src/unit-bjdata.cpp`), with a
comment naming the OSS-Fuzz issue. This way the input is checked by every CI run rather than only by OSS-Fuzz, and
it stays covered even if OSS-Fuzz later closes the report as not reproducible.
- **Keep the fuzzer drivers and the unit tests in sync.** The round-trip checks of the UBJSON and BJData drivers are
also run on a fixed corpus in the unit tests (see `tests/src/round_trip_corpus.hpp` and the "round-trip invariants"
test cases), so a regression shows up in CI first. When a driver's checks change, change the unit tests with them.
- **Record in the report whether the bug shipped.** OSS-Fuzz asks whether a crash was a short-lived regression or
affects a released version; answer it when the fix is merged, as it decides whether the fix needs a release note or
a security advisory (see the [security policy](../.github/SECURITY.md)).
After the fix is merged, OSS-Fuzz re-runs the reproducer on its next build and marks the report as verified and
closed. If it does not, the fix is incomplete.
+3
View File
@@ -42,6 +42,9 @@ dump() serializes any non-finite double the same deterministic way (as JSON
`null`, since JSON itself cannot represent NaN/Infinity), so comparing `null`, since JSON itself cannot represent NaN/Infinity), so comparing
dumps is stable under exactly the same values that break operator==. dumps is stable under exactly the same values that break operator==.
The unit tests run the same checks on a fixed corpus (see the "BJData round-trip
invariants" test case), so keep both in sync.
The provided function `LLVMFuzzerTestOneInput` can be used in different fuzzer The provided function `LLVMFuzzerTestOneInput` can be used in different fuzzer
drivers. drivers.
*/ */
+3
View File
@@ -21,6 +21,9 @@ array data, it performs the following steps:
- j4 = from_ubjson(vec3) - j4 = from_ubjson(vec3)
- assert(j1 == j4) - assert(j1 == j4)
The unit tests run the same checks on a fixed corpus (see the "UBJSON round-trip
invariants" test case), so keep both in sync.
The provided function `LLVMFuzzerTestOneInput` can be used in different fuzzer The provided function `LLVMFuzzerTestOneInput` can be used in different fuzzer
drivers. drivers.
*/ */
+213
View File
@@ -0,0 +1,213 @@
// __ _____ _____ _____
// __| | __| | | | JSON for Modern C++ (supporting code)
// | | |__ | | | | | | version 3.12.0
// |_____|_____|_____|_|___| https://github.com/nlohmann/json
//
// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
// SPDX-License-Identifier: MIT
#pragma once
#include <cmath> // nan
#include <cstddef> // size_t
#include <cstdint> // int32_t, int64_t, uint32_t, uint64_t
#include <limits> // numeric_limits
#include <random> // mt19937
#include <string> // string, to_string
#include <utility> // move
#include <vector> // vector
#include <nlohmann/json.hpp>
// Values for the round-trip property tests of the UBJSON and BJData writers.
//
// The fuzzer drivers (tests/src/fuzzer-parse_ubjson.cpp and
// fuzzer-parse_bjdata.cpp) check that anything the library parses can be
// serialized, parsed back, and serialized again without loss. Those checks
// only run at OSS-Fuzz, so a regression used to surface days later as an
// external report. The unit tests run the same checks on this corpus in CI.
//
// The corpus is deterministic: std::mt19937's output sequence is fixed by
// the standard, and it is used directly rather than through a distribution
// (whose results are implementation-defined).
namespace utils
{
class round_trip_corpus
{
public:
using json = nlohmann::json;
static std::vector<json> values()
{
round_trip_corpus corpus;
return corpus.build();
}
// whether a value contains a binary value, which a BJData or UBJSON round
// trip may turn into an array of integers
static bool contains_binary(const json& j)
{
if (j.is_binary())
{
return true;
}
if (j.is_structured())
{
for (const auto& element : j)
{
if (contains_binary(element))
{
return true;
}
}
}
return false;
}
private:
std::vector<json> atoms;
// a fixed seed is the point: the corpus must be the same in every run
std::mt19937 generator{42}; // NOLINT(cert-msc32-c,cert-msc51-cpp,bugprone-random-generator-seed)
round_trip_corpus()
: atoms
{
nullptr, true, false,
// integers at the boundaries of every UBJSON/BJData integer type
0, 1, -1, 127, 128, 255, 256, -128, -129,
32767, 32768, 65535, 65536, -32768, -32769,
(std::numeric_limits<std::int32_t>::min)(), (std::numeric_limits<std::int32_t>::max)(),
(std::numeric_limits<std::uint32_t>::max)(),
(std::numeric_limits<std::int64_t>::min)(), (std::numeric_limits<std::int64_t>::max)(),
static_cast<std::uint64_t>((std::numeric_limits<std::int64_t>::max)()) + 1u,
(std::numeric_limits<std::uint64_t>::max)(),
// floating-point numbers, including non-finite ones
0.0, -0.0, 1.5, -2.25, 3.4e38, (std::numeric_limits<double>::max)(),
std::nan(""), std::numeric_limits<double>::infinity(), -std::numeric_limits<double>::infinity(),
// strings, including a non-ASCII one and one longer than 255 bytes
"", "a", "\xC3\xA4", std::string(300, 'x'),
// binary values with and without subtype
json::binary({}), json::binary({1, 2, 255}), json::binary({0x80, 0x7F}, 42), json::binary({1}, 0)
}
{}
std::vector<json> build()
{
std::vector<json> result = atoms;
// each atom inside containers, including homogeneous ones that the
// writers encode as optimized (typed) containers
result.emplace_back(json::array());
result.emplace_back(json::object());
for (const auto& atom : atoms)
{
result.push_back(json::array({atom}));
result.push_back(json::array({atom, atom, atom}));
result.push_back(json::array({json::array({atom})}));
result.push_back(json::object({{"key", atom}}));
}
result.push_back(json::array({1, 1.5}));
result.push_back(json::array({-1, 255}));
result.push_back(json::array({"a", "b"}));
// deep, but well below any recursion or depth limit
json nested_array = 1;
json nested_object = 1;
for (int i = 0; i < 300; ++i)
{
nested_array = json::array({nested_array});
nested_object = json::object({{"key", nested_object}});
}
result.push_back(nested_array);
result.push_back(nested_object);
add_annotated_arrays(result);
add_random_values(result);
return result;
}
// objects in the JData annotated array format, which the BJData writer
// encodes as ND-arrays when the annotation describes a packed array, and
// as plain objects otherwise (see #5398, #5399, #5403, #5404, and #5542)
static void add_annotated_arrays(std::vector<json>& result)
{
const std::vector<json> types =
{
"uint8", "int8", "uint16", "int16", "uint32", "int32", "uint64", "int64",
"single", "double", "char", "byte", "bool", "unknown", 5, nullptr
};
const std::vector<json> sizes =
{
json::array(), {3}, {1, 3}, {3, 1}, {2, 3}, {2, 0}, {0, 2}, {2, 2, 2}, {-1, 2}, {2, 1.5},
"3", 3, nullptr, json::binary({})
};
const std::vector<json> data =
{
nullptr, 5, "s", json::object({{"a", 1}}), json::array(),
{1, 2, 3}, {1, 2, 3, 4, 5, 6}, {1, 2, 3, 4, 5, 6, 7, 8},
{1.5, 2.5, 3.5, 4.5, 5.5, 6.5}, {300, -300, 70000, -70000, 1, 2},
{"a", "b", "c", "d", "e", "f"}, {json::array({1, 2, 3}), json::array({4, 5, 6})}
};
for (const auto& type : types)
{
for (const auto& size : sizes)
{
for (const auto& d : data)
{
result.push_back({{"_ArrayType_", type}, {"_ArraySize_", size}, {"_ArrayData_", d}});
}
}
}
// incomplete annotations and annotations with an extra key
result.push_back({{"_ArraySize_", {2, 3}}, {"_ArrayData_", {1, 2, 3, 4, 5, 6}}});
result.push_back({{"_ArrayType_", "uint8"}, {"_ArrayData_", {1, 2, 3, 4, 5, 6}}});
result.push_back({{"_ArrayType_", "uint8"}, {"_ArraySize_", {2, 3}}});
result.push_back({{"_ArrayType_", "uint8"}, {"_ArraySize_", {2, 3}}, {"_ArrayData_", {1, 2, 3, 4, 5, 6}}, {"extra", 1}});
}
// random containers of atoms, both homogeneous and mixed
void add_random_values(std::vector<json>& result)
{
for (int i = 0; i < 1000; ++i)
{
result.push_back(random_value(0));
}
}
std::size_t random_below(std::size_t bound)
{
return generator() % bound;
}
json random_value(int depth)
{
const auto kind = random_below(10);
if (depth > 3 || kind < 5)
{
return atoms[random_below(atoms.size())];
}
json result = kind < 8 ? json::array() : json::object();
const auto count = random_below(5);
const bool homogeneous = random_below(2) == 0;
const json fixed = atoms[random_below(atoms.size())];
for (std::size_t i = 0; i < count; ++i)
{
json element = homogeneous ? fixed : random_value(depth + 1);
if (result.is_array())
{
result.push_back(std::move(element));
}
else
{
result[std::to_string(i)] = std::move(element);
}
}
return result;
}
};
} // namespace utils
+103
View File
@@ -19,6 +19,7 @@ using nlohmann::json;
#include <fstream> #include <fstream>
#include <set> #include <set>
#include "make_test_data_available.hpp" #include "make_test_data_available.hpp"
#include "round_trip_corpus.hpp"
#include "test_utils.hpp" #include "test_utils.hpp"
namespace namespace
@@ -2867,6 +2868,21 @@ TEST_CASE("BJData")
const auto out_num = json::to_bjdata(j_num); const auto out_num = json::to_bjdata(j_num);
CHECK(out_num.at(0) == '{'); CHECK(out_num.at(0) == '{');
CHECK(json::from_bjdata(out_num) == j_num); CHECK(json::from_bjdata(out_num) == j_num);
// OSS-Fuzz issue 474400817: an empty object _ArraySize_ was
// written as the ND-array header length, which from_bjdata()
// could not read back
const std::vector<uint8_t> input =
{
'[', '{', 'U', 11, '_', 'A', 'r', 'r', 'a', 'y', 'D', 'a', 't', 'a', '_', 'Z',
'U', 11, '_', 'A', 'r', 'r', 'a', 'y', 'T', 'y', 'p', 'e', '_', 'S', 'i', 5, 'i', 'n', 't', '1', '6',
'U', 11, '_', 'A', 'r', 'r', 'a', 'y', 'S', 'i', 'z', 'e', '_', '{', '}', '}', ']'
};
const json j1 = json::from_bjdata(input);
CHECK(j1 == json::parse(R"([{"_ArrayType_":"int16","_ArraySize_":{},"_ArrayData_":null}])"));
json j2;
CHECK_NOTHROW(j2 = json::from_bjdata(json::to_bjdata(j1, false, false)));
CHECK(j2 == j1);
} }
SECTION("ndarray with out-of-range _ArrayData_ elements stays as object") SECTION("ndarray with out-of-range _ArrayData_ elements stays as object")
@@ -4273,6 +4289,93 @@ TEST_CASE("BJData use_type requires use_size")
} }
} }
TEST_CASE("BJData round-trip invariants")
{
// This checks what the parse_bjdata_fuzzer driver checks (see
// tests/src/fuzzer-parse_bjdata.cpp), so that a regression shows up in CI
// rather than as an OSS-Fuzz report: every value from_bjdata() returns
// (j1) can be serialized with any combination of options, the result can
// be parsed back (j2), and serializing j2 again with the same options
// yields a value-equal result.
//
// Beyond the driver, this also checks that j2 equals j1 and that
// serializing j2 reproduces the exact bytes, both except for values that
// contain a binary value: a binary value is only written as a binary
// value with Draft 3's optimized binary array, and otherwise read back as
// an array of integers, for which the writer may choose different (but
// equally valid) type markers when it is serialized again (see #5494).
//
// Values are compared with dump() rather than operator==, because a NaN
// never compares equal to itself.
struct options
{
bool use_size;
bool use_type;
json::bjdata_version_t version;
};
const std::vector<options> all_options =
{
{false, false, json::bjdata_version_t::draft2},
{true, false, json::bjdata_version_t::draft2},
{true, true, json::bjdata_version_t::draft2},
{false, false, json::bjdata_version_t::draft3},
{true, false, json::bjdata_version_t::draft3},
{true, true, json::bjdata_version_t::draft3},
};
for (const auto& j0 : utils::round_trip_corpus::values())
{
// turn the corpus value into a value as from_bjdata() returns it
for (const auto& initial : all_options)
{
const json j1 = json::from_bjdata(json::to_bjdata(j0, initial.use_size, initial.use_type, initial.version));
const bool has_binary = utils::round_trip_corpus::contains_binary(j1);
for (const auto& o : all_options)
{
INFO("j1 = " << j1.dump() << ", use_size = " << o.use_size << ", use_type = " << o.use_type
<< ", draft3 = " << (o.version == json::bjdata_version_t::draft3));
const std::vector<std::uint8_t> vec = json::to_bjdata(j1, o.use_size, o.use_type, o.version);
json j2;
// anything the library writes must be parsable by the library
REQUIRE_NOTHROW(j2 = json::from_bjdata(vec));
const std::vector<std::uint8_t> vec2 = json::to_bjdata(j2, o.use_size, o.use_type, o.version);
CHECK(json::from_bjdata(vec2).dump() == j2.dump());
if (!has_binary)
{
CHECK(j2.dump() == j1.dump());
CHECK(vec2 == vec);
}
}
}
}
}
TEST_CASE("BJData round trip of a binary value is value-stable, not byte-stable")
{
// OSS-Fuzz issue 474480402: a Draft 3 optimized binary array is read as a
// binary value, which to_bjdata() writes in the default Draft 2 mode as a
// plain array of uint8 numbers. That is read back as an array of numbers,
// for which the writer then picks the smallest type marker, int8 ('i'),
// so re-serializing changes the bytes, but not the value. This is the
// exception described in the "Round trips" note of the BJData
// documentation, and why the fuzzer checks value stability (see #5494).
const std::vector<uint8_t> input = {'[', '$', 'B', '#', 'U', 1, 0x20};
const json j1 = json::from_bjdata(input);
CHECK(j1 == json::binary({0x20}));
const std::vector<uint8_t> vec = json::to_bjdata(j1, false, false);
CHECK(vec == std::vector<uint8_t>({'[', 'U', 0x20, ']'}));
const json j2 = json::from_bjdata(vec);
CHECK(j2 == json::array({0x20}));
const std::vector<uint8_t> vec2 = json::to_bjdata(j2, false, false);
CHECK(vec2 == std::vector<uint8_t>({'[', 'i', 0x20, ']'}));
CHECK(json::from_bjdata(vec2) == j2);
}
TEST_CASE("BJData roundtrips" * doctest::skip()) TEST_CASE("BJData roundtrips" * doctest::skip())
{ {
SECTION("input from self-generated BJData files") SECTION("input from self-generated BJData files")
+100
View File
@@ -361,6 +361,106 @@ TEST_CASE("Regression tests for extended diagnostics")
CHECK(p == o); CHECK(p == o);
} }
} }
SECTION("Regression test - erase() and update() must keep JSON_DIAGNOSTICS parent pointers of ordered_json members")
{
// ordered_json keeps its members in a vector: erasing a member
// re-constructs all members after it in place, and adding a key may
// reallocate the vector; both reset the parent pointers of the members
// that were moved
using nlohmann::ordered_json;
const auto check_parents = [](const ordered_json & j)
{
// const access, so operator[] cannot repair the parent pointers
CHECK_THROWS_WITH_AS(j["z"]["x"].at(0), "[json.exception.type_error.304] (/z/x) cannot use at() with number", ordered_json::type_error);
// must not trigger assert_invariant() in a debug/assert-enabled build
ordered_json const copy = j; // NOLINT(performance-unnecessary-copy-initialization)
CHECK(copy == j);
};
// erase(key)
{
ordered_json j = {{"a", 1}, {"z", {{"x", 1}}}};
CHECK(j.erase("a") == 1);
check_parents(j);
}
// erase(iterator)
{
ordered_json j = {{"a", 1}, {"z", {{"x", 1}}}};
j.erase(j.begin());
check_parents(j);
}
// erase(iterator, iterator)
{
ordered_json j = {{"a", 1}, {"b", 2}, {"z", {{"x", 1}}}};
j.erase(j.begin(), j.find("z"));
check_parents(j);
}
// patch() removes via erase(iterator)
{
ordered_json j = {{"a", 1}, {"z", {{"x", 1}}}};
j.patch_inplace(ordered_json::parse(R"([{"op": "remove", "path": "/a"}])"));
check_parents(j);
}
// update(j)
{
ordered_json j = {{"z", {{"x", 1}}}};
j.update({{"a", 1}, {"b", 2}});
check_parents(j);
}
// update(j, true), the outer and the nested vector both grow
{
ordered_json j = {{"z", {{"x", 1}}}};
j.update({{"z", {{"y", 2}}}, {"a", 1}}, true);
check_parents(j);
}
// update(j, true) around its descent bound, where the nested vectors
// grow while the objects are merged without recursing
for (const std::size_t depth :
{
nlohmann::detail::recursion_depth_limit() - 1, nlohmann::detail::recursion_depth_limit(), nlohmann::detail::recursion_depth_limit() + 2
})
{
ordered_json j = {{"z", {{"x", 1}}}};
ordered_json patch = {{"a", 1}, {"b", 2}, {"c", {{"d", 3}}}};
for (std::size_t i = 0; i < depth; ++i)
{
j = ordered_json{{"k", 0}, {"n", std::move(j)}};
patch = ordered_json{{"n", std::move(patch)}, {"l", 1}, {"m", 2}};
}
j.update(patch, true);
// must not trigger assert_invariant() on any level in a
// debug/assert-enabled build
ordered_json const copy = j; // NOLINT(performance-unnecessary-copy-initialization)
CHECK(copy == j);
}
// merge_patch() inserts "c" and removes "d" at /a/c, then inserts "e"
// at /a, which copies /a/c
{
auto j = ordered_json::parse(R"({"a": {"c": {"d": {}}}})");
j.merge_patch(ordered_json::parse(R"({"a": {"c": {"c": "s", "d": null}, "e": "s"}})"));
CHECK(j.dump() == R"({"a":{"c":{"c":"s"},"e":"s"}})");
auto const& constJ = j;
#if JSON_DIAGNOSTIC_POSITIONS
CHECK_THROWS_WITH_AS(constJ["a"]["c"]["c"].at(0), "[json.exception.type_error.304] (/a/c/c) (bytes 18-21) cannot use at() with string", ordered_json::type_error);
#else
CHECK_THROWS_WITH_AS(constJ["a"]["c"]["c"].at(0), "[json.exception.type_error.304] (/a/c/c) cannot use at() with string", ordered_json::type_error);
#endif
ordered_json const copy = j;
CHECK(copy == j);
}
}
} }
TEST_CASE("Better diagnostics past the descent bound of update() and merge_patch()") TEST_CASE("Better diagnostics past the descent bound of update() and merge_patch()")
+49 -1
View File
@@ -15,6 +15,7 @@ using nlohmann::json;
#include <fstream> #include <fstream>
#include <set> #include <set>
#include "make_test_data_available.hpp" #include "make_test_data_available.hpp"
#include "round_trip_corpus.hpp"
#include "test_utils.hpp" #include "test_utils.hpp"
namespace namespace
@@ -2265,7 +2266,9 @@ TEST_CASE("UBJSON optimized arrays of a valueless type are bounded")
SECTION("an excessive count is rejected") SECTION("an excessive count is rejected")
{ {
// 'l' is a big-endian int32: 0x7FFFFFFF elements, about 34 GB of value // 'l' is a big-endian int32: 0x7FFFFFFF elements, about 34 GB of value;
// OSS-Fuzz reported this shape as a parse_ubjson_fuzzer timeout
// (testcase 6347769435193344, no issue filed)
for (const auto marker : for (const auto marker :
{'Z', 'T', 'F' {'Z', 'T', 'F'
}) })
@@ -2817,6 +2820,51 @@ TEST_CASE("UBJSON use_type requires use_size")
} }
} }
TEST_CASE("UBJSON round-trip invariants")
{
// This checks what the parse_ubjson_fuzzer driver checks (see
// tests/src/fuzzer-parse_ubjson.cpp), so that a regression shows up in CI
// rather than as an OSS-Fuzz report: every value from_ubjson() returns
// (j1) can be serialized with any combination of options, the result can
// be parsed back (j2), and serializing j2 again with the same options
// reproduces the exact bytes. Beyond the driver, this also checks that j2
// equals j1. Values are compared with dump() rather than operator==,
// because a NaN never compares equal to itself.
struct options
{
bool use_size;
bool use_type;
};
const std::vector<options> all_options =
{
{false, false},
{true, false},
{true, true},
};
for (const auto& j0 : utils::round_trip_corpus::values())
{
// turn the corpus value into a value as from_ubjson() returns it; this
// has no binary values, as UBJSON writes them as arrays of integers
for (const auto& initial : all_options)
{
const json j1 = json::from_ubjson(json::to_ubjson(j0, initial.use_size, initial.use_type));
for (const auto& o : all_options)
{
INFO("j1 = " << j1.dump() << ", use_size = " << o.use_size << ", use_type = " << o.use_type);
const std::vector<std::uint8_t> vec = json::to_ubjson(j1, o.use_size, o.use_type);
json j2;
// anything the library writes must be parsable by the library
REQUIRE_NOTHROW(j2 = json::from_ubjson(vec));
CHECK(j2.dump() == j1.dump());
CHECK(json::to_ubjson(j2, o.use_size, o.use_type) == vec);
}
}
}
}
TEST_CASE("UBJSON roundtrips" * doctest::skip()) TEST_CASE("UBJSON roundtrips" * doctest::skip())
{ {
SECTION("input from self-generated UBJSON files") SECTION("input from self-generated UBJSON files")