Compare commits

..
Author SHA1 Message Date
Niels Lohmann aab1fa9e53 Copy the text of float tokens only when a SAX handler asks for it
After the digits are converted while scanning, copying the token text into
token_buffer (an out-of-line std::string::append) was about 10% of parsing
canada.json. The only reader is the string argument of
json_sax::number_float(), and the library's own handlers
(json_sax_dom_parser, json_sax_dom_callback_parser, json_sax_acceptor)
ignore it.

For contiguous input, the fast path now keeps a pointer to the token in the
input, and get_string() copies it on first use. The parser passes an empty
string to the handlers marked by the new internal trait
sax_ignores_number_text instead of calling get_string(), so json::parse()
and json::accept() never copy float text. User handlers, and
JSON_DIAGNOSTIC_POSITIONS (which reads get_string().size()), still see the
exact text. Keys and strings, which never have pending text, use the
unchecked get_string_token(). Input read through a window into a stream
(std::istream, FILE*) copies the text right away, because the next read may
refill the window.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-17 08:29:50 +02:00
Niels Lohmann 722b2c4334 Skip the digit accumulation checks without exceptions
The section compares error messages from both lexer paths and parses
out-of-range numbers, so it does not compile (the caught exception does not
exist) and would abort with JSON_NOEXCEPTION.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-17 08:29:49 +02:00
Niels Lohmann 81ca04f3ec Accumulate number digits while scanning them
The contiguous number fast path validated a token byte by byte, copied it
into token_buffer, and then converted it in out-of-line functions that
scanned the digits a second time: parse_float_native() called
parse_float_significand(), which returned its result through memory and
called std::any_of (out of line with GCC) twice per number, and
parse_integer_signed() divided by ten on every digit for its overflow check.

scan_number_bulk_contiguous() now accumulates the significand while it
validates the digits, eight at a time where at least eight bytes remain
(is_eight_digits() and parse_eight_digits(), after fast_float). Floats with
at most 19 digits go straight to decimal_to_float(); integers with at most
18 digits are stored without an overflow check (a narrower number type
still rejects values that do not round-trip). Longer tokens take the
existing path. has_nonzero_digit() is a plain loop and
parse_float_significand() is always inlined, which helps that path.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-17 08:29:48 +02:00
Niels Lohmann f25c6543aa Skip the exception-only checks of the stream input tests without exceptions
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-17 08:29:47 +02:00
Niels Lohmann c7a9780936 Add ABI tags for JSON_DISABLE_STREAM_BULK_SCAN and JSON_DISABLE_UNLOCKED_FILE_IO
Both macros select between two definitions of an input adapter, so
translation units compiled with different settings violated the ODR when
linked together. Like the other ABI-affecting macros, they now append a
tag to the inline namespace: _dsbs and _dufio. Their defaults move to
abi_macros.hpp so that json_fwd.hpp sees them, and the natvis file is
regenerated for the new namespaces.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-17 08:29:46 +02:00
Niels Lohmann 1d2526b1d8 Make parsing from std::istream and FILE* fast
Squashed onto develop cd6b428ef (CI fixes from #5860). Original commits:

- Let the lexer's bulk fast paths run on a window of the input
- Scan the get area of an input stream in bulk
- Lock a FILE once per parse instead of once per character
- Add benchmarks for parsing from a FILE* and for many values from a stream
- Avoid implicit unsigned char to char conversion in stream tests
- Make file_input_adapter::unlock() safe to call on a moved-from adapter

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-17 08:29:45 +02:00
Niels Lohmann 0e95fe8200 Split the slowest Valgrind tests out of unit-large_json and unit-bjdata (#5862)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-17 08:28:28 +02:00
Niels Lohmann cd6b428ef0 Fix CI failures on develop from the API performance stack (#5860)
Fixes every CI failure develop has had since the API performance stack (#5831-#5842) was merged: compiler warnings and errors (Clang, GCC 4.9/11, MSVC 2015/2017/2022), clang-tidy, cpplint, Cppcheck, IWYU, the examples check, the no-exceptions, diagnostic-positions, implicit-conversions and strict-NUL test configurations, and the MinGW link (now with LLD).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-16 19:16:21 +02:00
53 changed files with 114790 additions and 2065 deletions

No files matched your search

+10 -8
View File
@@ -166,19 +166,21 @@ jobs:
# link, but the binaries clang 11.0.1 and clang 18.1.8 then produce crash # link, but the binaries clang 11.0.1 and clang 18.1.8 then produce crash
# before doctest prints its first line - 39 of 102 tests on clang 18. # before doctest prints its first line - 39 of 102 tests on clang 18.
# Keep the objects small by splitting the test files instead. # Keep the objects small by splitting the test files instead.
# Objects with more than 65535 sections fail to link with "relocation # The tests link with LLD: GNU ld fails on objects with more than 32767
# truncated to fit: IMAGE_REL_AMD64_REL32 against `.rdata'": the MinGW # sections with "relocation truncated to fit: IMAGE_REL_AMD64_REL32
# linker stores the section an associative COMDAT section belongs to in # against `.rdata'", as it stores the section an associative COMDAT
# 16 bits (x_associated in binutils' include/coff/internal.h), so it # section belongs to in 16 bits (x_associated in binutils'
# discards the jump tables of inline functions together with the wrong # include/coff/internal.h) and so discards the jump tables of inline
# function. Each basic_json specialization a test file instantiates adds # functions together with the wrong function. Each basic_json
# many sections, so test a second one in a file of its own. # specialization and input type a test file instantiates adds many
# sections, so unit-msgpack.cpp and unit-regression2.cpp kept crossing
# that limit as the library grew.
- name: Run CMake - name: Run CMake
run: cmake -S . -B build ^ run: cmake -S . -B build ^
-DCMAKE_CXX_COMPILER="C:/Program Files/LLVM/bin/clang++.exe" ^ -DCMAKE_CXX_COMPILER="C:/Program Files/LLVM/bin/clang++.exe" ^
-DCMAKE_CXX_FLAGS="--target=x86_64-w64-mingw32 -stdlib=libstdc++ -pthread" ^ -DCMAKE_CXX_FLAGS="--target=x86_64-w64-mingw32 -stdlib=libstdc++ -pthread" ^
-DCMAKE_CXX_FLAGS_DEBUG="-g0" ^ -DCMAKE_CXX_FLAGS_DEBUG="-g0" ^
-DCMAKE_EXE_LINKER_FLAGS="-lwinpthread" ^ -DCMAKE_EXE_LINKER_FLAGS="-fuse-ld=lld -lwinpthread" ^
-G"MinGW Makefiles" ^ -G"MinGW Makefiles" ^
-DCMAKE_BUILD_TYPE=Debug ^ -DCMAKE_BUILD_TYPE=Debug ^
-DJSON_BuildTests=On -DJSON_BuildTests=On
-1
View File
@@ -36,7 +36,6 @@ cc_library(
"include/nlohmann/detail/input/parser.hpp", "include/nlohmann/detail/input/parser.hpp",
"include/nlohmann/detail/input/position_t.hpp", "include/nlohmann/detail/input/position_t.hpp",
"include/nlohmann/detail/input/pow5_table.hpp", "include/nlohmann/detail/input/pow5_table.hpp",
"include/nlohmann/detail/input/simd_scan.hpp",
"include/nlohmann/detail/input/string_scan.hpp", "include/nlohmann/detail/input/string_scan.hpp",
"include/nlohmann/detail/iterators/internal_iterator.hpp", "include/nlohmann/detail/iterators/internal_iterator.hpp",
"include/nlohmann/detail/iterators/iter_impl.hpp", "include/nlohmann/detail/iterators/iter_impl.hpp",
-1
View File
@@ -38,7 +38,6 @@ header. See also the [macro overview page](../../features/macros.md).
- [**JSON_HAS_THREE_WAY_COMPARISON**](json_has_three_way_comparison.md) - control 3-way comparison support - [**JSON_HAS_THREE_WAY_COMPARISON**](json_has_three_way_comparison.md) - control 3-way comparison support
- [**JSON_NO_AUTOMATIC_UDLS**](json_no_automatic_udls.md) - do not include the user-defined string literals (UDLs) automatically - [**JSON_NO_AUTOMATIC_UDLS**](json_no_automatic_udls.md) - do not include the user-defined string literals (UDLs) automatically
- [**JSON_NO_IO**](json_no_io.md) - switch off functions relying on certain C++ I/O headers - [**JSON_NO_IO**](json_no_io.md) - switch off functions relying on certain C++ I/O headers
- [**JSON_NO_SIMD**](json_no_simd.md) - use only portable code instead of vector instructions (SSE2, SSSE3, NEON)
- [**JSON_NO_THREAD_LOCAL**](json_no_thread_local.md) - switch off the use of `thread_local` storage - [**JSON_NO_THREAD_LOCAL**](json_no_thread_local.md) - switch off the use of `thread_local` storage
- [**JSON_SKIP_UNSUPPORTED_COMPILER_CHECK**](json_skip_unsupported_compiler_check.md) - do not warn about unsupported compilers - [**JSON_SKIP_UNSUPPORTED_COMPILER_CHECK**](json_skip_unsupported_compiler_check.md) - do not warn about unsupported compilers
- [**JSON_USE_GLOBAL_UDLS**](json_use_global_udls.md) - place user-defined string literals (UDLs) into the global namespace - [**JSON_USE_GLOBAL_UDLS**](json_use_global_udls.md) - place user-defined string literals (UDLs) into the global namespace
@@ -69,12 +69,11 @@ The default value is `0`: streams are scanned in bulk.
changes the characters in its get area in place, does not meet these requirements. Define the macro to `1` for changes the characters in its get area in place, does not meet these requirements. Define the macro to `1` for
such stream buffers. such stream buffers.
!!! warning "Define consistently" !!! note "ABI compatibility"
The macro selects between two definitions of the library's internal stream input adapter. It must therefore be The value of this macro is encoded in the [namespace](../../features/namespace.md) (tag `_dsbs`), resulting in
defined identically for **every** translation unit that includes the library; mixing translation units that define distinct symbol names. Translation units compiled with and without it can therefore be linked into the same program
it with ones that do not is an ODR violation. Prefer setting it as a compile definition on the target rather than without One Definition Rule (ODR) violations, but they cannot exchange instances of library types.
with `#!cpp #define` in individual source files.
!!! note "Other inputs" !!! note "Other inputs"
@@ -49,12 +49,11 @@ The default value is `0`: the file is locked once and read with the unlocked std
A program in which a SAX handler or parser callback waits for **another** thread that uses the **same** `FILE` A program in which a SAX handler or parser callback waits for **another** thread that uses the **same** `FILE`
object would deadlock. Define the macro to `1` for such programs. object would deadlock. Define the macro to `1` for such programs.
!!! warning "Define consistently" !!! note "ABI compatibility"
The macro selects between two definitions of the library's internal `FILE*` input adapter. It must therefore be The value of this macro is encoded in the [namespace](../../features/namespace.md) (tag `_dufio`), resulting in
defined identically for **every** translation unit that includes the library; mixing translation units that define distinct symbol names. Translation units compiled with and without it can therefore be linked into the same program
it with ones that do not is an ODR violation (and can leave a file locked). Prefer setting it as a compile without One Definition Rule (ODR) violations, but they cannot exchange instances of library types.
definition on the target rather than with `#!cpp #define` in individual source files.
!!! note "Other inputs" !!! note "Other inputs"
@@ -1,83 +0,0 @@
# JSON_NO_SIMD
```cpp
#define JSON_NO_SIMD
```
When defined, the library uses only portable C++ code where it would otherwise use vector instructions: when scanning
strings during parsing, and when writing floating-point numbers during serialization.
The macro is an escape hatch, for example for toolchains that cannot compile the vector code, for sanitizers or
analyzers that do not understand it, or to compare the speed of both variants. It does not change any result: values,
exceptions, and error positions are the same either way. Only the speed differs.
## Default definition
By default, `#!cpp JSON_NO_SIMD` is not defined, and vector instructions are used where the target has them.
```cpp
#undef JSON_NO_SIMD
```
## Notes
!!! note "Which instructions are used"
- **x86-64:** SSE2, which every x86-64 CPU has, finds the end of runs of plain string characters 16 bytes at a time
and writes the digits of floating-point numbers. The UTF-8 check of non-ASCII text needs SSSE3, which is not part
of x86-64: it is compiled for SSSE3 with a function attribute and used when the CPU has SSSE3 (every x86-64 CPU
since about 2011), else the portable check. Code compiled for SSSE3 anyway (e.g., with `-mssse3` or `-march=native`)
skips this check of the CPU. 32-bit x86 uses SSE2 for strings when the compiler targets it.
- **AArch64:** NEON, which every 64-bit Arm CPU has, checks UTF-8 text and long runs of plain string characters 16
bytes at a time, and writes the digits of floating-point numbers. With MSVC, only the floating-point digits use
NEON.
- **Other targets:** portable code only.
Strings are only scanned this way for contiguous input and for the characters a `#!cpp std::istream` holds in its
stream buffer (see [`JSON_DISABLE_STREAM_BULK_SCAN`](json_disable_stream_bulk_scan.md)); other inputs are read one
character at a time.
!!! note "Interaction with `JSON_USE_SIMDUTF`"
With [`JSON_USE_SIMDUTF`](json_use_simdutf.md), runs of string characters are first validated by simdutf; the code
described here is used for what simdutf does not accept, so that errors are reported at the same position.
`JSON_NO_SIMD` does not affect simdutf.
!!! warning "Define consistently"
The macro selects between two definitions of the library's internal scanning and formatting functions. It must
therefore be defined identically for **every** translation unit that includes the library; mixing translation units
that define it with ones that do not is an ODR violation. Prefer setting it as a compile definition on the target
rather than with `#!cpp #define` in individual source files.
## Examples
??? example "Example: switching off vector instructions"
The program parses and serializes the same values with and without the macro:
```cpp
#define JSON_NO_SIMD
#include <iostream>
#include <nlohmann/json.hpp>
using json = nlohmann::json;
int main()
{
const json j = json::parse(R"({"name": "日本語 text", "pi": 3.141592653589793})");
std::cout << j.dump() << '\n'; // {"name":"日本語 text","pi":3.141592653589793}
}
```
## See also
- [**parse**](../basic_json/parse.md) - deserialize from a compatible input
- [**dump**](../basic_json/dump.md) - serialization
- [**JSON_USE_SIMDUTF**](json_use_simdutf.md) - use the simdutf library to accelerate UTF-8 validation
- [**JSON_DISABLE_STREAM_BULK_SCAN**](json_disable_stream_bulk_scan.md) - read a `std::istream` one character at a time
- [Parsing: input sources and performance](../../features/parsing/index.md#input-sources-and-performance)
## Version history
- Added in version 3.13.0.
@@ -6,9 +6,8 @@
When defined, the parser validates the UTF-8 content of JSON strings that come from a **contiguous byte input** When defined, the parser validates the UTF-8 content of JSON strings that come from a **contiguous byte input**
(`std::string`, `std::vector<char>`/`<std::uint8_t>`, string literals, `const char*` ranges, …) using the (`std::string`, `std::vector<char>`/`<std::uint8_t>`, string literals, `const char*` ranges, …) using the
[simdutf](https://github.com/simdutf/simdutf) library instead of the built-in validator (which uses SSSE3 or NEON where [simdutf](https://github.com/simdutf/simdutf) library instead of the built-in scalar validator. On text with many
available, see [`JSON_NO_SIMD`](json_no_simd.md)). On text with many non-ASCII characters (e.g. CJK or emoji) this can non-ASCII characters (e.g. CJK or emoji) this can validate several times faster.
validate faster, depending on the CPU.
This is an **opt-in external dependency**. The library itself remains header-only and its behavior is unchanged: the This is an **opt-in external dependency**. The library itself remains header-only and its behavior is unchanged: the
same input is accepted or rejected either way, and every parse error is reported at the same position with the same same input is accepted or rejected either way, and every parse error is reported at the same position with the same
@@ -39,5 +39,6 @@ the library.
## Version history ## Version history
- Added in version 3.11.0. Changed inline namespace name in version 3.11.2. Added the ABI tag `_dp` in - Added in version 3.11.0. Changed inline namespace name in version 3.11.2. Added the ABI tag `_dp` in
version 3.12.0, and the ABI tags `_bics`, `_psp`, `_snul`, `_sbu8`, and `_ekmo` in version 3.13.0; see version 3.12.0, and the ABI tags `_bics`, `_psp`, `_snul`, `_sbu8`, `_ekmo`, `_dsbs`, and `_dufio` in version
3.13.0; see
[`nlohmann` Namespace](../../features/namespace.md#structure). [`nlohmann` Namespace](../../features/namespace.md#structure).
-7
View File
@@ -174,13 +174,6 @@ security reasons (e.g., Intel Software Guard Extensions (SGX)).
See [full documentation of `JSON_NO_IO`](../api/macros/json_no_io.md). See [full documentation of `JSON_NO_IO`](../api/macros/json_no_io.md).
## `JSON_NO_SIMD`
When defined, the library uses only portable code instead of vector instructions (SSE2, SSSE3, NEON) for scanning
strings and writing floating-point numbers. Results are the same either way. It is not defined by default.
See [full documentation of `JSON_NO_SIMD`](../api/macros/json_no_simd.md).
## `JSON_NO_THREAD_LOCAL` ## `JSON_NO_THREAD_LOCAL`
When defined, the library does not use `#!cpp thread_local` storage. Copying a value and comparing two values then When defined, the library does not use `#!cpp thread_local` storage. Copying a value and comparing two values then
+8 -2
View File
@@ -23,6 +23,10 @@ The complete default namespace name is derived as follows:
- [`JSON_STRICT_BINARY_UTF8`](../api/macros/json_strict_binary_utf8.md) defined non-zero appends `_sbu8`. - [`JSON_STRICT_BINARY_UTF8`](../api/macros/json_strict_binary_utf8.md) defined non-zero appends `_sbu8`.
- [`JSON_USE_OBJECTS_FOR_ENUM_KEYED_MAPS`](../api/macros/json_use_objects_for_enum_keyed_maps.md) defined non-zero - [`JSON_USE_OBJECTS_FOR_ENUM_KEYED_MAPS`](../api/macros/json_use_objects_for_enum_keyed_maps.md) defined non-zero
appends `_ekmo`. appends `_ekmo`.
- [`JSON_DISABLE_STREAM_BULK_SCAN`](../api/macros/json_disable_stream_bulk_scan.md) defined non-zero appends
`_dsbs`.
- [`JSON_DISABLE_UNLOCKED_FILE_IO`](../api/macros/json_disable_unlocked_file_io.md) defined non-zero appends
`_dufio`.
- The inline namespace ends with the suffix `_v` followed by the 3 components of the version number separated by - The inline namespace ends with the suffix `_v` followed by the 3 components of the version number separated by
underscores. To omit the version component, see [Disabling the version component](#disabling-the-version-component) underscores. To omit the version component, see [Disabling the version component](#disabling-the-version-component)
below. below.
@@ -105,5 +109,7 @@ follows:
- Added ABI tags `_bics` ([`JSON_BRACE_INIT_COPY_SEMANTICS`](../api/macros/json_brace_init_copy_semantics.md)), `_psp` - Added ABI tags `_bics` ([`JSON_BRACE_INIT_COPY_SEMANTICS`](../api/macros/json_brace_init_copy_semantics.md)), `_psp`
([`JSON_PRECISE_STREAM_POSITION`](../api/macros/json_precise_stream_position.md)), `_snul` ([`JSON_PRECISE_STREAM_POSITION`](../api/macros/json_precise_stream_position.md)), `_snul`
([`JSON_STRICT_NUL_HANDLING`](../api/macros/json_strict_nul_handling.md)), `_sbu8` ([`JSON_STRICT_NUL_HANDLING`](../api/macros/json_strict_nul_handling.md)), `_sbu8`
([`JSON_STRICT_BINARY_UTF8`](../api/macros/json_strict_binary_utf8.md)), and `_ekmo` ([`JSON_STRICT_BINARY_UTF8`](../api/macros/json_strict_binary_utf8.md)), `_ekmo`
([`JSON_USE_OBJECTS_FOR_ENUM_KEYED_MAPS`](../api/macros/json_use_objects_for_enum_keyed_maps.md)) in version 3.13.0. ([`JSON_USE_OBJECTS_FOR_ENUM_KEYED_MAPS`](../api/macros/json_use_objects_for_enum_keyed_maps.md)), `_dsbs`
([`JSON_DISABLE_STREAM_BULK_SCAN`](../api/macros/json_disable_stream_bulk_scan.md)), and `_dufio`
([`JSON_DISABLE_UNLOCKED_FILE_IO`](../api/macros/json_disable_unlocked_file_io.md)) in version 3.13.0.
@@ -79,10 +79,6 @@ follow the usual `std::streambuf` contract, and
[`JSON_DISABLE_UNLOCKED_FILE_IO`](../../api/macros/json_disable_unlocked_file_io.md) for programs in which another [`JSON_DISABLE_UNLOCKED_FILE_IO`](../../api/macros/json_disable_unlocked_file_io.md) for programs in which another
thread must use the same `#!cpp FILE` object while it is parsed. thread must use the same `#!cpp FILE` object while it is parsed.
Where the target has them, strings are scanned with vector instructions: SSE2 on x86-64 (and SSSE3 for UTF-8 text, when
the CPU has it) and NEON on AArch64. [`JSON_NO_SIMD`](../../api/macros/json_no_simd.md) selects portable code instead,
again without changing any result.
## SAX vs. DOM parsing ## SAX vs. DOM parsing
The library offers two parsing models: The library offers two parsing models:
+1 -1
View File
@@ -59,7 +59,7 @@ below are relative to `include/nlohmann`.
|-----------|----------| |-----------|----------|
| Value type enumeration | [`detail/value_t.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/value_t.hpp) | | Value type enumeration | [`detail/value_t.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/value_t.hpp) |
| Input adapters | [`detail/input/input_adapters.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/input_adapters.hpp) | | Input adapters | [`detail/input/input_adapters.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/input_adapters.hpp) |
| Lexer | [`detail/input/lexer.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/lexer.hpp), [`detail/input/number_parse.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/number_parse.hpp), [`detail/input/string_scan.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/string_scan.hpp), [`detail/input/simd_scan.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/simd_scan.hpp) | | Lexer | [`detail/input/lexer.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/lexer.hpp), [`detail/input/number_parse.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/number_parse.hpp), [`detail/input/string_scan.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/string_scan.hpp) |
| Parser | [`detail/input/parser.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/parser.hpp) | | Parser | [`detail/input/parser.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/parser.hpp) |
| SAX interface and DOM builders | [`detail/input/json_sax.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/json_sax.hpp) | | SAX interface and DOM builders | [`detail/input/json_sax.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/json_sax.hpp) |
| Binary format readers | [`detail/input/binary_reader.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/binary_reader.hpp) | | Binary format readers | [`detail/input/binary_reader.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/binary_reader.hpp) |
-1
View File
@@ -310,7 +310,6 @@ nav:
- 'JSON_NO_AUTOMATIC_UDLS': api/macros/json_no_automatic_udls.md - 'JSON_NO_AUTOMATIC_UDLS': api/macros/json_no_automatic_udls.md
- 'JSON_NOEXCEPTION': api/macros/json_noexception.md - 'JSON_NOEXCEPTION': api/macros/json_noexception.md
- 'JSON_NO_IO': api/macros/json_no_io.md - 'JSON_NO_IO': api/macros/json_no_io.md
- 'JSON_NO_SIMD': api/macros/json_no_simd.md
- 'JSON_NO_THREAD_LOCAL': api/macros/json_no_thread_local.md - 'JSON_NO_THREAD_LOCAL': api/macros/json_no_thread_local.md
- 'JSON_PRECISE_STREAM_POSITION': api/macros/json_precise_stream_position.md - 'JSON_PRECISE_STREAM_POSITION': api/macros/json_precise_stream_position.md
- 'JSON_SKIP_LIBRARY_VERSION_CHECK': api/macros/json_skip_library_version_check.md - 'JSON_SKIP_LIBRARY_VERSION_CHECK': api/macros/json_skip_library_version_check.md
+26 -4
View File
@@ -54,6 +54,14 @@
#define JSON_USE_OBJECTS_FOR_ENUM_KEYED_MAPS 0 #define JSON_USE_OBJECTS_FOR_ENUM_KEYED_MAPS 0
#endif #endif
#ifndef JSON_DISABLE_STREAM_BULK_SCAN
#define JSON_DISABLE_STREAM_BULK_SCAN 0
#endif
#ifndef JSON_DISABLE_UNLOCKED_FILE_IO
#define JSON_DISABLE_UNLOCKED_FILE_IO 0
#endif
#if JSON_DIAGNOSTICS #if JSON_DIAGNOSTICS
#define NLOHMANN_JSON_ABI_TAG_DIAGNOSTICS _diag #define NLOHMANN_JSON_ABI_TAG_DIAGNOSTICS _diag
#else #else
@@ -102,14 +110,26 @@
#define NLOHMANN_JSON_ABI_TAG_OBJECTS_FOR_ENUM_KEYED_MAPS #define NLOHMANN_JSON_ABI_TAG_OBJECTS_FOR_ENUM_KEYED_MAPS
#endif #endif
#if JSON_DISABLE_STREAM_BULK_SCAN
#define NLOHMANN_JSON_ABI_TAG_DISABLE_STREAM_BULK_SCAN _dsbs
#else
#define NLOHMANN_JSON_ABI_TAG_DISABLE_STREAM_BULK_SCAN
#endif
#if JSON_DISABLE_UNLOCKED_FILE_IO
#define NLOHMANN_JSON_ABI_TAG_DISABLE_UNLOCKED_FILE_IO _dufio
#else
#define NLOHMANN_JSON_ABI_TAG_DISABLE_UNLOCKED_FILE_IO
#endif
#ifndef NLOHMANN_JSON_NAMESPACE_NO_VERSION #ifndef NLOHMANN_JSON_NAMESPACE_NO_VERSION
#define NLOHMANN_JSON_NAMESPACE_NO_VERSION 0 #define NLOHMANN_JSON_NAMESPACE_NO_VERSION 0
#endif #endif
// Construct the namespace ABI tags component // Construct the namespace ABI tags component
#define NLOHMANN_JSON_ABI_TAGS_CONCAT_EX(a, b, c, d, e, f, g, h) json_abi ## a ## b ## c ## d ## e ## f ## g ## h #define NLOHMANN_JSON_ABI_TAGS_CONCAT_EX(a, b, c, d, e, f, g, h, i, j) json_abi ## a ## b ## c ## d ## e ## f ## g ## h ## i ## j
#define NLOHMANN_JSON_ABI_TAGS_CONCAT(a, b, c, d, e, f, g, h) \ #define NLOHMANN_JSON_ABI_TAGS_CONCAT(a, b, c, d, e, f, g, h, i, j) \
NLOHMANN_JSON_ABI_TAGS_CONCAT_EX(a, b, c, d, e, f, g, h) NLOHMANN_JSON_ABI_TAGS_CONCAT_EX(a, b, c, d, e, f, g, h, i, j)
#define NLOHMANN_JSON_ABI_TAGS \ #define NLOHMANN_JSON_ABI_TAGS \
NLOHMANN_JSON_ABI_TAGS_CONCAT( \ NLOHMANN_JSON_ABI_TAGS_CONCAT( \
@@ -120,7 +140,9 @@
NLOHMANN_JSON_ABI_TAG_PRECISE_STREAM_POSITION, \ NLOHMANN_JSON_ABI_TAG_PRECISE_STREAM_POSITION, \
NLOHMANN_JSON_ABI_TAG_STRICT_NUL_HANDLING, \ NLOHMANN_JSON_ABI_TAG_STRICT_NUL_HANDLING, \
NLOHMANN_JSON_ABI_TAG_STRICT_BINARY_UTF8, \ NLOHMANN_JSON_ABI_TAG_STRICT_BINARY_UTF8, \
NLOHMANN_JSON_ABI_TAG_OBJECTS_FOR_ENUM_KEYED_MAPS) NLOHMANN_JSON_ABI_TAG_OBJECTS_FOR_ENUM_KEYED_MAPS, \
NLOHMANN_JSON_ABI_TAG_DISABLE_STREAM_BULK_SCAN, \
NLOHMANN_JSON_ABI_TAG_DISABLE_UNLOCKED_FILE_IO)
// Construct the namespace version component // Construct the namespace version component
#define NLOHMANN_JSON_NAMESPACE_VERSION_CONCAT_EX(major, minor, patch) \ #define NLOHMANN_JSON_NAMESPACE_VERSION_CONCAT_EX(major, minor, patch) \
@@ -23,12 +23,12 @@
#endif #endif
// SSE2 (every x86-64 CPU) and NEON (every 64-bit Arm CPU) convert the 16 // SSE2 (every x86-64 CPU) and NEON (every 64-bit Arm CPU) convert the 16
// digits of a double at once (unless JSON_NO_SIMD selects the portable code) // digits of a double at once
#if !defined(JSON_NO_SIMD) && (defined(__x86_64__) || (defined(_M_X64) && !defined(_M_ARM64EC))) #if defined(__x86_64__) || (defined(_M_X64) && !defined(_M_ARM64EC))
#include <emmintrin.h> #include <emmintrin.h>
#define JSON_DTOA_SSE2 1 #define JSON_DTOA_SSE2 1
#define JSON_DTOA_NEON 0 #define JSON_DTOA_NEON 0
#elif !defined(JSON_NO_SIMD) && (defined(__aarch64__) || defined(_M_ARM64)) && !defined(_M_ARM64EC) && !defined(__ARM_BIG_ENDIAN) #elif (defined(__aarch64__) || defined(_M_ARM64)) && !defined(_M_ARM64EC) && !defined(__ARM_BIG_ENDIAN)
#include <arm_neon.h> #include <arm_neon.h>
#define JSON_DTOA_SSE2 0 #define JSON_DTOA_SSE2 0
#define JSON_DTOA_NEON 1 #define JSON_DTOA_NEON 1
@@ -13,7 +13,7 @@
#include <optional> // optional #include <optional> // optional
#endif #endif
#include <iterator> // begin, end #include <iterator> // begin, end, make_move_iterator, move_iterator
#include <memory> // allocator_traits #include <memory> // allocator_traits
#include <string> // basic_string, char_traits #include <string> // basic_string, char_traits
#include <tuple> // tuple, get #include <tuple> // tuple, get
@@ -48,6 +48,32 @@ namespace detail
* leave j with the type of the new value, but the pointer to the destroyed old one. * leave j with the type of the new value, but the pointer to the destroyed old one.
*/ */
// a move iterator over the elements of an rvalue container, unless they are
// const (e.g., std::set): libstdc++ before GCC 5 defines the reference of a
// std::move_iterator as value_type&&, which cannot bind to a const element
template<typename Iterator>
std::move_iterator<Iterator> make_move_iterator_if_mutable_impl(const Iterator& it, std::true_type /*mutable*/)
{
return std::make_move_iterator(it);
}
template<typename Iterator>
Iterator make_move_iterator_if_mutable_impl(const Iterator& it, std::false_type /*mutable*/)
{
return it;
}
template<typename Iterator>
using is_mutable_iterator = std::integral_constant < bool,
!std::is_const<typename std::remove_reference<decltype(*std::declval<Iterator&>())>::type>::value >;
template<typename Iterator>
auto make_move_iterator_if_mutable(const Iterator& it)
-> decltype(make_move_iterator_if_mutable_impl(it, is_mutable_iterator<Iterator> {}))
{
return make_move_iterator_if_mutable_impl(it, is_mutable_iterator<Iterator> {});
}
template<value_t> struct external_constructor; template<value_t> struct external_constructor;
template<> template<>
@@ -174,7 +200,7 @@ struct external_constructor<value_t::array>
using std::begin; using std::begin;
using std::end; using std::end;
const typename BasicJsonType::json_value value(j.template create<typename BasicJsonType::array_t>(std::make_move_iterator(begin(arr)), std::make_move_iterator(end(arr)))); const typename BasicJsonType::json_value value(j.template create<typename BasicJsonType::array_t>(make_move_iterator_if_mutable(begin(arr)), make_move_iterator_if_mutable(end(arr))));
j.replace_value(value_t::array, value); j.replace_value(value_t::array, value);
} }
@@ -235,7 +261,7 @@ struct external_constructor<value_t::object>
using std::begin; using std::begin;
using std::end; using std::end;
const typename BasicJsonType::json_value value(j.template create<typename BasicJsonType::object_t>(std::make_move_iterator(begin(obj)), std::make_move_iterator(end(obj)))); const typename BasicJsonType::json_value value(j.template create<typename BasicJsonType::object_t>(make_move_iterator_if_mutable(begin(obj)), make_move_iterator_if_mutable(end(obj))));
j.replace_value(value_t::object, value); j.replace_value(value_t::object, value);
} }
@@ -413,25 +439,27 @@ using is_movable_container_rvalue = negation<disjunction<
std::is_const<T>, std::is_const<T>,
has_dedicated_to_json<T>, has_dedicated_to_json<T>,
is_basic_json<T>, is_basic_json<T>,
is_range_view_type<T>,
is_enum_keyed_map_type<T>, is_enum_keyed_map_type<T>,
std::is_same<T, std::vector<bool>>>>; std::is_same<T, std::vector<bool>>>>;
// rvalue array-like containers // rvalue array-like containers; object types are ruled out before views are
// checked, as std::ranges::view<T> for the object_t of an ordered_json would
// need that ordered_json to be complete (GCC 11, see #3312)
template<typename BasicJsonType, typename T> template<typename BasicJsonType, typename T>
using is_rvalue_array_to_json = conjunction<is_movable_container_rvalue<T>, using is_rvalue_array_to_json = conjunction<is_movable_container_rvalue<T>,
negation<is_compatible_object_type<BasicJsonType, T>>,
negation<is_range_view_type<T>>,
is_compatible_array_type<BasicJsonType, T>, is_compatible_array_type<BasicJsonType, T>,
negation<disjunction< negation<disjunction<
std::is_same<T, typename BasicJsonType::array_t>, std::is_same<T, typename BasicJsonType::array_t>,
std::is_same<T, typename BasicJsonType::binary_t>, std::is_same<T, typename BasicJsonType::binary_t>,
is_compatible_object_type<BasicJsonType, T>,
is_compatible_string_type<BasicJsonType, T>, is_compatible_string_type<BasicJsonType, T>,
is_compatible_binary_type<BasicJsonType, T>>>>; is_compatible_binary_type<BasicJsonType, T>>>>;
// rvalue containers: their elements are moved instead of copied // rvalue containers: their elements are moved instead of copied
template<typename BasicJsonType, typename CompatibleArrayType, template<typename BasicJsonType, typename CompatibleArrayType,
enable_if_t<is_rvalue_array_to_json<BasicJsonType, CompatibleArrayType>::value, int> = 0> enable_if_t<is_rvalue_array_to_json<BasicJsonType, CompatibleArrayType>::value, int> = 0>
inline void to_json(BasicJsonType& j, CompatibleArrayType && arr) inline void to_json(BasicJsonType& j, CompatibleArrayType && arr) // NOLINT(cppcoreguidelines-missing-std-forward)
{ {
external_constructor<value_t::array>::construct_moving(j, arr); external_constructor<value_t::array>::construct_moving(j, arr);
} }
@@ -509,7 +537,8 @@ inline void to_json(BasicJsonType& j, const EnumKeyedMap& map)
} }
#endif #endif
// rvalue object-like containers // rvalue object-like containers (views have no key_type and mapped_type, so
// they are no compatible object type)
template<typename BasicJsonType, typename T> template<typename BasicJsonType, typename T>
using is_rvalue_object_to_json = conjunction<is_movable_container_rvalue<T>, using is_rvalue_object_to_json = conjunction<is_movable_container_rvalue<T>,
negation<std::is_same<T, typename BasicJsonType::object_t>>, negation<std::is_same<T, typename BasicJsonType::object_t>>,
@@ -517,7 +546,7 @@ using is_rvalue_object_to_json = conjunction<is_movable_container_rvalue<T>,
template<typename BasicJsonType, typename CompatibleObjectType, template<typename BasicJsonType, typename CompatibleObjectType,
enable_if_t<is_rvalue_object_to_json<BasicJsonType, CompatibleObjectType>::value, int> = 0> enable_if_t<is_rvalue_object_to_json<BasicJsonType, CompatibleObjectType>::value, int> = 0>
inline void to_json(BasicJsonType& j, CompatibleObjectType && obj) inline void to_json(BasicJsonType& j, CompatibleObjectType && obj) // NOLINT(cppcoreguidelines-missing-std-forward)
{ {
external_constructor<value_t::object>::construct_moving(j, obj); external_constructor<value_t::object>::construct_moving(j, obj);
} }
+1 -1
View File
@@ -185,7 +185,7 @@ JSON_HEDLEY_ALWAYS_INLINE shortest_decimal to_shortest(std::uint64_t bits) noexc
const uint128_parts p = umul192_hi128(p10.high, p10.low, bin_sig << static_cast<unsigned>(shift)); const uint128_parts p = umul192_hi128(p10.high, p10.low, bin_sig << static_cast<unsigned>(shift));
std::uint64_t integral = p.high >> static_cast<unsigned>(extra_shift); std::uint64_t integral = p.high >> static_cast<unsigned>(extra_shift);
// (no casts in the shifts: MSVC 2015 warns C4554 on them despite the parentheses) // (no casts in the shifts: MSVC 2015 warns C4554 on them despite the parentheses)
constexpr unsigned fractional_low_shift = static_cast<unsigned>(extra_shift); constexpr auto fractional_low_shift = static_cast<unsigned>(extra_shift);
constexpr unsigned fractional_high_shift = 64u - fractional_low_shift; constexpr unsigned fractional_high_shift = 64u - fractional_low_shift;
const std::uint64_t fractional = (p.high << fractional_high_shift) | (p.low >> fractional_low_shift); const std::uint64_t fractional = (p.high << fractional_high_shift) | (p.low >> fractional_low_shift);
std::uint64_t digit = 0; std::uint64_t digit = 0;
+1 -2
View File
@@ -494,8 +494,7 @@ JSON_HEDLEY_NO_RETURN JSON_HEDLEY_NEVER_INLINE void throw_patch_missing_member(c
static_cast<void>(string_type); static_cast<void>(string_type);
static_cast<void>(val); static_cast<void>(val);
// context-sensitive error message // context-sensitive error message
const std::string error_msg = (op == "op") ? std::string("operation") : concat("operation '", op, '\''); JSON_THROW(parse_error::create(exception_id::patch_invalid_operation, 0, concat((op == "op") ? std::string("operation") : concat("operation '", op, '\''), string_type ? " must have string member '" : " must have member '", member, "'"), &val));
JSON_THROW(parse_error::create(exception_id::patch_invalid_operation, 0, concat(error_msg, string_type ? " must have string member '" : " must have member '", member, "'"), &val));
} }
template<typename StringType, typename BasicJsonType> template<typename StringType, typename BasicJsonType>
@@ -91,10 +91,7 @@ class file_input_adapter
file_input_adapter& operator=(file_input_adapter&&) = delete; file_input_adapter& operator=(file_input_adapter&&) = delete;
~file_input_adapter() ~file_input_adapter()
{ {
if (m_file != nullptr) unlock();
{
unlock();
}
} }
std::char_traits<char>::int_type get_character() noexcept std::char_traits<char>::int_type get_character() noexcept
@@ -132,6 +129,11 @@ class file_input_adapter
void unlock() noexcept void unlock() noexcept
{ {
// a moved-from adapter has no file and holds no lock
if (m_file == nullptr)
{
return;
}
#if JSON_UNLOCKED_FILE_IO == 1 #if JSON_UNLOCKED_FILE_IO == 1
funlockfile(m_file); funlockfile(m_file);
#elif JSON_UNLOCKED_FILE_IO == 2 #elif JSON_UNLOCKED_FILE_IO == 2
+39 -72
View File
@@ -12,7 +12,6 @@
#include <cstddef> // size_t #include <cstddef> // size_t
#include <cstdint> // uint32_t #include <cstdint> // uint32_t
#include <cstdio> // snprintf #include <cstdio> // snprintf
#include <cstring> // memcmp
#include <initializer_list> // initializer_list #include <initializer_list> // initializer_list
#include <string> // char_traits, string #include <string> // char_traits, string
#include <utility> // move #include <utility> // move
@@ -418,43 +417,30 @@ class lexer : public lexer_base<BasicJsonType>
} }
/// contiguous input: bulk-append the run of ordinary characters and complete /// contiguous input: bulk-append the run of ordinary characters and complete
/// well-formed UTF-8 sequences starting at the current read position, then /// well-formed UTF-8 sequences starting at the current read position, leaving
/// read the first byte that needs individual handling (the closing quote, an /// the first byte that needs individual handling (the closing quote, an
/// escape, a control character, or an ill-formed UTF-8 byte) like get(). /// escape, a control character, or an ill-formed UTF-8 byte) for get()
/// That byte is taken from the buffer that was just scanned. Through get(), void scan_string_bulk(std::true_type /*bulk*/)
/// Apple Clang merged this read with the other get() calls of the scan
/// function, which load the read position again right after it was stored
/// (3% of json::accept() on jeopardy.json on Apple M1).
char_int_type get_after_string_run(std::true_type /*bulk*/)
{ {
// a pending unget must be consumed through the normal path first // a pending unget must be consumed through the normal path first
if (next_unget) if (next_unget)
{ {
return get(); return;
} }
const std::size_t remaining = ia.bulk_remaining(); const std::size_t remaining = ia.bulk_remaining();
if (remaining == 0) if (remaining == 0)
{ {
return get(); return;
} }
const auto* const raw = ia.bulk_data(); const auto* const data = reinterpret_cast<const unsigned char*>(ia.bulk_data());
const auto* const data = reinterpret_cast<const unsigned char*>(raw);
const std::size_t pos = lexer_string_bulk_run(data, remaining); const std::size_t pos = string_bulk_run(data, remaining);
if (pos != 0) if (pos == 0)
{ {
token_buffer.append(reinterpret_cast<const typename string_t::value_type*>(data), pos); return;
bulk_consume(reinterpret_cast<const char*>(data), pos);
} }
if (pos == remaining) token_buffer.append(reinterpret_cast<const typename string_t::value_type*>(data), pos);
{ bulk_consume(reinterpret_cast<const char*>(data), pos);
return get(); // end of the contiguous input
}
// what get() does, with the byte at hand
advance_position();
ia.bulk_skip(1);
current = char_traits<char_type>::to_int_type(raw[pos]);
return track_after_read();
} }
/// consume @a n characters at @a data (the adapter's bulk_data()) and do /// consume @a n characters at @a data (the adapter's bulk_data()) and do
@@ -487,10 +473,7 @@ class lexer : public lexer_base<BasicJsonType>
} }
/// streaming input: no bulk fast path /// streaming input: no bulk fast path
char_int_type get_after_string_run(std::false_type /*bulk*/) void scan_string_bulk(std::false_type /*bulk*/) const noexcept {}
{
return get();
}
/*! /*!
@brief scan a string literal @brief scan a string literal
@@ -518,8 +501,11 @@ class lexer : public lexer_base<BasicJsonType>
while (true) while (true)
{ {
// bulk-consume ordinary characters from contiguous input, then // bulk-consume ordinary characters from contiguous input, then
// get the next character, the one that ends the run // handle the next special byte through the switch below
switch (get_after_string_run(std::integral_constant<bool, bulk_scan>())) scan_string_bulk(std::integral_constant<bool, bulk_scan> {});
// get the next character
switch (get())
{ {
// end of file while parsing the string // end of file while parsing the string
case char_traits<char_type>::eof(): case char_traits<char_type>::eof():
@@ -1855,23 +1841,18 @@ scan_number_done:
} }
/*! /*!
@param[in] rest the literal text after its first character, which @param[in] literal_text the literal text to expect
is current (its length is a template parameter, so @param[in] length the length of the passed literal text
that the comparison of the contiguous fast path is @param[in] return_type the token type to return on success
a fixed-size load, not a call of memcmp)
@param[in] return_type the token type to return on success
*/ */
template<std::size_t N> JSON_HEDLEY_NON_NULL(2)
JSON_HEDLEY_NEVER_INLINE token_type scan_literal(const char (&rest)[N], token_type return_type) token_type scan_literal(const char_type* literal_text, const std::size_t length,
token_type return_type)
{ {
constexpr std::size_t length = N - 1; JSON_ASSERT(char_traits<char_type>::to_char_type(current) == literal_text[0]);
if (scan_literal_bulk(rest, length, std::integral_constant<bool, bulk_scan> {})) for (std::size_t i = 1; i < length; ++i)
{ {
return return_type; if (JSON_HEDLEY_UNLIKELY(char_traits<char_type>::to_char_type(get()) != literal_text[i]))
}
for (std::size_t i = 0; i < length; ++i)
{
if (JSON_HEDLEY_UNLIKELY(char_traits<char_type>::to_char_type(get()) != static_cast<char_type>(rest[i])))
{ {
error_message = "invalid literal"; error_message = "invalid literal";
return token_type::parse_error; return token_type::parse_error;
@@ -1880,27 +1861,6 @@ scan_number_done:
return return_type; return return_type;
} }
/// contiguous input: compare the rest of a literal at once; false (and
/// nothing consumed) if it does not match, so that the byte path reports
/// the error at the right character
JSON_HEDLEY_ALWAYS_INLINE bool scan_literal_bulk(const char* rest, const std::size_t length, std::true_type /*bulk*/)
{
if (next_unget || ia.bulk_remaining() < length
|| std::memcmp(ia.bulk_data(), rest, length) != 0)
{
return false;
}
bulk_consume(reinterpret_cast<const char*>(ia.bulk_data()), length);
current = static_cast<char_int_type>(static_cast<unsigned char>(rest[length - 1]));
return true;
}
/// streaming input: no bulk fast path
bool scan_literal_bulk(const char* /*rest*/, const std::size_t /*length*/, std::false_type /*bulk*/) const noexcept
{
return false;
}
///////////////////// /////////////////////
// input management // input management
///////////////////// /////////////////////
@@ -2526,14 +2486,21 @@ scan_number_done:
return token_type::value_separator; return token_type::value_separator;
// literals // literals
// (the text after the first character; no local arrays, which
// would give this function a stack canary on every token)
case 't': case 't':
return scan_literal("rue", token_type::literal_true); {
std::array<char_type, 4> true_literal = {{static_cast<char_type>('t'), static_cast<char_type>('r'), static_cast<char_type>('u'), static_cast<char_type>('e')}};
return scan_literal(true_literal.data(), true_literal.size(), token_type::literal_true);
}
case 'f': case 'f':
return scan_literal("alse", token_type::literal_false); {
std::array<char_type, 5> false_literal = {{static_cast<char_type>('f'), static_cast<char_type>('a'), static_cast<char_type>('l'), static_cast<char_type>('s'), static_cast<char_type>('e')}};
return scan_literal(false_literal.data(), false_literal.size(), token_type::literal_false);
}
case 'n': case 'n':
return scan_literal("ull", token_type::literal_null); {
std::array<char_type, 4> null_literal = {{static_cast<char_type>('n'), static_cast<char_type>('u'), static_cast<char_type>('l'), static_cast<char_type>('l')}};
return scan_literal(null_literal.data(), null_literal.size(), token_type::literal_null);
}
// string // string
case '\"': case '\"':
+5 -6
View File
@@ -244,9 +244,8 @@ class parser
bool sax_parse_internal(SAX* sax) bool sax_parse_internal(SAX* sax)
{ {
// stack to remember the hierarchy of structured values we are parsing // stack to remember the hierarchy of structured values we are parsing
// true = array; false = object (bytes, not std::vector<bool>, whose // true = array; false = object
// push_back() is a call with GCC) std::vector<bool> states;
std::vector<std::uint8_t> states;
// value to avoid a goto (see comment where set to true) // value to avoid a goto (see comment where set to true)
bool skip_to_state_evaluation = false; bool skip_to_state_evaluation = false;
@@ -291,7 +290,7 @@ class parser
} }
// remember we are now inside an object // remember we are now inside an object
states.push_back(0); states.push_back(false);
// parse values // parse values
get_token(); get_token();
@@ -316,7 +315,7 @@ class parser
} }
// remember we are now inside an array // remember we are now inside an array
states.push_back(1); states.push_back(true);
// parse values (no need to call get_token) // parse values (no need to call get_token)
continue; continue;
@@ -433,7 +432,7 @@ class parser
return true; return true;
} }
if (states.back() != 0) // array if (states.back()) // array
{ {
// comma -> next value // comma -> next value
// or end of array (ignore_trailing_commas = true) // or end of array (ignore_trailing_commas = true)
-314
View File
@@ -1,314 +0,0 @@
// __ _____ _____ _____
// __| | __| | | | JSON for Modern C++
// | | |__ | | | | | | version 3.12.0
// |_____|_____|_____|_|___| https://github.com/nlohmann/json
//
// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
// SPDX-FileCopyrightText: 2018-2025 The simdjson authors <https://github.com/simdjson/simdjson>
// SPDX-License-Identifier: MIT
#pragma once
#include <array> // array
#include <atomic> // atomic
#include <cstddef> // size_t
#include <cstdint> // uint8_t, uint64_t
#include <nlohmann/detail/bit_ops.hpp>
#include <nlohmann/detail/macro_scope.hpp>
// Vector code for runs of string bytes, used by the lexer for contiguous input
// (see string_bulk_run()). NEON (AArch64) and SSE2 (x86-64) belong to the
// baseline instruction sets and are used by default. The vector UTF-8 check
// needs NEON or SSSE3. SSSE3 is not part of x86-64, and the code must not
// depend on the flags of a translation unit (two translation units with
// different flags would have different definitions of the same inline
// functions): the check is compiled for SSSE3 with a function attribute and
// used where the CPU has SSSE3 (all x86-64 CPUs since about 2011), else the
// portable check. Code compiled for SSSE3 anyway (-mssse3 or more) skips the
// CPU check. JSON_NO_SIMD selects the portable code everywhere.
// NLOHMANN_JSON_NO_TARGET_ATTRIBUTE (for compilers that ignore the target
// attribute and then reject the SSSE3 intrinsics) keeps the check portable
// without disabling the SSE2 code.
#if !defined(JSON_NO_SIMD) && defined(__aarch64__) && (defined(__GNUC__) || defined(__clang__)) && defined(__BYTE_ORDER__) && defined(__ORDER_LITTLE_ENDIAN__) && __BYTE_ORDER__ == __ORDER_LITTLE_ENDIAN__
#include <arm_neon.h>
#define JSON_SCAN_NEON 1
#else
#define JSON_SCAN_NEON 0
#endif
// (x86 only: other targets can define __SSE2__ as well, e.g., WebAssembly with -msse2, but have no <cpuid.h>)
#if !defined(JSON_NO_SIMD) && !JSON_SCAN_NEON && (defined(__x86_64__) || defined(__i386__) || defined(_M_X64) || defined(_M_IX86)) && !defined(_M_ARM64EC) && (defined(__SSE2__) || defined(_M_X64) || (defined(_M_IX86_FP) && _M_IX86_FP >= 2))
#include <emmintrin.h>
#define JSON_SCAN_SSE2 1
#else
#define JSON_SCAN_SSE2 0
#endif
#if JSON_SCAN_SSE2 && defined(__SSSE3__)
#include <tmmintrin.h>
#define JSON_SCAN_SSSE3 1 // NOLINT(cppcoreguidelines-macro-to-enum,modernize-macro-to-enum)
#else
#define JSON_SCAN_SSSE3 0 // NOLINT(cppcoreguidelines-macro-to-enum,modernize-macro-to-enum)
#endif
#if JSON_SCAN_SSE2 && !JSON_SCAN_SSSE3 && !defined(NLOHMANN_JSON_NO_TARGET_ATTRIBUTE) && ((defined(__clang__) && __clang_major__ >= 4) || (defined(__GNUC__) && !defined(__clang__) && (__GNUC__ > 4 || (__GNUC__ == 4 && __GNUC_MINOR__ >= 9))))
// (GCC before 4.9 has no SSSE3 intrinsics without -mssse3)
#include <cpuid.h>
#include <tmmintrin.h>
#define JSON_SCAN_SSSE3_DISPATCH 1 // NOLINT(cppcoreguidelines-macro-to-enum,modernize-macro-to-enum)
#define JSON_SCAN_SSSE3_TARGET __attribute__((target("ssse3")))
#elif JSON_SCAN_SSE2 && !JSON_SCAN_SSSE3 && defined(_MSC_VER)
// (MSVC compiles intrinsics of any instruction set)
#include <intrin.h>
#include <tmmintrin.h>
#define JSON_SCAN_SSSE3_DISPATCH 1 // NOLINT(cppcoreguidelines-macro-to-enum,modernize-macro-to-enum)
#define JSON_SCAN_SSSE3_TARGET
#else
#define JSON_SCAN_SSSE3_DISPATCH 0 // NOLINT(cppcoreguidelines-macro-to-enum,modernize-macro-to-enum)
#define JSON_SCAN_SSSE3_TARGET
#endif
#define JSON_SCAN_VECTOR (JSON_SCAN_NEON || JSON_SCAN_SSE2)
#define JSON_SCAN_VECTOR_UTF8 (JSON_SCAN_NEON || JSON_SCAN_SSSE3 || JSON_SCAN_SSSE3_DISPATCH)
NLOHMANN_JSON_NAMESPACE_BEGIN
namespace detail
{
#if JSON_SCAN_VECTOR
/*!
@brief the first byte of a string run that is a quote, a backslash, a control
character, or not ASCII, 16 bytes per step
Stops at such a byte, or where fewer than 16 bytes are left (the caller tells
the two apart). A signed compare with 0x20 finds control characters and
non-ASCII bytes at once.
*/
JSON_HEDLEY_ALWAYS_INLINE const unsigned char* vector_plain_run(const unsigned char* p, const unsigned char* e) noexcept
{
while (e - p >= 16)
{
#if JSON_SCAN_NEON
const uint8x16_t in = vld1q_u8(p);
const uint8x16_t special = vorrq_u8(vorrq_u8(vceqq_u8(in, vdupq_n_u8('"')), vceqq_u8(in, vdupq_n_u8('\\'))),
vcltq_s8(vreinterpretq_s8_u8(in), vdupq_n_s8(0x20)));
// one nibble per byte (the usual NEON replacement of x86's movemask, see
// D. Kutenin, "Porting x86 vector bitmask optimizations to Arm NEON", 2022)
const std::uint64_t bits = vget_lane_u64(vreinterpret_u64_u8(vshrn_n_u16(vreinterpretq_u16_u8(special), 4)), 0);
if (bits != 0)
{
return p + (count_trailing_zeros(bits) >> 2u);
}
#else
const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(p))); // NOLINT(bugprone-casting-through-void)
const __m128i special = _mm_or_si128(_mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('"')), _mm_cmpeq_epi8(in, _mm_set1_epi8('\\'))),
_mm_cmplt_epi8(in, _mm_set1_epi8(0x20)));
const auto bits = static_cast<std::uint64_t>(static_cast<unsigned>(_mm_movemask_epi8(special)));
if (bits != 0)
{
return p + count_trailing_zeros(bits);
}
#endif
p += 16;
}
return p;
}
#endif
#if JSON_SCAN_VECTOR
/// vector_plain_run() for runs that are already longer than a few words:
/// inline with SSE2, out of line with NEON, where a larger string loop slows
/// down the rest of the lexer
#if JSON_SCAN_NEON
JSON_HEDLEY_NEVER_INLINE inline
#else
JSON_HEDLEY_ALWAYS_INLINE
#endif
const unsigned char* vector_plain_run_long(const unsigned char* p, const unsigned char* e) noexcept
{
return vector_plain_run(p, e);
}
#endif
#if JSON_SCAN_SSSE3_DISPATCH
/// whether the CPU has SSSE3 (CPUID leaf 1, ECX bit 9)
inline bool cpu_ssse3() noexcept
{
#if defined(_MSC_VER) && !defined(__clang__)
std::array<int, 4> regs {{}};
__cpuid(regs.data(), 1);
return (static_cast<unsigned>(regs[2]) & (1u << 9u)) != 0;
#else
unsigned eax = 0;
unsigned ebx = 0;
unsigned ecx = 0;
unsigned edx = 0;
return __get_cpuid(1, &eax, &ebx, &ecx, &edx) != 0 && (ecx & (1u << 9u)) != 0;
#endif
}
/// whether the CPU has SSSE3, asked once: the answer is kept in an atomic
/// that is initialized at compile time, so that neither a guard of a local
/// static nor a global constructor is needed (threads that ask at the same
/// time all store the same answer)
JSON_HEDLEY_ALWAYS_INLINE bool cpu_has_ssse3() noexcept
{
static std::atomic<int> known{0}; // 0: not asked yet, 1: no, 2: yes
int state = known.load(std::memory_order_relaxed);
if (JSON_HEDLEY_UNLIKELY(state == 0))
{
state = cpu_ssse3() ? 2 : 1;
known.store(state, std::memory_order_relaxed);
}
return state == 2;
}
#endif
#if JSON_SCAN_VECTOR_UTF8
/// Tables of the UTF-8 check of J. Keiser and D. Lemire, "Validating UTF-8 In
/// Less Than One Instruction Per Byte" (2021), as in simdjson ("lookup4"): each
/// maps a nibble (high and low nibble of the previous byte, high nibble of the
/// current byte) to the errors it allows; a byte pair is ill-formed if all
/// three have an error bit in common.
template<typename Dummy = void>
struct utf8_lookup4
{
static constexpr std::uint8_t too_short = 1u << 0u, too_long = 1u << 1u, overlong_3 = 1u << 2u, too_large = 1u << 3u;
static constexpr std::uint8_t surrogate = 1u << 4u, overlong_2 = 1u << 5u, too_large_1000 = 1u << 6u, overlong_4 = 1u << 6u;
static constexpr std::uint8_t two_conts = 1u << 7u, carry = too_short | too_long | two_conts;
static const std::array<std::uint8_t, 16> byte_1_high;
static const std::array<std::uint8_t, 16> byte_1_low;
static const std::array<std::uint8_t, 16> byte_2_high;
};
template<typename Dummy>
const std::array<std::uint8_t, 16> utf8_lookup4<Dummy>::byte_1_high =
{
{
too_long, too_long, too_long, too_long, too_long, too_long, too_long, too_long,
two_conts, two_conts, two_conts, two_conts,
too_short | overlong_2, too_short, too_short | overlong_3 | surrogate, too_short | too_large | too_large_1000 | overlong_4
}
};
template<typename Dummy>
const std::array<std::uint8_t, 16> utf8_lookup4<Dummy>::byte_1_low =
{
{
carry | overlong_3 | overlong_2 | overlong_4, carry | overlong_2, carry, carry,
carry | too_large, carry | too_large | too_large_1000, carry | too_large | too_large_1000, carry | too_large | too_large_1000,
carry | too_large | too_large_1000, carry | too_large | too_large_1000, carry | too_large | too_large_1000, carry | too_large | too_large_1000,
carry | too_large | too_large_1000, carry | too_large | too_large_1000 | surrogate, carry | too_large | too_large_1000, carry | too_large | too_large_1000
}
};
template<typename Dummy>
const std::array<std::uint8_t, 16> utf8_lookup4<Dummy>::byte_2_high =
{
{
too_short, too_short, too_short, too_short, too_short, too_short, too_short, too_short,
static_cast<std::uint8_t>(too_long | overlong_2 | two_conts | overlong_3 | too_large_1000 | overlong_4),
static_cast<std::uint8_t>(too_long | overlong_2 | two_conts | overlong_3 | too_large),
static_cast<std::uint8_t>(too_long | overlong_2 | two_conts | surrogate | too_large),
static_cast<std::uint8_t>(too_long | overlong_2 | two_conts | surrogate | too_large),
too_short, too_short, too_short, too_short
}
};
/*!
@brief the run of plain string bytes and well-formed UTF-8 at p (a character
boundary), 16 bytes per step
The first quote, backslash, or control character is found with vector
compares, and the UTF-8 check of J. Keiser and D. Lemire covers the bytes up
to it. Returns true with @a block at that byte; returns false with @a block at
the start of the 16 bytes where the check found ill-formed UTF-8 or where
fewer than 16 bytes are left, for the caller to finish one sequence at a time.
Out of line, so that no constants of the check occupy registers in the
caller's loop. On x86-64, it is compiled for SSSE3 (see cpu_has_ssse3()).
*/
JSON_SCAN_SSSE3_TARGET JSON_HEDLEY_NEVER_INLINE inline bool vector_utf8_run(const unsigned char* p, const unsigned char* e, const unsigned char*& block_out) noexcept
{
using lookup = utf8_lookup4<>;
const unsigned char* block = p;
#if JSON_SCAN_NEON
const uint8x16_t t1h = vld1q_u8(lookup::byte_1_high.data());
const uint8x16_t t1l = vld1q_u8(lookup::byte_1_low.data());
const uint8x16_t t2h = vld1q_u8(lookup::byte_2_high.data());
uint8x16_t prev = vdupq_n_u8(0);
while (e - block >= 16)
{
const uint8x16_t in = vld1q_u8(block);
const uint8x16_t special = vorrq_u8(vorrq_u8(vceqq_u8(in, vdupq_n_u8('"')), vceqq_u8(in, vdupq_n_u8('\\'))), vcltq_u8(in, vdupq_n_u8(0x20)));
const uint8x16_t prev1 = vextq_u8(prev, in, 15);
const uint8x16_t sc = vandq_u8(vandq_u8(vqtbl1q_u8(t1h, vshrq_n_u8(prev1, 4)), vqtbl1q_u8(t1l, vandq_u8(prev1, vdupq_n_u8(0x0F)))), vqtbl1q_u8(t2h, vshrq_n_u8(in, 4)));
const uint8x16_t must23 = vorrq_u8(vqsubq_u8(vextq_u8(prev, in, 14), vdupq_n_u8(0xE0 - 0x80)), vqsubq_u8(vextq_u8(prev, in, 13), vdupq_n_u8(0xF0 - 0x80)));
const uint8x16_t err = veorq_u8(vandq_u8(must23, vdupq_n_u8(0x80)), sc);
const std::uint64_t special_bits = vget_lane_u64(vreinterpret_u64_u8(vshrn_n_u16(vreinterpretq_u16_u8(special), 4)), 0);
const std::uint64_t err_bits = vget_lane_u64(vreinterpret_u64_u8(vshrn_n_u16(vreinterpretq_u16_u8(vtstq_u8(err, err)), 4)), 0);
if (special_bits != 0)
{
// errors up to the special byte count (an incomplete sequence
// before a quote shows at the quote); the bytes after it do not
const unsigned k = static_cast<unsigned>(count_trailing_zeros(special_bits)) >> 2u;
const std::uint64_t upto = k == 15 ? ~std::uint64_t{0} :
(std::uint64_t{1} << (4u * (k + 1u))) - 1u;
if ((err_bits & upto) == 0)
{
block_out = block + k;
return true;
}
break;
}
if (err_bits != 0)
{
break;
}
prev = in;
block += 16;
}
#else
// the same with SSSE3 (pshufb for the table lookups; nibbles from 16-bit
// shifts, as there are no byte shifts)
const __m128i t1h = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(lookup::byte_1_high.data()))); // NOLINT(bugprone-casting-through-void)
const __m128i t1l = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(lookup::byte_1_low.data()))); // NOLINT(bugprone-casting-through-void)
const __m128i t2h = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(lookup::byte_2_high.data()))); // NOLINT(bugprone-casting-through-void)
const __m128i nibble = _mm_set1_epi8(0x0F);
const __m128i zero = _mm_setzero_si128();
__m128i prev = zero;
while (e - block >= 16)
{
const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(block))); // NOLINT(bugprone-casting-through-void)
const __m128i special = _mm_or_si128(_mm_or_si128(_mm_cmpeq_epi8(in, _mm_set1_epi8('"')), _mm_cmpeq_epi8(in, _mm_set1_epi8('\\'))),
_mm_cmpeq_epi8(_mm_subs_epu8(in, _mm_set1_epi8(0x1F)), zero)); // in < 0x20
const __m128i prev1 = _mm_alignr_epi8(in, prev, 15);
const __m128i sc = _mm_and_si128(_mm_and_si128(_mm_shuffle_epi8(t1h, _mm_and_si128(_mm_srli_epi16(prev1, 4), nibble)),
_mm_shuffle_epi8(t1l, _mm_and_si128(prev1, nibble))),
_mm_shuffle_epi8(t2h, _mm_and_si128(_mm_srli_epi16(in, 4), nibble)));
const __m128i must23 = _mm_or_si128(_mm_subs_epu8(_mm_alignr_epi8(in, prev, 14), _mm_set1_epi8(0xE0 - 0x80)),
_mm_subs_epu8(_mm_alignr_epi8(in, prev, 13), _mm_set1_epi8(0xF0 - 0x80)));
const __m128i err = _mm_xor_si128(_mm_and_si128(must23, _mm_set1_epi8(static_cast<char>(-128))), sc);
const auto special_bits = static_cast<unsigned>(_mm_movemask_epi8(special));
const auto err_bits = ~static_cast<unsigned>(_mm_movemask_epi8(_mm_cmpeq_epi8(err, zero))) & 0xFFFFu;
if (special_bits != 0)
{
const auto k = static_cast<unsigned>(count_trailing_zeros(static_cast<std::uint64_t>(special_bits)));
if ((err_bits & ((2u << k) - 1u)) == 0)
{
block_out = block + k;
return true;
}
break;
}
if (err_bits != 0)
{
break;
}
prev = in;
block += 16;
}
#endif
block_out = block;
return false;
}
#endif
} // namespace detail
NLOHMANN_JSON_NAMESPACE_END
+16 -161
View File
@@ -14,7 +14,6 @@
#include <cstring> // memcpy #include <cstring> // memcpy
#include <nlohmann/detail/bit_ops.hpp> #include <nlohmann/detail/bit_ops.hpp>
#include <nlohmann/detail/input/simd_scan.hpp>
#include <nlohmann/detail/macro_scope.hpp> #include <nlohmann/detail/macro_scope.hpp>
// Optional SIMD backend for bulk UTF-8 validation. This is an opt-in external // Optional SIMD backend for bulk UTF-8 validation. This is an opt-in external
@@ -35,7 +34,7 @@
// This file contains the byte-level string-scanning helpers used by the lexer's // This file contains the byte-level string-scanning helpers used by the lexer's
// contiguous fast path. They operate purely on raw bytes (no dependency on the // contiguous fast path. They operate purely on raw bytes (no dependency on the
// lexer's template parameters) so they are free functions, keeping the lexer // lexer's template parameters) so they are free functions, keeping the lexer
// itself focused on the state machine; see lexer::get_after_string_run(). // itself focused on the state machine; see lexer::scan_string_bulk().
NLOHMANN_JSON_NAMESPACE_BEGIN NLOHMANN_JSON_NAMESPACE_BEGIN
namespace detail namespace detail
@@ -67,7 +66,7 @@ inline std::uint64_t swar_string_special(std::uint64_t v) noexcept
// return the index of the first is_string_special() byte in [data, data+n), or // return the index of the first is_string_special() byte in [data, data+n), or
// n if every byte is ordinary; scans 8 bytes at a time // n if every byte is ordinary; scans 8 bytes at a time
JSON_HEDLEY_ALWAYS_INLINE std::size_t find_string_special(const unsigned char* data, std::size_t n) noexcept inline std::size_t find_string_special(const unsigned char* data, std::size_t n) noexcept
{ {
std::size_t i = 0; std::size_t i = 0;
for (; i + 8 <= n; i += 8) for (; i + 8 <= n; i += 8)
@@ -236,143 +235,35 @@ inline std::size_t valid_utf8_prefix(const unsigned char* data, std::size_t n) n
return pos; return pos;
} }
#if JSON_SCAN_VECTOR_UTF8 // Scalar (C++11) computation of the bulk run length: the number of leading
/// the end of scan_string_vector from block, where the vector loop stopped // bytes in [data, data+n) that are ordinary ASCII or complete well-formed UTF-8
/// (ill-formed UTF-8, or fewer than 16 bytes left): one byte or sequence at a
/// time, from the start of a sequence that crosses into the block
inline const unsigned char* scan_string_finish(const unsigned char* p, const unsigned char* block, const unsigned char* e) noexcept
{
for (int i = 1; i <= 3 && block - i >= p; ++i)
{
const unsigned char c = block[-i];
if (c < 0x80)
{
break;
}
if (c >= 0xC0)
{
const int len = 2 + static_cast<int>(c >= 0xE0) + static_cast<int>(c >= 0xF0);
if (len > i)
{
block -= i;
}
break;
}
}
for (p = block; p != e;)
{
if (*p < 0x80)
{
if (is_string_special(*p))
{
return p;
}
++p;
continue;
}
const std::size_t n = validate_one_utf8(p, static_cast<std::size_t>(e - p));
if (n == 0)
{
return p;
}
p += n;
}
return p;
}
/// the rest of a string from p (a character boundary), with the vector UTF-8
/// check and the scalar check for the end of the block where it stopped
inline const unsigned char* scan_string_vector(const unsigned char* p, const unsigned char* e) noexcept
{
const unsigned char* block = p;
if (vector_utf8_run(p, e, block))
{
return block;
}
return scan_string_finish(p, block, e);
}
#endif
#if JSON_SCAN_VECTOR
/// the end of the run of plain string characters at @a p: the first quote,
/// backslash, control character, or non-ASCII byte, or @a e. The first words
/// are checked with SWAR (one; four with NEON, where the vector mask costs
/// more), as most keys and short values end there; longer runs continue 16
/// bytes at a time.
JSON_HEDLEY_ALWAYS_INLINE const unsigned char* plain_run_end(const unsigned char* p, const unsigned char* e) noexcept
{
constexpr int prefix_words = JSON_SCAN_NEON ? 4 : 1;
for (int words = prefix_words; words > 0 && e - p >= 8; --words)
{
const std::uint64_t special = swar_string_special(read_eight_bytes(p));
if (special != 0)
{
return p + (static_cast<std::size_t>(count_trailing_zeros(special)) / 8);
}
p += 8;
}
// the vector loop only for 16 bytes or more: the serializer passes a
// string without its closing quote, so most of its runs end at e, and a
// call of the out-of-line NEON loop for a shorter rest made dump() up to
// 10% slower
if (e - p >= 16)
{
p = vector_plain_run_long(p, e);
if (e - p >= 16)
{
return p; // the vector loop stopped at a special byte
}
}
return p + find_string_special(p, static_cast<std::size_t>(e - p));
}
#endif
// Computation of the bulk run length: the number of leading bytes in
// [data, data+n) that are ordinary ASCII or complete well-formed UTF-8
// sequences, stopping before the first byte that needs individual handling (the // sequences, stopping before the first byte that needs individual handling (the
// closing quote, an escape, a control character, or an ill-formed/truncated // closing quote, an escape, a control character, or an ill-formed/truncated
// sequence). ASCII is skipped 16 bytes at a time with SSE2 and 8 bytes at a // sequence). ASCII is skipped 8 bytes at a time.
// time otherwise; runs of non-ASCII text use the vector UTF-8 check where inline std::size_t scalar_string_bulk_run(const unsigned char* data, std::size_t n) noexcept
// available.
JSON_HEDLEY_ALWAYS_INLINE std::size_t scalar_string_bulk_run(const unsigned char* data, std::size_t n) noexcept
{ {
const unsigned char* const e = data + n; std::size_t pos = 0;
const unsigned char* p = data; while (pos < n)
while (p != e)
{ {
#if JSON_SCAN_VECTOR pos += find_string_special(data + pos, n - pos);
p = plain_run_end(p, e); if (pos >= n || data[pos] < 0x80u)
#else
p += find_string_special(p, static_cast<std::size_t>(e - p));
#endif
if (p == e || *p < 0x80u)
{ {
break; // end of buffer, or a quote/escape/control byte break; // end of buffer, or a quote/escape/control byte
} }
#if JSON_SCAN_VECTOR_UTF8
#if JSON_SCAN_SSSE3_DISPATCH
if (JSON_HEDLEY_LIKELY(cpu_has_ssse3()))
#endif
{
return static_cast<std::size_t>(scan_string_vector(p, e) - data);
}
#endif
#if !JSON_SCAN_VECTOR_UTF8 || JSON_SCAN_SSSE3_DISPATCH
// a run of multi-byte sequences (e.g. CJK text) is validated sequence // a run of multi-byte sequences (e.g. CJK text) is validated sequence
// by sequence without searching for the next special byte in between // by sequence without searching for the next special byte in between
do do
{ {
const std::size_t seq = validate_one_utf8(p, static_cast<std::size_t>(e - p)); const std::size_t seq = validate_one_utf8(data + pos, n - pos);
if (seq == 0) if (seq == 0)
{ {
return static_cast<std::size_t>(p - data); // ill-formed or truncated: let the byte path diagnose it return pos; // ill-formed or truncated: let the byte path diagnose it
} }
p += seq; pos += seq;
} }
while (p != e && *p >= 0x80u); while (pos < n && data[pos] >= 0x80u);
#endif
} }
return static_cast<std::size_t>(p - data); return pos;
} }
#if defined(JSON_USE_SIMDUTF) && defined(JSON_HAS_CPP_17) #if defined(JSON_USE_SIMDUTF) && defined(JSON_HAS_CPP_17)
@@ -413,7 +304,7 @@ inline std::size_t find_string_delimiter(const unsigned char* data, std::size_t
// next delimiter is validated in one shot by simdutf; on the rare failure the // next delimiter is validated in one shot by simdutf; on the rare failure the
// scalar helper recomputes the exact valid prefix so the byte path still // scalar helper recomputes the exact valid prefix so the byte path still
// produces the precise diagnostic. Without it, the pure scalar path is used. // produces the precise diagnostic. Without it, the pure scalar path is used.
JSON_HEDLEY_ALWAYS_INLINE std::size_t string_bulk_run(const unsigned char* data, std::size_t n) noexcept inline std::size_t string_bulk_run(const unsigned char* data, std::size_t n) noexcept
{ {
#if defined(JSON_USE_SIMDUTF) && defined(JSON_HAS_CPP_17) #if defined(JSON_USE_SIMDUTF) && defined(JSON_HAS_CPP_17)
const std::size_t run = find_string_delimiter(data, n); const std::size_t run = find_string_delimiter(data, n);
@@ -425,42 +316,6 @@ JSON_HEDLEY_ALWAYS_INLINE std::size_t string_bulk_run(const unsigned char* data,
return scalar_string_bulk_run(data, n); return scalar_string_bulk_run(data, n);
} }
#if JSON_SCAN_NEON
/// scalar_string_bulk_run() from @a start on, out of line (see
/// lexer_string_bulk_run())
JSON_HEDLEY_NEVER_INLINE inline std::size_t string_bulk_run_from(const unsigned char* data, std::size_t start, std::size_t n) noexcept
{
return start + scalar_string_bulk_run(data + start, n - start);
}
#endif
// string_bulk_run() for the lexer. With NEON, only the first four words are
// checked inline (most keys and short values end there), and longer runs and
// non-ASCII text continue out of line. The lexer's scan function runs once per
// token, and the vector code inlined into it needs more registers, which every
// token pays for: 5% of json::accept() on jeopardy.json with Apple Clang on
// Apple M1, even though no string of that file reaches the vector code. The
// serializer keeps string_bulk_run() inline, which is faster there.
JSON_HEDLEY_ALWAYS_INLINE std::size_t lexer_string_bulk_run(const unsigned char* data, std::size_t n) noexcept
{
#if JSON_SCAN_NEON && !(defined(JSON_USE_SIMDUTF) && defined(JSON_HAS_CPP_17))
std::size_t i = 0;
for (int words = 4; words > 0 && n - i >= 8; --words)
{
const std::uint64_t special = swar_string_special(read_eight_bytes(data + i));
if (special != 0)
{
const std::size_t pos = i + (static_cast<std::size_t>(count_trailing_zeros(special)) / 8);
return data[pos] < 0x80u ? pos : string_bulk_run_from(data, pos, n);
}
i += 8;
}
return string_bulk_run_from(data, i, n);
#else
return string_bulk_run(data, n);
#endif
}
// Decode the 4 hex digits at [data, data+4) - the digits following a `\u` // Decode the 4 hex digits at [data, data+4) - the digits following a `\u`
// escape - into a codepoint 0x0000..0xFFFF via one table lookup per byte // escape - into a codepoint 0x0000..0xFFFF via one table lookup per byte
// (after yyjson's read_hex_u16), or return -1 if any of the 4 bytes is not a // (after yyjson's read_hex_u16), or return -1 if any of the 4 bytes is not a
@@ -680,12 +680,12 @@ class iter_impl // NOLINT(cppcoreguidelines-special-member-functions,hicpp-speci
{ {
JSON_ASSERT(m_object != nullptr); JSON_ASSERT(m_object != nullptr);
if (JSON_HEDLEY_LIKELY(m_object->is_object())) if (JSON_HEDLEY_UNLIKELY(!m_object->is_object()))
{ {
return m_it.object_iterator->first; throw_literal<invalid_iterator>(exception_id::iterator_key_not_object, "cannot use key() for non-object iterators", m_object);
} }
throw_literal<invalid_iterator>(exception_id::iterator_key_not_object, "cannot use key() for non-object iterators", m_object); return m_it.object_iterator->first;
} }
/*! /*!
+4 -3
View File
@@ -11,6 +11,7 @@
#include <algorithm> // all_of #include <algorithm> // all_of
#include <cctype> // isdigit #include <cctype> // isdigit
#include <cerrno> // errno, ERANGE #include <cerrno> // errno, ERANGE
#include <cstddef> // size_t
#include <cstdlib> // strtoull #include <cstdlib> // strtoull
#ifndef JSON_NO_IO #ifndef JSON_NO_IO
#include <iosfwd> // ostream #include <iosfwd> // ostream
@@ -334,7 +335,7 @@ class json_pointer
{ {
break; // not a plain number: let strtoull decide break; // not a plain number: let strtoull decide
} }
acc = static_cast<size_type>(acc * 10 + static_cast<size_type>(c - '0')); acc = static_cast<size_type>((acc * 10) + static_cast<size_type>(c - '0'));
if (JSON_HEDLEY_UNLIKELY(i == s.size() - 1)) if (JSON_HEDLEY_UNLIKELY(i == s.size() - 1))
{ {
idx = acc; idx = acc;
@@ -529,10 +530,10 @@ class json_pointer
static std::size_t hash_of(std::size_t parent, const string_t& token) noexcept static std::size_t hash_of(std::size_t parent, const string_t& token) noexcept
{ {
std::size_t h = parent * static_cast<std::size_t>(0x9E3779B97F4A7C15ULL); std::size_t h = parent * static_cast<std::size_t>(0x9E3779B97F4A7C15ULL & (std::numeric_limits<std::size_t>::max)());
for (std::size_t i = 0; i < token.size(); ++i) for (std::size_t i = 0; i < token.size(); ++i)
{ {
h = (h ^ static_cast<unsigned char>(token[i])) * static_cast<std::size_t>(0x100000001B3ULL); h = (h ^ static_cast<unsigned char>(token[i])) * static_cast<std::size_t>(0x100000001B3ULL & (std::numeric_limits<std::size_t>::max)());
} }
return h ^ (h >> 29u); return h ^ (h >> 29u);
} }
-8
View File
@@ -895,11 +895,3 @@
#ifndef JSON_DISABLE_TUPLE_REFERENCE_CONVERSION #ifndef JSON_DISABLE_TUPLE_REFERENCE_CONVERSION
#define JSON_DISABLE_TUPLE_REFERENCE_CONVERSION 0 #define JSON_DISABLE_TUPLE_REFERENCE_CONVERSION 0
#endif #endif
#ifndef JSON_DISABLE_STREAM_BULK_SCAN
#define JSON_DISABLE_STREAM_BULK_SCAN 0
#endif
#ifndef JSON_DISABLE_UNLOCKED_FILE_IO
#define JSON_DISABLE_UNLOCKED_FILE_IO 0
#endif
+2 -9
View File
@@ -21,18 +21,9 @@
#undef JSON_NO_UNIQUE_ADDRESS #undef JSON_NO_UNIQUE_ADDRESS
#undef JSON_DISABLE_ENUM_SERIALIZATION #undef JSON_DISABLE_ENUM_SERIALIZATION
#undef JSON_DISABLE_TUPLE_REFERENCE_CONVERSION #undef JSON_DISABLE_TUPLE_REFERENCE_CONVERSION
#undef JSON_DISABLE_STREAM_BULK_SCAN
#undef JSON_DISABLE_UNLOCKED_FILE_IO
#undef JSON_UNLOCKED_FILE_IO #undef JSON_UNLOCKED_FILE_IO
#undef JSON_DTOA_SSE2 #undef JSON_DTOA_SSE2
#undef JSON_DTOA_NEON #undef JSON_DTOA_NEON
#undef JSON_SCAN_NEON
#undef JSON_SCAN_SSE2
#undef JSON_SCAN_SSSE3
#undef JSON_SCAN_SSSE3_DISPATCH
#undef JSON_SCAN_SSSE3_TARGET
#undef JSON_SCAN_VECTOR
#undef JSON_SCAN_VECTOR_UTF8
#ifndef JSON_TEST_KEEP_MACROS #ifndef JSON_TEST_KEEP_MACROS
#undef JSON_CATCH #undef JSON_CATCH
@@ -57,6 +48,8 @@
#undef JSON_STRICT_BINARY_UTF8 #undef JSON_STRICT_BINARY_UTF8
#undef JSON_DELETE_DEPRECATED_FUNCTIONS #undef JSON_DELETE_DEPRECATED_FUNCTIONS
#undef JSON_USE_OBJECTS_FOR_ENUM_KEYED_MAPS #undef JSON_USE_OBJECTS_FOR_ENUM_KEYED_MAPS
#undef JSON_DISABLE_STREAM_BULK_SCAN
#undef JSON_DISABLE_UNLOCKED_FILE_IO
#endif #endif
#include <nlohmann/thirdparty/hedley/hedley_undef.hpp> #include <nlohmann/thirdparty/hedley/hedley_undef.hpp>
+51 -49
View File
@@ -29,6 +29,7 @@
#endif // JSON_NO_IO #endif // JSON_NO_IO
#include <iterator> // make_move_iterator, random_access_iterator_tag #include <iterator> // make_move_iterator, random_access_iterator_tag
#include <limits> // numeric_limits #include <limits> // numeric_limits
#include <map> // map
#include <memory> // unique_ptr #include <memory> // unique_ptr
#include <set> // swap, operator!= #include <set> // swap, operator!=
#include <stdexcept> // length_error #include <stdexcept> // length_error
@@ -754,26 +755,20 @@ private:
// one level deep); stops at the first structured child it meets // one level deep); stops at the first structured child it meets
static bool all_children_leaves(const basic_json& v) noexcept static bool all_children_leaves(const basic_json& v) noexcept
{ {
// array_t only needs forward iterators (see custom_array_type)
if (v.m_data.m_type == value_t::array) if (v.m_data.m_type == value_t::array)
{ {
const auto& a = *v.m_data.m_value.array; const auto& a = *v.m_data.m_value.array;
for (auto it = a.rbegin(); it != a.rend(); ++it) return std::all_of(a.begin(), a.end(), [](const basic_json & el)
{ {
if (!has_no_children(*it)) return has_no_children(el);
{ });
return false;
}
}
return true;
} }
for (const auto& el : *v.m_data.m_value.object) const auto& o = *v.m_data.m_value.object;
return std::all_of(o.begin(), o.end(), [](const typename object_t::value_type & el)
{ {
if (!has_no_children(el.second)) return has_no_children(el.second);
{ });
return false;
}
}
return true;
} }
// The walk in destroy_container() may take the children of a // The walk in destroy_container() may take the children of a
@@ -1230,7 +1225,7 @@ public:
} }
template < typename O = object_t, detail::enable_if_t < detail::has_stable_elements<O>::value, int > = 0 > template < typename O = object_t, detail::enable_if_t < detail::has_stable_elements<O>::value, int > = 0 >
static constexpr std::size_t object_capacity() noexcept std::size_t object_capacity() const noexcept
{ {
return 0; return 0;
} }
@@ -1242,7 +1237,7 @@ public:
} }
template < typename O = object_t, detail::enable_if_t < !detail::has_stable_elements<O>::value && !detail::is_nlohmann_ordered_map<O>::value, int > = 0 > template < typename O = object_t, detail::enable_if_t < !detail::has_stable_elements<O>::value && !detail::is_nlohmann_ordered_map<O>::value, int > = 0 >
static constexpr std::size_t object_capacity() noexcept std::size_t object_capacity() const noexcept
{ {
return detail::unknown_size(); return detail::unknown_size();
} }
@@ -3043,18 +3038,21 @@ public:
JSON_HEDLEY_NO_RETURN JSON_HEDLEY_NEVER_INLINE JSON_HEDLEY_NO_RETURN JSON_HEDLEY_NEVER_INLINE
static void throw_iterator_range_of_null(const basic_json* obj) static void throw_iterator_range_of_null(const basic_json* obj)
{ {
static_cast<void>(obj); // unused when JSON_NOEXCEPTION is defined
JSON_THROW(invalid_iterator::create(detail::exception_id::iterator_range_of_null, detail::concat("cannot construct with iterators from ", obj->type_name()), obj)); JSON_THROW(invalid_iterator::create(detail::exception_id::iterator_range_of_null, detail::concat("cannot construct with iterators from ", obj->type_name()), obj));
} }
JSON_HEDLEY_NO_RETURN JSON_HEDLEY_NEVER_INLINE JSON_HEDLEY_NO_RETURN JSON_HEDLEY_NEVER_INLINE
static void throw_incompatible_reference_type(const basic_json& obj) static void throw_incompatible_reference_type(const basic_json& obj)
{ {
static_cast<void>(obj); // unused when JSON_NOEXCEPTION is defined
JSON_THROW(type_error::create(detail::exception_id::incompatible_reference_type, detail::concat("incompatible ReferenceType for get_ref, actual type is ", obj.type_name()), &obj)); JSON_THROW(type_error::create(detail::exception_id::incompatible_reference_type, detail::concat("incompatible ReferenceType for get_ref, actual type is ", obj.type_name()), &obj));
} }
JSON_HEDLEY_NO_RETURN JSON_HEDLEY_NEVER_INLINE JSON_HEDLEY_NO_RETURN JSON_HEDLEY_NEVER_INLINE
static void throw_array_index_exceeds_size_type(size_type idx) static void throw_array_index_exceeds_size_type(size_type idx)
{ {
static_cast<void>(idx); // unused when JSON_NOEXCEPTION is defined
JSON_THROW(std::length_error(detail::concat("array index ", std::to_string(idx), " exceeds size_type"))); JSON_THROW(std::length_error(detail::concat("array index ", std::to_string(idx), " exceeds size_type")));
} }
@@ -3062,6 +3060,8 @@ public:
JSON_HEDLEY_NO_RETURN JSON_HEDLEY_NEVER_INLINE JSON_HEDLEY_NO_RETURN JSON_HEDLEY_NEVER_INLINE
static void throw_array_index_out_of_range(size_type idx, const basic_json& j) static void throw_array_index_out_of_range(size_type idx, const basic_json& j)
{ {
static_cast<void>(idx); // unused when JSON_NOEXCEPTION is defined
static_cast<void>(j);
JSON_THROW(out_of_range::create(detail::exception_id::array_index_out_of_range, detail::concat("array index ", std::to_string(idx), " is out of range"), &j)); JSON_THROW(out_of_range::create(detail::exception_id::array_index_out_of_range, detail::concat("array index ", std::to_string(idx), " is out of range"), &j));
} }
@@ -3070,6 +3070,8 @@ public:
JSON_HEDLEY_NO_RETURN JSON_HEDLEY_NEVER_INLINE JSON_HEDLEY_NO_RETURN JSON_HEDLEY_NEVER_INLINE
static void throw_key_not_found(const KeyType& key, const basic_json& j) static void throw_key_not_found(const KeyType& key, const basic_json& j)
{ {
static_cast<void>(key); // unused when JSON_NOEXCEPTION is defined
static_cast<void>(j);
JSON_THROW(out_of_range::create(detail::exception_id::key_not_found, detail::concat("key '", key, "' not found"), &j)); JSON_THROW(out_of_range::create(detail::exception_id::key_not_found, detail::concat("key '", key, "' not found"), &j));
} }
@@ -4893,19 +4895,19 @@ public:
iterator insert(const_iterator pos, const basic_json& val) // NOLINT(performance-unnecessary-value-param) iterator insert(const_iterator pos, const basic_json& val) // NOLINT(performance-unnecessary-value-param)
{ {
// insert only works for arrays // insert only works for arrays
if (JSON_HEDLEY_LIKELY(is_array())) if (JSON_HEDLEY_UNLIKELY(!is_array()))
{ {
// check if iterator pos fits to this JSON value detail::throw_cannot_use_with(detail::exception_id::insert_wrong_type, "insert()", *this);
if (JSON_HEDLEY_UNLIKELY(pos.m_object != this))
{
detail::throw_literal<invalid_iterator>(detail::exception_id::iterator_from_other_value, "iterator does not fit current value", this);
}
// insert to array and return iterator
return insert_iterator(pos, val);
} }
detail::throw_cannot_use_with(detail::exception_id::insert_wrong_type, "insert()", *this); // check if iterator pos fits to this JSON value
if (JSON_HEDLEY_UNLIKELY(pos.m_object != this))
{
detail::throw_literal<invalid_iterator>(detail::exception_id::iterator_from_other_value, "iterator does not fit current value", this);
}
// insert to array and return iterator
return insert_iterator(pos, val);
} }
/// @brief inserts element into array /// @brief inserts element into array
@@ -4913,21 +4915,21 @@ public:
iterator insert(const_iterator pos, basic_json&& val) // NOLINT(performance-unnecessary-value-param) iterator insert(const_iterator pos, basic_json&& val) // NOLINT(performance-unnecessary-value-param)
{ {
// insert only works for arrays // insert only works for arrays
if (JSON_HEDLEY_LIKELY(is_array())) if (JSON_HEDLEY_UNLIKELY(!is_array()))
{ {
// check if iterator pos fits to this JSON value detail::throw_cannot_use_with(detail::exception_id::insert_wrong_type, "insert()", *this);
if (JSON_HEDLEY_UNLIKELY(pos.m_object != this))
{
detail::throw_literal<invalid_iterator>(detail::exception_id::iterator_from_other_value, "iterator does not fit current value", this);
}
// moving into a local first keeps this safe even if val aliases
// an element of this array
basic_json tmp(std::move(val));
return insert_iterator(pos, std::move(tmp));
} }
detail::throw_cannot_use_with(detail::exception_id::insert_wrong_type, "insert()", *this); // check if iterator pos fits to this JSON value
if (JSON_HEDLEY_UNLIKELY(pos.m_object != this))
{
detail::throw_literal<invalid_iterator>(detail::exception_id::iterator_from_other_value, "iterator does not fit current value", this);
}
// moving into a local first keeps this safe even if val aliases
// an element of this array
basic_json tmp(std::move(val));
return insert_iterator(pos, std::move(tmp));
} }
/// @brief inserts copies of element into array /// @brief inserts copies of element into array
@@ -4935,19 +4937,19 @@ public:
iterator insert(const_iterator pos, size_type cnt, const basic_json& val) // NOLINT(performance-unnecessary-value-param) iterator insert(const_iterator pos, size_type cnt, const basic_json& val) // NOLINT(performance-unnecessary-value-param)
{ {
// insert only works for arrays // insert only works for arrays
if (JSON_HEDLEY_LIKELY(is_array())) if (JSON_HEDLEY_UNLIKELY(!is_array()))
{ {
// check if iterator pos fits to this JSON value detail::throw_cannot_use_with(detail::exception_id::insert_wrong_type, "insert()", *this);
if (JSON_HEDLEY_UNLIKELY(pos.m_object != this))
{
detail::throw_literal<invalid_iterator>(detail::exception_id::iterator_from_other_value, "iterator does not fit current value", this);
}
// insert to array and return iterator
return insert_iterator(pos, cnt, val);
} }
detail::throw_cannot_use_with(detail::exception_id::insert_wrong_type, "insert()", *this); // check if iterator pos fits to this JSON value
if (JSON_HEDLEY_UNLIKELY(pos.m_object != this))
{
detail::throw_literal<invalid_iterator>(detail::exception_id::iterator_from_other_value, "iterator does not fit current value", this);
}
// insert to array and return iterator
return insert_iterator(pos, cnt, val);
} }
/// @brief inserts range of elements into array /// @brief inserts range of elements into array
@@ -5453,7 +5455,7 @@ public:
// inline fast path for two values of the same non-owning scalar type; anything // inline fast path for two values of the same non-owning scalar type; anything
// else falls through to the out-of-line general comparison // else falls through to the out-of-line general comparison
#define JSON_IMPLEMENT_SCALAR_FAST(op, null_result) \ #define JSON_IMPLEMENT_SCALAR_FAST(op, null_result) \
if (lhs.m_data.m_type == rhs.m_data.m_type) \ if (lhs.m_data.m_type == rhs.m_data.m_type) /* NOLINT(readability/braces) */ \
{ \ { \
switch (lhs.m_data.m_type) \ switch (lhs.m_data.m_type) \
{ \ { \
+1
View File
@@ -23,6 +23,7 @@
#include <nlohmann/detail/abi_macros.hpp> #include <nlohmann/detail/abi_macros.hpp>
#include <nlohmann/detail/macro_scope.hpp> #include <nlohmann/detail/macro_scope.hpp>
#include <nlohmann/detail/meta/type_traits.hpp> #include <nlohmann/detail/meta/type_traits.hpp>
#include <nlohmann/thirdparty/hedley/hedley.hpp>
NLOHMANN_JSON_NAMESPACE_BEGIN NLOHMANN_JSON_NAMESPACE_BEGIN
+113664
View File
File diff suppressed because it is too large. Load diff
+197 -651
View File
File diff suppressed because it is too large. Load diff
+26 -4
View File
@@ -71,6 +71,14 @@
#define JSON_USE_OBJECTS_FOR_ENUM_KEYED_MAPS 0 #define JSON_USE_OBJECTS_FOR_ENUM_KEYED_MAPS 0
#endif #endif
#ifndef JSON_DISABLE_STREAM_BULK_SCAN
#define JSON_DISABLE_STREAM_BULK_SCAN 0
#endif
#ifndef JSON_DISABLE_UNLOCKED_FILE_IO
#define JSON_DISABLE_UNLOCKED_FILE_IO 0
#endif
#if JSON_DIAGNOSTICS #if JSON_DIAGNOSTICS
#define NLOHMANN_JSON_ABI_TAG_DIAGNOSTICS _diag #define NLOHMANN_JSON_ABI_TAG_DIAGNOSTICS _diag
#else #else
@@ -119,14 +127,26 @@
#define NLOHMANN_JSON_ABI_TAG_OBJECTS_FOR_ENUM_KEYED_MAPS #define NLOHMANN_JSON_ABI_TAG_OBJECTS_FOR_ENUM_KEYED_MAPS
#endif #endif
#if JSON_DISABLE_STREAM_BULK_SCAN
#define NLOHMANN_JSON_ABI_TAG_DISABLE_STREAM_BULK_SCAN _dsbs
#else
#define NLOHMANN_JSON_ABI_TAG_DISABLE_STREAM_BULK_SCAN
#endif
#if JSON_DISABLE_UNLOCKED_FILE_IO
#define NLOHMANN_JSON_ABI_TAG_DISABLE_UNLOCKED_FILE_IO _dufio
#else
#define NLOHMANN_JSON_ABI_TAG_DISABLE_UNLOCKED_FILE_IO
#endif
#ifndef NLOHMANN_JSON_NAMESPACE_NO_VERSION #ifndef NLOHMANN_JSON_NAMESPACE_NO_VERSION
#define NLOHMANN_JSON_NAMESPACE_NO_VERSION 0 #define NLOHMANN_JSON_NAMESPACE_NO_VERSION 0
#endif #endif
// Construct the namespace ABI tags component // Construct the namespace ABI tags component
#define NLOHMANN_JSON_ABI_TAGS_CONCAT_EX(a, b, c, d, e, f, g, h) json_abi ## a ## b ## c ## d ## e ## f ## g ## h #define NLOHMANN_JSON_ABI_TAGS_CONCAT_EX(a, b, c, d, e, f, g, h, i, j) json_abi ## a ## b ## c ## d ## e ## f ## g ## h ## i ## j
#define NLOHMANN_JSON_ABI_TAGS_CONCAT(a, b, c, d, e, f, g, h) \ #define NLOHMANN_JSON_ABI_TAGS_CONCAT(a, b, c, d, e, f, g, h, i, j) \
NLOHMANN_JSON_ABI_TAGS_CONCAT_EX(a, b, c, d, e, f, g, h) NLOHMANN_JSON_ABI_TAGS_CONCAT_EX(a, b, c, d, e, f, g, h, i, j)
#define NLOHMANN_JSON_ABI_TAGS \ #define NLOHMANN_JSON_ABI_TAGS \
NLOHMANN_JSON_ABI_TAGS_CONCAT( \ NLOHMANN_JSON_ABI_TAGS_CONCAT( \
@@ -137,7 +157,9 @@
NLOHMANN_JSON_ABI_TAG_PRECISE_STREAM_POSITION, \ NLOHMANN_JSON_ABI_TAG_PRECISE_STREAM_POSITION, \
NLOHMANN_JSON_ABI_TAG_STRICT_NUL_HANDLING, \ NLOHMANN_JSON_ABI_TAG_STRICT_NUL_HANDLING, \
NLOHMANN_JSON_ABI_TAG_STRICT_BINARY_UTF8, \ NLOHMANN_JSON_ABI_TAG_STRICT_BINARY_UTF8, \
NLOHMANN_JSON_ABI_TAG_OBJECTS_FOR_ENUM_KEYED_MAPS) NLOHMANN_JSON_ABI_TAG_OBJECTS_FOR_ENUM_KEYED_MAPS, \
NLOHMANN_JSON_ABI_TAG_DISABLE_STREAM_BULK_SCAN, \
NLOHMANN_JSON_ABI_TAG_DISABLE_UNLOCKED_FILE_IO)
// Construct the namespace version component // Construct the namespace version component
#define NLOHMANN_JSON_NAMESPACE_VERSION_CONCAT_EX(major, minor, patch) \ #define NLOHMANN_JSON_NAMESPACE_VERSION_CONCAT_EX(major, minor, patch) \
+1 -20
View File
@@ -118,7 +118,7 @@ endif()
if (CMAKE_CXX_COMPILER_ID STREQUAL "MSVC") if (CMAKE_CXX_COMPILER_ID STREQUAL "MSVC")
# avoid stack overflow, see https://github.com/nlohmann/json/issues/2955 # avoid stack overflow, see https://github.com/nlohmann/json/issues/2955
json_test_set_test_options("test-bon8;test-cbor;test-msgpack;test-ubjson;test-bjdata;test-binary_formats" LINK_OPTIONS /STACK:4000000) json_test_set_test_options("test-bon8;test-cbor;test-msgpack;test-ubjson;test-bjdata;test-bjdata_round_trip_draft2;test-bjdata_round_trip_draft3;test-binary_formats" LINK_OPTIONS /STACK:4000000)
endif() endif()
# disable exceptions for test-disabled_exceptions # disable exceptions for test-disabled_exceptions
@@ -373,25 +373,6 @@ json_test_add_test_for(src/unit-diagnostic-positions.cpp
MAIN test_main CXX_STANDARDS ${test_cxx_standards} ${test_force} MAIN test_main CXX_STANDARDS ${test_cxx_standards} ${test_force}
) )
# the lexer again with the portable string scanning instead of SSE2/NEON, and on
# x86-64 with SSSE3 compiled in (no check of the CPU)
json_test_set_test_options(test-class_lexer_no_simd
COMPILE_DEFINITIONS JSON_NO_SIMD
)
json_test_add_test_for(src/unit-class_lexer.cpp
NAME test-class_lexer_no_simd
MAIN test_main CXX_STANDARDS ${test_cxx_standards} ${test_force}
)
if(CMAKE_SYSTEM_PROCESSOR MATCHES "^(x86_64|AMD64|amd64)$" AND NOT MSVC)
json_test_set_test_options(test-class_lexer_ssse3
COMPILE_OPTIONS -mssse3
)
json_test_add_test_for(src/unit-class_lexer.cpp
NAME test-class_lexer_ssse3
MAIN test_main CXX_STANDARDS ${test_cxx_standards} ${test_force}
)
endif()
# test reading from a std::istream again without scanning the get area in bulk # test reading from a std::istream again without scanning the get area in bulk
json_test_set_test_options(test-stream-bulk-scan_disabled json_test_set_test_options(test-stream-bulk-scan_disabled
COMPILE_DEFINITIONS JSON_DISABLE_STREAM_BULK_SCAN=1 COMPILE_DEFINITIONS JSON_DISABLE_STREAM_BULK_SCAN=1
+3 -1
View File
@@ -21,7 +21,9 @@ foreach(test default noversion)
target_compile_definitions(abi_config_${test}_all_tags PRIVATE target_compile_definitions(abi_config_${test}_all_tags PRIVATE
JSON_DIAGNOSTICS=1 JSON_DIAGNOSTICS=1
JSON_DIAGNOSTIC_POSITIONS=1 JSON_DIAGNOSTIC_POSITIONS=1
JSON_USE_LEGACY_DISCARDED_VALUE_COMPARISON=1) JSON_USE_LEGACY_DISCARDED_VALUE_COMPARISON=1
JSON_DISABLE_STREAM_BULK_SCAN=1
JSON_DISABLE_UNLOCKED_FILE_IO=1)
target_link_libraries(abi_config_${test}_all_tags PRIVATE abi_compat_main) target_link_libraries(abi_config_${test}_all_tags PRIVATE abi_compat_main)
add_test( add_test(
NAME test-abi_config_${test}_all_tags NAME test-abi_config_${test}_all_tags
+8
View File
@@ -52,6 +52,14 @@ TEST_CASE("default namespace")
expected += "_ekmo"; expected += "_ekmo";
#endif #endif
#if JSON_DISABLE_STREAM_BULK_SCAN
expected += "_dsbs";
#endif
#if JSON_DISABLE_UNLOCKED_FILE_IO
expected += "_dufio";
#endif
expected += "_v" STRINGIZE(NLOHMANN_JSON_VERSION_MAJOR); expected += "_v" STRINGIZE(NLOHMANN_JSON_VERSION_MAJOR);
expected += "_" STRINGIZE(NLOHMANN_JSON_VERSION_MINOR); expected += "_" STRINGIZE(NLOHMANN_JSON_VERSION_MINOR);
expected += "_" STRINGIZE(NLOHMANN_JSON_VERSION_PATCH) "::basic_json"; expected += "_" STRINGIZE(NLOHMANN_JSON_VERSION_PATCH) "::basic_json";
+8
View File
@@ -53,6 +53,14 @@ TEST_CASE("default namespace without version component")
expected += "_ekmo"; expected += "_ekmo";
#endif #endif
#if JSON_DISABLE_STREAM_BULK_SCAN
expected += "_dsbs";
#endif
#if JSON_DISABLE_UNLOCKED_FILE_IO
expected += "_dufio";
#endif
expected += "::basic_json"; expected += "::basic_json";
// fallback for Clang // fallback for Clang
+103
View File
@@ -0,0 +1,103 @@
// __ _____ _____ _____
// __| | __| | | | JSON for Modern C++ (supporting code)
// | | |__ | | | | | | version 3.12.0
// |_____|_____|_____|_|___| https://github.com/nlohmann/json
//
// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
// SPDX-License-Identifier: MIT
#pragma once
#include <cstdint> // uint8_t
#include <vector> // vector
#include "doctest_compatibility.h"
#include <nlohmann/json.hpp>
#include "round_trip_corpus.hpp"
// The BJData round-trip invariants, split by the BJData version that turns a
// corpus value into j1 (unit-bjdata_round_trip_draft2.cpp and
// unit-bjdata_round_trip_draft3.cpp). They were part of unit-bjdata.cpp, but
// took 80 % of its time; as separate tests, both halves run in parallel and
// stay below the ctest timeout under Valgrind.
namespace utils
{
// This checks what the parse_bjdata_fuzzer driver checks (see
// tests/src/fuzzer-parse_bjdata.cpp), so that a regression shows up in CI
// rather than as an OSS-Fuzz report: every value from_bjdata() returns
// (j1) can be serialized with any combination of options, the result can
// be parsed back (j2), and serializing j2 again with the same options
// yields a value-equal result.
//
// Beyond the driver, this also checks that j2 equals j1 and that
// serializing j2 reproduces the exact bytes, both except for values that
// contain a binary value: a binary value is only written as a binary
// value with Draft 3's optimized binary array, and otherwise read back as
// an array of integers, for which the writer may choose different (but
// equally valid) type markers when it is serialized again (see #5494).
//
// Values are compared with dump() rather than operator==, because a NaN
// never compares equal to itself.
//
// Only the corpus values written with @a initial_version become j1; each j1
// is then serialized with every combination of options, both versions
// included.
inline void check_bjdata_round_trip(const nlohmann::json::bjdata_version_t initial_version)
{
using nlohmann::json;
struct options
{
bool use_size;
bool use_type;
json::bjdata_version_t version;
};
const std::vector<options> all_options =
{
{false, false, json::bjdata_version_t::draft2},
{true, false, json::bjdata_version_t::draft2},
{true, true, json::bjdata_version_t::draft2},
{false, false, json::bjdata_version_t::draft3},
{true, false, json::bjdata_version_t::draft3},
{true, true, json::bjdata_version_t::draft3},
};
for (const auto& j0 : utils::round_trip_corpus::values())
{
// turn the corpus value into a value as from_bjdata() returns it
for (const auto& initial : all_options)
{
if (initial.version != initial_version)
{
continue;
}
const json j1 = json::from_bjdata(json::to_bjdata(j0, initial.use_size, initial.use_type, initial.version));
const bool has_binary = utils::round_trip_corpus::contains_binary(j1);
for (const auto& o : all_options)
{
INFO("j1 = " << j1.dump() << ", use_size = " << o.use_size << ", use_type = " << o.use_type
<< ", draft3 = " << (o.version == json::bjdata_version_t::draft3));
const std::vector<std::uint8_t> vec = json::to_bjdata(j1, o.use_size, o.use_type, o.version);
json j2;
// anything the library writes must be parsable by the library
REQUIRE_NOTHROW(j2 = json::from_bjdata(vec));
const std::vector<std::uint8_t> vec2 = json::to_bjdata(j2, o.use_size, o.use_type, o.version);
CHECK(json::from_bjdata(vec2).dump() == j2.dump());
if (!has_binary)
{
CHECK(j2.dump() == j1.dump());
CHECK(vec2 == vec);
}
}
}
}
}
} // namespace utils
-65
View File
@@ -28,7 +28,6 @@ using ordered_json = nlohmann::ordered_json;
#include <fstream> #include <fstream>
#include <set> #include <set>
#include "make_test_data_available.hpp" #include "make_test_data_available.hpp"
#include "round_trip_corpus.hpp"
#include "test_utils.hpp" #include "test_utils.hpp"
#include "sax_countdown.hpp" #include "sax_countdown.hpp"
using utils::SaxCountdown; using utils::SaxCountdown;
@@ -4441,70 +4440,6 @@ TEST_CASE("BJData use_type requires use_size")
} }
} }
TEST_CASE("BJData round-trip invariants")
{
// This checks what the parse_bjdata_fuzzer driver checks (see
// tests/src/fuzzer-parse_bjdata.cpp), so that a regression shows up in CI
// rather than as an OSS-Fuzz report: every value from_bjdata() returns
// (j1) can be serialized with any combination of options, the result can
// be parsed back (j2), and serializing j2 again with the same options
// yields a value-equal result.
//
// Beyond the driver, this also checks that j2 equals j1 and that
// serializing j2 reproduces the exact bytes, both except for values that
// contain a binary value: a binary value is only written as a binary
// value with Draft 3's optimized binary array, and otherwise read back as
// an array of integers, for which the writer may choose different (but
// equally valid) type markers when it is serialized again (see #5494).
//
// Values are compared with dump() rather than operator==, because a NaN
// never compares equal to itself.
struct options
{
bool use_size;
bool use_type;
json::bjdata_version_t version;
};
const std::vector<options> all_options =
{
{false, false, json::bjdata_version_t::draft2},
{true, false, json::bjdata_version_t::draft2},
{true, true, json::bjdata_version_t::draft2},
{false, false, json::bjdata_version_t::draft3},
{true, false, json::bjdata_version_t::draft3},
{true, true, json::bjdata_version_t::draft3},
};
for (const auto& j0 : utils::round_trip_corpus::values())
{
// turn the corpus value into a value as from_bjdata() returns it
for (const auto& initial : all_options)
{
const json j1 = json::from_bjdata(json::to_bjdata(j0, initial.use_size, initial.use_type, initial.version));
const bool has_binary = utils::round_trip_corpus::contains_binary(j1);
for (const auto& o : all_options)
{
INFO("j1 = " << j1.dump() << ", use_size = " << o.use_size << ", use_type = " << o.use_type
<< ", draft3 = " << (o.version == json::bjdata_version_t::draft3));
const std::vector<std::uint8_t> vec = json::to_bjdata(j1, o.use_size, o.use_type, o.version);
json j2;
// anything the library writes must be parsable by the library
REQUIRE_NOTHROW(j2 = json::from_bjdata(vec));
const std::vector<std::uint8_t> vec2 = json::to_bjdata(j2, o.use_size, o.use_type, o.version);
CHECK(json::from_bjdata(vec2).dump() == j2.dump());
if (!has_binary)
{
CHECK(j2.dump() == j1.dump());
CHECK(vec2 == vec);
}
}
}
}
}
TEST_CASE("BJData round trip of a binary value is value-stable, not byte-stable") TEST_CASE("BJData round trip of a binary value is value-stable, not byte-stable")
{ {
// OSS-Fuzz issue 474480402: a Draft 3 optimized binary array is read as a // OSS-Fuzz issue 474480402: a Draft 3 optimized binary array is read as a
@@ -0,0 +1,22 @@
// __ _____ _____ _____
// __| | __| | | | JSON for Modern C++ (supporting code)
// | | |__ | | | | | | version 3.12.0
// |_____|_____|_____|_|___| https://github.com/nlohmann/json
//
// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
// SPDX-License-Identifier: MIT
#include "doctest_compatibility.h"
#include <nlohmann/json.hpp>
using nlohmann::json;
#include "bjdata_round_trip.hpp"
// split from unit-bjdata.cpp, see bjdata_round_trip.hpp; the other half is in
// unit-bjdata_round_trip_draft3.cpp
TEST_CASE("BJData round-trip invariants (j1 written with Draft 2)")
{
utils::check_bjdata_round_trip(json::bjdata_version_t::draft2);
}
@@ -0,0 +1,22 @@
// __ _____ _____ _____
// __| | __| | | | JSON for Modern C++ (supporting code)
// | | |__ | | | | | | version 3.12.0
// |_____|_____|_____|_|___| https://github.com/nlohmann/json
//
// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
// SPDX-License-Identifier: MIT
#include "doctest_compatibility.h"
#include <nlohmann/json.hpp>
using nlohmann::json;
#include "bjdata_round_trip.hpp"
// split from unit-bjdata.cpp, see bjdata_round_trip.hpp; the other half is in
// unit-bjdata_round_trip_draft2.cpp
TEST_CASE("BJData round-trip invariants (j1 written with Draft 3)")
{
utils::check_bjdata_round_trip(json::bjdata_version_t::draft3);
}
+3 -125
View File
@@ -13,7 +13,6 @@
using nlohmann::json; using nlohmann::json;
#include <array> // array #include <array> // array
#include <cctype> // toupper
#include <cstdint> // uint32_t, uint64_t #include <cstdint> // uint32_t, uint64_t
#include <cstdio> // snprintf #include <cstdio> // snprintf
#include <cmath> // isfinite #include <cmath> // isfinite
@@ -336,6 +335,8 @@ TEST_CASE("lexer number fast path")
} }
} }
#if !defined(JSON_NOEXCEPTION)
// this section reports parse errors, which abort when exceptions are off
SECTION("digits accumulated while scanning") SECTION("digits accumulated while scanning")
{ {
// The fast path accumulates the digits while it validates them, eight // The fast path accumulates the digits while it validates them, eight
@@ -414,6 +415,7 @@ TEST_CASE("lexer number fast path")
} }
} }
} }
#endif
SECTION("integers that do not fit a narrower type become floats") SECTION("integers that do not fit a narrower type become floats")
{ {
@@ -597,63 +599,6 @@ TEST_CASE("lexer number fast path")
#endif #endif
} }
TEST_CASE("lexer literal fast path")
{
// For contiguous input, the rest of a literal is compared at once, and a
// mismatch is left to the byte path, which reports the error at the
// character that differs. Every truncation and every wrong character of
// each literal, at the end of the input or followed by more text, must
// give the byte path's result (a std::list is read one character at a
// time).
const auto outcome = [](const std::string & doc, bool byte_path) -> std::string
{
JSON_TRY
{
const std::list<char> chars(doc.begin(), doc.end());
return (byte_path ? json::parse(chars.begin(), chars.end()) : json::parse(doc)).dump();
}
JSON_CATCH (const json::exception & e)
{
return e.what();
}
};
std::vector<std::string> mismatches;
for (const std::string literal :
{"true", "false", "null"
})
{
std::vector<std::string> variants;
for (std::size_t length = 1; length <= literal.size(); ++length)
{
variants.push_back(literal.substr(0, length));
}
for (std::size_t i = 1; i < literal.size(); ++i)
{
std::string changed = literal;
changed[i] = 'X';
variants.push_back(changed);
changed[i] = static_cast<char>(std::toupper(static_cast<unsigned char>(literal[i])));
variants.push_back(changed);
}
for (const auto& variant : variants)
{
for (const std::string& doc :
{
variant, "[" + variant + "]", "[" + variant, variant + " ", variant + "1", "{\"k\":" + variant + "}"
})
{
if (outcome(doc, false) != outcome(doc, true))
{
mismatches.push_back(doc);
}
}
}
}
CAPTURE(mismatches)
CHECK(mismatches.empty());
}
TEST_CASE("lexer string fast path") TEST_CASE("lexer string fast path")
{ {
// Build a byte string from explicit values: a hex escape in a string // Build a byte string from explicit values: a hex escape in a string
@@ -774,73 +719,6 @@ TEST_CASE("lexer string fast path")
CAPTURE(mismatches) CAPTURE(mismatches)
CHECK(mismatches.empty()); CHECK(mismatches.empty());
} }
SECTION("special bytes at every offset of the vector blocks")
{
// With SSE2 or NEON, runs are scanned 16 bytes at a time (after one
// SWAR word, or four with NEON), and runs of non-ASCII text are
// checked 16 bytes at a time as well. Place every kind of byte that
// ends a run at each offset across several blocks, after ASCII and
// after CJK text, so sequences also straddle the block boundaries.
// The reference is a std::list, which has no contiguous storage and
// is therefore read one character at a time.
const auto byte_path = [](const std::string & doc) -> std::string
{
const std::list<char> chars(doc.begin(), doc.end());
JSON_TRY
{
return json::parse(chars.begin(), chars.end()).dump();
}
JSON_CATCH (const json::exception & e)
{
return e.what();
}
};
const std::vector<std::string> specials =
{
"\"", "\\", "\\u00e9", bytes({0x01}), bytes({0x1F}), bytes({0x7F}),
bytes({0xC3, 0xA9}), bytes({0xE4, 0xB8, 0xAD}), bytes({0xF0, 0x9F, 0x98, 0x80}),
bytes({0xFF}), bytes({0xC3}), bytes({0xE4, 0xB8}), bytes({0xED, 0xA0, 0x80}), bytes({0xC0, 0x80})
};
const std::string cjk = bytes({0xE4, 0xB8, 0xAD}); // U+4E2D
std::vector<std::string> mismatches;
for (std::size_t offset = 0; offset <= 80; ++offset)
{
std::string text;
while (text.size() < offset)
{
text += cjk;
}
for (const auto& prefix :
{
std::string(offset, 'a'), text
})
{
for (const auto& special : specials)
{
// the closing quote right behind, or in a later block
for (const int tail :
{
0, 1, 3, 20
})
{
std::string doc = "[\"";
doc += prefix;
doc += special;
doc.append(static_cast<std::size_t>(tail), 'b');
doc += "\"]";
if (outcome(doc, false) != byte_path(doc))
{
mismatches.push_back(doc);
}
}
}
}
}
CAPTURE(mismatches)
CHECK(mismatches.empty());
}
#endif #endif
// json::accept() never throws, so the ranges stay covered without exceptions // json::accept() never throws, so the ranges stay covered without exceptions
+7 -7
View File
@@ -198,11 +198,11 @@ TEST_CASE("copy constructor and destructor across value types")
const json src_uint = 7u; const json src_uint = 7u;
const json src_float = 2.5; const json src_float = 2.5;
json c_null(src_null); json c_null(src_null); // NOLINT(performance-unnecessary-copy-initialization): tests the copy constructor
json c_bool(src_bool); json c_bool(src_bool); // NOLINT(performance-unnecessary-copy-initialization): tests the copy constructor
json c_int(src_int); json c_int(src_int); // NOLINT(performance-unnecessary-copy-initialization): tests the copy constructor
json c_uint(src_uint); json c_uint(src_uint); // NOLINT(performance-unnecessary-copy-initialization): tests the copy constructor
json c_float(src_float); json c_float(src_float); // NOLINT(performance-unnecessary-copy-initialization): tests the copy constructor
CHECK(c_null.is_null()); CHECK(c_null.is_null());
CHECK(c_bool == true); CHECK(c_bool == true);
@@ -300,12 +300,12 @@ TEST_CASE("copy constructor and destructor across value types")
REQUIRE(v.size() == 70); REQUIRE(v.size() == 70);
std::vector<json> copy(v); std::vector<json> copy(v);
CHECK(copy == v); CHECK((copy == v));
// mutating the copy leaves the original alone // mutating the copy leaves the original alone
copy[5] = "changed"; copy[5] = "changed";
copy[6].push_back(1); copy[6].push_back(1);
CHECK(copy != v); CHECK((copy != v));
CHECK(v[5] == std::string(5, 'x')); CHECK(v[5] == std::string(5, 'x'));
CHECK(v[6].size() == 3); CHECK(v[6].size() == 3);
+2 -2
View File
@@ -817,8 +817,8 @@ template<class Key, class T, class IgnoredLess = std::less<Key>,
struct relocating_map : nlohmann::ordered_map<Key, T, IgnoredLess, Allocator> struct relocating_map : nlohmann::ordered_map<Key, T, IgnoredLess, Allocator>
{ {
using base_type = nlohmann::ordered_map<Key, T, IgnoredLess, Allocator>; using base_type = nlohmann::ordered_map<Key, T, IgnoredLess, Allocator>;
using typename base_type::iterator; using iterator = typename base_type::iterator;
using typename base_type::Container; using Container = typename base_type::Container;
using base_type::base_type; using base_type::base_type;
template<typename... Args> template<typename... Args>
+3
View File
@@ -170,6 +170,7 @@ TEST_CASE("parsing from a FILE*")
check_unlocked(f.get()); check_unlocked(f.get());
} }
#if !defined(JSON_NOEXCEPTION)
SECTION("the file is unlocked after a parse error") SECTION("the file is unlocked after a parse error")
{ {
const file_ptr f = open_with(name, "[1, 2, x]"); const file_ptr f = open_with(name, "[1, 2, x]");
@@ -186,6 +187,8 @@ TEST_CASE("parsing from a FILE*")
CHECK(std::string(rest.data()) == "]"); CHECK(std::string(rest.data()) == "]");
} }
#endif // !defined(JSON_NOEXCEPTION)
SECTION("the file can be parsed again and read with stdio afterwards") SECTION("the file can be parsed again and read with stdio afterwards")
{ {
const file_ptr f = open_with(name, "[1] [2] tail"); const file_ptr f = open_with(name, "[1] [2] tail");
+5 -4
View File
@@ -1637,6 +1637,7 @@ TEST_CASE_TEMPLATE("reverse iteration and member access", Json, nlohmann::json,
{ {
Json j = {1, 2, 3, 4}; Json j = {1, 2, 3, 4};
const std::vector<int> reversed = {4, 3, 2, 1};
auto it = j.end(); auto it = j.end();
std::vector<int> seen; std::vector<int> seen;
while (it != j.begin()) while (it != j.begin())
@@ -1644,7 +1645,7 @@ TEST_CASE_TEMPLATE("reverse iteration and member access", Json, nlohmann::json,
--it; --it;
seen.push_back(it->template get<int>()); seen.push_back(it->template get<int>());
} }
CHECK(seen == std::vector<int> {4, 3, 2, 1}); CHECK(seen == reversed);
auto it2 = j.end(); auto it2 = j.end();
auto old = it2--; auto old = it2--;
@@ -1656,14 +1657,14 @@ TEST_CASE_TEMPLATE("reverse iteration and member access", Json, nlohmann::json,
{ {
seen.push_back(r->template get<int>()); seen.push_back(r->template get<int>());
} }
CHECK(seen == std::vector<int> {4, 3, 2, 1}); CHECK(seen == reversed);
seen.clear(); seen.clear();
for (auto r = j.crbegin(); r != j.crend(); ++r) for (auto r = j.crbegin(); r != j.crend(); ++r)
{ {
seen.push_back(*r); seen.push_back(r->template get<int>());
} }
CHECK(seen == std::vector<int> {4, 3, 2, 1}); CHECK(seen == reversed);
} }
SECTION("object: -- and rbegin/rend") SECTION("object: -- and rbegin/rend")
+2 -2
View File
@@ -1230,11 +1230,11 @@ TEST_CASE("JSON pointer array index tokens match strtoull semantics")
SECTION("random tokens") SECTION("random tokens")
{ {
std::mt19937 gen(42); std::mt19937 gen(42); // NOLINT(cert-msc32-c,cert-msc51-cpp,bugprone-random-generator-seed): reproducible
const std::string alphabet = "0123456789012345678901234567890123456789-+ ex"; const std::string alphabet = "0123456789012345678901234567890123456789-+ ex";
for (int i = 0; i < 5000; ++i) for (int i = 0; i < 5000; ++i)
{ {
std::string token(static_cast<std::size_t>(gen() % 23), '0'); std::string token(gen() % 23, '0');
for (auto& c : token) for (auto& c : token)
{ {
c = alphabet[gen() % alphabet.size()]; c = alphabet[gen() % alphabet.size()];
-401
View File
@@ -13,8 +13,6 @@ using nlohmann::json;
#include <algorithm> #include <algorithm>
#include <string> #include <string>
#include <utility>
#include <vector>
TEST_CASE("tests on very large JSONs") TEST_CASE("tests on very large JSONs")
{ {
@@ -55,24 +53,6 @@ const json* innermost_value(const json& j, std::size_t& depth)
return current; return current;
} }
// The text of a value nested depth levels deep around the number 0. Level i is
// an array if pattern[i % pattern.size()] is '[', and otherwise an object with
// the single member "a", which every object type enumerates in the same order.
std::string nested_text(std::size_t depth, const std::string& pattern)
{
std::string text;
std::string closing;
for (std::size_t i = 0; i < depth; ++i)
{
const bool array = pattern[i % pattern.size()] == '[';
text += array ? "[" : "{\"a\":";
closing += array ? ']' : '}';
}
text += '0';
text.append(closing.rbegin(), closing.rend());
return text;
}
} // namespace } // namespace
TEST_CASE("tests on deeply nested JSONs") TEST_CASE("tests on deeply nested JSONs")
@@ -244,385 +224,4 @@ TEST_CASE("tests on deeply nested JSONs")
CHECK(*innermost_value(j, unused) == 0); CHECK(*innermost_value(j, unused) == 0);
} }
} }
SECTION("issue #5650 - stack overflow converting between specializations")
{
const std::vector<std::string> patterns = {"[", "{", "[{"};
SECTION("json to ordered_json")
{
for (const auto& pattern : patterns)
{
CAPTURE(pattern)
const std::string text = nested_text(depth, pattern);
const json j = json::parse(text);
const nlohmann::ordered_json converted = j;
CHECK(converted.dump() == text);
}
}
SECTION("ordered_json to json")
{
for (const auto& pattern : patterns)
{
CAPTURE(pattern)
const std::string text = nested_text(depth, pattern);
const nlohmann::ordered_json o = nlohmann::ordered_json::parse(text);
const json converted = o;
CHECK(converted.dump() == text);
}
}
SECTION("get<ordered_json>()")
{
for (const auto& pattern : patterns)
{
CAPTURE(pattern)
const std::string text = nested_text(depth, pattern);
const json j = json::parse(text);
CHECK(j.get<nlohmann::ordered_json>().dump() == text);
}
}
SECTION("depths around the bound of the recursive descent")
{
for (std::size_t d = 1; d <= 300; ++d)
{
CAPTURE(d)
for (const auto& pattern : patterns)
{
CAPTURE(pattern)
const std::string text = nested_text(d, pattern);
const json j = json::parse(text);
const nlohmann::ordered_json converted = j;
CHECK(converted.dump() == text);
const json back = converted;
CHECK(back.dump() == text);
}
}
}
SECTION("values below the bound are converted as values above it")
{
// Bury a value below the bound, where it is converted without the
// call stack, and compare it with the same value converted on its
// own by the containers' range constructors. Its objects have
// members that the two object types enumerate in different orders.
const auto bury = [](nlohmann::ordered_json value)
{
for (std::size_t i = 0; i < 200; ++i)
{
value = nlohmann::ordered_json::array({std::move(value)});
}
return value;
};
const auto dig = [](const json & value)
{
const json* current = &value;
for (std::size_t i = 0; i < 200; ++i)
{
current = &current->at(0);
}
return current;
};
nlohmann::ordered_json value = nlohmann::ordered_json::object();
value["z"] = {1, -2, 3U, 4.5, true, nullptr, "six", nlohmann::ordered_json::binary({7, 8}, 9),
nlohmann::ordered_json::binary({10}), nlohmann::ordered_json::array(), nlohmann::ordered_json::object()
};
value["y"] = {{"x", {{"w", 1}, {"v", 2}}}, {"u", {3, {{"t", 4}, {"s", 5}}}}};
value["r"] = nlohmann::ordered_json::array({nlohmann::ordered_json(nlohmann::ordered_json::value_t::discarded)});
const json converted_above = value;
const json buried = bury(value);
const json& converted_below = *dig(buried);
CHECK(converted_below.dump() == converted_above.dump());
CHECK(converted_below.at("z").at(7).get_binary().subtype() == 9);
CHECK_FALSE(converted_below.at("z").at(8).get_binary().has_subtype());
CHECK(converted_below.at("r").at(0).is_discarded());
// a discarded value is never equal to anything, so compare the rest
value.erase("r");
const json without_discarded_above = value;
const json without_discarded_buried = bury(value);
CHECK(*dig(without_discarded_buried) == without_discarded_above);
}
}
} }
namespace
{
json nested_array(const std::size_t depth, json leaf)
{
json j = std::move(leaf);
for (std::size_t i = 0; i < depth; ++i)
{
json a = json::array();
a.push_back(std::move(j));
j = std::move(a);
}
return j;
}
json nested_object(const std::size_t depth, json leaf)
{
json j = std::move(leaf);
for (std::size_t i = 0; i < depth; ++i)
{
json o = json::object();
o["k"] = std::move(j);
j = std::move(o);
}
return j;
}
} // namespace
TEST_CASE("issue #5392 - binary writers on deeply nested values")
{
// 200 is past the point where the writers stop recursing, and still
// shallow enough that from_* and operator== (which still recurse) are fine.
const json deep_array = nested_array(200, json(0));
const json deep_object = nested_object(200, json("x"));
const json empty_array = nested_array(200, json::array());
const json empty_object = nested_object(200, json::object());
const json mixed = nested_object(80, nested_array(80, json(true)));
SECTION("roundtrip past the recursion bound")
{
CHECK(json::from_cbor(json::to_cbor(deep_array)) == deep_array);
CHECK(json::from_msgpack(json::to_msgpack(deep_array)) == deep_array);
CHECK(json::from_ubjson(json::to_ubjson(deep_array)) == deep_array);
CHECK(json::from_ubjson(json::to_ubjson(deep_array, true, false)) == deep_array);
CHECK(json::from_ubjson(json::to_ubjson(deep_array, true, true)) == deep_array);
CHECK(json::from_bjdata(json::to_bjdata(deep_array)) == deep_array);
CHECK(json::from_bon8(json::to_bon8(deep_array)) == deep_array);
CHECK(json::from_cbor(json::to_cbor(deep_object)) == deep_object);
CHECK(json::from_msgpack(json::to_msgpack(deep_object)) == deep_object);
CHECK(json::from_ubjson(json::to_ubjson(deep_object)) == deep_object);
CHECK(json::from_ubjson(json::to_ubjson(deep_object, true, true)) == deep_object);
CHECK(json::from_bjdata(json::to_bjdata(deep_object)) == deep_object);
CHECK(json::from_bon8(json::to_bon8(deep_object)) == deep_object);
CHECK(json::from_cbor(json::to_cbor(empty_array)) == empty_array);
CHECK(json::from_msgpack(json::to_msgpack(empty_array)) == empty_array);
CHECK(json::from_ubjson(json::to_ubjson(empty_array)) == empty_array);
CHECK(json::from_ubjson(json::to_ubjson(empty_array, true, true)) == empty_array);
CHECK(json::from_bon8(json::to_bon8(empty_array)) == empty_array);
CHECK(json::from_cbor(json::to_cbor(empty_object)) == empty_object);
CHECK(json::from_msgpack(json::to_msgpack(empty_object)) == empty_object);
CHECK(json::from_ubjson(json::to_ubjson(empty_object)) == empty_object);
CHECK(json::from_bon8(json::to_bon8(empty_object)) == empty_object);
CHECK(json::from_cbor(json::to_cbor(mixed)) == mixed);
CHECK(json::from_msgpack(json::to_msgpack(mixed)) == mixed);
CHECK(json::from_ubjson(json::to_ubjson(mixed)) == mixed);
CHECK(json::from_bjdata(json::to_bjdata(mixed)) == mixed);
CHECK(json::from_bon8(json::to_bon8(mixed)) == mixed);
}
SECTION("the two ways of writing a value meet at the bound")
{
for (std::size_t depth = 120; depth <= 140; ++depth)
{
CAPTURE(depth)
const json array = nested_array(depth, json(7));
CHECK(json::from_cbor(json::to_cbor(array)) == array);
CHECK(json::from_msgpack(json::to_msgpack(array)) == array);
CHECK(json::from_ubjson(json::to_ubjson(array, true, true)) == array);
CHECK(json::from_bon8(json::to_bon8(array)) == array);
const json object = nested_object(depth, json(7));
CHECK(json::from_cbor(json::to_cbor(object)) == object);
CHECK(json::from_msgpack(json::to_msgpack(object)) == object);
CHECK(json::from_bjdata(json::to_bjdata(object)) == object);
CHECK(json::from_bon8(json::to_bon8(object)) == object);
}
}
SECTION("a BJData ndarray below the bound is still an ndarray")
{
const json ndarray = json({{"_ArrayType_", "uint8"}, {"_ArraySize_", {2, 3}}, {"_ArrayData_", {1, 2, 3, 4, 5, 6}}});
const json invalid = json({{"_ArrayType_", "nope"}, {"_ArraySize_", {1}}, {"_ArrayData_", {1}}});
const json deep_ndarray = nested_array(140, ndarray);
const json deep_invalid = nested_array(140, invalid);
CHECK(json::from_bjdata(json::to_bjdata(deep_ndarray)) == deep_ndarray);
CHECK(json::from_bjdata(json::to_bjdata(deep_invalid)) == deep_invalid);
CHECK(json::from_bjdata(json::to_bjdata(ndarray)) == ndarray);
}
SECTION("byte-exact across the switch-over")
{
// nested one-element arrays around the recursion bound: the exact
// bytes a writer produces do not depend on whether it stayed on the
// call stack or moved to the heap one partway through
for (const std::size_t depth :
{
nlohmann::detail::recursion_depth_limit() - 1, nlohmann::detail::recursion_depth_limit(),
nlohmann::detail::recursion_depth_limit() + 1, nlohmann::detail::recursion_depth_limit() + 2
})
{
CAPTURE(depth)
const json array = nested_array(depth, json(0));
std::vector<std::uint8_t> expected_cbor(depth, 0x81);
expected_cbor.push_back(0x00);
CHECK(json::to_cbor(array) == expected_cbor);
std::vector<std::uint8_t> expected_msgpack(depth, 0x91);
expected_msgpack.push_back(0x00);
CHECK(json::to_msgpack(array) == expected_msgpack);
std::string expected_ubjson(depth, '[');
expected_ubjson += "i";
expected_ubjson += '\0';
expected_ubjson.append(depth, ']');
const auto packed_ubjson = json::to_ubjson(array);
CHECK(std::string(packed_ubjson.begin(), packed_ubjson.end()) == expected_ubjson);
std::vector<std::uint8_t> expected_bon8(depth, 0x81);
expected_bon8.push_back(0x90);
CHECK(json::to_bon8(array) == expected_bon8);
}
}
SECTION("a deep object, and a BJData ndarray, past the recursion bound")
{
const std::size_t depth = nlohmann::detail::recursion_depth_limit() + 50;
const json object = nested_object(depth, json(42));
CHECK(json::from_cbor(json::to_cbor(object)) == object);
CHECK(json::from_msgpack(json::to_msgpack(object)) == object);
CHECK(json::from_ubjson(json::to_ubjson(object, true, true)) == object);
CHECK(json::from_bjdata(json::to_bjdata(object)) == object);
CHECK(json::from_bon8(json::to_bon8(object)) == object);
const json ndarray = json({{"_ArrayType_", "uint8"}, {"_ArraySize_", {2, 3}}, {"_ArrayData_", {1, 2, 3, 4, 5, 6}}});
const json deep_ndarray = nested_array(depth, ndarray);
CHECK(json::from_bjdata(json::to_bjdata(deep_ndarray)) == deep_ndarray);
}
SECTION("a discarded value past the recursion bound still throws type_error.321")
{
const std::size_t depth = nlohmann::detail::recursion_depth_limit() + 50;
const json discarded_leaf(json::value_t::discarded);
const json deep_discarded = nested_array(depth, discarded_leaf);
// with diagnostics, the message names the path to the discarded leaf
#if JSON_DIAGNOSTICS
std::string path;
for (std::size_t i = 0; i < depth; ++i)
{
path += "/0";
}
const std::string prefix = "[json.exception.type_error.321] (" + path + ") ";
#else
const std::string prefix = "[json.exception.type_error.321] ";
#endif
CHECK_THROWS_WITH_AS(json::to_cbor(deep_discarded), (prefix + "cannot serialize discarded value to CBOR").c_str(), json::type_error);
CHECK_THROWS_WITH_AS(json::to_msgpack(deep_discarded), (prefix + "cannot serialize discarded value to MessagePack").c_str(), json::type_error);
CHECK_THROWS_WITH_AS(json::to_ubjson(deep_discarded), (prefix + "cannot serialize discarded value to UBJSON").c_str(), json::type_error);
CHECK_THROWS_WITH_AS(json::to_bjdata(deep_discarded), (prefix + "cannot serialize discarded value to BJData").c_str(), json::type_error);
}
SECTION("BON8 past the recursion bound: discarded values and errors")
{
const std::size_t depth = nlohmann::detail::recursion_depth_limit() + 50;
// BON8 rejects a discarded value (type_error.321), deep or not
const json deep_discarded = nested_array(depth, json(json::value_t::discarded));
CHECK_THROWS_AS(json::to_bon8(deep_discarded), json::type_error);
// errors are thrown from below the bound as from above it
const json deep_invalid_string = nested_array(depth, json(std::string("\x80")));
CHECK_THROWS_AS(json::to_bon8(deep_invalid_string), json::type_error);
json deep_invalid_key = json::object();
deep_invalid_key[std::string("\x80")] = 1;
deep_invalid_key = nested_object(depth, deep_invalid_key);
CHECK_THROWS_AS(json::to_bon8(deep_invalid_key), json::type_error);
const json deep_too_large = nested_array(depth, json(9223372036854775808u));
CHECK_THROWS_AS(json::to_bon8(deep_too_large), json::out_of_range);
}
SECTION("does not overflow the C++ stack")
{
const std::size_t depth = 100000;
const json j = json::parse(std::string(depth, '[') + "0" + std::string(depth, ']'));
std::vector<std::uint8_t> packed;
CHECK_NOTHROW(packed = json::to_cbor(j));
CHECK(json::from_cbor(packed) == j);
CHECK_NOTHROW(packed = json::to_msgpack(j));
CHECK(json::from_msgpack(packed) == j);
CHECK_NOTHROW(packed = json::to_ubjson(j));
CHECK(json::from_ubjson(packed) == j);
CHECK_NOTHROW(packed = json::to_ubjson(j, true, false));
CHECK(json::from_ubjson(packed) == j);
CHECK_NOTHROW(packed = json::to_bjdata(j));
CHECK(json::from_bjdata(packed) == j);
CHECK_NOTHROW(packed = json::to_bon8(j));
CHECK(json::from_bon8(packed) == j);
}
SECTION("BON8 containers with more than four elements past the recursion bound")
{
// such containers are closed with 0xFE, which the iterative writer
// must emit when it leaves them; strings next to each other must
// still be separated with 0xFF
const std::size_t depth = nlohmann::detail::recursion_depth_limit() + 50;
json wide_array = json::array({1, "a", "b", json::array(), 5});
json wide_object = json({{"k1", "v"}, {"k2", 2}, {"k3", ""}, {"k4", json::object()}, {"k5", nullptr}});
for (std::size_t i = 0; i < depth; ++i)
{
wide_array = json::array({"s", wide_array, "t", wide_object, true});
wide_object = json({{"a", wide_object}, {"b", "x"}, {"c", "y"}, {"d", wide_array.size()}, {"e", "z"}});
}
CHECK(json::from_bon8(json::to_bon8(wide_array)) == wide_array);
CHECK(json::from_bon8(json::to_bon8(wide_object)) == wide_object);
}
SECTION("regression test for https://issues.oss-fuzz.com/issues/566583014")
{
// 200000 nested one-element CBOR arrays, the innermost holding null;
// round-tripping this used to recurse once per level on the way back
// out through to_cbor(), deep enough to overflow the stack
std::vector<std::uint8_t> v(200000, 0x81);
v.push_back(0xf6);
const json j = json::from_cbor(v);
CHECK(json::to_cbor(j) == v);
// the MessagePack analogue: fixarray of 1 nesting down to nil
std::vector<std::uint8_t> v_msgpack(200000, 0x91);
v_msgpack.push_back(0xc0);
const json j_msgpack = json::from_msgpack(v_msgpack);
CHECK(json::to_msgpack(j_msgpack) == v_msgpack);
}
SECTION("regression test for https://issues.oss-fuzz.com/issues/572238015")
{
// the BON8 analogue: 200000 nested one-element arrays down to null
std::vector<std::uint8_t> v(200000, 0x81);
v.push_back(0xFA);
const json j = json::from_bon8(v);
CHECK(json::to_bon8(j) == v);
}
}
@@ -0,0 +1,294 @@
// __ _____ _____ _____
// __| | __| | | | JSON for Modern C++ (supporting code)
// | | |__ | | | | | | version 3.12.0
// |_____|_____|_____|_|___| https://github.com/nlohmann/json
//
// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
// SPDX-License-Identifier: MIT
#include "doctest_compatibility.h"
#include <nlohmann/json.hpp>
using nlohmann::json;
#include <cstddef>
#include <cstdint>
#include <string>
#include <utility>
#include <vector>
// split from unit-large_json.cpp so that both run in parallel and stay below
// the ctest timeout under Valgrind
namespace
{
json nested_array(const std::size_t depth, json leaf)
{
json j = std::move(leaf);
for (std::size_t i = 0; i < depth; ++i)
{
json a = json::array();
a.push_back(std::move(j));
j = std::move(a);
}
return j;
}
json nested_object(const std::size_t depth, json leaf)
{
json j = std::move(leaf);
for (std::size_t i = 0; i < depth; ++i)
{
json o = json::object();
o["k"] = std::move(j);
j = std::move(o);
}
return j;
}
} // namespace
TEST_CASE("issue #5392 - binary writers on deeply nested values")
{
// 200 is past the point where the writers stop recursing, and still
// shallow enough that from_* and operator== (which still recurse) are fine.
const json deep_array = nested_array(200, json(0));
const json deep_object = nested_object(200, json("x"));
const json empty_array = nested_array(200, json::array());
const json empty_object = nested_object(200, json::object());
const json mixed = nested_object(80, nested_array(80, json(true)));
SECTION("roundtrip past the recursion bound")
{
CHECK(json::from_cbor(json::to_cbor(deep_array)) == deep_array);
CHECK(json::from_msgpack(json::to_msgpack(deep_array)) == deep_array);
CHECK(json::from_ubjson(json::to_ubjson(deep_array)) == deep_array);
CHECK(json::from_ubjson(json::to_ubjson(deep_array, true, false)) == deep_array);
CHECK(json::from_ubjson(json::to_ubjson(deep_array, true, true)) == deep_array);
CHECK(json::from_bjdata(json::to_bjdata(deep_array)) == deep_array);
CHECK(json::from_bon8(json::to_bon8(deep_array)) == deep_array);
CHECK(json::from_cbor(json::to_cbor(deep_object)) == deep_object);
CHECK(json::from_msgpack(json::to_msgpack(deep_object)) == deep_object);
CHECK(json::from_ubjson(json::to_ubjson(deep_object)) == deep_object);
CHECK(json::from_ubjson(json::to_ubjson(deep_object, true, true)) == deep_object);
CHECK(json::from_bjdata(json::to_bjdata(deep_object)) == deep_object);
CHECK(json::from_bon8(json::to_bon8(deep_object)) == deep_object);
CHECK(json::from_cbor(json::to_cbor(empty_array)) == empty_array);
CHECK(json::from_msgpack(json::to_msgpack(empty_array)) == empty_array);
CHECK(json::from_ubjson(json::to_ubjson(empty_array)) == empty_array);
CHECK(json::from_ubjson(json::to_ubjson(empty_array, true, true)) == empty_array);
CHECK(json::from_bon8(json::to_bon8(empty_array)) == empty_array);
CHECK(json::from_cbor(json::to_cbor(empty_object)) == empty_object);
CHECK(json::from_msgpack(json::to_msgpack(empty_object)) == empty_object);
CHECK(json::from_ubjson(json::to_ubjson(empty_object)) == empty_object);
CHECK(json::from_bon8(json::to_bon8(empty_object)) == empty_object);
CHECK(json::from_cbor(json::to_cbor(mixed)) == mixed);
CHECK(json::from_msgpack(json::to_msgpack(mixed)) == mixed);
CHECK(json::from_ubjson(json::to_ubjson(mixed)) == mixed);
CHECK(json::from_bjdata(json::to_bjdata(mixed)) == mixed);
CHECK(json::from_bon8(json::to_bon8(mixed)) == mixed);
}
SECTION("the two ways of writing a value meet at the bound")
{
for (std::size_t depth = 120; depth <= 140; ++depth)
{
CAPTURE(depth)
const json array = nested_array(depth, json(7));
CHECK(json::from_cbor(json::to_cbor(array)) == array);
CHECK(json::from_msgpack(json::to_msgpack(array)) == array);
CHECK(json::from_ubjson(json::to_ubjson(array, true, true)) == array);
CHECK(json::from_bon8(json::to_bon8(array)) == array);
const json object = nested_object(depth, json(7));
CHECK(json::from_cbor(json::to_cbor(object)) == object);
CHECK(json::from_msgpack(json::to_msgpack(object)) == object);
CHECK(json::from_bjdata(json::to_bjdata(object)) == object);
CHECK(json::from_bon8(json::to_bon8(object)) == object);
}
}
SECTION("a BJData ndarray below the bound is still an ndarray")
{
const json ndarray = json({{"_ArrayType_", "uint8"}, {"_ArraySize_", {2, 3}}, {"_ArrayData_", {1, 2, 3, 4, 5, 6}}});
const json invalid = json({{"_ArrayType_", "nope"}, {"_ArraySize_", {1}}, {"_ArrayData_", {1}}});
const json deep_ndarray = nested_array(140, ndarray);
const json deep_invalid = nested_array(140, invalid);
CHECK(json::from_bjdata(json::to_bjdata(deep_ndarray)) == deep_ndarray);
CHECK(json::from_bjdata(json::to_bjdata(deep_invalid)) == deep_invalid);
CHECK(json::from_bjdata(json::to_bjdata(ndarray)) == ndarray);
}
SECTION("byte-exact across the switch-over")
{
// nested one-element arrays around the recursion bound: the exact
// bytes a writer produces do not depend on whether it stayed on the
// call stack or moved to the heap one partway through
for (const std::size_t depth :
{
nlohmann::detail::recursion_depth_limit() - 1, nlohmann::detail::recursion_depth_limit(),
nlohmann::detail::recursion_depth_limit() + 1, nlohmann::detail::recursion_depth_limit() + 2
})
{
CAPTURE(depth)
const json array = nested_array(depth, json(0));
std::vector<std::uint8_t> expected_cbor(depth, 0x81);
expected_cbor.push_back(0x00);
CHECK(json::to_cbor(array) == expected_cbor);
std::vector<std::uint8_t> expected_msgpack(depth, 0x91);
expected_msgpack.push_back(0x00);
CHECK(json::to_msgpack(array) == expected_msgpack);
std::string expected_ubjson(depth, '[');
expected_ubjson += "i";
expected_ubjson += '\0';
expected_ubjson.append(depth, ']');
const auto packed_ubjson = json::to_ubjson(array);
CHECK(std::string(packed_ubjson.begin(), packed_ubjson.end()) == expected_ubjson);
std::vector<std::uint8_t> expected_bon8(depth, 0x81);
expected_bon8.push_back(0x90);
CHECK(json::to_bon8(array) == expected_bon8);
}
}
SECTION("a deep object, and a BJData ndarray, past the recursion bound")
{
const std::size_t depth = nlohmann::detail::recursion_depth_limit() + 50;
const json object = nested_object(depth, json(42));
CHECK(json::from_cbor(json::to_cbor(object)) == object);
CHECK(json::from_msgpack(json::to_msgpack(object)) == object);
CHECK(json::from_ubjson(json::to_ubjson(object, true, true)) == object);
CHECK(json::from_bjdata(json::to_bjdata(object)) == object);
CHECK(json::from_bon8(json::to_bon8(object)) == object);
const json ndarray = json({{"_ArrayType_", "uint8"}, {"_ArraySize_", {2, 3}}, {"_ArrayData_", {1, 2, 3, 4, 5, 6}}});
const json deep_ndarray = nested_array(depth, ndarray);
CHECK(json::from_bjdata(json::to_bjdata(deep_ndarray)) == deep_ndarray);
}
SECTION("a discarded value past the recursion bound still throws type_error.321")
{
const std::size_t depth = nlohmann::detail::recursion_depth_limit() + 50;
const json discarded_leaf(json::value_t::discarded);
const json deep_discarded = nested_array(depth, discarded_leaf);
// with diagnostics, the message names the path to the discarded leaf
#if JSON_DIAGNOSTICS
std::string path;
for (std::size_t i = 0; i < depth; ++i)
{
path += "/0";
}
const std::string prefix = "[json.exception.type_error.321] (" + path + ") ";
#else
const std::string prefix = "[json.exception.type_error.321] ";
#endif
CHECK_THROWS_WITH_AS(json::to_cbor(deep_discarded), (prefix + "cannot serialize discarded value to CBOR").c_str(), json::type_error);
CHECK_THROWS_WITH_AS(json::to_msgpack(deep_discarded), (prefix + "cannot serialize discarded value to MessagePack").c_str(), json::type_error);
CHECK_THROWS_WITH_AS(json::to_ubjson(deep_discarded), (prefix + "cannot serialize discarded value to UBJSON").c_str(), json::type_error);
CHECK_THROWS_WITH_AS(json::to_bjdata(deep_discarded), (prefix + "cannot serialize discarded value to BJData").c_str(), json::type_error);
}
SECTION("BON8 past the recursion bound: discarded values and errors")
{
const std::size_t depth = nlohmann::detail::recursion_depth_limit() + 50;
// BON8 rejects a discarded value (type_error.321), deep or not
const json deep_discarded = nested_array(depth, json(json::value_t::discarded));
CHECK_THROWS_AS(json::to_bon8(deep_discarded), json::type_error);
// errors are thrown from below the bound as from above it
const json deep_invalid_string = nested_array(depth, json(std::string("\x80")));
CHECK_THROWS_AS(json::to_bon8(deep_invalid_string), json::type_error);
json deep_invalid_key = json::object();
deep_invalid_key[std::string("\x80")] = 1;
deep_invalid_key = nested_object(depth, deep_invalid_key);
CHECK_THROWS_AS(json::to_bon8(deep_invalid_key), json::type_error);
const json deep_too_large = nested_array(depth, json(9223372036854775808u));
CHECK_THROWS_AS(json::to_bon8(deep_too_large), json::out_of_range);
}
SECTION("does not overflow the C++ stack")
{
const std::size_t depth = 100000;
const json j = json::parse(std::string(depth, '[') + "0" + std::string(depth, ']'));
std::vector<std::uint8_t> packed;
CHECK_NOTHROW(packed = json::to_cbor(j));
CHECK(json::from_cbor(packed) == j);
CHECK_NOTHROW(packed = json::to_msgpack(j));
CHECK(json::from_msgpack(packed) == j);
CHECK_NOTHROW(packed = json::to_ubjson(j));
CHECK(json::from_ubjson(packed) == j);
CHECK_NOTHROW(packed = json::to_ubjson(j, true, false));
CHECK(json::from_ubjson(packed) == j);
CHECK_NOTHROW(packed = json::to_bjdata(j));
CHECK(json::from_bjdata(packed) == j);
CHECK_NOTHROW(packed = json::to_bon8(j));
CHECK(json::from_bon8(packed) == j);
}
SECTION("BON8 containers with more than four elements past the recursion bound")
{
// such containers are closed with 0xFE, which the iterative writer
// must emit when it leaves them; strings next to each other must
// still be separated with 0xFF
const std::size_t depth = nlohmann::detail::recursion_depth_limit() + 50;
json wide_array = json::array({1, "a", "b", json::array(), 5});
json wide_object = json({{"k1", "v"}, {"k2", 2}, {"k3", ""}, {"k4", json::object()}, {"k5", nullptr}});
for (std::size_t i = 0; i < depth; ++i)
{
wide_array = json::array({"s", wide_array, "t", wide_object, true});
wide_object = json({{"a", wide_object}, {"b", "x"}, {"c", "y"}, {"d", wide_array.size()}, {"e", "z"}});
}
CHECK(json::from_bon8(json::to_bon8(wide_array)) == wide_array);
CHECK(json::from_bon8(json::to_bon8(wide_object)) == wide_object);
}
SECTION("regression test for https://issues.oss-fuzz.com/issues/566583014")
{
// 200000 nested one-element CBOR arrays, the innermost holding null;
// round-tripping this used to recurse once per level on the way back
// out through to_cbor(), deep enough to overflow the stack
std::vector<std::uint8_t> v(200000, 0x81);
v.push_back(0xf6);
const json j = json::from_cbor(v);
CHECK(json::to_cbor(j) == v);
// the MessagePack analogue: fixarray of 1 nesting down to nil
std::vector<std::uint8_t> v_msgpack(200000, 0x91);
v_msgpack.push_back(0xc0);
const json j_msgpack = json::from_msgpack(v_msgpack);
CHECK(json::to_msgpack(j_msgpack) == v_msgpack);
}
SECTION("regression test for https://issues.oss-fuzz.com/issues/572238015")
{
// the BON8 analogue: 200000 nested one-element arrays down to null
std::vector<std::uint8_t> v(200000, 0x81);
v.push_back(0xFA);
const json j = json::from_bon8(v);
CHECK(json::to_bon8(j) == v);
}
}
+156
View File
@@ -0,0 +1,156 @@
// __ _____ _____ _____
// __| | __| | | | JSON for Modern C++ (supporting code)
// | | |__ | | | | | | version 3.12.0
// |_____|_____|_____|_|___| https://github.com/nlohmann/json
//
// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
// SPDX-License-Identifier: MIT
#include "doctest_compatibility.h"
#include <nlohmann/json.hpp>
using nlohmann::json;
#include <cstddef>
#include <string>
#include <utility>
#include <vector>
// split from unit-large_json.cpp so that both run in parallel and stay below
// the ctest timeout under Valgrind
namespace
{
// The text of a value nested depth levels deep around the number 0. Level i is
// an array if pattern[i % pattern.size()] is '[', and otherwise an object with
// the single member "a", which every object type enumerates in the same order.
std::string nested_text(std::size_t depth, const std::string& pattern)
{
std::string text;
std::string closing;
for (std::size_t i = 0; i < depth; ++i)
{
const bool array = pattern[i % pattern.size()] == '[';
text += array ? "[" : "{\"a\":";
closing += array ? ']' : '}';
}
text += '0';
text.append(closing.rbegin(), closing.rend());
return text;
}
} // namespace
TEST_CASE("issue #5650 - stack overflow converting between specializations")
{
// deep enough to exhaust the call stack, but small enough to stay cheap:
// parsing is iterative, so building the values below costs little
const std::size_t depth = 100000;
const std::vector<std::string> patterns = {"[", "{", "[{"};
SECTION("json to ordered_json")
{
for (const auto& pattern : patterns)
{
CAPTURE(pattern)
const std::string text = nested_text(depth, pattern);
const json j = json::parse(text);
const nlohmann::ordered_json converted = j;
CHECK(converted.dump() == text);
}
}
SECTION("ordered_json to json")
{
for (const auto& pattern : patterns)
{
CAPTURE(pattern)
const std::string text = nested_text(depth, pattern);
const nlohmann::ordered_json o = nlohmann::ordered_json::parse(text);
const json converted = o;
CHECK(converted.dump() == text);
}
}
SECTION("get<ordered_json>()")
{
for (const auto& pattern : patterns)
{
CAPTURE(pattern)
const std::string text = nested_text(depth, pattern);
const json j = json::parse(text);
CHECK(j.get<nlohmann::ordered_json>().dump() == text);
}
}
SECTION("depths around the bound of the recursive descent")
{
for (std::size_t d = 1; d <= 300; ++d)
{
CAPTURE(d)
for (const auto& pattern : patterns)
{
CAPTURE(pattern)
const std::string text = nested_text(d, pattern);
const json j = json::parse(text);
const nlohmann::ordered_json converted = j;
CHECK(converted.dump() == text);
const json back = converted;
CHECK(back.dump() == text);
}
}
}
SECTION("values below the bound are converted as values above it")
{
// Bury a value below the bound, where it is converted without the
// call stack, and compare it with the same value converted on its
// own by the containers' range constructors. Its objects have
// members that the two object types enumerate in different orders.
const auto bury = [](nlohmann::ordered_json value)
{
for (std::size_t i = 0; i < 200; ++i)
{
value = nlohmann::ordered_json::array({std::move(value)});
}
return value;
};
const auto dig = [](const json & value)
{
const json* current = &value;
for (std::size_t i = 0; i < 200; ++i)
{
current = &current->at(0);
}
return current;
};
nlohmann::ordered_json value = nlohmann::ordered_json::object();
value["z"] = {1, -2, 3U, 4.5, true, nullptr, "six", nlohmann::ordered_json::binary({7, 8}, 9),
nlohmann::ordered_json::binary({10}), nlohmann::ordered_json::array(), nlohmann::ordered_json::object()
};
value["y"] = {{"x", {{"w", 1}, {"v", 2}}}, {"u", {3, {{"t", 4}, {"s", 5}}}}};
value["r"] = nlohmann::ordered_json::array({nlohmann::ordered_json(nlohmann::ordered_json::value_t::discarded)});
const json converted_above = value;
const json buried = bury(value);
const json& converted_below = *dig(buried);
CHECK(converted_below.dump() == converted_above.dump());
CHECK(converted_below.at("z").at(7).get_binary().subtype() == 9);
CHECK_FALSE(converted_below.at("z").at(8).get_binary().has_subtype());
CHECK(converted_below.at("r").at(0).is_discarded());
// a discarded value is never equal to anything, so compare the rest
value.erase("r");
const json without_discarded_above = value;
const json without_discarded_buried = bury(value);
CHECK(*dig(without_discarded_buried) == without_discarded_above);
}
}
+8 -2
View File
@@ -1374,7 +1374,13 @@ void check_dump_against_reference(const std::string& s, const bool with_strict =
}; };
for (const auto error_handler : error_handlers) for (const auto error_handler : error_handlers)
{ {
#if defined(JSON_NOEXCEPTION)
// a strict dump of invalid UTF-8 calls std::abort() instead of throwing
static_cast<void>(with_strict);
if (error_handler == json::error_handler_t::strict)
#else
if (!with_strict && error_handler == json::error_handler_t::strict) if (!with_strict && error_handler == json::error_handler_t::strict)
#endif
{ {
continue; continue;
} }
@@ -1429,7 +1435,7 @@ TEST_CASE("invalid UTF-8 at every position is handled like the reference")
"\xF4\x90\x80\x80", "\xF4\x90\x80\x80",
// sequences cut short by ASCII, a control character, a quote, a // sequences cut short by ASCII, a control character, a quote, a
// backslash, DEL, or the lead byte of a well-formed character // backslash, DEL, or the lead byte of a well-formed character
"\xC3" "a", "\xE2\x82" "\n", "\xE2\x82\"", "\xF0\x9F\x98\\", "\xC3\x7F", "\xC3\xC3\xA4", "\xE2\xF0\x9F\x98\x80", "\xC3" "a", "\xE2\x82" "\n", "\xE2\x82\"", "\xF0\x9F\x98\\", "\xC3\x7F", "\xC3\xC3\xA4", "\xE2\xF0\x9F\x98\x80", // NOLINT(bugprone-suspicious-missing-comma): "\xC3" "a" ends the hex escape
// truncated sequences // truncated sequences
"\xC3", "\xE2\x82", "\xF0\x9F\x98" "\xC3", "\xE2\x82", "\xF0\x9F\x98"
}; };
@@ -1489,7 +1495,7 @@ TEST_CASE("invalid UTF-8 at every position is handled like the reference")
{ {
for (const auto& pattern : patterns) for (const auto& pattern : patterns)
{ {
check_dump_against_reference(make_string(filler, pos, pattern, length - pos), bytes <= (filler.size() == 1 ? 17 : 8)); check_dump_against_reference(make_string(filler, pos, pattern, length - pos), bytes <= (filler.size() == 1 ? 17u : 8u));
} }
} }
} }
+7 -2
View File
@@ -171,8 +171,7 @@ std::vector<std::string> corpus()
R"("unterminated)", "\"control\x01character\"", "\"ill-formed \xc3\x28 UTF-8\"", R"("bad \x escape")", R"("unterminated)", "\"control\x01character\"", "\"ill-formed \xc3\x28 UTF-8\"", R"("bad \x escape")",
R"("\ud800 lone surrogate")", R"("\ud800 lone surrogate")",
// literals // literals
"true", "false", "null", "tru", "nul", "falsey", "[true,false,null]", "nulL", "truex", "[fals e]", "t", "true", "false", "null", "tru", "nul", "falsey",
"true1", "null null", "[false]",
// structures // structures
"[]", "{}", "[1,2,3]", R"({"a":1,"b":[true,false,null],"c":{"d":"e"}})", "[1,]", R"({"a" 1})", "[[[[[]]]]]", "[]", "{}", "[1,2,3]", R"({"a":1,"b":[true,false,null],"c":{"d":"e"}})", "[1,]", R"({"a" 1})", "[[[[[]]]]]",
// pretty-printed // pretty-printed
@@ -191,6 +190,7 @@ std::vector<std::string> corpus()
TEST_CASE("parsing from a std::istream scans the get area in bulk") TEST_CASE("parsing from a std::istream scans the get area in bulk")
{ {
#if !defined(JSON_NOEXCEPTION)
SECTION("results do not depend on the get area") SECTION("results do not depend on the get area")
{ {
const std::vector<read_mode> read_modes = {read_mode::extract, read_mode::parse, read_mode::parse_lenient}; const std::vector<read_mode> read_modes = {read_mode::extract, read_mode::parse, read_mode::parse_lenient};
@@ -211,6 +211,8 @@ TEST_CASE("parsing from a std::istream scans the get area in bulk")
} }
} }
#endif // !defined(JSON_NOEXCEPTION)
SECTION("values are the same as when parsing from a string") SECTION("values are the same as when parsing from a string")
{ {
const std::string input = R"({"name": "nlohmann::json", "numbers": [1, -2, 3.5, 4e10, 18446744073709551615],)" const std::string input = R"({"name": "nlohmann::json", "numbers": [1, -2, 3.5, 4e10, 18446744073709551615],)"
@@ -228,6 +230,7 @@ TEST_CASE("parsing from a std::istream scans the get area in bulk")
CHECK(json::parse(ss) == expected); CHECK(json::parse(ss) == expected);
} }
#if !defined(JSON_NOEXCEPTION)
SECTION("a token that spans several get areas is reported completely") SECTION("a token that spans several get areas is reported completely")
{ {
// the error is found after the string has crossed many get areas; the // the error is found after the string has crossed many get areas; the
@@ -282,6 +285,8 @@ TEST_CASE("parsing from a std::istream scans the get area in bulk")
} }
} }
#endif // !defined(JSON_NOEXCEPTION)
SECTION("many values from one stream") SECTION("many values from one stream")
{ {
std::string input; std::string input;
+4
View File
@@ -218,7 +218,11 @@ TEST_CASE("compliance tests from nativejson-benchmark")
n1e308[0] = '['; n1e308[0] = '[';
n1e308[1] = '1'; n1e308[1] = '1';
n1e308[310] = ']'; n1e308[310] = ']';
#if JSON_STRICT_NUL_HANDLING
n1e308.pop_back(); // a NUL byte after the value is a syntax error in strict mode
#else
n1e308[311] = '\0'; n1e308[311] = '\0';
#endif
TEST_DOUBLE(n1e308, 1E308); TEST_DOUBLE(n1e308, 1E308);
} }
+18 -9
View File
@@ -54,8 +54,8 @@ TEST_CASE("exception messages of out-of-line throw sites")
SECTION("at") SECTION("at")
{ {
const json carr = arr; const json& carr = arr;
const json cobj = obj; const json& cobj = obj;
CHECK_MSG((void)arr.at(static_cast<json::size_type>(10)), "[json.exception.out_of_range.401] array index 10 is out of range"); CHECK_MSG((void)arr.at(static_cast<json::size_type>(10)), "[json.exception.out_of_range.401] array index 10 is out of range");
CHECK_MSG((void)carr.at(static_cast<json::size_type>(3)), "[json.exception.out_of_range.401] array index 3 is out of range"); CHECK_MSG((void)carr.at(static_cast<json::size_type>(3)), "[json.exception.out_of_range.401] array index 3 is out of range");
CHECK_MSG((void)num.at(static_cast<json::size_type>(0)), "[json.exception.type_error.304] cannot use at() with number"); CHECK_MSG((void)num.at(static_cast<json::size_type>(0)), "[json.exception.type_error.304] cannot use at() with number");
@@ -65,7 +65,7 @@ TEST_CASE("exception messages of out-of-line throw sites")
CHECK_MSG((void)num.at("a"), "[json.exception.type_error.304] cannot use at() with number"); CHECK_MSG((void)num.at("a"), "[json.exception.type_error.304] cannot use at() with number");
nlohmann::ordered_json oobj = {{"a", 1}}; nlohmann::ordered_json oobj = {{"a", 1}};
const nlohmann::ordered_json coobj = oobj; const nlohmann::ordered_json& coobj = oobj;
CHECK_MSG((void)oobj.at("z"), "[json.exception.out_of_range.403] key 'z' not found"); CHECK_MSG((void)oobj.at("z"), "[json.exception.out_of_range.403] key 'z' not found");
CHECK_MSG((void)coobj.at("z"), "[json.exception.out_of_range.403] key 'z' not found"); CHECK_MSG((void)coobj.at("z"), "[json.exception.out_of_range.403] key 'z' not found");
CHECK_MSG((void)coobj.at(std::string("zz")), "[json.exception.out_of_range.403] key 'zz' not found"); CHECK_MSG((void)coobj.at(std::string("zz")), "[json.exception.out_of_range.403] key 'zz' not found");
@@ -94,9 +94,10 @@ TEST_CASE("exception messages of out-of-line throw sites")
CHECK_MSG(obj.update(arr.begin(), arr2.end()), "[json.exception.invalid_iterator.210] iterators do not fit"); CHECK_MSG(obj.update(arr.begin(), arr2.end()), "[json.exception.invalid_iterator.210] iterators do not fit");
} }
#if !defined(JSON_NOEXCEPTION) // patch_msg() catches the exceptions, which JSON_NOEXCEPTION turns into std::abort()
SECTION("patch") SECTION("patch")
{ {
const auto patch_msg = [](const char* doc, const char* patch) const auto patch_msg = [](const char* doc, const char* patch) -> std::string
{ {
try try
{ {
@@ -105,16 +106,23 @@ TEST_CASE("exception messages of out-of-line throw sites")
} }
catch (const json::exception& e) catch (const json::exception& e)
{ {
// with JSON_DIAGNOSTICS, what() names the offending value like "(/0) "; drop it // with JSON_DIAGNOSTICS, what() names the offending value like "(/0) ",
// and with JSON_DIAGNOSTIC_POSITIONS its position like "(bytes 0-2) "; drop both
std::string msg = e.what(); std::string msg = e.what();
const auto pos = msg.find(" (/"); for (const char* marker :
if (pos != std::string::npos) {
" (/", " (bytes "
})
{ {
msg.erase(pos + 1, msg.find(") ", pos) - pos + 1); const auto pos = msg.find(marker);
if (pos != std::string::npos)
{
msg.erase(pos + 1, msg.find(") ", pos) - pos + 1);
}
} }
return msg; return msg;
} }
return std::string(); return {};
}; };
CHECK(patch_msg("{}", "{}") == "[json.exception.parse_error.104] parse error: JSON patch must be an array of objects"); CHECK(patch_msg("{}", "{}") == "[json.exception.parse_error.104] parse error: JSON patch must be an array of objects");
@@ -132,4 +140,5 @@ TEST_CASE("exception messages of out-of-line throw sites")
CHECK(patch_msg("{\"a\":1}", "[{\"op\":\"test\",\"path\":\"/a\",\"value\":2}]") == "[json.exception.other_error.501] unsuccessful: {\"op\":\"test\",\"path\":\"/a\",\"value\":2}"); CHECK(patch_msg("{\"a\":1}", "[{\"op\":\"test\",\"path\":\"/a\",\"value\":2}]") == "[json.exception.other_error.501] unsuccessful: {\"op\":\"test\",\"path\":\"/a\",\"value\":2}");
CHECK(patch_msg("{\"a\":1}", "[{\"op\":\"move\",\"path\":\"/a\"}]") == "[json.exception.parse_error.105] parse error: operation 'move' must have member 'from'"); CHECK(patch_msg("{\"a\":1}", "[{\"op\":\"move\",\"path\":\"/a\"}]") == "[json.exception.parse_error.105] parse error: operation 'move' must have member 'from'");
} }
#endif
} }