Compare commits

..
Author SHA1 Message Date
Niels Lohmann e5f84e1ebf refactor: split the strict and non-strict paths in parser
Folding the release_lookahead() call into the existing strict check left
the "in strict mode" comment on an else-if branch, and made the strict
condition in sax_parse() redundant with the branch it followed.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-08-19 22:50:21 +02:00
Niels Lohmann 7fc3a7d87e docs: drop the whitespace-separator caveat from the parsing pages
The caveat added in #5343 describes the behavior this branch fixes: a
number no longer consumes the character that terminates it, so
concatenated values need no separator.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-08-19 22:46:12 +02:00
Niels Lohmann 43b689b9b6 Merge remote-tracking branch 'origin/develop' into claude/issue-5340-restore-unget
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-08-19 22:41:46 +02:00
Niels Lohmann a13902a33f docs: match the version history wording to the peek-based fix
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-08-04 14:44:23 +02:00
Niels Lohmann c021a09b08 fix: leave the character that terminates a number in the input
Read the character following a number without consuming it, instead of
consuming it and putting it back. input_stream_adapter now peeks with
sgetc() and only steps over the character when the next one is requested
or when the adapter is destroyed, so releasing it cannot fail - no
putback position is required from the streambuf.

Suggested by gregmarr in #5344.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-08-04 14:39:05 +02:00
Niels Lohmann e4aaf46d38 Merge branch 'develop' into claude/issue-5340-restore-unget
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-08-04 14:25:17 +02:00
Niels Lohmann 5bc24e876b tests: fix CI failures in the #5340 test helpers
Four CI failures, all in the new test code:

- GCC (-Werror=useless-cast): drop the `json(...)` wrapper around
  `json::parse(...)`, which already returns a `json`.
- GCC (-Werror=unused-result): assign the discarded `json::parse()`
  result to a dummy, the idiom used elsewhere in the test suite, and
  catch `json::parse_error&` for consistency.
- clang-tidy (google-default-arguments): remove the default argument
  from the `pbackfail()` override; `sungetc()` supplies the base
  declaration's default.
- MSVC (bad allocation): `no_putback_streambuf::underflow()` set a
  one-character get area without advancing `m_pos`, so an implementation
  whose `istream::get` peeks before it bumps re-read the same character
  forever. Keep no get area at all: `underflow()` peeks, `uflow()`
  consumes, and `sungetc()` still always lands in `pbackfail()`, which
  is what the test needs.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-08-03 19:42:06 +02:00
Niels Lohmann da7b9bdb3d fix: restore the character that terminates a number (#5340)
operator>> is documented to leave the stream positioned right after the
parsed value, so that concatenated JSON values can be read back to back.
That did not hold for numbers: a number is only terminated by the
character following it, and lexer::scan_number() reads that character
and calls unget() -- which is simulated and rewinds only the lexer's own
bookkeeping. input_stream_adapter consumes via sbumpc() with no matching
sungetc(), so the terminating character stayed consumed and the next
extraction started one byte too late ('1true' left the stream at 'rue').

Propagating unget() to the adapter directly does not work: next_unget
makes the following get() replay the cached character, so the terminator
would be delivered twice. Instead, restore the still-pending character
once at the end of a non-strict parse, where the input is handed back to
the caller:

- input_stream_adapter gains unget_character() (sungetc()) and advertises
  it via supports_unget, detected the same way as supports_seek.
- lexer::restore_pending_unget() turns a pending simulated unget of a
  real (non-EOF) character into a real one and clears next_unget so the
  character is not also replayed. It is a no-op for adapters that cannot
  unget, and reports failure when sungetc() fails, in which case the
  input is left as it was before.
- parser calls it on the three non-strict paths, i.e. for operator>> and
  sax_parse(strict = false).

Strict parse()/accept() are unaffected: they require the input to end
after the value, so the character is consumed by the end-of-input check
anyway. Parse error messages and reported positions are unchanged.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-08-01 07:33:11 +02:00
Niels Lohmann 634f49bc5b docs: qualify the operator>> stream positioning guarantee
operator>>'s notes state that it leaves the stream positioned right
after the parsed value, so that concatenated JSON values can be read
back to back. That does not hold when the value is a number: a number
is only terminated by the character that follows it, and the lexer's
unget() is simulated (it rewinds only the lexer's own bookkeeping),
so that character stays consumed from the stream.

Document the actual behaviour: the guarantee holds for all value types
except numbers, which must be followed by whitespace. Also qualify the
cross-reference on the JSON Lines page, which repeated the unqualified
claim.

Documentation only; the behaviour itself is tracked in #5340.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-08-01 07:27:53 +02:00
30 changed files with 1752 additions and 3088 deletions
+1 -3
View File
@@ -108,9 +108,7 @@ The tests are located in [`tests/src/unit-*.cpp`](https://github.com/nlohmann/js
are structured along the features of the library or the nature of the tests. Usually, it should be clear from the
context which existing file needs to be extended, and only very few cases require creating new test files.
When fixing a bug, edit `unit-regression3.cpp` and add a test case referencing the fixed issue. Its predecessors
`unit-regression1.cpp` and `unit-regression2.cpp` stay as they are: the MinGW linker fails on the object a file this
size produces, which is why the tests are spread over several files in the first place.
When fixing a bug, edit `unit-regression2.cpp` and add a section referencing the fixed issue.
#### Exceptions
+1 -11
View File
@@ -67,18 +67,8 @@ jobs:
${{ github.workspace }}/venv/bin/astyle --project=tools/astyle/.astylerc --suffix=none --quiet \
$INCLUDE_DIR/json.hpp $INCLUDE_DIR/json_fwd.hpp
# fail loudly if a directory is renamed or removed: find would only warn
# about the missing path and silently drop its files from the check
SOURCE_DIRS="docs/mkdocs/docs/examples include tests"
for DIR in $SOURCE_DIRS; do
if [ ! -d "$DIR" ]; then
echo "::error::source directory '$DIR' does not exist"
exit 1
fi
done
${{ github.workspace }}/venv/bin/astyle --project=tools/astyle/.astylerc --suffix=none --quiet \
$(find $SOURCE_DIRS -type f \( -name '*.hpp' -o -name '*.cpp' -o -name '*.cu' \) -not -path 'tests/thirdparty/*' -not -path 'tests/abi/include/nlohmann/*' | sort)
$(find docs/examples include tests -type f \( -name '*.hpp' -o -name '*.cpp' -o -name '*.cu' \) -not -path 'tests/thirdparty/*' -not -path 'tests/abi/include/nlohmann/*' | sort)
- name: Build patch and check for differences
id: diff
@@ -7,6 +7,7 @@ on:
- develop
paths:
- docs/mkdocs/**
- docs/examples/**
workflow_dispatch:
# we don't want to have concurrent jobs, and we don't want to cancel running jobs to avoid broken publications
+1 -1
View File
@@ -100,7 +100,7 @@ jobs:
container: ubuntu:focal
strategy:
matrix:
target: [ci_cmake_flags, ci_test_diagnostics, ci_test_diagnostic_positions, ci_test_noexceptions, ci_test_noimplicitconversions, ci_test_legacycomparison, ci_test_noglobaludls, ci_test_no_thread_local]
target: [ci_cmake_flags, ci_test_diagnostics, ci_test_diagnostic_positions, ci_test_noexceptions, ci_test_noimplicitconversions, ci_test_legacycomparison, ci_test_noglobaludls]
steps:
- name: Install build-essential
run: apt-get update ; apt-get install -y build-essential unzip wget git libssl-dev
-4
View File
@@ -158,10 +158,6 @@ jobs:
# to fit: IMAGE_REL_AMD64_SECREL against `.debug_line'" because the
# MinGW linker cannot relocate the debug sections this test produces.
# The tests are only built and run here, so the debug info is not used.
# Do not add -O1 here to shrink the objects further: it does make them
# link, but the binaries clang 11.0.1 and clang 18.1.8 then produce crash
# before doctest prints its first line - 39 of 102 tests on clang 18.
# Keep the objects small by splitting the test files instead.
- name: Run CMake
run: cmake -S . -B build ^
-DCMAKE_CXX_COMPILER="C:/Program Files/LLVM/bin/clang++.exe" ^
+1 -20
View File
@@ -242,25 +242,6 @@ add_custom_target(ci_test_noglobaludls
COMMENT "Compile and test with global UDLs disabled"
)
###############################################################################
# Disable thread-local storage.
###############################################################################
# Without thread-local storage, copying and comparing cannot bound their
# descent and handle every object and array without the call stack. Those paths
# are otherwise only reached by values nested deeper than the bound, so this
# target is what runs the whole test suite through them.
add_custom_target(ci_test_no_thread_local
COMMAND ${CMAKE_COMMAND}
-DCMAKE_BUILD_TYPE=Debug -GNinja
-DJSON_BuildTests=ON
-DCMAKE_CXX_FLAGS=-DJSON_NO_THREAD_LOCAL
-S${PROJECT_SOURCE_DIR} -B${PROJECT_BINARY_DIR}/build_no_thread_local
COMMAND ${CMAKE_COMMAND} --build ${PROJECT_BINARY_DIR}/build_no_thread_local
COMMAND cd ${PROJECT_BINARY_DIR}/build_no_thread_local && ${CMAKE_CTEST_COMMAND} --parallel ${N} --output-on-failure
COMMENT "Compile and test without thread-local storage"
)
###############################################################################
# Coverage.
###############################################################################
@@ -313,7 +294,7 @@ file(GLOB_RECURSE INDENT_FILES
${PROJECT_SOURCE_DIR}/tests/src/*.cpp
${PROJECT_SOURCE_DIR}/tests/src/*.hpp
${PROJECT_SOURCE_DIR}/tests/benchmarks/src/benchmarks.cpp
${PROJECT_SOURCE_DIR}/docs/mkdocs/docs/examples/*.cpp
${PROJECT_SOURCE_DIR}/docs/examples/*.cpp
)
set(include_dir ${PROJECT_SOURCE_DIR}/single_include/nlohmann)
+4 -1
View File
@@ -69,7 +69,8 @@ The SAX event lister must follow the interface of [`json_sax`](../json_sax/index
[`input_format_t`](input_format_t.md) for more information
`strict` (in)
: whether the input has to be consumed completely (optional, `#!cpp true` by default)
: whether the input has to be consumed completely (optional, `#!cpp true` by default); when `#!cpp false` and the
input is a `#!cpp std::istream`, the stream is left positioned right after the parsed value
`ignore_comments` (in)
: whether comments should be ignored and treated like whitespace (`#!cpp true`) or yield a parse error
@@ -136,6 +137,8 @@ A UTF-8 byte order mark is silently ignored.
- Added `ignore_trailing_commas` in version 3.13.0.
- Extended container support (1) to include types with lvalue-only ADL `begin`/`end` (matching `std::begin`/`std::end` semantics) in version 3.13.0.
- Extended overload (2) to accept heterogeneous iterator+sentinel pairs (C++20 ranges support) in version 3.13.0.
- Changed in version 4.0.0 to leave a `#!cpp std::istream` positioned right after the parsed value when `strict` is
`#!cpp false`; see [`operator>>`](../operator_gtgt.md#notes).
!!! warning "Deprecation"
-1
View File
@@ -22,7 +22,6 @@ header. See also the [macro overview page](../../features/macros.md).
- [**JSON_HAS_STD_FORMAT**](json_has_std_format.md) - control `std::format`/`std::formatter` support
- [**JSON_HAS_THREE_WAY_COMPARISON**](json_has_three_way_comparison.md) - control 3-way comparison support
- [**JSON_NO_IO**](json_no_io.md) - switch off functions relying on certain C++ I/O headers
- [**JSON_NO_THREAD_LOCAL**](json_no_thread_local.md) - switch off the use of `thread_local` storage
- [**JSON_SKIP_UNSUPPORTED_COMPILER_CHECK**](json_skip_unsupported_compiler_check.md) - do not warn about unsupported compilers
- [**JSON_USE_GLOBAL_UDLS**](json_use_global_udls.md) - place user-defined string literals (UDLs) into the global namespace
@@ -1,48 +0,0 @@
# JSON_NO_THREAD_LOCAL
```cpp
#define JSON_NO_THREAD_LOCAL
```
When defined, the library does not use `#!cpp thread_local` storage. This is relevant for the few environments whose
toolchain does not support it.
Copying a value and comparing two values both descend into the first levels by letting the containers copy or compare
themselves, and finish whatever is nested deeper than that without the call stack, so that neither can exhaust the stack
however deeply the values are nested. Each counts the levels it has descended into in a `#!cpp thread_local` variable, as
a counter shared between threads would be raced.
Without those counters, no descent can be bounded safely, so objects and arrays are copied and compared without the call
stack right away. Both keep working exactly as they do otherwise - the same values come out, the same comparisons hold,
and deeply nested values are handled just as safely - but both are slower, because the containers no longer copy or
compare themselves. Copying the benchmark documents takes 9% (`canada.json`) to 34% (`twitter.json`) longer, and
comparing two equal ones 10% (`citm_catalog.json`) to 90% (`canada.json`) longer.
## Default definition
By default, `#!cpp JSON_NO_THREAD_LOCAL` is not defined.
```cpp
#undef JSON_NO_THREAD_LOCAL
```
The library defines it by itself for Clang targeting MinGW, which does not survive the `#!cpp thread_local` storage:
copying a value segfaults there, with both old and current Clang versions, while GCC targeting MinGW is unaffected.
Copying and comparing fall back to working without the call stack there, as they do whenever the macro is defined.
## Examples
??? example
The code below forces the library not to use `#!cpp thread_local` storage.
```cpp
#define JSON_NO_THREAD_LOCAL 1
#include <nlohmann/json.hpp>
...
```
## Version history
- Added in version 3.12.1.
+16 -29
View File
@@ -33,41 +33,26 @@ A UTF-8 byte order mark is silently ignored.
Invalid Unicode escapes and unpaired surrogates in the input are reported as
[`parse_error.101`](../home/exceptions.md#jsonexceptionparse_error101) with a detailed message.
`operator>>` parses exactly one JSON value, so it can be called repeatedly to read a sequence of concatenated JSON
values from the same stream:
`operator>>` parses exactly one JSON value and leaves the stream positioned right after it, so it can be called
repeatedly to read a sequence of concatenated JSON values from the same stream:
```cpp
json j1, j2;
input >> j1; // parses the first value
input >> j2; // parses the next value
std::istringstream input("1true[2]");
json j1, j2, j3;
input >> j1; // j1 == 1, stream now positioned right after it
input >> j2; // j2 == true
input >> j3; // j3 == [2]
```
!!! warning "A number must be followed by whitespace"
!!! note "Changed behavior for numbers"
A number is only terminated by the character that follows it. That character is read from the stream to detect the
end of the number, and it is **not** put back. When a value that is a number is immediately followed by the next
value, the first character of that next value is lost:
A number is the only value whose end can be detected solely by reading the character that follows it. Up to
version 3.13.0 that character was consumed and not put back, so the stream was left one byte too far whenever a
number was immediately followed by another value: reading `1true` yielded `1` and left the stream at `rue`.
Values had to be separated by whitespace to work around this.
```cpp
std::istringstream input("1true");
json j1, j2;
input >> j1; // j1 == 1
input >> j2; // throws parse_error.101: the stream now starts at "rue"
```
Separating the values with whitespace avoids this, because the character that is eaten is then the separator:
```cpp
std::istringstream input("1 true");
json j1, j2;
input >> j1; // j1 == 1
input >> j2; // j2 == true
```
Only numbers are affected. Values ending in a self-delimiting character do not read past themselves, so
`truefalse`, `[1][2]`, `{"a":1}{"b":2}`, and `"a""b"` can be read back to back without a separator.
This is tracked in [#5340](https://github.com/nlohmann/json/issues/5340).
The terminating character is now only looked at and left in the stream, so no separator is required. Code that
relied on the extra byte being swallowed will observe it again.
Note that reading concatenated values does **not** work for [JSON Lines](../features/parsing/json_lines.md)
(newline-delimited JSON) input -- see that page for why and for the recommended alternative.
@@ -102,3 +87,5 @@ Note that reading concatenated values does **not** work for [JSON Lines](../feat
## Version history
- Added in version 1.0.0.
- Changed in version 4.0.0 to leave the character that terminates a number in the stream, so that the stream is
positioned right after the parsed value for every value type.
@@ -9,13 +9,13 @@ int main()
auto text = R"({"IDs": [116, 943], "Width": 800})";
// discard the array when the parser reads its opening bracket
json j_array_start = json::parse(text, [](int /*depth*/, json::parse_event_t event, json& /*parsed*/)
json j_array_start = json::parse(text, [](int /*depth*/, json::parse_event_t event, json & /*parsed*/)
{
return event != json::parse_event_t::array_start;
});
// discard the same array when the parser reads its closing bracket
json j_array_end = json::parse(text, [](int /*depth*/, json::parse_event_t event, json& /*parsed*/)
json j_array_end = json::parse(text, [](int /*depth*/, json::parse_event_t event, json & /*parsed*/)
{
return event != json::parse_event_t::array_end;
});
@@ -33,7 +33,7 @@ int main()
});
// discard the top-level object
json j_root = json::parse(text, [](int /*depth*/, json::parse_event_t event, json& /*parsed*/)
json j_root = json::parse(text, [](int /*depth*/, json::parse_event_t event, json & /*parsed*/)
{
return event != json::parse_event_t::object_end;
});
-8
View File
@@ -91,14 +91,6 @@ security reasons (e.g., Intel Software Guard Extensions (SGX)).
See [full documentation of `JSON_NO_IO`](../api/macros/json_no_io.md).
## `JSON_NO_THREAD_LOCAL`
When defined, the library does not use `#!cpp thread_local` storage. Copying a value and comparing two values then
always avoid the call stack rather than descending into a bounded number of levels first, which is slower but yields the
same values and the same comparisons.
See [full documentation of `JSON_NO_THREAD_LOCAL`](../api/macros/json_no_thread_local.md).
## `JSON_SKIP_LIBRARY_VERSION_CHECK`
When defined, the library will not create a compiler warning when a different version of the library was already
+1 -2
View File
@@ -40,8 +40,7 @@ what makes it possible to read several concatenated values from the same stream,
document followed by trailing bytes" is accepted rather than rejected. If you are validating conformance, or need to
reject any input that is not exactly one JSON document, prefer `parse`.
When using `operator>>` to read several concatenated values this way, a value that is a number must be followed by
whitespace, because `operator>>` consumes the character that terminates a number — see the
Values read this way do not need to be separated by whitespace; see the
[`operator>>` notes](../../api/operator_gtgt.md#notes) for details and examples.
## SAX vs. DOM parsing
@@ -49,5 +49,4 @@ JSON Lines input with more than one value is treated as invalid JSON by the [`pa
with a JSON Lines input does not work, because the parser will try to parse one value after the last one.
This is different from parsing a stream of *concatenated* (non-newline-delimited) JSON values, for which
`operator>>` does work, provided that a value that is a number is followed by whitespace -- see its
[notes](../../api/operator_gtgt.md#notes) for details.
`operator>>` does work -- see its [notes](../../api/operator_gtgt.md#notes) for details.
-1
View File
@@ -291,7 +291,6 @@ nav:
- 'JSON_HAS_THREE_WAY_COMPARISON': api/macros/json_has_three_way_comparison.md
- 'JSON_NOEXCEPTION': api/macros/json_noexception.md
- 'JSON_NO_IO': api/macros/json_no_io.md
- 'JSON_NO_THREAD_LOCAL': api/macros/json_no_thread_local.md
- 'JSON_SKIP_LIBRARY_VERSION_CHECK': api/macros/json_skip_library_version_check.md
- 'JSON_SKIP_UNSUPPORTED_COMPILER_CHECK': api/macros/json_skip_unsupported_compiler_check.md
- 'JSON_USE_GLOBAL_UDLS': api/macros/json_use_global_udls.md
+1 -1
View File
@@ -1,4 +1,4 @@
wheel==0.48.0
wheel==0.47.0
mkdocs==1.6.1 # documentation framework
mkdocs-git-revision-date-localized-plugin==1.5.3 # plugin "git-revision-date-localized"
@@ -101,6 +101,9 @@ class input_stream_adapter
// maintain ifstream flags, except eof
if (is != nullptr)
{
// consume the character last returned by get_character() unless it
// was given back with release_lookahead()
commit_lookahead();
is->clear(is->rdstate() & std::ios::eofbit);
}
}
@@ -115,29 +118,60 @@ class input_stream_adapter
input_stream_adapter& operator=(input_stream_adapter&&) = delete;
input_stream_adapter(input_stream_adapter&& rhs) noexcept
: is(rhs.is), sb(rhs.sb)
: is(rhs.is), sb(rhs.sb), lookahead(rhs.lookahead)
{
rhs.is = nullptr;
rhs.sb = nullptr;
rhs.lookahead = false;
}
// Whether the character last returned by get_character() can be given back
// to the input with release_lookahead().
static constexpr bool supports_lookahead = true;
// std::istream/std::streambuf use std::char_traits<char>::to_int_type, to
// ensure that std::char_traits<char>::eof() and the character 0xFF do not
// end up as the same value, e.g., 0xFFFFFFFF.
//
// The character is peeked rather than consumed: it is only stepped over
// once the next character is requested, or when the adapter is destroyed.
// Until then, release_lookahead() can leave it in the input.
std::char_traits<char>::int_type get_character()
{
auto res = sb->sbumpc();
if (lookahead)
{
// step over the character returned by the previous call
sb->sbumpc();
}
auto res = sb->sgetc();
// set eof manually, as we don't use the istream interface.
if (JSON_HEDLEY_UNLIKELY(res == std::char_traits<char>::eof()))
{
// there is nothing to step over next time
lookahead = false;
is->clear(is->rdstate() | std::ios::eofbit);
}
else
{
lookahead = true;
}
return res;
}
// Leave the character last returned by get_character() in the input, so
// that the next read from the stream - by this adapter or by the caller
// once parsing is done - sees it again. Unlike putting a consumed
// character back, this cannot fail.
void release_lookahead() noexcept
{
lookahead = false;
}
template<class T>
std::size_t get_elements(T* dest, std::size_t count = 1)
{
commit_lookahead();
auto res = static_cast<std::size_t>(sb->sgetn(reinterpret_cast<char*>(dest), static_cast<std::streamsize>(count * sizeof(T))));
if (JSON_HEDLEY_UNLIKELY(res < count * sizeof(T)))
{
@@ -147,9 +181,23 @@ class input_stream_adapter
}
private:
// Step over the character last returned by get_character(). The character
// has already been peeked successfully, so for every streambuf with a get
// area this is a pointer increment that cannot fail.
void commit_lookahead()
{
if (lookahead)
{
lookahead = false;
sb->sbumpc();
}
}
/// the associated input stream
std::istream* is = nullptr;
std::streambuf* sb = nullptr;
/// whether get_character() peeked a character that is not consumed yet
bool lookahead = false;
};
#endif // JSON_NO_IO
+62
View File
@@ -125,6 +125,24 @@ constexpr bool input_adapter_supports_seek(std::false_type /*detected*/)
return false;
}
// Detect whether an input adapter reads with one character of lookahead that
// can be left in the input (see input_stream_adapter::supports_lookahead),
// detected like supports_seek above.
template<typename InputAdapterType>
using detect_supports_lookahead = decltype(InputAdapterType::supports_lookahead);
template<typename InputAdapterType>
constexpr bool input_adapter_supports_lookahead(std::true_type /*detected*/)
{
return InputAdapterType::supports_lookahead;
}
template<typename InputAdapterType>
constexpr bool input_adapter_supports_lookahead(std::false_type /*detected*/)
{
return false;
}
/*!
@brief lexical analysis
@@ -146,6 +164,12 @@ class lexer : public lexer_base<BasicJsonType>
static constexpr bool lazy_token_string =
input_adapter_supports_seek<InputAdapterType>(is_detected<detect_supports_seek, InputAdapterType> {});
/// whether a simulated unget can be passed on to the input adapter, which
/// then leaves the character in the input; see
/// input_adapter_supports_lookahead
static constexpr bool can_release_lookahead =
input_adapter_supports_lookahead<InputAdapterType>(is_detected<detect_supports_lookahead, InputAdapterType> {});
public:
using token_type = typename lexer_base<BasicJsonType>::token_type;
@@ -1461,6 +1485,21 @@ scan_number_done:
uncapture_char(std::integral_constant<bool, lazy_token_string> {});
}
/// adapter without lookahead: nothing to do (see release_lookahead)
void release_lookahead_impl(std::false_type /*can_release*/) const noexcept {}
/// adapter with lookahead: leave the character in the input instead
void release_lookahead_impl(std::true_type /*can_release*/)
{
if (next_unget)
{
// the character is read from the input again rather than replayed
// from current, so the adapter must not step over it
next_unget = false;
ia.release_lookahead();
}
}
/// seekable adapter: nothing was captured, so nothing to undo
void uncapture_char(std::true_type /*lazy*/) const noexcept {}
@@ -1524,6 +1563,29 @@ scan_number_done:
return position;
}
/*!
@brief pass a pending simulated unget on to the input
unget() only rewinds the lexer's own bookkeeping, so the character that
terminated the last token (e.g. the character after a number) would still
be stepped over when the input adapter is done. Callers that hand the
input back to the user afterwards - operator>> and non-strict sax_parse -
call this once when scanning is done, so that the input is positioned
right after the value.
Adapters without lookahead (see input_adapter_supports_lookahead) are not
handed back to the user, so this is a no-op for them.
Scanning may continue after this call: @a next_unget is cleared, and the
character is read from the input again instead of being replayed from
@a current. A pending unget of EOF needs no special case, because reaching
EOF leaves no lookahead to release.
*/
void release_lookahead()
{
release_lookahead_impl(std::integral_constant<bool, can_release_lookahead> {});
}
#if JSON_DIAGNOSTIC_POSITIONS
/// return the offset of the first character of the last read token; unlike
/// the token's parsed value, this accounts for escape sequences
+45 -16
View File
@@ -99,13 +99,22 @@ class parser
json_sax_dom_callback_parser<BasicJsonType, InputAdapterType> sdp(result, callback, allow_exceptions, &m_lexer);
sax_parse_internal(&sdp);
// in strict mode, input must be completely read
if (strict && (get_token() != token_type::end_of_input))
if (strict)
{
sdp.parse_error(m_lexer.get_position(),
m_lexer.get_token_string(),
parse_error::create(101, m_lexer.get_position(),
exception_message(token_type::end_of_input, "value"), nullptr));
// in strict mode, input must be completely read
if (get_token() != token_type::end_of_input)
{
sdp.parse_error(m_lexer.get_position(),
m_lexer.get_token_string(),
parse_error::create(101, m_lexer.get_position(),
exception_message(token_type::end_of_input, "value"), nullptr));
}
}
else
{
// the caller keeps using the input: position it right after
// the value by leaving the character that terminated it
m_lexer.release_lookahead();
}
// in case of an error, return a discarded value
@@ -127,12 +136,20 @@ class parser
json_sax_dom_parser<BasicJsonType, InputAdapterType> sdp(result, allow_exceptions, &m_lexer);
sax_parse_internal(&sdp);
// in strict mode, input must be completely read
if (strict && (get_token() != token_type::end_of_input))
if (strict)
{
sdp.parse_error(m_lexer.get_position(),
m_lexer.get_token_string(),
parse_error::create(101, m_lexer.get_position(), exception_message(token_type::end_of_input, "value"), nullptr));
// in strict mode, input must be completely read
if (get_token() != token_type::end_of_input)
{
sdp.parse_error(m_lexer.get_position(),
m_lexer.get_token_string(),
parse_error::create(101, m_lexer.get_position(), exception_message(token_type::end_of_input, "value"), nullptr));
}
}
else
{
// see above
m_lexer.release_lookahead();
}
// in case of an error, return a discarded value
@@ -165,12 +182,24 @@ class parser
(void)detail::is_sax_static_asserts<SAX, BasicJsonType> {};
const bool result = sax_parse_internal(sax);
// strict mode: next byte must be EOF
if (result && strict && (get_token() != token_type::end_of_input))
if (result)
{
return sax->parse_error(m_lexer.get_position(),
m_lexer.get_token_string(),
parse_error::create(101, m_lexer.get_position(), exception_message(token_type::end_of_input, "value"), nullptr));
if (strict)
{
// strict mode: next byte must be EOF
if (get_token() != token_type::end_of_input)
{
return sax->parse_error(m_lexer.get_position(),
m_lexer.get_token_string(),
parse_error::create(101, m_lexer.get_position(), exception_message(token_type::end_of_input, "value"), nullptr));
}
}
else
{
// the caller keeps using the input: position it right after
// the value by leaving the character that terminated it
m_lexer.release_lookahead();
}
}
return result;
-9
View File
@@ -186,15 +186,6 @@
#define JSON_NO_UNIQUE_ADDRESS
#endif
// Clang targeting MinGW does not survive the thread_local storage the copy
// constructor uses to bound its descent: every test that copies a value
// segfaults with clang 11.0.1 and clang 18.1.8, while the same tests pass with
// GCC targeting MinGW and with every other toolchain the library is tested on.
// Copying works the same way without the counter, only more slowly.
#if !defined(JSON_NO_THREAD_LOCAL) && defined(__clang__) && defined(__MINGW32__)
#define JSON_NO_THREAD_LOCAL 1
#endif
// disable documentation warnings on clang
#if defined(__clang__)
#pragma clang diagnostic push
+62 -655
View File
@@ -28,14 +28,14 @@
#pragma GCC diagnostic ignored "-Wignored-attributes"
#endif
#include <algorithm> // all_of, find, for_each, none_of
#include <algorithm> // all_of, find, for_each
#include <cstddef> // nullptr_t, ptrdiff_t, size_t
#include <functional> // hash, less
#include <initializer_list> // initializer_list
#ifndef JSON_NO_IO
#include <iosfwd> // istream, ostream
#endif // JSON_NO_IO
#include <iterator> // make_move_iterator, random_access_iterator_tag
#include <iterator> // random_access_iterator_tag
#include <memory> // unique_ptr
#include <string> // string, stoi, to_string
#include <utility> // declval, forward, move, pair, swap
@@ -821,626 +821,6 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
return j;
}
/// the number of levels an operation descends into before it finishes the
/// value below it without the call stack
static constexpr std::size_t nesting_depth_limit()
{
return 128;
}
#ifndef JSON_NO_THREAD_LOCAL
/*!
@brief how many levels the operation going on in this thread has descended into
Copying a value and comparing two values share this count. The library never
nests one inside the other - copying a value does not compare one, and
comparing two values does not copy them - and where user code nests them
anyway, sharing the count only ends a descent sooner than it had to, which
costs a little speed and is never wrong.
A byte is enough: the count never exceeds the limit by more than the single
level that notices the limit has been reached.
*/
static std::size_t& nesting_depth() noexcept
{
static thread_local std::size_t depth = 0; // NOLINT(misc-use-internal-linkage)
return depth;
}
#endif
/*!
@brief whether a descent must stop here and finish without the call stack
@a may_descend says whether the operator descends at all; it is a constant
at every call site, and is passed rather than tested by the caller so that
the test does not become a constant condition there, which MSVC reports as
C4127.
*/
static bool nesting_depth_exhausted(bool may_descend = true) noexcept
{
#ifdef JSON_NO_THREAD_LOCAL
// without a count of its own per thread, a descent cannot be bounded
// without racing another one, so none is made
static_cast<void>(may_descend);
return true;
#else
return !may_descend || nesting_depth() >= nesting_depth_limit();
#endif
}
/*!
@brief counts one level of a bounded descent for as long as it runs
The constructor taking the count is for callers that have looked it up
already to test it: reaching thread-local storage is not free, and the path
that is taken almost every time should reach it once rather than twice. The
other is for callers that cannot look it up - the comparison operators are
written as a macro, and a macro cannot use the preprocessor.
*/
class nesting_depth_guard
{
public:
explicit nesting_depth_guard(std::size_t& depth) noexcept
: m_depth(&depth)
{
++*m_depth;
}
nesting_depth_guard() noexcept
: m_depth(countable())
{
if (m_depth != nullptr)
{
++*m_depth;
}
}
~nesting_depth_guard()
{
if (m_depth != nullptr)
{
--*m_depth;
}
}
nesting_depth_guard(const nesting_depth_guard&) = delete;
nesting_depth_guard& operator=(const nesting_depth_guard&) = delete;
nesting_depth_guard(nesting_depth_guard&&) = delete;
nesting_depth_guard& operator=(nesting_depth_guard&&) = delete;
private:
/// @brief the count to keep, or nullptr where there is none to keep
static std::size_t* countable() noexcept
{
#ifdef JSON_NO_THREAD_LOCAL
return nullptr;
#else
return &nesting_depth();
#endif
}
std::size_t* m_depth;
};
/// an entry of the iterative deep copy's worklist: a structured value and
/// the value that is to become its copy
using copy_worklist_t = std::vector<std::pair<const basic_json*, basic_json*>>;
/// scratch space to build the key skeleton of an object copy in one go
using copy_scratch_t = std::vector<std::pair<typename object_t::key_type, basic_json>>;
/// @brief copy everything of @a src into @a dst but its type and value
static void copy_metadata(const basic_json& src, basic_json& dst)
{
// a custom base class is only required to be copy-constructible and
// move-assignable, so the copy has to go through a temporary
static_cast<json_base_class_t&>(dst) = json_base_class_t(static_cast<const json_base_class_t&>(src));
#if JSON_DIAGNOSTIC_POSITIONS
dst.start_position = src.start_position;
dst.end_position = src.end_position;
#else
static_cast<void>(src);
static_cast<void>(dst);
#endif
}
/*!
@brief copy the value of @a src into @a dst, which must not be structured
Objects and arrays are left alone: creating those is the one thing the copy
constructor and @ref copy_shallow do differently from one another, and it is
the reason copying a value can descend at all.
*/
/// @note inlined on purpose: both callers have already told an object or an
/// array apart from the rest, and letting the compiler fold that test
/// into this switch is worth a few percent when copying a value made
/// mostly of numbers
JSON_HEDLEY_ALWAYS_INLINE
static void copy_leaf_value(const basic_json& src, basic_json& dst)
{
switch (src.m_data.m_type)
{
case value_t::string:
{
dst.m_data.m_value = *src.m_data.m_value.string;
break;
}
case value_t::binary:
{
dst.m_data.m_value = *src.m_data.m_value.binary;
break;
}
case value_t::boolean:
{
dst.m_data.m_value = src.m_data.m_value.boolean;
break;
}
case value_t::number_integer:
{
dst.m_data.m_value = src.m_data.m_value.number_integer;
break;
}
case value_t::number_unsigned:
{
dst.m_data.m_value = src.m_data.m_value.number_unsigned;
break;
}
case value_t::number_float:
{
dst.m_data.m_value = src.m_data.m_value.number_float;
break;
}
case value_t::object:
case value_t::array:
case value_t::null:
case value_t::discarded:
default:
break;
}
}
/*!
@brief copy everything of @a src into the null value @a dst but the children
Objects and arrays are not copied here; they are appended to @a worklist to
be created later by @ref copy_iteratively. Until that happens, @a dst remains
a null value, so that a partially built copy can be destroyed at any point
without ever violating the class invariants.
*/
static void copy_shallow(const basic_json& src, basic_json& dst, copy_worklist_t& worklist)
{
copy_metadata(src, dst);
if (src.m_data.m_type == value_t::object || src.m_data.m_type == value_t::array)
{
// defer: dst stays a null value until its container exists
worklist.emplace_back(&src, &dst);
return;
}
copy_leaf_value(src, dst);
// only now that the value exists may the type be set: had the creation
// of the value thrown, dst would have been left as a valid null value
dst.m_data.m_type = src.m_data.m_type;
}
/// @brief create the copy of the array @a src in @a dst
/// @note structured elements are appended to @a worklist instead
static void copy_array_level(const basic_json& src, basic_json& dst, copy_worklist_t& worklist)
{
const array_t& src_array = *src.m_data.m_value.array;
// create all elements up front: growing the array afterwards could
// invalidate the pointers that are handed to the worklist
dst.m_data.m_value.array = create<array_t>(src_array.size(), basic_json());
auto dst_it = dst.m_data.m_value.array->begin();
for (auto src_it = src_array.cbegin(); src_it != src_array.cend(); ++src_it, ++dst_it)
{
copy_shallow(*src_it, *dst_it, worklist);
}
}
/// @brief create the copy of the object @a src in @a dst
/// @note structured values are appended to @a worklist instead
static void copy_object_level(const basic_json& src, basic_json& dst,
copy_worklist_t& worklist, copy_scratch_t& scratch)
{
const object_t& src_object = *src.m_data.m_value.object;
// build the complete key skeleton and hand it to the object's range
// constructor: adding the keys one by one would be quadratic for object
// types that are backed by a vector, such as nlohmann::ordered_map
scratch.clear();
scratch.reserve(src_object.size());
for (const auto& element : src_object)
{
scratch.emplace_back(element.first, basic_json());
}
dst.m_data.m_value.object = create<object_t>(std::make_move_iterator(scratch.begin()),
std::make_move_iterator(scratch.end()));
scratch.clear();
// pair every value of the copy with its counterpart in the original;
// both are enumerated in the same order for every object type with a
// deterministic order, so the lookup is only needed for exotic ones
auto src_it = src_object.cbegin();
for (auto& element : *dst.m_data.m_value.object)
{
if (JSON_HEDLEY_LIKELY(src_it != src_object.cend() && src_it->first == element.first))
{
copy_shallow(src_it->second, element.second, worklist);
++src_it;
}
else
{
const auto found = src_object.find(element.first);
JSON_ASSERT(found != src_object.cend());
copy_shallow(found->second, element.second, worklist);
}
}
}
/*!
@brief deep-copy the object or array @a src into this value without recursing
The values whose copy has not been created yet are kept on an explicit
worklist rather than on the call stack. This is only reached for values
nested deeper than @ref nesting_depth_limit levels, which is why it copies
every container by hand instead of letting the container do it: the fast
ways of doing so would descend into the elements and defeat the purpose.
*/
void copy_iteratively(const basic_json& src)
{
copy_worklist_t worklist;
copy_scratch_t scratch;
const basic_json* src_value = &src;
basic_json* dst_value = this;
for (;;)
{
if (src_value->m_data.m_type == value_t::array)
{
copy_array_level(*src_value, *dst_value, worklist);
}
else
{
copy_object_level(*src_value, *dst_value, worklist, scratch);
}
// the container is complete and will not be modified again
dst_value->set_parents();
if (worklist.empty())
{
break;
}
const auto& next = worklist.back();
src_value = next.first;
dst_value = next.second;
worklist.pop_back();
// the value stops being a null value exactly here
dst_value->m_data.m_type = src_value->m_data.m_type;
}
}
/*!
@brief copy one level of the object or array @a src into this value
The container copies its own elements, which is the fastest way to fill it.
Every element that is structured itself comes back to @ref copy_structured.
*/
void copy_level(const basic_json& src)
{
if (m_data.m_type == value_t::object)
{
m_data.m_value = *src.m_data.m_value.object;
}
else
{
m_data.m_value = *src.m_data.m_value.array;
}
set_parents();
}
/*!
@brief deep-copy the object or array @a src into this value
Copying a container copies its elements, so a value nested deeply enough
used to exhaust the call stack. The descent is bounded here: the first
@ref nesting_depth_limit levels are copied by the containers themselves, just
as they always were, and anything below that is copied without the call
stack by @ref copy_iteratively. Copying a value can therefore no longer
exhaust the stack, however deeply it is nested, just like destroying one
cannot since #1436.
Nothing has to be scanned or built by hand to reach that: a value that is
not nested deeper than the limit - all but a vanishing minority - is copied
exactly as it was before, and this whole detour costs it one counter.
@sa https://github.com/nlohmann/json/issues/5387
*/
void copy_structured(const basic_json& src)
{
#ifndef JSON_NO_THREAD_LOCAL
std::size_t& depth = nesting_depth();
if (JSON_HEDLEY_LIKELY(depth < nesting_depth_limit()))
{
const nesting_depth_guard guard(depth);
copy_level(src);
return;
}
#endif
// Finish this value without descending any further. It is completed
// before this returns, so a copy made by a custom base class - or by
// anything else that runs while a copy is going on - is unaffected by
// the copy it is nested in.
copy_iteratively(src);
}
/// the result of comparing two values, including values that cannot be
/// ordered at all, such as a discarded value or a NaN
enum class compare_result { less, equal, greater, unordered };
#if JSON_HAS_THREE_WAY_COMPARISON
/// @brief the ordering that @a result stands for
static std::partial_ordering to_partial_ordering(compare_result result) noexcept // *NOPAD*
{
switch (result)
{
case compare_result::less:
return std::partial_ordering::less;
case compare_result::greater:
return std::partial_ordering::greater;
case compare_result::equal:
return std::partial_ordering::equivalent;
case compare_result::unordered:
default:
return std::partial_ordering::unordered;
}
}
#endif
/*!
@brief compare two values that are not both an array or both an object
Such a pair is compared by the operators themselves, which cannot descend
into it and therefore cannot recurse.
That holds for a pair whose types differ as much as for a pair of leaves: an
array and an object are told apart by their types alone, because an operator
only ever descends into two values of the same type. So `==` reports them as
unequal without looking inside either, and an ordering falls back to the
order of the types - an object sorts before an array - exactly as it does
for a value that is not nested deeply enough to get here.
*/
template<bool Ordered>
static compare_result compare_leaves(const_reference lhs, const_reference rhs) noexcept
{
if (lhs == rhs)
{
return compare_result::equal;
}
return order_leaves(lhs, rhs, std::integral_constant<bool, Ordered> {});
}
/*!
@brief compare two object keys
An object compares its entries as pairs of a key and a value, so its keys
are compared exactly as std::pair compares them: with < where the objects
are being ordered, and with == where they are only checked for equality.
Note that this is not the object's own comparator, which for a vector-backed
object type such as nlohmann::ordered_map tells equality rather than order.
*/
static compare_result compare_keys(const typename object_t::key_type& lhs,
const typename object_t::key_type& rhs,
std::true_type /*ordered*/)
{
if (lhs < rhs)
{
return compare_result::less;
}
if (rhs < lhs)
{
return compare_result::greater;
}
return compare_result::equal;
}
/// @brief check two object keys for equality
static compare_result compare_keys(const typename object_t::key_type& lhs,
const typename object_t::key_type& rhs,
std::false_type /*ordered*/)
{
return lhs == rhs ? compare_result::equal : compare_result::unordered;
}
/// @brief tell apart two values that are not equal
/// @note only instantiated where the values are being ordered, as a key or
/// string type is not required to be ordered to be compared for equality
static compare_result order_leaves(const_reference lhs, const_reference rhs, std::true_type /*ordered*/) noexcept
{
if (lhs < rhs)
{
return compare_result::less;
}
if (rhs < lhs)
{
return compare_result::greater;
}
return compare_result::unordered;
}
/// @brief report two values as not equal without ordering them
static compare_result order_leaves(const_reference /*lhs*/, const_reference /*rhs*/, std::false_type /*ordered*/) noexcept
{
return compare_result::unordered;
}
/*!
@brief compare @a lhs and @a rhs without descending into them
Reached once a comparison has descended @ref nesting_depth_limit levels, so
that comparing values cannot exhaust the call stack however deeply they are
nested. The two values are walked in lockstep on an explicit stack and
compared lexicographically, element by element in the order the containers
enumerate them - which is how the container types this library ships compare
themselves: a std::map enumerates its entries in key order, and
nlohmann::ordered_map in insertion order. An object type that enumerates its
entries in an unspecified order, such as std::unordered_map, compares them
pairwise instead; the difference could only ever show below the bound.
Note that the stack this walks with is allocated, while the comparison
operators are noexcept and the container comparison this replaces allocated
nothing. Failing that allocation therefore ends the process rather than
throwing. It only arises for values nested past the bound, and only when
memory has run out - where the same comparison used to exhaust the call
stack instead - but it is a way to fail that the operators did not have.
*/
template<bool Ordered>
static compare_result compare_iteratively(const_reference lhs, const_reference rhs,
const bool unordered_compares_equal) noexcept
{
/// a pair of containers being compared in lockstep
struct frame
{
const basic_json* lhs_value{nullptr};
const basic_json* rhs_value{nullptr};
typename array_t::const_iterator lhs_array_it{};
typename array_t::const_iterator rhs_array_it{};
typename object_t::const_iterator lhs_object_it{};
typename object_t::const_iterator rhs_object_it{};
};
std::vector<frame> stack;
const basic_json* left = &lhs;
const basic_json* right = &rhs;
for (;;)
{
const auto type = left->m_data.m_type;
if (type == right->m_data.m_type && (type == value_t::array || type == value_t::object))
{
// descend: the elements decide, and are compared further down
stack.emplace_back();
frame& pushed = stack.back();
pushed.lhs_value = left;
pushed.rhs_value = right;
if (type == value_t::array)
{
pushed.lhs_array_it = left->m_data.m_value.array->cbegin();
pushed.rhs_array_it = right->m_data.m_value.array->cbegin();
}
else
{
pushed.lhs_object_it = left->m_data.m_value.object->cbegin();
pushed.rhs_object_it = right->m_data.m_value.object->cbegin();
}
}
else
{
const compare_result result = compare_leaves<Ordered>(*left, *right);
// Values that cannot be ordered - a NaN, say - end an ordered
// comparison for std::lexicographical_compare_three_way, but
// std::lexicographical_compare treats them as equivalent and
// carries on with the next element. Both are reproduced here,
// so that a value nested too deeply to descend into compares
// exactly as one that is not.
if (result != compare_result::equal &&
!(unordered_compares_equal && result == compare_result::unordered))
{
return result;
}
}
// walk back up past the containers that are exhausted, then take the
// next pair of elements from the innermost one that is not
for (;;)
{
if (stack.empty())
{
return compare_result::equal;
}
frame& current = stack.back();
const bool is_object = current.lhs_value->m_data.m_type == value_t::object;
const bool lhs_done = is_object
? current.lhs_object_it == current.lhs_value->m_data.m_value.object->cend()
: current.lhs_array_it == current.lhs_value->m_data.m_value.array->cend();
const bool rhs_done = is_object
? current.rhs_object_it == current.rhs_value->m_data.m_value.object->cend()
: current.rhs_array_it == current.rhs_value->m_data.m_value.array->cend();
if (lhs_done || rhs_done)
{
// whichever ran out first holds the smaller container; if
// both did, they are equal and the container above decides
if (lhs_done != rhs_done)
{
return lhs_done ? compare_result::less : compare_result::greater;
}
stack.pop_back();
continue;
}
if (is_object)
{
// an entry is a key and a value, and the key decides first
const compare_result key_result =
compare_keys(current.lhs_object_it->first, current.rhs_object_it->first,
std::integral_constant<bool, Ordered> {});
if (key_result != compare_result::equal)
{
return key_result;
}
left = &(current.lhs_object_it->second);
right = &(current.rhs_object_it->second);
++current.lhs_object_it;
++current.rhs_object_it;
}
else
{
left = &(*current.lhs_array_it);
right = &(*current.rhs_array_it);
++current.lhs_array_it;
++current.rhs_array_it;
}
break;
}
}
}
public:
//////////////////////////
// JSON parser callback //
@@ -1820,15 +1200,60 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
// check of passed value is valid
other.assert_invariant();
if (m_data.m_type == value_t::object || m_data.m_type == value_t::array)
switch (m_data.m_type)
{
// copying the container directly would call this constructor again
// for every element, once per nesting level
copy_structured(other);
}
else
{
copy_leaf_value(other, *this);
case value_t::object:
{
m_data.m_value = *other.m_data.m_value.object;
break;
}
case value_t::array:
{
m_data.m_value = *other.m_data.m_value.array;
break;
}
case value_t::string:
{
m_data.m_value = *other.m_data.m_value.string;
break;
}
case value_t::boolean:
{
m_data.m_value = other.m_data.m_value.boolean;
break;
}
case value_t::number_integer:
{
m_data.m_value = other.m_data.m_value.number_integer;
break;
}
case value_t::number_unsigned:
{
m_data.m_value = other.m_data.m_value.number_unsigned;
break;
}
case value_t::number_float:
{
m_data.m_value = other.m_data.m_value.number_float;
break;
}
case value_t::binary:
{
m_data.m_value = *other.m_data.m_value.binary;
break;
}
case value_t::null:
case value_t::discarded:
default:
break;
}
set_parents();
@@ -4233,7 +3658,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
// because any negative signed value is smaller than any unsigned value.
// Otherwise, the non-negative signed value is cast to unsigned before the
// comparison to avoid wraparound.
#define JSON_IMPLEMENT_OPERATOR(op, null_result, unordered_result, default_result, deep_result, may_descend) \
#define JSON_IMPLEMENT_OPERATOR(op, null_result, unordered_result, default_result) \
const auto lhs_type = lhs.type(); \
const auto rhs_type = rhs.type(); \
\
@@ -4242,25 +3667,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
switch (lhs_type) \
{ \
case value_t::array: \
{ \
if (JSON_HEDLEY_UNLIKELY(nesting_depth_exhausted(may_descend))) \
{ \
return (deep_result); \
} \
const nesting_depth_guard guard; \
return (*lhs.m_data.m_value.array) op (*rhs.m_data.m_value.array); \
} \
\
\
case value_t::object: \
{ \
if (JSON_HEDLEY_UNLIKELY(nesting_depth_exhausted(may_descend))) \
{ \
return (deep_result); \
} \
const nesting_depth_guard guard; \
return (*lhs.m_data.m_value.object) op (*rhs.m_data.m_value.object); \
} \
\
\
case value_t::null: \
return (null_result); \
\
@@ -4360,8 +3771,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
#pragma GCC diagnostic ignored "-Wfloat-equal"
#endif
const_reference lhs = *this;
JSON_IMPLEMENT_OPERATOR( ==, true, false, false,
compare_iteratively<false>(lhs, rhs, false) == compare_result::equal, true)
JSON_IMPLEMENT_OPERATOR( ==, true, false, false)
#ifdef __GNUC__
#pragma GCC diagnostic pop
#endif
@@ -4386,8 +3796,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
JSON_IMPLEMENT_OPERATOR(<=>, // *NOPAD*
std::partial_ordering::equivalent,
std::partial_ordering::unordered,
lhs_type <=> rhs_type, // *NOPAD*
to_partial_ordering(compare_iteratively<true>(lhs, rhs, false)), true)
lhs_type <=> rhs_type) // *NOPAD*
}
/// @brief comparison: 3-way
@@ -4454,8 +3863,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wfloat-equal"
#endif
JSON_IMPLEMENT_OPERATOR( ==, true, false, false,
compare_iteratively<false>(lhs, rhs, false) == compare_result::equal, true)
JSON_IMPLEMENT_OPERATOR( ==, true, false, false)
#ifdef __GNUC__
#pragma GCC diagnostic pop
#endif
@@ -4511,8 +3919,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
// default_result is used if we cannot compare values. In that case,
// we compare types. Note we have to call the operator explicitly,
// because MSVC has problems otherwise.
JSON_IMPLEMENT_OPERATOR( <, false, false, operator<(lhs_type, rhs_type),
compare_iteratively<true>(lhs, rhs, true) == compare_result::less, false)
JSON_IMPLEMENT_OPERATOR( <, false, false, operator<(lhs_type, rhs_type))
}
/// @brief comparison: less than
File diff suppressed because it is too large Load Diff
-51
View File
@@ -216,57 +216,6 @@ TEST_CASE("controlled bad_alloc")
CHECK_THROWS_AS(my_json(s), std::bad_alloc&);
next_construct_fails = false;
}
SECTION("basic_json(const basic_json&) of a deeply nested value (#5387)")
{
// Copying a value nested deeper than the descent bound builds the
// copy from the top down: every value whose own copy has not been
// made yet stays a null value until it is. Failing an allocation
// part-way through is what proves such a half-built copy can still
// be destroyed.
//
// Which path the failure lands in depends on the build: the first
// allocation of a copy belongs to the outermost level, so here it
// is the descending one. Built with JSON_NO_THREAD_LOCAL - as the
// ci_test_no_thread_local target builds the whole suite - no
// descent is made at all and the very same failure lands in the
// iterative path instead, part-way through its worklist.
const auto check_deep_copy = [](bool objects)
{
CAPTURE(objects);
next_construct_fails = false;
// deeper than the 128 levels the copy constructor descends into
const std::size_t depth = 300;
my_json j = 1;
for (std::size_t i = 0; i < depth; ++i)
{
if (objects)
{
my_json wrapper = my_json::object();
wrapper["a"] = std::move(j);
j = std::move(wrapper);
}
else
{
j = my_json::array({std::move(j)});
}
}
// NOLINTNEXTLINE(performance-unnecessary-copy-initialization): the copy is what is tested
CHECK_NOTHROW(my_json(j));
next_construct_fails = true;
// NOLINTNEXTLINE(performance-unnecessary-copy-initialization): the copy is what is tested
CHECK_THROWS_AS(my_json(j), std::bad_alloc&);
next_construct_fails = false;
};
check_deep_copy(false);
check_deep_copy(true);
}
}
}
+173
View File
@@ -14,10 +14,15 @@ using nlohmann::json;
using namespace nlohmann::literals; // NOLINT(google-build-using-namespace)
#endif
#include <cstddef>
#include <iostream>
#include <iterator>
#include <sstream>
#include <streambuf>
#include <string>
#include <utility>
#include <valarray>
#include <vector>
#if defined(_WIN32)
#define NOMINMAX
@@ -219,6 +224,58 @@ class proxy_iterator
iterator* m_it = nullptr;
};
// A streambuf that keeps no get area at all and refuses every putback: with an
// empty get area, sungetc() always ends up in pbackfail(). Used to check that
// the character terminating a number is left in the input without relying on
// the streambuf being able to put a consumed character back.
class no_putback_streambuf : public std::streambuf
{
public:
explicit no_putback_streambuf(std::string s) : m_data(std::move(s)) {}
protected:
// peek at the next character without consuming it
int_type underflow() override
{
if (m_pos >= m_data.size())
{
return traits_type::eof();
}
return traits_type::to_int_type(m_data[m_pos]);
}
// consume the next character
int_type uflow() override
{
if (m_pos >= m_data.size())
{
return traits_type::eof();
}
return traits_type::to_int_type(m_data[m_pos++]);
}
int_type pbackfail(int_type /*c*/) override
{
return traits_type::eof();
}
private:
std::string m_data;
std::size_t m_pos = 0;
};
// read the characters that are left in a stream
std::string remaining(std::istream& is)
{
std::string result;
char c = 0;
while (is.get(c))
{
result += c;
}
return result;
}
// JSON_HAS_CPP_20
#if defined(__cpp_char8_t)
bool check_utf8()
@@ -1181,6 +1238,122 @@ TEST_CASE("deserialization")
}
}
SECTION("stream position after extraction (#5340)")
{
SECTION("a number does not consume the character that terminates it")
{
// a number is only terminated by the character following it; that
// character must be given back so the stream is positioned right
// after the value
const std::vector<std::pair<std::string, std::string>> tests =
{
{"1true", "true"},
{"1[2]", "[2]"},
{"1{}", "{}"},
{R"(1"a")", R"("a")"},
{"1 true", " true"},
{"12,", ","},
{"-0.5e3x", "x"},
{"1null", "null"}
};
for (const auto& test : tests)
{
CAPTURE(test.first);
std::istringstream ss(test.first);
json j;
ss >> j;
CHECK(j == json::parse(test.first.substr(0, test.first.size() - test.second.size())));
CHECK(remaining(ss) == test.second);
}
}
SECTION("values that are self-delimiting are unaffected")
{
const std::vector<std::pair<std::string, std::string>> tests =
{
{"truefalse", "false"},
{"[1][2]", "[2]"},
{R"({"a":1}{"b":2})", R"({"b":2})"},
{R"("a""b")", R"("b")"},
{"null null", " null"}
};
for (const auto& test : tests)
{
CAPTURE(test.first);
std::istringstream ss(test.first);
json j;
ss >> j;
CHECK(remaining(ss) == test.second);
}
}
SECTION("a number at the end of the input leaves nothing behind")
{
for (const std::string s :
{"1", "12", "-3.5e2", " 7 "
})
{
CAPTURE(s);
std::istringstream ss(s);
json j;
ss >> j;
CHECK(remaining(ss).find_first_not_of(" \t\n\r") == std::string::npos);
}
}
SECTION("repeated extraction of concatenated values")
{
std::istringstream ss(R"(1true[2]3"x"{"a":4}5)");
const std::vector<json> expected =
{
json(1), json(true), json::parse("[2]"), json(3),
json("x"), json::parse(R"({"a":4})"), json(5)
};
for (const auto& e : expected)
{
json j;
ss >> j;
CHECK(j == e);
}
}
SECTION("sax_parse with strict == false")
{
std::istringstream ss("1true");
SaxEventLogger l;
CHECK(json::sax_parse(ss, &l, nlohmann::detail::input_format_t::json, false));
CHECK(l.events.size() == 1);
CHECK(l.events[0] == "number_unsigned(1)");
CHECK(remaining(ss) == "true");
}
SECTION("strict parsing still rejects trailing data")
{
std::istringstream ss("1true");
json _;
CHECK_THROWS_WITH_AS(_ = json::parse(ss),
"[json.exception.parse_error.101] parse error at line 1, column 5: syntax error while parsing value - unexpected true literal; expected end of input", json::parse_error&);
std::istringstream ss2("1true");
CHECK_FALSE(json::accept(ss2));
}
SECTION("a streambuf that cannot put back is not needed")
{
// the terminating character is never consumed, so no putback
// position is required
no_putback_streambuf buf("1true");
std::istream is(&buf);
json j;
is >> j;
CHECK(j == json(1));
CHECK(remaining(is) == "true");
}
}
// build with C++20
// JSON_HAS_CPP_20
#if defined(__cpp_char8_t)
-66
View File
@@ -68,72 +68,6 @@ TEST_CASE("Better diagnostics with positions")
CHECK(j.end_pos() == root.size());
}
SECTION("copying keeps the positions of nested values (#5387)")
{
// Values nested deeper than the copy constructor's descent bound are
// copied without the call stack, on a path that has to carry the
// positions over itself; shallower ones copy their containers, which
// bring the positions along. Both sides of the bound are checked here.
const auto check_copy = [](std::size_t depth, bool objects)
{
CAPTURE(depth)
CAPTURE(objects)
const std::string opening = objects ? R"({"a":)" : "[";
const std::string closing = objects ? "}" : "]";
std::string text;
for (std::size_t i = 0; i < depth; ++i)
{
text += opening;
}
text += "12";
for (std::size_t i = 0; i < depth; ++i)
{
text += closing;
}
const json original = json::parse(text);
const json copy(original); // NOLINT(performance-unnecessary-copy-initialization)
const json* o = &original;
const json* c = &copy;
for (std::size_t level = 0; level <= depth; ++level)
{
CAPTURE(level)
REQUIRE(c->start_pos() == o->start_pos());
REQUIRE(c->end_pos() == o->end_pos());
if (level < depth)
{
o = objects ? &o->at("a") : &o->at(0);
c = objects ? &c->at("a") : &c->at(0);
}
}
};
const auto check_arrays = [&check_copy](std::size_t depth)
{
check_copy(depth, false);
};
const auto check_objects = [&check_copy](std::size_t depth)
{
check_copy(depth, true);
};
check_arrays(1);
check_arrays(127);
check_arrays(128);
check_arrays(129);
check_arrays(300);
check_objects(1);
check_objects(127);
check_objects(128);
check_objects(129);
check_objects(300);
}
SECTION("JSON patch add to primitive parent (#4292)")
{
// the JSON Patch "add" target /foo/bar/baz has a string parent
-57
View File
@@ -273,62 +273,5 @@ TEST_CASE("Regression tests for extended diagnostics")
CHECK(j1["numbers"]["two"] == 2);
CHECK(j1["string"] == "t");
}
SECTION("Regression test for issue #5387 - copying keeps the parents of nested values")
{
// A value nested deeper than the copy constructor's descent bound is
// copied without the call stack. Every container that path creates has
// to have the parents of its children set, or the JSON Pointer in the
// diagnostic is cut short.
const std::size_t depth = 300;
SECTION("objects")
{
json j = "not a number";
std::string pointer;
for (std::size_t i = 0; i < depth; ++i)
{
j = json{{"a", j}};
pointer += "/a";
}
json const copy(j); // NOLINT(performance-unnecessary-copy-initialization)
const json* inner = &copy;
for (std::size_t i = 0; i < depth; ++i)
{
inner = &inner->at("a");
}
std::string const expected = "[json.exception.type_error.302] (" + pointer + ") type must be number, but is string";
int i = 0;
CHECK_THROWS_WITH_AS(i = inner->get<int>(), expected.c_str(), json::type_error);
CHECK(i == 0);
}
SECTION("arrays")
{
json j = "not a number";
std::string pointer;
for (std::size_t i = 0; i < depth; ++i)
{
j = json::array({j});
pointer += "/0";
}
json const copy(j); // NOLINT(performance-unnecessary-copy-initialization)
const json* inner = &copy;
for (std::size_t i = 0; i < depth; ++i)
{
inner = &inner->at(0);
}
std::string const expected = "[json.exception.type_error.302] (" + pointer + ") type must be number, but is string";
int i = 0;
CHECK_THROWS_WITH_AS(i = inner->get<int>(), expected.c_str(), json::type_error);
CHECK(i == 0);
}
}
}
-199
View File
@@ -12,7 +12,6 @@
using nlohmann::json;
#include <algorithm>
#include <string>
TEST_CASE("tests on very large JSONs")
{
@@ -28,201 +27,3 @@ TEST_CASE("tests on very large JSONs")
}
}
namespace
{
// Descend a chain of single-element containers and return the value at its end,
// reporting the number of levels traversed in @a depth.
//
// The values in the test case below are nested far deeper than the call stack
// can follow, so they must not be inspected with operator== or dump(): both are
// still recursive and would overflow the stack themselves.
const json* innermost_value(const json& j, std::size_t& depth)
{
const json* current = &j;
depth = 0;
while ((current->is_array() || current->is_object()) && !current->empty())
{
current = current->is_array()
? &current->front()
: &current->begin().value();
++depth;
}
return current;
}
} // namespace
TEST_CASE("tests on deeply nested JSONs")
{
// deep enough to exhaust the call stack, but small enough to stay cheap:
// parsing is iterative, so building the values below costs little
const std::size_t depth = 100000;
SECTION("issue #5387 - stack overflow in the copy constructor")
{
SECTION("array")
{
const json j = json::parse(std::string(depth, '[') + '0' + std::string(depth, ']'));
const json copy(j); // NOLINT(performance-unnecessary-copy-initialization): the copy is what is tested
std::size_t copy_depth = 0;
CHECK(*innermost_value(copy, copy_depth) == 0);
CHECK(copy_depth == depth);
}
SECTION("object")
{
std::string s;
s.reserve((6 * depth) + 1);
for (std::size_t i = 0; i < depth; ++i)
{
s += "{\"a\":";
}
s += '1';
s.append(depth, '}');
const json j = json::parse(s);
const json copy(j); // NOLINT(performance-unnecessary-copy-initialization): the copy is what is tested
std::size_t copy_depth = 0;
CHECK(*innermost_value(copy, copy_depth) == 1);
CHECK(copy_depth == depth);
}
SECTION("copy assignment")
{
// operator=(basic_json) takes its argument by value, so the deep
// copy happens in the copy constructor
const json j = json::parse(std::string(depth, '[') + '0' + std::string(depth, ']'));
json target;
target = j;
std::size_t target_depth = 0;
CHECK(*innermost_value(target, target_depth) == 0);
CHECK(target_depth == depth);
}
SECTION("depths around the bound of the recursive descent")
{
// The copy constructor descends into a bounded number of levels and
// completes whatever is below that without the call stack. Cover
// every depth around that bound, so that the two ways of copying
// are known to meet cleanly - wherever the bound is set.
for (std::size_t d = 1; d <= 300; ++d)
{
CAPTURE(d);
const json array = json::parse(std::string(d, '[') + '0' + std::string(d, ']'));
const json array_copy(array); // NOLINT(performance-unnecessary-copy-initialization): the copy is what is tested
std::size_t array_depth = 0;
CHECK(*innermost_value(array_copy, array_depth) == 0);
CHECK(array_depth == d);
std::string object_text;
for (std::size_t i = 0; i < d; ++i)
{
object_text += "{\"a\":";
}
object_text += '1';
object_text.append(d, '}');
const json object = json::parse(object_text);
const json object_copy(object); // NOLINT(performance-unnecessary-copy-initialization): the copy is what is tested
std::size_t object_depth = 0;
CHECK(*innermost_value(object_copy, object_depth) == 1);
CHECK(object_depth == d);
}
}
SECTION("a value that is deep in one place only")
{
json j = json::object();
j["shallow"] = 1;
j["deep"] = json::parse(std::string(depth, '[') + '0' + std::string(depth, ']'));
j["also_shallow"] = json::array({1, 2, 3});
const json copy(j);
CHECK(copy["shallow"] == 1);
CHECK(copy["also_shallow"] == json::array({1, 2, 3}));
std::size_t deep_depth = 0;
CHECK(*innermost_value(copy["deep"], deep_depth) == 0);
CHECK(deep_depth == depth);
}
SECTION("comparing")
{
// Comparing used to descend once per level, and an ordered
// comparison used to compare every pair of elements twice, once in
// each direction, which took exponentially long in the nesting
// depth. Both are gone: these finish in milliseconds, where the
// second used to take longer than anyone would wait even for a
// value nested only a few dozen levels deep.
const std::string text = std::string(depth, '[') + '0' + std::string(depth, ']');
const json j = json::parse(text);
const json same = json::parse(text);
const json larger = json::parse(std::string(depth, '[') + '1' + std::string(depth, ']'));
CHECK(j == same);
CHECK_FALSE(j == larger);
CHECK(j != larger);
CHECK(j < larger);
CHECK_FALSE(larger < j);
CHECK(larger > j);
CHECK(j <= same);
CHECK(j >= same);
// a value that ends earlier is the smaller one
const json shorter = json::parse(std::string(depth - 1, '[') + '0' + std::string(depth - 1, ']'));
CHECK_FALSE(j == shorter);
}
SECTION("comparing objects")
{
std::string text;
text.reserve((6 * depth) + 1);
for (std::size_t i = 0; i < depth; ++i)
{
text += "{\"a\":";
}
text += '1';
text.append(depth, '}');
const json j = json::parse(text);
const json same = json::parse(text);
CHECK(j == same);
CHECK_FALSE(j != same);
CHECK(j <= same);
CHECK(j >= same);
}
SECTION("the copy is independent of the original")
{
const json j = json::parse(std::string(depth, '[') + '0' + std::string(depth, ']'));
json copy(j);
// reach the innermost value without recursing and replace it
json* current = &copy;
while (current->is_array() && !current->empty())
{
current = &current->front();
}
*current = 42;
std::size_t unused = 0;
CHECK(*innermost_value(copy, unused) == 42);
CHECK(*innermost_value(j, unused) == 0);
}
}
}
-34
View File
@@ -81,37 +81,3 @@ TEST_CASE("regression test for issue #3732 - iteration_proxy_value<iter_impl<ord
};
static_cast<void>(fn);
}
TEST_CASE("copying an ordered_json with nested values")
{
// ordered_map is backed by a vector, so copying an object that has
// structured values takes a different route than copying a std::map-backed
// one; see https://github.com/nlohmann/json/issues/5387
ordered_json oj;
oj["z"] = 1;
oj["a"]["y"] = 2;
oj["a"]["b"]["x"] = 3;
oj["m"] = {1, 2, {{"w", 4}}};
const ordered_json copy(oj);
SECTION("the copy is equal to the original")
{
CHECK(copy == oj);
CHECK(copy.dump() == oj.dump());
}
SECTION("the key order is preserved at every level")
{
CHECK(copy.dump() == R"({"z":1,"a":{"y":2,"b":{"x":3}},"m":[1,2,{"w":4}]})");
}
SECTION("the copy is independent of the original")
{
ordered_json mutated(oj);
mutated["a"]["b"]["x"] = 99;
CHECK(oj["a"]["b"]["x"] == 3);
CHECK(mutated["a"]["b"]["x"] == 99);
}
}
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff