Compare commits

..
Author SHA1 Message Date
Niels Lohmann 964b433dc7 Force inlining of the serializer helpers only in optimized builds
The seven serializer helpers marked JSON_HEDLEY_ALWAYS_INLINE are inlined
at -O0 as well, and GCC and Clang give every inlined copy its own stack
slots there. That made the recursive frame of dump_container() 2304 bytes
with Clang 21 (2048 with GCC 16), against 480 bytes in 3.12.0, so dump()
of a value nested beyond the default recursion depth limit needed 288 KiB
of stack in an unoptimized build and crashed in a 128 KiB musl thread.

Define JSON_INTERNAL_SERIALIZER_INLINE as JSON_HEDLEY_ALWAYS_INLINE when
__OPTIMIZE__ (or _MSC_VER) is defined and as plain inline otherwise, and
use it for these helpers. The frame drops to 320 bytes and dump() needs
48 KiB at the default limit. The -O2 object code is unchanged with Clang 21
and GCC 16.

Update the stack tables in json_recursion_depth_limit.md and
untrusted_input.md: copying and conversion are now the most demanding
operations at -O0.

Fixes #6116.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-11-13 10:25:10 +01:00
11 changed files with 135 additions and 578 deletions

No files matched your search

@@ -27,7 +27,7 @@ comparison of two [`json_view`](../basic_json_view/index.md) documents has a fix
and does not depend on it either. and does not depend on it either.
The call stack that these operations need grows with the limit: roughly *a few kilobytes plus a cost per level times the The call stack that these operations need grows with the limit: roughly *a few kilobytes plus a cost per level times the
limit*, where the cost per level is up to about 2.3 KiB in an unoptimized and up to about 0.5 KiB in an optimized build limit*, where the cost per level is up to about 1.8 KiB in an unoptimized and up to about 0.5 KiB in an optimized build
(see [Stack needed](#stack-needed)). The default of `128` levels is tuned for threads with ordinary stacks of a few (see [Stack needed](#stack-needed)). The default of `128` levels is tuned for threads with ordinary stacks of a few
hundred kilobytes or more. Set a lower limit if your threads have small stacks (RTOS tasks, fibers, threads of the musl hundred kilobytes or more. Set a lower limit if your threads have small stacks (RTOS tasks, fibers, threads of the musl
C C
@@ -67,7 +67,7 @@ When the macro is not defined, the library defines it to its default value.
the levels that are descended into are counted, so the count never exceeds the limit, not even when user code (a the levels that are descended into are counted, so the count never exceeds the limit, not even when user code (a
copy constructor of a custom base class, say) copies a value while a copy is going on. A byte holds `255` at copy constructor of a custom base class, say) copies a value while a copy is going on. A byte holds `255` at
most, and the limit stays below that, so that the count cannot wrap around. (More levels would hardly be most, and the limit stays below that, so that the count cannot wrap around. (More levels would hardly be
useful: `254` levels need up to 576 KiB of stack in an unoptimized build already, twice as much as the useful: `254` levels need up to 448 KiB of stack in an unoptimized build already, twice as much as the
default.) default.)
### Stack needed ### Stack needed
@@ -83,7 +83,7 @@ Unoptimized (`-O0`):
| Operation | limit 16 | limit 32 | limit 64 | limit 128 | limit 254 | | Operation | limit 16 | limit 32 | limit 64 | limit 128 | limit 254 |
|---|---:|---:|---:|---:|---:| |---|---:|---:|---:|---:|---:|
| `dump()` and `dump(indent)`, `operator<<` | 48 / 32 | 80 / 64 | 144 / 128 | 288 / 256 | 576 / 512 | | `dump()` and `dump(indent)`, `operator<<` | 16 / 16 | 16 / 16 | 32 / 32 | 48 / 48 | 96 / 96 |
| copy construction and assignment | 32 / 16 | 48 / 32 | 112 / 80 | 224 / 144 | 432 / 304 | | copy construction and assignment | 32 / 16 | 48 / 32 | 112 / 80 | 224 / 144 | 432 / 304 |
| `==` | 16 / 16 | 16 / 16 | 16 / 16 | 48 / 32 | 96 / 48 | | `==` | 16 / 16 | 16 / 16 | 16 / 16 | 48 / 32 | 96 / 48 |
| `<=>` (C++20) | 16 / 16 | 16 / 16 | 32 / 32 | 64 / 64 | 144 / 128 | | `<=>` (C++20) | 16 / 16 | 16 / 16 | 32 / 32 | 64 / 64 | 144 / 128 |
@@ -123,23 +123,24 @@ The rule of thumb is
> stack needed ≈ a few KiB + cost per level × limit > stack needed ≈ a few KiB + cost per level × limit
with a cost per level of up to 2.3 KiB (`dump` at `-O0` with Clang; 2.1 KiB with GCC), 1.7 KiB for copying and merging with a cost per level of up to 1.8 KiB (conversion to another `basic_json` type at `-O0` with Clang; 1.3 KiB with
(1.3 KiB with GCC), and 0.1 to 0.5 KiB for most other operations at `-O0`; and 0.5 KiB or less at `-O2`. The numbers GCC), 1.7 KiB for copying and merging (1.3 KiB with GCC), 0.4 KiB for `dump`, and 0.1 to 0.5 KiB for most other
come from the difference between the limits `128` and `254` in the tables, so they are rough: other compilers, operations at `-O0`; and 0.5 KiB or less at `-O2`. The numbers come from the difference between the limits `128` and
other targets (the frames of x86-64 code are about as large), other standard libraries, `JSON_DIAGNOSTICS`, and `254` in the tables, so they are rough: other compilers, other targets (the frames of x86-64 code are about as large),
sanitizers change them. Measure with your own tool chain (see the example below) rather than relying on the tables. other standard libraries, `JSON_DIAGNOSTICS`, and sanitizers change them. Measure with your own tool chain (see the
The stack that your own code and the operating system need comes on top of these numbers. example below) rather than relying on the tables. The stack that your own code and the operating system need comes on
top of these numbers.
Which limit to choose, for the stack that you can spare for the library (the largest limit of those measured Which limit to choose, for the stack that you can spare for the library (the largest limit of those measured
for which the slowest operation needs at most that much): for which the slowest operation needs at most that much):
| Stack for the library | optimized (`-O2`) | unoptimized (`-O0`) | | Stack for the library | optimized (`-O2`) | unoptimized (`-O0`) |
|---|---:|---:| |---|---:|---:|
| 32 KiB | 64 | 0 | | 32 KiB | 64 | 16 |
| 64 KiB | 128 | 16 | | 64 KiB | 128 | 32 |
| 128 KiB (the default stack of threads on musl) | 254 | 32 | | 128 KiB (the default stack of threads on musl) | 254 | 64 |
| 256 KiB | 254 | 64 | | 256 KiB | 254 | 128 |
| 512 KiB | 254 | 128 | | 512 KiB | 254 | 254 |
| 1 MiB or more | 254 | 254 | | 1 MiB or more | 254 | 254 |
### Speed ### Speed
@@ -64,13 +64,13 @@ stack for the nesting.)
The call stack still needs room for the levels that are recursed into, and the default limit assumes a stack of ordinary The call stack still needs room for the levels that are recursed into, and the default limit assumes a stack of ordinary
size. The stack that an operation needs is roughly proportional to the limit; with the default of 128 levels, the most size. The stack that an operation needs is roughly proportional to the limit; with the default of 128 levels, the most
demanding operation (`dump` in an unoptimized build) needed up to 288 KiB, and an optimized build needed 48 KiB or less, demanding operations (copying a value and converting it to another `basic_json` type, in an unoptimized build) needed
in measurements on macOS with arm64. The most demanding operation needed this much stack at the limits `0`, `32`, up to 224 KiB, and an optimized build needed 48 KiB or less, in measurements on macOS with arm64. The most demanding
`128`, and `254`: operation needed this much stack at the limits `0`, `32`, `128`, and `254`:
| Stack needed, in KiB | limit 0 | limit 32 | limit 128 (default) | limit 254 | | Stack needed, in KiB | limit 0 | limit 32 | limit 128 (default) | limit 254 |
|---|---:|---:|---:|---:| |---|---:|---:|---:|---:|
| unoptimized build (`-O0`) | 16 or less | 80 | 288 | 576 | | unoptimized build (`-O0`) | 16 or less | 64 | 224 | 448 |
| optimized build (`-O2`) | 16 or less | 16 or less | 48 | 112 | | optimized build (`-O2`) | 16 or less | 16 or less | 48 | 112 |
A thread with a smaller stack (a task of a real-time operating system, a fiber, or a thread of the musl C library, whose A thread with a smaller stack (a task of a real-time operating system, a fiber, or a thread of the musl C library, whose
-2
View File
@@ -140,8 +140,6 @@ Exceptions have ids 1xx.
This error indicates a syntax error while deserializing a JSON text. The error message describes that an unexpected token (character) was encountered, and the member `byte` indicates the error position. This error indicates a syntax error while deserializing a JSON text. The error message describes that an unexpected token (character) was encountered, and the member `byte` indicates the error position.
The message ends with the last read token (`last read: '...'`), with control characters written as `<U+XXXX>`. A token of more than 8192 characters is shown by its first and its last 4096 characters, separated by `...`.
!!! failure "Example message" !!! failure "Example message"
Input ended prematurely: Input ended prematurely:
+32 -109
View File
@@ -110,164 +110,87 @@ inline const char* empty_input_message() noexcept
return "attempting to parse an empty input; check that your input string or stream contains the expected JSON"; return "attempting to parse an empty input; check that your input string or stream contains the expected JSON";
} }
////////////////////
// input adapters //
////////////////////
#ifndef JSON_NO_IO
/*! /*!
@brief the characters of the last read token, kept for error messages
A growable array of characters for the part of a token that an input adapter A growable array of characters for the part of a token that an input adapter
copied out of its window, or that the lexer copies while it scans input it copied out of its window: unlike std::vector<char>::push_back(), appending a
cannot read again: unlike std::vector::push_back(), appending a character character takes no reference to a local, and is a few instructions when there
takes no reference to a local, and is a few instructions when there is room. is room.
The token is only needed for error messages, which show its first head()
characters and its last head() characters (see lexer::get_token_string). The
buffer therefore keeps the first head() and at least the last tail()
characters: when it is full and holds more than limit() characters, it drops
the characters in between instead of growing. That happens where it would
otherwise reallocate, so appending stays as cheap as before, and a long string
or a long run of whitespace read from a stream is not kept a second time.
*/ */
template<typename CharType> class token_carry_buffer
class token_text_buffer
{ {
public: public:
static constexpr std::size_t head() noexcept token_carry_buffer() = default;
{ token_carry_buffer(const token_carry_buffer&) = delete;
return 4096; token_carry_buffer& operator=(const token_carry_buffer&) = delete;
} token_carry_buffer(token_carry_buffer&& rhs) noexcept
/// a little more than head(), as the last character may be handed back
static constexpr std::size_t tail() noexcept
{
return head() + 16;
}
static constexpr std::size_t limit() noexcept
{
return head() + (2 * tail());
}
token_text_buffer() = default;
token_text_buffer(const token_text_buffer&) = delete;
token_text_buffer& operator=(const token_text_buffer&) = delete;
token_text_buffer(token_text_buffer&& rhs) noexcept
: m_data(std::move(rhs.m_data)), m_end(rhs.m_end), m_capacity_end(rhs.m_capacity_end) : m_data(std::move(rhs.m_data)), m_end(rhs.m_end), m_capacity_end(rhs.m_capacity_end)
{ {
rhs.m_end = rhs.m_capacity_end = nullptr; rhs.m_end = rhs.m_capacity_end = nullptr;
} }
token_text_buffer& operator=(token_text_buffer&& rhs) noexcept token_carry_buffer& operator=(token_carry_buffer&&) = delete;
{ ~token_carry_buffer() = default;
m_data = std::move(rhs.m_data);
m_end = rhs.m_end;
m_capacity_end = rhs.m_capacity_end;
rhs.m_end = rhs.m_capacity_end = nullptr;
return *this;
}
~token_text_buffer() = default;
void clear() noexcept void clear() noexcept
{ {
m_end = m_data.get(); m_end = m_data.get();
} }
bool empty() const noexcept void push_back(char c)
{
return m_end == m_data.get();
}
void push_back(CharType c)
{ {
if (JSON_HEDLEY_UNLIKELY(m_end == m_capacity_end)) if (JSON_HEDLEY_UNLIKELY(m_end == m_capacity_end))
{ {
make_room(1); grow(1);
} }
*m_end++ = c; *m_end++ = c;
} }
/// drop the last character void append(const char* first, const char* last)
void pop_back() noexcept
{
JSON_ASSERT(!empty());
--m_end;
}
void append(const CharType* first, const CharType* last)
{ {
const auto n = static_cast<std::size_t>(last - first); const auto n = static_cast<std::size_t>(last - first);
const auto room = static_cast<std::size_t>(m_capacity_end - m_end); const auto room = static_cast<std::size_t>(m_capacity_end - m_end);
if (JSON_HEDLEY_UNLIKELY(n > room)) if (n > room)
{ {
append_long(first, n); grow(n);
return;
} }
if (n != 0) if (n != 0)
{ {
std::memcpy(m_end, first, n * sizeof(CharType)); std::memcpy(m_end, first, n);
} }
m_end += n; m_end += n;
} }
/// replace the contents of @a out /// replace the contents of @a out
void copy_to(std::vector<CharType>& out) const void copy_to(std::vector<char>& out) const
{ {
out.assign(m_data.get(), m_end); out.assign(m_data.get(), m_end);
} }
private: private:
/// append @a n characters that do not fit, in pieces of at most tail() /// make room for @a extra more characters
/// characters, so that dropping the middle keeps the last tail() of them JSON_HEDLEY_NEVER_INLINE void grow(std::size_t extra)
JSON_HEDLEY_NEVER_INLINE void append_long(const CharType* first, std::size_t n)
{ {
while (n != 0) const auto size = static_cast<std::size_t>(m_end - m_data.get());
{ const auto capacity = static_cast<std::size_t>(m_capacity_end - m_data.get());
const std::size_t piece = n < tail() ? n : tail(); const std::size_t new_capacity = (std::max)(capacity < 32 ? static_cast<std::size_t>(32) : 2 * capacity, size + extra);
make_room(piece); std::unique_ptr<char[]> data(new char[new_capacity]); // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
std::memcpy(m_end, first, piece * sizeof(CharType));
m_end += piece;
first += piece;
n -= piece;
}
}
/// make room for @a extra (at most tail()) more characters: drop the
/// middle of a long token instead of growing beyond limit()
JSON_HEDLEY_NEVER_INLINE void make_room(std::size_t extra)
{
// nothing is allocated before the first character
std::size_t size = m_data ? static_cast<std::size_t>(m_end - m_data.get()) : 0;
if (size + extra > limit() && size > head() + tail())
{
std::memmove(m_data.get() + head(), m_end - tail(), tail() * sizeof(CharType));
m_end = m_data.get() + head() + tail();
size = head() + tail();
}
const std::size_t capacity = m_data ? static_cast<std::size_t>(m_capacity_end - m_data.get()) : 0;
if (size + extra > capacity)
{
// never more than limit(): from there on, the middle is dropped
const std::size_t doubled = capacity < 32 ? static_cast<std::size_t>(32) : 2 * capacity;
const std::size_t new_capacity = (std::max)(doubled < limit() ? doubled : limit(), size + extra);
std::unique_ptr<CharType[]> data(new CharType[new_capacity]); // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
if (size != 0) if (size != 0)
{ {
std::memcpy(data.get(), m_data.get(), size * sizeof(CharType)); std::memcpy(data.get(), m_data.get(), size);
} }
m_data = std::move(data); m_data = std::move(data);
m_end = m_data.get() + size; m_end = m_data.get() + size;
m_capacity_end = m_data.get() + new_capacity; m_capacity_end = m_data.get() + new_capacity;
} }
}
std::unique_ptr<CharType[]> m_data{}; // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays,readability-redundant-member-init) std::unique_ptr<char[]> m_data{}; // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays,readability-redundant-member-init)
CharType* m_end = nullptr; char* m_end = nullptr;
CharType* m_capacity_end = nullptr; char* m_capacity_end = nullptr;
}; };
/// the part of a token that an input adapter copied out of its window
using token_carry_buffer = token_text_buffer<char>;
////////////////////
// input adapters //
////////////////////
#ifndef JSON_NO_IO
/*! /*!
Input adapter for stdio file access. The file is locked for the lifetime of the Input adapter for stdio file access. The file is locked for the lifetime of the
adapter and read with the unlocked stdio functions (see adapter and read with the unlocked stdio functions (see
+8 -46
View File
@@ -494,10 +494,7 @@ class lexer : public lexer_base<BasicJsonType>
/// streaming adapter: capture the characters /// streaming adapter: capture the characters
void bulk_track_token(const char* data, std::size_t n, std::false_type /*lazy*/) void bulk_track_token(const char* data, std::size_t n, std::false_type /*lazy*/)
{ {
for (std::size_t i = 0; i < n; ++i) token_string.insert(token_string.end(), data, data + n);
{
token_string.push_back(static_cast<char_type>(data[i]));
}
} }
/// streaming input: no bulk fast path /// streaming input: no bulk fast path
@@ -2239,29 +2236,16 @@ scan_number_done:
const bool pending_real_unget = next_unget && current != char_traits<char_type>::eof(); const bool pending_real_unget = next_unget && current != char_traits<char_type>::eof();
const std::size_t stop = ia.get_consumed_count() - (pending_real_unget ? 1u : 0u); const std::size_t stop = ia.get_consumed_count() - (pending_real_unget ? 1u : 0u);
if (JSON_HEDLEY_LIKELY(stop >= token_string_start)) if (JSON_HEDLEY_LIKELY(stop >= token_string_start))
{
// only the beginning and the end of a long token are shown (see
// get_token_string), so copy only these
constexpr std::size_t head = token_text_buffer<char_type>::head();
constexpr std::size_t tail = token_text_buffer<char_type>::tail();
if (stop - token_string_start > head + tail)
{
ia.copy_consumed_range(token_string_start, token_string_start + head, out);
ia.copy_consumed_range(stop - tail, stop, out);
}
else
{ {
ia.copy_consumed_range(token_string_start, stop, out); ia.copy_consumed_range(token_string_start, stop, out);
} }
}
return out; return out;
} }
/// streaming adapter: the token was copied eagerly while scanning /// streaming adapter: the token was copied eagerly while scanning
const std::vector<char_type>& collect_token_chars(std::vector<char_type>& out, std::false_type /*lazy*/) const const std::vector<char_type>& collect_token_chars(std::vector<char_type>& /*out*/, std::false_type /*lazy*/) const
{ {
token_string.copy_to(out); return token_string;
return out;
} }
/// return the last read token (for errors only). Will never contain EOF /// return the last read token (for errors only). Will never contain EOF
@@ -2272,32 +2256,10 @@ scan_number_done:
std::vector<char_type> reconstructed; std::vector<char_type> reconstructed;
const std::vector<char_type>& chars = collect_token_chars(reconstructed, std::integral_constant<bool, lazy_token_string> {}); const std::vector<char_type>& chars = collect_token_chars(reconstructed, std::integral_constant<bool, lazy_token_string> {});
// a long token is shown by its first and its last head() characters; // escape control characters
// the buffers that keep it while it is read keep no more than these
// and a few more (see token_text_buffer)
constexpr std::size_t shown = token_text_buffer<char_type>::head();
std::string result; std::string result;
if (chars.size() > 2 * shown) for (const auto c : chars)
{ {
append_escaped(result, chars.begin(), chars.begin() + static_cast<std::ptrdiff_t>(shown));
result += "...";
append_escaped(result, chars.end() - static_cast<std::ptrdiff_t>(shown), chars.end());
}
else
{
append_escaped(result, chars.begin(), chars.end());
}
return result;
}
private:
/// append the characters [first, last) to @a result, escaping control characters
template<typename Iterator>
static void append_escaped(std::string& result, Iterator first, Iterator last)
{
for (; first != last; ++first)
{
const auto c = *first;
if (static_cast<unsigned char>(c) <= '\x1F') if (static_cast<unsigned char>(c) <= '\x1F')
{ {
// escape control characters // escape control characters
@@ -2311,9 +2273,9 @@ scan_number_done:
result.push_back(static_cast<std::string::value_type>(c)); result.push_back(static_cast<std::string::value_type>(c));
} }
} }
}
public: return result;
}
/// return syntax error message /// return syntax error message
JSON_HEDLEY_RETURNS_NON_NULL JSON_HEDLEY_RETURNS_NON_NULL
@@ -2563,7 +2525,7 @@ scan_number_done:
/// raw input token string for error messages; only populated for streaming /// raw input token string for error messages; only populated for streaming
/// adapters (seekable adapters reconstruct it lazily via token_string_start) /// adapters (seekable adapters reconstruct it lazily via token_string_start)
token_text_buffer<char_type> token_string {}; std::vector<char_type> token_string {};
/// start offset of the current token within the input, used to reconstruct /// start offset of the current token within the input, used to reconstruct
/// the last read token on error for seekable adapters (see collect_token_chars) /// the last read token on error for seekable adapters (see collect_token_chars)
+10
View File
@@ -230,6 +230,16 @@
// detail/recursion_depth_limit.hpp. Like JSON_NO_THREAD_LOCAL it is neither part // detail/recursion_depth_limit.hpp. Like JSON_NO_THREAD_LOCAL it is neither part
// of the ABI tag nor undefined at the end of json.hpp. // of the ABI tag nor undefined at the end of json.hpp.
// forced inlining of the serializer's helpers only in optimized builds: at -O0,
// GCC and Clang give every inlined copy its own stack slots, which made the
// recursive frame of serializer::dump_container() about 2.3 KB (MSVC does not
// expand __forceinline at /Ob0, so it keeps it)
#if defined(__OPTIMIZE__) || defined(_MSC_VER)
#define JSON_INTERNAL_SERIALIZER_INLINE JSON_HEDLEY_ALWAYS_INLINE
#else
#define JSON_INTERNAL_SERIALIZER_INLINE inline
#endif
// allow disabling exceptions // allow disabling exceptions
#if (defined(__cpp_exceptions) || defined(__EXCEPTIONS) || defined(_CPPUNWIND)) && !defined(JSON_NOEXCEPTION) #if (defined(__cpp_exceptions) || defined(__EXCEPTIONS) || defined(_CPPUNWIND)) && !defined(JSON_NOEXCEPTION)
#define JSON_THROW(exception) throw exception #define JSON_THROW(exception) throw exception
@@ -11,6 +11,7 @@
// clean up // clean up
#undef JSON_ASSERT #undef JSON_ASSERT
#undef JSON_INTERNAL_CATCH #undef JSON_INTERNAL_CATCH
#undef JSON_INTERNAL_SERIALIZER_INLINE
#undef JSON_THROW #undef JSON_THROW
#undef JSON_PRIVATE_UNLESS_TESTED #undef JSON_PRIVATE_UNLESS_TESTED
#undef NLOHMANN_BASIC_JSON_TPL_DECLARATION #undef NLOHMANN_BASIC_JSON_TPL_DECLARATION
@@ -265,7 +265,7 @@ class serializer
for a container; writing them from here spares each the entry and exit of for a container; writing them from here spares each the entry and exit of
that function. that function.
*/ */
JSON_HEDLEY_ALWAYS_INLINE void dump_element(const BasicJsonType& val, JSON_INTERNAL_SERIALIZER_INLINE void dump_element(const BasicJsonType& val,
const std::size_t current_indent, const std::size_t current_indent,
const std::size_t depth) const std::size_t depth)
{ {
@@ -971,7 +971,7 @@ class serializer
} }
/// write out the bytes pending in @ref string_buffer and reset @a bytes /// write out the bytes pending in @ref string_buffer and reset @a bytes
JSON_HEDLEY_ALWAYS_INLINE void emit_pending(std::size_t& bytes) JSON_INTERNAL_SERIALIZER_INLINE void emit_pending(std::size_t& bytes)
{ {
if (bytes != 0) if (bytes != 0)
{ {
@@ -983,7 +983,7 @@ class serializer
/// write buffer and reset index if fewer than 13 bytes are left, as this /// write buffer and reset index if fewer than 13 bytes are left, as this
/// is the maximal number of bytes to be written ("\uxxxx\uxxxx\0") for one /// is the maximal number of bytes to be written ("\uxxxx\uxxxx\0") for one
/// code point /// code point
JSON_HEDLEY_ALWAYS_INLINE void reserve_string_buffer(std::size_t& bytes) JSON_INTERNAL_SERIALIZER_INLINE void reserve_string_buffer(std::size_t& bytes)
{ {
if (string_buffer.size() - bytes < 13) if (string_buffer.size() - bytes < 13)
{ {
@@ -1344,7 +1344,7 @@ class serializer
inlining both into the pretty-printing loops once @ref dump_escaped inlining both into the pretty-printing loops once @ref dump_escaped
has grown, which made indented output up to 13% slower has grown, which made indented output up to 13% slower
*/ */
JSON_HEDLEY_ALWAYS_INLINE void put_indent(std::size_t indent) JSON_INTERNAL_SERIALIZER_INLINE void put_indent(std::size_t indent)
{ {
// closing braces at the outermost level ask for no indentation at all // closing braces at the outermost level ask for no indentation at all
if (indent == 0) if (indent == 0)
@@ -1398,7 +1398,7 @@ class serializer
its room and four stores of the indentation, see @ref put_short_literal_indent. its room and four stores of the indentation, see @ref put_short_literal_indent.
*/ */
template<std::size_t N> template<std::size_t N>
JSON_HEDLEY_ALWAYS_INLINE void put_literal_indent(const char (&s)[N], const std::size_t indent) // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays) JSON_INTERNAL_SERIALIZER_INLINE void put_literal_indent(const char (&s)[N], const std::size_t indent) // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
{ {
if (JSON_HEDLEY_LIKELY(put_short_literal_indent(s, indent))) if (JSON_HEDLEY_LIKELY(put_short_literal_indent(s, indent)))
{ {
@@ -1418,7 +1418,7 @@ class serializer
@a indent is wider or the write buffer lacks the room. @a indent is wider or the write buffer lacks the room.
*/ */
template<std::size_t N> template<std::size_t N>
JSON_HEDLEY_ALWAYS_INLINE bool put_short_literal_indent(const char (&s)[N], const std::size_t indent) // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays) JSON_INTERNAL_SERIALIZER_INLINE bool put_short_literal_indent(const char (&s)[N], const std::size_t indent) // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
{ {
constexpr std::size_t length = N - 1; constexpr std::size_t length = N - 1;
if (indent > 32 || write_buffer_pos + length + 32 > write_buffer.size()) if (indent > 32 || write_buffer_pos + length + 32 > write_buffer.size())
@@ -1445,7 +1445,7 @@ class serializer
write-through path for oversized runs. write-through path for oversized runs.
*/ */
template<std::size_t N> template<std::size_t N>
JSON_HEDLEY_ALWAYS_INLINE void put_literal(const char (&s)[N]) // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays) JSON_INTERNAL_SERIALIZER_INLINE void put_literal(const char (&s)[N]) // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
{ {
static_assert(N >= 2, "put_literal expects a non-empty string literal"); static_assert(N >= 2, "put_literal expects a non-empty string literal");
// the array bound counts the terminating NUL, which is not written // the array bound counts the terminating NUL, which is not written
+58 -162
View File
@@ -2603,6 +2603,16 @@ JSON_HEDLEY_DIAGNOSTIC_POP
// detail/recursion_depth_limit.hpp. Like JSON_NO_THREAD_LOCAL it is neither part // detail/recursion_depth_limit.hpp. Like JSON_NO_THREAD_LOCAL it is neither part
// of the ABI tag nor undefined at the end of json.hpp. // of the ABI tag nor undefined at the end of json.hpp.
// forced inlining of the serializer's helpers only in optimized builds: at -O0,
// GCC and Clang give every inlined copy its own stack slots, which made the
// recursive frame of serializer::dump_container() about 2.3 KB (MSVC does not
// expand __forceinline at /Ob0, so it keeps it)
#if defined(__OPTIMIZE__) || defined(_MSC_VER)
#define JSON_INTERNAL_SERIALIZER_INLINE JSON_HEDLEY_ALWAYS_INLINE
#else
#define JSON_INTERNAL_SERIALIZER_INLINE inline
#endif
// allow disabling exceptions // allow disabling exceptions
#if (defined(__cpp_exceptions) || defined(__EXCEPTIONS) || defined(_CPPUNWIND)) && !defined(JSON_NOEXCEPTION) #if (defined(__cpp_exceptions) || defined(__EXCEPTIONS) || defined(_CPPUNWIND)) && !defined(JSON_NOEXCEPTION)
#define JSON_THROW(exception) throw exception #define JSON_THROW(exception) throw exception
@@ -9323,164 +9333,87 @@ inline const char* empty_input_message() noexcept
return "attempting to parse an empty input; check that your input string or stream contains the expected JSON"; return "attempting to parse an empty input; check that your input string or stream contains the expected JSON";
} }
////////////////////
// input adapters //
////////////////////
#ifndef JSON_NO_IO
/*! /*!
@brief the characters of the last read token, kept for error messages
A growable array of characters for the part of a token that an input adapter A growable array of characters for the part of a token that an input adapter
copied out of its window, or that the lexer copies while it scans input it copied out of its window: unlike std::vector<char>::push_back(), appending a
cannot read again: unlike std::vector::push_back(), appending a character character takes no reference to a local, and is a few instructions when there
takes no reference to a local, and is a few instructions when there is room. is room.
The token is only needed for error messages, which show its first head()
characters and its last head() characters (see lexer::get_token_string). The
buffer therefore keeps the first head() and at least the last tail()
characters: when it is full and holds more than limit() characters, it drops
the characters in between instead of growing. That happens where it would
otherwise reallocate, so appending stays as cheap as before, and a long string
or a long run of whitespace read from a stream is not kept a second time.
*/ */
template<typename CharType> class token_carry_buffer
class token_text_buffer
{ {
public: public:
static constexpr std::size_t head() noexcept token_carry_buffer() = default;
{ token_carry_buffer(const token_carry_buffer&) = delete;
return 4096; token_carry_buffer& operator=(const token_carry_buffer&) = delete;
} token_carry_buffer(token_carry_buffer&& rhs) noexcept
/// a little more than head(), as the last character may be handed back
static constexpr std::size_t tail() noexcept
{
return head() + 16;
}
static constexpr std::size_t limit() noexcept
{
return head() + (2 * tail());
}
token_text_buffer() = default;
token_text_buffer(const token_text_buffer&) = delete;
token_text_buffer& operator=(const token_text_buffer&) = delete;
token_text_buffer(token_text_buffer&& rhs) noexcept
: m_data(std::move(rhs.m_data)), m_end(rhs.m_end), m_capacity_end(rhs.m_capacity_end) : m_data(std::move(rhs.m_data)), m_end(rhs.m_end), m_capacity_end(rhs.m_capacity_end)
{ {
rhs.m_end = rhs.m_capacity_end = nullptr; rhs.m_end = rhs.m_capacity_end = nullptr;
} }
token_text_buffer& operator=(token_text_buffer&& rhs) noexcept token_carry_buffer& operator=(token_carry_buffer&&) = delete;
{ ~token_carry_buffer() = default;
m_data = std::move(rhs.m_data);
m_end = rhs.m_end;
m_capacity_end = rhs.m_capacity_end;
rhs.m_end = rhs.m_capacity_end = nullptr;
return *this;
}
~token_text_buffer() = default;
void clear() noexcept void clear() noexcept
{ {
m_end = m_data.get(); m_end = m_data.get();
} }
bool empty() const noexcept void push_back(char c)
{
return m_end == m_data.get();
}
void push_back(CharType c)
{ {
if (JSON_HEDLEY_UNLIKELY(m_end == m_capacity_end)) if (JSON_HEDLEY_UNLIKELY(m_end == m_capacity_end))
{ {
make_room(1); grow(1);
} }
*m_end++ = c; *m_end++ = c;
} }
/// drop the last character void append(const char* first, const char* last)
void pop_back() noexcept
{
JSON_ASSERT(!empty());
--m_end;
}
void append(const CharType* first, const CharType* last)
{ {
const auto n = static_cast<std::size_t>(last - first); const auto n = static_cast<std::size_t>(last - first);
const auto room = static_cast<std::size_t>(m_capacity_end - m_end); const auto room = static_cast<std::size_t>(m_capacity_end - m_end);
if (JSON_HEDLEY_UNLIKELY(n > room)) if (n > room)
{ {
append_long(first, n); grow(n);
return;
} }
if (n != 0) if (n != 0)
{ {
std::memcpy(m_end, first, n * sizeof(CharType)); std::memcpy(m_end, first, n);
} }
m_end += n; m_end += n;
} }
/// replace the contents of @a out /// replace the contents of @a out
void copy_to(std::vector<CharType>& out) const void copy_to(std::vector<char>& out) const
{ {
out.assign(m_data.get(), m_end); out.assign(m_data.get(), m_end);
} }
private: private:
/// append @a n characters that do not fit, in pieces of at most tail() /// make room for @a extra more characters
/// characters, so that dropping the middle keeps the last tail() of them JSON_HEDLEY_NEVER_INLINE void grow(std::size_t extra)
JSON_HEDLEY_NEVER_INLINE void append_long(const CharType* first, std::size_t n)
{ {
while (n != 0) const auto size = static_cast<std::size_t>(m_end - m_data.get());
{ const auto capacity = static_cast<std::size_t>(m_capacity_end - m_data.get());
const std::size_t piece = n < tail() ? n : tail(); const std::size_t new_capacity = (std::max)(capacity < 32 ? static_cast<std::size_t>(32) : 2 * capacity, size + extra);
make_room(piece); std::unique_ptr<char[]> data(new char[new_capacity]); // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
std::memcpy(m_end, first, piece * sizeof(CharType));
m_end += piece;
first += piece;
n -= piece;
}
}
/// make room for @a extra (at most tail()) more characters: drop the
/// middle of a long token instead of growing beyond limit()
JSON_HEDLEY_NEVER_INLINE void make_room(std::size_t extra)
{
// nothing is allocated before the first character
std::size_t size = m_data ? static_cast<std::size_t>(m_end - m_data.get()) : 0;
if (size + extra > limit() && size > head() + tail())
{
std::memmove(m_data.get() + head(), m_end - tail(), tail() * sizeof(CharType));
m_end = m_data.get() + head() + tail();
size = head() + tail();
}
const std::size_t capacity = m_data ? static_cast<std::size_t>(m_capacity_end - m_data.get()) : 0;
if (size + extra > capacity)
{
// never more than limit(): from there on, the middle is dropped
const std::size_t doubled = capacity < 32 ? static_cast<std::size_t>(32) : 2 * capacity;
const std::size_t new_capacity = (std::max)(doubled < limit() ? doubled : limit(), size + extra);
std::unique_ptr<CharType[]> data(new CharType[new_capacity]); // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
if (size != 0) if (size != 0)
{ {
std::memcpy(data.get(), m_data.get(), size * sizeof(CharType)); std::memcpy(data.get(), m_data.get(), size);
} }
m_data = std::move(data); m_data = std::move(data);
m_end = m_data.get() + size; m_end = m_data.get() + size;
m_capacity_end = m_data.get() + new_capacity; m_capacity_end = m_data.get() + new_capacity;
} }
}
std::unique_ptr<CharType[]> m_data{}; // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays,readability-redundant-member-init) std::unique_ptr<char[]> m_data{}; // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays,readability-redundant-member-init)
CharType* m_end = nullptr; char* m_end = nullptr;
CharType* m_capacity_end = nullptr; char* m_capacity_end = nullptr;
}; };
/// the part of a token that an input adapter copied out of its window
using token_carry_buffer = token_text_buffer<char>;
////////////////////
// input adapters //
////////////////////
#ifndef JSON_NO_IO
/*! /*!
Input adapter for stdio file access. The file is locked for the lifetime of the Input adapter for stdio file access. The file is locked for the lifetime of the
adapter and read with the unlocked stdio functions (see adapter and read with the unlocked stdio functions (see
@@ -13975,10 +13908,7 @@ class lexer : public lexer_base<BasicJsonType>
/// streaming adapter: capture the characters /// streaming adapter: capture the characters
void bulk_track_token(const char* data, std::size_t n, std::false_type /*lazy*/) void bulk_track_token(const char* data, std::size_t n, std::false_type /*lazy*/)
{ {
for (std::size_t i = 0; i < n; ++i) token_string.insert(token_string.end(), data, data + n);
{
token_string.push_back(static_cast<char_type>(data[i]));
}
} }
/// streaming input: no bulk fast path /// streaming input: no bulk fast path
@@ -15720,29 +15650,16 @@ scan_number_done:
const bool pending_real_unget = next_unget && current != char_traits<char_type>::eof(); const bool pending_real_unget = next_unget && current != char_traits<char_type>::eof();
const std::size_t stop = ia.get_consumed_count() - (pending_real_unget ? 1u : 0u); const std::size_t stop = ia.get_consumed_count() - (pending_real_unget ? 1u : 0u);
if (JSON_HEDLEY_LIKELY(stop >= token_string_start)) if (JSON_HEDLEY_LIKELY(stop >= token_string_start))
{
// only the beginning and the end of a long token are shown (see
// get_token_string), so copy only these
constexpr std::size_t head = token_text_buffer<char_type>::head();
constexpr std::size_t tail = token_text_buffer<char_type>::tail();
if (stop - token_string_start > head + tail)
{
ia.copy_consumed_range(token_string_start, token_string_start + head, out);
ia.copy_consumed_range(stop - tail, stop, out);
}
else
{ {
ia.copy_consumed_range(token_string_start, stop, out); ia.copy_consumed_range(token_string_start, stop, out);
} }
}
return out; return out;
} }
/// streaming adapter: the token was copied eagerly while scanning /// streaming adapter: the token was copied eagerly while scanning
const std::vector<char_type>& collect_token_chars(std::vector<char_type>& out, std::false_type /*lazy*/) const const std::vector<char_type>& collect_token_chars(std::vector<char_type>& /*out*/, std::false_type /*lazy*/) const
{ {
token_string.copy_to(out); return token_string;
return out;
} }
/// return the last read token (for errors only). Will never contain EOF /// return the last read token (for errors only). Will never contain EOF
@@ -15753,32 +15670,10 @@ scan_number_done:
std::vector<char_type> reconstructed; std::vector<char_type> reconstructed;
const std::vector<char_type>& chars = collect_token_chars(reconstructed, std::integral_constant<bool, lazy_token_string> {}); const std::vector<char_type>& chars = collect_token_chars(reconstructed, std::integral_constant<bool, lazy_token_string> {});
// a long token is shown by its first and its last head() characters; // escape control characters
// the buffers that keep it while it is read keep no more than these
// and a few more (see token_text_buffer)
constexpr std::size_t shown = token_text_buffer<char_type>::head();
std::string result; std::string result;
if (chars.size() > 2 * shown) for (const auto c : chars)
{ {
append_escaped(result, chars.begin(), chars.begin() + static_cast<std::ptrdiff_t>(shown));
result += "...";
append_escaped(result, chars.end() - static_cast<std::ptrdiff_t>(shown), chars.end());
}
else
{
append_escaped(result, chars.begin(), chars.end());
}
return result;
}
private:
/// append the characters [first, last) to @a result, escaping control characters
template<typename Iterator>
static void append_escaped(std::string& result, Iterator first, Iterator last)
{
for (; first != last; ++first)
{
const auto c = *first;
if (static_cast<unsigned char>(c) <= '\x1F') if (static_cast<unsigned char>(c) <= '\x1F')
{ {
// escape control characters // escape control characters
@@ -15792,9 +15687,9 @@ scan_number_done:
result.push_back(static_cast<std::string::value_type>(c)); result.push_back(static_cast<std::string::value_type>(c));
} }
} }
}
public: return result;
}
/// return syntax error message /// return syntax error message
JSON_HEDLEY_RETURNS_NON_NULL JSON_HEDLEY_RETURNS_NON_NULL
@@ -16044,7 +15939,7 @@ scan_number_done:
/// raw input token string for error messages; only populated for streaming /// raw input token string for error messages; only populated for streaming
/// adapters (seekable adapters reconstruct it lazily via token_string_start) /// adapters (seekable adapters reconstruct it lazily via token_string_start)
token_text_buffer<char_type> token_string {}; std::vector<char_type> token_string {};
/// start offset of the current token within the input, used to reconstruct /// start offset of the current token within the input, used to reconstruct
/// the last read token on error for seekable adapters (see collect_token_chars) /// the last read token on error for seekable adapters (see collect_token_chars)
@@ -37100,7 +36995,7 @@ class serializer
for a container; writing them from here spares each the entry and exit of for a container; writing them from here spares each the entry and exit of
that function. that function.
*/ */
JSON_HEDLEY_ALWAYS_INLINE void dump_element(const BasicJsonType& val, JSON_INTERNAL_SERIALIZER_INLINE void dump_element(const BasicJsonType& val,
const std::size_t current_indent, const std::size_t current_indent,
const std::size_t depth) const std::size_t depth)
{ {
@@ -37806,7 +37701,7 @@ class serializer
} }
/// write out the bytes pending in @ref string_buffer and reset @a bytes /// write out the bytes pending in @ref string_buffer and reset @a bytes
JSON_HEDLEY_ALWAYS_INLINE void emit_pending(std::size_t& bytes) JSON_INTERNAL_SERIALIZER_INLINE void emit_pending(std::size_t& bytes)
{ {
if (bytes != 0) if (bytes != 0)
{ {
@@ -37818,7 +37713,7 @@ class serializer
/// write buffer and reset index if fewer than 13 bytes are left, as this /// write buffer and reset index if fewer than 13 bytes are left, as this
/// is the maximal number of bytes to be written ("\uxxxx\uxxxx\0") for one /// is the maximal number of bytes to be written ("\uxxxx\uxxxx\0") for one
/// code point /// code point
JSON_HEDLEY_ALWAYS_INLINE void reserve_string_buffer(std::size_t& bytes) JSON_INTERNAL_SERIALIZER_INLINE void reserve_string_buffer(std::size_t& bytes)
{ {
if (string_buffer.size() - bytes < 13) if (string_buffer.size() - bytes < 13)
{ {
@@ -38179,7 +38074,7 @@ class serializer
inlining both into the pretty-printing loops once @ref dump_escaped inlining both into the pretty-printing loops once @ref dump_escaped
has grown, which made indented output up to 13% slower has grown, which made indented output up to 13% slower
*/ */
JSON_HEDLEY_ALWAYS_INLINE void put_indent(std::size_t indent) JSON_INTERNAL_SERIALIZER_INLINE void put_indent(std::size_t indent)
{ {
// closing braces at the outermost level ask for no indentation at all // closing braces at the outermost level ask for no indentation at all
if (indent == 0) if (indent == 0)
@@ -38233,7 +38128,7 @@ class serializer
its room and four stores of the indentation, see @ref put_short_literal_indent. its room and four stores of the indentation, see @ref put_short_literal_indent.
*/ */
template<std::size_t N> template<std::size_t N>
JSON_HEDLEY_ALWAYS_INLINE void put_literal_indent(const char (&s)[N], const std::size_t indent) // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays) JSON_INTERNAL_SERIALIZER_INLINE void put_literal_indent(const char (&s)[N], const std::size_t indent) // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
{ {
if (JSON_HEDLEY_LIKELY(put_short_literal_indent(s, indent))) if (JSON_HEDLEY_LIKELY(put_short_literal_indent(s, indent)))
{ {
@@ -38253,7 +38148,7 @@ class serializer
@a indent is wider or the write buffer lacks the room. @a indent is wider or the write buffer lacks the room.
*/ */
template<std::size_t N> template<std::size_t N>
JSON_HEDLEY_ALWAYS_INLINE bool put_short_literal_indent(const char (&s)[N], const std::size_t indent) // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays) JSON_INTERNAL_SERIALIZER_INLINE bool put_short_literal_indent(const char (&s)[N], const std::size_t indent) // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
{ {
constexpr std::size_t length = N - 1; constexpr std::size_t length = N - 1;
if (indent > 32 || write_buffer_pos + length + 32 > write_buffer.size()) if (indent > 32 || write_buffer_pos + length + 32 > write_buffer.size())
@@ -38280,7 +38175,7 @@ class serializer
write-through path for oversized runs. write-through path for oversized runs.
*/ */
template<std::size_t N> template<std::size_t N>
JSON_HEDLEY_ALWAYS_INLINE void put_literal(const char (&s)[N]) // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays) JSON_INTERNAL_SERIALIZER_INLINE void put_literal(const char (&s)[N]) // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
{ {
static_assert(N >= 2, "put_literal expects a non-empty string literal"); static_assert(N >= 2, "put_literal expects a non-empty string literal");
// the array bound counts the terminating NUL, which is not written // the array bound counts the terminating NUL, which is not written
@@ -48000,6 +47895,7 @@ struct formatter<nlohmann::NLOHMANN_BASIC_JSON_TPL, char> // NOLINT(cert-dcl58-c
// clean up // clean up
#undef JSON_ASSERT #undef JSON_ASSERT
#undef JSON_INTERNAL_CATCH #undef JSON_INTERNAL_CATCH
#undef JSON_INTERNAL_SERIALIZER_INLINE
#undef JSON_THROW #undef JSON_THROW
#undef JSON_PRIVATE_UNLESS_TESTED #undef JSON_PRIVATE_UNLESS_TESTED
#undef NLOHMANN_BASIC_JSON_TPL_DECLARATION #undef NLOHMANN_BASIC_JSON_TPL_DECLARATION
-121
View File
@@ -12,7 +12,6 @@
#include <nlohmann/json.hpp> #include <nlohmann/json.hpp>
using nlohmann::json; using nlohmann::json;
#include <algorithm> // min
#include <array> // array #include <array> // array
#include <cctype> // toupper #include <cctype> // toupper
#include <cstdint> // uint32_t, uint64_t #include <cstdint> // uint32_t, uint64_t
@@ -2234,123 +2233,3 @@ TEST_CASE("serialization of short strings")
CHECK(json("caf\xC3\xA9").dump() == "\"caf\xC3\xA9\""); CHECK(json("caf\xC3\xA9").dump() == "\"caf\xC3\xA9\"");
CHECK(json("caf\xC3\xA9").dump(-1, ' ', true) == "\"caf\\u00e9\""); CHECK(json("caf\xC3\xA9").dump(-1, ' ', true) == "\"caf\\u00e9\"");
} }
TEST_CASE("token_text_buffer keeps the beginning and the end of a long token")
{
using buffer_t = nlohmann::detail::token_text_buffer<char>;
constexpr std::size_t head = buffer_t::head();
constexpr std::size_t tail = buffer_t::tail();
// the characters 0, 1, ..., 250, 0, 1, ...
const auto at = [](std::size_t i)
{
return static_cast<char>(i % 251);
};
// whether out holds the first head and the last tail characters of a
// token of n characters (and maybe more characters of its middle)
const auto keeps_ends = [&](const std::vector<char>& out, std::size_t n)
{
if (out.size() < head + tail || out.size() > buffer_t::limit())
{
return false;
}
for (std::size_t i = 0; i < head; ++i)
{
if (out[i] != at(i))
{
return false;
}
}
for (std::size_t i = 0; i < tail; ++i)
{
if (out[out.size() - tail + i] != at(n - tail + i))
{
return false;
}
}
return true;
};
SECTION("a short token is kept as it is")
{
buffer_t buffer;
CHECK(buffer.empty());
for (std::size_t i = 0; i < head + tail; ++i)
{
buffer.push_back(at(i));
}
std::vector<char> out;
buffer.copy_to(out);
REQUIRE(out.size() == head + tail);
bool same = true;
for (std::size_t i = 0; i < out.size(); ++i)
{
same = same && out[i] == at(i);
}
CHECK(same);
}
SECTION("characters added one by one")
{
for (const std::size_t n :
{
head + tail + 1, buffer_t::limit(), buffer_t::limit() + 1, static_cast<std::size_t>(100000)
})
{
CAPTURE(n)
buffer_t buffer;
for (std::size_t i = 0; i < n; ++i)
{
buffer.push_back(at(i));
}
std::vector<char> out;
buffer.copy_to(out);
CHECK(keeps_ends(out, n));
// the last character can be handed back
buffer.pop_back();
buffer.copy_to(out);
CHECK(keeps_ends(out, n - 1));
}
}
SECTION("characters added in pieces of different sizes")
{
std::vector<char> input(200000);
for (std::size_t i = 0; i < input.size(); ++i)
{
input[i] = at(i);
}
for (const std::size_t piece :
{
static_cast<std::size_t>(1), static_cast<std::size_t>(100), tail - 1, tail, tail + 1, static_cast<std::size_t>(65536), input.size()
})
{
CAPTURE(piece)
buffer_t buffer;
for (std::size_t pos = 0; pos < input.size(); pos += piece)
{
const std::size_t n = (std::min)(piece, input.size() - pos);
buffer.append(input.data() + pos, input.data() + pos + n);
}
std::vector<char> out;
buffer.copy_to(out);
CHECK(keeps_ends(out, input.size()));
}
}
SECTION("clear starts a new token")
{
buffer_t buffer;
for (std::size_t i = 0; i < 100000; ++i)
{
buffer.push_back('x');
}
buffer.clear();
CHECK(buffer.empty());
buffer.push_back('a');
std::vector<char> out;
buffer.copy_to(out);
CHECK(out == std::vector<char> {'a'});
}
}
-113
View File
@@ -3269,116 +3269,3 @@ TEST_CASE("a plain function is accepted as parser callback for containers")
CHECK((nlohmann::detail::can_compare_ne<std::string::const_iterator, std::string::const_iterator>::value)); CHECK((nlohmann::detail::can_compare_ne<std::string::const_iterator, std::string::const_iterator>::value));
CHECK((nlohmann::detail::can_compare_ne<const char*, const char*>::value)); CHECK((nlohmann::detail::can_compare_ne<const char*, const char*>::value));
} }
#if !defined(JSON_NOEXCEPTION)
TEST_CASE("parse error message of a long token")
{
// the message of a parse error shows the last read token; a long token is
// shown by its first and its last 4096 characters, and the message is the
// same for every kind of input
const auto message = [](const std::string & text, int kind) -> std::string
{
json _;
try
{
if (kind == 0)
{
_ = json::parse(text);
}
else if (kind == 1)
{
std::istringstream stream(text);
_ = json::parse(stream);
}
else if (kind == 2)
{
FILE* const file = std::tmpfile(); // NOLINT(cppcoreguidelines-owning-memory)
REQUIRE(file != nullptr);
CHECK(std::fwrite(text.data(), 1, text.size(), file) == text.size());
CHECK(std::fseek(file, 0, SEEK_SET) == 0);
try
{
_ = json::parse(file);
}
catch (...)
{
std::fclose(file); // NOLINT(cert-err33-c,cppcoreguidelines-owning-memory)
throw;
}
std::fclose(file); // NOLINT(cert-err33-c,cppcoreguidelines-owning-memory)
}
else
{
const std::list<char> chars(text.begin(), text.end());
_ = json::parse(chars.begin(), chars.end());
}
}
catch (const json::parse_error& e)
{
return e.what();
}
return "no error";
};
SECTION("a short token is shown completely")
{
const std::string text = "[\"" + std::string(8000, 'a') + "\x01\"]";
const std::string expected = "last read: '\"" + std::string(8000, 'a') + "<U+0001>'";
for (int kind = 0; kind < 4; ++kind)
{
CAPTURE(kind)
const std::string m = message(text, kind);
CHECK(m.size() > expected.size());
CHECK(m.compare(m.size() - expected.size(), expected.size(), expected) == 0);
}
}
SECTION("a long string")
{
const std::string text = "[\"" + std::string(100000, 'a') + "b\x01\"]";
// the token is '"', 100,000 times 'a', 'b', and the control character
const std::string expected = "last read: '\"" + std::string(4095, 'a') + "..." + std::string(4094, 'a') + "b<U+0001>'";
for (int kind = 0; kind < 4; ++kind)
{
CAPTURE(kind)
const std::string m = message(text, kind);
CHECK(m.size() > expected.size());
CHECK(m.compare(m.size() - expected.size(), expected.size(), expected) == 0);
}
}
SECTION("a long run of whitespace after a number")
{
// the token begins with the number and ends with the invalid character
const std::string text = "[1," + std::string(100000, '\n') + "?]";
std::string expected = "last read: '1,";
for (int i = 0; i < 4094; ++i)
{
expected += "<U+000A>";
}
expected += "...";
for (int i = 0; i < 4095; ++i)
{
expected += "<U+000A>";
}
expected += "?'";
for (int kind = 0; kind < 4; ++kind)
{
CAPTURE(kind)
const std::string m = message(text, kind);
CHECK(m.size() > expected.size());
CHECK(m.compare(m.size() - expected.size(), expected.size(), expected) == 0);
}
}
SECTION("valid input with a long run of whitespace")
{
const std::string text = "1" + std::string(100000, ' ');
for (int kind = 0; kind < 4; ++kind)
{
CAPTURE(kind)
CHECK(message(text, kind) == "no error");
}
}
}
#endif