Compare commits

..
Author SHA1 Message Date
Niels Lohmann 0efa09cf49 Fix clang-tidy performance-inefficient-string-concatenation in unit-json_view_builder
Fixes the ci_clang_tidy failure at tests/src/unit-json_view_builder.cpp:549.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-31 16:19:56 +01:00
Niels Lohmann 6491563cbb json_view: skip the digits of a fraction 16 bytes at a time (SSE2)
The digits after the decimal point were skipped by skip_digits(), 16 compare-and-branch
steps in a row: about 3.5 instructions per digit, and an exit branch that is
mispredicted when the lengths of the fractions vary. On x86 with SSE2, the
fraction is now skipped by skip_fraction_digits(): 16 bytes are compared at
once and the end of the run is taken from the bit mask, which is about a third
of the instructions for a fraction of 12 digits. The last bytes of the input
are skipped one at a time as before. Where there is no SSE2 (AArch64, other
targets, JSON_NO_SIMD) skip_fraction_digits() is skip_digits(): nothing changes
there. Integer digits keep skip_digits(), whose exit is predictable.

GCC unrolls the loop over the last bytes in every inlined copy of the new
function, which made the parse loop 4 KiB larger and parse() of documents with
hardly any fractions about 4 % slower. For GCC that loop is therefore out of
line (skip_digits_tail()); Clang is faster with it inline.

json_document::parse(), A/B in one binary, 6 code layouts, x86-64 Haswell,
time after / before (GCC 16.2 / Clang 23):

- numbers.json (10,001 floats): 0.715 / 0.680
- canada.json: 0.778 / 0.790
- mesh.json: 0.908 / 0.883
- marine_ik.json: 0.937 / 0.926
- mesh.pretty.json: 0.923 / 0.960

Files with few or no fractions and files with fractions of a fixed length
(20,000 prices with two decimals, 60,000 coordinates with six): between 0.98 and
1.014 on both compilers, except citm_catalog.json with GCC (1.05: the
instruction count is +0.3 %, and that row varies by +-7 % between builds of
unrelated variants on this machine).

New test: fractions of every length from 0 to 40 digits followed by every kind of
byte that ends a number or lies next to the digits in ASCII, in several
positions, and cut in every place near the end of the text, against json::parse()
(the length of the number is part of what is compared), for all option
combinations and with and without a terminating NUL. Checked on x86-64 with GCC 16.2,
Clang 23 and Apple clang under Rosetta, and with a deliberate off-by-one at the 16th byte.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-30 22:17:00 +01:00
10 changed files with 232 additions and 502 deletions

No files matched your search

@@ -13,15 +13,9 @@ function never throws an exception for invalid input, and the returned `#!cpp bo
is returned. is returned.
The result is the same as that of `BasicJsonType::accept()` by design; the difference is how the check is made. This The result is the same as that of `BasicJsonType::accept()` by design; the difference is how the check is made. This
function runs the document's builder, which checks the text as [`parse()`](parse.md) does, while function runs the document's index builder (and discards the index), while `BasicJsonType::accept()` runs the SAX
`BasicJsonType::accept()` runs the SAX parser with a handler that ignores every event. Use this one if you go on to parser with a handler that ignores every event. Use this one if you go on to parse the same input with
parse the same input with [`parse()`](parse.md) anyway, so that both use the same code; either is fine for a plain [`parse()`](parse.md) anyway, so that both use the same code; either is fine for a plain yes/no answer.
yes/no answer.
When both `ignore_comments` and `ignore_trailing_commas` are `#!cpp false` (the default), the builder writes no index.
With either option set, it builds the index as [`parse()`](parse.md) does and discards it: the result is the same, only
the time differs. (Neither option needs the index; the variant that skips it exists only for the default options to
keep the code size down.)
## Template parameters ## Template parameters
+19 -121
View File
@@ -69,8 +69,7 @@ struct parse_failure
}; };
/// FloatType: the number_float_t of the document, whose overflow parse() rejects /// FloatType: the number_float_t of the document, whose overflow parse() rejects
/// Store: false for accept(): the grammar, number, string and UTF-8 checks run as for parse(), but no node is written template<typename FloatType, bool Comments, bool TrailingCommas, bool NulIsEnd, bool Sentinel>
template<typename FloatType, bool Comments, bool TrailingCommas, bool NulIsEnd, bool Sentinel, bool Store>
class builder class builder
{ {
public: public:
@@ -246,87 +245,11 @@ class builder
return p; return p;
} }
/// the string from p, the first backslash, control character, ill-formed UTF-8 byte, or the end, checked and not
/// decoded (accept() stores nothing, so the arena is not written): the checks, errors and error positions of
/// slow_string(), for everything up to the closing quote
NLOHMANN_VIEW_NOINLINE decoded check_string(const unsigned char* p)
{
const unsigned char* const end = m_end;
for (;;)
{
if (p == end)
{
return failed(error_code::string_missing_quote, p);
}
const unsigned char c = *p;
if (c == '"')
{
++p;
return decoded{p, 0, 0};
}
if (c != '\\')
{
return failed(c < 0x20 ? error_code::string_control_character : error_code::string_utf8, p);
}
++p;
if (p == end)
{
return failed(error_code::string_missing_quote, p);
}
const unsigned char d = *p++;
if (d == 'u')
{
const int cp = hex4(p); // (moves p past the four digits)
if (NLOHMANN_VIEW_UNLIKELY(cp < 0))
{
return failed(error_code::string_unicode_hex, p);
}
if (NLOHMANN_VIEW_UNLIKELY((cp & 0xF800) == 0xD800)) // a surrogate
{
if (cp >= 0xDC00)
{
return failed(error_code::string_surrogate_low, p);
}
if (end - p < 2 || p[0] != '\\' || p[1] != 'u')
{
return failed(error_code::string_surrogate_high, p);
}
p += 2;
const int lo = hex4(p); // (moves p past the four digits)
if (lo < 0)
{
return failed(error_code::string_unicode_hex, p);
}
if (lo < 0xDC00 || lo > 0xDFFF)
{
return failed(error_code::string_surrogate_high, p);
}
}
}
else if (!(d == '"' || d == '\\' || d == '/' || d == 'b' || d == 'f' || d == 'n' || d == 'r' || d == 't'))
{
--p;
return failed(error_code::string_escape, p);
}
if (p != end && *p == '\\')
{
continue; // (escapes in a row: no run of text between them)
}
p = scan_string_run(p, end); // the text up to the next backslash, quote, control character, or ill-formed UTF-8
}
}
/// s points to the start of the string, p to a backslash, a control /// s points to the start of the string, p to a backslash, a control
/// character, an ill-formed UTF-8 byte, or the end: decode the string into /// character, an ill-formed UTF-8 byte, or the end: decode the string into
/// the arena /// the arena
NLOHMANN_VIEW_NOINLINE decoded slow_string(const unsigned char* s, const unsigned char* p) NLOHMANN_VIEW_NOINLINE decoded slow_string(const unsigned char* s, const unsigned char* p)
{ {
if (!enabled(Store))
{
// (accept(): the string is checked, not decoded)
static_cast<void>(s);
return check_string(p);
}
// single-character escapes; 0: invalid (and 'u', handled separately) // single-character escapes; 0: invalid (and 'u', handled separately)
static const std::array<char, 128> simple_escape = static const std::array<char, 128> simple_escape =
{ {
@@ -687,13 +610,10 @@ run_found:
NLOHMANN_VIEW_ALWAYS_INLINE bool run() NLOHMANN_VIEW_ALWAYS_INLINE bool run()
{ {
if (enabled(Store)) cold.m_doc.reserve(estimate_nodes(reinterpret_cast<const char*>(b), static_cast<std::size_t>(e - b)));
{ base = cold.m_doc.nodes;
cold.m_doc.reserve(estimate_nodes(reinterpret_cast<const char*>(b), static_cast<std::size_t>(e - b))); out = base;
base = cold.m_doc.nodes; cap = base + cold.m_doc.node_capacity;
out = base;
cap = base + cold.m_doc.node_capacity;
}
if (e - p >= 3 && p[0] == 0xEF && p[1] == 0xBB && p[2] == 0xBF) if (e - p >= 3 && p[0] == 0xEF && p[1] == 0xBB && p[2] == 0xBF)
{ {
@@ -877,10 +797,7 @@ close_object:
// an array pays nothing for this) // an array pays nothing for this)
if (NLOHMANN_VIEW_UNLIKELY(cur_count >= document_data::index_min_members)) if (NLOHMANN_VIEW_UNLIKELY(cur_count >= document_data::index_min_members))
{ {
if (enabled(Store)) cold.note_large_object(cur_idx);
{
cold.note_large_object(cur_idx);
}
} }
close_container: close_container:
@@ -1040,12 +957,6 @@ indent_done:
/// an integer's value); two stores on little-endian targets /// an integer's value); two stores on little-endian targets
NLOHMANN_VIEW_ALWAYS_INLINE node* push_node(node_kind k, std::uint8_t flags, std::uint16_t extra, std::size_t off, std::uint64_t second) NLOHMANN_VIEW_ALWAYS_INLINE node* push_node(node_kind k, std::uint8_t flags, std::uint16_t extra, std::size_t off, std::uint64_t second)
{ {
if (!enabled(Store))
{
// accept(): nothing is stored, so what computes the arguments is not run either. base and out stay
// null: their difference, which open() takes, is 0.
return out;
}
if (NLOHMANN_VIEW_UNLIKELY(out == cap)) if (NLOHMANN_VIEW_UNLIKELY(out == cap))
{ {
const auto n = static_cast<std::size_t>(out - base); const auto n = static_cast<std::size_t>(out - base);
@@ -1104,12 +1015,9 @@ indent_done:
NLOHMANN_VIEW_ALWAYS_INLINE void close() NLOHMANN_VIEW_ALWAYS_INLINE void close()
{ {
if (enabled(Store)) node& n = base[cur_idx];
{ n.len = cur_count;
node& n = base[cur_idx]; n.next = static_cast<std::uint32_t>(out - base) - cur_idx;
n.len = cur_count;
n.next = static_cast<std::uint32_t>(out - base) - cur_idx;
}
if (--depth != 0) if (--depth != 0)
{ {
if (NLOHMANN_VIEW_LIKELY(depth <= 64)) if (NLOHMANN_VIEW_LIKELY(depth <= 64))
@@ -1174,7 +1082,7 @@ indent_done:
{ {
++p; ++p;
const unsigned char* const f0 = p; const unsigned char* const f0 = p;
p = skip_digits(p, e); p = skip_fraction_digits(p, e);
if (NLOHMANN_VIEW_UNLIKELY(p == f0)) if (NLOHMANN_VIEW_UNLIKELY(p == f0))
{ {
return fail(error_code::number_after_dot); return fail(error_code::number_after_dot);
@@ -1284,44 +1192,34 @@ indent_done:
}; };
/// run the builder with compile-time options /// run the builder with compile-time options
template<typename FloatType, bool Comments, bool TrailingCommas, bool NulIsEnd, bool Store> template<typename FloatType, bool Comments, bool TrailingCommas, bool NulIsEnd>
inline bool build_with(document_data& d, const char* src, std::size_t size, bool sentinel, parse_failure& failure) inline bool build_with(document_data& d, const char* src, std::size_t size, bool sentinel, parse_failure& failure)
{ {
if (sentinel) if (sentinel)
{ {
builder<FloatType, Comments, TrailingCommas, NulIsEnd, true, Store> bld(d, src, size); builder<FloatType, Comments, TrailingCommas, NulIsEnd, true> bld(d, src, size);
const bool ok = bld.run(); const bool ok = bld.run();
failure = bld.failure(); failure = bld.failure();
return ok; return ok;
} }
builder<FloatType, Comments, TrailingCommas, NulIsEnd, false, Store> bld(d, src, size); builder<FloatType, Comments, TrailingCommas, NulIsEnd, false> bld(d, src, size);
const bool ok = bld.run(); const bool ok = bld.run();
failure = bld.failure(); failure = bld.failure();
return ok; return ok;
} }
/// sentinel: src[size] is readable and 0 (e.g. std::string); FloatType: the /// sentinel: src[size] is readable and 0 (e.g. std::string); FloatType: the
/// number_float_t of the document; validate_only: accept(), only the result is /// number_float_t of the document
/// used
template<typename FloatType, bool NulIsEnd> template<typename FloatType, bool NulIsEnd>
inline bool build(document_data& d, const char* src, std::size_t size, bool comments, bool trailing_commas, bool sentinel, bool validate_only, inline bool build(document_data& d, const char* src, std::size_t size, bool comments, bool trailing_commas, bool sentinel, parse_failure& failure)
parse_failure& failure)
{ {
if (validate_only && !comments && !trailing_commas)
{
// accept() of plain JSON: nothing is stored. The other option sets
// could skip the stores just as well; they keep the builder that
// stores only so that there are two more instantiations of the parse
// loop, not eight.
return build_with<FloatType, false, false, NulIsEnd, false>(d, src, size, sentinel, failure);
}
if (comments) if (comments)
{ {
return trailing_commas ? build_with<FloatType, true, true, NulIsEnd, true>(d, src, size, sentinel, failure) return trailing_commas ? build_with<FloatType, true, true, NulIsEnd>(d, src, size, sentinel, failure)
: build_with<FloatType, true, false, NulIsEnd, true>(d, src, size, sentinel, failure); : build_with<FloatType, true, false, NulIsEnd>(d, src, size, sentinel, failure);
} }
return trailing_commas ? build_with<FloatType, false, true, NulIsEnd, true>(d, src, size, sentinel, failure) return trailing_commas ? build_with<FloatType, false, true, NulIsEnd>(d, src, size, sentinel, failure)
: build_with<FloatType, false, false, NulIsEnd, true>(d, src, size, sentinel, failure); : build_with<FloatType, false, false, NulIsEnd>(d, src, size, sentinel, failure);
} }
} // namespace view } // namespace view
@@ -40,15 +40,6 @@
#define NLOHMANN_VIEW_NOINLINE #define NLOHMANN_VIEW_NOINLINE
#endif #endif
// GCC: a function compiled as written, without the clones in which interprocedural
// constant propagation drops an argument that every caller passes as the same constant
// (calling such a clone changes the code GCC generates for the parse loop that calls it)
#if defined(__GNUC__) && !defined(__clang__)
#define NLOHMANN_VIEW_NOCLONE __attribute__((noclone))
#else
#define NLOHMANN_VIEW_NOCLONE
#endif
#if defined(__GNUC__) || defined(__clang__) #if defined(__GNUC__) || defined(__clang__)
#define NLOHMANN_VIEW_NODISCARD __attribute__((warn_unused_result)) #define NLOHMANN_VIEW_NODISCARD __attribute__((warn_unused_result))
#elif defined(_MSC_VER) #elif defined(_MSC_VER)
@@ -15,7 +15,6 @@
#undef NLOHMANN_VIEW_UNLIKELY #undef NLOHMANN_VIEW_UNLIKELY
#undef NLOHMANN_VIEW_ALWAYS_INLINE #undef NLOHMANN_VIEW_ALWAYS_INLINE
#undef NLOHMANN_VIEW_NOINLINE #undef NLOHMANN_VIEW_NOINLINE
#undef NLOHMANN_VIEW_NOCLONE
#undef NLOHMANN_VIEW_NODISCARD #undef NLOHMANN_VIEW_NODISCARD
#undef NLOHMANN_VIEW_THROW #undef NLOHMANN_VIEW_THROW
#undef NLOHMANN_VIEW_LITTLE_ENDIAN #undef NLOHMANN_VIEW_LITTLE_ENDIAN
+51
View File
@@ -198,6 +198,57 @@ inline const unsigned char* find_block_comment_end(const unsigned char* p, const
return p; return p;
} }
/// The digits of a fraction, whose number varies from one number to the next: skip_digits() with 16 bytes compared at
/// once and the end of the run taken from a bit mask (SSE2), instead of a compare and a branch for each digit. That
/// is about a third of the instructions for a fraction of 12 digits, and no branch that is mispredicted when the
/// lengths vary. Where there is no SSE2, skip_digits() is used.
#if NLOHMANN_VIEW_SSE2
// The last bytes of the input, one at a time. GCC unrolls this loop in every inlined copy of the function below, which
// makes the parse loop 4 KiB larger and slows down documents that have hardly any fractions; out of line, it does
// not. Clang does better with the loop inline (a call in the parse loop costs it more than the unrolling).
#if defined(__GNUC__) && !defined(__clang__)
inline NLOHMANN_VIEW_NOINLINE const unsigned char* skip_digits_tail(const unsigned char* p, const unsigned char* e) noexcept
{
while (p != e && is_digit(*p))
{
++p;
}
return p;
}
#else
NLOHMANN_VIEW_ALWAYS_INLINE const unsigned char* skip_digits_tail(const unsigned char* p, const unsigned char* e) noexcept
{
while (p != e && is_digit(*p))
{
++p;
}
return p;
}
#endif
NLOHMANN_VIEW_ALWAYS_INLINE const unsigned char* skip_fraction_digits(const unsigned char* p, const unsigned char* e) noexcept
{
while (e - p >= 16)
{
const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(p))); // NOLINT(bugprone-casting-through-void)
// a digit is above '/' and below ':' (signed compares: a byte with the top bit set is negative, and neither)
const __m128i digits = _mm_and_si128(_mm_cmpgt_epi8(in, _mm_set1_epi8('/')), _mm_cmplt_epi8(in, _mm_set1_epi8(':')));
const auto not_digit = static_cast<std::uint64_t>(~static_cast<unsigned>(_mm_movemask_epi8(digits)) & 0xFFFFu);
if (not_digit != 0)
{
return p + count_trailing_zeros(not_digit);
}
p += 16;
}
return skip_digits_tail(p, e);
}
#else
NLOHMANN_VIEW_ALWAYS_INLINE const unsigned char* skip_fraction_digits(const unsigned char* p, const unsigned char* e) noexcept
{
return skip_digits(p, e);
}
#endif
/// powers of ten up to 10^19 as integers /// powers of ten up to 10^19 as integers
inline std::uint64_t int_pow10(unsigned k) noexcept inline std::uint64_t int_pow10(unsigned k) noexcept
{ {
+2 -5
View File
@@ -289,12 +289,9 @@ compares, and the UTF-8 check covers the bytes up to it. Returns where the
string scan stops, like scan_string_run: before ill-formed UTF-8 and for the string scan stops, like scan_string_run: before ill-formed UTF-8 and for the
last bytes of the input, the bytes are checked one sequence at a time. Out of last bytes of the input, the bytes are checked one sequence at a time. Out of
line, so that no constants of the check occupy registers in the parse loop. line, so that no constants of the check occupy registers in the parse loop.
On x86-64, it is compiled for SSSE3 (see cpu_has_ssse3()). Not cloned: with On x86-64, it is compiled for SSSE3 (see cpu_has_ssse3()).
the calls of accept()'s builder, GCC 16 would clone it without plain (always
string_plain()), and calling the clone made json_document::parse() run 0.7-1.2 %
more instructions.
*/ */
NLOHMANN_VIEW_SSSE3_TARGET NLOHMANN_VIEW_NOINLINE NLOHMANN_VIEW_NOCLONE inline const unsigned char* scan_string_vector(const unsigned char* p, const unsigned char* e, const std::uint8_t* plain) noexcept NLOHMANN_VIEW_SSSE3_TARGET NLOHMANN_VIEW_NOINLINE inline const unsigned char* scan_string_vector(const unsigned char* p, const unsigned char* e, const std::uint8_t* plain) noexcept
{ {
using lookup = utf8_lookup4<>; using lookup = utf8_lookup4<>;
const unsigned char* block = p; const unsigned char* block = p;
+19 -37
View File
@@ -1060,7 +1060,7 @@ class basic_json_document
const bool ignore_trailing_commas = false) const bool ignore_trailing_commas = false)
{ {
basic_json_document d; basic_json_document d;
d.read_range(first, last, error_mode(allow_exceptions), ignore_comments, ignore_trailing_commas); d.read_range(first, last, allow_exceptions, ignore_comments, ignore_trailing_commas);
return d; return d;
} }
@@ -1075,7 +1075,7 @@ class basic_json_document
basic_json_document d; basic_json_document d;
std::string buf; std::string buf;
append_input(buf, std::forward<InputType>(input)); append_input(buf, std::forward<InputType>(input));
d.build_owned(std::move(buf), error_mode(allow_exceptions), ignore_comments, ignore_trailing_commas); d.build_owned(std::move(buf), allow_exceptions, ignore_comments, ignore_trailing_commas);
return d; return d;
} }
@@ -1088,8 +1088,7 @@ class basic_json_document
static bool accept(InputType&& input, const bool ignore_comments = false, const bool ignore_trailing_commas = false) static bool accept(InputType&& input, const bool ignore_comments = false, const bool ignore_trailing_commas = false)
{ {
basic_json_document d; basic_json_document d;
d.read_kind(std::forward<InputType>(input), on_error::validate, ignore_comments, ignore_trailing_commas, d.read(std::forward<InputType>(input), false, ignore_comments, ignore_trailing_commas);
std::integral_constant<detail::view::input_kind, detail::view::classify_input<InputType>::value> {});
return !d.is_discarded(); return !d.is_discarded();
} }
@@ -1105,7 +1104,7 @@ class basic_json_document
const bool ignore_comments = false, const bool ignore_comments = false,
const bool ignore_trailing_commas = false) const bool ignore_trailing_commas = false)
{ {
read_kind(std::forward<InputType>(input), error_mode(allow_exceptions), ignore_comments, ignore_trailing_commas, read_kind(std::forward<InputType>(input), allow_exceptions, ignore_comments, ignore_trailing_commas,
std::integral_constant<detail::view::input_kind, detail::view::classify_input<InputType>::value> {}); std::integral_constant<detail::view::input_kind, detail::view::classify_input<InputType>::value> {});
} }
@@ -1464,22 +1463,6 @@ class basic_json_document
return idx; return idx;
} }
/// what a failed parse does
enum class on_error : std::uint8_t
{
discard, ///< the document is discarded
raise, ///< parse_error is thrown
validate ///< accept(): the document is discarded, and no node is stored while parsing
};
static on_error error_mode(bool allow_exceptions) noexcept
{
return allow_exceptions ? on_error::raise : on_error::discard;
}
/// NulIsEnd of the builder: a null byte ends the input unless strict_nul_handling
static constexpr bool nul_is_end = !detail::abi_config::strict_nul_handling;
/// create the storage (sized for the input) on first use /// create the storage (sized for the input) on first use
void ensure_data(const char* src, std::size_t size) void ensure_data(const char* src, std::size_t size)
{ {
@@ -1491,7 +1474,7 @@ class basic_json_document
/// parse a buffer the document takes ownership of (it replaces the /// parse a buffer the document takes ownership of (it replaces the
/// previous owned copy) /// previous owned copy)
void build_owned(std::string&& buf, on_error allow_exceptions, bool comments, bool trailing_commas) void build_owned(std::string&& buf, bool allow_exceptions, bool comments, bool trailing_commas)
{ {
ensure_data(buf.data(), buf.size()); ensure_data(buf.data(), buf.size());
m_data->owned = std::move(buf); m_data->owned = std::move(buf);
@@ -1501,7 +1484,7 @@ class basic_json_document
/// parse a copy of the input; the copy is appended to the cleared owned /// parse a copy of the input; the copy is appended to the cleared owned
/// buffer, so that its capacity is reused /// buffer, so that its capacity is reused
template<typename T> template<typename T>
void build_copy(T&& input, on_error allow_exceptions, bool comments, bool trailing_commas) void build_copy(T&& input, bool allow_exceptions, bool comments, bool trailing_commas)
{ {
if (!m_data) if (!m_data)
{ {
@@ -1522,7 +1505,7 @@ class basic_json_document
} }
/// sentinel: src[size] is readable and 0 (std::string, C strings) /// sentinel: src[size] is readable and 0 (std::string, C strings)
void build(const char* src, std::size_t size, on_error allow_exceptions, bool comments, bool trailing_commas, bool owned, bool sentinel) void build(const char* src, std::size_t size, bool allow_exceptions, bool comments, bool trailing_commas, bool owned, bool sentinel)
{ {
ensure_data(src, size); ensure_data(src, size);
document_data& d = *m_data; document_data& d = *m_data;
@@ -1549,8 +1532,7 @@ class basic_json_document
} }
else else
{ {
ok = detail::view::build<typename BasicJsonType::number_float_t, nul_is_end>(d, src, size, comments, trailing_commas, sentinel, ok = detail::view::build < typename BasicJsonType::number_float_t, !detail::abi_config::strict_nul_handling > (d, src, size, comments, trailing_commas, sentinel, failure);
allow_exceptions == on_error::validate, failure);
} }
if (NLOHMANN_VIEW_LIKELY(ok)) if (NLOHMANN_VIEW_LIKELY(ok))
{ {
@@ -1563,7 +1545,7 @@ class basic_json_document
return; return;
} }
std::vector<std::uint32_t>().swap(d.large_objects); std::vector<std::uint32_t>().swap(d.large_objects);
if (allow_exceptions == on_error::raise) if (allow_exceptions)
{ {
detail::view::throw_parse_failure<BasicJsonType>(failure, src, size, comments, trailing_commas); detail::view::throw_parse_failure<BasicJsonType>(failure, src, size, comments, trailing_commas);
} }
@@ -1571,13 +1553,13 @@ class basic_json_document
// --- input dispatch (see detail::view::input_kind) --- // --- input dispatch (see detail::view::input_kind) ---
void read_kind(std::string&& s, on_error ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::move_string> /*unused*/) void read_kind(std::string&& s, bool ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::move_string> /*unused*/)
{ {
build_owned(std::move(s), ae, c, tc); build_owned(std::move(s), ae, c, tc);
} }
template<typename CharT> template<typename CharT>
void read_kind(CharT* s, on_error ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::c_string> /*unused*/) void read_kind(CharT* s, bool ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::c_string> /*unused*/)
{ {
if (s == nullptr) if (s == nullptr)
{ {
@@ -1591,7 +1573,7 @@ class basic_json_document
} }
template<typename Array> template<typename Array>
void read_kind(Array& a, on_error ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::char_array> /*unused*/) void read_kind(Array& a, bool ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::char_array> /*unused*/)
{ {
using CharT = typename std::remove_cv<typename std::remove_extent<Array>::type>::type; using CharT = typename std::remove_cv<typename std::remove_extent<Array>::type>::type;
const std::size_t n = std::extent<Array>::value; const std::size_t n = std::extent<Array>::value;
@@ -1602,7 +1584,7 @@ class basic_json_document
} }
template<typename T> template<typename T>
void read_kind(const T& s, on_error ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::borrow_range> /*unused*/) void read_kind(const T& s, bool ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::borrow_range> /*unused*/)
{ {
// std::basic_string guarantees data()[size()] == 0: use it as sentinel // std::basic_string guarantees data()[size()] == 0: use it as sentinel
const auto size = static_cast<std::size_t>(s.size()); const auto size = static_cast<std::size_t>(s.size());
@@ -1611,13 +1593,13 @@ class basic_json_document
} }
template<typename T> template<typename T>
void read_kind(const T& s, on_error ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::copy_range> /*unused*/) void read_kind(const T& s, bool ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::copy_range> /*unused*/)
{ {
build_copy(s, ae, c, tc); build_copy(s, ae, c, tc);
} }
template<typename CharT> template<typename CharT>
void read_kind(CharT* s, on_error ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::wide_c_string> /*unused*/) void read_kind(CharT* s, bool ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::wide_c_string> /*unused*/)
{ {
if (s == nullptr) if (s == nullptr)
{ {
@@ -1629,13 +1611,13 @@ class basic_json_document
} }
template<typename T> template<typename T>
void read_kind(T&& input, on_error ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::adapter> /*unused*/) void read_kind(T&& input, bool ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::adapter> /*unused*/)
{ {
build_copy(std::forward<T>(input), ae, c, tc); build_copy(std::forward<T>(input), ae, c, tc);
} }
template<typename IteratorType> template<typename IteratorType>
void read_range(IteratorType first, IteratorType last, on_error ae, bool c, bool tc) void read_range(IteratorType first, IteratorType last, bool ae, bool c, bool tc)
{ {
using value_type = typename std::remove_cv<typename std::iterator_traits<IteratorType>::value_type>::type; using value_type = typename std::remove_cv<typename std::iterator_traits<IteratorType>::value_type>::type;
// borrow the range where the library's input adapter scans it as one // borrow the range where the library's input adapter scans it as one
@@ -1646,14 +1628,14 @@ class basic_json_document
} }
template<typename IteratorType> template<typename IteratorType>
void read_range_impl(IteratorType first, IteratorType last, on_error ae, bool c, bool tc, std::true_type /*contiguous bytes*/) void read_range_impl(IteratorType first, IteratorType last, bool ae, bool c, bool tc, std::true_type /*contiguous bytes*/)
{ {
const auto size = static_cast<std::size_t>(std::distance(first, last)); const auto size = static_cast<std::size_t>(std::distance(first, last));
build(size == 0 ? "" : reinterpret_cast<const char*>(&*first), size, ae, c, tc, false, size == 0); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast) build(size == 0 ? "" : reinterpret_cast<const char*>(&*first), size, ae, c, tc, false, size == 0); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
} }
template<typename IteratorType> template<typename IteratorType>
void read_range_impl(IteratorType first, IteratorType last, on_error ae, bool c, bool tc, std::false_type /*other*/) void read_range_impl(IteratorType first, IteratorType last, bool ae, bool c, bool tc, std::false_type /*other*/)
{ {
build_copy(detail::view::make_adapter_input(detail::input_adapter(first, last)), ae, c, tc); build_copy(detail::view::make_adapter_input(detail::input_adapter(first, last)), ae, c, tc);
} }
+91 -173
View File
@@ -157,15 +157,6 @@
#define NLOHMANN_VIEW_NOINLINE #define NLOHMANN_VIEW_NOINLINE
#endif #endif
// GCC: a function compiled as written, without the clones in which interprocedural
// constant propagation drops an argument that every caller passes as the same constant
// (calling such a clone changes the code GCC generates for the parse loop that calls it)
#if defined(__GNUC__) && !defined(__clang__)
#define NLOHMANN_VIEW_NOCLONE __attribute__((noclone))
#else
#define NLOHMANN_VIEW_NOCLONE
#endif
#if defined(__GNUC__) || defined(__clang__) #if defined(__GNUC__) || defined(__clang__)
#define NLOHMANN_VIEW_NODISCARD __attribute__((warn_unused_result)) #define NLOHMANN_VIEW_NODISCARD __attribute__((warn_unused_result))
#elif defined(_MSC_VER) #elif defined(_MSC_VER)
@@ -957,12 +948,9 @@ compares, and the UTF-8 check covers the bytes up to it. Returns where the
string scan stops, like scan_string_run: before ill-formed UTF-8 and for the string scan stops, like scan_string_run: before ill-formed UTF-8 and for the
last bytes of the input, the bytes are checked one sequence at a time. Out of last bytes of the input, the bytes are checked one sequence at a time. Out of
line, so that no constants of the check occupy registers in the parse loop. line, so that no constants of the check occupy registers in the parse loop.
On x86-64, it is compiled for SSSE3 (see cpu_has_ssse3()). Not cloned: with On x86-64, it is compiled for SSSE3 (see cpu_has_ssse3()).
the calls of accept()'s builder, GCC 16 would clone it without plain (always
string_plain()), and calling the clone made json_document::parse() run 0.7-1.2 %
more instructions.
*/ */
NLOHMANN_VIEW_SSSE3_TARGET NLOHMANN_VIEW_NOINLINE NLOHMANN_VIEW_NOCLONE inline const unsigned char* scan_string_vector(const unsigned char* p, const unsigned char* e, const std::uint8_t* plain) noexcept NLOHMANN_VIEW_SSSE3_TARGET NLOHMANN_VIEW_NOINLINE inline const unsigned char* scan_string_vector(const unsigned char* p, const unsigned char* e, const std::uint8_t* plain) noexcept
{ {
using lookup = utf8_lookup4<>; using lookup = utf8_lookup4<>;
const unsigned char* block = p; const unsigned char* block = p;
@@ -1230,6 +1218,57 @@ inline const unsigned char* find_block_comment_end(const unsigned char* p, const
return p; return p;
} }
/// The digits of a fraction, whose number varies from one number to the next: skip_digits() with 16 bytes compared at
/// once and the end of the run taken from a bit mask (SSE2), instead of a compare and a branch for each digit. That
/// is about a third of the instructions for a fraction of 12 digits, and no branch that is mispredicted when the
/// lengths vary. Where there is no SSE2, skip_digits() is used.
#if NLOHMANN_VIEW_SSE2
// The last bytes of the input, one at a time. GCC unrolls this loop in every inlined copy of the function below, which
// makes the parse loop 4 KiB larger and slows down documents that have hardly any fractions; out of line, it does
// not. Clang does better with the loop inline (a call in the parse loop costs it more than the unrolling).
#if defined(__GNUC__) && !defined(__clang__)
inline NLOHMANN_VIEW_NOINLINE const unsigned char* skip_digits_tail(const unsigned char* p, const unsigned char* e) noexcept
{
while (p != e && is_digit(*p))
{
++p;
}
return p;
}
#else
NLOHMANN_VIEW_ALWAYS_INLINE const unsigned char* skip_digits_tail(const unsigned char* p, const unsigned char* e) noexcept
{
while (p != e && is_digit(*p))
{
++p;
}
return p;
}
#endif
NLOHMANN_VIEW_ALWAYS_INLINE const unsigned char* skip_fraction_digits(const unsigned char* p, const unsigned char* e) noexcept
{
while (e - p >= 16)
{
const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(p))); // NOLINT(bugprone-casting-through-void)
// a digit is above '/' and below ':' (signed compares: a byte with the top bit set is negative, and neither)
const __m128i digits = _mm_and_si128(_mm_cmpgt_epi8(in, _mm_set1_epi8('/')), _mm_cmplt_epi8(in, _mm_set1_epi8(':')));
const auto not_digit = static_cast<std::uint64_t>(~static_cast<unsigned>(_mm_movemask_epi8(digits)) & 0xFFFFu);
if (not_digit != 0)
{
return p + count_trailing_zeros(not_digit);
}
p += 16;
}
return skip_digits_tail(p, e);
}
#else
NLOHMANN_VIEW_ALWAYS_INLINE const unsigned char* skip_fraction_digits(const unsigned char* p, const unsigned char* e) noexcept
{
return skip_digits(p, e);
}
#endif
/// powers of ten up to 10^19 as integers /// powers of ten up to 10^19 as integers
inline std::uint64_t int_pow10(unsigned k) noexcept inline std::uint64_t int_pow10(unsigned k) noexcept
{ {
@@ -1323,8 +1362,7 @@ struct parse_failure
}; };
/// FloatType: the number_float_t of the document, whose overflow parse() rejects /// FloatType: the number_float_t of the document, whose overflow parse() rejects
/// Store: false for accept(): the grammar, number, string and UTF-8 checks run as for parse(), but no node is written template<typename FloatType, bool Comments, bool TrailingCommas, bool NulIsEnd, bool Sentinel>
template<typename FloatType, bool Comments, bool TrailingCommas, bool NulIsEnd, bool Sentinel, bool Store>
class builder class builder
{ {
public: public:
@@ -1500,87 +1538,11 @@ class builder
return p; return p;
} }
/// the string from p, the first backslash, control character, ill-formed UTF-8 byte, or the end, checked and not
/// decoded (accept() stores nothing, so the arena is not written): the checks, errors and error positions of
/// slow_string(), for everything up to the closing quote
NLOHMANN_VIEW_NOINLINE decoded check_string(const unsigned char* p)
{
const unsigned char* const end = m_end;
for (;;)
{
if (p == end)
{
return failed(error_code::string_missing_quote, p);
}
const unsigned char c = *p;
if (c == '"')
{
++p;
return decoded{p, 0, 0};
}
if (c != '\\')
{
return failed(c < 0x20 ? error_code::string_control_character : error_code::string_utf8, p);
}
++p;
if (p == end)
{
return failed(error_code::string_missing_quote, p);
}
const unsigned char d = *p++;
if (d == 'u')
{
const int cp = hex4(p); // (moves p past the four digits)
if (NLOHMANN_VIEW_UNLIKELY(cp < 0))
{
return failed(error_code::string_unicode_hex, p);
}
if (NLOHMANN_VIEW_UNLIKELY((cp & 0xF800) == 0xD800)) // a surrogate
{
if (cp >= 0xDC00)
{
return failed(error_code::string_surrogate_low, p);
}
if (end - p < 2 || p[0] != '\\' || p[1] != 'u')
{
return failed(error_code::string_surrogate_high, p);
}
p += 2;
const int lo = hex4(p); // (moves p past the four digits)
if (lo < 0)
{
return failed(error_code::string_unicode_hex, p);
}
if (lo < 0xDC00 || lo > 0xDFFF)
{
return failed(error_code::string_surrogate_high, p);
}
}
}
else if (!(d == '"' || d == '\\' || d == '/' || d == 'b' || d == 'f' || d == 'n' || d == 'r' || d == 't'))
{
--p;
return failed(error_code::string_escape, p);
}
if (p != end && *p == '\\')
{
continue; // (escapes in a row: no run of text between them)
}
p = scan_string_run(p, end); // the text up to the next backslash, quote, control character, or ill-formed UTF-8
}
}
/// s points to the start of the string, p to a backslash, a control /// s points to the start of the string, p to a backslash, a control
/// character, an ill-formed UTF-8 byte, or the end: decode the string into /// character, an ill-formed UTF-8 byte, or the end: decode the string into
/// the arena /// the arena
NLOHMANN_VIEW_NOINLINE decoded slow_string(const unsigned char* s, const unsigned char* p) NLOHMANN_VIEW_NOINLINE decoded slow_string(const unsigned char* s, const unsigned char* p)
{ {
if (!enabled(Store))
{
// (accept(): the string is checked, not decoded)
static_cast<void>(s);
return check_string(p);
}
// single-character escapes; 0: invalid (and 'u', handled separately) // single-character escapes; 0: invalid (and 'u', handled separately)
static const std::array<char, 128> simple_escape = static const std::array<char, 128> simple_escape =
{ {
@@ -1941,13 +1903,10 @@ run_found:
NLOHMANN_VIEW_ALWAYS_INLINE bool run() NLOHMANN_VIEW_ALWAYS_INLINE bool run()
{ {
if (enabled(Store)) cold.m_doc.reserve(estimate_nodes(reinterpret_cast<const char*>(b), static_cast<std::size_t>(e - b)));
{ base = cold.m_doc.nodes;
cold.m_doc.reserve(estimate_nodes(reinterpret_cast<const char*>(b), static_cast<std::size_t>(e - b))); out = base;
base = cold.m_doc.nodes; cap = base + cold.m_doc.node_capacity;
out = base;
cap = base + cold.m_doc.node_capacity;
}
if (e - p >= 3 && p[0] == 0xEF && p[1] == 0xBB && p[2] == 0xBF) if (e - p >= 3 && p[0] == 0xEF && p[1] == 0xBB && p[2] == 0xBF)
{ {
@@ -2131,10 +2090,7 @@ close_object:
// an array pays nothing for this) // an array pays nothing for this)
if (NLOHMANN_VIEW_UNLIKELY(cur_count >= document_data::index_min_members)) if (NLOHMANN_VIEW_UNLIKELY(cur_count >= document_data::index_min_members))
{ {
if (enabled(Store)) cold.note_large_object(cur_idx);
{
cold.note_large_object(cur_idx);
}
} }
close_container: close_container:
@@ -2294,12 +2250,6 @@ indent_done:
/// an integer's value); two stores on little-endian targets /// an integer's value); two stores on little-endian targets
NLOHMANN_VIEW_ALWAYS_INLINE node* push_node(node_kind k, std::uint8_t flags, std::uint16_t extra, std::size_t off, std::uint64_t second) NLOHMANN_VIEW_ALWAYS_INLINE node* push_node(node_kind k, std::uint8_t flags, std::uint16_t extra, std::size_t off, std::uint64_t second)
{ {
if (!enabled(Store))
{
// accept(): nothing is stored, so what computes the arguments is not run either. base and out stay
// null: their difference, which open() takes, is 0.
return out;
}
if (NLOHMANN_VIEW_UNLIKELY(out == cap)) if (NLOHMANN_VIEW_UNLIKELY(out == cap))
{ {
const auto n = static_cast<std::size_t>(out - base); const auto n = static_cast<std::size_t>(out - base);
@@ -2358,12 +2308,9 @@ indent_done:
NLOHMANN_VIEW_ALWAYS_INLINE void close() NLOHMANN_VIEW_ALWAYS_INLINE void close()
{ {
if (enabled(Store)) node& n = base[cur_idx];
{ n.len = cur_count;
node& n = base[cur_idx]; n.next = static_cast<std::uint32_t>(out - base) - cur_idx;
n.len = cur_count;
n.next = static_cast<std::uint32_t>(out - base) - cur_idx;
}
if (--depth != 0) if (--depth != 0)
{ {
if (NLOHMANN_VIEW_LIKELY(depth <= 64)) if (NLOHMANN_VIEW_LIKELY(depth <= 64))
@@ -2428,7 +2375,7 @@ indent_done:
{ {
++p; ++p;
const unsigned char* const f0 = p; const unsigned char* const f0 = p;
p = skip_digits(p, e); p = skip_fraction_digits(p, e);
if (NLOHMANN_VIEW_UNLIKELY(p == f0)) if (NLOHMANN_VIEW_UNLIKELY(p == f0))
{ {
return fail(error_code::number_after_dot); return fail(error_code::number_after_dot);
@@ -2538,44 +2485,34 @@ indent_done:
}; };
/// run the builder with compile-time options /// run the builder with compile-time options
template<typename FloatType, bool Comments, bool TrailingCommas, bool NulIsEnd, bool Store> template<typename FloatType, bool Comments, bool TrailingCommas, bool NulIsEnd>
inline bool build_with(document_data& d, const char* src, std::size_t size, bool sentinel, parse_failure& failure) inline bool build_with(document_data& d, const char* src, std::size_t size, bool sentinel, parse_failure& failure)
{ {
if (sentinel) if (sentinel)
{ {
builder<FloatType, Comments, TrailingCommas, NulIsEnd, true, Store> bld(d, src, size); builder<FloatType, Comments, TrailingCommas, NulIsEnd, true> bld(d, src, size);
const bool ok = bld.run(); const bool ok = bld.run();
failure = bld.failure(); failure = bld.failure();
return ok; return ok;
} }
builder<FloatType, Comments, TrailingCommas, NulIsEnd, false, Store> bld(d, src, size); builder<FloatType, Comments, TrailingCommas, NulIsEnd, false> bld(d, src, size);
const bool ok = bld.run(); const bool ok = bld.run();
failure = bld.failure(); failure = bld.failure();
return ok; return ok;
} }
/// sentinel: src[size] is readable and 0 (e.g. std::string); FloatType: the /// sentinel: src[size] is readable and 0 (e.g. std::string); FloatType: the
/// number_float_t of the document; validate_only: accept(), only the result is /// number_float_t of the document
/// used
template<typename FloatType, bool NulIsEnd> template<typename FloatType, bool NulIsEnd>
inline bool build(document_data& d, const char* src, std::size_t size, bool comments, bool trailing_commas, bool sentinel, bool validate_only, inline bool build(document_data& d, const char* src, std::size_t size, bool comments, bool trailing_commas, bool sentinel, parse_failure& failure)
parse_failure& failure)
{ {
if (validate_only && !comments && !trailing_commas)
{
// accept() of plain JSON: nothing is stored. The other option sets
// could skip the stores just as well; they keep the builder that
// stores only so that there are two more instantiations of the parse
// loop, not eight.
return build_with<FloatType, false, false, NulIsEnd, false>(d, src, size, sentinel, failure);
}
if (comments) if (comments)
{ {
return trailing_commas ? build_with<FloatType, true, true, NulIsEnd, true>(d, src, size, sentinel, failure) return trailing_commas ? build_with<FloatType, true, true, NulIsEnd>(d, src, size, sentinel, failure)
: build_with<FloatType, true, false, NulIsEnd, true>(d, src, size, sentinel, failure); : build_with<FloatType, true, false, NulIsEnd>(d, src, size, sentinel, failure);
} }
return trailing_commas ? build_with<FloatType, false, true, NulIsEnd, true>(d, src, size, sentinel, failure) return trailing_commas ? build_with<FloatType, false, true, NulIsEnd>(d, src, size, sentinel, failure)
: build_with<FloatType, false, false, NulIsEnd, true>(d, src, size, sentinel, failure); : build_with<FloatType, false, false, NulIsEnd>(d, src, size, sentinel, failure);
} }
} // namespace view } // namespace view
@@ -10215,7 +10152,7 @@ class basic_json_document
const bool ignore_trailing_commas = false) const bool ignore_trailing_commas = false)
{ {
basic_json_document d; basic_json_document d;
d.read_range(first, last, error_mode(allow_exceptions), ignore_comments, ignore_trailing_commas); d.read_range(first, last, allow_exceptions, ignore_comments, ignore_trailing_commas);
return d; return d;
} }
@@ -10230,7 +10167,7 @@ class basic_json_document
basic_json_document d; basic_json_document d;
std::string buf; std::string buf;
append_input(buf, std::forward<InputType>(input)); append_input(buf, std::forward<InputType>(input));
d.build_owned(std::move(buf), error_mode(allow_exceptions), ignore_comments, ignore_trailing_commas); d.build_owned(std::move(buf), allow_exceptions, ignore_comments, ignore_trailing_commas);
return d; return d;
} }
@@ -10243,8 +10180,7 @@ class basic_json_document
static bool accept(InputType&& input, const bool ignore_comments = false, const bool ignore_trailing_commas = false) static bool accept(InputType&& input, const bool ignore_comments = false, const bool ignore_trailing_commas = false)
{ {
basic_json_document d; basic_json_document d;
d.read_kind(std::forward<InputType>(input), on_error::validate, ignore_comments, ignore_trailing_commas, d.read(std::forward<InputType>(input), false, ignore_comments, ignore_trailing_commas);
std::integral_constant<detail::view::input_kind, detail::view::classify_input<InputType>::value> {});
return !d.is_discarded(); return !d.is_discarded();
} }
@@ -10260,7 +10196,7 @@ class basic_json_document
const bool ignore_comments = false, const bool ignore_comments = false,
const bool ignore_trailing_commas = false) const bool ignore_trailing_commas = false)
{ {
read_kind(std::forward<InputType>(input), error_mode(allow_exceptions), ignore_comments, ignore_trailing_commas, read_kind(std::forward<InputType>(input), allow_exceptions, ignore_comments, ignore_trailing_commas,
std::integral_constant<detail::view::input_kind, detail::view::classify_input<InputType>::value> {}); std::integral_constant<detail::view::input_kind, detail::view::classify_input<InputType>::value> {});
} }
@@ -10619,22 +10555,6 @@ class basic_json_document
return idx; return idx;
} }
/// what a failed parse does
enum class on_error : std::uint8_t
{
discard, ///< the document is discarded
raise, ///< parse_error is thrown
validate ///< accept(): the document is discarded, and no node is stored while parsing
};
static on_error error_mode(bool allow_exceptions) noexcept
{
return allow_exceptions ? on_error::raise : on_error::discard;
}
/// NulIsEnd of the builder: a null byte ends the input unless strict_nul_handling
static constexpr bool nul_is_end = !detail::abi_config::strict_nul_handling;
/// create the storage (sized for the input) on first use /// create the storage (sized for the input) on first use
void ensure_data(const char* src, std::size_t size) void ensure_data(const char* src, std::size_t size)
{ {
@@ -10646,7 +10566,7 @@ class basic_json_document
/// parse a buffer the document takes ownership of (it replaces the /// parse a buffer the document takes ownership of (it replaces the
/// previous owned copy) /// previous owned copy)
void build_owned(std::string&& buf, on_error allow_exceptions, bool comments, bool trailing_commas) void build_owned(std::string&& buf, bool allow_exceptions, bool comments, bool trailing_commas)
{ {
ensure_data(buf.data(), buf.size()); ensure_data(buf.data(), buf.size());
m_data->owned = std::move(buf); m_data->owned = std::move(buf);
@@ -10656,7 +10576,7 @@ class basic_json_document
/// parse a copy of the input; the copy is appended to the cleared owned /// parse a copy of the input; the copy is appended to the cleared owned
/// buffer, so that its capacity is reused /// buffer, so that its capacity is reused
template<typename T> template<typename T>
void build_copy(T&& input, on_error allow_exceptions, bool comments, bool trailing_commas) void build_copy(T&& input, bool allow_exceptions, bool comments, bool trailing_commas)
{ {
if (!m_data) if (!m_data)
{ {
@@ -10677,7 +10597,7 @@ class basic_json_document
} }
/// sentinel: src[size] is readable and 0 (std::string, C strings) /// sentinel: src[size] is readable and 0 (std::string, C strings)
void build(const char* src, std::size_t size, on_error allow_exceptions, bool comments, bool trailing_commas, bool owned, bool sentinel) void build(const char* src, std::size_t size, bool allow_exceptions, bool comments, bool trailing_commas, bool owned, bool sentinel)
{ {
ensure_data(src, size); ensure_data(src, size);
document_data& d = *m_data; document_data& d = *m_data;
@@ -10704,8 +10624,7 @@ class basic_json_document
} }
else else
{ {
ok = detail::view::build<typename BasicJsonType::number_float_t, nul_is_end>(d, src, size, comments, trailing_commas, sentinel, ok = detail::view::build < typename BasicJsonType::number_float_t, !detail::abi_config::strict_nul_handling > (d, src, size, comments, trailing_commas, sentinel, failure);
allow_exceptions == on_error::validate, failure);
} }
if (NLOHMANN_VIEW_LIKELY(ok)) if (NLOHMANN_VIEW_LIKELY(ok))
{ {
@@ -10718,7 +10637,7 @@ class basic_json_document
return; return;
} }
std::vector<std::uint32_t>().swap(d.large_objects); std::vector<std::uint32_t>().swap(d.large_objects);
if (allow_exceptions == on_error::raise) if (allow_exceptions)
{ {
detail::view::throw_parse_failure<BasicJsonType>(failure, src, size, comments, trailing_commas); detail::view::throw_parse_failure<BasicJsonType>(failure, src, size, comments, trailing_commas);
} }
@@ -10726,13 +10645,13 @@ class basic_json_document
// --- input dispatch (see detail::view::input_kind) --- // --- input dispatch (see detail::view::input_kind) ---
void read_kind(std::string&& s, on_error ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::move_string> /*unused*/) void read_kind(std::string&& s, bool ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::move_string> /*unused*/)
{ {
build_owned(std::move(s), ae, c, tc); build_owned(std::move(s), ae, c, tc);
} }
template<typename CharT> template<typename CharT>
void read_kind(CharT* s, on_error ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::c_string> /*unused*/) void read_kind(CharT* s, bool ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::c_string> /*unused*/)
{ {
if (s == nullptr) if (s == nullptr)
{ {
@@ -10746,7 +10665,7 @@ class basic_json_document
} }
template<typename Array> template<typename Array>
void read_kind(Array& a, on_error ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::char_array> /*unused*/) void read_kind(Array& a, bool ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::char_array> /*unused*/)
{ {
using CharT = typename std::remove_cv<typename std::remove_extent<Array>::type>::type; using CharT = typename std::remove_cv<typename std::remove_extent<Array>::type>::type;
const std::size_t n = std::extent<Array>::value; const std::size_t n = std::extent<Array>::value;
@@ -10757,7 +10676,7 @@ class basic_json_document
} }
template<typename T> template<typename T>
void read_kind(const T& s, on_error ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::borrow_range> /*unused*/) void read_kind(const T& s, bool ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::borrow_range> /*unused*/)
{ {
// std::basic_string guarantees data()[size()] == 0: use it as sentinel // std::basic_string guarantees data()[size()] == 0: use it as sentinel
const auto size = static_cast<std::size_t>(s.size()); const auto size = static_cast<std::size_t>(s.size());
@@ -10766,13 +10685,13 @@ class basic_json_document
} }
template<typename T> template<typename T>
void read_kind(const T& s, on_error ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::copy_range> /*unused*/) void read_kind(const T& s, bool ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::copy_range> /*unused*/)
{ {
build_copy(s, ae, c, tc); build_copy(s, ae, c, tc);
} }
template<typename CharT> template<typename CharT>
void read_kind(CharT* s, on_error ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::wide_c_string> /*unused*/) void read_kind(CharT* s, bool ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::wide_c_string> /*unused*/)
{ {
if (s == nullptr) if (s == nullptr)
{ {
@@ -10784,13 +10703,13 @@ class basic_json_document
} }
template<typename T> template<typename T>
void read_kind(T&& input, on_error ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::adapter> /*unused*/) void read_kind(T&& input, bool ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::adapter> /*unused*/)
{ {
build_copy(std::forward<T>(input), ae, c, tc); build_copy(std::forward<T>(input), ae, c, tc);
} }
template<typename IteratorType> template<typename IteratorType>
void read_range(IteratorType first, IteratorType last, on_error ae, bool c, bool tc) void read_range(IteratorType first, IteratorType last, bool ae, bool c, bool tc)
{ {
using value_type = typename std::remove_cv<typename std::iterator_traits<IteratorType>::value_type>::type; using value_type = typename std::remove_cv<typename std::iterator_traits<IteratorType>::value_type>::type;
// borrow the range where the library's input adapter scans it as one // borrow the range where the library's input adapter scans it as one
@@ -10801,14 +10720,14 @@ class basic_json_document
} }
template<typename IteratorType> template<typename IteratorType>
void read_range_impl(IteratorType first, IteratorType last, on_error ae, bool c, bool tc, std::true_type /*contiguous bytes*/) void read_range_impl(IteratorType first, IteratorType last, bool ae, bool c, bool tc, std::true_type /*contiguous bytes*/)
{ {
const auto size = static_cast<std::size_t>(std::distance(first, last)); const auto size = static_cast<std::size_t>(std::distance(first, last));
build(size == 0 ? "" : reinterpret_cast<const char*>(&*first), size, ae, c, tc, false, size == 0); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast) build(size == 0 ? "" : reinterpret_cast<const char*>(&*first), size, ae, c, tc, false, size == 0); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
} }
template<typename IteratorType> template<typename IteratorType>
void read_range_impl(IteratorType first, IteratorType last, on_error ae, bool c, bool tc, std::false_type /*other*/) void read_range_impl(IteratorType first, IteratorType last, bool ae, bool c, bool tc, std::false_type /*other*/)
{ {
build_copy(detail::view::make_adapter_input(detail::input_adapter(first, last)), ae, c, tc); build_copy(detail::view::make_adapter_input(detail::input_adapter(first, last)), ae, c, tc);
} }
@@ -10939,7 +10858,6 @@ class tuple_element<N, ::nlohmann::detail::view::view_item<View>> // NOLINT(cert
#undef NLOHMANN_VIEW_UNLIKELY #undef NLOHMANN_VIEW_UNLIKELY
#undef NLOHMANN_VIEW_ALWAYS_INLINE #undef NLOHMANN_VIEW_ALWAYS_INLINE
#undef NLOHMANN_VIEW_NOINLINE #undef NLOHMANN_VIEW_NOINLINE
#undef NLOHMANN_VIEW_NOCLONE
#undef NLOHMANN_VIEW_NODISCARD #undef NLOHMANN_VIEW_NODISCARD
#undef NLOHMANN_VIEW_THROW #undef NLOHMANN_VIEW_THROW
#undef NLOHMANN_VIEW_LITTLE_ENDIAN #undef NLOHMANN_VIEW_LITTLE_ENDIAN
+46 -146
View File
@@ -13,7 +13,6 @@
#include <nlohmann/detail/view/builder.hpp> #include <nlohmann/detail/view/builder.hpp>
#include <nlohmann/detail/view/materialize.hpp> #include <nlohmann/detail/view/materialize.hpp>
#include <nlohmann/detail/view/string_ref.hpp> #include <nlohmann/detail/view/string_ref.hpp>
#include <nlohmann/json_view.hpp> // json_document::accept
#else #else
#include <nlohmann/json_view.hpp> // the single header contains the internal headers #include <nlohmann/json_view.hpp> // the single header contains the internal headers
#endif #endif
@@ -60,8 +59,7 @@ built build(const std::string& text, bool comments, bool trailing_commas, bool s
r.copy.assign(text.begin(), text.end()); r.copy.assign(text.begin(), text.end());
src = r.copy.data(); src = r.copy.data();
} }
constexpr bool nul_is_end = !nlohmann::detail::abi_config::strict_nul_handling; r.ok = nlohmann::detail::view::build < FloatType, !nlohmann::detail::abi_config::strict_nul_handling > (*r.data, src, text.size(), comments, trailing_commas, sentinel, r.failure);
r.ok = nlohmann::detail::view::build<FloatType, nul_is_end>(*r.data, src, text.size(), comments, trailing_commas, sentinel, false, r.failure);
r.data->src = src; r.data->src = src;
r.data->base[0] = src; r.data->base[0] = src;
r.data->base[1] = r.data->arena.data(); r.data->base[1] = r.data->arena.data();
@@ -174,22 +172,6 @@ void check_string(const std::string& content)
} }
} }
// accept() stores no nodes: it must accept and reject what json::accept does, for all options and for inputs with
// and without a terminating NUL
void check_accept(const std::string& text)
{
CAPTURE(text)
const std::vector<char> bytes(text.begin(), text.end());
for (int options = 0; options < 4; ++options)
{
const bool comments = (options & 1) != 0;
const bool trailing_commas = (options & 2) != 0;
const bool accepted = json::accept(text, comments, trailing_commas);
CHECK(nlohmann::json_document::accept(text, comments, trailing_commas) == accepted);
CHECK(nlohmann::json_document::accept(bytes, comments, trailing_commas) == accepted);
}
}
// a small deterministic generator of documents // a small deterministic generator of documents
struct generator struct generator
{ {
@@ -441,133 +423,6 @@ TEST_CASE("json_view builder")
} }
} }
SECTION("accept() without an index")
{
for (const char* text :
{
"", " ", "\n", "null", "nul", "true", "[", "]", "[]", "{}", "[1,2,3]", "[1,2,3", "[1,2,3,]", "{\"a\":1,}", "{\"a\"}", "{\"a\":}",
"[1] x", "[1][2]", "\xEF\xBB\xBF[1]", "[\"a\\", "[\"\\u12\"]", "[\"\\ud800\"]", "[\"\x01\"]", "[\"\xff\"]", "[01]", "[1.]",
"[.5]", "[-]", "[1e]", "// c\n[1]", "/* c */ {\"a\" /* x */ : 1}", "/* c", "[1] // c", "1e400", "-1e400", "123456789012345678901234567890",
"{\"a\":1,\"a\":2}", "[[[]]]", "\"\\u00e4\\n\\ud83d\\ude00\""
})
{
check_accept(text);
}
// a null byte inside the text, and after it
check_accept(std::string("[1,\0]", 5));
check_accept(std::string("[1]\0", 4));
check_accept(std::string("\0", 1));
// generated documents and damaged copies, as for the builder above
generator g;
for (int i = 0; i < 2000; ++i)
{
std::string text;
g.value(text, 0);
check_accept(text);
std::string damaged = text;
const auto at = static_cast<std::size_t>(g.r(static_cast<int>(damaged.size())));
static const char replacements[] = {'x', '"', '\\', ',', ':', ']', '}', '[', '{', '1', '-', '.', 'e', '\0', '\n', '/'}; // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
damaged[at] = replacements[g.r(16)];
check_accept(damaged);
check_accept(text.substr(0, at));
}
// nesting beyond the 64 levels that are kept on the builder's own stack, closed and not closed
for (const std::size_t depth :
{
63u, 64u, 65u, 66u, 200u, 1000u
})
{
check_accept(std::string(depth, '[') + std::string(depth, ']'));
check_accept(std::string(depth, '[') + std::string(depth - 1, ']'));
std::string objects;
for (std::size_t i = 0; i < depth; ++i)
{
objects += "{\"k\":";
}
objects += "1" + std::string(depth, '}');
check_accept(objects);
check_accept(objects.substr(0, objects.size() - 1));
}
// objects with enough members to get a hash index when they are indexed (and duplicate keys)
std::string large = "{";
for (int i = 0; i < 300; ++i)
{
large += (i == 0 ? "\"k" : ",\"k") + std::to_string(i % 200) + "\":" + std::to_string(i);
}
check_accept(large + "}");
check_accept(large);
check_accept("[" + large + "},{" + large.substr(1) + "}]");
// the overflow check uses the floating-point type of the document
using float_json = nlohmann::basic_json<std::map, std::vector, std::string, bool, std::int64_t, std::uint64_t, float>;
for (const char* text :
{
"1e39", "-1e39", "3.4028235e38", "3.4028236e38", "[1.5e38, 3.5e38]", "{\"a\": 1e-50, \"b\": 1e39}", "340282356779733661637539395458142568448"
})
{
CAPTURE(text)
CHECK(nlohmann::basic_json_document<float_json>::accept(text) == float_json::accept(text));
}
// strings are checked and not decoded: runs of text of every length from 0 to 40 between valid and invalid
// escapes, UTF-8 sequences and control characters, in values and in keys, cut near the end of the text
const std::array<const char*, 9> valid = {{"\\n", "\\\"", "\\\\", "\\/", "\\t", "\\u00e4", "\\u20ac", "\\ud83d\\ude00", "\\u0000"}};
const std::array<const char*, 12> invalid = {{"\\x", "\\ud800", "\\udc00", "\\u12G4", "\\u12", "\\ud83dx", "\\ud83d\\u0041", "\\ud83d\\ud83d", "\\", "\x01", "\x1f", "\x7f"}};
const std::array<const char*, 6> utf8 = {{"\xc3\xa4", "\xe2\x82\xac", "\xf0\x9f\x98\x80", "\xc3", "\xff", "\x80"}};
std::mt19937 rng(2029); // NOLINT(cert-msc32-c,cert-msc51-cpp,bugprone-random-generator-seed): reproducible
for (std::size_t run = 0; run <= 40; ++run)
{
for (int i = 0; i < 30; ++i)
{
std::string content = std::string(rng() % 20, 'a');
const auto pieces = 1 + (rng() % 6);
for (unsigned k = 0; k < pieces; ++k)
{
content += valid[rng() % valid.size()];
content += std::string(k == 0 ? run : rng() % 41, static_cast<char>('b' + (rng() % 20)));
switch (rng() % 8)
{
case 0:
content += utf8[rng() % 3];
break;
case 1:
content += utf8[rng() % utf8.size()];
break;
case 2:
content += invalid[rng() % invalid.size()];
break;
default:
break;
}
}
for (const std::string& text :
{
"[\"" + content + "\"]", "{\"" + content + "\":1}", "[\"" + content, "[\"" + content.substr(0, content.size() - (rng() % 4)) + "\"]"
})
{
check_accept(text);
}
}
}
}
SECTION("escapes in a row")
{
for (const char* content :
{
"\\n", "\\n\\t\\\\", "\\u00e9\\u00e9\\u4e2d", "\\u00e9\\u00e", "\\u00e9\\", "\\n\\q", "\\n\\ud83d\\ude00\\n",
"\\ud83d\\ude00\\ud83d", "\\n\\n\\n abc \\n\\n", "\\n\x01", "\\n\xC3\x28", "\\n\\"
})
{
check_accept(std::string("[\"") + content + "\"]");
check_accept(std::string("[\"") + content);
check_accept(std::string("{\"") + content + "\":1}");
}
}
SECTION("test files") SECTION("test files")
{ {
for (const char* name : for (const char* name :
@@ -668,6 +523,51 @@ TEST_CASE("json_view builder: strings across vector blocks")
} }
} }
TEST_CASE("json_view builder: fraction digits")
{
// The digits of a fraction are skipped 16 bytes at a time where there are
// that many left (SSE2), and one at a time after that. Fractions of every
// length from 0 to 40 digits, followed by every kind of byte that ends a
// number or lies next to the digits in ASCII ('/' and ':'), as values, as
// keys' neighbours, and at the very end of the text (a vector without a
// terminating NUL). The length of the number is part of the value that
// the test compares with json::parse().
const std::string followers = ",]}/:.eE+-x \n\t\"09";
std::mt19937 rng(2028); // NOLINT(cert-msc32-c,cert-msc51-cpp,bugprone-random-generator-seed): reproducible
for (std::size_t digits = 0; digits <= 40; ++digits)
{
for (int i = 0; i < 3; ++i)
{
std::string fraction;
for (std::size_t k = 0; k < digits; ++k)
{
fraction += static_cast<char>('0' + (rng() % 10));
}
const std::string number = std::string(rng() % 2 == 0 ? "-" : "") + std::to_string(rng() % 100000) + "." + fraction;
check_same(number);
check_same("[" + number + "]");
std::string pair = "[" + number;
pair += ",";
pair += number;
pair += "]";
check_same(pair);
check_same("{\"a\":" + number + "}");
for (const char follower : followers)
{
check_same(number + follower);
check_same("[" + number + follower + "]");
check_same("[" + number + follower + std::string(rng() % 20, '1'));
}
// everything near the end of the text: the number cut in every place
for (std::size_t cut = 1; cut <= 18 && cut <= number.size(); ++cut)
{
check_same(number.substr(0, number.size() - cut));
check_same("[" + number.substr(0, number.size() - cut));
}
}
}
}
TEST_CASE("json_view builder: escapes between runs of text") TEST_CASE("json_view builder: escapes between runs of text")
{ {
// The text between escapes is copied 16 bytes at a time where there are // The text between escapes is copied 16 bytes at a time where there are
+1 -1
View File
@@ -25,7 +25,7 @@
#endif #endif
// the view's own macros do not leak // the view's own macros do not leak
#if defined(NLOHMANN_VIEW_LIKELY) || defined(NLOHMANN_VIEW_UNLIKELY) || defined(NLOHMANN_VIEW_ALWAYS_INLINE) || defined(NLOHMANN_VIEW_NOINLINE) || defined(NLOHMANN_VIEW_NOCLONE) \ #if defined(NLOHMANN_VIEW_LIKELY) || defined(NLOHMANN_VIEW_UNLIKELY) || defined(NLOHMANN_VIEW_ALWAYS_INLINE) || defined(NLOHMANN_VIEW_NOINLINE) \
|| defined(NLOHMANN_VIEW_NODISCARD) || defined(NLOHMANN_VIEW_THROW) || defined(NLOHMANN_VIEW_HAS_CPP_17) || defined(NLOHMANN_VIEW_LITTLE_ENDIAN) \ || defined(NLOHMANN_VIEW_NODISCARD) || defined(NLOHMANN_VIEW_THROW) || defined(NLOHMANN_VIEW_HAS_CPP_17) || defined(NLOHMANN_VIEW_LITTLE_ENDIAN) \
|| defined(NLOHMANN_VIEW_REPEAT16) || defined(NLOHMANN_VIEW_REPEAT16)
#error "json_view.hpp leaks a macro" #error "json_view.hpp leaks a macro"