mirror of
https://github.com/nlohmann/json.git
synced 2026-10-22 22:08:55 +00:00
operator== of views (view/view, view/json, and editable documents) built two
vectors of members for every object it entered, sorted them with stable_sort
for every object, and converted each number to a basic_json value. It now:
- keeps the members of all open objects in one vector (reserved with
geometric growth, so that nested objects do not reallocate), and orders
them by insertion sort for up to 16 members, or not at all when the keys
ascend already;
- compares numbers of one type directly, and builds a basic_json only for
numbers of different types;
- compares short texts (keys, numbers, most strings) with two overlapping
loads (bytes_equal(), in node.hpp);
- first walks both documents in lockstep, in the order of their text, which
finds most pairs of equal values without the work of the comparison. The
walk only enters an object if the comparison would visit its members in the
same order (the object type keeps the order, the keys ascend, or all values
are scalars), so a difference is never found later than before, and it
records the containers it found different, so that the comparison does not
walk them again;
- rejects values of different types and arrays of different sizes before any
of this, and, for an object type that keeps the order, objects whose first
(up to four) members have different keys, which ends a comparison that
differs near the start without preparing the members.
Results are unchanged: numbers compare by value across types, duplicate keys
resolve as parse() resolves them, and objects compare in document order for
ordered_json and by key otherwise.
Instructions of one comparison of a document with an equal copy, macOS arm64
(develop -> now), json_document / ordered_json_document / editable:
twitter 16.1M -> 7.0M 17.7M -> 1.5M 16.6M -> 7.3M
citm 45.0M -> 3.4M 47.0M -> 3.2M 45.7M -> 4.3M
canada 111.4M -> 13.7M 111.4M -> 13.7M 113.3M -> 15.0M
jeopardy 26.2M -> 3.7M 27.3M -> 3.7M 26.6M -> 12.1M
and of a view with a basic_json value:
twitter 11.4M -> 5.9M 11.2M -> 7.3M 11.6M -> 5.9M
citm 33.9M -> 9.7M 32.6M -> 9.1M 34.6M -> 9.8M
canada 92.9M -> 32.1M 92.9M -> 32.1M 93.9M -> 32.8M
and Linux instruction counts (GCC 13 / Clang 18) of the same comparisons:
document with an equal copy json_document ordered_json_document editable
twitter -49 % / -52 % -89 % / -89 % -48 % / -50 %
citm -90 % / -89 % -91 % / -90 % -88 % / -87 %
jeopardy (4,000 questions) -85 % / -85 % -86 % / -86 % -46 % / -49 %
view with a basic_json value
twitter -41 % / -45 % -28 % / -35 % -45 % / -44 %
citm -63 % / -65 % -63 % / -66 % -65 % / -64 %
jeopardy -39 % / -43 % -14 % / -24 % -44 % / -43 %
A sweep that places one difference at about 2,100 positions of eight
documents (a token at an even spacing, the first and last tokens, the first,
middle and last member of the root removed, one appended, the document
wrapped) in five kinds of comparison (view/view of read-only and of editable
documents, view/json, json_document, editable/view) finds the same results as
before and no placement slower on the Mac (Clang); on Linux no placement is
slower than 3 % (Clang and GCC).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
209 lines
8.3 KiB
C++
209 lines
8.3 KiB
C++
// __ _____ _____ _____
|
|
// __| | __| | | | JSON for Modern C++
|
|
// | | |__ | | | | | | version 3.12.0
|
|
// |_____|_____|_____|_|___| https://github.com/nlohmann/json
|
|
//
|
|
// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
|
|
// SPDX-License-Identifier: MIT
|
|
|
|
#pragma once
|
|
|
|
#include <cstddef> // size_t
|
|
#include <cstdint> // uint8_t, uint16_t, uint32_t, uint64_t
|
|
#include <cstring> // memcpy
|
|
|
|
#include <nlohmann/json.hpp>
|
|
#include <nlohmann/detail/view/macro_scope.hpp>
|
|
|
|
NLOHMANN_JSON_NAMESPACE_BEGIN
|
|
namespace detail
|
|
{
|
|
namespace view
|
|
{
|
|
|
|
/// The largest input a document accepts, in bytes. Offsets and node counts are
|
|
/// 32 bits wide; the limit keeps 16 bytes (the width of the scanner's steps)
|
|
/// below 2^32, so that a position one step past the end of the text fits.
|
|
constexpr std::size_t max_input_size() noexcept
|
|
{
|
|
return 0xFFFFFFEFu;
|
|
}
|
|
|
|
/// node flags
|
|
struct node_flags
|
|
{
|
|
static constexpr std::uint8_t escaped = 1; ///< string payload lives in the decode arena, not the source
|
|
static constexpr std::uint8_t edited = 2; ///< string or number token lives in the edit arena (editable documents)
|
|
static constexpr std::uint8_t storage = escaped | edited; ///< mask: where a string or number token lives (index into document_data::base)
|
|
static constexpr std::uint8_t is_true = 4; ///< boolean value
|
|
static constexpr std::uint8_t moved = 8; ///< array/object: the elements live in a separate sequence (editable documents)
|
|
static constexpr std::uint8_t is_new = 16; ///< written by an edit: no source position
|
|
static constexpr std::uint8_t linked = 32; ///< an entry of a moved sequence links to this value (editable documents): its extent in the parsed layout no longer matters
|
|
};
|
|
|
|
/// kind of an entry of an edited sequence that stands for a value stored
|
|
/// elsewhere (the address of the value's node is kept in the len/next bytes)
|
|
constexpr std::uint8_t kind_link = 10;
|
|
|
|
/// One entry of the flat index, in document order. An object's members are
|
|
/// stored as key node followed by the value's subtree. Integers keep their
|
|
/// converted 64-bit value in the len/next bytes (the node after a scalar is
|
|
/// always the next one, and the token length follows from `extra`).
|
|
struct node
|
|
{
|
|
std::uint8_t kind; ///< value_t, or kind_link
|
|
std::uint8_t flags; ///< node_flags
|
|
std::uint16_t extra; ///< numbers: integer digits (low byte) and fraction digits (high byte), 255 = "many"; objects: number of the object's key hash index (1-based, 0 = none); otherwise 0
|
|
std::uint32_t off; ///< source offset (string content, number token, literal, bracket); arena offset if escaped/edited; number of the element sequence if moved
|
|
std::uint32_t len; ///< string: decoded bytes; float: token bytes; array/object: element count
|
|
std::uint32_t next; ///< array/object: number of nodes of the subtree (its extent in the enclosing sequence)
|
|
};
|
|
static_assert(sizeof(node) == 16, "node must stay 16 bytes");
|
|
|
|
// the node kinds are value_t values; the test below (object and array are 1
|
|
// and 2) and those of the number kinds depend on this numbering
|
|
static_assert(static_cast<std::uint8_t>(value_t::null) == 0 && static_cast<std::uint8_t>(value_t::object) == 1
|
|
&& static_cast<std::uint8_t>(value_t::array) == 2 && static_cast<std::uint8_t>(value_t::string) == 3
|
|
&& static_cast<std::uint8_t>(value_t::boolean) == 4 && static_cast<std::uint8_t>(value_t::number_integer) == 5
|
|
&& static_cast<std::uint8_t>(value_t::number_unsigned) == 6 && static_cast<std::uint8_t>(value_t::number_float) == 7,
|
|
"the node format depends on the numbering of value_t");
|
|
|
|
NLOHMANN_VIEW_ALWAYS_INLINE bool is_container(const node& n) noexcept
|
|
{
|
|
return static_cast<unsigned>(n.kind) - 1u <= 1u;
|
|
}
|
|
|
|
/// the value a link node stands for
|
|
NLOHMANN_VIEW_ALWAYS_INLINE const node* link_target(const node& n) noexcept
|
|
{
|
|
const node* t = nullptr;
|
|
std::memcpy(static_cast<void*>(&t), reinterpret_cast<const unsigned char*>(&n) + 8, sizeof(const node*)); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
|
|
return t;
|
|
}
|
|
|
|
/// let the entry n stand for the value at target (and mark the value)
|
|
inline void make_link(node& n, node* target) noexcept
|
|
{
|
|
target->flags = static_cast<std::uint8_t>(target->flags | node_flags::linked);
|
|
n = node{};
|
|
n.kind = kind_link;
|
|
std::memcpy(reinterpret_cast<unsigned char*>(&n) + 8, static_cast<const void*>(&target), sizeof(const node*)); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
|
|
}
|
|
|
|
/// the converted value of an integer node: len is its low half, next its high
|
|
/// half (on little-endian targets the two words are the value in memory)
|
|
NLOHMANN_VIEW_ALWAYS_INLINE std::uint64_t integer_bits(const node& n) noexcept
|
|
{
|
|
#if NLOHMANN_VIEW_LITTLE_ENDIAN
|
|
std::uint64_t v = 0;
|
|
std::memcpy(&v, reinterpret_cast<const unsigned char*>(&n) + 8, 8); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
|
|
return v;
|
|
#else
|
|
return static_cast<std::uint64_t>(n.len) | (static_cast<std::uint64_t>(n.next) << 32);
|
|
#endif
|
|
}
|
|
|
|
NLOHMANN_VIEW_ALWAYS_INLINE void set_integer_bits(node& n, std::uint64_t v) noexcept
|
|
{
|
|
#if NLOHMANN_VIEW_LITTLE_ENDIAN
|
|
std::memcpy(reinterpret_cast<unsigned char*>(&n) + 8, &v, 8); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
|
|
#else
|
|
n.len = static_cast<std::uint32_t>(v);
|
|
n.next = static_cast<std::uint32_t>(v >> 32);
|
|
#endif
|
|
}
|
|
|
|
/// token length of a number node
|
|
NLOHMANN_VIEW_ALWAYS_INLINE std::uint32_t number_length(const node& n) noexcept
|
|
{
|
|
return n.kind == static_cast<std::uint8_t>(value_t::number_float) ? n.len
|
|
: (n.extra & 0xFFu) + (n.kind == static_cast<std::uint8_t>(value_t::number_integer) ? 1u : 0u);
|
|
}
|
|
|
|
/// whether n bytes are equal; the short texts of JSON (keys, numbers, most
|
|
/// strings) are compared by two overlapping loads instead of a library call
|
|
NLOHMANN_VIEW_ALWAYS_INLINE bool bytes_equal(const char* x, const char* y, std::size_t n) noexcept
|
|
{
|
|
const auto load8 = [](const char* p)
|
|
{
|
|
std::uint64_t v = 0;
|
|
std::memcpy(&v, p, sizeof(v));
|
|
return v;
|
|
};
|
|
const auto load4 = [](const char* p)
|
|
{
|
|
std::uint32_t v = 0;
|
|
std::memcpy(&v, p, sizeof(v));
|
|
return v;
|
|
};
|
|
if (n >= 8)
|
|
{
|
|
std::size_t i = 0;
|
|
for (; i + 8 < n; i += 8)
|
|
{
|
|
if (load8(x + i) != load8(y + i))
|
|
{
|
|
return false;
|
|
}
|
|
}
|
|
return load8(x + n - 8) == load8(y + n - 8);
|
|
}
|
|
if (n >= 4)
|
|
{
|
|
return load4(x) == load4(y) && load4(x + n - 4) == load4(y + n - 4);
|
|
}
|
|
for (std::size_t i = 0; i < n; ++i)
|
|
{
|
|
if (x[i] != y[i])
|
|
{
|
|
return false;
|
|
}
|
|
}
|
|
return true;
|
|
}
|
|
|
|
/// whether the n1 bytes at x are before the n2 bytes at y (memcmp() order, then the shorter first);
|
|
/// the first difference is found eight bytes at a time
|
|
NLOHMANN_VIEW_ALWAYS_INLINE bool bytes_less(const char* x, std::size_t n1, const char* y, std::size_t n2) noexcept
|
|
{
|
|
const std::size_t n = n1 < n2 ? n1 : n2;
|
|
std::size_t i = 0;
|
|
for (; i + 8 <= n; i += 8)
|
|
{
|
|
const std::uint64_t u = read_eight_bytes(x + i);
|
|
const std::uint64_t v = read_eight_bytes(y + i);
|
|
if (u != v)
|
|
{
|
|
const auto shift = static_cast<unsigned>(count_trailing_zeros(u ^ v)) & ~7u; // (the first byte that differs)
|
|
return ((u >> shift) & 0xFFu) < ((v >> shift) & 0xFFu);
|
|
}
|
|
}
|
|
for (; i < n; ++i)
|
|
{
|
|
if (x[i] != y[i])
|
|
{
|
|
return static_cast<unsigned char>(x[i]) < static_cast<unsigned char>(y[i]);
|
|
}
|
|
}
|
|
return n1 < n2;
|
|
}
|
|
|
|
/// estimated number of nodes for an input of `size` bytes (one node per ~12
|
|
/// bytes covers typical documents without regrowth)
|
|
inline std::size_t estimate_nodes(std::size_t size) noexcept
|
|
{
|
|
return (size / 12) + 16;
|
|
}
|
|
|
|
/// estimated number of nodes for the input [src, src + size): pretty-printed
|
|
/// input (whitespace after the first byte) needs about a node per 12 bytes,
|
|
/// minified input up to one per 4 (yyjson tells the two apart the same way)
|
|
inline std::size_t estimate_nodes(const char* src, std::size_t size) noexcept
|
|
{
|
|
return size >= 2 && (src[1] == ' ' || src[1] == '\n' || src[1] == '\r' || src[1] == '\t') ? estimate_nodes(size) : (size / 4) + 16;
|
|
}
|
|
|
|
} // namespace view
|
|
} // namespace detail
|
|
NLOHMANN_JSON_NAMESPACE_END
|