mirror of
https://github.com/nlohmann/json.git
synced 2026-10-25 15:28:54 +00:00
* Add shared detail helpers for the #5891 cleanup Add detail::is_primitive(value_t), detail::hex4_lower(), detail::utf8_shape(), detail::input_format_name(), detail::empty_input_message(), the exception ids 116, 319, 320 and 416 to detail::exception_id, and make detail::is_ascii_copyable() constexpr. Later commits switch the duplicated code over to them. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 1.4: BSON writers share one write-pass UTF-8 sanitizer (#5891) write_bson_entry_header and write_bson_string re-derived the error-handler policy differently. Both now use sanitize_utf8_for_bson_write, which treats strict like keep (the size pass already validated) and sanitizes replace/ignore into the bytes the size pass measured. No context plumbing is needed. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.1.1: one MessagePack container prefix helper (#5891) Array and object prefixes differed only in the fix marker and the 16/32-bit markers; both now delegate to write_msgpack_container_prefix. Output is byte-identical. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.1.2: one iterative walker for CBOR and MessagePack (#5891) write_cbor_iterative and write_msgpack_iterative (and their *_value_or_push helpers) were identical but for the head, scalar, and key writers. They are now one template over a small policy struct, so everything inlines. BON8 keeps its own walker (open string run, closing marker), noted in a comment. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.1.3, 2.1.8: one UBJSON container start helper, shared frame base (#5891) write_ubjson_start_array/_object are now write_ubjson_start_container<IsArray>, keeping the array-only begin()+1 scan and excessive_valueless rule. ubjson_frame derives from binary_container_frame instead of duplicating it. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.1.4: write_bytes helper for the byte-string write sites (#5891) The 13 oa.write_characters(reinterpret_cast<const CharType*>(x.data()), n) sites now call write_bytes(const void*, size_t); copy_characters sites and the buffer-based writes are untouched. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.1.5: shared object-key writer for CBOR and MessagePack (#5891) write_cbor_key and write_msgpack_key were the same SFINAE pair apart from the string/scalar writers; both now delegate to write_object_key<Walk>. UBJSON key and to_format_length stay as they are. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.3.10: writer-side format names from input_format_name (#5891) The CBOR/MessagePack/UBJSON/BSON/BJData/BON8 literals in throw_on_discarded and throw_integer_out_of_range calls now come from detail::input_format_name; the texts are identical, and UBJSON vs BJData still follows use_bjdata. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * serializer: constexpr ascii table, hex4_lower, dedupe helpers (#5891) 1.1: build ascii_copyable() at compile time from is_ascii_copyable and test find_ascii_copyable_run against it for every byte. 1.2/2.5.4: use detail::hex4_lower in both \uXXXX writers. 2.1.6: factor the repeated flush/emit blocks in dump_escaped_impl into always-inline helpers. 2.1.7: comment why the escape copy does not use put_buffer. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 1.6: use string_t-aware to_string/concat in meta() (#5891) Replaces the four std::to_string sites; rendered text unchanged. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 1.7: use detail::is_primitive in json.hpp switches (#5891) has_no_children, set_parents, max_size and the two diff walkers. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.2.1: factor insert()/iterator throw helpers (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.2.2: share swap() overloads via swap_impl (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.2.4: factor primitive reset and range check in erase()/range ctor (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.2.6: share push_back array bodies via push_back_array_impl (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.2.7: route parse/accept/sax_parse through *_impl helpers (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.2.8: share the vector-returning to_* bodies via to_binary_vector (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.2.10: merge destroy_string/destroy_binary into destroy_pointer (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.4.1/1.7: exceptions: share what_impl, flatten diagnostics switch (#5891) The four create() bodies build their message through one protected static helper; the diagnostics parent walk handles array/object and ignores all other types. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.4.2/2.4.3/1.7: json_pointer: share slash and escape checks, drop primitive case lists (#5891) The leading-slash and invalid '~' escape checks are shared helpers; the reference-token walks test object and array first and treat everything else as primitive. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 1.7/2.4.9: iter_impl, iteration_proxy: drop primitive case lists, share compare preamble (#5891) Object and array stay the first branches; everything else is the primitive case. operator== and operator< share the different-container check. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.4.5/2.4.6: from_json: share get_number_value and reserve_if_possible (#5891) The three number cases of get_arithmetic_value and the arithmetic from_json share one helper (boolean stays accepted only by the latter); the array and object reserve overload pairs become one. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.4.8: ordered_map: share erase_impl between the key erase overloads (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 1.7: json_pointer: flatten walk drops the primitive case list (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 1.2/1.3/1.5/2.5.1/2.5.2/2.5.5: share helpers in the view code (#5891) - detail::exception_id at every numeric id; throw_* helpers take it - hex_byte, hex4_lower, utf8_shape replace local hex/table code - throw_array_index_out_of_range/_past_end, throw_key_not_found - has_native_float_format replaces the hand-written IEEE check No change of messages, ids or output. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 1.5: use utf8_shape in get_bon8_string, add exhaustive UTF-8 validator test (#5891) get_bon8_string takes the valid range of the second byte from utf8_shape() (0xF5..0xF7 stay errors, unget behavior unchanged). validate_one_utf8 keeps its if-chain (hot bulk path); the new test checks decode, validate_one_utf8, utf8_shape, json::accept and the BON8 reader against each other for every lead and second byte. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.3.1: get_cbor_sized_bytes for the CBOR chunk getters (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.3.2: get_string_with_length for the plain-order string sites (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.3.3, 2.3.4, 2.3.10: unsupported_key_type, empty_input_message, input_format_name in the readers (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.3.7, 2.3.9: read_object_key and leave_container_and_check for CBOR/MessagePack/BON8 (#5891) UBJSON keys (different getter signature) and its container tail stay as is. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.3.5: json_sax_dom_error_state base for the DOM SAX parsers (#5891) Holds errored, allow_exceptions, parse_error() and is_errored(). The scalar handlers stay in each class (parse hot path). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.3.6: table of control-character messages in the lexer (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.3.8: finish_input helper for the strict-EOF block in the parser (#5891) sax_parse keeps its if (result) guard and return value. Also adds the clang-tidy NOLINT for the 2.3.6 message table. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.5.7: share parse_array_index between json_pointer and json_view (#5891) Move json_pointer::parse_array_index() and its status enum to namespace detail (json_pointer keeps a forwarding member), and let the json_view pointer resolution use it instead of its own copy of the RFC 6901 index grammar. Both accepted the same tokens with the same outcome. Also update the comment on detail::decode(), which listed four of the five UTF-8 validators and now points to the test that keeps them in sync. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Amalgamate (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.4.5: keep the inlined number cases in from_json(ArithmeticType&) (#5891) Calling get_number_value() from from_json(ArithmeticType&) made GetArray<std::array<int, 1000>> 2.4x slower (it changes the inlining of the std::array conversion), so that overload keeps its own switch, with a comment. get_arithmetic_value() still uses the shared helper. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 2.5.7: force-inline detail::parse_array_index (#5891) Moving parse_array_index out of json_pointer stopped clang from inlining it into the pointer resolution (JsonPointerAt/citm_catalog +17%, JsonPointerAt/twitter +8%); JSON_HEDLEY_ALWAYS_INLINE restores the baseline. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 1.5: keep get_bon8_string's own UTF-8 ranges (#5891) Taking the second-byte range from utf8_shape() made BON8 decoding 2-3% slower (FromBinaryShape/scalars, FromBinaryBuffer, alignment-pinned builds), so the byte path keeps its hand-written ranges. The exhaustive validator test still checks that it accepts the same sequences as the other validators. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Amalgamate (#5891) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix CI failures: constexpr, NRVO, cppcheck, IWYU (#5891) - json_sax_dom_error_state::is_errored() cannot be constexpr: the class is not a literal type (GCC 4.8, GCC -Wpedantic, Clang 3.5). - to_binary_vector: destroy the vector writer before returning, as the output_vector_sink sets the size of the vector in its destructor. Without NRVO (MSVC Debug in C++14 mode), the vector was moved out before it was resized. - parser::sax_parse: return false instead of result after the early exit (cppcheck identicalConditionAfterEarlyExit). - json.hpp: export detail/exceptions.hpp for IWYU, so json_view.hpp can name exception_id without including a detail header. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address review: throw helpers take input_format_t, simpler type checks - throw_on_discarded() and throw_integer_out_of_range() take an input_format_t and call input_format_name() themselves - has_no_children(), set_parents() and max_size() check array, then object, instead of calling is_primitive() first - parse_impl() chains like accept_impl(); sax_parse_impl() uses if/else - std::array for the control character messages; negation<> in write_object_key(); clarify the is_primitive() comment Signed-off-by: Niels Lohmann <mail@nlohmann.me> * json_view: use exception_id in the value() overloads Signed-off-by: Niels Lohmann <mail@nlohmann.me> * json_sax: assert the invariant of place_container for the static analyzer Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>
745 lines
26 KiB
C++
745 lines
26 KiB
C++
// __ _____ _____ _____
|
|
// __| | __| | | | JSON for Modern C++
|
|
// | | |__ | | | | | | version 3.12.0
|
|
// |_____|_____|_____|_|___| https://github.com/nlohmann/json
|
|
//
|
|
// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
|
|
// SPDX-License-Identifier: MIT
|
|
|
|
#pragma once
|
|
|
|
#include <algorithm> // min, sort
|
|
#include <array> // array
|
|
#include <cstddef> // offsetof, size_t
|
|
#include <cstdint> // int64_t, uint8_t, uint16_t, uint32_t, uint64_t
|
|
#include <cstring> // memcmp, memcpy, memset
|
|
#include <limits> // numeric_limits
|
|
#include <string> // string
|
|
#include <vector> // vector
|
|
|
|
#include <nlohmann/json.hpp>
|
|
#include <nlohmann/detail/view/document_data.hpp>
|
|
#include <nlohmann/detail/view/errors.hpp>
|
|
#include <nlohmann/detail/view/macro_scope.hpp>
|
|
#include <nlohmann/detail/view/node.hpp>
|
|
#include <nlohmann/detail/view/number.hpp>
|
|
#include <nlohmann/detail/view/object_index.hpp>
|
|
#include <nlohmann/detail/view/scan.hpp>
|
|
|
|
// Images: a document stored so that loading it needs no parsing.
|
|
//
|
|
// Layout (little-endian): a 64-byte header, the nodes, the text (the source,
|
|
// followed by the number tokens written by edits), a NUL, the decoded strings
|
|
// (followed by the strings written by edits), a NUL. The idea is that of
|
|
// zero-copy formats such as FlatBuffers (https://github.com/google/flatbuffers)
|
|
// and YaFF (https://github.com/yandex/yaff); no code is taken from them.
|
|
// check_image follows the idea of FlatBuffers' Verifier (bounds and
|
|
// structure) and also checks what the parser guarantees about strings and
|
|
// numbers, so that reading and serializing a checked image is safe and yields
|
|
// valid JSON.
|
|
|
|
NLOHMANN_JSON_NAMESPACE_BEGIN
|
|
namespace detail
|
|
{
|
|
namespace view
|
|
{
|
|
|
|
/// how load() checks an image
|
|
enum class image_check
|
|
{
|
|
/// everything the parser guarantees: structure and bounds, strings (valid
|
|
/// UTF-8; source strings without quotes, backslashes, and control
|
|
/// characters), and numbers (well-formed, matching the stored values)
|
|
full,
|
|
/// structure and bounds only: reading and serializing are safe, but a
|
|
/// crafted image can yield invalid UTF-8, strings that serialize to
|
|
/// invalid JSON, or numbers that differ from their text
|
|
bounds,
|
|
/// none: for images from a trusted source only (a damaged image is
|
|
/// undefined behavior)
|
|
none,
|
|
};
|
|
|
|
struct image_header
|
|
{
|
|
std::array<char, 4> magic; ///< "NJVI"
|
|
std::uint32_t version; ///< 1
|
|
std::uint64_t node_count;
|
|
std::uint64_t text_size;
|
|
std::uint64_t arena_size;
|
|
std::array<std::uint64_t, 4> reserved; ///< zero (for later versions)
|
|
};
|
|
static_assert(sizeof(image_header) == 64, "the image header must be 64 bytes");
|
|
|
|
constexpr std::uint32_t image_version = 1;
|
|
|
|
/// the largest node count and text or string size of an image (as for parsed
|
|
/// documents, offsets and counts must fit 32 bits)
|
|
constexpr std::uint64_t image_limit = 0xFFFFFFF0u;
|
|
|
|
/// Copy the current structure of an edited document into nodes in document
|
|
/// order, as the parser would have written them. Text written by edits is
|
|
/// appended to text_tail (number tokens) and arena_tail (strings); floats that
|
|
/// are not finite become null, as dump() writes them.
|
|
inline void compact_nodes(const document_data& d, std::size_t arena_size, std::vector<node>& out, std::string& text_tail, std::string& arena_tail)
|
|
{
|
|
struct frame
|
|
{
|
|
const node* cur;
|
|
const node* end;
|
|
std::size_t index; ///< the container's node in out
|
|
std::uint32_t count;
|
|
bool object;
|
|
};
|
|
std::vector<frame> stack;
|
|
const auto string_node = [&](const node & s)
|
|
{
|
|
node r = s;
|
|
r.extra = 0;
|
|
r.flags = static_cast<std::uint8_t>(s.flags & node_flags::storage);
|
|
if (r.flags == node_flags::edited)
|
|
{
|
|
r.off = static_cast<std::uint32_t>(arena_size + arena_tail.size());
|
|
arena_tail.append(d.str(s), s.len);
|
|
r.flags = node_flags::escaped;
|
|
}
|
|
return r;
|
|
};
|
|
const auto copy_node = [&](const node * v)
|
|
{
|
|
node r = *v;
|
|
switch (static_cast<value_t>(v->kind))
|
|
{
|
|
case value_t::object:
|
|
case value_t::array:
|
|
r.flags = 0;
|
|
r.extra = 0;
|
|
r.off = (v->flags & (node_flags::moved | node_flags::is_new)) != 0 ? 0 : v->off;
|
|
r.len = 0; // counted below
|
|
r.next = 0; // set when the container is complete
|
|
stack.push_back(frame{d.first_child_edited(v), d.child_end_edited(v), out.size(), 0, v->kind == static_cast<std::uint8_t>(value_t::object)});
|
|
break;
|
|
case value_t::string:
|
|
r = string_node(*v);
|
|
break;
|
|
case value_t::number_integer:
|
|
case value_t::number_unsigned:
|
|
if ((v->flags & node_flags::storage) == node_flags::edited)
|
|
{
|
|
r.off = static_cast<std::uint32_t>(d.size + text_tail.size());
|
|
text_tail.append(d.str(*v), number_length(*v));
|
|
}
|
|
r.flags = 0;
|
|
break;
|
|
case value_t::number_float:
|
|
if ((v->flags & node_flags::storage) == node_flags::edited)
|
|
{
|
|
const char* const t = d.str(*v);
|
|
if (t[0] == 'n' || t[0] == 'i' || (v->len > 1 && t[1] == 'i'))
|
|
{
|
|
r = node{}; // nan and infinity: null, as dump() writes them
|
|
r.kind = static_cast<std::uint8_t>(value_t::null);
|
|
break;
|
|
}
|
|
r.off = static_cast<std::uint32_t>(d.size + text_tail.size());
|
|
text_tail.append(t, v->len);
|
|
r.extra = 0xFFFFu; // the digit layout is not recorded
|
|
}
|
|
r.flags = 0;
|
|
break;
|
|
case value_t::boolean:
|
|
r.flags = static_cast<std::uint8_t>(v->flags & node_flags::is_true);
|
|
break;
|
|
case value_t::null:
|
|
case value_t::binary:
|
|
case value_t::discarded:
|
|
default:
|
|
r.flags = 0;
|
|
break;
|
|
}
|
|
out.push_back(r);
|
|
};
|
|
copy_node(d.nodes);
|
|
while (!stack.empty())
|
|
{
|
|
const frame top = stack.back();
|
|
if (top.cur == top.end)
|
|
{
|
|
node& c = out[top.index];
|
|
c.len = top.count;
|
|
c.next = static_cast<std::uint32_t>(out.size() - top.index);
|
|
stack.pop_back();
|
|
continue;
|
|
}
|
|
if (top.object)
|
|
{
|
|
out.push_back(string_node(*top.cur));
|
|
}
|
|
const node* const value = top.cur + (top.object ? 1 : 0);
|
|
stack.back().cur = document_data::after(value);
|
|
++stack.back().count;
|
|
copy_node(document_data::deref(value)); // may grow the stack
|
|
}
|
|
}
|
|
|
|
/// the document as an image
|
|
inline std::vector<std::uint8_t> save_image(const document_data& d)
|
|
{
|
|
#if !NLOHMANN_VIEW_LITTLE_ENDIAN
|
|
throw_type_error(exception_id::document_image_unsupported, "json_document images need a little-endian target"); // LCOV_EXCL_LINE
|
|
#endif
|
|
const std::size_t arena_size = d.arena_size;
|
|
std::vector<node> compacted;
|
|
std::string text_tail;
|
|
std::string arena_tail;
|
|
if (d.edits)
|
|
{
|
|
compact_nodes(d, arena_size, compacted, text_tail, arena_tail);
|
|
}
|
|
const node* const nodes = d.edits ? compacted.data() : d.nodes;
|
|
const std::size_t count = d.edits ? compacted.size() : d.node_count;
|
|
const std::size_t text_size = d.size + text_tail.size();
|
|
const std::size_t total_arena = arena_size + arena_tail.size();
|
|
if (NLOHMANN_VIEW_UNLIKELY(text_size >= image_limit || total_arena >= image_limit || count >= image_limit))
|
|
{
|
|
// LCOV_EXCL_START (4 GiB)
|
|
throw_out_of_range(exception_id::document_too_large, "images of 4 GiB or more are not supported by json_document");
|
|
// LCOV_EXCL_STOP
|
|
}
|
|
image_header h{};
|
|
h.magic = {{'N', 'J', 'V', 'I'}};
|
|
h.version = image_version;
|
|
h.node_count = count;
|
|
h.text_size = text_size;
|
|
h.arena_size = total_arena;
|
|
std::vector<std::uint8_t> image(sizeof(h) + (count * sizeof(node)) + text_size + 1 + total_arena + 1);
|
|
std::uint8_t* o = image.data();
|
|
std::memcpy(o, &h, sizeof(h));
|
|
o += sizeof(h);
|
|
std::memcpy(o, nodes, count * sizeof(node));
|
|
// the hash indexes are rebuilt by load(), whose check rejects any other
|
|
// extra of an object, and the same document saves to the same bytes
|
|
for (std::size_t i = 0; i < count; ++i)
|
|
{
|
|
if (nodes[i].kind == static_cast<std::uint8_t>(value_t::object) && nodes[i].extra != 0)
|
|
{
|
|
std::memset(o + (i * sizeof(node)) + offsetof(node, extra), 0, sizeof(nodes[i].extra));
|
|
}
|
|
}
|
|
o += count * sizeof(node);
|
|
const auto append = [&o](const char* s, std::size_t n)
|
|
{
|
|
if (n != 0)
|
|
{
|
|
std::memcpy(o, s, n);
|
|
o += n;
|
|
}
|
|
};
|
|
append(d.src, d.size);
|
|
append(text_tail.data(), text_tail.size());
|
|
*o++ = 0;
|
|
append(d.base[1], arena_size);
|
|
append(arena_tail.data(), arena_tail.size());
|
|
*o = 0;
|
|
return image;
|
|
}
|
|
|
|
/// the parts of a number token that the checks need
|
|
struct number_token
|
|
{
|
|
const unsigned char* int_start;
|
|
std::size_t int_digits;
|
|
std::size_t frac_digits;
|
|
std::int64_t exponent;
|
|
bool negative;
|
|
bool is_float; ///< a fraction or an exponent
|
|
};
|
|
|
|
/// whether [s, s + len) is a JSON number (the grammar the parser accepts)
|
|
inline bool scan_number_token(const unsigned char* s, std::size_t len, number_token& t)
|
|
{
|
|
const unsigned char* const e = s + len;
|
|
const unsigned char* p = s;
|
|
t.negative = p != e && *p == '-';
|
|
p += t.negative ? 1 : 0;
|
|
if (p == e)
|
|
{
|
|
return false;
|
|
}
|
|
t.int_start = p;
|
|
if (*p == '0')
|
|
{
|
|
++p;
|
|
}
|
|
else if (*p >= '1' && *p <= '9')
|
|
{
|
|
do
|
|
{
|
|
++p;
|
|
}
|
|
while (p != e && is_digit(*p));
|
|
}
|
|
else
|
|
{
|
|
return false;
|
|
}
|
|
t.int_digits = static_cast<std::size_t>(p - t.int_start);
|
|
t.frac_digits = 0;
|
|
t.is_float = false;
|
|
if (p != e && *p == '.')
|
|
{
|
|
const unsigned char* const f0 = ++p;
|
|
while (p != e && is_digit(*p))
|
|
{
|
|
++p;
|
|
}
|
|
if (p == f0)
|
|
{
|
|
return false;
|
|
}
|
|
t.frac_digits = static_cast<std::size_t>(p - f0);
|
|
t.is_float = true;
|
|
}
|
|
t.exponent = 0;
|
|
if (p != e && (*p | 0x20u) == 'e')
|
|
{
|
|
++p;
|
|
const bool exp_negative = p != e && *p == '-';
|
|
p += (p != e && (*p == '+' || *p == '-')) ? 1 : 0;
|
|
if (p == e || !is_digit(*p))
|
|
{
|
|
return false;
|
|
}
|
|
do
|
|
{
|
|
t.exponent = t.exponent < 100000 ? (t.exponent * 10) + (*p - '0') : t.exponent;
|
|
++p;
|
|
}
|
|
while (p != e && is_digit(*p));
|
|
t.exponent = exp_negative ? -t.exponent : t.exponent;
|
|
t.is_float = true;
|
|
}
|
|
return p == e;
|
|
}
|
|
|
|
/// the digit layout the parser records for a float token (compaction
|
|
/// writes "many" instead; the caller accepts both)
|
|
inline std::uint16_t float_layout(const number_token& t)
|
|
{
|
|
return static_cast<std::uint16_t>((std::min)(t.int_digits, std::size_t{255}) | ((std::min)(t.frac_digits, std::size_t{255}) << 8u));
|
|
}
|
|
|
|
/// whether a float token (well-formed, per scan_number_token) is finite as
|
|
/// double; parse() rejects floats that overflow, and as there, only a number
|
|
/// whose magnitude could reach 1e308 needs the conversion
|
|
inline bool float_token_finite(const unsigned char* s, std::size_t len, const number_token& t)
|
|
{
|
|
if (static_cast<std::int64_t>(t.int_digits) + t.exponent > 300)
|
|
{
|
|
node n{};
|
|
n.kind = static_cast<std::uint8_t>(value_t::number_float);
|
|
n.len = static_cast<std::uint32_t>(len);
|
|
const auto v = float_value<double>(reinterpret_cast<const char*>(s), n); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
|
|
return v <= (std::numeric_limits<double>::max)() && v >= -(std::numeric_limits<double>::max)();
|
|
}
|
|
return true;
|
|
}
|
|
|
|
/// whether an integer node matches its token the way the parser records it
|
|
/// (after the bounds check; the token has at most 256 characters)
|
|
inline bool check_integer(const node& n, const unsigned char* text)
|
|
{
|
|
const std::size_t len = number_length(n);
|
|
const unsigned char* const s = text + n.off;
|
|
number_token t{};
|
|
if (!scan_number_token(s, len, t))
|
|
{
|
|
return false;
|
|
}
|
|
// the token's value is the stored one; number_integer nodes of edits can
|
|
// be non-negative (as basic_json keeps the type of a value)
|
|
const bool integer = n.kind == static_cast<std::uint8_t>(value_t::number_integer);
|
|
if (t.is_float || t.int_digits > 20 || (t.negative && !integer))
|
|
{
|
|
return false;
|
|
}
|
|
// (at most 19 digits cannot overflow; 20 digits are compared with 2^64 - 1)
|
|
if (t.int_digits == 20 && std::memcmp(t.int_start, "18446744073709551615", 20) > 0)
|
|
{
|
|
return false;
|
|
}
|
|
std::uint64_t m = 0;
|
|
for (const unsigned char* d = t.int_start; d != t.int_start + t.int_digits; ++d)
|
|
{
|
|
m = (m * 10) + static_cast<std::uint64_t>(*d - '0');
|
|
}
|
|
if (integer && m > (t.negative ? std::uint64_t{1} << 63u : (std::uint64_t{1} << 63u) - 1))
|
|
{
|
|
return false;
|
|
}
|
|
return integer_bits(n) == (t.negative ? 0 - m : m);
|
|
}
|
|
|
|
/// a byte range of the text or of the decoded strings
|
|
struct byte_range
|
|
{
|
|
std::uint32_t off;
|
|
std::uint32_t len;
|
|
};
|
|
|
|
/// a float token and the digit layout its node records
|
|
struct float_range
|
|
{
|
|
std::uint32_t off;
|
|
std::uint32_t len;
|
|
std::uint16_t extra;
|
|
};
|
|
|
|
/// Run check once for each distinct range of ranges (which are within bounds
|
|
/// and not empty). Ranges that are not identical must not overlap: save()
|
|
/// writes every string and token to a place of its own, and nodes that share
|
|
/// a value (copies within a document) share the whole range. This bounds the
|
|
/// work by the size of the text, however many nodes point to the same bytes.
|
|
template<typename Check>
|
|
bool check_distinct_ranges(std::vector<byte_range>& ranges, Check check)
|
|
{
|
|
std::sort(ranges.begin(), ranges.end(), [](const byte_range & a, const byte_range & b)
|
|
{
|
|
return a.off != b.off ? a.off < b.off : a.len < b.len;
|
|
});
|
|
std::size_t end = 0;
|
|
for (std::size_t i = 0; i < ranges.size(); ++i)
|
|
{
|
|
const byte_range r = ranges[i];
|
|
if (i != 0 && r.off == ranges[i - 1].off && r.len == ranges[i - 1].len)
|
|
{
|
|
continue;
|
|
}
|
|
if (r.off < end || !check(r))
|
|
{
|
|
return false;
|
|
}
|
|
end = static_cast<std::size_t>(r.off) + r.len;
|
|
}
|
|
return true;
|
|
}
|
|
|
|
/// Check the float nodes: each token once (nodes of the same token must
|
|
/// record layouts that match it), tokens must not overlap.
|
|
inline bool check_float_ranges(std::vector<float_range>& ranges, const unsigned char* text)
|
|
{
|
|
std::sort(ranges.begin(), ranges.end(), [](const float_range & a, const float_range & b)
|
|
{
|
|
if (a.off != b.off)
|
|
{
|
|
return a.off < b.off;
|
|
}
|
|
if (a.len != b.len)
|
|
{
|
|
return a.len < b.len;
|
|
}
|
|
return a.extra < b.extra;
|
|
});
|
|
std::size_t end = 0;
|
|
std::size_t i = 0;
|
|
while (i < ranges.size())
|
|
{
|
|
const float_range r = ranges[i];
|
|
if (r.off < end)
|
|
{
|
|
return false;
|
|
}
|
|
number_token t{};
|
|
const unsigned char* const s = text + r.off;
|
|
if (!scan_number_token(s, r.len, t) || !float_token_finite(s, r.len, t))
|
|
{
|
|
return false;
|
|
}
|
|
// the digit layout the parser records (or "many", as compaction writes it)
|
|
const std::uint16_t layout = float_layout(t);
|
|
for (; i < ranges.size() && ranges[i].off == r.off && ranges[i].len == r.len; ++i)
|
|
{
|
|
if (ranges[i].extra != layout && ranges[i].extra != 0xFFFFu)
|
|
{
|
|
return false;
|
|
}
|
|
}
|
|
end = static_cast<std::size_t>(r.off) + r.len;
|
|
}
|
|
return true;
|
|
}
|
|
|
|
/// the content checks of check_image: source strings as the parser leaves
|
|
/// them (no quotes, backslashes, or control characters), decoded strings
|
|
/// (valid UTF-8), and float tokens
|
|
inline bool check_contents(const unsigned char* text, const unsigned char* arena,
|
|
std::vector<byte_range>& source_strings, std::vector<byte_range>& decoded_strings,
|
|
std::vector<float_range>& floats)
|
|
{
|
|
return check_distinct_ranges(source_strings, [text](const byte_range & r)
|
|
{
|
|
const unsigned char* const b = text + r.off;
|
|
return scan_string_run(b, b + r.len) == b + r.len;
|
|
})
|
|
&& check_distinct_ranges(decoded_strings, [arena](const byte_range & r)
|
|
{
|
|
return valid_utf8_prefix(arena + r.off, r.len) == r.len;
|
|
})
|
|
&& check_float_ranges(floats, text);
|
|
}
|
|
|
|
/// Check the nodes of a loaded image against its text and decoded strings:
|
|
/// kinds, flags, and `extra`; extents and element counts of arrays and
|
|
/// objects; keys; bounds; string contents (source strings as the parser
|
|
/// leaves them: no quotes, backslashes, or control characters; all strings
|
|
/// valid UTF-8); and number tokens.
|
|
///
|
|
/// The structure and the bounds are checked node by node. The contents of
|
|
/// strings and of float tokens are checked afterwards, once for each distinct
|
|
/// range (see check_distinct_ranges), so that the full check is linear in the
|
|
/// size of the image plus the sorting of the ranges, and not in the number of
|
|
/// nodes times the size of the text.
|
|
inline bool check_image(const node* nodes, std::size_t count, const unsigned char* text, std::size_t text_size,
|
|
const unsigned char* arena, std::size_t arena_size, bool full)
|
|
{
|
|
struct frame
|
|
{
|
|
std::size_t end;
|
|
std::uint32_t len;
|
|
std::uint32_t seen;
|
|
bool object;
|
|
bool expect_key;
|
|
};
|
|
std::vector<frame> stack;
|
|
std::vector<byte_range> source_strings;
|
|
std::vector<byte_range> decoded_strings;
|
|
std::vector<float_range> floats;
|
|
const auto check_string = [&](const node & n) -> bool
|
|
{
|
|
if ((n.flags & ~node_flags::escaped) != 0 || n.extra != 0)
|
|
{
|
|
return false;
|
|
}
|
|
const bool decoded = (n.flags & node_flags::escaped) != 0;
|
|
const std::size_t limit = decoded ? arena_size : text_size;
|
|
if (n.off > limit || n.len > limit - n.off)
|
|
{
|
|
return false;
|
|
}
|
|
if (full && n.len != 0)
|
|
{
|
|
(decoded ? decoded_strings : source_strings).push_back(byte_range{n.off, n.len});
|
|
}
|
|
return true;
|
|
};
|
|
// bounds of a number token; the recorded digit layout must lie within it
|
|
const auto number_in_bounds = [&](const node & n) -> bool
|
|
{
|
|
const std::size_t len = number_length(n);
|
|
if (len == 0 || n.off > text_size || len > text_size - n.off)
|
|
{
|
|
return false;
|
|
}
|
|
if (n.kind != static_cast<std::uint8_t>(value_t::number_float))
|
|
{
|
|
return (n.extra >> 8u) == 0;
|
|
}
|
|
// float_value() reads the sign, the integer digits, and the point and
|
|
// fraction digits the layout records (a layout of more than 19 digits
|
|
// means the general conversion, which stays within the token)
|
|
const std::size_t int_digits = n.extra & 0xFFu;
|
|
const std::size_t frac_digits = n.extra >> 8u;
|
|
const std::size_t need = (text[n.off] == '-' ? 1u : 0u) + int_digits + (frac_digits != 0 ? frac_digits + 1 : 0);
|
|
return int_digits + frac_digits > 19 || need <= len;
|
|
};
|
|
std::size_t i = 0;
|
|
for (;;)
|
|
{
|
|
// close finished arrays and objects
|
|
while (!stack.empty() && i == stack.back().end)
|
|
{
|
|
const frame f = stack.back();
|
|
if (f.seen != f.len || (f.object && !f.expect_key))
|
|
{
|
|
return false;
|
|
}
|
|
stack.pop_back();
|
|
if (!stack.empty())
|
|
{
|
|
++stack.back().seen;
|
|
stack.back().expect_key = true;
|
|
}
|
|
}
|
|
if (i == count)
|
|
{
|
|
return stack.empty() && (!full || check_contents(text, arena, source_strings, decoded_strings, floats));
|
|
}
|
|
if (i != 0 && stack.empty())
|
|
{
|
|
return false; // nodes after the root
|
|
}
|
|
const node& n = nodes[i];
|
|
if (!stack.empty() && stack.back().object && stack.back().expect_key)
|
|
{
|
|
if (n.kind != static_cast<std::uint8_t>(value_t::string) || !check_string(n))
|
|
{
|
|
return false;
|
|
}
|
|
stack.back().expect_key = false;
|
|
++i;
|
|
continue;
|
|
}
|
|
bool complete = true;
|
|
switch (static_cast<value_t>(n.kind))
|
|
{
|
|
case value_t::null:
|
|
// (the offset of a literal is read to size the output of dump())
|
|
if (n.flags != 0 || n.extra != 0 || n.off > text_size)
|
|
{
|
|
return false;
|
|
}
|
|
break;
|
|
case value_t::boolean:
|
|
if ((n.flags & ~node_flags::is_true) != 0 || n.extra != 0 || n.off > text_size)
|
|
{
|
|
return false;
|
|
}
|
|
break;
|
|
case value_t::string:
|
|
if (!check_string(n))
|
|
{
|
|
return false;
|
|
}
|
|
break;
|
|
case value_t::number_integer:
|
|
case value_t::number_unsigned:
|
|
case value_t::number_float:
|
|
if (n.flags != 0 || !number_in_bounds(n))
|
|
{
|
|
return false;
|
|
}
|
|
if (full)
|
|
{
|
|
// (the contents of float tokens are checked later; the token of an
|
|
// integer has at most 256 characters)
|
|
if (n.kind == static_cast<std::uint8_t>(value_t::number_float))
|
|
{
|
|
floats.push_back(float_range{n.off, n.len, n.extra});
|
|
}
|
|
else if (!check_integer(n, text))
|
|
{
|
|
return false;
|
|
}
|
|
}
|
|
break;
|
|
case value_t::array:
|
|
case value_t::object:
|
|
{
|
|
const std::size_t limit = stack.empty() ? count : stack.back().end;
|
|
if (n.flags != 0 || n.extra != 0 || n.next == 0 || n.next > limit - i || n.off > text_size)
|
|
{
|
|
return false;
|
|
}
|
|
stack.push_back(frame{i + n.next, n.len, 0, n.kind == static_cast<std::uint8_t>(value_t::object), true});
|
|
complete = false;
|
|
break;
|
|
}
|
|
case value_t::binary:
|
|
case value_t::discarded:
|
|
default:
|
|
return false;
|
|
}
|
|
++i;
|
|
if (complete && !stack.empty())
|
|
{
|
|
++stack.back().seen;
|
|
stack.back().expect_key = true;
|
|
}
|
|
}
|
|
}
|
|
|
|
[[noreturn]] NLOHMANN_VIEW_NOINLINE inline void throw_invalid_image(const char* what)
|
|
{
|
|
throw_parse_error(exception_id::invalid_document_image, concat("invalid json_document image: ", what));
|
|
}
|
|
|
|
/// Read an image into d. The text and the decoded strings stay in the image;
|
|
/// the nodes are copied (so that they are aligned, and edits can change them).
|
|
inline void load_image(document_data& d, const std::uint8_t* image, std::size_t size, image_check check)
|
|
{
|
|
#if !NLOHMANN_VIEW_LITTLE_ENDIAN
|
|
throw_type_error(exception_id::document_image_unsupported, "json_document images need a little-endian target"); // LCOV_EXCL_LINE
|
|
#endif
|
|
if (image == nullptr || size < sizeof(image_header))
|
|
{
|
|
throw_invalid_image("too short");
|
|
}
|
|
image_header h{};
|
|
std::memcpy(&h, image, sizeof(h));
|
|
// (the reserved fields are for later versions)
|
|
if (std::memcmp(h.magic.data(), "NJVI", 4) != 0 || h.version != image_version
|
|
|| (h.reserved[0] | h.reserved[1] | h.reserved[2] | h.reserved[3]) != 0)
|
|
{
|
|
throw_invalid_image("unknown format");
|
|
}
|
|
const std::size_t room = size - sizeof(h);
|
|
if (h.node_count == 0 || h.node_count > room / sizeof(node) || h.node_count >= image_limit || h.text_size >= image_limit || h.arena_size >= image_limit)
|
|
{
|
|
throw_invalid_image("sizes out of range");
|
|
}
|
|
const auto count = static_cast<std::size_t>(h.node_count);
|
|
const auto text_size = static_cast<std::size_t>(h.text_size);
|
|
const auto arena_size = static_cast<std::size_t>(h.arena_size);
|
|
const std::size_t text_at = sizeof(h) + (count * sizeof(node));
|
|
// the text, a NUL, the decoded strings, a NUL, and nothing after them
|
|
if (size - text_at < 2 || text_size > size - text_at - 2 || arena_size != size - text_at - text_size - 2
|
|
|| image[text_at + text_size] != 0 || image[size - 1] != 0)
|
|
{
|
|
throw_invalid_image("sizes out of range");
|
|
}
|
|
|
|
d.discarded = true;
|
|
d.edits.reset();
|
|
d.base[2] = nullptr;
|
|
d.owned.clear();
|
|
if (d.owned_image.empty() || image != d.owned_image.data())
|
|
{
|
|
d.owned_image.clear();
|
|
}
|
|
d.arena.clear();
|
|
d.indexes.clear();
|
|
d.index_slots.clear();
|
|
d.large_objects.clear();
|
|
d.node_count = 0;
|
|
d.reserve(count);
|
|
std::memcpy(d.nodes, image + sizeof(h), count * sizeof(node));
|
|
d.node_count = count;
|
|
const std::uint8_t* const text = image + text_at;
|
|
const std::uint8_t* const arena = text + text_size + 1;
|
|
d.src = reinterpret_cast<const char*>(text); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
|
|
d.size = text_size;
|
|
d.base[0] = d.src;
|
|
d.base[1] = reinterpret_cast<const char*>(arena); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
|
|
d.arena_size = arena_size;
|
|
if (check != image_check::none && !check_image(d.nodes, count, text, text_size, arena, arena_size, check == image_check::full))
|
|
{
|
|
throw_invalid_image("the check failed");
|
|
}
|
|
// the hash indexes of large objects, as after parsing
|
|
for (std::size_t i = 0; i < count; ++i)
|
|
{
|
|
const node& n = d.nodes[i];
|
|
if (n.kind == static_cast<std::uint8_t>(value_t::object) && n.len >= document_data::index_min_members)
|
|
{
|
|
d.large_objects.push_back(static_cast<std::uint32_t>(i));
|
|
}
|
|
}
|
|
build_object_indexes(d);
|
|
std::vector<std::uint32_t>().swap(d.large_objects); // (only needed while building)
|
|
d.discarded = false;
|
|
}
|
|
|
|
} // namespace view
|
|
} // namespace detail
|
|
NLOHMANN_JSON_NAMESPACE_END
|