Compare commits

..
Author SHA1 Message Date
Niels Lohmann 0395ad0992 Pass the complete text of a UBJSON/BJData high-precision number to SAX
For a high-precision number that is a floating-point number, the
UBJSON/BJData reader passed lexer::get_token_string() as the source
text to SAX number_float(). That helper is meant for error messages and
shows a token of more than 8192 characters only by its first and last
4096 characters around "...", so a SAX handler received a shortened
string for a long number.

Pass the number text the lexer collected while scanning the number
(lexer::get_string()) instead. It is complete and, as before, holds
only the number, without any whitespace or byte order mark around it
in the payload. Error messages still use get_token_string().

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-11-16 08:27:07 +01:00
10 changed files with 56 additions and 189 deletions

No files matched your search

@@ -100,8 +100,7 @@ points apply to every edit:
[`root()`](root.md). New values (and the element sequences of an edited array/object) go to storage owned by the [`root()`](root.md). New values (and the element sequences of an edited array/object) go to storage owned by the
document, allocated on demand. document, allocated on demand.
- **A view keeps referring to the same value.** After [`set`](set.md) replaces the value a view refers to, that view - **A view keeps referring to the same value.** After [`set`](set.md) replaces the value a view refers to, that view
sees the new value; a view of a value that a later edit replaces or drops keeps showing what it last held. Such a sees the new value; a view of a value that a later edit replaces or drops keeps showing what it last held. An edit
view can still be edited: the edit changes only that detached value, never anything reachable from `root()`. An edit
of an array or object, however, **invalidates the iterators taken over it** (its members may now live in a of an array or object, however, **invalidates the iterators taken over it** (its members may now live in a
different sequence), and a string obtained with [`get_string()`](../basic_json_view/get_string.md) stays valid even different sequence), and a string obtained with [`get_string()`](../basic_json_view/get_string.md) stays valid even
as further edits happen (earlier buffers of edited text are kept alive, not overwritten). as further edits happen (earlier buffers of edited text are kept alive, not overwritten).
@@ -164,9 +164,7 @@ general.
The same switch happens, for the same reason, when overload 1. replaces a multi-node array/object value with a The same switch happens, for the same reason, when overload 1. replaces a multi-node array/object value with a
scalar: the *parent's* element sequence is what has to switch to links, not `target` itself, because the parent scalar: the *parent's* element sequence is what has to switch to links, not `target` itself, because the parent
originally stepped over `target`'s whole subtree by its node count, which no longer applies once `target` is a originally stepped over `target`'s whole subtree by its node count, which no longer applies once `target` is a
one-node scalar. This also holds when `target` is no longer part of the document (an edit erased or replaced it, or one-node scalar.
a value that contains it): if the container that `target` was in still uses its parsed layout, it switches to links
(otherwise nothing steps over `target` any more), and only the dropped values change.
## Examples ## Examples
@@ -4096,6 +4096,10 @@ class binary_reader
auto number_lexer = detail::lexer<BasicJsonType, ia_type>(detail::input_adapter(number_vector), false); auto number_lexer = detail::lexer<BasicJsonType, ia_type>(detail::input_adapter(number_vector), false);
const auto result_number = number_lexer.scan(); const auto result_number = number_lexer.scan();
const auto number_string = number_lexer.get_token_string(); const auto number_string = number_lexer.get_token_string();
// the number as the lexer read it, for SAX number_float(): unlike
// get_token_string(), which is meant for error messages and shortens
// long tokens, it is complete; the next scan() replaces it
const string_t number_text = number_lexer.get_string();
const auto result_remainder = number_lexer.scan(); const auto result_remainder = number_lexer.scan();
using token_type = typename detail::lexer_base<BasicJsonType>::token_type; using token_type = typename detail::lexer_base<BasicJsonType>::token_type;
@@ -4122,10 +4126,7 @@ class binary_reader
number_string, number_string,
out_of_range::create(exception_id::number_overflow, concat("number overflow parsing '", number_string, '\''), nullptr)); out_of_range::create(exception_id::number_overflow, concat("number overflow parsing '", number_string, '\''), nullptr));
} }
// number_string is a std::string, while the SAX interface takes a return sax->number_float(parsed_float, number_text);
// string_t; convert explicitly, as the two are only implicitly
// convertible for some string types
return sax->number_float(parsed_float, string_t(number_string.data(), number_string.size()));
} }
case token_type::uninitialized: case token_type::uninitialized:
case token_type::literal_true: case token_type::literal_true:
+2 -15
View File
@@ -384,22 +384,9 @@ class editor
// for the container is linear in the size of the document, so // for the container is linear in the size of the document, so
// links (which are marked in the slot) avoid it. // links (which are marked in the slot) avoid it.
node* const p = parent_known ? parent : find_parent(m_doc, slot); node* const p = parent_known ? parent : find_parent(m_doc, slot);
if (p != nullptr) if (p != nullptr && ((p->flags & node_flags::moved) == 0 || moved_capacity(m_doc, p) == 0))
{ {
if ((p->flags & node_flags::moved) == 0 || moved_capacity(m_doc, p) == 0) block_of(m_doc, p, 0);
{
block_of(m_doc, p, 0);
}
}
else
{
// the slot is no longer part of the document, but the
// (dropped) container it is in may still step over it
node* const q = find_layout_parent(m_doc, slot);
if (q != nullptr)
{
block_of(m_doc, q, 0);
}
} }
} }
*slot = e.scalar; *slot = e.scalar;
+1 -38
View File
@@ -194,9 +194,7 @@ inline node* block_of(document_data& d, node* n, std::size_t extra)
h->len = n->len; h->len = n->len;
h->next = static_cast<std::uint32_t>(used); h->next = static_cast<std::uint32_t>(used);
node* o = h + 1; node* o = h + 1;
// (the bound on o keeps the copy inside the block even if the elements for (const node* c = d.first_child_edited(n), *e = d.child_end_edited(n); c != e;)
// disagreed with the element count)
for (const node* c = d.first_child_edited(n), *e = d.child_end_edited(n), *const full = h + used; c != e && o != full;)
{ {
if (object) if (object)
{ {
@@ -278,41 +276,6 @@ inline node* find_parent(const document_data& d, const node* target)
} }
} }
/// The container whose contiguous layout (in the parsed index, or in a new
/// value) steps over `target`, also when an edit dropped the container or a
/// value that encloses it; nullptr if there is none, or if the container no
/// longer walks that layout (its elements moved). An edit never gives a node a
/// larger extent than it was parsed or encoded with (a scalar is one node, an
/// assigned array/object keeps the extent of the slot), so the containers whose
/// extent covers `target` are its original ancestors: the nearest one before it
/// is its original parent. The root of a new value that is the element sequence
/// of another container is walked by that container (which find_parent() finds
/// while it still uses the sequence), not by itself.
inline node* find_layout_parent(const document_data& d, const node* target)
{
const std::less<const node*> lt{};
const node* lo = d.nodes;
if (lt(target, lo) || !lt(target, d.nodes + d.node_count))
{
auto it = d.edits->regions.upper_bound(target);
if (it == d.edits->regions.begin())
{
return nullptr; // LCOV_EXCL_LINE (an array/object with elements is in the index or a new value)
}
--it;
lo = it->second != nullptr ? it->first + 1 : it->first;
}
for (const node* c = target; c != lo;)
{
--c;
if (is_container(*c) && lt(target, c + c->next))
{
return (c->flags & node_flags::moved) == 0 ? const_cast<node*>(c) : nullptr; // NOLINT(cppcoreguidelines-pro-type-const-cast): the nodes belong to the document
}
}
return nullptr;
}
} // namespace view } // namespace view
} // namespace detail } // namespace detail
NLOHMANN_JSON_NAMESPACE_END NLOHMANN_JSON_NAMESPACE_END
+5 -4
View File
@@ -22774,6 +22774,10 @@ class binary_reader
auto number_lexer = detail::lexer<BasicJsonType, ia_type>(detail::input_adapter(number_vector), false); auto number_lexer = detail::lexer<BasicJsonType, ia_type>(detail::input_adapter(number_vector), false);
const auto result_number = number_lexer.scan(); const auto result_number = number_lexer.scan();
const auto number_string = number_lexer.get_token_string(); const auto number_string = number_lexer.get_token_string();
// the number as the lexer read it, for SAX number_float(): unlike
// get_token_string(), which is meant for error messages and shortens
// long tokens, it is complete; the next scan() replaces it
const string_t number_text = number_lexer.get_string();
const auto result_remainder = number_lexer.scan(); const auto result_remainder = number_lexer.scan();
using token_type = typename detail::lexer_base<BasicJsonType>::token_type; using token_type = typename detail::lexer_base<BasicJsonType>::token_type;
@@ -22800,10 +22804,7 @@ class binary_reader
number_string, number_string,
out_of_range::create(exception_id::number_overflow, concat("number overflow parsing '", number_string, '\''), nullptr)); out_of_range::create(exception_id::number_overflow, concat("number overflow parsing '", number_string, '\''), nullptr));
} }
// number_string is a std::string, while the SAX interface takes a return sax->number_float(parsed_float, number_text);
// string_t; convert explicitly, as the two are only implicitly
// convertible for some string types
return sax->number_float(parsed_float, string_t(number_string.data(), number_string.size()));
} }
case token_type::uninitialized: case token_type::uninitialized:
case token_type::literal_true: case token_type::literal_true:
+3 -53
View File
@@ -5564,9 +5564,7 @@ inline node* block_of(document_data& d, node* n, std::size_t extra)
h->len = n->len; h->len = n->len;
h->next = static_cast<std::uint32_t>(used); h->next = static_cast<std::uint32_t>(used);
node* o = h + 1; node* o = h + 1;
// (the bound on o keeps the copy inside the block even if the elements for (const node* c = d.first_child_edited(n), *e = d.child_end_edited(n); c != e;)
// disagreed with the element count)
for (const node* c = d.first_child_edited(n), *e = d.child_end_edited(n), *const full = h + used; c != e && o != full;)
{ {
if (object) if (object)
{ {
@@ -5648,41 +5646,6 @@ inline node* find_parent(const document_data& d, const node* target)
} }
} }
/// The container whose contiguous layout (in the parsed index, or in a new
/// value) steps over `target`, also when an edit dropped the container or a
/// value that encloses it; nullptr if there is none, or if the container no
/// longer walks that layout (its elements moved). An edit never gives a node a
/// larger extent than it was parsed or encoded with (a scalar is one node, an
/// assigned array/object keeps the extent of the slot), so the containers whose
/// extent covers `target` are its original ancestors: the nearest one before it
/// is its original parent. The root of a new value that is the element sequence
/// of another container is walked by that container (which find_parent() finds
/// while it still uses the sequence), not by itself.
inline node* find_layout_parent(const document_data& d, const node* target)
{
const std::less<const node*> lt{};
const node* lo = d.nodes;
if (lt(target, lo) || !lt(target, d.nodes + d.node_count))
{
auto it = d.edits->regions.upper_bound(target);
if (it == d.edits->regions.begin())
{
return nullptr; // LCOV_EXCL_LINE (an array/object with elements is in the index or a new value)
}
--it;
lo = it->second != nullptr ? it->first + 1 : it->first;
}
for (const node* c = target; c != lo;)
{
--c;
if (is_container(*c) && lt(target, c + c->next))
{
return (c->flags & node_flags::moved) == 0 ? const_cast<node*>(c) : nullptr; // NOLINT(cppcoreguidelines-pro-type-const-cast): the nodes belong to the document
}
}
return nullptr;
}
} // namespace view } // namespace view
} // namespace detail } // namespace detail
NLOHMANN_JSON_NAMESPACE_END NLOHMANN_JSON_NAMESPACE_END
@@ -6559,22 +6522,9 @@ class editor
// for the container is linear in the size of the document, so // for the container is linear in the size of the document, so
// links (which are marked in the slot) avoid it. // links (which are marked in the slot) avoid it.
node* const p = parent_known ? parent : find_parent(m_doc, slot); node* const p = parent_known ? parent : find_parent(m_doc, slot);
if (p != nullptr) if (p != nullptr && ((p->flags & node_flags::moved) == 0 || moved_capacity(m_doc, p) == 0))
{ {
if ((p->flags & node_flags::moved) == 0 || moved_capacity(m_doc, p) == 0) block_of(m_doc, p, 0);
{
block_of(m_doc, p, 0);
}
}
else
{
// the slot is no longer part of the document, but the
// (dropped) container it is in may still step over it
node* const q = find_layout_parent(m_doc, slot);
if (q != nullptr)
{
block_of(m_doc, q, 0);
}
} }
} }
*slot = e.scalar; *slot = e.scalar;
+19
View File
@@ -1278,6 +1278,25 @@ TEST_CASE("BJData")
CHECK(j.dump() == "3.141592653589793"); CHECK(j.dump() == "3.141592653589793");
} }
SECTION("long floating-point number")
{
// the SAX parser receives the number text as it is in the
// input, also when it is longer than the part of a token
// that error messages show
const std::vector<std::size_t> lengths = {8192, 8193, 10000};
for (const auto length : lengths)
{
CAPTURE(length)
const std::string number = "1." + std::string(length - 2, '3');
std::vector<std::uint8_t> vec = {'H', 'I', static_cast<std::uint8_t>(length & 0xFF), static_cast<std::uint8_t>(length >> 8)};
vec.insert(vec.end(), number.begin(), number.end());
SaxEventLogger logger;
CHECK(json::sax_parse(vec, &logger, json::input_format_t::bjdata));
CHECK(logger.events == std::vector<std::string>({"number_float(" + number + ")"}));
CHECK(json::from_bjdata(vec).is_number_float());
}
}
SECTION("errors") SECTION("errors")
{ {
// error while parsing length // error while parsing length
-70
View File
@@ -589,76 +589,6 @@ TEST_CASE("json_view edits: a container that was added is replaced by a scalar")
CHECK(d.root().materialize() == json::parse(R"({"a":[1,6,{"k":"s"}]})")); CHECK(d.root().materialize() == json::parse(R"({"a":[1,6,{"k":"s"}]})"));
} }
TEST_CASE("json_view edits: a scalar replaces an array or object that is no longer part of the document")
{
// the dropped container that the array/object is in still steps over its
// parsed extent: that container switches to links first, as a live one does
SECTION("members of an erased object")
{
json_editable_document d = json_editable_document::parse(R"({"p": {"a": [1, [2]], "b": 3, "e": []}, "q": 0})");
const json_editable_view p = d.root()["p"];
const json_editable_view a = p["a"];
d.erase(d.root(), "p");
d.set(a[1], 5);
CHECK(p.dump() == R"({"a":[1,5],"b":3,"e":[]})");
d.set(a, 5);
CHECK(p.size() == 3);
CHECK(p.dump() == R"({"a":5,"b":3,"e":[]})");
d.push_back(p["e"], 6);
CHECK(p.dump() == R"({"a":5,"b":3,"e":[6]})");
CHECK(d.root().dump() == R"({"q":0})");
}
SECTION("an array whose elements moved")
{
json_editable_document d = json_editable_document::parse(R"({"p": {"a": [1, 2], "b": 3}, "q": 0})");
const json_editable_view p = d.root()["p"];
const json_editable_view a = p["a"];
d.erase(d.root(), "p");
d.push_back(a, 4); // (the extent of a stays)
d.set(a, 5);
CHECK(p.dump() == R"({"a":5,"b":3})");
CHECK(d.root().dump() == R"({"q":0})");
}
SECTION("an element of an erased array, which grows afterwards")
{
json_editable_document d = json_editable_document::parse(R"({"p": [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], "q": 0})");
const json_editable_view p = d.root()["p"];
d.erase(d.root(), "p");
d.set(p[0], 5);
d.push_back(p, 1);
CHECK(p.dump() == "[5,1]");
CHECK(d.root().dump() == R"({"q":0})");
}
SECTION("values in an erased new value")
{
json_editable_document d = json_editable_document::parse("[0]");
const json_editable_view x = d.push_back(d.root(), json::parse("[[[1, 2]], 3]"));
const json_editable_view y = x[0];
d.erase(d.root(), 1);
d.set(y[0], 5);
CHECK(x.dump() == "[[5],3]");
d.set(y, 5);
CHECK(x.dump() == "[5,3]");
CHECK(d.root().dump() == "[0]");
}
SECTION("values in the elements that an array had before it was replaced")
{
json_editable_document d = json_editable_document::parse("[0]");
d.set(d.root()[0], json::parse("[[[1, 2]]]"));
const json_editable_view y = d.root()[0][0];
d.set(d.root()[0], json::array({3}));
d.set(y[0], 5);
CHECK(y.dump() == "[5]");
d.set(y, 5);
CHECK(y.dump() == "5");
CHECK(d.root().dump() == "[[3]]");
}
}
#if !defined(JSON_NOEXCEPTION) #if !defined(JSON_NOEXCEPTION)
TEST_CASE("json_view edits: the size of a text arena") TEST_CASE("json_view edits: the size of a text arena")
{ {
+19
View File
@@ -793,6 +793,25 @@ TEST_CASE("UBJSON")
CHECK(j.dump() == "3.141592653589793"); CHECK(j.dump() == "3.141592653589793");
} }
SECTION("long floating-point number")
{
// the SAX parser receives the number text as it is in the
// input, also when it is longer than the part of a token
// that error messages show
const std::vector<std::size_t> lengths = {8192, 8193, 10000};
for (const auto length : lengths)
{
CAPTURE(length)
const std::string number = "1." + std::string(length - 2, '3');
std::vector<std::uint8_t> vec = {'H', 'I', static_cast<std::uint8_t>(length >> 8), static_cast<std::uint8_t>(length & 0xFF)};
vec.insert(vec.end(), number.begin(), number.end());
SaxEventLogger logger;
CHECK(json::sax_parse(vec, &logger, json::input_format_t::ubjson));
CHECK(logger.events == std::vector<std::string>({"number_float(" + number + ")"}));
CHECK(json::from_ubjson(vec).is_number_float());
}
}
SECTION("errors") SECTION("errors")
{ {
// error while parsing length // error while parsing length