Compare commits

..
Author SHA1 Message Date
Niels Lohmann d490ad8541 Fix a scalar set() through a view of a value that an edit dropped
set(target, scalar) on an array/object that spans more than one node first
switches the container that target is in to links, since that container
steps over target's parsed subtree by its node count. The container was
looked up from root(); for a target that is no longer part of the document
(an edit erased or replaced it, or a value that contains it) the lookup
found nothing and the scalar was written anyway. The dropped container then
read past its nodes when walked, and a later edit of it made block_of()
write more links than the block it allocated holds.

When the lookup from root() fails, look for the container whose contiguous
layout still steps over the slot: the nearest container before it in its
allocation whose extent covers it (edits never enlarge an extent, so this
is the original parent). If that container still uses its layout, it
switches to links as a live parent does. Every other edit through a view of
a dropped value already worked and changed only the detached value; the
scalar set now does the same, as documented.

Also stop block_of() when its block is full, so that it never writes past
the capacity it allocated.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-11-16 07:41:42 +01:00
10 changed files with 186 additions and 224 deletions

No files matched your search

@@ -100,7 +100,8 @@ points apply to every edit:
[`root()`](root.md). New values (and the element sequences of an edited array/object) go to storage owned by the [`root()`](root.md). New values (and the element sequences of an edited array/object) go to storage owned by the
document, allocated on demand. document, allocated on demand.
- **A view keeps referring to the same value.** After [`set`](set.md) replaces the value a view refers to, that view - **A view keeps referring to the same value.** After [`set`](set.md) replaces the value a view refers to, that view
sees the new value; a view of a value that a later edit replaces or drops keeps showing what it last held. An edit sees the new value; a view of a value that a later edit replaces or drops keeps showing what it last held. Such a
view can still be edited: the edit changes only that detached value, never anything reachable from `root()`. An edit
of an array or object, however, **invalidates the iterators taken over it** (its members may now live in a of an array or object, however, **invalidates the iterators taken over it** (its members may now live in a
different sequence), and a string obtained with [`get_string()`](../basic_json_view/get_string.md) stays valid even different sequence), and a string obtained with [`get_string()`](../basic_json_view/get_string.md) stays valid even
as further edits happen (earlier buffers of edited text are kept alive, not overwritten). as further edits happen (earlier buffers of edited text are kept alive, not overwritten).
@@ -164,7 +164,9 @@ general.
The same switch happens, for the same reason, when overload 1. replaces a multi-node array/object value with a The same switch happens, for the same reason, when overload 1. replaces a multi-node array/object value with a
scalar: the *parent's* element sequence is what has to switch to links, not `target` itself, because the parent scalar: the *parent's* element sequence is what has to switch to links, not `target` itself, because the parent
originally stepped over `target`'s whole subtree by its node count, which no longer applies once `target` is a originally stepped over `target`'s whole subtree by its node count, which no longer applies once `target` is a
one-node scalar. one-node scalar. This also holds when `target` is no longer part of the document (an edit erased or replaced it, or
a value that contains it): if the container that `target` was in still uses its parsed layout, it switches to links
(otherwise nothing steps over `target` any more), and only the dropped values change.
## Examples ## Examples
+1 -1
View File
@@ -140,7 +140,7 @@ Exceptions have ids 1xx.
This error indicates a syntax error while deserializing a JSON text. The error message describes that an unexpected token (character) was encountered, and the member `byte` indicates the error position. This error indicates a syntax error while deserializing a JSON text. The error message describes that an unexpected token (character) was encountered, and the member `byte` indicates the error position.
The message ends with the last read token (`last read: '...'`), with control characters written as `<U+XXXX>`. A token of more than 8192 bytes is shown by its first and its last 4096 bytes, separated by `...`; each part is shortened by up to three bytes so that it does not end or begin within a UTF-8 character. The message ends with the last read token (`last read: '...'`), with control characters written as `<U+XXXX>`. A token of more than 8192 characters is shown by its first and its last 4096 characters, separated by `...`.
!!! failure "Example message" !!! failure "Example message"
+2 -56
View File
@@ -2279,14 +2279,9 @@ scan_number_done:
std::string result; std::string result;
if (chars.size() > 2 * shown) if (chars.size() > 2 * shown)
{ {
// the lexer reads bytes (wide characters arrive as UTF-8), so append_escaped(result, chars.begin(), chars.begin() + static_cast<std::ptrdiff_t>(shown));
// move both cuts to character boundaries to keep the message
// valid UTF-8; this only shortens the two parts
const auto head_end = chars.begin() + static_cast<std::ptrdiff_t>(length_without_partial_utf8_at_end(chars.data(), shown));
const auto tail_begin = chars.end() - static_cast<std::ptrdiff_t>(shown);
append_escaped(result, chars.begin(), head_end);
result += "..."; result += "...";
append_escaped(result, tail_begin + static_cast<std::ptrdiff_t>(partial_utf8_length_at_start(&*tail_begin)), chars.end()); append_escaped(result, chars.end() - static_cast<std::ptrdiff_t>(shown), chars.end());
} }
else else
{ {
@@ -2296,55 +2291,6 @@ scan_number_done:
} }
private: private:
/// whether @a c is a UTF-8 continuation byte
static bool is_utf8_continuation(const char_type c) noexcept
{
return (static_cast<unsigned char>(c) & 0xC0u) == 0x80u;
}
/*!
@brief the length of the first @a n characters at @a data without a UTF-8
character that begins there but does not end there
Looks only at the last 4 of these characters: in valid UTF-8, at most 3
bytes of a character can be before the cut.
*/
static std::size_t length_without_partial_utf8_at_end(const char_type* data, const std::size_t n) noexcept
{
std::size_t lead = n - 1;
while (lead > n - 4 && is_utf8_continuation(data[lead]))
{
--lead;
}
const auto c = static_cast<unsigned char>(data[lead]);
std::size_t length = 1;
if (c >= 0xF0u)
{
length = 4;
}
else if (c >= 0xE0u)
{
length = 3;
}
else if (c >= 0xC0u)
{
length = 2;
}
return lead + length > n ? lead : n;
}
/// the number of UTF-8 continuation bytes (at most 3) at the start of @a data, which
/// belong to a character that begins before it
static std::size_t partial_utf8_length_at_start(const char_type* data) noexcept
{
std::size_t skip = 0;
while (skip < 3 && is_utf8_continuation(data[skip]))
{
++skip;
}
return skip;
}
/// append the characters [first, last) to @a result, escaping control characters /// append the characters [first, last) to @a result, escaping control characters
template<typename Iterator> template<typename Iterator>
static void append_escaped(std::string& result, Iterator first, Iterator last) static void append_escaped(std::string& result, Iterator first, Iterator last)
+15 -2
View File
@@ -384,9 +384,22 @@ class editor
// for the container is linear in the size of the document, so // for the container is linear in the size of the document, so
// links (which are marked in the slot) avoid it. // links (which are marked in the slot) avoid it.
node* const p = parent_known ? parent : find_parent(m_doc, slot); node* const p = parent_known ? parent : find_parent(m_doc, slot);
if (p != nullptr && ((p->flags & node_flags::moved) == 0 || moved_capacity(m_doc, p) == 0)) if (p != nullptr)
{ {
block_of(m_doc, p, 0); if ((p->flags & node_flags::moved) == 0 || moved_capacity(m_doc, p) == 0)
{
block_of(m_doc, p, 0);
}
}
else
{
// the slot is no longer part of the document, but the
// (dropped) container it is in may still step over it
node* const q = find_layout_parent(m_doc, slot);
if (q != nullptr)
{
block_of(m_doc, q, 0);
}
} }
} }
*slot = e.scalar; *slot = e.scalar;
+38 -1
View File
@@ -194,7 +194,9 @@ inline node* block_of(document_data& d, node* n, std::size_t extra)
h->len = n->len; h->len = n->len;
h->next = static_cast<std::uint32_t>(used); h->next = static_cast<std::uint32_t>(used);
node* o = h + 1; node* o = h + 1;
for (const node* c = d.first_child_edited(n), *e = d.child_end_edited(n); c != e;) // (the bound on o keeps the copy inside the block even if the elements
// disagreed with the element count)
for (const node* c = d.first_child_edited(n), *e = d.child_end_edited(n), *const full = h + used; c != e && o != full;)
{ {
if (object) if (object)
{ {
@@ -276,6 +278,41 @@ inline node* find_parent(const document_data& d, const node* target)
} }
} }
/// The container whose contiguous layout (in the parsed index, or in a new
/// value) steps over `target`, also when an edit dropped the container or a
/// value that encloses it; nullptr if there is none, or if the container no
/// longer walks that layout (its elements moved). An edit never gives a node a
/// larger extent than it was parsed or encoded with (a scalar is one node, an
/// assigned array/object keeps the extent of the slot), so the containers whose
/// extent covers `target` are its original ancestors: the nearest one before it
/// is its original parent. The root of a new value that is the element sequence
/// of another container is walked by that container (which find_parent() finds
/// while it still uses the sequence), not by itself.
inline node* find_layout_parent(const document_data& d, const node* target)
{
const std::less<const node*> lt{};
const node* lo = d.nodes;
if (lt(target, lo) || !lt(target, d.nodes + d.node_count))
{
auto it = d.edits->regions.upper_bound(target);
if (it == d.edits->regions.begin())
{
return nullptr; // LCOV_EXCL_LINE (an array/object with elements is in the index or a new value)
}
--it;
lo = it->second != nullptr ? it->first + 1 : it->first;
}
for (const node* c = target; c != lo;)
{
--c;
if (is_container(*c) && lt(target, c + c->next))
{
return (c->flags & node_flags::moved) == 0 ? const_cast<node*>(c) : nullptr; // NOLINT(cppcoreguidelines-pro-type-const-cast): the nodes belong to the document
}
}
return nullptr;
}
} // namespace view } // namespace view
} // namespace detail } // namespace detail
NLOHMANN_JSON_NAMESPACE_END NLOHMANN_JSON_NAMESPACE_END
+2 -56
View File
@@ -15782,14 +15782,9 @@ scan_number_done:
std::string result; std::string result;
if (chars.size() > 2 * shown) if (chars.size() > 2 * shown)
{ {
// the lexer reads bytes (wide characters arrive as UTF-8), so append_escaped(result, chars.begin(), chars.begin() + static_cast<std::ptrdiff_t>(shown));
// move both cuts to character boundaries to keep the message
// valid UTF-8; this only shortens the two parts
const auto head_end = chars.begin() + static_cast<std::ptrdiff_t>(length_without_partial_utf8_at_end(chars.data(), shown));
const auto tail_begin = chars.end() - static_cast<std::ptrdiff_t>(shown);
append_escaped(result, chars.begin(), head_end);
result += "..."; result += "...";
append_escaped(result, tail_begin + static_cast<std::ptrdiff_t>(partial_utf8_length_at_start(&*tail_begin)), chars.end()); append_escaped(result, chars.end() - static_cast<std::ptrdiff_t>(shown), chars.end());
} }
else else
{ {
@@ -15799,55 +15794,6 @@ scan_number_done:
} }
private: private:
/// whether @a c is a UTF-8 continuation byte
static bool is_utf8_continuation(const char_type c) noexcept
{
return (static_cast<unsigned char>(c) & 0xC0u) == 0x80u;
}
/*!
@brief the length of the first @a n characters at @a data without a UTF-8
character that begins there but does not end there
Looks only at the last 4 of these characters: in valid UTF-8, at most 3
bytes of a character can be before the cut.
*/
static std::size_t length_without_partial_utf8_at_end(const char_type* data, const std::size_t n) noexcept
{
std::size_t lead = n - 1;
while (lead > n - 4 && is_utf8_continuation(data[lead]))
{
--lead;
}
const auto c = static_cast<unsigned char>(data[lead]);
std::size_t length = 1;
if (c >= 0xF0u)
{
length = 4;
}
else if (c >= 0xE0u)
{
length = 3;
}
else if (c >= 0xC0u)
{
length = 2;
}
return lead + length > n ? lead : n;
}
/// the number of UTF-8 continuation bytes (at most 3) at the start of @a data, which
/// belong to a character that begins before it
static std::size_t partial_utf8_length_at_start(const char_type* data) noexcept
{
std::size_t skip = 0;
while (skip < 3 && is_utf8_continuation(data[skip]))
{
++skip;
}
return skip;
}
/// append the characters [first, last) to @a result, escaping control characters /// append the characters [first, last) to @a result, escaping control characters
template<typename Iterator> template<typename Iterator>
static void append_escaped(std::string& result, Iterator first, Iterator last) static void append_escaped(std::string& result, Iterator first, Iterator last)
+53 -3
View File
@@ -5564,7 +5564,9 @@ inline node* block_of(document_data& d, node* n, std::size_t extra)
h->len = n->len; h->len = n->len;
h->next = static_cast<std::uint32_t>(used); h->next = static_cast<std::uint32_t>(used);
node* o = h + 1; node* o = h + 1;
for (const node* c = d.first_child_edited(n), *e = d.child_end_edited(n); c != e;) // (the bound on o keeps the copy inside the block even if the elements
// disagreed with the element count)
for (const node* c = d.first_child_edited(n), *e = d.child_end_edited(n), *const full = h + used; c != e && o != full;)
{ {
if (object) if (object)
{ {
@@ -5646,6 +5648,41 @@ inline node* find_parent(const document_data& d, const node* target)
} }
} }
/// The container whose contiguous layout (in the parsed index, or in a new
/// value) steps over `target`, also when an edit dropped the container or a
/// value that encloses it; nullptr if there is none, or if the container no
/// longer walks that layout (its elements moved). An edit never gives a node a
/// larger extent than it was parsed or encoded with (a scalar is one node, an
/// assigned array/object keeps the extent of the slot), so the containers whose
/// extent covers `target` are its original ancestors: the nearest one before it
/// is its original parent. The root of a new value that is the element sequence
/// of another container is walked by that container (which find_parent() finds
/// while it still uses the sequence), not by itself.
inline node* find_layout_parent(const document_data& d, const node* target)
{
const std::less<const node*> lt{};
const node* lo = d.nodes;
if (lt(target, lo) || !lt(target, d.nodes + d.node_count))
{
auto it = d.edits->regions.upper_bound(target);
if (it == d.edits->regions.begin())
{
return nullptr; // LCOV_EXCL_LINE (an array/object with elements is in the index or a new value)
}
--it;
lo = it->second != nullptr ? it->first + 1 : it->first;
}
for (const node* c = target; c != lo;)
{
--c;
if (is_container(*c) && lt(target, c + c->next))
{
return (c->flags & node_flags::moved) == 0 ? const_cast<node*>(c) : nullptr; // NOLINT(cppcoreguidelines-pro-type-const-cast): the nodes belong to the document
}
}
return nullptr;
}
} // namespace view } // namespace view
} // namespace detail } // namespace detail
NLOHMANN_JSON_NAMESPACE_END NLOHMANN_JSON_NAMESPACE_END
@@ -6522,9 +6559,22 @@ class editor
// for the container is linear in the size of the document, so // for the container is linear in the size of the document, so
// links (which are marked in the slot) avoid it. // links (which are marked in the slot) avoid it.
node* const p = parent_known ? parent : find_parent(m_doc, slot); node* const p = parent_known ? parent : find_parent(m_doc, slot);
if (p != nullptr && ((p->flags & node_flags::moved) == 0 || moved_capacity(m_doc, p) == 0)) if (p != nullptr)
{ {
block_of(m_doc, p, 0); if ((p->flags & node_flags::moved) == 0 || moved_capacity(m_doc, p) == 0)
{
block_of(m_doc, p, 0);
}
}
else
{
// the slot is no longer part of the document, but the
// (dropped) container it is in may still step over it
node* const q = find_layout_parent(m_doc, slot);
if (q != nullptr)
{
block_of(m_doc, q, 0);
}
} }
} }
*slot = e.scalar; *slot = e.scalar;
-103
View File
@@ -3380,108 +3380,5 @@ TEST_CASE("parse error message of a long token")
CHECK(message(text, kind) == "no error"); CHECK(message(text, kind) == "no error");
} }
} }
SECTION("a long string of multi-byte characters")
{
// the parts of a long token are cut at character boundaries, so the
// message of valid UTF-8 input is valid UTF-8, also for wide strings
const auto last_read = [](const std::string & m) -> std::string
{
const auto pos = m.find("last read: '");
return pos == std::string::npos ? std::string() : m.substr(pos + 12, m.size() - pos - 13);
};
const auto utf8 = [](const char32_t c) -> std::string
{
std::string result;
if (c < 0x800)
{
result.push_back(static_cast<char>(0xC0 | (c >> 6)));
}
else if (c < 0x10000)
{
result.push_back(static_cast<char>(0xE0 | (c >> 12)));
result.push_back(static_cast<char>(0x80 | ((c >> 6) & 0x3F)));
}
else
{
result.push_back(static_cast<char>(0xF0 | (c >> 18)));
result.push_back(static_cast<char>(0x80 | ((c >> 12) & 0x3F)));
result.push_back(static_cast<char>(0x80 | ((c >> 6) & 0x3F)));
}
result.push_back(static_cast<char>(0x80 | (c & 0x3F)));
return result;
};
// characters of 2, 3, and 4 bytes
const std::vector<char32_t> characters = {0xE9, 0x20AC, 0x1F600};
for (const char32_t c : characters)
{
for (std::size_t offset = 0; offset < 4; ++offset)
{
CAPTURE(static_cast<std::uint32_t>(c))
CAPTURE(offset)
// an unterminated string: the token is the whole input
std::string text = "\"" + std::string(offset, 'a');
std::u32string text32 = U"\"" + std::u32string(offset, U'a');
while (text.size() < 9000)
{
text += utf8(c);
text32.push_back(c);
}
std::u16string text16;
for (const char32_t c32 : text32)
{
if (c32 < 0x10000)
{
text16.push_back(static_cast<char16_t>(c32));
}
else
{
text16.push_back(static_cast<char16_t>(0xD800 + ((c32 - 0x10000) >> 10)));
text16.push_back(static_cast<char16_t>(0xDC00 + ((c32 - 0x10000) & 0x3FF)));
}
}
const std::string shown = last_read(message(text, 0));
const auto dots = shown.find("...");
REQUIRE(dots != std::string::npos);
// the first and the last at most 4096 bytes, without parts of characters
CHECK(dots <= 4096);
CHECK(dots > 4092);
CHECK(text.compare(0, dots, shown, 0, dots) == 0);
const std::size_t tail = shown.size() - dots - 3;
CHECK(tail <= 4096);
CHECK(tail > 4092);
CHECK(text.compare(text.size() - tail, tail, shown, dots + 3, tail) == 0);
CHECK_NOTHROW(json(shown).dump());
for (int kind = 1; kind < 4; ++kind)
{
CAPTURE(kind)
CHECK(last_read(message(text, kind)) == shown);
}
json _;
std::string m32;
std::string m16;
try
{
_ = json::parse(text32);
}
catch (const json::parse_error& e)
{
m32 = e.what();
}
try
{
_ = json::parse(text16);
}
catch (const json::parse_error& e)
{
m16 = e.what();
}
CHECK(last_read(m32) == shown);
CHECK(last_read(m16) == shown);
}
}
}
} }
#endif #endif
+70
View File
@@ -589,6 +589,76 @@ TEST_CASE("json_view edits: a container that was added is replaced by a scalar")
CHECK(d.root().materialize() == json::parse(R"({"a":[1,6,{"k":"s"}]})")); CHECK(d.root().materialize() == json::parse(R"({"a":[1,6,{"k":"s"}]})"));
} }
TEST_CASE("json_view edits: a scalar replaces an array or object that is no longer part of the document")
{
// the dropped container that the array/object is in still steps over its
// parsed extent: that container switches to links first, as a live one does
SECTION("members of an erased object")
{
json_editable_document d = json_editable_document::parse(R"({"p": {"a": [1, [2]], "b": 3, "e": []}, "q": 0})");
const json_editable_view p = d.root()["p"];
const json_editable_view a = p["a"];
d.erase(d.root(), "p");
d.set(a[1], 5);
CHECK(p.dump() == R"({"a":[1,5],"b":3,"e":[]})");
d.set(a, 5);
CHECK(p.size() == 3);
CHECK(p.dump() == R"({"a":5,"b":3,"e":[]})");
d.push_back(p["e"], 6);
CHECK(p.dump() == R"({"a":5,"b":3,"e":[6]})");
CHECK(d.root().dump() == R"({"q":0})");
}
SECTION("an array whose elements moved")
{
json_editable_document d = json_editable_document::parse(R"({"p": {"a": [1, 2], "b": 3}, "q": 0})");
const json_editable_view p = d.root()["p"];
const json_editable_view a = p["a"];
d.erase(d.root(), "p");
d.push_back(a, 4); // (the extent of a stays)
d.set(a, 5);
CHECK(p.dump() == R"({"a":5,"b":3})");
CHECK(d.root().dump() == R"({"q":0})");
}
SECTION("an element of an erased array, which grows afterwards")
{
json_editable_document d = json_editable_document::parse(R"({"p": [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], "q": 0})");
const json_editable_view p = d.root()["p"];
d.erase(d.root(), "p");
d.set(p[0], 5);
d.push_back(p, 1);
CHECK(p.dump() == "[5,1]");
CHECK(d.root().dump() == R"({"q":0})");
}
SECTION("values in an erased new value")
{
json_editable_document d = json_editable_document::parse("[0]");
const json_editable_view x = d.push_back(d.root(), json::parse("[[[1, 2]], 3]"));
const json_editable_view y = x[0];
d.erase(d.root(), 1);
d.set(y[0], 5);
CHECK(x.dump() == "[[5],3]");
d.set(y, 5);
CHECK(x.dump() == "[5,3]");
CHECK(d.root().dump() == "[0]");
}
SECTION("values in the elements that an array had before it was replaced")
{
json_editable_document d = json_editable_document::parse("[0]");
d.set(d.root()[0], json::parse("[[[1, 2]]]"));
const json_editable_view y = d.root()[0][0];
d.set(d.root()[0], json::array({3}));
d.set(y[0], 5);
CHECK(y.dump() == "[5]");
d.set(y, 5);
CHECK(y.dump() == "5");
CHECK(d.root().dump() == "[[3]]");
}
}
#if !defined(JSON_NOEXCEPTION) #if !defined(JSON_NOEXCEPTION)
TEST_CASE("json_view edits: the size of a text arena") TEST_CASE("json_view edits: the size of a text arena")
{ {