Compare commits

..
Author SHA1 Message Date
Niels Lohmann 40a14c4ed8 Move the scanned string into the value instead of copying it
The SAX interface documents that the string handed to json_sax::string() may
be moved from, and the DOM handlers already move the one handed to binary().
string() did not, so every string value was copy-constructed out of the
lexer's token buffer, which then kept the buffer alive at its high-water mark
until the next token overwrote it.

Moving hands that buffer to the new value instead. The allocation count is
unchanged - the value needed one either way - but the copy is gone.

  jeopardy      247.3 ms -> 240.9 ms  (-2.6%)
  citm_catalog    4.61 ms ->   4.48 ms (-2.8%)
  40k 30-char strings 7.80 ms -> 7.64 ms (-2.1%)

Note this deliberately does not extend to the object key. Moving the key
hands the lexer's buffer - sized for the largest token seen so far - to a key
that is usually short, so the next value has to grow a fresh buffer. Measured,
that costs 11.9% on a document of many small keys with longer values.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-09-03 07:42:17 +02:00
Niels Lohmann 65d5e66042 Do not search a container for the value the callback rejected
When a parser callback rejects a value, the placeholder stored for it has to
be removed from its parent again. remove_discarded_value() found it by
scanning the parent from the beginning, so filtering a container cost one
scan per rejected member - quadratic in the number of members of a single
container.

A rejected value can only ever be the one most recently added to its parent:
the last element of an array, or the placeholder key() stored under the
current key in an object. Record that key alongside the existing
key_keep_stack, and for a container record it again alongside ref_stack so
end_object()/end_array() can find it in the parent. Removal is then O(1) for
an array and O(log n) for an object, and finding nothing there means nothing
was stored, so there is nothing to remove.

The key for a container is read before handle_value() may consume it, so it
is also correct when the callback rejects the container at its start event
and it never reaches its parent at all.

Discarding half the members of one object, before -> after:

     members     value rejected    container rejected at start
      16 000    392 ms -> 3.7 ms      803 ms ->  8.2 ms
      64 000   6238 ms -> 14.6 ms   12651 ms -> 32.2 ms
     128 000  25339 ms -> 30.6 ms

Results are unchanged: 48 000 randomized documents parsed under 12 different
filtering callbacks - covering duplicate keys, empty keys, rejected keys and
containers rejected at both their start and end events - produce byte
identical output before and after.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-09-03 07:42:16 +02:00
Niels Lohmann d9b8559880 Benchmark parsing of pretty-printed JSON
Every input in the benchmark corpus is minified or only lightly spaced, so
none of them exercise the lexer's whitespace handling. Real-world JSON is
frequently indented - configuration files, pretty-printed API responses,
anything kept under version control - where the insignificant whitespace can
outweigh the data itself.

Add a ParseIndented family that re-serializes each document with an
indentation and parses that. The content is identical to the matching
ParseString row, so the pair isolates the cost of the whitespace alone.
Measured here, parsing the indented form costs 16-34% more than the minified
form of the same document, which nothing in the suite currently reports.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-09-03 07:42:15 +02:00
Niels LohmannandClaude 5c331d1fab Take the output adapter by reference at the serializer ctor
Per review: the serializer still holds the adapter as a non-owning
pointer, but the constructor now takes output_adapter_protocol<char>&
and takes its address internally, so every call site passes a
reference. A reference cannot be null and reads as a borrow, which
makes the lifetime contract harder to get wrong than handing over a
raw pointer. The stored member and the write path are unchanged.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-09-03 07:42:14 +02:00
Claude 386dc225fc Update the serializer test for the non-owning adapter ctor
check_escaped constructed the serializer with output_adapter<char>(ss),
which produced the old owning output_adapter_t. The ctor now takes a
non-owning output_adapter_protocol<char>*, so build the concrete
output_stream_adapter on the stack and pass its address, matching how
dump() and operator<< now call it.

Signed-off-by: Claude <noreply@anthropic.com>
2026-09-03 07:42:13 +02:00
Claude 3ab813565c Stop dump() from heap-allocating its output adapter per call
The serializer held its output sink as output_adapter_t<char>
(a std::shared_ptr<output_adapter_protocol<char>>), which dump() and
operator<< built via make_shared -- one heap allocation per call for a
sink that only wraps a reference to the caller's string or stream.

Hold the sink as a non-owning output_adapter_protocol<char>* instead and
construct the concrete adapter on the stack at the call site. The write
path (o->write_characters) is unchanged, so output is byte-for-byte
identical; a compact dump() of a small object drops from 2 heap
allocations to 1 (only the returned string remains), ~3% faster.

Completes the per-call allocation cleanup on this branch, which already
removed the indent_string buffer (both were reported in #5413).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1oJ2ggRHS37zeVe94QTA1
Signed-off-by: Claude <noreply@anthropic.com>
2026-09-03 07:42:12 +02:00
7 changed files with 278 additions and 37 deletions
+83 -11
View File
@@ -222,7 +222,9 @@ class json_sax_dom_parser
bool string(string_t& val) bool string(string_t& val)
{ {
handle_value(val); // the interface allows moving the value (see json_sax::string), which
// hands the lexer's buffer to the new value instead of copying it
handle_value(std::move(val));
return true; return true;
} }
@@ -532,7 +534,8 @@ class json_sax_dom_callback_parser
bool string(string_t& val) bool string(string_t& val)
{ {
handle_value(val); // see json_sax_dom_parser::string()
handle_value(std::move(val));
return true; return true;
} }
@@ -548,6 +551,11 @@ class json_sax_dom_callback_parser
const bool keep = callback(static_cast<int>(ref_stack.size()), parse_event_t::object_start, discarded); const bool keep = callback(static_cast<int>(ref_stack.size()), parse_event_t::object_start, discarded);
keep_stack.push_back(keep); keep_stack.push_back(keep);
// the key this object will be stored under, read before handle_value()
// may consume it; kept in lockstep with ref_stack so end_object() can
// find the object in its parent again
container_key_stack.push_back(current_key());
auto val = handle_value(BasicJsonType::value_t::object, true); auto val = handle_value(BasicJsonType::value_t::object, true);
ref_stack.push_back(val.second); ref_stack.push_back(val.second);
@@ -581,6 +589,9 @@ class json_sax_dom_callback_parser
// check callback for the key // check callback for the key
const bool keep = callback(static_cast<int>(ref_stack.size()), parse_event_t::key, k); const bool keep = callback(static_cast<int>(ref_stack.size()), parse_event_t::key, k);
key_keep_stack.push_back(keep); key_keep_stack.push_back(keep);
// remember the key so a rejected value can be erased without searching
// the object for it (kept in lockstep with key_keep_stack)
key_stack.push_back(val);
// add discarded value at the given key and store the reference for later // add discarded value at the given key and store the reference for later
if (keep && ref_stack.back()) if (keep && ref_stack.back())
@@ -622,13 +633,16 @@ class json_sax_dom_callback_parser
JSON_ASSERT(!ref_stack.empty()); JSON_ASSERT(!ref_stack.empty());
JSON_ASSERT(!keep_stack.empty()); JSON_ASSERT(!keep_stack.empty());
JSON_ASSERT(!container_key_stack.empty());
ref_stack.pop_back(); ref_stack.pop_back();
keep_stack.pop_back(); keep_stack.pop_back();
const string_t object_key = std::move(container_key_stack.back());
container_key_stack.pop_back();
if (!ref_stack.empty() && ref_stack.back() && ref_stack.back()->is_structured()) if (!ref_stack.empty() && ref_stack.back() && ref_stack.back()->is_structured())
{ {
// remove discarded value // remove discarded value
remove_discarded_value(*ref_stack.back()); remove_discarded_value(*ref_stack.back(), object_key);
} }
return true; return true;
@@ -639,6 +653,9 @@ class json_sax_dom_callback_parser
const bool keep = callback(static_cast<int>(ref_stack.size()), parse_event_t::array_start, discarded); const bool keep = callback(static_cast<int>(ref_stack.size()), parse_event_t::array_start, discarded);
keep_stack.push_back(keep); keep_stack.push_back(keep);
// see start_object()
container_key_stack.push_back(current_key());
auto val = handle_value(BasicJsonType::value_t::array, true); auto val = handle_value(BasicJsonType::value_t::array, true);
ref_stack.push_back(val.second); ref_stack.push_back(val.second);
@@ -701,8 +718,11 @@ class json_sax_dom_callback_parser
JSON_ASSERT(!ref_stack.empty()); JSON_ASSERT(!ref_stack.empty());
JSON_ASSERT(!keep_stack.empty()); JSON_ASSERT(!keep_stack.empty());
JSON_ASSERT(!container_key_stack.empty());
ref_stack.pop_back(); ref_stack.pop_back();
keep_stack.pop_back(); keep_stack.pop_back();
const string_t object_key = std::move(container_key_stack.back());
container_key_stack.pop_back();
// remove discarded value // remove discarded value
if (!ref_stack.empty() && ref_stack.back()) if (!ref_stack.empty() && ref_stack.back())
@@ -716,7 +736,7 @@ class json_sax_dom_callback_parser
// the array is either still stored under its key or was never // the array is either still stored under its key or was never
// stored, leaving the placeholder key() wrote; both show up as // stored, leaving the placeholder key() wrote; both show up as
// a discarded member of the parent object // a discarded member of the parent object
remove_discarded_value(*ref_stack.back()); remove_discarded_value(*ref_stack.back(), object_key);
} }
} }
@@ -809,15 +829,56 @@ class json_sax_dom_callback_parser
} }
#endif #endif
/// remove the discarded value the callback rejected from its parent /*!
static void remove_discarded_value(BasicJsonType& parent) @brief the key the value now being handled will be stored under
Empty unless the enclosing container is an object, in which case it is the
key of the pending key() event. Read before handle_value() consumes that
key, so it is also correct when the value never reaches its parent.
*/
string_t current_key() const
{ {
for (auto it = parent.begin(); it != parent.end(); ++it) if (!ref_stack.empty() && ref_stack.back() && ref_stack.back()->is_object()
&& !key_stack.empty())
{ {
if (it->is_discarded()) return key_stack.back();
}
return string_t{};
}
/*!
@brief remove the discarded value the callback rejected from its parent
A rejected value can only ever be the one most recently added to @a parent:
the last element of an array, or the placeholder key() stored under @a key
in an object. Looking there directly makes this O(1) resp. O(log n), where
searching @a parent for it made a filtering parse quadratic in the number of
members of a single container.
Finding no discarded value there means none was stored in the first place -
the callback rejected the value before it reached its parent - so there is
nothing to remove.
@param[in,out] parent the container to remove the rejected value from
@param[in] key the key the value was stored under; unused for arrays
*/
static void remove_discarded_value(BasicJsonType& parent, const string_t& key)
{ {
parent.erase(it); if (parent.is_array())
break; {
auto& array = *parent.m_data.m_value.array;
if (!array.empty() && array.back().is_discarded())
{
array.pop_back();
}
}
else if (parent.is_object())
{
auto& object = *parent.m_data.m_value.object;
const auto it = object.find(key);
if (it != object.end() && it->second.is_discarded())
{
object.erase(it);
} }
} }
} }
@@ -867,11 +928,14 @@ class json_sax_dom_callback_parser
if (!ref_stack.empty() && ref_stack.back() && ref_stack.back()->is_object()) if (!ref_stack.empty() && ref_stack.back() && ref_stack.back()->is_object())
{ {
JSON_ASSERT(!key_keep_stack.empty()); JSON_ASSERT(!key_keep_stack.empty());
JSON_ASSERT(!key_stack.empty());
const bool placeholder_stored = key_keep_stack.back(); const bool placeholder_stored = key_keep_stack.back();
key_keep_stack.pop_back(); key_keep_stack.pop_back();
const string_t key = std::move(key_stack.back());
key_stack.pop_back();
if (placeholder_stored) if (placeholder_stored)
{ {
remove_discarded_value(*ref_stack.back()); remove_discarded_value(*ref_stack.back(), key);
} }
} }
return {false, nullptr}; return {false, nullptr};
@@ -904,8 +968,10 @@ class json_sax_dom_callback_parser
JSON_ASSERT(ref_stack.back()->is_object()); JSON_ASSERT(ref_stack.back()->is_object());
// check if we should store an element for the current key // check if we should store an element for the current key
JSON_ASSERT(!key_keep_stack.empty()); JSON_ASSERT(!key_keep_stack.empty());
JSON_ASSERT(!key_stack.empty());
const bool store_element = key_keep_stack.back(); const bool store_element = key_keep_stack.back();
key_keep_stack.pop_back(); key_keep_stack.pop_back();
key_stack.pop_back();
if (!store_element) if (!store_element)
{ {
@@ -925,6 +991,12 @@ class json_sax_dom_callback_parser
std::vector<bool> keep_stack {}; // NOLINT(readability-redundant-member-init) std::vector<bool> keep_stack {}; // NOLINT(readability-redundant-member-init)
/// stack to manage which object keys to keep /// stack to manage which object keys to keep
std::vector<bool> key_keep_stack {}; // NOLINT(readability-redundant-member-init) std::vector<bool> key_keep_stack {}; // NOLINT(readability-redundant-member-init)
/// the keys key() stored a placeholder for, in lockstep with key_keep_stack
std::vector<string_t> key_stack {}; // NOLINT(readability-redundant-member-init)
/// for each open container, the key it is stored under in its parent
/// object, in lockstep with ref_stack; unused where the parent is not an
/// object
std::vector<string_t> container_key_stack {}; // NOLINT(readability-redundant-member-init)
/// helper to hold the reference for the next object element /// helper to hold the reference for the next object element
BasicJsonType* object_element = nullptr; BasicJsonType* object_element = nullptr;
/// whether a syntax error occurred /// whether a syntax error occurred
@@ -63,13 +63,14 @@ class serializer
public: public:
/*! /*!
@param[in] s output stream to serialize to @param[in] s output adapter to serialize to; not owned by the serializer,
so it must outlive it (it lives at the call site)
@param[in] ichar indentation character to use @param[in] ichar indentation character to use
@param[in] error_handler_ how to react on decoding errors @param[in] error_handler_ how to react on decoding errors
*/ */
serializer(output_adapter_t<char> s, const char ichar, serializer(output_adapter_protocol<char>& s, const char ichar,
error_handler_t error_handler_ = error_handler_t::strict) error_handler_t error_handler_ = error_handler_t::strict)
: o(std::move(s)) : o(&s)
, loc(std::localeconv()) , loc(std::localeconv())
, thousands_sep(loc->thousands_sep == nullptr ? '\0' : std::char_traits<char>::to_char_type(* (loc->thousands_sep))) , thousands_sep(loc->thousands_sep == nullptr ? '\0' : std::char_traits<char>::to_char_type(* (loc->thousands_sep)))
, decimal_point(loc->decimal_point == nullptr ? '\0' : std::char_traits<char>::to_char_type(* (loc->decimal_point))) , decimal_point(loc->decimal_point == nullptr ? '\0' : std::char_traits<char>::to_char_type(* (loc->decimal_point)))
@@ -1688,8 +1689,8 @@ class serializer
} }
private: private:
/// the output of the serializer /// the output of the serializer (non-owning; the adapter lives at the call site)
output_adapter_t<char> o = nullptr; output_adapter_protocol<char>* o = nullptr;
/// a (hopefully) large enough character buffer /// a (hopefully) large enough character buffer
std::array<char, 64> number_buffer{{}}; std::array<char, 64> number_buffer{{}};
+4 -2
View File
@@ -1341,7 +1341,8 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
const error_handler_t error_handler = error_handler_t::strict) const const error_handler_t error_handler = error_handler_t::strict) const
{ {
string_t result; string_t result;
serializer s(detail::output_adapter<char, string_t>(result), indent_char, error_handler); detail::output_string_adapter<char, string_t> string_adapter(result);
serializer s(string_adapter, indent_char, error_handler);
if (indent >= 0) if (indent >= 0)
{ {
@@ -4055,7 +4056,8 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
o.width(0); o.width(0);
// do the actual serialization // do the actual serialization
serializer s(detail::output_adapter<char>(o), o.fill()); detail::output_stream_adapter<char> stream_adapter(o);
serializer s(stream_adapter, o.fill());
s.dump(j, pretty_print, false, static_cast<unsigned int>(indentation)); s.dump(j, pretty_print, false, static_cast<unsigned int>(indentation));
return o; return o;
} }
+93 -18
View File
@@ -10739,7 +10739,9 @@ class json_sax_dom_parser
bool string(string_t& val) bool string(string_t& val)
{ {
handle_value(val); // the interface allows moving the value (see json_sax::string), which
// hands the lexer's buffer to the new value instead of copying it
handle_value(std::move(val));
return true; return true;
} }
@@ -11049,7 +11051,8 @@ class json_sax_dom_callback_parser
bool string(string_t& val) bool string(string_t& val)
{ {
handle_value(val); // see json_sax_dom_parser::string()
handle_value(std::move(val));
return true; return true;
} }
@@ -11065,6 +11068,11 @@ class json_sax_dom_callback_parser
const bool keep = callback(static_cast<int>(ref_stack.size()), parse_event_t::object_start, discarded); const bool keep = callback(static_cast<int>(ref_stack.size()), parse_event_t::object_start, discarded);
keep_stack.push_back(keep); keep_stack.push_back(keep);
// the key this object will be stored under, read before handle_value()
// may consume it; kept in lockstep with ref_stack so end_object() can
// find the object in its parent again
container_key_stack.push_back(current_key());
auto val = handle_value(BasicJsonType::value_t::object, true); auto val = handle_value(BasicJsonType::value_t::object, true);
ref_stack.push_back(val.second); ref_stack.push_back(val.second);
@@ -11098,6 +11106,9 @@ class json_sax_dom_callback_parser
// check callback for the key // check callback for the key
const bool keep = callback(static_cast<int>(ref_stack.size()), parse_event_t::key, k); const bool keep = callback(static_cast<int>(ref_stack.size()), parse_event_t::key, k);
key_keep_stack.push_back(keep); key_keep_stack.push_back(keep);
// remember the key so a rejected value can be erased without searching
// the object for it (kept in lockstep with key_keep_stack)
key_stack.push_back(val);
// add discarded value at the given key and store the reference for later // add discarded value at the given key and store the reference for later
if (keep && ref_stack.back()) if (keep && ref_stack.back())
@@ -11139,13 +11150,16 @@ class json_sax_dom_callback_parser
JSON_ASSERT(!ref_stack.empty()); JSON_ASSERT(!ref_stack.empty());
JSON_ASSERT(!keep_stack.empty()); JSON_ASSERT(!keep_stack.empty());
JSON_ASSERT(!container_key_stack.empty());
ref_stack.pop_back(); ref_stack.pop_back();
keep_stack.pop_back(); keep_stack.pop_back();
const string_t object_key = std::move(container_key_stack.back());
container_key_stack.pop_back();
if (!ref_stack.empty() && ref_stack.back() && ref_stack.back()->is_structured()) if (!ref_stack.empty() && ref_stack.back() && ref_stack.back()->is_structured())
{ {
// remove discarded value // remove discarded value
remove_discarded_value(*ref_stack.back()); remove_discarded_value(*ref_stack.back(), object_key);
} }
return true; return true;
@@ -11156,6 +11170,9 @@ class json_sax_dom_callback_parser
const bool keep = callback(static_cast<int>(ref_stack.size()), parse_event_t::array_start, discarded); const bool keep = callback(static_cast<int>(ref_stack.size()), parse_event_t::array_start, discarded);
keep_stack.push_back(keep); keep_stack.push_back(keep);
// see start_object()
container_key_stack.push_back(current_key());
auto val = handle_value(BasicJsonType::value_t::array, true); auto val = handle_value(BasicJsonType::value_t::array, true);
ref_stack.push_back(val.second); ref_stack.push_back(val.second);
@@ -11218,8 +11235,11 @@ class json_sax_dom_callback_parser
JSON_ASSERT(!ref_stack.empty()); JSON_ASSERT(!ref_stack.empty());
JSON_ASSERT(!keep_stack.empty()); JSON_ASSERT(!keep_stack.empty());
JSON_ASSERT(!container_key_stack.empty());
ref_stack.pop_back(); ref_stack.pop_back();
keep_stack.pop_back(); keep_stack.pop_back();
const string_t object_key = std::move(container_key_stack.back());
container_key_stack.pop_back();
// remove discarded value // remove discarded value
if (!ref_stack.empty() && ref_stack.back()) if (!ref_stack.empty() && ref_stack.back())
@@ -11233,7 +11253,7 @@ class json_sax_dom_callback_parser
// the array is either still stored under its key or was never // the array is either still stored under its key or was never
// stored, leaving the placeholder key() wrote; both show up as // stored, leaving the placeholder key() wrote; both show up as
// a discarded member of the parent object // a discarded member of the parent object
remove_discarded_value(*ref_stack.back()); remove_discarded_value(*ref_stack.back(), object_key);
} }
} }
@@ -11326,15 +11346,56 @@ class json_sax_dom_callback_parser
} }
#endif #endif
/// remove the discarded value the callback rejected from its parent /*!
static void remove_discarded_value(BasicJsonType& parent) @brief the key the value now being handled will be stored under
Empty unless the enclosing container is an object, in which case it is the
key of the pending key() event. Read before handle_value() consumes that
key, so it is also correct when the value never reaches its parent.
*/
string_t current_key() const
{ {
for (auto it = parent.begin(); it != parent.end(); ++it) if (!ref_stack.empty() && ref_stack.back() && ref_stack.back()->is_object()
&& !key_stack.empty())
{ {
if (it->is_discarded()) return key_stack.back();
}
return string_t{};
}
/*!
@brief remove the discarded value the callback rejected from its parent
A rejected value can only ever be the one most recently added to @a parent:
the last element of an array, or the placeholder key() stored under @a key
in an object. Looking there directly makes this O(1) resp. O(log n), where
searching @a parent for it made a filtering parse quadratic in the number of
members of a single container.
Finding no discarded value there means none was stored in the first place -
the callback rejected the value before it reached its parent - so there is
nothing to remove.
@param[in,out] parent the container to remove the rejected value from
@param[in] key the key the value was stored under; unused for arrays
*/
static void remove_discarded_value(BasicJsonType& parent, const string_t& key)
{ {
parent.erase(it); if (parent.is_array())
break; {
auto& array = *parent.m_data.m_value.array;
if (!array.empty() && array.back().is_discarded())
{
array.pop_back();
}
}
else if (parent.is_object())
{
auto& object = *parent.m_data.m_value.object;
const auto it = object.find(key);
if (it != object.end() && it->second.is_discarded())
{
object.erase(it);
} }
} }
} }
@@ -11384,11 +11445,14 @@ class json_sax_dom_callback_parser
if (!ref_stack.empty() && ref_stack.back() && ref_stack.back()->is_object()) if (!ref_stack.empty() && ref_stack.back() && ref_stack.back()->is_object())
{ {
JSON_ASSERT(!key_keep_stack.empty()); JSON_ASSERT(!key_keep_stack.empty());
JSON_ASSERT(!key_stack.empty());
const bool placeholder_stored = key_keep_stack.back(); const bool placeholder_stored = key_keep_stack.back();
key_keep_stack.pop_back(); key_keep_stack.pop_back();
const string_t key = std::move(key_stack.back());
key_stack.pop_back();
if (placeholder_stored) if (placeholder_stored)
{ {
remove_discarded_value(*ref_stack.back()); remove_discarded_value(*ref_stack.back(), key);
} }
} }
return {false, nullptr}; return {false, nullptr};
@@ -11421,8 +11485,10 @@ class json_sax_dom_callback_parser
JSON_ASSERT(ref_stack.back()->is_object()); JSON_ASSERT(ref_stack.back()->is_object());
// check if we should store an element for the current key // check if we should store an element for the current key
JSON_ASSERT(!key_keep_stack.empty()); JSON_ASSERT(!key_keep_stack.empty());
JSON_ASSERT(!key_stack.empty());
const bool store_element = key_keep_stack.back(); const bool store_element = key_keep_stack.back();
key_keep_stack.pop_back(); key_keep_stack.pop_back();
key_stack.pop_back();
if (!store_element) if (!store_element)
{ {
@@ -11442,6 +11508,12 @@ class json_sax_dom_callback_parser
std::vector<bool> keep_stack {}; // NOLINT(readability-redundant-member-init) std::vector<bool> keep_stack {}; // NOLINT(readability-redundant-member-init)
/// stack to manage which object keys to keep /// stack to manage which object keys to keep
std::vector<bool> key_keep_stack {}; // NOLINT(readability-redundant-member-init) std::vector<bool> key_keep_stack {}; // NOLINT(readability-redundant-member-init)
/// the keys key() stored a placeholder for, in lockstep with key_keep_stack
std::vector<string_t> key_stack {}; // NOLINT(readability-redundant-member-init)
/// for each open container, the key it is stored under in its parent
/// object, in lockstep with ref_stack; unused where the parent is not an
/// object
std::vector<string_t> container_key_stack {}; // NOLINT(readability-redundant-member-init)
/// helper to hold the reference for the next object element /// helper to hold the reference for the next object element
BasicJsonType* object_element = nullptr; BasicJsonType* object_element = nullptr;
/// whether a syntax error occurred /// whether a syntax error occurred
@@ -21123,13 +21195,14 @@ class serializer
public: public:
/*! /*!
@param[in] s output stream to serialize to @param[in] s output adapter to serialize to; not owned by the serializer,
so it must outlive it (it lives at the call site)
@param[in] ichar indentation character to use @param[in] ichar indentation character to use
@param[in] error_handler_ how to react on decoding errors @param[in] error_handler_ how to react on decoding errors
*/ */
serializer(output_adapter_t<char> s, const char ichar, serializer(output_adapter_protocol<char>& s, const char ichar,
error_handler_t error_handler_ = error_handler_t::strict) error_handler_t error_handler_ = error_handler_t::strict)
: o(std::move(s)) : o(&s)
, loc(std::localeconv()) , loc(std::localeconv())
, thousands_sep(loc->thousands_sep == nullptr ? '\0' : std::char_traits<char>::to_char_type(* (loc->thousands_sep))) , thousands_sep(loc->thousands_sep == nullptr ? '\0' : std::char_traits<char>::to_char_type(* (loc->thousands_sep)))
, decimal_point(loc->decimal_point == nullptr ? '\0' : std::char_traits<char>::to_char_type(* (loc->decimal_point))) , decimal_point(loc->decimal_point == nullptr ? '\0' : std::char_traits<char>::to_char_type(* (loc->decimal_point)))
@@ -22748,8 +22821,8 @@ class serializer
} }
private: private:
/// the output of the serializer /// the output of the serializer (non-owning; the adapter lives at the call site)
output_adapter_t<char> o = nullptr; output_adapter_protocol<char>* o = nullptr;
/// a (hopefully) large enough character buffer /// a (hopefully) large enough character buffer
std::array<char, 64> number_buffer{{}}; std::array<char, 64> number_buffer{{}};
@@ -24451,7 +24524,8 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
const error_handler_t error_handler = error_handler_t::strict) const const error_handler_t error_handler = error_handler_t::strict) const
{ {
string_t result; string_t result;
serializer s(detail::output_adapter<char, string_t>(result), indent_char, error_handler); detail::output_string_adapter<char, string_t> string_adapter(result);
serializer s(string_adapter, indent_char, error_handler);
if (indent >= 0) if (indent >= 0)
{ {
@@ -27165,7 +27239,8 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
o.width(0); o.width(0);
// do the actual serialization // do the actual serialization
serializer s(detail::output_adapter<char>(o), o.fill()); detail::output_stream_adapter<char> stream_adapter(o);
serializer s(stream_adapter, o.fill());
s.dump(j, pretty_print, false, static_cast<unsigned int>(indentation)); s.dump(j, pretty_print, false, static_cast<unsigned int>(indentation));
return o; return o;
} }
+38
View File
@@ -81,6 +81,44 @@ BENCHMARK_CAPTURE(ParseString, signed_ints, TEST_DATA_DIRECTORY "/regressi
BENCHMARK_CAPTURE(ParseString, unsigned_ints, TEST_DATA_DIRECTORY "/regression/unsigned_ints.json"); BENCHMARK_CAPTURE(ParseString, unsigned_ints, TEST_DATA_DIRECTORY "/regression/unsigned_ints.json");
BENCHMARK_CAPTURE(ParseString, small_signed_ints, TEST_DATA_DIRECTORY "/regression/small_signed_ints.json"); BENCHMARK_CAPTURE(ParseString, small_signed_ints, TEST_DATA_DIRECTORY "/regression/small_signed_ints.json");
//////////////////////////////////////////////////////////////////////////////
// parse pretty-printed JSON from string
//
// Every file in the corpus above is minified or only lightly spaced, so none of
// them exercise the lexer's whitespace handling. Real-world JSON is frequently
// indented - configuration files, pretty-printed API responses, anything kept
// under version control - where insignificant whitespace can outweigh the data.
// Re-serializing a document with an indentation and parsing that keeps the
// content identical to the ParseString row above, so the pair isolates the cost
// of the whitespace alone.
//////////////////////////////////////////////////////////////////////////////
static void ParseIndented(benchmark::State& state, const char* filename, int indent)
{
std::ifstream f(filename);
std::string str((std::istreambuf_iterator<char>(f)), std::istreambuf_iterator<char>());
const std::string indented = json::parse(str).dump(indent);
while (state.KeepRunning())
{
state.PauseTiming();
auto* j = new json();
state.ResumeTiming();
*j = json::parse(indented);
state.PauseTiming();
delete j;
state.ResumeTiming();
}
state.SetBytesProcessed(state.iterations() * indented.size());
}
BENCHMARK_CAPTURE(ParseIndented, jeopardy / 4, TEST_DATA_DIRECTORY "/jeopardy/jeopardy.json", 4);
BENCHMARK_CAPTURE(ParseIndented, canada / 4, TEST_DATA_DIRECTORY "/nativejson-benchmark/canada.json", 4);
BENCHMARK_CAPTURE(ParseIndented, citm_catalog / 4, TEST_DATA_DIRECTORY "/nativejson-benchmark/citm_catalog.json", 4);
BENCHMARK_CAPTURE(ParseIndented, twitter / 4, TEST_DATA_DIRECTORY "/nativejson-benchmark/twitter.json", 4);
////////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////////
// serialize JSON // serialize JSON
////////////////////////////////////////////////////////////////////////////// //////////////////////////////////////////////////////////////////////////////
+52
View File
@@ -1564,6 +1564,58 @@ TEST_CASE("parser class")
CHECK (j_filtered2 == json({{"foo", {1, 2}}})); CHECK (j_filtered2 == json({{"foo", {1, 2}}}));
} }
SECTION("filter many members of one container")
{
// Rejecting a value makes the parser remove the placeholder its key
// event stored. Locating that placeholder used to be a scan of the
// whole parent, which made filtering a large container quadratic:
// 128k members took ~25 s. These cases keep many members alive
// while discarding many others, so the removal cost is the whole
// point; they run in milliseconds when the placeholder is erased
// directly.
constexpr int count = 20000;
std::string s = "{";
for (int i = 0; i < count; ++i)
{
// "a<i>" is kept, "z<i>" is discarded
s += "\"a" + std::to_string(i) + "\":" + std::to_string(i) + ",";
s += "\"z" + std::to_string(i) + "\":-1,";
}
s.back() = '}';
const json j_values = json::parse(s, [](int /*unused*/, json::parse_event_t e, const json & parsed) noexcept
{
return !(e == json::parse_event_t::value && parsed == json(-1));
});
CHECK(j_values.size() == count);
CHECK(j_values.at("a0") == json(0));
CHECK(j_values.at("a" + std::to_string(count - 1)) == json(count - 1));
CHECK_FALSE(j_values.contains("z0"));
CHECK_FALSE(j_values.contains("z" + std::to_string(count - 1)));
// the same, but discarding whole containers rather than values,
// which takes the end_object()/end_array() removal path
std::string s_nested = "{";
for (int i = 0; i < count; ++i)
{
s_nested += "\"a" + std::to_string(i) + "\":" + std::to_string(i) + ",";
s_nested += "\"z" + std::to_string(i) + "\":[1,2],";
}
s_nested.back() = '}';
const json j_arrays = json::parse(s_nested, [](int /*unused*/, json::parse_event_t e, const json& /*unused*/) noexcept
{
return e != json::parse_event_t::array_end;
});
CHECK(j_arrays.size() == count);
CHECK(j_arrays.at("a0") == json(0));
CHECK_FALSE(j_arrays.contains("z0"));
CHECK_FALSE(j_arrays.contains("z" + std::to_string(count - 1)));
}
SECTION("filter specific events") SECTION("filter specific events")
{ {
SECTION("first closing event") SECTION("first closing event")
+2 -1
View File
@@ -98,7 +98,8 @@ void check_escaped(const char* original, const char* escaped = "", bool ensure_a
void check_escaped(const char* original, const char* escaped, const bool ensure_ascii) void check_escaped(const char* original, const char* escaped, const bool ensure_ascii)
{ {
std::stringstream ss; std::stringstream ss;
json::serializer s(nlohmann::detail::output_adapter<char>(ss), ' '); nlohmann::detail::output_stream_adapter<char> adapter(ss);
json::serializer s(adapter, ' ');
s.dump_escaped(original, ensure_ascii); s.dump_escaped(original, ensure_ascii);
s.flush(); // dump_escaped writes into the serializer's internal buffer s.flush(); // dump_escaped writes into the serializer's internal buffer
CHECK(ss.str() == escaped); CHECK(ss.str() == escaped);