mirror of
https://github.com/nlohmann/json.git
synced 2026-09-11 18:57:58 +00:00
Niels LohmannandClaude Opus 4.8
ff80ed3295
* Add SWAR bulk fast path to string serialization (dump_escaped) When ensure_ascii is false, dump_escaped previously ran every byte of every string and object key through the UTF-8 DFA decoder, even for the common case of ordinary text with nothing to escape. This mirrors the per-byte cost the parser had before the contiguous fast paths. At a character boundary, bulk-copy the longest run of bytes that need no escaping using string_bulk_run() - the same SWAR scanner and UTF-8 bulk validator the lexer's contiguous path uses - and only fall back to the byte-at-a-time DFA loop for the first byte that needs individual handling (a quote, backslash, control character, or ill-formed/truncated UTF-8). Because every "hard" or invalid byte is still processed by the unchanged byte path, escaping output and error handling (including strict-mode error 316 position and message) are byte-identical to before. The ensure_ascii=true path is unchanged: it must escape non-ASCII and 0x7F, which string_bulk_run does not stop on, so a separate predicate would be needed for it. Verified byte-for-byte identical dump output against the pre-change implementation across ~20k randomized byte strings plus curated edge cases (all escapes, control chars, valid multibyte, surrogates, overlong, truncated sequences) for both ensure_ascii settings and all three error handlers, in C++11/17/20 at -O2/-O3. Throughput (g++ -O3, ensure_ascii=false, vs pre-change): long ASCII strings 4.2x twitter-like objects 2.3x dense CJK 1.4x (further headroom with JSON_USE_SIMDUTF) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Buffer serializer output and add ensure_ascii string fast path Two further serialization speedups on top of the ensure_ascii=false bulk copy, both reusing the SWAR primitives in detail/input/string_scan.hpp. 1. Internal write buffer (devirtualization). Every structural character ('{', '"', ',', ...) previously went straight to the output adapter through a virtual call. Route all writes through put_char/put_chars into a 1 KiB buffer that flushes in bulk; the public dump() flushes once the top-level value is done (the recursive worker is split out as dump_internal). Runs larger than the buffer are written straight through, so large payloads are not copied twice. This is the dominant cost for object/array-heavy values. 2. ensure_ascii fast path. dump_escaped previously ran the UTF-8 DFA over every byte when escaping non-ASCII. Add find_ascii_copyable_run() (a SWAR scan stopping at '"', '\\', < 0x20, 0x7F, and >= 0x80) so runs of printable ASCII are bulk-copied, with the byte path handling each escape/non-ASCII byte exactly as before. Behavior is unchanged: dump output is byte-for-byte identical to the previous implementation across ~20k randomized byte strings plus curated edge cases (all escapes, control chars, 0x7F, valid multibyte, surrogates, overlong, truncated), for object/array/pretty output, both ensure_ascii settings, and all three error handlers, in C++11/17/20 at -O2/-O3. New unit tests cover the buffer flush boundaries, the escape and 0x7F handling, multibyte under both settings, and invalid-UTF-8 handling. Throughput (g++ -O3, vs the ensure_ascii=false-only baseline): long ASCII, ensure_ascii=0 4.2x long ASCII, ensure_ascii=1 4.1x twitter-like objects 2.7x dense CJK 1.8x Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Flush serializer buffer in dump_escaped unit test test-convenience failed (macOS finished first; the failure is platform-independent) because check_escaped() calls the internal serializer::dump_escaped() directly and then reads the output stream. Since dump_escaped() now writes into the serializer's internal write buffer, the bytes were still buffered and the stream was empty. Expose flush() under JSON_PRIVATE_UNLESS_TESTED (same visibility as dump_escaped) and flush in check_escaped() before inspecting the output. Per-string flushing inside dump_escaped() was rejected on purpose: it would defeat the buffering that makes object/array-heavy dumps faster. Library behavior is unchanged (flush()'s body is identical; only its access label moved). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Avoid deep recursion in serialization write-buffer test The "many small structural writes exceed the write buffer" subcase built a 1100-deep nested array and dumped it to force >1024 consecutive single-character writes through put_char (exercising the write buffer's flush-when-full branch). dump() recurses per nesting level, so on MSVC debug builds (smaller default stack, larger frames) this overflowed the stack and crashed test-serialization; Linux/macOS have enough headroom to hide it. Replace the nesting with a flat array of 500 empty strings. Each element emits '"', '"', ',' via put_char, so the dump is a long run of single-character writes (1501 bytes > the 1024-byte buffer) at nesting depth two, hitting the same flush branch without deep recursion. Library code is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Split the write-buffer helpers and write indentation directly Follow-up to @gregmarr's review: put_chars() was doing four unrelated jobs, so give the two that can be made safe their own entry points. - put_literal(): takes the literal by reference and deduces the length from the array bound, so the 27 hand-counted lengths at the call sites can no longer drift from the literals they describe. A literal is checked at compile time to fit the buffer, so this path needs no write-through branch. - put_buffer(): takes the fixed-size buffer itself rather than a bare pointer, so the length can be checked against the buffer's own bound. - put_indent(): memsets the indentation into the write buffer, filling and flushing it as needed. This removes indent_string entirely, and with it both bugs of #5186: the indentation string was grown by doubling, which is not enough when indent_step more than doubles it (a heap over-read - dump(2000) read 2000 bytes out of a 1024-byte string), and the grown part was filled with a space instead of the configured indent_char. next_indent() keeps that PR's assertion against the unsigned indentation accumulation wrapping on deep nesting. put_chars() keeps the two cases that are genuinely a pointer and a count: the run-length copies out of the string being escaped, and to_chars() output. Tests cover an indent_step wider than the write buffer, a non-space indentation character past the old growth point, and nesting whose accumulated indentation spans several buffer-fulls. All three fail against develop. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fill the indentation buffer once instead of once per flush @gregmarr's point on the fill-and-flush loop: flushing does not disturb what the write buffer holds, so an indentation spanning several buffer-fulls only has to be written into the buffer once and can then be handed to the adapter as many times as needed. The loop re-filled it every time, doing work it already knew was there. put_indent() now fills the room left in the buffer, and if anything remains, flushes, fills the buffer once, and re-flushes that same content. It also returns early for a zero-width indentation, which is what the closing brace of every outermost value asks for. Measured over a dump(), counting memset calls and bytes inside put_indent: indent before after 4 1 call / 4 B 1 call / 4 B 2000 2 calls / 2000 B 2 calls / 2046 B 100000 98 calls / 100000 B 2 calls / 2046 B The wide case is now constant work rather than proportional to the indentation width; ordinary widths are unchanged. Tests extended to cover several whole buffer-fulls and an exact multiple of the buffer size. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Tighten the write-buffer helpers after review More of @gregmarr's review on the put_* split: - Reattach the put_chars() doc comment, which the new helpers had been inserted in front of, leaving it describing put_indent(). - Compute the literal length once in put_literal() instead of spelling N - 1 at each use. - Add put_string(str, start, end), which keeps the pointer arithmetic and the bounds assertions inside the function instead of at the call site. With dump_float()'s to_chars() output moved onto put_buffer() as well, put_chars() now has no callers outside put_string()/put_buffer(): nothing passes a bare pointer and a count any more. - Carry the indentation as std::size_t rather than unsigned int. It is a size, it is compared and combined with buffer sizes throughout, and the casts in put_indent() disappear. next_indent() keeps its assertion, which is far harder to trip on a 64-bit size_t but still reachable where that is 32 bits. No output change: pretty and compact dumps, binary values included, are byte-identical to develop. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Silence avoid-c-arrays on put_literal's array reference clang-tidy flags the reference-to-array parameter under cppcoreguidelines/hicpp/modernize-avoid-c-arrays, and the CI treats warnings as errors. Binding to the array is the whole point here - it is what lets the length be deduced from the literal instead of hand-written at the call site - so suppress it the same way from_json(), to_json() and get_to() already suppress it for their own T (&arr)[N] parameters. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Bound the descent of dump() Serializing a container serializes its elements, so dump() descended into one call per nesting level. A value nested deeply enough exhausted the call stack and terminated the process with a segmentation fault - no exception, nothing the caller could catch. Parsing such a value works, as the parser is iterative, and so does destroying one, as #1436 made destruction iterative. Bound how far the descent goes rather than take the call stack away from it. The first 128 levels are written by exactly the code that always wrote them, and only below that does dump_iteratively write out what is left, keeping the containers it has entered on an explicit stack. Serializing can therefore no longer exhaust the stack, however deeply a value is nested, while a value nested less deeply than the bound pays only for one comparison per container. Writing every value that way instead measured between 2% and 20% slower - 20% on object-heavy documents - which is why the descent is kept for all but the values that cannot afford it. The bound costs nothing measurable: between -1.4% and +1.2% across compact and pretty output of number, integer, string, object-heavy, wide-object and deeply nested documents. The output is unchanged for every value. Both ways of writing a container emit the separator in front of every element but the first, rather than after every element but the last, which puts exactly one between each pair and none at the end. This fixes #5387 for dump(). The copy constructor is fixed in #5389. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fold ensure_ascii into the escaper and write bytes without dump_integer Two hot spots that the write buffer and the bulk scanner left behind. dump_escaped took ensure_ascii as a runtime flag and tested it inside the loop, once per character run, although it cannot change while a string is written. It is now a template parameter, dispatched once per string, which folds the choice of scanner and lets each of the two be inlined into a loop of its own. This is the hottest loop in the serializer: it runs over every string and every object key. A binary value's bytes went through dump_integer, which counts digits and does 64-bit arithmetic for a number that is always in [0, 255]. dump_byte writes the three digits it takes at most straight into the write buffer instead. Any byte type that is not a plain unsigned byte is still left to dump_integer, whose representation of it may differ. Measured against the previous commit (medians of 9 interleaved runs, clang -O3): binary values -33.8%, dense CJK with ensure_ascii -20.6%, key-heavy objects -17.8%, deeply nested pretty output -17.9%, dense CJK without ensure_ascii -11.8%, object-heavy documents -9.3% compact and -9.5% pretty, a small value dumped in a loop -21.4%, wide objects -2.3%. Arrays of plain ASCII strings measured 3.5% to 4.2% slower, the one shape that loses; number and integer arrays are unchanged. Also tried and dropped: leaving the write and string buffers uninitialized rather than zeroing 1.5 KB per dump() call. It is worth -30% on small values, but two nearly identical string workloads moved 18% apart in opposite directions, so the measurements did not support it. The output is unchanged for every value: the differential now also covers every one of the 256 byte values, alone and together, in both binary layouts. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Write a byte without walking a pointer over the buffer clang-tidy's misc-const-correctness reads the pointer dump_byte advanced over the write buffer as one whose pointee could be const. Index the buffer instead, which says the same thing without a raw pointer at all. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Parenthesize the reserve arithmetic in the deep-nesting test clang-tidy's readability-math-missing-parentheses wants the multiplication spelled out in reserve(6 * depth + 1), and CI treats its warnings as errors. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Do not scan for a copyable run that cannot exist Under ensure_ascii, dump_escaped() calls find_ascii_copyable_run() at every character boundary. When the text is dense non-ASCII - CJK, where every byte is >= 0x80 - the scanner stops on its first byte and returns zero, so its SWAR block runs once per character and buys nothing, on top of the escaping that still has to happen afterwards. A run can only be non-empty when the first byte is one the scanner may copy, so test that single byte before calling it. Runs that do exist are found exactly as before, so the bulk-copy win is unchanged; only the calls that were always going to return zero are skipped. Output is unchanged: the dump digest over canada/citm/twitter, in compact, pretty and ensure_ascii form, matches develop byte for byte. dump(ensure_ascii=true) develop before after CJK text 3.54ms 4.25ms 3.36ms CJK, no ASCII at all 3.09ms 4.02ms 3.02ms Latin-1-ish text 4.39ms 3.04ms 2.93ms plain ASCII 3.92ms 0.80ms 0.79ms Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address review of the write-buffer helpers Three points from @gregmarr's review: put_chars() is gone. It was the only entry point taking a bare pointer and a count, and it existed only so put_string() and put_buffer() had something to delegate to. Its body now lives in put_string(), and put_buffer() is put_string(buffer, 0, length) - std::array already carries data() and size(), so it satisfies the same interface a string does. Nothing appends characters without a bound any more. dump_escaped()'s documentation block was duplicated. The dispatcher was inserted between the original comment and the function it described, and the comment was copied rather than split. The worker now has its own short comment saying why ensure_ascii is a template parameter. The local in dump_byte() is deliberate, and is now documented as such: writing through write_buffer[] is a char write, which may alias any object, so with write_buffer_pos updated in place the compiler must reload and store it around every digit. Measured on a dump of a 4 MiB binary value, 18.0 ms without the local against 7.4 ms with it. Output is unchanged: byte-identical dumps across 77 files in compact, pretty, ensure_ascii, pretty+ascii, indent 600 and tab-indent form. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address review: drop unneeded backslash-escapes and duplicate scan loop '"' does not need escaping in a char literal, unlike in a string literal. find_ascii_copyable_run() also duplicated the byte-at-a-time search that already exists as the loop's own scalar tail; break into it instead of re-deriving the offset in a second, near-identical loop. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Move pretty_print, ensure_ascii and indent_step into the serializer None of these change over the life of a serializer, unlike current_indent and depth, which do change on every recursive call. They are now captured once in the constructor - matching indent_char and error_handler - instead of being threaded through dump(), dump_internal(), dump_iteratively(), dump_value() and dump_escaped() on every call. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Stop the serializer from holding onto std::localeconv()'s pointer loc was only ever read twice, immediately, to seed thousands_sep and decimal_point; nothing else in the class used it. A local in the constructor body serves the same purpose without keeping the pointer around for the serializer's lifetime. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Keep thousands_sep/decimal_point const via a small locale_chars struct const members can't be assigned in a constructor body, so seeding them from std::localeconv() meant either dropping const or holding onto the lconv* for longer than needed. A sub-object computes both from the pointer in its own constructor and is itself initialized in serializer's mem-initializer-list, so the two chars stay const, std::localeconv() is still called exactly once, and nothing outlives the constructor. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
619 lines
24 KiB
C++
619 lines
24 KiB
C++
// __ _____ _____ _____
|
|
// __| | __| | | | JSON for Modern C++ (supporting code)
|
|
// | | |__ | | | | | | version 3.12.0
|
|
// |_____|_____|_____|_|___| https://github.com/nlohmann/json
|
|
//
|
|
// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
|
|
// SPDX-License-Identifier: MIT
|
|
|
|
#include "doctest_compatibility.h"
|
|
|
|
#include <nlohmann/json.hpp>
|
|
using nlohmann::json;
|
|
|
|
#include <array>
|
|
#include <sstream>
|
|
#include <iomanip>
|
|
|
|
#include "test_utils.hpp"
|
|
|
|
TEST_CASE("serialization")
|
|
{
|
|
SECTION("operator<<")
|
|
{
|
|
SECTION("no given width")
|
|
{
|
|
std::stringstream ss;
|
|
const json j = {"foo", 1, 2, 3, false, {{"one", 1}}};
|
|
ss << j;
|
|
CHECK(ss.str() == "[\"foo\",1,2,3,false,{\"one\":1}]");
|
|
}
|
|
|
|
SECTION("given width")
|
|
{
|
|
std::stringstream ss;
|
|
const json j = {"foo", 1, 2, 3, false, {{"one", 1}}};
|
|
ss << std::setw(4) << j;
|
|
CHECK(ss.str() ==
|
|
"[\n \"foo\",\n 1,\n 2,\n 3,\n false,\n {\n \"one\": 1\n }\n]");
|
|
}
|
|
|
|
SECTION("given fill")
|
|
{
|
|
std::stringstream ss;
|
|
const json j = {"foo", 1, 2, 3, false, {{"one", 1}}};
|
|
ss << std::setw(1) << std::setfill('\t') << j;
|
|
CHECK(ss.str() ==
|
|
"[\n\t\"foo\",\n\t1,\n\t2,\n\t3,\n\tfalse,\n\t{\n\t\t\"one\": 1\n\t}\n]");
|
|
}
|
|
}
|
|
|
|
SECTION("operator>>")
|
|
{
|
|
SECTION("no given width")
|
|
{
|
|
std::stringstream ss;
|
|
const json j = {"foo", 1, 2, 3, false, {{"one", 1}}};
|
|
j >> ss;
|
|
CHECK(ss.str() == "[\"foo\",1,2,3,false,{\"one\":1}]");
|
|
}
|
|
|
|
SECTION("given width")
|
|
{
|
|
std::stringstream ss;
|
|
const json j = {"foo", 1, 2, 3, false, {{"one", 1}}};
|
|
ss.width(4);
|
|
j >> ss;
|
|
CHECK(ss.str() ==
|
|
"[\n \"foo\",\n 1,\n 2,\n 3,\n false,\n {\n \"one\": 1\n }\n]");
|
|
}
|
|
|
|
SECTION("given fill")
|
|
{
|
|
std::stringstream ss;
|
|
const json j = {"foo", 1, 2, 3, false, {{"one", 1}}};
|
|
ss.width(1);
|
|
ss.fill('\t');
|
|
j >> ss;
|
|
CHECK(ss.str() ==
|
|
"[\n\t\"foo\",\n\t1,\n\t2,\n\t3,\n\tfalse,\n\t{\n\t\t\"one\": 1\n\t}\n]");
|
|
}
|
|
}
|
|
|
|
SECTION("dump")
|
|
{
|
|
SECTION("invalid character")
|
|
{
|
|
const json j = "ä\xA9ü";
|
|
|
|
// dump() is nodiscard; the exception is thrown by dump() itself before it would return
|
|
CHECK_THROWS_WITH_AS(utils::ignore_return_value(j.dump()), "[json.exception.type_error.316] invalid UTF-8 byte at index 2: 0xA9", json::type_error&);
|
|
CHECK_THROWS_WITH_AS(utils::ignore_return_value(j.dump(1, ' ', false, json::error_handler_t::strict)), "[json.exception.type_error.316] invalid UTF-8 byte at index 2: 0xA9", json::type_error&);
|
|
CHECK(j.dump(-1, ' ', false, json::error_handler_t::ignore) == "\"äü\"");
|
|
CHECK(j.dump(-1, ' ', false, json::error_handler_t::replace) == "\"ä\xEF\xBF\xBDü\"");
|
|
CHECK(j.dump(-1, ' ', true, json::error_handler_t::replace) == "\"\\u00e4\\ufffd\\u00fc\"");
|
|
}
|
|
|
|
SECTION("ending with incomplete character")
|
|
{
|
|
const json j = "123\xC2";
|
|
|
|
// dump() is nodiscard; the exception is thrown by dump() itself before it would return
|
|
CHECK_THROWS_WITH_AS(utils::ignore_return_value(j.dump()), "[json.exception.type_error.316] incomplete UTF-8 string; last byte: 0xC2", json::type_error&);
|
|
CHECK_THROWS_AS(utils::ignore_return_value(j.dump(1, ' ', false, json::error_handler_t::strict)), json::type_error&);
|
|
CHECK(j.dump(-1, ' ', false, json::error_handler_t::ignore) == "\"123\"");
|
|
CHECK(j.dump(-1, ' ', false, json::error_handler_t::replace) == "\"123\xEF\xBF\xBD\"");
|
|
CHECK(j.dump(-1, ' ', true, json::error_handler_t::replace) == "\"123\\ufffd\"");
|
|
}
|
|
|
|
SECTION("unexpected character")
|
|
{
|
|
const json j = "123\xF1\xB0\x34\x35\x36";
|
|
|
|
// dump() is nodiscard; the exception is thrown by dump() itself before it would return
|
|
CHECK_THROWS_WITH_AS(utils::ignore_return_value(j.dump()), "[json.exception.type_error.316] invalid UTF-8 byte at index 5: 0x34", json::type_error&);
|
|
CHECK_THROWS_AS(utils::ignore_return_value(j.dump(1, ' ', false, json::error_handler_t::strict)), json::type_error&);
|
|
CHECK(j.dump(-1, ' ', false, json::error_handler_t::ignore) == "\"123456\"");
|
|
CHECK(j.dump(-1, ' ', false, json::error_handler_t::replace) == "\"123\xEF\xBF\xBD\x34\x35\x36\"");
|
|
CHECK(j.dump(-1, ' ', true, json::error_handler_t::replace) == "\"123\\ufffd456\"");
|
|
}
|
|
|
|
SECTION("U+FFFD Substitution of Maximal Subparts")
|
|
{
|
|
// Some tests (mostly) from
|
|
// https://www.unicode.org/versions/Unicode11.0.0/ch03.pdf
|
|
// Section 3.9 -- U+FFFD Substitution of Maximal Subparts
|
|
|
|
auto test = [&](std::string const & input, std::string const & expected)
|
|
{
|
|
const json j = input;
|
|
CHECK(j.dump(-1, ' ', true, json::error_handler_t::replace) == "\"" + expected + "\"");
|
|
};
|
|
|
|
test("\xC2", "\\ufffd");
|
|
test("\xC2\x41\x42", "\\ufffd" "\x41" "\x42");
|
|
test("\xC2\xF4", "\\ufffd" "\\ufffd");
|
|
|
|
test("\xF0\x80\x80\x41", "\\ufffd" "\\ufffd" "\\ufffd" "\x41");
|
|
test("\xF1\x80\x80\x41", "\\ufffd" "\x41");
|
|
test("\xF2\x80\x80\x41", "\\ufffd" "\x41");
|
|
test("\xF3\x80\x80\x41", "\\ufffd" "\x41");
|
|
test("\xF4\x80\x80\x41", "\\ufffd" "\x41");
|
|
test("\xF5\x80\x80\x41", "\\ufffd" "\\ufffd" "\\ufffd" "\x41");
|
|
|
|
test("\xF0\x90\x80\x41", "\\ufffd" "\x41");
|
|
test("\xF1\x90\x80\x41", "\\ufffd" "\x41");
|
|
test("\xF2\x90\x80\x41", "\\ufffd" "\x41");
|
|
test("\xF3\x90\x80\x41", "\\ufffd" "\x41");
|
|
test("\xF4\x90\x80\x41", "\\ufffd" "\\ufffd" "\\ufffd" "\x41");
|
|
test("\xF5\x90\x80\x41", "\\ufffd" "\\ufffd" "\\ufffd" "\x41");
|
|
|
|
test("\xC0\xAF\xE0\x80\xBF\xF0\x81\x82\x41", "\\ufffd" "\\ufffd" "\\ufffd" "\\ufffd" "\\ufffd" "\\ufffd" "\\ufffd" "\\ufffd" "\x41");
|
|
test("\xED\xA0\x80\xED\xBF\xBF\xED\xAF\x41", "\\ufffd" "\\ufffd" "\\ufffd" "\\ufffd" "\\ufffd" "\\ufffd" "\\ufffd" "\\ufffd" "\x41");
|
|
test("\xF4\x91\x92\x93\xFF\x41\x80\xBF\x42", "\\ufffd" "\\ufffd" "\\ufffd" "\\ufffd" "\\ufffd" "\x41" "\\ufffd""\\ufffd" "\x42");
|
|
test("\xE1\x80\xE2\xF0\x91\x92\xF1\xBF\x41", "\\ufffd" "\\ufffd" "\\ufffd" "\\ufffd" "\x41");
|
|
}
|
|
}
|
|
|
|
SECTION("to_string")
|
|
{
|
|
auto test = [&](std::string const & input, std::string const & expected)
|
|
{
|
|
using std::to_string;
|
|
const json j = input;
|
|
CHECK(to_string(j) == "\"" + expected + "\"");
|
|
};
|
|
|
|
test(R"({"x":5,"y":6})", R"({\"x\":5,\"y\":6})");
|
|
test("{\"x\":[10,null,null,null]}", R"({\"x\":[10,null,null,null]})");
|
|
test("test", "test");
|
|
test("[3,\"false\",false]", R"([3,\"false\",false])");
|
|
}
|
|
}
|
|
|
|
TEST_CASE_TEMPLATE("serialization for extreme integer values", T, int32_t, uint32_t, int64_t, uint64_t) // NOLINT(readability-math-missing-parentheses, bugprone-throwing-static-initialization)
|
|
{
|
|
SECTION("minimum")
|
|
{
|
|
constexpr auto minimum = (std::numeric_limits<T>::min)();
|
|
const json j = minimum;
|
|
CHECK(j.dump() == std::to_string(minimum));
|
|
}
|
|
|
|
SECTION("maximum")
|
|
{
|
|
constexpr auto maximum = (std::numeric_limits<T>::max)();
|
|
const json j = maximum;
|
|
CHECK(j.dump() == std::to_string(maximum));
|
|
}
|
|
}
|
|
|
|
TEST_CASE("dump with binary values")
|
|
{
|
|
auto binary = json::binary({1, 2, 3, 4});
|
|
auto binary_empty = json::binary({});
|
|
auto binary_with_subtype = json::binary({1, 2, 3, 4}, 128);
|
|
auto binary_empty_with_subtype = json::binary({}, 128);
|
|
|
|
const json object = {{"key", binary}};
|
|
const json object_empty = {{"key", binary_empty}};
|
|
const json object_with_subtype = {{"key", binary_with_subtype}};
|
|
const json object_empty_with_subtype = {{"key", binary_empty_with_subtype}};
|
|
|
|
const json array = {"value", 1, binary};
|
|
const json array_empty = {"value", 1, binary_empty};
|
|
const json array_with_subtype = {"value", 1, binary_with_subtype};
|
|
const json array_empty_with_subtype = {"value", 1, binary_empty_with_subtype};
|
|
|
|
SECTION("normal")
|
|
{
|
|
CHECK(binary.dump() == "{\"bytes\":[1,2,3,4],\"subtype\":null}");
|
|
CHECK(binary_empty.dump() == "{\"bytes\":[],\"subtype\":null}");
|
|
CHECK(binary_with_subtype.dump() == "{\"bytes\":[1,2,3,4],\"subtype\":128}");
|
|
CHECK(binary_empty_with_subtype.dump() == "{\"bytes\":[],\"subtype\":128}");
|
|
|
|
CHECK(object.dump() == "{\"key\":{\"bytes\":[1,2,3,4],\"subtype\":null}}");
|
|
CHECK(object_empty.dump() == "{\"key\":{\"bytes\":[],\"subtype\":null}}");
|
|
CHECK(object_with_subtype.dump() == "{\"key\":{\"bytes\":[1,2,3,4],\"subtype\":128}}");
|
|
CHECK(object_empty_with_subtype.dump() == "{\"key\":{\"bytes\":[],\"subtype\":128}}");
|
|
|
|
CHECK(array.dump() == "[\"value\",1,{\"bytes\":[1,2,3,4],\"subtype\":null}]");
|
|
CHECK(array_empty.dump() == "[\"value\",1,{\"bytes\":[],\"subtype\":null}]");
|
|
CHECK(array_with_subtype.dump() == "[\"value\",1,{\"bytes\":[1,2,3,4],\"subtype\":128}]");
|
|
CHECK(array_empty_with_subtype.dump() == "[\"value\",1,{\"bytes\":[],\"subtype\":128}]");
|
|
}
|
|
|
|
SECTION("pretty-printed")
|
|
{
|
|
CHECK(binary.dump(4) == "{\n"
|
|
" \"bytes\": [1, 2, 3, 4],\n"
|
|
" \"subtype\": null\n"
|
|
"}");
|
|
CHECK(binary_empty.dump(4) == "{\n"
|
|
" \"bytes\": [],\n"
|
|
" \"subtype\": null\n"
|
|
"}");
|
|
CHECK(binary_with_subtype.dump(4) == "{\n"
|
|
" \"bytes\": [1, 2, 3, 4],\n"
|
|
" \"subtype\": 128\n"
|
|
"}");
|
|
CHECK(binary_empty_with_subtype.dump(4) == "{\n"
|
|
" \"bytes\": [],\n"
|
|
" \"subtype\": 128\n"
|
|
"}");
|
|
|
|
CHECK(object.dump(4) == "{\n"
|
|
" \"key\": {\n"
|
|
" \"bytes\": [1, 2, 3, 4],\n"
|
|
" \"subtype\": null\n"
|
|
" }\n"
|
|
"}");
|
|
CHECK(object_empty.dump(4) == "{\n"
|
|
" \"key\": {\n"
|
|
" \"bytes\": [],\n"
|
|
" \"subtype\": null\n"
|
|
" }\n"
|
|
"}");
|
|
CHECK(object_with_subtype.dump(4) == "{\n"
|
|
" \"key\": {\n"
|
|
" \"bytes\": [1, 2, 3, 4],\n"
|
|
" \"subtype\": 128\n"
|
|
" }\n"
|
|
"}");
|
|
CHECK(object_empty_with_subtype.dump(4) == "{\n"
|
|
" \"key\": {\n"
|
|
" \"bytes\": [],\n"
|
|
" \"subtype\": 128\n"
|
|
" }\n"
|
|
"}");
|
|
|
|
CHECK(array.dump(4) == "[\n"
|
|
" \"value\",\n"
|
|
" 1,\n"
|
|
" {\n"
|
|
" \"bytes\": [1, 2, 3, 4],\n"
|
|
" \"subtype\": null\n"
|
|
" }\n"
|
|
"]");
|
|
CHECK(array_empty.dump(4) == "[\n"
|
|
" \"value\",\n"
|
|
" 1,\n"
|
|
" {\n"
|
|
" \"bytes\": [],\n"
|
|
" \"subtype\": null\n"
|
|
" }\n"
|
|
"]");
|
|
CHECK(array_with_subtype.dump(4) == "[\n"
|
|
" \"value\",\n"
|
|
" 1,\n"
|
|
" {\n"
|
|
" \"bytes\": [1, 2, 3, 4],\n"
|
|
" \"subtype\": 128\n"
|
|
" }\n"
|
|
"]");
|
|
CHECK(array_empty_with_subtype.dump(4) == "[\n"
|
|
" \"value\",\n"
|
|
" 1,\n"
|
|
" {\n"
|
|
" \"bytes\": [],\n"
|
|
" \"subtype\": 128\n"
|
|
" }\n"
|
|
"]");
|
|
}
|
|
}
|
|
|
|
TEST_CASE("dump for basic_json with long double number_float_t")
|
|
{
|
|
// Custom basic_json instantiation with long double as NumberFloatType.
|
|
// On platforms where long double is wider than double (e.g. GCC/Clang on
|
|
// Linux/macOS x86_64), dump() goes through the snprintf path in
|
|
// serializer::dump_float(x, std::false_type). That branch must use the
|
|
// "%.*Lg" format specifier; using "%.*g" with a long double argument is
|
|
// undefined behavior and corrupts the output.
|
|
using long_double_json = nlohmann::basic_json<std::map, std::vector, std::string,
|
|
bool, std::int64_t, std::uint64_t, long double>;
|
|
|
|
SECTION("round-trip dump/parse")
|
|
{
|
|
constexpr std::array<long double, 13> values =
|
|
{
|
|
{
|
|
0.0L, -0.0L, 1.0L, -1.0L,
|
|
0.5L, -0.5L, 1.5L, -2.25L,
|
|
1.23e45L, 1.23e-45L,
|
|
(std::numeric_limits<long double>::min)(),
|
|
std::numeric_limits<long double>::lowest(),
|
|
(std::numeric_limits<long double>::max)()
|
|
}
|
|
};
|
|
|
|
for (long double v : values)
|
|
{
|
|
const long_double_json j = v;
|
|
const auto s = j.dump();
|
|
const auto j2 = long_double_json::parse(s);
|
|
CHECK(j2.template get<long double>() == v);
|
|
}
|
|
}
|
|
|
|
SECTION("exact dump string for simple values")
|
|
{
|
|
CHECK(long_double_json(0.5L).dump() == "0.5");
|
|
CHECK(long_double_json(-0.5L).dump() == "-0.5");
|
|
CHECK(long_double_json(1.5L).dump() == "1.5");
|
|
CHECK(long_double_json(-2.25L).dump() == "-2.25");
|
|
CHECK(long_double_json(0.0L).dump() == "0.0");
|
|
CHECK(long_double_json(1.0L).dump() == "1.0");
|
|
CHECK(long_double_json(-1.0L).dump() == "-1.0");
|
|
CHECK(long_double_json(100.0L).dump() == "100.0");
|
|
}
|
|
|
|
SECTION("NaN and infinity dump as null")
|
|
{
|
|
CHECK(long_double_json(std::numeric_limits<long double>::quiet_NaN()).dump() == "null");
|
|
|
|
// Probe the platform's runtime behavior — `volatile` forces a runtime
|
|
// call rather than constexpr-folding to a known answer at compile time.
|
|
// Skip the infinity assertions if std::isfinite() doesn't actually
|
|
// recognize long double infinity on this platform (notably, Valgrind
|
|
// 3.22's x87 80-bit emulation reports +/-inf as a large finite value).
|
|
// TODO(rusloker): remove this guard once Valgrind's 80-bit long double
|
|
// support ships (Valgrind bug https://bugs.kde.org/show_bug.cgi?id=197915,
|
|
// ASSIGNED since 2009 — the Valgrind project tracks its bugs on
|
|
// bugs.kde.org) and the minimum supported Valgrind version contains it.
|
|
const volatile long double inf_probe = std::numeric_limits<long double>::infinity();
|
|
if (!std::isfinite(inf_probe))
|
|
{
|
|
CHECK(long_double_json(std::numeric_limits<long double>::infinity()).dump() == "null");
|
|
CHECK(long_double_json(-std::numeric_limits<long double>::infinity()).dump() == "null");
|
|
}
|
|
}
|
|
|
|
SECTION("dump output matches double for exactly-representable values")
|
|
{
|
|
auto check_same = [](long double v_ld, double v_d)
|
|
{
|
|
const long_double_json j_ld = v_ld;
|
|
const json j_d = v_d;
|
|
CHECK(j_ld.dump() == j_d.dump());
|
|
};
|
|
|
|
check_same(0.0L, 0.0);
|
|
check_same(0.5L, 0.5);
|
|
check_same(-0.5L, -0.5);
|
|
check_same(1.5L, 1.5);
|
|
check_same(-2.25L, -2.25);
|
|
check_same(1.0L, 1.0);
|
|
check_same(100.0L, 100.0);
|
|
}
|
|
}
|
|
|
|
TEST_CASE("serialization of strings (bulk fast path)")
|
|
{
|
|
// These cases exercise the SWAR bulk-copy fast path in dump_escaped and the
|
|
// internal write buffer: long runs, escapes interrupting runs, 0x7F/DEL,
|
|
// multibyte UTF-8 under both ensure_ascii settings, and payloads larger than
|
|
// the write buffer.
|
|
|
|
SECTION("long unescaped ASCII exceeds the write buffer")
|
|
{
|
|
const std::string big(3000, 'a');
|
|
const json j = big;
|
|
CHECK(j.dump() == '"' + big + '"');
|
|
CHECK(j.dump(-1, ' ', true) == '"' + big + '"');
|
|
// round-trips
|
|
CHECK(json::parse(j.dump()) == j);
|
|
}
|
|
|
|
SECTION("runs interrupted by escapes")
|
|
{
|
|
const json j = std::string(500, 'x') + "\n\"\\" + std::string(500, 'y');
|
|
const std::string out = j.dump();
|
|
CHECK(out == '"' + std::string(500, 'x') + "\\n\\\"\\\\" + std::string(500, 'y') + '"');
|
|
CHECK(json::parse(out) == j);
|
|
}
|
|
|
|
SECTION("DEL (0x7F) depends on ensure_ascii")
|
|
{
|
|
const json j = std::string("a\x7f" "b");
|
|
CHECK(j.dump(-1, ' ', false) == "\"a\x7f" "b\""); // copied verbatim
|
|
CHECK(j.dump(-1, ' ', true) == "\"a\\u007fb\""); // escaped
|
|
}
|
|
|
|
SECTION("multibyte UTF-8 under both ensure_ascii settings")
|
|
{
|
|
const json j = std::string("A\xc3\xa9\xe4\xbd\xa0\xf0\x9f\x98\x80Z"); // A é 你 😀 Z
|
|
// not escaping non-ASCII: bytes are copied through the bulk validator
|
|
CHECK(j.dump(-1, ' ', false) == "\"A\xc3\xa9\xe4\xbd\xa0\xf0\x9f\x98\x80Z\"");
|
|
// ensure_ascii: escaped (with a surrogate pair for the emoji)
|
|
CHECK(j.dump(-1, ' ', true) == "\"A\\u00e9\\u4f60\\ud83d\\ude00Z\"");
|
|
CHECK(json::parse(j.dump(-1, ' ', true)) == j);
|
|
}
|
|
|
|
SECTION("many small structural writes exceed the write buffer")
|
|
{
|
|
json arr = json::array();
|
|
for (int i = 0; i < 2000; ++i)
|
|
{
|
|
arr.push_back(i);
|
|
}
|
|
const std::string out = arr.dump();
|
|
CHECK(out.front() == '[');
|
|
CHECK(out.back() == ']');
|
|
CHECK(json::parse(out) == arr);
|
|
|
|
json obj = json::object();
|
|
for (int i = 0; i < 500; ++i)
|
|
{
|
|
obj["key" + std::to_string(i)] = i;
|
|
}
|
|
CHECK(json::parse(obj.dump()) == obj);
|
|
CHECK(json::parse(obj.dump(2)) == obj);
|
|
|
|
// an array of many empty strings emits a long run of single-character
|
|
// writes ('"', '"', ',') at shallow nesting depth, so the write buffer
|
|
// fills and flushes mid-run without the deep recursion that would
|
|
// overflow the stack on some debug builds
|
|
json many_empty = json::array();
|
|
for (int i = 0; i < 500; ++i)
|
|
{
|
|
many_empty.push_back("");
|
|
}
|
|
const std::string out2 = many_empty.dump();
|
|
CHECK(out2.size() > 1024); // spans multiple write-buffer flushes
|
|
CHECK(out2.front() == '[');
|
|
CHECK(out2.back() == ']');
|
|
CHECK(json::parse(out2) == many_empty);
|
|
}
|
|
|
|
SECTION("invalid UTF-8 handling is unaffected by the fast path")
|
|
{
|
|
const json j = std::string("valid\xff" "more");
|
|
CHECK_THROWS_WITH_AS(j.dump(), "[json.exception.type_error.316] invalid UTF-8 byte at index 5: 0xFF", json::type_error&);
|
|
CHECK(j.dump(-1, ' ', false, json::error_handler_t::replace) == "\"valid\xef\xbf\xbd" "more\"");
|
|
CHECK(j.dump(-1, ' ', true, json::error_handler_t::replace) == "\"valid\\ufffdmore\"");
|
|
CHECK(j.dump(-1, ' ', false, json::error_handler_t::ignore) == "\"validmore\"");
|
|
}
|
|
}
|
|
|
|
TEST_CASE("indentation is written straight into the write buffer")
|
|
{
|
|
// put_indent() memsets the indentation into the write buffer instead of
|
|
// copying it out of a pre-grown indentation string. These cases cover an
|
|
// indentation wider than the buffer, a non-space indentation character, and
|
|
// nesting deep enough that the accumulated indentation spans several
|
|
// buffer-fulls - the situations the old grow-a-string approach got wrong.
|
|
|
|
SECTION("indent_step wider than the write buffer")
|
|
{
|
|
const json j = {{"a", 1}};
|
|
// 2000 > the 1024-byte write buffer, and > the 512 the indentation
|
|
// string used to start at
|
|
CHECK(j.dump(2000) == "{\n" + std::string(2000, ' ') + "\"a\": 1\n}");
|
|
// several whole buffer-fulls, so the buffer is refilled once and then
|
|
// flushed repeatedly
|
|
CHECK(j.dump(5000) == "{\n" + std::string(5000, ' ') + "\"a\": 1\n}");
|
|
CHECK(j.dump(5000, '\t') == "{\n" + std::string(5000, '\t') + "\"a\": 1\n}");
|
|
// an exact multiple of the buffer size
|
|
CHECK(j.dump(4096) == "{\n" + std::string(4096, ' ') + "\"a\": 1\n}");
|
|
}
|
|
|
|
SECTION("a non-space indentation character is used throughout")
|
|
{
|
|
const json j = {{"a", 1}};
|
|
// 600 is past the point where the indentation used to be grown, which
|
|
// is where a hard-coded space would have shown up
|
|
CHECK(j.dump(600, '\t') == "{\n" + std::string(600, '\t') + "\"a\": 1\n}");
|
|
CHECK(j.dump(3, '.') == "{\n...\"a\": 1\n}");
|
|
}
|
|
|
|
SECTION("accumulated indentation spans several buffer-fulls")
|
|
{
|
|
// five levels deep at 400 per level: the innermost value is indented by
|
|
// 2000 characters, reached in steps that each straddle the buffer end
|
|
json j = json::array({1});
|
|
for (int i = 0; i < 4; ++i)
|
|
{
|
|
j = json::array({j});
|
|
}
|
|
|
|
const std::string out = j.dump(400);
|
|
CHECK(out.find(std::string("\n") + std::string(2000, ' ') + "1\n") != std::string::npos);
|
|
CHECK(json::parse(out) == j);
|
|
}
|
|
|
|
SECTION("indentation is unchanged for ordinary widths")
|
|
{
|
|
const json j = {{"a", {1, 2}}, {"b", nullptr}};
|
|
CHECK(j.dump(2) == "{\n \"a\": [\n 1,\n 2\n ],\n \"b\": null\n}");
|
|
CHECK(j.dump(0) == "{\n\"a\": [\n1,\n2\n],\n\"b\": null\n}");
|
|
}
|
|
}
|
|
|
|
TEST_CASE("serialization of deeply nested values")
|
|
{
|
|
// dump() descends into a bounded number of levels and writes out whatever
|
|
// is nested deeper than that without the call stack; see
|
|
// https://github.com/nlohmann/json/issues/5387
|
|
|
|
SECTION("nested deeper than the call stack could follow")
|
|
{
|
|
// parsing is iterative, so building these costs little
|
|
const std::size_t depth = 100000;
|
|
|
|
const std::string array_text = std::string(depth, '[') + '0' + std::string(depth, ']');
|
|
CHECK(json::parse(array_text).dump() == array_text);
|
|
|
|
std::string object_text;
|
|
object_text.reserve((6 * depth) + 1);
|
|
for (std::size_t i = 0; i < depth; ++i)
|
|
{
|
|
object_text += "{\"a\":";
|
|
}
|
|
object_text += '1';
|
|
object_text.append(depth, '}');
|
|
CHECK(json::parse(object_text).dump() == object_text);
|
|
}
|
|
|
|
SECTION("depths around the bound of the descent")
|
|
{
|
|
// Cover every depth around the bound, so that the two ways of writing a
|
|
// value are known to meet cleanly - wherever the bound is set.
|
|
for (std::size_t d = 1; d <= 300; ++d)
|
|
{
|
|
CAPTURE(d);
|
|
|
|
const std::string array_text = std::string(d, '[') + '7' + std::string(d, ']');
|
|
CHECK(json::parse(array_text).dump() == array_text);
|
|
|
|
std::string object_text;
|
|
for (std::size_t i = 0; i < d; ++i)
|
|
{
|
|
object_text += "{\"k\":";
|
|
}
|
|
object_text += '7';
|
|
object_text.append(d, '}');
|
|
CHECK(json::parse(object_text).dump() == object_text);
|
|
}
|
|
}
|
|
|
|
SECTION("pretty-printing across the bound")
|
|
{
|
|
for (std::size_t d = 120; d <= 140; ++d)
|
|
{
|
|
CAPTURE(d);
|
|
|
|
const json j = json::parse(std::string(d, '[') + '7' + std::string(d, ']'));
|
|
|
|
std::string expected;
|
|
for (std::size_t i = 0; i < d; ++i)
|
|
{
|
|
expected += std::string(2 * i, ' ') + "[\n";
|
|
}
|
|
expected += std::string(2 * d, ' ') + '7';
|
|
for (std::size_t i = d; i > 0; --i)
|
|
{
|
|
expected += '\n' + std::string(2 * (i - 1), ' ') + ']';
|
|
}
|
|
|
|
CHECK(j.dump(2) == expected);
|
|
}
|
|
}
|
|
|
|
SECTION("an empty container below the bound")
|
|
{
|
|
// an empty container is written out in full and never descended into,
|
|
// so it must not gain a newline when it is reached iteratively
|
|
for (std::size_t d = 125; d <= 135; ++d)
|
|
{
|
|
CAPTURE(d);
|
|
|
|
const std::string compact = std::string(d, '[') + "[]" + std::string(d, ']');
|
|
CHECK(json::parse(compact).dump() == compact);
|
|
|
|
const std::string with_object = std::string(d, '[') + "{}" + std::string(d, ']');
|
|
CHECK(json::parse(with_object).dump() == with_object);
|
|
}
|
|
}
|
|
}
|