mirror of
https://github.com/nlohmann/json.git
synced 2026-09-29 03:00:31 +00:00
Add JSON_PRECISE_STREAM_POSITION to leave the character that terminates a number in the stream (#5344)
* docs: qualify the operator>> stream positioning guarantee operator>>'s notes state that it leaves the stream positioned right after the parsed value, so that concatenated JSON values can be read back to back. That does not hold when the value is a number: a number is only terminated by the character that follows it, and the lexer's unget() is simulated (it rewinds only the lexer's own bookkeeping), so that character stays consumed from the stream. Document the actual behaviour: the guarantee holds for all value types except numbers, which must be followed by whitespace. Also qualify the cross-reference on the JSON Lines page, which repeated the unqualified claim. Documentation only; the behaviour itself is tracked in #5340. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * fix: restore the character that terminates a number (#5340) operator>> is documented to leave the stream positioned right after the parsed value, so that concatenated JSON values can be read back to back. That did not hold for numbers: a number is only terminated by the character following it, and lexer::scan_number() reads that character and calls unget() -- which is simulated and rewinds only the lexer's own bookkeeping. input_stream_adapter consumes via sbumpc() with no matching sungetc(), so the terminating character stayed consumed and the next extraction started one byte too late ('1true' left the stream at 'rue'). Propagating unget() to the adapter directly does not work: next_unget makes the following get() replay the cached character, so the terminator would be delivered twice. Instead, restore the still-pending character once at the end of a non-strict parse, where the input is handed back to the caller: - input_stream_adapter gains unget_character() (sungetc()) and advertises it via supports_unget, detected the same way as supports_seek. - lexer::restore_pending_unget() turns a pending simulated unget of a real (non-EOF) character into a real one and clears next_unget so the character is not also replayed. It is a no-op for adapters that cannot unget, and reports failure when sungetc() fails, in which case the input is left as it was before. - parser calls it on the three non-strict paths, i.e. for operator>> and sax_parse(strict = false). Strict parse()/accept() are unaffected: they require the input to end after the value, so the character is consumed by the end-of-input check anyway. Parse error messages and reported positions are unchanged. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * tests: fix CI failures in the #5340 test helpers Four CI failures, all in the new test code: - GCC (-Werror=useless-cast): drop the `json(...)` wrapper around `json::parse(...)`, which already returns a `json`. - GCC (-Werror=unused-result): assign the discarded `json::parse()` result to a dummy, the idiom used elsewhere in the test suite, and catch `json::parse_error&` for consistency. - clang-tidy (google-default-arguments): remove the default argument from the `pbackfail()` override; `sungetc()` supplies the base declaration's default. - MSVC (bad allocation): `no_putback_streambuf::underflow()` set a one-character get area without advancing `m_pos`, so an implementation whose `istream::get` peeks before it bumps re-read the same character forever. Keep no get area at all: `underflow()` peeks, `uflow()` consumes, and `sungetc()` still always lands in `pbackfail()`, which is what the test needs. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * fix: leave the character that terminates a number in the input Read the character following a number without consuming it, instead of consuming it and putting it back. input_stream_adapter now peeks with sgetc() and only steps over the character when the next one is requested or when the adapter is destroyed, so releasing it cannot fail - no putback position is required from the streambuf. Suggested by gregmarr in #5344. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: match the version history wording to the peek-based fix Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: drop the whitespace-separator caveat from the parsing pages The caveat added in #5343 describes the behavior this branch fixes: a number no longer consumes the character that terminates it, so concatenated values need no separator. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * refactor: split the strict and non-strict paths in parser Folding the release_lookahead() call into the existing strict check left the "in strict mode" comment on an else-if branch, and made the strict condition in sax_parse() redundant with the branch it followed. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Put the stream position fix behind JSON_PRECISE_STREAM_POSITION Leaving the character that terminates a number in the stream is observable: reading "1,2,3" with repeated operator>> works today only because the comma after each number is swallowed, and std::getline after a number skips the line break. Both break with the fix, so make it opt-in for 3.x, as suggested by @gregmarr in the review. - JSON_PRECISE_STREAM_POSITION (default 0) selects the peek-based input_stream_adapter. Without it, the adapter is the consuming one from develop and has no supports_lookahead, so lexer::release_lookahead() and the parser's calls to it compile to nothing. - The macro changes input_stream_adapter's layout and member functions, so it gets the ABI tag _psp, after _bics. The ABI config tests, the natvis generator, and nlohmann_json.natvis (regenerated) know the tag. - The tests for the fix move to unit-precise-stream-position.cpp, which defines the macro itself and runs in every build, and gain the two cases above. unit-deserialization.cpp pins the default behavior instead. - The docs describe the default behavior again and point to the new macro page; version history says "added in 3.13.0, planned default in 4.0.0". Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>
This commit is contained in:
Niels Lohmann
@@ -36,6 +36,10 @@ TEST_CASE("default namespace")
|
||||
expected += "_bics";
|
||||
#endif
|
||||
|
||||
#if JSON_PRECISE_STREAM_POSITION
|
||||
expected += "_psp";
|
||||
#endif
|
||||
|
||||
expected += "_v" STRINGIZE(NLOHMANN_JSON_VERSION_MAJOR);
|
||||
expected += "_" STRINGIZE(NLOHMANN_JSON_VERSION_MINOR);
|
||||
expected += "_" STRINGIZE(NLOHMANN_JSON_VERSION_PATCH) "::basic_json";
|
||||
|
||||
@@ -37,6 +37,10 @@ TEST_CASE("default namespace without version component")
|
||||
expected += "_bics";
|
||||
#endif
|
||||
|
||||
#if JSON_PRECISE_STREAM_POSITION
|
||||
expected += "_psp";
|
||||
#endif
|
||||
|
||||
expected += "::basic_json";
|
||||
|
||||
// fallback for Clang
|
||||
|
||||
@@ -25,6 +25,7 @@ using nlohmann::json;
|
||||
#include <iostream>
|
||||
#include <iterator>
|
||||
#include <sstream>
|
||||
#include <string>
|
||||
#include <valarray>
|
||||
|
||||
#if defined(_WIN32)
|
||||
@@ -1233,6 +1234,57 @@ TEST_CASE("deserialization")
|
||||
}
|
||||
}
|
||||
|
||||
SECTION("stream position after extraction without JSON_PRECISE_STREAM_POSITION (#5340)")
|
||||
{
|
||||
// By default, the character that terminates a number is consumed, so
|
||||
// the stream is left one byte too far after a number (and only after a
|
||||
// number). JSON_PRECISE_STREAM_POSITION changes this; see
|
||||
// unit-precise-stream-position.cpp. These checks pin the default.
|
||||
const auto remaining = [](std::istream & is)
|
||||
{
|
||||
return std::string(std::istreambuf_iterator<char>(is), std::istreambuf_iterator<char>());
|
||||
};
|
||||
|
||||
SECTION("the character after a number is consumed")
|
||||
{
|
||||
std::istringstream ss("1true");
|
||||
json j;
|
||||
ss >> j;
|
||||
CHECK(j == 1);
|
||||
CHECK(remaining(ss) == "rue");
|
||||
}
|
||||
|
||||
SECTION("the character after other values is not consumed")
|
||||
{
|
||||
std::istringstream ss("[1]true");
|
||||
json j;
|
||||
ss >> j;
|
||||
CHECK(j == json::parse("[1]"));
|
||||
CHECK(remaining(ss) == "true");
|
||||
}
|
||||
|
||||
SECTION("comma-separated numbers can be read one by one")
|
||||
{
|
||||
std::istringstream ss("1,2,3");
|
||||
json j1, j2, j3;
|
||||
ss >> j1 >> j2 >> j3;
|
||||
CHECK(j1 == 1);
|
||||
CHECK(j2 == 2);
|
||||
CHECK(j3 == 3);
|
||||
}
|
||||
|
||||
SECTION("std::getline after a number skips the line break")
|
||||
{
|
||||
std::istringstream ss("42\nfoo");
|
||||
json j;
|
||||
std::string line;
|
||||
ss >> j;
|
||||
std::getline(ss, line);
|
||||
CHECK(j == 42);
|
||||
CHECK(line == "foo");
|
||||
}
|
||||
}
|
||||
|
||||
// build with C++20
|
||||
// JSON_HAS_CPP_20
|
||||
#if defined(__cpp_char8_t)
|
||||
|
||||
@@ -0,0 +1,237 @@
|
||||
// __ _____ _____ _____
|
||||
// __| | __| | | | JSON for Modern C++ (supporting code)
|
||||
// | | |__ | | | | | | version 3.12.0
|
||||
// |_____|_____|_____|_|___| https://github.com/nlohmann/json
|
||||
//
|
||||
// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
|
||||
// SPDX-License-Identifier: MIT
|
||||
|
||||
#include "doctest_compatibility.h"
|
||||
|
||||
// This file tests the opt-in JSON_PRECISE_STREAM_POSITION, so it defines the
|
||||
// macro itself rather than relying on a -D flag, and runs in every build. The
|
||||
// default behavior is pinned in unit-deserialization.cpp.
|
||||
#ifdef JSON_PRECISE_STREAM_POSITION
|
||||
#undef JSON_PRECISE_STREAM_POSITION
|
||||
#endif
|
||||
|
||||
#define JSON_PRECISE_STREAM_POSITION 1
|
||||
|
||||
#include <nlohmann/json.hpp>
|
||||
using nlohmann::json;
|
||||
|
||||
#include <cstddef>
|
||||
#include <sstream>
|
||||
#include <streambuf>
|
||||
#include <string>
|
||||
#include <utility>
|
||||
#include <vector>
|
||||
|
||||
#define STRINGIZE_EX(x) #x
|
||||
#define STRINGIZE(x) STRINGIZE_EX(x)
|
||||
|
||||
namespace
|
||||
{
|
||||
// A streambuf that keeps no get area at all and refuses every putback: with an
|
||||
// empty get area, sungetc() always ends up in pbackfail(). Used to check that
|
||||
// the character terminating a number is left in the input without relying on
|
||||
// the streambuf being able to put a consumed character back.
|
||||
class no_putback_streambuf : public std::streambuf
|
||||
{
|
||||
public:
|
||||
explicit no_putback_streambuf(std::string s) : m_data(std::move(s)) {}
|
||||
|
||||
protected:
|
||||
// peek at the next character without consuming it
|
||||
int_type underflow() override
|
||||
{
|
||||
if (m_pos >= m_data.size())
|
||||
{
|
||||
return traits_type::eof();
|
||||
}
|
||||
return traits_type::to_int_type(m_data[m_pos]);
|
||||
}
|
||||
|
||||
// consume the next character
|
||||
int_type uflow() override
|
||||
{
|
||||
if (m_pos >= m_data.size())
|
||||
{
|
||||
return traits_type::eof();
|
||||
}
|
||||
return traits_type::to_int_type(m_data[m_pos++]);
|
||||
}
|
||||
|
||||
int_type pbackfail(int_type /*c*/) override
|
||||
{
|
||||
return traits_type::eof();
|
||||
}
|
||||
|
||||
private:
|
||||
std::string m_data;
|
||||
std::size_t m_pos = 0;
|
||||
};
|
||||
|
||||
// read the characters that are left in a stream
|
||||
std::string remaining(std::istream& is)
|
||||
{
|
||||
std::string result;
|
||||
char c = 0;
|
||||
while (is.get(c))
|
||||
{
|
||||
result += c;
|
||||
}
|
||||
return result;
|
||||
}
|
||||
} // namespace
|
||||
|
||||
TEST_CASE("JSON_PRECISE_STREAM_POSITION")
|
||||
{
|
||||
SECTION("the macro is part of the ABI tag")
|
||||
{
|
||||
const std::string ns = STRINGIZE(NLOHMANN_JSON_NAMESPACE);
|
||||
// other tags may come before it, e.g. json_abi_diag_psp
|
||||
CHECK(ns.find("_psp") != std::string::npos);
|
||||
}
|
||||
|
||||
SECTION("a number does not consume the character that terminates it")
|
||||
{
|
||||
// a number is only terminated by the character following it; that
|
||||
// character must be given back so the stream is positioned right
|
||||
// after the value
|
||||
const std::vector<std::pair<std::string, std::string>> tests =
|
||||
{
|
||||
{"1true", "true"},
|
||||
{"1[2]", "[2]"},
|
||||
{"1{}", "{}"},
|
||||
{R"(1"a")", R"("a")"},
|
||||
{"1 true", " true"},
|
||||
{"12,", ","},
|
||||
{"-0.5e3x", "x"},
|
||||
{"1null", "null"}
|
||||
};
|
||||
|
||||
for (const auto& test : tests)
|
||||
{
|
||||
CAPTURE(test.first);
|
||||
std::istringstream ss(test.first);
|
||||
json j;
|
||||
ss >> j;
|
||||
CHECK(j == json::parse(test.first.substr(0, test.first.size() - test.second.size())));
|
||||
CHECK(remaining(ss) == test.second);
|
||||
}
|
||||
}
|
||||
|
||||
SECTION("values that are self-delimiting are unaffected")
|
||||
{
|
||||
const std::vector<std::pair<std::string, std::string>> tests =
|
||||
{
|
||||
{"truefalse", "false"},
|
||||
{"[1][2]", "[2]"},
|
||||
{R"({"a":1}{"b":2})", R"({"b":2})"},
|
||||
{R"("a""b")", R"("b")"},
|
||||
{"null null", " null"}
|
||||
};
|
||||
|
||||
for (const auto& test : tests)
|
||||
{
|
||||
CAPTURE(test.first);
|
||||
std::istringstream ss(test.first);
|
||||
json j;
|
||||
ss >> j;
|
||||
CHECK(remaining(ss) == test.second);
|
||||
}
|
||||
}
|
||||
|
||||
SECTION("a number at the end of the input leaves nothing behind")
|
||||
{
|
||||
for (const std::string s :
|
||||
{"1", "12", "-3.5e2", " 7 "
|
||||
})
|
||||
{
|
||||
CAPTURE(s);
|
||||
std::istringstream ss(s);
|
||||
json j;
|
||||
ss >> j;
|
||||
CHECK(remaining(ss).find_first_not_of(" \t\n\r") == std::string::npos);
|
||||
}
|
||||
}
|
||||
|
||||
SECTION("repeated extraction of concatenated values")
|
||||
{
|
||||
std::istringstream ss(R"(1true[2]3"x"{"a":4}5)");
|
||||
const std::vector<json> expected =
|
||||
{
|
||||
json(1), json(true), json::parse("[2]"), json(3),
|
||||
json("x"), json::parse(R"({"a":4})"), json(5)
|
||||
};
|
||||
|
||||
for (const auto& e : expected)
|
||||
{
|
||||
json j;
|
||||
ss >> j;
|
||||
CHECK(j == e);
|
||||
}
|
||||
}
|
||||
|
||||
SECTION("differences to the default behavior")
|
||||
{
|
||||
// both of these work by accident without the macro, because the
|
||||
// character after a number is swallowed; see unit-deserialization.cpp
|
||||
|
||||
SECTION("a separator after a number is not skipped")
|
||||
{
|
||||
std::istringstream ss("1,2");
|
||||
json j;
|
||||
ss >> j;
|
||||
CHECK(j == 1);
|
||||
CHECK_THROWS_AS(ss >> j, json::parse_error&);
|
||||
}
|
||||
|
||||
SECTION("std::getline after a number sees the line break")
|
||||
{
|
||||
std::istringstream ss("42\nfoo");
|
||||
json j;
|
||||
std::string line;
|
||||
ss >> j;
|
||||
std::getline(ss, line);
|
||||
CHECK(j == 42);
|
||||
CHECK(line.empty());
|
||||
std::getline(ss, line);
|
||||
CHECK(line == "foo");
|
||||
}
|
||||
}
|
||||
|
||||
SECTION("sax_parse with strict == false")
|
||||
{
|
||||
std::istringstream ss("1true");
|
||||
json j;
|
||||
nlohmann::detail::json_sax_dom_parser<json, nlohmann::detail::input_stream_adapter> sdp(j, true);
|
||||
CHECK(json::sax_parse(ss, &sdp, nlohmann::detail::input_format_t::json, false));
|
||||
CHECK(j == 1);
|
||||
CHECK(remaining(ss) == "true");
|
||||
}
|
||||
|
||||
SECTION("strict parsing still rejects trailing data")
|
||||
{
|
||||
std::istringstream ss("1true");
|
||||
json _;
|
||||
CHECK_THROWS_WITH_AS(_ = json::parse(ss),
|
||||
"[json.exception.parse_error.101] parse error at line 1, column 5: syntax error while parsing value - unexpected true literal; expected end of input", json::parse_error&);
|
||||
|
||||
std::istringstream ss2("1true");
|
||||
CHECK_FALSE(json::accept(ss2));
|
||||
}
|
||||
|
||||
SECTION("a streambuf that cannot put back is not needed")
|
||||
{
|
||||
// the terminating character is never consumed, so no putback
|
||||
// position is required
|
||||
no_putback_streambuf buf("1true");
|
||||
std::istream is(&buf);
|
||||
json j;
|
||||
is >> j;
|
||||
CHECK(j == json(1));
|
||||
CHECK(remaining(is) == "true");
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user