Compare commits

..
Author SHA1 Message Date
Niels Lohmann 1d4dfc3af0 Report the parse errors of the position test via SAX
The test caught json::parse_error, so it aborted in the
ci_test_noexceptions build, where JSON_NOEXCEPTION turns the throw into
std::abort(). sax_parse() hands the error position and the exception
object to parse_error() without throwing, so the test now records them
there and runs in every configuration.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-13 11:22:48 +02:00
Niels Lohmann db3d98de07 Move the std::vector<std::byte> binary type tests out of unit-regression3
The #4804/#4209 tests instantiated a second basic_json specialization
(with_binary_t<std::vector<std::byte>>), including its parser and CBOR
reader. With clang 18, the C++20 object of unit-regression3.cpp had
32656 COFF sections on develop; the lazy position helpers pushed it to
32975, past what the MinGW linker of the Windows clang jobs handles
("relocation truncated to fit: IMAGE_REL_AMD64_REL32 against .rdata").
unit-custom-binary-type.cpp already instantiates the same type, so the
tests move there; unit-regression3.cpp drops to 27616 sections.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-13 09:40:36 +02:00
Niels Lohmann 3dbed2a0cb Derive the lexer position lazily for seekable input
The lexer counted the read position on every character: two increments
and a newline check per get(), all on members kept in memory. For
seekable input (pointers, strings, other random-access byte ranges) the
position can be derived instead: chars_read_total is the adapter's read
count (including reads at EOF) minus a pending unget, and line and
column are recomputed from the consumed input only when get_position()
is called, i.e. on parse errors. JSON_DIAGNOSTIC_POSITIONS uses the new
O(1) get_chars_read_total(). Stream, file, wide-string and user-defined
adapters keep the eager counters.

Parsing pretty-printed input gets 8-12% faster and accept() 12-27%
(canada, citm_catalog, twitter; clang 21 arm64 and GCC 13 x86-64).
Error messages, byte offsets, SAX positions and diagnostic positions are
unchanged (differential comparison against develop on 548k inputs).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-13 08:52:49 +02:00
14 changed files with 399 additions and 257 deletions

No files matched your search

+7 -19
View File
@@ -610,8 +610,9 @@ foreach(SRC_FILE ${SRC_FILES})
add_executable(single_${RELATIVE_SRC_FILE} EXCLUDE_FROM_ALL ${PROJECT_BINARY_DIR}/src_single/${RELATIVE_SRC_FILE}.cpp)
target_include_directories(single_${RELATIVE_SRC_FILE} PRIVATE ${PROJECT_SOURCE_DIR}/include)
target_compile_features(single_${RELATIVE_SRC_FILE} PRIVATE cxx_std_11)
if(RELATIVE_SRC_FILE STREQUAL "json")
# see below: report the diagnostics of json.hpp without --error, so they do not fail the build
if(RELATIVE_SRC_FILE STREQUAL "json" OR RELATIVE_SRC_FILE STREQUAL "json_literals")
# see below: report the diagnostics of json.hpp and json_literals.hpp without --error, so they
# do not fail the build
set_property(TARGET single_${RELATIVE_SRC_FILE} PROPERTY CXX_INCLUDE_WHAT_YOU_USE ${IWYU_TOOL} -Xiwyu --max_line_length=300)
else()
set_property(TARGET single_${RELATIVE_SRC_FILE} PROPERTY CXX_INCLUDE_WHAT_YOU_USE "${iwyu_path_and_options}")
@@ -625,31 +626,18 @@ foreach(SRC_FILE ${SRC_FILES})
# reporting its diagnostics (informational, via CXX_INCLUDE_WHAT_YOU_USE above) but exclude it
# from the hard gate below so a fresh IWYU/compiler combination does not fail this target on a
# nondeterministic suggestion for a header that already re-exports everything on purpose.
if(NOT RELATIVE_SRC_FILE STREQUAL "json")
# json_literals.hpp and json.hpp include each other on purpose (json.hpp includes it at its end
# unless JSON_NO_AUTOMATIC_UDLS is defined), and IWYU, not following the cycle, suggests replacing
# json.hpp with json_fwd.hpp although the literals need the complete basic_json; exclude it, too.
if(NOT RELATIVE_SRC_FILE STREQUAL "json" AND NOT RELATIVE_SRC_FILE STREQUAL "json_literals")
list(APPEND single_binaries_tus src_single/${RELATIVE_SRC_FILE}.cpp)
endif()
endforeach()
# The checks above look at each header on its own. Client code that includes only <nlohmann/json.hpp>
# must not be told by include analyzers to include another nlohmann header for what it uses
# (issue #5812): json.hpp marks those includes "IWYU pragma: export", and both IWYU and clang-tidy's
# misc-include-cleaner (disabled in .clang-tidy for the library itself) check these clients.
file(GLOB CONSUMER_FILES ${PROJECT_SOURCE_DIR}/tests/iwyu_consumer/*.cpp)
foreach(CONSUMER_FILE ${CONSUMER_FILES})
get_filename_component(CONSUMER_NAME "${CONSUMER_FILE}" NAME_WE)
add_executable(iwyu_consumer_${CONSUMER_NAME} EXCLUDE_FROM_ALL ${CONSUMER_FILE})
target_include_directories(iwyu_consumer_${CONSUMER_NAME} PRIVATE ${PROJECT_SOURCE_DIR}/include)
target_compile_features(iwyu_consumer_${CONSUMER_NAME} PRIVATE cxx_std_11)
list(APPEND single_binaries iwyu_consumer_${CONSUMER_NAME})
list(APPEND single_binaries_tus ${CONSUMER_FILE})
endforeach()
if(IWYU_TOOL_PY)
add_custom_target(ci_single_binaries
DEPENDS ${single_binaries}
COMMAND ${IWYU_TOOL_PY} -p ${PROJECT_BINARY_DIR} ${single_binaries_tus} -- ${iwyu_options}
COMMAND ${CLANG_TIDY_TOOL} -p ${PROJECT_BINARY_DIR} --checks=-*,misc-include-cleaner --warnings-as-errors=* ${CONSUMER_FILES}
VERBATIM
COMMENT "Check if headers are self-contained"
)
else()
+3 -1
View File
@@ -185,7 +185,9 @@ The lexer detects two optional extensions at compile time. Only `iterator_input_
random-access input of single-byte characters:
- `supports_seek`, `get_consumed_count()`, and `copy_consumed_range()` let the lexer reconstruct already consumed input
for error messages instead of copying every character it reads.
for error messages instead of copying every character it reads. With `get_read_count()` and
`count_consumed_lines()`, the lexer also derives the read position from the adapter and counts lines only when an
error needs them, instead of counting characters and lines on every read.
- `supports_bulk_scan`, `bulk_data()`, `bulk_remaining()`, and `bulk_skip()` let the lexer scan strings directly in
contiguous memory, several bytes at a time.
@@ -8,12 +8,12 @@
#pragma once
#include <algorithm> // min
#include <algorithm> // count, find, min
#include <array> // array
#include <cstddef> // size_t
#include <cstdint> // uint32_t
#include <cstring> // strlen
#include <iterator> // begin, end, iterator_traits, random_access_iterator_tag, distance, next
#include <iterator> // begin, end, iterator_traits, random_access_iterator_tag, distance, next, reverse_iterator
#include <streambuf> // streambuf
#include <string> // string, char_traits
#include <type_traits> // enable_if, is_base_of, is_pointer, is_integral, remove_pointer
@@ -297,6 +297,7 @@ class iterator_input_adapter
return result;
}
++eof_reads;
return char_traits<char_type>::eof();
}
@@ -306,6 +307,26 @@ class iterator_input_adapter
return static_cast<std::size_t>(std::distance(begin, current));
}
// number of get_character() calls so far, including those that returned
// EOF; the lexer derives its position from it (see lexer::lazy_position)
std::size_t get_read_count() const
{
return get_consumed_count() + eof_reads;
}
// number of newlines in the consumed range [0, last_index); @a line_start
// is set to the index after the last of them (0 if there is none); only
// valid when supports_seek
std::size_t count_consumed_lines(std::size_t last_index, std::size_t& line_start) const
{
const auto to = std::next(begin, static_cast<typename std::iterator_traits<IteratorType>::difference_type>(last_index));
const auto newline = char_traits<char_type>::to_char_type('\n');
using reverse_it = std::reverse_iterator<IteratorType>;
const auto last_newline = std::find(reverse_it(to), reverse_it(begin), newline);
line_start = static_cast<std::size_t>(std::distance(begin, last_newline.base()));
return static_cast<std::size_t>(std::count(begin, last_newline.base(), newline));
}
// append the already-consumed characters in the half-open range
// [first_index, last_index) to @a out; only valid when supports_seek
template<typename ContainerType>
@@ -422,6 +443,8 @@ class iterator_input_adapter
IteratorType begin;
IteratorType current;
SentinelType end;
/// number of get_character() calls that returned EOF
std::size_t eof_reads = 0;
template<typename BaseInputAdapter, size_t T>
friend struct wide_string_input_helper;
+3 -3
View File
@@ -217,7 +217,7 @@ struct diagnostic_positions
{
// Lexer has read the first character of the container, so
// subtract 1 from the position to get the correct start position.
v.start_position = lexer->get_position() - 1;
v.start_position = lexer->get_chars_read_total() - 1;
}
}
@@ -231,7 +231,7 @@ struct diagnostic_positions
if (lexer)
{
// Lexer's position is past the closing brace or bracket, so set that as the end position.
v.end_position = lexer->get_position();
v.end_position = lexer->get_chars_read_total();
}
}
@@ -247,7 +247,7 @@ struct diagnostic_positions
// Lexer has read past the current field value, so set the end position to the current position.
// The start position will be set below based on the length of the string representation
// of the value.
v.end_position = lexer->get_position();
v.end_position = lexer->get_chars_read_total();
switch (v.type())
{
+95 -20
View File
@@ -186,6 +186,15 @@ class lexer : public lexer_base<BasicJsonType>
static constexpr bool lazy_token_string =
input_adapter_supports_seek<InputAdapterType>(is_detected<detect_supports_seek, InputAdapterType> {});
/// whether the read position is derived from the input adapter on demand
/// instead of being counted on every scanned character. Seekable adapters
/// know how much input they consumed and can revisit it, so
/// chars_read_total is the adapter's read count (minus a pending unget),
/// and line and column are recomputed from the consumed input only when
/// get_position() is called (errors). Counting per character kept two
/// increments and a newline check in the hot path of every get().
static constexpr bool lazy_position = lazy_token_string;
/// whether a simulated unget can be passed on to the input adapter, which
/// then leaves the character in the input; see
/// input_adapter_supports_lookahead
@@ -195,7 +204,9 @@ class lexer : public lexer_base<BasicJsonType>
/// whether string scanning may bulk-consume runs of ordinary characters
/// directly from a contiguous input buffer (SWAR fast path). This requires
/// the token to be reconstructible lazily (lazy_token_string), so bypassing
/// the per-character capture in get() cannot lose error diagnostics.
/// the per-character capture in get() cannot lose error diagnostics. It
/// also implies lazy_position, so the bulk paths need not advance the
/// position counters.
static constexpr bool bulk_scan =
lazy_token_string
&& input_adapter_supports_bulk_scan<InputAdapterType>(is_detected<detect_supports_bulk_scan, InputAdapterType> {});
@@ -331,10 +342,6 @@ class lexer : public lexer_base<BasicJsonType>
}
token_buffer.append(reinterpret_cast<const typename string_t::value_type*>(data), pos);
ia.bulk_skip(pos);
// the run contains no newline (all bytes < 0x20 are treated as special),
// so only the flat character counters advance
position.chars_read_total += pos;
position.chars_read_current_line += pos;
}
/// streaming input: no bulk fast path
@@ -1567,8 +1574,6 @@ scan_number_done:
if (JSON_HEDLEY_LIKELY(integer_result != token_type::uninitialized))
{
ia.bulk_skip(len - 1);
position.chars_read_total += (len - 1);
position.chars_read_current_line += (len - 1);
return integer_result;
}
// The value does not fit an integer, so this token converts as a
@@ -1585,8 +1590,6 @@ scan_number_done:
decimal_point_position = dot_index;
ia.bulk_skip(len - 1);
position.chars_read_total += (len - 1);
position.chars_read_current_line += (len - 1);
return convert_number(number_type, mantissa_end);
}
@@ -1637,7 +1640,7 @@ scan_number_done:
#if JSON_DIAGNOSTIC_POSITIONS
// the first character of the token has already been read, hence the -1
token_start_position = position.chars_read_total - 1;
token_start_position = get_chars_read_total() - 1;
#endif
note_token_start(std::integral_constant<bool, lazy_token_string> {});
@@ -1690,6 +1693,15 @@ scan_number_done:
/// per-character position counters (line-count-on-'\n' bookkeeping is
/// handled afterwards, in track_after_read(), once `current` is known)
void advance_position() noexcept
{
advance_position_impl(std::integral_constant<bool, lazy_position> {});
}
/// seekable adapter: the position is derived from the adapter on demand
void advance_position_impl(std::true_type /*lazy*/) const noexcept {}
/// streaming adapter: count the character
void advance_position_impl(std::false_type /*lazy*/) noexcept
{
++position.chars_read_total;
++position.chars_read_current_line;
@@ -1703,7 +1715,16 @@ scan_number_done:
// seekable adapters reconstruct the token lazily on error (see
// get_token_string), so the eager per-character copy is skipped
capture_char(std::integral_constant<bool, lazy_token_string> {});
track_newline(std::integral_constant<bool, lazy_position> {});
return current;
}
/// seekable adapter: lines are counted on demand in get_position()
void track_newline(std::true_type /*lazy*/) const noexcept {}
/// streaming adapter: start a new line after reading a newline
void track_newline(std::false_type /*lazy*/) noexcept
{
if (current == '\n')
{
++position.lines_read;
@@ -1712,8 +1733,6 @@ scan_number_done:
chars_read_before_newline = position.chars_read_current_line;
position.chars_read_current_line = 0;
}
return current;
}
/*!
@@ -1757,12 +1776,22 @@ scan_number_done:
We implement unget by setting variable next_unget to true. The input is not
changed - we just simulate ungetting by modifying chars_read_total,
chars_read_current_line, and token_string. The next call to get() will
behave as if the unget character is read again.
behave as if the unget character is read again. With lazy_position, the
pending next_unget is all get_chars_read_total() needs.
*/
void unget()
{
next_unget = true;
unget_position(std::integral_constant<bool, lazy_position> {});
uncapture_char(std::integral_constant<bool, lazy_token_string> {});
}
/// seekable adapter: get_chars_read_total() accounts for next_unget
void unget_position(std::true_type /*lazy*/) const noexcept {}
/// streaming adapter: step the counters back over the ungotten character
void unget_position(std::false_type /*lazy*/) noexcept
{
--position.chars_read_total;
// in case we "unget" a newline, we have to also decrement the lines_read
@@ -1785,8 +1814,6 @@ scan_number_done:
{
--position.chars_read_current_line;
}
uncapture_char(std::integral_constant<bool, lazy_token_string> {});
}
/// adapter without lookahead: nothing to do (see release_lookahead)
@@ -1857,10 +1884,21 @@ scan_number_done:
// diagnostics
/////////////////////
/// return position of last read token
constexpr position_t get_position() const noexcept
/*!
@brief return the number of characters read so far
Unlike get_position(), this is O(1) for all adapters, so it may be called
for every value (JSON_DIAGNOSTIC_POSITIONS).
*/
std::size_t get_chars_read_total() const noexcept
{
return position;
return get_chars_read_total_impl(std::integral_constant<bool, lazy_position> {});
}
/// return position of last read token
position_t get_position() const
{
return get_position_impl(std::integral_constant<bool, lazy_position> {});
}
/*!
@@ -1897,6 +1935,43 @@ scan_number_done:
}
#endif
private:
/// seekable adapter: every get() either read from the adapter (including
/// reads at EOF, which get() counts as characters as well) or replayed an
/// ungotten character, so the count is the adapter's read count minus a
/// still pending unget
std::size_t get_chars_read_total_impl(std::true_type /*lazy*/) const noexcept
{
return ia.get_read_count() - (next_unget ? 1u : 0u);
}
/// streaming adapter: the counter is maintained by get() and unget()
std::size_t get_chars_read_total_impl(std::false_type /*lazy*/) const noexcept
{
return position.chars_read_total;
}
/// seekable adapter: count the lines in the consumed input; O(input read
/// so far), but only called on errors
position_t get_position_impl(std::true_type /*lazy*/) const
{
position_t result;
result.chars_read_total = get_chars_read_total();
// reads at EOF count as characters but are not part of the input
const std::size_t consumed = (std::min)(result.chars_read_total, ia.get_consumed_count());
std::size_t line_start = 0;
result.lines_read = ia.count_consumed_lines(consumed, line_start);
result.chars_read_current_line = result.chars_read_total - line_start;
return result;
}
/// streaming adapter: the position is maintained by get() and unget()
position_t get_position_impl(std::false_type /*lazy*/) const noexcept
{
return position;
}
public:
/// seekable adapter: rebuild the last read token from the input on demand
const std::vector<char_type>& collect_token_chars(std::vector<char_type>& out, std::true_type /*lazy*/) const
{
@@ -2012,7 +2087,7 @@ scan_number_done:
token_type scan()
{
// initially, skip the BOM
if (position.chars_read_total == 0 && !skip_bom())
if (get_chars_read_total() == 0 && !skip_bom())
{
error_message = "invalid BOM; must be 0xEF 0xBB 0xBF if given";
return token_type::parse_error;
@@ -2039,7 +2114,7 @@ scan_number_done:
token_type scan_expecting(token_type expected_type)
{
JSON_ASSERT(expected_type == token_type::name_separator || expected_type == token_type::value_separator);
JSON_ASSERT(position.chars_read_total > 0);
JSON_ASSERT(get_chars_read_total() > 0);
const char_int_type expected_char = static_cast<unsigned char>((expected_type == token_type::name_separator) ? ':' : ',');
skip_whitespace();
if (JSON_HEDLEY_LIKELY(current == expected_char))
+5 -3
View File
@@ -370,7 +370,7 @@ class parser
}
case token_type::end_of_input:
{
if (JSON_HEDLEY_UNLIKELY(m_lexer.get_position().chars_read_total == 1))
if (JSON_HEDLEY_UNLIKELY(m_lexer.get_chars_read_total() == 1))
{
return syntax_error(*sax, "attempting to parse an empty input; check that your input string or stream contains the expected JSON");
}
@@ -520,8 +520,10 @@ class parser
{
// MSVC 2015 reports C4100 (unreferenced parameter) when SAX::parse_error is static
static_cast<void>(sax);
return sax.parse_error(m_lexer.get_position(), m_lexer.get_token_string(),
parse_error::create(exception_id::syntax_error, m_lexer.get_position(), message, nullptr));
// computing the position is O(input read so far) for seekable input
const position_t position = m_lexer.get_position();
return sax.parse_error(position, m_lexer.get_token_string(),
parse_error::create(exception_id::syntax_error, position, message, nullptr));
}
std::string exception_message(const token_type expected, const std::string& context)
+6 -11
View File
@@ -42,14 +42,9 @@
// each of these, not only the forward declarations from json_fwd.hpp, so IWYU's suggestion to
// drop them (nothing in this file otherwise names the type) would break every downstream
// translation unit that relies on basic_json<>'s defaults actually being usable.
// export: client code names what these headers declare (nlohmann::json and ordered_json from
// json_fwd.hpp, adl_serializer for specializations, json_ref behind brace initialization, the
// NLOHMANN_DEFINE_* and NLOHMANN_JSON_NAMESPACE_* macros, and the literals at the end of this file), and include analyzers must
// attribute it to this header, not tell clients to include the internal ones (issue #5812).
// tests/iwyu_consumer checks this.
#include <nlohmann/adl_serializer.hpp> // IWYU pragma: export
#include <nlohmann/adl_serializer.hpp> // IWYU pragma: keep
#include <nlohmann/byte_container_with_subtype.hpp>
#include <nlohmann/detail/abi_macros.hpp> // IWYU pragma: export
#include <nlohmann/detail/abi_macros.hpp>
#include <nlohmann/detail/conversions/from_json.hpp> // IWYU pragma: keep
#include <nlohmann/detail/conversions/to_json.hpp> // IWYU pragma: keep
#include <nlohmann/detail/exceptions.hpp>
@@ -66,8 +61,8 @@
#include <nlohmann/detail/iterators/primitive_iterator.hpp>
#include <nlohmann/detail/json_custom_base_class.hpp>
#include <nlohmann/detail/json_pointer.hpp>
#include <nlohmann/detail/json_ref.hpp> // IWYU pragma: export
#include <nlohmann/detail/macro_scope.hpp> // IWYU pragma: export
#include <nlohmann/detail/json_ref.hpp>
#include <nlohmann/detail/macro_scope.hpp>
#include <nlohmann/detail/meta/detected.hpp>
#include <nlohmann/detail/string_concat.hpp>
#include <nlohmann/detail/string_escape.hpp>
@@ -79,7 +74,7 @@
#include <nlohmann/detail/output/serializer.hpp>
#include <nlohmann/detail/recursion_depth_limit.hpp>
#include <nlohmann/detail/value_t.hpp>
#include <nlohmann/json_fwd.hpp> // IWYU pragma: export
#include <nlohmann/json_fwd.hpp>
#include <nlohmann/ordered_map.hpp> // IWYU pragma: keep
#include <nlohmann/thirdparty/hedley/hedley.hpp>
@@ -7800,7 +7795,7 @@ struct formatter<nlohmann::NLOHMANN_BASIC_JSON_TPL, char> // NOLINT(cert-dcl58-c
// where needed.
#ifndef JSON_NO_AUTOMATIC_UDLS
// NOLINTNEXTLINE(misc-header-include-cycle): json_literals.hpp includes this header
#include <nlohmann/json_literals.hpp> // IWYU pragma: export
#include <nlohmann/json_literals.hpp>
#endif
#endif // INCLUDE_NLOHMANN_JSON_HPP_
+134 -39
View File
@@ -42,11 +42,6 @@
// each of these, not only the forward declarations from json_fwd.hpp, so IWYU's suggestion to
// drop them (nothing in this file otherwise names the type) would break every downstream
// translation unit that relies on basic_json<>'s defaults actually being usable.
// export: client code names what these headers declare (nlohmann::json and ordered_json from
// json_fwd.hpp, adl_serializer for specializations, json_ref behind brace initialization, the
// NLOHMANN_DEFINE_* and NLOHMANN_JSON_NAMESPACE_* macros, and the literals at the end of this file), and include analyzers must
// attribute it to this header, not tell clients to include the internal ones (issue #5812).
// tests/iwyu_consumer checks this.
// #include <nlohmann/adl_serializer.hpp>
// __ _____ _____ _____
// __| | __| | | | JSON for Modern C++
@@ -7538,7 +7533,7 @@ struct adl_serializer
};
NLOHMANN_JSON_NAMESPACE_END
// IWYU pragma: export
// IWYU pragma: keep
// #include <nlohmann/byte_container_with_subtype.hpp>
// __ _____ _____ _____
// __| | __| | | | JSON for Modern C++
@@ -7646,7 +7641,7 @@ class byte_container_with_subtype : public BinaryType
NLOHMANN_JSON_NAMESPACE_END
// #include <nlohmann/detail/abi_macros.hpp>
// IWYU pragma: export
// #include <nlohmann/detail/conversions/from_json.hpp>
// IWYU pragma: keep
// #include <nlohmann/detail/conversions/to_json.hpp>
@@ -7968,12 +7963,12 @@ NLOHMANN_JSON_NAMESPACE_END
#include <algorithm> // min
#include <algorithm> // count, find, min
#include <array> // array
#include <cstddef> // size_t
#include <cstdint> // uint32_t
#include <cstring> // strlen
#include <iterator> // begin, end, iterator_traits, random_access_iterator_tag, distance, next
#include <iterator> // begin, end, iterator_traits, random_access_iterator_tag, distance, next, reverse_iterator
#include <streambuf> // streambuf
#include <string> // string, char_traits
#include <type_traits> // enable_if, is_base_of, is_pointer, is_integral, remove_pointer
@@ -8262,6 +8257,7 @@ class iterator_input_adapter
return result;
}
++eof_reads;
return char_traits<char_type>::eof();
}
@@ -8271,6 +8267,26 @@ class iterator_input_adapter
return static_cast<std::size_t>(std::distance(begin, current));
}
// number of get_character() calls so far, including those that returned
// EOF; the lexer derives its position from it (see lexer::lazy_position)
std::size_t get_read_count() const
{
return get_consumed_count() + eof_reads;
}
// number of newlines in the consumed range [0, last_index); @a line_start
// is set to the index after the last of them (0 if there is none); only
// valid when supports_seek
std::size_t count_consumed_lines(std::size_t last_index, std::size_t& line_start) const
{
const auto to = std::next(begin, static_cast<typename std::iterator_traits<IteratorType>::difference_type>(last_index));
const auto newline = char_traits<char_type>::to_char_type('\n');
using reverse_it = std::reverse_iterator<IteratorType>;
const auto last_newline = std::find(reverse_it(to), reverse_it(begin), newline);
line_start = static_cast<std::size_t>(std::distance(begin, last_newline.base()));
return static_cast<std::size_t>(std::count(begin, last_newline.base(), newline));
}
// append the already-consumed characters in the half-open range
// [first_index, last_index) to @a out; only valid when supports_seek
template<typename ContainerType>
@@ -8387,6 +8403,8 @@ class iterator_input_adapter
IteratorType begin;
IteratorType current;
SentinelType end;
/// number of get_character() calls that returned EOF
std::size_t eof_reads = 0;
template<typename BaseInputAdapter, size_t T>
friend struct wide_string_input_helper;
@@ -10589,6 +10607,15 @@ class lexer : public lexer_base<BasicJsonType>
static constexpr bool lazy_token_string =
input_adapter_supports_seek<InputAdapterType>(is_detected<detect_supports_seek, InputAdapterType> {});
/// whether the read position is derived from the input adapter on demand
/// instead of being counted on every scanned character. Seekable adapters
/// know how much input they consumed and can revisit it, so
/// chars_read_total is the adapter's read count (minus a pending unget),
/// and line and column are recomputed from the consumed input only when
/// get_position() is called (errors). Counting per character kept two
/// increments and a newline check in the hot path of every get().
static constexpr bool lazy_position = lazy_token_string;
/// whether a simulated unget can be passed on to the input adapter, which
/// then leaves the character in the input; see
/// input_adapter_supports_lookahead
@@ -10598,7 +10625,9 @@ class lexer : public lexer_base<BasicJsonType>
/// whether string scanning may bulk-consume runs of ordinary characters
/// directly from a contiguous input buffer (SWAR fast path). This requires
/// the token to be reconstructible lazily (lazy_token_string), so bypassing
/// the per-character capture in get() cannot lose error diagnostics.
/// the per-character capture in get() cannot lose error diagnostics. It
/// also implies lazy_position, so the bulk paths need not advance the
/// position counters.
static constexpr bool bulk_scan =
lazy_token_string
&& input_adapter_supports_bulk_scan<InputAdapterType>(is_detected<detect_supports_bulk_scan, InputAdapterType> {});
@@ -10734,10 +10763,6 @@ class lexer : public lexer_base<BasicJsonType>
}
token_buffer.append(reinterpret_cast<const typename string_t::value_type*>(data), pos);
ia.bulk_skip(pos);
// the run contains no newline (all bytes < 0x20 are treated as special),
// so only the flat character counters advance
position.chars_read_total += pos;
position.chars_read_current_line += pos;
}
/// streaming input: no bulk fast path
@@ -11970,8 +11995,6 @@ scan_number_done:
if (JSON_HEDLEY_LIKELY(integer_result != token_type::uninitialized))
{
ia.bulk_skip(len - 1);
position.chars_read_total += (len - 1);
position.chars_read_current_line += (len - 1);
return integer_result;
}
// The value does not fit an integer, so this token converts as a
@@ -11988,8 +12011,6 @@ scan_number_done:
decimal_point_position = dot_index;
ia.bulk_skip(len - 1);
position.chars_read_total += (len - 1);
position.chars_read_current_line += (len - 1);
return convert_number(number_type, mantissa_end);
}
@@ -12040,7 +12061,7 @@ scan_number_done:
#if JSON_DIAGNOSTIC_POSITIONS
// the first character of the token has already been read, hence the -1
token_start_position = position.chars_read_total - 1;
token_start_position = get_chars_read_total() - 1;
#endif
note_token_start(std::integral_constant<bool, lazy_token_string> {});
@@ -12093,6 +12114,15 @@ scan_number_done:
/// per-character position counters (line-count-on-'\n' bookkeeping is
/// handled afterwards, in track_after_read(), once `current` is known)
void advance_position() noexcept
{
advance_position_impl(std::integral_constant<bool, lazy_position> {});
}
/// seekable adapter: the position is derived from the adapter on demand
void advance_position_impl(std::true_type /*lazy*/) const noexcept {}
/// streaming adapter: count the character
void advance_position_impl(std::false_type /*lazy*/) noexcept
{
++position.chars_read_total;
++position.chars_read_current_line;
@@ -12106,7 +12136,16 @@ scan_number_done:
// seekable adapters reconstruct the token lazily on error (see
// get_token_string), so the eager per-character copy is skipped
capture_char(std::integral_constant<bool, lazy_token_string> {});
track_newline(std::integral_constant<bool, lazy_position> {});
return current;
}
/// seekable adapter: lines are counted on demand in get_position()
void track_newline(std::true_type /*lazy*/) const noexcept {}
/// streaming adapter: start a new line after reading a newline
void track_newline(std::false_type /*lazy*/) noexcept
{
if (current == '\n')
{
++position.lines_read;
@@ -12115,8 +12154,6 @@ scan_number_done:
chars_read_before_newline = position.chars_read_current_line;
position.chars_read_current_line = 0;
}
return current;
}
/*!
@@ -12160,12 +12197,22 @@ scan_number_done:
We implement unget by setting variable next_unget to true. The input is not
changed - we just simulate ungetting by modifying chars_read_total,
chars_read_current_line, and token_string. The next call to get() will
behave as if the unget character is read again.
behave as if the unget character is read again. With lazy_position, the
pending next_unget is all get_chars_read_total() needs.
*/
void unget()
{
next_unget = true;
unget_position(std::integral_constant<bool, lazy_position> {});
uncapture_char(std::integral_constant<bool, lazy_token_string> {});
}
/// seekable adapter: get_chars_read_total() accounts for next_unget
void unget_position(std::true_type /*lazy*/) const noexcept {}
/// streaming adapter: step the counters back over the ungotten character
void unget_position(std::false_type /*lazy*/) noexcept
{
--position.chars_read_total;
// in case we "unget" a newline, we have to also decrement the lines_read
@@ -12188,8 +12235,6 @@ scan_number_done:
{
--position.chars_read_current_line;
}
uncapture_char(std::integral_constant<bool, lazy_token_string> {});
}
/// adapter without lookahead: nothing to do (see release_lookahead)
@@ -12260,10 +12305,21 @@ scan_number_done:
// diagnostics
/////////////////////
/// return position of last read token
constexpr position_t get_position() const noexcept
/*!
@brief return the number of characters read so far
Unlike get_position(), this is O(1) for all adapters, so it may be called
for every value (JSON_DIAGNOSTIC_POSITIONS).
*/
std::size_t get_chars_read_total() const noexcept
{
return position;
return get_chars_read_total_impl(std::integral_constant<bool, lazy_position> {});
}
/// return position of last read token
position_t get_position() const
{
return get_position_impl(std::integral_constant<bool, lazy_position> {});
}
/*!
@@ -12300,6 +12356,43 @@ scan_number_done:
}
#endif
private:
/// seekable adapter: every get() either read from the adapter (including
/// reads at EOF, which get() counts as characters as well) or replayed an
/// ungotten character, so the count is the adapter's read count minus a
/// still pending unget
std::size_t get_chars_read_total_impl(std::true_type /*lazy*/) const noexcept
{
return ia.get_read_count() - (next_unget ? 1u : 0u);
}
/// streaming adapter: the counter is maintained by get() and unget()
std::size_t get_chars_read_total_impl(std::false_type /*lazy*/) const noexcept
{
return position.chars_read_total;
}
/// seekable adapter: count the lines in the consumed input; O(input read
/// so far), but only called on errors
position_t get_position_impl(std::true_type /*lazy*/) const
{
position_t result;
result.chars_read_total = get_chars_read_total();
// reads at EOF count as characters but are not part of the input
const std::size_t consumed = (std::min)(result.chars_read_total, ia.get_consumed_count());
std::size_t line_start = 0;
result.lines_read = ia.count_consumed_lines(consumed, line_start);
result.chars_read_current_line = result.chars_read_total - line_start;
return result;
}
/// streaming adapter: the position is maintained by get() and unget()
position_t get_position_impl(std::false_type /*lazy*/) const noexcept
{
return position;
}
public:
/// seekable adapter: rebuild the last read token from the input on demand
const std::vector<char_type>& collect_token_chars(std::vector<char_type>& out, std::true_type /*lazy*/) const
{
@@ -12415,7 +12508,7 @@ scan_number_done:
token_type scan()
{
// initially, skip the BOM
if (position.chars_read_total == 0 && !skip_bom())
if (get_chars_read_total() == 0 && !skip_bom())
{
error_message = "invalid BOM; must be 0xEF 0xBB 0xBF if given";
return token_type::parse_error;
@@ -12442,7 +12535,7 @@ scan_number_done:
token_type scan_expecting(token_type expected_type)
{
JSON_ASSERT(expected_type == token_type::name_separator || expected_type == token_type::value_separator);
JSON_ASSERT(position.chars_read_total > 0);
JSON_ASSERT(get_chars_read_total() > 0);
const char_int_type expected_char = static_cast<unsigned char>((expected_type == token_type::name_separator) ? ':' : ',');
skip_whitespace();
if (JSON_HEDLEY_LIKELY(current == expected_char))
@@ -12796,7 +12889,7 @@ struct diagnostic_positions
{
// Lexer has read the first character of the container, so
// subtract 1 from the position to get the correct start position.
v.start_position = lexer->get_position() - 1;
v.start_position = lexer->get_chars_read_total() - 1;
}
}
@@ -12810,7 +12903,7 @@ struct diagnostic_positions
if (lexer)
{
// Lexer's position is past the closing brace or bracket, so set that as the end position.
v.end_position = lexer->get_position();
v.end_position = lexer->get_chars_read_total();
}
}
@@ -12826,7 +12919,7 @@ struct diagnostic_positions
// Lexer has read past the current field value, so set the end position to the current position.
// The start position will be set below based on the length of the string representation
// of the value.
v.end_position = lexer->get_position();
v.end_position = lexer->get_chars_read_total();
switch (v.type())
{
@@ -18669,7 +18762,7 @@ class parser
}
case token_type::end_of_input:
{
if (JSON_HEDLEY_UNLIKELY(m_lexer.get_position().chars_read_total == 1))
if (JSON_HEDLEY_UNLIKELY(m_lexer.get_chars_read_total() == 1))
{
return syntax_error(*sax, "attempting to parse an empty input; check that your input string or stream contains the expected JSON");
}
@@ -18819,8 +18912,10 @@ class parser
{
// MSVC 2015 reports C4100 (unreferenced parameter) when SAX::parse_error is static
static_cast<void>(sax);
return sax.parse_error(m_lexer.get_position(), m_lexer.get_token_string(),
parse_error::create(exception_id::syntax_error, m_lexer.get_position(), message, nullptr));
// computing the position is O(input read so far) for seekable input
const position_t position = m_lexer.get_position();
return sax.parse_error(position, m_lexer.get_token_string(),
parse_error::create(exception_id::syntax_error, position, message, nullptr));
}
std::string exception_message(const token_type expected, const std::string& context)
@@ -21415,9 +21510,9 @@ class json_ref
} // namespace detail
NLOHMANN_JSON_NAMESPACE_END
// IWYU pragma: export
// #include <nlohmann/detail/macro_scope.hpp>
// IWYU pragma: export
// #include <nlohmann/detail/meta/detected.hpp>
// #include <nlohmann/detail/string_concat.hpp>
@@ -27781,7 +27876,7 @@ NLOHMANN_JSON_NAMESPACE_END
// #include <nlohmann/detail/value_t.hpp>
// #include <nlohmann/json_fwd.hpp>
// IWYU pragma: export
// #include <nlohmann/ordered_map.hpp>
// __ _____ _____ _____
// __| | __| | | | JSON for Modern C++
@@ -36214,7 +36309,7 @@ NLOHMANN_JSON_NAMESPACE_END
#undef NLOHMANN_JSON_LITERAL_OPERATOR
#endif // INCLUDE_NLOHMANN_JSON_LITERALS_HPP_
// IWYU pragma: export
#endif
#endif // INCLUDE_NLOHMANN_JSON_HPP_
-62
View File
@@ -1,62 +0,0 @@
// __ _____ _____ _____
// __| | __| | | | JSON for Modern C++ (supporting code)
// | | |__ | | | | | | version 3.12.0
// |_____|_____|_____|_|___| https://github.com/nlohmann/json
//
// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
// SPDX-License-Identifier: MIT
// See values.cpp: the conversion macros and nlohmann::adl_serializer must be available through
// <nlohmann/json.hpp> alone. NLOHMANN_JSON_SERIALIZE_ENUM (from the same header as
// NLOHMANN_DEFINE_TYPE_INTRUSIVE) is left out: IWYU attributes the basic_json comparison in its
// expansion to an unrelated standard header, the ambiguity described for json.hpp in cmake/ci.cmake.
#include <string>
#include <nlohmann/json.hpp>
namespace ns
{
struct person
{
std::string name;
int age;
NLOHMANN_DEFINE_TYPE_INTRUSIVE(person, name, age)
};
struct legacy
{
int value;
};
} // namespace ns
namespace nlohmann
{
template<>
struct adl_serializer<ns::legacy>
{
static void to_json(json& j, const ns::legacy& l)
{
j = l.value;
}
static void from_json(const json& j, ns::legacy& l)
{
l.value = j.get<int>();
}
};
} // namespace nlohmann
int main()
{
ns::person ada;
ada.name = "Ada";
ada.age = 36;
ns::legacy one;
one.value = 1;
const nlohmann::json j = ada;
const auto p = j.get<ns::person>();
const auto l = nlohmann::json(one).get<ns::legacy>();
return p.age - 36 + l.value - 1;
}
-19
View File
@@ -1,19 +0,0 @@
// __ _____ _____ _____
// __| | __| | | | JSON for Modern C++ (supporting code)
// | | |__ | | | | | | version 3.12.0
// |_____|_____|_____|_|___| https://github.com/nlohmann/json
//
// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
// SPDX-License-Identifier: MIT
// See values.cpp: the user-defined literals must be available through <nlohmann/json.hpp> alone.
#include <nlohmann/json.hpp>
using namespace nlohmann::literals; // NOLINT(google-build-using-namespace)
int main()
{
const auto j = R"({"a": [1, 2]})"_json;
return j.at("/a/1"_json_pointer).get<int>() - 2;
}
-30
View File
@@ -1,30 +0,0 @@
// __ _____ _____ _____
// __| | __| | | | JSON for Modern C++ (supporting code)
// | | |__ | | | | | | version 3.12.0
// |_____|_____|_____|_|___| https://github.com/nlohmann/json
//
// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
// SPDX-License-Identifier: MIT
// Uses the library the way a client does, including only <nlohmann/json.hpp>. ci_single_binaries
// runs include-what-you-use and clang-tidy's misc-include-cleaner on this file and fails if either
// asks for another nlohmann header, which would mean json.hpp does not export it (issue #5812).
#include <cstdint>
#include <string>
#include <vector>
#include <nlohmann/json.hpp>
int main()
{
// brace initialization (nlohmann::json, detail::json_ref)
nlohmann::json j = {{"pi", 3.141}, {"list", {1, 0, 2}}};
const nlohmann::ordered_json o = {{"b", 1}, {"a", 2}};
j["parsed"] = nlohmann::json::parse(R"({"happy": true})");
j["binary"] = nlohmann::json::binary(std::vector<std::uint8_t> {0xCA, 0xFE}, 42);
const std::string s = j.dump() + o.dump();
return s.empty() ? 1 : 0;
}
+57
View File
@@ -8,6 +8,13 @@
#include "doctest_compatibility.h"
// capture whether JSON_DISABLE_ENUM_SERIALIZATION was enabled on the command
// line before including json.hpp, which #undefs it; std::byte is an enum, so
// converting std::vector<std::byte> needs enum serialization
#if defined(JSON_DISABLE_ENUM_SERIALIZATION) && (JSON_DISABLE_ENUM_SERIALIZATION == 1)
#define SKIP_TESTS_FOR_ENUM_SERIALIZATION
#endif
#include <nlohmann/json.hpp>
#include <cstdint>
@@ -85,3 +92,53 @@ TEST_CASE("binary type whose value type is not std::uint8_t")
}
#endif
}
#ifdef JSON_HAS_CPP_17
// moved from unit-regression3.cpp, so that file does not instantiate a second
// basic_json specialization (MinGW section limit, see windows.yml)
TEST_CASE("std::vector<std::byte> as binary type")
{
SECTION("issue #4804: from_cbor incompatible with std::vector<std::byte> as binary_t")
{
const std::vector<std::uint8_t> data = {0x80};
const auto decoded = byte_binary_json::from_cbor(data);
CHECK((decoded == byte_binary_json::array()));
}
#ifndef SKIP_TESTS_FOR_ENUM_SERIALIZATION
SECTION("discussion #4209 - custom BinaryType direct assignment and round-tripping")
{
// Test that assigning a custom BinaryType directly creates a binary value, not an array
const std::vector<std::byte> original{std::byte{1}, std::byte{2}, std::byte{3}};
const byte_binary_json j = original;
CHECK(j.is_binary());
CHECK(!j.is_array());
// Test round-tripping: extracting the binary value back as the custom container type
const auto extracted = j.get<std::vector<std::byte>>();
CHECK(extracted == original);
// Test that the default json alias behavior is unchanged: std::vector<uint8_t> -> array
const nlohmann::json default_json = std::vector<std::uint8_t> {1, 2, 3};
CHECK(default_json.is_array());
CHECK(!default_json.is_binary());
}
SECTION("discussion #4209 - custom BinaryType extraction from parsed array")
{
// Test that extracting a custom BinaryType from a parsed JSON array still works
// (not just from a binary-typed node)
const auto j = byte_binary_json::parse("[1,2,3]");
CHECK(j.is_array());
CHECK(!j.is_binary());
// Extracting as custom BinaryType should work from arrays
const auto extracted = j.get<std::vector<std::byte>>();
CHECK(extracted.size() == 3);
CHECK(extracted[0] == std::byte{1});
CHECK(extracted[1] == std::byte{2});
CHECK(extracted[2] == std::byte{3});
}
#endif
}
#endif
+64
View File
@@ -35,6 +35,7 @@ using nlohmann::json;
#include <sstream>
#include <string>
#include <valarray>
#include <vector>
#include "sax_event_loggers.hpp"
@@ -1250,3 +1251,66 @@ TEST_CASE_TEMPLATE("deserialization of different character types (UTF-32)", T, c
CHECK(json::sax_parse(v, &l));
CHECK(l.events.size() == 1);
}
namespace
{
// records the position and message of a parse error; reporting the error via
// SAX does not throw, so the test also runs with JSON_NOEXCEPTION
struct SaxErrorRecorder : public SaxEventLogger
{
bool parse_error(std::size_t position, const std::string& /*last_token*/, const json::exception& ex) override
{
error = std::to_string(position) + " " + ex.what();
return false;
}
std::string error {}; // NOLINT(readability-redundant-member-init)
};
} // namespace
TEST_CASE("parse error positions are the same for contiguous and stream input")
{
// contiguous input derives the position from the input on error, stream
// input counts it while reading (see lexer::lazy_position); the cases
// cover newlines before the error, ungetting a newline after a number,
// newlines inside a string, a comment, and reads at the end of input
struct test_case
{
std::string input;
std::size_t byte;
std::string message;
};
const std::vector<test_case> cases =
{
{"[\n 1,\n 2\n", 12, "[json.exception.parse_error.101] parse error at line 4, column 1: syntax error while parsing array - unexpected end of input; expected ']'"},
{"[1\n,2,\n]", 8, "[json.exception.parse_error.101] parse error at line 3, column 1: syntax error while parsing value - unexpected ']'; expected '[', '{', or a literal"},
{"[1\n x]", 5, "[json.exception.parse_error.101] parse error at line 2, column 2: syntax error while parsing array - invalid literal; last read: '1<U+000A> x'; expected ']'"},
{"1\n2", 3, "[json.exception.parse_error.101] parse error at line 2, column 1: syntax error while parsing value - unexpected number literal; expected end of input"},
{
"{\"a\":\n\"b\nc\"}", 9, R"([json.exception.parse_error.101] parse error at line 3, column 0: syntax error while parsing value - invalid string: control character U+000A (LF) must be escaped to \u000A or \n; last read: '"b<U+000A>')"
},
{"/* a\n b */\n x", 13, "[json.exception.parse_error.101] parse error at line 3, column 2: syntax error while parsing value - invalid literal; last read: '/* a<U+000A> b */<U+000A> x'"},
{"\n\n", 3, "[json.exception.parse_error.101] parse error at line 3, column 1: syntax error while parsing value - unexpected end of input; expected '[', '{', or a literal"},
{"[\"a\",\n 1e\n]", 10, "[json.exception.parse_error.101] parse error at line 3, column 0: syntax error while parsing value - invalid number; expected '+', '-', or digit after exponent; last read: '1e<U+000A>'"},
{"[tru\n]", 5, "[json.exception.parse_error.101] parse error at line 2, column 0: syntax error while parsing value - invalid literal; last read: '[tru<U+000A>'"},
};
for (const auto& c : cases)
{
CAPTURE(c.input)
const std::string expected = std::to_string(c.byte) + " " + c.message;
SaxErrorRecorder string_sax;
CHECK_FALSE(json::sax_parse(c.input, &string_sax, json::input_format_t::json, true, true));
CHECK(string_sax.error == expected);
SaxErrorRecorder pointer_sax;
CHECK_FALSE(json::sax_parse(c.input.data(), c.input.data() + c.input.size(), &pointer_sax, json::input_format_t::json, true, true));
CHECK(pointer_sax.error == expected);
std::istringstream is(c.input);
SaxErrorRecorder stream_sax;
CHECK_FALSE(json::sax_parse(is, &stream_sax, json::input_format_t::json, true, true));
CHECK(stream_sax.error == expected);
}
}
-48
View File
@@ -58,11 +58,6 @@ using ordered_json = nlohmann::ordered_json;
#include <optional>
#elif __has_include(<experimental/optional>)
#endif
/////////////////////////////////////////////////////////////////////
// for #4804
/////////////////////////////////////////////////////////////////////
using json_4804 = nlohmann::json::with_binary_t<std::vector<std::byte>>;
#endif
#ifdef JSON_HAS_CPP_20
@@ -458,49 +453,6 @@ TEST_CASE("regression tests 3")
#endif
#ifdef JSON_HAS_CPP_17
SECTION("issue #4804: from_cbor incompatible with std::vector<std::byte> as binary_t")
{
const std::vector<std::uint8_t> data = {0x80};
const auto decoded = json_4804::from_cbor(data);
CHECK((decoded == json_4804::array()));
}
#ifndef SKIP_TESTS_FOR_ENUM_SERIALIZATION
SECTION("discussion #4209 - custom BinaryType direct assignment and round-tripping")
{
// Test that assigning a custom BinaryType directly creates a binary value, not an array
const std::vector<std::byte> original{std::byte{1}, std::byte{2}, std::byte{3}};
const json_4804 j = original;
CHECK(j.is_binary());
CHECK(!j.is_array());
// Test round-tripping: extracting the binary value back as the custom container type
const auto extracted = j.get<std::vector<std::byte>>();
CHECK(extracted == original);
// Test that the default json alias behavior is unchanged: std::vector<uint8_t> -> array
const json default_json = std::vector<std::uint8_t> {1, 2, 3};
CHECK(default_json.is_array());
CHECK(!default_json.is_binary());
}
SECTION("discussion #4209 - custom BinaryType extraction from parsed array")
{
// Test that extracting a custom BinaryType from a parsed JSON array still works
// (not just from a binary-typed node)
const auto j = json_4804::parse("[1,2,3]");
CHECK(j.is_array());
CHECK(!j.is_binary());
// Extracting as custom BinaryType should work from arrays
const auto extracted = j.get<std::vector<std::byte>>();
CHECK(extracted.size() == 3);
CHECK(extracted[0] == std::byte{1});
CHECK(extracted[1] == std::byte{2});
CHECK(extracted[2] == std::byte{3});
}
#endif
SECTION("issue #5046 - implicit conversion of return json to std::optional no longer implicit")
{
const json jval{};