Add iterator+sentinel tests and docs for binary deserializers (#5265)

* Add iterator+sentinel tests and docs for binary deserializers

This commit extends the C++20 ranges support (iterator+sentinel pairs) to the
binary format deserializers from_cbor, from_msgpack, from_ubjson, from_bjdata,
and from_bson, matching what was already done for parse(), accept(), and
sax_parse().

Changes:
- Add istreambuf_sentinel helper to test_utils.hpp for EOF detection in tests
- Add 5 new test cases that read binary files directly via
  std::istreambuf_iterator<char> + sentinel, without pre-buffering
- Update documentation for all 5 from_* functions to document overload (3)
  with SentinelType parameter
- All tests pass; verified against existing test suite data
- Fix potential buffer over-read warning in heterogeneous iterator test

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Merge iterator+sentinel overloads and fix ambiguity/CI issues

Address PR review feedback and CI failures:

- Merge the separate same-type and sentinel-type iterator overloads of
  parse(), accept(), sax_parse(), and the five from_* binary deserializers
  into a single overload with SentinelType defaulted to IteratorType,
  as suggested in review. Applied the same simplification to the
  detail::input_adapter() free functions.
- Fix a latent ambiguity: some compilers (e.g. GCC 4.8) unreliably SFINAE
  the operator!= detection for std::nullptr_t against container/string
  types, making calls like parse(s, nullptr, ...) ambiguous with the
  compatible-input overload. can_compare_ne now explicitly excludes
  std::nullptr_t as a SentinelType.
- Use a named enable_if_t template parameter instead of an unnamed
  function parameter for the SFINAE guard, fixing a clang-tidy
  hicpp-named-parameter/readability-named-parameter failure.
- Update parse.md, accept.md, sax_parse.md, and the five from_*.md pages
  to document the merged overload instead of separate (2)/(3) overloads,
  also fixing an over-160-char line that broke the documentation
  style_check CI job.
- Rework the BSON iterator+sentinel test to parse a BSON file already
  present in the test suite instead of writing/deleting a temp file.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Fix -Wunneeded-internal-declaration for CustomSentinel in test

CustomSentinel lives in an anonymous namespace (internal linkage), and
the library's parse loop only ever evaluates the iterator-first
direction (it != last), so the reversed-order friend operator!= was
never referenced. Clang's -Weverything flags such unused internal
declarations as an error. Drop the unused overload; the used direction
is enough to satisfy can_compare_ne's either-order detection.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Fix clang-tidy hicpp-named-parameter and misc-const-correctness

- Drop the unused reversed-order operator!= overload from
  utils::istreambuf_sentinel (only iterator != sentinel is ever
  evaluated) and name the remaining friend's sentinel parameter, fixing
  hicpp-named-parameter/readability-named-parameter.
- Mark the istreambuf_iterator first/last helper variable const in the
  five binary-format sentinel tests, fixing misc-const-correctness.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Fix clang-tidy misc-const-correctness in heterogeneous sentinel test

json_str is only read via .data()/.size() and never reassigned, so
clang-tidy correctly flags it as const-able. Verified against the exact
CI job (silkeh/clang:dev, ci_clang_tidy target) by running clang-tidy
directly on this file plus the five binary-format sentinel tests
touched by prior commits; all are now clean.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
This commit is contained in:
Niels Lohmann
2026-07-11 15:09:28 +02:00
committed by GitHub
parent fe2bcc080f
commit ca76c37650
18 changed files with 366 additions and 117 deletions
+13 -5
View File
@@ -8,8 +8,8 @@ static bool accept(InputType&& i,
const bool ignore_trailing_commas = false);
// (2)
template<typename IteratorType>
static bool accept(IteratorType first, IteratorType last,
template<typename IteratorType, typename SentinelType = IteratorType>
static bool accept(IteratorType first, SentinelType last,
const bool ignore_comments = false,
const bool ignore_trailing_commas = false);
```
@@ -17,10 +17,11 @@ static bool accept(IteratorType first, IteratorType last,
Checks whether the input is valid JSON.
1. Reads from a compatible input.
2. Reads from a pair of character iterators
2. Reads from a pair of character iterators, or an iterator and a sentinel of a different type (C++20 ranges support)
The value_type of the iterator must be an integral type with a size of 1, 2, or 4 bytes, which will be interpreted
respectively as UTF-8, UTF-16, and UTF-32.
respectively as UTF-8, UTF-16, and UTF-32. If `SentinelType` differs from `IteratorType`, it must be comparable to
the iterator type with `operator!=`.
Unlike the [`parse()`](parse.md) function, this function neither throws an exception in case of invalid JSON input
(i.e., a parse error) nor creates diagnostic information.
@@ -44,6 +45,12 @@ Unlike the [`parse()`](parse.md) function, this function neither throws an excep
- a pair of `std::string::iterator` or `std::vector<std::uint8_t>::iterator`
- a pair of pointers such as `ptr` and `ptr + len`
`SentinelType`
: defaults to `IteratorType`; may be a different type comparable to `IteratorType` via `operator!=`, for instance.
- a custom sentinel type for C++20 ranges
- `std::counted_iterator` with a different sentinel type
## Parameters
`i` (in)
@@ -61,7 +68,7 @@ Unlike the [`parse()`](parse.md) function, this function neither throws an excep
: iterator to the start of the character range
`last` (in)
: iterator to the end of the character range
: iterator to the end of the character range, or a sentinel value that compares equal to the end iterator with `operator!=`
## Return value
@@ -112,6 +119,7 @@ A UTF-8 byte order mark is silently ignored.
- Changed [runtime assertion](../../features/assertions.md) in case of `FILE*` null pointers to exception in version 3.12.0.
- Added `ignore_trailing_commas` in version 3.13.0.
- Extended container support (1) to include types with lvalue-only ADL `begin`/`end` (matching `std::begin`/`std::end` semantics) in version 3.13.0.
- Extended overload (2) to accept heterogeneous iterator+sentinel pairs (C++20 ranges support) in version 3.13.0.
!!! warning "Deprecation"
@@ -7,8 +7,8 @@ static basic_json from_bjdata(InputType&& i,
const bool strict = true,
const bool allow_exceptions = true);
// (2)
template<typename IteratorType>
static basic_json from_bjdata(IteratorType first, IteratorType last,
template<typename IteratorType, typename SentinelType = IteratorType>
static basic_json from_bjdata(IteratorType first, SentinelType last,
const bool strict = true,
const bool allow_exceptions = true);
```
@@ -16,7 +16,7 @@ static basic_json from_bjdata(IteratorType first, IteratorType last,
Deserializes a given input to a JSON value using the BJData (Binary JData) serialization format.
1. Reads from a compatible input.
2. Reads from an iterator range.
2. Reads from an iterator range, or an iterator and a sentinel of a different type (C++20 ranges support).
The exact mapping and its limitations are described on a [dedicated page](../../features/binary_formats/bjdata.md).
@@ -35,6 +35,10 @@ The exact mapping and its limitations are described on a [dedicated page](../../
`IteratorType`
: a compatible iterator type
`SentinelType`
: defaults to `IteratorType`; may be a different type comparable to `IteratorType` via `operator!=`, for instance a
custom sentinel type for C++20 ranges
## Parameters
`i` (in)
@@ -44,7 +48,7 @@ The exact mapping and its limitations are described on a [dedicated page](../../
: iterator to the start of the input
`last` (in)
: iterator to the end of the input
: iterator to the end of the input, or a sentinel value that compares equal to the end iterator with `operator!=`
`strict` (in)
: whether to expect the input to be consumed until EOF (`#!cpp true` by default)
@@ -103,3 +107,4 @@ Linear in the size of the input.
- Added in version 3.11.0.
- Extended container support (1) to include types with lvalue-only ADL `begin`/`end` (matching `std::begin`/`std::end` semantics) in version 3.13.0.
- Extended overload (2) to accept heterogeneous iterator+sentinel pairs (C++20 ranges support) in version 3.13.0.
+9 -4
View File
@@ -7,8 +7,8 @@ static basic_json from_bson(InputType&& i,
const bool strict = true,
const bool allow_exceptions = true);
// (2)
template<typename IteratorType>
static basic_json from_bson(IteratorType first, IteratorType last,
template<typename IteratorType, typename SentinelType = IteratorType>
static basic_json from_bson(IteratorType first, SentinelType last,
const bool strict = true,
const bool allow_exceptions = true);
```
@@ -16,7 +16,7 @@ static basic_json from_bson(IteratorType first, IteratorType last,
Deserializes a given input to a JSON value using the BSON (Binary JSON) serialization format.
1. Reads from a compatible input.
2. Reads from an iterator range.
2. Reads from an iterator range, or an iterator and a sentinel of a different type (C++20 ranges support).
The exact mapping and its limitations are described on a [dedicated page](../../features/binary_formats/bson.md).
@@ -35,6 +35,10 @@ The exact mapping and its limitations are described on a [dedicated page](../../
`IteratorType`
: a compatible iterator type
`SentinelType`
: defaults to `IteratorType`; may be a different type comparable to `IteratorType` via `operator!=`, for instance a
custom sentinel type for C++20 ranges
## Parameters
`i` (in)
@@ -44,7 +48,7 @@ The exact mapping and its limitations are described on a [dedicated page](../../
: iterator to the start of the input
`last` (in)
: iterator to the end of the input
: iterator to the end of the input, or a sentinel value that compares equal to the end iterator with `operator!=`
`strict` (in)
: whether to expect the input to be consumed until EOF (`#!cpp true` by default)
@@ -103,6 +107,7 @@ Linear in the size of the input.
- Added in version 3.4.0.
- Extended container support (1) to include types with lvalue-only ADL `begin`/`end` (matching `std::begin`/`std::end` semantics) in version 3.13.0.
- Extended overload (2) to accept heterogeneous iterator+sentinel pairs (C++20 ranges support) in version 3.13.0.
!!! warning "Deprecation"
+9 -4
View File
@@ -9,8 +9,8 @@ static basic_json from_cbor(InputType&& i,
const cbor_tag_handler_t tag_handler = cbor_tag_handler_t::error);
// (2)
template<typename IteratorType>
static basic_json from_cbor(IteratorType first, IteratorType last,
template<typename IteratorType, typename SentinelType = IteratorType>
static basic_json from_cbor(IteratorType first, SentinelType last,
const bool strict = true,
const bool allow_exceptions = true,
const cbor_tag_handler_t tag_handler = cbor_tag_handler_t::error);
@@ -19,7 +19,7 @@ static basic_json from_cbor(IteratorType first, IteratorType last,
Deserializes a given input to a JSON value using the CBOR (Concise Binary Object Representation) serialization format.
1. Reads from a compatible input.
2. Reads from an iterator range.
2. Reads from an iterator range, or an iterator and a sentinel of a different type (C++20 ranges support).
The exact mapping and its limitations are described on a [dedicated page](../../features/binary_formats/cbor.md).
@@ -38,6 +38,10 @@ The exact mapping and its limitations are described on a [dedicated page](../../
`IteratorType`
: a compatible iterator type
`SentinelType`
: defaults to `IteratorType`; may be a different type comparable to `IteratorType` via `operator!=`, for instance a
custom sentinel type for C++20 ranges
## Parameters
`i` (in)
@@ -47,7 +51,7 @@ The exact mapping and its limitations are described on a [dedicated page](../../
: iterator to the start of the input
`last` (in)
: iterator to the end of the input
: iterator to the end of the input, or a sentinel value that compares equal to the end iterator with `operator!=`
`strict` (in)
: whether to expect the input to be consumed until EOF (`#!cpp true` by default)
@@ -113,6 +117,7 @@ Linear in the size of the input.
- Added `allow_exceptions` parameter in version 3.2.0.
- Added `tag_handler` parameter in version 3.9.0.
- Extended container support (1) to include types with lvalue-only ADL `begin`/`end` (matching `std::begin`/`std::end` semantics) in version 3.13.0.
- Extended overload (2) to accept heterogeneous iterator+sentinel pairs (C++20 ranges support) in version 3.13.0.
!!! warning "Deprecation"
@@ -7,8 +7,8 @@ static basic_json from_msgpack(InputType&& i,
const bool strict = true,
const bool allow_exceptions = true);
// (2)
template<typename IteratorType>
static basic_json from_msgpack(IteratorType first, IteratorType last,
template<typename IteratorType, typename SentinelType = IteratorType>
static basic_json from_msgpack(IteratorType first, SentinelType last,
const bool strict = true,
const bool allow_exceptions = true);
```
@@ -16,7 +16,7 @@ static basic_json from_msgpack(IteratorType first, IteratorType last,
Deserializes a given input to a JSON value using the MessagePack serialization format.
1. Reads from a compatible input.
2. Reads from an iterator range.
2. Reads from an iterator range, or an iterator and a sentinel of a different type (C++20 ranges support).
The exact mapping and its limitations are described on a [dedicated page](../../features/binary_formats/messagepack.md).
@@ -35,6 +35,10 @@ The exact mapping and its limitations are described on a [dedicated page](../../
`IteratorType`
: a compatible iterator type
`SentinelType`
: defaults to `IteratorType`; may be a different type comparable to `IteratorType` via `operator!=`, for instance a
custom sentinel type for C++20 ranges
## Parameters
`i` (in)
@@ -44,7 +48,7 @@ The exact mapping and its limitations are described on a [dedicated page](../../
: iterator to the start of the input
`last` (in)
: iterator to the end of the input
: iterator to the end of the input, or a sentinel value that compares equal to the end iterator with `operator!=`
`strict` (in)
: whether to expect the input to be consumed until EOF (`#!cpp true` by default)
@@ -105,6 +109,7 @@ Linear in the size of the input.
- Changed to consume input adapters, removed `start_index` parameter, and added `strict` parameter in version 3.0.0.
- Added `allow_exceptions` parameter in version 3.2.0.
- Extended container support (1) to include types with lvalue-only ADL `begin`/`end` (matching `std::begin`/`std::end` semantics) in version 3.13.0.
- Extended overload (2) to accept heterogeneous iterator+sentinel pairs (C++20 ranges support) in version 3.13.0.
!!! warning "Deprecation"
@@ -7,8 +7,8 @@ static basic_json from_ubjson(InputType&& i,
const bool strict = true,
const bool allow_exceptions = true);
// (2)
template<typename IteratorType>
static basic_json from_ubjson(IteratorType first, IteratorType last,
template<typename IteratorType, typename SentinelType = IteratorType>
static basic_json from_ubjson(IteratorType first, SentinelType last,
const bool strict = true,
const bool allow_exceptions = true);
```
@@ -16,7 +16,7 @@ static basic_json from_ubjson(IteratorType first, IteratorType last,
Deserializes a given input to a JSON value using the UBJSON (Universal Binary JSON) serialization format.
1. Reads from a compatible input.
2. Reads from an iterator range.
2. Reads from an iterator range, or an iterator and a sentinel of a different type (C++20 ranges support).
The exact mapping and its limitations are described on a [dedicated page](../../features/binary_formats/ubjson.md).
@@ -35,6 +35,10 @@ The exact mapping and its limitations are described on a [dedicated page](../../
`IteratorType`
: a compatible iterator type
`SentinelType`
: defaults to `IteratorType`; may be a different type comparable to `IteratorType` via `operator!=`, for instance a
custom sentinel type for C++20 ranges
## Parameters
`i` (in)
@@ -44,7 +48,7 @@ The exact mapping and its limitations are described on a [dedicated page](../../
: iterator to the start of the input
`last` (in)
: iterator to the end of the input
: iterator to the end of the input, or a sentinel value that compares equal to the end iterator with `operator!=`
`strict` (in)
: whether to expect the input to be consumed until EOF (`#!cpp true` by default)
@@ -104,6 +108,7 @@ Linear in the size of the input.
- Added in version 3.1.0.
- Added `allow_exceptions` parameter in version 3.2.0.
- Extended container support (1) to include types with lvalue-only ADL `begin`/`end` (matching `std::begin`/`std::end` semantics) in version 3.13.0.
- Extended overload (2) to accept heterogeneous iterator+sentinel pairs (C++20 ranges support) in version 3.13.0.
!!! warning "Deprecation"
+13 -5
View File
@@ -10,8 +10,8 @@ static basic_json parse(InputType&& i,
const bool ignore_trailing_commas = false);
// (2)
template<typename IteratorType>
static basic_json parse(IteratorType first, IteratorType last,
template<typename IteratorType, typename SentinelType = IteratorType>
static basic_json parse(IteratorType first, SentinelType last,
const parser_callback_t cb = nullptr,
const bool allow_exceptions = true,
const bool ignore_comments = false,
@@ -19,10 +19,11 @@ static basic_json parse(IteratorType first, IteratorType last,
```
1. Deserialize from a compatible input.
2. Deserialize from a pair of character iterators
2. Deserialize from a pair of character iterators, or an iterator and a sentinel of a different type (C++20 ranges support)
The `value_type` of the iterator must be an integral type with size of 1, 2, or 4 bytes, which will be interpreted
respectively as UTF-8, UTF-16, and UTF-32.
respectively as UTF-8, UTF-16, and UTF-32. If `SentinelType` differs from `IteratorType`, it must be comparable to
the iterator type with `operator!=`.
## Template parameters
@@ -43,6 +44,12 @@ static basic_json parse(IteratorType first, IteratorType last,
- a pair of `std::string::iterator` or `std::vector<std::uint8_t>::iterator`
- a pair of pointers such as `ptr` and `ptr + len`
`SentinelType`
: defaults to `IteratorType`; may be a different type comparable to `IteratorType` via `operator!=`, for instance.
- a custom sentinel type for C++20 ranges
- `std::counted_iterator` with a different sentinel type
## Parameters
`i` (in)
@@ -67,7 +74,7 @@ static basic_json parse(IteratorType first, IteratorType last,
: iterator to the start of a character range
`last` (in)
: iterator to the end of a character range
: iterator to the end of a character range, or a sentinel value that compares equal to the end iterator with `operator!=`
## Return value
@@ -238,6 +245,7 @@ Invalid Unicode escapes and unpaired surrogates in the input are reported as
- Changed [runtime assertion](../../features/assertions.md) in case of `FILE*` null pointers to exception in version 3.12.0.
- Added `ignore_trailing_commas` in version 3.13.0.
- Extended container support (1) to include types with lvalue-only ADL `begin`/`end` (matching `std::begin`/`std::end` semantics) in version 3.13.0.
- Extended overload (2) to accept heterogeneous iterator+sentinel pairs (C++20 ranges support) in version 3.13.0.
!!! warning "Deprecation"
+10 -5
View File
@@ -11,8 +11,8 @@ static bool sax_parse(InputType&& i,
const bool ignore_trailing_commas = false);
// (2)
template<class IteratorType, class SAX>
static bool sax_parse(IteratorType first, IteratorType last,
template<class IteratorType, class SAX, class SentinelType = IteratorType>
static bool sax_parse(IteratorType first, SentinelType last,
SAX* sax,
input_format_t format = input_format_t::json,
const bool strict = true,
@@ -23,10 +23,11 @@ static bool sax_parse(IteratorType first, IteratorType last,
Read from input and generate SAX events
1. Read from a compatible input.
2. Read from a pair of character iterators
2. Read from a pair of character iterators, or an iterator and a sentinel of a different type (C++20 ranges support)
The value_type of the iterator must be an integral type with a size of 1, 2, or 4 bytes, which will be interpreted
respectively as UTF-8, UTF-16, and UTF-32.
respectively as UTF-8, UTF-16, and UTF-32. If `SentinelType` differs from `IteratorType`, it must be comparable to
the iterator type with `operator!=`.
The SAX event lister must follow the interface of [`json_sax`](../json_sax/index.md).
@@ -46,6 +47,9 @@ The SAX event lister must follow the interface of [`json_sax`](../json_sax/index
: a compatible iterator type for overload (2); a pair of character iterators whose `value_type` is an integral type
with a size of 1, 2, or 4 bytes (interpreted respectively as UTF-8, UTF-16, and UTF-32)
`SentinelType`
: defaults to `IteratorType`; may be a different type comparable to `IteratorType` via `operator!=`, for overload (2)
`SAX`
: a class fulfilling the SAX event listener interface; see [`json_sax`](../json_sax/index.md)
@@ -76,7 +80,7 @@ The SAX event lister must follow the interface of [`json_sax`](../json_sax/index
: iterator to the start of a character range
`last` (in)
: iterator to the end of a character range
: iterator to the end of a character range, or a sentinel value that compares equal to the end iterator with `operator!=`
## Return value
@@ -128,6 +132,7 @@ A UTF-8 byte order mark is silently ignored.
- Ignoring comments via `ignore_comments` added in version 3.9.0.
- Added `ignore_trailing_commas` in version 3.13.0.
- Extended container support (1) to include types with lvalue-only ADL `begin`/`end` (matching `std::begin`/`std::end` semantics) in version 3.13.0.
- Extended overload (2) to accept heterogeneous iterator+sentinel pairs (C++20 ranges support) in version 3.13.0.
!!! warning "Deprecation"