Add iterator+sentinel tests and docs for binary deserializers (#5265)

* Add iterator+sentinel tests and docs for binary deserializers

This commit extends the C++20 ranges support (iterator+sentinel pairs) to the
binary format deserializers from_cbor, from_msgpack, from_ubjson, from_bjdata,
and from_bson, matching what was already done for parse(), accept(), and
sax_parse().

Changes:
- Add istreambuf_sentinel helper to test_utils.hpp for EOF detection in tests
- Add 5 new test cases that read binary files directly via
  std::istreambuf_iterator<char> + sentinel, without pre-buffering
- Update documentation for all 5 from_* functions to document overload (3)
  with SentinelType parameter
- All tests pass; verified against existing test suite data
- Fix potential buffer over-read warning in heterogeneous iterator test

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Merge iterator+sentinel overloads and fix ambiguity/CI issues

Address PR review feedback and CI failures:

- Merge the separate same-type and sentinel-type iterator overloads of
  parse(), accept(), sax_parse(), and the five from_* binary deserializers
  into a single overload with SentinelType defaulted to IteratorType,
  as suggested in review. Applied the same simplification to the
  detail::input_adapter() free functions.
- Fix a latent ambiguity: some compilers (e.g. GCC 4.8) unreliably SFINAE
  the operator!= detection for std::nullptr_t against container/string
  types, making calls like parse(s, nullptr, ...) ambiguous with the
  compatible-input overload. can_compare_ne now explicitly excludes
  std::nullptr_t as a SentinelType.
- Use a named enable_if_t template parameter instead of an unnamed
  function parameter for the SFINAE guard, fixing a clang-tidy
  hicpp-named-parameter/readability-named-parameter failure.
- Update parse.md, accept.md, sax_parse.md, and the five from_*.md pages
  to document the merged overload instead of separate (2)/(3) overloads,
  also fixing an over-160-char line that broke the documentation
  style_check CI job.
- Rework the BSON iterator+sentinel test to parse a BSON file already
  present in the test suite instead of writing/deleting a temp file.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Fix -Wunneeded-internal-declaration for CustomSentinel in test

CustomSentinel lives in an anonymous namespace (internal linkage), and
the library's parse loop only ever evaluates the iterator-first
direction (it != last), so the reversed-order friend operator!= was
never referenced. Clang's -Weverything flags such unused internal
declarations as an error. Drop the unused overload; the used direction
is enough to satisfy can_compare_ne's either-order detection.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Fix clang-tidy hicpp-named-parameter and misc-const-correctness

- Drop the unused reversed-order operator!= overload from
  utils::istreambuf_sentinel (only iterator != sentinel is ever
  evaluated) and name the remaining friend's sentinel parameter, fixing
  hicpp-named-parameter/readability-named-parameter.
- Mark the istreambuf_iterator first/last helper variable const in the
  five binary-format sentinel tests, fixing misc-const-correctness.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Fix clang-tidy misc-const-correctness in heterogeneous sentinel test

json_str is only read via .data()/.size() and never reassigned, so
clang-tidy correctly flags it as const-able. Verified against the exact
CI job (silkeh/clang:dev, ci_clang_tidy target) by running clang-tidy
directly on this file plus the five binary-format sentinel tests
touched by prior commits; all are now clean.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

tests/src/test_utils.hpp

@@ -30,4 +30,17 @@ inline std::vector<std::uint8_t> read_binary_file(const std::string& filename)
     return byte_vector;
 }
 
+// sentinel for istreambuf_iterator; compares != true until EOF is reached
+// lets tests read a file directly via the new iterator+sentinel overloads
+// instead of buffering the whole file into a vector first.
+// Only the iterator-first direction (it != sentinel) is ever evaluated by
+// the library's parse loop, so no reversed-order overload is needed.
+struct istreambuf_sentinel
+{
+    friend bool operator!=(const std::istreambuf_iterator<char>& it, const istreambuf_sentinel& /*unused*/) noexcept
+    {
+        return it != std::istreambuf_iterator<char>();
+    }
+};
+
 } // namespace utils

tests/src/unit-bjdata.cpp

@@ -3699,6 +3699,15 @@ TEST_CASE("Universal Binary JSON Specification Examples 1")
     }
 }
 
+TEST_CASE("Parse BJData directly from a file using iterator and sentinel")
+{
+    std::string const filename = TEST_DATA_DIRECTORY "/json_testsuite/sample.json.bjdata";
+    std::ifstream file(filename, std::ios::binary);
+    const std::istreambuf_iterator<char> first(file);
+    const json parsed = json::from_bjdata(first, utils::istreambuf_sentinel{});
+    CHECK((parsed.is_object() || parsed.is_array()));
+}
+
 #if !defined(JSON_NOEXCEPTION)
 TEST_CASE("all BJData first bytes")
 {

tests/src/unit-bson.cpp

@@ -1208,6 +1208,19 @@ TEST_CASE("BSON numerical data")
     }
 }
 
+TEST_CASE("Parse BSON directly from a file using iterator and sentinel")
+{
+    std::string const filename = TEST_DATA_DIRECTORY "/json.org/1.json";
+
+    std::ifstream f_json(filename);
+    const json expected = json::parse(f_json);
+
+    std::ifstream file(filename + ".bson", std::ios::binary);
+    const std::istreambuf_iterator<char> first(file);
+    const json parsed = json::from_bson(first, utils::istreambuf_sentinel{});
+    CHECK(parsed == expected);
+}
+
 TEST_CASE("BSON roundtrips" * doctest::skip())
 {
     SECTION("reference files")

tests/src/unit-cbor.cpp

@@ -1921,6 +1921,15 @@ TEST_CASE("single CBOR roundtrip")
     }
 }
 
+TEST_CASE("Parse CBOR directly from a file using iterator and sentinel")
+{
+    std::string const filename = TEST_DATA_DIRECTORY "/json_testsuite/sample.json.cbor";
+    std::ifstream file(filename, std::ios::binary);
+    const std::istreambuf_iterator<char> first(file);
+    const json parsed = json::from_cbor(first, utils::istreambuf_sentinel{});
+    CHECK((parsed.is_object() || parsed.is_array()));
+}
+
 #if !defined(JSON_NOEXCEPTION)
 TEST_CASE("CBOR regressions")
 {

tests/src/unit-msgpack.cpp

@@ -1641,6 +1641,15 @@ TEST_CASE("single MessagePack roundtrip")
     }
 }
 
+TEST_CASE("Parse MessagePack directly from a file using iterator and sentinel")
+{
+    std::string const filename = TEST_DATA_DIRECTORY "/json_testsuite/sample.json.msgpack";
+    std::ifstream file(filename, std::ios::binary);
+    const std::istreambuf_iterator<char> first(file);
+    const json parsed = json::from_msgpack(first, utils::istreambuf_sentinel{});
+    CHECK((parsed.is_object() || parsed.is_array()));
+}
+
 TEST_CASE("MessagePack roundtrips" * doctest::skip())
 {
     SECTION("input from msgpack-python")

tests/src/unit-ubjson.cpp

@@ -2393,6 +2393,15 @@ TEST_CASE("Universal Binary JSON Specification Examples 1")
     }
 }
 
+TEST_CASE("Parse UBJSON directly from a file using iterator and sentinel")
+{
+    std::string const filename = TEST_DATA_DIRECTORY "/json_testsuite/sample.json.ubjson";
+    std::ifstream file(filename, std::ios::binary);
+    const std::istreambuf_iterator<char> first(file);
+    const json parsed = json::from_ubjson(first, utils::istreambuf_sentinel{});
+    CHECK((parsed.is_object() || parsed.is_array()));
+}
+
 #if !defined(JSON_NOEXCEPTION)
 TEST_CASE("all UBJSON first bytes")
 {

tests/src/unit-user_defined_input.cpp

@@ -168,4 +168,37 @@ TEST_CASE("Custom iterator")
     CHECK(as_json.at(3) == 4);
 }
 
+// Custom sentinel type for testing heterogeneous iterator+sentinel support
+struct CustomSentinel
+{
+    const char* end_ptr;
+
+    // only the iterator-first direction (it != sentinel) is ever evaluated by
+    // the library's parse loop; a reversed-order overload would go unused and
+    // trip -Wunneeded-internal-declaration under -Weverything
+    friend bool operator!=(const char* it, const CustomSentinel& sentinel)
+    {
+        return it != sentinel.end_ptr;
+    }
+};
+
+TEST_CASE("Parse with heterogeneous iterator and sentinel types")
+{
+    const std::string json_str = R"({"key":"value"})";
+    const char* end_ptr = json_str.data() + json_str.size();
+
+    // Parse using pointer and sentinel (different types)
+    json j = json::parse(json_str.data(), CustomSentinel{end_ptr});
+    CHECK(j["key"] == "value");
+
+    // Accept using pointer and sentinel
+    CHECK(json::accept(json_str.data(), CustomSentinel{end_ptr}));
+
+    // Test that the same-type case still works
+    std::string raw_data = R"([1,2,3])";
+    std::list<char> data(raw_data.begin(), raw_data.end());
+    json j2 = json::parse(data.begin(), data.end());
+    CHECK(j2.at(0) == 1);
+}
+
 } // namespace