test: drop a trailing NUL byte from the nativejson-benchmark 1e308 fixture

The "doubles" compliance test built its 1e308 literal as a std::string
sized to 312 bytes with an explicit trailing '\0' at index 311, relying
on the old NUL-as-EOF lexer behavior removed earlier in this branch: with
that removed, the NUL is ordinary (invalid) trailing data instead of
being silently swallowed, so json::parse() correctly rejected it with
parse_error.101 - breaking CI across every compiler/platform, since this
one nativejson-benchmark TEST_CASE is exercised by every test job.

No padding is needed at all: json::parse(const std::string&) already
uses the string's own size(), not a NUL terminator. Drop the trailing
byte and shrink the string by one to match (index 310 - the new last
index - is where ']' now goes), fixing this the same way the other
pre-existing tests in this branch that depended on the same bug were
already fixed.

Signed-off-by: Niels Lohmann <niels.lohmann@gmail.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N4RQ1Ahan5YAGbnAQGjZTY

docs/mkdocs/docs/api/basic_json/sax_parse.md

@@ -69,7 +69,8 @@ The SAX event lister must follow the interface of [`json_sax`](../json_sax/index
      [`input_format_t`](input_format_t.md) for more information
 
 `strict` (in)
-:   whether the input has to be consumed completely (optional, `#!cpp true` by default)
+:   whether the input has to be consumed completely (optional, `#!cpp true` by default); when `#!cpp false` and the
+    input is a `#!cpp std::istream`, the stream is left positioned right after the parsed value
 
 `ignore_comments` (in)
 :   whether comments should be ignored and treated like whitespace (`#!cpp true`) or yield a parse error
@@ -136,6 +137,8 @@ A UTF-8 byte order mark is silently ignored.
 - Added `ignore_trailing_commas` in version 3.13.0.
 - Extended container support (1) to include types with lvalue-only ADL `begin`/`end` (matching `std::begin`/`std::end` semantics) in version 3.13.0.
 - Extended overload (2) to accept heterogeneous iterator+sentinel pairs (C++20 ranges support) in version 3.13.0.
+- Changed in version 4.0.0 to leave a `#!cpp std::istream` positioned right after the parsed value when `strict` is
+  `#!cpp false`; see [`operator>>`](../operator_gtgt.md#notes).
 
 !!! warning "Deprecation"
 

docs/mkdocs/docs/api/operator_gtgt.md

@@ -33,41 +33,26 @@ A UTF-8 byte order mark is silently ignored.
 Invalid Unicode escapes and unpaired surrogates in the input are reported as
 [`parse_error.101`](../home/exceptions.md#jsonexceptionparse_error101) with a detailed message.
 
-`operator>>` parses exactly one JSON value, so it can be called repeatedly to read a sequence of concatenated JSON
-values from the same stream:
+`operator>>` parses exactly one JSON value and leaves the stream positioned right after it, so it can be called
+repeatedly to read a sequence of concatenated JSON values from the same stream:
 
 ```cpp
-json j1, j2;
-input >> j1;  // parses the first value
-input >> j2;  // parses the next value
+std::istringstream input("1true[2]");
+json j1, j2, j3;
+input >> j1;  // j1 == 1,      stream now positioned right after it
+input >> j2;  // j2 == true
+input >> j3;  // j3 == [2]
 ```
 
-!!! warning "A number must be followed by whitespace"
+!!! note "Changed behavior for numbers"
 
-    A number is only terminated by the character that follows it. That character is read from the stream to detect the
-    end of the number, and it is **not** put back. When a value that is a number is immediately followed by the next
-    value, the first character of that next value is lost:
+    A number is the only value whose end can be detected solely by reading the character that follows it. Up to
+    version 3.13.0 that character was consumed and not put back, so the stream was left one byte too far whenever a
+    number was immediately followed by another value: reading `1true` yielded `1` and left the stream at `rue`.
+    Values had to be separated by whitespace to work around this.
 
-    ```cpp
-    std::istringstream input("1true");
-    json j1, j2;
-    input >> j1;  // j1 == 1
-    input >> j2;  // throws parse_error.101: the stream now starts at "rue"
-    ```
-
-    Separating the values with whitespace avoids this, because the character that is eaten is then the separator:
-
-    ```cpp
-    std::istringstream input("1 true");
-    json j1, j2;
-    input >> j1;  // j1 == 1
-    input >> j2;  // j2 == true
-    ```
-
-    Only numbers are affected. Values ending in a self-delimiting character do not read past themselves, so
-    `truefalse`, `[1][2]`, `{"a":1}{"b":2}`, and `"a""b"` can be read back to back without a separator.
-
-    This is tracked in [#5340](https://github.com/nlohmann/json/issues/5340).
+    The terminating character is now only looked at and left in the stream, so no separator is required. Code that
+    relied on the extra byte being swallowed will observe it again.
 
 Note that reading concatenated values does **not** work for [JSON Lines](../features/parsing/json_lines.md)
 (newline-delimited JSON) input -- see that page for why and for the recommended alternative.
@@ -102,3 +87,5 @@ Note that reading concatenated values does **not** work for [JSON Lines](../feat
 ## Version history
 
 - Added in version 1.0.0.
+- Changed in version 4.0.0 to leave the character that terminates a number in the stream, so that the stream is
+  positioned right after the parsed value for every value type.

docs/mkdocs/docs/features/parsing/index.md

@@ -40,8 +40,7 @@ what makes it possible to read several concatenated values from the same stream,
 document followed by trailing bytes" is accepted rather than rejected. If you are validating conformance, or need to
 reject any input that is not exactly one JSON document, prefer `parse`.
 
-When using `operator>>` to read several concatenated values this way, a value that is a number must be followed by
-whitespace, because `operator>>` consumes the character that terminates a number — see the
+Values read this way do not need to be separated by whitespace; see the
 [`operator>>` notes](../../api/operator_gtgt.md#notes) for details and examples.
 
 ## SAX vs. DOM parsing

docs/mkdocs/docs/features/parsing/json_lines.md

@@ -49,5 +49,4 @@ JSON Lines input with more than one value is treated as invalid JSON by the [`pa
     with a JSON Lines input does not work, because the parser will try to parse one value after the last one.
 
     This is different from parsing a stream of *concatenated* (non-newline-delimited) JSON values, for which
-    `operator>>` does work, provided that a value that is a number is followed by whitespace -- see its
-    [notes](../../api/operator_gtgt.md#notes) for details.
+    `operator>>` does work -- see its [notes](../../api/operator_gtgt.md#notes) for details.

docs/mkdocs/docs/home/faq.md

@@ -90,6 +90,21 @@ The library supports **Unicode input** as follows:
 In most cases, the parser is right to complain, because the input is not UTF-8 encoded. This is especially true for Microsoft Windows, where Latin-1 or ISO 8859-1 is often the standard encoding.
 
 
+### NUL bytes in the input
+
+!!! question
+
+    Why does parsing fail with "invalid literal" or "unexpected additional data" when my input contains a `'\0'` (NUL) byte?
+
+A `'\0'` byte that occurs inside or at the end of the input is **not** treated as end-of-input; it is treated as an ordinary, invalid byte, exactly like any other unexpected byte in that position. [RFC 8259](https://tools.ietf.org/html/rfc8259.html) does not give the NUL byte any special end-of-text meaning, so a JSON text that is embedded in a larger byte sequence (for instance, a `std::string` with a trailing `'\0'` appended, or a buffer that happens to be zero-padded) will yield a `parse_error.101`, the same error you would get for any other unexpected trailing or misplaced byte:
+
+- If the NUL byte follows a complete value, parsing fails with the usual "expected end of input" message the library also gives for any other unexpected trailing byte (e.g., `json::parse(std::string("123") + '\0')` fails the same way `json::parse("123x")` does).
+- If the NUL byte occurs where a value is expected (for instance, at the very start of the input, or right after a `:` or `,`), the library reports "invalid literal".
+- A NUL byte inside a quoted string still needs to be escaped as `\u0000`, as required by [RFC 8259](https://tools.ietf.org/html/rfc8259.html#section-7); an unescaped NUL there is reported separately as a control character that must be escaped.
+
+Only the length actually passed to the parser matters here: parsing a `const char*` (for example, a string literal) uses `strlen()`-like semantics and therefore never sees the terminating NUL, so `json::parse("[1,2,3]")` is unaffected. What is affected is input that explicitly includes a NUL byte as data, such as a `std::string` with `'\0'` appended, or an iterator range/container whose end includes it.
+
+
 ### Wide string handling
 
 !!! question

include/nlohmann/detail/input/input_adapters.hpp

@@ -101,6 +101,9 @@ class input_stream_adapter
         // maintain ifstream flags, except eof
         if (is != nullptr)
         {
+            // consume the character last returned by get_character() unless it
+            // was given back with release_lookahead()
+            commit_lookahead();
             is->clear(is->rdstate() & std::ios::eofbit);
         }
     }
@@ -115,29 +118,60 @@ class input_stream_adapter
     input_stream_adapter& operator=(input_stream_adapter&&) = delete;
 
     input_stream_adapter(input_stream_adapter&& rhs) noexcept
-        : is(rhs.is), sb(rhs.sb)
+        : is(rhs.is), sb(rhs.sb), lookahead(rhs.lookahead)
     {
         rhs.is = nullptr;
         rhs.sb = nullptr;
+        rhs.lookahead = false;
     }
 
+    // Whether the character last returned by get_character() can be given back
+    // to the input with release_lookahead().
+    static constexpr bool supports_lookahead = true;
+
     // std::istream/std::streambuf use std::char_traits<char>::to_int_type, to
     // ensure that std::char_traits<char>::eof() and the character 0xFF do not
     // end up as the same value, e.g., 0xFFFFFFFF.
+    //
+    // The character is peeked rather than consumed: it is only stepped over
+    // once the next character is requested, or when the adapter is destroyed.
+    // Until then, release_lookahead() can leave it in the input.
     std::char_traits<char>::int_type get_character()
     {
-        auto res = sb->sbumpc();
+        if (lookahead)
+        {
+            // step over the character returned by the previous call
+            sb->sbumpc();
+        }
+
+        auto res = sb->sgetc();
         // set eof manually, as we don't use the istream interface.
         if (JSON_HEDLEY_UNLIKELY(res == std::char_traits<char>::eof()))
         {
+            // there is nothing to step over next time
+            lookahead = false;
             is->clear(is->rdstate() | std::ios::eofbit);
         }
+        else
+        {
+            lookahead = true;
+        }
         return res;
     }
 
+    // Leave the character last returned by get_character() in the input, so
+    // that the next read from the stream - by this adapter or by the caller
+    // once parsing is done - sees it again. Unlike putting a consumed
+    // character back, this cannot fail.
+    void release_lookahead() noexcept
+    {
+        lookahead = false;
+    }
+
     template<class T>
     std::size_t get_elements(T* dest, std::size_t count = 1)
     {
+        commit_lookahead();
         auto res = static_cast<std::size_t>(sb->sgetn(reinterpret_cast<char*>(dest), static_cast<std::streamsize>(count * sizeof(T))));
         if (JSON_HEDLEY_UNLIKELY(res < count * sizeof(T)))
         {
@@ -147,9 +181,23 @@ class input_stream_adapter
     }
 
   private:
+    // Step over the character last returned by get_character(). The character
+    // has already been peeked successfully, so for every streambuf with a get
+    // area this is a pointer increment that cannot fail.
+    void commit_lookahead()
+    {
+        if (lookahead)
+        {
+            lookahead = false;
+            sb->sbumpc();
+        }
+    }
+
     /// the associated input stream
     std::istream* is = nullptr;
     std::streambuf* sb = nullptr;
+    /// whether get_character() peeked a character that is not consumed yet
+    bool lookahead = false;
 };
 #endif  // JSON_NO_IO
 
@@ -649,7 +697,29 @@ contiguous_bytes_input_adapter input_adapter(CharT b)
     return input_adapter(ptr, ptr + length); // cppcheck-suppress[nullPointerArithmeticRedundantCheck]
 }
 
-template<typename T, std::size_t N>
+// char arrays are usually string literals (e.g. json::parse("[1,2,3]")),
+// which the compiler pads with a trailing '\0' that is not part of the
+// text to parse; mirror the const char* overload above (which computes
+// its length with strlen()) and exclude a single trailing NUL terminator,
+// if present, so parsing a literal behaves the same whether the argument
+// decays to a pointer or binds directly to this array overload.
+template < typename T, std::size_t N,
+           typename std::enable_if<std::is_same<typename std::remove_cv<T>::type, char>::value, int>::type = 0 >
+contiguous_bytes_input_adapter input_adapter(T (&array)[N]) // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
+{
+    std::size_t length = N;
+    if (length > 0 && array[length - 1] == 0)
+    {
+        --length;
+    }
+    const auto* ptr = static_cast<const char*>(array);
+    return input_adapter(ptr, ptr + length);
+}
+
+// all other arrays (e.g. byte arrays used for binary formats) are passed
+// through unchanged, trailing zero byte included, since it may be data.
+template < typename T, std::size_t N,
+           typename std::enable_if < !std::is_same<typename std::remove_cv<T>::type, char>::value, int >::type = 0 >
 auto input_adapter(T (&array)[N]) -> decltype(input_adapter(array, array + N)) // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
 {
     return input_adapter(array, array + N);

include/nlohmann/detail/input/lexer.hpp

@@ -125,6 +125,24 @@ constexpr bool input_adapter_supports_seek(std::false_type /*detected*/)
     return false;
 }
 
+// Detect whether an input adapter reads with one character of lookahead that
+// can be left in the input (see input_stream_adapter::supports_lookahead),
+// detected like supports_seek above.
+template<typename InputAdapterType>
+using detect_supports_lookahead = decltype(InputAdapterType::supports_lookahead);
+
+template<typename InputAdapterType>
+constexpr bool input_adapter_supports_lookahead(std::true_type /*detected*/)
+{
+    return InputAdapterType::supports_lookahead;
+}
+
+template<typename InputAdapterType>
+constexpr bool input_adapter_supports_lookahead(std::false_type /*detected*/)
+{
+    return false;
+}
+
 /*!
 @brief lexical analysis
 
@@ -146,6 +164,12 @@ class lexer : public lexer_base<BasicJsonType>
     static constexpr bool lazy_token_string =
         input_adapter_supports_seek<InputAdapterType>(is_detected<detect_supports_seek, InputAdapterType> {});
 
+    /// whether a simulated unget can be passed on to the input adapter, which
+    /// then leaves the character in the input; see
+    /// input_adapter_supports_lookahead
+    static constexpr bool can_release_lookahead =
+        input_adapter_supports_lookahead<InputAdapterType>(is_detected<detect_supports_lookahead, InputAdapterType> {});
+
   public:
     using token_type = typename lexer_base<BasicJsonType>::token_type;
 
@@ -884,7 +908,6 @@ class lexer : public lexer_base<BasicJsonType>
                         case '\n':
                         case '\r':
                         case char_traits<char_type>::eof():
-                        case '\0':
                             return true;
 
                         default:
@@ -903,7 +926,6 @@ class lexer : public lexer_base<BasicJsonType>
                     switch (get())
                     {
                         case char_traits<char_type>::eof():
-                        case '\0':
                         {
                             error_message = "invalid comment; missing closing '*/'";
                             return false;
@@ -1461,6 +1483,21 @@ scan_number_done:
         uncapture_char(std::integral_constant<bool, lazy_token_string> {});
     }
 
+    /// adapter without lookahead: nothing to do (see release_lookahead)
+    void release_lookahead_impl(std::false_type /*can_release*/) const noexcept {}
+
+    /// adapter with lookahead: leave the character in the input instead
+    void release_lookahead_impl(std::true_type /*can_release*/)
+    {
+        if (next_unget)
+        {
+            // the character is read from the input again rather than replayed
+            // from current, so the adapter must not step over it
+            next_unget = false;
+            ia.release_lookahead();
+        }
+    }
+
     /// seekable adapter: nothing was captured, so nothing to undo
     void uncapture_char(std::true_type /*lazy*/) const noexcept {}
 
@@ -1524,6 +1561,29 @@ scan_number_done:
         return position;
     }
 
+    /*!
+    @brief pass a pending simulated unget on to the input
+
+    unget() only rewinds the lexer's own bookkeeping, so the character that
+    terminated the last token (e.g. the character after a number) would still
+    be stepped over when the input adapter is done. Callers that hand the
+    input back to the user afterwards - operator>> and non-strict sax_parse -
+    call this once when scanning is done, so that the input is positioned
+    right after the value.
+
+    Adapters without lookahead (see input_adapter_supports_lookahead) are not
+    handed back to the user, so this is a no-op for them.
+
+    Scanning may continue after this call: @a next_unget is cleared, and the
+    character is read from the input again instead of being replayed from
+    @a current. A pending unget of EOF needs no special case, because reaching
+    EOF leaves no lookahead to release.
+    */
+    void release_lookahead()
+    {
+        release_lookahead_impl(std::integral_constant<bool, can_release_lookahead> {});
+    }
+
 #if JSON_DIAGNOSTIC_POSITIONS
     /// return the offset of the first character of the last read token; unlike
     /// the token's parsed value, this accounts for escape sequences
@@ -1696,9 +1756,7 @@ scan_number_done:
             case '9':
                 return scan_number();
 
-            // end of input (the null byte is needed when parsing from
-            // string literals)
-            case '\0':
+            // end of input
             case char_traits<char_type>::eof():
                 return token_type::end_of_input;
 

include/nlohmann/detail/input/parser.hpp

@@ -99,14 +99,23 @@ class parser
             json_sax_dom_callback_parser<BasicJsonType, InputAdapterType> sdp(result, callback, allow_exceptions, &m_lexer);
             sax_parse_internal(&sdp);
 
+            if (strict)
+            {
                 // in strict mode, input must be completely read
-            if (strict && (get_token() != token_type::end_of_input))
+                if (get_token() != token_type::end_of_input)
                 {
                     sdp.parse_error(m_lexer.get_position(),
                                     m_lexer.get_token_string(),
                                     parse_error::create(101, m_lexer.get_position(),
                                                         exception_message(token_type::end_of_input, "value"), nullptr));
                 }
+            }
+            else
+            {
+                // the caller keeps using the input: position it right after
+                // the value by leaving the character that terminated it
+                m_lexer.release_lookahead();
+            }
 
             // in case of an error, return a discarded value
             if (sdp.is_errored())
@@ -127,13 +136,21 @@ class parser
             json_sax_dom_parser<BasicJsonType, InputAdapterType> sdp(result, allow_exceptions, &m_lexer);
             sax_parse_internal(&sdp);
 
+            if (strict)
+            {
                 // in strict mode, input must be completely read
-            if (strict && (get_token() != token_type::end_of_input))
+                if (get_token() != token_type::end_of_input)
                 {
                     sdp.parse_error(m_lexer.get_position(),
                                     m_lexer.get_token_string(),
                                     parse_error::create(101, m_lexer.get_position(), exception_message(token_type::end_of_input, "value"), nullptr));
                 }
+            }
+            else
+            {
+                // see above
+                m_lexer.release_lookahead();
+            }
 
             // in case of an error, return a discarded value
             if (sdp.is_errored())
@@ -165,13 +182,25 @@ class parser
         (void)detail::is_sax_static_asserts<SAX, BasicJsonType> {};
         const bool result = sax_parse_internal(sax);
 
+        if (result)
+        {
+            if (strict)
+            {
                 // strict mode: next byte must be EOF
-        if (result && strict && (get_token() != token_type::end_of_input))
+                if (get_token() != token_type::end_of_input)
                 {
                     return sax->parse_error(m_lexer.get_position(),
                                             m_lexer.get_token_string(),
                                             parse_error::create(101, m_lexer.get_position(), exception_message(token_type::end_of_input, "value"), nullptr));
                 }
+            }
+            else
+            {
+                // the caller keeps using the input: position it right after
+                // the value by leaving the character that terminated it
+                m_lexer.release_lookahead();
+            }
+        }
 
         return result;
     }

single_include/nlohmann/json.hpp

@@ -7104,6 +7104,9 @@ class input_stream_adapter
         // maintain ifstream flags, except eof
         if (is != nullptr)
         {
+            // consume the character last returned by get_character() unless it
+            // was given back with release_lookahead()
+            commit_lookahead();
             is->clear(is->rdstate() & std::ios::eofbit);
         }
     }
@@ -7118,29 +7121,60 @@ class input_stream_adapter
     input_stream_adapter& operator=(input_stream_adapter&&) = delete;
 
     input_stream_adapter(input_stream_adapter&& rhs) noexcept
-        : is(rhs.is), sb(rhs.sb)
+        : is(rhs.is), sb(rhs.sb), lookahead(rhs.lookahead)
     {
         rhs.is = nullptr;
         rhs.sb = nullptr;
+        rhs.lookahead = false;
     }
 
+    // Whether the character last returned by get_character() can be given back
+    // to the input with release_lookahead().
+    static constexpr bool supports_lookahead = true;
+
     // std::istream/std::streambuf use std::char_traits<char>::to_int_type, to
     // ensure that std::char_traits<char>::eof() and the character 0xFF do not
     // end up as the same value, e.g., 0xFFFFFFFF.
+    //
+    // The character is peeked rather than consumed: it is only stepped over
+    // once the next character is requested, or when the adapter is destroyed.
+    // Until then, release_lookahead() can leave it in the input.
     std::char_traits<char>::int_type get_character()
     {
-        auto res = sb->sbumpc();
+        if (lookahead)
+        {
+            // step over the character returned by the previous call
+            sb->sbumpc();
+        }
+
+        auto res = sb->sgetc();
         // set eof manually, as we don't use the istream interface.
         if (JSON_HEDLEY_UNLIKELY(res == std::char_traits<char>::eof()))
         {
+            // there is nothing to step over next time
+            lookahead = false;
             is->clear(is->rdstate() | std::ios::eofbit);
         }
+        else
+        {
+            lookahead = true;
+        }
         return res;
     }
 
+    // Leave the character last returned by get_character() in the input, so
+    // that the next read from the stream - by this adapter or by the caller
+    // once parsing is done - sees it again. Unlike putting a consumed
+    // character back, this cannot fail.
+    void release_lookahead() noexcept
+    {
+        lookahead = false;
+    }
+
     template<class T>
     std::size_t get_elements(T* dest, std::size_t count = 1)
     {
+        commit_lookahead();
         auto res = static_cast<std::size_t>(sb->sgetn(reinterpret_cast<char*>(dest), static_cast<std::streamsize>(count * sizeof(T))));
         if (JSON_HEDLEY_UNLIKELY(res < count * sizeof(T)))
         {
@@ -7150,9 +7184,23 @@ class input_stream_adapter
     }
 
   private:
+    // Step over the character last returned by get_character(). The character
+    // has already been peeked successfully, so for every streambuf with a get
+    // area this is a pointer increment that cannot fail.
+    void commit_lookahead()
+    {
+        if (lookahead)
+        {
+            lookahead = false;
+            sb->sbumpc();
+        }
+    }
+
     /// the associated input stream
     std::istream* is = nullptr;
     std::streambuf* sb = nullptr;
+    /// whether get_character() peeked a character that is not consumed yet
+    bool lookahead = false;
 };
 #endif  // JSON_NO_IO
 
@@ -7652,7 +7700,29 @@ contiguous_bytes_input_adapter input_adapter(CharT b)
     return input_adapter(ptr, ptr + length); // cppcheck-suppress[nullPointerArithmeticRedundantCheck]
 }
 
-template<typename T, std::size_t N>
+// char arrays are usually string literals (e.g. json::parse("[1,2,3]")),
+// which the compiler pads with a trailing '\0' that is not part of the
+// text to parse; mirror the const char* overload above (which computes
+// its length with strlen()) and exclude a single trailing NUL terminator,
+// if present, so parsing a literal behaves the same whether the argument
+// decays to a pointer or binds directly to this array overload.
+template < typename T, std::size_t N,
+           typename std::enable_if<std::is_same<typename std::remove_cv<T>::type, char>::value, int>::type = 0 >
+contiguous_bytes_input_adapter input_adapter(T (&array)[N]) // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
+{
+    std::size_t length = N;
+    if (length > 0 && array[length - 1] == 0)
+    {
+        --length;
+    }
+    const auto* ptr = static_cast<const char*>(array);
+    return input_adapter(ptr, ptr + length);
+}
+
+// all other arrays (e.g. byte arrays used for binary formats) are passed
+// through unchanged, trailing zero byte included, since it may be data.
+template < typename T, std::size_t N,
+           typename std::enable_if < !std::is_same<typename std::remove_cv<T>::type, char>::value, int >::type = 0 >
 auto input_adapter(T (&array)[N]) -> decltype(input_adapter(array, array + N)) // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
 {
     return input_adapter(array, array + N);
@@ -7843,6 +7913,24 @@ constexpr bool input_adapter_supports_seek(std::false_type /*detected*/)
     return false;
 }
 
+// Detect whether an input adapter reads with one character of lookahead that
+// can be left in the input (see input_stream_adapter::supports_lookahead),
+// detected like supports_seek above.
+template<typename InputAdapterType>
+using detect_supports_lookahead = decltype(InputAdapterType::supports_lookahead);
+
+template<typename InputAdapterType>
+constexpr bool input_adapter_supports_lookahead(std::true_type /*detected*/)
+{
+    return InputAdapterType::supports_lookahead;
+}
+
+template<typename InputAdapterType>
+constexpr bool input_adapter_supports_lookahead(std::false_type /*detected*/)
+{
+    return false;
+}
+
 /*!
 @brief lexical analysis
 
@@ -7864,6 +7952,12 @@ class lexer : public lexer_base<BasicJsonType>
     static constexpr bool lazy_token_string =
         input_adapter_supports_seek<InputAdapterType>(is_detected<detect_supports_seek, InputAdapterType> {});
 
+    /// whether a simulated unget can be passed on to the input adapter, which
+    /// then leaves the character in the input; see
+    /// input_adapter_supports_lookahead
+    static constexpr bool can_release_lookahead =
+        input_adapter_supports_lookahead<InputAdapterType>(is_detected<detect_supports_lookahead, InputAdapterType> {});
+
   public:
     using token_type = typename lexer_base<BasicJsonType>::token_type;
 
@@ -8602,7 +8696,6 @@ class lexer : public lexer_base<BasicJsonType>
                         case '\n':
                         case '\r':
                         case char_traits<char_type>::eof():
-                        case '\0':
                             return true;
 
                         default:
@@ -8621,7 +8714,6 @@ class lexer : public lexer_base<BasicJsonType>
                     switch (get())
                     {
                         case char_traits<char_type>::eof():
-                        case '\0':
                         {
                             error_message = "invalid comment; missing closing '*/'";
                             return false;
@@ -9179,6 +9271,21 @@ scan_number_done:
         uncapture_char(std::integral_constant<bool, lazy_token_string> {});
     }
 
+    /// adapter without lookahead: nothing to do (see release_lookahead)
+    void release_lookahead_impl(std::false_type /*can_release*/) const noexcept {}
+
+    /// adapter with lookahead: leave the character in the input instead
+    void release_lookahead_impl(std::true_type /*can_release*/)
+    {
+        if (next_unget)
+        {
+            // the character is read from the input again rather than replayed
+            // from current, so the adapter must not step over it
+            next_unget = false;
+            ia.release_lookahead();
+        }
+    }
+
     /// seekable adapter: nothing was captured, so nothing to undo
     void uncapture_char(std::true_type /*lazy*/) const noexcept {}
 
@@ -9242,6 +9349,29 @@ scan_number_done:
         return position;
     }
 
+    /*!
+    @brief pass a pending simulated unget on to the input
+
+    unget() only rewinds the lexer's own bookkeeping, so the character that
+    terminated the last token (e.g. the character after a number) would still
+    be stepped over when the input adapter is done. Callers that hand the
+    input back to the user afterwards - operator>> and non-strict sax_parse -
+    call this once when scanning is done, so that the input is positioned
+    right after the value.
+
+    Adapters without lookahead (see input_adapter_supports_lookahead) are not
+    handed back to the user, so this is a no-op for them.
+
+    Scanning may continue after this call: @a next_unget is cleared, and the
+    character is read from the input again instead of being replayed from
+    @a current. A pending unget of EOF needs no special case, because reaching
+    EOF leaves no lookahead to release.
+    */
+    void release_lookahead()
+    {
+        release_lookahead_impl(std::integral_constant<bool, can_release_lookahead> {});
+    }
+
 #if JSON_DIAGNOSTIC_POSITIONS
     /// return the offset of the first character of the last read token; unlike
     /// the token's parsed value, this accounts for escape sequences
@@ -9414,9 +9544,7 @@ scan_number_done:
             case '9':
                 return scan_number();
 
-            // end of input (the null byte is needed when parsing from
-            // string literals)
-            case '\0':
+            // end of input
             case char_traits<char_type>::eof():
                 return token_type::end_of_input;
 
@@ -14005,14 +14133,23 @@ class parser
             json_sax_dom_callback_parser<BasicJsonType, InputAdapterType> sdp(result, callback, allow_exceptions, &m_lexer);
             sax_parse_internal(&sdp);
 
+            if (strict)
+            {
                 // in strict mode, input must be completely read
-            if (strict && (get_token() != token_type::end_of_input))
+                if (get_token() != token_type::end_of_input)
                 {
                     sdp.parse_error(m_lexer.get_position(),
                                     m_lexer.get_token_string(),
                                     parse_error::create(101, m_lexer.get_position(),
                                                         exception_message(token_type::end_of_input, "value"), nullptr));
                 }
+            }
+            else
+            {
+                // the caller keeps using the input: position it right after
+                // the value by leaving the character that terminated it
+                m_lexer.release_lookahead();
+            }
 
             // in case of an error, return a discarded value
             if (sdp.is_errored())
@@ -14033,13 +14170,21 @@ class parser
             json_sax_dom_parser<BasicJsonType, InputAdapterType> sdp(result, allow_exceptions, &m_lexer);
             sax_parse_internal(&sdp);
 
+            if (strict)
+            {
                 // in strict mode, input must be completely read
-            if (strict && (get_token() != token_type::end_of_input))
+                if (get_token() != token_type::end_of_input)
                 {
                     sdp.parse_error(m_lexer.get_position(),
                                     m_lexer.get_token_string(),
                                     parse_error::create(101, m_lexer.get_position(), exception_message(token_type::end_of_input, "value"), nullptr));
                 }
+            }
+            else
+            {
+                // see above
+                m_lexer.release_lookahead();
+            }
 
             // in case of an error, return a discarded value
             if (sdp.is_errored())
@@ -14071,13 +14216,25 @@ class parser
         (void)detail::is_sax_static_asserts<SAX, BasicJsonType> {};
         const bool result = sax_parse_internal(sax);
 
+        if (result)
+        {
+            if (strict)
+            {
                 // strict mode: next byte must be EOF
-        if (result && strict && (get_token() != token_type::end_of_input))
+                if (get_token() != token_type::end_of_input)
                 {
                     return sax->parse_error(m_lexer.get_position(),
                                             m_lexer.get_token_string(),
                                             parse_error::create(101, m_lexer.get_position(), exception_message(token_type::end_of_input, "value"), nullptr));
                 }
+            }
+            else
+            {
+                // the caller keeps using the input: position it right after
+                // the value by leaving the character that terminated it
+                m_lexer.release_lookahead();
+            }
+        }
 
         return result;
     }

tests/src/unit-class_parser.cpp

@@ -455,6 +455,59 @@ TEST_CASE("parser class")
                     json _;
                     CHECK_THROWS_WITH_AS(_ = json::parse(s.begin(), s.end()), "[json.exception.parse_error.101] parse error at line 1, column 2: syntax error while parsing value - invalid string: control character U+0000 (NUL) must be escaped to \\u0000; last read: '\"<U+0000>'", json::parse_error&);
                 }
+
+                SECTION("a NUL byte is not end of input (#5530)")
+                {
+                    // a NUL byte is an ordinary byte like any other; unlike EOF,
+                    // it does not implicitly end the input
+                    json _;
+
+                    // a NUL byte right after a complete value is unexpected
+                    // trailing data, not end of input
+                    std::string trailing_nul = "123";
+                    trailing_nul.push_back('\0');
+                    CHECK_THROWS_WITH_AS(_ = json::parse(trailing_nul),
+                                         "[json.exception.parse_error.101] parse error at line 1, column 4: syntax error while parsing value - invalid literal; last read: '123<U+0000>'; expected end of input", json::parse_error&);
+
+                    // a NUL byte where a value is expected is an ordinary
+                    // invalid byte
+                    CHECK_THROWS_WITH_AS(_ = json::parse(std::string(1, '\0')),
+                                         "[json.exception.parse_error.101] parse error at line 1, column 1: syntax error while parsing value - invalid literal; last read: '<U+0000>'", json::parse_error&);
+
+                    std::string missing_value = "{\"a\":";
+                    missing_value.push_back('\0');
+                    CHECK_THROWS_WITH_AS(_ = json::parse(missing_value),
+                                         "[json.exception.parse_error.101] parse error at line 1, column 6: syntax error while parsing value - invalid literal; last read: '\"a\":<U+0000>'", json::parse_error&);
+
+                    // an embedded NUL byte inside a single-line comment does
+                    // not stop the comment-skip loop early; scanning
+                    // continues to the actual end of the comment (a real
+                    // newline, or EOF)
+                    std::string line_comment = "// hello";
+                    line_comment.push_back('\0');
+                    line_comment += "world\n123";
+                    CHECK(json::parse(line_comment, nullptr, true, true) == json(123));
+
+                    // same for a multi-line comment: an embedded NUL byte
+                    // does not stop the scan for the closing '*/'
+                    std::string block_comment = "/* hello";
+                    block_comment.push_back('\0');
+                    block_comment += "world */123";
+                    CHECK(json::parse(block_comment, nullptr, true, true) == json(123));
+
+                    // regression check: the plain C-string convenience
+                    // overload is unaffected, because it never sees a NUL
+                    // byte in the first place - see the FAQ entry on NUL
+                    // bytes in the input
+                    CHECK(json::parse("123") == json(123));
+
+                    // regression check: the existing test above (a NUL byte
+                    // inside a quoted string) must still be rejected exactly
+                    // as before
+                    std::string in_string = "\"1\"";
+                    in_string[1] = '\0';
+                    CHECK_THROWS_WITH_AS(_ = json::parse(in_string.begin(), in_string.end()), "[json.exception.parse_error.101] parse error at line 1, column 2: syntax error while parsing value - invalid string: control character U+0000 (NUL) must be escaped to \\u0000; last read: '\"<U+0000>'", json::parse_error&);
+                }
             }
 
             SECTION("escaped")
@@ -1636,7 +1689,11 @@ TEST_CASE("parser class")
 
         SECTION("from std::array")
         {
-            std::array<uint8_t, 5> v { {'t', 'r', 'u', 'e'} };
+            // note: the array is sized to hold exactly "true" and no more;
+            // a trailing NUL byte (as a 5-element array with only 4
+            // initializers would implicitly zero-pad) is not end-of-input
+            // but ordinary (invalid, trailing) data - see issue #5530
+            std::array<uint8_t, 4> v { {'t', 'r', 'u', 'e'} };
             json j;
             json::parser(nlohmann::detail::input_adapter(std::begin(v), std::end(v))).parse(true, j);
             CHECK(j == json(true));
@@ -1777,7 +1834,7 @@ TEST_CASE("parser class")
     {
         json _;
         CHECK_THROWS_WITH_AS(_ = json::parse("/a", nullptr, true, true), "[json.exception.parse_error.101] parse error at line 1, column 2: syntax error while parsing value - invalid comment; expecting '/' or '*' after '/'; last read: '/a'", json::parse_error);
-        CHECK_THROWS_WITH_AS(_ = json::parse("/*", nullptr, true, true), "[json.exception.parse_error.101] parse error at line 1, column 3: syntax error while parsing value - invalid comment; missing closing '*/'; last read: '/*<U+0000>'", json::parse_error);
+        CHECK_THROWS_WITH_AS(_ = json::parse("/*", nullptr, true, true), "[json.exception.parse_error.101] parse error at line 1, column 3: syntax error while parsing value - invalid comment; missing closing '*/'; last read: '/*'", json::parse_error);
     }
 }
 

tests/src/unit-class_parser_diagnostic_positions.cpp

@@ -1592,7 +1592,11 @@ TEST_CASE("parser class")
 
         SECTION("from std::array")
         {
-            std::array<uint8_t, 5> v { {'t', 'r', 'u', 'e'} };
+            // note: the array is sized to hold exactly "true" and no more;
+            // a trailing NUL byte (as a 5-element array with only 4
+            // initializers would implicitly zero-pad) is not end-of-input
+            // but ordinary (invalid, trailing) data - see issue #5530
+            std::array<uint8_t, 4> v { {'t', 'r', 'u', 'e'} };
             json j;
             json::parser(nlohmann::detail::input_adapter(std::begin(v), std::end(v))).parse(true, j);
             CHECK(j == json(true));
@@ -1733,7 +1737,7 @@ TEST_CASE("parser class")
     {
         json _;
         CHECK_THROWS_WITH_AS(_ = json::parse("/a", nullptr, true, true), "[json.exception.parse_error.101] parse error at line 1, column 2: syntax error while parsing value - invalid comment; expecting '/' or '*' after '/'; last read: '/a'", json::parse_error);
-        CHECK_THROWS_WITH_AS(_ = json::parse("/*", nullptr, true, true), "[json.exception.parse_error.101] parse error at line 1, column 3: syntax error while parsing value - invalid comment; missing closing '*/'; last read: '/*<U+0000>'", json::parse_error);
+        CHECK_THROWS_WITH_AS(_ = json::parse("/*", nullptr, true, true), "[json.exception.parse_error.101] parse error at line 1, column 3: syntax error while parsing value - invalid comment; missing closing '*/'; last read: '/*'", json::parse_error);
     }
 
     // Macro for all test cases for start_pos and end_pos

tests/src/unit-deserialization.cpp

@@ -14,10 +14,15 @@ using nlohmann::json;
     using namespace nlohmann::literals; // NOLINT(google-build-using-namespace)
 #endif
 
+#include <cstddef>
 #include <iostream>
 #include <iterator>
 #include <sstream>
+#include <streambuf>
+#include <string>
+#include <utility>
 #include <valarray>
+#include <vector>
 
 #if defined(_WIN32)
     #define NOMINMAX
@@ -219,6 +224,58 @@ class proxy_iterator
     iterator* m_it = nullptr;
 };
 
+// A streambuf that keeps no get area at all and refuses every putback: with an
+// empty get area, sungetc() always ends up in pbackfail(). Used to check that
+// the character terminating a number is left in the input without relying on
+// the streambuf being able to put a consumed character back.
+class no_putback_streambuf : public std::streambuf
+{
+  public:
+    explicit no_putback_streambuf(std::string s) : m_data(std::move(s)) {}
+
+  protected:
+    // peek at the next character without consuming it
+    int_type underflow() override
+    {
+        if (m_pos >= m_data.size())
+        {
+            return traits_type::eof();
+        }
+        return traits_type::to_int_type(m_data[m_pos]);
+    }
+
+    // consume the next character
+    int_type uflow() override
+    {
+        if (m_pos >= m_data.size())
+        {
+            return traits_type::eof();
+        }
+        return traits_type::to_int_type(m_data[m_pos++]);
+    }
+
+    int_type pbackfail(int_type /*c*/) override
+    {
+        return traits_type::eof();
+    }
+
+  private:
+    std::string m_data;
+    std::size_t m_pos = 0;
+};
+
+// read the characters that are left in a stream
+std::string remaining(std::istream& is)
+{
+    std::string result;
+    char c = 0;
+    while (is.get(c))
+    {
+        result += c;
+    }
+    return result;
+}
+
 // JSON_HAS_CPP_20
 #if defined(__cpp_char8_t)
 bool check_utf8()
@@ -453,7 +510,9 @@ TEST_CASE("deserialization")
 
             SECTION("from std::array")
             {
-                std::array<uint8_t, 5> const v { {'t', 'r', 'u', 'e'} };
+                // note: sized to hold exactly "true"; a trailing NUL byte is
+                // not end-of-input but ordinary trailing data - see #5530
+                std::array<uint8_t, 4> const v { {'t', 'r', 'u', 'e'} };
                 CHECK(json::parse(v) == json(true));
                 CHECK(json::accept(v));
 
@@ -549,7 +608,9 @@ TEST_CASE("deserialization")
 
             SECTION("from std::array")
             {
-                std::array<uint8_t, 5> v { {'t', 'r', 'u', 'e'} };
+                // note: sized to hold exactly "true"; a trailing NUL byte is
+                // not end-of-input but ordinary trailing data - see #5530
+                std::array<uint8_t, 4> v { {'t', 'r', 'u', 'e'} };
                 CHECK(json::parse(std::begin(v), std::end(v)) == json(true));
                 CHECK(json::accept(std::begin(v), std::end(v)));
 
@@ -1181,6 +1242,122 @@ TEST_CASE("deserialization")
         }
     }
 
+    SECTION("stream position after extraction (#5340)")
+    {
+        SECTION("a number does not consume the character that terminates it")
+        {
+            // a number is only terminated by the character following it; that
+            // character must be given back so the stream is positioned right
+            // after the value
+            const std::vector<std::pair<std::string, std::string>> tests =
+            {
+                {"1true", "true"},
+                {"1[2]", "[2]"},
+                {"1{}", "{}"},
+                {R"(1"a")", R"("a")"},
+                {"1 true", " true"},
+                {"12,", ","},
+                {"-0.5e3x", "x"},
+                {"1null", "null"}
+            };
+
+            for (const auto& test : tests)
+            {
+                CAPTURE(test.first);
+                std::istringstream ss(test.first);
+                json j;
+                ss >> j;
+                CHECK(j == json::parse(test.first.substr(0, test.first.size() - test.second.size())));
+                CHECK(remaining(ss) == test.second);
+            }
+        }
+
+        SECTION("values that are self-delimiting are unaffected")
+        {
+            const std::vector<std::pair<std::string, std::string>> tests =
+            {
+                {"truefalse", "false"},
+                {"[1][2]", "[2]"},
+                {R"({"a":1}{"b":2})", R"({"b":2})"},
+                {R"("a""b")", R"("b")"},
+                {"null null", " null"}
+            };
+
+            for (const auto& test : tests)
+            {
+                CAPTURE(test.first);
+                std::istringstream ss(test.first);
+                json j;
+                ss >> j;
+                CHECK(remaining(ss) == test.second);
+            }
+        }
+
+        SECTION("a number at the end of the input leaves nothing behind")
+        {
+            for (const std::string s :
+                    {"1", "12", "-3.5e2", " 7 "
+                    })
+            {
+                CAPTURE(s);
+                std::istringstream ss(s);
+                json j;
+                ss >> j;
+                CHECK(remaining(ss).find_first_not_of(" \t\n\r") == std::string::npos);
+            }
+        }
+
+        SECTION("repeated extraction of concatenated values")
+        {
+            std::istringstream ss(R"(1true[2]3"x"{"a":4}5)");
+            const std::vector<json> expected =
+            {
+                json(1), json(true), json::parse("[2]"), json(3),
+                json("x"), json::parse(R"({"a":4})"), json(5)
+            };
+
+            for (const auto& e : expected)
+            {
+                json j;
+                ss >> j;
+                CHECK(j == e);
+            }
+        }
+
+        SECTION("sax_parse with strict == false")
+        {
+            std::istringstream ss("1true");
+            SaxEventLogger l;
+            CHECK(json::sax_parse(ss, &l, nlohmann::detail::input_format_t::json, false));
+            CHECK(l.events.size() == 1);
+            CHECK(l.events[0] == "number_unsigned(1)");
+            CHECK(remaining(ss) == "true");
+        }
+
+        SECTION("strict parsing still rejects trailing data")
+        {
+            std::istringstream ss("1true");
+            json _;
+            CHECK_THROWS_WITH_AS(_ = json::parse(ss),
+                                 "[json.exception.parse_error.101] parse error at line 1, column 5: syntax error while parsing value - unexpected true literal; expected end of input", json::parse_error&);
+
+            std::istringstream ss2("1true");
+            CHECK_FALSE(json::accept(ss2));
+        }
+
+        SECTION("a streambuf that cannot put back is not needed")
+        {
+            // the terminating character is never consumed, so no putback
+            // position is required
+            no_putback_streambuf buf("1true");
+            std::istream is(&buf);
+            json j;
+            is >> j;
+            CHECK(j == json(1));
+            CHECK(remaining(is) == "true");
+        }
+    }
+
     // build with C++20
     // JSON_HAS_CPP_20
 #if defined(__cpp_char8_t)

tests/src/unit-regression2.cpp

@@ -804,7 +804,10 @@ TEST_CASE("regression tests 2")
     SECTION("issue #2546 - parsing containers of std::byte")
     {
         const char DATA[] = R"("Hello, world!")"; // NOLINT(misc-const-correctness,cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
-        const auto s = std::as_bytes(std::span(DATA));
+        // exclude the array's implicit trailing '\0': it is not part of the
+        // JSON text and, since #5530, is no longer silently treated as
+        // end-of-input, but as ordinary (invalid, trailing) data
+        const auto s = std::as_bytes(std::span(DATA, std::size(DATA) - 1));
         const json j = json::parse(s);
         CHECK(j.dump() == "\"Hello, world!\"");
     }

tests/src/unit-testsuites.cpp

@@ -214,11 +214,13 @@ TEST_CASE("compliance tests from nativejson-benchmark")
                     5708990770823839524233143877797980545530986496.0);
 
         {
-            std::string n1e308(312, '0');   // '1' followed by 308 '0'
+            // note: no trailing NUL byte - a NUL is ordinary (invalid) data
+            // now, not end-of-input, so it is no longer needed (or valid)
+            // padding here; see issue #5530
+            std::string n1e308(311, '0');   // '1' followed by 308 '0'
             n1e308[0] = '[';
             n1e308[1] = '1';
             n1e308[310] = ']';
-            n1e308[311] = '\0';
             TEST_DOUBLE(n1e308, 1E308);
         }