fix: treat a NUL byte in the input as an ordinary byte, not EOF

The lexer's token dispatch had `case '\0':` fall through to the same
`end_of_input` handling as the real end-of-file sentinel, with a comment
claiming the NUL case was "needed when parsing from string literals".
That rationale no longer holds: input_adapter(const char*) already uses
strlen() to compute its range, so it never hands the lexer a trailing
NUL, and the const-char* overload is the only "string literal" path the
comment could be referring to. In practice, `case '\0':` only ever fired
on a genuine embedded or trailing NUL byte in real input data (e.g. a
std::string with '\0' appended), which was then silently swallowed as if
it were EOF instead of producing the parse_error.101 any other
unexpected byte gets. Two more spots in the comment-skipping logic had
the same NUL-as-EOF idiom, stopping a `//` or `/* */` comment scan early
at an embedded NUL instead of continuing to the real terminator.

Removing all three still left one real regression: input_adapter's
T(&array)[N] overload (used for a string literal like
json::parse("123"), as opposed to a decayed const char* pointer) passes
the array's full extent through unchanged, trailing '\0' included. That
path was relying on the lexer's old NUL-as-EOF behavior to make ordinary
literal parsing work at all. It now gets its own strlen()-like handling
for char arrays specifically: a single trailing NUL terminator is
excluded, mirroring the pointer overload, while non-char arrays (e.g.
uint8_t buffers for binary formats) are left untouched since a trailing
zero byte there may be data.

Also updates a few existing tests that (mostly incidentally) depended on
a trailing NUL being swallowed - std::array<uint8_t, 5>{"true"} left the
5th element zero-initialized - and adds an FAQ entry.

Signed-off-by: Niels Lohmann <niels.lohmann@gmail.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N4RQ1Ahan5YAGbnAQGjZTY

docs/mkdocs/docs/home/faq.md

@@ -90,6 +90,21 @@ The library supports **Unicode input** as follows:
 In most cases, the parser is right to complain, because the input is not UTF-8 encoded. This is especially true for Microsoft Windows, where Latin-1 or ISO 8859-1 is often the standard encoding.
 
 
+### NUL bytes in the input
+
+!!! question
+
+    Why does parsing fail with "invalid literal" or "unexpected additional data" when my input contains a `'\0'` (NUL) byte?
+
+A `'\0'` byte that occurs inside or at the end of the input is **not** treated as end-of-input; it is treated as an ordinary, invalid byte, exactly like any other unexpected byte in that position. [RFC 8259](https://tools.ietf.org/html/rfc8259.html) does not give the NUL byte any special end-of-text meaning, so a JSON text that is embedded in a larger byte sequence (for instance, a `std::string` with a trailing `'\0'` appended, or a buffer that happens to be zero-padded) will yield a `parse_error.101`, the same error you would get for any other unexpected trailing or misplaced byte:
+
+- If the NUL byte follows a complete value, parsing fails with the usual "expected end of input" message the library also gives for any other unexpected trailing byte (e.g., `json::parse(std::string("123") + '\0')` fails the same way `json::parse("123x")` does).
+- If the NUL byte occurs where a value is expected (for instance, at the very start of the input, or right after a `:` or `,`), the library reports "invalid literal".
+- A NUL byte inside a quoted string still needs to be escaped as `\u0000`, as required by [RFC 8259](https://tools.ietf.org/html/rfc8259.html#section-7); an unescaped NUL there is reported separately as a control character that must be escaped.
+
+Only the length actually passed to the parser matters here: parsing a `const char*` (for example, a string literal) uses `strlen()`-like semantics and therefore never sees the terminating NUL, so `json::parse("[1,2,3]")` is unaffected. What is affected is input that explicitly includes a NUL byte as data, such as a `std::string` with `'\0'` appended, or an iterator range/container whose end includes it.
+
+
 ### Wide string handling
 
 !!! question

include/nlohmann/detail/input/input_adapters.hpp

@@ -697,7 +697,29 @@ contiguous_bytes_input_adapter input_adapter(CharT b)
     return input_adapter(ptr, ptr + length); // cppcheck-suppress[nullPointerArithmeticRedundantCheck]
 }
 
-template<typename T, std::size_t N>
+// char arrays are usually string literals (e.g. json::parse("[1,2,3]")),
+// which the compiler pads with a trailing '\0' that is not part of the
+// text to parse; mirror the const char* overload above (which computes
+// its length with strlen()) and exclude a single trailing NUL terminator,
+// if present, so parsing a literal behaves the same whether the argument
+// decays to a pointer or binds directly to this array overload.
+template < typename T, std::size_t N,
+           typename std::enable_if<std::is_same<typename std::remove_cv<T>::type, char>::value, int>::type = 0 >
+contiguous_bytes_input_adapter input_adapter(T (&array)[N]) // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
+{
+    std::size_t length = N;
+    if (length > 0 && array[length - 1] == 0)
+    {
+        --length;
+    }
+    const auto* ptr = static_cast<const char*>(array);
+    return input_adapter(ptr, ptr + length);
+}
+
+// all other arrays (e.g. byte arrays used for binary formats) are passed
+// through unchanged, trailing zero byte included, since it may be data.
+template < typename T, std::size_t N,
+           typename std::enable_if < !std::is_same<typename std::remove_cv<T>::type, char>::value, int >::type = 0 >
 auto input_adapter(T (&array)[N]) -> decltype(input_adapter(array, array + N)) // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
 {
     return input_adapter(array, array + N);

include/nlohmann/detail/input/lexer.hpp

@@ -908,7 +908,6 @@ class lexer : public lexer_base<BasicJsonType>
                         case '\n':
                         case '\r':
                         case char_traits<char_type>::eof():
-                        case '\0':
                             return true;
 
                         default:
@@ -927,7 +926,6 @@ class lexer : public lexer_base<BasicJsonType>
                     switch (get())
                     {
                         case char_traits<char_type>::eof():
-                        case '\0':
                         {
                             error_message = "invalid comment; missing closing '*/'";
                             return false;
@@ -1758,9 +1756,7 @@ scan_number_done:
             case '9':
                 return scan_number();
 
-            // end of input (the null byte is needed when parsing from
-            // string literals)
-            case '\0':
+            // end of input
             case char_traits<char_type>::eof():
                 return token_type::end_of_input;
 

single_include/nlohmann/json.hpp

@@ -7700,7 +7700,29 @@ contiguous_bytes_input_adapter input_adapter(CharT b)
     return input_adapter(ptr, ptr + length); // cppcheck-suppress[nullPointerArithmeticRedundantCheck]
 }
 
-template<typename T, std::size_t N>
+// char arrays are usually string literals (e.g. json::parse("[1,2,3]")),
+// which the compiler pads with a trailing '\0' that is not part of the
+// text to parse; mirror the const char* overload above (which computes
+// its length with strlen()) and exclude a single trailing NUL terminator,
+// if present, so parsing a literal behaves the same whether the argument
+// decays to a pointer or binds directly to this array overload.
+template < typename T, std::size_t N,
+           typename std::enable_if<std::is_same<typename std::remove_cv<T>::type, char>::value, int>::type = 0 >
+contiguous_bytes_input_adapter input_adapter(T (&array)[N]) // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
+{
+    std::size_t length = N;
+    if (length > 0 && array[length - 1] == 0)
+    {
+        --length;
+    }
+    const auto* ptr = static_cast<const char*>(array);
+    return input_adapter(ptr, ptr + length);
+}
+
+// all other arrays (e.g. byte arrays used for binary formats) are passed
+// through unchanged, trailing zero byte included, since it may be data.
+template < typename T, std::size_t N,
+           typename std::enable_if < !std::is_same<typename std::remove_cv<T>::type, char>::value, int >::type = 0 >
 auto input_adapter(T (&array)[N]) -> decltype(input_adapter(array, array + N)) // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
 {
     return input_adapter(array, array + N);
@@ -8674,7 +8696,6 @@ class lexer : public lexer_base<BasicJsonType>
                         case '\n':
                         case '\r':
                         case char_traits<char_type>::eof():
-                        case '\0':
                             return true;
 
                         default:
@@ -8693,7 +8714,6 @@ class lexer : public lexer_base<BasicJsonType>
                     switch (get())
                     {
                         case char_traits<char_type>::eof():
-                        case '\0':
                         {
                             error_message = "invalid comment; missing closing '*/'";
                             return false;
@@ -9524,9 +9544,7 @@ scan_number_done:
             case '9':
                 return scan_number();
 
-            // end of input (the null byte is needed when parsing from
-            // string literals)
-            case '\0':
+            // end of input
             case char_traits<char_type>::eof():
                 return token_type::end_of_input;
 

tests/src/unit-class_parser.cpp

@@ -455,6 +455,59 @@ TEST_CASE("parser class")
                     json _;
                     CHECK_THROWS_WITH_AS(_ = json::parse(s.begin(), s.end()), "[json.exception.parse_error.101] parse error at line 1, column 2: syntax error while parsing value - invalid string: control character U+0000 (NUL) must be escaped to \\u0000; last read: '\"<U+0000>'", json::parse_error&);
                 }
+
+                SECTION("a NUL byte is not end of input (#5530)")
+                {
+                    // a NUL byte is an ordinary byte like any other; unlike EOF,
+                    // it does not implicitly end the input
+                    json _;
+
+                    // a NUL byte right after a complete value is unexpected
+                    // trailing data, not end of input
+                    std::string trailing_nul = "123";
+                    trailing_nul.push_back('\0');
+                    CHECK_THROWS_WITH_AS(_ = json::parse(trailing_nul),
+                                         "[json.exception.parse_error.101] parse error at line 1, column 4: syntax error while parsing value - invalid literal; last read: '123<U+0000>'; expected end of input", json::parse_error&);
+
+                    // a NUL byte where a value is expected is an ordinary
+                    // invalid byte
+                    CHECK_THROWS_WITH_AS(_ = json::parse(std::string(1, '\0')),
+                                         "[json.exception.parse_error.101] parse error at line 1, column 1: syntax error while parsing value - invalid literal; last read: '<U+0000>'", json::parse_error&);
+
+                    std::string missing_value = "{\"a\":";
+                    missing_value.push_back('\0');
+                    CHECK_THROWS_WITH_AS(_ = json::parse(missing_value),
+                                         "[json.exception.parse_error.101] parse error at line 1, column 6: syntax error while parsing value - invalid literal; last read: '\"a\":<U+0000>'", json::parse_error&);
+
+                    // an embedded NUL byte inside a single-line comment does
+                    // not stop the comment-skip loop early; scanning
+                    // continues to the actual end of the comment (a real
+                    // newline, or EOF)
+                    std::string line_comment = "// hello";
+                    line_comment.push_back('\0');
+                    line_comment += "world\n123";
+                    CHECK(json::parse(line_comment, nullptr, true, true) == json(123));
+
+                    // same for a multi-line comment: an embedded NUL byte
+                    // does not stop the scan for the closing '*/'
+                    std::string block_comment = "/* hello";
+                    block_comment.push_back('\0');
+                    block_comment += "world */123";
+                    CHECK(json::parse(block_comment, nullptr, true, true) == json(123));
+
+                    // regression check: the plain C-string convenience
+                    // overload is unaffected, because it never sees a NUL
+                    // byte in the first place - see the FAQ entry on NUL
+                    // bytes in the input
+                    CHECK(json::parse("123") == json(123));
+
+                    // regression check: the existing test above (a NUL byte
+                    // inside a quoted string) must still be rejected exactly
+                    // as before
+                    std::string in_string = "\"1\"";
+                    in_string[1] = '\0';
+                    CHECK_THROWS_WITH_AS(_ = json::parse(in_string.begin(), in_string.end()), "[json.exception.parse_error.101] parse error at line 1, column 2: syntax error while parsing value - invalid string: control character U+0000 (NUL) must be escaped to \\u0000; last read: '\"<U+0000>'", json::parse_error&);
+                }
             }
 
             SECTION("escaped")
@@ -1636,7 +1689,11 @@ TEST_CASE("parser class")
 
         SECTION("from std::array")
         {
-            std::array<uint8_t, 5> v { {'t', 'r', 'u', 'e'} };
+            // note: the array is sized to hold exactly "true" and no more;
+            // a trailing NUL byte (as a 5-element array with only 4
+            // initializers would implicitly zero-pad) is not end-of-input
+            // but ordinary (invalid, trailing) data - see issue #5530
+            std::array<uint8_t, 4> v { {'t', 'r', 'u', 'e'} };
             json j;
             json::parser(nlohmann::detail::input_adapter(std::begin(v), std::end(v))).parse(true, j);
             CHECK(j == json(true));
@@ -1777,7 +1834,7 @@ TEST_CASE("parser class")
     {
         json _;
         CHECK_THROWS_WITH_AS(_ = json::parse("/a", nullptr, true, true), "[json.exception.parse_error.101] parse error at line 1, column 2: syntax error while parsing value - invalid comment; expecting '/' or '*' after '/'; last read: '/a'", json::parse_error);
-        CHECK_THROWS_WITH_AS(_ = json::parse("/*", nullptr, true, true), "[json.exception.parse_error.101] parse error at line 1, column 3: syntax error while parsing value - invalid comment; missing closing '*/'; last read: '/*<U+0000>'", json::parse_error);
+        CHECK_THROWS_WITH_AS(_ = json::parse("/*", nullptr, true, true), "[json.exception.parse_error.101] parse error at line 1, column 3: syntax error while parsing value - invalid comment; missing closing '*/'; last read: '/*'", json::parse_error);
     }
 }
 

tests/src/unit-class_parser_diagnostic_positions.cpp

@@ -1592,7 +1592,11 @@ TEST_CASE("parser class")
 
         SECTION("from std::array")
         {
-            std::array<uint8_t, 5> v { {'t', 'r', 'u', 'e'} };
+            // note: the array is sized to hold exactly "true" and no more;
+            // a trailing NUL byte (as a 5-element array with only 4
+            // initializers would implicitly zero-pad) is not end-of-input
+            // but ordinary (invalid, trailing) data - see issue #5530
+            std::array<uint8_t, 4> v { {'t', 'r', 'u', 'e'} };
             json j;
             json::parser(nlohmann::detail::input_adapter(std::begin(v), std::end(v))).parse(true, j);
             CHECK(j == json(true));
@@ -1733,7 +1737,7 @@ TEST_CASE("parser class")
     {
         json _;
         CHECK_THROWS_WITH_AS(_ = json::parse("/a", nullptr, true, true), "[json.exception.parse_error.101] parse error at line 1, column 2: syntax error while parsing value - invalid comment; expecting '/' or '*' after '/'; last read: '/a'", json::parse_error);
-        CHECK_THROWS_WITH_AS(_ = json::parse("/*", nullptr, true, true), "[json.exception.parse_error.101] parse error at line 1, column 3: syntax error while parsing value - invalid comment; missing closing '*/'; last read: '/*<U+0000>'", json::parse_error);
+        CHECK_THROWS_WITH_AS(_ = json::parse("/*", nullptr, true, true), "[json.exception.parse_error.101] parse error at line 1, column 3: syntax error while parsing value - invalid comment; missing closing '*/'; last read: '/*'", json::parse_error);
     }
 
     // Macro for all test cases for start_pos and end_pos

tests/src/unit-deserialization.cpp

@@ -510,7 +510,9 @@ TEST_CASE("deserialization")
 
             SECTION("from std::array")
             {
-                std::array<uint8_t, 5> const v { {'t', 'r', 'u', 'e'} };
+                // note: sized to hold exactly "true"; a trailing NUL byte is
+                // not end-of-input but ordinary trailing data - see #5530
+                std::array<uint8_t, 4> const v { {'t', 'r', 'u', 'e'} };
                 CHECK(json::parse(v) == json(true));
                 CHECK(json::accept(v));
 
@@ -606,7 +608,9 @@ TEST_CASE("deserialization")
 
             SECTION("from std::array")
             {
-                std::array<uint8_t, 5> v { {'t', 'r', 'u', 'e'} };
+                // note: sized to hold exactly "true"; a trailing NUL byte is
+                // not end-of-input but ordinary trailing data - see #5530
+                std::array<uint8_t, 4> v { {'t', 'r', 'u', 'e'} };
                 CHECK(json::parse(std::begin(v), std::end(v)) == json(true));
                 CHECK(json::accept(std::begin(v), std::end(v)));
 

tests/src/unit-regression2.cpp

@@ -804,7 +804,10 @@ TEST_CASE("regression tests 2")
     SECTION("issue #2546 - parsing containers of std::byte")
     {
         const char DATA[] = R"("Hello, world!")"; // NOLINT(misc-const-correctness,cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
-        const auto s = std::as_bytes(std::span(DATA));
+        // exclude the array's implicit trailing '\0': it is not part of the
+        // JSON text and, since #5530, is no longer silently treated as
+        // end-of-input, but as ordinary (invalid, trailing) data
+        const auto s = std::as_bytes(std::span(DATA, std::size(DATA) - 1));
         const json j = json::parse(s);
         CHECK(j.dump() == "\"Hello, world!\"");
     }