Report a missing 0xFF after a final BON8 string also without strict

A BON8 string ends at the first byte that cannot continue it. When a
string ends a value and is not terminated by 0xFF, that byte belongs to
whatever follows the value, but the reader has already taken it from the
input and keeps it in its push-back buffer. With strict = true the
end-of-input check reports it; with strict = false the buffer was
discarded, so a following value in a stream lost its first byte or bytes
without an error ({0x61, 0x90} read "a" and then dropped the 0 that
follows).

Report parse_error.110 "string at the end of the value is not terminated
by 0xFF" when the push-back buffer is not empty once a BON8 value is
complete and strict is false, as bon8.md requires the final 0xFF. Output
of to_bon8 always has it, so canonical input is not affected.

Fixes #6110.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

docs/mkdocs/docs/api/basic_json/from_bon8.md

@@ -69,8 +69,10 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va
 
 ## Exceptions
 
-- Throws [parse_error.110](../../home/exceptions.md#jsonexceptionparse_error110) if the given input ends prematurely or
-  the end of the file was not reached when `strict` was set to true
+- Throws [parse_error.110](../../home/exceptions.md#jsonexceptionparse_error110) if the given input ends prematurely,
+  if the end of the file was not reached when `strict` was set to true, or if a string at the end of the value is not
+  terminated by 0xFF although a byte follows it (also when `strict` is false: the byte that ends the string is
+  taken from the input, so the next value would be lost)
 - Throws [parse_error.112](../../home/exceptions.md#jsonexceptionparse_error112) if a parse error occurs, for instance
   an invalid byte, a string that is not valid UTF-8, or an object key that is not a string
 

docs/mkdocs/docs/features/binary_formats/bon8.md

@@ -181,7 +181,10 @@ of the message are reported as [parse_error.110](../../home/exceptions.md#jsonex
     arrays and objects with up to four elements that are terminated by 0xFE, unsorted object keys, or a 0xFF after a
     string that would also end without it, are accepted. A second 0xFF is not a terminator but an empty string.
 
-    Strings must be valid UTF-8, and a string at the very end of a message must be terminated by 0xFF.
+    Strings must be valid UTF-8, and a string at the very end of a message must be terminated by 0xFF. This also
+    holds when `strict` is false and more data follows the value: the byte that would end the string belongs to the
+    next value, and reading it would remove it from the input, so the missing 0xFF is reported as
+    [parse_error.110](../../home/exceptions.md#jsonexceptionparse_error110).
 
     Duplicate object keys are accepted; the last value wins (`ordered_json` keeps the position of the first occurrence).
     The specification does not say how to treat them; the HikoGUI reference decoder keeps the first value.

docs/mkdocs/docs/home/exceptions.md

@@ -140,8 +140,6 @@ Exceptions have ids 1xx.
 
 This error indicates a syntax error while deserializing a JSON text. The error message describes that an unexpected token (character) was encountered, and the member `byte` indicates the error position.
 
-The message ends with the last read token (`last read: '...'`), with control characters written as `<U+XXXX>`. A token of more than 8192 characters is shown by its first and its last 4096 characters, separated by `...`.
-
 !!! failure "Example message"
 
     Input ended prematurely:

include/nlohmann/detail/input/binary_reader.hpp

@@ -177,6 +177,17 @@ class binary_reader
                 JSON_ASSERT(false); // NOLINT(cert-dcl03-c,hicpp-static-assert,misc-static-assert) LCOV_EXCL_LINE
         }
 
+        // BON8: a string ends at the first byte that cannot continue it. If
+        // that byte follows the value, the reader has already taken it from
+        // the input and holds it in bon8_pushback. Without the end-of-input
+        // check below it would be dropped, and the next read would start
+        // behind it, so report the missing 0xFF instead.
+        if (result && !strict && input_format == input_format_t::bon8 && bon8_pushback_size != 0)
+        {
+            get_bon8();
+            return last_byte_error(exception_id::unexpected_end_of_input, "string at the end of the value is not terminated by 0xFF", "value", "");
+        }
+
         // strict mode: next byte must be EOF
         if (result && strict)
         {

include/nlohmann/detail/input/input_adapters.hpp

@@ -110,164 +110,87 @@ inline const char* empty_input_message() noexcept
     return "attempting to parse an empty input; check that your input string or stream contains the expected JSON";
 }
 
+////////////////////
+// input adapters //
+////////////////////
+
+#ifndef JSON_NO_IO
 /*!
-@brief the characters of the last read token, kept for error messages
-
 A growable array of characters for the part of a token that an input adapter
-copied out of its window, or that the lexer copies while it scans input it
-cannot read again: unlike std::vector::push_back(), appending a character
-takes no reference to a local, and is a few instructions when there is room.
-
-The token is only needed for error messages, which show its first head()
-characters and its last head() characters (see lexer::get_token_string). The
-buffer therefore keeps the first head() and at least the last tail()
-characters: when it is full and holds more than limit() characters, it drops
-the characters in between instead of growing. That happens where it would
-otherwise reallocate, so appending stays as cheap as before, and a long string
-or a long run of whitespace read from a stream is not kept a second time.
+copied out of its window: unlike std::vector<char>::push_back(), appending a
+character takes no reference to a local, and is a few instructions when there
+is room.
 */
-template<typename CharType>
-class token_text_buffer
+class token_carry_buffer
 {
   public:
-    static constexpr std::size_t head() noexcept
-    {
-        return 4096;
-    }
-    /// a little more than head(), as the last character may be handed back
-    static constexpr std::size_t tail() noexcept
-    {
-        return head() + 16;
-    }
-    static constexpr std::size_t limit() noexcept
-    {
-        return head() + (2 * tail());
-    }
-
-    token_text_buffer() = default;
-    token_text_buffer(const token_text_buffer&) = delete;
-    token_text_buffer& operator=(const token_text_buffer&) = delete;
-    token_text_buffer(token_text_buffer&& rhs) noexcept
+    token_carry_buffer() = default;
+    token_carry_buffer(const token_carry_buffer&) = delete;
+    token_carry_buffer& operator=(const token_carry_buffer&) = delete;
+    token_carry_buffer(token_carry_buffer&& rhs) noexcept
         : m_data(std::move(rhs.m_data)), m_end(rhs.m_end), m_capacity_end(rhs.m_capacity_end)
     {
         rhs.m_end = rhs.m_capacity_end = nullptr;
     }
-    token_text_buffer& operator=(token_text_buffer&& rhs) noexcept
-    {
-        m_data = std::move(rhs.m_data);
-        m_end = rhs.m_end;
-        m_capacity_end = rhs.m_capacity_end;
-        rhs.m_end = rhs.m_capacity_end = nullptr;
-        return *this;
-    }
-    ~token_text_buffer() = default;
+    token_carry_buffer& operator=(token_carry_buffer&&) = delete;
+    ~token_carry_buffer() = default;
 
     void clear() noexcept
     {
         m_end = m_data.get();
     }
 
-    bool empty() const noexcept
-    {
-        return m_end == m_data.get();
-    }
-
-    void push_back(CharType c)
+    void push_back(char c)
     {
         if (JSON_HEDLEY_UNLIKELY(m_end == m_capacity_end))
         {
-            make_room(1);
+            grow(1);
         }
         *m_end++ = c;
     }
 
-    /// drop the last character
-    void pop_back() noexcept
-    {
-        JSON_ASSERT(!empty());
-        --m_end;
-    }
-
-    void append(const CharType* first, const CharType* last)
+    void append(const char* first, const char* last)
     {
         const auto n = static_cast<std::size_t>(last - first);
         const auto room = static_cast<std::size_t>(m_capacity_end - m_end);
-        if (JSON_HEDLEY_UNLIKELY(n > room))
+        if (n > room)
         {
-            append_long(first, n);
-            return;
+            grow(n);
         }
         if (n != 0)
         {
-            std::memcpy(m_end, first, n * sizeof(CharType));
+            std::memcpy(m_end, first, n);
         }
         m_end += n;
     }
 
     /// replace the contents of @a out
-    void copy_to(std::vector<CharType>& out) const
+    void copy_to(std::vector<char>& out) const
     {
         out.assign(m_data.get(), m_end);
     }
 
   private:
-    /// append @a n characters that do not fit, in pieces of at most tail()
-    /// characters, so that dropping the middle keeps the last tail() of them
-    JSON_HEDLEY_NEVER_INLINE void append_long(const CharType* first, std::size_t n)
+    /// make room for @a extra more characters
+    JSON_HEDLEY_NEVER_INLINE void grow(std::size_t extra)
     {
-        while (n != 0)
+        const auto size = static_cast<std::size_t>(m_end - m_data.get());
+        const auto capacity = static_cast<std::size_t>(m_capacity_end - m_data.get());
+        const std::size_t new_capacity = (std::max)(capacity < 32 ? static_cast<std::size_t>(32) : 2 * capacity, size + extra);
+        std::unique_ptr<char[]> data(new char[new_capacity]); // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
+        if (size != 0)
         {
-            const std::size_t piece = n < tail() ? n : tail();
-            make_room(piece);
-            std::memcpy(m_end, first, piece * sizeof(CharType));
-            m_end += piece;
-            first += piece;
-            n -= piece;
+            std::memcpy(data.get(), m_data.get(), size);
         }
+        m_data = std::move(data);
+        m_end = m_data.get() + size;
+        m_capacity_end = m_data.get() + new_capacity;
     }
 
-    /// make room for @a extra (at most tail()) more characters: drop the
-    /// middle of a long token instead of growing beyond limit()
-    JSON_HEDLEY_NEVER_INLINE void make_room(std::size_t extra)
-    {
-        // nothing is allocated before the first character
-        std::size_t size = m_data ? static_cast<std::size_t>(m_end - m_data.get()) : 0;
-        if (size + extra > limit() && size > head() + tail())
-        {
-            std::memmove(m_data.get() + head(), m_end - tail(), tail() * sizeof(CharType));
-            m_end = m_data.get() + head() + tail();
-            size = head() + tail();
-        }
-        const std::size_t capacity = m_data ? static_cast<std::size_t>(m_capacity_end - m_data.get()) : 0;
-        if (size + extra > capacity)
-        {
-            // never more than limit(): from there on, the middle is dropped
-            const std::size_t doubled = capacity < 32 ? static_cast<std::size_t>(32) : 2 * capacity;
-            const std::size_t new_capacity = (std::max)(doubled < limit() ? doubled : limit(), size + extra);
-            std::unique_ptr<CharType[]> data(new CharType[new_capacity]); // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
-            if (size != 0)
-            {
-                std::memcpy(data.get(), m_data.get(), size * sizeof(CharType));
-            }
-            m_data = std::move(data);
-            m_end = m_data.get() + size;
-            m_capacity_end = m_data.get() + new_capacity;
-        }
-    }
-
-    std::unique_ptr<CharType[]> m_data{}; // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays,readability-redundant-member-init)
-    CharType* m_end = nullptr;
-    CharType* m_capacity_end = nullptr;
+    std::unique_ptr<char[]> m_data{}; // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays,readability-redundant-member-init)
+    char* m_end = nullptr;
+    char* m_capacity_end = nullptr;
 };
-
-/// the part of a token that an input adapter copied out of its window
-using token_carry_buffer = token_text_buffer<char>;
-
-////////////////////
-// input adapters //
-////////////////////
-
-#ifndef JSON_NO_IO
 /*!
 Input adapter for stdio file access. The file is locked for the lifetime of the
 adapter and read with the unlocked stdio functions (see

include/nlohmann/detail/input/lexer.hpp

@@ -494,10 +494,7 @@ class lexer : public lexer_base<BasicJsonType>
     /// streaming adapter: capture the characters
     void bulk_track_token(const char* data, std::size_t n, std::false_type /*lazy*/)
     {
-        for (std::size_t i = 0; i < n; ++i)
-        {
-            token_string.push_back(static_cast<char_type>(data[i]));
-        }
+        token_string.insert(token_string.end(), data, data + n);
     }
 
     /// streaming input: no bulk fast path
@@ -2240,28 +2237,15 @@ scan_number_done:
         const std::size_t stop = ia.get_consumed_count() - (pending_real_unget ? 1u : 0u);
         if (JSON_HEDLEY_LIKELY(stop >= token_string_start))
         {
-            // only the beginning and the end of a long token are shown (see
-            // get_token_string), so copy only these
-            constexpr std::size_t head = token_text_buffer<char_type>::head();
-            constexpr std::size_t tail = token_text_buffer<char_type>::tail();
-            if (stop - token_string_start > head + tail)
-            {
-                ia.copy_consumed_range(token_string_start, token_string_start + head, out);
-                ia.copy_consumed_range(stop - tail, stop, out);
-            }
-            else
-            {
-                ia.copy_consumed_range(token_string_start, stop, out);
-            }
+            ia.copy_consumed_range(token_string_start, stop, out);
         }
         return out;
     }
 
     /// streaming adapter: the token was copied eagerly while scanning
-    const std::vector<char_type>& collect_token_chars(std::vector<char_type>& out, std::false_type /*lazy*/) const
+    const std::vector<char_type>& collect_token_chars(std::vector<char_type>& /*out*/, std::false_type /*lazy*/) const
     {
-        token_string.copy_to(out);
-        return out;
+        return token_string;
     }
 
     /// return the last read token (for errors only).  Will never contain EOF
@@ -2272,32 +2256,10 @@ scan_number_done:
         std::vector<char_type> reconstructed;
         const std::vector<char_type>& chars = collect_token_chars(reconstructed, std::integral_constant<bool, lazy_token_string> {});
 
-        // a long token is shown by its first and its last head() characters;
-        // the buffers that keep it while it is read keep no more than these
-        // and a few more (see token_text_buffer)
-        constexpr std::size_t shown = token_text_buffer<char_type>::head();
+        // escape control characters
         std::string result;
-        if (chars.size() > 2 * shown)
+        for (const auto c : chars)
         {
-            append_escaped(result, chars.begin(), chars.begin() + static_cast<std::ptrdiff_t>(shown));
-            result += "...";
-            append_escaped(result, chars.end() - static_cast<std::ptrdiff_t>(shown), chars.end());
-        }
-        else
-        {
-            append_escaped(result, chars.begin(), chars.end());
-        }
-        return result;
-    }
-
-  private:
-    /// append the characters [first, last) to @a result, escaping control characters
-    template<typename Iterator>
-    static void append_escaped(std::string& result, Iterator first, Iterator last)
-    {
-        for (; first != last; ++first)
-        {
-            const auto c = *first;
             if (static_cast<unsigned char>(c) <= '\x1F')
             {
                 // escape control characters
@@ -2311,9 +2273,9 @@ scan_number_done:
                 result.push_back(static_cast<std::string::value_type>(c));
             }
         }
-    }
 
-  public:
+        return result;
+    }
 
     /// return syntax error message
     JSON_HEDLEY_RETURNS_NON_NULL
@@ -2563,7 +2525,7 @@ scan_number_done:
 
     /// raw input token string for error messages; only populated for streaming
     /// adapters (seekable adapters reconstruct it lazily via token_string_start)
-    token_text_buffer<char_type> token_string {};
+    std::vector<char_type> token_string {};
 
     /// start offset of the current token within the input, used to reconstruct
     /// the last read token on error for seekable adapters (see collect_token_chars)

single_include/nlohmann/json.hpp

@@ -9323,164 +9323,87 @@ inline const char* empty_input_message() noexcept
     return "attempting to parse an empty input; check that your input string or stream contains the expected JSON";
 }
 
+////////////////////
+// input adapters //
+////////////////////
+
+#ifndef JSON_NO_IO
 /*!
-@brief the characters of the last read token, kept for error messages
-
 A growable array of characters for the part of a token that an input adapter
-copied out of its window, or that the lexer copies while it scans input it
-cannot read again: unlike std::vector::push_back(), appending a character
-takes no reference to a local, and is a few instructions when there is room.
-
-The token is only needed for error messages, which show its first head()
-characters and its last head() characters (see lexer::get_token_string). The
-buffer therefore keeps the first head() and at least the last tail()
-characters: when it is full and holds more than limit() characters, it drops
-the characters in between instead of growing. That happens where it would
-otherwise reallocate, so appending stays as cheap as before, and a long string
-or a long run of whitespace read from a stream is not kept a second time.
+copied out of its window: unlike std::vector<char>::push_back(), appending a
+character takes no reference to a local, and is a few instructions when there
+is room.
 */
-template<typename CharType>
-class token_text_buffer
+class token_carry_buffer
 {
   public:
-    static constexpr std::size_t head() noexcept
-    {
-        return 4096;
-    }
-    /// a little more than head(), as the last character may be handed back
-    static constexpr std::size_t tail() noexcept
-    {
-        return head() + 16;
-    }
-    static constexpr std::size_t limit() noexcept
-    {
-        return head() + (2 * tail());
-    }
-
-    token_text_buffer() = default;
-    token_text_buffer(const token_text_buffer&) = delete;
-    token_text_buffer& operator=(const token_text_buffer&) = delete;
-    token_text_buffer(token_text_buffer&& rhs) noexcept
+    token_carry_buffer() = default;
+    token_carry_buffer(const token_carry_buffer&) = delete;
+    token_carry_buffer& operator=(const token_carry_buffer&) = delete;
+    token_carry_buffer(token_carry_buffer&& rhs) noexcept
         : m_data(std::move(rhs.m_data)), m_end(rhs.m_end), m_capacity_end(rhs.m_capacity_end)
     {
         rhs.m_end = rhs.m_capacity_end = nullptr;
     }
-    token_text_buffer& operator=(token_text_buffer&& rhs) noexcept
-    {
-        m_data = std::move(rhs.m_data);
-        m_end = rhs.m_end;
-        m_capacity_end = rhs.m_capacity_end;
-        rhs.m_end = rhs.m_capacity_end = nullptr;
-        return *this;
-    }
-    ~token_text_buffer() = default;
+    token_carry_buffer& operator=(token_carry_buffer&&) = delete;
+    ~token_carry_buffer() = default;
 
     void clear() noexcept
     {
         m_end = m_data.get();
     }
 
-    bool empty() const noexcept
-    {
-        return m_end == m_data.get();
-    }
-
-    void push_back(CharType c)
+    void push_back(char c)
     {
         if (JSON_HEDLEY_UNLIKELY(m_end == m_capacity_end))
         {
-            make_room(1);
+            grow(1);
         }
         *m_end++ = c;
     }
 
-    /// drop the last character
-    void pop_back() noexcept
-    {
-        JSON_ASSERT(!empty());
-        --m_end;
-    }
-
-    void append(const CharType* first, const CharType* last)
+    void append(const char* first, const char* last)
     {
         const auto n = static_cast<std::size_t>(last - first);
         const auto room = static_cast<std::size_t>(m_capacity_end - m_end);
-        if (JSON_HEDLEY_UNLIKELY(n > room))
+        if (n > room)
         {
-            append_long(first, n);
-            return;
+            grow(n);
         }
         if (n != 0)
         {
-            std::memcpy(m_end, first, n * sizeof(CharType));
+            std::memcpy(m_end, first, n);
         }
         m_end += n;
     }
 
     /// replace the contents of @a out
-    void copy_to(std::vector<CharType>& out) const
+    void copy_to(std::vector<char>& out) const
     {
         out.assign(m_data.get(), m_end);
     }
 
   private:
-    /// append @a n characters that do not fit, in pieces of at most tail()
-    /// characters, so that dropping the middle keeps the last tail() of them
-    JSON_HEDLEY_NEVER_INLINE void append_long(const CharType* first, std::size_t n)
+    /// make room for @a extra more characters
+    JSON_HEDLEY_NEVER_INLINE void grow(std::size_t extra)
     {
-        while (n != 0)
+        const auto size = static_cast<std::size_t>(m_end - m_data.get());
+        const auto capacity = static_cast<std::size_t>(m_capacity_end - m_data.get());
+        const std::size_t new_capacity = (std::max)(capacity < 32 ? static_cast<std::size_t>(32) : 2 * capacity, size + extra);
+        std::unique_ptr<char[]> data(new char[new_capacity]); // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
+        if (size != 0)
         {
-            const std::size_t piece = n < tail() ? n : tail();
-            make_room(piece);
-            std::memcpy(m_end, first, piece * sizeof(CharType));
-            m_end += piece;
-            first += piece;
-            n -= piece;
+            std::memcpy(data.get(), m_data.get(), size);
         }
+        m_data = std::move(data);
+        m_end = m_data.get() + size;
+        m_capacity_end = m_data.get() + new_capacity;
     }
 
-    /// make room for @a extra (at most tail()) more characters: drop the
-    /// middle of a long token instead of growing beyond limit()
-    JSON_HEDLEY_NEVER_INLINE void make_room(std::size_t extra)
-    {
-        // nothing is allocated before the first character
-        std::size_t size = m_data ? static_cast<std::size_t>(m_end - m_data.get()) : 0;
-        if (size + extra > limit() && size > head() + tail())
-        {
-            std::memmove(m_data.get() + head(), m_end - tail(), tail() * sizeof(CharType));
-            m_end = m_data.get() + head() + tail();
-            size = head() + tail();
-        }
-        const std::size_t capacity = m_data ? static_cast<std::size_t>(m_capacity_end - m_data.get()) : 0;
-        if (size + extra > capacity)
-        {
-            // never more than limit(): from there on, the middle is dropped
-            const std::size_t doubled = capacity < 32 ? static_cast<std::size_t>(32) : 2 * capacity;
-            const std::size_t new_capacity = (std::max)(doubled < limit() ? doubled : limit(), size + extra);
-            std::unique_ptr<CharType[]> data(new CharType[new_capacity]); // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
-            if (size != 0)
-            {
-                std::memcpy(data.get(), m_data.get(), size * sizeof(CharType));
-            }
-            m_data = std::move(data);
-            m_end = m_data.get() + size;
-            m_capacity_end = m_data.get() + new_capacity;
-        }
-    }
-
-    std::unique_ptr<CharType[]> m_data{}; // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays,readability-redundant-member-init)
-    CharType* m_end = nullptr;
-    CharType* m_capacity_end = nullptr;
+    std::unique_ptr<char[]> m_data{}; // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays,readability-redundant-member-init)
+    char* m_end = nullptr;
+    char* m_capacity_end = nullptr;
 };
-
-/// the part of a token that an input adapter copied out of its window
-using token_carry_buffer = token_text_buffer<char>;
-
-////////////////////
-// input adapters //
-////////////////////
-
-#ifndef JSON_NO_IO
 /*!
 Input adapter for stdio file access. The file is locked for the lifetime of the
 adapter and read with the unlocked stdio functions (see
@@ -13975,10 +13898,7 @@ class lexer : public lexer_base<BasicJsonType>
     /// streaming adapter: capture the characters
     void bulk_track_token(const char* data, std::size_t n, std::false_type /*lazy*/)
     {
-        for (std::size_t i = 0; i < n; ++i)
-        {
-            token_string.push_back(static_cast<char_type>(data[i]));
-        }
+        token_string.insert(token_string.end(), data, data + n);
     }
 
     /// streaming input: no bulk fast path
@@ -15721,28 +15641,15 @@ scan_number_done:
         const std::size_t stop = ia.get_consumed_count() - (pending_real_unget ? 1u : 0u);
         if (JSON_HEDLEY_LIKELY(stop >= token_string_start))
         {
-            // only the beginning and the end of a long token are shown (see
-            // get_token_string), so copy only these
-            constexpr std::size_t head = token_text_buffer<char_type>::head();
-            constexpr std::size_t tail = token_text_buffer<char_type>::tail();
-            if (stop - token_string_start > head + tail)
-            {
-                ia.copy_consumed_range(token_string_start, token_string_start + head, out);
-                ia.copy_consumed_range(stop - tail, stop, out);
-            }
-            else
-            {
-                ia.copy_consumed_range(token_string_start, stop, out);
-            }
+            ia.copy_consumed_range(token_string_start, stop, out);
         }
         return out;
     }
 
     /// streaming adapter: the token was copied eagerly while scanning
-    const std::vector<char_type>& collect_token_chars(std::vector<char_type>& out, std::false_type /*lazy*/) const
+    const std::vector<char_type>& collect_token_chars(std::vector<char_type>& /*out*/, std::false_type /*lazy*/) const
     {
-        token_string.copy_to(out);
-        return out;
+        return token_string;
     }
 
     /// return the last read token (for errors only).  Will never contain EOF
@@ -15753,32 +15660,10 @@ scan_number_done:
         std::vector<char_type> reconstructed;
         const std::vector<char_type>& chars = collect_token_chars(reconstructed, std::integral_constant<bool, lazy_token_string> {});
 
-        // a long token is shown by its first and its last head() characters;
-        // the buffers that keep it while it is read keep no more than these
-        // and a few more (see token_text_buffer)
-        constexpr std::size_t shown = token_text_buffer<char_type>::head();
+        // escape control characters
         std::string result;
-        if (chars.size() > 2 * shown)
+        for (const auto c : chars)
         {
-            append_escaped(result, chars.begin(), chars.begin() + static_cast<std::ptrdiff_t>(shown));
-            result += "...";
-            append_escaped(result, chars.end() - static_cast<std::ptrdiff_t>(shown), chars.end());
-        }
-        else
-        {
-            append_escaped(result, chars.begin(), chars.end());
-        }
-        return result;
-    }
-
-  private:
-    /// append the characters [first, last) to @a result, escaping control characters
-    template<typename Iterator>
-    static void append_escaped(std::string& result, Iterator first, Iterator last)
-    {
-        for (; first != last; ++first)
-        {
-            const auto c = *first;
             if (static_cast<unsigned char>(c) <= '\x1F')
             {
                 // escape control characters
@@ -15792,9 +15677,9 @@ scan_number_done:
                 result.push_back(static_cast<std::string::value_type>(c));
             }
         }
-    }
 
-  public:
+        return result;
+    }
 
     /// return syntax error message
     JSON_HEDLEY_RETURNS_NON_NULL
@@ -16044,7 +15929,7 @@ scan_number_done:
 
     /// raw input token string for error messages; only populated for streaming
     /// adapters (seekable adapters reconstruct it lazily via token_string_start)
-    token_text_buffer<char_type> token_string {};
+    std::vector<char_type> token_string {};
 
     /// start offset of the current token within the input, used to reconstruct
     /// the last read token on error for seekable adapters (see collect_token_chars)
@@ -18683,6 +18568,17 @@ class binary_reader
                 JSON_ASSERT(false); // NOLINT(cert-dcl03-c,hicpp-static-assert,misc-static-assert) LCOV_EXCL_LINE
         }
 
+        // BON8: a string ends at the first byte that cannot continue it. If
+        // that byte follows the value, the reader has already taken it from
+        // the input and holds it in bon8_pushback. Without the end-of-input
+        // check below it would be dropped, and the next read would start
+        // behind it, so report the missing 0xFF instead.
+        if (result && !strict && input_format == input_format_t::bon8 && bon8_pushback_size != 0)
+        {
+            get_bon8();
+            return last_byte_error(exception_id::unexpected_end_of_input, "string at the end of the value is not terminated by 0xFF", "value", "");
+        }
+
         // strict mode: next byte must be EOF
         if (result && strict)
         {

tests/src/unit-bon8.cpp

@@ -608,14 +608,26 @@ TEST_CASE("BON8")
             CHECK_THROWS_WITH_AS(_ = json::from_bon8(vec), "[json.exception.parse_error.110] parse error at byte 2: syntax error while parsing BON8 value: expected end of input; last byte: 0x90", json::parse_error&);
             CHECK(json::from_bon8(vec, true, false).is_discarded());
 
-            // the byte after a string that ends a container
+            // the byte after a string that ends a container: without strict, the
+            // missing 0xFF is reported, as the byte has been taken from the input
             const bytes vec2 = {0x81, 'a', 0x90};
-            CHECK(json::from_bon8(vec2, false) == json({"a"}));
+            CHECK_THROWS_WITH_AS(_ = json::from_bon8(vec2, false), "[json.exception.parse_error.110] parse error at byte 3: syntax error while parsing BON8 value: string at the end of the value is not terminated by 0xFF; last byte: 0x90", json::parse_error&);
+            CHECK(json::from_bon8(vec2, false, false).is_discarded());
             CHECK_THROWS_WITH_AS(_ = json::from_bon8(vec2), "[json.exception.parse_error.110] parse error at byte 3: syntax error while parsing BON8 value: expected end of input; last byte: 0x90", json::parse_error&);
 
             const bytes vec3 = {0x81, 'a', 0xC2, 0x00};
-            CHECK(json::from_bon8(vec3, false) == json({"a"}));
+            CHECK_THROWS_WITH_AS(_ = json::from_bon8(vec3, false), "[json.exception.parse_error.110] parse error at byte 3: syntax error while parsing BON8 value: string at the end of the value is not terminated by 0xFF; last byte: 0xC2", json::parse_error&);
             CHECK_THROWS_WITH_AS(_ = json::from_bon8(vec3), "[json.exception.parse_error.110] parse error at byte 3: syntax error while parsing BON8 value: expected end of input; last byte: 0xC2", json::parse_error&);
+
+            // with a terminated string, strict = false leaves the next value in the stream
+            std::istringstream terminated(std::string("\x61\xFF\x90", 3));
+            CHECK(json::from_bon8(terminated, false) == "a");
+            CHECK(terminated.tellg() == 2);
+            CHECK(json::from_bon8(terminated, false) == 0);
+
+            // without the terminator, the next value would be lost
+            std::istringstream unterminated(std::string("\x61\x90", 2));
+            CHECK_THROWS_WITH_AS(_ = json::from_bon8(unterminated, false), "[json.exception.parse_error.110] parse error at byte 2: syntax error while parsing BON8 value: string at the end of the value is not terminated by 0xFF; last byte: 0x90", json::parse_error&);
         }
     }
 

tests/src/unit-class_lexer.cpp

@@ -12,7 +12,6 @@
 #include <nlohmann/json.hpp>
 using nlohmann::json;
 
-#include <algorithm> // min
 #include <array> // array
 #include <cctype> // toupper
 #include <cstdint> // uint32_t, uint64_t
@@ -2234,123 +2233,3 @@ TEST_CASE("serialization of short strings")
     CHECK(json("caf\xC3\xA9").dump() == "\"caf\xC3\xA9\"");
     CHECK(json("caf\xC3\xA9").dump(-1, ' ', true) == "\"caf\\u00e9\"");
 }
-
-TEST_CASE("token_text_buffer keeps the beginning and the end of a long token")
-{
-    using buffer_t = nlohmann::detail::token_text_buffer<char>;
-    constexpr std::size_t head = buffer_t::head();
-    constexpr std::size_t tail = buffer_t::tail();
-
-    // the characters 0, 1, ..., 250, 0, 1, ...
-    const auto at = [](std::size_t i)
-    {
-        return static_cast<char>(i % 251);
-    };
-    // whether out holds the first head and the last tail characters of a
-    // token of n characters (and maybe more characters of its middle)
-    const auto keeps_ends = [&](const std::vector<char>& out, std::size_t n)
-    {
-        if (out.size() < head + tail || out.size() > buffer_t::limit())
-        {
-            return false;
-        }
-        for (std::size_t i = 0; i < head; ++i)
-        {
-            if (out[i] != at(i))
-            {
-                return false;
-            }
-        }
-        for (std::size_t i = 0; i < tail; ++i)
-        {
-            if (out[out.size() - tail + i] != at(n - tail + i))
-            {
-                return false;
-            }
-        }
-        return true;
-    };
-
-    SECTION("a short token is kept as it is")
-    {
-        buffer_t buffer;
-        CHECK(buffer.empty());
-        for (std::size_t i = 0; i < head + tail; ++i)
-        {
-            buffer.push_back(at(i));
-        }
-        std::vector<char> out;
-        buffer.copy_to(out);
-        REQUIRE(out.size() == head + tail);
-        bool same = true;
-        for (std::size_t i = 0; i < out.size(); ++i)
-        {
-            same = same && out[i] == at(i);
-        }
-        CHECK(same);
-    }
-
-    SECTION("characters added one by one")
-    {
-        for (const std::size_t n :
-                {
-                    head + tail + 1, buffer_t::limit(), buffer_t::limit() + 1, static_cast<std::size_t>(100000)
-                })
-        {
-            CAPTURE(n)
-            buffer_t buffer;
-            for (std::size_t i = 0; i < n; ++i)
-            {
-                buffer.push_back(at(i));
-            }
-            std::vector<char> out;
-            buffer.copy_to(out);
-            CHECK(keeps_ends(out, n));
-
-            // the last character can be handed back
-            buffer.pop_back();
-            buffer.copy_to(out);
-            CHECK(keeps_ends(out, n - 1));
-        }
-    }
-
-    SECTION("characters added in pieces of different sizes")
-    {
-        std::vector<char> input(200000);
-        for (std::size_t i = 0; i < input.size(); ++i)
-        {
-            input[i] = at(i);
-        }
-        for (const std::size_t piece :
-                {
-                    static_cast<std::size_t>(1), static_cast<std::size_t>(100), tail - 1, tail, tail + 1, static_cast<std::size_t>(65536), input.size()
-                })
-        {
-            CAPTURE(piece)
-            buffer_t buffer;
-            for (std::size_t pos = 0; pos < input.size(); pos += piece)
-            {
-                const std::size_t n = (std::min)(piece, input.size() - pos);
-                buffer.append(input.data() + pos, input.data() + pos + n);
-            }
-            std::vector<char> out;
-            buffer.copy_to(out);
-            CHECK(keeps_ends(out, input.size()));
-        }
-    }
-
-    SECTION("clear starts a new token")
-    {
-        buffer_t buffer;
-        for (std::size_t i = 0; i < 100000; ++i)
-        {
-            buffer.push_back('x');
-        }
-        buffer.clear();
-        CHECK(buffer.empty());
-        buffer.push_back('a');
-        std::vector<char> out;
-        buffer.copy_to(out);
-        CHECK(out == std::vector<char> {'a'});
-    }
-}

tests/src/unit-class_parser.cpp

@@ -3269,116 +3269,3 @@ TEST_CASE("a plain function is accepted as parser callback for containers")
     CHECK((nlohmann::detail::can_compare_ne<std::string::const_iterator, std::string::const_iterator>::value));
     CHECK((nlohmann::detail::can_compare_ne<const char*, const char*>::value));
 }
-
-#if !defined(JSON_NOEXCEPTION)
-TEST_CASE("parse error message of a long token")
-{
-    // the message of a parse error shows the last read token; a long token is
-    // shown by its first and its last 4096 characters, and the message is the
-    // same for every kind of input
-    const auto message = [](const std::string & text, int kind) -> std::string
-    {
-        json _;
-        try
-        {
-            if (kind == 0)
-            {
-                _ = json::parse(text);
-            }
-            else if (kind == 1)
-            {
-                std::istringstream stream(text);
-                _ = json::parse(stream);
-            }
-            else if (kind == 2)
-            {
-                FILE* const file = std::tmpfile(); // NOLINT(cppcoreguidelines-owning-memory)
-                REQUIRE(file != nullptr);
-                CHECK(std::fwrite(text.data(), 1, text.size(), file) == text.size());
-                CHECK(std::fseek(file, 0, SEEK_SET) == 0);
-                try
-                {
-                    _ = json::parse(file);
-                }
-                catch (...)
-                {
-                    std::fclose(file); // NOLINT(cert-err33-c,cppcoreguidelines-owning-memory)
-                    throw;
-                }
-                std::fclose(file); // NOLINT(cert-err33-c,cppcoreguidelines-owning-memory)
-            }
-            else
-            {
-                const std::list<char> chars(text.begin(), text.end());
-                _ = json::parse(chars.begin(), chars.end());
-            }
-        }
-        catch (const json::parse_error& e)
-        {
-            return e.what();
-        }
-        return "no error";
-    };
-
-    SECTION("a short token is shown completely")
-    {
-        const std::string text = "[\"" + std::string(8000, 'a') + "\x01\"]";
-        const std::string expected = "last read: '\"" + std::string(8000, 'a') + "<U+0001>'";
-        for (int kind = 0; kind < 4; ++kind)
-        {
-            CAPTURE(kind)
-            const std::string m = message(text, kind);
-            CHECK(m.size() > expected.size());
-            CHECK(m.compare(m.size() - expected.size(), expected.size(), expected) == 0);
-        }
-    }
-
-    SECTION("a long string")
-    {
-        const std::string text = "[\"" + std::string(100000, 'a') + "b\x01\"]";
-        // the token is '"', 100,000 times 'a', 'b', and the control character
-        const std::string expected = "last read: '\"" + std::string(4095, 'a') + "..." + std::string(4094, 'a') + "b<U+0001>'";
-        for (int kind = 0; kind < 4; ++kind)
-        {
-            CAPTURE(kind)
-            const std::string m = message(text, kind);
-            CHECK(m.size() > expected.size());
-            CHECK(m.compare(m.size() - expected.size(), expected.size(), expected) == 0);
-        }
-    }
-
-    SECTION("a long run of whitespace after a number")
-    {
-        // the token begins with the number and ends with the invalid character
-        const std::string text = "[1," + std::string(100000, '\n') + "?]";
-        std::string expected = "last read: '1,";
-        for (int i = 0; i < 4094; ++i)
-        {
-            expected += "<U+000A>";
-        }
-        expected += "...";
-        for (int i = 0; i < 4095; ++i)
-        {
-            expected += "<U+000A>";
-        }
-        expected += "?'";
-        for (int kind = 0; kind < 4; ++kind)
-        {
-            CAPTURE(kind)
-            const std::string m = message(text, kind);
-            CHECK(m.size() > expected.size());
-            CHECK(m.compare(m.size() - expected.size(), expected.size(), expected) == 0);
-        }
-    }
-
-    SECTION("valid input with a long run of whitespace")
-    {
-        const std::string text = "1" + std::string(100000, ' ');
-        for (int kind = 0; kind < 4; ++kind)
-        {
-            CAPTURE(kind)
-            CHECK(message(text, kind) == "no error");
-        }
-    }
-}
-#endif