Report a missing 0xFF after a final BON8 string also without strict

A BON8 string ends at the first byte that cannot continue it. When a
string ends a value and is not terminated by 0xFF, that byte belongs to
whatever follows the value, but the reader has already taken it from the
input and keeps it in its push-back buffer. With strict = true the
end-of-input check reports it; with strict = false the buffer was
discarded, so a following value in a stream lost its first byte or bytes
without an error ({0x61, 0x90} read "a" and then dropped the 0 that
follows).

Report parse_error.110 "string at the end of the value is not terminated
by 0xFF" when the push-back buffer is not empty once a BON8 value is
complete and strict is false, as bon8.md requires the final 0xFF. Output
of to_bon8 always has it, so canonical input is not affected.

Fixes #6110.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

docs/mkdocs/docs/api/basic_json/from_bon8.md

@@ -69,8 +69,10 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va
 
 ## Exceptions
 
-- Throws [parse_error.110](../../home/exceptions.md#jsonexceptionparse_error110) if the given input ends prematurely or
+- Throws [parse_error.110](../../home/exceptions.md#jsonexceptionparse_error110) if the given input ends prematurely,
-  the end of the file was not reached when `strict` was set to true
+  if the end of the file was not reached when `strict` was set to true, or if a string at the end of the value is not
+  terminated by 0xFF although a byte follows it (also when `strict` is false: the byte that ends the string is
+  taken from the input, so the next value would be lost)
 - Throws [parse_error.112](../../home/exceptions.md#jsonexceptionparse_error112) if a parse error occurs, for instance
   an invalid byte, a string that is not valid UTF-8, or an object key that is not a string
 

docs/mkdocs/docs/api/macros/json_recursion_depth_limit.md

@@ -27,7 +27,7 @@ comparison of two [`json_view`](../basic_json_view/index.md) documents has a fix
 and does not depend on it either.
 
 The call stack that these operations need grows with the limit: roughly *a few kilobytes plus a cost per level times the
-limit*, where the cost per level is up to about 1.8 KiB in an unoptimized and up to about 0.5 KiB in an optimized build
+limit*, where the cost per level is up to about 2.3 KiB in an unoptimized and up to about 0.5 KiB in an optimized build
 (see [Stack needed](#stack-needed)). The default of `128` levels is tuned for threads with ordinary stacks of a few
 hundred kilobytes or more. Set a lower limit if your threads have small stacks (RTOS tasks, fibers, threads of the musl
 C
@@ -67,7 +67,7 @@ When the macro is not defined, the library defines it to its default value.
       the levels that are descended into are counted, so the count never exceeds the limit, not even when user code (a
       copy constructor of a custom base class, say) copies a value while a copy is going on. A byte holds `255` at
       most, and the limit stays below that, so that the count cannot wrap around. (More levels would hardly be
-      useful: `254` levels need up to 448 KiB of stack in an unoptimized build already, twice as much as the
+      useful: `254` levels need up to 576 KiB of stack in an unoptimized build already, twice as much as the
       default.)
 
 ### Stack needed
@@ -83,7 +83,7 @@ Unoptimized (`-O0`):
 
 | Operation | limit 16 | limit 32 | limit 64 | limit 128 | limit 254 |
 |---|---:|---:|---:|---:|---:|
-| `dump()` and `dump(indent)`, `operator<<` | 16 / 16 | 16 / 16 | 32 / 32 | 48 / 48 | 96 / 96 |
+| `dump()` and `dump(indent)`, `operator<<` | 48 / 32 | 80 / 64 | 144 / 128 | 288 / 256 | 576 / 512 |
 | copy construction and assignment | 32 / 16 | 48 / 32 | 112 / 80 | 224 / 144 | 432 / 304 |
 | `==` | 16 / 16 | 16 / 16 | 16 / 16 | 48 / 32 | 96 / 48 |
 | `<=>` (C++20) | 16 / 16 | 16 / 16 | 32 / 32 | 64 / 64 | 144 / 128 |
@@ -123,24 +123,23 @@ The rule of thumb is
 
 > stack needed ≈ a few KiB + cost per level × limit
 
-with a cost per level of up to 1.8 KiB (conversion to another `basic_json` type at `-O0` with Clang; 1.3 KiB with
+with a cost per level of up to 2.3 KiB (`dump` at `-O0` with Clang; 2.1 KiB with GCC), 1.7 KiB for copying and merging
-GCC), 1.7 KiB for copying and merging (1.3 KiB with GCC), 0.4 KiB for `dump`, and 0.1 to 0.5 KiB for most other
+(1.3 KiB with GCC), and 0.1 to 0.5 KiB for most other operations at `-O0`; and 0.5 KiB or less at `-O2`. The numbers
-operations at `-O0`; and 0.5 KiB or less at `-O2`. The numbers come from the difference between the limits `128` and
+come from the difference between the limits `128` and `254` in the tables, so they are rough: other compilers,
-`254` in the tables, so they are rough: other compilers, other targets (the frames of x86-64 code are about as large),
+other targets (the frames of x86-64 code are about as large), other standard libraries, `JSON_DIAGNOSTICS`, and
-other standard libraries, `JSON_DIAGNOSTICS`, and sanitizers change them. Measure with your own tool chain (see the
+sanitizers change them. Measure with your own tool chain (see the example below) rather than relying on the tables.
-example below) rather than relying on the tables. The stack that your own code and the operating system need comes on
+The stack that your own code and the operating system need comes on top of these numbers.
-top of these numbers.
 
 Which limit to choose, for the stack that you can spare for the library (the largest limit of those measured
 for which the slowest operation needs at most that much):
 
 | Stack for the library | optimized (`-O2`) | unoptimized (`-O0`) |
 |---|---:|---:|
-| 32 KiB | 64 | 16 |
+| 32 KiB | 64 | 0 |
-| 64 KiB | 128 | 32 |
+| 64 KiB | 128 | 16 |
-| 128 KiB (the default stack of threads on musl) | 254 | 64 |
+| 128 KiB (the default stack of threads on musl) | 254 | 32 |
-| 256 KiB | 254 | 128 |
+| 256 KiB | 254 | 64 |
-| 512 KiB | 254 | 254 |
+| 512 KiB | 254 | 128 |
 | 1 MiB or more | 254 | 254 |
 
 ### Speed

docs/mkdocs/docs/features/binary_formats/bon8.md

@@ -181,7 +181,10 @@ of the message are reported as [parse_error.110](../../home/exceptions.md#jsonex
     arrays and objects with up to four elements that are terminated by 0xFE, unsorted object keys, or a 0xFF after a
     string that would also end without it, are accepted. A second 0xFF is not a terminator but an empty string.
 
-    Strings must be valid UTF-8, and a string at the very end of a message must be terminated by 0xFF.
+    Strings must be valid UTF-8, and a string at the very end of a message must be terminated by 0xFF. This also
+    holds when `strict` is false and more data follows the value: the byte that would end the string belongs to the
+    next value, and reading it would remove it from the input, so the missing 0xFF is reported as
+    [parse_error.110](../../home/exceptions.md#jsonexceptionparse_error110).
 
     Duplicate object keys are accepted; the last value wins (`ordered_json` keeps the position of the first occurrence).
     The specification does not say how to treat them; the HikoGUI reference decoder keeps the first value.

docs/mkdocs/docs/features/parsing/untrusted_input.md

@@ -64,13 +64,13 @@ stack for the nesting.)
 
 The call stack still needs room for the levels that are recursed into, and the default limit assumes a stack of ordinary
 size. The stack that an operation needs is roughly proportional to the limit; with the default of 128 levels, the most
-demanding operations (copying a value and converting it to another `basic_json` type, in an unoptimized build) needed
+demanding operation (`dump` in an unoptimized build) needed up to 288 KiB, and an optimized build needed 48 KiB or less,
-up to 224 KiB, and an optimized build needed 48 KiB or less, in measurements on macOS with arm64. The most demanding
+in measurements on macOS with arm64. The most demanding operation needed this much stack at the limits `0`, `32`,
-operation needed this much stack at the limits `0`, `32`, `128`, and `254`:
+`128`, and `254`:
 
 | Stack needed, in KiB | limit 0 | limit 32 | limit 128 (default) | limit 254 |
 |---|---:|---:|---:|---:|
-| unoptimized build (`-O0`) | 16 or less | 64 | 224 | 448 |
+| unoptimized build (`-O0`) | 16 or less | 80 | 288 | 576 |
 | optimized build (`-O2`) | 16 or less | 16 or less | 48 | 112 |
 
 A thread with a smaller stack (a task of a real-time operating system, a fiber, or a thread of the musl C library, whose

include/nlohmann/detail/input/binary_reader.hpp

@@ -177,6 +177,17 @@ class binary_reader
                 JSON_ASSERT(false); // NOLINT(cert-dcl03-c,hicpp-static-assert,misc-static-assert) LCOV_EXCL_LINE
         }
 
+        // BON8: a string ends at the first byte that cannot continue it. If
+        // that byte follows the value, the reader has already taken it from
+        // the input and holds it in bon8_pushback. Without the end-of-input
+        // check below it would be dropped, and the next read would start
+        // behind it, so report the missing 0xFF instead.
+        if (result && !strict && input_format == input_format_t::bon8 && bon8_pushback_size != 0)
+        {
+            get_bon8();
+            return last_byte_error(exception_id::unexpected_end_of_input, "string at the end of the value is not terminated by 0xFF", "value", "");
+        }
+
         // strict mode: next byte must be EOF
         if (result && strict)
         {

include/nlohmann/detail/macro_scope.hpp

@@ -230,16 +230,6 @@
 // detail/recursion_depth_limit.hpp. Like JSON_NO_THREAD_LOCAL it is neither part
 // of the ABI tag nor undefined at the end of json.hpp.
 
-// forced inlining of the serializer's helpers only in optimized builds: at -O0,
-// GCC and Clang give every inlined copy its own stack slots, which made the
-// recursive frame of serializer::dump_container() about 2.3 KB (MSVC does not
-// expand __forceinline at /Ob0, so it keeps it)
-#if defined(__OPTIMIZE__) || defined(_MSC_VER)
-    #define JSON_INTERNAL_SERIALIZER_INLINE JSON_HEDLEY_ALWAYS_INLINE
-#else
-    #define JSON_INTERNAL_SERIALIZER_INLINE inline
-#endif
-
 // allow disabling exceptions
 #if (defined(__cpp_exceptions) || defined(__EXCEPTIONS) || defined(_CPPUNWIND)) && !defined(JSON_NOEXCEPTION)
     #define JSON_THROW(exception) throw exception

include/nlohmann/detail/macro_unscope.hpp

@@ -11,7 +11,6 @@
 // clean up
 #undef JSON_ASSERT
 #undef JSON_INTERNAL_CATCH
-#undef JSON_INTERNAL_SERIALIZER_INLINE
 #undef JSON_THROW
 #undef JSON_PRIVATE_UNLESS_TESTED
 #undef NLOHMANN_BASIC_JSON_TPL_DECLARATION

include/nlohmann/detail/output/serializer.hpp

@@ -265,7 +265,7 @@ class serializer
     for a container; writing them from here spares each the entry and exit of
     that function.
     */
-    JSON_INTERNAL_SERIALIZER_INLINE void dump_element(const BasicJsonType& val,
+    JSON_HEDLEY_ALWAYS_INLINE void dump_element(const BasicJsonType& val,
             const std::size_t current_indent,
             const std::size_t depth)
     {
@@ -971,7 +971,7 @@ class serializer
     }
 
     /// write out the bytes pending in @ref string_buffer and reset @a bytes
-    JSON_INTERNAL_SERIALIZER_INLINE void emit_pending(std::size_t& bytes)
+    JSON_HEDLEY_ALWAYS_INLINE void emit_pending(std::size_t& bytes)
     {
         if (bytes != 0)
         {
@@ -983,7 +983,7 @@ class serializer
     /// write buffer and reset index if fewer than 13 bytes are left, as this
     /// is the maximal number of bytes to be written ("\uxxxx\uxxxx\0") for one
     /// code point
-    JSON_INTERNAL_SERIALIZER_INLINE void reserve_string_buffer(std::size_t& bytes)
+    JSON_HEDLEY_ALWAYS_INLINE void reserve_string_buffer(std::size_t& bytes)
     {
         if (string_buffer.size() - bytes < 13)
         {
@@ -1344,7 +1344,7 @@ class serializer
           inlining both into the pretty-printing loops once @ref dump_escaped
           has grown, which made indented output up to 13% slower
     */
-    JSON_INTERNAL_SERIALIZER_INLINE void put_indent(std::size_t indent)
+    JSON_HEDLEY_ALWAYS_INLINE void put_indent(std::size_t indent)
     {
         // closing braces at the outermost level ask for no indentation at all
         if (indent == 0)
@@ -1398,7 +1398,7 @@ class serializer
     its room and four stores of the indentation, see @ref put_short_literal_indent.
     */
     template<std::size_t N>
-    JSON_INTERNAL_SERIALIZER_INLINE void put_literal_indent(const char (&s)[N], const std::size_t indent) // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
+    JSON_HEDLEY_ALWAYS_INLINE void put_literal_indent(const char (&s)[N], const std::size_t indent) // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
     {
         if (JSON_HEDLEY_LIKELY(put_short_literal_indent(s, indent)))
         {
@@ -1418,7 +1418,7 @@ class serializer
     @a indent is wider or the write buffer lacks the room.
     */
     template<std::size_t N>
-    JSON_INTERNAL_SERIALIZER_INLINE bool put_short_literal_indent(const char (&s)[N], const std::size_t indent) // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
+    JSON_HEDLEY_ALWAYS_INLINE bool put_short_literal_indent(const char (&s)[N], const std::size_t indent) // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
     {
         constexpr std::size_t length = N - 1;
         if (indent > 32 || write_buffer_pos + length + 32 > write_buffer.size())
@@ -1445,7 +1445,7 @@ class serializer
     write-through path for oversized runs.
     */
     template<std::size_t N>
-    JSON_INTERNAL_SERIALIZER_INLINE void put_literal(const char (&s)[N]) // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
+    JSON_HEDLEY_ALWAYS_INLINE void put_literal(const char (&s)[N]) // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
     {
         static_assert(N >= 2, "put_literal expects a non-empty string literal");
         // the array bound counts the terminating NUL, which is not written

single_include/nlohmann/json.hpp

@@ -2603,16 +2603,6 @@ JSON_HEDLEY_DIAGNOSTIC_POP
 // detail/recursion_depth_limit.hpp. Like JSON_NO_THREAD_LOCAL it is neither part
 // of the ABI tag nor undefined at the end of json.hpp.
 
-// forced inlining of the serializer's helpers only in optimized builds: at -O0,
-// GCC and Clang give every inlined copy its own stack slots, which made the
-// recursive frame of serializer::dump_container() about 2.3 KB (MSVC does not
-// expand __forceinline at /Ob0, so it keeps it)
-#if defined(__OPTIMIZE__) || defined(_MSC_VER)
-    #define JSON_INTERNAL_SERIALIZER_INLINE JSON_HEDLEY_ALWAYS_INLINE
-#else
-    #define JSON_INTERNAL_SERIALIZER_INLINE inline
-#endif
-
 // allow disabling exceptions
 #if (defined(__cpp_exceptions) || defined(__EXCEPTIONS) || defined(_CPPUNWIND)) && !defined(JSON_NOEXCEPTION)
     #define JSON_THROW(exception) throw exception
@@ -18578,6 +18568,17 @@ class binary_reader
                 JSON_ASSERT(false); // NOLINT(cert-dcl03-c,hicpp-static-assert,misc-static-assert) LCOV_EXCL_LINE
         }
 
+        // BON8: a string ends at the first byte that cannot continue it. If
+        // that byte follows the value, the reader has already taken it from
+        // the input and holds it in bon8_pushback. Without the end-of-input
+        // check below it would be dropped, and the next read would start
+        // behind it, so report the missing 0xFF instead.
+        if (result && !strict && input_format == input_format_t::bon8 && bon8_pushback_size != 0)
+        {
+            get_bon8();
+            return last_byte_error(exception_id::unexpected_end_of_input, "string at the end of the value is not terminated by 0xFF", "value", "");
+        }
+
         // strict mode: next byte must be EOF
         if (result && strict)
         {
@@ -36995,7 +36996,7 @@ class serializer
     for a container; writing them from here spares each the entry and exit of
     that function.
     */
-    JSON_INTERNAL_SERIALIZER_INLINE void dump_element(const BasicJsonType& val,
+    JSON_HEDLEY_ALWAYS_INLINE void dump_element(const BasicJsonType& val,
             const std::size_t current_indent,
             const std::size_t depth)
     {
@@ -37701,7 +37702,7 @@ class serializer
     }
 
     /// write out the bytes pending in @ref string_buffer and reset @a bytes
-    JSON_INTERNAL_SERIALIZER_INLINE void emit_pending(std::size_t& bytes)
+    JSON_HEDLEY_ALWAYS_INLINE void emit_pending(std::size_t& bytes)
     {
         if (bytes != 0)
         {
@@ -37713,7 +37714,7 @@ class serializer
     /// write buffer and reset index if fewer than 13 bytes are left, as this
     /// is the maximal number of bytes to be written ("\uxxxx\uxxxx\0") for one
     /// code point
-    JSON_INTERNAL_SERIALIZER_INLINE void reserve_string_buffer(std::size_t& bytes)
+    JSON_HEDLEY_ALWAYS_INLINE void reserve_string_buffer(std::size_t& bytes)
     {
         if (string_buffer.size() - bytes < 13)
         {
@@ -38074,7 +38075,7 @@ class serializer
           inlining both into the pretty-printing loops once @ref dump_escaped
           has grown, which made indented output up to 13% slower
     */
-    JSON_INTERNAL_SERIALIZER_INLINE void put_indent(std::size_t indent)
+    JSON_HEDLEY_ALWAYS_INLINE void put_indent(std::size_t indent)
     {
         // closing braces at the outermost level ask for no indentation at all
         if (indent == 0)
@@ -38128,7 +38129,7 @@ class serializer
     its room and four stores of the indentation, see @ref put_short_literal_indent.
     */
     template<std::size_t N>
-    JSON_INTERNAL_SERIALIZER_INLINE void put_literal_indent(const char (&s)[N], const std::size_t indent) // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
+    JSON_HEDLEY_ALWAYS_INLINE void put_literal_indent(const char (&s)[N], const std::size_t indent) // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
     {
         if (JSON_HEDLEY_LIKELY(put_short_literal_indent(s, indent)))
         {
@@ -38148,7 +38149,7 @@ class serializer
     @a indent is wider or the write buffer lacks the room.
     */
     template<std::size_t N>
-    JSON_INTERNAL_SERIALIZER_INLINE bool put_short_literal_indent(const char (&s)[N], const std::size_t indent) // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
+    JSON_HEDLEY_ALWAYS_INLINE bool put_short_literal_indent(const char (&s)[N], const std::size_t indent) // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
     {
         constexpr std::size_t length = N - 1;
         if (indent > 32 || write_buffer_pos + length + 32 > write_buffer.size())
@@ -38175,7 +38176,7 @@ class serializer
     write-through path for oversized runs.
     */
     template<std::size_t N>
-    JSON_INTERNAL_SERIALIZER_INLINE void put_literal(const char (&s)[N]) // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
+    JSON_HEDLEY_ALWAYS_INLINE void put_literal(const char (&s)[N]) // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
     {
         static_assert(N >= 2, "put_literal expects a non-empty string literal");
         // the array bound counts the terminating NUL, which is not written
@@ -47895,7 +47896,6 @@ struct formatter<nlohmann::NLOHMANN_BASIC_JSON_TPL, char> // NOLINT(cert-dcl58-c
 // clean up
 #undef JSON_ASSERT
 #undef JSON_INTERNAL_CATCH
-#undef JSON_INTERNAL_SERIALIZER_INLINE
 #undef JSON_THROW
 #undef JSON_PRIVATE_UNLESS_TESTED
 #undef NLOHMANN_BASIC_JSON_TPL_DECLARATION

tests/src/unit-bon8.cpp

@@ -608,14 +608,26 @@ TEST_CASE("BON8")
             CHECK_THROWS_WITH_AS(_ = json::from_bon8(vec), "[json.exception.parse_error.110] parse error at byte 2: syntax error while parsing BON8 value: expected end of input; last byte: 0x90", json::parse_error&);
             CHECK(json::from_bon8(vec, true, false).is_discarded());
 
-            // the byte after a string that ends a container
+            // the byte after a string that ends a container: without strict, the
+            // missing 0xFF is reported, as the byte has been taken from the input
             const bytes vec2 = {0x81, 'a', 0x90};
-            CHECK(json::from_bon8(vec2, false) == json({"a"}));
+            CHECK_THROWS_WITH_AS(_ = json::from_bon8(vec2, false), "[json.exception.parse_error.110] parse error at byte 3: syntax error while parsing BON8 value: string at the end of the value is not terminated by 0xFF; last byte: 0x90", json::parse_error&);
+            CHECK(json::from_bon8(vec2, false, false).is_discarded());
             CHECK_THROWS_WITH_AS(_ = json::from_bon8(vec2), "[json.exception.parse_error.110] parse error at byte 3: syntax error while parsing BON8 value: expected end of input; last byte: 0x90", json::parse_error&);
 
             const bytes vec3 = {0x81, 'a', 0xC2, 0x00};
-            CHECK(json::from_bon8(vec3, false) == json({"a"}));
+            CHECK_THROWS_WITH_AS(_ = json::from_bon8(vec3, false), "[json.exception.parse_error.110] parse error at byte 3: syntax error while parsing BON8 value: string at the end of the value is not terminated by 0xFF; last byte: 0xC2", json::parse_error&);
             CHECK_THROWS_WITH_AS(_ = json::from_bon8(vec3), "[json.exception.parse_error.110] parse error at byte 3: syntax error while parsing BON8 value: expected end of input; last byte: 0xC2", json::parse_error&);
+
+            // with a terminated string, strict = false leaves the next value in the stream
+            std::istringstream terminated(std::string("\x61\xFF\x90", 3));
+            CHECK(json::from_bon8(terminated, false) == "a");
+            CHECK(terminated.tellg() == 2);
+            CHECK(json::from_bon8(terminated, false) == 0);
+
+            // without the terminator, the next value would be lost
+            std::istringstream unterminated(std::string("\x61\x90", 2));
+            CHECK_THROWS_WITH_AS(_ = json::from_bon8(unterminated, false), "[json.exception.parse_error.110] parse error at byte 2: syntax error while parsing BON8 value: string at the end of the value is not terminated by 0xFF; last byte: 0x90", json::parse_error&);
         }
     }