Report a missing 0xFF after a final BON8 string also without strict

A BON8 string ends at the first byte that cannot continue it. When a
string ends a value and is not terminated by 0xFF, that byte belongs to
whatever follows the value, but the reader has already taken it from the
input and keeps it in its push-back buffer. With strict = true the
end-of-input check reports it; with strict = false the buffer was
discarded, so a following value in a stream lost its first byte or bytes
without an error ({0x61, 0x90} read "a" and then dropped the 0 that
follows).

Report parse_error.110 "string at the end of the value is not terminated
by 0xFF" when the push-back buffer is not empty once a BON8 value is
complete and strict is false, as bon8.md requires the final 0xFF. Output
of to_bon8 always has it, so canonical input is not affected.

Fixes #6110.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

docs/mkdocs/docs/api/basic_json/from_bon8.md

@@ -69,8 +69,10 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va
 
 ## Exceptions
 
-- Throws [parse_error.110](../../home/exceptions.md#jsonexceptionparse_error110) if the given input ends prematurely or
+- Throws [parse_error.110](../../home/exceptions.md#jsonexceptionparse_error110) if the given input ends prematurely,
-  the end of the file was not reached when `strict` was set to true
+  if the end of the file was not reached when `strict` was set to true, or if a string at the end of the value is not
+  terminated by 0xFF although a byte follows it (also when `strict` is false: the byte that ends the string is
+  taken from the input, so the next value would be lost)
 - Throws [parse_error.112](../../home/exceptions.md#jsonexceptionparse_error112) if a parse error occurs, for instance
   an invalid byte, a string that is not valid UTF-8, or an object key that is not a string
 

docs/mkdocs/docs/features/binary_formats/bon8.md

@@ -181,7 +181,10 @@ of the message are reported as [parse_error.110](../../home/exceptions.md#jsonex
     arrays and objects with up to four elements that are terminated by 0xFE, unsorted object keys, or a 0xFF after a
     string that would also end without it, are accepted. A second 0xFF is not a terminator but an empty string.
 
-    Strings must be valid UTF-8, and a string at the very end of a message must be terminated by 0xFF.
+    Strings must be valid UTF-8, and a string at the very end of a message must be terminated by 0xFF. This also
+    holds when `strict` is false and more data follows the value: the byte that would end the string belongs to the
+    next value, and reading it would remove it from the input, so the missing 0xFF is reported as
+    [parse_error.110](../../home/exceptions.md#jsonexceptionparse_error110).
 
     Duplicate object keys are accepted; the last value wins (`ordered_json` keeps the position of the first occurrence).
     The specification does not say how to treat them; the HikoGUI reference decoder keeps the first value.

include/nlohmann/detail/input/binary_reader.hpp

@@ -177,6 +177,17 @@ class binary_reader
                 JSON_ASSERT(false); // NOLINT(cert-dcl03-c,hicpp-static-assert,misc-static-assert) LCOV_EXCL_LINE
         }
 
+        // BON8: a string ends at the first byte that cannot continue it. If
+        // that byte follows the value, the reader has already taken it from
+        // the input and holds it in bon8_pushback. Without the end-of-input
+        // check below it would be dropped, and the next read would start
+        // behind it, so report the missing 0xFF instead.
+        if (result && !strict && input_format == input_format_t::bon8 && bon8_pushback_size != 0)
+        {
+            get_bon8();
+            return last_byte_error(exception_id::unexpected_end_of_input, "string at the end of the value is not terminated by 0xFF", "value", "");
+        }
+
         // strict mode: next byte must be EOF
         if (result && strict)
         {

single_include/nlohmann/json.hpp

@@ -18568,6 +18568,17 @@ class binary_reader
                 JSON_ASSERT(false); // NOLINT(cert-dcl03-c,hicpp-static-assert,misc-static-assert) LCOV_EXCL_LINE
         }
 
+        // BON8: a string ends at the first byte that cannot continue it. If
+        // that byte follows the value, the reader has already taken it from
+        // the input and holds it in bon8_pushback. Without the end-of-input
+        // check below it would be dropped, and the next read would start
+        // behind it, so report the missing 0xFF instead.
+        if (result && !strict && input_format == input_format_t::bon8 && bon8_pushback_size != 0)
+        {
+            get_bon8();
+            return last_byte_error(exception_id::unexpected_end_of_input, "string at the end of the value is not terminated by 0xFF", "value", "");
+        }
+
         // strict mode: next byte must be EOF
         if (result && strict)
         {

tests/src/unit-bon8.cpp

@@ -608,14 +608,26 @@ TEST_CASE("BON8")
             CHECK_THROWS_WITH_AS(_ = json::from_bon8(vec), "[json.exception.parse_error.110] parse error at byte 2: syntax error while parsing BON8 value: expected end of input; last byte: 0x90", json::parse_error&);
             CHECK(json::from_bon8(vec, true, false).is_discarded());
 
-            // the byte after a string that ends a container
+            // the byte after a string that ends a container: without strict, the
+            // missing 0xFF is reported, as the byte has been taken from the input
             const bytes vec2 = {0x81, 'a', 0x90};
-            CHECK(json::from_bon8(vec2, false) == json({"a"}));
+            CHECK_THROWS_WITH_AS(_ = json::from_bon8(vec2, false), "[json.exception.parse_error.110] parse error at byte 3: syntax error while parsing BON8 value: string at the end of the value is not terminated by 0xFF; last byte: 0x90", json::parse_error&);
+            CHECK(json::from_bon8(vec2, false, false).is_discarded());
             CHECK_THROWS_WITH_AS(_ = json::from_bon8(vec2), "[json.exception.parse_error.110] parse error at byte 3: syntax error while parsing BON8 value: expected end of input; last byte: 0x90", json::parse_error&);
 
             const bytes vec3 = {0x81, 'a', 0xC2, 0x00};
-            CHECK(json::from_bon8(vec3, false) == json({"a"}));
+            CHECK_THROWS_WITH_AS(_ = json::from_bon8(vec3, false), "[json.exception.parse_error.110] parse error at byte 3: syntax error while parsing BON8 value: string at the end of the value is not terminated by 0xFF; last byte: 0xC2", json::parse_error&);
             CHECK_THROWS_WITH_AS(_ = json::from_bon8(vec3), "[json.exception.parse_error.110] parse error at byte 3: syntax error while parsing BON8 value: expected end of input; last byte: 0xC2", json::parse_error&);
+
+            // with a terminated string, strict = false leaves the next value in the stream
+            std::istringstream terminated(std::string("\x61\xFF\x90", 3));
+            CHECK(json::from_bon8(terminated, false) == "a");
+            CHECK(terminated.tellg() == 2);
+            CHECK(json::from_bon8(terminated, false) == 0);
+
+            // without the terminator, the next value would be lost
+            std::istringstream unterminated(std::string("\x61\x90", 2));
+            CHECK_THROWS_WITH_AS(_ = json::from_bon8(unterminated, false), "[json.exception.parse_error.110] parse error at byte 2: syntax error while parsing BON8 value: string at the end of the value is not terminated by 0xFF; last byte: 0x90", json::parse_error&);
         }
     }