fix: restore the character that terminates a number (#5340)

operator>> is documented to leave the stream positioned right after the
parsed value, so that concatenated JSON values can be read back to back.
That did not hold for numbers: a number is only terminated by the
character following it, and lexer::scan_number() reads that character
and calls unget() -- which is simulated and rewinds only the lexer's own
bookkeeping. input_stream_adapter consumes via sbumpc() with no matching
sungetc(), so the terminating character stayed consumed and the next
extraction started one byte too late ('1true' left the stream at 'rue').

Propagating unget() to the adapter directly does not work: next_unget
makes the following get() replay the cached character, so the terminator
would be delivered twice. Instead, restore the still-pending character
once at the end of a non-strict parse, where the input is handed back to
the caller:

- input_stream_adapter gains unget_character() (sungetc()) and advertises
  it via supports_unget, detected the same way as supports_seek.
- lexer::restore_pending_unget() turns a pending simulated unget of a
  real (non-EOF) character into a real one and clears next_unget so the
  character is not also replayed. It is a no-op for adapters that cannot
  unget, and reports failure when sungetc() fails, in which case the
  input is left as it was before.
- parser calls it on the three non-strict paths, i.e. for operator>> and
  sax_parse(strict = false).

Strict parse()/accept() are unaffected: they require the input to end
after the value, so the character is consumed by the end-of-input check
anyway. Parse error messages and reported positions are unchanged.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

include/nlohmann/detail/input/parser.hpp

@@ -99,8 +99,14 @@ class parser
             json_sax_dom_callback_parser<BasicJsonType, InputAdapterType> sdp(result, callback, allow_exceptions, &m_lexer);
             sax_parse_internal(&sdp);
 
+            if (!strict)
+            {
+                // the caller keeps using the input: position it right after
+                // the value by giving back the character that terminated it
+                m_lexer.restore_pending_unget();
+            }
             // in strict mode, input must be completely read
-            if (strict && (get_token() != token_type::end_of_input))
+            else if (get_token() != token_type::end_of_input)
             {
                 sdp.parse_error(m_lexer.get_position(),
                                 m_lexer.get_token_string(),
@@ -127,8 +133,13 @@ class parser
             json_sax_dom_parser<BasicJsonType, InputAdapterType> sdp(result, allow_exceptions, &m_lexer);
             sax_parse_internal(&sdp);
 
+            if (!strict)
+            {
+                // see above
+                m_lexer.restore_pending_unget();
+            }
             // in strict mode, input must be completely read
-            if (strict && (get_token() != token_type::end_of_input))
+            else if (get_token() != token_type::end_of_input)
             {
                 sdp.parse_error(m_lexer.get_position(),
                                 m_lexer.get_token_string(),
@@ -165,8 +176,14 @@ class parser
         (void)detail::is_sax_static_asserts<SAX, BasicJsonType> {};
         const bool result = sax_parse_internal(sax);
 
+        if (result && !strict)
+        {
+            // the caller keeps using the input: position it right after the
+            // value by giving back the character that terminated it
+            m_lexer.restore_pending_unget();
+        }
         // strict mode: next byte must be EOF
-        if (result && strict && (get_token() != token_type::end_of_input))
+        else if (result && strict && (get_token() != token_type::end_of_input))
         {
             return sax->parse_error(m_lexer.get_position(),
                                     m_lexer.get_token_string(),