Hold the token carry buffer in a std::unique_ptr

clang-tidy rejected the raw new[]/delete[] of token_carry_buffer
(cppcoreguidelines-owning-memory) and the capacity computation
(readability-use-std-min-max). The storage is now owned by a
std::unique_ptr<char[]>, as in json_view's edit storage, and the new
capacity is a std::max.

istream parse instructions (Apple clang, twitter, citm, canada, floats)
are unchanged within noise.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

docs/mkdocs/docs/api/basic_json/accept.md

@@ -23,9 +23,6 @@ Checks whether the input is valid JSON.
     respectively as UTF-8, UTF-16, and UTF-32. If `SentinelType` differs from `IteratorType`, it must be comparable to
     the iterator type with `operator!=`.
     
-    Ill-formed UTF-16 and UTF-32 sequences are rejected: a lone surrogate, a high surrogate that is not followed by its
-    low surrogate, and a code point above `U+10FFFF` all make this function return `false`.
-
 Unlike the [`parse()`](parse.md) function, this function neither throws an exception in case of invalid JSON input
 (i.e., a parse error) nor creates diagnostic information.
 
@@ -38,12 +35,9 @@ Unlike the [`parse()`](parse.md) function, this function neither throws an excep
     - a `#!c FILE` pointer (throws if null)
     - a C-style array of characters
     - a pointer to a null-terminated string of single byte characters (throws if null)
-    - a `std::string` or a `std::string_view` (C++17)
+    - a `std::string`
     - a container `obj` for which `begin(obj)` and `end(obj)` produce a valid pair of iterators
       (as found via ADL or member functions, with semantics compatible to `std::begin` and `std::end`)
-    - wide characters (`wchar_t`, `char16_t`, `char32_t`), which are converted to UTF-8: a wide string object
-      (`std::wstring`, `std::u16string`, `std::u32string`) or an array (e.g. a wide string literal); a pointer to a
-      null-terminated wide string (e.g. `std::u16string::c_str()`) is not supported
 
 `IteratorType`
 :   a compatible iterator type, for instance.
@@ -131,21 +125,6 @@ outside of a string, invalid) byte; see the [FAQ entry](../../home/faq.md#nul-by
     --8<-- "examples/accept__iterator_pair.output"
     ```
 
-??? example "Example: (3) reading from UTF-16 strings"
-
-    The encoding is deduced from the character type of the input. String literals like `u"..."`, `U"..."`, and
-    `L"..."` work as well.
-
-    ```cpp
-    --8<-- "examples/accept__wide_string.cpp"
-    ```
-
-    Output:
-
-    ```json
-    --8<-- "examples/accept__wide_string.output"
-    ```
-
 ## See also
 
 - [parse](parse.md) - deserialize from a compatible input
@@ -172,9 +151,6 @@ outside of a string, invalid) byte; see the [FAQ entry](../../home/faq.md#nul-by
   ([`JSON_DISABLE_STREAM_BULK_SCAN`](../macros/json_disable_stream_bulk_scan.md)).
 - Parsing from a `FILE*` locks the file once instead of once per character in version 3.13.0
   ([`JSON_DISABLE_UNLOCKED_FILE_IO`](../macros/json_disable_unlocked_file_io.md)).
-- Ill-formed UTF-16 and UTF-32 input (a lone surrogate, a high surrogate that is not followed by its low surrogate, or
-  a code point above `U+10FFFF`) is rejected in version 3.13.0; before that, some such sequences were accepted or
-  ended the input silently.
 
 !!! warning "Deprecation"
 

docs/mkdocs/docs/api/basic_json/index.md

@@ -44,22 +44,6 @@ are collected in [Template Parameter Requirements](../../features/types/template
 - [**json**](../json.md) - default specialization
 - [**ordered_json**](../ordered_json.md) - a specialization that maintains the insertion order of object keys
 
-## Documents and views
-
-Besides `basic_json`, the library offers a second way to hold a parsed JSON text: a
-[`basic_json_document`](../basic_json_document/index.md), which owns or borrows the text and holds a flat index of the
-values in it, and a [`basic_json_view`](../basic_json_view/index.md), a read-only handle to one value of such a document.
-Both are defined in the header `<nlohmann/json_view.hpp>`; see
-[Zero-copy JSON views](../../features/json_view.md) for the concept and the trade-offs.
-
-- [**json_document**](../json_document.md) / [**ordered_json_document**](../ordered_json_document.md) - read-only
-  documents of the `basic_json` specialization they belong to
-- [**json_editable_document**](../json_editable_document.md) /
-  [**ordered_json_editable_document**](../ordered_json_editable_document.md) - documents that can be modified in place
-- [**json_view**](../json_view.md) / [**ordered_json_view**](../ordered_json_view.md) - views of a document
-- [**json_editable_view**](../json_editable_view.md) /
-  [**ordered_json_editable_view**](../ordered_json_editable_view.md) - views of an editable document
-
 ## Iterator invalidation
 
 All operations that add values to an **array** ([`push_back`](push_back.md) , [`operator+=`](operator+=.md),
@@ -359,9 +343,6 @@ Access to the JSON value
 ## See also
 
 - [RFC 8259: The JavaScript Object Notation (JSON) Data Interchange Format](https://tools.ietf.org/html/rfc8259)
-- [basic_json_document](../basic_json_document/index.md) / [basic_json_view](../basic_json_view/index.md) - parse a JSON
-  text into a flat index and read it through a view
-- [Zero-copy JSON views](../../features/json_view.md) - when to use a document and a view instead of a `basic_json`
 
 ## Version history
 

docs/mkdocs/docs/api/basic_json/parse.md

@@ -25,10 +25,6 @@ static basic_json parse(IteratorType first, SentinelType last,
     respectively as UTF-8, UTF-16, and UTF-32. If `SentinelType` differs from `IteratorType`, it must be comparable to
     the iterator type with `operator!=`.
 
-    Ill-formed UTF-16 and UTF-32 sequences are rejected: a lone surrogate, a high surrogate that is not followed by its
-    low surrogate, and a code point above `U+10FFFF` all yield a
-    [`parse_error.101`](../../home/exceptions.md#jsonexceptionparse_error101).
-
 ## Template parameters
 
 `InputType`
@@ -38,12 +34,9 @@ static basic_json parse(IteratorType first, SentinelType last,
     - a `FILE` pointer (throws if null)
     - a C-style array of characters
     - a pointer to a null-terminated string of single byte characters (throws if null)
-    - a `std::string` or a `std::string_view` (C++17)
+    - a `std::string`
     - a container `obj` for which `begin(obj)` and `end(obj)` produce a valid pair of iterators
       (as found via ADL or member functions, with semantics compatible to `std::begin` and `std::end`)
-    - wide characters (`wchar_t`, `char16_t`, `char32_t`), which are converted to UTF-8: a wide string object
-      (`std::wstring`, `std::u16string`, `std::u32string`) or an array (e.g. a wide string literal); a pointer to a
-      null-terminated wide string (e.g. `std::u16string::c_str()`) is not supported
 
 `IteratorType`
 :   a compatible iterator type, for instance.
@@ -119,13 +112,6 @@ By default, a `'\0'` (NUL) byte anywhere in the input is treated as end of input
 outside of a string, invalid) byte; see the [FAQ entry](../../home/faq.md#nul-bytes-in-the-input) for details and the
 [`JSON_STRICT_NUL_HANDLING`](../macros/json_strict_nul_handling.md) macro to opt into rejecting it instead.
 
-!!! tip "Parsing into a document instead of a value"
-
-    [`basic_json_document::parse`](../basic_json_document/parse.md) accepts the same inputs and the same options, but
-    keeps the result as a flat index of the parsed values instead of a tree of `basic_json` values. It is faster and uses
-    less memory when only parts of the document are read, and it also accepts a pointer to a null-terminated wide string,
-    which `parse()` does not. See [Zero-copy JSON views](../../features/json_view.md).
-
 !!! tip "Performance"
 
     Contiguous inputs (strings, string views, vectors, arrays, pointer ranges) are the fastest to parse. A
@@ -264,22 +250,6 @@ outside of a string, invalid) byte; see the [FAQ entry](../../home/faq.md#nul-by
     --8<-- "examples/trailing_commas.output"
     ```
 
-??? example "Example: parse from UTF-16, UTF-32, and wide strings"
-
-    The encoding is deduced from the character type of the input; `std::u16string` and `std::u32string` are interpreted
-    as UTF-16 and UTF-32, and `std::wstring` uses the platform's `wchar_t` width. String literals like `u"..."`,
-    `U"..."`, and `L"..."` work as well, as do iterators over wide characters.
-
-    ```cpp
-    --8<-- "examples/parse__wide_string.cpp"
-    ```
-
-    Output:
-
-    ```json
-    --8<-- "examples/parse__wide_string.output"
-    ```
-
 ## See also
 
 - [accept](accept.md) - check if the input is valid JSON
@@ -310,9 +280,6 @@ outside of a string, invalid) byte; see the [FAQ entry](../../home/faq.md#nul-by
   time).
 - Parsing from a `FILE*` locks the file once instead of once per character in version 3.13.0
   ([`JSON_DISABLE_UNLOCKED_FILE_IO`](../macros/json_disable_unlocked_file_io.md) restores `std::fgetc`).
-- Ill-formed UTF-16 and UTF-32 input (a lone surrogate, a high surrogate that is not followed by its low surrogate, or
-  a code point above `U+10FFFF`) is rejected with `parse_error.101` in version 3.13.0; before that, some such
-  sequences were accepted or ended the input silently.
 
 !!! warning "Deprecation"
 

docs/mkdocs/docs/api/basic_json/sax_parse.md

@@ -31,10 +31,6 @@ Read from input and generate SAX events
     respectively as UTF-8, UTF-16, and UTF-32. If `SentinelType` differs from `IteratorType`, it must be comparable to
     the iterator type with `operator!=`.
 
-    Ill-formed UTF-16 and UTF-32 sequences are rejected: a lone surrogate, a high surrogate that is not followed by its
-    low surrogate, and a code point above `U+10FFFF` all yield a
-    [`parse_error.101`](../../home/exceptions.md#jsonexceptionparse_error101).
-
 The SAX event lister must follow the interface of [`json_sax`](../json_sax/index.md).
 
 ## Template parameters
@@ -46,12 +42,8 @@ The SAX event lister must follow the interface of [`json_sax`](../json_sax/index
     - a `FILE` pointer
     - a C-style array of characters
     - a pointer to a null-terminated string of single byte characters
-    - a `std::string` or a `std::string_view` (C++17)
     - a container `obj` for which `begin(obj)` and `end(obj)` produce a valid pair of iterators
       (as found via ADL or member functions, with semantics compatible to `std::begin` and `std::end`)
-    - wide characters (`wchar_t`, `char16_t`, `char32_t`), which are converted to UTF-8: a wide string object
-      (`std::wstring`, `std::u16string`, `std::u32string`) or an array (e.g. a wide string literal); a pointer to a
-      null-terminated wide string (e.g. `std::u16string::c_str()`) is not supported
 
 `IteratorType`
 :   a compatible iterator type for overload (2); a pair of character iterators whose `value_type` is an integral type
@@ -168,9 +160,6 @@ use the same file with stdio functions, but it must not wait for another thread
   ([`JSON_DISABLE_STREAM_BULK_SCAN`](../macros/json_disable_stream_bulk_scan.md)).
 - Parsing from a `FILE*` locks the file once instead of once per character in version 3.13.0
   ([`JSON_DISABLE_UNLOCKED_FILE_IO`](../macros/json_disable_unlocked_file_io.md)).
-- Ill-formed UTF-16 and UTF-32 input (a lone surrogate, a high surrogate that is not followed by its low surrogate, or
-  a code point above `U+10FFFF`) is rejected with `parse_error.101` in version 3.13.0; before that, some such
-  sequences were accepted or ended the input silently.
 
 !!! warning "Deprecation"
 

docs/mkdocs/docs/api/basic_json_document/index.md

@@ -115,12 +115,6 @@ points apply to every edit:
   [`source_offset()`](../basic_json_view/source_offset.md) of a value introduced by an edit is
   `#!cpp static_cast<std::size_t>(-1)`, the same value it reports for a decoded string.
 
-## See also
-
-- [basic_json_view](../basic_json_view/index.md) - the read-only handle to a value of a document
-- [json_document](../json_document.md) - the document type of the default specialization
-- [Zero-copy JSON views](../../features/json_view.md) - the concept behind documents and views
-
 ## Version history
 
 - Added in version 3.13.0.

docs/mkdocs/docs/api/basic_json_document/parse.md

@@ -21,18 +21,17 @@ static basic_json_document parse(IteratorType first, IteratorType last,
 
 Both overloads accept the same JSON text as [`BasicJsonType::parse()`](../basic_json/parse.md), with the same
 `ignore_comments`/`ignore_trailing_commas` options, but build a [`basic_json_document`](index.md) (a flat index into
-the input) instead of a tree of `BasicJsonType` values. They take the same inputs as `BasicJsonType::parse()`, plus a
-pointer to a null-terminated wide string, which `BasicJsonType::parse()` does not accept. Byte inputs that stay alive
-can be borrowed; all others (streams, `#!cpp FILE*`, wide characters, non-contiguous iterators) are read into a copy
-that the document owns -- see the Notes. Use [`parse_copy()`](parse_copy.md) to get an owned copy of a borrowable input
-as well.
+the input) instead of a tree of `BasicJsonType` values. They take the same inputs as `BasicJsonType::parse()`. Byte
+inputs that stay alive can be borrowed; all others (streams, `#!cpp FILE*`, wide characters, non-contiguous
+iterators) are read into a copy that the document owns -- see the Notes. Use [`parse_copy()`](parse_copy.md) to get an
+owned copy of a borrowable input as well.
 
 ## Template parameters
 
 `InputType`
 :   A compatible input, for instance:
 
-    - a `#!cpp std::string`, a `#!cpp std::string_view` (C++17), or a C-style array of single-byte characters
+    - a `#!cpp std::string`, `#!cpp std::string_view`, or a C-style array of single-byte characters
     - a pointer to a null-terminated string of single-byte characters (`#!cpp char`, `#!cpp signed char`,
       `#!cpp unsigned char`, `#!cpp std::uint8_t`)
     - a container for which `#!cpp obj.data()` and `#!cpp obj.size()` give contiguous single-byte access, e.g.
@@ -48,10 +47,6 @@ as well.
     `#!cpp std::string::iterator`; the iterators of single-byte characters are borrowed or read like the byte inputs
     above, those of wide characters are converted to UTF-8
 
-Ill-formed UTF-16 and UTF-32 sequences are rejected: a lone surrogate, a high surrogate that is not followed by its low
-surrogate, and a code point above `U+10FFFF` all yield a
-[`parse_error.101`](../../home/exceptions.md#jsonexceptionparse_error101).
-
 ## Parameters
 
 `input` (in)
@@ -160,5 +155,3 @@ buffer of a given length, pass a pair of pointers: `#!cpp parse(ptr, ptr + len)`
 ## Version history
 
 - Added in version 3.13.0.
-- Ill-formed UTF-16 and UTF-32 input (a lone surrogate, a high surrogate that is not followed by its low surrogate, or
-  a code point above `U+10FFFF`) is rejected with `parse_error.101`.

docs/mkdocs/docs/api/basic_json_view/index.md

@@ -143,12 +143,6 @@ still refers to it -- including ones taken before the change -- reads the new va
 
 - [**source_offset**](source_offset.md) - byte offset of this value in the document's source text
 
-## See also
-
-- [basic_json_document](../basic_json_document/index.md) - the document a view refers into
-- [materialize](materialize.md) - build the `BasicJsonType` value of a subtree
-- [Zero-copy JSON views](../../features/json_view.md) - the concept behind documents and views
-
 ## Version history
 
 - Added in version 3.13.0.

docs/mkdocs/docs/examples/accept__wide_string.cpp

@@ -1,18 +0,0 @@
-#include <iostream>
-#include <string>
-#include <nlohmann/json.hpp>
-
-using json = nlohmann::json;
-
-int main()
-{
-    // a valid JSON text, given as UTF-16
-    const std::u16string valid_text = u"{\"name\":\"\u00e9\"}";
-
-    // an invalid JSON text, given as UTF-16
-    const std::u16string invalid_text = u"{\"name\":\"\u00e9\",}";
-
-    std::cout << std::boolalpha
-              << json::accept(valid_text) << ' '
-              << json::accept(invalid_text) << '\n';
-}

docs/mkdocs/docs/examples/accept__wide_string.output

@@ -1 +0,0 @@
-true false

docs/mkdocs/docs/examples/parse__wide_string.cpp

@@ -1,27 +0,0 @@
-#include <iostream>
-#include <string>
-#include <nlohmann/json.hpp>
-
-using json = nlohmann::json;
-
-int main()
-{
-    // the same JSON text, given in different encodings
-    const std::string utf8 = "{\"name\":\"\u00e9\"}";
-    const std::u16string utf16 = u"{\"name\":\"\u00e9\"}";
-    const std::u32string utf32 = U"{\"name\":\"\u00e9\"}";
-    const std::wstring wide = L"{\"name\":\"\u00e9\"}";
-
-    // the encoding is deduced from the character type of the input, and all
-    // of them yield the same value
-    std::cout << json::parse(utf8) << '\n'
-              << json::parse(utf16) << '\n'
-              << json::parse(utf32) << '\n'
-              << json::parse(wide) << '\n';
-
-    // a wide string literal works as well: u"..." is UTF-16, U"..." is UTF-32
-    std::cout << json::parse(u"[1, 2, 3]") << '\n';
-
-    // iterators over wide characters work as well
-    std::cout << json::parse(utf16.begin(), utf16.end()) << '\n';
-}

docs/mkdocs/docs/examples/parse__wide_string.output

@@ -1,6 +0,0 @@
-{"name":"é"}
-{"name":"é"}
-{"name":"é"}
-{"name":"é"}
-[1,2,3]
-{"name":"é"}

docs/mkdocs/docs/features/conversions.md

@@ -38,16 +38,6 @@ which avoids repeating it:
     --8<-- "examples/get_to.output"
     ```
 
-!!! info "Getting values out of a document or a view"
-
-    A [view](../api/basic_json_view/index.md) offers the same `get<T>()` conversion, but reads it straight from the
-    document's index, without building a `basic_json` value first. Two conversions never copy at all:
-    [`get_string()`](../api/basic_json_view/get_string.md) returns a string as a view into the source text, and
-    [`number_token()`](../api/basic_json_view/number_token.md) returns a number exactly as it was written. Every other
-    type is obtained from a subtree that is built on demand with
-    [`materialize()`](../api/basic_json_view/materialize.md). See
-    [Zero-copy JSON views](json_view.md#getting-values-out-without-copying).
-
 The library already knows how to convert to and from the scalar types and the STL containers (such as
 `#!cpp std::vector`, `#!cpp std::map`, `#!cpp std::array`, `#!cpp std::optional`, and many more). Converting a JSON
 object back to a `#!cpp std::map` or a JSON array back to a `#!cpp std::vector` therefore works without any extra code:
@@ -244,4 +234,3 @@ The conversions above are built in for standard types. To make the same syntax w
 - [`get_ref`](../api/basic_json/get_ref.md) / [`get_ptr`](../api/basic_json/get_ptr.md) - access the stored value without copying
 - [Arbitrary types conversions](arbitrary_types.md) - support your own types
 - [`JSON_USE_IMPLICIT_CONVERSIONS`](../api/macros/json_use_implicit_conversions.md) - toggle implicit conversions
-- [Zero-copy JSON views](json_view.md) - get values out of a document or a view

docs/mkdocs/docs/features/creating_values.md

@@ -1,9 +1,7 @@
 # Creating JSON values
 
 There are several ways to create a JSON value in memory. This page gives an overview; to read a value from JSON text
-instead, see [parsing](parsing/index.md). A JSON text can also be parsed into a
-[document](../api/basic_json_document/index.md) and read through a [view](../api/basic_json_view/index.md) instead of
-into a value; see [Zero-copy JSON views](json_view.md).
+instead, see [parsing](parsing/index.md).
 
 ## From C++ values
 
@@ -103,4 +101,3 @@ string `#!json "42"`.
 - [`operator""_json`](../api/operator_literal_json.md) - the `_json` literal
 - [Converting values](conversions.md) - which C++ types can be used
 - [Parsing](parsing/index.md) - create a value from JSON text
-- [Zero-copy JSON views](json_view.md) - create a document and a view from JSON text instead of a value

docs/mkdocs/docs/features/macros.md

@@ -287,23 +287,6 @@ external dependency and is not defined by default.
 
 See [full documentation of `JSON_USE_SIMDUTF`](../api/macros/json_use_simdutf.md).
 
-## `JSON_VIEW_NO_SIMD`
-
-When defined, the parser of [`basic_json_document`](../api/basic_json_document/index.md)
-(`<nlohmann/json_view.hpp>`) uses only portable C++ to scan strings, instead of vector instructions (SSE2 on x86-64,
-NEON on AArch64). The same input is accepted or rejected either way; only the speed differs. It is not defined by
-default.
-
-See [full documentation of `JSON_VIEW_NO_SIMD`](../api/macros/json_view_no_simd.md).
-
-## `JSON_VIEW_USE_SSSE3`
-
-When defined on x86-64, the parser of [`basic_json_document`](../api/basic_json_document/index.md)
-(`<nlohmann/json_view.hpp>`) validates non-ASCII text in strings with SSSE3 without asking the CPU first. It is not
-defined by default.
-
-See [full documentation of `JSON_VIEW_USE_SSSE3`](../api/macros/json_view_use_ssse3.md).
-
 ## `NLOHMANN_DEFINE_TYPE_*(...)`, `NLOHMANN_DEFINE_DERIVED_TYPE_*(...)`
 
 The library defines 12 macros to simplify the serialization/deserialization of types. See the page on

docs/mkdocs/docs/features/modifying_values.md

@@ -85,16 +85,6 @@ json a = {1, 2, 3, 4};
 a.erase(1);             // [1,3,4]  (erase by index)
 ```
 
-## Editing a document
-
-The operations above work on a `basic_json` value. A JSON text that was parsed into a
-[document](json_view.md#editing-a-document) can also be changed in place, without building a `basic_json` tree for the
-parts that are not touched: [`set`](../api/basic_json_document/set.md) replaces a value or an object member,
-[`push_back`](../api/basic_json_document/push_back.md) appends to an array,
-[`insert`](../api/basic_json_document/insert.md) inserts into an array, and
-[`erase`](../api/basic_json_document/erase.md) removes a member or an element. These functions are available on the
-editable document types, such as [`json_editable_document`](../api/json_editable_document.md).
-
 ## See also
 
 - [`push_back`](../api/basic_json/push_back.md) / [`emplace_back`](../api/basic_json/emplace_back.md) - append to an array
@@ -103,4 +93,3 @@ editable document types, such as [`json_editable_document`](../api/json_editable
 - [`merge_patch`](../api/basic_json/merge_patch.md) - apply an RFC 7396 merge patch
 - [`erase`](../api/basic_json/erase.md) / [`clear`](../api/basic_json/clear.md) - remove elements
 - [JSON Patch and Diff](json_patch.md) and [JSON Merge Patch](merge_patch.md) - structured modifications
-- [Editing a document](json_view.md#editing-a-document) - change a parsed document in place

docs/mkdocs/docs/features/parsing/index.md

@@ -18,7 +18,6 @@ flowchart LR
 The [`parse`](../../api/basic_json/parse.md) function reads a JSON value from an input. The input can be
 
 - a string (`#!cpp std::string`, C string, or string literal),
-- a wide string (`#!cpp std::u16string`, `#!cpp std::u32string`, or `#!cpp std::wstring`, or a wide string literal),
 - a `#!cpp std::istream` (e.g., an `#!cpp std::ifstream` reading from a file),
 - a `#!cpp FILE*` pointer,
 - a pair of iterators over a contiguous range (e.g., a `#!cpp std::vector<std::uint8_t>`), or
@@ -33,18 +32,12 @@ std::ifstream f("example.json");
 json data = json::parse(f);
 ```
 
-The input must be encoded in UTF-8; other encodings are not supported. The exception is wide-character input, which
-is read as UTF-16 or UTF-32 -- depending on the character type -- and converted to UTF-8; see the
-[`parse`](../../api/basic_json/parse.md) page for an example. A single input may contain only one JSON value.
+The input must be encoded in UTF-8; other encodings are not supported. A single input may contain only one JSON value.
 Inputs consisting of multiple values separated by newlines are handled by the [JSON Lines](json_lines.md) format.
 
 By default, the library rejects comments and trailing commas. Both can be enabled with parameters of the `parse`
 function — see [comments](../comments.md) and [trailing commas](../trailing_commas.md).
 
-[`basic_json_document::parse`](../../api/basic_json_document/parse.md) accepts the same inputs with the same options,
-but keeps the result in a document rather than in a `basic_json` value; see
-[Documents and views](#documents-and-views) below.
-
 ## Strictness and trailing data
 
 [`parse`](../../api/basic_json/parse.md) reads a single JSON value and requires the whole input to be consumed: any
@@ -105,24 +98,6 @@ The library offers two parsing models:
 You can influence a DOM parse without switching to the SAX interface by passing a
 [parser callback](parser_callbacks.md), which is called during parsing and can, for example, discard parts of the input.
 
-## Documents and views
-
-A third model sits between DOM and SAX: [`basic_json_document::parse`](../../api/basic_json_document/parse.md) parses
-the input into a document that keeps the text and a flat index of the values in it, and
-[`basic_json_view`](../../api/basic_json_view/index.md) gives read-only access to the values through that index.
-Strings and numbers stay in the source text and are only decoded when they are read, which is faster and uses less
-memory than building a tree of `basic_json` values when only parts of the document are needed. A document can also be
-modified in place, and it can be saved and loaded again without parsing it.
-
-```cpp
-#include <nlohmann/json_view.hpp>
-
-json_document doc = json_document::parse(R"({"happy": true, "pi": 3.141})");
-std::cout << doc.root()["happy"].get<bool>() << '\n';
-```
-
-See [Zero-copy JSON views](../json_view.md) for the concept and the trade-offs.
-
 ## Exceptions
 
 When the input is not valid JSON, the `parse` function throws an exception by default. If exceptions are undesired or
@@ -133,7 +108,6 @@ options.
 ## See also
 
 - [`parse`](../../api/basic_json/parse.md) - deserialize from a compatible input
-- [`basic_json_document::parse`](../../api/basic_json_document/parse.md) - deserialize into a document instead of a value
 - [`accept`](../../api/basic_json/accept.md) - check if the input is valid JSON
 - [`sax_parse`](../../api/basic_json/sax_parse.md) - generate SAX events
 - [JSON Lines](json_lines.md) - parse newline-delimited JSON
@@ -141,4 +115,3 @@ options.
 - [SAX interface](sax_interface.md) - implement a custom SAX handler
 - [parsing and exceptions](parse_exceptions.md) - control error handling
 - [parsing untrusted input](untrusted_input.md) - what to consider when parsing input from untrusted sources
-- [Zero-copy JSON views](../json_view.md) - parse into a document and read it through a view

docs/mkdocs/docs/features/parsing/untrusted_input.md

@@ -81,9 +81,6 @@ Invalid UTF-8 is rejected while parsing, not just while serializing:
 
 - In JSON text, an ill-formed UTF-8 byte in a string is a
   [`parse_error.101`](../../home/exceptions.md#jsonexceptionparse_error101) ("invalid string: ill-formed UTF-8 byte").
-- In UTF-16 or UTF-32 input, an ill-formed sequence is a
-  [`parse_error.101`](../../home/exceptions.md#jsonexceptionparse_error101) as well: a lone surrogate, a high surrogate
-  that is not followed by its low surrogate, and a code point above `U+10FFFF` are all rejected.
 - In a binary format, a string that is not valid UTF-8 is a
   [`parse_error.113`](../../home/exceptions.md#jsonexceptionparse_error113).
 

docs/mkdocs/docs/home/debugging.md

@@ -1,7 +1,7 @@
 # Debugging
 
-This page collects the library's built-in debugger integrations and other debugging-related features. They are not
-covered by any single page elsewhere in the docs, so they are collected here.
+This page collects the library's built-in debugger integrations and other debugging-related features. They are
+not linked from a single place elsewhere in the docs, so are collected here.
 
 ## Visual Studio (natvis)
 

docs/mkdocs/docs/home/exceptions.md

@@ -99,8 +99,7 @@ As this global context comes at the price of storing one additional pointer per
 
     Now the exception message contains a JSON Pointer `/address/housenumber` that indicates which value has the wrong type.
 
-See [documentation of `JSON_DIAGNOSTICS`](../api/macros/json_diagnostics.md) for more information, and
-[Debugging](debugging.md) for the debugger integrations of the library.
+See [documentation of `JSON_DIAGNOSTICS`](../api/macros/json_diagnostics.md) for more information.
 
 ## Parse errors
 

docs/mkdocs/docs/home/faq.md

@@ -81,9 +81,8 @@ rejected by default, and the library never adds them when serializing.
 
 The library supports **Unicode input** as follows:
 
-- **UTF-8** encoded input is supported, which is the default encoding for JSON, according to [RFC 8259](https://tools.ietf.org/html/rfc8259.html#section-8.1).
-- `std::u16string` and `std::u32string` can be parsed, assuming UTF-16 and UTF-32 encoding, respectively; `std::wstring` works as well and is interpreted as UTF-16 or UTF-32, depending on the width of the platform's `wchar_t`. These encodings cannot be read from a `FILE*` or an `std::istream`, which carry bytes only.
-- Ill-formed UTF-16 or UTF-32 input is rejected: a lone surrogate, a high surrogate that is not followed by its low surrogate, and a code point above `U+10FFFF` all yield a parse error. The same holds for iterators over 2- or 4-byte characters.
+- Only **UTF-8** encoded input is supported, which is the default encoding for JSON, according to [RFC 8259](https://tools.ietf.org/html/rfc8259.html#section-8.1).
+- `std::u16string` and `std::u32string` can be parsed, assuming UTF-16 and UTF-32 encoding, respectively. These encodings are not supported when reading from files or other input containers.
 - Other encodings such as Latin-1 or ISO 8859-1 are **not** supported and will yield parse or serialization errors.
 - The library will not replace [Unicode noncharacters](http://www.unicode.org/faq/private_use.html#nonchar1).
 - Invalid surrogates (e.g., incomplete pairs such as `\uDEAD`) will yield parse errors.
@@ -147,7 +146,7 @@ json::parse(std::string("\"") + '\0' + "\""); // throws parse_error.101: control
 
     Why are wide strings (e.g., `std::wstring`) dumped as arrays of numbers?
 
-As described [above](#parse-errors-reading-non-ascii-characters), the library stores its strings as UTF-8. To store a wide string, you need to change the encoding.
+As described [above](#parse-errors-reading-non-ascii-characters), the library assumes UTF-8 as encoding.  To store a wide string, you need to change the encoding.
 
 !!! example
 

docs/mkdocs/docs/index.md

@@ -6,9 +6,7 @@ JSON for Modern C++ is a header-only C++11 library that turns JSON into a first-
 magic of modern C++ so that creating, reading, and modifying JSON values feels as natural as it does in languages like
 Python. The whole library is available as a single header, `json.hpp`, with no dependencies, no subproject, and no
 complex build system to set up; a companion header, `json_fwd.hpp`, provides forward declarations to keep compile times
-down, `<nlohmann/json_literals.hpp>` holds the user-defined string literals on their own, and
-`<nlohmann/json_view.hpp>` adds zero-copy views and editable documents. See
-[header-only integration](integration/index.md) for details. It is heavily unit-tested with 100% code
+down. See [header-only integration](integration/index.md) for details. It is heavily unit-tested with 100% code
 coverage, checked with Valgrind and the Clang Sanitizers for memory leaks, and continuously fuzz-tested by Google
 OSS-Fuzz.
 
@@ -60,8 +58,7 @@ Conda, and more), and pkg-config.
 
     ---
 
-    Creating, parsing, accessing, and serializing JSON values, JSON Pointer/Patch, binary formats, zero-copy views,
-    and more.
+    Creating, parsing, accessing, and serializing JSON values, JSON Pointer/Patch, binary formats, and more.
 
     [:octicons-arrow-right-24: Features](features/index.md)
 
@@ -77,8 +74,7 @@ Conda, and more), and pkg-config.
 
     ---
 
-    The complete reference for `basic_json` and its member functions, types, and related classes, as well as for
-    `json_document` and `json_view`.
+    The complete reference for `basic_json` and its member functions, types, and related classes.
 
     [:octicons-arrow-right-24: API documentation](api/basic_json/index.md)
 

include/nlohmann/detail/input/input_adapters.hpp

@@ -8,17 +8,18 @@
 
 #pragma once
 
-#include <algorithm> // count, find, min
+#include <algorithm> // count, find, max, min
 #include <array> // array
 #include <cstddef> // size_t
 #include <cstdint> // uint32_t
 #include <cstring> // strlen
 #include <iterator> // begin, end, iterator_traits, random_access_iterator_tag, distance, next, reverse_iterator
 #include <limits> // numeric_limits
+#include <memory> // unique_ptr
 #include <streambuf> // streambuf
 #include <string> // string, char_traits
 #include <type_traits> // enable_if, is_base_of, is_pointer, is_integral, remove_pointer
-#include <utility> // pair, declval
+#include <utility> // declval, move, pair
 #include <vector> // vector
 
 #ifndef JSON_NO_IO
@@ -72,6 +73,81 @@ enum class input_format_t { json, cbor, msgpack, ubjson, bson, bjdata, bon8 };
 
 #ifndef JSON_NO_IO
 /*!
+A growable array of characters for the part of a token that an input adapter
+copied out of its window: unlike std::vector<char>::push_back(), appending a
+character takes no reference to a local, and is a few instructions when there
+is room.
+*/
+class token_carry_buffer
+{
+  public:
+    token_carry_buffer() = default;
+    token_carry_buffer(const token_carry_buffer&) = delete;
+    token_carry_buffer& operator=(const token_carry_buffer&) = delete;
+    token_carry_buffer(token_carry_buffer&& rhs) noexcept
+        : m_data(std::move(rhs.m_data)), m_end(rhs.m_end), m_capacity_end(rhs.m_capacity_end)
+    {
+        rhs.m_end = rhs.m_capacity_end = nullptr;
+    }
+    token_carry_buffer& operator=(token_carry_buffer&&) = delete;
+    ~token_carry_buffer() = default;
+
+    void clear() noexcept
+    {
+        m_end = m_data.get();
+    }
+
+    void push_back(char c)
+    {
+        if (JSON_HEDLEY_UNLIKELY(m_end == m_capacity_end))
+        {
+            grow(1);
+        }
+        *m_end++ = c;
+    }
+
+    void append(const char* first, const char* last)
+    {
+        const auto n = static_cast<std::size_t>(last - first);
+        if (n > static_cast<std::size_t>(m_capacity_end - m_end))
+        {
+            grow(n);
+        }
+        if (n != 0)
+        {
+            std::memcpy(m_end, first, n);
+        }
+        m_end += n;
+    }
+
+    /// replace the contents of @a out
+    void copy_to(std::vector<char>& out) const
+    {
+        out.assign(m_data.get(), m_end);
+    }
+
+  private:
+    /// make room for @a extra more characters
+    JSON_HEDLEY_NEVER_INLINE void grow(std::size_t extra)
+    {
+        const auto size = static_cast<std::size_t>(m_end - m_data.get());
+        const auto capacity = static_cast<std::size_t>(m_capacity_end - m_data.get());
+        const std::size_t new_capacity = (std::max)(capacity < 32 ? static_cast<std::size_t>(32) : 2 * capacity, size + extra);
+        std::unique_ptr<char[]> data(new char[new_capacity]); // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
+        if (size != 0)
+        {
+            std::memcpy(data.get(), m_data.get(), size);
+        }
+        m_data = std::move(data);
+        m_end = m_data.get() + size;
+        m_capacity_end = m_data.get() + new_capacity;
+    }
+
+    std::unique_ptr<char[]> m_data{}; // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays,readability-redundant-member-init)
+    char* m_end = nullptr;
+    char* m_capacity_end = nullptr;
+};
+/*!
 Input adapter for stdio file access. This adapter read only 1 byte and do not use any
  buffer. This adapter is a very low level adapter.
 */
@@ -209,6 +285,7 @@ class input_stream_adapter
         , token_segment(rhs.token_segment)
         , current_in_get_area(rhs.current_in_get_area)
         , tracking_token(rhs.tracking_token)
+        , no_get_area(rhs.no_get_area)
 #endif
     {
         rhs.is = nullptr;
@@ -295,37 +372,20 @@ class input_stream_adapter
     std::char_traits<char>::int_type get_character()
     {
 #if !JSON_DISABLE_STREAM_BULK_SCAN
-        // sbumpc() refills the get area if it is exhausted
-        const bool refill = streambuf_access::get_ptr(sb) == streambuf_access::get_end(sb);
-        if (JSON_HEDLEY_UNLIKELY(refill))
-        {
-            save_token_segment();
-        }
-#endif
-
-        auto res = sb->sbumpc();
-        // set eof manually, as we don't use the istream interface.
-        if (JSON_HEDLEY_UNLIKELY(res == std::char_traits<char>::eof()))
-        {
-#if !JSON_DISABLE_STREAM_BULK_SCAN
-            current_in_get_area = false;
-#endif
-            is->clear(is->rdstate() | std::ios::eofbit);
-        }
-#if !JSON_DISABLE_STREAM_BULK_SCAN
-        else if (JSON_HEDLEY_UNLIKELY(refill))
-        {
-            // the consumed character is at gptr() - 1, unless the streambuf
-            // has no get area and returned it from uflow()
-            const char* const next = streambuf_access::get_ptr(sb);
-            after_refill(res, next != nullptr && next != streambuf_access::get_begin(sb));
-        }
-        else
+        // the character is in the get area: step over it, as sbumpc() does
+        const char* const next = streambuf_access::get_ptr(sb);
+        if (JSON_HEDLEY_LIKELY(next != streambuf_access::get_end(sb)))
         {
             current_in_get_area = true;
+            streambuf_access::bump(sb, 1);
+            return std::char_traits<char>::to_int_type(*next);
+        }
+        if (no_get_area)
+        {
+            return get_character_no_get_area();
         }
 #endif
-        return res;
+        return get_character_slow();
     }
 #endif
 
@@ -334,6 +394,18 @@ class input_stream_adapter
     {
 #if JSON_PRECISE_STREAM_POSITION
         commit_lookahead();
+#endif
+#if !JSON_DISABLE_STREAM_BULK_SCAN
+        // the elements are in the get area: copy them, as sgetn() does, but
+        // without the virtual call
+        const char* const first = streambuf_access::get_ptr(sb);
+        const auto available = static_cast<std::size_t>(streambuf_access::get_end(sb) - first);
+        if (JSON_HEDLEY_LIKELY(count * sizeof(T) <= available && count * sizeof(T) <= static_cast<std::size_t>((std::numeric_limits<int>::max)())))
+        {
+            std::memcpy(dest, first, count * sizeof(T));
+            streambuf_access::bump(sb, static_cast<int>(count * sizeof(T)));
+            return count * sizeof(T);
+        }
 #endif
         auto res = static_cast<std::size_t>(sb->sgetn(reinterpret_cast<char*>(dest), static_cast<std::streamsize>(count * sizeof(T))));
         if (JSON_HEDLEY_UNLIKELY(res < count * sizeof(T)))
@@ -421,7 +493,7 @@ class input_stream_adapter
     // last one if @a drop_last.
     void copy_token(std::vector<char>& out, bool drop_last) const
     {
-        out = token_carry;
+        token_carry.copy_to(out);
         if (token_segment != nullptr)
         {
             out.insert(out.end(), token_segment, current_end());
@@ -434,6 +506,47 @@ class input_stream_adapter
 #endif
 
   private:
+#if !JSON_PRECISE_STREAM_POSITION
+    // the get area is exhausted, or there is none
+#if !JSON_DISABLE_STREAM_BULK_SCAN
+    JSON_HEDLEY_NEVER_INLINE
+#endif
+    std::char_traits<char>::int_type get_character_slow()
+    {
+#if !JSON_DISABLE_STREAM_BULK_SCAN
+        // sbumpc() refills the get area if it is exhausted
+        const bool refill = streambuf_access::get_ptr(sb) == streambuf_access::get_end(sb);
+        if (JSON_HEDLEY_UNLIKELY(refill))
+        {
+            save_token_segment();
+        }
+#endif
+
+        auto res = sb->sbumpc();
+        // set eof manually, as we don't use the istream interface.
+        if (JSON_HEDLEY_UNLIKELY(res == std::char_traits<char>::eof()))
+        {
+#if !JSON_DISABLE_STREAM_BULK_SCAN
+            current_in_get_area = false;
+#endif
+            is->clear(is->rdstate() | std::ios::eofbit);
+        }
+#if !JSON_DISABLE_STREAM_BULK_SCAN
+        else if (JSON_HEDLEY_UNLIKELY(refill))
+        {
+            // the consumed character is at gptr() - 1, unless the streambuf
+            // has no get area and returned it from uflow()
+            const char* const next = streambuf_access::get_ptr(sb);
+            after_refill(res, next != nullptr && next != streambuf_access::get_begin(sb));
+        }
+        else
+        {
+            current_in_get_area = true;
+        }
+#endif
+        return res;
+    }
+#endif
 #if !JSON_DISABLE_STREAM_BULK_SCAN
     // Reads the protected get-area pointers of any std::streambuf. Naming the
     // members through a derived class makes forming the member pointers
@@ -457,6 +570,11 @@ class input_stream_adapter
         {
             (s->*(&streambuf_access::gbump))(n);
         }
+        // sbumpc() for an exhausted get area
+        static std::streambuf::int_type next_character(std::streambuf* s)
+        {
+            return (s->*(&streambuf_access::uflow))();
+        }
     };
 
     // one past the character last returned by get_character(), if that is in
@@ -477,16 +595,53 @@ class input_stream_adapter
     {
         if (token_segment != nullptr)
         {
-            token_carry.insert(token_carry.end(), token_segment, streambuf_access::get_end(sb));
+            token_carry.append(token_segment, streambuf_access::get_end(sb));
             token_segment = nullptr;
         }
     }
 
+#if !JSON_PRECISE_STREAM_POSITION
+    // The streambuf has no get area (e.g., std::cin synchronized with stdio):
+    // every character comes from uflow(), and the token is kept by value.
+    JSON_HEDLEY_NEVER_INLINE std::char_traits<char>::int_type get_character_no_get_area()
+    {
+        // the get area is exhausted, so this is what sbumpc() does
+        const auto res = streambuf_access::next_character(sb);
+        if (JSON_HEDLEY_UNLIKELY(res == std::char_traits<char>::eof() || streambuf_access::get_end(sb) != nullptr))
+        {
+            return get_character_no_get_area_other(res);
+        }
+        // current_in_get_area is false: only the first character of a get
+        // area, or the end of the input, is read here
+        if (tracking_token)
+        {
+            token_carry.push_back(std::char_traits<char>::to_char_type(res));
+        }
+        return res;
+    }
+
+    // @a res came from uflow() of a streambuf without a get area: it is the
+    // end of the input, or the streambuf has a get area now
+    JSON_HEDLEY_NEVER_INLINE std::char_traits<char>::int_type get_character_no_get_area_other(std::char_traits<char>::int_type res)
+    {
+        if (res == std::char_traits<char>::eof())
+        {
+            current_in_get_area = false;
+            is->clear(is->rdstate() | std::ios::eofbit);
+            return res;
+        }
+        // the consumed character is at gptr() - 1
+        after_refill(res, streambuf_access::get_ptr(sb) != streambuf_access::get_begin(sb));
+        return res;
+    }
+#endif
+
     // character @a c was just read into a new get area (@a in_get_area) or
     // returned without one
     void after_refill(std::char_traits<char>::int_type c, bool in_get_area)
     {
         current_in_get_area = in_get_area;
+        no_get_area = !in_get_area;
         if (!tracking_token)
         {
             return;
@@ -525,13 +680,16 @@ class input_stream_adapter
 #endif
 #if !JSON_DISABLE_STREAM_BULK_SCAN
     /// the part of the current token that was in earlier get areas
-    std::vector<char> token_carry{}; // NOLINT(readability-redundant-member-init)
+    token_carry_buffer token_carry{}; // NOLINT(readability-redundant-member-init)
     /// start of the current token's part in the get area, or nullptr
     const char* token_segment = nullptr;
     /// whether the character last returned by get_character() is in the get area
     bool current_in_get_area = false;
     /// whether a token is tracked at all; the binary readers never start one
     bool tracking_token = false;
+    /// whether the streambuf had no get area when it was last asked for a
+    /// character
+    bool no_get_area = false;
 #endif
 };
 #endif  // JSON_NO_IO

single_include/nlohmann/json.hpp

@@ -8468,17 +8468,18 @@ NLOHMANN_JSON_NAMESPACE_END
 
 
 
-#include <algorithm> // count, find, min
+#include <algorithm> // count, find, max, min
 #include <array> // array
 #include <cstddef> // size_t
 #include <cstdint> // uint32_t
 #include <cstring> // strlen
 #include <iterator> // begin, end, iterator_traits, random_access_iterator_tag, distance, next, reverse_iterator
 #include <limits> // numeric_limits
+#include <memory> // unique_ptr
 #include <streambuf> // streambuf
 #include <string> // string, char_traits
 #include <type_traits> // enable_if, is_base_of, is_pointer, is_integral, remove_pointer
-#include <utility> // pair, declval
+#include <utility> // declval, move, pair
 #include <vector> // vector
 
 #ifndef JSON_NO_IO
@@ -8537,6 +8538,81 @@ enum class input_format_t { json, cbor, msgpack, ubjson, bson, bjdata, bon8 };
 
 #ifndef JSON_NO_IO
 /*!
+A growable array of characters for the part of a token that an input adapter
+copied out of its window: unlike std::vector<char>::push_back(), appending a
+character takes no reference to a local, and is a few instructions when there
+is room.
+*/
+class token_carry_buffer
+{
+  public:
+    token_carry_buffer() = default;
+    token_carry_buffer(const token_carry_buffer&) = delete;
+    token_carry_buffer& operator=(const token_carry_buffer&) = delete;
+    token_carry_buffer(token_carry_buffer&& rhs) noexcept
+        : m_data(std::move(rhs.m_data)), m_end(rhs.m_end), m_capacity_end(rhs.m_capacity_end)
+    {
+        rhs.m_end = rhs.m_capacity_end = nullptr;
+    }
+    token_carry_buffer& operator=(token_carry_buffer&&) = delete;
+    ~token_carry_buffer() = default;
+
+    void clear() noexcept
+    {
+        m_end = m_data.get();
+    }
+
+    void push_back(char c)
+    {
+        if (JSON_HEDLEY_UNLIKELY(m_end == m_capacity_end))
+        {
+            grow(1);
+        }
+        *m_end++ = c;
+    }
+
+    void append(const char* first, const char* last)
+    {
+        const auto n = static_cast<std::size_t>(last - first);
+        if (n > static_cast<std::size_t>(m_capacity_end - m_end))
+        {
+            grow(n);
+        }
+        if (n != 0)
+        {
+            std::memcpy(m_end, first, n);
+        }
+        m_end += n;
+    }
+
+    /// replace the contents of @a out
+    void copy_to(std::vector<char>& out) const
+    {
+        out.assign(m_data.get(), m_end);
+    }
+
+  private:
+    /// make room for @a extra more characters
+    JSON_HEDLEY_NEVER_INLINE void grow(std::size_t extra)
+    {
+        const auto size = static_cast<std::size_t>(m_end - m_data.get());
+        const auto capacity = static_cast<std::size_t>(m_capacity_end - m_data.get());
+        const std::size_t new_capacity = (std::max)(capacity < 32 ? static_cast<std::size_t>(32) : 2 * capacity, size + extra);
+        std::unique_ptr<char[]> data(new char[new_capacity]); // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
+        if (size != 0)
+        {
+            std::memcpy(data.get(), m_data.get(), size);
+        }
+        m_data = std::move(data);
+        m_end = m_data.get() + size;
+        m_capacity_end = m_data.get() + new_capacity;
+    }
+
+    std::unique_ptr<char[]> m_data{}; // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays,readability-redundant-member-init)
+    char* m_end = nullptr;
+    char* m_capacity_end = nullptr;
+};
+/*!
 Input adapter for stdio file access. This adapter read only 1 byte and do not use any
  buffer. This adapter is a very low level adapter.
 */
@@ -8674,6 +8750,7 @@ class input_stream_adapter
         , token_segment(rhs.token_segment)
         , current_in_get_area(rhs.current_in_get_area)
         , tracking_token(rhs.tracking_token)
+        , no_get_area(rhs.no_get_area)
 #endif
     {
         rhs.is = nullptr;
@@ -8760,37 +8837,20 @@ class input_stream_adapter
     std::char_traits<char>::int_type get_character()
     {
 #if !JSON_DISABLE_STREAM_BULK_SCAN
-        // sbumpc() refills the get area if it is exhausted
-        const bool refill = streambuf_access::get_ptr(sb) == streambuf_access::get_end(sb);
-        if (JSON_HEDLEY_UNLIKELY(refill))
-        {
-            save_token_segment();
-        }
-#endif
-
-        auto res = sb->sbumpc();
-        // set eof manually, as we don't use the istream interface.
-        if (JSON_HEDLEY_UNLIKELY(res == std::char_traits<char>::eof()))
-        {
-#if !JSON_DISABLE_STREAM_BULK_SCAN
-            current_in_get_area = false;
-#endif
-            is->clear(is->rdstate() | std::ios::eofbit);
-        }
-#if !JSON_DISABLE_STREAM_BULK_SCAN
-        else if (JSON_HEDLEY_UNLIKELY(refill))
-        {
-            // the consumed character is at gptr() - 1, unless the streambuf
-            // has no get area and returned it from uflow()
-            const char* const next = streambuf_access::get_ptr(sb);
-            after_refill(res, next != nullptr && next != streambuf_access::get_begin(sb));
-        }
-        else
+        // the character is in the get area: step over it, as sbumpc() does
+        const char* const next = streambuf_access::get_ptr(sb);
+        if (JSON_HEDLEY_LIKELY(next != streambuf_access::get_end(sb)))
         {
             current_in_get_area = true;
+            streambuf_access::bump(sb, 1);
+            return std::char_traits<char>::to_int_type(*next);
+        }
+        if (no_get_area)
+        {
+            return get_character_no_get_area();
         }
 #endif
-        return res;
+        return get_character_slow();
     }
 #endif
 
@@ -8799,6 +8859,18 @@ class input_stream_adapter
     {
 #if JSON_PRECISE_STREAM_POSITION
         commit_lookahead();
+#endif
+#if !JSON_DISABLE_STREAM_BULK_SCAN
+        // the elements are in the get area: copy them, as sgetn() does, but
+        // without the virtual call
+        const char* const first = streambuf_access::get_ptr(sb);
+        const auto available = static_cast<std::size_t>(streambuf_access::get_end(sb) - first);
+        if (JSON_HEDLEY_LIKELY(count * sizeof(T) <= available && count * sizeof(T) <= static_cast<std::size_t>((std::numeric_limits<int>::max)())))
+        {
+            std::memcpy(dest, first, count * sizeof(T));
+            streambuf_access::bump(sb, static_cast<int>(count * sizeof(T)));
+            return count * sizeof(T);
+        }
 #endif
         auto res = static_cast<std::size_t>(sb->sgetn(reinterpret_cast<char*>(dest), static_cast<std::streamsize>(count * sizeof(T))));
         if (JSON_HEDLEY_UNLIKELY(res < count * sizeof(T)))
@@ -8886,7 +8958,7 @@ class input_stream_adapter
     // last one if @a drop_last.
     void copy_token(std::vector<char>& out, bool drop_last) const
     {
-        out = token_carry;
+        token_carry.copy_to(out);
         if (token_segment != nullptr)
         {
             out.insert(out.end(), token_segment, current_end());
@@ -8899,6 +8971,47 @@ class input_stream_adapter
 #endif
 
   private:
+#if !JSON_PRECISE_STREAM_POSITION
+    // the get area is exhausted, or there is none
+#if !JSON_DISABLE_STREAM_BULK_SCAN
+    JSON_HEDLEY_NEVER_INLINE
+#endif
+    std::char_traits<char>::int_type get_character_slow()
+    {
+#if !JSON_DISABLE_STREAM_BULK_SCAN
+        // sbumpc() refills the get area if it is exhausted
+        const bool refill = streambuf_access::get_ptr(sb) == streambuf_access::get_end(sb);
+        if (JSON_HEDLEY_UNLIKELY(refill))
+        {
+            save_token_segment();
+        }
+#endif
+
+        auto res = sb->sbumpc();
+        // set eof manually, as we don't use the istream interface.
+        if (JSON_HEDLEY_UNLIKELY(res == std::char_traits<char>::eof()))
+        {
+#if !JSON_DISABLE_STREAM_BULK_SCAN
+            current_in_get_area = false;
+#endif
+            is->clear(is->rdstate() | std::ios::eofbit);
+        }
+#if !JSON_DISABLE_STREAM_BULK_SCAN
+        else if (JSON_HEDLEY_UNLIKELY(refill))
+        {
+            // the consumed character is at gptr() - 1, unless the streambuf
+            // has no get area and returned it from uflow()
+            const char* const next = streambuf_access::get_ptr(sb);
+            after_refill(res, next != nullptr && next != streambuf_access::get_begin(sb));
+        }
+        else
+        {
+            current_in_get_area = true;
+        }
+#endif
+        return res;
+    }
+#endif
 #if !JSON_DISABLE_STREAM_BULK_SCAN
     // Reads the protected get-area pointers of any std::streambuf. Naming the
     // members through a derived class makes forming the member pointers
@@ -8922,6 +9035,11 @@ class input_stream_adapter
         {
             (s->*(&streambuf_access::gbump))(n);
         }
+        // sbumpc() for an exhausted get area
+        static std::streambuf::int_type next_character(std::streambuf* s)
+        {
+            return (s->*(&streambuf_access::uflow))();
+        }
     };
 
     // one past the character last returned by get_character(), if that is in
@@ -8942,16 +9060,53 @@ class input_stream_adapter
     {
         if (token_segment != nullptr)
         {
-            token_carry.insert(token_carry.end(), token_segment, streambuf_access::get_end(sb));
+            token_carry.append(token_segment, streambuf_access::get_end(sb));
             token_segment = nullptr;
         }
     }
 
+#if !JSON_PRECISE_STREAM_POSITION
+    // The streambuf has no get area (e.g., std::cin synchronized with stdio):
+    // every character comes from uflow(), and the token is kept by value.
+    JSON_HEDLEY_NEVER_INLINE std::char_traits<char>::int_type get_character_no_get_area()
+    {
+        // the get area is exhausted, so this is what sbumpc() does
+        const auto res = streambuf_access::next_character(sb);
+        if (JSON_HEDLEY_UNLIKELY(res == std::char_traits<char>::eof() || streambuf_access::get_end(sb) != nullptr))
+        {
+            return get_character_no_get_area_other(res);
+        }
+        // current_in_get_area is false: only the first character of a get
+        // area, or the end of the input, is read here
+        if (tracking_token)
+        {
+            token_carry.push_back(std::char_traits<char>::to_char_type(res));
+        }
+        return res;
+    }
+
+    // @a res came from uflow() of a streambuf without a get area: it is the
+    // end of the input, or the streambuf has a get area now
+    JSON_HEDLEY_NEVER_INLINE std::char_traits<char>::int_type get_character_no_get_area_other(std::char_traits<char>::int_type res)
+    {
+        if (res == std::char_traits<char>::eof())
+        {
+            current_in_get_area = false;
+            is->clear(is->rdstate() | std::ios::eofbit);
+            return res;
+        }
+        // the consumed character is at gptr() - 1
+        after_refill(res, streambuf_access::get_ptr(sb) != streambuf_access::get_begin(sb));
+        return res;
+    }
+#endif
+
     // character @a c was just read into a new get area (@a in_get_area) or
     // returned without one
     void after_refill(std::char_traits<char>::int_type c, bool in_get_area)
     {
         current_in_get_area = in_get_area;
+        no_get_area = !in_get_area;
         if (!tracking_token)
         {
             return;
@@ -8990,13 +9145,16 @@ class input_stream_adapter
 #endif
 #if !JSON_DISABLE_STREAM_BULK_SCAN
     /// the part of the current token that was in earlier get areas
-    std::vector<char> token_carry{}; // NOLINT(readability-redundant-member-init)
+    token_carry_buffer token_carry{}; // NOLINT(readability-redundant-member-init)
     /// start of the current token's part in the get area, or nullptr
     const char* token_segment = nullptr;
     /// whether the character last returned by get_character() is in the get area
     bool current_in_get_area = false;
     /// whether a token is tracked at all; the binary readers never start one
     bool tracking_token = false;
+    /// whether the streambuf had no get area when it was last asked for a
+    /// character
+    bool no_get_area = false;
 #endif
 };
 #endif  // JSON_NO_IO

tests/src/unit-stream-bulk-scan.cpp

@@ -48,14 +48,16 @@ namespace
 class chunked_streambuf : public std::streambuf
 {
   public:
-    chunked_streambuf(std::string s, std::size_t chunk_size)
-        : m_data(std::move(s)), m_chunk_size(chunk_size)
+    // The first @a unbuffered characters are served without a get area, the
+    // rest in get areas of @a chunk_size.
+    chunked_streambuf(std::string s, std::size_t chunk_size, std::size_t unbuffered = 0)
+        : m_data(std::move(s)), m_chunk_size(chunk_size), m_unbuffered(unbuffered)
     {}
 
   protected:
     int_type underflow() override
     {
-        if (m_chunk_size == 0)
+        if (m_chunk_size == 0 || m_pos < m_unbuffered)
         {
             return m_pos < m_data.size() ? traits_type::to_int_type(m_data[m_pos]) : traits_type::eof();
         }
@@ -77,7 +79,7 @@ class chunked_streambuf : public std::streambuf
 
     int_type uflow() override
     {
-        if (m_chunk_size == 0)
+        if (m_chunk_size == 0 || m_pos < m_unbuffered)
         {
             return m_pos < m_data.size() ? traits_type::to_int_type(m_data[m_pos++]) : traits_type::eof();
         }
@@ -88,6 +90,7 @@ class chunked_streambuf : public std::streambuf
     std::string m_data;
     std::size_t m_pos = 0;
     std::size_t m_chunk_size;
+    std::size_t m_unbuffered;
 };
 
 // how a test reads from the stream
@@ -101,9 +104,9 @@ enum class read_mode
 // Read @a input from a chunked_streambuf and describe everything observable:
 // the values or the exception, the stream state after each value, and the
 // characters left in the stream.
-std::string observe(const std::string& input, std::size_t chunk_size, read_mode mode)
+std::string observe(const std::string& input, std::size_t chunk_size, read_mode mode, std::size_t unbuffered = 0)
 {
-    chunked_streambuf buf(input, chunk_size);
+    chunked_streambuf buf(input, chunk_size, unbuffered);
     std::istream is(&buf);
     std::string result;
     for (int i = 0; i < 4; ++i)
@@ -212,6 +215,31 @@ TEST_CASE("parsing from a std::istream scans the get area in bulk")
         }
     }
 
+    SECTION("results do not depend on a get area that appears late")
+    {
+        // the first characters come without a get area, then the streambuf
+        // starts to use one: the token that is read at that point is complete
+        for (const auto& input : corpus())
+        {
+            const std::string expected = observe(input, 0, read_mode::extract);
+            for (const std::size_t unbuffered :
+                    {
+                        1u, 2u, 3u, 5u, 8u, 13u, 21u
+                    })
+            {
+                for (const std::size_t chunk_size :
+                        {
+                            1u, 2u, 3u, 7u, 64u
+                        })
+                {
+                    CAPTURE(input)
+                    CAPTURE(unbuffered)
+                    CAPTURE(chunk_size)
+                    CHECK(observe(input, chunk_size, read_mode::extract, unbuffered) == expected);
+                }
+            }
+        }
+    }
 #endif // !defined(JSON_NOEXCEPTION)
 
     SECTION("values are the same as when parsing from a string")