Scan the buffer of a FILE in bulk with glibc and the Apple C library (#6030)

* Scan the buffer of a FILE in bulk with glibc and the Apple C library

Parsing from a FILE* still read one character at a time through
getc_unlocked(), while a std::istream has its get area scanned in bulk.
file_input_adapter now offers the lexer the same kind of window into the
buffer of the FILE: it reads the fields that the C library's own
getc_unlocked() macro reads (glibc: _IO_read_ptr and _IO_read_end; Apple:
_p and _r) and consumes exactly the characters getc_unlocked() would
return, so the file is left where it was left before. The part of the
current token in a buffer that is about to be refilled is kept for error
messages, as for streams. The window needs the file locked for the whole
parse, so JSON_DISABLE_UNLOCKED_FILE_IO and MemorySanitizer builds turn it
off; uClibc, which defines __GLIBC__ but has other fields, does not get it.

parse() from a FILE* gets 2.0-2.6x as fast on an Apple M1 Max and 1.7-2.2x
on x86-64 (glibc, GCC 15 and Clang 21), accept() 2.9-4.4x; that is about as
fast as parsing from a std::ifstream. The new tests read with FILE buffers
of 1 to 70 bytes and unbuffered files, after ungetc(), with SAX positions
and with binary strings across the buffer's edges.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Tests: no setvbuf() buffer below 2 bytes on Windows

The Microsoft C runtime rejects a _IOFBF buffer size below 2 as an
invalid parameter and ends the process (0xc0000409), which made
test-file_input fail on all MSVC and clang-cl jobs.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Fix CI: astyle formatting in unit-file_input.cpp

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Tests: limit the multi-character ungetc() test to glibc and Apple

The Windows C runtime only supports one pushed-back character, which
broke the Windows CI jobs.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz
Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Tests: sweep_allocator::construct() accepts pointers to const

Recent libc++ (emsdk latest) probes construct(const T*, const T&) in an
unevaluated context while checking for trivial copies; the static_cast to
void* in the noexcept specification then fails to compile.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz
(cherry picked from commit 794bf70823c12a2a1cb2a8a95f34dfb6357746a8)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Tests: guard sweep_allocator::construct() against const types instead of const_cast

The const_cast added in the previous commit is rejected by
cppcoreguidelines-pro-type-const-cast (clang-tidy). Make construct() not
viable for const U instead, which is what libc++ probes for.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz
(cherry picked from commit de099fa607001b710862fd1bbecef3609c54cbc4)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Fix CI: astyle and clang-tidy for sweep_allocator::construct()

astyle 3.4.13 reformats the enable_if<!...> default template argument
(Check amalgamation). Delete the const overload instead, which makes the
libc++ probe for const objects fail in the SFINAE context just the same
and needs no <type_traits>.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01N5NasEsrzHhSNouxSoEwqz
Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>

docs/mkdocs/docs/api/macros/json_disable_unlocked_file_io.md

@@ -5,7 +5,9 @@
 ```
 
 When defined to `1`, parsing from a `#!cpp FILE*` reads every character with `std::fgetc`, as in version 3.12.0 and
-earlier, instead of locking the file once and reading it with the unlocked stdio functions.
+earlier, instead of locking the file once and reading it with the unlocked stdio functions. This also turns off the
+scan of the `FILE`'s buffer in bulk (see below), which [`JSON_DISABLE_STREAM_BULK_SCAN`](json_disable_stream_bulk_scan.md)
+does not affect.
 
 It does not change any result: values, exceptions, and the position of the file after parsing are the same either way.
 Only the speed and the locking of the file differ (see the notes on locking below).
@@ -37,9 +39,17 @@ The default value is `0`: the file is locked once and read with the unlocked std
     | Microsoft (MSVC, clang-cl)                                 | `_lock_file`, `_fgetc_nolock`, `_unlock_file`       |
     | others (e.g., musl, MinGW, newlib)                         | `std::fgetc`, as before                             |
 
-    This makes parsing from a `FILE*` 3 to 4 times as fast with the Apple C library and 1.1 to 1.3 times as fast with
-    glibc, whose locking is cheaper. The binary formats ([`from_cbor`](../basic_json/from_cbor.md),
-    [`from_msgpack`](../basic_json/from_msgpack.md), ...) read from a `FILE*` the same way.
+    With glibc and the Apple C library, the parser also scans the characters that the `FILE` already holds in its
+    buffer in bulk, as it scans the get area of a stream (see
+    [`JSON_DISABLE_STREAM_BULK_SCAN`](json_disable_stream_bulk_scan.md)): it reads the same fields of the `FILE` that
+    the C library's own `getc_unlocked` reads, and it consumes exactly the characters that `getc_unlocked` would have
+    returned, so the file is left where it was left before.
+
+    Locking the file once makes parsing from a `FILE*` 3 to 4 times as fast with the Apple C library and 1.1 to 1.3
+    times as fast with glibc, whose locking is cheaper. Scanning the buffer in bulk makes `parse()` another 1.7 to 2.6
+    times as fast (and `accept()` 2.9 to 4.4 times), about as fast as parsing from a `std::ifstream`. The binary
+    formats ([`from_cbor`](../basic_json/from_cbor.md), [`from_msgpack`](../basic_json/from_msgpack.md), ...) read
+    from a `FILE*` the same way.
 
 !!! note "Locking"
 

docs/mkdocs/docs/features/parsing/index.md

@@ -77,7 +77,7 @@ them faster than others:
 |--------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------|
 | strings, string views, string literals, contiguous containers, pointer or iterator ranges over contiguous memory | the whole input is scanned in bulk: runs of string characters, numbers, and indentation are recognized several bytes at a time |
 | `#!cpp std::istream` (`#!cpp std::ifstream`, `#!cpp std::istringstream`, ...)            | the characters the stream buffer holds are scanned in bulk like a string; the stream buffer is asked for more when they are used up |
-| `#!cpp FILE*`                                                                              | one character at a time, with the file locked once for the whole parse              |
+| `#!cpp FILE*`                                                                              | with glibc and the Apple C library, the characters the `FILE` holds in its buffer are scanned in bulk like a stream's; elsewhere one character at a time; the file is locked once for the whole parse |
 | other iterator ranges (e.g., `#!cpp std::list<char>`, `#!cpp std::istreambuf_iterator<char>`), wide strings | one character at a time                                                             |
 
 Parsing a file from a `#!cpp std::ifstream` is therefore only about 10 to 20 % slower than reading the whole file into a

docs/mkdocs/docs/features/performance.md

@@ -27,8 +27,8 @@ A `#!cpp std::istream` (including `#!cpp std::ifstream`) gets the same bulk scan
 characters its stream buffer already holds (the *get area*) and asks the stream buffer for more only when they are used
 up. Nothing is read ahead, so the stream is left exactly where it would be after reading one character at a time, and
 [`operator>>`](../api/operator_gtgt.md) can still read several values from one stream. A stream buffer without a get
-area is read one character at a time. A `#!cpp FILE*` is still read one character at a time, but `file_input_adapter`
-locks the file once per parse instead of once per character. See
+area is read one character at a time. A `#!cpp FILE*` is locked once per parse instead of once per character, and with
+glibc and the Apple C library, the characters in its buffer are scanned in bulk as a stream's get area is. See
 [input sources and performance](parsing/index.md#input-sources-and-performance) for the full table. According to the
 description of [#5828](https://github.com/nlohmann/json/pull/5828), parsing from a stream is within about 10 to 20 % of
 parsing an equivalent `#!cpp std::string`.