diff --git a/docs/mkdocs/docs/api/basic_json/accept.md b/docs/mkdocs/docs/api/basic_json/accept.md index 4abf5d36e..41c6969a1 100644 --- a/docs/mkdocs/docs/api/basic_json/accept.md +++ b/docs/mkdocs/docs/api/basic_json/accept.md @@ -134,6 +134,8 @@ outside of a string, invalid) byte; see the [FAQ entry](../../home/faq.md#nul-by instead of treating it as end of input - [`JSON_DISABLE_STREAM_BULK_SCAN`](../macros/json_disable_stream_bulk_scan.md) - read a `std::istream` one character at a time +- [`JSON_DISABLE_UNLOCKED_FILE_IO`](../macros/json_disable_unlocked_file_io.md) - read a `FILE*` with the locking stdio + functions ## Version history @@ -147,6 +149,8 @@ outside of a string, invalid) byte; see the [FAQ entry](../../home/faq.md#nul-by it as end of input; planned to become the default in version 4.0.0. - Parsing from an `std::istream` scans the characters in its stream buffer in bulk in version 3.13.0 ([`JSON_DISABLE_STREAM_BULK_SCAN`](../macros/json_disable_stream_bulk_scan.md)). +- Parsing from a `FILE*` locks the file once instead of once per character in version 3.13.0 + ([`JSON_DISABLE_UNLOCKED_FILE_IO`](../macros/json_disable_unlocked_file_io.md)). !!! warning "Deprecation" diff --git a/docs/mkdocs/docs/api/basic_json/parse.md b/docs/mkdocs/docs/api/basic_json/parse.md index 7ac05b94b..f2e7eda69 100644 --- a/docs/mkdocs/docs/api/basic_json/parse.md +++ b/docs/mkdocs/docs/api/basic_json/parse.md @@ -116,9 +116,11 @@ outside of a string, invalid) byte; see the [FAQ entry](../../home/faq.md#nul-by Contiguous inputs (strings, string views, vectors, arrays, pointer ranges) are the fastest to parse. A `#!cpp std::istream` comes close: the parser scans the characters its stream buffer holds in bulk, without reading - ahead, so the stream is left exactly where it would be after reading one character at a time. See - [input sources and performance](../../features/parsing/index.md#input-sources-and-performance) and - [`JSON_DISABLE_STREAM_BULK_SCAN`](../macros/json_disable_stream_bulk_scan.md). + ahead, so the stream is left exactly where it would be after reading one character at a time. A `#!cpp FILE*` is + read one character at a time, but the file is locked only once, so other threads that use the same `FILE` object + wait until parsing is done. See [input sources and performance](../../features/parsing/index.md#input-sources-and-performance), + [`JSON_DISABLE_STREAM_BULK_SCAN`](../macros/json_disable_stream_bulk_scan.md), and + [`JSON_DISABLE_UNLOCKED_FILE_IO`](../macros/json_disable_unlocked_file_io.md). ## Examples @@ -257,6 +259,8 @@ outside of a string, invalid) byte; see the [FAQ entry](../../home/faq.md#nul-by instead of treating it as end of input - [`JSON_DISABLE_STREAM_BULK_SCAN`](../macros/json_disable_stream_bulk_scan.md) - read a `std::istream` one character at a time +- [`JSON_DISABLE_UNLOCKED_FILE_IO`](../macros/json_disable_unlocked_file_io.md) - read a `FILE*` with the locking stdio + functions ## Version history @@ -274,6 +278,8 @@ outside of a string, invalid) byte; see the [FAQ entry](../../home/faq.md#nul-by - Parsing from an `std::istream` scans the characters in its stream buffer in bulk in version 3.13.0 ([`JSON_DISABLE_STREAM_BULK_SCAN`](../macros/json_disable_stream_bulk_scan.md) restores reading one character at a time). +- Parsing from a `FILE*` locks the file once instead of once per character in version 3.13.0 + ([`JSON_DISABLE_UNLOCKED_FILE_IO`](../macros/json_disable_unlocked_file_io.md) restores `std::fgetc`). !!! warning "Deprecation" diff --git a/docs/mkdocs/docs/api/basic_json/sax_parse.md b/docs/mkdocs/docs/api/basic_json/sax_parse.md index 811cea6ff..b13147b05 100644 --- a/docs/mkdocs/docs/api/basic_json/sax_parse.md +++ b/docs/mkdocs/docs/api/basic_json/sax_parse.md @@ -116,6 +116,10 @@ consumer `sax` has a super-linear complexity. A UTF-8 byte order mark is silently ignored. +When reading from a `#!c FILE*`, the file is locked while it is parsed. The lock is recursive, so the SAX handler may +use the same file with stdio functions, but it must not wait for another thread that uses the same `FILE` object; see +[`JSON_DISABLE_UNLOCKED_FILE_IO`](../macros/json_disable_unlocked_file_io.md). + ## Examples ??? example @@ -139,6 +143,8 @@ A UTF-8 byte order mark is silently ignored. - [accept](accept.md) - check if the input is valid JSON - [`JSON_DISABLE_STREAM_BULK_SCAN`](../macros/json_disable_stream_bulk_scan.md) - read a `std::istream` one character at a time +- [`JSON_DISABLE_UNLOCKED_FILE_IO`](../macros/json_disable_unlocked_file_io.md) - read a `FILE*` with the locking stdio + functions ## Version history @@ -152,6 +158,8 @@ A UTF-8 byte order mark is silently ignored. after the parsed value when `strict` is `#!cpp false`. - Parsing from an `std::istream` scans the characters in its stream buffer in bulk in version 3.13.0 ([`JSON_DISABLE_STREAM_BULK_SCAN`](../macros/json_disable_stream_bulk_scan.md)). +- Parsing from a `FILE*` locks the file once instead of once per character in version 3.13.0 + ([`JSON_DISABLE_UNLOCKED_FILE_IO`](../macros/json_disable_unlocked_file_io.md)). !!! warning "Deprecation" diff --git a/docs/mkdocs/docs/api/macros/index.md b/docs/mkdocs/docs/api/macros/index.md index 149262989..82e3ac8e3 100644 --- a/docs/mkdocs/docs/api/macros/index.md +++ b/docs/mkdocs/docs/api/macros/index.md @@ -18,6 +18,8 @@ header. See also the [macro overview page](../../features/macros.md). - [**JSON_DISABLE_STREAM_BULK_SCAN**](json_disable_stream_bulk_scan.md) - read a `std::istream` one character at a time instead of scanning its stream buffer in bulk +- [**JSON_DISABLE_UNLOCKED_FILE_IO**](json_disable_unlocked_file_io.md) - read a `FILE*` with `std::fgetc` instead of + locking it once and using the unlocked stdio functions - [**JSON_PRECISE_STREAM_POSITION**](json_precise_stream_position.md) - opt in to leaving an input stream positioned right after a parsed number - [**JSON_STRICT_BINARY_UTF8**](json_strict_binary_utf8.md) - opt in to checking strings for valid UTF-8 in the CBOR, diff --git a/docs/mkdocs/docs/api/macros/json_disable_stream_bulk_scan.md b/docs/mkdocs/docs/api/macros/json_disable_stream_bulk_scan.md index 5a765061c..5ecedb9ac 100644 --- a/docs/mkdocs/docs/api/macros/json_disable_stream_bulk_scan.md +++ b/docs/mkdocs/docs/api/macros/json_disable_stream_bulk_scan.md @@ -79,7 +79,8 @@ The default value is `0`: streams are scanned in bulk. !!! note "Other inputs" Strings, string views, vectors, arrays, and pointer or iterator ranges over contiguous memory are scanned in bulk - anyway. Reading from a `#!cpp FILE*` is not affected by this macro. + anyway. Reading from a `#!cpp FILE*` is not affected by this macro; see + [`JSON_DISABLE_UNLOCKED_FILE_IO`](json_disable_unlocked_file_io.md) for that input. ## Examples @@ -135,6 +136,8 @@ The default value is `0`: streams are scanned in bulk. - [**sax_parse**](../basic_json/sax_parse.md) - generate SAX events - [**JSON_PRECISE_STREAM_POSITION**](json_precise_stream_position.md) - leave an input stream positioned right after a parsed number +- [**JSON_DISABLE_UNLOCKED_FILE_IO**](json_disable_unlocked_file_io.md) - read a `FILE*` with the locking stdio + functions - [Parsing: input sources and performance](../../features/parsing/index.md#input-sources-and-performance) ## Version history diff --git a/docs/mkdocs/docs/api/macros/json_disable_unlocked_file_io.md b/docs/mkdocs/docs/api/macros/json_disable_unlocked_file_io.md new file mode 100644 index 000000000..056110736 --- /dev/null +++ b/docs/mkdocs/docs/api/macros/json_disable_unlocked_file_io.md @@ -0,0 +1,113 @@ +# JSON_DISABLE_UNLOCKED_FILE_IO + +```cpp +#define JSON_DISABLE_UNLOCKED_FILE_IO /* value */ +``` + +When defined to `1`, parsing from a `#!cpp FILE*` reads every character with `std::fgetc`, as in version 3.12.0 and +earlier, instead of locking the file once and reading it with the unlocked stdio functions. + +It does not change any result: values, exceptions, and the position of the file after parsing are the same either way. +Only the speed and the locking of the file differ (see the notes on locking below). + +## Default definition + +The default value is `0`: the file is locked once and read with the unlocked stdio functions where they are available. + +```cpp +#define JSON_DISABLE_UNLOCKED_FILE_IO 0 +``` + +## Notes + +!!! note "Background" + + The C standard library makes every `std::fgetc` call thread-safe by locking and unlocking the `FILE` object. When + parsing from a `FILE*`, the parser reads one character at a time, so this happens for every character of the input. + + Instead, the parser locks the file once when parsing starts and unlocks it when parsing ends - also when an + exception is thrown - and reads the characters in between without locking: + + | C library | functions | + |------------------------------------------------------------|----------------------------------------------------| + | glibc (Linux), Apple (macOS, iOS), Android, FreeBSD, NetBSD, OpenBSD, DragonFly BSD | `flockfile`, `getc_unlocked`, `funlockfile` (POSIX) | + | Microsoft (MSVC, clang-cl) | `_lock_file`, `_fgetc_nolock`, `_unlock_file` | + | others (e.g., musl, MinGW, newlib) | `std::fgetc`, as before | + + This makes parsing from a `FILE*` 3 to 4 times as fast with the Apple C library and 1.1 to 1.3 times as fast with + glibc, whose locking is cheaper. The binary formats ([`from_cbor`](../basic_json/from_cbor.md), + [`from_msgpack`](../basic_json/from_msgpack.md), ...) read from a `FILE*` the same way. + +!!! note "Locking" + + While a value is parsed from a `FILE*`, the parsing thread holds the file's lock. The lock is recursive, so the + parsing thread itself - for instance, a [SAX](../json_sax/index.md) handler or a + [parser callback](../basic_json/parser_callback_t.md) - can keep using the file with stdio functions such as + `std::ftell`. Other threads that use the same `FILE` object wait until the parser is done. Before, they could + interleave their reads with the parser's, which made the parsed input unpredictable anyway. + + A program in which a SAX handler or parser callback waits for **another** thread that uses the **same** `FILE` + object would deadlock. Define the macro to `1` for such programs. + +!!! warning "Define consistently" + + The macro selects between two definitions of the library's internal `FILE*` input adapter. It must therefore be + defined identically for **every** translation unit that includes the library; mixing translation units that define + it with ones that do not is an ODR violation (and can leave a file locked). Prefer setting it as a compile + definition on the target rather than with `#!cpp #define` in individual source files. + +!!! note "Other inputs" + + Reading from a `#!cpp std::istream` is not affected by this macro; see + [`JSON_DISABLE_STREAM_BULK_SCAN`](json_disable_stream_bulk_scan.md) for that input. + +## Examples + +??? example "Example: parsing from a `FILE*` (default)" + + The file is locked once while it is parsed: + + ```cpp + #include + #include + + using json = nlohmann::json; + + int main() + { + std::FILE* f = std::fopen("example.json", "rb"); + json j = json::parse(f); // f is locked until parse() returns + std::fclose(f); + } + ``` + +??? example "Example: locking for every character (macro defined to 1)" + + With the macro, every character is read with `std::fgetc`, with the same result: + + ```cpp + #define JSON_DISABLE_UNLOCKED_FILE_IO 1 + #include + #include + + using json = nlohmann::json; + + int main() + { + std::FILE* f = std::fopen("example.json", "rb"); + json j = json::parse(f); // std::fgetc for every character + std::fclose(f); + } + ``` + +## See also + +- [**parse**](../basic_json/parse.md) - deserialize from a compatible input +- [**sax_parse**](../basic_json/sax_parse.md) - generate SAX events +- [**JSON_DISABLE_STREAM_BULK_SCAN**](json_disable_stream_bulk_scan.md) - read a `std::istream` one character at a + time +- [Parsing: input sources and performance](../../features/parsing/index.md#input-sources-and-performance) + +## Version history + +- Added in version 3.13.0. diff --git a/docs/mkdocs/docs/features/macros.md b/docs/mkdocs/docs/features/macros.md index 396862170..05bf2a4c9 100644 --- a/docs/mkdocs/docs/features/macros.md +++ b/docs/mkdocs/docs/features/macros.md @@ -149,6 +149,14 @@ is planned to become the default in version 4.0.0. See [full documentation of `JSON_DISABLE_TUPLE_REFERENCE_CONVERSION`](../api/macros/json_disable_tuple_reference_conversion.md). +## `JSON_DISABLE_UNLOCKED_FILE_IO` + +When defined to `1`, parsing from a `#!cpp FILE*` reads every character with `std::fgetc`, which locks the file each +time, instead of locking the file once and reading it with the unlocked stdio functions. Results are the same either +way. The default value is `0`. + +See [full documentation of `JSON_DISABLE_UNLOCKED_FILE_IO`](../api/macros/json_disable_unlocked_file_io.md). + ## `JSON_NO_AUTOMATIC_UDLS` When defined, `` does not include `` with the user-defined string literals diff --git a/docs/mkdocs/docs/features/parsing/index.md b/docs/mkdocs/docs/features/parsing/index.md index 3099b7751..f823786be 100644 --- a/docs/mkdocs/docs/features/parsing/index.md +++ b/docs/mkdocs/docs/features/parsing/index.md @@ -64,7 +64,7 @@ them faster than others: |--------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------| | strings, string views, string literals, contiguous containers, pointer or iterator ranges over contiguous memory | the whole input is scanned in bulk: runs of string characters, numbers, and indentation are recognized several bytes at a time | | `#!cpp std::istream` (`#!cpp std::ifstream`, `#!cpp std::istringstream`, ...) | the characters the stream buffer holds are scanned in bulk like a string; the stream buffer is asked for more when they are used up | -| `#!cpp FILE*` | one character at a time | +| `#!cpp FILE*` | one character at a time, with the file locked once for the whole parse | | other iterator ranges (e.g., `#!cpp std::list`, `#!cpp std::istreambuf_iterator`), wide strings | one character at a time | Parsing a file from a `#!cpp std::ifstream` is therefore only about 10 to 20 % slower than reading the whole file into a @@ -73,9 +73,11 @@ ahead: after parsing, it is positioned exactly as if every character had been re [`operator>>`](../../api/operator_gtgt.md) can still read several values from one stream (see also [JSON Lines](json_lines.md)). -The macro [`JSON_DISABLE_STREAM_BULK_SCAN`](../../api/macros/json_disable_stream_bulk_scan.md) switches the bulk -scanning of streams off without changing any result, for stream buffers that do not follow the usual `std::streambuf` -contract. +Two macros switch these optimizations off without changing any result: +[`JSON_DISABLE_STREAM_BULK_SCAN`](../../api/macros/json_disable_stream_bulk_scan.md) for stream buffers that do not +follow the usual `std::streambuf` contract, and +[`JSON_DISABLE_UNLOCKED_FILE_IO`](../../api/macros/json_disable_unlocked_file_io.md) for programs in which another +thread must use the same `#!cpp FILE` object while it is parsed. ## SAX vs. DOM parsing diff --git a/docs/mkdocs/mkdocs.yml b/docs/mkdocs/mkdocs.yml index be17eeb0e..806effd70 100644 --- a/docs/mkdocs/mkdocs.yml +++ b/docs/mkdocs/mkdocs.yml @@ -299,6 +299,7 @@ nav: - 'JSON_DISABLE_ENUM_SERIALIZATION': api/macros/json_disable_enum_serialization.md - 'JSON_DISABLE_STREAM_BULK_SCAN': api/macros/json_disable_stream_bulk_scan.md - 'JSON_DISABLE_TUPLE_REFERENCE_CONVERSION': api/macros/json_disable_tuple_reference_conversion.md + - 'JSON_DISABLE_UNLOCKED_FILE_IO': api/macros/json_disable_unlocked_file_io.md - 'JSON_HAS_CPP_11, JSON_HAS_CPP_14, JSON_HAS_CPP_17, JSON_HAS_CPP_20, JSON_HAS_CPP_23, JSON_HAS_CPP_26': api/macros/json_has_cpp_11.md - 'JSON_HAS_EXPERIMENTAL_FILESYSTEM, JSON_HAS_FILESYSTEM': api/macros/json_has_filesystem.md - 'JSON_HAS_RANGE_VIEW_CONVERSION': api/macros/json_has_range_view_conversion.md diff --git a/include/nlohmann/detail/input/input_adapters.hpp b/include/nlohmann/detail/input/input_adapters.hpp index 4102dae3a..72d4c1ba6 100644 --- a/include/nlohmann/detail/input/input_adapters.hpp +++ b/include/nlohmann/detail/input/input_adapters.hpp @@ -32,6 +32,23 @@ #include #include +#ifndef JSON_NO_IO + // Which unlocked stdio functions file_input_adapter uses (see + // JSON_DISABLE_UNLOCKED_FILE_IO): 1 = POSIX flockfile/getc_unlocked, + // 2 = Microsoft CRT _lock_file/_fgetc_nolock, 0 = none (std::fgetc, which + // locks the file for every character). Only C libraries known to declare + // the functions are listed; the macros are tested after , which + // defines __GLIBC__. + #if JSON_DISABLE_UNLOCKED_FILE_IO + #define JSON_UNLOCKED_FILE_IO 0 + #elif defined(_MSC_VER) + #define JSON_UNLOCKED_FILE_IO 2 + #elif defined(__GLIBC__) || defined(__APPLE__) || defined(__ANDROID__) || defined(__FreeBSD__) || defined(__NetBSD__) || defined(__OpenBSD__) || defined(__DragonFly__) + #define JSON_UNLOCKED_FILE_IO 1 + #else + #define JSON_UNLOCKED_FILE_IO 0 + #endif +#endif // JSON_NO_IO NLOHMANN_JSON_NAMESPACE_BEGIN namespace detail @@ -59,18 +76,38 @@ class file_input_adapter : m_file(f) { JSON_ASSERT(m_file != nullptr); + lock(); } // make class move-only file_input_adapter(const file_input_adapter&) = delete; - file_input_adapter(file_input_adapter&&) noexcept = default; + file_input_adapter(file_input_adapter&& rhs) noexcept + : m_file(rhs.m_file) + { + // the lock is owned by the thread, not the adapter: hand it over + rhs.m_file = nullptr; + } file_input_adapter& operator=(const file_input_adapter&) = delete; file_input_adapter& operator=(file_input_adapter&&) = delete; - ~file_input_adapter() = default; + ~file_input_adapter() + { + if (m_file != nullptr) + { + unlock(); + } + } std::char_traits::int_type get_character() noexcept { + // the file is locked for the lifetime of the adapter, so the + // per-character lock of std::fgetc can be skipped +#if JSON_UNLOCKED_FILE_IO == 1 + return getc_unlocked(m_file); +#elif JSON_UNLOCKED_FILE_IO == 2 + return _fgetc_nolock(m_file); +#else return std::fgetc(m_file); +#endif } // returns the number of characters successfully read @@ -81,6 +118,27 @@ class file_input_adapter } private: + // Lock the file once for the whole parse instead of once per character. + // The lock is recursive, so fread() and the user's own stdio calls on the + // same thread keep working. + void lock() noexcept + { +#if JSON_UNLOCKED_FILE_IO == 1 + flockfile(m_file); +#elif JSON_UNLOCKED_FILE_IO == 2 + _lock_file(m_file); +#endif + } + + void unlock() noexcept + { +#if JSON_UNLOCKED_FILE_IO == 1 + funlockfile(m_file); +#elif JSON_UNLOCKED_FILE_IO == 2 + _unlock_file(m_file); +#endif + } + /// the file pointer to read from std::FILE* m_file; }; diff --git a/include/nlohmann/detail/macro_scope.hpp b/include/nlohmann/detail/macro_scope.hpp index 83cceaabd..653e1a600 100644 --- a/include/nlohmann/detail/macro_scope.hpp +++ b/include/nlohmann/detail/macro_scope.hpp @@ -899,3 +899,7 @@ #ifndef JSON_DISABLE_STREAM_BULK_SCAN #define JSON_DISABLE_STREAM_BULK_SCAN 0 #endif + +#ifndef JSON_DISABLE_UNLOCKED_FILE_IO + #define JSON_DISABLE_UNLOCKED_FILE_IO 0 +#endif diff --git a/include/nlohmann/detail/macro_unscope.hpp b/include/nlohmann/detail/macro_unscope.hpp index c80e1168f..5e17e1f86 100644 --- a/include/nlohmann/detail/macro_unscope.hpp +++ b/include/nlohmann/detail/macro_unscope.hpp @@ -22,6 +22,8 @@ #undef JSON_DISABLE_ENUM_SERIALIZATION #undef JSON_DISABLE_TUPLE_REFERENCE_CONVERSION #undef JSON_DISABLE_STREAM_BULK_SCAN +#undef JSON_DISABLE_UNLOCKED_FILE_IO +#undef JSON_UNLOCKED_FILE_IO #ifndef JSON_TEST_KEEP_MACROS #undef JSON_CATCH diff --git a/single_include/nlohmann/json.hpp b/single_include/nlohmann/json.hpp index e3afae1f8..6ed411bd6 100644 --- a/single_include/nlohmann/json.hpp +++ b/single_include/nlohmann/json.hpp @@ -3242,6 +3242,10 @@ JSON_HEDLEY_DIAGNOSTIC_POP #define JSON_DISABLE_STREAM_BULK_SCAN 0 #endif +#ifndef JSON_DISABLE_UNLOCKED_FILE_IO + #define JSON_DISABLE_UNLOCKED_FILE_IO 0 +#endif + #if JSON_HAS_THREE_WAY_COMPARISON #include // partial_ordering #endif @@ -8001,6 +8005,23 @@ NLOHMANN_JSON_NAMESPACE_END // #include +#ifndef JSON_NO_IO + // Which unlocked stdio functions file_input_adapter uses (see + // JSON_DISABLE_UNLOCKED_FILE_IO): 1 = POSIX flockfile/getc_unlocked, + // 2 = Microsoft CRT _lock_file/_fgetc_nolock, 0 = none (std::fgetc, which + // locks the file for every character). Only C libraries known to declare + // the functions are listed; the macros are tested after , which + // defines __GLIBC__. + #if JSON_DISABLE_UNLOCKED_FILE_IO + #define JSON_UNLOCKED_FILE_IO 0 + #elif defined(_MSC_VER) + #define JSON_UNLOCKED_FILE_IO 2 + #elif defined(__GLIBC__) || defined(__APPLE__) || defined(__ANDROID__) || defined(__FreeBSD__) || defined(__NetBSD__) || defined(__OpenBSD__) || defined(__DragonFly__) + #define JSON_UNLOCKED_FILE_IO 1 + #else + #define JSON_UNLOCKED_FILE_IO 0 + #endif +#endif // JSON_NO_IO NLOHMANN_JSON_NAMESPACE_BEGIN namespace detail @@ -8028,18 +8049,38 @@ class file_input_adapter : m_file(f) { JSON_ASSERT(m_file != nullptr); + lock(); } // make class move-only file_input_adapter(const file_input_adapter&) = delete; - file_input_adapter(file_input_adapter&&) noexcept = default; + file_input_adapter(file_input_adapter&& rhs) noexcept + : m_file(rhs.m_file) + { + // the lock is owned by the thread, not the adapter: hand it over + rhs.m_file = nullptr; + } file_input_adapter& operator=(const file_input_adapter&) = delete; file_input_adapter& operator=(file_input_adapter&&) = delete; - ~file_input_adapter() = default; + ~file_input_adapter() + { + if (m_file != nullptr) + { + unlock(); + } + } std::char_traits::int_type get_character() noexcept { + // the file is locked for the lifetime of the adapter, so the + // per-character lock of std::fgetc can be skipped +#if JSON_UNLOCKED_FILE_IO == 1 + return getc_unlocked(m_file); +#elif JSON_UNLOCKED_FILE_IO == 2 + return _fgetc_nolock(m_file); +#else return std::fgetc(m_file); +#endif } // returns the number of characters successfully read @@ -8050,6 +8091,27 @@ class file_input_adapter } private: + // Lock the file once for the whole parse instead of once per character. + // The lock is recursive, so fread() and the user's own stdio calls on the + // same thread keep working. + void lock() noexcept + { +#if JSON_UNLOCKED_FILE_IO == 1 + flockfile(m_file); +#elif JSON_UNLOCKED_FILE_IO == 2 + _lock_file(m_file); +#endif + } + + void unlock() noexcept + { +#if JSON_UNLOCKED_FILE_IO == 1 + funlockfile(m_file); +#elif JSON_UNLOCKED_FILE_IO == 2 + _unlock_file(m_file); +#endif + } + /// the file pointer to read from std::FILE* m_file; }; @@ -36476,6 +36538,8 @@ struct formatter // NOLINT(cert-dcl58-c #undef JSON_DISABLE_ENUM_SERIALIZATION #undef JSON_DISABLE_TUPLE_REFERENCE_CONVERSION #undef JSON_DISABLE_STREAM_BULK_SCAN +#undef JSON_DISABLE_UNLOCKED_FILE_IO +#undef JSON_UNLOCKED_FILE_IO #ifndef JSON_TEST_KEEP_MACROS #undef JSON_CATCH diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index a314bfbda..28bf2e7fa 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -285,6 +285,10 @@ if (JSON_TestSimdutf) endif() endif() +# test-file_input checks from a second thread that parsing from a FILE* releases the file's lock +find_package(Threads REQUIRED) +json_test_set_test_options("test-file_input;test-file_input_unlocked_io_disabled" LINK_LIBRARIES Threads::Threads) + # *DO* use json_test_set_test_options() above this line json_test_should_build_32bit_test(json_32bit_test json_32bit_test_only "${JSON_32bitTest}") @@ -387,6 +391,15 @@ json_test_add_test_for(src/unit-stream-bulk-scan.cpp MAIN test_main CXX_STANDARDS ${test_cxx_standards} ${test_force} ) +# test reading from a FILE* again with the locking stdio functions +json_test_set_test_options(test-file_input_unlocked_io_disabled + COMPILE_DEFINITIONS JSON_DISABLE_UNLOCKED_FILE_IO=1 +) +json_test_add_test_for(src/unit-file_input.cpp + NAME test-file_input_unlocked_io_disabled + MAIN test_main CXX_STANDARDS ${test_cxx_standards} ${test_force} +) + endif() # *DO NOT* use json_test_set_test_options() below this line diff --git a/tests/src/unit-file_input.cpp b/tests/src/unit-file_input.cpp new file mode 100644 index 000000000..eaa378aa8 --- /dev/null +++ b/tests/src/unit-file_input.cpp @@ -0,0 +1,236 @@ +// __ _____ _____ _____ +// __| | __| | | | JSON for Modern C++ (supporting code) +// | | |__ | | | | | | version 3.12.0 +// |_____|_____|_____|_|___| https://github.com/nlohmann/json +// +// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann +// SPDX-License-Identifier: MIT + +#include "doctest_compatibility.h" + +#include +using nlohmann::json; + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +// Parsing from a FILE* locks the file once and reads it with the unlocked +// stdio functions (see JSON_DISABLE_UNLOCKED_FILE_IO). These tests pin that +// this is not observable: the results are the same, the file is unlocked +// again when parsing ends - also with an exception - and the same thread can +// keep using the file while it is locked, e.g., from a SAX callback. + +namespace +{ +// a file name of its own for every run: the test runs for several C++ +// standards and configurations at the same time, in the same directory +std::string temp_file_name() +{ + std::random_device rd; + return "unit-file_input-" + std::to_string(rd()) + ".tmp"; +} + +using file_ptr = std::unique_ptr; + +// write @a content to the file @a name and open it for reading +file_ptr open_with(const char* name, const std::string& content) +{ + { + std::ofstream file(name, std::ios::binary); + file << content; + } + return {std::fopen(name, "rb"), &std::fclose}; +} + +// The stdio lock is recursive, so whether parsing released it can only be +// seen from another thread: ftrylockfile() fails there while this thread +// holds the lock. ftrylockfile() is POSIX; elsewhere, the check is skipped. +#if defined(__GLIBC__) || defined(__APPLE__) || defined(__FreeBSD__) || defined(__NetBSD__) || defined(__OpenBSD__) +void check_unlocked(std::FILE* f) +{ + int result = -1; + std::thread other([f, &result] + { + result = ftrylockfile(f); + if (result == 0) + { + funlockfile(f); + } + }); + other.join(); + CHECK(result == 0); +} +#else +void check_unlocked(std::FILE* /*f*/) {} +#endif + +// SAX consumer that also reads the position of the file it is parsing, which +// takes the file's lock again on the same thread +class position_reading_sax : public nlohmann::json_sax +{ + public: + explicit position_reading_sax(std::FILE* f) : m_file(f) {} + position_reading_sax(const position_reading_sax&) = delete; + position_reading_sax(position_reading_sax&&) = delete; + position_reading_sax& operator=(const position_reading_sax&) = delete; + position_reading_sax& operator=(position_reading_sax&&) = delete; + ~position_reading_sax() override = default; + + bool null() override + { + return record(); + } + bool boolean(bool /*val*/) override + { + return record(); + } + bool number_integer(number_integer_t /*val*/) override + { + return record(); + } + bool number_unsigned(number_unsigned_t /*val*/) override + { + return record(); + } + bool number_float(number_float_t /*val*/, const string_t& /*s*/) override + { + return record(); + } + bool string(string_t& /*val*/) override + { + return record(); + } + bool binary(binary_t& /*val*/) override + { + return record(); + } + bool start_object(std::size_t /*elements*/) override + { + return record(); + } + bool key(string_t& /*val*/) override + { + return record(); + } + bool end_object() override + { + return record(); + } + bool start_array(std::size_t /*elements*/) override + { + return record(); + } + bool end_array() override + { + return record(); + } + bool parse_error(std::size_t /*position*/, const std::string& /*last_token*/, const nlohmann::detail::exception& /*ex*/) override + { + return false; + } + + const std::vector& positions() const + { + return m_positions; + } + + private: + bool record() + { + m_positions.push_back(std::ftell(m_file)); + return true; + } + + std::FILE* m_file; + std::vector m_positions{}; // NOLINT(readability-redundant-member-init) +}; +} // namespace + +TEST_CASE("parsing from a FILE*") +{ + const std::string temp_name = temp_file_name(); + const char* const name = temp_name.c_str(); + const std::string input = "{\"a\": [1, -2.5, \"three\", true, null], \"b\": {\"c\": \"caf\xc3\xa9\"}}"; + + SECTION("values are the same as when parsing from a string") + { + const file_ptr f = open_with(name, input); + REQUIRE(f != nullptr); + CHECK(json::parse(f.get()) == json::parse(input)); + // the whole input was consumed + std::array rest{}; + CHECK(std::fgets(rest.data(), static_cast(rest.size()), f.get()) == nullptr); + check_unlocked(f.get()); + } + + SECTION("the file is unlocked after a parse error") + { + const file_ptr f = open_with(name, "[1, 2, x]"); + REQUIRE(f != nullptr); + json _; + CHECK_THROWS_WITH_AS(_ = json::parse(f.get()), + "[json.exception.parse_error.101] parse error at line 1, column 8: syntax error while parsing value - invalid literal; last read: '2, x'", + json::parse_error&); + check_unlocked(f.get()); + + // the file is still usable: the rest of the input is there + std::array rest{}; + CHECK(std::fgets(rest.data(), static_cast(rest.size()), f.get()) != nullptr); + CHECK(std::string(rest.data()) == "]"); + } + + SECTION("the file can be parsed again and read with stdio afterwards") + { + const file_ptr f = open_with(name, "[1] [2] tail"); + REQUIRE(f != nullptr); + + // sax_parse with strict == false stops after the first value; an array + // ends with its closing bracket, so the space after it is not read + json j; + nlohmann::detail::json_sax_dom_parser sax1(j); + CHECK(json::sax_parse(f.get(), &sax1, json::input_format_t::json, false)); + CHECK(j == json({1})); + check_unlocked(f.get()); + + json k; + nlohmann::detail::json_sax_dom_parser sax2(k); + CHECK(json::sax_parse(f.get(), &sax2, json::input_format_t::json, false)); + CHECK(k == json({2})); + check_unlocked(f.get()); + + std::array rest{}; + CHECK(std::fgets(rest.data(), static_cast(rest.size()), f.get()) != nullptr); + CHECK(std::string(rest.data()) == " tail"); + } + + SECTION("the same thread can use the file while it is parsed") + { + const file_ptr f = open_with(name, "[10, 20, 30]"); + REQUIRE(f != nullptr); + position_reading_sax sax(f.get()); + CHECK(json::sax_parse(f.get(), &sax)); + // start_array, three numbers, end_array + CHECK(sax.positions().size() == 5); + CHECK(sax.positions().front() == 1); + check_unlocked(f.get()); + } + + SECTION("binary formats read from a FILE*") + { + const json j = {{"string", std::string(300, 'x')}, {"array", {1, 2, 3}}, {"number", 1.5}}; + const std::vector cbor = json::to_cbor(j); + const file_ptr f = open_with(name, std::string(cbor.begin(), cbor.end())); + REQUIRE(f != nullptr); + CHECK(json::from_cbor(f.get()) == j); + check_unlocked(f.get()); + } + + static_cast(std::remove(name)); +}