Docs: reading and writing files; patch_inplace() is not atomic (#6017)

docs/mkdocs/docs/features/files.md

@@ -0,0 +1,130 @@
+# Reading and writing files
+
+The library has no functions that take a file name. It reads from and writes to streams, so files are handled with
+`#!cpp std::ifstream` and `#!cpp std::ofstream` (or a `#!cpp FILE*` when reading). This page collects the recipes for
+the common cases and the mistakes that show up again and again.
+
+## Reading a JSON file
+
+Pass the stream to [`parse`](../api/basic_json/parse.md) directly, and check that the file was opened:
+
+```cpp
+#include <fstream>
+#include <stdexcept>
+#include <string>
+#include <nlohmann/json.hpp>
+
+using json = nlohmann::json;
+
+json read_json_file(const std::string& path)
+{
+    std::ifstream f(path);
+    if (!f.is_open())
+    {
+        throw std::runtime_error("cannot open " + path);
+    }
+    return json::parse(f);
+}
+```
+
+- **Check that the file is open.** A stream that failed to open looks like an empty input, so `parse` reports
+  [`parse_error.101`](../home/exceptions.md#jsonexceptionparse_error101) ("unexpected end of input") at line 1,
+  column 1 instead of saying that the file is missing. A relative path is resolved against the working directory of the
+  running program, which is often not the directory of the source file (see the
+  [FAQ](../home/faq.md#parsing-a-file-fails-with-unexpected-end-of-input)).
+- **Don't read the file into a string first.** Parsing from a `#!cpp std::ifstream` is only about 10 to 20 % slower
+  than reading the whole file into a string with one `read()` and parsing that, and it needs no memory for the copy (see
+  [input sources and performance](parsing/index.md#input-sources-and-performance)). The common way to read a file with
+  `#!cpp std::istreambuf_iterator<char>` is so slow that it takes about twice as long as parsing from the stream. Read
+  the file into a string only if you need the text itself, for example for an error message. This advice is for
+  `#!cpp json::parse`. [`json_document`](json_view.md) reads a stream into a buffer that it owns anyway, so there is no
+  copy to save: reading the file into a `#!cpp std::string` yourself and passing that as an lvalue, which the document
+  borrows, was about 6 to 12 % faster than parsing from the stream, with the same single buffer.
+- `parse` requires the file to hold exactly one JSON value. Files with one value per line are read with
+  [JSON Lines](parsing/json_lines.md); files too large for memory are covered in
+  [Large, streamed, and partial input](parsing/large_and_partial_input.md).
+
+## Writing a JSON file
+
+[`dump`](../api/basic_json/dump.md) the value into a string first, then write the string:
+
+```cpp
+void write_json_file(const std::string& path, const json& j)
+{
+    const std::string text = j.dump(4) + '\n';  // may throw: nothing is written yet
+
+    std::ofstream f(path, std::ios::binary);
+    if (!f.is_open())
+    {
+        throw std::runtime_error("cannot open " + path);
+    }
+    f << text;
+    f.close();
+    if (f.fail())
+    {
+        throw std::runtime_error("cannot write " + path);
+    }
+}
+```
+
+- **Serialize before opening the file.** Opening a `#!cpp std::ofstream` truncates the file. `dump` and
+  [`operator<<`](../api/operator_ltlt.md) throw [`type_error.316`](../home/exceptions.md#jsonexceptiontype_error316)
+  when a string is not valid UTF-8 (see [handling invalid UTF-8](serialization.md#handling-invalid-utf-8)), and
+  `#!cpp f << std::setw(4) << j` may already have written part of the value when it throws. Serializing into a string
+  first leaves the old file untouched in that case.
+- **Check the stream after closing it.** A full disk or a missing permission sets the stream's error state; it does not
+  throw unless you enabled exceptions on the stream with `exceptions()`.
+- `std::ios::binary` writes the text as it is. Without it, Windows writes `\r\n` for every newline of a pretty-printed
+  value.
+
+## Replacing a file without corrupting it
+
+The library never writes a file itself, so this section is not about the library. It is general C++ advice for a
+problem users of the library keep running into.
+
+A program that crashes or is killed while it writes leaves a truncated file behind. To make sure readers see either the
+old or the new contents, write to a temporary file in the same directory and rename it over the original
+(C++17, `#!cpp <filesystem>`):
+
+```cpp
+#include <filesystem>
+
+void replace_json_file(const std::filesystem::path& path, const json& j)
+{
+    std::filesystem::path tmp = path;
+    tmp += ".tmp";  // use a unique name if several processes may write the file
+
+    write_json_file(tmp.string(), j);
+    std::filesystem::rename(tmp, path);  // replaces path
+}
+```
+
+The temporary file must be in the same directory (more precisely, on the same file system) as the target: a rename
+across file systems does not fall back to a copy, it fails and throws `#!cpp std::filesystem::filesystem_error`. On
+POSIX systems, the rename atomically replaces the file. Standard C++ has no way to flush a file to the disk (`fsync`),
+so this protects against a crashed program, but not necessarily against a power loss.
+
+## Binary formats
+
+The [binary formats](binary_formats/index.md) work the same way, with the streams opened in binary mode:
+
+```cpp
+// read
+std::ifstream in("data.cbor", std::ios::binary);
+json j = json::from_cbor(in);
+
+// write
+const std::vector<std::uint8_t> bytes = json::to_cbor(j);
+std::ofstream out("data.cbor", std::ios::binary);
+out.write(reinterpret_cast<const char*>(bytes.data()), static_cast<std::streamsize>(bytes.size()));
+```
+
+Without `std::ios::binary`, Windows turns the bytes `0x0D 0x0A` into `0x0A` and may stop at a byte `0x1A` while reading,
+and turns `0x0A` into `0x0D 0x0A` while writing, which corrupts the data.
+
+## See also
+
+- [Parsing](parsing/index.md) - the inputs `parse` accepts
+- [Serialization](serialization.md) - the options of `dump`
+- [`parse`](../api/basic_json/parse.md), [`dump`](../api/basic_json/dump.md)
+- [`operator>>`](../api/operator_gtgt.md), [`operator<<`](../api/operator_ltlt.md)

docs/mkdocs/docs/features/index.md

@@ -46,6 +46,8 @@ C++ types, and finally serialize it again.
   [BJData](binary_formats/bjdata.md), [BON8](binary_formats/bon8.md), [BSON](binary_formats/bson.md),
   [CBOR](binary_formats/cbor.md), [MessagePack](binary_formats/messagepack.md), or [UBJSON](binary_formats/ubjson.md).
 - [Binary values](binary_values.md) — store and exchange raw byte sequences.
+- [Reading and writing files](files.md) — read and write files safely, including replacing a file without
+  corrupting it.
 
 ## How values are stored and configured
 

docs/mkdocs/docs/features/json_patch.md

@@ -29,7 +29,8 @@ other three continue on its result.
 
     [`patch`](../api/basic_json/patch.md) leaves the original value unchanged and returns the patched result as a copy.
     If the document is large and the original value is no longer needed,
-    [`patch_inplace`](../api/basic_json/patch_inplace.md) applies the same operations in place instead.
+    [`patch_inplace`](../api/basic_json/patch_inplace.md) applies the same operations in place instead. Unlike
+    `patch`, it is not atomic: if an operation fails, the operations before it stay applied.
 
 !!! note "Duplicate member names"
 

docs/mkdocs/docs/features/json_view.md

@@ -160,7 +160,9 @@ document: `#!cpp auto v = json_document::parse(text).root();` does not compile.
   wide. The limit is 16 bytes below 2^32, because the parser reads in 16-byte steps, and a position one step past the
   end must still fit.
 - **A stream is always read to its end.** There is no partial/streaming read of an `#!cpp std::istream`: the stream
-  is read to its end into a buffer the document owns, and parsing starts after that.
+  is read to its end into a buffer the document owns, and parsing starts after that. For a file, reading it into a
+  `#!cpp std::string` yourself and parsing that (an lvalue is borrowed) is therefore not more expensive in memory, and
+  was about 6 to 12 % faster than parsing from an `#!cpp std::ifstream`.
 - **Fewer source positions after edits.** With
   [`JSON_DIAGNOSTIC_POSITIONS`](../api/macros/json_diagnostic_positions.md) enabled,
   [`materialize()`](../api/basic_json_view/materialize.md) sets the same positions as `#!cpp basic_json::parse()`.

docs/mkdocs/docs/features/parsing/index.md

@@ -35,6 +35,8 @@ std::ifstream f("example.json");
 json data = json::parse(f);
 ```
 
+See [Reading and writing files](../files.md) for checking that the file was opened and for writing files safely.
+
 The input must be encoded in UTF-8; other encodings are not supported. The exception is wide-character input, which
 is read as UTF-16 or UTF-32 — depending on the character type — and converted to UTF-8; see the
 [`parse`](../../api/basic_json/parse.md) page for an example. A single input may contain only one JSON value.
@@ -84,6 +86,9 @@ ahead: after parsing, it is positioned exactly as if every character had been re
 [`operator>>`](../../api/operator_gtgt.md) can still read several values from one stream (see also
 [JSON Lines](json_lines.md)).
 
+This does not hold for [`json_document`](../json_view.md): it reads a stream into a buffer it owns, so the stream saves
+no memory, and reading the file into a `#!cpp std::string` yourself and borrowing it is about 6 to 12 % faster.
+
 Two macros switch these optimizations off without changing any result:
 [`JSON_DISABLE_STREAM_BULK_SCAN`](../../api/macros/json_disable_stream_bulk_scan.md) for stream buffers that do not
 follow the usual `std::streambuf` contract, and

docs/mkdocs/docs/features/performance.md

@@ -39,7 +39,9 @@ iterator pair over non-contiguous but random-access storage (e.g. `#!cpp std::de
 optimization but not the second, since the byte-scanning fast path requires contiguous storage.
 
 Practically: a file can be passed directly to `parse`, which also avoids the memory for a copy of the file. Reading it
-into a string first is no longer necessary for speed:
+into a string first is no longer necessary for speed (this is for `#!cpp json::parse`; a
+[`json_document`](json_view.md) reads a stream into an owned buffer anyway, so for it reading the file into a string
+and borrowing that is the faster way):
 
 ```cpp
 // fast: the stream buffer's get area is scanned in bulk

docs/mkdocs/docs/features/serialization.md

@@ -17,6 +17,9 @@ To write a value directly to a stream (for example, a file or `#!cpp std::cout`)
 std::cout << j << std::endl;
 ```
 
+To write a file without leaving a truncated file behind when serialization fails, see
+[Reading and writing files](files.md#writing-a-json-file).
+
 !!! note "String, not raw value"
 
     `dump` always returns a **JSON text**. Serializing a JSON string therefore includes the surrounding quotes and