Merge develop

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

docs/mkdocs/docs/api/basic_json/parser_callback_t.md

@@ -63,6 +63,15 @@ called:
 Whether the JSON value which called the function during parsing should be kept (`#!cpp true`) or not (`#!cpp false`). In
 the latter case, it is skipped completely, or replaced by `null` if it is the top-level value.
 
+## Notes
+
+For a `parse_event_t::key` event, `parsed` holds the decoded object name, with the same guarantees as the `val`
+parameter of [`json_sax::key`](../json_sax/key.md#notes): all escape sequences are decoded and `\uXXXX` escapes are
+stored as UTF-8, so different spellings of the same name yield identical strings; the name is validated completely
+before the callback is called, so a name with an invalid escape sequence or ill-formed UTF-8 causes a parse error and
+no `key` event; no Unicode normalization or case conversion is applied; and a name that occurs several times in the
+same object yields one `key` event per occurrence.
+
 ## Examples
 
 ??? example "Example: skip an object key while parsing"

docs/mkdocs/docs/api/json_sax/key.md

@@ -19,6 +19,23 @@ Whether parsing should proceed.
 
 It is safe to move the passed object key value.
 
+When parsing JSON text, `val` is the decoded object name, not its spelling in the input:
+
+- All escape sequences (`\"`, `\\`, `\/`, `\b`, `\f`, `\n`, `\r`, `\t`, `\uXXXX`, and surrogate pairs) are
+  decoded, and `\uXXXX` escapes are stored as UTF-8. Different spellings of the same name, such as `#!json "/"`
+  and `#!json "\/"`, or `#!json "\u00e9"` and `#!json "\u00E9"`, yield identical strings. An escaped `\u0000` yields
+  an embedded NUL byte.
+- The name is scanned and validated completely before `key` is called. If it contains an invalid escape sequence, an
+  unpaired surrogate, a control character, or ill-formed UTF-8, the parse fails with a
+  [`parse_error`](../../home/exceptions.md#parse-errors), and `key` is not called for it.
+- No Unicode normalization or case conversion is applied: names that differ in any byte are different strings.
+- `key` is called before the following `:` is read, so a parse error may still follow for the same member.
+- A name that occurs several times in the same object is passed once per occurrence.
+
+For the binary formats, object keys contain no escape sequences. For CBOR, MessagePack, UBJSON, and BSON, `val` holds
+the bytes stored in the input, and [`sax_parse`](../basic_json/sax_parse.md) passes them unchanged, without checking them
+for valid UTF-8. BON8 encodes characters itself, so an ill-formed name is a parse error and `key` is not called for it.
+
 ## Examples
 
 ??? example
@@ -45,3 +62,7 @@ It is safe to move the passed object key value.
 ## Version history
 
 - Added in version 3.2.0.
+- The behavior described in the notes above (decoded names, validation before `key` is called, no normalization, `key`
+  before the `:` check, one call per duplicate, bytes of binary formats passed unchanged) is the same in all versions
+  from 3.2.0 on, and for the `key` event of a [parser callback](../basic_json/parser_callback_t.md) from 3.0.0 on. BSON
+  is only supported from version 3.4.0, and BON8 only from version 3.13.0.

docs/mkdocs/docs/features/parsing/sax_interface.md

@@ -64,6 +64,9 @@ bool parse_error(std::size_t position, const std::string& last_token, const json
 
 The return value of each function determines whether parsing should proceed.
 
+Object keys are passed to `key` decoded and validated: escape sequences are already resolved, so different spellings of
+the same name yield the same string. See [`key`](../../api/json_sax/key.md#notes) for the exact guarantees.
+
 ## Stepping through the events
 
 The events for a small input, step by step, and what two different handlers do with them. Press *Play* or step with *Next*:

tests/src/unit-class_parser.cpp

@@ -3030,3 +3030,139 @@ TEST_CASE("a json_sax_dom_parser keeps the values read before a parse error")
         CHECK(read_before_error<nlohmann::ordered_json>(c.first) == c.second);
     }
 }
+
+TEST_CASE("object keys are delivered decoded and validated")
+{
+    // json_sax::key and the key event of a parser callback see the decoded name, and
+    // only after the whole name was scanned and validated
+    const auto keys_of_format = [](const std::string & text, const json::input_format_t format)
+    {
+        SaxEventLogger l;
+        json::sax_parse(text, &l, format);
+        std::vector<std::string> keys;
+        for (const auto& e : l.events)
+        {
+            if (e.compare(0, 4, "key(") == 0)
+            {
+                keys.push_back(e.substr(4, e.size() - 5));
+            }
+        }
+        return keys;
+    };
+
+    const auto keys_of = [&keys_of_format](const std::string & text)
+    {
+        return keys_of_format(text, json::input_format_t::json);
+    };
+
+    const auto callback_keys_of = [](const std::string & text)
+    {
+        std::vector<std::string> keys;
+        const json::parser_callback_t cb = [&keys](int /*depth*/, json::parse_event_t event, json & parsed)
+        {
+            if (event == json::parse_event_t::key)
+            {
+                keys.push_back(parsed.get<std::string>());
+            }
+            return true;
+        };
+        const json parsed = json::parse(text, cb, false);
+        static_cast<void>(parsed);
+        return keys;
+    };
+
+    SECTION("different spellings of a name yield the same string")
+    {
+        const std::string e_acute = "\xc3\xa9";
+        const std::string text = std::string(R"({"/":1,"\/":2,"\u002f":3,"\u00e9":4,"\u00E9":5,")") + e_acute + R"(":6})";
+        const std::vector<std::string> expected = {"/", "/", "/", e_acute, e_acute, e_acute};
+        CHECK(keys_of(text) == expected);
+        CHECK(callback_keys_of(text) == expected);
+    }
+
+    SECTION("escape sequences are decoded")
+    {
+        const std::string simple = R"({"\"\\\/\b\f\n\r\t":1})";
+        const std::vector<std::string> simple_expected = {std::string("\"\\/\b\f\n\r\t")};
+        CHECK(keys_of(simple) == simple_expected);
+        CHECK(callback_keys_of(simple) == simple_expected);
+
+        const std::string nul = R"({"\u0000":1})";
+        const std::vector<std::string> nul_expected = {std::string(1, '\0')};
+        CHECK(keys_of(nul) == nul_expected);
+        CHECK(callback_keys_of(nul) == nul_expected);
+
+        const std::string pair = R"({"\ud83d\ude00":1})";
+        const std::vector<std::string> pair_expected = {"\xf0\x9f\x98\x80"};
+        CHECK(keys_of(pair) == pair_expected);
+        CHECK(callback_keys_of(pair) == pair_expected);
+    }
+
+    SECTION("no normalization and no case conversion")
+    {
+        const std::string text = std::string("{\"\xc3\xa9\":1,\"e\xcc\x81\":2,\"A\":3,\"a\":4}");
+        const std::vector<std::string> expected = {"\xc3\xa9", "e\xcc\x81", "A", "a"};
+        CHECK(keys_of(text) == expected);
+        CHECK(callback_keys_of(text) == expected);
+    }
+
+    SECTION("a name that is not valid is not delivered")
+    {
+        const std::vector<std::string> texts =
+        {
+            R"({"\u12G4":1})",       // malformed escape
+            R"({"\udc00":1})",       // lone low surrogate
+            R"({"\ud800":1})",       // lone high surrogate
+            R"({"\ud800\u0041":1})", // high surrogate not followed by a low one
+            R"({"\x":1})",           // unknown escape
+            std::string("{\"a\x01\":1}"),   // control character
+            std::string("{\"\xff\":1}")     // ill-formed UTF-8
+        };
+        for (const auto& text : texts)
+        {
+            CAPTURE(text)
+            CHECK(!json::accept(text));
+            CHECK(keys_of(text).empty());
+            CHECK(callback_keys_of(text).empty());
+        }
+    }
+
+    SECTION("a name is delivered before the following colon is read")
+    {
+        const std::string text = R"({"a" 1})";
+        const std::vector<std::string> expected = {"a"};
+        CHECK(!json::accept(text));
+        CHECK(keys_of(text) == expected);
+        CHECK(callback_keys_of(text) == expected);
+    }
+
+    SECTION("a repeated name is delivered once per occurrence")
+    {
+        const std::string text = R"({"a":1,"a":2})";
+        const std::vector<std::string> expected = {"a", "a"};
+        CHECK(keys_of(text) == expected);
+        CHECK(callback_keys_of(text) == expected);
+    }
+
+    SECTION("binary formats pass the stored bytes through")
+    {
+        const std::string stored = "\xff\\";
+        const std::vector<std::string> expected = {stored};
+        const auto keys_of_bytes = [&keys_of_format](const std::vector<std::uint8_t>& v, json::input_format_t format)
+        {
+            return keys_of_format(std::string(v.begin(), v.end()), format);
+        };
+
+        CHECK(keys_of_bytes({0xA1, 0x62, 0xFF, 0x5C, 0x01}, json::input_format_t::cbor) == expected);
+        CHECK(keys_of_bytes({0x81, 0xA2, 0xFF, 0x5C, 0x01}, json::input_format_t::msgpack) == expected);
+        CHECK(keys_of_bytes({'{', 'i', 2, 0xFF, 0x5C, 'i', 1, '}'}, json::input_format_t::ubjson) == expected);
+        CHECK(keys_of_bytes({0x0D, 0, 0, 0, 0x10, 0xFF, 0x5C, 0, 1, 0, 0, 0, 0}, json::input_format_t::bson) == expected);
+    }
+
+    SECTION("BON8 rejects ill-formed UTF-8 in a name")
+    {
+        const std::vector<std::string> valid = {"\\"};
+        CHECK(keys_of_format(std::string("\x87\x5C\x91"), json::input_format_t::bon8) == valid);
+        CHECK(keys_of_format(std::string("\x87\xC3\x28\x91"), json::input_format_t::bon8).empty());
+    }
+}

tests/src/unit-json_view_image.cpp

@@ -185,8 +185,10 @@ std::uint32_t rng()
 
 TEST_CASE("json_view images: round trips")
 {
-    static_assert(json_view_image_test::has_save<json_document>::value && json_view_image_test::has_load<json_document>::value, "save() and load() on a little-endian target");
-    static_assert(json_view_image_test::has_save<ordered_json_editable_document>::value && json_view_image_test::has_load<ordered_json_editable_document>::value, "save() and load() on a little-endian target");
+    static_assert(json_view_image_test::has_save<json_document>::value, "save() on a little-endian target");
+    static_assert(json_view_image_test::has_load<json_document>::value, "load() on a little-endian target");
+    static_assert(json_view_image_test::has_save<ordered_json_editable_document>::value, "save() on a little-endian target");
+    static_assert(json_view_image_test::has_load<ordered_json_editable_document>::value, "load() on a little-endian target");
 
     SECTION("small documents")
     {