Merge branch 'develop' into fix/json_pointer_create_object_5357

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

docs/mkdocs/docs/api/basic_json/operator[].md

@@ -128,10 +128,13 @@ Strong exception safety: if an exception occurs, the original value stays intact
 
         When the JSON pointer traverses intermediate levels that don't exist at all yet (not just a missing
         leaf), each missing level is created as an array or an object depending on whether the corresponding
-        pointer token parses as a non-negative integer: a numeric token creates an array, a non-numeric token
-        creates an object. For example, on an initially `#!json null` value, `/foo/0/0/0` creates nested arrays,
-        while `/foo/one/one/one` creates nested objects. This is not specified by the JSON Pointer RFC; it is
-        this library's own, intentional disambiguation rule. See also [JSON Pointer](../../features/json_pointer.md).
+        pointer token is a valid array index: the token `0`, a sequence of digits that does not begin with `0`,
+        or the token `-` creates an array, and every other token creates an object. For example, on an
+        initially `#!json null` value, `/foo/0/0/0` creates nested arrays, while `/foo/one/one/one` creates
+        nested objects. Tokens such as `01` or the empty token cannot be array indices (cf. RFC 6901, Sect. 4)
+        and therefore create objects, just as they would if the level already existed as an object. This is not
+        specified by the JSON Pointer RFC; it is this library's own, intentional disambiguation rule. See also
+        [JSON Pointer](../../features/json_pointer.md).
 
 ## Examples
 

docs/mkdocs/docs/features/arbitrary_types.md

@@ -79,7 +79,6 @@ Some important things:
 * When using `get<your_type>()`, `your_type` **MUST** be [DefaultConstructible](https://en.cppreference.com/w/cpp/named_req/DefaultConstructible). (There is a way to bypass this requirement described later.)
 * In function `from_json`, use function [`at()`](../api/basic_json/at.md) to access the object values rather than `operator[]`. In case a key does not exist, `at` throws an exception that you can handle, whereas `operator[]` exhibits undefined behavior.
 * You do not need to add serializers or deserializers for STL types like `std::vector`: the library already implements these.
-* If you control the type, consider defining `to_json`/`from_json` as `friend` functions inside the class ("hidden friends"). Argument-dependent lookup then only finds them for your type, which also avoids a [GCC < 11 compilation error](../home/faq.md#incomplete-detector-type-with-gcc-11).
 
 
 ## Simplify your life with macros

docs/mkdocs/docs/home/faq.md

@@ -307,51 +307,6 @@ APP_CPPFLAGS += -frtti -fexceptions
 The code compiles successfully with [Android NDK](https://developer.android.com/ndk/index.html?hl=ml), Revision 9 - 11 (and possibly later) and [CrystaX's Android NDK](https://www.crystax.net/en/android/ndk) version 10.
 
 
-### Incomplete `detector` type with GCC < 11
-
-!!! question
-
-    Why does GCC 10 or older fail with `invalid use of incomplete type 'struct nlohmann::detail::detector<..., to_json_function, ...>'` for a type that holds an `optional` member?
-
-This happens with GCC 10 and older in C++11/C++14 mode when all of these hold:
-
-- a class `Holder` has an `optional<Dummy>` member (e.g., `boost::optional`),
-- `Dummy` has a constructor taking a `json` value, and
-- `to_json` for `Holder` is a free function in the namespace of `Dummy`.
-
-```cpp
-class Dummy {
-  public:
-    explicit Dummy(const nlohmann::json& j);
-};
-
-class Holder {
-    boost::optional<Dummy> d;
-};
-
-void to_json(nlohmann::json& j, const Holder& h);  // triggers the error
-```
-
-To decide whether `Dummy` is copyable, the compiler checks whether a `Dummy` can be converted to `json`. That check
-looks up `to_json` via argument-dependent lookup, finds the unrelated `to_json` for `Holder`, and eventually asks again
-whether `Dummy` is copyable. GCC before version 11 turns this cycle into a hard error; GCC 11 and later, Clang, and
-C++17 mode compile the code. The same error shows up without this library whenever a constrained converting constructor
-is involved, so the library can't avoid it.
-
-To work around this, define `to_json` (and `from_json`) as a *hidden friend* inside the class. That way,
-argument-dependent lookup only finds it for `Holder`:
-
-```cpp
-class Holder {
-    boost::optional<Dummy> d;
-
-    friend void to_json(nlohmann::json& j, const Holder& h) { /* ... */ }
-};
-```
-
-The [`NLOHMANN_DEFINE_TYPE_INTRUSIVE`](../api/macros/nlohmann_define_type_intrusive.md) macros define hidden friends as
-well. See [#3669](https://github.com/nlohmann/json/issues/3669) for details.
-
 ### Missing STL function
 
 !!! question "Questions"

include/nlohmann/detail/json_pointer.hpp

@@ -416,15 +416,19 @@ class json_pointer
             // convert null values to arrays or objects before continuing
             if (ptr->is_null())
             {
-                // check if the reference token is a number
-                const bool nums =
-                    std::all_of(reference_token.begin(), reference_token.end(),
-                                [](const unsigned char x)
+                // check if the reference token is a valid array index, that is
+                // a nonempty sequence of digits without a leading '0'
+                // (cf. RFC 6901, Sect. 4); tokens that could never be a valid
+                // array index (such as "01" or "") are treated as object keys
+                const bool nums = !reference_token.empty()
+                                  && (reference_token.size() == 1 || reference_token[0] != '0')
+                                  && std::all_of(reference_token.begin(), reference_token.end(),
+                                                 [](const unsigned char x)
                 {
                     return std::isdigit(x);
                 });
 
-                // change value to an array for numbers or "-" or to object otherwise
+                // change value to an array for array indices or "-" or to object otherwise
                 *ptr = (nums || reference_token == "-")
                        ? detail::value_t::array
                        : detail::value_t::object;

single_include/nlohmann/json.hpp

@@ -19060,15 +19060,19 @@ class json_pointer
             // convert null values to arrays or objects before continuing
             if (ptr->is_null())
             {
-                // check if the reference token is a number
-                const bool nums =
-                    std::all_of(reference_token.begin(), reference_token.end(),
-                                [](const unsigned char x)
+                // check if the reference token is a valid array index, that is
+                // a nonempty sequence of digits without a leading '0'
+                // (cf. RFC 6901, Sect. 4); tokens that could never be a valid
+                // array index (such as "01" or "") are treated as object keys
+                const bool nums = !reference_token.empty()
+                                  && (reference_token.size() == 1 || reference_token[0] != '0')
+                                  && std::all_of(reference_token.begin(), reference_token.end(),
+                                                 [](const unsigned char x)
                 {
                     return std::isdigit(x);
                 });
 
-                // change value to an array for numbers or "-" or to object otherwise
+                // change value to an array for array indices or "-" or to object otherwise
                 *ptr = (nums || reference_token == "-")
                        ? detail::value_t::array
                        : detail::value_t::object;

tests/src/unit-json_pointer.cpp

@@ -438,6 +438,72 @@ TEST_CASE("JSON pointers")
         }
     }
 
+    SECTION("creating intermediate levels")
+    {
+        SECTION("tokens that are valid array indices create arrays")
+        {
+            json j;
+            j["/0"_json_pointer] = 1;
+            CHECK(j == json({1}));
+
+            json j2;
+            j2["/2"_json_pointer] = 1;
+            CHECK(j2 == json({nullptr, nullptr, 1}));
+
+            json j3;
+            j3["/-"_json_pointer] = 1;
+            CHECK(j3 == json({1}));
+
+            json j4;
+            j4["/foo/0/0"_json_pointer] = 1;
+            CHECK(j4 == json({{"foo", {{1}}}}));
+        }
+
+        SECTION("tokens that are no valid array indices create objects")
+        {
+            json j;
+            j["/one"_json_pointer] = 1;
+            CHECK(j == json({{"one", 1}}));
+
+            // leading '0' can never be a valid array index (RFC 6901, Sect. 4)
+            json j2;
+            j2["/01"_json_pointer] = 1;
+            CHECK(j2 == json({{"01", 1}}));
+
+            // the empty token is a valid object key, but no valid array index
+            json j3;
+            j3["/"_json_pointer] = 1;
+            CHECK(j3 == json({{"", 1}}));
+        }
+
+        SECTION("creating a level yields the same result as reusing it (#5357)")
+        {
+            json j;
+            j["/a/b/01/d"_json_pointer] = "value";
+
+            json j_init = json::object();
+            j_init["/a/b"_json_pointer] = json::object();
+            j_init["/a/b/01/d"_json_pointer] = "value";
+
+            const json expected = json::parse(R"({"a":{"b":{"01":{"d":"value"}}}})");
+            CHECK(j == expected);
+            CHECK(j_init == expected);
+
+            // unflatten uses the same key
+            const json flat = {{"/a/b/01/d", "value"}};
+            CHECK(flat.unflatten() == expected);
+        }
+
+        SECTION("existing arrays still reject invalid indices")
+        {
+            json j = {1, 2, 3};
+            CHECK_THROWS_WITH_AS(j["/01"_json_pointer],
+                                 "[json.exception.parse_error.106] parse error: array index '01' must not begin with '0'", json::parse_error&);
+            CHECK_THROWS_WITH_AS(j.at("/01"_json_pointer),
+                                 "[json.exception.parse_error.106] parse error: array index '01' must not begin with '0'", json::parse_error&);
+        }
+    }
+
     SECTION("flatten")
     {
         json j =

tests/src/unit-regression2.cpp

@@ -241,52 +241,6 @@ class my_allocator : public std::allocator<T>
     };
 };
 
-/////////////////////////////////////////////////////////////////////
-// for #3669
-/////////////////////////////////////////////////////////////////////
-
-// mimics boost::optional's converting constructor, whose SFINAE check asks
-// whether T is constructible from const U&
-template<class T, class Arg>
-struct issue3669_is_constructible
-{
-    template<class T2, class A2, class = decltype(T2(std::declval<A2>()))>
-    static char test(int);
-    template<class, class>
-    static long test(...);
-    static constexpr bool value = sizeof(test<T, Arg>(0)) == 1;
-};
-
-template<class T>
-class issue3669_optional
-{
-  public:
-    issue3669_optional() = default;
-    template<class U>
-    issue3669_optional(const issue3669_optional<U>& /*unused*/, // NOLINT(google-explicit-constructor,hicpp-explicit-conversions)
-                       typename std::enable_if<issue3669_is_constructible<T, const U&>::value, bool>::type /*unused*/ = true) {}
-};
-
-class Issue3669Dummy
-{
-  public:
-    explicit Issue3669Dummy(const json& /*unused*/) {}
-};
-
-class Issue3669Holder
-{
-    issue3669_optional<Issue3669Dummy> d{};
-
-    // GCC < 11 (C++11/14) rejects a free to_json(json&, const Issue3669Holder&)
-    // here, because ADL for Issue3669Dummy finds it and closes an instantiation
-    // cycle; a hidden friend is only visible to ADL for Issue3669Holder
-    friend void to_json(json& j, const Issue3669Holder& h)
-    {
-        static_cast<void>(h.d); // silence -Wunused-private-field
-        j = "holder";
-    }
-};
-
 TEST_CASE("regression tests 2")
 {
     SECTION("issue #1001 - Fix memory leak during parser callback")
@@ -812,14 +766,6 @@ TEST_CASE("regression tests 2")
         CHECK(j == k);
     }
 
-    SECTION("issue #3669 - invalid use of incomplete type with optional member and to_json")
-    {
-        const Issue3669Holder h{};
-        const Issue3669Holder h2(h); // NOLINT(performance-unnecessary-copy-initialization)
-        const json j = h2;
-        CHECK(j == "holder");
-    }
-
 }
 
 TEST_CASE("regression test - parser callback must not lose a duplicate key's prior value")