Merge branch 'develop' into fix/json_pointer_create_object_5357

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

docs/mkdocs/docs/api/basic_json/binary_t.md

@@ -64,7 +64,18 @@ values of that type directly to a `basic_json` instance, and they will automatic
 rather than arrays:
 
 ```cpp
-using custom_json = nlohmann::ordered_json::with_binary_t<std::vector<std::byte>>;
+using custom_json = nlohmann::basic_json<
+    nlohmann::ordered_map,  // ObjectType
+    std::vector,            // ArrayType
+    std::string,            // StringType
+    bool,                   // BooleanType
+    std::int64_t,           // NumberIntegerType
+    std::uint64_t,          // NumberUnsignedType
+    double,                 // NumberFloatType
+    std::allocator,         // AllocatorType
+    nlohmann::adl_serializer,
+    std::vector<std::byte>  // Custom BinaryType
+>;
 
 std::vector<std::byte> data{std::byte{1}, std::byte{2}, std::byte{3}};
 custom_json j = data;  // Creates a binary value, not an array

docs/mkdocs/docs/api/basic_json/operator[].md

@@ -137,10 +137,13 @@ Strong exception safety: if an exception occurs, the original value stays intact
 
         When the JSON pointer traverses intermediate levels that don't exist at all yet (not just a missing
         leaf), each missing level is created as an array or an object depending on whether the corresponding
-        pointer token parses as a non-negative integer: a numeric token creates an array, a non-numeric token
-        creates an object. For example, on an initially `#!json null` value, `/foo/0/0/0` creates nested arrays,
-        while `/foo/one/one/one` creates nested objects. This is not specified by the JSON Pointer RFC; it is
-        this library's own, intentional disambiguation rule. See also [JSON Pointer](../../features/json_pointer.md).
+        pointer token is a valid array index: the token `0`, a sequence of digits that does not begin with `0`,
+        or the token `-` creates an array, and every other token creates an object. For example, on an
+        initially `#!json null` value, `/foo/0/0/0` creates nested arrays, while `/foo/one/one/one` creates
+        nested objects. Tokens such as `01` or the empty token cannot be array indices (cf. RFC 6901, Sect. 4)
+        and therefore create objects, just as they would if the level already existed as an object. This is not
+        specified by the JSON Pointer RFC; it is this library's own, intentional disambiguation rule. See also
+        [JSON Pointer](../../features/json_pointer.md).
 
 !!! warning "Deprecation"
 

docs/mkdocs/docs/community/roadmap.md

@@ -36,26 +36,13 @@ work items are tracked in the [GitHub milestones](https://github.com/nlohmann/js
 ## API stability
 
 Releases follow [semantic versioning](https://semver.org): a minor or patch release of version 3.x does not break code
-that uses the public API, unless that code opts in to a change with a macro as described [below](#version-40). In
-particular, a 3.x release does not:
+that uses the public API. In particular, a 3.x release does not:
 
-- make breaking changes to the signature of a function: the types or order of its existing parameters, its return type,
-  its `noexcept` or `constexpr` specifier, or the const-ness of a member function. New parameters may be added if they
-  have a default value;
-- remove or rename a function or class, or change the template parameters of a public class template;
+- change the signature of a function (its parameter types, return type, number of parameters, or the const-ness of a
+  member function);
+- remove or rename a function or class;
 - change which exceptions a function throws, or the [exception ids](../home/exceptions.md);
-- change access specifiers, or change or remove existing default arguments. New default arguments may be added;
-- change the JSON type that a valid input parses to, or the text that `dump()` produces for a valid value;
-- accept input that was rejected before, or reject input that was accepted before;
-- change the order in which the keys of an object are iterated. The default type sorts keys, and
-  [`ordered_json`](../api/ordered_json.md) keeps insertion order;
-- change when iterators, pointers, or references are invalidated, or the state of a moved-from `basic_json`;
-- add or remove implicit conversions from `basic_json`;
-- change how `to_json` and `from_json` functions are found, or the behavior of
-  [`adl_serializer`](../api/adl_serializer/index.md);
-- add pure virtual functions to the [`json_sax`](../api/json_sax/index.md) interface;
-- remove, rename, renumber, or add enumerators of `value_t`;
-- remove or rename a documented macro, CMake option, CMake target, or header, or change what a documented macro does.
+- change access specifiers or default arguments.
 
 Exceptions to these rules, for instance when fixing a bug requires changing the exception a function throws, are
 documented in the [release notes](../home/releases.md).
@@ -64,14 +51,13 @@ The following are **not** part of the public API and may change in any release,
 
 - The text of exception messages returned by `what()`. Use the [exception id](../home/exceptions.md) to tell errors
   apart.
-- The ABI, including `sizeof(basic_json)` and the memory layout of its values. The
-  [versioned inline namespace](../features/namespace.md) turns mixing versions into a link error.
-- The hash values returned by `std::hash` for `basic_json`. Numbers that compare equal still hash equally.
+- The ABI, including `sizeof(basic_json)` and the memory layout of its values. Recompile your code when you upgrade the
+  library. The [versioned inline namespace](../features/namespace.md) turns mixing versions into a link error.
 - Everything in namespace `nlohmann::detail`, and macros and type traits that are not documented in the
   [API reference](../api/basic_json/index.md).
 
-Breaking changes are only added behind a macro whose default keeps the 3.x behavior. See [Version 4.0](#version-40) and
-the [macro overview](../features/macros.md).
+Changes that would break the public API are only added behind a macro whose default keeps the 3.x behavior, see
+[Version 4.0](#version-40).
 
 ## Version 4.0
 

docs/mkdocs/docs/examples/as_base_class.cpp

@@ -15,7 +15,19 @@ class base_class_with_hidden_members
     }
 };
 
-using json = nlohmann::json::with_base_class_t<base_class_with_hidden_members>;
+using json = nlohmann::basic_json <
+             std::map,
+             std::vector,
+             std::string,
+             bool,
+             std::int64_t,
+             std::uint64_t,
+             double,
+             std::allocator,
+             nlohmann::adl_serializer,
+             std::vector<std::uint8_t>,
+             base_class_with_hidden_members
+             >;
 
 int main()
 {

docs/mkdocs/docs/examples/contains__json_pointer.cpp

@@ -19,9 +19,25 @@ int main()
               << j.contains("/array/1"_json_pointer) << '\n'
               << j.contains("/array/-"_json_pointer) << '\n'
               << j.contains("/array/4"_json_pointer) << '\n'
-              << j.contains("/baz"_json_pointer) << '\n'
-              // an array index with a leading '0' is not found
-              << j.contains("/array/01"_json_pointer) << '\n'
-              // an array index that is not a number is not found
-              << j.contains("/array/one"_json_pointer) << std::endl;
+              << j.contains("/baz"_json_pointer) << std::endl;
+
+    try
+    {
+        // try to use an array index with leading '0'
+        j.contains("/array/01"_json_pointer);
+    }
+    catch (const json::parse_error& e)
+    {
+        std::cout << e.what() << '\n';
+    }
+
+    try
+    {
+        // try to use an array index that is not a number
+        j.contains("/array/one"_json_pointer);
+    }
+    catch (const json::parse_error& e)
+    {
+        std::cout << e.what() << '\n';
+    }
 }

docs/mkdocs/docs/examples/contains__json_pointer.output

@@ -5,5 +5,3 @@ true
 false
 false
 false
-false
-false

docs/mkdocs/docs/examples/parse_error.cpp

@@ -8,7 +8,7 @@ int main()
     try
     {
         // parsing input with a syntax error
-        json j = json::parse("[1,2,3,]");
+        json::parse("[1,2,3,]");
     }
     catch (const json::parse_error& e)
     {

include/nlohmann/detail/json_pointer.hpp

@@ -473,15 +473,19 @@ class json_pointer
             // convert null values to arrays or objects before continuing
             if (ptr->is_null())
             {
-                // check if the reference token is a number
-                const bool nums =
-                    std::all_of(reference_token.begin(), reference_token.end(),
-                                [](const unsigned char x)
+                // check if the reference token is a valid array index, that is
+                // a nonempty sequence of digits without a leading '0'
+                // (cf. RFC 6901, Sect. 4); tokens that could never be a valid
+                // array index (such as "01" or "") are treated as object keys
+                const bool nums = !reference_token.empty()
+                                  && (reference_token.size() == 1 || reference_token[0] != '0')
+                                  && std::all_of(reference_token.begin(), reference_token.end(),
+                                                 [](const unsigned char x)
                 {
                     return std::isdigit(x);
                 });
 
-                // change value to an array for numbers or "-" or to object otherwise
+                // change value to an array for array indices or "-" or to object otherwise
                 *ptr = (nums || reference_token == "-")
                        ? detail::value_t::array
                        : detail::value_t::object;

single_include/nlohmann/json.hpp

@@ -20391,15 +20391,19 @@ class json_pointer
             // convert null values to arrays or objects before continuing
             if (ptr->is_null())
             {
-                // check if the reference token is a number
-                const bool nums =
-                    std::all_of(reference_token.begin(), reference_token.end(),
-                                [](const unsigned char x)
+                // check if the reference token is a valid array index, that is
+                // a nonempty sequence of digits without a leading '0'
+                // (cf. RFC 6901, Sect. 4); tokens that could never be a valid
+                // array index (such as "01" or "") are treated as object keys
+                const bool nums = !reference_token.empty()
+                                  && (reference_token.size() == 1 || reference_token[0] != '0')
+                                  && std::all_of(reference_token.begin(), reference_token.end(),
+                                                 [](const unsigned char x)
                 {
                     return std::isdigit(x);
                 });
 
-                // change value to an array for numbers or "-" or to object otherwise
+                // change value to an array for array indices or "-" or to object otherwise
                 *ptr = (nums || reference_token == "-")
                        ? detail::value_t::array
                        : detail::value_t::object;

tests/CMakeLists.txt

@@ -138,8 +138,7 @@ json_test_set_test_options(test-disabled_exceptions
 # only the #972 regression test needs thirdparty/fifo_map on its include path
 json_test_set_test_options(test-regression1 LINK_LIBRARIES fifo_map_include)
 
-# Regression test for GCC's false -Warray-bounds error with JSON_DIAGNOSTICS (#5742, fixed in #5585). It only
-# showed up when optimizing, so build this test with -O3 and the warning as an error.
+# GCC's false -Warray-bounds error with JSON_DIAGNOSTICS only shows up when optimizing (#5742).
 # -O3 makes the optimizer-driven warnings of the ci_test_gcc flag set (-Winline,
 # -Wsuggest-attribute=...) fire on the library's inline functions; they are not
 # what this test checks, so turn them off for it.

tests/src/unit-bson.cpp

@@ -46,7 +46,9 @@ class huge_binary_t : public std::vector<std::uint8_t>
     }
 };
 
-using huge_binary_json = nlohmann::json::with_binary_t<huge_binary_t>;
+using huge_binary_json = nlohmann::basic_json <
+                         std::map, std::vector, std::string, bool, std::int64_t, std::uint64_t,
+                         double, std::allocator, nlohmann::adl_serializer, huge_binary_t, void >;
 
 // a string type that can be made to report a size beyond INT32_MAX without
 // allocating that much memory, so BSON length overflow can be tested for
@@ -94,7 +96,9 @@ class huge_string_t : public std::string
     bool pretend_huge = false;
 };
 
-using huge_string_json = nlohmann::json::with_string_t<huge_string_t>;
+using huge_string_json = nlohmann::basic_json <
+                         std::map, std::vector, huge_string_t, bool, std::int64_t, std::uint64_t,
+                         double, std::allocator, nlohmann::adl_serializer, std::vector<std::uint8_t>, void >;
 } // namespace
 
 TEST_CASE("BSON")

tests/src/unit-custom-base-class.cpp

@@ -400,7 +400,20 @@ class base_class_with_hidden_members
     std::size_t m_size = 42;
 };
 
-using json_with_hidden_base_members = nlohmann::json::with_base_class_t<base_class_with_hidden_members>;
+using json_with_hidden_base_members =
+    nlohmann::basic_json <
+    std::map,
+    std::vector,
+    std::string,
+    bool,
+    std::int64_t,
+    std::uint64_t,
+    double,
+    std::allocator,
+    nlohmann::adl_serializer,
+    std::vector<std::uint8_t>,
+    base_class_with_hidden_members
+    >;
 
 TEST_CASE("JSON Node as_base_class")
 {
@@ -446,7 +459,19 @@ struct const_member_base
     const int id = 7; // NOLINT(misc-non-private-member-variables-in-classes)
 };
 
-using json_with_const_base = nlohmann::json::with_base_class_t<const_member_base>;
+using json_with_const_base = nlohmann::basic_json <
+                             std::map,
+                             std::vector,
+                             std::string,
+                             bool,
+                             std::int64_t,
+                             std::uint64_t,
+                             double,
+                             std::allocator,
+                             nlohmann::adl_serializer,
+                             std::vector<std::uint8_t>,
+                             const_member_base
+                             >;
 
 // build an array nested @a depth levels deep, with the innermost value 1;
 // every level is constructed (never assigned), since const_member_base does

tests/src/unit-custom-binary-type.cpp

@@ -26,11 +26,15 @@ namespace
 
 // a BinaryType whose value type is signed: the elements must still be
 // processed as the numbers 0..255
-using char_binary_json = nlohmann::json::with_binary_t<std::vector<char>>;
+using char_binary_json = nlohmann::basic_json <
+                         std::map, std::vector, std::string, bool, std::int64_t, std::uint64_t,
+                         double, std::allocator, nlohmann::adl_serializer, std::vector<char>, void >;
 
 #ifdef JSON_HAS_CPP_17
     // a BinaryType whose value type is not an integer type at all
-    using byte_binary_json = nlohmann::json::with_binary_t<std::vector<std::byte>>;
+    using byte_binary_json = nlohmann::basic_json <
+    std::map, std::vector, std::string, bool, std::int64_t, std::uint64_t,
+    double, std::allocator, nlohmann::adl_serializer, std::vector<std::byte>, void >;
 #endif
 
 } // namespace

tests/src/unit-json_pointer.cpp

@@ -466,6 +466,72 @@ TEST_CASE("JSON pointers")
         }
     }
 
+    SECTION("creating intermediate levels")
+    {
+        SECTION("tokens that are valid array indices create arrays")
+        {
+            json j;
+            j["/0"_json_pointer] = 1;
+            CHECK(j == json({1}));
+
+            json j2;
+            j2["/2"_json_pointer] = 1;
+            CHECK(j2 == json({nullptr, nullptr, 1}));
+
+            json j3;
+            j3["/-"_json_pointer] = 1;
+            CHECK(j3 == json({1}));
+
+            json j4;
+            j4["/foo/0/0"_json_pointer] = 1;
+            CHECK(j4 == json({{"foo", {{1}}}}));
+        }
+
+        SECTION("tokens that are no valid array indices create objects")
+        {
+            json j;
+            j["/one"_json_pointer] = 1;
+            CHECK(j == json({{"one", 1}}));
+
+            // leading '0' can never be a valid array index (RFC 6901, Sect. 4)
+            json j2;
+            j2["/01"_json_pointer] = 1;
+            CHECK(j2 == json({{"01", 1}}));
+
+            // the empty token is a valid object key, but no valid array index
+            json j3;
+            j3["/"_json_pointer] = 1;
+            CHECK(j3 == json({{"", 1}}));
+        }
+
+        SECTION("creating a level yields the same result as reusing it (#5357)")
+        {
+            json j;
+            j["/a/b/01/d"_json_pointer] = "value";
+
+            json j_init = json::object();
+            j_init["/a/b"_json_pointer] = json::object();
+            j_init["/a/b/01/d"_json_pointer] = "value";
+
+            const json expected = json::parse(R"({"a":{"b":{"01":{"d":"value"}}}})");
+            CHECK(j == expected);
+            CHECK(j_init == expected);
+
+            // unflatten uses the same key
+            const json flat = {{"/a/b/01/d", "value"}};
+            CHECK(flat.unflatten() == expected);
+        }
+
+        SECTION("existing arrays still reject invalid indices")
+        {
+            json j = {1, 2, 3};
+            CHECK_THROWS_WITH_AS(j["/01"_json_pointer],
+                                 "[json.exception.parse_error.106] parse error: array index '01' must not begin with '0'", json::parse_error&);
+            CHECK_THROWS_WITH_AS(j.at("/01"_json_pointer),
+                                 "[json.exception.parse_error.106] parse error: array index '01' must not begin with '0'", json::parse_error&);
+        }
+    }
+
     SECTION("flatten")
     {
         json j =

tests/src/unit-msgpack.cpp

@@ -2201,7 +2201,10 @@ struct huge_array : std::vector<T, A>
     }
 };
 
-using huge_array_json = nlohmann::json::with_array_t<huge_array>;
+using huge_array_json = nlohmann::basic_json <
+                        std::map, huge_array, std::string, bool, std::int64_t, std::uint64_t,
+                        double, std::allocator, nlohmann::adl_serializer,
+                        std::vector<std::uint8_t>, void >;
 
 TEST_CASE("MessagePack Size above uint32 for array")
 {
@@ -2246,7 +2249,18 @@ template<typename K, typename V,
     }
 };
 
-using huge_object_json = nlohmann::json::with_object_t<huge_map>;
+using huge_object_json = nlohmann::basic_json <
+                         huge_map,
+                         std::vector,
+                         std::string,
+                         bool,
+                         std::int64_t,
+                         std::uint64_t,
+                         double,
+                         std::allocator,
+                         nlohmann::adl_serializer,
+                         std::vector<std::uint8_t>,
+                         void >;
 
 TEST_CASE("MessagePack Size above uint32 for object")
 {
@@ -2281,7 +2295,18 @@ struct huge_string : std::string
     }
 };
 
-using huge_string_json = nlohmann::json::with_string_t<huge_string>;
+using huge_string_json = nlohmann::basic_json <
+                         std::map,
+                         std::vector,
+                         huge_string,
+                         bool,
+                         std::int64_t,
+                         std::uint64_t,
+                         double,
+                         std::allocator,
+                         nlohmann::adl_serializer,
+                         std::vector<std::uint8_t>,
+                         void >;
 
 TEST_CASE("MessagePack Size above uint32 for string")
 {
@@ -2304,7 +2329,18 @@ struct huge_binary : std::vector<std::uint8_t>
     }
 };
 
-using huge_binary_json = nlohmann::json::with_binary_t<huge_binary>;
+using huge_binary_json = nlohmann::basic_json <
+                         std::map,
+                         std::vector,
+                         std::string,
+                         bool,
+                         std::int64_t,
+                         std::uint64_t,
+                         double,
+                         std::allocator,
+                         nlohmann::adl_serializer,
+                         huge_binary,
+                         void >;
 
 TEST_CASE("MessagePack Size above uint32 for binary")
 {
@@ -2354,10 +2390,14 @@ class beyond_uint32_string_t : public std::string
     }
 };
 
-using beyond_uint32_string_json = nlohmann::json::with_string_t<beyond_uint32_string_t>;
+using beyond_uint32_string_json = nlohmann::basic_json <
+                                  std::map, std::vector, beyond_uint32_string_t, bool, std::int64_t, std::uint64_t,
+                                  double, std::allocator, nlohmann::adl_serializer, std::vector<std::uint8_t>, void >;
 #endif
 
-using beyond_uint32_binary_json = nlohmann::json::with_binary_t<beyond_uint32_binary_t>;
+using beyond_uint32_binary_json = nlohmann::basic_json <
+                                  std::map, std::vector, std::string, bool, std::int64_t, std::uint64_t,
+                                  double, std::allocator, nlohmann::adl_serializer, beyond_uint32_binary_t, void >;
 } // namespace
 
 TEST_CASE("MessagePack lengths beyond UINT32_MAX cannot be serialized")

tests/src/unit-ordered_json2.cpp

@@ -217,7 +217,16 @@ void int_to_string(alt_string& target, std::size_t value)
     target = std::to_string(value).c_str();
 }
 
-using alt_json = nlohmann::json::with_string_t<alt_string>;
+using alt_json = nlohmann::basic_json <
+                 std::map,
+                 std::vector,
+                 alt_string,
+                 bool,
+                 std::int64_t,
+                 std::uint64_t,
+                 double,
+                 std::allocator,
+                 nlohmann::adl_serializer >;
 
 bool operator<(const char* op1, const alt_string& op2) noexcept
 {