Merge branch 'develop' into fix/json_pointer_create_object_5357

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

docs/mkdocs/docs/api/basic_json/operator[].md

@@ -137,10 +137,13 @@ Strong exception safety: if an exception occurs, the original value stays intact
 
         When the JSON pointer traverses intermediate levels that don't exist at all yet (not just a missing
         leaf), each missing level is created as an array or an object depending on whether the corresponding
-        pointer token parses as a non-negative integer: a numeric token creates an array, a non-numeric token
-        creates an object. For example, on an initially `#!json null` value, `/foo/0/0/0` creates nested arrays,
-        while `/foo/one/one/one` creates nested objects. This is not specified by the JSON Pointer RFC; it is
-        this library's own, intentional disambiguation rule. See also [JSON Pointer](../../features/json_pointer.md).
+        pointer token is a valid array index: the token `0`, a sequence of digits that does not begin with `0`,
+        or the token `-` creates an array, and every other token creates an object. For example, on an
+        initially `#!json null` value, `/foo/0/0/0` creates nested arrays, while `/foo/one/one/one` creates
+        nested objects. Tokens such as `01` or the empty token cannot be array indices (cf. RFC 6901, Sect. 4)
+        and therefore create objects, just as they would if the level already existed as an object. This is not
+        specified by the JSON Pointer RFC; it is this library's own, intentional disambiguation rule. See also
+        [JSON Pointer](../../features/json_pointer.md).
 
 !!! warning "Deprecation"
 

docs/mkdocs/docs/community/roadmap.md

@@ -36,26 +36,13 @@ work items are tracked in the [GitHub milestones](https://github.com/nlohmann/js
 ## API stability
 
 Releases follow [semantic versioning](https://semver.org): a minor or patch release of version 3.x does not break code
-that uses the public API, unless that code opts in to a change with a macro as described [below](#version-40). In
-particular, a 3.x release does not:
+that uses the public API. In particular, a 3.x release does not:
 
-- make breaking changes to the signature of a function: the types or order of its existing parameters, its return type,
-  its `noexcept` or `constexpr` specifier, or the const-ness of a member function. New parameters may be added if they
-  have a default value;
-- remove or rename a function or class, or change the template parameters of a public class template;
+- change the signature of a function (its parameter types, return type, number of parameters, or the const-ness of a
+  member function);
+- remove or rename a function or class;
 - change which exceptions a function throws, or the [exception ids](../home/exceptions.md);
-- change access specifiers, or change or remove existing default arguments. New default arguments may be added;
-- change the JSON type that a valid input parses to, or the text that `dump()` produces for a valid value;
-- accept input that was rejected before, or reject input that was accepted before;
-- change the order in which the keys of an object are iterated. The default type sorts keys, and
-  [`ordered_json`](../api/ordered_json.md) keeps insertion order;
-- change when iterators, pointers, or references are invalidated, or the state of a moved-from `basic_json`;
-- add or remove implicit conversions from `basic_json`;
-- change how `to_json` and `from_json` functions are found, or the behavior of
-  [`adl_serializer`](../api/adl_serializer/index.md);
-- add pure virtual functions to the [`json_sax`](../api/json_sax/index.md) interface;
-- remove, rename, renumber, or add enumerators of `value_t`;
-- remove or rename a documented macro, CMake option, CMake target, or header, or change what a documented macro does.
+- change access specifiers or default arguments.
 
 Exceptions to these rules, for instance when fixing a bug requires changing the exception a function throws, are
 documented in the [release notes](../home/releases.md).
@@ -64,14 +51,13 @@ The following are **not** part of the public API and may change in any release,
 
 - The text of exception messages returned by `what()`. Use the [exception id](../home/exceptions.md) to tell errors
   apart.
-- The ABI, including `sizeof(basic_json)` and the memory layout of its values. The
-  [versioned inline namespace](../features/namespace.md) turns mixing versions into a link error.
-- The hash values returned by `std::hash` for `basic_json`. Numbers that compare equal still hash equally.
+- The ABI, including `sizeof(basic_json)` and the memory layout of its values. Recompile your code when you upgrade the
+  library. The [versioned inline namespace](../features/namespace.md) turns mixing versions into a link error.
 - Everything in namespace `nlohmann::detail`, and macros and type traits that are not documented in the
   [API reference](../api/basic_json/index.md).
 
-Breaking changes are only added behind a macro whose default keeps the 3.x behavior. See [Version 4.0](#version-40) and
-the [macro overview](../features/macros.md).
+Changes that would break the public API are only added behind a macro whose default keeps the 3.x behavior, see
+[Version 4.0](#version-40).
 
 ## Version 4.0
 

docs/mkdocs/docs/examples/contains__json_pointer.cpp

@@ -19,9 +19,25 @@ int main()
               << j.contains("/array/1"_json_pointer) << '\n'
               << j.contains("/array/-"_json_pointer) << '\n'
               << j.contains("/array/4"_json_pointer) << '\n'
-              << j.contains("/baz"_json_pointer) << '\n'
-              // an array index with a leading '0' is not found
-              << j.contains("/array/01"_json_pointer) << '\n'
-              // an array index that is not a number is not found
-              << j.contains("/array/one"_json_pointer) << std::endl;
+              << j.contains("/baz"_json_pointer) << std::endl;
+
+    try
+    {
+        // try to use an array index with leading '0'
+        j.contains("/array/01"_json_pointer);
+    }
+    catch (const json::parse_error& e)
+    {
+        std::cout << e.what() << '\n';
+    }
+
+    try
+    {
+        // try to use an array index that is not a number
+        j.contains("/array/one"_json_pointer);
+    }
+    catch (const json::parse_error& e)
+    {
+        std::cout << e.what() << '\n';
+    }
 }

docs/mkdocs/docs/examples/contains__json_pointer.output

@@ -5,5 +5,3 @@ true
 false
 false
 false
-false
-false

docs/mkdocs/docs/examples/parse_error.cpp

@@ -8,7 +8,7 @@ int main()
     try
     {
         // parsing input with a syntax error
-        json j = json::parse("[1,2,3,]");
+        json::parse("[1,2,3,]");
     }
     catch (const json::parse_error& e)
     {

include/nlohmann/detail/json_pointer.hpp

@@ -473,15 +473,19 @@ class json_pointer
             // convert null values to arrays or objects before continuing
             if (ptr->is_null())
             {
-                // check if the reference token is a number
-                const bool nums =
-                    std::all_of(reference_token.begin(), reference_token.end(),
-                                [](const unsigned char x)
+                // check if the reference token is a valid array index, that is
+                // a nonempty sequence of digits without a leading '0'
+                // (cf. RFC 6901, Sect. 4); tokens that could never be a valid
+                // array index (such as "01" or "") are treated as object keys
+                const bool nums = !reference_token.empty()
+                                  && (reference_token.size() == 1 || reference_token[0] != '0')
+                                  && std::all_of(reference_token.begin(), reference_token.end(),
+                                                 [](const unsigned char x)
                 {
                     return std::isdigit(x);
                 });
 
-                // change value to an array for numbers or "-" or to object otherwise
+                // change value to an array for array indices or "-" or to object otherwise
                 *ptr = (nums || reference_token == "-")
                        ? detail::value_t::array
                        : detail::value_t::object;

include/nlohmann/detail/output/binary_writer.hpp

@@ -85,12 +85,6 @@ template<typename BasicJsonType, typename CharType, typename OutputSinkType = ou
 class binary_writer
 {
     using string_t = typename BasicJsonType::string_t;
-
-    /// an object key as string_t: a reference when object_t::key_type already is
-    /// string_t, otherwise a converted copy that outlives sanitize_utf8_for_write's result
-    using object_key_string_t = typename std::conditional <
-                                std::is_same<typename BasicJsonType::object_t::key_type, string_t>::value,
-                                const string_t&, string_t >::type;
     using binary_t = typename BasicJsonType::binary_t;
     using number_float_t = typename BasicJsonType::number_float_t;
 
@@ -825,10 +819,8 @@ class binary_writer
 
                 for (const auto& el : *j.m_data.m_value.object)
                 {
-                    // a converted key must outlive the reference returned by sanitize_utf8_for_write
-                    const object_key_string_t key_string = el.first;
                     string_t storage;
-                    const string_t& key = sanitize_utf8_for_write(key_string, j, storage);
+                    const string_t& key = sanitize_utf8_for_write(el.first, j, storage);
                     write_number_with_ubjson_prefix(key.size(), true, use_bjdata);
                     oa.write_characters(
                           reinterpret_cast<const CharType*>(key.data()),
@@ -1374,10 +1366,8 @@ class binary_writer
                     continue;
                 }
 
-                // a converted key must outlive the reference returned by sanitize_utf8_for_write
-                const object_key_string_t key_string = current.object_it->first;
                 string_t storage;
-                const string_t& key = sanitize_utf8_for_write(key_string, j, storage);
+                const string_t& key = sanitize_utf8_for_write(current.object_it->first, j, storage);
                 write_number_with_ubjson_prefix(key.size(), true, use_bjdata);
                 oa.write_characters(
                       reinterpret_cast<const CharType*>(key.data()),
@@ -2740,11 +2730,6 @@ class binary_writer
     itself in every case but a sanitized `replace`/`ignore` one, so @a
     storage must outlive the returned reference only then.
 
-    @a s must be an lvalue that outlives the returned reference. An object key
-    whose `key_type` is not @ref string_t must therefore first be converted
-    into a named string_t (see @ref object_key_string_t); the deleted overload
-    below enforces this at compile time.
-
     @param[in] s        the string (value or object key) to write
     @param[in] context  the value @a s belongs to (for diagnostics)
     @param[out] storage  backing storage for a sanitized copy
@@ -2774,10 +2759,6 @@ class binary_writer
         }
     }
 
-    /// deleted: anything but a string_t would bind a temporary that dies before the returned reference is used
-    template < typename T, enable_if_t < !std::is_same<T, string_t>::value, int > = 0 >
-    const string_t& sanitize_utf8_for_write(const T& /*s*/, const BasicJsonType& /*context*/, string_t& /*storage*/) const = delete;
-
     /*!
     @brief write an integer in the shortest encoding
 

single_include/nlohmann/json.hpp

@@ -20391,15 +20391,19 @@ class json_pointer
             // convert null values to arrays or objects before continuing
             if (ptr->is_null())
             {
-                // check if the reference token is a number
-                const bool nums =
-                    std::all_of(reference_token.begin(), reference_token.end(),
-                                [](const unsigned char x)
+                // check if the reference token is a valid array index, that is
+                // a nonempty sequence of digits without a leading '0'
+                // (cf. RFC 6901, Sect. 4); tokens that could never be a valid
+                // array index (such as "01" or "") are treated as object keys
+                const bool nums = !reference_token.empty()
+                                  && (reference_token.size() == 1 || reference_token[0] != '0')
+                                  && std::all_of(reference_token.begin(), reference_token.end(),
+                                                 [](const unsigned char x)
                 {
                     return std::isdigit(x);
                 });
 
-                // change value to an array for numbers or "-" or to object otherwise
+                // change value to an array for array indices or "-" or to object otherwise
                 *ptr = (nums || reference_token == "-")
                        ? detail::value_t::array
                        : detail::value_t::object;
@@ -21590,12 +21594,6 @@ template<typename BasicJsonType, typename CharType, typename OutputSinkType = ou
 class binary_writer
 {
     using string_t = typename BasicJsonType::string_t;
-
-    /// an object key as string_t: a reference when object_t::key_type already is
-    /// string_t, otherwise a converted copy that outlives sanitize_utf8_for_write's result
-    using object_key_string_t = typename std::conditional <
-                                std::is_same<typename BasicJsonType::object_t::key_type, string_t>::value,
-                                const string_t&, string_t >::type;
     using binary_t = typename BasicJsonType::binary_t;
     using number_float_t = typename BasicJsonType::number_float_t;
 
@@ -22330,10 +22328,8 @@ class binary_writer
 
                 for (const auto& el : *j.m_data.m_value.object)
                 {
-                    // a converted key must outlive the reference returned by sanitize_utf8_for_write
-                    const object_key_string_t key_string = el.first;
                     string_t storage;
-                    const string_t& key = sanitize_utf8_for_write(key_string, j, storage);
+                    const string_t& key = sanitize_utf8_for_write(el.first, j, storage);
                     write_number_with_ubjson_prefix(key.size(), true, use_bjdata);
                     oa.write_characters(
                           reinterpret_cast<const CharType*>(key.data()),
@@ -22879,10 +22875,8 @@ class binary_writer
                     continue;
                 }
 
-                // a converted key must outlive the reference returned by sanitize_utf8_for_write
-                const object_key_string_t key_string = current.object_it->first;
                 string_t storage;
-                const string_t& key = sanitize_utf8_for_write(key_string, j, storage);
+                const string_t& key = sanitize_utf8_for_write(current.object_it->first, j, storage);
                 write_number_with_ubjson_prefix(key.size(), true, use_bjdata);
                 oa.write_characters(
                       reinterpret_cast<const CharType*>(key.data()),
@@ -24245,11 +24239,6 @@ class binary_writer
     itself in every case but a sanitized `replace`/`ignore` one, so @a
     storage must outlive the returned reference only then.
 
-    @a s must be an lvalue that outlives the returned reference. An object key
-    whose `key_type` is not @ref string_t must therefore first be converted
-    into a named string_t (see @ref object_key_string_t); the deleted overload
-    below enforces this at compile time.
-
     @param[in] s        the string (value or object key) to write
     @param[in] context  the value @a s belongs to (for diagnostics)
     @param[out] storage  backing storage for a sanitized copy
@@ -24279,10 +24268,6 @@ class binary_writer
         }
     }
 
-    /// deleted: anything but a string_t would bind a temporary that dies before the returned reference is used
-    template < typename T, enable_if_t < !std::is_same<T, string_t>::value, int > = 0 >
-    const string_t& sanitize_utf8_for_write(const T& /*s*/, const BasicJsonType& /*context*/, string_t& /*storage*/) const = delete;
-
     /*!
     @brief write an integer in the shortest encoding
 

tests/CMakeLists.txt

@@ -138,8 +138,7 @@ json_test_set_test_options(test-disabled_exceptions
 # only the #972 regression test needs thirdparty/fifo_map on its include path
 json_test_set_test_options(test-regression1 LINK_LIBRARIES fifo_map_include)
 
-# Regression test for GCC's false -Warray-bounds error with JSON_DIAGNOSTICS (#5742, fixed in #5585). It only
-# showed up when optimizing, so build this test with -O3 and the warning as an error.
+# GCC's false -Warray-bounds error with JSON_DIAGNOSTICS only shows up when optimizing (#5742).
 # -O3 makes the optimizer-driven warnings of the ci_test_gcc flag set (-Winline,
 # -Wsuggest-attribute=...) fire on the library's inline functions; they are not
 # what this test checks, so turn them off for it.

tests/src/unit-binary_utf8_error_handler.cpp

@@ -12,10 +12,7 @@
 #include <nlohmann/json.hpp>
 using nlohmann::json;
 
-#include <map>
-#include <memory>
 #include <string>
-#include <utility>
 #include <vector>
 
 namespace
@@ -58,49 +55,6 @@ std::string dump_and_parse(const std::string& raw, eh error_handler)
     return json::parse(json(raw).dump(-1, ' ', false, error_handler)).get<std::string>();
 }
 
-// an object key type that is not string_t, but converts implicitly to it
-class converting_key
-{
-  public:
-    converting_key(const char* s) : m_value(s) {} // NOLINT(google-explicit-constructor,hicpp-explicit-conversions)
-    converting_key(std::string s) : m_value(std::move(s)) {} // NOLINT(google-explicit-constructor,hicpp-explicit-conversions)
-
-    // the conversion yields a temporary string_t
-    operator std::string() const // NOLINT(google-explicit-constructor,hicpp-explicit-conversions)
-    {
-        return m_value;
-    }
-
-    // read by the exception messages when JSON_DIAGNOSTICS is enabled
-    const char* data() const noexcept
-    {
-        return m_value.data();
-    }
-
-    friend bool operator<(const converting_key& lhs, const converting_key& rhs)
-    {
-        return lhs.m_value < rhs.m_value;
-    }
-
-  private:
-    std::string m_value;
-};
-
-// ObjectType using converting_key; the Key template argument is ignored
-template<typename Key, typename Value, typename Compare, typename Allocator>
-class converting_key_object : public std::map<converting_key, Value, std::less<converting_key>, // NOLINT(modernize-use-transparent-functors)
-    typename std::allocator_traits<Allocator>::template rebind_alloc<std::pair<const converting_key, Value>>>
-{
-    using base_type = std::map<converting_key, Value, std::less<converting_key>, // NOLINT(modernize-use-transparent-functors)
-                               typename std::allocator_traits<Allocator>::template rebind_alloc<std::pair<const converting_key, Value>>>;
-
-                             public:
-                               using base_type::base_type;
-                               using base_type::operator=;
-};
-
-using converting_key_json = nlohmann::basic_json<converting_key_object>;
-
 } // namespace
 
 TEST_CASE("UTF-8 error_handler for the binary readers and writers")
@@ -416,109 +370,3 @@ TEST_CASE("UTF-8 error_handler for the binary readers and writers")
         CHECK(json::from_bson(bson_bytes)["k"].get<std::string>() == ill_formed_cases()[0].bytes);
     }
 }
-
-// The UBJSON and BJData writers bind the (possibly sanitized) key to a const
-// string_t&. If key_type is not string_t but converts to it, the converted
-// temporary must outlive that reference; this was a use-after-scope found by
-// AddressSanitizer. Keys exceed the small string optimization on purpose.
-TEST_CASE("UBJSON and BJData writers with an object_t whose key_type is not string_t")
-{
-    const std::string long_prefix(70, 'k');
-
-    SECTION("well-formed keys, every error_handler")
-    {
-        const std::string key1 = long_prefix + "-first";
-        const std::string key2 = long_prefix + "-second";
-
-        converting_key_json::object_t o;
-        o.emplace(converting_key(key1), 1);
-        o.emplace(converting_key(key2), "value");
-        const converting_key_json v(std::move(o));
-
-        json expected;
-        expected[key1] = 1;
-        expected[key2] = "value";
-
-        const bool combos[3][2] = {{false, false}, {true, false}, {true, true}};
-        for (const auto h : all_handlers())
-        {
-            CAPTURE(static_cast<int>(h))
-            for (const auto& combo : combos)
-            {
-                const bool use_count = combo[0];
-                const bool use_type = combo[1];
-                CAPTURE(use_count)
-                CAPTURE(use_type)
-
-                CHECK(json::from_ubjson(converting_key_json::to_ubjson(v, use_count, use_type, h)) == expected);
-                CHECK(json::from_bjdata(converting_key_json::to_bjdata(v, use_count, use_type, json::bjdata_version_t::draft2, h)) == expected);
-                CHECK(json::from_bjdata(converting_key_json::to_bjdata(v, use_count, use_type, json::bjdata_version_t::draft3, h)) == expected);
-            }
-        }
-    }
-
-    SECTION("ill-formed keys")
-    {
-        for (const auto& c : ill_formed_cases())
-        {
-            CAPTURE(c.name)
-            const std::string key = long_prefix + c.bytes;
-
-            converting_key_json::object_t o;
-            o.emplace(converting_key(key), 1);
-            const converting_key_json v(std::move(o));
-
-            CHECK_THROWS_AS(converting_key_json::to_ubjson(v, false, false, eh::strict), converting_key_json::type_error&);
-            CHECK_THROWS_AS(converting_key_json::to_bjdata(v, false, false, json::bjdata_version_t::draft2, eh::strict), converting_key_json::type_error&);
-
-            for (const auto h :
-                    {
-                        eh::replace, eh::ignore
-                    })
-            {
-                CAPTURE(static_cast<int>(h))
-                const std::string expected = dump_and_parse(key, h);
-
-                CHECK(json::from_ubjson(converting_key_json::to_ubjson(v, false, false, h)).begin().key() == expected);
-                CHECK(json::from_bjdata(converting_key_json::to_bjdata(v, false, false, json::bjdata_version_t::draft2, h)).begin().key() == expected);
-            }
-
-            CHECK(json::from_ubjson(converting_key_json::to_ubjson(v, false, false, eh::keep)).begin().key() == key);
-            CHECK(json::from_bjdata(converting_key_json::to_bjdata(v, false, false, json::bjdata_version_t::draft2, eh::keep)).begin().key() == key);
-        }
-    }
-
-    SECTION("nested deeper than the recursion limit")
-    {
-        // wrap the previous value, innermost first
-        converting_key_json v = 42;
-        json expected = 42;
-        for (int i = 199; i >= 0; --i)
-        {
-            const std::string key = "level-" + std::to_string(i) + "-" + std::string(64, 'x');
-
-            converting_key_json::object_t o;
-            o.emplace(converting_key(key), std::move(v));
-            v = converting_key_json(std::move(o));
-
-            json e;
-            e[key] = std::move(expected);
-            expected = std::move(e);
-        }
-
-        for (const auto h : all_handlers())
-        {
-            CAPTURE(static_cast<int>(h))
-            for (const bool use_count :
-                    {
-                        false, true
-                    })
-            {
-                CAPTURE(use_count)
-
-                CHECK(json::from_ubjson(converting_key_json::to_ubjson(v, use_count, false, h)) == expected);
-                CHECK(json::from_bjdata(converting_key_json::to_bjdata(v, use_count, false, json::bjdata_version_t::draft2, h)) == expected);
-            }
-        }
-    }
-}

tests/src/unit-json_pointer.cpp

@@ -466,6 +466,72 @@ TEST_CASE("JSON pointers")
         }
     }
 
+    SECTION("creating intermediate levels")
+    {
+        SECTION("tokens that are valid array indices create arrays")
+        {
+            json j;
+            j["/0"_json_pointer] = 1;
+            CHECK(j == json({1}));
+
+            json j2;
+            j2["/2"_json_pointer] = 1;
+            CHECK(j2 == json({nullptr, nullptr, 1}));
+
+            json j3;
+            j3["/-"_json_pointer] = 1;
+            CHECK(j3 == json({1}));
+
+            json j4;
+            j4["/foo/0/0"_json_pointer] = 1;
+            CHECK(j4 == json({{"foo", {{1}}}}));
+        }
+
+        SECTION("tokens that are no valid array indices create objects")
+        {
+            json j;
+            j["/one"_json_pointer] = 1;
+            CHECK(j == json({{"one", 1}}));
+
+            // leading '0' can never be a valid array index (RFC 6901, Sect. 4)
+            json j2;
+            j2["/01"_json_pointer] = 1;
+            CHECK(j2 == json({{"01", 1}}));
+
+            // the empty token is a valid object key, but no valid array index
+            json j3;
+            j3["/"_json_pointer] = 1;
+            CHECK(j3 == json({{"", 1}}));
+        }
+
+        SECTION("creating a level yields the same result as reusing it (#5357)")
+        {
+            json j;
+            j["/a/b/01/d"_json_pointer] = "value";
+
+            json j_init = json::object();
+            j_init["/a/b"_json_pointer] = json::object();
+            j_init["/a/b/01/d"_json_pointer] = "value";
+
+            const json expected = json::parse(R"({"a":{"b":{"01":{"d":"value"}}}})");
+            CHECK(j == expected);
+            CHECK(j_init == expected);
+
+            // unflatten uses the same key
+            const json flat = {{"/a/b/01/d", "value"}};
+            CHECK(flat.unflatten() == expected);
+        }
+
+        SECTION("existing arrays still reject invalid indices")
+        {
+            json j = {1, 2, 3};
+            CHECK_THROWS_WITH_AS(j["/01"_json_pointer],
+                                 "[json.exception.parse_error.106] parse error: array index '01' must not begin with '0'", json::parse_error&);
+            CHECK_THROWS_WITH_AS(j.at("/01"_json_pointer),
+                                 "[json.exception.parse_error.106] parse error: array index '01' must not begin with '0'", json::parse_error&);
+        }
+    }
+
     SECTION("flatten")
     {
         json j =