Read a std::u8string from a JSON string and store it as one in every configuration (#6075)

* Read a std::u8string from a JSON string and store it as one in every configuration

In C++20, json(std::u8string) was stored as a JSON string only when
std::filesystem::path support was available (the to_json overload sat
inside the JSON_HAS_FILESYSTEM guard), and there was no matching
from_json, so get<std::u8string>() threw type_error.302 ("type must be
array, but is string"). Without std::filesystem, the same expression
still produced an array of code units (the 3.12.0 behavior).

Move the to_json overload out of the filesystem guard so that a
std::u8string is a string in every configuration, build the value
directly from the code units (this also works for a custom string_t
such as the tests' alt_string), and add a from_json overload that reads
a JSON string. The array of numbers written by 3.12.0 is still accepted;
other types throw type_error.302 ("type must be string or array").
std::u8string_view and char8_t arrays are unchanged.

The tests now run in C++20/23 (the file mentions JSON_HAS_CPP_20/23),
also without std::filesystem (test-conversions2_no_filesystem), and with
ordered_json and a custom string type. The new behavior is documented.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Fill the std::u8string with resize and indexing instead of push_back

The push_back loop is not vectorized: reading a 1 kB string took about
2.2 us against 0.39 us with resize and direct element access (clang -O2).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Write the std::u8string from_json template header in the compact form

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Ignore libstdc++ char to char8_t sign change in the clang sanitizer job

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

include/nlohmann/detail/conversions/from_json.hpp

@@ -785,6 +785,37 @@ void from_json(const BasicJsonType& j, std::unordered_map<Key, Value, Hash, KeyE
     from_json_pair_array_to_map(j, m);
 }
 
+#if defined(__cpp_lib_char8_t)
+// a std::u8string is read from a JSON string; the array of code units written by
+// versions before 3.13.0 is still accepted
+template<typename BasicJsonType, typename Tr, typename Allocator,
+         enable_if_t<std::is_same<typename json_type_of_t<BasicJsonType>::string_t::value_type, char>::value, int> = 0>
+inline void from_json(const BasicJsonType& j, std::basic_string<char8_t, Tr, Allocator>& s)
+{
+    if (j.is_string())
+    {
+        const auto& str = string_value(j);
+        const char* const data = str.data();
+        const std::size_t size = str.size();
+        std::basic_string<char8_t, Tr, Allocator> ret(s.get_allocator());
+        ret.resize(size);
+        for (std::size_t i = 0; i < size; ++i)
+        {
+            ret[i] = static_cast<char8_t>(static_cast<unsigned char>(data[i]));
+        }
+        s = std::move(ret);
+    }
+    else if (j.is_array())
+    {
+        from_json_array_impl(j, s, priority_tag<3> {});
+    }
+    else
+    {
+        throw_type_must_be("string or array", j);
+    }
+}
+#endif
+
 #if JSON_HAS_FILESYSTEM || JSON_HAS_EXPERIMENTAL_FILESYSTEM
 
 // Workaround for MSVC 19.51 (and possibly later): in large cpp files, the compiler may fail to resolve with generic has_from_json (issue #4996)

include/nlohmann/detail/conversions/to_json.hpp

@@ -625,16 +625,20 @@ inline void to_json(BasicJsonType& j, const T& t)
     to_json_tuple_impl(j, t, make_index_sequence<std::tuple_size<T>::value> {});
 }
 
-#if JSON_HAS_FILESYSTEM || JSON_HAS_EXPERIMENTAL_FILESYSTEM
+// a std::u8string is a JSON string (it was an array of code units before 3.13.0);
+// defined outside the std::filesystem guard so that the result does not depend
+// on whether std::filesystem::path is supported
 #if defined(__cpp_lib_char8_t)
 template<typename BasicJsonType, typename Tr, typename Allocator>
 inline void to_json(BasicJsonType& j, const std::basic_string<char8_t, Tr, Allocator>& s)
 {
-    using OtherAllocator = typename std::allocator_traits<Allocator>::template rebind_alloc<char>;
-    j = std::basic_string<char, std::char_traits<char>, OtherAllocator>(s.begin(), s.end(), s.get_allocator());
+    using string_t = typename BasicJsonType::string_t;
+    // char may alias any object; the code units are copied unchanged
+    j = string_t(reinterpret_cast<const char*>(s.data()), s.size());
 }
 #endif
 
+#if JSON_HAS_FILESYSTEM || JSON_HAS_EXPERIMENTAL_FILESYSTEM
 // Workaround for MSVC 19.51 (and possibly later): in large cpp files, the compiler may fail to resolve with generic has_to_json (issue #4996)
 template<typename BasicJsonType>
 struct has_to_json<BasicJsonType, std_fs::path, void> : std::true_type {};