Merge branch 'develop' into claude/3453-breaking-change-flags-c937b4

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

docs/mkdocs/docs/api/macros/json_brace_init_copy_semantics.md

@@ -115,3 +115,4 @@ The default value is `0` (disabled — existing behavior is preserved).
 ## Version history
 
 - Added in version 3.13.0.
+- Planned to become the default (with the macro removed) in version 4.0.0.

docs/mkdocs/docs/community/roadmap.md

@@ -14,7 +14,7 @@ work items are tracked in the [GitHub milestones](https://github.com/nlohmann/js
   opt-in.
 - **Keep the 3.x public API stable.** Releases follow [semantic versioning](https://semver.org). Changes that would
   break existing code are only added behind a feature macro, so users can opt in and test their code before a next
-  major release.
+  major release, see [Version 4.0](#version-40).
 - **Support a broad range of compilers and platforms.** The [CI](quality_assurance.md) keeps testing old and new
   versions of GCC, Clang, MSVC, and other compilers on Linux, macOS, and Windows.
 - **Keep the quality assurance up.** Every change keeps the test coverage at 100%, passes the static and dynamic
@@ -37,7 +37,65 @@ work items are tracked in the [GitHub milestones](https://github.com/nlohmann/js
 
 ## Version 4.0
 
-There is no decision yet on whether or when a version 4.0 with breaking changes will be released. Proposals that need
-a major version, for instance stricter type conversions, are collected in issue
-[#3453](https://github.com/nlohmann/json/issues/3453). Until then, such changes are only added as opt-in behavior
-behind feature macros.
+There is no release date for version 4.0 yet. Proposals that need a major version, for instance stricter type
+conversions, are collected in issue [#3453](https://github.com/nlohmann/json/issues/3453).
+
+!!! note "Not final"
+
+    The plan for version 4.0 described below is not final and may still change: macros may be added to or removed from
+    the list, and planned defaults may be revised. Any such change will be documented on this page.
+
+### Trying out 4.0 today
+
+Version 4.0 will not be developed on a separate branch. Instead, every breaking change is first added to a 3.x release
+behind a macro whose default keeps the 3.x behavior. Version 4.0 then switches the defaults and removes the macros.
+Version 4.0 is therefore the sum of these macros: you can try it on the 3.x release train today by defining each macro
+to its 4.0 value and fixing what no longer compiles or behaves differently. Once your code works with all of them, it
+is ready for version 4.0.
+
+The following macros guard changes that are planned to become the default in version 4.0:
+
+| Macro                                                                                                            | 3.x default | 4.0 behavior                                                                                                                  | CMake option                                                                                                       | Added  |
+|------------------------------------------------------------------------------------------------------------------|-------------|-------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------|--------|
+| [`JSON_USE_IMPLICIT_CONVERSIONS`](../api/macros/json_use_implicit_conversions.md)                                | `1`         | `0`: no implicit conversions from `basic_json` to other types; use [`get`](../api/basic_json/get.md) instead                  | [`JSON_ImplicitConversions`](../integration/cmake.md#json_implicitconversions)                                     | 3.9.0  |
+| [`JSON_USE_GLOBAL_UDLS`](../api/macros/json_use_global_udls.md)                                                  | `1`         | `0`: the string literals `_json` and `_json_pointer` are only available in namespace `nlohmann::literals`                    | [`JSON_GlobalUDLs`](../integration/cmake.md#json_globaludls)                                                       | 3.11.0 |
+| [`JSON_USE_LEGACY_DISCARDED_VALUE_COMPARISON`](../api/macros/json_use_legacy_discarded_value_comparison.md)      | `0`         | removed: the deprecated legacy comparison of discarded values can no longer be enabled                                        | [`JSON_LegacyDiscardedValueComparison`](../integration/cmake.md#json_legacydiscardedvaluecomparison)               | 3.11.0 |
+| [`JSON_BRACE_INIT_COPY_SEMANTICS`](../api/macros/json_brace_init_copy_semantics.md)                              | `0`         | `1`: single-element brace initialization such as `#!cpp json j{obj};` copies the element instead of creating an array         | –                                                                                                                  | 3.13.0 |
+| [`JSON_PRECISE_STREAM_POSITION`](../api/macros/json_precise_stream_position.md)                                  | `0`         | `1`: reading from a stream does not consume the character after a number                                                      | –                                                                                                                  | 3.13.0 |
+| [`JSON_STRICT_NUL_HANDLING`](../api/macros/json_strict_nul_handling.md)                                          | `0`         | `1`: a NUL byte in the input is a parse error instead of the end of input                                                     | [`JSON_StrictNulHandling`](../integration/cmake.md#json_strictnulhandling)                                         | 3.13.0 |
+
+For example, the following makes a 3.x release behave like version 4.0 with respect to these changes:
+
+```cpp
+#define JSON_USE_IMPLICIT_CONVERSIONS 0
+#define JSON_USE_GLOBAL_UDLS 0
+#define JSON_USE_LEGACY_DISCARDED_VALUE_COMPARISON 0
+#define JSON_BRACE_INIT_COPY_SEMANTICS 1
+#define JSON_PRECISE_STREAM_POSITION 1
+#define JSON_STRICT_NUL_HANDLING 1
+#include <nlohmann/json.hpp>
+```
+
+The macros must be defined before the library header is included; setting them once in the build system is the easiest
+way to achieve this.
+
+### Removal of deprecated functions
+
+Version 4.0 will remove all deprecated functions. Compiling with deprecation warnings enabled shows which of them your
+code still uses. The [migration guide](../integration/migration_guide.md#replace-deprecated-functions) shows how to
+replace each of them.
+
+| Deprecated                                                                                                                                                                                                                                         | Since  | Migration                                                                        |
+|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------|----------------------------------------------------------------------------------|
+| `#!cpp operator<<(basic_json&, std::istream&)`                                                                                                                                                                                                     | 3.0.0  | [Parsing](../integration/migration_guide.md#parsing)                             |
+| `#!cpp operator>>(const basic_json&, std::ostream&)`                                                                                                                                                                                               | 3.0.0  | [Miscellaneous functions](../integration/migration_guide.md#miscellaneous-functions) |
+| `iterator_wrapper`                                                                                                                                                                                                                                 | 3.1.0  | [Miscellaneous functions](../integration/migration_guide.md#miscellaneous-functions) |
+| [`parse`](../api/basic_json/parse.md), [`accept`](../api/basic_json/accept.md), and [`sax_parse`](../api/basic_json/sax_parse.md) with an initializer list `{ptr, len}` or `{first, last}`                                                         | 3.8.0  | [Parsing](../integration/migration_guide.md#parsing)                             |
+| [`from_bson`](../api/basic_json/from_bson.md), [`from_cbor`](../api/basic_json/from_cbor.md), [`from_msgpack`](../api/basic_json/from_msgpack.md), and [`from_ubjson`](../api/basic_json/from_ubjson.md) with `(ptr, len)` or an initializer list | 3.8.0  | [Parsing](../integration/migration_guide.md#parsing)                             |
+| [`json_pointer::operator string_t`](../api/json_pointer/operator_string_t.md)                                                                                                                                                                      | 3.11.0 | [JSON Pointers](../integration/migration_guide.md#json-pointers)                 |
+| [`json_pointer`](../api/json_pointer/index.md) with a `basic_json` type as template argument, and the overloads of `value`, `contains`, `operator[]`, and `at` accepting such a pointer                                                           | 3.11.0 | [JSON Pointers](../integration/migration_guide.md#json-pointers)                 |
+| Comparing a [`json_pointer`](../api/json_pointer/index.md) with a string via [`operator==`](../api/json_pointer/operator_eq.md) or [`operator!=`](../api/json_pointer/operator_ne.md)                                                              | 3.11.2 | [JSON Pointers](../integration/migration_guide.md#json-pointers)                 |
+
+The deprecated legacy comparison of discarded values is controlled by a macro and therefore listed in the table above.
+
+New breaking changes will follow the same path: they are added to these tables when they land in a 3.x release.

docs/mkdocs/docs/integration/migration_guide.md

@@ -1,10 +1,13 @@
 # Migration Guide
 
 This page collects some guidelines on how to future-proof your code for future versions of this library.
+The [roadmap](../community/roadmap.md#version-40) lists what will change in version 4.0, including the macros that let
+you try its behavior with a 3.x release; this page describes how to adjust your code.
 
 ## Replace deprecated functions
 
-The following functions have been deprecated and will be removed in the next major version (i.e., 4.0.0). All
+The following functions have been deprecated and will be removed in the next major version (i.e., 4.0.0), see the
+[roadmap](../community/roadmap.md#removal-of-deprecated-functions) for an overview. All
 deprecations are annotated with
 [`HEDLEY_DEPRECATED_FOR`](https://nemequ.github.io/hedley/api-reference.html#HEDLEY_DEPRECATED_FOR) to report which
 function to use instead.
@@ -34,8 +37,9 @@ function to use instead.
   [`accept`](../api/basic_json/accept.md), [`sax_parse`](../api/basic_json/sax_parse.md),
   [`from_cbor`](../api/basic_json/from_cbor.md), [`from_msgpack`](../api/basic_json/from_msgpack.md),
   [`from_ubjson`](../api/basic_json/from_ubjson.md), and [`from_bson`](../api/basic_json/from_bson.md) via initializer
-  lists is deprecated since 3.8.0. Instead, pass two iterators; for instance, call `from_cbor(ptr, ptr+len)` instead of
-  `from_cbor({ptr, len})`.
+  lists is deprecated since 3.8.0. The same holds for passing a pointer and a length as two arguments to the `from_*`
+  functions. Instead, pass two iterators; for instance, call `from_cbor(ptr, ptr+len)` instead of
+  `from_cbor({ptr, len})` or `from_cbor(ptr, len)`.
 
     === "Deprecated"
   

include/nlohmann/detail/input/input_adapters.hpp

@@ -445,10 +445,8 @@ struct wide_string_input_helper<BaseInputAdapter, 4>
         }
         else
         {
-            // get the current character; converted to an unsigned type so that
-            // a negative unit (wint_t is signed on some platforms) is not
-            // mistaken for an ASCII character or for EOF
-            const auto wc = static_cast<std::uint32_t>(input.get_character());
+            // get the current character
+            const auto wc = input.get_character();
 
             // UTF-32 to UTF-8 encoding
             if (wc < 0x80)
@@ -543,11 +541,9 @@ struct wide_string_input_helper<BaseInputAdapter, 2>
                 bool valid_pair = false;
                 if (wc <= 0xDBFF && JSON_HEDLEY_UNLIKELY(!input.empty()))
                 {
-                    // only consume the next unit if it completes the pair
-                    const auto wc2 = static_cast<unsigned int>(*input.current);
+                    const auto wc2 = static_cast<unsigned int>(input.get_character());
                     if (0xDC00 <= wc2 && wc2 <= 0xDFFF)
                     {
-                        input.get_character();
                         const auto charcode = 0x10000u + (((static_cast<unsigned int>(wc) & 0x3FFu) << 10u) | (wc2 & 0x3FFu));
                         utf8_bytes[0] = static_cast<std::char_traits<char>::int_type>(0xF0u | (charcode >> 18u));
                         utf8_bytes[1] = static_cast<std::char_traits<char>::int_type>(0x80u | ((charcode >> 12u) & 0x3Fu));
@@ -560,8 +556,7 @@ struct wide_string_input_helper<BaseInputAdapter, 2>
 
                 if (!valid_pair)
                 {
-                    // emit a byte that is never valid UTF-8 (see the UTF-32 case)
-                    utf8_bytes[0] = 0xFF;
+                    utf8_bytes[0] = static_cast<std::char_traits<char>::int_type>(wc);
                     utf8_bytes_filled = 1;
                 }
             }

single_include/nlohmann/json.hpp

@@ -7989,10 +7989,8 @@ struct wide_string_input_helper<BaseInputAdapter, 4>
         }
         else
         {
-            // get the current character; converted to an unsigned type so that
-            // a negative unit (wint_t is signed on some platforms) is not
-            // mistaken for an ASCII character or for EOF
-            const auto wc = static_cast<std::uint32_t>(input.get_character());
+            // get the current character
+            const auto wc = input.get_character();
 
             // UTF-32 to UTF-8 encoding
             if (wc < 0x80)
@@ -8087,11 +8085,9 @@ struct wide_string_input_helper<BaseInputAdapter, 2>
                 bool valid_pair = false;
                 if (wc <= 0xDBFF && JSON_HEDLEY_UNLIKELY(!input.empty()))
                 {
-                    // only consume the next unit if it completes the pair
-                    const auto wc2 = static_cast<unsigned int>(*input.current);
+                    const auto wc2 = static_cast<unsigned int>(input.get_character());
                     if (0xDC00 <= wc2 && wc2 <= 0xDFFF)
                     {
-                        input.get_character();
                         const auto charcode = 0x10000u + (((static_cast<unsigned int>(wc) & 0x3FFu) << 10u) | (wc2 & 0x3FFu));
                         utf8_bytes[0] = static_cast<std::char_traits<char>::int_type>(0xF0u | (charcode >> 18u));
                         utf8_bytes[1] = static_cast<std::char_traits<char>::int_type>(0x80u | ((charcode >> 12u) & 0x3Fu));
@@ -8104,8 +8100,7 @@ struct wide_string_input_helper<BaseInputAdapter, 2>
 
                 if (!valid_pair)
                 {
-                    // emit a byte that is never valid UTF-8 (see the UTF-32 case)
-                    utf8_bytes[0] = 0xFF;
+                    utf8_bytes[0] = static_cast<std::char_traits<char>::int_type>(wc);
                     utf8_bytes_filled = 1;
                 }
             }

tests/src/unit-wstring.cpp

@@ -8,7 +8,6 @@
 
 #include "doctest_compatibility.h"
 
-#include <cwchar>
 #include <nlohmann/json.hpp>
 using nlohmann::json;
 
@@ -99,15 +98,15 @@ TEST_CASE("wide strings")
             CHECK_THROWS_AS(_ = json::parse(w), json::parse_error&);
 
             // a lone low surrogate cannot start a pair
-            CHECK_THROWS_WITH_AS(_ = json::parse(std::u16string{u'"', 0xDC00, u'"'}), "[json.exception.parse_error.101] parse error at line 1, column 2: syntax error while parsing value - invalid string: ill-formed UTF-8 byte; last read: '\"\xFF'", json::parse_error&);
+            CHECK_THROWS_WITH_AS(_ = json::parse(std::u16string{u'"', 0xDC00, u'"'}), "[json.exception.parse_error.101] parse error at line 1, column 2: syntax error while parsing value - invalid string: ill-formed UTF-8 byte; last read: '\"<U+0000>'", json::parse_error&);
             // a high surrogate followed by a non-low-surrogate unit is invalid
-            CHECK_THROWS_WITH_AS(_ = json::parse(std::u16string{u'"', 0xD800, u'a', u'"'}), "[json.exception.parse_error.101] parse error at line 1, column 2: syntax error while parsing value - invalid string: ill-formed UTF-8 byte; last read: '\"\xFF'", json::parse_error&);
+            CHECK_THROWS_WITH_AS(_ = json::parse(std::u16string{u'"', 0xD800, u'a', u'"'}), "[json.exception.parse_error.101] parse error at line 1, column 2: syntax error while parsing value - invalid string: ill-formed UTF-8 byte; last read: '\"<U+0000>'", json::parse_error&);
             // ... also when the unit is above the low surrogates
-            CHECK_THROWS_WITH_AS(_ = json::parse(std::u16string{u'"', 0xD800, 0xE000, u'"'}), "[json.exception.parse_error.101] parse error at line 1, column 2: syntax error while parsing value - invalid string: ill-formed UTF-8 byte; last read: '\"\xFF'", json::parse_error&);
+            CHECK_THROWS_WITH_AS(_ = json::parse(std::u16string{u'"', 0xD800, 0xE000, u'"'}), "[json.exception.parse_error.101] parse error at line 1, column 2: syntax error while parsing value - invalid string: ill-formed UTF-8 byte; last read: '\"<U+0000>'", json::parse_error&);
             // a lone low surrogate must not swallow the following unit: pairing
             // it with any second unit would produce valid UTF-8, so the error
             // has to report an ill-formed byte at the surrogate's own position
-            CHECK_THROWS_WITH_AS(_ = json::parse(std::u16string{u'"', 0xDC00, u'a', u'"'}), "[json.exception.parse_error.101] parse error at line 1, column 2: syntax error while parsing value - invalid string: ill-formed UTF-8 byte; last read: '\"\xFF'", json::parse_error&);
+            CHECK_THROWS_WITH_AS(_ = json::parse(std::u16string{u'"', 0xDC00, u'a', u'"'}), "[json.exception.parse_error.101] parse error at line 1, column 2: syntax error while parsing value - invalid string: ill-formed UTF-8 byte; last read: '\"<U+0000>'", json::parse_error&);
             // a valid surrogate pair is still decoded (U+1F600)
             CHECK(json::parse(std::u16string{u'"', 0xD83D, 0xDE00, u'"'}).get<std::string>() == "\xF0\x9F\x98\x80");
         }
@@ -142,56 +141,5 @@ TEST_CASE("wide strings")
             CHECK_THROWS_WITH_AS(_ = json::parse(std::u32string{U'"', static_cast<char32_t>(0xFFFFFFFF), U'"'}), "[json.exception.parse_error.101] parse error at line 1, column 2: syntax error while parsing value - invalid string: ill-formed UTF-8 byte; last read: '\"\xFF'", json::parse_error&);
         }
     }
-
-    SECTION("malformed wide-string input outside strings (#5645)")
-    {
-        json _;
-
-        // a lone low surrogate inside a literal must not be truncated to its
-        // low byte and mistaken for the letter the literal expects next
-        // (0xDC72 truncates to 'r', which is what "true" expects after 't')
-        CHECK_THROWS_WITH_AS(_ = json::parse(std::u16string{u't', static_cast<char16_t>(0xDC72), u'u', u'e'}),
-                             "[json.exception.parse_error.101] parse error at line 1, column 2: syntax error while parsing value - invalid literal; last read: 't\xFF'", json::parse_error&);
-
-        // ... also when the lone surrogate is the last unit of the input
-        CHECK_THROWS_WITH_AS(_ = json::parse(std::u16string{u'f', u'a', u'l', u's', static_cast<char16_t>(0xDD65)}),
-                             "[json.exception.parse_error.101] parse error at line 1, column 5: syntax error while parsing value - invalid literal; last read: 'fals\xFF'", json::parse_error&);
-
-        // a high surrogate followed by a unit that is not its low surrogate
-        // must not silently swallow that unit
-        CHECK_THROWS_WITH_AS(_ = json::parse(std::u16string{u't', static_cast<char16_t>(0xD872), u'X', u'u', u'e'}),
-                             "[json.exception.parse_error.101] parse error at line 1, column 2: syntax error while parsing value - invalid literal; last read: 't\xFF'", json::parse_error&);
-
-        // ... in particular, if the swallowed unit is the newline that ends a
-        // // comment, the comment must not extend over the following line
-        CHECK(json::parse(std::u16string{u'[', u'1', u' ', u'/', u'/', static_cast<char16_t>(0xD800), u'\n',
-                                         u',', u'2', u' ', u'/', u'/', u'\n', u']'},
-                          nullptr, true, /*ignore_comments*/true) == json::parse("[1,2]"));
-        CHECK(json::accept(std::u16string{u'[', u'1', u' ', u'/', u'/', static_cast<char16_t>(0xD800), u'\n',
-                                          u',', u'2', u' ', u'/', u'/', u'\n', u']'}, /*ignore_comments*/true));
-
-        // cases 5 and 6 use a 32-bit wchar_t (Linux, macOS, the BSDs) to reach
-        // the UTF-32 helper tested above via u32string; the 16-bit wchar_t of
-        // Windows goes through the UTF-16 helper instead, already covered by
-        // the u16string cases above
-#if WCHAR_MAX > 0xFFFFu
-        // a negative wchar_t must not be mistaken for
-        // char_traits<char>::eof() and silently end the input, letting
-        // trailing garbage pass the strict end-of-input check (only observable
-        // where wint_t is signed, e.g. macOS/the BSDs; on Linux wint_t is
-        // unsigned and this was already handled by #5348)
-        std::wstring w = L"[1]";
-        w.push_back(static_cast<wchar_t>(-1));
-        w += L"garbage";
-        CHECK(!json::accept(w));
-        CHECK_THROWS_WITH_AS(_ = json::parse(w),
-                             "[json.exception.parse_error.101] parse error at line 1, column 4: syntax error while parsing value - invalid literal; last read: '1]\xFF'; expected end of input", json::parse_error&);
-
-        // other negative wchar_t units must not be truncated to their low
-        // byte (0xFFFFFF72 truncates to 'r', as in the u16string case above)
-        CHECK_THROWS_WITH_AS(_ = json::parse(std::wstring{L't', static_cast<wchar_t>(0xFFFFFF72), L'u', L'e'}),
-                             "[json.exception.parse_error.101] parse error at line 1, column 2: syntax error while parsing value - invalid literal; last read: 't\xFF'", json::parse_error&);
-#endif
-    }
 }
 #endif