Merge remote-tracking branch 'origin/develop' into claude/issue-5387-duplicate-check-bd7853

Resolves conflicts from develop's binary-reader and serializer rewrites,
and the unit-regression2.cpp/unit-regression3.cpp split (#5511): took
develop's version of the split test files wholesale (content-equivalent,
confirmed by comparing SECTION/TEST_CASE names), and combined the
ci_cmake_options CI matrix (ci_test_simdutf + ci_test_no_thread_local).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

docs/mkdocs/docs/api/macros/index.md

@@ -25,6 +25,7 @@ header. See also the [macro overview page](../../features/macros.md).
 - [**JSON_NO_THREAD_LOCAL**](json_no_thread_local.md) - switch off the use of `thread_local` storage
 - [**JSON_SKIP_UNSUPPORTED_COMPILER_CHECK**](json_skip_unsupported_compiler_check.md) - do not warn about unsupported compilers
 - [**JSON_USE_GLOBAL_UDLS**](json_use_global_udls.md) - place user-defined string literals (UDLs) into the global namespace
+- [**JSON_USE_SIMDUTF**](json_use_simdutf.md) - use the simdutf library to accelerate UTF-8 validation
 
 ## Library version
 

docs/mkdocs/docs/api/macros/json_use_simdutf.md

@@ -0,0 +1,71 @@
+# JSON_USE_SIMDUTF
+
+```cpp
+#define JSON_USE_SIMDUTF
+```
+
+When defined, the parser validates the UTF-8 content of JSON strings that come from a **contiguous byte input**
+(`std::string`, `std::vector<char>`/`<std::uint8_t>`, string literals, `const char*` ranges, …) using the
+[simdutf](https://github.com/simdutf/simdutf) library instead of the built-in scalar validator. On text with many
+non-ASCII characters (e.g. CJK or emoji) this can validate several times faster.
+
+This is an **opt-in external dependency**. The library itself remains header-only and its behavior is unchanged: the
+same input is accepted or rejected either way, and every parse error is reported at the same position with the same
+message (simdutf is only used to fast-path *valid* runs; anything it flags falls back to the scalar path so the exact
+diagnostic is preserved). Streaming inputs (files, `std::istream`, wide strings, user-defined adapters) always use the
+scalar path.
+
+When `JSON_USE_SIMDUTF` is defined you must make the `simdutf.h` header available on the include path and link the
+simdutf library. When it is not defined, no simdutf header is included and there is no dependency.
+
+!!! note "Requires C++17"
+
+    simdutf requires C++17 and its header rejects older standards with an `#!cpp #error`. The backend is therefore only
+    compiled in from C++17 on. In C++11 and C++14 the macro has no effect and the scalar validator is used, which
+    accepts and rejects exactly the same input -- only throughput differs. Setting the macro project-wide is therefore
+    safe even when some translation units are built with an older standard.
+
+!!! warning "Define consistently"
+
+    The macro selects between two definitions of the same inline validation function. It must therefore be defined
+    identically for **every** translation unit that includes the library; mixing translation units that define it with
+    ones that do not is an ODR violation. Prefer setting it as a compile definition on the target rather than with
+    `#!cpp #define` in individual source files.
+
+## Default definition
+
+By default, `#!cpp JSON_USE_SIMDUTF` is not defined and the portable C++11 scalar validator is used.
+
+```cpp
+#undef JSON_USE_SIMDUTF
+```
+
+## Examples
+
+??? example
+
+    The code below enables the simdutf backend for UTF-8 validation.
+
+    ```cpp
+    #define JSON_USE_SIMDUTF 1
+    #include <nlohmann/json.hpp>
+
+    ...
+    ```
+
+    The project must also link against simdutf, e.g. with CMake:
+
+    ```cmake
+    target_compile_definitions(your_target PRIVATE JSON_USE_SIMDUTF)
+    target_link_libraries(your_target PRIVATE simdutf::simdutf)
+    ```
+
+!!! hint "Testing this configuration"
+
+    The unit tests can be built against the simdutf backend with the CMake option `JSON_TestSimdutf` (`OFF` by
+    default), which fetches simdutf and defines `JSON_USE_SIMDUTF` for every test target. The `ci_test_simdutf` target
+    runs the whole test suite in that configuration.
+
+## Version history
+
+- Added in version 3.13.0.

docs/mkdocs/docs/features/binary_formats/ubjson.md

@@ -69,6 +69,13 @@ The library uses the following mapping from JSON values types to UBJSON types ac
     Note that `use_size = true` alone may result in larger representations - the benefit of this parameter is that the
     receiving side is immediately informed on the number of elements of the container.
 
+    An array whose type marker is `Z` (null), `T` (true) or `F` (false) stores no payload at all, because the marker
+    already is the value. Its declared count is therefore the only thing that decides how much memory the receiving side
+    allocates, and a handful of bytes can describe billions of elements. `from_ubjson` rejects such an array with
+    [`out_of_range.408`](../../home/exceptions.md#jsonexceptionout_of_range408) when the count exceeds 1,048,576
+    (`1 << 20`), and `to_ubjson` writes longer arrays of these types without the annotation, so any value it produces
+    can be read back.
+
 !!! info "Binary values"
 
     If the JSON data contains the binary type, the value stored is a list of integers, as suggested by the UBJSON

docs/mkdocs/docs/features/macros.md

@@ -144,6 +144,14 @@ behavior is deprecated and switched off (`0`) by default.
 
 See [full documentation of `JSON_USE_LEGACY_DISCARDED_VALUE_COMPARISON`](../api/macros/json_use_legacy_discarded_value_comparison.md).
 
+## `JSON_USE_SIMDUTF`
+
+When defined, UTF-8 validation of JSON strings read from contiguous byte input is delegated to the
+[simdutf](https://github.com/simdutf/simdutf) library instead of the built-in scalar validator. This is an opt-in
+external dependency and is not defined by default.
+
+See [full documentation of `JSON_USE_SIMDUTF`](../api/macros/json_use_simdutf.md).
+
 ## `NLOHMANN_DEFINE_TYPE_*(...)`, `NLOHMANN_DEFINE_DERIVED_TYPE_*(...)`
 
 The library defines 12 macros to simplify the serialization/deserialization of types. See the page on

docs/mkdocs/docs/home/exceptions.md

@@ -868,6 +868,12 @@ The size of an array or object in a [binary format](../features/binary_formats/i
 the size following `#` for [UBJSON](../features/binary_formats/ubjson.md)/[BJData](../features/binary_formats/bjdata.md),
 or the encoded length for [CBOR](../features/binary_formats/cbor.md).
 
+The exception is also thrown for a [UBJSON](../features/binary_formats/ubjson.md) array of a type that is encoded by its
+marker alone (`Z`, `T` or `F`) whose declared count exceeds 1,048,576 (`1 << 20`). Such an array has no payload, so its
+count alone decides how much memory is allocated, and a handful of bytes would otherwise describe billions of values.
+[`to_ubjson`](../api/basic_json/to_ubjson.md) writes longer arrays of these types without the size and type annotation,
+so any value it produces can still be read back.
+
 !!! failure "Example messages"
 
     ```
@@ -879,6 +885,9 @@ or the encoded length for [CBOR](../features/binary_formats/cbor.md).
     ```
     [json.exception.out_of_range.408] syntax error while parsing CBOR size: excessive map size
     ```
+    ```
+    [json.exception.out_of_range.408] syntax error while parsing UBJSON size: excessive array size
+    ```
 
 ### json.exception.out_of_range.409
 

docs/mkdocs/mkdocs.yml

@@ -297,6 +297,7 @@ nav:
           - 'JSON_USE_GLOBAL_UDLS': api/macros/json_use_global_udls.md
           - 'JSON_USE_IMPLICIT_CONVERSIONS': api/macros/json_use_implicit_conversions.md
           - 'JSON_USE_LEGACY_DISCARDED_VALUE_COMPARISON': api/macros/json_use_legacy_discarded_value_comparison.md
+          - 'JSON_USE_SIMDUTF': api/macros/json_use_simdutf.md
           - 'NLOHMANN_DEFINE_DERIVED_TYPE_INTRUSIVE, NLOHMANN_DEFINE_DERIVED_TYPE_INTRUSIVE_WITH_DEFAULT, NLOHMANN_DEFINE_DERIVED_TYPE_INTRUSIVE_ONLY_SERIALIZE, NLOHMANN_DEFINE_DERIVED_TYPE_NON_INTRUSIVE, NLOHMANN_DEFINE_DERIVED_TYPE_NON_INTRUSIVE_WITH_DEFAULT, NLOHMANN_DEFINE_DERIVED_TYPE_NON_INTRUSIVE_ONLY_SERIALIZE': api/macros/nlohmann_define_derived_type.md
           - 'NLOHMANN_DEFINE_TYPE_INTRUSIVE, NLOHMANN_DEFINE_TYPE_INTRUSIVE_WITH_DEFAULT, NLOHMANN_DEFINE_TYPE_INTRUSIVE_ONLY_SERIALIZE': api/macros/nlohmann_define_type_intrusive.md
           - 'NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE, NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE_WITH_DEFAULT, NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE_ONLY_SERIALIZE': api/macros/nlohmann_define_type_non_intrusive.md