Reserve output capacity up front for binary serialization

The vector-returning to_cbor/to_msgpack/to_ubjson/to_bjdata/to_bson grew the output buffer purely by geometric reallocation. Reserving an estimate up front avoids the early reallocations, which is the dominant per-byte cost for array/object-heavy output. The estimate (binary_reserve_hint) is deliberately conservative and safe against untrusted input: it consults only the top-level element count (O(1), no walk of the DOM), guards the multiplication against overflow, and clamps the result to a fixed 1 MiB ceiling, so a large or hostile DOM can never force an oversized allocation here. The buffer still grows geometrically past the hint, so an underestimate only costs a few later reallocations; scalars/strings/binary are written in one shot and get no hint. Reserving capacity does not change the bytes produced. Throughput (g++/clang -O3, vs the previous commit): cbor int array +10% / +13% cbor object array +20% / +38% Output is byte-for-byte identical to develop across the binary differential corpus. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Encode big-endian numbers with a byte swap instead of std::reverse
2026-08-11 11:43:18 +00:00 · 2026-08-04 08:57:53 +02:00 · 2026-08-04 08:54:04 +02:00 · 2026-08-04 08:54:04 +02:00 · 2026-08-04 08:54:03 +02:00
24 changed files with 768 additions and 644 deletions
@@ -38,14 +38,14 @@ jobs:

    # Initializes the CodeQL tools for scanning.
    - name: Initialize CodeQL
-      uses: github/codeql-action/init@d1ba80a13dd99fba24a470575428917156a28b43 # v4.37.5
+      uses: github/codeql-action/init@e4fba868fa4b1b91e1fdab776edc8cfbe6e9fb81 # v4.37.3
      with:
        languages: c-cpp

    # Autobuild attempts to build any compiled languages  (C/C++, C#, or Java).
    # If this step fails, then you should remove it and run the build manually (see below)
    - name: Autobuild
-      uses: github/codeql-action/autobuild@d1ba80a13dd99fba24a470575428917156a28b43 # v4.37.5
+      uses: github/codeql-action/autobuild@e4fba868fa4b1b91e1fdab776edc8cfbe6e9fb81 # v4.37.3

    - name: Perform CodeQL Analysis
-      uses: github/codeql-action/analyze@d1ba80a13dd99fba24a470575428917156a28b43 # v4.37.5
+      uses: github/codeql-action/analyze@e4fba868fa4b1b91e1fdab776edc8cfbe6e9fb81 # v4.37.3
@@ -43,6 +43,6 @@ jobs:
          output: 'flawfinder_results.sarif'

      - name: Upload analysis results to GitHub Security tab
-        uses: github/codeql-action/upload-sarif@d1ba80a13dd99fba24a470575428917156a28b43 # v4.37.5
+        uses: github/codeql-action/upload-sarif@e4fba868fa4b1b91e1fdab776edc8cfbe6e9fb81 # v4.37.3
        with:
          sarif_file: ${{github.workspace}}/flawfinder_results.sarif
@@ -76,6 +76,6 @@ jobs:

      # Upload the results to GitHub's code scanning dashboard.
      - name: "Upload to code-scanning"
-        uses: github/codeql-action/upload-sarif@d1ba80a13dd99fba24a470575428917156a28b43 # v4.37.5
+        uses: github/codeql-action/upload-sarif@e4fba868fa4b1b91e1fdab776edc8cfbe6e9fb81 # v4.37.3
        with:
          sarif_file: results.sarif
@@ -61,7 +61,7 @@ jobs:

      # Upload SARIF file generated in previous step
      - name: Upload SARIF file
-        uses: github/codeql-action/upload-sarif@d1ba80a13dd99fba24a470575428917156a28b43 # v4.37.5
+        uses: github/codeql-action/upload-sarif@e4fba868fa4b1b91e1fdab776edc8cfbe6e9fb81 # v4.37.3
        with:
          sarif_file: semgrep.sarif
        if: always()
@@ -25,7 +25,7 @@ jobs:
        with:
          persist-credentials: false
      - name: Get latest CMake and ninja
-        uses: lukka/get-cmake@4a7d025fc60f00db0c7b44ebf783d19b52444830 # v4.4.1
+        uses: lukka/get-cmake@e6906078ebd1ccb8ce51ab4626ac46a1b5a517e3 # v4.4.0
      - name: Run CMake
        run: cmake -S . -B build -DJSON_CI=On
      - name: Build
@@ -47,7 +47,7 @@ jobs:
        with:
          persist-credentials: false
      - name: Get latest CMake and ninja
-        uses: lukka/get-cmake@4a7d025fc60f00db0c7b44ebf783d19b52444830 # v4.4.1
+        uses: lukka/get-cmake@e6906078ebd1ccb8ce51ab4626ac46a1b5a517e3 # v4.4.0
      - name: Run CMake
        run: cmake -S . -B build -DJSON_CI=On
      - name: Build
@@ -70,7 +70,7 @@ jobs:
        with:
          persist-credentials: false
      - name: Get latest CMake and ninja
-        uses: lukka/get-cmake@4a7d025fc60f00db0c7b44ebf783d19b52444830 # v4.4.1
+        uses: lukka/get-cmake@e6906078ebd1ccb8ce51ab4626ac46a1b5a517e3 # v4.4.0
      - name: Run CMake
        run: cmake -S . -B build -DJSON_CI=On
      - name: Build
@@ -89,7 +89,7 @@ jobs:
        with:
          persist-credentials: false
      - name: Get latest CMake and ninja
-        uses: lukka/get-cmake@4a7d025fc60f00db0c7b44ebf783d19b52444830 # v4.4.1
+        uses: lukka/get-cmake@e6906078ebd1ccb8ce51ab4626ac46a1b5a517e3 # v4.4.0
      - name: Run CMake
        run: cmake -S . -B build -DJSON_CI=On
      - name: Build
@@ -108,7 +108,7 @@ jobs:
        with:
          persist-credentials: false
      - name: Get latest CMake and ninja
-        uses: lukka/get-cmake@4a7d025fc60f00db0c7b44ebf783d19b52444830 # v4.4.1
+        uses: lukka/get-cmake@e6906078ebd1ccb8ce51ab4626ac46a1b5a517e3 # v4.4.0
      - name: Run CMake
        run: cmake -S . -B build -DJSON_CI=On
      - name: Build
@@ -184,7 +184,7 @@ jobs:
        with:
          persist-credentials: false
      - name: Get latest CMake and ninja
-        uses: lukka/get-cmake@4a7d025fc60f00db0c7b44ebf783d19b52444830 # v4.4.1
+        uses: lukka/get-cmake@e6906078ebd1ccb8ce51ab4626ac46a1b5a517e3 # v4.4.0
      - name: Run CMake
        run: CXX=g++-${{ matrix.compiler }} cmake -S . -B build -DJSON_CI=On
      - name: Build
@@ -202,7 +202,7 @@ jobs:
        with:
          persist-credentials: false
      - name: Get latest CMake and ninja
-        uses: lukka/get-cmake@4a7d025fc60f00db0c7b44ebf783d19b52444830 # v4.4.1
+        uses: lukka/get-cmake@e6906078ebd1ccb8ce51ab4626ac46a1b5a517e3 # v4.4.0
      - name: Run CMake
        run: cmake -S . -B build -DJSON_CI=On
      - name: Build
@@ -212,14 +212,14 @@ jobs:
    runs-on: ubuntu-latest
    strategy:
      matrix:
-        compiler: ['3.4', '3.5', '3.6', '3.7', '3.8', '3.9', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15-bullseye', '16', '17', '18', '19', '20', '21', '22', 'latest']
+        compiler: ['3.4', '3.5', '3.6', '3.7', '3.8', '3.9', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15-bullseye', '16', '17', '18', '19', '20', 'latest']
    container: silkeh/clang:${{ matrix.compiler }}
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
        with:
          persist-credentials: false
      - name: Get latest CMake and ninja
-        uses: lukka/get-cmake@4a7d025fc60f00db0c7b44ebf783d19b52444830 # v4.4.1
+        uses: lukka/get-cmake@e6906078ebd1ccb8ce51ab4626ac46a1b5a517e3 # v4.4.0
      - name: Set env FORCE_STDCPPFS_FLAG for clang 7 / 8 / 9 / 10
        run: echo "JSON_FORCED_GLOBAL_COMPILE_OPTIONS=-DJSON_HAS_FILESYSTEM=0;-DJSON_HAS_EXPERIMENTAL_FILESYSTEM=0" >> "$GITHUB_ENV"
        if: ${{ matrix.compiler == '7' || matrix.compiler == '8' || matrix.compiler == '9' || matrix.compiler == '10' }}
@@ -239,7 +239,7 @@ jobs:
        with:
          persist-credentials: false
      - name: Get latest CMake and ninja
-        uses: lukka/get-cmake@4a7d025fc60f00db0c7b44ebf783d19b52444830 # v4.4.1
+        uses: lukka/get-cmake@e6906078ebd1ccb8ce51ab4626ac46a1b5a517e3 # v4.4.0
      - name: Run CMake
        run: cmake -S . -B build -DJSON_CI=On
      - name: Build
@@ -259,7 +259,7 @@ jobs:
        with:
          persist-credentials: false
      - name: Get latest CMake and ninja
-        uses: lukka/get-cmake@4a7d025fc60f00db0c7b44ebf783d19b52444830 # v4.4.1
+        uses: lukka/get-cmake@e6906078ebd1ccb8ce51ab4626ac46a1b5a517e3 # v4.4.0
      - name: Run CMake
        run: cmake -S . -B build -DJSON_CI=On
      - name: Build with libc++
@@ -286,7 +286,7 @@ jobs:
        with:
          persist-credentials: false
      - name: Get latest CMake and ninja
-        uses: lukka/get-cmake@4a7d025fc60f00db0c7b44ebf783d19b52444830 # v4.4.1
+        uses: lukka/get-cmake@e6906078ebd1ccb8ce51ab4626ac46a1b5a517e3 # v4.4.0
      - name: Run CMake
        run: cmake -S . -B build -DJSON_CI=On
      - name: Build
@@ -306,7 +306,7 @@ jobs:
      # import-std support. Its opt-in token is CMake-version-specific, so pin
      # CMake to the version whose token is set in tests/module_cpp20/CMakeLists.txt.
      - name: Get pinned CMake and ninja
-        uses: lukka/get-cmake@4a7d025fc60f00db0c7b44ebf783d19b52444830 # v4.4.1
+        uses: lukka/get-cmake@e6906078ebd1ccb8ce51ab4626ac46a1b5a517e3 # v4.4.0
        with:
          cmakeVersion: 4.3.4
      # Clang: the std library module is provided by libc++ (the image's libstdc++
@@ -332,7 +332,7 @@ jobs:
        with:
          persist-credentials: false
      - name: Get latest CMake and ninja
-        uses: lukka/get-cmake@4a7d025fc60f00db0c7b44ebf783d19b52444830 # v4.4.1
+        uses: lukka/get-cmake@e6906078ebd1ccb8ce51ab4626ac46a1b5a517e3 # v4.4.0
      - name: Run CMake
        run: cmake -S . -B build -DJSON_CI=On
      - name: Build
@@ -347,7 +347,7 @@ jobs:
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
      - name: Get latest CMake and ninja
-        uses: lukka/get-cmake@4a7d025fc60f00db0c7b44ebf783d19b52444830 # v4.4.1
+        uses: lukka/get-cmake@e6906078ebd1ccb8ce51ab4626ac46a1b5a517e3 # v4.4.0
      - name: Run CMake
        run: cmake -S . -B build -DJSON_CI=On
      - name: Build
@@ -359,7 +359,7 @@ jobs:
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
      - name: Get latest CMake and ninja
-        uses: lukka/get-cmake@4a7d025fc60f00db0c7b44ebf783d19b52444830 # v4.4.1
+        uses: lukka/get-cmake@e6906078ebd1ccb8ce51ab4626ac46a1b5a517e3 # v4.4.0
      - name: Run CMake
        run: cmake -S . -B build -DJSON_CI=On
      - name: Build
@@ -379,7 +379,7 @@ jobs:
        with:
          persist-credentials: false
      - name: Get latest CMake and ninja
-        uses: lukka/get-cmake@4a7d025fc60f00db0c7b44ebf783d19b52444830 # v4.4.1
+        uses: lukka/get-cmake@e6906078ebd1ccb8ce51ab4626ac46a1b5a517e3 # v4.4.0
      - name: Run CMake
        run: cmake -S . -B build -DCMAKE_TOOLCHAIN_FILE=$EMSDK/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake -GNinja
      - name: Build
@@ -88,7 +88,7 @@ jobs:
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
      - name: Get latest CMake and ninja
-        uses: lukka/get-cmake@4a7d025fc60f00db0c7b44ebf783d19b52444830 # v4.4.1
+        uses: lukka/get-cmake@e6906078ebd1ccb8ce51ab4626ac46a1b5a517e3 # v4.4.0
      - name: Set extra CXX_FLAGS for latest std_version
        # /wd5285 silences C5285 emitted by the bundled third-party doctest.h, which
        # specializes std::tuple (newly diagnosed by the VS2026 v145 toolset)
@@ -153,16 +153,10 @@ jobs:
        with:
          platform: x64
          version: 12.2.0 # https://github.com/egor-tensin/setup-mingw/issues/14
-      # CMAKE_CXX_FLAGS_DEBUG is overridden to drop the default -g: linking
-      # test-regression2_cpp20 intermittently fails with "relocation truncated
-      # to fit: IMAGE_REL_AMD64_SECREL against `.debug_line'" because the
-      # MinGW linker cannot relocate the debug sections this test produces.
-      # The tests are only built and run here, so the debug info is not used.
      - name: Run CMake
        run: cmake -S . -B build ^
             -DCMAKE_CXX_COMPILER="C:/Program Files/LLVM/bin/clang++.exe" ^
             -DCMAKE_CXX_FLAGS="--target=x86_64-w64-mingw32 -stdlib=libstdc++ -pthread" ^
-             -DCMAKE_CXX_FLAGS_DEBUG="-g0" ^
             -DCMAKE_EXE_LINKER_FLAGS="-lwinpthread" ^
             -G"MinGW Makefiles" ^
             -DCMAKE_BUILD_TYPE=Debug ^
@@ -199,7 +193,7 @@ jobs:
      # import-std support. Its opt-in token is CMake-version-specific, so pin
      # CMake to the version whose token is set in tests/module_cpp20/CMakeLists.txt.
      - name: Get pinned CMake and ninja
-        uses: lukka/get-cmake@4a7d025fc60f00db0c7b44ebf783d19b52444830 # v4.4.1
+        uses: lukka/get-cmake@e6906078ebd1ccb8ce51ab4626ac46a1b5a517e3 # v4.4.0
        with:
          cmakeVersion: 4.3.4
      - name: Run CMake (Debug)
@@ -42,7 +42,6 @@
  - [Specializing enum conversion](#specializing-enum-conversion)
  - [Binary formats (BSON, CBOR, MessagePack, UBJSON, and BJData)](#binary-formats-bson-cbor-messagepack-ubjson-and-bjdata)
 - [Customers](#customers)
- [Ecosystem](#ecosystem)
 - [Supported compilers](#supported-compilers)
 - [Integration](#integration)
  - [CMake](#cmake)
@@ -1187,11 +1186,6 @@ The library is used in multiple projects, applications, operating systems, etc.

 [![logos of customers using the library](docs/mkdocs/docs/images/customers.png)](https://json.nlohmann.me/home/customers/)

-## Ecosystem
-
-Beyond projects that use the library, there are third-party projects that build on top of it - schema validators,
-language bindings, format converters, and the like. See the curated [Ecosystem](https://json.nlohmann.me/community/ecosystem/) page.
-
 ## Supported compilers

 Though it's 2026 already, the support for C++11 is still a bit sparse. Currently, the following compilers are known to work:
@@ -1,40 +0,0 @@
-# Ecosystem
-
-The projects below build on top of `nlohmann::json` rather than merely using it - schema validators, language
-bindings, format converters, and similar building blocks. The list is not exhaustive, and is curated rather than
-automatically generated. If you maintain or know of a project that belongs here,
-[please let me know](mailto:mail@nlohmann.me).
-
-For products, applications, and organizations that use the library, see [Customers](../home/customers.md) instead.
-
-## Schema validation
-
- [**json-schema-validator**](https://github.com/pboettch/json-schema-validator), a JSON Schema (draft 7) validator
-  with human-readable error messages
-
-## Serialization and reflection
-
- [**nlohmann_json_reflect**](https://github.com/1261385937/nlohmann_json_reflect), a reflection extension for
-  (de)serializing nested containers-in-structs-in-containers
-
-## Encodings
-
- [**base-encode-decode**](https://github.com/saxonnicholls/base-encode-decode), a header-only Base64/32/16/8/4/2
-  (and DNA/RNA) encoding library, with an adapter that serializes binary data through `nlohmann::json`
-
-## Language bindings and interop
-
- [**pybind11_json**](https://github.com/pybind/pybind11_json), a bidirectional type caster between
-  `nlohmann::json` and Python objects for [pybind11](https://github.com/pybind/pybind11) bindings
- [**nanobind_json**](https://github.com/ianhbell/nanobind_json), the same idea for
-  [nanobind](https://github.com/wjakob/nanobind) bindings
- [**nlohmann_json_qt**](https://github.com/dpurgin/nlohmann_json_qt), deserialization helpers for Qt types
-  (`QString`, `QUrl`, `QDateTime`, `QVector`, ...) from `nlohmann::json`
- [**vulkan2json**](https://github.com/Fadis/vulkan2json), serialization and deserialization of Vulkan API structs
-
-## Format converters
-
- [**tojson**](https://github.com/mircodz/tojson), a header-only converter between YAML/XML documents and
-  `nlohmann::json`
- [**json2xml**](https://github.com/testillano/json2xml), a header-only converter from `nlohmann::json` to XML for
-  simple configuration documents
@@ -1,6 +1,5 @@
 # Community

- [Ecosystem](ecosystem.md) - third-party projects built on top of this library
 - [Code of Conduct](code_of_conduct.md) - the rules and norms of this project
 - [Contribution Guidelines](contribution_guidelines.md) - guidelines how to contribute to this project
 - [Governance](governance.md) - the governance model of this project
@@ -66,7 +66,6 @@ Note: Some modern features (like C++20 ranges or filesystem support) may be disa
        | Clang 20.1.1                                 | x86_64       | Ubuntu 22.04.1 LTS                | GitHub    |
        | Clang 20.1.8 with GNU-like command-line      | x86_64       | Windows Server 2022 (Build 20348) | GitHub    |
        | Clang 21.1.8                                 | x86_64       | Ubuntu 22.04.1 LTS                | GitHub    |
-        | Clang 22.1.8                                 | x86_64       | Ubuntu 22.04.1 LTS                | GitHub    |
        | CUDA 11.8.0 (nvcc)                           | x86_64       | Ubuntu 22.04 LTS                  | GitHub    |
        | CUDA 12.1.1 (nvcc)                           | x86_64       | Ubuntu 22.04 LTS                  | GitHub    |
        | CUDA 12.6.3 (nvcc)                           | x86_64       | Ubuntu 22.04 LTS                  | GitHub    |
@@ -116,19 +116,9 @@ The library uses the following mapping from JSON values types to BJData types ac
    ```

    Likewise, when a JSON object in the above form is serialized using
-    [`to_bjdata`](../../api/basic_json/to_bjdata.md), it is automatically converted into a compact BJData ND-array. When
-    the 1-dimensional vector stored in `"_ArraySize_"` contains a single integer or two integers with one being 1, a
-    regular 1-D optimized array is generated instead.
-
-    An object is only converted if the annotation actually describes a packed array; otherwise it is serialized as a
-    regular JSON object. This requires all of the following:
-
-    - `"_ArrayType_"` is one of `uint8`, `int8`, `uint16`, `int16`, `uint32`, `int32`, `uint64`, `int64`, `single`,
-      `double`, `char`, or `byte`,
-    - every entry of `"_ArraySize_"` is a non-negative integer, and their product is representable as a `std::size_t`,
-    - `"_ArrayData_"` holds exactly that many elements, and
-    - every element of `"_ArrayData_"` is a number of the kind named by `"_ArrayType_"` (a floating-point number for
-      `single` and `double`, an integer otherwise).
+    [`to_bjdata`](../../api/basic_json/to_bjdata.md), it is automatically converted into a compact BJData ND-array. The
+    only exception is, that when the 1-dimensional vector stored in `"_ArraySize_"` contains a single integer or two
+    integers with one being 1, a regular 1-D optimized array is generated.

    The current version of this library does not yet support automatic detection of and conversion from a nested JSON
    array input to a BJData ND-array.
@@ -308,7 +308,6 @@ nav:
          - 'NLOHMANN_JSON_VERSION_MAJOR, NLOHMANN_JSON_VERSION_MINOR, NLOHMANN_JSON_VERSION_PATCH': api/macros/nlohmann_json_version_major.md
  - Community:
      - community/index.md
-      - community/ecosystem.md
      - "Code of Conduct": community/code_of_conduct.md
      - community/contribution_guidelines.md
      - community/quality_assurance.md
@@ -465,6 +465,15 @@ class binary_reader
    // CBOR //
    //////////

+    /*!
+    @param[in] get_char  whether a new character should be retrieved from the
+                         input (true) or whether the last read character should
+                         be considered instead (false)
+    @param[in] tag_handler how CBOR tags should be treated
+
+    @return whether a valid CBOR value was passed to the SAX parser
+    */
+
    template<typename NumberType>
    bool get_cbor_negative_integer()
    {
@@ -483,14 +492,6 @@ class binary_reader
        return sax->number_integer(static_cast<number_integer_t>(-1) - static_cast<number_integer_t>(number));
    }

-    /*!
-    @param[in] get_char  whether a new character should be retrieved from the
-                         input (true) or whether the last read character should
-                         be considered instead (false)
-    @param[in] tag_handler how CBOR tags should be treated
-
-    @return whether a valid CBOR value was passed to the SAX parser
-    */
    bool parse_cbor_internal(const bool get_char,
                             const cbor_tag_handler_t tag_handler)
    {
@@ -1987,11 +1988,7 @@ class binary_reader
    {
        if (get_char)
        {
-            // no get_ignore_noop() here: the byte read next must be a string
-            // length type specification, and a no-op ('N') is not valid in
-            // that position. No-ops at positions where a value may appear are
-            // already consumed by the callers via get_ignore_noop().
-            get();
+            get();  // TODO(niels): may we ignore N here?
        }

        if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format, "value")))
@@ -345,12 +345,8 @@ struct wide_string_input_helper<BaseInputAdapter, 4>
            }
            else
            {
-                // A code point above U+10FFFF has no UTF-8 encoding. Passing the
-                // unit through would narrow it to int, where 0xFFFFFFFF becomes
-                // char_traits<char>::eof() and would end the input silently, so
-                // emit a byte that is never valid UTF-8 and let the decoder
-                // reject it.
-                utf8_bytes[0] = 0xFF;
+                // unknown character
+                utf8_bytes[0] = static_cast<std::char_traits<char>::int_type>(wc);
                utf8_bytes_filled = 1;
            }
        }
@@ -370,10 +370,8 @@ class json_sax_dom_parser

                case value_t::string:
                {
-                    // escape sequences make the token longer than the value it
-                    // parses to, so the start position cannot be derived from
-                    // the value; use the offset the lexer recorded instead
-                    v.start_position = m_lexer_ref->get_token_start_position();
+                    // include the length of the quotes, which is 2
+                    v.start_position = v.end_position - v.m_data.m_value.string->size() - 2;
                    break;
                }

@@ -771,10 +769,8 @@ class json_sax_dom_callback_parser

                case value_t::string:
                {
-                    // escape sequences make the token longer than the value it
-                    // parses to, so the start position cannot be derived from
-                    // the value; use the offset the lexer recorded instead
-                    v.start_position = m_lexer_ref->get_token_start_position();
+                    // include the length of the quotes, which is 2
+                    v.start_position = v.end_position - v.m_data.m_value.string->size() - 2;
                    break;
                }

@@ -1357,11 +1357,6 @@ scan_number_done:
        token_buffer.clear();
        decimal_point_position = std::string::npos;

-#if JSON_DIAGNOSTIC_POSITIONS
-        // the first character of the token has already been read, hence the -1
-        token_start_position = position.chars_read_total - 1;
-#endif
-
        note_token_start(std::integral_constant<bool, lazy_token_string> {});
    }

@@ -1524,15 +1519,6 @@ scan_number_done:
        return position;
    }

-#if JSON_DIAGNOSTIC_POSITIONS
-    /// return the offset of the first character of the last read token; unlike
-    /// the token's parsed value, this accounts for escape sequences
-    constexpr std::size_t get_token_start_position() const noexcept
-    {
-        return token_start_position;
-    }
-#endif
-
    /// seekable adapter: rebuild the last read token from the input on demand
    const std::vector<char_type>& collect_token_chars(std::vector<char_type>& out, std::true_type /*lazy*/) const
    {
@@ -1733,12 +1719,6 @@ scan_number_done:
    /// the last read token on error for seekable adapters (see collect_token_chars)
    std::size_t token_string_start = 0;

-#if JSON_DIAGNOSTIC_POSITIONS
-    /// start offset of the current token within the input, used to report
-    /// diagnostic positions (see reset())
-    std::size_t token_start_position = 0;
-#endif
-
    /// buffer for variable-length tokens (numbers, strings)
    string_t token_buffer {};

@@ -13,6 +13,7 @@
 #include <iterator> // back_inserter
 #include <memory> // shared_ptr, make_shared
 #include <string> // basic_string
+#include <utility> // move
 #include <vector> // vector

 #ifndef JSON_NO_IO
@@ -118,6 +119,72 @@ class output_string_adapter : public output_adapter_protocol<CharType>
    StringType& str;
 };

+/// @brief non-virtual output sink writing into a std::vector
+///
+/// Unlike output_vector_adapter, this sink is not part of the virtual
+/// output_adapter_protocol hierarchy: it is passed to binary_writer by value as
+/// a template parameter, so write_character()/write_characters() are ordinary
+/// (inlinable) calls with no vtable lookup and no shared_ptr. It is used for the
+/// common `to_cbor`/`to_msgpack`/... into a std::vector.
+template<typename CharType, typename AllocatorType = std::allocator<CharType>>
+class output_vector_sink
+{
+  public:
+    explicit output_vector_sink(std::vector<CharType, AllocatorType>& vec) noexcept
+        : v(vec)
+    {}
+
+    void write_character(CharType c)
+    {
+        v.push_back(c);
+    }
+
+    // no JSON_HEDLEY_NON_NULL here: binary_writer legitimately passes a null
+    // pointer with length 0 for empty strings/binary values. Appending an empty
+    // range is a no-op; the type-erased path tolerates this via the (unattributed)
+    // virtual base, and the concrete sink must do the same.
+    void write_characters(const CharType* s, std::size_t length)
+    {
+        v.insert(v.end(), s, s + length);
+    }
+
+  private:
+    std::vector<CharType, AllocatorType>& v;
+};
+
+/// @brief output sink forwarding to a type-erased output adapter
+///
+/// Wraps the polymorphic output_adapter_t so the same binary_writer template can
+/// also target arbitrary adapters (output streams, strings, user-provided
+/// adapters) via the `output_adapter`-based overloads. Each write still goes
+/// through one virtual call, exactly as before; only the concrete sinks above
+/// avoid it.
+template<typename CharType>
+class output_adapter_sink
+{
+  public:
+    explicit output_adapter_sink(output_adapter_t<CharType> adapter)
+        : oa(std::move(adapter))
+    {
+        JSON_ASSERT(oa);
+    }
+
+    void write_character(CharType c)
+    {
+        oa->write_character(c);
+    }
+
+    // no JSON_HEDLEY_NON_NULL: forwards (null, 0) for empty payloads, exactly as
+    // the type-erased path already did before this sink existed
+    void write_characters(const CharType* s, std::size_t length)
+    {
+        oa->write_characters(s, length);
+    }
+
+  private:
+    output_adapter_t<CharType> oa = nullptr;
+};
+
 template<typename CharType, typename StringType = std::basic_string<CharType>>
 class output_adapter
 {
@@ -140,7 +140,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
    friend ::nlohmann::detail::serializer<basic_json>;
    template<typename BasicJsonType>
    friend class ::nlohmann::detail::iter_impl;
-    template<typename BasicJsonType, typename CharType>
+    template<typename BasicJsonType, typename CharType, typename OutputSinkType>
    friend class ::nlohmann::detail::binary_writer;
    template<typename BasicJsonType, typename InputType, typename SAX>
    friend class ::nlohmann::detail::binary_reader;
@@ -4327,7 +4327,9 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
    static std::vector<std::uint8_t> to_cbor(const basic_json& j)
    {
        std::vector<std::uint8_t> result;
-        to_cbor(j, result);
+        result.reserve(detail::binary_reserve_hint(j));
+        detail::binary_writer<basic_json, std::uint8_t, detail::output_vector_sink<std::uint8_t>>(
+            detail::output_vector_sink<std::uint8_t>(result)).write_cbor(j);
        return result;
    }

@@ -4350,7 +4352,9 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
    static std::vector<std::uint8_t> to_msgpack(const basic_json& j)
    {
        std::vector<std::uint8_t> result;
-        to_msgpack(j, result);
+        result.reserve(detail::binary_reserve_hint(j));
+        detail::binary_writer<basic_json, std::uint8_t, detail::output_vector_sink<std::uint8_t>>(
+            detail::output_vector_sink<std::uint8_t>(result)).write_msgpack(j);
        return result;
    }

@@ -4375,7 +4379,9 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
            const bool use_type = false)
    {
        std::vector<std::uint8_t> result;
-        to_ubjson(j, result, use_size, use_type);
+        result.reserve(detail::binary_reserve_hint(j));
+        detail::binary_writer<basic_json, std::uint8_t, detail::output_vector_sink<std::uint8_t>>(
+            detail::output_vector_sink<std::uint8_t>(result)).write_ubjson(j, use_size, use_type);
        return result;
    }

@@ -4403,7 +4409,9 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
            const bjdata_version_t version = bjdata_version_t::draft2)
    {
        std::vector<std::uint8_t> result;
-        to_bjdata(j, result, use_size, use_type, version);
+        result.reserve(detail::binary_reserve_hint(j));
+        detail::binary_writer<basic_json, std::uint8_t, detail::output_vector_sink<std::uint8_t>>(
+            detail::output_vector_sink<std::uint8_t>(result)).write_ubjson(j, use_size, use_type, true, true, version);
        return result;
    }

@@ -4430,7 +4438,9 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
    static std::vector<std::uint8_t> to_bson(const basic_json& j)
    {
        std::vector<std::uint8_t> result;
-        to_bson(j, result);
+        result.reserve(detail::binary_reserve_hint(j));
+        detail::binary_writer<basic_json, std::uint8_t, detail::output_vector_sink<std::uint8_t>>(
+            detail::output_vector_sink<std::uint8_t>(result)).write_bson(j);
        return result;
    }

@@ -2730,27 +2730,6 @@ TEST_CASE("BJData")
                CHECK(json::from_bjdata(json::to_bjdata(j_type), true, true) == j_type);
                CHECK(json::from_bjdata(json::to_bjdata(j_size), true, true) == j_size);
            }
-
-            SECTION("ndarray whose dimensions overflow stays as object")
-            {
-                // the product of the dimensions wraps around std::size_t to 0
-                // and so matches the size of the empty _ArrayData_; writing this
-                // as an ndarray would announce an element count no reader can
-                // honor, so it has to stay a plain object
-                json j_overflow = json({{"_ArrayData_", json::array()}, {"_ArraySize_", {9223372036854775808ull, 2}}, {"_ArrayType_", "uint8"}});
-                CHECK(json::from_bjdata(json::to_bjdata(j_overflow), true, true) == j_overflow);
-
-                // a single dimension that does not fit into std::size_t is
-                // rejected for the same reason (only observable where
-                // std::size_t is narrower than 64 bit)
-                json j_huge = json({{"_ArrayData_", json::array()}, {"_ArraySize_", {18446744073709551615ull}}, {"_ArrayType_", "uint8"}});
-                CHECK(json::from_bjdata(json::to_bjdata(j_huge), true, true) == j_huge);
-
-                // a well-formed ndarray is still encoded as one
-                json j_ok = json({{"_ArrayData_", {1, 2, 3, 4, 5, 6}}, {"_ArraySize_", {2, 3}}, {"_ArrayType_", "uint8"}});
-                CHECK(json::to_bjdata(j_ok) == std::vector<uint8_t>({'[', '$', 'U', '#', '[', 'i', 2, 'i', 3, ']', 1, 2, 3, 4, 5, 6}));
-                CHECK(json::from_bjdata(json::to_bjdata(j_ok), true, true) == j_ok);
-            }
        }
    }

@@ -38,36 +38,6 @@ TEST_CASE("Better diagnostics with positions")
                             "[json.exception.type_error.302] type must be number, but is string", json::type_error);
    }

-    SECTION("positions of strings containing escape sequences")
-    {
-        // escape sequences make the token longer than the string it parses to,
-        // so the positions must not be derived from the parsed value's length
-        const auto check = [](const std::string & text, const std::string & token)
-        {
-            CAPTURE(text)
-            CAPTURE(token)
-            const json j = json::parse(text);
-            const json& v = j.at("a");
-            CHECK(text.substr(v.start_pos(), v.end_pos() - v.start_pos()) == token);
-        };
-
-        check(R"({"a":"plain"})", R"("plain")");
-        check(R"({"a":"tab\there"})", R"("tab\there")");
-        check(R"({"a":"\n\n\n\n\n\n"})", R"("\n\n\n\n\n\n")");
-        check(R"({"a":"\""})", R"("\"")");
-        check(R"({"a":"\\"})", R"("\\")");
-        check(R"({"a":"é"})", R"("é")");
-        check(R"({"a":"🌞"})", R"("🌞")");
-        check("{\"a\":\"\xc3\xa9\"}", "\"\xc3\xa9\"");  // multi-byte UTF-8, no escapes
-
-        // a string at the root, where an escape would otherwise push the
-        // reported start position past the opening quote
-        const std::string root = R"("a\tb")";
-        const json j = json::parse(root);
-        CHECK(j.start_pos() == 0);
-        CHECK(j.end_pos() == root.size());
-    }
-
    SECTION("JSON patch add to primitive parent (#4292)")
    {
        // the JSON Patch "add" target /foo/bar/baz has a string parent
@@ -1713,44 +1713,6 @@ TEST_CASE("UBJSON")
            CHECK(json::to_ubjson(json::from_ubjson(s_L)) == s_i);
        }

-        SECTION("no-op markers")
-        {
-            // A no-op ('N') is valid wherever a value may start; it is consumed
-            // by get_ignore_noop() before the value is read. It is not valid
-            // where a string length type specification is expected.
-
-            SECTION("accepted where a value may start")
-            {
-                // at top level, also repeated
-                CHECK(json::from_ubjson(std::vector<uint8_t>({'N', 'i', 1})) == json(1));
-                CHECK(json::from_ubjson(std::vector<uint8_t>({'N', 'N', 'N', 'i', 1})) == json(1));
-
-                // inside an array of unknown size, before and after an element
-                CHECK(json::from_ubjson(std::vector<uint8_t>({'[', 'N', 'i', 1, ']'})) == json({1}));
-                CHECK(json::from_ubjson(std::vector<uint8_t>({'[', 'i', 1, 'N', ']'})) == json({1}));
-
-                // inside an object of unknown size: before a key, between key
-                // and value, and before the closing '}'
-                CHECK(json::from_ubjson(std::vector<uint8_t>({'{', 'N', 'U', 1, 'a', 'i', 1, '}'})) == json({{"a", 1}}));
-                CHECK(json::from_ubjson(std::vector<uint8_t>({'{', 'U', 1, 'a', 'N', 'i', 1, '}'})) == json({{"a", 1}}));
-                CHECK(json::from_ubjson(std::vector<uint8_t>({'{', 'U', 1, 'a', 'i', 1, 'N', '}'})) == json({{"a", 1}}));
-            }
-
-            SECTION("rejected where a length type specification is expected")
-            {
-                json _;
-
-                // after the 'S' marker of a string value
-                std::vector<uint8_t> const v_S = {'S', 'N', 'U', 1, 'a'};
-                CHECK_THROWS_WITH_AS(_ = json::from_ubjson(v_S), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing UBJSON string: expected length type specification (U, i, I, l, L); last byte: 0x4E", json::parse_error&);
-
-                // as the key length of an object with a known size, where
-                // no-ops are not permitted in the first place
-                std::vector<uint8_t> const v_key = {'{', '#', 'i', 1, 'N', 'U', 1, 'a', 'i', 1};
-                CHECK_THROWS_WITH_AS(_ = json::from_ubjson(v_key), "[json.exception.parse_error.113] parse error at byte 5: syntax error while parsing UBJSON string: expected length type specification (U, i, I, l, L); last byte: 0x4E", json::parse_error&);
-            }
-        }
-
        SECTION("number")
        {
            SECTION("float")
@@ -125,16 +125,6 @@ TEST_CASE("wide strings")
            std::u32string const w = U"\"\x110000";
            json _;
            CHECK_THROWS_AS(_ = json::parse(w), json::parse_error&);
-
-            // a code unit above U+10FFFF must not be narrowed onto the EOF
-            // sentinel: 0xFFFFFFFF would otherwise end the document silently and
-            // let everything following it pass the strict end-of-input check
-            std::u32string const trailing{U'[', U'1', U']', static_cast<char32_t>(0xFFFFFFFF), U'x'};
-            CHECK_THROWS_WITH_AS(_ = json::parse(trailing), "[json.exception.parse_error.101] parse error at line 1, column 4: syntax error while parsing value - invalid literal; last read: '1]\xFF'; expected end of input", json::parse_error&);
-            CHECK(!json::accept(trailing));
-
-            // the same unit inside a string is reported as an ill-formed byte
-            CHECK_THROWS_WITH_AS(_ = json::parse(std::u32string{U'"', static_cast<char32_t>(0xFFFFFFFF), U'"'}), "[json.exception.parse_error.101] parse error at line 1, column 2: syntax error while parsing value - invalid string: ill-formed UTF-8 byte; last read: '\"\xFF'", json::parse_error&);
        }
    }
 }
Author	SHA1	Message	Date
Niels LohmannandClaude Opus 4.8	8895cfb9c8	Reserve output capacity up front for binary serialization The vector-returning to_cbor/to_msgpack/to_ubjson/to_bjdata/to_bson grew the output buffer purely by geometric reallocation. Reserving an estimate up front avoids the early reallocations, which is the dominant per-byte cost for array/object-heavy output. The estimate (binary_reserve_hint) is deliberately conservative and safe against untrusted input: it consults only the top-level element count (O(1), no walk of the DOM), guards the multiplication against overflow, and clamps the result to a fixed 1 MiB ceiling, so a large or hostile DOM can never force an oversized allocation here. The buffer still grows geometrically past the hint, so an underestimate only costs a few later reallocations; scalars/strings/binary are written in one shot and get no hint. Reserving capacity does not change the bytes produced. Throughput (g++/clang -O3, vs the previous commit): cbor int array +10% / +13% cbor object array +20% / +38% Output is byte-for-byte identical to develop across the binary differential corpus. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-08-04 08:57:53 +02:00
Niels LohmannandClaude Opus 4.8	96ad89718c	Encode big-endian numbers with a byte swap instead of std::reverse write_number() reordered multi-byte numbers for the big-endian formats (CBOR/MessagePack/UBJSON) with std::reverse over the byte array. GCC lowered only some sizes to a bswap; clang kept a scalar byte shuffle (0 bswap instructions in the CBOR number path). Replace the reverse with size-dispatched __builtin_bswap16/32/64 helpers (portable shift fallback for other compilers; std::reverse retained for exotic sizes such as a long double number_float_t). Codegen: the CBOR number path now emits bswap on both compilers (gcc 2 -> 16, clang 0 -> 4). Output is byte-for-byte identical to the previous implementation across the binary differential corpus. Throughput (isolated vs the std::reverse version, best of 9): CBOR int64 array gcc +7% clang +10% CBOR uint16 array gcc +27% clang flat Modest but consistent on number-dense encodings; negligible on string/blob-heavy output, as expected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-08-04 08:54:04 +02:00
Niels LohmannandClaude Opus 4.8	b93899b9c4	Fix CI failures from binary_writer output-sink change Four CI jobs failed on the initial commit; all are addressed here without changing any output (binary encodings remain byte-for-byte identical to develop across the differential corpus): 1. ci_test_gcc / cuda (-Werror=duplicated-branches): for number_float_t == float, static_cast<float>(n) is the identity, so write_compact_float's two branches are intentionally identical. Once the concrete vector sink is inlined, GCC constant-folds and diagnoses this (the type-erased path hid it behind a non-inlined virtual call). Silence -Wduplicated-branches for GCC (clang has no such warning) alongside the existing -Wfloat-equal pragma. 2. ci_static_analysis_clang (UBSan nonnull-attribute): binary_writer passes a null pointer with length 0 for empty strings/binary. output_vector_sink / output_adapter_sink declared write_characters JSON_HEDLEY_NON_NULL, so the sanitizer flagged the (harmless) zero-length call once the sink was called directly rather than through the attribute-free virtual base. Drop the attribute from both sinks, matching the pre-existing behavior. 3. ci_cpplint (build/include_what_you_use): output_adapter_sink uses std::move; add #include <utility>. 4. ci_cuda_example (nvcc 11.8): NVCC's front end rejects the default template argument on the binary_writer alias template. Revert the alias to its original single-parameter form (relying on binary_writer's own defaulted OutputSinkType) and spell out the full type in the vector-sink convenience functions. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-08-04 08:54:04 +02:00
Niels LohmannandClaude Opus 4.8	a42095fd01	Devirtualize binary_writer via a value-type output sink to_cbor/to_msgpack/to_ubjson/to_bjdata/to_bson wrote every byte through output_adapter_t, a shared_ptr<output_adapter_protocol> whose write_character/write_characters are virtual. Unlike the lexer (templated on a concrete InputAdapterType), the binary writer never got that treatment, so binary output paid a vtable lookup per byte and a make_shared per call. Template binary_writer on an OutputSinkType and give it two concrete, non-virtual sinks: - output_vector_sink: appends straight into a std::vector (push_back / insert), used by the vector-returning to_* convenience functions. No vtable, no shared_ptr; the writes inline. - output_adapter_sink: forwards to a type-erased output_adapter_t, so the existing to_*(j, output_adapter) overloads (streams, strings, custom adapters) keep working exactly as before -- one virtual call each, unchanged. binary_writer keeps a convenience constructor taking output_adapter_t (building the default output_adapter_sink), so the adapter overloads are untouched; only the convenience functions switch to the vector sink. The friend declaration and the basic_json binary_writer alias gain the new (defaulted) template parameter. Output is byte-for-byte identical: verified across ~3000 randomized values plus curated edge cases (all scalar widths, strings with invalid UTF-8, binary, nested arrays/objects) for CBOR, MessagePack, UBJSON (both size/type settings), BJData, and BSON, plus the output_adapter path, in C++11/17/20. Warning-clean under clang -Weverything and the gcc pedantic set; clang-tidy clean on the changed headers; make check-amalgamation clean. Throughput (g++ -O3, vs develop): scalar-dense binary output such as integer arrays ~1.4x; many small to_cbor calls ~1.04x (DOM traversal bound); string/blob-heavy output unchanged (already bulk-bound). No workload regressed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG Signed-off-by: Niels Lohmann <mail@nlohmann.me>	2026-08-04 08:54:03 +02:00