Compare commits

..
10 Commits
Author SHA1 Message Date
Niels Lohmann fd72d68d78 Do not dispatch the SSSE3 string scan with GCC's C++20 modules
GCC does not keep the target("ssse3") attribute on functions it reads
back from a module interface, so the module build (ci_module_cpp20,
gcc:latest) rejected the intrinsics in vector_utf8_run() with "needs isa
option -mssse3". With __cpp_modules, GCC now uses the portable UTF-8 check
like other compilers without the target attribute; the SSE2 scan stays,
and -mssse3 or more still compiles the SSSE3 check directly. Clang is not
affected.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-17 08:29:53 +02:00
Niels Lohmann eeccff6eb3 Define string_bulk_run_from() only where lexer_string_bulk_run() calls it
With NEON and simdutf in C++17, lexer_string_bulk_run() takes the
string_bulk_run() path, so the out-of-line helper was unused.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-17 08:29:52 +02:00
Niels Lohmann 44983f8c84 Keep the string scan from slowing down short strings and dump() on Arm
Three benchmarks were 7-9% slower than develop on Apple M1 (Apple Clang,
-O3 -flto): Accept/jeopardy, Dump/citm_catalog, and Dump/jeopardy.

- dump(): the serializer passes a string without its closing quote, so
  plain_run_end() ran out of input and called the out-of-line NEON loop
  for every string, with fewer than 16 bytes left. The loop is now only
  called for 16 bytes or more.
- accept(): no string of jeopardy.json reaches the vector UTF-8 check,
  yet compiling it into the lexer's scan function made it 5% slower: the
  function runs once per token and now saved more registers and kept a
  stack slot. lexer_string_bulk_run() checks the first four words inline
  and continues out of line, and the serializer keeps string_bulk_run()
  inline, which is faster there.
- The byte that ends a run is read from the scanned buffer instead of
  through get(), which Apple Clang merged with the other get() calls that
  load the read position again right after it was stored (3%).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-17 08:29:51 +02:00
Niels Lohmann 6368bf154d Scan strings with SSE2, SSSE3, or NEON
The string fast path found the end of a run of plain characters 8 bytes at
a time (SWAR) and validated non-ASCII text one sequence at a time, calling
validate_one_utf8() out of line for each.

The vector code of json_view's scanner, in the new shared header
detail/input/simd_scan.hpp, now does both 16 bytes at a time: SSE2 finds
the end of a run on x86-64, and the UTF-8 check of Keiser and Lemire runs
with NEON on AArch64 and with SSSE3 on x86-64. SSSE3 is not part of x86-64,
so the check is compiled for it with a function attribute and used when the
CPU has it (cached in a constant-initialized atomic), else the scalar check;
code compiled with -mssse3 skips the CPU check. The first word of a run is
still checked with SWAR (the first four with NEON, where a vector mask
costs more), as most keys and short values end there, and the scalar
helpers are always inlined. On anything else, the scalar code is unchanged.

The new macro JSON_NO_SIMD selects the portable code everywhere, including
the SSE2/NEON digit formatting of the floating-point writer. The lexer tests
now also run with JSON_NO_SIMD and, on x86-64, with -mssse3, and a new test
places every byte that ends a run at each offset across several 16-byte
blocks, after ASCII and after CJK text.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-17 08:29:51 +02:00
Niels Lohmann aab1fa9e53 Copy the text of float tokens only when a SAX handler asks for it
After the digits are converted while scanning, copying the token text into
token_buffer (an out-of-line std::string::append) was about 10% of parsing
canada.json. The only reader is the string argument of
json_sax::number_float(), and the library's own handlers
(json_sax_dom_parser, json_sax_dom_callback_parser, json_sax_acceptor)
ignore it.

For contiguous input, the fast path now keeps a pointer to the token in the
input, and get_string() copies it on first use. The parser passes an empty
string to the handlers marked by the new internal trait
sax_ignores_number_text instead of calling get_string(), so json::parse()
and json::accept() never copy float text. User handlers, and
JSON_DIAGNOSTIC_POSITIONS (which reads get_string().size()), still see the
exact text. Keys and strings, which never have pending text, use the
unchecked get_string_token(). Input read through a window into a stream
(std::istream, FILE*) copies the text right away, because the next read may
refill the window.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-17 08:29:50 +02:00
Niels Lohmann 722b2c4334 Skip the digit accumulation checks without exceptions
The section compares error messages from both lexer paths and parses
out-of-range numbers, so it does not compile (the caught exception does not
exist) and would abort with JSON_NOEXCEPTION.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-17 08:29:49 +02:00
Niels Lohmann 81ca04f3ec Accumulate number digits while scanning them
The contiguous number fast path validated a token byte by byte, copied it
into token_buffer, and then converted it in out-of-line functions that
scanned the digits a second time: parse_float_native() called
parse_float_significand(), which returned its result through memory and
called std::any_of (out of line with GCC) twice per number, and
parse_integer_signed() divided by ten on every digit for its overflow check.

scan_number_bulk_contiguous() now accumulates the significand while it
validates the digits, eight at a time where at least eight bytes remain
(is_eight_digits() and parse_eight_digits(), after fast_float). Floats with
at most 19 digits go straight to decimal_to_float(); integers with at most
18 digits are stored without an overflow check (a narrower number type
still rejects values that do not round-trip). Longer tokens take the
existing path. has_nonzero_digit() is a plain loop and
parse_float_significand() is always inlined, which helps that path.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-17 08:29:48 +02:00
Niels Lohmann f25c6543aa Skip the exception-only checks of the stream input tests without exceptions
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-17 08:29:47 +02:00
Niels Lohmann c7a9780936 Add ABI tags for JSON_DISABLE_STREAM_BULK_SCAN and JSON_DISABLE_UNLOCKED_FILE_IO
Both macros select between two definitions of an input adapter, so
translation units compiled with different settings violated the ODR when
linked together. Like the other ABI-affecting macros, they now append a
tag to the inline namespace: _dsbs and _dufio. Their defaults move to
abi_macros.hpp so that json_fwd.hpp sees them, and the natvis file is
regenerated for the new namespaces.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-17 08:29:46 +02:00
Niels Lohmann 1d2526b1d8 Make parsing from std::istream and FILE* fast
Squashed onto develop cd6b428ef (CI fixes from #5860). Original commits:

- Let the lexer's bulk fast paths run on a window of the input
- Scan the get area of an input stream in bulk
- Lock a FILE once per parse instead of once per character
- Add benchmarks for parsing from a FILE* and for many values from a stream
- Avoid implicit unsigned char to char conversion in stream tests
- Make file_input_adapter::unlock() safe to call on a moved-from adapter

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-17 08:29:45 +02:00
9 changed files with 38 additions and 108 deletions

No files matched your search

+1 -1
View File
@@ -37,7 +37,7 @@ jobs:
path: pr/
check:
runs-on: ubuntu-26.04
runs-on: ubuntu-latest
env:
MAIN_DIR: ${{ github.workspace }}/main
INCLUDE_DIR: ${{ github.workspace }}/main/single_include/nlohmann
+1 -1
View File
@@ -21,7 +21,7 @@ permissions:
jobs:
CodeQL-Build:
runs-on: ubuntu-26.04
runs-on: ubuntu-latest
permissions:
security-events: write
+1 -1
View File
@@ -33,7 +33,7 @@ jobs:
security-events: write # for github/codeql-action/upload-sarif to upload SARIF results
actions: read # only required for a private repository by github/codeql-action/upload-sarif to get the Action run status
name: Scan
runs-on: ubuntu-26.04
runs-on: ubuntu-latest
steps:
- name: Harden Runner
uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1
+6 -6
View File
@@ -32,7 +32,7 @@ jobs:
run: cmake --build build --target ci_test_gcc
ci_meson_install:
runs-on: ubuntu-26.04
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
@@ -77,7 +77,7 @@ jobs:
cmake --build build-import-split
ci_infer:
runs-on: ubuntu-26.04
runs-on: ubuntu-latest
steps:
- name: Harden Runner
uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1
@@ -106,7 +106,7 @@ jobs:
if-no-files-found: ignore
ci_static_analysis_ubuntu:
runs-on: ubuntu-26.04
runs-on: ubuntu-latest
strategy:
matrix:
target: [ci_test_single_header, ci_cppcheck, ci_cpplint, ci_reproducible_tests, ci_non_git_tests, ci_offline_testdata, ci_reuse_compliance, ci_test_valgrind]
@@ -175,7 +175,7 @@ jobs:
run: cmake --build build --target ${{ matrix.target }}
ci_test_coverage:
runs-on: ubuntu-26.04
runs-on: ubuntu-latest
steps:
- name: Harden Runner
uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1
@@ -430,7 +430,7 @@ jobs:
run: cmake --build build --target ci_nvhpc
ci_emscripten:
runs-on: ubuntu-26.04
runs-on: ubuntu-latest
steps:
- name: Harden Runner
uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1
@@ -450,7 +450,7 @@ jobs:
run: cmake --build build
ci_test_documentation:
runs-on: ubuntu-26.04
runs-on: ubuntu-latest
strategy:
matrix:
target: [ci_test_examples, ci_test_build_documentation, ci_test_documentation_mermaid]
+3 -3
View File
@@ -361,10 +361,10 @@ add_custom_target(ci_test_coverage
COMMAND ${CMAKE_COMMAND} --build ${PROJECT_BINARY_DIR}/build_coverage32
COMMAND cd ${PROJECT_BINARY_DIR}/build_coverage32 && ${CMAKE_CTEST_COMMAND} --parallel ${N} --output-on-failure
COMMAND ${LCOV_TOOL} --directory . --capture --output-file json.info --rc branch_coverage=1 --rc geninfo_unexecuted_blocks=1 --ignore-errors mismatch --ignore-errors unused --ignore-errors inconsistent
COMMAND ${LCOV_TOOL} -e json.info ${SRC_FILES} --output-file json.info.filtered --rc branch_coverage=1 --ignore-errors unused --ignore-errors inconsistent
COMMAND ${LCOV_TOOL} --directory . --capture --output-file json.info --rc branch_coverage=1 --rc geninfo_unexecuted_blocks=1 --ignore-errors mismatch --ignore-errors unused
COMMAND ${LCOV_TOOL} -e json.info ${SRC_FILES} --output-file json.info.filtered --rc branch_coverage=1 --ignore-errors unused
COMMAND ${CMAKE_SOURCE_DIR}/tests/thirdparty/imapdl/filterbr.py json.info.filtered > json.info.filtered.noexcept
COMMAND ${GENHTML_TOOL} --title "JSON for Modern C++" --legend --demangle-cpp --output-directory html --show-details --branch-coverage --ignore-errors inconsistent json.info.filtered.noexcept
COMMAND ${GENHTML_TOOL} --title "JSON for Modern C++" --legend --demangle-cpp --output-directory html --show-details --branch-coverage json.info.filtered.noexcept
COMMENT "Compile and test with coverage"
)
@@ -135,9 +135,6 @@ Note: Some modern features (like C++20 ranges or filesystem support) may be disa
headers are included.
- [x] On Windows, the library is compiled with `<Windows.h>` being included to detect and avoid common bugs (see
[`unit-windows_h.cpp`](https://github.com/nlohmann/json/blob/develop/tests/src/unit-windows_h.cpp)).
- [x] The library is compiled with Qt's keyword macros (`slots`, `signals`, `emit`, `foreach`, `forever`) being defined
to make sure it does not use them as identifiers (see
[`unit-qt_keywords.cpp`](https://github.com/nlohmann/json/blob/develop/tests/src/unit-qt_keywords.cpp)).
- [x] The library is compiled with exceptions disabled to support alternative means of error handling.
## Stable public API
+13 -13
View File
@@ -525,7 +525,7 @@ class json_pointer
{}
};
std::vector<slot> table;
std::vector<slot> slots;
std::size_t used = 0;
static std::size_t hash_of(std::size_t parent, const string_t& token) noexcept
@@ -540,19 +540,19 @@ class json_pointer
void grow()
{
std::vector<slot> old(table.size() * 2);
old.swap(table);
const std::size_t mask = table.size() - 1;
std::vector<slot> old(slots.size() * 2);
old.swap(slots);
const std::size_t mask = slots.size() - 1;
for (const auto& s : old)
{
if (s.token != nullptr)
{
std::size_t i = s.hash & mask;
while (table[i].token != nullptr)
while (slots[i].token != nullptr)
{
i = (i + 1) & mask;
}
table[i] = s;
slots[i] = s;
}
}
}
@@ -565,7 +565,7 @@ class json_pointer
// start with the empty prefix only
prefix_tree()
: table(16)
: slots(16)
, parent(1, 0)
, is_array(1, false)
{}
@@ -575,20 +575,20 @@ class json_pointer
std::size_t add_child(std::size_t id, const string_t& reference_token)
{
const std::size_t hash = hash_of(id, reference_token);
const std::size_t mask = table.size() - 1;
const std::size_t mask = slots.size() - 1;
std::size_t i = hash & mask;
while (table[i].token != nullptr)
while (slots[i].token != nullptr)
{
if (table[i].hash == hash && table[i].parent == id && *table[i].token == reference_token)
if (slots[i].hash == hash && slots[i].parent == id && *slots[i].token == reference_token)
{
return table[i].id;
return slots[i].id;
}
i = (i + 1) & mask;
}
// new prefix
const std::size_t child = parent.size();
table[i] = slot(hash, id, &reference_token, child);
slots[i] = slot(hash, id, &reference_token, child);
if (reference_token == "0")
{
is_array[id] = true;
@@ -597,7 +597,7 @@ class json_pointer
is_array.push_back(false);
// keep the load factor below 1/2
if (++used * 2 > table.size())
if (++used * 2 > slots.size())
{
grow();
}
+13 -13
View File
@@ -22595,7 +22595,7 @@ class json_pointer
{}
};
std::vector<slot> table;
std::vector<slot> slots;
std::size_t used = 0;
static std::size_t hash_of(std::size_t parent, const string_t& token) noexcept
@@ -22610,19 +22610,19 @@ class json_pointer
void grow()
{
std::vector<slot> old(table.size() * 2);
old.swap(table);
const std::size_t mask = table.size() - 1;
std::vector<slot> old(slots.size() * 2);
old.swap(slots);
const std::size_t mask = slots.size() - 1;
for (const auto& s : old)
{
if (s.token != nullptr)
{
std::size_t i = s.hash & mask;
while (table[i].token != nullptr)
while (slots[i].token != nullptr)
{
i = (i + 1) & mask;
}
table[i] = s;
slots[i] = s;
}
}
}
@@ -22635,7 +22635,7 @@ class json_pointer
// start with the empty prefix only
prefix_tree()
: table(16)
: slots(16)
, parent(1, 0)
, is_array(1, false)
{}
@@ -22645,20 +22645,20 @@ class json_pointer
std::size_t add_child(std::size_t id, const string_t& reference_token)
{
const std::size_t hash = hash_of(id, reference_token);
const std::size_t mask = table.size() - 1;
const std::size_t mask = slots.size() - 1;
std::size_t i = hash & mask;
while (table[i].token != nullptr)
while (slots[i].token != nullptr)
{
if (table[i].hash == hash && table[i].parent == id && *table[i].token == reference_token)
if (slots[i].hash == hash && slots[i].parent == id && *slots[i].token == reference_token)
{
return table[i].id;
return slots[i].id;
}
i = (i + 1) & mask;
}
// new prefix
const std::size_t child = parent.size();
table[i] = slot(hash, id, &reference_token, child);
slots[i] = slot(hash, id, &reference_token, child);
if (reference_token == "0")
{
is_array[id] = true;
@@ -22667,7 +22667,7 @@ class json_pointer
is_array.push_back(false);
// keep the load factor below 1/2
if (++used * 2 > table.size())
if (++used * 2 > slots.size())
{
grow();
}
-67
View File
@@ -1,67 +0,0 @@
// __ _____ _____ _____
// __| | __| | | | JSON for Modern C++ (supporting code)
// | | |__ | | | | | | version 3.12.0
// |_____|_____|_____|_|___| https://github.com/nlohmann/json
//
// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
// SPDX-License-Identifier: MIT
#include "doctest_compatibility.h"
// Qt defines its keywords as macros (see qtmetamacros.h and qforeach.h) unless
// QT_NO_KEYWORDS is defined, and it is common to include Qt headers before this
// library; it must therefore not use these words as identifiers (see #5865)
#define slots
#define signals public
#define emit
#define Q_FOREACH(variable, container) for (variable : container) // NOLINT(bugprone-macro-parentheses): variable is a declaration, as in Qt
#define foreach Q_FOREACH
#define forever for (;;)
#include <nlohmann/json.hpp>
using nlohmann::json;
TEST_CASE("Qt keyword macros")
{
SECTION("parse and dump")
{
const json j = json::parse(R"({"a": [1, 2.5, "x", null, true]})");
CHECK(j.dump() == R"({"a":[1,2.5,"x",null,true]})");
}
SECTION("flatten and unflatten")
{
const json j = {{"a", {{"b", {1, 2}}, {"c", "d"}}}, {"e", nullptr}};
CHECK(j.flatten().unflatten() == j);
}
SECTION("JSON pointer")
{
json j = {{"a", {{"b", {1, 2}}}}};
CHECK(j.at(json::json_pointer("/a/b/1")) == 2);
j[json::json_pointer("/a/c")] = 3;
CHECK(j.contains(json::json_pointer("/a/c")));
}
SECTION("binary formats")
{
const json j = {{"a", {1, 2, 3}}, {"b", "c"}};
CHECK(json::from_cbor(json::to_cbor(j)) == j);
CHECK(json::from_msgpack(json::to_msgpack(j)) == j);
CHECK(json::from_ubjson(json::to_ubjson(j)) == j);
CHECK(json::from_bjdata(json::to_bjdata(j)) == j);
CHECK(json::from_bson(json::to_bson(j)) == j);
}
SECTION("patch and diff")
{
const json source = {{"a", 1}, {"b", {1, 2}}};
const json target = {{"a", 2}, {"c", "d"}};
CHECK(source.patch(json::diff(source, target)) == target);
}
}
// testing for the macros also keeps -Wunused-macros from rejecting them
#if !defined(slots) || !defined(signals) || !defined(emit) || !defined(Q_FOREACH) || !defined(foreach) || !defined(forever)
#error "the Qt keyword macros must stay defined for the tests above"
#endif