amorfo77/json
1
0
Fork 0
mirror of https://github.com/nlohmann/json.git synced 2026-08-31 21:47:15 +00:00
Code Issues Packages Projects Releases Wiki Activity

Compare commits

merge into: amorfo77/json:fix/json_pointer_create_object_5357
Branches Tags
amorfo77/json:develop amorfo77/json:claude/dump-adapter-devirt amorfo77/json:claude/dump-function-performance-hcj2bl amorfo77/json:claude/parser-performance-research-4hkdf8 amorfo77/json:claude/basic-json-template-assumptions-tyc76m amorfo77/json:dependabot/pip/tools/astyle/astyle-3.6.18 amorfo77/json:gh-pages amorfo77/json:claude/issue-5387-comparison-binary amorfo77/json:claude/issue-5387-duplicate-check-bd7853 amorfo77/json:claude/todo-49-fix-bd32cc amorfo77/json:claude/issue-5340-restore-unget amorfo77/json:claude/binary-writer-output-sinks amorfo77/json:fix/json_pointer_create_object_5357 amorfo77/json:claude/ci-issues-pr-5237-drd9bw amorfo77/json:claude/code-simplification-cqwaup amorfo77/json:fix-indent_string-resize amorfo77/json:claude/todo-191-plan-508110 amorfo77/json:claude/todo-183-plan-267c0c amorfo77/json:claude/github-issue-review-b531d3 amorfo77/json:upgrade_doctest amorfo77/json:bson-sizes amorfo77/json:nvcc-image amorfo77/json:issue4755-cppcheck-warning amorfo77/json:add-ci-step-windows amorfo77/json:master amorfo77/json:issue4552-ignore amorfo77/json:issue3989 amorfo77/json:clang-tidy amorfo77/json:clang-format amorfo77/json:bon8 amorfo77/json:coverity_scan amorfo77/json:issue2615 amorfo77/json:feature/optional amorfo77/json:issue2228
amorfo77/json:v3.12.0 amorfo77/json:v3.11.3 amorfo77/json:v3.11.2 amorfo77/json:v3.11.1 amorfo77/json:v3.11.0 amorfo77/json:v3.10.5 amorfo77/json:v3.10.4 amorfo77/json:v3.10.3 amorfo77/json:v3.10.2 amorfo77/json:v3.10.1 amorfo77/json:v3.10.0 amorfo77/json:v3.9.1 amorfo77/json:v3.9.0 amorfo77/json:v3.8.0 amorfo77/json:v3.7.3 amorfo77/json:v3.7.2 amorfo77/json:v3.7.1 amorfo77/json:v3.7.0 amorfo77/json:v3.6.1 amorfo77/json:3.6.1 amorfo77/json:v3.6.0 amorfo77/json:v3.5.0 amorfo77/json:v3.4.0 amorfo77/json:v3.3.0 amorfo77/json:3.3.0 amorfo77/json:v3.2.0 amorfo77/json:3.2.0 amorfo77/json:v3.1.2 amorfo77/json:3.1.2 amorfo77/json:v3.1.1 amorfo77/json:v3.1.0 amorfo77/json:3.1.0 amorfo77/json:v3.0.1 amorfo77/json:3.0.1 amorfo77/json:v3.0.0 amorfo77/json:3.0.0 amorfo77/json:v2.1.1 amorfo77/json:2.1.1 amorfo77/json:v2.1.0 amorfo77/json:2.1.0 amorfo77/json:v2.0.10 amorfo77/json:2.0.10 amorfo77/json:v2.0.9 amorfo77/json:2.0.9 amorfo77/json:v2.0.8 amorfo77/json:2.0.8 amorfo77/json:v2.0.7 amorfo77/json:v2.0.6 amorfo77/json:v2.0.5 amorfo77/json:v2.0.4 amorfo77/json:v2.0.3 amorfo77/json:v2.0.2 amorfo77/json:v2.0.1 amorfo77/json:v2.0.0 amorfo77/json:v1.1.0 amorfo77/json:v1.0.0 amorfo77/json:v1.0.0-rc1
..
pull from: amorfo77/json:claude/dump-adapter-devirt
Branches Tags
amorfo77/json:claude/dump-adapter-devirt amorfo77/json:claude/dump-function-performance-hcj2bl amorfo77/json:claude/parser-performance-research-4hkdf8 amorfo77/json:develop amorfo77/json:claude/basic-json-template-assumptions-tyc76m amorfo77/json:dependabot/pip/tools/astyle/astyle-3.6.18 amorfo77/json:gh-pages amorfo77/json:claude/issue-5387-comparison-binary amorfo77/json:claude/issue-5387-duplicate-check-bd7853 amorfo77/json:claude/todo-49-fix-bd32cc amorfo77/json:claude/issue-5340-restore-unget amorfo77/json:claude/binary-writer-output-sinks amorfo77/json:fix/json_pointer_create_object_5357 amorfo77/json:claude/ci-issues-pr-5237-drd9bw amorfo77/json:claude/code-simplification-cqwaup amorfo77/json:fix-indent_string-resize amorfo77/json:claude/todo-191-plan-508110 amorfo77/json:claude/todo-183-plan-267c0c amorfo77/json:claude/github-issue-review-b531d3 amorfo77/json:upgrade_doctest amorfo77/json:bson-sizes amorfo77/json:nvcc-image amorfo77/json:issue4755-cppcheck-warning amorfo77/json:add-ci-step-windows amorfo77/json:master amorfo77/json:issue4552-ignore amorfo77/json:issue3989 amorfo77/json:clang-tidy amorfo77/json:clang-format amorfo77/json:bon8 amorfo77/json:coverity_scan amorfo77/json:issue2615 amorfo77/json:feature/optional amorfo77/json:issue2228
amorfo77/json:v3.12.0 amorfo77/json:v3.11.3 amorfo77/json:v3.11.2 amorfo77/json:v3.11.1 amorfo77/json:v3.11.0 amorfo77/json:v3.10.5 amorfo77/json:v3.10.4 amorfo77/json:v3.10.3 amorfo77/json:v3.10.2 amorfo77/json:v3.10.1 amorfo77/json:v3.10.0 amorfo77/json:v3.9.1 amorfo77/json:v3.9.0 amorfo77/json:v3.8.0 amorfo77/json:v3.7.3 amorfo77/json:v3.7.2 amorfo77/json:v3.7.1 amorfo77/json:v3.7.0 amorfo77/json:v3.6.1 amorfo77/json:3.6.1 amorfo77/json:v3.6.0 amorfo77/json:v3.5.0 amorfo77/json:v3.4.0 amorfo77/json:v3.3.0 amorfo77/json:3.3.0 amorfo77/json:v3.2.0 amorfo77/json:3.2.0 amorfo77/json:v3.1.2 amorfo77/json:3.1.2 amorfo77/json:v3.1.1 amorfo77/json:v3.1.0 amorfo77/json:3.1.0 amorfo77/json:v3.0.1 amorfo77/json:3.0.1 amorfo77/json:v3.0.0 amorfo77/json:3.0.0 amorfo77/json:v2.1.1 amorfo77/json:2.1.1 amorfo77/json:v2.1.0 amorfo77/json:2.1.0 amorfo77/json:v2.0.10 amorfo77/json:2.0.10 amorfo77/json:v2.0.9 amorfo77/json:2.0.9 amorfo77/json:v2.0.8 amorfo77/json:2.0.8 amorfo77/json:v2.0.7 amorfo77/json:v2.0.6 amorfo77/json:v2.0.5 amorfo77/json:v2.0.4 amorfo77/json:v2.0.3 amorfo77/json:v2.0.2 amorfo77/json:v2.0.1 amorfo77/json:v2.0.0 amorfo77/json:v1.1.0 amorfo77/json:v1.0.0 amorfo77/json:v1.0.0-rc1
80 Commits
fix/json_pointer_create_object_5357 .. claude/dump-adapter-devirt
Author SHA1 Message Date
Claude b8bc5e0d59 Update the serializer test for the non-owning adapter ctor 
check_escaped constructed the serializer with output_adapter<char>(ss),
which produced the old owning output_adapter_t. The ctor now takes a
non-owning output_adapter_protocol<char>*, so build the concrete
output_stream_adapter on the stack and pass its address, matching how
dump() and operator<< now call it.

Signed-off-by: Claude <noreply@anthropic.com>
 2026-08-31 23:25:06 +02:00
Claude 2992ca9f08 Stop dump() from heap-allocating its output adapter per call 
The serializer held its output sink as output_adapter_t<char>
(a std::shared_ptr<output_adapter_protocol<char>>), which dump() and
operator<< built via make_shared -- one heap allocation per call for a
sink that only wraps a reference to the caller's string or stream.

Hold the sink as a non-owning output_adapter_protocol<char>* instead and
construct the concrete adapter on the stack at the call site. The write
path (o->write_characters) is unchanged, so output is byte-for-byte
identical; a compact dump() of a small object drops from 2 heap
allocations to 1 (only the returned string remains), ~3% faster.

Completes the per-call allocation cleanup on this branch, which already
removed the indent_string buffer (both were reported in #5413).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L1oJ2ggRHS37zeVe94QTA1
Signed-off-by: Claude <noreply@anthropic.com>
 2026-08-31 23:25:06 +02:00
Niels Lohmann 120429dee1 Parenthesize the reserve arithmetic in the deep-nesting test 
clang-tidy's readability-math-missing-parentheses wants the multiplication
spelled out in reserve(6 * depth + 1), and CI treats its warnings as errors.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:05 +02:00
Niels Lohmann cb1aa15abb Write a byte without walking a pointer over the buffer 
clang-tidy's misc-const-correctness reads the pointer dump_byte advanced over
the write buffer as one whose pointee could be const. Index the buffer
instead, which says the same thing without a raw pointer at all.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:05 +02:00
Niels Lohmann 975ad592a2 Fold ensure_ascii into the escaper and write bytes without dump_integer 
Two hot spots that the write buffer and the bulk scanner left behind.

dump_escaped took ensure_ascii as a runtime flag and tested it inside the
loop, once per character run, although it cannot change while a string is
written. It is now a template parameter, dispatched once per string, which
folds the choice of scanner and lets each of the two be inlined into a loop
of its own. This is the hottest loop in the serializer: it runs over every
string and every object key.

A binary value's bytes went through dump_integer, which counts digits and
does 64-bit arithmetic for a number that is always in [0, 255]. dump_byte
writes the three digits it takes at most straight into the write buffer
instead. Any byte type that is not a plain unsigned byte is still left to
dump_integer, whose representation of it may differ.

Measured against the previous commit (medians of 9 interleaved runs, clang
-O3): binary values -33.8%, dense CJK with ensure_ascii -20.6%, key-heavy
objects -17.8%, deeply nested pretty output -17.9%, dense CJK without
ensure_ascii -11.8%, object-heavy documents -9.3% compact and -9.5% pretty,
a small value dumped in a loop -21.4%, wide objects -2.3%. Arrays of plain
ASCII strings measured 3.5% to 4.2% slower, the one shape that loses; number
and integer arrays are unchanged.

Also tried and dropped: leaving the write and string buffers uninitialized
rather than zeroing 1.5 KB per dump() call. It is worth -30% on small values,
but two nearly identical string workloads moved 18% apart in opposite
directions, so the measurements did not support it.

The output is unchanged for every value: the differential now also covers
every one of the 256 byte values, alone and together, in both binary layouts.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:05 +02:00
Niels Lohmann bd032281fd Bound the descent of dump() 
Serializing a container serializes its elements, so dump() descended into one
call per nesting level. A value nested deeply enough exhausted the call stack
and terminated the process with a segmentation fault - no exception, nothing
the caller could catch. Parsing such a value works, as the parser is
iterative, and so does destroying one, as #1436 made destruction iterative.

Bound how far the descent goes rather than take the call stack away from it.
The first 128 levels are written by exactly the code that always wrote them,
and only below that does dump_iteratively write out what is left, keeping the
containers it has entered on an explicit stack. Serializing can therefore no
longer exhaust the stack, however deeply a value is nested, while a value
nested less deeply than the bound pays only for one comparison per container.

Writing every value that way instead measured between 2% and 20% slower - 20%
on object-heavy documents - which is why the descent is kept for all but the
values that cannot afford it. The bound costs nothing measurable: between
-1.4% and +1.2% across compact and pretty output of number, integer, string,
object-heavy, wide-object and deeply nested documents.

The output is unchanged for every value. Both ways of writing a container
emit the separator in front of every element but the first, rather than
after every element but the last, which puts exactly one between each pair
and none at the end.

This fixes #5387 for dump(). The copy constructor is fixed in #5389.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:05 +02:00
Niels Lohmann 2df81a33aa Silence avoid-c-arrays on put_literal's array reference 
clang-tidy flags the reference-to-array parameter under
cppcoreguidelines/hicpp/modernize-avoid-c-arrays, and the CI treats warnings as
errors. Binding to the array is the whole point here - it is what lets the
length be deduced from the literal instead of hand-written at the call site - so
suppress it the same way from_json(), to_json() and get_to() already suppress it
for their own T (&arr)[N] parameters.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:05 +02:00
Niels Lohmann f0e591465d Tighten the write-buffer helpers after review 
More of @gregmarr's review on the put_* split:

- Reattach the put_chars() doc comment, which the new helpers had been
  inserted in front of, leaving it describing put_indent().

- Compute the literal length once in put_literal() instead of spelling N - 1
  at each use.

- Add put_string(str, start, end), which keeps the pointer arithmetic and the
  bounds assertions inside the function instead of at the call site. With
  dump_float()'s to_chars() output moved onto put_buffer() as well, put_chars()
  now has no callers outside put_string()/put_buffer(): nothing passes a bare
  pointer and a count any more.

- Carry the indentation as std::size_t rather than unsigned int. It is a size,
  it is compared and combined with buffer sizes throughout, and the casts in
  put_indent() disappear. next_indent() keeps its assertion, which is far
  harder to trip on a 64-bit size_t but still reachable where that is 32 bits.

No output change: pretty and compact dumps, binary values included, are
byte-identical to develop.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:05 +02:00
Niels Lohmann bfb780b90d Fill the indentation buffer once instead of once per flush 
@gregmarr's point on the fill-and-flush loop: flushing does not disturb what
the write buffer holds, so an indentation spanning several buffer-fulls only
has to be written into the buffer once and can then be handed to the adapter
as many times as needed. The loop re-filled it every time, doing work it
already knew was there.

put_indent() now fills the room left in the buffer, and if anything remains,
flushes, fills the buffer once, and re-flushes that same content. It also
returns early for a zero-width indentation, which is what the closing brace of
every outermost value asks for.

Measured over a dump(), counting memset calls and bytes inside put_indent:

    indent       before              after
         4       1 call /     4 B    1 call /     4 B
      2000       2 calls /  2000 B   2 calls /  2046 B
    100000      98 calls / 100000 B  2 calls /  2046 B

The wide case is now constant work rather than proportional to the indentation
width; ordinary widths are unchanged. Tests extended to cover several whole
buffer-fulls and an exact multiple of the buffer size.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:05 +02:00
Niels Lohmann b2d2916064 Split the write-buffer helpers and write indentation directly 
Follow-up to @gregmarr's review: put_chars() was doing four unrelated jobs, so
give the two that can be made safe their own entry points.

- put_literal(): takes the literal by reference and deduces the length from the
  array bound, so the 27 hand-counted lengths at the call sites can no longer
  drift from the literals they describe. A literal is checked at compile time to
  fit the buffer, so this path needs no write-through branch.

- put_buffer(): takes the fixed-size buffer itself rather than a bare pointer,
  so the length can be checked against the buffer's own bound.

- put_indent(): memsets the indentation into the write buffer, filling and
  flushing it as needed. This removes indent_string entirely, and with it both
  bugs of #5186: the indentation string was grown by doubling, which is not
  enough when indent_step more than doubles it (a heap over-read - dump(2000)
  read 2000 bytes out of a 1024-byte string), and the grown part was filled with
  a space instead of the configured indent_char. next_indent() keeps that PR's
  assertion against the unsigned indentation accumulation wrapping on deep
  nesting.

put_chars() keeps the two cases that are genuinely a pointer and a count: the
run-length copies out of the string being escaped, and to_chars() output.

Tests cover an indent_step wider than the write buffer, a non-space indentation
character past the old growth point, and nesting whose accumulated indentation
spans several buffer-fulls. All three fail against develop.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:05 +02:00
Niels LohmannandClaude Opus 4.8 6ac71e00ff Avoid deep recursion in serialization write-buffer test 
The "many small structural writes exceed the write buffer" subcase built
a 1100-deep nested array and dumped it to force >1024 consecutive
single-character writes through put_char (exercising the write buffer's
flush-when-full branch). dump() recurses per nesting level, so on MSVC
debug builds (smaller default stack, larger frames) this overflowed the
stack and crashed test-serialization; Linux/macOS have enough headroom to
hide it.

Replace the nesting with a flat array of 500 empty strings. Each element
emits '"', '"', ',' via put_char, so the dump is a long run of
single-character writes (1501 bytes > the 1024-byte buffer) at nesting
depth two, hitting the same flush branch without deep recursion. Library
code is unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:05 +02:00
Niels LohmannandClaude Opus 4.8 151421828d Flush serializer buffer in dump_escaped unit test 
test-convenience failed (macOS finished first; the failure is
platform-independent) because check_escaped() calls the internal
serializer::dump_escaped() directly and then reads the output stream.
Since dump_escaped() now writes into the serializer's internal write
buffer, the bytes were still buffered and the stream was empty.

Expose flush() under JSON_PRIVATE_UNLESS_TESTED (same visibility as
dump_escaped) and flush in check_escaped() before inspecting the output.
Per-string flushing inside dump_escaped() was rejected on purpose: it
would defeat the buffering that makes object/array-heavy dumps faster.
Library behavior is unchanged (flush()'s body is identical; only its
access label moved).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:05 +02:00
Niels LohmannandClaude Opus 4.8 aab82fec94 Buffer serializer output and add ensure_ascii string fast path 
Two further serialization speedups on top of the ensure_ascii=false bulk
copy, both reusing the SWAR primitives in detail/input/string_scan.hpp.

1. Internal write buffer (devirtualization). Every structural character
   ('{', '"', ',', ...) previously went straight to the output adapter
   through a virtual call. Route all writes through put_char/put_chars
   into a 1 KiB buffer that flushes in bulk; the public dump() flushes
   once the top-level value is done (the recursive worker is split out as
   dump_internal). Runs larger than the buffer are written straight
   through, so large payloads are not copied twice. This is the dominant
   cost for object/array-heavy values.

2. ensure_ascii fast path. dump_escaped previously ran the UTF-8 DFA over
   every byte when escaping non-ASCII. Add find_ascii_copyable_run() (a
   SWAR scan stopping at '"', '\\', < 0x20, 0x7F, and >= 0x80) so runs of
   printable ASCII are bulk-copied, with the byte path handling each
   escape/non-ASCII byte exactly as before.

Behavior is unchanged: dump output is byte-for-byte identical to the
previous implementation across ~20k randomized byte strings plus curated
edge cases (all escapes, control chars, 0x7F, valid multibyte,
surrogates, overlong, truncated), for object/array/pretty output, both
ensure_ascii settings, and all three error handlers, in C++11/17/20 at
-O2/-O3. New unit tests cover the buffer flush boundaries, the escape and
0x7F handling, multibyte under both settings, and invalid-UTF-8 handling.

Throughput (g++ -O3, vs the ensure_ascii=false-only baseline):
  long ASCII, ensure_ascii=0   4.2x
  long ASCII, ensure_ascii=1   4.1x
  twitter-like objects         2.7x
  dense CJK                    1.8x

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:05 +02:00
Niels LohmannandClaude Opus 4.8 41d793b347 Add SWAR bulk fast path to string serialization (dump_escaped) 
When ensure_ascii is false, dump_escaped previously ran every byte of
every string and object key through the UTF-8 DFA decoder, even for the
common case of ordinary text with nothing to escape. This mirrors the
per-byte cost the parser had before the contiguous fast paths.

At a character boundary, bulk-copy the longest run of bytes that need no
escaping using string_bulk_run() - the same SWAR scanner and UTF-8 bulk
validator the lexer's contiguous path uses - and only fall back to the
byte-at-a-time DFA loop for the first byte that needs individual handling
(a quote, backslash, control character, or ill-formed/truncated UTF-8).
Because every "hard" or invalid byte is still processed by the unchanged
byte path, escaping output and error handling (including strict-mode
error 316 position and message) are byte-identical to before.

The ensure_ascii=true path is unchanged: it must escape non-ASCII and
0x7F, which string_bulk_run does not stop on, so a separate predicate
would be needed for it.

Verified byte-for-byte identical dump output against the pre-change
implementation across ~20k randomized byte strings plus curated edge
cases (all escapes, control chars, valid multibyte, surrogates,
overlong, truncated sequences) for both ensure_ascii settings and all
three error handlers, in C++11/17/20 at -O2/-O3.

Throughput (g++ -O3, ensure_ascii=false, vs pre-change):
  long ASCII strings   4.2x
  twitter-like objects 2.3x
  dense CJK            1.4x  (further headroom with JSON_USE_SIMDUTF)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:05 +02:00
Niels LohmannandClaude 0386005cbd Warn when JSON_TestSimdutf cannot reach the simdutf backend 
simdutf needs C++17: without it the dependency does not even compile, and with
a C++17 compiler but no C++17-or-later standard under test it builds and then
goes unused. Either way the option silently did nothing useful, or broke the
configure step outright.

Resolve the tested standards first, then check them: when none of them can
reach simdutf, skip the dependency and say so, naming which of the two reasons
applies and how to fix it. The tests then run against the scalar validator,
which is what would have happened anyway.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:02 +02:00
Niels LohmannandClaude c9b9d7fc74 Test the simdutf backend in CI 
JSON_USE_SIMDUTF was documented and shipped but never compiled by anything in
the repository, so nothing held the backend to the behavior the docs promise.

Add JSON_TestSimdutf (OFF by default), which fetches simdutf and defines
JSON_USE_SIMDUTF for every test target, and a ci_test_simdutf target that runs
the whole suite in that configuration. Because simdutf needs C++17, the suite is
built at C++11 as well, so one job covers both the scalar fallback with the
macro defined and simdutf itself.

The dependency hangs off test_main, whose usage requirements every test target
inherits. The library target and the installed CMake package are deliberately
untouched: making nlohmann_json link simdutf would put a find_dependency() in
the exported package, which is a separate decision.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:02 +02:00
Niels LohmannandClaude 45f1f8f17a Only compile the simdutf backend from C++17 on 
simdutf.h rejects anything below C++17 with an #error, so defining
JSON_USE_SIMDUTF in a C++11 or C++14 translation unit did not fail with a
message about simdutf being unavailable - it failed to compile at all, taking
the library's C++11 support with it. Nothing caught this because no build ever
compiled that path.

Gate the include and both uses on JSON_HAS_CPP_17, the same way number_parse.hpp
gates std::from_chars. Below C++17 the macro now has no effect and the scalar
validator runs; it accepts and rejects exactly the same input, so the macro is
safe to set project-wide even when some translation units use an older standard.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:02 +02:00
Niels LohmannandClaude fbf3f28b41 Require a container's value_type to match what data() points at 
is_contiguous_byte_container accepted any type with a data() returning a
pointer to a single-byte integral plus a size(). That is duck typing: the two
members say nothing about size() counting the units data() points at.

A type where it does not - fixed-size records, say - was routed to the
pointer-based adapter and parsed as [data(), data() + size()) bytes, silently
truncating input the iterator-based adapter had read in full:

    struct record_buffer {
        using value_type = std::array<char, 4>;
        std::string bytes;
        const char* data() const;                        // raw bytes
        std::size_t size() const;                        // in records
        const char* begin() const; const char* end() const;
    };
    json::parse(record_buffer{"[1,2,3,4,5]"});           // parse error at column 3

Requiring the container's own value_type to be that same element type ties the
two together. Every contiguous standard container satisfies it, so std::string,
std::vector<char>, std::array<char, N> and std::string_view keep the fast path;
anything else falls back to the iterator-based adapter, which is always correct.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:02 +02:00
Niels LohmannandClaude 3e30fc3bf1 Do not repeat the integer conversion that just failed 
scan_number_bulk_contiguous() converts an integer token straight from the input
buffer. When the value does not fit, it materializes token_buffer and calls
convert_number(), which tried the very same integer conversion again before
falling back to floating point.

Recording the outcome in number_type skips the second attempt. The resulting
token type and value are unchanged: convert_number() reached the float tail
either way.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:02 +02:00
Niels LohmannandClaude ecd8003be1 Pin the number error position against a non-newline terminator 
The comment claimed the reported column is the one the offending token starts
at. It is the column reached after the token's last character - which is the
actual point of the unget() change: a number terminated by a newline now
reports what the same number terminated by a space always did.

Assert that equality directly, and add a multi-character token where the start
and end columns differ, so the invariant cannot be read off a single-character
example.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:02 +02:00
Niels LohmannandClaude f334d0e433 Address review comments on the simdutf backend 
string_bulk_run() had the same `return scalar_string_bulk_run(...)` in both
arms of the `#if`. The simdutf arm already falls through when validation
fails, so a single return after the `#endif` says the same thing.

The JSON_USE_SIMDUTF example showed `#include <simdutf.h>`, which
string_scan.hpp already does under the same guard; users only have to put the
header on the include path and link the library, not include it themselves.

Set the version history entry to 3.13.0, matching the other macro pages
documenting unreleased features.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MXDi7NTMKAmoArUZSKMc4T
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:02 +02:00
Niels Lohmann f800004f49 Reattach convert_number's documentation 
@gregmarr spotted that convert_integer() was inserted between convert_number()
and its doc block, leaving convert_integer() with two stacked blocks and
convert_number() with none. Comment only; no code change.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:02 +02:00
Niels Lohmann 22e4061d03 Avoid escaped literals in the counted-iterator diagnostics list 
clang-tidy reads "[\"\\ud834\"]" as a literal better written raw, and the
two literals written next to each other in "[\"a\x01""b\"]" as a missing
comma. The concatenation was there to stop the hex escape swallowing the
following character; build those documents from explicit bytes instead and
use raw strings elsewhere. The byte sequences are unchanged.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:02 +02:00
Niels Lohmann 3f7760ebba Satisfy clang-tidy in the new bulk scanner tests 
- give the helper lambdas an explicit std::string return type and return
  braced initializer lists (modernize-return-braced-init-list)
- replace the C-style array of test cases with a std::vector
  (modernize-avoid-c-arrays)
- silence pro-type-member-init on the two brace-initialized aggregates;
  default member initializers would stop them being aggregates in C++11

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:02 +02:00
Niels Lohmann 22d05f5ef5 Amalgamate source code 
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:02 +02:00
Niels Lohmann 0197a0e98b Gate the C++20 iterator classification on JSON_HAS_RANGES 
Making supports_bulk_scan depend on iterator_is_contiguous meant the trait
is now instantiated for every adapter, not only when get_elements() is
called. On standard libraries with an incomplete <ranges> that is fatal:
libstdc++ 10 evaluates std::contiguous_iterator<std::counted_iterator<T*>>
by calling std::to_address, which needs an operator-> its counted_iterator
does not have, so satisfaction checking is a hard error rather than false.
Reported by clang 14 + libstdc++ 10.

JSON_HAS_RANGES already encodes exactly this ("libstdc++ < 11 has incomplete
C++20 ranges", #4440), so require it for the C++20 branch. Affected
toolchains fall back to the pointer-only test and the byte-at-a-time
scanner, which parses identically, just without the bulk fast paths.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:02 +02:00
Niels Lohmann 56d6b7b36e Fix shadowed locals and guard the exception-dependent tests 
Two problems in the tests added for the bulk scanners, both found by CI:

- the inner `const json j` in the counted-iterator diagnostics shadowed the
  one declared at test-case scope, which -Wshadow rejects on GCC and clang
  and C4456 rejects on MSVC under /WX; rename them
- the new parity checks parse deliberately invalid input, which calls
  std::abort() rather than throwing when JSON_NOEXCEPTION is defined, so
  they would have crashed the no-exception build; guard them the way the
  other tests do

json::accept() does not abort, so the UTF-8 range assertions stay compiled
without exceptions and keep covering validate_one_utf8() there.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:02 +02:00
Niels Lohmann 89334dd55f Format the new string fast path test with astyle 
The pinned astyle expands a braced-init-list used as a range-for range onto
several lines; hoist the two offsets into a named vector instead, which reads
better and leaves nothing for astyle to reformat.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:02 +02:00
Niels Lohmann 4170fe0b39 Cover the bulk string and UTF-8 scanners 
These paths had no dedicated tests and rested on differential fuzzing only.
Add three sections, all comparing the contiguous scanner against the
byte-at-a-time one on the parsed value and on the exact error message:

- every string of length 1..3 over an alphabet of ordinary ASCII, both
  specials, a control byte, escape characters, UTF-8 lead and continuation
  bytes, and a byte that is never valid - each at offset 0 and offset 9, so
  the bulk scanner sees them with and without a run behind them
- every kind of run-ending byte at each offset across two 8-byte SWAR words,
  so multibyte sequences also straddle the word boundary
- the boundaries of every range validate_one_utf8() recognizes: shortest and
  longest encodings, overlongs, both ends of the surrogate block, U+10FFFF
  and just past it, and truncated sequences

Verified to fail if the bulk validator accepts surrogates, and if the SWAR
word test stops detecting control characters.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:02 +02:00
Niels Lohmann 9e4b3b128e Amalgamate source code 
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:02 +02:00
Niels Lohmann 34c58fa8d2 Skip token_buffer for integers on the contiguous path 
An integer token does not need token_buffer: the number_integer and
number_unsigned SAX callbacks take only the value, and the overflow
diagnostic rebuilds the text from the input via get_token_string(). Convert
straight from the input buffer and materialize the token only for the
floating-point tail, which still needs a NUL-terminated buffer for strtod.

JSON_DIAGNOSTIC_POSITIONS derives a number's start position from
get_string().size(), so the copy is kept when that is enabled.

The integer dispatch is factored into convert_integer() and shared with
convert_number(), so both scanners keep using one implementation.

Integer-heavy input, 400k values, -O3:

              parse    accept
  gcc 16      +14%     +23%
  clang       +15%     +22%

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:02 +02:00
Niels Lohmann 4fc04afc8f Lock the two number grammars together with a parity test 
The JSON number grammar is encoded twice: as the scan_number() state machine
and as the contiguous fast path. The fast path declining on anything it does
not recognize keeps most divergence harmless, but if it ever accepted
something the state machine rejects the result would be a silent correctness
bug, and the existing test only pinned a hand-written list of numbers.

Enumerate every string of length 1..4 over "01.eE+-" (2800 tokens) and
require both paths to agree on the parsed value and on the exact error
message. Verified to fail if the fast path's grammar is perturbed.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:02 +02:00
Niels Lohmann 27f5812bc5 Do not discard the parse result in the error position check 
json::parse is declared warn_unused_result, and CHECK_THROWS_WITH_AS
evaluates its expression as a discarded statement, so the assertion broke
the -Werror builds (GCC -Werror=unused-result, MSVC C4834 under /WX).
Compare against the helper that already captures the message instead.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:02 +02:00
Niels Lohmann 079c297fa7 Fix the SentinelType example on iterator_input_adapter 
The comment offered "a C++20 sentinel or counted_iterator" as examples of a
SentinelType, but std::counted_iterator is the IteratorType - the sentinel it
pairs with is std::default_sentinel_t. #5268 corrected the same wording in the
API documentation and left the code comment behind.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:02 +02:00
Niels Lohmann 0f8724d646 Cover the counted-iterator bulk scan paths 
Sized sentinels newly reach the bulk string/number scanners and the
seek-based token reconstruction, so exercise both:

- diagnostics that quote the offending token, which are rebuilt from the
  consumed input via copy_consumed_range()
- inputs whose count ends before the underlying buffer does, including a
  closing quote that exists only behind the count, a cut inside an 8-byte
  SWAR stride, and a cut inside a UTF-8 sequence

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:02 +02:00
Niels Lohmann 21b6fec01f Amalgamate source code 
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:02 +02:00
Niels Lohmann 22688bf40d Document JSON_USE_SIMDUTF on the macro overview page 
The macro was only listed in the API macro index; add it to the supported
macros overview alongside the other JSON_USE_* macros, and note that it
selects between two definitions of the same inline function and so must be
defined identically in every translation unit.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:02 +02:00
Niels Lohmann f7b068eff6 Extend the bulk scan fast paths to sized sentinels 
supports_bulk_scan required IteratorType and SentinelType to be the same
type, which excluded std::counted_iterator paired with std::default_sentinel_t
- the combination #5268 had already enabled for the memcpy fast path. Such
input fell back to the byte-at-a-time scanner even though it is contiguous
and its remaining length is computable in O(1).

Factor the "distance is computable in O(1)" test into sentinel_is_sized and
use it for iterator_is_contiguous, supports_seek, and supports_bulk_scan
alike, and share the std::ranges::distance/std::distance dispatch through a
remaining_count() helper.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:02 +02:00
Niels Lohmann c74e7a23aa Restore the column when ungetting a newline 
The contiguous number fast path never reads the character that terminates
a number token, while scan_number() reads it and then ungets it. When that
character is a newline, get() has already cleared chars_read_current_line,
and unget() could only restore lines_read - leaving the column at 0. The
two paths therefore reported different columns for the same document:

    json::parse("[01\n]")      -> line 1, column 3
    json::parse(stringstream)  -> line 1, column 0

Remember the column the newline was read at so unget() can restore it.
Both paths now report the position the offending token actually starts at,
which also fixes the pre-existing column-0 artifact for streaming input.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:02 +02:00
Niels LohmannandClaude Opus 4.8 80dfbc3a30 Guard <charconv> include with __has_include for GCC 7 
GCC 7 sets __cplusplus to the C++17 value under -std=gnu++1z, so
JSON_HAS_CPP_17 is defined, but its libstdc++ ships no <charconv> header
(added in GCC 8; floating-point from_chars in GCC 11). The unconditional
"#if defined(JSON_HAS_CPP_17) #include <charconv>" therefore failed to
compile there: "fatal error: charconv: No such file or directory" in the
ci_test_compilers_gcc (7) job.

Wrap the include in __has_include(<charconv>), mirroring the library's
existing handling of <version> and <filesystem> in macro_scope.hpp. When
the header is absent, __cpp_lib_to_chars stays undefined and
parse_float_from_chars() takes its scalar fallback, so the from_chars use
site (already gated on __cpp_lib_to_chars) is never reached. GCC 8-10,
which have <charconv> but no floating-point from_chars, are unaffected:
they include the header but still take the fallback. GCC 11+ is unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:02 +02:00
Niels LohmannandClaude Opus 4.8 ea3dd23f29 Disable Clinger float fast path under extended FP precision (x87) 
The contiguous number fast path uses a Clinger-style exact algorithm
(significand * 10^scale in double arithmetic), which is only correctly
rounded when double operations are evaluated in true 53-bit precision.
On the x87 FPU used by 32-bit x86 (FLT_EVAL_METHOD == 2) the single
multiply/divide is computed in 80-bit and then double-rounded to double,
so a small fraction of values land 1 ULP off.

This surfaced as test-cbor_cpp11 and test-msgpack_cpp11 failing on the
mingw (x86) job for regression/floats.json: the C++17 builds pass because
they take the correctly-rounded std::from_chars path, while C++11 falls
back to parse_float_fast(). A 5M-sample check over shortest round-trip
decimals reproduces it: 0 divergences with 53-bit doubles, ~1 in 25 000
with 80-bit intermediates; declining to std::strtod fixes all of them.

Guard parse_float_fast() on FLT_EVAL_METHOD so it declines whenever the
platform evaluates doubles in extended precision, letting the caller use
the correctly-rounded std::from_chars / std::strtod path instead. On
mainstream x86-64/ARM64 (FLT_EVAL_METHOD == 0) the fast path is unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:02 +02:00
Niels LohmannandClaude Opus 4.8 6a0c3ee9f1 Guard from_chars use on JSON_HAS_CPP_17, not just __cpp_lib_to_chars 
libstdc++ 15 defines __cpp_lib_to_chars even in C++14 mode (via
bits/version.h pulled in by other headers), but <charconv> is only included
under JSON_HAS_CPP_17. That made parse_float_from_chars() reference
std::from_chars without the header in C++14 builds, breaking gcc-latest,
icpx, and the offline-testdata jobs.

Gate the use on JSON_HAS_CPP_17 && __cpp_lib_to_chars so it matches the
include condition exactly; C++11/14 always take the scalar fallback.
Verified by forcing __cpp_lib_to_chars in a C++14 build: the guard
suppresses std::from_chars and it compiles. C++17 behavior is unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:02 +02:00
Niels LohmannandClaude Opus 4.8 4dfa01c07e Use std::from_chars (Eisel-Lemire) for float conversion when available 
The Clinger fast path is exact only for the "easy" subset (<=19 significant
digits, |exp10| <= 22); high-precision and scientific floats fall through to
strtod, where the failed Clinger attempt actually makes parsing a net loss.
std::from_chars implements the Eisel-Lemire algorithm in modern standard
libraries: locale-independent, correctly rounded, and fast over the whole
value range.

convert_number() now tries parse_float_from_chars() first (guarded by
__cpp_lib_to_chars, so C++11 and libc++-without-float-support keep the
Clinger + strtod path unchanged), then Clinger, then strtof. from_chars is
used only when it consumes the entire token; a partial parse means a non-'.'
locale decimal point, and an under-/overflow (result_out_of_range) also
declines - in both cases the existing strtod fallback supplies the exact
value and the well-defined +/-inf/0 the parser expects, side-stepping the
P4168 divergence between implementations. float and long double now get the
fast path too (Clinger was double-only).

Measured, C++17, g++ 13 -O3, json::parse/accept:
  - canada-style floats: ~unchanged (Clinger already covered them)
  - high-precision (17 digits):  parse 2.1x, accept 2.5x
  - scientific (17 digits + exp): parse 3.6x, accept 4.1x

Verified: C++11 (Clinger/strtod) and C++17 (from_chars) parse every value -
including subnormals, boundary values, and 1e9999/1e-9999 over-/underflow -
to bit-identical results; 2M number-fuzz clean; conversions/deserialization/
locale/number-fast-path suites pass in both C++11 and C++17; clang-tidy
clean; warning-clean on g++ and clang in C++11/17/20.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:02 +02:00
Niels LohmannandClaude Opus 4.8 980b5f344d Satisfy clang-tidy: parenthesize math and drop unused forwarding reference 
The CI clang-tidy (newer than the locally available version) reported two
additional checks on the new code:

- readability-math-missing-parentheses: parenthesize the (a * b) + c digit
  accumulations in number_parse.hpp.
- cppcoreguidelines-missing-std-forward: the contiguous-byte-container
  input_adapter overload took a forwarding reference but only reads
  data()/size() and never forwards it. It is already disjoint from the
  generic container overload via SFINAE, so a plain const& is correct and
  clearer (and keeps the container alive for the whole parse just as before).

No behavior change; char_type and routing are unchanged (std::string and
std::vector<std::uint8_t> still take the pointer adapter with char/uint8_t
char_type), CBOR/MsgPack round-trips and the 2M number fuzz still pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:02 +02:00
Niels LohmannandClaude Opus 4.8 7a522b00dc Fix number fast path for custom string types without assign() 
The contiguous number fast path materialized token_buffer with
token_buffer.assign(data, len), but string_t is only required to provide
the minimal interface the rest of the lexer uses (push_back, append,
clear, operator[], ...). Custom string types such as the test's alt_string
do not implement assign(), so scan_number_bulk_contiguous() failed to
compile for them (unit-alt-string), breaking the gcc/clang standards and
old-compiler CI jobs.

reset() already clears token_buffer, so fill it with append() - which
alt_string and std::string both provide and which the string fast path
already relies on - instead of assign().

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:02 +02:00
Niels LohmannandClaude Opus 4.8 2bc08ada75 Fix clang-tidy findings and document JSON_USE_SIMDUTF in the nav 
- number_parse.hpp: use std::array for the powers-of-ten table
  (avoid-c-arrays) and `auto` for the cast-initialized result
  (modernize-use-auto), matching the codebase style (cf. the serializer's
  utf8d table). Indexing casts keep the -Wsign-conversion build clean.
- add JSON_USE_SIMDUTF to the mkdocs navigation so the macro page is
  reachable.

No behavior change; clang-tidy is clean on the new headers and the
amalgamation is regenerated.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:02 +02:00
Niels LohmannandClaude Opus 4.8 f2915a86ac Move byte-level scan/parse helpers out of lexer.hpp 
lexer.hpp had grown by ~600 lines of byte-level helpers that have no
dependency on the lexer's template parameters and clutter the state
machine. Move them, unchanged, into two focused headers as free functions
in namespace detail:

- number_parse.hpp: parse_integer_unsigned/parse_integer_signed (now
  templated on the number type) and parse_float_fast (Clinger's exact
  double fast path, with the decimal point passed as an argument instead of
  read from a lexer member).
- string_scan.hpp: the SWAR string helpers (is_string_special,
  swar_string_special, find_string_special, validate_one_utf8,
  scalar_string_bulk_run) and the backend-dispatched string_bulk_run,
  including the optional simdutf include and find_string_delimiter.

lexer.hpp now includes these and calls the free functions; the methods that
touch lexer state (scan_string, scan_number, scan_string_bulk,
scan_number_bulk_contiguous, convert_number) stay put. This is a pure code
move with no behavior change: lexer.hpp drops from 2357 to 1934 lines, the
now-unused <cstdint>/<cstring>/<limits> includes are removed, and the
free-function form makes the SWAR helpers reusable elsewhere (e.g. the
serializer's string escaping).

Verified: default and JSON_USE_SIMDUTF builds compile; 2,000,000 number and
2,000,000 arbitrary-byte-string differential-fuzz documents parse
identically to before; lexer/parser/conversions/deserialization/locale/
diagnostic-position suites pass (20,576 assertions); warning-clean on g++
and clang in C++11/17/20; the amalgamation regenerates and passes
check-amalgamation.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:02 +02:00
Niels LohmannandClaude Opus 4.8 1a6fb6a27b Add a contiguous fast path for scanning numbers 
scan_number() reads a number one character at a time through the input
adapter (get()) and appends each byte to token_buffer (add()) before
converting. For contiguous input, the per-character get()/add() overhead
dominates: it is roughly two thirds of the time spent on number-heavy
parsing, far more than the value conversion itself.

Add scan_number_bulk_contiguous(), which parses the whole number token
straight from the input buffer: it validates and classifies the extent
with the same grammar as scan_number()'s state machine, materializes
token_buffer in one copy (substituting the locale decimal point exactly as
scan_number() does), advances the adapter, and reuses the shared
convert_number() tail. On anything it does not recognize as a well-formed
number it makes no state change and returns token_type::uninitialized, so
the caller falls back to scan_number(), which then produces the exact
diagnostic. Errors and their positions are therefore unchanged.

The conversion tail is factored out of scan_number() into convert_number()
so both scanners share it; the fast path is selected by tag dispatch on the
existing bulk_scan capability, so streaming/wide/user adapters are
unaffected.

Measured on pointer input, g++ 13 -O3:
  - integers: parse +65%, accept +98%
  - floats:   parse +39%, accept +70%

Verified: 2,000,000 randomized number documents (including overflow-range
integers, long digit strings and %.17g doubles) parse identically via the
contiguous path and the streaming byte path, matching value, type and
round-trip text; the locale suite and existing parser/lexer/conversions/
deserialization tests pass; a new "lexer number fast path" test checks
contiguous-vs-streaming parity, token classification, and that malformed
numbers are rejected identically. Pure C++11, no intrinsics.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:02 +02:00
Niels LohmannandClaude Opus 4.8 642c77f534 Add optional simdutf backend for bulk UTF-8 validation (JSON_USE_SIMDUTF) 
The bulk string scanner validates UTF-8 straight from a contiguous buffer.
The scalar validator caps at ~0.3-0.7 GB/s on non-ASCII text; a SIMD
validator reaches several GB/s. Rather than hand-rolling SIMD UTF-8
validation (easy to get subtly wrong - a from-scratch SSE attempt rejected
valid CJK), wire in the vetted simdutf library behind an opt-in switch.

simdutf is not header-only (it ships simdutf.cpp and uses runtime CPU
dispatch), so it is not vendored: defining JSON_USE_SIMDUTF includes
<simdutf.h> and routes the bulk validator through simdutf::validate_utf8;
the project supplies and links simdutf. Undefined (the default), nothing
external is included and the portable C++11 scalar path is used, so the
library stays header-only and its baseline behavior is unchanged.

Design keeps behavior identical either way:
- scan_string_bulk() now finds the run up to the next quote/escape/control
  byte (non-ASCII allowed) and validates it in one shot; on the rare
  validation failure it recomputes the exact valid prefix with the scalar
  helper, so ill-formed input still falls through to the byte path and is
  reported at the same position with the same message.
- the per-sequence scalar path is factored into scalar_string_bulk_run()
  and is the default backend; the refactor is behavior-preserving and does
  not change scalar throughput.

Verified: default and JSON_USE_SIMDUTF builds accept/reject/parse
identically across 2,000,000 arbitrary-byte documents and 1,000,000
mixed-escape/UTF-8 documents (differential fuzz vs the streaming byte
path); lexer/parser/diagnostic-position/deserialization suites pass under
both configurations (20,188 assertions with the backend enabled);
warning-clean on g++ and clang, C++11 and C++20, both configurations.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:02 +02:00
Niels LohmannandClaude Opus 4.8 72cadadfc5 Route contiguous byte containers through the pointer adapter (fast paths in C++11) 
json::parse(std::string) - the most common entry point - did not benefit
from the contiguous fast paths (bulk string scanning, UTF-8 bulk
validation, memcpy for binary formats) in C++11..17: std::string::iterator
is a library wrapper, not a raw pointer, and pre-C++20 there is no portable
way to prove it contiguous, so supports_bulk_scan was false. Only raw
pointers, string literals, and C-arrays (and, in C++20, anything modelling
std::contiguous_iterator) took the fast path.

Detect contiguous single-byte containers (std::string, std::vector<char>,
std::vector<std::uint8_t>, std::string_view, ...) via is_contiguous_byte_
container and route them through an iterator_input_adapter built from
data()/data()+size(). The generic iterator-based container overload is
constrained to exclude these, so the two overloads are disjoint and there
is no ambiguity (a plain competing overload loses to the greedy
forwarding-reference container overload on reference binding, and a factory
partial-specialization is ambiguous - both were tried and rejected).

The pointer keeps the container's own element type, so char_type - and
therefore all parsing behavior - is byte-for-byte identical to the iterator
path (const char* for std::string, const std::uint8_t* for
std::vector<std::uint8_t>); only the raw pointer additionally turns on the
fast paths. Lifetimes are unchanged: the container outlives the adapter for
the full parse expression, exactly as the iterators it replaces did.

Measured, C++11, json::parse/accept(std::string), g++ 13:

  long ASCII strings:  accept 201 -> 3200 MB/s  (~16x),  parse 174 -> 1444
  dense CJK:           accept 263 ->  697 MB/s  (~2.6x)
  short strings:       accept 163 ->  243 MB/s  (~1.5x)

Verified: char_type preserved for std::string (char) and
std::vector<std::uint8_t> (uint8_t); CBOR/MsgPack round-trips from
std::vector<std::uint8_t> unchanged; 1,000,000 randomized documents accept
and parse identically via std::string and via std::istream;
deserialization/user-defined-input/parser/lexer/conversions/diagnostic-
position suites pass (20,480 assertions); warning-clean on g++ and clang in
C++11/17/20.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:02 +02:00
Niels LohmannandClaude Opus 4.8 2c15b0617b Validate UTF-8 in the bulk string scanner (portable ~2-3x on non-ASCII text) 
The SWAR bulk string path stopped at the first non-ASCII byte and handed
every multibyte character to the byte-at-a-time scanner, whose per-byte
get()/next_byte_in_range()/add() machinery runs at roughly half the speed
of validating straight from the buffer. As a result, dense non-ASCII text
(CJK, emoji, accented Latin) parsed ~10-15x slower than ASCII.

Fold well-formed UTF-8 into the bulk run: scan_string_bulk() now, on a
non-ASCII lead byte, validates one sequence with validate_one_utf8() -
which mirrors scan_string()'s per-byte switch ranges exactly (rejecting
overlong forms, surrogates, and out-of-range code points) - and appends it
in place, continuing until the closing quote, an escape, a control byte,
or an ill-formed sequence. All error handling still defers to the byte
path, so error messages and positions are byte-for-byte unchanged.

Because only well-formed content is fast-pathed and every rejection falls
through to the existing scanner, behavior is identical; the win is purely
throughput. Measured on pointer input (accept, string values discarded):

  content        g++ 13         clang 18
  dense CJK      277 -> 648     ~605  MB/s   (~2.3x)
  dense emoji    299 -> 857     ~702  MB/s   (~2.6-2.9x)
  mixed 90% ASCII 246 -> 331    ~334  MB/s   (~1.35x)
  pure ASCII     unchanged (~3.2 / 4.1 GB/s)

Verified: 2,000,000 randomized documents built from arbitrary bytes
(overlong, surrogate, truncated, out-of-range sequences) accept/reject and
parse identically via the contiguous path and the streaming byte path;
lexer/parser/diagnostic-position/deserialization/conversions suites pass
unchanged. Pure C++11, no intrinsics.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:02 +02:00
Niels LohmannandClaude Opus 4.8 4d606cdae5 Add SWAR bulk string scanning for contiguous input (simdjson-style) 
scan_string() read the input one character at a time through the input
adapter and classified every byte with a large switch. For contiguous
byte buffers we can instead scan 8 bytes at a time with a SWAR word test
that finds the first byte needing individual handling (the closing quote,
an escape, a control character, or a non-ASCII UTF-8 byte) and bulk-append
the ordinary run in one go.

- input adapters expose supports_bulk_scan / bulk_data / bulk_remaining /
  bulk_skip for provably-contiguous, same-type, 1-byte iterator ranges
  (raw pointers in every standard; std::string/std::vector/std::array and
  friends additionally in C++20 via std::contiguous_iterator).
- the lexer gains a bulk_scan capability (gated on lazy_token_string so
  bypassing the per-character capture cannot lose error diagnostics) and a
  scan_string_bulk() fast path; streaming/wide/user adapters are unchanged
  and keep the byte-at-a-time scanner.

The run contains no newline (all bytes < 0x20 are treated as special), so
position bookkeeping stays exact, and error tokens are still reconstructed
lazily from the consumed byte range. The SWAR special-byte test is pure
uint64_t arithmetic - no intrinsics, no runtime dispatch, C++11-clean.

Measured on representative data, pointer input, g++ 13 -O3
(string values discarded by accept() see the largest gains):

  long ASCII strings:  DOM +4.5x,  SAX +14x,   accept +17x  (to ~2 GB/s)
  short strings:       DOM +15%,   SAX +62%,   accept +85%
  escape-heavy:        DOM +31%,   SAX +26%,   accept +28%

Same-input parity verified: 200k randomized documents (escapes, multibyte
UTF-8, surrogate pairs) accept/parse identically via the contiguous SWAR
path and the streaming byte path; unit lexer/parser/diagnostic-position/
deserialization/conversions suites pass unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:02 +02:00
Niels LohmannandClaude Opus 4.8 a5895127b6 Speed up number parsing in the lexer (fast paths from the fast_float/simdjson world) 
The number scanner converted its already-validated digit buffer with
std::strtoull/std::strtoll/std::strtod. Those pull in locale and errno
machinery and dominate number-heavy parsing (strtod runs at ~6 M/s).

Replace them with dedicated parsers over the validated buffer:

- parse_integer_unsigned / parse_integer_signed: accumulate digits with
  overflow detection, falling back to the float path on overflow exactly
  as the strtoull/strtoll round-trip check did. Overflow behavior is
  unchanged for narrower or wider custom number types.

- parse_float_fast: Clinger's exact fast path for `double` (<=19 significant
  digits, |exp10| <= 22, significand < 2^53), where significand * 10^exp is
  exact under IEEE round-to-nearest. This is the same fast path used by
  fast_float/simdjson. It is bit-identical to strtod on this subset and
  declines (falling back to strtod) otherwise. Only `double` uses it; float
  and long double keep std::strtof/std::strtold via a templated overload.

Measured on representative data (g++ 13, -O3):
  - integers:  DOM parse +11%, SAX +25-34%
  - floats:    DOM parse +37%, SAX +70%  (clang: float DOM ~1.9x)

No dependencies added; header-only and C++11-clean. Existing parser,
lexer, conversion and deserialization unit tests pass unchanged; a
3M-value random-double fuzz matches strtod bit-for-bit.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AXcDtEma2PjxgmPS9cQGzA
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-31 23:25:02 +02:00
dependabot[bot] c864bb36da Bump mkdocs-git-revision-date-localized-plugin in /docs/mkdocs (#5444) 
Bumps [mkdocs-git-revision-date-localized-plugin](https://github.com/timvink/mkdocs-git-revision-date-localized-plugin) from 1.5.3 to 1.5.4.
- [Release notes](https://github.com/timvink/mkdocs-git-revision-date-localized-plugin/releases)
- [Commits](https://github.com/timvink/mkdocs-git-revision-date-localized-plugin/compare/v1.5.3...v1.5.4)

---
updated-dependencies:
- dependency-name: mkdocs-git-revision-date-localized-plugin
  dependency-version: 1.5.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-08-31 23:06:13 +02:00
dependabot[bot] 129d2891ed Bump the codeql-action group with 4 updates (#5446) 
Bumps the codeql-action group with 4 updates: [github/codeql-action/init](https://github.com/github/codeql-action), [github/codeql-action/autobuild](https://github.com/github/codeql-action), [github/codeql-action/analyze](https://github.com/github/codeql-action) and [github/codeql-action/upload-sarif](https://github.com/github/codeql-action).


Updates `github/codeql-action/init` from 4.37.7 to 4.37.8
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/ff2f1c621b7f889edc0d3c761ac2e6a3f8cdb0dd...db488ddef3bf6cb639b32c2e9a7c0a7ea8271d28)

Updates `github/codeql-action/autobuild` from 4.37.7 to 4.37.8
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/ff2f1c621b7f889edc0d3c761ac2e6a3f8cdb0dd...db488ddef3bf6cb639b32c2e9a7c0a7ea8271d28)

Updates `github/codeql-action/analyze` from 4.37.7 to 4.37.8
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/ff2f1c621b7f889edc0d3c761ac2e6a3f8cdb0dd...db488ddef3bf6cb639b32c2e9a7c0a7ea8271d28)

Updates `github/codeql-action/upload-sarif` from 4.37.7 to 4.37.8
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/ff2f1c621b7f889edc0d3c761ac2e6a3f8cdb0dd...db488ddef3bf6cb639b32c2e9a7c0a7ea8271d28)

---
updated-dependencies:
- dependency-name: github/codeql-action/init
  dependency-version: 4.37.8
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: codeql-action
- dependency-name: github/codeql-action/autobuild
  dependency-version: 4.37.8
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: codeql-action
- dependency-name: github/codeql-action/analyze
  dependency-version: 4.37.8
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: codeql-action
- dependency-name: github/codeql-action/upload-sarif
  dependency-version: 4.37.8
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: codeql-action
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-08-31 23:05:44 +02:00
elix3r 35705d79d8 Fix update(merge_objects=true) throwing on primitive-to-object merge (#5414) 
When merge_objects is true, recurse only if the existing value is an
object. Otherwise overwrite, matching the documented "all other values
are overwritten as usual" behavior.

Fixes #5402

Signed-off-by: elix3r <157088510+22elix3r@users.noreply.github.com>
 2026-08-28 13:38:28 +01:00
dependabot[bot] 892be68ca4 Bump the codeql-action group across 1 directory with 4 updates (#5388) 
Bumps the codeql-action group with 4 updates in the / directory: [github/codeql-action/init](https://github.com/github/codeql-action), [github/codeql-action/autobuild](https://github.com/github/codeql-action), [github/codeql-action/analyze](https://github.com/github/codeql-action) and [github/codeql-action/upload-sarif](https://github.com/github/codeql-action).


Updates `github/codeql-action/init` from 4.37.6 to 4.37.7
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/5595ccaf912efad79be6eef63a5619ff05969be3...ff2f1c621b7f889edc0d3c761ac2e6a3f8cdb0dd)

Updates `github/codeql-action/autobuild` from 4.37.6 to 4.37.7
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/5595ccaf912efad79be6eef63a5619ff05969be3...ff2f1c621b7f889edc0d3c761ac2e6a3f8cdb0dd)

Updates `github/codeql-action/analyze` from 4.37.6 to 4.37.7
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/5595ccaf912efad79be6eef63a5619ff05969be3...ff2f1c621b7f889edc0d3c761ac2e6a3f8cdb0dd)

Updates `github/codeql-action/upload-sarif` from 4.37.6 to 4.37.7
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/5595ccaf912efad79be6eef63a5619ff05969be3...ff2f1c621b7f889edc0d3c761ac2e6a3f8cdb0dd)

---
updated-dependencies:
- dependency-name: github/codeql-action/analyze
  dependency-version: 4.37.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: codeql-action
- dependency-name: github/codeql-action/autobuild
  dependency-version: 4.37.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: codeql-action
- dependency-name: github/codeql-action/init
  dependency-version: 4.37.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: codeql-action
- dependency-name: github/codeql-action/upload-sarif
  dependency-version: 4.37.7
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: codeql-action
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-08-28 13:30:33 +01:00
whn 1ac268d409 docs: correct to_bson complexity (#5334) 
Signed-off-by: whn <142425816+Whning0513@users.noreply.github.com>
 2026-08-28 13:27:18 +01:00
Niels LohmannandClaude 3fa93dac65 docs: document std::pair/std::tuple serializing as an object for string-keyed pairs (#5442) 
A std::pair or std::tuple whose every element is itself a two-element array
with a string first element (e.g. std::pair<std::string, int>) serializes to
a JSON object instead of a JSON array, because to_json builds the value with a
brace initializer and the initializer-list object-detection rule fires. The
resulting object cannot be read back into the original type and collapses
duplicate keys. Document this quirk in the conversions guide, together with the
unaffected cases and the idiom to force an array.


Claude-Session: https://claude.ai/code/session_016cwQq8WQRFzQcGQtbJTtJg

Signed-off-by: Claude <noreply@anthropic.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-08-28 13:26:01 +01:00
dependabot[bot] 1876493f87 Bump step-security/harden-runner from 2.20.1 to 2.21.0 (#5394) 
Bumps [step-security/harden-runner](https://github.com/step-security/harden-runner) from 2.20.1 to 2.21.0.
- [Release notes](https://github.com/step-security/harden-runner/releases)
- [Commits](https://github.com/step-security/harden-runner/compare/b09bb98e06d4d774595224525879c09bc6e98c40...05e31511f85b41b11d1cf0ef85d0992719546e2c)

---
updated-dependencies:
- dependency-name: step-security/harden-runner
  dependency-version: 2.21.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-08-27 07:39:56 +01:00
whn 01853ed6bc docs: document lenient BSON input handling (#5333) 
Signed-off-by: whn <142425816+Whning0513@users.noreply.github.com>
 2026-08-25 07:18:50 +01:00
Krishnanand G 2f025f401e Throw other_error.502 when UBJSON use_type is set without use_size (#5380) 
* Throw other_error.502 when UBJSON use_type is set without use_size

Fixes #5321

Signed-off-by: Krishnanand G <118352827+Krishnanand-G@users.noreply.github.com>

* Scope UBJSON use_type check to container branches and expand tests

Signed-off-by: Krishnanand G <118352827+Krishnanand-G@users.noreply.github.com>

* Re-amalgamate single_include/json.hpp

The previous commit updated the split headers but the amalgamated
file didn't go back through astyle before I committed it, so CI's
amalgamation check caught formatting drift in json_fwd.hpp and a
few noexcept clauses in basic_json, plus one doc example. None of
it touches the UBJSON logic. Applied the patch CI generated to
bring single_include back in sync.

Signed-off-by: Krishnanand G <118352827+Krishnanand-G@users.noreply.github.com>

---------

Signed-off-by: Krishnanand G <118352827+Krishnanand-G@users.noreply.github.com>
 2026-08-25 07:16:10 +01:00
Niels Lohmann 734fd305a1 Format-check the documentation examples in CI (#5386) 
* Reformat parser_callback_t example with astyle

The file uses "json & /*parsed*/" in three lambda parameter lists, which
astyle rewrites to "json& /*parsed*/" per --align-reference=type. The
drift went unnoticed because CI never format-checked the documentation
examples; "make pretty" does cover them.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Format-check the documentation examples in CI

The examples live in docs/mkdocs/docs/examples, but both format checks
still referenced the long-gone docs/examples path:

- check_amalgamation.yml passed it to find, which printed an error for
  the missing path and carried on, so astyle only ever saw include and
  tests. The step still exited 0.
- ci.cmake globbed it into INDENT_FILES, and a GLOB_RECURSE over a
  missing directory silently yields nothing, so the ci_test_amalgamation
  target skipped the examples too.

Either way the 231 example files have never been format-checked. Point
both at the real path, and guard the workflow with an explicit directory
check so a future rename fails the job instead of quietly shrinking the
file list again.

Also drop the dead docs/examples/** path filter from
publish_documentation.yml; docs/mkdocs/** already covers the examples.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-20 12:32:50 +02:00
dependabot[bot] 36187cacfb ⬆️ Bump wheel from 0.47.0 to 0.48.0 in /docs/mkdocs (#5385) 
Bumps [wheel](https://github.com/pypa/wheel) from 0.47.0 to 0.48.0.
- [Release notes](https://github.com/pypa/wheel/releases)
- [Changelog](https://github.com/pypa/wheel/blob/main/docs/news.rst)
- [Commits](https://github.com/pypa/wheel/compare/0.47.0...0.48.0)

---
updated-dependencies:
- dependency-name: wheel
  dependency-version: 0.48.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-08-20 12:32:34 +02:00
Sahil_Kamate b5378e8deb Fix CBOR tag handlers not recognizing tags 0-5 and 21-23 (#5331) 
* Fix CBOR tag handlers not recognizing tags 0-5 and 21-23

The tagged-item switch in binary_reader::parse_cbor_internal() only handled
head bytes 0xC6-0xD4 and 0xD8-0xDB. Bytes 0xC0-0xC5 (tags 0-5: date/time,
epoch, bignum, decimal, bigfloat) and 0xD5-0xD7 (tags 21-23: base64url,
base64, base16 conversion hints) fell through to the default case and were
reported as invalid bytes, even under cbor_tag_handler_t::ignore and ::store,
despite being valid CBOR major-type-6 tags per RFC 8949.

Add the missing case labels so the full 0xC0-0xDB range is handled
uniformly. Extend the "Tagged values" test in unit-cbor.cpp to cover
0xC0-0xD7, and update the CBOR docs to state the corrected tag range.

Fixes #5315

Signed-off-by: sahilkamate03 <45514385+sahilkamate03@users.noreply.github.com>

* Fix stale CBOR tag docs and add store-mode binary-payload test

The "Incomplete mapping" warning still listed tags 0-5 (date/time,
bignum, decimal fraction, bigfloat) and 21-23 (expected conversions)
as unsupported, even though they now parse correctly under
cbor_tag_handler_t::ignore/store, same as 0xC6..0xD4/0xD8..0xDB.
Remove those five bullets and cross-reference the "Tagged items"
warning below, matching the equivalent docs fix landed independently
in PR #5367.

Also add a cbor_tag_handler_t::store test that wraps a binary
payload (not just a string) for every byte in 0xC0..0xD7, confirming
these tags are unwrapped the same way as 0xC6..0xD4 rather than
mistaken for the 0xD8..0xDB binary-subtype marker syntax, per review
feedback on #5331.

Signed-off-by: sahilkamate03 <45514385+sahilkamate03@users.noreply.github.com>

---------

Signed-off-by: sahilkamate03 <45514385+sahilkamate03@users.noreply.github.com>
 2026-08-19 20:19:47 +02:00
dependabot[bot] ce87157d4e ⬆️ Bump the codeql-action group across 1 directory with 4 updates (#5379) 
Bumps the codeql-action group with 4 updates in the / directory: [github/codeql-action/init](https://github.com/github/codeql-action), [github/codeql-action/autobuild](https://github.com/github/codeql-action), [github/codeql-action/analyze](https://github.com/github/codeql-action) and [github/codeql-action/upload-sarif](https://github.com/github/codeql-action).


Updates `github/codeql-action/init` from 4.37.5 to 4.37.6
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/d1ba80a13dd99fba24a470575428917156a28b43...5595ccaf912efad79be6eef63a5619ff05969be3)

Updates `github/codeql-action/autobuild` from 4.37.5 to 4.37.6
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/d1ba80a13dd99fba24a470575428917156a28b43...5595ccaf912efad79be6eef63a5619ff05969be3)

Updates `github/codeql-action/analyze` from 4.37.5 to 4.37.6
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/d1ba80a13dd99fba24a470575428917156a28b43...5595ccaf912efad79be6eef63a5619ff05969be3)

Updates `github/codeql-action/upload-sarif` from 4.37.5 to 4.37.6
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/d1ba80a13dd99fba24a470575428917156a28b43...5595ccaf912efad79be6eef63a5619ff05969be3)

---
updated-dependencies:
- dependency-name: github/codeql-action/analyze
  dependency-version: 4.37.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: codeql-action
- dependency-name: github/codeql-action/autobuild
  dependency-version: 4.37.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: codeql-action
- dependency-name: github/codeql-action/init
  dependency-version: 4.37.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: codeql-action
- dependency-name: github/codeql-action/upload-sarif
  dependency-version: 4.37.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: codeql-action
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-08-18 21:47:20 +02:00
dependabot[bot] cdf52ae9be ⬆️ Bump lukka/get-cmake from 4.4.1 to 4.4.2 (#5373) 
Bumps [lukka/get-cmake](https://github.com/lukka/get-cmake) from 4.4.1 to 4.4.2.
- [Release notes](https://github.com/lukka/get-cmake/releases)
- [Changelog](https://github.com/lukka/get-cmake/blob/main/RELEASE_PROCESS.md)
- [Commits](https://github.com/lukka/get-cmake/compare/4a7d025fc60f00db0c7b44ebf783d19b52444830...fffaaafeea488556c2c12dad60690008bc1caacb)

---
updated-dependencies:
- dependency-name: lukka/get-cmake
  dependency-version: 4.4.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-08-12 21:18:24 +02:00
dependabot[bot] 146ba55453 ⬆️ Bump step-security/harden-runner from 2.20.0 to 2.20.1 (#5375) 
Bumps [step-security/harden-runner](https://github.com/step-security/harden-runner) from 2.20.0 to 2.20.1.
- [Release notes](https://github.com/step-security/harden-runner/releases)
- [Commits](https://github.com/step-security/harden-runner/compare/bf7454d06d71f1098171f2acdf0cd4708d7b5920...b09bb98e06d4d774595224525879c09bc6e98c40)

---
updated-dependencies:
- dependency-name: step-security/harden-runner
  dependency-version: 2.20.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-08-12 09:21:46 +02:00
dependabot[bot] e6978ba50c ⬆️ Bump the codeql-action group with 4 updates (#5372) 
Bumps the codeql-action group with 4 updates: [github/codeql-action/init](https://github.com/github/codeql-action), [github/codeql-action/autobuild](https://github.com/github/codeql-action), [github/codeql-action/analyze](https://github.com/github/codeql-action) and [github/codeql-action/upload-sarif](https://github.com/github/codeql-action).


Updates `github/codeql-action/init` from 4.37.4 to 4.37.5
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/f205ea1c3313d32999d8d6a48b4f6530d4437b38...d1ba80a13dd99fba24a470575428917156a28b43)

Updates `github/codeql-action/autobuild` from 4.37.4 to 4.37.5
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/f205ea1c3313d32999d8d6a48b4f6530d4437b38...d1ba80a13dd99fba24a470575428917156a28b43)

Updates `github/codeql-action/analyze` from 4.37.4 to 4.37.5
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/f205ea1c3313d32999d8d6a48b4f6530d4437b38...d1ba80a13dd99fba24a470575428917156a28b43)

Updates `github/codeql-action/upload-sarif` from 4.37.4 to 4.37.5
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/f205ea1c3313d32999d8d6a48b4f6530d4437b38...d1ba80a13dd99fba24a470575428917156a28b43)

---
updated-dependencies:
- dependency-name: github/codeql-action/init
  dependency-version: 4.37.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: codeql-action
- dependency-name: github/codeql-action/autobuild
  dependency-version: 4.37.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: codeql-action
- dependency-name: github/codeql-action/analyze
  dependency-version: 4.37.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: codeql-action
- dependency-name: github/codeql-action/upload-sarif
  dependency-version: 4.37.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: codeql-action
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-08-12 09:21:41 +02:00
ljcjclljc 6285225fd0 Fix integer comparison bug (#5211) 
* Fix integer comparison bug

Signed-off-by: ljccjlljc <939159710@qq.com>

* commit

Signed-off-by: ljccjlljc <939159710@qq.com>

* Remove generated CI artifacts and update amalgamation

Signed-off-by: ljccjlljc <939159710@qq.com>

* Silence cpplint braces warning in comparison macro

Signed-off-by: ljccjlljc <939159710@qq.com>

* Update amalgamation after cpplint fix

Signed-off-by: ljccjlljc <939159710@qq.com>

* Add mixed signed and unsigned comparison regression test

Signed-off-by: ljccjlljc <939159710@qq.com>

* Clarify mixed signed and unsigned comparison handling

Signed-off-by: ljccjlljc <939159710@qq.com>

* Expand mixed signed and unsigned comparison tests

Signed-off-by: ljccjlljc <939159710@qq.com>

---------

Signed-off-by: ljccjlljc <939159710@qq.com>
 2026-08-12 09:21:19 +02:00
dependabot[bot] 21af527e75 ⬆️ Bump the codeql-action group with 4 updates (#5365) 2026-08-07 19:30:42 +02:00
Niels Lohmann 23518f54fe Add an Ecosystem page for third-party projects built on nlohmann::json (#5369) 2026-08-07 19:29:58 +02:00
Dmitry 1c136a66c4 Move the CBOR doc block to the function it describes (#5363) 
The block documenting get_char and tag_handler sat above
get_cbor_negative_integer(), which takes neither, so Doxygen attached it
there and parse_cbor_internal() was left undocumented.

Comment placement only.

Signed-off-by: Dmitry <45711841+darkdi@users.noreply.github.com>
 2026-08-06 08:30:15 +02:00
dependabot[bot] c1c19a7bcd ⬆️ Bump lukka/get-cmake from 4.4.0 to 4.4.1 (#5364) 
Bumps [lukka/get-cmake](https://github.com/lukka/get-cmake) from 4.4.0 to 4.4.1.
- [Release notes](https://github.com/lukka/get-cmake/releases)
- [Changelog](https://github.com/lukka/get-cmake/blob/main/RELEASE_PROCESS.md)
- [Commits](https://github.com/lukka/get-cmake/compare/e6906078ebd1ccb8ce51ab4626ac46a1b5a517e3...4a7d025fc60f00db0c7b44ebf783d19b52444830)

---
updated-dependencies:
- dependency-name: lukka/get-cmake
  dependency-version: 4.4.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-08-06 08:30:00 +02:00
Niels Lohmann bacdabd176 Fix start_pos() for strings containing escape sequences (#5361) 
The diagnostic position of a string value was derived by subtracting the
parsed value's length from the end position. Escape sequences make the
source token longer than the value it parses to, so the reported start
position landed inside the string, one byte off per escape sequence:

    input: {"a":"\n\n\n\n\n\n"}
      start_pos() == 11, so the reported range covered  n\n\n\n"
      instead of the documented "\n\n\n\n\n\n"

This contradicts the documented behavior of start_pos(), which is the
position of the opening quote, and it also corrupted the "(bytes N-M)"
part of JSON_DIAGNOSTICS exception messages. Strings with multi-byte
UTF-8 but no escapes were unaffected, which is why this went unnoticed.

Record the offset of the token in the lexer when it starts scanning and
use that, instead of reconstructing it from the parsed value. Booleans,
null and numbers already reported correct positions and are unchanged.

The new lexer member and accessor are compiled only when
JSON_DIAGNOSTIC_POSITIONS is enabled, which is already part of the ABI
tag, so the default build is unaffected.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-05 16:01:23 +02:00
Niels Lohmann d5647e6a3b Resolve the TODO(niels) in get_ubjson_string (#5355) 
The comment asked whether the no-op marker 'N' may be ignored when a
string is read. It may not: at that point the next byte must be a string
length type specification, and 'N' is not one. No-ops at positions where
a value may start are already consumed by the callers through
get_ignore_noop(), so nothing is lost by not skipping them here.

Replace the TODO with a comment stating that, and add regression tests
pinning both directions: a no-op is accepted at top level (also
repeated), before and after an array element, and before an object key,
between key and value, and before the closing brace of an object of
unknown size; it is rejected where a length type specification is
expected, i.e. after the 'S' marker of a string value and as the key
length of an object of known size.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-05 14:47:40 +02:00
Niels Lohmann 9a091d2b82 Do not write BJData ndarrays whose size overflows std::size_t (#5362) 
* Do not write BJData ndarrays whose size overflows std::size_t

write_bjdata_ndarray() multiplied the _ArraySize_ dimensions into a
std::size_t without checking for overflow. A product that wraps around
to a value that happens to match the size of _ArrayData_ passed the
length check, and the writer emitted an ndarray header announcing an
element count that cannot be represented:

    {"_ArrayType_":"uint8","_ArraySize_":[9223372036854775808,2],"_ArrayData_":[]}

was encoded as 5b 24 55 23 5b 4d 00 00 00 00 00 00 00 80 69 02 5d, an
ndarray of 2^64 elements followed by no data. Reading that back throws
out_of_range.408 ("excessive ndarray size caused overflow"), so to_bjdata
produced output that from_bjdata rejects. This is reachable by parsing
untrusted JSON and re-encoding it as BJData.

Mirror the overflow check the binary reader already performs, and also
reject a single dimension that does not fit into std::size_t, which the
previous cast silently truncated where std::size_t is narrower than 64
bits. Such objects now fall back to a plain object encoding, which is
what the surrounding type and length validation already does for
annotations it cannot represent, and they round-trip unchanged.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Document when to_bjdata converts a JData annotation to an ND-array

The BJData page described the 1-D vector case as the only situation in
which an object carrying _ArrayType_/_ArraySize_/_ArrayData_ is not
written as a compact ND-array. The writer has always had several other
fallbacks -- an unknown _ArrayType_, a dimension that is not a
non-negative integer, an _ArrayData_ whose length does not match the
product of the dimensions, and elements that are not numbers of the
annotated kind -- all of which cause the value to be serialized as a
regular JSON object instead.

Spell out the conditions, including the size-overflow check added in the
preceding commit, so the documented behavior matches the implementation.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-05 13:44:15 +02:00
Niels Lohmann b890b4cba3 CI: build the MinGW Clang matrix without debug info (#5360) 
Linking test-regression2_cpp20 intermittently fails with

  unit-regression2.cpp.obj:(.debug_info+0x16): relocation truncated to
  fit: IMAGE_REL_AMD64_SECREL against `.debug_line'

The failure moves between matrix entries from run to run, and the same
commit can pass and fail on consecutive runs, so it is the size of the
debug sections rather than any one Clang version.

The jobs only build and run the tests, so override CMAKE_CXX_FLAGS_DEBUG
to drop the default -g. Everything else about the Debug build is
unchanged: no optimization flag is added and NDEBUG stays undefined, so
JSON_ASSERT remains active.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
 2026-08-05 13:44:05 +02:00
Angadi56 dca9d49a33 reject out-of-range code points in UTF-32 wide-string input (#5348) 
* reject out-of-range code points in UTF-32 wide-string input

Signed-off-by: Angadi Yashaswini <angadi@digiscrypt.com>

* remove useless cast to char_traits<char>::int_type

Signed-off-by: Angadi Yashaswini <angadi@digiscrypt.com>

---------

Signed-off-by: Angadi Yashaswini <angadi@digiscrypt.com>
 2026-08-05 13:43:36 +02:00
Petr Bělohlávek acd87e2336 CI: Add clang 21&22 to Ubuntu CLang build matrix (#5347) 
* Add clang 21 to ubuntu build matrix (CI)

Signed-off-by: Petr Belohlavek <me@petrbel.cz>

* Add clang 22 to ubuntu build matrix (CI)

Signed-off-by: Petr Belohlavek <me@petrbel.cz>

* Register Clang 22.1.8 to quality_assurance.md

Signed-off-by: Petr Belohlavek <me@petrbel.cz>

---------

Signed-off-by: Petr Belohlavek <me@petrbel.cz>
 2026-08-04 16:14:35 +02:00
58 changed files with 5389 additions and 527 deletions
Expand all files Collapse all files
.github/workflows/check_amalgamation.yml
+13 -3
View File
@@ -11,7 +11,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Harden Runner
uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0
uses: step-security/harden-runner@05e31511f85b41b11d1cf0ef85d0992719546e2c # v2.21.0
with:
egress-policy: audit
@@ -34,7 +34,7 @@ jobs:
steps:
- name: Harden Runner
uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0
uses: step-security/harden-runner@05e31511f85b41b11d1cf0ef85d0992719546e2c # v2.21.0
with:
egress-policy: audit
@@ -67,8 +67,18 @@ jobs:
${{ github.workspace }}/venv/bin/astyle --project=tools/astyle/.astylerc --suffix=none --quiet \
$INCLUDE_DIR/json.hpp $INCLUDE_DIR/json_fwd.hpp
# fail loudly if a directory is renamed or removed: find would only warn
# about the missing path and silently drop its files from the check
SOURCE_DIRS="docs/mkdocs/docs/examples include tests"
for DIR in $SOURCE_DIRS; do
if [ ! -d "$DIR" ]; then
echo "::error::source directory '$DIR' does not exist"
exit 1
fi
done
${{ github.workspace }}/venv/bin/astyle --project=tools/astyle/.astylerc --suffix=none --quiet \
$(find docs/examples include tests -type f \( -name '*.hpp' -o -name '*.cpp' -o -name '*.cu' \) -not -path 'tests/thirdparty/*' -not -path 'tests/abi/include/nlohmann/*' | sort)
$(find $SOURCE_DIRS -type f \( -name '*.hpp' -o -name '*.cpp' -o -name '*.cu' \) -not -path 'tests/thirdparty/*' -not -path 'tests/abi/include/nlohmann/*' | sort)
- name: Build patch and check for differences
id: diff
.github/workflows/cifuzz.yml
+1 -1
View File
@@ -9,7 +9,7 @@ jobs:
runs-on: ubuntu-22.04
steps:
- name: Harden Runner
uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0
uses: step-security/harden-runner@05e31511f85b41b11d1cf0ef85d0992719546e2c # v2.21.0
with:
egress-policy: audit
.github/workflows/codeql-analysis.yml
+4 -4
View File
@@ -27,7 +27,7 @@ jobs:
steps:
- name: Harden Runner
uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0
uses: step-security/harden-runner@05e31511f85b41b11d1cf0ef85d0992719546e2c # v2.21.0
with:
egress-policy: audit
@@ -38,14 +38,14 @@ jobs:
# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@e4fba868fa4b1b91e1fdab776edc8cfbe6e9fb81 # v4.37.3
uses: github/codeql-action/init@db488ddef3bf6cb639b32c2e9a7c0a7ea8271d28 # v4.37.8
with:
languages: c-cpp
# Autobuild attempts to build any compiled languages (C/C++, C#, or Java).
# If this step fails, then you should remove it and run the build manually (see below)
- name: Autobuild
uses: github/codeql-action/autobuild@e4fba868fa4b1b91e1fdab776edc8cfbe6e9fb81 # v4.37.3
uses: github/codeql-action/autobuild@db488ddef3bf6cb639b32c2e9a7c0a7ea8271d28 # v4.37.8
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@e4fba868fa4b1b91e1fdab776edc8cfbe6e9fb81 # v4.37.3
uses: github/codeql-action/analyze@db488ddef3bf6cb639b32c2e9a7c0a7ea8271d28 # v4.37.8
.github/workflows/comment_check_amalgamation.yml
+1 -1
View File
@@ -19,7 +19,7 @@ jobs:
pull-requests: write
steps:
- name: Harden Runner
uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0
uses: step-security/harden-runner@05e31511f85b41b11d1cf0ef85d0992719546e2c # v2.21.0
with:
egress-policy: audit
.github/workflows/dependency-review.yml
+1 -1
View File
@@ -17,7 +17,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Harden Runner
uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0
uses: step-security/harden-runner@05e31511f85b41b11d1cf0ef85d0992719546e2c # v2.21.0
with:
egress-policy: audit
.github/workflows/flawfinder.yml
+2 -2
View File
@@ -27,7 +27,7 @@ jobs:
security-events: write
steps:
- name: Harden Runner
uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0
uses: step-security/harden-runner@05e31511f85b41b11d1cf0ef85d0992719546e2c # v2.21.0
with:
egress-policy: audit
@@ -43,6 +43,6 @@ jobs:
output: 'flawfinder_results.sarif'
- name: Upload analysis results to GitHub Security tab
uses: github/codeql-action/upload-sarif@e4fba868fa4b1b91e1fdab776edc8cfbe6e9fb81 # v4.37.3
uses: github/codeql-action/upload-sarif@db488ddef3bf6cb639b32c2e9a7c0a7ea8271d28 # v4.37.8
with:
sarif_file: ${{github.workspace}}/flawfinder_results.sarif
.github/workflows/labeler.yml
+1 -1
View File
@@ -17,7 +17,7 @@ jobs:
steps:
- name: Harden Runner
uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0
uses: step-security/harden-runner@05e31511f85b41b11d1cf0ef85d0992719546e2c # v2.21.0
with:
egress-policy: audit
.github/workflows/publish_documentation.yml
+1 -2
View File
@@ -7,7 +7,6 @@ on:
- develop
paths:
- docs/mkdocs/**
- docs/examples/**
workflow_dispatch:
# we don't want to have concurrent jobs, and we don't want to cancel running jobs to avoid broken publications
@@ -27,7 +26,7 @@ jobs:
runs-on: ubuntu-22.04
steps:
- name: Harden Runner
uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0
uses: step-security/harden-runner@05e31511f85b41b11d1cf0ef85d0992719546e2c # v2.21.0
with:
egress-policy: audit
.github/workflows/scorecards.yml
+2 -2
View File
@@ -36,7 +36,7 @@ jobs:
steps:
- name: Harden Runner
uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0
uses: step-security/harden-runner@05e31511f85b41b11d1cf0ef85d0992719546e2c # v2.21.0
with:
egress-policy: audit
@@ -76,6 +76,6 @@ jobs:
# Upload the results to GitHub's code scanning dashboard.
- name: "Upload to code-scanning"
uses: github/codeql-action/upload-sarif@e4fba868fa4b1b91e1fdab776edc8cfbe6e9fb81 # v4.37.3
uses: github/codeql-action/upload-sarif@db488ddef3bf6cb639b32c2e9a7c0a7ea8271d28 # v4.37.8
with:
sarif_file: results.sarif
.github/workflows/semgrep.yml
+2 -2
View File
@@ -32,7 +32,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Harden Runner
uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0
uses: step-security/harden-runner@05e31511f85b41b11d1cf0ef85d0992719546e2c # v2.21.0
with:
egress-policy: audit
@@ -61,7 +61,7 @@ jobs:
# Upload SARIF file generated in previous step
- name: Upload SARIF file
uses: github/codeql-action/upload-sarif@e4fba868fa4b1b91e1fdab776edc8cfbe6e9fb81 # v4.37.3
uses: github/codeql-action/upload-sarif@db488ddef3bf6cb639b32c2e9a7c0a7ea8271d28 # v4.37.8
with:
sarif_file: semgrep.sarif
if: always()
.github/workflows/stale.yml
+1 -1
View File
@@ -16,7 +16,7 @@ jobs:
steps:
- name: Harden Runner
uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0
uses: step-security/harden-runner@05e31511f85b41b11d1cf0ef85d0992719546e2c # v2.21.0
with:
egress-policy: audit
.github/workflows/ubuntu.yml
+23 -23
View File
@@ -25,7 +25,7 @@ jobs:
with:
persist-credentials: false
- name: Get latest CMake and ninja
uses: lukka/get-cmake@e6906078ebd1ccb8ce51ab4626ac46a1b5a517e3 # v4.4.0
uses: lukka/get-cmake@fffaaafeea488556c2c12dad60690008bc1caacb # v4.4.2
- name: Run CMake
run: cmake -S . -B build -DJSON_CI=On
- name: Build
@@ -35,7 +35,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Harden Runner
uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0
uses: step-security/harden-runner@05e31511f85b41b11d1cf0ef85d0992719546e2c # v2.21.0
with:
egress-policy: audit
@@ -47,7 +47,7 @@ jobs:
with:
persist-credentials: false
- name: Get latest CMake and ninja
uses: lukka/get-cmake@e6906078ebd1ccb8ce51ab4626ac46a1b5a517e3 # v4.4.0
uses: lukka/get-cmake@fffaaafeea488556c2c12dad60690008bc1caacb # v4.4.2
- name: Run CMake
run: cmake -S . -B build -DJSON_CI=On
- name: Build
@@ -60,7 +60,7 @@ jobs:
target: [ci_test_amalgamation, ci_test_single_header, ci_cppcheck, ci_cpplint, ci_reproducible_tests, ci_non_git_tests, ci_offline_testdata, ci_reuse_compliance, ci_test_valgrind]
steps:
- name: Harden Runner
uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0
uses: step-security/harden-runner@05e31511f85b41b11d1cf0ef85d0992719546e2c # v2.21.0
with:
egress-policy: audit
@@ -70,7 +70,7 @@ jobs:
with:
persist-credentials: false
- name: Get latest CMake and ninja
uses: lukka/get-cmake@e6906078ebd1ccb8ce51ab4626ac46a1b5a517e3 # v4.4.0
uses: lukka/get-cmake@fffaaafeea488556c2c12dad60690008bc1caacb # v4.4.2
- name: Run CMake
run: cmake -S . -B build -DJSON_CI=On
- name: Build
@@ -89,7 +89,7 @@ jobs:
with:
persist-credentials: false
- name: Get latest CMake and ninja
uses: lukka/get-cmake@e6906078ebd1ccb8ce51ab4626ac46a1b5a517e3 # v4.4.0
uses: lukka/get-cmake@fffaaafeea488556c2c12dad60690008bc1caacb # v4.4.2
- name: Run CMake
run: cmake -S . -B build -DJSON_CI=On
- name: Build
@@ -100,7 +100,7 @@ jobs:
container: ubuntu:focal
strategy:
matrix:
target: [ci_cmake_flags, ci_test_diagnostics, ci_test_diagnostic_positions, ci_test_noexceptions, ci_test_noimplicitconversions, ci_test_legacycomparison, ci_test_noglobaludls]
target: [ci_cmake_flags, ci_test_diagnostics, ci_test_diagnostic_positions, ci_test_noexceptions, ci_test_noimplicitconversions, ci_test_legacycomparison, ci_test_noglobaludls, ci_test_simdutf]
steps:
- name: Install build-essential
run: apt-get update ; apt-get install -y build-essential unzip wget git libssl-dev
@@ -108,7 +108,7 @@ jobs:
with:
persist-credentials: false
- name: Get latest CMake and ninja
uses: lukka/get-cmake@e6906078ebd1ccb8ce51ab4626ac46a1b5a517e3 # v4.4.0
uses: lukka/get-cmake@fffaaafeea488556c2c12dad60690008bc1caacb # v4.4.2
- name: Run CMake
run: cmake -S . -B build -DJSON_CI=On
- name: Build
@@ -118,7 +118,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Harden Runner
uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0
uses: step-security/harden-runner@05e31511f85b41b11d1cf0ef85d0992719546e2c # v2.21.0
with:
egress-policy: audit
@@ -184,7 +184,7 @@ jobs:
with:
persist-credentials: false
- name: Get latest CMake and ninja
uses: lukka/get-cmake@e6906078ebd1ccb8ce51ab4626ac46a1b5a517e3 # v4.4.0
uses: lukka/get-cmake@fffaaafeea488556c2c12dad60690008bc1caacb # v4.4.2
- name: Run CMake
run: CXX=g++-${{ matrix.compiler }} cmake -S . -B build -DJSON_CI=On
- name: Build
@@ -202,7 +202,7 @@ jobs:
with:
persist-credentials: false
- name: Get latest CMake and ninja
uses: lukka/get-cmake@e6906078ebd1ccb8ce51ab4626ac46a1b5a517e3 # v4.4.0
uses: lukka/get-cmake@fffaaafeea488556c2c12dad60690008bc1caacb # v4.4.2
- name: Run CMake
run: cmake -S . -B build -DJSON_CI=On
- name: Build
@@ -212,14 +212,14 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
compiler: ['3.4', '3.5', '3.6', '3.7', '3.8', '3.9', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15-bullseye', '16', '17', '18', '19', '20', 'latest']
compiler: ['3.4', '3.5', '3.6', '3.7', '3.8', '3.9', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15-bullseye', '16', '17', '18', '19', '20', '21', '22', 'latest']
container: silkeh/clang:${{ matrix.compiler }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Get latest CMake and ninja
uses: lukka/get-cmake@e6906078ebd1ccb8ce51ab4626ac46a1b5a517e3 # v4.4.0
uses: lukka/get-cmake@fffaaafeea488556c2c12dad60690008bc1caacb # v4.4.2
- name: Set env FORCE_STDCPPFS_FLAG for clang 7 / 8 / 9 / 10
run: echo "JSON_FORCED_GLOBAL_COMPILE_OPTIONS=-DJSON_HAS_FILESYSTEM=0;-DJSON_HAS_EXPERIMENTAL_FILESYSTEM=0" >> "$GITHUB_ENV"
if: ${{ matrix.compiler == '7' || matrix.compiler == '8' || matrix.compiler == '9' || matrix.compiler == '10' }}
@@ -239,7 +239,7 @@ jobs:
with:
persist-credentials: false
- name: Get latest CMake and ninja
uses: lukka/get-cmake@e6906078ebd1ccb8ce51ab4626ac46a1b5a517e3 # v4.4.0
uses: lukka/get-cmake@fffaaafeea488556c2c12dad60690008bc1caacb # v4.4.2
- name: Run CMake
run: cmake -S . -B build -DJSON_CI=On
- name: Build
@@ -259,7 +259,7 @@ jobs:
with:
persist-credentials: false
- name: Get latest CMake and ninja
uses: lukka/get-cmake@e6906078ebd1ccb8ce51ab4626ac46a1b5a517e3 # v4.4.0
uses: lukka/get-cmake@fffaaafeea488556c2c12dad60690008bc1caacb # v4.4.2
- name: Run CMake
run: cmake -S . -B build -DJSON_CI=On
- name: Build with libc++
@@ -286,7 +286,7 @@ jobs:
with:
persist-credentials: false
- name: Get latest CMake and ninja
uses: lukka/get-cmake@e6906078ebd1ccb8ce51ab4626ac46a1b5a517e3 # v4.4.0
uses: lukka/get-cmake@fffaaafeea488556c2c12dad60690008bc1caacb # v4.4.2
- name: Run CMake
run: cmake -S . -B build -DJSON_CI=On
- name: Build
@@ -306,7 +306,7 @@ jobs:
# import-std support. Its opt-in token is CMake-version-specific, so pin
# CMake to the version whose token is set in tests/module_cpp20/CMakeLists.txt.
- name: Get pinned CMake and ninja
uses: lukka/get-cmake@e6906078ebd1ccb8ce51ab4626ac46a1b5a517e3 # v4.4.0
uses: lukka/get-cmake@fffaaafeea488556c2c12dad60690008bc1caacb # v4.4.2
with:
cmakeVersion: 4.3.4
# Clang: the std library module is provided by libc++ (the image's libstdc++
@@ -332,7 +332,7 @@ jobs:
with:
persist-credentials: false
- name: Get latest CMake and ninja
uses: lukka/get-cmake@e6906078ebd1ccb8ce51ab4626ac46a1b5a517e3 # v4.4.0
uses: lukka/get-cmake@fffaaafeea488556c2c12dad60690008bc1caacb # v4.4.2
- name: Run CMake
run: cmake -S . -B build -DJSON_CI=On
- name: Build
@@ -347,7 +347,7 @@ jobs:
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Get latest CMake and ninja
uses: lukka/get-cmake@e6906078ebd1ccb8ce51ab4626ac46a1b5a517e3 # v4.4.0
uses: lukka/get-cmake@fffaaafeea488556c2c12dad60690008bc1caacb # v4.4.2
- name: Run CMake
run: cmake -S . -B build -DJSON_CI=On
- name: Build
@@ -359,7 +359,7 @@ jobs:
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Get latest CMake and ninja
uses: lukka/get-cmake@e6906078ebd1ccb8ce51ab4626ac46a1b5a517e3 # v4.4.0
uses: lukka/get-cmake@fffaaafeea488556c2c12dad60690008bc1caacb # v4.4.2
- name: Run CMake
run: cmake -S . -B build -DJSON_CI=On
- name: Build
@@ -369,7 +369,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Harden Runner
uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0
uses: step-security/harden-runner@05e31511f85b41b11d1cf0ef85d0992719546e2c # v2.21.0
with:
egress-policy: audit
@@ -379,7 +379,7 @@ jobs:
with:
persist-credentials: false
- name: Get latest CMake and ninja
uses: lukka/get-cmake@e6906078ebd1ccb8ce51ab4626ac46a1b5a517e3 # v4.4.0
uses: lukka/get-cmake@fffaaafeea488556c2c12dad60690008bc1caacb # v4.4.2
- name: Run CMake
run: cmake -S . -B build -DCMAKE_TOOLCHAIN_FILE=$EMSDK/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake -GNinja
- name: Build
@@ -392,7 +392,7 @@ jobs:
target: [ci_test_examples, ci_test_build_documentation]
steps:
- name: Harden Runner
uses: step-security/harden-runner@bf7454d06d71f1098171f2acdf0cd4708d7b5920 # v2.20.0
uses: step-security/harden-runner@05e31511f85b41b11d1cf0ef85d0992719546e2c # v2.21.0
with:
egress-policy: audit
.github/workflows/windows.yml
+8 -2
View File
@@ -88,7 +88,7 @@ jobs:
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Get latest CMake and ninja
uses: lukka/get-cmake@e6906078ebd1ccb8ce51ab4626ac46a1b5a517e3 # v4.4.0
uses: lukka/get-cmake@fffaaafeea488556c2c12dad60690008bc1caacb # v4.4.2
- name: Set extra CXX_FLAGS for latest std_version
# /wd5285 silences C5285 emitted by the bundled third-party doctest.h, which
# specializes std::tuple (newly diagnosed by the VS2026 v145 toolset)
@@ -153,10 +153,16 @@ jobs:
with:
platform: x64
version: 12.2.0 # https://github.com/egor-tensin/setup-mingw/issues/14
# CMAKE_CXX_FLAGS_DEBUG is overridden to drop the default -g: linking
# test-regression2_cpp20 intermittently fails with "relocation truncated
# to fit: IMAGE_REL_AMD64_SECREL against `.debug_line'" because the
# MinGW linker cannot relocate the debug sections this test produces.
# The tests are only built and run here, so the debug info is not used.
- name: Run CMake
run: cmake -S . -B build ^
-DCMAKE_CXX_COMPILER="C:/Program Files/LLVM/bin/clang++.exe" ^
-DCMAKE_CXX_FLAGS="--target=x86_64-w64-mingw32 -stdlib=libstdc++ -pthread" ^
-DCMAKE_CXX_FLAGS_DEBUG="-g0" ^
-DCMAKE_EXE_LINKER_FLAGS="-lwinpthread" ^
-G"MinGW Makefiles" ^
-DCMAKE_BUILD_TYPE=Debug ^
@@ -193,7 +199,7 @@ jobs:
# import-std support. Its opt-in token is CMake-version-specific, so pin
# CMake to the version whose token is set in tests/module_cpp20/CMakeLists.txt.
- name: Get pinned CMake and ninja
uses: lukka/get-cmake@e6906078ebd1ccb8ce51ab4626ac46a1b5a517e3 # v4.4.0
uses: lukka/get-cmake@fffaaafeea488556c2c12dad60690008bc1caacb # v4.4.2
with:
cmakeVersion: 4.3.4
- name: Run CMake (Debug)
README.md
+6
View File
@@ -42,6 +42,7 @@
- [Specializing enum conversion](#specializing-enum-conversion)
- [Binary formats (BSON, CBOR, MessagePack, UBJSON, and BJData)](#binary-formats-bson-cbor-messagepack-ubjson-and-bjdata)
- [Customers](#customers)
- [Ecosystem](#ecosystem)
- [Supported compilers](#supported-compilers)
- [Integration](#integration)
- [CMake](#cmake)
@@ -1186,6 +1187,11 @@ The library is used in multiple projects, applications, operating systems, etc.
[![logos of customers using the library](docs/mkdocs/docs/images/customers.png)](https://json.nlohmann.me/home/customers/)
## Ecosystem
Beyond projects that use the library, there are third-party projects that build on top of it - schema validators,
language bindings, format converters, and the like. See the curated [Ecosystem](https://json.nlohmann.me/community/ecosystem/) page.
## Supported compilers
Though it's 2026 already, the support for C++11 is still a bit sparse. Currently, the following compilers are known to work:
cmake/ci.cmake
+19 -1
View File
@@ -212,6 +212,24 @@ add_custom_target(ci_test_legacycomparison
COMMENT "Compile and test with legacy discarded value comparison enabled"
)
###############################################################################
# Validate UTF-8 with simdutf.
###############################################################################
add_custom_target(ci_test_simdutf
COMMAND ${CMAKE_COMMAND}
-DCMAKE_BUILD_TYPE=Debug -GNinja
-DJSON_BuildTests=ON -DJSON_TestSimdutf=ON
# simdutf needs C++17, so the library falls back to its scalar validator
# below that: build the suite at C++11 to cover the fallback with the macro
# defined, and at C++17 to run every test against simdutf itself
"-DJSON_TestStandards=11\;17"
-S${PROJECT_SOURCE_DIR} -B${PROJECT_BINARY_DIR}/build_simdutf
COMMAND ${CMAKE_COMMAND} --build ${PROJECT_BINARY_DIR}/build_simdutf
COMMAND cd ${PROJECT_BINARY_DIR}/build_simdutf && ${CMAKE_CTEST_COMMAND} --parallel ${N} --output-on-failure
COMMENT "Compile and test with simdutf UTF-8 validation enabled"
)
###############################################################################
# Enable brace-init copy semantics.
###############################################################################
@@ -294,7 +312,7 @@ file(GLOB_RECURSE INDENT_FILES
${PROJECT_SOURCE_DIR}/tests/src/*.cpp
${PROJECT_SOURCE_DIR}/tests/src/*.hpp
${PROJECT_SOURCE_DIR}/tests/benchmarks/src/benchmarks.cpp
${PROJECT_SOURCE_DIR}/docs/examples/*.cpp
${PROJECT_SOURCE_DIR}/docs/mkdocs/docs/examples/*.cpp
)
set(include_dir ${PROJECT_SOURCE_DIR}/single_include/nlohmann)
docs/mkdocs/docs/api/basic_json/operator[].md
+4 -7
View File
@@ -128,13 +128,10 @@ Strong exception safety: if an exception occurs, the original value stays intact
When the JSON pointer traverses intermediate levels that don't exist at all yet (not just a missing
leaf), each missing level is created as an array or an object depending on whether the corresponding
pointer token is a valid array index: a token that is a nonempty sequence of digits without a leading
`0` (or the token `-`) creates an array, and every other token creates an object. For example, on an
initially `#!json null` value, `/foo/0/0/0` creates nested arrays, while `/foo/one/one/one` creates
nested objects. Tokens such as `01` or the empty token cannot be array indices (cf. RFC 6901, Sect. 4)
and therefore create objects, just as they would if the level already existed as an object. This is not
specified by the JSON Pointer RFC; it is this library's own, intentional disambiguation rule. See also
[JSON Pointer](../../features/json_pointer.md).
pointer token parses as a non-negative integer: a numeric token creates an array, a non-numeric token
creates an object. For example, on an initially `#!json null` value, `/foo/0/0/0` creates nested arrays,
while `/foo/one/one/one` creates nested objects. This is not specified by the JSON Pointer RFC; it is
this library's own, intentional disambiguation rule. See also [JSON Pointer](../../features/json_pointer.md).
## Examples
docs/mkdocs/docs/api/basic_json/to_bjdata.md
+5
View File
@@ -52,6 +52,11 @@ optional, `#!cpp bjdata_version_t::draft2` by default.
Strong guarantee: if an exception is thrown, there are no changes in the JSON value.
## Exceptions
- Throws [`other_error.502`](../../home/exceptions.md#jsonexceptionother_error502) if `use_type` is true and `use_size`
is false.
## Complexity
Linear in the size of the JSON value `j`.
docs/mkdocs/docs/api/basic_json/to_bson.md
+3 -1
View File
@@ -46,7 +46,9 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va
## Complexity
Linear in the size of the JSON value `j`.
Proportional to the size of the JSON value `j` multiplied by its maximum nesting
depth, `O(n × d)`. BSON length prefixes are computed recursively before nested
values are written.
## Examples
docs/mkdocs/docs/api/basic_json/to_ubjson.md
+5
View File
@@ -45,6 +45,11 @@ The exact mapping and its limitations are described on a [dedicated page](../../
Strong guarantee: if an exception is thrown, there are no changes in the JSON value.
## Exceptions
- Throws [`other_error.502`](../../home/exceptions.md#jsonexceptionother_error502) if `use_type` is true and `use_size`
is false.
## Complexity
Linear in the size of the JSON value `j`.
docs/mkdocs/docs/api/macros/index.md
+1
View File
@@ -24,6 +24,7 @@ header. See also the [macro overview page](../../features/macros.md).
- [**JSON_NO_IO**](json_no_io.md) - switch off functions relying on certain C++ I/O headers
- [**JSON_SKIP_UNSUPPORTED_COMPILER_CHECK**](json_skip_unsupported_compiler_check.md) - do not warn about unsupported compilers
- [**JSON_USE_GLOBAL_UDLS**](json_use_global_udls.md) - place user-defined string literals (UDLs) into the global namespace
- [**JSON_USE_SIMDUTF**](json_use_simdutf.md) - use the simdutf library to accelerate UTF-8 validation
## Library version
docs/mkdocs/docs/api/macros/json_use_simdutf.md
+71
View File
@@ -0,0 +1,71 @@
# JSON_USE_SIMDUTF
```cpp
#define JSON_USE_SIMDUTF
```
When defined, the parser validates the UTF-8 content of JSON strings that come from a **contiguous byte input**
(`std::string`, `std::vector<char>`/`<std::uint8_t>`, string literals, `const char*` ranges, …) using the
[simdutf](https://github.com/simdutf/simdutf) library instead of the built-in scalar validator. On text with many
non-ASCII characters (e.g. CJK or emoji) this can validate several times faster.
This is an **opt-in external dependency**. The library itself remains header-only and its behavior is unchanged: the
same input is accepted or rejected either way, and every parse error is reported at the same position with the same
message (simdutf is only used to fast-path *valid* runs; anything it flags falls back to the scalar path so the exact
diagnostic is preserved). Streaming inputs (files, `std::istream`, wide strings, user-defined adapters) always use the
scalar path.
When `JSON_USE_SIMDUTF` is defined you must make the `simdutf.h` header available on the include path and link the
simdutf library. When it is not defined, no simdutf header is included and there is no dependency.
!!! note "Requires C++17"
simdutf requires C++17 and its header rejects older standards with an `#!cpp #error`. The backend is therefore only
compiled in from C++17 on. In C++11 and C++14 the macro has no effect and the scalar validator is used, which
accepts and rejects exactly the same input -- only throughput differs. Setting the macro project-wide is therefore
safe even when some translation units are built with an older standard.
!!! warning "Define consistently"
The macro selects between two definitions of the same inline validation function. It must therefore be defined
identically for **every** translation unit that includes the library; mixing translation units that define it with
ones that do not is an ODR violation. Prefer setting it as a compile definition on the target rather than with
`#!cpp #define` in individual source files.
## Default definition
By default, `#!cpp JSON_USE_SIMDUTF` is not defined and the portable C++11 scalar validator is used.
```cpp
#undef JSON_USE_SIMDUTF
```
## Examples
??? example
The code below enables the simdutf backend for UTF-8 validation.
```cpp
#define JSON_USE_SIMDUTF 1
#include <nlohmann/json.hpp>
...
```
The project must also link against simdutf, e.g. with CMake:
```cmake
target_compile_definitions(your_target PRIVATE JSON_USE_SIMDUTF)
target_link_libraries(your_target PRIVATE simdutf::simdutf)
```
!!! hint "Testing this configuration"
The unit tests can be built against the simdutf backend with the CMake option `JSON_TestSimdutf` (`OFF` by
default), which fetches simdutf and defines `JSON_USE_SIMDUTF` for every test target. The `ci_test_simdutf` target
runs the whole test suite in that configuration.
## Version history
- Added in version 3.13.0.
docs/mkdocs/docs/community/ecosystem.md
+40
View File
@@ -0,0 +1,40 @@
# Ecosystem
The projects below build on top of `nlohmann::json` rather than merely using it - schema validators, language
bindings, format converters, and similar building blocks. The list is not exhaustive, and is curated rather than
automatically generated. If you maintain or know of a project that belongs here,
[please let me know](mailto:mail@nlohmann.me).
For products, applications, and organizations that use the library, see [Customers](../home/customers.md) instead.
## Schema validation
- [**json-schema-validator**](https://github.com/pboettch/json-schema-validator), a JSON Schema (draft 7) validator
with human-readable error messages
## Serialization and reflection
- [**nlohmann_json_reflect**](https://github.com/1261385937/nlohmann_json_reflect), a reflection extension for
(de)serializing nested containers-in-structs-in-containers
## Encodings
- [**base-encode-decode**](https://github.com/saxonnicholls/base-encode-decode), a header-only Base64/32/16/8/4/2
(and DNA/RNA) encoding library, with an adapter that serializes binary data through `nlohmann::json`
## Language bindings and interop
- [**pybind11_json**](https://github.com/pybind/pybind11_json), a bidirectional type caster between
`nlohmann::json` and Python objects for [pybind11](https://github.com/pybind/pybind11) bindings
- [**nanobind_json**](https://github.com/ianhbell/nanobind_json), the same idea for
[nanobind](https://github.com/wjakob/nanobind) bindings
- [**nlohmann_json_qt**](https://github.com/dpurgin/nlohmann_json_qt), deserialization helpers for Qt types
(`QString`, `QUrl`, `QDateTime`, `QVector`, ...) from `nlohmann::json`
- [**vulkan2json**](https://github.com/Fadis/vulkan2json), serialization and deserialization of Vulkan API structs
## Format converters
- [**tojson**](https://github.com/mircodz/tojson), a header-only converter between YAML/XML documents and
`nlohmann::json`
- [**json2xml**](https://github.com/testillano/json2xml), a header-only converter from `nlohmann::json` to XML for
simple configuration documents
docs/mkdocs/docs/community/index.md
+1
View File
@@ -1,5 +1,6 @@
# Community
- [Ecosystem](ecosystem.md) - third-party projects built on top of this library
- [Code of Conduct](code_of_conduct.md) - the rules and norms of this project
- [Contribution Guidelines](contribution_guidelines.md) - guidelines how to contribute to this project
- [Governance](governance.md) - the governance model of this project
docs/mkdocs/docs/community/quality_assurance.md
+1
View File
@@ -66,6 +66,7 @@ Note: Some modern features (like C++20 ranges or filesystem support) may be disa
| Clang 20.1.1 | x86_64 | Ubuntu 22.04.1 LTS | GitHub |
| Clang 20.1.8 with GNU-like command-line | x86_64 | Windows Server 2022 (Build 20348) | GitHub |
| Clang 21.1.8 | x86_64 | Ubuntu 22.04.1 LTS | GitHub |
| Clang 22.1.8 | x86_64 | Ubuntu 22.04.1 LTS | GitHub |
| CUDA 11.8.0 (nvcc) | x86_64 | Ubuntu 22.04 LTS | GitHub |
| CUDA 12.1.1 (nvcc) | x86_64 | Ubuntu 22.04 LTS | GitHub |
| CUDA 12.6.3 (nvcc) | x86_64 | Ubuntu 22.04 LTS | GitHub |
docs/mkdocs/docs/examples/parser_callback_t.cpp
+3 -3
View File
@@ -9,13 +9,13 @@ int main()
auto text = R"({"IDs": [116, 943], "Width": 800})";
// discard the array when the parser reads its opening bracket
json j_array_start = json::parse(text, [](int /*depth*/, json::parse_event_t event, json & /*parsed*/)
json j_array_start = json::parse(text, [](int /*depth*/, json::parse_event_t event, json& /*parsed*/)
{
return event != json::parse_event_t::array_start;
});
// discard the same array when the parser reads its closing bracket
json j_array_end = json::parse(text, [](int /*depth*/, json::parse_event_t event, json & /*parsed*/)
json j_array_end = json::parse(text, [](int /*depth*/, json::parse_event_t event, json& /*parsed*/)
{
return event != json::parse_event_t::array_end;
});
@@ -33,7 +33,7 @@ int main()
});
// discard the top-level object
json j_root = json::parse(text, [](int /*depth*/, json::parse_event_t event, json & /*parsed*/)
json j_root = json::parse(text, [](int /*depth*/, json::parse_event_t event, json& /*parsed*/)
{
return event != json::parse_event_t::object_end;
});
docs/mkdocs/docs/features/binary_formats/bjdata.md
+13 -3
View File
@@ -116,9 +116,19 @@ The library uses the following mapping from JSON values types to BJData types ac
```
Likewise, when a JSON object in the above form is serialized using
[`to_bjdata`](../../api/basic_json/to_bjdata.md), it is automatically converted into a compact BJData ND-array. The
only exception is, that when the 1-dimensional vector stored in `"_ArraySize_"` contains a single integer or two
integers with one being 1, a regular 1-D optimized array is generated.
[`to_bjdata`](../../api/basic_json/to_bjdata.md), it is automatically converted into a compact BJData ND-array. When
the 1-dimensional vector stored in `"_ArraySize_"` contains a single integer or two integers with one being 1, a
regular 1-D optimized array is generated instead.
An object is only converted if the annotation actually describes a packed array; otherwise it is serialized as a
regular JSON object. This requires all of the following:
- `"_ArrayType_"` is one of `uint8`, `int8`, `uint16`, `int16`, `uint32`, `int32`, `uint64`, `int64`, `single`,
`double`, `char`, or `byte`,
- every entry of `"_ArraySize_"` is a non-negative integer, and their product is representable as a `std::size_t`,
- `"_ArrayData_"` holds exactly that many elements, and
- every element of `"_ArrayData_"` is a number of the kind named by `"_ArrayType_"` (a floating-point number for
`single` and `double`, an integer otherwise).
The current version of this library does not yet support automatic detection of and conversion from a nested JSON
array input to a BJData ND-array.
docs/mkdocs/docs/features/binary_formats/bson.md
+11
View File
@@ -98,6 +98,17 @@ The library maps BSON record types to JSON value types as follows:
This library deserializes BSON type `0x11` (Timestamp) as a `number_unsigned` value. The 64-bit value is preserved,
but the Timestamp type information is not.
!!! warning "Lenient BSON input handling"
The BSON reader is lenient in a few areas where the BSON specification is more restrictive:
- array element keys are not checked against the required decimal sequence (`0`, `1`, `2`, ...),
- any non-zero byte is accepted as `true` for the boolean type, and
- the payload for binary subtype `0x02` is returned as-is, including its inner length prefix.
If BSON input must be validated for strict specification compliance, validate it separately before passing it to
`from_bson()`.
??? example
```cpp
docs/mkdocs/docs/features/binary_formats/cbor.md
+3 -6
View File
@@ -160,14 +160,11 @@ The library maps CBOR types to JSON value types as follows:
The mapping is **incomplete** in the sense that not all CBOR types can be converted to a JSON value. The following CBOR types are not supported and will yield parse errors:
- date/time (0xC0..0xC1)
- bignum (0xC2..0xC3)
- decimal fraction (0xC4)
- bigfloat (0xC5)
- expected conversions (0xD5..0xD7)
- simple values (0xE0..0xF3, 0xF8)
- undefined (0xF7)
Tagged items (0xC0..0xDB) are not interpreted either; see the note on tagged items below.
!!! warning "Negative integer overflow"
CBOR negative integers (major type 1) are decoded as `-1 - n`. If the encoded magnitude `n` is too large for the
@@ -181,7 +178,7 @@ The library maps CBOR types to JSON value types as follows:
!!! warning "Tagged items"
Tagged items will throw a parse error by default. They can be ignored by passing `cbor_tag_handler_t::ignore` to function `from_cbor`. They can be stored by passing `cbor_tag_handler_t::store` to function `from_cbor`.
Tagged items (0xC0..0xDB) will throw a parse error by default. They can be ignored by passing `cbor_tag_handler_t::ignore` to function `from_cbor`, in which case the tag is skipped and the enclosed data item is parsed on its own. They can be stored by passing `cbor_tag_handler_t::store` to function `from_cbor`. Note that no tag is ever interpreted: for instance, a text string tagged with tag 0 (date/time) stays a string.
??? example
docs/mkdocs/docs/features/conversions.md
+24
View File
@@ -54,6 +54,30 @@ json j = {1.0, "hello", 42};
auto t = j.get<std::tuple<double, std::string, int>>(); // {1.0, "hello", 42}
```
!!! warning "Serializing a `std::pair`/`std::tuple` whose every element is a string-keyed pair"
When *every* element of a `#!cpp std::pair` or `#!cpp std::tuple` is itself a two-element array whose first
element is a string (for example `#!cpp std::pair<std::string, int>`), serializing it produces a JSON **object**
instead of the expected array:
```cpp
using kv = std::pair<std::string, int>;
json j = std::pair<kv, kv>{{"a", 1}, {"b", 2}}; // {"a":1,"b":2}, not [["a",1],["b",2]]
```
This is a consequence of the [brace-initializer object-detection rule](creating_values.md): the same rule that
lets `#!cpp json{{"a", 1}, {"b", 2}}` create an object also fires here. The resulting object cannot be read back
into the original type (`#!cpp get<std::pair<kv, kv>>()` throws [`type_error.302`](../home/exceptions.md#jsonexceptiontype_error302)),
and duplicate keys collapse into one, losing elements. This only affects `#!cpp std::pair`/`#!cpp std::tuple`
themselves; a `#!cpp std::vector<std::pair<std::string, int>>`, or a pair/tuple with at least one element that is
not a string-keyed pair, serializes to an array as expected. To force an array, build one explicitly from the
elements with [`array`](../api/basic_json/array.md):
```cpp
std::pair<kv, kv> p{{"a", 1}, {"b", 2}};
json a = json::array({p.first, p.second}); // [["a",1],["b",2]]
```
!!! info "Extracting references into a tuple"
A tuple type may also hold references (e.g. `#!cpp std::tuple<double&, std::string&>`) to avoid copying: `get`
docs/mkdocs/docs/features/macros.md
+8
View File
@@ -137,6 +137,14 @@ behavior is deprecated and switched off (`0`) by default.
See [full documentation of `JSON_USE_LEGACY_DISCARDED_VALUE_COMPARISON`](../api/macros/json_use_legacy_discarded_value_comparison.md).
## `JSON_USE_SIMDUTF`
When defined, UTF-8 validation of JSON strings read from contiguous byte input is delegated to the
[simdutf](https://github.com/simdutf/simdutf) library instead of the built-in scalar validator. This is an opt-in
external dependency and is not defined by default.
See [full documentation of `JSON_USE_SIMDUTF`](../api/macros/json_use_simdutf.md).
## `NLOHMANN_DEFINE_TYPE_*(...)`, `NLOHMANN_DEFINE_DERIVED_TYPE_*(...)`
The library defines 12 macros to simplify the serialization/deserialization of types. See the page on
docs/mkdocs/docs/home/exceptions.md
+16
View File
@@ -965,3 +965,19 @@ A JSON Patch operation 'test' failed. The unsuccessful operation is also printed
```
[json.exception.other_error.501] unsuccessful: {"op":"test","path":"/baz","value":"bar"}
```
### json.exception.other_error.502
[`to_ubjson`](../api/basic_json/to_ubjson.md) and [`to_bjdata`](../api/basic_json/to_bjdata.md) were called with
`use_type = true` but `use_size = false`. UBJSON requires a size marker (`#`) after a type marker (`$`).
!!! failure "Example message"
```
[json.exception.other_error.502] use_type requires use_size = true
```
!!! note
This exception was added in version 3.13.0. Before that, debug builds aborted on an assertion and release builds
wrote a `$` marker without `#`, which [`from_ubjson`](../api/basic_json/from_ubjson.md) then rejected.
docs/mkdocs/mkdocs.yml
+2
View File
@@ -296,6 +296,7 @@ nav:
- 'JSON_USE_GLOBAL_UDLS': api/macros/json_use_global_udls.md
- 'JSON_USE_IMPLICIT_CONVERSIONS': api/macros/json_use_implicit_conversions.md
- 'JSON_USE_LEGACY_DISCARDED_VALUE_COMPARISON': api/macros/json_use_legacy_discarded_value_comparison.md
- 'JSON_USE_SIMDUTF': api/macros/json_use_simdutf.md
- 'NLOHMANN_DEFINE_DERIVED_TYPE_INTRUSIVE, NLOHMANN_DEFINE_DERIVED_TYPE_INTRUSIVE_WITH_DEFAULT, NLOHMANN_DEFINE_DERIVED_TYPE_INTRUSIVE_ONLY_SERIALIZE, NLOHMANN_DEFINE_DERIVED_TYPE_NON_INTRUSIVE, NLOHMANN_DEFINE_DERIVED_TYPE_NON_INTRUSIVE_WITH_DEFAULT, NLOHMANN_DEFINE_DERIVED_TYPE_NON_INTRUSIVE_ONLY_SERIALIZE': api/macros/nlohmann_define_derived_type.md
- 'NLOHMANN_DEFINE_TYPE_INTRUSIVE, NLOHMANN_DEFINE_TYPE_INTRUSIVE_WITH_DEFAULT, NLOHMANN_DEFINE_TYPE_INTRUSIVE_ONLY_SERIALIZE': api/macros/nlohmann_define_type_intrusive.md
- 'NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE, NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE_WITH_DEFAULT, NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE_ONLY_SERIALIZE': api/macros/nlohmann_define_type_non_intrusive.md
@@ -308,6 +309,7 @@ nav:
- 'NLOHMANN_JSON_VERSION_MAJOR, NLOHMANN_JSON_VERSION_MINOR, NLOHMANN_JSON_VERSION_PATCH': api/macros/nlohmann_json_version_major.md
- Community:
- community/index.md
- community/ecosystem.md
- "Code of Conduct": community/code_of_conduct.md
- community/contribution_guidelines.md
- community/quality_assurance.md
docs/mkdocs/requirements.txt
+2 -2
View File
@@ -1,7 +1,7 @@
wheel==0.47.0
wheel==0.48.0
mkdocs==1.6.1 # documentation framework
mkdocs-git-revision-date-localized-plugin==1.5.3 # plugin "git-revision-date-localized"
mkdocs-git-revision-date-localized-plugin==1.5.4 # plugin "git-revision-date-localized"
mkdocs-material==9.7.7 # theme for mkdocs
mkdocs-material-extensions==1.3.1 # extensions
mkdocs-minify-plugin==0.8.0 # plugin "minify"
include/nlohmann/detail/input/binary_reader.hpp
+23 -11
View File
@@ -465,15 +465,6 @@ class binary_reader
// CBOR //
//////////
/*!
@param[in] get_char whether a new character should be retrieved from the
input (true) or whether the last read character should
be considered instead (false)
@param[in] tag_handler how CBOR tags should be treated
@return whether a valid CBOR value was passed to the SAX parser
*/
template<typename NumberType>
bool get_cbor_negative_integer()
{
@@ -492,6 +483,14 @@ class binary_reader
return sax->number_integer(static_cast<number_integer_t>(-1) - static_cast<number_integer_t>(number));
}
/*!
@param[in] get_char whether a new character should be retrieved from the
input (true) or whether the last read character should
be considered instead (false)
@param[in] tag_handler how CBOR tags should be treated
@return whether a valid CBOR value was passed to the SAX parser
*/
bool parse_cbor_internal(const bool get_char,
const cbor_tag_handler_t tag_handler)
{
@@ -774,7 +773,13 @@ class binary_reader
case 0xBF: // map (indefinite length)
return get_cbor_object(detail::unknown_size(), tag_handler);
case 0xC6: // tagged item
case 0xC0: // tagged item
case 0xC1:
case 0xC2:
case 0xC3:
case 0xC4:
case 0xC5:
case 0xC6:
case 0xC7:
case 0xC8:
case 0xC9:
@@ -789,6 +794,9 @@ class binary_reader
case 0xD2:
case 0xD3:
case 0xD4:
case 0xD5:
case 0xD6:
case 0xD7:
case 0xD8: // tagged item (1 byte follows)
case 0xD9: // tagged item (2 bytes follow)
case 0xDA: // tagged item (4 bytes follow)
@@ -1988,7 +1996,11 @@ class binary_reader
{
if (get_char)
{
get(); // TODO(niels): may we ignore N here?
// no get_ignore_noop() here: the byte read next must be a string
// length type specification, and a no-op ('N') is not valid in
// that position. No-ops at positions where a value may appear are
// already consumed by the callers via get_ignore_noop().
get();
}
if (JSON_HEDLEY_UNLIKELY(!unexpect_eof(input_format, "value")))
include/nlohmann/detail/input/input_adapters.hpp
+137 -23
View File
@@ -155,11 +155,31 @@ class input_stream_adapter
// General-purpose iterator-based adapter. It might not be as fast as
// theoretically possible for some containers, but it is extremely versatile.
// SentinelType defaults to IteratorType for backward compatibility, but may
// be a different type (e.g., a C++20 sentinel or counted_iterator).
// SentinelType defaults to IteratorType for backward compatibility, but may be
// a different type, e.g. a C++20 sentinel such as std::default_sentinel_t when
// IteratorType is a std::counted_iterator.
template<typename IteratorType, typename SentinelType = IteratorType>
class iterator_input_adapter
{
// Whether the number of elements between two positions can be computed in
// O(1): either the iterator and the sentinel have the same type (plain
// std::distance) or, in C++20, the sentinel is a sized sentinel for the
// iterator (std::ranges::distance), e.g. std::default_sentinel_t paired
// with std::counted_iterator.
//
// JSON_HAS_RANGES gates the C++20 branch: on standard libraries with an
// incomplete <ranges> (libstdc++ < 11, see #4440) evaluating
// std::contiguous_iterator on a std::counted_iterator is a hard error
// instead of yielding false, and these traits are instantiated for every
// adapter. Such toolchains fall back to the pointer-only test and simply
// use the byte-at-a-time scanner.
static constexpr bool sentinel_is_sized =
#if JSON_HAS_RANGES && defined(__cpp_lib_concepts) && defined(JSON_HAS_CPP_20)
std::is_same<IteratorType, SentinelType>::value || std::sized_sentinel_for<SentinelType, IteratorType>;
#else
std::is_same<IteratorType, SentinelType>::value;
#endif
public:
using char_type = typename std::iterator_traits<IteratorType>::value_type;
@@ -171,7 +191,7 @@ class iterator_input_adapter
// in wide_string_input_adapter, which does not expose this).
static constexpr bool supports_seek =
std::is_same<typename std::iterator_traits<IteratorType>::iterator_category, std::random_access_iterator_tag>::value
&& std::is_same<IteratorType, SentinelType>::value
&& sentinel_is_sized
&& sizeof(char_type) == 1;
iterator_input_adapter(IteratorType first, SentinelType last)
@@ -219,30 +239,60 @@ class iterator_input_adapter
private:
// whether IteratorType refers to a contiguous range and therefore supports
// a std::memcpy fast path (pointers always do; in C++20 we can also detect
// library iterators such as those of std::vector and std::string).
// Computing the available element count needs either same-type iterators
// (plain std::distance) or, in C++20, a sized sentinel (std::ranges::distance),
// e.g. std::counted_iterator paired with std::default_sentinel_t.
static constexpr bool iterator_is_contiguous =
#if defined(__cpp_lib_concepts) && defined(JSON_HAS_CPP_20)
(std::is_same<IteratorType, SentinelType>::value || std::sized_sentinel_for<SentinelType, IteratorType>)
&& (std::contiguous_iterator<IteratorType> || std::is_pointer<IteratorType>::value);
// library iterators such as those of std::vector and std::string). The
// available element count must also be computable in O(1), hence
// sentinel_is_sized.
static constexpr bool iterator_is_contiguous = sentinel_is_sized &&
#if JSON_HAS_RANGES && defined(__cpp_lib_concepts) && defined(JSON_HAS_CPP_20)
(std::contiguous_iterator<IteratorType> || std::is_pointer<IteratorType>::value);
#else
std::is_same<IteratorType, SentinelType>::value && std::is_pointer<IteratorType>::value;
std::is_pointer<IteratorType>::value;
#endif
// number of unread elements in [current, end)
std::size_t remaining_count() const
{
#if JSON_HAS_RANGES && defined(__cpp_lib_concepts) && defined(JSON_HAS_CPP_20)
// std::ranges::distance also supports sized sentinels of a different
// type (e.g. std::counted_iterator + std::default_sentinel_t)
return static_cast<std::size_t>(std::ranges::distance(current, end));
#else
return static_cast<std::size_t>(std::distance(current, end));
#endif
}
public:
// Whether the remaining input is a single contiguous block of 1-byte
// elements that the lexer can inspect directly (used for the SWAR string
// fast path).
static constexpr bool supports_bulk_scan =
iterator_is_contiguous && sizeof(char_type) == 1;
// Pointer to the next unread element; only valid when bulk_remaining() > 0.
const char_type* bulk_data() const
{
return &*current;
}
// Number of unread elements available as one contiguous block.
std::size_t bulk_remaining() const
{
return remaining_count();
}
// Consume @a n elements previously inspected via bulk_data().
void bulk_skip(std::size_t n)
{
std::advance(current, static_cast<typename std::iterator_traits<IteratorType>::difference_type>(n));
}
private:
// contiguous fast path: bulk copy the remaining range with std::memcpy
template<class T>
std::size_t get_elements_impl(T* dest, std::size_t count, std::true_type /*contiguous*/)
{
const std::size_t wanted = count * sizeof(T);
#if defined(__cpp_lib_concepts) && defined(JSON_HAS_CPP_20)
// std::ranges::distance also supports sized sentinels of a different
// type (e.g. std::counted_iterator + std::default_sentinel_t)
const std::size_t available = static_cast<std::size_t>(std::ranges::distance(current, end)) * sizeof(char_type);
#else
const std::size_t available = static_cast<std::size_t>(std::distance(current, end)) * sizeof(char_type);
#endif
const std::size_t available = remaining_count() * sizeof(char_type);
const std::size_t copied = (std::min)(wanted, available);
if (JSON_HEDLEY_LIKELY(copied != 0))
{
@@ -345,8 +395,12 @@ struct wide_string_input_helper<BaseInputAdapter, 4>
}
else
{
// unknown character
utf8_bytes[0] = static_cast<std::char_traits<char>::int_type>(wc);
// A code point above U+10FFFF has no UTF-8 encoding. Passing the
// unit through would narrow it to int, where 0xFFFFFFFF becomes
// char_traits<char>::eof() and would end the input silently, so
// emit a byte that is never valid UTF-8 and let the decoder
// reject it.
utf8_bytes[0] = 0xFF;
utf8_bytes_filled = 1;
}
}
@@ -566,6 +620,46 @@ typename iterator_input_adapter_factory<IteratorType, SentinelType>::adapter_typ
return factory_type::create(first, last);
}
// The element type a container's data() points at, cv-qualifiers removed.
// Ill-formed - and therefore SFINAE-friendly - for types without data().
template<typename ContainerType>
using container_data_t = typename std::remove_cv<typename std::remove_pointer <
decltype(std::declval<const ContainerType&>().data()) >::type >::type;
// The container's own element type, cv-qualifiers removed. It is looked up on
// the bare type so it is also found when ContainerType is deduced as a
// reference by the forwarding-reference overload below.
template<typename ContainerType>
using container_value_t = typename std::remove_cv <
typename std::remove_cv<typename std::remove_reference<ContainerType>::type>::type::value_type >::type;
// Detect a container that stores its elements contiguously as single bytes
// (std::string, std::vector<char/unsigned char>, std::array<char, N>,
// std::string_view, ...). Such inputs are wrapped in a pointer-based adapter so
// they benefit from the contiguous fast paths (bulk string scanning, memcpy for
// binary formats) in every C++ standard - not only in C++20, where the standard
// library iterators model std::contiguous_iterator and are detected directly.
//
// data() and size() on their own would be duck typing: they say nothing about
// size() counting the units data() points at, and reading [data(), data() +
// size()) as bytes would be wrong for a type where it does not. Requiring the
// container's own value_type to be that same single-byte element ties the two
// together; every contiguous standard container satisfies it. Anything else
// keeps the iterator-based adapter, which is always correct - only slower.
template<typename ContainerType, typename = void>
struct is_contiguous_byte_container : std::false_type {};
template<typename ContainerType>
struct is_contiguous_byte_container < ContainerType, void_t <
container_data_t<ContainerType>,
container_value_t<ContainerType>,
decltype(std::declval<const ContainerType&>().size()) >>
: std::integral_constant < bool,
std::is_pointer<decltype(std::declval<const ContainerType&>().data())>::value&&
std::is_integral<container_data_t<ContainerType>>::value&&
sizeof(container_data_t<ContainerType>) == 1 &&
std::is_same<container_data_t<ContainerType>, container_value_t<ContainerType>>::value > {};
// Convenience shorthand from container to iterator
// Enables ADL on begin(container) and end(container)
// Encloses the using declarations in namespace for not to leak them to outside scope
@@ -593,12 +687,32 @@ struct container_input_adapter_factory< ContainerType,
} // namespace container_input_adapter_factory_impl
template<typename ContainerType>
typename container_input_adapter_factory_impl::container_input_adapter_factory<ContainerType>::adapter_type input_adapter(ContainerType&& container)
// General container path (iterator-based). Contiguous single-byte containers
// are excluded here and routed through the pointer-based overload below.
template < typename ContainerType,
enable_if_t < !is_contiguous_byte_container<ContainerType>::value, int > = 0 >
typename container_input_adapter_factory_impl::container_input_adapter_factory<ContainerType>::adapter_type input_adapter(ContainerType && container)
{
return container_input_adapter_factory_impl::container_input_adapter_factory<ContainerType>::create(std::forward<ContainerType>(container));
}
// Contiguous single-byte containers (std::string, std::vector<char>, ...) are
// wrapped in a pointer-based adapter so the contiguous fast paths apply in every
// standard. The pointer keeps the container's own element type (const char* for
// std::string, const std::uint8_t* for std::vector<std::uint8_t>, ...), so the
// resulting char_type - and therefore the parsing behavior - is byte-for-byte
// identical to the iterator-based path; only the raw pointer additionally
// enables the bulk fast paths. The container outlives the adapter for the whole
// parse (temporaries live until the end of the full expression), exactly as the
// iterators it replaces did.
template < typename ContainerType,
enable_if_t < is_contiguous_byte_container<ContainerType>::value, int > = 0 >
auto input_adapter(const ContainerType& container)
-> decltype(input_adapter(container.data(), container.data() + container.size()))
{
return input_adapter(container.data(), container.data() + container.size());
}
// specialization for std::string
using string_input_adapter_type = decltype(input_adapter(std::declval<std::string>()));
include/nlohmann/detail/input/json_sax.hpp
+8 -4
View File
@@ -370,8 +370,10 @@ class json_sax_dom_parser
case value_t::string:
{
// include the length of the quotes, which is 2
v.start_position = v.end_position - v.m_data.m_value.string->size() - 2;
// escape sequences make the token longer than the value it
// parses to, so the start position cannot be derived from
// the value; use the offset the lexer recorded instead
v.start_position = m_lexer_ref->get_token_start_position();
break;
}
@@ -769,8 +771,10 @@ class json_sax_dom_callback_parser
case value_t::string:
{
// include the length of the quotes, which is 2
v.start_position = v.end_position - v.m_data.m_value.string->size() - 2;
// escape sequences make the token longer than the value it
// parses to, so the start position cannot be derived from
// the value; use the offset the lexer recorded instead
v.start_position = m_lexer_ref->get_token_start_position();
break;
}
include/nlohmann/detail/input/lexer.hpp
+312 -27
View File
@@ -19,7 +19,9 @@
#include <vector> // vector
#include <nlohmann/detail/input/input_adapters.hpp>
#include <nlohmann/detail/input/number_parse.hpp>
#include <nlohmann/detail/input/position_t.hpp>
#include <nlohmann/detail/input/string_scan.hpp>
#include <nlohmann/detail/macro_scope.hpp>
#include <nlohmann/detail/meta/type_traits.hpp>
@@ -125,6 +127,25 @@ constexpr bool input_adapter_supports_seek(std::false_type /*detected*/)
return false;
}
// Detect whether an input adapter exposes a contiguous byte block that the
// lexer can scan directly (see iterator_input_adapter::supports_bulk_scan).
// Adapters without the flag - file, stream, wide-string, user-defined - fall
// back to the character-at-a-time string scanner.
template<typename InputAdapterType>
using detect_supports_bulk_scan = decltype(InputAdapterType::supports_bulk_scan);
template<typename InputAdapterType>
constexpr bool input_adapter_supports_bulk_scan(std::true_type /*detected*/)
{
return InputAdapterType::supports_bulk_scan;
}
template<typename InputAdapterType>
constexpr bool input_adapter_supports_bulk_scan(std::false_type /*detected*/)
{
return false;
}
/*!
@brief lexical analysis
@@ -146,6 +167,14 @@ class lexer : public lexer_base<BasicJsonType>
static constexpr bool lazy_token_string =
input_adapter_supports_seek<InputAdapterType>(is_detected<detect_supports_seek, InputAdapterType> {});
/// whether string scanning may bulk-consume runs of ordinary characters
/// directly from a contiguous input buffer (SWAR fast path). This requires
/// the token to be reconstructible lazily (lazy_token_string), so bypassing
/// the per-character capture in get() cannot lose error diagnostics.
static constexpr bool bulk_scan =
lazy_token_string
&& input_adapter_supports_bulk_scan<InputAdapterType>(is_detected<detect_supports_bulk_scan, InputAdapterType> {});
public:
using token_type = typename lexer_base<BasicJsonType>::token_type;
@@ -265,6 +294,40 @@ class lexer : public lexer_base<BasicJsonType>
return true;
}
/// contiguous input: bulk-append the run of ordinary characters and complete
/// well-formed UTF-8 sequences starting at the current read position, leaving
/// the first byte that needs individual handling (the closing quote, an
/// escape, a control character, or an ill-formed UTF-8 byte) for get()
void scan_string_bulk(std::true_type /*bulk*/)
{
// a pending unget must be consumed through the normal path first
if (next_unget)
{
return;
}
const std::size_t remaining = ia.bulk_remaining();
if (remaining == 0)
{
return;
}
const auto* const data = reinterpret_cast<const unsigned char*>(ia.bulk_data());
const std::size_t pos = string_bulk_run(data, remaining);
if (pos == 0)
{
return;
}
token_buffer.append(reinterpret_cast<const typename string_t::value_type*>(data), pos);
ia.bulk_skip(pos);
// the run contains no newline (all bytes < 0x20 are treated as special),
// so only the flat character counters advance
position.chars_read_total += pos;
position.chars_read_current_line += pos;
}
/// streaming input: no bulk fast path
void scan_string_bulk(std::false_type /*bulk*/) const noexcept {}
/*!
@brief scan a string literal
@@ -290,6 +353,10 @@ class lexer : public lexer_base<BasicJsonType>
while (true)
{
// bulk-consume ordinary characters from contiguous input, then
// handle the next special byte through the switch below
scan_string_bulk(std::integral_constant<bool, bulk_scan> {});
// get the next character
switch (get())
{
@@ -1279,45 +1346,78 @@ scan_number_done:
// we are done scanning a number)
unget();
char* endptr = nullptr; // NOLINT(misc-const-correctness,cppcoreguidelines-pro-type-vararg,hicpp-vararg)
errno = 0;
return convert_number(number_type);
}
// try to parse integers first and fall back to floats
/*!
@brief convert an already-validated integer token to its value
The digit sequence in [first, last) has been validated by the caller, so a
dedicated parser can avoid the locale/errno overhead of std::strtoull.
@return the token type on success; token_type::uninitialized if @a
number_type is not an integer type or the value does not fit, in
which case the caller falls back to the floating-point conversion
(matching the previous std::strtoull/std::strtoll behavior)
*/
token_type convert_integer(token_type number_type, const char* first, const char* last)
{
if (number_type == token_type::value_unsigned)
{
const auto x = std::strtoull(token_buffer.data(), &endptr, 10);
// we checked the number format before
JSON_ASSERT(endptr == token_buffer.data() + token_buffer.size());
if (errno != ERANGE)
if (parse_integer_unsigned(first, last, value_unsigned))
{
value_unsigned = static_cast<number_unsigned_t>(x);
if (value_unsigned == x)
{
return token_type::value_unsigned;
}
return token_type::value_unsigned;
}
}
else if (number_type == token_type::value_integer)
{
const auto x = std::strtoll(token_buffer.data(), &endptr, 10);
// we checked the number format before
JSON_ASSERT(endptr == token_buffer.data() + token_buffer.size());
if (errno != ERANGE)
if (parse_integer_signed(first, last, value_integer))
{
value_integer = static_cast<number_integer_t>(x);
if (value_integer == x)
{
return token_type::value_integer;
}
return token_type::value_integer;
}
}
return token_type::uninitialized;
}
/*!
@brief convert the number text in token_buffer to its value and token type
The digit sequence in token_buffer has already been validated (by the
scan_number() state machine or by the contiguous fast path) and holds the
locale decimal point in place of '.'. Integers are parsed first and fall
back to floating point on overflow. This is shared so both scanners produce
identical results.
*/
token_type convert_number(token_type number_type)
{
const char* const num_begin = token_buffer.data();
const char* const num_end = num_begin + token_buffer.size();
if (number_type != token_type::value_float)
{
const token_type integer_result = convert_integer(number_type, num_begin, num_end);
if (integer_result != token_type::uninitialized)
{
return integer_result;
}
}
// this code is reached if we parse a floating-point number or if an
// integer conversion above failed
// integer conversion above overflowed. Prefer std::from_chars
// (Eisel-Lemire, locale-independent, correctly rounded) when available;
// otherwise the exact Clinger fast path (double only); otherwise the
// locale-aware strtof/strtod.
if (parse_float_from_chars(num_begin, num_end, value_float))
{
return token_type::value_float;
}
if (parse_float_fast(num_begin, num_end, decimal_point_char, value_float))
{
return token_type::value_float;
}
char* endptr = nullptr; // NOLINT(misc-const-correctness,cppcoreguidelines-pro-type-vararg,hicpp-vararg)
strtof(value_float, token_buffer.data(), &endptr);
// we checked the number format before
@@ -1326,6 +1426,156 @@ scan_number_done:
return token_type::value_float;
}
/*!
@brief contiguous fast path for scanning a number
Parses the whole number token straight from the input buffer, avoiding the
per-character get()/add() of scan_number(). On success it fills token_buffer
(with the locale decimal point substituted, as scan_number() does) and
returns the token type. On anything it does not fully recognize as a
well-formed number it makes no state change and returns
token_type::uninitialized, so the caller falls back to scan_number(), which
then produces the exact diagnostic. @a current is the first digit or the
leading minus (already read); the remaining bytes are taken from the adapter.
*/
token_type scan_number_bulk_contiguous()
{
// a pending unget offsets the buffer position from current; fall back
if (next_unget)
{
return token_type::uninitialized;
}
const std::size_t rem = ia.bulk_remaining();
if (rem == 0)
{
// the first digit is the last input byte; let scan_number() finish
return token_type::uninitialized;
}
// the byte before the next unread one is current (contiguous input)
const char* const data = reinterpret_cast<const char*>(ia.bulk_data()) - 1;
const std::size_t avail = rem + 1;
// validate + classify the number extent (mirrors scan_number()'s grammar)
std::size_t i = 0;
std::size_t dot_index = std::string::npos;
token_type number_type = token_type::value_unsigned;
if (data[0] == '-')
{
number_type = token_type::value_integer;
i = 1;
if (i >= avail)
{
return token_type::uninitialized;
}
}
if (data[i] == '0')
{
++i;
}
else if (data[i] >= '1' && data[i] <= '9')
{
++i;
while (i < avail && data[i] >= '0' && data[i] <= '9')
{
++i;
}
}
else
{
return token_type::uninitialized;
}
if (i < avail && data[i] == '.')
{
number_type = token_type::value_float;
dot_index = i;
++i;
if (i >= avail || !(data[i] >= '0' && data[i] <= '9'))
{
return token_type::uninitialized;
}
while (i < avail && data[i] >= '0' && data[i] <= '9')
{
++i;
}
}
if (i < avail && (data[i] == 'e' || data[i] == 'E'))
{
number_type = token_type::value_float;
++i;
if (i < avail && (data[i] == '+' || data[i] == '-'))
{
++i;
}
if (i >= avail || !(data[i] >= '0' && data[i] <= '9'))
{
return token_type::uninitialized;
}
while (i < avail && data[i] >= '0' && data[i] <= '9')
{
++i;
}
}
const std::size_t len = i;
// reset() records where this token starts (for diagnostics), so it has
// to run before the input position advances below
reset();
// An integer token needs no token_buffer: the SAX callbacks for
// number_integer/number_unsigned take only the value, and the overflow
// diagnostic rebuilds the text from the input. Convert straight from the
// input buffer and leave token_buffer empty. (JSON_DIAGNOSTIC_POSITIONS
// derives a number's start position from get_string().size(), so there
// the token still has to be materialized.)
#if !JSON_DIAGNOSTIC_POSITIONS
if (number_type != token_type::value_float)
{
const token_type integer_result = convert_integer(number_type, data, data + len);
if (JSON_HEDLEY_LIKELY(integer_result != token_type::uninitialized))
{
ia.bulk_skip(len - 1);
position.chars_read_total += (len - 1);
position.chars_read_current_line += (len - 1);
return integer_result;
}
// The value does not fit an integer, so this token converts as a
// float. Recording that here keeps convert_number() below from
// repeating the integer attempt that just failed.
number_type = token_type::value_float;
}
#endif
// materialize the token exactly as scan_number() would, substituting the
// locale decimal point so convert_number()'s strtof fallback stays valid.
// reset() already cleared token_buffer, so append() fills it (assign() is
// avoided because custom string_t types need not provide it)
token_buffer.append(reinterpret_cast<const typename string_t::value_type*>(data), len);
if (dot_index != std::string::npos)
{
token_buffer[dot_index] = static_cast<typename string_t::value_type>(decimal_point_char);
decimal_point_position = dot_index;
}
ia.bulk_skip(len - 1);
position.chars_read_total += (len - 1);
position.chars_read_current_line += (len - 1);
return convert_number(number_type);
}
/// contiguous input: try the number fast path, else the byte-path scanner
token_type scan_number_dispatch(std::true_type /*bulk*/)
{
const token_type t = scan_number_bulk_contiguous();
return (t != token_type::uninitialized) ? t : scan_number();
}
/// streaming input: always use the byte-path scanner
token_type scan_number_dispatch(std::false_type /*bulk*/)
{
return scan_number();
}
/*!
@param[in] literal_text the literal text to expect
@param[in] length the length of the passed literal text
@@ -1357,6 +1607,11 @@ scan_number_done:
token_buffer.clear();
decimal_point_position = std::string::npos;
#if JSON_DIAGNOSTIC_POSITIONS
// the first character of the token has already been read, hence the -1
token_start_position = position.chars_read_total - 1;
#endif
note_token_start(std::integral_constant<bool, lazy_token_string> {});
}
@@ -1408,6 +1663,9 @@ scan_number_done:
if (current == '\n')
{
++position.lines_read;
// remember the column the newline was read at: chars_read_current_line
// is about to be cleared, and a matching unget() cannot reconstruct it
chars_read_before_newline = position.chars_read_current_line;
position.chars_read_current_line = 0;
}
@@ -1441,12 +1699,20 @@ scan_number_done:
--position.chars_read_total;
// in case we "unget" a newline, we have to also decrement the lines_read
// and restore the column that get() cleared when it saw the newline;
// chars_read_current_line == 0 can only mean the last get() read one
if (position.chars_read_current_line == 0)
{
if (position.lines_read > 0)
{
--position.lines_read;
}
// chars_read_before_newline counts the newline itself, which is the
// character being ungotten, hence the -1
position.chars_read_current_line = (chars_read_before_newline > 0)
? chars_read_before_newline - 1
: 0;
}
else
{
@@ -1519,6 +1785,15 @@ scan_number_done:
return position;
}
#if JSON_DIAGNOSTIC_POSITIONS
/// return the offset of the first character of the last read token; unlike
/// the token's parsed value, this accounts for escape sequences
constexpr std::size_t get_token_start_position() const noexcept
{
return token_start_position;
}
#endif
/// seekable adapter: rebuild the last read token from the input on demand
const std::vector<char_type>& collect_token_chars(std::vector<char_type>& out, std::true_type /*lazy*/) const
{
@@ -1680,7 +1955,7 @@ scan_number_done:
case '7':
case '8':
case '9':
return scan_number();
return scan_number_dispatch(std::integral_constant<bool, bulk_scan> {});
// end of input (the null byte is needed when parsing from
// string literals)
@@ -1711,6 +1986,10 @@ scan_number_done:
/// the start position of the current token
position_t position {};
/// the value chars_read_current_line had when the last newline was read, so
/// that unget() can restore the column instead of leaving it at 0
std::size_t chars_read_before_newline = 0;
/// raw input token string for error messages; only populated for streaming
/// adapters (seekable adapters reconstruct it lazily via token_string_start)
std::vector<char_type> token_string {};
@@ -1719,6 +1998,12 @@ scan_number_done:
/// the last read token on error for seekable adapters (see collect_token_chars)
std::size_t token_string_start = 0;
#if JSON_DIAGNOSTIC_POSITIONS
/// start offset of the current token within the input, used to report
/// diagnostic positions (see reset())
std::size_t token_start_position = 0;
#endif
/// buffer for variable-length tokens (numbers, strings)
string_t token_buffer {};
include/nlohmann/detail/input/number_parse.hpp
+302
View File
@@ -0,0 +1,302 @@
// __ _____ _____ _____
// __| | __| | | | JSON for Modern C++
// | | |__ | | | | | | version 3.12.0
// |_____|_____|_____|_|___| https://github.com/nlohmann/json
//
// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
// SPDX-License-Identifier: MIT
#pragma once
#include <array> // array
#include <cfloat> // FLT_EVAL_METHOD
#include <cstddef> // size_t
#include <cstdint> // int64_t, uint64_t
#include <limits> // numeric_limits
#include <nlohmann/detail/macro_scope.hpp>
// std::from_chars lives in <charconv>, but being in C++17 mode does not
// guarantee the header exists: GCC 7 sets __cplusplus to C++17 yet ships no
// <charconv> (added in GCC 8; floating-point support in GCC 11). Guard the
// include with __has_include so such toolchains fall back to the scalar path.
#if defined(JSON_HAS_CPP_17) && defined(__has_include)
#if __has_include(<charconv>)
#include <charconv> // from_chars (only used when __cpp_lib_to_chars is defined)
#include <system_error> // errc
#endif
#endif
// This file contains the value-conversion helpers used by the lexer to turn an
// already-validated number token into a value, without the locale/errno
// overhead of std::strtoull/std::strtod. They are free functions so the lexer
// stays focused on scanning; see lexer::convert_number().
NLOHMANN_JSON_NAMESPACE_BEGIN
namespace detail
{
/*!
@brief fast integer parser for an already-validated unsigned integer
The number scanner has already checked that [first, last) is a valid JSON
integer, so this only needs to accumulate the digits and detect overflow. This
avoids the locale/errno machinery of std::strtoull, which dominates
integer-heavy inputs.
@param[in] first pointer to the first character (a digit)
@param[in] last pointer past the last character
@param[out] value the parsed value on success
@return true if the value fit into @a NumberUnsignedType; false on overflow, in
which case the caller falls back to floating-point parsing (matching the
previous std::strtoull behavior)
*/
template<typename NumberUnsignedType>
bool parse_integer_unsigned(const char* first, const char* last, NumberUnsignedType& value) noexcept
{
// accumulate in the widest unsigned type used by the previous strtoull
// path so the overflow behavior is unchanged for custom number types
std::uint64_t x = 0;
constexpr std::uint64_t cutoff = (std::numeric_limits<std::uint64_t>::max)() / 10u;
constexpr std::uint64_t cutlim = (std::numeric_limits<std::uint64_t>::max)() % 10u;
for (const char* p = first; p != last; ++p)
{
const auto digit = static_cast<std::uint64_t>(static_cast<unsigned char>(*p) - static_cast<unsigned char>('0'));
if (JSON_HEDLEY_UNLIKELY(x > cutoff || (x == cutoff && digit > cutlim)))
{
return false;
}
x = (x * 10u) + digit;
}
value = static_cast<NumberUnsignedType>(x);
// reject values that do not round-trip into a narrower NumberUnsignedType
return static_cast<std::uint64_t>(value) == x;
}
/*!
@brief fast integer parser for an already-validated negative integer
@param[in] first pointer to the leading '-'
@param[in] last pointer past the last character
@param[out] value the parsed (negative) value on success
@return true on success; false on overflow (caller falls back to float)
*/
template<typename NumberIntegerType>
bool parse_integer_signed(const char* first, const char* last, NumberIntegerType& value) noexcept
{
// the state machine only reaches the signed path via a leading '-'
JSON_ASSERT(first != last && *first == '-');
std::uint64_t magnitude = 0;
// |INT64_MIN| == INT64_MAX + 1; this is the largest admissible magnitude
constexpr std::uint64_t limit = static_cast<std::uint64_t>((std::numeric_limits<std::int64_t>::max)()) + 1u;
for (const char* p = first + 1; p != last; ++p)
{
const auto digit = static_cast<std::uint64_t>(static_cast<unsigned char>(*p) - static_cast<unsigned char>('0'));
if (JSON_HEDLEY_UNLIKELY(magnitude > (limit - digit) / 10u))
{
return false;
}
magnitude = (magnitude * 10u) + digit;
}
const std::int64_t x = (magnitude == limit)
? (std::numeric_limits<std::int64_t>::min)()
: -static_cast<std::int64_t>(magnitude);
value = static_cast<NumberIntegerType>(x);
// reject values that do not round-trip into a narrower NumberIntegerType
return static_cast<std::int64_t>(value) == x;
}
/*!
@brief exact fast path for parsing a `double` (Clinger's algorithm)
For the common case - at most 19 significant digits, a decimal exponent in
[-22, 22], and a significand below 2^53 - the value equals significand *
10^exp computed in IEEE-754 double arithmetic, which is exact under
round-to-nearest because both operands are exactly representable. This is the
same fast path used by fast_float/simdjson; the general cases are left to
std::strtod. The parser only activates for number_float_t == double; float and
long double keep the std::strtof/std::strtold paths (see the templated overload
below).
@param[in] first pointer to the first character of the number
@param[in] last pointer past the last character
@param[in] decimal_point the (locale-dependent) decimal point character
@param[out] out the parsed value on success
@return true if the value was parsed exactly; false to fall back to strtod
*/
template<typename DecimalPointType>
bool parse_float_fast(const char* first, const char* last, DecimalPointType decimal_point, double& out) noexcept
{
#if defined(FLT_EVAL_METHOD) && FLT_EVAL_METHOD != 0
// Clinger's fast path is only exact when double operations are evaluated in
// true double precision. On platforms that keep intermediates in extended
// precision (e.g. the x87 FPU on 32-bit x86, where FLT_EVAL_METHOD == 2) the
// single significand * 10^scale step is double-rounded and can be 1 ULP off,
// so decline and let the caller fall back to the correctly-rounded
// std::from_chars / std::strtod path.
static_cast<void>(first);
static_cast<void>(last);
static_cast<void>(decimal_point);
static_cast<void>(out);
return false;
#else
static const std::array<double, 23> powers_of_ten =
{
{
1e0, 1e1, 1e2, 1e3, 1e4, 1e5, 1e6, 1e7, 1e8, 1e9, 1e10, 1e11,
1e12, 1e13, 1e14, 1e15, 1e16, 1e17, 1e18, 1e19, 1e20, 1e21, 1e22
}
};
const char* p = first;
bool negative = false;
if (p != last && (*p == '-' || *p == '+'))
{
negative = (*p == '-');
++p;
}
std::uint64_t significand = 0;
int num_digits = 0;
int fractional_digits = 0;
bool seen_dot = false;
bool any_digit = false;
for (; p != last; ++p)
{
const char c = *p;
if (c >= '0' && c <= '9')
{
any_digit = true;
if (JSON_HEDLEY_UNLIKELY(num_digits >= 19))
{
return false; // significand may not fit into uint64_t
}
significand = (significand * 10u) + static_cast<std::uint64_t>(c - '0');
++num_digits;
fractional_digits += static_cast<int>(seen_dot);
}
else if (static_cast<DecimalPointType>(c) == decimal_point)
{
if (JSON_HEDLEY_UNLIKELY(seen_dot))
{
return false;
}
seen_dot = true;
}
else if (c == 'e' || c == 'E')
{
++p;
break;
}
else
{
return false;
}
}
if (JSON_HEDLEY_UNLIKELY(!any_digit))
{
return false;
}
int exponent = 0;
if (p != last) // an exponent part remains
{
bool exp_negative = false;
if (p != last && (*p == '-' || *p == '+'))
{
exp_negative = (*p == '-');
++p;
}
bool any_exp_digit = false;
for (; p != last; ++p)
{
if (JSON_HEDLEY_UNLIKELY(*p < '0' || *p > '9'))
{
return false;
}
exponent = (exponent * 10) + (*p - '0');
any_exp_digit = true;
if (JSON_HEDLEY_UNLIKELY(exponent > 9999))
{
return false;
}
}
if (JSON_HEDLEY_UNLIKELY(!any_exp_digit))
{
return false;
}
if (exp_negative)
{
exponent = -exponent;
}
}
const int scale = exponent - fractional_digits;
if (JSON_HEDLEY_UNLIKELY(significand >= (static_cast<std::uint64_t>(1) << 53)))
{
return false; // significand not exactly representable as double
}
auto result = static_cast<double>(significand);
if (scale >= 0)
{
if (JSON_HEDLEY_UNLIKELY(scale > 22))
{
return false;
}
result *= powers_of_ten[static_cast<std::size_t>(scale)];
}
else
{
if (JSON_HEDLEY_UNLIKELY(-scale > 22))
{
return false;
}
result /= powers_of_ten[static_cast<std::size_t>(-scale)];
}
out = negative ? -result : result;
return true;
#endif
}
/// fast float path is only exact for `double`; decline for float/long double
template<typename DecimalPointType, typename FloatType>
bool parse_float_fast(const char* /*first*/, const char* /*last*/, DecimalPointType /*decimal_point*/, FloatType& /*out*/) noexcept
{
return false;
}
/*!
@brief parse a float with std::from_chars (Eisel-Lemire) when available
std::from_chars is locale-independent, correctly rounded, and - via the
Eisel-Lemire algorithm in modern standard libraries - much faster than strtod
over the whole value range (not just the Clinger subset). It is used only when
__cpp_lib_to_chars indicates full floating-point support and only when it
consumes the entire token ([first, last)); a partial parse means the buffer
uses a non-'.' locale decimal point, in which case the caller falls back to the
locale-aware path. An under-/overflow (result_out_of_range) also declines, so
the caller's strtod fallback supplies the well-defined ±inf/0 result the parser
expects (side-stepping the P4168 divergence between implementations).
@return true if the value was parsed exactly and fully; false to fall back
*/
template<typename FloatType>
bool parse_float_from_chars(const char* first, const char* last, FloatType& out) noexcept
{
// JSON_HAS_CPP_17 must gate the use as well as the <charconv> include above:
// some standard libraries (e.g. libstdc++ 15) define __cpp_lib_to_chars even
// in C++14 mode, where <charconv> is not included.
#if defined(JSON_HAS_CPP_17) && defined(__cpp_lib_to_chars)
const auto result = std::from_chars(first, last, out);
return result.ec == std::errc() && result.ptr == last;
#else
static_cast<void>(first);
static_cast<void>(last);
static_cast<void>(out);
return false;
#endif
}
} // namespace detail
NLOHMANN_JSON_NAMESPACE_END
include/nlohmann/detail/input/string_scan.hpp
+293
View File
@@ -0,0 +1,293 @@
// __ _____ _____ _____
// __| | __| | | | JSON for Modern C++
// | | |__ | | | | | | version 3.12.0
// |_____|_____|_____|_|___| https://github.com/nlohmann/json
//
// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
// SPDX-License-Identifier: MIT
#pragma once
#include <cstddef> // size_t
#include <cstdint> // uint64_t
#include <cstring> // memcpy
#include <nlohmann/detail/macro_scope.hpp>
// Optional SIMD backend for bulk UTF-8 validation. This is an opt-in external
// dependency: nlohmann/json itself stays header-only and the C++11 scalar
// validator below is always available; defining JSON_USE_SIMDUTF additionally
// requires the simdutf headers on the include path and linking the simdutf
// library. See string_bulk_run().
//
// simdutf.h itself requires C++17 - it rejects older standards with an #error -
// so the backend is only compiled in from C++17 on. Below that the macro has no
// effect and the scalar validator is used; it accepts and rejects exactly the
// same input, so only throughput differs. macro_scope.hpp is included above to
// have JSON_HAS_CPP_17 available for this test.
#if defined(JSON_USE_SIMDUTF) && defined(JSON_HAS_CPP_17)
#include <simdutf.h>
#endif
// This file contains the byte-level string-scanning helpers used by the lexer's
// contiguous fast path. They operate purely on raw bytes (no dependency on the
// lexer's template parameters) so they are free functions, keeping the lexer
// itself focused on the state machine; see lexer::scan_string_bulk().
NLOHMANN_JSON_NAMESPACE_BEGIN
namespace detail
{
// classify a single byte as needing individual string handling: the closing
// quote, an escape, a control character, or a non-ASCII (UTF-8)
// lead/continuation byte. Ordinary bytes (0x20..0x7F except '"' and '\\') are
// copied verbatim, which the bulk scanner does 8 bytes at a time.
inline bool is_string_special(unsigned char c) noexcept
{
return c == '\"' || c == '\\' || c < 0x20u || c >= 0x80u;
}
// SWAR helper: return a word whose high bit is set in every byte of @a v that
// is_string_special(); zero if the 8 bytes are all ordinary.
inline std::uint64_t swar_string_special(std::uint64_t v) noexcept
{
constexpr std::uint64_t ones = 0x0101010101010101ull;
constexpr std::uint64_t high = 0x8080808080808080ull;
const std::uint64_t q = v ^ 0x2222222222222222ull; // '"' (0x22)
const std::uint64_t b = v ^ 0x5C5C5C5C5C5C5C5Cull; // '\\' (0x5C)
const std::uint64_t has_quote = (q - ones) & ~q & high;
const std::uint64_t has_backslash = (b - ones) & ~b & high;
const std::uint64_t has_control = (v - 0x2020202020202020ull) & ~v & high; // < 0x20
const std::uint64_t has_non_ascii = v & high; // >= 0x80
return has_quote | has_backslash | has_control | has_non_ascii;
}
// return the index of the first is_string_special() byte in [data, data+n), or
// n if every byte is ordinary; scans 8 bytes at a time
inline std::size_t find_string_special(const unsigned char* data, std::size_t n) noexcept
{
std::size_t i = 0;
for (; i + 8 <= n; i += 8)
{
std::uint64_t word = 0;
std::memcpy(&word, data + i, sizeof(word));
if (swar_string_special(word) != 0)
{
// a special byte is in this word; locate it (endian-agnostic)
for (std::size_t j = 0; j < 8; ++j)
{
if (is_string_special(data[i + j]))
{
return i + j;
}
}
}
}
for (; i < n; ++i)
{
if (is_string_special(data[i]))
{
return i;
}
}
return n;
}
// classify a byte as one the serializer must NOT copy verbatim when
// ensure_ascii is requested: the closing quote, an escape, a control character
// (< 0x20), DEL (0x7F), or any non-ASCII byte (>= 0x80). Everything else -
// printable ASCII except '"' and '\\' - is emitted unchanged. Note this differs
// from is_string_special() only in that 0x7F is also a stop (it is escaped as
// \u007f under ensure_ascii).
inline bool is_ascii_copyable(unsigned char c) noexcept
{
return c >= 0x20u && c < 0x7Fu && c != '\"' && c != '\\';
}
// return the index of the first byte in [data, data+n) that is NOT
// is_ascii_copyable(), or n if every byte can be copied verbatim; scans 8 bytes
// at a time. Used by the serializer's ensure_ascii fast path.
inline std::size_t find_ascii_copyable_run(const unsigned char* data, std::size_t n) noexcept
{
constexpr std::uint64_t ones = 0x0101010101010101ull;
constexpr std::uint64_t high = 0x8080808080808080ull;
std::size_t i = 0;
for (; i + 8 <= n; i += 8)
{
std::uint64_t v = 0;
std::memcpy(&v, data + i, sizeof(v));
const std::uint64_t q = v ^ 0x2222222222222222ull; // '"' (0x22)
const std::uint64_t b = v ^ 0x5C5C5C5C5C5C5C5Cull; // '\\' (0x5C)
const std::uint64_t d = v ^ 0x7F7F7F7F7F7F7F7Full; // DEL (0x7F)
const std::uint64_t stop = ((q - ones) & ~q & high) // == '"'
| ((b - ones) & ~b & high) // == '\\'
| ((d - ones) & ~d & high) // == 0x7F
| ((v - 0x2020202020202020ull) & ~v & high) // < 0x20
| (v & high); // >= 0x80
if (stop != 0)
{
for (std::size_t j = 0; j < 8; ++j)
{
if (!is_ascii_copyable(data[i + j]))
{
return i + j;
}
}
}
}
for (; i < n; ++i)
{
if (!is_ascii_copyable(data[i]))
{
return i;
}
}
return n;
}
// Validate one UTF-8 sequence at the front of [data, data+avail). Returns its
// length (2..4) only when the bytes form a *well-formed* sequence using exactly
// the same ranges as scan_string()'s per-byte switch, so the bulk path accepts
// precisely what the byte path accepts. Returns 0 for anything that is invalid,
// incomplete, or that the byte path must diagnose (the caller then defers to
// that path, keeping error messages unchanged). Lead bytes < 0x80 are handled
// by the caller and never passed here.
inline std::size_t validate_one_utf8(const unsigned char* data, std::size_t avail) noexcept
{
const unsigned char c0 = data[0];
if (c0 >= 0xC2 && c0 <= 0xDF) // U+0080..U+07FF
{
if (avail >= 2 && data[1] >= 0x80 && data[1] <= 0xBF)
{
return 2;
}
}
else if (c0 == 0xE0) // U+0800..U+0FFF
{
if (avail >= 3 && data[1] >= 0xA0 && data[1] <= 0xBF && data[2] >= 0x80 && data[2] <= 0xBF)
{
return 3;
}
}
else if ((c0 >= 0xE1 && c0 <= 0xEC) || c0 == 0xEE || c0 == 0xEF) // U+1000..U+CFFF, U+E000..U+FFFF
{
if (avail >= 3 && data[1] >= 0x80 && data[1] <= 0xBF && data[2] >= 0x80 && data[2] <= 0xBF)
{
return 3;
}
}
else if (c0 == 0xED) // U+D000..U+D7FF (excludes surrogates)
{
if (avail >= 3 && data[1] >= 0x80 && data[1] <= 0x9F && data[2] >= 0x80 && data[2] <= 0xBF)
{
return 3;
}
}
else if (c0 == 0xF0) // U+10000..U+3FFFF
{
if (avail >= 4 && data[1] >= 0x90 && data[1] <= 0xBF && data[2] >= 0x80 && data[2] <= 0xBF && data[3] >= 0x80 && data[3] <= 0xBF)
{
return 4;
}
}
else if (c0 >= 0xF1 && c0 <= 0xF3) // U+40000..U+FFFFF
{
if (avail >= 4 && data[1] >= 0x80 && data[1] <= 0xBF && data[2] >= 0x80 && data[2] <= 0xBF && data[3] >= 0x80 && data[3] <= 0xBF)
{
return 4;
}
}
else if (c0 == 0xF4) // U+100000..U+10FFFF
{
if (avail >= 4 && data[1] >= 0x80 && data[1] <= 0x8F && data[2] >= 0x80 && data[2] <= 0xBF && data[3] >= 0x80 && data[3] <= 0xBF)
{
return 4;
}
}
return 0; // invalid, incomplete, or must be diagnosed by the byte path
}
// Scalar (C++11) computation of the bulk run length: the number of leading
// bytes in [data, data+n) that are ordinary ASCII or complete well-formed UTF-8
// sequences, stopping before the first byte that needs individual handling (the
// closing quote, an escape, a control character, or an ill-formed/truncated
// sequence). ASCII is skipped 8 bytes at a time.
inline std::size_t scalar_string_bulk_run(const unsigned char* data, std::size_t n) noexcept
{
std::size_t pos = 0;
while (pos < n)
{
pos += find_string_special(data + pos, n - pos);
if (pos >= n || data[pos] < 0x80u)
{
break; // end of buffer, or a quote/escape/control byte
}
const std::size_t seq = validate_one_utf8(data + pos, n - pos);
if (seq == 0)
{
break; // ill-formed or truncated: let the byte path diagnose it
}
pos += seq;
}
return pos;
}
#if defined(JSON_USE_SIMDUTF) && defined(JSON_HAS_CPP_17)
// Index of the first quote/escape/control byte in [data, data+n) (non-ASCII
// bytes are *not* stops here - the whole run is handed to simdutf), or n.
inline std::size_t find_string_delimiter(const unsigned char* data, std::size_t n) noexcept
{
constexpr std::uint64_t ones = 0x0101010101010101ull;
constexpr std::uint64_t high = 0x8080808080808080ull;
std::size_t i = 0;
for (; i + 8 <= n; i += 8)
{
std::uint64_t v = 0;
std::memcpy(&v, data + i, sizeof(v));
const std::uint64_t q = v ^ 0x2222222222222222ull;
const std::uint64_t b = v ^ 0x5C5C5C5C5C5C5C5Cull;
const std::uint64_t hit = ((q - ones) & ~q & high)
| ((b - ones) & ~b & high)
| ((v - 0x2020202020202020ull) & ~v & high);
if (hit != 0)
{
for (std::size_t j = 0; j < 8; ++j)
{
const unsigned char c = data[i + j];
if (c == '\"' || c == '\\' || c < 0x20u)
{
return i + j;
}
}
}
}
for (; i < n; ++i)
{
const unsigned char c = data[i];
if (c == '\"' || c == '\\' || c < 0x20u)
{
return i;
}
}
return n;
}
#endif
// Backend-dispatched bulk run length. With JSON_USE_SIMDUTF the run up to the
// next delimiter is validated in one shot by simdutf; on the rare failure the
// scalar helper recomputes the exact valid prefix so the byte path still
// produces the precise diagnostic. Without it, the pure scalar path is used.
inline std::size_t string_bulk_run(const unsigned char* data, std::size_t n) noexcept
{
#if defined(JSON_USE_SIMDUTF) && defined(JSON_HAS_CPP_17)
const std::size_t run = find_string_delimiter(data, n);
if (run != 0 && simdutf::validate_utf8(reinterpret_cast<const char*>(data), run))
{
return run;
}
#endif
return scalar_string_bulk_run(data, n);
}
} // namespace detail
NLOHMANN_JSON_NAMESPACE_END
include/nlohmann/detail/json_pointer.hpp
+5 -9
View File
@@ -396,19 +396,15 @@ class json_pointer
// convert null values to arrays or objects before continuing
if (ptr->is_null())
{
// check if the reference token is a valid array index, that is
// a nonempty sequence of digits without a leading '0'
// (cf. RFC 6901, Sect. 4); tokens that could never be a valid
// array index (such as "01" or "") are treated as object keys
const bool nums = !reference_token.empty()
&& (reference_token.size() == 1 || reference_token[0] != '0')
&& std::all_of(reference_token.begin(), reference_token.end(),
[](const unsigned char x)
// check if the reference token is a number
const bool nums =
std::all_of(reference_token.begin(), reference_token.end(),
[](const unsigned char x)
{
return std::isdigit(x);
});
// change value to an array for array indices or "-" or to object otherwise
// change value to an array for numbers or "-" or to object otherwise
*ptr = (nums || reference_token == "-")
? detail::value_t::array
: detail::value_t::object;
include/nlohmann/detail/output/binary_writer.hpp
+29 -4
View File
@@ -813,7 +813,10 @@ class binary_writer
bool prefix_required = true;
if (use_type && !j.m_data.m_value.array->empty())
{
JSON_ASSERT(use_count);
if (!use_count)
{
JSON_THROW(other_error::create(502, "use_type requires use_size = true", &j));
}
const CharType first_prefix = ubjson_prefix(j.front(), use_bjdata);
const bool same_prefix = std::all_of(j.begin() + 1, j.end(),
[this, first_prefix, use_bjdata](const BasicJsonType & v)
@@ -859,7 +862,10 @@ class binary_writer
if (use_type && (bjdata_draft3 || !j.m_data.m_value.binary->empty()))
{
JSON_ASSERT(use_count);
if (!use_count)
{
JSON_THROW(other_error::create(502, "use_type requires use_size = true", &j));
}
oa->write_character(to_char_type('$'));
oa->write_character(bjdata_draft3 ? 'B' : 'U');
}
@@ -911,7 +917,10 @@ class binary_writer
bool prefix_required = true;
if (use_type && !j.m_data.m_value.object->empty())
{
JSON_ASSERT(use_count);
if (!use_count)
{
JSON_THROW(other_error::create(502, "use_type requires use_size = true", &j));
}
const CharType first_prefix = ubjson_prefix(j.front(), use_bjdata);
const bool same_prefix = std::all_of(j.begin(), j.end(),
[this, first_prefix, use_bjdata](const BasicJsonType & v)
@@ -1670,7 +1679,23 @@ class binary_writer
{
return true;
}
len *= static_cast<std::size_t>(el.template get<std::uint64_t>());
// a dimension that does not fit into std::size_t, or a product that
// overflows it, would wrap around and could match the size of
// _ArrayData_ by accident; the resulting header announces an
// element count that no reader can honor (the binary reader rejects
// it with out_of_range.408), so encode as a plain object instead
const auto dim = el.template get<std::uint64_t>();
if (!value_in_range_of<std::size_t>(dim))
{
return true;
}
const auto dim_size = static_cast<std::size_t>(dim);
if (dim_size != 0 && len > (std::numeric_limits<std::size_t>::max)() / dim_size)
{
return true;
}
len *= dim_size;
}
key = "_ArrayData_";
include/nlohmann/detail/output/serializer.hpp
+801 -110
View File
File diff suppressed because it is too large Load Diff
include/nlohmann/json.hpp
+22 -7
View File
@@ -1341,11 +1341,12 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
const error_handler_t error_handler = error_handler_t::strict) const
{
string_t result;
serializer s(detail::output_adapter<char, string_t>(result), indent_char, error_handler);
detail::output_string_adapter<char, string_t> string_adapter(result);
serializer s(&string_adapter, indent_char, error_handler);
if (indent >= 0)
{
s.dump(*this, true, ensure_ascii, static_cast<unsigned int>(indent));
s.dump(*this, true, ensure_ascii, static_cast<std::size_t>(indent));
}
else
{
@@ -3516,7 +3517,10 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
if (merge_objects && it.value().is_object())
{
auto it2 = m_data.m_value.object->find(it.key());
if (it2 != m_data.m_value.object->end())
// Only recurse when the existing value is itself an object.
// Otherwise overwrite, matching the documented "all other values
// are overwritten as usual" behavior (see #5402).
if (it2 != m_data.m_value.object->end() && it2->second.is_object())
{
it2->second.update(it.value(), true);
#if JSON_DIAGNOSTICS
@@ -3652,6 +3656,12 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
// note parentheses around operands are necessary; see
// https://github.com/nlohmann/json/issues/1530
// Mixed signed/unsigned integer comparisons check whether the signed value
// is negative before casting. If it is, the comparison is performed with
// the fixed values -1 and 1, which preserves the ordering relationship
// because any negative signed value is smaller than any unsigned value.
// Otherwise, the non-negative signed value is cast to unsigned before the
// comparison to avoid wraparound.
#define JSON_IMPLEMENT_OPERATOR(op, null_result, unordered_result, default_result) \
const auto lhs_type = lhs.type(); \
const auto rhs_type = rhs.type(); \
@@ -3710,12 +3720,16 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
} \
else if (lhs_type == value_t::number_unsigned && rhs_type == value_t::number_integer) \
{ \
return static_cast<number_integer_t>(lhs.m_data.m_value.number_unsigned) op rhs.m_data.m_value.number_integer; \
return (rhs.m_data.m_value.number_integer < 0) \
? (number_integer_t(1) op number_integer_t(-1)) \
: (lhs.m_data.m_value.number_unsigned op static_cast<number_unsigned_t>(rhs.m_data.m_value.number_integer)); \
} \
else if (lhs_type == value_t::number_integer && rhs_type == value_t::number_unsigned) \
{ \
return lhs.m_data.m_value.number_integer op static_cast<number_integer_t>(rhs.m_data.m_value.number_unsigned); \
} \
return (lhs.m_data.m_value.number_integer < 0) \
? (number_integer_t(-1) op number_integer_t(1)) \
: (static_cast<number_unsigned_t>(lhs.m_data.m_value.number_integer) op rhs.m_data.m_value.number_unsigned); \
} \
else if(compares_unordered(lhs, rhs))\
{\
return (unordered_result);\
@@ -4042,7 +4056,8 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
o.width(0);
// do the actual serialization
serializer s(detail::output_adapter<char>(o), o.fill());
detail::output_stream_adapter<char> stream_adapter(o);
serializer s(&stream_adapter, o.fill());
s.dump(j, pretty_print, false, static_cast<unsigned int>(indentation));
return o;
}
single_include/nlohmann/json.hpp
+1937 -195
View File
File diff suppressed because it is too large Load Diff
tests/CMakeLists.txt
+68
View File
@@ -2,6 +2,9 @@ cmake_minimum_required(VERSION 3.13...4.0)
option(JSON_Valgrind "Execute test suite with Valgrind." OFF)
option(JSON_FastTests "Skip expensive/slow tests." OFF)
option(JSON_TestSimdutf "Build the unit tests against the simdutf UTF-8 validation backend." OFF)
set(JSON_SIMDUTF_VERSION 9.1.0 CACHE STRING "The simdutf version used by JSON_TestSimdutf.")
set(JSON_32bitTest AUTO CACHE STRING "Enable the 32bit unit test (ON/OFF/AUTO/ONLY).")
set(JSON_TestStandards "" CACHE STRING "The list of standards to test explicitly.")
@@ -149,6 +152,71 @@ if(test_force)
endif()
message(STATUS "${msg}")
#############################################################################
# optionally validate UTF-8 with simdutf (JSON_USE_SIMDUTF)
#############################################################################
# The simdutf backend is opt-in and not vendored, so it is fetched here rather
# than being a checked-in dependency. Everything below hangs off test_main,
# whose usage requirements every test target inherits; the library target and
# the installed CMake package are deliberately left untouched.
if (JSON_TestSimdutf)
# simdutf requires C++17, both to compile itself and to be reachable from
# the library, which keeps its scalar validator below that. Find a tested
# standard that satisfies it.
set(simdutf_standard "")
foreach(cxx_standard ${test_cxx_standards})
if(NOT cxx_standard LESS 17 AND compiler_supports_cpp_${cxx_standard})
set(simdutf_standard ${cxx_standard})
break()
endif()
endforeach()
if("${simdutf_standard}" STREQUAL "")
# Building simdutf would fail outright without a C++17 compiler, and
# even with one it would go unused if no C++17-or-later standard is
# tested. Say so and fall back to the scalar validator rather than
# failing the build.
if(NOT compiler_supports_cpp_17)
set(simdutf_reason "the compiler does not support C++17")
else()
set(simdutf_reason "no tested standard is C++17 or later (testing ${msg_standards})")
endif()
message(WARNING
"JSON_TestSimdutf is enabled, but ${simdutf_reason}. simdutf requires C++17, so it "
"is not fetched and JSON_USE_SIMDUTF is not defined: the tests run against the "
"built-in scalar UTF-8 validator instead. Set JSON_TestStandards to include 17 or "
"later, or build with a compiler that supports C++17.")
else()
if (CMAKE_VERSION VERSION_LESS 3.18)
message(FATAL_ERROR "JSON_TestSimdutf requires CMake 3.18 or later (simdutf's minimum).")
endif()
include(FetchContent)
# simdutf builds its tests and tools by default, and its tests pull
# further dependencies of their own; only the library is needed here
set(SIMDUTF_TESTS OFF CACHE BOOL "" FORCE)
set(SIMDUTF_TOOLS OFF CACHE BOOL "" FORCE)
set(SIMDUTF_BENCHMARKS OFF CACHE BOOL "" FORCE)
set(SIMDUTF_ICONV OFF CACHE BOOL "" FORCE)
FetchContent_Declare(simdutf
URL https://github.com/simdutf/simdutf/archive/refs/tags/v${JSON_SIMDUTF_VERSION}.tar.gz
DOWNLOAD_EXTRACT_TIMESTAMP TRUE
)
FetchContent_MakeAvailable(simdutf)
target_compile_definitions(test_main PUBLIC JSON_USE_SIMDUTF)
target_link_libraries(test_main PUBLIC simdutf::simdutf)
# simdutf.h requires C++17; below that the library keeps its scalar
# validator, so any C++11/14 test targets exercise the fallback and the
# C++17-and-later ones exercise simdutf. Both must agree.
message(STATUS "UTF-8 validation delegated to simdutf ${JSON_SIMDUTF_VERSION} for C++17 and later (JSON_USE_SIMDUTF)")
endif()
endif()
# *DO* use json_test_set_test_options() above this line
json_test_should_build_32bit_test(json_32bit_test json_32bit_test_only "${JSON_32bitTest}")
tests/src/unit-bjdata.cpp
+63
View File
@@ -2730,6 +2730,27 @@ TEST_CASE("BJData")
CHECK(json::from_bjdata(json::to_bjdata(j_type), true, true) == j_type);
CHECK(json::from_bjdata(json::to_bjdata(j_size), true, true) == j_size);
}
SECTION("ndarray whose dimensions overflow stays as object")
{
// the product of the dimensions wraps around std::size_t to 0
// and so matches the size of the empty _ArrayData_; writing this
// as an ndarray would announce an element count no reader can
// honor, so it has to stay a plain object
json j_overflow = json({{"_ArrayData_", json::array()}, {"_ArraySize_", {9223372036854775808ull, 2}}, {"_ArrayType_", "uint8"}});
CHECK(json::from_bjdata(json::to_bjdata(j_overflow), true, true) == j_overflow);
// a single dimension that does not fit into std::size_t is
// rejected for the same reason (only observable where
// std::size_t is narrower than 64 bit)
json j_huge = json({{"_ArrayData_", json::array()}, {"_ArraySize_", {18446744073709551615ull}}, {"_ArrayType_", "uint8"}});
CHECK(json::from_bjdata(json::to_bjdata(j_huge), true, true) == j_huge);
// a well-formed ndarray is still encoded as one
json j_ok = json({{"_ArrayData_", {1, 2, 3, 4, 5, 6}}, {"_ArraySize_", {2, 3}}, {"_ArrayType_", "uint8"}});
CHECK(json::to_bjdata(j_ok) == std::vector<uint8_t>({'[', '$', 'U', '#', '[', 'i', 2, 'i', 3, ']', 1, 2, 3, 4, 5, 6}));
CHECK(json::from_bjdata(json::to_bjdata(j_ok), true, true) == j_ok);
}
}
}
@@ -3822,6 +3843,48 @@ TEST_CASE("all BJData first bytes")
}
#endif
TEST_CASE("BJData use_type requires use_size")
{
SECTION("non-empty object throws other_error.502")
{
const json j = {{"a", 1}, {"b", 2}};
CHECK_THROWS_WITH_AS(json::to_bjdata(j, false, true),
"[json.exception.other_error.502] use_type requires use_size = true",
json::other_error&);
}
SECTION("non-empty array throws other_error.502")
{
const json j = {1, 2, 3};
CHECK_THROWS_WITH_AS(json::to_bjdata(j, false, true),
"[json.exception.other_error.502] use_type requires use_size = true",
json::other_error&);
}
SECTION("scalars do not throw with use_type=true, use_count=false")
{
CHECK_NOTHROW(json::to_bjdata(42, false, true));
CHECK_NOTHROW(json::to_bjdata(3.14, false, true));
CHECK_NOTHROW(json::to_bjdata("hello", false, true));
CHECK_NOTHROW(json::to_bjdata(true, false, true));
CHECK_NOTHROW(json::to_bjdata(nullptr, false, true));
}
SECTION("empty containers do not throw with use_type=true, use_count=false")
{
CHECK_NOTHROW(json::to_bjdata(json::array(), false, true));
CHECK_NOTHROW(json::to_bjdata(json::object(), false, true));
}
SECTION("valid combinations on non-empty containers")
{
const json j = {{"a", 1}, {"b", 2}};
CHECK_NOTHROW(json::to_bjdata(j, false, false));
CHECK_NOTHROW(json::to_bjdata(j, true, false));
CHECK_NOTHROW(json::to_bjdata(j, true, true));
}
}
TEST_CASE("BJData roundtrips" * doctest::skip())
{
SECTION("input from self-generated BJData files")
tests/src/unit-cbor.cpp
+13 -2
View File
@@ -2565,11 +2565,16 @@ TEST_CASE("Tagged values")
const json j = "s";
auto v = json::to_cbor(j);
SECTION("0xC6..0xD4")
const json j_bin_payload = json::binary(std::vector<std::uint8_t> {0x01, 0x02, 0x03});
auto v_bin_payload = json::to_cbor(j_bin_payload);
SECTION("0xC0..0xD7")
{
for (const auto b : std::vector<std::uint8_t>
{
0xC6, 0xC7, 0xC8, 0xC9, 0xCA, 0xCB, 0xCC, 0xCD, 0xCE, 0xCF, 0xD0, 0xD1, 0xD2, 0xD3, 0xD4
0xC0, 0xC1, 0xC2, 0xC3, 0xC4, 0xC5,
0xC6, 0xC7, 0xC8, 0xC9, 0xCA, 0xCB, 0xCC, 0xCD, 0xCE, 0xCF, 0xD0, 0xD1, 0xD2, 0xD3, 0xD4,
0xD5, 0xD6, 0xD7
})
{
CAPTURE(b);
@@ -2589,6 +2594,12 @@ TEST_CASE("Tagged values")
auto j_tagged_stored = json::from_cbor(v_tagged, true, true, json::cbor_tag_handler_t::store);
CHECK(j_tagged_stored == j);
auto v_binary_tagged = v_bin_payload;
v_binary_tagged.insert(v_binary_tagged.begin(), b);
auto j_binary_tagged_stored = json::from_cbor(v_binary_tagged, true, true, json::cbor_tag_handler_t::store);
CHECK(j_binary_tagged_stored == j_bin_payload);
CHECK(!j_binary_tagged_stored.get_binary().has_subtype());
}
}
tests/src/unit-class_lexer.cpp
+382
View File
@@ -12,6 +12,10 @@
#include <nlohmann/json.hpp>
using nlohmann::json;
#include <sstream> // stringstream
#include <string> // string
#include <vector> // vector
namespace
{
// shortcut to scan a string literal
@@ -224,3 +228,381 @@ TEST_CASE("lexer class")
CHECK((scan_string("/**//**//**/", true) == json::lexer::token_type::end_of_input));
}
}
TEST_CASE("lexer number fast path")
{
// The contiguous fast path (used for pointer/string input) must agree with
// the streaming byte path (used for std::istream) on token type, numeric
// value, and round-trip text for every well-formed number, and reject the
// same malformed numbers with the same message.
SECTION("contiguous vs streaming parity")
{
const std::vector<std::string> numbers =
{
"0", "-0", "1", "-1", "42", "-42", "10", "100", "1234567890",
"0.0", "-0.0", "3.14", "-3.14", "0.5", "-0.001", "123.456789",
"1e0", "1E0", "1e10", "1e-10", "1e+10", "1.5e3", "-2.5E-4",
"9223372036854775807", // INT64_MAX -> unsigned
"9223372036854775808", // INT64_MAX + 1 -> unsigned
"18446744073709551615", // UINT64_MAX -> unsigned
"18446744073709551616", // UINT64_MAX + 1 -> float
"-9223372036854775808", // INT64_MIN -> integer
"-9223372036854775809", // INT64_MIN - 1 -> float
"123456789012345678901234567890", // huge -> float
"0.30000000000000004", "2.2250738585072014e-308", "1e308",
// high-precision / wide-exponent values that exercise the
// std::from_chars (Eisel-Lemire) path beyond the Clinger subset
"1.7976931348623157e308", "1.2345678901234567e-250",
"9007199254740993", "5e-324", "1e-320"
};
for (const auto& n : numbers)
{
const std::string doc = "[" + n + "]";
// contiguous fast path
const json a = json::parse(doc);
// streaming byte path
std::stringstream ss(doc);
const json b = json::parse(ss);
CAPTURE(n);
CHECK(a == b);
CHECK(a.dump() == b.dump());
CHECK(a[0].type() == b[0].type());
}
}
SECTION("token type classification")
{
CHECK((scan_string("0") == json::lexer::token_type::value_unsigned));
CHECK((scan_string("-1") == json::lexer::token_type::value_integer));
CHECK((scan_string("1.5") == json::lexer::token_type::value_float));
CHECK((scan_string("1e5") == json::lexer::token_type::value_float));
CHECK((scan_string("18446744073709551615") == json::lexer::token_type::value_unsigned));
CHECK((scan_string("18446744073709551616") == json::lexer::token_type::value_float));
CHECK((scan_string("-9223372036854775808") == json::lexer::token_type::value_integer));
CHECK((scan_string("-9223372036854775809") == json::lexer::token_type::value_float));
}
SECTION("malformed numbers are rejected identically")
{
for (const char* bad :
{"-", "1.", "1e", "1e+", "1.2e", "01", "-01", "1..2", "1.2.3"
})
{
CAPTURE(bad);
// the contiguous fast path must decline and let the byte path report
const std::string doc = std::string("[") + bad + "]";
CHECK_FALSE(json::accept(doc));
std::stringstream ss(doc);
CHECK_FALSE(json::accept(ss));
}
}
#if !defined(JSON_NOEXCEPTION)
// these sections parse invalid input, which aborts when exceptions are off
SECTION("exhaustive grammar parity with the streaming path")
{
// The JSON number grammar is encoded twice: once as the scan_number()
// state machine and once as the contiguous fast path. Enumerate every
// short string over the number alphabet and require the two encodings to
// agree exactly - on acceptance, on the reported error, and on the parsed
// value - so they cannot drift apart.
const std::string alphabet = "01.eE+-";
// full outcome of parsing @a doc, so a mismatch in type, value, or error
// message is caught, not just a mismatch in acceptance
const auto outcome = [](const std::string & doc, bool streaming) -> std::string
{
try
{
if (streaming)
{
std::stringstream ss(doc);
const json j = json::parse(ss);
return std::string(j[0].type_name()) + '|' + j.dump();
}
const json j = json::parse(doc);
return std::string(j[0].type_name()) + '|' + j.dump();
}
catch (const json::parse_error& e)
{
return {e.what()};
}
};
std::vector<std::string> mismatches;
std::vector<std::string> tokens{""};
for (std::size_t length = 1; length <= 4; ++length)
{
std::vector<std::string> next;
next.reserve(tokens.size() * alphabet.size());
for (const auto& prefix : tokens)
{
for (const char c : alphabet)
{
next.push_back(prefix + c);
}
}
tokens = next;
for (const auto& token : tokens)
{
const std::string doc = "[" + token + "]";
if (outcome(doc, false) != outcome(doc, true))
{
mismatches.push_back(doc);
}
}
}
// 7 + 49 + 343 + 2401 tokens
CHECK(tokens.size() == 2401);
CAPTURE(mismatches);
CHECK(mismatches.empty());
}
SECTION("error positions match the streaming path")
{
// Rejecting identically is not enough: the fast path must also report the
// error at the same position as the byte path. A number directly followed
// by a newline is the interesting case, because the byte path reaches the
// newline (which resets the column) and then ungets it.
// returns the parse_error message, or "" if the document parsed
const auto contiguous_error = [](const std::string & doc) -> std::string
{
try
{
const json j = json::parse(doc);
static_cast<void>(j);
}
catch (const json::parse_error& e)
{
return {e.what()};
}
return {};
};
const auto streaming_error = [](const std::string & doc) -> std::string
{
try
{
std::stringstream ss(doc);
const json j = json::parse(ss);
static_cast<void>(j);
}
catch (const json::parse_error& e)
{
return {e.what()};
}
return {};
};
for (const char* bad :
{"[01\n]", "[00\n]", "[-01\n]", "{1\n}", "[1\n2]", "[1.2.3\n]",
"[1 \n2]", "[\n1\n2]", "1\n2", "[01\r\n]", "[1e\n]", "[-\n]"
})
{
CAPTURE(bad);
const std::string doc = bad;
const std::string contiguous_what = contiguous_error(doc);
CHECK_FALSE(contiguous_what.empty());
CHECK(contiguous_what == streaming_error(doc));
}
// A number terminated by a newline must report the same position as the
// same number terminated by anything else: scan_number() reads the
// terminator and ungets it, so the reported column is the one reached
// after the number's last character - not the 0 that an unget() across
// the newline used to leave behind.
CHECK(contiguous_error("[01\n]") == contiguous_error("[01 ]"));
CHECK(contiguous_error("[01\n]") ==
"[json.exception.parse_error.101] parse error at line 1, column 3: "
"syntax error while parsing array - unexpected number literal; expected ']'");
// the same for a multi-character token, where the column of the last
// character (the '3' of "-2.5e3") differs from the column it starts at
CHECK(contiguous_error("null -2.5e3\nfalse") == contiguous_error("null -2.5e3 false"));
CHECK(contiguous_error("null -2.5e3\nfalse") ==
"[json.exception.parse_error.101] parse error at line 1, column 11: "
"syntax error while parsing value - unexpected number literal; expected end of input");
}
#endif
}
TEST_CASE("lexer string fast path")
{
// Build a byte string from explicit values: a hex escape in a string
// literal swallows every following hex digit, which makes sequences like
// "\xC3\xA9b" mean something other than they look like.
const auto bytes = [](std::initializer_list<int> values)
{
std::string result;
for (const int value : values)
{
result.push_back(static_cast<char>(value));
}
return result;
};
#if !defined(JSON_NOEXCEPTION)
// the full outcome of parsing @a doc: the parsed value, or the exact error
// message, so a mismatch in either is caught. Only usable with exceptions
// on: parsing invalid input aborts when they are off.
const auto outcome = [](const std::string & doc, bool streaming) -> std::string
{
try
{
if (streaming)
{
std::stringstream ss(doc);
const json j = json::parse(ss);
return j.dump();
}
const json j = json::parse(doc);
return j.dump();
}
// not just parse_error: if a bulk scanner ever let ill-formed UTF-8
// through, dump() would throw type_error.316, and that has to surface
// as a reported mismatch rather than as an uncaught exception
catch (const json::exception& e)
{
return {e.what()};
}
};
#endif
// once at the start of the string, once past the first 8-byte SWAR word, so
// the bulk scanner sees each case with and without a run behind it
const std::vector<std::size_t> offsets{0, 9};
#if !defined(JSON_NOEXCEPTION)
SECTION("exhaustive contiguous vs streaming parity")
{
// ordinary ASCII, both specials, a control byte, characters that make
// the preceding backslash a valid escape, a UTF-8 lead byte of each
// length, a continuation byte, and a byte that is never valid
const std::vector<std::string> alphabet =
{
"a", "\"", "\\", "n", "u", "0", bytes({0x01}),
bytes({0xC3}), bytes({0xA9}), bytes({0xE4}), bytes({0xF0}),
bytes({0x80}), bytes({0xFF})
};
std::vector<std::string> mismatches;
std::vector<std::string> tokens{""};
for (std::size_t length = 1; length <= 3; ++length)
{
std::vector<std::string> next;
next.reserve(tokens.size() * alphabet.size());
for (const auto& prefix : tokens)
{
for (const auto& symbol : alphabet)
{
next.push_back(prefix + symbol);
}
}
tokens = next;
for (const auto& token : tokens)
{
for (const std::size_t offset : offsets)
{
const std::string doc = "[\"" + std::string(offset, 'a') + token + "\"]";
if (outcome(doc, false) != outcome(doc, true))
{
mismatches.push_back(doc);
}
}
}
}
// 13 + 169 + 2197 tokens, each at two offsets
CHECK(tokens.size() == 2197);
CAPTURE(mismatches);
CHECK(mismatches.empty());
}
SECTION("special bytes at every offset of the SWAR stride")
{
// The bulk scanner consumes 8 bytes at a time and then a tail; place
// every kind of byte that ends a run at each offset across two words,
// so multibyte sequences also straddle the word boundary.
const std::vector<std::string> specials =
{
"\"", "\\", bytes({0x01}), bytes({0x1F}), bytes({0x7F}),
bytes({0xC3, 0xA9}), bytes({0xE4, 0xB8, 0xAD}), bytes({0xF0, 0x9F, 0x98, 0x80}),
bytes({0xFF}), bytes({0xC3}), bytes({0xE4, 0xB8})
};
std::vector<std::string> mismatches;
for (std::size_t offset = 0; offset <= 17; ++offset)
{
for (const auto& special : specials)
{
const std::string doc = "[\"" + std::string(offset, 'a') + special + "\"]";
if (outcome(doc, false) != outcome(doc, true))
{
mismatches.push_back(doc);
}
}
}
CAPTURE(mismatches);
CHECK(mismatches.empty());
}
#endif
// json::accept() never throws, so the ranges stay covered without exceptions
SECTION("UTF-8 ranges are accepted and rejected as documented")
{
// The bulk validator must accept exactly what the byte-at-a-time
// scanner accepts, so pin the boundaries of every range it recognizes.
// aggregate, only ever brace-initialized below; default member
// initializers would stop it being an aggregate in C++11
struct utf8_case // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init)
{
std::string sequence;
bool valid;
const char* description;
};
const std::vector<utf8_case> cases =
{
{bytes({0xC2, 0x80}), true, "U+0080, shortest two-byte"},
{bytes({0xDF, 0xBF}), true, "U+07FF, longest two-byte"},
{bytes({0xC1, 0xBF}), false, "overlong two-byte"},
{bytes({0xC2, 0x7F}), false, "two-byte with bad continuation"},
{bytes({0xE0, 0xA0, 0x80}), true, "U+0800, shortest three-byte"},
{bytes({0xE0, 0x9F, 0xBF}), false, "overlong three-byte"},
{bytes({0xED, 0x9F, 0xBF}), true, "U+D7FF, just below the surrogates"},
{bytes({0xED, 0xA0, 0x80}), false, "surrogate U+D800"},
{bytes({0xED, 0xBF, 0xBF}), false, "surrogate U+DFFF"},
{bytes({0xEE, 0x80, 0x80}), true, "U+E000, just above the surrogates"},
{bytes({0xEF, 0xBF, 0xBF}), true, "U+FFFF"},
{bytes({0xF0, 0x90, 0x80, 0x80}), true, "U+10000, shortest four-byte"},
{bytes({0xF0, 0x8F, 0xBF, 0xBF}), false, "overlong four-byte"},
{bytes({0xF4, 0x8F, 0xBF, 0xBF}), true, "U+10FFFF, highest code point"},
{bytes({0xF4, 0x90, 0x80, 0x80}), false, "above U+10FFFF"},
{bytes({0xF5, 0x80, 0x80, 0x80}), false, "lead byte out of range"},
{bytes({0x80}), false, "bare continuation byte"},
{bytes({0xFF}), false, "byte that never appears in UTF-8"},
{bytes({0xC3}), false, "truncated two-byte"},
{bytes({0xE4, 0xB8}), false, "truncated three-byte"},
{bytes({0xF0, 0x9F, 0x98}), false, "truncated four-byte"}
};
for (const auto& test_case : cases)
{
CAPTURE(test_case.description);
for (const std::size_t offset : offsets)
{
CAPTURE(offset);
const std::string doc = "[\"" + std::string(offset, 'a') + test_case.sequence + "\"]";
CHECK(json::accept(doc) == test_case.valid);
#if !defined(JSON_NOEXCEPTION)
CHECK(outcome(doc, false) == outcome(doc, true));
#endif
}
}
}
}
tests/src/unit-comparison.cpp
+71
View File
@@ -15,6 +15,8 @@
#include "doctest_compatibility.h"
#include <cstdint>
#define JSON_TESTS_PRIVATE
#include <nlohmann/json.hpp>
using nlohmann::json;
@@ -255,6 +257,75 @@ TEST_CASE("lexicographical comparison operators")
{f_, f_, f_, f_, f_, f_, f_, f_, f_, f_, f_, f_, f_, f_, f_, f_, f_, f_, f_, f_, f_, f_}, // 21
};
SECTION("signed/unsigned mixed comparison above INT64_MAX")
{
const json above_int64_max = static_cast<std::uint64_t>((std::numeric_limits<std::int64_t>::max)()) + 1ULL;
const json max_uint64 = (std::numeric_limits<std::uint64_t>::max)();
const json negative_one = -1;
const json one = 1;
const json max_int64 = (std::numeric_limits<std::int64_t>::max)();
CHECK_FALSE(above_int64_max == negative_one);
CHECK(above_int64_max != negative_one);
CHECK(negative_one < above_int64_max);
CHECK(negative_one <= above_int64_max);
CHECK_FALSE(negative_one > above_int64_max);
CHECK_FALSE(negative_one >= above_int64_max);
CHECK_FALSE(above_int64_max < negative_one);
CHECK_FALSE(above_int64_max <= negative_one);
CHECK(above_int64_max > negative_one);
CHECK(above_int64_max >= negative_one);
CHECK(negative_one != above_int64_max);
CHECK_FALSE(negative_one == above_int64_max);
CHECK_FALSE(max_uint64 == negative_one);
CHECK(max_uint64 != negative_one);
CHECK(negative_one < max_uint64);
CHECK(negative_one <= max_uint64);
CHECK_FALSE(negative_one > max_uint64);
CHECK_FALSE(negative_one >= max_uint64);
CHECK_FALSE(max_uint64 < negative_one);
CHECK_FALSE(max_uint64 <= negative_one);
CHECK(max_uint64 > negative_one);
CHECK(max_uint64 >= negative_one);
CHECK(negative_one != max_uint64);
CHECK_FALSE(negative_one == max_uint64);
CHECK_FALSE(one == above_int64_max);
CHECK(one != above_int64_max);
CHECK(one < above_int64_max);
CHECK(one <= above_int64_max);
CHECK_FALSE(one > above_int64_max);
CHECK_FALSE(one >= above_int64_max);
CHECK_FALSE(above_int64_max < one);
CHECK_FALSE(above_int64_max <= one);
CHECK(above_int64_max > one);
CHECK(above_int64_max >= one);
CHECK_FALSE(max_int64 == above_int64_max);
CHECK(max_int64 != above_int64_max);
CHECK(max_int64 < above_int64_max);
CHECK(max_int64 <= above_int64_max);
CHECK_FALSE(max_int64 > above_int64_max);
CHECK_FALSE(max_int64 >= above_int64_max);
CHECK_FALSE(above_int64_max < max_int64);
CHECK_FALSE(above_int64_max <= max_int64);
CHECK(above_int64_max > max_int64);
CHECK(above_int64_max >= max_int64);
#if JSON_HAS_THREE_WAY_COMPARISON
// JSON_HAS_CPP_20 (do not remove; see note at top of file)
CHECK((negative_one <=> above_int64_max) == std::partial_ordering::less); // *NOPAD*
CHECK((above_int64_max <=> negative_one) == std::partial_ordering::greater); // *NOPAD*
CHECK((negative_one <=> max_uint64) == std::partial_ordering::less); // *NOPAD*
CHECK((max_uint64 <=> negative_one) == std::partial_ordering::greater); // *NOPAD*
CHECK((one <=> above_int64_max) == std::partial_ordering::less); // *NOPAD*
CHECK((above_int64_max <=> one) == std::partial_ordering::greater); // *NOPAD*
CHECK((max_int64 <=> above_int64_max) == std::partial_ordering::less); // *NOPAD*
CHECK((above_int64_max <=> max_int64) == std::partial_ordering::greater); // *NOPAD*
#endif
}
SECTION("compares unordered")
{
std::vector<std::vector<bool>> expected =
tests/src/unit-convenience.cpp
+3 -1
View File
@@ -98,8 +98,10 @@ void check_escaped(const char* original, const char* escaped = "", bool ensure_a
void check_escaped(const char* original, const char* escaped, const bool ensure_ascii)
{
std::stringstream ss;
json::serializer s(nlohmann::detail::output_adapter<char>(ss), ' ');
nlohmann::detail::output_stream_adapter<char> adapter(ss);
json::serializer s(&adapter, ' ');
s.dump_escaped(original, ensure_ascii);
s.flush(); // dump_escaped writes into the serializer's internal buffer
CHECK(ss.str() == escaped);
}
} // namespace
tests/src/unit-diagnostic-positions.cpp
+30
View File
@@ -38,6 +38,36 @@ TEST_CASE("Better diagnostics with positions")
"[json.exception.type_error.302] type must be number, but is string", json::type_error);
}
SECTION("positions of strings containing escape sequences")
{
// escape sequences make the token longer than the string it parses to,
// so the positions must not be derived from the parsed value's length
const auto check = [](const std::string & text, const std::string & token)
{
CAPTURE(text)
CAPTURE(token)
const json j = json::parse(text);
const json& v = j.at("a");
CHECK(text.substr(v.start_pos(), v.end_pos() - v.start_pos()) == token);
};
check(R"({"a":"plain"})", R"("plain")");
check(R"({"a":"tab\there"})", R"("tab\there")");
check(R"({"a":"\n\n\n\n\n\n"})", R"("\n\n\n\n\n\n")");
check(R"({"a":"\""})", R"("\"")");
check(R"({"a":"\\"})", R"("\\")");
check(R"({"a":"é"})", R"("é")");
check(R"({"a":"🌞"})", R"("🌞")");
check("{\"a\":\"\xc3\xa9\"}", "\"\xc3\xa9\""); // multi-byte UTF-8, no escapes
// a string at the root, where an escape would otherwise push the
// reported start position past the opening quote
const std::string root = R"("a\tb")";
const json j = json::parse(root);
CHECK(j.start_pos() == 0);
CHECK(j.end_pos() == root.size());
}
SECTION("JSON patch add to primitive parent (#4292)")
{
// the JSON Patch "add" target /foo/bar/baz has a string parent
tests/src/unit-json_pointer.cpp
-66
View File
@@ -396,72 +396,6 @@ TEST_CASE("JSON pointers")
}
}
SECTION("creating intermediate levels")
{
SECTION("tokens that are valid array indices create arrays")
{
json j;
j["/0"_json_pointer] = 1;
CHECK(j == json({1}));
json j2;
j2["/2"_json_pointer] = 1;
CHECK(j2 == json({nullptr, nullptr, 1}));
json j3;
j3["/-"_json_pointer] = 1;
CHECK(j3 == json({1}));
json j4;
j4["/foo/0/0"_json_pointer] = 1;
CHECK(j4 == json({{"foo", {{1}}}}));
}
SECTION("tokens that are no valid array indices create objects")
{
json j;
j["/one"_json_pointer] = 1;
CHECK(j == json({{"one", 1}}));
// leading '0' can never be a valid array index (RFC 6901, Sect. 4)
json j2;
j2["/01"_json_pointer] = 1;
CHECK(j2 == json({{"01", 1}}));
// the empty token is a valid object key, but no valid array index
json j3;
j3["/"_json_pointer] = 1;
CHECK(j3 == json({{"", 1}}));
}
SECTION("creating a level yields the same result as reusing it (#5357)")
{
json j;
j["/a/b/01/d"_json_pointer] = "value";
json j_init = json::object();
j_init["/a/b"_json_pointer] = json::object();
j_init["/a/b/01/d"_json_pointer] = "value";
const json expected = json::parse(R"({"a":{"b":{"01":{"d":"value"}}}})");
CHECK(j == expected);
CHECK(j_init == expected);
// unflatten uses the same key
const json flat = {{"/a/b/01/d", "value"}};
CHECK(flat.unflatten() == expected);
}
SECTION("existing arrays still reject invalid indices")
{
json j = {1, 2, 3};
CHECK_THROWS_WITH_AS(j["/01"_json_pointer],
"[json.exception.parse_error.106] parse error: array index '01' must not begin with '0'", json::parse_error&);
CHECK_THROWS_WITH_AS(j.at("/01"_json_pointer),
"[json.exception.parse_error.106] parse error: array index '01' must not begin with '0'", json::parse_error&);
}
}
SECTION("flatten")
{
json j =
tests/src/unit-modifiers.cpp
+24
View File
@@ -801,6 +801,30 @@ TEST_CASE("modifiers")
j1.update(j2, true);
CHECK(j1 == json({{"string", "t"}, {"numbers", 1}}));
}
SECTION("overwrite primitive with object")
{
json j1 = {{"k", 1}};
json const j2 = {{"k", {{"x", 2}}}};
j1.update(j2, true);
CHECK(j1 == json({{"k", {{"x", 2}}}}));
}
SECTION("overwrite array with object")
{
json j1 = {{"k", {1, 2}}};
json const j2 = {{"k", {{"x", 2}}}};
j1.update(j2, true);
CHECK(j1 == json({{"k", {{"x", 2}}}}));
}
SECTION("overwrite nested primitive with object")
{
json j1 = {{"k", {{"inner", 1}}}};
json const j2 = {{"k", {{"inner", {{"x", 2}}}}}};
j1.update(j2, true);
CHECK(j1 == json({{"k", {{"inner", {{"x", 2}}}}}}));
}
}
}
}
tests/src/unit-regression2.cpp
+11
View File
@@ -1555,4 +1555,15 @@ TEST_CASE("issue #5338 - truncated CBOR tagged binary subtype is rejected")
}
}
TEST_CASE("issue #5402 - update(merge_objects=true) overwrites a primitive with an object")
{
json t = {{"k", 1}};
t.update(json{{"k", {{"x", 2}}}}, true);
CHECK(t == json({{"k", {{"x", 2}}}}));
json mixed = {{"keep", {{"a", 1}}}, {"replace", 1}};
mixed.update(json{{"keep", {{"b", 2}}}, {"replace", {{"x", 2}}}}, true);
CHECK(mixed == json({{"keep", {{"a", 1}, {"b", 2}}}, {"replace", {{"x", 2}}}}));
}
DOCTEST_CLANG_SUPPRESS_WARNING_POP
tests/src/unit-serialization.cpp
+229
View File
@@ -382,3 +382,232 @@ TEST_CASE("dump for basic_json with long double number_float_t")
check_same(100.0L, 100.0);
}
}
TEST_CASE("serialization of strings (bulk fast path)")
{
// These cases exercise the SWAR bulk-copy fast path in dump_escaped and the
// internal write buffer: long runs, escapes interrupting runs, 0x7F/DEL,
// multibyte UTF-8 under both ensure_ascii settings, and payloads larger than
// the write buffer.
SECTION("long unescaped ASCII exceeds the write buffer")
{
const std::string big(3000, 'a');
const json j = big;
CHECK(j.dump() == '"' + big + '"');
CHECK(j.dump(-1, ' ', true) == '"' + big + '"');
// round-trips
CHECK(json::parse(j.dump()) == j);
}
SECTION("runs interrupted by escapes")
{
const json j = std::string(500, 'x') + "\n\"\\" + std::string(500, 'y');
const std::string out = j.dump();
CHECK(out == '"' + std::string(500, 'x') + "\\n\\\"\\\\" + std::string(500, 'y') + '"');
CHECK(json::parse(out) == j);
}
SECTION("DEL (0x7F) depends on ensure_ascii")
{
const json j = std::string("a\x7f" "b");
CHECK(j.dump(-1, ' ', false) == "\"a\x7f" "b\""); // copied verbatim
CHECK(j.dump(-1, ' ', true) == "\"a\\u007fb\""); // escaped
}
SECTION("multibyte UTF-8 under both ensure_ascii settings")
{
const json j = std::string("A\xc3\xa9\xe4\xbd\xa0\xf0\x9f\x98\x80Z"); // A é 你 😀 Z
// not escaping non-ASCII: bytes are copied through the bulk validator
CHECK(j.dump(-1, ' ', false) == "\"A\xc3\xa9\xe4\xbd\xa0\xf0\x9f\x98\x80Z\"");
// ensure_ascii: escaped (with a surrogate pair for the emoji)
CHECK(j.dump(-1, ' ', true) == "\"A\\u00e9\\u4f60\\ud83d\\ude00Z\"");
CHECK(json::parse(j.dump(-1, ' ', true)) == j);
}
SECTION("many small structural writes exceed the write buffer")
{
json arr = json::array();
for (int i = 0; i < 2000; ++i)
{
arr.push_back(i);
}
const std::string out = arr.dump();
CHECK(out.front() == '[');
CHECK(out.back() == ']');
CHECK(json::parse(out) == arr);
json obj = json::object();
for (int i = 0; i < 500; ++i)
{
obj["key" + std::to_string(i)] = i;
}
CHECK(json::parse(obj.dump()) == obj);
CHECK(json::parse(obj.dump(2)) == obj);
// an array of many empty strings emits a long run of single-character
// writes ('"', '"', ',') at shallow nesting depth, so the write buffer
// fills and flushes mid-run without the deep recursion that would
// overflow the stack on some debug builds
json many_empty = json::array();
for (int i = 0; i < 500; ++i)
{
many_empty.push_back("");
}
const std::string out2 = many_empty.dump();
CHECK(out2.size() > 1024); // spans multiple write-buffer flushes
CHECK(out2.front() == '[');
CHECK(out2.back() == ']');
CHECK(json::parse(out2) == many_empty);
}
SECTION("invalid UTF-8 handling is unaffected by the fast path")
{
const json j = std::string("valid\xff" "more");
CHECK_THROWS_WITH_AS(j.dump(), "[json.exception.type_error.316] invalid UTF-8 byte at index 5: 0xFF", json::type_error&);
CHECK(j.dump(-1, ' ', false, json::error_handler_t::replace) == "\"valid\xef\xbf\xbd" "more\"");
CHECK(j.dump(-1, ' ', true, json::error_handler_t::replace) == "\"valid\\ufffdmore\"");
CHECK(j.dump(-1, ' ', false, json::error_handler_t::ignore) == "\"validmore\"");
}
}
TEST_CASE("indentation is written straight into the write buffer")
{
// put_indent() memsets the indentation into the write buffer instead of
// copying it out of a pre-grown indentation string. These cases cover an
// indentation wider than the buffer, a non-space indentation character, and
// nesting deep enough that the accumulated indentation spans several
// buffer-fulls - the situations the old grow-a-string approach got wrong.
SECTION("indent_step wider than the write buffer")
{
const json j = {{"a", 1}};
// 2000 > the 1024-byte write buffer, and > the 512 the indentation
// string used to start at
CHECK(j.dump(2000) == "{\n" + std::string(2000, ' ') + "\"a\": 1\n}");
// several whole buffer-fulls, so the buffer is refilled once and then
// flushed repeatedly
CHECK(j.dump(5000) == "{\n" + std::string(5000, ' ') + "\"a\": 1\n}");
CHECK(j.dump(5000, '\t') == "{\n" + std::string(5000, '\t') + "\"a\": 1\n}");
// an exact multiple of the buffer size
CHECK(j.dump(4096) == "{\n" + std::string(4096, ' ') + "\"a\": 1\n}");
}
SECTION("a non-space indentation character is used throughout")
{
const json j = {{"a", 1}};
// 600 is past the point where the indentation used to be grown, which
// is where a hard-coded space would have shown up
CHECK(j.dump(600, '\t') == "{\n" + std::string(600, '\t') + "\"a\": 1\n}");
CHECK(j.dump(3, '.') == "{\n...\"a\": 1\n}");
}
SECTION("accumulated indentation spans several buffer-fulls")
{
// five levels deep at 400 per level: the innermost value is indented by
// 2000 characters, reached in steps that each straddle the buffer end
json j = json::array({1});
for (int i = 0; i < 4; ++i)
{
j = json::array({j});
}
const std::string out = j.dump(400);
CHECK(out.find(std::string("\n") + std::string(2000, ' ') + "1\n") != std::string::npos);
CHECK(json::parse(out) == j);
}
SECTION("indentation is unchanged for ordinary widths")
{
const json j = {{"a", {1, 2}}, {"b", nullptr}};
CHECK(j.dump(2) == "{\n \"a\": [\n 1,\n 2\n ],\n \"b\": null\n}");
CHECK(j.dump(0) == "{\n\"a\": [\n1,\n2\n],\n\"b\": null\n}");
}
}
TEST_CASE("serialization of deeply nested values")
{
// dump() descends into a bounded number of levels and writes out whatever
// is nested deeper than that without the call stack; see
// https://github.com/nlohmann/json/issues/5387
SECTION("nested deeper than the call stack could follow")
{
// parsing is iterative, so building these costs little
const std::size_t depth = 100000;
const std::string array_text = std::string(depth, '[') + '0' + std::string(depth, ']');
CHECK(json::parse(array_text).dump() == array_text);
std::string object_text;
object_text.reserve((6 * depth) + 1);
for (std::size_t i = 0; i < depth; ++i)
{
object_text += "{\"a\":";
}
object_text += '1';
object_text.append(depth, '}');
CHECK(json::parse(object_text).dump() == object_text);
}
SECTION("depths around the bound of the descent")
{
// Cover every depth around the bound, so that the two ways of writing a
// value are known to meet cleanly - wherever the bound is set.
for (std::size_t d = 1; d <= 300; ++d)
{
CAPTURE(d);
const std::string array_text = std::string(d, '[') + '7' + std::string(d, ']');
CHECK(json::parse(array_text).dump() == array_text);
std::string object_text;
for (std::size_t i = 0; i < d; ++i)
{
object_text += "{\"k\":";
}
object_text += '7';
object_text.append(d, '}');
CHECK(json::parse(object_text).dump() == object_text);
}
}
SECTION("pretty-printing across the bound")
{
for (std::size_t d = 120; d <= 140; ++d)
{
CAPTURE(d);
const json j = json::parse(std::string(d, '[') + '7' + std::string(d, ']'));
std::string expected;
for (std::size_t i = 0; i < d; ++i)
{
expected += std::string(2 * i, ' ') + "[\n";
}
expected += std::string(2 * d, ' ') + '7';
for (std::size_t i = d; i > 0; --i)
{
expected += '\n' + std::string(2 * (i - 1), ' ') + ']';
}
CHECK(j.dump(2) == expected);
}
}
SECTION("an empty container below the bound")
{
// an empty container is written out in full and never descended into,
// so it must not gain a newline when it is reached iteratively
for (std::size_t d = 125; d <= 135; ++d)
{
CAPTURE(d);
const std::string compact = std::string(d, '[') + "[]" + std::string(d, ']');
CHECK(json::parse(compact).dump() == compact);
const std::string with_object = std::string(d, '[') + "{}" + std::string(d, ']');
CHECK(json::parse(with_object).dump() == with_object);
}
}
}
tests/src/unit-ubjson.cpp
+80
View File
@@ -1713,6 +1713,44 @@ TEST_CASE("UBJSON")
CHECK(json::to_ubjson(json::from_ubjson(s_L)) == s_i);
}
SECTION("no-op markers")
{
// A no-op ('N') is valid wherever a value may start; it is consumed
// by get_ignore_noop() before the value is read. It is not valid
// where a string length type specification is expected.
SECTION("accepted where a value may start")
{
// at top level, also repeated
CHECK(json::from_ubjson(std::vector<uint8_t>({'N', 'i', 1})) == json(1));
CHECK(json::from_ubjson(std::vector<uint8_t>({'N', 'N', 'N', 'i', 1})) == json(1));
// inside an array of unknown size, before and after an element
CHECK(json::from_ubjson(std::vector<uint8_t>({'[', 'N', 'i', 1, ']'})) == json({1}));
CHECK(json::from_ubjson(std::vector<uint8_t>({'[', 'i', 1, 'N', ']'})) == json({1}));
// inside an object of unknown size: before a key, between key
// and value, and before the closing '}'
CHECK(json::from_ubjson(std::vector<uint8_t>({'{', 'N', 'U', 1, 'a', 'i', 1, '}'})) == json({{"a", 1}}));
CHECK(json::from_ubjson(std::vector<uint8_t>({'{', 'U', 1, 'a', 'N', 'i', 1, '}'})) == json({{"a", 1}}));
CHECK(json::from_ubjson(std::vector<uint8_t>({'{', 'U', 1, 'a', 'i', 1, 'N', '}'})) == json({{"a", 1}}));
}
SECTION("rejected where a length type specification is expected")
{
json _;
// after the 'S' marker of a string value
std::vector<uint8_t> const v_S = {'S', 'N', 'U', 1, 'a'};
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(v_S), "[json.exception.parse_error.113] parse error at byte 2: syntax error while parsing UBJSON string: expected length type specification (U, i, I, l, L); last byte: 0x4E", json::parse_error&);
// as the key length of an object with a known size, where
// no-ops are not permitted in the first place
std::vector<uint8_t> const v_key = {'{', '#', 'i', 1, 'N', 'U', 1, 'a', 'i', 1};
CHECK_THROWS_WITH_AS(_ = json::from_ubjson(v_key), "[json.exception.parse_error.113] parse error at byte 5: syntax error while parsing UBJSON string: expected length type specification (U, i, I, l, L); last byte: 0x4E", json::parse_error&);
}
}
SECTION("number")
{
SECTION("float")
@@ -2465,6 +2503,48 @@ TEST_CASE("all UBJSON first bytes")
}
#endif
TEST_CASE("UBJSON use_type requires use_size")
{
SECTION("non-empty array throws other_error.502")
{
const json j = {1, 2, 3};
CHECK_THROWS_WITH_AS(json::to_ubjson(j, false, true),
"[json.exception.other_error.502] use_type requires use_size = true",
json::other_error&);
}
SECTION("non-empty object throws other_error.502")
{
const json j = {{"a", 1}, {"b", 2}};
CHECK_THROWS_WITH_AS(json::to_ubjson(j, false, true),
"[json.exception.other_error.502] use_type requires use_size = true",
json::other_error&);
}
SECTION("scalars do not throw with use_type=true, use_count=false")
{
CHECK_NOTHROW(json::to_ubjson(42, false, true));
CHECK_NOTHROW(json::to_ubjson(3.14, false, true));
CHECK_NOTHROW(json::to_ubjson("hello", false, true));
CHECK_NOTHROW(json::to_ubjson(true, false, true));
CHECK_NOTHROW(json::to_ubjson(nullptr, false, true));
}
SECTION("empty containers do not throw with use_type=true, use_count=false")
{
CHECK_NOTHROW(json::to_ubjson(json::array(), false, true));
CHECK_NOTHROW(json::to_ubjson(json::object(), false, true));
}
SECTION("valid combinations on non-empty containers")
{
const json j = {1, 2, 3};
CHECK_NOTHROW(json::to_ubjson(j, false, false));
CHECK_NOTHROW(json::to_ubjson(j, true, false));
CHECK_NOTHROW(json::to_ubjson(j, true, true));
}
}
TEST_CASE("UBJSON roundtrips" * doctest::skip())
{
SECTION("input from self-generated UBJSON files")
tests/src/unit-user_defined_input.cpp
+238
View File
@@ -18,7 +18,12 @@
#include <nlohmann/json.hpp>
using nlohmann::json;
#include <array> // array
#include <cstddef> // size_t
#include <cstdint> // uint8_t
#include <list>
#include <string> // string
#include <vector> // vector
#if defined(__cpp_lib_concepts) && defined(JSON_HAS_CPP_20)
#include <iterator>
@@ -212,6 +217,65 @@ TEST_CASE("Parse with heterogeneous iterator and sentinel types")
CHECK(j2.at(0) == 1);
}
// A type whose data() hands out raw bytes but whose size() counts something
// else - here fixed-size records. Reading [data(), data() + size()) as bytes
// would silently truncate the input, so data() and size() alone must not be
// taken as evidence of contiguous byte storage.
struct record_buffer
{
using value_type = std::array<char, 4>;
std::string bytes;
const char* data() const noexcept
{
return bytes.data();
}
std::size_t size() const noexcept
{
return bytes.size() / sizeof(value_type);
}
const char* begin() const noexcept
{
return bytes.data();
}
const char* end() const noexcept
{
return bytes.data() + bytes.size();
}
};
TEST_CASE("Contiguous byte containers take the pointer adapter")
{
// Containers with contiguous single-byte storage are routed through the
// pointer-based adapter so the bulk fast paths apply in every standard, not
// only in C++20 where the library iterators model std::contiguous_iterator.
CHECK(nlohmann::detail::is_contiguous_byte_container<std::string>::value);
CHECK(nlohmann::detail::is_contiguous_byte_container<std::vector<char>>::value);
CHECK(nlohmann::detail::is_contiguous_byte_container<std::vector<std::uint8_t>>::value);
CHECK(nlohmann::detail::is_contiguous_byte_container<std::array<char, 4>>::value);
// input_adapter() takes its container by forwarding reference, so the trait
// is also asked about reference types
CHECK(nlohmann::detail::is_contiguous_byte_container<std::string&>::value);
CHECK(nlohmann::detail::is_contiguous_byte_container<const std::string&>::value);
// everything else keeps the iterator-based adapter
CHECK_FALSE(nlohmann::detail::is_contiguous_byte_container<std::list<char>>::value);
CHECK_FALSE(nlohmann::detail::is_contiguous_byte_container<std::vector<int>>::value);
CHECK_FALSE(nlohmann::detail::is_contiguous_byte_container<const char*>::value);
// including a type that has data() and size() but whose size() does not
// count the units data() points at: its value_type says so
CHECK_FALSE(nlohmann::detail::is_contiguous_byte_container<record_buffer>::value);
// and such a container still parses through its iterators, in full - taking
// it for a byte container would stop after data() + size() bytes
const record_buffer buffer{"[1,2,3,4,5]"};
CHECK(buffer.size() * sizeof(record_buffer::value_type) < buffer.bytes.size());
CHECK(json::parse(buffer) == json({1, 2, 3, 4, 5}));
}
#if defined(__cpp_lib_concepts) && defined(JSON_HAS_CPP_20)
// JSON_HAS_CPP_20 (do not remove; see note at top of file)
TEST_CASE("Parse with std::counted_iterator and std::default_sentinel_t")
@@ -228,6 +292,180 @@ TEST_CASE("Parse with std::counted_iterator and std::default_sentinel_t")
const std::counted_iterator<iterator_type> first2(json_str.begin(), len);
CHECK(json::accept(first2, std::default_sentinel));
}
TEST_CASE("std::counted_iterator reaches the contiguous fast paths")
{
// A sized sentinel makes the remaining element count computable in O(1), so
// std::counted_iterator over a contiguous iterator must reach the same bulk
// string/number scanners as a plain pointer - not just the byte-at-a-time
// fallback (see #5268 for the equivalent memcpy fast path).
#if JSON_HAS_RANGES
// JSON_HAS_RANGES is 0 on standard libraries with an incomplete <ranges>
// (libstdc++ < 11, libc++ < 16), where the adapter deliberately falls back
// to the byte-at-a-time scanner; everything below still has to work there.
using adapter_type = nlohmann::detail::iterator_input_adapter<std::counted_iterator<const char*>, std::default_sentinel_t>;
CHECK(adapter_type::supports_bulk_scan);
CHECK(adapter_type::supports_seek);
#endif
// exercise every fast path: long ASCII run, multibyte UTF-8, escapes, and
// integer/floating-point numbers
const std::string json_str =
R"({"ascii":"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",)"
"\"utf8\":\"\xe4\xb8\xad\xe6\x96\x87\xf0\x9f\x98\x80\xc3\xa9\","
R"("escaped":"aéb\n\\","ints":[0,-1,18446744073709551615,-9223372036854775808],)"
R"("floats":[1.5,-2.25e3,0.30000000000000004]})";
const auto len = static_cast<std::iter_difference_t<const char*>>(json_str.size());
const std::counted_iterator<const char*> first(json_str.data(), len);
const json j = json::parse(first, std::default_sentinel);
// parsing through the pointer adapter must give exactly the same result
CHECK(j == json::parse(json_str));
#if !defined(JSON_NOEXCEPTION)
// Diagnostics that quote the offending token are reconstructed from the
// already-consumed input (supports_seek), a path a sized sentinel only
// reaches now; check a few that include the "last read" text. Parsing
// invalid input aborts when exceptions are off, hence the guard.
// Raw strings and explicit bytes: an escaped literal and two literals
// written next to each other both read as mistakes to static analysis.
const auto byte = [](int value)
{
return std::string(1, static_cast<char>(value));
};
const std::vector<std::string> diagnostic_docs =
{
"1\nx",
"truX",
"[tru]",
R"("abc)",
R"(["\ud834"])",
R"(["a)" + byte(0x01) + R"(b"])",
R"([")" + byte(0xC3) + byte(0x28) + R"("])",
"[1e]",
R"(["aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaX)"
};
for (const auto& text : diagnostic_docs)
{
CAPTURE(text);
const std::counted_iterator<const char*> it(text.data(), static_cast<std::iter_difference_t<const char*>>(text.size()));
std::string counted_message;
std::string string_message;
try
{
const json counted_result = json::parse(it, std::default_sentinel);
static_cast<void>(counted_result);
}
catch (const json::parse_error& e)
{
counted_message = e.what();
}
try
{
const json string_result = json::parse(text);
static_cast<void>(string_result);
}
catch (const json::parse_error& e)
{
string_message = e.what();
}
CHECK_FALSE(counted_message.empty());
CHECK(counted_message == string_message);
}
// and errors must still be reported identically
const std::string bad = "[01\n]";
const std::counted_iterator<const char*> bad_first(bad.data(), static_cast<std::iter_difference_t<const char*>>(bad.size()));
std::string counted_what;
std::string string_what;
try
{
const json counted_result = json::parse(bad_first, std::default_sentinel);
static_cast<void>(counted_result);
}
catch (const json::parse_error& e)
{
counted_what = e.what();
}
try
{
const json string_result = json::parse(bad);
static_cast<void>(string_result);
}
catch (const json::parse_error& e)
{
string_what = e.what();
}
CHECK_FALSE(counted_what.empty());
CHECK(counted_what == string_what);
#endif
}
#if !defined(JSON_NOEXCEPTION)
// several cases below are truncated on purpose, and parsing invalid input
// aborts when exceptions are off
TEST_CASE("std::counted_iterator bulk scanning stops at the counted end")
{
// The count, not the size of the underlying buffer, is the end of the
// input: the bulk scanners must never look at the bytes behind it, even
// though they are readable. Each case is compared against parsing the
// equivalent prefix as a std::string.
const auto via_counted = [](const std::string & buf, std::size_t n) -> std::string
{
const std::counted_iterator<const char*> first(buf.data(), static_cast<std::iter_difference_t<const char*>>(n));
try
{
const json j = json::parse(first, std::default_sentinel);
return "OK|" + j.dump();
}
catch (const json::parse_error& e)
{
return {e.what()};
}
};
const auto via_prefix = [](const std::string & buf, std::size_t n) -> std::string
{
try
{
const json j = json::parse(buf.substr(0, n));
return "OK|" + j.dump();
}
catch (const json::parse_error& e)
{
return {e.what()};
}
};
struct testcase // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init)
{
const char* buffer;
std::size_t count;
};
const std::vector<testcase> cases =
{
{"[\"abc\"]____TRAILING____", 7}, // exact fit, tail hidden
{"[\"abcdefghijklmnop\"]____", 8}, // cut inside a string
{"[\"abc\"]____", 6}, // cut just before the closing quote
{"[12345]xxxxx", 4}, // cut inside a number
{"[123]999999", 5}, // number ends exactly at the count
{"[\"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\"]", 12}, // closing quote only behind the count
{"[\"aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa\"]", 19}, // cut inside an 8-byte SWAR stride
{"[\"\xe4\xb8\xad\xe6\x96\x87\"]", 5}, // cut inside a UTF-8 sequence
{"[\"\xe4\xb8\xad\xe6\x96\x87\"]____", 10}, // complete UTF-8, tail hidden
{"[1.25e3]TRAILINGDIGITS999", 7}, // number token reaches the count
};
for (const auto& tc : cases)
{
CAPTURE(tc.buffer);
CAPTURE(tc.count);
const std::string buffer = tc.buffer;
CHECK(via_counted(buffer, tc.count) == via_prefix(buffer, tc.count));
}
}
#endif
#endif
} // namespace
tests/src/unit-wstring.cpp
+10
View File
@@ -125,6 +125,16 @@ TEST_CASE("wide strings")
std::u32string const w = U"\"\x110000";
json _;
CHECK_THROWS_AS(_ = json::parse(w), json::parse_error&);
// a code unit above U+10FFFF must not be narrowed onto the EOF
// sentinel: 0xFFFFFFFF would otherwise end the document silently and
// let everything following it pass the strict end-of-input check
std::u32string const trailing{U'[', U'1', U']', static_cast<char32_t>(0xFFFFFFFF), U'x'};
CHECK_THROWS_WITH_AS(_ = json::parse(trailing), "[json.exception.parse_error.101] parse error at line 1, column 4: syntax error while parsing value - invalid literal; last read: '1]\xFF'; expected end of input", json::parse_error&);
CHECK(!json::accept(trailing));
// the same unit inside a string is reported as an ill-formed byte
CHECK_THROWS_WITH_AS(_ = json::parse(std::u32string{U'"', static_cast<char32_t>(0xFFFFFFFF), U'"'}), "[json.exception.parse_error.101] parse error at line 1, column 2: syntax error while parsing value - invalid string: ill-formed UTF-8 byte; last read: '\"\xFF'", json::parse_error&);
}
}
}