Compare commits

..
Author SHA1 Message Date
Niels Lohmann cf9bc9f71b Merge branch 'ci/tiers' into ci/windows-macos-sets
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-30 15:36:35 +01:00
Niels Lohmann 9e1d35eda9 Merge branch 'ci/path-filters' into ci/tiers
Keep the tiered job layout and apply the harden-runner 2.22.2 bump
from #5962.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-30 15:36:22 +01:00
Niels Lohmann 4b538d80f6 Merge branch 'develop' into ci/path-filters
Keep ci_test_documentation in repository_checks.yml and apply the
harden-runner 2.22.2 bump from #5962 there.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-30 15:35:39 +01:00
Niels Lohmann 91a8503f83 Merge ci/tiers into ci/windows-macos-sets
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-28 22:09:54 +01:00
Niels Lohmann 66ee5d1b84 Merge ci/path-filters into ci/tiers
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-28 22:09:54 +01:00
Niels Lohmann 7a4e6b7a5d Merge develop into ci/path-filters
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-28 22:09:53 +01:00
Niels Lohmann a4c4e92dbd Merge wip/5949 into ci/windows-macos-sets
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-28 20:32:12 +01:00
Niels Lohmann ae57b09bc6 Merge wip/5948 into ci/tiers
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-28 20:32:11 +01:00
Niels Lohmann fd37b04813 Merge origin/develop into ci/path-filters
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-28 20:32:10 +01:00
Niels Lohmann e89d5fdbdd Merge ci/tiers
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-28 07:11:45 +01:00
Niels Lohmann d587214ab2 Merge ci/path-filters
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-28 07:11:44 +01:00
Niels Lohmann bd442d5d07 Merge develop
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-28 07:11:43 +01:00
Niels Lohmann 4c3e92b96e Split the macOS and Windows jobs into a PR set, a full set, and a nightly set
The 19 macOS and 32 Windows jobs ran on every push to every pull request.
They now use the same sets as the Ubuntu jobs:

- PR set (16): four Visual Studio 2022 and two Visual Studio 2026
  configurations, ARM64 Debug, Clang 20.1.8 with MinGW, MinGW x86, the
  C++20 module test, Xcode 16.0, 16.4, 26.3 and 26.6, C++14 on the
  latest Xcode, and the Swift Package Manager check.
- full set (34): every other Visual Studio configuration, Clang
  version, clang-cl, Xcode version and C++ standard.
- nightly set (1): 64-bit MinGW, which takes about 68 minutes. A failed
  nightly run opens or updates an issue.

No configuration is removed: tools/ci_jobs/ci_jobs.py --compare lists
157 configurations before and after, and all jobs keep their runner,
steps and environment. ci_jobs.py now checks all three workflows and
expands matrices that consist only of include entries correctly.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-27 19:45:39 +01:00
Niels Lohmann 087df7b0c2 Split the Ubuntu jobs into a PR set, a full set, and a nightly set
All ~100 Ubuntu jobs ran on every push to every pull request. They now
belong to three sets, chosen by each job's if: condition:

- PR set (16 configurations): every push to a pull request and every
  push. Together with the checks that always run, these jobs would
  have caught at least 92 % of the failing commits of Sep/Oct 2026.
- full set (73 configurations): after the PR set passed, on pushes and
  on pull requests with the "🚀 ready to merge" label.
- nightly set (10 configurations): Infer, Valgrind, coverage, icpc,
  icpx, nvhpc, CUDA, and ARM64; nightly on develop, on pushes to master
  and release/*, and by hand. A failed nightly run opens or updates an
  issue.

A matrix whose entries belong to different sets is split into one job
per set; the jobs share their steps through YAML anchors. No
configuration is removed: tools/ci_jobs/ci_jobs.py --compare lists 150
configurations before and after. The same tool checks in CI that every
job belongs to exactly one set, that the full set waits for the whole
PR set, and that the job table in quality_assurance.md is up to date.

The documentation describes the sets in a new section "How CI runs",
and the QA pipeline figure shows the new badges.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-27 19:40:21 +01:00
Niels Lohmann eae3c8217b Split the checks of check_path_filters.py into smaller functions
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-27 19:39:20 +01:00
Niels Lohmann 960c234cc6 Do not report the excluded files that check_path_filters.py lists itself
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-27 18:58:36 +01:00
Niels Lohmann b6e24d19a9 Skip the compiler workflows for pull requests that change no build input
ubuntu.yml, macos.yml and windows.yml run about 155 jobs on every pull
request, including those that change only the documentation. They now
have a pull_request path filter: it starts with '**' and excludes the
documentation, Markdown files, GitHub metadata, other workflows and the
license files, so a new file runs them unless it is excluded on purpose.
Pushes to develop, master and release/* are not filtered.

The documentation jobs and the REUSE check move from ubuntu.yml to the
new repository_checks.yml, which has no path filter and also runs
tools/check_path_filters. That check keeps the three filters identical
and fails when a file the compiler jobs keep mentions an excluded file.
It found two: tools/check_build_options reads the Meson section of
package_managers.md, and the swiftpm job builds the Swift example from
the documentation; both are included in the filter.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-27 18:35:03 +01:00
110 changed files with 4049 additions and 27271 deletions

No files matched your search

+9 -2
View File
@@ -111,8 +111,6 @@ context which existing file needs to be extended, and only very few cases requir
When fixing a bug, edit `unit-regression3.cpp` and add a section referencing the fixed issue.
`unit-regression2.cpp` holds the older tests; the two files exist because a single one grew large enough for the
MinGW linker to fail relocating it, so please keep adding to the smaller file rather than growing the larger one.
Tests that call `sax_parse` go into `unit-sax_parse.cpp` rather than into a large test file: every call instantiates
the parser and the binary reader that recover from errors, which grows a test file considerably.
#### Exceptions
@@ -156,6 +154,15 @@ and each page's title (H1) and declaration by
[`docs/docset/generate_docset.py`](https://github.com/nlohmann/json/blob/develop/docs/docset/generate_docset.py);
`make build` reports API pages that cannot be classified.
A pull request that changes only the documentation, Markdown files, GitHub metadata, or license files skips the
compiler workflows (Ubuntu, macOS, Windows); the documentation and license checks still run. See
[`tools/check_path_filters`](https://github.com/nlohmann/json/tree/develop/tools/check_path_filters) for the list of
skipped files and what to do when a compiler job starts reading one of them.
Most compiler jobs run only once a maintainer adds the label "🚀 ready to merge" to a pull request; until then, a
smaller set of jobs runs on every push. See
[How CI runs](https://json.nlohmann.me/community/quality_assurance/#how-ci-runs) for the sets and when they run.
### Amalgamate the source code
The single-header files
+1 -1
View File
@@ -24,7 +24,7 @@ jobs:
permissions:
actions: write
runs-on: ubuntu-26.04
runs-on: ubuntu-latest
steps:
- name: Harden Runner
+1 -1
View File
@@ -19,7 +19,7 @@ permissions:
jobs:
save:
if: github.event_name == 'pull_request'
runs-on: ubuntu-26.04
runs-on: ubuntu-latest
steps:
- name: Harden Runner
uses: step-security/harden-runner@dc2769abcc0fcfbadfc1df54081c1608579ef6b2 # v2.22.2
+1 -1
View File
@@ -16,7 +16,7 @@ concurrency:
jobs:
check_docs_links:
if: github.repository == 'nlohmann/json'
runs-on: ubuntu-26.04
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- name: Harden Runner
@@ -12,7 +12,7 @@ jobs:
comment:
# push runs on develop have no PR to comment on (and no "pr" artifact)
if: ${{ github.event.workflow_run.conclusion == 'failure' && github.event.workflow_run.event == 'pull_request' }}
runs-on: ubuntu-26.04
runs-on: ubuntu-latest
permissions:
contents: read
actions: read
+1 -1
View File
@@ -18,7 +18,7 @@ permissions:
jobs:
dependency-review:
runs-on: ubuntu-26.04
runs-on: ubuntu-latest
steps:
- name: Harden Runner
uses: step-security/harden-runner@dc2769abcc0fcfbadfc1df54081c1608579ef6b2 # v2.22.2
+1 -1
View File
@@ -24,7 +24,7 @@ on:
jobs:
flawfinder:
name: Flawfinder
runs-on: ubuntu-26.04
runs-on: ubuntu-latest
permissions:
actions: read
contents: read
+1 -1
View File
@@ -19,7 +19,7 @@ jobs:
contents: read
pull-requests: write
runs-on: ubuntu-26.04
runs-on: ubuntu-latest
steps:
- name: Harden Runner
+83 -6
View File
@@ -7,25 +7,60 @@ on:
- master
- release/*
pull_request:
# labeled: the "🚀 ready to merge" label starts the full set (see below)
types: [opened, synchronize, reopened, labeled]
# Skip pull requests that change only files no job here reads. The list is checked by
# tools/check_path_filters/check_path_filters.py (repository_checks.yml); change both together.
paths:
- '**'
- '!docs/**'
- '!**.md'
- '!.github/*'
- '!.github/ISSUE_TEMPLATE/**'
- '!.github/external_ci/**'
- '!.github/workflows/**'
- '!.reuse/**'
- '!LICENSES/**'
- '!CITATION.cff'
- '!REUSE.toml'
# read by tools/check_build_options in the ci_meson_install job (ubuntu.yml)
- 'docs/mkdocs/docs/integration/package_managers.md'
# built by the swiftpm job (macos.yml)
- 'docs/mkdocs/docs/integration/swift/**'
- '.github/workflows/macos.yml'
workflow_dispatch:
# A run triggered by any other label gets a group of its own, so it does not
# cancel the run of the last push; its jobs are skipped.
concurrency:
group: ${{ github.workflow }}-${{ github.ref || github.run_id }}
group: ${{ github.workflow }}-${{ github.event_name == 'schedule' && 'nightly' || github.ref || github.run_id }}${{ github.event.action == 'labeled' && github.event.label.name != '🚀 ready to merge' && format('-{0}', github.run_id) || '' }}
cancel-in-progress: true
permissions:
contents: read
# The jobs are split into a PR set and a full set like in ubuntu.yml (see "How
# CI runs" in docs/mkdocs/docs/community/quality_assurance.md): the PR set runs
# on every push; the full set after the PR set passed, on pushes and on pull
# requests with the "🚀 ready to merge" label. tools/ci_jobs checks that every
# job belongs to exactly one set.
jobs:
###########################################################################
# PR set
###########################################################################
macos-15:
if: &pr_set ${{ github.event_name != 'schedule' && (github.event.action != 'labeled' || github.event.label.name == '🚀 ready to merge') }}
runs-on: macos-15 # https://github.com/actions/runner-images/blob/main/images/macos/macos-15-Readme.md
strategy:
matrix:
xcode: ['16.0', '16.1', '16.2', '16.3', '16.4', '26.0.1', '26.1.1', '26.2', '26.3']
xcode: ['16.0', '16.4', '26.3']
env:
DEVELOPER_DIR: /Applications/Xcode_${{ matrix.xcode }}.app/Contents/Developer
steps:
steps: &xcode_steps
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
@@ -37,10 +72,11 @@ jobs:
run: cd build ; ctest -j 10 --output-on-failure
macos-26:
if: *pr_set
runs-on: macos-26 # https://github.com/actions/runner-images/blob/main/images/macos/macos-26-arm64-Readme.md
strategy:
matrix:
xcode: ['26.4.1', '26.5', '26.6']
xcode: ['26.6']
env:
DEVELOPER_DIR: /Applications/Xcode_${{ matrix.xcode }}.app/Contents/Developer
@@ -56,12 +92,13 @@ jobs:
run: cd build ; ctest -j 10 --output-on-failure
xcode_standards:
if: *pr_set
runs-on: macos-latest
strategy:
matrix:
standard: [11, 14, 17, 20, 23, 26]
standard: [14]
steps:
steps: &xcode_standards_steps
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
@@ -73,6 +110,7 @@ jobs:
run: cd build ; ctest -j 10 --output-on-failure
swiftpm:
if: *pr_set
runs-on: macos-15
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
@@ -106,3 +144,42 @@ jobs:
EOF
cd /tmp/json-swiftpm-consumer
swift build
###########################################################################
# full set: starts after the PR set passed
###########################################################################
macos-15_full:
needs: &pr_set_jobs [macos-15, macos-26, xcode_standards, swiftpm]
if: &full_set ${{ github.event_name == 'push' || github.event_name == 'workflow_dispatch' || contains(github.event.pull_request.labels.*.name, '🚀 ready to merge') }}
runs-on: macos-15 # https://github.com/actions/runner-images/blob/main/images/macos/macos-15-Readme.md
strategy:
matrix:
xcode: ['16.1', '16.2', '16.3', '26.0.1', '26.1.1', '26.2']
env:
DEVELOPER_DIR: /Applications/Xcode_${{ matrix.xcode }}.app/Contents/Developer
steps: *xcode_steps
macos-26_full:
needs: *pr_set_jobs
if: *full_set
runs-on: macos-26 # https://github.com/actions/runner-images/blob/main/images/macos/macos-26-arm64-Readme.md
strategy:
matrix:
xcode: ['26.4.1', '26.5']
env:
DEVELOPER_DIR: /Applications/Xcode_${{ matrix.xcode }}.app/Contents/Developer
steps: *xcode_steps
xcode_standards_full:
needs: *pr_set_jobs
if: *full_set
runs-on: macos-latest
strategy:
matrix:
standard: [11, 17, 20, 23, 26]
steps: *xcode_standards_steps
+1 -1
View File
@@ -34,7 +34,7 @@ jobs:
contents: write
if: github.repository == 'nlohmann/json'
runs-on: ubuntu-26.04
runs-on: ubuntu-latest
steps:
- name: Harden Runner
uses: step-security/harden-runner@dc2769abcc0fcfbadfc1df54081c1608579ef6b2 # v2.22.2
+83
View File
@@ -0,0 +1,83 @@
name: Repository checks
# Checks that concern files the compiler workflows (ubuntu.yml, macos.yml,
# windows.yml) skip with their path filter: the documentation, the license
# information, and the CI configuration itself. This workflow has no path filter.
on:
push:
branches:
- develop
- master
- release/*
pull_request:
workflow_dispatch:
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.ref || github.run_id }}
cancel-in-progress: true
jobs:
check_ci_configuration:
runs-on: ubuntu-26.04
steps:
- name: Harden Runner
uses: step-security/harden-runner@dc2769abcc0fcfbadfc1df54081c1608579ef6b2 # v2.22.2
with:
egress-policy: audit
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Check the path filters of the compiler workflows
run: python3 tools/check_path_filters/check_path_filters.py
- name: Check the job sets of the compiler workflows and their documentation
run: |
python3 -m venv venv
venv/bin/pip install --quiet -r tools/ci_jobs/requirements.txt
venv/bin/python tools/ci_jobs/ci_jobs.py
ci_reuse_compliance:
runs-on: ubuntu-26.04
steps:
- name: Harden Runner
uses: step-security/harden-runner@dc2769abcc0fcfbadfc1df54081c1608579ef6b2 # v2.22.2
with:
egress-policy: audit
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Get latest CMake and ninja
uses: lukka/get-cmake@a89e43e22a96abafcd7a62db652a65ecd33052c9 # v4.4.4
- name: Run CMake
run: cmake -S . -B build -DJSON_CI=On
- name: Build
run: cmake --build build --target ci_reuse_compliance
ci_test_documentation:
runs-on: ubuntu-26.04
strategy:
matrix:
target: [ci_test_examples, ci_test_build_documentation, ci_test_documentation_mermaid]
steps:
- name: Harden Runner
uses: step-security/harden-runner@dc2769abcc0fcfbadfc1df54081c1608579ef6b2 # v2.22.2
with:
egress-policy: audit
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
# the git-revision-date-localized plugin needs the history; a shallow clone makes the strict build fail
fetch-depth: ${{ matrix.target == 'ci_test_build_documentation' && '0' || '1' }}
- uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
if: matrix.target == 'ci_test_documentation_mermaid'
with:
node-version: 24
- name: Run CMake
run: cmake -S . -B build -DJSON_CI=On
- name: Build
run: cmake --build build --target ${{ matrix.target }}
+1 -1
View File
@@ -24,7 +24,7 @@ permissions:
jobs:
analysis:
name: Scorecard analysis
runs-on: ubuntu-26.04
runs-on: ubuntu-latest
permissions:
# Needed to upload the results to code-scanning dashboard.
security-events: write
+1 -1
View File
@@ -8,7 +8,7 @@ permissions:
jobs:
stale:
runs-on: ubuntu-26.04
runs-on: ubuntu-latest
permissions:
issues: write
+427 -267
View File
@@ -7,18 +7,63 @@ on:
- master
- release/*
pull_request:
# labeled: the "🚀 ready to merge" label starts the full set (see below)
types: [opened, synchronize, reopened, labeled]
# Skip pull requests that change only files no job here reads. The list is checked by
# tools/check_path_filters/check_path_filters.py (repository_checks.yml); change both together.
paths:
- '**'
- '!docs/**'
- '!**.md'
- '!.github/*'
- '!.github/ISSUE_TEMPLATE/**'
- '!.github/external_ci/**'
- '!.github/workflows/**'
- '!.reuse/**'
- '!LICENSES/**'
- '!CITATION.cff'
- '!REUSE.toml'
# read by tools/check_build_options in the ci_meson_install job (ubuntu.yml)
- 'docs/mkdocs/docs/integration/package_managers.md'
# built by the swiftpm job (macos.yml)
- 'docs/mkdocs/docs/integration/swift/**'
- '.github/workflows/ubuntu.yml'
schedule:
# the nightly set (see below), on develop
- cron: '0 2 * * *'
workflow_dispatch:
permissions:
contents: read
# A run triggered by any other label gets a group of its own, so it does not
# cancel the run of the last push; its jobs are skipped.
concurrency:
group: ${{ github.workflow }}-${{ github.ref || github.run_id }}
group: ${{ github.workflow }}-${{ github.event_name == 'schedule' && 'nightly' || github.ref || github.run_id }}${{ github.event.action == 'labeled' && github.event.label.name != '🚀 ready to merge' && format('-{0}', github.run_id) || '' }}
cancel-in-progress: true
# The jobs are split into three sets (see "How CI runs" in
# docs/mkdocs/docs/community/quality_assurance.md):
#
# - PR set: every pull request push and every push; chosen because these jobs
# caught 92 % of the failing commits in Sep/Oct 2026.
# - full set: after the PR set passed, on pushes and on pull requests with the
# "🚀 ready to merge" label; every remaining compiler, version and standard.
# - nightly set: slow tools and rare compilers, nightly on develop, on pushes
# to master and release/*, and when started by hand.
#
# A matrix whose entries belong to different sets is split into one job per
# set; the jobs share their steps through YAML anchors. tools/ci_jobs checks
# that every job belongs to exactly one set.
jobs:
###########################################################################
# PR set
###########################################################################
ci_test_gcc:
runs-on: ubuntu-26.04
if: &pr_set ${{ github.event_name != 'schedule' && (github.event.action != 'labeled' || github.event.label.name == '🚀 ready to merge') }}
runs-on: ubuntu-latest
container: gcc:latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
@@ -32,6 +77,7 @@ jobs:
run: cmake --build build --target ci_test_gcc
ci_meson_install:
if: *pr_set
runs-on: ubuntu-26.04
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
@@ -76,7 +122,342 @@ jobs:
cmake -S tests/cmake_import/project -B build-import-split -DCMAKE_PREFIX_PATH=${{ github.workspace }}/install-split
cmake --build build-import-split
ci_static_analysis_ubuntu:
if: *pr_set
runs-on: ubuntu-26.04
strategy:
matrix:
target: [ci_cpplint]
steps: &static_analysis_ubuntu_steps
- name: Harden Runner
uses: step-security/harden-runner@dc2769abcc0fcfbadfc1df54081c1608579ef6b2 # v2.22.2
with:
egress-policy: audit
- name: Install Valgrind
if: matrix.target == 'ci_test_valgrind'
run: sudo apt-get update ; sudo apt-get install -y valgrind
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Get latest CMake and ninja
uses: lukka/get-cmake@a89e43e22a96abafcd7a62db652a65ecd33052c9 # v4.4.4
- name: Run CMake
run: cmake -S . -B build -DJSON_CI=On
- name: Build
run: cmake --build build --target ${{ matrix.target }}
ci_static_analysis_clang:
if: *pr_set
runs-on: ubuntu-latest
container: silkeh/clang:22
strategy:
matrix:
target: [ci_test_clang, ci_clang_tidy, ci_test_clang_sanitizer, ci_single_binaries]
steps: &static_analysis_clang_steps
- name: Install git, clang-tools, iwyu (ci_single_binaries), and unzip
run: apt-get update ; apt-get install -y git clang-tools iwyu unzip
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Get latest CMake and ninja
uses: lukka/get-cmake@a89e43e22a96abafcd7a62db652a65ecd33052c9 # v4.4.4
- name: Run CMake
run: cmake -S . -B build -DJSON_CI=On
- name: Build
run: cmake --build build --target ${{ matrix.target }}
ci_cmake_options:
if: *pr_set
runs-on: ubuntu-latest
container: ubuntu:24.04
strategy:
matrix:
target: [ci_test_diagnostics, ci_test_diagnostic_positions, ci_test_noexceptions, ci_test_noglobaludls]
steps: &cmake_options_steps
- name: Install build-essential
run: apt-get update ; apt-get install -y build-essential unzip wget git
# the CMake 3.5.0 release binary that ci_cmake_flags downloads links
# against libidn.so.11, which Ubuntu 24.04 no longer ships
- name: Install libidn11 for CMake 3.5.0
if: matrix.target == 'ci_cmake_flags'
run: |
wget -q http://archive.ubuntu.com/ubuntu/pool/main/libi/libidn/libidn11_1.33-2.2ubuntu2_amd64.deb
dpkg -i libidn11_1.33-2.2ubuntu2_amd64.deb
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Get latest CMake and ninja
uses: lukka/get-cmake@a89e43e22a96abafcd7a62db652a65ecd33052c9 # v4.4.4
- name: Run CMake
run: cmake -S . -B build -DJSON_CI=On
- name: Build
run: cmake --build build --target ${{ matrix.target }}
ci_test_compilers_gcc_old:
if: *pr_set
runs-on: ubuntu-latest
strategy:
matrix:
compiler: ['4.8']
# official gcc:4.8/4.9/5/6 images fail to check out code (too old for
# actions/checkout); install the old compilers on top of official ubuntu:20.04
# instead, mirroring what the (now retired) custom json-ci image did.
container: ubuntu:20.04
steps: &compilers_gcc_old_steps
- name: Install g++-${{ matrix.compiler }}
run: |
export DEBIAN_FRONTEND=noninteractive
apt-get update
apt-get install -y --no-install-recommends software-properties-common ca-certificates gnupg make git
# add-apt-repository resolves the PPA through the Launchpad API,
# which intermittently times out or fails the team lookup (the plain
# "deb ..." sources below never hit Launchpad and never flake).
# Retry with backoff so a transient Launchpad blip does not fail CI.
for attempt in 1 2 3 4 5; do
add-apt-repository -y ppa:ubuntu-toolchain-r/test && break
echo "::warning::add-apt-repository ppa:ubuntu-toolchain-r/test failed (attempt ${attempt}/5); retrying"
sleep $((attempt * 10))
done
apt-add-repository -y "deb http://archive.ubuntu.com/ubuntu/ bionic main"
apt-add-repository -y "deb http://archive.ubuntu.com/ubuntu/ bionic universe"
apt-add-repository -y "deb http://archive.ubuntu.com/ubuntu/ xenial main"
apt-add-repository -y "deb http://archive.ubuntu.com/ubuntu/ xenial universe"
apt-add-repository -y "deb http://archive.ubuntu.com/ubuntu/ xenial-updates main"
apt-add-repository -y "deb http://archive.ubuntu.com/ubuntu/ xenial-updates universe"
apt-get update
apt-get install -y --no-install-recommends g++-${{ matrix.compiler }}
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Get latest CMake and ninja
uses: lukka/get-cmake@a89e43e22a96abafcd7a62db652a65ecd33052c9 # v4.4.4
- name: Run CMake
run: CXX=g++-${{ matrix.compiler }} cmake -S . -B build -DJSON_CI=On
- name: Build
run: cmake --build build --target ci_test_compiler_g++-${{ matrix.compiler }}
ci_test_compilers_clang:
if: *pr_set
runs-on: ubuntu-latest
strategy:
matrix:
compiler: ['3.4']
container: silkeh/clang:${{ matrix.compiler }}
steps: &compilers_clang_steps
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Get latest CMake and ninja
uses: lukka/get-cmake@a89e43e22a96abafcd7a62db652a65ecd33052c9 # v4.4.4
- name: Disable filesystem support (std::filesystem is unusable) for clang 7 / 8 / 9 / 10
run: echo "JSON_FORCED_GLOBAL_COMPILE_OPTIONS=-DJSON_HAS_FILESYSTEM=0;-DJSON_HAS_EXPERIMENTAL_FILESYSTEM=0" >> "$GITHUB_ENV"
if: ${{ matrix.compiler == '7' || matrix.compiler == '8' || matrix.compiler == '9' || matrix.compiler == '10' }}
- name: Run CMake
run: cmake -S . -B build -DJSON_CI=On
- name: Build
run: cmake --build build --target ci_test_compiler_default
ci_test_standards_gcc:
if: *pr_set
runs-on: ubuntu-latest
container: gcc:latest
strategy:
matrix:
standard: [26]
steps: &standards_gcc_steps
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Get latest CMake and ninja
uses: lukka/get-cmake@a89e43e22a96abafcd7a62db652a65ecd33052c9 # v4.4.4
- name: Run CMake
run: cmake -S . -B build -DJSON_CI=On
- name: Build
run: cmake --build build --target ci_test_gcc_cxx${{ matrix.standard }}
ci_test_standards_clang:
if: *pr_set
runs-on: ubuntu-latest
container: silkeh/clang:latest
strategy:
matrix:
standard: [26]
stdlib: [libcxx]
steps: &standards_clang_steps
- name: Install git and unzip
run: apt-get update ; apt-get install -y git unzip
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Get latest CMake and ninja
uses: lukka/get-cmake@a89e43e22a96abafcd7a62db652a65ecd33052c9 # v4.4.4
- name: Run CMake
run: cmake -S . -B build -DJSON_CI=On
- name: Build with libc++
run: cmake --build build --target ci_test_clang_libcxx_cxx${{ matrix.standard }}
if: ${{ matrix.stdlib == 'libcxx' }}
- name: Build with libstdc++
run: cmake --build build --target ci_test_clang_cxx${{ matrix.standard }}
if: ${{ matrix.stdlib == 'libstdcxx' }}
ci_emscripten:
if: *pr_set
runs-on: ubuntu-26.04
steps:
- name: Harden Runner
uses: step-security/harden-runner@dc2769abcc0fcfbadfc1df54081c1608579ef6b2 # v2.22.2
with:
egress-policy: audit
- name: Install emscripten
uses: mymindstorm/setup-emsdk@4528d102f7230f0e7b276855c01ea1159be0e984 # v16
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Get latest CMake and ninja
uses: lukka/get-cmake@a89e43e22a96abafcd7a62db652a65ecd33052c9 # v4.4.4
- name: Run CMake
run: cmake -S . -B build -DCMAKE_TOOLCHAIN_FILE=$EMSDK/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake -GNinja
- name: Build
run: cmake --build build
###########################################################################
# full set: starts after the PR set passed
###########################################################################
ci_static_analysis_ubuntu_full:
needs: &pr_set_jobs [ci_test_gcc, ci_meson_install, ci_static_analysis_ubuntu, ci_static_analysis_clang, ci_cmake_options, ci_test_compilers_gcc_old, ci_test_compilers_clang, ci_test_standards_gcc, ci_test_standards_clang, ci_emscripten]
if: &full_set ${{ github.event_name == 'push' || github.event_name == 'workflow_dispatch' || contains(github.event.pull_request.labels.*.name, '🚀 ready to merge') }}
runs-on: ubuntu-26.04
strategy:
matrix:
target: [ci_test_single_header, ci_cppcheck, ci_reproducible_tests, ci_non_git_tests, ci_offline_testdata]
steps: *static_analysis_ubuntu_steps
ci_static_analysis_clang_full:
needs: *pr_set_jobs
if: *full_set
runs-on: ubuntu-latest
container: silkeh/clang:22
strategy:
matrix:
target: [ci_clang_analyze]
steps: *static_analysis_clang_steps
ci_cmake_options_full:
needs: *pr_set_jobs
if: *full_set
runs-on: ubuntu-latest
container: ubuntu:24.04
strategy:
matrix:
target: [ci_cmake_flags, ci_test_noimplicitconversions, ci_test_legacycomparison, ci_test_disableenumserialization, ci_test_disabletuplereferenceconversion, ci_test_skiplibraryversioncheck, ci_test_simdutf, ci_test_strict_nul_handling, ci_test_delete_deprecated_functions, ci_test_no_thread_local]
steps: *cmake_options_steps
ci_test_compilers_gcc_old_full:
needs: *pr_set_jobs
if: *full_set
runs-on: ubuntu-latest
strategy:
matrix:
compiler: ['4.9', '5', '6']
container: ubuntu:20.04
steps: *compilers_gcc_old_steps
ci_test_compilers_gcc_full:
needs: *pr_set_jobs
if: *full_set
runs-on: ubuntu-latest
strategy:
matrix:
# older GCC docker images (4, 5, 6) fail to check out code
compiler: ['7', '8', '9', '10', '11', '12', '13', '14', '15', '16', 'latest']
container: gcc:${{ matrix.compiler }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Get latest CMake and ninja
uses: lukka/get-cmake@a89e43e22a96abafcd7a62db652a65ecd33052c9 # v4.4.4
- name: Run CMake
run: cmake -S . -B build -DJSON_CI=On
- name: Build
run: cmake --build build --target ci_test_compiler_default
ci_test_compilers_clang_full:
needs: *pr_set_jobs
if: *full_set
runs-on: ubuntu-latest
strategy:
matrix:
compiler: ['3.5', '3.6', '3.7', '3.8', '3.9', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15-bullseye', '16', '17', '18', '19', '20', '21', '22', 'latest']
container: silkeh/clang:${{ matrix.compiler }}
steps: *compilers_clang_steps
ci_test_standards_gcc_full:
needs: *pr_set_jobs
if: *full_set
runs-on: ubuntu-latest
container: gcc:latest
strategy:
matrix:
standard: [11, 14, 17, 20, 23]
steps: *standards_gcc_steps
ci_test_standards_clang_full:
needs: *pr_set_jobs
if: *full_set
runs-on: ubuntu-latest
container: silkeh/clang:latest
strategy:
matrix:
standard: [11, 14, 17, 20, 23, 26]
stdlib: [libcxx, libstdcxx]
exclude:
# in the PR set
- standard: 26
stdlib: libcxx
steps: *standards_clang_steps
ci_module_cpp20:
needs: *pr_set_jobs
if: *full_set
strategy:
matrix:
container: ['gcc:latest', 'silkeh/clang:latest']
runs-on: ubuntu-latest
container: ${{ matrix.container }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
# The module test uses `import std;`, which needs CMake's experimental
# import-std support. Its opt-in token is CMake-version-specific, so pin
# CMake to the version whose token is set in tests/module_cpp20/CMakeLists.txt.
- name: Get pinned CMake and ninja
uses: lukka/get-cmake@a89e43e22a96abafcd7a62db652a65ecd33052c9 # v4.4.4
with:
cmakeVersion: 4.3.4
# Clang: the std library module is provided by libc++ (the image's libstdc++
# ships none), and the image's libc++ module manifest has a broken relative
# path — repoint it at the real module sources.
- name: Use libc++ and fix its module manifest path (Clang)
if: matrix.container == 'silkeh/clang:latest'
run: |
echo "CXXFLAGS=-stdlib=libc++" >> "$GITHUB_ENV"
mkdir -p /usr/lib/share && ln -sf /usr/lib/llvm-*/share/libc++ /usr/lib/share/libc++
- name: Run CMake
run: cmake -S . -B build -DJSON_CI=On
- name: Build
run: cmake --build build --target ci_module_cpp20
###########################################################################
# nightly set
###########################################################################
ci_infer:
if: &nightly_set ${{ github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' || (github.event_name == 'push' && github.ref != 'refs/heads/develop') }}
runs-on: ubuntu-26.04
steps:
- name: Harden Runner
@@ -105,76 +486,16 @@ jobs:
path: ${{ github.workspace }}/build/build_infer/infer-out/report.txt
if-no-files-found: ignore
ci_static_analysis_ubuntu:
ci_static_analysis_ubuntu_nightly:
if: *nightly_set
runs-on: ubuntu-26.04
strategy:
matrix:
target: [ci_test_single_header, ci_cppcheck, ci_cpplint, ci_reproducible_tests, ci_non_git_tests, ci_offline_testdata, ci_reuse_compliance, ci_test_valgrind]
steps:
- name: Harden Runner
uses: step-security/harden-runner@dc2769abcc0fcfbadfc1df54081c1608579ef6b2 # v2.22.2
with:
egress-policy: audit
- name: Install Valgrind
if: matrix.target == 'ci_test_valgrind'
run: sudo apt-get update ; sudo apt-get install -y valgrind
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Get latest CMake and ninja
uses: lukka/get-cmake@a89e43e22a96abafcd7a62db652a65ecd33052c9 # v4.4.4
- name: Run CMake
run: cmake -S . -B build -DJSON_CI=On
- name: Build
run: cmake --build build --target ${{ matrix.target }}
ci_static_analysis_clang:
runs-on: ubuntu-26.04
container: silkeh/clang:22
strategy:
matrix:
target: [ci_test_clang, ci_clang_tidy, ci_test_clang_sanitizer, ci_clang_analyze, ci_single_binaries]
steps:
- name: Install git, clang-tools, iwyu (ci_single_binaries), and unzip
run: apt-get update ; apt-get install -y git clang-tools iwyu unzip
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Get latest CMake and ninja
uses: lukka/get-cmake@a89e43e22a96abafcd7a62db652a65ecd33052c9 # v4.4.4
- name: Run CMake
run: cmake -S . -B build -DJSON_CI=On
- name: Build
run: cmake --build build --target ${{ matrix.target }}
ci_cmake_options:
runs-on: ubuntu-26.04
container: ubuntu:24.04
strategy:
matrix:
target: [ci_cmake_flags, ci_test_diagnostics, ci_test_diagnostic_positions, ci_test_noexceptions, ci_test_noimplicitconversions, ci_test_legacycomparison, ci_test_noglobaludls, ci_test_disableenumserialization, ci_test_disabletuplereferenceconversion, ci_test_skiplibraryversioncheck, ci_test_simdutf, ci_test_strict_nul_handling, ci_test_delete_deprecated_functions, ci_test_no_thread_local]
steps:
- name: Install build-essential
run: apt-get update ; apt-get install -y build-essential unzip wget git
# the CMake 3.5.0 release binary that ci_cmake_flags downloads links
# against libidn.so.11, which Ubuntu 24.04 no longer ships
- name: Install libidn11 for CMake 3.5.0
if: matrix.target == 'ci_cmake_flags'
run: |
wget -q http://archive.ubuntu.com/ubuntu/pool/main/libi/libidn/libidn11_1.33-2.2ubuntu2_amd64.deb
dpkg -i libidn11_1.33-2.2ubuntu2_amd64.deb
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Get latest CMake and ninja
uses: lukka/get-cmake@a89e43e22a96abafcd7a62db652a65ecd33052c9 # v4.4.4
- name: Run CMake
run: cmake -S . -B build -DJSON_CI=On
- name: Build
run: cmake --build build --target ${{ matrix.target }}
target: [ci_test_valgrind]
steps: *static_analysis_ubuntu_steps
ci_test_coverage:
if: *nightly_set
runs-on: ubuntu-26.04
steps:
- name: Harden Runner
@@ -208,129 +529,9 @@ jobs:
path-to-lcov: ${{ github.workspace }}/build/json.info.filtered.noexcept
fail-on-error: false
ci_test_compilers_gcc_old:
runs-on: ubuntu-26.04
strategy:
matrix:
compiler: ['4.8', '4.9', '5', '6']
# official gcc:4.8/4.9/5/6 images fail to check out code (too old for
# actions/checkout); install the old compilers on top of official ubuntu:20.04
# instead, mirroring what the (now retired) custom json-ci image did.
container: ubuntu:20.04
steps:
- name: Install g++-${{ matrix.compiler }}
run: |
export DEBIAN_FRONTEND=noninteractive
apt-get update
apt-get install -y --no-install-recommends software-properties-common ca-certificates gnupg make git
# add-apt-repository resolves the PPA through the Launchpad API,
# which intermittently times out or fails the team lookup (the plain
# "deb ..." sources below never hit Launchpad and never flake).
# Retry with backoff so a transient Launchpad blip does not fail CI.
for attempt in 1 2 3 4 5; do
add-apt-repository -y ppa:ubuntu-toolchain-r/test && break
echo "::warning::add-apt-repository ppa:ubuntu-toolchain-r/test failed (attempt ${attempt}/5); retrying"
sleep $((attempt * 10))
done
apt-add-repository -y "deb http://archive.ubuntu.com/ubuntu/ bionic main"
apt-add-repository -y "deb http://archive.ubuntu.com/ubuntu/ bionic universe"
apt-add-repository -y "deb http://archive.ubuntu.com/ubuntu/ xenial main"
apt-add-repository -y "deb http://archive.ubuntu.com/ubuntu/ xenial universe"
apt-add-repository -y "deb http://archive.ubuntu.com/ubuntu/ xenial-updates main"
apt-add-repository -y "deb http://archive.ubuntu.com/ubuntu/ xenial-updates universe"
apt-get update
apt-get install -y --no-install-recommends g++-${{ matrix.compiler }}
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Get latest CMake and ninja
uses: lukka/get-cmake@a89e43e22a96abafcd7a62db652a65ecd33052c9 # v4.4.4
- name: Run CMake
run: CXX=g++-${{ matrix.compiler }} cmake -S . -B build -DJSON_CI=On
- name: Build
run: cmake --build build --target ci_test_compiler_g++-${{ matrix.compiler }}
ci_test_compilers_gcc:
runs-on: ubuntu-26.04
strategy:
matrix:
# older GCC docker images (4, 5, 6) fail to check out code
compiler: ['7', '8', '9', '10', '11', '12', '13', '14', '15', '16', 'latest']
container: gcc:${{ matrix.compiler }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Get latest CMake and ninja
uses: lukka/get-cmake@a89e43e22a96abafcd7a62db652a65ecd33052c9 # v4.4.4
- name: Run CMake
run: cmake -S . -B build -DJSON_CI=On
- name: Build
run: cmake --build build --target ci_test_compiler_default
ci_test_compilers_clang:
runs-on: ubuntu-26.04
strategy:
matrix:
compiler: ['3.4', '3.5', '3.6', '3.7', '3.8', '3.9', '4', '5', '6', '7', '8', '9', '10', '11', '12', '13', '14', '15-bullseye', '16', '17', '18', '19', '20', '21', '22', 'latest']
container: silkeh/clang:${{ matrix.compiler }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Get latest CMake and ninja
uses: lukka/get-cmake@a89e43e22a96abafcd7a62db652a65ecd33052c9 # v4.4.4
- name: Disable filesystem support (std::filesystem is unusable) for clang 7 / 8 / 9 / 10
run: echo "JSON_FORCED_GLOBAL_COMPILE_OPTIONS=-DJSON_HAS_FILESYSTEM=0;-DJSON_HAS_EXPERIMENTAL_FILESYSTEM=0" >> "$GITHUB_ENV"
if: ${{ matrix.compiler == '7' || matrix.compiler == '8' || matrix.compiler == '9' || matrix.compiler == '10' }}
- name: Run CMake
run: cmake -S . -B build -DJSON_CI=On
- name: Build
run: cmake --build build --target ci_test_compiler_default
ci_test_standards_gcc:
runs-on: ubuntu-26.04
container: gcc:latest
strategy:
matrix:
standard: [11, 14, 17, 20, 23, 26]
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Get latest CMake and ninja
uses: lukka/get-cmake@a89e43e22a96abafcd7a62db652a65ecd33052c9 # v4.4.4
- name: Run CMake
run: cmake -S . -B build -DJSON_CI=On
- name: Build
run: cmake --build build --target ci_test_gcc_cxx${{ matrix.standard }}
ci_test_standards_clang:
runs-on: ubuntu-26.04
container: silkeh/clang:latest
strategy:
matrix:
standard: [11, 14, 17, 20, 23, 26]
stdlib: [libcxx, libstdcxx]
steps:
- name: Install git and unzip
run: apt-get update ; apt-get install -y git unzip
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Get latest CMake and ninja
uses: lukka/get-cmake@a89e43e22a96abafcd7a62db652a65ecd33052c9 # v4.4.4
- name: Run CMake
run: cmake -S . -B build -DJSON_CI=On
- name: Build with libc++
run: cmake --build build --target ci_test_clang_libcxx_cxx${{ matrix.standard }}
if: ${{ matrix.stdlib == 'libcxx' }}
- name: Build with libstdc++
run: cmake --build build --target ci_test_clang_cxx${{ matrix.standard }}
if: ${{ matrix.stdlib == 'libstdcxx' }}
ci_cuda_example:
runs-on: ubuntu-26.04
if: *nightly_set
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
@@ -352,38 +553,9 @@ jobs:
- name: Build
run: cmake --build build --target ci_cuda_example
ci_module_cpp20:
strategy:
matrix:
container: ['gcc:latest', 'silkeh/clang:latest']
runs-on: ubuntu-26.04
container: ${{ matrix.container }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
# The module test uses `import std;`, which needs CMake's experimental
# import-std support. Its opt-in token is CMake-version-specific, so pin
# CMake to the version whose token is set in tests/module_cpp20/CMakeLists.txt.
- name: Get pinned CMake and ninja
uses: lukka/get-cmake@a89e43e22a96abafcd7a62db652a65ecd33052c9 # v4.4.4
with:
cmakeVersion: 4.3.4
# Clang: the std library module is provided by libc++ (the image's libstdc++
# ships none), and the image's libc++ module manifest has a broken relative
# path — repoint it at the real module sources.
- name: Use libc++ and fix its module manifest path (Clang)
if: matrix.container == 'silkeh/clang:latest'
run: |
echo "CXXFLAGS=-stdlib=libc++" >> "$GITHUB_ENV"
mkdir -p /usr/lib/share && ln -sf /usr/lib/llvm-*/share/libc++ /usr/lib/share/libc++
- name: Run CMake
run: cmake -S . -B build -DJSON_CI=On
- name: Build
run: cmake --build build --target ci_module_cpp20
ci_icpc:
runs-on: ubuntu-26.04
if: *nightly_set
runs-on: ubuntu-latest
# Intel discontinued the classic icc/icpc compiler in oneAPI 2024.0; this is
# Intel's own last officially published image that still includes it.
container: intel/oneapi-hpckit:2023.2.1-devel-ubuntu22.04
@@ -402,7 +574,8 @@ jobs:
run: cmake --build build --target ci_icpc
ci_icpx:
runs-on: ubuntu-26.04
if: *nightly_set
runs-on: ubuntu-latest
container: intel/oneapi-hpckit:latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
@@ -416,7 +589,8 @@ jobs:
run: cmake --build build --target ci_icpx
ci_nvhpc:
runs-on: ubuntu-26.04
if: *nightly_set
runs-on: ubuntu-latest
container: nvcr.io/nvidia/nvhpc:25.5-devel-cuda12.9-ubuntu22.04
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
@@ -429,54 +603,10 @@ jobs:
- name: Build
run: cmake --build build --target ci_nvhpc
ci_emscripten:
runs-on: ubuntu-26.04
steps:
- name: Harden Runner
uses: step-security/harden-runner@dc2769abcc0fcfbadfc1df54081c1608579ef6b2 # v2.22.2
with:
egress-policy: audit
- name: Install emscripten
uses: mymindstorm/setup-emsdk@4528d102f7230f0e7b276855c01ea1159be0e984 # v16
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Get latest CMake and ninja
uses: lukka/get-cmake@a89e43e22a96abafcd7a62db652a65ecd33052c9 # v4.4.4
- name: Run CMake
run: cmake -S . -B build -DCMAKE_TOOLCHAIN_FILE=$EMSDK/upstream/emscripten/cmake/Modules/Platform/Emscripten.cmake -GNinja
- name: Build
run: cmake --build build
ci_test_documentation:
runs-on: ubuntu-26.04
strategy:
matrix:
target: [ci_test_examples, ci_test_build_documentation, ci_test_documentation_mermaid]
steps:
- name: Harden Runner
uses: step-security/harden-runner@dc2769abcc0fcfbadfc1df54081c1608579ef6b2 # v2.22.2
with:
egress-policy: audit
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
# the git-revision-date-localized plugin needs the history; a shallow clone makes the strict build fail
fetch-depth: ${{ matrix.target == 'ci_test_build_documentation' && '0' || '1' }}
- uses: actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0
if: matrix.target == 'ci_test_documentation_mermaid'
with:
node-version: 24
- name: Run CMake
run: cmake -S . -B build -DJSON_CI=On
- name: Build
run: cmake --build build --target ${{ matrix.target }}
# Native Linux ARM64 runner with the latest GCC. The ubuntu-24.04-arm label is
# only available for public repositories.
ci_test_arm64:
if: *nightly_set
runs-on: ubuntu-24.04-arm
container: gcc:latest
steps:
@@ -491,3 +621,33 @@ jobs:
run: cmake -S . -B build -DJSON_CI=On
- name: Build
run: cmake --build build --target ci_test_compiler_default
# A failure of a scheduled run opens an issue, or comments on the open one.
nightly_report:
needs: [ci_infer, ci_static_analysis_ubuntu_nightly, ci_test_coverage, ci_cuda_example, ci_icpc, ci_icpx, ci_nvhpc, ci_test_arm64]
if: ${{ always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure') }}
runs-on: ubuntu-26.04
permissions:
issues: write
steps:
- name: Harden Runner
uses: step-security/harden-runner@dc2769abcc0fcfbadfc1df54081c1608579ef6b2 # v2.22.2
with:
egress-policy: audit
- name: Open or update the nightly failure issue
env:
GH_TOKEN: ${{ github.token }}
GH_REPO: ${{ github.repository }}
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
WORKFLOW: ${{ github.workflow }}
SHA: ${{ github.sha }}
run: |
title="Nightly CI failure (${WORKFLOW})"
body="The nightly run of ${WORKFLOW} on ${SHA:0:10} failed: ${RUN_URL}"
number=$(gh issue list --state open --search "in:title \"${title}\"" --json number,title --jq "map(select(.title == \"${title}\")) | .[0].number // empty")
if [ -n "$number" ]; then
gh issue comment "$number" --body "$body"
else
gh issue create --title "$title" --body "$body"
fi
+215 -33
View File
@@ -7,23 +7,62 @@ on:
- master
- release/*
pull_request:
# labeled: the "🚀 ready to merge" label starts the full set (see below)
types: [opened, synchronize, reopened, labeled]
# Skip pull requests that change only files no job here reads. The list is checked by
# tools/check_path_filters/check_path_filters.py (repository_checks.yml); change both together.
paths:
- '**'
- '!docs/**'
- '!**.md'
- '!.github/*'
- '!.github/ISSUE_TEMPLATE/**'
- '!.github/external_ci/**'
- '!.github/workflows/**'
- '!.reuse/**'
- '!LICENSES/**'
- '!CITATION.cff'
- '!REUSE.toml'
# read by tools/check_build_options in the ci_meson_install job (ubuntu.yml)
- 'docs/mkdocs/docs/integration/package_managers.md'
# built by the swiftpm job (macos.yml)
- 'docs/mkdocs/docs/integration/swift/**'
- '.github/workflows/windows.yml'
schedule:
# the nightly set (see below), on develop
- cron: '0 2 * * *'
workflow_dispatch:
permissions:
contents: read
# A run triggered by any other label gets a group of its own, so it does not
# cancel the run of the last push; its jobs are skipped.
concurrency:
group: ${{ github.workflow }}-${{ github.ref || github.run_id }}
group: ${{ github.workflow }}-${{ github.event_name == 'schedule' && 'nightly' || github.ref || github.run_id }}${{ github.event.action == 'labeled' && github.event.label.name != '🚀 ready to merge' && format('-{0}', github.run_id) || '' }}
cancel-in-progress: true
# The jobs are split into three sets like in ubuntu.yml (see "How CI runs" in
# docs/mkdocs/docs/community/quality_assurance.md): the PR set runs on every
# push; the full set after the PR set passed, on pushes and on pull requests
# with the "🚀 ready to merge" label; the nightly set nightly on develop, on
# pushes to master and release/*, and when started by hand. tools/ci_jobs
# checks that every job belongs to exactly one set.
jobs:
###########################################################################
# PR set
###########################################################################
mingw:
if: &pr_set ${{ github.event_name != 'schedule' && (github.event.action != 'labeled' || github.event.label.name == '🚀 ready to merge') }}
runs-on: windows-2022
strategy:
matrix:
architecture: [x64, x86]
architecture: [x86]
steps:
steps: &mingw_steps
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
@@ -40,15 +79,26 @@ jobs:
run: cd build ; ctest -j 10 -C Debug --output-on-failure
msvc:
if: *pr_set
strategy:
matrix:
build_type: [Debug, Release]
architecture: [Win32, x64]
std_version: [default, latest]
include:
- build_type: Debug
architecture: x64
std_version: default
- build_type: Release
architecture: x64
std_version: default
- build_type: Debug
architecture: Win32
std_version: default
- build_type: Release
architecture: Win32
std_version: latest
runs-on: windows-2022
steps:
steps: &msvc_steps
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
@@ -77,15 +127,20 @@ jobs:
# Visual Studio 2026 (v145 toolset) on the windows-2025 image. The "Visual Studio
# 18 2026" generator requires CMake 4.2+, so a recent CMake is fetched explicitly.
msvc-vs2026:
if: *pr_set
strategy:
matrix:
build_type: [Debug, Release]
architecture: [Win32, x64]
std_version: [default, latest]
include:
- build_type: Debug
architecture: x64
std_version: latest
- build_type: Release
architecture: Win32
std_version: default
runs-on: windows-2025
steps:
steps: &msvc_vs2026_steps
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
@@ -117,13 +172,14 @@ jobs:
# Native ARM64 Windows runner with the MSVC ARM64 toolset. The windows-11-arm
# label is only available for public repositories.
msvc-arm64:
if: *pr_set
strategy:
matrix:
build_type: [Debug, Release]
build_type: [Debug]
runs-on: windows-11-arm
steps:
steps: &msvc_arm64_steps
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
@@ -141,12 +197,13 @@ jobs:
run: cd build ; ctest -j 10 -C ${{ matrix.build_type }} --output-on-failure
clang:
if: *pr_set
runs-on: windows-2022
strategy:
matrix:
version: [11.0.1, 12.0.1, 13.0.1, 14.0.6, 15.0.7, 16.0.6, 18.1.8, 19.1.7, 20.1.8]
version: [20.1.8]
steps:
steps: &clang_steps
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
@@ -189,24 +246,8 @@ jobs:
- name: Test
run: cd build ; ctest -j 10 -C Debug --output-on-failure
clang-cl:
runs-on: windows-2022
strategy:
matrix:
architecture: [Win32, x64]
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Run CMake
run: cmake -S . -B build -G "Visual Studio 17 2022" -A ${{ matrix.architecture }} -T ClangCL -DJSON_BuildTests=On
- name: Build
run: cmake --build build --config Debug --parallel 10
- name: Test
run: cd build ; ctest -j 10 -C Debug --output-on-failure
ci_module_cpp20:
if: *pr_set
runs-on: windows-2022
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
@@ -225,4 +266,145 @@ jobs:
run: cmake -S . -B build -G "Visual Studio 17 2022" -DJSON_CI=ON -DCMAKE_CXX_FLAGS="/permissive- /std:c++latest /utf-8 /EHsc /W4 /WX"
- name: Build
run: cmake --build build --config Debug --target ci_module_cpp20
###########################################################################
# full set: starts after the PR set passed
###########################################################################
msvc_full:
needs: &pr_set_jobs [mingw, msvc, msvc-vs2026, msvc-arm64, clang, ci_module_cpp20]
if: &full_set ${{ github.event_name == 'push' || github.event_name == 'workflow_dispatch' || contains(github.event.pull_request.labels.*.name, '🚀 ready to merge') }}
strategy:
matrix:
build_type: [Debug, Release]
architecture: [Win32, x64]
std_version: [default, latest]
exclude:
# in the PR set
- build_type: Debug
architecture: x64
std_version: default
- build_type: Release
architecture: x64
std_version: default
- build_type: Debug
architecture: Win32
std_version: default
- build_type: Release
architecture: Win32
std_version: latest
runs-on: windows-2022
steps: *msvc_steps
# Visual Studio 2026 (v145 toolset) on the windows-2025 image. The "Visual Studio
# 18 2026" generator requires CMake 4.2+, so a recent CMake is fetched explicitly.
msvc-vs2026_full:
needs: *pr_set_jobs
if: *full_set
strategy:
matrix:
build_type: [Debug, Release]
architecture: [Win32, x64]
std_version: [default, latest]
exclude:
# in the PR set
- build_type: Debug
architecture: x64
std_version: latest
- build_type: Release
architecture: Win32
std_version: default
runs-on: windows-2025
steps: *msvc_vs2026_steps
# Native ARM64 Windows runner with the MSVC ARM64 toolset. The windows-11-arm
# label is only available for public repositories.
msvc-arm64_full:
needs: *pr_set_jobs
if: *full_set
strategy:
matrix:
build_type: [Release]
runs-on: windows-11-arm
steps: *msvc_arm64_steps
clang_full:
needs: *pr_set_jobs
if: *full_set
runs-on: windows-2022
strategy:
matrix:
version: [11.0.1, 12.0.1, 13.0.1, 14.0.6, 15.0.7, 16.0.6, 18.1.8, 19.1.7]
steps: *clang_steps
clang-cl_full:
needs: *pr_set_jobs
if: *full_set
runs-on: windows-2022
strategy:
matrix:
architecture: [Win32, x64]
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Run CMake
run: cmake -S . -B build -G "Visual Studio 17 2022" -A ${{ matrix.architecture }} -T ClangCL -DJSON_BuildTests=On
- name: Build
run: cmake --build build --config Debug --parallel 10
- name: Test
run: cd build ; ctest -j 10 -C Debug --output-on-failure
###########################################################################
# nightly set
###########################################################################
mingw_nightly:
if: ${{ github.event_name == 'schedule' || github.event_name == 'workflow_dispatch' || (github.event_name == 'push' && github.ref != 'refs/heads/develop') }}
runs-on: windows-2022
strategy:
matrix:
architecture: [x64]
steps: *mingw_steps
# A failure of a scheduled run opens an issue, or comments on the open one.
nightly_report:
needs: [mingw_nightly]
if: ${{ always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure') }}
runs-on: ubuntu-26.04
permissions:
issues: write
steps:
- name: Harden Runner
uses: step-security/harden-runner@ccd8616d44fd3846e67624a50d5aad6d37bf2d25 # v2.22.1
with:
egress-policy: audit
- name: Open or update the nightly failure issue
env:
GH_TOKEN: ${{ github.token }}
GH_REPO: ${{ github.repository }}
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
WORKFLOW: ${{ github.workflow }}
SHA: ${{ github.sha }}
run: |
title="Nightly CI failure (${WORKFLOW})"
body="The nightly run of ${WORKFLOW} on ${SHA:0:10} failed: ${RUN_URL}"
number=$(gh issue list --state open --search "in:title \"${title}\"" --json number,title --jq "map(select(.title == \"${title}\")) | .[0].number // empty")
if [ -n "$number" ]; then
gh issue comment "$number" --body "$body"
else
gh issue create --title "$title" --body "$body"
fi
+2 -2
View File
@@ -495,7 +495,7 @@ bool key(string_t& val);
bool parse_error(std::size_t position, const std::string& last_token, const detail::exception& ex);
```
The return value of each function determines whether parsing should proceed. For `parse_error`, returning `true` [recovers from the error](https://json.nlohmann.me/features/parsing/error_recovery/): the parser repairs the input and continues.
The return value of each function determines whether parsing should proceed.
To implement your own SAX handler, proceed as follows:
@@ -503,7 +503,7 @@ To implement your own SAX handler, proceed as follows:
2. Create an object of your SAX interface class, e.g. `my_sax`.
3. Call `bool json::sax_parse(input, &my_sax)`; where the first parameter can be any input like a string or an input stream and the second parameter is a pointer to your SAX interface.
Note the `sax_parse` function only returns a `bool` indicating whether the input was parsed without errors and no SAX event returned `false`. It does not return a `json` value - it is up to you to decide what to do with the SAX events. Furthermore, no exceptions are thrown in case of a parse error -- it is up to you what to do with the exception object passed to your `parse_error` implementation. Internally, the SAX interface is used for the DOM parser (class `json_sax_dom_parser`) as well as the acceptor (`json_sax_acceptor`), see file [`json_sax.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/json_sax.hpp).
Note the `sax_parse` function only returns a `bool` indicating the result of the last executed SAX event. It does not return a `json` value - it is up to you to decide what to do with the SAX events. Furthermore, no exceptions are thrown in case of a parse error -- it is up to you what to do with the exception object passed to your `parse_error` implementation. Internally, the SAX interface is used for the DOM parser (class `json_sax_dom_parser`) as well as the acceptor (`json_sax_acceptor`), see file [`json_sax.hpp`](https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/json_sax.hpp).
### STL-like access
+1 -4
View File
@@ -104,9 +104,7 @@ The SAX event lister must follow the interface of [`json_sax`](../json_sax/index
## Return value
`#!cpp true` if the input was parsed without errors and no SAX event returned `#!cpp false`; `#!cpp false` otherwise.
In particular, the result is `#!cpp false` for input with errors, even if the SAX parser recovered from all of them
(see [error recovery](../../features/parsing/error_recovery.md)).
return value of the last processed SAX event
## Exception safety
@@ -165,7 +163,6 @@ use the same file with stdio functions, but it must not wait for another thread
- Added `ignore_trailing_commas` in version 3.13.0.
- Added `tag_handler` in version 3.13.0.
- Extended container support (1) to include types with lvalue-only ADL `begin`/`end` (matching `std::begin`/`std::end` semantics) in version 3.13.0.
- Recovering from parse errors (see [`parse_error`](../json_sax/parse_error.md)) added in version 3.13.0.
- Extended overload (2) to accept heterogeneous iterator+sentinel pairs (C++20 ranges support) in version 3.13.0.
- `JSON_PRECISE_STREAM_POSITION` added in version 3.13.0 to optionally leave a `#!cpp std::istream` positioned right
after the parsed value when `strict` is `#!cpp false`.
@@ -13,15 +13,9 @@ function never throws an exception for invalid input, and the returned `#!cpp bo
is returned.
The result is the same as that of `BasicJsonType::accept()` by design; the difference is how the check is made. This
function runs the document's builder, which checks the text as [`parse()`](parse.md) does, while
`BasicJsonType::accept()` runs the SAX parser with a handler that ignores every event. Use this one if you go on to
parse the same input with [`parse()`](parse.md) anyway, so that both use the same code; either is fine for a plain
yes/no answer.
When both `ignore_comments` and `ignore_trailing_commas` are `#!cpp false` (the default), the builder writes no index.
With either option set, it builds the index as [`parse()`](parse.md) does and discards it: the result is the same, only
the time differs. (Neither option needs the index; the variant that skips it exists only for the default options to
keep the code size down.)
function runs the document's index builder (and discards the index), while `BasicJsonType::accept()` runs the SAX
parser with a handler that ignores every event. Use this one if you go on to parse the same input with
[`parse()`](parse.md) anyway, so that both use the same code; either is fine for a plain yes/no answer.
## Template parameters
+3 -4
View File
@@ -68,10 +68,9 @@ Strong guarantee: there are no changes to the view or the document it refers to.
- Throws [`out_of_range.404`](../../home/exceptions.md#jsonexceptionout_of_range404) if `ptr` cannot be resolved
because a reference token is used on a primitive value.
As for `BasicJsonType`, with [`JSON_DIAGNOSTICS`](../macros/json_diagnostics.md) these exceptions name the JSON
Pointer path of the value within the document, and with
[`JSON_DIAGNOSTIC_POSITIONS`](../macros/json_diagnostic_positions.md) its bytes in the source text (when
[`materialize()`](materialize.md) sets them).
None of these exceptions carry a [`JSON_DIAGNOSTICS`](../macros/json_diagnostics.md) path: the view has no
`BasicJsonType` value to point at, so the exception is created without one, even if `BasicJsonType` was built with
`JSON_DIAGNOSTICS` enabled.
## Complexity
+3 -3
View File
@@ -20,9 +20,9 @@ Strong guarantee: there are no changes to the view or the document it refers to.
Throws [`invalid_iterator.214`](../../home/exceptions.md#jsonexceptioninvalid_iterator214) if the view is
[null](is_null.md) or [discarded](is_discarded.md), or if it is an empty array or object.
As for `BasicJsonType`, with [`JSON_DIAGNOSTICS`](../macros/json_diagnostics.md) the exception names the JSON Pointer
path of the value within the document, and with [`JSON_DIAGNOSTIC_POSITIONS`](../macros/json_diagnostic_positions.md)
its bytes in the source text (when [`materialize()`](materialize.md) sets them).
This exception does not carry a [`JSON_DIAGNOSTICS`](../macros/json_diagnostics.md) path: the view has no
`BasicJsonType` value to point at, so the exception is created without one, even if `BasicJsonType` was built with
`JSON_DIAGNOSTICS` enabled.
## Complexity
@@ -20,9 +20,9 @@ Strong guarantee: there are no changes to the view or the document it refers to.
Throws [`invalid_iterator.214`](../../home/exceptions.md#jsonexceptioninvalid_iterator214) if the view is
[null](is_null.md) or [discarded](is_discarded.md), or if it is an empty array or object.
As for `BasicJsonType`, with [`JSON_DIAGNOSTICS`](../macros/json_diagnostics.md) the exception names the JSON Pointer
path of the value within the document, and with [`JSON_DIAGNOSTIC_POSITIONS`](../macros/json_diagnostic_positions.md)
its bytes in the source text (when [`materialize()`](materialize.md) sets them).
This exception does not carry a [`JSON_DIAGNOSTICS`](../macros/json_diagnostics.md) path: the view has no
`BasicJsonType` value to point at, so the exception is created without one, even if `BasicJsonType` was built with
`JSON_DIAGNOSTICS` enabled.
## Complexity
+8 -7
View File
@@ -80,13 +80,14 @@ Strong guarantee: there are no changes to the view or the document it refers to.
- For every other `T`: whatever [`materialize().get<T>()`](../basic_json/get.md) throws — typically `type_error.302`,
or whatever a user-provided `from_json()` throws.
The exceptions thrown directly by this function or by a `from_json()` that reads the view (the first four bullets
above) carry the context of those of `BasicJsonType`: with [`JSON_DIAGNOSTICS`](../macros/json_diagnostics.md) the
JSON Pointer path of the value within the document, and with
[`JSON_DIAGNOSTIC_POSITIONS`](../macros/json_diagnostic_positions.md) its bytes in the source text. An exception
thrown while converting through `materialize()` (the last bullet) is thrown by the materialized `BasicJsonType` value
instead: its path is relative to the root of the view's subtree, because `materialize()` builds only the subtree
(with parent pointers), so it is empty if the failing value is the subtree's root itself.
None of the exceptions thrown directly by this function or by a `from_json()` that reads the view (the first four
bullets above) carry a
[`JSON_DIAGNOSTICS`](../macros/json_diagnostics.md) path: the view has no `BasicJsonType` value to point at. An
exception thrown while converting through `materialize()` (the last bullet) is different: it is thrown by a real
`BasicJsonType` value, so it **does** carry a `JSON_DIAGNOSTICS` path if `BasicJsonType` was built with it enabled.
That path is relative to the root of the view's subtree, because `materialize()` builds only the subtree (with parent
pointers): it is empty if the failing value is the subtree's root itself, and it never is the path within the whole
document.
## Complexity
@@ -22,9 +22,9 @@ Strong guarantee: there are no changes to the view or the document it refers to.
Throws [`type_error.302`](../../home/exceptions.md#jsonexceptiontype_error302) if the value is not a string; example:
`"type must be string, but is array"`.
As for `BasicJsonType`, with [`JSON_DIAGNOSTICS`](../macros/json_diagnostics.md) the exception names the JSON Pointer
path of the value within the document, and with [`JSON_DIAGNOSTIC_POSITIONS`](../macros/json_diagnostic_positions.md)
its bytes in the source text (when [`materialize()`](materialize.md) sets them).
This exception does not carry a [`JSON_DIAGNOSTICS`](../macros/json_diagnostics.md) path: the view has no
`BasicJsonType` value to point at, so the exception is created without one, even if `BasicJsonType` was built with
`JSON_DIAGNOSTICS` enabled.
## Complexity
@@ -22,9 +22,9 @@ Strong guarantee: there are no changes to the view or the document it refers to.
Throws [`type_error.302`](../../home/exceptions.md#jsonexceptiontype_error302) if the value is not a number; example:
`"type must be number, but is string"`.
As for `BasicJsonType`, with [`JSON_DIAGNOSTICS`](../macros/json_diagnostics.md) the exception names the JSON Pointer
path of the value within the document, and with [`JSON_DIAGNOSTIC_POSITIONS`](../macros/json_diagnostic_positions.md)
its bytes in the source text (when [`materialize()`](materialize.md) sets them).
This exception does not carry a [`JSON_DIAGNOSTICS`](../macros/json_diagnostics.md) path: the view has no
`BasicJsonType` value to point at, so the exception is created without one, even if `BasicJsonType` was built with
`JSON_DIAGNOSTICS` enabled.
## Complexity
@@ -73,10 +73,9 @@ Strong guarantee: there are no changes to the view or the document it refers to.
- [`parse_error.109`](../../home/exceptions.md#jsonexceptionparse_error109) if an array index in `ptr` is not a
number.
As for `BasicJsonType`, with [`JSON_DIAGNOSTICS`](../macros/json_diagnostics.md) these exceptions name the JSON
Pointer path of the value within the document, and with
[`JSON_DIAGNOSTIC_POSITIONS`](../macros/json_diagnostic_positions.md) its bytes in the source text (when
[`materialize()`](materialize.md) sets them).
None of these exceptions carry a [`JSON_DIAGNOSTICS`](../macros/json_diagnostics.md) path: the view has no
`BasicJsonType` value to point at, so the exception is created without one, even if `BasicJsonType` was built with
`JSON_DIAGNOSTICS` enabled.
## Complexity
@@ -61,10 +61,9 @@ Strong guarantee: there are no changes to the view or the document it refers to.
`default_value` instead of throwing, exactly as [`BasicJsonType::value`](../basic_json/value.md) catches
`out_of_range` and returns `default_value`.
As for `BasicJsonType`, with [`JSON_DIAGNOSTICS`](../macros/json_diagnostics.md) these exceptions name the JSON
Pointer path of the value within the document, and with
[`JSON_DIAGNOSTIC_POSITIONS`](../macros/json_diagnostic_positions.md) its bytes in the source text (when
[`materialize()`](materialize.md) sets them).
None of these exceptions carry a [`JSON_DIAGNOSTICS`](../macros/json_diagnostics.md) path: the view has no
`BasicJsonType` value to point at, so the exception is created without one, even if `BasicJsonType` was built with
`JSON_DIAGNOSTICS` enabled.
## Complexity
+1 -2
View File
@@ -7,8 +7,7 @@ struct json_sax;
This class describes the SAX interface used by [sax_parse](../basic_json/sax_parse.md). Each function is called in
different situations while the input is parsed. The boolean return value informs the parser whether to continue
processing the input; for [`parse_error`](parse_error.md), it decides whether to
[recover from the error](../../features/parsing/error_recovery.md).
processing the input.
For instance, parsing the JSON text `{"a": [1, true]}` triggers the following callbacks, in order:
+2 -26
View File
@@ -21,18 +21,11 @@ A parse error occurred.
## Return value
Whether to recover from the error:
- `#!cpp false` stops parsing.
- `#!cpp true` recovers from the error: the error is repaired and parsing continues. If that is not possible, which
happens in the binary formats when the end of the item with the error is unknown, the value read so far is completed
and parsing stops. See [error recovery](../../features/parsing/error_recovery.md) for how errors are repaired.
Either way, [`sax_parse`](../basic_json/sax_parse.md) returns `#!cpp false`.
Whether parsing should proceed (**must return `#!cpp false`**).
## Examples
??? example "Example: (1) the SAX interface"
??? example
The example below shows how the SAX interface is used.
@@ -46,30 +39,13 @@ Either way, [`sax_parse`](../basic_json/sax_parse.md) returns `#!cpp false`.
--8<-- "examples/sax_parse.output"
```
??? example "Example: (2) recovering from errors"
The example below shows how a SAX parser recovers from errors.
```cpp
--8<-- "examples/sax_parse__error_recovery.cpp"
```
Output:
```
--8<-- "examples/sax_parse__error_recovery.output"
```
## See also
- [`sax_parse`](../basic_json/sax_parse.md) - SAX parser
- [Parsing and Exceptions](../../features/parsing/parse_exceptions.md) - the article on handling parse errors without
exceptions
- [Error Recovery](../../features/parsing/error_recovery.md) - the article on recovering from parse errors
- [SAX Interface](../../features/parsing/sax_interface.md) - guide to event-based parsing
## Version history
- Added in version 3.2.0.
- Returning `#!cpp true` recovers from the error since version 3.13.0; before, parsing stopped, but the result of
[`sax_parse`](../basic_json/sax_parse.md) could be wrong.
@@ -13,6 +13,80 @@ describe each of them.
<p class="nj-caption" markdown>The checks grouped by the stage at which they run, from push or pull request through the merge into develop to a release. The badge on each card says whether the check runs on every push or PR, on a schedule, or continuously.</p>
## How CI runs
Not every job runs for every change. The compiler jobs on Ubuntu, macOS, and Windows are split into three sets, so a
pull request gets fast feedback and the long jobs run when they matter:
| Set | When it runs | What it contains |
|-------------|-------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| PR set | every push to a pull request, and every push to `develop`, `master`, and `release/*` | GCC and Clang with all warnings, Clang-Tidy, sanitizers, IWYU, cpplint, the oldest and newest GCC and Clang, common CMake options, Meson, Emscripten, four Visual Studio 2022 and two Visual Studio 2026 configurations, MinGW, Clang on Windows, ARM64 Windows, five Xcode versions, Swift Package Manager; with the checks that always run, they would have caught at least 92 % of the failing commits of September and October 2026 |
| full set | after the PR set passed: on pull requests with the label :rocket: ready to merge, and on every push | every other compiler, version, C++ standard, and CMake option, the Clang Static Analyzer, Cppcheck, and the C++20 module tests |
| nightly set | every night on `develop`, on every push to `master` and `release/*`, and when started by hand | Infer, Valgrind, coverage, the Intel and NVIDIA compilers, CUDA, ARM64 Linux, and 64-bit MinGW; a failure opens an issue |
The documentation and license checks and the amalgamation check run on every push and pull request; AppVeyor (Visual
Studio 2015, 2017, and 2019) has its own schedule. A pull request that changes only the documentation, Markdown files, GitHub metadata, or license files
skips the compiler workflows (see [`tools/check_path_filters`](https://github.com/nlohmann/json/tree/develop/tools/check_path_filters)).
A maintainer adds the label :rocket: ready to merge when a pull request is ready; the full set then runs, and again on
every later push to that pull request. A pull request is merged only when the full set passed.
??? abstract "Jobs of the compiler workflows and their sets"
The table is generated by [`tools/ci_jobs`](https://github.com/nlohmann/json/tree/develop/tools/ci_jobs), which
also checks in CI that it is up to date and that every job belongs to exactly one set.
<!-- ci_jobs: begin -->
| Set | Workflow | Job | Configurations |
|-----|----------|-----|----------------|
| PR set | macos.yml | `macos-15` | 16.0; 16.4; 26.3 |
| PR set | macos.yml | `macos-26` | 26.6 |
| PR set | macos.yml | `xcode_standards` | 14 |
| PR set | macos.yml | `swiftpm` | - |
| PR set | ubuntu.yml | `ci_test_gcc` | - |
| PR set | ubuntu.yml | `ci_meson_install` | - |
| PR set | ubuntu.yml | `ci_static_analysis_ubuntu` | ci_cpplint |
| PR set | ubuntu.yml | `ci_static_analysis_clang` | ci_test_clang; ci_clang_tidy; ci_test_clang_sanitizer; ci_single_binaries |
| PR set | ubuntu.yml | `ci_cmake_options` | ci_test_diagnostics; ci_test_diagnostic_positions; ci_test_noexceptions; ci_test_noglobaludls |
| PR set | ubuntu.yml | `ci_test_compilers_gcc_old` | 4.8 |
| PR set | ubuntu.yml | `ci_test_compilers_clang` | 3.4 |
| PR set | ubuntu.yml | `ci_test_standards_gcc` | 26 |
| PR set | ubuntu.yml | `ci_test_standards_clang` | 26, libcxx |
| PR set | ubuntu.yml | `ci_emscripten` | - |
| PR set | windows.yml | `mingw` | x86 |
| PR set | windows.yml | `msvc` | Debug, x64, default; Release, x64, default; Debug, Win32, default; Release, Win32, latest |
| PR set | windows.yml | `msvc-vs2026` | Debug, x64, latest; Release, Win32, default |
| PR set | windows.yml | `msvc-arm64` | Debug |
| PR set | windows.yml | `clang` | 20.1.8 |
| PR set | windows.yml | `ci_module_cpp20` | - |
| full set | macos.yml | `macos-15_full` | 16.1; 16.2; 16.3; 26.0.1; 26.1.1; 26.2 |
| full set | macos.yml | `macos-26_full` | 26.4.1; 26.5 |
| full set | macos.yml | `xcode_standards_full` | 11; 17; 20; 23; 26 |
| full set | ubuntu.yml | `ci_static_analysis_ubuntu_full` | ci_test_single_header; ci_cppcheck; ci_reproducible_tests; ci_non_git_tests; ci_offline_testdata |
| full set | ubuntu.yml | `ci_static_analysis_clang_full` | ci_clang_analyze |
| full set | ubuntu.yml | `ci_cmake_options_full` | ci_cmake_flags; ci_test_noimplicitconversions; ci_test_legacycomparison; ci_test_disableenumserialization; ci_test_disabletuplereferenceconversion; ci_test_skiplibraryversioncheck; ci_test_simdutf; ci_test_strict_nul_handling; ci_test_delete_deprecated_functions; ci_test_no_thread_local |
| full set | ubuntu.yml | `ci_test_compilers_gcc_old_full` | 4.9; 5; 6 |
| full set | ubuntu.yml | `ci_test_compilers_gcc_full` | 7; 8; 9; 10; 11; 12; 13; 14; 15; 16; latest |
| full set | ubuntu.yml | `ci_test_compilers_clang_full` | 3.5; 3.6; 3.7; 3.8; 3.9; 4; 5; 6; 7; 8; 9; 10; 11; 12; 13; 14; 15-bullseye; 16; 17; 18; 19; 20; 21; 22; latest |
| full set | ubuntu.yml | `ci_test_standards_gcc_full` | 11; 14; 17; 20; 23 |
| full set | ubuntu.yml | `ci_test_standards_clang_full` | 11, libcxx; 11, libstdcxx; 14, libcxx; 14, libstdcxx; 17, libcxx; 17, libstdcxx; 20, libcxx; 20, libstdcxx; 23, libcxx; 23, libstdcxx; 26, libstdcxx |
| full set | ubuntu.yml | `ci_module_cpp20` | gcc:latest; silkeh/clang:latest |
| full set | windows.yml | `msvc_full` | Debug, Win32, latest; Debug, x64, latest; Release, Win32, default; Release, x64, latest |
| full set | windows.yml | `msvc-vs2026_full` | Debug, Win32, default; Debug, Win32, latest; Debug, x64, default; Release, Win32, latest; Release, x64, default; Release, x64, latest |
| full set | windows.yml | `msvc-arm64_full` | Release |
| full set | windows.yml | `clang_full` | 11.0.1; 12.0.1; 13.0.1; 14.0.6; 15.0.7; 16.0.6; 18.1.8; 19.1.7 |
| full set | windows.yml | `clang-cl_full` | Win32; x64 |
| nightly set | ubuntu.yml | `ci_infer` | - |
| nightly set | ubuntu.yml | `ci_static_analysis_ubuntu_nightly` | ci_test_valgrind |
| nightly set | ubuntu.yml | `ci_test_coverage` | - |
| nightly set | ubuntu.yml | `ci_cuda_example` | 11.8.0; 12.1.1; 12.6.3 |
| nightly set | ubuntu.yml | `ci_icpc` | - |
| nightly set | ubuntu.yml | `ci_icpx` | - |
| nightly set | ubuntu.yml | `ci_nvhpc` | - |
| nightly set | ubuntu.yml | `ci_test_arm64` | - |
| nightly set | windows.yml | `mingw_nightly` | x64 |
<!-- ci_jobs: end -->
## C++ language compliance and compiler compatibility
!!! success "Requirement: Compiler support"
@@ -1,43 +0,0 @@
#include <iostream>
#include <iomanip>
#include <nlohmann/json.hpp>
using json = nlohmann::json;
// a SAX parser that creates a JSON value like json::parse does, but that
// recovers from parse errors instead of stopping at the first one
class recovering_parser : public nlohmann::detail::json_sax_dom_parser<json>
{
public:
explicit recovering_parser(json& result)
: nlohmann::detail::json_sax_dom_parser<json>(result, false)
{}
bool parse_error(std::size_t position,
const std::string& /*last_token*/,
const json::exception& ex)
{
std::cout << "byte " << position << ": " << ex.what() << '\n';
// repair the input and continue
return true;
}
};
int main()
{
// JSON text with several mistakes that ends too early
const std::string text = R"({
"name": "Hello World",
"tags": ["a" "b",],
"valid": tru,
"size": 1.,
"nested": {"x": 1)";
json result;
recovering_parser sax(result);
const bool valid = json::sax_parse(text, &sax);
std::cout << "\nvalid JSON: " << std::boolalpha << valid << '\n'
<< std::setw(4) << result << std::endl;
}
@@ -1,19 +0,0 @@
byte 49: [json.exception.parse_error.101] parse error at line 3, column 20: syntax error while parsing array - unexpected string literal; expected ']'
byte 51: [json.exception.parse_error.101] parse error at line 3, column 22: syntax error while parsing value - unexpected ']'; expected '[', '{', or a literal
byte 70: [json.exception.parse_error.101] parse error at line 4, column 17: syntax error while parsing value - invalid literal; last read: '"valid": tru,'
byte 86: [json.exception.parse_error.101] parse error at line 5, column 15: syntax error while parsing value - invalid number; expected digit after '.'; last read: '1.,'
byte 109: [json.exception.parse_error.101] parse error at line 6, column 22: syntax error while parsing object - unexpected end of input; expected '}'
valid JSON: false
{
"name": "Hello World",
"nested": {
"x": 1
},
"size": 1,
"tags": [
"a",
"b"
],
"valid": null
}
+15 -47
View File
@@ -134,21 +134,9 @@ document: `#!cpp auto v = json_document::parse(text).root();` does not compile.
- **Macros.** [`JSON_STRICT_NUL_HANDLING`](../api/macros/json_strict_nul_handling.md) and
[`JSON_NOEXCEPTION`](../api/macros/json_noexception.md)/[`JSON_THROW_USER`](../api/macros/json_throw_user.md)
behave the same way they do for `<nlohmann/json.hpp>`. These are the macros that change what the parser accepts or
how it reports errors. Two more macros add context to the exceptions of views, as they do for those of
`#!cpp basic_json`:
- With [`JSON_DIAGNOSTICS`](../api/macros/json_diagnostics.md), an exception names the JSON Pointer path of the
value within the document.
- With [`JSON_DIAGNOSTIC_POSITIONS`](../api/macros/json_diagnostic_positions.md), an exception names the bytes of
the value in the source text.
A view has neither parent pointers nor stored positions, so both are found from the document's index when the
exception is thrown. This takes time linear in the size of the document. One exception has neither:
`key()` of an iterator over an array or a primitive value
([`invalid_iterator.207`](../home/exceptions.md#jsonexceptioninvalid_iterator207)), because a view's iterator does
not know the value it iterates over.
The other macros only affect the values `materialize()` produces, through `BasicJsonType`.
how it reports errors. The others either do not apply ([`JSON_DIAGNOSTICS`](../api/macros/json_diagnostics.md)
needs parent pointers in `basic_json` values) or only affect the values `materialize()` produces, through
`BasicJsonType`.
## What is different
@@ -179,6 +167,10 @@ document: `#!cpp auto v = json_document::parse(text).root();` does not compile.
[`at`](../api/basic_json_view/at.md) throws for every missing value.
- **Duplicate keys: lookups find the first member, `parse()` keeps the last.** See [Duplicate keys](#duplicate-keys)
below.
- **No [`JSON_DIAGNOSTICS`](../api/macros/json_diagnostics.md) path.** Exceptions thrown by `basic_json_view`'s own
element access and lookup functions never carry the JSON Pointer path `JSON_DIAGNOSTICS` would otherwise add: the
view has no `basic_json` value to point at, so the exception is created without one, regardless of how
`BasicJsonType` was built.
- **Ordering comparisons are not provided** by `basic_json_view` — there is no `#!cpp operator<`.
[`operator==`](../api/basic_json_view/operator_eq.md) and [`operator!=`](../api/basic_json_view/operator_ne.md) are
provided, though: two views, or a view and a `BasicJsonType` value, compare equal exactly when
@@ -261,34 +253,6 @@ Two conversions never copy at all:
Both results are only valid as long as the view — and, for a string with no escapes, the borrowed source text — is.
### `string_view` members
A struct whose member is a `#!cpp std::string_view` (C++17) can be decoded from a view with the
`NLOHMANN_DEFINE_TYPE_*` macros, and the member is not copied either: for a string without escape sequences it points
into the document's source text, and for a string with escape sequences into the document's buffer of decoded strings.
The member is valid as long as the document is alive and, for a document that
[borrows its source text](#ownership-and-lifetime) (as [`parse()`](../api/basic_json_document/parse.md) does for
an lvalue `#!cpp std::string`, a `#!cpp std::string_view`, or a C string), as long as that text is.
```cpp
struct record
{
std::string_view name;
int count;
NLOHMANN_DEFINE_TYPE_INTRUSIVE(record, name, count)
};
const std::string text = R"({"name": "widget", "count": 3})";
const json_document doc = json_document::parse(text); // borrows text
const record r = doc.root().get<record>(); // r.name points into text, valid while text and doc are
```
!!! warning "Decoding into a `string_view` from a `json` value dangles at once"
The same struct also compiles with `#!cpp json::parse(text).get<record>()`, but there `name` points into the
temporary `json` value, which is destroyed at the end of the statement. Use a named
[`json_document`](../api/json_document.md), as above, to decode such a struct.
## Writing generic code
A view's type names its `BasicJsonType` and its [`view_mode`](../api/view_mode.md):
@@ -365,10 +329,14 @@ Its body may only use what views and `basic_json` have in common: the type check
a view does not have; such a type is converted through `materialize()`. A `from_json()` for `json` only is
converted through `materialize()` as well.
One thing differs when a `from_json()` reads the view: members looked up by name (`at()`, `value()`, ... and thus
the macros) find the **first** member with a repeated key, see [Duplicate keys](#duplicate-keys);
`#!cpp materialize().get<T>()` would see the last one. Containers built from an object, such as `#!cpp std::map`,
keep the **last** value either way, as [`parse()`](../api/basic_json/parse.md) does.
Two things differ when a `from_json()` reads the view:
- **Duplicate keys.** Members looked up by name (`at()`, `value()`, ... and thus the macros) find the **first** member
with a repeated key, see [Duplicate keys](#duplicate-keys); `#!cpp materialize().get<T>()` would see the last one.
Containers built from an object, such as `#!cpp std::map`, keep the **last** value either way, as
[`parse()`](../api/basic_json/parse.md) does.
- **[`JSON_DIAGNOSTICS`](../api/macros/json_diagnostics.md).** Exceptions thrown by the view carry no path to the
value.
`to_json()` is unaffected: it writes `basic_json` values, never views.
@@ -1,122 +0,0 @@
# Error Recovery
By default, parsing stops at the first error. With the [SAX interface](sax_interface.md), you can instead ask the
parser to *recover*: to repair the error and continue, so that you get as much as possible out of malformed input, for
instance a file that was cut off, JSON edited by hand, or the output of a language model.
## Recovering from errors
The SAX parser's [`parse_error`](../../api/json_sax/parse_error.md) function is called for every error. Its return value
decides what happens next:
- `#!cpp false` stops parsing. This is what the SAX parsers of the library do, so [`parse`](../../api/basic_json/parse.md)
and [`accept`](../../api/basic_json/accept.md) never recover.
- `#!cpp true` repairs the error and continues parsing.
When recovering, the SAX parser still receives well-formed events: every `start_object` or `start_array` is followed by
the matching `end_object` or `end_array`, and every `key` is followed by exactly one value. A SAX parser that creates a
JSON value, such as the one in the example below, therefore gets a complete value. Parsing always ends, and
[`sax_parse`](../../api/basic_json/sax_parse.md) returns `#!cpp false` for input that is not valid JSON, even if every
error was repaired. Each token is reported at most once, and the SAX parser can stop at any error by returning
`#!cpp false`.
!!! example
The example below derives a SAX parser from the library's parser for `json` values (`json_sax_dom_parser`),
and recovers from all errors.
```cpp
--8<-- "examples/sax_parse__error_recovery.cpp"
```
Output:
```
--8<-- "examples/sax_parse__error_recovery.output"
```
## How errors are repaired
Each error is repaired with the smallest local edit: a missing separator is inserted, a stray token is removed, what can
be read of a broken string or number is kept, and a value that cannot be read at all becomes `#!json null`.
| Mistake | Repair | Example | Result |
|---------------------------|--------------------------------------------------------------------------------|------------------------------------------|----------------------------|
| missing `,` or `:` | inserted | `#!json [1 2]`, `#!json {"a" 1}` | `[1,2]`, `{"a":1}` |
| missing value | `#!json null` for an object key or between commas in an array | `#!json {"a":}`, `#!json [1,,2]` | `{"a":null}`, `[1,null,2]` |
| trailing comma | removed | `#!json [1,2,]` | `[1,2]` |
| broken string | invalid escapes and bytes are replaced (see below); a line break ends the string | `#!json ["a\qb"]` | `["aqb"]` |
| broken number | the longest valid beginning is kept | `#!json [1., 2e+]` | `[1,2]` |
| unreadable value | `#!json null` | `#!json [1, NaN, tru]` | `[1,null,null]` |
| number too large | passed as infinity, together with its text | `#!json [1e999]` | infinity (see below) |
| stray `:` | removed | `#!json ["a":1]` | `["a",1]` |
| member without a key | skipped up to the next `,` or `}` | `#!json {1:2, "b":3}` | `{"b":3}` |
| wrong closing bracket | closes the innermost array or object | `#!json {"a":[1,2}, "b":3}` | `{"a":[1,2],"b":3}` |
| input ends too early | all open arrays and objects are closed | `#!json {"a":[1,2` | `{"a":[1,2]}` |
| text before the value | skipped | `#!json )]}'{"a":1}` | `{"a":1}` |
In a string, an unknown escape like `\q` stands for the escaped character (`q`), as in JavaScript. An invalid `\u`
escape, a lone surrogate, and ill-formed UTF-8 are each replaced by U+FFFD (REPLACEMENT CHARACTER), and control
characters are kept. A string without its closing quote ends at the next line break or at the end of the input.
The input after the top-level value is not repaired: as without recovery, it is reported as an error, and parsing stops.
## Binary formats
The binary formats ([BJData](../binary_formats/bjdata.md), [BON8](../binary_formats/bon8.md),
[BSON](../binary_formats/bson.md), [CBOR](../binary_formats/cbor.md), [MessagePack](../binary_formats/messagepack.md),
and [UBJSON](../binary_formats/ubjson.md)) have no delimiters to find the next value by. So what can be repaired depends
on whether the end of the item with the error is known, a distinction that
[RFC 8949, Section 5.3](https://www.rfc-editor.org/rfc/rfc8949.html#section-5.3) makes for CBOR, too.
If the item is complete, but cannot be passed on as it is, it is replaced, and parsing continues after it:
| Mistake | Formats | Repair |
|---------------------------------------------------------------------|-------------------------|-------------------------------------------------------------------------|
| tag, if `tag_handler` is `cbor_tag_handler_t::error` (the default) | CBOR | ignored |
| simple value other than `false`, `true`, and `null`, like undefined | CBOR | `#!json null` |
| number too large for a custom `number_float_t`, like `float` | all | infinity |
| character (`C`) that is not ASCII | BJData, UBJSON | U+FFFD |
| invalid high-precision number (`H`) | BJData, UBJSON | the longest valid beginning is kept, as for JSON text, or `#!json null` |
| high-precision number too large | BJData, UBJSON | passed as infinity, together with its text |
| object key that is not a string | BON8, CBOR, MessagePack | the member is skipped |
| element of a type the library does not read, like ObjectId or date | BSON | `#!json null` |
| string without its terminator | BSON | kept |
| document whose size does not match its content | BSON | kept |
CBOR tags and simple values are repaired as [RFC 8949, Section 6.1](https://www.rfc-editor.org/rfc/rfc8949.html#section-6.1)
suggests for converting CBOR to JSON. By default, [`sax_parse`](../../api/basic_json/sax_parse.md) reports every tag as
an error; when recovering, tags are then ignored like with
[`cbor_tag_handler_t::ignore`](../../api/basic_json/cbor_tag_handler_t.md). Strings that are not valid UTF-8 are no
error: like [`from_cbor`](../../api/basic_json/from_cbor.md) and the other functions by default, `sax_parse` passes
them on as they are.
After any other error, the end of the item is unknown: the input ended, a byte is not a valid type marker, or a size
cannot be right. Parsing then stops, and the value read so far is completed: a key that waits for its value gets
`#!json null`, and all open arrays and objects are closed. This keeps everything before the error of an input that was
cut off. The exception is BSON, which stores the size of every document: an element whose end is unknown gets
`#!json null`, the rest of its document is skipped, and parsing continues after the document.
## Limitations
- A repair is a guess. For example, `#!json {"a" "b": 1}` could be meant as `#!json {"a": "b"}` or as
`#!json {"a": null, "b": 1}`; it is repaired to the former. Treat recovered values as a best effort, and check the
reported errors.
- A closing bracket always closes the innermost array or object. If a bracket is missing rather than wrong, the
repair differs from the intention: `#!json {"a": {"b": [1, 2}, "c": 3}` is repaired to
`#!json {"a": {"b": [1, 2], "c": 3}}`, although `#!json {"a": {"b": [1, 2]}, "c": 3}` may have been meant.
- Keys without quotes, and strings in single quotes, are not supported; such members are skipped.
- In the binary formats, a member that is skipped because its key is not a string is lost, and so are the elements of a
BSON document after one whose end is unknown.
- A number that is too large for `number_float_t` is passed as positive or negative infinity. The SAX parser's
`number_float` also gets the number's text, but a JSON value cannot store it, and
[`dump`](../../api/basic_json/dump.md) serializes infinity as `#!json null`.
- When parsing is not strict (see [`sax_parse`](../../api/basic_json/sax_parse.md)), a repair may read parts of the
input after the value, for instance of the next value in a stream of concatenated values.
## See also
- [SAX interface](sax_interface.md) - implement a custom SAX handler
- [`parse_error`](../../api/json_sax/parse_error.md) - the SAX event for parse errors
- [`sax_parse`](../../api/basic_json/sax_parse.md) - generate SAX events
- [parsing and exceptions](parse_exceptions.md) - control error handling
+1 -2
View File
@@ -141,7 +141,7 @@ See [Zero-copy JSON views](../json_view.md) for the concept and the trade-offs.
When the input is not valid JSON, the `parse` function throws an exception by default. If exceptions are undesired or
unavailable, the parser can instead return a discarded value, or [`accept`](../../api/basic_json/accept.md) can be used
to only check whether an input is valid JSON. See [parsing and exceptions](parse_exceptions.md) for the available
options. To get as much as possible out of malformed input, a SAX parser can [recover from errors](error_recovery.md).
options.
## See also
@@ -155,7 +155,6 @@ options. To get as much as possible out of malformed input, a SAX parser can [re
- [SAX interface](sax_interface.md) - implement a custom SAX handler
- [Zero-copy JSON views](../json_view.md) - parse into a read-only document and read it through views
- [parsing and exceptions](parse_exceptions.md) - control error handling
- [error recovery](error_recovery.md) - get as much as possible out of malformed input
- [parsing untrusted input](untrusted_input.md) - what to consider when parsing input from untrusted sources
- [`operator>>`](../../api/operator_gtgt.md) - deserialize from stream
- [`json_sax`](../../api/json_sax/index.md) - documentation of the SAX interface
@@ -68,8 +68,7 @@ bool parse_error(std::size_t position,
const json::exception& ex);
```
The return value decides whether to stop parsing (`#!cpp false`) or to repair the error and continue
(`#!cpp true`); see [error recovery](error_recovery.md) for the latter.
The return value indicates whether the parsing should continue, so the function should usually return `#!cpp false`.
??? example "Example: report parse errors without exceptions"
@@ -62,8 +62,7 @@ bool key(string_t& val);
bool parse_error(std::size_t position, const std::string& last_token, const json::exception& ex);
```
The return value of each function determines whether parsing should proceed. For `parse_error`, returning
`#!cpp true` [recovers from the error](error_recovery.md).
The return value of each function determines whether parsing should proceed.
Object keys are passed to `key` decoded and validated: escape sequences are already resolved, so different spellings of
the same name yield the same string. See [`key`](../../api/json_sax/key.md#notes) for the exact guarantees.
@@ -102,7 +101,7 @@ To implement your own SAX handler, proceed as follows:
2. Create an object of your SAX interface class, e.g. `my_sax`.
3. Call `#!cpp bool json::sax_parse(input, &my_sax);` where the first parameter can be any input like a string or an input stream and the second parameter is a pointer to your SAX interface.
Note the `sax_parse` function only returns a `#!cpp bool` indicating whether the input was parsed without errors and no SAX event returned `#!cpp false`. It does not return `json` value - it is up to you to decide what to do with the SAX events. Furthermore, no exceptions are thrown in case of a parse error - it is up to you what to do with the exception object passed to your `parse_error` implementation. Internally, the SAX interface is used for the DOM parser (class `json_sax_dom_parser`) as well as the acceptor (`json_sax_acceptor`), see file `json_sax.hpp`.
Note the `sax_parse` function only returns a `#!cpp bool` indicating the result of the last executed SAX event. It does not return `json` value - it is up to you to decide what to do with the SAX events. Furthermore, no exceptions are thrown in case of a parse error - it is up to you what to do with the exception object passed to your `parse_error` implementation. Internally, the SAX interface is used for the DOM parser (class `json_sax_dom_parser`) as well as the acceptor (`json_sax_acceptor`), see file `json_sax.hpp`.
## See also
+1 -8
View File
@@ -10,13 +10,6 @@ at its root, a [Natvis](https://learn.microsoft.com/en-us/visualstudio/debugger/
file that gives `json`/`ordered_json` values a friendly, key/value debugger view instead of showing raw internal
fields, when debugging with the MSVC debug engine (`cppvsdbg`) in Visual Studio or VS Code.
The file also has visualizers for `json_document` and `json_view` (and their `ordered_json`/editable variants). They
show the value of the document or view, and its elements as an expandable list. Because a document is only a flat index
into the source text, the strings of the direct elements are shown, but the elements of nested arrays and objects are
shown as index nodes (their kind, size, and numbers; strings only by length). Arrays and objects that have been edited
in a `json_editable_document` are not expanded. The visualizers read the internal layout of the MSVC STL (`std::array`,
`std::unique_ptr`).
The checked-in visualizers cover namespace variants with zero, one, or two ABI tags. Configurations with three or more
ABI tags are not covered and may show raw internal fields instead. To generate a Natvis file for such a configuration:
@@ -36,7 +29,7 @@ ABI tags are not covered and may show raw internal fields instead. To generate a
4. Use the generated file instead of `nlohmann_json.natvis`.
`--abi-tags` can be repeated for several configurations, and `--all` generates every combination (about 60 MB for 11
`--abi-tags` can be repeated for several configurations, and `--all` generates every combination (about 60 MB for 12
ABI tags, doubling with each further tag). See
[`tools/generate_natvis`](https://github.com/nlohmann/json/tree/develop/tools/generate_natvis) for all options.
+174 -160
View File
@@ -1,26 +1,31 @@
<svg class="nj-fig nj-exceptions-qa nj-exceptions-qa--wide" viewBox="0 0 760 808" role="img" aria-labelledby="nj-qa-title-w nj-qa-desc-w" xmlns="http://www.w3.org/2000/svg">
<svg class="nj-fig nj-exceptions-qa nj-exceptions-qa--wide" viewBox="0 0 760 862" role="img" aria-labelledby="nj-qa-title-w nj-qa-desc-w" xmlns="http://www.w3.org/2000/svg">
<title id="nj-qa-title-w">Quality assurance pipeline</title>
<desc id="nj-qa-desc-w">Every push or pull request starts these checks in parallel: Build matrix (every push / PR): 50+ compilers: G&#8203;CC, Clang, M&#8203;SVC, AppleClang, Intel, NVIDIA, ... C++11, 14, 17, 20, 23, and 26 libc++ and libstdc++ warnings: Clang -Weverything, G&#8203;CC 300+ warnings; Unit tests (doctest) (every push / PR): every public A&#8203;PI function every exception id and message all Unicode code points single header and split headers custom number, string, array, and object types; Coverage (every push / PR): unit tests cover all lines of the code base; Static analysis (every push / PR, scheduled): Clang-Tidy, Cppcheck, Infer Clang Static Analyzer (52 checkers), I&#8203;WYU, Codacy CodeQL, Semgrep, Flawfinder (also scheduled); Dynamic analysis (every push / PR): runtime assertions Valgrind (Memcheck) address and undefined behavior sanitizers, integer overflow, nullability violations; Fuzzing (every PR, continuous): CIFuzz on every pull request OSS-Fuzz: billions of inputs every crash fixed with a reproducing unit test; Style (every push / PR): Artistic Style (astyle), enforced in the C&#8203;I cpplint with 61 rules; Docs and examples (every push / PR, scheduled): every documentation example is tested; changed output fails site built with warnings as errors; links checked regularly (scheduled); Integration, supply chain (every push / PR, scheduled): CMake 3.5, 3.31.6, and 4.0.0 Meson, Swift Package Manager dependency review on every PR Dependabot daily, OpenSSF Scorecard, R&#8203;EUSE. When they pass, the change is merged into develop; a release is made only when no vulnerability alert is open.</desc>
<desc id="nj-qa-desc-w">A push or pull request starts these checks; the badges say when each runs: Build matrix (every push / PR, before merge): 50+ compilers: G&#8203;CC, Clang, M&#8203;SVC, AppleClang, Intel, NVIDIA, ... C++11, 14, 17, 20, 23, and 26 libc++ and libstdc++ warnings: Clang -Weverything, G&#8203;CC 300+ warnings a subset on every push and PR, all before merge; Intel, NVIDIA, C&#8203;UDA, and ARM64 nightly; Unit tests (doctest) (every push / PR): every public A&#8203;PI function every exception id and message all Unicode code points single header and split headers custom number, string, array, and object types; Coverage (nightly): unit tests cover all lines of the code base; Static analysis (every push / PR, scheduled): Clang-Tidy, Cppcheck, Infer Clang Static Analyzer (52 checkers), I&#8203;WYU, Codacy CodeQL, Semgrep, Flawfinder (also scheduled); Infer nightly; Dynamic analysis (every push / PR, nightly): runtime assertions Valgrind (Memcheck), nightly address and undefined behavior sanitizers, integer overflow, nullability violations; Fuzzing (every PR, continuous): CIFuzz on every pull request OSS-Fuzz: billions of inputs every crash fixed with a reproducing unit test; Style (every push / PR): Artistic Style (astyle), enforced in the C&#8203;I cpplint with 61 rules; Docs and examples (every push / PR, scheduled): every documentation example is tested; changed output fails site built with warnings as errors; links checked regularly (scheduled); Integration, supply chain (every push / PR, scheduled): CMake 3.5, 3.31.6, and 4.0.0 Meson, Swift Package Manager dependency review on every PR Dependabot daily, OpenSSF Scorecard, R&#8203;EUSE. When they pass, the change is merged into develop; a release is made only when no vulnerability alert is open.</desc>
<defs><marker id="nj-qa-arrow-w" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto"><path class="fg-head" d="M0 1 L10 5 L0 9 z"/></marker></defs>
<rect class="nj-exceptions-groupbox" x="12" y="70" width="736" height="594" rx="12"/>
<rect class="nj-exceptions-groupbox" x="12" y="70" width="736" height="648" rx="12"/>
<rect class="nj-exceptions-stage" x="265" y="8" width="230" height="34" rx="17"/>
<text class="nj-exceptions-stagetext" x="380" y="30">push or pull request</text>
<line class="nj-exceptions-edge" x1="380" y1="42" x2="380" y2="68" marker-end="url(#nj-qa-arrow-w)"/>
<text class="nj-exceptions-group" x="26" y="94">checks, run in parallel on GitHub Actions and AppVeyor</text>
<g class="nj-exceptions-card">
<rect class="nj-exceptions-box" x="26" y="110" width="228" height="184" rx="8"/>
<rect class="nj-exceptions-box" x="26" y="110" width="228" height="238" rx="8"/>
<text class="nj-exceptions-ctitle" x="40" y="135">Build matrix</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--push" x="40" y="146" width="110.5" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--push" x="95.25" y="160">every push / PR</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--push" x="156.5" y="146" width="91.6" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--push" x="202.3" y="160">before merge</text>
<text class="nj-exceptions-line" x="40" y="188">50+ compilers: G&#8203;CC, Clang, M&#8203;SVC,</text>
<text class="nj-exceptions-line" x="40" y="206">AppleClang, Intel, NVIDIA, ...</text>
<text class="nj-exceptions-line" x="40" y="224">C++11, 14, 17, 20, 23, and 26</text>
<text class="nj-exceptions-line" x="40" y="242">libc++ and libstdc++</text>
<text class="nj-exceptions-line" x="40" y="260">warnings: Clang -Weverything,</text>
<text class="nj-exceptions-line" x="40" y="278">G&#8203;CC 300+ warnings</text>
<text class="nj-exceptions-line" x="40" y="296">a subset on every push and PR,</text>
<text class="nj-exceptions-line" x="40" y="314">all before merge; Intel, NVIDIA,</text>
<text class="nj-exceptions-line" x="40" y="332">C&#8203;UDA, and ARM64 nightly</text>
</g>
<g class="nj-exceptions-card">
<rect class="nj-exceptions-box" x="266" y="110" width="228" height="184" rx="8"/>
<rect class="nj-exceptions-box" x="266" y="110" width="228" height="238" rx="8"/>
<text class="nj-exceptions-ctitle" x="280" y="135">Unit tests (doctest)</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--push" x="280" y="146" width="110.5" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--push" x="335.25" y="160">every push / PR</text>
@@ -32,209 +37,218 @@
<text class="nj-exceptions-line" x="280" y="278">and object types</text>
</g>
<g class="nj-exceptions-card">
<rect class="nj-exceptions-box" x="506" y="110" width="228" height="184" rx="8"/>
<rect class="nj-exceptions-box" x="506" y="110" width="228" height="238" rx="8"/>
<text class="nj-exceptions-ctitle" x="520" y="135">Coverage</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--push" x="520" y="146" width="110.5" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--push" x="575.25" y="160">every push / PR</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--sched" x="520" y="146" width="60.1" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--sched" x="550.05" y="160">nightly</text>
<text class="nj-exceptions-line" x="520" y="188">unit tests cover all lines</text>
<text class="nj-exceptions-line" x="520" y="206">of the code base</text>
</g>
<g class="nj-exceptions-card">
<rect class="nj-exceptions-box" x="26" y="306" width="228" height="166" rx="8"/>
<text class="nj-exceptions-ctitle" x="40" y="331">Static analysis</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--push" x="40" y="342" width="110.5" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--push" x="95.25" y="356">every push / PR</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--sched" x="156.5" y="342" width="72.7" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--sched" x="192.85" y="356">scheduled</text>
<text class="nj-exceptions-line" x="40" y="384">Clang-Tidy, Cppcheck, Infer</text>
<text class="nj-exceptions-line" x="40" y="402">Clang Static Analyzer (52</text>
<text class="nj-exceptions-line" x="40" y="420">checkers), I&#8203;WYU, Codacy</text>
<text class="nj-exceptions-line" x="40" y="438">CodeQL, Semgrep, Flawfinder</text>
<text class="nj-exceptions-line" x="40" y="456">(also scheduled)</text>
<rect class="nj-exceptions-box" x="26" y="360" width="228" height="166" rx="8"/>
<text class="nj-exceptions-ctitle" x="40" y="385">Static analysis</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--push" x="40" y="396" width="110.5" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--push" x="95.25" y="410">every push / PR</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--sched" x="156.5" y="396" width="72.7" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--sched" x="192.85" y="410">scheduled</text>
<text class="nj-exceptions-line" x="40" y="438">Clang-Tidy, Cppcheck, Infer</text>
<text class="nj-exceptions-line" x="40" y="456">Clang Static Analyzer (52</text>
<text class="nj-exceptions-line" x="40" y="474">checkers), I&#8203;WYU, Codacy</text>
<text class="nj-exceptions-line" x="40" y="492">CodeQL, Semgrep, Flawfinder</text>
<text class="nj-exceptions-line" x="40" y="510">(also scheduled); Infer nightly</text>
</g>
<g class="nj-exceptions-card">
<rect class="nj-exceptions-box" x="266" y="306" width="228" height="166" rx="8"/>
<text class="nj-exceptions-ctitle" x="280" y="331">Dynamic analysis</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--push" x="280" y="342" width="110.5" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--push" x="335.25" y="356">every push / PR</text>
<text class="nj-exceptions-line" x="280" y="384">runtime assertions</text>
<text class="nj-exceptions-line" x="280" y="402">Valgrind (Memcheck)</text>
<text class="nj-exceptions-line" x="280" y="420">address and undefined behavior</text>
<text class="nj-exceptions-line" x="280" y="438">sanitizers, integer overflow,</text>
<text class="nj-exceptions-line" x="280" y="456">nullability violations</text>
<rect class="nj-exceptions-box" x="266" y="360" width="228" height="166" rx="8"/>
<text class="nj-exceptions-ctitle" x="280" y="385">Dynamic analysis</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--push" x="280" y="396" width="110.5" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--push" x="335.25" y="410">every push / PR</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--sched" x="396.5" y="396" width="60.1" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--sched" x="426.55" y="410">nightly</text>
<text class="nj-exceptions-line" x="280" y="438">runtime assertions</text>
<text class="nj-exceptions-line" x="280" y="456">Valgrind (Memcheck), nightly</text>
<text class="nj-exceptions-line" x="280" y="474">address and undefined behavior</text>
<text class="nj-exceptions-line" x="280" y="492">sanitizers, integer overflow,</text>
<text class="nj-exceptions-line" x="280" y="510">nullability violations</text>
</g>
<g class="nj-exceptions-card">
<rect class="nj-exceptions-box" x="506" y="306" width="228" height="166" rx="8"/>
<text class="nj-exceptions-ctitle" x="520" y="331">Fuzzing</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--push" x="520" y="342" width="66.4" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--push" x="553.2" y="356">every PR</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--cont" x="592.4" y="342" width="79" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--cont" x="631.9" y="356">continuous</text>
<text class="nj-exceptions-line" x="520" y="384">CIFuzz on every pull request</text>
<text class="nj-exceptions-line" x="520" y="402">OSS-Fuzz: billions of inputs</text>
<text class="nj-exceptions-line" x="520" y="420">every crash fixed with a</text>
<text class="nj-exceptions-line" x="520" y="438">reproducing unit test</text>
<rect class="nj-exceptions-box" x="506" y="360" width="228" height="166" rx="8"/>
<text class="nj-exceptions-ctitle" x="520" y="385">Fuzzing</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--push" x="520" y="396" width="66.4" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--push" x="553.2" y="410">every PR</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--cont" x="592.4" y="396" width="79" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--cont" x="631.9" y="410">continuous</text>
<text class="nj-exceptions-line" x="520" y="438">CIFuzz on every pull request</text>
<text class="nj-exceptions-line" x="520" y="456">OSS-Fuzz: billions of inputs</text>
<text class="nj-exceptions-line" x="520" y="474">every crash fixed with a</text>
<text class="nj-exceptions-line" x="520" y="492">reproducing unit test</text>
</g>
<g class="nj-exceptions-card">
<rect class="nj-exceptions-box" x="26" y="484" width="228" height="166" rx="8"/>
<text class="nj-exceptions-ctitle" x="40" y="509">Style</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--push" x="40" y="520" width="110.5" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--push" x="95.25" y="534">every push / PR</text>
<text class="nj-exceptions-line" x="40" y="562">Artistic Style (astyle),</text>
<text class="nj-exceptions-line" x="40" y="580">enforced in the C&#8203;I</text>
<text class="nj-exceptions-line" x="40" y="598">cpplint with 61 rules</text>
<rect class="nj-exceptions-box" x="26" y="538" width="228" height="166" rx="8"/>
<text class="nj-exceptions-ctitle" x="40" y="563">Style</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--push" x="40" y="574" width="110.5" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--push" x="95.25" y="588">every push / PR</text>
<text class="nj-exceptions-line" x="40" y="616">Artistic Style (astyle),</text>
<text class="nj-exceptions-line" x="40" y="634">enforced in the C&#8203;I</text>
<text class="nj-exceptions-line" x="40" y="652">cpplint with 61 rules</text>
</g>
<g class="nj-exceptions-card">
<rect class="nj-exceptions-box" x="266" y="484" width="228" height="166" rx="8"/>
<text class="nj-exceptions-ctitle" x="280" y="509">Docs and examples</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--push" x="280" y="520" width="110.5" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--push" x="335.25" y="534">every push / PR</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--sched" x="396.5" y="520" width="72.7" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--sched" x="432.85" y="534">scheduled</text>
<text class="nj-exceptions-line" x="280" y="562">every documentation example is</text>
<text class="nj-exceptions-line" x="280" y="580">tested; changed output fails</text>
<text class="nj-exceptions-line" x="280" y="598">site built with warnings</text>
<text class="nj-exceptions-line" x="280" y="616">as errors; links checked</text>
<text class="nj-exceptions-line" x="280" y="634">regularly (scheduled)</text>
<rect class="nj-exceptions-box" x="266" y="538" width="228" height="166" rx="8"/>
<text class="nj-exceptions-ctitle" x="280" y="563">Docs and examples</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--push" x="280" y="574" width="110.5" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--push" x="335.25" y="588">every push / PR</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--sched" x="396.5" y="574" width="72.7" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--sched" x="432.85" y="588">scheduled</text>
<text class="nj-exceptions-line" x="280" y="616">every documentation example is</text>
<text class="nj-exceptions-line" x="280" y="634">tested; changed output fails</text>
<text class="nj-exceptions-line" x="280" y="652">site built with warnings</text>
<text class="nj-exceptions-line" x="280" y="670">as errors; links checked</text>
<text class="nj-exceptions-line" x="280" y="688">regularly (scheduled)</text>
</g>
<g class="nj-exceptions-card">
<rect class="nj-exceptions-box" x="506" y="484" width="228" height="166" rx="8"/>
<text class="nj-exceptions-ctitle" x="520" y="509">Integration, supply chain</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--push" x="520" y="520" width="110.5" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--push" x="575.25" y="534">every push / PR</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--sched" x="636.5" y="520" width="72.7" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--sched" x="672.85" y="534">scheduled</text>
<text class="nj-exceptions-line" x="520" y="562">CMake 3.5, 3.31.6, and 4.0.0</text>
<text class="nj-exceptions-line" x="520" y="580">Meson, Swift Package Manager</text>
<text class="nj-exceptions-line" x="520" y="598">dependency review on every PR</text>
<text class="nj-exceptions-line" x="520" y="616">Dependabot daily, OpenSSF</text>
<text class="nj-exceptions-line" x="520" y="634">Scorecard, R&#8203;EUSE</text>
<rect class="nj-exceptions-box" x="506" y="538" width="228" height="166" rx="8"/>
<text class="nj-exceptions-ctitle" x="520" y="563">Integration, supply chain</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--push" x="520" y="574" width="110.5" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--push" x="575.25" y="588">every push / PR</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--sched" x="636.5" y="574" width="72.7" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--sched" x="672.85" y="588">scheduled</text>
<text class="nj-exceptions-line" x="520" y="616">CMake 3.5, 3.31.6, and 4.0.0</text>
<text class="nj-exceptions-line" x="520" y="634">Meson, Swift Package Manager</text>
<text class="nj-exceptions-line" x="520" y="652">dependency review on every PR</text>
<text class="nj-exceptions-line" x="520" y="670">Dependabot daily, OpenSSF</text>
<text class="nj-exceptions-line" x="520" y="688">Scorecard, R&#8203;EUSE</text>
</g>
<line class="nj-exceptions-edge" x1="380" y1="664" x2="380" y2="690" marker-end="url(#nj-qa-arrow-w)"/>
<rect class="nj-exceptions-stage nj-exceptions-stage--merge" x="265" y="692" width="230" height="34" rx="17"/>
<text class="nj-exceptions-stagetext" x="380" y="714">merge into develop</text>
<line class="nj-exceptions-edge" x1="380" y1="726" x2="380" y2="752" marker-end="url(#nj-qa-arrow-w)"/>
<rect class="nj-exceptions-stage nj-exceptions-stage--merge" x="230" y="754" width="300" height="46" rx="23"/>
<text class="nj-exceptions-stagetext" x="380" y="776">release</text>
<text class="nj-exceptions-stagesub" x="380" y="792">no open vulnerability alert</text>
<line class="nj-exceptions-edge" x1="380" y1="718" x2="380" y2="744" marker-end="url(#nj-qa-arrow-w)"/>
<rect class="nj-exceptions-stage nj-exceptions-stage--merge" x="265" y="746" width="230" height="34" rx="17"/>
<text class="nj-exceptions-stagetext" x="380" y="768">merge into develop</text>
<line class="nj-exceptions-edge" x1="380" y1="780" x2="380" y2="806" marker-end="url(#nj-qa-arrow-w)"/>
<rect class="nj-exceptions-stage nj-exceptions-stage--merge" x="230" y="808" width="300" height="46" rx="23"/>
<text class="nj-exceptions-stagetext" x="380" y="830">release</text>
<text class="nj-exceptions-stagesub" x="380" y="846">no open vulnerability alert</text>
</svg>
<svg class="nj-fig nj-exceptions-qa nj-exceptions-qa--narrow" viewBox="0 0 360 1786" role="img" aria-labelledby="nj-qa-title-n nj-qa-desc-n" xmlns="http://www.w3.org/2000/svg">
<svg class="nj-fig nj-exceptions-qa nj-exceptions-qa--narrow" viewBox="0 0 360 1840" role="img" aria-labelledby="nj-qa-title-n nj-qa-desc-n" xmlns="http://www.w3.org/2000/svg">
<title id="nj-qa-title-n">Quality assurance pipeline</title>
<desc id="nj-qa-desc-n">Every push or pull request starts these checks in parallel: Build matrix (every push / PR): 50+ compilers: G&#8203;CC, Clang, M&#8203;SVC, AppleClang, Intel, NVIDIA, ... C++11, 14, 17, 20, 23, and 26 libc++ and libstdc++ warnings: Clang -Weverything, G&#8203;CC 300+ warnings; Unit tests (doctest) (every push / PR): every public A&#8203;PI function every exception id and message all Unicode code points single header and split headers custom number, string, array, and object types; Coverage (every push / PR): unit tests cover all lines of the code base; Static analysis (every push / PR, scheduled): Clang-Tidy, Cppcheck, Infer Clang Static Analyzer (52 checkers), I&#8203;WYU, Codacy CodeQL, Semgrep, Flawfinder (also scheduled); Dynamic analysis (every push / PR): runtime assertions Valgrind (Memcheck) address and undefined behavior sanitizers, integer overflow, nullability violations; Fuzzing (every PR, continuous): CIFuzz on every pull request OSS-Fuzz: billions of inputs every crash fixed with a reproducing unit test; Style (every push / PR): Artistic Style (astyle), enforced in the C&#8203;I cpplint with 61 rules; Docs and examples (every push / PR, scheduled): every documentation example is tested; changed output fails site built with warnings as errors; links checked regularly (scheduled); Integration, supply chain (every push / PR, scheduled): CMake 3.5, 3.31.6, and 4.0.0 Meson, Swift Package Manager dependency review on every PR Dependabot daily, OpenSSF Scorecard, R&#8203;EUSE. When they pass, the change is merged into develop; a release is made only when no vulnerability alert is open.</desc>
<desc id="nj-qa-desc-n">A push or pull request starts these checks; the badges say when each runs: Build matrix (every push / PR, before merge): 50+ compilers: G&#8203;CC, Clang, M&#8203;SVC, AppleClang, Intel, NVIDIA, ... C++11, 14, 17, 20, 23, and 26 libc++ and libstdc++ warnings: Clang -Weverything, G&#8203;CC 300+ warnings a subset on every push and PR, all before merge; Intel, NVIDIA, C&#8203;UDA, and ARM64 nightly; Unit tests (doctest) (every push / PR): every public A&#8203;PI function every exception id and message all Unicode code points single header and split headers custom number, string, array, and object types; Coverage (nightly): unit tests cover all lines of the code base; Static analysis (every push / PR, scheduled): Clang-Tidy, Cppcheck, Infer Clang Static Analyzer (52 checkers), I&#8203;WYU, Codacy CodeQL, Semgrep, Flawfinder (also scheduled); Infer nightly; Dynamic analysis (every push / PR, nightly): runtime assertions Valgrind (Memcheck), nightly address and undefined behavior sanitizers, integer overflow, nullability violations; Fuzzing (every PR, continuous): CIFuzz on every pull request OSS-Fuzz: billions of inputs every crash fixed with a reproducing unit test; Style (every push / PR): Artistic Style (astyle), enforced in the C&#8203;I cpplint with 61 rules; Docs and examples (every push / PR, scheduled): every documentation example is tested; changed output fails site built with warnings as errors; links checked regularly (scheduled); Integration, supply chain (every push / PR, scheduled): CMake 3.5, 3.31.6, and 4.0.0 Meson, Swift Package Manager dependency review on every PR Dependabot daily, OpenSSF Scorecard, R&#8203;EUSE. When they pass, the change is merged into develop; a release is made only when no vulnerability alert is open.</desc>
<defs><marker id="nj-qa-arrow-n" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto"><path class="fg-head" d="M0 1 L10 5 L0 9 z"/></marker></defs>
<rect class="nj-exceptions-groupbox" x="12" y="70" width="336" height="1572" rx="12"/>
<rect class="nj-exceptions-groupbox" x="12" y="70" width="336" height="1626" rx="12"/>
<rect class="nj-exceptions-stage" x="65" y="8" width="230" height="34" rx="17"/>
<text class="nj-exceptions-stagetext" x="180" y="30">push or pull request</text>
<line class="nj-exceptions-edge" x1="180" y1="42" x2="180" y2="68" marker-end="url(#nj-qa-arrow-n)"/>
<text class="nj-exceptions-group" x="26" y="94">checks, run in parallel</text>
<g class="nj-exceptions-card">
<rect class="nj-exceptions-box" x="26" y="110" width="308" height="184" rx="8"/>
<rect class="nj-exceptions-box" x="26" y="110" width="308" height="238" rx="8"/>
<text class="nj-exceptions-ctitle" x="40" y="135">Build matrix</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--push" x="40" y="146" width="110.5" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--push" x="95.25" y="160">every push / PR</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--push" x="156.5" y="146" width="91.6" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--push" x="202.3" y="160">before merge</text>
<text class="nj-exceptions-line" x="40" y="188">50+ compilers: G&#8203;CC, Clang, M&#8203;SVC,</text>
<text class="nj-exceptions-line" x="40" y="206">AppleClang, Intel, NVIDIA, ...</text>
<text class="nj-exceptions-line" x="40" y="224">C++11, 14, 17, 20, 23, and 26</text>
<text class="nj-exceptions-line" x="40" y="242">libc++ and libstdc++</text>
<text class="nj-exceptions-line" x="40" y="260">warnings: Clang -Weverything,</text>
<text class="nj-exceptions-line" x="40" y="278">G&#8203;CC 300+ warnings</text>
<text class="nj-exceptions-line" x="40" y="296">a subset on every push and PR,</text>
<text class="nj-exceptions-line" x="40" y="314">all before merge; Intel, NVIDIA,</text>
<text class="nj-exceptions-line" x="40" y="332">C&#8203;UDA, and ARM64 nightly</text>
</g>
<g class="nj-exceptions-card">
<rect class="nj-exceptions-box" x="26" y="306" width="308" height="184" rx="8"/>
<text class="nj-exceptions-ctitle" x="40" y="331">Unit tests (doctest)</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--push" x="40" y="342" width="110.5" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--push" x="95.25" y="356">every push / PR</text>
<text class="nj-exceptions-line" x="40" y="384">every public A&#8203;PI function</text>
<text class="nj-exceptions-line" x="40" y="402">every exception id and message</text>
<text class="nj-exceptions-line" x="40" y="420">all Unicode code points</text>
<text class="nj-exceptions-line" x="40" y="438">single header and split headers</text>
<text class="nj-exceptions-line" x="40" y="456">custom number, string, array,</text>
<text class="nj-exceptions-line" x="40" y="474">and object types</text>
<rect class="nj-exceptions-box" x="26" y="360" width="308" height="184" rx="8"/>
<text class="nj-exceptions-ctitle" x="40" y="385">Unit tests (doctest)</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--push" x="40" y="396" width="110.5" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--push" x="95.25" y="410">every push / PR</text>
<text class="nj-exceptions-line" x="40" y="438">every public A&#8203;PI function</text>
<text class="nj-exceptions-line" x="40" y="456">every exception id and message</text>
<text class="nj-exceptions-line" x="40" y="474">all Unicode code points</text>
<text class="nj-exceptions-line" x="40" y="492">single header and split headers</text>
<text class="nj-exceptions-line" x="40" y="510">custom number, string, array,</text>
<text class="nj-exceptions-line" x="40" y="528">and object types</text>
</g>
<g class="nj-exceptions-card">
<rect class="nj-exceptions-box" x="26" y="502" width="308" height="112" rx="8"/>
<text class="nj-exceptions-ctitle" x="40" y="527">Coverage</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--push" x="40" y="538" width="110.5" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--push" x="95.25" y="552">every push / PR</text>
<text class="nj-exceptions-line" x="40" y="580">unit tests cover all lines</text>
<text class="nj-exceptions-line" x="40" y="598">of the code base</text>
<rect class="nj-exceptions-box" x="26" y="556" width="308" height="112" rx="8"/>
<text class="nj-exceptions-ctitle" x="40" y="581">Coverage</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--sched" x="40" y="592" width="60.1" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--sched" x="70.05" y="606">nightly</text>
<text class="nj-exceptions-line" x="40" y="634">unit tests cover all lines</text>
<text class="nj-exceptions-line" x="40" y="652">of the code base</text>
</g>
<g class="nj-exceptions-card">
<rect class="nj-exceptions-box" x="26" y="626" width="308" height="166" rx="8"/>
<text class="nj-exceptions-ctitle" x="40" y="651">Static analysis</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--push" x="40" y="662" width="110.5" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--push" x="95.25" y="676">every push / PR</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--sched" x="156.5" y="662" width="72.7" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--sched" x="192.85" y="676">scheduled</text>
<text class="nj-exceptions-line" x="40" y="704">Clang-Tidy, Cppcheck, Infer</text>
<text class="nj-exceptions-line" x="40" y="722">Clang Static Analyzer (52</text>
<text class="nj-exceptions-line" x="40" y="740">checkers), I&#8203;WYU, Codacy</text>
<text class="nj-exceptions-line" x="40" y="758">CodeQL, Semgrep, Flawfinder</text>
<text class="nj-exceptions-line" x="40" y="776">(also scheduled)</text>
<rect class="nj-exceptions-box" x="26" y="680" width="308" height="166" rx="8"/>
<text class="nj-exceptions-ctitle" x="40" y="705">Static analysis</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--push" x="40" y="716" width="110.5" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--push" x="95.25" y="730">every push / PR</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--sched" x="156.5" y="716" width="72.7" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--sched" x="192.85" y="730">scheduled</text>
<text class="nj-exceptions-line" x="40" y="758">Clang-Tidy, Cppcheck, Infer</text>
<text class="nj-exceptions-line" x="40" y="776">Clang Static Analyzer (52</text>
<text class="nj-exceptions-line" x="40" y="794">checkers), I&#8203;WYU, Codacy</text>
<text class="nj-exceptions-line" x="40" y="812">CodeQL, Semgrep, Flawfinder</text>
<text class="nj-exceptions-line" x="40" y="830">(also scheduled); Infer nightly</text>
</g>
<g class="nj-exceptions-card">
<rect class="nj-exceptions-box" x="26" y="804" width="308" height="166" rx="8"/>
<text class="nj-exceptions-ctitle" x="40" y="829">Dynamic analysis</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--push" x="40" y="840" width="110.5" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--push" x="95.25" y="854">every push / PR</text>
<text class="nj-exceptions-line" x="40" y="882">runtime assertions</text>
<text class="nj-exceptions-line" x="40" y="900">Valgrind (Memcheck)</text>
<text class="nj-exceptions-line" x="40" y="918">address and undefined behavior</text>
<text class="nj-exceptions-line" x="40" y="936">sanitizers, integer overflow,</text>
<text class="nj-exceptions-line" x="40" y="954">nullability violations</text>
<rect class="nj-exceptions-box" x="26" y="858" width="308" height="166" rx="8"/>
<text class="nj-exceptions-ctitle" x="40" y="883">Dynamic analysis</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--push" x="40" y="894" width="110.5" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--push" x="95.25" y="908">every push / PR</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--sched" x="156.5" y="894" width="60.1" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--sched" x="186.55" y="908">nightly</text>
<text class="nj-exceptions-line" x="40" y="936">runtime assertions</text>
<text class="nj-exceptions-line" x="40" y="954">Valgrind (Memcheck), nightly</text>
<text class="nj-exceptions-line" x="40" y="972">address and undefined behavior</text>
<text class="nj-exceptions-line" x="40" y="990">sanitizers, integer overflow,</text>
<text class="nj-exceptions-line" x="40" y="1008">nullability violations</text>
</g>
<g class="nj-exceptions-card">
<rect class="nj-exceptions-box" x="26" y="982" width="308" height="148" rx="8"/>
<text class="nj-exceptions-ctitle" x="40" y="1007">Fuzzing</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--push" x="40" y="1018" width="66.4" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--push" x="73.2" y="1032">every PR</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--cont" x="112.4" y="1018" width="79" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--cont" x="151.9" y="1032">continuous</text>
<text class="nj-exceptions-line" x="40" y="1060">CIFuzz on every pull request</text>
<text class="nj-exceptions-line" x="40" y="1078">OSS-Fuzz: billions of inputs</text>
<text class="nj-exceptions-line" x="40" y="1096">every crash fixed with a</text>
<text class="nj-exceptions-line" x="40" y="1114">reproducing unit test</text>
<rect class="nj-exceptions-box" x="26" y="1036" width="308" height="148" rx="8"/>
<text class="nj-exceptions-ctitle" x="40" y="1061">Fuzzing</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--push" x="40" y="1072" width="66.4" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--push" x="73.2" y="1086">every PR</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--cont" x="112.4" y="1072" width="79" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--cont" x="151.9" y="1086">continuous</text>
<text class="nj-exceptions-line" x="40" y="1114">CIFuzz on every pull request</text>
<text class="nj-exceptions-line" x="40" y="1132">OSS-Fuzz: billions of inputs</text>
<text class="nj-exceptions-line" x="40" y="1150">every crash fixed with a</text>
<text class="nj-exceptions-line" x="40" y="1168">reproducing unit test</text>
</g>
<g class="nj-exceptions-card">
<rect class="nj-exceptions-box" x="26" y="1142" width="308" height="130" rx="8"/>
<text class="nj-exceptions-ctitle" x="40" y="1167">Style</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--push" x="40" y="1178" width="110.5" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--push" x="95.25" y="1192">every push / PR</text>
<text class="nj-exceptions-line" x="40" y="1220">Artistic Style (astyle),</text>
<text class="nj-exceptions-line" x="40" y="1238">enforced in the C&#8203;I</text>
<text class="nj-exceptions-line" x="40" y="1256">cpplint with 61 rules</text>
<rect class="nj-exceptions-box" x="26" y="1196" width="308" height="130" rx="8"/>
<text class="nj-exceptions-ctitle" x="40" y="1221">Style</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--push" x="40" y="1232" width="110.5" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--push" x="95.25" y="1246">every push / PR</text>
<text class="nj-exceptions-line" x="40" y="1274">Artistic Style (astyle),</text>
<text class="nj-exceptions-line" x="40" y="1292">enforced in the C&#8203;I</text>
<text class="nj-exceptions-line" x="40" y="1310">cpplint with 61 rules</text>
</g>
<g class="nj-exceptions-card">
<rect class="nj-exceptions-box" x="26" y="1284" width="308" height="166" rx="8"/>
<text class="nj-exceptions-ctitle" x="40" y="1309">Docs and examples</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--push" x="40" y="1320" width="110.5" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--push" x="95.25" y="1334">every push / PR</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--sched" x="156.5" y="1320" width="72.7" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--sched" x="192.85" y="1334">scheduled</text>
<text class="nj-exceptions-line" x="40" y="1362">every documentation example is</text>
<text class="nj-exceptions-line" x="40" y="1380">tested; changed output fails</text>
<text class="nj-exceptions-line" x="40" y="1398">site built with warnings</text>
<text class="nj-exceptions-line" x="40" y="1416">as errors; links checked</text>
<text class="nj-exceptions-line" x="40" y="1434">regularly (scheduled)</text>
<rect class="nj-exceptions-box" x="26" y="1338" width="308" height="166" rx="8"/>
<text class="nj-exceptions-ctitle" x="40" y="1363">Docs and examples</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--push" x="40" y="1374" width="110.5" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--push" x="95.25" y="1388">every push / PR</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--sched" x="156.5" y="1374" width="72.7" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--sched" x="192.85" y="1388">scheduled</text>
<text class="nj-exceptions-line" x="40" y="1416">every documentation example is</text>
<text class="nj-exceptions-line" x="40" y="1434">tested; changed output fails</text>
<text class="nj-exceptions-line" x="40" y="1452">site built with warnings</text>
<text class="nj-exceptions-line" x="40" y="1470">as errors; links checked</text>
<text class="nj-exceptions-line" x="40" y="1488">regularly (scheduled)</text>
</g>
<g class="nj-exceptions-card">
<rect class="nj-exceptions-box" x="26" y="1462" width="308" height="166" rx="8"/>
<text class="nj-exceptions-ctitle" x="40" y="1487">Integration, supply chain</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--push" x="40" y="1498" width="110.5" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--push" x="95.25" y="1512">every push / PR</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--sched" x="156.5" y="1498" width="72.7" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--sched" x="192.85" y="1512">scheduled</text>
<text class="nj-exceptions-line" x="40" y="1540">CMake 3.5, 3.31.6, and 4.0.0</text>
<text class="nj-exceptions-line" x="40" y="1558">Meson, Swift Package Manager</text>
<text class="nj-exceptions-line" x="40" y="1576">dependency review on every PR</text>
<text class="nj-exceptions-line" x="40" y="1594">Dependabot daily, OpenSSF</text>
<text class="nj-exceptions-line" x="40" y="1612">Scorecard, R&#8203;EUSE</text>
<rect class="nj-exceptions-box" x="26" y="1516" width="308" height="166" rx="8"/>
<text class="nj-exceptions-ctitle" x="40" y="1541">Integration, supply chain</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--push" x="40" y="1552" width="110.5" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--push" x="95.25" y="1566">every push / PR</text>
<rect class="nj-exceptions-badge nj-exceptions-badge--sched" x="156.5" y="1552" width="72.7" height="20" rx="10"/>
<text class="nj-exceptions-badgetext nj-exceptions-badgetext--sched" x="192.85" y="1566">scheduled</text>
<text class="nj-exceptions-line" x="40" y="1594">CMake 3.5, 3.31.6, and 4.0.0</text>
<text class="nj-exceptions-line" x="40" y="1612">Meson, Swift Package Manager</text>
<text class="nj-exceptions-line" x="40" y="1630">dependency review on every PR</text>
<text class="nj-exceptions-line" x="40" y="1648">Dependabot daily, OpenSSF</text>
<text class="nj-exceptions-line" x="40" y="1666">Scorecard, R&#8203;EUSE</text>
</g>
<line class="nj-exceptions-edge" x1="180" y1="1642" x2="180" y2="1668" marker-end="url(#nj-qa-arrow-n)"/>
<rect class="nj-exceptions-stage nj-exceptions-stage--merge" x="65" y="1670" width="230" height="34" rx="17"/>
<text class="nj-exceptions-stagetext" x="180" y="1692">merge into develop</text>
<line class="nj-exceptions-edge" x1="180" y1="1704" x2="180" y2="1730" marker-end="url(#nj-qa-arrow-n)"/>
<rect class="nj-exceptions-stage nj-exceptions-stage--merge" x="30" y="1732" width="300" height="46" rx="23"/>
<text class="nj-exceptions-stagetext" x="180" y="1754">release</text>
<text class="nj-exceptions-stagesub" x="180" y="1770">no open vulnerability alert</text>
<line class="nj-exceptions-edge" x1="180" y1="1696" x2="180" y2="1722" marker-end="url(#nj-qa-arrow-n)"/>
<rect class="nj-exceptions-stage nj-exceptions-stage--merge" x="65" y="1724" width="230" height="34" rx="17"/>
<text class="nj-exceptions-stagetext" x="180" y="1746">merge into develop</text>
<line class="nj-exceptions-edge" x1="180" y1="1758" x2="180" y2="1784" marker-end="url(#nj-qa-arrow-n)"/>
<rect class="nj-exceptions-stage nj-exceptions-stage--merge" x="30" y="1786" width="300" height="46" rx="23"/>
<text class="nj-exceptions-stagetext" x="180" y="1808">release</text>
<text class="nj-exceptions-stagesub" x="180" y="1824">no open vulnerability alert</text>
</svg>

Before

Width:  |  Height:  |  Size: 22 KiB

After

Width:  |  Height:  |  Size: 23 KiB

-1
View File
@@ -142,7 +142,6 @@ nav:
- features/performance.md
- Parsing:
- features/parsing/index.md
- features/parsing/error_recovery.md
- features/parsing/json_lines.md
- features/parsing/large_and_partial_input.md
- features/parsing/parse_exceptions.md
+11 -6
View File
@@ -22,19 +22,24 @@ P = "nj-exceptions-"
TRIGGERS = {
"push": ("every push / PR", "push"),
"pr": ("every PR", "push"),
"merge": ("before merge", "push"),
"nightly": ("nightly", "sched"),
"sched": ("scheduled", "sched"),
"cont": ("continuous", "cont"),
}
# title, triggers, lines (at most 31 characters, so they fit a card of the three-column drawing)
LANES = [
("Build matrix", ["push"], [
("Build matrix", ["push", "merge"], [
"50+ compilers: GCC, Clang, MSVC,",
"AppleClang, Intel, NVIDIA, ...",
"C++11, 14, 17, 20, 23, and 26",
"libc++ and libstdc++",
"warnings: Clang -Weverything,",
"GCC 300+ warnings",
"a subset on every push and PR,",
"all before merge; Intel, NVIDIA,",
"CUDA, and ARM64 nightly",
]),
("Unit tests (doctest)", ["push"], [
"every public API function",
@@ -44,7 +49,7 @@ LANES = [
"custom number, string, array,",
"and object types",
]),
("Coverage", ["push"], [
("Coverage", ["nightly"], [
"unit tests cover all lines",
"of the code base",
]),
@@ -53,11 +58,11 @@ LANES = [
"Clang Static Analyzer (52",
"checkers), IWYU, Codacy",
"CodeQL, Semgrep, Flawfinder",
"(also scheduled)",
"(also scheduled); Infer nightly",
]),
("Dynamic analysis", ["push"], [
("Dynamic analysis", ["push", "nightly"], [
"runtime assertions",
"Valgrind (Memcheck)",
"Valgrind (Memcheck), nightly",
"address and undefined behavior",
"sanitizers, integer overflow,",
"nullability violations",
@@ -178,7 +183,7 @@ def drawing(W, cols, suffix):
defs = ('<defs><marker id="%s" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto">'
'<path class="fg-head" d="M0 1 L10 5 L0 9 z"/></marker></defs>' % mid)
tid, did = "nj-qa-title-" + suffix, "nj-qa-desc-" + suffix
desc = ("Every push or pull request starts these checks in parallel: " +
desc = ("A push or pull request starts these checks; the badges say when each runs: " +
"; ".join("%s (%s): %s" % (t, ", ".join(TRIGGERS[k][0] for k in trig), " ".join(lines)) for t, trig, lines in LANES) +
". When they pass, the change is merged into develop; a release is made only when no vulnerability alert is open.")
out = ['<svg class="nj-fig %sqa %sqa--%s" viewBox="0 0 %d %d" role="img" aria-labelledby="%s %s" xmlns="http://www.w3.org/2000/svg">'
+1 -23
View File
@@ -431,28 +431,6 @@ std::nullptr_t exception_context(const View& /*unused*/) noexcept
return nullptr;
}
namespace view
{
// the path and bytes of the value of a view (defined in <nlohmann/detail/view/errors.hpp>)
template<typename View>
std::string diagnostics_of(const View& v);
} // namespace view
// what goes in front of the message of an exception about j: nothing for a
// basic_json value (exception_context() passes the value itself), the context
// basic_json would add (JSON_DIAGNOSTICS, JSON_DIAGNOSTIC_POSITIONS) for a view
template<typename BasicJsonType, enable_if_t<is_basic_json<BasicJsonType>::value, int> = 0>
const char* exception_prefix(const BasicJsonType& /*unused*/) noexcept
{
return "";
}
template<typename View, enable_if_t<is_json_view<View>::value, int> = 0>
std::string exception_prefix(const View& v)
{
return view::diagnostics_of(v);
}
/*!
@brief throws because @a j does not have the type a conversion expects
@param[in] expected the expected type(s), e.g. "array" or "binary or array"
@@ -464,7 +442,7 @@ JSON_HEDLEY_NO_RETURN JSON_HEDLEY_NEVER_INLINE void throw_type_must_be(const cha
{
static_cast<void>(expected); // unused when JSON_NOEXCEPTION is defined
static_cast<void>(j);
JSON_THROW(type_error::create(exception_id::type_mismatch, concat(exception_prefix(j), "type must be ", expected, ", but is ", j.type_name()), exception_context(j)));
JSON_THROW(type_error::create(exception_id::type_mismatch, concat("type must be ", expected, ", but is ", j.type_name()), exception_context(j)));
}
/*!
File diff suppressed because it is too large. Load diff
@@ -796,22 +796,17 @@ class iterator_input_adapter
return get_consumed_count() + eof_reads;
}
// number of newlines in the consumed range [first_index, last_index); if
// there is one, @a line_start is set to the index after the last of them,
// otherwise it is left unchanged; only valid when supports_seek
std::size_t count_consumed_lines(std::size_t first_index, std::size_t last_index, std::size_t& line_start) const
// number of newlines in the consumed range [0, last_index); @a line_start
// is set to the index after the last of them (0 if there is none); only
// valid when supports_seek
std::size_t count_consumed_lines(std::size_t last_index, std::size_t& line_start) const
{
using difference_type = typename std::iterator_traits<IteratorType>::difference_type;
const auto from = std::next(begin, static_cast<difference_type>(first_index));
const auto to = std::next(from, static_cast<difference_type>(last_index - first_index));
const auto to = std::next(begin, static_cast<typename std::iterator_traits<IteratorType>::difference_type>(last_index));
const auto newline = char_traits<char_type>::to_char_type('\n');
using reverse_it = std::reverse_iterator<IteratorType>;
const auto last_newline = std::find(reverse_it(to), reverse_it(from), newline);
if (last_newline.base() != from)
{
line_start = static_cast<std::size_t>(std::distance(begin, last_newline.base()));
}
return static_cast<std::size_t>(std::count(from, last_newline.base(), newline));
const auto last_newline = std::find(reverse_it(to), reverse_it(begin), newline);
line_start = static_cast<std::size_t>(std::distance(begin, last_newline.base()));
return static_cast<std::size_t>(std::count(begin, last_newline.base(), newline));
}
// append the already-consumed characters in the half-open range
+3 -10
View File
@@ -138,9 +138,7 @@ struct json_sax
@param[in] position the position in the input where the error occurs
@param[in] last_token the last read token
@param[in] ex an exception object describing the error
@return whether to recover from the error: false stops parsing; true
repairs the error and continues, or, if that is not possible,
stops after completing the value read so far
@return whether parsing should proceed (must return false)
*/
virtual bool parse_error(std::size_t position,
const std::string& last_token,
@@ -487,9 +485,6 @@ stored in place as it arrives, so after a parse error the value holds what was
read up to it.
@tparam BasicJsonType the JSON type
@tparam InputAdapterType the input adapter of the lexer that can be passed to
the constructor to record diagnostic positions; it
does not matter if no lexer is passed
@tparam Collect whether to collect the values of containers of unknown size on
a stack (only for a caller that does not use the value after a
parse error)
@@ -498,7 +493,7 @@ read up to it.
it, so any type with its three functions works (as
materialize() of json_view uses it)
*/
template<typename BasicJsonType, typename InputAdapterType = string_input_adapter_type, bool Collect = false, typename LexerType = lexer<BasicJsonType, InputAdapterType>>
template<typename BasicJsonType, typename InputAdapterType, bool Collect = false, typename LexerType = lexer<BasicJsonType, InputAdapterType>>
class json_sax_dom_parser : public json_sax_dom_error_state
{
public:
@@ -804,8 +799,6 @@ class json_sax_dom_parser : public json_sax_dom_error_state
JSON_HEDLEY_NEVER_INLINE
void grow()
{
// the stack starts with local_capacity frames and only grows
JSON_ASSERT(capacity != 0);
const std::size_t new_capacity = capacity * 2;
std::unique_ptr<frame[]> new_heap(new frame[new_capacity]); // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
for (std::size_t i = 0; i < count; ++i)
@@ -1621,7 +1614,7 @@ class json_sax_dom_binary_parser : public json_sax_dom_error_state
lexer_t* m_lexer_ref = nullptr;
};
template<typename BasicJsonType, typename InputAdapterType = string_input_adapter_type>
template<typename BasicJsonType, typename InputAdapterType>
class json_sax_dom_callback_parser : public json_sax_dom_error_state
{
public:
File diff suppressed because it is too large. Load diff
+30 -743
View File
@@ -72,14 +72,14 @@ This class implements a parser for JSON text. Nested arrays and objects are trac
stack instead of recursion, so deeply nested input does not exhaust the call stack, and what is read
is reported as SAX events.
*/
template<typename BasicJsonType, typename InputAdapterType, bool DiscardValues = false>
template<typename BasicJsonType, typename InputAdapterType>
class parser
{
using number_integer_t = typename BasicJsonType::number_integer_t;
using number_unsigned_t = typename BasicJsonType::number_unsigned_t;
using number_float_t = typename BasicJsonType::number_float_t;
using string_t = typename BasicJsonType::string_t;
using lexer_t = lexer<BasicJsonType, InputAdapterType, DiscardValues>;
using lexer_t = lexer<BasicJsonType, InputAdapterType>;
using token_type = typename lexer_t::token_type;
public:
@@ -88,9 +88,10 @@ class parser
parser_callback_t<BasicJsonType> cb = nullptr,
const bool allow_exceptions_ = true,
const bool ignore_comments = false,
const bool ignore_trailing_commas_ = false)
const bool ignore_trailing_commas_ = false,
const bool discard_number_values_ = false)
: callback(std::move(cb))
, m_lexer(std::move(adapter), ignore_comments)
, m_lexer(std::move(adapter), ignore_comments, discard_number_values_)
, allow_exceptions(allow_exceptions_)
, ignore_trailing_commas(ignore_trailing_commas_)
{
@@ -152,72 +153,25 @@ class parser
bool accept(const bool strict = true)
{
json_sax_acceptor<BasicJsonType> sax_acceptor;
return sax_parse_impl<false>(&sax_acceptor, strict);
return sax_parse(&sax_acceptor, strict);
}
/*!
@brief public SAX interface
If the SAX parser's parse_error() returns true, the parser recovers from
the error: it repairs the input and continues (see #3989).
@param[in] sax the SAX parser
@param[in] strict whether to expect the last token to be EOF
@return whether the input was parsed without errors and no SAX event
returned false
*/
template<typename SAX>
JSON_HEDLEY_NON_NULL(2)
bool sax_parse(SAX* sax, const bool strict = true)
{
return sax_parse_impl<true>(sax, strict);
}
private:
/// what sax_parse_internal() does after an object key was expected
enum class next_step : std::uint8_t
{
/// stop parsing
stop,
/// parse a value that begins with last_token
parse_value,
/// evaluate the state of the innermost container, which reads
/// last_token again
evaluate_state
};
template<bool AllowRecovery, typename SAX>
JSON_HEDLEY_NON_NULL(2)
bool sax_parse_impl(SAX* sax, const bool strict)
{
(void)detail::is_sax_static_asserts<SAX, BasicJsonType> {};
const bool result = sax_parse_internal<AllowRecovery>(sax);
const bool result = sax_parse_internal(sax);
if (result)
{
if (strict)
{
// strict mode: next byte must be EOF; after recovering from an
// error, the end of the input may already have been read
if (last_token != token_type::end_of_input && get_token() != token_type::end_of_input)
{
// the value is complete, so there is nothing to recover
static_cast<void>(report_error(sax, parse_error::create(exception_id::syntax_error, m_lexer.get_position(), exception_message(token_type::end_of_input, "value"), nullptr),
std::integral_constant<bool, AllowRecovery> {}));
return false;
}
}
else
{
// the caller keeps using the input: position it right after
// the value by leaving the character that terminated it
m_lexer.release_lookahead();
}
return finish_input(*sax, strict);
}
return result && !error_reported;
return false;
}
private:
/// the text of the current float token for @a sax (see sax_ignores_number_text)
template<typename SAX>
const string_t& number_float_text(SAX* /*sax*/)
@@ -285,26 +239,17 @@ class parser
template<typename DomSax>
bool parse_dom(DomSax& sdp, const bool strict)
{
sax_parse_internal<false>(&sdp);
sax_parse_internal(&sdp);
static_cast<void>(finish_input(sdp, strict));
return !sdp.is_errored();
}
/*!
@brief parse a JSON value and pass it to a SAX parser
@tparam AllowRecovery whether to recover from an error if the SAX parser's
parse_error() returns true; false for the SAX parsers
of parse() and accept(), which never do, so that no
code for recovering is generated for them
*/
template<bool AllowRecovery, typename SAX>
template<typename SAX>
JSON_HEDLEY_NON_NULL(2)
bool sax_parse_internal(SAX* sax)
{
const std::integral_constant<bool, AllowRecovery> allow_recovery{};
// value to avoid a goto (see comment where set to true)
bool skip_to_state_evaluation = false;
@@ -343,18 +288,10 @@ class parser
break;
}
// remember we are now inside an object
states.push_back(0);
// parse key (the steps of parse_key(), which are
// repeated here and below for speed)
// parse key
if (JSON_HEDLEY_UNLIKELY(last_token != token_type::value_string))
{
if (!continue_after(key_error(sax, allow_recovery, false), skip_to_state_evaluation))
{
return false;
}
continue;
return syntax_error(*sax, exception_message(token_type::value_string, "object key"));
}
if (JSON_HEDLEY_UNLIKELY(!sax->key(m_lexer.get_string_token())))
{
@@ -364,13 +301,12 @@ class parser
// parse separator (:)
if (JSON_HEDLEY_UNLIKELY(!get_token_expecting(token_type::name_separator)))
{
if (!continue_after(key_error(sax, allow_recovery, true), skip_to_state_evaluation))
{
return false;
}
continue;
return syntax_error(*sax, exception_message(token_type::name_separator, "object separator"));
}
// remember we are now inside an object
states.push_back(0);
// parse values
get_token();
continue;
@@ -406,11 +342,9 @@ class parser
if (JSON_HEDLEY_UNLIKELY(!std::isfinite(res)))
{
if (!overflow_error(sax, res, allow_recovery))
{
return false;
}
break;
return sax->parse_error(m_lexer.get_position(),
m_lexer.get_token_string(),
out_of_range::create(exception_id::number_overflow, concat("number overflow parsing '", m_lexer.get_token_string(), '\''), nullptr));
}
if (JSON_HEDLEY_UNLIKELY(!sax->number_float(res, number_float_text(sax))))
@@ -478,63 +412,16 @@ class parser
case token_type::parse_error:
{
// using "uninitialized" to avoid an "expected" message
if (!report_error(sax, parse_error::create(exception_id::syntax_error, m_lexer.get_position(), exception_message(token_type::uninitialized, "value"), nullptr), allow_recovery))
{
return false;
}
// recover: keep what can be read of the token
recover_token(allow_recovery);
if (last_token != token_type::uninitialized)
{
// a string or a number
continue;
}
if (states.empty())
{
// look for the value after the garbage
if (!skip_to_value(allow_recovery))
{
return false;
}
continue;
}
// nothing could be read
if (JSON_HEDLEY_UNLIKELY(!sax->null()))
{
return false;
}
break;
return syntax_error(*sax, exception_message(token_type::uninitialized, "value"));
}
case token_type::end_of_input:
{
if (JSON_HEDLEY_UNLIKELY(m_lexer.get_chars_read_total() == 1))
{
// there is nothing to recover
static_cast<void>(report_error(sax, parse_error::create(exception_id::syntax_error, m_lexer.get_position(),
empty_input_message(), nullptr), allow_recovery));
return false;
return syntax_error(*sax, empty_input_message());
}
if (!report_error(sax, parse_error::create(exception_id::syntax_error, m_lexer.get_position(), exception_message(token_type::literal_or_value, "value"), nullptr), allow_recovery))
{
return false;
}
// recover: the input ends where a value is missing
if (states.empty())
{
// there is no value
return false;
}
if (!recover_missing_value(sax, states, allow_recovery))
{
return false;
}
// the state evaluation reads the token again
unget_token(allow_recovery);
skip_to_state_evaluation = true;
continue;
return syntax_error(*sax, exception_message(token_type::literal_or_value, "value"));
}
case token_type::uninitialized:
case token_type::end_array:
@@ -544,35 +431,7 @@ class parser
case token_type::literal_or_value:
default: // the last token was unexpected
{
if (!report_error(sax, parse_error::create(exception_id::syntax_error, m_lexer.get_position(), exception_message(token_type::literal_or_value, "value"), nullptr), allow_recovery))
{
return false;
}
// recover
if (states.empty())
{
// look for the value after the garbage
if (!skip_to_value(allow_recovery))
{
return false;
}
continue;
}
if (last_token == token_type::name_separator)
{
// a stray ':'; the value may follow
get_token();
continue;
}
if (!recover_missing_value(sax, states, allow_recovery))
{
return false;
}
// the state evaluation reads the token again
unget_token(allow_recovery);
skip_to_state_evaluation = true;
continue;
return syntax_error(*sax, exception_message(token_type::literal_or_value, "value"));
}
}
}
@@ -623,30 +482,7 @@ class parser
continue;
}
if (!report_error(sax, parse_error::create(exception_id::syntax_error, m_lexer.get_position(), exception_message(token_type::end_array, "array"), nullptr), allow_recovery))
{
return false;
}
// recover
if (last_token == token_type::end_of_input)
{
// the input ends inside the array
return close_containers(sax, states, allow_recovery);
}
if (last_token == token_type::end_object)
{
// a wrong closing bracket closes the innermost container
if (JSON_HEDLEY_UNLIKELY(!sax->end_array()))
{
return false;
}
states.pop_back();
skip_to_state_evaluation = true;
}
// otherwise, a missing ',' (or a stray ':', which value
// parsing drops): the next value begins here
continue;
return syntax_error(*sax, exception_message(token_type::end_array, "array"));
}
// states.back() is false -> object
@@ -663,12 +499,9 @@ class parser
// parse key
if (JSON_HEDLEY_UNLIKELY(last_token != token_type::value_string))
{
if (!continue_after(key_error(sax, allow_recovery, false), skip_to_state_evaluation))
{
return false;
}
continue;
return syntax_error(*sax, exception_message(token_type::value_string, "object key"));
}
if (JSON_HEDLEY_UNLIKELY(!sax->key(m_lexer.get_string_token())))
{
return false;
@@ -677,11 +510,7 @@ class parser
// parse separator (:)
if (JSON_HEDLEY_UNLIKELY(!get_token_expecting(token_type::name_separator)))
{
if (!continue_after(key_error(sax, allow_recovery, true), skip_to_state_evaluation))
{
return false;
}
continue;
return syntax_error(*sax, exception_message(token_type::name_separator, "object separator"));
}
// parse values
@@ -709,546 +538,10 @@ class parser
continue;
}
if (!report_error(sax, parse_error::create(exception_id::syntax_error, m_lexer.get_position(), exception_message(token_type::end_object, "object"), nullptr), allow_recovery))
{
return false;
}
// recover
if (last_token == token_type::end_of_input)
{
// the input ends inside the object
return close_containers(sax, states, allow_recovery);
}
if (last_token == token_type::end_array)
{
// a wrong closing bracket closes the innermost container
if (JSON_HEDLEY_UNLIKELY(!sax->end_object()))
{
return false;
}
states.pop_back();
skip_to_state_evaluation = true;
continue;
}
if (!continue_after(recover_member(sax, allow_recovery), skip_to_state_evaluation))
{
return false;
}
return syntax_error(*sax, exception_message(token_type::end_object, "object"));
}
}
/*!
@brief continue sax_parse_internal() after a recovery
@return whether to continue parsing
*/
bool continue_after(const next_step step, bool& skip_to_state_evaluation)
{
if (step == next_step::evaluate_state)
{
// the state evaluation reads the token again
m_lexer.unget_token();
skip_to_state_evaluation = true;
}
return step != next_step::stop;
}
/// the parser for parse() and accept() never recovers: stop parsing
JSON_INTERNAL_ALWAYS_INLINE
static std::false_type continue_after(std::false_type /*step*/, bool& /*skip_to_state_evaluation*/) noexcept
{
return {};
}
/*!
@brief parse an object key and the name separator (:) after it
last_token is the token where the key is expected. sax_parse_internal()
repeats these steps rather than calling this function, which is used
when recovering from an error.
@return next_step::parse_value if the value follows, with last_token its
first token; next_step::evaluate_state if the object's state is
to be evaluated after recovering from an error; next_step::stop
to stop parsing
*/
template<typename SAX>
next_step parse_key(SAX* sax)
{
const std::true_type allow_recovery{};
if (JSON_HEDLEY_UNLIKELY(last_token != token_type::value_string))
{
return key_error(sax, allow_recovery, false);
}
if (JSON_HEDLEY_UNLIKELY(!sax->key(m_lexer.get_string_token())))
{
return next_step::stop;
}
// parse separator (:)
if (JSON_HEDLEY_UNLIKELY(get_token() != token_type::name_separator))
{
return key_error(sax, allow_recovery, true);
}
// the value begins with the next token
get_token();
return next_step::parse_value;
}
/*!
@brief report a number that is too large for number_float_t, and recover
from the error by passing the value on; the SAX parser gets the
number's text as well
This is a separate function, as reading other numbers is measurably
slower if the error is handled where they are read.
@param[in] sax the SAX parser
@param[in] value the value that is not finite
@return whether to continue parsing
*/
template<typename SAX>
bool overflow_error(SAX* sax, const number_float_t value, std::true_type allow_recovery)
{
if (!report_error(sax, out_of_range::create(exception_id::number_overflow, concat("number overflow parsing '", m_lexer.get_token_string(), '\''), nullptr), allow_recovery))
{
return false;
}
return sax->number_float(value, number_float_text(sax));
}
/// @copydoc overflow_error
template<typename SAX>
JSON_INTERNAL_ALWAYS_INLINE
std::false_type overflow_error(SAX* sax, const number_float_t /*value*/, std::false_type allow_recovery)
{
return report_error(sax, out_of_range::create(exception_id::number_overflow, concat("number overflow parsing '", m_lexer.get_token_string(), '\''), nullptr), allow_recovery);
}
/*!
@brief report a missing key, or a missing name separator (:) after the
key; the parser for parse() and accept() never recovers
@param[in] key_read whether the key was read, so that the name separator
is missing
@return std::false_type, see report_error()
*/
template<typename SAX>
JSON_INTERNAL_ALWAYS_INLINE
std::false_type key_error(SAX* sax, std::false_type allow_recovery, const bool key_read)
{
return report_error(sax, parse_error::create(exception_id::syntax_error, m_lexer.get_position(), key_read
? exception_message(token_type::name_separator, "object separator")
: exception_message(token_type::value_string, "object key"), nullptr), allow_recovery);
}
/*!
@brief report a missing key, or a missing name separator (:) after the
key, and recover from it
@param[in] key_read whether the key was read, so that the name separator
is missing
*/
template<typename SAX>
next_step key_error(SAX* sax, std::true_type allow_recovery, const bool key_read)
{
if (!key_read)
{
if (!report_error(sax, parse_error::create(exception_id::syntax_error, m_lexer.get_position(), exception_message(token_type::value_string, "object key"), nullptr), allow_recovery))
{
return next_step::stop;
}
return recover_key(sax);
}
if (!report_error(sax, parse_error::create(exception_id::syntax_error, m_lexer.get_position(), exception_message(token_type::name_separator, "object separator"), nullptr), allow_recovery))
{
return next_step::stop;
}
return recover_name_separator(sax);
}
/////////////////////
// error recovery
/////////////////////
/*
The functions below repair an error after the SAX parser's parse_error()
returned true (see #3989). Each mistake is repaired by the smallest local
edit: a missing ',' or ':' is inserted, a stray token is removed, what can
be read of an invalid string or number is kept (see
lexer::recover_token()), a missing value becomes null, a wrong closing
bracket closes the innermost container, and the end of the input closes
all of them. The events stay balanced, and every key() is followed by
exactly one value.
A repair hands a token to the state evaluation, by returning it to the
lexer (lexer::unget_token()) so that the state evaluation reads it again,
only if it is ',', ']', '}', or the end of the input. The state evaluation
hands a token to value or key parsing only if it is none of them, so a
token is never handed back and forth. Every other step reads a token or
closes a container, so parsing always ends.
*/
/*!
@brief report an error to the SAX parser; the parser for parse() and
accept() never recovers
@return std::false_type rather than false: its value is known where the
function is called even if the call is not inlined, so the code
for recovering is not generated
*/
template<typename SAX, typename Exception>
JSON_INTERNAL_ALWAYS_INLINE
std::false_type report_error(SAX* sax, const Exception& ex, std::false_type /*allow_recovery*/)
{
static_cast<void>(sax); // MSVC 2015 does not count calling a static parse_error() as using it
error_reported = true;
static_cast<void>(sax->parse_error(m_lexer.get_position(), m_lexer.get_token_string(), ex));
return {};
}
/*!
@brief report an error to the SAX parser
@return whether to recover from the error
*/
template<typename SAX, typename Exception>
bool report_error(SAX* sax, const Exception& ex, std::true_type /*allow_recovery*/)
{
static_cast<void>(sax); // MSVC 2015 does not count calling a static parse_error() as using it
const std::size_t position = m_lexer.get_position().chars_read_total;
if (error_reported && position == last_error_position && last_token == last_error_token)
{
// a repair handed on the token of the error it repaired; the
// token was reported already, and the SAX parser asked to recover
return true;
}
error_reported = true;
last_error_position = position;
last_error_token = last_token;
if (!sax->parse_error(m_lexer.get_position(), m_lexer.get_token_string(), ex))
{
return false;
}
// the token string of the next error begins here
m_lexer.restart_token_string();
return true;
}
/*!
@brief keep what can be read of the token that the lexer rejected
The error was reported for the rejected token, so it is not reported again
for the token it is repaired to (see lexer::recover_token()).
*/
token_type recover_token()
{
last_token = m_lexer.recover_token();
last_error_position = m_lexer.get_position().chars_read_total;
last_error_token = last_token;
return last_token;
}
// The functions below are called from sax_parse_internal() after an error
// was reported. The parser for parse() and accept() stopped there, so for
// it they are stubs: the code for recovering is then not even referenced,
// and unoptimized builds do not emit it either.
/// @copydoc recover_token()
JSON_INTERNAL_ALWAYS_INLINE
token_type recover_token(std::true_type /*allow_recovery*/)
{
return recover_token();
}
/// @copydoc recover_token()
JSON_INTERNAL_ALWAYS_INLINE
static std::false_type recover_token(std::false_type /*allow_recovery*/) noexcept
{
return {};
}
/// return the token that was read last to the lexer (see lexer::unget_token())
JSON_INTERNAL_ALWAYS_INLINE
void unget_token(std::true_type /*allow_recovery*/)
{
m_lexer.unget_token();
}
/// @copydoc unget_token(std::true_type)
JSON_INTERNAL_ALWAYS_INLINE
static void unget_token(std::false_type /*allow_recovery*/) noexcept {}
/// @copydoc skip_to_value
JSON_INTERNAL_ALWAYS_INLINE
static std::false_type skip_to_value(std::false_type /*allow_recovery*/) noexcept
{
return {};
}
/// @copydoc recover_missing_value
template<typename SAX>
JSON_INTERNAL_ALWAYS_INLINE
static std::false_type recover_missing_value(SAX* /*sax*/, const std::vector<std::uint8_t>& /*states*/, std::false_type /*allow_recovery*/) noexcept
{
return {};
}
/// pass the end events of all open containers
template<typename SAX>
bool close_containers(SAX* sax, std::vector<std::uint8_t>& states, std::true_type /*allow_recovery*/)
{
while (!states.empty())
{
const bool is_array = states.back() != 0;
states.pop_back();
if (JSON_HEDLEY_UNLIKELY(is_array ? !sax->end_array() : !sax->end_object()))
{
return false;
}
}
return true;
}
/// @copydoc close_containers
template<typename SAX>
JSON_INTERNAL_ALWAYS_INLINE
static std::false_type close_containers(SAX* /*sax*/, std::vector<std::uint8_t>& /*states*/, std::false_type /*allow_recovery*/) noexcept
{
return {};
}
/*!
@brief read tokens until one begins a value, skipping everything before
the top-level value
@return whether a value begins with last_token
*/
bool skip_to_value(std::true_type /*allow_recovery*/)
{
while (true)
{
switch (get_token())
{
case token_type::begin_array:
case token_type::begin_object:
case token_type::literal_false:
case token_type::literal_null:
case token_type::literal_true:
case token_type::value_float:
case token_type::value_integer:
case token_type::value_string:
case token_type::value_unsigned:
return true;
case token_type::end_of_input:
return false;
case token_type::parse_error:
recover_token();
if (last_token != token_type::uninitialized)
{
return true;
}
break;
case token_type::uninitialized:
case token_type::end_array:
case token_type::end_object:
case token_type::name_separator:
case token_type::value_separator:
case token_type::literal_or_value:
default:
break;
}
}
}
/*!
@brief skip the rest of an object member that cannot be read
Reads tokens, beginning with last_token, until a ',', '}', or ']' that is
not inside a container that begins in the skipped tokens, or the end of
the input.
*/
void skip_member()
{
std::size_t depth = 0;
while (true)
{
switch (last_token)
{
case token_type::begin_array:
case token_type::begin_object:
++depth;
break;
case token_type::end_array:
case token_type::end_object:
if (depth == 0)
{
return;
}
--depth;
break;
case token_type::value_separator:
if (depth == 0)
{
return;
}
break;
case token_type::end_of_input:
return;
case token_type::parse_error:
recover_token();
break;
case token_type::uninitialized:
case token_type::literal_true:
case token_type::literal_false:
case token_type::literal_null:
case token_type::value_string:
case token_type::value_unsigned:
case token_type::value_integer:
case token_type::value_float:
case token_type::name_separator:
case token_type::literal_or_value:
default:
break;
}
get_token();
}
}
/*!
@brief pass a value where it is missing
last_token is ',', ']', '}', or the end of the input, where a value was
expected. In an object, the key gets null; in an array, a ',' where a
value is missing stands for null (as in JavaScript), while an array that
ends there just ends.
*/
template<typename SAX>
bool recover_missing_value(SAX* sax, const std::vector<std::uint8_t>& states, std::true_type /*allow_recovery*/)
{
JSON_ASSERT(!states.empty());
if (states.back() == 0 || last_token == token_type::value_separator)
{
return sax->null();
}
return true;
}
/// recover from a missing key; last_token is where it was expected
template<typename SAX>
next_step recover_key(SAX* sax)
{
switch (last_token)
{
case token_type::value_separator:
case token_type::end_object:
case token_type::end_array:
case token_type::end_of_input:
// no member: the object's state handles the token
return next_step::evaluate_state;
case token_type::parse_error:
recover_token();
if (last_token == token_type::value_string)
{
// a key that could be repaired
return parse_key(sax);
}
skip_member();
return next_step::evaluate_state;
case token_type::uninitialized:
case token_type::literal_true:
case token_type::literal_false:
case token_type::literal_null:
case token_type::value_string:
case token_type::value_unsigned:
case token_type::value_integer:
case token_type::value_float:
case token_type::begin_array:
case token_type::begin_object:
case token_type::name_separator:
case token_type::literal_or_value:
default:
// a member without a key
skip_member();
return next_step::evaluate_state;
}
}
/// recover from a missing name separator (:) after the key; last_token
/// is where it was expected
template<typename SAX>
next_step recover_name_separator(SAX* sax)
{
switch (last_token)
{
case token_type::value_separator:
case token_type::end_object:
case token_type::end_array:
case token_type::end_of_input:
// the value is missing as well
return sax->null() ? next_step::evaluate_state : next_step::stop;
case token_type::uninitialized:
case token_type::literal_true:
case token_type::literal_false:
case token_type::literal_null:
case token_type::value_string:
case token_type::value_unsigned:
case token_type::value_integer:
case token_type::value_float:
case token_type::begin_array:
case token_type::begin_object:
case token_type::name_separator:
case token_type::parse_error:
case token_type::literal_or_value:
default:
// a missing ':'; the value begins here
return next_step::parse_value;
}
}
/// recover from a token after an object member that is neither ',' nor
/// '}' (nor ']' or the end of the input, which the caller handles)
template<typename SAX>
next_step recover_member(SAX* sax, std::true_type /*allow_recovery*/)
{
if (last_token == token_type::parse_error)
{
recover_token();
}
if (last_token == token_type::value_string)
{
// a missing ','; the next key begins here
return parse_key(sax);
}
skip_member();
return next_step::evaluate_state;
}
/// the parser for parse() and accept() never recovers (and does not come
/// here, as report_error() returned false)
template<typename SAX>
JSON_INTERNAL_ALWAYS_INLINE
std::false_type recover_member(SAX* /*sax*/, std::false_type /*allow_recovery*/) const noexcept
{
return {};
}
/// get next token from lexer
token_type get_token()
{
@@ -1319,12 +612,6 @@ class parser
const bool allow_exceptions = true;
/// whether trailing commas in objects and arrays should be ignored (true) or signaled as errors (false)
const bool ignore_trailing_commas = false;
/// whether an error was reported to the SAX parser
bool error_reported = false;
/// the position of the last reported error
std::size_t last_error_position = 0;
/// the token of the last reported error
token_type last_error_token = token_type::uninitialized;
};
} // namespace detail
@@ -322,78 +322,6 @@ JSON_SCAN_SSSE3_TARGET JSON_HEDLEY_NEVER_INLINE inline bool vector_utf8_run(cons
block_out = block;
return false;
}
/*!
@brief skip 16-byte blocks of well-formed UTF-8 at p (a character boundary)
Like vector_utf8_run, but quotes, backslashes, and control characters are
ordinary characters. Returns the start of the first block in which the check
found an error (which may belong to a sequence begun in the block before) or
where fewer than 16 bytes are left; a sequence may be cut at that point, so the
caller resumes at its start, one sequence at a time.
*/
JSON_SCAN_SSSE3_TARGET JSON_HEDLEY_NEVER_INLINE inline const unsigned char* vector_utf8_valid_blocks(const unsigned char* block, const unsigned char* e) noexcept
{
using lookup = utf8_lookup4<>;
#if JSON_SCAN_NEON
const uint8x16_t t1h = vld1q_u8(lookup::byte_1_high.data());
const uint8x16_t t1l = vld1q_u8(lookup::byte_1_low.data());
const uint8x16_t t2h = vld1q_u8(lookup::byte_2_high.data());
uint8x16_t prev = vdupq_n_u8(0);
while (e - block >= 16)
{
const uint8x16_t in = vld1q_u8(block);
if (vmaxvq_u8(vorrq_u8(in, prev)) < 0x80)
{
// ASCII here, and no sequence from the block before to finish
prev = in;
block += 16;
continue;
}
const uint8x16_t prev1 = vextq_u8(prev, in, 15);
const uint8x16_t sc = vandq_u8(vandq_u8(vqtbl1q_u8(t1h, vshrq_n_u8(prev1, 4)), vqtbl1q_u8(t1l, vandq_u8(prev1, vdupq_n_u8(0x0F)))), vqtbl1q_u8(t2h, vshrq_n_u8(in, 4)));
const uint8x16_t must23 = vorrq_u8(vqsubq_u8(vextq_u8(prev, in, 14), vdupq_n_u8(0xE0 - 0x80)), vqsubq_u8(vextq_u8(prev, in, 13), vdupq_n_u8(0xF0 - 0x80)));
const uint8x16_t err = veorq_u8(vandq_u8(must23, vdupq_n_u8(0x80)), sc);
if (vmaxvq_u8(err) != 0)
{
break;
}
prev = in;
block += 16;
}
#else
const __m128i t1h = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(lookup::byte_1_high.data()))); // NOLINT(bugprone-casting-through-void)
const __m128i t1l = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(lookup::byte_1_low.data()))); // NOLINT(bugprone-casting-through-void)
const __m128i t2h = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(lookup::byte_2_high.data()))); // NOLINT(bugprone-casting-through-void)
const __m128i nibble = _mm_set1_epi8(0x0F);
const __m128i zero = _mm_setzero_si128();
__m128i prev = zero;
while (e - block >= 16)
{
const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(block))); // NOLINT(bugprone-casting-through-void)
if (_mm_movemask_epi8(_mm_or_si128(in, prev)) == 0)
{
prev = in;
block += 16;
continue;
}
const __m128i prev1 = _mm_alignr_epi8(in, prev, 15);
const __m128i sc = _mm_and_si128(_mm_and_si128(_mm_shuffle_epi8(t1h, _mm_and_si128(_mm_srli_epi16(prev1, 4), nibble)),
_mm_shuffle_epi8(t1l, _mm_and_si128(prev1, nibble))),
_mm_shuffle_epi8(t2h, _mm_and_si128(_mm_srli_epi16(in, 4), nibble)));
const __m128i must23 = _mm_or_si128(_mm_subs_epu8(_mm_alignr_epi8(in, prev, 14), _mm_set1_epi8(0xE0 - 0x80)),
_mm_subs_epu8(_mm_alignr_epi8(in, prev, 13), _mm_set1_epi8(0xF0 - 0x80)));
const __m128i err = _mm_xor_si128(_mm_and_si128(must23, _mm_set1_epi8(static_cast<char>(-128))), sc);
if (_mm_movemask_epi8(_mm_cmpeq_epi8(err, zero)) != 0xFFFF)
{
break;
}
prev = in;
block += 16;
}
#endif
return block;
}
#endif
} // namespace detail
@@ -90,34 +90,6 @@ JSON_HEDLEY_ALWAYS_INLINE std::size_t find_string_special(const unsigned char* d
return n;
}
// return the index of the first 0x00 byte in [data, data+n), or n if there is
// none; scans 8 bytes at a time (BSON's C-style strings)
JSON_HEDLEY_ALWAYS_INLINE std::size_t find_zero_byte(const unsigned char* data, std::size_t n) noexcept
{
constexpr std::uint64_t ones = 0x0101010101010101ull;
constexpr std::uint64_t high = 0x8080808080808080ull;
std::size_t i = 0;
for (; i + 8 <= n; i += 8)
{
const std::uint64_t v = read_eight_bytes(data + i);
const std::uint64_t zero = (v - ones) & ~v & high;
if (zero != 0)
{
// the lowest flagged byte is the first zero byte (see
// find_string_special())
return i + (static_cast<std::size_t>(count_trailing_zeros(zero)) / 8);
}
}
for (; i < n; ++i)
{
if (data[i] == 0x00)
{
return i;
}
}
return n;
}
// classify a byte as one the serializer must NOT copy verbatim when
// ensure_ascii is requested: the closing quote, an escape, a control character
// (< 0x20), DEL (0x7F), or any non-ASCII byte (>= 0x80). Everything else -
@@ -264,64 +236,6 @@ inline std::size_t valid_utf8_prefix(const unsigned char* data, std::size_t n) n
return pos;
}
/*!
@brief whether [data, data+n) is valid UTF-8, for checking whole strings
The same answer as valid_utf8_prefix(data, n) == n. Short ASCII strings, the
most common keys and values, are decided in one test, and long strings are
checked 16 bytes at a time where the vector check is available.
*/
inline bool valid_utf8_string(const unsigned char* data, std::size_t n) noexcept
{
constexpr std::uint64_t high = 0x8080808080808080ull;
if (n <= 16)
{
// two overlapping words cover 8 to 16 bytes, one partial word fewer
std::uint64_t first = 0;
std::uint64_t last = 0;
if (n >= 8)
{
std::memcpy(&first, data, sizeof(first));
std::memcpy(&last, data + n - 8, sizeof(last));
}
else if (n != 0)
{
std::memcpy(&first, data, n);
}
if (((first | last) & high) == 0)
{
return true;
}
}
#if JSON_SCAN_VECTOR_UTF8
else if (n >= 32
#if JSON_SCAN_SSSE3_DISPATCH
&& cpu_has_ssse3()
#endif
)
{
const unsigned char* block = vector_utf8_valid_blocks(data, data + n);
// the rest from the start of a sequence that may be cut at block
for (int i = 1; i <= 3 && block - i >= data; ++i)
{
const unsigned char c = block[-i];
if (c < 0x80)
{
break;
}
if (c >= 0xC0)
{
block -= i;
break;
}
}
const auto checked = static_cast<std::size_t>(block - data);
return checked + valid_utf8_prefix(block, n - checked) == n;
}
#endif
return valid_utf8_prefix(data, n) == n;
}
#if JSON_SCAN_VECTOR_UTF8
/// the end of scan_string_vector from block, where the vector loop stopped
/// (ill-formed UTF-8, or fewer than 16 bytes left): one byte or sequence at a
+1 -12
View File
@@ -215,17 +215,6 @@
#define JSON_NO_UNIQUE_ADDRESS
#endif
// Inlines small functions even in unoptimized builds, so that they are not
// emitted. The parsers for parse(), accept(), and from_*() use it for the
// functions that stand in for the code recovering from errors (see #3989).
// MSVC is left to decide, as it warns (C4714) where it does not inline a
// __forceinline function.
#if defined(_MSC_VER) && !defined(__clang__)
#define JSON_INTERNAL_ALWAYS_INLINE
#else
#define JSON_INTERNAL_ALWAYS_INLINE JSON_HEDLEY_ALWAYS_INLINE
#endif
// Clang targeting MinGW does not survive the thread_local storage the copy
// constructor uses to bound its descent: every test that copies a value
// segfaults with clang 11.0.1 and clang 18.1.8, while the same tests pass with
@@ -358,7 +347,7 @@
return ej_pair.second == j; \
}); \
if (it != std::end(m)) e = it->first; \
else ::nlohmann::detail::templated_json_throw<nlohmann::detail::out_of_range>(nlohmann::detail::out_of_range::create(nlohmann::detail::exception_id::value_out_of_range, nlohmann::detail::concat(nlohmann::detail::exception_prefix(j), "enum value out of range for " #ENUM_TYPE ": ", nlohmann::detail::from_json_dump(j)), nlohmann::detail::exception_context(j))); \
else ::nlohmann::detail::templated_json_throw<nlohmann::detail::out_of_range>(nlohmann::detail::out_of_range::create(nlohmann::detail::exception_id::value_out_of_range, nlohmann::detail::concat("enum value out of range for " #ENUM_TYPE ": ", nlohmann::detail::from_json_dump(j)), nlohmann::detail::exception_context(j))); \
}
// Ugly macros to avoid uglier copy-paste when specializing basic_json. They
@@ -19,7 +19,6 @@
#undef NLOHMANN_CAN_CALL_STD_FUNC_IMPL
#undef JSON_INLINE_VARIABLE
#undef JSON_NO_UNIQUE_ADDRESS
#undef JSON_INTERNAL_ALWAYS_INLINE
#undef JSON_DISABLE_ENUM_SERIALIZATION
#undef JSON_DISABLE_TUPLE_REFERENCE_CONVERSION
#undef JSON_UNLOCKED_FILE_IO
+32 -286
View File
@@ -2088,8 +2088,7 @@ class binary_writer
/// arrays: the first index with one more digit; only maintained by
/// @ref calc_bson_sizes
std::uint64_t index_boundary = 10;
/// @ref calc_bson_sizes: where its size goes in the table; @ref
/// write_bson_document_single_pass: the position of its size field
/// @ref calc_bson_sizes only: where its size goes in the table
std::size_t size_slot;
/// @ref calc_bson_sizes only: the size of its entries seen so far
std::size_t entries_size = 0;
@@ -2340,199 +2339,6 @@ class binary_writer
}
}
template<typename SinkType>
using sink_written_t = decltype(std::declval<const SinkType&>().written());
/// whether every integer fits the 64 bits BSON has, so that writing an
/// array of numbers needs no range check
static constexpr bool bson_integers_fit_64_bits =
std::numeric_limits<typename BasicJsonType::number_integer_t>::is_specialized
&& std::numeric_limits<typename BasicJsonType::number_integer_t>::digits <= std::numeric_limits<std::int64_t>::digits
&& std::numeric_limits<typename BasicJsonType::number_unsigned_t>::is_specialized
&& std::numeric_limits<typename BasicJsonType::number_unsigned_t>::digits <= std::numeric_limits<std::uint64_t>::digits;
/*!
@brief whether BSON documents can be written in a single pass: the sink
can overwrite what it has written (output_vector_sink), and
@ref bson_integers_fit_64_bits
*/
using bson_single_pass_sink = conjunction<is_detected<sink_written_t, OutputSinkType>, std::integral_constant<bool, bson_integers_fit_64_bits>>;
/// write a placeholder for the size of an object or array whose entries
/// follow; @return its position, for @ref close_bson_container
std::size_t write_bson_size_placeholder()
{
const std::size_t pos = oa.written();
// zero in any byte order
const std::array<CharType, sizeof(std::int32_t)> zero{};
oa.write_characters(zero.data(), zero.size());
return pos;
}
/// end the object or array whose size is at @a pos and write its size
/// there: everything from that field to the terminating null byte
/// @return false (and the size is not written) if it does not fit
bool close_bson_container(const std::size_t pos)
{
oa.write_character(to_char_type(0x00));
const std::size_t size = oa.written() - pos;
if (JSON_HEDLEY_UNLIKELY(!value_in_range_of<std::int32_t>(size)))
{
return false;
}
const auto le_size = static_cast<std::uint32_t>(size);
oa.overwrite(pos, is_little_endian ? le_size : byteswap(le_size));
return true;
}
/*!
@brief check the value @a j of a BSON entry as @ref calc_bson_value_size
does, for @ref write_bson_document_single_pass
@return at least the size of the value
A number, boolean, or null needs no check there (its integers have at most
64 bits) and has at most 8 characters; a discarded value is rejected with
the same exception when it is written.
*/
std::size_t check_bson_value(const BasicJsonType& j)
{
return (j.is_string() || j.is_binary()) ? calc_bson_value_size(j) : 8;
}
/// whether an entry of @a entry_size characters keeps the document that
/// begins at @a start, closed by @a open_containers null bytes, within
/// the 32-bit size of BSON
bool bson_entry_fits(const std::size_t start, const std::size_t entry_size, const std::size_t open_containers) const noexcept
{
const auto limit = static_cast<std::size_t>((std::numeric_limits<std::int32_t>::max)());
const std::size_t written = oa.written() - start;
return written <= limit && entry_size <= limit - written && open_containers <= limit - written - entry_size;
}
/*!
@brief Serializes the JSON object @a document as a BSON document in a
single pass, writing the size of each object and array once it is
complete
Checks every key and value as @ref calc_bson_sizes does, in the same
order, so the same exception is thrown; the output written up to then is
discarded with the vector it was written to.
@return false as soon as the document or a string, binary value, object,
or array in it would be larger than the 32-bit BSON sizes allow;
nothing large has been written then, and the caller writes the
document in two passes, which reports that error
*/
template<typename SinkType = OutputSinkType, typename std::enable_if<is_detected<sink_written_t, SinkType>::value, int>::type = 0>
bool write_bson_document_single_pass(const BasicJsonType& document)
{
const std::size_t start = oa.written();
// the object or array whose entries are being written (with the
// position of its size), and the ones it is in
bson_frame current(&document, write_bson_size_placeholder());
std::vector<bson_frame> parents;
while (true)
{
const BasicJsonType* nested = nullptr;
std::size_t nested_size_pos = 0;
if (current.value->is_object())
{
const auto& object = *current.value->m_data.m_value.object;
while (nested == nullptr && current.member != object.cend())
{
const auto& el = *current.member;
++current.member;
const std::size_t header_size = calc_bson_entry_header_size(el.first, el.second);
const std::size_t value_size = el.second.is_structured() ? 0 : check_bson_value(el.second);
if (JSON_HEDLEY_UNLIKELY(!bson_entry_fits(start, header_size + value_size, parents.size() + 1)))
{
return false;
}
if (el.second.is_structured())
{
write_bson_entry_header(el.first, el.second.is_object() ? 0x03 : 0x04);
nested_size_pos = write_bson_size_placeholder();
nested = &el.second;
}
else
{
write_bson_value(el.first, el.second);
}
}
}
else
{
const auto& array = *current.value->m_data.m_value.array;
while (nested == nullptr && current.index < array.size())
{
if (is_bson_fixed_size(array[current.index]))
{
write_bson_array_fixed_entries(current);
if (current.index == array.size())
{
break;
}
}
// a string, binary value, object or array
const BasicJsonType& el = array[current.index];
const string_t index_name(current.index_name.data(), current.index_name_size);
current.next_index();
const std::size_t value_size = el.is_structured() ? 0 : check_bson_value(el);
if (JSON_HEDLEY_UNLIKELY(!bson_entry_fits(start, 2 + index_name.size() + value_size, parents.size() + 1)))
{
return false;
}
if (el.is_structured())
{
write_bson_entry_header(index_name, el.is_object() ? 0x03 : 0x04);
nested_size_pos = write_bson_size_placeholder();
nested = &el;
}
else
{
write_bson_value(index_name, el);
}
}
}
if (nested != nullptr)
{
if (nested->is_array())
{
// an array of booleans, numbers and null is written without a frame on the stack
bson_frame leaf(nested, nested_size_pos);
write_bson_array_fixed_entries(leaf);
if (leaf.index == nested->m_data.m_value.array->size())
{
if (JSON_HEDLEY_UNLIKELY(!close_bson_container(nested_size_pos)))
{
return false;
}
continue;
}
parents.push_back(std::move(current));
current = std::move(leaf);
continue;
}
parents.push_back(std::move(current));
current = bson_frame(nested, nested_size_pos);
continue;
}
if (JSON_HEDLEY_UNLIKELY(!close_bson_container(current.size_slot)))
{
return false;
}
if (parents.empty())
{
return true;
}
current = std::move(parents.back());
parents.pop_back();
}
}
/*!
@brief Serializes the JSON object @a document as a BSON document
@@ -2544,39 +2350,6 @@ class binary_writer
@pre document.type() == value_t::object
*/
void write_bson_document(const BasicJsonType& document)
{
write_bson_document(document, bson_single_pass_sink());
}
/// the sink can overwrite: one pass, unless the error handler replaces or
/// drops ill-formed UTF-8, which changes the sizes the checks compute
void write_bson_document(const BasicJsonType& document, std::true_type /*single_pass*/)
{
const std::size_t start = oa.written();
if ((error_handler == error_handler_t::keep || error_handler == error_handler_t::strict)
&& write_bson_document_single_pass(document))
{
return;
}
// a length does not fit: start over, so that it is reported as the
// two passes report it, before anything is written
oa.rewind(start);
write_bson_document_two_pass(document);
}
void write_bson_document(const BasicJsonType& document, std::false_type /*single_pass*/)
{
write_bson_document_two_pass(document);
}
/*!
@brief Serializes the JSON object @a document as a BSON document: the
sizes of all objects and arrays first, then the entries
@param[in] document the JSON object to serialize
@pre document.type() == value_t::object
*/
void write_bson_document_two_pass(const BasicJsonType& document)
{
std::vector<std::size_t> nested_sizes;
const std::size_t document_size = calc_bson_sizes(document, nested_sizes);
@@ -3307,25 +3080,6 @@ class binary_writer
}
}
/// write the head and the characters of @a value, at most
/// max_text_in_reservation of them, in one reservation
JSON_HEDLEY_ALWAYS_INLINE void write_cbor_string_reserved(const string_t& value)
{
CharType* const out = oa.reserve(value.size() + 2);
CharType* p = out;
if (value.size() <= 0x17)
{
*p++ = to_char_type(static_cast<std::uint8_t>(0x60 + value.size()));
}
else
{
*p++ = to_char_type(0x78);
*p++ = to_char_type(static_cast<std::uint8_t>(value.size()));
}
copy_characters(p, reinterpret_cast<const CharType*>(value.data()), value.size());
oa.commit(static_cast<std::size_t>(p - out) + value.size());
}
/*!
@brief write a CBOR text string
@@ -3347,7 +3101,19 @@ class binary_writer
// the string is written as it is: head and characters in one reservation
if (JSON_HEDLEY_LIKELY(value.size() <= max_text_in_reservation))
{
write_cbor_string_reserved(value);
CharType* const out = oa.reserve(value.size() + 2);
CharType* p = out;
if (value.size() <= 0x17)
{
*p++ = to_char_type(static_cast<std::uint8_t>(0x60 + value.size()));
}
else
{
*p++ = to_char_type(0x78);
*p++ = to_char_type(static_cast<std::uint8_t>(value.size()));
}
copy_characters(p, reinterpret_cast<const CharType*>(value.data()), value.size());
oa.commit(static_cast<std::size_t>(p - out) + value.size());
return;
}
}
@@ -3361,13 +3127,6 @@ class binary_writer
JSON_HEDLEY_NEVER_INLINE
void write_cbor_checked_string(const string_t& value, const BasicJsonType& context)
{
if (error_handler == error_handler_t::strict && value.size() <= max_text_in_reservation)
{
check_utf8(value, context);
write_cbor_string_reserved(value);
return;
}
string_t storage;
const string_t& sanitized = sanitize_utf8_for_write(value, context, storage);
@@ -3382,27 +3141,6 @@ class binary_writer
// MsgPack //
/////////////
/// write the head and the characters of @a value, at most
/// max_text_in_reservation of them, in one reservation
JSON_HEDLEY_ALWAYS_INLINE void write_msgpack_string_reserved(const string_t& value)
{
CharType* const out = oa.reserve(value.size() + 2);
CharType* p = out;
if (value.size() <= 31)
{
// fixstr
*p++ = to_char_type(static_cast<std::uint8_t>(0xA0 | value.size()));
}
else
{
// str 8
*p++ = to_char_type(0xD9);
*p++ = to_char_type(static_cast<std::uint8_t>(value.size()));
}
copy_characters(p, reinterpret_cast<const CharType*>(value.data()), value.size());
oa.commit(static_cast<std::size_t>(p - out) + value.size());
}
/*!
@brief write a MessagePack str
@@ -3420,7 +3158,21 @@ class binary_writer
// the string is written as it is: head and characters in one reservation
if (JSON_HEDLEY_LIKELY(value.size() <= max_text_in_reservation))
{
write_msgpack_string_reserved(value);
CharType* const out = oa.reserve(value.size() + 2);
CharType* p = out;
if (value.size() <= 31)
{
// fixstr
*p++ = to_char_type(static_cast<std::uint8_t>(0xA0 | value.size()));
}
else
{
// str 8
*p++ = to_char_type(0xD9);
*p++ = to_char_type(static_cast<std::uint8_t>(value.size()));
}
copy_characters(p, reinterpret_cast<const CharType*>(value.data()), value.size());
oa.commit(static_cast<std::size_t>(p - out) + value.size());
return;
}
}
@@ -3433,13 +3185,6 @@ class binary_writer
JSON_HEDLEY_NEVER_INLINE
void write_msgpack_checked_string(const string_t& value, const BasicJsonType& context)
{
if (error_handler == error_handler_t::strict && value.size() <= max_text_in_reservation)
{
check_utf8(value, context);
write_msgpack_string_reserved(value);
return;
}
string_t storage;
const string_t& sanitized = sanitize_utf8_for_write(value, context, storage);
@@ -5049,9 +4794,10 @@ class binary_writer
static void check_utf8(const string_t& s, const BasicJsonType& context)
{
const auto* data = reinterpret_cast<const unsigned char*>(s.data());
if (JSON_HEDLEY_UNLIKELY(!valid_utf8_string(data, s.size())))
const std::size_t valid = valid_utf8_prefix(data, s.size());
if (JSON_HEDLEY_UNLIKELY(valid != s.size()))
{
throw_invalid_utf8(data, valid_utf8_prefix(data, s.size()), context);
throw_invalid_utf8(data, valid, context);
}
}
@@ -11,7 +11,6 @@
#include <algorithm> // max
#include <array> // array
#include <cstddef> // size_t
#include <cstdint> // uint32_t
#include <cstring> // memcpy
#include <memory> // shared_ptr, make_shared
#include <string> // basic_string
@@ -228,31 +227,6 @@ class output_vector_sink
void flush() noexcept
{}
/// the number of characters written so far, the position of the next one
std::size_t written() const noexcept
{
return position();
}
/// @brief replace the 4 bytes written before at position @a pos with
/// those of @a n, in the byte order of the machine
///
/// Lets a writer put a length in front of data it has written since, such
/// as the size of a BSON document, without computing it in advance.
void overwrite(const std::size_t pos, const std::uint32_t n) noexcept
{
static_assert(sizeof(CharType) == 1, "overwrite() writes bytes");
JSON_ASSERT(pos + sizeof(n) <= position());
std::memcpy(v->data() + pos, &n, sizeof(n));
}
/// drop the characters written from position @a pos on
void rewind(const std::size_t pos) noexcept
{
JSON_ASSERT(pos <= position());
cursor = v->data() + pos;
}
private:
/// number of characters of *v written so far
std::size_t position() const noexcept
@@ -1977,23 +1977,6 @@ class serializer
return;
}
// 17 to 20 digits: the first one to four, then twice eight
if (magnitude_fits_64_bits(abs_value, std::integral_constant<bool, magnitude_has_at_most_64_bits>()))
{
const auto value = static_cast<std::uint64_t>(abs_value);
const std::uint64_t top = value / 10000000000000000u; // 1 to 1844
const std::uint64_t low16 = value - (top * 10000000000000000u);
const std::uint64_t middle = low16 / 100000000u;
const std::uint64_t first = dtoa_impl::eight_digit_bytes(top);
const auto leading = static_cast<unsigned>(count_leading_zeros(first) / 8); // (first is not 0)
char* const p = buffer_ptr + n_chars;
dtoa_impl::store_msb_first(p, (first << (8 * leading)) + 0x3030303030303030u);
dtoa_impl::store_msb_first(p + 8 - leading, dtoa_impl::eight_digit_bytes(middle) + 0x3030303030303030u);
dtoa_impl::store_msb_first(p + 16 - leading, dtoa_impl::eight_digit_bytes(low16 - (middle * 100000000u)) + 0x3030303030303030u);
write_buffer_pos += n_chars + 24 - leading;
return;
}
n_chars += count_digits(abs_value);
// spare 1 byte for '\0'
@@ -2139,23 +2122,6 @@ class serializer
}
}
/// whether every number_magnitude_t fits 64 bits
static constexpr bool magnitude_has_at_most_64_bits = std::numeric_limits<number_magnitude_t>::is_specialized
&& std::numeric_limits<number_magnitude_t>::digits <= 64;
/// whether @a x fits 64 bits: always if number_magnitude_t has at most
/// 64 bits (selected by type, as the comparison would be reported as
/// always true), else compared
static bool magnitude_fits_64_bits(const number_magnitude_t& /*unused*/, std::true_type /*unused*/) noexcept
{
return true;
}
static bool magnitude_fits_64_bits(const number_magnitude_t& x, std::false_type /*unused*/) noexcept
{
return x <= 18446744073709551615u;
}
/*
* Helper function for dump_integer
*
-1
View File
@@ -13,7 +13,6 @@
#include <cstddef> // size_t
#include <cstdint> // uint8_t, uint32_t
#include <string> // string, to_string
#include <utility> // move
#include <nlohmann/detail/abi_macros.hpp>
#include <nlohmann/detail/macro_scope.hpp>
+19 -136
View File
@@ -69,8 +69,7 @@ struct parse_failure
};
/// FloatType: the number_float_t of the document, whose overflow parse() rejects
/// Store: false for accept(): the grammar, number, string and UTF-8 checks run as for parse(), but no node is written
template<typename FloatType, bool Comments, bool TrailingCommas, bool NulIsEnd, bool Sentinel, bool Store>
template<typename FloatType, bool Comments, bool TrailingCommas, bool NulIsEnd, bool Sentinel>
class builder
{
public:
@@ -246,87 +245,11 @@ class builder
return p;
}
/// the string from p, the first backslash, control character, ill-formed UTF-8 byte, or the end, checked and not
/// decoded (accept() stores nothing, so the arena is not written): the checks, errors and error positions of
/// slow_string(), for everything up to the closing quote
NLOHMANN_VIEW_NOINLINE decoded check_string(const unsigned char* p)
{
const unsigned char* const end = m_end;
for (;;)
{
if (p == end)
{
return failed(error_code::string_missing_quote, p);
}
const unsigned char c = *p;
if (c == '"')
{
++p;
return decoded{p, 0, 0};
}
if (c != '\\')
{
return failed(c < 0x20 ? error_code::string_control_character : error_code::string_utf8, p);
}
++p;
if (p == end)
{
return failed(error_code::string_missing_quote, p);
}
const unsigned char d = *p++;
if (d == 'u')
{
const int cp = hex4(p); // (moves p past the four digits)
if (NLOHMANN_VIEW_UNLIKELY(cp < 0))
{
return failed(error_code::string_unicode_hex, p);
}
if (NLOHMANN_VIEW_UNLIKELY((cp & 0xF800) == 0xD800)) // a surrogate
{
if (cp >= 0xDC00)
{
return failed(error_code::string_surrogate_low, p);
}
if (end - p < 2 || p[0] != '\\' || p[1] != 'u')
{
return failed(error_code::string_surrogate_high, p);
}
p += 2;
const int lo = hex4(p); // (moves p past the four digits)
if (lo < 0)
{
return failed(error_code::string_unicode_hex, p);
}
if (lo < 0xDC00 || lo > 0xDFFF)
{
return failed(error_code::string_surrogate_high, p);
}
}
}
else if (!(d == '"' || d == '\\' || d == '/' || d == 'b' || d == 'f' || d == 'n' || d == 'r' || d == 't'))
{
--p;
return failed(error_code::string_escape, p);
}
if (p != end && *p == '\\')
{
continue; // (escapes in a row: no run of text between them)
}
p = scan_string_run(p, end); // the text up to the next backslash, quote, control character, or ill-formed UTF-8
}
}
/// s points to the start of the string, p to a backslash, a control
/// character, an ill-formed UTF-8 byte, or the end: decode the string into
/// the arena
NLOHMANN_VIEW_NOINLINE decoded slow_string(const unsigned char* s, const unsigned char* p)
{
if (!enabled(Store))
{
// (accept(): the string is checked, not decoded)
static_cast<void>(s);
return check_string(p);
}
// single-character escapes; 0: invalid (and 'u', handled separately)
static const std::array<char, 128> simple_escape =
{
@@ -420,21 +343,6 @@ class builder
}
arena_ensure(w, w_end, 4);
w = put_utf8(w, cp);
// A chain of unicode escapes ("\u3042\u3044\u3046", as Python's json.dumps writes non-ASCII text) is
// decoded here, without going round the loop above for each of them. Anything else (a surrogate, a
// bad digit, fewer than six bytes) leaves p at the backslash: the loop above handles it, and reports
// the same error at the same position as it would without this loop.
while (end - p >= 6 && p[0] == '\\' && p[1] == 'u')
{
const int next = hex_codepoint(p + 2);
if (next < 0 || (next & 0xF800) == 0xD800)
{
break;
}
p += 6;
arena_ensure(w, w_end, 4);
w = put_utf8(w, next);
}
}
else if (NLOHMANN_VIEW_LIKELY(d < 128 && simple_escape[d] != 0))
{
@@ -702,13 +610,10 @@ run_found:
NLOHMANN_VIEW_ALWAYS_INLINE bool run()
{
if (enabled(Store))
{
cold.m_doc.reserve(estimate_nodes(reinterpret_cast<const char*>(b), static_cast<std::size_t>(e - b)));
base = cold.m_doc.nodes;
out = base;
cap = base + cold.m_doc.node_capacity;
}
cold.m_doc.reserve(estimate_nodes(reinterpret_cast<const char*>(b), static_cast<std::size_t>(e - b)));
base = cold.m_doc.nodes;
out = base;
cap = base + cold.m_doc.node_capacity;
if (e - p >= 3 && p[0] == 0xEF && p[1] == 0xBB && p[2] == 0xBF)
{
@@ -892,10 +797,7 @@ close_object:
// an array pays nothing for this)
if (NLOHMANN_VIEW_UNLIKELY(cur_count >= document_data::index_min_members))
{
if (enabled(Store))
{
cold.note_large_object(cur_idx);
}
cold.note_large_object(cur_idx);
}
close_container:
@@ -1055,12 +957,6 @@ indent_done:
/// an integer's value); two stores on little-endian targets
NLOHMANN_VIEW_ALWAYS_INLINE node* push_node(node_kind k, std::uint8_t flags, std::uint16_t extra, std::size_t off, std::uint64_t second)
{
if (!enabled(Store))
{
// accept(): nothing is stored, so what computes the arguments is not run either. base and out stay
// null: their difference, which open() takes, is 0.
return out;
}
if (NLOHMANN_VIEW_UNLIKELY(out == cap))
{
const auto n = static_cast<std::size_t>(out - base);
@@ -1119,12 +1015,9 @@ indent_done:
NLOHMANN_VIEW_ALWAYS_INLINE void close()
{
if (enabled(Store))
{
node& n = base[cur_idx];
n.len = cur_count;
n.next = static_cast<std::uint32_t>(out - base) - cur_idx;
}
node& n = base[cur_idx];
n.len = cur_count;
n.next = static_cast<std::uint32_t>(out - base) - cur_idx;
if (--depth != 0)
{
if (NLOHMANN_VIEW_LIKELY(depth <= 64))
@@ -1189,7 +1082,7 @@ indent_done:
{
++p;
const unsigned char* const f0 = p;
p = skip_fraction_digits(p, e);
p = skip_digits(p, e);
if (NLOHMANN_VIEW_UNLIKELY(p == f0))
{
return fail(error_code::number_after_dot);
@@ -1299,44 +1192,34 @@ indent_done:
};
/// run the builder with compile-time options
template<typename FloatType, bool Comments, bool TrailingCommas, bool NulIsEnd, bool Store>
template<typename FloatType, bool Comments, bool TrailingCommas, bool NulIsEnd>
inline bool build_with(document_data& d, const char* src, std::size_t size, bool sentinel, parse_failure& failure)
{
if (sentinel)
{
builder<FloatType, Comments, TrailingCommas, NulIsEnd, true, Store> bld(d, src, size);
builder<FloatType, Comments, TrailingCommas, NulIsEnd, true> bld(d, src, size);
const bool ok = bld.run();
failure = bld.failure();
return ok;
}
builder<FloatType, Comments, TrailingCommas, NulIsEnd, false, Store> bld(d, src, size);
builder<FloatType, Comments, TrailingCommas, NulIsEnd, false> bld(d, src, size);
const bool ok = bld.run();
failure = bld.failure();
return ok;
}
/// sentinel: src[size] is readable and 0 (e.g. std::string); FloatType: the
/// number_float_t of the document; validate_only: accept(), only the result is
/// used
/// number_float_t of the document
template<typename FloatType, bool NulIsEnd>
inline bool build(document_data& d, const char* src, std::size_t size, bool comments, bool trailing_commas, bool sentinel, bool validate_only,
parse_failure& failure)
inline bool build(document_data& d, const char* src, std::size_t size, bool comments, bool trailing_commas, bool sentinel, parse_failure& failure)
{
if (validate_only && !comments && !trailing_commas)
{
// accept() of plain JSON: nothing is stored. The other option sets
// could skip the stores just as well; they keep the builder that
// stores only so that there are two more instantiations of the parse
// loop, not eight.
return build_with<FloatType, false, false, NulIsEnd, false>(d, src, size, sentinel, failure);
}
if (comments)
{
return trailing_commas ? build_with<FloatType, true, true, NulIsEnd, true>(d, src, size, sentinel, failure)
: build_with<FloatType, true, false, NulIsEnd, true>(d, src, size, sentinel, failure);
return trailing_commas ? build_with<FloatType, true, true, NulIsEnd>(d, src, size, sentinel, failure)
: build_with<FloatType, true, false, NulIsEnd>(d, src, size, sentinel, failure);
}
return trailing_commas ? build_with<FloatType, false, true, NulIsEnd, true>(d, src, size, sentinel, failure)
: build_with<FloatType, false, false, NulIsEnd, true>(d, src, size, sentinel, failure);
return trailing_commas ? build_with<FloatType, false, true, NulIsEnd>(d, src, size, sentinel, failure)
: build_with<FloatType, false, false, NulIsEnd>(d, src, size, sentinel, failure);
}
} // namespace view
+11 -11
View File
@@ -157,7 +157,7 @@ class editor
node* const o = own(object);
if (o->kind != node_kind::object && o->kind != node_kind::null)
{
throw_type_error(site_of(object), exception_id::subscript_wrong_type, "cannot use operator[] with a string argument with ", object.type_name());
throw_type_error(exception_id::subscript_wrong_type, "cannot use operator[] with a string argument with ", object.type_name());
}
check_utf8(key.data(), key.size());
const encoded e = encode(std::forward<V>(value));
@@ -208,9 +208,9 @@ class editor
node* const a = own(array);
if (a->kind != node_kind::array)
{
throw_type_error(site_of(array), exception_id::subscript_wrong_type, "cannot use operator[] with a numeric argument with ", array.type_name());
throw_type_error(exception_id::subscript_wrong_type, "cannot use operator[] with a numeric argument with ", array.type_name());
}
check_index(a, idx, a->len);
check_index(idx, a->len);
const encoded e = encode(std::forward<V>(value));
node* const slot = const_cast<node*>(nav::value(element_at<true>(m_doc, a, idx))); // NOLINT(cppcoreguidelines-pro-type-const-cast)
assign(slot, e, a, true);
@@ -225,7 +225,7 @@ class editor
node* const a = own(array);
if (a->kind != node_kind::array && a->kind != node_kind::null)
{
throw_type_error(site_of(array), exception_id::push_back_wrong_type, "cannot use push_back() with ", array.type_name());
throw_type_error(exception_id::push_back_wrong_type, "cannot use push_back() with ", array.type_name());
}
const encoded e = encode(std::forward<V>(value));
if (a->kind == node_kind::null)
@@ -249,9 +249,9 @@ class editor
node* const a = own(array);
if (a->kind != node_kind::array)
{
throw_type_error(site_of(array), exception_id::insert_wrong_type, "cannot use insert() with ", array.type_name());
throw_type_error(exception_id::insert_wrong_type, "cannot use insert() with ", array.type_name());
}
check_index(a, idx, a->len + 1);
check_index(idx, a->len + 1);
const encoded e = encode(std::forward<V>(value));
node* const slot = new_slot(e);
node* const h = block_of(m_doc, a, 1);
@@ -269,7 +269,7 @@ class editor
node* const o = own(object);
if (o->kind != node_kind::object)
{
throw_type_error(site_of(object), exception_id::erase_wrong_type, "cannot use erase() with ", object.type_name());
throw_type_error(exception_id::erase_wrong_type, "cannot use erase() with ", object.type_name());
}
for (const node* k = nav::first(m_doc, o), *end = nav::end(m_doc, o); k != end; k = document_data::after(k + 1))
{
@@ -287,9 +287,9 @@ class editor
node* const a = own(array);
if (a->kind != node_kind::array)
{
throw_type_error(site_of(array), exception_id::erase_wrong_type, "cannot use erase() with ", array.type_name());
throw_type_error(exception_id::erase_wrong_type, "cannot use erase() with ", array.type_name());
}
check_index(a, idx, a->len);
check_index(idx, a->len);
node* const h = block_of(m_doc, a, 0);
std::memmove(h + 1 + idx, h + 2 + idx, (h->next - 2 - idx) * sizeof(node));
--h->next;
@@ -316,11 +316,11 @@ class editor
return const_cast<node*>(v.m_node); // NOLINT(cppcoreguidelines-pro-type-const-cast): the nodes belong to this document
}
void check_index(const node* array, std::size_t idx, std::size_t limit) const
static void check_index(std::size_t idx, std::size_t limit)
{
if (idx >= limit)
{
throw_array_index_out_of_range(error_site(&m_doc, array), idx);
throw_array_index_out_of_range(idx);
}
}
+10 -328
View File
@@ -10,15 +10,11 @@
#include <algorithm> // min
#include <cstddef> // size_t
#include <string> // string, to_string
#include <vector> // vector
#include <string> // string
#include <nlohmann/json.hpp>
#include <nlohmann/detail/view/builder.hpp>
#include <nlohmann/detail/view/document_data.hpp>
#include <nlohmann/detail/view/macro_scope.hpp>
#include <nlohmann/detail/view/materialize.hpp>
#include <nlohmann/detail/view/node.hpp>
NLOHMANN_JSON_NAMESPACE_BEGIN
namespace detail
@@ -26,293 +22,8 @@ namespace detail
namespace view
{
/*!
@brief the value an exception is about
basic_json passes the value to the exception, which then names its path
(JSON_DIAGNOSTICS) and its bytes in the source (JSON_DIAGNOSTIC_POSITIONS). A
view's value has no parent pointer and no stored positions: both are found
from the document and the node, when the exception is thrown (so only the
error path pays for them). Without the two macros, a site is empty.
*/
class error_site
{
public:
error_site() noexcept = default;
#if JSON_DIAGNOSTICS || JSON_DIAGNOSTIC_POSITIONS
error_site(const document_data* d, const node* n) noexcept
: m_doc(d), m_node(n)
{}
const document_data* doc() const noexcept
{
return m_doc;
}
const node* value() const noexcept
{
return m_node;
}
private:
const document_data* m_doc = nullptr;
const node* m_node = nullptr;
#else
error_site(const document_data* /*d*/, const node* /*n*/) noexcept {}
#endif
};
/// the site of a view's value (the views make this a friend)
struct view_access
{
template<typename View>
static error_site site(const View& v) noexcept
{
return {v.m_doc, v.m_node};
}
};
template<typename View>
error_site site_of(const View& v) noexcept
{
static_assert(is_json_view<View>::value, "the site of a value is only defined for views");
return view_access::site(v);
}
#if JSON_DIAGNOSTICS
/*!
@brief the JSON pointer tokens from the root of document @a d to node @a target
Nodes have no parent index, so the path is found by descending from the root:
before the first edit, only into the subtree that contains the target (a
container spans the next `next` nodes); after edits, elements can live in
other sequences, so every container is searched. Iterative (the nesting depth
is limited by memory only). False if the target is not found (then tokens is
unspecified).
*/
template<bool Editable>
bool value_path(const document_data& d, const node* target, std::vector<std::string>& tokens)
{
using nav = navigation<Editable>;
struct frame
{
const node* pos;
const node* end;
std::size_t index;
bool object;
};
const node* const root = d.nodes;
if (target == root)
{
return true;
}
if (!is_container(*root))
{
return false;
}
std::vector<frame> stack;
stack.push_back(frame{nav::first(d, root), nav::end(d, root), 0, root->kind == node_kind::object});
// (tokens[i] leads from stack[i] to stack[i + 1], or to the target)
for (;;)
{
// a copy: pop_back() and push_back() below change the stack
const frame f = stack.back();
if (f.pos == f.end)
{
stack.pop_back();
if (stack.empty())
{
return false;
}
tokens.pop_back();
continue;
}
const node* entry = f.pos;
std::string token;
if (f.object)
{
// (as basic_json: the key up to a null byte)
token.assign(d.str(*entry), entry->len);
const std::size_t nul = token.find('\0');
if (nul != std::string::npos)
{
token.resize(nul);
}
++entry;
}
else
{
token = std::to_string(f.index);
}
const node* const child = nav::value(entry);
if (child == target)
{
tokens.push_back(token);
return true;
}
stack.back().pos = document_data::after(entry);
++stack.back().index;
if (is_container(*child) && (nav::editable() || (target > child && target < child + child->next)))
{
tokens.push_back(token);
stack.push_back(frame{nav::first(d, child), nav::end(d, child), 0, child->kind == node_kind::object});
}
}
}
#endif
#if JSON_DIAGNOSTIC_POSITIONS
/*!
@brief the bytes [start, end) of node @a n in the source, as materialize() sets them
Scalars as position_source::set_scalar; an array or object from its opening
bracket to its closing bracket, which is found by following the chain of last
elements to a scalar (or an empty container) and scanning from there to the
closing brackets. None (npos) for values written by edits, and for containers
whose last element has none, as with materialize().
*/
template<bool Editable>
void value_range(const document_data& d, const node* n, std::size_t& start, std::size_t& end)
{
using nav = navigation<Editable>;
position_source p;
start = std::string::npos;
end = std::string::npos;
if (!is_container(*n))
{
p.set_scalar(d, *n);
start = p.start;
end = p.end;
return;
}
if ((n->flags & node_flags::is_new) != 0 || n->off >= d.size)
{
return;
}
// the open brackets: n, then the last element as long as it is a non-empty container
std::size_t closing = 0;
std::size_t pos = std::string::npos;
for (const node* c = n;;)
{
if (!is_container(*c))
{
p.set_scalar(d, *c);
pos = p.end;
break;
}
if ((c->flags & node_flags::is_new) != 0 || c->off >= d.size)
{
return;
}
++closing;
const node* const first = nav::first(d, c);
const node* const last_end = nav::end(d, c);
if (first == last_end)
{
pos = std::size_t{c->off} + 1;
break;
}
const std::size_t value_offset = c->kind == node_kind::object ? 1 : 0;
const node* last = first;
for (const node* e = document_data::after(first + value_offset); e != last_end; e = document_data::after(e + value_offset))
{
last = e;
}
c = nav::value(last + value_offset);
}
for (; closing != 0 && pos != std::string::npos; --closing)
{
pos = position_source::container_end(d.base[0], d.size, pos);
}
if (pos != std::string::npos)
{
start = n->off;
end = pos;
}
}
#endif
/// the context basic_json puts in front of the message of an exception about
/// a value: "(/path) " with JSON_DIAGNOSTICS (none for the root), then
/// "(bytes a-b) " with JSON_DIAGNOSTIC_POSITIONS (with JSON_DIAGNOSTICS, only
/// if there is a path)
///
/// This and the throw helpers that take an error_site are templates, so that
/// the path and range code is instantiated only where an exception can be
/// thrown: instantiated in every translation unit (as a non-template inline
/// function makes it), it changed GCC's early inlining of unrelated view
/// functions, so that GCC 13 no longer inlined a recursive walk over a view
/// into itself (2.7 times the instructions).
#if JSON_DIAGNOSTICS || JSON_DIAGNOSTIC_POSITIONS
template<typename Unused = void>
std::string diagnostics(const error_site site)
{
if (site.doc() == nullptr || site.value() == nullptr)
{
return "";
}
const document_data& d = *site.doc();
const bool edited = d.edits != nullptr;
std::string context;
#if JSON_DIAGNOSTICS
std::vector<std::string> tokens;
const bool found = edited ? value_path<true>(d, site.value(), tokens) : value_path<false>(d, site.value(), tokens);
if (!found || tokens.empty())
{
// context is still empty; returning it keeps the copy elided (-Wnrvo)
return context;
}
std::string path;
for (const std::string& token : tokens)
{
path.push_back('/');
path += detail::escape(token);
}
context = concat('(', path, ") ");
#endif
#if JSON_DIAGNOSTIC_POSITIONS
std::size_t start = 0;
std::size_t end = 0;
if (edited)
{
value_range<true>(d, site.value(), start, end);
}
else
{
value_range<false>(d, site.value(), start, end);
}
if (start != std::string::npos && end != std::string::npos)
{
context += concat("(bytes ", std::to_string(start), "-", std::to_string(end), ") ");
}
#endif
return context;
}
#else
template<typename Unused = void>
std::string diagnostics(const error_site /*site*/)
{
return "";
}
#endif
/// the context of an exception about view @a v (see exception_prefix())
template<typename View>
std::string diagnostics_of(const View& v)
{
return diagnostics<>(site_of(v));
}
// Exceptions are thrown out of line, so that the accessors that may throw stay
// small enough to be inlined. The overloads with an error_site put its context
// in front of the message.
//
// The overloads without one are for exceptions that are not about a value of a
// document: limits and invalid input of the editing and image functions, a
// discarded view (it has no node), a view of another document (its path would
// be one in that document), and those that basic_json creates without a
// context too (the errors of json_pointer, invalid UTF-8). The one exception
// about a value that has no site is key(), see view_iterator::key().
// small enough to be inlined.
[[noreturn]] NLOHMANN_VIEW_NOINLINE inline void throw_type_error(exception_id id, const char* prefix, const char* type)
{
@@ -334,24 +45,6 @@ std::string diagnostics_of(const View& v)
NLOHMANN_VIEW_THROW(invalid_iterator::create(id, msg, nullptr));
}
template<typename Unused = void>
[[noreturn]] NLOHMANN_VIEW_NOINLINE void throw_type_error(const error_site site, exception_id id, const char* prefix, const char* type)
{
NLOHMANN_VIEW_THROW(type_error::create(id, concat(diagnostics<>(site), prefix, type), nullptr));
}
template<typename Unused = void>
[[noreturn]] NLOHMANN_VIEW_NOINLINE void throw_out_of_range(const error_site site, exception_id id, const std::string& msg)
{
NLOHMANN_VIEW_THROW(out_of_range::create(id, concat(diagnostics<>(site), msg), nullptr));
}
template<typename Unused = void>
[[noreturn]] NLOHMANN_VIEW_NOINLINE void throw_invalid_iterator(const error_site site, exception_id id, const char* msg)
{
NLOHMANN_VIEW_THROW(invalid_iterator::create(id, concat(diagnostics<>(site), msg), nullptr));
}
/// a parse error without a position (as those of json_pointer)
[[noreturn]] NLOHMANN_VIEW_NOINLINE inline void throw_parse_error(exception_id id, const std::string& msg)
{
@@ -360,23 +53,22 @@ template<typename Unused = void>
/// out_of_range.401: the array index @a idx is not below the size
template<typename Index>
[[noreturn]] NLOHMANN_VIEW_NOINLINE void throw_array_index_out_of_range(const error_site site, const Index idx)
[[noreturn]] NLOHMANN_VIEW_NOINLINE void throw_array_index_out_of_range(const Index idx)
{
throw_out_of_range(site, exception_id::array_index_out_of_range, concat("array index ", std::to_string(idx), " is out of range"));
throw_out_of_range(exception_id::array_index_out_of_range, concat("array index ", std::to_string(idx), " is out of range"));
}
/// out_of_range.402: the JSON Pointer token '-' of an array of @a size elements
template<typename Unused = void>
[[noreturn]] NLOHMANN_VIEW_NOINLINE void throw_array_index_past_end(const error_site site, std::size_t size)
[[noreturn]] NLOHMANN_VIEW_NOINLINE inline void throw_array_index_past_end(std::size_t size)
{
throw_out_of_range(site, exception_id::pointer_past_the_end_index, concat("array index '-' (", std::to_string(size), ") is out of range"));
throw_out_of_range(exception_id::pointer_past_the_end_index, concat("array index '-' (", std::to_string(size), ") is out of range"));
}
/// out_of_range.403: the object has no member @a key
template<typename Key>
[[noreturn]] NLOHMANN_VIEW_NOINLINE void throw_key_not_found(const error_site site, const Key& key)
[[noreturn]] NLOHMANN_VIEW_NOINLINE void throw_key_not_found(const Key& key)
{
throw_out_of_range(site, exception_id::key_not_found, concat("key '", key, "' not found"));
throw_out_of_range(exception_id::key_not_found, concat("key '", key, "' not found"));
}
/// SAX handler of the error replay: accepts every event without storing it,
@@ -452,21 +144,11 @@ struct throwing_sax
/// (a template, as in json_sax_dom_parser: the parser passes the concrete
/// exception type, which is thrown as is)
///
/// It is not [[noreturn]]: sax_parse() recovers from an error if
/// parse_error() returns true, and MSVC reports that code as unreachable
/// (C4702) for a handler that never returns. The flag hides that from it.
template<class Exception>
bool parse_error(std::size_t /*unused*/, const std::string& /*unused*/, const Exception& ex)
[[noreturn]] bool parse_error(std::size_t /*unused*/, const std::string& /*unused*/, const Exception& ex)
{
if (throwing)
{
NLOHMANN_VIEW_THROW(ex);
}
return false; // LCOV_EXCL_LINE
NLOHMANN_VIEW_THROW(ex);
}
bool throwing = true;
};
/*!
+1 -4
View File
@@ -105,10 +105,7 @@ class view_iterator
}
/// the key of the current object member; throws invalid_iterator.207 for
/// other iterators, like basic_json's iterators (but without the context
/// JSON_DIAGNOSTICS and JSON_DIAGNOSTIC_POSITIONS add: the iterator does not
/// know the value it iterates over, and keeping it would make iterators
/// larger, which keeps GCC from inlining recursive walks)
/// other iterators, like basic_json's iterators
string_view_t key() const
{
if (NLOHMANN_VIEW_UNLIKELY(m_value_offset == 0))
@@ -40,15 +40,6 @@
#define NLOHMANN_VIEW_NOINLINE
#endif
// GCC: a function compiled as written, without the clones in which interprocedural
// constant propagation drops an argument that every caller passes as the same constant
// (calling such a clone changes the code GCC generates for the parse loop that calls it)
#if defined(__GNUC__) && !defined(__clang__)
#define NLOHMANN_VIEW_NOCLONE __attribute__((noclone))
#else
#define NLOHMANN_VIEW_NOCLONE
#endif
#if defined(__GNUC__) || defined(__clang__)
#define NLOHMANN_VIEW_NODISCARD __attribute__((warn_unused_result))
#elif defined(_MSC_VER)
@@ -15,7 +15,6 @@
#undef NLOHMANN_VIEW_UNLIKELY
#undef NLOHMANN_VIEW_ALWAYS_INLINE
#undef NLOHMANN_VIEW_NOINLINE
#undef NLOHMANN_VIEW_NOCLONE
#undef NLOHMANN_VIEW_NODISCARD
#undef NLOHMANN_VIEW_THROW
#undef NLOHMANN_VIEW_LITTLE_ENDIAN
+8 -163
View File
@@ -19,7 +19,6 @@
#include <nlohmann/detail/view/macro_scope.hpp>
#include <nlohmann/detail/view/node.hpp>
#include <nlohmann/detail/view/scan.hpp>
#include <nlohmann/detail/view/simd.hpp>
NLOHMANN_JSON_NAMESPACE_BEGIN
namespace detail
@@ -104,56 +103,12 @@ NLOHMANN_VIEW_NOINLINE FloatType float_value(const char* first, const node& n)
return convert_float<FloatType>(first, last, dot, mantissa_end);
}
#if NLOHMANN_VIEW_VECTOR
/*!
@brief the value of the first 16 digits of a run of digits with a point after
int_digits of them (or none, if int_digits >= 16)
The 17 bytes at p are read at once: the bytes after the point move down by one,
then pairs, quadruples, and octets of digits are combined by multiplications
(after simdjson). The bytes [p, p + 32) must be readable. Bytes that are not
digits (only in a damaged image loaded with image_check::bounds) give a wrong
value, but no undefined behavior.
*/
NLOHMANN_VIEW_ALWAYS_INLINE std::uint64_t sixteen_digits(const unsigned char* p, unsigned int_digits) noexcept
{
#if NLOHMANN_VIEW_NEON
const uint8x16_t low = vld1q_u8(p);
const uint8x16_t index = vcombine_u8(vcreate_u8(0x0706050403020100u), vcreate_u8(0x0F0E0D0C0B0A0908u));
const uint8x16_t before_point = vcltq_u8(index, vdupq_n_u8(static_cast<std::uint8_t>(int_digits)));
const uint8x16_t digits = vsubq_u8(vbslq_u8(before_point, low, vextq_u8(low, vld1q_u8(p + 16), 1)), vdupq_n_u8('0'));
const uint16x8_t twos = vpaddlq_u8(vmulq_u8(digits, vreinterpretq_u8_u16(vdupq_n_u16(0x010A))));
const uint32x4_t fours = vpaddlq_u16(vmulq_u16(twos, vreinterpretq_u16_u32(vdupq_n_u32(0x00010064))));
const uint64x2_t eights = vpaddlq_u32(vmulq_u32(fours, vreinterpretq_u32_u64(vdupq_n_u64(0x0000000100002710u))));
return (vgetq_lane_u64(eights, 0) * 100000000u) + vgetq_lane_u64(eights, 1);
#else
// NOLINTBEGIN(portability-simd-intrinsics)
const __m128i low = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(p))); // NOLINT(bugprone-casting-through-void)
const __m128i high = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(p + 16))); // NOLINT(bugprone-casting-through-void)
const __m128i before_point = _mm_cmpgt_epi8(_mm_set1_epi8(static_cast<char>(int_digits)), _mm_setr_epi8(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15));
const __m128i moved = _mm_or_si128(_mm_srli_si128(low, 1), _mm_slli_si128(high, 15));
const __m128i digits = _mm_sub_epi8(_mm_or_si128(_mm_and_si128(before_point, low), _mm_andnot_si128(before_point, moved)), _mm_set1_epi8('0'));
// (16-bit lanes: SSE2 has no multiply-add of bytes)
const __m128i tens = _mm_setr_epi16(10, 1, 10, 1, 10, 1, 10, 1);
const __m128i twos = _mm_packs_epi32(_mm_madd_epi16(_mm_unpacklo_epi8(digits, _mm_setzero_si128()), tens),
_mm_madd_epi16(_mm_unpackhi_epi8(digits, _mm_setzero_si128()), tens));
const __m128i fours = _mm_madd_epi16(twos, _mm_setr_epi16(100, 1, 100, 1, 100, 1, 100, 1));
const __m128i eights = _mm_madd_epi16(_mm_packs_epi32(fours, fours), _mm_setr_epi16(10000, 1, 10000, 1, 10000, 1, 10000, 1));
const auto upper = static_cast<std::uint32_t>(_mm_cvtsi128_si32(eights));
const auto lower = static_cast<std::uint32_t>(_mm_cvtsi128_si32(_mm_srli_si128(eights, 4)));
// NOLINTEND(portability-simd-intrinsics)
return (std::uint64_t{upper} * 100000000u) + lower;
#endif
}
#endif
/*!
@brief the digits of a float token with at most 19 digits, from its layout
The digit layout recorded while parsing says where the integer digits, the
fraction digits, and the exponent are, so the digits are read eight at a
time without scanning (with SIMD, a token of 16 to 19 digits sixteen at a
time).
time without scanning.
@param[in] p first character of the token
@param[in] e end of the token
@@ -163,34 +118,15 @@ NLOHMANN_VIEW_ALWAYS_INLINE float_significand layout_decimal(const unsigned char
{
const bool negative = *p == '-';
p += negative ? 1 : 0;
std::uint64_t w = 0;
std::uint64_t w = parse_upto19(p, int_digits, limit);
p += int_digits;
std::int64_t q = 0;
#if NLOHMANN_VIEW_VECTOR
const unsigned count = int_digits + frac_digits;
if (count >= 16 && limit - p >= 32)
if (frac_digits != 0)
{
// (the point, if any, follows the integer digits; the digits after
// the first 16 are at most three)
w = sixteen_digits(p, int_digits);
for (unsigned i = 16; i < count; ++i)
{
w = (w * 10) + static_cast<std::uint64_t>(p[i + (i >= int_digits ? 1 : 0)] - '0');
}
p += count + (frac_digits != 0 ? 1 : 0);
w = (w * int_pow10(frac_digits)) + parse_upto19(p + 1, frac_digits, limit);
p += 1 + frac_digits;
q = -static_cast<std::int64_t>(frac_digits);
}
else /* NOLINT(readability/braces) */
#endif
{
w = parse_upto19(p, int_digits, limit);
p += int_digits;
if (frac_digits != 0)
{
w = (w * int_pow10(frac_digits)) + parse_upto19(p + 1, frac_digits, limit);
p += 1 + frac_digits;
q = -static_cast<std::int64_t>(frac_digits);
}
}
if (p != e)
{
// [eE][+-]digits; huge exponents saturate (the parser rejected
@@ -225,22 +161,12 @@ NLOHMANN_VIEW_ALWAYS_INLINE float_significand layout_decimal(const unsigned char
The result is correctly rounded by the lexer's conversion
(detail::decimal_to_float(): Clinger's fast path where both operands are
exact, else the Eisel-Lemire algorithm, which needs no fallback for up to 19
digits), so it is the value parse() produces. Tokens of 16 or more digits go
to Eisel-Lemire directly: the fast path takes few of them (none of 17 digits),
and its check is a branch that data like GeoJSON coordinates mispredicts.
digits), so it is the value parse() produces.
*/
template<typename FloatType>
NLOHMANN_VIEW_ALWAYS_INLINE FloatType layout_float(const unsigned char* p, const unsigned char* e, unsigned int_digits, unsigned frac_digits, const unsigned char* limit) noexcept
{
const float_significand d = layout_decimal(p, e, int_digits, frac_digits, limit);
if (int_digits + frac_digits >= 16)
{
using format = ieee_binary_format<std::numeric_limits<FloatType>::digits>;
native_float_t<FloatType> value{};
float_from_bits(eisel_lemire<format>(d.exponent, d.w) | (d.negative ? (std::uint64_t{1} << format::sign_bit()) : 0u), value);
return value;
}
return decimal_to_float<FloatType>(d);
return decimal_to_float<FloatType>(layout_decimal(p, e, int_digits, frac_digits, limit));
}
/// the value of a float set by an edit: its token (the shortest round-trip
@@ -259,80 +185,6 @@ NLOHMANN_VIEW_NOINLINE FloatType edited_float(const char* token, const node& n)
return float_value<FloatType>(token, n);
}
#if NLOHMANN_VIEW_VECTOR
/// eisel_lemire() for the token w * 10^q of coordinate_float()
template<typename Format>
NLOHMANN_VIEW_ALWAYS_INLINE std::uint64_t coordinate_bits(std::int64_t q, std::uint64_t w, Format /*format*/) noexcept
{
return eisel_lemire<Format>(q, w);
}
/*!
@brief (binary64) eisel_lemire() without the cases that tokens of 16 or 17
digits with 5 or more after the point cannot reach: a value of 0, a
subnormal, an infinity, or a tie (q < -4)
*/
NLOHMANN_VIEW_ALWAYS_INLINE std::uint64_t coordinate_bits(std::int64_t q, std::uint64_t w, ieee_binary_format<53> format) noexcept
{
if (NLOHMANN_VIEW_UNLIKELY(w == 0 || q > -5))
{
return eisel_lemire<ieee_binary_format<53>>(q, w);
}
static_cast<void>(format);
const int lz = count_leading_zeros(w);
const std::uint64_t normalized = w << static_cast<unsigned>(lz);
const auto index = static_cast<std::size_t>(2 * (q - pow5_128_smallest_power));
const uint128_parts product = full_multiplication(normalized, pow5_128()[index]);
std::uint64_t high = product.high;
if (NLOHMANN_VIEW_UNLIKELY((high & 0x1FFu) == 0x1FFu))
{
// (the lower bits may carry into the result)
const std::uint64_t second = full_multiplication(normalized, pow5_128()[index + 1]).high;
high += product.low + second < product.low ? 1u : 0u;
}
const auto upperbit = static_cast<unsigned>(high >> 63u);
std::uint64_t mantissa = high >> (upperbit + 9);
auto power2 = static_cast<std::uint64_t>(((217706 * q) >> 16) + 63 + static_cast<std::int64_t>(upperbit) - lz + 1023);
mantissa = (mantissa + (mantissa & 1u)) >> 1u;
const bool carry = mantissa >= (std::uint64_t{2} << 52u);
mantissa = carry ? (std::uint64_t{1} << 52u) : mantissa;
power2 += carry ? 1u : 0u;
return (mantissa & ~(std::uint64_t{1} << 52u)) | (power2 << 52u);
}
/*!
@brief (binary32 and binary64) the value of a float token of the source with
16 or 17 digits, a point, and no exponent, the shape of most coordinates
written with full precision; whether the token of n has this shape
These tokens skip the general layout of layout_decimal(): no digits beyond
the 17th, no exponent (and Eisel-Lemire directly, as in layout_float()).
*/
template<typename FloatType>
NLOHMANN_VIEW_ALWAYS_INLINE bool coordinate_float(const document_data& d, const node& n, FloatType& value) noexcept
{
const unsigned int_digits = n.extra & 0xFFu;
const unsigned frac_digits = n.extra >> 8u;
const unsigned count = int_digits + frac_digits;
const auto* const first = reinterpret_cast<const unsigned char*>(d.src + n.off); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
const unsigned negative = first[0] == '-' ? 1u : 0u;
if (count - 16u > 1u || frac_digits == 0 || n.len != negative + count + 1 || d.size - n.off < 33)
{
return false;
}
const unsigned char* const digits = first + negative;
const std::uint64_t w = sixteen_digits(digits, int_digits);
// (the 17th digit follows the point)
const std::uint64_t w17 = (w * 10) + static_cast<std::uint64_t>(digits[17] - '0');
using format = ieee_binary_format<std::numeric_limits<FloatType>::digits>;
native_float_t<FloatType> result{};
float_from_bits(coordinate_bits(-static_cast<std::int64_t>(frac_digits), count == 17 ? w17 : w, format{}) | (static_cast<std::uint64_t>(negative) << format::sign_bit()), result);
value = result;
return true;
}
#endif
/// the value of the float token of a node, as parse() converts it; floats and
/// doubles with at most 19 digits are converted from the digit layout
template<typename FloatType>
@@ -348,13 +200,6 @@ FloatType float_value(const document_data& d, const node& n)
template<typename FloatType>
FloatType float_value(const document_data& d, const node& n, std::true_type /*binary32 or binary64*/)
{
#if NLOHMANN_VIEW_VECTOR
FloatType value {};
if (coordinate_float(d, n, value))
{
return value;
}
#endif
const unsigned int_digits = n.extra & 0xFFu;
const unsigned frac_digits = n.extra >> 8u;
if (NLOHMANN_VIEW_LIKELY(int_digits + frac_digits <= 19)) // (255 marks "many")
+4 -5
View File
@@ -44,7 +44,6 @@ index_status array_index(const StringType& s, std::size_t& idx) noexcept
}
/// throws the exception json_pointer::array_index throws for this status
/// (without a context, as json_pointer::array_index)
template<typename StringType>
[[noreturn]] NLOHMANN_VIEW_NOINLINE void throw_array_index_error(index_status status, const StringType& s)
{
@@ -84,7 +83,7 @@ View resolve_pointer(View cur, const Tokens& tokens, pointer_mode mode)
{
if (mode == pointer_mode::checked)
{
throw_key_not_found(site_of(cur), token);
throw_key_not_found(token);
}
return View();
}
@@ -96,7 +95,7 @@ View resolve_pointer(View cur, const Tokens& tokens, pointer_mode mode)
{
if (throwing)
{
throw_array_index_past_end(site_of(cur), cur.size());
throw_array_index_past_end(cur.size());
}
return View();
}
@@ -115,7 +114,7 @@ View resolve_pointer(View cur, const Tokens& tokens, pointer_mode mode)
{
if (mode == pointer_mode::checked)
{
throw_array_index_out_of_range(site_of(cur), idx);
throw_array_index_out_of_range(idx);
}
return View();
}
@@ -125,7 +124,7 @@ View resolve_pointer(View cur, const Tokens& tokens, pointer_mode mode)
{
if (throwing)
{
throw_out_of_range(site_of(cur), exception_id::pointer_unresolved, concat("unresolved reference token '", token, "'"));
throw_out_of_range(exception_id::pointer_unresolved, concat("unresolved reference token '", token, "'"));
}
return View();
}
-51
View File
@@ -198,57 +198,6 @@ inline const unsigned char* find_block_comment_end(const unsigned char* p, const
return p;
}
/// The digits of a fraction, whose number varies from one number to the next: skip_digits() with 16 bytes compared at
/// once and the end of the run taken from a bit mask (SSE2), instead of a compare and a branch for each digit. That
/// is about a third of the instructions for a fraction of 12 digits, and no branch that is mispredicted when the
/// lengths vary. Where there is no SSE2, skip_digits() is used.
#if NLOHMANN_VIEW_SSE2
// The last bytes of the input, one at a time. GCC unrolls this loop in every inlined copy of the function below, which
// makes the parse loop 4 KiB larger and slows down documents that have hardly any fractions; out of line, it does
// not. Clang does better with the loop inline (a call in the parse loop costs it more than the unrolling).
#if defined(__GNUC__) && !defined(__clang__)
inline NLOHMANN_VIEW_NOINLINE const unsigned char* skip_digits_tail(const unsigned char* p, const unsigned char* e) noexcept
{
while (p != e && is_digit(*p))
{
++p;
}
return p;
}
#else
NLOHMANN_VIEW_ALWAYS_INLINE const unsigned char* skip_digits_tail(const unsigned char* p, const unsigned char* e) noexcept
{
while (p != e && is_digit(*p))
{
++p;
}
return p;
}
#endif
NLOHMANN_VIEW_ALWAYS_INLINE const unsigned char* skip_fraction_digits(const unsigned char* p, const unsigned char* e) noexcept
{
while (e - p >= 16)
{
const __m128i in = _mm_loadu_si128(static_cast<const __m128i*>(static_cast<const void*>(p))); // NOLINT(bugprone-casting-through-void)
// a digit is above '/' and below ':' (signed compares: a byte with the top bit set is negative, and neither)
const __m128i digits = _mm_and_si128(_mm_cmpgt_epi8(in, _mm_set1_epi8('/')), _mm_cmplt_epi8(in, _mm_set1_epi8(':')));
const auto not_digit = static_cast<std::uint64_t>(~static_cast<unsigned>(_mm_movemask_epi8(digits)) & 0xFFFFu);
if (not_digit != 0)
{
return p + count_trailing_zeros(not_digit);
}
p += 16;
}
return skip_digits_tail(p, e);
}
#else
NLOHMANN_VIEW_ALWAYS_INLINE const unsigned char* skip_fraction_digits(const unsigned char* p, const unsigned char* e) noexcept
{
return skip_digits(p, e);
}
#endif
/// powers of ten up to 10^19 as integers
inline std::uint64_t int_pow10(unsigned k) noexcept
{
+1 -403
View File
@@ -14,9 +14,7 @@
#include <cstddef> // size_t
#include <cstdint> // uint8_t, uint32_t
#include <cstring> // memcpy, memset
#include <functional> // less
#include <limits> // numeric_limits
#include <memory> // unique_ptr
#include <type_traits> // integral_constant
#include <vector> // vector
@@ -386,143 +384,6 @@ struct depth_counter<false>
}
};
/*!
@brief where the double x nearest to a token w * 10^q lies, for 10^15 <= w <
10^17 and -17 <= q <= -1 (Eisel-Lemire, then a fixed-point position)
dump() writes the shortest decimal in the rounding interval of x (the closest
one if there are several, see zmij::to_shortest()). Its digits are those of a
multiple of 10^g of the token's last digit, with g = dec_exp + 1 - q for x's
decimal exponent dec_exp: the multiple of 10^g nearest to x if it lies in the
interval (at most one does), else the multiple of 10^(g - 1) nearest to x
(see token_digits_1() and token_digits_2()). For g <= 0, the token itself is
in the interval.
The position is exact in a 64-bit fixed-point number with 56 bits after the
point (of the token's last digit): x / 10^q = 2c * 5^-q / 2^t for x = c * 2^e2
and t = q + 1 - e2, with at most 56 bits after the point here, and |x / 10^q -
w| is at most half the spacing of the doubles, below 12 units. So only the low
64 bits of 2c * 5^-q are needed. c and e2 come from Eisel-Lemire's product
directly (as eisel_lemire() rounds it: no tie is possible for q < -4, and its
second product is added where it can carry into the result).
@param[out] offset x / 10^q - w + 32, with 56 bits after the point
@param[out] radius half the spacing of the doubles at x, likewise (+ 1 if
the interval is closed)
@return g; 3 if the digits are not decided here (a power of two, whose
interval is not symmetric, or out of range)
*/
NLOHMANN_VIEW_ALWAYS_INLINE int locate_token(std::uint64_t w, int q, std::uint64_t& offset, std::uint64_t& radius) noexcept
{
const int lz = count_leading_zeros(w);
const std::uint64_t normalized = w << static_cast<unsigned>(lz);
const auto index = static_cast<std::size_t>(2 * (q - pow5_128_smallest_power));
const uint128_parts product = full_multiplication(normalized, pow5_128()[index]);
std::uint64_t high = product.high;
if (NLOHMANN_VIEW_UNLIKELY((high & 0x1FFu) == 0x1FFu))
{
// (as eisel_lemire(): the lower bits may carry into the result)
const std::uint64_t second = full_multiplication(normalized, pow5_128()[index + 1]).high;
high += product.low + second < product.low ? 1u : 0u;
}
const auto upperbit = static_cast<unsigned>(high >> 63u);
const std::uint64_t mantissa = high >> (upperbit + 9);
const std::uint64_t c = (mantissa + (mantissa & 1u)) >> 1u;
// e2 as eisel_lemire() computes the exponent (log2(10) ~ 217706 / 2^16)
const int e2 = static_cast<int>((217706 * static_cast<std::int64_t>(q)) >> 16) + 63 + static_cast<int>(upperbit) - lz - 52;
const int t = q + 1 - e2;
const int g = ((e2 * 315653) >> 20) + 1 - q; // (dec_exp as in zmij::to_shortest(), for a regular x)
static const std::array<std::uint64_t, 17> fives = {{1u, 5u, 25u, 125u, 625u, 3125u, 15625u, 78125u, 390625u, 1953125u, 9765625u, 48828125u, 244140625u, 1220703125u, 6103515625u, 30517578125u, 152587890625u}};
const std::uint64_t five = fives[static_cast<std::size_t>(-q)]; // 5^-q
const unsigned s = static_cast<unsigned>(56 - t) & 63u;
offset = ((2 * c * five) << s) - (w << 56u) + (std::uint64_t{32} << 56u);
radius = (five << s) + ((c & 1u) == 0 ? 1u : 0u);
// (c = 2^52: a power of two; c = 2^53: rounded up to one. With 8 to 16
// fraction digits, as the batch's tokens have, t is in [15, 40] and g at
// most 2. x lies halfway between two candidates only if its distance from
// the token is a whole unit (g == 2) or a half (g == 1): such tokens are
// left to the full conversion)
const bool decided = c - ((std::uint64_t{1} << 52u) + 1) < (std::uint64_t{1} << 52u) - 1
&& (offset & ((std::uint64_t{1} << 55u) - 1)) != 0;
return decided ? g : 3;
}
/// all bits set if b, else none
NLOHMANN_VIEW_ALWAYS_INLINE std::uint64_t mask_if(bool b) noexcept
{
return std::uint64_t{0} - static_cast<std::uint64_t>(b);
}
/// a if b, else c, without a branch (the conditions of token_digits_1/2() are
/// as good as random, and GCC compiled their ?: as branches: one misprediction
/// per float of canada.json on x86-64)
NLOHMANN_VIEW_ALWAYS_INLINE std::uint64_t select_if(bool b, std::uint64_t a, std::uint64_t c) noexcept
{
return c ^ ((a ^ c) & mask_if(b));
}
/// the distance of a position (see locate_token()) from the biased point 32
NLOHMANN_VIEW_ALWAYS_INLINE std::uint64_t biased_distance(std::uint64_t position) noexcept
{
constexpr std::uint64_t center = std::uint64_t{32} << 56u;
// |position - center|, without a branch: negated (complemented, plus 1) if below
const std::uint64_t below = mask_if(position < center);
return ((position - center) ^ below) - below;
}
/*!
@brief (g == 1) the digits dump() writes for a token, from the position of its
double (see locate_token()): the multiple of 10 nearest to x if it is in the
interval, else w rounded to x (an exact tie between two candidates, like
754570942.33984373, whose double is 754570942.33984375, is excluded by
locate_token())
With |x - w| at most the radius, which is below half the spacing of the
candidates, the multiple of 10 nearest to x is w - r10 or the one above it.
*/
NLOHMANN_VIEW_ALWAYS_INLINE std::uint64_t token_digits_1(std::uint64_t w, std::uint64_t offset, std::uint64_t radius, std::uint64_t r10) noexcept
{
constexpr std::uint64_t one = std::uint64_t{1} << 56u;
const std::uint64_t from = offset + (r10 * one); // x - (w - r10) (+ 32)
const bool up = from > 37 * one; // (nearer to the multiple above)
const bool shorter = biased_distance(from - (mask_if(up) & (10 * one))) < radius;
const std::uint64_t a = w - r10 + (mask_if(up) & 10u);
const std::uint64_t b = w + ((offset + (one / 2)) >> 56u) - 32; // w rounded to x
return select_if(shorter, a, b);
}
/*!
@brief (g == 2) the digits dump() writes for a token, as token_digits_1(): the
multiple of 100 nearest to x if it is in the interval, else the multiple of 10
nearest to x
*/
NLOHMANN_VIEW_ALWAYS_INLINE std::uint64_t token_digits_2(std::uint64_t w, std::uint64_t offset, std::uint64_t radius, std::uint64_t r10, std::uint64_t r100) noexcept
{
constexpr std::uint64_t one = std::uint64_t{1} << 56u;
const std::uint64_t from = offset + (r100 * one);
const bool up = from > 82 * one;
const bool shorter = biased_distance(from - (mask_if(up) & (100 * one))) < radius;
const std::uint64_t a = w - r100 + (mask_if(up) & 100u);
// x - (w - r10), in [-12, 21]: the multiple of 10 nearest to x is that one,
// or one or two above, or one below
const std::uint64_t y = offset + (r10 * one);
const std::uint64_t steps = static_cast<std::uint64_t>(y > 37 * one) + static_cast<std::uint64_t>(y > 47 * one);
const bool down = y < 27 * one;
const std::uint64_t b = w - r10 + (10 * steps) - (mask_if(down) & 10u);
return select_if(shorter, a, b);
}
/// the four ASCII digits of x < 10^4, the first in the lowest byte (as
/// little-endian memory holds them)
NLOHMANN_VIEW_ALWAYS_INLINE std::uint32_t four_digit_text(std::uint32_t x) noexcept
{
const std::uint32_t ab = (x * 5243u) >> 19u; // x / 100
std::uint32_t v = ab | ((x - (ab * 100u)) << 16u);
const std::uint32_t a = ((v * 103u) >> 10u) & 0x000F000Fu; // / 10 per 16-bit lane
v = a | ((v - (a * 10u)) << 8u);
return v + 0x30303030u;
}
/// how the view's dump() writes a value
struct dump_style
{
@@ -850,7 +711,6 @@ class view_serializer
std::array<frame, 32> buffer; // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init): written before read
std::vector<frame> heap;
small_stack<frame> stack(buffer.data(), buffer.size(), heap);
float_batch batch;
const char* const src = m_doc.src;
const char* const src_end = src + m_doc.size;
char* w = m_out.cursor();
@@ -990,7 +850,7 @@ class view_serializer
else if (enabled(fast_double::value))
{
room(64);
w = write_double_ahead(w, src, n, batch);
w = write_double_at(w, *n);
}
else
{
@@ -1015,20 +875,6 @@ class view_serializer
case value_t::array:
{
const bool object = n->kind == node_kind::object;
// (read-only documents, compact output) a pair of floats,
// like the coordinates of GeoJSON, without a frame
if (enabled(!Editable && !Pretty && !SourceNumbers && fast_double::value) && n->next == 3 && n->len == 2
&& n[1].kind == node_kind::number_float && n[2].kind == node_kind::number_float)
{
room(2 * 64);
*w++ = '[';
w = write_double_ahead(w, src, n + 1, batch);
*w++ = ',';
w = write_double_ahead(w, src, n + 2, batch);
*w++ = ']';
n += 2; // (the element before the next node)
break;
}
room(8);
if (n->len == 0)
{
@@ -1138,254 +984,6 @@ class view_serializer
}
}
/// (float_batch) the mark of digits whose rounding carried beyond the
/// token's last four (see mark())
static constexpr std::uint64_t carried() noexcept
{
return std::uint64_t{1} << 63u;
}
/// floats of 16 or 17 digits ahead of the walk, converted together (see
/// convert_ahead()), with their digits as mark() keeps them
struct float_batch
{
static constexpr std::size_t capacity() noexcept
{
return 256;
}
/// (on the heap, allocated by the first conversion: a frame of more
/// than a page made every dump of a small document slower)
struct entries
{
std::array<const node*, 256> at; // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init): written before read
std::array<std::uint64_t, 256> digits; // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init): written before read
};
std::unique_ptr<entries> storage{};
const node** at = nullptr;
std::uint64_t* digits = nullptr;
std::size_t count = 0;
std::size_t next = 0;
const node* scanned = nullptr; ///< where the last scan of the node array ended
};
/// (doubles) the float at n as dump() writes it, at w (64 bytes of room):
/// from the batch if it holds n
NLOHMANN_VIEW_ALWAYS_INLINE char* write_double_ahead(char* w, const char* src, const node* n, float_batch& batch)
{
#if NLOHMANN_VIEW_VECTOR && NLOHMANN_VIEW_LITTLE_ENDIAN
// (digits 0: not written by the batch; carried(): rounding carried
// beyond the last four digits)
if (batch.next != batch.count && batch.at[batch.next] == n && batch.digits[batch.next] - 1 < carried() - 1)
{
return write_token_digits(w, src, *n, batch.digits[batch.next++]);
}
return write_double_beside(w, n, batch);
#else
static_cast<void>(src);
static_cast<void>(batch);
return write_double_at(w, *n);
#endif
}
#if NLOHMANN_VIEW_VECTOR && NLOHMANN_VIEW_LITTLE_ENDIAN
/// write_double_ahead() for a float that is not the batch's next one (out
/// of line: the walk is faster with less code in its loop)
NLOHMANN_VIEW_NOINLINE char* write_double_beside(char* w, const node* n, float_batch& batch)
{
// (an editable document's walk can skip nodes of the array)
const std::less<const node*> before{}; // NOLINT(modernize-use-transparent-functors) C++11 has no std::less<>
while (batch.next != batch.count && before(batch.at[batch.next], n))
{
++batch.next;
}
if ((batch.next != batch.count && batch.at[batch.next] == n) || convert_ahead(n, batch))
{
const std::uint64_t digits = batch.digits[batch.next++];
if ((digits & carried()) != 0)
{
return write_carried_digits(w, *n, digits & ~carried());
}
if (digits != 0)
{
return write_token_digits(w, m_doc.src, *n, digits);
}
}
return write_double_at(w, *n);
}
/*!
@brief convert the floats of 16 or 17 digits among the next nodes of the
array at once; whether n is the first of them
Such tokens (like the coordinates of GeoJSON, written with 17 digits) are
written as the token with its last four digits replaced
(write_token_digits()). Each step (the digits, locate_token(),
token_digits_1/2()) is a long chain of dependent operations; one loop per
step over many floats lets the processor overlap the floats, which it does
not across the steps of one float (on Apple M1, one loop for the digits and
locate_token() made the dump of canada.json 12 % slower). Each node of the array
is scanned once; a node that the walk does not reach in order (in an
editable document) is written without the batch.
*/
NLOHMANN_VIEW_NOINLINE bool convert_ahead(const node* n, float_batch& batch)
{
const std::less<const node*> before{}; // NOLINT(modernize-use-transparent-functors) C++11 has no std::less<>
const node* const last = m_doc.nodes + m_doc.node_count;
if (before(n, batch.scanned) || before(n, m_doc.nodes) || !before(n, last))
{
return false;
}
const char* const src = m_doc.src;
const char* const src_end = src + m_doc.size;
const node* const stop = last - n > 1024 ? n + 1024 : last;
batch.count = 0;
batch.next = 0;
// the digits of each token (and its last four from the text)
std::array<std::uint64_t, 256> digits; // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init): written before read
std::array<int, 256> exponent; // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init): written before read
std::array<std::uint32_t, 256> last_four; // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init): written before read
const node* p = n;
if (!batch.storage)
{
batch.storage.reset(new typename float_batch::entries); // NOLINT(cppcoreguidelines-owning-memory,modernize-make-unique): C++11
batch.at = batch.storage->at.data();
batch.digits = batch.storage->digits.data();
}
for (; p != stop && batch.count != float_batch::capacity(); ++p)
{
// a token of the source: 16 or 17 digits, at least 8 after the
// point, no exponent, 32 bytes to read
const unsigned frac_digits = p->extra >> 8u;
const unsigned int_digits = p->extra & 0xFFu;
const unsigned count = int_digits + frac_digits;
if (p->kind == node_kind::number_float && (p->flags & node_flags::storage) == 0
&& count - 16u <= 1u && frac_digits >= 8 && src_end - (src + p->off) >= 33)
{
const auto* const token = reinterpret_cast<const unsigned char*>(src + p->off); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
const unsigned sign = token[0] == '-' ? 1 : 0;
const unsigned char* const first = token + sign;
const std::uint64_t w = sixteen_digits(first, int_digits);
// (the 17th digit follows the point; a token with an exponent
// gets digits outside the range of locate_token(), and is not
// written by the batch. A token 0.xxx with 16 significant
// digits is: its double is written as "0." and its digits too)
const std::uint64_t w17 = (w * 10) + static_cast<std::uint64_t>(first[17] - '0');
const std::size_t i = batch.count++;
batch.at[i] = p;
if (p->len != sign + count + 1)
{
digits[i] = 0;
}
else
{
digits[i] = count == 17 ? w17 : w;
}
exponent[i] = -static_cast<int>(frac_digits);
// (byte 0: the last digit, byte 1: the last two, bits 16-29: the last four)
std::uint32_t four = 0;
std::memcpy(&four, token + p->len - 4, 4);
four -= 0x30303030u;
const std::uint32_t pairs = ((four & 0x0F0F0F0Fu) * 2561u) >> 8u; // (byte 0, 2: the pairs)
last_four[i] = (four >> 24u) | ((pairs >> 16u & 0xFFu) << 8u) | ((((pairs & 0x00FF00FFu) * 6553601u) >> 16u) << 16u);
}
}
batch.scanned = p;
// the position of each double, and the floats of either kind
std::array<std::uint64_t, 256> offset; // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init): written before read
std::array<std::uint64_t, 256> radius; // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init): written before read
std::array<std::uint8_t, 256> kind_1; // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init): written before read
std::array<std::uint8_t, 256> kind_2; // NOLINT(cppcoreguidelines-pro-type-member-init,hicpp-member-init): written before read
std::size_t count_1 = 0;
std::size_t count_2 = 0;
for (std::size_t i = 0; i != batch.count; ++i)
{
const std::uint64_t w = digits[i];
// (the range excludes the values of bytes that are not digits, only
// in a damaged image loaded with image_check::bounds)
const int g = w - 1000000000000000u < 99000000000000000u ? locate_token(w, exponent[i], offset[i], radius[i]) : 3;
batch.digits[i] = mask_if(g <= 0) & ((last_four[i] >> 16u) + 1);
kind_1[count_1] = static_cast<std::uint8_t>(i);
count_1 += static_cast<std::size_t>(g == 1);
kind_2[count_2] = static_cast<std::uint8_t>(i);
count_2 += static_cast<std::size_t>(g == 2);
}
// the digits to write, in one loop per kind (see mark())
for (std::size_t k = 0; k != count_1; ++k)
{
const std::size_t i = kind_1[k];
const std::uint64_t s = token_digits_1(digits[i], offset[i], radius[i], last_four[i] & 0xFFu);
batch.digits[i] = mark(digits[i], s, last_four[i] >> 16u);
}
for (std::size_t k = 0; k != count_2; ++k)
{
const std::size_t i = kind_2[k];
const std::uint64_t s = token_digits_2(digits[i], offset[i], radius[i], last_four[i] & 0xFFu, (last_four[i] >> 8u) & 0xFFu);
batch.digits[i] = mark(digits[i], s, last_four[i] >> 16u);
}
return batch.count != 0 && batch.at[0] == n;
}
/*!
@brief a float token of 16 or 17 digits as dump() writes it, at w (64 bytes
of room), from the digits of its double (see locate_token()) whose
rounding carried beyond the token's last four digits
(like 43.42027299999999 -> 43.420273): the digits go to the writer of
Zmij's results as its 15 or 16 leading digits and one more, which omits
the trailing zeros.
*/
NLOHMANN_VIEW_NOINLINE char* write_carried_digits(char* w, const node& n, std::uint64_t digits) const noexcept
{
const int exponent = -(n.extra >> 8u);
const std::uint64_t leading = digits / 10;
const auto last = static_cast<unsigned char>(digits - (leading * 10));
const bool seventeen = digits >= 10000000000000000u;
*w = '-';
w += m_doc.src[n.off] == '-' ? 1 : 0;
return ::nlohmann::detail::dtoa_impl::write_shortest(w, seventeen ? ::nlohmann::detail::zmij::shortest_decimal{leading, exponent, last, last != 0}
: ::nlohmann::detail::zmij::shortest_decimal{digits, exponent - 1, 0, false});
}
/// the digits s of a token with the digits w and the last four digits
/// r4, as the batch keeps them: the new last four digits + 1 (only those
/// are written over the token's), or s marked if it differs from w beyond
/// them (0 stays for a float the batch does not write)
static std::uint64_t mark(std::uint64_t w, std::uint64_t s, std::uint64_t r4) noexcept
{
const std::uint64_t last = s - (w - r4);
return last < 10000 ? last + 1 : s | carried();
}
/*!
@brief a float token of 16 or 17 digits as dump() writes it, at w (64 bytes
of room): the token with its last four digits replaced by digits - 1 (see
mark(); the others are the token's), and without trailing zeros after the
point, but one
*/
char* write_token_digits(char* w, const char* src, const node& n, std::uint64_t digits)
{
const char* const token = src + n.off;
std::memcpy(w, token, 32);
char* const end = w + n.len;
const std::uint32_t last = four_digit_text(static_cast<std::uint32_t>(digits - 1));
std::memcpy(end - 4, &last, 4);
// (the token's text before the new digits, from the source: reading
// the output back would wait for the stores)
std::uint64_t before = 0;
std::memcpy(&before, token + n.len - 12, 8);
const std::uint32_t last_zeros = last ^ 0x30303030u;
const std::uint64_t before_zeros = before ^ 0x3030303030303030u;
if (NLOHMANN_VIEW_UNLIKELY(last_zeros == 0 && before_zeros == 0))
{
return write_double_at(w, n); // (twelve or more trailing zeros)
}
const unsigned zeros = last_zeros != 0 ? (static_cast<unsigned>(count_leading_zeros(last_zeros)) / 8) - 4
: 4 + (static_cast<unsigned>(count_leading_zeros(before_zeros)) / 8);
const unsigned frac_digits = n.extra >> 8u;
return end - (zeros < frac_digits ? zeros : frac_digits - 1);
}
#endif
/// a float node as dump() writes it
void write_float_node(const node& n)
{
+2 -5
View File
@@ -289,12 +289,9 @@ compares, and the UTF-8 check covers the bytes up to it. Returns where the
string scan stops, like scan_string_run: before ill-formed UTF-8 and for the
last bytes of the input, the bytes are checked one sequence at a time. Out of
line, so that no constants of the check occupy registers in the parse loop.
On x86-64, it is compiled for SSSE3 (see cpu_has_ssse3()). Not cloned: with
the calls of accept()'s builder, GCC 16 would clone it without plain (always
string_plain()), and calling the clone made json_document::parse() run 0.7-1.2 %
more instructions.
On x86-64, it is compiled for SSSE3 (see cpu_has_ssse3()).
*/
NLOHMANN_VIEW_SSSE3_TARGET NLOHMANN_VIEW_NOINLINE NLOHMANN_VIEW_NOCLONE inline const unsigned char* scan_string_vector(const unsigned char* p, const unsigned char* e, const std::uint8_t* plain) noexcept
NLOHMANN_VIEW_SSSE3_TARGET NLOHMANN_VIEW_NOINLINE inline const unsigned char* scan_string_vector(const unsigned char* p, const unsigned char* e, const std::uint8_t* plain) noexcept
{
using lookup = utf8_lookup4<>;
const unsigned char* block = p;
+3 -3
View File
@@ -60,7 +60,7 @@ NLOHMANN_VIEW_ALWAYS_INLINE T arithmetic_value(const document_data& d, const nod
case value_t::binary:
case value_t::discarded:
default:
throw_type_error(error_site(&d, &n), exception_id::type_mismatch, "type must be number, but is ", value_type_name(static_cast<value_t>(n.kind)));
throw_type_error(exception_id::type_mismatch, "type must be number, but is ", value_type_name(static_cast<value_t>(n.kind)));
}
}
@@ -70,7 +70,7 @@ std::vector<U, A> vector_value(const View& v)
{
if (NLOHMANN_VIEW_UNLIKELY(!v.is_array()))
{
throw_type_error(site_of(v), exception_id::type_mismatch, "type must be array, but is ", v.type_name());
throw_type_error(exception_id::type_mismatch, "type must be array, but is ", v.type_name());
}
std::vector<U, A> r;
r.reserve(v.size());
@@ -88,7 +88,7 @@ Map map_value(const View& v)
{
if (NLOHMANN_VIEW_UNLIKELY(!v.is_object()))
{
throw_type_error(site_of(v), exception_id::type_mismatch, "type must be object, but is ", v.type_name());
throw_type_error(exception_id::type_mismatch, "type must be object, but is ", v.type_name());
}
Map r;
for (auto it = v.begin(); it != v.end(); ++it)
+7 -9
View File
@@ -150,14 +150,14 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
// can be restored when json_pointer backwards compatibility is removed
// friend ::nlohmann::json_pointer<StringType>;
template<typename BasicJsonType, typename InputType, bool DiscardValues>
template<typename BasicJsonType, typename InputType>
friend class ::nlohmann::detail::parser;
friend ::nlohmann::detail::serializer<basic_json>;
template<typename BasicJsonType>
friend class ::nlohmann::detail::iter_impl;
template<typename BasicJsonType, typename CharType, typename OutputSinkType>
friend class ::nlohmann::detail::binary_writer;
template<typename BasicJsonType, typename InputType, typename SAX, bool AllowRecovery>
template<typename BasicJsonType, typename InputType, typename SAX>
friend class ::nlohmann::detail::binary_reader;
template<typename BasicJsonType, typename InputAdapterType, bool Collect, typename LexerType>
friend class ::nlohmann::detail::json_sax_dom_parser;
@@ -183,11 +183,12 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
detail::parser_callback_t<basic_json>cb = nullptr,
const bool allow_exceptions = true,
const bool ignore_comments = false,
const bool ignore_trailing_commas = false
const bool ignore_trailing_commas = false,
const bool discard_number_values = false
)
{
return ::nlohmann::detail::parser<basic_json, InputAdapterType>(std::move(adapter),
std::move(cb), allow_exceptions, ignore_comments, ignore_trailing_commas);
std::move(cb), allow_exceptions, ignore_comments, ignore_trailing_commas, discard_number_values);
}
private:
@@ -3116,9 +3117,7 @@ public:
const bool ignore_comments,
const bool ignore_trailing_commas)
{
// the parser and lexer that discard the values of tokens (see lexer)
return ::nlohmann::detail::parser<basic_json, typename std::decay<InputAdapterType>::type, true>(std::forward<InputAdapterType>(adapter),
nullptr, false, ignore_comments, ignore_trailing_commas).accept(true);
return parser(std::forward<InputAdapterType>(adapter), nullptr, false, ignore_comments, ignore_trailing_commas, true).accept(true);
}
/// generate SAX events from the input behind an already created adapter
@@ -3134,8 +3133,7 @@ public:
{
return parser(std::move(ia), nullptr, true, ignore_comments, ignore_trailing_commas).sax_parse(sax, strict);
}
// the SAX parser may recover from errors (see parse_error())
return detail::binary_reader<basic_json, InputAdapterType, SAX, true>(std::move(ia), format).sax_parse(sax, strict, tag_handler);
return detail::binary_reader<basic_json, InputAdapterType, SAX>(std::move(ia), format).sax_parse(sax, strict, tag_handler);
}
/// serialize @a j into a new vector; @a write receives the vector-backed writer
+48 -70
View File
@@ -285,7 +285,7 @@ class basic_json_view
{
return {};
}
detail::view::throw_type_error(site(), detail::exception_id::subscript_wrong_type, "cannot use operator[] with a string argument with ", type_name());
detail::view::throw_type_error(detail::exception_id::subscript_wrong_type, "cannot use operator[] with a string argument with ", type_name());
}
return lookup(key);
}
@@ -311,7 +311,7 @@ class basic_json_view
{
return {};
}
detail::view::throw_type_error(site(), detail::exception_id::subscript_wrong_type, "cannot use operator[] with a numeric argument with ", type_name());
detail::view::throw_type_error(detail::exception_id::subscript_wrong_type, "cannot use operator[] with a numeric argument with ", type_name());
}
return idx < m_node->len ? element(idx) : basic_json_view();
}
@@ -344,12 +344,12 @@ class basic_json_view
{
if (NLOHMANN_VIEW_UNLIKELY(!is_object()))
{
detail::view::throw_type_error(site(), detail::exception_id::at_wrong_type, "cannot use at() with ", type_name());
detail::view::throw_type_error(detail::exception_id::at_wrong_type, "cannot use at() with ", type_name());
}
const basic_json_view r = lookup(key);
if (NLOHMANN_VIEW_UNLIKELY(r.is_discarded()))
{
detail::view::throw_key_not_found(site(), std::string(key.data(), key.size()));
detail::view::throw_key_not_found(std::string(key.data(), key.size()));
}
return r;
}
@@ -370,11 +370,11 @@ class basic_json_view
{
if (NLOHMANN_VIEW_UNLIKELY(!is_array()))
{
detail::view::throw_type_error(site(), detail::exception_id::at_wrong_type, "cannot use at() with ", type_name());
detail::view::throw_type_error(detail::exception_id::at_wrong_type, "cannot use at() with ", type_name());
}
if (NLOHMANN_VIEW_UNLIKELY(idx >= m_node->len))
{
detail::view::throw_array_index_out_of_range(site(), idx);
detail::view::throw_array_index_out_of_range(idx);
}
return element(idx);
}
@@ -402,7 +402,7 @@ class basic_json_view
{
if (NLOHMANN_VIEW_UNLIKELY(!is_object()))
{
detail::view::throw_type_error(site(), detail::exception_id::value_wrong_type, "cannot use value() with ", type_name());
detail::view::throw_type_error(detail::exception_id::value_wrong_type, "cannot use value() with ", type_name());
}
const basic_json_view r = lookup(key);
return r.is_discarded() ? default_value : r.template get<T>();
@@ -412,7 +412,7 @@ class basic_json_view
{
if (NLOHMANN_VIEW_UNLIKELY(!is_object()))
{
detail::view::throw_type_error(site(), detail::exception_id::value_wrong_type, "cannot use value() with ", type_name());
detail::view::throw_type_error(detail::exception_id::value_wrong_type, "cannot use value() with ", type_name());
}
// (the default is made only if there is no such member)
const basic_json_view r = lookup(key);
@@ -427,7 +427,7 @@ class basic_json_view
{
if (NLOHMANN_VIEW_UNLIKELY(!is_structured()))
{
detail::view::throw_type_error(site(), detail::exception_id::value_wrong_type, "cannot use value() with ", type_name());
detail::view::throw_type_error(detail::exception_id::value_wrong_type, "cannot use value() with ", type_name());
}
const basic_json_view r = detail::view::resolve_pointer(*this, detail::json_pointer_access::reference_tokens(ptr), detail::view::pointer_mode::value);
return r.is_discarded() ? default_value : r.template get<T>();
@@ -437,7 +437,7 @@ class basic_json_view
{
if (NLOHMANN_VIEW_UNLIKELY(!is_structured()))
{
detail::view::throw_type_error(site(), detail::exception_id::value_wrong_type, "cannot use value() with ", type_name());
detail::view::throw_type_error(detail::exception_id::value_wrong_type, "cannot use value() with ", type_name());
}
const basic_json_view r = detail::view::resolve_pointer(*this, detail::json_pointer_access::reference_tokens(ptr), detail::view::pointer_mode::value);
return r.is_discarded() ? string_t(default_value) : r.template get<string_t>();
@@ -450,7 +450,7 @@ class basic_json_view
const iterator it = begin();
if (NLOHMANN_VIEW_UNLIKELY(it == end()))
{
detail::view::throw_invalid_iterator(site(), detail::exception_id::iterator_value_unavailable, "cannot get value");
detail::view::throw_invalid_iterator(detail::exception_id::iterator_value_unavailable, "cannot get value");
}
return *it;
}
@@ -606,7 +606,7 @@ class basic_json_view
{
if (NLOHMANN_VIEW_UNLIKELY(!is_string()))
{
detail::view::throw_type_error(site(), detail::exception_id::type_mismatch, "type must be string, but is ", type_name());
detail::view::throw_type_error(detail::exception_id::type_mismatch, "type must be string, but is ", type_name());
}
return {m_doc->str(*m_node), m_node->len};
}
@@ -618,7 +618,7 @@ class basic_json_view
{
if (NLOHMANN_VIEW_UNLIKELY(!is_number()))
{
detail::view::throw_type_error(site(), detail::exception_id::type_mismatch, "type must be number, but is ", type_name());
detail::view::throw_type_error(detail::exception_id::type_mismatch, "type must be number, but is ", type_name());
}
return {m_doc->str(*m_node), detail::view::number_length(*m_node)};
}
@@ -764,19 +764,15 @@ class basic_json_view
template<typename, view_mode> friend class basic_json_view;
template<typename, typename> friend class detail::view::editor;
template<typename, typename> friend class detail::view::view_side;
friend struct detail::view::view_access;
friend iterator;
/// whether the view is one of an editable document
static constexpr bool is_editable = Editable;
basic_json_view(const document_data* d, const node* n) noexcept
: m_doc(d), m_node(n)
{}
/// this value as the context of an exception
detail::view::error_site site() const noexcept
{
return {m_doc, m_node};
}
using json_side_t = detail::view::json_side<BasicJsonType, string_view_t>;
static detail::view::view_side<BasicJsonType, basic_json_view> side(const basic_json_view& v) noexcept
@@ -883,7 +879,7 @@ class basic_json_view
{
if (NLOHMANN_VIEW_UNLIKELY(!is_boolean()))
{
detail::view::throw_type_error(site(), detail::exception_id::type_mismatch, "type must be boolean, but is ", type_name());
detail::view::throw_type_error(detail::exception_id::type_mismatch, "type must be boolean, but is ", type_name());
}
return (m_node->flags & detail::view::node_flags::is_true) != 0;
}
@@ -893,7 +889,7 @@ class basic_json_view
{
if (NLOHMANN_VIEW_UNLIKELY(m_node == nullptr))
{
detail::view::throw_type_error(site(), detail::exception_id::type_mismatch, "type must be number, but is ", type_name());
detail::view::throw_type_error(detail::exception_id::type_mismatch, "type must be number, but is ", type_name());
}
return detail::view::arithmetic_value<T, BasicJsonType>(*m_doc, *m_node);
}
@@ -902,7 +898,7 @@ class basic_json_view
{
if (NLOHMANN_VIEW_UNLIKELY(!is_null()))
{
detail::view::throw_type_error(site(), detail::exception_id::type_mismatch, "type must be null, but is ", type_name());
detail::view::throw_type_error(detail::exception_id::type_mismatch, "type must be null, but is ", type_name());
}
return nullptr;
}
@@ -1067,7 +1063,7 @@ class basic_json_document
const bool ignore_trailing_commas = false)
{
basic_json_document d;
d.read_range(first, last, error_mode(allow_exceptions), ignore_comments, ignore_trailing_commas);
d.read_range(first, last, allow_exceptions, ignore_comments, ignore_trailing_commas);
return d;
}
@@ -1082,7 +1078,7 @@ class basic_json_document
basic_json_document d;
std::string buf;
append_input(buf, std::forward<InputType>(input));
d.build_owned(std::move(buf), error_mode(allow_exceptions), ignore_comments, ignore_trailing_commas);
d.build_owned(std::move(buf), allow_exceptions, ignore_comments, ignore_trailing_commas);
return d;
}
@@ -1095,8 +1091,7 @@ class basic_json_document
static bool accept(InputType&& input, const bool ignore_comments = false, const bool ignore_trailing_commas = false)
{
basic_json_document d;
d.read_kind(std::forward<InputType>(input), on_error::validate, ignore_comments, ignore_trailing_commas,
std::integral_constant<detail::view::input_kind, detail::view::classify_input<InputType>::value> {});
d.read(std::forward<InputType>(input), false, ignore_comments, ignore_trailing_commas);
return !d.is_discarded();
}
@@ -1112,7 +1107,7 @@ class basic_json_document
const bool ignore_comments = false,
const bool ignore_trailing_commas = false)
{
read_kind(std::forward<InputType>(input), error_mode(allow_exceptions), ignore_comments, ignore_trailing_commas,
read_kind(std::forward<InputType>(input), allow_exceptions, ignore_comments, ignore_trailing_commas,
std::integral_constant<detail::view::input_kind, detail::view::classify_input<InputType>::value> {});
}
@@ -1317,7 +1312,7 @@ class basic_json_document
template<typename I, typename V, typename std::enable_if<detail::view::is_non_bool_integral<I>::value, int>::type = 0>
view_type set(view_type array, I idx, V && value)
{
return editor().set(array, index(array, idx), std::forward<V>(value));
return editor().set(array, index(idx), std::forward<V>(value));
}
/// set the value at a JSON pointer: its parent must exist; an object
@@ -1354,7 +1349,7 @@ class basic_json_document
const std::size_t idx = pointer_index(token);
if (idx >= 0xFFFFFFFFu)
{
detail::view::throw_array_index_out_of_range(detail::view::site_of(parent), idx);
detail::view::throw_array_index_out_of_range(idx);
}
for (std::size_t i = 0; i < idx; ++i)
{
@@ -1388,7 +1383,7 @@ class basic_json_document
template<typename I, typename V, typename std::enable_if<detail::view::is_non_bool_integral<I>::value, int>::type = 0>
view_type insert(view_type array, I idx, V && value)
{
return editor().insert(array, index(array, idx), std::forward<V>(value));
return editor().insert(array, index(idx), std::forward<V>(value));
}
/// remove all members with this key; returns their number
@@ -1401,7 +1396,7 @@ class basic_json_document
template<typename I, typename std::enable_if<detail::view::is_non_bool_integral<I>::value, int>::type = 0>
void erase(view_type array, I idx)
{
editor().erase(array, index(array, idx));
editor().erase(array, index(idx));
}
/// remove the value at a JSON pointer; returns the number of removed
@@ -1435,25 +1430,25 @@ class basic_json_document
return detail::view::editor<BasicJsonType, view_type>(*m_data);
}
/// an index into @a array (out_of_range.401 if negative)
/// an index (out_of_range.401 if negative)
template<typename I>
static std::size_t index(const view_type& array, I idx)
static std::size_t index(I idx)
{
return index(array, idx, std::is_signed<I> {});
return index(idx, std::is_signed<I> {});
}
template<typename I>
static std::size_t index(const view_type& array, I idx, std::true_type /*signed*/)
static std::size_t index(I idx, std::true_type /*signed*/)
{
if (idx < 0)
{
detail::view::throw_array_index_out_of_range(detail::view::site_of(array), idx);
detail::view::throw_array_index_out_of_range(idx);
}
return static_cast<std::size_t>(idx);
}
template<typename I>
static std::size_t index(const view_type& /*array*/, I idx, std::false_type /*unsigned*/)
static std::size_t index(I idx, std::false_type /*unsigned*/)
{
return static_cast<std::size_t>(idx);
}
@@ -1471,22 +1466,6 @@ class basic_json_document
return idx;
}
/// what a failed parse does
enum class on_error : std::uint8_t
{
discard, ///< the document is discarded
raise, ///< parse_error is thrown
validate ///< accept(): the document is discarded, and no node is stored while parsing
};
static on_error error_mode(bool allow_exceptions) noexcept
{
return allow_exceptions ? on_error::raise : on_error::discard;
}
/// NulIsEnd of the builder: a null byte ends the input unless strict_nul_handling
static constexpr bool nul_is_end = !detail::abi_config::strict_nul_handling;
/// create the storage (sized for the input) on first use
void ensure_data(const char* src, std::size_t size)
{
@@ -1498,7 +1477,7 @@ class basic_json_document
/// parse a buffer the document takes ownership of (it replaces the
/// previous owned copy)
void build_owned(std::string&& buf, on_error allow_exceptions, bool comments, bool trailing_commas)
void build_owned(std::string&& buf, bool allow_exceptions, bool comments, bool trailing_commas)
{
ensure_data(buf.data(), buf.size());
m_data->owned = std::move(buf);
@@ -1508,7 +1487,7 @@ class basic_json_document
/// parse a copy of the input; the copy is appended to the cleared owned
/// buffer, so that its capacity is reused
template<typename T>
void build_copy(T&& input, on_error allow_exceptions, bool comments, bool trailing_commas)
void build_copy(T&& input, bool allow_exceptions, bool comments, bool trailing_commas)
{
if (!m_data)
{
@@ -1529,7 +1508,7 @@ class basic_json_document
}
/// sentinel: src[size] is readable and 0 (std::string, C strings)
void build(const char* src, std::size_t size, on_error allow_exceptions, bool comments, bool trailing_commas, bool owned, bool sentinel)
void build(const char* src, std::size_t size, bool allow_exceptions, bool comments, bool trailing_commas, bool owned, bool sentinel)
{
ensure_data(src, size);
document_data& d = *m_data;
@@ -1556,8 +1535,7 @@ class basic_json_document
}
else
{
ok = detail::view::build<typename BasicJsonType::number_float_t, nul_is_end>(d, src, size, comments, trailing_commas, sentinel,
allow_exceptions == on_error::validate, failure);
ok = detail::view::build < typename BasicJsonType::number_float_t, !detail::abi_config::strict_nul_handling > (d, src, size, comments, trailing_commas, sentinel, failure);
}
if (NLOHMANN_VIEW_LIKELY(ok))
{
@@ -1570,7 +1548,7 @@ class basic_json_document
return;
}
std::vector<std::uint32_t>().swap(d.large_objects);
if (allow_exceptions == on_error::raise)
if (allow_exceptions)
{
detail::view::throw_parse_failure<BasicJsonType>(failure, src, size, comments, trailing_commas);
}
@@ -1578,13 +1556,13 @@ class basic_json_document
// --- input dispatch (see detail::view::input_kind) ---
void read_kind(std::string&& s, on_error ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::move_string> /*unused*/)
void read_kind(std::string&& s, bool ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::move_string> /*unused*/)
{
build_owned(std::move(s), ae, c, tc);
}
template<typename CharT>
void read_kind(CharT* s, on_error ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::c_string> /*unused*/)
void read_kind(CharT* s, bool ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::c_string> /*unused*/)
{
if (s == nullptr)
{
@@ -1598,7 +1576,7 @@ class basic_json_document
}
template<typename Array>
void read_kind(Array& a, on_error ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::char_array> /*unused*/)
void read_kind(Array& a, bool ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::char_array> /*unused*/)
{
using CharT = typename std::remove_cv<typename std::remove_extent<Array>::type>::type;
const std::size_t n = std::extent<Array>::value;
@@ -1609,7 +1587,7 @@ class basic_json_document
}
template<typename T>
void read_kind(const T& s, on_error ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::borrow_range> /*unused*/)
void read_kind(const T& s, bool ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::borrow_range> /*unused*/)
{
// std::basic_string guarantees data()[size()] == 0: use it as sentinel
const auto size = static_cast<std::size_t>(s.size());
@@ -1618,13 +1596,13 @@ class basic_json_document
}
template<typename T>
void read_kind(const T& s, on_error ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::copy_range> /*unused*/)
void read_kind(const T& s, bool ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::copy_range> /*unused*/)
{
build_copy(s, ae, c, tc);
}
template<typename CharT>
void read_kind(CharT* s, on_error ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::wide_c_string> /*unused*/)
void read_kind(CharT* s, bool ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::wide_c_string> /*unused*/)
{
if (s == nullptr)
{
@@ -1636,13 +1614,13 @@ class basic_json_document
}
template<typename T>
void read_kind(T&& input, on_error ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::adapter> /*unused*/)
void read_kind(T&& input, bool ae, bool c, bool tc, std::integral_constant<input_kind, input_kind::adapter> /*unused*/)
{
build_copy(std::forward<T>(input), ae, c, tc);
}
template<typename IteratorType>
void read_range(IteratorType first, IteratorType last, on_error ae, bool c, bool tc)
void read_range(IteratorType first, IteratorType last, bool ae, bool c, bool tc)
{
using value_type = typename std::remove_cv<typename std::iterator_traits<IteratorType>::value_type>::type;
// borrow the range where the library's input adapter scans it as one
@@ -1653,14 +1631,14 @@ class basic_json_document
}
template<typename IteratorType>
void read_range_impl(IteratorType first, IteratorType last, on_error ae, bool c, bool tc, std::true_type /*contiguous bytes*/)
void read_range_impl(IteratorType first, IteratorType last, bool ae, bool c, bool tc, std::true_type /*contiguous bytes*/)
{
const auto size = static_cast<std::size_t>(std::distance(first, last));
build(size == 0 ? "" : reinterpret_cast<const char*>(&*first), size, ae, c, tc, false, size == 0); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
}
template<typename IteratorType>
void read_range_impl(IteratorType first, IteratorType last, on_error ae, bool c, bool tc, std::false_type /*other*/)
void read_range_impl(IteratorType first, IteratorType last, bool ae, bool c, bool tc, std::false_type /*other*/)
{
build_copy(detail::view::make_adapter_input(detail::input_adapter(first, last)), ae, c, tc);
}
-11880
View File
File diff suppressed because it is too large. Load diff
+501 -3963
View File
File diff suppressed because it is too large. Load diff
+559 -1645
View File
File diff suppressed because it is too large. Load diff
-13
View File
@@ -47,10 +47,6 @@ dumps is stable under exactly the same values that break operator==.
The unit tests run the same checks on a fixed corpus (see the "BJData round-trip
invariants" test case), so keep both in sync.
Furthermore, it reads data with a SAX parser that recovers from every error
and checks that the events are balanced, that reading ends, and that it
reports an error exactly when from_bjdata() fails (see #3989).
Finally, it reads the data from a stream, a non-contiguous iterator range, and
a FILE*, which read byte by byte, and checks that each gives the same value or
error (id, message, byte position) as the pointer range, which takes the bulk
@@ -61,7 +57,6 @@ drivers.
*/
#include "fuzzer_common.hpp"
#include "fuzzer-recovering_checker.hpp"
namespace
{
@@ -86,9 +81,6 @@ static bool is_value_stable(const json& lhs, const json& rhs)
// see http://llvm.org/docs/LibFuzzer.html
extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
{
// recover from all errors, reading from memory and from a stream
const bool recovered_without_errors = check_recovering_parse(data, size, json::input_format_t::bjdata).errors == 0;
std::vector<uint8_t> const vec1(data, data + size);
// step 0: parse input without exceptions
@@ -107,9 +99,6 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
// without exceptions, the same input must give the same value
assert(!noexcept_threw && !j_noexcept.is_discarded() && same_value(j_noexcept, j1));
// the recovering parser must not have reported an error either
assert(recovered_without_errors);
try
{
// step 2.1: round trip without adding size annotations to container types
@@ -143,7 +132,6 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
{
// parse errors are ok, because input may be random bytes
assert(parsed || noexcept_threw || j_noexcept.is_discarded());
assert(parsed || !recovered_without_errors);
}
catch (const json::type_error&)
{
@@ -154,7 +142,6 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
{
// out of range errors may happen if provided sizes are excessive
assert(parsed || noexcept_threw || j_noexcept.is_discarded());
assert(parsed || !recovered_without_errors);
}
// step 3: the other input adapters must read the data alike
-13
View File
@@ -17,10 +17,6 @@ array data, it performs the following steps:
- j2 = from_bon8(vec)
- assert(to_bon8(j2) == vec)
Furthermore, it reads data with a SAX parser that recovers from every error
and checks that the events are balanced, that reading ends, and that it
reports an error exactly when from_bon8() fails (see #3989).
Finally, it reads the data from a stream, a non-contiguous iterator range, and
a FILE*, which read byte by byte, and checks that each gives the same value or
error (id, message, byte position) as the pointer range, which takes the bulk
@@ -31,7 +27,6 @@ drivers.
*/
#include "fuzzer_common.hpp"
#include "fuzzer-recovering_checker.hpp"
namespace
{
@@ -49,9 +44,6 @@ struct read_bon8
// see http://llvm.org/docs/LibFuzzer.html
extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
{
// recover from all errors, reading from memory and from a stream
const bool recovered_without_errors = check_recovering_parse(data, size, json::input_format_t::bon8).errors == 0;
std::vector<uint8_t> const vec1(data, data + size);
// step 0: parse input without exceptions
@@ -70,9 +62,6 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
// without exceptions, the same input must give the same value
assert(!noexcept_threw && !j_noexcept.is_discarded() && same_value(j_noexcept, j1));
// the recovering parser must not have reported an error either
assert(recovered_without_errors);
try
{
// step 2: round trip
@@ -94,7 +83,6 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
{
// parse errors are ok, because input may be random bytes
assert(parsed || noexcept_threw || j_noexcept.is_discarded());
assert(parsed || !recovered_without_errors);
}
catch (const json::type_error&)
{
@@ -105,7 +93,6 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
{
// out of range errors may happen if provided sizes are excessive
assert(parsed || noexcept_threw || j_noexcept.is_discarded());
assert(parsed || !recovered_without_errors);
}
// step 3: the other input adapters must read the data alike
-13
View File
@@ -17,10 +17,6 @@ array data, it performs the following steps:
- j2 = from_bson(vec)
- assert(to_bson(j2) == vec)
Furthermore, it reads data with a SAX parser that recovers from every error
and checks that the events are balanced, that reading ends, and that it
reports an error exactly when from_bson() fails (see #3989).
Finally, it reads the data from a stream, a non-contiguous iterator range, and
a FILE*, which read byte by byte, and checks that each gives the same value or
error (id, message, byte position) as the pointer range, which takes the bulk
@@ -31,7 +27,6 @@ drivers.
*/
#include "fuzzer_common.hpp"
#include "fuzzer-recovering_checker.hpp"
namespace
{
@@ -49,9 +44,6 @@ struct read_bson
// see http://llvm.org/docs/LibFuzzer.html
extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
{
// recover from all errors, reading from memory and from a stream
const bool recovered_without_errors = check_recovering_parse(data, size, json::input_format_t::bson).errors == 0;
std::vector<uint8_t> const vec1(data, data + size);
// step 0: parse input without exceptions
@@ -70,9 +62,6 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
// without exceptions, the same input must give the same value
assert(!noexcept_threw && !j_noexcept.is_discarded() && same_value(j_noexcept, j1));
// the recovering parser must not have reported an error either
assert(recovered_without_errors);
try
{
// step 2: round trip
@@ -94,7 +83,6 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
{
// parse errors are ok, because input may be random bytes
assert(parsed || noexcept_threw || j_noexcept.is_discarded());
assert(parsed || !recovered_without_errors);
}
catch (const json::type_error&)
{
@@ -105,7 +93,6 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
{
// out of range errors can occur during parsing, too
assert(parsed || noexcept_threw || j_noexcept.is_discarded());
assert(parsed || !recovered_without_errors);
}
// step 3: the other input adapters must read the data alike
-13
View File
@@ -17,10 +17,6 @@ array data, it performs the following steps:
- j2 = from_cbor(vec)
- assert(to_cbor(j2) == vec)
Furthermore, it reads data with a SAX parser that recovers from every error
and checks that the events are balanced, that reading ends, and that it
reports an error exactly when from_cbor() fails (see #3989).
Finally, it reads the data from a stream, a non-contiguous iterator range, and
a FILE*, which read byte by byte, and checks that each gives the same value or
error (id, message, byte position) as the pointer range, which takes the bulk
@@ -31,7 +27,6 @@ drivers.
*/
#include "fuzzer_common.hpp"
#include "fuzzer-recovering_checker.hpp"
namespace
{
@@ -49,9 +44,6 @@ struct read_cbor
// see http://llvm.org/docs/LibFuzzer.html
extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
{
// recover from all errors, reading from memory and from a stream
const bool recovered_without_errors = check_recovering_parse(data, size, json::input_format_t::cbor).errors == 0;
std::vector<uint8_t> const vec1(data, data + size);
// step 0: parse input without exceptions
@@ -70,9 +62,6 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
// without exceptions, the same input must give the same value
assert(!noexcept_threw && !j_noexcept.is_discarded() && same_value(j_noexcept, j1));
// the recovering parser must not have reported an error either
assert(recovered_without_errors);
try
{
// step 2: round trip
@@ -94,7 +83,6 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
{
// parse errors are ok, because input may be random bytes
assert(parsed || noexcept_threw || j_noexcept.is_discarded());
assert(parsed || !recovered_without_errors);
}
catch (const json::type_error&)
{
@@ -105,7 +93,6 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
{
// out of range errors can occur during parsing, too
assert(parsed || noexcept_threw || j_noexcept.is_discarded());
assert(parsed || !recovered_without_errors);
}
// step 3: the other input adapters must read the data alike
-12
View File
@@ -24,16 +24,11 @@ array data, it performs the following steps:
The pointer range takes the bulk fast paths, the other input adapters read byte
by byte; see check_input_adapters().
Furthermore, it parses data with a SAX parser that recovers from every error
and checks that the events are balanced, that parsing ends, and that valid
input is parsed without errors (see #3989).
The provided function `LLVMFuzzerTestOneInput` can be used in different fuzzer
drivers.
*/
#include "fuzzer_common.hpp"
#include "fuzzer-recovering_checker.hpp"
namespace
{
@@ -64,13 +59,6 @@ struct accept_json
// see http://llvm.org/docs/LibFuzzer.html
extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
{
// recover from all errors, reading from memory and from a stream
{
const auto checker = check_recovering_parse(data, size, json::input_format_t::json);
assert(checker.events <= (4 * size) + 4);
assert((checker.errors == 0) == json::accept(data, data + size));
}
// step 0: parse input without exceptions
bool noexcept_threw = false;
json const j_noexcept = parse_without_exceptions([&] { return json::parse(data, data + size, nullptr, false); }, noexcept_threw);
-13
View File
@@ -17,10 +17,6 @@ array data, it performs the following steps:
- j2 = from_msgpack(vec)
- assert(to_msgpack(j2) == vec)
Furthermore, it reads data with a SAX parser that recovers from every error
and checks that the events are balanced, that reading ends, and that it
reports an error exactly when from_msgpack() fails (see #3989).
Finally, it reads the data from a stream, a non-contiguous iterator range, and
a FILE*, which read byte by byte, and checks that each gives the same value or
error (id, message, byte position) as the pointer range, which takes the bulk
@@ -31,7 +27,6 @@ drivers.
*/
#include "fuzzer_common.hpp"
#include "fuzzer-recovering_checker.hpp"
namespace
{
@@ -49,9 +44,6 @@ struct read_msgpack
// see http://llvm.org/docs/LibFuzzer.html
extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
{
// recover from all errors, reading from memory and from a stream
const bool recovered_without_errors = check_recovering_parse(data, size, json::input_format_t::msgpack).errors == 0;
std::vector<uint8_t> const vec1(data, data + size);
// step 0: parse input without exceptions
@@ -70,9 +62,6 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
// without exceptions, the same input must give the same value
assert(!noexcept_threw && !j_noexcept.is_discarded() && same_value(j_noexcept, j1));
// the recovering parser must not have reported an error either
assert(recovered_without_errors);
try
{
// step 2: round trip
@@ -94,7 +83,6 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
{
// parse errors are ok, because input may be random bytes
assert(parsed || noexcept_threw || j_noexcept.is_discarded());
assert(parsed || !recovered_without_errors);
}
catch (const json::type_error&)
{
@@ -105,7 +93,6 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
{
// out of range errors may happen if provided sizes are excessive
assert(parsed || noexcept_threw || j_noexcept.is_discarded());
assert(parsed || !recovered_without_errors);
}
// step 3: the other input adapters must read the data alike
-13
View File
@@ -26,10 +26,6 @@ array data, it performs the following steps:
The unit tests run the same checks on a fixed corpus (see the "UBJSON round-trip
invariants" test case), so keep both in sync.
Furthermore, it reads data with a SAX parser that recovers from every error
and checks that the events are balanced, that reading ends, and that it
reports an error exactly when from_ubjson() fails (see #3989).
Finally, it reads the data from a stream, a non-contiguous iterator range, and
a FILE*, which read byte by byte, and checks that each gives the same value or
error (id, message, byte position) as the pointer range, which takes the bulk
@@ -40,7 +36,6 @@ drivers.
*/
#include "fuzzer_common.hpp"
#include "fuzzer-recovering_checker.hpp"
namespace
{
@@ -58,9 +53,6 @@ struct read_ubjson
// see http://llvm.org/docs/LibFuzzer.html
extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
{
// recover from all errors, reading from memory and from a stream
const bool recovered_without_errors = check_recovering_parse(data, size, json::input_format_t::ubjson).errors == 0;
std::vector<uint8_t> const vec1(data, data + size);
// step 0: parse input without exceptions
@@ -79,9 +71,6 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
// without exceptions, the same input must give the same value
assert(!noexcept_threw && !j_noexcept.is_discarded() && same_value(j_noexcept, j1));
// the recovering parser must not have reported an error either
assert(recovered_without_errors);
try
{
// step 2.1: round trip without adding size annotations to container types
@@ -113,7 +102,6 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
{
// parse errors are ok, because input may be random bytes
assert(parsed || noexcept_threw || j_noexcept.is_discarded());
assert(parsed || !recovered_without_errors);
}
catch (const json::type_error&)
{
@@ -124,7 +112,6 @@ extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size)
{
// out of range errors may happen if provided sizes are excessive
assert(parsed || noexcept_threw || j_noexcept.is_discarded());
assert(parsed || !recovered_without_errors);
}
// step 3: the other input adapters must read the data alike
-154
View File
@@ -1,154 +0,0 @@
// __ _____ _____ _____
// __| | __| | | | JSON for Modern C++ (supporting code)
// | | |__ | | | | | | version 3.12.0
// |_____|_____|_____|_|___| https://github.com/nlohmann/json
//
// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
// SPDX-License-Identifier: MIT
#pragma once
#include <cassert>
#include <cstddef>
#include <cstdint>
#include <sstream>
#include <string>
#include <vector>
#include <nlohmann/json.hpp>
namespace
{
// a SAX parser that recovers from every error and checks that the events are
// balanced and that every key is followed by exactly one value
class recovering_checker : public nlohmann::json_sax<nlohmann::json>
{
public:
bool null() override
{
return value();
}
bool boolean(bool /*val*/) override
{
return value();
}
bool number_integer(number_integer_t /*val*/) override
{
return value();
}
bool number_unsigned(number_unsigned_t /*val*/) override
{
return value();
}
bool number_float(number_float_t /*val*/, const string_t& /*s*/) override
{
return value();
}
bool string(string_t& /*val*/) override
{
return value();
}
bool binary(binary_t& /*val*/) override
{
return value();
}
bool start_object(std::size_t /*elements*/) override
{
value();
stack.push_back('o');
return true;
}
bool key(string_t& /*val*/) override
{
++events;
assert(!stack.empty() && stack.back() == 'o');
stack.back() = 'v';
return true;
}
bool end_object() override
{
++events;
assert(!stack.empty() && stack.back() == 'o');
stack.pop_back();
return true;
}
bool start_array(std::size_t /*elements*/) override
{
value();
stack.push_back('a');
return true;
}
bool end_array() override
{
++events;
assert(!stack.empty() && stack.back() == 'a');
stack.pop_back();
return true;
}
bool parse_error(std::size_t /*position*/, const std::string& /*last_token*/, const nlohmann::detail::exception& /*ex*/) override
{
++errors;
return true;
}
bool complete() const
{
return stack.empty();
}
std::size_t events = 0;
std::size_t errors = 0;
private:
bool value()
{
++events;
if (!stack.empty())
{
// an array element, or the value of a key
assert(stack.back() != 'o');
if (stack.back() == 'v')
{
stack.back() = 'o';
}
}
return true;
}
// 'a' for an array, 'o' for an object that expects a key, 'v' for an
// object that expects the value of a key
std::vector<char> stack {}; // NOLINT(readability-redundant-member-init)
};
/// parses @a data with a recovering_checker from memory and from a stream,
/// checks that both see the same, that the events are balanced, and that the
/// number of errors is bounded, and returns the checker (see #3989)
inline recovering_checker check_recovering_parse(const std::uint8_t* data, const std::size_t size, const nlohmann::json::input_format_t format)
{
recovering_checker checker;
const bool ok = nlohmann::json::sax_parse(data, data + size, &checker, format);
assert(checker.complete());
assert(checker.errors <= size + 1);
assert(ok == (checker.errors == 0));
std::istringstream stream(std::string(reinterpret_cast<const char*>(data), size));
recovering_checker stream_checker;
assert(nlohmann::json::sax_parse(stream, &stream_checker, format) == ok);
assert(stream_checker.complete());
assert(stream_checker.events == checker.events);
assert(stream_checker.errors == checker.errors);
return checker;
}
} // namespace
-22
View File
@@ -15,7 +15,6 @@
#include <nlohmann/json_view.hpp>
#include <algorithm> // any_of
#include <cstddef> // size_t
#include <functional> // function
#include <random> // mt19937
#include <string> // string
@@ -89,27 +88,6 @@ inline bool has_duplicate_keys(const nlohmann::ordered_json_view& v)
});
}
// the message of an exception about the value at path (a JSON pointer, "" for
// the root) whose source bytes are [start, end): the context basic_json adds
// with JSON_DIAGNOSTICS and JSON_DIAGNOSTIC_POSITIONS, if enabled, then what
inline std::string message(const std::string& id, const std::string& path, std::size_t start, std::size_t end, const std::string& what)
{
#if JSON_DIAGNOSTIC_POSITIONS
const std::string positions = "(bytes " + std::to_string(start) + "-" + std::to_string(end) + ") ";
#else
const std::string positions;
static_cast<void>(start);
static_cast<void>(end);
#endif
#if JSON_DIAGNOSTICS
const std::string context = path.empty() ? "" : "(" + path + ") " + positions;
#else
const std::string& context = positions;
static_cast<void>(path);
#endif
return "[json.exception." + id + "] " + context + what;
}
#if !defined(JSON_NOEXCEPTION)
// the exception a call throws, or "" if it throws none
inline std::string exception_of_call(const std::function<void()>& f)
+1 -3
View File
@@ -131,13 +131,11 @@ struct SaxEventLogger : public nlohmann::json_sax<nlohmann::json>
{
errored = true;
events.push_back("parse_error(" + std::to_string(position) + ")");
return recover;
return false;
}
std::vector<std::string> events {}; // NOLINT(readability-redundant-member-init)
bool errored = false;
/// whether parse_error() asks the parser to recover from the error (see #3989)
bool recover = false;
};
struct SaxEventLoggerExitAfterStartObject : public SaxEventLogger
-40
View File
@@ -432,46 +432,6 @@ TEST_CASE("alternative string type")
CHECK(j2.dump() == R"({"/foo/0":"bar","/foo/1":"baz"})");
}
SECTION("error recovery")
{
// a SAX parser that recovers from every error (see #3989)
struct recovering_parser : nlohmann::detail::json_sax_dom_parser<alt_json>
{
explicit recovering_parser(alt_json& j)
: nlohmann::detail::json_sax_dom_parser<alt_json>(j, false)
{}
// sax_parse() calls the SAX parser's own parse_error(), so hiding
// the one of the base class is what recovering takes
// NOLINTNEXTLINE(bugprone-derived-method-shadowing-base-method)
bool parse_error(std::size_t /*unused*/, const std::string& /*unused*/, const nlohmann::detail::exception& /*unused*/)
{
++errors;
return true;
}
std::size_t errors = 0;
};
alt_json j;
recovering_parser sax(j);
// not inside CHECK(): MSVC reads the escape in a stringized raw string
const std::string input = R"([1., "a\qb", tru, {"k" 2}])";
CHECK(!alt_json::sax_parse(input, &sax));
CHECK(sax.errors == 4);
CHECK(j.dump() == R"([1,"aqb",null,{"k":2}])");
// a UBJSON high-precision number, a CBOR key that is not a string
alt_json u;
recovering_parser ubjson_sax(u);
CHECK(!alt_json::sax_parse(std::vector<std::uint8_t> {'[', 'H', 'i', 2, '1', '.', ']'}, &ubjson_sax, alt_json::input_format_t::ubjson));
CHECK(u.dump() == "[1]");
alt_json c;
recovering_parser cbor_sax(c);
CHECK(!alt_json::sax_parse(std::vector<std::uint8_t> {0xA2, 0x01, 0x02, 0x61, 'a', 0x03}, &cbor_sax, alt_json::input_format_t::cbor));
CHECK(c.dump() == R"({"a":3})");
}
SECTION("conversion between basic_json specializations (#2649)")
{
// explicit conversions are always possible
-77
View File
@@ -26,14 +26,11 @@ using nlohmann::ordered_json;
#include <limits>
#include <iostream>
#include <fstream>
#include <list>
#include <set>
#include "make_test_data_available.hpp"
#include "test_utils.hpp"
#include "sax_countdown.hpp"
#include "sax_event_loggers.hpp"
using utils::SaxCountdown;
using utils::SaxEventLogger;
// trait_test_arg and the "value_in_range_of trait" TEST_CASE_TEMPLATE_DEFINE
@@ -4749,77 +4746,3 @@ TEST_CASE("BJData optimized container type marker is validated for count 0")
CHECK(json::from_bjdata(vb) == json::binary({}));
}
}
TEST_CASE("BJData optimized arrays from contiguous and non-contiguous input")
{
// contiguous input reads the numbers of an array with a type marker in
// one loop, other input number by number; both must give the same events,
// value, or error for the complete document and for every truncation of
// it, and stop at every event a SAX handler rejects
json document =
{
{"u8", {0, 200, 255}},
{"i8", {-5, 100, -128}},
{"i16", {-30000, 1000, 30000}},
{"i32", {-2000000000, 5, 70000}},
{"i64", {-9000000000000000000LL, 1, 5000000000LL}},
{"f64", {1.5, -0.1, 2.5}},
{"pairs", {{1.25, -3.75}, {1.25, -3.75}, json::array()}},
{"mixed", {1, "a", 2.5, nullptr}},
{"u16", {40000, 1, 65535}},
{"u32", {4000000000U, 1, 70000}},
{"u64", {18446744073709551615ULL, 1, 5000000000ULL}},
{"nd", {{"_ArrayType_", "double"}, {"_ArraySize_", {2, 2}}, {"_ArrayData_", {1.5, -2.5, 3.5, -4.5}}}},
{"nd8", {{"_ArrayType_", "uint8"}, {"_ArraySize_", {3}}, {"_ArrayData_", {1, 2, 3}}}},
};
const std::vector<std::uint8_t> encoded = json::to_bjdata(document, true, true);
CHECK(json::from_bjdata(encoded) == document);
for (std::size_t length = 0; length <= encoded.size(); ++length)
{
CAPTURE(length)
const std::vector<std::uint8_t> contiguous(encoded.begin(), encoded.begin() + static_cast<std::ptrdiff_t>(length));
const std::list<std::uint8_t> bytewise(contiguous.begin(), contiguous.end());
CHECK(json::from_bjdata(contiguous, true, false).dump() == json::from_bjdata(bytewise.begin(), bytewise.end(), true, false).dump());
#if !defined(JSON_NOEXCEPTION)
std::string contiguous_error;
std::string bytewise_error;
try
{
json _;
_ = json::from_bjdata(contiguous);
}
catch (const json::exception& e)
{
contiguous_error = e.what();
}
try
{
json _;
_ = json::from_bjdata(bytewise.begin(), bytewise.end());
}
catch (const json::exception& e)
{
bytewise_error = e.what();
}
CHECK(contiguous_error == bytewise_error);
#endif
SaxEventLogger from_contiguous;
SaxEventLogger from_bytewise;
CHECK(json::sax_parse(contiguous, &from_contiguous, json::input_format_t::bjdata)
== json::sax_parse(bytewise.begin(), bytewise.end(), &from_bytewise, json::input_format_t::bjdata));
CHECK(from_contiguous.events == from_bytewise.events);
}
for (int events = 0; events <= 80; ++events)
{
CAPTURE(events)
const std::list<std::uint8_t> bytewise(encoded.begin(), encoded.end());
SaxCountdown from_contiguous(events);
SaxCountdown from_bytewise(events);
CHECK(json::sax_parse(encoded, &from_contiguous, json::input_format_t::bjdata)
== json::sax_parse(bytewise.begin(), bytewise.end(), &from_bytewise, json::input_format_t::bjdata));
}
}
+2 -224
View File
@@ -19,11 +19,9 @@
#include <nlohmann/json.hpp>
using nlohmann::json;
#include <array>
#include <cstdint>
#include <fstream>
#include <limits>
#include <list>
#include <sstream>
#include <vector>
#include "make_test_data_available.hpp"
@@ -1166,13 +1164,7 @@ TEST_CASE("BSON keys from contiguous and stream input")
// contiguous input reads a key up to its \x00-byte in one step, a stream
// reads it byte by byte; both must give the same value or error for the
// complete document and for every truncation of it
json j = {{"", true}, {"k", {1, 2, 3}}, {std::string(40, 'x'), {{"nested key", "value"}}}};
// keys and array element names ("0" to "11") that end at every position of
// an 8-byte word, which contiguous input scans at a time
for (std::size_t length = 1; length <= 17; ++length)
{
j[std::string(length, static_cast<char>('a' + length))] = json::array({0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, {11}});
}
const json j = {{"", true}, {"k", {1, 2, 3}}, {std::string(40, 'x'), {{"nested key", "value"}}}};
const std::vector<std::uint8_t> bson = json::to_bson(j);
CHECK(json::from_bson(bson) == j);
@@ -1192,7 +1184,7 @@ TEST_CASE("BSON keys from contiguous and stream input")
}
try
{
std::istringstream stream(std::string(reinterpret_cast<const char*>(input.data()), input.size()));
std::istringstream stream(std::string(input.begin(), input.end()));
from_stream = json::from_bson(stream).dump();
}
catch (const json::parse_error& e)
@@ -1200,72 +1192,8 @@ TEST_CASE("BSON keys from contiguous and stream input")
from_stream = e.what();
}
CHECK(from_vector == from_stream);
// error_handler_t::strict checks the array element names in place on
// contiguous input and byte by byte otherwise; the names are
// well-formed, so both give the same value or error as above
std::string strict_from_vector;
std::string strict_from_list;
try
{
strict_from_vector = json::from_bson(input, true, true, json::error_handler_t::strict).dump();
}
catch (const json::parse_error& e)
{
strict_from_vector = e.what();
}
try
{
const std::list<std::uint8_t> list(input.begin(), input.end());
strict_from_list = json::from_bson(list.begin(), list.end(), true, true, json::error_handler_t::strict).dump();
}
catch (const json::parse_error& e)
{
strict_from_list = e.what();
}
CHECK(strict_from_vector == from_vector);
CHECK(strict_from_list == from_vector);
}
}
TEST_CASE("BSON array element names")
{
// an array's element names are not part of the value, so they are not
// passed on; error_handler_t::strict still rejects ill-formed UTF-8 in them
const std::vector<std::uint8_t> bson =
{
0x1D, 0x00, 0x00, 0x00, // size (little endian)
0x04, 'a', 0x00, // entry: array "a"
0x15, 0x00, 0x00, 0x00, // size (little endian)
0x08, 0xFF, 0x00, 0x01, // entry: boolean named "\xFF", true
0x08, '1', '2', '3', '4', '5', '6', '7', '8', '9', 0x00, 0x00, // entry: boolean named "123456789", false
0x00, // end of the array
0x00 // end of the document
};
const json expected = {{"a", {true, false}}};
CHECK(json::from_bson(bson) == expected);
CHECK(json::from_bson(bson, true, true, json::error_handler_t::replace) == expected);
CHECK(json::from_bson(bson, true, true, json::error_handler_t::ignore) == expected);
json _;
CHECK_THROWS_WITH_AS(_ = json::from_bson(bson, true, true, json::error_handler_t::strict), "[json.exception.parse_error.113] parse error at byte 14: syntax error while parsing BSON key: invalid string: ill-formed UTF-8 byte", json::parse_error&);
std::istringstream stream(std::string(reinterpret_cast<const char*>(bson.data()), bson.size()));
CHECK(json::from_bson(stream) == expected);
// input that is not contiguous reads the names byte by byte and must reject
// the same name
const std::list<std::uint8_t> list(bson.begin(), bson.end());
CHECK_THROWS_WITH_AS(_ = json::from_bson(list.begin(), list.end(), true, true, json::error_handler_t::strict), "[json.exception.parse_error.113] parse error at byte 14: syntax error while parsing BSON key: invalid string: ill-formed UTF-8 byte", json::parse_error&);
// without exceptions, the rejected name ends the parse
CHECK(json::from_bson(bson, true, false, json::error_handler_t::strict).is_discarded());
// well-formed names pass the check without being stored
const json valid = {{"a", {true, false, {{"k", json::array({1, 2})}}}}};
CHECK(json::from_bson(json::to_bson(valid), true, true, json::error_handler_t::strict) == valid);
}
#endif
TEST_CASE("Negative size of binary value")
@@ -2339,153 +2267,3 @@ TEST_CASE("BSON large strings and binaries (chunked reader)")
}
}
}
TEST_CASE("BSON written in one pass and in two passes")
{
// to_bson() into a returned vector writes each size once the object or
// array is complete; through an output adapter, all sizes are computed
// first. Both must produce the same bytes and throw the same exceptions.
const auto two_pass = [](const json & j, const json::error_handler_t handler)
{
std::vector<std::uint8_t> result;
json::to_bson(j, result, handler);
return result;
};
const auto check_same = [&](const json & j, const json::error_handler_t handler)
{
CAPTURE(j.dump(-1, ' ', true, json::error_handler_t::replace))
CAPTURE(static_cast<int>(handler))
#if !defined(JSON_NOEXCEPTION)
std::string one_pass_error;
std::string two_pass_error;
std::vector<std::uint8_t> one;
std::vector<std::uint8_t> two;
try
{
one = json::to_bson(j, handler);
}
catch (const json::exception& e)
{
one_pass_error = e.what();
}
try
{
two = two_pass(j, handler);
}
catch (const json::exception& e)
{
two_pass_error = e.what();
}
CHECK(one_pass_error == two_pass_error);
CHECK(one == two);
#else
CHECK(json::to_bson(j, handler) == two_pass(j, handler));
#endif
};
const std::array<json::error_handler_t, 4> handlers = {{
json::error_handler_t::keep, json::error_handler_t::strict,
json::error_handler_t::replace, json::error_handler_t::ignore
}
};
SECTION("corpus")
{
for (const auto& value : utils::round_trip_corpus::values())
{
const json document = value.is_object() ? value : json{{"value", value}};
for (const auto handler : handlers)
{
check_same(document, handler);
}
}
}
SECTION("nesting and arrays of every length class")
{
json document = json::object();
document["empty object"] = json::object();
document["empty array"] = json::array();
for (const std::size_t n :
{
1u, 9u, 10u, 11u, 99u, 100u, 101u, 1000u
})
{
json numbers = json::array();
json mixed = json::array();
for (std::size_t i = 0; i < n; ++i)
{
numbers.push_back(i % 3 == 0 ? json(static_cast<double>(i) / 3) : json(static_cast<std::int64_t>(i) * 1000003));
switch (i % 4)
{
case 0:
mixed.push_back(json("s" + std::to_string(i)));
break;
case 1:
mixed.push_back(json::array({i, json::object({{"k", i}})}));
break;
case 2:
mixed.push_back(json(nullptr));
break;
default:
mixed.push_back(json(i % 2 == 0));
break;
}
}
document["numbers " + std::to_string(n)] = numbers;
document["mixed " + std::to_string(n)] = mixed;
document["nested " + std::to_string(n)] = json::object({{"a", numbers}, {"b", json::array({mixed, json::object()})}});
}
document["binary"] = json::binary({1, 2, 3}, 0x80);
document["binary old"] = json::binary({1, 2, 3}, 0x02);
for (const auto handler : handlers)
{
check_same(document, handler);
}
}
#if !defined(JSON_NOEXCEPTION)
// every entry here throws, which aborts without exceptions
SECTION("errors at every position")
{
// an entry that cannot be written, at each place among valid ones
const std::string bad_utf8 = "\xC3\x28";
const std::array<json, 7> bad_entries =
{
{
json::object({{std::string("a\0b", 3), 1}}),
json::object({{"s", bad_utf8}}),
json::object({{bad_utf8, 1}}),
json::array({1, 2, bad_utf8}),
json::object({{"b", json::binary({1}, 0x100)}}),
json::object({{"d", json(json::value_t::discarded)}}),
json::array({json(json::value_t::discarded)}),
}
};
for (const auto& bad : bad_entries)
{
for (std::size_t pos = 0; pos < 4; ++pos)
{
json document = json::object();
json list = json::array();
for (std::size_t i = 0; i < 4; ++i)
{
if (i == pos)
{
document["x" + std::to_string(i)] = bad;
list.push_back(bad);
}
document["n" + std::to_string(i)] = json::array({i, "str", json::object({{"k", i}})});
list.push_back(i);
}
document["list"] = list;
for (const auto handler : handlers)
{
check_same(document, handler);
}
}
}
}
#endif
}
-74
View File
@@ -29,9 +29,7 @@ using nlohmann::json;
#include "round_trip_corpus.hpp"
#include "test_utils.hpp"
#include "sax_countdown.hpp"
#include "sax_event_loggers.hpp"
using utils::SaxCountdown;
using utils::SaxEventLogger;
TEST_CASE("CBOR")
@@ -2448,78 +2446,6 @@ TEST_CASE("CBOR SAX parsing stops at every event")
CHECK(count_events(std::vector<std::uint8_t>({0xBF, 0x61, 'a', 0x9F, 0x01, 0xFF, 0xFF})) == 6);
}
TEST_CASE("CBOR arrays of scalars from contiguous and non-contiguous input")
{
// contiguous input reads the integers, floats, booleans, and nulls an
// array begins with in one loop, other input value by value; both must
// give the same events, value, or error for the complete document and for
// every truncation of it, and stop at every event a SAX handler rejects
const std::vector<std::uint8_t> cbor =
{
0xA3, // map(3)
0x61, 'a', // "a"
0x98, 0x16, // array(22)
0x00, 0x17, 0x18, 0xFF, 0x19, 0x01, 0x00, // 0, 23, 255, 256
0x1A, 0x00, 0x01, 0x00, 0x00, // 65536
0x1B, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, // 18446744073709551615
0x20, 0x37, 0x38, 0xFF, 0x39, 0x01, 0x00, // -1, -24, -256, -257
0x3A, 0x00, 0x01, 0x00, 0x00, // -65537
0x3B, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, // -18446744073709551616 (a float)
0x3B, 0x7F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, // -9223372036854775808
0xF4, 0xF5, 0xF6, // false, true, null
0xFA, 0x3F, 0xC0, 0x00, 0x00, // 1.5 (float32)
0xFB, 0xBF, 0xB9, 0x99, 0x99, 0x99, 0x99, 0x99, 0x9A, // -0.1
0xF9, 0x3C, 0x00, // 1.0 (float16, ends the loop)
0x01, 0x61, 'a', // 1, "a"
0xFB, 0x40, 0x04, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, // 2.5
0x61, 'b', // "b"
0x83, // array(3)
0x82, 0xFB, 0x3F, 0xF4, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, // [1.25,
0xFB, 0xC0, 0x0E, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, // -3.75]
0x82, 0xFB, 0x3F, 0xF4, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, // [1.25,
0xFB, 0xC0, 0x0E, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, // -3.75]
0x80, // []
0x61, 'c', // "c"
0x84, 0xA0, 0x01, 0xC1, 0x02, 0x81, 0x03 // [{}, 1, tag(1) 2, [3]]
};
const auto tags = json::cbor_tag_handler_t::ignore;
const json j = json::from_cbor(cbor, true, true, tags);
CHECK(j["a"].size() == 22);
CHECK(j["a"][5] == 18446744073709551615u);
CHECK(j["a"][11].is_number_float());
CHECK(j["a"][11] == -18446744073709551616.0);
CHECK(j["a"][12] == (std::numeric_limits<std::int64_t>::min)());
CHECK(j["a"][17] == -0.1);
CHECK(j["b"] == json::parse("[[1.25, -3.75], [1.25, -3.75], []]"));
CHECK(j["c"] == json::parse("[{}, 1, 2, [3]]"));
for (std::size_t length = 0; length <= cbor.size(); ++length)
{
CAPTURE(length)
const std::vector<std::uint8_t> contiguous(cbor.begin(), cbor.begin() + static_cast<std::ptrdiff_t>(length));
const std::list<std::uint8_t> bytewise(contiguous.begin(), contiguous.end());
CHECK(json::from_cbor(contiguous, true, false, tags).dump() == json::from_cbor(bytewise.begin(), bytewise.end(), true, false, tags).dump());
SaxEventLogger from_contiguous;
SaxEventLogger from_bytewise;
CHECK(json::sax_parse(contiguous, &from_contiguous, json::input_format_t::cbor, true, false, false, tags)
== json::sax_parse(bytewise.begin(), bytewise.end(), &from_bytewise, json::input_format_t::cbor, true, false, false, tags));
CHECK(from_contiguous.events == from_bytewise.events);
}
for (int events = 0; events <= 60; ++events)
{
CAPTURE(events)
const std::list<std::uint8_t> bytewise(cbor.begin(), cbor.end());
SaxCountdown from_contiguous(events);
SaxCountdown from_bytewise(events);
CHECK(json::sax_parse(cbor, &from_contiguous, json::input_format_t::cbor, true, false, false, tags)
== json::sax_parse(bytewise.begin(), bytewise.end(), &from_bytewise, json::input_format_t::cbor, true, false, false, tags));
}
}
TEST_CASE("CBOR indefinite-length strings do not recurse per chunk")
{
// Reading an indefinite-length string or byte array used to call itself
-706
View File
@@ -2953,651 +2953,6 @@ TEST_CASE("diagnostic positions: value lifetime, input adapters, and SAX")
}
#endif
namespace
{
/// builds a value like json::parse(), but asks the parser to recover from
/// errors (see #3989), and checks that the events it receives are balanced
class RecoveringDomParser
{
public:
explicit RecoveringDomParser(json& j, std::size_t max_errors_ = static_cast<std::size_t>(-1))
: dom(j, false)
, max_errors(max_errors_)
{}
bool null()
{
value();
return dom.null();
}
bool boolean(bool val)
{
value();
return dom.boolean(val);
}
bool number_integer(json::number_integer_t val)
{
value();
return dom.number_integer(val);
}
bool number_unsigned(json::number_unsigned_t val)
{
value();
return dom.number_unsigned(val);
}
bool number_float(json::number_float_t val, const std::string& s)
{
value();
return dom.number_float(val, s);
}
bool string(std::string& val)
{
value();
return dom.string(val);
}
bool binary(json::binary_t& val)
{
value();
return dom.binary(val);
}
bool start_object(std::size_t elements)
{
value();
stack.push_back('o');
return dom.start_object(elements);
}
bool key(std::string& val)
{
++events;
if (stack.empty() || stack.back() != 'o')
{
well_formed = false;
return false;
}
stack.back() = 'v';
return dom.key(val);
}
bool end_object()
{
++events;
if (stack.empty() || stack.back() != 'o')
{
well_formed = false;
return false;
}
stack.pop_back();
return dom.end_object();
}
bool start_array(std::size_t elements)
{
value();
stack.push_back('a');
return dom.start_array(elements);
}
bool end_array()
{
++events;
if (stack.empty() || stack.back() != 'a')
{
well_formed = false;
return false;
}
stack.pop_back();
return dom.end_array();
}
bool parse_error(std::size_t /*unused*/, const std::string& /*unused*/, const json::exception& ex)
{
errors.emplace_back(ex.what());
return errors.size() < max_errors;
}
/// whether the events were balanced and every key was followed by a value
bool balanced() const
{
return well_formed && stack.empty();
}
/// builds the value
nlohmann::detail::json_sax_dom_parser<json> dom;
std::vector<std::string> errors {}; // NOLINT(readability-redundant-member-init)
std::size_t events = 0;
/// the open containers: 'a' for an array, 'o' for an object that expects
/// a key, 'v' for an object that expects the value of a key
std::vector<char> stack {}; // NOLINT(readability-redundant-member-init)
bool well_formed = true;
std::size_t max_errors;
private:
/// a value is passed: it is an array element, or the value of a key
void value()
{
++events;
if (!stack.empty())
{
if (stack.back() == 'v')
{
stack.back() = 'o';
}
else if (stack.back() == 'o')
{
// a value without a key
well_formed = false;
}
}
}
};
struct RecoveryResult
{
json value;
std::vector<std::string> errors;
std::size_t events;
bool ok;
bool balanced;
};
template<typename InputType>
RecoveryResult parse_recovering(InputType&& input, const bool strict = true,
const bool ignore_comments = false, const bool ignore_trailing_commas = false)
{
json j;
RecoveringDomParser sax(j);
const bool ok = json::sax_parse(std::forward<InputType>(input), &sax, json::input_format_t::json,
strict, ignore_comments, ignore_trailing_commas);
return {j, sax.errors, sax.events, ok, sax.balanced()};
}
/// stops after a number of events, but recovers from errors
class RecoveringCountdown
{
public:
explicit RecoveringCountdown(const int count)
: countdown(count)
{}
bool null()
{
return countdown.null();
}
bool boolean(bool val)
{
return countdown.boolean(val);
}
bool number_integer(json::number_integer_t val)
{
return countdown.number_integer(val);
}
bool number_unsigned(json::number_unsigned_t val)
{
return countdown.number_unsigned(val);
}
bool number_float(json::number_float_t val, const std::string& s)
{
return countdown.number_float(val, s);
}
bool string(std::string& val)
{
return countdown.string(val);
}
bool binary(json::binary_t& val)
{
return countdown.binary(val);
}
bool start_object(std::size_t elements)
{
return countdown.start_object(elements);
}
bool key(std::string& val)
{
return countdown.key(val);
}
bool end_object()
{
return countdown.end_object();
}
bool start_array(std::size_t elements)
{
return countdown.start_array(elements);
}
bool end_array()
{
return countdown.end_array();
}
bool parse_error(std::size_t /*position*/, const std::string& /*last_token*/, const json::exception& /*ex*/) // NOLINT(readability-convert-member-functions-to-static)
{
return true;
}
private:
SaxCountdown countdown;
};
/// a repaired input: the value it is repaired to, and the number of errors
struct Repair
{
const char* input;
const char* expected;
std::size_t errors;
};
} // namespace
TEST_CASE("parser error recovery (#3989)")
{
SECTION("repairs")
{
const std::vector<Repair> repairs =
{
// a missing separator is inserted
{"[1 2]", "[1,2]", 1},
{R"({"a":1 "b":2})", R"({"a":1,"b":2})", 1},
{R"({"a" 1})", R"({"a":1})", 1},
{"[1 tru 2]", "[1,null,2]", 2},
{R"({"a" "b": 1})", R"({"a":"b"})", 2},
// a missing value is null in an object; in an array, a ',' stands
// for null, while an array that ends there just ends
{R"({"a":})", R"({"a":null})", 1},
{R"({"a"})", R"({"a":null})", 1},
{R"({"a","b":1})", R"({"a":null,"b":1})", 1},
{"[1,,2]", "[1,null,2]", 1},
{"[,1]", "[null,1]", 1},
{"[1,]", "[1]", 1},
{"[1,2,3,]", "[1,2,3]", 1},
{R"({"a":1,})", R"({"a":1})", 1},
// a broken string keeps what can be read
{R"(["a\qb"])", R"(["aqb"])", 1},
{R"({"na\me":1})", R"({"name":1})", 1},
{"[\"\xFF\"]", R"(["\uFFFD"])", 1},
{"[\"a\xC3(\"]", R"(["a\uFFFD("])", 1},
{"[\"\xE2\x82\"]", R"(["\uFFFD"])", 1},
{"[\"\xC3\\\\\", 1]", R"(["\uFFFD\\",1])", 1},
{R"(["\u12"])", R"(["\uFFFD"])", 1},
{R"(["\u12G4"])", R"(["\uFFFDG4"])", 1},
{R"(["\uDC00x"])", R"(["\uFFFDx"])", 1},
{R"(["\uD800x"])", R"(["\uFFFDx"])", 1},
{R"(["\uD800\u0041"])", R"(["\uFFFDA"])", 1},
{R"(["\uD800\uD800\uDC00"])", R"(["\uFFFD\uD800\uDC00"])", 1},
{R"(["\uD800\uD800\uD800x"])", R"(["\uFFFD\uFFFD\uFFFDx"])", 1},
{
R"(["\uD800\"x", 1])", R"(["\uFFFD\"x",1])", 1
},
{R"(["\uD800\q"])", R"(["\uFFFDq"])", 1},
{"[\"a\tb\"]", R"(["a\tb"])", 1},
{R"(["a\qb\u0041\x"])", R"(["aqbAx"])", 1},
// a broken number keeps its longest valid prefix
{"[1.]", "[1]", 1},
{"[-2.]", "[-2]", 1},
{"[1.5e]", "[1.5]", 1},
{"[1e+]", "[1]", 1},
{"[1.x2, 3]", "[1,3]", 1},
// what cannot be read at all is null
{"[1,NaN,3]", "[1,null,3]", 1},
{"[tru]", "[null]", 1},
{"[-]", "[null]", 1},
{R"({"a":Infinity})", R"({"a":null})", 1},
// a stray token is dropped
{"[:1]", "[1]", 1},
{R"(["a":1])", R"(["a",1])", 1},
{R"({"a"::1})", R"({"a":1})", 1},
// a member that cannot be read is skipped
{R"({1:2,"b":3})", R"({"b":3})", 1},
{R"({"a":1 2})", R"({"a":1})", 1},
{R"({,"a":1})", R"({"a":1})", 1},
{R"({"a":1,,"b":2})", R"({"a":1,"b":2})", 1},
{"{a:1}", "{}", 1},
{R"({"a":1 [1,{"b":2}], "c":3})", R"({"a":1,"c":3})", 1},
{R"([{1}, "a"])", R"([{},"a"])", 1},
// a wrong closing bracket closes the innermost container
{R"({"a":[1,2}, "b":3})", R"({"a":[1,2],"b":3})", 1},
{R"([{"a":1], 2])", R"([{"a":1},2])", 1},
{"{]", "{}", 1},
{"[}", "[]", 1},
// the end of the input closes all containers
{R"({"a":[1,2)", R"({"a":[1,2]})", 1},
{"[", "[]", 1},
{"{", "{}", 1},
{R"({"a")", R"({"a":null})", 1},
{R"({"a":)", R"({"a":null})", 1},
{"[1,", "[1]", 1},
{"[[[1", "[[[1]]]", 1},
{
R"(["abc)", R"(["abc"])", 2
},
{"[1,tr", "[1,null]", 2},
{"\"abc", "\"abc\"", 1},
{"[\"ab\ncd\"]", R"(["ab",null,"]"])", 4},
// what comes before the top-level value is skipped
{")]}'\n{\"a\":1}", R"({"a":1})", 1},
{R"(data: {"a":1})", R"({"a":1})", 1},
{"\xEF\xBB[1]", "[1]", 1},
// what comes after it is an error that ends parsing
{R"({"a":1}})", R"({"a":1})", 1},
{"[1}]", "[1]", 2},
{"[1] [2]", "[1]", 1},
};
for (const auto& repair : repairs)
{
CAPTURE(repair.input)
const auto result = parse_recovering(std::string(repair.input));
CHECK(!result.ok);
CHECK(result.balanced);
CHECK(result.value == json::parse(repair.expected));
CHECK(result.errors.size() == repair.errors);
}
}
SECTION("number overflow")
{
const auto result = parse_recovering(std::string("[1e999,-1e999]"));
CHECK(!result.ok);
CHECK(result.balanced);
CHECK(result.errors.size() == 2);
CHECK(result.errors[0] == "[json.exception.out_of_range.406] number overflow parsing '1e999'");
REQUIRE(result.value.size() == 2);
CHECK(result.value[0].is_number_float());
CHECK(result.value[0].get<double>() == std::numeric_limits<double>::infinity());
CHECK(result.value[1].get<double>() == -std::numeric_limits<double>::infinity());
// the SAX parser gets the number's text
SaxEventLogger logger;
logger.recover = true;
CHECK(!json::sax_parse("1e999", &logger));
CHECK(logger.events == std::vector<std::string>({"parse_error(5)", "number_float(1e999)"}));
}
SECTION("nothing to recover")
{
for (const std::string s :
{
"", " ", "]", "tru", "NaN", ",:", "/* comment"
})
{
CAPTURE(s)
const auto result = parse_recovering(s, true, true);
CHECK(!result.ok);
CHECK(result.balanced);
CHECK(result.events == 0);
CHECK(result.value == nullptr);
CHECK(result.errors.size() == 1);
}
}
SECTION("error messages")
{
// the first error is reported as without recovery
for (const std::string s :
{
"[1 2]", R"({"a":1 "b":2})", R"({"a" 1})", R"({"a":})", "[1,]", "[1.]",
R"(["a\qb"])", "[1e999]", "{1:2}", R"({"a":[1,2}})", "[1,", "[1] [2]", "{a:1}"
})
{
CAPTURE(s)
const auto result = parse_recovering(s);
REQUIRE(!result.errors.empty());
json _;
CHECK_THROWS_WITH_STD_STR(_ = json::parse(s), result.errors.front());
}
// the token of an error begins where the previous error was
const auto result = parse_recovering(std::string("[tru, fals, nul]"));
CHECK(result.errors == std::vector<std::string>(
{
"[json.exception.parse_error.101] parse error at line 1, column 5: syntax error while parsing value - invalid literal; last read: '[tru,'",
"[json.exception.parse_error.101] parse error at line 1, column 11: syntax error while parsing value - invalid literal; last read: ', fals,'",
"[json.exception.parse_error.101] parse error at line 1, column 16: syntax error while parsing value - invalid literal; last read: ', nul]'"
}));
CHECK(result.value == json::parse("[null,null,null]"));
}
SECTION("events")
{
// see #4522
SaxEventLogger logger;
logger.recover = true;
CHECK(!json::sax_parse(R"([{1}, "a"])", &logger));
CHECK(logger.events == std::vector<std::string>(
{
"start_array()", "start_object()", "parse_error(3)", "end_object()", "string(a)", "end_array()"
}));
}
SECTION("options")
{
SECTION("strict")
{
const auto result = parse_recovering(std::string("[1 2] [3]"), false);
CHECK(!result.ok);
CHECK(result.value == json::parse("[1,2]"));
CHECK(result.errors.size() == 1);
}
SECTION("ignore_trailing_commas")
{
for (const std::string s :
{
"[1,]", R"({"a":1,})", "[[1,],]"
})
{
CAPTURE(s)
const auto result = parse_recovering(s, true, false, true);
CHECK(result.ok);
CHECK(result.errors.empty());
}
auto result = parse_recovering(std::string("[1,,]"), true, false, true);
CHECK(result.value == json::parse("[1,null]"));
CHECK(result.errors.size() == 1);
result = parse_recovering(std::string(R"({"a":1,,})"), true, false, true);
CHECK(result.value == json::parse(R"({"a":1})"));
CHECK(result.errors.size() == 1);
}
SECTION("ignore_comments")
{
auto result = parse_recovering(std::string("[1 /* one */ 2]"), true, true);
CHECK(result.value == json::parse("[1,2]"));
CHECK(result.errors.size() == 1);
// a comment that is not closed runs to the end of the input, which
// is not reported again
result = parse_recovering(std::string("[1, 2 /* unterminated"), true, true);
CHECK(result.balanced);
CHECK(result.value == json::parse("[1,2]"));
CHECK(result.errors.size() == 1);
// a '/' that does not begin a comment is garbage
result = parse_recovering(std::string("[1, /x, 2]"), true, true);
CHECK(result.balanced);
CHECK(result.value == json::parse("[1,null,2]"));
CHECK(result.errors.size() == 1);
}
}
SECTION("null bytes")
{
// a null byte ends the input, unless JSON_STRICT_NUL_HANDLING is set
const auto result = parse_recovering(std::string("[1,\0x", 5));
CHECK(result.balanced);
CHECK(!result.ok);
#ifdef JSON_TEST_STRICT_NUL_HANDLING_ENABLED
CHECK(result.value == json::parse("[1,null]"));
#else
CHECK(result.value == json::parse("[1]"));
CHECK(result.errors.size() == 1);
#endif
const auto in_string = parse_recovering(std::string("[\"a\0b\"]", 7));
CHECK(in_string.balanced);
#ifdef JSON_TEST_STRICT_NUL_HANDLING_ENABLED
CHECK(in_string.value == json::array({std::string("a\0b", 3)}));
#else
CHECK(in_string.value == json::parse(R"(["a"])"));
#endif
}
SECTION("the SAX parser stops recovering")
{
json j;
RecoveringDomParser sax(j, 2);
CHECK(!json::sax_parse("[1 2 3 4 5]", &sax));
CHECK(sax.errors.size() == 2);
// an error at a delimiter that an invalid token consumed is reported
// to the SAX parser, too
json j2;
RecoveringDomParser sax2(j2, 2);
CHECK(!json::sax_parse("[tru}, 1]", &sax2));
CHECK(sax2.errors.size() == 2);
}
SECTION("an event stops parsing during a repair")
{
// start_object() and key() are passed, then null() for the missing
// value returns false
RecoveringCountdown countdown(2);
CHECK(!json::sax_parse(R"({"a":})", &countdown));
// the end of the input: end_array() for the second array returns false
RecoveringCountdown countdown2(4);
CHECK(!json::sax_parse("[[1", &countdown2));
}
SECTION("input adapters")
{
// the lexer reads contiguous and streaming input differently, and it
// puts back a character that ended an invalid token
for (const std::string s :
{
"[1 2]", "[tru}, 1]", R"({"a" "b\q", "c":[1.x, 2}})", "[\"\xFF\xC3(\", -, 1e+]", "{a:1,\"b\":2", ")]}' [1]"
})
{
CAPTURE(s)
const auto reference = parse_recovering(s);
CHECK(reference.balanced);
const auto from_c_string = parse_recovering(s.c_str());
CHECK(from_c_string.value == reference.value);
CHECK(from_c_string.errors == reference.errors);
const std::list<char> l(s.begin(), s.end());
json j;
RecoveringDomParser sax(j);
CHECK(!json::sax_parse(l.begin(), l.end(), &sax));
CHECK(j == reference.value);
CHECK(sax.errors == reference.errors);
std::istringstream ss(s);
const auto from_stream = parse_recovering(ss);
CHECK(from_stream.value == reference.value);
CHECK(from_stream.errors == reference.errors);
}
}
SECTION("long runs of errors")
{
// no error may copy all the input read before it
const auto closing = parse_recovering("[" + std::string(100000, '}'));
CHECK(closing.balanced);
CHECK(closing.value == json::array());
const auto garbage = parse_recovering("[" + std::string(100000, 'x') + "]");
CHECK(garbage.balanced);
CHECK(garbage.errors.size() == 1);
const auto commas = parse_recovering("{" + std::string(100000, ',') + "}");
CHECK(commas.balanced);
CHECK(commas.value == json::object());
}
SECTION("mutations of valid input")
{
// whatever the input, the events are balanced, every error is reported
// at most once, and valid input is parsed as usual
const std::vector<std::string> documents =
{
R"({"name": "value", "list": [1, -2.5, true, null, {"x": [[]]}], "e": "\u00e9"})",
R"([{"a": [1, 2, {"b": "c"}]}, [], {}, "\ud83d\ude00", 1e10])",
"{\"\xC3\xA9\": \"\xF0\x9F\x98\x80\"}",
R"( {"k" : [ "v" , 0 ] } )",
};
// each character that can be inserted, including a null byte
const std::string insertions("[]{},:\"x\\\0\xFF", 11);
std::vector<std::string> inputs;
for (const auto& doc : documents)
{
for (std::size_t i = 0; i <= doc.size(); ++i)
{
inputs.push_back(doc.substr(0, i));
if (i < doc.size())
{
inputs.push_back(doc.substr(0, i) + doc.substr(i + 1));
}
for (const char c : insertions)
{
inputs.push_back(doc.substr(0, i) + c + doc.substr(i));
}
}
}
for (const auto& s : inputs)
{
CAPTURE(s)
const auto result = parse_recovering(s);
CHECK(result.balanced);
CHECK(result.errors.size() <= s.size() + 1);
CHECK(result.events <= (4 * s.size()) + 4);
if (json::accept(s))
{
CHECK(result.ok);
CHECK(result.errors.empty());
CHECK(result.value == json::parse(s));
}
else
{
CHECK(!result.ok);
CHECK(!result.errors.empty());
}
}
}
}
TEST_CASE("accept() agrees with parse() on floats around the overflow limit")
{
// accept() does not convert a float that cannot overflow; the tokens
@@ -3627,67 +2982,6 @@ TEST_CASE("accept() agrees with parse() on floats around the overflow limit")
}
}
TEST_CASE("accept() agrees with parse() on strings, which it checks without copying them")
{
// accept() scans strings and object keys with a separate instantiation of
// the string scanner that does not copy them into the token buffer; it
// must accept and reject exactly what parse() does
const std::vector<std::string> pieces =
{
"", "abc", R"(\"\\\/\b\f\n\r\t)", R"(\u0041\u00e4\u20AC)", "\\ud83d\\ude00", "\\uD83D\\uDE00",
"\\ud800", "\\udc00", "\\ud800\\u0041", "\\ud800x", "\\udbff\\udfff", "\\u12", "\\u12g4", "\\x", "\\",
"\xc3\xa4", "\xe2\x82\xac", "\xf0\x9f\x98\x80", "\xf4\x8f\xbf\xbf", "\xc0\xaf", "\xed\xa0\x80", "\xf4\x90\x80\x80",
"\xff", "\xe2\x82", "\xc3", "\x01", "\x1f", "\x7f", "\n", std::string(1, '\0')
};
for (std::size_t run = 0; run <= 70; ++run)
{
for (const std::string& piece : pieces)
{
// a run of ordinary characters puts the piece at every offset of the bulk scanners
const std::string str = std::string(run, 'a') + piece;
std::string opened = "\"";
opened += str;
std::string closed = opened;
closed += '"';
std::vector<std::string> texts;
texts.push_back(closed);
texts.emplace_back("[");
texts.back() += closed;
texts.back() += ']';
texts.emplace_back("{");
texts.back() += closed;
texts.back() += ":1}";
texts.emplace_back("{\"k\":");
texts.back() += closed;
texts.back() += ',';
texts.back() += closed;
texts.back() += ":[";
texts.back() += closed;
texts.back() += "]}";
texts.emplace_back("[");
texts.back() += opened;
for (const std::string& text : texts)
{
CAPTURE(text)
for (int options = 0; options < 4; ++options)
{
const bool ignore_comments = (options & 1) != 0;
const bool ignore_trailing_commas = (options & 2) != 0;
const bool parses = !json::parse(text, nullptr, false, ignore_comments, ignore_trailing_commas).is_discarded();
CHECK(json::accept(text, ignore_comments, ignore_trailing_commas) == parses);
std::istringstream stream(text);
CHECK(json::accept(stream, ignore_comments, ignore_trailing_commas) == parses);
const std::list<char> chars(text.begin(), text.end());
CHECK(json::accept(chars.begin(), chars.end(), ignore_comments, ignore_trailing_commas) == parses);
}
}
}
}
}
namespace
{
// a DOM parser that reports parse errors without throwing, as in the sax_no_exception example
-1
View File
@@ -403,7 +403,6 @@ TEST_CASE("guard page")
CHECK(nlohmann::detail::find_string_special(p, n) <= n);
CHECK(nlohmann::detail::find_ascii_copyable_run(p, n) <= n);
CHECK(nlohmann::detail::valid_utf8_prefix(p, n) <= n);
CHECK(nlohmann::detail::valid_utf8_string(p, n) == (nlohmann::detail::valid_utf8_prefix(p, n) == n));
#if defined(JSON_USE_SIMDUTF) && defined(JSON_HAS_CPP_17)
CHECK(nlohmann::detail::find_string_delimiter(p, n) <= n);
#endif
+71 -39
View File
@@ -225,7 +225,7 @@ TEST_CASE("json_view")
#if JSON_DIAGNOSTICS
// the parents are set, so errors name the path
const json m = materialized<json_document>(R"({"a":{"b":[1]}})");
CHECK_THROWS_WITH_AS(m.at("a").at("b").at(0).at("x"), json_view_test::message("type_error.304", "/a/b/0", 11, 12, "cannot use at() with number").c_str(), json::type_error&);
CHECK_THROWS_WITH_AS(m.at("a").at("b").at(0).at("x"), "[json.exception.type_error.304] (/a/b/0) cannot use at() with number", json::type_error&);
#endif
}
@@ -750,6 +750,19 @@ std::string exception_of(F f)
return "";
}
// the exception a call on a basic_json value throws, without the byte range of
// the value (JSON_DIAGNOSTIC_POSITIONS), which the exceptions of views lack
template<typename F>
std::string json_exception_of(F f)
{
std::string what = exception_of(f);
const std::size_t at = what.find("] (bytes ");
if (at != std::string::npos)
{
what.erase(at + 2, what.find(") ", at) - at);
}
return what;
}
#endif
// compares a view with the ordered_json value materialize() gives for it:
@@ -958,25 +971,24 @@ TEST_CASE("json_view element access and iteration")
#if !defined(JSON_NOEXCEPTION)
if (!j.is_object())
{
CHECK(exception_of([&] { static_cast<void>(v["a"]); }) == exception_of([&] { static_cast<void>(j["a"]); }));
CHECK(exception_of([&] { static_cast<void>(v["a"]); }) == json_exception_of([&] { static_cast<void>(j["a"]); }));
}
if (!j.is_array())
{
CHECK(exception_of([&] { static_cast<void>(v[0]); }) == exception_of([&] { static_cast<void>(j[0]); }));
CHECK(exception_of([&] { static_cast<void>(v[0]); }) == json_exception_of([&] { static_cast<void>(j[0]); }));
}
CHECK(exception_of([&] { static_cast<void>(v.at("a")); }) == exception_of([&] { static_cast<void>(j.at("a")); }));
CHECK(exception_of([&] { static_cast<void>(v.at("missing")); }) == exception_of([&] { static_cast<void>(j.at("missing")); }));
CHECK(exception_of([&] { static_cast<void>(v.at(0)); }) == exception_of([&] { static_cast<void>(j.at(0)); }));
CHECK(exception_of([&] { static_cast<void>(v.at(5)); }) == exception_of([&] { static_cast<void>(j.at(5)); }));
CHECK(exception_of([&] { static_cast<void>(v.at("a")); }) == json_exception_of([&] { static_cast<void>(j.at("a")); }));
CHECK(exception_of([&] { static_cast<void>(v.at("missing")); }) == json_exception_of([&] { static_cast<void>(j.at("missing")); }));
CHECK(exception_of([&] { static_cast<void>(v.at(0)); }) == json_exception_of([&] { static_cast<void>(j.at(0)); }));
CHECK(exception_of([&] { static_cast<void>(v.at(5)); }) == json_exception_of([&] { static_cast<void>(j.at(5)); }));
if (!(j.is_object() && j.empty())) // (key() of an end iterator)
{
// (without the context of basic_json: view iterators do not know the value they iterate over)
CHECK(exception_of([&] { static_cast<void>(v.begin().key()); }) == (j.is_object() ? "" : "[json.exception.invalid_iterator.207] cannot use key() for non-object iterators"));
CHECK(exception_of([&] { static_cast<void>(v.begin().key()); }) == json_exception_of([&] { static_cast<void>(j.begin().key()); }));
}
if (!j.empty() || j.is_null())
{
CHECK(exception_of([&] { static_cast<void>(v.front()); }) == exception_of([&] { static_cast<void>(j.front()); }));
CHECK(exception_of([&] { static_cast<void>(v.back()); }) == exception_of([&] { static_cast<void>(j.back()); }));
CHECK(exception_of([&] { static_cast<void>(v.front()); }) == json_exception_of([&] { static_cast<void>(j.front()); }));
CHECK(exception_of([&] { static_cast<void>(v.back()); }) == json_exception_of([&] { static_cast<void>(j.back()); }));
}
#endif
CHECK(v.contains("a") == j.contains("a"));
@@ -988,8 +1000,8 @@ TEST_CASE("json_view element access and iteration")
const json_document d = json_document::parse(R"({"a":[]})");
CHECK(d.root()["b"].is_discarded());
CHECK(d.root()["a"][0].is_discarded());
CHECK_THROWS_WITH_AS(d.root()["a"].front(), json_view_test::message("invalid_iterator.214", "/a", 5, 7, "cannot get value").c_str(), json::invalid_iterator&);
CHECK_THROWS_WITH_AS(d.root()["a"].back(), json_view_test::message("invalid_iterator.214", "/a", 5, 7, "cannot get value").c_str(), json::invalid_iterator&);
CHECK_THROWS_WITH_AS(d.root()["a"].front(), "[json.exception.invalid_iterator.214] cannot get value", json::invalid_iterator&);
CHECK_THROWS_WITH_AS(d.root()["a"].back(), "[json.exception.invalid_iterator.214] cannot get value", json::invalid_iterator&);
const json_view invalid{};
CHECK(invalid.begin() == invalid.end());
CHECK(std::string(invalid.type_name()) == "discarded");
@@ -1020,10 +1032,10 @@ TEST_CASE("json_view element access and iteration")
CHECK(v[json_view::json_pointer("/a/b/5")][0].is_discarded());
#if !defined(JSON_NOEXCEPTION)
// type errors on values that are not discarded stay
CHECK_THROWS_WITH_AS(v[0], json_view_test::message("type_error.305", "", 0, 29, "cannot use operator[] with a numeric argument with object").c_str(), json::type_error&);
CHECK_THROWS_WITH_AS(v["a"]["b"]["c"], json_view_test::message("type_error.305", "/a/b", 10, 17, "cannot use operator[] with a string argument with array").c_str(), json::type_error&);
CHECK_THROWS_WITH_AS(v["s"]["c"], json_view_test::message("type_error.305", "/s", 23, 28, "cannot use operator[] with a string argument with string").c_str(), json::type_error&);
CHECK_THROWS_WITH_AS(v["s"][0], json_view_test::message("type_error.305", "/s", 23, 28, "cannot use operator[] with a numeric argument with string").c_str(), json::type_error&);
CHECK_THROWS_WITH_AS(v[0], "[json.exception.type_error.305] cannot use operator[] with a numeric argument with object", json::type_error&);
CHECK_THROWS_WITH_AS(v["a"]["b"]["c"], "[json.exception.type_error.305] cannot use operator[] with a string argument with array", json::type_error&);
CHECK_THROWS_WITH_AS(v["s"]["c"], "[json.exception.type_error.305] cannot use operator[] with a string argument with string", json::type_error&);
CHECK_THROWS_WITH_AS(v["s"][0], "[json.exception.type_error.305] cannot use operator[] with a numeric argument with string", json::type_error&);
CHECK_THROWS_AS(v["a"]["b"][0]["c"], json::type_error&);
CHECK_THROWS_AS(v["s"][json_view::json_pointer("/x")], json::out_of_range&);
// at() keeps throwing on a discarded view
@@ -1098,9 +1110,9 @@ TEST_CASE("json_view element access and iteration")
CHECK(v[(std::numeric_limits<std::size_t>::max)()].is_discarded());
CHECK(v[std::numeric_limits<int>::max()].is_discarded());
#if !defined(JSON_NOEXCEPTION)
CHECK_THROWS_WITH_AS(v.at(3), json_view_test::message("out_of_range.401", "", 0, 10, "array index 3 is out of range").c_str(), json::out_of_range&);
CHECK_THROWS_WITH_AS(v.at(3u), json_view_test::message("out_of_range.401", "", 0, 10, "array index 3 is out of range").c_str(), json::out_of_range&);
CHECK_THROWS_WITH_AS(v.at(std::int64_t(3)), json_view_test::message("out_of_range.401", "", 0, 10, "array index 3 is out of range").c_str(), json::out_of_range&);
CHECK_THROWS_WITH_AS(v.at(3), "[json.exception.out_of_range.401] array index 3 is out of range", json::out_of_range&);
CHECK_THROWS_WITH_AS(v.at(3u), "[json.exception.out_of_range.401] array index 3 is out of range", json::out_of_range&);
CHECK_THROWS_WITH_AS(v.at(std::int64_t(3)), "[json.exception.out_of_range.401] array index 3 is out of range", json::out_of_range&);
CHECK_THROWS_AS(v.at(-1), json::out_of_range&);
CHECK_THROWS_AS(v.at(-1L), json::out_of_range&);
CHECK_THROWS_AS(v.at(std::int64_t(-1)), json::out_of_range&);
@@ -1234,6 +1246,26 @@ TEST_CASE("json_view element access and iteration")
namespace
{
#if !defined(JSON_NOEXCEPTION)
// an exception message without the context that basic_json adds with
// JSON_DIAGNOSTICS ("(/path) ") and JSON_DIAGNOSTIC_POSITIONS ("(bytes 1-2) ");
// the view's exceptions have no such context
std::string without_path(std::string msg)
{
for (const char* prefix :
{"] (/", "] (bytes "
})
{
const std::size_t open = msg.find(prefix);
if (open != std::string::npos)
{
msg.erase(open + 2, msg.find(") ", open) + 2 - (open + 2));
}
}
return msg;
}
#endif
// the bits of a float, to compare values bit for bit
std::uint64_t bits(double x)
{
@@ -1304,18 +1336,18 @@ void check_values(const ordered_json_view& v, const ordered_json& j, const std::
// conversions to the wrong type throw what basic_json throws
if (!j.is_number())
{
CHECK(exception_of([&] { static_cast<void>(v.get<int>()); }) == exception_of([&] { static_cast<void>(j.get<int>()); }));
CHECK(exception_of([&] { static_cast<void>(v.get<int>()); }) == without_path(exception_of([&] { static_cast<void>(j.get<int>()); })));
}
CHECK(exception_of([&] { static_cast<void>(v.get<bool>()); }) == exception_of([&] { static_cast<void>(j.get<bool>()); }));
CHECK(exception_of([&] { static_cast<void>(v.get<std::string>()); }) == exception_of([&] { static_cast<void>(j.get<std::string>()); }));
CHECK(exception_of([&] { static_cast<void>(v.get<std::nullptr_t>()); }) == exception_of([&] { static_cast<void>(j.get<std::nullptr_t>()); }));
CHECK(exception_of([&] { static_cast<void>(v.get<bool>()); }) == without_path(exception_of([&] { static_cast<void>(j.get<bool>()); })));
CHECK(exception_of([&] { static_cast<void>(v.get<std::string>()); }) == without_path(exception_of([&] { static_cast<void>(j.get<std::string>()); })));
CHECK(exception_of([&] { static_cast<void>(v.get<std::nullptr_t>()); }) == without_path(exception_of([&] { static_cast<void>(j.get<std::nullptr_t>()); })));
if (!j.is_array())
{
CHECK(exception_of([&] { static_cast<void>(v.get<std::vector<int>>()); }) == exception_of([&] { static_cast<void>(j.get<std::vector<int>>()); }));
CHECK(exception_of([&] { static_cast<void>(v.get<std::vector<int>>()); }) == without_path(exception_of([&] { static_cast<void>(j.get<std::vector<int>>()); })));
}
if (!j.is_object())
{
CHECK(exception_of([&] { static_cast<void>(v.get<std::map<std::string, int>>()); }) == exception_of([&] { static_cast<void>(j.get<std::map<std::string, int>>()); }));
CHECK(exception_of([&] { static_cast<void>(v.get<std::map<std::string, int>>()); }) == without_path(exception_of([&] { static_cast<void>(j.get<std::map<std::string, int>>()); })));
}
#endif
@@ -1458,8 +1490,8 @@ TEST_CASE("json_view values")
CHECK(v[3].number_token() == "123456789012345678901234567890");
CHECK(v[4].number_token() == "-12");
CHECK(v[5].number_token() == "7");
CHECK_THROWS_WITH_AS(v[6].number_token(), json_view_test::message("type_error.302", "/6", 56, 59, "type must be number, but is string").c_str(), json::type_error&);
CHECK_THROWS_WITH_AS(v.get_string(), json_view_test::message("type_error.302", "", 0, 60, "type must be string, but is array").c_str(), json::type_error&);
CHECK_THROWS_WITH_AS(v[6].number_token(), "[json.exception.type_error.302] type must be number, but is string", json::type_error&);
CHECK_THROWS_WITH_AS(v.get_string(), "[json.exception.type_error.302] type must be string, but is array", json::type_error&);
}
SECTION("conversions")
@@ -1533,15 +1565,15 @@ TEST_CASE("json_view values")
CHECK(v["o"]["x"].value(json::json_pointer("/1"), 0) == j["o"]["x"].value(json::json_pointer("/1"), 0));
CHECK(v["o"]["x"].value(json::json_pointer("/7"), 3) == j["o"]["x"].value(json::json_pointer("/7"), 3));
#if !defined(JSON_NOEXCEPTION)
CHECK(exception_of([&] { static_cast<void>(v["o"]["x"].value("k", 0)); }) == exception_of([&] { static_cast<void>(j["o"]["x"].value("k", 0)); }));
CHECK(exception_of([&] { static_cast<void>(v.value("s", 0)); }) == exception_of([&] { static_cast<void>(j.value("s", 0)); }));
CHECK(exception_of([&] { static_cast<void>(v["n"].value("x", 0)); }) == exception_of([&] { static_cast<void>(j["n"].value("x", 0)); }));
CHECK(exception_of([&] { static_cast<void>(v["n"].value(json::json_pointer("/x"), 0)); }) == exception_of([&] { static_cast<void>(j["n"].value(json::json_pointer("/x"), 0)); }));
CHECK(exception_of([&] { static_cast<void>(v["o"]["x"].value("k", 0)); }) == without_path(exception_of([&] { static_cast<void>(j["o"]["x"].value("k", 0)); })));
CHECK(exception_of([&] { static_cast<void>(v.value("s", 0)); }) == without_path(exception_of([&] { static_cast<void>(j.value("s", 0)); })));
CHECK(exception_of([&] { static_cast<void>(v["n"].value("x", 0)); }) == without_path(exception_of([&] { static_cast<void>(j["n"].value("x", 0)); })));
CHECK(exception_of([&] { static_cast<void>(v["n"].value(json::json_pointer("/x"), 0)); }) == without_path(exception_of([&] { static_cast<void>(j["n"].value(json::json_pointer("/x"), 0)); })));
// string defaults: the member is not a string, or the value is neither an object nor an array
CHECK(exception_of([&] { static_cast<void>(v.value("n", "default")); }) == exception_of([&] { static_cast<void>(j.value("n", "default")); }));
CHECK(exception_of([&] { static_cast<void>(v["n"].value("x", "default")); }) == exception_of([&] { static_cast<void>(j["n"].value("x", "default")); }));
CHECK(exception_of([&] { static_cast<void>(v.value(json::json_pointer("/n"), "default")); }) == exception_of([&] { static_cast<void>(j.value(json::json_pointer("/n"), "default")); }));
CHECK(exception_of([&] { static_cast<void>(v["s"].value(json::json_pointer("/x"), "default")); }) == exception_of([&] { static_cast<void>(j["s"].value(json::json_pointer("/x"), "default")); }));
CHECK(exception_of([&] { static_cast<void>(v.value("n", "default")); }) == without_path(exception_of([&] { static_cast<void>(j.value("n", "default")); })));
CHECK(exception_of([&] { static_cast<void>(v["n"].value("x", "default")); }) == without_path(exception_of([&] { static_cast<void>(j["n"].value("x", "default")); })));
CHECK(exception_of([&] { static_cast<void>(v.value(json::json_pointer("/n"), "default")); }) == without_path(exception_of([&] { static_cast<void>(j.value(json::json_pointer("/n"), "default")); })));
CHECK(exception_of([&] { static_cast<void>(v["s"].value(json::json_pointer("/x"), "default")); }) == without_path(exception_of([&] { static_cast<void>(j["s"].value(json::json_pointer("/x"), "default")); })));
#endif
// (defaults and members longer than the short string optimization)
CHECK(v.value("s", "a default longer than fifteen characters") == "text");
@@ -1600,7 +1632,7 @@ TEST_CASE("json_view JSON pointers")
{
CAPTURE(pointer)
const json::json_pointer p(pointer);
const std::string at_error = exception_of([&] { static_cast<void>(j.at(p)); });
const std::string at_error = without_path(exception_of([&] { static_cast<void>(j.at(p)); }));
CHECK(exception_of([&] { static_cast<void>(v.at(p)); }) == at_error);
if (at_error.empty())
{
@@ -1614,7 +1646,7 @@ TEST_CASE("json_view JSON pointers")
}
else
{
CHECK(exception_of([&] { static_cast<void>(v[p]); }) == exception_of([&] { static_cast<void>(j[p]); }));
CHECK(exception_of([&] { static_cast<void>(v[p]); }) == without_path(exception_of([&] { static_cast<void>(j[p]); })));
}
// (basic_json::contains() throws out_of_range.404 for an empty
// array index token, although it is not meant to throw; the view
@@ -1625,7 +1657,7 @@ TEST_CASE("json_view JSON pointers")
static_cast<void>(found);
});
CHECK(v.contains(p) == (contains_error.empty() && j.contains(p)));
CHECK(exception_of([&] { static_cast<void>(v.value(p, 5)); }) == exception_of([&] { static_cast<void>(j.value(p, 5)); }));
CHECK(exception_of([&] { static_cast<void>(v.value(p, 5)); }) == without_path(exception_of([&] { static_cast<void>(j.value(p, 5)); })));
}
}
#endif
+1 -247
View File
@@ -13,7 +13,6 @@
#include <nlohmann/detail/view/builder.hpp>
#include <nlohmann/detail/view/materialize.hpp>
#include <nlohmann/detail/view/string_ref.hpp>
#include <nlohmann/json_view.hpp> // json_document::accept
#else
#include <nlohmann/json_view.hpp> // the single header contains the internal headers
#endif
@@ -60,8 +59,7 @@ built build(const std::string& text, bool comments, bool trailing_commas, bool s
r.copy.assign(text.begin(), text.end());
src = r.copy.data();
}
constexpr bool nul_is_end = !nlohmann::detail::abi_config::strict_nul_handling;
r.ok = nlohmann::detail::view::build<FloatType, nul_is_end>(*r.data, src, text.size(), comments, trailing_commas, sentinel, false, r.failure);
r.ok = nlohmann::detail::view::build < FloatType, !nlohmann::detail::abi_config::strict_nul_handling > (*r.data, src, text.size(), comments, trailing_commas, sentinel, r.failure);
r.data->src = src;
r.data->base[0] = src;
r.data->base[1] = r.data->arena.data();
@@ -174,22 +172,6 @@ void check_string(const std::string& content)
}
}
// accept() stores no nodes: it must accept and reject what json::accept does, for all options and for inputs with
// and without a terminating NUL
void check_accept(const std::string& text)
{
CAPTURE(text)
const std::vector<char> bytes(text.begin(), text.end());
for (int options = 0; options < 4; ++options)
{
const bool comments = (options & 1) != 0;
const bool trailing_commas = (options & 2) != 0;
const bool accepted = json::accept(text, comments, trailing_commas);
CHECK(nlohmann::json_document::accept(text, comments, trailing_commas) == accepted);
CHECK(nlohmann::json_document::accept(bytes, comments, trailing_commas) == accepted);
}
}
// a small deterministic generator of documents
struct generator
{
@@ -441,133 +423,6 @@ TEST_CASE("json_view builder")
}
}
SECTION("accept() without an index")
{
for (const char* text :
{
"", " ", "\n", "null", "nul", "true", "[", "]", "[]", "{}", "[1,2,3]", "[1,2,3", "[1,2,3,]", "{\"a\":1,}", "{\"a\"}", "{\"a\":}",
"[1] x", "[1][2]", "\xEF\xBB\xBF[1]", "[\"a\\", R"(["\u12"])", R"(["\ud800"])", "[\"\x01\"]", "[\"\xff\"]", "[01]", "[1.]",
"[.5]", "[-]", "[1e]", "// c\n[1]", "/* c */ {\"a\" /* x */ : 1}", "/* c", "[1] // c", "1e400", "-1e400", "123456789012345678901234567890",
R"({"a":1,"a":2})", "[[[]]]", R"("\u00e4\n\ud83d\ude00")"
})
{
check_accept(text);
}
// a null byte inside the text, and after it
check_accept(std::string("[1,\0]", 5));
check_accept(std::string("[1]\0", 4));
check_accept(std::string("\0", 1));
// generated documents and damaged copies, as for the builder above
generator g;
for (int i = 0; i < 2000; ++i)
{
std::string text;
g.value(text, 0);
check_accept(text);
std::string damaged = text;
const auto at = static_cast<std::size_t>(g.r(static_cast<int>(damaged.size())));
static const char replacements[] = {'x', '"', '\\', ',', ':', ']', '}', '[', '{', '1', '-', '.', 'e', '\0', '\n', '/'}; // NOLINT(cppcoreguidelines-avoid-c-arrays,hicpp-avoid-c-arrays,modernize-avoid-c-arrays)
damaged[at] = replacements[g.r(16)];
check_accept(damaged);
check_accept(text.substr(0, at));
}
// nesting beyond the 64 levels that are kept on the builder's own stack, closed and not closed
for (const std::size_t depth :
{
63u, 64u, 65u, 66u, 200u, 1000u
})
{
check_accept(std::string(depth, '[') + std::string(depth, ']'));
check_accept(std::string(depth, '[') + std::string(depth - 1, ']'));
std::string objects;
for (std::size_t i = 0; i < depth; ++i)
{
objects += "{\"k\":";
}
objects += "1" + std::string(depth, '}');
check_accept(objects);
check_accept(objects.substr(0, objects.size() - 1));
}
// objects with enough members to get a hash index when they are indexed (and duplicate keys)
std::string large = "{";
for (int i = 0; i < 300; ++i)
{
large += (i == 0 ? "\"k" : ",\"k") + std::to_string(i % 200) + "\":" + std::to_string(i);
}
check_accept(large + "}");
check_accept(large);
check_accept("[" + large + "},{" + large.substr(1) + "}]");
// the overflow check uses the floating-point type of the document
using float_json = nlohmann::basic_json<std::map, std::vector, std::string, bool, std::int64_t, std::uint64_t, float>;
for (const char* text :
{
"1e39", "-1e39", "3.4028235e38", "3.4028236e38", "[1.5e38, 3.5e38]", R"({"a": 1e-50, "b": 1e39})", "340282356779733661637539395458142568448"
})
{
CAPTURE(text)
CHECK(nlohmann::basic_json_document<float_json>::accept(text) == float_json::accept(text));
}
// strings are checked and not decoded: runs of text of every length from 0 to 40 between valid and invalid
// escapes, UTF-8 sequences and control characters, in values and in keys, cut near the end of the text
const std::array<const char*, 9> valid = {{"\\n", "\\\"", "\\\\", "\\/", "\\t", "\\u00e4", "\\u20ac", "\\ud83d\\ude00", "\\u0000"}};
const std::array<const char*, 12> invalid = {{"\\x", "\\ud800", "\\udc00", "\\u12G4", "\\u12", "\\ud83dx", "\\ud83d\\u0041", "\\ud83d\\ud83d", "\\", "\x01", "\x1f", "\x7f"}};
const std::array<const char*, 6> utf8 = {{"\xc3\xa4", "\xe2\x82\xac", "\xf0\x9f\x98\x80", "\xc3", "\xff", "\x80"}};
std::mt19937 rng(2029); // NOLINT(cert-msc32-c,cert-msc51-cpp,bugprone-random-generator-seed): reproducible
for (std::size_t run = 0; run <= 40; ++run)
{
for (int i = 0; i < 30; ++i)
{
std::string content = std::string(rng() % 20, 'a');
const auto pieces = 1 + (rng() % 6);
for (unsigned k = 0; k < pieces; ++k)
{
content += valid[rng() % valid.size()];
content += std::string(k == 0 ? run : rng() % 41, static_cast<char>('b' + (rng() % 20)));
switch (rng() % 8)
{
case 0:
content += utf8[rng() % 3];
break;
case 1:
content += utf8[rng() % utf8.size()];
break;
case 2:
content += invalid[rng() % invalid.size()];
break;
default:
break;
}
}
for (const std::string& text :
{
"[\"" + content + "\"]", "{\"" + content + "\":1}", "[\"" + content, "[\"" + content.substr(0, content.size() - (rng() % 4)) + "\"]"
})
{
check_accept(text);
}
}
}
}
SECTION("escapes in a row")
{
for (const char* content :
{
"\\n", R"(\n\t\\)", R"(\u00e9\u00e9\u4e2d)", "\\u00e9\\u00e", "\\u00e9\\", "\\n\\q", R"(\n\ud83d\ude00\n)",
R"(\ud83d\ude00\ud83d)", R"(\n\n\n abc \n\n)", "\\n\x01", "\\n\xC3\x28", "\\n\\"
})
{
check_accept(std::string("[\"") + content + "\"]");
check_accept(std::string("[\"") + content);
check_accept(std::string("{\"") + content + "\":1}");
}
}
SECTION("test files")
{
for (const char* name :
@@ -668,54 +523,6 @@ TEST_CASE("json_view builder: strings across vector blocks")
}
}
TEST_CASE("json_view builder: fraction digits")
{
// The digits of a fraction are skipped 16 bytes at a time where there are
// that many left (SSE2), and one at a time after that. Fractions of every
// length from 0 to 40 digits, followed by every kind of byte that ends a
// number or lies next to the digits in ASCII ('/' and ':'), as values, as
// keys' neighbours, and at the very end of the text (a vector without a
// terminating NUL). The length of the number is part of the value that
// the test compares with json::parse().
const std::string followers = ",]}/:.eE+-x \n\t\"09";
std::mt19937 rng(2028); // NOLINT(cert-msc32-c,cert-msc51-cpp,bugprone-random-generator-seed): reproducible
for (std::size_t digits = 0; digits <= 40; ++digits)
{
for (int i = 0; i < 3; ++i)
{
std::string fraction;
for (std::size_t k = 0; k < digits; ++k)
{
fraction += static_cast<char>('0' + (rng() % 10));
}
const std::string number = std::string(rng() % 2 == 0 ? "-" : "") + std::to_string(rng() % 100000) + "." + fraction;
check_same(number);
check_same("[" + number + "]");
std::string pair;
pair.reserve((number.size() * 2) + 3);
pair += '[';
pair += number;
pair += ',';
pair += number;
pair += ']';
check_same(pair);
check_same("{\"a\":" + number + "}");
for (const char follower : followers)
{
check_same(number + follower);
check_same("[" + number + follower + "]");
check_same("[" + number + follower + std::string(rng() % 20, '1'));
}
// everything near the end of the text: the number cut in every place
for (std::size_t cut = 1; cut <= 18 && cut <= number.size(); ++cut)
{
check_same(number.substr(0, number.size() - cut));
check_same("[" + number.substr(0, number.size() - cut));
}
}
}
}
TEST_CASE("json_view builder: escapes between runs of text")
{
// The text between escapes is copied 16 bytes at a time where there are
@@ -768,59 +575,6 @@ TEST_CASE("json_view builder: escapes between runs of text")
}
}
TEST_CASE("json_view builder: chains of unicode escapes")
{
// Consecutive \uXXXX escapes are decoded in a loop of their own. Chains of
// every length from 1 to 40 of escapes that give one, two, and three
// bytes of UTF-8, in upper and lower case, followed by what ends a chain:
// the end of the string, text, another escape, UTF-8, and everything that
// is not a valid escape (a bad digit at every position of the last
// escape, a lone surrogate, a surrogate pair, a cut in every place).
const std::array<const char*, 8> chain = {{"\\u0041", "\\u00e4", "\\u00E4", "\\u20ac", "\\u3042", "\\uFFFF", "\\u0000", "\\u07ff"}};
const std::array<const char*, 12> ends = {{"", "abc", "\\n", "\\u00e9", "\\ud83d\\ude00", "\\ud83d", "\\ude00", "\\ud83d\\u0041", "\\u12G4", "\\u12", "\xc3\xa4", "\\"}};
std::mt19937 rng(2027); // NOLINT(cert-msc32-c,cert-msc51-cpp,bugprone-random-generator-seed): reproducible
for (std::size_t length = 1; length <= 40; ++length)
{
for (int i = 0; i < 20; ++i)
{
std::string content = std::string(rng() % 20, 'a');
for (std::size_t k = 0; k < length; ++k)
{
content += chain[rng() % chain.size()];
}
const std::string whole = content;
for (const char* end : ends)
{
check_string(whole + end);
check_string(whole + end + std::string(rng() % 19, 'z'));
}
// a bad digit at every position of every escape near the end of the chain
for (std::size_t back = 1; back <= 6 && back <= length; ++back)
{
for (std::size_t digit = 2; digit < 6; ++digit)
{
std::string broken = whole;
broken[broken.size() - (6 * back) + digit] = "gG\"\x01 z"[rng() % 6];
check_string(broken);
}
}
// the text cut in every place of the last three escapes
for (std::size_t cut = 0; cut <= 18 && cut <= whole.size(); ++cut)
{
check_string(whole.substr(0, whole.size() - cut));
}
}
}
// a chain longer than the first growth of the arena, in a key and in a value
std::string long_chain;
for (int i = 0; i < 3000; ++i)
{
long_chain += chain[static_cast<std::size_t>(i) % chain.size()];
}
check_string(long_chain);
check_string(long_chain + "\\ud83d\\ude00" + long_chain);
}
TEST_CASE("json_view node integer bits")
{
using nlohmann::detail::view::integer_bits;
+672 -85
View File
@@ -35,9 +35,9 @@ using json_view_test::has_duplicate_keys;
#include <string>
#include <vector>
// These tests were split off unit-json_view.cpp, whose object file got too large for the MinGW
// linker (see json_view_test_helpers.hpp). Like that file, this one is also built
// with C++17, as it mentions JSON_HAS_CPP_17.
// These tests were split off unit-json_view.cpp, whose object file got too large
// for the MinGW linker (see json_view_test_helpers.hpp). Like that file, this one
// is also built with C++17, as it mentions JSON_HAS_CPP_17.
TEST_CASE("json_view dump")
{
@@ -150,88 +150,6 @@ TEST_CASE("json_view dump")
CHECK(many_doc.root().dump() == json::parse(many_tokens).dump());
}
// tokens of 16 and 17 digits without an exponent (like GeoJSON
// coordinates), which dump() converts in batches: doubles exactly
// halfway between two candidates for the shortest digits, random
// digits, and digits near a shorter decimal (...99999997,
// ...00000003), whose rounding carries; in pairs and among other
// values, so that the batches are interrupted; the last token near the
// end of the text
{
std::mt19937_64 tokens(5627); // NOLINT(cert-msc32-c,cert-msc51-cpp,bugprone-random-generator-seed)
const auto draw = [&tokens](std::size_t n)
{
const std::uint64_t r = tokens() % n;
return static_cast<std::size_t>(r);
};
std::string coordinates = "[754570942.33984373,257769445.56835938,858106901.94921875,97944964.876953125,"
"[0.5,1e-7],[1.5,-0.0,2.5],[1E2,12.5,0.1,3.25,4.5,5.75,6.125,7.0,8.5],[],[1.5],[1,2.5],[2.5,\"x\"],[[1.5],2.5],";
for (int i = 0; i < 20000; ++i)
{
const std::size_t length = 16 + draw(2);
std::string digits(1, static_cast<char>('1' + draw(9)));
const std::size_t kept = draw(3) == 0 ? length : 2 + draw(length - 3);
for (std::size_t k = 1; k < kept; ++k)
{
digits += static_cast<char>('0' + draw(10));
}
digits += std::string(length - kept, draw(2) == 0 ? '9' : '0');
digits.back() = static_cast<char>('0' + draw(10));
const std::size_t point = 1 + draw(9);
const std::string token = (draw(2) == 0 ? "-" : "") + digits.substr(0, point) + "." + digits.substr(point);
if (i != 0)
{
coordinates += ',';
}
if (draw(8) == 0)
{
coordinates += R"({"x":)";
coordinates += token;
coordinates += R"(,"s":"a"})";
}
else
{
coordinates += '[';
coordinates += token;
coordinates += ',';
coordinates += token;
coordinates += ']';
}
}
coordinates += ",-65.613616999999977]";
const json_document parsed = json_document::parse(coordinates);
const ordered_json j = ordered_json::parse(coordinates);
CHECK(parsed.root().dump() == j.dump());
CHECK(parsed.root().dump(1) == j.dump(1));
CHECK(parsed.root().dump(-1, ' ', true) == j.dump(-1, ' ', true));
CHECK(parsed.root().back().get<double>() == j.back().get<double>());
CHECK(parsed.root().back().dump() == "-65.61361699999998");
std::size_t i = 0;
for (const json_view v : parsed.root())
{
const ordered_json& e = j[i++];
CHECK(v.dump() == e.dump());
if (v.is_array() && !v.empty() && v[0].is_number())
{
CHECK(v[0].get<double>() == e[0].get<double>());
}
}
// an editable document: edits move some arrays, so that the walk
// leaves the order of the parsed nodes
ordered_json_editable_document editable = ordered_json_editable_document::parse(coordinates);
ordered_json expected = j;
for (std::size_t k = 0; k < 20000; k += 97)
{
if (expected[k].is_array())
{
editable.push_back(editable.root()[k], ordered_json(1.5));
expected[k].push_back(1.5);
}
}
CHECK(editable.root().dump() == expected.dump());
}
// random doubles, written as parse() and dump() would
std::mt19937_64 rng(1170); // NOLINT(cert-msc32-c,cert-msc51-cpp,bugprone-random-generator-seed)
std::string many = "[";
@@ -253,4 +171,673 @@ TEST_CASE("json_view dump")
const nlohmann::basic_json_document<json_float> float_document = nlohmann::basic_json_document<json_float>::parse("[0.1, 1.5e10, 3.4028235e38]");
CHECK(float_document.root().dump() == json_float::parse("[0.1, 1.5e10, 3.4028235e38]").dump());
}
SECTION("members in document order, all of them")
{
const json_document d = json_document::parse(R"({"b": 1, "a": 2, "b": 3})");
CHECK(d.root().dump() == R"({"b":1,"a":2,"b":3})");
CHECK(d.root().dump(1) == "{\n \"b\": 1,\n \"a\": 2,\n \"b\": 3\n}");
// dump() writes all members, though the lookup finds the first
CHECK(d.root()["b"].dump() == "1");
}
SECTION("deep nesting")
{
const std::string deep = std::string(100000, '[') + std::string(100000, ']');
const json_document deep_document = json_document::parse(deep);
CHECK(deep_document.root().dump() == deep);
}
SECTION("ensure_ascii: a character that needs an escape at each position of a string")
{
// string lengths around the width of the vector steps, in a string value, a key, and a value near the end of the text
for (const char* const special :
{
"\xC3\xA9", "\xE4\xB8\xAD", "\xF0\x9F\x98\x80", "\x7F", "\\n", "\\u0001", "\\\"", "\\\\"
})
{
for (std::size_t length = 0; length <= 40; ++length)
{
for (std::size_t position = 0; position <= length; ++position)
{
std::string content(length, 'a');
content.insert(position, special);
const std::array<std::string, 3> texts =
{
{
"\"" + content + "\"",
"[\"" + content + "\"]",
"{\"" + content + "\":1}",
}
};
for (const std::string& text : texts)
{
CAPTURE(text)
const ordered_json_document d = ordered_json_document::parse(text);
const ordered_json j = ordered_json::parse(text);
CHECK(d.root().dump(-1, ' ', true) == j.dump(-1, ' ', true));
CHECK(d.root().dump(2, ' ', true) == j.dump(2, ' ', true));
CHECK(d.root().dump() == j.dump());
}
}
}
}
}
SECTION("ensure_ascii: every character of the Basic Multilingual Plane and some beyond")
{
// (the two- and three-byte sequences, including the ranges that need a check: E0, ED, EF BF BF)
std::string text = "[";
std::size_t in_chunk = 0;
const auto append = [&](std::uint32_t cp)
{
if (cp < 0x80)
{
if (cp < 0x20 || cp == '"' || cp == '\\')
{
return;
}
text += static_cast<char>(cp);
}
else if (cp < 0x800)
{
text += static_cast<char>(0xC0 | (cp >> 6));
text += static_cast<char>(0x80 | (cp & 0x3F));
}
else if (cp < 0x10000)
{
text += static_cast<char>(0xE0 | (cp >> 12));
text += static_cast<char>(0x80 | ((cp >> 6) & 0x3F));
text += static_cast<char>(0x80 | (cp & 0x3F));
}
else
{
text += static_cast<char>(0xF0 | (cp >> 18));
text += static_cast<char>(0x80 | ((cp >> 12) & 0x3F));
text += static_cast<char>(0x80 | ((cp >> 6) & 0x3F));
text += static_cast<char>(0x80 | (cp & 0x3F));
}
};
for (std::uint32_t cp = 0x20; cp < 0x10400; ++cp)
{
if (cp >= 0xD800 && cp < 0xE000)
{
continue;
}
if (in_chunk == 0)
{
text += text.size() > 1 ? ",\"" : "\"";
}
append(cp);
if (++in_chunk == 37)
{
text += "\"";
in_chunk = 0;
}
}
text += in_chunk != 0 ? "\"]" : "]";
const ordered_json_document d = ordered_json_document::parse(text);
const ordered_json j = ordered_json::parse(text);
CHECK(d.root().dump(-1, ' ', true) == j.dump(-1, ' ', true));
CHECK(d.root().dump(2, ' ', true) == j.dump(2, ' ', true));
CHECK(d.root().dump() == j.dump());
CHECK(d.root().dump(2) == j.dump(2));
}
SECTION("deep nesting, indented")
{
// indentations of up to and beyond the width of the writer's fixed-size copy
for (const int depth :
{
1, 5, 20, 70, 300
})
{
std::string text;
for (int i = 0; i < depth; ++i)
{
text += i % 2 == 0 ? "[1," : R"({"k":2,"\u00e9":)";
}
text += "[]";
for (int i = depth - 1; i >= 0; --i)
{
text += i % 2 == 0 ? "]" : "}";
}
const ordered_json_document d = ordered_json_document::parse(text);
const ordered_json j = ordered_json::parse(text);
for (const int indent :
{
0, 1, 2, 3, 8
})
{
CHECK(d.root().dump(indent) == j.dump(indent));
CHECK(d.root().dump(indent, '\t') == j.dump(indent, '\t'));
}
}
}
SECTION("the output buffer of a small value is small")
{
// an escaped key after the value: its node lies in the arena, so the
// source extent of the value cannot be read from the next node
const std::string big(100000, 'a');
const std::string text = R"({"small":1,"list":[1,2,3],"k\n":")" + big + R"("})";
const json_document d = json_document::parse(text);
const auto small = d.root()["small"].dump();
CHECK(small == "1");
CHECK(small.capacity() < 4096);
const auto list = d.root()["list"].dump();
CHECK(list == "[1,2,3]");
CHECK(list.capacity() < 4096);
CHECK(d.root()["list"].dump(2).capacity() < 4096);
// the whole document and the large value are unaffected
CHECK(d.root().dump() == ordered_json::parse(text).dump());
CHECK(d.root()["k\n"].dump() == "\"" + big + "\"");
}
SECTION("output that outgrows the estimate")
{
// ensure_ascii writes six bytes for each two-byte character
std::string chars;
for (int i = 0; i < 5000; ++i)
{
chars += "\xC3\xA9";
}
const json_document d = json_document::parse(R"({"a":")" + chars + R"(","k\n":1})");
const json expected = json::parse("\"" + chars + "\"");
CHECK(d.root()["a"].dump(-1, ' ', true) == expected.dump(-1, ' ', true));
CHECK(d.root()["a"].dump(-1, ' ', true).size() == 2 + (5000 * 6));
}
SECTION("ensure_ascii with strings of the source, decoded strings, and edited strings")
{
// plain ASCII, non-ASCII, escaped (decoded), and mixed strings, as values and as keys;
// the second text has a DEL, which ensure_ascii writes as an escape
for (const char* const del :
{"", "\x7F"
})
{
const std::string text = std::string("{\"plain\":\"abc\",\"caf\xC3\xA9\":\"\xC3\xA9\xE2\x82\xAC\xF0\x9F\x98\x80\",")
+ "\"esc\\n\\\"\":\"a\\tb\\u00e4\\ud83d\\ude00\",\"del\":\"a" + del + "b\",\"\":\"\"," // NOLINT(modernize-raw-string-literal)
+ "\"list\":[\"x\",\"\xC3\xBC\",\"\\\\\",\"long ASCII string with more than thirty-two bytes in it\","
+ "\"long string with more than thirty-two bytes and one non-ASCII byte \xC3\xB6\"],"
+ "\"n\":[1,2.5,-0,true,null]}";
const ordered_json j = ordered_json::parse(text);
const ordered_json_document d = ordered_json_document::parse(text);
CHECK(d.root().dump(-1, ' ', true) == j.dump(-1, ' ', true));
CHECK(d.root().dump(4, ' ', true) == j.dump(4, ' ', true));
CHECK(d.root().dump() == j.dump());
CHECK(d.root()["list"].dump(-1, ' ', true) == j["list"].dump(-1, ' ', true));
CHECK(d.root()["del"].dump(-1, ' ', true) == j["del"].dump(-1, ' ', true));
// strings that an edit wrote (the source strings that they replace were plain)
ordered_json_editable_document e = ordered_json_editable_document::parse(text);
ordered_json k = j;
e.set(e.root()["plain"], std::string("\xC3\xA4gypten"));
k["plain"] = "\xC3\xA4gypten";
e.set(e.root()["list"][0], std::string("ascii after edit"));
k["list"][0] = "ascii after edit";
e.set(e.root()["list"][1], std::string("\xC3\xA4"));
k["list"][1] = "\xC3\xA4";
e.set(e.root()["del"], std::string("tab\there"));
k["del"] = "tab\there";
CHECK(e.root().dump(-1, ' ', true) == k.dump(-1, ' ', true));
CHECK(e.root().dump(2, ' ', true) == k.dump(2, ' ', true));
CHECK(e.root().dump() == k.dump());
// strings that were edited from or to ASCII text
e.set(e.root()["list"][2], std::string("\xC3\xA4"));
k["list"][2] = "\xC3\xA4";
e.set(e.root()["list"][4], std::string("plain again"));
k["list"][4] = "plain again";
CHECK(e.root().dump(-1, ' ', true) == k.dump(-1, ' ', true));
}
}
SECTION("ensure_ascii with DEL or a non-ASCII character at every position")
{
// the check whether a string of the source has no byte >= 0x7F reads
// eight bytes at a time and then the rest one by one
for (std::size_t length = 1; length <= 20; ++length)
{
for (std::size_t pos = 0; pos < length; ++pos)
{
for (const char* const c :
{
"\x7F", "\x7E", "\xC3\xA4", "\xC3\xBF" // NOLINT(modernize-raw-string-literal)
})
{
const std::string s = std::string(pos, 'a') + c + std::string(length - pos - 1, 'b');
std::string text = "[\"";
text.append(s).append("\",{\"").append(s).append("\":1}]");
const json_document d = json_document::parse(text);
const json j = json::parse(text);
CHECK(d.root().dump(-1, ' ', true) == j.dump(-1, ' ', true));
CHECK(d.root().dump(1, ' ', true) == j.dump(1, ' ', true));
}
}
}
}
SECTION("streams and discarded views")
{
const json_document d = json_document::parse(R"({"a": [1, 2]})");
std::ostringstream compact;
compact << d.root();
CHECK(compact.str() == R"({"a":[1,2]})");
std::ostringstream pretty;
pretty << std::setw(2) << std::setfill('.') << d.root() << d.root()["a"];
CHECK(pretty.str() == "{\n..\"a\": [\n....1,\n....2\n..]\n}[1,2]");
CHECK(json_view().dump() == json(json::value_t::discarded).dump());
}
}
TEST_CASE("json_view large objects")
{
// objects with 128 members or more are looked up with a hash index
for (const std::size_t members :
{
127u, 128u, 129u, 10000u
})
{
CAPTURE(members)
std::string text = "{";
for (std::size_t i = 0; i < members; ++i)
{
text += (i != 0 ? ",\"" : "\"") + std::string(i % 23, 'k') + std::to_string(i) + (i % 7 == 0 ? "\\n" : "") + "\":" + std::to_string(i);
}
text += R"(,"":"empty key","k1":"a duplicate of an earlier key"})";
const json_document d = json_document::parse(text);
const json_view v = d.root();
const json j = json::parse(text);
for (std::size_t i = 0; i < members; ++i)
{
const std::string key = std::string(i % 23, 'k') + std::to_string(i) + (i % 7 == 0 ? "\n" : "");
CHECK(v[key].get<std::size_t>() == i);
CHECK(v.contains(key));
CHECK(v.find(key).key() == key);
CHECK(v.at(key).get<std::size_t>() == i);
CHECK(!v.contains(key + "x"));
}
CHECK(v[""].get_string() == "empty key");
CHECK(v["k1"].get<int>() == 1); // the first of duplicate keys, as for small objects
CHECK(!v.contains("missing"));
CHECK_THROWS_WITH_AS(v.at("missing"), "[json.exception.out_of_range.403] key 'missing' not found", json::out_of_range&);
CHECK(v == j);
CHECK(v.materialize() == j);
}
SECTION("duplicate keys: the first member wins, with and without a table")
{
// an object of `total` members: the keys "k0".."k<n-1>" in order, then
// three keys repeated twice more (one copy in the middle, one at the
// end), and two keys repeated once; the value of a member is its
// position, so that the first member of a key can be told apart
struct member
{
std::string key;
std::size_t position;
};
const auto make_members = [](std::size_t total)
{
std::vector<std::string> keys;
for (std::size_t i = 0; i + 8 < total; ++i)
{
keys.push_back("k" + std::to_string(i));
}
const std::size_t n = keys.size();
const std::array<std::size_t, 3> triple = {{3, 17, n - 1}};
const std::array<std::size_t, 2> twice = {{5, n / 2}};
std::vector<std::string> ordered = keys;
for (const std::size_t i : triple)
{
ordered.insert(ordered.begin() + static_cast<std::ptrdiff_t>(ordered.size() / 2), keys[i]);
}
for (const std::size_t i : twice)
{
ordered.insert(ordered.begin() + static_cast<std::ptrdiff_t>(ordered.size() / 3), keys[i]);
}
for (const std::size_t i : triple)
{
ordered.push_back(keys[i]);
}
std::vector<member> result;
result.reserve(ordered.size());
for (std::size_t i = 0; i < ordered.size(); ++i)
{
result.push_back({ordered[i], i});
}
return result;
};
// 100 and 127 members: no table; 128 members and more: a table
for (const std::size_t total :
{
100u, 127u, 128u, 200u, 5000u
})
{
CAPTURE(total)
const std::vector<member> members = make_members(total);
REQUIRE(members.size() >= total);
REQUIRE(members.size() >= 100);
std::string text = "{";
std::map<std::string, std::size_t> first;
std::map<std::string, std::size_t> last;
std::map<std::string, std::size_t> occurrences;
for (const member& m : members)
{
text += (text.size() > 1 ? ",\"" : "\"") + m.key + "\":" + std::to_string(m.position);
first.insert({m.key, m.position});
last[m.key] = m.position;
++occurrences[m.key];
}
text += '}';
REQUIRE(first.size() < members.size());
const json_document d = json_document::parse(text);
const json_view v = d.root();
const json j = json::parse(text);
CHECK(v.size() == members.size()); // every occurrence is visited
const json m = v.materialize();
for (const auto& entry : first)
{
CAPTURE(entry.first)
const std::size_t expected = entry.second;
CHECK(v[entry.first].get<std::size_t>() == expected);
CHECK(v.at(entry.first).get<std::size_t>() == expected);
CHECK(v.find(entry.first).value().get<std::size_t>() == expected);
CHECK(v.value(entry.first, std::size_t{0}) == expected);
CHECK(v.contains(entry.first));
CHECK(v.count(entry.first) == occurrences[entry.first]);
const std::string pointer = "/" + entry.first;
CHECK(v[json::json_pointer(pointer)].get<std::size_t>() == expected);
CHECK(v.at(json::json_pointer(pointer)).get<std::size_t>() == expected);
CHECK(v.value(json::json_pointer(pointer), std::size_t{0}) == expected);
CHECK(v.contains(json::json_pointer(pointer)));
// materialize() and parse() keep the last value instead
CHECK(j[entry.first].get<std::size_t>() == last.at(entry.first));
CHECK(m[entry.first].get<std::size_t>() == last.at(entry.first));
}
CHECK(!v.contains("k"));
CHECK(v["missing"].is_discarded());
CHECK(v.materialize() == j);
CHECK(v == j);
}
}
SECTION("colliding keys")
{
// the hash is not seeded, so keys that all land in one place must not
// make the table build quadratic: such an object gets no table and is
// searched linearly
constexpr std::size_t members = 300;
constexpr std::size_t table_size = 1024; // the table size for 300 members: the next power of two >= 600
std::vector<std::string> colliding;
std::vector<std::string> spread;
for (std::uint64_t counter = 0; colliding.size() < members || spread.size() < members; ++counter)
{
std::string key(8, 'a');
std::uint64_t x = counter;
for (std::size_t i = 0; i < 8; ++i, x /= 26)
{
key[i] = static_cast<char>('a' + (x % 26));
}
const bool lands_in_slot_zero = (nlohmann::detail::view::key_hash(key.data(), key.size()) & (table_size - 1)) == 0;
if (lands_in_slot_zero && colliding.size() < members)
{
colliding.push_back(key);
}
else if (!lands_in_slot_zero && spread.size() < members)
{
spread.push_back(key);
}
}
const auto make_text = [](const std::vector<std::string>& keys)
{
std::string text = "{";
for (std::size_t i = 0; i < keys.size(); ++i)
{
text += (i != 0 ? ",\"" : "\"") + keys[i] + "\":" + std::to_string(i % 10);
}
return text + "}";
};
const std::string colliding_text = make_text(colliding);
const std::string spread_text = make_text(spread);
json_document with_collisions = json_document::parse(colliding_text);
json_document without_collisions = json_document::parse(spread_text);
for (const auto* pair :
{
&colliding, &spread
})
{
const json_view v = (pair == &colliding ? with_collisions : without_collisions).root();
for (std::size_t i = 0; i < members; ++i)
{
CAPTURE(i)
CHECK(v[(*pair)[i]].get<std::size_t>() == i % 10);
CHECK(v.at((*pair)[i]).get<std::size_t>() == i % 10);
CHECK(v.find((*pair)[i]).key() == (*pair)[i]);
CHECK(!v.contains((*pair)[i] + "x"));
}
CHECK(!v.contains("missing"));
}
CHECK(with_collisions.root() == json::parse(colliding_text));
// only the object with the spread keys got a table (both texts have the
// same length, so the tables are the only difference)
with_collisions.shrink_to_fit();
without_collisions.shrink_to_fit();
CHECK(without_collisions.memory_usage() >= with_collisions.memory_usage() + (table_size * sizeof(std::uint32_t)));
}
SECTION("keys that differ in their last bytes are spread over the table")
{
// The hash must spread the usual families of keys (numbered names, hex
// strings, dates, short names): a key farther than index_max_displacement
// from its home slot would leave the object without a table. The keys
// are placed with linear probing, as the table does.
const auto max_displacement = [](const std::vector<std::string>& keys)
{
std::size_t slots = 16;
while (slots < 2 * keys.size())
{
slots *= 2;
}
std::vector<bool> used(slots, false);
std::size_t farthest = 0;
for (const std::string& key : keys)
{
std::size_t i = nlohmann::detail::conditional_static_cast<std::size_t>(nlohmann::detail::view::key_hash(key.data(), key.size())) & (slots - 1);
std::size_t distance = 0;
while (used[i])
{
i = (i + 1) & (slots - 1);
++distance;
}
used[i] = true;
farthest = (std::max)(farthest, distance);
}
return farthest;
};
// the keys of a printf-like pattern with one %d, %0Nd or %0Nx, formatted with a stream
// (a format string that is not a literal is a Flawfinder error)
const auto numbered = [](const char* format, int count, int step)
{
const std::string pattern(format);
const std::size_t percent = pattern.find('%');
std::size_t conversion = percent + 1;
std::size_t width = 0;
while (pattern[conversion] >= '0' && pattern[conversion] <= '9')
{
width = (width * 10) + static_cast<std::size_t>(pattern[conversion] - '0');
++conversion;
}
const bool hex = pattern[conversion] == 'x';
std::vector<std::string> keys;
for (int i = 0; i < count; ++i)
{
std::ostringstream key;
key << pattern.substr(0, percent) << std::setfill('0') << std::setw(static_cast<int>(width))
<< (hex ? std::hex : std::dec) << (i * step) << pattern.substr(conversion + 1);
keys.push_back(key.str());
}
return keys;
};
for (const char* format :
{"key%d", "%d", "item_%04d", "a%05d", "%03dxyz", "id%d", "user_%d", "%d_suffix", "0x%04x", "A%07d", "12345678%d", "1234567%d", "123456789%d",
"configuration_parameter_value_%04d_extra", "2026-01-%02dT00:00:00Z"
})
{
CAPTURE(format)
CHECK(max_displacement(numbered(format, std::string(format).find("%02d") != std::string::npos ? 60 : 1000, 1)) < nlohmann::detail::view::index_max_displacement); // NOLINT(abseil-string-find-str-contains)
CHECK(max_displacement(numbered(format, 600, 7)) < nlohmann::detail::view::index_max_displacement);
}
std::mt19937_64 rng(7); // NOLINT(cert-msc32-c,cert-msc51-cpp,bugprone-random-generator-seed)
for (const std::size_t length :
{
std::size_t{1}, std::size_t{2}, std::size_t{3}, std::size_t{5}, std::size_t{7}, std::size_t{8}, std::size_t{9}, std::size_t{12}, std::size_t{16}, std::size_t{17}, std::size_t{33}
})
{
CAPTURE(length)
std::set<std::string> distinct;
// (there are 26 and 676 keys of one and two letters)
std::size_t wanted = 700;
if (length == 1)
{
wanted = 26;
}
else if (length == 2)
{
wanted = 500;
}
while (distinct.size() < wanted)
{
std::string key(length, 'a');
for (char& c : key)
{
c = static_cast<char>('a' + (rng() % 26));
}
distinct.insert(key);
}
CHECK(max_displacement(std::vector<std::string>(distinct.begin(), distinct.end())) < nlohmann::detail::view::index_max_displacement);
}
}
SECTION("keys of every length that differ in one byte")
{
// In a large object, a key is found by comparing its bytes with those of the member the
// table points to: up to 16 bytes as two overlapping words, longer ones with memcmp.
// Keys of every length from 0 to 40, and keys that differ from them in one byte.
std::mt19937_64 rng(2026); // NOLINT(cert-msc32-c,cert-msc51-cpp,bugprone-random-generator-seed)
std::vector<std::string> keys;
std::set<std::string> distinct;
for (std::size_t length = 0; length <= 40; ++length)
{
for (int copy = 0; copy < 5; ++copy)
{
std::string key(length, 'a');
for (char& c : key)
{
c = static_cast<char>('a' + (rng() % 26));
}
if (distinct.insert(key).second)
{
keys.push_back(key);
}
}
}
std::string text = "{";
for (std::size_t i = 0; i < keys.size(); ++i)
{
text += (i != 0 ? ",\"" : "\"") + keys[i] + "\":" + std::to_string(i);
}
text += "}";
REQUIRE(keys.size() >= 128); // (a table is built for 128 members)
const json_document d = json_document::parse(text);
const json_view v = d.root();
for (std::size_t i = 0; i < keys.size(); ++i)
{
CAPTURE(keys[i])
CHECK(v[keys[i]].get<std::size_t>() == i);
CHECK(v.at(keys[i]).get<std::size_t>() == i);
CHECK(v.contains(keys[i]));
for (std::size_t at = 0; at < keys[i].size(); ++at)
{
std::string other = keys[i];
other[at] = other[at] == 'z' ? 'y' : 'z';
CAPTURE(other)
CHECK(v.contains(other) == (distinct.count(other) != 0));
}
CHECK(v.contains(keys[i] + "a") == (distinct.count(keys[i] + "a") != 0));
if (!keys[i].empty())
{
CHECK(v.contains(keys[i].substr(1)) == (distinct.count(keys[i].substr(1)) != 0));
}
}
}
SECTION("shrink_to_fit releases the tables' spare capacity")
{
const auto make_text = [](int objects, int members)
{
std::string text = "[";
for (int object = 0; object < objects; ++object)
{
text += object != 0 ? ",{" : "{";
for (int i = 0, n = members + object; i < n; ++i)
{
text += (i != 0 ? ",\"" : "\"") + std::to_string(i) + "\":" + std::to_string(i);
}
text += '}';
}
return text + "]";
};
const std::string small_text = make_text(5, 150);
const std::string big_text = make_text(40, 400);
// reading a big text, and then a small one, leaves the spare capacity
// of the big one: shrink_to_fit() brings the document to the size of
// one parsed from the small text alone
json_document d = json_document::parse(big_text);
const std::size_t big = d.memory_usage();
d.read(small_text);
CHECK(d.memory_usage() >= big);
d.shrink_to_fit();
json_document fresh = json_document::parse(small_text);
fresh.shrink_to_fit();
CHECK(d.memory_usage() == fresh.memory_usage());
CHECK(d.memory_usage() < big / 2);
CHECK(d.root() == json::parse(small_text));
for (int object = 0; object < 5; ++object)
{
const json_view v = d.root()[static_cast<std::size_t>(object)];
for (int i = 0, n = 150 + object; i < n; ++i)
{
CHECK(v[std::to_string(i)].get<int>() == i);
}
}
}
SECTION("nested, reused, and in arrays")
{
std::string inner = "{";
for (int i = 0; i < 300; ++i)
{
inner += (i != 0 ? ",\"m" : "\"m") + std::to_string(i) + "\":" + std::to_string(i);
}
inner += '}';
const std::string text = "[" + inner + ",{\"x\":" + inner + "}," + inner + "]";
json_document d = json_document::parse(text);
CHECK(d.root()[0]["m299"].get<int>() == 299);
CHECK(d.root()[1]["x"]["m150"].get<int>() == 150);
CHECK(d.root()[2]["m0"].get<int>() == 0);
const std::size_t with_index = d.memory_usage();
d.read(std::string("{\"small\": 1}"));
CHECK(d.root()["small"].get<int>() == 1);
d.read(text);
CHECK(d.root()[2]["m7"].get<int>() == 7);
CHECK(d.memory_usage() >= with_index / 2);
}
}
-298
View File
@@ -1,298 +0,0 @@
// __ _____ _____ _____
// __| | __| | | | JSON for Modern C++ (supporting code)
// | | |__ | | | | | | version 3.12.0
// |_____|_____|_____|_|___| https://github.com/nlohmann/json
//
// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
// SPDX-License-Identifier: MIT
#include "doctest_compatibility.h"
#include <nlohmann/json_view.hpp>
using nlohmann::json;
using nlohmann::ordered_json;
using nlohmann::json_document;
using nlohmann::ordered_json_editable_document;
using nlohmann::json_view;
using nlohmann::ordered_json_document;
using nlohmann::ordered_json_view;
#include <algorithm>
#include <array>
#include <cmath>
#include <cstddef>
#include <cstdint>
#include <cstring>
#include <iomanip>
#include <map>
#include <random>
#include <set>
#include <sstream>
#include <string>
#include <vector>
// These tests were split off unit-json_view_dump.cpp, whose object file got too large for the MinGW
// linker (see json_view_test_helpers.hpp). Like that file, this one is also built
// with C++17, as it mentions JSON_HAS_CPP_17.
TEST_CASE("json_view dump: escaping, nesting, and output buffers")
{
SECTION("members in document order, all of them")
{
const json_document d = json_document::parse(R"({"b": 1, "a": 2, "b": 3})");
CHECK(d.root().dump() == R"({"b":1,"a":2,"b":3})");
CHECK(d.root().dump(1) == "{\n \"b\": 1,\n \"a\": 2,\n \"b\": 3\n}");
// dump() writes all members, though the lookup finds the first
CHECK(d.root()["b"].dump() == "1");
}
SECTION("deep nesting")
{
const std::string deep = std::string(100000, '[') + std::string(100000, ']');
const json_document deep_document = json_document::parse(deep);
CHECK(deep_document.root().dump() == deep);
}
SECTION("ensure_ascii: a character that needs an escape at each position of a string")
{
// string lengths around the width of the vector steps, in a string value, a key, and a value near the end of the text
for (const char* const special :
{
"\xC3\xA9", "\xE4\xB8\xAD", "\xF0\x9F\x98\x80", "\x7F", "\\n", "\\u0001", "\\\"", "\\\\"
})
{
for (std::size_t length = 0; length <= 40; ++length)
{
for (std::size_t position = 0; position <= length; ++position)
{
std::string content(length, 'a');
content.insert(position, special);
const std::array<std::string, 3> texts =
{
{
"\"" + content + "\"",
"[\"" + content + "\"]",
"{\"" + content + "\":1}",
}
};
for (const std::string& text : texts)
{
CAPTURE(text)
const ordered_json_document d = ordered_json_document::parse(text);
const ordered_json j = ordered_json::parse(text);
CHECK(d.root().dump(-1, ' ', true) == j.dump(-1, ' ', true));
CHECK(d.root().dump(2, ' ', true) == j.dump(2, ' ', true));
CHECK(d.root().dump() == j.dump());
}
}
}
}
}
SECTION("ensure_ascii: every character of the Basic Multilingual Plane and some beyond")
{
// (the two- and three-byte sequences, including the ranges that need a check: E0, ED, EF BF BF)
std::string text = "[";
std::size_t in_chunk = 0;
const auto append = [&](std::uint32_t cp)
{
if (cp < 0x80)
{
if (cp < 0x20 || cp == '"' || cp == '\\')
{
return;
}
text += static_cast<char>(cp);
}
else if (cp < 0x800)
{
text += static_cast<char>(0xC0 | (cp >> 6));
text += static_cast<char>(0x80 | (cp & 0x3F));
}
else if (cp < 0x10000)
{
text += static_cast<char>(0xE0 | (cp >> 12));
text += static_cast<char>(0x80 | ((cp >> 6) & 0x3F));
text += static_cast<char>(0x80 | (cp & 0x3F));
}
else
{
text += static_cast<char>(0xF0 | (cp >> 18));
text += static_cast<char>(0x80 | ((cp >> 12) & 0x3F));
text += static_cast<char>(0x80 | ((cp >> 6) & 0x3F));
text += static_cast<char>(0x80 | (cp & 0x3F));
}
};
for (std::uint32_t cp = 0x20; cp < 0x10400; ++cp)
{
if (cp >= 0xD800 && cp < 0xE000)
{
continue;
}
if (in_chunk == 0)
{
text += text.size() > 1 ? ",\"" : "\"";
}
append(cp);
if (++in_chunk == 37)
{
text += "\"";
in_chunk = 0;
}
}
text += in_chunk != 0 ? "\"]" : "]";
const ordered_json_document d = ordered_json_document::parse(text);
const ordered_json j = ordered_json::parse(text);
CHECK(d.root().dump(-1, ' ', true) == j.dump(-1, ' ', true));
CHECK(d.root().dump(2, ' ', true) == j.dump(2, ' ', true));
CHECK(d.root().dump() == j.dump());
CHECK(d.root().dump(2) == j.dump(2));
}
SECTION("deep nesting, indented")
{
// indentations of up to and beyond the width of the writer's fixed-size copy
for (const int depth :
{
1, 5, 20, 70, 300
})
{
std::string text;
for (int i = 0; i < depth; ++i)
{
text += i % 2 == 0 ? "[1," : R"({"k":2,"\u00e9":)";
}
text += "[]";
for (int i = depth - 1; i >= 0; --i)
{
text += i % 2 == 0 ? "]" : "}";
}
const ordered_json_document d = ordered_json_document::parse(text);
const ordered_json j = ordered_json::parse(text);
for (const int indent :
{
0, 1, 2, 3, 8
})
{
CHECK(d.root().dump(indent) == j.dump(indent));
CHECK(d.root().dump(indent, '\t') == j.dump(indent, '\t'));
}
}
}
SECTION("the output buffer of a small value is small")
{
// an escaped key after the value: its node lies in the arena, so the
// source extent of the value cannot be read from the next node
const std::string big(100000, 'a');
const std::string text = R"({"small":1,"list":[1,2,3],"k\n":")" + big + R"("})";
const json_document d = json_document::parse(text);
const auto small = d.root()["small"].dump();
CHECK(small == "1");
CHECK(small.capacity() < 4096);
const auto list = d.root()["list"].dump();
CHECK(list == "[1,2,3]");
CHECK(list.capacity() < 4096);
CHECK(d.root()["list"].dump(2).capacity() < 4096);
// the whole document and the large value are unaffected
CHECK(d.root().dump() == ordered_json::parse(text).dump());
CHECK(d.root()["k\n"].dump() == "\"" + big + "\"");
}
SECTION("output that outgrows the estimate")
{
// ensure_ascii writes six bytes for each two-byte character
std::string chars;
for (int i = 0; i < 5000; ++i)
{
chars += "\xC3\xA9";
}
const json_document d = json_document::parse(R"({"a":")" + chars + R"(","k\n":1})");
const json expected = json::parse("\"" + chars + "\"");
CHECK(d.root()["a"].dump(-1, ' ', true) == expected.dump(-1, ' ', true));
CHECK(d.root()["a"].dump(-1, ' ', true).size() == 2 + (5000 * 6));
}
SECTION("ensure_ascii with strings of the source, decoded strings, and edited strings")
{
// plain ASCII, non-ASCII, escaped (decoded), and mixed strings, as values and as keys;
// the second text has a DEL, which ensure_ascii writes as an escape
for (const char* const del :
{"", "\x7F"
})
{
const std::string text = std::string("{\"plain\":\"abc\",\"caf\xC3\xA9\":\"\xC3\xA9\xE2\x82\xAC\xF0\x9F\x98\x80\",")
+ "\"esc\\n\\\"\":\"a\\tb\\u00e4\\ud83d\\ude00\",\"del\":\"a" + del + "b\",\"\":\"\"," // NOLINT(modernize-raw-string-literal)
+ "\"list\":[\"x\",\"\xC3\xBC\",\"\\\\\",\"long ASCII string with more than thirty-two bytes in it\","
+ "\"long string with more than thirty-two bytes and one non-ASCII byte \xC3\xB6\"],"
+ "\"n\":[1,2.5,-0,true,null]}";
const ordered_json j = ordered_json::parse(text);
const ordered_json_document d = ordered_json_document::parse(text);
CHECK(d.root().dump(-1, ' ', true) == j.dump(-1, ' ', true));
CHECK(d.root().dump(4, ' ', true) == j.dump(4, ' ', true));
CHECK(d.root().dump() == j.dump());
CHECK(d.root()["list"].dump(-1, ' ', true) == j["list"].dump(-1, ' ', true));
CHECK(d.root()["del"].dump(-1, ' ', true) == j["del"].dump(-1, ' ', true));
// strings that an edit wrote (the source strings that they replace were plain)
ordered_json_editable_document e = ordered_json_editable_document::parse(text);
ordered_json k = j;
e.set(e.root()["plain"], std::string("\xC3\xA4gypten"));
k["plain"] = "\xC3\xA4gypten";
e.set(e.root()["list"][0], std::string("ascii after edit"));
k["list"][0] = "ascii after edit";
e.set(e.root()["list"][1], std::string("\xC3\xA4"));
k["list"][1] = "\xC3\xA4";
e.set(e.root()["del"], std::string("tab\there"));
k["del"] = "tab\there";
CHECK(e.root().dump(-1, ' ', true) == k.dump(-1, ' ', true));
CHECK(e.root().dump(2, ' ', true) == k.dump(2, ' ', true));
CHECK(e.root().dump() == k.dump());
// strings that were edited from or to ASCII text
e.set(e.root()["list"][2], std::string("\xC3\xA4"));
k["list"][2] = "\xC3\xA4";
e.set(e.root()["list"][4], std::string("plain again"));
k["list"][4] = "plain again";
CHECK(e.root().dump(-1, ' ', true) == k.dump(-1, ' ', true));
}
}
SECTION("ensure_ascii with DEL or a non-ASCII character at every position")
{
// the check whether a string of the source has no byte >= 0x7F reads
// eight bytes at a time and then the rest one by one
for (std::size_t length = 1; length <= 20; ++length)
{
for (std::size_t pos = 0; pos < length; ++pos)
{
for (const char* const c :
{
"\x7F", "\x7E", "\xC3\xA4", "\xC3\xBF" // NOLINT(modernize-raw-string-literal)
})
{
const std::string s = std::string(pos, 'a') + c + std::string(length - pos - 1, 'b');
std::string text = "[\"";
text.append(s).append("\",{\"").append(s).append("\":1}]");
const json_document d = json_document::parse(text);
const json j = json::parse(text);
CHECK(d.root().dump(-1, ' ', true) == j.dump(-1, ' ', true));
CHECK(d.root().dump(1, ' ', true) == j.dump(1, ' ', true));
}
}
}
}
SECTION("streams and discarded views")
{
const json_document d = json_document::parse(R"({"a": [1, 2]})");
std::ostringstream compact;
compact << d.root();
CHECK(compact.str() == R"({"a":[1,2]})");
std::ostringstream pretty;
pretty << std::setw(2) << std::setfill('.') << d.root() << d.root()["a"];
CHECK(pretty.str() == "{\n..\"a\": [\n....1,\n....2\n..]\n}[1,2]");
CHECK(json_view().dump() == json(json::value_t::discarded).dump());
}
}
+11 -11
View File
@@ -532,18 +532,18 @@ TEST_CASE("json_view edits: errors")
CHECK_THROWS_WITH_AS(d.set(other_editable.root(), 1), "[json.exception.invalid_iterator.202] view does not belong to this document", json::invalid_iterator&);
CHECK_THROWS_WITH_AS(d.set(json_editable_view(), 1), "[json.exception.invalid_iterator.202] view does not belong to this document", json::invalid_iterator&);
CHECK_THROWS_WITH_AS(d.set(root["a"], "k", 1), json_view_test::message("type_error.305", "/a", 21, 27, "cannot use operator[] with a string argument with array").c_str(), json::type_error&);
CHECK_THROWS_WITH_AS(d.set(root["o"], 0, 1), json_view_test::message("type_error.305", "/o", 6, 14, "cannot use operator[] with a numeric argument with object").c_str(), json::type_error&);
CHECK_THROWS_WITH_AS(d.set(root["a"], 2, 1), json_view_test::message("out_of_range.401", "/a", 21, 27, "array index 2 is out of range").c_str(), json::out_of_range&);
CHECK_THROWS_WITH_AS(d.set(root["a"], -1, 1), json_view_test::message("out_of_range.401", "/a", 21, 27, "array index -1 is out of range").c_str(), json::out_of_range&);
CHECK_THROWS_WITH_AS(d.push_back(root["o"], 1), json_view_test::message("type_error.308", "/o", 6, 14, "cannot use push_back() with object").c_str(), json::type_error&);
CHECK_THROWS_WITH_AS(d.insert(root["n"], 0, 1), json_view_test::message("type_error.309", "/n", 34, 35, "cannot use insert() with number").c_str(), json::type_error&);
CHECK_THROWS_WITH_AS(d.insert(root["a"], 3, 1), json_view_test::message("out_of_range.401", "/a", 21, 27, "array index 3 is out of range").c_str(), json::out_of_range&);
CHECK_THROWS_WITH_AS(d.erase(root["n"], "k"), json_view_test::message("type_error.307", "/n", 34, 35, "cannot use erase() with number").c_str(), json::type_error&);
CHECK_THROWS_WITH_AS(d.erase(root["o"], 0), json_view_test::message("type_error.307", "/o", 6, 14, "cannot use erase() with object").c_str(), json::type_error&);
CHECK_THROWS_WITH_AS(d.erase(root["a"], 2), json_view_test::message("out_of_range.401", "/a", 21, 27, "array index 2 is out of range").c_str(), json::out_of_range&);
CHECK_THROWS_WITH_AS(d.set(root["a"], "k", 1), "[json.exception.type_error.305] cannot use operator[] with a string argument with array", json::type_error&);
CHECK_THROWS_WITH_AS(d.set(root["o"], 0, 1), "[json.exception.type_error.305] cannot use operator[] with a numeric argument with object", json::type_error&);
CHECK_THROWS_WITH_AS(d.set(root["a"], 2, 1), "[json.exception.out_of_range.401] array index 2 is out of range", json::out_of_range&);
CHECK_THROWS_WITH_AS(d.set(root["a"], -1, 1), "[json.exception.out_of_range.401] array index -1 is out of range", json::out_of_range&);
CHECK_THROWS_WITH_AS(d.push_back(root["o"], 1), "[json.exception.type_error.308] cannot use push_back() with object", json::type_error&);
CHECK_THROWS_WITH_AS(d.insert(root["n"], 0, 1), "[json.exception.type_error.309] cannot use insert() with number", json::type_error&);
CHECK_THROWS_WITH_AS(d.insert(root["a"], 3, 1), "[json.exception.out_of_range.401] array index 3 is out of range", json::out_of_range&);
CHECK_THROWS_WITH_AS(d.erase(root["n"], "k"), "[json.exception.type_error.307] cannot use erase() with number", json::type_error&);
CHECK_THROWS_WITH_AS(d.erase(root["o"], 0), "[json.exception.type_error.307] cannot use erase() with object", json::type_error&);
CHECK_THROWS_WITH_AS(d.erase(root["a"], 2), "[json.exception.out_of_range.401] array index 2 is out of range", json::out_of_range&);
CHECK_THROWS_WITH_AS(d.erase(json::json_pointer("")), "[json.exception.out_of_range.405] JSON pointer has no parent", json::out_of_range&);
CHECK_THROWS_WITH_AS(d.erase(json::json_pointer("/missing/x")), json_view_test::message("out_of_range.403", "", 0, 47, "key 'missing' not found").c_str(), json::out_of_range&);
CHECK_THROWS_WITH_AS(d.erase(json::json_pointer("/missing/x")), "[json.exception.out_of_range.403] key 'missing' not found", json::out_of_range&);
CHECK_THROWS_WITH_AS(d.set(json::json_pointer("/a/01"), 1), "[json.exception.parse_error.106] parse error: array index '01' must not begin with '0'", json::parse_error&);
CHECK_THROWS_WITH_AS(d.set(root, json_editable_view()), "[json.exception.type_error.302] type must be a value, but is discarded", json::type_error&);
CHECK_THROWS_WITH_AS(d.set(root, json::binary({1, 2})), "[json.exception.type_error.319] cannot store a binary value in a json_document", json::type_error&);
+7 -94
View File
@@ -39,7 +39,6 @@ using nlohmann::ordered_json_document;
#ifdef JSON_HAS_CPP_17
#include <optional>
#include <string_view>
#endif
// NLOHMANN_JSON_SERIALIZE_ENUM uses a static std::pair; same() compares floating-point values on purpose
@@ -338,7 +337,8 @@ TEST_CASE("json_view from_json")
CHECK(d.root().get<std::unordered_map<std::string, int>>().at("a") == 3);
}
#if !defined(JSON_NOEXCEPTION)
// (the exceptions of views have neither the path nor the byte range of the value)
#if !defined(JSON_NOEXCEPTION) && !JSON_DIAGNOSTICS && !JSON_DIAGNOSTIC_POSITIONS
SECTION("std types: the same errors as from the materialized value")
{
const json_document d = json_document::parse(R"([[1, "x"], {"a": 1}, "text", [1], 5, null])");
@@ -403,10 +403,10 @@ TEST_CASE("json_view from_json")
CHECK(d.root()[4].get<strict_color>() == strict_color::red);
#if !defined(JSON_NOEXCEPTION)
const json j = d.root().materialize();
#if !JSON_DIAGNOSTICS && !JSON_DIAGNOSTIC_POSITIONS
CHECK(exception_of_call([&] { static_cast<void>(d.root()[2].get<strict_color>()); }) == "[json.exception.out_of_range.410] enum value out of range for strict_color: \"purple\"");
#endif
#if !JSON_DIAGNOSTICS && !JSON_DIAGNOSTIC_POSITIONS
CHECK(exception_of_call([&] { static_cast<void>(d.root()[2].get<strict_color>()); }) == exception_of_call([&] { static_cast<void>(j[2].get<strict_color>()); }));
#endif
#endif
}
@@ -443,30 +443,6 @@ TEST_CASE("json_view from_json")
CHECK(d.root().materialize().get<point>().x == 3);
}
SECTION("escaped keys: members are found by their decoded names")
{
// keys are decoded before they are compared, as by parse(); a lookup
// that compares the raw bytes of the source would miss these members
const json_document d = json_document::parse(R"([{"\u0078": 1, "y": 2}, {"V\u0061lue": 3, "Te\u0078t": "t"}, {"\u0062": "escaped"}, {"x": 1, "\u0078": 3, "y": 2}])");
const json j = d.root().materialize();
CHECK(d.root()[0].get<point>().x == 1);
CHECK(d.root()[0].get<point>().y == 2);
CHECK((d.root()[0].get<point>() == j[0].get<point>()));
CHECK(d.root()[1].get<with_names>().value == 3);
CHECK(d.root()[1].get<with_names>().text == "t");
CHECK(d.root()[2].get<with_defaults>().b == "escaped");
CHECK(d.root()[2].get<with_defaults>().a == 1);
// a duplicate key written once plain and once escaped: still the first member
CHECK(d.root()[3].get<point>().x == 1);
CHECK(j[3].get<point>().x == 3);
#if !defined(JSON_NOEXCEPTION) && !JSON_DIAGNOSTICS && !JSON_DIAGNOSTIC_POSITIONS
// an escaped backslash is part of the decoded key: "x\\" is not "x"
const json_document e = json_document::parse(R"({"x\\": 1, "y": 2})");
CHECK(exception_of_call([&] { static_cast<void>(e.root().get<point>()); }) == exception_of_call([&] { static_cast<void>(e.root().materialize().get<point>()); }));
CHECK(exception_of_call([&] { static_cast<void>(e.root().get<point>()); }) == "[json.exception.out_of_range.403] key 'x' not found");
#endif
}
SECTION("editable documents and ordered_json")
{
json_editable_document e = json_editable_document::parse(R"({"x": 1, "y": 2})");
@@ -476,9 +452,11 @@ TEST_CASE("json_view from_json")
CHECK(o.root().get<shape>().center.y == 2);
}
#if !defined(JSON_NOEXCEPTION)
#if !defined(JSON_NOEXCEPTION) && !JSON_DIAGNOSTICS && !JSON_DIAGNOSTIC_POSITIONS
SECTION("errors are those of the same conversion of a json value")
{
// (with JSON_DIAGNOSTICS, those of json name the path of the value,
// and with JSON_DIAGNOSTIC_POSITIONS its bytes)
const json_document d = json_document::parse(R"([{"x": 1}, {"x": "1", "y": 2}, [1, 2], {"id": "7", "c": "red"}])");
const json j = d.root().materialize();
for (std::size_t i = 0; i < 3; ++i)
@@ -491,70 +469,5 @@ TEST_CASE("json_view from_json")
#endif
}
#ifdef JSON_HAS_CPP_17
namespace view_from_json
{
// a string_view member is not copied: it points into the source text or, for a
// string with escape sequences, into the buffer of decoded strings of the document
struct named_view
{
// initialized for GCC -Weffc++
// NOLINTNEXTLINE(readability-redundant-member-init)
std::string_view name{};
int n = 0;
NLOHMANN_DEFINE_TYPE_INTRUSIVE(named_view, name, n)
};
} // namespace view_from_json
TEST_CASE("json_view from_json: string_view members borrow from the document")
{
using view_from_json::named_view;
// whether all of s is inside the bytes of text
const auto within = [](std::string_view s, std::string_view text)
{
const auto first = reinterpret_cast<std::uintptr_t>(text.data());
const auto begin = reinterpret_cast<std::uintptr_t>(s.data());
return begin >= first && begin + s.size() <= first + text.size();
};
SECTION("a string without escapes points into the source text")
{
const std::string text = R"({"name": "hello", "n": 1})";
const json_document d = json_document::parse(text);
REQUIRE_FALSE(d.owns_source());
const auto r = d.root().get<named_view>();
CHECK(r.name == "hello");
CHECK(r.n == 1);
CHECK(within(r.name, text));
CHECK(within(r.name, d.source()));
}
SECTION("a string with escapes points into the document, not into the source text")
{
const std::string text = R"({"name": "a\nb", "n": 2})";
json_document d = json_document::parse(text);
REQUIRE_FALSE(d.owns_source());
const auto r = d.root().get<named_view>();
CHECK(r.name == "a\nb");
CHECK(r.n == 2);
CHECK_FALSE(within(r.name, text));
// valid as long as the document is alive, even if the document is moved
const json_document moved = std::move(d);
CHECK(r.name == "a\nb");
CHECK(moved.root().get<named_view>().name == r.name);
}
SECTION("a document that owns its text keeps the members readable while it is alive")
{
const json_document d = json_document::parse(std::string(R"({"name": "owned", "n": 3})"));
REQUIRE(d.owns_source());
const auto r = d.root().get<named_view>();
CHECK(r.name == "owned");
CHECK(within(r.name, d.source()));
}
}
#endif
DOCTEST_CLANG_SUPPRESS_WARNING_POP
DOCTEST_GCC_SUPPRESS_WARNING_POP
-544
View File
@@ -1,544 +0,0 @@
// __ _____ _____ _____
// __| | __| | | | JSON for Modern C++ (supporting code)
// | | |__ | | | | | | version 3.12.0
// |_____|_____|_____|_|___| https://github.com/nlohmann/json
//
// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
// SPDX-License-Identifier: MIT
#include "doctest_compatibility.h"
#include <nlohmann/json_view.hpp>
using nlohmann::json;
using nlohmann::ordered_json;
using nlohmann::json_document;
using nlohmann::ordered_json_editable_document;
using nlohmann::json_view;
using nlohmann::ordered_json_document;
using nlohmann::ordered_json_view;
#include "json_view_test_helpers.hpp"
using json_view_test::generator;
#include <algorithm>
#include <array>
#include <cmath>
#include <cstddef>
#include <cstdint>
#include <cstring>
#include <iomanip>
#include <map>
#include <random>
#include <set>
#include <sstream>
#include <string>
#include <vector>
// These tests were split off unit-json_view_dump.cpp, whose object file got too large for the MinGW
// linker (see json_view_test_helpers.hpp). Like that file, this one is also built
// with C++17, as it mentions JSON_HAS_CPP_17.
TEST_CASE("json_view comparison")
{
SECTION("equality of the values parse() produces")
{
generator g;
std::vector<std::string> texts;
for (int i = 0; i < 600; ++i)
{
std::string text;
g.value(text, 0);
texts.push_back(text);
// the same value written differently: sorted keys, canonical numbers
texts.push_back(json::parse(text).dump(1));
}
for (std::size_t i = 0; i + 2 < texts.size(); ++i)
{
for (std::size_t k = i; k < i + 3; ++k)
{
CAPTURE(texts[i])
CAPTURE(texts[k])
const json_document a = json_document::parse(texts[i]);
const json_document b = json_document::parse(texts[k]);
const json ja = json::parse(texts[i]);
const json jb = json::parse(texts[k]);
CHECK((a.root() == b.root()) == (ja == jb));
CHECK((a.root() != b.root()) == (ja != jb));
CHECK((a.root() == jb) == (ja == jb));
CHECK((jb == a.root()) == (ja == jb));
CHECK((a.root() != jb) == (ja != jb));
CHECK((jb != a.root()) == (ja != jb));
// ordered_json compares members in order
const ordered_json_document oa = ordered_json_document::parse(texts[i]);
const ordered_json_document ob = ordered_json_document::parse(texts[k]);
const ordered_json oja = ordered_json::parse(texts[i]);
const ordered_json ojb = ordered_json::parse(texts[k]);
CHECK((oa.root() == ob.root()) == (oja == ojb));
CHECK((oa.root() == ojb) == (oja == ojb));
}
}
}
SECTION("numbers, duplicate keys, member order")
{
const auto same = [](const char* x, const char* y)
{
const json_document dx = json_document::parse(x);
const json_document dy = json_document::parse(y);
return dx.root() == dy.root();
};
CHECK(same("1", "1.0"));
CHECK(same("[1, -1, 2.5]", "[1.0, -1.0, 25e-1]"));
CHECK(!same("1", "1.5"));
CHECK(same("18446744073709551615", "18446744073709551615"));
CHECK(same(R"({"a": 1, "a": 2})", R"({"a": 2})"));
CHECK(!same(R"({"a": 1, "a": 2})", R"({"a": 1})"));
CHECK(same(R"({"a": 1, "b": 2})", R"({"b": 2, "a": 1})"));
CHECK(!same(R"({"a": 1})", R"({"a": 1, "b": 2})"));
CHECK(!same("[1, 2]", "[2, 1]"));
CHECK(!same("\"a\"", "\"b\""));
CHECK(same("\"\\u00e9\"", "\"\xc3\xa9\""));
CHECK(!same("null", "false"));
CHECK(!same("[]", "{}"));
const ordered_json_document dup = ordered_json_document::parse(R"({"a": 1, "b": 2, "a": 3})");
const ordered_json_document last = ordered_json_document::parse(R"({"a": 3, "b": 2})");
CHECK(dup.root() == last.root());
// (== compares as parse() resolves duplicates, a lookup finds the first)
CHECK(dup.root()["a"].materialize() == 1);
CHECK(dup.root()["a"] != last.root()["a"]);
const ordered_json_document ab = ordered_json_document::parse(R"({"a": 1, "b": 2})");
const ordered_json_document ba = ordered_json_document::parse(R"({"b": 2, "a": 1})");
CHECK(ab.root() != ba.root());
// discarded values compare as basic_json's do
const json discarded(json::value_t::discarded);
CHECK((json_view() == json_view()) == (discarded == discarded)); // NOLINT(readability-container-size-empty): operator== is tested
CHECK((json_view() == discarded) == (discarded == discarded)); // NOLINT(readability-container-size-empty)
const json_document null_document = json_document::parse("null");
CHECK(!(json_view() == null_document.root())); // NOLINT(readability-container-size-empty)
CHECK(!(null_document.root() == discarded));
}
SECTION("deep nesting")
{
const std::string deep = std::string(100000, '[') + std::string(100000, ']');
const json_document a = json_document::parse(deep);
const json_document b = json_document::parse(deep);
CHECK(a.root() == b.root());
CHECK(a.root() == json::parse(deep));
const std::string other = std::string(100000, '[') + "1" + std::string(100000, ']');
const json_document c = json_document::parse(other);
CHECK(a.root() != c.root());
}
}
TEST_CASE("json_view large objects")
{
// objects with 128 members or more are looked up with a hash index
for (const std::size_t members :
{
127u, 128u, 129u, 10000u
})
{
CAPTURE(members)
std::string text = "{";
for (std::size_t i = 0; i < members; ++i)
{
text += (i != 0 ? ",\"" : "\"") + std::string(i % 23, 'k') + std::to_string(i) + (i % 7 == 0 ? "\\n" : "") + "\":" + std::to_string(i);
}
text += R"(,"":"empty key","k1":"a duplicate of an earlier key"})";
const json_document d = json_document::parse(text);
const json_view v = d.root();
const json j = json::parse(text);
for (std::size_t i = 0; i < members; ++i)
{
const std::string key = std::string(i % 23, 'k') + std::to_string(i) + (i % 7 == 0 ? "\n" : "");
CHECK(v[key].get<std::size_t>() == i);
CHECK(v.contains(key));
CHECK(v.find(key).key() == key);
CHECK(v.at(key).get<std::size_t>() == i);
CHECK(!v.contains(key + "x"));
}
CHECK(v[""].get_string() == "empty key");
CHECK(v["k1"].get<int>() == 1); // the first of duplicate keys, as for small objects
CHECK(!v.contains("missing"));
CHECK_THROWS_WITH_AS(v.at("missing"), json_view_test::message("out_of_range.403", "", 0, text.size(), "key 'missing' not found").c_str(), json::out_of_range&);
CHECK(v == j);
CHECK(v.materialize() == j);
}
SECTION("duplicate keys: the first member wins, with and without a table")
{
// an object of `total` members: the keys "k0".."k<n-1>" in order, then
// three keys repeated twice more (one copy in the middle, one at the
// end), and two keys repeated once; the value of a member is its
// position, so that the first member of a key can be told apart
struct member
{
std::string key;
std::size_t position;
};
const auto make_members = [](std::size_t total)
{
std::vector<std::string> keys;
for (std::size_t i = 0; i + 8 < total; ++i)
{
keys.push_back("k" + std::to_string(i));
}
const std::size_t n = keys.size();
const std::array<std::size_t, 3> triple = {{3, 17, n - 1}};
const std::array<std::size_t, 2> twice = {{5, n / 2}};
std::vector<std::string> ordered = keys;
for (const std::size_t i : triple)
{
ordered.insert(ordered.begin() + static_cast<std::ptrdiff_t>(ordered.size() / 2), keys[i]);
}
for (const std::size_t i : twice)
{
ordered.insert(ordered.begin() + static_cast<std::ptrdiff_t>(ordered.size() / 3), keys[i]);
}
for (const std::size_t i : triple)
{
ordered.push_back(keys[i]);
}
std::vector<member> result;
result.reserve(ordered.size());
for (std::size_t i = 0; i < ordered.size(); ++i)
{
result.push_back({ordered[i], i});
}
return result;
};
// 100 and 127 members: no table; 128 members and more: a table
for (const std::size_t total :
{
100u, 127u, 128u, 200u, 5000u
})
{
CAPTURE(total)
const std::vector<member> members = make_members(total);
REQUIRE(members.size() >= total);
REQUIRE(members.size() >= 100);
std::string text = "{";
std::map<std::string, std::size_t> first;
std::map<std::string, std::size_t> last;
std::map<std::string, std::size_t> occurrences;
for (const member& m : members)
{
text += (text.size() > 1 ? ",\"" : "\"") + m.key + "\":" + std::to_string(m.position);
first.insert({m.key, m.position});
last[m.key] = m.position;
++occurrences[m.key];
}
text += '}';
REQUIRE(first.size() < members.size());
const json_document d = json_document::parse(text);
const json_view v = d.root();
const json j = json::parse(text);
CHECK(v.size() == members.size()); // every occurrence is visited
const json m = v.materialize();
for (const auto& entry : first)
{
CAPTURE(entry.first)
const std::size_t expected = entry.second;
CHECK(v[entry.first].get<std::size_t>() == expected);
CHECK(v.at(entry.first).get<std::size_t>() == expected);
CHECK(v.find(entry.first).value().get<std::size_t>() == expected);
CHECK(v.value(entry.first, std::size_t{0}) == expected);
CHECK(v.contains(entry.first));
CHECK(v.count(entry.first) == occurrences[entry.first]);
const std::string pointer = "/" + entry.first;
CHECK(v[json::json_pointer(pointer)].get<std::size_t>() == expected);
CHECK(v.at(json::json_pointer(pointer)).get<std::size_t>() == expected);
CHECK(v.value(json::json_pointer(pointer), std::size_t{0}) == expected);
CHECK(v.contains(json::json_pointer(pointer)));
// materialize() and parse() keep the last value instead
CHECK(j[entry.first].get<std::size_t>() == last.at(entry.first));
CHECK(m[entry.first].get<std::size_t>() == last.at(entry.first));
}
CHECK(!v.contains("k"));
CHECK(v["missing"].is_discarded());
CHECK(v.materialize() == j);
CHECK(v == j);
}
}
SECTION("colliding keys")
{
// the hash is not seeded, so keys that all land in one place must not
// make the table build quadratic: such an object gets no table and is
// searched linearly
constexpr std::size_t members = 300;
constexpr std::size_t table_size = 1024; // the table size for 300 members: the next power of two >= 600
std::vector<std::string> colliding;
std::vector<std::string> spread;
for (std::uint64_t counter = 0; colliding.size() < members || spread.size() < members; ++counter)
{
std::string key(8, 'a');
std::uint64_t x = counter;
for (std::size_t i = 0; i < 8; ++i, x /= 26)
{
key[i] = static_cast<char>('a' + (x % 26));
}
const bool lands_in_slot_zero = (nlohmann::detail::view::key_hash(key.data(), key.size()) & (table_size - 1)) == 0;
if (lands_in_slot_zero && colliding.size() < members)
{
colliding.push_back(key);
}
else if (!lands_in_slot_zero && spread.size() < members)
{
spread.push_back(key);
}
}
const auto make_text = [](const std::vector<std::string>& keys)
{
std::string text = "{";
for (std::size_t i = 0; i < keys.size(); ++i)
{
text += (i != 0 ? ",\"" : "\"") + keys[i] + "\":" + std::to_string(i % 10);
}
return text + "}";
};
const std::string colliding_text = make_text(colliding);
const std::string spread_text = make_text(spread);
json_document with_collisions = json_document::parse(colliding_text);
json_document without_collisions = json_document::parse(spread_text);
for (const auto* pair :
{
&colliding, &spread
})
{
const json_view v = (pair == &colliding ? with_collisions : without_collisions).root();
for (std::size_t i = 0; i < members; ++i)
{
CAPTURE(i)
CHECK(v[(*pair)[i]].get<std::size_t>() == i % 10);
CHECK(v.at((*pair)[i]).get<std::size_t>() == i % 10);
CHECK(v.find((*pair)[i]).key() == (*pair)[i]);
CHECK(!v.contains((*pair)[i] + "x"));
}
CHECK(!v.contains("missing"));
}
CHECK(with_collisions.root() == json::parse(colliding_text));
// only the object with the spread keys got a table (both texts have the
// same length, so the tables are the only difference)
with_collisions.shrink_to_fit();
without_collisions.shrink_to_fit();
CHECK(without_collisions.memory_usage() >= with_collisions.memory_usage() + (table_size * sizeof(std::uint32_t)));
}
SECTION("keys that differ in their last bytes are spread over the table")
{
// The hash must spread the usual families of keys (numbered names, hex
// strings, dates, short names): a key farther than index_max_displacement
// from its home slot would leave the object without a table. The keys
// are placed with linear probing, as the table does.
const auto max_displacement = [](const std::vector<std::string>& keys)
{
std::size_t slots = 16;
while (slots < 2 * keys.size())
{
slots *= 2;
}
std::vector<bool> used(slots, false);
std::size_t farthest = 0;
for (const std::string& key : keys)
{
std::size_t i = nlohmann::detail::conditional_static_cast<std::size_t>(nlohmann::detail::view::key_hash(key.data(), key.size())) & (slots - 1);
std::size_t distance = 0;
while (used[i])
{
i = (i + 1) & (slots - 1);
++distance;
}
used[i] = true;
farthest = (std::max)(farthest, distance);
}
return farthest;
};
// the keys of a printf-like pattern with one %d, %0Nd or %0Nx, formatted with a stream
// (a format string that is not a literal is a Flawfinder error)
const auto numbered = [](const char* format, int count, int step)
{
const std::string pattern(format);
const std::size_t percent = pattern.find('%');
std::size_t conversion = percent + 1;
std::size_t width = 0;
while (pattern[conversion] >= '0' && pattern[conversion] <= '9')
{
width = (width * 10) + static_cast<std::size_t>(pattern[conversion] - '0');
++conversion;
}
const bool hex = pattern[conversion] == 'x';
std::vector<std::string> keys;
for (int i = 0; i < count; ++i)
{
std::ostringstream key;
key << pattern.substr(0, percent) << std::setfill('0') << std::setw(static_cast<int>(width))
<< (hex ? std::hex : std::dec) << (i * step) << pattern.substr(conversion + 1);
keys.push_back(key.str());
}
return keys;
};
for (const char* format :
{"key%d", "%d", "item_%04d", "a%05d", "%03dxyz", "id%d", "user_%d", "%d_suffix", "0x%04x", "A%07d", "12345678%d", "1234567%d", "123456789%d",
"configuration_parameter_value_%04d_extra", "2026-01-%02dT00:00:00Z"
})
{
CAPTURE(format)
CHECK(max_displacement(numbered(format, std::string(format).find("%02d") != std::string::npos ? 60 : 1000, 1)) < nlohmann::detail::view::index_max_displacement); // NOLINT(abseil-string-find-str-contains)
CHECK(max_displacement(numbered(format, 600, 7)) < nlohmann::detail::view::index_max_displacement);
}
std::mt19937_64 rng(7); // NOLINT(cert-msc32-c,cert-msc51-cpp,bugprone-random-generator-seed)
for (const std::size_t length :
{
std::size_t{1}, std::size_t{2}, std::size_t{3}, std::size_t{5}, std::size_t{7}, std::size_t{8}, std::size_t{9}, std::size_t{12}, std::size_t{16}, std::size_t{17}, std::size_t{33}
})
{
CAPTURE(length)
std::set<std::string> distinct;
// (there are 26 and 676 keys of one and two letters)
std::size_t wanted = 700;
if (length == 1)
{
wanted = 26;
}
else if (length == 2)
{
wanted = 500;
}
while (distinct.size() < wanted)
{
std::string key(length, 'a');
for (char& c : key)
{
c = static_cast<char>('a' + (rng() % 26));
}
distinct.insert(key);
}
CHECK(max_displacement(std::vector<std::string>(distinct.begin(), distinct.end())) < nlohmann::detail::view::index_max_displacement);
}
}
SECTION("keys of every length that differ in one byte")
{
// In a large object, a key is found by comparing its bytes with those of the member the
// table points to: up to 16 bytes as two overlapping words, longer ones with memcmp.
// Keys of every length from 0 to 40, and keys that differ from them in one byte.
std::mt19937_64 rng(2026); // NOLINT(cert-msc32-c,cert-msc51-cpp,bugprone-random-generator-seed)
std::vector<std::string> keys;
std::set<std::string> distinct;
for (std::size_t length = 0; length <= 40; ++length)
{
for (int copy = 0; copy < 5; ++copy)
{
std::string key(length, 'a');
for (char& c : key)
{
c = static_cast<char>('a' + (rng() % 26));
}
if (distinct.insert(key).second)
{
keys.push_back(key);
}
}
}
std::string text = "{";
for (std::size_t i = 0; i < keys.size(); ++i)
{
text += (i != 0 ? ",\"" : "\"") + keys[i] + "\":" + std::to_string(i);
}
text += "}";
REQUIRE(keys.size() >= 128); // (a table is built for 128 members)
const json_document d = json_document::parse(text);
const json_view v = d.root();
for (std::size_t i = 0; i < keys.size(); ++i)
{
CAPTURE(keys[i])
CHECK(v[keys[i]].get<std::size_t>() == i);
CHECK(v.at(keys[i]).get<std::size_t>() == i);
CHECK(v.contains(keys[i]));
for (std::size_t at = 0; at < keys[i].size(); ++at)
{
std::string other = keys[i];
other[at] = other[at] == 'z' ? 'y' : 'z';
CAPTURE(other)
CHECK(v.contains(other) == (distinct.count(other) != 0));
}
CHECK(v.contains(keys[i] + "a") == (distinct.count(keys[i] + "a") != 0));
if (!keys[i].empty())
{
CHECK(v.contains(keys[i].substr(1)) == (distinct.count(keys[i].substr(1)) != 0));
}
}
}
SECTION("shrink_to_fit releases the tables' spare capacity")
{
const auto make_text = [](int objects, int members)
{
std::string text = "[";
for (int object = 0; object < objects; ++object)
{
text += object != 0 ? ",{" : "{";
for (int i = 0, n = members + object; i < n; ++i)
{
text += (i != 0 ? ",\"" : "\"") + std::to_string(i) + "\":" + std::to_string(i);
}
text += '}';
}
return text + "]";
};
const std::string small_text = make_text(5, 150);
const std::string big_text = make_text(40, 400);
// reading a big text, and then a small one, leaves the spare capacity
// of the big one: shrink_to_fit() brings the document to the size of
// one parsed from the small text alone
json_document d = json_document::parse(big_text);
const std::size_t big = d.memory_usage();
d.read(small_text);
CHECK(d.memory_usage() >= big);
d.shrink_to_fit();
json_document fresh = json_document::parse(small_text);
fresh.shrink_to_fit();
CHECK(d.memory_usage() == fresh.memory_usage());
CHECK(d.memory_usage() < big / 2);
CHECK(d.root() == json::parse(small_text));
for (int object = 0; object < 5; ++object)
{
const json_view v = d.root()[static_cast<std::size_t>(object)];
for (int i = 0, n = 150 + object; i < n; ++i)
{
CHECK(v[std::to_string(i)].get<int>() == i);
}
}
}
SECTION("nested, reused, and in arrays")
{
std::string inner = "{";
for (int i = 0; i < 300; ++i)
{
inner += (i != 0 ? ",\"m" : "\"m") + std::to_string(i) + "\":" + std::to_string(i);
}
inner += '}';
const std::string text = "[" + inner + ",{\"x\":" + inner + "}," + inner + "]";
json_document d = json_document::parse(text);
CHECK(d.root()[0]["m299"].get<int>() == 299);
CHECK(d.root()[1]["x"]["m150"].get<int>() == 150);
CHECK(d.root()[2]["m0"].get<int>() == 0);
const std::size_t with_index = d.memory_usage();
d.read(std::string("{\"small\": 1}"));
CHECK(d.root()["small"].get<int>() == 1);
d.read(text);
CHECK(d.root()[2]["m7"].get<int>() == 7);
CHECK(d.memory_usage() >= with_index / 2);
}
}
+1 -1
View File
@@ -25,7 +25,7 @@
#endif
// the view's own macros do not leak
#if defined(NLOHMANN_VIEW_LIKELY) || defined(NLOHMANN_VIEW_UNLIKELY) || defined(NLOHMANN_VIEW_ALWAYS_INLINE) || defined(NLOHMANN_VIEW_NOINLINE) || defined(NLOHMANN_VIEW_NOCLONE) \
#if defined(NLOHMANN_VIEW_LIKELY) || defined(NLOHMANN_VIEW_UNLIKELY) || defined(NLOHMANN_VIEW_ALWAYS_INLINE) || defined(NLOHMANN_VIEW_NOINLINE) \
|| defined(NLOHMANN_VIEW_NODISCARD) || defined(NLOHMANN_VIEW_THROW) || defined(NLOHMANN_VIEW_HAS_CPP_17) || defined(NLOHMANN_VIEW_LITTLE_ENDIAN) \
|| defined(NLOHMANN_VIEW_REPEAT16)
#error "json_view.hpp leaks a macro"
-227
View File
@@ -81,112 +81,6 @@ std::string range(const json& j)
{
return j.start_pos() == npos && j.end_pos() == npos ? "none" : std::to_string(j.start_pos()) + "-" + std::to_string(j.end_pos());
}
#if !defined(JSON_NOEXCEPTION)
using json_view_test::exception_of_call;
// the exceptions about every value of v are those about the value at the same
// place in j, a value of the whole document from parse() or materialize() (so
// that its parents and positions are set); members whose key repeats are
// skipped (the view finds the first one, j holds the last)
template<typename View>
void check_exceptions(const View& v, const json& j)
{
CAPTURE(j.dump())
if (!v.is_number())
{
CHECK(exception_of_call([&] { static_cast<void>(v.template get<int>()); }) == exception_of_call([&] { static_cast<void>(j.get<int>()); }));
}
if (!v.is_string())
{
CHECK(exception_of_call([&] { static_cast<void>(v.template get<std::string>()); }) == exception_of_call([&] { static_cast<void>(j.get<std::string>()); }));
}
if (!v.is_array())
{
CHECK(exception_of_call([&] { static_cast<void>(v.template get<std::vector<int>>()); }) == exception_of_call([&] { static_cast<void>(j.get<std::vector<int>>()); }));
}
CHECK(exception_of_call([&] { static_cast<void>(v.at("missing")); }) == exception_of_call([&] { static_cast<void>(j.at("missing")); }));
CHECK(exception_of_call([&] { static_cast<void>(v.at(1000)); }) == exception_of_call([&] { static_cast<void>(j.at(1000)); }));
CHECK(exception_of_call([&] { static_cast<void>(v.at(json::json_pointer("/missing"))); }) == exception_of_call([&] { static_cast<void>(j.at(json::json_pointer("/missing"))); }));
CHECK(exception_of_call([&] { static_cast<void>(v.at(json::json_pointer("/-"))); }) == exception_of_call([&] { static_cast<void>(j.at(json::json_pointer("/-"))); }));
CHECK(exception_of_call([&] { static_cast<void>(v.value("k", 0)); }) == exception_of_call([&] { static_cast<void>(j.value("k", 0)); }));
if (!v.is_object())
{
CHECK(exception_of_call([&] { static_cast<void>(v["k"]); }) == exception_of_call([&] { static_cast<void>(j["k"]); }));
// (without the context of basic_json: view iterators do not know the value they iterate over)
CHECK(exception_of_call([&] { static_cast<void>(v.begin().key()); }) == "[json.exception.invalid_iterator.207] cannot use key() for non-object iterators");
}
if (!v.is_array())
{
CHECK(exception_of_call([&] { static_cast<void>(v[0]); }) == exception_of_call([&] { static_cast<void>(j[0]); }));
}
if (v.is_null())
{
CHECK(exception_of_call([&] { static_cast<void>(v.front()); }) == exception_of_call([&] { static_cast<void>(j.front()); }));
}
if (v.is_object())
{
for (auto it = v.begin(); it != v.end(); ++it)
{
const std::string key(it.key().data(), it.key().size());
if (v.count(key) == 1)
{
check_exceptions(it.value(), j.at(key));
}
}
}
else if (v.is_array())
{
for (std::size_t i = 0; i < v.size(); ++i)
{
check_exceptions(v[i], j[i]);
}
}
}
// as check_exceptions, for edits that fail (they change neither d nor j)
void check_edit_exceptions(json_editable_document& d, const json_editable_document::view_type& v, json& j)
{
CAPTURE(j.dump())
if (!v.is_object() && !v.is_null())
{
CHECK(exception_of_call([&] { d.set(v, "k", 1); }) == exception_of_call([&] { j["k"] = 1; }));
CHECK(exception_of_call([&] { d.erase(v, "k"); }) == exception_of_call([&] { j.erase("k"); }));
}
if (!v.is_array() && !v.is_null())
{
CHECK(exception_of_call([&] { d.set(v, 0, 1); }) == exception_of_call([&] { j[0] = 1; }));
CHECK(exception_of_call([&] { d.push_back(v, 1); }) == exception_of_call([&] { j.push_back(1); }));
}
if (!v.is_array())
{
CHECK(exception_of_call([&] { d.insert(v, 0, 1); }) == exception_of_call([&] { j.insert(j.cbegin(), 1); }));
CHECK(exception_of_call([&] { d.erase(v, 0); }) == exception_of_call([&] { j.erase(0); }));
}
else
{
CHECK(exception_of_call([&] { d.erase(v, 1000); }) == exception_of_call([&] { j.erase(1000); }));
}
if (v.is_object())
{
for (auto it = v.begin(); it != v.end(); ++it)
{
const std::string key(it.key().data(), it.key().size());
if (v.count(key) == 1)
{
check_edit_exceptions(d, it.value(), j.at(key));
}
}
}
else if (v.is_array())
{
for (std::size_t i = 0; i < v.size(); ++i)
{
check_edit_exceptions(d, v[i], j[i]);
}
}
}
#endif
} // namespace
TEST_CASE("json_view materialize() source positions")
@@ -451,124 +345,3 @@ TEST_CASE("json_view materialize() source positions")
}
#endif
}
#if !defined(JSON_NOEXCEPTION)
TEST_CASE("json_view exceptions name the path and the bytes of the value")
{
SECTION("as those of basic_json")
{
const std::vector<std::string> texts =
{
"null", "true", "42", "\"s\"", "[]", "{}", "[1, [2, [3, []]]]", R"({"a": {"b": [1, {"c": null}]}, "d": "e"})",
R"({"a/b": {"~": [1]}, "\u0041\n": {"x": "\t"}, "": [{}]})",
R"([1 /* ] */, {"a": [] // }
, "b": {} } , ])",
"\n\t[\r\n 1 ,\n {\n \"a\" : [ ]\n }\n]\n",
};
for (const auto& text : texts)
{
CAPTURE(text)
const json j = json::parse(text, nullptr, true, true, true);
const json_document d = json_document::parse(text, true, true, true);
check_exceptions(d.root(), j);
json_editable_document e = json_editable_document::parse(text, true, true, true);
check_exceptions(e.root(), j);
json m = e.root().materialize();
check_edit_exceptions(e, e.root(), m);
}
}
SECTION("generated documents")
{
json_view_test::generator gen;
for (int i = 0; i < 100; ++i)
{
std::string text;
gen.value(text, 0);
CAPTURE(text)
const json_document d = json_document::parse(text);
check_exceptions(d.root(), json::parse(text));
}
}
SECTION("edited documents: as those of materialize()")
{
const std::string text = R"({"a": [1, 2], "b": {"c": "\n", "d": true}, "e": [3, [4]], "f": "x"})";
json_editable_document d = json_editable_document::parse(text);
d.set(d.root()["f"], 5);
d.push_back(d.root()["a"], json::parse(R"({"n": [null]})"));
d.erase(d.root()["e"], 0);
d.set(d.root()["b"], "c2", d.root()["e"]);
d.insert(d.root()["e"], 0, "new");
json m = d.root().materialize();
check_exceptions(d.root(), m);
check_edit_exceptions(d, d.root(), m);
// (a value inside a new container: a path, but no bytes)
std::string expected = "[json.exception.type_error.302] ";
#if JSON_DIAGNOSTICS
expected += "(/a/2/n/0) ";
#endif
expected += "type must be number, but is null";
CHECK(exception_of_call([&] { static_cast<void>(d.root()["a"][2]["n"][0].get<int>()); }) == expected);
}
SECTION("deep documents with siblings after every nested container")
{
// the search for the path pushes and pops one frame per level (the
// stack reallocates while it grows, and shrinks again at every
// sibling): run under a sanitizer, a reference to a popped or moved
// frame would show here
std::string text = "7";
for (int level = 0; level < 40; ++level)
{
text.insert(0, R"([[1, [2]], )");
text += R"(, {"k": [3, {"l": [4]}]}, [5, [6, [7]]]])";
}
CAPTURE(text)
const json j = json::parse(text);
const json_document d = json_document::parse(text);
check_exceptions(d.root(), j);
json_editable_document e = json_editable_document::parse(text);
e.set(e.root()[0][0], 9);
check_exceptions(e.root(), e.root().materialize());
}
SECTION("examples")
{
const std::string text = R"({"people": [{"name": "a", "age": 1}, {"name": "c", "age": "x"}], "o": {"k": 1}})";
const json_document d = json_document::parse(text);
std::string expected_age = "[json.exception.type_error.302] ";
#if JSON_DIAGNOSTICS
expected_age += "(/people/1/age) ";
#endif
expected_age += "(bytes 58-61) type must be number, but is string";
CHECK(exception_of_call([&] { static_cast<void>(d.root()["people"][1]["age"].get<int>()); }) == expected_age);
std::string expected_zz = "[json.exception.out_of_range.403] ";
#if JSON_DIAGNOSTICS
expected_zz += "(/people/0) ";
#endif
expected_zz += "(bytes 12-35) key 'zz' not found";
CHECK(exception_of_call([&] { static_cast<void>(d.root().at(json::json_pointer("/people/0/zz"))); }) == expected_zz);
// the root has no path, and with JSON_DIAGNOSTICS then no bytes either (as basic_json)
std::string expected_missing = "[json.exception.out_of_range.403] ";
#if !JSON_DIAGNOSTICS
expected_missing += "(bytes 0-79) ";
#endif
expected_missing += "key 'missing' not found";
CHECK(exception_of_call([&] { static_cast<void>(d.root().at("missing")); }) == expected_missing);
}
SECTION("duplicate keys: the value the view finds")
{
const json_document d = json_document::parse(R"({"x": "1", "x": "22"})");
std::string expected = "[json.exception.type_error.302] ";
#if JSON_DIAGNOSTICS
expected += "(/x) ";
#endif
expected += "(bytes 6-9) type must be number, but is string";
CHECK(exception_of_call([&] { static_cast<void>(d.root()["x"].get<int>()); }) == expected);
}
}
#endif
+73 -99
View File
@@ -27,15 +27,12 @@ using nlohmann::json;
#include <sstream>
#include <iomanip>
#include <limits>
#include <list>
#include <set>
#include "make_test_data_available.hpp"
#include "round_trip_corpus.hpp"
#include "test_utils.hpp"
#include "sax_countdown.hpp"
#include "sax_event_loggers.hpp"
using utils::SaxCountdown;
using utils::SaxEventLogger;
TEST_CASE("MessagePack")
@@ -1648,6 +1645,30 @@ TEST_CASE("MessagePack")
}
}
}
SECTION("SAX aborts")
{
SECTION("start_array(len)")
{
std::vector<uint8_t> const v = {0x93, 0x01, 0x02, 0x03};
SaxCountdown scp(0);
CHECK(!json::sax_parse(v, &scp, json::input_format_t::msgpack));
}
SECTION("start_object(len)")
{
std::vector<uint8_t> const v = {0x81, 0xa3, 0x66, 0x6F, 0x6F, 0xc2};
SaxCountdown scp(0);
CHECK(!json::sax_parse(v, &scp, json::input_format_t::msgpack));
}
SECTION("key()")
{
std::vector<uint8_t> const v = {0x81, 0xa3, 0x66, 0x6F, 0x6F, 0xc2};
SaxCountdown scp(1);
CHECK(!json::sax_parse(v, &scp, json::input_format_t::msgpack));
}
}
}
TEST_CASE("issue #5405 - array reserve for definite-length MessagePack arrays")
@@ -1718,6 +1739,21 @@ TEST_CASE("issue #5405 - array reserve for definite-length MessagePack arrays")
CHECK(json::from_msgpack(packed) == j);
}
}
SECTION("a user-defined SAX consumer is unaffected by the internal DOM reserve optimization")
{
// the reserve() call is local to json_sax_dom_parser / json_sax_dom_callback_parser;
// a custom SAX consumer that does not touch a DOM array sees identical events
json j = json::array();
for (int i = 0; i < 100; ++i)
{
j.push_back(i);
}
const auto packed = json::to_msgpack(j);
SaxCountdown scp(1000000); // large enough to never trigger an abort
CHECK(json::sax_parse(packed, &scp, json::input_format_t::msgpack));
}
}
TEST_CASE("regression test - MessagePack ext type rejects a subtype that doesn't fit a single byte")
@@ -1756,6 +1792,15 @@ TEST_CASE("MessagePack nesting does not consume the call stack")
CHECK(json::from_msgpack(input, true, false).is_discarded());
}
SECTION("a well-formed deep value is read through the SAX interface")
{
std::vector<uint8_t> input(300000, 0x91);
input.push_back(0x01); // innermost value
SaxCountdown accept_all(600001);
CHECK(json::sax_parse(input, &accept_all, json::input_format_t::msgpack));
}
SECTION("a well-formed deep value is read into a value")
{
const std::size_t depth = 10000;
@@ -1803,6 +1848,31 @@ TEST_CASE("MessagePack input that cannot be read is discarded by every overload"
#endif
}
TEST_CASE("MessagePack SAX parsing stops at every event")
{
// Containers are opened and closed by the loop that reads them; a SAX
// handler that rejects any event - including the end of a nested
// container - must stop the parse right there.
const auto count_events = [](const std::vector<std::uint8_t>& input)
{
int events = 0;
while (true)
{
SaxCountdown scp(events);
if (json::sax_parse(input, &scp, json::input_format_t::msgpack))
{
return events;
}
++events;
REQUIRE(events < 1000);
}
};
// 20 events: every container kind closes inside another one
const json j = json::parse(R"({"a": [1, {"b": []}], "c": {"d": [[2]]}})");
CHECK(count_events(json::to_msgpack(j)) == 20);
}
TEST_CASE("single MessagePack roundtrip")
{
SECTION("sample.json")
@@ -2142,99 +2212,3 @@ TEST_CASE("MessagePack large strings and binaries (chunked reader)")
}
}
}
TEST_CASE("MessagePack arrays of scalars from contiguous and non-contiguous input")
{
// contiguous input reads the integers, floats, booleans, and nils an
// array begins with in one loop, other input value by value; both must
// give the same events, value, or error for the complete document and for
// every truncation of it, and stop at every event a SAX handler rejects
const std::vector<std::uint8_t> msgpack =
{
0x83, // map(3)
0xA1, 'a', // "a"
0xDC, 0x00, 0x16, // array 16 (22)
0x00, 0x7F, 0xCC, 0xFF, 0xCD, 0x01, 0x00, // 0, 127, 255, 256
0xCE, 0x00, 0x01, 0x00, 0x00, // 65536
0xCF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, // 18446744073709551615
0xFF, 0xE0, 0xD0, 0x80, 0xD1, 0xFF, 0x00, // -1, -32, -128, -256
0xD2, 0xFF, 0xFE, 0xFF, 0xFF, // -65537
0xD3, 0x80, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, // -9223372036854775808
0xC2, 0xC3, 0xC0, // false, true, nil
0xCA, 0x3F, 0xC0, 0x00, 0x00, // 1.5 (float 32)
0xCB, 0xBF, 0xB9, 0x99, 0x99, 0x99, 0x99, 0x99, 0x9A, // -0.1
0xA1, 'x', // "x" (ends the loop)
0x01, // 1
0xCB, 0x40, 0x04, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, // 2.5
0xD9, 0x01, 'y', // "y" (str 8)
0xC0, // nil
0xA1, 'b', // "b"
0x93, // fixarray(3)
0x92, 0xCB, 0x3F, 0xF4, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, // [1.25,
0xCB, 0xC0, 0x0E, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, // -3.75]
0x92, 0xCB, 0x3F, 0xF4, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, // [1.25,
0xCB, 0xC0, 0x0E, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, // -3.75]
0x90, // []
0xA1, 'c', // "c"
0x94, 0x80, 0x01, 0xC4, 0x01, 0x05, 0x91, 0x03 // [{}, 1, bin 8, [3]]
};
const json j = json::from_msgpack(msgpack);
CHECK(j["a"].size() == 22);
CHECK(j["a"][5] == 18446744073709551615u);
CHECK(j["a"][7] == -32);
CHECK(j["a"][11] == (std::numeric_limits<std::int64_t>::min)());
CHECK(j["a"][15] == 1.5);
CHECK(j["a"][16] == -0.1);
CHECK(j["a"][17] == "x");
CHECK(j["b"] == json::parse("[[1.25, -3.75], [1.25, -3.75], []]"));
CHECK(j["c"][2].is_binary());
for (std::size_t length = 0; length <= msgpack.size(); ++length)
{
CAPTURE(length)
const std::vector<std::uint8_t> contiguous(msgpack.begin(), msgpack.begin() + static_cast<std::ptrdiff_t>(length));
const std::list<std::uint8_t> bytewise(contiguous.begin(), contiguous.end());
CHECK(json::from_msgpack(contiguous, true, false).dump() == json::from_msgpack(bytewise.begin(), bytewise.end(), true, false).dump());
#if !defined(JSON_NOEXCEPTION)
std::string contiguous_error;
std::string bytewise_error;
try
{
json _;
_ = json::from_msgpack(contiguous);
}
catch (const json::parse_error& e)
{
contiguous_error = e.what();
}
try
{
json _;
_ = json::from_msgpack(bytewise.begin(), bytewise.end());
}
catch (const json::parse_error& e)
{
bytewise_error = e.what();
}
CHECK(contiguous_error == bytewise_error);
#endif
SaxEventLogger from_contiguous;
SaxEventLogger from_bytewise;
CHECK(json::sax_parse(contiguous, &from_contiguous, json::input_format_t::msgpack)
== json::sax_parse(bytewise.begin(), bytewise.end(), &from_bytewise, json::input_format_t::msgpack));
CHECK(from_contiguous.events == from_bytewise.events);
}
for (int events = 0; events <= 60; ++events)
{
CAPTURE(events)
const std::list<std::uint8_t> bytewise(msgpack.begin(), msgpack.end());
SaxCountdown from_contiguous(events);
SaxCountdown from_bytewise(events);
CHECK(json::sax_parse(msgpack, &from_contiguous, json::input_format_t::msgpack)
== json::sax_parse(bytewise.begin(), bytewise.end(), &from_bytewise, json::input_format_t::msgpack));
}
}
+52
View File
@@ -22,6 +22,14 @@
// scoped enum, so get<std::byte>() (needed below to get<std::vector<std::byte>>()
// from a plain JSON array, not just from an already-binary value) relies on
// enum serialization being enabled
// capture whether JSON_DELETE_DEPRECATED_FUNCTIONS was enabled on the command
// line *before* including json.hpp, since the library #undefs it once the header
// has been fully processed (see include/nlohmann/detail/macro_unscope.hpp); the
// tests of deprecated functions are skipped if these functions are deleted
#if defined(JSON_DELETE_DEPRECATED_FUNCTIONS) && (JSON_DELETE_DEPRECATED_FUNCTIONS == 1)
#define JSON_TEST_DEPRECATED_FUNCTIONS_DELETED
#endif
#if defined(JSON_DISABLE_ENUM_SERIALIZATION) && (JSON_DISABLE_ENUM_SERIALIZATION == 1)
#define SKIP_TESTS_FOR_ENUM_SERIALIZATION
#endif
@@ -802,6 +810,50 @@ TEST_CASE("issue #5338 - truncated CBOR tagged binary subtype is rejected")
}
}
TEST_CASE("issue #5676 - SAX parsing of CBOR tags")
{
const json expected = json::binary({1, 2, 3}, 42);
const auto cbor = json::to_cbor(expected);
nlohmann::detail::json_sax_acceptor<json> acceptor;
CHECK_FALSE(json::sax_parse(cbor, &acceptor, json::input_format_t::cbor));
CHECK_FALSE(json::sax_parse(cbor, &acceptor, json::input_format_t::cbor,
true, false, false, json::cbor_tag_handler_t::error));
CHECK(json::sax_parse(cbor, &acceptor, json::input_format_t::cbor,
true, false, false, json::cbor_tag_handler_t::ignore));
json parsed;
nlohmann::detail::json_sax_dom_parser<json, nlohmann::detail::string_input_adapter_type> sax(parsed);
CHECK(json::sax_parse(cbor, &sax, json::input_format_t::cbor,
true, false, false, json::cbor_tag_handler_t::store));
CHECK(parsed == expected);
json iterator_parsed;
nlohmann::detail::json_sax_dom_parser<json, nlohmann::detail::string_input_adapter_type> iterator_sax(iterator_parsed);
CHECK(json::sax_parse(cbor.begin(), cbor.end(), &iterator_sax, json::input_format_t::cbor,
true, false, false, json::cbor_tag_handler_t::store));
CHECK(iterator_parsed == expected);
#ifndef JSON_TEST_DEPRECATED_FUNCTIONS_DELETED
json span_parsed;
nlohmann::detail::json_sax_dom_parser<json, nlohmann::detail::string_input_adapter_type> span_sax(span_parsed);
CHECK(json::sax_parse(nlohmann::detail::span_input_adapter(cbor.data(), cbor.size()), &span_sax,
json::input_format_t::cbor, true, false, false, json::cbor_tag_handler_t::store));
CHECK(span_parsed == expected);
#endif
const std::string text = "null";
CHECK(json::sax_parse(text, &acceptor, json::input_format_t::json,
true, false, false, json::cbor_tag_handler_t::store));
CHECK(json::sax_parse(text.begin(), text.end(), &acceptor, json::input_format_t::json,
true, false, false, json::cbor_tag_handler_t::store));
#ifndef JSON_TEST_DEPRECATED_FUNCTIONS_DELETED
CHECK(json::sax_parse(nlohmann::detail::span_input_adapter(text.data(), text.size()), &acceptor,
json::input_format_t::json, true, false, false, json::cbor_tag_handler_t::store));
#endif
}
TEST_CASE("issue #5402 - update(merge_objects=true) overwrites a primitive with an object")
{
json t = {{"k", 1}};
-781
View File
@@ -1,781 +0,0 @@
// __ _____ _____ _____
// __| | __| | | | JSON for Modern C++ (supporting code)
// | | |__ | | | | | | version 3.12.0
// |_____|_____|_____|_|___| https://github.com/nlohmann/json
//
// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
// SPDX-License-Identifier: MIT
/////////////////////////////////////////////////////////////////////
// Tests that call basic_json::sax_parse have a file of their own: every
// sax_parse call instantiates the parser and binary reader that recover from
// errors (see #3989), and in unit-regression2.cpp, unit-regression3.cpp, and
// unit-msgpack.cpp this made the objects too large for the MinGW linker to
// relocate (see #5511).
/////////////////////////////////////////////////////////////////////
#include "doctest_compatibility.h"
// capture whether JSON_DELETE_DEPRECATED_FUNCTIONS was enabled on the command
// line *before* including json.hpp, since the library #undefs it once the header
// has been fully processed (see include/nlohmann/detail/macro_unscope.hpp); the
// tests of deprecated functions are skipped if these functions are deleted
#if defined(JSON_DELETE_DEPRECATED_FUNCTIONS) && (JSON_DELETE_DEPRECATED_FUNCTIONS == 1)
#define JSON_TEST_DEPRECATED_FUNCTIONS_DELETED
#endif
#include <nlohmann/json.hpp>
using json = nlohmann::json;
#include <cmath>
#include <cstddef>
#include <cstdint>
#include <initializer_list>
#include <map>
#include <string>
#include <utility>
#include <vector>
#include "sax_countdown.hpp"
using utils::SaxCountdown;
// a narrow number_float_t, so that a double read from binary input can
// overflow it
using float_json = nlohmann::basic_json<std::map, std::vector, std::string, bool, std::int64_t, std::uint64_t, float>;
DOCTEST_CLANG_SUPPRESS_WARNING_PUSH
DOCTEST_CLANG_SUPPRESS_WARNING("-Wexit-time-destructors")
namespace
{
/// builds a value from SAX events, asks the parser to recover from its first
/// 100 errors, and checks that the events are balanced (see #3989)
template<typename BasicJsonType>
class BasicRecoveringParser
{
public:
explicit BasicRecoveringParser(BasicJsonType& j)
: dom(j, false)
{}
bool null()
{
value();
return dom.null();
}
bool boolean(bool val)
{
value();
return dom.boolean(val);
}
bool number_integer(typename BasicJsonType::number_integer_t val)
{
value();
return dom.number_integer(val);
}
bool number_unsigned(typename BasicJsonType::number_unsigned_t val)
{
value();
return dom.number_unsigned(val);
}
bool number_float(typename BasicJsonType::number_float_t val, const std::string& s)
{
value();
return dom.number_float(val, s);
}
bool string(std::string& val)
{
value();
return dom.string(val);
}
bool binary(typename BasicJsonType::binary_t& val)
{
value();
return dom.binary(val);
}
bool start_object(std::size_t elements)
{
value();
stack.push_back('o');
return dom.start_object(elements);
}
bool key(std::string& val)
{
if (stack.empty() || stack.back() != 'o')
{
well_formed = false;
return false;
}
stack.back() = 'v';
return dom.key(val);
}
bool end_object()
{
if (stack.empty() || stack.back() != 'o')
{
well_formed = false;
return false;
}
stack.pop_back();
return dom.end_object();
}
bool start_array(std::size_t elements)
{
value();
stack.push_back('a');
return dom.start_array(elements);
}
bool end_array()
{
if (stack.empty() || stack.back() != 'a')
{
well_formed = false;
return false;
}
stack.pop_back();
return dom.end_array();
}
bool parse_error(std::size_t /*unused*/, const std::string& /*unused*/, const json::exception& ex)
{
messages.emplace_back(ex.what());
// a limit, so that a reader that does not stop fails the test
// instead of making it hang
return ++errors < 100;
}
/// whether the events were balanced and every key was followed by a value
bool balanced() const
{
return well_formed && stack.empty();
}
/// builds the value
nlohmann::detail::json_sax_dom_parser<BasicJsonType> dom;
std::size_t errors = 0;
std::vector<std::string> messages {}; // NOLINT(readability-redundant-member-init)
std::vector<char> stack {}; // NOLINT(readability-redundant-member-init)
bool well_formed = true;
private:
void value()
{
if (!stack.empty())
{
if (stack.back() == 'v')
{
stack.back() = 'o';
}
else if (stack.back() == 'o')
{
well_formed = false;
}
}
}
};
using RecoveringParser = BasicRecoveringParser<json>;
struct BinaryParseResult
{
json value;
std::size_t errors;
std::vector<std::string> messages;
bool ok;
bool balanced;
};
BinaryParseResult parse_binary_recovering(const std::vector<std::uint8_t>& input, const json::input_format_t format)
{
json j;
RecoveringParser sax(j);
const bool ok = json::sax_parse(input, &sax, format);
return {j, sax.errors, sax.messages, ok, sax.balanced()};
}
#if !defined(JSON_NOEXCEPTION)
/// the message of the exception that reading @a input into a JSON value
/// throws, or an empty string if reading succeeds
std::string binary_error_message(const std::vector<std::uint8_t>& input, const json::input_format_t format)
{
try
{
json _;
switch (format)
{
case json::input_format_t::cbor:
_ = json::from_cbor(input);
break;
case json::input_format_t::msgpack:
_ = json::from_msgpack(input);
break;
case json::input_format_t::ubjson:
_ = json::from_ubjson(input);
break;
case json::input_format_t::bjdata:
_ = json::from_bjdata(input);
break;
case json::input_format_t::bson:
_ = json::from_bson(input);
break;
case json::input_format_t::bon8:
_ = json::from_bon8(input);
break;
case json::input_format_t::json:
default:
break;
}
}
catch (const json::exception& e)
{
return e.what();
}
return "";
}
#endif
/// a BSON element: its type, its name, and its value
std::vector<std::uint8_t> bson_element(const std::uint8_t type, const std::string& name, const std::vector<std::uint8_t>& value)
{
std::vector<std::uint8_t> result = {type};
result.insert(result.end(), name.begin(), name.end());
result.push_back(0x00);
result.insert(result.end(), value.begin(), value.end());
return result;
}
/// a BSON document of the given elements; @a size_offset is added to the
/// size it declares
std::vector<std::uint8_t> bson_document(const std::vector<std::vector<std::uint8_t>>& elements, const int size_offset = 0)
{
std::vector<std::uint8_t> body;
for (const auto& element : elements)
{
body.insert(body.end(), element.begin(), element.end());
}
const auto size = static_cast<std::uint32_t>(static_cast<int>(body.size()) + 5 + size_offset);
std::vector<std::uint8_t> result = {static_cast<std::uint8_t>(size & 0xFFu), static_cast<std::uint8_t>((size >> 8u) & 0xFFu),
static_cast<std::uint8_t>((size >> 16u) & 0xFFu), static_cast<std::uint8_t>((size >> 24u) & 0xFFu)
};
result.insert(result.end(), body.begin(), body.end());
result.push_back(0x00);
return result;
}
/// a BSON int32 value
std::vector<std::uint8_t> bson_int32(const std::int32_t value)
{
const auto u = static_cast<std::uint32_t>(value);
return {static_cast<std::uint8_t>(u & 0xFFu), static_cast<std::uint8_t>((u >> 8u) & 0xFFu),
static_cast<std::uint8_t>((u >> 16u) & 0xFFu), static_cast<std::uint8_t>((u >> 24u) & 0xFFu)};
}
/// a BSON string value, whose length is @a length_offset off
std::vector<std::uint8_t> bson_string(const std::string& value, const std::int32_t length_offset = 0)
{
auto result = bson_int32(static_cast<std::int32_t>(value.size() + 1) + length_offset);
result.insert(result.end(), value.begin(), value.end());
result.push_back(0x00);
return result;
}
/// @a count bytes of value 0xAB
std::vector<std::uint8_t> bytes(const std::size_t count)
{
return std::vector<std::uint8_t>(count, 0xAB);
}
/// the bytes of @a parts, one after the other
std::vector<std::uint8_t> concatenated(std::initializer_list<std::vector<std::uint8_t>> parts)
{
std::vector<std::uint8_t> result;
for (const auto& part : parts)
{
result.insert(result.end(), part.begin(), part.end());
}
return result;
}
/// U+FFFD REPLACEMENT CHARACTER
std::string replacement_character()
{
return "\xEF\xBF\xBD";
}
} // namespace
TEST_CASE("regression test - #3989 SAX parse_error() returning true")
{
SECTION("binary formats complete what was read before the input ends")
{
const json j = {{"a", {1, -2, {{"b", "c"}}, json::array()}}, {"d", {{"e", nullptr}, {"f", true}}}, {"g", 1.5}, {"h", json::binary({1, 2, 3})}};
const std::vector<std::pair<json::input_format_t, std::vector<std::uint8_t>>> encodings =
{
{json::input_format_t::cbor, json::to_cbor(j)},
{json::input_format_t::msgpack, json::to_msgpack(j)},
{json::input_format_t::ubjson, json::to_ubjson(j)},
{json::input_format_t::ubjson, json::to_ubjson(j, true, true)},
{json::input_format_t::bjdata, json::to_bjdata(j)},
{json::input_format_t::bjdata, json::to_bjdata(j, true, true)},
{json::input_format_t::bson, json::to_bson(j)},
{json::input_format_t::bon8, json::to_bon8(j)},
};
for (const auto& encoding : encodings)
{
const auto format = encoding.first;
const auto& bytes = encoding.second;
CAPTURE(format)
// every prefix is truncated input
for (std::size_t length = 0; length < bytes.size(); ++length)
{
CAPTURE(length)
const auto result = parse_binary_recovering(std::vector<std::uint8_t>(bytes.begin(), bytes.begin() + static_cast<std::ptrdiff_t>(length)), format);
CHECK(!result.ok);
CHECK(result.errors == 1);
CHECK(result.balanced);
}
// the complete input is read as usual (binary values do not
// round-trip through every format, so compare with a plain parse)
json expected;
nlohmann::detail::json_sax_dom_parser<json> dom(expected);
CHECK(json::sax_parse(bytes, &dom, format));
const auto complete = parse_binary_recovering(bytes, format);
CHECK(complete.ok);
CHECK(complete.errors == 0);
CHECK(complete.value == expected);
// a byte after the value
auto trailing_bytes = bytes;
trailing_bytes.push_back(0x01);
const auto trailing = parse_binary_recovering(trailing_bytes, format);
CHECK(!trailing.ok);
CHECK(trailing.errors == 1);
CHECK(trailing.value == expected);
}
}
SECTION("containers without an end")
{
// these made the readers loop, or read on, after the error
const auto cbor_array = parse_binary_recovering({0x9F}, json::input_format_t::cbor);
CHECK(cbor_array.errors == 1);
CHECK(cbor_array.value == json::array());
const auto cbor_map = parse_binary_recovering({0xBF, 0x61, 'a'}, json::input_format_t::cbor);
CHECK(cbor_map.errors == 1);
CHECK(cbor_map.value == json({{"a", nullptr}}));
const auto msgpack_array = parse_binary_recovering({0xDD, 0xFF, 0xFF, 0xFF, 0xFF}, json::input_format_t::msgpack);
CHECK(msgpack_array.errors == 1);
CHECK(msgpack_array.value == json::array());
const auto msgpack_map = parse_binary_recovering({0x81, 0xA1, 'a', 0x92, 0x01}, json::input_format_t::msgpack);
CHECK(msgpack_map.errors == 1);
CHECK(msgpack_map.value == json({{"a", {1}}}));
}
SECTION("BJData ndarray")
{
// a 2x3 int8 array with two of its six elements; the annotated array
// format opens an object and two arrays of its own
const auto result = parse_binary_recovering({'[', '$', 'i', '#', '[', '$', 'i', '#', 'i', 2, 2, 3, 1, 2}, json::input_format_t::bjdata);
CHECK(result.errors == 1);
CHECK(result.balanced);
CHECK(result.value == json({{"_ArrayType_", "int8"}, {"_ArraySize_", {2, 3}}, {"_ArrayData_", {1, 2}}}));
}
SECTION("binary formats repair items whose end is known")
{
struct Repair
{
json::input_format_t format;
std::vector<std::uint8_t> input;
json expected;
std::size_t errors;
};
const std::vector<Repair> repairs =
{
// CBOR: tags are ignored (here tag 1 and the self-describe tag 55799)
{json::input_format_t::cbor, {0x82, 0xC1, 0x05, 0xD9, 0xD9, 0xF7, 0x06}, {5, 6}, 2},
// CBOR: undefined and other simple values become null
{json::input_format_t::cbor, {0x84, 0xF7, 0xE0, 0xF8, 0x20, 0x01}, {nullptr, nullptr, nullptr, 1}, 3},
// CBOR: members whose key is not a string are skipped, whatever their key and value
{json::input_format_t::cbor, {0xA4, 0x01, 0x02, 0x82, 0x01, 0x02, 0xA1, 0x61, 'x', 0x9F, 0xFF, 0xC1, 0x01, 0x5F, 0x41, 0x00, 0xFF, 0x61, 'a', 0x03}, {{"a", 3}}, 3},
{json::input_format_t::cbor, {0xBF, 0xF5, 0xBF, 0x61, 'x', 0x7F, 0x61, 'y', 0xFF, 0xFF, 0x61, 'a', 0x03, 0xFF}, {{"a", 3}}, 1},
// MessagePack: members whose key is not a string are skipped
{json::input_format_t::msgpack, {0x84, 0x01, 0x02, 0x81, 0xA1, 'x', 0x01, 0x92, 0x01, 0x02, 0xD4, 0x01, 0x02, 0xC0, 0xA1, 'a', 0x04}, {{"a", 4}}, 3},
// UBJSON: a char that is not ASCII becomes U+FFFD
{json::input_format_t::ubjson, {'[', 'C', 0x80, 'C', 'A', ']'}, {replacement_character(), "A"}, 1},
// UBJSON: the longest beginning of a high-precision number is kept
{json::input_format_t::ubjson, {'[', 'H', 'i', 5, '1', '2', 'a', 'b', 'c', 'H', 'i', 2, '1', '.', 'H', 'i', 3, 'a', 'b', 'c', 'H', 'i', 3, '4', '.', '5', ']'}, {12, 1, nullptr, 4.5}, 3},
// BJData, too
{json::input_format_t::bjdata, {'[', 'C', 0xFF, 'H', 'i', 2, '-', '1', 'H', 'i', 2, '-', 'x', ']'}, {replacement_character(), -1, nullptr}, 2},
// a NUL ends a high-precision number, as it ends JSON text
{json::input_format_t::ubjson, {'[', 'H', 'i', 4, '1', '2', 0, '9', 'H', 'i', 2, 0, '1', 'i', 3, ']'}, {12, nullptr, 3}, 2},
// BON8: members whose key is not a string are skipped
{json::input_format_t::bon8, {0x89, 0x91, 0x92, 0xC9, 0x40, 0x82, 0x91, 0x92, 0x61, 0x93}, {{"a", 3}}, 2},
{json::input_format_t::bon8, {0x8B, 0x91, 0x85, 0x91, 0xFE, 0xFA, 0x8B, 'x', 0x91, 0xFE, 0x61, 0x93, 0xFE}, {{"a", 3}}, 2},
// BSON: elements of types the library does not read become null
{
json::input_format_t::bson, bson_document(
{
bson_element(0x07, "_id", bytes(12)), // ObjectId
bson_element(0x09, "date", bytes(8)), // UTC datetime
bson_element(0x13, "decimal", bytes(16)), // 128-bit decimal
bson_element(0x0B, "regex", {'a', '+', 0, 'i', 0}), // regular expression
bson_element(0x0D, "code", bson_string("f()")), // JavaScript code
bson_element(0x0E, "symbol", bson_string("s")), // symbol
bson_element(0x0C, "pointer", concatenated({bson_string("c"), bytes(12)})), // DBPointer
bson_element(0x0F, "scope", concatenated({bson_int32(15), bson_string("g"), bson_document({})})), // code with scope
bson_element(0x06, "undefined", {}), // undefined
bson_element(0xFF, "min", {}), // min key
bson_element(0x7F, "max", {}), // max key
bson_element(0x10, "z", bson_int32(7)),
}),
{{"_id", nullptr}, {"date", nullptr}, {"decimal", nullptr}, {"regex", nullptr}, {"code", nullptr}, {"symbol", nullptr}, {"pointer", nullptr}, {"scope", nullptr}, {"undefined", nullptr}, {"min", nullptr}, {"max", nullptr}, {"z", 7}},
11
},
// BSON: an element of an unknown type becomes null, and the rest of its document is skipped
{
json::input_format_t::bson, bson_document(
{
bson_element(0x03, "inner", bson_document({bson_element(0x10, "a", bson_int32(1)), bson_element(0x42, "x", bytes(3)), bson_element(0x10, "b", bson_int32(2))})),
bson_element(0x04, "array", bson_document({bson_element(0x10, "0", bson_int32(1)), bson_element(0x42, "1", bytes(3))})),
bson_element(0x10, "after", bson_int32(3)),
}),
{{"inner", {{"a", 1}, {"x", nullptr}}}, {"array", {1, nullptr}}, {"after", 3}},
2
},
// BSON: so does a string or byte array whose length cannot be right
{
json::input_format_t::bson, bson_document(
{
bson_element(0x03, "inner", bson_document({bson_element(0x02, "s", bson_string("abc", -10)), bson_element(0x10, "b", bson_int32(2))})),
bson_element(0x03, "bin", bson_document({bson_element(0x05, "b", concatenated({bson_int32(-1), bytes(1)})), bson_element(0x10, "b", bson_int32(2))})),
bson_element(0x10, "after", bson_int32(3)),
}),
{{"inner", {{"s", nullptr}}}, {"bin", {{"b", nullptr}}}, {"after", 3}},
2
},
// BSON: a string without its terminator, and a document whose size does not match, are kept
{
json::input_format_t::bson, bson_document(
{
bson_element(0x02, "s", {2, 0, 0, 0, 'a', 'X'}),
bson_element(0x03, "inner", bson_document({bson_element(0x10, "a", bson_int32(1))}, 1)),
}),
{{"s", "a"}, {"inner", {{"a", 1}}}},
2
},
// BSON: so is a document whose size is too small for its content,
// which is read up to its terminator
{
json::input_format_t::bson, bson_document(
{
bson_element(0x03, "inner", bson_document({bson_element(0x10, "a", bson_int32(1)), bson_element(0x10, "b", bson_int32(2))}, -7)),
bson_element(0x10, "after", bson_int32(3)),
}),
{{"inner", {{"a", 1}, {"b", 2}}}, {"after", 3}},
1
},
};
for (const auto& repair : repairs)
{
CAPTURE(repair.format)
CAPTURE(repair.input)
const auto result = parse_binary_recovering(repair.input, repair.format);
CHECK(!result.ok);
CHECK(result.balanced);
CHECK(result.errors == repair.errors);
CHECK(result.value == repair.expected);
REQUIRE(!result.messages.empty());
#if !defined(JSON_NOEXCEPTION)
// the first error is the one reported without recovering; under
// JSON_NOEXCEPTION, reading without recovering aborts instead of
// throwing, so there is no message to compare with
CHECK(result.messages.front() == binary_error_message(repair.input, repair.format));
#endif
}
}
SECTION("binary formats repair numbers that are out of range")
{
// CBOR: a double too large for a float number_float_t
float_json cbor;
BasicRecoveringParser<float_json> sax(cbor);
const std::vector<std::uint8_t> cbor_input = {0x82, 0xFB, 0x7E, 0x37, 0xE4, 0x3C, 0x88, 0x00, 0x75, 0x9C, 0x01}; // [1e300, 1]
CHECK(!float_json::sax_parse(cbor_input, &sax, float_json::input_format_t::cbor));
CHECK(sax.errors == 1);
CHECK(sax.messages.front() == "[json.exception.out_of_range.406] syntax error while parsing CBOR value: number overflow");
REQUIRE(cbor.size() == 2);
CHECK(std::isinf(cbor[0].get<float>()));
CHECK(cbor[1] == 1);
// UBJSON: a high-precision number too large for number_float_t
const auto ubjson = parse_binary_recovering({'H', 'i', 5, '1', 'e', '9', '9', '9'}, json::input_format_t::ubjson);
CHECK(ubjson.errors == 1);
CHECK(ubjson.value.is_number_float());
CHECK(std::isinf(ubjson.value.get<double>()));
}
SECTION("binary formats stop where the end of an item is not known")
{
// a byte that begins no item
const auto cbor = parse_binary_recovering({0x82, 0x01, 0x1C, 0x02}, json::input_format_t::cbor);
CHECK(cbor.errors == 1);
CHECK(cbor.value == json({1}));
// a key that is no item: the unused MessagePack byte, a CBOR break
// in a map of known size, and the end of a BON8 container
const auto msgpack = parse_binary_recovering({0x82, 0xA1, 'a', 0x01, 0xC1, 0x02}, json::input_format_t::msgpack);
CHECK(msgpack.errors == 1);
CHECK(msgpack.value == json({{"a", 1}}));
const auto cbor_break = parse_binary_recovering({0xA2, 0x61, 'a', 0x01, 0xFF, 0x02}, json::input_format_t::cbor);
CHECK(cbor_break.errors == 1);
CHECK(cbor_break.value == json({{"a", 1}}));
const auto bon8 = parse_binary_recovering({0x88, 0x61, 0x91, 0xFE}, json::input_format_t::bon8);
CHECK(bon8.errors == 1);
CHECK(bon8.value == json({{"a", 1}}));
// an indefinite-length string inside an indefinite-length string
const auto nested = parse_binary_recovering({0x82, 0x01, 0x7F, 0x7F, 0x61, 'a', 0xFF, 0xFF}, json::input_format_t::cbor);
CHECK(nested.errors == 1);
CHECK(nested.value == json({1}));
// a BJData ndarray whose element type is not valid: the type is
// checked before the object that would hold the ndarray is begun
const auto ndarray = parse_binary_recovering({'[', '$', 0x01, '#', '[', '$', 'i', '#', 'i', 2, 2, 3}, json::input_format_t::bjdata);
CHECK(ndarray.errors == 1);
CHECK(ndarray.balanced);
CHECK(ndarray.value == nullptr);
// a skipped member that the input ends in
const auto truncated = parse_binary_recovering({0xA2, 0x01, 0x82, 0x01}, json::input_format_t::cbor);
CHECK(truncated.errors == 2);
CHECK(truncated.balanced);
CHECK(truncated.value == json::object());
// a BSON element of an unknown type in a document whose size cannot be
// right: the size is reported when it is read, and the element ends
// the document, as its end is not known
const auto bson = parse_binary_recovering(bson_document({bson_element(0x10, "a", bson_int32(1)), bson_element(0x42, "x", bytes(3))}, -10), json::input_format_t::bson);
CHECK(bson.errors == 2);
CHECK(bson.value == json({{"a", 1}, {"x", nullptr}}));
}
SECTION("changed bytes in binary input")
{
const json j = {{"a", {1, -2, {{"b", "c"}}, json::array()}}, {"d", {{"e", nullptr}, {"f", true}}}, {"g", 1.5}, {"h", json::binary({1, 2, 3})}, {"i", "\xC3\xA4"}};
const std::vector<std::pair<json::input_format_t, std::vector<std::uint8_t>>> encodings =
{
{json::input_format_t::cbor, json::to_cbor(j)},
{json::input_format_t::msgpack, json::to_msgpack(j)},
{json::input_format_t::ubjson, json::to_ubjson(j)},
{json::input_format_t::ubjson, json::to_ubjson(j, true, true)},
{json::input_format_t::bjdata, json::to_bjdata(j)},
{json::input_format_t::bjdata, json::to_bjdata(j, true, true)},
{json::input_format_t::bson, json::to_bson(j)},
{json::input_format_t::bon8, json::to_bon8(j)},
};
const std::vector<std::uint8_t> replacements = {0x00, 0x01, 0x7F, 0x80, 0xC1, 0xD9, 0xE0, 0xF7, 0xFE, 0xFF};
for (const auto& encoding : encodings)
{
const auto format = encoding.first;
const auto& original = encoding.second;
CAPTURE(format)
std::vector<std::vector<std::uint8_t>> inputs;
for (std::size_t position = 0; position < original.size(); ++position)
{
for (const auto replacement : replacements)
{
auto changed = original;
changed[position] = replacement;
inputs.push_back(changed);
}
auto removed = original;
removed.erase(removed.begin() + static_cast<std::ptrdiff_t>(position));
inputs.push_back(removed);
}
for (const auto& input : inputs)
{
CAPTURE(input)
const auto result = parse_binary_recovering(input, format);
CHECK(result.balanced);
CHECK(result.errors <= input.size() + 1);
#if !defined(JSON_NOEXCEPTION)
// an error is reported exactly if reading into a JSON value
// fails, and the first one is the same (under JSON_NOEXCEPTION,
// that reading aborts instead of throwing)
const auto message = binary_error_message(input, format);
CHECK(result.ok == message.empty());
if (!result.ok && result.errors < 100)
{
CHECK(result.messages.front() == message);
}
#endif
}
}
}
SECTION("JSON text")
{
// the parser stopped, but reported success
json j;
RecoveringParser sax(j);
CHECK(!json::sax_parse("[1,2,3,]", &sax));
CHECK(sax.errors == 1);
CHECK(j == json({1, 2, 3}));
}
SECTION("the SAX parsers of the library stop")
{
json _;
CHECK(json::from_cbor(std::vector<std::uint8_t> {0x9F}, true, false).is_discarded());
CHECK_THROWS_WITH_AS(_ = json::from_cbor(std::vector<std::uint8_t> {0x9F}), "[json.exception.parse_error.110] parse error at byte 2: syntax error while parsing CBOR value: unexpected end of input", json::parse_error&);
CHECK(json::parse("[1,2,3,]", nullptr, false).is_discarded());
CHECK(!json::accept("[1,2,3,]"));
}
}
TEST_CASE("issue #5676 - SAX parsing of CBOR tags")
{
const json expected = json::binary({1, 2, 3}, 42);
const auto cbor = json::to_cbor(expected);
nlohmann::detail::json_sax_acceptor<json> acceptor;
CHECK_FALSE(json::sax_parse(cbor, &acceptor, json::input_format_t::cbor));
CHECK_FALSE(json::sax_parse(cbor, &acceptor, json::input_format_t::cbor,
true, false, false, json::cbor_tag_handler_t::error));
CHECK(json::sax_parse(cbor, &acceptor, json::input_format_t::cbor,
true, false, false, json::cbor_tag_handler_t::ignore));
json parsed;
nlohmann::detail::json_sax_dom_parser<json, nlohmann::detail::string_input_adapter_type> sax(parsed);
CHECK(json::sax_parse(cbor, &sax, json::input_format_t::cbor,
true, false, false, json::cbor_tag_handler_t::store));
CHECK(parsed == expected);
json iterator_parsed;
nlohmann::detail::json_sax_dom_parser<json, nlohmann::detail::string_input_adapter_type> iterator_sax(iterator_parsed);
CHECK(json::sax_parse(cbor.begin(), cbor.end(), &iterator_sax, json::input_format_t::cbor,
true, false, false, json::cbor_tag_handler_t::store));
CHECK(iterator_parsed == expected);
#ifndef JSON_TEST_DEPRECATED_FUNCTIONS_DELETED
json span_parsed;
nlohmann::detail::json_sax_dom_parser<json, nlohmann::detail::string_input_adapter_type> span_sax(span_parsed);
CHECK(json::sax_parse(nlohmann::detail::span_input_adapter(cbor.data(), cbor.size()), &span_sax,
json::input_format_t::cbor, true, false, false, json::cbor_tag_handler_t::store));
CHECK(span_parsed == expected);
#endif
const std::string text = "null";
CHECK(json::sax_parse(text, &acceptor, json::input_format_t::json,
true, false, false, json::cbor_tag_handler_t::store));
CHECK(json::sax_parse(text.begin(), text.end(), &acceptor, json::input_format_t::json,
true, false, false, json::cbor_tag_handler_t::store));
#ifndef JSON_TEST_DEPRECATED_FUNCTIONS_DELETED
CHECK(json::sax_parse(nlohmann::detail::span_input_adapter(text.data(), text.size()), &acceptor,
json::input_format_t::json, true, false, false, json::cbor_tag_handler_t::store));
#endif
}
TEST_CASE("MessagePack SAX aborts")
{
SECTION("start_array(len)")
{
std::vector<uint8_t> const v = {0x93, 0x01, 0x02, 0x03};
SaxCountdown scp(0);
CHECK(!json::sax_parse(v, &scp, json::input_format_t::msgpack));
}
SECTION("start_object(len)")
{
std::vector<uint8_t> const v = {0x81, 0xa3, 0x66, 0x6F, 0x6F, 0xc2};
SaxCountdown scp(0);
CHECK(!json::sax_parse(v, &scp, json::input_format_t::msgpack));
}
SECTION("key()")
{
std::vector<uint8_t> const v = {0x81, 0xa3, 0x66, 0x6F, 0x6F, 0xc2};
SaxCountdown scp(1);
CHECK(!json::sax_parse(v, &scp, json::input_format_t::msgpack));
}
}
TEST_CASE("issue #5405 - a user-defined SAX consumer is unaffected by the internal DOM reserve optimization")
{
// the reserve() call is local to json_sax_dom_parser / json_sax_dom_callback_parser;
// a custom SAX consumer that does not touch a DOM array sees identical events
json j = json::array();
for (int i = 0; i < 100; ++i)
{
j.push_back(i);
}
const auto packed = json::to_msgpack(j);
SaxCountdown scp(1000000); // large enough to never trigger an abort
CHECK(json::sax_parse(packed, &scp, json::input_format_t::msgpack));
}
TEST_CASE("MessagePack nesting does not consume the call stack - SAX interface")
{
// see the test case of the same name in unit-msgpack.cpp (#5104)
std::vector<uint8_t> input(300000, 0x91);
input.push_back(0x01); // innermost value
SaxCountdown accept_all(600001);
CHECK(json::sax_parse(input, &accept_all, json::input_format_t::msgpack));
}
TEST_CASE("MessagePack SAX parsing stops at every event")
{
// Containers are opened and closed by the loop that reads them; a SAX
// handler that rejects any event - including the end of a nested
// container - must stop the parse right there.
const auto count_events = [](const std::vector<std::uint8_t>& input)
{
int events = 0;
while (true)
{
SaxCountdown scp(events);
if (json::sax_parse(input, &scp, json::input_format_t::msgpack))
{
return events;
}
++events;
REQUIRE(events < 1000);
}
};
// 20 events: every container kind closes inside another one
const json j = json::parse(R"({"a": [1, {"b": []}], "c": {"d": [[2]]}})");
CHECK(count_events(json::to_msgpack(j)) == 20);
}
DOCTEST_CLANG_SUPPRESS_WARNING_POP
-46
View File
@@ -23,7 +23,6 @@ using nlohmann::json;
#include <cstdint>
#include <sstream>
#include <iomanip>
#include <limits>
#include <string>
#include <vector>
@@ -249,51 +248,6 @@ TEST_CASE_TEMPLATE("serialization for extreme integer values", T, int32_t, uint3
}
}
TEST_CASE("integers of every digit count are written like std::to_string")
{
// dump_integer() writes up to 16 digits in two groups of eight and 17 to
// 20 digits as one to four digits and two groups of eight
std::vector<std::uint64_t> values;
std::uint64_t power = 1;
for (int digits = 1; digits <= 20; ++digits)
{
values.push_back(power); // 1 followed by zeros
values.push_back(power + 1);
values.push_back((power * 2) - 1);
if (digits < 20)
{
values.push_back((power * 10) - 1); // all 9s
values.push_back((power * 10) - power); // 9 followed by zeros
power *= 10;
}
}
values.push_back((std::numeric_limits<std::uint64_t>::max)());
values.push_back(static_cast<std::uint64_t>((std::numeric_limits<std::int64_t>::max)()));
values.push_back(static_cast<std::uint64_t>((std::numeric_limits<std::int64_t>::max)()) + 1);
std::uint64_t state = 0x9E3779B97F4A7C15u;
for (int i = 0; i < 20000; ++i)
{
state = (state * 6364136223846793005u) + 1442695040888963407u;
values.push_back(state >> (state % 61)); // every bit length
}
for (const auto v : values)
{
CAPTURE(v)
CHECK(json(v).dump() == std::to_string(v));
if (v <= static_cast<std::uint64_t>((std::numeric_limits<std::int64_t>::max)()))
{
const auto s = static_cast<std::int64_t>(v);
CHECK(json(s).dump() == std::to_string(s));
CHECK(json(-s).dump() == std::to_string(-s));
}
}
CHECK(json((std::numeric_limits<std::int64_t>::min)()).dump() == "-9223372036854775808");
// in a container, so the digits are followed by other output
CHECK(json::array({std::uint64_t{18446744073709551615u}, std::int64_t{-1234567890123456789}, 1}).dump() == "[18446744073709551615,-1234567890123456789,1]");
}
TEST_CASE("dump with binary values")
{
auto binary = json::binary({1, 2, 3, 4});
Loaded 100 of 110 files, more files were not shown because too many files have changed in this diff. Show more