Compare commits

..
Author SHA1 Message Date
Niels Lohmann f7823b5afe Merge branch 'develop' into techdebt/5717-tooling-cleanup
Conflicts:
- Makefile: kept develop's json_literals.hpp copy rule next to the new natvis/macro_builder_check targets; release now copies and signs json_literals.hpp (develop) and hashes every release_files artifact (this branch)

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-09-30 20:35:10 +02:00
Niels Lohmann 7aeb98414e Extend tools/macro_builder to also generate DOUBLE_PASTE and TYPE_BODY
tools/macro_builder only emitted NLOHMANN_JSON_EXPAND..PASTE64, so two
other tables that scale with the same max_args stayed hand-maintained
with nothing checking them: NLOHMANN_JSON_DOUBLE_PASTE (added by hand
in #4563 for the *_WITH_NAMES macros) and the 64-slot dispatch table of
NLOHMANN_JSON_TYPE_BODY (the #4041 zero-member/one-or-more-member
switch). Both tables pass one macro name per slot to the same
NLOHMANN_JSON_GET_MACRO dispatch as PASTE, so they can drift out of
sync with max_args exactly the way _dp did in the ABI tag list fixed
by #5544.

Extend main.cpp with build_double_paste_code() (same recursive-doubling
shape as build_paste_code(), but DOUBLE_PASTE consumes two arguments
per member, so an even slot index falls back to the next lower odd
DOUBLE_PASTE<N>) and build_type_body_table() (max_args - 1 MEMBERS
slots and one trailing EMPTY slot, 8 per line, matching how it is
written by hand today). Add a "type_body" argument that selects the
TYPE_BODY block, since it lives at a separate location in
macro_scope.hpp from the EXPAND..DOUBLE_PASTE63 block; plain invocation
is unchanged apart from covering the extended range. No longer emit
the tool's old trailing blank line, so its output is directly diffable
without post-processing.

Verified with c++ -std=c++11: running the tool (with and without
"type_body") and piping the raw output through the pinned astyle
reproduces both blocks of the current macro_scope.hpp byte for byte.
tests/src/unit-udt_macro.cpp (all NLOHMANN_DEFINE_TYPE_*/_WITH_NAMES/
zero-member variants) passes unchanged under -std=c++11 and -std=c++17
with -fsanitize=address,undefined.

Add a "macro_builder_check" Makefile target that builds main.cpp,
regenerates both blocks into a scratch directory inside the repository
(astyle's --project lookup needs the target files under the same tree
as .astylerc, unlike an external /tmp directory), and diffs them
against the corresponding ranges of macro_scope.hpp; wire it into
check-amalgamation next to the natvis check. Wire the same regeneration
into check_amalgamation.yml, splicing the (still unindented) generated
blocks back into the PR's own macro_scope.hpp before the existing
astyle/amalgamation step runs, so that step's own tree-wide astyle
pass both indents them and folds any drift into the amalgamation
patch/diff the workflow already produces.

Unlike amalgamate.py and generate_natvis.py, this step builds
tools/macro_builder/main.cpp from the pull request's own checkout
($MAIN_DIR) rather than a separate checkout of tools/ at develop: this
tool has no independent source of truth to regenerate against (its
README documents that it must reproduce macro_scope.hpp byte for
byte), so a develop-pinned copy would only reproduce the
generate_natvis.py trap fixed in a previous commit on this branch,
where a PR that teaches the tool to cover more of the file fails its
own CI until that PR merges and updates the develop copy.

Add tools/macro_builder/README.md documentation for both new tables
and the two-invocation usage, and a short pointer comment above
NLOHMANN_JSON_TYPE_BODY (the EXPAND pointer already covered the first
block; extended its wording to include DOUBLE_PASTE63).

Closes #5717 item 5 in full, completing what the documentation-only
"Document tools/macro_builder..." commit already on this branch left
open (that commit's README/pointer-comment half stands; it also covers
item 7).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-09-30 18:35:18 +02:00
Niels Lohmann a436ba2775 Fix check_amalgamation.yml: pass --version to generate_natvis.py
bff45f111 (#5717 item 3) made --version optional in
tools/generate_natvis/generate_natvis.py and wired the workflow's new
"Regenerate nlohmann_json.natvis" step to call it without --version,
relying on the script deriving the version from abi_macros.hpp itself.

But NATVIS_TOOL_DIR is checked out from develop, the same way TOOL_DIR
already is for amalgamate.py, precisely so an in-flight PR's tooling
changes cannot mark themselves clean. Until this PR (or an equivalent)
merges to develop, that checkout is the old generate_natvis.py, whose
--version argument is still required=True. The new step's invocation
of "generate_natvis.py $MAIN_DIR" (no --version) then fails argparse
on this PR's own CI run with "the following arguments are required:
--version", before the check ever gets to compare output.

Extract the version from $MAIN_DIR's own abi_macros.hpp in the
workflow and always pass it as --version. That satisfies the old
script's required argument and is accepted as an explicit override by
the new one, so the step behaves the same whether NATVIS_TOOL_DIR holds
the pre- or post-merge tool, and stays correct for later PRs that bump
the version.

Verified by running the workflow step's shell logic locally against
both the pre-#5717 generate_natvis.py (checked out at 633de8e44) and
the new one: both produce the identical nlohmann_json.natvis as the
committed file.

Review fix for #5717 item 3 (PR #5735).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-09-30 18:19:58 +02:00
Niels Lohmann 33f28ee972 Strip the leading "./" find(1) prefix from release hashes.txt entries
bc6e7db72 (#5717 item 1) switched the release target's shasum line from
naming files by hand to $$(find . -type f -not -name '*.asc' | sort),
so a newly shipped header is hashed automatically. Run from inside
release_files, that find prints paths as "./json.hpp" instead of
"json.hpp", so hashes.txt lists "./json.hpp" etc. instead of the plain
filenames it always used. shasum -c still verifies "./json.hpp" fine,
but it is a needless cosmetic regression for anyone reading the file
or matching it against release notes.

Strip the "./" prefix with sed before sorting, keeping the filenames
exactly as before while still hashing every artifact automatically.

Review fix for #5717 item 1 (PR #5735).

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-09-30 18:19:47 +02:00
Niels Lohmann bff45f1111 Derive generate_natvis.py's ABI tag list and version from abi_macros.hpp
generate_natvis.py hard-coded abi_tags = ['_diag', '_ldvcmp', '_dp',
'_bics', '_psp', '_snul'] and required --version on the command line.
The source of truth is include/nlohmann/detail/abi_macros.hpp: the
NLOHMANN_JSON_ABI_TAG_* defines, the argument order of
NLOHMANN_JSON_ABI_TAGS_CONCAT, and NLOHMANN_JSON_VERSION_MAJOR/MINOR/
PATCH. Nothing checked that the copies stayed in sync, and they have
drifted apart before: _dp was added in #4517 but missed here until
#5544, and 3.11.3 shipped json_abi_v3_11_2 namespaces (#4340).

Parse the tag list (in NLOHMANN_JSON_ABI_TAGS_CONCAT order) and the
version from abi_macros.hpp instead of hard-coding them. Make
--version optional (falling back to the parsed version) and default
the output directory to the repository root the script lives in.

Add a "natvis" Makefile target that runs the script, and extend
check-amalgamation to regenerate nlohmann_json.natvis and fail on a
diff, the same way it already does for the amalgamated headers and
BUILD.bazel. Wire the same regeneration into check_amalgamation.yml,
using the tool copy checked out from develop (as the workflow already
does for amalgamate.py) and installing jinja2 from
tools/generate_natvis/requirements.txt. Update the tool's README to
say it must be re-run after adding an ABI tag or bumping the version.

Verified: a run against develop produces no diff (with either the
default or an explicit --version 3.12.0); adding a dummy
NLOHMANN_JSON_ABI_TAG_* without a matching #define makes the script
fail loudly instead of silently omitting the tag; xmllint --noout
passes on the regenerated file; and running the script from a
directory other than the one being checked (simulating the workflow's
separate tool checkout) against this repository root also produces no
diff.

Overlaps #5600, which added _ekmo to the same hand-written abi_tags
line and regenerated the file.

#5717 item 3

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-09-30 18:10:56 +02:00
Niels Lohmann d7af0f7d3b Fix stale install/usage instructions in the vendored amalgamate README
tools/amalgamate/README.md is the unmodified upstream text and no
longer matches how the tool is used here:

- It named a Bitbucket origin that no longer exists; CHANGES.md
  already tracks the GitHub mirror commit this copy is based on.
- It asked for Python 2.7, but CI and the Makefile run the script
  with python3.
- It told readers to run ./test.sh (not vendored) and install to
  /usr/local/bin; in this repository the tool runs through
  `make amalgamate`.
- Its usage synopsis showed `-v` taking no argument, but the script's
  own argparser requires `choices=["yes", "no"]`, so that form fails
  with "argument -v/--verbose: expected one argument". The Makefile
  calls it as `--verbose=yes`.
- It pointed at test/source.c.json and test/include.h.json, which are
  not vendored; the configs actually used are config_json.json and
  config_json_fwd.json.

Rewrote only the Installing and Using sections to match; left the
"Here be dragons" caveats and the rest of the vendored code untouched
to avoid diverging further from upstream.

Overlaps #5615, which edits amalgamate.py, this README and CHANGES.md.

#5717 item 6

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-09-30 18:01:07 +02:00
Niels Lohmann 79c8289285 Remove stale Travis comment above check-amalgamation
check-amalgamation carried "Note: this target is called by Travis",
left over from before the project switched off Travis CI. The prior
Makefile cleanup commit removed the other stale Travis-era leftovers
(the doctest help entry, pretty_format, and the benchmarks/ path in
clean) but missed this comment.

#5717 item 4

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-09-30 18:00:21 +02:00
Niels Lohmann b730630eba Remove the broken fuzz_testing* Makefile targets
fuzz_testing and fuzz_testing_{bon8,bson,cbor,msgpack,ubjson} seeded
fuzz-testing/testcases from tests/data, which was removed in dbf1a1f41
(2020) when the test data moved to the external json_test_data repo.
The find command found nothing, but the pipeline's exit status was
that of xargs, so the recipe still reported success with an empty
corpus, and the printed afl-fuzz command would refuse to start.

The recipes were also six near-identical copies with unquoted -name
patterns, used the legacy CXX=afl-clang++, and fuzzing-start/stop were
missing from both the help output and .PHONY. tests/fuzzing.md already
documents the working flow (download json_test_data, then
`make -C tests fuzzers`), so replace the six broken targets and their
help lines with a single pointer to that document instead of trying
to keep six copies of a fragile shell pipeline in sync.

This does not affect OSS-Fuzz, which builds through tests/Makefile.

Overlaps #5621, which adds a seventh copy of the same broken line for
fuzz_testing_json_view.

#5717 item 2

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-09-30 18:00:09 +02:00
Niels Lohmann bc6e7db724 Hash every release_files artifact, not a hardcoded subset
The `release` target signed and copied json_fwd.hpp into release_files
alongside json.hpp, but the shasum line that writes hashes.txt only
listed json.hpp, include.zip and json.tar.xz. Users could not verify
the published json_fwd.hpp against hashes.txt.

Hash every file in release_files except the .asc signatures instead
of naming files by hand, so a newly shipped header (such as the
json_literals.hpp that #5610 adds to this target) cannot be missed
again.

Only affects the generated hashes.txt release artifact; the library
itself is unaffected.

Overlaps #5610, which touches the same lines to add json_literals.hpp
to the release target.

#5717 item 1

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-09-30 17:59:49 +02:00
Niels Lohmann ea9c4a2501 Document tools/macro_builder and tidy up serve_header.py
tools/macro_builder generates the NLOHMANN_JSON_EXPAND,
NLOHMANN_JSON_GET_MACRO and NLOHMANN_JSON_PASTE* macros in
macro_scope.hpp, but nothing referred to it. Add a README that explains
what it generates, how to run it and where the output goes, and which
dependent tables (NLOHMANN_JSON_DOUBLE_PASTE, NLOHMANN_JSON_TYPE_BODY)
are maintained by hand. Point to it from a comment above
NLOHMANN_JSON_EXPAND. The generator itself is unchanged; following the
README reproduces the header byte for byte.

In serve_header.py, drop the LGTM suppression (LGTM.com shut down in
2022), replace the """.""" placeholder docstrings with real ones, and
import socket and ssl at module level. DualStackServer.server_bind uses
socket, which was only imported under __main__; when the module was
imported instead, the NameError was swallowed and IPV6_V6ONLY was not
cleared.

The header change is a comment only; behavior, API and ABI are
unchanged.

Part of #5717

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-09-30 09:45:44 +02:00
Niels Lohmann 83477108b5 Remove dead doctest help entry and pretty_format target from Makefile
The top-level Makefile still carried three leftovers:

- The help text listed a "doctest" target that was removed in #4560,
  so "make doctest" fails with "No rule to make target". The example
  check now runs as "make check_output -C docs".
- "pretty_format" ran clang-format on all sources, but .clang-format
  was deleted in #4573, so the target reformatted everything in the
  default LLVM style, against the Artistic Style formatting that
  "make pretty" applies and CI enforces.
- "clean" removed benchmarks/files/numbers/*.json, a directory that no
  longer exists since the benchmarks moved to tests/benchmarks (#3462).

Only maintainer tooling changes; the library is not affected.

Part of #5717

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-09-30 09:38:23 +02:00
22 changed files with 1326 additions and 3066 deletions
+56
View File
@@ -35,6 +35,7 @@ jobs:
MAIN_DIR: ${{ github.workspace }}/main
INCLUDE_DIR: ${{ github.workspace }}/main/single_include/nlohmann
TOOL_DIR: ${{ github.workspace }}/tools/tools/amalgamate
NATVIS_TOOL_DIR: ${{ github.workspace }}/tools/tools/generate_natvis
steps:
- name: Harden Runner
@@ -61,6 +62,48 @@ jobs:
python3 -mvenv venv
venv/bin/pip3 install -r $MAIN_DIR/tools/astyle/requirements.txt
- name: Install generate_natvis dependencies
run: pip3 install -r $NATVIS_TOOL_DIR/requirements.txt
- name: Regenerate the tools/macro_builder tables in macro_scope.hpp
run: |
cd $MAIN_DIR
# Built from this PR's own tools/macro_builder/main.cpp, not a
# develop checkout: unlike amalgamate.py and generate_natvis.py,
# this tool has no other source of truth to check against (its
# own README documents that it must reproduce macro_scope.hpp
# byte for byte), so there is nothing to gain from checking out a
# separate copy, and doing so would make this step fail on a PR
# that adds support for a new dispatch table until that PR itself
# merges to develop, the same way generate_natvis.py's --version
# requirement briefly did.
TMPDIR=$(mktemp -d ./macro_builder_check.XXXXXX)
c++ -std=c++11 tools/macro_builder/main.cpp -o "$TMPDIR/macro_builder"
"$TMPDIR/macro_builder" > "$TMPDIR/paste.hpp"
"$TMPDIR/macro_builder" type_body > "$TMPDIR/type_body.hpp"
# Splice the (still unindented) generated blocks back into
# macro_scope.hpp; the astyle pass below indents their
# continuation lines the same way it does for the rest of
# include/, so a correctly regenerated file comes out unchanged.
awk -v newfile="$TMPDIR/paste.hpp" '
BEGIN { while ((getline line < newfile) > 0) { new = new line "\n" } }
/^#define NLOHMANN_JSON_EXPAND\( x \) x$/ { printf "%s", new; skip=1 }
skip && /^#define NLOHMANN_JSON_DOUBLE_PASTE63\(/ { skip=0; next }
skip { next }
{ print }
' include/nlohmann/detail/macro_scope.hpp > "$TMPDIR/macro_scope_1.hpp"
awk -v newfile="$TMPDIR/type_body.hpp" '
BEGIN { while ((getline line < newfile) > 0) { new = new line "\n" } }
/^#define NLOHMANN_JSON_TYPE_BODY\(Prefix, \.\.\.\)/ { printf "%s", new; skip=1 }
skip && /^[[:space:]]*NLOHMANN_JSON_TYPE_BODY_SENTINEL\)\)$/ { skip=0; next }
skip { next }
{ print }
' "$TMPDIR/macro_scope_1.hpp" > "$TMPDIR/macro_scope_2.hpp"
mv "$TMPDIR/macro_scope_2.hpp" include/nlohmann/detail/macro_scope.hpp
rm -rf "$TMPDIR"
- name: Regenerate amalgamation, formatting, and BUILD.bazel
run: |
cd $MAIN_DIR
@@ -88,6 +131,19 @@ jobs:
${{ github.workspace }}/venv/bin/astyle --project=tools/astyle/.astylerc --suffix=none --quiet \
$(find $SOURCE_DIRS -type f \( -name '*.hpp' -o -name '*.cpp' -o -name '*.cu' \) -not -path 'tests/thirdparty/*' -not -path 'tests/abi/include/nlohmann/*' | sort)
- name: Regenerate nlohmann_json.natvis
run: |
cd $MAIN_DIR
# Pass --version explicitly so this step also works with the tool
# copy from develop before this repository's own generate_natvis.py
# learns to derive the version itself: the older script requires
# --version, and the newer one accepts it as an explicit override.
ABI_MACROS=include/nlohmann/detail/abi_macros.hpp
VERSION_MAJOR=$(grep -m1 'define NLOHMANN_JSON_VERSION_MAJOR' $ABI_MACROS | grep -o '[0-9]\+')
VERSION_MINOR=$(grep -m1 'define NLOHMANN_JSON_VERSION_MINOR' $ABI_MACROS | grep -o '[0-9]\+')
VERSION_PATCH=$(grep -m1 'define NLOHMANN_JSON_VERSION_PATCH' $ABI_MACROS | grep -o '[0-9]\+')
python3 $NATVIS_TOOL_DIR/generate_natvis.py --version "$VERSION_MAJOR.$VERSION_MINOR.$VERSION_PATCH" $MAIN_DIR
- name: Build patch and check for differences
id: diff
run: |
+32 -83
View File
@@ -1,4 +1,4 @@
.PHONY: pretty clean ChangeLog.md release update_hedley update_hedley_undef BUILD.bazel
.PHONY: pretty clean ChangeLog.md release update_hedley update_hedley_undef BUILD.bazel natvis macro_builder_check
##########################################################################
# configuration
@@ -24,6 +24,9 @@ AMALGAMATED_FWD_FILE=single_include/nlohmann/json_fwd.hpp
# json_literals.hpp only includes <nlohmann/json.hpp>, so it is copied verbatim
AMALGAMATED_LITERALS_FILE=single_include/nlohmann/json_literals.hpp
# the header with the argument-counting macros generated by tools/macro_builder
MACRO_SCOPE_HPP=include/nlohmann/detail/macro_scope.hpp
##########################################################################
# documentation of the Makefile's targets
@@ -36,13 +39,9 @@ all:
@echo "ChangeLog.md - generate ChangeLog file"
@echo "check-amalgamation - check whether sources have been amalgamated and BUILD.bazel is up to date"
@echo "clean - remove built files"
@echo "doctest - compile example files and check their output"
@echo "fuzz_testing - prepare fuzz testing of the JSON parser"
@echo "fuzz_testing_bon8 - prepare fuzz testing of the BON8 parser"
@echo "fuzz_testing_bson - prepare fuzz testing of the BSON parser"
@echo "fuzz_testing_cbor - prepare fuzz testing of the CBOR parser"
@echo "fuzz_testing_msgpack - prepare fuzz testing of the MessagePack parser"
@echo "fuzz_testing_ubjson - prepare fuzz testing of the UBJSON parser"
@echo "fuzzing - see tests/fuzzing.md for how to build and run the fuzzers"
@echo "macro_builder_check - check that macro_scope.hpp matches tools/macro_builder's output"
@echo "natvis - regenerate nlohmann_json.natvis from the current ABI tags and version"
@echo "pretty - beautify code with Artistic Style"
@echo "run_benchmarks - build and run benchmarks"
@echo "update_hedley - download Hedley and regenerate hedley.hpp / hedley_undef.hpp"
@@ -61,74 +60,6 @@ run_benchmarks:
cd cmake-build-benchmarks ; ./json_benchmarks
##########################################################################
# fuzzing
##########################################################################
# the overall fuzz testing target
fuzz_testing:
rm -fr fuzz-testing
mkdir -p fuzz-testing fuzz-testing/testcases fuzz-testing/out
$(MAKE) parse_afl_fuzzer -C tests CXX=afl-clang++
mv tests/parse_afl_fuzzer fuzz-testing/fuzzer
find tests/data/json_tests -size -5k -name *json | xargs -I{} cp "{}" fuzz-testing/testcases
@echo "Execute: afl-fuzz -i fuzz-testing/testcases -o fuzz-testing/out fuzz-testing/fuzzer"
fuzz_testing_bon8:
rm -fr fuzz-testing
mkdir -p fuzz-testing fuzz-testing/testcases fuzz-testing/out
$(MAKE) parse_bon8_fuzzer -C tests CXX=afl-clang++
mv tests/parse_bon8_fuzzer fuzz-testing/fuzzer
find tests/data -size -5k -name *.bon8 | xargs -I{} cp "{}" fuzz-testing/testcases
@echo "Execute: afl-fuzz -i fuzz-testing/testcases -o fuzz-testing/out fuzz-testing/fuzzer"
fuzz_testing_bson:
rm -fr fuzz-testing
mkdir -p fuzz-testing fuzz-testing/testcases fuzz-testing/out
$(MAKE) parse_bson_fuzzer -C tests CXX=afl-clang++
mv tests/parse_bson_fuzzer fuzz-testing/fuzzer
find tests/data -size -5k -name *.bson | xargs -I{} cp "{}" fuzz-testing/testcases
@echo "Execute: afl-fuzz -i fuzz-testing/testcases -o fuzz-testing/out fuzz-testing/fuzzer"
fuzz_testing_cbor:
rm -fr fuzz-testing
mkdir -p fuzz-testing fuzz-testing/testcases fuzz-testing/out
$(MAKE) parse_cbor_fuzzer -C tests CXX=afl-clang++
mv tests/parse_cbor_fuzzer fuzz-testing/fuzzer
find tests/data -size -5k -name *.cbor | xargs -I{} cp "{}" fuzz-testing/testcases
@echo "Execute: afl-fuzz -i fuzz-testing/testcases -o fuzz-testing/out fuzz-testing/fuzzer"
fuzz_testing_msgpack:
rm -fr fuzz-testing
mkdir -p fuzz-testing fuzz-testing/testcases fuzz-testing/out
$(MAKE) parse_msgpack_fuzzer -C tests CXX=afl-clang++
mv tests/parse_msgpack_fuzzer fuzz-testing/fuzzer
find tests/data -size -5k -name *.msgpack | xargs -I{} cp "{}" fuzz-testing/testcases
@echo "Execute: afl-fuzz -i fuzz-testing/testcases -o fuzz-testing/out fuzz-testing/fuzzer"
fuzz_testing_ubjson:
rm -fr fuzz-testing
mkdir -p fuzz-testing fuzz-testing/testcases fuzz-testing/out
$(MAKE) parse_ubjson_fuzzer -C tests CXX=afl-clang++
mv tests/parse_ubjson_fuzzer fuzz-testing/fuzzer
find tests/data -size -5k -name *.ubjson | xargs -I{} cp "{}" fuzz-testing/testcases
@echo "Execute: afl-fuzz -i fuzz-testing/testcases -o fuzz-testing/out fuzz-testing/fuzzer"
fuzzing-start:
afl-fuzz -S fuzzer1 -i fuzz-testing/testcases -o fuzz-testing/out fuzz-testing/fuzzer > /dev/null &
afl-fuzz -S fuzzer2 -i fuzz-testing/testcases -o fuzz-testing/out fuzz-testing/fuzzer > /dev/null &
afl-fuzz -S fuzzer3 -i fuzz-testing/testcases -o fuzz-testing/out fuzz-testing/fuzzer > /dev/null &
afl-fuzz -S fuzzer4 -i fuzz-testing/testcases -o fuzz-testing/out fuzz-testing/fuzzer > /dev/null &
afl-fuzz -S fuzzer5 -i fuzz-testing/testcases -o fuzz-testing/out fuzz-testing/fuzzer > /dev/null &
afl-fuzz -S fuzzer6 -i fuzz-testing/testcases -o fuzz-testing/out fuzz-testing/fuzzer > /dev/null &
afl-fuzz -S fuzzer7 -i fuzz-testing/testcases -o fuzz-testing/out fuzz-testing/fuzzer > /dev/null &
afl-fuzz -M fuzzer0 -i fuzz-testing/testcases -o fuzz-testing/out fuzz-testing/fuzzer
fuzzing-stop:
-killall fuzzer
-killall afl-fuzz
##########################################################################
# Static analysis
##########################################################################
@@ -158,10 +89,6 @@ install_astyle:
pretty: install_astyle
$(ASTYLE) --project=tools/astyle/.astylerc $(SRCS) $(TESTS_SRCS) $(AMALGAMATED_FILE) $(AMALGAMATED_FWD_FILE) $(AMALGAMATED_LITERALS_FILE) docs/mkdocs/docs/examples/*.cpp
# call the Clang-Format on all source files
pretty_format:
for FILE in $(SRCS) $(TESTS_SRCS) $(AMALGAMATED_FILE) docs/mkdocs/docs/examples/*.cpp; do echo $$FILE; clang-format -i $$FILE; done
# create single header files and pretty print
amalgamate: $(AMALGAMATED_FILE) $(AMALGAMATED_FWD_FILE) $(AMALGAMATED_LITERALS_FILE)
$(MAKE) pretty
@@ -178,8 +105,26 @@ $(AMALGAMATED_FWD_FILE): $(SRCS)
$(AMALGAMATED_LITERALS_FILE): include/nlohmann/json_literals.hpp
cp include/nlohmann/json_literals.hpp $(AMALGAMATED_LITERALS_FILE)
# regenerate nlohmann_json.natvis from the ABI tags and version in include/nlohmann/detail/abi_macros.hpp
natvis:
python3 tools/generate_natvis/generate_natvis.py .
# regenerate the two tools/macro_builder blocks of $(MACRO_SCOPE_HPP) (see its README.md) and diff against the
# checked-in header; phony, because it never writes $(MACRO_SCOPE_HPP) itself
macro_builder_check:
@set -e; \
TMPDIR=$$(mktemp -d ./macro_builder_check.XXXXXX); \
trap 'rm -rf "$$TMPDIR"' EXIT; \
$(CXX) -std=c++11 tools/macro_builder/main.cpp -o "$$TMPDIR/macro_builder"; \
"$$TMPDIR/macro_builder" > "$$TMPDIR/paste.hpp"; \
"$$TMPDIR/macro_builder" type_body > "$$TMPDIR/type_body.hpp"; \
$(ASTYLE) --project=tools/astyle/.astylerc --suffix=none --quiet "$$TMPDIR/paste.hpp" "$$TMPDIR/type_body.hpp"; \
sed -n '/^#define NLOHMANN_JSON_EXPAND( x ) x$$/,/^#define NLOHMANN_JSON_DOUBLE_PASTE63(/p' $(MACRO_SCOPE_HPP) > "$$TMPDIR/paste_actual.hpp"; \
sed -n '/^#define NLOHMANN_JSON_TYPE_BODY(Prefix, \.\.\.)/,/^ NLOHMANN_JSON_TYPE_BODY_SENTINEL))$$/p' $(MACRO_SCOPE_HPP) > "$$TMPDIR/type_body_actual.hpp"; \
diff "$$TMPDIR/paste.hpp" "$$TMPDIR/paste_actual.hpp" || (echo "===================================================================\n $(MACRO_SCOPE_HPP) (NLOHMANN_JSON_EXPAND..NLOHMANN_JSON_DOUBLE_PASTE63) is out of date!\n Regenerate it, see tools/macro_builder/README.md.\n===================================================================" ; exit 1); \
diff "$$TMPDIR/type_body.hpp" "$$TMPDIR/type_body_actual.hpp" || (echo "===================================================================\n $(MACRO_SCOPE_HPP) (NLOHMANN_JSON_TYPE_BODY) is out of date!\n Regenerate it, see tools/macro_builder/README.md.\n===================================================================" ; exit 1)
# check if file single_include/nlohmann/json.hpp has been amalgamated from the nlohmann sources
# Note: this target is called by Travis
check-amalgamation:
@mv $(AMALGAMATED_FILE) $(AMALGAMATED_FILE)~
@mv $(AMALGAMATED_FWD_FILE) $(AMALGAMATED_FWD_FILE)~
@@ -195,6 +140,11 @@ check-amalgamation:
@$(MAKE) BUILD.bazel
@diff BUILD.bazel BUILD.bazel~ || (echo "===================================================================\n BUILD.bazel is out of date! Please run 'make BUILD.bazel'.\n===================================================================" ; mv BUILD.bazel~ BUILD.bazel ; false)
@mv BUILD.bazel~ BUILD.bazel
@mv nlohmann_json.natvis nlohmann_json.natvis~
@$(MAKE) natvis
@diff nlohmann_json.natvis nlohmann_json.natvis~ || (echo "===================================================================\n nlohmann_json.natvis is out of date! Please run 'make natvis'.\n===================================================================" ; mv nlohmann_json.natvis~ nlohmann_json.natvis ; false)
@mv nlohmann_json.natvis~ nlohmann_json.natvis
@$(MAKE) macro_builder_check
# generate the Bazel BUILD file; phony, because a removed header would not trigger a rebuild
BUILD.bazel:
@@ -246,7 +196,7 @@ release: include.zip json.tar.xz
cp $(AMALGAMATED_FWD_FILE) release_files
cp $(AMALGAMATED_LITERALS_FILE) release_files
mv $(AMALGAMATED_FILE).asc $(AMALGAMATED_FWD_FILE).asc $(AMALGAMATED_LITERALS_FILE).asc json.tar.xz json.tar.xz.asc include.zip include.zip.asc release_files
cd release_files ; shasum -a 256 json.hpp include.zip json.tar.xz > hashes.txt
cd release_files ; shasum -a 256 $$(find . -type f -not -name '*.asc' | sed 's|^\./||' | sort) > hashes.txt
##########################################################################
@@ -256,7 +206,6 @@ release: include.zip json.tar.xz
# clean up
clean:
rm -fr fuzz fuzz-testing *.dSYM tests/*.dSYM
rm -fr benchmarks/files/numbers/*.json
rm -fr cmake-build-benchmarks fuzz-testing cmake-build-pvs-studio release_files
$(MAKE) clean -Cdocs
+1 -1
View File
@@ -1395,7 +1395,7 @@ THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR I
- The class contains a slightly modified version of the Grisu2 algorithm from Florian Loitsch which is licensed under the [MIT License](https://opensource.org/licenses/MIT) (see above). Copyright &copy; 2009 [Florian Loitsch](https://florian.loitsch.com/)
- The class contains a copy of [Hedley](https://nemequ.github.io/hedley/) from Evan Nemerson which is licensed as [CC0-1.0](https://creativecommons.org/publicdomain/zero/1.0/).
- The class contains parts of [Google Abseil](https://github.com/abseil/abseil-cpp) which is licensed under the [Apache 2.0 License](https://opensource.org/licenses/Apache-2.0).
- The class contains an adapted version of the Eisel-Lemire algorithm, its table of powers of five, and its digit comparison for long numbers from [fast_float](https://github.com/fastfloat/fast_float) by Daniel Lemire and contributors, which is available under the [MIT License](https://opensource.org/licenses/MIT) (used here), the Apache 2.0 License, and the Boost Software License. Copyright &copy; 2021 The fast_float authors
- The class contains an adapted version of the Eisel-Lemire algorithm and its table of powers of five from [fast_float](https://github.com/fastfloat/fast_float) by Daniel Lemire and contributors, which is available under the [MIT License](https://opensource.org/licenses/MIT) (used here), the Apache 2.0 License, and the Boost Software License. Copyright &copy; 2021 The fast_float authors
<img align="right" src="https://git.fsfe.org/reuse/reuse-ci/raw/branch/master/reuse-horizontal.png" alt="REUSE Software">
@@ -23,10 +23,9 @@ type to use.
## Template parameters
`NumberFloatType`
: the type to store floating-point numbers. The parser converts `#!cpp float`, `#!cpp double`, and a
`#!cpp long double` that is IEEE 754 binary64 itself and other `#!cpp long double` formats with
`#!cpp std::from_chars` or `#!cpp std::strtold`, and serialization falls back to `#!cpp std::snprintf`, so the
type must be `#!cpp float`, `#!cpp double`, or `#!cpp long double`. The
: the type to store floating-point numbers. Parsing and serialization are implemented in terms of
`#!cpp std::strtof`/`#!cpp std::strtod`/`#!cpp std::strtold` and `#!cpp std::snprintf`, so the type must be
`#!cpp float`, `#!cpp double`, or `#!cpp long double`. The
[binary formats](../../features/binary_formats/index.md) additionally require `#!cpp float` or `#!cpp double`,
because they have no encoding for `#!cpp long double`. See
[Template Parameter Requirements](../../features/types/template_parameters.md#numberfloattype).
@@ -71,11 +71,10 @@ otherwise, it uses unsigned integer storage.
- Numbers with a decimal digit or scientific notation are always stored as `#!c double`.
- The number types can be changed, see [Template number types](#template-number-types).
- The library converts integers and floating-point numbers itself, independent of the locale. Floating-point
numbers are correctly rounded (to nearest, ties to even). Only a `#!c long double` that is not IEEE 754 binary64
(e.g., the 80-bit x87 format) is converted with `#!cpp std::from_chars` where available, or with
[`std::strtold`](https://en.cppreference.com/w/cpp/string/byte/strtof), which gets the decimal point of the
current locale, also one longer than one byte (e.g., in `fa_IR.UTF-8`).
- As of version 3.9.1, the conversion is realized by
[`std::strtoull`](https://en.cppreference.com/w/cpp/string/byte/strtoul),
[`std::strtoll`](https://en.cppreference.com/w/cpp/string/byte/strtol), and
[`std::strtod`](https://en.cppreference.com/w/cpp/string/byte/strtof), respectively.
!!! example "Examples"
@@ -86,10 +85,10 @@ otherwise, it uses unsigned integer storage.
### Number limits
- Any 64-bit signed or unsigned integer can be stored without loss of precision.
- Numbers exceeding the limits of `#!c double` (i.e., numbers whose rounded value is not satisfying
- Numbers exceeding the limits of `#!c double` (i.e., numbers that after conversion via
[`std::strtod`](https://en.cppreference.com/w/cpp/string/byte/strtof) are not satisfying
[`std::isfinite`](https://en.cppreference.com/w/cpp/numeric/math/isfinite) such as `#!c 1E400`) will throw exception
[`json.exception.out_of_range.406`](../../home/exceptions.md#jsonexceptionout_of_range406) during parsing. Numbers too
small for `#!c double` (such as `#!c 1E-400`) become zero, with the sign of the number.
[`json.exception.out_of_range.406`](../../home/exceptions.md#jsonexceptionout_of_range406) during parsing.
- Floating-point numbers are rounded to the next number representable as `double`. For instance
`#!c 3.141592653589793238462643383279` is stored as [`0x400921fb54442d18`](https://float.exposed/0x400921fb54442d18).
This is the same behavior as the code `#!c double x = 3.141592653589793238462643383279;`.
@@ -26,9 +26,8 @@ Requirements are split into two groups:
diagnosed with dedicated error messages, and violating most of them results in a compiler error somewhere inside
the library. Four violations are not caught at compile time at all:
- A [`StringType`](#stringtype) whose `data()` is not null-terminated compiles and silently misparses numbers
stored as a `#!cpp long double` that is not IEEE 754 binary64 (e.g., the 80-bit x87 format), because the lexer
hands the buffer to `#!cpp std::strtold`.
- A [`StringType`](#stringtype) whose `data()` is not null-terminated compiles and silently misparses numbers,
because the lexer hands the buffer to `#!cpp std::strtoull`/`#!cpp std::strtoll`/`#!cpp std::strtod`.
- A stateful [`AllocatorType`](#allocatortype) compiles and silently ignores its state: allocation, deallocation,
and [`get_allocator()`](../../api/basic_json/get_allocator.md) each use a different default-constructed instance.
- The two [cross-specialization conversions](#cross-specialization-conversions) below. These abort on an assertion
@@ -536,10 +535,8 @@ therefore silently changes parse results rather than raising an error. See
`NumberFloatType` must be one of `#!cpp float`, `#!cpp double`, or `#!cpp long double`:
- The [parser](../parsing/index.md) converts number literals to `#!cpp float`, `#!cpp double`, and a
`#!cpp long double` that is IEEE 754 binary64 itself; other `#!cpp long double` formats are converted with
`#!cpp std::from_chars` where available, or with `#!cpp std::strtold`. The library provides overloads for exactly
these three types.
- The [parser](../parsing/index.md) converts number literals with `#!cpp std::strtof`, `#!cpp std::strtod`, or
`#!cpp std::strtold`; the library provides overloads for exactly these three types.
- [`dump`](../../api/basic_json/dump.md) falls back to `#!cpp std::snprintf` with the `%g` and `%Lg` conversion
specifiers, for which the library likewise provides only `#!cpp double` and `#!cpp long double` overloads
(`#!cpp float` is promoted to `#!cpp double`).
+1 -1
View File
@@ -20,4 +20,4 @@ The class contains a slightly modified version of the Grisu2 algorithm from Flor
The class contains a copy of [Hedley](https://nemequ.github.io/hedley/) from Evan Nemerson which is licensed as [CC0-1.0](https://creativecommons.org/publicdomain/zero/1.0/).
The class contains an adapted version of the Eisel-Lemire algorithm, its table of powers of five, and its digit comparison for long numbers from [fast_float](https://github.com/fastfloat/fast_float) by Daniel Lemire and contributors, which is available under the [MIT License](https://opensource.org/licenses/MIT) (used here), the Apache 2.0 License, and the Boost Software License. Copyright &copy; 2021 The fast_float authors
The class contains an adapted version of the Eisel-Lemire algorithm and its table of powers of five from [fast_float](https://github.com/fastfloat/fast_float) by Daniel Lemire and contributors, which is available under the [MIT License](https://opensource.org/licenses/MIT) (used here), the Apache 2.0 License, and the Boost Software License. Copyright &copy; 2021 The fast_float authors
+2 -26
View File
@@ -39,25 +39,6 @@ inline int count_leading_zeros(std::uint64_t x) noexcept
#endif
}
/// number of trailing zero bits of x (x != 0)
inline int count_trailing_zeros(std::uint64_t x) noexcept
{
#if defined(__GNUC__) || defined(__clang__)
return __builtin_ctzll(x);
#else
int n = 0;
for (int shift = 32; shift != 0; shift >>= 1)
{
if ((x << (64 - shift)) == 0)
{
n += shift;
x >>= shift;
}
}
return n;
#endif
}
/// the 128-bit product of two 64-bit numbers
struct uint128_parts
{
@@ -87,19 +68,14 @@ inline uint128_parts full_multiplication(std::uint64_t a, std::uint64_t b) noexc
/// eight bytes as a little-endian word (compilers fold this into one load on
/// little-endian targets)
inline std::uint64_t read_eight_bytes(const unsigned char* b) noexcept
inline std::uint64_t read_eight_bytes(const char* p) noexcept
{
const auto* b = reinterpret_cast<const unsigned char*>(p); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
return static_cast<std::uint64_t>(b[0]) | (static_cast<std::uint64_t>(b[1]) << 8u)
| (static_cast<std::uint64_t>(b[2]) << 16u) | (static_cast<std::uint64_t>(b[3]) << 24u)
| (static_cast<std::uint64_t>(b[4]) << 32u) | (static_cast<std::uint64_t>(b[5]) << 40u)
| (static_cast<std::uint64_t>(b[6]) << 48u) | (static_cast<std::uint64_t>(b[7]) << 56u);
}
/// eight bytes as a little-endian word
inline std::uint64_t read_eight_bytes(const char* p) noexcept
{
return read_eight_bytes(reinterpret_cast<const unsigned char*>(p)); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
}
} // namespace detail
NLOHMANN_JSON_NAMESPACE_END
+10 -59
View File
@@ -219,44 +219,6 @@ class lexer : public lexer_base<BasicJsonType>
// scan functions
/////////////////////
/// contiguous input: try to decode the 4 hex digits following `\u`
/// directly from the input buffer via hex_codepoint(), instead of 4 calls
/// to get(). On success, advances the adapter and the position counters
/// exactly as those 4 get() calls would (a hex digit is never '\n', so
/// only the flat counters move) and leaves @a current holding the last of
/// the 4 digits, just as the last such get() would; the codepoint is
/// written to @a out. Makes no state change and returns false - for a
/// pending unget, fewer than 4 remaining bytes, or any of the 4 bytes not
/// being a hex digit - so the caller falls back unchanged to the
/// per-character loop, which then reports the same diagnostic (stopping
/// at the first invalid digit) as before this optimization.
bool get_codepoint_bulk(std::true_type /*bulk*/, int& out)
{
if (next_unget || ia.bulk_remaining() < 4)
{
return false;
}
const char_type* const raw = ia.bulk_data();
const int codepoint = hex_codepoint(reinterpret_cast<const unsigned char*>(raw));
if (codepoint < 0)
{
return false;
}
ia.bulk_skip(4);
// a hex digit is never a newline, so only the flat counters advance
position.chars_read_total += 4;
position.chars_read_current_line += 4;
current = char_traits<char_type>::to_int_type(raw[3]);
out = codepoint;
return true;
}
/// streaming input: no bulk fast path
bool get_codepoint_bulk(std::false_type /*bulk*/, int& /*out*/) const noexcept
{
return false;
}
/*!
@brief get codepoint from 4 hex characters following `\u`
@@ -276,14 +238,6 @@ class lexer : public lexer_base<BasicJsonType>
{
// this function only makes sense after reading `\u`
JSON_ASSERT(current == 'u');
// contiguous input: decode all 4 hex digits directly from the buffer
int fast_codepoint = 0;
if (get_codepoint_bulk(std::integral_constant<bool, bulk_scan> {}, fast_codepoint))
{
return fast_codepoint;
}
int codepoint = 0;
const auto factors = { 12u, 8u, 4u, 0u };
@@ -1105,11 +1059,9 @@ class lexer : public lexer_base<BasicJsonType>
token_type::parse_error otherwise
@note The scanner is independent of the current locale: token_buffer
always holds `.`. The conversion of float and double does not use
the locale either. Only the std::strtold fallback of
convert_number() for long double formats other than binary64
depends on it, and it looks up the decimal point right before
converting (see detail::convert_float_locale_aware()).
always holds `.`. Only the std::strtod fallback of convert_number()
depends on the locale, and it looks up the decimal point right
before converting (see detail::convert_float_locale_aware()).
*/
token_type scan_number() // lgtm [cpp/use-of-goto] `goto` is used in this function to implement the number-parsing state machine described above. By design, any finite input will eventually reach the "done" state or return token_type::parse_error. In each intermediate state, 1 byte of the input is appended to the token_buffer vector, and only the already initialized variables token_buffer, number_type, and error_message are manipulated.
{
@@ -1122,7 +1074,7 @@ class lexer : public lexer_base<BasicJsonType>
// offset just past the last mantissa byte in token_buffer (i.e. the
// index of 'e'/'E', or the whole token when there is no exponent).
// convert_number() uses it to split the token; npos means
// convert_number() uses it to count significant digits; npos means
// "not seen an exponent yet" and is resolved at scan_number_done
std::size_t mantissa_end = std::string::npos;
@@ -1452,8 +1404,8 @@ scan_number_done:
@param[in] mantissa_end offset just past the last mantissa byte in
token_buffer (the index of 'e'/'E', or
token_buffer.size() when there is no exponent);
with decimal_point_position, it locates the parts
of a float token without scanning it again
used to skip Clinger's fast path when it cannot
possibly succeed - see detail::mantissa_fits_clinger()
*/
token_type convert_number(token_type number_type, std::size_t mantissa_end)
{
@@ -1522,11 +1474,10 @@ scan_number_done:
}
// this code is reached if we parse a floating-point number or if an
// integer conversion above overflowed. float and double (and long
// double where it is binary64) are converted by the library itself,
// correctly rounded and independent of the locale; other long double
// formats use std::from_chars when available, otherwise the
// locale-aware strtold.
// integer conversion above overflowed. Prefer std::from_chars
// (Eisel-Lemire, locale-independent, correctly rounded) when available;
// otherwise the exact Clinger fast path (double only); otherwise the
// locale-aware strtof/strtod/strtold.
if (convert_float_fast(num_begin, num_end, decimal_point_position, mantissa_end, value_float))
{
return token_type::value_float;
File diff suppressed because it is too large Load Diff
+29 -70
View File
@@ -8,12 +8,10 @@
#pragma once
#include <array> // array
#include <cstddef> // size_t
#include <cstdint> // uint64_t, uint8_t
#include <cstdint> // uint64_t
#include <cstring> // memcpy
#include <nlohmann/detail/bit_ops.hpp>
#include <nlohmann/detail/macro_scope.hpp>
// Optional SIMD backend for bulk UTF-8 validation. This is an opt-in external
@@ -71,12 +69,18 @@ inline std::size_t find_string_special(const unsigned char* data, std::size_t n)
std::size_t i = 0;
for (; i + 8 <= n; i += 8)
{
const std::uint64_t special = swar_string_special(read_eight_bytes(data + i));
if (special != 0)
std::uint64_t word = 0;
std::memcpy(&word, data + i, sizeof(word));
if (swar_string_special(word) != 0)
{
// the lowest flagged byte is the first special one: the borrows of
// the subtractions can only flag bytes above a true hit
return i + (static_cast<std::size_t>(count_trailing_zeros(special)) / 8);
// a special byte is in this word; locate it (endian-agnostic)
for (std::size_t j = 0; j < 8; ++j)
{
if (is_string_special(data[i + j]))
{
return i + j;
}
}
}
}
for (; i < n; ++i)
@@ -110,7 +114,8 @@ inline std::size_t find_ascii_copyable_run(const unsigned char* data, std::size_
std::size_t i = 0;
for (; i + 8 <= n; i += 8)
{
const std::uint64_t v = read_eight_bytes(data + i);
std::uint64_t v = 0;
std::memcpy(&v, data + i, sizeof(v));
const std::uint64_t q = v ^ 0x2222222222222222ull; // '"' (0x22)
const std::uint64_t b = v ^ 0x5C5C5C5C5C5C5C5Cull; // '\\' (0x5C)
const std::uint64_t d = v ^ 0x7F7F7F7F7F7F7F7Full; // DEL (0x7F)
@@ -121,9 +126,7 @@ inline std::size_t find_ascii_copyable_run(const unsigned char* data, std::size_
| (v & high); // >= 0x80
if (stop != 0)
{
// the lowest flagged byte is the first one to stop at (see
// find_string_special())
return i + (static_cast<std::size_t>(count_trailing_zeros(stop)) / 8);
break;
}
}
for (; i < n; ++i)
@@ -250,18 +253,12 @@ inline std::size_t scalar_string_bulk_run(const unsigned char* data, std::size_t
{
break; // end of buffer, or a quote/escape/control byte
}
// a run of multi-byte sequences (e.g. CJK text) is validated sequence
// by sequence without searching for the next special byte in between
do
const std::size_t seq = validate_one_utf8(data + pos, n - pos);
if (seq == 0)
{
const std::size_t seq = validate_one_utf8(data + pos, n - pos);
if (seq == 0)
{
return pos; // ill-formed or truncated: let the byte path diagnose it
}
pos += seq;
break; // ill-formed or truncated: let the byte path diagnose it
}
while (pos < n && data[pos] >= 0x80u);
pos += seq;
}
return pos;
}
@@ -276,7 +273,8 @@ inline std::size_t find_string_delimiter(const unsigned char* data, std::size_t
std::size_t i = 0;
for (; i + 8 <= n; i += 8)
{
const std::uint64_t v = read_eight_bytes(data + i);
std::uint64_t v = 0;
std::memcpy(&v, data + i, sizeof(v));
const std::uint64_t q = v ^ 0x2222222222222222ull;
const std::uint64_t b = v ^ 0x5C5C5C5C5C5C5C5Cull;
const std::uint64_t hit = ((q - ones) & ~q & high)
@@ -284,8 +282,14 @@ inline std::size_t find_string_delimiter(const unsigned char* data, std::size_t
| ((v - 0x2020202020202020ull) & ~v & high);
if (hit != 0)
{
// the lowest flagged byte is the first delimiter (see find_string_special())
return i + (static_cast<std::size_t>(count_trailing_zeros(hit)) / 8);
for (std::size_t j = 0; j < 8; ++j)
{
const unsigned char c = data[i + j];
if (c == '\"' || c == '\\' || c < 0x20u)
{
return i + j;
}
}
}
}
for (; i < n; ++i)
@@ -316,50 +320,5 @@ inline std::size_t string_bulk_run(const unsigned char* data, std::size_t n) noe
return scalar_string_bulk_run(data, n);
}
// Decode the 4 hex digits at [data, data+4) - the digits following a `\u`
// escape - into a codepoint 0x0000..0xFFFF via one table lookup per byte
// (after yyjson's read_hex_u16), or return -1 if any of the 4 bytes is not a
// hex digit ('0'..'9', 'A'..'F', 'a'..'f'). The caller must already have
// checked that 4 bytes are available; used by lexer::get_codepoint()'s
// contiguous fast path. On -1 it falls back to the byte-at-a-time loop, which
// stops at the first invalid digit, so the reported error and position are
// unaffected by this fast path.
inline int hex_codepoint(const unsigned char* data) noexcept
{
static const std::array<std::uint8_t, 256> hex_digit_table = // NOLINT(cppcoreguidelines-avoid-non-const-global-variables)
{
{
0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, // 00..0F
0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, // 10..1F
0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, // 20..2F
0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, // 30..3F ('0'..'9')
0xFF, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, // 40..4F ('A'..'F')
0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, // 50..5F
0xFF, 0x0A, 0x0B, 0x0C, 0x0D, 0x0E, 0x0F, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, // 60..6F ('a'..'f')
0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, // 70..7F
0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, // 80..8F
0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, // 90..9F
0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, // A0..AF
0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, // B0..BF
0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, // C0..CF
0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, // D0..DF
0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, // E0..EF
0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF // F0..FF
}
};
const std::uint8_t d0 = hex_digit_table[data[0]];
const std::uint8_t d1 = hex_digit_table[data[1]];
const std::uint8_t d2 = hex_digit_table[data[2]];
const std::uint8_t d3 = hex_digit_table[data[3]];
// every valid digit is <= 0xF; the combined OR only exceeds it if at
// least one of the four bytes was not a hex digit (looked up as 0xFF)
if ((d0 | d1 | d2 | d3) > 0x0F)
{
return -1;
}
return (d0 << 12) | (d1 << 8) | (d2 << 4) | d3;
}
} // namespace detail
NLOHMANN_JSON_NAMESPACE_END
+5
View File
@@ -359,6 +359,7 @@ void templated_json_throw(ExceptionType exception)
// Macros to simplify conversion from/to types
// NLOHMANN_JSON_EXPAND to NLOHMANN_JSON_DOUBLE_PASTE63 are generated by tools/macro_builder (see its README.md)
#define NLOHMANN_JSON_EXPAND( x ) x
#define NLOHMANN_JSON_GET_MACRO(_1, _2, _3, _4, _5, _6, _7, _8, _9, _10, _11, _12, _13, _14, _15, _16, _17, _18, _19, _20, _21, _22, _23, _24, _25, _26, _27, _28, _29, _30, _31, _32, _33, _34, _35, _36, _37, _38, _39, _40, _41, _42, _43, _44, _45, _46, _47, _48, _49, _50, _51, _52, _53, _54, _55, _56, _57, _58, _59, _60, _61, _62, _63, _64, NAME,...) NAME
#define NLOHMANN_JSON_PASTE(...) NLOHMANN_JSON_EXPAND(NLOHMANN_JSON_GET_MACRO(__VA_ARGS__, \
@@ -621,6 +622,10 @@ void templated_json_throw(ExceptionType exception)
// arguments, so dispatching on Type,BaseType,member... directly would run out
// one slot early and cap the derived-type macros at 62 members instead of the
// 63 that NLOHMANN_JSON_PASTE supports.
//
// The slot table below (down to the closing NLOHMANN_JSON_TYPE_BODY_SENTINEL))
// is generated by tools/macro_builder (see its README.md; run with the
// "type_body" argument).
#define NLOHMANN_JSON_TYPE_BODY(Prefix, ...) NLOHMANN_JSON_EXPAND(NLOHMANN_JSON_GET_MACRO(__VA_ARGS__, \
Prefix ## MEMBERS, Prefix ## MEMBERS, Prefix ## MEMBERS, Prefix ## MEMBERS, Prefix ## MEMBERS, Prefix ## MEMBERS, Prefix ## MEMBERS, Prefix ## MEMBERS, \
Prefix ## MEMBERS, Prefix ## MEMBERS, Prefix ## MEMBERS, Prefix ## MEMBERS, Prefix ## MEMBERS, Prefix ## MEMBERS, Prefix ## MEMBERS, Prefix ## MEMBERS, \
File diff suppressed because it is too large Load Diff
-599
View File
@@ -1,599 +0,0 @@
// __ _____ _____ _____
// __| | __| | | | JSON for Modern C++ (supporting code)
// | | |__ | | | | | | version 3.12.0
// |_____|_____|_____|_|___| https://github.com/nlohmann/json
//
// SPDX-FileCopyrightText: 2013-2026 Niels Lohmann <https://nlohmann.me>
// SPDX-License-Identifier: MIT
#pragma once
#include <array> // array
#include <cstdint> // uint32_t, uint64_t
// Number tokens that are hard to round correctly, with the IEEE-754 binary64
// and binary32 bits of their correctly rounded values (ties to even; infinity
// for an overflow, a signed zero for an underflow).
//
// For doubles and floats around 0, the smallest normal number, 1, 2^24, 2^53,
// 0.1, and the largest finite number, and for random ones, the exact midpoint
// m to the next number gives: m, m with one unit more and less in the last
// digit, m with "01" and "0...01" appended, m with trailing zeros, and m cut
// after 17 to 30 digits (rounded down and up, so that the rounding is decided
// after the 19th digit), in fixed and exponent notation, 30% of them negative.
// Tokens longer than 80 characters are left out, except for four of 700 digits
// and more. Zeros, underflow, overflow, huge exponents, and integers beyond 64
// bits complete the set. Of the 508 tokens, 134 (as double) and 150 (as
// float) need the exact comparison with the midpoint (detail::digit_comparison()).
//
// The expected bits were computed with exact rational arithmetic in Python
// (fractions.Fraction) and cross-checked with Python's float(); strtod_l and
// strtof_l of Apple's libc and of glibc agree. Generated by
// compact_hard_cases.py 5 (with hard_cases.py), see the pull request that
// added this file.
namespace float_hard_cases
{
struct hard_case
{
const char* token;
std::uint64_t bits64;
std::uint32_t bits32;
};
inline const std::array<hard_case, 508>& cases()
{
static const std::array<hard_case, 508> table =
{
{
{"-2.4703282292062327e-324", 0x8000000000000000u, 0x80000000u},
{"24703282292062328e-340", 0x0000000000000001u, 0x00000000u},
{"247032822920623272e-341", 0x0000000000000000u, 0x00000000u},
{"-0.2470328229206232721e-323", 0x8000000000000001u, 0x80000000u},
{"-0.24703282292062327208e-323", 0x8000000000000000u, 0x80000000u},
{"-2.4703282292062327209e-324", 0x8000000000000001u, 0x80000000u},
{"2.47032822920623272088e-324", 0x0000000000000000u, 0x00000000u},
{"247032822920623272089e-344", 0x0000000000000001u, 0x00000000u},
{"-247032822920623272088284396434e-353", 0x8000000000000000u, 0x80000000u},
{"0.247032822920623272088284396435e-323", 0x0000000000000001u, 0x00000000u},
{"-74109846876186981e-340", 0x8000000000000001u, 0x80000000u},
{"0.74109846876186982e-323", 0x0000000000000002u, 0x00000000u},
{"-0.7410984687618698162e-323", 0x8000000000000001u, 0x80000000u},
{"-7.410984687618698163e-324", 0x8000000000000002u, 0x80000000u},
{"7.4109846876186981626e-324", 0x0000000000000001u, 0x00000000u},
{"-74109846876186981627e-343", 0x8000000000000002u, 0x80000000u},
{"-741098468761869816264e-344", 0x8000000000000001u, 0x80000000u},
{"0.741098468761869816265e-323", 0x0000000000000002u, 0x00000000u},
{"0.741098468761869816264853189302e-323", 0x0000000000000001u, 0x00000000u},
{"-7.41098468761869816264853189303e-324", 0x8000000000000002u, 0x80000000u},
{"0.22250738585072006e-307", 0x000FFFFFFFFFFFFEu, 0x00000000u},
{"2.2250738585072007e-308", 0x000FFFFFFFFFFFFFu, 0x00000000u},
{"2.225073858507200641e-308", 0x000FFFFFFFFFFFFEu, 0x00000000u},
{"-2225073858507200642e-326", 0x800FFFFFFFFFFFFFu, 0x80000000u},
{"22250738585072006419e-327", 0x000FFFFFFFFFFFFEu, 0x00000000u},
{"0.2225073858507200642e-307", 0x000FFFFFFFFFFFFFu, 0x00000000u},
{"0.222507385850720064199e-307", 0x000FFFFFFFFFFFFEu, 0x00000000u},
{"2.225073858507200642e-308", 0x000FFFFFFFFFFFFFu, 0x00000000u},
{"-2.22507385850720064199176395546e-308", 0x800FFFFFFFFFFFFEu, 0x80000000u},
{"222507385850720064199176395547e-337", 0x000FFFFFFFFFFFFFu, 0x00000000u},
{"-2.2250738585072011e-308", 0x800FFFFFFFFFFFFFu, 0x80000000u},
{"-22250738585072012e-324", 0x8010000000000000u, 0x80000000u},
{"-2225073858507201136e-326", 0x800FFFFFFFFFFFFFu, 0x80000000u},
{"0.2225073858507201137e-307", 0x0010000000000000u, 0x00000000u},
{"0.2225073858507201136e-307", 0x000FFFFFFFFFFFFFu, 0x00000000u},
{"-2.2250738585072011361e-308", 0x8010000000000000u, 0x80000000u},
{"2.22507385850720113605e-308", 0x000FFFFFFFFFFFFFu, 0x00000000u},
{"222507385850720113606e-328", 0x0010000000000000u, 0x00000000u},
{"22250738585072011360574097967e-336", 0x000FFFFFFFFFFFFFu, 0x00000000u},
{"0.222507385850720113605740979671e-307", 0x0010000000000000u, 0x00000000u},
{"22250738585072016e-324", 0x0010000000000000u, 0x00000000u},
{"0.22250738585072017e-307", 0x0010000000000001u, 0x00000000u},
{"0.222507385850720163e-307", 0x0010000000000000u, 0x00000000u},
{"2.225073858507201631e-308", 0x0010000000000001u, 0x00000000u},
{"-2.2250738585072016301e-308", 0x8010000000000000u, 0x80000000u},
{"22250738585072016302e-327", 0x0010000000000001u, 0x00000000u},
{"-222507385850720163012e-328", 0x8010000000000000u, 0x80000000u},
{"0.222507385850720163013e-307", 0x0010000000000001u, 0x00000000u},
{"0.222507385850720163012305563795e-307", 0x0010000000000000u, 0x00000000u},
{"-2.22507385850720163012305563796e-308", 0x8010000000000001u, 0x80000000u},
{"0.17976931348623156E+309", 0x7FEFFFFFFFFFFFFEu, 0x7F800000u},
{"1.7976931348623157e308", 0x7FEFFFFFFFFFFFFFu, 0x7F800000u},
{"1.797693134862315608e308", 0x7FEFFFFFFFFFFFFEu, 0x7F800000u},
{"-1797693134862315609e290", 0xFFEFFFFFFFFFFFFFu, 0xFF800000u},
{"-17976931348623156083e289", 0xFFEFFFFFFFFFFFFEu, 0xFF800000u},
{"-0.17976931348623156084E+309", 0xFFEFFFFFFFFFFFFFu, 0xFF800000u},
{"0.179769313486231560835E+309", 0x7FEFFFFFFFFFFFFEu, 0x7F800000u},
{"-1.79769313486231560836e308", 0xFFEFFFFFFFFFFFFFu, 0xFF800000u},
{"1.79769313486231560835325876058e308", 0x7FEFFFFFFFFFFFFEu, 0x7F800000u},
{"179769313486231560835325876059e279", 0x7FEFFFFFFFFFFFFFu, 0x7F800000u},
{"1.7976931348623158e308", 0x7FEFFFFFFFFFFFFFu, 0x7F800000u},
{"17976931348623159e292", 0x7FF0000000000000u, 0x7F800000u},
{"1797693134862315807e290", 0x7FEFFFFFFFFFFFFFu, 0x7F800000u},
{"0.1797693134862315808E+309", 0x7FF0000000000000u, 0x7F800000u},
{"0.17976931348623158079E+309", 0x7FEFFFFFFFFFFFFFu, 0x7F800000u},
{"-1.797693134862315808e308", 0xFFF0000000000000u, 0xFF800000u},
{"1.79769313486231580793e308", 0x7FEFFFFFFFFFFFFFu, 0x7F800000u},
{"179769313486231580794e288", 0x7FF0000000000000u, 0x7F800000u},
{"179769313486231580793728971405e279", 0x7FEFFFFFFFFFFFFFu, 0x7F800000u},
{"-0.179769313486231580793728971406E+309", 0xFFF0000000000000u, 0xFF800000u},
{"100000000000000011102230246251565404236316680908203125e-53", 0x3FF0000000000000u, 0x3F800000u},
{"-1.00000000000000011102230246251565404236316680908203126", 0xBFF0000000000001u, 0xBF800000u},
{"1.00000000000000011102230246251565404236316680908203124e0", 0x3FF0000000000000u, 0x3F800000u},
{"10000000000000001110223024625156540423631668090820312501e-55", 0x3FF0000000000001u, 0x3F800000u},
{"1.00000000000000011102230246251565404236316680908203125000000000000000000001", 0x3FF0000000000001u, 0x3F800000u},
{"10000000000000001e-16", 0x3FF0000000000000u, 0x3F800000u},
{"1.0000000000000002", 0x3FF0000000000001u, 0x3F800000u},
{"1.000000000000000111", 0x3FF0000000000000u, 0x3F800000u},
{"1.000000000000000112e0", 0x3FF0000000000001u, 0x3F800000u},
{"1.000000000000000111e0", 0x3FF0000000000000u, 0x3F800000u},
{"-10000000000000001111e-19", 0xBFF0000000000001u, 0xBF800000u},
{"-100000000000000011102e-20", 0xBFF0000000000000u, 0xBF800000u},
{"-1.00000000000000011103", 0xBFF0000000000001u, 0xBF800000u},
{"1.00000000000000011102230246251", 0x3FF0000000000000u, 0x3F800000u},
{"1.00000000000000011102230246252e0", 0x3FF0000000000001u, 0x3F800000u},
{"-0.999999999999999944488848768742172978818416595458984375", 0xBFF0000000000000u, 0xBF800000u},
{"-9.99999999999999944488848768742172978818416595458984376e-1", 0xBFF0000000000000u, 0xBF800000u},
{"999999999999999944488848768742172978818416595458984374e-54", 0x3FEFFFFFFFFFFFFFu, 0x3F800000u},
{"0.99999999999999994448884876874217297881841659545898437501", 0x3FF0000000000000u, 0x3F800000u},
{"9.99999999999999944488848768742172978818416595458984375000000000000000000001e-1", 0x3FF0000000000000u, 0x3F800000u},
{"-0.99999999999999994", 0xBFEFFFFFFFFFFFFFu, 0xBF800000u},
{"9.9999999999999995e-1", 0x3FF0000000000000u, 0x3F800000u},
{"9.999999999999999444e-1", 0x3FEFFFFFFFFFFFFFu, 0x3F800000u},
{"9999999999999999445e-19", 0x3FF0000000000000u, 0x3F800000u},
{"99999999999999994448e-20", 0x3FEFFFFFFFFFFFFFu, 0x3F800000u},
{"-0.99999999999999994449", 0xBFF0000000000000u, 0xBF800000u},
{"0.999999999999999944488", 0x3FEFFFFFFFFFFFFFu, 0x3F800000u},
{"-9.99999999999999944489e-1", 0xBFF0000000000000u, 0xBF800000u},
{"9.99999999999999944488848768742e-1", 0x3FEFFFFFFFFFFFFFu, 0x3F800000u},
{"999999999999999944488848768743e-30", 0x3FF0000000000000u, 0x3F800000u},
{"-9.007199254740993e15", 0xC340000000000000u, 0xDA000000u},
{"9007199254740994e0", 0x4340000000000001u, 0x5A000000u},
{"9007199254740992", 0x4340000000000000u, 0x5A000000u},
{"9.00719925474099301e15", 0x4340000000000001u, 0x5A000000u},
{"9007199254740993000000000000000000001e-21", 0x4340000000000001u, 0x5A000000u},
{"-9007199254740993.000000000000000000000000000000", 0xC340000000000000u, 0xDA000000u},
{"90071992547409915e-1", 0x4340000000000000u, 0x5A000000u},
{"-9007199254740991.6", 0xC340000000000000u, 0xDA000000u},
{"9.0071992547409914e15", 0x433FFFFFFFFFFFFFu, 0x5A000000u},
{"-9007199254740991501e-3", 0xC340000000000000u, 0xDA000000u},
{"9007199254740991.5000000000000000000001", 0x4340000000000000u, 0x5A000000u},
{"9.0071992547409915000000000000000000000000000000e15", 0x4340000000000000u, 0x5A000000u},
{"0.100000000000000012490009027033011079765856266021728515625", 0x3FB999999999999Au, 0x3DCCCCCDu},
{"1.00000000000000012490009027033011079765856266021728515626e-1", 0x3FB999999999999Bu, 0x3DCCCCCDu},
{"100000000000000012490009027033011079765856266021728515624e-57", 0x3FB999999999999Au, 0x3DCCCCCDu},
{"0.10000000000000001249000902703301107976585626602172851562501", 0x3FB999999999999Bu, 0x3DCCCCCDu},
{"0.10000000000000001", 0x3FB999999999999Au, 0x3DCCCCCDu},
{"1.0000000000000002e-1", 0x3FB999999999999Bu, 0x3DCCCCCDu},
{"-1.000000000000000124e-1", 0xBFB999999999999Au, 0xBDCCCCCDu},
{"1000000000000000125e-19", 0x3FB999999999999Bu, 0x3DCCCCCDu},
{"-10000000000000001249e-20", 0xBFB999999999999Au, 0xBDCCCCCDu},
{"0.1000000000000000125", 0x3FB999999999999Bu, 0x3DCCCCCDu},
{"0.10000000000000001249", 0x3FB999999999999Au, 0x3DCCCCCDu},
{"1.00000000000000012491e-1", 0x3FB999999999999Bu, 0x3DCCCCCDu},
{"1.00000000000000012490009027033e-1", 0x3FB999999999999Au, 0x3DCCCCCDu},
{"100000000000000012490009027034e-30", 0x3FB999999999999Bu, 0x3DCCCCCDu},
{"2.45134755833537796875e14", 0x42EBDE5C4164D83Au, 0x575EF2E2u},
{"-245134755833537796876e-6", 0xC2EBDE5C4164D83Au, 0xD75EF2E2u},
{"-245134755833537.796874", 0xC2EBDE5C4164D839u, 0xD75EF2E2u},
{"2.4513475583353779687501e14", 0x42EBDE5C4164D83Au, 0x575EF2E2u},
{"245134755833537796875000000000000000000001e-27", 0x42EBDE5C4164D83Au, 0x575EF2E2u},
{"245134755833537.796875000000000000000000000000000000", 0x42EBDE5C4164D83Au, 0x575EF2E2u},
{"2.4513475583353779e14", 0x42EBDE5C4164D839u, 0x575EF2E2u},
{"2451347558335378e-1", 0x42EBDE5C4164D83Au, 0x575EF2E2u},
{"2451347558335377968e-4", 0x42EBDE5C4164D839u, 0x575EF2E2u},
{"245134755833537.7969", 0x42EBDE5C4164D83Au, 0x575EF2E2u},
{"245134755833537.79687", 0x42EBDE5C4164D839u, 0x575EF2E2u},
{"2.4513475583353779688e14", 0x42EBDE5C4164D83Au, 0x575EF2E2u},
{"181510327827821147441864013671875e-23", 0x41DB0C11CB91CE38u, 0x4ED8608Eu},
{"-1815103278.27821147441864013671876", 0xC1DB0C11CB91CE38u, 0xCED8608Eu},
{"1.81510327827821147441864013671874e9", 0x41DB0C11CB91CE37u, 0x4ED8608Eu},
{"18151032782782114744186401367187501e-25", 0x41DB0C11CB91CE38u, 0x4ED8608Eu},
{"1815103278.27821147441864013671875000000000000000000001", 0x41DB0C11CB91CE38u, 0x4ED8608Eu},
{"-1.81510327827821147441864013671875000000000000000000000000000000e9", 0xC1DB0C11CB91CE38u, 0xCED8608Eu},
{"18151032782782114e-7", 0x41DB0C11CB91CE37u, 0x4ED8608Eu},
{"1815103278.2782115", 0x41DB0C11CB91CE38u, 0x4ED8608Eu},
{"1815103278.278211474", 0x41DB0C11CB91CE37u, 0x4ED8608Eu},
{"-1.815103278278211475e9", 0xC1DB0C11CB91CE38u, 0xCED8608Eu},
{"1.8151032782782114744e9", 0x41DB0C11CB91CE37u, 0x4ED8608Eu},
{"-18151032782782114745e-10", 0xC1DB0C11CB91CE38u, 0xCED8608Eu},
{"181510327827821147441e-11", 0x41DB0C11CB91CE37u, 0x4ED8608Eu},
{"1815103278.27821147442", 0x41DB0C11CB91CE38u, 0x4ED8608Eu},
{"1815103278.27821147441864013671", 0x41DB0C11CB91CE37u, 0x4ED8608Eu},
{"1.81510327827821147441864013672e9", 0x41DB0C11CB91CE38u, 0x4ED8608Eu},
{"3809325632181785344", 0x43CA6EB8BD69FE2Au, 0x5E5375C6u},
{"3.809325632181785345e18", 0x43CA6EB8BD69FE2Au, 0x5E5375C6u},
{"3809325632181785343e0", 0x43CA6EB8BD69FE29u, 0x5E5375C6u},
{"3809325632181785344.01", 0x43CA6EB8BD69FE2Au, 0x5E5375C6u},
{"3.809325632181785344000000000000000000001e18", 0x43CA6EB8BD69FE2Au, 0x5E5375C6u},
{"3809325632181785344000000000000000000000000000000e-30", 0x43CA6EB8BD69FE2Au, 0x5E5375C6u},
{"3809325632181785300", 0x43CA6EB8BD69FE29u, 0x5E5375C6u},
{"3.8093256321817854e18", 0x43CA6EB8BD69FE2Au, 0x5E5375C6u},
{"4.046966549916366943359375e12", 0x428D7210076CE2F0u, 0x546B9080u},
{"4046966549916366943359376e-12", 0x428D7210076CE2F0u, 0x546B9080u},
{"4046966549916.366943359374", 0x428D7210076CE2EFu, 0x546B9080u},
{"4.04696654991636694335937501e12", 0x428D7210076CE2F0u, 0x546B9080u},
{"4046966549916366943359375000000000000000000001e-33", 0x428D7210076CE2F0u, 0x546B9080u},
{"-4046966549916.366943359375000000000000000000000000000000", 0xC28D7210076CE2F0u, 0xD46B9080u},
{"4.0469665499163669e12", 0x428D7210076CE2EFu, 0x546B9080u},
{"4046966549916367e-3", 0x428D7210076CE2F0u, 0x546B9080u},
{"4046966549916366943e-6", 0x428D7210076CE2EFu, 0x546B9080u},
{"4046966549916.366944", 0x428D7210076CE2F0u, 0x546B9080u},
{"4046966549916.3669433", 0x428D7210076CE2EFu, 0x546B9080u},
{"4.0469665499163669434e12", 0x428D7210076CE2F0u, 0x546B9080u},
{"-4.04696654991636694335e12", 0xC28D7210076CE2EFu, 0xD46B9080u},
{"404696654991636694336e-8", 0x428D7210076CE2F0u, 0x546B9080u},
{"28093802557000874e154", 0x63529C3B77330BDBu, 0x7F800000u},
{"0.28093802557000875E+171", 0x63529C3B77330BDCu, 0x7F800000u},
{"0.2809380255700087447E+171", 0x63529C3B77330BDBu, 0x7F800000u},
{"2.809380255700087448e170", 0x63529C3B77330BDCu, 0x7F800000u},
{"2.8093802557000874472e170", 0x63529C3B77330BDBu, 0x7F800000u},
{"28093802557000874473e151", 0x63529C3B77330BDCu, 0x7F800000u},
{"280938025570008744728e150", 0x63529C3B77330BDBu, 0x7F800000u},
{"-0.280938025570008744729E+171", 0xE3529C3B77330BDCu, 0xFF800000u},
{"-0.280938025570008744728403667979E+171", 0xE3529C3B77330BDBu, 0xFF800000u},
{"2.8093802557000874472840366798e170", 0x63529C3B77330BDCu, 0x7F800000u},
{"0.39523280297734525e-154", 0x1FE0F51BF17FD374u, 0x00000000u},
{"-3.9523280297734526e-155", 0x9FE0F51BF17FD375u, 0x80000000u},
{"3.952328029773452547e-155", 0x1FE0F51BF17FD374u, 0x00000000u},
{"3952328029773452548e-173", 0x1FE0F51BF17FD375u, 0x00000000u},
{"-39523280297734525478e-174", 0x9FE0F51BF17FD374u, 0x80000000u},
{"0.39523280297734525479e-154", 0x1FE0F51BF17FD375u, 0x00000000u},
{"-0.395232802977345254787e-154", 0x9FE0F51BF17FD374u, 0x80000000u},
{"-3.95232802977345254788e-155", 0x9FE0F51BF17FD375u, 0x80000000u},
{"-3.95232802977345254787245825501e-155", 0x9FE0F51BF17FD374u, 0x80000000u},
{"395232802977345254787245825502e-184", 0x1FE0F51BF17FD375u, 0x00000000u},
{"-1.0790205420931879e-276", 0x86A3209CA6233255u, 0x80000000u},
{"-1079020542093188e-291", 0x86A3209CA6233256u, 0x80000000u},
{"-1079020542093187947e-294", 0x86A3209CA6233255u, 0x80000000u},
{"0.1079020542093187948e-275", 0x06A3209CA6233256u, 0x00000000u},
{"0.1079020542093187947e-275", 0x06A3209CA6233255u, 0x00000000u},
{"1.0790205420931879471e-276", 0x06A3209CA6233256u, 0x00000000u},
{"1.07902054209318794701e-276", 0x06A3209CA6233255u, 0x00000000u},
{"-107902054209318794702e-296", 0x86A3209CA6233256u, 0x80000000u},
{"107902054209318794701153285302e-305", 0x06A3209CA6233255u, 0x00000000u},
{"-0.107902054209318794701153285303e-275", 0x86A3209CA6233256u, 0x80000000u},
{"58530471071351308e-228", 0x1413B446E6A16A3Bu, 0x00000000u},
{"-0.58530471071351309e-211", 0x9413B446E6A16A3Cu, 0x80000000u},
{"0.5853047107135130893e-211", 0x1413B446E6A16A3Bu, 0x00000000u},
{"-5.853047107135130894e-212", 0x9413B446E6A16A3Cu, 0x80000000u},
{"5.853047107135130893e-212", 0x1413B446E6A16A3Bu, 0x00000000u},
{"58530471071351308931e-231", 0x1413B446E6A16A3Cu, 0x00000000u},
{"-585304710713513089304e-232", 0x9413B446E6A16A3Bu, 0x80000000u},
{"0.585304710713513089305e-211", 0x1413B446E6A16A3Cu, 0x00000000u},
{"-0.585304710713513089304248824438e-211", 0x9413B446E6A16A3Bu, 0x80000000u},
{"5.85304710713513089304248824439e-212", 0x1413B446E6A16A3Cu, 0x00000000u},
{"0.19334214893983531e-78", 0x2F96ECBF1CFB10F6u, 0x00000000u},
{"1.9334214893983532e-79", 0x2F96ECBF1CFB10F7u, 0x00000000u},
{"1.933421489398353102e-79", 0x2F96ECBF1CFB10F6u, 0x00000000u},
{"1933421489398353103e-97", 0x2F96ECBF1CFB10F7u, 0x00000000u},
{"19334214893983531023e-98", 0x2F96ECBF1CFB10F6u, 0x00000000u},
{"0.19334214893983531024e-78", 0x2F96ECBF1CFB10F7u, 0x00000000u},
{"0.193342148939835310231e-78", 0x2F96ECBF1CFB10F6u, 0x00000000u},
{"1.93342148939835310232e-79", 0x2F96ECBF1CFB10F7u, 0x00000000u},
{"-1.93342148939835310231359014704e-79", 0xAF96ECBF1CFB10F6u, 0x80000000u},
{"193342148939835310231359014705e-108", 0x2F96ECBF1CFB10F7u, 0x00000000u},
{"2.9873358928024455e227", 0x6F2938807814E8A2u, 0x7F800000u},
{"29873358928024456e211", 0x6F2938807814E8A3u, 0x7F800000u},
{"298733589280244551e210", 0x6F2938807814E8A2u, 0x7F800000u},
{"0.2987335892802445511E+228", 0x6F2938807814E8A3u, 0x7F800000u},
{"0.29873358928024455109E+228", 0x6F2938807814E8A2u, 0x7F800000u},
{"2.987335892802445511e227", 0x6F2938807814E8A3u, 0x7F800000u},
{"-2.98733589280244551098e227", 0xEF2938807814E8A2u, 0xFF800000u},
{"-298733589280244551099e207", 0xEF2938807814E8A3u, 0xFF800000u},
{"298733589280244551098081559931e198", 0x6F2938807814E8A2u, 0x7F800000u},
{"0.298733589280244551098081559932E+228", 0x6F2938807814E8A3u, 0x7F800000u},
{"7.0064923216240853e-46", 0x3690000000000000u, 0x00000000u},
{"70064923216240854e-62", 0x3690000000000000u, 0x00000001u},
{"7006492321624085354e-64", 0x3690000000000000u, 0x00000000u},
{"-0.7006492321624085355e-45", 0xB690000000000000u, 0x80000001u},
{"0.70064923216240853546e-45", 0x3690000000000000u, 0x00000000u},
{"7.0064923216240853547e-46", 0x3690000000000000u, 0x00000001u},
{"7.00649232162408535461e-46", 0x3690000000000000u, 0x00000000u},
{"-700649232162408535462e-66", 0xB690000000000000u, 0x80000001u},
{"-700649232162408535461864791644e-75", 0xB690000000000000u, 0x80000000u},
{"0.700649232162408535461864791645e-45", 0x3690000000000000u, 0x00000001u},
{"21019476964872256e-61", 0x36A8000000000000u, 0x00000001u},
{"0.21019476964872257e-44", 0x36A8000000000000u, 0x00000002u},
{"-0.2101947696487225606e-44", 0xB6A8000000000000u, 0x80000001u},
{"-2.101947696487225607e-45", 0xB6A8000000000000u, 0x80000002u},
{"2.1019476964872256063e-45", 0x36A8000000000000u, 0x00000001u},
{"21019476964872256064e-64", 0x36A8000000000000u, 0x00000002u},
{"-210194769648722560638e-65", 0xB6A8000000000000u, 0x80000001u},
{"-0.210194769648722560639e-44", 0xB6A8000000000000u, 0x80000002u},
{"-0.210194769648722560638559437493e-44", 0xB6A8000000000000u, 0x80000001u},
{"2.10194769648722560638559437494e-45", 0x36A8000000000000u, 0x00000002u},
{"0.11754941406275178e-37", 0x380FFFFFA0000000u, 0x007FFFFEu},
{"1.1754941406275179e-38", 0x380FFFFFA0000000u, 0x007FFFFFu},
{"-1.175494140627517859e-38", 0xB80FFFFFA0000000u, 0x807FFFFEu},
{"117549414062751786e-55", 0x380FFFFFA0000000u, 0x007FFFFFu},
{"-11754941406275178592e-57", 0xB80FFFFFA0000000u, 0x807FFFFEu},
{"0.11754941406275178593e-37", 0x380FFFFFA0000000u, 0x007FFFFFu},
{"0.117549414062751785924e-37", 0x380FFFFFA0000000u, 0x007FFFFEu},
{"1.17549414062751785925e-38", 0x380FFFFFA0000000u, 0x007FFFFFu},
{"-1.17549414062751785924617589866e-38", 0xB80FFFFFA0000000u, 0x807FFFFEu},
{"117549414062751785924617589867e-67", 0x380FFFFFA0000000u, 0x007FFFFFu},
{"1.1754942807573642e-38", 0x380FFFFFDFFFFFFFu, 0x007FFFFFu},
{"11754942807573643e-54", 0x380FFFFFE0000000u, 0x00800000u},
{"1175494280757364291e-56", 0x380FFFFFE0000000u, 0x007FFFFFu},
{"0.1175494280757364292e-37", 0x380FFFFFE0000000u, 0x00800000u},
{"0.11754942807573642917e-37", 0x380FFFFFE0000000u, 0x007FFFFFu},
{"-1.1754942807573642918e-38", 0xB80FFFFFE0000000u, 0x80800000u},
{"-1.17549428075736429172e-38", 0xB80FFFFFE0000000u, 0x807FFFFFu},
{"-117549428075736429173e-58", 0xB80FFFFFE0000000u, 0x80800000u},
{"117549428075736429172788299103e-67", 0x380FFFFFE0000000u, 0x007FFFFFu},
{"0.117549428075736429172788299104e-37", 0x380FFFFFE0000000u, 0x00800000u},
{"11754944208872107e-54", 0x3810000010000000u, 0x00800000u},
{"-0.11754944208872108e-37", 0xB810000010000000u, 0x80800001u},
{"0.1175494420887210724e-37", 0x3810000010000000u, 0x00800000u},
{"1.175494420887210725e-38", 0x3810000010000000u, 0x00800001u},
{"1.1754944208872107242e-38", 0x3810000010000000u, 0x00800000u},
{"-11754944208872107243e-57", 0xB810000010000000u, 0x80800001u},
{"-11754944208872107242e-57", 0xB810000010000000u, 0x80800000u},
{"0.117549442088721072421e-37", 0x3810000010000000u, 0x00800001u},
{"0.11754944208872107242095900834e-37", 0x3810000010000000u, 0x00800000u},
{"-1.17549442088721072420959008341e-38", 0xB810000010000000u, 0x80800001u},
{"340282336497324057985868971510891282432", 0x47EFFFFFD0000000u, 0x7F7FFFFEu},
{"-3.40282336497324057985868971510891282433e38", 0xC7EFFFFFD0000000u, 0xFF7FFFFFu},
{"340282336497324057985868971510891282431e0", 0x47EFFFFFD0000000u, 0x7F7FFFFEu},
{"340282336497324057985868971510891282432.01", 0x47EFFFFFD0000000u, 0x7F7FFFFFu},
{"3.40282336497324057985868971510891282432000000000000000000001e38", 0x47EFFFFFD0000000u, 0x7F7FFFFFu},
{"340282336497324057985868971510891282432000000000000000000000000000000e-30", 0x47EFFFFFD0000000u, 0x7F7FFFFEu},
{"340282336497324050000000000000000000000", 0x47EFFFFFD0000000u, 0x7F7FFFFEu},
{"3.4028233649732406e38", 0x47EFFFFFD0000000u, 0x7F7FFFFFu},
{"3.402823364973240579e38", 0x47EFFFFFD0000000u, 0x7F7FFFFEu},
{"340282336497324058e21", 0x47EFFFFFD0000000u, 0x7F7FFFFFu},
{"34028233649732405798e19", 0x47EFFFFFD0000000u, 0x7F7FFFFEu},
{"340282336497324057990000000000000000000", 0x47EFFFFFD0000000u, 0x7F7FFFFFu},
{"340282336497324057985000000000000000000", 0x47EFFFFFD0000000u, 0x7F7FFFFEu},
{"3.40282336497324057986e38", 0x47EFFFFFD0000000u, 0x7F7FFFFFu},
{"3.4028233649732405798586897151e38", 0x47EFFFFFD0000000u, 0x7F7FFFFEu},
{"340282336497324057985868971511e9", 0x47EFFFFFD0000000u, 0x7F7FFFFFu},
{"3.40282356779733661637539395458142568448e38", 0x47EFFFFFF0000000u, 0x7F800000u},
{"-340282356779733661637539395458142568449e0", 0xC7EFFFFFF0000000u, 0xFF800000u},
{"340282356779733661637539395458142568447", 0x47EFFFFFF0000000u, 0x7F7FFFFFu},
{"3.4028235677973366163753939545814256844801e38", 0x47EFFFFFF0000000u, 0x7F800000u},
{"340282356779733661637539395458142568448000000000000000000001e-21", 0x47EFFFFFF0000000u, 0x7F800000u},
{"340282356779733661637539395458142568448.000000000000000000000000000000", 0x47EFFFFFF0000000u, 0x7F800000u},
{"3.4028235677973366e38", 0x47EFFFFFF0000000u, 0x7F7FFFFFu},
{"-34028235677973367e22", 0xC7EFFFFFF0000000u, 0xFF800000u},
{"3402823567797336616e20", 0x47EFFFFFF0000000u, 0x7F7FFFFFu},
{"340282356779733661700000000000000000000", 0x47EFFFFFF0000000u, 0x7F800000u},
{"340282356779733661630000000000000000000", 0x47EFFFFFF0000000u, 0x7F7FFFFFu},
{"3.4028235677973366164e38", 0x47EFFFFFF0000000u, 0x7F800000u},
{"3.40282356779733661637e38", 0x47EFFFFFF0000000u, 0x7F7FFFFFu},
{"340282356779733661638e18", 0x47EFFFFFF0000000u, 0x7F800000u},
{"-340282356779733661637539395458e9", 0xC7EFFFFFF0000000u, 0xFF7FFFFFu},
{"340282356779733661637539395459000000000", 0x47EFFFFFF0000000u, 0x7F800000u},
{"-1000000059604644775390625e-24", 0xBFF0000010000000u, 0xBF800000u},
{"-1.000000059604644775390626", 0xBFF0000010000000u, 0xBF800001u},
{"-1.000000059604644775390624e0", 0xBFF0000010000000u, 0xBF800000u},
{"100000005960464477539062501e-26", 0x3FF0000010000000u, 0x3F800001u},
{"1.000000059604644775390625000000000000000000001", 0x3FF0000010000000u, 0x3F800001u},
{"1.000000059604644775390625000000000000000000000000000000e0", 0x3FF0000010000000u, 0x3F800000u},
{"-10000000596046447e-16", 0xBFF0000010000000u, 0xBF800000u},
{"1.0000000596046448", 0x3FF0000010000000u, 0x3F800001u},
{"1.000000059604644775", 0x3FF0000010000000u, 0x3F800000u},
{"-1.000000059604644776e0", 0xBFF0000010000000u, 0xBF800001u},
{"-1.0000000596046447753e0", 0xBFF0000010000000u, 0xBF800000u},
{"10000000596046447754e-19", 0x3FF0000010000000u, 0x3F800001u},
{"100000005960464477539e-20", 0x3FF0000010000000u, 0x3F800000u},
{"-1.0000000596046447754", 0xBFF0000010000000u, 0xBF800001u},
{"0.9999999701976776123046875", 0x3FEFFFFFF0000000u, 0x3F800000u},
{"9.999999701976776123046876e-1", 0x3FEFFFFFF0000000u, 0x3F800000u},
{"9999999701976776123046874e-25", 0x3FEFFFFFF0000000u, 0x3F7FFFFFu},
{"0.999999970197677612304687501", 0x3FEFFFFFF0000000u, 0x3F800000u},
{"-9.999999701976776123046875000000000000000000001e-1", 0xBFEFFFFFF0000000u, 0xBF800000u},
{"-9999999701976776123046875000000000000000000000000000000e-55", 0xBFEFFFFFF0000000u, 0xBF800000u},
{"0.99999997019767761", 0x3FEFFFFFF0000000u, 0x3F7FFFFFu},
{"-9.9999997019767762e-1", 0xBFEFFFFFF0000000u, 0xBF800000u},
{"-9.999999701976776123e-1", 0xBFEFFFFFF0000000u, 0xBF7FFFFFu},
{"9999999701976776124e-19", 0x3FEFFFFFF0000000u, 0x3F800000u},
{"9999999701976776123e-19", 0x3FEFFFFFF0000000u, 0x3F7FFFFFu},
{"0.99999997019767761231", 0x3FEFFFFFF0000000u, 0x3F800000u},
{"-0.999999970197677612304", 0xBFEFFFFFF0000000u, 0xBF7FFFFFu},
{"9.99999970197677612305e-1", 0x3FEFFFFFF0000000u, 0x3F800000u},
{"-1.6777217e7", 0xC170000010000000u, 0xCB800000u},
{"16777218e0", 0x4170000020000000u, 0x4B800001u},
{"16777216", 0x4170000000000000u, 0x4B800000u},
{"-1.677721701e7", 0xC17000001028F5C3u, 0xCB800001u},
{"-16777217000000000000000000001e-21", 0xC170000010000000u, 0xCB800001u},
{"16777217.000000000000000000000000000000", 0x4170000010000000u, 0x4B800000u},
{"167772155e-1", 0x416FFFFFF0000000u, 0x4B800000u},
{"16777215.6", 0x416FFFFFF3333333u, 0x4B800000u},
{"1.67772154e7", 0x416FFFFFECCCCCCDu, 0x4B7FFFFFu},
{"16777215501e-3", 0x416FFFFFF0083127u, 0x4B800000u},
{"-16777215.5000000000000000000001", 0xC16FFFFFF0000000u, 0xCB800000u},
{"-1.67772155000000000000000000000000000000e7", 0xC16FFFFFF0000000u, 0xCB800000u},
{"0.1000000052154064178466796875", 0x3FB99999B0000000u, 0x3DCCCCCEu},
{"1.000000052154064178466796876e-1", 0x3FB99999B0000000u, 0x3DCCCCCEu},
{"-1000000052154064178466796874e-28", 0xBFB99999B0000000u, 0xBDCCCCCDu},
{"0.100000005215406417846679687501", 0x3FB99999B0000000u, 0x3DCCCCCEu},
{"1.000000052154064178466796875000000000000000000001e-1", 0x3FB99999B0000000u, 0x3DCCCCCEu},
{"-1000000052154064178466796875000000000000000000000000000000e-58", 0xBFB99999B0000000u, 0xBDCCCCCEu},
{"-0.10000000521540641", 0xBFB99999AFFFFFFFu, 0xBDCCCCCDu},
{"-1.0000000521540642e-1", 0xBFB99999B0000000u, 0xBDCCCCCEu},
{"1.000000052154064178e-1", 0x3FB99999B0000000u, 0x3DCCCCCDu},
{"-1000000052154064179e-19", 0xBFB99999B0000000u, 0xBDCCCCCEu},
{"10000000521540641784e-20", 0x3FB99999B0000000u, 0x3DCCCCCDu},
{"0.10000000521540641785", 0x3FB99999B0000000u, 0x3DCCCCCEu},
{"0.100000005215406417846", 0x3FB99999B0000000u, 0x3DCCCCCDu},
{"1.00000005215406417847e-1", 0x3FB99999B0000000u, 0x3DCCCCCEu},
{"5.429001220703125e3", 0x40B5350050000000u, 0x45A9A802u},
{"-5429001220703126e-12", 0xC0B5350050000001u, 0xC5A9A803u},
{"-5429.001220703124", 0xC0B535004FFFFFFFu, 0xC5A9A802u},
{"5.42900122070312501e3", 0x40B5350050000000u, 0x45A9A803u},
{"5429001220703125000000000000000000001e-33", 0x40B5350050000000u, 0x45A9A803u},
{"-5429.001220703125000000000000000000000000000000", 0xC0B5350050000000u, 0xC5A9A802u},
{"503719056e0", 0x41BE062490000000u, 0x4DF03124u},
{"503719057", 0x41BE062491000000u, 0x4DF03125u},
{"5.03719055e8", 0x41BE06248F000000u, 0x4DF03124u},
{"50371905601e-2", 0x41BE062490028F5Cu, 0x4DF03125u},
{"503719056.000000000000000000001", 0x41BE062490000000u, 0x4DF03125u},
{"5.03719056000000000000000000000000000000e8", 0x41BE062490000000u, 0x4DF03124u},
{"-92331620", 0xC196037990000000u, 0xCCB01BCCu},
{"9.233163e7", 0x41960379B8000000u, 0x4CB01BCEu},
{"9233161e1", 0x4196037968000000u, 0x4CB01BCBu},
{"92331620.1", 0x4196037990666666u, 0x4CB01BCDu},
{"9.233162000000000000000000001e7", 0x4196037990000000u, 0x4CB01BCDu},
{"9233162000000000000000000000000000000e-29", 0x4196037990000000u, 0x4CB01BCCu},
{"3.002458625e6", 0x4146E82D50000000u, 0x4A37416Au},
{"3002458626e-3", 0x4146E82D5020C49Cu, 0x4A37416Bu},
{"3002458.624", 0x4146E82D4FDF3B64u, 0x4A37416Au},
{"-3.00245862501e6", 0xC146E82D500053E3u, 0xCA37416Bu},
{"3002458625000000000000000000001e-24", 0x4146E82D50000000u, 0x4A37416Bu},
{"3002458.625000000000000000000000000000000", 0x4146E82D50000000u, 0x4A37416Au},
{"-1095485584696182596504479582065262592e1", 0xC7A07BA830000000u, 0xFD03DD42u},
{"10954855846961825965044795820652625930", 0x47A07BA830000000u, 0x7D03DD42u},
{"1.095485584696182596504479582065262591e37", 0x47A07BA830000000u, 0x7D03DD41u},
{"109548558469618259650447958206526259201e-1", 0x47A07BA830000000u, 0x7D03DD42u},
{"-10954855846961825965044795820652625920.00000000000000000001", 0xC7A07BA830000000u, 0xFD03DD42u},
{"1.095485584696182596504479582065262592000000000000000000000000000000e37", 0x47A07BA830000000u, 0x7D03DD42u},
{"10954855846961825e21", 0x47A07BA830000000u, 0x7D03DD41u},
{"10954855846961826000000000000000000000", 0x47A07BA830000000u, 0x7D03DD42u},
{"10954855846961825960000000000000000000", 0x47A07BA830000000u, 0x7D03DD41u},
{"1.095485584696182597e37", 0x47A07BA830000000u, 0x7D03DD42u},
{"1.0954855846961825965e37", 0x47A07BA830000000u, 0x7D03DD41u},
{"-10954855846961825966e18", 0xC7A07BA830000000u, 0xFD03DD42u},
{"10954855846961825965e18", 0x47A07BA830000000u, 0x7D03DD41u},
{"10954855846961825965100000000000000000", 0x47A07BA830000000u, 0x7D03DD42u},
{"10954855846961825965044795820600000000", 0x47A07BA830000000u, 0x7D03DD41u},
{"-1.09548558469618259650447958207e37", 0xC7A07BA830000000u, 0xFD03DD42u},
{"1.6449216019182103706535606608388384863861375606575165875256061553955078126e-21", 0x3B9F125A50000000u, 0x1CF892D3u},
{"-16449216019182103706535606608388384863861375606575165875256061553955078124e-94", 0xBB9F125A50000000u, 0x9CF892D2u},
{"-0.0000000000000000000016449216019182103", 0xBB9F125A50000000u, 0x9CF892D2u},
{"-1.6449216019182104e-21", 0xBB9F125A50000000u, 0x9CF892D3u},
{"-1.64492160191821037e-21", 0xBB9F125A50000000u, 0x9CF892D2u},
{"-1644921601918210371e-39", 0xBB9F125A50000000u, 0x9CF892D3u},
{"-16449216019182103706e-40", 0xBB9F125A50000000u, 0x9CF892D2u},
{"0.0000000000000000000016449216019182103707", 0x3B9F125A50000000u, 0x1CF892D3u},
{"0.00000000000000000000164492160191821037065", 0x3B9F125A50000000u, 0x1CF892D2u},
{"1.64492160191821037066e-21", 0x3B9F125A50000000u, 0x1CF892D3u},
{"1.64492160191821037065356066083e-21", 0x3B9F125A50000000u, 0x1CF892D2u},
{"164492160191821037065356066084e-50", 0x3B9F125A50000000u, 0x1CF892D3u},
{"6.565061509609222412109375e-1", 0x3FE5021930000000u, 0x3F2810CAu},
{"6565061509609222412109376e-25", 0x3FE5021930000000u, 0x3F2810CAu},
{"0.6565061509609222412109374", 0x3FE5021930000000u, 0x3F2810C9u},
{"6.56506150960922241210937501e-1", 0x3FE5021930000000u, 0x3F2810CAu},
{"6565061509609222412109375000000000000000000001e-46", 0x3FE5021930000000u, 0x3F2810CAu},
{"-0.6565061509609222412109375000000000000000000000000000000", 0xBFE5021930000000u, 0xBF2810CAu},
{"6.5650615096092224e-1", 0x3FE5021930000000u, 0x3F2810C9u},
{"-65650615096092225e-17", 0xBFE5021930000000u, 0xBF2810CAu},
{"6565061509609222412e-19", 0x3FE5021930000000u, 0x3F2810C9u},
{"0.6565061509609222413", 0x3FE5021930000000u, 0x3F2810CAu},
{"0.65650615096092224121", 0x3FE5021930000000u, 0x3F2810C9u},
{"6.5650615096092224122e-1", 0x3FE5021930000000u, 0x3F2810CAu},
{"-6.5650615096092224121e-1", 0xBFE5021930000000u, 0xBF2810C9u},
{"656506150960922241211e-21", 0x3FE5021930000000u, 0x3F2810CAu},
{"18014627239033005156980393746124491372029297053813934326171875e-77", 0x3CA9F63970000000u, 0x254FB1CCu},
{"0.00000000000000018014627239033005156980393746124491372029297053813934326171876", 0x3CA9F63970000000u, 0x254FB1CCu},
{"-1.8014627239033005156980393746124491372029297053813934326171874e-16", 0xBCA9F63970000000u, 0xA54FB1CBu},
{"1801462723903300515698039374612449137202929705381393432617187501e-79", 0x3CA9F63970000000u, 0x254FB1CCu},
{"18014627239033005e-32", 0x3CA9F63970000000u, 0x254FB1CBu},
{"0.00000000000000018014627239033006", 0x3CA9F63970000000u, 0x254FB1CCu},
{"0.0000000000000001801462723903300515", 0x3CA9F63970000000u, 0x254FB1CBu},
{"-1.801462723903300516e-16", 0xBCA9F63970000000u, 0xA54FB1CCu},
{"-1.8014627239033005156e-16", 0xBCA9F63970000000u, 0xA54FB1CBu},
{"18014627239033005157e-35", 0x3CA9F63970000000u, 0x254FB1CCu},
{"180146272390330051569e-36", 0x3CA9F63970000000u, 0x254FB1CBu},
{"-0.00000000000000018014627239033005157", 0xBCA9F63970000000u, 0xA54FB1CCu},
{"0.000000000000000180146272390330051569803937461", 0x3CA9F63970000000u, 0x254FB1CBu},
{"1.80146272390330051569803937462e-16", 0x3CA9F63970000000u, 0x254FB1CCu},
{"0.05534819327294826507568359375", 0x3FAC569930000000u, 0x3D62B4CAu},
{"-5.534819327294826507568359376e-2", 0xBFAC569930000000u, 0xBD62B4CAu},
{"5534819327294826507568359374e-29", 0x3FAC569930000000u, 0x3D62B4C9u},
{"-0.0553481932729482650756835937501", 0xBFAC569930000000u, 0xBD62B4CAu},
{"5.534819327294826507568359375000000000000000000001e-2", 0x3FAC569930000000u, 0x3D62B4CAu},
{"5534819327294826507568359375000000000000000000000000000000e-59", 0x3FAC569930000000u, 0x3D62B4CAu},
{"0.055348193272948265", 0x3FAC569930000000u, 0x3D62B4C9u},
{"5.5348193272948266e-2", 0x3FAC569930000000u, 0x3D62B4CAu},
{"5.534819327294826507e-2", 0x3FAC569930000000u, 0x3D62B4C9u},
{"5534819327294826508e-20", 0x3FAC569930000000u, 0x3D62B4CAu},
{"55348193272948265075e-21", 0x3FAC569930000000u, 0x3D62B4C9u},
{"-0.055348193272948265076", 0xBFAC569930000000u, 0xBD62B4CAu},
{"0.0553481932729482650756", 0x3FAC569930000000u, 0x3D62B4C9u},
{"-5.53481932729482650757e-2", 0xBFAC569930000000u, 0xBD62B4CAu},
{"5.179692133247783258005389047985340416e36", 0x478F2C9450000000u, 0x7C7964A2u},
{"5179692133247783258005389047985340417e0", 0x478F2C9450000000u, 0x7C7964A3u},
{"5179692133247783258005389047985340415", 0x478F2C9450000000u, 0x7C7964A2u},
{"-5.17969213324778325800538904798534041601e36", 0xC78F2C9450000000u, 0xFC7964A3u},
{"-5179692133247783258005389047985340416000000000000000000001e-21", 0xC78F2C9450000000u, 0xFC7964A3u},
{"-5179692133247783258005389047985340416.000000000000000000000000000000", 0xC78F2C9450000000u, 0xFC7964A2u},
{"5.1796921332477832e36", 0x478F2C9450000000u, 0x7C7964A2u},
{"51796921332477833e20", 0x478F2C9450000000u, 0x7C7964A3u},
{"-5179692133247783258e18", 0xC78F2C9450000000u, 0xFC7964A2u},
{"5179692133247783259000000000000000000", 0x478F2C9450000000u, 0x7C7964A3u},
{"5179692133247783258000000000000000000", 0x478F2C9450000000u, 0x7C7964A2u},
{"5.1796921332477832581e36", 0x478F2C9450000000u, 0x7C7964A3u},
{"-5.179692133247783258e36", 0xC78F2C9450000000u, 0xFC7964A2u},
{"-517969213324778325801e16", 0xC78F2C9450000000u, 0xFC7964A3u},
{"517969213324778325800538904798e7", 0x478F2C9450000000u, 0x7C7964A2u},
{"5179692133247783258005389047990000000", 0x478F2C9450000000u, 0x7C7964A3u},
{
"0.22250738585072011360574097967091319759348195463516456480234261097248222220210769455165295239081350"
"8791414915891303962110687008643869459464552765720740782062174337998814106326732925355228688137214901"
"2981122451451889849057222307285255133155755015914397476397983411801999323962548289017107081850690630"
"6666559949382757725720157630626906633326475653000092458883164330377797918696120494973903778297049050"
"5108060994073026293712895895000358379996720725430436028407889577179615094551674824347103070260914462"
"1572289880258182545180325707018860872113128079512233426288368622321503775666622503982534335974568884"
"4239002654981983854879482922068947216898310996983658468140228542433306603398508864458040010349339704"
"2756718644338377048603786162277173854562306587467901408672332763671875e-307", 0x0010000000000000u, 0x00000000u
},
{
"2.22507385850720113605740979670913197593481954635164564802342610972482222202107694551652952390813508"
"7914149158913039621106870086438694594645527657207407820621743379988141063267329253552286881372149012"
"9811224514518898490572223072852551331557550159143974763979834118019993239625482890171070818506906306"
"6665599493827577257201576306269066333264756530000924588831643303777979186961204949739037782970490505"
"1080609940730262937128958950003583799967207254304360284078895771796150945516748243471030702609144621"
"5722898802581825451803257070188608721131280795122334262883686223215037756666225039825343359745688844"
"2390026549819838548794829220689472168983109969836584681402285424333066033985088644580400103493397042"
"756718644338377048603786162277173854562306587467901408672332763671875000000000000000000001e-308", 0x0010000000000000u, 0x00000000u
},
{
"0.11754942807573642917278829910357665133228589927589904276829631184250030649651730385585324256680905"
"8189392089843750000000000000000000000000000000000000000000000000000000000000000000000000000000000000"
"0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"
"0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"
"0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"
"0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"
"0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"
"00000000000000000000000000000000000000000000000000000000000000000e-37", 0x380FFFFFE0000000u, 0x00800000u
},
{
"1175494280757364291727882991035766513322858992758990427682963118425003064965173038558532425668090581"
"8939208984375000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"
"0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"
"0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"
"0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"
"0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"
"0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000"
"00000000000001e-751", 0x380FFFFFE0000000u, 0x00800000u
},
{"0", 0x0000000000000000u, 0x00000000u},
{"-0", 0x8000000000000000u, 0x80000000u},
{"0.0", 0x0000000000000000u, 0x00000000u},
{"-0.0", 0x8000000000000000u, 0x80000000u},
{"0e999999999999999999999", 0x0000000000000000u, 0x00000000u},
{"-0.000e-99999", 0x8000000000000000u, 0x80000000u},
{"1e-400", 0x0000000000000000u, 0x00000000u},
{"-1e-400", 0x8000000000000000u, 0x80000000u},
{"1e400", 0x7FF0000000000000u, 0x7F800000u},
{"-1e400", 0xFFF0000000000000u, 0xFF800000u},
{"1e-50", 0x358DEE7A4AD4B81Fu, 0x00000000u},
{"-1e-50", 0xB58DEE7A4AD4B81Fu, 0x80000000u},
{"1e39", 0x48078287F49C4A1Du, 0x7F800000u},
{"-1e39", 0xC8078287F49C4A1Du, 0xFF800000u},
{"1e99999999999999999999999999", 0x7FF0000000000000u, 0x7F800000u},
{"1e-99999999999999999999999999", 0x0000000000000000u, 0x00000000u},
{"1e0000000000000000000000000000000000000000308", 0x7FE1CCF385EBC8A0u, 0x7F800000u},
{"123456789012345678901234567890e-30", 0x3FBF9ADD3746F65Fu, 0x3DFCD6EAu},
{"18446744073709551615", 0x43F0000000000000u, 0x5F800000u},
{"18446744073709551616", 0x43F0000000000000u, 0x5F800000u},
{"-9223372036854775808", 0xC3E0000000000000u, 0xDF000000u},
{"-9223372036854775809", 0xC3E0000000000000u, 0xDF000000u},
}
};
return table;
}
} // namespace float_hard_cases
+106 -553
View File
@@ -13,19 +13,15 @@
using nlohmann::json;
#include <array> // array
#include <cfloat> // FLT_EVAL_METHOD
#include <cstdint> // uint32_t, uint64_t
#include <cstdio> // snprintf
#include <cstdlib> // strtod
#include <cstring> // memcpy
#include <map> // map
#include <random> // mt19937
#include <sstream> // stringstream
#include <string> // string
#include <utility> // pair
#include <vector> // vector
#include "float_hard_cases.hpp"
namespace
{
// shortcut to scan a string literal
@@ -261,7 +257,7 @@ TEST_CASE("lexer number fast path")
"123456789012345678901234567890", // huge -> float
"0.30000000000000004", "2.2250738585072014e-308", "1e308",
// high-precision / wide-exponent values that exercise the
// Eisel-Lemire path beyond the Clinger subset
// std::from_chars (Eisel-Lemire) path beyond the Clinger subset
"1.7976931348623157e308", "1.2345678901234567e-250",
"9007199254740993", "5e-324", "1e-320"
};
@@ -283,18 +279,20 @@ TEST_CASE("lexer number fast path")
}
}
SECTION("significant digits around Clinger's fast path")
SECTION("significant-digit gate for the Clinger fast path")
{
// Clinger's fast path needs a significand of at most 2^53, which
// tokens with 17 or more significant digits exceed. The conversion
// splits the token at the positions the scanners recorded, so leading
// zeros must not count as digits - "0.1234567890123456" has 16
// significant digits, not 17 - and both scanners must agree.
// Clinger's fast path needs a significand below 2^53, so it cannot
// succeed once the mantissa has 17 or more significant digits (the
// significand would be at least 10^16). The lexer skips the attempt
// there. That is only allowed to save work: every value must still come
// out bit-exactly, and both scanners must agree. In particular the gate
// must not fire for tokens whose leading zeros merely look like extra
// digits - "0.1234567890123456" has 16 significant digits, not 17.
const std::vector<std::string> numbers =
{
"1234567890123456", // 16 significant digits
"12345678901234567", // 17
"123456789012345678", // 18
"12345678901234567", // 17 -> attempt skipped
"123456789012345678", // 18 -> attempt skipped
"0.1234567890123456", // 16: the leading "0" is not significant
"0.12345678901234567", // 17
"0.00000000000000001", // 1, in a long token
@@ -665,323 +663,46 @@ TEST_CASE("lexer string fast path")
}
}
TEST_CASE("lexer escape fast path")
TEST_CASE("parse_float_fast declines what it cannot convert exactly")
{
// json::accept() never throws, so this section stays covered without
// exceptions; it pins which of the cases below are valid/invalid and
// checks the contiguous and streaming paths agree on that classification.
SECTION("accept() parity")
// The lexer only hands well-formed numbers to parse_float_fast, so the
// malformed ones below can only be passed to it directly. Declining is
// always safe: the caller then falls back to a slower, exact conversion.
const auto fast = [](const std::string & s, double & out)
{
const std::vector<std::pair<std::string, bool>> cases =
{
{"\\u0041", true}, {"\\u00e4", true}, {"\\u00E4", true},
{"\\uD83D\\uDE00", true},
{"\\u12", false}, {"\\u12G4", false}, {"\\uXYZW", false},
{"\\uD800", false}, {"\\uD800A", false}, {"\\uD800\\u0041", false},
{"\\uDC00", false}, {"\\u", false}
};
for (const auto& c : cases)
{
for (const std::size_t offset :
{
std::size_t{0}, std::size_t{9}
})
{
const std::string doc = "[\"" + std::string(offset, 'a') + c.first + "\"]";
CAPTURE(doc);
CHECK(json::accept(doc) == c.second);
std::stringstream ss(doc);
CHECK(json::accept(ss) == c.second);
}
}
}
#if !defined(JSON_NOEXCEPTION)
// the full outcome of parsing @a doc: the parsed value, or the exact
// error message, so a mismatch in either is caught
const auto outcome = [](const std::string & doc, bool streaming) -> std::string
{
try
{
if (streaming)
{
std::stringstream ss(doc);
const json j = json::parse(ss);
return j.dump();
}
const json j = json::parse(doc);
return j.dump();
}
catch (const json::exception& e)
{
return {e.what()};
}
return nlohmann::detail::parse_float_fast(s.data(), s.data() + s.size(), out);
};
double out = 0;
SECTION("contiguous vs streaming parity")
{
const std::vector<std::string> escapes =
{
"\\u0041", // "A"
"\\u00e4", // "ä" (lowercase hex)
"\\u00E4", // "ä" (uppercase hex)
"\\uD83D\\uDE00", // valid surrogate pair (an emoji)
"\\u12", // truncated: only 2 hex digits before the closing quote
"\\u12G4", // invalid hex digit at the 3rd position
"\\uXYZW", // all 4 bytes invalid
"\\uD800", // lone high surrogate, string ends right after
"\\uD800A", // high surrogate not followed by another \u escape
"\\uD800\\u0041", // high surrogate followed by \u, but not a low surrogate
"\\uDC00", // lone low surrogate
"\\u", // '\u' with nothing after (closing quote right away)
};
// once at the start of the string and once past the first 8-byte SWAR
// word of the outer string_bulk_run, so the escape is reached both
// right after the opening quote and mid-run
for (const auto& escape : escapes)
{
for (const std::size_t offset :
{
std::size_t{0}, std::size_t{9}
})
{
const std::string doc = "[\"" + std::string(offset, 'a') + escape + "\"]";
CAPTURE(doc);
CHECK(outcome(doc, false) == outcome(doc, true));
}
// the escape is the last thing before end of input: no closing
// quote at all
const std::string truncated_doc = "[\"" + escape;
CAPTURE(truncated_doc);
CHECK(outcome(truncated_doc, false) == outcome(truncated_doc, true));
}
}
SECTION("truncated \\u escape at every distance from the end of input")
{
// ia.bulk_remaining() must correctly report fewer than 4 bytes for
// every possible count of trailing hex-looking bytes (0, 1, 2, or 3)
// before end of input, so the fast path declines and the byte path
// alone reports the "must be followed by 4 hex digits" error, at the
// same position, in every case
for (const std::string& tail :
{
std::string{}, std::string("1"), std::string("12"), std::string("123")
})
{
const std::string doc = "[\"\\u" + tail;
CAPTURE(doc);
CHECK(outcome(doc, false) == outcome(doc, true));
CHECK(outcome(doc, false).find("must be followed by 4 hex digits") != std::string::npos);
}
}
SECTION("invalid hex digit at every position of the 4")
{
// the fast path must decline for *any* invalid byte among the 4, not
// just the first, and the byte path must then stop at exactly that
// position - same as it always has
for (std::size_t bad_pos = 0; bad_pos < 4; ++bad_pos)
{
std::string digits = "1234";
digits[bad_pos] = 'g'; // not a hex digit
const std::string doc = "[\"\\u" + digits + "\"]";
CAPTURE(doc);
CHECK(outcome(doc, false) == outcome(doc, true));
CHECK(outcome(doc, false).find("must be followed by 4 hex digits") != std::string::npos);
}
}
SECTION("random escapes")
{
// A seeded PRNG builds the 4 bytes following `\u` from a mix of hex
// digits and non-hex bytes, at varying distances from the start of
// the string, to compare the two scanners on many more shapes than
// are practical to enumerate by hand.
std::mt19937 gen(7654321); // NOLINT(cert-msc32-c,cert-msc51-cpp)
const std::string hex_alphabet = "0123456789AaBbCcDdEeFf";
std::uniform_int_distribution<std::size_t> pick_hex(0, hex_alphabet.size() - 1);
std::uniform_int_distribution<int> pick_byte(1, 255); // never NUL
std::uniform_int_distribution<int> pick_is_hex(0, 4); // 4-in-5 chance of a hex digit
std::uniform_int_distribution<std::size_t> pick_offset(0, 12);
std::vector<std::string> mismatches;
for (int iter = 0; iter < 3000; ++iter)
{
std::string digits;
for (int i = 0; i < 4; ++i)
{
if (pick_is_hex(gen) != 0)
{
digits += hex_alphabet[pick_hex(gen)];
}
else
{
char c = static_cast<char>(pick_byte(gen));
if (c == '"' || c == '\\')
{
// keep the string well-formed apart from the escape
// itself, so any mismatch is attributable to the \u
// handling and not to an unrelated quote/escape
c = 'z';
}
digits += c;
}
}
const std::string doc = "[\"" + std::string(pick_offset(gen), 'a') + "\\u" + digits + "\"]";
if (outcome(doc, false) != outcome(doc, true))
{
mismatches.push_back(doc);
}
}
CAPTURE(mismatches);
CHECK(mismatches.empty());
}
#if defined(FLT_EVAL_METHOD) && FLT_EVAL_METHOD != 0
// without true double precision, the fast path declines everything
CHECK_FALSE(fast("1.5", out));
#else
CHECK(fast("1.5", out));
CHECK(out == 1.5);
CHECK(fast("+2.5e1", out));
CHECK(out == 25.0);
CHECK(fast("-25E-1", out));
CHECK(out == -2.5);
CHECK(fast("1e", out));
CHECK(out == 1.0);
#endif
}
namespace
{
// the index of the decimal point (or npos) and of the end of the mantissa of a
// number token, which the lexer records while scanning it
std::pair<std::size_t, std::size_t> float_token_layout(const std::string& s)
{
std::size_t dot = std::string::npos;
std::size_t mantissa_end = s.size();
for (std::size_t i = 0; i < s.size(); ++i)
{
if (s[i] == '.')
{
dot = i;
}
else if (s[i] == 'e' || s[i] == 'E')
{
mantissa_end = i;
break;
}
}
return {dot, mantissa_end};
}
// not a number
CHECK_FALSE(fast("", out));
CHECK_FALSE(fast("-", out));
CHECK_FALSE(fast(".", out));
CHECK_FALSE(fast("1.2.3", out));
CHECK_FALSE(fast("1x", out));
CHECK_FALSE(fast("1e+", out));
CHECK_FALSE(fast("1e1x", out));
template<typename FloatType>
FloatType parse_native(const std::string& s)
{
const auto layout = float_token_layout(s);
return nlohmann::detail::parse_float_native<FloatType>(s.data(), s.data() + s.size(), layout.first, layout.second);
}
std::uint64_t bits_of(double d)
{
std::uint64_t b = 0;
std::memcpy(&b, &d, sizeof(b));
return b;
}
std::uint32_t bits_of(float f)
{
std::uint32_t b = 0;
std::memcpy(&b, &f, sizeof(b));
return b;
}
std::uint64_t native_bits64(const std::string& s)
{
return bits_of(parse_native<double>(s));
}
std::uint32_t native_bits32(const std::string& s)
{
return bits_of(parse_native<float>(s));
}
} // namespace
TEST_CASE("parse_float_native rounds correctly")
{
SECTION("double")
{
CHECK(native_bits64("1.5") == 0x3FF8000000000000u);
CHECK(native_bits64("0.1") == 0x3FB999999999999Au);
CHECK(native_bits64("-0.0") == 0x8000000000000000u);
CHECK(native_bits64("0e999999999999999999999") == 0u);
// 2^53 + 1 is exactly between two doubles: ties to even, unless more digits follow
CHECK(native_bits64("9007199254740993") == 0x4340000000000000u);
CHECK(native_bits64("9007199254740993.0000000000000000001") == 0x4340000000000001u);
CHECK(native_bits64("9007199254740992.9999999999999999999") == 0x4340000000000000u);
// 1 + 2^-53 exactly (a tie), and one unit in the 55th digit around it
CHECK(native_bits64("1.00000000000000011102230246251565404236316680908203125") == 0x3FF0000000000000u);
CHECK(native_bits64("1.00000000000000011102230246251565404236316680908203126") == 0x3FF0000000000001u);
CHECK(native_bits64("1.00000000000000011102230246251565404236316680908203124") == 0x3FF0000000000000u);
// subnormal and overflow boundaries
CHECK(native_bits64("2.4703282292062327e-324") == 0u);
CHECK(native_bits64("2.4703282292062328e-324") == 1u);
CHECK(native_bits64("2.2250738585072011e-308") == 0x000FFFFFFFFFFFFFu);
CHECK(native_bits64("2.2250738585072012e-308") == 0x0010000000000000u);
CHECK(native_bits64("1.7976931348623157e308") == 0x7FEFFFFFFFFFFFFFu);
CHECK(native_bits64("1.7976931348623159e308") == 0x7FF0000000000000u);
CHECK(native_bits64("-1e400") == 0xFFF0000000000000u);
CHECK(native_bits64("-1e-400") == 0x8000000000000000u);
// exponents and zeros far beyond the range cancel out
CHECK(native_bits64("0." + std::string(1000, '0') + "1e1001") == 0x3FF0000000000000u);
CHECK(native_bits64("1" + std::string(1000, '0') + "e-1000") == 0x3FF0000000000000u);
CHECK(native_bits64("1e-99999999999999999999999") == 0u);
CHECK(native_bits64("1E+99999999999999999999999") == 0x7FF0000000000000u);
// more digits than any midpoint has (769): only whether a nonzero digit follows matters
const std::string tie = "1.00000000000000011102230246251565404236316680908203125";
CHECK(native_bits64(tie + std::string(800, '0')) == 0x3FF0000000000000u);
CHECK(native_bits64(tie + std::string(800, '0') + "1") == 0x3FF0000000000001u);
}
SECTION("float")
{
CHECK(native_bits32("1.5") == 0x3FC00000u);
CHECK(native_bits32("0.1") == 0x3DCCCCCDu);
CHECK(native_bits32("-0.0") == 0x80000000u);
// 2^24 + 1 is exactly between two floats
CHECK(native_bits32("16777217") == 0x4B800000u);
CHECK(native_bits32("16777217.000000000000000000001") == 0x4B800001u);
CHECK(native_bits32("16777218.999999999999999999999") == 0x4B800001u);
CHECK(native_bits32("16777219") == 0x4B800002u);
// subnormal and overflow boundaries
CHECK(native_bits32("3.4028235677973366e38") == 0x7F7FFFFFu);
CHECK(native_bits32("3.4028235677973367e38") == 0x7F800000u);
CHECK(native_bits32("7.006492321624085e-46") == 0u);
CHECK(native_bits32("7.006492321624086e-46") == 1u);
CHECK(native_bits32("1.1754942e-38") == 0x007FFFFFu);
CHECK(native_bits32("-1.17549435e-38") == 0x80800000u);
CHECK(native_bits32("1e39") == 0x7F800000u);
CHECK(native_bits32("-1e-50") == 0x80000000u);
// not rounded through double: its double would round to another float
CHECK(native_bits32("1.00000005960464477539062500000000001") == 0x3F800001u);
CHECK(native_bits32("9007199254740993") == 0x5A000000u);
}
SECTION("the conversion shared with other parsers")
{
// convert_float() gives the lexer's results, for every type
const std::vector<std::string> tokens =
{
"0", "-0.0", "1.5", "0.1", "1e-400", "-2.5E+3", "123456789012345678901234567890",
"9007199254740993.0000000000000000001", "4.9406564584124654e-324"
};
using float_json = nlohmann::basic_json<std::map, std::vector, std::string, bool, std::int64_t, std::uint64_t, float>;
using long_double_json = nlohmann::basic_json<std::map, std::vector, std::string, bool, std::int64_t, std::uint64_t, long double>;
for (const auto& t : tokens)
{
CAPTURE(t);
const auto layout = float_token_layout(t);
const char* const first = t.data();
const char* const last = first + t.size();
const auto d = nlohmann::detail::convert_float<double>(first, last, layout.first, layout.second);
const auto f = nlohmann::detail::convert_float<float>(first, last, layout.first, layout.second);
const auto ld = nlohmann::detail::convert_float<long double>(first, last, layout.first, layout.second);
CHECK(bits_of(d) == bits_of(json::parse(t).get<double>()));
CHECK(bits_of(f) == bits_of(float_json::parse(t).get<float>()));
CHECK(ld == long_double_json::parse(t).get<long double>());
}
}
// numbers that are not represented exactly on the fast path
CHECK_FALSE(fast("12345678901234567890", out));
CHECK_FALSE(fast("1e10000", out));
CHECK_FALSE(fast("9007199254740993", out));
CHECK_FALSE(fast("1e23", out));
CHECK_FALSE(fast("1e-23", out));
}
namespace
@@ -1085,6 +806,40 @@ std::size_t big_bit_length(const big_uint& a)
}
return n;
}
std::uint64_t bits_of(double d)
{
std::uint64_t b = 0;
std::memcpy(&b, &d, sizeof(b));
return b;
}
bool eisel_lemire(const std::string& s, double& out)
{
return nlohmann::detail::parse_float_eisel_lemire(s.data(), s.data() + s.size(), out);
}
// significant digits of a token, without trailing zeros
std::size_t significant_digits(const std::string& s)
{
std::string digits;
for (const char c : s)
{
if (c == 'e' || c == 'E')
{
break;
}
if (c >= '0' && c <= '9' && !(digits.empty() && c == '0'))
{
digits += c;
}
}
while (!digits.empty() && digits.back() == '0')
{
digits.pop_back();
}
return digits.size();
}
} // namespace
TEST_CASE("Eisel-Lemire float conversion")
@@ -1482,33 +1237,26 @@ TEST_CASE("Eisel-Lemire float conversion")
for (const auto& c : known)
{
CAPTURE(c.first);
CHECK(native_bits64(c.first) == c.second);
double out = 0;
if (eisel_lemire(c.first, out))
{
CHECK(bits_of(out) == c.second);
}
else
{
// only tokens with more than 19 significant digits are left to
// strtod: those whose value lies too close to a tie
CHECK(significant_digits(c.first) > 19);
}
}
}
SECTION("binary32")
{
using binary32 = nlohmann::detail::ieee_binary_format<24>;
CHECK(nlohmann::detail::eisel_lemire<binary32>(0, 1) == 0x3F800000u);
CHECK(nlohmann::detail::eisel_lemire<binary32>(-1, 1) == 0x3DCCCCCDu);
CHECK(nlohmann::detail::eisel_lemire<binary32>(-1, 15) == 0x3FC00000u);
CHECK(nlohmann::detail::eisel_lemire<binary32>(0, 16777217) == 0x4B800000u); // tie, to even
CHECK(nlohmann::detail::eisel_lemire<binary32>(0, 16777219) == 0x4B800002u); // tie, to even
CHECK(nlohmann::detail::eisel_lemire<binary32>(-45, 1) == 0x00000001u);
CHECK(nlohmann::detail::eisel_lemire<binary32>(-46, 7) == 0x00000000u);
CHECK(nlohmann::detail::eisel_lemire<binary32>(-46, 8) == 0x00000001u);
CHECK(nlohmann::detail::eisel_lemire<binary32>(-65, 9999999999999999999u) == 0x00000000u);
CHECK(nlohmann::detail::eisel_lemire<binary32>(20, 3402823466385288598u) == 0x7F7FFFFFu);
CHECK(nlohmann::detail::eisel_lemire<binary32>(20, 3402823669209384635u) == 0x7F800000u);
CHECK(nlohmann::detail::eisel_lemire<binary32>(39, 1) == 0x7F800000u);
CHECK(nlohmann::detail::eisel_lemire<binary32>(-5, 0) == 0x00000000u);
}
SECTION("round trip")
{
// every double written by to_chars and read back, and its 17-digit
// form with trailing digits that make the token longer than 19 digits
// every double written by to_chars and read back, also with trailing
// digits that make the token longer than 19 digits
std::uint64_t state = 5295;
std::size_t declined = 0;
for (int i = 0; i < 200000; ++i)
{
state ^= state << 13u;
@@ -1530,51 +1278,30 @@ TEST_CASE("Eisel-Lemire float conversion")
const char* end = nlohmann::detail::to_chars(buffer.data(), buffer.data() + buffer.size(), d);
const std::string token(buffer.data(), static_cast<std::size_t>(end - buffer.data()));
CAPTURE(token);
CHECK(native_bits64(token) == b);
double out = 0;
REQUIRE(eisel_lemire(token, out));
CHECK(bits_of(out) == b);
// insert digits before the exponent of the 17-digit form: that
// form lies strictly inside the rounding interval of the double
// (the shortest one may lie on its boundary), and the digits move
// it by far less than the distance to the boundary, so the value
// must not change
std::array<char, 64> digits17{};
static_cast<void>(std::snprintf(digits17.data(), digits17.size(), "%.17g", d)); // NOLINT(cppcoreguidelines-pro-type-vararg,hicpp-vararg)
std::string longer = digits17.data();
// insert digits before the exponent: the value moves by far less
// than the distance to the rounding boundary, so it must not change
std::string longer = token;
const std::size_t e = longer.find('e');
const std::size_t dot = longer.find('.');
const std::string extra = dot == std::string::npos ? ".000000000000000000001" : "000000000000000000001";
longer.insert(e == std::string::npos ? longer.size() : e, extra);
CAPTURE(longer);
CHECK(native_bits64(longer) == b);
}
}
SECTION("round trip, binary32")
{
std::uint32_t state = 5295;
for (int i = 0; i < 100000; ++i)
{
state ^= state << 13u;
state ^= state >> 17u;
state ^= state << 5u;
std::uint32_t b = state;
if ((b & 0x7F800000u) == 0x7F800000u)
if (eisel_lemire(longer, out))
{
continue; // infinity or NaN
CHECK(bits_of(out) == b);
}
if (i % 4 == 0)
else
{
b &= 0x807FFFFFu; // subnormals
// w and w + 1 round differently: only when the value is very
// close to a rounding boundary
++declined;
}
float f = 0;
std::memcpy(&f, &b, sizeof(f));
std::array<char, 64> buffer{};
const char* end = nlohmann::detail::to_chars(buffer.data(), buffer.data() + buffer.size(), f);
const std::string token(buffer.data(), static_cast<std::size_t>(end - buffer.data()));
CAPTURE(token);
CHECK(native_bits32(token) == b);
}
CHECK(declined < 1000); // 107 of the 200,000
}
SECTION("used by the lexer")
@@ -1588,177 +1315,3 @@ TEST_CASE("Eisel-Lemire float conversion")
"[json.exception.out_of_range.406] number overflow parsing '1.7976931348623159e308'", json::out_of_range&);
}
}
namespace
{
using float_json = nlohmann::basic_json<std::map, std::vector, std::string, bool, std::int64_t, std::uint64_t, float>;
// the bits of the float that parse() gives for a token, via both scanners;
// the value must be the same for both
template<typename Json, typename Bits>
void check_parse(const std::string& token, Bits expected, Bits infinity)
{
std::stringstream stream(token);
if ((expected & ~(Bits{1} << (8 * sizeof(Bits) - 1))) == infinity)
{
Json _;
CHECK_THROWS_WITH_AS(_ = Json::parse(token), ("[json.exception.out_of_range.406] number overflow parsing '" + token + "'").c_str(), typename Json::out_of_range&);
CHECK_THROWS_WITH_AS(_ = Json::parse(stream), ("[json.exception.out_of_range.406] number overflow parsing '" + token + "'").c_str(), typename Json::out_of_range&);
return;
}
const Json contiguous = Json::parse(token);
const Json streamed = Json::parse(stream);
if (contiguous.is_number_float()) // not an integer that fits
{
CHECK(bits_of(contiguous.template get<typename Json::number_float_t>()) == expected);
CHECK(bits_of(streamed.template get<typename Json::number_float_t>()) == expected);
}
else
{
CHECK(streamed.is_number_integer());
}
}
} // namespace
TEST_CASE("float conversion of hard cases")
{
// see float_hard_cases.hpp
for (const auto& c : float_hard_cases::cases())
{
const std::string token = c.token;
CAPTURE(token);
CHECK(native_bits64(token) == c.bits64);
CHECK(native_bits32(token) == c.bits32);
check_parse<json>(token, c.bits64, std::uint64_t{0x7FF0000000000000u});
check_parse<float_json>(token, c.bits32, std::uint32_t{0x7F800000u});
}
}
TEST_CASE("float overflow and underflow in the parser")
{
SECTION("double")
{
check_parse<json>("1.7976931348623157e308", std::uint64_t{0x7FEFFFFFFFFFFFFFu}, std::uint64_t{0x7FF0000000000000u});
check_parse<json>("1.7976931348623159e308", std::uint64_t{0x7FF0000000000000u}, std::uint64_t{0x7FF0000000000000u});
check_parse<json>("-1e309", std::uint64_t{0xFFF0000000000000u}, std::uint64_t{0x7FF0000000000000u});
check_parse<json>("1" + std::string(400, '0'), std::uint64_t{0x7FF0000000000000u}, std::uint64_t{0x7FF0000000000000u});
check_parse<json>("1e99999999999999999999", std::uint64_t{0x7FF0000000000000u}, std::uint64_t{0x7FF0000000000000u});
// an underflow gives a zero with the sign of the token
check_parse<json>("1e-400", std::uint64_t{0}, std::uint64_t{0x7FF0000000000000u});
check_parse<json>("-1e-400", std::uint64_t{0x8000000000000000u}, std::uint64_t{0x7FF0000000000000u});
check_parse<json>("-2.4703282292062327e-324", std::uint64_t{0x8000000000000000u}, std::uint64_t{0x7FF0000000000000u});
check_parse<json>("0." + std::string(400, '0') + "1", std::uint64_t{0}, std::uint64_t{0x7FF0000000000000u});
}
SECTION("float")
{
check_parse<float_json>("3.4028234e38", std::uint32_t{0x7F7FFFFFu}, std::uint32_t{0x7F800000u});
check_parse<float_json>("3.4028236e38", std::uint32_t{0x7F800000u}, std::uint32_t{0x7F800000u});
check_parse<float_json>("-1e39", std::uint32_t{0xFF800000u}, std::uint32_t{0x7F800000u});
check_parse<float_json>("1e-46", std::uint32_t{0}, std::uint32_t{0x7F800000u});
check_parse<float_json>("-1e-46", std::uint32_t{0x80000000u}, std::uint32_t{0x7F800000u});
check_parse<float_json>("-7.006492321624085e-46", std::uint32_t{0x80000000u}, std::uint32_t{0x7F800000u});
check_parse<float_json>("-7.006492321624086e-46", std::uint32_t{0x80000001u}, std::uint32_t{0x7F800000u});
}
}
TEST_CASE("string scanning kernels")
{
// the word-at-a-time kernels must stop exactly where a byte-by-byte scan
// stops, for any content, length, and alignment
const auto reference_special = [](const unsigned char* data, std::size_t n)
{
std::size_t i = 0;
while (i < n && !nlohmann::detail::is_string_special(data[i]))
{
++i;
}
return i;
};
const auto reference_copyable = [](const unsigned char* data, std::size_t n)
{
std::size_t i = 0;
while (i < n && nlohmann::detail::is_ascii_copyable(data[i]))
{
++i;
}
return i;
};
const auto reference_bulk_run = [](const unsigned char* data, std::size_t n)
{
std::size_t i = 0;
while (i < n)
{
if (data[i] < 0x80u)
{
if (nlohmann::detail::is_string_special(data[i]))
{
break;
}
++i;
continue;
}
const std::size_t seq = nlohmann::detail::validate_one_utf8(data + i, n - i);
if (seq == 0)
{
break;
}
i += seq;
}
return i;
};
// pieces: ordinary ASCII, stops, DEL, well-formed sequences of every
// length, and ill-formed or truncated ones
const std::vector<std::string> pieces =
{
"a", "Z", " ", "~", "0123456789", "\"", "\\", std::string(1, '\0'), "\n", "\x1F", "\x7F",
"\xC3\xA4", "\xE2\x82\xAC", "\xE6\x97\xA5\xE6\x9C\xAC", "\xF0\x9F\x98\x80", "\xED\x9F\xBF",
"\x80", "\xC0\x80", "\xC3", "\xE2\x82", "\xED\xA0\x80", "\xF4\x90\x80\x80", "\xFF",
};
std::uint64_t state = 5295;
const auto next = [&state]()
{
state ^= state << 13u;
state ^= state >> 7u;
state ^= state << 17u;
return state;
};
// the upper half as a 32-bit value: converts to std::size_t implicitly on
// every platform (a cast of std::uint64_t is useless where both are the
// same type, and required where std::size_t is 32 bits wide)
const auto next_small = [&next]()
{
return static_cast<std::uint32_t>(next() >> 32u);
};
for (int round = 0; round < 100000; ++round)
{
// mostly ordinary text, so that runs span several words
std::string text(next_small() % 8u, '.');
const std::size_t count = next_small() % 12u;
for (std::size_t k = 0; k < count; ++k)
{
const std::size_t p = (next() % 4 == 0) ? next_small() % pieces.size() : 0;
text += pieces[p];
text += std::string(next_small() % 10u, 'x');
}
const auto* data = reinterpret_cast<const unsigned char*>(text.data()); // NOLINT(cppcoreguidelines-pro-type-reinterpret-cast)
for (std::size_t offset = 0; offset < 3 && offset <= text.size(); ++offset)
{
const std::size_t n = text.size() - offset;
CAPTURE(text);
CAPTURE(offset);
CHECK(nlohmann::detail::find_string_special(data + offset, n) == reference_special(data + offset, n));
CHECK(nlohmann::detail::find_ascii_copyable_run(data + offset, n) == reference_copyable(data + offset, n));
CHECK(nlohmann::detail::scalar_string_bulk_run(data + offset, n) == reference_bulk_run(data + offset, n));
}
}
// the trailing-zero count, whichever implementation the compiler gets
for (int k = 0; k < 64; ++k)
{
const std::uint64_t bit = std::uint64_t{1} << k;
CHECK(nlohmann::detail::count_trailing_zeros(bit) == k);
CHECK(nlohmann::detail::count_trailing_zeros(bit | (bit << 1u) | 0x8000000000000000u) == k);
}
}
+8 -25
View File
@@ -257,11 +257,10 @@ struct LocaleSwitchingSax final: public nlohmann::json_sax<json>
TEST_CASE("locale changes between lexer construction and number conversion (#5198)")
{
// float and double are converted without the locale. A long double that
// is not binary64 can take the strtold fallback, which honors the locale
// that is current at conversion time. The numbers are chosen so that it
// does: too many significant digits for Clinger's fast path, an underflow
// that std::from_chars rejects, and a plain value.
// The numbers are chosen so that the conversion also takes the strtod
// fallback, which honors the locale that is current at conversion time:
// too many significant digits for Clinger's fast path, an underflow that
// std::from_chars rejects, and a plain value.
const std::vector<std::string> numbers = {"3.14159265358979323846", "1.5e-400", "12.34", "-0.000123456789012345678"};
std::string text = "[";
for (const auto& n : numbers)
@@ -325,8 +324,7 @@ TEST_CASE("locale changes between lexer construction and number conversion (#519
}
}
// a long double goes through std::strtold unless it is binary64 or
// std::from_chars supports it
// a long double goes through std::strtold unless std::from_chars supports it
{
bool switched = false;
const auto cb = [&](int /*depth*/, long_double_json::parse_event_t event, long_double_json& /*parsed*/) noexcept
@@ -352,15 +350,8 @@ TEST_CASE("locale with a multi-byte decimal point")
{
// Some locales use a decimal point that is not a single character, e.g.
// U+066B ARABIC DECIMAL SEPARATOR (two bytes in UTF-8). It cannot be
// substituted in place for '.', so the strtold fallback (only for long
// double formats other than binary64) converts a copy of the token with
// the whole decimal point instead (#5660). The values must be those of the
// "C" locale.
using long_double_json = nlohmann::basic_json<std::map, std::vector, std::string, bool, std::int64_t, std::uint64_t, long double>;
const char* const long_double_numbers = "[3.14159265358979323846, 1.5e-400, -0.000123456789012345678]";
REQUIRE(std::setlocale(LC_NUMERIC, "C") != nullptr);
const long_double_json expected_long_double = long_double_json::parse(long_double_numbers);
// substituted in place for '.', so the strtod fallback stops early. The
// conversion must still terminate rather than retry forever.
const std::array<const char*, 6> names = {{"ar_EG.UTF-8", "ar_SA.UTF-8", "fa_IR.UTF-8", "ps_AF.UTF-8", "ar_EG", "fa_IR"}};
bool tested = false;
for (const char* name : names)
@@ -378,20 +369,12 @@ TEST_CASE("locale with a multi-byte decimal point")
tested = true;
// too many significant digits for Clinger's fast path, and an underflow
// that std::from_chars rejects: double does not depend on the locale
// that std::from_chars rejects: both reach the strtod fallback
json j;
CHECK_NOTHROW(j = json::parse("[3.14159265358979323846, 1.5e-400, -0.000123456789012345678]"));
CHECK(j.is_array());
CHECK(j[0] == 3.14159265358979323846);
CHECK(j[1] == 0.0);
CHECK(j[2] == -0.000123456789012345678);
CHECK(json::accept("3.14159265358979323846"));
// a long double that reaches the strtold fallback is not truncated
long_double_json ld;
CHECK_NOTHROW(ld = long_double_json::parse(long_double_numbers));
CHECK(ld == expected_long_double);
// a value the locale-independent paths convert is not affected
CHECK(json::parse("12.5") == 12.5);
}
+15 -11
View File
@@ -1,8 +1,8 @@
# amalgamate.py - Amalgamate C source and header files
Origin: https://bitbucket.org/erikedlund/amalgamate
Mirror: https://github.com/edlund/amalgamate
Origin: https://github.com/edlund/amalgamate (formerly hosted at
https://bitbucket.org/erikedlund/amalgamate, which no longer exists; see
`CHANGES.md` for the upstream commit this copy is based on)
`amalgamate.py` aims to make it easy to use SQLite-style C source and header
amalgamation in projects.
@@ -41,21 +41,22 @@ results.
## Installing amalgamate.py
Python v.2.7.0 or higher is required.
Python 3 is required.
`amalgamate.py` can be tested and installed using the following commands:
./test.sh && sudo -k cp ./amalgamate.py /usr/local/bin/
In this repository, `amalgamate.py` is not installed separately; it is run in
place through `make amalgamate`, which calls it once for `json.hpp` and once
for `json_fwd.hpp` (see the root `Makefile`).
## Using amalgamate.py
amalgamate.py [-v] -c path/to/config.json -s path/to/source/dir \
[-p path/to/prologue.(c|h)]
amalgamate.py -c path/to/config.json -s path/to/source/dir \
[-p path/to/prologue.(c|h)] [--verbose=yes|no]
* The `-c, --config` option should specify the path to a JSON config file which
lists the source files, include paths and where to write the resulting
amalgamation. Have a look at `test/source.c.json` and `test/include.h.json`
to see two examples.
amalgamation. `config_json.json` and `config_json_fwd.json` in this
directory are the configs used for `json.hpp` and `json_fwd.hpp`; each
sets `target`, `sources` and `include_paths`.
The optional `external` list names include paths that are kept as `#include`
directives instead of being inlined, e.g. `["nlohmann/json.hpp"]` for a header
@@ -68,3 +69,6 @@ Python v.2.7.0 or higher is required.
* The `-p, --prologue` option should specify the path to a file which will be
added to the beginning of the amalgamation. It is optional.
* The `-v, --verbose` option takes `yes` or `no` (for example
`--verbose=yes`, as used by the Makefile). It is optional.
+19 -1
View File
@@ -2,8 +2,26 @@
Generate the Natvis debugger visualization file for all supported namespace combinations.
The ABI tag list and the library version are parsed from
`include/nlohmann/detail/abi_macros.hpp`, so this script must be re-run (via
`make natvis`) whenever an `NLOHMANN_JSON_ABI_TAG_*` macro is added to that
file or the library version is bumped — otherwise the committed
`nlohmann_json.natvis` drifts from the header it visualizes, and
`make check-amalgamation` fails.
## Usage
```shell
./generate_natvis.py --version X.Y.Z output_directory/
make natvis
```
or, equivalently:
```shell
./generate_natvis.py [--version X.Y.Z] [repository_root/]
```
`--version` and the output/repository-root directory both default to values
derived from this script's own location, so they only need to be given
explicitly when generating a Natvis file for a different checkout or a
version other than the one in `abi_macros.hpp`.
+58 -4
View File
@@ -7,21 +7,75 @@ import os
import re
import sys
# Directory of the repository, assuming this script stays at
# tools/generate_natvis/generate_natvis.py. Used only as the default value
# for the "output" argument below.
REPO_ROOT = os.path.normpath(os.path.join(sys.path[0], '..', '..'))
def semver(v):
if not re.fullmatch(r'\d+\.\d+\.\d+', v):
raise ValueError
return v
def abi_info(repo_root):
"""Derive the ABI tag list (in NLOHMANN_JSON_ABI_TAGS_CONCAT order) and the
library version from <repo_root>/include/nlohmann/detail/abi_macros.hpp,
so this script cannot drift from the header it visualizes."""
abi_macros_hpp = os.path.join(repo_root, 'include', 'nlohmann', 'detail', 'abi_macros.hpp')
with open(abi_macros_hpp) as f:
content = f.read()
# find the NLOHMANN_JSON_ABI_TAGS_CONCAT(...) invocation that lists the
# NLOHMANN_JSON_ABI_TAG_* identifiers in order (not its own #define, which
# only names its formal parameters a, b, c, ...)
tag_idents = None
for args in re.findall(r'NLOHMANN_JSON_ABI_TAGS_CONCAT\(\s*(.*?)\)', content, re.S):
idents = re.findall(r'NLOHMANN_JSON_ABI_TAG_\w+', args)
if idents:
tag_idents = idents
break
if not tag_idents:
raise ValueError(f'could not find NLOHMANN_JSON_ABI_TAGS_CONCAT(...) in {abi_macros_hpp}')
abi_tags = []
for ident in tag_idents:
# each tag is #define'd to its suffix (e.g. _diag) when the matching
# JSON_* option is enabled, and to nothing in the #else branch; only
# the non-empty definition matches here
match = re.search(r'#define\s+' + re.escape(ident) + r'\s+(_\w+)\s*\n', content)
if not match:
raise ValueError(f'could not find a non-empty #define for {ident} in {abi_macros_hpp}')
abi_tags.append(match.group(1))
version = {}
for part in ('MAJOR', 'MINOR', 'PATCH'):
match = re.search(r'#define\s+NLOHMANN_JSON_VERSION_' + part + r'\s+(\d+)', content)
if not match:
raise ValueError(f'could not find NLOHMANN_JSON_VERSION_{part} in {abi_macros_hpp}')
version[part] = match.group(1)
return abi_tags, '{MAJOR}.{MINOR}.{PATCH}'.format(**version)
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument('--version', required=True, type=semver, help='Library version number')
parser.add_argument('output', help='Output directory for nlohmann_json.natvis')
parser.add_argument('--version', type=semver,
help='Library version number (default: parsed from '
'include/nlohmann/detail/abi_macros.hpp below "output")')
parser.add_argument('output', nargs='?', default=REPO_ROOT,
help='Repository root: where include/nlohmann/detail/abi_macros.hpp is '
'read from and where nlohmann_json.natvis is written '
'(default: the repository root this script lives in)')
args = parser.parse_args()
derived_tags, derived_version = abi_info(args.output)
namespaces = ['nlohmann']
abi_prefix = 'json_abi'
abi_tags = ['_diag', '_ldvcmp', '_dp', '_bics', '_psp', '_snul']
version = '_v' + args.version.replace('.', '_')
abi_tags = derived_tags
version = '_v' + (args.version or derived_version).replace('.', '_')
inline_namespaces = []
# generate all combinations of inline namespace names
+51
View File
@@ -0,0 +1,51 @@
# macro_builder
Generates the argument-counting macros behind the `NLOHMANN_DEFINE_TYPE_*` and `NLOHMANN_DEFINE_DERIVED_TYPE_*`
macros in [`include/nlohmann/detail/macro_scope.hpp`](../../include/nlohmann/detail/macro_scope.hpp):
- `NLOHMANN_JSON_EXPAND`
- `NLOHMANN_JSON_GET_MACRO`, which selects a macro by the number of its arguments (64 slots)
- `NLOHMANN_JSON_PASTE`, which calls a function-like macro for each member, and its helpers `NLOHMANN_JSON_PASTE2`
to `NLOHMANN_JSON_PASTE64`
- `NLOHMANN_JSON_DOUBLE_PASTE`, which the `*_WITH_NAMES` macros use to call a function-like macro for each
(JSON name, member) pair, and its helpers `NLOHMANN_JSON_DOUBLE_PASTE3` to `NLOHMANN_JSON_DOUBLE_PASTE63`
- the slot table of `NLOHMANN_JSON_TYPE_BODY`, which dispatches `NLOHMANN_DEFINE_TYPE_*(Type)` (no further
arguments) to the zero-member implementation and every other argument count to the one-or-more-member
implementation
The number of slots (`max_args` in [`main.cpp`](main.cpp)) sets the member limit of these macros.
`NLOHMANN_JSON_PASTE` and `NLOHMANN_JSON_TYPE_BODY` take the function/prefix as their first argument, so 64 slots
allow 63 members; `NLOHMANN_JSON_DOUBLE_PASTE` additionally consumes its members two at a time (name, member), so
it only defines the odd helpers up to `NLOHMANN_JSON_DOUBLE_PASTE63`.
## Usage
From the project root:
```shell
c++ -std=c++11 tools/macro_builder/main.cpp -o macro_builder
./macro_builder
./macro_builder type_body
```
1. Run `./macro_builder` (no arguments). In `include/nlohmann/detail/macro_scope.hpp`, replace the lines from
`#define NLOHMANN_JSON_EXPAND( x ) x` to the `#define NLOHMANN_JSON_DOUBLE_PASTE63(...)` line with the output,
without its trailing empty line.
2. Run `./macro_builder type_body`. Replace the lines from `#define NLOHMANN_JSON_TYPE_BODY(Prefix, ...)` to the
`NLOHMANN_JSON_TYPE_BODY_SENTINEL))` line with the output.
3. Run `make amalgamate`. It updates `single_include/nlohmann/json.hpp` and runs `make pretty`, which indents the
continuation lines that the tool writes unindented.
With an unchanged `main.cpp`, these steps reproduce both blocks of `macro_scope.hpp` byte for byte. `make
macro_builder_check` (also run by CI, see `.github/workflows/check_amalgamation.yml`) automates this: it builds
`main.cpp`, regenerates both blocks, and fails on a diff against the checked-in header.
## Maintained by hand
The tool does not generate everything that depends on the number of slots. When changing `max_args`, also update:
- the documented limit of 63 members in `docs/mkdocs/docs` and the tests at that limit in
`tests/src/unit-udt_macro.cpp`
All three tables pass one macro name per slot to `NLOHMANN_JSON_GET_MACRO`, so they need exactly as many entries as
it has slots.
+76 -11
View File
@@ -1,10 +1,14 @@
#include <cstdlib>
#include <iostream>
#include <sstream>
#include <string>
using namespace std;
void build_code(int max_args)
// Builds NLOHMANN_JSON_EXPAND, NLOHMANN_JSON_GET_MACRO, and the
// NLOHMANN_JSON_PASTE / NLOHMANN_JSON_PASTE2..PASTE<max_args> dispatch table
// and recursive definitions.
string build_paste_code(int max_args)
{
stringstream ss;
ss << "#define NLOHMANN_JSON_EXPAND( x ) x" << endl;
@@ -12,32 +16,93 @@ void build_code(int max_args)
for (int i = 0 ; i < max_args ; i++)
ss << "_" << i + 1 << ", ";
ss << "NAME,...) NAME" << endl;
ss << "#define NLOHMANN_JSON_PASTE(...) NLOHMANN_JSON_EXPAND(NLOHMANN_JSON_GET_MACRO(__VA_ARGS__, \\" << endl;
for (int i = max_args ; i > 1 ; i--)
ss << "NLOHMANN_JSON_PASTE" << i << ", \\" << endl;
ss << "NLOHMANN_JSON_PASTE1)(__VA_ARGS__))" << endl;
ss << "#define NLOHMANN_JSON_PASTE2(func, v1) func(v1)" << endl;
for (int i = 3 ; i <= max_args ; i++)
{
ss << "#define NLOHMANN_JSON_PASTE" << i << "(func, ";
ss << "#define NLOHMANN_JSON_PASTE" << i << "(func, ";
for (int j = 1 ; j < i -1 ; j++)
ss << "v" << j << ", ";
ss << "v" << j << ", ";
ss << "v" << i-1 << ") NLOHMANN_JSON_PASTE2(func, v1) NLOHMANN_JSON_PASTE" << i-1 << "(func, ";
for (int j = 2 ; j < i-1 ; j++)
ss << "v" << j << ", ";
ss << "v" << i-1 << ")" << endl;
}
cout << ss.str() << endl;
return ss.str();
}
int main(int argc, char** argv)
// Builds the NLOHMANN_JSON_DOUBLE_PASTE dispatch table and recursive
// definitions used by the *_WITH_NAMES macros. Its GET_MACRO dispatch reuses
// the same max_args slots as NLOHMANN_JSON_PASTE, but DOUBLE_PASTE consumes
// its arguments two at a time (name, member), so an even slot count falls
// back to the next lower odd NLOHMANN_JSON_DOUBLE_PASTE<N>.
string build_double_paste_code(int max_args)
{
stringstream ss;
ss << "#define NLOHMANN_JSON_DOUBLE_PASTE(...) NLOHMANN_JSON_EXPAND(NLOHMANN_JSON_GET_MACRO(__VA_ARGS__, \\" << endl;
for (int i = max_args ; i > 1 ; i--)
{
int k = (i % 2 == 1) ? i : i - 1;
ss << "NLOHMANN_JSON_DOUBLE_PASTE" << k << ", \\" << endl;
}
ss << "NLOHMANN_JSON_DOUBLE_PASTE1)(__VA_ARGS__))" << endl;
ss << "#define NLOHMANN_JSON_DOUBLE_PASTE3(func, v1, v2) func(v1, v2)" << endl;
for (int k = 5 ; k <= max_args - 1 ; k += 2)
{
ss << "#define NLOHMANN_JSON_DOUBLE_PASTE" << k << "(func, ";
for (int j = 1 ; j < k - 1 ; j++)
ss << "v" << j << ", ";
ss << "v" << k - 1 << ") NLOHMANN_JSON_DOUBLE_PASTE3(func, v1, v2) NLOHMANN_JSON_DOUBLE_PASTE" << k - 2 << "(func, ";
for (int j = 3 ; j < k - 1 ; j++)
ss << "v" << j << ", ";
ss << "v" << k - 1 << ")" << endl;
}
return ss.str();
}
// Builds the NLOHMANN_JSON_TYPE_BODY dispatch table: max_args - 1 slots
// selecting the *_MEMBERS implementation and a final slot selecting
// *_EMPTY, so NLOHMANN_DEFINE_TYPE_*(Type) with no further arguments still
// resolves (issue #4041).
string build_type_body_table(int max_args)
{
stringstream ss;
ss << "#define NLOHMANN_JSON_TYPE_BODY(Prefix, ...) NLOHMANN_JSON_EXPAND(NLOHMANN_JSON_GET_MACRO(__VA_ARGS__, \\" << endl;
const int per_line = 8;
for (int i = 1 ; i <= max_args ; i++)
{
ss << (i == max_args ? "Prefix ## EMPTY" : "Prefix ## MEMBERS") << ", ";
if (i % per_line == 0)
ss << "\\" << endl;
}
ss << "NLOHMANN_JSON_TYPE_BODY_SENTINEL))" << endl;
return ss.str();
}
int main(int argc, char** argv)
{
int max_args = 64;
build_code(max_args);
// With "type_body", print only the NLOHMANN_JSON_TYPE_BODY dispatch
// table (a separate insertion point in macro_scope.hpp); otherwise
// print the EXPAND/GET_MACRO/PASTE/DOUBLE_PASTE block that precedes it.
if (argc > 1 && string(argv[1]) == "type_body")
{
cout << build_type_body_table(max_args);
}
else
{
cout << build_paste_code(max_args) << build_double_paste_code(max_args);
}
return 0;
}
+11 -11
View File
@@ -5,6 +5,8 @@ import logging
import os
import re
import shutil
import socket
import ssl
import sys
import subprocess
@@ -34,7 +36,7 @@ JSON_VERSION_RE = re.compile(r'\s*#\s*define\s+NLOHMANN_JSON_VERSION_MAJOR\s+')
class ExitHandler(logging.StreamHandler):
def __init__(self, level):
"""."""
"""Exit the process on log records at or above level."""
super().__init__()
self.level = level
@@ -54,7 +56,7 @@ def is_project_root(test_dir='.'):
class DirectoryEventBucket:
def __init__(self, callback, delay=1.2, threshold=0.8):
"""."""
"""Batch directory events and pass their common path to callback."""
self.delay = delay
self.threshold = timedelta(seconds=threshold)
self.callback = callback
@@ -99,7 +101,7 @@ class WorkTree:
make_command = 'make'
def __init__(self, root_dir, tree_dir):
"""."""
"""Track the working tree at tree_dir and its amalgamated header."""
self.root_dir = root_dir
self.tree_dir = tree_dir
self.rel_dir = os.path.relpath(tree_dir, root_dir)
@@ -114,11 +116,11 @@ class WorkTree:
self.build_time = t.strftime(DATETIME_FORMAT)
def __hash__(self):
"""."""
"""Hash by working tree directory."""
return hash((self.tree_dir))
def __eq__(self, other):
"""."""
"""Compare by working tree directory."""
if not isinstance(other, type(self)):
return NotImplemented
return self.tree_dir == other.tree_dir
@@ -150,7 +152,7 @@ class WorkTree:
class WorkTrees(FileSystemEventHandler):
def __init__(self, root_dir):
"""."""
"""Find the working trees below root_dir and watch it for changes."""
super().__init__()
self.root_dir = root_dir
self.trees = set([])
@@ -250,11 +252,11 @@ class WorkTrees(FileSystemEventHandler):
self.observer.stop()
self.observer.join()
class HeaderRequestHandler(SimpleHTTPRequestHandler): # lgtm[py/missing-call-to-init]
class HeaderRequestHandler(SimpleHTTPRequestHandler):
cors_origins = DEFAULT_CORS_ORIGINS
def __init__(self, request, client_address, server):
"""."""
"""Handle a request for a header below the working trees' root directory."""
self.worktrees = server.worktrees
self.worktree = None
try:
@@ -336,7 +338,7 @@ class HeaderRequestHandler(SimpleHTTPRequestHandler): # lgtm[py/missing-call-to-
class DualStackServer(ThreadingHTTPServer):
def __init__(self, addr, worktrees):
"""."""
"""Serve the headers of worktrees on addr."""
self.worktrees = worktrees
super().__init__(addr, HeaderRequestHandler)
@@ -349,8 +351,6 @@ class DualStackServer(ThreadingHTTPServer):
if __name__ == '__main__':
import argparse
import ssl
import socket
import yaml
# exit code