Compare commits

...
6 Commits
Author SHA1 Message Date
Niels LohmannandDylan Baker ed8ba0201f Install CMake package config files with Meson, and add Meson options (#5587)
* Install CMake package config files with Meson, and add Meson options

Squashed onto develop from:
- Install CMake package config files with Meson
- meson: Indent code inside an if block
- meson: set a minimum Meson version
- meson: use `override_dependency()` to set dependencies
- meson: use `install_subdir` for headers
- meson: set the C++ standard to C++11
- meson: handle single header and multiheader the same way CMake does
- meson: add support for the GlobalUDLs option
- meson: add support for the ImplictConversions option
- Add meson information to FILES.md
- Complete the Meson options and match CMake's compile definitions
- Add the JSON_* definitions to the CMake pkg-config file
- Document the Meson options and check them in CI
- Fix the Meson CMake target for includedir or datadir outside the prefix
- Avoid //include in Meson-generated CMake target for prefix /
- Add DisableTupleReferenceConversion to Meson and the CMake pkg-config file
- Check in CI that Meson and pkg-config offer the CMake options
- Remove accidentally committed Python bytecode

Co-authored-by: Dylan Baker <dylan@pnwbakers.com>
Signed-off-by: Dylan Baker <dylan@pnwbakers.com>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Add the StrictBinaryUTF8 and DeleteDeprecatedFunctions Meson options

JSON_StrictBinaryUTF8 (#5741) and JSON_DeleteDeprecatedFunctions (#5755)
arrived with develop but were missing from the Meson build and the
pkg-config file, so check_build_options failed. Both Meson options
default to false and add JSON_STRICT_BINARY_UTF8=1 and
JSON_DELETE_DEPRECATED_FUNCTIONS=1 to the dependency, the pkg-config
file, and the generated CMake target; CMake's pkg-config file now
carries both definitions too. The ci_meson_install job sets and checks
them in its non-default install.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Dylan Baker <dylan@pnwbakers.com>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Co-authored-by: Dylan Baker <dylan@pnwbakers.com>
2026-10-07 22:20:05 +02:00
Niels LohmannandAfonso Januário a5f5d3059b Make std::hash<basic_json> consistent with operator== for numbers (#5772)
* Make std::hash<basic_json> consistent with operator== for numbers

operator== converts between number_integer, number_unsigned, and
number_float before comparing, so json(0), json(0U), and json(0.0)
all compare equal. hash() folded the specific value_t into the
result for each of the three numeric cases, giving each a distinct
hash and breaking the standard Hash requirement that a == b implies
hash(a) == hash(b). A std::unordered_set could therefore hold all
three as separate elements even though they compare equal.

hash() now treats all three numeric variants the same way: it
converts the value to number_float_t and combines it with a single
shared type tag, so any two numbers operator== considers equal hash
identically regardless of which internal type actually holds them.

Updated the accompanying test to check this consistency directly
(including via an actual unordered_set) instead of asserting that 0,
0U, and 0.0 hash differently, since that assumption was the bug.
Also corrected the function's own doc comment and the std::hash API
docs, which described the old behavior as intended.

Fixes #5400

Signed-off-by: Afonso Januário <afonso-januario@hotmail.com>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Remove now-unused number_integer_t/number_unsigned_t typedefs in hash()

Merging the three numeric branches into one that only reads
number_float_t left these two aliases unused, which several CI
configurations treat as a build error under -Wunused-local-typedefs.

Signed-off-by: Afonso Januário <afonso-januario@hotmail.com>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Mark the unordered_set in the hash regression test const

clang-tidy's misc-const-correctness check flagged it: the set is
never mutated after construction, only read via size().

Signed-off-by: Afonso Januário <afonso-januario@hotmail.com>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Normalize -0.0 in number hashes and test range ends

operator== compares numbers exactly since #5459, so equal numbers
share one value and convert to the same number_float_t. Update the
comment accordingly, map -0.0 to 0.0 before hashing (std::hash need
not do that), and test -0.0 and the ends of the integer ranges. Show
hash(0.0) in the docs example and note the change in the version
history.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Clarify hash documentation after review

- Say "may hash differently" for null, false, and numbers, since a
  collision across types is possible.
- Name the storage types (signed integer, unsigned integer,
  floating-point number) instead of example literals.
- Explain that the hash survives converting an integer to
  number_float_t but not the lossy conversion back, and that unequal
  numbers may share a hash.
- State that the example hash values are illustrative only and vary by
  platform, compiler, compiler version, and library version.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Afonso Januário <afonso-januario@hotmail.com>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Co-authored-by: Afonso Januário <afonso-januario@hotmail.com>
2026-10-07 19:18:51 +02:00
43fe8928e1 Stop binary writers overflowing the stack on deep values (#5781)
* Stop binary writers overflowing the stack on deep values

to_cbor, to_msgpack, and to_ubjson recurse once per nesting level.
The parser is iterative, so a value the library accepts can crash on
the way back out.

Keep the existing recursive path for the first 128 levels and finish
anything deeper on a heap stack. Output is unchanged. BSON is left
alone because its extra size walk is a separate change.

Rebased onto the value-type output sink. The heap frames now initialize
every member, which is what -Weffc++ was rejecting.

See #5392.

Signed-off-by: ayush-singh-0601 <singhayush062006@gmail.com>
(cherry picked from commit cf65ac438f)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Redesign the iterative binary writers around a shared recursion depth limit

Address the open review on the non-recursive CBOR/MessagePack/UBJSON/BJData
writers (#5518):

- Delete the CBOR array/object prefix helpers; both the recursive and
  iterative paths call write_cbor_head(), which already existed on develop.
- MessagePack: share one write_msgpack_array_prefix()/write_msgpack_object_prefix()
  helper per container kind between the recursive and iterative paths, both
  going through to_msgpack_length() so an over-long container throws
  out_of_range.412 identically either way.
- Reuse detail::recursion_depth_limit() instead of a separate constant, the
  same bound serializer::dump() and write_bson_document() already use.
- Redesign the frames after bson_frame/dump_frame: only a container with
  elements is ever pushed, its header is written at the point it is pushed,
  and the iterator is set in the frame's constructor instead of a
  default-then-assign two-step with a since-removed "started" flag. The
  UBJSON frame keeps only the value pointer, the per-element prefix_required
  flag, and the iterator; write_closer and is_object are no longer stored,
  since the former is always !use_count (use_count is constant for the whole
  document) and the latter follows from value->is_object().
- Factor the BJData ND-array shape check into is_bjdata_ndarray(), used by
  both the recursive object case and the iterative pushing logic.
- Give the frame classes the GCC -Weffc++ treatment already used for
  diff_frame: a noexcept converting constructor plus the five special members
  defaulted with no explicit noexcept.
- Fix two @ref self-references in write_cbor/write_msgpack/write_ubjson's own
  doc comments to point at the public to_cbor/to_msgpack/to_ubjson/to_bjdata
  API instead.
- The iterative object-key write for CBOR/MessagePack now runs the same
  strict-mode check_utf8() against the parent object as diagnostics context
  that the recursive path already ran, so the two paths raise identical
  diagnostics across the switch-over.
- Rewrite the tests: round trips instead of a bare size check, byte-exact
  comparisons against the recursive output at depths around the bound, a
  deep object and a BJData ND-array past the bound, a deep discarded value
  (type_error.321), and the OSS-Fuzz 566583014 CBOR/MessagePack regression.

BSON is unaffected by this change; it already walks its documents
iteratively and is covered separately by #5553.

Co-authored-by: ayush-singh-0601 <179524189+ayush-singh-0601@users.noreply.github.com>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: ayush-singh-0601 <singhayush062006@gmail.com>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Co-authored-by: ayush-singh-0601 <singhayush062006@gmail.com>
Co-authored-by: ayush-singh-0601 <179524189+ayush-singh-0601@users.noreply.github.com>
2026-10-07 19:18:20 +02:00
Niels Lohmann 3c465beb61 Add tests for error_handler_t::keep in dump() (#4555)
Squashed onto develop from:
- Add error_handler_t::keep to copy invalid UTF-8 bytes unchanged
- Mention error_handler_t::keep in the README

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-07 19:17:57 +02:00
Niels LohmannandMihnea Magheru e752688b52 Add a natvis fallback visualizer for detail::json_default_base (#5588)
* Add a natvis fallback visualizer for detail::json_default_base

Squashed onto develop from:
- Add a type in the natvis template for detail::json_default_base
- Document the json_default_base natvis fallback and regenerate natvis
- Match json_default_base in both its current and 3.12.0 namespace

Co-authored-by: Mihnea Magheru <sakuntalle@yahoo.com>
Signed-off-by: Mihnea Magheru <sakuntalle@yahoo.com>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>

* Take the natvis template from the PR in the amalgamation check

The check ran generate_natvis.py from a develop checkout, which loads
nlohmann_json.natvis.j2 from its own directory. A PR that changes the
template was therefore checked against develop's template and always
failed. Copy the PR's template next to the develop script before
running it.

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

---------

Signed-off-by: Mihnea Magheru <sakuntalle@yahoo.com>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Co-authored-by: Mihnea Magheru <sakuntalle@yahoo.com>
2026-10-07 19:17:41 +02:00
Niels Lohmann 069ace74af Round-trip BJData ND-array annotations exactly (single precision, key order) (#5707)
Squashed onto develop from:
- Round-trip BJData ND-array annotations exactly (single precision, key order)

Signed-off-by: Niels Lohmann <mail@nlohmann.me>
2026-10-07 19:17:18 +02:00
32 changed files with 24785 additions and 372 deletions

No files matched your search

+7
View File
@@ -149,6 +149,13 @@ jobs:
VERSION_MAJOR=$(grep -m1 'define NLOHMANN_JSON_VERSION_MAJOR' $ABI_MACROS | grep -o '[0-9]\+')
VERSION_MINOR=$(grep -m1 'define NLOHMANN_JSON_VERSION_MINOR' $ABI_MACROS | grep -o '[0-9]\+')
VERSION_PATCH=$(grep -m1 'define NLOHMANN_JSON_VERSION_PATCH' $ABI_MACROS | grep -o '[0-9]\+')
# The template is what this check verifies the committed natvis
# file against, so take it from this PR rather than from the
# develop checkout: generate_natvis.py loads the template from its
# own directory, and with develop's copy a PR that changes the
# template could never pass. Like the macro_builder step above,
# this only renders PR content in a job without credentials.
cp tools/generate_natvis/nlohmann_json.natvis.j2 $NATVIS_TOOL_DIR/
python3 $NATVIS_TOOL_DIR/generate_natvis.py --version "$VERSION_MAJOR.$VERSION_MINOR.$VERSION_PATCH" $MAIN_DIR
- name: Build patch and check for differences
+45
View File
@@ -31,6 +31,51 @@ jobs:
- name: Build
run: cmake --build build --target ci_test_gcc
ci_meson_install:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Get latest CMake and ninja
uses: lukka/get-cmake@fffaaafeea488556c2c12dad60690008bc1caacb # v4.4.2
- name: Check that Meson and pkg-config offer the CMake options
run: make check_build_options
- name: Install Meson
run: pip install meson
- name: Install with Meson
run: |
meson setup build-meson --prefix=${{ github.workspace }}/install
meson install -C build-meson
- name: Use the installed package with find_package
run: |
cmake -S tests/cmake_import/project -B build-import -DCMAKE_PREFIX_PATH=${{ github.workspace }}/install
cmake --build build-import
- name: Install with Meson and non-default options
run: |
meson setup build-meson-options --prefix=${{ github.workspace }}/install-options -DMultipleHeaders=true -DDiagnostics=true -DGlobalUDLs=false -DDisableTupleReferenceConversion=true -DStrictBinaryUTF8=true -DDeleteDeprecatedFunctions=true
meson install -C build-meson-options
- name: Check that the options reach the installed files
run: |
test -d install-options/include/nlohmann/detail
cflags=$(PKG_CONFIG_PATH=${{ github.workspace }}/install-options/share/pkgconfig pkg-config --cflags nlohmann_json)
echo "$cflags"
echo "$cflags" | grep -q -- '-DJSON_DIAGNOSTICS=1'
echo "$cflags" | grep -q -- '-DJSON_USE_GLOBAL_UDLS=0'
echo "$cflags" | grep -q -- '-DJSON_DISABLE_TUPLE_REFERENCE_CONVERSION=1'
echo "$cflags" | grep -q -- '-DJSON_STRICT_BINARY_UTF8=1'
echo "$cflags" | grep -q -- '-DJSON_DELETE_DEPRECATED_FUNCTIONS=1'
grep -q 'JSON_USE_GLOBAL_UDLS=0;JSON_DISABLE_TUPLE_REFERENCE_CONVERSION=1;JSON_DIAGNOSTICS=1' install-options/share/cmake/nlohmann_json/nlohmann_jsonTargets.cmake
grep -q 'JSON_STRICT_BINARY_UTF8=1;JSON_DELETE_DEPRECATED_FUNCTIONS=1' install-options/share/cmake/nlohmann_json/nlohmann_jsonTargets.cmake
cmake -S tests/cmake_import/project -B build-import-options -DCMAKE_PREFIX_PATH=${{ github.workspace }}/install-options
cmake --build build-import-options
- name: Install with Meson and the include directory outside the prefix
run: |
meson setup build-meson-split --prefix=${{ github.workspace }}/install-split --includedir=${{ github.workspace }}/install-split-dev/include
meson install -C build-meson-split
cmake -S tests/cmake_import/project -B build-import-split -DCMAKE_PREFIX_PATH=${{ github.workspace }}/install-split
cmake --build build-import-split
ci_infer:
runs-on: ubuntu-latest
steps:
+33 -1
View File
@@ -189,7 +189,39 @@ if (MSVC)
)
endif()
# Install a pkg-config file, so other tools can find this.
# Install a pkg-config file, so other tools can find this. It carries the same
# compile definitions as the target above.
set(NLOHMANN_JSON_PKGCONFIG_CFLAGS "")
if (NOT JSON_GlobalUDLs)
string(APPEND NLOHMANN_JSON_PKGCONFIG_CFLAGS " -DJSON_USE_GLOBAL_UDLS=0")
endif()
if (NOT JSON_ImplicitConversions)
string(APPEND NLOHMANN_JSON_PKGCONFIG_CFLAGS " -DJSON_USE_IMPLICIT_CONVERSIONS=0")
endif()
if (JSON_DisableEnumSerialization)
string(APPEND NLOHMANN_JSON_PKGCONFIG_CFLAGS " -DJSON_DISABLE_ENUM_SERIALIZATION=1")
endif()
if (JSON_DisableTupleReferenceConversion)
string(APPEND NLOHMANN_JSON_PKGCONFIG_CFLAGS " -DJSON_DISABLE_TUPLE_REFERENCE_CONVERSION=1")
endif()
if (JSON_Diagnostics)
string(APPEND NLOHMANN_JSON_PKGCONFIG_CFLAGS " -DJSON_DIAGNOSTICS=1")
endif()
if (JSON_Diagnostic_Positions)
string(APPEND NLOHMANN_JSON_PKGCONFIG_CFLAGS " -DJSON_DIAGNOSTIC_POSITIONS=1")
endif()
if (JSON_LegacyDiscardedValueComparison)
string(APPEND NLOHMANN_JSON_PKGCONFIG_CFLAGS " -DJSON_USE_LEGACY_DISCARDED_VALUE_COMPARISON=1")
endif()
if (JSON_StrictNulHandling)
string(APPEND NLOHMANN_JSON_PKGCONFIG_CFLAGS " -DJSON_STRICT_NUL_HANDLING=1")
endif()
if (JSON_StrictBinaryUTF8)
string(APPEND NLOHMANN_JSON_PKGCONFIG_CFLAGS " -DJSON_STRICT_BINARY_UTF8=1")
endif()
if (JSON_DeleteDeprecatedFunctions)
string(APPEND NLOHMANN_JSON_PKGCONFIG_CFLAGS " -DJSON_DELETE_DEPRECATED_FUNCTIONS=1")
endif()
configure_file(
"${CMAKE_CURRENT_SOURCE_DIR}/cmake/pkg-config.pc.in"
"${CMAKE_CURRENT_BINARY_DIR}/${PROJECT_NAME}.pc"
+24 -2
View File
@@ -249,9 +249,31 @@ make BUILD.bazel
The "Check amalgamation" workflow fails if the file is out of date.
### `meson.build`
### `meson.build` and `meson_options.txt`
The build definition for the [Meson](https://mesonbuild.com) build system.
Meson build definitions suitable for use as a subproject ("wrap" in Meson terminology).
Projects wishing to use the wrap can execute:
```sh
meson wrap install nlohmann_json
```
Which allows Meson to build from source when a system provided dependency isn't available.
To build directly:
```sh
meson setup builddir
ninja -C builddir
```
`meson_options.txt` defines the options, which mirror the CMake options that change the library's target (for example,
`-DDiagnostics=true`). Meson requires this file next to `meson.build`, so it is also part of `include.zip`. `make check_build_options`
([`tools/check_build_options`](tools/check_build_options/README.md)) checks in CI that both files and the pkg-config files
stay in sync with the CMake options.
When installing, `meson.build` installs the headers, a pkg-config file, and the CMake package config files, so that
`find_package(nlohmann_json)` works. As Meson cannot generate `nlohmann_jsonTargets.cmake` itself, it is created from
the template `cmake/nlohmann_jsonTargets.cmake.in`, which is only used by Meson.
### `Package.swift`
+6 -2
View File
@@ -1,4 +1,4 @@
.PHONY: pretty clean ChangeLog.md release update_hedley update_hedley_undef BUILD.bazel natvis macro_builder_check
.PHONY: pretty clean ChangeLog.md release update_hedley update_hedley_undef BUILD.bazel natvis macro_builder_check check_build_options
##########################################################################
# configuration
@@ -127,6 +127,10 @@ macro_builder_check:
diff "$$TMPDIR/paste.hpp" "$$TMPDIR/paste_actual.hpp" || (echo "===================================================================\n $(MACRO_SCOPE_HPP) (NLOHMANN_JSON_EXPAND..NLOHMANN_JSON_DOUBLE_PASTE63) is out of date!\n Regenerate it, see tools/macro_builder/README.md.\n===================================================================" ; exit 1); \
diff "$$TMPDIR/type_body.hpp" "$$TMPDIR/type_body_actual.hpp" || (echo "===================================================================\n $(MACRO_SCOPE_HPP) (NLOHMANN_JSON_TYPE_BODY) is out of date!\n Regenerate it, see tools/macro_builder/README.md.\n===================================================================" ; exit 1)
# check that the Meson build and the pkg-config files offer the options of the CMake target
check_build_options:
python3 tools/check_build_options/check_build_options.py .
# check if file single_include/nlohmann/json.hpp has been amalgamated from the nlohmann sources
check-amalgamation:
@mv $(AMALGAMATED_FILE) $(AMALGAMATED_FILE)~
@@ -184,7 +188,7 @@ json.tar.xz:
# We use `-X` to make the resulting ZIP file reproducible, see
# <https://content.pivotal.io/blog/barriers-to-deterministic-reproducible-zip-files>.
include.zip: BUILD.bazel
zip -9 --recurse-paths -X include.zip $(SRCS) $(AMALGAMATED_FILE) $(AMALGAMATED_FWD_FILE) $(AMALGAMATED_LITERALS_FILE) BUILD.bazel MODULE.bazel meson.build LICENSE.MIT
zip -9 --recurse-paths -X include.zip $(SRCS) $(AMALGAMATED_FILE) $(AMALGAMATED_FWD_FILE) $(AMALGAMATED_LITERALS_FILE) BUILD.bazel MODULE.bazel meson.build meson_options.txt LICENSE.MIT
# Create the files for a release and add signatures and hashes.
release: include.zip json.tar.xz
+2 -2
View File
@@ -361,7 +361,7 @@ std::cout << j_string << " == " << serialized_string << std::endl;
[`.dump()`](https://json.nlohmann.me/api/basic_json/dump/) returns the originally stored string value.
Note the library only supports UTF-8. When you store strings with different encodings in the library, calling [`dump()`](https://json.nlohmann.me/api/basic_json/dump/) may throw an exception unless `json::error_handler_t::replace` or `json::error_handler_t::ignore` are used as error handlers.
Note the library only supports UTF-8. When you store strings with different encodings in the library, calling [`dump()`](https://json.nlohmann.me/api/basic_json/dump/) may throw an exception unless `json::error_handler_t::replace`, `json::error_handler_t::ignore`, or `json::error_handler_t::keep` are used as error handlers.
#### To/from streams (e.g., files, string streams)
@@ -1917,7 +1917,7 @@ The library supports **Unicode input** as follows:
- [Unicode noncharacters](https://www.unicode.org/faq/private_use.html#nonchar1) will not be replaced by the library.
- Invalid surrogates (e.g., incomplete pairs such as `\uDEAD`) will yield parse errors.
- The strings stored in the library are UTF-8 encoded. When using the default string type (`std::string`), note that its length/size functions return the number of stored bytes rather than the number of characters or glyphs.
- When you store strings with different encodings in the library, calling [`dump()`](https://json.nlohmann.me/api/basic_json/dump/) may throw an exception unless `json::error_handler_t::replace` or `json::error_handler_t::ignore` are used as error handlers.
- When you store strings with different encodings in the library, calling [`dump()`](https://json.nlohmann.me/api/basic_json/dump/) may throw an exception unless `json::error_handler_t::replace`, `json::error_handler_t::ignore`, or `json::error_handler_t::keep` are used as error handlers.
- To store wide strings (e.g., `std::wstring`), you need to convert them to a UTF-8 encoded `std::string` before, see [an example](https://json.nlohmann.me/home/faq/#wide-string-handling).
### Comments in JSON
+42
View File
@@ -0,0 +1,42 @@
# Imported target for installations made with Meson (see meson.build).
#
# CMake installations generate this file with install(EXPORT ...). Meson cannot
# do that, but as the library is header-only, the target only needs an include
# directory, the C++ standard, and the compile definitions of the options that
# differ from their defaults. Paths are computed relative to this file so that
# the installation can be relocated (e.g., into a sysroot), unless includedir or
# datadir is outside the prefix.
if(TARGET @PROJECT_NAME@::@NLOHMANN_JSON_TARGET_NAME@)
return()
endif()
get_filename_component(_IMPORT_PREFIX "${CMAKE_CURRENT_LIST_DIR}/@NLOHMANN_JSON_CONFIG_TO_PREFIX@" ABSOLUTE)
# As in CMake's generated file: avoid "//include" for an installation to "/".
if(_IMPORT_PREFIX STREQUAL "/")
set(_IMPORT_PREFIX "")
endif()
add_library(@PROJECT_NAME@::@NLOHMANN_JSON_TARGET_NAME@ INTERFACE IMPORTED)
set_target_properties(@PROJECT_NAME@::@NLOHMANN_JSON_TARGET_NAME@ PROPERTIES
INTERFACE_INCLUDE_DIRECTORIES "@NLOHMANN_JSON_INCLUDE_DIR@"
)
if(CMAKE_VERSION VERSION_LESS 3.8)
set_target_properties(@PROJECT_NAME@::@NLOHMANN_JSON_TARGET_NAME@ PROPERTIES
INTERFACE_COMPILE_FEATURES cxx_range_for
)
else()
set_target_properties(@PROJECT_NAME@::@NLOHMANN_JSON_TARGET_NAME@ PROPERTIES
INTERFACE_COMPILE_FEATURES cxx_std_11
)
endif()
set(_NLOHMANN_JSON_COMPILE_DEFINITIONS "@NLOHMANN_JSON_COMPILE_DEFINITIONS@")
if(_NLOHMANN_JSON_COMPILE_DEFINITIONS)
set_target_properties(@PROJECT_NAME@::@NLOHMANN_JSON_TARGET_NAME@ PROPERTIES
INTERFACE_COMPILE_DEFINITIONS "${_NLOHMANN_JSON_COMPILE_DEFINITIONS}"
)
endif()
unset(_NLOHMANN_JSON_COMPILE_DEFINITIONS)
unset(_IMPORT_PREFIX)
+1 -1
View File
@@ -4,4 +4,4 @@ includedir=${prefix}/@CMAKE_INSTALL_INCLUDEDIR@
Name: @PROJECT_NAME@
Description: JSON for Modern C++
Version: @PROJECT_VERSION@
Cflags: -I${includedir}
Cflags: -I${includedir}@NLOHMANN_JSON_PKGCONFIG_CFLAGS@
+12 -3
View File
@@ -7,8 +7,14 @@ namespace std {
```
Return a hash value for a JSON object. The hash function tries to rely on `std::hash` where possible. Furthermore, the
type of the JSON value is taken into account to have different hash values for `#!json null`, `#!cpp 0`, `#!cpp 0U`, and
`#!cpp false`, etc.
type of the JSON value is taken into account, so `#!json null`, `#!cpp false`, and numbers may hash differently from
each other. Numbers that compare equal under [`operator==`](operator_eq.md) always hash equally, regardless of
whether they are stored as signed integer, unsigned integer, or floating-point number.
Numbers are hashed by their value converted to `number_float_t`. Converting an integer to `number_float_t` therefore
keeps its hash, but converting a floating-point number to an integer type is lossy and can change it: `#!cpp 0.5`
converts to `#!cpp 0`, which need not have the same hash. Unequal numbers can also share a hash value, for example two
large integers that convert to the same `number_float_t`.
## Examples
@@ -26,7 +32,8 @@ type of the JSON value is taken into account to have different hash values for `
--8<-- "examples/std_hash.output"
```
Note the output is platform-dependent.
The hash values shown are examples only. They depend on the platform, the compiler, and the compiler version, and
they can change between versions of this library. Do not persist them or rely on specific values.
## See also
@@ -36,3 +43,5 @@ type of the JSON value is taken into account to have different hash values for `
- Added in version 1.0.0.
- Extended for arbitrary basic_json types in version 3.10.5.
- Numbers that compare equal hash equally since version 3.13.0; before, `#!cpp 0`, `#!cpp 0U`, and `#!cpp 0.0` had
different hash values.
+1
View File
@@ -11,6 +11,7 @@ int main()
<< "hash(false) = " << std::hash<json> {}(json(false)) << '\n'
<< "hash(0) = " << std::hash<json> {}(json(0)) << '\n'
<< "hash(0U) = " << std::hash<json> {}(json(0U)) << '\n'
<< "hash(0.0) = " << std::hash<json> {}(json(0.0)) << '\n'
<< "hash(\"\") = " << std::hash<json> {}(json("")) << '\n'
<< "hash({}) = " << std::hash<json> {}(json::object()) << '\n'
<< "hash([]) = " << std::hash<json> {}(json::array()) << '\n'
+5 -4
View File
@@ -1,8 +1,9 @@
hash(null) = 2654435769
hash(false) = 2654436030
hash(0) = 2654436095
hash(0U) = 2654436156
hash("") = 6142509191626859748
hash(0) = 2654436221
hash(0U) = 2654436221
hash(0.0) = 2654436221
hash("") = 11160318156688833227
hash({}) = 2654435832
hash([]) = 2654435899
hash({"hello": "world"}) = 4469488738203676328
hash({"hello": "world"}) = 3701319991624763853
@@ -141,8 +141,14 @@ The library uses the following mapping from JSON values types to BJData types ac
parsed back as a regular array,
- every entry of `"_ArraySize_"` is a positive integer, and their product is representable as a `std::size_t`,
- `"_ArrayData_"` is an array holding exactly that many elements, and
- every element of `"_ArrayData_"` is a number of the kind named by `"_ArrayType_"` (a floating-point number for
`single` and `double`, an integer otherwise).
- every element of `"_ArrayData_"` is a number of the kind named by `"_ArrayType_"`: for the integer types, a
value that fits the named width; for `double`, any value; for `single`, a value that survives narrowing to
`float` and back without change (for instance, `0.1` does not, since it is not exactly representable as
`float`).
An annotated object is always read back with its keys in the order shown above, `"_ArrayType_"`, `"_ArraySize_"`,
`"_ArrayData_"`, regardless of the order the ND-array's header stores them in on the wire. This matters for
`ordered_json`, whose comparison takes key order into account.
The current version of this library does not yet support automatic detection of and conversion from a nested JSON
array input to a BJData ND-array.
@@ -64,6 +64,7 @@ serialization fails by default. The fourth argument of `dump` selects an
- `strict` (default) — throw a [`type_error.316`](../home/exceptions.md#jsonexceptiontype_error316) exception.
- `replace` — replace invalid bytes with the Unicode replacement character U+FFFD (`�`).
- `ignore` — silently drop invalid bytes.
- `keep` — copy invalid bytes to the output unchanged; the result is not valid UTF-8.
??? example "Example: serialize invalid UTF-8 with different error handlers"
+1
View File
@@ -771,6 +771,7 @@ as well for a string value or object key that is not valid UTF-8 if their `error
- Pass an error handler as last parameter to the `dump()` function to avoid this exception:
- `json::error_handler_t::replace` will replace invalid bytes sequences with `U+FFFD`
- `json::error_handler_t::ignore` will silently ignore invalid byte sequences
- `json::error_handler_t::keep` will copy invalid byte sequences to the output unchanged
### json.exception.type_error.317
+1 -1
View File
@@ -85,7 +85,7 @@ The library supports **Unicode input** as follows:
- The library will not replace [Unicode noncharacters](http://www.unicode.org/faq/private_use.html#nonchar1).
- Invalid surrogates (e.g., incomplete pairs such as `\uDEAD`) will yield parse errors.
- The strings stored in the library are UTF-8 encoded. When using the default string type (`std::string`), note that its length/size functions return the number of stored bytes rather than the number of characters or glyphs.
- When you store strings with different encodings in the library, calling [`dump()`](../api/basic_json/dump.md) may throw an exception unless `json::error_handler_t::replace` or `json::error_handler_t::ignore` are used as error handlers.
- When you store strings with different encodings in the library, calling [`dump()`](../api/basic_json/dump.md) may throw an exception unless `json::error_handler_t::replace`, `json::error_handler_t::ignore`, or `json::error_handler_t::keep` are used as error handlers.
In most cases, the parser is right to complain, because the input is not UTF-8 encoded. This is especially true for Microsoft Windows, where Latin-1 or ISO 8859-1 is often the standard encoding.
@@ -125,10 +125,24 @@ meson wrap install nlohmann_json
Please see the Meson project for any issues regarding the packaging.
The provided `meson.build` can also be used as an alternative to CMake for installing `nlohmann_json` system-wide in
which case a [pkg-config](pkg-config.md) file is installed. To use it, have your build system require the
`nlohmann_json` pkg-config dependency. In Meson, it is preferred to use the
[`dependency()`](https://mesonbuild.com/Reference-manual.html#dependency) object with a subproject fallback, rather than
using the subproject directly.
which case a [pkg-config](pkg-config.md) file and the CMake package config files are installed. To use it, have your build system require
the `nlohmann_json` pkg-config dependency, or use [`find_package(nlohmann_json)`](cmake.md#external) in CMake. In Meson,
it is preferred to use the [`dependency()`](https://mesonbuild.com/Reference-manual.html#dependency) object with a
subproject fallback, rather than using the subproject directly.
The options that change the library's configuration are available in Meson as well, named like the
[CMake options](cmake.md#cmake-options) without the `JSON_` prefix: `MultipleHeaders`, `GlobalUDLs`,
`ImplicitConversions`, `DisableEnumSerialization`, `DisableTupleReferenceConversion`, `Diagnostics`,
`Diagnostic_Positions`, `LegacyDiscardedValueComparison`, `StrictNulHandling`, `StrictBinaryUTF8`, and
`DeleteDeprecatedFunctions`. They have the same defaults as in CMake, except that
`MultipleHeaders` is `false`. Set them with `-D` when setting up the build, or with the subproject name as prefix when
the library is used as a subproject:
```shell
meson setup build -Dnlohmann_json:Diagnostics=true
```
The resulting compile definitions are part of the Meson dependency, the pkg-config file, and the CMake target.
??? example "Example: Wrap"
+19 -16
View File
@@ -35,8 +35,10 @@ std::size_t hash_iteratively(const BasicJsonType& j);
@brief hash a JSON value
The hash function tries to rely on std::hash where possible. Furthermore, the
type of the JSON value is taken into account to have different hash values for
null, 0, 0U, and false, etc.
type of the JSON value is taken into account, so null, false, and numbers may
hash differently from each other, but any two numbers that compare equal
under operator== hash equally regardless of which of number_integer,
number_unsigned, or number_float actually holds the value.
Hashing an array or an object hashes its elements, which used to call this
function again once per nesting level, so a value nested deeply enough
@@ -55,8 +57,6 @@ template<typename BasicJsonType>
std::size_t hash(const BasicJsonType& j, const std::size_t depth = 0)
{
using string_t = typename BasicJsonType::string_t;
using number_integer_t = typename BasicJsonType::number_integer_t;
using number_unsigned_t = typename BasicJsonType::number_unsigned_t;
using number_float_t = typename BasicJsonType::number_float_t;
const auto type = static_cast<std::size_t>(j.type());
@@ -113,21 +113,24 @@ std::size_t hash(const BasicJsonType& j, const std::size_t depth = 0)
}
case BasicJsonType::value_t::number_integer:
{
const auto h = std::hash<number_integer_t> {}(j.template get<number_integer_t>());
return combine(type, h);
}
case BasicJsonType::value_t::number_unsigned:
{
const auto h = std::hash<number_unsigned_t> {}(j.template get<number_unsigned_t>());
return combine(type, h);
}
case BasicJsonType::value_t::number_float:
{
const auto h = std::hash<number_float_t> {}(j.template get<number_float_t>());
return combine(type, h);
// operator== compares numbers by their mathematical value across
// number_integer, number_unsigned, and number_float, so equal
// numbers of different internal types (0, 0U, 0.0) must hash the
// same. Two equal numbers have the same value, which converts to
// the same number_float_t, so all numbers share one type tag and
// hash that converted value. Adding zero turns -0.0 (equal to 0)
// into 0.0, as std::hash need not map both to the same hash.
// The converse does not hold: converting a number_float_t value
// to an integer type is lossy, so the result can hash
// differently, and unequal numbers that convert to the same
// number_float_t (e.g., 2^53 and 2^53 + 1) share a hash.
const auto number_type = static_cast<std::size_t>(BasicJsonType::value_t::number_float);
const auto value = j.template get<number_float_t>() + static_cast<number_float_t>(0);
const auto h = std::hash<number_float_t> {}(value);
return combine(number_type, h);
}
case BasicJsonType::value_t::binary:
+39 -18
View File
@@ -2716,10 +2716,15 @@ class binary_reader
is_ndarray can only return `true` when its initial value
is `false`
@param[in] prefix type marker if already read, otherwise set to 0
@param[in] ndarray_dtype the element type marker of the enclosing bjdata ndarray if
already known (it precedes the dimension vector read here),
otherwise 0; used to emit the "_ArrayType_" annotation key
before "_ArraySize_" if a dimension vector turns out to
describe an ndarray
@return whether size determination completed
*/
bool get_ubjson_size_value(std::size_t& result, bool& is_ndarray, char_int_type prefix = 0)
bool get_ubjson_size_value(std::size_t& result, bool& is_ndarray, char_int_type prefix = 0, char_int_type ndarray_dtype = 0)
{
if (prefix == 0)
{
@@ -2832,8 +2837,34 @@ class binary_reader
}
}
if (JSON_HEDLEY_UNLIKELY(!sax->start_object(3)))
{
return false;
}
// the element type precedes the dimension vector (see get_ubjson_size_type)
// and is passed down as ndarray_dtype; emit it here so the annotation keys
// follow the documented _ArrayType_, _ArraySize_, _ArrayData_ order
if (ndarray_dtype != 0)
{
const char* type_name = bjd_type_name(ndarray_dtype);
if (JSON_HEDLEY_UNLIKELY(type_name == nullptr))
{
auto last_token = get_token_string();
return sax->parse_error(chars_read, last_token, parse_error::create(112, chars_read,
exception_message("invalid byte: 0x" + last_token, "type"), nullptr));
}
string_t type_key = "_ArrayType_";
string_t type = type_name; // sax->string() takes a reference
if (JSON_HEDLEY_UNLIKELY(!sax->key(type_key) || !sax->string(type)))
{
return false;
}
}
string_t key = "_ArraySize_";
if (JSON_HEDLEY_UNLIKELY(!sax->start_object(3) || !sax->key(key) || !sax->start_array(dim.size())))
if (JSON_HEDLEY_UNLIKELY(!sax->key(key) || !sax->start_array(dim.size())))
{
return false;
}
@@ -2941,7 +2972,7 @@ class binary_reader
exception_message(concat("expected '#' after type information; last byte: 0x", last_token), "size"), nullptr));
}
const bool is_error = get_ubjson_size_value(result.first, is_ndarray);
const bool is_error = get_ubjson_size_value(result.first, is_ndarray, 0, result.second);
// an ndarray was read here only if the flag flipped; when it was
// seeded true, get_ubjson_size_value() already rejected the nested
// dimension vector
@@ -3134,27 +3165,17 @@ class binary_reader
if (input_format == input_format_t::bjdata && size_and_type.first != npos && (size_and_type.second & (1 << 8)) != 0)
{
size_and_type.second &= ~(static_cast<char_int_type>(1) << 8); // use bit 8 to indicate ndarray, here we remove the bit to restore the type marker
const char* type_name = bjd_type_name(size_and_type.second);
string_t key = "_ArrayType_";
if (JSON_HEDLEY_UNLIKELY(type_name == nullptr))
{
auto last_token = get_token_string();
return sax->parse_error(chars_read, last_token, parse_error::create(112, chars_read,
exception_message("invalid byte: 0x" + last_token, "type"), nullptr));
}
string_t type = type_name; // sax->string() takes a reference
if (JSON_HEDLEY_UNLIKELY(!sax->key(key) || !sax->string(type)))
{
return false;
}
// the "_ArrayType_" and "_ArraySize_" annotation keys were already emitted by
// get_ubjson_size_value() (the type marker is known before the dimension vector
// that determines size_and_type.first is read, so it is emitted first there to
// match the documented _ArrayType_, _ArraySize_, _ArrayData_ key order)
if (size_and_type.second == 'C' || size_and_type.second == 'B')
{
size_and_type.second = 'U';
}
key = "_ArrayData_";
string_t key = "_ArrayData_";
if (JSON_HEDLEY_UNLIKELY(!sax->key(key) || !sax->start_array(size_and_type.first) ))
{
return false;
+589 -130
View File
@@ -28,6 +28,7 @@
#include <nlohmann/detail/macro_scope.hpp>
#include <nlohmann/detail/output/error_handler.hpp>
#include <nlohmann/detail/output/output_adapters.hpp>
#include <nlohmann/detail/recursion_depth_limit.hpp>
#include <nlohmann/detail/string_concat.hpp>
#include <nlohmann/detail/string_utils.hpp>
@@ -156,13 +157,30 @@ class binary_writer
}
/*!
@param[in] j JSON value to serialize
@param[in] j JSON value to serialize
@param[in] depth nesting level of @a j, counted from the top-level value
passed to @ref basic_json::to_cbor
@throw type_error.316 if a string value or an object key is not valid
UTF-8
@throw type_error.321 if @a j or a value nested in it is discarded
Serializing a container descends into its elements, so a value nested deeply
enough used to exhaust the call stack and terminate the process with no
exception to catch. The descent is bounded here: once @ref recursion_depth_limit
levels have been entered, @ref write_cbor_iterative writes out what is left
without the call stack. A value nested less deeply than that - all but a
vanishing minority - is written by exactly the code that always wrote it.
@sa https://github.com/nlohmann/json/issues/5392
*/
void write_cbor(const BasicJsonType& j)
void write_cbor(const BasicJsonType& j, const std::size_t depth = 0)
{
if (JSON_HEDLEY_UNLIKELY(depth >= recursion_depth_limit()) && (j.is_array() || j.is_object()))
{
write_cbor_iterative(j);
return;
}
switch (j.type())
{
case value_t::null:
@@ -244,10 +262,9 @@ class binary_writer
// step 1: write control byte and the array size
write_cbor_head(0x80, j.m_data.m_value.array->size());
// step 2: write each element
for (const auto& el : *j.m_data.m_value.array)
{
write_cbor(el);
write_cbor(el, depth + 1);
}
break;
}
@@ -302,7 +319,6 @@ class binary_writer
// step 1: write control byte and the object size
write_cbor_head(0xA0, j.m_data.m_value.object->size());
// step 2: write each element
for (const auto& el : *j.m_data.m_value.object)
{
// el.first is checked here, against the object as
@@ -317,7 +333,7 @@ class binary_writer
check_utf8(el.first, j);
}
write_cbor(el.first);
write_cbor(el.second);
write_cbor(el.second, depth + 1);
}
break;
}
@@ -383,11 +399,22 @@ class binary_writer
}
/*!
@param[in] j JSON value to serialize
@param[in] j JSON value to serialize
@param[in] depth nesting level of @a j, counted from the top-level value
passed to @ref basic_json::to_msgpack
@throw type_error.321 if @a j or a value nested in it is discarded
@sa @ref write_cbor
@sa https://github.com/nlohmann/json/issues/5392
*/
void write_msgpack(const BasicJsonType& j)
void write_msgpack(const BasicJsonType& j, const std::size_t depth = 0)
{
if (JSON_HEDLEY_UNLIKELY(depth >= recursion_depth_limit()) && (j.is_array() || j.is_object()))
{
write_msgpack_iterative(j);
return;
}
switch (j.type())
{
case value_t::null: // nil
@@ -503,29 +530,11 @@ class binary_writer
case value_t::array:
{
// step 1: write control byte and the array size
const auto N = to_msgpack_length(j.m_data.m_value.array->size(), j);
if (N <= 15)
{
// fixarray
write_number(static_cast<std::uint8_t>(0x90 | N));
}
else if (N <= (std::numeric_limits<std::uint16_t>::max)())
{
// array 16
oa.write_character(to_char_type(0xDC));
write_number(static_cast<std::uint16_t>(N));
}
else
{
// array 32
oa.write_character(to_char_type(0xDD));
write_number(static_cast<std::uint32_t>(N));
}
write_msgpack_array_prefix(j.m_data.m_value.array->size(), j);
// step 2: write each element
for (const auto& el : *j.m_data.m_value.array)
{
write_msgpack(el);
write_msgpack(el, depth + 1);
}
break;
}
@@ -621,26 +630,8 @@ class binary_writer
case value_t::object:
{
// step 1: write control byte and the object size
const auto N = to_msgpack_length(j.m_data.m_value.object->size(), j);
if (N <= 15)
{
// fixmap
write_number(static_cast<std::uint8_t>(0x80 | (N & 0xF)));
}
else if (N <= (std::numeric_limits<std::uint16_t>::max)())
{
// map 16
oa.write_character(to_char_type(0xDE));
write_number(static_cast<std::uint16_t>(N));
}
else
{
// map 32
oa.write_character(to_char_type(0xDF));
write_number(static_cast<std::uint32_t>(N));
}
write_msgpack_object_prefix(j.m_data.m_value.object->size(), j);
// step 2: write each element
for (const auto& el : *j.m_data.m_value.object)
{
// as in write_cbor, el.first is checked here against the
@@ -651,7 +642,7 @@ class binary_writer
check_utf8(el.first, j);
}
write_msgpack(el.first);
write_msgpack(el.second);
write_msgpack(el.second, depth + 1);
}
break;
}
@@ -669,14 +660,26 @@ class binary_writer
@param[in] add_prefix whether prefixes need to be used for this value
@param[in] use_bjdata whether write in BJData format, default is false
@param[in] bjdata_version which BJData version to use, default is draft2
@param[in] depth nesting level of @a j, counted from the top-level value
passed to @ref basic_json::to_ubjson or @ref basic_json::to_bjdata
@throw type_error.316 if a string value or an object key is not valid
UTF-8
@throw type_error.321 if @a j or a value nested in it is discarded
@sa @ref write_cbor
@sa https://github.com/nlohmann/json/issues/5392
*/
void write_ubjson(const BasicJsonType& j, const bool use_count,
const bool use_type, const bool add_prefix = true,
const bool use_bjdata = false, const bjdata_version_t bjdata_version = bjdata_version_t::draft2)
const bool use_bjdata = false, const bjdata_version_t bjdata_version = bjdata_version_t::draft2,
const std::size_t depth = 0)
{
if (JSON_HEDLEY_UNLIKELY(depth >= recursion_depth_limit()) && (j.is_array() || j.is_object()))
{
write_ubjson_iterative(j, use_count, use_type, add_prefix, use_bjdata, bjdata_version);
return;
}
const bool bjdata_draft3 = use_bjdata && bjdata_version == bjdata_version_t::draft3;
switch (j.type())
@@ -737,55 +740,15 @@ class binary_writer
case value_t::array:
{
if (add_prefix)
{
oa.write_character(to_char_type('['));
}
bool prefix_required = true;
if (use_type && !j.m_data.m_value.array->empty())
{
if (!use_count)
{
JSON_THROW(other_error::create(502, "use_type requires use_size = true", &j));
}
const CharType first_prefix = ubjson_prefix(j.front(), use_bjdata);
const bool same_prefix = std::all_of(j.begin() + 1, j.end(),
[this, first_prefix, use_bjdata](const BasicJsonType & v)
{
return ubjson_prefix(v, use_bjdata) == first_prefix;
});
// an optimized array of a valueless type carries no payload, so a
// reader has nothing but the declared count to bound the allocation
// by and refuses an excessive one. Write the unoptimized form for
// those, at one byte per element, so the result can be read back.
// Objects are not affected: every element is preceded by its key.
const bool valueless_type = (first_prefix == 'Z' || first_prefix == 'T' || first_prefix == 'F');
const bool excessive_valueless = valueless_type
&& j.m_data.m_value.array->size() > detail::max_valueless_container_size;
if (same_prefix && !excessive_valueless
&& !(use_bjdata && is_bjdata_excluded_type_marker(first_prefix)))
{
prefix_required = false;
oa.write_character(to_char_type('$'));
oa.write_character(first_prefix);
}
}
if (use_count)
{
oa.write_character(to_char_type('#'));
write_number_with_ubjson_prefix(j.m_data.m_value.array->size(), true, use_bjdata);
}
const bool write_closer = write_ubjson_start_array(j, use_count, use_type, add_prefix, use_bjdata, prefix_required);
for (const auto& el : *j.m_data.m_value.array)
{
write_ubjson(el, use_count, use_type, prefix_required, use_bjdata, bjdata_version);
write_ubjson(el, use_count, use_type, prefix_required, use_bjdata, bjdata_version, depth + 1);
}
if (!use_count)
if (write_closer)
{
oa.write_character(to_char_type(']'));
}
@@ -843,7 +806,7 @@ class binary_writer
case value_t::object:
{
if (use_bjdata && j.m_data.m_value.object->size() == 3 && j.m_data.m_value.object->find("_ArrayType_") != j.m_data.m_value.object->end() && j.m_data.m_value.object->find("_ArraySize_") != j.m_data.m_value.object->end() && j.m_data.m_value.object->find("_ArrayData_") != j.m_data.m_value.object->end())
if (use_bjdata && is_bjdata_ndarray(j))
{
if (!write_bjdata_ndarray(*j.m_data.m_value.object, use_count, use_type, bjdata_version)) // decode bjdata ndarray in the JData format (https://github.com/NeuroJSON/jdata)
{
@@ -851,38 +814,8 @@ class binary_writer
}
}
if (add_prefix)
{
oa.write_character(to_char_type('{'));
}
bool prefix_required = true;
if (use_type && !j.m_data.m_value.object->empty())
{
if (!use_count)
{
JSON_THROW(other_error::create(502, "use_type requires use_size = true", &j));
}
const CharType first_prefix = ubjson_prefix(j.front(), use_bjdata);
const bool same_prefix = std::all_of(j.begin(), j.end(),
[this, first_prefix, use_bjdata](const BasicJsonType & v)
{
return ubjson_prefix(v, use_bjdata) == first_prefix;
});
if (same_prefix && !(use_bjdata && is_bjdata_excluded_type_marker(first_prefix)))
{
prefix_required = false;
oa.write_character(to_char_type('$'));
oa.write_character(first_prefix);
}
}
if (use_count)
{
oa.write_character(to_char_type('#'));
write_number_with_ubjson_prefix(j.m_data.m_value.object->size(), true, use_bjdata);
}
const bool write_closer = write_ubjson_start_object(j, use_count, use_type, add_prefix, use_bjdata, prefix_required);
for (const auto& el : *j.m_data.m_value.object)
{
@@ -892,10 +825,10 @@ class binary_writer
oa.write_characters(
reinterpret_cast<const CharType*>(key.data()),
key.size());
write_ubjson(el.second, use_count, use_type, prefix_required, use_bjdata, bjdata_version);
write_ubjson(el.second, use_count, use_type, prefix_required, use_bjdata, bjdata_version, depth + 1);
}
if (!use_count)
if (write_closer)
{
oa.write_character(to_char_type('}'));
}
@@ -936,6 +869,517 @@ class binary_writer
JSON_THROW(type_error::create(321, concat("cannot serialize discarded value to ", format_name), &j));
}
void write_msgpack_array_prefix(const std::size_t N, const BasicJsonType& j)
{
const auto n = to_msgpack_length(N, j);
if (n <= 15)
{
// fixarray
write_number(static_cast<std::uint8_t>(0x90 | n));
}
else if (n <= (std::numeric_limits<std::uint16_t>::max)())
{
// array 16
oa.write_character(to_char_type(0xDC));
write_number(static_cast<std::uint16_t>(n));
}
else
{
// array 32
oa.write_character(to_char_type(0xDD));
write_number(static_cast<std::uint32_t>(n));
}
}
void write_msgpack_object_prefix(const std::size_t N, const BasicJsonType& j)
{
const auto n = to_msgpack_length(N, j);
if (n <= 15)
{
// fixmap
write_number(static_cast<std::uint8_t>(0x80 | (n & 0xF)));
}
else if (n <= (std::numeric_limits<std::uint16_t>::max)())
{
// map 16
oa.write_character(to_char_type(0xDE));
write_number(static_cast<std::uint16_t>(n));
}
else
{
// map 32
oa.write_character(to_char_type(0xDF));
write_number(static_cast<std::uint32_t>(n));
}
}
/// @brief a CBOR or MessagePack array or object whose elements
/// @ref write_cbor_iterative or @ref write_msgpack_iterative is
/// still writing
struct binary_container_frame
{
explicit binary_container_frame(const BasicJsonType* value_) noexcept
: value(value_)
{
if (value->is_object())
{
object_it = value->m_data.m_value.object->cbegin();
}
else
{
array_it = value->m_data.m_value.array->cbegin();
}
}
// declared for GCC's -Weffc++, which asks for them in a class with
// pointer members and a non-trivial destructor; the exception
// specifications are left implicit, as GCC 4.8 rejects explicit ones
// that differ from them
binary_container_frame(const binary_container_frame&) = default;
binary_container_frame(binary_container_frame&&) = default;
binary_container_frame& operator=(const binary_container_frame&) = default;
binary_container_frame& operator=(binary_container_frame&&) = default;
~binary_container_frame() = default;
/// the array or object being written
const BasicJsonType* value;
/// value's elements still to write; which of the two is live follows
/// from the type of value. They are kept side by side rather than in
/// a union, which would need its special members written out by
/// hand, see detail/iterators/internal_iterator.hpp
typename BasicJsonType::object_t::const_iterator object_it{};
typename BasicJsonType::array_t::const_iterator array_it{};
};
/*!
@brief write @a j with @ref write_cbor, or write its header and push a
frame for @ref write_cbor_iterative to continue with its elements
A scalar, and an empty array or object, are written out in full: there is
nothing below them for @ref write_cbor_iterative to come back to, so
nothing is pushed for them.
*/
void write_cbor_value_or_push(const BasicJsonType& j, std::vector<binary_container_frame>& stack)
{
if (j.is_array())
{
write_cbor_head(0x80, j.m_data.m_value.array->size());
if (!j.m_data.m_value.array->empty())
{
stack.emplace_back(&j);
}
return;
}
if (j.is_object())
{
write_cbor_head(0xA0, j.m_data.m_value.object->size());
if (!j.m_data.m_value.object->empty())
{
stack.emplace_back(&j);
}
return;
}
write_cbor(j);
}
/*!
@brief write out @a root and everything below it without the call stack
Emits the same bytes as @ref write_cbor, keeping the containers it has
entered on an explicit stack instead of descending into them. Only reached
for values nested deeper than @ref recursion_depth_limit, which is why it
is not written for speed.
*/
void write_cbor_iterative(const BasicJsonType& root)
{
// only a container with elements is ever pushed; see write_cbor_value_or_push
std::vector<binary_container_frame> stack;
write_cbor_value_or_push(root, stack);
while (!stack.empty())
{
const binary_container_frame current = stack.back();
if (current.value->is_array())
{
const auto& array = *current.value->m_data.m_value.array;
if (current.array_it == array.cend())
{
stack.pop_back();
continue;
}
// read the child before pushing: entering it can move every frame
const BasicJsonType* child = &(*current.array_it);
++stack.back().array_it;
write_cbor_value_or_push(*child, stack);
}
else
{
const auto& object = *current.value->m_data.m_value.object;
if (current.object_it == object.cend())
{
stack.pop_back();
continue;
}
// el.first is checked here, against the object as diagnostics
// context, like the matching check in write_cbor's object case
if (error_handler == error_handler_t::strict)
{
check_utf8(current.object_it->first, *current.value);
}
write_cbor(current.object_it->first);
const BasicJsonType* child = &(current.object_it->second);
++stack.back().object_it;
write_cbor_value_or_push(*child, stack);
}
}
}
/*!
@brief write @a j with @ref write_msgpack, or write its header and push a
frame for @ref write_msgpack_iterative to continue with its elements
@sa @ref write_cbor_value_or_push
*/
void write_msgpack_value_or_push(const BasicJsonType& j, std::vector<binary_container_frame>& stack)
{
if (j.is_array())
{
write_msgpack_array_prefix(j.m_data.m_value.array->size(), j);
if (!j.m_data.m_value.array->empty())
{
stack.emplace_back(&j);
}
return;
}
if (j.is_object())
{
write_msgpack_object_prefix(j.m_data.m_value.object->size(), j);
if (!j.m_data.m_value.object->empty())
{
stack.emplace_back(&j);
}
return;
}
write_msgpack(j);
}
/*!
@brief write out @a root and everything below it without the call stack
@sa @ref write_cbor_iterative
*/
void write_msgpack_iterative(const BasicJsonType& root)
{
std::vector<binary_container_frame> stack;
write_msgpack_value_or_push(root, stack);
while (!stack.empty())
{
const binary_container_frame current = stack.back();
if (current.value->is_array())
{
const auto& array = *current.value->m_data.m_value.array;
if (current.array_it == array.cend())
{
stack.pop_back();
continue;
}
const BasicJsonType* child = &(*current.array_it);
++stack.back().array_it;
write_msgpack_value_or_push(*child, stack);
}
else
{
const auto& object = *current.value->m_data.m_value.object;
if (current.object_it == object.cend())
{
stack.pop_back();
continue;
}
if (error_handler == error_handler_t::strict)
{
check_utf8(current.object_it->first, *current.value);
}
write_msgpack(current.object_it->first);
const BasicJsonType* child = &(current.object_it->second);
++stack.back().object_it;
write_msgpack_value_or_push(*child, stack);
}
}
}
/// @return true when a closing ']' still has to be written after the elements
bool write_ubjson_start_array(const BasicJsonType& j, const bool use_count, const bool use_type,
const bool add_prefix, const bool use_bjdata, bool& prefix_required)
{
prefix_required = true;
if (add_prefix)
{
oa.write_character(to_char_type('['));
}
if (use_type && !j.m_data.m_value.array->empty())
{
if (!use_count)
{
JSON_THROW(other_error::create(502, "use_type requires use_size = true", &j));
}
const CharType first_prefix = ubjson_prefix(j.front(), use_bjdata);
const bool same_prefix = std::all_of(j.begin() + 1, j.end(),
[this, first_prefix, use_bjdata](const BasicJsonType & v)
{
return ubjson_prefix(v, use_bjdata) == first_prefix;
});
// an optimized array of a valueless type carries no payload, so a
// reader has nothing but the declared count to bound the allocation
// by and refuses an excessive one. Write the unoptimized form for
// those, at one byte per element, so the result can be read back.
// Objects are not affected: every element is preceded by its key.
const bool valueless_type = (first_prefix == 'Z' || first_prefix == 'T' || first_prefix == 'F');
const bool excessive_valueless = valueless_type
&& j.m_data.m_value.array->size() > detail::max_valueless_container_size;
if (same_prefix && !excessive_valueless
&& !(use_bjdata && is_bjdata_excluded_type_marker(first_prefix)))
{
prefix_required = false;
oa.write_character(to_char_type('$'));
oa.write_character(first_prefix);
}
}
if (use_count)
{
oa.write_character(to_char_type('#'));
write_number_with_ubjson_prefix(j.m_data.m_value.array->size(), true, use_bjdata);
}
return !use_count;
}
/// @return true when a closing '}' still has to be written after the elements
bool write_ubjson_start_object(const BasicJsonType& j, const bool use_count, const bool use_type,
const bool add_prefix, const bool use_bjdata, bool& prefix_required)
{
prefix_required = true;
if (add_prefix)
{
oa.write_character(to_char_type('{'));
}
if (use_type && !j.m_data.m_value.object->empty())
{
if (!use_count)
{
JSON_THROW(other_error::create(502, "use_type requires use_size = true", &j));
}
const CharType first_prefix = ubjson_prefix(j.front(), use_bjdata);
const bool same_prefix = std::all_of(j.begin(), j.end(),
[this, first_prefix, use_bjdata](const BasicJsonType & v)
{
return ubjson_prefix(v, use_bjdata) == first_prefix;
});
if (same_prefix && !(use_bjdata && is_bjdata_excluded_type_marker(first_prefix)))
{
prefix_required = false;
oa.write_character(to_char_type('$'));
oa.write_character(first_prefix);
}
}
if (use_count)
{
oa.write_character(to_char_type('#'));
write_number_with_ubjson_prefix(j.m_data.m_value.object->size(), true, use_bjdata);
}
return !use_count;
}
/*!
@brief whether @a j is a BJData ND-array annotation object
(https://github.com/NeuroJSON/jdata)
Used by both the recursive object case of @ref write_ubjson and
@ref write_ubjson_value_or_push, which must agree on what counts as an
ND-array: @a j is only actually written as one once @ref
write_bjdata_ndarray has also accepted its contents.
@pre @a j.is_object()
*/
static bool is_bjdata_ndarray(const BasicJsonType& j)
{
const auto& object = *j.m_data.m_value.object;
return object.size() == 3
&& object.find("_ArrayType_") != object.end()
&& object.find("_ArraySize_") != object.end()
&& object.find("_ArrayData_") != object.end();
}
/// @brief an object or array @ref write_ubjson_iterative is still writing
/// the elements of
struct ubjson_frame
{
ubjson_frame(const BasicJsonType* value_, const bool prefix_required_) noexcept
: value(value_)
, prefix_required(prefix_required_)
{
if (value->is_object())
{
object_it = value->m_data.m_value.object->cbegin();
}
else
{
array_it = value->m_data.m_value.array->cbegin();
}
}
// declared for GCC's -Weffc++, which asks for them in a class with
// pointer members and a non-trivial destructor; the exception
// specifications are left implicit, as GCC 4.8 rejects explicit ones
// that differ from them
ubjson_frame(const ubjson_frame&) = default;
ubjson_frame(ubjson_frame&&) = default;
ubjson_frame& operator=(const ubjson_frame&) = default;
ubjson_frame& operator=(ubjson_frame&&) = default;
~ubjson_frame() = default;
/// the array or object being written
const BasicJsonType* value;
/// whether value's elements each carry their own type marker; an
/// optimized ($type) container writes it once for all of them instead
bool prefix_required;
typename BasicJsonType::object_t::const_iterator object_it{};
typename BasicJsonType::array_t::const_iterator array_it{};
};
/*!
@brief write @a j with @ref write_ubjson, or write its header and push a
frame for @ref write_ubjson_iterative to continue with its elements
@param[in] add_prefix whether @a j's own type marker is written now (the
elements of an optimized container, and everything below the
top level, never repeat it)
@sa @ref write_cbor_value_or_push
*/
void write_ubjson_value_or_push(const BasicJsonType& j, const bool add_prefix, const bool use_count,
const bool use_type, const bool use_bjdata, const bjdata_version_t bjdata_version,
std::vector<ubjson_frame>& stack)
{
if (!j.is_array() && !j.is_object())
{
write_ubjson(j, use_count, use_type, add_prefix, use_bjdata, bjdata_version);
return;
}
if (use_bjdata && j.is_object() && is_bjdata_ndarray(j)
&& !write_bjdata_ndarray(*j.m_data.m_value.object, use_count, use_type, bjdata_version))
{
// fully written as an ND-array: nothing below it to come back to
return;
}
const bool is_array = j.is_array();
bool prefix_required = true;
if (is_array)
{
write_ubjson_start_array(j, use_count, use_type, add_prefix, use_bjdata, prefix_required);
}
else
{
write_ubjson_start_object(j, use_count, use_type, add_prefix, use_bjdata, prefix_required);
}
const bool empty = is_array ? j.m_data.m_value.array->empty() : j.m_data.m_value.object->empty();
if (!empty)
{
stack.emplace_back(&j, prefix_required);
return;
}
// write_ubjson_start_array/_object return !use_count, i.e. whether a
// closer still has to be written; use_count is constant for the whole
// document, so that is recomputed here instead of being carried along
if (!use_count)
{
oa.write_character(to_char_type(is_array ? ']' : '}'));
}
}
/*!
@brief write out @a root and everything below it without the call stack
@sa @ref write_cbor_iterative
*/
void write_ubjson_iterative(const BasicJsonType& root, const bool use_count, const bool use_type,
const bool add_prefix, const bool use_bjdata, const bjdata_version_t bjdata_version)
{
std::vector<ubjson_frame> stack;
write_ubjson_value_or_push(root, add_prefix, use_count, use_type, use_bjdata, bjdata_version, stack);
while (!stack.empty())
{
const ubjson_frame current = stack.back();
const BasicJsonType& j = *current.value;
if (j.is_array())
{
const auto& array = *j.m_data.m_value.array;
if (current.array_it == array.cend())
{
if (!use_count)
{
oa.write_character(to_char_type(']'));
}
stack.pop_back();
continue;
}
const BasicJsonType* child = &(*current.array_it);
const bool child_prefix = current.prefix_required;
++stack.back().array_it;
write_ubjson_value_or_push(*child, child_prefix, use_count, use_type, use_bjdata, bjdata_version, stack);
}
else
{
const auto& object = *j.m_data.m_value.object;
if (current.object_it == object.cend())
{
if (!use_count)
{
oa.write_character(to_char_type('}'));
}
stack.pop_back();
continue;
}
string_t storage;
const string_t& key = sanitize_utf8_for_write(current.object_it->first, j, storage);
write_number_with_ubjson_prefix(key.size(), true, use_bjdata);
oa.write_characters(
reinterpret_cast<const CharType*>(key.data()),
key.size());
const BasicJsonType* child = &(current.object_it->second);
const bool child_prefix = current.prefix_required;
++stack.back().object_it;
write_ubjson_value_or_push(*child, child_prefix, use_count, use_type, use_bjdata, bjdata_version, stack);
}
}
}
//////////
// BSON //
//////////
@@ -1839,16 +2283,31 @@ class binary_writer
/*!
@brief validate (dry_run) or write one BJData ND-array element of dtype 'd' (single precision)
@return whether @a el's value fits a float without overflow; always true when @a dry_run is false
@return whether @a el's value survives narrowing to float and back without any change
(so the ND-array round-trips exactly), or is infinite or NaN; always true when
@a dry_run is false
*/
bool write_bjdata_ndarray_float_element(const BasicJsonType& el, const bool dry_run)
{
const auto dval = el.template get<double>();
if (dry_run)
{
return !std::isfinite(dval) ||
(dval >= static_cast<double>(std::numeric_limits<float>::lowest()) &&
dval <= static_cast<double>((std::numeric_limits<float>::max)()));
#ifdef __GNUC__
JSON_HEDLEY_DIAGNOSTIC_PUSH
JSON_HEDLEY_PRAGMA(GCC diagnostic ignored "-Wfloat-equal")
#endif
// a value that would be rounded (rather than exactly represented) by the
// narrowing to float is treated like an out-of-range integer element; this
// is the same criterion write_compact_float() uses for CBOR/MessagePack
const bool in_range = std::isnan(dval) ||
(dval >= static_cast<double>(std::numeric_limits<float>::lowest()) &&
dval <= static_cast<double>((std::numeric_limits<float>::max)()) &&
static_cast<double>(static_cast<float>(dval)) == dval) ||
std::isinf(dval);
#ifdef __GNUC__
JSON_HEDLEY_DIAGNOSTIC_POP
#endif
return in_range;
}
write_number(static_cast<float>(dval), true);
return true;
+105 -9
View File
@@ -2,24 +2,120 @@ project('nlohmann_json',
'cpp',
version : '3.12.0',
license : 'MIT',
meson_version : '>= 0.64',
default_options: ['cpp_std=c++11'],
)
if get_option('MultipleHeaders')
incdir = 'include'
else
incdir = 'single_include'
endif
# The same compile definitions as the CMake target (see target_compile_definitions
# in CMakeLists.txt): only an option that differs from its default adds one.
json_defines = []
if not get_option('GlobalUDLs')
json_defines += 'JSON_USE_GLOBAL_UDLS=0'
endif
if not get_option('ImplicitConversions')
json_defines += 'JSON_USE_IMPLICIT_CONVERSIONS=0'
endif
if get_option('DisableEnumSerialization')
json_defines += 'JSON_DISABLE_ENUM_SERIALIZATION=1'
endif
if get_option('DisableTupleReferenceConversion')
json_defines += 'JSON_DISABLE_TUPLE_REFERENCE_CONVERSION=1'
endif
if get_option('Diagnostics')
json_defines += 'JSON_DIAGNOSTICS=1'
endif
if get_option('Diagnostic_Positions')
json_defines += 'JSON_DIAGNOSTIC_POSITIONS=1'
endif
if get_option('LegacyDiscardedValueComparison')
json_defines += 'JSON_USE_LEGACY_DISCARDED_VALUE_COMPARISON=1'
endif
if get_option('StrictNulHandling')
json_defines += 'JSON_STRICT_NUL_HANDLING=1'
endif
if get_option('StrictBinaryUTF8')
json_defines += 'JSON_STRICT_BINARY_UTF8=1'
endif
if get_option('DeleteDeprecatedFunctions')
json_defines += 'JSON_DELETE_DEPRECATED_FUNCTIONS=1'
endif
cpp_args = []
foreach define : json_defines
cpp_args += '-D' + define
endforeach
nlohmann_json_dep = declare_dependency(
include_directories: include_directories('single_include')
compile_args: cpp_args,
include_directories: include_directories(incdir)
)
meson.override_dependency('nlohmann_json', nlohmann_json_dep)
# The multi-header version under the name earlier versions of this file used
nlohmann_json_multiple_headers = declare_dependency(
compile_args: cpp_args,
include_directories: include_directories('include')
)
if not meson.is_subproject()
install_headers('single_include/nlohmann/json.hpp', subdir: 'nlohmann')
install_headers('single_include/nlohmann/json_fwd.hpp', subdir: 'nlohmann')
install_headers('single_include/nlohmann/json_literals.hpp', subdir: 'nlohmann')
install_subdir(
incdir / 'nlohmann',
install_dir: get_option('includedir'),
install_tag: 'devel',
)
pkgc = import('pkgconfig')
pkgc.generate(name: 'nlohmann_json',
version: meson.project_version(),
description: 'JSON for Modern C++'
)
pkgc = import('pkgconfig')
pkgc.generate(name: 'nlohmann_json',
version: meson.project_version(),
description: 'JSON for Modern C++',
extra_cflags: cpp_args,
install_dir: get_option('datadir') / 'pkgconfig',
)
# CMake package config files, so that find_package(nlohmann_json) works. The
# include directory is given relative to the config files, so that the
# installation can be relocated. This is not possible if includedir or datadir
# is an absolute path outside the prefix (e.g., with the separate outputs of
# Nix); then the absolute include directory is used, as CMake does.
fs = import('fs')
cmake_install_dir = get_option('datadir') / 'cmake' / meson.project_name()
cmake_to_prefix = []
if fs.is_absolute(get_option('includedir')) or fs.is_absolute(cmake_install_dir)
cmake_include_dir = (get_option('prefix') / get_option('includedir')).replace('\\', '/')
else
foreach component : cmake_install_dir.split('/')
cmake_to_prefix += '..'
endforeach
cmake_include_dir = '${_IMPORT_PREFIX}/' + get_option('includedir')
endif
cmake_conf = configuration_data()
cmake_conf.set('PROJECT_NAME', meson.project_name())
cmake_conf.set('PROJECT_VERSION', meson.project_version())
cmake_conf.set('PROJECT_VERSION_MAJOR', meson.project_version().split('.')[0])
cmake_conf.set('NLOHMANN_JSON_TARGET_NAME', meson.project_name())
cmake_conf.set('NLOHMANN_JSON_TARGETS_EXPORT_NAME', meson.project_name() + 'Targets')
cmake_conf.set('NLOHMANN_JSON_INCLUDE_DIR', cmake_include_dir)
cmake_conf.set('NLOHMANN_JSON_CONFIG_TO_PREFIX', '/'.join(cmake_to_prefix))
cmake_conf.set('NLOHMANN_JSON_COMPILE_DEFINITIONS', ';'.join(json_defines))
foreach cmake_file : [
['cmake/config.cmake.in', 'nlohmann_jsonConfig.cmake'],
['cmake/nlohmann_jsonConfigVersion.cmake.in', 'nlohmann_jsonConfigVersion.cmake'],
['cmake/nlohmann_jsonTargets.cmake.in', 'nlohmann_jsonTargets.cmake'],
]
configure_file(
input: cmake_file[0],
output: cmake_file[1],
configuration: cmake_conf,
format: 'cmake@',
install_dir: cmake_install_dir,
)
endforeach
endif
+66
View File
@@ -0,0 +1,66 @@
option(
'MultipleHeaders',
type: 'boolean',
value: false,
description: 'Use non-amalgamated version of the library',
)
option(
'GlobalUDLs',
type: 'boolean',
value: true,
description: 'Place user-defined string literals in the global namespace',
)
option(
'ImplicitConversions',
type: 'boolean',
value: true,
description: 'Enable implicit conversions',
)
option(
'DisableEnumSerialization',
type: 'boolean',
value: false,
description: 'Disable default integer enum serialization',
)
option(
'DisableTupleReferenceConversion',
type: 'boolean',
value: false,
description: 'Disable conversion from a one-element tuple of a JSON reference',
)
option(
'Diagnostics',
type: 'boolean',
value: false,
description: 'Use extended diagnostic messages',
)
option(
'Diagnostic_Positions',
type: 'boolean',
value: false,
description: 'Enable diagnostic positions',
)
option(
'LegacyDiscardedValueComparison',
type: 'boolean',
value: false,
description: 'Enable legacy discarded value comparison',
)
option(
'StrictNulHandling',
type: 'boolean',
value: false,
description: 'Enable strict NUL-byte handling',
)
option(
'StrictBinaryUTF8',
type: 'boolean',
value: false,
description: 'Enable UTF-8 checks in the CBOR, UBJSON, BJData, and BSON writers',
)
option(
'DeleteDeprecatedFunctions',
type: 'boolean',
value: false,
description: 'Delete the deprecated functions instead of only deprecating them',
)
+22572
View File
File diff suppressed because it is too large. Load diff
+648 -164
View File
@@ -7583,8 +7583,10 @@ std::size_t hash_iteratively(const BasicJsonType& j);
@brief hash a JSON value
The hash function tries to rely on std::hash where possible. Furthermore, the
type of the JSON value is taken into account to have different hash values for
null, 0, 0U, and false, etc.
type of the JSON value is taken into account, so null, false, and numbers may
hash differently from each other, but any two numbers that compare equal
under operator== hash equally regardless of which of number_integer,
number_unsigned, or number_float actually holds the value.
Hashing an array or an object hashes its elements, which used to call this
function again once per nesting level, so a value nested deeply enough
@@ -7603,8 +7605,6 @@ template<typename BasicJsonType>
std::size_t hash(const BasicJsonType& j, const std::size_t depth = 0)
{
using string_t = typename BasicJsonType::string_t;
using number_integer_t = typename BasicJsonType::number_integer_t;
using number_unsigned_t = typename BasicJsonType::number_unsigned_t;
using number_float_t = typename BasicJsonType::number_float_t;
const auto type = static_cast<std::size_t>(j.type());
@@ -7661,21 +7661,24 @@ std::size_t hash(const BasicJsonType& j, const std::size_t depth = 0)
}
case BasicJsonType::value_t::number_integer:
{
const auto h = std::hash<number_integer_t> {}(j.template get<number_integer_t>());
return combine(type, h);
}
case BasicJsonType::value_t::number_unsigned:
{
const auto h = std::hash<number_unsigned_t> {}(j.template get<number_unsigned_t>());
return combine(type, h);
}
case BasicJsonType::value_t::number_float:
{
const auto h = std::hash<number_float_t> {}(j.template get<number_float_t>());
return combine(type, h);
// operator== compares numbers by their mathematical value across
// number_integer, number_unsigned, and number_float, so equal
// numbers of different internal types (0, 0U, 0.0) must hash the
// same. Two equal numbers have the same value, which converts to
// the same number_float_t, so all numbers share one type tag and
// hash that converted value. Adding zero turns -0.0 (equal to 0)
// into 0.0, as std::hash need not map both to the same hash.
// The converse does not hold: converting a number_float_t value
// to an integer type is lossy, so the result can hash
// differently, and unequal numbers that convert to the same
// number_float_t (e.g., 2^53 and 2^53 + 1) share a hash.
const auto number_type = static_cast<std::size_t>(BasicJsonType::value_t::number_float);
const auto value = j.template get<number_float_t>() + static_cast<number_float_t>(0);
const auto h = std::hash<number_float_t> {}(value);
return combine(number_type, h);
}
case BasicJsonType::value_t::binary:
@@ -16412,10 +16415,15 @@ class binary_reader
is_ndarray can only return `true` when its initial value
is `false`
@param[in] prefix type marker if already read, otherwise set to 0
@param[in] ndarray_dtype the element type marker of the enclosing bjdata ndarray if
already known (it precedes the dimension vector read here),
otherwise 0; used to emit the "_ArrayType_" annotation key
before "_ArraySize_" if a dimension vector turns out to
describe an ndarray
@return whether size determination completed
*/
bool get_ubjson_size_value(std::size_t& result, bool& is_ndarray, char_int_type prefix = 0)
bool get_ubjson_size_value(std::size_t& result, bool& is_ndarray, char_int_type prefix = 0, char_int_type ndarray_dtype = 0)
{
if (prefix == 0)
{
@@ -16528,8 +16536,34 @@ class binary_reader
}
}
if (JSON_HEDLEY_UNLIKELY(!sax->start_object(3)))
{
return false;
}
// the element type precedes the dimension vector (see get_ubjson_size_type)
// and is passed down as ndarray_dtype; emit it here so the annotation keys
// follow the documented _ArrayType_, _ArraySize_, _ArrayData_ order
if (ndarray_dtype != 0)
{
const char* type_name = bjd_type_name(ndarray_dtype);
if (JSON_HEDLEY_UNLIKELY(type_name == nullptr))
{
auto last_token = get_token_string();
return sax->parse_error(chars_read, last_token, parse_error::create(112, chars_read,
exception_message("invalid byte: 0x" + last_token, "type"), nullptr));
}
string_t type_key = "_ArrayType_";
string_t type = type_name; // sax->string() takes a reference
if (JSON_HEDLEY_UNLIKELY(!sax->key(type_key) || !sax->string(type)))
{
return false;
}
}
string_t key = "_ArraySize_";
if (JSON_HEDLEY_UNLIKELY(!sax->start_object(3) || !sax->key(key) || !sax->start_array(dim.size())))
if (JSON_HEDLEY_UNLIKELY(!sax->key(key) || !sax->start_array(dim.size())))
{
return false;
}
@@ -16637,7 +16671,7 @@ class binary_reader
exception_message(concat("expected '#' after type information; last byte: 0x", last_token), "size"), nullptr));
}
const bool is_error = get_ubjson_size_value(result.first, is_ndarray);
const bool is_error = get_ubjson_size_value(result.first, is_ndarray, 0, result.second);
// an ndarray was read here only if the flag flipped; when it was
// seeded true, get_ubjson_size_value() already rejected the nested
// dimension vector
@@ -16830,27 +16864,17 @@ class binary_reader
if (input_format == input_format_t::bjdata && size_and_type.first != npos && (size_and_type.second & (1 << 8)) != 0)
{
size_and_type.second &= ~(static_cast<char_int_type>(1) << 8); // use bit 8 to indicate ndarray, here we remove the bit to restore the type marker
const char* type_name = bjd_type_name(size_and_type.second);
string_t key = "_ArrayType_";
if (JSON_HEDLEY_UNLIKELY(type_name == nullptr))
{
auto last_token = get_token_string();
return sax->parse_error(chars_read, last_token, parse_error::create(112, chars_read,
exception_message("invalid byte: 0x" + last_token, "type"), nullptr));
}
string_t type = type_name; // sax->string() takes a reference
if (JSON_HEDLEY_UNLIKELY(!sax->key(key) || !sax->string(type)))
{
return false;
}
// the "_ArrayType_" and "_ArraySize_" annotation keys were already emitted by
// get_ubjson_size_value() (the type marker is known before the dimension vector
// that determines size_and_type.first is read, so it is emitted first there to
// match the documented _ArrayType_, _ArraySize_, _ArrayData_ key order)
if (size_and_type.second == 'C' || size_and_type.second == 'B')
{
size_and_type.second = 'U';
}
key = "_ArrayData_";
string_t key = "_ArrayData_";
if (JSON_HEDLEY_UNLIKELY(!sax->key(key) || !sax->start_array(size_and_type.first) ))
{
return false;
@@ -21497,6 +21521,8 @@ class output_adapter
} // namespace detail
NLOHMANN_JSON_NAMESPACE_END
// #include <nlohmann/detail/recursion_depth_limit.hpp>
// #include <nlohmann/detail/string_concat.hpp>
// #include <nlohmann/detail/string_utils.hpp>
@@ -21627,13 +21653,30 @@ class binary_writer
}
/*!
@param[in] j JSON value to serialize
@param[in] j JSON value to serialize
@param[in] depth nesting level of @a j, counted from the top-level value
passed to @ref basic_json::to_cbor
@throw type_error.316 if a string value or an object key is not valid
UTF-8
@throw type_error.321 if @a j or a value nested in it is discarded
Serializing a container descends into its elements, so a value nested deeply
enough used to exhaust the call stack and terminate the process with no
exception to catch. The descent is bounded here: once @ref recursion_depth_limit
levels have been entered, @ref write_cbor_iterative writes out what is left
without the call stack. A value nested less deeply than that - all but a
vanishing minority - is written by exactly the code that always wrote it.
@sa https://github.com/nlohmann/json/issues/5392
*/
void write_cbor(const BasicJsonType& j)
void write_cbor(const BasicJsonType& j, const std::size_t depth = 0)
{
if (JSON_HEDLEY_UNLIKELY(depth >= recursion_depth_limit()) && (j.is_array() || j.is_object()))
{
write_cbor_iterative(j);
return;
}
switch (j.type())
{
case value_t::null:
@@ -21715,10 +21758,9 @@ class binary_writer
// step 1: write control byte and the array size
write_cbor_head(0x80, j.m_data.m_value.array->size());
// step 2: write each element
for (const auto& el : *j.m_data.m_value.array)
{
write_cbor(el);
write_cbor(el, depth + 1);
}
break;
}
@@ -21773,7 +21815,6 @@ class binary_writer
// step 1: write control byte and the object size
write_cbor_head(0xA0, j.m_data.m_value.object->size());
// step 2: write each element
for (const auto& el : *j.m_data.m_value.object)
{
// el.first is checked here, against the object as
@@ -21788,7 +21829,7 @@ class binary_writer
check_utf8(el.first, j);
}
write_cbor(el.first);
write_cbor(el.second);
write_cbor(el.second, depth + 1);
}
break;
}
@@ -21854,11 +21895,22 @@ class binary_writer
}
/*!
@param[in] j JSON value to serialize
@param[in] j JSON value to serialize
@param[in] depth nesting level of @a j, counted from the top-level value
passed to @ref basic_json::to_msgpack
@throw type_error.321 if @a j or a value nested in it is discarded
@sa @ref write_cbor
@sa https://github.com/nlohmann/json/issues/5392
*/
void write_msgpack(const BasicJsonType& j)
void write_msgpack(const BasicJsonType& j, const std::size_t depth = 0)
{
if (JSON_HEDLEY_UNLIKELY(depth >= recursion_depth_limit()) && (j.is_array() || j.is_object()))
{
write_msgpack_iterative(j);
return;
}
switch (j.type())
{
case value_t::null: // nil
@@ -21974,29 +22026,11 @@ class binary_writer
case value_t::array:
{
// step 1: write control byte and the array size
const auto N = to_msgpack_length(j.m_data.m_value.array->size(), j);
if (N <= 15)
{
// fixarray
write_number(static_cast<std::uint8_t>(0x90 | N));
}
else if (N <= (std::numeric_limits<std::uint16_t>::max)())
{
// array 16
oa.write_character(to_char_type(0xDC));
write_number(static_cast<std::uint16_t>(N));
}
else
{
// array 32
oa.write_character(to_char_type(0xDD));
write_number(static_cast<std::uint32_t>(N));
}
write_msgpack_array_prefix(j.m_data.m_value.array->size(), j);
// step 2: write each element
for (const auto& el : *j.m_data.m_value.array)
{
write_msgpack(el);
write_msgpack(el, depth + 1);
}
break;
}
@@ -22092,26 +22126,8 @@ class binary_writer
case value_t::object:
{
// step 1: write control byte and the object size
const auto N = to_msgpack_length(j.m_data.m_value.object->size(), j);
if (N <= 15)
{
// fixmap
write_number(static_cast<std::uint8_t>(0x80 | (N & 0xF)));
}
else if (N <= (std::numeric_limits<std::uint16_t>::max)())
{
// map 16
oa.write_character(to_char_type(0xDE));
write_number(static_cast<std::uint16_t>(N));
}
else
{
// map 32
oa.write_character(to_char_type(0xDF));
write_number(static_cast<std::uint32_t>(N));
}
write_msgpack_object_prefix(j.m_data.m_value.object->size(), j);
// step 2: write each element
for (const auto& el : *j.m_data.m_value.object)
{
// as in write_cbor, el.first is checked here against the
@@ -22122,7 +22138,7 @@ class binary_writer
check_utf8(el.first, j);
}
write_msgpack(el.first);
write_msgpack(el.second);
write_msgpack(el.second, depth + 1);
}
break;
}
@@ -22140,14 +22156,26 @@ class binary_writer
@param[in] add_prefix whether prefixes need to be used for this value
@param[in] use_bjdata whether write in BJData format, default is false
@param[in] bjdata_version which BJData version to use, default is draft2
@param[in] depth nesting level of @a j, counted from the top-level value
passed to @ref basic_json::to_ubjson or @ref basic_json::to_bjdata
@throw type_error.316 if a string value or an object key is not valid
UTF-8
@throw type_error.321 if @a j or a value nested in it is discarded
@sa @ref write_cbor
@sa https://github.com/nlohmann/json/issues/5392
*/
void write_ubjson(const BasicJsonType& j, const bool use_count,
const bool use_type, const bool add_prefix = true,
const bool use_bjdata = false, const bjdata_version_t bjdata_version = bjdata_version_t::draft2)
const bool use_bjdata = false, const bjdata_version_t bjdata_version = bjdata_version_t::draft2,
const std::size_t depth = 0)
{
if (JSON_HEDLEY_UNLIKELY(depth >= recursion_depth_limit()) && (j.is_array() || j.is_object()))
{
write_ubjson_iterative(j, use_count, use_type, add_prefix, use_bjdata, bjdata_version);
return;
}
const bool bjdata_draft3 = use_bjdata && bjdata_version == bjdata_version_t::draft3;
switch (j.type())
@@ -22208,55 +22236,15 @@ class binary_writer
case value_t::array:
{
if (add_prefix)
{
oa.write_character(to_char_type('['));
}
bool prefix_required = true;
if (use_type && !j.m_data.m_value.array->empty())
{
if (!use_count)
{
JSON_THROW(other_error::create(502, "use_type requires use_size = true", &j));
}
const CharType first_prefix = ubjson_prefix(j.front(), use_bjdata);
const bool same_prefix = std::all_of(j.begin() + 1, j.end(),
[this, first_prefix, use_bjdata](const BasicJsonType & v)
{
return ubjson_prefix(v, use_bjdata) == first_prefix;
});
// an optimized array of a valueless type carries no payload, so a
// reader has nothing but the declared count to bound the allocation
// by and refuses an excessive one. Write the unoptimized form for
// those, at one byte per element, so the result can be read back.
// Objects are not affected: every element is preceded by its key.
const bool valueless_type = (first_prefix == 'Z' || first_prefix == 'T' || first_prefix == 'F');
const bool excessive_valueless = valueless_type
&& j.m_data.m_value.array->size() > detail::max_valueless_container_size;
if (same_prefix && !excessive_valueless
&& !(use_bjdata && is_bjdata_excluded_type_marker(first_prefix)))
{
prefix_required = false;
oa.write_character(to_char_type('$'));
oa.write_character(first_prefix);
}
}
if (use_count)
{
oa.write_character(to_char_type('#'));
write_number_with_ubjson_prefix(j.m_data.m_value.array->size(), true, use_bjdata);
}
const bool write_closer = write_ubjson_start_array(j, use_count, use_type, add_prefix, use_bjdata, prefix_required);
for (const auto& el : *j.m_data.m_value.array)
{
write_ubjson(el, use_count, use_type, prefix_required, use_bjdata, bjdata_version);
write_ubjson(el, use_count, use_type, prefix_required, use_bjdata, bjdata_version, depth + 1);
}
if (!use_count)
if (write_closer)
{
oa.write_character(to_char_type(']'));
}
@@ -22314,7 +22302,7 @@ class binary_writer
case value_t::object:
{
if (use_bjdata && j.m_data.m_value.object->size() == 3 && j.m_data.m_value.object->find("_ArrayType_") != j.m_data.m_value.object->end() && j.m_data.m_value.object->find("_ArraySize_") != j.m_data.m_value.object->end() && j.m_data.m_value.object->find("_ArrayData_") != j.m_data.m_value.object->end())
if (use_bjdata && is_bjdata_ndarray(j))
{
if (!write_bjdata_ndarray(*j.m_data.m_value.object, use_count, use_type, bjdata_version)) // decode bjdata ndarray in the JData format (https://github.com/NeuroJSON/jdata)
{
@@ -22322,38 +22310,8 @@ class binary_writer
}
}
if (add_prefix)
{
oa.write_character(to_char_type('{'));
}
bool prefix_required = true;
if (use_type && !j.m_data.m_value.object->empty())
{
if (!use_count)
{
JSON_THROW(other_error::create(502, "use_type requires use_size = true", &j));
}
const CharType first_prefix = ubjson_prefix(j.front(), use_bjdata);
const bool same_prefix = std::all_of(j.begin(), j.end(),
[this, first_prefix, use_bjdata](const BasicJsonType & v)
{
return ubjson_prefix(v, use_bjdata) == first_prefix;
});
if (same_prefix && !(use_bjdata && is_bjdata_excluded_type_marker(first_prefix)))
{
prefix_required = false;
oa.write_character(to_char_type('$'));
oa.write_character(first_prefix);
}
}
if (use_count)
{
oa.write_character(to_char_type('#'));
write_number_with_ubjson_prefix(j.m_data.m_value.object->size(), true, use_bjdata);
}
const bool write_closer = write_ubjson_start_object(j, use_count, use_type, add_prefix, use_bjdata, prefix_required);
for (const auto& el : *j.m_data.m_value.object)
{
@@ -22363,10 +22321,10 @@ class binary_writer
oa.write_characters(
reinterpret_cast<const CharType*>(key.data()),
key.size());
write_ubjson(el.second, use_count, use_type, prefix_required, use_bjdata, bjdata_version);
write_ubjson(el.second, use_count, use_type, prefix_required, use_bjdata, bjdata_version, depth + 1);
}
if (!use_count)
if (write_closer)
{
oa.write_character(to_char_type('}'));
}
@@ -22407,6 +22365,517 @@ class binary_writer
JSON_THROW(type_error::create(321, concat("cannot serialize discarded value to ", format_name), &j));
}
void write_msgpack_array_prefix(const std::size_t N, const BasicJsonType& j)
{
const auto n = to_msgpack_length(N, j);
if (n <= 15)
{
// fixarray
write_number(static_cast<std::uint8_t>(0x90 | n));
}
else if (n <= (std::numeric_limits<std::uint16_t>::max)())
{
// array 16
oa.write_character(to_char_type(0xDC));
write_number(static_cast<std::uint16_t>(n));
}
else
{
// array 32
oa.write_character(to_char_type(0xDD));
write_number(static_cast<std::uint32_t>(n));
}
}
void write_msgpack_object_prefix(const std::size_t N, const BasicJsonType& j)
{
const auto n = to_msgpack_length(N, j);
if (n <= 15)
{
// fixmap
write_number(static_cast<std::uint8_t>(0x80 | (n & 0xF)));
}
else if (n <= (std::numeric_limits<std::uint16_t>::max)())
{
// map 16
oa.write_character(to_char_type(0xDE));
write_number(static_cast<std::uint16_t>(n));
}
else
{
// map 32
oa.write_character(to_char_type(0xDF));
write_number(static_cast<std::uint32_t>(n));
}
}
/// @brief a CBOR or MessagePack array or object whose elements
/// @ref write_cbor_iterative or @ref write_msgpack_iterative is
/// still writing
struct binary_container_frame
{
explicit binary_container_frame(const BasicJsonType* value_) noexcept
: value(value_)
{
if (value->is_object())
{
object_it = value->m_data.m_value.object->cbegin();
}
else
{
array_it = value->m_data.m_value.array->cbegin();
}
}
// declared for GCC's -Weffc++, which asks for them in a class with
// pointer members and a non-trivial destructor; the exception
// specifications are left implicit, as GCC 4.8 rejects explicit ones
// that differ from them
binary_container_frame(const binary_container_frame&) = default;
binary_container_frame(binary_container_frame&&) = default;
binary_container_frame& operator=(const binary_container_frame&) = default;
binary_container_frame& operator=(binary_container_frame&&) = default;
~binary_container_frame() = default;
/// the array or object being written
const BasicJsonType* value;
/// value's elements still to write; which of the two is live follows
/// from the type of value. They are kept side by side rather than in
/// a union, which would need its special members written out by
/// hand, see detail/iterators/internal_iterator.hpp
typename BasicJsonType::object_t::const_iterator object_it{};
typename BasicJsonType::array_t::const_iterator array_it{};
};
/*!
@brief write @a j with @ref write_cbor, or write its header and push a
frame for @ref write_cbor_iterative to continue with its elements
A scalar, and an empty array or object, are written out in full: there is
nothing below them for @ref write_cbor_iterative to come back to, so
nothing is pushed for them.
*/
void write_cbor_value_or_push(const BasicJsonType& j, std::vector<binary_container_frame>& stack)
{
if (j.is_array())
{
write_cbor_head(0x80, j.m_data.m_value.array->size());
if (!j.m_data.m_value.array->empty())
{
stack.emplace_back(&j);
}
return;
}
if (j.is_object())
{
write_cbor_head(0xA0, j.m_data.m_value.object->size());
if (!j.m_data.m_value.object->empty())
{
stack.emplace_back(&j);
}
return;
}
write_cbor(j);
}
/*!
@brief write out @a root and everything below it without the call stack
Emits the same bytes as @ref write_cbor, keeping the containers it has
entered on an explicit stack instead of descending into them. Only reached
for values nested deeper than @ref recursion_depth_limit, which is why it
is not written for speed.
*/
void write_cbor_iterative(const BasicJsonType& root)
{
// only a container with elements is ever pushed; see write_cbor_value_or_push
std::vector<binary_container_frame> stack;
write_cbor_value_or_push(root, stack);
while (!stack.empty())
{
const binary_container_frame current = stack.back();
if (current.value->is_array())
{
const auto& array = *current.value->m_data.m_value.array;
if (current.array_it == array.cend())
{
stack.pop_back();
continue;
}
// read the child before pushing: entering it can move every frame
const BasicJsonType* child = &(*current.array_it);
++stack.back().array_it;
write_cbor_value_or_push(*child, stack);
}
else
{
const auto& object = *current.value->m_data.m_value.object;
if (current.object_it == object.cend())
{
stack.pop_back();
continue;
}
// el.first is checked here, against the object as diagnostics
// context, like the matching check in write_cbor's object case
if (error_handler == error_handler_t::strict)
{
check_utf8(current.object_it->first, *current.value);
}
write_cbor(current.object_it->first);
const BasicJsonType* child = &(current.object_it->second);
++stack.back().object_it;
write_cbor_value_or_push(*child, stack);
}
}
}
/*!
@brief write @a j with @ref write_msgpack, or write its header and push a
frame for @ref write_msgpack_iterative to continue with its elements
@sa @ref write_cbor_value_or_push
*/
void write_msgpack_value_or_push(const BasicJsonType& j, std::vector<binary_container_frame>& stack)
{
if (j.is_array())
{
write_msgpack_array_prefix(j.m_data.m_value.array->size(), j);
if (!j.m_data.m_value.array->empty())
{
stack.emplace_back(&j);
}
return;
}
if (j.is_object())
{
write_msgpack_object_prefix(j.m_data.m_value.object->size(), j);
if (!j.m_data.m_value.object->empty())
{
stack.emplace_back(&j);
}
return;
}
write_msgpack(j);
}
/*!
@brief write out @a root and everything below it without the call stack
@sa @ref write_cbor_iterative
*/
void write_msgpack_iterative(const BasicJsonType& root)
{
std::vector<binary_container_frame> stack;
write_msgpack_value_or_push(root, stack);
while (!stack.empty())
{
const binary_container_frame current = stack.back();
if (current.value->is_array())
{
const auto& array = *current.value->m_data.m_value.array;
if (current.array_it == array.cend())
{
stack.pop_back();
continue;
}
const BasicJsonType* child = &(*current.array_it);
++stack.back().array_it;
write_msgpack_value_or_push(*child, stack);
}
else
{
const auto& object = *current.value->m_data.m_value.object;
if (current.object_it == object.cend())
{
stack.pop_back();
continue;
}
if (error_handler == error_handler_t::strict)
{
check_utf8(current.object_it->first, *current.value);
}
write_msgpack(current.object_it->first);
const BasicJsonType* child = &(current.object_it->second);
++stack.back().object_it;
write_msgpack_value_or_push(*child, stack);
}
}
}
/// @return true when a closing ']' still has to be written after the elements
bool write_ubjson_start_array(const BasicJsonType& j, const bool use_count, const bool use_type,
const bool add_prefix, const bool use_bjdata, bool& prefix_required)
{
prefix_required = true;
if (add_prefix)
{
oa.write_character(to_char_type('['));
}
if (use_type && !j.m_data.m_value.array->empty())
{
if (!use_count)
{
JSON_THROW(other_error::create(502, "use_type requires use_size = true", &j));
}
const CharType first_prefix = ubjson_prefix(j.front(), use_bjdata);
const bool same_prefix = std::all_of(j.begin() + 1, j.end(),
[this, first_prefix, use_bjdata](const BasicJsonType & v)
{
return ubjson_prefix(v, use_bjdata) == first_prefix;
});
// an optimized array of a valueless type carries no payload, so a
// reader has nothing but the declared count to bound the allocation
// by and refuses an excessive one. Write the unoptimized form for
// those, at one byte per element, so the result can be read back.
// Objects are not affected: every element is preceded by its key.
const bool valueless_type = (first_prefix == 'Z' || first_prefix == 'T' || first_prefix == 'F');
const bool excessive_valueless = valueless_type
&& j.m_data.m_value.array->size() > detail::max_valueless_container_size;
if (same_prefix && !excessive_valueless
&& !(use_bjdata && is_bjdata_excluded_type_marker(first_prefix)))
{
prefix_required = false;
oa.write_character(to_char_type('$'));
oa.write_character(first_prefix);
}
}
if (use_count)
{
oa.write_character(to_char_type('#'));
write_number_with_ubjson_prefix(j.m_data.m_value.array->size(), true, use_bjdata);
}
return !use_count;
}
/// @return true when a closing '}' still has to be written after the elements
bool write_ubjson_start_object(const BasicJsonType& j, const bool use_count, const bool use_type,
const bool add_prefix, const bool use_bjdata, bool& prefix_required)
{
prefix_required = true;
if (add_prefix)
{
oa.write_character(to_char_type('{'));
}
if (use_type && !j.m_data.m_value.object->empty())
{
if (!use_count)
{
JSON_THROW(other_error::create(502, "use_type requires use_size = true", &j));
}
const CharType first_prefix = ubjson_prefix(j.front(), use_bjdata);
const bool same_prefix = std::all_of(j.begin(), j.end(),
[this, first_prefix, use_bjdata](const BasicJsonType & v)
{
return ubjson_prefix(v, use_bjdata) == first_prefix;
});
if (same_prefix && !(use_bjdata && is_bjdata_excluded_type_marker(first_prefix)))
{
prefix_required = false;
oa.write_character(to_char_type('$'));
oa.write_character(first_prefix);
}
}
if (use_count)
{
oa.write_character(to_char_type('#'));
write_number_with_ubjson_prefix(j.m_data.m_value.object->size(), true, use_bjdata);
}
return !use_count;
}
/*!
@brief whether @a j is a BJData ND-array annotation object
(https://github.com/NeuroJSON/jdata)
Used by both the recursive object case of @ref write_ubjson and
@ref write_ubjson_value_or_push, which must agree on what counts as an
ND-array: @a j is only actually written as one once @ref
write_bjdata_ndarray has also accepted its contents.
@pre @a j.is_object()
*/
static bool is_bjdata_ndarray(const BasicJsonType& j)
{
const auto& object = *j.m_data.m_value.object;
return object.size() == 3
&& object.find("_ArrayType_") != object.end()
&& object.find("_ArraySize_") != object.end()
&& object.find("_ArrayData_") != object.end();
}
/// @brief an object or array @ref write_ubjson_iterative is still writing
/// the elements of
struct ubjson_frame
{
ubjson_frame(const BasicJsonType* value_, const bool prefix_required_) noexcept
: value(value_)
, prefix_required(prefix_required_)
{
if (value->is_object())
{
object_it = value->m_data.m_value.object->cbegin();
}
else
{
array_it = value->m_data.m_value.array->cbegin();
}
}
// declared for GCC's -Weffc++, which asks for them in a class with
// pointer members and a non-trivial destructor; the exception
// specifications are left implicit, as GCC 4.8 rejects explicit ones
// that differ from them
ubjson_frame(const ubjson_frame&) = default;
ubjson_frame(ubjson_frame&&) = default;
ubjson_frame& operator=(const ubjson_frame&) = default;
ubjson_frame& operator=(ubjson_frame&&) = default;
~ubjson_frame() = default;
/// the array or object being written
const BasicJsonType* value;
/// whether value's elements each carry their own type marker; an
/// optimized ($type) container writes it once for all of them instead
bool prefix_required;
typename BasicJsonType::object_t::const_iterator object_it{};
typename BasicJsonType::array_t::const_iterator array_it{};
};
/*!
@brief write @a j with @ref write_ubjson, or write its header and push a
frame for @ref write_ubjson_iterative to continue with its elements
@param[in] add_prefix whether @a j's own type marker is written now (the
elements of an optimized container, and everything below the
top level, never repeat it)
@sa @ref write_cbor_value_or_push
*/
void write_ubjson_value_or_push(const BasicJsonType& j, const bool add_prefix, const bool use_count,
const bool use_type, const bool use_bjdata, const bjdata_version_t bjdata_version,
std::vector<ubjson_frame>& stack)
{
if (!j.is_array() && !j.is_object())
{
write_ubjson(j, use_count, use_type, add_prefix, use_bjdata, bjdata_version);
return;
}
if (use_bjdata && j.is_object() && is_bjdata_ndarray(j)
&& !write_bjdata_ndarray(*j.m_data.m_value.object, use_count, use_type, bjdata_version))
{
// fully written as an ND-array: nothing below it to come back to
return;
}
const bool is_array = j.is_array();
bool prefix_required = true;
if (is_array)
{
write_ubjson_start_array(j, use_count, use_type, add_prefix, use_bjdata, prefix_required);
}
else
{
write_ubjson_start_object(j, use_count, use_type, add_prefix, use_bjdata, prefix_required);
}
const bool empty = is_array ? j.m_data.m_value.array->empty() : j.m_data.m_value.object->empty();
if (!empty)
{
stack.emplace_back(&j, prefix_required);
return;
}
// write_ubjson_start_array/_object return !use_count, i.e. whether a
// closer still has to be written; use_count is constant for the whole
// document, so that is recomputed here instead of being carried along
if (!use_count)
{
oa.write_character(to_char_type(is_array ? ']' : '}'));
}
}
/*!
@brief write out @a root and everything below it without the call stack
@sa @ref write_cbor_iterative
*/
void write_ubjson_iterative(const BasicJsonType& root, const bool use_count, const bool use_type,
const bool add_prefix, const bool use_bjdata, const bjdata_version_t bjdata_version)
{
std::vector<ubjson_frame> stack;
write_ubjson_value_or_push(root, add_prefix, use_count, use_type, use_bjdata, bjdata_version, stack);
while (!stack.empty())
{
const ubjson_frame current = stack.back();
const BasicJsonType& j = *current.value;
if (j.is_array())
{
const auto& array = *j.m_data.m_value.array;
if (current.array_it == array.cend())
{
if (!use_count)
{
oa.write_character(to_char_type(']'));
}
stack.pop_back();
continue;
}
const BasicJsonType* child = &(*current.array_it);
const bool child_prefix = current.prefix_required;
++stack.back().array_it;
write_ubjson_value_or_push(*child, child_prefix, use_count, use_type, use_bjdata, bjdata_version, stack);
}
else
{
const auto& object = *j.m_data.m_value.object;
if (current.object_it == object.cend())
{
if (!use_count)
{
oa.write_character(to_char_type('}'));
}
stack.pop_back();
continue;
}
string_t storage;
const string_t& key = sanitize_utf8_for_write(current.object_it->first, j, storage);
write_number_with_ubjson_prefix(key.size(), true, use_bjdata);
oa.write_characters(
reinterpret_cast<const CharType*>(key.data()),
key.size());
const BasicJsonType* child = &(current.object_it->second);
const bool child_prefix = current.prefix_required;
++stack.back().object_it;
write_ubjson_value_or_push(*child, child_prefix, use_count, use_type, use_bjdata, bjdata_version, stack);
}
}
}
//////////
// BSON //
//////////
@@ -23310,16 +23779,31 @@ class binary_writer
/*!
@brief validate (dry_run) or write one BJData ND-array element of dtype 'd' (single precision)
@return whether @a el's value fits a float without overflow; always true when @a dry_run is false
@return whether @a el's value survives narrowing to float and back without any change
(so the ND-array round-trips exactly), or is infinite or NaN; always true when
@a dry_run is false
*/
bool write_bjdata_ndarray_float_element(const BasicJsonType& el, const bool dry_run)
{
const auto dval = el.template get<double>();
if (dry_run)
{
return !std::isfinite(dval) ||
(dval >= static_cast<double>(std::numeric_limits<float>::lowest()) &&
dval <= static_cast<double>((std::numeric_limits<float>::max)()));
#ifdef __GNUC__
JSON_HEDLEY_DIAGNOSTIC_PUSH
JSON_HEDLEY_PRAGMA(GCC diagnostic ignored "-Wfloat-equal")
#endif
// a value that would be rounded (rather than exactly represented) by the
// narrowing to float is treated like an out-of-range integer element; this
// is the same criterion write_compact_float() uses for CBOR/MessagePack
const bool in_range = std::isnan(dval) ||
(dval >= static_cast<double>(std::numeric_limits<float>::lowest()) &&
dval <= static_cast<double>((std::numeric_limits<float>::max)()) &&
static_cast<double>(static_cast<float>(dval)) == dval) ||
std::isinf(dval);
#ifdef __GNUC__
JSON_HEDLEY_DIAGNOSTIC_POP
#endif
return in_range;
}
write_number(static_cast<float>(dval), true);
return true;
+42 -4
View File
@@ -19,6 +19,7 @@
#include <nlohmann/json.hpp>
using nlohmann::json;
using ordered_json = nlohmann::ordered_json;
#include <algorithm>
#include <climits>
@@ -2258,29 +2259,33 @@ TEST_CASE("BJData")
SECTION("start_array() in ndarray _ArraySize_")
{
// _ArrayType_ (2 events: key + string) is now emitted before
// _ArraySize_ (see GitHub issue #5661), which shifts the events
// below later by the same 2 events
std::vector<uint8_t> const v = {'[', '$', 'i', '#', '[', '$', 'i', '#', 'i', 2, 2, 1, 1, 2};
SaxCountdown scp(2);
SaxCountdown scp(4);
CHECK_FALSE(json::sax_parse(v, &scp, json::input_format_t::bjdata));
}
SECTION("number_integer() in ndarray _ArraySize_")
{
std::vector<uint8_t> const v = {'[', '$', 'U', '#', '[', '$', 'i', '#', 'i', 2, 2, 1, 1, 2};
SaxCountdown scp(3);
SaxCountdown scp(5);
CHECK_FALSE(json::sax_parse(v, &scp, json::input_format_t::bjdata));
}
SECTION("key() in ndarray _ArrayType_")
{
// _ArrayType_ is emitted right after start_object(), before _ArraySize_
std::vector<uint8_t> const v = {'[', '$', 'U', '#', '[', '$', 'U', '#', 'i', 2, 2, 2, 1, 2, 3, 4};
SaxCountdown scp(6);
SaxCountdown scp(1);
CHECK_FALSE(json::sax_parse(v, &scp, json::input_format_t::bjdata));
}
SECTION("string() in ndarray _ArrayType_")
{
std::vector<uint8_t> const v = {'[', '$', 'U', '#', '[', '$', 'U', '#', 'i', 2, 2, 2, 1, 2, 3, 4};
SaxCountdown scp(7);
SaxCountdown scp(2);
CHECK_FALSE(json::sax_parse(v, &scp, json::input_format_t::bjdata));
}
@@ -2883,6 +2888,22 @@ TEST_CASE("BJData")
CHECK(out_single.at(0) == '{');
CHECK(json::from_bjdata(out_single) == j_single);
// a double element that is finite and within the range of "single"
// but is not exactly representable as a float, so narrowing it would
// silently round it (0.1 is read back as 0.10000000149011612); this,
// like the overflow case above, falls back to a plain object (see
// GitHub issue #5661)
json const j_single_rounded = json({{"_ArrayType_", "single"}, {"_ArraySize_", {2, 1}}, {"_ArrayData_", {1.5, 0.1}}});
const auto out_single_rounded = json::to_bjdata(j_single_rounded);
CHECK(out_single_rounded.at(0) == '{');
CHECK(json::from_bjdata(out_single_rounded) == j_single_rounded);
// a double element that underflows to 0 when narrowed to "single"
json const j_single_underflow = json({{"_ArrayType_", "single"}, {"_ArraySize_", {2, 1}}, {"_ArrayData_", {1.5, 1e-300}}});
const auto out_single_underflow = json::to_bjdata(j_single_underflow);
CHECK(out_single_underflow.at(0) == '{');
CHECK(json::from_bjdata(out_single_underflow) == j_single_underflow);
// in-range boundary values still use the compact ndarray encoding
json const j_uint8_ok = json({{"_ArrayType_", "uint8"}, {"_ArraySize_", {2, 1}}, {"_ArrayData_", {0, 255}}});
CHECK(json::to_bjdata(j_uint8_ok) == std::vector<uint8_t>({'[', '$', 'U', '#', '[', 'i', 2, 'i', 1, ']', 0, 255}));
@@ -2896,6 +2917,23 @@ TEST_CASE("BJData")
CHECK(json::from_bjdata(out_single_ok) == json({{"_ArrayType_", "single"}, {"_ArraySize_", {2, 1}}, {"_ArrayData_", {1.5f, -1.5f}}}));
}
SECTION("ndarray annotation keys are read back in the documented order")
{
// from_bjdata() must emit the annotation object's keys in the order
// used throughout the documentation, _ArrayType_, _ArraySize_,
// _ArrayData_: the type marker precedes the dimension vector on the
// wire (see get_ubjson_size_type()), so it is known, and emitted,
// before _ArraySize_. For a plain json this key order is invisible
// (its comparison ignores it), but for an ordered_json it is not (see
// GitHub issue #5661).
const ordered_json o = ordered_json::parse(R"({"_ArrayType_":"uint8","_ArraySize_":[2,2],"_ArrayData_":[1,2,3,4]})");
const auto packed = ordered_json::to_bjdata(o);
CHECK(packed.at(0) == '[');
const ordered_json o_back = ordered_json::from_bjdata(packed);
CHECK(o_back == o);
CHECK(o_back.dump() == o.dump());
}
SECTION("ndarray that would not be read back as an annotated object stays as object")
{
// the reader only restores an annotated object from an ND-array
+39 -8
View File
@@ -12,8 +12,10 @@
using json = nlohmann::json;
using ordered_json = nlohmann::ordered_json;
#include <limits>
#include <set>
#include <string>
#include <unordered_set>
namespace
{
@@ -91,6 +93,9 @@ TEST_CASE("hash<nlohmann::json>")
// Collect hashes for different JSON values and make sure that they are distinct
// We cannot compare against fixed values, because the implementation of
// std::hash may differ between compilers.
//
// numbers that compare equal under operator== (0 == 0U == 0.0) must hash
// equally, so they are only inserted once below and checked separately.
std::set<std::size_t> hashes;
@@ -107,10 +112,7 @@ TEST_CASE("hash<nlohmann::json>")
// number
hashes.insert(std::hash<json> {}(json(0)));
hashes.insert(std::hash<json> {}(json(static_cast<unsigned>(0))));
hashes.insert(std::hash<json> {}(json(-1)));
hashes.insert(std::hash<json> {}(json(0.0)));
hashes.insert(std::hash<json> {}(json(42.23)));
// array
@@ -132,7 +134,36 @@ TEST_CASE("hash<nlohmann::json>")
// discarded
hashes.insert(std::hash<json> {}(json(json::value_t::discarded)));
CHECK(hashes.size() == 21);
CHECK(hashes.size() == 19);
// numbers that compare equal under operator== must hash equally,
// regardless of which of number_integer, number_unsigned, or
// number_float actually holds the value
CHECK(json(0) == json(static_cast<unsigned>(0)));
CHECK(json(0) == json(0.0));
CHECK(std::hash<json> {}(json(0)) == std::hash<json> {}(json(static_cast<unsigned>(0))));
CHECK(std::hash<json> {}(json(0)) == std::hash<json> {}(json(0.0)));
CHECK(std::hash<json> {}(json(-1)) == std::hash<json> {}(json(-1.0)));
// a std::unordered_set relies on this same consistency between == and hash
const std::unordered_set<json> numbers {json(0), json(static_cast<unsigned>(0)), json(0.0)};
CHECK(numbers.size() == 1);
// -0.0 compares equal to 0 and 0.0
CHECK(json(-0.0) == json(0));
CHECK(std::hash<json> {}(json(-0.0)) == std::hash<json> {}(json(0)));
CHECK(std::hash<json> {}(json(-0.0)) == std::hash<json> {}(json(0.0)));
// the ends of the integer ranges, which equal floats exactly
const auto int_min = (std::numeric_limits<json::number_integer_t>::min)();
const auto int_max = (std::numeric_limits<json::number_integer_t>::max)();
const auto two_63 = json::number_unsigned_t(1) << 63U;
CHECK(json(int_min) == json(-9223372036854775808.0));
CHECK(std::hash<json> {}(json(int_min)) == std::hash<json> {}(json(-9223372036854775808.0)));
CHECK(json(two_63) == json(9223372036854775808.0));
CHECK(std::hash<json> {}(json(two_63)) == std::hash<json> {}(json(9223372036854775808.0)));
CHECK(json(json::number_unsigned_t(int_max)) == json(int_max));
CHECK(std::hash<json> {}(json(json::number_unsigned_t(int_max))) == std::hash<json> {}(json(int_max)));
}
TEST_CASE("hash<nlohmann::ordered_json>")
@@ -156,10 +187,7 @@ TEST_CASE("hash<nlohmann::ordered_json>")
// number
hashes.insert(std::hash<ordered_json> {}(ordered_json(0)));
hashes.insert(std::hash<ordered_json> {}(ordered_json(static_cast<unsigned>(0))));
hashes.insert(std::hash<ordered_json> {}(ordered_json(-1)));
hashes.insert(std::hash<ordered_json> {}(ordered_json(0.0)));
hashes.insert(std::hash<ordered_json> {}(ordered_json(42.23)));
// array
@@ -181,7 +209,10 @@ TEST_CASE("hash<nlohmann::ordered_json>")
// discarded
hashes.insert(std::hash<ordered_json> {}(ordered_json(ordered_json::value_t::discarded)));
CHECK(hashes.size() == 21);
CHECK(hashes.size() == 19);
CHECK(std::hash<ordered_json> {}(ordered_json(0)) == std::hash<ordered_json> {}(ordered_json(static_cast<unsigned>(0))));
CHECK(std::hash<ordered_json> {}(ordered_json(0)) == std::hash<ordered_json> {}(ordered_json(0.0)));
}
TEST_CASE("hash of deeply nested values")
+197
View File
@@ -13,6 +13,7 @@ using nlohmann::json;
#include <algorithm>
#include <string>
#include <utility>
#include <vector>
TEST_CASE("tests on very large JSONs")
@@ -354,3 +355,199 @@ TEST_CASE("tests on deeply nested JSONs")
}
}
namespace
{
json nested_array(const std::size_t depth, json leaf)
{
json j = std::move(leaf);
for (std::size_t i = 0; i < depth; ++i)
{
json a = json::array();
a.push_back(std::move(j));
j = std::move(a);
}
return j;
}
json nested_object(const std::size_t depth, json leaf)
{
json j = std::move(leaf);
for (std::size_t i = 0; i < depth; ++i)
{
json o = json::object();
o["k"] = std::move(j);
j = std::move(o);
}
return j;
}
} // namespace
TEST_CASE("issue #5392 - binary writers on deeply nested values")
{
// 200 is past the point where the writers stop recursing, and still
// shallow enough that from_* and operator== (which still recurse) are fine.
const json deep_array = nested_array(200, json(0));
const json deep_object = nested_object(200, json("x"));
const json empty_array = nested_array(200, json::array());
const json empty_object = nested_object(200, json::object());
const json mixed = nested_object(80, nested_array(80, json(true)));
SECTION("roundtrip past the recursion bound")
{
CHECK(json::from_cbor(json::to_cbor(deep_array)) == deep_array);
CHECK(json::from_msgpack(json::to_msgpack(deep_array)) == deep_array);
CHECK(json::from_ubjson(json::to_ubjson(deep_array)) == deep_array);
CHECK(json::from_ubjson(json::to_ubjson(deep_array, true, false)) == deep_array);
CHECK(json::from_ubjson(json::to_ubjson(deep_array, true, true)) == deep_array);
CHECK(json::from_bjdata(json::to_bjdata(deep_array)) == deep_array);
CHECK(json::from_cbor(json::to_cbor(deep_object)) == deep_object);
CHECK(json::from_msgpack(json::to_msgpack(deep_object)) == deep_object);
CHECK(json::from_ubjson(json::to_ubjson(deep_object)) == deep_object);
CHECK(json::from_ubjson(json::to_ubjson(deep_object, true, true)) == deep_object);
CHECK(json::from_bjdata(json::to_bjdata(deep_object)) == deep_object);
CHECK(json::from_cbor(json::to_cbor(empty_array)) == empty_array);
CHECK(json::from_msgpack(json::to_msgpack(empty_array)) == empty_array);
CHECK(json::from_ubjson(json::to_ubjson(empty_array)) == empty_array);
CHECK(json::from_ubjson(json::to_ubjson(empty_array, true, true)) == empty_array);
CHECK(json::from_cbor(json::to_cbor(empty_object)) == empty_object);
CHECK(json::from_msgpack(json::to_msgpack(empty_object)) == empty_object);
CHECK(json::from_ubjson(json::to_ubjson(empty_object)) == empty_object);
CHECK(json::from_cbor(json::to_cbor(mixed)) == mixed);
CHECK(json::from_msgpack(json::to_msgpack(mixed)) == mixed);
CHECK(json::from_ubjson(json::to_ubjson(mixed)) == mixed);
CHECK(json::from_bjdata(json::to_bjdata(mixed)) == mixed);
}
SECTION("the two ways of writing a value meet at the bound")
{
for (std::size_t depth = 120; depth <= 140; ++depth)
{
CAPTURE(depth);
const json array = nested_array(depth, json(7));
CHECK(json::from_cbor(json::to_cbor(array)) == array);
CHECK(json::from_msgpack(json::to_msgpack(array)) == array);
CHECK(json::from_ubjson(json::to_ubjson(array, true, true)) == array);
const json object = nested_object(depth, json(7));
CHECK(json::from_cbor(json::to_cbor(object)) == object);
CHECK(json::from_msgpack(json::to_msgpack(object)) == object);
CHECK(json::from_bjdata(json::to_bjdata(object)) == object);
}
}
SECTION("a BJData ndarray below the bound is still an ndarray")
{
const json ndarray = json({{"_ArrayType_", "uint8"}, {"_ArraySize_", {2, 3}}, {"_ArrayData_", {1, 2, 3, 4, 5, 6}}});
const json invalid = json({{"_ArrayType_", "nope"}, {"_ArraySize_", {1}}, {"_ArrayData_", {1}}});
const json deep_ndarray = nested_array(140, ndarray);
const json deep_invalid = nested_array(140, invalid);
CHECK(json::from_bjdata(json::to_bjdata(deep_ndarray)) == deep_ndarray);
CHECK(json::from_bjdata(json::to_bjdata(deep_invalid)) == deep_invalid);
CHECK(json::from_bjdata(json::to_bjdata(ndarray)) == ndarray);
}
SECTION("byte-exact across the switch-over")
{
// nested one-element arrays around the recursion bound: the exact
// bytes a writer produces do not depend on whether it stayed on the
// call stack or moved to the heap one partway through
for (const std::size_t depth :
{
nlohmann::detail::recursion_depth_limit() - 1, nlohmann::detail::recursion_depth_limit(),
nlohmann::detail::recursion_depth_limit() + 1, nlohmann::detail::recursion_depth_limit() + 2
})
{
CAPTURE(depth);
const json array = nested_array(depth, json(0));
std::vector<std::uint8_t> expected_cbor(depth, 0x81);
expected_cbor.push_back(0x00);
CHECK(json::to_cbor(array) == expected_cbor);
std::vector<std::uint8_t> expected_msgpack(depth, 0x91);
expected_msgpack.push_back(0x00);
CHECK(json::to_msgpack(array) == expected_msgpack);
std::string expected_ubjson(depth, '[');
expected_ubjson += "i";
expected_ubjson += '\0';
expected_ubjson.append(depth, ']');
const auto packed_ubjson = json::to_ubjson(array);
CHECK(std::string(packed_ubjson.begin(), packed_ubjson.end()) == expected_ubjson);
}
}
SECTION("a deep object, and a BJData ndarray, past the recursion bound")
{
const std::size_t depth = nlohmann::detail::recursion_depth_limit() + 50;
const json object = nested_object(depth, json(42));
CHECK(json::from_cbor(json::to_cbor(object)) == object);
CHECK(json::from_msgpack(json::to_msgpack(object)) == object);
CHECK(json::from_ubjson(json::to_ubjson(object, true, true)) == object);
CHECK(json::from_bjdata(json::to_bjdata(object)) == object);
const json ndarray = json({{"_ArrayType_", "uint8"}, {"_ArraySize_", {2, 3}}, {"_ArrayData_", {1, 2, 3, 4, 5, 6}}});
const json deep_ndarray = nested_array(depth, ndarray);
CHECK(json::from_bjdata(json::to_bjdata(deep_ndarray)) == deep_ndarray);
}
SECTION("a discarded value past the recursion bound still throws type_error.321")
{
const std::size_t depth = nlohmann::detail::recursion_depth_limit() + 50;
const json discarded_leaf(json::value_t::discarded);
const json deep_discarded = nested_array(depth, discarded_leaf);
CHECK_THROWS_WITH_AS(json::to_cbor(deep_discarded), "[json.exception.type_error.321] cannot serialize discarded value to CBOR", json::type_error);
CHECK_THROWS_WITH_AS(json::to_msgpack(deep_discarded), "[json.exception.type_error.321] cannot serialize discarded value to MessagePack", json::type_error);
CHECK_THROWS_WITH_AS(json::to_ubjson(deep_discarded), "[json.exception.type_error.321] cannot serialize discarded value to UBJSON", json::type_error);
CHECK_THROWS_WITH_AS(json::to_bjdata(deep_discarded), "[json.exception.type_error.321] cannot serialize discarded value to BJData", json::type_error);
}
SECTION("does not overflow the C++ stack")
{
const std::size_t depth = 100000;
const json j = json::parse(std::string(depth, '[') + "0" + std::string(depth, ']'));
std::vector<std::uint8_t> packed;
CHECK_NOTHROW(packed = json::to_cbor(j));
CHECK(json::from_cbor(packed) == j);
CHECK_NOTHROW(packed = json::to_msgpack(j));
CHECK(json::from_msgpack(packed) == j);
CHECK_NOTHROW(packed = json::to_ubjson(j));
CHECK(json::from_ubjson(packed) == j);
CHECK_NOTHROW(packed = json::to_ubjson(j, true, false));
CHECK(json::from_ubjson(packed) == j);
CHECK_NOTHROW(packed = json::to_bjdata(j));
CHECK(json::from_bjdata(packed) == j);
}
SECTION("regression test for https://issues.oss-fuzz.com/issues/566583014")
{
// 200000 nested one-element CBOR arrays, the innermost holding null;
// round-tripping this used to recurse once per level on the way back
// out through to_cbor(), deep enough to overflow the stack
std::vector<std::uint8_t> v(200000, 0x81);
v.push_back(0xf6);
const json j = json::from_cbor(v);
CHECK(json::to_cbor(j) == v);
// the MessagePack analogue: fixarray of 1 nesting down to nil
std::vector<std::uint8_t> v_msgpack(200000, 0x91);
v_msgpack.push_back(0xc0);
const json j_msgpack = json::from_msgpack(v_msgpack);
CHECK(json::to_msgpack(j_msgpack) == v_msgpack);
}
}
+9
View File
@@ -888,6 +888,15 @@ TEST_CASE("regression tests 2")
CHECK(j == k);
}
SECTION("issue #4552 - UTF-8 invalid characters are not always ignored when dumping with error_handler_t::ignore")
{
json node;
node["test"] = "test\334\005";
CHECK(node.dump(-1, ' ', false, json::error_handler_t::ignore) == "{\"test\":\"test\\u0005\"}");
CHECK(node.dump(-1, ' ', false, json::error_handler_t::keep) == "{\"test\":\"test\334\\u0005\"}");
CHECK(node.dump(-1, ' ', true, json::error_handler_t::keep) == "{\"test\":\"test\334\\u0005\"}");
}
#ifdef JSON_HAS_CPP_17
SECTION("issue #5066 - MSVC converts json to std::variant<json> via the conversion operator")
{
+37
View File
@@ -102,6 +102,8 @@ TEST_CASE("serialization")
CHECK(j.dump(-1, ' ', false, json::error_handler_t::ignore) == "\"äü\"");
CHECK(j.dump(-1, ' ', false, json::error_handler_t::replace) == "\"ä\xEF\xBF\xBDü\"");
CHECK(j.dump(-1, ' ', true, json::error_handler_t::replace) == "\"\\u00e4\\ufffd\\u00fc\"");
CHECK(j.dump(-1, ' ', false, json::error_handler_t::keep) == "\"ä\xA9ü\"");
CHECK(j.dump(-1, ' ', true, json::error_handler_t::keep) == "\"\\u00e4\xA9\\u00fc\"");
}
SECTION("invalid character (regression guard for shared UTF-8 decoder, see #5529)")
@@ -124,6 +126,8 @@ TEST_CASE("serialization")
CHECK(j.dump(-1, ' ', false, json::error_handler_t::ignore) == "\"123\"");
CHECK(j.dump(-1, ' ', false, json::error_handler_t::replace) == "\"123\xEF\xBF\xBD\"");
CHECK(j.dump(-1, ' ', true, json::error_handler_t::replace) == "\"123\\ufffd\"");
CHECK(j.dump(-1, ' ', false, json::error_handler_t::keep) == "\"123\xC2\"");
CHECK(j.dump(-1, ' ', true, json::error_handler_t::keep) == "\"123\xC2\"");
}
SECTION("unexpected character")
@@ -136,6 +140,39 @@ TEST_CASE("serialization")
CHECK(j.dump(-1, ' ', false, json::error_handler_t::ignore) == "\"123456\"");
CHECK(j.dump(-1, ' ', false, json::error_handler_t::replace) == "\"123\xEF\xBF\xBD\x34\x35\x36\"");
CHECK(j.dump(-1, ' ', true, json::error_handler_t::replace) == "\"123\\ufffd456\"");
CHECK(j.dump(-1, ' ', false, json::error_handler_t::keep) == "\"123\xF1\xB0\x34\x35\x36\"");
CHECK(j.dump(-1, ' ', true, json::error_handler_t::keep) == "\"123\xF1\xB0\x34\x35\x36\"");
}
SECTION("keep: valid characters are still escaped")
{
// an invalid byte followed by characters that must be escaped
const json j = "\xC2\"\\\n\xFF\x05";
CHECK(j.dump(-1, ' ', false, json::error_handler_t::keep) == "\"\xC2\\\"\\\\\\n\xFF\\u0005\"");
CHECK(j.dump(-1, ' ', true, json::error_handler_t::keep) == "\"\xC2\\\"\\\\\\n\xFF\\u0005\"");
}
SECTION("keep: truncated multibyte sequences")
{
CHECK(json("\xF0\x9F\x98").dump(-1, ' ', false, json::error_handler_t::keep) == "\"\xF0\x9F\x98\"");
CHECK(json("\xF0\x9F\x98").dump(-1, ' ', true, json::error_handler_t::keep) == "\"\xF0\x9F\x98\"");
CHECK(json("\xF0\x9F\x98" "a").dump(-1, ' ', false, json::error_handler_t::keep) == "\"\xF0\x9F\x98" "a\"");
CHECK(json("\xF0\x9F\x98" "a").dump(-1, ' ', true, json::error_handler_t::keep) == "\"\xF0\x9F\x98" "a\"");
}
SECTION("keep: long string with many invalid bytes")
{
// exceeds the internal string buffer several times
std::string input;
std::string expected = "\"";
for (int i = 0; i < 2000; ++i)
{
input += "\xFF\xE2\x82\n\xC3\xA4";
expected += "\xFF\xE2\x82\\n\xC3\xA4";
}
expected += "\"";
const json j = input;
CHECK(j.dump(-1, ' ', false, json::error_handler_t::keep) == expected);
}
SECTION("U+FFFD Substitution of Maximal Subparts")
+25 -1
View File
@@ -11,6 +11,7 @@
#include <nlohmann/json.hpp>
using nlohmann::json;
#include <algorithm>
#include <fstream>
#include <string>
#include "make_test_data_available.hpp"
@@ -67,8 +68,11 @@ void check_utf8dump(bool success_expected, int byte1, int byte2 = -1, int byte3
static std::string s_replaced2;
static std::string s_replaced_ascii;
static std::string s_replaced2_ascii;
static std::string s_kept;
static std::string s_kept2;
static std::string s_kept_ascii;
// dumping with ignore/replace must not throw in any case
// dumping with ignore/replace/keep must not throw in any case
s_ignored = j.dump(-1, ' ', false, json::error_handler_t::ignore);
s_ignored2 = j2.dump(-1, ' ', false, json::error_handler_t::ignore);
s_ignored_ascii = j.dump(-1, ' ', true, json::error_handler_t::ignore);
@@ -77,6 +81,9 @@ void check_utf8dump(bool success_expected, int byte1, int byte2 = -1, int byte3
s_replaced2 = j2.dump(-1, ' ', false, json::error_handler_t::replace);
s_replaced_ascii = j.dump(-1, ' ', true, json::error_handler_t::replace);
s_replaced2_ascii = j2.dump(-1, ' ', true, json::error_handler_t::replace);
s_kept = j.dump(-1, ' ', false, json::error_handler_t::keep);
s_kept2 = j2.dump(-1, ' ', false, json::error_handler_t::keep);
s_kept_ascii = j.dump(-1, ' ', true, json::error_handler_t::keep);
if (success_expected)
{
@@ -86,6 +93,7 @@ void check_utf8dump(bool success_expected, int byte1, int byte2 = -1, int byte3
// all dumps should agree on the string
CHECK(s_strict == s_ignored);
CHECK(s_strict == s_replaced);
CHECK(s_strict == s_kept);
}
else
{
@@ -97,6 +105,20 @@ void check_utf8dump(bool success_expected, int byte1, int byte2 = -1, int byte3
// check that replace string contains a replacement character
CHECK(s_replaced.find("\xEF\xBF\xBD") != std::string::npos);
// ignore drops the invalid bytes, keep copies them
CHECK(s_ignored != s_kept);
CHECK(s_ignored_ascii != s_kept_ascii);
// unless a byte needs escaping, keep copies the input unchanged
const bool needs_escaping = std::any_of(json_string.begin(), json_string.end(), [](char c)
{
return static_cast<unsigned char>(c) < 0x20 || c == '"' || c == '\\';
});
if (!needs_escaping)
{
CHECK(s_kept == "\"" + json_string + "\"");
}
}
// check that prefix and suffix are preserved
@@ -108,6 +130,8 @@ void check_utf8dump(bool success_expected, int byte1, int byte2 = -1, int byte3
CHECK(s_replaced2.substr(s_replaced2.size() - 4, 3) == "xyz");
CHECK(s_replaced2_ascii.substr(1, 3) == "abc");
CHECK(s_replaced2_ascii.substr(s_replaced2_ascii.size() - 4, 3) == "xyz");
CHECK(s_kept2.substr(1, 3) == "abc");
CHECK(s_kept2.substr(s_kept2.size() - 4, 3) == "xyz");
}
void check_utf8string(bool success_expected, int byte1, int byte2, int byte3, int byte4);
+21
View File
@@ -0,0 +1,21 @@
# check_build_options
Checks that the Meson build and the pkg-config files offer the same options as the CMake target, so that a new CMake
option is not forgotten in one of them.
The compile definitions of the CMake target (`target_compile_definitions` in [`CMakeLists.txt`](../../CMakeLists.txt))
are the reference. When you add an option there, also add it to
- the pkg-config block in `CMakeLists.txt` (`NLOHMANN_JSON_PKGCONFIG_CFLAGS`),
- [`meson_options.txt`](../../meson_options.txt), named without the `JSON_` prefix and with the same default,
- [`meson.build`](../../meson.build) (`json_defines`), and
- the list of Meson options in
[`docs/mkdocs/docs/integration/package_managers.md`](../../docs/mkdocs/docs/integration/package_managers.md).
Run the check with
```shell
make check_build_options
```
It needs only Python 3 and runs in the `ci_meson_install` CI job.
+144
View File
@@ -0,0 +1,144 @@
#!/usr/bin/env python3
"""Check that the Meson build and the pkg-config files offer the CMake options.
The compile definitions of the CMake target (target_compile_definitions in
CMakeLists.txt) are the reference. For every option used there, the script
checks that
- the CMake pkg-config file adds the same definition under the same condition,
- meson_options.txt has a boolean option of the same name without the "JSON_"
prefix and with the same default,
- meson.build adds the same definition under the same condition, and
- the Meson section of the package manager documentation lists the option.
Meson's MultipleHeaders option selects the include directory and adds no
definition; it is the only Meson option without a definition.
"""
import argparse
import os
import re
import sys
REPO_ROOT = os.path.normpath(os.path.join(sys.path[0], '..', '..'))
DOCS = os.path.join('docs', 'mkdocs', 'docs', 'integration', 'package_managers.md')
# Meson options that add no compile definition, with their default
MESON_ONLY = {'MultipleHeaders': 'false'}
def read(root, path):
with open(os.path.join(root, path), encoding='utf-8') as f:
return f.read()
def cmake_target_definitions(cmake):
"""Return {option: (definition, add_if_on)} from target_compile_definitions."""
block = re.search(r'target_compile_definitions\(\s*\$\{NLOHMANN_JSON_TARGET_NAME\}\s*INTERFACE(.*?)\n\)', cmake, re.S)
if not block:
sys.exit('CMakeLists.txt: target_compile_definitions of the target not found')
result = {}
for line in block.group(1).split('\n'):
line = line.strip()
if not line:
continue
m = re.fullmatch(r'\$<\$<NOT:\$<BOOL:\$\{JSON_(\w+)\}>>:(\w+=\w+)>', line)
if m:
result[m.group(1)] = (m.group(2), False)
continue
m = re.fullmatch(r'\$<\$<BOOL:\$\{JSON_(\w+)\}>:(\w+=\w+)>', line)
if m:
result[m.group(1)] = (m.group(2), True)
continue
sys.exit(f'CMakeLists.txt: unexpected line in target_compile_definitions: {line}')
return result
def cmake_defaults(cmake):
"""Return {option: 'true'/'false'} for the option() calls of the form JSON_<name>."""
return {m.group(1): 'true' if m.group(2) == 'ON' else 'false'
for m in re.finditer(r'^option\(JSON_(\w+)\s+"[^"]*"\s+(ON|OFF)\)', cmake, re.M)}
def cmake_pkgconfig_definitions(cmake):
"""Return {option: (definition, add_if_on)} from the pkg-config block."""
return {m.group(2): (m.group(3), m.group(1) is None)
for m in re.finditer(r'if \((NOT )?JSON_(\w+)\)\s*\n\s*string\(APPEND NLOHMANN_JSON_PKGCONFIG_CFLAGS " -D(\w+=\w+)"\)', cmake)}
def meson_options(options):
"""Return {option: default} for the boolean options in meson_options.txt."""
result = {}
for block in re.findall(r'option\((.*?)\)', options, re.S):
name = re.search(r"'(\w+)'", block).group(1)
kind = re.search(r"type\s*:\s*'(\w+)'", block)
value = re.search(r'value\s*:\s*(\w+)', block)
result[name] = value.group(1) if kind and kind.group(1) == 'boolean' and value else None
return result
def meson_definitions(meson):
"""Return {option: (definition, add_if_on)} from meson.build."""
return {m.group(2): (m.group(3), m.group(1) is None)
for m in re.finditer(r"if (not )?get_option\('(\w+)'\)\s*\n\s*json_defines \+= '(\w+=\w+)'", meson)}
def describe(definition):
name, add_if_on = definition
return f'{name} if {"enabled" if add_if_on else "disabled"}'
def compare(errors, where, expected, actual):
for option, definition in expected.items():
if option not in actual:
errors.append(f'{where}: no definition for option {option} (expected {describe(definition)})')
elif actual[option] != definition:
errors.append(f'{where}: option {option} adds {describe(actual[option])}, expected {describe(definition)}')
for option in actual.keys() - expected.keys():
errors.append(f'{where}: definition for option {option}, which the CMake target does not have')
def main():
parser = argparse.ArgumentParser(description=__doc__.split('\n')[0])
parser.add_argument('root', nargs='?', default=REPO_ROOT, help='repository root (default: %(default)s)')
root = parser.parse_args().root
cmake = read(root, 'CMakeLists.txt')
reference = cmake_target_definitions(cmake)
defaults = cmake_defaults(cmake)
errors = []
compare(errors, 'CMakeLists.txt (pkg-config)', reference, cmake_pkgconfig_definitions(cmake))
compare(errors, 'meson.build', reference, meson_definitions(read(root, 'meson.build')))
options = meson_options(read(root, 'meson_options.txt'))
for option in reference:
if option not in defaults:
errors.append(f'CMakeLists.txt: no option(JSON_{option} ... ON|OFF)')
elif option not in options:
errors.append(f'meson_options.txt: option {option} missing')
elif options[option] != defaults[option]:
errors.append(f'meson_options.txt: option {option} must be boolean with value {defaults[option]} as in CMake')
for option, default in MESON_ONLY.items():
if options.get(option) != default:
errors.append(f'meson_options.txt: option {option} must be boolean with value {default}')
for option in options.keys() - reference.keys() - MESON_ONLY.keys():
errors.append(f'meson_options.txt: option {option} has no counterpart in the CMake target')
docs = read(root, DOCS)
for option in sorted(set(options) & (reference.keys() | MESON_ONLY.keys())):
if f'`{option}`' not in docs:
errors.append(f'{DOCS}: Meson option {option} not listed')
for error in errors:
print(error, file=sys.stderr)
if errors:
print('The Meson build and the pkg-config files must offer the options of the CMake target; see '
'tools/check_build_options/README.md.', file=sys.stderr)
return 1
print(f'OK: {len(reference)} options with compile definitions agree between CMake, pkg-config, and Meson.')
return 0
if __name__ == '__main__':
sys.exit(main())
@@ -36,5 +36,31 @@
</Expand>
</Type>
<!-- Fallback for when the basic_json entry above does not match: json_default_base is the (empty) default
base class of basic_json, and base class visualizers are inherited by derived types and evaluated
against the derived object, so m_data is accessible here. The class lives in {{ ns }} after
3.12.0 (#5238) and in {{ ns }}::detail up to 3.12.0, so both names are listed. -->
{% for default_base in ['json_default_base', 'detail::json_default_base'] %}
<Type Name="{{ ns }}::{{ default_base }}">
<DisplayString Condition="m_data.m_type == {{ ns }}::detail::value_t::null">null</DisplayString>
<DisplayString Condition="m_data.m_type == {{ ns }}::detail::value_t::object">{*(m_data.m_value.object)}</DisplayString>
<DisplayString Condition="m_data.m_type == {{ ns }}::detail::value_t::array">{*(m_data.m_value.array)}</DisplayString>
<DisplayString Condition="m_data.m_type == {{ ns }}::detail::value_t::string">{*(m_data.m_value.string)}</DisplayString>
<DisplayString Condition="m_data.m_type == {{ ns }}::detail::value_t::boolean">{m_data.m_value.boolean}</DisplayString>
<DisplayString Condition="m_data.m_type == {{ ns }}::detail::value_t::number_integer">{m_data.m_value.number_integer}</DisplayString>
<DisplayString Condition="m_data.m_type == {{ ns }}::detail::value_t::number_unsigned">{m_data.m_value.number_unsigned}</DisplayString>
<DisplayString Condition="m_data.m_type == {{ ns }}::detail::value_t::number_float">{m_data.m_value.number_float}</DisplayString>
<DisplayString Condition="m_data.m_type == {{ ns }}::detail::value_t::discarded">discarded</DisplayString>
<Expand>
<ExpandedItem Condition="m_data.m_type == {{ ns }}::detail::value_t::object">
*(m_data.m_value.object),view(simple)
</ExpandedItem>
<ExpandedItem Condition="m_data.m_type == {{ ns }}::detail::value_t::array">
*(m_data.m_value.array),view(simple)
</ExpandedItem>
</Expand>
</Type>
{% endfor %}
{% endfor %}
</AutoVisualizer>