mirror of
https://github.com/nlohmann/json.git
synced 2026-09-27 18:20:32 +00:00
Niels Lohmann
1e101ecac1
* Add BON8 support Add to_bon8/from_bon8 and input_format_t::bon8 for BON8, a binary format that uses the byte values that cannot begin a UTF-8 character as type markers, so strings need no length prefix. It is the most compact of the supported binary formats on the benchmark files. The reader is non-recursive like the other binary readers. A string ends at the first byte that cannot continue it, so the reader hands the one or two bytes it reads past a string back to the value that follows. The writer produces the canonical representation of the specification, except for NFC normalization; its output is identical to that of the reference implementation (HikoGUI) on all files of the test data. The round-trip tests need the .bon8 files of json_test_data 3.2.0. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address review comments - Reuse detail::validate_one_utf8 to check strings in to_bon8; the error now names the first byte of the invalid sequence. - Document that to_bon8 leaves bytes in the output adapter on an exception, and that string_open is only an output of write_bon8_marker. - Explain why the pushback buffer of the BON8 reader cannot overflow. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Select the BON8 float prefix by type get_bon8_float_prefix only depends on the type of its argument, so make the type a template parameter instead of passing an unused value. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Rename a test variable that Flawfinder mistakes for read() Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix the BON8 CI failures - compare the float in write_bon8_float with number_float_t constants, so GCC does not warn about a float-to-double conversion - mark check_bon8_utf8's context as used when exceptions are disabled - choose the compact float prefix in a helper rather than with nested conditional operators (clang-tidy) - use auto for the cast in the BON8 integer reader (clang-tidy) - write the int32 minimum test values as long long literals (MSVC C4146) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Amalgamate Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Read BON8 strings in bulk from contiguous input - copy the valid UTF-8 of a string in one step when the input is contiguous (twitter.json is read in 1.68 instead of 2.52 ms, jeopardy.json in 196 instead of 297 ms, close to CBOR and MessagePack) - share the new valid_utf8_prefix() with the writer's UTF-8 check, which now skips ASCII 8 bytes at a time - let the fuzzer check that contiguous and stream input give the same value or error, and test both paths in the unit tests - clarify that a second 0xFF after a string is an empty string Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Link the BON8 functions from the other binary format pages Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Name the bulk scan flag after the input, not BON8 Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Read BSON keys in bulk from contiguous input BSON keys (and array indices) are C-style strings, which were read byte by byte. For contiguous input they are now read up to their \x00-byte in one step, using the same bulk_scan flag as BON8 strings: twitter.json is read in 1.46 instead of 2.01 ms, citm_catalog.json in 2.93 instead of 3.33 ms, jeopardy.json in 182 instead of 207 ms. canada.json, whose keys are almost all one-digit array indices, takes 2 % longer. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix the BON8 CI failures of the bulk-read tests - skip the contiguous-versus-stream tests of BON8 strings and BSON keys when exceptions are disabled: they catch the parse errors of invalid input, and without exceptions the library aborts instead - use static_cast for the int64 test value (google-readability-casting) Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Move the explicit basic_json instantiation into its own test file Linking test-regression3_cpp20 with clang and MinGW failed with "relocation truncated to fit: IMAGE_REL_AMD64_REL32 against `.rdata'", as test-regression2 did before #5511. The explicit instantiation of basic_json<> for #4825 compiles every member function, including the BON8 reader and writer, into that object, and it was already close to the limit (2,226,104 bytes on develop, 2,234,960 with BON8; clang -O1, C++20). Give the instantiation a file of its own: unit-regression3 is now 1,594,736 bytes and unit-explicit_instantiation 1,095,064. The new file mentions JSON_HAS_CPP_17 and JSON_HAS_CPP_20 so it keeps being built for the C++17 standard the regression was about. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Convert the bytes of the BON8 test strings explicitly The str() helper constructed a std::string from a byte range, which converts each unsigned char implicitly; -fsanitize=integer reports that for bytes of 0x80 and above (ci_test_clang_sanitizer). Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>
92 lines
4.6 KiB
CMake
92 lines
4.6 KiB
CMake
set(JSON_TEST_DATA_URL https://github.com/nlohmann/json_test_data)
|
|
set(JSON_TEST_DATA_VERSION 3.2.0)
|
|
|
|
include(ExternalProject)
|
|
|
|
# if variable is set, use test data from given directory rather than downloading them
|
|
if(JSON_TestDataDirectory)
|
|
message(STATUS "Using test data in ${JSON_TestDataDirectory}.")
|
|
add_custom_target(download_test_data)
|
|
file(WRITE ${CMAKE_BINARY_DIR}/include/test_data.hpp "#define TEST_DATA_DIRECTORY \"${JSON_TestDataDirectory}\"\n")
|
|
else()
|
|
# create a header with the path to the downloaded test data
|
|
file(WRITE ${CMAKE_BINARY_DIR}/include/test_data.hpp "#define TEST_DATA_DIRECTORY \"${CMAKE_BINARY_DIR}/test_files\"\n")
|
|
|
|
# download test data from the GitHub tag source archive
|
|
ExternalProject_Add(download_test_data_project
|
|
URL "${JSON_TEST_DATA_URL}/archive/refs/tags/v${JSON_TEST_DATA_VERSION}.zip"
|
|
SOURCE_DIR "${CMAKE_BINARY_DIR}/test_files"
|
|
CONFIGURE_COMMAND ""
|
|
BUILD_COMMAND ""
|
|
INSTALL_COMMAND ""
|
|
LOG_DOWNLOAD TRUE
|
|
DOWNLOAD_EXTRACT_TIMESTAMP TRUE
|
|
EXCLUDE_FROM_ALL TRUE
|
|
)
|
|
|
|
# target to download test data
|
|
add_custom_target(download_test_data
|
|
DEPENDS download_test_data_project
|
|
)
|
|
endif()
|
|
|
|
# determine the operating system (for debug and support purposes)
|
|
find_program(UNAME_COMMAND uname)
|
|
find_program(LSB_RELEASE_COMMAND lsb_release)
|
|
find_program(SW_VERS_COMMAND sw_vers)
|
|
set(OS_VERSION_STRINGS "${CMAKE_SYSTEM}")
|
|
if (CMAKE_HOST_WIN32)
|
|
# "ver" is a cmd.exe builtin rather than a standalone executable, so it
|
|
# cannot be located with find_program and must be invoked through cmd
|
|
execute_process(COMMAND cmd /c ver OUTPUT_VARIABLE VER_COMMAND_RESULT ERROR_QUIET)
|
|
string(STRIP "${VER_COMMAND_RESULT}" VER_COMMAND_RESULT)
|
|
if (VER_COMMAND_RESULT)
|
|
set(OS_VERSION_STRINGS "${OS_VERSION_STRINGS}; ${VER_COMMAND_RESULT}")
|
|
endif()
|
|
endif()
|
|
if (SW_VERS_COMMAND)
|
|
execute_process(COMMAND ${SW_VERS_COMMAND} OUTPUT_VARIABLE SW_VERS_COMMAND_RESULT OUTPUT_STRIP_TRAILING_WHITESPACE ERROR_QUIET)
|
|
string(REGEX REPLACE "[ ]*\n" "; " SW_VERS_COMMAND_RESULT "${SW_VERS_COMMAND_RESULT}")
|
|
set(OS_VERSION_STRINGS "${OS_VERSION_STRINGS}; ${SW_VERS_COMMAND_RESULT}")
|
|
endif()
|
|
if (LSB_RELEASE_COMMAND)
|
|
execute_process(COMMAND ${LSB_RELEASE_COMMAND} -a OUTPUT_VARIABLE LSB_RELEASE_COMMAND_RESULT OUTPUT_STRIP_TRAILING_WHITESPACE ERROR_QUIET)
|
|
string(REGEX REPLACE "[ ]*\n" "; " LSB_RELEASE_COMMAND_RESULT "${LSB_RELEASE_COMMAND_RESULT}")
|
|
set(OS_VERSION_STRINGS "${OS_VERSION_STRINGS}; ${LSB_RELEASE_COMMAND_RESULT}")
|
|
endif()
|
|
if (UNAME_COMMAND)
|
|
execute_process(COMMAND ${UNAME_COMMAND} -a OUTPUT_VARIABLE UNAME_COMMAND_RESULT OUTPUT_STRIP_TRAILING_WHITESPACE ERROR_QUIET)
|
|
set(OS_VERSION_STRINGS "${OS_VERSION_STRINGS}; ${UNAME_COMMAND_RESULT}")
|
|
endif()
|
|
|
|
message(STATUS "Operating system: ${OS_VERSION_STRINGS}")
|
|
|
|
# determine the compiler (for debug and support purposes)
|
|
if (MSVC)
|
|
execute_process(COMMAND ${CMAKE_CXX_COMPILER} OUTPUT_VARIABLE CXX_VERSION_RESULT OUTPUT_STRIP_TRAILING_WHITESPACE ERROR_VARIABLE CXX_VERSION_RESULT ERROR_STRIP_TRAILING_WHITESPACE)
|
|
set(CXX_VERSION_RESULT "${CXX_VERSION_RESULT}; MSVC_VERSION=${MSVC_VERSION}; MSVC_TOOLSET_VERSION=${MSVC_TOOLSET_VERSION}")
|
|
else()
|
|
execute_process(COMMAND ${CMAKE_CXX_COMPILER} --version OUTPUT_VARIABLE CXX_VERSION_RESULT OUTPUT_STRIP_TRAILING_WHITESPACE)
|
|
endif()
|
|
string(REGEX REPLACE "[ ]*\n" "; " CXX_VERSION_RESULT "${CXX_VERSION_RESULT}")
|
|
message(STATUS "Compiler: ${CXX_VERSION_RESULT}")
|
|
|
|
# determine used C++ standard library (for debug and support purposes)
|
|
if(CMAKE_CROSSCOMPILING)
|
|
set(LIBCPP_VERSION_OUTPUT_CACHED "could not be detected due to cross-compiling" CACHE STRING "Detected C++ standard library version")
|
|
endif()
|
|
if(NOT DEFINED LIBCPP_VERSION_OUTPUT_CACHED)
|
|
try_run(RUN_RESULT_VAR COMPILE_RESULT_VAR
|
|
"${CMAKE_BINARY_DIR}" SOURCES "${CMAKE_CURRENT_LIST_DIR}/detect_libcpp_version.cpp"
|
|
RUN_OUTPUT_VARIABLE LIBCPP_VERSION_OUTPUT
|
|
COMPILE_OUTPUT_VARIABLE LIBCPP_VERSION_COMPILE_OUTPUT
|
|
)
|
|
if(NOT LIBCPP_VERSION_OUTPUT)
|
|
set(LIBCPP_VERSION_OUTPUT "Unknown")
|
|
message(AUTHOR_WARNING "Failed to compile cmake/detect_libcpp_version to detect the used C++ standard library. This does not affect the library or the test cases. Please still create an issue at https://github.com/nlohmann/json to investigate this.\n${LIBCPP_VERSION_COMPILE_OUTPUT}")
|
|
endif()
|
|
set(LIBCPP_VERSION_OUTPUT_CACHED "${LIBCPP_VERSION_OUTPUT}" CACHE STRING "Detected C++ standard library version")
|
|
endif()
|
|
|
|
message(STATUS "C++ standard library: ${LIBCPP_VERSION_OUTPUT_CACHED}")
|