A NUL byte anywhere in the input is currently treated the same as real
end of input, rather than raising parse_error.101 like any other
unexpected byte (documented in the previous commit's FAQ entry). A full
unconditional fix was tried in PR #5532 but rejected as too risky to
ship by default: any caller could depend on the current behavior, even
unknowingly (e.g. a zero-padded buffer). On PR #5534, gregmarr proposed
a compile-time opt-in flag instead, and the maintainer agreed, wanting
it available now and defaulting to the corrected behavior in 4.0.0.
This mirrors the existing JSON_BRACE_INIT_COPY_SEMANTICS precedent as
closely as sensible:
- JSON_STRICT_NUL_HANDLING defaults to 0 (off); the three lexer sites
that treat '\0' as EOF/comment-terminator are gated with
`#if !JSON_STRICT_NUL_HANDLING` so the default-off behavior is
byte-for-byte identical to today's.
- input_adapters.hpp's `T (&array)[N]` overload additionally trims a
single trailing '\0' from a `char` array (e.g. a string literal like
`json::parse("123")`) when the macro is on, so that case keeps
working; every other element type (unsigned char, std::uint8_t, ...)
always keeps its full extent. This intentionally does *not* reuse the
existing strlen()-based pointer overload via SFINAE-excluding `char`
from the array overload, as originally sketched for this change: that
approach is ambiguous against the newer generic container overload
added since PR #5532, and even where it compiles, strlen()-scanning a
`char` array that is not NUL-terminated within its bounds reads past
the end of the array (confirmed with AddressSanitizer). Trimming only
a single trailing byte, without scanning, avoids both problems.
- Documented via docs/mkdocs/docs/api/macros/json_strict_nul_handling.md,
linked from the macros index/nav/features page, the FAQ entry, and
the parse/accept/operator>> reference pages.
- Tested in unit-class_parser.cpp and unit-deserialization.cpp, default
state unguarded and opt-in state guarded. Since the library itself
#undefs the macro at the end of json.hpp (as JSON_BRACE_INIT_COPY_SEMANTICS
already does), a plain `#if defined(JSON_STRICT_NUL_HANDLING)` guard
after the include never actually triggers; the tests instead capture
the command-line value into a test-local macro before including the
header. A few pre-existing fixtures elsewhere (std::array<uint8_t, N>
sized one larger than their literal, relying on value-initialization
to silently add a trailing zero byte) needed the same one-byte
adjustment to keep passing under the opt-in behavior.
Unlike the precedent, this adds a proper `JSON_StrictNulHandling` CMake
option (rather than a raw -DCMAKE_CXX_FLAGS injection) and wires its
ci_test_strict_nul_handling target into the ci_cmake_options job matrix
in .github/workflows/ubuntu.yml, so the opt-in build is actually
exercised in CI -- closing the one gap in the precedent's own CI setup
(ci_test_brace_init_copy_semantics is defined but never referenced by
any workflow, so it has never actually run).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Migrate ci_icpc/ci_test_compilers_gcc_old/ci_infer off custom json-ci image
Replaces the last three consumers of ghcr.io/nlohmann/json-ci with official
images: ci_icpc now uses Intel's own intel/oneapi-hpckit:2023.2.1-devel-ubuntu22.04
(the last release with classic icc/icpc before Intel dropped it in oneAPI
2024.0), ci_test_compilers_gcc_old installs old GCCs on official ubuntu:20.04
via the same PPA/archive setup the custom image used (working around
actions/checkout's incompatibility with official gcc:4/5/6 images), and
ci_infer runs directly on ubuntu-latest, fetching Facebook's official Infer
release tarball inline instead of a maintained image. No job in
ubuntu.yml references the custom image anymore.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Update quality-assurance compiler table for CI image migration
Reflects the ci_icpc/ci_test_compilers_gcc_old container migration: the
old-GCC jobs (4.8/4.9/5/6) now compile inside official ubuntu:20.04 rather
than the custom Focal-based json-ci image (same OS, just now attributed to
the official image), and ci_icpc now uses Intel's official
intel/oneapi-hpckit:2023.2.1-devel-ubuntu22.04, bumping the reported ICC
version from 2021.5.0 to 2021.10.0 and the OS from Ubuntu 20.04.3 to 22.04.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix CI failures from the json-ci image migration
- ci_icpc: the official intel/oneapi-hpckit image has no CMake preinstalled
(the custom image bundled one); add the missing lukka/get-cmake step.
- ci_test_compilers_gcc_old: official ubuntu:20.04 has no build tool, so
CMake's default Unix Makefiles generator failed with "CMAKE_MAKE_PROGRAM
is not set"; install make alongside the PPA-provided g++.
- ci_infer: Infer v1.1.0's bundled Clang frontend can't parse GCC 14's
headers (ubuntu-latest's default toolchain), failing with parse errors in
<bits/unicode.h>; bump to the latest release, v1.3.0, whose newer bundled
frontend understands them (release asset also renamed upstream from
infer-linux64-v1.1.0.tar.xz to infer-linux-x86_64-v1.3.0.tar.xz).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix ci_test_compilers_gcc_old: g++-6 missing from xenial-only archives
My inline PPA/archive replication only added the xenial main/universe
suites, but g++-6 isn't available there ("has no installation candidate").
The original custom Dockerfile also pulled from bionic main/universe and
xenial-updates main/universe; add those back to match.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix ci_test_compilers_gcc_old: install git for CMake's FetchContent tests
Official ubuntu:20.04 ships no git at all (actions/checkout only succeeded
via its API-download fallback). The cmake_fetch_content(2) tests invoke
CMake's own ExternalProject_Add, which needs a real git binary and failed
with "could not find git for clone of json-populate". Install git alongside
the other build prerequisites.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix ci_icpc: drop redundant setvars.sh sourcing
Unlike the old custom image, the official intel/oneapi-hpckit image already
has the oneAPI environment (icc/icpc on PATH) baked in at the container
level. Explicitly re-sourcing setvars.sh in the Build step failed with
"setvars.sh has already been run. Skipping re-execution." (exit code 3,
aborting the step under `sh -e`). Drop the now-unnecessary sourcing.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix ci_icpc: exclude classic ICC from the std::span regression test
Bumping to Intel's official intel/oneapi-hpckit:2023.2.1 image (see previous
commit) also bumped classic icc/icpc from 2021.5.0 to 2021.10.0. The newer
version's __has_include(<span>) now returns true, but it still can't
actually compile std::span/std::as_bytes usage:
error: namespace "std" has no member "as_bytes"
error: namespace "std" has no member "span"
Exclude __ICC/__INTEL_COMPILER the same way _LIBCPP_VERSION is already
excluded for issue #4490.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix amalgamation/style check: indent comment per astyle
Verified with the pinned astyle 3.4.13 (make install_astyle) locally;
no further diff.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix ci_icpc: skip UTF-8 u8-literal comparison test on classic ICC
test-deserialization_cpp20 failed:
ERROR: CHECK( j2["emoji"] == "😀" ) is NOT correct!
check_utf8() only guards against MSVC's ANSI-codepage quirk (its docstring
example), but classic ICC has an analogous problem: it doesn't encode a
narrow string literal containing non-ASCII source characters as UTF-8,
so comparing a decoded u8R"(...)" literal against a narrow literal with
the same characters fails. Extend the existing guard.
Verified with the pinned astyle 3.4.13; no diff.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Add implementation to retrieve start and end positions of json during parse
* Add more unit tests and add start/stop parsing for arrays
* Add raw value for all types
* Add more tests and fix compiler warning
* Amalgamate
* Fix CLang GCC warnings
* Fix error in build
* Style using astyle 3.1
* Fix whitespace changes
* revert
* more whitespace reverts
* Address PR comments
* Fix failing issues
* More whitespace reverts
* Address remaining PR comments
* Address comments
* Switch to using custom base class instead of default basic_json
* Adding a basic using for a json using the new base class. Also address PR comments and fix CI failures
* Address decltype comments
* Diagnostic positions macro (#4)
Co-authored-by: Sush Shringarputale <sushring@linux.microsoft.com>
* Fix missed include deletion
* Add docs and address other PR comments (#5)
* Add docs and address other PR comments
---------
Co-authored-by: Sush Shringarputale <sushring@linux.microsoft.com>
* Address new PR comments and fix CI tests for documentation
* Update documentation based on feedback (#6)
---------
Co-authored-by: Sush Shringarputale <sushring@linux.microsoft.com>
* Address std::size_t and other comments
* Fix new CI issues
* Fix lcov
* Improve lcov case with update to handle_diagnostic_positions call for discarded values
* Fix indentation of LCOV_EXCL_STOP comments
* fix amalgamation astyle issue
---------
Co-authored-by: Sush Shringarputale <sushring@linux.microsoft.com>
* Possible fix for #4485
Throw's an exception when i is nullptr,
also added a testcase for this scenario though most likely in the wrong test file.cpp
* quick cleanup
* Fix compile issues
* moved tests around, changed exceptions, removed a possibly unneeded include
* add back include <memory> for testing something
* Ninja doesn't like not having a \n, at end of file, adding it back
* update input_adapter file to deal with empty/null file ptr.
* ran make pretty
* added test for inputadapter
* ran make amalgamate
* Update tests/src/unit-deserialization.cpp
Co-authored-by: Niels Lohmann <niels.lohmann@gmail.com>
* Update tests/src/unit-deserialization.cpp
Co-authored-by: Niels Lohmann <niels.lohmann@gmail.com>
* Update input adapters.hpp with new includes
* fix unabigious use of _, (there was a double declare)
* did the amalagamate
* rm duplicate includes
* make amalgamate again
* reorder
* amalgamate
* moved it above
* amalgamate
---------
Co-authored-by: Jordan <jordan-hoang@users.noreply.github.com>
Co-authored-by: Niels Lohmann <niels.lohmann@gmail.com>
* Move UDLs into nlohmann::literals::json_literals namespace
* Add 'using namespace' to unit tests
* Add 'using namespace' to examples
* Add 'using namespace' to README
* Move UDL mkdocs pages out of basic_json/
* Update documentation
* Update docset index
* Add JSON_GlobalUDLs CMake option
* Add unit test
* Build examples without global UDLs
* Add CI target
When parsing a string containing two JSON values using iterators, after
parsing, iterator_input_adapter should have advanced to the first
character of the second value.
Add a unit test to check that this is true.