Several members were required only because of how the library happened to
be written, not because the functionality needs them. Dropping them widens
the set of usable string and array types, and one of them was also a
performance problem.
string_t:
- c_str() is gone. Every call site already knew the length and passed it
along, so data() is enough. The one place that did not, the diagnostics
path in exceptions.hpp, now builds the token from data() and size(),
which also stops it from truncating keys that contain a null byte.
- back() is gone; the serializer indexes the last character instead.
- find(str, pos), replace(), and substr() are gone. escape() and
unescape() rebuilt the string with one replace() per escaped character,
which moves the tail every time: escaping a string of n characters that
all need escaping cost O(n^2). Both now scan with find_first_of() -- a
member the pointer parser already required -- and append whole runs, so
the common case is one search and one copy. Escaping 64000 tildes drops
from 717 ms to 20 ms; a string with nothing to escape gets faster too
(8.4 ms to 5.8 ms), because the scan is still a single memchr per pass.
json_pointer::split() takes its reference tokens with the
(const char*, size_type) constructor rather than substr().
- json_pointer::to_string() accumulates with concat<string_t> instead of
letting concat default to std::string and converting afterwards, so
streaming a json_pointer no longer requires string_t to be assignable
from a std::string.
array_t:
- at(size_type) is gone. basic_json::at(size_type) checked the index by
calling array_t::at() and translating std::out_of_range, which also
required the array type to throw that exact exception. It now compares
against size() and uses operator[]. The thrown exception, its message,
and the behaviour under JSON_NOEXCEPTION are unchanged.
The BSON writer wrote the terminating null byte out of the string's own
buffer (size() + 1). It now writes the byte itself, so string_t::data()
need not be null-terminated for to_bson().
The tests pin the reduced API: alt_string loses the five dropped members
and gains coverage of the escaping paths, and a std::vector whose at() is
hidden is used as an ArrayType.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Three places built a std::string and handed it to something expecting a
string_t: the UBJSON high-precision number reader, which every binary reader
instantiates, and the BSON writer's array element size calculation and write.
That silently required string_t to be implicitly convertible from std::string,
which std::string itself and types with a string_view conversion satisfy, but
many string types do not.
Construct the string_t explicitly from the data and size, which the
requirements already cover. This makes boost::container::string, eastl::string,
std::pmr::string, and std::basic_string with a custom allocator work as
StringType, none of which could previously be used with any binary format.
Add binary format coverage to the alt_string test, which had none, including a
UBJSON high-precision number -- the case that goes through the reader path.
BSON stays uncovered there: it additionally needs string_t::find(value_type),
which alt_string does not provide.
Also record which containers from Boost, Abseil, and EASTL work for each
template parameter, and correct two claims: std::pmr::string is usable after
this change, and tsl::ordered_map is not usable at all, because its iterators
expose the mapped value as const while basic_json modifies it in place.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
unflatten() decided between array and object by looking at the first reference
token it happened to see for a node: it started an array only when that token
was 0. With a sorted object type the token 0 always arrives first, so the
result was correct by accident; with an object type whose iteration order is
unspecified, {"/c/2":3,"/c/1":2,"/c/0":1} unflattened to an object with the
keys "0", "1", and "2" instead of an array.
Collect the pointer prefixes that have a reference token 0 among their children
before building the result, and let get_and_create() consult that set. The
outcome is now independent of the iteration order and matches, for every input,
what a sorted object type produced before: a value is restored as an array if
and only if one of its keys is 0. Iterating the flattened object in a different
order would have been simpler, but it would have changed the key order of the
result for insertion-ordered object types.
The serializer, std::hash, and the UBJSON writer converted the elements of a
binary value to an integer implicitly, which does not compile for a BinaryType
whose value type is std::byte, and which made dump() write the bytes of a
signed value type as negative numbers. Convert to std::uint8_t explicitly in
all three places, so every byte type dumps as 0..255. The default
std::vector<std::uint8_t> configuration is unaffected.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018hxZxz8svM54c6ATEvXp5E
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Throw other_error.502 when UBJSON use_type is set without use_size
Fixes#5321
Signed-off-by: Krishnanand G <118352827+Krishnanand-G@users.noreply.github.com>
* Scope UBJSON use_type check to container branches and expand tests
Signed-off-by: Krishnanand G <118352827+Krishnanand-G@users.noreply.github.com>
* Re-amalgamate single_include/json.hpp
The previous commit updated the split headers but the amalgamated
file didn't go back through astyle before I committed it, so CI's
amalgamation check caught formatting drift in json_fwd.hpp and a
few noexcept clauses in basic_json, plus one doc example. None of
it touches the UBJSON logic. Applied the patch CI generated to
bring single_include back in sync.
Signed-off-by: Krishnanand G <118352827+Krishnanand-G@users.noreply.github.com>
---------
Signed-off-by: Krishnanand G <118352827+Krishnanand-G@users.noreply.github.com>
* Do not write BJData ndarrays whose size overflows std::size_t
write_bjdata_ndarray() multiplied the _ArraySize_ dimensions into a
std::size_t without checking for overflow. A product that wraps around
to a value that happens to match the size of _ArrayData_ passed the
length check, and the writer emitted an ndarray header announcing an
element count that cannot be represented:
{"_ArrayType_":"uint8","_ArraySize_":[9223372036854775808,2],"_ArrayData_":[]}
was encoded as 5b 24 55 23 5b 4d 00 00 00 00 00 00 00 80 69 02 5d, an
ndarray of 2^64 elements followed by no data. Reading that back throws
out_of_range.408 ("excessive ndarray size caused overflow"), so to_bjdata
produced output that from_bjdata rejects. This is reachable by parsing
untrusted JSON and re-encoding it as BJData.
Mirror the overflow check the binary reader already performs, and also
reject a single dimension that does not fit into std::size_t, which the
previous cast silently truncated where std::size_t is narrower than 64
bits. Such objects now fall back to a plain object encoding, which is
what the surrounding type and length validation already does for
annotations it cannot represent, and they round-trip unchanged.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Document when to_bjdata converts a JData annotation to an ND-array
The BJData page described the 1-D vector case as the only situation in
which an object carrying _ArrayType_/_ArraySize_/_ArrayData_ is not
written as a compact ND-array. The writer has always had several other
fallbacks -- an unknown _ArrayType_, a dimension that is not a
non-negative integer, an _ArrayData_ whose length does not match the
product of the dimensions, and elements that are not numbers of the
annotated kind -- all of which cause the value to be serialized as a
regular JSON object instead.
Spell out the conditions, including the size-overflow check added in the
preceding commit, so the documented behavior matches the implementation.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* validate ndarray element types in write_bjdata_ndarray
Signed-off-by: Angadi Yashaswini <angadi@digiscrypt.com>
* read ndarray elements through get<> instead of a fixed union member
_ArrayType_ names the wire type, not how the value is stored: parsing
keeps a non-negative integer as number_unsigned while the C++ API keeps
an int literal as number_integer. Selecting the union member from the
type marker therefore reads the inactive alternative for one of the two,
so read through get<> instead, which dispatches on the active member.
Also reject a negative _ArraySize_ entry, which is not a usable
dimension, and cover the parse-built path in the tests.
Signed-off-by: Angadi Yashaswini <angadi@digiscrypt.com>
---------
Signed-off-by: Angadi Yashaswini <angadi@digiscrypt.com>
* Replace snprintf with a branch-free writer for \uXXXX escapes
dump_escaped called std::snprintf(..., "\u%04x", ...) once per escaped
code point in the string serialization hot path. snprintf re-parses
the format string and pulls in locale/printf machinery on every call,
which is far heavier than the fixed 6-/12-byte output warrants. This
is hot for any string containing control characters, and for all
non-ASCII text when ensure_ascii is set.
Replace it with write_u_escape, a small helper that writes the escape
directly into string_buffer via a nibble-to-hex lookup table, mirroring
the existing hand-rolled dump_integer fast path in the same file.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix clang-tidy avoid-c-arrays warning in write_u_escape
Use a const char* rather than a char[] lookup table, matching the
existing hex_bytes helper in the same file.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* ♻️ adjust write_u_escape signature
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Add versioned inline namespace
Add a versioned inline namespace to prevent ABI issues when linking code
using multiple library versions.
* Add namespace macros
* Encode ABI information in inline namespace
Add _diag suffix to inline namespace if JSON_DIAGNOSTICS is enabled, and
_ldvcmp suffix if JSON_USE_LEGACY_DISCARDED_VALUE_COMPARISON is enabled.
* Move ABI-affecting macros into abi_macros.hpp
* Move std_fs namespace definition into std_fs.hpp
* Remove std_fs namespace from unit test
* Format more files in tests directory
* Add unit tests
* Update documentation
* Fix GDB pretty printer
* fixup! Add namespace macros
* Derive ABI prefix from NLOHMANN_JSON_VERSION_*
* Add benchmark for cbor serialization and cbor binary data serialization
* Use std::vector::insert method instead of std::copy in output_vector_adapter::write_characters
This change increases a lot the performance when writing lots of binary data.
* support UBJSON-derived Binary JData (BJData) format
* fix Codacy warning
* partially fix VS compilation errors
* fix additional VS errors
* fix more VS compilation errors
* fix additional warnings and errors for clang and msvc
* add more tests to cover the new bjdata types
* add tests for optimized ndarray, improve coverage, fix clang/gcc warnings
* gcc warn useless conversion but msvc gives an error
* fix ci_test errors
* complete test coverage, fix ci_test errors
* add half precision error test
* fix No newline at end of file error by clang
* simplify endian condition, format unit-bjdata
* remove broken test due to alloc limit
* full coverage, I hope
* move bjdata new markers from default to the same level as ubjson markers
* fix ci errors, add tests for new bjdata switch structure
* make is_bjdata const after using initializer list
* remove the unwanted assert
* move is_bjdata to an optional param to write_ubjson
* pass use_bjdata via output adapter
* revert order to avoid msvc 2015 unreferenced formal param error
* update BJData Spect V1 Draft-2 URL after spec release
* amalgamate code
* code polishing following @gregmarr's feedback
* make use_bjdata a non-default parameter
* fix ci error, remove unwanted param comment
* encode and decode bjdata ndarray in jdata annotations, enable roundtrip tests
* partially fix ci errors, add tests to improve coverage
* polish patch to remove ci errors
* fix a ndarray dim vector condition
* fix clang tidy error
* add sax test cases for ndarray
* add additional sax event tests
* adjust sax event numbering
* fix sax tests
* ndarray can only be used with array containers, discard if used in object
* complete test coverage
* disable [{SHTFNZ in optimized type due to security risks in #2793 and hampered readability
* fix ci error
* move OutputIsLittleEndian from tparam to param to replace use_bjdata
* fix ci clang gcc error
* fix ci static analysis error
* update json_test_data to 3.1.0, enable file-based bjdata unit tests
* fix stack overflow error on msvc 2019 and 2022
* use https link, update sax_parse_error after rebase
* make input_format const and use initializer
* return bool for write_bjdata_ndarray
* test write_bjdata_ndarray return value as boolean
* fix ci error
* Make exception context optional
Change exception context parameter to pointer and replace context with
nullptr where appropriate.
* Support escaping other string types
* Add string concatenation function
Add variadic concat() function for concatenating char *, char, and
string types.
* Replace string concatenations using + with concat()
* Template json_pointer on string type
Change json_pointer from being templated on basic_json to being
templated on string type.
* Add unit test for #3388Closes#3388.
* Fix regression test for #2958
* Add backwards compatibility with json_pointer<basic_json>
* Update json_pointer docs
* Allow comparing different json_pointers
* Update version numbers
Niels Lohmann
Krishnanand G
Angadi56
Luke Banicevic
Kirill Lokotkov
Charles Cabergs
NmPassTHFan
Michael Valladolid
Nebojša Cvetković
Raphael Grimmandbarcode
Romain Reignier
Qianqian Fang
Niels Lohmann
Alex Beregszaszi
Pierre Hallot