mirror of
https://github.com/nlohmann/json.git
synced 2026-09-25 01:10:30 +00:00
1054b2097e721e8a455285a1b528cc72caf859b7
142
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  Niels LohmannandClaude
|
1054b2097e |
Speed up binary writing: value-type output sink + byte-swap number encoding (#5286)
* Devirtualize binary_writer via a value-type output sink to_cbor/to_msgpack/to_ubjson/to_bjdata/to_bson wrote every byte through output_adapter_t, a shared_ptr<output_adapter_protocol> whose write_character/write_characters are virtual. Unlike the lexer (templated on a concrete InputAdapterType), the binary writer never got that treatment, so binary output paid a vtable lookup per byte and a make_shared per call. Template binary_writer on an OutputSinkType and give it two concrete, non-virtual sinks: - output_vector_sink: appends straight into a std::vector (push_back / insert), used by the vector-returning to_* convenience functions. No vtable, no shared_ptr; the writes inline. - output_adapter_sink: forwards to a type-erased output_adapter_t, so the existing to_*(j, output_adapter) overloads (streams, strings, custom adapters) keep working exactly as before -- one virtual call each, unchanged. binary_writer keeps a convenience constructor taking output_adapter_t (building the default output_adapter_sink), so the adapter overloads are untouched; only the convenience functions switch to the vector sink. The friend declaration and the basic_json binary_writer alias gain the new (defaulted) template parameter. Output is byte-for-byte identical: verified across ~3000 randomized values plus curated edge cases (all scalar widths, strings with invalid UTF-8, binary, nested arrays/objects) for CBOR, MessagePack, UBJSON (both size/type settings), BJData, and BSON, plus the output_adapter path, in C++11/17/20. Warning-clean under clang -Weverything and the gcc pedantic set; clang-tidy clean on the changed headers; make check-amalgamation clean. Throughput (g++ -O3, vs develop): scalar-dense binary output such as integer arrays ~1.4x; many small to_cbor calls ~1.04x (DOM traversal bound); string/blob-heavy output unchanged (already bulk-bound). No workload regressed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix CI failures from binary_writer output-sink change Four CI jobs failed on the initial commit; all are addressed here without changing any output (binary encodings remain byte-for-byte identical to develop across the differential corpus): 1. ci_test_gcc / cuda (-Werror=duplicated-branches): for number_float_t == float, static_cast<float>(n) is the identity, so write_compact_float's two branches are intentionally identical. Once the concrete vector sink is inlined, GCC constant-folds and diagnoses this (the type-erased path hid it behind a non-inlined virtual call). Silence -Wduplicated-branches for GCC (clang has no such warning) alongside the existing -Wfloat-equal pragma. 2. ci_static_analysis_clang (UBSan nonnull-attribute): binary_writer passes a null pointer with length 0 for empty strings/binary. output_vector_sink / output_adapter_sink declared write_characters JSON_HEDLEY_NON_NULL, so the sanitizer flagged the (harmless) zero-length call once the sink was called directly rather than through the attribute-free virtual base. Drop the attribute from both sinks, matching the pre-existing behavior. 3. ci_cpplint (build/include_what_you_use): output_adapter_sink uses std::move; add #include <utility>. 4. ci_cuda_example (nvcc 11.8): NVCC's front end rejects the default template argument on the binary_writer alias template. Revert the alias to its original single-parameter form (relying on binary_writer's own defaulted OutputSinkType) and spell out the full type in the vector-sink convenience functions. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Encode big-endian numbers with a byte swap instead of std::reverse write_number() reordered multi-byte numbers for the big-endian formats (CBOR/MessagePack/UBJSON) with std::reverse over the byte array. GCC lowered only some sizes to a bswap; clang kept a scalar byte shuffle (0 bswap instructions in the CBOR number path). Replace the reverse with size-dispatched __builtin_bswap16/32/64 helpers (portable shift fallback for other compilers; std::reverse retained for exotic sizes such as a long double number_float_t). Codegen: the CBOR number path now emits bswap on both compilers (gcc 2 -> 16, clang 0 -> 4). Output is byte-for-byte identical to the previous implementation across the binary differential corpus. Throughput (isolated vs the std::reverse version, best of 9): CBOR int64 array gcc +7% clang +10% CBOR uint16 array gcc +27% clang flat Modest but consistent on number-dense encodings; negligible on string/blob-heavy output, as expected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Reserve output capacity up front for binary serialization The vector-returning to_cbor/to_msgpack/to_ubjson/to_bjdata/to_bson grew the output buffer purely by geometric reallocation. Reserving an estimate up front avoids the early reallocations, which is the dominant per-byte cost for array/object-heavy output. The estimate (binary_reserve_hint) is deliberately conservative and safe against untrusted input: it consults only the top-level element count (O(1), no walk of the DOM), guards the multiplication against overflow, and clamps the result to a fixed 1 MiB ceiling, so a large or hostile DOM can never force an oversized allocation here. The buffer still grows geometrically past the hint, so an underestimate only costs a few later reallocations; scalars/strings/binary are written in one shot and get no hint. Reserving capacity does not change the bytes produced. Throughput (g++/clang -O3, vs the previous commit): cbor int array +10% / +13% cbor object array +20% / +38% Output is byte-for-byte identical to develop across the binary differential corpus. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address review findings on the binary writer output sinks - binary_reserve_hint(): the 4-bytes-per-element estimate over-reserved by up to 4x for arrays of small scalars (CBOR encodes 0..23 in one byte), and the returned vector kept that capacity. Make the hint a strict lower bound on the encoded size instead, which also removes the 1 MiB clamp whose branch no test could reach (the largest container in the suite has 65793 elements). - Guard the -Wduplicated-branches pragma with __GNUC__ >= 7. The warning does not exist before GCC 7, so naming it made GCC 4.8/4.9/5/6 - which the CI matrix still builds - warn under -Wpragmas on every including translation unit, breaking downstream -Werror builds. - Constrain the adapter constructor of binary_writer with the enable_if its documentation already claimed, so a writer over some other sink type is no longer advertised as constructible from an output adapter. - Let output_vector_adapter wrap output_vector_sink rather than duplicating the append logic, so the type-erased and templated paths share one implementation. - Collapse the three copies of the memcpy/byte_swap/memcpy dance into a single byte_swap_buffer() helper, and add the MSVC _byteswap_* intrinsics so MSVC no longer falls back to the scalar shuffle this change exists to eliminate. - Add a vector_writer() helper for the five vector-returning to_* overloads instead of spelling out the writer type at each call site, and drop a dead default member initializer on output_adapter_sink. - New tests: the vector sink and the adapter sink must produce identical bytes for every format (the two to_* overloads no longer delegate to each other and could otherwise drift), and binary_reserve_hint() must never exceed the size actually written. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Route the -Wduplicated-branches pragma through Hedley Match #5485, which moved the binary writer's hand-rolled diagnostic pragmas onto JSON_HEDLEY_PRAGMA (merged into develop while this branch was open). The devirtualization's -Wduplicated-branches suppression in write_compact_float was the one raw '#pragma GCC diagnostic' left; it now uses JSON_HEDLEY_PRAGMA like the adjacent -Wfloat-equal line, still guarded to GCC >= 7 and non-clang (the warning exists only there). Co-Authored-By: Claude <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XAYM1qhSA2FDaDcGfPW3fG Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Niels Lohmann
|
0b20b7e622 |
Reject MessagePack/BSON binary subtypes that don't fit their wire format (#5469)
* Reject MessagePack/BSON binary subtypes that don't fit their wire format Both formats store byte_container_with_subtype's subtype (a uint64_t) in a single byte. The writers cast to std::int8_t/std::uint8_t without a range check, so subtypes above 255 were silently truncated modulo 256 instead of raising an error. Throw out_of_range.413 instead when the subtype exceeds the representable range of 0-255. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Move the new binary-subtype regression test out of unit-regression2.cpp unit-regression2.cpp is already at the edge of what the MinGW linker can relocate; adding this test's ~26 lines tips test-regression2_cpp20 (clang, Windows) over into "relocation truncated to fit: IMAGE_REL_AMD64_REL32 against `.rdata'" (see |
||
|
Niels Lohmann
|
c41152e620 |
Fall back to plain-object encoding when to_bjdata()'s _ArrayType_ annotation is not a string (#5494)
* Fall back to plain-object encoding when _ArrayType_ is not a string write_bjdata_ndarray() looked up _ArrayType_ by calling get<string_t>() directly, which throws type_error.302 when the annotation is not a string (e.g. a number, null, boolean, array, or object). Per the documented BJData ndarray contract, an object only qualifies for the compact ndarray encoding if _ArrayType_ names a known type; anything else must fall back to plain-object encoding, the same way an unknown type-name string already does. Add an is_string() check before the get<string_t>() call so a non-string _ArrayType_ takes the existing "unrecognized type name" fallback path instead of throwing. Fixes #5398. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Relax the BJData fuzzer's round-trip check from byte-exact to value-exact Fixing #5398 lets to_bjdata() proceed past the object it used to reject, which exposed a pre-existing, unrelated round-trip quirk to the fuzzer: a binary_t value serialized through the non-optimized ("$U#"-less) array encoding is parsed back as a plain array of numbers, since from_bjdata() has no way to tell "array of uint8 numbers" apart from "array of bytes" without that optimized header. Re-serializing that plain array then goes through the generic smallest-type writer, which - unrelated to this PR, and long predating it - prefers the 'i' (int8) marker over 'U' (uint8) for values that fit both, so the re-encoded bytes can differ from the original even though both decode to the same value. This is not introduced by the #5398 fix; the same divergence reproduces from a bare json::binary_t value with no _ArrayType_ annotation involved at all, on the commit immediately preceding it. A general fix would mean changing the shared UBJSON/BJData smallest-type selection that hundreds of existing tests pin to 'i' for small positive integers, which is out of scope and too risky for this PR. Update fuzzer-parse_bjdata.cpp's round-trip assertions to check that re-serializing is value-stable (from_bjdata(to_bjdata(j)) == j) rather than byte-exact, matching the guarantee BJData actually provides, and add a regression test in unit-bjdata.cpp using the exact OSS-Fuzz input that documents the behavior. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Compare dump()s instead of json values in the BJData fuzzer's round-trip check The value-stability assertion added to fix the earlier OSS-Fuzz crash (json::from_bjdata(to_bjdata(j2)) == j2) itself broke on a NaN payload: IEEE 754 NaN is never equal to itself, so operator== reports two structurally-identical trees containing a non-finite double as different -- not a round-trip bug, just NaN's ordinary non-reflexivity. dump() serializes any non-finite double the same deterministic way (as JSON null, since JSON cannot represent NaN or Infinity), so comparing dumps is stable under exactly the values that break operator==. Verified against both the original OSS-Fuzz crash input and the new one (0x68 0x68 0x7c, which decodes to a NaN), plus a local 2.5M-case random-input sweep with no failures. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
502e9d66f6 |
Fix to_bjdata() emitting the Draft-3-only 'B' marker in default Draft-2 mode (#5479)
* Fix to_bjdata() emitting the Draft-3-only 'B' marker in default Draft-2 mode _ArrayType_ = "byte" mapped unconditionally to the BJData type marker 'B', regardless of the requested bjdata_version. 'B' is defined only by BJData Draft 3; with the default version (draft2), this produced a stream that is invalid for Draft 2 and, unlike every other _ArrayType_, round-tripped back as a binary value instead of the original annotated object. Only accept "byte" / emit 'B' when bjdata_version selects Draft 3. Under Draft 2, fall back to the same plain-object encoding used elsewhere in this function for other invalid-annotation cases, so the value round-trips correctly. Fixes #5404. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Future-proof the Draft-3-only 'B' marker gate @gregmarr pointed out that dtype == 'B' && bjdata_version != draft3 only future-proofs by accident, since bjdata_version_t currently has exactly two values. Compare with < instead, so a later draft that keeps the 'B' marker valid does not need this gate revisited. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels LohmannandClaude Opus 5
|
c72f37a40d |
Support custom object/array types and improve template parameter handling (#5443)
* docs: document the implicit requirements on basic_json's template parameters The requirements that basic_json places on its eleven template parameters were only implied by how the library uses the resulting object_t, array_t, string_t, etc. Consumers had to discover them by trial and error. Add "Template Parameter Requirements" collecting them, split into what is always required and what is only required when a particular part of the API is instantiated. Notable findings that were previously undocumented: - ObjectType must provide a key_compare member type (actual_object_comparator names object_t::key_compare in both arms of a std::conditional), and its third template parameter is used as a comparator, so std::unordered_map cannot be used without a wrapper. - ArrayType must provide capacity() -- push_back(), emplace_back(), operator+=(), and operator[](size_type) call it unconditionally -- and needs random-access iterators, so std::deque and std::list do not work. - StringType needs contiguous, null-terminated data(), a one-byte value_type, and either assignability from std::to_string or an ADL int_to_string(). - NumberFloatType must be float, double, or long double for parsing and serialization; the integer types must satisfy std::is_integral. - AllocatorType must be stateless, support incomplete types, and use plain pointers. - BooleanType and the number types are union members and must be trivial. Link the new page from the basic_json overview, the types feature page, and the individual type alias pages, and correct the container examples given for ObjectType (std::unordered_map) and ArrayType (std::list), which do not work. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018hxZxz8svM54c6ATEvXp5E Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix object_comparator_t for object types without key_compare detail::actual_object_comparator selected between object_t::key_compare and default_object_comparator_t with std::conditional. Both type arguments of std::conditional are named eagerly, so object_t::key_compare had to exist regardless of the condition, and the has_key_compare guard added in 3.11.0 never took effect: any ObjectType without a key_compare member type failed to compile while instantiating basic_json itself. Use detected_or_t instead, which resolves through a SFINAE partial specialization and only names object_t::key_compare when it exists. The selected type is unchanged for every object type that compiled before, so object_comparator_t -- a public member type -- keeps its meaning and ABI. has_key_compare had no other users and is removed. Add a regression test using an adapter around std::unordered_map, which has no key_compare; it fails to compile without this change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018hxZxz8svM54c6ATEvXp5E Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: list the types that are known to work for each template parameter Follow up on the template parameter requirements page: state, for every template parameter, which concrete types work and where they stop working. Each entry was verified by compiling and running a common workload (DOM access, dump, parse, CBOR/MessagePack round-trip, flatten, hash) against that instantiation. Findings worth calling out: - ObjectType no longer needs a key_compare member type, so the std::unordered_map adapter only has to restore the template argument order. A hash-ordered ObjectType works everywhere except unflatten(), which reconstructs an array only when it meets the reference token 0 before the other indices. - ArrayType: std::deque works when wrapped to add capacity(); std::list does not. - StringType: std::pmr::string and std::basic_string with a custom allocator compile for the DOM, dump, and parse, but not for the binary readers, flatten, or diff, because the library assigns std::string values to string_t and int_to_string cannot be overloaded for a type in namespace std. - NumberFloatType: long double works for dump and parse but not for the binary formats, which have no encoding for it. - BinaryType: std::vector<std::byte> supports assignment, get, and the binary formats, but neither dump nor std::hash<basic_json>. Also record the object_comparator_t fix in its version history. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018hxZxz8svM54c6ATEvXp5E Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix unflatten and binary dumping for non-default configurations unflatten() decided between array and object by looking at the first reference token it happened to see for a node: it started an array only when that token was 0. With a sorted object type the token 0 always arrives first, so the result was correct by accident; with an object type whose iteration order is unspecified, {"/c/2":3,"/c/1":2,"/c/0":1} unflattened to an object with the keys "0", "1", and "2" instead of an array. Collect the pointer prefixes that have a reference token 0 among their children before building the result, and let get_and_create() consult that set. The outcome is now independent of the iteration order and matches, for every input, what a sorted object type produced before: a value is restored as an array if and only if one of its keys is 0. Iterating the flattened object in a different order would have been simpler, but it would have changed the key order of the result for insertion-ordered object types. The serializer, std::hash, and the UBJSON writer converted the elements of a binary value to an integer implicitly, which does not compile for a BinaryType whose value type is std::byte, and which made dump() write the bytes of a signed value type as negative numbers. Convert to std::uint8_t explicitly in all three places, so every byte type dumps as 0..255. The default std::vector<std::uint8_t> configuration is unaffected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018hxZxz8svM54c6ATEvXp5E Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: note which Abseil containers can be used as template arguments Checked against Abseil release 20250127.0 with the same workload as the other entries on the page (DOM access, dump, parse, CBOR/MessagePack/UBJSON round-trip, flatten, hash), with and without JSON_DIAGNOSTICS. absl::flat_hash_map and absl::node_hash_map work as ObjectType through an adapter that restores the template argument order and makes erase(iterator) return the following iterator, which Abseil's returns as void. The page now carries that adapter, and notes that absl::flat_hash_map does not keep references to the mapped values valid across insertions while absl::node_hash_map does. Both have a capacity() member, so JSON_DIAGNOSTICS already refreshes the parent pointers conservatively for them. absl::btree_map and absl::InlinedVector cannot be used at all: object_t and array_t are formed while basic_json is still incomplete, and both inspect their value type at class scope. std::map and std::vector are required by the standard to tolerate this, third-party containers generally are not, so the page states the constraint on its own rather than only per container. absl::InlinedVector does work as BinaryType, where it is instantiated with a complete type. absl::FixedArray and absl::Cord are not usable. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018hxZxz8svM54c6ATEvXp5E Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Relax the ArrayType and ObjectType requirements Two requirements forced users of otherwise suitable containers to write a wrapper, and neither was load-bearing. array_t::capacity() was read in push_back(), emplace_back(), operator+=(), and operator[](size_type), but set_parent() only looks at the value under JSON_DIAGNOSTICS; without diagnostics it was computed and discarded. Read it through array_capacity(), which reports unknown_size() when diagnostics are off or when the array type has no capacity() at all, and treat an unknown capacity as "the elements may have moved" so the parent pointers are refreshed conservatively. std::deque now works as ArrayType, in both builds, and capacity() is no longer named at all in a default build. Since the capacity is now only meaningful for array insertions, it moves out of set_parent() into set_parent_after_array_insert(). basic_json::erase(iterator) assigned the object's erase() return value, which requires the container to return the following iterator. Abseil's hash maps return void to avoid computing a successor the caller may not need. Detect that and compute the successor before erasing; containers that return an iterator, including the vector-backed ordered_map where a precomputed successor would be wrong, keep the existing path. Together these leave an Abseil hash map needing only an alias that restores the template argument order, and no adapter at all for std::deque. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018hxZxz8svM54c6ATEvXp5E Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Do not require string_t to be convertible from std::string Three places built a std::string and handed it to something expecting a string_t: the UBJSON high-precision number reader, which every binary reader instantiates, and the BSON writer's array element size calculation and write. That silently required string_t to be implicitly convertible from std::string, which std::string itself and types with a string_view conversion satisfy, but many string types do not. Construct the string_t explicitly from the data and size, which the requirements already cover. This makes boost::container::string, eastl::string, std::pmr::string, and std::basic_string with a custom allocator work as StringType, none of which could previously be used with any binary format. Add binary format coverage to the alt_string test, which had none, including a UBJSON high-precision number -- the case that goes through the reader path. BSON stays uncovered there: it additionally needs string_t::find(value_type), which alt_string does not provide. Also record which containers from Boost, Abseil, and EASTL work for each template parameter, and correct two claims: std::pmr::string is usable after this change, and tsl::ordered_map is not usable at all, because its iterators expose the mapped value as const while basic_json modifies it in place. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: record compatibility for the common header-only hash maps ankerl::unordered_dense (map and segmented_map), phmap (flat_hash_map and node_hash_map), and robin_hood::unordered_flat_map all work as ObjectType through the same adapter as Abseil's and Boost's hash maps, which only has to restore the template argument order. phmap::btree_map and robin_hood::unordered_node_map do not: like the other btree containers they require a complete value type. Note that none of these hash maps defines key_compare, so every one of them depends on object_comparator_t falling back to default_object_comparator_t. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: record Folly and the remaining vector replacements Folly works, with the caveat that its headers need C++20: folly::fbstring as StringType, folly::fbvector and folly::small_vector as ArrayType, folly::fbvector<std::uint8_t> as BinaryType, and folly::F14NodeMap as ObjectType through the usual argument-order adapter. folly::F14FastMap is the exception and requires a complete value type. For ArrayType, boost::container::devector, boost::container::static_vector (within its fixed capacity), and std::pmr::vector work as well. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: cover fifo_map, gtl, folly::sorted_vector_map, and Qt nlohmann::fifo_map works through the adapter that has always been documented for it, and preserves the insertion order. Restore its mention in the object order page, which was dropped together with the tsl::ordered_map one: unlike ordered_map it keeps a lookup index, so it is the insertion-ordered option without the quadratic cost. gtl::flat_hash_map and folly::sorted_vector_map work as well, the latter through an alias that drops the allocator, whose value type it disagrees on. gtl::btree_map does not, for the same reason as the other btree containers. None of the Qt containers can be used, each for its own reason: QMap has no value_type, QHash iterators yield the mapped value rather than a pair, QList has no max_size(), QByteArray spells empty() as isEmpty(), and QString is UTF-16. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: qualify the std::pmr::string support claim Listing std::pmr::string as fully supported was an overclaim: it was only ever checked with the default memory resource, which is not what PMR is for. basic_json cannot be given an allocator or a memory resource, so a pmr string inside a value always allocates from std::pmr::get_default_resource(), and assigning an arena-backed string into a value silently drops its resource, because polymorphic_allocator does not propagate on copy construction. Passing polymorphic_allocator as AllocatorType does not compile either. Only the process-global set_default_resource() redirects these allocations. Say so, and separate the row from std::basic_string with a custom stateless allocator, which is unaffected. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: remove a duplicated StringType compatibility section The StringType section carried two 'Compatible types' tables and two copies of the reference-implementation tip. The second table was a stale copy from before the binary format string fixes and still listed std::pmr::string and std::basic_string with a custom allocator as unusable, contradicting the corrected table a few lines above it, and it dragged along the old explanation that blamed int_to_string. Drop the stale copy and put the surviving table before the notes, so the 'see below' in the std::pmr::string row points forwards. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: correct the template parameter requirements after independent verification Every claim on the page was re-checked by compiling and running it, including the rows that say a type cannot be used, which were checked to fail for the documented reason and not merely to fail. Twenty-four claims were wrong. The most consequential: the incomplete-type constraint applies to ObjectType only. object_t is instantiated inside the class definition, because it is probed for key_compare; array_t is only named there and is not instantiated until basic_json is complete. So eastl::vector, QList and QVector are not excluded by incomplete types at all -- they simply have no max_size() -- and absl::InlinedVector is excluded for a subtler reason of its own. Further corrections: ObjectType does not need erase(key), which has a fallback, but does need at(key) for UBJSON output; only == and < are used, or == and <=> under C++20, not all six; the documented adapter does not fit ankerl or robin_hood. ArrayType needs no initializer-list insert, and value_type, the (count, value) constructor and swappability are per-function, not always. BinaryType needs a range insert for CBOR indefinite-length byte strings and does not need push_back. StringType needs append(const StringType&) unconditionally, and does not need operator!= or operator== against const char*; empty(), resize(n) and reserve(n) are per-subsystem; int_to_string is needed by diff, items and std::hash rather than by JSON Pointer or flatten. BooleanType must be implicitly convertible from bool, and JSONSerializer's second parameter need not carry a default. std::pmr::string was wrong in the other direction this time: a moved-in string does keep its memory resource, and later growth allocates from it. Only copies land on the default resource. Five requirement violations are not caught at compile time rather than the two the page claimed; they are now listed together up front. Split every compatibility table into what works and what does not, as the reasons in the second half are the useful part. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Reduce the string_t and array_t members the library requires Several members were required only because of how the library happened to be written, not because the functionality needs them. Dropping them widens the set of usable string and array types, and one of them was also a performance problem. string_t: - c_str() is gone. Every call site already knew the length and passed it along, so data() is enough. The one place that did not, the diagnostics path in exceptions.hpp, now builds the token from data() and size(), which also stops it from truncating keys that contain a null byte. - back() is gone; the serializer indexes the last character instead. - find(str, pos), replace(), and substr() are gone. escape() and unescape() rebuilt the string with one replace() per escaped character, which moves the tail every time: escaping a string of n characters that all need escaping cost O(n^2). Both now scan with find_first_of() -- a member the pointer parser already required -- and append whole runs, so the common case is one search and one copy. Escaping 64000 tildes drops from 717 ms to 20 ms; a string with nothing to escape gets faster too (8.4 ms to 5.8 ms), because the scan is still a single memchr per pass. json_pointer::split() takes its reference tokens with the (const char*, size_type) constructor rather than substr(). - json_pointer::to_string() accumulates with concat<string_t> instead of letting concat default to std::string and converting afterwards, so streaming a json_pointer no longer requires string_t to be assignable from a std::string. array_t: - at(size_type) is gone. basic_json::at(size_type) checked the index by calling array_t::at() and translating std::out_of_range, which also required the array type to throw that exact exception. It now compares against size() and uses operator[]. The thrown exception, its message, and the behaviour under JSON_NOEXCEPTION are unchanged. The BSON writer wrote the terminating null byte out of the string's own buffer (size() + 1). It now writes the byte itself, so string_t::data() need not be null-terminated for to_bson(). The tests pin the reduced API: alt_string loses the five dropped members and gains coverage of the escaping paths, and a std::vector whose at() is hidden is used as an ArrayType. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: record the reduced string_t and array_t requirements Drop c_str(), back(), find(str, pos), replace(), and substr() from the StringType requirements and at(size_type) from the ArrayType ones, and note the string assignment the JSON pointer code performs. Streaming a json_pointer no longer needs assignability from a std::string. Add the non-null-terminated data() to the list of violations that are not diagnosed at compile time -- it was described in the StringType section but missing from the summary at the top -- and correct the QString row, which no longer fails for the c_str() it lacks. JSON_CATCH_USER no longer wraps a catch of std::out_of_range: the last one went away with array_t::at(). Describe what the library actually catches. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Use character literals for the signed BinaryType test MSVC rejects char(0xFF) with C4310 (cast truncates constant value), which the Windows workflow treats as an error. The character literals carry the same byte values without a narrowing cast. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Do not instantiate a hash map with an incomplete basic_json in the tests object_t is probed for key_compare inside the definition of basic_json, so it is instantiated while basic_json is still incomplete. Whether a hash map survives that depends on the standard library: libstdc++ 9 needs the size of the mapped type to instantiate std::unordered_map's node type and rejects the adapter, which broke the GCC 9 builds. The test now derives its no-key_compare object type from std::map -- which does cope -- and shadows the inherited key_compare member type with an entity that is not a type, so the library's probe finds none, exactly as for a hash map. The unflatten() order-independence checks in unit-json_pointer already cover the behaviour that the unordered object type was there for. The limitation is documented for std::unordered_map. Also address two Clang-Tidy findings the earlier commits introduced: erase_from_object() declares its iterator with auto, and at(size_type) checks the type first and then falls through to the return instead of throwing from an else branch. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Keep diagnostic key paths null-terminated Building the token from data() and size() kept an embedded null byte in the key, and since what() hands out a C string, that truncated the whole message rather than just the key: to_bson() on a key containing U+0000 reported "[json.exception.out_of_range.409] (/en" instead of the full explanation. This broke test-bson under JSON_DIAGNOSTICS. Constructing from data() alone stops at the first null byte, which is what c_str() did before, so the message is unchanged -- without requiring string_t to provide c_str(). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Do not parse the value in the array-at() test JSON_DIAGNOSTIC_POSITIONS adds the byte range of the value to the exception message, which a parsed value has and an in-memory one does not, so the two message checks failed in that configuration. Build the array in memory instead of parsing it; the test is about at(size_type) not needing array_t::at(), and the byte range is beside the point. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Move the custom BinaryType tests into their own translation unit The two sections added to unit-regression2.cpp brought a third full basic_json instantiation into a translation unit that was already large. With Clang on MinGW that pushed the object over the reach of a 32-bit relocation and test-regression2_cpp20.exe failed to link: relocation truncated to fit: IMAGE_REL_AMD64_REL32 against `.rdata' unit-regression2.cpp is restored to exactly what it was before, and the coverage moves to unit-custom-binary-type.cpp, next to the object and array type tests it belongs with. The signed value type is now also covered in C++11, where std::byte is not available. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Do not require the container iterators to be nothrow move constructible iter_impl declared its defaulted move operations noexcept. The exception specification a defaulted function gets implicitly follows from its members, here internal_iterator, which holds the object and array iterators. libstdc++ gives std::deque's iterator a user-provided copy constructor without noexcept before version 11, so the implicit specification is noexcept(false) and does not match the declared one. That deletes the function -- and with g++ 4.8, which predates CWG 1778, it is an error outright: error: function 'iter_impl<basic_json<std::map, std::deque> >::iter_impl( iter_impl&&)' defaulted on its first declaration with an exception-specification that differs from the implicit declaration So std::deque, which this branch documents as a usable array type, could not be used with an older standard library. Leaving the specification to be computed cannot mismatch; iteration_proxy_value already spells out the same condition next door. The default configuration is unaffected: json::iterator, json::const_iterator and ordered_json::iterator stay nothrow move constructible and move assignable, which the test now checks so it cannot regress unnoticed. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address two Clang-Tidy findings the custom container tests exposed Both come from instantiating basic_json with containers other than the default ones, and neither shows up with the Clang-Tidy version available outside CI: - insert(const_iterator, basic_json&&) forwards its by-value iterator to the const-reference overload. performance-unnecessary-value-param asks for the copy to be a move; it only fires for an iterator that is not trivially copyable, as std::deque's is not. The NOLINT on the function does not cover it, because the finding is reported where the parameter is used rather than where it is declared. Move it, which is what the check asks for and is a (very small) improvement in its own right. - cppcoreguidelines-use-enum-class rejects the unnamed enum that shadowed the inherited key_compare member type. An enum class would not do, since it declares a type of that name and the probe would find it again; a member function declaration hides the name just as well. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Assert the iterators' exception specification relative to the container The test pinned that nlohmann::json's iterators stay nothrow movable after iter_impl's defaulted move operations lost their declared noexcept. That is not a property of the library, though: the exception specification is now computed from the container iterators, so it holds only for standard library implementations whose iterators are themselves nothrow movable. MSVC's checked iterators before VS2017 are not -- _Iterator_base12 registers the iterator with the container's debug proxy in a copy constructor that carries no noexcept -- so the assertions fail on a Visual Studio 2015 debug build, which is the one debug configuration in the AppVeyor matrix and has no counterpart in the GitHub Actions matrix. Assert what the change actually guarantees instead: the iterators are nothrow movable exactly when the object and array iterators they are built from are. That still pins the default configuration against a silent regression, and it is true whatever the standard library provides. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Detect a void-returning erase() through a named trait erase_from_object() distinguished its two overloads with a decltype of a member call written inline in a default template argument. Every other detection in the library goes through the detector machinery in detected.hpp instead -- has_erase_with_key_type is the same question about the same member function -- and the inline form is the one shape older compilers are least reliable about. Express it the same way: detect_erase_with_iterator plus is_detected_exact, both of which the library already relies on elsewhere. No behaviour changes. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Give the custom container types only the constructors the library uses The three container types in the new tests inherited every constructor of their base with using Base::Base. That asks for more than the test needs: the library builds an object or an array by default construction, by copy or move, and -- when converting between two basic_json types or from an initializer list -- from an iterator range. Declaring those directly makes the requirement visible in the test, and keeps object types out of a corner where a compiler has to declare std::map's whole constructor set for a derived class while basic_json is still incomplete. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Temporarily disable the new custom container tests AppVeyor is the only CI that builds MSVC 2015 and 2017, and it has now rejected three heads of this branch. Its build log is not reachable from where this is being worked on, so the verdict is a single bit and the cause has to be narrowed down by bisection. Everything else stays: the library changes, the reduced alt_string, and the unflatten() tests. If AppVeyor passes with these three translation units disabled, the cause is one of the six basic_json instantiations they add; if it fails, it is in the library. Either way this commit is reverted. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Guard the disabled tests with a macro rather than #if 0 Clang-Tidy's readability-avoid-unconditional-preprocessor-if rejects a literal #if 0. Use a macro that is never defined instead, which the check does not look at. Still temporary, and reverted together with the previous commit. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Re-enable the array and binary container tests AppVeyor passed with all three new translation units disabled, so the library changes, the reduced alt_string, and the unflatten() tests are fine on MSVC 2015 and 2017; the cause is one of the six basic_json instantiations the new tests add. Bring back two of the three. If AppVeyor passes again, the cause is in unit-custom-object-type.cpp, which is the one still disabled; if it fails, it is in one of these two and needs one more split. Still temporary. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Diagnose two silently violated template parameter requirements Both were on the list of requirements that are not caught at compile time and corrupt values rather than failing, and both are a plain size comparison: - A BinaryType whose value_type is wider than one byte, which the readers and writers reinterpret as raw bytes anyway. - A NumberUnsignedType too narrow to hold the absolute value of every NumberIntegerType value, which makes basic_json(INT64_MIN).dump() yield -0 for std::int64_t with std::uint32_t. Neither static_assert rejects a configuration that worked before: both only fire where the result was already wrong. Also add the two comments the review asked for, in write_bson_string() and calc_bson_array_size(), matching the ones their counterparts already carry. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Align the template parameter tables and record what is now diagnosed Every table in the page is reformatted so each column is exactly as wide as its widest cell, which is what the review asked for in a dozen places: the separator rows that ran two dashes long, the stray spaces, and the columns padded well past their content. The row listing six containers that require a complete mapped type is split in two so that one cell no longer sets the width of the whole table. Content changes: NumberUnsignedType is described as any unsigned integer type at least as wide as NumberIntegerType rather than any unsigned integer type; the two requirements that are now static_asserts move out of the list of violations that are not caught at compile time; and the two places that require a non-const operator[] say why data() will not do (std::string has no non-const data() before C++17). Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Bisect the other way: only the object container tests The previous head touched only docs/, which AppVeyor's only_commits filter skips, so it produced no build and no status at all -- the pull request looked green without ever having been built on MSVC 2015 or 2017. Swap the guards instead of repeating that step: unit-custom-object-type.cpp is enabled and the array and binary translation units are disabled. AppVeyor already passed with all three disabled, so a failure here pins the cause on no_key_compare_json or void_erase_json, and a pass pins it on the array or binary file. Still temporary. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Split the two object types apart AppVeyor failed with only unit-custom-object-type.cpp enabled and passed with all three new translation units disabled, so the cause is one of the two object types in this file and not the array or binary ones. Guard out void_erase_map and leave no_key_compare_map, which separates the two constructs under suspicion: shadowing the inherited key_compare member type with an entity that is not a type, and hiding the inherited erase with a void-returning overload. A failure here points at the first, a pass at the second. Still temporary. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Build the no-key_compare object type by composition, not inheritance The "object type without key_compare" test failed on AppVeyor's MSVC 2017 jobs (/std:c++17): its no_key_compare_map derived publicly from std::map and shadowed the inherited key_compare type with a same-named member function, relying on ordinary member hiding to make key_compare unreachable as a type for the library's detection trait. MSVC 2017 does not honor that hiding for a typename-qualified lookup performed from outside the class and still resolves key_compare to the base's comparator type, so object_comparator_t incorrectly picked it up instead of falling back to default_object_comparator_t. Wrapping a std::map by composition instead removes the base class entirely, so there is no key_compare to find under any lookup rule, on any compiler. Also drops the now-unneeded JSON_BISECT_CUSTOM_CONTAINER_TESTS guard left over from narrowing this down: the void_erase_map test in the same file was never the cause and is re-enabled unconditionally. Verified locally with clang++ and g++ under C++17 and C++20. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Re-enable the array and binary custom-container tests unit-custom-array-type.cpp and unit-custom-binary-type.cpp were still guarded behind JSON_BISECT_CUSTOM_CONTAINER_TESTS from bisecting the AppVeyor failure fixed in |
||
|
Niels Lohmann
|
44f8ec30e9 |
Bound UBJSON optimized arrays of a valueless type (#5504)
* Bound UBJSON optimized arrays of a valueless type An element of type 'Z' (null), 'T' (true) or 'F' (false) is encoded by its type marker alone, so an optimized UBJSON array of one of those has no payload: reading an element consumes no input at all. Its declared count is therefore the only thing that decides how much is allocated, and nothing bounded it. "[$Z#l" and a four-byte count is nine bytes of input describing two billion values; #2793 reports 35 GB and 150 seconds from ten bytes, and OSS-Fuzz has an out-of-memory and a timeout report for the same shape. Every other type costs at least one byte per element, so the end of the input bounds it. 'N' (no-op) is already skipped rather than stored. Objects are not affected either: each element is preceded by its key, which costs bytes. And BJData already refuses these markers as an optimized type, so this is a plain UBJSON matter. Reject a count above 1,048,576 elements for those three types with out_of_range.408, the code this reader already uses for a declared size it will not honour. The check runs before the SAX start event, so no container is opened and then abandoned. Rejecting on the read side alone would break the guarantee that anything to_ubjson() writes can be read back, and would trip the round-trip assertion in fuzzer-parse_ubjson.cpp. So the writer falls back to the unoptimized encoding, one byte per element, for arrays of these types above the same limit. Its decision depends only on the array's size, which is identical for a value and for anything parsed back from it, so the round trip is stable. No existing test changes: the largest such count in the test suite is 65,793. The excessive-size test that already used this shape still passes, now rejected a little earlier than by the max_size() check it used to reach. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Note the 1,048,576 valueless-array limit as (1 << 20) in the docs Addresses review feedback from @gregmarr on PR #5504: spell out the binary/hex form next to the decimal count so it reads as the round power-of-two it is, matching how include/nlohmann/detail/input/binary_reader.hpp defines max_valueless_container_size. Applied in both docs/exceptions.md and ubjson.md, as requested. Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
bfb07786cd |
Fix to_bjdata() silently truncating out-of-range _ArrayData_ elements (#5473)
write_bjdata_ndarray() validated that each _ArrayData_ element matched the number kind (integer vs. float) named by _ArrayType_, but not its range. An element that did not fit the target C++ type (e.g. 256 for "uint8") was silently wrapped by the static_cast used to write it, or, for "single", silently overflowed to infinity. Range-check each element against the type named by _ArrayType_ before writing it, reusing the existing fallback path that already encodes the annotated object as a plain object for other invalid-annotation cases in this function. Fixes #5403. Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
d6660cf718 |
Route hand-rolled diagnostic pragmas through Hedley (#5485)
* Route hand-rolled diagnostic pragmas through Hedley
Several places in the library hand-roll compiler diagnostic suppression
with raw `#pragma`/`#ifdef __GNUC__`/`#ifdef __clang__` guards instead of
using the Hedley primitives already bundled and used elsewhere
(JSON_HEDLEY_DIAGNOSTIC_PUSH/POP, JSON_HEDLEY_PRAGMA, ...). Converted six
of the seven listed push/pop pairs to use those primitives instead of
raw `#pragma GCC diagnostic`/`#pragma clang diagnostic` text:
- include/nlohmann/json.hpp (~3770, ~3863): -Wfloat-equal
- include/nlohmann/detail/conversions/to_chars.hpp (~1078): -Wfloat-equal
- include/nlohmann/detail/output/binary_writer.hpp (~1844): -Wfloat-equal
- include/nlohmann/detail/iterators/iteration_proxy.hpp (~211): -Wmismatched-tags
- include/nlohmann/detail/exceptions.hpp (~36): -Wweak-vtables
iteration_proxy.hpp did not previously include macro_scope.hpp itself
(it only compiled because some other header included earlier in
json.hpp happened to pull macro_scope.hpp in first); it now includes it
directly like the other detail headers that use Hedley macros, so it is
self-contained.
Each push/pop pair now uses JSON_HEDLEY_DIAGNOSTIC_PUSH/POP
unconditionally (a no-op on compilers that don't need it) and wraps the
actual `#pragma ... diagnostic ignored` text in JSON_HEDLEY_PRAGMA so it
goes through Hedley's _Pragma()-based emission instead of a raw #pragma
line, while keeping the original `#ifdef __GNUC__` / `#if
defined(__clang__)` guard around the ignored-pragma itself.
Deviation from the issue's suggested transformation: the issue's example
replaces the `#ifdef __GNUC__` guard with `#if
JSON_HEDLEY_HAS_WARNING("-Wfloat-equal")`. JSON_HEDLEY_HAS_WARNING is
implemented purely via Clang's `__has_warning` builtin and evaluates to
0 on real GCC (`#define JSON_HEDLEY_HAS_WARNING(warning) (0)` when
`__has_warning` is not defined), so adopting it verbatim would silently
stop suppressing -Wfloat-equal on GCC -- a real regression, not just a
style change. The existing `#ifdef __GNUC__` / `#if defined(__clang__)`
guards were kept for the ignored-pragma to stay behavior-preserving, and
only the push/pop/pragma-emission mechanism was routed through Hedley.
Two of the seven locations from the issue (the -Wignored-attributes
push at the very top of json.hpp and its matching pop after
`#include <nlohmann/detail/macro_unscope.hpp>`) were intentionally left
unconverted:
- The push, at the very top of json.hpp, runs before
`detail/macro_scope.hpp` (and therefore hedley.hpp) has been included
anywhere in the translation unit, so JSON_HEDLEY_DIAGNOSTIC_PUSH is not
yet defined at that point.
- The pop runs after `macro_unscope.hpp`, which -- via hedley_undef.hpp
-- has already #undef'd every JSON_HEDLEY_* macro (by design, see
#5408) precisely so they don't leak to users, so JSON_HEDLEY_DIAGNOSTIC_POP
is no longer defined by the time the pop is reached either.
Making this one pair work would require either hoisting the ~2000
line vendored hedley.hpp to the very top of the amalgamated single
header (a much bigger structural change to single_include than a pure
mechanism swap) or special-casing this one pop ahead of the general
macro cleanup. Both are riskier than the mechanical, behavior-preserving
change requested, so this pair was left as-is.
## Validation
- Compiled include/nlohmann/json.hpp and single_include/nlohmann/json.hpp
with `-Wall -Wextra -Wfloat-equal -Wmismatched-tags -Wweak-vtables`
(clang, which self-identifies as __GNUC__ too): no warnings, same as
before the change.
- Compiled and ran tests/src/unit-to_chars.cpp, unit-conversions.cpp,
unit-iterators1.cpp, unit-iterators2.cpp, and unit-class_parser.cpp
against the fixed include/: all pass.
- Compiled unit-msgpack.cpp, unit-bjdata.cpp, and unit-ubjson.cpp (which
exercise binary_writer.hpp's write_compact_float extensively): all
compile cleanly; the vast majority of assertions pass (the only
failures are pre-existing environment issues unrelated to this change
-- missing generated test-data files, not code correctness).
- Ran `make amalgamate`; the single_include diff is limited to exactly
the lines touched in include/, with no unrelated reordering.
- No real (non-Apple) GCC was available in this environment to test
directly; the `_Pragma("GCC diagnostic ...")` text emitted by
JSON_HEDLEY_PRAGMA is byte-identical to the prior `#pragma GCC
diagnostic ...` text, and the `#ifdef __GNUC__` guard is unchanged, so
GCC's behavior is expected to be identical. CI covers the GCC matrix.
This PR is stacked on top of #5475 (issue-5408-hedley-undef-leak) since
both touch the same files; only the last commit here is new.
Fixes #5409.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Guard JSON_HEDLEY_DIAGNOSTIC_PUSH/POP with the same compiler check as the pragma they bracket
Addresses review feedback from @gregmarr on PR #5485: the push/pop calls
were unconditional, so compilers other than the one the ignored-pragma
targets (e.g. MSVC, or GCC where the pair only applies under __clang__)
now did a needless push/pop with nothing suppressed in between. Move the
existing #ifdef __GNUC__ / #if defined(__clang__) guard to also cover the
push/pop, restoring the original zero-overhead behavior on other compilers
while still emitting the pragma itself through Hedley.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
|
||
 Qatadaha Bin Matloob
|
09b6b6b5ba |
Fix to_bjdata() emitting unparsable output when _ArraySize_ is not an array (#5455)
* Fix to_bjdata() emitting unparsable output when _ArraySize_ is not an array
write_bjdata_ndarray() never checked that _ArraySize_ is an array. The shape
is written verbatim as the header length, so a null shape emitted 'Z' and an
object shape emitted '{' after the '#', neither of which from_bjdata()
accepts, and the round-trip guarantee in the BJData docs was broken.
Both slipped through the existing validation: for null, empty() is true so
the element count starts at 0 and the per-dimension loop never runs, and for
an object the loop walks its values, which can satisfy the non-negative
integer check. When _ArrayData_ then matched that count, the writer took the
ndarray path.
Require the shape to be an array, so anything else falls back to a plain
object encoding that round-trips, as the fallback rule in the docs already
specifies.
Signed-off-by: qatcod <79017227+qatcod@users.noreply.github.com>
* Document that _ArraySize_ must be an array in the ndarray requirements
The list at bjdata.md is the exhaustive set of conditions for the ndarray
encoding, but it only implied this one through 'every entry of'.
Signed-off-by: qatcod <79017227+qatcod@users.noreply.github.com>
---------
Signed-off-by: qatcod <79017227+qatcod@users.noreply.github.com>
|
||
 Krishnanand G
|
2f025f401e |
Throw other_error.502 when UBJSON use_type is set without use_size (#5380)
* Throw other_error.502 when UBJSON use_type is set without use_size Fixes #5321 Signed-off-by: Krishnanand G <118352827+Krishnanand-G@users.noreply.github.com> * Scope UBJSON use_type check to container branches and expand tests Signed-off-by: Krishnanand G <118352827+Krishnanand-G@users.noreply.github.com> * Re-amalgamate single_include/json.hpp The previous commit updated the split headers but the amalgamated file didn't go back through astyle before I committed it, so CI's amalgamation check caught formatting drift in json_fwd.hpp and a few noexcept clauses in basic_json, plus one doc example. None of it touches the UBJSON logic. Applied the patch CI generated to bring single_include back in sync. Signed-off-by: Krishnanand G <118352827+Krishnanand-G@users.noreply.github.com> --------- Signed-off-by: Krishnanand G <118352827+Krishnanand-G@users.noreply.github.com> |
||
|
Niels Lohmann
|
9a091d2b82 |
Do not write BJData ndarrays whose size overflows std::size_t (#5362)
* Do not write BJData ndarrays whose size overflows std::size_t
write_bjdata_ndarray() multiplied the _ArraySize_ dimensions into a
std::size_t without checking for overflow. A product that wraps around
to a value that happens to match the size of _ArrayData_ passed the
length check, and the writer emitted an ndarray header announcing an
element count that cannot be represented:
{"_ArrayType_":"uint8","_ArraySize_":[9223372036854775808,2],"_ArrayData_":[]}
was encoded as 5b 24 55 23 5b 4d 00 00 00 00 00 00 00 80 69 02 5d, an
ndarray of 2^64 elements followed by no data. Reading that back throws
out_of_range.408 ("excessive ndarray size caused overflow"), so to_bjdata
produced output that from_bjdata rejects. This is reachable by parsing
untrusted JSON and re-encoding it as BJData.
Mirror the overflow check the binary reader already performs, and also
reject a single dimension that does not fit into std::size_t, which the
previous cast silently truncated where std::size_t is narrower than 64
bits. Such objects now fall back to a plain object encoding, which is
what the surrounding type and length validation already does for
annotations it cannot represent, and they round-trip unchanged.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Document when to_bjdata converts a JData annotation to an ND-array
The BJData page described the 1-D vector case as the only situation in
which an object carrying _ArrayType_/_ArraySize_/_ArrayData_ is not
written as a compact ND-array. The writer has always had several other
fallbacks -- an unknown _ArrayType_, a dimension that is not a
non-negative integer, an _ArrayData_ whose length does not match the
product of the dimensions, and elements that are not numbers of the
annotated kind -- all of which cause the value to be serialized as a
regular JSON object instead.
Spell out the conditions, including the size-overflow check added in the
preceding commit, so the documented behavior matches the implementation.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
|
||
 Angadi56
|
fd72ecfc8c |
validate ndarray element types in write_bjdata_ndarray (#5301)
* validate ndarray element types in write_bjdata_ndarray Signed-off-by: Angadi Yashaswini <angadi@digiscrypt.com> * read ndarray elements through get<> instead of a fixed union member _ArrayType_ names the wire type, not how the value is stored: parsing keeps a non-negative integer as number_unsigned while the C++ API keeps an int literal as number_integer. Selecting the union member from the type marker therefore reads the inactive alternative for one of the two, so read through get<> instead, which dispatches on the active member. Also reject a negative _ArraySize_ entry, which is not a usable dimension, and cover the parse-built path in the tests. Signed-off-by: Angadi Yashaswini <angadi@digiscrypt.com> --------- Signed-off-by: Angadi Yashaswini <angadi@digiscrypt.com> |
||
 Luke Banicevic
|
2e23687092 | to_bson() silently emits corrupt documents when a length exceeds INT32_MAX (#5314) | ||
|
Luke Banicevic
|
8ec98e2c9e | adding cleanups to bson writer (#5313) | ||
|
Niels Lohmann
|
31dd15b258 |
Fix ambiguous static_cast (#5221)
* 🐛 fix ambiguous static_cast Signed-off-by: Niels Lohmann <mail@nlohmann.me> * ✅ add regression test Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🐛 fix warning Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🚨 fix warning Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
515d994acb |
📄 adjust year (#5044)
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
54be9b04f0 | 📄 update REUSE (#4960) | ||
|
Niels Lohmann
|
b19f058465 |
Encode infinity and NaN as float for MsgPack and CBOR (#4802)
* 🚸 encode infinity and NaN as float for MsgPack and CBOR Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🚨 suppress warnings Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
 NmPassTHFan
|
46e7cd3dc2 | Replace deprecated std::is_trivial in C++26 (#4775) | ||
|
Niels Lohmann
|
9110918cf8 |
Fix typos (#4748)
* ✏️ fix typos Signed-off-by: Niels Lohmann <mail@nlohmann.me> * ✏️ address review comments Signed-off-by: Niels Lohmann <mail@nlohmann.me> * ✏️ address review comments Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
88c92e605c |
Fix compilation failure and warnings with NVHPC (#4744)
* 🚨 fix warnings Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🚨 fix warnings Signed-off-by: Niels Lohmann <mail@nlohmann.me> * ⚗️ enable ranges support Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🔥 remove ci_nvhpc job Signed-off-by: Niels Lohmann <mail@nlohmann.me> * ⚗️ enable ranges support Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🔥 remove ci_nvhpc job Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 🚨 fix warning Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
1705bfe914 |
🔖 set version to 3.12.0 (#4727)
Signed-off-by: Niels Lohmann <mail@nlohmann.me> |
||
|
Niels Lohmann
|
f06604fce0 |
Bump the copyright years (#4606)
* 📄 bump the copyright years Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 📄 bump the copyright years Signed-off-by: Niels Lohmann <mail@nlohmann.me> * 📄 bump the copyright years Signed-off-by: Niels Lohmann <niels.lohmann@gmail.com> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me> Signed-off-by: Niels Lohmann <niels.lohmann@gmail.com> |
||
 Michael Valladolid
|
2d42229f4d |
Support BSON uint64 de/serialization (#4590)
* Support BSON uint64 de/serialization Signed-off-by: Michael Valladolid <mikevalladolid@gmail.com> * Treat 0x11 as uint64 and not timestamp specific Signed-off-by: Michael Valladolid <mikevalladolid@gmail.com> --------- Signed-off-by: Michael Valladolid <mikevalladolid@gmail.com> |
||
 Nebojša Cvetković
|
48e7b4c23b | BJData Fixes (#4588) | ||
|
Nebojša Cvetković
|
2e50d5b2f3 | BJData optimized binary array type (#4513) | ||
|
Niels Lohmann
|
1b9a9d1f21 |
Update licenses (#4521)
* 📄 update licenses * 📄 update licenses |
||
|
Niels Lohmann
|
1825117e63 |
Another desperate try to fix the CI (#4489)
* 🚨 fix warning * 💚 update actions * 🚨 fix warning * 🚨 fix warning * 🚨 fix warning * 💚 update actions * 💚 update actions * 🚨 fix warning * 🚨 fix warning * 💚 update actions * 🚨 fix warning * 💚 update actions * 💚 update actions * 💚 update actions * 🚨 fix warning * 🚨 fix warning * 🚨 fix warning * 🚨 fix warning * 💚 update actions * 💚 update actions * 🚨 fix warning * 💚 update actions * 💚 update actions * 💚 update actions * 💚 update actions * 💚 update actions |
||
|
Niels Lohmann
|
9cca280a4d | JSON for Modern C++ 3.11.3 (#4222) | ||
|
Niels Lohmann
|
f56c6e2e30 | Update documentation for the next release (#4216) | ||
  Raphael Grimmandbarcode
|
bbe337c3a3 |
Prevent memory leak when exception is thrown in adl_serializer::to_json (#3901)
Co-authored-by: barcode <barcode@example.com> |
||
 Niels LohmannandFlorian Albrechtskirchinger
|
58bd97e2b1 |
Add clang-tools to required tools for ci_static_analysis_clang (#3724)
* 💚 add clang-tools to required tools for ci_static_analysis_clang * 🚨 update Clang-Tidy warning selection * 🚨 fix Clang-Tidy warnings * 🚨 fix Clang-Tidy warnings * 🚨 fix Clang-Tidy warnings * 🚨 fix Clang-Tidy warnings * 🚨 fix Clang-Tidy warnings * 🚨 fix Clang-Tidy warnings * 🚨 fix Clang-Tidy warnings * 🚨 fix Clang-Tidy warnings * 🚨 fix Clang-Tidy warnings * 🚨 fix Clang-Tidy warnings * 🚨 fix Clang-Tidy warnings (#3738) * ⏪ revert fix * ⏪ revert fix * 🚨 fix Clang-Tidy warnings (#3739) Co-authored-by: Florian Albrechtskirchinger <falbrechtskirchinger@gmail.com> |
||
|
Niels Lohmann
|
9d69186291 | 🔖 set version to 3.11.2 | ||
|
Niels Lohmann
|
f2020da0dd | 🔖 set version to 3.11.1 | ||
|
Niels Lohmann
|
ce0e13ccea | 🔖 set version to 3.11.0 | ||
|
Florian Albrechtskirchinger
|
d909f80960 |
Add versioned, ABI-tagged inline namespace and namespace macros (#3590)
* Add versioned inline namespace Add a versioned inline namespace to prevent ABI issues when linking code using multiple library versions. * Add namespace macros * Encode ABI information in inline namespace Add _diag suffix to inline namespace if JSON_DIAGNOSTICS is enabled, and _ldvcmp suffix if JSON_USE_LEGACY_DISCARDED_VALUE_COMPARISON is enabled. * Move ABI-affecting macros into abi_macros.hpp * Move std_fs namespace definition into std_fs.hpp * Remove std_fs namespace from unit test * Format more files in tests directory * Add unit tests * Update documentation * Fix GDB pretty printer * fixup! Add namespace macros * Derive ABI prefix from NLOHMANN_JSON_VERSION_* |
||
|
Niels LohmannandFlorian Albrechtskirchinger
|
527da54dcb |
Use REUSE framework (#3546)
* 📄 add licenses * 👷 add REUSE compliance check * 📝 add badge for REUSE Co-authored-by: Florian Albrechtskirchinger <falbrechtskirchinger@gmail.com> |
||
 Qianqian Fang
|
ee51661481 |
Support UBJSON-derived Binary JData (BJData) format (#3336)
* support UBJSON-derived Binary JData (BJData) format
* fix Codacy warning
* partially fix VS compilation errors
* fix additional VS errors
* fix more VS compilation errors
* fix additional warnings and errors for clang and msvc
* add more tests to cover the new bjdata types
* add tests for optimized ndarray, improve coverage, fix clang/gcc warnings
* gcc warn useless conversion but msvc gives an error
* fix ci_test errors
* complete test coverage, fix ci_test errors
* add half precision error test
* fix No newline at end of file error by clang
* simplify endian condition, format unit-bjdata
* remove broken test due to alloc limit
* full coverage, I hope
* move bjdata new markers from default to the same level as ubjson markers
* fix ci errors, add tests for new bjdata switch structure
* make is_bjdata const after using initializer list
* remove the unwanted assert
* move is_bjdata to an optional param to write_ubjson
* pass use_bjdata via output adapter
* revert order to avoid msvc 2015 unreferenced formal param error
* update BJData Spect V1 Draft-2 URL after spec release
* amalgamate code
* code polishing following @gregmarr's feedback
* make use_bjdata a non-default parameter
* fix ci error, remove unwanted param comment
* encode and decode bjdata ndarray in jdata annotations, enable roundtrip tests
* partially fix ci errors, add tests to improve coverage
* polish patch to remove ci errors
* fix a ndarray dim vector condition
* fix clang tidy error
* add sax test cases for ndarray
* add additional sax event tests
* adjust sax event numbering
* fix sax tests
* ndarray can only be used with array containers, discard if used in object
* complete test coverage
* disable [{SHTFNZ in optimized type due to security risks in #2793 and hampered readability
* fix ci error
* move OutputIsLittleEndian from tparam to param to replace use_bjdata
* fix ci clang gcc error
* fix ci static analysis error
* update json_test_data to 3.1.0, enable file-based bjdata unit tests
* fix stack overflow error on msvc 2019 and 2022
* use https link, update sax_parse_error after rebase
* make input_format const and use initializer
* return bool for write_bjdata_ndarray
* test write_bjdata_ndarray return value as boolean
* fix ci error
|
||
|
Florian Albrechtskirchinger
|
616caea27a |
Re-template json_pointer on string type (#3415)
* Make exception context optional Change exception context parameter to pointer and replace context with nullptr where appropriate. * Support escaping other string types * Add string concatenation function Add variadic concat() function for concatenating char *, char, and string types. * Replace string concatenations using + with concat() * Template json_pointer on string type Change json_pointer from being templated on basic_json to being templated on string type. * Add unit test for #3388 Closes #3388. * Fix regression test for #2958 * Add backwards compatibility with json_pointer<basic_json> * Update json_pointer docs * Allow comparing different json_pointers * Update version numbers |
||
 Niels Lohmann
|
6d3115924c |
Add C++17 copies of the test binaries (#3101)
* ⚗️ add C++17 copies of the test binaries * ⚗️ use proper header for filesystem * 🚨 fix warnings * ⚗️ do not use too old compilers with C++17 * ✅ add test * 🔨 add more constraints #3097 * ⚗️ use fix from https://github.com/nlohmann/json/pull/3101#issuecomment-998788786 * ⚗️ use fix from https://gcc.gnu.org/bugzilla/show_bug.cgi?id=90050 * 👷 use published CI image |
||
 Alex Beregszaszi
|
834918018e | Fix spelling (#3125) | ||
 Pierre Hallot
|
433604843d | Fix extra ";" clang warnings | ||
|
Niels Lohmann
|
0a0eb7a850 |
Merge branch 'develop' of https://github.com/nlohmann/json into issue2863
Conflicts: .github/workflows/windows.yml |
||
 Ferry Huberts
|
e7a69b8d31 |
Adjust more files after actually building like the CI
Signed-off-by: Ferry Huberts <ferry.huberts@pelagic.nl> |
||
|
Niels Lohmann
|
e20f3f95eb | 🚨 guard GCC pragmas #2924 | ||
|
Niels Lohmann
|
8ae31a56a5 | Merge branch 'develop' of https://github.com/nlohmann/json into issue2824 | ||
|
Niels Lohmann
|
b7db1d68d9 | ♻️ fix CBOR and BSON | ||
 Sven Fink
|
095aae19fe | Supress -Wfloat-equal on intended float comparisions | ||
|
Niels Lohmann
|
c3df4ffd5a | 🚨 fix warning | ||
|
Niels Lohmann
|
6f551930e5 |
🚨 add new CI and fix warnings (#2561)
* ⚗️ move CI targets to CMake * ♻️ add target for cpplint * ♻️ add target for self-contained binaries * ♻️ add targets for iwyu and infer * 🔊 add version output * ♻️ add target for oclint * 🚨 fix warnings * ♻️ rename targets * ♻️ use iwyu properly * 🚨 fix warnings * ♻️ use iwyu properly * ♻️ add target for benchmarks * ♻️ add target for CMake flags * 👷 use GitHub Actions * ⚗️ try to install Clang 11 * ⚗️ try to install GCC 11 * ⚗️ try to install Clang 11 * ⚗️ try to install GCC 11 * ⚗️ add clang analyze target * 🔥 remove Google Benchmark * ⬆️ Google Benchmark 1.5.2 * 🔥 use fetchcontent * 🐧 add target to download a Linux version of CMake * 🔨 fix dependency * 🚨 fix includes * 🚨 fix comment * 🔧 adjust flags for GCC 11.0.0 20210110 (experimental) * 🐳 user Docker image to run CI * 🔧 add target for Valgrind * 👷 add target for Valgrind tests * ⚗️ add Dart * ⏪ remove Dart * ⚗️ do not call ctest in test subdirectory * ⚗️ download test data explicitly * ⚗️ only execute Valgrind tests * ⚗️ fix labels * 🔥 remove unneeded jobs * 🔨 cleanup * 🐛 fix OCLint call * ✅ add targets for offline and git-independent tests * ✅ add targets for C++ language versions and reproducible tests * 🔨 clean up * 👷 add CI steps for cppcheck and cpplint * 🚨 fix warnings from Clang-Tidy * 👷 add CI steps for Clang-Tidy * 🚨 fix warnings * 🔧 select proper binary * 🚨 fix warnings * 🚨 suppress some unhelpful warnings * 🚨 fix warnings * 🎨 fix format * 🚨 fix warnings * 👷 add CI steps for Sanitizers * 🚨 fix warnings * ⚡ add optimization to sanitizer build * 🚨 fix warnings * 🚨 add missing header * 🚨 fix warnings * 👷 add CI step for coverage * 👷 add CI steps for disabled exceptions and implicit conversions * 🚨 fix warnings * 👷 add CI steps for checking indentation * 🐛 fix variable use * 💚 fix build * ➖ remove CircleCI * 👷 add CI step for diagnostics * 🚨 fix warning * 🔥 clean Travis |