VS 2015 does not accept std::string::npos in a constant expression (C2131),
and the constant is only used at run time.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* json_view: source positions in materialize() with JSON_DIAGNOSTIC_POSITIONS
With JSON_DIAGNOSTIC_POSITIONS, materialize() now gives every value the
start_pos()/end_pos() that basic_json::parse() would give it, so exceptions
about materialized values name the same bytes (#5864).
- json_sax_dom_parser takes the lexer type as a defaulted template
parameter; materialize() passes position_source, which reports the
positions it derives from the index. diagnostic_positions treats npos as
"no position" and keeps ranges all-or-nothing.
- Strings with escapes keep the offset of their opening quote plus one in
`next` (only with the macro); materialize() scans from there to the closing
quote. The end of an array or object is found by scanning from the end of
its last element over whitespace, comments and a trailing comma.
- In editable documents, values written by edits, arrays and objects whose
elements changed, and containers whose last element has no position get
none. Images keep the flag is_new, and load() checks that `next` of a
string is within the text.
Nothing changes on the parse path without the macro. The view's own
exceptions still carry no byte range, so the tests that compare them with
those of materialized values skip it under the macro, as they skip the path
under JSON_DIAGNOSTICS.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Indent comment in unit-json_view_from_json.cpp
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address review: clearer position code and docs
- position_source gets clear(); set_scalar() returns early for new values,
switches on node_kind, and uses no local copies; the one position_source
per materialize() call is documented
- no npos + 1 for containers without a position; a comment on why the end
of a container is reported as its closing bracket
- less duplication under #if in materialize() and the builder
- diagnostic_positions sets npos explicitly
- docs: positions of basic_json values after changes, positions in images
depend on how the document was parsed, when next is set, and the cost of
positions while parsing
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Share the comment scanners of the view builder with materialize()
skip_line_comment() and find_block_comment_end() in scan.hpp are used by
the builder's comment() and by position_source::container_end(), which
had its own copy.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address review: if/else for container positions, clearer next field
- the opening bracket of a container sets positions with an if/else instead
of a ternary, and the frame takes its start from positions
- the closing bracket of a container is found in nested if/else; the parent's
last_end is taken from positions, so it is npos whenever no range is set
- architecture.md: the next field reads "or 0 if it was not; all other
values: 0" instead of "else 0; otherwise 0"
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix GCC unused-result and clang-tidy raw-string findings in unit-json_view_positions.cpp
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix self-aliasing in json_pointer::operator/= and patch_inplace
json_pointer::operator/=(const json_pointer&) passed iterators into its
own token vector to insert() when a pointer was appended to itself
(p /= p), which is undefined behavior; libstdc++ produced "/a/b//" for
"/a/b". The pointer now appends a copy in that case.
basic_json::patch_inplace iterated the patch while changing *this, so
j.patch_inplace(j) read freed memory. It now applies a copy of the
patch when the patch is the value itself. A patch stored inside the
value remains undefined behavior and is documented as such; copying the
patch unconditionally would double the time of a large patch.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix JSON_DIAGNOSTICS test crash and clang-tidy hit in patch_inplace self-aliasing
The spare-capacity test reallocated non-empty elements through get_ref(), which leaves stale parent pointers under JSON_DIAGNOSTICS; reserve while the array is empty instead. Suppress performance-unnecessary-copy-initialization for the copy that is needed because *this is modified.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Detect a patch stored inside the patched value with JSON_DIAGNOSTICS
patch_inplace applied a copy only when the patch was the value itself. With JSON_DIAGNOSTICS the parent pointers make it cheap to also detect a patch nested inside the value, so that case is handled the same way. Without diagnostics it stays documented undefined behavior.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Silence misc-const-correctness for patch_aliases_this
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix clang-tidy findings in the operator/= self-append test
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
MSVC 2015 treats two function templates that differ only in a decltype
return type as a redefinition (C2995), which breaks all three Visual
Studio 2015 jobs on AppVeyor. Overload on the view's mode in the
parameter type instead; template argument deduction ignores the
read-only-to-editable conversion, so each view still selects the
overload of its own mode.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix clang-tidy findings in unit-json_view_from_json.cpp
Silence misc-use-internal-linkage on the macro-generated ADL functions,
reconcile -Weffc++ with readability-redundant-member-init, use auto for
get<T>() initializers, and use value() on the optional access.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Satisfy bugprone-unchecked-optional-access in unit-json_view_from_json.cpp
clang-tidy 22 also flags value(); check has_value() in the same expression.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
GCC 13 compiled the ?: selections of token_digits_1/2(), biased_distance()
and the batch's classification loop as branches. Their conditions are as
good as random, so dumping canada.json mispredicted about one branch per
float (callgrind --branch-sim: 220k per dump with GCC, 97k with Clang),
and the dump took 21-26 % longer than with Clang. The selections are now
masks (mask_if(), select_if()), which no compiler turns into a branch.
canada.json dump on the Haswell VPS (dprobe, best of 60, 3 interleaved
rounds): GCC 13 4.74-5.04 ms -> 3.98-4.24 ms, Clang 18 unchanged
(3.93-4.06 -> 3.82-3.98 ms); mispredictions per dump 220k -> 98k with GCC.
Instructions per dump: GCC 38.2M -> 40.2M, Clang 34.7M -> 35.0M. Apple
M1 Max, Apple clang 21: 0.5-1.8 % fewer cycles per dump in 4 of 4 pairs.
In bench_all, json_view's dump of canada.json against yyjson goes from
0.75-0.78 to 0.91 with GCC.
Output unchanged: dump() equals json::parse(text).dump() for 4e7 random
16/17-digit, %.17g and short tokens (singles and pairs) with GCC 13 on
x86-64 and 6e7 with Apple clang on arm64; unit-json_view_dump passes with
GCC 13 and with Clang under ASan/UBSan.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The test relies on a copied fifo_map restarting its timestamps. Copying a
basic_json does that only on the recursive path; with JSON_NO_THREAD_LOCAL
the copy is built member by member and keeps correct timestamps, so the
premise was missing and the test failed (ci_test_no_thread_local, MinGW
clang). Copy the object_t explicitly to create the state on every path.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
- unit-bson: initialize the SAX recorder's member (-Weffc++)
- unit-json_patch: build the reference document independently instead of
copying it (performance-unnecessary-copy-initialization)
- unit-json_view_builder: use node_kind::link instead of an out-of-range
enum cast (clang-analyzer-optin.core.EnumCastOutOfRange)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Spell out that numbers comparing equal produce no operation even if one is
zero and the other negative zero, in diff.md and features/json_patch.md.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Keep the value unchanged when operator[] with a JSON pointer throws
The non-const operator[](json_pointer) is documented with strong exception
safety, but get_unchecked created intermediate values while walking the
pointer and kept them when a later reference token failed, e.g.
json j; j["/a/99999999999999999999999"_json_pointer] threw 404 and left
{"a":[]}; padding an existing array or appending with "-" behaved alike,
and so did std::length_error/std::bad_alloc from a huge index.
get_unchecked now only walks existing values up to the last reference
token. The last token's key insertion or array growth is strongly
exception safe by itself. Any other change goes through
get_unchecked_create, which records the one change it makes to an existing
value (null converted, key inserted, array grown) in a RAII guard and
reverts it if a later step throws; everything else it creates lies below
that change. The guard needs no try/catch, so JSON_NOEXCEPTION builds are
unaffected. Successful calls create the same values as before.
Docs: state the guarantee in the exceptions list and version history of
operator[], fix the complexity of the pointer overloads of operator[] and
at, and warn that padding arrays up to an untrusted index allocates memory
proportional to the index (operator[].md, untrusted_input.md).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Make the strongly exception-safe pointer operator[] faster than before
The previous commit made get_unchecked strongly exception safe but slower
with GCC 16 when it creates values (up to +6 % instructions for appending
with "-"). This restructures it so that no benchmark row is slower than
before the strong guarantee, with GCC 16, Clang 22 and Apple Clang.
- get_unchecked walks existing values in one loop that inserts nothing.
The last reference token is resolved in that walk: "-" appends with
emplace_back(), an index grows the array with operator[], and a missing
key is inserted with emplace_hint() at the position the walk's
lower_bound() found, so the key is searched once.
- With std::less<string_t> or std::less<> as comparator, the walk checks
the key at the lower bound for equality instead of equivalence, which is
the same for these comparators and cheaper.
- get_unchecked_create (not inlined) handles all other changes. Its guard
swaps a null value with a fresh null value instead of copying it and
reverts the change in a non-inlined function, so a successful call pays
one compare for it. As every null value converted afterwards is a fresh
one, it is converted in place instead of assigning a new array or object.
- Keys are added to new objects with emplace(), which also replaces the
basic_json::operator[] call for ordered_json.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix rollback of pointer operator[] for unusual object and string types
Follow-up to the strong exception guarantee of the non-const
operator[](json_pointer):
- get_unchecked_create assumed that emplace() always inserts the key that
the walk did not find. For object types whose lookup and emplace()
disagree (e.g. a copied fifo_map), nothing was inserted, the rollback
erased the existing member, and the walk continued with a value of the
wrong type (assertion, type_error.305, or a crash). It now checks the
result of emplace(), arms the rollback only if a key was inserted, and
otherwise continues the walk with the member found, as basic_json's
operator[] does: such pointers throw out_of_range.404 and leave the value
unchanged, as before the strong guarantee.
- The rollback erased the inserted key by key, which converts string_t to
object_t::key_type and may allocate inside a noexcept function
(std::terminate on bad_alloc with custom key types). It now erases by the
iterator emplace() returned, and shrinks a grown array with erase() at
the end instead of resize(). The documentation states the remaining
requirement for custom template parameters.
- Creating values no longer calls string_t::begin(), so the non-const
operator[](json_pointer) compiles with string types that lack it (as the
alternative string type of the tests).
- Correct the @throw lists of get_unchecked and get_unchecked_create.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address review comments on operator[](json_pointer) rollback
- undo() shrinks the array with resize() instead of erase()
- comments say what "will be" done in get_unchecked_create
- tests: avoid a clang-tidy performance-unnecessary-copy-initialization
hit (unit-alt-string) and the word "access" in section names, which
Flawfinder flags as race/access
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Let the pointer rollback guard call undo() unconditionally
The check in the destructor and JSON_HEDLEY_NEVER_INLINE on undo() made no
measurable difference: instruction counts of j[ptr] are identical with
GCC 13 -O3 and within 0.4 % with clang, timings are within noise.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: new design for the documentation site
Apply the brand to the Material for MkDocs site: Inter and JetBrains Mono
served by the site (upright and real italic), brand colours, blue header band
with the syntax pattern, wordmark in the header, new home page (hero, cards,
animated feature tour), 404 page, polish for tabs, search, footer, focus
rings, tables and code blocks, and theme features (code copy, previous/next
footer, TOC follow, search suggestions).
Draw the number railroad diagram, the parser callback events and the iterator
illustrations as inline SVGs that follow the light and dark scheme. The
generator scripts are in docs/mkdocs/scripts/figures; the feature slideshow
(json-light.svg, json-dark.svg, replacing json.gif on the home page) is built
by docs/mkdocs/scripts/slideshow. Decision nodes in five flowcharts become
hexagons, which need far less room.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: add figures, table captions and a macro chooser
- Figures with numbered captions: SAX stepper, json_view layout and edit
stepper, exception map, JSON pointer and patch walkers, QA pipeline,
scaling and binary size charts, roadmap lanes, release timeline,
integration decision tree and namespace builder
- Table captions (nj-table-caption) on 9 pages
- Compile times: modules and PCH charts, filesystem section, re-measured
Figure 1, links between features/modules.md and compile_times.md
- Interactive macro chooser for features/arbitrary_types.md
- Link check: exclude URLs that answer the plugin with 400/401/419 or are
dead (customers.md)
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: macro chooser generates adl_serializer for non-default-constructible types
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: cross-link concept pages and API pages in See also sections
Add See also links in both directions between the concept pages (features,
home, integration) and the API pages, write API links in code font, give every
entry a comment after a dash, and show the callback event figure on the parser
callbacks page.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: address review comments on the new design
- Home: add a "Latest release" button
- Sponsors: update the list and group it by tier
- Features: list all feature pages on the index page
- Releases: add highlights per release, link the 4.0 roadmap, add the first commit to the timeline
- Exceptions: prefix all example messages with the exception name
- API: no code font for format names in the See also links of the to_/from_ pages
- Merge Patch: show the document before, the patch, and the result
- Replace "--" with typographic dashes, mark external links with an icon, center Mermaid diagrams
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* json_view: pointer tokens that cannot be array indices create objects
editable_document::set() chose an array for a null parent whenever the
token consisted of digits, so "01" threw parse_error.106 and the empty
token threw, while basic_json::operator[](json_pointer) has created an
object for such tokens since #5358. This broke
unit-json_view_edit_values on develop. Use the same RFC 6901 array index
test as basic_json.
Also re-indent the unflatten test that the astyle check flagged.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix the other red jobs on develop
- unit-bson: initialize recorder::events in its declaration (-Weffc++)
- unit-json_view_from_json: suppress -Wexit-time-destructors for the
serialize-enum tables and -Wfloat-equal for same()
- unit-pointer_access: @infer-ignore NULLPTR_DEREFERENCE for the
dereferences after get_ptr() that Infer reports
- json_view.hpp: IWYU no_include for the detail headers that the view
code uses through nlohmann/json.hpp
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix GCC warnings in the json_view from_json test
GCC (-Weffc++, -Wfloat-equal) rejected unit-json_view_from_json.cpp
(ci_test_standards_gcc (20)): initialize the members of shape and
with_names, and suppress -Wfloat-equal for GCC like for clang.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
- Size the buffer of decoded strings on its first growth from the input (a
quarter of it plus 16 bytes, at most 4 KiB) instead of 272 bytes, so that a
document with escapes does not grow it twice, with a zero-filled copy each
time.
- Allocate the node index of documents with an estimate of up to 1024 nodes
(was 256) together with the header, which saves one allocation. When the
estimate is too low (dense numbers), that block stays unused: at most 16 KiB
instead of 4 KiB. shrink_to_fit() cannot release it separately; the
documentation says so.
Fresh parse of a 2.5 KB minified tweet: 993 to 958 ns (-3.5 %), traverse and
select -3 %, with no row consistently slower (Apple M1 Max, 30 rounds, three
runs). Dense numeric documents parse about 9 % faster and keep up to 16 KiB
more memory.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: writing generic code for json_view
A "Writing generic code" section on the json_view page: why a function
taking json_view accepts no other view, the three signatures for
editable, read-only, or either kind of view (passed by value), why the
modes do not convert, is_json_view, and how a from_json() reads views
(constrained to json values and views; the macros and the library's
conversions do it already) with what differs then (duplicate keys,
JSON_DIAGNOSTICS paths).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: name the documents the generic view function accepts
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: views of read-only documents convert to editable views
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* json_view: the library's std conversions read views directly
The library's from_json() overloads (containers, maps, std::array, C
arrays, pairs, tuples, optional, valarray, forward_list, enums, binary,
paths, and scalars) accept views: their traits use the view's
basic_json type, and the few accesses that differ (get_ptr) go through
small helpers. from_json() of a view now finds them; only the probe
still looks up by ADL only. get<T>() of a view thus converts these types
without materialize(), with the same values and exceptions; from an
object, a repeated key keeps the last value, as parse() does.
The view keeps its direct paths for vectors of the types it converts
directly (faster than from_json()) and for containers of views; its
other vector and map paths are replaced by from_json(). For basic_json,
std::valarray and std::forward_list are read in one pass (40 % and 3.5 %
fewer instructions); all other conversions are unchanged.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* json_view: enable_if_not_view_t instead of '!' in the from_json() templates
astyle spaces out the template argument list when it contains a negation;
the alias spells the condition with negation<> and leaves the lines alone.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* json_view: compact the template parameter lists astyle had spaced out
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
get<T>() and get_to() of a view call T's from_json() with the view itself
when that from_json() accepts views but not an unrelated probe type, and
fall back to materialize() otherwise. The from_json() functions of the
NLOHMANN_DEFINE_TYPE_*, NLOHMANN_DEFINE_DERIVED_TYPE_* and
NLOHMANN_JSON_SERIALIZE_ENUM(_STRICT) macros accept basic_json values and
views; a derived type whose base reads only json reads the base part
through materialize(). The library's own from_json() overloads are not
called with views: from_json() of a view (and of the probe) only looks up
functions by ADL.
New public trait nlohmann::is_json_view<T>, and the member type
basic_json_type of basic_json_view. With duplicate keys, the members of
such a T are those the view's lookups find (the first ones).
bench_decode.cpp measures decoding into structs (4-6x faster than
materialize().get<T>() on twitter, citm_catalog, and synthetic records).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The view's vector code follows JSON_NO_SIMD, like the lexer and the float
writer, and skips the CPU check of SSSE3 when the code is compiled for
SSSE3 anyway (__SSSE3__), like the lexer. JSON_VIEW_NO_SIMD and
JSON_VIEW_USE_SSSE3 and their ABI tags _vns and _vssse3 are removed; they
were never released. JSON_NO_SIMD keeps its tag _ns.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The image format is little-endian only. Instead of throwing
type_error.320 at runtime, save_image(), load_image() and the members
save() and load() do not exist on a big-endian target, so a call fails to
compile, like set() on a read-only document. type_error.320 remains for
saving a discarded document.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* json_view: view_mode enum instead of bool Editable, node_kind for node::kind
basic_json_document and basic_json_view take a view_mode (read_only, the
default, or editable) instead of a bool: basic_json_document<json, true>
did not say what true means. The shortcut aliases are unchanged.
The kind of a node is a node_kind (the value_t values plus link, 10 as
before), which removes the static_cast<std::uint8_t>(value_t::...) of
every comparison. The node and image layouts are unchanged.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* json_view: node_kind in the node writers, spell out read_only, comment fixes
The builder and the edit functions take and store a node_kind instead of
casting a value_t; the numbering assertions and the comments now speak of
node_kind, and the link kind notes that it follows value_t::discarded.
The read-only aliases name view_mode::read_only like the editable ones.
view_mode.md says the mode is a compile-time property of the type.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* json_view: views of read-only documents convert to editable views
A read-only document has no edits, so a view of it reads correctly as a
view of an editable document before its first edit. The implicit
conversion lets a function taking a json_editable_view accept the views
of both modes. There is no conversion the other way: a read-only view of
an editable document would read wrong values after the next edit.
The mode comparisons in the cross-mode operator==/!= use an
is_other_mode trait, which astyle keeps compact.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: fix BON8 documentation gaps found in the format review
Verified each claim with small C++ probes before writing it.
- index.md: BON8 serialization is also incomplete for invalid UTF-8.
- bon8.md: document duplicate keys (last wins), the error ids for
malformed input incl. 0xF5..0xF7 + 0x80..0xBF, exact string
termination wording, wide integers and discarded values in the
unsupported list, and title two untitled admonitions.
- exceptions.md: to_bon8 in type_error.316, BON8 in parse_error.112
(with example) and .113, wide integers in out_of_range.407.
- binary_values.md: bytes 0..39 take one byte in BON8.
- from_bon8.md: note on strict UTF-8 and duplicate keys.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address review of the BON8 documentation
Restore the correct statement that marker bytes cannot begin a UTF-8 character (0x80..0xBF occur in valid UTF-8 as continuation bytes), and reword two sentences.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: drop the discarded-value remark from the BON8 page
Discarded values are not mentioned on the individual format pages.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: do not list invalid UTF-8 strings as a BON8 serialization gap
JSON text must be valid UTF-8, and to_bon8 rejects ill-formed strings by design (type_error.316), as documented on the BON8 page. Restore the original cell and realign the table.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Read ND-arrays of half floats and enforce the char limit in ND-arrays
BJData N-dimensional arrays with element type 'h' (float16) failed to
read with a misleading parse_error.112, because the reader's dtype name
table lacked 'h'. It now maps to "half", the JData name. The writer still
does not pack "half"; such an object is written as a plain object and
reads back equal.
BJData limits 'C' (char) to 0..127 in Draft 2 and Draft 3, and the 1-D
reader enforces this, but ND-arrays with _ArrayType_ "char" wrote values
up to 255 as 'C' elements and read them back without a check. The writer
now only packs values in 0..127 (others fall back to a plain object that
round-trips), and the reader rejects 'C' elements above 127 with
parse_error.113 like the 1-D reader.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: correct and complete the BJData documentation
Verify the BJData chapter against the library and the Draft 2/3/4
specifications and fix what did not match:
- bjdata.md: correct the list of unused markers (N, not Z), the ND-array
example (3x2, not 2x3), the conditions under which an object is packed
as an ND-array (exactly three keys, row vectors, float-only elements for
single/double, char 0..127, byte only in Draft 3), the binary table row
and notes (which flag/draft combinations write [$B), the "$" type
restriction (Draft 3, Z is excluded too), the H marker in the
deserialization table, and the claim of a complete deserialization
mapping. Document the supported drafts, that Draft 4 (SoA containers,
extension type E) and column-major ND-arrays are rejected, ND-arrays of
half floats, duplicate object keys, and link the Draft 3 specifications
instead of master.
- Add an API page for basic_json::bjdata_version_t with an example, and
link it from the nav, the API index and to_bjdata.
- Update binary_formats/index.md and binary_values.md, which still claimed
that BJData has no binary type, and fix a wrong byte comment there.
- Document parse_error.115 and the out_of_range.408 cases for from_bjdata,
the precise other_error.502 condition for to_bjdata, and reword the note
on truncated unsigned integers in exceptions.md (only ND-array elements
were truncated before 3.13.0).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Regenerate single_include and apply astyle
Result of make amalgamate (including make pretty) for the preceding changes.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address review of the BJData documentation
The $ type restriction exists since Draft 2 (Draft 3 added B), $ requires a count, ND char above 127 is rejected on read, and the new API page is listed in alphabetical order in the nav.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: format tables, use the Oxford comma
Reformat the tables changed by this PR (BJData serialization and
deserialization mappings, completeness and binary-value tables) so all
rows are aligned, and add the Oxford comma to lists of three or more
items in the added text.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Write and read the inner length of BSON binary subtype 0x02
The BSON specification defines the payload of the "binary (old)" subtype
0x02 as an int32 followed by that many bytes. to_bson wrote the plain
payload instead, so decoders such as pymongo and libbson-based readers
rejected the bytes, and from_bson returned the inner length as part of the
payload (documented as lenient behavior in gh-5333).
The writer now emits the outer length (payload + 4), the subtype, the inner
int32 holding the payload size, and the payload. The reader requires an
outer length of at least 4 and an inner length equal to the outer length
minus 4 (the cases the bson-corpus lists as decode errors), reports parse
error 112 otherwise, and returns only the payload. Both directions change
together so that the round trip keeps working.
The single-reservation fast path for scalar entries declines subtype 0x02
because it has no room for the inner length.
This is a deliberate exception to the opt-in rule for behavior changes:
the previous output was invalid BSON.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Check the declared BSON document size before reading its content
The BSON reader compared a document's declared size with the number of bytes
read only when it reached the terminator. The declared size never bounded
the reads, so a SAX handler saw the keys and values of a document whose size
was too small before parse error 112 was reported.
The container frame now stores the position where the document ends
according to its declared size. open_bson_document rejects a size below 5
(size prefix plus terminator) and an embedded document that does not end
before the terminator of its enclosing document; the element loop rejects an
element whose type byte leaves no room for the name terminator and the
document terminator. The terminator check stays and still catches a declared
size that is too large. No input that was accepted before is rejected now;
documents with a wrong size are reported earlier, with fewer SAX events, a
different byte position and, in three cases, a different message. The error
id (parse_error.112) is unchanged and strict=false still does not skip the
check.
The tests that asserted the old messages and positions are updated, with the
outer sizes of two inputs corrected so that they still exercise the embedded
document. New tests cover the SAX events, a size below 5, and an embedded
document that fits exactly.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Document BSON subtype 0x02, document size checks, and lenient reads
Describe the inner length of binary subtype 0x02 that to_bson now writes and
from_bson checks and removes. Add the document size checks (parse_error.112,
not affected by strict=false) to bson.md and from_bson.md, and extend the
list of lenient reads with the UUID and MD5 payload lengths (not checked),
duplicate keys (last value wins), embedded NUL bytes in strings, and the
ignored trailing bytes with strict=false. State that the members of an object
are written in iteration order.
Correct the statement that element names are never validated on read: they
are checked and sanitized like string values by the error handler. Add the
BSON message to the examples of out_of_range.415 and type_error.321, the
subtype limit of 255 to the binary values page, CBOR to the formats that use
byte_container_with_subtype, and fix a typo in the incomplete mapping box.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Regenerate single_include and apply astyle
Result of make amalgamate (including make pretty) for the preceding changes.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address review of the BSON documentation
Describe exactly when a too small document size is detected and note that subtype 2 data written by earlier versions is not readable.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: use the Oxford comma, drop discarded-value remarks
Add the serial comma in a list added by this PR and remove the BSON
example of type_error.321 (with the type_error.317 remark) so the docs do
not mention discarded values for BSON; restore the original singular
heading of that section. No tables were changed by this PR.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix -Weffc++ in the BSON recorder test and a clang-tidy use-auto hit
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix clang-tidy readability-redundant-member-init in BSON test
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Validate the optimized-container type marker for count 0 in UBJSON and BJData
The '$' type marker of an optimized container was only checked when an
element was read, so containers with count 0 accepted any marker:
`[$x#i0`, `{$x#i0` and `[$E#i0` parsed as empty containers although a
non-empty container with the same marker is rejected and the UBJSON
specification requires a legal marker after '$'.
get_ubjson_size_type() now checks the marker right after '$' against the
allowed set (UBJSON: Z N T F i U I l L d D H C S [ {; BJData: B U i u I m
l M L h d D C, with the existing exclusion of T F Z N S H [ {) and reports
parse_error.112 "marker ... is not a permitted optimized array type".
Input that was accepted and is now rejected: an optimized container with
count 0 and an invalid '$' type marker. Valid input is unaffected. Two
existing tests that used an invalid marker incidentally ("[$#" and a
negative-count case) were adapted.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Correct and complete the UBJSON documentation
Fix statements that disagree with the behavior (checked with probes):
- `Z` is null; the unused marker is `N`.
- `$` requires `#`; use_type with use_size=false throws other_error.502 only
for non-empty arrays, objects and binary values.
- NaN and infinity are written as IEEE 754 float64 and read back unchanged;
the UBJSON specification encodes infinity as null.
- `H` is documented (semantics, parse_error.115, out_of_range.406); the
"complete mapping" claim is qualified.
- use_type "same type" means same marker.
- Long Z/T/F arrays lose only the `$` annotation; the `#` count and the
element markers are kept.
- from_ubjson lists the 112, 113, 115, 406 and 408 causes that occur for
UBJSON, without the BJData-only ND-array wording.
- Duplicate object keys keep the last value; the `$` type marker is
validated even for count 0.
- Smaller fixes: float32 row, "map" -> "object", https link, exception 110
strict case, to_ubjson history of discarded values.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Regenerate single_include and apply astyle
Result of make amalgamate (including make pretty) for the preceding changes.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address review of the UBJSON documentation
Clarify that empty containers carry no $ annotation of their own and note that a counted N object is not supported.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: format tables, use the Oxford comma, drop discarded-value remarks
Reformat the two UBJSON mapping tables that this PR changed (uniform column
widths, padded cells), use the serial comma in lists of three or more in the
added text, and restore the original type_error.321 bullet in to_ubjson.md.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix narrowing of the optimized-container count in BJData and UBJSON tests
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Document MessagePack ext types and fix related doc errors
Add an "Extension types (ext, fixext)" section to the MessagePack page:
ext/fixext are read as binary, the signed ext type is exposed as an
unsigned subtype 0..255 (-1 reads as 255), Timestamp is not interpreted,
subtypes are always written as ext/fixext, and subtypes above 255 throw
out_of_range.415. Add the 0xC1 row to the deserialization table, note
that duplicate keys keep the last value, and correct the NaN/Infinity
explanation, the "31 bytes / 15 elements" wording, the ill-formed UTF-8
sentence, and the generic 0xC1-key error message.
Add fixext16 and the unsigned subtype mapping to binary_t.md and
binary_values.md, reword "maximum of the MessagePack ext type" in
to_msgpack.md and exceptions.md, and mark MessagePack deserialization
as incomplete (non-string keys) in the binary formats overview.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address review of the MessagePack documentation
Name parse_error.113 for a 0xC1 map key, move the duplicate-key sentence out of the ext section, and keep the 0xC1 row in marker order.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Skip CBOR tags in front of map keys and validate indefinite text chunks
A tag in front of a text string map key was rejected with parse_error.113
even with cbor_tag_handler_t::ignore or store, although tags in front of
values are handled. The tags are now skipped iteratively before the key is
read; the handler error still rejects them.
The chunks of an indefinite-length text string were concatenated before
the single UTF-8 check, so a code point split over two chunks, or an
invalid chunk, was accepted. RFC 8949, Section 3.2.3 requires every chunk
to be valid on its own. With error_handler_t::strict, replace and ignore
each chunk is now checked; keep is unchanged. The chunks are appended with
append(ptr, len), so StringType needs nothing new.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: correct and extend the CBOR documentation
Fix the negative-integer rows (they used MessagePack boundaries), the
number_integer range of the 0x1B row, and the half-precision entry in the
list of unused types; correct the NaN history (0xF9 since 3.8.0) and the
tagged-item wording (only the tag directly in front of a byte string is
stored, and only in the 0xD8..0xDB form). Document that to_cbor is not
RFC 8949 deterministic, that duplicate keys are accepted (last wins),
tags in front of keys, per-chunk UTF-8 validation, the exceptions
parse_error.113, out_of_range.406/408 of from_cbor, the pointer and
C array input traps, the partially written output on exceptions, and the
store/ignore results in the binary values example. Drop the claim that
the CBOR reader needs StringType::append(const StringType&) and
BinaryType::insert(pos, first, last).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Regenerate single_include and apply astyle
Result of make amalgamate (including make pretty) for the preceding changes.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Address review of the CBOR changes
Cite RFC 8949 Section 5.3.1 for duplicate keys, describe the out_of_range.408 condition precisely, rewrap a long line, and update the stale get_string comment (per-chunk UTF-8 check). single_include regenerated.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: drop discarded-value remarks
Remove the mention that a discarded value cannot be serialized to CBOR
(type_error.321) from the exception-safety and mapping sections. The
integer table and all added prose already follow the table-format and
Oxford-comma rules.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* create objects for tokens that cannot be array indices (#5357)
When operator[](json_pointer) traverses a level that does not exist yet,
the null value is turned into an array or an object depending on the
reference token. The check only tested whether all characters are digits,
so tokens that can never be a valid array index selected an array and
then failed:
- "01" (and any other token with a leading '0') threw parse_error.106
- the empty token threw out_of_range.404
Both tokens are valid object keys, and both work when the level already
exists as an object, so creating the level changed the outcome. Test the
token against the RFC 6901, Sect. 4 grammar for array indices instead, so
that such tokens create an object. This only affects pointers that threw
before.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Clarify that the token 0 is a valid array index in operator[] docs
The note described valid array indices as "a nonempty sequence of digits
without a leading 0", which read literally excludes the token "0" itself,
contradicting both the code and the /foo/0/0/0 example in the next
sentence. Spell out the RFC 6901 grammar instead.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Throw out_of_range.411 for JSON Patch add into a null parent (#5917)
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MKJf48vnGtTE25KscmjfaW
Signed-off-by: Niels Lohmann <niels.lohmann@gmail.com>
* docs: make the diff() and patch() documentation accurate; add tests
Document the exceptions to the diff round-trip invariant (NaN, discarded
values, number types), the path parameter, non-minimal patches, array and
object operation order, the ordered_json fallback, binary values in
serialized patches, the real complexity, duplicate member names in
operations, test on missing paths, and the exact out_of_range.403 message.
Add tests for these behaviors.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <niels.lohmann@gmail.com>
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Fix inaccurate or missing statements about unflatten (third round-trip
failure, empty token, typo), value (type_error.306 before pointer lookup),
contains (pointer overload), parent_pointer and operator<< (bad_alloc),
out_of_range.410 example, template parameter requirements (resize(n)), and
document the URI fragment form and non-unique member names of RFC 6901.
Add tests for the unflatten and value() cases.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Flawfinder 2.0.20 has an --exclude PATTERN option. Scan `./` and exclude
`*/single_include/*` so that new top-level directories are covered
automatically. The scanned file set is identical to the explicit
directory list.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
- Name std::hash only once the object type has emplace_new_key: VS 2015
static-asserts on std::hash of a type without one (alt_string) instead of
reporting a substitution failure (C2338).
- Replace the std::copy calls on frames by loops: VS 2015 reports raw
pointers handed to std::copy as C4996, an error under /WX.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: byte layout sections, glossary, and QA page updates
- Move "Byte layout" of each binary format to its own section before
"Serialization" and describe the idea of the format.
- Glossary: add LLDB and other terms used throughout the documentation.
- quality_assurance: correct Clang Static Analyzer and CMake entries; add
CodeQL, Semgrep, Flawfinder, CIFuzz, OpenSSF Scorecard, REUSE, Meson/SwiftPM,
and reproducibility checks.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: drop unused length colour from CBOR and MessagePack byte layout legends
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Document clang-tidy exclusions and drop unneeded ones
Re-enable every excluded check in a run of clang-tidy 22 over all tests
and remove the seven that never fire (cert-err58-cpp,
clang-analyzer-webkit.NoUncountedMemberChecker,
cppcoreguidelines-virtual-class-destructor, google-readability-function-size,
hicpp-function-size, hicpp-no-assembler, readability-function-size).
Document the reason for every remaining exclusion at the top of the file.
Exclude in advance the checks that only exist in clang-tidy 23 and are
noise here (bugprone-signed-bitwise, *-explicit-constructor,
readability-trailing-comma, modernize-use-string-view), plus
modernize-use-integer-sign-comparison, which needs C++20 std::cmp_less
and flags deliberate casts.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix clang-tidy findings in documentation examples and replace redundant aliases
Apply clang-tidy fixes to the documentation examples:
- performance-avoid-endl: std::endl -> '\n'
- readability-qualified-auto: auto -> auto* / const auto*
- performance-for-range-copy: copy -> const reference
- google-readability-casting: C-style cast -> static_cast
Replace redundant aliases (readability-redundant-qualified-alias) such as
"using json = nlohmann::json;" with "using nlohmann::json;" in the
documentation, README, examples, tests, and benchmarks. Aliases at class
scope and aliases of other types (with_base_class_t etc.) are kept.
No changes to the library or its public API. All changed examples were
compiled and run; their output matches the stored .output files (except
max_size and meta, which depend on the platform and compiler).
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Format doc examples with the pinned astyle
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Docs: fill documentation gaps found in user questions
Add pages on class-hierarchy/smart-pointer/variant/chrono recipes, working
without exceptions, and large, streamed and partial input. Extend the FAQ
(23 entries, Known bugs first), parse error documentation (empty-input
checklist, line/column, diagnostic positions), element access, conversions,
creating values, ADL troubleshooting, memory, and build/distribution notes.
Add 14 examples with generated output. Correct the statement that parse errors
cannot be recovered from with -fno-exceptions.
The topics come from recurring questions on Stack Overflow, Reddit, Hacker
News, YouTube, and the project's GitHub discussions and issues.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* docs: declare range-for lists in variables to keep astyle quiet
astyle reformats a braced init list placed directly in a range-for
header; declare the list first so check_amalgamation passes.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Niels Lohmann
