mirror of
https://github.com/nlohmann/json.git
synced 2026-11-15 16:16:42 +00:00
Niels Lohmann
99adff001d
* Make unflatten() linear in the length of the keys
unflatten() decides in a first pass which prefixes of the keys denote arrays
(those that have a reference token 0 among their children), and looks up
prefixes again in the second pass. A prefix was identified by hashing it, and
by comparing it byte by byte with the entry in the set, from its beginning.
A key with n tokens therefore cost n^2/2 bytes: 100,000 tokens "/0" took
2.4 s (3.12.0: 5 ms), and so did "/0" followed by "/1" tokens, because every
new level looks its prefix up. Keys in the order of an unsorted object type
(ordered_json) defeated the shortcut for keys that share a beginning with the
previous one: four keys were enough to be quadratic.
Prefixes of less than 64 bytes are kept in a hash table and compared as
before, which costs a bounded number of bytes. Longer prefixes are kept in a
tree whose nodes are looked up by their parent and the text of one token. The
node of a prefix is found by walking the tokens of the key from the root, so
all prefixes of a key together cost one pass over the key, and no prefix is
compared as a whole. The nodes along the key walked last are kept, so keys
that share a beginning do not walk it again. The array/object decision and
the independence from the order of the keys are unchanged.
The hash function is seeded with a value derived from the address of a local
variable, which address space layout randomization changes between
processes, so keys cannot be crafted to collide in advance: with the fixed
constants, 20,000 keys of 16 + 2 bytes with the same hash took 0.5 s (random
keys, and 3.12.0: 7 ms). Where addresses are not randomized, or when the seed
is known, k short prefixes can still collide and cost k^2 / 2 comparisons of
fewer than 64 bytes. Results do not depend on the seed.
Measurements (Apple clang -O2, arm64), unflatten() of one key with n tokens:
tokens "/0" n=25k n=50k n=100k n=200k
before 175 ms 617 ms 2437 ms 9729 ms
after 4 ms 6 ms 12 ms 27 ms
3.12.0 1 ms 3 ms 5 ms 10 ms
"/0" then "/1" n=25k n=50k n=100k n=200k
before 153 ms 603 ms 2436 ms 9678 ms
after 1 ms 3 ms 5 ms 10 ms
3.12.0 1 ms 3 ms 5 ms 11 ms
ordered_json, 4 keys /0*m/a0, /10, /0*m/a1, /11
m=25k m=50k m=100k m=200k
before 397 ms 1402 ms 5191 ms 20723 ms
after 4 ms 7 ms 15 ms 33 ms
3.12.0 2 ms 3 ms 8 ms 17 ms
Ordinary documents (flatten() then unflatten(), instructions retired): within
0.3 % of before for canada, citm_catalog, twitter, jeopardy, mesh,
update-center and random from the benchmark corpus.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Fix -Wuseless-cast in array_prefix_set::reseed
uintptr_t is already uint64_t on LP64 Linux, so the explicit cast is
useless there; rely on the implicit (widening) conversion instead.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Throw type_error.313 when an unflatten key is a prefix of an earlier key
With an object type that keeps the insertion order (ordered_json), a key
such as "/a" can come after "/a/b". The final assignment then replaced the
object built for "/a/b" without an error, so {"/a/b":2,"/a":1} gave
{"a":1}. json never saw this because std::map visits the shorter key first.
Throw type_error.313 if the target of the final assignment is no longer
null, as json does for the same members. A null value keeps the container,
so {"/a/b":2,"/a":null} gives {"a":{"b":2}} for both object types.
Fixes #6105.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
---------
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
Documentation
Generate documentation
Note on documentation: The source files contain links to the online documentation at https://json.nlohmann.me.
This URL provides the most recent documentation and also applies to previous versions. Documentation for deprecated
functions is not removed; instead, it is marked as deprecated.
If you want to view the documentation for a specific tag or commit hash, you can generate it locally as follows (example
using tag v3.12.0):
git clone https://github.com/nlohmann/json.git
cd json
git checkout v3.12.0
make install_venv serve -C docs/mkdocs
Open http://127.0.0.1:8000/ in your browser. Replace any URL in the source code that points to
https://json.nlohmann.me with http://127.0.0.1:8000 to view the documentation for the selected tag or commit hash.