When a parser callback rejects a value, the placeholder stored for it has to
be removed from its parent again. remove_discarded_value() found it by
scanning the parent from the beginning, so filtering a container cost one
scan per rejected member - quadratic in the number of members of a single
container.
A rejected value can only ever be the one most recently added to its parent:
the last element of an array, or the placeholder key() stored under the
current key in an object. Record that key alongside the existing
key_keep_stack, and for a container record it again alongside ref_stack so
end_object()/end_array() can find it in the parent. Removal is then O(1) for
an array and O(log n) for an object, and finding nothing there means nothing
was stored, so there is nothing to remove.
The key for a container is read before handle_value() may consume it, so it
is also correct when the callback rejects the container at its start event
and it never reaches its parent at all.
Discarding half the members of one object, before -> after:
members value rejected container rejected at start
16 000 392 ms -> 3.7 ms 803 ms -> 8.2 ms
64 000 6238 ms -> 14.6 ms 12651 ms -> 32.2 ms
128 000 25339 ms -> 30.6 ms
Results are unchanged: 48 000 randomized documents parsed under 12 different
filtering callbacks - covering duplicate keys, empty keys, rejected keys and
containers rejected at both their start and end events - produce byte
identical output before and after.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
The diagnostic position of a string value was derived by subtracting the
parsed value's length from the end position. Escape sequences make the
source token longer than the value it parses to, so the reported start
position landed inside the string, one byte off per escape sequence:
input: {"a":"\n\n\n\n\n\n"}
start_pos() == 11, so the reported range covered n\n\n\n"
instead of the documented "\n\n\n\n\n\n"
This contradicts the documented behavior of start_pos(), which is the
position of the opening quote, and it also corrupted the "(bytes N-M)"
part of JSON_DIAGNOSTICS exception messages. Strings with multi-byte
UTF-8 but no escapes were unaffected, which is why this went unnoticed.
Record the offset of the token in the lexer when it starts scanning and
use that, instead of reconstructing it from the parsed value. Booleans,
null and numbers already reported correct positions and are unchanged.
The new lexer member and accessor are compiled only when
JSON_DIAGNOSTIC_POSITIONS is enabled, which is already part of the ABI
tag, so the default build is unaffected.
Signed-off-by: Niels Lohmann <mail@nlohmann.me>
* Add implementation to retrieve start and end positions of json during parse
* Add more unit tests and add start/stop parsing for arrays
* Add raw value for all types
* Add more tests and fix compiler warning
* Amalgamate
* Fix CLang GCC warnings
* Fix error in build
* Style using astyle 3.1
* Fix whitespace changes
* revert
* more whitespace reverts
* Address PR comments
* Fix failing issues
* More whitespace reverts
* Address remaining PR comments
* Address comments
* Switch to using custom base class instead of default basic_json
* Adding a basic using for a json using the new base class. Also address PR comments and fix CI failures
* Address decltype comments
* Diagnostic positions macro (#4)
Co-authored-by: Sush Shringarputale <sushring@linux.microsoft.com>
* Fix missed include deletion
* Add docs and address other PR comments (#5)
* Add docs and address other PR comments
---------
Co-authored-by: Sush Shringarputale <sushring@linux.microsoft.com>
* Address new PR comments and fix CI tests for documentation
* Update documentation based on feedback (#6)
---------
Co-authored-by: Sush Shringarputale <sushring@linux.microsoft.com>
* Address std::size_t and other comments
* Fix new CI issues
* Fix lcov
* Improve lcov case with update to handle_diagnostic_positions call for discarded values
* Fix indentation of LCOV_EXCL_STOP comments
* fix amalgamation astyle issue
---------
Co-authored-by: Sush Shringarputale <sushring@linux.microsoft.com>
* Add versioned inline namespace
Add a versioned inline namespace to prevent ABI issues when linking code
using multiple library versions.
* Add namespace macros
* Encode ABI information in inline namespace
Add _diag suffix to inline namespace if JSON_DIAGNOSTICS is enabled, and
_ldvcmp suffix if JSON_USE_LEGACY_DISCARDED_VALUE_COMPARISON is enabled.
* Move ABI-affecting macros into abi_macros.hpp
* Move std_fs namespace definition into std_fs.hpp
* Remove std_fs namespace from unit test
* Format more files in tests directory
* Add unit tests
* Update documentation
* Fix GDB pretty printer
* fixup! Add namespace macros
* Derive ABI prefix from NLOHMANN_JSON_VERSION_*
* Make exception context optional
Change exception context parameter to pointer and replace context with
nullptr where appropriate.
* Support escaping other string types
* Add string concatenation function
Add variadic concat() function for concatenating char *, char, and
string types.
* Replace string concatenations using + with concat()
* Template json_pointer on string type
Change json_pointer from being templated on basic_json to being
templated on string type.
* Add unit test for #3388Closes#3388.
* Fix regression test for #2958
* Add backwards compatibility with json_pointer<basic_json>
* Update json_pointer docs
* Allow comparing different json_pointers
* Update version numbers
Niels Lohmann
Angadi56
Michael Valladolid
Sushrut ShringarputaleandSush Shringarputale
Raphael Grimmandbarcode
Florian Albrechtskirchinger
Niels Lohmann
aodq
chenguoping
Michael Reilly