Replace std::equal with an explicitly-bounded loop in the move prefix check

The three-iterator std::equal(first1, last1, first2) form has no
explicit end iterator for the second range, which a static analyzer
(Flawfinder, CWE-126) flags as a potential over-read even though the
preceding size comparison already guarantees the second range is long
enough. Rather than argue the point, make the bound visible in the code
itself via an explicit loop -- every access to ptr.reference_tokens is
now guarded by the same index the loop condition bounds against
from_size.

(The C++14 four-iterator std::equal(first1, last1, first2, last2) form
was tried first as a more minimal fix, but this codebase targets C++11
and that overload is not safely usable under -std=c++11 with all
supported standard library implementations.)

Signed-off-by: Niels Lohmann <mail@nlohmann.me>

docs/mkdocs/docs/api/basic_json/patch.md

@@ -34,6 +34,10 @@ Strong guarantee: if an exception is thrown, there are no changes in the JSON va
   ("add", "remove", "move")
 - Throws [`out_of_range.411`](../../home/exceptions.md#jsonexceptionout_of_range411) if an "add" operation's target
   location has a parent that is neither an object nor an array.
+- Throws [`out_of_range.413`](../../home/exceptions.md#jsonexceptionout_of_range413) if a "remove" operation's target
+  location has a parent that is neither an object nor an array.
+- Throws [`out_of_range.414`](../../home/exceptions.md#jsonexceptionout_of_range414) if a "move" operation's "from"
+  location is a proper prefix of its "path" location.
 - Throws [`other_error.501`](../../home/exceptions.md#jsonexceptionother_error501) if "test" operation was
   unsuccessful.
 
@@ -75,3 +79,7 @@ is thrown. In any case, the original value is not changed: the patch is applied
 - Added in version 2.0.0.
 - Added [`out_of_range.411`](../../home/exceptions.md#jsonexceptionout_of_range411) and stopped relying on an internal assertion when an "add" operation's
   target location has a non-object/non-array parent in version 3.13.0.
+- Added [`out_of_range.413`](../../home/exceptions.md#jsonexceptionout_of_range413) and stopped silently ignoring a "remove" operation whose target
+  location has a non-object/non-array parent in version 3.13.0.
+- Added [`out_of_range.414`](../../home/exceptions.md#jsonexceptionout_of_range414) and rejected a "move" operation whose "from" location is a proper
+  prefix of its "path" location instead of silently producing a corrupted result in version 3.13.0.

docs/mkdocs/docs/api/basic_json/patch_inplace.md

@@ -30,6 +30,10 @@ No guarantees, value may be corrupted by an unsuccessful patch operation.
   ("add", "remove", "move")
 - Throws [`out_of_range.411`](../../home/exceptions.md#jsonexceptionout_of_range411) if an "add" operation's target
   location has a parent that is neither an object nor an array.
+- Throws [`out_of_range.413`](../../home/exceptions.md#jsonexceptionout_of_range413) if a "remove" operation's target
+  location has a parent that is neither an object nor an array.
+- Throws [`out_of_range.414`](../../home/exceptions.md#jsonexceptionout_of_range414) if a "move" operation's "from"
+  location is a proper prefix of its "path" location.
 - Throws [`other_error.501`](../../home/exceptions.md#jsonexceptionother_error501) if "test" operation was
   unsuccessful.
 
@@ -72,3 +76,7 @@ function throws an exception.
 - Added in version 3.11.0.
 - Added [`out_of_range.411`](../../home/exceptions.md#jsonexceptionout_of_range411) and stopped relying on an internal assertion when an "add" operation's
   target location has a non-object/non-array parent in version 3.13.0.
+- Added [`out_of_range.413`](../../home/exceptions.md#jsonexceptionout_of_range413) and stopped silently ignoring a "remove" operation whose target
+  location has a non-object/non-array parent in version 3.13.0.
+- Added [`out_of_range.414`](../../home/exceptions.md#jsonexceptionout_of_range414) and rejected a "move" operation whose "from" location is a proper
+  prefix of its "path" location instead of silently producing a corrupted result in version 3.13.0.

docs/mkdocs/docs/home/exceptions.md

@@ -933,6 +933,34 @@ BSON stores the length of documents, arrays, strings, and binary values in a sig
     [`to_bson`](../api/basic_json/to_bson.md) produced documents with negative length prefixes that
     [`from_bson`](../api/basic_json/from_bson.md) rejected.
 
+### json.exception.out_of_range.413
+
+A JSON Patch `remove` operation cannot be applied because the target location's parent is neither an object nor an array. Per [RFC 6902](https://datatracker.ietf.org/doc/html/rfc6902), a `remove` target must reference a member of an existing object or an element of an existing array; a primitive value (string, number, boolean, etc.) or `null` has no members or elements to remove.
+
+!!! failure "Example message"
+
+    ```
+    cannot remove value: the JSON Patch 'remove' target's parent is of type number, but must be an object or array
+    ```
+
+!!! note
+
+    This exception was added in version 3.13.0. Before that, this situation was silently ignored (the `remove` operation had no effect).
+
+### json.exception.out_of_range.414
+
+A JSON Patch `move` operation's `"from"` location is a proper prefix of its `"path"` location. Per [RFC 6902](https://datatracker.ietf.org/doc/html/rfc6902) (section 4.4), a location cannot be moved into one of its own children.
+
+!!! failure "Example message"
+
+    ```
+    cannot move value: 'from' path '/0' is a proper prefix of 'path' '/0/0'
+    ```
+
+!!! note
+
+    This exception was added in version 3.13.0. Before that, this situation could succeed with a corrupted result: for an array target, removing the "from" element before the "add" step shifted subsequent indices, so "path" silently re-resolved to a different element than intended.
+
 ## Further exceptions
 
 This exception is thrown in case of errors that cannot be classified with the

include/nlohmann/detail/input/lexer.hpp

@@ -149,11 +149,10 @@ class lexer : public lexer_base<BasicJsonType>
   public:
     using token_type = typename lexer_base<BasicJsonType>::token_type;
 
-    explicit lexer(InputAdapterType&& adapter, bool ignore_comments_ = false, bool discard_number_values_ = false) noexcept
+    explicit lexer(InputAdapterType&& adapter, bool ignore_comments_ = false) noexcept
         : ia(std::move(adapter))
         , ignore_comments(ignore_comments_)
         , decimal_point_char(static_cast<char_int_type>(get_decimal_point()))
-        , discard_number_values(discard_number_values_)
     {}
 
     // deleted because of pointer members
@@ -1280,58 +1279,6 @@ scan_number_done:
         // we are done scanning a number)
         unget();
 
-        // If the caller does not need the converted value (only whether the
-        // input is syntactically valid; see json_sax_acceptor/accept()), an
-        // unsigned/integer token can be reported without calling
-        // strtoull()/strtoll() at all, *provided* we can already tell from
-        // the digit count alone that the conversion cannot overflow 64 bits.
-        // Such tokens are always finite and are accepted unconditionally by
-        // the parser regardless of their actual value (parser::sax_parse_internal()
-        // never checks finiteness for value_unsigned/value_integer), so the
-        // classification below is all that is needed.
-        //
-        // A decimal number with up to 18 digits is always representable in
-        // both std::uint64_t and std::int64_t (18 nines is ~1e18, well below
-        // both UINT64_MAX ~1.8e19 and INT64_MAX ~9.2e18), so strtoull()/strtoll()
-        // could not have set errno to ERANGE for it. Numbers with more digits
-        // (rare in practice) fall through to the exact code below, unchanged,
-        // so their handling -- including reclassification to value_float when
-        // the value overflows 64 bits, and rejection when it is not even
-        // finite as a double -- is bit-for-bit identical to before this
-        // optimization.
-        //
-        // Note this reasons about std::uint64_t/std::int64_t, not about
-        // number_unsigned_t/number_integer_t (BasicJsonType's own, possibly
-        // narrower, template parameters -- e.g. std::uint32_t). That is fine
-        // *only* because discard_number_values is exclusively set by
-        // accept() (see json.hpp), and accept() always parses through the
-        // library's own json_sax_acceptor -- never a user-supplied SAX
-        // consumer -- whose number_unsigned()/number_integer()/number_float()
-        // callbacks unconditionally discard their argument and return true.
-        // So for every caller that can reach this branch, neither the token
-        // classification below nor the eventual (possibly narrowed, and on
-        // this fast path left stale/unset) value_unsigned/value_integer is
-        // ever consulted -- an unsigned/integer token is accepted outright,
-        // and even a >18-digit token that this fast path deliberately falls
-        // through for is, once reclassified to value_float, still finite
-        // (and thus accepted) for any digit count that fits in number_unsigned_t
-        // or number_integer_t regardless of that type's width. If this
-        // function is ever taught to run with discard_number_values true for
-        // a caller that *does* read the converted value, this reasoning (and
-        // the fast path below) would need to be revisited.
-        if (discard_number_values)
-        {
-            constexpr std::size_t safe_digit_count = 18;
-            if (number_type == token_type::value_unsigned && token_buffer.size() <= safe_digit_count)
-            {
-                return token_type::value_unsigned;
-            }
-            if (number_type == token_type::value_integer && token_buffer.size() - 1 <= safe_digit_count)
-            {
-                return token_type::value_integer;
-            }
-        }
-
         char* endptr = nullptr; // NOLINT(misc-const-correctness,cppcoreguidelines-pro-type-vararg,hicpp-vararg)
         errno = 0;
 
@@ -1446,7 +1393,8 @@ scan_number_done:
     */
     char_int_type get()
     {
-        advance_position();
+        ++position.chars_read_total;
+        ++position.chars_read_current_line;
 
         if (next_unget)
         {
@@ -1458,23 +1406,6 @@ scan_number_done:
             current = ia.get_character();
         }
 
-        return track_after_read();
-    }
-
-    /// shared head of get() / get_ignoring_pending_unget(): bump the
-    /// per-character position counters (line-count-on-'\n' bookkeeping is
-    /// handled afterwards, in track_after_read(), once `current` is known)
-    void advance_position() noexcept
-    {
-        ++position.chars_read_total;
-        ++position.chars_read_current_line;
-    }
-
-    /// shared tail of get() / get_ignoring_pending_unget(): capture the
-    /// character for error messages (if needed) and update line/column
-    /// bookkeeping for the character now in `current`
-    char_int_type track_after_read()
-    {
         // seekable adapters reconstruct the token lazily on error (see
         // get_token_string), so the eager per-character copy is skipped
         capture_char(std::integral_constant<bool, lazy_token_string> {});
@@ -1488,29 +1419,6 @@ scan_number_done:
         return current;
     }
 
-    /*!
-    @brief like get(), but for call sites that can prove no unget() is pending
-
-    get() has to check the `next_unget` flag on every call, because a
-    previous token may have ended with unget() (e.g. scan_number() always
-    ungets the character that terminated the number, so the next call to
-    scan() can see it again). skip_whitespace() reads that first,
-    possibly-ungotten character via a plain get(), but every further
-    character it reads is guaranteed to be a fresh read: nothing between
-    those calls invokes unget(). This variant skips the (otherwise always
-    false) next_unget branch for those calls; it is not a general
-    replacement for get().
-    */
-    char_int_type get_ignoring_pending_unget()
-    {
-        JSON_ASSERT(!next_unget);
-
-        advance_position();
-        current = ia.get_character();
-
-        return track_after_read();
-    }
-
     /// seekable adapter: nothing to capture, the token is rebuilt on error
     void capture_char(std::true_type /*lazy*/) const noexcept {}
 
@@ -1704,37 +1612,13 @@ scan_number_done:
         return true;
     }
 
-    /// whether `current` is one of the four JSON whitespace characters
-    bool current_is_whitespace() const noexcept
-    {
-        return current == ' ' || current == '\t' || current == '\n' || current == '\r';
-    }
-
     void skip_whitespace()
     {
-        // the first character may be a pending unget() left over from the
-        // previous token (see get_ignoring_pending_unget()); every
-        // subsequent character read by this loop is guaranteed fresh, since
-        // nothing below calls unget()
-        get();
-
-        if (!current_is_whitespace())
-        {
-            return;
-        }
-
-        // this is written as an if-guarded do-while (rather than a plain
-        // while loop) because that shape is what lets both GCC and Clang
-        // keep the input adapter's read pointer in a register across
-        // iterations; the equivalent while-loop measurably defeated that
-        // optimization in testing, turning long whitespace runs (e.g. the
-        // indentation of pretty-printed JSON) from a register-only loop
-        // into one that reloads the pointer from memory every character
         do
         {
-            get_ignoring_pending_unget();
+            get();
         }
-        while (current_is_whitespace());
+        while (current == ' ' || current == '\t' || current == '\n' || current == '\r');
     }
 
     token_type scan()
@@ -1870,13 +1754,6 @@ scan_number_done:
     const char_int_type decimal_point_char = '.';
     /// the position of the decimal point in the input
     std::size_t decimal_point_position = std::string::npos;
-
-    /// whether the caller (e.g. accept()/json_sax_acceptor) only needs the
-    /// token classification and never looks at the converted numeric value;
-    /// when set, scan_number() may skip strtoull()/strtoll() for
-    /// value_unsigned/value_integer tokens whose digit count guarantees they
-    /// fit into 64 bits (see scan_number())
-    const bool discard_number_values = false;
 };
 
 }  // namespace detail

include/nlohmann/detail/input/parser.hpp

@@ -72,10 +72,9 @@ class parser
                     parser_callback_t<BasicJsonType> cb = nullptr,
                     const bool allow_exceptions_ = true,
                     const bool ignore_comments = false,
-                    const bool ignore_trailing_commas_ = false,
-                    const bool discard_number_values_ = false)
+                    const bool ignore_trailing_commas_ = false)
         : callback(std::move(cb))
-        , m_lexer(std::move(adapter), ignore_comments, discard_number_values_)
+        , m_lexer(std::move(adapter), ignore_comments)
         , allow_exceptions(allow_exceptions_)
         , ignore_trailing_commas(ignore_trailing_commas_)
     {

include/nlohmann/json.hpp

@@ -164,12 +164,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         detail::parser_callback_t<basic_json>cb = nullptr,
         const bool allow_exceptions = true,
         const bool ignore_comments = false,
-        const bool ignore_trailing_commas = false,
-        const bool discard_number_values = false
+        const bool ignore_trailing_commas = false
     )
     {
         return ::nlohmann::detail::parser<basic_json, InputAdapterType>(std::move(adapter),
-               std::move(cb), allow_exceptions, ignore_comments, ignore_trailing_commas, discard_number_values);
+               std::move(cb), allow_exceptions, ignore_comments, ignore_trailing_commas);
     }
 
   private:
@@ -4134,7 +4133,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
                        const bool ignore_comments = false,
                        const bool ignore_trailing_commas = false)
     {
-        return parser(detail::input_adapter(std::forward<InputType>(i)), nullptr, false, ignore_comments, ignore_trailing_commas, true).accept(true);
+        return parser(detail::input_adapter(std::forward<InputType>(i)), nullptr, false, ignore_comments, ignore_trailing_commas).accept(true);
     }
 
     /// @brief check if the input is valid JSON (iterator pair, or iterator+sentinel pair for C++20 ranges support)
@@ -4145,7 +4144,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
                        const bool ignore_comments = false,
                        const bool ignore_trailing_commas = false)
     {
-        return parser(detail::input_adapter(std::move(first), std::move(last)), nullptr, false, ignore_comments, ignore_trailing_commas, true).accept(true);
+        return parser(detail::input_adapter(std::move(first), std::move(last)), nullptr, false, ignore_comments, ignore_trailing_commas).accept(true);
     }
 
     JSON_HEDLEY_WARN_UNUSED_RESULT
@@ -4154,7 +4153,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
                        const bool ignore_comments = false,
                        const bool ignore_trailing_commas = false)
     {
-        return parser(i.get(), nullptr, false, ignore_comments, ignore_trailing_commas, true).accept(true);
+        return parser(i.get(), nullptr, false, ignore_comments, ignore_trailing_commas).accept(true);
     }
 
     /// @brief generate SAX events
@@ -4940,6 +4939,12 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
                 // note erase performs range check
                 parent.erase(json_pointer::template array_index<basic_json_t>(last_path));
             }
+            else
+            {
+                // the parent of a "remove" target must be an object or array
+                // (see #5396)
+                JSON_THROW(out_of_range::create(413, detail::concat("cannot remove value: the JSON Patch 'remove' target's parent is of type ", parent.type_name(), ", but must be an object or array"), &parent));
+            }
         };
 
         // type check: top level value must be an array
@@ -5017,6 +5022,31 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
                     const auto from_path = get_value("move", "from", true).template get<string_t>();
                     json_pointer from_ptr(from_path);
 
+                    // RFC 6902 (section 4.4) forbids "from" from being a
+                    // proper prefix of "path": a location cannot be moved
+                    // into one of its own children. Compare the pointers'
+                    // reference tokens (already unescaped by json_pointer's
+                    // parser) rather than the raw pointer strings, since a
+                    // token may itself contain an escaped '/' or '~' that
+                    // would defeat a naive string-prefix comparison. This is
+                    // written as an explicit, manually-bounded loop (rather
+                    // than std::equal(first1, last1, first2), whose second
+                    // range has no explicit end iterator) so every access to
+                    // ptr.reference_tokens is visibly guarded by the same
+                    // index the loop condition already bounds against
+                    // from_size -- from_size < ptr.reference_tokens.size()
+                    // is checked once, up front, before the loop runs at all.
+                    const auto from_size = from_ptr.reference_tokens.size();
+                    bool from_is_proper_prefix_of_path = from_size < ptr.reference_tokens.size();
+                    for (std::size_t i = 0; from_is_proper_prefix_of_path && i < from_size; ++i)
+                    {
+                        from_is_proper_prefix_of_path = from_ptr.reference_tokens[i] == ptr.reference_tokens[i];
+                    }
+                    if (JSON_HEDLEY_UNLIKELY(from_is_proper_prefix_of_path))
+                    {
+                        JSON_THROW(out_of_range::create(414, detail::concat("cannot move value: 'from' path '", from_path, "' is a proper prefix of 'path' '", path, "'"), &result));
+                    }
+
                     // the "from" location must exist - use at()
                     basic_json const v = result.at(from_ptr);
 

single_include/nlohmann/json.hpp

@@ -7932,11 +7932,10 @@ class lexer : public lexer_base<BasicJsonType>
   public:
     using token_type = typename lexer_base<BasicJsonType>::token_type;
 
-    explicit lexer(InputAdapterType&& adapter, bool ignore_comments_ = false, bool discard_number_values_ = false) noexcept
+    explicit lexer(InputAdapterType&& adapter, bool ignore_comments_ = false) noexcept
         : ia(std::move(adapter))
         , ignore_comments(ignore_comments_)
         , decimal_point_char(static_cast<char_int_type>(get_decimal_point()))
-        , discard_number_values(discard_number_values_)
     {}
 
     // deleted because of pointer members
@@ -9063,58 +9062,6 @@ scan_number_done:
         // we are done scanning a number)
         unget();
 
-        // If the caller does not need the converted value (only whether the
-        // input is syntactically valid; see json_sax_acceptor/accept()), an
-        // unsigned/integer token can be reported without calling
-        // strtoull()/strtoll() at all, *provided* we can already tell from
-        // the digit count alone that the conversion cannot overflow 64 bits.
-        // Such tokens are always finite and are accepted unconditionally by
-        // the parser regardless of their actual value (parser::sax_parse_internal()
-        // never checks finiteness for value_unsigned/value_integer), so the
-        // classification below is all that is needed.
-        //
-        // A decimal number with up to 18 digits is always representable in
-        // both std::uint64_t and std::int64_t (18 nines is ~1e18, well below
-        // both UINT64_MAX ~1.8e19 and INT64_MAX ~9.2e18), so strtoull()/strtoll()
-        // could not have set errno to ERANGE for it. Numbers with more digits
-        // (rare in practice) fall through to the exact code below, unchanged,
-        // so their handling -- including reclassification to value_float when
-        // the value overflows 64 bits, and rejection when it is not even
-        // finite as a double -- is bit-for-bit identical to before this
-        // optimization.
-        //
-        // Note this reasons about std::uint64_t/std::int64_t, not about
-        // number_unsigned_t/number_integer_t (BasicJsonType's own, possibly
-        // narrower, template parameters -- e.g. std::uint32_t). That is fine
-        // *only* because discard_number_values is exclusively set by
-        // accept() (see json.hpp), and accept() always parses through the
-        // library's own json_sax_acceptor -- never a user-supplied SAX
-        // consumer -- whose number_unsigned()/number_integer()/number_float()
-        // callbacks unconditionally discard their argument and return true.
-        // So for every caller that can reach this branch, neither the token
-        // classification below nor the eventual (possibly narrowed, and on
-        // this fast path left stale/unset) value_unsigned/value_integer is
-        // ever consulted -- an unsigned/integer token is accepted outright,
-        // and even a >18-digit token that this fast path deliberately falls
-        // through for is, once reclassified to value_float, still finite
-        // (and thus accepted) for any digit count that fits in number_unsigned_t
-        // or number_integer_t regardless of that type's width. If this
-        // function is ever taught to run with discard_number_values true for
-        // a caller that *does* read the converted value, this reasoning (and
-        // the fast path below) would need to be revisited.
-        if (discard_number_values)
-        {
-            constexpr std::size_t safe_digit_count = 18;
-            if (number_type == token_type::value_unsigned && token_buffer.size() <= safe_digit_count)
-            {
-                return token_type::value_unsigned;
-            }
-            if (number_type == token_type::value_integer && token_buffer.size() - 1 <= safe_digit_count)
-            {
-                return token_type::value_integer;
-            }
-        }
-
         char* endptr = nullptr; // NOLINT(misc-const-correctness,cppcoreguidelines-pro-type-vararg,hicpp-vararg)
         errno = 0;
 
@@ -9229,7 +9176,8 @@ scan_number_done:
     */
     char_int_type get()
     {
-        advance_position();
+        ++position.chars_read_total;
+        ++position.chars_read_current_line;
 
         if (next_unget)
         {
@@ -9241,23 +9189,6 @@ scan_number_done:
             current = ia.get_character();
         }
 
-        return track_after_read();
-    }
-
-    /// shared head of get() / get_ignoring_pending_unget(): bump the
-    /// per-character position counters (line-count-on-'\n' bookkeeping is
-    /// handled afterwards, in track_after_read(), once `current` is known)
-    void advance_position() noexcept
-    {
-        ++position.chars_read_total;
-        ++position.chars_read_current_line;
-    }
-
-    /// shared tail of get() / get_ignoring_pending_unget(): capture the
-    /// character for error messages (if needed) and update line/column
-    /// bookkeeping for the character now in `current`
-    char_int_type track_after_read()
-    {
         // seekable adapters reconstruct the token lazily on error (see
         // get_token_string), so the eager per-character copy is skipped
         capture_char(std::integral_constant<bool, lazy_token_string> {});
@@ -9271,29 +9202,6 @@ scan_number_done:
         return current;
     }
 
-    /*!
-    @brief like get(), but for call sites that can prove no unget() is pending
-
-    get() has to check the `next_unget` flag on every call, because a
-    previous token may have ended with unget() (e.g. scan_number() always
-    ungets the character that terminated the number, so the next call to
-    scan() can see it again). skip_whitespace() reads that first,
-    possibly-ungotten character via a plain get(), but every further
-    character it reads is guaranteed to be a fresh read: nothing between
-    those calls invokes unget(). This variant skips the (otherwise always
-    false) next_unget branch for those calls; it is not a general
-    replacement for get().
-    */
-    char_int_type get_ignoring_pending_unget()
-    {
-        JSON_ASSERT(!next_unget);
-
-        advance_position();
-        current = ia.get_character();
-
-        return track_after_read();
-    }
-
     /// seekable adapter: nothing to capture, the token is rebuilt on error
     void capture_char(std::true_type /*lazy*/) const noexcept {}
 
@@ -9487,37 +9395,13 @@ scan_number_done:
         return true;
     }
 
-    /// whether `current` is one of the four JSON whitespace characters
-    bool current_is_whitespace() const noexcept
-    {
-        return current == ' ' || current == '\t' || current == '\n' || current == '\r';
-    }
-
     void skip_whitespace()
     {
-        // the first character may be a pending unget() left over from the
-        // previous token (see get_ignoring_pending_unget()); every
-        // subsequent character read by this loop is guaranteed fresh, since
-        // nothing below calls unget()
-        get();
-
-        if (!current_is_whitespace())
-        {
-            return;
-        }
-
-        // this is written as an if-guarded do-while (rather than a plain
-        // while loop) because that shape is what lets both GCC and Clang
-        // keep the input adapter's read pointer in a register across
-        // iterations; the equivalent while-loop measurably defeated that
-        // optimization in testing, turning long whitespace runs (e.g. the
-        // indentation of pretty-printed JSON) from a register-only loop
-        // into one that reloads the pointer from memory every character
         do
         {
-            get_ignoring_pending_unget();
+            get();
         }
-        while (current_is_whitespace());
+        while (current == ' ' || current == '\t' || current == '\n' || current == '\r');
     }
 
     token_type scan()
@@ -9653,13 +9537,6 @@ scan_number_done:
     const char_int_type decimal_point_char = '.';
     /// the position of the decimal point in the input
     std::size_t decimal_point_position = std::string::npos;
-
-    /// whether the caller (e.g. accept()/json_sax_acceptor) only needs the
-    /// token classification and never looks at the converted numeric value;
-    /// when set, scan_number() may skip strtoull()/strtoll() for
-    /// value_unsigned/value_integer tokens whose digit count guarantees they
-    /// fit into 64 bits (see scan_number())
-    const bool discard_number_values = false;
 };
 
 }  // namespace detail
@@ -14166,10 +14043,9 @@ class parser
                     parser_callback_t<BasicJsonType> cb = nullptr,
                     const bool allow_exceptions_ = true,
                     const bool ignore_comments = false,
-                    const bool ignore_trailing_commas_ = false,
-                    const bool discard_number_values_ = false)
+                    const bool ignore_trailing_commas_ = false)
         : callback(std::move(cb))
-        , m_lexer(std::move(adapter), ignore_comments, discard_number_values_)
+        , m_lexer(std::move(adapter), ignore_comments)
         , allow_exceptions(allow_exceptions_)
         , ignore_trailing_commas(ignore_trailing_commas_)
     {
@@ -21716,12 +21592,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
         detail::parser_callback_t<basic_json>cb = nullptr,
         const bool allow_exceptions = true,
         const bool ignore_comments = false,
-        const bool ignore_trailing_commas = false,
-        const bool discard_number_values = false
+        const bool ignore_trailing_commas = false
                                  )
     {
         return ::nlohmann::detail::parser<basic_json, InputAdapterType>(std::move(adapter),
-            std::move(cb), allow_exceptions, ignore_comments, ignore_trailing_commas, discard_number_values);
+            std::move(cb), allow_exceptions, ignore_comments, ignore_trailing_commas);
     }
 
   private:
@@ -25686,7 +25561,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
                        const bool ignore_comments = false,
                        const bool ignore_trailing_commas = false)
     {
-        return parser(detail::input_adapter(std::forward<InputType>(i)), nullptr, false, ignore_comments, ignore_trailing_commas, true).accept(true);
+        return parser(detail::input_adapter(std::forward<InputType>(i)), nullptr, false, ignore_comments, ignore_trailing_commas).accept(true);
     }
 
     /// @brief check if the input is valid JSON (iterator pair, or iterator+sentinel pair for C++20 ranges support)
@@ -25697,7 +25572,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
                        const bool ignore_comments = false,
                        const bool ignore_trailing_commas = false)
     {
-        return parser(detail::input_adapter(std::move(first), std::move(last)), nullptr, false, ignore_comments, ignore_trailing_commas, true).accept(true);
+        return parser(detail::input_adapter(std::move(first), std::move(last)), nullptr, false, ignore_comments, ignore_trailing_commas).accept(true);
     }
 
     JSON_HEDLEY_WARN_UNUSED_RESULT
@@ -25706,7 +25581,7 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
                        const bool ignore_comments = false,
                        const bool ignore_trailing_commas = false)
     {
-        return parser(i.get(), nullptr, false, ignore_comments, ignore_trailing_commas, true).accept(true);
+        return parser(i.get(), nullptr, false, ignore_comments, ignore_trailing_commas).accept(true);
     }
 
     /// @brief generate SAX events
@@ -26492,6 +26367,12 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
                 // note erase performs range check
                 parent.erase(json_pointer::template array_index<basic_json_t>(last_path));
             }
+            else
+            {
+                // the parent of a "remove" target must be an object or array
+                // (see #5396)
+                JSON_THROW(out_of_range::create(413, detail::concat("cannot remove value: the JSON Patch 'remove' target's parent is of type ", parent.type_name(), ", but must be an object or array"), &parent));
+            }
         };
 
         // type check: top level value must be an array
@@ -26569,6 +26450,31 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec
                     const auto from_path = get_value("move", "from", true).template get<string_t>();
                     json_pointer from_ptr(from_path);
 
+                    // RFC 6902 (section 4.4) forbids "from" from being a
+                    // proper prefix of "path": a location cannot be moved
+                    // into one of its own children. Compare the pointers'
+                    // reference tokens (already unescaped by json_pointer's
+                    // parser) rather than the raw pointer strings, since a
+                    // token may itself contain an escaped '/' or '~' that
+                    // would defeat a naive string-prefix comparison. This is
+                    // written as an explicit, manually-bounded loop (rather
+                    // than std::equal(first1, last1, first2), whose second
+                    // range has no explicit end iterator) so every access to
+                    // ptr.reference_tokens is visibly guarded by the same
+                    // index the loop condition already bounds against
+                    // from_size -- from_size < ptr.reference_tokens.size()
+                    // is checked once, up front, before the loop runs at all.
+                    const auto from_size = from_ptr.reference_tokens.size();
+                    bool from_is_proper_prefix_of_path = from_size < ptr.reference_tokens.size();
+                    for (std::size_t i = 0; from_is_proper_prefix_of_path && i < from_size; ++i)
+                    {
+                        from_is_proper_prefix_of_path = from_ptr.reference_tokens[i] == ptr.reference_tokens[i];
+                    }
+                    if (JSON_HEDLEY_UNLIKELY(from_is_proper_prefix_of_path))
+                    {
+                        JSON_THROW(out_of_range::create(414, detail::concat("cannot move value: 'from' path '", from_path, "' is a proper prefix of 'path' '", path, "'"), &result));
+                    }
+
                     // the "from" location must exist - use at()
                     basic_json const v = result.at(from_ptr);
 

tests/src/unit-class_parser.cpp

@@ -930,94 +930,6 @@ TEST_CASE("parser class")
                 CHECK(accept_helper("+1") == false);
                 CHECK(accept_helper("+0") == false);
             }
-
-            SECTION("issue #5411 - skip conversion when accept() does not need the numeric value")
-            {
-                // lexer::scan_number() may skip strtoull()/strtoll() for
-                // value_unsigned/value_integer tokens when the caller (e.g.
-                // json::accept()) does not need the converted value, as long
-                // as the digit count alone guarantees no 64-bit overflow (see
-                // the "safe_digit_count" fast path in scan_number()). This
-                // differential test checks that json::accept() (which enables
-                // the fast path) and json::parse() (which never does) always
-                // agree, over a corpus that exercises both the fast path
-                // (<=18 digits) and the untouched, exact fallback path (>=19
-                // digits) -- including reclassification of huge digit-only
-                // integers to a (possibly non-finite) floating-point value.
-                const std::vector<std::pair<std::string, bool>> cases =
-                {
-                    // normal small/large integers, both signs
-                    {"0", true}, {"1", true}, {"-1", true}, {"42", true}, {"-42", true},
-                    {"123456789", true}, {"-123456789", true},
-
-                    // digit-count boundary around the 18-digit safe cutoff (both signs)
-                    {std::string(17, '9'), true},
-                    {std::string(18, '9'), true},
-                    {std::string(19, '9'), true},
-                    {std::string(20, '9'), true},
-                    {"-" + std::string(17, '9'), true},
-                    {"-" + std::string(18, '9'), true},
-                    {"-" + std::string(19, '9'), true},
-                    {"-" + std::string(20, '9'), true},
-
-                    // 64-bit boundaries
-                    {"9223372036854775807", true},    // INT64_MAX
-                    {"-9223372036854775808", true},   // INT64_MIN
-                    {"18446744073709551615", true},   // UINT64_MAX
-                    {"18446744073709551616", true},   // UINT64_MAX + 1 (overflows uint64_t, finite double)
-
-                    // the 28-digit example from the issue: overflows uint64_t
-                    // but is finite as a double, so the scanner reclassifies
-                    // it to value_float and it is accepted
-                    {"9999999999999999999999999999", true},
-
-                    // huge digit-only integers that overflow even a double -> rejected
-                    {std::string(309, '9'), false},
-                    {std::string(400, '9'), false},
-                    {"1" + std::string(400, '0'), false},
-
-                    // 1e999 / 1e400 style overflow -> rejected
-                    {"1e999", false},
-                    {"1e400", false},
-                    {"-1e999", false},
-                    {"1E999", false},
-
-                    // values straddling DBL_MAX
-                    {"1.7976931348623157e308", true},   // <= DBL_MAX, finite
-                    {"1.7976931348623159e308", false},  // > DBL_MAX, overflows to inf
-
-                    // a mix of other valid/invalid numeric syntax
-                    {"3.14159", true},
-                    {"-0.0", true},
-                    {"1.0e10", true},
-                    {"01", false},
-                    {"-", false},
-                    {"1.", false},
-                    {"1e", false},
-                    {"+1", false},
-                };
-
-                for (const auto& c : cases)
-                {
-                    const std::string& number = c.first;
-                    const bool expected = c.second;
-                    CAPTURE(number)
-                    CAPTURE(expected)
-
-                    // accept() takes the fast path (skips conversion when possible)
-                    CHECK(json::accept(number) == expected);
-
-                    // parse() always performs the full conversion; it must agree
-                    json j;
-                    CHECK_NOTHROW(json::parser(nlohmann::detail::input_adapter(number), nullptr, false).parse(true, j));
-                    CHECK(!j.is_discarded() == expected);
-
-                    // wrap in an array so get_token() is exercised beyond the
-                    // very first (constructor-time) scan as well
-                    const std::string wrapped = "[" + number + "," + number + "]";
-                    CHECK(json::accept(wrapped) == expected);
-                }
-            }
         }
     }
 
@@ -1482,62 +1394,6 @@ TEST_CASE("parser class")
         CHECK(accept_helper("\"\\uD80C\\uFFFF\"") == false);
     }
 
-    SECTION("issue #5412 - whitespace skipping bookkeeping (compact vs. pretty-printed)")
-    {
-        // lexer::skip_whitespace() reads its first character with get() (to
-        // honor a possibly pending unget() from the previous token) and every
-        // further whitespace character with get_ignoring_pending_unget() (a
-        // get() variant that skips the then-always-false next_unget check).
-        // This must not change the reported byte offset, line, or column of
-        // a syntax error, even when a long run of whitespace containing
-        // multiple newlines is skipped beforehand (as with pretty-printed
-        // input). The expected values below were captured from the
-        // unmodified do-while(get()) loop, so any regression that miscounts
-        // characters or newlines while skipping whitespace changes them.
-        const auto check_error = [](const std::string & input, std::size_t expected_byte,
-                                    const std::string & expected_what)
-        {
-            CAPTURE(input)
-            try
-            {
-                json _ = json::parse(input);
-                FAIL_CHECK("expected a parse_error, but parsing succeeded");
-            }
-            catch (const json::parse_error& e)
-            {
-                CHECK(e.byte == expected_byte);
-                CHECK(std::string(e.what()) == expected_what);
-            }
-        };
-
-        // a nested document, serialized both compactly and pretty-printed
-        // (dump(4)), each truncated right before the final closing '}' so
-        // that the parser hits EOF after skipping all of the (in the
-        // pretty-printed case, substantial) indentation whitespace
-        const json doc =
-        {
-            {"a", 1},
-            {"b", json::array({true, false, nullptr, "x"})},
-            {"c", json::object({{"d", 3.14}, {"e", json::array({1, 2, 3})}})}
-        };
-
-        const std::string compact = doc.dump();
-        const std::string pretty = doc.dump(4);
-
-        check_error(compact.substr(0, compact.size() - 1), 60,
-                    "[json.exception.parse_error.101] parse error at line 1, column 60: syntax error while parsing object - unexpected end of input; expected '}'");
-        check_error(pretty.substr(0, pretty.size() - 1), 193,
-                    "[json.exception.parse_error.101] parse error at line 17, column 1: syntax error while parsing object - unexpected end of input; expected '}'");
-
-        // an invalid token appearing after several indented, multi-line
-        // whitespace runs vs. the same document without any of that
-        // whitespace
-        check_error("{\n    \"a\": 1,\n    \"b\": [\n        true,\n        false\n    ],\n    \"c\": @\n}", 70,
-                    "[json.exception.parse_error.101] parse error at line 7, column 10: syntax error while parsing value - invalid literal; last read: '\"c\": @'");
-        check_error("{\"a\":1,\"b\":[true,false],\"c\":@}", 29,
-                    "[json.exception.parse_error.101] parse error at line 1, column 29: syntax error while parsing value - invalid literal; last read: '\"c\":@'");
-    }
-
     SECTION("tests found by mutate++")
     {
         // test case to make sure no comma precedes the first key

tests/src/unit-json_patch.cpp

@@ -1389,6 +1389,172 @@ TEST_CASE("JSON patch - add to a primitive parent (regression #4292)")
     }
 }
 
+TEST_CASE("JSON patch - remove with primitive or null parent (regression #5396)")
+{
+    // Regression test for https://github.com/nlohmann/json/issues/5396
+    //
+    // RFC 6902 (§4.2) requires the target location of a "remove" operation
+    // to exist. When the target's parent resolves to a primitive value or
+    // null, the operation must fail. Previously operation_remove silently
+    // did nothing in this case (neither the "is_object" nor the "is_array"
+    // branch matched, and there was no final "else"), so the patch appeared
+    // to succeed without changing the document. It now throws
+    // out_of_range.413.
+
+    SECTION("parent is a primitive (number)")
+    {
+        json const doc = {{"a", 1}};
+        json const patch = {{{"op", "remove"}, {"path", "/a/b"}}};
+#if JSON_DIAGNOSTICS
+        CHECK_THROWS_WITH_AS(doc.patch(patch), "[json.exception.out_of_range.413] (/a) cannot remove value: the JSON Patch 'remove' target's parent is of type number, but must be an object or array", json::out_of_range&);
+#else
+        CHECK_THROWS_WITH_AS(doc.patch(patch), "[json.exception.out_of_range.413] cannot remove value: the JSON Patch 'remove' target's parent is of type number, but must be an object or array", json::out_of_range&);
+#endif
+    }
+
+    SECTION("parent is a primitive (string)")
+    {
+        json const doc = {{"foo", {{"bar", "a string"}}}};
+        json const patch = {{{"op", "remove"}, {"path", "/foo/bar/baz"}}};
+#if JSON_DIAGNOSTICS
+        CHECK_THROWS_WITH_AS(doc.patch(patch), "[json.exception.out_of_range.413] (/foo/bar) cannot remove value: the JSON Patch 'remove' target's parent is of type string, but must be an object or array", json::out_of_range&);
+#else
+        CHECK_THROWS_WITH_AS(doc.patch(patch), "[json.exception.out_of_range.413] cannot remove value: the JSON Patch 'remove' target's parent is of type string, but must be an object or array", json::out_of_range&);
+#endif
+    }
+
+    SECTION("top-level document is null")
+    {
+        json const doc = nullptr;
+        json const patch = {{{"op", "remove"}, {"path", "/a"}}};
+        CHECK_THROWS_WITH_AS(doc.patch(patch), "[json.exception.out_of_range.413] cannot remove value: the JSON Patch 'remove' target's parent is of type null, but must be an object or array", json::out_of_range&);
+    }
+
+    SECTION("legitimate removes still work")
+    {
+        // object member
+        json const doc1 = {{"a", 1}, {"b", 2}};
+        json const patch1 = {{{"op", "remove"}, {"path", "/a"}}};
+        CHECK(doc1.patch(patch1) == json({{"b", 2}}));
+
+        // array element
+        json const doc2 = R"([1, 2, 3])"_json;
+        json const patch2 = {{{"op", "remove"}, {"path", "/1"}}};
+        CHECK(doc2.patch(patch2) == R"([1, 3])"_json);
+    }
+}
+
+TEST_CASE("JSON patch - move where 'from' is a proper prefix of 'path' (regression #5397)")
+{
+    // Regression test for https://github.com/nlohmann/json/issues/5397
+    //
+    // RFC 6902 (§4.4) forbids "from" from being a proper prefix of "path"
+    // for a "move" operation: "a location cannot be moved into one of its
+    // children." "move" is implemented as remove-then-add; for an object
+    // target this happened to throw anyway as a side effect of the "add"
+    // step re-resolving through the now-removed parent, but for an array
+    // target the removal shifted subsequent indices, so "path" silently
+    // re-resolved to a different element and the operation "succeeded"
+    // with a corrupted result. It now throws out_of_range.414 for both
+    // object and array targets.
+
+    SECTION("array target (from the issue)")
+    {
+        json const doc = R"([[1,2],[3]])"_json;
+        json const patch = {{{"op", "move"}, {"from", "/0"}, {"path", "/0/0"}}};
+        CHECK_THROWS_WITH_AS(doc.patch(patch), "[json.exception.out_of_range.414] cannot move value: 'from' path '/0' is a proper prefix of 'path' '/0/0'", json::out_of_range&);
+    }
+
+    SECTION("object target")
+    {
+        json const doc = R"({"a": {"b": 1}})"_json;
+        json const patch = {{{"op", "move"}, {"from", "/a"}, {"path", "/a/b"}}};
+        CHECK_THROWS_WITH_AS(doc.patch(patch), "[json.exception.out_of_range.414] cannot move value: 'from' path '/a' is a proper prefix of 'path' '/a/b'", json::out_of_range&);
+    }
+
+    SECTION("from == path is not a proper prefix and must not be rejected")
+    {
+        // "from" equal to "path" is a no-op move; it is not a *proper*
+        // prefix relationship, so this new check must not reject it.
+        json const doc = R"({"a": 1, "b": 2})"_json;
+        json const patch = {{{"op", "move"}, {"from", "/a"}, {"path", "/a"}}};
+        CHECK(doc.patch(patch) == doc);
+    }
+
+    SECTION("raw string prefix that is not a pointer-token prefix must be allowed")
+    {
+        // "/ab" is a string-prefix of "/abc/x" as raw text, but "ab" and
+        // "abc" are different reference tokens, so this is NOT a
+        // pointer-token prefix relationship and the move must succeed.
+        // This is the key case proving the check compares tokens, not
+        // raw pointer text (a naive std::string prefix/rfind check on
+        // the undecoded pointer would wrongly reject this).
+        json const doc = R"({"ab": 1, "abc": {"x": 2}})"_json;
+        json const patch = {{{"op", "move"}, {"from", "/ab"}, {"path", "/abc/x"}}};
+        json const result = R"({"abc": {"x": 1}})"_json;
+        CHECK(doc.patch(patch) == result);
+    }
+
+    SECTION("escaped reference tokens are compared unescaped")
+    {
+        // "from" is the single token "a/b" (escaped as "a~1b"); "path"
+        // addresses member "x" of that same value, so "from" is a
+        // proper (token-level) prefix of "path" and must be rejected.
+        json const doc = R"({"a/b": {"x": 1}})"_json;
+        json const patch = {{{"op", "move"}, {"from", "/a~1b"}, {"path", "/a~1b/x"}}};
+        CHECK_THROWS_WITH_AS(doc.patch(patch), "[json.exception.out_of_range.414] cannot move value: 'from' path '/a~1b' is a proper prefix of 'path' '/a~1b/x'", json::out_of_range&);
+    }
+
+    SECTION("ordinary valid moves still work")
+    {
+        // unrelated top-level members
+        json const doc1 = R"({"a": 1, "b": 2})"_json;
+        json const patch1 = {{{"op", "move"}, {"from", "/a"}, {"path", "/c"}}};
+        CHECK(doc1.patch(patch1) == R"({"b": 2, "c": 1})"_json);
+
+        // sibling paths that share a textual prefix but are unrelated
+        json const doc2 = R"({"a": {"x": 1}, "b": {"y": 2}})"_json;
+        json const patch2 = {{{"op", "move"}, {"from", "/a/x"}, {"path", "/b/z"}}};
+        CHECK(doc2.patch(patch2) == R"({"a": {}, "b": {"y": 2, "z": 1}})"_json);
+
+        // "path" is a proper prefix of "from" (the reverse relationship,
+        // which RFC 6902 does not forbid)
+        json const doc3 = R"({"a": {"b": 1}})"_json;
+        json const patch3 = {{{"op", "move"}, {"from", "/a/b"}, {"path", "/a"}}};
+        CHECK(doc3.patch(patch3) == R"({"a": 1})"_json);
+    }
+
+    SECTION("root 'from' is a proper prefix of every non-root 'path'")
+    {
+        // the whole document is a proper prefix of any location inside it
+        json const doc = R"({"a": 1})"_json;
+        json const patch = {{{"op", "move"}, {"from", ""}, {"path", "/a"}}};
+        CHECK_THROWS_WITH_AS(doc.patch(patch), "[json.exception.out_of_range.414] cannot move value: 'from' path '' is a proper prefix of 'path' '/a'", json::out_of_range&);
+    }
+
+    SECTION("root 'path' is never a proper prefix violation for a non-root 'from'")
+    {
+        // the reverse of the above: moving a non-root location to the root
+        // is the "path is a prefix of from" relationship, which RFC 6902
+        // permits (already covered generally above; this pins the root
+        // case specifically, since root is the one path with no reference
+        // tokens at all)
+        json const doc = R"({"a": {"b": 1}})"_json;
+        json const patch = {{{"op", "move"}, {"from", "/a"}, {"path", ""}}};
+        CHECK(doc.patch(patch) == R"({"b": 1})"_json);
+    }
+
+    SECTION("the array-append token '-' is an ordinary child token")
+    {
+        // "-" (append-to-array) addresses a location *inside* the array,
+        // so "from" pointing at the array is still a proper prefix of
+        // "path" ending in "-" and must be rejected like any other child.
+        json const doc = R"({"a": [1, 2]})"_json;
+        json const patch = {{{"op", "move"}, {"from", "/a"}, {"path", "/a/-"}}};
+        CHECK_THROWS_WITH_AS(doc.patch(patch), "[json.exception.out_of_range.414] cannot move value: 'from' path '/a' is a proper prefix of 'path' '/a/-'", json::out_of_range&);
+    }
+}
+
 TEST_CASE("JSON patch - diff emits array removals in descending index order")
 {
     SECTION("array shrunk to empty")