mirror of
https://github.com/nlohmann/json.git
synced 2026-10-27 00:08:55 +00:00
Fix BSON binary subtype 0x02 and check the declared document size earlier (#5930)
* Write and read the inner length of BSON binary subtype 0x02 The BSON specification defines the payload of the "binary (old)" subtype 0x02 as an int32 followed by that many bytes. to_bson wrote the plain payload instead, so decoders such as pymongo and libbson-based readers rejected the bytes, and from_bson returned the inner length as part of the payload (documented as lenient behavior in gh-5333). The writer now emits the outer length (payload + 4), the subtype, the inner int32 holding the payload size, and the payload. The reader requires an outer length of at least 4 and an inner length equal to the outer length minus 4 (the cases the bson-corpus lists as decode errors), reports parse error 112 otherwise, and returns only the payload. Both directions change together so that the round trip keeps working. The single-reservation fast path for scalar entries declines subtype 0x02 because it has no room for the inner length. This is a deliberate exception to the opt-in rule for behavior changes: the previous output was invalid BSON. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Check the declared BSON document size before reading its content The BSON reader compared a document's declared size with the number of bytes read only when it reached the terminator. The declared size never bounded the reads, so a SAX handler saw the keys and values of a document whose size was too small before parse error 112 was reported. The container frame now stores the position where the document ends according to its declared size. open_bson_document rejects a size below 5 (size prefix plus terminator) and an embedded document that does not end before the terminator of its enclosing document; the element loop rejects an element whose type byte leaves no room for the name terminator and the document terminator. The terminator check stays and still catches a declared size that is too large. No input that was accepted before is rejected now; documents with a wrong size are reported earlier, with fewer SAX events, a different byte position and, in three cases, a different message. The error id (parse_error.112) is unchanged and strict=false still does not skip the check. The tests that asserted the old messages and positions are updated, with the outer sizes of two inputs corrected so that they still exercise the embedded document. New tests cover the SAX events, a size below 5, and an embedded document that fits exactly. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Document BSON subtype 0x02, document size checks, and lenient reads Describe the inner length of binary subtype 0x02 that to_bson now writes and from_bson checks and removes. Add the document size checks (parse_error.112, not affected by strict=false) to bson.md and from_bson.md, and extend the list of lenient reads with the UUID and MD5 payload lengths (not checked), duplicate keys (last value wins), embedded NUL bytes in strings, and the ignored trailing bytes with strict=false. State that the members of an object are written in iteration order. Correct the statement that element names are never validated on read: they are checked and sanitized like string values by the error handler. Add the BSON message to the examples of out_of_range.415 and type_error.321, the subtype limit of 255 to the binary values page, CBOR to the formats that use byte_container_with_subtype, and fix a typo in the incomplete mapping box. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Regenerate single_include and apply astyle Result of make amalgamate (including make pretty) for the preceding changes. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Address review of the BSON documentation Describe exactly when a too small document size is detected and note that subtype 2 data written by earlier versions is not readable. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * docs: use the Oxford comma, drop discarded-value remarks Add the serial comma in a list added by this PR and remove the BSON example of type_error.321 (with the type_error.317 remark) so the docs do not mention discarded values for BSON; restore the original singular heading of that section. No tables were changed by this PR. Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix -Weffc++ in the BSON recorder test and a clang-tidy use-auto hit Signed-off-by: Niels Lohmann <mail@nlohmann.me> * Fix clang-tidy readability-redundant-member-init in BSON test Signed-off-by: Niels Lohmann <mail@nlohmann.me> --------- Signed-off-by: Niels Lohmann <mail@nlohmann.me>
This commit is contained in:
Niels Lohmann authored and 
GitHub committed
1 parent
0c3570d44a
commit
dc768a7d5d
10 files changed
+491
-35
No files matched your search
+249
-17
@@ -802,12 +802,11 @@ TEST_CASE("BSON")
|
||||
CHECK(json::from_bson(result, true, false) == j);
|
||||
}
|
||||
|
||||
SECTION("binary member with subtype 0x02 (old binary) keeps its inner length prefix (lenient parsing)")
|
||||
SECTION("binary member with subtype 0x02 (old binary) carries an inner length")
|
||||
{
|
||||
// documented lenient behavior (see gh-5333): the payload for
|
||||
// binary subtype 0x02 ("old binary") is returned as-is,
|
||||
// including its own inner 4-byte length prefix; it is not
|
||||
// stripped or reinterpreted
|
||||
// the payload of binary subtype 0x02 ("old binary") starts with an
|
||||
// int32 that repeats the payload size; the bytes below are what
|
||||
// pymongo and the bson-corpus (binary.json) use
|
||||
std::vector<std::uint8_t> const input =
|
||||
{
|
||||
0x17, 0x00, 0x00, 0x00, // size (little endian)
|
||||
@@ -815,16 +814,87 @@ TEST_CASE("BSON")
|
||||
|
||||
0x06, 0x00, 0x00, 0x00, // size of binary (little endian)
|
||||
0x02, // "old binary" subtype
|
||||
0x02, 0x00, 0x00, 0x00, // inner length prefix (part of the old-binary payload)
|
||||
0x02, 0x00, 0x00, 0x00, // inner length
|
||||
0x68, 0x69, // payload ('h', 'i')
|
||||
|
||||
0x00 // end marker
|
||||
};
|
||||
|
||||
// the inner length prefix is part of the (unmodified) payload
|
||||
const std::vector<std::uint8_t> expected_payload = {0x02, 0x00, 0x00, 0x00, 0x68, 0x69};
|
||||
const json expected = { { "entry", json::binary(expected_payload, 0x02) } };
|
||||
// the inner length is not part of the payload
|
||||
const json expected = { { "entry", json::binary({0x68, 0x69}, 0x02) } };
|
||||
CHECK(json::from_bson(input) == expected);
|
||||
CHECK(json::to_bson(expected) == input);
|
||||
CHECK(json::from_bson(json::to_bson(expected)) == expected);
|
||||
}
|
||||
|
||||
SECTION("binary member with subtype 0x02 (old binary) and empty payload")
|
||||
{
|
||||
std::vector<std::uint8_t> const input =
|
||||
{
|
||||
0x15, 0x00, 0x00, 0x00, // size (little endian)
|
||||
0x05, 'e', 'n', 't', 'r', 'y', '\x00', // entry: binary
|
||||
0x04, 0x00, 0x00, 0x00, // size of binary (little endian)
|
||||
0x02, // "old binary" subtype
|
||||
0x00, 0x00, 0x00, 0x00, // inner length
|
||||
0x00 // end marker
|
||||
};
|
||||
|
||||
const json expected = { { "entry", json::binary({}, 0x02) } };
|
||||
CHECK(json::from_bson(input) == expected);
|
||||
CHECK(json::to_bson(expected) == input);
|
||||
}
|
||||
|
||||
SECTION("binary member with subtype 0x02 (old binary) with an invalid inner length")
|
||||
{
|
||||
std::vector<std::uint8_t> input =
|
||||
{
|
||||
0x17, 0x00, 0x00, 0x00, // size (little endian)
|
||||
0x05, 'e', 'n', 't', 'r', 'y', '\x00', // entry: binary
|
||||
0x06, 0x00, 0x00, 0x00, // size of binary (little endian)
|
||||
0x02, // "old binary" subtype
|
||||
0x02, 0x00, 0x00, 0x00, // inner length
|
||||
0x68, 0x69, // payload ('h', 'i')
|
||||
0x00 // end marker
|
||||
};
|
||||
|
||||
SECTION("inner length too long")
|
||||
{
|
||||
input[19] = 0x03;
|
||||
}
|
||||
|
||||
SECTION("inner length too short")
|
||||
{
|
||||
input[19] = 0x01;
|
||||
}
|
||||
|
||||
SECTION("inner length negative")
|
||||
{
|
||||
input[19] = 0xFF;
|
||||
input[20] = 0xFF;
|
||||
input[21] = 0xFF;
|
||||
input[22] = 0xFF;
|
||||
}
|
||||
|
||||
CAPTURE(input)
|
||||
json _;
|
||||
CHECK_THROWS_AS(_ = json::from_bson(input), json::parse_error&);
|
||||
}
|
||||
|
||||
SECTION("binary member with subtype 0x02 (old binary) without an inner length")
|
||||
{
|
||||
// the payload is shorter than the inner length field
|
||||
std::vector<std::uint8_t> const input =
|
||||
{
|
||||
0x14, 0x00, 0x00, 0x00, // size (little endian)
|
||||
0x05, 'e', 'n', 't', 'r', 'y', '\x00', // entry: binary
|
||||
0x03, 0x00, 0x00, 0x00, // size of binary (little endian)
|
||||
0x02, // "old binary" subtype
|
||||
0x01, 0x02, 0x03, // payload without an inner length
|
||||
0x00 // end marker
|
||||
};
|
||||
|
||||
json _;
|
||||
CHECK_THROWS_AS(_ = json::from_bson(input), json::parse_error&);
|
||||
}
|
||||
|
||||
SECTION("Some more complex document")
|
||||
@@ -1178,7 +1248,7 @@ TEST_CASE("BSON document size mismatch")
|
||||
SECTION("top-level document with a negative size")
|
||||
{
|
||||
std::vector<std::uint8_t> const input = {0xFF, 0xFF, 0xFF, 0xFF, 0x00};
|
||||
CHECK_THROWS_WITH_AS(_ = json::from_bson(input), "[json.exception.parse_error.112] parse error at byte 5: syntax error while parsing BSON document: document size -1 does not match the number of bytes read (5)", json::parse_error&);
|
||||
CHECK_THROWS_WITH_AS(_ = json::from_bson(input), "[json.exception.parse_error.112] parse error at byte 4: syntax error while parsing BSON document: document size -1 is less than 5", json::parse_error&);
|
||||
CHECK(json::from_bson(input, true, false).is_discarded());
|
||||
}
|
||||
|
||||
@@ -1190,7 +1260,7 @@ TEST_CASE("BSON document size mismatch")
|
||||
// enclosing document instead of "d"
|
||||
std::vector<std::uint8_t> const input =
|
||||
{
|
||||
0x00, 0x00, 0x00, 0x00, // outer size
|
||||
0x14, 0x00, 0x00, 0x00, // outer size
|
||||
0x03, 'd', 0x00, // entry: embedded document "d"
|
||||
0xFF, 0xFF, 0xFF, 0x7F, // embedded size 0x7FFFFFFF
|
||||
0x0A, 'a', 0x00, // entry: null "a"
|
||||
@@ -1198,7 +1268,24 @@ TEST_CASE("BSON document size mismatch")
|
||||
0x08, 'h', 0x00, 0x01, // entry: bool "h" = true
|
||||
0x00 // outer end marker
|
||||
};
|
||||
CHECK_THROWS_WITH_AS(_ = json::from_bson(input), "[json.exception.parse_error.112] parse error at byte 15: syntax error while parsing BSON document: document size 2147483647 does not match the number of bytes read (8)", json::parse_error&);
|
||||
CHECK_THROWS_WITH_AS(_ = json::from_bson(input), "[json.exception.parse_error.112] parse error at byte 11: syntax error while parsing BSON document: document size 2147483647 exceeds the size of the enclosing document", json::parse_error&);
|
||||
CHECK(json::from_bson(input, true, false).is_discarded());
|
||||
}
|
||||
|
||||
SECTION("embedded document whose size is smaller than its content")
|
||||
{
|
||||
// {"a":null} is 8 bytes, but the length prefix claims 7
|
||||
std::vector<std::uint8_t> const input =
|
||||
{
|
||||
0x14, 0x00, 0x00, 0x00, // outer size
|
||||
0x03, 'd', 0x00, // entry: embedded document "d"
|
||||
0x07, 0x00, 0x00, 0x00, // embedded size 7 (real is 8)
|
||||
0x0A, 'a', 0x00, // entry: null "a"
|
||||
0x00, // embedded end marker
|
||||
0x08, 'h', 0x00, 0x01, // entry: bool "h" = true
|
||||
0x00 // outer end marker
|
||||
};
|
||||
CHECK_THROWS_WITH_AS(_ = json::from_bson(input), "[json.exception.parse_error.112] parse error at byte 15: syntax error while parsing BSON document: document size 7 does not match the number of bytes read (8)", json::parse_error&);
|
||||
CHECK(json::from_bson(input, true, false).is_discarded());
|
||||
}
|
||||
|
||||
@@ -1207,18 +1294,163 @@ TEST_CASE("BSON document size mismatch")
|
||||
// array [42] is 12 bytes, but the length prefix claims 13
|
||||
std::vector<std::uint8_t> const input =
|
||||
{
|
||||
0x00, 0x00, 0x00, 0x00, // outer size
|
||||
0x14, 0x00, 0x00, 0x00, // outer size
|
||||
0x04, 'a', 0x00, // entry: array "a"
|
||||
0x0D, 0x00, 0x00, 0x00, // array size 13 (real is 12)
|
||||
0x10, '0', 0x00, 0x2A, 0x00, 0x00, 0x00, // entry: int32 "0" = 42
|
||||
0x00, // array end marker
|
||||
0x00 // outer end marker
|
||||
};
|
||||
CHECK_THROWS_WITH_AS(_ = json::from_bson(input), "[json.exception.parse_error.112] parse error at byte 19: syntax error while parsing BSON document: document size 13 does not match the number of bytes read (12)", json::parse_error&);
|
||||
CHECK_THROWS_WITH_AS(_ = json::from_bson(input), "[json.exception.parse_error.112] parse error at byte 11: syntax error while parsing BSON document: document size 13 exceeds the size of the enclosing document", json::parse_error&);
|
||||
CHECK(json::from_bson(input, true, false).is_discarded());
|
||||
}
|
||||
}
|
||||
|
||||
TEST_CASE("BSON declared document size bounds the content")
|
||||
{
|
||||
// records the events that a SAX parser receives
|
||||
struct recorder : nlohmann::json_sax<json>
|
||||
{
|
||||
std::string events;
|
||||
|
||||
bool null() override
|
||||
{
|
||||
events += "null ";
|
||||
return true;
|
||||
}
|
||||
bool boolean(bool /*val*/) override
|
||||
{
|
||||
events += "boolean ";
|
||||
return true;
|
||||
}
|
||||
bool number_integer(json::number_integer_t /*val*/) override
|
||||
{
|
||||
events += "integer ";
|
||||
return true;
|
||||
}
|
||||
bool number_unsigned(json::number_unsigned_t /*val*/) override
|
||||
{
|
||||
events += "unsigned ";
|
||||
return true;
|
||||
}
|
||||
bool number_float(json::number_float_t /*val*/, const std::string& /*s*/) override
|
||||
{
|
||||
events += "float ";
|
||||
return true;
|
||||
}
|
||||
bool string(std::string& /*val*/) override
|
||||
{
|
||||
events += "string ";
|
||||
return true;
|
||||
}
|
||||
bool binary(json::binary_t& /*val*/) override
|
||||
{
|
||||
events += "binary ";
|
||||
return true;
|
||||
}
|
||||
bool start_object(std::size_t /*elements*/) override
|
||||
{
|
||||
events += "{ ";
|
||||
return true;
|
||||
}
|
||||
bool key(std::string& /*val*/) override
|
||||
{
|
||||
events += "key ";
|
||||
return true;
|
||||
}
|
||||
bool end_object() override
|
||||
{
|
||||
events += "} ";
|
||||
return true;
|
||||
}
|
||||
bool start_array(std::size_t /*elements*/) override
|
||||
{
|
||||
events += "[ ";
|
||||
return true;
|
||||
}
|
||||
bool end_array() override
|
||||
{
|
||||
events += "] ";
|
||||
return true;
|
||||
}
|
||||
bool parse_error(std::size_t /*position*/, const std::string& /*last_token*/, const json::exception& /*ex*/) override
|
||||
{
|
||||
events += "error ";
|
||||
return false;
|
||||
}
|
||||
};
|
||||
|
||||
SECTION("the content of a too small document is not reported")
|
||||
{
|
||||
// {"a":1} is 12 bytes; the length prefix claims 5, so the element
|
||||
// does not fit
|
||||
std::vector<std::uint8_t> const input =
|
||||
{
|
||||
0x05, 0x00, 0x00, 0x00,
|
||||
0x10, 'a', 0x00, 0x01, 0x00, 0x00, 0x00,
|
||||
0x00
|
||||
};
|
||||
recorder rec;
|
||||
CHECK(!json::sax_parse(input, &rec, json::input_format_t::bson));
|
||||
CHECK(rec.events == "{ error ");
|
||||
|
||||
// also without strict mode
|
||||
recorder rec2;
|
||||
CHECK(!json::sax_parse(input, &rec2, json::input_format_t::bson, false));
|
||||
CHECK(rec2.events == "{ error ");
|
||||
}
|
||||
|
||||
SECTION("an embedded document that does not fit its parent is not opened")
|
||||
{
|
||||
// the embedded document ends where the enclosing one does, so there
|
||||
// is no room for the terminator of the enclosing document
|
||||
std::vector<std::uint8_t> const input =
|
||||
{
|
||||
0x0C, 0x00, 0x00, 0x00,
|
||||
0x03, 'd', 0x00,
|
||||
0x05, 0x00, 0x00, 0x00, 0x00,
|
||||
0x00
|
||||
};
|
||||
recorder rec;
|
||||
CHECK(!json::sax_parse(input, &rec, json::input_format_t::bson));
|
||||
CHECK(rec.events == "{ key error ");
|
||||
}
|
||||
|
||||
SECTION("an embedded document that fits exactly")
|
||||
{
|
||||
std::vector<std::uint8_t> const input =
|
||||
{
|
||||
0x0D, 0x00, 0x00, 0x00,
|
||||
0x03, 'd', 0x00,
|
||||
0x05, 0x00, 0x00, 0x00, 0x00,
|
||||
0x00
|
||||
};
|
||||
CHECK(json::from_bson(input) == json({{"d", json::object()}}));
|
||||
}
|
||||
|
||||
SECTION("a document size that is less than 5")
|
||||
{
|
||||
for (const std::int32_t size :
|
||||
{
|
||||
0, 1, 4, -1, (std::numeric_limits<std::int32_t>::min)()
|
||||
})
|
||||
{
|
||||
CAPTURE(size)
|
||||
std::vector<std::uint8_t> input =
|
||||
{
|
||||
0x00, 0x00, 0x00, 0x00, 0x00
|
||||
};
|
||||
input[0] = static_cast<std::uint8_t>(size & 0xFF);
|
||||
input[1] = static_cast<std::uint8_t>((size >> 8) & 0xFF);
|
||||
input[2] = static_cast<std::uint8_t>((size >> 16) & 0xFF);
|
||||
input[3] = static_cast<std::uint8_t>((size >> 24) & 0xFF);
|
||||
json _;
|
||||
CHECK_THROWS_AS(_ = json::from_bson(input), json::parse_error&);
|
||||
CHECK(json::from_bson(input, true, false).is_discarded());
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
TEST_CASE("BSON nesting does not consume the call stack")
|
||||
{
|
||||
// An embedded document or array used to be read by calling back into the
|
||||
@@ -1923,7 +2155,7 @@ TEST_CASE("Invalid document size handling")
|
||||
{
|
||||
std::vector<std::uint8_t> const v = {0x04, 0x00, 0x00, 0x00, 0x00};
|
||||
json _;
|
||||
CHECK_THROWS_WITH_AS(_ = json::from_bson(v), "[json.exception.parse_error.112] parse error at byte 5: syntax error while parsing BSON document: document size 4 does not match the number of bytes read (5)", json::parse_error&);
|
||||
CHECK_THROWS_WITH_AS(_ = json::from_bson(v), "[json.exception.parse_error.112] parse error at byte 4: syntax error while parsing BSON document: document size 4 is less than 5", json::parse_error&);
|
||||
CHECK(json::from_bson(v, true, false).is_discarded());
|
||||
}
|
||||
|
||||
@@ -1938,7 +2170,7 @@ TEST_CASE("Invalid document size handling")
|
||||
0x00
|
||||
};
|
||||
json _;
|
||||
CHECK_THROWS_WITH_AS(_ = json::from_bson(v), "[json.exception.parse_error.112] parse error at byte 16: syntax error while parsing BSON document: document size 5 does not match the number of bytes read (16)", json::parse_error&);
|
||||
CHECK_THROWS_WITH_AS(_ = json::from_bson(v), "[json.exception.parse_error.112] parse error at byte 5: syntax error while parsing BSON document: document size 5 is too small for its content", json::parse_error&);
|
||||
CHECK(json::from_bson(v, true, false).is_discarded());
|
||||
}
|
||||
|
||||
@@ -1969,7 +2201,7 @@ TEST_CASE("Invalid document size handling")
|
||||
0x00 // object terminator
|
||||
};
|
||||
json _;
|
||||
CHECK_THROWS_WITH_AS(_ = json::from_bson(v), "[json.exception.parse_error.112] parse error at byte 19: syntax error while parsing BSON document: document size 5 does not match the number of bytes read (12)", json::parse_error&);
|
||||
CHECK_THROWS_WITH_AS(_ = json::from_bson(v), "[json.exception.parse_error.112] parse error at byte 12: syntax error while parsing BSON document: document size 5 is too small for its content", json::parse_error&);
|
||||
CHECK(json::from_bson(v, true, false).is_discarded());
|
||||
}
|
||||
|
||||
|
||||
Reference in new issue
Block a user