get_context_for_document() always materialized every document id a user
can see into a Python list, even for a superuser (or no user at all),
passing it through as a SQL IN filter. For a superuser, that's the whole
library:
- Past ~32,763 documents, this crashes outright:
sqlite3.OperationalError: too many SQL variables (SQLite's
SQLITE_MAX_VARIABLE_NUMBER is 32766 by default, and the query already
binds embedding + k + a NE self-exclusion clause alongside the ids).
- Below that cliff, vec0's IN-list evaluation is a nested loop (strncmp
per row per allowed id), so it's quadratic in library size for no
reason -- the filter was never going to exclude anything.
get_objects_for_user_owner_aware() already returns every Document for a
superuser (guardian's own with_superuser shortcut), so skipping straight
to document_ids=None changes nothing about which documents are
considered, only how we get there.
Also:
- Drop the pointless sorted() in _document_id_filters(): a SQL IN clause
doesn't care about order, so it was pure overhead on every call.
- Add a hard guard in _build_where() so a future regression (or a
legitimately huge permission-restricted user) fails closed -- no rows,
a logged warning -- instead of a cryptic OperationalError. Since this
filter scopes document access, failing closed rather than skipping the
filter is the only safe way to handle an oversized list.
First item from VECTOR_STORE_PERF_BACKLOG.md (an audit done alongside
perf/13314-vecstore-point-delete, deferred to its own branch since that
one was already large).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Fix: Validate and limit chat question input in ChatStreamingView
Add max_length=4000 to ChatStreamingSerializer.q and replace the bare
request.data["q"] read with proper serializer.is_valid(raise_exception=True)
so oversized or missing questions are rejected with HTTP 400 before
reaching the LLM.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Fix: Add defensive prompt framing to mark document content as untrusted
* Also adds a system prompt which is treated higher that this is untrusted stuff
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
stumpylogandClaude Sonnet 5
shamoonandGitHub
