This brings users without an embedding backend configured to closer
parity with those who do. Reuse the search backend to locate similar
documents and use them to provide the LLM with the better suggestion pool
to draw from
AI Suggestions previously invented near-duplicate metadata because the classification
prompt had no knowledge of the installation's own taxonomy. This surfaces
a small, ranked, permission-filtered set of existing tags/document
types/correspondents/storage paths - drawn from the document's RAG
neighbors plus its own already-assigned metadata - so the model prefers
reusing what already exists.
The LLM response schema now returns existing_ids (IDs of reused
candidates) separately from new_names (genuinely new suggestions).
Only new_names goes through localization and fuzzy name-matching;
existing_ids is resolved deterministically and never touched by the
localization pass, so exact matches can no longer be silently
corrupted by translation.
get_context_for_document() always materialized every document id a user
can see into a Python list, even for a superuser (or no user at all),
passing it through as a SQL IN filter. For a superuser, that's the whole
library:
- Past ~32,763 documents, this crashes outright:
sqlite3.OperationalError: too many SQL variables (SQLite's
SQLITE_MAX_VARIABLE_NUMBER is 32766 by default, and the query already
binds embedding + k + a NE self-exclusion clause alongside the ids).
- Below that cliff, vec0's IN-list evaluation is a nested loop (strncmp
per row per allowed id), so it's quadratic in library size for no
reason -- the filter was never going to exclude anything.
get_objects_for_user_owner_aware() already returns every Document for a
superuser (guardian's own with_superuser shortcut), so skipping straight
to document_ids=None changes nothing about which documents are
considered, only how we get there.
Also:
- Drop the pointless sorted() in _document_id_filters(): a SQL IN clause
doesn't care about order, so it was pure overhead on every call.
- Add a hard guard in _build_where() so a future regression (or a
legitimately huge permission-restricted user) fails closed -- no rows,
a logged warning -- instead of a cryptic OperationalError. Since this
filter scopes document access, failing closed rather than skipping the
filter is the only safe way to handle an oversized list.
First item from VECTOR_STORE_PERF_BACKLOG.md (an audit done alongside
perf/13314-vecstore-point-delete, deferred to its own branch since that
one was already large).
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Fix: Validate and limit chat question input in ChatStreamingView
Add max_length=4000 to ChatStreamingSerializer.q and replace the bare
request.data["q"] read with proper serializer.is_valid(raise_exception=True)
so oversized or missing questions are rejected with HTTP 400 before
reaching the LLM.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Fix: Add defensive prompt framing to mark document content as untrusted
* Also adds a system prompt which is treated higher that this is untrusted stuff
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
stumpylog
shamoon
