Perf/fix: stop building a full-library IN filter for unrestricted RAG context (#13441)

get_context_for_document() always materialized every document id a user
can see into a Python list, even for a superuser (or no user at all),
passing it through as a SQL IN filter. For a superuser, that's the whole
library:

- Past ~32,763 documents, this crashes outright:
  sqlite3.OperationalError: too many SQL variables (SQLite's
  SQLITE_MAX_VARIABLE_NUMBER is 32766 by default, and the query already
  binds embedding + k + a NE self-exclusion clause alongside the ids).
- Below that cliff, vec0's IN-list evaluation is a nested loop (strncmp
  per row per allowed id), so it's quadratic in library size for no
  reason -- the filter was never going to exclude anything.

get_objects_for_user_owner_aware() already returns every Document for a
superuser (guardian's own with_superuser shortcut), so skipping straight
to document_ids=None changes nothing about which documents are
considered, only how we get there.

Also:
- Drop the pointless sorted() in _document_id_filters(): a SQL IN clause
  doesn't care about order, so it was pure overhead on every call.
- Add a hard guard in _build_where() so a future regression (or a
  legitimately huge permission-restricted user) fails closed -- no rows,
  a logged warning -- instead of a cryptic OperationalError. Since this
  filter scopes document access, failing closed rather than skipping the
  filter is the only safe way to handle an oversized list.

First item from VECTOR_STORE_PERF_BACKLOG.md (an audit done alongside
perf/13314-vecstore-point-delete, deferred to its own branch since that
one was already large).

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

src/paperless_ai/vector_store.py

@@ -75,6 +75,17 @@ class _Row(NamedTuple):
     embedding: bytes
 
 
+# _build_where(): the largest IN value list translated into bound SQL
+# parameters. SQLite's own hard limit (SQLITE_MAX_VARIABLE_NUMBER) is 32766
+# by default; this leaves headroom below that for the query's other bound
+# parameters (the embedding blob, k, and any NE clause) and for the limit
+# itself to move. An IN filter this large should not happen in practice --
+# callers are expected to pass None (no filter) rather than every id when
+# the filter would not actually narrow anything -- so this is a guard
+# against a future regression, not a normal code path.
+_MAX_IN_VALUES = 32700
+
+
 def _pack(embedding: Sequence[float]) -> bytes:
     return struct.pack(f"{len(embedding)}f", *embedding)
 
@@ -119,6 +130,21 @@ def _build_where(filters: MetadataFilters | None) -> tuple[str, list[int]]:
             if not values:  # pragma: no cover
                 clauses.append("1 = 0")
                 continue
+            if len(values) > _MAX_IN_VALUES:
+                # Fail closed (see the empty-clauses case below) rather than
+                # let SQLite raise "too many SQL variables" past its own
+                # limit: this filter scopes document access, so an IN list
+                # too large to safely bind must match no rows, never widen
+                # the scope to "everything" by accident.
+                logger.warning(
+                    "Refusing to build an IN filter on %r with %d values "
+                    "(over the %d-value safety limit); returning no rows.",
+                    f.key,
+                    len(values),
+                    _MAX_IN_VALUES,
+                )
+                clauses.append("1 = 0")
+                continue
             placeholders = ",".join("?" for _ in values)
             clauses.append(f"{f.key} IN ({placeholders})")
             params.extend(values)