mirror of
https://github.com/paperless-ngx/paperless-ngx.git
synced 2026-08-08 03:43:20 +00:00
e35452beeb
get_context_for_document() always materialized every document id a user can see into a Python list, even for a superuser (or no user at all), passing it through as a SQL IN filter. For a superuser, that's the whole library: - Past ~32,763 documents, this crashes outright: sqlite3.OperationalError: too many SQL variables (SQLite's SQLITE_MAX_VARIABLE_NUMBER is 32766 by default, and the query already binds embedding + k + a NE self-exclusion clause alongside the ids). - Below that cliff, vec0's IN-list evaluation is a nested loop (strncmp per row per allowed id), so it's quadratic in library size for no reason -- the filter was never going to exclude anything. get_objects_for_user_owner_aware() already returns every Document for a superuser (guardian's own with_superuser shortcut), so skipping straight to document_ids=None changes nothing about which documents are considered, only how we get there. Also: - Drop the pointless sorted() in _document_id_filters(): a SQL IN clause doesn't care about order, so it was pure overhead on every call. - Add a hard guard in _build_where() so a future regression (or a legitimately huge permission-restricted user) fails closed -- no rows, a logged warning -- instead of a cryptic OperationalError. Since this filter scopes document access, failing closed rather than skipping the filter is the only safe way to handle an oversized list. First item from VECTOR_STORE_PERF_BACKLOG.md (an audit done alongside perf/13314-vecstore-point-delete, deferred to its own branch since that one was already large). Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>