The vec0 default of 1024 rows per vector chunk makes each chunk a
multi-megabyte blob. On spinning disks, indexing a document took about
2.5 seconds and re-indexing one about 13 seconds. With 32 rows per
chunk the same operations take about 35 and 180 milliseconds, with the
write-ahead log a third of the size and no cost in query speed or file
size.
Pair that with 64 KiB SQLite pages, which made cold scans about 35%
faster and warm ones about 45% faster at no write cost once the chunks
are small. The page size is set before the file enters WAL mode, so it
applies to new files.
Document chat held the shared index read lock for the whole request,
including the LLM stream, which can take minutes. A compaction or
migration swap waiting for exclusive access could time out behind it
and be skipped until the next run.
Now that the answer is synthesized from the nodes already retrieved,
nothing after the vector search touches the store. Hold the lock only
while loading the index and searching, and release it before synthesis.
Document chat searched the vector store twice per question: once
explicitly to build the reference list, and again inside
RetrieverQueryEngine.query(), which retrieves before synthesizing. That
meant two query-embedding calls and two full vector searches each time.
The engine added nothing beyond the retriever call (no node
postprocessors are configured), so drop it and synthesize directly from
the nodes already retrieved. The streaming response is unchanged.
* Fix: match Apple Mail tag colors case-insensitively
* Fix: reject OAuth callbacks with an unrecognized scope
* Fix: refuse to refresh OAuth tokens for non-OAuth accounts or missing refresh tokens
* Fix: parse datetime strings in the format_datetime filter and reject invalid ones
* Fix: reject a PAPERLESS_URL without a hostname
* Fix: report a database error in the system status instead of failing with a 500
* Fix: surface the database error when refreshing a document before moving its files
* Fix: report no unapplied migrations when the applied migrations cannot be read
* Fix: return the parsed value directly in format_datetime for typing
* Chore: annotate the paperless_dirs fixture in the refresh failure test
* Allow null for the latest_migration system status type
---------
Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
* Chore: Import dateparser lazily so processes that never parse dates skip it
* Chore: Import turbohtml lazily in the mail parser
* Chore: Import lxml lazily in the SVG validator
* Chore: Import paperless_ai.indexing lazily in document tasks
* Chore: Import paperless_ai helpers lazily in views
* Chore: Read the OCRmyPDF sRGB profile on first use instead of at import
* Chore: Import pikepdf and img2pdf lazily so startup does not load them
* Chore: Reuse the initialized Django ASGI application for HTTP routing
* Chore: drop get_groups_with_only_permission
It worked around get_groups_with_perms lacking only_with_perms_in, which
django-guardian added in 3.1.0. The pinned version is newer.
* Chore: consolidate direct grant lookups into get_permission_grantees
Six places listed who holds a permission on an object, each with its own
query and result shape, and one had already drifted into expanding group
members.
* Chore: drop the unreachable empty viewer grant default
get_permission_grantees returns an entry for every requested document.
* Chore: drop set_permissions_for_object in favor of the bulk helper
The same grant rules were implemented twice, and only the bulk version
validated every action before writing, so the single-object one could leave
a partial change behind.
* Chore: type the set-permissions dict
Callers passed whatever they had, including querysets of User and Group
objects that only worked because id__in accepts a subquery.
* Refactor: add pure pdf_ops module with real-PDF tests
* Drops tests we don't relly need + adds the typing annotation
* Now that I'm sure it works, re-work these tests to mock pdf_ops, rather than pikepdf
* Validates pages ahead of time in build_pdfs rather than during the building
* Adds the given/when/then and some additional cleanups of pytest style
* This is redundant now
shamoon
GitHub Actions
Trenton H
Ben Petty