Performance: Tantivy indexing optimization (#13053)

* Tantivy: get permissions by chunks

-40% indexing time compared to previous commit

* Make progress bar process one by one with chunk

-15% indexing time compared to previous commit

* Prefetch FK + iterate over chunk from SQL

Prefetch additional needed data (note user, custom field content)

-20% indexing time compared to previous commit

* Reindex: increase Tantivy heap size from 128 to 512MB

Gains probably vary depending on the machine,
but it seems a sweet spot compatible with low-end hardware.

* Reindex: optimization on permission fetching and autocomplete word set

-10% indexing time compared to previous commit

* Autocomplete analyzer python->rust

Splits words with underscore compared to the python analyzer.
E.g.: "blue_print" -> ["blue", "print"]
It can still be found with the "blue_print" keyword,
as the search string is also split in two words.

-50% indexing time compared to previous commit (indexing is twice faster!)

* Index bigram for CJK content only

Inedxing time slightly longer (~3%),
but since the non-CJK content is not indexed,
bigram searchs will be slightly optimized.

* Fix group-based view_document permissions missing from bulk rebuild

_bulk_get_viewer_ids only queried UserObjectPermission, dropping the
group-permission expansion that get_users_with_perms(with_group_users=True)
performs for the non-batched per-document indexing path. A user who could
only see a document via group membership would lose search access to it
after any full reindex.

Also query GroupObjectPermission and expand group membership to user ids,
matching the existing single-document behavior.

* Yield (document, viewer_ids) pairs from _DocumentViewerStream

Previously _DocumentViewerStream.__iter__ yielded plain Document objects
while the matching viewer ids were exposed through a separate mutable
attribute (viewer_ids_by_pk), overwritten each time the generator crossed
a chunk boundary. rebuild() read that attribute out-of-band per document.

This only worked because the current iter_wrapper (a plain progress-bar
passthrough) happens to consume the stream in strict lock-step with no
lookahead. Any wrapper that buffers, batches, or reorders would silently
pair a document with the wrong chunk's viewer ids. Yield the pair directly
so the association travels with the document regardless of how iter_wrapper
consumes the stream, and drop the now-unneeded viewer_ids_by_pk attribute.

* Add --heap-size-mb CLI arg to document_index reindex

writer_heap_bytes was hardcoded at 512MB with no way to tune it. Expose it
as a manual-rebuild-only CLI arg rather than a settings/env var, per review
feedback, so lower-memory hosts can reduce it without a wider config
surface. Defaults to unset so TantivyBackend.rebuild's own default stays
the single source of truth.

---------

Co-authored-by: stumpylog <797416+stumpylog@users.noreply.github.com>

src/documents/search/_tokenizer.py

@@ -154,6 +154,23 @@ def simple_search_tokens(text: str) -> list[str]:
     return _SIMPLE_SEARCH_ANALYZER.analyze(text)
 
 
+# Autocomplete word extraction: tokenize -> lowercase -> ascii_fold in a single
+# Rust pass. Uses the simple tokenizer so extracted words match how document
+# content is actually indexed (the content tokenizer _paperless_text also uses
+# simple()), replacing a Python regex scan plus per-token folding.
+_AUTOCOMPLETE_ANALYZER: Final = (
+    tantivy.TextAnalyzerBuilder(tantivy.Tokenizer.simple())
+    .filter(tantivy.Filter.lowercase())
+    .filter(tantivy.Filter.ascii_fold())
+    .build()
+)
+
+
+def autocomplete_tokens(text: str) -> list[str]:
+    """Tokenize text into normalized autocomplete words (lowercased, ascii-folded)."""
+    return _AUTOCOMPLETE_ANALYZER.analyze(text)
+
+
 def ascii_fold(text: str) -> str:
     """Fold text to ASCII using the same mapping as the content tokenizers.