Full category (not just the ruff-0.16 default subset). 35 hits:
- 4 PLW0108 (unnecessary lambda) autofixed
- 6 PLW2901 (loop/with variable shadowed) renamed to distinct names
- 2 PLW1510 (subprocess.run without explicit check) given check=False,
matching existing behavior exactly
- 6 PLW0602 (global declared but never assigned) removed - these were
all in-place mutations (.append/.insert), not reassignments, so
`global` was already a no-op
- 7 PLW0603 (global statement) suppressed with noqa - these are
genuine lazy-init singletons with no class to hold the state;
refactoring them is a separate, larger change
- 6 PLW1508 (non-str/None env var default) fixed using the existing
get_int_from_env/get_float_from_env typed helpers instead of raw
os.getenv, which also fixes a real bug: LOGROTATE_MAX_SIZE and
LOGROTATE_MAX_BACKUPS were never wrapped in int(), so a string env
var value would have flowed into RotatingFileHandler as a string
- 1 PLW1641 (__eq__ without __hash__) fixed by adding __hash__ to
PlaceholderString
* Fix: exclude next-period start from relative date-range filters
Tantivy's [lo TO hi] range is inclusive on both ends, but computed upper
bounds (keyword ranges, YYYY/YYYYMM/YYYYMMDD tokens) represent the start of
the next period. Use half-open [lo TO hi} for those so e.g. "previous month"
no longer matches the 1st of the current month.
* Adds a regression test down to the second check for the hi range
* Tantivy: get permissions by chunks
-40% indexing time compared to previous commit
* Make progress bar process one by one with chunk
-15% indexing time compared to previous commit
* Prefetch FK + iterate over chunk from SQL
Prefetch additional needed data (note user, custom field content)
-20% indexing time compared to previous commit
* Reindex: increase Tantivy heap size from 128 to 512MB
Gains probably vary depending on the machine,
but it seems a sweet spot compatible with low-end hardware.
* Reindex: optimization on permission fetching and autocomplete word set
-10% indexing time compared to previous commit
* Autocomplete analyzer python->rust
Splits words with underscore compared to the python analyzer.
E.g.: "blue_print" -> ["blue", "print"]
It can still be found with the "blue_print" keyword,
as the search string is also split in two words.
-50% indexing time compared to previous commit (indexing is twice faster!)
* Index bigram for CJK content only
Inedxing time slightly longer (~3%),
but since the non-CJK content is not indexed,
bigram searchs will be slightly optimized.
* Fix group-based view_document permissions missing from bulk rebuild
_bulk_get_viewer_ids only queried UserObjectPermission, dropping the
group-permission expansion that get_users_with_perms(with_group_users=True)
performs for the non-batched per-document indexing path. A user who could
only see a document via group membership would lose search access to it
after any full reindex.
Also query GroupObjectPermission and expand group membership to user ids,
matching the existing single-document behavior.
* Yield (document, viewer_ids) pairs from _DocumentViewerStream
Previously _DocumentViewerStream.__iter__ yielded plain Document objects
while the matching viewer ids were exposed through a separate mutable
attribute (viewer_ids_by_pk), overwritten each time the generator crossed
a chunk boundary. rebuild() read that attribute out-of-band per document.
This only worked because the current iter_wrapper (a plain progress-bar
passthrough) happens to consume the stream in strict lock-step with no
lookahead. Any wrapper that buffers, batches, or reorders would silently
pair a document with the wrong chunk's viewer ids. Yield the pair directly
so the association travels with the document regardless of how iter_wrapper
consumes the stream, and drop the now-unneeded viewer_ids_by_pk attribute.
* Add --heap-size-mb CLI arg to document_index reindex
writer_heap_bytes was hardcoded at 512MB with no way to tune it. Expose it
as a manual-rebuild-only CLI arg rather than a settings/env var, per review
feedback, so lower-memory hosts can reduce it without a wider config
surface. Defaults to unset so TantivyBackend.rebuild's own default stays
the single source of truth.
---------
Co-authored-by: stumpylog <797416+stumpylog@users.noreply.github.com>
Implements and tests a retry with backoff + jitter for aquring the index update lock. If we still can't get it, dispatch a celery task to handle it later instead (also with retry)
Signed-off-by: stumpylog <797416+stumpylog@users.noreply.github.com>
stumpylog
shamoonandGitHub
Matthias MastandGitHub
