Per Playwright's own docs, routeFromHAR's notFound: 'fallback' sends
unmatched requests straight to the network -- it does not chain to
other, earlier-registered page.route() handlers via route.fallback()
the way I'd assumed. Since Playwright checks routes in
reverse-registration order, a handler registered in beforeEach (i.e.
before the test body's routeFromHAR call) is checked AFTER routeFromHAR,
whose catch-all pattern intercepts everything first and sends any miss
straight to the (nonexistent, in e2e) network -- my stub never got a
chance to run. Confirmed by a second CI failure with the same "Failed
to fetch" symptom even after the CORS-header fix.
Moved the mockFilterSelectionData(page) call to immediately after each
test's own routeFromHAR(...) call instead, so it's registered later and
checked first for that specific URL, with routeFromHAR's broader
pattern still handling everything else as before.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The e2e app (localhost:4200) calls the backend at localhost:8000, a
cross-origin request from the browser's perspective. Recorded HAR
responses carry the real backend's Access-Control-Allow-Origin header,
which is why they work; the stub added in the previous commit didn't
set one, so the browser rejected the fulfilled response as a CORS
violation -- same symptom as the original bug (TypeError: Failed to
fetch), confirmed via a fresh CI run after the first e2e fix attempt.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
CI (https://github.com/paperless-ngx/paperless-ngx/actions/runs/29777331059)
failed every e2e test that reloads the document list, with the browser
throwing "TypeError: Failed to fetch". Root cause: the e2e harness only
starts the Angular dev server (no real backend), and relies entirely on
page.routeFromHAR(..., { notFound: 'fallback' }) to mock API responses,
falling through to the real network for anything not in a recorded HAR.
The new GET /api/documents/filter_selection_data/ request added by this
branch fires on every non-search reload() and isn't in any pre-recorded
HAR fixture, so it fell through to a real network call with nothing
listening on the other end.
Added a shared mockFilterSelectionData() helper that stubs an empty
response for that endpoint, registered via test.beforeEach() before each
affected test's own routeFromHAR() call in every spec file that
exercises document-list reload (document-list, dashboard,
global-permissions, settings, document-detail). Playwright resolves
routes in reverse-registration order, so the later-registered HAR route
checks first, calls route.fallback() on a miss, and defers back to this
earlier-registered stub instead of hitting the network.
Not verified locally (Playwright browser binaries aren't installed in
this environment and the user preferred not to install them); relying
on CI to confirm.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Cleanups from a code-simplification pass over the prior commit:
- Extract the repeated "build HttpParams from a plain object, skipping
null/undefined" loop (previously duplicated in list(), getFew(), and the
new getFilterSelectionData()) into a single shared
AbstractPaperlessService.withParams() helper.
- document-list-view.service.ts: fire the filter_selection_data request
concurrently with the primary list request instead of nesting it inside
the list request's success callback. They're independent (the follow-up
only needs filterRules, not the list response), so there's no reason to
wait on one before starting the other. Extracted into
loadFilterSelectionData() for readability, and guarded the error path so
a failed list request cancels the concurrent follow-up too rather than
letting it resolve afterward and clobber the error-state reset.
- views.py: give filter_selection_data a lean base queryset
(_base_document_queryset) instead of the full get_queryset(), which
carries select_related/prefetch_related meant for serializing full
Document objects. Benchmarked this specifically: no measured query-time
impact in this path, since those calls were never iterating the
queryset directly and were already inert here -- keeping it for
clarity, not as a performance claim.
- Test fallout: since filter_selection_data now fires unconditionally on
every non-search reload() rather than only after a successful primary
flush, added a generic drain step to both spec files' afterEach so
individual tests don't each need to know about this follow-up request
unless they specifically assert on its response.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The document overview sent include_selection_data=true on every list
request (page/filter/sort change), computing 5 correlated Count(DISTINCT)
aggregations over the full matching queryset inline before the list could
render. At scale (hundreds of thousands of documents, unfiltered) this is
the confirmed root cause of the document overview "eternally loading" in
paperless-ngx/paperless-ngx#13161.
Split the aggregation into its own endpoint, GET
/api/documents/filter_selection_data/, filter-scoped the same way the
list endpoint already resolves matches (no document ID enumeration
needed). The frontend now fetches it as a separate, non-blocking request
after the list has already rendered, for plain (non-search) browsing.
Full-text search keeps computing it inline since results are already
narrowed by the search backend first.
include_selection_data never shipped in a stable release (introduced
this beta cycle, not present on main), so the plain list endpoint simply
stops acting on it rather than needing a deprecation path.
Also closes a latent localStorage leak between
document-list-view.service.spec.ts tests (filterRules persisted via
localStorage were never cleared, only sessionStorage was) that this
change's new URL-dependent follow-up request exposed, and fixes two
tests that were unknowingly relying on that leaked state to pass.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
apply_assignment_to_document() mutated tags directly on the shared document instance via add_nested_tags(), whose m2m_changed signal triggers update_filename_and_move_files() -> instance.refresh_from_db(), discarding any not-yet-saved fields (e.g. storage_path) staged by an earlier-ordered action in the same workflow. Apply tag changes to a freshly-fetched instance instead, matching the pattern already used in apply_removal_to_document().
Adds indexes on Document.checksum, Document.page_count, and a composite
(owner, created) index on Document, plus composite (field, value_*)
indexes on CustomFieldInstance for each typed value column.
Benchmarked against a 100k-document / 200k-custom-field-instance SQLite
dataset: Document.checksum lookups ~69x faster, page_count sort ~40x,
owner+created list queries ~6x. CustomFieldInstance composite indexes
show large gains (10-24x) on realistic narrow-selectivity queries
matching the workflow scheduler and custom-field search code paths.
Django's condition() decorator invokes etag_func and last_modified_func
separately, and the view itself may resolve again -- each call to
resolve_effective_document_by_pk() was redoing the same root/version
DB lookups. Memoize the resolution on the request object so a single
thumb/metadata/preview request resolves the effective document once
instead of up to three times.
Related to paperless-ngx/paperless-ngx#13161.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
The Merge and Push Manifest check isn't materialized until the
build-arch matrix completes (its `if` gates on a job output), which
routinely takes longer than the action's 60s discovery window,
causing the release workflow to bail before Docker builds finish.
* Tantivy: get permissions by chunks
-40% indexing time compared to previous commit
* Make progress bar process one by one with chunk
-15% indexing time compared to previous commit
* Prefetch FK + iterate over chunk from SQL
Prefetch additional needed data (note user, custom field content)
-20% indexing time compared to previous commit
* Reindex: increase Tantivy heap size from 128 to 512MB
Gains probably vary depending on the machine,
but it seems a sweet spot compatible with low-end hardware.
* Reindex: optimization on permission fetching and autocomplete word set
-10% indexing time compared to previous commit
* Autocomplete analyzer python->rust
Splits words with underscore compared to the python analyzer.
E.g.: "blue_print" -> ["blue", "print"]
It can still be found with the "blue_print" keyword,
as the search string is also split in two words.
-50% indexing time compared to previous commit (indexing is twice faster!)
* Index bigram for CJK content only
Inedxing time slightly longer (~3%),
but since the non-CJK content is not indexed,
bigram searchs will be slightly optimized.
* Fix group-based view_document permissions missing from bulk rebuild
_bulk_get_viewer_ids only queried UserObjectPermission, dropping the
group-permission expansion that get_users_with_perms(with_group_users=True)
performs for the non-batched per-document indexing path. A user who could
only see a document via group membership would lose search access to it
after any full reindex.
Also query GroupObjectPermission and expand group membership to user ids,
matching the existing single-document behavior.
* Yield (document, viewer_ids) pairs from _DocumentViewerStream
Previously _DocumentViewerStream.__iter__ yielded plain Document objects
while the matching viewer ids were exposed through a separate mutable
attribute (viewer_ids_by_pk), overwritten each time the generator crossed
a chunk boundary. rebuild() read that attribute out-of-band per document.
This only worked because the current iter_wrapper (a plain progress-bar
passthrough) happens to consume the stream in strict lock-step with no
lookahead. Any wrapper that buffers, batches, or reorders would silently
pair a document with the wrong chunk's viewer ids. Yield the pair directly
so the association travels with the document regardless of how iter_wrapper
consumes the stream, and drop the now-unneeded viewer_ids_by_pk attribute.
* Add --heap-size-mb CLI arg to document_index reindex
writer_heap_bytes was hardcoded at 512MB with no way to tune it. Expose it
as a manual-rebuild-only CLI arg rather than a settings/env var, per review
feedback, so lower-memory hosts can reduce it without a wider config
surface. Defaults to unset so TantivyBackend.rebuild's own default stays
the single source of truth.
---------
Co-authored-by: stumpylog <797416+stumpylog@users.noreply.github.com>
Trenton HolmesandClaude Sonnet 5
shamoonandGitHub
GitHub Actions
dependabot[bot]andGitHub
Hugo Beauzée-LuyssenandGitHub
Albert MikaelyanandGitHub
