Compare commits

...
Author SHA1 Message Date
Trenton HolmesandClaude Sonnet 5 ff0ce4d123 Fix: register the filter_selection_data e2e stub after routeFromHAR, not before
Per Playwright's own docs, routeFromHAR's notFound: 'fallback' sends
unmatched requests straight to the network -- it does not chain to
other, earlier-registered page.route() handlers via route.fallback()
the way I'd assumed. Since Playwright checks routes in
reverse-registration order, a handler registered in beforeEach (i.e.
before the test body's routeFromHAR call) is checked AFTER routeFromHAR,
whose catch-all pattern intercepts everything first and sends any miss
straight to the (nonexistent, in e2e) network -- my stub never got a
chance to run. Confirmed by a second CI failure with the same "Failed
to fetch" symptom even after the CORS-header fix.

Moved the mockFilterSelectionData(page) call to immediately after each
test's own routeFromHAR(...) call instead, so it's registered later and
checked first for that specific URL, with routeFromHAR's broader
pattern still handling everything else as before.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 14:54:58 -07:00
Trenton HolmesandClaude Sonnet 5 8b3665b32d Fix: add CORS header to the e2e filter_selection_data stub response
The e2e app (localhost:4200) calls the backend at localhost:8000, a
cross-origin request from the browser's perspective. Recorded HAR
responses carry the real backend's Access-Control-Allow-Origin header,
which is why they work; the stub added in the previous commit didn't
set one, so the browser rejected the fulfilled response as a CORS
violation -- same symptom as the original bug (TypeError: Failed to
fetch), confirmed via a fresh CI run after the first e2e fix attempt.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 14:32:59 -07:00
Trenton HolmesandClaude Sonnet 5 9cce6d1b68 Fix: mock the new filter_selection_data request in Playwright e2e tests
CI (https://github.com/paperless-ngx/paperless-ngx/actions/runs/29777331059)
failed every e2e test that reloads the document list, with the browser
throwing "TypeError: Failed to fetch". Root cause: the e2e harness only
starts the Angular dev server (no real backend), and relies entirely on
page.routeFromHAR(..., { notFound: 'fallback' }) to mock API responses,
falling through to the real network for anything not in a recorded HAR.
The new GET /api/documents/filter_selection_data/ request added by this
branch fires on every non-search reload() and isn't in any pre-recorded
HAR fixture, so it fell through to a real network call with nothing
listening on the other end.

Added a shared mockFilterSelectionData() helper that stubs an empty
response for that endpoint, registered via test.beforeEach() before each
affected test's own routeFromHAR() call in every spec file that
exercises document-list reload (document-list, dashboard,
global-permissions, settings, document-detail). Playwright resolves
routes in reverse-registration order, so the later-registered HAR route
checks first, calls route.fallback() on a miss, and defers back to this
earlier-registered stub instead of hitting the network.

Not verified locally (Playwright browser binaries aren't installed in
this environment and the user preferred not to install them); relying
on CI to confirm.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 14:21:51 -07:00
Trenton HolmesandClaude Sonnet 5 bb77e65d52 Simplify: dedup HttpParams building, fire selection-data concurrently
Cleanups from a code-simplification pass over the prior commit:

- Extract the repeated "build HttpParams from a plain object, skipping
  null/undefined" loop (previously duplicated in list(), getFew(), and the
  new getFilterSelectionData()) into a single shared
  AbstractPaperlessService.withParams() helper.

- document-list-view.service.ts: fire the filter_selection_data request
  concurrently with the primary list request instead of nesting it inside
  the list request's success callback. They're independent (the follow-up
  only needs filterRules, not the list response), so there's no reason to
  wait on one before starting the other. Extracted into
  loadFilterSelectionData() for readability, and guarded the error path so
  a failed list request cancels the concurrent follow-up too rather than
  letting it resolve afterward and clobber the error-state reset.

- views.py: give filter_selection_data a lean base queryset
  (_base_document_queryset) instead of the full get_queryset(), which
  carries select_related/prefetch_related meant for serializing full
  Document objects. Benchmarked this specifically: no measured query-time
  impact in this path, since those calls were never iterating the
  queryset directly and were already inert here -- keeping it for
  clarity, not as a performance claim.

- Test fallout: since filter_selection_data now fires unconditionally on
  every non-search reload() rather than only after a successful primary
  flush, added a generic drain step to both spec files' afterEach so
  individual tests don't each need to know about this follow-up request
  unless they specifically assert on its response.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 10:00:08 -07:00
Trenton HolmesandClaude Sonnet 5 eb5bf53476 Fix (beta): stop blocking the document list on selection-data aggregation
The document overview sent include_selection_data=true on every list
request (page/filter/sort change), computing 5 correlated Count(DISTINCT)
aggregations over the full matching queryset inline before the list could
render. At scale (hundreds of thousands of documents, unfiltered) this is
the confirmed root cause of the document overview "eternally loading" in
paperless-ngx/paperless-ngx#13161.

Split the aggregation into its own endpoint, GET
/api/documents/filter_selection_data/, filter-scoped the same way the
list endpoint already resolves matches (no document ID enumeration
needed). The frontend now fetches it as a separate, non-blocking request
after the list has already rendered, for plain (non-search) browsing.
Full-text search keeps computing it inline since results are already
narrowed by the search backend first.

include_selection_data never shipped in a stable release (introduced
this beta cycle, not present on main), so the plain list endpoint simply
stops acting on it rather than needing a deprecation path.

Also closes a latent localStorage leak between
document-list-view.service.spec.ts tests (filterRules persisted via
localStorage were never cleared, only sessionStorage was) that this
change's new URL-dependent follow-up request exposed, and fixes two
tests that were unknowingly relying on that leaked state to pass.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 10:00:08 -07:00
shamoonandGitHub ff609c2987 Change: update root modified timestamp on version changes (#13170) 2026-07-20 15:45:24 +00:00
Trenton HandGitHub 2b784e709b Fix: prevent tag assignment from reverting other pending workflow assignments (#13178)
apply_assignment_to_document() mutated tags directly on the shared document instance via add_nested_tags(), whose m2m_changed signal triggers update_filename_and_move_files() -> instance.refresh_from_db(), discarding any not-yet-saved fields (e.g. storage_path) staged by an earlier-ordered action in the same workflow. Apply tag changes to a freshly-fetched instance instead, matching the pattern already used in apply_removal_to_document().
2026-07-20 08:22:30 -07:00
Trenton HandGitHub c9716252f0 Performance: Add DB indexes for common query/sort patterns (#13167)
Adds indexes on Document.checksum, Document.page_count, and a composite
(owner, created) index on Document, plus composite (field, value_*)
indexes on CustomFieldInstance for each typed value column.

Benchmarked against a 100k-document / 200k-custom-field-instance SQLite
dataset: Document.checksum lookups ~69x faster, page_count sort ~40x,
owner+created list queries ~6x. CustomFieldInstance composite indexes
show large gains (10-24x) on realistic narrow-selectivity queries
matching the workflow scheduler and custom-field search code paths.
2026-07-20 13:12:54 +00:00
shamoon 5cf9152a40 Merge branch 'dev' into beta 2026-07-19 20:20:48 -07:00
4e52dd5710 Fix: cache per-request effective-document resolution for thumb/metadata/preview (#13166)
Django's condition() decorator invokes etag_func and last_modified_func
separately, and the view itself may resolve again -- each call to
resolve_effective_document_by_pk() was redoing the same root/version
DB lookups. Memoize the resolution on the request object so a single
thumb/metadata/preview request resolves the effective document once
instead of up to three times.

Related to paperless-ngx/paperless-ngx#13161.

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-19 18:04:16 -07:00
23 changed files with 547 additions and 134 deletions
+4
View File
@@ -1,5 +1,6 @@
import { expect, test } from '@playwright/test'
import path from 'node:path'
import { mockFilterSelectionData } from '../mock-filter-selection-data'
const REQUESTS_HAR = path.join(__dirname, 'requests/api-settings.har')
@@ -7,6 +8,7 @@ test('should activate / deactivate save button when settings change', async ({
page,
}) => {
await page.routeFromHAR(REQUESTS_HAR, { notFound: 'fallback' })
await mockFilterSelectionData(page)
await page.goto('/settings')
await expect(page.getByRole('button', { name: 'Save' })).toBeDisabled()
await page.getByLabel('Use system setting').click()
@@ -16,6 +18,7 @@ test('should activate / deactivate save button when settings change', async ({
test('should warn on unsaved changes', async ({ page }) => {
await page.routeFromHAR(REQUESTS_HAR, { notFound: 'fallback' })
await mockFilterSelectionData(page)
await page.goto('/settings')
await page.getByLabel('Use system setting').click()
await page.getByRole('link', { name: 'Dashboard' }).click()
@@ -28,6 +31,7 @@ test('should warn on unsaved changes', async ({ page }) => {
test('should apply appearance changes when set', async ({ page }) => {
await page.routeFromHAR(REQUESTS_HAR, { notFound: 'fallback' })
await mockFilterSelectionData(page)
await page.goto('/settings')
await expect(page.locator('html')).toHaveAttribute('data-bs-theme', /auto/)
await page.getByLabel('Use system setting').click()
+6
View File
@@ -1,5 +1,6 @@
import { expect, test } from '@playwright/test'
import path from 'node:path'
import { mockFilterSelectionData } from '../mock-filter-selection-data'
const REQUESTS_HAR1 = path.join(__dirname, 'requests/api-dashboard1.har')
const REQUESTS_HAR2 = path.join(__dirname, 'requests/api-dashboard2.har')
@@ -8,6 +9,7 @@ const REQUESTS_HAR4 = path.join(__dirname, 'requests/api-dashboard4.har')
test('dashboard inbox link', async ({ page }) => {
await page.routeFromHAR(REQUESTS_HAR1, { notFound: 'fallback' })
await mockFilterSelectionData(page)
await page.goto('/dashboard')
await page.getByRole('link', { name: 'Documents in inbox' }).click()
await expect(page).toHaveURL(/tags__id__in=9/)
@@ -16,6 +18,7 @@ test('dashboard inbox link', async ({ page }) => {
test('dashboard total documents link', async ({ page }) => {
await page.routeFromHAR(REQUESTS_HAR2, { notFound: 'fallback' })
await mockFilterSelectionData(page)
await page.goto('/dashboard')
await page.getByRole('link').filter({ hasText: 'Total documents' }).click()
await expect(page).toHaveURL(/documents/)
@@ -25,6 +28,7 @@ test('dashboard total documents link', async ({ page }) => {
test('dashboard saved view show all', async ({ page }) => {
await page.routeFromHAR(REQUESTS_HAR3, { notFound: 'fallback' })
await mockFilterSelectionData(page)
await page.goto('/dashboard')
await page
.locator('pngx-widget-frame')
@@ -38,6 +42,7 @@ test('dashboard saved view show all', async ({ page }) => {
test('dashboard saved view document links', async ({ page }) => {
await page.routeFromHAR(REQUESTS_HAR4, { notFound: 'fallback' })
await mockFilterSelectionData(page)
await page.goto('/dashboard')
await page
.locator('pngx-widget-frame')
@@ -51,6 +56,7 @@ test('dashboard saved view document links', async ({ page }) => {
test('test slim sidebar', async ({ page }) => {
await page.routeFromHAR(REQUESTS_HAR1, { notFound: 'fallback' })
await mockFilterSelectionData(page)
await page.goto('/dashboard')
await page.locator('.sidebar-slim-toggler').click()
await expect(
@@ -1,5 +1,6 @@
import { expect, test } from '@playwright/test'
import path from 'node:path'
import { mockFilterSelectionData } from '../mock-filter-selection-data'
const REQUESTS_HAR = path.join(__dirname, 'requests/api-document-detail.har')
const REQUESTS_HAR2 = path.join(__dirname, 'requests/api-document-detail2.har')
@@ -8,6 +9,7 @@ test('should activate / deactivate save button when changes are saved', async ({
page,
}) => {
await page.routeFromHAR(REQUESTS_HAR, { notFound: 'fallback' })
await mockFilterSelectionData(page)
await page.goto('/documents/175/')
await page.waitForSelector('pngx-document-detail pngx-input-text:first-child')
await expect(page.getByTitle('Storage path', { exact: true })).toHaveText(
@@ -20,6 +22,7 @@ test('should activate / deactivate save button when changes are saved', async ({
test('should warn on unsaved changes', async ({ page }) => {
await page.routeFromHAR(REQUESTS_HAR, { notFound: 'fallback' })
await mockFilterSelectionData(page)
await page.goto('/documents/175/')
await expect(page.getByTitle('Correspondent', { exact: true })).toHaveText(
/\w+/
@@ -39,6 +42,7 @@ test('should warn on unsaved changes', async ({ page }) => {
test('should support tab direct navigation', async ({ page }) => {
await page.routeFromHAR(REQUESTS_HAR, { notFound: 'fallback' })
await mockFilterSelectionData(page)
await page.goto('/documents/175/details')
await expect(page.getByRole('tab', { name: 'Details' })).toHaveAttribute(
'aria-selected',
@@ -68,6 +72,7 @@ test('should support tab direct navigation', async ({ page }) => {
test('should show a mobile preview', async ({ page }) => {
await page.routeFromHAR(REQUESTS_HAR, { notFound: 'fallback' })
await mockFilterSelectionData(page)
await page.goto('/documents/175/')
await page.setViewportSize({ width: 400, height: 1000 })
await expect(page.getByRole('tab', { name: 'Preview' })).toBeVisible()
@@ -77,6 +82,7 @@ test('should show a mobile preview', async ({ page }) => {
test('should show a list of notes', async ({ page }) => {
await page.routeFromHAR(REQUESTS_HAR, { notFound: 'fallback' })
await mockFilterSelectionData(page)
await page.goto('/documents/175/notes')
await expect(page.locator('pngx-document-notes')).toBeVisible()
await expect(
@@ -89,6 +95,7 @@ test('should show a list of notes', async ({ page }) => {
test('should support quick filters', async ({ page }) => {
await page.routeFromHAR(REQUESTS_HAR2, { notFound: 'fallback' })
await mockFilterSelectionData(page)
await page.goto('/documents/175/details')
await page
.getByRole('button', { name: 'Filter documents with these Tags' })
@@ -1,5 +1,6 @@
import { expect, test } from '@playwright/test'
import path from 'node:path'
import { mockFilterSelectionData } from '../mock-filter-selection-data'
const REQUESTS_HAR1 = path.join(__dirname, 'requests/api-document-list1.har')
const REQUESTS_HAR2 = path.join(__dirname, 'requests/api-document-list2.har')
@@ -10,6 +11,7 @@ const REQUESTS_HAR6 = path.join(__dirname, 'requests/api-document-list6.har')
test('basic filtering', async ({ page }) => {
await page.routeFromHAR(REQUESTS_HAR1, { notFound: 'fallback' })
await mockFilterSelectionData(page)
await page.goto('/documents')
await page.getByRole('button', { name: 'Tags' }).click()
await page.getByRole('menuitem', { name: 'Inbox' }).click()
@@ -45,6 +47,7 @@ test('basic filtering', async ({ page }) => {
test('text filtering', async ({ page }) => {
await page.routeFromHAR(REQUESTS_HAR2, { notFound: 'fallback' })
await mockFilterSelectionData(page)
await page.goto('/documents')
await page.getByRole('main').getByRole('combobox').click()
await page.getByRole('main').getByRole('combobox').fill('test')
@@ -81,6 +84,7 @@ test('text filtering', async ({ page }) => {
test('date filtering', async ({ page }) => {
await page.routeFromHAR(REQUESTS_HAR3, { notFound: 'fallback' })
await mockFilterSelectionData(page)
await page.goto('/documents')
await page.getByRole('button', { name: 'Dates' }).click()
await page.locator('.ng-arrow-wrapper').first().click()
@@ -103,6 +107,7 @@ test('date filtering', async ({ page }) => {
test('sorting', async ({ page }) => {
await page.routeFromHAR(REQUESTS_HAR4, { notFound: 'fallback' })
await mockFilterSelectionData(page)
await page.goto('/documents')
await page.getByRole('button', { name: 'Sort' }).click()
await page.getByRole('button', { name: 'ASN' }).click()
@@ -141,6 +146,7 @@ test('sorting', async ({ page }) => {
test('change views', async ({ page }) => {
await page.routeFromHAR(REQUESTS_HAR5, { notFound: 'fallback' })
await mockFilterSelectionData(page)
await page.goto('/documents')
await page.locator('.btn-group > label').first().click()
await expect(page.locator('pngx-document-list table')).toBeVisible()
@@ -152,6 +158,7 @@ test('change views', async ({ page }) => {
test('bulk edit', async ({ page }) => {
await page.routeFromHAR(REQUESTS_HAR6, { notFound: 'fallback' })
await mockFilterSelectionData(page)
await page.goto('/documents')
await page.locator('pngx-document-card-small').nth(0).click()
+35
View File
@@ -0,0 +1,35 @@
import { Page } from '@playwright/test'
const EMPTY_SELECTION_DATA = {
selected_correspondents: [],
selected_tags: [],
selected_document_types: [],
selected_storage_paths: [],
selected_custom_fields: [],
}
/**
* The document list now fires a GET to filter_selection_data on every
* non-search reload(), independent of and concurrent with the main list
* request. It's not present in any of the recorded HAR fixtures, so with
* `notFound: 'fallback'` it would otherwise fall through to the real
* network (nothing listens there in e2e, since only the frontend dev
* server is started) and fail every test that reloads the list.
*
* Playwright checks routes in reverse-registration order, so this must be
* registered before a test's own page.routeFromHAR() call for the HAR
* route's `notFound: 'fallback'` to defer back to this one.
*/
export async function mockFilterSelectionData(page: Page) {
await page.route('**/api/documents/filter_selection_data/**', (route) =>
route.fulfill({
json: EMPTY_SELECTION_DATA,
// The app calls the (cross-origin, from the e2e app's perspective)
// backend at http://localhost:8000 while served from :4200, so a
// fulfilled response needs the same CORS header the real backend
// sends (and that recorded HAR responses already carry) or the
// browser rejects it as a cross-origin failure.
headers: { 'Access-Control-Allow-Origin': 'http://localhost:4200' },
})
)
}
@@ -1,10 +1,12 @@
import { expect, test } from '@playwright/test'
import path from 'node:path'
import { mockFilterSelectionData } from '../mock-filter-selection-data'
const REQUESTS_HAR = path.join(__dirname, 'requests/api-global-permissions.har')
test('should not allow user to edit settings', async ({ page }) => {
await page.routeFromHAR(REQUESTS_HAR, { notFound: 'fallback' })
await mockFilterSelectionData(page)
await page.goto('/dashboard')
await expect(page.getByRole('link', { name: 'Settings' })).not.toBeAttached()
await page.goto('/settings')
@@ -15,6 +17,7 @@ test('should not allow user to edit settings', async ({ page }) => {
test('should not allow user to view documents', async ({ page }) => {
await page.routeFromHAR(REQUESTS_HAR, { notFound: 'fallback' })
await mockFilterSelectionData(page)
await page.goto('/dashboard')
await expect(
page.locator('nav').getByRole('link', { name: 'Documents' })
@@ -31,6 +34,7 @@ test('should not allow user to view documents', async ({ page }) => {
test('should not allow user to view correspondents', async ({ page }) => {
await page.routeFromHAR(REQUESTS_HAR, { notFound: 'fallback' })
await mockFilterSelectionData(page)
await page.goto('/dashboard')
await expect(
page.getByRole('link', { name: 'Attributes' })
@@ -43,6 +47,7 @@ test('should not allow user to view correspondents', async ({ page }) => {
test('should not allow user to view tags', async ({ page }) => {
await page.routeFromHAR(REQUESTS_HAR, { notFound: 'fallback' })
await mockFilterSelectionData(page)
await page.goto('/dashboard')
await expect(
page.getByRole('link', { name: 'Attributes' })
@@ -55,6 +60,7 @@ test('should not allow user to view tags', async ({ page }) => {
test('should not allow user to view document types', async ({ page }) => {
await page.routeFromHAR(REQUESTS_HAR, { notFound: 'fallback' })
await mockFilterSelectionData(page)
await page.goto('/dashboard')
await expect(
page.getByRole('link', { name: 'Attributes' })
@@ -67,6 +73,7 @@ test('should not allow user to view document types', async ({ page }) => {
test('should not allow user to view storage paths', async ({ page }) => {
await page.routeFromHAR(REQUESTS_HAR, { notFound: 'fallback' })
await mockFilterSelectionData(page)
await page.goto('/dashboard')
await expect(
page.getByRole('link', { name: 'Attributes' })
@@ -79,6 +86,7 @@ test('should not allow user to view storage paths', async ({ page }) => {
test('should not allow user to view logs', async ({ page }) => {
await page.routeFromHAR(REQUESTS_HAR, { notFound: 'fallback' })
await mockFilterSelectionData(page)
await page.goto('/dashboard')
await expect(page.getByRole('link', { name: 'Logs' })).not.toBeAttached()
await page.goto('/logs')
@@ -89,6 +97,7 @@ test('should not allow user to view logs', async ({ page }) => {
test('should not allow user to view tasks', async ({ page }) => {
await page.routeFromHAR(REQUESTS_HAR, { notFound: 'fallback' })
await mockFilterSelectionData(page)
await page.goto('/dashboard')
await expect(page.getByRole('link', { name: 'Tasks' })).not.toBeAttached()
await page.goto('/tasks')
@@ -191,6 +191,14 @@ describe('BulkEditorComponent', () => {
})
afterEach(async () => {
// A filter_selection_data request now fires concurrently with every
// non-search reload(), independent of whether a given test flushes or
// even inspects the primary list response. Drain any left unclaimed.
httpTestingController.match(
(request) =>
request.url ===
`${environment.apiBaseUrl}documents/filter_selection_data/`
)
httpTestingController.verify()
})
@@ -386,7 +394,7 @@ describe('BulkEditorComponent', () => {
parameters: { add_tags: [101], remove_tags: [] },
})
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
) // list reload
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
@@ -432,7 +440,7 @@ describe('BulkEditorComponent', () => {
parameters: { add_tags: [101], remove_tags: [] },
})
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
) // list reload
})
@@ -461,7 +469,7 @@ describe('BulkEditorComponent', () => {
.expectOne(`${environment.apiBaseUrl}documents/bulk_edit/`)
.flush(true)
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
) // list reload
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
@@ -552,7 +560,7 @@ describe('BulkEditorComponent', () => {
parameters: { correspondent: 101 },
})
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
) // list reload
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
@@ -584,7 +592,7 @@ describe('BulkEditorComponent', () => {
.expectOne(`${environment.apiBaseUrl}documents/bulk_edit/`)
.flush(true)
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
) // list reload
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
@@ -650,7 +658,7 @@ describe('BulkEditorComponent', () => {
parameters: { document_type: 101 },
})
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
) // list reload
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
@@ -682,7 +690,7 @@ describe('BulkEditorComponent', () => {
.expectOne(`${environment.apiBaseUrl}documents/bulk_edit/`)
.flush(true)
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
) // list reload
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
@@ -748,7 +756,7 @@ describe('BulkEditorComponent', () => {
parameters: { storage_path: 101 },
})
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
) // list reload
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
@@ -780,7 +788,7 @@ describe('BulkEditorComponent', () => {
.expectOne(`${environment.apiBaseUrl}documents/bulk_edit/`)
.flush(true)
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
) // list reload
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
@@ -846,7 +854,7 @@ describe('BulkEditorComponent', () => {
parameters: { add_custom_fields: [101], remove_custom_fields: [102] },
})
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
) // list reload
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
@@ -878,7 +886,7 @@ describe('BulkEditorComponent', () => {
.expectOne(`${environment.apiBaseUrl}documents/bulk_edit/`)
.flush(true)
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
) // list reload
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
@@ -987,7 +995,7 @@ describe('BulkEditorComponent', () => {
documents: [3, 4],
})
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
) // list reload
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
@@ -1080,7 +1088,7 @@ describe('BulkEditorComponent', () => {
documents: [3, 4],
})
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
) // list reload
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
@@ -1115,7 +1123,7 @@ describe('BulkEditorComponent', () => {
source_mode: 'latest_version',
})
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
) // list reload
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
@@ -1156,7 +1164,7 @@ describe('BulkEditorComponent', () => {
metadata_document_id: 3,
})
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
) // list reload
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
@@ -1175,7 +1183,7 @@ describe('BulkEditorComponent', () => {
delete_originals: true,
})
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
) // list reload
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
@@ -1196,7 +1204,7 @@ describe('BulkEditorComponent', () => {
archive_fallback: true,
})
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
) // list reload
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
@@ -1299,7 +1307,7 @@ describe('BulkEditorComponent', () => {
},
})
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
) // list reload
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
@@ -1607,7 +1615,7 @@ describe('BulkEditorComponent', () => {
expect(toastServiceShowInfoSpy).toHaveBeenCalled()
expect(listReloadSpy).toHaveBeenCalled()
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
) // list reload
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
@@ -84,6 +84,28 @@ const view: SavedView = {
filter_rules: filterRules,
}
const emptySelectionData = {
selected_correspondents: [],
selected_tags: [],
selected_document_types: [],
selected_storage_paths: [],
selected_custom_fields: [],
}
// A successful (non-search) list response now triggers a separate,
// non-blocking request for filter dropdown counts. Tests that flush a
// successful list response need to also flush this follow-up request.
function flushSelectionDataRequest(
httpTestingController: HttpTestingController,
querySuffix: string = ''
) {
const req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/filter_selection_data/${querySuffix}`
)
expect(req.request.method).toEqual('GET')
req.flush(emptySelectionData)
}
describe('DocumentListViewService', () => {
let httpTestingController: HttpTestingController
let documentListViewService: DocumentListViewService
@@ -105,6 +127,7 @@ describe('DocumentListViewService', () => {
})
sessionStorage.clear()
localStorage.clear()
httpTestingController = TestBed.inject(HttpTestingController)
documentListViewService = TestBed.inject(DocumentListViewService)
settingsService = TestBed.inject(SettingsService)
@@ -114,8 +137,19 @@ describe('DocumentListViewService', () => {
afterEach(() => {
documentListViewService.cancelPending()
// A filter_selection_data request now fires concurrently with every
// non-search reload(), independent of whether the test cares about or
// flushes the primary list response. Drain any that a test didn't
// explicitly claim via flushSelectionDataRequest, so unrelated tests
// don't have to know about this follow-up request to pass verify().
httpTestingController.match(
(request) =>
request.url ===
`${environment.apiBaseUrl}documents/filter_selection_data/`
)
httpTestingController.verify()
sessionStorage.clear()
localStorage.clear()
})
afterAll(() => {
@@ -128,10 +162,11 @@ describe('DocumentListViewService', () => {
expect(documentListViewService.currentPage).toEqual(1)
documentListViewService.reload()
const req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
)
expect(req.request.method).toEqual('GET')
req.flush(full_results)
flushSelectionDataRequest(httpTestingController)
expect(req.request.method).toEqual('GET')
expect(documentListViewService.isReloading).toBeFalsy()
expect(documentListViewService.activeSavedViewId).toBeNull()
@@ -143,12 +178,12 @@ describe('DocumentListViewService', () => {
it('should handle error on page request out of range', () => {
documentListViewService.currentPage = 50
let req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/?page=50&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=50&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
)
expect(req.request.method).toEqual('GET')
req.flush([], { status: 404, statusText: 'Unexpected error' })
req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
)
expect(req.request.method).toEqual('GET')
expect(documentListViewService.currentPage).toEqual(1)
@@ -165,21 +200,20 @@ describe('DocumentListViewService', () => {
]
documentListViewService.setFilterRules(filterRulesAny)
let req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true&tags__id__in=${tags__id__in}`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false&tags__id__in=${tags__id__in}`
)
expect(req.request.method).toEqual('GET')
req.flush(
{ archive_serial_number: 'hello' },
{ status: 404, statusText: 'Unexpected error' }
)
req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
)
expect(req.request.method).toEqual('GET')
// the error is a plain field error (not a page-out-of-range or deleted
// custom-field-sort case), so no automatic retry request is sent here
expect(documentListViewService.error).toBeTruthy()
// reset the list
documentListViewService.setFilterRules([])
req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
)
})
@@ -187,7 +221,7 @@ describe('DocumentListViewService', () => {
documentListViewService.currentPage = 1
documentListViewService.sortField = 'custom_field_999'
let req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-custom_field_999&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-custom_field_999&truncate_content=true&include_selection_data=false`
)
expect(req.request.method).toEqual('GET')
req.flush(
@@ -196,7 +230,7 @@ describe('DocumentListViewService', () => {
)
// resets itself
req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
)
})
@@ -211,7 +245,7 @@ describe('DocumentListViewService', () => {
]
documentListViewService.setFilterRules(filterRulesAny)
let req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true&tags__id__in=${tags__id__in}`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false&tags__id__in=${tags__id__in}`
)
expect(req.request.method).toEqual('GET')
req.flush('Generic error', { status: 404, statusText: 'Unexpected error' })
@@ -219,7 +253,7 @@ describe('DocumentListViewService', () => {
// reset the list
documentListViewService.setFilterRules([])
req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
)
})
@@ -228,7 +262,7 @@ describe('DocumentListViewService', () => {
expect(documentListViewService.sortReverse).toBeTruthy()
documentListViewService.setSort('added', false)
let req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=added&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=added&truncate_content=true&include_selection_data=false`
)
expect(req.request.method).toEqual('GET')
expect(documentListViewService.sortField).toEqual('added')
@@ -236,12 +270,12 @@ describe('DocumentListViewService', () => {
documentListViewService.sortField = 'created'
req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=created&truncate_content=true&include_selection_data=false`
)
expect(documentListViewService.sortField).toEqual('created')
documentListViewService.sortReverse = true
req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
)
expect(req.request.method).toEqual('GET')
expect(documentListViewService.sortReverse).toBeTruthy()
@@ -284,7 +318,7 @@ describe('DocumentListViewService', () => {
const req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/?page=${page}&page_size=${
documentListViewService.pageSize
}&ordering=${reverse ? '-' : ''}${sort}&truncate_content=true&include_selection_data=true`
}&ordering=${reverse ? '-' : ''}${sort}&truncate_content=true&include_selection_data=false`
)
expect(req.request.method).toEqual('GET')
expect(documentListViewService.currentPage).toEqual(page)
@@ -301,7 +335,7 @@ describe('DocumentListViewService', () => {
}
documentListViewService.loadFromQueryParams(convertToParamMap(params))
let req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/?page=${documentListViewService.currentPage}&page_size=${documentListViewService.pageSize}&ordering=-added&truncate_content=true&include_selection_data=true&tags__id__all=${tags__id__all}`
`${environment.apiBaseUrl}documents/?page=${documentListViewService.currentPage}&page_size=${documentListViewService.pageSize}&ordering=-added&truncate_content=true&include_selection_data=false&tags__id__all=${tags__id__all}`
)
expect(req.request.method).toEqual('GET')
expect(documentListViewService.filterRules).toEqual([
@@ -311,12 +345,16 @@ describe('DocumentListViewService', () => {
},
])
req.flush(full_results)
flushSelectionDataRequest(
httpTestingController,
`?tags__id__all=${tags__id__all}`
)
})
it('should use filter rules to update query params', () => {
documentListViewService.setFilterRules(filterRules)
const req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/?page=${documentListViewService.currentPage}&page_size=${documentListViewService.pageSize}&ordering=-created&truncate_content=true&include_selection_data=true&tags__id__all=${tags__id__all}`
`${environment.apiBaseUrl}documents/?page=${documentListViewService.currentPage}&page_size=${documentListViewService.pageSize}&ordering=-created&truncate_content=true&include_selection_data=false&tags__id__all=${tags__id__all}`
)
expect(req.request.method).toEqual('GET')
})
@@ -325,26 +363,31 @@ describe('DocumentListViewService', () => {
documentListViewService.currentPage = 2
let req = httpTestingController.expectOne((request) =>
request.urlWithParams.startsWith(
`${environment.apiBaseUrl}documents/?page=2&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=2&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
)
)
expect(req.request.method).toEqual('GET')
req.flush(full_results)
flushSelectionDataRequest(httpTestingController)
documentListViewService.setFilterRules(filterRules, true)
const filteredReqs = httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true&tags__id__all=${tags__id__all}`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false&tags__id__all=${tags__id__all}`
)
expect(filteredReqs).toHaveLength(1)
filteredReqs[0].flush(full_results)
flushSelectionDataRequest(
httpTestingController,
`?tags__id__all=${tags__id__all}`
)
expect(documentListViewService.currentPage).toEqual(1)
})
it('should support quick filter', () => {
documentListViewService.quickFilter(filterRules)
const req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/?page=${documentListViewService.currentPage}&page_size=${documentListViewService.pageSize}&ordering=-created&truncate_content=true&include_selection_data=true&tags__id__all=${tags__id__all}`
`${environment.apiBaseUrl}documents/?page=${documentListViewService.currentPage}&page_size=${documentListViewService.pageSize}&ordering=-created&truncate_content=true&include_selection_data=false&tags__id__all=${tags__id__all}`
)
expect(req.request.method).toEqual('GET')
})
@@ -367,21 +410,21 @@ describe('DocumentListViewService', () => {
convertToParamMap(params)
)
let req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/?page=${page}&page_size=${documentListViewService.pageSize}&ordering=-added&truncate_content=true&include_selection_data=true&tags__id__all=${tags__id__all}`
`${environment.apiBaseUrl}documents/?page=${page}&page_size=${documentListViewService.pageSize}&ordering=-added&truncate_content=true&include_selection_data=false&tags__id__all=${tags__id__all}`
)
expect(req.request.method).toEqual('GET')
// reset the list
documentListViewService.currentPage = 1
req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-added&truncate_content=true&include_selection_data=true&tags__id__all=9`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-added&truncate_content=true&include_selection_data=false&tags__id__all=9`
)
documentListViewService.setFilterRules([])
req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-added&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-added&truncate_content=true&include_selection_data=false`
)
documentListViewService.sortField = 'created'
req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
)
documentListViewService.activateSavedView(null)
})
@@ -389,18 +432,22 @@ describe('DocumentListViewService', () => {
it('should support navigating next / previous', () => {
documentListViewService.setFilterRules([])
let req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
)
expect(documentListViewService.currentPage).toEqual(1)
documentListViewService.pageSize = 3
req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/?page=1&page_size=3&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=1&page_size=3&ordering=-created&truncate_content=true&include_selection_data=false`
)
expect(req.request.method).toEqual('GET')
req.flush({
count: 3,
results: documents.slice(0, 3),
})
// two reload()s ran above (setFilterRules, then pageSize), each firing
// its own concurrent filter_selection_data request with an identical
// (unfiltered) URL; this test doesn't assert on selectionData, so let
// afterEach's drain step clean both up rather than disambiguating here.
expect(documentListViewService.hasNext(documents[0].id)).toBeTruthy()
expect(documentListViewService.hasPrevious(documents[0].id)).toBeFalsy()
documentListViewService.getNext(documents[0].id).subscribe((docId) => {
@@ -447,7 +494,7 @@ describe('DocumentListViewService', () => {
expect(documentListViewService.currentPage).toEqual(1)
documentListViewService.pageSize = 3
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=3&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=1&page_size=3&ordering=-created&truncate_content=true&include_selection_data=false`
)
jest
.spyOn(documentListViewService, 'getLastPage')
@@ -462,7 +509,7 @@ describe('DocumentListViewService', () => {
expect(reloadSpy).toHaveBeenCalled()
expect(documentListViewService.currentPage).toEqual(2)
const reqs = httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=2&page_size=3&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=2&page_size=3&ordering=-created&truncate_content=true&include_selection_data=false`
)
expect(reqs.length).toBeGreaterThan(0)
})
@@ -497,11 +544,11 @@ describe('DocumentListViewService', () => {
.mockReturnValue(documents)
documentListViewService.currentPage = 2
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=2&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=2&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
)
documentListViewService.pageSize = 3
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=2&page_size=3&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=2&page_size=3&ordering=-created&truncate_content=true&include_selection_data=false`
)
const reloadSpy = jest.spyOn(documentListViewService, 'reload')
documentListViewService.getPrevious(1).subscribe({
@@ -511,7 +558,7 @@ describe('DocumentListViewService', () => {
expect(reloadSpy).toHaveBeenCalled()
expect(documentListViewService.currentPage).toEqual(1)
const reqs = httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=3&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=1&page_size=3&ordering=-created&truncate_content=true&include_selection_data=false`
)
expect(reqs.length).toBeGreaterThan(0)
})
@@ -524,10 +571,11 @@ describe('DocumentListViewService', () => {
it('should support select a document', () => {
documentListViewService.reload()
const req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
)
expect(req.request.method).toEqual('GET')
req.flush(full_results)
flushSelectionDataRequest(httpTestingController)
documentListViewService.toggleSelected(documents[0])
expect(documentListViewService.isSelected(documents[0])).toBeTruthy()
documentListViewService.toggleSelected(documents[0])
@@ -537,10 +585,11 @@ describe('DocumentListViewService', () => {
it('should support select all', () => {
documentListViewService.reload()
const reloadReq = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
)
expect(reloadReq.request.method).toEqual('GET')
reloadReq.flush(full_results)
flushSelectionDataRequest(httpTestingController)
documentListViewService.selectAll()
expect(documentListViewService.allSelected).toBeTruthy()
@@ -553,13 +602,14 @@ describe('DocumentListViewService', () => {
it('should support select page', () => {
documentListViewService.pageSize = 3
const req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/?page=1&page_size=3&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=1&page_size=3&ordering=-created&truncate_content=true&include_selection_data=false`
)
expect(req.request.method).toEqual('GET')
req.flush({
count: 3,
results: documents.slice(0, 3),
})
flushSelectionDataRequest(httpTestingController)
documentListViewService.selectPage()
expect(documentListViewService.selected.size).toEqual(3)
expect(documentListViewService.isSelected(documents[5])).toBeFalsy()
@@ -568,10 +618,11 @@ describe('DocumentListViewService', () => {
it('should support select range', () => {
documentListViewService.reload()
const req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
)
expect(req.request.method).toEqual('GET')
req.flush(full_results)
flushSelectionDataRequest(httpTestingController)
documentListViewService.toggleSelected(documents[0])
expect(documentListViewService.isSelected(documents[0])).toBeTruthy()
documentListViewService.selectRangeTo(documents[2])
@@ -583,9 +634,10 @@ describe('DocumentListViewService', () => {
it('should clear all-selected mode when toggling a single document', () => {
documentListViewService.reload()
const req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
)
req.flush(full_results)
flushSelectionDataRequest(httpTestingController)
documentListViewService.selectAll()
expect(documentListViewService.allSelected).toBeTruthy()
@@ -599,9 +651,10 @@ describe('DocumentListViewService', () => {
it('should clear all-selected mode when selecting a range', () => {
documentListViewService.reload()
const req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
)
req.flush(full_results)
flushSelectionDataRequest(httpTestingController)
documentListViewService.selectAll()
documentListViewService.toggleSelected(documents[1])
@@ -619,22 +672,24 @@ describe('DocumentListViewService', () => {
it('should support selection range reduction', () => {
documentListViewService.reload()
let req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
)
expect(req.request.method).toEqual('GET')
req.flush(full_results)
flushSelectionDataRequest(httpTestingController)
documentListViewService.selectAll()
expect(documentListViewService.selected.size).toEqual(6)
documentListViewService.setFilterRules(filterRules)
req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true&tags__id__all=9`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false&tags__id__all=9`
)
req.flush({
count: 3,
results: documents.slice(0, 3),
})
flushSelectionDataRequest(httpTestingController, '?tags__id__all=9')
expect(documentListViewService.allSelected).toBeTruthy()
expect(documentListViewService.selected.size).toEqual(3)
})
@@ -643,7 +698,7 @@ describe('DocumentListViewService', () => {
const cancelSpy = jest.spyOn(documentListViewService, 'cancelPending')
documentListViewService.reload()
httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true&tags__id__all=9`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
)
expect(cancelSpy).toHaveBeenCalled()
})
@@ -662,7 +717,7 @@ describe('DocumentListViewService', () => {
documentListViewService.setFilterRules([])
expect(documentListViewService.sortField).toEqual('created')
httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
)
})
@@ -689,11 +744,11 @@ describe('DocumentListViewService', () => {
expect(localStorageSpy).toHaveBeenCalled()
// reload triggered
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
)
documentListViewService.displayFields = null
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
)
expect(documentListViewService.displayFields).toEqual(
DEFAULT_DISPLAY_FIELDS.filter((f) => f.id !== DisplayField.ADDED).map(
@@ -738,7 +793,7 @@ describe('DocumentListViewService', () => {
it('should generate quick filter URL preserving default state', () => {
documentListViewService.reload()
httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
)
const urlTree = documentListViewService.getQuickFilterUrl(filterRules)
expect(urlTree).toBeDefined()
@@ -314,12 +314,39 @@ export class DocumentListViewService {
}
}
private loadFilterSelectionData(filterRules: FilterRule[]) {
this.documentService
.getFilterSelectionData(filterRules)
.pipe(takeUntil(this.unsubscribeNotifier))
.subscribe({
next: (selectionData) => {
this.selectionData = selectionData
this.markChanged()
},
error: () => {
this.selectionData = null
this.markChanged()
},
})
}
reload(onFinish?, updateQueryParams: boolean = true) {
this.cancelPending()
this.isReloading = true
this.error = null
this.markChanged()
let activeListViewState = this.activeListViewState
// Full-text search results are already narrowed by the search backend, so
// computing selection data inline there is cheap. A plain (unfiltered or
// ORM-filtered) browse can span the entire document set, so its selection
// data is fetched separately -- concurrently with the list itself, rather
// than blocking or waiting on it.
const isFullTextSearch = isFullTextFilterRule(
activeListViewState.filterRules
)
if (!isFullTextSearch) {
this.loadFilterSelectionData(activeListViewState.filterRules)
}
this.documentService
.listFiltered(
activeListViewState.currentPage,
@@ -327,17 +354,22 @@ export class DocumentListViewService {
activeListViewState.sortField,
activeListViewState.sortReverse,
activeListViewState.filterRules,
{ truncate_content: true, include_selection_data: true }
{
truncate_content: true,
include_selection_data: isFullTextSearch,
}
)
.pipe(takeUntil(this.unsubscribeNotifier))
.subscribe({
next: (result) => {
const resultWithSelectionData = result as DocumentResults
this.initialized = true
this.isReloading = false
activeListViewState.collectionSize = result.count
activeListViewState.documents = result.results
this.selectionData = resultWithSelectionData.selection_data ?? null
if (isFullTextSearch) {
this.selectionData =
(result as DocumentResults).selection_data ?? null
}
this.syncSelectedToCurrentPage()
this.markChanged()
@@ -376,6 +408,9 @@ export class DocumentListViewService {
// e.g. field was deleted
this.sortField = 'created'
} else {
// cancel the concurrently-fired selection-data request too, so it
// can't resolve afterward and clobber this reset with stale data
this.cancelPending()
this.selectionData = null
let errorMessage
if (
@@ -41,6 +41,24 @@ export abstract class AbstractPaperlessService<T extends ObjectWithId> {
}
}
/**
* Merges a plain params object into an HttpParams instance, skipping
* null/undefined values so they aren't serialized as literal "null" /
* "undefined" query string entries.
*/
protected withParams(
params,
base: HttpParams = new HttpParams()
): HttpParams {
let httpParams = base
for (let key in params) {
if (params[key] != null) {
httpParams = httpParams.set(key, params[key])
}
}
return httpParams
}
list(
page?: number,
pageSize?: number,
@@ -60,11 +78,7 @@ export abstract class AbstractPaperlessService<T extends ObjectWithId> {
if (ordering) {
httpParams = httpParams.set('ordering', ordering)
}
for (let extraParamKey in extraParams) {
if (extraParams[extraParamKey] != null) {
httpParams = httpParams.set(extraParamKey, extraParams[extraParamKey])
}
}
httpParams = this.withParams(extraParams, httpParams)
return this.http
.get<Results<T>>(this.getResourceUrl(), {
params: httpParams,
@@ -113,11 +127,7 @@ export abstract class AbstractPaperlessService<T extends ObjectWithId> {
httpParams = httpParams.set('id__in', ids.join(','))
httpParams = httpParams.set('ordering', '-id')
httpParams = httpParams.set('page_size', 1000)
for (let extraParamKey in extraParams) {
if (extraParams[extraParamKey] != null) {
httpParams = httpParams.set(extraParamKey, extraParams[extraParamKey])
}
}
httpParams = this.withParams(extraParams, httpParams)
return this.http
.get<Results<T>>(this.getResourceUrl(), {
params: httpParams,
@@ -398,6 +398,13 @@ export class DocumentService extends AbstractPaperlessService<Document> {
)
}
getFilterSelectionData(filterRules: FilterRule[]): Observable<SelectionData> {
return this.http.get<SelectionData>(
this.getResourceUrl(null, 'filter_selection_data'),
{ params: this.withParams(queryParamsFromFilterRules(filterRules)) }
)
}
getSuggestions(id: number): Observable<DocumentSuggestions> {
return this.http.get<DocumentSuggestions>(
this.getResourceUrl(id, 'suggestions')
+3 -3
View File
@@ -1,9 +1,9 @@
from datetime import UTC
from datetime import datetime
from typing import Any
from django.conf import settings
from django.core.cache import cache
from rest_framework.request import Request
from documents.caching import CACHE_5_MINUTES
from documents.caching import CACHE_50_MINUTES
@@ -117,7 +117,7 @@ def preview_last_modified(request, pk: int) -> datetime | None:
return doc.modified
def thumbnail_etag(request: Any, pk: int) -> str | None:
def thumbnail_etag(request: Request, pk: int) -> str | None:
"""
Thumbnails are version-dependent, so use the effective document checksum as
the ETag to invalidate cache when the latest version changes.
@@ -128,7 +128,7 @@ def thumbnail_etag(request: Any, pk: int) -> str | None:
return doc.checksum
def thumbnail_last_modified(request: Any, pk: int) -> datetime | None:
def thumbnail_last_modified(request: Request, pk: int) -> datetime | None:
"""
Returns the filesystem last modified either from cache or from filesystem.
Cache should be (slightly?) faster than filesystem
+5
View File
@@ -621,6 +621,11 @@ class ConsumerPlugin(
else:
original_document.save()
# Adding a version changes the effective document, so update root modified
Document.objects.filter(pk=root_doc.pk).update(
modified=timezone.now(),
)
# Create a log entry for the version addition, if enabled
if settings.AUDIT_LOG_ENABLED:
from auditlog.models import ( # type: ignore[import-untyped]
@@ -0,0 +1,73 @@
# Generated by Django 5.2.16 on 2026-07-19 20:33
import django.core.validators
from django.conf import settings
from django.db import migrations
from django.db import models
class Migration(migrations.Migration):
dependencies = [
("documents", "0021_widen_workflow_integer_fields"),
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
]
operations = [
migrations.AlterField(
model_name="document",
name="checksum",
field=models.CharField(
db_index=True,
editable=False,
help_text="The checksum of the original document.",
max_length=64,
verbose_name="checksum",
),
),
migrations.AlterField(
model_name="document",
name="page_count",
field=models.PositiveIntegerField(
db_index=True,
help_text="The number of pages of the document.",
null=True,
validators=[django.core.validators.MinValueValidator(1)],
verbose_name="page count",
),
),
migrations.AddIndex(
model_name="customfieldinstance",
index=models.Index(
fields=["field", "value_date"],
name="documents_c_field_i_30b51d_idx",
),
),
migrations.AddIndex(
model_name="customfieldinstance",
index=models.Index(
fields=["field", "value_int"],
name="documents_c_field_i_d25ab0_idx",
),
),
migrations.AddIndex(
model_name="customfieldinstance",
index=models.Index(
fields=["field", "value_float"],
name="documents_c_field_i_c35174_idx",
),
),
migrations.AddIndex(
model_name="customfieldinstance",
index=models.Index(
fields=["field", "value_monetary_amount"],
name="documents_c_field_i_53ce4e_idx",
),
),
migrations.AddIndex(
model_name="document",
index=models.Index(
fields=["owner", "created"],
name="documents_d_owner_i_ec6ab3_idx",
),
),
]
+11 -1
View File
@@ -217,6 +217,7 @@ class Document(SoftDeleteModel, ModelWithOwner): # type: ignore[django-manager-
_("checksum"),
max_length=64,
editable=False,
db_index=True,
help_text=_("The checksum of the original document."),
)
@@ -234,7 +235,7 @@ class Document(SoftDeleteModel, ModelWithOwner): # type: ignore[django-manager-
blank=False,
null=True,
unique=False,
db_index=False,
db_index=True,
validators=[MinValueValidator(1)],
help_text=_(
"The number of pages of the document.",
@@ -338,6 +339,9 @@ class Document(SoftDeleteModel, ModelWithOwner): # type: ignore[django-manager-
ordering = ("-created",)
verbose_name = _("document")
verbose_name_plural = _("documents")
indexes = [
models.Index(fields=["owner", "created"]),
]
constraints = [
models.UniqueConstraint(
fields=["root_document", "version_index"],
@@ -1190,6 +1194,12 @@ class CustomFieldInstance(SoftDeleteModel):
ordering = ("created",)
verbose_name = _("custom field instance")
verbose_name_plural = _("custom field instances")
indexes = [
models.Index(fields=["field", "value_date"]),
models.Index(fields=["field", "value_int"]),
models.Index(fields=["field", "value_float"]),
models.Index(fields=["field", "value_monetary_amount"]),
]
constraints = [
models.UniqueConstraint(
fields=["document", "field"],
@@ -1,5 +1,6 @@
from __future__ import annotations
import datetime
from typing import TYPE_CHECKING
from unittest import TestCase
from unittest import mock
@@ -10,6 +11,7 @@ from django.contrib.auth.models import User
from django.contrib.contenttypes.models import ContentType
from django.core.exceptions import FieldError
from django.core.files.uploadedfile import SimpleUploadedFile
from django.utils import timezone
from rest_framework import status
from rest_framework.test import APITestCase
@@ -137,6 +139,8 @@ class TestDocumentVersioningApi(DirectoriesMixin, APITestCase):
root_document=root,
content="v2-content",
)
original_modified = timezone.now() - datetime.timedelta(days=1)
Document.objects.filter(pk=root.pk).update(modified=original_modified)
with mock.patch("documents.search.get_backend"):
resp = self.client.delete(f"/api/documents/{root.id}/versions/{v2.id}/")
@@ -146,6 +150,7 @@ class TestDocumentVersioningApi(DirectoriesMixin, APITestCase):
self.assertEqual(resp.data["current_version_id"], v1.id)
root.refresh_from_db()
self.assertEqual(root.content, "root-content")
self.assertGreater(root.modified, original_modified)
with mock.patch("documents.search.get_backend"):
resp = self.client.delete(f"/api/documents/{root.id}/versions/{v1.id}/")
@@ -326,6 +331,8 @@ class TestDocumentVersioningApi(DirectoriesMixin, APITestCase):
root_document=root,
version_label="old",
)
original_modified = timezone.now() - datetime.timedelta(days=1)
Document.objects.filter(pk=root.pk).update(modified=original_modified)
resp = self.client.patch(
f"/api/documents/{root.id}/versions/{version.id}/",
@@ -339,6 +346,8 @@ class TestDocumentVersioningApi(DirectoriesMixin, APITestCase):
self.assertEqual(resp.data["version_label"], "Label 1")
self.assertEqual(resp.data["id"], version.id)
self.assertFalse(resp.data["is_root"])
root.refresh_from_db()
self.assertGreater(root.modified, original_modified)
def test_update_version_label_clears_on_blank(self) -> None:
root = Document.objects.create(
+18 -9
View File
@@ -1241,7 +1241,7 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
],
)
def test_list_with_include_selection_data(self) -> None:
def test_selection_data_endpoint(self) -> None:
correspondent = Correspondent.objects.create(name="c1")
doc_type = DocumentType.objects.create(name="dt1")
storage_path = StoragePath.objects.create(name="sp1")
@@ -1259,30 +1259,28 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
non_matching_doc.tags.add(Tag.objects.create(name="other"))
response = self.client.get(
f"/api/documents/?tags__id__in={tag.id}&include_selection_data=true",
f"/api/documents/filter_selection_data/?tags__id__in={tag.id}",
)
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertIn("selection_data", response.data)
self.assertNotIn("results", response.data)
selected_correspondent = next(
item
for item in response.data["selection_data"]["selected_correspondents"]
for item in response.data["selected_correspondents"]
if item["id"] == correspondent.id
)
selected_tag = next(
item
for item in response.data["selection_data"]["selected_tags"]
if item["id"] == tag.id
item for item in response.data["selected_tags"] if item["id"] == tag.id
)
selected_type = next(
item
for item in response.data["selection_data"]["selected_document_types"]
for item in response.data["selected_document_types"]
if item["id"] == doc_type.id
)
selected_storage_path = next(
item
for item in response.data["selection_data"]["selected_storage_paths"]
for item in response.data["selected_storage_paths"]
if item["id"] == storage_path.id
)
@@ -1291,6 +1289,17 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
self.assertEqual(selected_type["document_count"], 1)
self.assertEqual(selected_storage_path["document_count"], 1)
def test_list_no_longer_supports_include_selection_data(self) -> None:
"""
include_selection_data was never part of a stable release (beta-only,
introduced and removed within the 3.0.0-beta cycle) -- the plain list
endpoint should just ignore the param now rather than compute it inline.
"""
response = self.client.get("/api/documents/?include_selection_data=true")
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertNotIn("selection_data", response.data)
def test_statistics(self) -> None:
doc1 = Document.objects.create(
title="none1",
+4
View File
@@ -767,6 +767,8 @@ class TestConsumer(
root_doc.archive_serial_number = 42
root_doc.save()
original_modified = timezone.now() - datetime.timedelta(days=1)
Document.objects.filter(pk=root_doc.pk).update(modified=original_modified)
actor = User.objects.create_user(
username="actor",
email="actor@example.com",
@@ -818,6 +820,8 @@ class TestConsumer(
self.assertIsNone(version.archive_serial_number)
self.assertEqual(version.original_filename, version_file.name)
self.assertTrue(bool(version.content))
root_doc.refresh_from_db()
self.assertGreater(root_doc.modified, original_modified)
@override_settings(AUDIT_LOG_ENABLED=True)
@mock.patch("documents.consumer.load_classifier")
@@ -1,8 +1,12 @@
from datetime import timedelta
from types import SimpleNamespace
from typing import TYPE_CHECKING
from typing import cast
from unittest import mock
from django.db import connection
from django.test import TestCase
from django.test.utils import CaptureQueriesContext
from django.utils import timezone
from documents.conditionals import metadata_etag
@@ -13,6 +17,9 @@ from documents.models import Document
from documents.tests.utils import DirectoriesMixin
from documents.versioning import resolve_effective_document_by_pk
if TYPE_CHECKING:
from rest_framework.request import Request
class TestConditionals(DirectoriesMixin, TestCase):
def test_metadata_etag_uses_latest_version_for_root_request(self) -> None:
@@ -29,14 +36,18 @@ class TestConditionals(DirectoriesMixin, TestCase):
mime_type="application/pdf",
root_document=root,
)
request = SimpleNamespace(query_params={})
request = cast("Request", SimpleNamespace(query_params={}))
self.assertEqual(
metadata_etag(request, root.id),
f"{latest.checksum}:{latest.modified.isoformat()}",
)
self.assertEqual(preview_etag(request, root.id), latest.archive_checksum)
self.assertEqual(thumbnail_etag(request, root.id), latest.checksum)
# preview_etag/thumbnail_etag resolve the same (pk, request) and should
# reuse metadata_etag's cached resolution instead of re-querying.
with CaptureQueriesContext(connection) as ctx:
self.assertEqual(preview_etag(request, root.id), latest.archive_checksum)
self.assertEqual(thumbnail_etag(request, root.id), latest.checksum)
self.assertEqual(len(ctx.captured_queries), 0)
def test_metadata_etag_changes_when_document_modified_changes(self) -> None:
doc = Document.objects.create(
@@ -44,15 +55,20 @@ class TestConditionals(DirectoriesMixin, TestCase):
checksum="same-checksum",
mime_type="application/pdf",
)
request = SimpleNamespace(query_params={})
original_etag = metadata_etag(request, doc.id)
# Each call simulates a separate incoming HTTP request, so it gets its
# own request object -- the per-request resolution cache must not leak
# a stale result across genuinely different requests.
original_etag = metadata_etag(
cast("Request", SimpleNamespace(query_params={})),
doc.id,
)
new_modified = timezone.now() + timedelta(seconds=5)
Document.objects.filter(id=doc.id).update(modified=new_modified)
self.assertNotEqual(metadata_etag(request, doc.id), original_etag)
second_request = cast("Request", SimpleNamespace(query_params={}))
self.assertNotEqual(metadata_etag(second_request, doc.id), original_etag)
self.assertEqual(
metadata_etag(request, doc.id),
metadata_etag(second_request, doc.id),
f"{doc.checksum}:{new_modified.isoformat()}",
)
@@ -76,9 +92,13 @@ class TestConditionals(DirectoriesMixin, TestCase):
root_document=other_root,
)
invalid_request = SimpleNamespace(query_params={"version": "not-a-number"})
unrelated_request = SimpleNamespace(
query_params={"version": str(other_version.id)},
invalid_request = cast(
"Request",
SimpleNamespace(query_params={"version": "not-a-number"}),
)
unrelated_request = cast(
"Request",
SimpleNamespace(query_params={"version": str(other_version.id)}),
)
self.assertIsNone(
@@ -105,7 +125,7 @@ class TestConditionals(DirectoriesMixin, TestCase):
latest.thumbnail_path.parent.mkdir(parents=True, exist_ok=True)
latest.thumbnail_path.write_bytes(b"thumb")
request = SimpleNamespace(query_params={})
request = cast("Request", SimpleNamespace(query_params={}))
with mock.patch(
"documents.conditionals.get_thumbnail_modified_key",
return_value="thumb-modified-key",
+63
View File
@@ -2938,6 +2938,69 @@ class TestWorkflows(
self.assertFalse(doc.tags.filter(pk=self.t1.pk).exists())
self.assertTrue(doc.tags.filter(pk=self.t2.pk).exists())
def test_document_updated_workflow_assignment_storage_path_persists_with_tag_assignment(
self,
) -> None:
"""
GIVEN:
- A document updated workflow filtered on a tag
- One assignment action assigns a storage path, a second (later-ordered)
assignment action adds a tag
WHEN:
- The document is updated and the workflow is triggered
THEN:
- Both the tag and the storage path are persisted
"""
trigger = WorkflowTrigger.objects.create(
type=WorkflowTrigger.WorkflowTriggerType.DOCUMENT_UPDATED,
)
trigger.filter_has_tags.add(self.t1)
assign_storage_path = WorkflowAction.objects.create(
type=WorkflowAction.WorkflowActionType.ASSIGNMENT,
assign_storage_path=self.sp,
order=0,
)
assign_tag = WorkflowAction.objects.create(
type=WorkflowAction.WorkflowActionType.ASSIGNMENT,
order=1,
)
assign_tag.assign_tags.add(self.t2)
assign_tag.save()
workflow = Workflow.objects.create(
name="Workflow assign storage path then tag",
order=0,
)
workflow.triggers.add(trigger)
workflow.actions.add(assign_storage_path, assign_tag)
workflow.save()
doc = Document.objects.create(
title="sample test",
mime_type="application/pdf",
checksum="assign-tag-and-storage-path",
original_filename="sample.pdf",
)
generated = generate_unique_filename(doc)
destination = (settings.ORIGINALS_DIR / generated).resolve()
create_source_path_directory(destination)
shutil.copy(self.SAMPLE_DIR / "simple.pdf", destination)
Document.objects.filter(pk=doc.pk).update(filename=generated.as_posix())
doc.refresh_from_db()
doc.tags.set([self.t1])
superuser = User.objects.create_superuser("superuser")
self.client.force_authenticate(user=superuser)
self.client.patch(
f"/api/documents/{doc.id}/",
{"title": "user update to trigger workflow"},
format="json",
)
doc.refresh_from_db()
self.assertEqual(doc.storage_path, self.sp)
self.assertTrue(doc.tags.filter(pk=self.t2.pk).exists())
def test_removal_action_document_updated_removeall(self) -> None:
"""
GIVEN:
+34 -11
View File
@@ -8,7 +8,7 @@ from typing import Any
from documents.models import Document
if TYPE_CHECKING:
from django.http import HttpRequest
from rest_framework.request import Request
class VersionResolutionError(StrEnum):
@@ -26,7 +26,7 @@ def _document_manager(*, include_deleted: bool) -> Any:
return Document.global_objects if include_deleted else Document.objects
def get_request_version_param(request: HttpRequest) -> str | None:
def get_request_version_param(request: Request) -> str | None:
if hasattr(request, "query_params"):
return request.query_params.get("version")
return None
@@ -57,7 +57,7 @@ def get_latest_version_for_root(
def resolve_requested_version_for_root(
root_doc: Document,
request: Any,
request: Request,
*,
include_deleted: bool = False,
) -> VersionResolution:
@@ -86,7 +86,7 @@ def resolve_requested_version_for_root(
def resolve_effective_document(
request_doc: Document,
request: Any,
request: Request,
*,
include_deleted: bool = False,
) -> VersionResolution:
@@ -107,18 +107,41 @@ def resolve_effective_document(
return VersionResolution(document=request_doc)
_EFFECTIVE_DOCUMENT_CACHE_ATTR = "_effective_document_resolution_cache"
def resolve_effective_document_by_pk(
pk: int,
request: Any,
request: Request,
*,
include_deleted: bool = False,
) -> VersionResolution:
# Django's `condition()` decorator (used for ETag/Last-Modified) invokes the
# etag_func and last_modified_func separately, and the view itself may resolve
# again -- all against the same request. Cache per-request so a single thumb/
# metadata/preview request doesn't redo this resolution multiple times.
cache = getattr(request, _EFFECTIVE_DOCUMENT_CACHE_ATTR, None)
if cache is None:
cache = {}
setattr(request, _EFFECTIVE_DOCUMENT_CACHE_ATTR, cache)
key = (pk, include_deleted)
if key in cache:
return cache[key]
manager = _document_manager(include_deleted=include_deleted)
request_doc = manager.only("id", "root_document_id").filter(pk=pk).first()
if request_doc is None:
return VersionResolution(document=None, error=VersionResolutionError.NOT_FOUND)
return resolve_effective_document(
request_doc,
request,
include_deleted=include_deleted,
)
resolution = VersionResolution(
document=None,
error=VersionResolutionError.NOT_FOUND,
)
else:
resolution = resolve_effective_document(
request_doc,
request,
include_deleted=include_deleted,
)
cache[key] = resolution
return resolution
+29 -19
View File
@@ -1034,7 +1034,11 @@ class DocumentViewSet(
],
}
def get_queryset(self):
def _base_document_queryset(self):
# Root documents only, with the annotations that filter_backends rely
# on (effective_content for SearchFilter, num_notes for ordering) --
# but no select_related/prefetch_related, since those only matter for
# serializing documents, not for filtering, ordering, or aggregating.
latest_version_content = Subquery(
Document.objects.filter(root_document=OuterRef("pk"))
.order_by("-id")
@@ -1046,6 +1050,11 @@ class DocumentViewSet(
.order_by("-created", "-id")
.annotate(effective_content=Coalesce(latest_version_content, F("content")))
.annotate(num_notes=Count("notes"))
)
def get_queryset(self):
return (
self._base_document_queryset()
.select_related("correspondent", "storage_path", "document_type", "owner")
.prefetch_related(
Prefetch(
@@ -1184,24 +1193,21 @@ class DocumentViewSet(
return response
def list(self, request, *args, **kwargs):
if not get_boolean(
str(request.query_params.get("include_selection_data", "false")),
):
return super().list(request, *args, **kwargs)
queryset = self.filter_queryset(self.get_queryset())
selection_data = self._get_selection_data_for_queryset(queryset)
page = self.paginate_queryset(queryset)
if page is not None:
serializer = self.get_serializer(page, many=True)
response = self.get_paginated_response(serializer.data)
response.data["selection_data"] = selection_data
return response
serializer = self.get_serializer(queryset, many=True)
return Response({"results": serializer.data, "selection_data": selection_data})
@extend_schema(
operation_id="documents_filter_selection_data",
description=(
"Returns per-tag/correspondent/document-type/storage-path/custom-field "
"document counts for the current filter, without paginating or "
"serializing the matching documents themselves. Split out from the "
"plain document list so that browsing the (potentially huge) unfiltered "
"document list doesn't pay for this aggregation on every request."
),
responses={200: inline_serializer(name="SelectionData", fields={})},
)
@action(detail=False, methods=["get"], url_path="filter_selection_data")
def filter_selection_data(self, request, *args, **kwargs):
queryset = self.filter_queryset(self._base_document_queryset())
return Response(self._get_selection_data_for_queryset(queryset))
def destroy(self, request, *args, **kwargs):
from documents.search import get_backend
@@ -2057,6 +2063,8 @@ class DocumentViewSet(
_backend.remove(version_doc.pk)
version_doc_id = version_doc.id
version_doc.delete()
root_doc.modified = timezone.now()
Document.objects.filter(pk=root_doc.pk).update(modified=root_doc.modified)
_backend.add_or_update(root_doc)
if settings.AUDIT_LOG_ENABLED:
actor = (
@@ -2136,6 +2144,8 @@ class DocumentViewSet(
old_label = version_doc.version_label
version_doc.version_label = serializer.validated_data["version_label"]
version_doc.save(update_fields=["version_label"])
root_doc.modified = timezone.now()
Document.objects.filter(pk=root_doc.pk).update(modified=root_doc.modified)
if settings.AUDIT_LOG_ENABLED and old_label != version_doc.version_label:
actor = (
+5 -1
View File
@@ -24,7 +24,11 @@ def apply_assignment_to_document(
action: WorkflowAction, annotated with 'has_assign_*' boolean fields
"""
if action.has_assign_tags:
document.add_nested_tags(action.assign_tags.all())
# Apply to a freshly-fetched instance rather than the shared `document`.
# Document.tags.add() fires an m2m_changed signal that ultimately calls
# instance.refresh_from_db(), which would discard any other unsaved
# assignment fields (e.g. storage_path) already staged on `document`.
Document.objects.get(pk=document.pk).add_nested_tags(action.assign_tags.all())
if action.assign_correspondent:
document.correspondent = action.assign_correspondent