Fix: register the filter_selection_data e2e stub after routeFromHAR, not before

Per Playwright's own docs, routeFromHAR's notFound: 'fallback' sends unmatched requests straight to the network -- it does not chain to other, earlier-registered page.route() handlers via route.fallback() the way I'd assumed. Since Playwright checks routes in reverse-registration order, a handler registered in beforeEach (i.e. before the test body's routeFromHAR call) is checked AFTER routeFromHAR, whose catch-all pattern intercepts everything first and sends any miss straight to the (nonexistent, in e2e) network -- my stub never got a chance to run. Confirmed by a second CI failure with the same "Failed to fetch" symptom even after the CORS-header fix. Moved the mockFilterSelectionData(page) call to immediately after each test's own routeFromHAR(...) call instead, so it's registered later and checked first for that specific URL, with routeFromHAR's broader pattern still handling everything else as before. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Fix: add CORS header to the e2e filter_selection_data stub response
2026-07-29 15:24:54 +00:00 · 2026-07-20 14:54:58 -07:00 · 2026-07-20 14:32:59 -07:00 · 2026-07-20 14:21:51 -07:00 · 2026-07-20 10:00:08 -07:00 · 2026-07-20 10:00:08 -07:00
19 changed files with 860 additions and 1756 deletions
@@ -1,662 +0,0 @@
 # Mail Fetch Re-download Fix Implementation Plan
 > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
 **Goal:** Stop `paperless_mail` from re-downloading full IMAP message bodies for mail it has already handled, and make sure attachment-less mail under attachments-only rules is recorded so it stops matching the search forever.
 **Architecture:** In `_handle_mail_rule`, replace the single "fetch every matching message's full body" call with a cheap `UID SEARCH` first, subtract UIDs already in `ProcessedMail`, and fetch bodies only for the remainder (batched). Separately, extract the existing "record `PROCESSED_WO_CONSUMPTION`" block into a shared helper and call it from the no-attachments-under-attachments-only-scope early return in `_handle_message`, which currently skips it entirely.
 **Tech Stack:** Django (pytest-django), `imap_tools` (`MailBox.uids()`, `AND` query builder), existing `ProcessedMail` model — no new dependencies, no migrations.
 ## Global Constraints
 - No new required settings and no DB migrations for existing installs (spec: "No new required settings/migrations for existing installs").
 - Batch size for body fetches is a hardcoded module constant (`MAIL_FETCH_BATCH_SIZE = 500`), not user-configurable.
 - No mailbox action (mark-read/flag/tag/move/delete) is applied for the no-attachment/no-consumption case — only a `ProcessedMail` row is written, matching the existing precedent for other no-consumption outcomes.
 - Every path that currently produces a document or applies a mail action must behave exactly as before (no change to `_process_attachments`'/`_process_eml`'s consuming behavior, action application, or the existing per-message `already_processed` dedup check, which stays in place as a safety net).
 - Follow the existing per-message `uid_validity` matching semantics exactly: when `self._current_uid_validity is not None`, only match `ProcessedMail` rows with the same `uid_validity` or `uid_validity IS NULL`; when it is `None` (server didn't report UIDVALIDITY), fall back to matching on `(rule, uid, folder)` alone, ignoring `uid_validity`. (This exact conditional already exists at `mail.py:722-726` for the per-message check — the new UID-diff query must replicate it or several existing uidvalidity tests will break.)
 ---
 ### Task 1: Add UID search support to the `BogusMailBox` test double
 **Files:**
 - Modify: `src/paperless_mail/tests/test_mail.py:137-171` (`BogusMailBox.fetch`)
 - Test: `src/paperless_mail/tests/test_mail.py` (new test in `class TestMail`)
 **Interfaces:**
 - Produces: `BogusMailBox.uids(self, criteria, charset="") -> list[str]`, filtering with the same rules as `fetch()`, returning only UIDs (no message bodies). `BogusMailBox.fetch()` gains support for a `UID <comma-list>` criteria token (used by real code's `AND(uid=[...])` queries).
 - Consumes: nothing new — this is test infrastructure only, no production code changes in this task.
 - [ ] **Step 1: Write the failing test**
 Add to `class TestMail` in `src/paperless_mail/tests/test_mail.py` (near the other `BogusMailBox`-adjacent tests, e.g. after `test_handle_message`):
 ```python
    def test_bogus_mailbox_uids_and_uid_criteria(self) -> None:
        mailbox = self.mailMocker.bogus_mailbox
        all_messages = list(mailbox.messages)
        # uids() returns the UIDs of unseen messages, no bodies needed to call it
        unseen_uids = mailbox.uids("(UNSEEN)")
        self.assertEqual(
            set(unseen_uids),
            {m.uid for m in all_messages if not m.seen},
        )
        # fetch() with an explicit UID criteria returns only the matching messages
        target_uid = all_messages[0].uid
        from imap_tools import AND
        fetched = mailbox.fetch(AND(uid=[target_uid]), mark_seen=False)
        self.assertEqual([m.uid for m in fetched], [target_uid])
 ```
 - [ ] **Step 2: Run test to verify it fails**
 Run: `uv run pytest --override-ini="addopts=" src/paperless_mail/tests/test_mail.py::TestMail::test_bogus_mailbox_uids_and_uid_criteria -v`
 Expected: FAIL with `AttributeError: 'BogusMailBox' object has no attribute 'uids'` (and/or the UID-criteria fetch returning all 3 default messages instead of 1, since `fetch()` doesn't understand a `UID` token yet).
 - [ ] **Step 3: Implement `uids()` and `UID` criteria support**
 Replace `src/paperless_mail/tests/test_mail.py:137-171`:
 ```python
    def fetch(self, criteria, mark_seen, charset="", *, bulk=True):
        return self._filter_messages(criteria)
    def uids(self, criteria, charset="") -> list[str]:
        return [m.uid for m in self._filter_messages(criteria)]
    def _filter_messages(self, criteria):
        msg = self.messages
        criteria = str(criteria).strip("()").split(" ")
        if "UNSEEN" in criteria:
            msg = filter(lambda m: not m.seen, msg)
        if "SUBJECT" in criteria:
            subject = criteria[criteria.index("SUBJECT") + 1].strip('"')
            msg = filter(lambda m: subject in m.subject, msg)
        if "BODY" in criteria:
            body = criteria[criteria.index("BODY") + 1].strip('"')
            msg = filter(lambda m: body in m.text, msg)
        if "FROM" in criteria:
            from_ = criteria[criteria.index("FROM") + 1].strip('"')
            msg = filter(lambda m: from_ in m.from_, msg)
        if "TO" in criteria:
            to_ = criteria[criteria.index("TO") + 1].strip('"')
            msg = filter(lambda m: any(to_ in to_addr for to_addr in m.to), msg)
        if "UNFLAGGED" in criteria:
            msg = filter(lambda m: not m.flagged, msg)
        if "UNKEYWORD" in criteria:
            tag = criteria[criteria.index("UNKEYWORD") + 1].strip("'")
            msg = filter(lambda m: tag not in m.flags, msg)
        if "(X-GM-LABELS" in criteria:  # ['NOT', '(X-GM-LABELS', '"processed"']
            msg = filter(lambda m: "processed" not in m.flags, msg)
        if "UID" in criteria:
            uid_list = criteria[criteria.index("UID") + 1].split(",")
            msg = filter(lambda m: m.uid in uid_list, msg)
        return list(msg)
 ```
 This is a pure refactor of the existing filtering logic into `_filter_messages`, reused by both `fetch()` (unchanged behavior) and the new `uids()`, plus one new `if "UID" in criteria` branch.
 - [ ] **Step 4: Run test to verify it passes**
 Run: `uv run pytest --override-ini="addopts=" src/paperless_mail/tests/test_mail.py::TestMail::test_bogus_mailbox_uids_and_uid_criteria -v`
 Expected: PASS
 - [ ] **Step 5: Run the full existing `test_mail.py` suite to confirm the refactor didn't change `fetch()` behavior**
 Run: `uv run pytest --override-ini="addopts=" src/paperless_mail/tests/test_mail.py -v`
 Expected: all tests PASS (same pass count as before this task)
 - [ ] **Step 6: Commit**
 ```bash
 git add src/paperless_mail/tests/test_mail.py
 git commit -m "test: add BogusMailBox.uids() and UID criteria support"
 ```
 ---
 ### Task 2: Record `ProcessedMail` for attachment-less mail under attachments-only rules
 **Files:**
 - Modify: `src/paperless_mail/mail.py:749-793` (`_handle_message`)
 - Modify: `src/paperless_mail/mail.py:952-979` (`_process_attachments`, tail)
 - Test: `src/paperless_mail/tests/test_mail.py:540-548` (`test_handle_empty_message`, rewritten)
 **Interfaces:**
 - Produces: `MailAccountHandler._record_processed_without_consumption(self, message: MailMessage, rule: MailRule) -> None` — idempotently writes a `ProcessedMail(status="PROCESSED_WO_CONSUMPTION")` row for `(rule, message.uid, rule.folder)` if one doesn't already exist for the current `self._current_uid_validity`. No mailbox action is applied.
 - Consumes: `self._current_uid_validity` (already set by `_handle_mail_rule` before `_handle_message` is called; `None` when called directly, e.g. in unit tests).
 - [ ] **Step 1: Write the failing test**
 Replace `src/paperless_mail/tests/test_mail.py:540-548` (`test_handle_empty_message`):
 ```python
    def test_handle_empty_message(self) -> None:
        message = self.mailMocker.messageBuilder.create_message(
            subject="No attachments here",
            attachments=[],
        )
        account = MailAccount.objects.create()
        rule = MailRule.objects.create(
            account=account,
            consumption_scope=MailRule.ConsumptionScope.ATTACHMENTS_ONLY,
        )
        result = self.mail_account_handler._handle_message(message, rule)
        self.mailMocker._queue_consumption_tasks_mock.assert_not_called()
        self.assertEqual(result, 0)
        processed = ProcessedMail.objects.get(
            rule=rule,
            uid=message.uid,
            folder=rule.folder,
        )
        self.assertEqual(processed.status, "PROCESSED_WO_CONSUMPTION")
        # Calling it again must not create a second row
        self.mail_account_handler._handle_message(message, rule)
        self.assertEqual(
            ProcessedMail.objects.filter(
                rule=rule,
                uid=message.uid,
                folder=rule.folder,
            ).count(),
            1,
        )
 ```
 - [ ] **Step 2: Run test to verify it fails**
 Run: `uv run pytest --override-ini="addopts=" src/paperless_mail/tests/test_mail.py::TestMail::test_handle_empty_message -v`
 Expected: FAIL — `ProcessedMail.DoesNotExist` (no row is created by the current early return).
 - [ ] **Step 3: Add the shared helper and call it from both places**
 Insert a new method right after `_handle_message` ends, i.e. after `src/paperless_mail/mail.py:793` (`return processed_elements`), before `def filename_inclusion_matches`:
 ```python
    def _record_processed_without_consumption(
        self,
        message: MailMessage,
        rule: MailRule,
    ) -> None:
        if not ProcessedMail.objects.filter(
            rule=rule,
            uid=message.uid,
            folder=rule.folder,
            uid_validity=self._current_uid_validity,
        ).exists():
            ProcessedMail.objects.create(
                rule=rule,
                folder=rule.folder,
                uid=message.uid,
                uid_validity=self._current_uid_validity,
                subject=message.subject,
                received=make_aware(message.date)
                if is_naive(message.date)
                else message.date,
                status="PROCESSED_WO_CONSUMPTION",
            )
 ```
 Then modify `_handle_message`'s early return at `src/paperless_mail/mail.py:756-760`:
 ```python
        # Skip Message handling when only attachments are to be processed but
        # message doesn't have any.
        if (
            not message.attachments
            and rule.consumption_scope == MailRule.ConsumptionScope.ATTACHMENTS_ONLY
        ):
            return processed_elements
 ```
 becomes:
 ```python
        # Skip Message handling when only attachments are to be processed but
        # message doesn't have any.
        if (
            not message.attachments
            and rule.consumption_scope == MailRule.ConsumptionScope.ATTACHMENTS_ONLY
        ):
            self._record_processed_without_consumption(message, rule)
            return processed_elements
 ```
 Then replace the tail of `_process_attachments` at `src/paperless_mail/mail.py:952-979`:
 ```python
        if len(consume_tasks) > 0:
            queue_consumption_tasks(
                consume_tasks=consume_tasks,
                rule=rule,
                message=message,
                uid_validity=self._current_uid_validity,
            )
        else:
            # No files to consume, just mark as processed if it wasn't by .eml processing
            if not ProcessedMail.objects.filter(
                rule=rule,
                uid=message.uid,
                folder=rule.folder,
                uid_validity=self._current_uid_validity,
            ).exists():
                ProcessedMail.objects.create(
                    rule=rule,
                    folder=rule.folder,
                    uid=message.uid,
                    uid_validity=self._current_uid_validity,
                    subject=message.subject,
                    received=make_aware(message.date)
                    if is_naive(message.date)
                    else message.date,
                    status="PROCESSED_WO_CONSUMPTION",
                )
        return processed_attachments
 ```
 with:
 ```python
        if len(consume_tasks) > 0:
            queue_consumption_tasks(
                consume_tasks=consume_tasks,
                rule=rule,
                message=message,
                uid_validity=self._current_uid_validity,
            )
        else:
            # No files to consume, just mark as processed if it wasn't by .eml processing
            self._record_processed_without_consumption(message, rule)
        return processed_attachments
 ```
 - [ ] **Step 4: Run test to verify it passes**
 Run: `uv run pytest --override-ini="addopts=" src/paperless_mail/tests/test_mail.py::TestMail::test_handle_empty_message -v`
 Expected: PASS
 - [ ] **Step 5: Run the full `test_mail.py` suite (regression check for the `_process_attachments` refactor)**
 Run: `uv run pytest --override-ini="addopts=" src/paperless_mail/tests/test_mail.py -v`
 Expected: all PASS
 - [ ] **Step 6: Commit**
 ```bash
 git add src/paperless_mail/mail.py src/paperless_mail/tests/test_mail.py
 git commit -m "fix: record ProcessedMail for attachment-less mail under attachments-only rules"
 ```
 ---
 ### Task 3: Diff UIDs against `ProcessedMail` before fetching message bodies
 **Files:**
 - Modify: `src/paperless_mail/mail.py:641-693` (`_handle_mail_rule`, the fetch section)
 - Test: `src/paperless_mail/tests/test_mail.py` (new test in `class TestMail`)
 **Interfaces:**
 - Consumes: `MailAccountHandler._record_processed_without_consumption` (Task 2), `ProcessedMail` model, `imap_tools.AND`, `M.uids()` (Task 1's `BogusMailBox.uids()` in tests; `imap_tools.MailBox.uids()` in production).
 - Produces: no new public interface — `_handle_mail_rule`'s external behavior (return value, exceptions raised) is unchanged; only its internal fetch strategy changes. `_handle_mail_rule` now returns `0` immediately, without calling `M.fetch()` at all, when every UID matching the search criteria already has a `ProcessedMail` row.
 - [ ] **Step 1: Write the failing test**
 Add to `class TestMail` in `src/paperless_mail/tests/test_mail.py`:
 ```python
    def test_handle_mail_account_skips_body_fetch_for_already_processed_mail(
        self,
    ) -> None:
        """
        GIVEN:
            - An attachment-less mail under an attachments-only mark-read rule,
              already recorded as PROCESSED_WO_CONSUMPTION
        WHEN:
            - The mail account is processed again and the mail still matches the
              search criteria (it was never marked read, since no mail action is
              applied for the no-consumption case)
        THEN:
            - No IMAP body fetch happens for that mail; only the cheap UID search runs.
        """
        account = MailAccount.objects.create(
            name="test",
            imap_server="",
            username="admin",
            password="secret",
        )
        rule = MailRule.objects.create(
            name="testrule",
            account=account,
            action=MailRule.MailAction.MARK_READ,
            consumption_scope=MailRule.ConsumptionScope.ATTACHMENTS_ONLY,
        )
        message = self.mailMocker.messageBuilder.create_message(
            subject="No attachment",
            attachments=[],
        )
        self.mailMocker.bogus_mailbox.messages = [message]
        self.mailMocker.bogus_mailbox.updateClient()
        # First run: records ProcessedMail without consuming anything.
        self.mail_account_handler.handle_mail_account(account)
        self.assertTrue(
            ProcessedMail.objects.filter(
                rule=rule,
                uid=message.uid,
                folder=rule.folder,
            ).exists(),
        )
        self.mailMocker._queue_consumption_tasks_mock.assert_not_called()
        # Second run: message still matches UNSEEN (mark-read action never ran),
        # but its body must not be downloaded again.
        with mock.patch.object(
            self.mailMocker.bogus_mailbox,
            "fetch",
            wraps=self.mailMocker.bogus_mailbox.fetch,
        ) as fetch_spy:
            self.mail_account_handler.handle_mail_account(account)
        fetch_spy.assert_not_called()
 ```
 - [ ] **Step 2: Run test to verify it fails**
 Run: `uv run pytest --override-ini="addopts=" src/paperless_mail/tests/test_mail.py::TestMail::test_handle_mail_account_skips_body_fetch_for_already_processed_mail -v`
 Expected: FAIL — `fetch_spy.assert_not_called()` fails because the current code always calls `M.fetch()` for every message matching the search criteria, regardless of `ProcessedMail`.
 - [ ] **Step 3: Implement the UID diff**
 Replace `src/paperless_mail/mail.py:683-693`:
 ```python
        try:
            messages = M.fetch(
                criteria=criterias,
                mark_seen=False,
                charset=rule.account.character_set,
                bulk=True,
            )
        except Exception as err:
            raise MailError(
                f"Rule {rule}: Error while fetching folder {rule.folder}",
            ) from err
 ```
 with:
 ```python
        try:
            all_uids = set(
                M.uids(criteria=criterias, charset=rule.account.character_set),
            )
        except Exception as err:
            raise MailError(
                f"Rule {rule}: Error while searching folder {rule.folder}",
            ) from err
        processed_uids_qs = ProcessedMail.objects.filter(
            rule=rule,
            folder=rule.folder,
            uid__in=all_uids,
        )
        if self._current_uid_validity is not None:
            processed_uids_qs = processed_uids_qs.filter(
                Q(uid_validity=self._current_uid_validity)
                | Q(uid_validity__isnull=True),
            )
        processed_uids = set(processed_uids_qs.values_list("uid", flat=True))
        new_uids = all_uids - processed_uids
        if not new_uids:
            self.log.debug(
                f"Rule {rule}: No new mail matching criteria {criterias}",
            )
            return 0
        try:
            messages = M.fetch(
                criteria=AND(uid=list(new_uids)),
                mark_seen=False,
                charset=rule.account.character_set,
                bulk=True,
            )
        except Exception as err:
            raise MailError(
                f"Rule {rule}: Error while fetching folder {rule.folder}",
            ) from err
 ```
 Note the `uid_validity` handling here deliberately mirrors the existing per-message check at `mail.py:722-726` exactly (see Global Constraints) — the extra `Q(...)` filter is applied only when `self._current_uid_validity is not None`.
 - [ ] **Step 4: Run test to verify it passes**
 Run: `uv run pytest --override-ini="addopts=" src/paperless_mail/tests/test_mail.py::TestMail::test_handle_mail_account_skips_body_fetch_for_already_processed_mail -v`
 Expected: PASS
 - [ ] **Step 5: Run the full `test_mail.py` suite (regression check, especially the uidvalidity tests)**
 Run: `uv run pytest --override-ini="addopts=" src/paperless_mail/tests/test_mail.py -v`
 Expected: all PASS, including:
 - `test_handle_mail_account_skip_duplicate_uids_from_fetch`
 - `test_handle_mail_account_skips_mail_already_processed_in_same_uidvalidity`
 - `test_handle_mail_account_processes_mail_after_uidvalidity_change`
 - `test_handle_mail_account_skips_mail_processed_before_uidvalidity_tracking`
 - `test_handle_mail_account_processes_mail_when_uidvalidity_unavailable`
 - `test_handle_mail_account_skips_mail_when_uidvalidity_unavailable_but_prior_record_exists`
 - `test_handle_mail_account_overlapping_rules_only_first_consumes`
 If any of these fail, the `uid_validity` branching in Step 3 does not match `mail.py:722-726` closely enough — re-check against the Global Constraints note before changing test expectations.
 - [ ] **Step 6: Commit**
 ```bash
 git add src/paperless_mail/mail.py src/paperless_mail/tests/test_mail.py
 git commit -m "fix: diff UIDs against ProcessedMail before fetching message bodies"
 ```
 ---
 ### Task 4: Batch the body fetch for large backlogs
 **Files:**
 - Modify: `src/paperless_mail/mail.py:74-76` (module-level constant)
 - Modify: `src/paperless_mail/mail.py` (the `M.fetch(...)` block added in Task 3)
 - Test: `src/paperless_mail/tests/test_mail.py` (new test in `class TestMail`)
 **Interfaces:**
 - Produces: module-level constant `paperless_mail.mail.MAIL_FETCH_BATCH_SIZE: int = 500`.
 - Consumes: `itertools` (already imported at `mail.py:2`), `sorted_new_uids` derived from Task 3's `new_uids` set.
 - [ ] **Step 1: Write the failing test**
 Add to `class TestMail` in `src/paperless_mail/tests/test_mail.py`. Add `from paperless_mail.mail import MAIL_FETCH_BATCH_SIZE` to the imports at the top of the file (alongside the other `from paperless_mail.mail import ...` lines at `test_mail.py:35-38`).
 ```python
    @mock.patch("paperless_mail.mail.MAIL_FETCH_BATCH_SIZE", 5)
    def test_handle_mail_account_batches_body_fetch_for_large_backlog(self) -> None:
        """
        GIVEN:
            - More new/unprocessed mail than MAIL_FETCH_BATCH_SIZE
        WHEN:
            - The mail account is processed
        THEN:
            - The body fetch is issued in multiple batches
            - Every message is still processed (none dropped at a batch boundary)
        """
        account = MailAccount.objects.create(
            name="test",
            imap_server="",
            username="admin",
            password="secret",
        )
        rule = MailRule.objects.create(
            name="testrule",
            account=account,
            action=MailRule.MailAction.MARK_READ,
            consumption_scope=MailRule.ConsumptionScope.ATTACHMENTS_ONLY,
        )
        message_count = 12  # more than the patched batch size of 5
        self.mailMocker.bogus_mailbox.messages = [
            self.mailMocker.messageBuilder.create_message(
                subject=f"No attachment {i}",
                attachments=[],
            )
            for i in range(message_count)
        ]
        self.mailMocker.bogus_mailbox.updateClient()
        with mock.patch.object(
            self.mailMocker.bogus_mailbox,
            "fetch",
            wraps=self.mailMocker.bogus_mailbox.fetch,
        ) as fetch_spy:
            self.mail_account_handler.handle_mail_account(account)
        # ceil(12 / 5) == 3 batches
        self.assertEqual(fetch_spy.call_count, 3)
        self.assertEqual(
            ProcessedMail.objects.filter(rule=rule).count(),
            message_count,
        )
 ```
 - [ ] **Step 2: Run test to verify it fails**
 Run: `uv run pytest --override-ini="addopts=" src/paperless_mail/tests/test_mail.py::TestMail::test_handle_mail_account_batches_body_fetch_for_large_backlog -v`
 Expected: FAIL — either an `ImportError` for `MAIL_FETCH_BATCH_SIZE` (doesn't exist yet) or, once that import is stubbed out, `fetch_spy.call_count == 1` instead of `3` (Task 3's implementation fetches all new UIDs in one call).
 - [ ] **Step 3: Add the constant and batch the fetch**
 Insert after `src/paperless_mail/mail.py:74` (right after the `APPLE_MAIL_TAG_COLORS` dict closes, before `class MailError`):
 ```python
 MAIL_FETCH_BATCH_SIZE = 500
 ```
 Replace the `M.fetch(...)` block added in Task 3 (Task 3 Step 3's final `try/except`):
 ```python
        try:
            messages = M.fetch(
                criteria=AND(uid=list(new_uids)),
                mark_seen=False,
                charset=rule.account.character_set,
                bulk=True,
            )
        except Exception as err:
            raise MailError(
                f"Rule {rule}: Error while fetching folder {rule.folder}",
            ) from err
 ```
 with:
 ```python
        sorted_new_uids = sorted(new_uids, key=int)
        message_batches = []
        for batch_start in range(0, len(sorted_new_uids), MAIL_FETCH_BATCH_SIZE):
            batch = sorted_new_uids[batch_start : batch_start + MAIL_FETCH_BATCH_SIZE]
            try:
                message_batches.append(
                    M.fetch(
                        criteria=AND(uid=batch),
                        mark_seen=False,
                        charset=rule.account.character_set,
                        bulk=True,
                    ),
                )
            except Exception as err:
                raise MailError(
                    f"Rule {rule}: Error while fetching folder {rule.folder}",
                ) from err
        messages = itertools.chain(*message_batches)
 ```
 - [ ] **Step 4: Run test to verify it passes**
 Run: `uv run pytest --override-ini="addopts=" src/paperless_mail/tests/test_mail.py::TestMail::test_handle_mail_account_batches_body_fetch_for_large_backlog -v`
 Expected: PASS
 - [ ] **Step 5: Run the full `test_mail.py` suite**
 Run: `uv run pytest --override-ini="addopts=" src/paperless_mail/tests/test_mail.py -v`
 Expected: all PASS
 - [ ] **Step 6: Commit**
 ```bash
 git add src/paperless_mail/mail.py src/paperless_mail/tests/test_mail.py
 git commit -m "perf: batch body fetches when many new UIDs are pending"
 ```
 ---
 ### Task 5: Full regression pass and lint
 **Files:** none (verification only, plus any fixups this step surfaces)
 - [ ] **Step 1: Run the full backend test suite with coverage/parallelism as configured**
 Run: `uv run pytest src/paperless_mail/`
 Expected: all `paperless_mail` tests PASS
 - [ ] **Step 2: Run the full project test suite**
 Run: `uv run pytest`
 Expected: all tests PASS (no regressions outside `paperless_mail`)
 - [ ] **Step 3: Lint and format**
 Run: `uv run ruff check src/paperless_mail/` and `uv run ruff format --check src/paperless_mail/`
 Expected: no errors. If `ruff format --check` reports files needing formatting, run `uv run ruff format src/paperless_mail/` and re-check.
 - [ ] **Step 4: Type check against the frozen baselines**
 Run: `uv run mypy src/paperless_mail/mail.py` (or the project's standard mypy invocation) and confirm no new violations beyond `.mypy-baseline.txt`.
 Expected: no new errors introduced by this change.
 - [ ] **Step 5: Commit any fixups**
 If Steps 1-4 required changes:
 ```bash
 git add -u
 git commit -m "chore: fix lint/type fallout from mail fetch fix"
 ```
 If no changes were needed, skip this step — nothing to commit.
@@ -1,216 +0,0 @@
 # Mail fetch fix: avoid re-downloading already-handled IMAP mail
 ## Problem
 `paperless_mail`'s `_handle_mail_rule` (`src/paperless_mail/mail.py:641`) fetches
 the full RFC822 body of every message an IMAP `SEARCH` matches (via
 `M.fetch(criteria=criterias, ..., bulk=True)`), then does de-duplication in
 Python afterward. Two things compound into a real bug (reported upstream in
 [paperless-ngx#13175](https://github.com/paperless-ngx/paperless-ngx/issues/13175)):
 1. **Full-body fetch happens before dedup.** The de-dup check against
   `ProcessedMail` (`mail.py:717-731`) happens per-message, after the body has
   already been downloaded for every matching message. Steady-state cost is
   _O(everything the search matches)_, not _O(new since last run)_.
 2. **Attachment-less mail under an attachments-only rule is never recorded.**
   `_handle_message` (`mail.py:749`) returns early, before any attachment
   processing, when a message has no attachments and
   `rule.consumption_scope == ATTACHMENTS_ONLY`:
   ```python
   if (
       not message.attachments
       and rule.consumption_scope == MailRule.ConsumptionScope.ATTACHMENTS_ONLY
   ):
       return processed_elements  # 0, no ProcessedMail row, no mail action applied
   ```
   No `ProcessedMail` row is written and no mail action (mark-read/flag/tag)
   is applied, since both only happen via `queue_consumption_tasks`, which is
   only reached from the attachment/eml processing paths. The message keeps
   matching the search (e.g. `UNSEEN`) forever.
 Combined: an inbox where most mail has no attachments gets fully re-downloaded
 on every scheduled run (default every 10 minutes), indefinitely, for large
 mailboxes -- confirmed against the current codebase, not user
 misconfiguration.
 ## Goals
 - Stop re-downloading full message bodies for mail that has already been
  handled (processed into a document, or determined to produce nothing).
 - Record `ProcessedMail` for the no-attachment / attachments-only early-return
  case, so it participates in dedup like every other terminal outcome.
 - No new required settings or DB migrations for existing installs.
 - Graceful behavior on first run against a large existing mailbox (no single
  giant IMAP command, no giant in-memory batch).
 - Preserve existing action/consumption semantics exactly for every path that
  currently produces a document or applies a mail action.
 ## Non-goals
 - Changing what mail actions (mark-read/flag/tag/move/delete) get applied, or
  when.
 - Applying a mail action to the no-attachment early-return case (explicitly
  out of scope -- see Decisions).
 - Making batch size user-configurable.
 - Touching EML_ONLY / EVERYTHING consumption scopes' semantics (they already
  don't hit the early-return branch described above).
 ## Decisions
 These were settled during brainstorming and are load-bearing for the design
 below:
 - **Fix both halves together.** Recording `ProcessedMail` alone does not fix
  the bandwidth problem: without a mail action applied, an attachment-less
  message stays unseen and keeps matching the search criteria, so its body
  gets re-downloaded every run regardless of whether it gets reprocessed.
  Fixing only the bandwidth side without recording `ProcessedMail` would mean
  the UID-diff step never has anything to exclude for these messages. They
  must ship together.
 - **No mailbox action for the no-attachment case.** Only a `ProcessedMail`
  row is written; the message is not marked read/flagged/moved/deleted. This
  matches the existing precedent for other no-consumption outcomes
  (`mail.py:960-977`, e.g. an attachment present but filtered out or an
  unsupported mime type) and avoids changing the user's mailbox state for
  mail paperless previously never touched.
 - **UID-diff-before-fetch, not a BODYSTRUCTURE probe or SEARCH-side
  exclusion.** Considered and rejected:
  - BODYSTRUCTURE probing (fetch structure only, decide whether to fetch full
    body) only helps the attachments-only scope, not the general
    already-processed case, and `imap_tools` doesn't cleanly expose a
    structure-only fetch.
  - Excluding already-processed UIDs directly in the IMAP `SEARCH` criteria
    (`NOT UID (...)`) was rejected because for tens of thousands of
    already-processed UIDs the excluded-UID list itself blows up the command
    size -- worse than the two-step approach.
 - **Batch size is a hardcoded constant**, not a new setting, per the
  "no new required settings" goal.
 ## Design
 ### `_handle_mail_rule` (`mail.py:641`)
 Replace the single `M.fetch(criteria=criterias, ...)` call with:
 1. `self._current_uid_validity` computed as today (unchanged, already first).
 2. `criterias = make_criterias(...)` (unchanged).
 3. `all_uids = set(M.uids(criteria=criterias, charset=rule.account.character_set))`
   -- a `UID SEARCH`, no bodies.
 4. Query already-processed UIDs in one DB round trip, reusing the same
   uid_validity matching already used per-message:
   ```python
   processed_uids = set(
       ProcessedMail.objects.filter(
           rule=rule, folder=rule.folder, uid__in=all_uids,
       ).filter(
           Q(uid_validity=self._current_uid_validity) | Q(uid_validity__isnull=True),
       ).values_list("uid", flat=True)
   )
   ```
 5. `new_uids = all_uids - processed_uids`. If empty: log at debug level and
   `return 0` -- no body fetch at all. This is the steady-state case.
 6. Otherwise, iterate `new_uids` in fixed-size batches of
   `MAIL_FETCH_BATCH_SIZE = 500` (module-level constant), calling
   `M.fetch(criteria=AND(uid=batch), mark_seen=False, charset=rule.account.character_set, bulk=True)`
   per batch. Chain the resulting message iterators into the existing
   per-message loop (`mail.py:699+`) unchanged.
 7. The existing per-message `already_processed` DB check
   (`mail.py:717-731`) stays in place unchanged, as a safety net against
   races (e.g. concurrent rule runs against the same folder in this pass) and
   to keep behavior identical if steps 3/4 ever disagree with it.
 ### New helper: `_record_processed_without_consumption`
 Extract the existing dedup-and-create block at `mail.py:960-977` into a
 method on the same class:
 ```python
 def _record_processed_without_consumption(self, message, rule) -> None:
    if not ProcessedMail.objects.filter(
        rule=rule,
        uid=message.uid,
        folder=rule.folder,
        uid_validity=self._current_uid_validity,
    ).exists():
        ProcessedMail.objects.create(
            rule=rule,
            folder=rule.folder,
            uid=message.uid,
            uid_validity=self._current_uid_validity,
            subject=message.subject,
            received=make_aware(message.date) if is_naive(message.date) else message.date,
            status="PROCESSED_WO_CONSUMPTION",
        )
 ```
 Call sites:
 - `_process_attachments`'s existing no-consumables branch (replaces the
  inline block, same behavior).
 - `_handle_message`'s early return (`mail.py:756-760`), newly, for the
  no-attachments-under-attachments-only-scope case.
 ### Data flow (steady state, nothing new)
 search criteria (unchanged) -> `UID SEARCH` for matching UIDs (cheap) ->
 subtract UIDs already in `ProcessedMail` (cheap, DB-only) -> zero new UIDs ->
 no fetch, no body download.
 ### Data flow (new mail present)
 ... same as above -> some UIDs remain -> fetch bodies only for those,
 batched -> existing per-message processing, action application, and
 `ProcessedMail` recording, unchanged except the no-attachment early return
 now also calls `_record_processed_without_consumption`.
 ## Error handling
 - `M.uids(...)` raising is wrapped in the same `try/except` that currently
  wraps `M.fetch(...)` (`mail.py:683-693`), surfacing as `MailError`
  identically to today's search-failure behavior.
 - A batch's `M.fetch(...)` raising mid-loop is caught by the same
  `try/except`, applied per-batch. One failing batch fails the whole rule
  run for this pass -- matching today's all-or-nothing semantics (currently
  a single failed fetch already fails the whole rule).
 - Empty `new_uids` short-circuits before any fetch, so it can't hit fetch
  error paths at all for the common case.
 - `_record_processed_without_consumption` reuses the existing
  `ProcessedMail.objects.create` call already in production use; no new
  failure mode introduced.
 ## Testing
 In `src/paperless_mail/tests/test_mail.py`:
 - Extend the `BogusMailBox` test double with a `uids()` method that mirrors
  its existing `fetch()` criteria matching but returns only UIDs.
 - New test: an attachment-less message under an attachments-only mark-read
  rule is recorded as `PROCESSED_WO_CONSUMPTION` on the first run. On a
  second run, with the same message still unseen server-side, assert no
  body fetch happens (only `M.uids()` is called) -- i.e. steady state costs
  one search and zero body downloads.
 - New test: a folder with more new UIDs than `MAIL_FETCH_BATCH_SIZE` results
  in multiple batched `fetch` calls, and all messages are still processed
  (no messages dropped at a batch boundary).
 - Update existing `test_handle_empty_message` to assert a
  `PROCESSED_WO_CONSUMPTION` row now exists for the no-attachment case.
 - Regression: existing tests for `EML_ONLY`/`EVERYTHING` scopes and the
  multi-rule-same-folder dedup (`consumed_messages` set) should pass
  unchanged, since neither the per-message `already_processed` check nor the
  `consumed_messages` set logic is touched by this design.
 ## Open questions / risks
 - `imap_tools`'s `AND(uid=batch)` criteria builder needs to be confirmed to
  produce a valid `UID FETCH <list>` command at the batch sizes used here;
  covered by the batching test above.
 - If a mail server doesn't support `UID SEARCH` the same way it supports the
  existing `fetch`'s implicit search, `M.uids()` could behave differently
  from today's `M.fetch()` on some edge-case server. Existing project test
  coverage (via `BogusMailBox`) won't catch server-specific quirks; this is
  the same class of risk as any IMAP-behavior assumption already baked into
  this module.
@@ -47,7 +47,7 @@ dependencies = [
  "gotenberg-client~=0.14.0",
  "httpx-oauth~=0.16",
  "ijson>=3.2",
-  "imap-tools~=1.14.0",
+  "imap-tools~=1.13.0",
  "jinja2~=3.1.5",
  "langdetect~=1.0.9",
  "llama-index-core>=0.14.22",
@@ -1,5 +1,6 @@
 import { expect, test } from '@playwright/test'
 import path from 'node:path'
 import { mockFilterSelectionData } from '../mock-filter-selection-data'
 const REQUESTS_HAR = path.join(__dirname, 'requests/api-settings.har')
@@ -7,6 +8,7 @@ test('should activate / deactivate save button when settings change', async ({
  page,
 }) => {
  await page.routeFromHAR(REQUESTS_HAR, { notFound: 'fallback' })
  await mockFilterSelectionData(page)
  await page.goto('/settings')
  await expect(page.getByRole('button', { name: 'Save' })).toBeDisabled()
  await page.getByLabel('Use system setting').click()
@@ -16,6 +18,7 @@ test('should activate / deactivate save button when settings change', async ({
 test('should warn on unsaved changes', async ({ page }) => {
  await page.routeFromHAR(REQUESTS_HAR, { notFound: 'fallback' })
  await mockFilterSelectionData(page)
  await page.goto('/settings')
  await page.getByLabel('Use system setting').click()
  await page.getByRole('link', { name: 'Dashboard' }).click()
@@ -28,6 +31,7 @@ test('should warn on unsaved changes', async ({ page }) => {
 test('should apply appearance changes when set', async ({ page }) => {
  await page.routeFromHAR(REQUESTS_HAR, { notFound: 'fallback' })
  await mockFilterSelectionData(page)
  await page.goto('/settings')
  await expect(page.locator('html')).toHaveAttribute('data-bs-theme', /auto/)
  await page.getByLabel('Use system setting').click()
@@ -1,5 +1,6 @@
 import { expect, test } from '@playwright/test'
 import path from 'node:path'
 import { mockFilterSelectionData } from '../mock-filter-selection-data'
 const REQUESTS_HAR1 = path.join(__dirname, 'requests/api-dashboard1.har')
 const REQUESTS_HAR2 = path.join(__dirname, 'requests/api-dashboard2.har')
@@ -8,6 +9,7 @@ const REQUESTS_HAR4 = path.join(__dirname, 'requests/api-dashboard4.har')
 test('dashboard inbox link', async ({ page }) => {
  await page.routeFromHAR(REQUESTS_HAR1, { notFound: 'fallback' })
  await mockFilterSelectionData(page)
  await page.goto('/dashboard')
  await page.getByRole('link', { name: 'Documents in inbox' }).click()
  await expect(page).toHaveURL(/tags__id__in=9/)
@@ -16,6 +18,7 @@ test('dashboard inbox link', async ({ page }) => {
 test('dashboard total documents link', async ({ page }) => {
  await page.routeFromHAR(REQUESTS_HAR2, { notFound: 'fallback' })
  await mockFilterSelectionData(page)
  await page.goto('/dashboard')
  await page.getByRole('link').filter({ hasText: 'Total documents' }).click()
  await expect(page).toHaveURL(/documents/)
@@ -25,6 +28,7 @@ test('dashboard total documents link', async ({ page }) => {
 test('dashboard saved view show all', async ({ page }) => {
  await page.routeFromHAR(REQUESTS_HAR3, { notFound: 'fallback' })
  await mockFilterSelectionData(page)
  await page.goto('/dashboard')
  await page
    .locator('pngx-widget-frame')
@@ -38,6 +42,7 @@ test('dashboard saved view show all', async ({ page }) => {
 test('dashboard saved view document links', async ({ page }) => {
  await page.routeFromHAR(REQUESTS_HAR4, { notFound: 'fallback' })
  await mockFilterSelectionData(page)
  await page.goto('/dashboard')
  await page
    .locator('pngx-widget-frame')
@@ -51,6 +56,7 @@ test('dashboard saved view document links', async ({ page }) => {
 test('test slim sidebar', async ({ page }) => {
  await page.routeFromHAR(REQUESTS_HAR1, { notFound: 'fallback' })
  await mockFilterSelectionData(page)
  await page.goto('/dashboard')
  await page.locator('.sidebar-slim-toggler').click()
  await expect(
@@ -1,5 +1,6 @@
 import { expect, test } from '@playwright/test'
 import path from 'node:path'
 import { mockFilterSelectionData } from '../mock-filter-selection-data'
 const REQUESTS_HAR = path.join(__dirname, 'requests/api-document-detail.har')
 const REQUESTS_HAR2 = path.join(__dirname, 'requests/api-document-detail2.har')
@@ -8,6 +9,7 @@ test('should activate / deactivate save button when changes are saved', async ({
  page,
 }) => {
  await page.routeFromHAR(REQUESTS_HAR, { notFound: 'fallback' })
  await mockFilterSelectionData(page)
  await page.goto('/documents/175/')
  await page.waitForSelector('pngx-document-detail pngx-input-text:first-child')
  await expect(page.getByTitle('Storage path', { exact: true })).toHaveText(
@@ -20,6 +22,7 @@ test('should activate / deactivate save button when changes are saved', async ({
 test('should warn on unsaved changes', async ({ page }) => {
  await page.routeFromHAR(REQUESTS_HAR, { notFound: 'fallback' })
  await mockFilterSelectionData(page)
  await page.goto('/documents/175/')
  await expect(page.getByTitle('Correspondent', { exact: true })).toHaveText(
    /\w+/
@@ -39,6 +42,7 @@ test('should warn on unsaved changes', async ({ page }) => {
 test('should support tab direct navigation', async ({ page }) => {
  await page.routeFromHAR(REQUESTS_HAR, { notFound: 'fallback' })
  await mockFilterSelectionData(page)
  await page.goto('/documents/175/details')
  await expect(page.getByRole('tab', { name: 'Details' })).toHaveAttribute(
    'aria-selected',
@@ -68,6 +72,7 @@ test('should support tab direct navigation', async ({ page }) => {
 test('should show a mobile preview', async ({ page }) => {
  await page.routeFromHAR(REQUESTS_HAR, { notFound: 'fallback' })
  await mockFilterSelectionData(page)
  await page.goto('/documents/175/')
  await page.setViewportSize({ width: 400, height: 1000 })
  await expect(page.getByRole('tab', { name: 'Preview' })).toBeVisible()
@@ -77,6 +82,7 @@ test('should show a mobile preview', async ({ page }) => {
 test('should show a list of notes', async ({ page }) => {
  await page.routeFromHAR(REQUESTS_HAR, { notFound: 'fallback' })
  await mockFilterSelectionData(page)
  await page.goto('/documents/175/notes')
  await expect(page.locator('pngx-document-notes')).toBeVisible()
  await expect(
@@ -89,6 +95,7 @@ test('should show a list of notes', async ({ page }) => {
 test('should support quick filters', async ({ page }) => {
  await page.routeFromHAR(REQUESTS_HAR2, { notFound: 'fallback' })
  await mockFilterSelectionData(page)
  await page.goto('/documents/175/details')
  await page
    .getByRole('button', { name: 'Filter documents with these Tags' })
@@ -1,5 +1,6 @@
 import { expect, test } from '@playwright/test'
 import path from 'node:path'
 import { mockFilterSelectionData } from '../mock-filter-selection-data'
 const REQUESTS_HAR1 = path.join(__dirname, 'requests/api-document-list1.har')
 const REQUESTS_HAR2 = path.join(__dirname, 'requests/api-document-list2.har')
@@ -10,6 +11,7 @@ const REQUESTS_HAR6 = path.join(__dirname, 'requests/api-document-list6.har')
 test('basic filtering', async ({ page }) => {
  await page.routeFromHAR(REQUESTS_HAR1, { notFound: 'fallback' })
  await mockFilterSelectionData(page)
  await page.goto('/documents')
  await page.getByRole('button', { name: 'Tags' }).click()
  await page.getByRole('menuitem', { name: 'Inbox' }).click()
@@ -45,6 +47,7 @@ test('basic filtering', async ({ page }) => {
 test('text filtering', async ({ page }) => {
  await page.routeFromHAR(REQUESTS_HAR2, { notFound: 'fallback' })
  await mockFilterSelectionData(page)
  await page.goto('/documents')
  await page.getByRole('main').getByRole('combobox').click()
  await page.getByRole('main').getByRole('combobox').fill('test')
@@ -81,6 +84,7 @@ test('text filtering', async ({ page }) => {
 test('date filtering', async ({ page }) => {
  await page.routeFromHAR(REQUESTS_HAR3, { notFound: 'fallback' })
  await mockFilterSelectionData(page)
  await page.goto('/documents')
  await page.getByRole('button', { name: 'Dates' }).click()
  await page.locator('.ng-arrow-wrapper').first().click()
@@ -103,6 +107,7 @@ test('date filtering', async ({ page }) => {
 test('sorting', async ({ page }) => {
  await page.routeFromHAR(REQUESTS_HAR4, { notFound: 'fallback' })
  await mockFilterSelectionData(page)
  await page.goto('/documents')
  await page.getByRole('button', { name: 'Sort' }).click()
  await page.getByRole('button', { name: 'ASN' }).click()
@@ -141,6 +146,7 @@ test('sorting', async ({ page }) => {
 test('change views', async ({ page }) => {
  await page.routeFromHAR(REQUESTS_HAR5, { notFound: 'fallback' })
  await mockFilterSelectionData(page)
  await page.goto('/documents')
  await page.locator('.btn-group > label').first().click()
  await expect(page.locator('pngx-document-list table')).toBeVisible()
@@ -152,6 +158,7 @@ test('change views', async ({ page }) => {
 test('bulk edit', async ({ page }) => {
  await page.routeFromHAR(REQUESTS_HAR6, { notFound: 'fallback' })
  await mockFilterSelectionData(page)
  await page.goto('/documents')
  await page.locator('pngx-document-card-small').nth(0).click()
@@ -0,0 +1,35 @@
 import { Page } from '@playwright/test'
 const EMPTY_SELECTION_DATA = {
  selected_correspondents: [],
  selected_tags: [],
  selected_document_types: [],
  selected_storage_paths: [],
  selected_custom_fields: [],
 }
 /**
 * The document list now fires a GET to filter_selection_data on every
 * non-search reload(), independent of and concurrent with the main list
 * request. It's not present in any of the recorded HAR fixtures, so with
 * `notFound: 'fallback'` it would otherwise fall through to the real
 * network (nothing listens there in e2e, since only the frontend dev
 * server is started) and fail every test that reloads the list.
 *
 * Playwright checks routes in reverse-registration order, so this must be
 * registered before a test's own page.routeFromHAR() call for the HAR
 * route's `notFound: 'fallback'` to defer back to this one.
 */
 export async function mockFilterSelectionData(page: Page) {
  await page.route('**/api/documents/filter_selection_data/**', (route) =>
    route.fulfill({
      json: EMPTY_SELECTION_DATA,
      // The app calls the (cross-origin, from the e2e app's perspective)
      // backend at http://localhost:8000 while served from :4200, so a
      // fulfilled response needs the same CORS header the real backend
      // sends (and that recorded HAR responses already carry) or the
      // browser rejects it as a cross-origin failure.
      headers: { 'Access-Control-Allow-Origin': 'http://localhost:4200' },
    })
  )
 }
@@ -1,10 +1,12 @@
 import { expect, test } from '@playwright/test'
 import path from 'node:path'
 import { mockFilterSelectionData } from '../mock-filter-selection-data'
 const REQUESTS_HAR = path.join(__dirname, 'requests/api-global-permissions.har')
 test('should not allow user to edit settings', async ({ page }) => {
  await page.routeFromHAR(REQUESTS_HAR, { notFound: 'fallback' })
  await mockFilterSelectionData(page)
  await page.goto('/dashboard')
  await expect(page.getByRole('link', { name: 'Settings' })).not.toBeAttached()
  await page.goto('/settings')
@@ -15,6 +17,7 @@ test('should not allow user to edit settings', async ({ page }) => {
 test('should not allow user to view documents', async ({ page }) => {
  await page.routeFromHAR(REQUESTS_HAR, { notFound: 'fallback' })
  await mockFilterSelectionData(page)
  await page.goto('/dashboard')
  await expect(
    page.locator('nav').getByRole('link', { name: 'Documents' })
@@ -31,6 +34,7 @@ test('should not allow user to view documents', async ({ page }) => {
 test('should not allow user to view correspondents', async ({ page }) => {
  await page.routeFromHAR(REQUESTS_HAR, { notFound: 'fallback' })
  await mockFilterSelectionData(page)
  await page.goto('/dashboard')
  await expect(
    page.getByRole('link', { name: 'Attributes' })
@@ -43,6 +47,7 @@ test('should not allow user to view correspondents', async ({ page }) => {
 test('should not allow user to view tags', async ({ page }) => {
  await page.routeFromHAR(REQUESTS_HAR, { notFound: 'fallback' })
  await mockFilterSelectionData(page)
  await page.goto('/dashboard')
  await expect(
    page.getByRole('link', { name: 'Attributes' })
@@ -55,6 +60,7 @@ test('should not allow user to view tags', async ({ page }) => {
 test('should not allow user to view document types', async ({ page }) => {
  await page.routeFromHAR(REQUESTS_HAR, { notFound: 'fallback' })
  await mockFilterSelectionData(page)
  await page.goto('/dashboard')
  await expect(
    page.getByRole('link', { name: 'Attributes' })
@@ -67,6 +73,7 @@ test('should not allow user to view document types', async ({ page }) => {
 test('should not allow user to view storage paths', async ({ page }) => {
  await page.routeFromHAR(REQUESTS_HAR, { notFound: 'fallback' })
  await mockFilterSelectionData(page)
  await page.goto('/dashboard')
  await expect(
    page.getByRole('link', { name: 'Attributes' })
@@ -79,6 +86,7 @@ test('should not allow user to view storage paths', async ({ page }) => {
 test('should not allow user to view logs', async ({ page }) => {
  await page.routeFromHAR(REQUESTS_HAR, { notFound: 'fallback' })
  await mockFilterSelectionData(page)
  await page.goto('/dashboard')
  await expect(page.getByRole('link', { name: 'Logs' })).not.toBeAttached()
  await page.goto('/logs')
@@ -89,6 +97,7 @@ test('should not allow user to view logs', async ({ page }) => {
 test('should not allow user to view tasks', async ({ page }) => {
  await page.routeFromHAR(REQUESTS_HAR, { notFound: 'fallback' })
  await mockFilterSelectionData(page)
  await page.goto('/dashboard')
  await expect(page.getByRole('link', { name: 'Tasks' })).not.toBeAttached()
  await page.goto('/tasks')
@@ -191,6 +191,14 @@ describe('BulkEditorComponent', () => {
  })
  afterEach(async () => {
    // A filter_selection_data request now fires concurrently with every
    // non-search reload(), independent of whether a given test flushes or
    // even inspects the primary list response. Drain any left unclaimed.
    httpTestingController.match(
      (request) =>
        request.url ===
        `${environment.apiBaseUrl}documents/filter_selection_data/`
    )
    httpTestingController.verify()
  })
@@ -386,7 +394,7 @@ describe('BulkEditorComponent', () => {
      parameters: { add_tags: [101], remove_tags: [] },
    })
    httpTestingController.match(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
    ) // list reload
    httpTestingController.match(
      `${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
@@ -432,7 +440,7 @@ describe('BulkEditorComponent', () => {
      parameters: { add_tags: [101], remove_tags: [] },
    })
    httpTestingController.match(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
    ) // list reload
  })
@@ -461,7 +469,7 @@ describe('BulkEditorComponent', () => {
      .expectOne(`${environment.apiBaseUrl}documents/bulk_edit/`)
      .flush(true)
    httpTestingController.match(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
    ) // list reload
    httpTestingController.match(
      `${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
@@ -552,7 +560,7 @@ describe('BulkEditorComponent', () => {
      parameters: { correspondent: 101 },
    })
    httpTestingController.match(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
    ) // list reload
    httpTestingController.match(
      `${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
@@ -584,7 +592,7 @@ describe('BulkEditorComponent', () => {
      .expectOne(`${environment.apiBaseUrl}documents/bulk_edit/`)
      .flush(true)
    httpTestingController.match(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
    ) // list reload
    httpTestingController.match(
      `${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
@@ -650,7 +658,7 @@ describe('BulkEditorComponent', () => {
      parameters: { document_type: 101 },
    })
    httpTestingController.match(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
    ) // list reload
    httpTestingController.match(
      `${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
@@ -682,7 +690,7 @@ describe('BulkEditorComponent', () => {
      .expectOne(`${environment.apiBaseUrl}documents/bulk_edit/`)
      .flush(true)
    httpTestingController.match(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
    ) // list reload
    httpTestingController.match(
      `${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
@@ -748,7 +756,7 @@ describe('BulkEditorComponent', () => {
      parameters: { storage_path: 101 },
    })
    httpTestingController.match(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
    ) // list reload
    httpTestingController.match(
      `${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
@@ -780,7 +788,7 @@ describe('BulkEditorComponent', () => {
      .expectOne(`${environment.apiBaseUrl}documents/bulk_edit/`)
      .flush(true)
    httpTestingController.match(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
    ) // list reload
    httpTestingController.match(
      `${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
@@ -846,7 +854,7 @@ describe('BulkEditorComponent', () => {
      parameters: { add_custom_fields: [101], remove_custom_fields: [102] },
    })
    httpTestingController.match(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
    ) // list reload
    httpTestingController.match(
      `${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
@@ -878,7 +886,7 @@ describe('BulkEditorComponent', () => {
      .expectOne(`${environment.apiBaseUrl}documents/bulk_edit/`)
      .flush(true)
    httpTestingController.match(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
    ) // list reload
    httpTestingController.match(
      `${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
@@ -987,7 +995,7 @@ describe('BulkEditorComponent', () => {
      documents: [3, 4],
    })
    httpTestingController.match(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
    ) // list reload
    httpTestingController.match(
      `${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
@@ -1080,7 +1088,7 @@ describe('BulkEditorComponent', () => {
      documents: [3, 4],
    })
    httpTestingController.match(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
    ) // list reload
    httpTestingController.match(
      `${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
@@ -1115,7 +1123,7 @@ describe('BulkEditorComponent', () => {
      source_mode: 'latest_version',
    })
    httpTestingController.match(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
    ) // list reload
    httpTestingController.match(
      `${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
@@ -1156,7 +1164,7 @@ describe('BulkEditorComponent', () => {
      metadata_document_id: 3,
    })
    httpTestingController.match(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
    ) // list reload
    httpTestingController.match(
      `${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
@@ -1175,7 +1183,7 @@ describe('BulkEditorComponent', () => {
      delete_originals: true,
    })
    httpTestingController.match(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
    ) // list reload
    httpTestingController.match(
      `${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
@@ -1196,7 +1204,7 @@ describe('BulkEditorComponent', () => {
      archive_fallback: true,
    })
    httpTestingController.match(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
    ) // list reload
    httpTestingController.match(
      `${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
@@ -1299,7 +1307,7 @@ describe('BulkEditorComponent', () => {
      },
    })
    httpTestingController.match(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
    ) // list reload
    httpTestingController.match(
      `${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
@@ -1607,7 +1615,7 @@ describe('BulkEditorComponent', () => {
    expect(toastServiceShowInfoSpy).toHaveBeenCalled()
    expect(listReloadSpy).toHaveBeenCalled()
    httpTestingController.match(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
    ) // list reload
    httpTestingController.match(
      `${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
@@ -84,6 +84,28 @@ const view: SavedView = {
  filter_rules: filterRules,
 }
 const emptySelectionData = {
  selected_correspondents: [],
  selected_tags: [],
  selected_document_types: [],
  selected_storage_paths: [],
  selected_custom_fields: [],
 }
 // A successful (non-search) list response now triggers a separate,
 // non-blocking request for filter dropdown counts. Tests that flush a
 // successful list response need to also flush this follow-up request.
 function flushSelectionDataRequest(
  httpTestingController: HttpTestingController,
  querySuffix: string = ''
 ) {
  const req = httpTestingController.expectOne(
    `${environment.apiBaseUrl}documents/filter_selection_data/${querySuffix}`
  )
  expect(req.request.method).toEqual('GET')
  req.flush(emptySelectionData)
 }
 describe('DocumentListViewService', () => {
  let httpTestingController: HttpTestingController
  let documentListViewService: DocumentListViewService
@@ -105,6 +127,7 @@ describe('DocumentListViewService', () => {
    })
    sessionStorage.clear()
    localStorage.clear()
    httpTestingController = TestBed.inject(HttpTestingController)
    documentListViewService = TestBed.inject(DocumentListViewService)
    settingsService = TestBed.inject(SettingsService)
@@ -114,8 +137,19 @@ describe('DocumentListViewService', () => {
  afterEach(() => {
    documentListViewService.cancelPending()
    // A filter_selection_data request now fires concurrently with every
    // non-search reload(), independent of whether the test cares about or
    // flushes the primary list response. Drain any that a test didn't
    // explicitly claim via flushSelectionDataRequest, so unrelated tests
    // don't have to know about this follow-up request to pass verify().
    httpTestingController.match(
      (request) =>
        request.url ===
        `${environment.apiBaseUrl}documents/filter_selection_data/`
    )
    httpTestingController.verify()
    sessionStorage.clear()
    localStorage.clear()
  })
  afterAll(() => {
@@ -128,10 +162,11 @@ describe('DocumentListViewService', () => {
    expect(documentListViewService.currentPage).toEqual(1)
    documentListViewService.reload()
    const req = httpTestingController.expectOne(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
    )
    expect(req.request.method).toEqual('GET')
    req.flush(full_results)
    flushSelectionDataRequest(httpTestingController)
    expect(req.request.method).toEqual('GET')
    expect(documentListViewService.isReloading).toBeFalsy()
    expect(documentListViewService.activeSavedViewId).toBeNull()
@@ -143,12 +178,12 @@ describe('DocumentListViewService', () => {
  it('should handle error on page request out of range', () => {
    documentListViewService.currentPage = 50
    let req = httpTestingController.expectOne(
-      `${environment.apiBaseUrl}documents/?page=50&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=50&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
    )
    expect(req.request.method).toEqual('GET')
    req.flush([], { status: 404, statusText: 'Unexpected error' })
    req = httpTestingController.expectOne(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
    )
    expect(req.request.method).toEqual('GET')
    expect(documentListViewService.currentPage).toEqual(1)
@@ -165,21 +200,20 @@ describe('DocumentListViewService', () => {
    ]
    documentListViewService.setFilterRules(filterRulesAny)
    let req = httpTestingController.expectOne(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true&tags__id__in=${tags__id__in}`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false&tags__id__in=${tags__id__in}`
    )
    expect(req.request.method).toEqual('GET')
    req.flush(
      { archive_serial_number: 'hello' },
      { status: 404, statusText: 'Unexpected error' }
    )
-    req = httpTestingController.expectOne(
+    // the error is a plain field error (not a page-out-of-range or deleted
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
+    // custom-field-sort case), so no automatic retry request is sent here
-    )
+    expect(documentListViewService.error).toBeTruthy()
    expect(req.request.method).toEqual('GET')
    // reset the list
    documentListViewService.setFilterRules([])
    req = httpTestingController.expectOne(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
    )
  })
@@ -187,7 +221,7 @@ describe('DocumentListViewService', () => {
    documentListViewService.currentPage = 1
    documentListViewService.sortField = 'custom_field_999'
    let req = httpTestingController.expectOne(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-custom_field_999&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-custom_field_999&truncate_content=true&include_selection_data=false`
    )
    expect(req.request.method).toEqual('GET')
    req.flush(
@@ -196,7 +230,7 @@ describe('DocumentListViewService', () => {
    )
    // resets itself
    req = httpTestingController.expectOne(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
    )
  })
@@ -211,7 +245,7 @@ describe('DocumentListViewService', () => {
    ]
    documentListViewService.setFilterRules(filterRulesAny)
    let req = httpTestingController.expectOne(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true&tags__id__in=${tags__id__in}`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false&tags__id__in=${tags__id__in}`
    )
    expect(req.request.method).toEqual('GET')
    req.flush('Generic error', { status: 404, statusText: 'Unexpected error' })
@@ -219,7 +253,7 @@ describe('DocumentListViewService', () => {
    // reset the list
    documentListViewService.setFilterRules([])
    req = httpTestingController.expectOne(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
    )
  })
@@ -228,7 +262,7 @@ describe('DocumentListViewService', () => {
    expect(documentListViewService.sortReverse).toBeTruthy()
    documentListViewService.setSort('added', false)
    let req = httpTestingController.expectOne(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=added&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=added&truncate_content=true&include_selection_data=false`
    )
    expect(req.request.method).toEqual('GET')
    expect(documentListViewService.sortField).toEqual('added')
@@ -236,12 +270,12 @@ describe('DocumentListViewService', () => {
    documentListViewService.sortField = 'created'
    req = httpTestingController.expectOne(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=created&truncate_content=true&include_selection_data=false`
    )
    expect(documentListViewService.sortField).toEqual('created')
    documentListViewService.sortReverse = true
    req = httpTestingController.expectOne(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
    )
    expect(req.request.method).toEqual('GET')
    expect(documentListViewService.sortReverse).toBeTruthy()
@@ -284,7 +318,7 @@ describe('DocumentListViewService', () => {
    const req = httpTestingController.expectOne(
      `${environment.apiBaseUrl}documents/?page=${page}&page_size=${
        documentListViewService.pageSize
-      }&ordering=${reverse ? '-' : ''}${sort}&truncate_content=true&include_selection_data=true`
+      }&ordering=${reverse ? '-' : ''}${sort}&truncate_content=true&include_selection_data=false`
    )
    expect(req.request.method).toEqual('GET')
    expect(documentListViewService.currentPage).toEqual(page)
@@ -301,7 +335,7 @@ describe('DocumentListViewService', () => {
    }
    documentListViewService.loadFromQueryParams(convertToParamMap(params))
    let req = httpTestingController.expectOne(
-      `${environment.apiBaseUrl}documents/?page=${documentListViewService.currentPage}&page_size=${documentListViewService.pageSize}&ordering=-added&truncate_content=true&include_selection_data=true&tags__id__all=${tags__id__all}`
+      `${environment.apiBaseUrl}documents/?page=${documentListViewService.currentPage}&page_size=${documentListViewService.pageSize}&ordering=-added&truncate_content=true&include_selection_data=false&tags__id__all=${tags__id__all}`
    )
    expect(req.request.method).toEqual('GET')
    expect(documentListViewService.filterRules).toEqual([
@@ -311,12 +345,16 @@ describe('DocumentListViewService', () => {
      },
    ])
    req.flush(full_results)
    flushSelectionDataRequest(
      httpTestingController,
      `?tags__id__all=${tags__id__all}`
    )
  })
  it('should use filter rules to update query params', () => {
    documentListViewService.setFilterRules(filterRules)
    const req = httpTestingController.expectOne(
-      `${environment.apiBaseUrl}documents/?page=${documentListViewService.currentPage}&page_size=${documentListViewService.pageSize}&ordering=-created&truncate_content=true&include_selection_data=true&tags__id__all=${tags__id__all}`
+      `${environment.apiBaseUrl}documents/?page=${documentListViewService.currentPage}&page_size=${documentListViewService.pageSize}&ordering=-created&truncate_content=true&include_selection_data=false&tags__id__all=${tags__id__all}`
    )
    expect(req.request.method).toEqual('GET')
  })
@@ -325,26 +363,31 @@ describe('DocumentListViewService', () => {
    documentListViewService.currentPage = 2
    let req = httpTestingController.expectOne((request) =>
      request.urlWithParams.startsWith(
-        `${environment.apiBaseUrl}documents/?page=2&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
+        `${environment.apiBaseUrl}documents/?page=2&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
      )
    )
    expect(req.request.method).toEqual('GET')
    req.flush(full_results)
    flushSelectionDataRequest(httpTestingController)
    documentListViewService.setFilterRules(filterRules, true)
    const filteredReqs = httpTestingController.match(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true&tags__id__all=${tags__id__all}`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false&tags__id__all=${tags__id__all}`
    )
    expect(filteredReqs).toHaveLength(1)
    filteredReqs[0].flush(full_results)
    flushSelectionDataRequest(
      httpTestingController,
      `?tags__id__all=${tags__id__all}`
    )
    expect(documentListViewService.currentPage).toEqual(1)
  })
  it('should support quick filter', () => {
    documentListViewService.quickFilter(filterRules)
    const req = httpTestingController.expectOne(
-      `${environment.apiBaseUrl}documents/?page=${documentListViewService.currentPage}&page_size=${documentListViewService.pageSize}&ordering=-created&truncate_content=true&include_selection_data=true&tags__id__all=${tags__id__all}`
+      `${environment.apiBaseUrl}documents/?page=${documentListViewService.currentPage}&page_size=${documentListViewService.pageSize}&ordering=-created&truncate_content=true&include_selection_data=false&tags__id__all=${tags__id__all}`
    )
    expect(req.request.method).toEqual('GET')
  })
@@ -367,21 +410,21 @@ describe('DocumentListViewService', () => {
      convertToParamMap(params)
    )
    let req = httpTestingController.expectOne(
-      `${environment.apiBaseUrl}documents/?page=${page}&page_size=${documentListViewService.pageSize}&ordering=-added&truncate_content=true&include_selection_data=true&tags__id__all=${tags__id__all}`
+      `${environment.apiBaseUrl}documents/?page=${page}&page_size=${documentListViewService.pageSize}&ordering=-added&truncate_content=true&include_selection_data=false&tags__id__all=${tags__id__all}`
    )
    expect(req.request.method).toEqual('GET')
    // reset the list
    documentListViewService.currentPage = 1
    req = httpTestingController.expectOne(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-added&truncate_content=true&include_selection_data=true&tags__id__all=9`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-added&truncate_content=true&include_selection_data=false&tags__id__all=9`
    )
    documentListViewService.setFilterRules([])
    req = httpTestingController.expectOne(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-added&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-added&truncate_content=true&include_selection_data=false`
    )
    documentListViewService.sortField = 'created'
    req = httpTestingController.expectOne(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
    )
    documentListViewService.activateSavedView(null)
  })
@@ -389,18 +432,22 @@ describe('DocumentListViewService', () => {
  it('should support navigating next / previous', () => {
    documentListViewService.setFilterRules([])
    let req = httpTestingController.expectOne(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
    )
    expect(documentListViewService.currentPage).toEqual(1)
    documentListViewService.pageSize = 3
    req = httpTestingController.expectOne(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=3&ordering=-created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=3&ordering=-created&truncate_content=true&include_selection_data=false`
    )
    expect(req.request.method).toEqual('GET')
    req.flush({
      count: 3,
      results: documents.slice(0, 3),
    })
    // two reload()s ran above (setFilterRules, then pageSize), each firing
    // its own concurrent filter_selection_data request with an identical
    // (unfiltered) URL; this test doesn't assert on selectionData, so let
    // afterEach's drain step clean both up rather than disambiguating here.
    expect(documentListViewService.hasNext(documents[0].id)).toBeTruthy()
    expect(documentListViewService.hasPrevious(documents[0].id)).toBeFalsy()
    documentListViewService.getNext(documents[0].id).subscribe((docId) => {
@@ -447,7 +494,7 @@ describe('DocumentListViewService', () => {
    expect(documentListViewService.currentPage).toEqual(1)
    documentListViewService.pageSize = 3
    httpTestingController.match(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=3&ordering=-created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=3&ordering=-created&truncate_content=true&include_selection_data=false`
    )
    jest
      .spyOn(documentListViewService, 'getLastPage')
@@ -462,7 +509,7 @@ describe('DocumentListViewService', () => {
    expect(reloadSpy).toHaveBeenCalled()
    expect(documentListViewService.currentPage).toEqual(2)
    const reqs = httpTestingController.match(
-      `${environment.apiBaseUrl}documents/?page=2&page_size=3&ordering=-created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=2&page_size=3&ordering=-created&truncate_content=true&include_selection_data=false`
    )
    expect(reqs.length).toBeGreaterThan(0)
  })
@@ -497,11 +544,11 @@ describe('DocumentListViewService', () => {
      .mockReturnValue(documents)
    documentListViewService.currentPage = 2
    httpTestingController.match(
-      `${environment.apiBaseUrl}documents/?page=2&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=2&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
    )
    documentListViewService.pageSize = 3
    httpTestingController.match(
-      `${environment.apiBaseUrl}documents/?page=2&page_size=3&ordering=-created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=2&page_size=3&ordering=-created&truncate_content=true&include_selection_data=false`
    )
    const reloadSpy = jest.spyOn(documentListViewService, 'reload')
    documentListViewService.getPrevious(1).subscribe({
@@ -511,7 +558,7 @@ describe('DocumentListViewService', () => {
    expect(reloadSpy).toHaveBeenCalled()
    expect(documentListViewService.currentPage).toEqual(1)
    const reqs = httpTestingController.match(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=3&ordering=-created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=3&ordering=-created&truncate_content=true&include_selection_data=false`
    )
    expect(reqs.length).toBeGreaterThan(0)
  })
@@ -524,10 +571,11 @@ describe('DocumentListViewService', () => {
  it('should support select a document', () => {
    documentListViewService.reload()
    const req = httpTestingController.expectOne(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
    )
    expect(req.request.method).toEqual('GET')
    req.flush(full_results)
    flushSelectionDataRequest(httpTestingController)
    documentListViewService.toggleSelected(documents[0])
    expect(documentListViewService.isSelected(documents[0])).toBeTruthy()
    documentListViewService.toggleSelected(documents[0])
@@ -537,10 +585,11 @@ describe('DocumentListViewService', () => {
  it('should support select all', () => {
    documentListViewService.reload()
    const reloadReq = httpTestingController.expectOne(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
    )
    expect(reloadReq.request.method).toEqual('GET')
    reloadReq.flush(full_results)
    flushSelectionDataRequest(httpTestingController)
    documentListViewService.selectAll()
    expect(documentListViewService.allSelected).toBeTruthy()
@@ -553,13 +602,14 @@ describe('DocumentListViewService', () => {
  it('should support select page', () => {
    documentListViewService.pageSize = 3
    const req = httpTestingController.expectOne(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=3&ordering=-created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=3&ordering=-created&truncate_content=true&include_selection_data=false`
    )
    expect(req.request.method).toEqual('GET')
    req.flush({
      count: 3,
      results: documents.slice(0, 3),
    })
    flushSelectionDataRequest(httpTestingController)
    documentListViewService.selectPage()
    expect(documentListViewService.selected.size).toEqual(3)
    expect(documentListViewService.isSelected(documents[5])).toBeFalsy()
@@ -568,10 +618,11 @@ describe('DocumentListViewService', () => {
  it('should support select range', () => {
    documentListViewService.reload()
    const req = httpTestingController.expectOne(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
    )
    expect(req.request.method).toEqual('GET')
    req.flush(full_results)
    flushSelectionDataRequest(httpTestingController)
    documentListViewService.toggleSelected(documents[0])
    expect(documentListViewService.isSelected(documents[0])).toBeTruthy()
    documentListViewService.selectRangeTo(documents[2])
@@ -583,9 +634,10 @@ describe('DocumentListViewService', () => {
  it('should clear all-selected mode when toggling a single document', () => {
    documentListViewService.reload()
    const req = httpTestingController.expectOne(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
    )
    req.flush(full_results)
    flushSelectionDataRequest(httpTestingController)
    documentListViewService.selectAll()
    expect(documentListViewService.allSelected).toBeTruthy()
@@ -599,9 +651,10 @@ describe('DocumentListViewService', () => {
  it('should clear all-selected mode when selecting a range', () => {
    documentListViewService.reload()
    const req = httpTestingController.expectOne(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
    )
    req.flush(full_results)
    flushSelectionDataRequest(httpTestingController)
    documentListViewService.selectAll()
    documentListViewService.toggleSelected(documents[1])
@@ -619,22 +672,24 @@ describe('DocumentListViewService', () => {
  it('should support selection range reduction', () => {
    documentListViewService.reload()
    let req = httpTestingController.expectOne(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
    )
    expect(req.request.method).toEqual('GET')
    req.flush(full_results)
    flushSelectionDataRequest(httpTestingController)
    documentListViewService.selectAll()
    expect(documentListViewService.selected.size).toEqual(6)
    documentListViewService.setFilterRules(filterRules)
    req = httpTestingController.expectOne(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true&tags__id__all=9`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false&tags__id__all=9`
    )
    req.flush({
      count: 3,
      results: documents.slice(0, 3),
    })
    flushSelectionDataRequest(httpTestingController, '?tags__id__all=9')
    expect(documentListViewService.allSelected).toBeTruthy()
    expect(documentListViewService.selected.size).toEqual(3)
  })
@@ -643,7 +698,7 @@ describe('DocumentListViewService', () => {
    const cancelSpy = jest.spyOn(documentListViewService, 'cancelPending')
    documentListViewService.reload()
    httpTestingController.expectOne(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true&tags__id__all=9`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
    )
    expect(cancelSpy).toHaveBeenCalled()
  })
@@ -662,7 +717,7 @@ describe('DocumentListViewService', () => {
    documentListViewService.setFilterRules([])
    expect(documentListViewService.sortField).toEqual('created')
    httpTestingController.expectOne(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
    )
  })
@@ -689,11 +744,11 @@ describe('DocumentListViewService', () => {
    expect(localStorageSpy).toHaveBeenCalled()
    // reload triggered
    httpTestingController.match(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
    )
    documentListViewService.displayFields = null
    httpTestingController.match(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
    )
    expect(documentListViewService.displayFields).toEqual(
      DEFAULT_DISPLAY_FIELDS.filter((f) => f.id !== DisplayField.ADDED).map(
@@ -738,7 +793,7 @@ describe('DocumentListViewService', () => {
  it('should generate quick filter URL preserving default state', () => {
    documentListViewService.reload()
    httpTestingController.expectOne(
-      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
+      `${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=false`
    )
    const urlTree = documentListViewService.getQuickFilterUrl(filterRules)
    expect(urlTree).toBeDefined()
@@ -314,12 +314,39 @@ export class DocumentListViewService {
    }
  }
  private loadFilterSelectionData(filterRules: FilterRule[]) {
    this.documentService
      .getFilterSelectionData(filterRules)
      .pipe(takeUntil(this.unsubscribeNotifier))
      .subscribe({
        next: (selectionData) => {
          this.selectionData = selectionData
          this.markChanged()
        },
        error: () => {
          this.selectionData = null
          this.markChanged()
        },
      })
  }
  reload(onFinish?, updateQueryParams: boolean = true) {
    this.cancelPending()
    this.isReloading = true
    this.error = null
    this.markChanged()
    let activeListViewState = this.activeListViewState
    // Full-text search results are already narrowed by the search backend, so
    // computing selection data inline there is cheap. A plain (unfiltered or
    // ORM-filtered) browse can span the entire document set, so its selection
    // data is fetched separately -- concurrently with the list itself, rather
    // than blocking or waiting on it.
    const isFullTextSearch = isFullTextFilterRule(
      activeListViewState.filterRules
    )
    if (!isFullTextSearch) {
      this.loadFilterSelectionData(activeListViewState.filterRules)
    }
    this.documentService
      .listFiltered(
        activeListViewState.currentPage,
@@ -327,17 +354,22 @@ export class DocumentListViewService {
        activeListViewState.sortField,
        activeListViewState.sortReverse,
        activeListViewState.filterRules,
-        { truncate_content: true, include_selection_data: true }
+        {
          truncate_content: true,
          include_selection_data: isFullTextSearch,
        }
      )
      .pipe(takeUntil(this.unsubscribeNotifier))
      .subscribe({
        next: (result) => {
          const resultWithSelectionData = result as DocumentResults
          this.initialized = true
          this.isReloading = false
          activeListViewState.collectionSize = result.count
          activeListViewState.documents = result.results
-          this.selectionData = resultWithSelectionData.selection_data ?? null
+          if (isFullTextSearch) {
            this.selectionData =
              (result as DocumentResults).selection_data ?? null
          }
          this.syncSelectedToCurrentPage()
          this.markChanged()
@@ -376,6 +408,9 @@ export class DocumentListViewService {
            // e.g. field was deleted
            this.sortField = 'created'
          } else {
            // cancel the concurrently-fired selection-data request too, so it
            // can't resolve afterward and clobber this reset with stale data
            this.cancelPending()
            this.selectionData = null
            let errorMessage
            if (
@@ -41,6 +41,24 @@ export abstract class AbstractPaperlessService<T extends ObjectWithId> {
    }
  }
  /**
   * Merges a plain params object into an HttpParams instance, skipping
   * null/undefined values so they aren't serialized as literal "null" /
   * "undefined" query string entries.
   */
  protected withParams(
    params,
    base: HttpParams = new HttpParams()
  ): HttpParams {
    let httpParams = base
    for (let key in params) {
      if (params[key] != null) {
        httpParams = httpParams.set(key, params[key])
      }
    }
    return httpParams
  }
  list(
    page?: number,
    pageSize?: number,
@@ -60,11 +78,7 @@ export abstract class AbstractPaperlessService<T extends ObjectWithId> {
    if (ordering) {
      httpParams = httpParams.set('ordering', ordering)
    }
-    for (let extraParamKey in extraParams) {
+    httpParams = this.withParams(extraParams, httpParams)
      if (extraParams[extraParamKey] != null) {
        httpParams = httpParams.set(extraParamKey, extraParams[extraParamKey])
      }
    }
    return this.http
      .get<Results<T>>(this.getResourceUrl(), {
        params: httpParams,
@@ -113,11 +127,7 @@ export abstract class AbstractPaperlessService<T extends ObjectWithId> {
    httpParams = httpParams.set('id__in', ids.join(','))
    httpParams = httpParams.set('ordering', '-id')
    httpParams = httpParams.set('page_size', 1000)
-    for (let extraParamKey in extraParams) {
+    httpParams = this.withParams(extraParams, httpParams)
      if (extraParams[extraParamKey] != null) {
        httpParams = httpParams.set(extraParamKey, extraParams[extraParamKey])
      }
    }
    return this.http
      .get<Results<T>>(this.getResourceUrl(), {
        params: httpParams,
@@ -398,6 +398,13 @@ export class DocumentService extends AbstractPaperlessService<Document> {
    )
  }
  getFilterSelectionData(filterRules: FilterRule[]): Observable<SelectionData> {
    return this.http.get<SelectionData>(
      this.getResourceUrl(null, 'filter_selection_data'),
      { params: this.withParams(queryParamsFromFilterRules(filterRules)) }
    )
  }
  getSuggestions(id: number): Observable<DocumentSuggestions> {
    return this.http.get<DocumentSuggestions>(
      this.getResourceUrl(id, 'suggestions')
@@ -1241,7 +1241,7 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
            ],
        )
-    def test_list_with_include_selection_data(self) -> None:
+    def test_selection_data_endpoint(self) -> None:
        correspondent = Correspondent.objects.create(name="c1")
        doc_type = DocumentType.objects.create(name="dt1")
        storage_path = StoragePath.objects.create(name="sp1")
@@ -1259,30 +1259,28 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
        non_matching_doc.tags.add(Tag.objects.create(name="other"))
        response = self.client.get(
-            f"/api/documents/?tags__id__in={tag.id}&include_selection_data=true",
+            f"/api/documents/filter_selection_data/?tags__id__in={tag.id}",
        )
        self.assertEqual(response.status_code, status.HTTP_200_OK)
-        self.assertIn("selection_data", response.data)
+        self.assertNotIn("results", response.data)
        selected_correspondent = next(
            item
-            for item in response.data["selection_data"]["selected_correspondents"]
+            for item in response.data["selected_correspondents"]
            if item["id"] == correspondent.id
        )
        selected_tag = next(
-            item
+            item for item in response.data["selected_tags"] if item["id"] == tag.id
            for item in response.data["selection_data"]["selected_tags"]
            if item["id"] == tag.id
        )
        selected_type = next(
            item
-            for item in response.data["selection_data"]["selected_document_types"]
+            for item in response.data["selected_document_types"]
            if item["id"] == doc_type.id
        )
        selected_storage_path = next(
            item
-            for item in response.data["selection_data"]["selected_storage_paths"]
+            for item in response.data["selected_storage_paths"]
            if item["id"] == storage_path.id
        )
@@ -1291,6 +1289,17 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
        self.assertEqual(selected_type["document_count"], 1)
        self.assertEqual(selected_storage_path["document_count"], 1)
    def test_list_no_longer_supports_include_selection_data(self) -> None:
        """
        include_selection_data was never part of a stable release (beta-only,
        introduced and removed within the 3.0.0-beta cycle) -- the plain list
        endpoint should just ignore the param now rather than compute it inline.
        """
        response = self.client.get("/api/documents/?include_selection_data=true")
        self.assertEqual(response.status_code, status.HTTP_200_OK)
        self.assertNotIn("selection_data", response.data)
    def test_statistics(self) -> None:
        doc1 = Document.objects.create(
            title="none1",
@@ -1034,29 +1034,27 @@ class DocumentViewSet(
            ],
        }
-    def get_queryset(self):
+    def _base_document_queryset(self):
        # Root documents only, with the annotations that filter_backends rely
        # on (effective_content for SearchFilter, num_notes for ordering) --
        # but no select_related/prefetch_related, since those only matter for
        # serializing documents, not for filtering, ordering, or aggregating.
        latest_version_content = Subquery(
            Document.objects.filter(root_document=OuterRef("pk"))
            .order_by("-id")
            .values("content")[:1],
        )
        # A correlated subquery avoids the LEFT JOIN + Count() this used to
        # be, which forced a GROUP BY aggregate over every matching document
        # before the query could even be sorted or limited.
        note_count = Subquery(
            Note.objects.filter(document=OuterRef("pk"))
            .order_by()
            .values("document")
            .annotate(count=Count("pk"))
            .values("count"),
            output_field=IntegerField(),
        )
        return (
            Document.objects.filter(root_document__isnull=True)
            .distinct()
            .order_by("-created", "-id")
            .annotate(effective_content=Coalesce(latest_version_content, F("content")))
-            .annotate(num_notes=Coalesce(note_count, 0))
+            .annotate(num_notes=Count("notes"))
        )
    def get_queryset(self):
        return (
            self._base_document_queryset()
            .select_related("correspondent", "storage_path", "document_type", "owner")
            .prefetch_related(
                Prefetch(
@@ -1195,24 +1193,21 @@ class DocumentViewSet(
        return response
-    def list(self, request, *args, **kwargs):
+    @extend_schema(
-        if not get_boolean(
+        operation_id="documents_filter_selection_data",
-            str(request.query_params.get("include_selection_data", "false")),
+        description=(
-        ):
+            "Returns per-tag/correspondent/document-type/storage-path/custom-field "
-            return super().list(request, *args, **kwargs)
+            "document counts for the current filter, without paginating or "
-
+            "serializing the matching documents themselves. Split out from the "
-        queryset = self.filter_queryset(self.get_queryset())
+            "plain document list so that browsing the (potentially huge) unfiltered "
-        selection_data = self._get_selection_data_for_queryset(queryset)
+            "document list doesn't pay for this aggregation on every request."
-
+        ),
-        page = self.paginate_queryset(queryset)
+        responses={200: inline_serializer(name="SelectionData", fields={})},
-        if page is not None:
+    )
-            serializer = self.get_serializer(page, many=True)
+    @action(detail=False, methods=["get"], url_path="filter_selection_data")
-            response = self.get_paginated_response(serializer.data)
+    def filter_selection_data(self, request, *args, **kwargs):
-            response.data["selection_data"] = selection_data
+        queryset = self.filter_queryset(self._base_document_queryset())
-            return response
+        return Response(self._get_selection_data_for_queryset(queryset))
        serializer = self.get_serializer(queryset, many=True)
        return Response({"results": serializer.data, "selection_data": selection_data})
    def destroy(self, request, *args, **kwargs):
        from documents.search import get_backend
@@ -73,8 +73,6 @@ APPLE_MAIL_TAG_COLORS = {
    "grey": ["$MailFlagBit1", "$MailFlagBit2"],
 }
 MAIL_FETCH_BATCH_SIZE = 500
 class MailError(Exception):
    pass
@@ -682,41 +680,12 @@ class MailAccountHandler(LoggingMixin):
            f"Rule {rule}: Searching folder with criteria {criterias}",
        )
        try:
            all_uids = set(
                M.uids(criteria=criterias, charset=rule.account.character_set),
            )
        except Exception as err:
            raise MailError(
                f"Rule {rule}: Error while searching folder {rule.folder}",
            ) from err
        processed_uids_qs = ProcessedMail.objects.filter(
            rule=rule,
            folder=rule.folder,
            uid__in=all_uids,
        )
        if self._current_uid_validity is not None:
            processed_uids_qs = processed_uids_qs.filter(
                Q(uid_validity=self._current_uid_validity)
                | Q(uid_validity__isnull=True),
            )
        processed_uids = set(processed_uids_qs.values_list("uid", flat=True))
        new_uids = all_uids - processed_uids
        if not new_uids:
            self.log.debug(
                f"Rule {rule}: No new mail matching criteria {criterias}",
            )
            return 0
        sorted_new_uids = sorted(new_uids, key=int)
        try:
            messages = M.fetch(
-                uid_list=sorted_new_uids,
+                criteria=criterias,
                mark_seen=False,
-                bulk=MAIL_FETCH_BATCH_SIZE,
+                charset=rule.account.character_set,
                bulk=True,
            )
        except Exception as err:
            raise MailError(
@@ -788,7 +757,6 @@ class MailAccountHandler(LoggingMixin):
            not message.attachments
            and rule.consumption_scope == MailRule.ConsumptionScope.ATTACHMENTS_ONLY
        ):
            self._record_processed_without_consumption(message, rule)
            return processed_elements
        self.log.debug(
@@ -824,25 +792,6 @@ class MailAccountHandler(LoggingMixin):
        return processed_elements
    def _record_processed_without_consumption(
        self,
        message: MailMessage,
        rule: MailRule,
    ) -> None:
        ProcessedMail.objects.get_or_create(
            rule=rule,
            uid=message.uid,
            folder=rule.folder,
            uid_validity=self._current_uid_validity,
            defaults={
                "subject": message.subject,
                "received": make_aware(message.date)
                if is_naive(message.date)
                else message.date,
                "status": "PROCESSED_WO_CONSUMPTION",
            },
        )
    def filename_inclusion_matches(
        self,
        filter_attachment_filename_include: str | None,
@@ -1009,7 +958,23 @@ class MailAccountHandler(LoggingMixin):
            )
        else:
            # No files to consume, just mark as processed if it wasn't by .eml processing
-            self._record_processed_without_consumption(message, rule)
+            if not ProcessedMail.objects.filter(
                rule=rule,
                uid=message.uid,
                folder=rule.folder,
                uid_validity=self._current_uid_validity,
            ).exists():
                ProcessedMail.objects.create(
                    rule=rule,
                    folder=rule.folder,
                    uid=message.uid,
                    uid_validity=self._current_uid_validity,
                    subject=message.subject,
                    received=make_aware(message.date)
                    if is_naive(message.date)
                    else message.date,
                    status="PROCESSED_WO_CONSUMPTION",
                )
        return processed_attachments
@@ -134,23 +134,7 @@ class BogusMailBox(AbstractContextManager):
        if username != self.USERNAME or access_token != self.ACCESS_TOKEN:
            raise MailboxLoginError("BAD", "OK")
-    def fetch(
+    def fetch(self, criteria, mark_seen, charset="", *, bulk=True):
        self,
        criteria="ALL",
        charset="",
        *,
        mark_seen=True,
        bulk=True,
        uid_list=None,
    ):
        if uid_list is not None:
            return [m for m in self.messages if m.uid in uid_list]
        return self._filter_messages(criteria)
    def uids(self, criteria, charset="") -> list[str]:
        return [m.uid for m in self._filter_messages(criteria)]
    def _filter_messages(self, criteria):
        msg = self.messages
        criteria = str(criteria).strip("()").split(" ")
@@ -184,10 +168,6 @@ class BogusMailBox(AbstractContextManager):
        if "(X-GM-LABELS" in criteria:  # ['NOT', '(X-GM-LABELS', '"processed"']
            msg = filter(lambda m: "processed" not in m.flags, msg)
        if "UID" in criteria:
            uid_list = criteria[criteria.index("UID") + 1].split(",")
            msg = filter(lambda m: m.uid in uid_list, msg)
        return list(msg)
    def delete(self, uid_list) -> None:
@@ -426,7 +406,7 @@ def assert_eventually_equals(
    deadline = time.time() + timeout
    while time.time() < deadline:
        if getter_fn() == expected_value:
-            return
+            return None
        time.sleep(interval)
    actual = getter_fn()
    raise AssertionError(f"Expected {expected_value}, but got {actual}")
@@ -445,58 +425,6 @@ class TestMail(
        super().setUp()
    @mock.patch("paperless_mail.mail.MAIL_FETCH_BATCH_SIZE", 5)
    def test_handle_mail_account_batches_body_fetch_for_large_backlog(self) -> None:
        """
        GIVEN:
            - More new/unprocessed mail than MAIL_FETCH_BATCH_SIZE
        WHEN:
            - The mail account is processed
        THEN:
            - The body fetch is issued once, with all UIDs and the configured batch size
              handed to imap_tools so it can bulk-fetch in batches server-side
            - Every message is still processed (none dropped at a batch boundary)
        """
        account = MailAccount.objects.create(
            name="test",
            imap_server="",
            username="admin",
            password="secret",
        )
        rule = MailRule.objects.create(
            name="testrule",
            account=account,
            action=MailRule.MailAction.MARK_READ,
            consumption_scope=MailRule.ConsumptionScope.ATTACHMENTS_ONLY,
        )
        message_count = 12  # more than the patched batch size of 5
        self.mailMocker.bogus_mailbox.messages = [
            self.mailMocker.messageBuilder.create_message(
                subject=f"No attachment {i}",
                attachments=[],
            )
            for i in range(message_count)
        ]
        self.mailMocker.bogus_mailbox.updateClient()
        with mock.patch.object(
            self.mailMocker.bogus_mailbox,
            "fetch",
            wraps=self.mailMocker.bogus_mailbox.fetch,
        ) as fetch_spy:
            self.mail_account_handler.handle_mail_account(account)
        # A single fetch() call hands the full UID list and batch size to imap_tools,
        # which does its own bulk-fetching in batches of MAIL_FETCH_BATCH_SIZE.
        fetch_spy.assert_called_once()
        self.assertEqual(fetch_spy.call_args.kwargs["bulk"], 5)
        self.assertEqual(len(fetch_spy.call_args.kwargs["uid_list"]), message_count)
        self.assertEqual(
            ProcessedMail.objects.filter(rule=rule).count(),
            message_count,
        )
    def test_get_correspondent(self) -> None:
        message = namedtuple("MailMessage", [])
        message.from_ = "someone@somewhere.com"
@@ -609,59 +537,17 @@ class TestMail(
            ],
        )
    def test_bogus_mailbox_uids_and_uid_criteria(self) -> None:
        mailbox = self.mailMocker.bogus_mailbox
        all_messages = list(mailbox.messages)
        # uids() returns the UIDs of unseen messages, no bodies needed to call it
        unseen_uids = mailbox.uids("(UNSEEN)")
        self.assertEqual(
            set(unseen_uids),
            {m.uid for m in all_messages if not m.seen},
        )
        # fetch() with an explicit UID criteria returns only the matching messages
        target_uid = all_messages[0].uid
        from imap_tools import AND
        fetched = mailbox.fetch(AND(uid=[target_uid]), mark_seen=False)
        self.assertEqual([m.uid for m in fetched], [target_uid])
    def test_handle_empty_message(self) -> None:
-        message = self.mailMocker.messageBuilder.create_message(
+        message = namedtuple("MailMessage", [])
            subject="No attachments here",
            attachments=[],
        )
-        account = MailAccount.objects.create()
+        message.attachments = []
-        rule = MailRule.objects.create(
+        rule = MailRule()
            account=account,
            consumption_scope=MailRule.ConsumptionScope.ATTACHMENTS_ONLY,
        )
        result = self.mail_account_handler._handle_message(message, rule)
        self.mailMocker._queue_consumption_tasks_mock.assert_not_called()
        self.assertEqual(result, 0)
        processed = ProcessedMail.objects.get(
            rule=rule,
            uid=message.uid,
            folder=rule.folder,
        )
        self.assertEqual(processed.status, "PROCESSED_WO_CONSUMPTION")
        # Calling it again must not create a second row
        self.mail_account_handler._handle_message(message, rule)
        self.assertEqual(
            ProcessedMail.objects.filter(
                rule=rule,
                uid=message.uid,
                folder=rule.folder,
            ).count(),
            1,
        )
    def test_handle_unknown_mime_type(self) -> None:
        message = self.mailMocker.messageBuilder.create_message(
            attachments=[
@@ -1026,62 +912,6 @@ class TestMail(
        ]
        self.assertEqual(queued_rule.id, first_rule.id)
    def test_handle_mail_account_skips_body_fetch_for_already_processed_mail(
        self,
    ) -> None:
        """
        GIVEN:
            - An attachment-less mail under an attachments-only mark-read rule,
              already recorded as PROCESSED_WO_CONSUMPTION
        WHEN:
            - The mail account is processed again and the mail still matches the
              search criteria (it was never marked read, since no mail action is
              applied for the no-consumption case)
        THEN:
            - No IMAP body fetch happens for that mail; only the cheap UID search runs.
        """
        account = MailAccount.objects.create(
            name="test",
            imap_server="",
            username="admin",
            password="secret",
        )
        rule = MailRule.objects.create(
            name="testrule",
            account=account,
            action=MailRule.MailAction.MARK_READ,
            consumption_scope=MailRule.ConsumptionScope.ATTACHMENTS_ONLY,
        )
        message = self.mailMocker.messageBuilder.create_message(
            subject="No attachment",
            attachments=[],
        )
        self.mailMocker.bogus_mailbox.messages = [message]
        self.mailMocker.bogus_mailbox.updateClient()
        # First run: records ProcessedMail without consuming anything.
        self.mail_account_handler.handle_mail_account(account)
        self.assertTrue(
            ProcessedMail.objects.filter(
                rule=rule,
                uid=message.uid,
                folder=rule.folder,
            ).exists(),
        )
        self.mailMocker._queue_consumption_tasks_mock.assert_not_called()
        # Second run: message still matches UNSEEN (mark-read action never ran),
        # but its body must not be downloaded again.
        with mock.patch.object(
            self.mailMocker.bogus_mailbox,
            "fetch",
            wraps=self.mailMocker.bogus_mailbox.fetch,
        ) as fetch_spy:
            self.mail_account_handler.handle_mail_account(account)
        fetch_spy.assert_not_called()
    def test_handle_mail_account_skip_duplicate_uids_from_fetch(self) -> None:
        """
        GIVEN:
@@ -1687,7 +1517,7 @@ class TestMail(
            if message.from_ == "amazon@amazon.de":
                raise ValueError("Does not compute.")
            else:
-                return
+                return None
        m.side_effect = get_correspondent_fake
Author	SHA1	Message	Date
Trenton HolmesandClaude Sonnet 5	ff0ce4d123	Fix: register the filter_selection_data e2e stub after routeFromHAR, not before Per Playwright's own docs, routeFromHAR's notFound: 'fallback' sends unmatched requests straight to the network -- it does not chain to other, earlier-registered page.route() handlers via route.fallback() the way I'd assumed. Since Playwright checks routes in reverse-registration order, a handler registered in beforeEach (i.e. before the test body's routeFromHAR call) is checked AFTER routeFromHAR, whose catch-all pattern intercepts everything first and sends any miss straight to the (nonexistent, in e2e) network -- my stub never got a chance to run. Confirmed by a second CI failure with the same "Failed to fetch" symptom even after the CORS-header fix. Moved the mockFilterSelectionData(page) call to immediately after each test's own routeFromHAR(...) call instead, so it's registered later and checked first for that specific URL, with routeFromHAR's broader pattern still handling everything else as before. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-20 14:54:58 -07:00
Trenton HolmesandClaude Sonnet 5	8b3665b32d	Fix: add CORS header to the e2e filter_selection_data stub response The e2e app (localhost:4200) calls the backend at localhost:8000, a cross-origin request from the browser's perspective. Recorded HAR responses carry the real backend's Access-Control-Allow-Origin header, which is why they work; the stub added in the previous commit didn't set one, so the browser rejected the fulfilled response as a CORS violation -- same symptom as the original bug (TypeError: Failed to fetch), confirmed via a fresh CI run after the first e2e fix attempt. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-20 14:32:59 -07:00
Trenton HolmesandClaude Sonnet 5	9cce6d1b68	Fix: mock the new filter_selection_data request in Playwright e2e tests CI (https://github.com/paperless-ngx/paperless-ngx/actions/runs/29777331059) failed every e2e test that reloads the document list, with the browser throwing "TypeError: Failed to fetch". Root cause: the e2e harness only starts the Angular dev server (no real backend), and relies entirely on page.routeFromHAR(..., { notFound: 'fallback' }) to mock API responses, falling through to the real network for anything not in a recorded HAR. The new GET /api/documents/filter_selection_data/ request added by this branch fires on every non-search reload() and isn't in any pre-recorded HAR fixture, so it fell through to a real network call with nothing listening on the other end. Added a shared mockFilterSelectionData() helper that stubs an empty response for that endpoint, registered via test.beforeEach() before each affected test's own routeFromHAR() call in every spec file that exercises document-list reload (document-list, dashboard, global-permissions, settings, document-detail). Playwright resolves routes in reverse-registration order, so the later-registered HAR route checks first, calls route.fallback() on a miss, and defers back to this earlier-registered stub instead of hitting the network. Not verified locally (Playwright browser binaries aren't installed in this environment and the user preferred not to install them); relying on CI to confirm. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-20 14:21:51 -07:00
Trenton HolmesandClaude Sonnet 5	bb77e65d52	Simplify: dedup HttpParams building, fire selection-data concurrently Cleanups from a code-simplification pass over the prior commit: - Extract the repeated "build HttpParams from a plain object, skipping null/undefined" loop (previously duplicated in list(), getFew(), and the new getFilterSelectionData()) into a single shared AbstractPaperlessService.withParams() helper. - document-list-view.service.ts: fire the filter_selection_data request concurrently with the primary list request instead of nesting it inside the list request's success callback. They're independent (the follow-up only needs filterRules, not the list response), so there's no reason to wait on one before starting the other. Extracted into loadFilterSelectionData() for readability, and guarded the error path so a failed list request cancels the concurrent follow-up too rather than letting it resolve afterward and clobber the error-state reset. - views.py: give filter_selection_data a lean base queryset (_base_document_queryset) instead of the full get_queryset(), which carries select_related/prefetch_related meant for serializing full Document objects. Benchmarked this specifically: no measured query-time impact in this path, since those calls were never iterating the queryset directly and were already inert here -- keeping it for clarity, not as a performance claim. - Test fallout: since filter_selection_data now fires unconditionally on every non-search reload() rather than only after a successful primary flush, added a generic drain step to both spec files' afterEach so individual tests don't each need to know about this follow-up request unless they specifically assert on its response. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-20 10:00:08 -07:00
Trenton HolmesandClaude Sonnet 5	eb5bf53476	Fix (beta): stop blocking the document list on selection-data aggregation The document overview sent include_selection_data=true on every list request (page/filter/sort change), computing 5 correlated Count(DISTINCT) aggregations over the full matching queryset inline before the list could render. At scale (hundreds of thousands of documents, unfiltered) this is the confirmed root cause of the document overview "eternally loading" in paperless-ngx/paperless-ngx#13161. Split the aggregation into its own endpoint, GET /api/documents/filter_selection_data/, filter-scoped the same way the list endpoint already resolves matches (no document ID enumeration needed). The frontend now fetches it as a separate, non-blocking request after the list has already rendered, for plain (non-search) browsing. Full-text search keeps computing it inline since results are already narrowed by the search backend first. include_selection_data never shipped in a stable release (introduced this beta cycle, not present on main), so the plain list endpoint simply stops acting on it rather than needing a deprecation path. Also closes a latent localStorage leak between document-list-view.service.spec.ts tests (filterRules persisted via localStorage were never cleared, only sessionStorage was) that this change's new URL-dependent follow-up request exposed, and fixes two tests that were unknowingly relying on that leaked state to pass. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-20 10:00:08 -07:00