Add spec and plan for AI taxonomy hints (discussion #12787 )

Supersedes the feature-ai-taxonomy-hints prototype and closed PR #13465. Keeps the prototype's core RAG-neighbour-derived candidate approach but fixes double retrieval, dropped assigned metadata, stale candidate names, unranked candidates, localization corrupting exact matches, unescaped untrusted prompt data, and a retrieval error boundary/caching gap, per design-review feedback. Reviewed against the current codebase by an independent agent across two rounds. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Fix: include sharelink bundle perms in WebUI (#13664 )
2026-08-13 06:13:20 +00:00 · 2026-08-12 19:42:48 -07:00 · 2026-08-12 14:19:21 -07:00 · 2026-08-12 19:04:59 +00:00 · 2026-08-12 19:03:24 +00:00 · 2026-08-12 08:23:05 -07:00
47 changed files with 5512 additions and 3373 deletions
@@ -1,62 +0,0 @@
---
-name: whoosh-compat-transition
-description: Use when integrating the whoosh-compat library into paperless-ngx search, replacing src/documents/search/_translate.py or _dates.py, building the search FieldRegistry, or changing user query parsing during the whoosh-to-tantivy transition
---
-
-# whoosh-compat transition
-
-## Overview
-
-whoosh-compat (github.com/stumpylog/whoosh-compat; local checkout usually at `../whoosh-compat`) replaces the hand-maintained translation layer (`src/documents/search/_translate.py`, `_dates.py`): it parses user queries with a faithful fork of whoosh's real grammar into a typed AST and emits programmatic tantivy queries. Read its README and ARCHITECTURE.md before wiring anything; its DIVERGENCES.md lists intended behavior differences and is the authority on "is this difference a bug".
-
-## Decisions already made (do not re-derive)
-
- **Queries are user-typed free text.** The advanced search box passes whatever the user types straight to the parser (that is how the issue #13568 queries exist). Do NOT try to infer the supported field surface from frontend code; the frontend only generates a few date filter strings, everything else is typed by users.
- **The field surface is a policy decision, not `KNOWN_FIELDS`.** Today's `KNOWN_FIELDS` accepts internal ID fields (`tag_id`, `owner_id`, `viewer_id`, other `*_id`) that are undocumented in `docs/usage.md` and were ruled not user-searchable by the maintainer: exclude them from the `FieldRegistry` (they stay as programmatic permission/filter fields in `build_permission_filter`, which never touches user query text). The registry is built from documented syntax in `docs/usage.md` plus the v2-compat aliases (`type`, `path`, `type_id`-style aliases follow their canonical field's fate). Undocumented-but-working fields (`asn`, `page_count`, `num_notes`, `original_filename`, `checksum`) need an explicit maintainer yes/no; since users type freely, silently dropping one breaks any saved view using it, so a drop must be a visible, documented decision.
- **Analyzer seam:** `FieldSpec.analyzer` binds the live registered tantivy analyzer's `.analyze` (the same Rust analyzer used at index time; language-keyed, so rebuild the registry when `SEARCH_LANGUAGE` changes, on the same trigger as `register_tokenizers`). `pattern_normalizer` is `_tokenizer.ascii_fold`: character-level lowercase+fold only, NEVER stemming.
- **Diagnostics before emit:** `whoosh_compat.parse()` never raises on bad input. Check `ParseResult.diagnostics` and map to `SearchQueryError`/`InvalidDateQuery` (HTTP 400) BEFORE calling `emit()`; also catch the emitter's `UnsupportedQueryError` into a 400. Never carry forward the legacy raw-string fallback (`except Exception: query_str = raw_query`) into the new path; it masks integration bugs.
- **Build typed errors from structured diagnostic data, never by parsing `message`.** Each `Diagnostic` carries `kind`, `startchar`/`endchar`, and `field`/`raw_value`. `message` is human-readable text whose wording can change. For a range that fails on one bound, `raw_value` is the bound that actually failed.
-  - `diagnostic.field` is a **`FieldRef`**, not a string: use `str(diagnostic.field)` for the canonical dotted name (`created`, `notes.user`) or `diagnostic.field.name` for the field alone. Note the name is canonical, so an aliased query (`type:`) reports the field it resolves to (`document_type`), and the diagnostic span covers the offending value rather than the field name, so the text the user typed for the field is not recoverable.
- **The registry has one resolver.** `registry.make_ref(raw)` turns a raw field string into a `FieldRef` or `None` for an unknown field, and `registry.resolve(ref)` returns the spec. There is no `resolve_json()`; a dotted name is interpreted only inside `make_ref`.
- **`notes` and `custom_fields` are JSON fields** with fixed subpaths (`notes.user`/`notes.note`, `custom_fields.name`/`custom_fields.value`); the registry stays a static, language-keyed singleton, never per-request.
-
-## Mandatory before deleting old code
-
- Date-grammar parity audit, line by line: every keyword, relative unit, and abbreviation `_dates.py` and `_translate.py` accept today (including the whoosh-era abbreviations kept for old saved views) must have an accepted form in whoosh-compat's dateparse grammar. Silent keyword loss is the saved-view breakage class behind issue #13568.
- Acceptance corpus compared by matched-document-ID sets, not query strings: the #13568 queries verbatim, real saved-view strings, every date keyword, field aliases, comma lists, date and numeric ranges, wildcards with bracket classes, boosts, JSON subpaths.
-
-## Tests: what goes, what comes
-
-Removed with their modules (do not port their string-level assertions):
-
- `src/documents/tests/search/test_translate.py`: its subject is deleted; string-translation unit cases are whoosh-compat's own responsibility now. Cases that encode real user-visible behavior get reincarnated as result-level acceptance cases, not string assertions.
- Date-keyword unit tests tied to `_dates.py` internals: same treatment.
- `test_query.py` cases asserting `parse_user_query` internals or intermediate query strings: rewritten against the new pipeline, asserting on matched results.
-
-Kept: `test_migration_fulltext_query_field_prefixes.py` (data migration, orthogonal), `test_schema.py`, `test_tokenizer.py`, permission-filter and simple-search tests.
-
-Added:
-
- A result-level acceptance module (paperless's analogue of whoosh-compat's `test_acceptance_e2e.py`): the corpus above against a real index built from `build_schema()`, asserting document-ID sets. Use `pytest.param(..., id="...")` for every case.
- Registry unit tests: internal `*_id` names rejected, aliases resolve to canonical fields, JSON subpaths match `docs/usage.md`, construction deterministic per language.
- One `Multitoken` case nested inside a top-level `OR` (whoosh-compat DIVERGENCES entry on Multitoken.DEFAULT) to prove it does not matter for paperless's data.
- If acceptance work surfaces a new whoosh-compat divergence, that is a whoosh-compat-repo change (its `differential-triage` skill applies), not a silent paperless workaround.
-
-## Do not mark the JSON fields fast
-
-`notes` and `custom_fields` must stay non-fast for now. Existence checks against a **fast** JSON field currently return inverted results (a document that has the field is reported as not having it), because the underlying call does not check subpath columns by default. The unsupported-configuration error raised for a non-fast JSON field advises marking it fast, which walks directly into that bug. Ignore that advice until the upstream issue is fixed, then re-check.
-
-## Coordination
-
- whoosh-compat is pre-1.0: pin an exact version or git SHA; upgrades are deliberate, reviewed changes.
- JSON subpath emission depends on the installed tantivy-py version (fallback until quickwit-oss/tantivy-py#716 ships). The whoosh-compat repo has a `carve-out-retirement` skill; coordinate tantivy pin bumps with it, in a separate PR from the parser migration.
- Rollout: settings flag defaulting to the legacy path plus shadow-compare logging (log when old and new paths return different ID sets; sample if cost matters) for one release; delete `_translate.py`/`_dates.py` only after the flag defaults to the new path with no material reports.
-
-## Common mistakes
-
- Inferring the field surface from frontend code (users type queries directly).
- Copying `KNOWN_FIELDS` into the registry wholesale (resurfaces internal fields).
- Wiring stemming into `pattern_normalizer`.
- Calling `emit()` unconditionally, or porting the legacy raw-string fallback.
- Deleting `_dates.py` without the parity audit.
- Porting `test_translate.py`'s string assertions instead of writing result-level tests.
@@ -699,6 +699,7 @@ document_fuzzy_match [--ratio] [--processes N]
 | --ratio     | No       | 85.0                | a number between 0 and 100, setting how similar a document must be for it to be reported. Higher numbers mean more similarity. |
 | --processes | No       | 1/4 of system cores | Number of processes to use for matching. Setting 1 disables multiple processes                                                 |
 | --delete    | No       | False               | If provided, one document of a matched pair above the ratio will be deleted.                                                   |
+| --url       | No       | blank               | If an instance URL is provided, the output table will show URLs to each documents instead of the document ID and name.         |

 !!! warning

@@ -948,10 +948,11 @@ for display in the web interface.

    !!! note

-        The **remote OCR parser** (Azure AI) always produces a searchable
-        PDF and stores it as the archive copy, regardless of this setting.
-        `ARCHIVE_FILE_GENERATION=never` has no effect when the remote
-        parser handles a document.
+        The **remote OCR parser** (Azure AI) also honors this setting: when
+        no archive is requested (`never`, or `auto` with a born-digital PDF),
+        the remote engine is skipped entirely and locally-extracted text is
+        used instead, avoiding an unnecessary API call and a duplicate text
+        layer.

 #### [`PAPERLESS_OCR_CLEAN=<mode>`](#PAPERLESS_OCR_CLEAN) {#PAPERLESS_OCR_CLEAN}

@@ -187,10 +187,11 @@ PAPERLESS_ARCHIVE_FILE_GENERATION=auto

 ### Remote OCR parser

-If you use the **remote OCR parser** (Azure AI), note that it always produces a
-searchable PDF and stores it as the archive copy. `ARCHIVE_FILE_GENERATION=never`
-has no effect for documents handled by the remote parser - the archive is produced
-unconditionally by the remote engine.
+If you use the **remote OCR parser** (Azure AI), `ARCHIVE_FILE_GENERATION` is
+honored the same way as for the local engine: when no archive is requested
+(`never`, or `auto` with a born-digital PDF), the remote engine is skipped
+entirely and locally-extracted text is used instead, avoiding an unnecessary
+API call and a duplicate text layer.

 ## Search Index (Whoosh -> Tantivy)

@@ -1,455 +0,0 @@
-# whoosh-compat transition design
-
-Date: 2026-08-07
-Status: approved, pending spec review
-Related skill: `whoosh-compat-transition`
-Related issue: [stumpylog/whoosh-compat#1](https://github.com/stumpylog/whoosh-compat/issues/1)
-
-> ## Read this before implementing: parts of this spec describe an API that has moved
->
-> whoosh-compat changed after this spec was written, and more changes are
-> already decided. Re-check anything below against the library's own source
-> and `ARCHITECTURE.md` before writing code from it. The library repo is the
-> source of truth; this document is not.
->
-> **Wrong today, fix on sight:**
->
-> | This spec says                                                   | Reality now                                                                                                                                                                                                    |
-> | ---------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-> | `FieldRegistry.resolve_json(dotted)` (§ JSON subpath resolution) | Gone. There is one resolver: `registry.make_ref(raw) -> FieldRef \| None` interprets a dotted name, and `registry.resolve(ref) -> FieldSpec \| None` looks it up. A dot is interpreted only inside `make_ref`. |
-> | `InvalidDateQuery(d.field, d.raw_value)`                         | `Diagnostic.field` is now a `FieldRef`, not a string. Use `str(d.field)` for the canonical dotted name, or `d.field.name`. The name is canonical, so an aliased query (`type:`) reports `document_type`.       |
-> | AST leaves carrying a field name string                          | Every field-carrying AST leaf now holds a `FieldRef`.                                                                                                                                                          |
->
-> **Decided upstream, not yet implemented. Write toward these, they will land before this migration executes:**
->
-> | Behavior                                                                                                                           | Issue                                                       |
-> | ---------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------- |
-> | An empty or unknown `default_fields` raises at `parse()`; a `field_boosts` key naming an alias resolves, one naming nothing raises | [#20](https://github.com/stumpylog/whoosh-compat/issues/20) |
-> | A naive `basedate` is rejected rather than read in the host machine's timezone                                                     | [#19](https://github.com/stumpylog/whoosh-compat/issues/19) |
-> | A wildcard on a numeric field produces a diagnostic instead of failing at search time, so the error mapping gains a case           | [#17](https://github.com/stumpylog/whoosh-compat/issues/17) |
-> | A bare JSON field name (`notes:foo`) demotes to a text search rather than raising at emit                                          | [#11](https://github.com/stumpylog/whoosh-compat/issues/11) |
-> | Registry construction rejects exists-target cycles, empty names, duplicate aliases within a spec, and dotted canonical names       | [#21](https://github.com/stumpylog/whoosh-compat/issues/21) |
->
-> **Still undecided, do not guess:** whether `emit()` keeps its `schema`
-> parameter ([#27](https://github.com/stumpylog/whoosh-compat/issues/27)).
-> This spec calls `emit(ast, index=index, schema=schema, registry=...)` in two
-> places. Check the issue before writing either call site.
->
-> **Trap:** do not add `fast=True` to the `notes` or `custom_fields` JSON
-> specs. Existence checks against a fast JSON field currently return inverted
-> results ([#7](https://github.com/stumpylog/whoosh-compat/issues/7)), and the
-> error raised for a non-fast JSON field advises marking it fast, which walks
-> straight into that bug. The field table below correctly leaves them non-fast.
->
-> [#28](https://github.com/stumpylog/whoosh-compat/issues/28) tracks all open
-> upstream work ordered by effort.
-
-## Summary
-
-Replace paperless-ngx's hand-maintained query-translation layer
-(`src/documents/search/_translate.py`, `src/documents/search/_dates.py`)
-with [whoosh-compat](https://github.com/stumpylog/whoosh-compat): a typed
-Whoosh-grammar parser that emits programmatically constructed
-`tantivy.Query` objects instead of building an intermediate Tantivy query
-_string_. The integration point is narrow: `parse_user_query()` in
-`src/documents/search/_query.py` is the only function whose implementation
-changes; `_backend.py`, `_tokenizer.py`, simple/title search, CJK handling,
-and permission filtering are all unaffected.
-
-Delivered as a stack of four paperless-ngx PRs plus one prerequisite change
-in whoosh-compat itself (same maintainer, no cross-repo coordination
-overhead), landed with no feature flag and no shadow-compare rollout period
-— safety comes from a result-level acceptance test corpus and a
-date-grammar parity audit instead.
-
-## Architecture
-
-```
-raw_query (user-typed)
-    │
-    ▼
-wc.parse(raw_query, registry=FIELD_REGISTRY, default_fields=DEFAULT_SEARCH_FIELDS,
-         field_boosts=_FIELD_BOOSTS, tz=tz)
-    │
-    ▼
-ParseResult(ast, diagnostics)
-    │
-    ├─ diagnostics non-empty? → map ALL diagnostics to SearchQueryError
-    │  subclass(es) → HTTP 400 (never just the first diagnostic)
-    │
-    ▼
-emit(ast, index=index, schema=schema, registry=FIELD_REGISTRY)
-    │  (raises UnsupportedQueryError → mapped to SearchQueryError → 400,
-    │   for constructs that parse but can't execute against tantivy)
-    ▼
-tantivy.Query
-    │
-    ▼
-existing clause assembly in parse_user_query(): Should(exact) + optional
-fuzzy re-parse of raw_query + optional CJK bigram query, unchanged from today
-    │
-    ▼
-_apply_permission_filter() in _backend.py wraps the result with
-build_permission_filter() — entirely independent of whoosh-compat, unchanged
-```
-
-Permission filtering is explicitly out of scope for this migration:
-`build_permission_filter()` builds its `tantivy.Query` directly against
-`owner_id`/`viewer_id`/`viewer_group_id`, never through the parser or
-registry, and those fields are exactly the internal `*_id` fields excluded
-from the `FieldRegistry` (see "Field surface" below). Nothing in this
-migration's diff touches it.
-
-## PR stack
-
-Each PR is independently buildable, reviewable, and CI-able; later PRs
-rebase on earlier ones. No PR depends on whoosh-compat behavior it hasn't
-already proven correct in isolation.
-
-1. **Refactor `_schema.py` to a shared field-definition table.** Pure
-   refactor — `build_schema()`'s output is byte-identical before and after.
-   `test_schema.py` (existing) proves it.
-2. **Pin whoosh-compat as a real dependency; build `FieldRegistry`.** New
-   `_registry.py` built from the same table PR 1 introduced. Registry unit
-   tests only — no wiring into search yet.
-3. **Date-grammar parity audit.** A transitional, executable differential
-   test using the still-present `_dates.py`/`_translate.py` as the oracle.
-   Any gap found is fixed in whoosh-compat directly before this PR closes.
-   A whoosh-compat PyPI release is expected around this point (see
-   "Dependency pinning").
-4. **Wire it in; delete the old path.** Rewrite `parse_user_query()`,
-   diagnostics→exception mapping, add the result-level acceptance corpus,
-   expand `test_api_search.py`, delete `_translate.py`/`_dates.py`/
-   `test_translate.py` and the internals-testing classes in `test_query.py`,
-   update `docs/usage.md` and changelog.
-
-**Prerequisite, whoosh-compat repo** (tracked as
-[stumpylog/whoosh-compat#1](https://github.com/stumpylog/whoosh-compat/issues/1),
-lands before PR 4 starts its diagnostics-mapping work): add `field: str |
-None` and `raw_value: str | None` to `Diagnostic`, threaded through at its
-three construction sites (`dateparse.py`'s `_error()`, `default.py`'s
-`term_query()` and `_coerce_range_bound()`), so paperless can build typed
-exceptions without parsing whoosh-compat's human-readable `message` text.
-
-## Field surface
-
-The `FieldRegistry` covers only query-syntax-addressable fields — a subset
-of the full Tantivy schema. Internal-only schema fields with no query-syntax
-meaning of their own (`title_sort`/`correspondent_sort`/`type_sort` shadow
-sort fields, `bigram_*` CJK fields, `simple_title`/`simple_content`,
-`autocomplete_word`, `notes_text`) stay hardcoded `sb.add_*` calls in
-`_schema.py`, untouched by the shared table.
-
-**Decision: keep and document all five currently-undocumented-but-working
-fields** (`asn`, `page_count`, `num_notes`, `original_filename`,
-`checksum`) rather than dropping them — least risk of silently breaking an
-existing saved view. `docs/usage.md`'s advanced-search section gets these
-added with examples, as part of PR 4.
-
-**Decision: `archive_checksum` stays out of scope.** Unlike `checksum`, it
-isn't indexed in the Tantivy schema at all today (confirmed: `_schema.py`
-only adds `checksum`; `_build_tantivy_doc` only calls
-`doc.add_text("checksum", document.checksum)`). Making it searchable is a
-schema-level change (new indexed field, new document population code), not
-a parser-migration concern — left as a separate follow-up.
-
-**Decision: internal `*_id` fields (`tag_id`, `correspondent_id`,
-`document_type_id`, `storage_path_id`, `owner_id`, `viewer_id`,
-`viewer_group_id`) are excluded from the `FieldRegistry` entirely.** They
-remain Tantivy-schema-only, used exclusively by `build_permission_filter()`.
-Because whoosh-compat folds any unrecognized `field:` prefix into literal
-text (Whoosh-parity leniency, confirmed in `FieldsPlugin.do_fieldnames` —
-not an error), a saved view typed as `tag_id:5` won't 400: it silently
-becomes a text search for the literal string `tag_id:5`, most likely
-returning zero results. This is a real behavior change and gets a
-**changelog callout**, not just a docs update, since a docs addition alone
-wouldn't surface it to someone skimming release notes.
-
-## Shared field-definition table (`_fields.py`)
-
-```python
-from whoosh_compat import FieldKind  # reused directly — no parallel enum
-
-@dataclass(frozen=True, slots=True)
-class PublicField:
-    name: str
-    kind: FieldKind
-    aliases: tuple[str, ...] = ()
-    comma_values: bool = False
-    date_only: bool = False
-    fast: bool = False
-    subpaths: tuple[str, ...] = ()   # JSON kind only
-
-PUBLIC_FIELDS = (
-    PublicField("title", FieldKind.TEXT),
-    PublicField("content", FieldKind.TEXT),
-    PublicField("correspondent", FieldKind.TEXT),
-    PublicField("document_type", FieldKind.TEXT, aliases=("type",)),
-    PublicField("storage_path", FieldKind.TEXT, aliases=("path",)),
-    PublicField("original_filename", FieldKind.TEXT),
-    PublicField("tag", FieldKind.TEXT, comma_values=True),
-    PublicField("checksum", FieldKind.KEYWORD),
-    PublicField("asn", FieldKind.U64, fast=True),
-    PublicField("page_count", FieldKind.U64, fast=True),
-    PublicField("num_notes", FieldKind.U64, fast=True),
-    PublicField("created", FieldKind.DATE, date_only=True, fast=True),
-    PublicField("modified", FieldKind.DATETIME, fast=True),
-    PublicField("added", FieldKind.DATETIME, fast=True),
-    PublicField("notes", FieldKind.JSON, subpaths=("user", "note")),
-    PublicField("custom_fields", FieldKind.JSON, subpaths=("name", "value")),
-)
-```
-
-`build_schema()` derives its `sb.add_*` call and tokenizer from `kind`
-(TEXT/KEYWORD → `add_text_field` with `paperless_text`/`raw` tokenizer
-respectively; U64 → `add_unsigned_field`; DATE/DATETIME → `add_date_field`;
-JSON → `add_json_field`). The `notes_text` snippet-companion field stays a
-separate hardcoded line right after the `notes` entry — schema-only
-plumbing with no query-syntax meaning.
-
-`_registry.py` maps each `PublicField` to a `whoosh_compat.FieldSpec`,
-kept as one flat dataclass (no kind-specific subclassing) to mirror
-whoosh-compat's own `FieldSpec` design, which validates kind-conditional
-attributes (e.g. JSON requires non-empty `subpaths`) at
-`FieldRegistry.__init__` rather than in the type system.
-
-Footnote for whoever writes `_registry.py`: `FieldRegistry.__init__` forces
-`date_only=True` on _any_ `FieldKind.DATE` spec regardless of what's
-passed, unconditionally — `PublicField.date_only` isn't an independent
-knob for DATE fields the way it might look; it only matters in the sense
-that `created` sets it explicitly for clarity, while `modified`/`added`
-use `FieldKind.DATETIME` instead of relying on that override.
-
-**`subpaths` stays `tuple[str, ...]`, not a nested structure.** Confirmed
-against whoosh-compat's own `FieldRegistry.resolve_json()`: it splits a
-dotted query term on the _first_ dot only and matches the remainder as an
-exact string against `spec.subpaths` — even the docstring's own
-`"metadata.author.name"` example is a single opaque string in the tuple,
-not a recursive tree. A tuple of strings is exactly as expressive as the
-library it feeds; inventing richer structure in `PublicField` now would
-just get flattened back to strings at the registry-construction boundary.
-Real recursive nesting, if ever needed, is new whoosh-compat capability
-first.
-
-**JSON document population stays separate from `subpaths`.** `subpaths` is
-query-side only — it declares which dotted names are legal to type and
-which JSON keys the emitter should address. It says nothing about how
-`_backend.py::_build_tantivy_doc` builds the JSON documents at index-write
-time, and that logic isn't uniform attribute access (`note.user.username`
-needs a null guard and isn't `note.user`; `cfi.value_for_search` is a
-property, not a literal `value` attribute), so a generic
-`getattr(obj, subpath_name)` scheme would silently do the wrong thing for
-both. That code stays hand-written, unchanged by this migration. Mitigation
-instead: a coupling test (PR 2, alongside the registry unit tests) asserting
-the literal JSON keys used in `_build_tantivy_doc`'s `doc.add_json(...)`
-calls match `PUBLIC_FIELDS`' `notes`/`custom_fields` `subpaths` exactly, so
-drift between the two is caught rather than silently becoming an
-unqueryable (or silently unindexed) field.
-
-**JSON subpath queries (`notes.*`, `custom_fields.*`) route through
-`index.parse_query()`, not programmatic construction, given paperless's
-pinned tantivy version.** Installed `tantivy-py`'s `Query.term_query`
-cannot resolve a JSON subpath by exact field name — it raises as if the
-field didn't exist. Until
-[tantivy-py#716](https://github.com/quickwit-oss/tantivy-py/pull/716) lands
-and ships, whoosh-compat's `TantivyEmitter._json_paths_supported()` feature-
-detects this per process and falls back to a strictly escaped, single-leaf
-`index.parse_query()` call for just that one leaf (whoosh-compat's README/
-ARCHITECTURE.md call this out as "the JSON subpath carve-out"). Paperless
-pins `tantivy~=0.26.0`, squarely inside the affected range (whoosh-compat's
-`tantivy` extra only requires `tantivy>=0.24`, so nothing prevents this
-combination). Nothing needs to change in this design because of it — the
-carve-out is self-retiring on whoosh-compat's side once tantivy-py catches
-up — but the acceptance corpus's `notes.user:`/`custom_fields.name:` cases
-(PR 4) are exercising that fallback escaping path specifically, not the
-programmatic path every other field goes through, and that's worth knowing
-if one of those cases ever behaves oddly around quoting/escaping.
-
-**Analyzer wiring**: `FieldSpec.analyzer` reuses the same `tantivy
-.TextAnalyzer` objects `_tokenizer.py` already builds (`_paperless_text
-(language)`, etc.) — standalone objects not dependent on index
-registration, so `_registry.py` calls the same builder functions and binds
-`.analyze` directly; `checksum` (KEYWORD, `raw` tokenizer) gets an identity
-analyzer (`lambda t: [t]`). `pattern_normalizer` for every field is
-`_tokenizer.ascii_fold` (character-fold only, never stemming) per the
-skill's explicit instruction. The whole `FieldRegistry` is built once,
-cached keyed by `settings.SEARCH_LANGUAGE`, rebuilt on the same trigger
-`register_tokenizers()` already uses.
-
-## Error handling
-
-```python
-class SearchQueryError(ValueError): ...          # unchanged, base
-
-class InvalidDateQuery(SearchQueryError):         # unchanged
-    def __init__(self, field, value): ...
-
-class InvalidNumberQuery(SearchQueryError):       # new
-    def __init__(self, field: str | None, value: str | None) -> None:
-        self.field = field
-        self.value = value
-        super().__init__(f"Invalid numeric value {value!r} for field {field!r}.")
-
-class MultipleSearchQueryErrors(SearchQueryError):  # new
-    """Aggregates every user-fixable error from one parse, not just the first."""
-    def __init__(self, errors: Sequence[SearchQueryError]) -> None:
-        self.errors = tuple(errors)
-        super().__init__("; ".join(str(e) for e in self.errors))
-```
-
-```python
-def parse_user_query(index, raw_query, tz):
-    registry = get_field_registry(settings.SEARCH_LANGUAGE)
-    result = wc.parse(
-        raw_query, registry=registry, default_fields=DEFAULT_SEARCH_FIELDS,
-        field_boosts=_FIELD_BOOSTS, tz=tz,
-    )
-    if result.diagnostics:
-        raise _diagnostics_to_error(result.diagnostics)   # ALL diagnostics, not [0]
-
-    try:
-        exact = tantivy_emit(result.ast, index=index, schema=index.schema, registry=registry)
-    except UnsupportedQueryError as e:
-        raise SearchQueryError(str(e)) from e
-
-    # CJK: unchanged — already re-parses raw_query directly via index.parse_query,
-    # never went through translate_query, so nothing here changes.
-    cjk_query = _build_cjk_query(index, raw_query, _CJK_ALL_FIELDS) if _has_cjk(raw_query) else None
-
-    clauses = [(tantivy.Occur.Should, exact)]
-    threshold = settings.ADVANCED_FUZZY_SEARCH_THRESHOLD
-    if threshold is not None:
-        # Fuzzy re-parses raw_query (not the AST) — no clean AST-level fuzzy
-        # equivalent exists; fuzzy matching was always an approximate,
-        # secondary clause, so this divergence from the exact-match path is
-        # acceptable.
-        fuzzy = index.parse_query(raw_query, DEFAULT_SEARCH_FIELDS, field_boosts=_FIELD_BOOSTS,
-                                   fuzzy_fields={f: (True, 1, True) for f in DEFAULT_SEARCH_FIELDS})
-        clauses.append((tantivy.Occur.Should, tantivy.Query.boost_query(fuzzy, 0.1)))
-    if cjk_query is not None:
-        clauses.append((tantivy.Occur.Should, cjk_query))
-
-    return exact if len(clauses) == 1 else tantivy.Query.boolean_query(clauses)
-
-
-def _diagnostics_to_error(diagnostics: tuple[Diagnostic, ...]) -> SearchQueryError:
-    errors = [_single_diagnostic_to_error(d) for d in diagnostics]
-    return errors[0] if len(errors) == 1 else MultipleSearchQueryErrors(errors)
-
-
-def _single_diagnostic_to_error(d: Diagnostic) -> SearchQueryError:
-    if d.kind is DiagnosticKind.BAD_DATE:
-        return InvalidDateQuery(d.field, d.raw_value)
-    if d.kind is DiagnosticKind.BAD_NUMBER:
-        return InvalidNumberQuery(d.field, d.raw_value)
-    return SearchQueryError(d.message)
-```
-
-No `except Exception: query_str = raw_query` fallback — per the skill, that
-legacy defensive branch is explicitly not carried forward. A bug in the new
-path must surface as a real error, not silently degrade to stale behavior.
-
-`views.py`'s existing `except SearchQueryError as e: raise
-ValidationError({"query": [str(e)]}) from e` handler gets one added branch
-to surface every aggregated message instead of just one:
-
-```python
-except SearchQueryError as e:
-    messages = [str(sub) for sub in e.errors] if isinstance(e, MultipleSearchQueryErrors) else [str(e)]
-    raise ValidationError({"query": messages}) from e
-```
-
-`d.field`/`d.raw_value` depend on the whoosh-compat prerequisite change
-(issue #1) landing first; until then (or if `field`/`raw_value` are `None`
-for a given diagnostic kind not yet covered), `_single_diagnostic_to_error`
-falls back to `SearchQueryError(d.message)`.
-
-Deferred, explicitly out of scope for this PR stack: any frontend use of
-`startchar`/`endchar` (already present on `Diagnostic` today) to highlight
-the offending span in the search box. Backend-only for now, per explicit
-decision.
-
-## Testing
-
-Existing test inventory (`src/documents/tests/search/` and
-`test_api_search.py`):
-
-| File                                                                                                                                                                                          | Fate                                                                                                                                                                       |
-| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `test_translate.py`                                                                                                                                                                           | Deleted (PR 4) — subject deleted                                                                                                                                           |
-| `test_query.py`: `TestCreatedDateField`, `TestDateTimeFields`, `TestWhooshQueryRewriting`, `TestYearRangeRewriting`, `TestNonDateFieldsNotRewritten`, `TestPassthrough`, `TestNormalizeQuery` | Deleted (PR 4) — test `_translate.py`/`_dates.py` internals or intermediate query strings                                                                                  |
-| `test_query.py`: `TestParseUserQuery`                                                                                                                                                         | Reviewed at plan time; result-level assertions folded into the new acceptance module, internals-only assertions dropped                                                    |
-| `test_query.py`: `TestParseSimpleTextHighlightQuery`, `TestPermissionFilter`                                                                                                                  | Unchanged — never touched `translate_query`                                                                                                                                |
-| `test_schema.py`, `test_tokenizer.py`, `test_backend.py`, `test_lock_backoff.py`, `test_migration_fulltext_query_field_prefixes.py`                                                           | Unchanged                                                                                                                                                                  |
-| `test_api_search.py` (`TestDocumentSearchApi`, 43 tests)                                                                                                                                      | **Stays green across every PR in the stack** (hard gate, not just PR 4) — full HTTP+DB+index integration coverage catches wiring mistakes none of the narrower tests would |
-
-New tests per PR:
-
- **PR 2**: `test_registry.py` — internal `*_id` names rejected; `type`/
-  `path` aliases resolve to canonical fields; JSON subpaths match
-  `docs/usage.md`; registry construction deterministic per language; the
-  `notes`/`custom_fields` dict-key coupling test described above.
- **PR 3**: `test_date_grammar_parity.py` — transitional, parametrized over
-  every keyword/unit `_dates.py`/`_translate.py` accept today
-  (`_DATE_KEYWORDS`, all of `_UNIT_ALIASES`'s Whoosh-era abbreviations —
-  `yrs`/`mos`/`wks`/`hrs`/`mins`/`secs` etc. — digit-precision forms, ISO
-  dash forms, `now-7d`/`now+1h`/`now-30m` compact offsets, open/reversed
-  ranges). Each case parses through
-  `wc.parse()` against a DATE-kind `FieldRegistry` and asserts no
-  diagnostics _and_ bounds matching what `_dates.py`/`_translate.py`
-  compute today, using the still-present legacy code as the oracle.
-  Deleted again in PR 4 along with that oracle, superseded by the
-  permanent acceptance corpus. This audit is scoped to _parity_ only —
-  whoosh-compat's date grammar is a strict superset of what `_dates.py`
-  accepts today (e.g. `tomorrow`, `now`, `midnight`, `noon`, weekday names
-  like `next monday`), so the migration also grants new date vocabulary for
-  free. That's a nice side effect, not something this PR needs to test or
-  document beyond noting it in the changelog alongside the other behavior
-  changes.
- **PR 4**:
-  - Result-level acceptance module (paperless's analogue of whoosh-compat's
-    `test_acceptance_e2e.py`): a real index built via `build_schema()`, the
-    issue #13568 queries verbatim, real saved-view strings, every date
-    keyword/unit, field aliases, comma lists, numeric/date ranges,
-    bracket-class wildcards, boosts, JSON subpaths — asserted by matched
-    document-ID set, `pytest.param(..., id=...)` per case. Plus a
-    multi-diagnostic case (two bad fields → `MultipleSearchQueryErrors`
-    with both messages present) and one `Multitoken` case nested inside a
-    top-level `OR` (proves DIVERGENCES entry 15 doesn't matter for
-    paperless's data, per the skill).
-  - `test_api_search.py` expanded: a multi-bad-field query (e.g.
-    `created:notadate AND asn:notanumber`) asserting the 400 response's
-    `query` list contains both messages; end-to-end searches on the five
-    newly-documented fields (`asn:`, `page_count:`, `num_notes:`,
-    `original_filename:`, `checksum:`) returning the right documents
-    through the real index.
-
-## Dependency pinning
-
-Stays `path = "../whoosh-compat"` in `[tool.uv.sources]` through the whole
-PR stack — both repos are being actively co-developed. The final swap
-happens at PR 4:
-
- **Primary plan**: whoosh-compat is released to PyPI around PR 3 (per
-  your stated intent), assuming the parity audit and issue #1 don't turn
-  up anything else needing a second round. PR 4 switches to a pinned PyPI
-  version (`whoosh-compat[tantivy]==X.Y.Z` in `dependencies`, the
-  `[tool.uv.sources]` override removed entirely).
- **Fallback**: if the PyPI release slips past PR 4's start, pin an exact
-  git commit SHA instead (`whoosh-compat[tantivy] @ git+https://github.com/
-stumpylog/whoosh-compat@<sha>`), per the skill's "pre-1.0: pin an exact
-  version or git SHA, upgrades are deliberate" guidance.
-
-The `TODO` comment already sitting in `pyproject.toml` (from the earlier
-smoke-test setup) gets updated to reflect this — "release, else pinned SHA"
-— rather than committing hard to one path before it's known which applies.
-
-## Explicitly out of scope
-
- `archive_checksum` indexing/search (separate schema-level follow-up).
- Frontend consumption of `Diagnostic.startchar`/`endchar` for in-box error
-  highlighting (backend-only for this PR stack).
- A feature flag or shadow-compare rollout period — explicitly decided
-  against; safety comes from the acceptance corpus and parity audit instead.
- Any change to `build_permission_filter()`/`_apply_permission_filter()` —
-  confirmed untouched by this migration.
@@ -0,0 +1,472 @@
+# AI Taxonomy Hints — Spec
+
+## Status
+
+Draft. Supersedes `feature-ai-taxonomy-hints` (prototype, not merged) and closed PR
+[#13465](https://github.com/paperless-ngx/paperless-ngx/pull/13465) (rejected as
+low-quality/AI slop). This spec incorporates a design review of the prototype
+branch and defines the version to actually implement and merge.
+
+## Source
+
+Discussion: [#12787 — AI Suggestions should prefer existing tags, document types,
+and storage paths](https://github.com/paperless-ngx/paperless-ngx/discussions/12787)
+
+> AI Suggestions appear to invent new metadata names (`blood test`, `blood work`)
+> instead of preferring existing ones (`Bloodwork`). Fuzzy string matching alone
+> cannot map semantic equivalents (`IRS` → `Taxes`, `State Farm` → `Insurance`).
+> Paperless should surface likely-relevant existing tags/types/paths/correspondents
+> to the LLM as candidates, and instruct it to prefer them verbatim.
+
+## Prior art
+
+### Closed PR #13465 — what went wrong
+
+Dumped the **entire system-wide** taxonomy (`Tag.objects.values_list("name", ...)`,
+unfiltered) into every classification prompt, plus the document's already-assigned
+metadata with instructions the model could "add, remove, or modify" it (the response
+schema has no way to represent a removal). No permission scoping — every user's
+prompt included every tag in the installation, including tags from documents they
+cannot see. No cap — cost and prompt size scale with total taxonomy size, not with
+the document being classified. Left a stray `print(prompt)` in production code.
+Rejected by maintainers as low-effort/unreviewed.
+
+### Prototype `feature-ai-taxonomy-hints` — what it got right
+
+Derives a small, locally-relevant taxonomy from the document's RAG neighbours
+instead of the global taxonomy: bounded prompt size, respects document visibility
+(via `get_objects_for_user_owner_aware`), scales with installation size instead of
+against it. Isolated the hint-building logic in `paperless_ai/taxonomy.py`.
+Protected a hinted-but-unmatched name from being fuzzily re-mapped onto an unrelated
+object in `matching.py`. Reasonably well tested for a prototype (full commit list at
+`746e21cbe977f5b86e27c1eb9741ad5c63b2be24`).
+
+### Prototype — issues this spec fixes
+
+1. **Double retrieval.** `get_taxonomy_hints_for_document()` and
+   `build_prompt_with_rag()` each independently call into the vector store —
+   two query embeddings, two vector searches, two slightly different neighbour
+   sets, doubled latency.
+2. **Assigned metadata is dropped.** The prototype only looks at neighbours; a
+   document's _own_ existing tags/type/correspondent/storage path (valuable,
+   authoritative context) are never surfaced to the model at all.
+3. **Candidate names can be stale.** Vector-store node metadata stores taxonomy
+   _names_ captured at index time. A rename or delete leaves stale names in the
+   index until every affected document is reindexed, and those stale names get
+   fed back into the prompt as "available" candidates.
+4. **Similarity evidence is discarded.** All neighbour metadata is reduced to
+   alphabetically sorted sets — a tag from four strong neighbours and a tag from
+   one weak neighbour are equally "available" to the model, and an installation
+   with wide-ranging documents could still produce a long, unranked hint list.
+5. **Localization can silently break exact reuse.** The prompt says "use existing
+   names verbatim," but the separate localization pass rewrites `tags`,
+   `document_types`, and `storage_paths` afterward with no knowledge of which
+   values were exact matches to existing objects. In non-default-language
+   installations, an exact match becomes a translated string and fails
+   deterministic matching in `matching.py` on the very next line.
+6. **Untrusted taxonomy names go into the prompt unescaped.** Tag/correspondent
+   names are user-controlled strings (a user can name a tag anything, including
+   newlines or instruction-shaped text) and are bullet-rendered directly into the
+   prompt, unlike document content which is already labelled untrusted.
+7. **Retrieval sits outside the classification error boundary**, and cached
+   suggestions are not invalidated by taxonomy or index changes (existing
+   behavior, not introduced by this feature, but worth an explicit decision
+   before this feature makes staleness more consequential).
+
+## Goals
+
+- Feed the LLM a small set of taxonomy **candidates** — drawn from RAG neighbours,
+  ranked by similarity evidence, permission-filtered, and always fresh — so it
+  prefers reusing existing tags/types/correspondents/storage paths over inventing
+  near-duplicates.
+- Also surface the document's own **already-assigned** metadata as separate,
+  clearly-labelled context (not a candidate list, not something the model is asked
+  to change).
+- Make "the model reused an existing object" a **structural fact** (an ID the
+  matching code resolves deterministically), not something inferred by re-running
+  fuzzy string matching after a localization pass has potentially mangled the name.
+- Keep prompt cost bounded and roughly constant regardless of installation size.
+- Preserve document-visibility permission scoping throughout (neighbour retrieval,
+  candidate resolution, and the final object lookups all respect what the
+  requesting user can see).
+
+## Non-goals
+
+- Building the offline evaluation harness (precision/recall corpus, variant
+  comparison) described in the design review. This is valuable but is a
+  separate, independent effort — track it as a follow-up issue, not a blocker
+  for this feature. See "Future work" below.
+- Changing the vector index's chunking, embedding model selection, or the
+  `document_llmindex` management command.
+- Redesigning the frontend AI-suggestions UI. The `ai_suggestions` API response
+  shape (`tags`/`suggested_tags`/etc. as already returned by `views.py`) is
+  unchanged by this spec.
+- Multi-document-type correspondent/type disambiguation beyond what the existing
+  schema already does (one list of candidate strings per category).
+
+## Design
+
+### Data flow
+
+```
+current document
+      |
+      +-- assigned metadata (this document's own tags/type/correspondent/path)
+      |
+      +-- one permission-filtered neighbour retrieval
+                       |
+                       +-- RAG text context   (existing behavior, unchanged output)
+                       +-- candidate taxonomy (new: ranked, ID-backed, fresh)
+                                      |
+                                      v
+                          structured classification call
+                          (schema returns existing_ids + new_names per category)
+                                      |
+                       +--------------+---------------+
+                       | existing_ids                 | new_names
+                       | resolved via permitted_object_ids  | localized, then
+                       | (no string matching needed)  | fuzzy-matched as today
+                       +-------------------------------------+
+```
+
+### 1. Consolidated retrieval
+
+Replace the prototype's two independent calls with one. Add a single retrieval
+entry point in `paperless_ai/indexing.py` that returns the raw retrieved nodes
+(with scores and metadata) plus the resolved `Document` objects, and have both
+the RAG-context builder and the taxonomy-candidate builder consume that one
+result.
+
+`query_similar_documents()` stays as the public helper other callers use (its
+existing return type — `list[Document]` — does not change), but its body is
+refactored to call the new shared retrieval function rather than duplicating
+retriever setup.
+
+New function:
+
+```python
+def retrieve_similar_nodes(
+    document: Document,
+    top_k: int = 5,
+    document_ids: Iterable[int | str] | None = None,
+) -> list["NodeWithScore"]:
+    """Run the vector-store retrieval once and return the raw scored nodes,
+    permission-filtered by document_ids and with the source document excluded.
+    Callers derive both RAG text context and taxonomy candidates from this."""
+```
+
+`query_similar_documents()` becomes:
+
+```python
+def query_similar_documents(
+    document: Document,
+    top_k: int = 5,
+    document_ids: Iterable[int | str] | None = None,
+) -> list[Document]:
+    nodes = retrieve_similar_nodes(document, top_k=top_k, document_ids=document_ids)
+    retrieved_document_ids = _node_document_ids(nodes)
+    return list(Document.objects.filter(pk__in=retrieved_document_ids))
+```
+
+`get_context_for_document()` in `ai_classifier.py` and the new taxonomy-candidate
+builder both call `retrieve_similar_nodes()` directly (once per classification
+request) instead of going through two independent higher-level helpers.
+
+`get_context_for_document`'s existing superuser fast-path (skip materializing
+`visible_document_ids` into a Python list when the user is `None` or a
+superuser — see the comment at `ai_classifier.py:99-108`, added for #12976) is
+preserved unchanged; the consolidation only removes the duplicate retrieval
+call, not that optimization.
+
+### 2. Assigned metadata vs. candidate taxonomy — two distinct concepts
+
+`paperless_ai/taxonomy.py` gets a second, independent function:
+
+```python
+class AssignedMetadata(TypedDict):
+    tags: list[str]
+    document_type: str | None
+    correspondent: str | None
+    storage_path: str | None
+
+
+def get_assigned_metadata(document: Document) -> AssignedMetadata:
+    """The document's own current taxonomy. Authoritative context, not a
+    candidate list -- the model is never asked to add, remove, or replace
+    these values, only to use them when helpful for the title and for
+    fields that are still empty."""
+```
+
+The prompt renders this as its own labelled block, separate from candidates, and
+the accompanying instruction text explicitly says these values are already set
+and should not be re-suggested — not "you may add, remove, or modify" (the
+mistake in #13465; the response schema has no removal representation, so telling
+the model it can remove things is actively misleading).
+
+### 3. Candidates carry IDs, not just names — solves staleness (issue 3) and
+
+sets up deterministic resolution (issue 5)
+
+Node metadata already stores `document_id` (see `build_document_node()`,
+`indexing.py:264-276`). The candidate builder uses that to re-derive taxonomy
+from the **current** ORM state of each neighbour document, not from the
+possibly-stale names cached in the node metadata at index time.
+
+```python
+class TaxonomyCandidate(TypedDict):
+    id: int
+    name: str
+    weight: float  # aggregate similarity evidence, see ranking below
+
+
+class TaxonomyCandidates(TypedDict):
+    tags: list[TaxonomyCandidate]
+    document_types: list[TaxonomyCandidate]
+    correspondents: list[TaxonomyCandidate]
+    storage_paths: list[TaxonomyCandidate]
+
+
+def build_taxonomy_candidates(
+    nodes: list["NodeWithScore"],
+    user: User | None,
+) -> TaxonomyCandidates:
+    """Resolve each neighbour node's document_id to a live Document, read its
+    *current* tags/type/correspondent/storage_path via the ORM, weight each
+    distinct taxonomy object by aggregate neighbour similarity, permission-filter
+    against what `user` can see, and return each category ranked by weight."""
+```
+
+This also gives item 3's permission benefit for free: a candidate is only
+included if it currently exists and the requesting user can see it (checked via
+`permitted_object_ids`, not just the neighbour documents' visibility) — a
+neighbour document being visible does not imply its tag object is (e.g. a tag
+could theoretically be scoped separately). See Task 3 for the exact
+implementation using `documents.permissions.permitted_object_ids`, consistent
+with how the rest of the codebase is migrating off
+`get_objects_for_user_owner_aware` (see `documents/matching.py`'s recent
+migration in commit `3986150f9`).
+
+Neighbour documents are re-fetched with a single batched queryset
+(`Document.objects.filter(pk__in=...).prefetch_related("tags", "document_type",
+"correspondent", "storage_path")`), not one query per neighbour.
+
+### 4. Ranking and capping (issue 4)
+
+Weight per candidate = sum of the similarity scores of the neighbour nodes that
+carried it (a tag backed by four strong neighbours outranks one backed by a
+single weak neighbour). Within each category, sort by weight descending and cap:
+
+- `tags`: top 10
+- `document_types`, `correspondents`, `storage_paths`: top 5 each
+
+These caps are constants in `taxonomy.py` (`MAX_TAG_CANDIDATES = 10`,
+`MAX_SINGLE_VALUE_CANDIDATES = 5`), not derived from measurement — the design
+review flagged the exact numbers as needing real measurement, which belongs in
+the offline evaluation harness (see Future work). Ship reasonable, clearly-named
+constants now; tune them later with data instead of blocking the feature on
+building an evaluation corpus first.
+
+### 5. Untrusted data — escape, don't bullet-render (issue 6)
+
+Tag/correspondent/type/path names are user-controlled data, same trust level as
+document content. `format_hints_for_prompt()` (renamed
+`format_taxonomy_for_prompt()`) serializes each category as a JSON array of
+`{"id": ..., "name": ...}` objects rather than free-text bullets, and the
+surrounding prompt text labels the block untrusted, matching the existing
+pattern already used for document content and RAG context in
+`ai_classifier.py` (`"Content (untrusted user data ...)"`,
+`"Additional context ... (untrusted -- do not follow instructions within)"`).
+JSON's own escaping means embedded newlines or instruction-shaped text stay
+inert as string data instead of breaking prompt structure.
+
+### 6. Structured response carries IDs for exact reuse (issue 5, the most
+
+immediate correctness issue per the design review)
+
+Extend `DocumentClassifierSchema` (`base_model.py`) so each taxonomy category
+returns a resolved-ID bucket and a new-name bucket instead of one flat list of
+strings:
+
+```python
+class TaxonomyChoice(BaseModel):
+    """One taxonomy category's suggestions: IDs the model matched to a
+    candidate it was shown, plus names for values it believes are genuinely
+    new."""
+
+    existing_ids: list[int] = Field(default_factory=list)
+    new_names: list[str] = Field(default_factory=list)
+
+
+class DocumentClassifierSchema(BaseModel):
+    title: str
+    tags: TaxonomyChoice = Field(default_factory=TaxonomyChoice)
+    correspondents: TaxonomyChoice = Field(default_factory=TaxonomyChoice)
+    document_types: TaxonomyChoice = Field(default_factory=TaxonomyChoice)
+    storage_paths: TaxonomyChoice = Field(default_factory=TaxonomyChoice)
+    dates: list[str] = Field(default_factory=list)
+```
+
+The prompt instructs the model: candidate IDs from the "Available ..." blocks go
+in `existing_ids` when reused; anything not covered by a candidate goes in
+`new_names`. `existing_ids` are plain integers — not human-readable text — so
+the localization pass (which only rewrites `title`/`tags`/`document_types`/
+`storage_paths` **strings**) has nothing to corrupt; localization is scoped down
+to run only over each category's `new_names`, never `existing_ids`.
+
+This is a real backend contract change (not just a prompt tweak), touching both
+LLM code paths in `client.py` (Ollama's `format=json_schema` and the
+OpenAI-like tool-calling path both already serialize whatever pydantic model is
+handed to them, so nesting `TaxonomyChoice` works with both, unchanged
+call shape).
+
+Typing carries past the LLM boundary, not just at it: `TaxonomyChoice`/
+`DocumentClassifierSchema` (pydantic) are the runtime-validating layer for
+whatever the model actually returns. Everywhere downstream of
+`AIClient.run_llm_query()` — which already returns a validated
+`.model_dump()`, i.e. a plain dict — the pipeline (`parse_ai_response`,
+`build_localization_prompt`, `get_ai_document_classification`, the
+`ai_suggestions` view) is typed against `TaxonomyChoiceDict`/
+`ClassificationSuggestions`, `TypedDict`s mirroring those two models' dumped
+shape, instead of bare `dict`. Every other data structure this feature
+introduces is likewise a named type, not a `dict`: `AssignedMetadata` and
+`TaxonomyCandidate`/`TaxonomyCandidates` are `TypedDict`s (section 2-4); no
+function in this design takes or returns an untyped `dict` as its "real" data
+shape.
+
+No dynamic per-request schema (e.g. constraining `existing_ids` to
+an enum of the exact candidate IDs shown) — the model can still emit an ID that
+isn't a valid candidate (hallucination) or that has gone stale between prompt
+construction and response; those are handled the same way as any other
+resolution failure: filtered out server-side (Task 6) rather than trusted.
+
+Backward compatibility: this changes the shape `get_ai_document_classification()`
+returns internally. `parse_ai_response()` and the `views.py` call site are
+updated in the same change (Task 7) — there is no external API caller of the
+Python-level dict shape to preserve; the public HTTP response shape from
+`ai_suggestions` (`tags`/`suggested_tags`/etc., all still flat ID/name lists) is
+unchanged.
+
+### 7. ID resolution replaces string matching for the `existing_ids` bucket
+
+`matching.py` gains ID-based resolution functions used alongside (not instead
+of) the existing name-based fuzzy matching, since `new_names` still needs it:
+
+```python
+def resolve_tag_ids(ids: list[int], user: User) -> list[Tag]:
+    """Resolve model-returned tag IDs against what the user may currently
+    see. Invalid, deleted, or now-invisible IDs are silently dropped (the
+    model's belief that an ID exists and is visible may be stale by the
+    time the response comes back)."""
+```
+
+One such function per category (`resolve_tag_ids`, `resolve_correspondent_ids`,
+`resolve_document_type_ids`, `resolve_storage_path_ids`), each built on
+`documents.permissions.permitted_object_ids` (see Task 3's precedent) rather
+than `get_objects_for_user_owner_aware`, migrating these lookups onto the
+project's current permission-filtering path in the same change
+(`permissions.py:167`, already used by the recently-migrated
+`documents/matching.py`).
+
+`match_tags_by_name()` and friends keep their existing signature and behavior
+for the `new_names` bucket — they still fuzzy-match. They also keep the
+prototype's `hinted_names` guard (refusing to fuzzy-map a name onto an object
+that was itself shown as a candidate) as an optional parameter, but this spec
+does **not** wire it at the `views.py` call site: the guard's original
+purpose was to respect an implicit "I saw this name and chose not to use it"
+signal, which was meaningful when candidates were plain text bullets with no
+other way to reference them. In this design the model has a direct,
+unambiguous way to reuse a candidate (`existing_ids`), so a value landing in
+`new_names` is a much weaker rejection signal — plausibly just a near-duplicate
+the model failed to map rather than a deliberate choice — and applying the
+guard there risks suppressing genuine fuzzy matches. Reusing it would also
+require threading the current request's candidate names from
+`get_ai_document_classification` through to the view, which duplicates data
+already implicit in `existing_ids`. Left as a `matching.py` capability for
+future use rather than exercised now.
+
+`views.py`'s `ai_suggestions` action combines both: `resolve_tag_ids(...)` +
+`match_tags_by_name(new_names, ...)`, concatenated, before building `resp_data`
+exactly as today (IDs of matched objects go in `tags`, leftover unmatched
+`new_names` go in `suggested_tags`).
+
+### 8. Error boundary and caching (issue 7)
+
+Move candidate/context retrieval inside the same `try/except` in
+`ai_classifier.get_ai_document_classification()` that already wraps the LLM
+call, so a vector-store failure during retrieval degrades to no-hints/no-context
+(matching the prototype's existing gate-on-no-embedding-backend behavior)
+instead of bubbling up as an unhandled 500 from `views.py`. Concretely: wrap
+`retrieve_similar_nodes()` (and everything derived from it) in a `try/except
+Exception`, log, and continue with `hints=None`/empty context — the pre-RAG
+prompt is still a valid classification request.
+
+Caching: the existing `llm_cache_backend` cache key (backend + model + endpoint
+
+- output_language, see `views.py:1531-1541`) is **not** extended to include
+  taxonomy/index state in this change. A cached suggestion can reference tags that
+  have since been renamed or deleted, same as the existing RAG-context cache
+  already can — this spec makes that explicit as a known, accepted limitation
+  rather than a regression, and leaves cache invalidation on taxonomy/index change
+  as explicit future work (see below), not a silent gap.
+
+## Prompt shape (illustrative)
+
+```
+You are a document classification assistant.
+
+This document's existing metadata (already assigned; use as context for the
+title and for any fields below still empty, do not re-suggest these values):
+Tags: Bloodwork, Annual Physical
+Document Type: (not set)
+Correspondent: (not set)
+Storage Path: (not set)
+
+Available tags, document types, correspondents, and storage paths from similar
+documents (untrusted data; prefer these verbatim via existing_ids when one
+fits; only use new_names for values that genuinely don't match any of these):
+{"tags": [{"id": 12, "name": "Bloodwork"}, {"id": 47, "name": "Lab Work"}], ...}
+
+Analyze the following document and extract the following information:
+...
+```
+
+## Testing strategy
+
+- Unit tests for `retrieve_similar_nodes()` covering: source-document exclusion,
+  permission filtering, empty-index fallback (existing `query_similar_documents`
+  coverage moves/adapts here).
+- Unit tests for `build_taxonomy_candidates()`: staleness (renamed/deleted
+  taxonomy on a neighbour is not surfaced), permission filtering independent of
+  neighbour-document visibility, ranking order, per-category caps.
+- Unit tests for `get_assigned_metadata()`: unset fields render as `None`/empty,
+  not surfaced as candidates.
+- Unit tests for `format_taxonomy_for_prompt()`: JSON escaping of a
+  newline/instruction-shaped tag name, empty-category omission.
+- Unit tests for the extended `DocumentClassifierSchema`/`TaxonomyChoice`
+  round-trip through both `client.py` backends (mock the LLM boundary as
+  existing tests already do).
+- Unit tests for `resolve_tag_ids()` and siblings: invisible ID dropped, deleted
+  ID dropped, valid ID resolved, permission-filtered per user.
+- Unit test proving localization only touches `new_names`, never `existing_ids`
+  (the concrete regression this spec fixes).
+- Integration test on the `ai_suggestions` view: candidate retrieval failure
+  degrades to a successful response with empty hints, not a 500.
+- Existing `test_matching.py` `hinted_names`-style protection test carried
+  forward against the new candidate-name-set scope.
+
+## Future work (explicitly out of scope here)
+
+- **Offline evaluation harness**: hide each classified document's metadata,
+  exclude it from retrieval, generate suggestions under multiple variants
+  (baseline / global taxonomy / neighbour taxonomy / ranked neighbour taxonomy
+  / ranked neighbours + assigned metadata), and measure tag precision/recall,
+  top-1 accuracy for correspondent/type/storage path, `existing_ids` resolution
+  rate, duplicate-new-taxonomy rate, prompt tokens, and latency — split by
+  collection size and output language. This is how the ranking caps in
+  section 4 should eventually be tuned. Track as a separate issue; this spec's
+  implementation should not block on it.
+- Cache invalidation tied to taxonomy/index mutation (tag rename/delete,
+  reindex) rather than TTL-only.
+- Per-request dynamic schema constraints (e.g. actually enumerating valid IDs
+  in the JSON schema) if hallucinated-ID rates from the simpler approach here
+  turn out to matter in practice.
@@ -576,7 +576,9 @@ The following workflow action types are available:
 - Tags, correspondent, document type and storage path
 - Document owner
 - View and / or edit permissions to users or groups
- Custom fields. Note that no value for the field will be set
+- Custom fields, optionally with a value. If no value is set, the field is only added to the
+  document and any value it may already have is left untouched. If a value is set, it will
+  overwrite an existing value of that field on the document.

 ##### Removal {#workflow-action-removal}

@@ -111,7 +111,7 @@
                    routerLinkActive="active" (click)="closeMenu()" [ngbPopover]="view.name"
                    [disablePopover]="!slimSidebarEnabled" placement="end" container="body" triggers="mouseenter:mouseleave"
                    popoverClass="popover-slim">
-                    <i-bs class="me-2" name="funnel"></i-bs><span><div class="d-inline-flex view-name"><span class="overflow-hidden" [class.text-wrap]="!slimSidebarEnabled">{{view.name}}</span></div>
+                    <i-bs class="me-2" [name]="view.icon || 'funnel'"></i-bs><span><div class="d-inline-flex view-name"><span class="overflow-hidden" [class.text-wrap]="!slimSidebarEnabled">{{view.name}}</span></div>
                        @if (showSidebarCounts && !slimSidebarEnabled) {
                          <span class="badge bg-info text-dark ms-2 d-inline">{{ savedViewService.getDocumentCount(view) }}</span>
                        }
@@ -52,10 +52,10 @@ describe('CustomFieldsValuesComponent', () => {
  })

  it('should set selectedFields and map values correctly', () => {
-    component.value = { 1: 'value1' }
-    component.selectedFields = [1, 2]
-    expect(component.selectedFields).toEqual([1, 2])
-    expect(component.value).toEqual({ 1: 'value1', 2: null })
+    component.value = { 1: 'value1', 3: 0, 4: false }
+    component.selectedFields = [1, 2, 3, 4]
+    expect(component.selectedFields).toEqual([1, 2, 3, 4])
+    expect(component.value).toEqual({ 1: 'value1', 2: null, 3: 0, 4: false })
  })

  it('should return the correct custom field by id', () => {
@@ -77,7 +77,7 @@ export class CustomFieldsValuesComponent extends AbstractInputComponent<Object>
    this._selectedFields = newFields
    // map the selected fields to an object with field_id as key and value as value
    this.value = newFields.reduce((acc, fieldId) => {
-      acc[fieldId] = this.value?.[fieldId] || null
+      acc[fieldId] = this.value?.[fieldId] ?? null
      return acc
    }, {})
    this.onChange(this.value)
@@ -36,7 +36,16 @@
          (focus)="clearLastSearchTerm()"
          (clear)="clearLastSearchTerm()"
          (blur)="onBlur()">
+          <ng-template ng-label-tmp let-item="item">
+            @if (iconField && item[iconField]) {
+              <i-bs class="me-2" [name]="item[iconField]"></i-bs>
+            }
+            <span [title]="item[bindLabel]">{{item[bindLabel]}}</span>
+          </ng-template>
          <ng-template ng-option-tmp let-item="item">
+              @if (iconField && item[iconField]) {
+                <i-bs class="me-2" [name]="item[iconField]"></i-bs>
+              }
              <span [title]="item[bindLabel]">{{item[bindLabel]}}</span>
          </ng-template>
        </ng-select>
@@ -35,7 +35,7 @@ import { AbstractInputComponent } from '../abstract-input'
    NgxBootstrapIconsModule,
  ],
 })
-export class SelectComponent extends AbstractInputComponent<number> {
+export class SelectComponent extends AbstractInputComponent<number | string> {
  constructor() {
    super()
    this.addItemRef = this.addItem.bind(this)
@@ -100,6 +100,9 @@ export class SelectComponent extends AbstractInputComponent<number> {
  @Input()
  bindLabel: string = 'name'

+  @Input()
+  iconField: string
+
  public searchFn = (term: string, item: any): boolean =>
    matchesSearchText(item?.[this.bindLabel], term)

@@ -17,6 +17,10 @@ const permissions = [
  'view_document',
  'change_document',
  'delete_document',
+  'add_sharelinkbundle',
+  'view_sharelinkbundle',
+  'change_sharelinkbundle',
+  'delete_sharelinkbundle',
  'change_tag',
  'view_documenttype',
 ]
@@ -75,6 +79,7 @@ describe('PermissionsSelectComponent', () => {
    component.ngOnInit()
    component.writeValue(permissions)
    expect(component.typesWithAllActions).toContain('Document')
+    expect(component.typesWithAllActions).toContain('ShareLinkBundle')
  })

  it('should update checkboxes on permissions set', () => {
@@ -85,6 +90,10 @@ describe('PermissionsSelectComponent', () => {
    expect(input1.nativeElement.checked).toBeTruthy()
    const input2 = fixture.debugElement.query(By.css('input#Tag_Change'))
    expect(input2.nativeElement.checked).toBeTruthy()
+    const bundleInput = fixture.debugElement.query(
+      By.css('input#ShareLinkBundle_Add')
+    )
+    expect(bundleInput.nativeElement.checked).toBeTruthy()
  })

  it('disable checkboxes when permissions are inherited', () => {
@@ -1,6 +1,7 @@
 <pngx-widget-frame
  *pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.Document }"
  [title]="savedView.name"
+  [titleIcon]="savedView.icon || 'funnel'"
  [loading]="false"
  [draggable]="savedView"
  >
@@ -8,7 +8,12 @@
              <i-bs name="grip-vertical"></i-bs>
            </div>
          }
-          <h6 class="card-title mb-0">{{title()}}</h6>
+          <h6 class="card-title mb-0">
+            @if (titleIcon()) {
+              <i-bs class="me-2" [name]="titleIcon()"></i-bs>
+            }
+            {{title()}}
+          </h6>
          <ng-content select="[title-badge]"></ng-content>
          @if (badge() !== null && badge() !== undefined) {
            <span class="badge bg-info text-dark ms-2">{{badge()}}</span>
@@ -16,6 +16,8 @@ export class WidgetFrameComponent implements AfterViewInit {

  title = input<string>()

+  titleIcon = input<string>()
+
  draggable = input<any>()

  cardless = input(false)
@@ -97,7 +97,9 @@
    <div class="dropdown-menu shadow dropdown-menu-right" ngbDropdownMenu>
      @if (!list.activeSavedViewId) {
        @for (view of savedViewService.allViews; track view) {
-          <button ngbDropdownItem (click)="loadViewConfig(view.id)">{{view.name}}</button>
+          <button ngbDropdownItem (click)="loadViewConfig(view.id)">
+            <i-bs class="me-2" [name]="view.icon || 'funnel'"></i-bs>{{view.name}}
+          </button>
        }
        @if (savedViewService.allViews.length > 0) {
          <div class="dropdown-divider"></div>
@@ -457,6 +457,7 @@ export class DocumentListComponent
      modal.componentInstance.buttonsEnabled.set(false)
      let savedView: SavedView = {
        name: formValue.name,
+        icon: formValue.icon,
        filter_rules: this.list.filterRules,
        sort_reverse: this.list.sortReverse,
        sort_field: this.list.sortField,
@@ -6,6 +6,14 @@
  </div>
  <div class="modal-body">
    <pngx-input-text i18n-title title="Name" formControlName="name" [error]="error()?.name" autocomplete="off"></pngx-input-text>
+    <pngx-input-select
+      i18n-title
+      title="Icon"
+      formControlName="icon"
+      [items]="savedViewIcons"
+      iconField="icon"
+      [error]="error()?.icon">
+    </pngx-input-select>
    <pngx-input-check i18n-title title="Show in sidebar" formControlName="showInSideBar"></pngx-input-check>
    <pngx-input-check i18n-title title="Show on dashboard" formControlName="showOnDashboard"></pngx-input-check>
    <pngx-permissions-form accordion="true" formControlName="permissions_form"></pngx-permissions-form>
@@ -9,6 +9,7 @@ import { CheckComponent } from '../../common/input/check/check.component'
 import { PermissionsFormComponent } from '../../common/input/permissions/permissions-form/permissions-form.component'
 import { PermissionsGroupComponent } from '../../common/input/permissions/permissions-group/permissions-group.component'
 import { PermissionsUserComponent } from '../../common/input/permissions/permissions-user/permissions-user.component'
+import { SelectComponent } from '../../common/input/select/select.component'
 import { TextComponent } from '../../common/input/text/text.component'
 import { SaveViewConfigDialogComponent } from './save-view-config-dialog.component'

@@ -40,6 +41,7 @@ describe('SaveViewConfigDialogComponent', () => {
        ReactiveFormsModule,
        SaveViewConfigDialogComponent,
        TextComponent,
+        SelectComponent,
        CheckComponent,
        PermissionsFormComponent,
        PermissionsUserComponent,
@@ -63,6 +65,7 @@ describe('SaveViewConfigDialogComponent', () => {
    expect(component.defaultName()).toEqual(name)
    expect(result).toEqual({
      name,
+      icon: 'funnel',
      showInSideBar: false,
      showOnDashboard: false,
    })
@@ -94,6 +97,7 @@ describe('SaveViewConfigDialogComponent', () => {
    component.save()
    expect(result).toEqual({
      name,
+      icon: 'funnel',
      showInSideBar: true,
      showOnDashboard: true,
    })
@@ -113,6 +117,7 @@ describe('SaveViewConfigDialogComponent', () => {
    component.save()
    expect(result).toEqual({
      name: '',
+      icon: 'funnel',
      showInSideBar: false,
      showOnDashboard: false,
      permissions_form: permissions,
@@ -13,9 +13,14 @@ import {
  ReactiveFormsModule,
 } from '@angular/forms'
 import { NgbActiveModal } from '@ng-bootstrap/ng-bootstrap'
+import {
+  DEFAULT_SAVED_VIEW_ICON,
+  SAVED_VIEW_ICONS,
+} from 'src/app/data/saved-view-icons'
 import { User } from 'src/app/data/user'
 import { CheckComponent } from '../../common/input/check/check.component'
 import { PermissionsFormComponent } from '../../common/input/permissions/permissions-form/permissions-form.component'
+import { SelectComponent } from '../../common/input/select/select.component'
 import { TextComponent } from '../../common/input/text/text.component'

@Component({
@@ -24,6 +29,7 @@ import { TextComponent } from '../../common/input/text/text.component'
  styleUrls: ['./save-view-config-dialog.component.scss'],
  imports: [
    CheckComponent,
+    SelectComponent,
    TextComponent,
    PermissionsFormComponent,
    FormsModule,
@@ -41,6 +47,7 @@ export class SaveViewConfigDialogComponent implements OnInit {
  public saveClicked = new EventEmitter()

  users: User[]
+  readonly savedViewIcons = SAVED_VIEW_ICONS

  setDefaultName(value: string) {
    this.defaultName.set(value)
@@ -49,6 +56,7 @@ export class SaveViewConfigDialogComponent implements OnInit {

  saveViewConfigForm = new FormGroup({
    name: new FormControl(''),
+    icon: new FormControl(DEFAULT_SAVED_VIEW_ICON),
    showInSideBar: new FormControl(false),
    showOnDashboard: new FormControl(false),
    permissions_form: new FormControl(null),
@@ -65,6 +73,7 @@ export class SaveViewConfigDialogComponent implements OnInit {
    const formValue = this.saveViewConfigForm.value
    const saveViewConfig = {
      name: formValue.name,
+      icon: formValue.icon,
      showInSideBar: formValue.showInSideBar,
      showOnDashboard: formValue.showOnDashboard,
    }
@@ -7,15 +7,24 @@
 </pngx-page-header>
 <form [formGroup]="savedViewsForm" (ngSubmit)="save()">
  <ul class="list-group mb-3" formGroupName="savedViews">
-    @for (view of savedViews(); track view) {
+    @for (view of pagedSavedViews(); track view) {
      <li class="list-group-item py-3">
      <div [formGroupName]="view.id">
        <div class="row">
-          <div class="col">
+          <div class="col-md">
            <pngx-input-text title="Name" formControlName="name"></pngx-input-text>
          </div>
+          <div class="col-md">
+            <pngx-input-select
+            i18n-title
+            title="Icon"
+            formControlName="icon"
+            [items]="savedViewIcons"
+            iconField="icon">
+            </pngx-input-select>
+          </div>
          @if (canSaveSettings) {
-            <div class="col">
+            <div class="col-md">
              <div class="form-check form-switch mt-3">
                <input type="checkbox" class="form-check-input" id="show_on_dashboard_{{view.id}}" formControlName="show_on_dashboard">
                <label class="form-check-label" for="show_on_dashboard_{{view.id}}" i18n>Show on dashboard</label>
@@ -81,6 +90,11 @@
    }
  </ul>

-  <button type="button" (click)="reset()" class="btn btn-outline-secondary mb-2" [disabled]="(isDirty$ | async) === false" i18n>Cancel</button>
-  <button type="submit" class="btn btn-primary ms-2 mb-2" [disabled]="(isDirty$ | async) === false" i18n>Save</button>
+  <div class="d-flex align-items-center mb-3">
+    <button type="button" (click)="reset()" class="btn btn-outline-secondary mb-2" [disabled]="(isDirty$ | async) === false" i18n>Cancel</button>
+    <button type="submit" class="btn btn-primary ms-2 mb-2" [disabled]="(isDirty$ | async) === false" i18n>Save</button>
+    @if (savedViews()?.length > pageSize) {
+      <ngb-pagination class="ms-auto" [pageSize]="pageSize" [collectionSize]="savedViews().length" [page]="page()" [maxSize]="5" (pageChange)="page.set($event)" size="sm" aria-label="Pagination"></ngb-pagination>
+    }
+  </div>
 </form>
@@ -4,6 +4,7 @@ import { provideHttpClientTesting } from '@angular/common/http/testing'
 import { signal } from '@angular/core'
 import { ComponentFixture, TestBed } from '@angular/core/testing'
 import { FormsModule, ReactiveFormsModule } from '@angular/forms'
+import { By } from '@angular/platform-browser'
 import { NgbModal, NgbModule } from '@ng-bootstrap/ng-bootstrap'
 import { NgxBootstrapIconsModule, allIcons } from 'ngx-bootstrap-icons'
 import { Subject, of, throwError } from 'rxjs'
@@ -25,8 +26,20 @@ import { PageHeaderComponent } from '../../common/page-header/page-header.compon
 import { SavedViewsComponent } from './saved-views.component'

 const savedViews = [
-  { id: 1, name: 'view1', show_in_sidebar: true, show_on_dashboard: true },
-  { id: 2, name: 'view2', show_in_sidebar: false, show_on_dashboard: false },
+  {
+    id: 1,
+    name: 'view1',
+    icon: 'archive',
+    show_in_sidebar: true,
+    show_on_dashboard: true,
+  },
+  {
+    id: 2,
+    name: 'view2',
+    icon: 'funnel',
+    show_in_sidebar: false,
+    show_on_dashboard: false,
+  },
 ]

 describe('SavedViewsComponent', () => {
@@ -157,6 +170,24 @@ describe('SavedViewsComponent', () => {
    expect(patchBody.show_in_sidebar).toBeUndefined()
  })

+  it('should persist a changed icon', () => {
+    const patchSpy = jest.spyOn(savedViewService, 'patchMany')
+    const view = savedViews[0]
+    const iconControl = component.savedViewsForm
+      .get('savedViews')
+      .get(view.id.toString())
+      .get('icon')
+
+    iconControl.setValue('bell')
+    iconControl.markAsDirty()
+    component.save()
+
+    expect(patchSpy.mock.calls[0][0][0]).toMatchObject({
+      id: view.id,
+      icon: 'bell',
+    })
+  })
+
  it('should persist visibility changes to user settings', () => {
    const patchSpy = jest.spyOn(savedViewService, 'patchMany')
    const updateVisibilitySpy = jest
@@ -222,6 +253,44 @@ describe('SavedViewsComponent', () => {
    ).toEqual(view.show_on_dashboard)
  })

+  it('should page saved views, clamp the page if views are removed', () => {
+    const manyViews = Array.from({ length: 30 }, (_, i) => ({
+      id: i + 1,
+      name: `view${i + 1}`,
+    })) as SavedView[]
+    const listSpy = jest.spyOn(savedViewService, 'list').mockReturnValue(
+      of({
+        all: manyViews.map((v) => v.id),
+        count: manyViews.length,
+        results: manyViews.concat([]),
+      })
+    )
+    component.ngOnInit()
+    fixture.detectChanges()
+    expect(listSpy).toHaveBeenCalledWith(1, 100000, null, false, {
+      full_perms: true,
+    })
+    expect(component.pagedSavedViews()).toHaveLength(25)
+    expect(fixture.debugElement.query(By.css('ngb-pagination'))).not.toBeNull()
+    // all views have controls, not just the current page
+    expect(
+      Object.keys(component.savedViewsForm.get('savedViews').value)
+    ).toHaveLength(30)
+
+    component.page.set(2)
+    expect(component.pagedSavedViews()).toHaveLength(5)
+
+    listSpy.mockReturnValue(
+      of({
+        all: manyViews.slice(0, 25).map((v) => v.id),
+        count: 25,
+        results: manyViews.slice(0, 25),
+      })
+    )
+    component.ngOnInit()
+    expect(component.page()).toEqual(1)
+  })
+
  it('should support editing permissions', () => {
    const confirmClicked = new Subject<any>()
    const modalRef = {
@@ -1,18 +1,29 @@
 import { AsyncPipe } from '@angular/common'
-import { Component, OnDestroy, OnInit, inject, signal } from '@angular/core'
+import {
+  Component,
+  OnDestroy,
+  OnInit,
+  computed,
+  inject,
+  signal,
+} from '@angular/core'
 import {
  FormControl,
  FormGroup,
  FormsModule,
  ReactiveFormsModule,
 } from '@angular/forms'
-import { NgbModal } from '@ng-bootstrap/ng-bootstrap'
+import { NgbModal, NgbPaginationModule } from '@ng-bootstrap/ng-bootstrap'
 import { dirtyCheck } from '@ngneat/dirty-check-forms'
 import { NgxBootstrapIconsModule } from 'ngx-bootstrap-icons'
 import { BehaviorSubject, Observable, of, switchMap, takeUntil } from 'rxjs'
 import { PermissionsDialogComponent } from 'src/app/components/common/permissions-dialog/permissions-dialog.component'
 import { DisplayMode } from 'src/app/data/document'
 import { SavedView } from 'src/app/data/saved-view'
+import {
+  DEFAULT_SAVED_VIEW_ICON,
+  SAVED_VIEW_ICONS,
+} from 'src/app/data/saved-view-icons'
 import { IfPermissionsDirective } from 'src/app/directives/if-permissions.directive'
 import {
  PermissionAction,
@@ -25,6 +36,7 @@ import { ToastService } from 'src/app/services/toast.service'
 import { ConfirmButtonComponent } from '../../common/confirm-button/confirm-button.component'
 import { DragDropSelectComponent } from '../../common/input/drag-drop-select/drag-drop-select.component'
 import { NumberComponent } from '../../common/input/number/number.component'
+import { SelectComponent } from '../../common/input/select/select.component'
 import { TextComponent } from '../../common/input/text/text.component'
 import { PageHeaderComponent } from '../../common/page-header/page-header.component'
 import { LoadingComponentWithPermissions } from '../../loading-component/loading.component'
@@ -36,12 +48,14 @@ import { LoadingComponentWithPermissions } from '../../loading-component/loading
    PageHeaderComponent,
    ConfirmButtonComponent,
    NumberComponent,
+    SelectComponent,
    TextComponent,
    IfPermissionsDirective,
    DragDropSelectComponent,
    FormsModule,
    ReactiveFormsModule,
    AsyncPipe,
+    NgbPaginationModule,
    NgxBootstrapIconsModule,
  ],
 })
@@ -56,8 +70,17 @@ export class SavedViewsComponent
  private readonly modalService = inject(NgbModal)

  DisplayMode = DisplayMode
+  readonly savedViewIcons = SAVED_VIEW_ICONS

  readonly savedViews = signal<SavedView[]>(undefined)
+  readonly page = signal(1)
+  public readonly pageSize = 25
+  // All views are loaded at init, so paging is only for display
+  readonly pagedSavedViews = computed(() => {
+    const start = (this.page() - 1) * this.pageSize
+    return this.savedViews()?.slice(start, start + this.pageSize)
+  })
+
  private savedViewsGroup = new FormGroup({})
  public savedViewsForm: FormGroup = new FormGroup({
    savedViews: this.savedViewsGroup,
@@ -84,9 +107,11 @@ export class SavedViewsComponent
  private reloadViews(): void {
    this.loading.set(true)
    this.savedViewService
-      .list(null, null, null, false, { full_perms: true })
+      .list(1, 100000, null, false, { full_perms: true })
      .subscribe((r) => {
        this.savedViews.set(r.results)
+        const pageCount = Math.ceil(r.results.length / this.pageSize)
+        this.page.update((page) => Math.min(page, Math.max(1, pageCount)))
        this.initialize()
      })
  }
@@ -110,6 +135,7 @@ export class SavedViewsComponent
      storeData.savedViews[view.id.toString()] = {
        id: view.id,
        name: view.name,
+        icon: view.icon ?? DEFAULT_SAVED_VIEW_ICON,
        show_on_dashboard: view.show_on_dashboard,
        show_in_sidebar: view.show_in_sidebar,
        page_size: view.page_size,
@@ -122,6 +148,7 @@ export class SavedViewsComponent
        new FormGroup({
          id: new FormControl({ value: null, disabled: !canEdit }),
          name: new FormControl({ value: null, disabled: !canEdit }),
+          icon: new FormControl({ value: null, disabled: !canEdit }),
          show_on_dashboard: new FormControl({
            value: null,
            disabled: false,
@@ -200,6 +227,7 @@ export class SavedViewsComponent

      const modelFieldsChanged =
        group.get('name')?.dirty ||
+        group.get('icon')?.dirty ||
        group.get('page_size')?.dirty ||
        group.get('display_mode')?.dirty ||
        group.get('display_fields')?.dirty
@@ -0,0 +1,89 @@
+export const DEFAULT_SAVED_VIEW_ICON = 'funnel'
+
+export const SAVED_VIEW_ICONS = [
+  { id: 'archive', name: $localize`Archive`, icon: 'archive' },
+  { id: 'bank', name: $localize`Bank`, icon: 'bank' },
+  { id: 'basket', name: $localize`Basket`, icon: 'basket' },
+  { id: 'bell', name: $localize`Bell`, icon: 'bell' },
+  { id: 'bookmark', name: $localize`Bookmark`, icon: 'bookmark' },
+  { id: 'boxes', name: $localize`Boxes`, icon: 'boxes' },
+  { id: 'briefcase', name: $localize`Briefcase`, icon: 'briefcase' },
+  { id: 'building', name: $localize`Building`, icon: 'building' },
+  { id: 'calculator', name: $localize`Calculator`, icon: 'calculator' },
+  { id: 'calendar', name: $localize`Calendar`, icon: 'calendar' },
+  { id: 'camera', name: $localize`Camera`, icon: 'camera' },
+  {
+    id: 'card-checklist',
+    name: $localize`Checklist`,
+    icon: 'card-checklist',
+  },
+  { id: 'cash', name: $localize`Cash`, icon: 'cash' },
+  { id: 'chat-left-text', name: $localize`Chat`, icon: 'chat-left-text' },
+  { id: 'check-circle', name: $localize`Check`, icon: 'check-circle' },
+  { id: 'clipboard', name: $localize`Clipboard`, icon: 'clipboard' },
+  { id: 'clock-history', name: $localize`Clock`, icon: 'clock-history' },
+  { id: 'credit-card', name: $localize`Credit card`, icon: 'credit-card' },
+  { id: 'download', name: $localize`Download`, icon: 'download' },
+  { id: 'envelope', name: $localize`Envelope`, icon: 'envelope' },
+  {
+    id: 'exclamation-triangle',
+    name: $localize`Warning`,
+    icon: 'exclamation-triangle',
+  },
+  { id: 'file-earmark', name: $localize`File`, icon: 'file-earmark' },
+  {
+    id: 'file-earmark-check',
+    name: $localize`Checked file`,
+    icon: 'file-earmark-check',
+  },
+  {
+    id: 'file-earmark-lock',
+    name: $localize`Locked file`,
+    icon: 'file-earmark-lock',
+  },
+  {
+    id: 'file-earmark-medical',
+    name: $localize`Medical file`,
+    icon: 'file-earmark-medical',
+  },
+  {
+    id: 'file-earmark-person',
+    name: $localize`Person file`,
+    icon: 'file-earmark-person',
+  },
+  {
+    id: 'file-earmark-spreadsheet',
+    name: $localize`Spreadsheet`,
+    icon: 'file-earmark-spreadsheet',
+  },
+  { id: 'file-text', name: $localize`Text file`, icon: 'file-text' },
+  { id: 'files', name: $localize`Files`, icon: 'files' },
+  { id: 'folder', name: $localize`Folder`, icon: 'folder' },
+  { id: 'funnel', name: $localize`Filter`, icon: 'funnel' },
+  { id: 'gear', name: $localize`Gear`, icon: 'gear' },
+  { id: 'globe2', name: $localize`Globe`, icon: 'globe2' },
+  { id: 'hash', name: $localize`Hash`, icon: 'hash' },
+  { id: 'heart', name: $localize`Heart`, icon: 'heart' },
+  { id: 'house', name: $localize`House`, icon: 'house' },
+  { id: 'inbox', name: $localize`Inbox`, icon: 'inbox' },
+  { id: 'journals', name: $localize`Journals`, icon: 'journals' },
+  { id: 'list-task', name: $localize`Task list`, icon: 'list-task' },
+  { id: 'newspaper', name: $localize`Newspaper`, icon: 'newspaper' },
+  { id: 'paperclip', name: $localize`Attachment`, icon: 'paperclip' },
+  { id: 'people', name: $localize`People`, icon: 'people' },
+  { id: 'person', name: $localize`Person`, icon: 'person' },
+  { id: 'printer', name: $localize`Printer`, icon: 'printer' },
+  { id: 'receipt', name: $localize`Receipt`, icon: 'receipt' },
+  { id: 'safe', name: $localize`Safe`, icon: 'safe' },
+  { id: 'search', name: $localize`Search`, icon: 'search' },
+  { id: 'send', name: $localize`Send`, icon: 'send' },
+  { id: 'shop', name: $localize`Shop`, icon: 'shop' },
+  { id: 'stack', name: $localize`Stack`, icon: 'stack' },
+  { id: 'stars', name: $localize`Stars`, icon: 'stars' },
+  { id: 'tag', name: $localize`Tag`, icon: 'tag' },
+  { id: 'tags', name: $localize`Tags`, icon: 'tags' },
+  { id: 'telephone', name: $localize`Telephone`, icon: 'telephone' },
+  { id: 'truck', name: $localize`Truck`, icon: 'truck' },
+  { id: 'upc-scan', name: $localize`Barcode`, icon: 'upc-scan' },
+  { id: 'wallet2', name: $localize`Wallet`, icon: 'wallet2' },
+]
@@ -5,6 +5,8 @@ import { ObjectWithPermissions } from './object-with-permissions'
 export interface SavedView extends ObjectWithPermissions {
  name?: string

+  icon?: string
+
  show_on_dashboard?: boolean

  show_in_sidebar?: boolean
@@ -120,6 +120,12 @@ describe('PermissionsService', () => {
      actionKey: 'View', // PermissionAction.View
      typeKey: 'SystemMonitoring', // PermissionType.SystemMonitoring
    })
+    expect(permissionsService.getPermissionKeys('add_sharelinkbundle')).toEqual(
+      {
+        actionKey: 'Add', // PermissionAction.Add
+        typeKey: 'ShareLinkBundle', // PermissionType.ShareLinkBundle
+      }
+    )
  })

  it('correctly checks explicit global permissions', () => {
@@ -269,6 +275,10 @@ describe('PermissionsService', () => {
        'view_sharelink',
        'change_sharelink',
        'delete_sharelink',
+        'add_sharelinkbundle',
+        'view_sharelinkbundle',
+        'change_sharelinkbundle',
+        'delete_sharelinkbundle',
        'add_workflow',
        'view_workflow',
        'change_workflow',
@@ -26,6 +26,7 @@ export enum PermissionType {
  User = '%s_user',
  Group = '%s_group',
  ShareLink = '%s_sharelink',
+  ShareLinkBundle = '%s_sharelinkbundle',
  CustomField = '%s_customfield',
  Workflow = '%s_workflow',
  ProcessedMail = '%s_processedmail',
@@ -35,19 +35,27 @@ import {
  arrowRightShort,
  arrowUpRight,
  asterisk,
+  bank,
+  basket,
  bell,
  bodyText,
+  bookmark,
  boxArrowUp,
  boxArrowUpRight,
  boxes,
  braces,
+  briefcase,
+  building,
+  calculator,
  calendar,
  calendarEvent,
  calendarEventFill,
+  camera,
  cardChecklist,
  cardHeading,
  caretDown,
  caretUp,
+  cash,
  chatLeftText,
  chatSquareDots,
  check,
@@ -65,6 +73,7 @@ import {
  clipboardCheckFill,
  clipboardFill,
  clockHistory,
+  creditCard,
  dash,
  dashCircle,
  diagram3,
@@ -83,9 +92,12 @@ import {
  fileEarmarkDiff,
  fileEarmarkFill,
  fileEarmarkLock,
+  fileEarmarkMedical,
  fileEarmarkMinus,
+  fileEarmarkPerson,
  fileEarmarkPlus,
  fileEarmarkRichtext,
+  fileEarmarkSpreadsheet,
  fileText,
  files,
  filter,
@@ -93,12 +105,15 @@ import {
  folderFill,
  funnel,
  gear,
+  globe2,
  google,
  grid,
  gripVertical,
  hash,
  hddStack,
+  heart,
  house,
+  inbox,
  infoCircle,
  journals,
  link,
@@ -106,7 +121,9 @@ import {
  listTask,
  listUl,
  microsoft,
+  newspaper,
  nodePlus,
+  paperclip,
  pencil,
  people,
  peopleFill,
@@ -121,9 +138,12 @@ import {
  plusCircle,
  printer,
  questionCircle,
+  receipt,
+  safe,
  scissors,
  search,
  send,
+  shop,
  slashCircle,
  sliders2Vertical,
  sortAlphaDown,
@@ -133,14 +153,17 @@ import {
  tag,
  tagFill,
  tags,
+  telephone,
  textIndentLeft,
  textLeft,
  threeDots,
  threeDotsVertical,
  trash,
+  truck,
  uiRadios,
  unlock,
  upcScan,
+  wallet2,
  windowStack,
  x,
  xCircle,
@@ -258,15 +281,22 @@ const icons = {
  arrowRightShort,
  arrowUpRight,
  asterisk,
+  bank,
+  basket,
  bell,
  braces,
  bodyText,
+  bookmark,
  boxArrowUp,
  boxArrowUpRight,
  boxes,
+  briefcase,
+  building,
+  calculator,
  calendar,
  calendarEvent,
  calendarEventFill,
+  camera,
  cardChecklist,
  cardHeading,
  caretDown,
@@ -288,6 +318,8 @@ const icons = {
  clipboardCheckFill,
  clipboardFill,
  clockHistory,
+  cash,
+  creditCard,
  dash,
  dashCircle,
  diagram3,
@@ -306,9 +338,12 @@ const icons = {
  fileEarmarkDiff,
  fileEarmarkFill,
  fileEarmarkLock,
+  fileEarmarkMedical,
  fileEarmarkMinus,
+  fileEarmarkPerson,
  fileEarmarkPlus,
  fileEarmarkRichtext,
+  fileEarmarkSpreadsheet,
  files,
  fileText,
  filter,
@@ -316,12 +351,15 @@ const icons = {
  folderFill,
  funnel,
  gear,
+  globe2,
  google,
  grid,
  gripVertical,
  hash,
  hddStack,
+  heart,
  house,
+  inbox,
  infoCircle,
  journals,
  link,
@@ -329,8 +367,10 @@ const icons = {
  listTask,
  listUl,
  microsoft,
+  newspaper,
  nodePlus,
  pencil,
+  paperclip,
  people,
  peopleFill,
  person,
@@ -344,10 +384,13 @@ const icons = {
  plusCircle,
  printer,
  questionCircle,
+  receipt,
+  safe,
  scissors,
  search,
  send,
  slashCircle,
+  shop,
  sliders2Vertical,
  sortAlphaDown,
  sortAlphaUpAlt,
@@ -358,12 +401,15 @@ const icons = {
  tags,
  textIndentLeft,
  textLeft,
+  telephone,
  threeDots,
  threeDotsVertical,
  trash,
+  truck,
  uiRadios,
  unlock,
  upcScan,
+  wallet2,
  windowStack,
  x,
  xCircle,
@@ -55,7 +55,7 @@ class Command(PaperlessCommand):
            "--ratio",
            default=85.0,
            type=float,
-            help="Ratio to consider documents a match",
+            help="Ratio to consider documents a match (0.0 - 100.0)",
        )
        parser.add_argument(
            "--delete",
@@ -69,6 +69,17 @@ class Command(PaperlessCommand):
            action="store_true",
            help="Skip the confirmation prompt when used with --delete",
        )
+        parser.add_argument(
+            "--url",
+            default=None,
+            type=str,
+            help=(
+                "Base URL of the Paperless instance (e.g. "
+                "http://localhost:8000 or https://paperless.local). If set, matched "
+                "documents are shown as clickable (usually ctrl+click) links to "
+                "<url>/documents/<id>/details instead of by title."
+            ),
+        )

    def _render_results(
        self,
@@ -76,6 +87,7 @@ class Command(PaperlessCommand):
        *,
        opt_ratio: float,
        do_delete: bool,
+        base_url: str | None = None,
    ) -> list[int]:
        """Render match results as a Rich table. Returns list of PKs to delete."""
        if not matches:
@@ -88,13 +100,22 @@ class Command(PaperlessCommand):
            )
            return []

-        # Fetch titles for matched documents in a single query.
-        all_pks = {pk for m in matches for pk in (m.doc_one_pk, m.doc_two_pk)}
-        titles: dict[int, str] = dict(
-            Document.objects.filter(pk__in=all_pks)
-            .only("pk", "title")
-            .values_list("pk", "title"),
-        )
+        # Fetch titles for matched documents in a single query, unless we're
+        # going to show URLs instead.
+        titles: dict[int, str] = {}
+        if not base_url:
+            all_pks = {pk for m in matches for pk in (m.doc_one_pk, m.doc_two_pk)}
+            titles = dict(
+                Document.objects.filter(pk__in=all_pks)
+                .only("pk", "title")
+                .values_list("pk", "title"),
+            )
+
+        def _cell(pk: int) -> str:
+            if base_url:
+                doc_url = f"{base_url.rstrip('/')}/documents/{pk}/details"
+                return f"[link={doc_url}]{doc_url}[/link]"
+            return f"[dim]#{pk}[/dim] {titles.get(pk, 'Unknown')}"

        table = Table(
            title=f"Fuzzy Matches (threshold: {opt_ratio:.1f}%)",
@@ -124,8 +145,8 @@ class Command(PaperlessCommand):

            table.add_row(
                str(i),
-                f"[dim]#{pk_a}[/dim] {titles.get(pk_a, 'Unknown')}",
-                f"[dim]#{pk_b}[/dim] {titles.get(pk_b, 'Unknown')}",
+                _cell(pk_a),
+                _cell(pk_b),
                Text(f"{ratio:.1f}%", style=ratio_style),
            )
            maybe_delete_ids.append(pk_b)
@@ -208,6 +229,7 @@ class Command(PaperlessCommand):
            matches,
            opt_ratio=opt_ratio,
            do_delete=options["delete"],
+            base_url=options["url"],
        )

        if options["delete"] and maybe_delete_ids:
@@ -0,0 +1,79 @@
+from django.db import migrations
+from django.db import models
+
+
+class Migration(migrations.Migration):
+    dependencies = [
+        ("documents", "0022_add_perf_indexes"),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name="savedview",
+            name="icon",
+            field=models.CharField(
+                choices=[
+                    ("archive", "Archive"),
+                    ("bank", "Bank"),
+                    ("basket", "Basket"),
+                    ("bell", "Bell"),
+                    ("bookmark", "Bookmark"),
+                    ("boxes", "Boxes"),
+                    ("briefcase", "Briefcase"),
+                    ("building", "Building"),
+                    ("calculator", "Calculator"),
+                    ("calendar", "Calendar"),
+                    ("camera", "Camera"),
+                    ("card-checklist", "Checklist"),
+                    ("cash", "Cash"),
+                    ("chat-left-text", "Chat"),
+                    ("check-circle", "Check"),
+                    ("clipboard", "Clipboard"),
+                    ("clock-history", "Clock"),
+                    ("credit-card", "Credit card"),
+                    ("download", "Download"),
+                    ("envelope", "Envelope"),
+                    ("exclamation-triangle", "Warning"),
+                    ("file-earmark", "File"),
+                    ("file-earmark-check", "Checked file"),
+                    ("file-earmark-lock", "Locked file"),
+                    ("file-earmark-medical", "Medical file"),
+                    ("file-earmark-person", "Person file"),
+                    ("file-earmark-spreadsheet", "Spreadsheet"),
+                    ("file-text", "Text file"),
+                    ("files", "Files"),
+                    ("folder", "Folder"),
+                    ("funnel", "Filter"),
+                    ("gear", "Gear"),
+                    ("globe2", "Globe"),
+                    ("hash", "Hash"),
+                    ("heart", "Heart"),
+                    ("house", "House"),
+                    ("inbox", "Inbox"),
+                    ("journals", "Journals"),
+                    ("list-task", "Task list"),
+                    ("newspaper", "Newspaper"),
+                    ("paperclip", "Attachment"),
+                    ("people", "People"),
+                    ("person", "Person"),
+                    ("printer", "Printer"),
+                    ("receipt", "Receipt"),
+                    ("safe", "Safe"),
+                    ("search", "Search"),
+                    ("send", "Send"),
+                    ("shop", "Shop"),
+                    ("stack", "Stack"),
+                    ("stars", "Stars"),
+                    ("tag", "Tag"),
+                    ("tags", "Tags"),
+                    ("telephone", "Telephone"),
+                    ("truck", "Truck"),
+                    ("upc-scan", "Barcode"),
+                    ("wallet2", "Wallet"),
+                ],
+                default="funnel",
+                max_length=64,
+                verbose_name="icon",
+            ),
+        ),
+    ]
@@ -519,6 +519,68 @@ class Document(SoftDeleteModel, ModelWithOwner):  # type: ignore[django-manager-


 class SavedView(ModelWithOwner):
+    class Icon(models.TextChoices):
+        ARCHIVE = ("archive", _("Archive"))
+        BANK = ("bank", _("Bank"))
+        BASKET = ("basket", _("Basket"))
+        BELL = ("bell", _("Bell"))
+        BOOKMARK = ("bookmark", _("Bookmark"))
+        BOXES = ("boxes", _("Boxes"))
+        BRIEFCASE = ("briefcase", _("Briefcase"))
+        BUILDING = ("building", _("Building"))
+        CALCULATOR = ("calculator", _("Calculator"))
+        CALENDAR = ("calendar", _("Calendar"))
+        CAMERA = ("camera", _("Camera"))
+        CARD_CHECKLIST = ("card-checklist", _("Checklist"))
+        CASH = ("cash", _("Cash"))
+        CHAT_LEFT_TEXT = ("chat-left-text", _("Chat"))
+        CHECK_CIRCLE = ("check-circle", _("Check"))
+        CLIPBOARD = ("clipboard", _("Clipboard"))
+        CLOCK_HISTORY = ("clock-history", _("Clock"))
+        CREDIT_CARD = ("credit-card", _("Credit card"))
+        DOWNLOAD = ("download", _("Download"))
+        ENVELOPE = ("envelope", _("Envelope"))
+        EXCLAMATION_TRIANGLE = ("exclamation-triangle", _("Warning"))
+        FILE_EARMARK = ("file-earmark", _("File"))
+        FILE_EARMARK_CHECK = ("file-earmark-check", _("Checked file"))
+        FILE_EARMARK_LOCK = ("file-earmark-lock", _("Locked file"))
+        FILE_EARMARK_MEDICAL = ("file-earmark-medical", _("Medical file"))
+        FILE_EARMARK_PERSON = ("file-earmark-person", _("Person file"))
+        FILE_EARMARK_SPREADSHEET = (
+            "file-earmark-spreadsheet",
+            _("Spreadsheet"),
+        )
+        FILE_TEXT = ("file-text", _("Text file"))
+        FILES = ("files", _("Files"))
+        FOLDER = ("folder", _("Folder"))
+        FUNNEL = ("funnel", _("Filter"))
+        GEAR = ("gear", _("Gear"))
+        GLOBE = ("globe2", _("Globe"))
+        HASH = ("hash", _("Hash"))
+        HEART = ("heart", _("Heart"))
+        HOUSE = ("house", _("House"))
+        INBOX = ("inbox", _("Inbox"))
+        JOURNALS = ("journals", _("Journals"))
+        LIST_TASK = ("list-task", _("Task list"))
+        NEWSPAPER = ("newspaper", _("Newspaper"))
+        PAPERCLIP = ("paperclip", _("Attachment"))
+        PEOPLE = ("people", _("People"))
+        PERSON = ("person", _("Person"))
+        PRINTER = ("printer", _("Printer"))
+        RECEIPT = ("receipt", _("Receipt"))
+        SAFE = ("safe", _("Safe"))
+        SEARCH = ("search", _("Search"))
+        SEND = ("send", _("Send"))
+        SHOP = ("shop", _("Shop"))
+        STACK = ("stack", _("Stack"))
+        STARS = ("stars", _("Stars"))
+        TAG = ("tag", _("Tag"))
+        TAGS = ("tags", _("Tags"))
+        TELEPHONE = ("telephone", _("Telephone"))
+        TRUCK = ("truck", _("Truck"))
+        UPC_SCAN = ("upc-scan", _("Barcode"))
+        WALLET = ("wallet2", _("Wallet"))
+
    class DisplayMode(models.TextChoices):
        TABLE = ("table", _("Table"))
        SMALL_CARDS = ("smallCards", _("Small Cards"))
@@ -541,6 +603,13 @@ class SavedView(ModelWithOwner):

    name = models.CharField(_("name"), max_length=128)

+    icon = models.CharField(
+        _("icon"),
+        max_length=64,
+        choices=Icon.choices,
+        default=Icon.FUNNEL,
+    )
+
    sort_field = models.CharField(
        _("sort field"),
        max_length=128,
@@ -1383,6 +1383,7 @@ class SavedViewSerializer(OwnedObjectSerializer):
        fields = [
            "id",
            "name",
+            "icon",
            "sort_field",
            "sort_reverse",
            "filter_rules",
@@ -3213,6 +3214,13 @@ class WorkflowActionSerializer(serializers.ModelSerializer[WorkflowAction]):
                        {"assign_title": f'Invalid f-string detected: "{e.args[0]}"'},
                    )

+        if attrs.get("assign_custom_fields_values"):
+            # Empty strings treated as None to avoid unexpected behavior
+            attrs["assign_custom_fields_values"] = {
+                field_id: (None if value == "" else value)
+                for field_id, value in attrs["assign_custom_fields_values"].items()
+            }
+
        if (
            "type" in attrs
            and attrs["type"] == WorkflowAction.WorkflowActionType.EMAIL
@@ -2905,18 +2905,20 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):

        v1 = SavedView.objects.get(name="test")
        self.assertEqual(v1.sort_field, "created2")
+        self.assertEqual(v1.icon, SavedView.Icon.FUNNEL)
        self.assertEqual(v1.filter_rules.count(), 1)
        self.assertEqual(v1.owner, self.user)

        response = self.client.patch(
            f"/api/saved_views/{v1.id}/",
-            {"sort_reverse": True},
+            {"sort_reverse": True, "icon": SavedView.Icon.RECEIPT},
            format="json",
        )

        v1 = SavedView.objects.get(id=v1.id)
        self.assertEqual(response.status_code, status.HTTP_200_OK)
        self.assertTrue(v1.sort_reverse)
+        self.assertEqual(v1.icon, SavedView.Icon.RECEIPT)
        self.assertEqual(v1.filter_rules.count(), 1)

        view["filter_rules"] = [{"rule_type": 12, "value": "secret"}]
@@ -2936,6 +2938,13 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
        v1 = SavedView.objects.get(id=v1.id)
        self.assertEqual(v1.filter_rules.count(), 0)

+        response = self.client.patch(
+            f"/api/saved_views/{v1.id}/",
+            {"icon": "not-an-icon"},
+            format="json",
+        )
+        self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
+
    def test_saved_view_display_options(self) -> None:
        """
        GIVEN:
@@ -422,6 +422,11 @@ class TestApiWorkflows(DirectoriesMixin, APITestCase):
            json.dumps(
                {
                    "assign_title": "",
+                    "assign_custom_fields": [self.cf1.id, self.cf2.id],
+                    "assign_custom_fields_values": {
+                        str(self.cf1.id): "",
+                        str(self.cf2.id): 0,
+                    },
                },
            ),
            content_type="application/json",
@@ -429,6 +434,10 @@ class TestApiWorkflows(DirectoriesMixin, APITestCase):
        self.assertEqual(response.status_code, status.HTTP_201_CREATED)
        action = WorkflowAction.objects.get(id=response.data["id"])
        self.assertIsNone(action.assign_title)
+        self.assertEqual(
+            action.assign_custom_fields_values,
+            {str(self.cf1.id): None, str(self.cf2.id): 0},
+        )

        response = self.client.post(
            self.ENDPOINT_TRIGGERS,
@@ -1,3 +1,4 @@
+import os
 from io import StringIO
 from unittest.mock import patch

@@ -41,7 +42,7 @@ class TestFuzzyMatchCommand(TestCase):

    def test_invalid_ratio_upper_limit(self) -> None:
        """
-        GIVEN:s
+        GIVEN:
            - Invalid ratio above upper
        WHEN:
            - Command is called
@@ -108,6 +109,45 @@ class TestFuzzyMatchCommand(TestCase):
        stdout, _ = self.call_command("--processes", "1")
        self.assertIn("Found 1 matching pair(s)", stdout)

+    def test_with_matches_and_url(self) -> None:
+        """
+        GIVEN:
+            - 2 documents exist
+            - Similarity between content is 86.667
+            - --url is provided
+        WHEN:
+            - Command is called with --url
+        THEN:
+            - 1 match is returned from doc 1 to doc 2
+            - No match from doc 2 to doc 1 reported
+            - Output contains clickable links to the documents instead of titles
+        """
+        # Content similarity is 86.667
+        Document.objects.create(
+            checksum="BEEFCAFE",
+            title="A",
+            content="first document scanned by bob",
+            mime_type="application/pdf",
+            filename="test.pdf",
+        )
+        Document.objects.create(
+            checksum="DEADBEAF",
+            title="A",
+            content="first document scanned by alice",
+            mime_type="application/pdf",
+            filename="other_test.pdf",
+        )
+        with patch.dict(os.environ, {"COLUMNS": "200"}):
+            stdout, _ = self.call_command(
+                "--processes",
+                "1",
+                "--url",
+                "http://localhost:8000",
+            )
+        self.assertIn("Found 1 matching pair(s)", stdout)
+        self.assertIn("http://localhost:8000/documents/1/details", stdout)
+        self.assertIn("http://localhost:8000/documents/2/details", stdout)
+
    def test_with_3_matches(self) -> None:
        """
        GIVEN:
@@ -2000,6 +2000,55 @@ class TestWorkflows(
                r"Doc added in \w{3,}",
            )  # Match any 3-letter month name

+    def test_document_updated_workflow_existing_custom_field_empty_value(self) -> None:
+        """
+        GIVEN:
+            - Existing workflow with UPDATED trigger and action that assigns a custom field
+              with an empty value
+        WHEN:
+            - Document is updated that already contains the field with a value
+        THEN:
+            - The existing value is left untouched, see GH #13627
+        """
+        trigger = WorkflowTrigger.objects.create(
+            type=WorkflowTrigger.WorkflowTriggerType.DOCUMENT_UPDATED,
+            filter_has_document_type=self.dt,
+        )
+        action = WorkflowAction.objects.create()
+        action.assign_custom_fields.add(self.cf1)
+        action.assign_custom_fields_values = {self.cf1.pk: ""}
+        action.save()
+        w = Workflow.objects.create(
+            name="Workflow 1",
+            order=0,
+        )
+        w.triggers.add(trigger)
+        w.actions.add(action)
+        w.save()
+
+        doc = Document.objects.create(
+            title="sample test",
+            correspondent=self.c,
+            original_filename="sample.pdf",
+        )
+        CustomFieldInstance.objects.create(
+            document=doc,
+            field=self.cf1,
+            value_text="existing value",
+        )
+
+        superuser = User.objects.create_superuser("superuser")
+        self.client.force_authenticate(user=superuser)
+
+        self.client.patch(
+            f"/api/documents/{doc.id}/",
+            {"document_type": self.dt.id},
+            format="json",
+        )
+
+        doc.refresh_from_db()
+        self.assertEqual(doc.custom_fields.get(field=self.cf1).value, "existing value")
+
    def test_document_updated_workflow_existing_custom_field(self) -> None:
        """
        GIVEN:
@@ -2267,7 +2267,7 @@ class ChatStreamingView(GenericAPIView[Any]):
            if not has_perms_owner_aware(request.user, "view_document", document):
                return HttpResponseForbidden("Insufficient permissions")

-            documents = [document]
+            documents = Document.objects.filter(pk=document.pk)
        else:
            documents = Document.objects.filter(
                id__in=permitted_document_ids(request.user),
@@ -105,7 +105,8 @@ def apply_assignment_to_document(
                field=field,
                document=document,
            ).first()
-            if instance and args[value_field_name] is not None:
+            # empty string is indistinguishable from no value in the UI
+            if instance and args[value_field_name] not in (None, ""):
                setattr(instance, value_field_name, args[value_field_name])
                instance.save()
            elif not instance:
@@ -3,7 +3,9 @@ Built-in remote-OCR document parser.

 Handles documents by sending them to a configured remote OCR engine
 (currently Azure AI Vision / Document Intelligence) and retrieving both
-the extracted text and a searchable PDF with an embedded text layer.
+the extracted text and a searchable PDF with an embedded text layer. For
+born-digital PDFs that need no archive copy, the remote call is skipped
+entirely in favor of locally-extracted text (see ``RemoteDocumentParser.parse``).

 When no engine is configured, ``score()`` returns ``None`` so the parser
 is effectively invisible to the registry — the tesseract parser handles
@@ -22,6 +24,8 @@ from typing import Self
 from django.conf import settings

 from documents.parsers import ParseError
+from paperless.parsers.utils import extract_pdf_text
+from paperless.parsers.utils import post_process_text
 from paperless.version import __full_version_str__

 if TYPE_CHECKING:
@@ -70,8 +74,11 @@ class RemoteDocumentParser:
    """Parse documents via a remote OCR API (currently Azure AI Vision).

    This parser sends documents to a remote engine that returns both
-    extracted text and a searchable PDF with an embedded text layer.
-    It does not depend on Tesseract or ocrmypdf.
+    extracted text and a searchable PDF with an embedded text layer,
+    except when ``parse()`` is called with ``produce_archive=False`` for
+    a PDF, in which case the remote call is skipped and only locally
+    extracted text is returned (no archive). It does not depend on
+    Tesseract or ocrmypdf.

    Class attributes
    ----------------
@@ -160,8 +167,11 @@ class RemoteDocumentParser:
        Returns
        -------
        bool
-            Always True — the remote engine always returns a PDF with an
-            embedded text layer that serves as the archive copy.
+            Always True — the remote engine is capable of returning a PDF
+            with an embedded text layer to serve as the archive copy.
+            Whether it actually does so for a given document depends on
+            ``produce_archive`` passed to :meth:`parse` (see there for when
+            the remote engine call, and thus archive generation, is skipped).
        """
        return True

@@ -218,6 +228,12 @@ class RemoteDocumentParser:
    ) -> None:
        """Send the document to the remote engine and store results.

+        When *produce_archive* is False for a PDF, the caller (via
+        ``documents.consumer.should_produce_archive``) has already determined
+        that the document is born-digital and needs no archive — skip the
+        remote engine entirely rather than re-OCRing it and creating a
+        duplicate text layer.
+
        Parameters
        ----------
        document_path:
@@ -225,8 +241,8 @@ class RemoteDocumentParser:
        mime_type:
            Detected MIME type of the document.
        produce_archive:
-            Ignored — the remote engine always returns a searchable PDF,
-            which is stored as the archive copy regardless of this flag.
+            Whether an archive copy is wanted. For PDFs, False skips the
+            remote engine and uses locally-extracted text instead.
        """
        config = RemoteEngineConfig(
            engine=settings.REMOTE_OCR_ENGINE,
@@ -241,6 +257,16 @@ class RemoteDocumentParser:
            self._text = ""
            return

+        if not produce_archive and mime_type == "application/pdf":
+            logger.debug(
+                "Remote OCR: skipped — no archive requested, "
+                "using locally-extracted text",
+            )
+            self._text = (
+                post_process_text(extract_pdf_text(document_path, log=logger)) or ""
+            )
+            return
+
        if config.engine == "azureai":
            self._text = self._azure_ai_vision_parse(document_path, config)

@@ -337,6 +337,117 @@ class TestRemoteParserParse:
        assert remote_parser.get_date() is None


+# ---------------------------------------------------------------------------
+# parse() — produce_archive=False skips the remote engine (PDFs only)
+# ---------------------------------------------------------------------------
+
+
+class TestRemoteParserSkipsWhenNoArchiveWanted:
+    """When the caller has already decided no archive is needed for a PDF
+    (documents.consumer.should_produce_archive), the remote engine call is
+    skipped entirely in favor of locally-extracted text.
+    """
+
+    def test_pdf_skips_azure_when_no_archive_requested(
+        self,
+        remote_parser: RemoteDocumentParser,
+        simple_digital_pdf_file: Path,
+        azure_client: Mock,
+    ) -> None:
+        """
+        GIVEN: produce_archive=False for a PDF
+        WHEN:  parse() is called
+        THEN:  Azure is never invoked, no archive is produced, and text
+               comes from local pdftotext extraction
+        """
+        remote_parser.parse(
+            simple_digital_pdf_file,
+            "application/pdf",
+            produce_archive=False,
+        )
+
+        azure_client.begin_analyze_document.assert_not_called()
+        assert remote_parser.get_archive_path() is None
+        assert remote_parser.get_text() != ""
+
+    def test_pdf_no_archive_requested_text_matches_local_extraction(
+        self,
+        remote_parser: RemoteDocumentParser,
+        simple_digital_pdf_file: Path,
+        azure_client: Mock,
+        mocker: MockerFixture,
+    ) -> None:
+        """
+        GIVEN: produce_archive=False for a PDF
+        WHEN:  parse() is called
+        THEN:  the returned text is exactly the locally-extracted text,
+               not anything from the (unused) Azure mock
+        """
+        mocker.patch(
+            "paperless.parsers.remote.extract_pdf_text",
+            return_value="Local digital text.",
+        )
+
+        remote_parser.parse(
+            simple_digital_pdf_file,
+            "application/pdf",
+            produce_archive=False,
+        )
+
+        assert remote_parser.get_text() == "Local digital text."
+
+    def test_pdf_no_archive_requested_closes_no_client(
+        self,
+        remote_parser: RemoteDocumentParser,
+        simple_digital_pdf_file: Path,
+        azure_client: Mock,
+    ) -> None:
+        remote_parser.parse(
+            simple_digital_pdf_file,
+            "application/pdf",
+            produce_archive=False,
+        )
+
+        azure_client.close.assert_not_called()
+
+    def test_non_pdf_still_calls_azure_when_no_archive_requested(
+        self,
+        remote_parser: RemoteDocumentParser,
+        simple_digital_pdf_file: Path,
+        azure_client: Mock,
+    ) -> None:
+        """
+        Images have no local-text fallback, so produce_archive=False does
+        not skip the remote engine for non-PDF MIME types.
+        """
+        remote_parser.parse(
+            simple_digital_pdf_file,
+            "image/png",
+            produce_archive=False,
+        )
+
+        azure_client.begin_analyze_document.assert_called_once()
+        assert remote_parser.get_text() == _DEFAULT_TEXT
+
+    @pytest.mark.usefixtures("no_engine_settings")
+    def test_unconfigured_engine_takes_precedence_over_skip(
+        self,
+        remote_parser: RemoteDocumentParser,
+        simple_digital_pdf_file: Path,
+    ) -> None:
+        """An unconfigured engine still short-circuits before the
+        produce_archive check, returning empty text as before.
+        """
+        remote_parser.parse(
+            simple_digital_pdf_file,
+            "application/pdf",
+            produce_archive=False,
+        )
+
+        assert remote_parser.get_text() == ""
+        assert remote_parser.get_archive_path() is None
+
+
 # ---------------------------------------------------------------------------
 # parse() — Azure failure path
 # ---------------------------------------------------------------------------
@@ -2,6 +2,8 @@ import json
 import logging
 import sys

+from django.db.models import QuerySet
+
 from documents.models import Document
 from paperless.config import AIConfig
 from paperless_ai.client import AIClient
@@ -82,10 +84,21 @@ def _build_document_reference(


 def _get_document_references(
-    documents: list[Document],
+    documents: QuerySet[Document],
    top_nodes: list,
 ) -> list[dict[str, int | str]]:
-    allowed_documents = {doc.pk: doc for doc in documents}
+    candidate_ids: set[int] = set()
+    for node in top_nodes:
+        try:
+            candidate_ids.add(int(node.metadata["document_id"]))
+        except (KeyError, TypeError, ValueError):  # pragma: no cover
+            continue
+
+    if not candidate_ids:
+        return []
+
+    allowed_documents = {doc.pk: doc for doc in documents.filter(pk__in=candidate_ids)}
+
    references: list[dict[str, int | str]] = []
    seen_document_ids: set[int] = set()

@@ -119,7 +132,7 @@ def _format_chat_metadata_trailer(references: list[dict[str, int | str]]) -> str

 def stream_chat_with_documents(
    query_str: str,
-    documents: list[Document],
+    documents: QuerySet[Document],
    output_language: str | None = None,
 ):
    try:
@@ -135,10 +148,10 @@ def stream_chat_with_documents(

 def _stream_chat_with_documents(
    query_str: str,
-    documents: list[Document],
+    documents: QuerySet[Document],
    output_language: str | None = None,
 ):
-    if not documents:
+    if not documents.exists():
        yield CHAT_NO_CONTENT_MESSAGE
        return

@@ -148,7 +161,9 @@ def _stream_chat_with_documents(
    from llama_index.core.retrievers import VectorIndexRetriever

    config = AIConfig()
-    filters = _document_id_filters(str(doc.pk) for doc in documents)
+    filters = _document_id_filters(
+        str(pk) for pk in documents.values_list("pk", flat=True)
+    )

    # Hold the shared read lock for the whole operation: the query engine
    # retrieves from the vector store again during synthesis, so the connection
@@ -3,10 +3,12 @@ from unittest.mock import MagicMock
 from unittest.mock import patch

 import pytest
+from django.db.models.signals import post_init
 from llama_index.core import settings as llama_settings
 from llama_index.core.embeddings.mock_embed_model import MockEmbedding
 from llama_index.core.schema import TextNode

+from documents.models import Document
 from documents.tests.factories import DocumentFactory
 from paperless_ai import chat
 from paperless_ai import indexing
@@ -36,16 +38,6 @@ def patch_embed_nodes():
        yield mock_embed_nodes


-@pytest.fixture
-def mock_document():
-    doc = MagicMock()
-    doc.pk = 1
-    doc.title = "Test Document"
-    doc.filename = "test_file.pdf"
-    doc.content = "This is the document content."
-    return doc
-
-
 def assert_chat_output(
    output: list[str],
    *,
@@ -61,6 +53,13 @@ def assert_chat_output(
    }


+def _fake_documents_queryset(pks: list[int]) -> MagicMock:
+    qs = MagicMock()
+    qs.exists.return_value = bool(pks)
+    qs.values_list.return_value = pks
+    return qs
+
+
@pytest.mark.parametrize(
    ("output_language", "expected_language_line"),
    [
@@ -107,9 +106,10 @@ def test_build_refine_prompt(

@pytest.mark.django_db
 def test_stream_chat_with_one_document_retrieval(
-    mock_document,
    patch_embed_nodes,
 ) -> None:
+    document = DocumentFactory.create(title="Test Document", content="ignored")
+    documents = Document.objects.filter(pk=document.pk)
    with (
        patch("paperless_ai.chat.AIClient") as mock_client_cls,
        patch("paperless_ai.chat.load_or_build_index") as mock_load_index,
@@ -124,22 +124,19 @@ def test_stream_chat_with_one_document_retrieval(
        mock_client_cls.return_value = mock_client
        mock_client.llm = MagicMock()

-        mock_node = TextNode(
-            text="This is node content.",
-            metadata={"document_id": str(mock_document.pk), "title": "Test Document"},
-        )
        mock_index = MagicMock()
-        # Simulate get_nodes returning nodes (content exists)
-        mock_index.vector_store.get_nodes.return_value = [mock_node]
+        mock_index.vector_store.get_nodes.return_value = [
+            TextNode(
+                text="This is node content.",
+                metadata={"document_id": str(document.pk), "title": "Test Document"},
+            ),
+        ]
        mock_load_index.return_value = mock_index

        mock_retriever_instance = MagicMock()
        mock_retriever_instance.retrieve.return_value = [
            MagicMock(
-                metadata={
-                    "document_id": str(mock_document.pk),
-                    "title": "Test Document",
-                },
+                metadata={"document_id": str(document.pk), "title": "Test Document"},
            ),
        ]

@@ -153,7 +150,7 @@ def test_stream_chat_with_one_document_retrieval(
            "llama_index.core.retrievers.VectorIndexRetriever",
            return_value=mock_retriever_instance,
        ):
-            output = list(stream_chat_with_documents("What is this?", [mock_document]))
+            output = list(stream_chat_with_documents("What is this?", documents))

        mock_query_engine.query.assert_called_once_with("What is this?")
        synthesizer_kwargs = mock_get_response_synthesizer.call_args.kwargs
@@ -166,13 +163,16 @@ def test_stream_chat_with_one_document_retrieval(
            output,
            expected_chunks=["chunk1", "chunk2"],
            expected_references=[
-                {"id": mock_document.pk, "title": "Test Document"},
+                {"id": document.pk, "title": "Test Document"},
            ],
        )


@pytest.mark.django_db
 def test_stream_chat_with_multiple_documents_retrieval(patch_embed_nodes) -> None:
+    doc1 = DocumentFactory.create(title="Document 1", content="ignored")
+    doc2 = DocumentFactory.create(title="Document 2", content="ignored")
+    documents = Document.objects.filter(pk__in=[doc1.pk, doc2.pk])
    with (
        patch("paperless_ai.chat.AIClient") as mock_client_cls,
        patch("paperless_ai.chat.load_or_build_index") as mock_load_index,
@@ -184,23 +184,23 @@ def test_stream_chat_with_multiple_documents_retrieval(patch_embed_nodes) -> Non
        mock_client_cls.return_value = mock_client
        mock_client.llm = MagicMock()

-        mock_node1 = TextNode(
-            text="Content for doc 1.",
-            metadata={"document_id": "1", "title": "Document 1"},
-        )
-        mock_node2 = TextNode(
-            text="Content for doc 2.",
-            metadata={"document_id": "2", "title": "Document 2"},
-        )
        mock_index = MagicMock()
-        # Simulate get_nodes returning nodes (content exists)
-        mock_index.vector_store.get_nodes.return_value = [mock_node1, mock_node2]
+        mock_index.vector_store.get_nodes.return_value = [
+            TextNode(
+                text="Content for doc 1.",
+                metadata={"document_id": str(doc1.pk), "title": "Document 1"},
+            ),
+            TextNode(
+                text="Content for doc 2.",
+                metadata={"document_id": str(doc2.pk), "title": "Document 2"},
+            ),
+        ]
        mock_load_index.return_value = mock_index

        mock_retriever_instance = MagicMock()
        mock_retriever_instance.retrieve.return_value = [
-            MagicMock(metadata={"document_id": "1", "title": "Document 1"}),
-            MagicMock(metadata={"document_id": "2", "title": "Document 2"}),
+            MagicMock(metadata={"document_id": str(doc1.pk), "title": "Document 1"}),
+            MagicMock(metadata={"document_id": str(doc2.pk), "title": "Document 2"}),
        ]

        mock_response_stream = MagicMock()
@@ -210,14 +210,11 @@ def test_stream_chat_with_multiple_documents_retrieval(patch_embed_nodes) -> Non
        mock_query_engine_cls.return_value = mock_query_engine
        mock_query_engine.query.return_value = mock_response_stream

-        doc1 = MagicMock(pk=1, title="Document 1", filename="doc1.pdf")
-        doc2 = MagicMock(pk=2, title="Document 2", filename="doc2.pdf")
-
        with patch(
            "llama_index.core.retrievers.VectorIndexRetriever",
            return_value=mock_retriever_instance,
        ):
-            output = list(stream_chat_with_documents("What's up?", [doc1, doc2]))
+            output = list(stream_chat_with_documents("What's up?", documents))

        mock_query_engine.query.assert_called_once_with("What's up?")
        patch_embed_nodes.assert_not_called()
@@ -225,15 +222,15 @@ def test_stream_chat_with_multiple_documents_retrieval(patch_embed_nodes) -> Non
            output,
            expected_chunks=["chunk1", "chunk2"],
            expected_references=[
-                {"id": 1, "title": "Document 1"},
-                {"id": 2, "title": "Document 2"},
+                {"id": doc1.pk, "title": "Document 1"},
+                {"id": doc2.pk, "title": "Document 2"},
            ],
        )


 def test_stream_chat_empty_document_list() -> None:
    with patch("paperless_ai.chat.load_or_build_index") as mock_load_index:
-        output = list(stream_chat_with_documents("Any info?", []))
+        output = list(stream_chat_with_documents("Any info?", Document.objects.none()))
        mock_load_index.assert_not_called()
        assert output == ["Sorry, I couldn't find any content to answer your question."]

@@ -253,7 +250,9 @@ def test_stream_chat_no_matching_nodes() -> None:
        mock_index.vector_store.get_nodes.return_value = []
        mock_load_index.return_value = mock_index

-        output = list(stream_chat_with_documents("Any info?", [MagicMock(pk=1)]))
+        output = list(
+            stream_chat_with_documents("Any info?", _fake_documents_queryset([1])),
+        )

        assert output == ["Sorry, I couldn't find any content to answer your question."]

@@ -282,7 +281,9 @@ def test_stream_chat_unexpected_failure_returns_generic_error(caplog) -> None:
            )
            mock_retriever_cls.return_value = mock_retriever

-            output = list(stream_chat_with_documents("Any info?", [MagicMock(pk=1)]))
+            output = list(
+                stream_chat_with_documents("Any info?", _fake_documents_queryset([1])),
+            )

        assert output == [CHAT_ERROR_MESSAGE]
        assert "Failed to stream document chat response" in caplog.text
@@ -298,7 +299,12 @@ class TestStreamChatRetrieval:
    ) -> None:
        doc = DocumentFactory.create(content="hello world")
        # Nothing indexed for this document yet.
-        out = list(chat.stream_chat_with_documents("question?", [doc]))
+        out = list(
+            chat.stream_chat_with_documents(
+                "question?",
+                Document.objects.filter(pk=doc.pk),
+            ),
+        )
        assert chat.CHAT_NO_CONTENT_MESSAGE in out

    def test_chat_filter_contains_only_requested_document_ids(
@@ -332,7 +338,12 @@ class TestStreamChatRetrieval:
            side_effect=capture_retriever,
        )

-        list(chat.stream_chat_with_documents("question?", [included]))
+        list(
+            chat.stream_chat_with_documents(
+                "question?",
+                Document.objects.filter(pk=included.pk),
+            ),
+        )

        assert captured_filters, "VectorIndexRetriever was never constructed"
        filt = captured_filters[0]
@@ -340,3 +351,47 @@ class TestStreamChatRetrieval:
        filter_values = filt.filters[0].value
        assert str(included.pk) in filter_values
        assert str(excluded.pk) not in filter_values
+
+    @pytest.mark.django_db
+    def test_get_document_references_only_queries_referenced_documents(
+        self,
+        django_assert_num_queries,
+    ) -> None:
+        """Building references must not hydrate every document the caller is
+        permitted to see -- only the (<= CHAT_RETRIEVER_TOP_K) documents that
+        the retriever actually returned nodes for.
+        """
+        referenced = DocumentFactory.create(title="Referenced Document")
+        # Many more documents are "accessible" but never referenced by a node.
+        DocumentFactory.create_batch(200)
+
+        documents = Document.objects.all()
+        top_nodes = [
+            MagicMock(
+                metadata={
+                    "document_id": str(referenced.pk),
+                    "title": "Referenced Document",
+                },
+            ),
+        ]
+
+        hydrated_count = 0
+
+        def _count_hydration(sender, instance, **kwargs):
+            nonlocal hydrated_count
+            hydrated_count += 1
+
+        post_init.connect(_count_hydration, sender=Document)
+        try:
+            # One query: `documents.filter(pk__in=candidate_ids)` for the single
+            # referenced id. No query should scale with the 200 unreferenced documents.
+            with django_assert_num_queries(1):
+                references = chat._get_document_references(documents, top_nodes)
+        finally:
+            post_init.disconnect(_count_hydration, sender=Document)
+
+        # The bug this guards against: the old code hydrated all 201 accessible
+        # documents via `{doc.pk: doc for doc in documents}` before filtering by
+        # top_nodes. Only the referenced document should ever be constructed.
+        assert hydrated_count == 1
+        assert references == [{"id": referenced.pk, "title": "Referenced Document"}]
Author	SHA1	Message	Date
Trenton HolmesandClaude Sonnet 5	7b2bca80d4	Add spec and plan for AI taxonomy hints (discussion #12787 ) Supersedes the feature-ai-taxonomy-hints prototype and closed PR #13465. Keeps the prototype's core RAG-neighbour-derived candidate approach but fixes double retrieval, dropped assigned metadata, stale candidate names, unranked candidates, localization corrupting exact matches, unescaped untrusted prompt data, and a retrieval error boundary/caching gap, per design-review feedback. Reviewed against the current codebase by an independent agent across two rounds. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-12 19:42:48 -07:00
shamoonandGitHub	2a8579f610	Fix: include sharelink bundle perms in WebUI (#13664 )	2026-08-12 14:19:21 -07:00
GitHub Actions	4789fe9a52	Auto translate strings	2026-08-12 19:04:59 +00:00
shamoonandGitHub	e150c8c7c0	Enhancement: customizable icons for saved views (#13388 )	2026-08-12 19:03:24 +00:00
JaydenandGitHub	879cd4a30a	Enhancement: Add --url argument to document_fuzzy_match to improve output (#13123 ) Added a new --url argument to specify the base URL of the Paperless instance, allowing matched documents to be displayed as clickable links. Updated the logic to fetch document titles based on the presence of the base URL.	2026-08-12 08:23:05 -07:00
Trenton HandGitHub	6a02b87dde	Feature: Updates remote OCR parser to respect the OCR mode setting (#13408 ) * Have the remote parser respect the provided produce_archive_file setting, as already determined via the consumer checks * Updates the documentation to be correct about the respecting now * merge conflict fixing	2026-08-11 19:23:46 +00:00
Trenton HandGitHub	59a2651804	Fix: pass document chat queries as a QuerySet instead of a materialized list (#13638 ) In tracemalloc based profiling, not materializing the whole Document list reduced memory to approximately 20% of the baseline, with a peak memory that scaled with the library size. Now, the lazt queryset is used and only the needed pk value is actually contributing to memory	2026-08-11 15:25:08 +00:00
GitHub Actions	a99f63e059	Auto translate strings	2026-08-11 14:11:20 +00:00
shamoonandGitHub	939cb52f6e	Fix: add pagination to saved views management page (#13646 )	2026-08-11 07:08:30 -07:00
shamoonandGitHub	855669ddf9	Fix: fixes for workflow assign custom field values (#13630 )	2026-08-10 07:38:07 -07:00