Compare commits

..
Author SHA1 Message Date
stumpylog 3bd51461dc docs: flag where the transition spec and plan describe a moved API
The library changed after these were written and more changes are already
decided upstream. Records what is wrong today, what to write toward, the
one question still open, and the fast-JSON-field trap, rather than
silently leaving code that would fail on contact.
2026-08-11 14:24:24 -07:00
stumpylog 812e784a68 chore: update transition guidance for the current whoosh-compat API
Field references became a typed value rather than a dotted string, so
diagnostics carry one too and the registry exposes a single resolver.
Also records that the JSON fields must stay non-fast while existence
checks against a fast JSON field return inverted results.
2026-08-11 14:21:52 -07:00
stumpylogandClaude Fable 5 c86edaa0e8 chore: build typed search errors from structured diagnostic data
Diagnostics now carry field and raw_value, so the transition guidance
points at those instead of parsing human-readable message text.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-11 09:58:49 -07:00
stumpylogandClaude Sonnet 5 d8c9d22ea1 docs: add whoosh-compat transition implementation plan
16 bite-sized, TDD tasks across the design spec's 4-PR stack, each with
a suggested subagent type/model for delegated execution. Test/fixture
code in the acceptance-corpus and API-expansion tasks was verified
against the real codebase (documents/tests/search/conftest.py's
existing backend/index fixtures, test_backend.py's pytestmark
convention, CustomFieldInstance's typed value_text field) rather than
guessed, and the date-grammar parity test's AST-shape assumption was
confirmed by actually running whoosh_compat.parse() against a real
DATE FieldRegistry.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-09 21:27:46 -07:00
stumpylogandClaude Sonnet 5 50ed8c060b docs: fold agent-review findings into whoosh-compat transition spec
Agent review (source-verified against both repos) confirmed the spec's
claims accurate throughout, with one real gap: the JSON-subpath
tantivy-py carve-out (index.parse_query fallback for notes.*/
custom_fields.* until tantivy-py#716 ships) interacts with paperless's
pinned tantivy~=0.26.0 and wasn't mentioned. Also added two footnotes:
FieldRegistry forces date_only=True on any DATE spec regardless of the
PublicField default, and the date-grammar parity audit implicitly
grants new keyword vocabulary as a side effect.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-09 21:27:46 -07:00
stumpylogandClaude Sonnet 5 00631146ff docs: add whoosh-compat transition design spec
Design for replacing _translate.py/_dates.py with whoosh-compat: shared
field-definition table driving both the Tantivy schema and the query
FieldRegistry, diagnostics->exception mapping (aggregating all errors,
not just the first), a 4-PR stack with no rollout flag, and a
result-level acceptance corpus + date-grammar parity audit as the
safety net instead.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-09 21:27:46 -07:00
stumpylogandClaude Fable 5 d09caf480c chore: add whoosh-compat transition skill
Encodes the settled integration decisions for replacing the
hand-maintained search translation layer with whoosh-compat:
user-typed query surface policy, analyzer seam, diagnostics-before-emit
contract, mandatory date parity audit, test churn, and rollout plan.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-09 21:27:46 -07:00
69 changed files with 3081 additions and 2767 deletions
@@ -0,0 +1,62 @@
---
name: whoosh-compat-transition
description: Use when integrating the whoosh-compat library into paperless-ngx search, replacing src/documents/search/_translate.py or _dates.py, building the search FieldRegistry, or changing user query parsing during the whoosh-to-tantivy transition
---
# whoosh-compat transition
## Overview
whoosh-compat (github.com/stumpylog/whoosh-compat; local checkout usually at `../whoosh-compat`) replaces the hand-maintained translation layer (`src/documents/search/_translate.py`, `_dates.py`): it parses user queries with a faithful fork of whoosh's real grammar into a typed AST and emits programmatic tantivy queries. Read its README and ARCHITECTURE.md before wiring anything; its DIVERGENCES.md lists intended behavior differences and is the authority on "is this difference a bug".
## Decisions already made (do not re-derive)
- **Queries are user-typed free text.** The advanced search box passes whatever the user types straight to the parser (that is how the issue #13568 queries exist). Do NOT try to infer the supported field surface from frontend code; the frontend only generates a few date filter strings, everything else is typed by users.
- **The field surface is a policy decision, not `KNOWN_FIELDS`.** Today's `KNOWN_FIELDS` accepts internal ID fields (`tag_id`, `owner_id`, `viewer_id`, other `*_id`) that are undocumented in `docs/usage.md` and were ruled not user-searchable by the maintainer: exclude them from the `FieldRegistry` (they stay as programmatic permission/filter fields in `build_permission_filter`, which never touches user query text). The registry is built from documented syntax in `docs/usage.md` plus the v2-compat aliases (`type`, `path`, `type_id`-style aliases follow their canonical field's fate). Undocumented-but-working fields (`asn`, `page_count`, `num_notes`, `original_filename`, `checksum`) need an explicit maintainer yes/no; since users type freely, silently dropping one breaks any saved view using it, so a drop must be a visible, documented decision.
- **Analyzer seam:** `FieldSpec.analyzer` binds the live registered tantivy analyzer's `.analyze` (the same Rust analyzer used at index time; language-keyed, so rebuild the registry when `SEARCH_LANGUAGE` changes, on the same trigger as `register_tokenizers`). `pattern_normalizer` is `_tokenizer.ascii_fold`: character-level lowercase+fold only, NEVER stemming.
- **Diagnostics before emit:** `whoosh_compat.parse()` never raises on bad input. Check `ParseResult.diagnostics` and map to `SearchQueryError`/`InvalidDateQuery` (HTTP 400) BEFORE calling `emit()`; also catch the emitter's `UnsupportedQueryError` into a 400. Never carry forward the legacy raw-string fallback (`except Exception: query_str = raw_query`) into the new path; it masks integration bugs.
- **Build typed errors from structured diagnostic data, never by parsing `message`.** Each `Diagnostic` carries `kind`, `startchar`/`endchar`, and `field`/`raw_value`. `message` is human-readable text whose wording can change. For a range that fails on one bound, `raw_value` is the bound that actually failed.
- `diagnostic.field` is a **`FieldRef`**, not a string: use `str(diagnostic.field)` for the canonical dotted name (`created`, `notes.user`) or `diagnostic.field.name` for the field alone. Note the name is canonical, so an aliased query (`type:`) reports the field it resolves to (`document_type`), and the diagnostic span covers the offending value rather than the field name, so the text the user typed for the field is not recoverable.
- **The registry has one resolver.** `registry.make_ref(raw)` turns a raw field string into a `FieldRef` or `None` for an unknown field, and `registry.resolve(ref)` returns the spec. There is no `resolve_json()`; a dotted name is interpreted only inside `make_ref`.
- **`notes` and `custom_fields` are JSON fields** with fixed subpaths (`notes.user`/`notes.note`, `custom_fields.name`/`custom_fields.value`); the registry stays a static, language-keyed singleton, never per-request.
## Mandatory before deleting old code
- Date-grammar parity audit, line by line: every keyword, relative unit, and abbreviation `_dates.py` and `_translate.py` accept today (including the whoosh-era abbreviations kept for old saved views) must have an accepted form in whoosh-compat's dateparse grammar. Silent keyword loss is the saved-view breakage class behind issue #13568.
- Acceptance corpus compared by matched-document-ID sets, not query strings: the #13568 queries verbatim, real saved-view strings, every date keyword, field aliases, comma lists, date and numeric ranges, wildcards with bracket classes, boosts, JSON subpaths.
## Tests: what goes, what comes
Removed with their modules (do not port their string-level assertions):
- `src/documents/tests/search/test_translate.py`: its subject is deleted; string-translation unit cases are whoosh-compat's own responsibility now. Cases that encode real user-visible behavior get reincarnated as result-level acceptance cases, not string assertions.
- Date-keyword unit tests tied to `_dates.py` internals: same treatment.
- `test_query.py` cases asserting `parse_user_query` internals or intermediate query strings: rewritten against the new pipeline, asserting on matched results.
Kept: `test_migration_fulltext_query_field_prefixes.py` (data migration, orthogonal), `test_schema.py`, `test_tokenizer.py`, permission-filter and simple-search tests.
Added:
- A result-level acceptance module (paperless's analogue of whoosh-compat's `test_acceptance_e2e.py`): the corpus above against a real index built from `build_schema()`, asserting document-ID sets. Use `pytest.param(..., id="...")` for every case.
- Registry unit tests: internal `*_id` names rejected, aliases resolve to canonical fields, JSON subpaths match `docs/usage.md`, construction deterministic per language.
- One `Multitoken` case nested inside a top-level `OR` (whoosh-compat DIVERGENCES entry on Multitoken.DEFAULT) to prove it does not matter for paperless's data.
- If acceptance work surfaces a new whoosh-compat divergence, that is a whoosh-compat-repo change (its `differential-triage` skill applies), not a silent paperless workaround.
## Do not mark the JSON fields fast
`notes` and `custom_fields` must stay non-fast for now. Existence checks against a **fast** JSON field currently return inverted results (a document that has the field is reported as not having it), because the underlying call does not check subpath columns by default. The unsupported-configuration error raised for a non-fast JSON field advises marking it fast, which walks directly into that bug. Ignore that advice until the upstream issue is fixed, then re-check.
## Coordination
- whoosh-compat is pre-1.0: pin an exact version or git SHA; upgrades are deliberate, reviewed changes.
- JSON subpath emission depends on the installed tantivy-py version (fallback until quickwit-oss/tantivy-py#716 ships). The whoosh-compat repo has a `carve-out-retirement` skill; coordinate tantivy pin bumps with it, in a separate PR from the parser migration.
- Rollout: settings flag defaulting to the legacy path plus shadow-compare logging (log when old and new paths return different ID sets; sample if cost matters) for one release; delete `_translate.py`/`_dates.py` only after the flag defaults to the new path with no material reports.
## Common mistakes
- Inferring the field surface from frontend code (users type queries directly).
- Copying `KNOWN_FIELDS` into the registry wholesale (resurfaces internal fields).
- Wiring stemming into `pattern_normalizer`.
- Calling `emit()` unconditionally, or porting the legacy raw-string fallback.
- Deleting `_dates.py` without the parity audit.
- Porting `test_translate.py`'s string assertions instead of writing result-level tests.
-1
View File
@@ -699,7 +699,6 @@ document_fuzzy_match [--ratio] [--processes N]
| --ratio | No | 85.0 | a number between 0 and 100, setting how similar a document must be for it to be reported. Higher numbers mean more similarity. |
| --processes | No | 1/4 of system cores | Number of processes to use for matching. Setting 1 disables multiple processes |
| --delete | No | False | If provided, one document of a matched pair above the ratio will be deleted. |
| --url | No | blank | If an instance URL is provided, the output table will show URLs to each documents instead of the document ID and name. |
!!! warning
+4 -17
View File
@@ -948,11 +948,10 @@ for display in the web interface.
!!! note
The **remote OCR parser** (Azure AI) also honors this setting: when
no archive is requested (`never`, or `auto` with a born-digital PDF),
the remote engine is skipped entirely and locally-extracted text is
used instead, avoiding an unnecessary API call and a duplicate text
layer.
The **remote OCR parser** (Azure AI) always produces a searchable
PDF and stores it as the archive copy, regardless of this setting.
`ARCHIVE_FILE_GENERATION=never` has no effect when the remote
parser handles a document.
#### [`PAPERLESS_OCR_CLEAN=<mode>`](#PAPERLESS_OCR_CLEAN) {#PAPERLESS_OCR_CLEAN}
@@ -2048,18 +2047,6 @@ password. All of these options come from their similarly-named [Django settings]
Defaults to None.
#### [`PAPERLESS_REMOTE_OCR_MODE=<str>`](#PAPERLESS_REMOTE_OCR_MODE) {#PAPERLESS_REMOTE_OCR_MODE}
: Which documents are sent to the remote OCR engine.
- `always`: every document of a supported file type is sent to the remote
engine, bypassing the local OCR engine.
- `workflow_only`: documents are processed locally unless a workflow
explicitly enables remote OCR for them, letting you use the remote engine
selectively.
Defaults to "always".
## AI {#ai}
#### [`PAPERLESS_AI_ENABLED=<bool>`](#PAPERLESS_AI_ENABLED) {#PAPERLESS_AI_ENABLED}
-14
View File
@@ -456,20 +456,6 @@ def score(
return 10
```
**Remote services**
If your parser sends document content to a remote service, declare it:
```python
class MyCustomParser:
uses_remote_service = True
```
Paperless-ngx excludes such parsers when the document being consumed has not
been marked for remote processing, so users can keep remote OCR off by default
and enable it selectively with a workflow. Parsers that do not declare the
attribute are treated as fully local and are always considered.
**Archive and rendition flags**
```python
+4 -5
View File
@@ -187,11 +187,10 @@ PAPERLESS_ARCHIVE_FILE_GENERATION=auto
### Remote OCR parser
If you use the **remote OCR parser** (Azure AI), `ARCHIVE_FILE_GENERATION` is
honored the same way as for the local engine: when no archive is requested
(`never`, or `auto` with a born-digital PDF), the remote engine is skipped
entirely and locally-extracted text is used instead, avoiding an unnecessary
API call and a duplicate text layer.
If you use the **remote OCR parser** (Azure AI), note that it always produces a
searchable PDF and stores it as the archive copy. `ARCHIVE_FILE_GENERATION=never`
has no effect for documents handled by the remote parser - the archive is produced
unconditionally by the remote engine.
## Search Index (Whoosh -> Tantivy)
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,455 @@
# whoosh-compat transition design
Date: 2026-08-07
Status: approved, pending spec review
Related skill: `whoosh-compat-transition`
Related issue: [stumpylog/whoosh-compat#1](https://github.com/stumpylog/whoosh-compat/issues/1)
> ## Read this before implementing: parts of this spec describe an API that has moved
>
> whoosh-compat changed after this spec was written, and more changes are
> already decided. Re-check anything below against the library's own source
> and `ARCHITECTURE.md` before writing code from it. The library repo is the
> source of truth; this document is not.
>
> **Wrong today, fix on sight:**
>
> | This spec says | Reality now |
> | ---------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
> | `FieldRegistry.resolve_json(dotted)` (§ JSON subpath resolution) | Gone. There is one resolver: `registry.make_ref(raw) -> FieldRef \| None` interprets a dotted name, and `registry.resolve(ref) -> FieldSpec \| None` looks it up. A dot is interpreted only inside `make_ref`. |
> | `InvalidDateQuery(d.field, d.raw_value)` | `Diagnostic.field` is now a `FieldRef`, not a string. Use `str(d.field)` for the canonical dotted name, or `d.field.name`. The name is canonical, so an aliased query (`type:`) reports `document_type`. |
> | AST leaves carrying a field name string | Every field-carrying AST leaf now holds a `FieldRef`. |
>
> **Decided upstream, not yet implemented. Write toward these, they will land before this migration executes:**
>
> | Behavior | Issue |
> | ---------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------- |
> | An empty or unknown `default_fields` raises at `parse()`; a `field_boosts` key naming an alias resolves, one naming nothing raises | [#20](https://github.com/stumpylog/whoosh-compat/issues/20) |
> | A naive `basedate` is rejected rather than read in the host machine's timezone | [#19](https://github.com/stumpylog/whoosh-compat/issues/19) |
> | A wildcard on a numeric field produces a diagnostic instead of failing at search time, so the error mapping gains a case | [#17](https://github.com/stumpylog/whoosh-compat/issues/17) |
> | A bare JSON field name (`notes:foo`) demotes to a text search rather than raising at emit | [#11](https://github.com/stumpylog/whoosh-compat/issues/11) |
> | Registry construction rejects exists-target cycles, empty names, duplicate aliases within a spec, and dotted canonical names | [#21](https://github.com/stumpylog/whoosh-compat/issues/21) |
>
> **Still undecided, do not guess:** whether `emit()` keeps its `schema`
> parameter ([#27](https://github.com/stumpylog/whoosh-compat/issues/27)).
> This spec calls `emit(ast, index=index, schema=schema, registry=...)` in two
> places. Check the issue before writing either call site.
>
> **Trap:** do not add `fast=True` to the `notes` or `custom_fields` JSON
> specs. Existence checks against a fast JSON field currently return inverted
> results ([#7](https://github.com/stumpylog/whoosh-compat/issues/7)), and the
> error raised for a non-fast JSON field advises marking it fast, which walks
> straight into that bug. The field table below correctly leaves them non-fast.
>
> [#28](https://github.com/stumpylog/whoosh-compat/issues/28) tracks all open
> upstream work ordered by effort.
## Summary
Replace paperless-ngx's hand-maintained query-translation layer
(`src/documents/search/_translate.py`, `src/documents/search/_dates.py`)
with [whoosh-compat](https://github.com/stumpylog/whoosh-compat): a typed
Whoosh-grammar parser that emits programmatically constructed
`tantivy.Query` objects instead of building an intermediate Tantivy query
_string_. The integration point is narrow: `parse_user_query()` in
`src/documents/search/_query.py` is the only function whose implementation
changes; `_backend.py`, `_tokenizer.py`, simple/title search, CJK handling,
and permission filtering are all unaffected.
Delivered as a stack of four paperless-ngx PRs plus one prerequisite change
in whoosh-compat itself (same maintainer, no cross-repo coordination
overhead), landed with no feature flag and no shadow-compare rollout period
— safety comes from a result-level acceptance test corpus and a
date-grammar parity audit instead.
## Architecture
```
raw_query (user-typed)
wc.parse(raw_query, registry=FIELD_REGISTRY, default_fields=DEFAULT_SEARCH_FIELDS,
field_boosts=_FIELD_BOOSTS, tz=tz)
ParseResult(ast, diagnostics)
├─ diagnostics non-empty? → map ALL diagnostics to SearchQueryError
│ subclass(es) → HTTP 400 (never just the first diagnostic)
emit(ast, index=index, schema=schema, registry=FIELD_REGISTRY)
│ (raises UnsupportedQueryError → mapped to SearchQueryError → 400,
│ for constructs that parse but can't execute against tantivy)
tantivy.Query
existing clause assembly in parse_user_query(): Should(exact) + optional
fuzzy re-parse of raw_query + optional CJK bigram query, unchanged from today
_apply_permission_filter() in _backend.py wraps the result with
build_permission_filter() — entirely independent of whoosh-compat, unchanged
```
Permission filtering is explicitly out of scope for this migration:
`build_permission_filter()` builds its `tantivy.Query` directly against
`owner_id`/`viewer_id`/`viewer_group_id`, never through the parser or
registry, and those fields are exactly the internal `*_id` fields excluded
from the `FieldRegistry` (see "Field surface" below). Nothing in this
migration's diff touches it.
## PR stack
Each PR is independently buildable, reviewable, and CI-able; later PRs
rebase on earlier ones. No PR depends on whoosh-compat behavior it hasn't
already proven correct in isolation.
1. **Refactor `_schema.py` to a shared field-definition table.** Pure
refactor — `build_schema()`'s output is byte-identical before and after.
`test_schema.py` (existing) proves it.
2. **Pin whoosh-compat as a real dependency; build `FieldRegistry`.** New
`_registry.py` built from the same table PR 1 introduced. Registry unit
tests only — no wiring into search yet.
3. **Date-grammar parity audit.** A transitional, executable differential
test using the still-present `_dates.py`/`_translate.py` as the oracle.
Any gap found is fixed in whoosh-compat directly before this PR closes.
A whoosh-compat PyPI release is expected around this point (see
"Dependency pinning").
4. **Wire it in; delete the old path.** Rewrite `parse_user_query()`,
diagnostics→exception mapping, add the result-level acceptance corpus,
expand `test_api_search.py`, delete `_translate.py`/`_dates.py`/
`test_translate.py` and the internals-testing classes in `test_query.py`,
update `docs/usage.md` and changelog.
**Prerequisite, whoosh-compat repo** (tracked as
[stumpylog/whoosh-compat#1](https://github.com/stumpylog/whoosh-compat/issues/1),
lands before PR 4 starts its diagnostics-mapping work): add `field: str |
None` and `raw_value: str | None` to `Diagnostic`, threaded through at its
three construction sites (`dateparse.py`'s `_error()`, `default.py`'s
`term_query()` and `_coerce_range_bound()`), so paperless can build typed
exceptions without parsing whoosh-compat's human-readable `message` text.
## Field surface
The `FieldRegistry` covers only query-syntax-addressable fields — a subset
of the full Tantivy schema. Internal-only schema fields with no query-syntax
meaning of their own (`title_sort`/`correspondent_sort`/`type_sort` shadow
sort fields, `bigram_*` CJK fields, `simple_title`/`simple_content`,
`autocomplete_word`, `notes_text`) stay hardcoded `sb.add_*` calls in
`_schema.py`, untouched by the shared table.
**Decision: keep and document all five currently-undocumented-but-working
fields** (`asn`, `page_count`, `num_notes`, `original_filename`,
`checksum`) rather than dropping them — least risk of silently breaking an
existing saved view. `docs/usage.md`'s advanced-search section gets these
added with examples, as part of PR 4.
**Decision: `archive_checksum` stays out of scope.** Unlike `checksum`, it
isn't indexed in the Tantivy schema at all today (confirmed: `_schema.py`
only adds `checksum`; `_build_tantivy_doc` only calls
`doc.add_text("checksum", document.checksum)`). Making it searchable is a
schema-level change (new indexed field, new document population code), not
a parser-migration concern — left as a separate follow-up.
**Decision: internal `*_id` fields (`tag_id`, `correspondent_id`,
`document_type_id`, `storage_path_id`, `owner_id`, `viewer_id`,
`viewer_group_id`) are excluded from the `FieldRegistry` entirely.** They
remain Tantivy-schema-only, used exclusively by `build_permission_filter()`.
Because whoosh-compat folds any unrecognized `field:` prefix into literal
text (Whoosh-parity leniency, confirmed in `FieldsPlugin.do_fieldnames`
not an error), a saved view typed as `tag_id:5` won't 400: it silently
becomes a text search for the literal string `tag_id:5`, most likely
returning zero results. This is a real behavior change and gets a
**changelog callout**, not just a docs update, since a docs addition alone
wouldn't surface it to someone skimming release notes.
## Shared field-definition table (`_fields.py`)
```python
from whoosh_compat import FieldKind # reused directly — no parallel enum
@dataclass(frozen=True, slots=True)
class PublicField:
name: str
kind: FieldKind
aliases: tuple[str, ...] = ()
comma_values: bool = False
date_only: bool = False
fast: bool = False
subpaths: tuple[str, ...] = () # JSON kind only
PUBLIC_FIELDS = (
PublicField("title", FieldKind.TEXT),
PublicField("content", FieldKind.TEXT),
PublicField("correspondent", FieldKind.TEXT),
PublicField("document_type", FieldKind.TEXT, aliases=("type",)),
PublicField("storage_path", FieldKind.TEXT, aliases=("path",)),
PublicField("original_filename", FieldKind.TEXT),
PublicField("tag", FieldKind.TEXT, comma_values=True),
PublicField("checksum", FieldKind.KEYWORD),
PublicField("asn", FieldKind.U64, fast=True),
PublicField("page_count", FieldKind.U64, fast=True),
PublicField("num_notes", FieldKind.U64, fast=True),
PublicField("created", FieldKind.DATE, date_only=True, fast=True),
PublicField("modified", FieldKind.DATETIME, fast=True),
PublicField("added", FieldKind.DATETIME, fast=True),
PublicField("notes", FieldKind.JSON, subpaths=("user", "note")),
PublicField("custom_fields", FieldKind.JSON, subpaths=("name", "value")),
)
```
`build_schema()` derives its `sb.add_*` call and tokenizer from `kind`
(TEXT/KEYWORD → `add_text_field` with `paperless_text`/`raw` tokenizer
respectively; U64 → `add_unsigned_field`; DATE/DATETIME → `add_date_field`;
JSON → `add_json_field`). The `notes_text` snippet-companion field stays a
separate hardcoded line right after the `notes` entry — schema-only
plumbing with no query-syntax meaning.
`_registry.py` maps each `PublicField` to a `whoosh_compat.FieldSpec`,
kept as one flat dataclass (no kind-specific subclassing) to mirror
whoosh-compat's own `FieldSpec` design, which validates kind-conditional
attributes (e.g. JSON requires non-empty `subpaths`) at
`FieldRegistry.__init__` rather than in the type system.
Footnote for whoever writes `_registry.py`: `FieldRegistry.__init__` forces
`date_only=True` on _any_ `FieldKind.DATE` spec regardless of what's
passed, unconditionally — `PublicField.date_only` isn't an independent
knob for DATE fields the way it might look; it only matters in the sense
that `created` sets it explicitly for clarity, while `modified`/`added`
use `FieldKind.DATETIME` instead of relying on that override.
**`subpaths` stays `tuple[str, ...]`, not a nested structure.** Confirmed
against whoosh-compat's own `FieldRegistry.resolve_json()`: it splits a
dotted query term on the _first_ dot only and matches the remainder as an
exact string against `spec.subpaths` — even the docstring's own
`"metadata.author.name"` example is a single opaque string in the tuple,
not a recursive tree. A tuple of strings is exactly as expressive as the
library it feeds; inventing richer structure in `PublicField` now would
just get flattened back to strings at the registry-construction boundary.
Real recursive nesting, if ever needed, is new whoosh-compat capability
first.
**JSON document population stays separate from `subpaths`.** `subpaths` is
query-side only — it declares which dotted names are legal to type and
which JSON keys the emitter should address. It says nothing about how
`_backend.py::_build_tantivy_doc` builds the JSON documents at index-write
time, and that logic isn't uniform attribute access (`note.user.username`
needs a null guard and isn't `note.user`; `cfi.value_for_search` is a
property, not a literal `value` attribute), so a generic
`getattr(obj, subpath_name)` scheme would silently do the wrong thing for
both. That code stays hand-written, unchanged by this migration. Mitigation
instead: a coupling test (PR 2, alongside the registry unit tests) asserting
the literal JSON keys used in `_build_tantivy_doc`'s `doc.add_json(...)`
calls match `PUBLIC_FIELDS`' `notes`/`custom_fields` `subpaths` exactly, so
drift between the two is caught rather than silently becoming an
unqueryable (or silently unindexed) field.
**JSON subpath queries (`notes.*`, `custom_fields.*`) route through
`index.parse_query()`, not programmatic construction, given paperless's
pinned tantivy version.** Installed `tantivy-py`'s `Query.term_query`
cannot resolve a JSON subpath by exact field name — it raises as if the
field didn't exist. Until
[tantivy-py#716](https://github.com/quickwit-oss/tantivy-py/pull/716) lands
and ships, whoosh-compat's `TantivyEmitter._json_paths_supported()` feature-
detects this per process and falls back to a strictly escaped, single-leaf
`index.parse_query()` call for just that one leaf (whoosh-compat's README/
ARCHITECTURE.md call this out as "the JSON subpath carve-out"). Paperless
pins `tantivy~=0.26.0`, squarely inside the affected range (whoosh-compat's
`tantivy` extra only requires `tantivy>=0.24`, so nothing prevents this
combination). Nothing needs to change in this design because of it — the
carve-out is self-retiring on whoosh-compat's side once tantivy-py catches
up — but the acceptance corpus's `notes.user:`/`custom_fields.name:` cases
(PR 4) are exercising that fallback escaping path specifically, not the
programmatic path every other field goes through, and that's worth knowing
if one of those cases ever behaves oddly around quoting/escaping.
**Analyzer wiring**: `FieldSpec.analyzer` reuses the same `tantivy
.TextAnalyzer` objects `_tokenizer.py` already builds (`_paperless_text
(language)`, etc.) — standalone objects not dependent on index
registration, so `_registry.py` calls the same builder functions and binds
`.analyze` directly; `checksum` (KEYWORD, `raw` tokenizer) gets an identity
analyzer (`lambda t: [t]`). `pattern_normalizer` for every field is
`_tokenizer.ascii_fold` (character-fold only, never stemming) per the
skill's explicit instruction. The whole `FieldRegistry` is built once,
cached keyed by `settings.SEARCH_LANGUAGE`, rebuilt on the same trigger
`register_tokenizers()` already uses.
## Error handling
```python
class SearchQueryError(ValueError): ... # unchanged, base
class InvalidDateQuery(SearchQueryError): # unchanged
def __init__(self, field, value): ...
class InvalidNumberQuery(SearchQueryError): # new
def __init__(self, field: str | None, value: str | None) -> None:
self.field = field
self.value = value
super().__init__(f"Invalid numeric value {value!r} for field {field!r}.")
class MultipleSearchQueryErrors(SearchQueryError): # new
"""Aggregates every user-fixable error from one parse, not just the first."""
def __init__(self, errors: Sequence[SearchQueryError]) -> None:
self.errors = tuple(errors)
super().__init__("; ".join(str(e) for e in self.errors))
```
```python
def parse_user_query(index, raw_query, tz):
registry = get_field_registry(settings.SEARCH_LANGUAGE)
result = wc.parse(
raw_query, registry=registry, default_fields=DEFAULT_SEARCH_FIELDS,
field_boosts=_FIELD_BOOSTS, tz=tz,
)
if result.diagnostics:
raise _diagnostics_to_error(result.diagnostics) # ALL diagnostics, not [0]
try:
exact = tantivy_emit(result.ast, index=index, schema=index.schema, registry=registry)
except UnsupportedQueryError as e:
raise SearchQueryError(str(e)) from e
# CJK: unchanged — already re-parses raw_query directly via index.parse_query,
# never went through translate_query, so nothing here changes.
cjk_query = _build_cjk_query(index, raw_query, _CJK_ALL_FIELDS) if _has_cjk(raw_query) else None
clauses = [(tantivy.Occur.Should, exact)]
threshold = settings.ADVANCED_FUZZY_SEARCH_THRESHOLD
if threshold is not None:
# Fuzzy re-parses raw_query (not the AST) — no clean AST-level fuzzy
# equivalent exists; fuzzy matching was always an approximate,
# secondary clause, so this divergence from the exact-match path is
# acceptable.
fuzzy = index.parse_query(raw_query, DEFAULT_SEARCH_FIELDS, field_boosts=_FIELD_BOOSTS,
fuzzy_fields={f: (True, 1, True) for f in DEFAULT_SEARCH_FIELDS})
clauses.append((tantivy.Occur.Should, tantivy.Query.boost_query(fuzzy, 0.1)))
if cjk_query is not None:
clauses.append((tantivy.Occur.Should, cjk_query))
return exact if len(clauses) == 1 else tantivy.Query.boolean_query(clauses)
def _diagnostics_to_error(diagnostics: tuple[Diagnostic, ...]) -> SearchQueryError:
errors = [_single_diagnostic_to_error(d) for d in diagnostics]
return errors[0] if len(errors) == 1 else MultipleSearchQueryErrors(errors)
def _single_diagnostic_to_error(d: Diagnostic) -> SearchQueryError:
if d.kind is DiagnosticKind.BAD_DATE:
return InvalidDateQuery(d.field, d.raw_value)
if d.kind is DiagnosticKind.BAD_NUMBER:
return InvalidNumberQuery(d.field, d.raw_value)
return SearchQueryError(d.message)
```
No `except Exception: query_str = raw_query` fallback — per the skill, that
legacy defensive branch is explicitly not carried forward. A bug in the new
path must surface as a real error, not silently degrade to stale behavior.
`views.py`'s existing `except SearchQueryError as e: raise
ValidationError({"query": [str(e)]}) from e` handler gets one added branch
to surface every aggregated message instead of just one:
```python
except SearchQueryError as e:
messages = [str(sub) for sub in e.errors] if isinstance(e, MultipleSearchQueryErrors) else [str(e)]
raise ValidationError({"query": messages}) from e
```
`d.field`/`d.raw_value` depend on the whoosh-compat prerequisite change
(issue #1) landing first; until then (or if `field`/`raw_value` are `None`
for a given diagnostic kind not yet covered), `_single_diagnostic_to_error`
falls back to `SearchQueryError(d.message)`.
Deferred, explicitly out of scope for this PR stack: any frontend use of
`startchar`/`endchar` (already present on `Diagnostic` today) to highlight
the offending span in the search box. Backend-only for now, per explicit
decision.
## Testing
Existing test inventory (`src/documents/tests/search/` and
`test_api_search.py`):
| File | Fate |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `test_translate.py` | Deleted (PR 4) — subject deleted |
| `test_query.py`: `TestCreatedDateField`, `TestDateTimeFields`, `TestWhooshQueryRewriting`, `TestYearRangeRewriting`, `TestNonDateFieldsNotRewritten`, `TestPassthrough`, `TestNormalizeQuery` | Deleted (PR 4) — test `_translate.py`/`_dates.py` internals or intermediate query strings |
| `test_query.py`: `TestParseUserQuery` | Reviewed at plan time; result-level assertions folded into the new acceptance module, internals-only assertions dropped |
| `test_query.py`: `TestParseSimpleTextHighlightQuery`, `TestPermissionFilter` | Unchanged — never touched `translate_query` |
| `test_schema.py`, `test_tokenizer.py`, `test_backend.py`, `test_lock_backoff.py`, `test_migration_fulltext_query_field_prefixes.py` | Unchanged |
| `test_api_search.py` (`TestDocumentSearchApi`, 43 tests) | **Stays green across every PR in the stack** (hard gate, not just PR 4) — full HTTP+DB+index integration coverage catches wiring mistakes none of the narrower tests would |
New tests per PR:
- **PR 2**: `test_registry.py` — internal `*_id` names rejected; `type`/
`path` aliases resolve to canonical fields; JSON subpaths match
`docs/usage.md`; registry construction deterministic per language; the
`notes`/`custom_fields` dict-key coupling test described above.
- **PR 3**: `test_date_grammar_parity.py` — transitional, parametrized over
every keyword/unit `_dates.py`/`_translate.py` accept today
(`_DATE_KEYWORDS`, all of `_UNIT_ALIASES`'s Whoosh-era abbreviations —
`yrs`/`mos`/`wks`/`hrs`/`mins`/`secs` etc. — digit-precision forms, ISO
dash forms, `now-7d`/`now+1h`/`now-30m` compact offsets, open/reversed
ranges). Each case parses through
`wc.parse()` against a DATE-kind `FieldRegistry` and asserts no
diagnostics _and_ bounds matching what `_dates.py`/`_translate.py`
compute today, using the still-present legacy code as the oracle.
Deleted again in PR 4 along with that oracle, superseded by the
permanent acceptance corpus. This audit is scoped to _parity_ only —
whoosh-compat's date grammar is a strict superset of what `_dates.py`
accepts today (e.g. `tomorrow`, `now`, `midnight`, `noon`, weekday names
like `next monday`), so the migration also grants new date vocabulary for
free. That's a nice side effect, not something this PR needs to test or
document beyond noting it in the changelog alongside the other behavior
changes.
- **PR 4**:
- Result-level acceptance module (paperless's analogue of whoosh-compat's
`test_acceptance_e2e.py`): a real index built via `build_schema()`, the
issue #13568 queries verbatim, real saved-view strings, every date
keyword/unit, field aliases, comma lists, numeric/date ranges,
bracket-class wildcards, boosts, JSON subpaths — asserted by matched
document-ID set, `pytest.param(..., id=...)` per case. Plus a
multi-diagnostic case (two bad fields → `MultipleSearchQueryErrors`
with both messages present) and one `Multitoken` case nested inside a
top-level `OR` (proves DIVERGENCES entry 15 doesn't matter for
paperless's data, per the skill).
- `test_api_search.py` expanded: a multi-bad-field query (e.g.
`created:notadate AND asn:notanumber`) asserting the 400 response's
`query` list contains both messages; end-to-end searches on the five
newly-documented fields (`asn:`, `page_count:`, `num_notes:`,
`original_filename:`, `checksum:`) returning the right documents
through the real index.
## Dependency pinning
Stays `path = "../whoosh-compat"` in `[tool.uv.sources]` through the whole
PR stack — both repos are being actively co-developed. The final swap
happens at PR 4:
- **Primary plan**: whoosh-compat is released to PyPI around PR 3 (per
your stated intent), assuming the parity audit and issue #1 don't turn
up anything else needing a second round. PR 4 switches to a pinned PyPI
version (`whoosh-compat[tantivy]==X.Y.Z` in `dependencies`, the
`[tool.uv.sources]` override removed entirely).
- **Fallback**: if the PyPI release slips past PR 4's start, pin an exact
git commit SHA instead (`whoosh-compat[tantivy] @ git+https://github.com/
stumpylog/whoosh-compat@<sha>`), per the skill's "pre-1.0: pin an exact
version or git SHA, upgrades are deliberate" guidance.
The `TODO` comment already sitting in `pyproject.toml` (from the earlier
smoke-test setup) gets updated to reflect this — "release, else pinned SHA"
— rather than committing hard to one path before it's known which applies.
## Explicitly out of scope
- `archive_checksum` indexing/search (separate schema-level follow-up).
- Frontend consumption of `Diagnostic.startchar`/`endchar` for in-box error
highlighting (backend-only for this PR stack).
- A feature flag or shadow-compare rollout period — explicitly decided
against; safety comes from the acceptance corpus and parity audit instead.
- Any change to `build_permission_filter()`/`_apply_permission_filter()`
confirmed untouched by this migration.
+2 -9
View File
@@ -576,9 +576,7 @@ The following workflow action types are available:
- Tags, correspondent, document type and storage path
- Document owner
- View and / or edit permissions to users or groups
- Custom fields, optionally with a value. If no value is set, the field is only added to the
document and any value it may already have is left untouched. If a value is set, it will
overwrite an existing value of that field on the document.
- Custom fields. Note that no value for the field will be set
##### Removal {#workflow-action-removal}
@@ -1086,16 +1084,11 @@ Paperless-ngx supports performing OCR on documents using remote services. At the
[Microsoft's Azure "Document Intelligence" service](https://azure.microsoft.com/en-us/products/ai-services/ai-document-intelligence).
This is of course a paid service (with a free tier) which requires an Azure account and subscription. Azure AI is not affiliated with
Paperless-ngx in any way. When enabled, Paperless-ngx will automatically send appropriate documents to Azure for OCR processing, bypassing
the local OCR engine. See the [configuration](configuration.md#PAPERLESS_REMOTE_OCR_ENGINE) options for more details. These
settings can be supplied as environment variables or via **Application Configuration**.
the local OCR engine. See the [configuration](configuration.md#PAPERLESS_REMOTE_OCR_ENGINE) options for more details.
Additionally, when using a commercial service with this feature, consider both potential costs as well as any associated file size
or page limitations (e.g. with a free tier).
By default, every document of a supported file type is sent to the remote engine. To use it more selectively, set the
[remote OCR mode](configuration.md#PAPERLESS_REMOTE_OCR_MODE) to `workflow_only`. Documents are then processed locally
unless a workflow explicitly enables remote OCR for them, so you can limit the remote engine to particular documents.
## Architecture
Paperless-ngx consists of the following components:
+99 -491
View File
File diff suppressed because it is too large Load Diff
@@ -14,48 +14,43 @@
<a ngbNavLink>{{category}}</a>
<ng-template ngbNavContent>
<div class="p-3">
@for (section of getCategorySections(category); track section) {
@if (section) {
<h5 class="mt-4 mb-3">{{section}}</h5>
}
<div class="row row-cols-1 row-cols-md-2 row-cols-lg-3 g-2">
@for (option of getCategoryOptions(category, section); track option.key) {
<div class="col">
<div class="card bg-light">
<div class="card-body">
<div class="card-title d-flex align-items-center">
<h6 class="mb-0">
{{option.title}}
</h6>
<a class="btn btn-sm btn-link" title="Read the documentation about this setting" i18n-title [href]="getDocsUrl(option.config_key)" target="_blank" referrerpolicy="no-referrer">
<i-bs name="info-circle"></i-bs>
</a>
@if (isSet(option.key)) {
<button type="button" class="btn btn-sm btn-link text-danger ms-auto pe-0" title="Reset" i18n-title (click)="resetOption(option.key)">
<i-bs class="me-1" name="x"></i-bs><ng-container i18n>Reset</ng-container>
</button>
}
</div>
<div class="mb-n3">
@switch (option.type) {
@case (ConfigOptionType.Select) { <pngx-input-select [formControlName]="option.key" [error]="errors[option.key]" [items]="option.choices" [allowNull]="true"></pngx-input-select> }
@case (ConfigOptionType.Number) { <pngx-input-number [formControlName]="option.key" [error]="errors[option.key]" [showAdd]="false"></pngx-input-number> }
@case (ConfigOptionType.Boolean) { <pngx-input-switch [formControlName]="option.key" [error]="errors[option.key]" [showUnsetNote]="true" [horizontal]="true" title="Enable" i18n-title></pngx-input-switch> }
@case (ConfigOptionType.String) { <pngx-input-text [formControlName]="option.key" [error]="errors[option.key]"></pngx-input-text> }
@case (ConfigOptionType.JSON) { <pngx-input-text [formControlName]="option.key" [error]="errors[option.key]"></pngx-input-text> }
@case (ConfigOptionType.File) { <pngx-input-file [formControlName]="option.key" (upload)="uploadFile($event, option.key)" [error]="errors[option.key]"></pngx-input-file> }
@case (ConfigOptionType.Password) { <pngx-input-password [formControlName]="option.key" [error]="errors[option.key]"></pngx-input-password> }
}
</div>
@if (option.note) {
<div class="form-text fst-italic">{{option.note}}</div>
<div class="row row-cols-1 row-cols-md-2 row-cols-lg-3 g-2">
@for (option of getCategoryOptions(category); track option.key) {
<div class="col">
<div class="card bg-light">
<div class="card-body">
<div class="card-title d-flex align-items-center">
<h6 class="mb-0">
{{option.title}}
</h6>
<a class="btn btn-sm btn-link" title="Read the documentation about this setting" i18n-title [href]="getDocsUrl(option.config_key)" target="_blank" referrerpolicy="no-referrer">
<i-bs name="info-circle"></i-bs>
</a>
@if (isSet(option.key)) {
<button type="button" class="btn btn-sm btn-link text-danger ms-auto pe-0" title="Reset" i18n-title (click)="resetOption(option.key)">
<i-bs class="me-1" name="x"></i-bs><ng-container i18n>Reset</ng-container>
</button>
}
</div>
<div class="mb-n3">
@switch (option.type) {
@case (ConfigOptionType.Select) { <pngx-input-select [formControlName]="option.key" [error]="errors[option.key]" [items]="option.choices" [allowNull]="true"></pngx-input-select> }
@case (ConfigOptionType.Number) { <pngx-input-number [formControlName]="option.key" [error]="errors[option.key]" [showAdd]="false"></pngx-input-number> }
@case (ConfigOptionType.Boolean) { <pngx-input-switch [formControlName]="option.key" [error]="errors[option.key]" [showUnsetNote]="true" [horizontal]="true" title="Enable" i18n-title></pngx-input-switch> }
@case (ConfigOptionType.String) { <pngx-input-text [formControlName]="option.key" [error]="errors[option.key]"></pngx-input-text> }
@case (ConfigOptionType.JSON) { <pngx-input-text [formControlName]="option.key" [error]="errors[option.key]"></pngx-input-text> }
@case (ConfigOptionType.File) { <pngx-input-file [formControlName]="option.key" (upload)="uploadFile($event, option.key)" [error]="errors[option.key]"></pngx-input-file> }
@case (ConfigOptionType.Password) { <pngx-input-password [formControlName]="option.key" [error]="errors[option.key]"></pngx-input-password> }
}
</div>
@if (option.note) {
<div class="form-text fst-italic">{{option.note}}</div>
}
</div>
</div>
}
</div>
}
</div>
}
</div>
</div>
</ng-template>
</li>
@@ -8,11 +8,7 @@ import { NgbModule } from '@ng-bootstrap/ng-bootstrap'
import { NgSelectModule } from '@ng-select/ng-select'
import { NgxBootstrapIconsModule, allIcons } from 'ngx-bootstrap-icons'
import { of, throwError } from 'rxjs'
import {
ConfigCategory,
ConfigSection,
OutputTypeConfig,
} from 'src/app/data/paperless-config'
import { OutputTypeConfig } from 'src/app/data/paperless-config'
import { ConfigService } from 'src/app/services/config.service'
import { SettingsService } from 'src/app/services/settings.service'
import { ToastService } from 'src/app/services/toast.service'
@@ -162,24 +158,4 @@ describe('ConfigComponent', () => {
component.resetOption('barcodes_enabled')
expect(component.configForm.get('barcodes_enabled').value).toBeNull()
})
it('should group options into sections within a category, or not', () => {
const sections = component.getCategorySections(ConfigCategory.OCR)
expect(sections).toEqual([null, ConfigSection.RemoteOCR])
expect(
component
.getCategoryOptions(ConfigCategory.OCR)
.map((option) => option.key)
).toContain('output_type')
expect(
component
.getCategoryOptions(ConfigCategory.OCR, ConfigSection.RemoteOCR)
.map((option) => option.key)
).toEqual([
'remote_ocr_engine',
'remote_ocr_api_key',
'remote_ocr_endpoint',
'remote_ocr_mode',
])
})
})
@@ -74,20 +74,8 @@ export class ConfigComponent
return Object.values(ConfigCategory)
}
getCategorySections(category: string): string[] {
return [
...new Set(
PaperlessConfigOptions.filter((o) => o.category === category).map(
(o) => o.section ?? null // null means no section
)
),
]
}
getCategoryOptions(category: string, section: string = null): ConfigOption[] {
return PaperlessConfigOptions.filter(
(o) => o.category === category && (o.section ?? null) === section
)
getCategoryOptions(category: string): ConfigOption[] {
return PaperlessConfigOptions.filter((o) => o.category === category)
}
initialConfig: PaperlessConfig
@@ -111,7 +111,7 @@
routerLinkActive="active" (click)="closeMenu()" [ngbPopover]="view.name"
[disablePopover]="!slimSidebarEnabled" placement="end" container="body" triggers="mouseenter:mouseleave"
popoverClass="popover-slim">
<i-bs class="me-2" [name]="view.icon || 'funnel'"></i-bs><span><div class="d-inline-flex view-name"><span class="overflow-hidden" [class.text-wrap]="!slimSidebarEnabled">{{view.name}}</span></div>
<i-bs class="me-2" name="funnel"></i-bs><span><div class="d-inline-flex view-name"><span class="overflow-hidden" [class.text-wrap]="!slimSidebarEnabled">{{view.name}}</span></div>
@if (showSidebarCounts && !slimSidebarEnabled) {
<span class="badge bg-info text-dark ms-2 d-inline">{{ savedViewService.getDocumentCount(view) }}</span>
}
@@ -52,10 +52,10 @@ describe('CustomFieldsValuesComponent', () => {
})
it('should set selectedFields and map values correctly', () => {
component.value = { 1: 'value1', 3: 0, 4: false }
component.selectedFields = [1, 2, 3, 4]
expect(component.selectedFields).toEqual([1, 2, 3, 4])
expect(component.value).toEqual({ 1: 'value1', 2: null, 3: 0, 4: false })
component.value = { 1: 'value1' }
component.selectedFields = [1, 2]
expect(component.selectedFields).toEqual([1, 2])
expect(component.value).toEqual({ 1: 'value1', 2: null })
})
it('should return the correct custom field by id', () => {
@@ -77,7 +77,7 @@ export class CustomFieldsValuesComponent extends AbstractInputComponent<Object>
this._selectedFields = newFields
// map the selected fields to an object with field_id as key and value as value
this.value = newFields.reduce((acc, fieldId) => {
acc[fieldId] = this.value?.[fieldId] ?? null
acc[fieldId] = this.value?.[fieldId] || null
return acc
}, {})
this.onChange(this.value)
@@ -36,16 +36,7 @@
(focus)="clearLastSearchTerm()"
(clear)="clearLastSearchTerm()"
(blur)="onBlur()">
<ng-template ng-label-tmp let-item="item">
@if (iconField && item[iconField]) {
<i-bs class="me-2" [name]="item[iconField]"></i-bs>
}
<span [title]="item[bindLabel]">{{item[bindLabel]}}</span>
</ng-template>
<ng-template ng-option-tmp let-item="item">
@if (iconField && item[iconField]) {
<i-bs class="me-2" [name]="item[iconField]"></i-bs>
}
<span [title]="item[bindLabel]">{{item[bindLabel]}}</span>
</ng-template>
</ng-select>
@@ -35,7 +35,7 @@ import { AbstractInputComponent } from '../abstract-input'
NgxBootstrapIconsModule,
],
})
export class SelectComponent extends AbstractInputComponent<number | string> {
export class SelectComponent extends AbstractInputComponent<number> {
constructor() {
super()
this.addItemRef = this.addItem.bind(this)
@@ -100,9 +100,6 @@ export class SelectComponent extends AbstractInputComponent<number | string> {
@Input()
bindLabel: string = 'name'
@Input()
iconField: string
public searchFn = (term: string, item: any): boolean =>
matchesSearchText(item?.[this.bindLabel], term)
@@ -17,10 +17,6 @@ const permissions = [
'view_document',
'change_document',
'delete_document',
'add_sharelinkbundle',
'view_sharelinkbundle',
'change_sharelinkbundle',
'delete_sharelinkbundle',
'change_tag',
'view_documenttype',
]
@@ -79,7 +75,6 @@ describe('PermissionsSelectComponent', () => {
component.ngOnInit()
component.writeValue(permissions)
expect(component.typesWithAllActions).toContain('Document')
expect(component.typesWithAllActions).toContain('ShareLinkBundle')
})
it('should update checkboxes on permissions set', () => {
@@ -90,10 +85,6 @@ describe('PermissionsSelectComponent', () => {
expect(input1.nativeElement.checked).toBeTruthy()
const input2 = fixture.debugElement.query(By.css('input#Tag_Change'))
expect(input2.nativeElement.checked).toBeTruthy()
const bundleInput = fixture.debugElement.query(
By.css('input#ShareLinkBundle_Add')
)
expect(bundleInput.nativeElement.checked).toBeTruthy()
})
it('disable checkboxes when permissions are inherited', () => {
@@ -1,7 +1,6 @@
<pngx-widget-frame
*pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.Document }"
[title]="savedView.name"
[titleIcon]="savedView.icon || 'funnel'"
[loading]="false"
[draggable]="savedView"
>
@@ -8,12 +8,7 @@
<i-bs name="grip-vertical"></i-bs>
</div>
}
<h6 class="card-title mb-0">
@if (titleIcon()) {
<i-bs class="me-2" [name]="titleIcon()"></i-bs>
}
{{title()}}
</h6>
<h6 class="card-title mb-0">{{title()}}</h6>
<ng-content select="[title-badge]"></ng-content>
@if (badge() !== null && badge() !== undefined) {
<span class="badge bg-info text-dark ms-2">{{badge()}}</span>
@@ -16,8 +16,6 @@ export class WidgetFrameComponent implements AfterViewInit {
title = input<string>()
titleIcon = input<string>()
draggable = input<any>()
cardless = input(false)
@@ -97,9 +97,7 @@
<div class="dropdown-menu shadow dropdown-menu-right" ngbDropdownMenu>
@if (!list.activeSavedViewId) {
@for (view of savedViewService.allViews; track view) {
<button ngbDropdownItem (click)="loadViewConfig(view.id)">
<i-bs class="me-2" [name]="view.icon || 'funnel'"></i-bs>{{view.name}}
</button>
<button ngbDropdownItem (click)="loadViewConfig(view.id)">{{view.name}}</button>
}
@if (savedViewService.allViews.length > 0) {
<div class="dropdown-divider"></div>
@@ -457,7 +457,6 @@ export class DocumentListComponent
modal.componentInstance.buttonsEnabled.set(false)
let savedView: SavedView = {
name: formValue.name,
icon: formValue.icon,
filter_rules: this.list.filterRules,
sort_reverse: this.list.sortReverse,
sort_field: this.list.sortField,
@@ -6,14 +6,6 @@
</div>
<div class="modal-body">
<pngx-input-text i18n-title title="Name" formControlName="name" [error]="error()?.name" autocomplete="off"></pngx-input-text>
<pngx-input-select
i18n-title
title="Icon"
formControlName="icon"
[items]="savedViewIcons"
iconField="icon"
[error]="error()?.icon">
</pngx-input-select>
<pngx-input-check i18n-title title="Show in sidebar" formControlName="showInSideBar"></pngx-input-check>
<pngx-input-check i18n-title title="Show on dashboard" formControlName="showOnDashboard"></pngx-input-check>
<pngx-permissions-form accordion="true" formControlName="permissions_form"></pngx-permissions-form>
@@ -9,7 +9,6 @@ import { CheckComponent } from '../../common/input/check/check.component'
import { PermissionsFormComponent } from '../../common/input/permissions/permissions-form/permissions-form.component'
import { PermissionsGroupComponent } from '../../common/input/permissions/permissions-group/permissions-group.component'
import { PermissionsUserComponent } from '../../common/input/permissions/permissions-user/permissions-user.component'
import { SelectComponent } from '../../common/input/select/select.component'
import { TextComponent } from '../../common/input/text/text.component'
import { SaveViewConfigDialogComponent } from './save-view-config-dialog.component'
@@ -41,7 +40,6 @@ describe('SaveViewConfigDialogComponent', () => {
ReactiveFormsModule,
SaveViewConfigDialogComponent,
TextComponent,
SelectComponent,
CheckComponent,
PermissionsFormComponent,
PermissionsUserComponent,
@@ -65,7 +63,6 @@ describe('SaveViewConfigDialogComponent', () => {
expect(component.defaultName()).toEqual(name)
expect(result).toEqual({
name,
icon: 'funnel',
showInSideBar: false,
showOnDashboard: false,
})
@@ -97,7 +94,6 @@ describe('SaveViewConfigDialogComponent', () => {
component.save()
expect(result).toEqual({
name,
icon: 'funnel',
showInSideBar: true,
showOnDashboard: true,
})
@@ -117,7 +113,6 @@ describe('SaveViewConfigDialogComponent', () => {
component.save()
expect(result).toEqual({
name: '',
icon: 'funnel',
showInSideBar: false,
showOnDashboard: false,
permissions_form: permissions,
@@ -13,14 +13,9 @@ import {
ReactiveFormsModule,
} from '@angular/forms'
import { NgbActiveModal } from '@ng-bootstrap/ng-bootstrap'
import {
DEFAULT_SAVED_VIEW_ICON,
SAVED_VIEW_ICONS,
} from 'src/app/data/saved-view-icons'
import { User } from 'src/app/data/user'
import { CheckComponent } from '../../common/input/check/check.component'
import { PermissionsFormComponent } from '../../common/input/permissions/permissions-form/permissions-form.component'
import { SelectComponent } from '../../common/input/select/select.component'
import { TextComponent } from '../../common/input/text/text.component'
@Component({
@@ -29,7 +24,6 @@ import { TextComponent } from '../../common/input/text/text.component'
styleUrls: ['./save-view-config-dialog.component.scss'],
imports: [
CheckComponent,
SelectComponent,
TextComponent,
PermissionsFormComponent,
FormsModule,
@@ -47,7 +41,6 @@ export class SaveViewConfigDialogComponent implements OnInit {
public saveClicked = new EventEmitter()
users: User[]
readonly savedViewIcons = SAVED_VIEW_ICONS
setDefaultName(value: string) {
this.defaultName.set(value)
@@ -56,7 +49,6 @@ export class SaveViewConfigDialogComponent implements OnInit {
saveViewConfigForm = new FormGroup({
name: new FormControl(''),
icon: new FormControl(DEFAULT_SAVED_VIEW_ICON),
showInSideBar: new FormControl(false),
showOnDashboard: new FormControl(false),
permissions_form: new FormControl(null),
@@ -73,7 +65,6 @@ export class SaveViewConfigDialogComponent implements OnInit {
const formValue = this.saveViewConfigForm.value
const saveViewConfig = {
name: formValue.name,
icon: formValue.icon,
showInSideBar: formValue.showInSideBar,
showOnDashboard: formValue.showOnDashboard,
}
@@ -7,24 +7,15 @@
</pngx-page-header>
<form [formGroup]="savedViewsForm" (ngSubmit)="save()">
<ul class="list-group mb-3" formGroupName="savedViews">
@for (view of pagedSavedViews(); track view) {
@for (view of savedViews(); track view) {
<li class="list-group-item py-3">
<div [formGroupName]="view.id">
<div class="row">
<div class="col-md">
<div class="col">
<pngx-input-text title="Name" formControlName="name"></pngx-input-text>
</div>
<div class="col-md">
<pngx-input-select
i18n-title
title="Icon"
formControlName="icon"
[items]="savedViewIcons"
iconField="icon">
</pngx-input-select>
</div>
@if (canSaveSettings) {
<div class="col-md">
<div class="col">
<div class="form-check form-switch mt-3">
<input type="checkbox" class="form-check-input" id="show_on_dashboard_{{view.id}}" formControlName="show_on_dashboard">
<label class="form-check-label" for="show_on_dashboard_{{view.id}}" i18n>Show on dashboard</label>
@@ -90,11 +81,6 @@
}
</ul>
<div class="d-flex align-items-center mb-3">
<button type="button" (click)="reset()" class="btn btn-outline-secondary mb-2" [disabled]="(isDirty$ | async) === false" i18n>Cancel</button>
<button type="submit" class="btn btn-primary ms-2 mb-2" [disabled]="(isDirty$ | async) === false" i18n>Save</button>
@if (savedViews()?.length > pageSize) {
<ngb-pagination class="ms-auto" [pageSize]="pageSize" [collectionSize]="savedViews().length" [page]="page()" [maxSize]="5" (pageChange)="page.set($event)" size="sm" aria-label="Pagination"></ngb-pagination>
}
</div>
<button type="button" (click)="reset()" class="btn btn-outline-secondary mb-2" [disabled]="(isDirty$ | async) === false" i18n>Cancel</button>
<button type="submit" class="btn btn-primary ms-2 mb-2" [disabled]="(isDirty$ | async) === false" i18n>Save</button>
</form>
@@ -4,7 +4,6 @@ import { provideHttpClientTesting } from '@angular/common/http/testing'
import { signal } from '@angular/core'
import { ComponentFixture, TestBed } from '@angular/core/testing'
import { FormsModule, ReactiveFormsModule } from '@angular/forms'
import { By } from '@angular/platform-browser'
import { NgbModal, NgbModule } from '@ng-bootstrap/ng-bootstrap'
import { NgxBootstrapIconsModule, allIcons } from 'ngx-bootstrap-icons'
import { Subject, of, throwError } from 'rxjs'
@@ -26,20 +25,8 @@ import { PageHeaderComponent } from '../../common/page-header/page-header.compon
import { SavedViewsComponent } from './saved-views.component'
const savedViews = [
{
id: 1,
name: 'view1',
icon: 'archive',
show_in_sidebar: true,
show_on_dashboard: true,
},
{
id: 2,
name: 'view2',
icon: 'funnel',
show_in_sidebar: false,
show_on_dashboard: false,
},
{ id: 1, name: 'view1', show_in_sidebar: true, show_on_dashboard: true },
{ id: 2, name: 'view2', show_in_sidebar: false, show_on_dashboard: false },
]
describe('SavedViewsComponent', () => {
@@ -170,24 +157,6 @@ describe('SavedViewsComponent', () => {
expect(patchBody.show_in_sidebar).toBeUndefined()
})
it('should persist a changed icon', () => {
const patchSpy = jest.spyOn(savedViewService, 'patchMany')
const view = savedViews[0]
const iconControl = component.savedViewsForm
.get('savedViews')
.get(view.id.toString())
.get('icon')
iconControl.setValue('bell')
iconControl.markAsDirty()
component.save()
expect(patchSpy.mock.calls[0][0][0]).toMatchObject({
id: view.id,
icon: 'bell',
})
})
it('should persist visibility changes to user settings', () => {
const patchSpy = jest.spyOn(savedViewService, 'patchMany')
const updateVisibilitySpy = jest
@@ -253,44 +222,6 @@ describe('SavedViewsComponent', () => {
).toEqual(view.show_on_dashboard)
})
it('should page saved views, clamp the page if views are removed', () => {
const manyViews = Array.from({ length: 30 }, (_, i) => ({
id: i + 1,
name: `view${i + 1}`,
})) as SavedView[]
const listSpy = jest.spyOn(savedViewService, 'list').mockReturnValue(
of({
all: manyViews.map((v) => v.id),
count: manyViews.length,
results: manyViews.concat([]),
})
)
component.ngOnInit()
fixture.detectChanges()
expect(listSpy).toHaveBeenCalledWith(1, 100000, null, false, {
full_perms: true,
})
expect(component.pagedSavedViews()).toHaveLength(25)
expect(fixture.debugElement.query(By.css('ngb-pagination'))).not.toBeNull()
// all views have controls, not just the current page
expect(
Object.keys(component.savedViewsForm.get('savedViews').value)
).toHaveLength(30)
component.page.set(2)
expect(component.pagedSavedViews()).toHaveLength(5)
listSpy.mockReturnValue(
of({
all: manyViews.slice(0, 25).map((v) => v.id),
count: 25,
results: manyViews.slice(0, 25),
})
)
component.ngOnInit()
expect(component.page()).toEqual(1)
})
it('should support editing permissions', () => {
const confirmClicked = new Subject<any>()
const modalRef = {
@@ -1,29 +1,18 @@
import { AsyncPipe } from '@angular/common'
import {
Component,
OnDestroy,
OnInit,
computed,
inject,
signal,
} from '@angular/core'
import { Component, OnDestroy, OnInit, inject, signal } from '@angular/core'
import {
FormControl,
FormGroup,
FormsModule,
ReactiveFormsModule,
} from '@angular/forms'
import { NgbModal, NgbPaginationModule } from '@ng-bootstrap/ng-bootstrap'
import { NgbModal } from '@ng-bootstrap/ng-bootstrap'
import { dirtyCheck } from '@ngneat/dirty-check-forms'
import { NgxBootstrapIconsModule } from 'ngx-bootstrap-icons'
import { BehaviorSubject, Observable, of, switchMap, takeUntil } from 'rxjs'
import { PermissionsDialogComponent } from 'src/app/components/common/permissions-dialog/permissions-dialog.component'
import { DisplayMode } from 'src/app/data/document'
import { SavedView } from 'src/app/data/saved-view'
import {
DEFAULT_SAVED_VIEW_ICON,
SAVED_VIEW_ICONS,
} from 'src/app/data/saved-view-icons'
import { IfPermissionsDirective } from 'src/app/directives/if-permissions.directive'
import {
PermissionAction,
@@ -36,7 +25,6 @@ import { ToastService } from 'src/app/services/toast.service'
import { ConfirmButtonComponent } from '../../common/confirm-button/confirm-button.component'
import { DragDropSelectComponent } from '../../common/input/drag-drop-select/drag-drop-select.component'
import { NumberComponent } from '../../common/input/number/number.component'
import { SelectComponent } from '../../common/input/select/select.component'
import { TextComponent } from '../../common/input/text/text.component'
import { PageHeaderComponent } from '../../common/page-header/page-header.component'
import { LoadingComponentWithPermissions } from '../../loading-component/loading.component'
@@ -48,14 +36,12 @@ import { LoadingComponentWithPermissions } from '../../loading-component/loading
PageHeaderComponent,
ConfirmButtonComponent,
NumberComponent,
SelectComponent,
TextComponent,
IfPermissionsDirective,
DragDropSelectComponent,
FormsModule,
ReactiveFormsModule,
AsyncPipe,
NgbPaginationModule,
NgxBootstrapIconsModule,
],
})
@@ -70,17 +56,8 @@ export class SavedViewsComponent
private readonly modalService = inject(NgbModal)
DisplayMode = DisplayMode
readonly savedViewIcons = SAVED_VIEW_ICONS
readonly savedViews = signal<SavedView[]>(undefined)
readonly page = signal(1)
public readonly pageSize = 25
// All views are loaded at init, so paging is only for display
readonly pagedSavedViews = computed(() => {
const start = (this.page() - 1) * this.pageSize
return this.savedViews()?.slice(start, start + this.pageSize)
})
private savedViewsGroup = new FormGroup({})
public savedViewsForm: FormGroup = new FormGroup({
savedViews: this.savedViewsGroup,
@@ -107,11 +84,9 @@ export class SavedViewsComponent
private reloadViews(): void {
this.loading.set(true)
this.savedViewService
.list(1, 100000, null, false, { full_perms: true })
.list(null, null, null, false, { full_perms: true })
.subscribe((r) => {
this.savedViews.set(r.results)
const pageCount = Math.ceil(r.results.length / this.pageSize)
this.page.update((page) => Math.min(page, Math.max(1, pageCount)))
this.initialize()
})
}
@@ -135,7 +110,6 @@ export class SavedViewsComponent
storeData.savedViews[view.id.toString()] = {
id: view.id,
name: view.name,
icon: view.icon ?? DEFAULT_SAVED_VIEW_ICON,
show_on_dashboard: view.show_on_dashboard,
show_in_sidebar: view.show_in_sidebar,
page_size: view.page_size,
@@ -148,7 +122,6 @@ export class SavedViewsComponent
new FormGroup({
id: new FormControl({ value: null, disabled: !canEdit }),
name: new FormControl({ value: null, disabled: !canEdit }),
icon: new FormControl({ value: null, disabled: !canEdit }),
show_on_dashboard: new FormControl({
value: null,
disabled: false,
@@ -227,7 +200,6 @@ export class SavedViewsComponent
const modelFieldsChanged =
group.get('name')?.dirty ||
group.get('icon')?.dirty ||
group.get('page_size')?.dirty ||
group.get('display_mode')?.dirty ||
group.get('display_fields')?.dirty
-55
View File
@@ -54,10 +54,6 @@ export const ConfigCategory = {
AI: $localize`AI Settings`,
}
export const ConfigSection = {
RemoteOCR: $localize`Remote OCR`,
}
export const LLMEmbeddingBackendConfig = {
OPENAI_LIKE: 'openai-like',
HUGGINGFACE: 'huggingface',
@@ -69,15 +65,6 @@ export const LLMBackendConfig = {
OLLAMA: 'ollama',
}
export const RemoteOCREngineConfig = {
AZURE_AI: 'azureai',
}
export const RemoteOCRModeConfig = {
ALWAYS: 'always',
WORKFLOW_ONLY: 'workflow_only',
}
export interface ConfigOption {
key: string
title: string
@@ -85,7 +72,6 @@ export interface ConfigOption {
choices?: Array<{ id: string; name: string }>
config_key?: string
category: string
section?: string
note?: string
}
@@ -195,43 +181,6 @@ export const PaperlessConfigOptions: ConfigOption[] = [
config_key: 'PAPERLESS_OCR_USER_ARGS',
category: ConfigCategory.OCR,
},
{
key: 'remote_ocr_engine',
title: $localize`Remote OCR Engine`,
type: ConfigOptionType.Select,
choices: mapToItems(RemoteOCREngineConfig),
config_key: 'PAPERLESS_REMOTE_OCR_ENGINE',
category: ConfigCategory.OCR,
section: ConfigSection.RemoteOCR,
note: $localize`Enabling remote OCR sends documents to a third-party service for processing. Consider the privacy implications as well as potential costs before enabling.`,
},
{
key: 'remote_ocr_api_key',
title: $localize`Remote OCR API Key`,
type: ConfigOptionType.Password,
config_key: 'PAPERLESS_REMOTE_OCR_API_KEY',
category: ConfigCategory.OCR,
section: ConfigSection.RemoteOCR,
},
{
key: 'remote_ocr_endpoint',
title: $localize`Remote OCR Endpoint`,
type: ConfigOptionType.String,
config_key: 'PAPERLESS_REMOTE_OCR_ENDPOINT',
category: ConfigCategory.OCR,
section: ConfigSection.RemoteOCR,
note: $localize`Required when using the Azure AI engine.`,
},
{
key: 'remote_ocr_mode',
title: $localize`Remote OCR Mode`,
type: ConfigOptionType.Select,
choices: mapToItems(RemoteOCRModeConfig),
config_key: 'PAPERLESS_REMOTE_OCR_MODE',
category: ConfigCategory.OCR,
section: ConfigSection.RemoteOCR,
note: $localize`Which documents are sent to the remote engine. Use 'workflow_only' to keep remote OCR off unless a workflow enables it for a document.`,
},
{
key: 'app_logo',
title: $localize`Application Logo`,
@@ -449,10 +398,6 @@ export interface PaperlessConfig extends ObjectWithId {
barcode_enable_tag: boolean
barcode_tag_mapping: object
barcode_tag_split: boolean
remote_ocr_engine: string
remote_ocr_api_key: string
remote_ocr_endpoint: string
remote_ocr_mode: string
ai_enabled: boolean
llm_embedding_backend: string
llm_embedding_model: string
-89
View File
@@ -1,89 +0,0 @@
export const DEFAULT_SAVED_VIEW_ICON = 'funnel'
export const SAVED_VIEW_ICONS = [
{ id: 'archive', name: $localize`Archive`, icon: 'archive' },
{ id: 'bank', name: $localize`Bank`, icon: 'bank' },
{ id: 'basket', name: $localize`Basket`, icon: 'basket' },
{ id: 'bell', name: $localize`Bell`, icon: 'bell' },
{ id: 'bookmark', name: $localize`Bookmark`, icon: 'bookmark' },
{ id: 'boxes', name: $localize`Boxes`, icon: 'boxes' },
{ id: 'briefcase', name: $localize`Briefcase`, icon: 'briefcase' },
{ id: 'building', name: $localize`Building`, icon: 'building' },
{ id: 'calculator', name: $localize`Calculator`, icon: 'calculator' },
{ id: 'calendar', name: $localize`Calendar`, icon: 'calendar' },
{ id: 'camera', name: $localize`Camera`, icon: 'camera' },
{
id: 'card-checklist',
name: $localize`Checklist`,
icon: 'card-checklist',
},
{ id: 'cash', name: $localize`Cash`, icon: 'cash' },
{ id: 'chat-left-text', name: $localize`Chat`, icon: 'chat-left-text' },
{ id: 'check-circle', name: $localize`Check`, icon: 'check-circle' },
{ id: 'clipboard', name: $localize`Clipboard`, icon: 'clipboard' },
{ id: 'clock-history', name: $localize`Clock`, icon: 'clock-history' },
{ id: 'credit-card', name: $localize`Credit card`, icon: 'credit-card' },
{ id: 'download', name: $localize`Download`, icon: 'download' },
{ id: 'envelope', name: $localize`Envelope`, icon: 'envelope' },
{
id: 'exclamation-triangle',
name: $localize`Warning`,
icon: 'exclamation-triangle',
},
{ id: 'file-earmark', name: $localize`File`, icon: 'file-earmark' },
{
id: 'file-earmark-check',
name: $localize`Checked file`,
icon: 'file-earmark-check',
},
{
id: 'file-earmark-lock',
name: $localize`Locked file`,
icon: 'file-earmark-lock',
},
{
id: 'file-earmark-medical',
name: $localize`Medical file`,
icon: 'file-earmark-medical',
},
{
id: 'file-earmark-person',
name: $localize`Person file`,
icon: 'file-earmark-person',
},
{
id: 'file-earmark-spreadsheet',
name: $localize`Spreadsheet`,
icon: 'file-earmark-spreadsheet',
},
{ id: 'file-text', name: $localize`Text file`, icon: 'file-text' },
{ id: 'files', name: $localize`Files`, icon: 'files' },
{ id: 'folder', name: $localize`Folder`, icon: 'folder' },
{ id: 'funnel', name: $localize`Filter`, icon: 'funnel' },
{ id: 'gear', name: $localize`Gear`, icon: 'gear' },
{ id: 'globe2', name: $localize`Globe`, icon: 'globe2' },
{ id: 'hash', name: $localize`Hash`, icon: 'hash' },
{ id: 'heart', name: $localize`Heart`, icon: 'heart' },
{ id: 'house', name: $localize`House`, icon: 'house' },
{ id: 'inbox', name: $localize`Inbox`, icon: 'inbox' },
{ id: 'journals', name: $localize`Journals`, icon: 'journals' },
{ id: 'list-task', name: $localize`Task list`, icon: 'list-task' },
{ id: 'newspaper', name: $localize`Newspaper`, icon: 'newspaper' },
{ id: 'paperclip', name: $localize`Attachment`, icon: 'paperclip' },
{ id: 'people', name: $localize`People`, icon: 'people' },
{ id: 'person', name: $localize`Person`, icon: 'person' },
{ id: 'printer', name: $localize`Printer`, icon: 'printer' },
{ id: 'receipt', name: $localize`Receipt`, icon: 'receipt' },
{ id: 'safe', name: $localize`Safe`, icon: 'safe' },
{ id: 'search', name: $localize`Search`, icon: 'search' },
{ id: 'send', name: $localize`Send`, icon: 'send' },
{ id: 'shop', name: $localize`Shop`, icon: 'shop' },
{ id: 'stack', name: $localize`Stack`, icon: 'stack' },
{ id: 'stars', name: $localize`Stars`, icon: 'stars' },
{ id: 'tag', name: $localize`Tag`, icon: 'tag' },
{ id: 'tags', name: $localize`Tags`, icon: 'tags' },
{ id: 'telephone', name: $localize`Telephone`, icon: 'telephone' },
{ id: 'truck', name: $localize`Truck`, icon: 'truck' },
{ id: 'upc-scan', name: $localize`Barcode`, icon: 'upc-scan' },
{ id: 'wallet2', name: $localize`Wallet`, icon: 'wallet2' },
]
-2
View File
@@ -5,8 +5,6 @@ import { ObjectWithPermissions } from './object-with-permissions'
export interface SavedView extends ObjectWithPermissions {
name?: string
icon?: string
show_on_dashboard?: boolean
show_in_sidebar?: boolean
@@ -120,12 +120,6 @@ describe('PermissionsService', () => {
actionKey: 'View', // PermissionAction.View
typeKey: 'SystemMonitoring', // PermissionType.SystemMonitoring
})
expect(permissionsService.getPermissionKeys('add_sharelinkbundle')).toEqual(
{
actionKey: 'Add', // PermissionAction.Add
typeKey: 'ShareLinkBundle', // PermissionType.ShareLinkBundle
}
)
})
it('correctly checks explicit global permissions', () => {
@@ -275,10 +269,6 @@ describe('PermissionsService', () => {
'view_sharelink',
'change_sharelink',
'delete_sharelink',
'add_sharelinkbundle',
'view_sharelinkbundle',
'change_sharelinkbundle',
'delete_sharelinkbundle',
'add_workflow',
'view_workflow',
'change_workflow',
@@ -26,7 +26,6 @@ export enum PermissionType {
User = '%s_user',
Group = '%s_group',
ShareLink = '%s_sharelink',
ShareLinkBundle = '%s_sharelinkbundle',
CustomField = '%s_customfield',
Workflow = '%s_workflow',
ProcessedMail = '%s_processedmail',
-46
View File
@@ -35,27 +35,19 @@ import {
arrowRightShort,
arrowUpRight,
asterisk,
bank,
basket,
bell,
bodyText,
bookmark,
boxArrowUp,
boxArrowUpRight,
boxes,
braces,
briefcase,
building,
calculator,
calendar,
calendarEvent,
calendarEventFill,
camera,
cardChecklist,
cardHeading,
caretDown,
caretUp,
cash,
chatLeftText,
chatSquareDots,
check,
@@ -73,7 +65,6 @@ import {
clipboardCheckFill,
clipboardFill,
clockHistory,
creditCard,
dash,
dashCircle,
diagram3,
@@ -92,12 +83,9 @@ import {
fileEarmarkDiff,
fileEarmarkFill,
fileEarmarkLock,
fileEarmarkMedical,
fileEarmarkMinus,
fileEarmarkPerson,
fileEarmarkPlus,
fileEarmarkRichtext,
fileEarmarkSpreadsheet,
fileText,
files,
filter,
@@ -105,15 +93,12 @@ import {
folderFill,
funnel,
gear,
globe2,
google,
grid,
gripVertical,
hash,
hddStack,
heart,
house,
inbox,
infoCircle,
journals,
link,
@@ -121,9 +106,7 @@ import {
listTask,
listUl,
microsoft,
newspaper,
nodePlus,
paperclip,
pencil,
people,
peopleFill,
@@ -138,12 +121,9 @@ import {
plusCircle,
printer,
questionCircle,
receipt,
safe,
scissors,
search,
send,
shop,
slashCircle,
sliders2Vertical,
sortAlphaDown,
@@ -153,17 +133,14 @@ import {
tag,
tagFill,
tags,
telephone,
textIndentLeft,
textLeft,
threeDots,
threeDotsVertical,
trash,
truck,
uiRadios,
unlock,
upcScan,
wallet2,
windowStack,
x,
xCircle,
@@ -281,22 +258,15 @@ const icons = {
arrowRightShort,
arrowUpRight,
asterisk,
bank,
basket,
bell,
braces,
bodyText,
bookmark,
boxArrowUp,
boxArrowUpRight,
boxes,
briefcase,
building,
calculator,
calendar,
calendarEvent,
calendarEventFill,
camera,
cardChecklist,
cardHeading,
caretDown,
@@ -318,8 +288,6 @@ const icons = {
clipboardCheckFill,
clipboardFill,
clockHistory,
cash,
creditCard,
dash,
dashCircle,
diagram3,
@@ -338,12 +306,9 @@ const icons = {
fileEarmarkDiff,
fileEarmarkFill,
fileEarmarkLock,
fileEarmarkMedical,
fileEarmarkMinus,
fileEarmarkPerson,
fileEarmarkPlus,
fileEarmarkRichtext,
fileEarmarkSpreadsheet,
files,
fileText,
filter,
@@ -351,15 +316,12 @@ const icons = {
folderFill,
funnel,
gear,
globe2,
google,
grid,
gripVertical,
hash,
hddStack,
heart,
house,
inbox,
infoCircle,
journals,
link,
@@ -367,10 +329,8 @@ const icons = {
listTask,
listUl,
microsoft,
newspaper,
nodePlus,
pencil,
paperclip,
people,
peopleFill,
person,
@@ -384,13 +344,10 @@ const icons = {
plusCircle,
printer,
questionCircle,
receipt,
safe,
scissors,
search,
send,
slashCircle,
shop,
sliders2Vertical,
sortAlphaDown,
sortAlphaUpAlt,
@@ -401,15 +358,12 @@ const icons = {
tags,
textIndentLeft,
textLeft,
telephone,
threeDots,
threeDotsVertical,
trash,
truck,
uiRadios,
unlock,
upcScan,
wallet2,
windowStack,
x,
xCircle,
-8
View File
@@ -53,7 +53,6 @@ from documents.utils import copy_basic_file_stats
from documents.utils import copy_file_with_basic_stats
from documents.utils import run_subprocess
from paperless.config import OcrConfig
from paperless.config import RemoteOCRConfig
from paperless.models import ArchiveFileGenerationChoices
from paperless.parsers import ParserContext
from paperless.parsers import ParserProtocol
@@ -452,19 +451,12 @@ class ConsumerPlugin(
except Exception as e:
self.log.error(f"Error attempting to clean PDF: {e}")
# Workflows have already run at this point, so the metadata knows
# whether this document was singled out for remote OCR
allow_remote = (
self.metadata.remote_ocr or RemoteOCRConfig().remote_ocr_by_default
)
# Based on the mime type, get the parser for that type
parser_class: type[ParserProtocol] | None = (
get_parser_registry().get_parser_for_file(
mime_type,
self.filename,
self.working_copy,
allow_remote=allow_remote,
)
)
if not parser_class:
-3
View File
@@ -34,7 +34,6 @@ class DocumentMetadataOverrides:
skip_asn_if_exists: bool = False
version_label: str | None = None
actor_id: int | None = None
remote_ocr: bool = False
def update(self, other: "DocumentMetadataOverrides") -> "DocumentMetadataOverrides":
"""
@@ -58,8 +57,6 @@ class DocumentMetadataOverrides:
self.actor_id = other.actor_id
if other.skip_asn_if_exists:
self.skip_asn_if_exists = True
if other.remote_ocr:
self.remote_ocr = True
if other.version_label is not None:
self.version_label = other.version_label
@@ -55,7 +55,7 @@ class Command(PaperlessCommand):
"--ratio",
default=85.0,
type=float,
help="Ratio to consider documents a match (0.0 - 100.0)",
help="Ratio to consider documents a match",
)
parser.add_argument(
"--delete",
@@ -69,17 +69,6 @@ class Command(PaperlessCommand):
action="store_true",
help="Skip the confirmation prompt when used with --delete",
)
parser.add_argument(
"--url",
default=None,
type=str,
help=(
"Base URL of the Paperless instance (e.g. "
"http://localhost:8000 or https://paperless.local). If set, matched "
"documents are shown as clickable (usually ctrl+click) links to "
"<url>/documents/<id>/details instead of by title."
),
)
def _render_results(
self,
@@ -87,7 +76,6 @@ class Command(PaperlessCommand):
*,
opt_ratio: float,
do_delete: bool,
base_url: str | None = None,
) -> list[int]:
"""Render match results as a Rich table. Returns list of PKs to delete."""
if not matches:
@@ -100,22 +88,13 @@ class Command(PaperlessCommand):
)
return []
# Fetch titles for matched documents in a single query, unless we're
# going to show URLs instead.
titles: dict[int, str] = {}
if not base_url:
all_pks = {pk for m in matches for pk in (m.doc_one_pk, m.doc_two_pk)}
titles = dict(
Document.objects.filter(pk__in=all_pks)
.only("pk", "title")
.values_list("pk", "title"),
)
def _cell(pk: int) -> str:
if base_url:
doc_url = f"{base_url.rstrip('/')}/documents/{pk}/details"
return f"[link={doc_url}]{doc_url}[/link]"
return f"[dim]#{pk}[/dim] {titles.get(pk, 'Unknown')}"
# Fetch titles for matched documents in a single query.
all_pks = {pk for m in matches for pk in (m.doc_one_pk, m.doc_two_pk)}
titles: dict[int, str] = dict(
Document.objects.filter(pk__in=all_pks)
.only("pk", "title")
.values_list("pk", "title"),
)
table = Table(
title=f"Fuzzy Matches (threshold: {opt_ratio:.1f}%)",
@@ -145,8 +124,8 @@ class Command(PaperlessCommand):
table.add_row(
str(i),
_cell(pk_a),
_cell(pk_b),
f"[dim]#{pk_a}[/dim] {titles.get(pk_a, 'Unknown')}",
f"[dim]#{pk_b}[/dim] {titles.get(pk_b, 'Unknown')}",
Text(f"{ratio:.1f}%", style=ratio_style),
)
maybe_delete_ids.append(pk_b)
@@ -229,7 +208,6 @@ class Command(PaperlessCommand):
matches,
opt_ratio=opt_ratio,
do_delete=options["delete"],
base_url=options["url"],
)
if options["delete"] and maybe_delete_ids:
@@ -1,79 +0,0 @@
from django.db import migrations
from django.db import models
class Migration(migrations.Migration):
dependencies = [
("documents", "0022_add_perf_indexes"),
]
operations = [
migrations.AddField(
model_name="savedview",
name="icon",
field=models.CharField(
choices=[
("archive", "Archive"),
("bank", "Bank"),
("basket", "Basket"),
("bell", "Bell"),
("bookmark", "Bookmark"),
("boxes", "Boxes"),
("briefcase", "Briefcase"),
("building", "Building"),
("calculator", "Calculator"),
("calendar", "Calendar"),
("camera", "Camera"),
("card-checklist", "Checklist"),
("cash", "Cash"),
("chat-left-text", "Chat"),
("check-circle", "Check"),
("clipboard", "Clipboard"),
("clock-history", "Clock"),
("credit-card", "Credit card"),
("download", "Download"),
("envelope", "Envelope"),
("exclamation-triangle", "Warning"),
("file-earmark", "File"),
("file-earmark-check", "Checked file"),
("file-earmark-lock", "Locked file"),
("file-earmark-medical", "Medical file"),
("file-earmark-person", "Person file"),
("file-earmark-spreadsheet", "Spreadsheet"),
("file-text", "Text file"),
("files", "Files"),
("folder", "Folder"),
("funnel", "Filter"),
("gear", "Gear"),
("globe2", "Globe"),
("hash", "Hash"),
("heart", "Heart"),
("house", "House"),
("inbox", "Inbox"),
("journals", "Journals"),
("list-task", "Task list"),
("newspaper", "Newspaper"),
("paperclip", "Attachment"),
("people", "People"),
("person", "Person"),
("printer", "Printer"),
("receipt", "Receipt"),
("safe", "Safe"),
("search", "Search"),
("send", "Send"),
("shop", "Shop"),
("stack", "Stack"),
("stars", "Stars"),
("tag", "Tag"),
("tags", "Tags"),
("telephone", "Telephone"),
("truck", "Truck"),
("upc-scan", "Barcode"),
("wallet2", "Wallet"),
],
default="funnel",
max_length=64,
verbose_name="icon",
),
),
]
-69
View File
@@ -519,68 +519,6 @@ class Document(SoftDeleteModel, ModelWithOwner): # type: ignore[django-manager-
class SavedView(ModelWithOwner):
class Icon(models.TextChoices):
ARCHIVE = ("archive", _("Archive"))
BANK = ("bank", _("Bank"))
BASKET = ("basket", _("Basket"))
BELL = ("bell", _("Bell"))
BOOKMARK = ("bookmark", _("Bookmark"))
BOXES = ("boxes", _("Boxes"))
BRIEFCASE = ("briefcase", _("Briefcase"))
BUILDING = ("building", _("Building"))
CALCULATOR = ("calculator", _("Calculator"))
CALENDAR = ("calendar", _("Calendar"))
CAMERA = ("camera", _("Camera"))
CARD_CHECKLIST = ("card-checklist", _("Checklist"))
CASH = ("cash", _("Cash"))
CHAT_LEFT_TEXT = ("chat-left-text", _("Chat"))
CHECK_CIRCLE = ("check-circle", _("Check"))
CLIPBOARD = ("clipboard", _("Clipboard"))
CLOCK_HISTORY = ("clock-history", _("Clock"))
CREDIT_CARD = ("credit-card", _("Credit card"))
DOWNLOAD = ("download", _("Download"))
ENVELOPE = ("envelope", _("Envelope"))
EXCLAMATION_TRIANGLE = ("exclamation-triangle", _("Warning"))
FILE_EARMARK = ("file-earmark", _("File"))
FILE_EARMARK_CHECK = ("file-earmark-check", _("Checked file"))
FILE_EARMARK_LOCK = ("file-earmark-lock", _("Locked file"))
FILE_EARMARK_MEDICAL = ("file-earmark-medical", _("Medical file"))
FILE_EARMARK_PERSON = ("file-earmark-person", _("Person file"))
FILE_EARMARK_SPREADSHEET = (
"file-earmark-spreadsheet",
_("Spreadsheet"),
)
FILE_TEXT = ("file-text", _("Text file"))
FILES = ("files", _("Files"))
FOLDER = ("folder", _("Folder"))
FUNNEL = ("funnel", _("Filter"))
GEAR = ("gear", _("Gear"))
GLOBE = ("globe2", _("Globe"))
HASH = ("hash", _("Hash"))
HEART = ("heart", _("Heart"))
HOUSE = ("house", _("House"))
INBOX = ("inbox", _("Inbox"))
JOURNALS = ("journals", _("Journals"))
LIST_TASK = ("list-task", _("Task list"))
NEWSPAPER = ("newspaper", _("Newspaper"))
PAPERCLIP = ("paperclip", _("Attachment"))
PEOPLE = ("people", _("People"))
PERSON = ("person", _("Person"))
PRINTER = ("printer", _("Printer"))
RECEIPT = ("receipt", _("Receipt"))
SAFE = ("safe", _("Safe"))
SEARCH = ("search", _("Search"))
SEND = ("send", _("Send"))
SHOP = ("shop", _("Shop"))
STACK = ("stack", _("Stack"))
STARS = ("stars", _("Stars"))
TAG = ("tag", _("Tag"))
TAGS = ("tags", _("Tags"))
TELEPHONE = ("telephone", _("Telephone"))
TRUCK = ("truck", _("Truck"))
UPC_SCAN = ("upc-scan", _("Barcode"))
WALLET = ("wallet2", _("Wallet"))
class DisplayMode(models.TextChoices):
TABLE = ("table", _("Table"))
SMALL_CARDS = ("smallCards", _("Small Cards"))
@@ -603,13 +541,6 @@ class SavedView(ModelWithOwner):
name = models.CharField(_("name"), max_length=128)
icon = models.CharField(
_("icon"),
max_length=64,
choices=Icon.choices,
default=Icon.FUNNEL,
)
sort_field = models.CharField(
_("sort field"),
max_length=128,
-8
View File
@@ -1383,7 +1383,6 @@ class SavedViewSerializer(OwnedObjectSerializer):
fields = [
"id",
"name",
"icon",
"sort_field",
"sort_reverse",
"filter_rules",
@@ -3214,13 +3213,6 @@ class WorkflowActionSerializer(serializers.ModelSerializer[WorkflowAction]):
{"assign_title": f'Invalid f-string detected: "{e.args[0]}"'},
)
if attrs.get("assign_custom_fields_values"):
# Empty strings treated as None to avoid unexpected behavior
attrs["assign_custom_fields_values"] = {
field_id: (None if value == "" else value)
for field_id, value in attrs["assign_custom_fields_values"].items()
}
if (
"type" in attrs
and attrs["type"] == WorkflowAction.WorkflowActionType.EMAIL
+1 -10
View File
@@ -66,7 +66,6 @@ from documents.utils import compute_checksum
from documents.utils import identity
from documents.workflows.utils import get_workflows_for_trigger
from paperless.config import AIConfig
from paperless.config import RemoteOCRConfig
from paperless.logging import consume_task_id
from paperless.parsers import ParserContext
from paperless.parsers.registry import get_parser_registry
@@ -338,17 +337,10 @@ def bulk_update_documents(document_ids) -> None:
@shared_task
def update_document_content_maybe_archive_file(
document_id,
*,
remote_ocr: bool = False,
) -> None:
def update_document_content_maybe_archive_file(document_id) -> None:
"""
Re-creates OCR content and thumbnail for a document, and archive file if
it exists.
Remote OCR is used only when the engine is configured to handle everything
or if explicitly asked for via ``remote_ocr``.
"""
document = Document.objects.get(id=document_id)
@@ -358,7 +350,6 @@ def update_document_content_maybe_archive_file(
mime_type,
document.original_filename or "",
document.source_path,
allow_remote=remote_ocr or RemoteOCRConfig().remote_ocr_by_default,
)
if not parser_class:
@@ -72,10 +72,6 @@ class TestApiAppConfig(DirectoriesMixin, APITestCase):
"barcode_enable_tag": None,
"barcode_tag_mapping": None,
"barcode_tag_split": None,
"remote_ocr_engine": None,
"remote_ocr_api_key": None,
"remote_ocr_endpoint": None,
"remote_ocr_mode": None,
"ai_enabled": False,
"llm_embedding_backend": None,
"llm_embedding_model": None,
@@ -874,49 +870,6 @@ class TestApiAppConfig(DirectoriesMixin, APITestCase):
config.refresh_from_db()
self.assertEqual(config.llm_api_key, None)
def test_update_remote_ocr_api_key(self) -> None:
"""
GIVEN:
- Existing config with remote_ocr_api_key specified
WHEN:
- API to update remote_ocr_api_key is called with all *s
- API to update remote_ocr_api_key is called with empty string
THEN:
- remote_ocr_api_key is unchanged
- remote_ocr_api_key is set to None
"""
config = ApplicationConfiguration.objects.first()
assert config is not None
config.remote_ocr_api_key = "1234567890"
config.save()
# Test with all *
response = self.client.patch(
f"{self.ENDPOINT}1/",
json.dumps(
{
"remote_ocr_api_key": "*" * 32,
},
),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_200_OK)
config.refresh_from_db()
self.assertEqual(config.remote_ocr_api_key, "1234567890")
# Test with empty string
response = self.client.patch(
f"{self.ENDPOINT}1/",
json.dumps(
{
"remote_ocr_api_key": "",
},
),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_200_OK)
config.refresh_from_db()
self.assertEqual(config.remote_ocr_api_key, None)
def test_enable_ai_index_triggers_update(self) -> None:
"""
GIVEN:
+1 -10
View File
@@ -2905,20 +2905,18 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
v1 = SavedView.objects.get(name="test")
self.assertEqual(v1.sort_field, "created2")
self.assertEqual(v1.icon, SavedView.Icon.FUNNEL)
self.assertEqual(v1.filter_rules.count(), 1)
self.assertEqual(v1.owner, self.user)
response = self.client.patch(
f"/api/saved_views/{v1.id}/",
{"sort_reverse": True, "icon": SavedView.Icon.RECEIPT},
{"sort_reverse": True},
format="json",
)
v1 = SavedView.objects.get(id=v1.id)
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertTrue(v1.sort_reverse)
self.assertEqual(v1.icon, SavedView.Icon.RECEIPT)
self.assertEqual(v1.filter_rules.count(), 1)
view["filter_rules"] = [{"rule_type": 12, "value": "secret"}]
@@ -2938,13 +2936,6 @@ class TestDocumentApi(DirectoriesMixin, ConsumeTaskMixin, APITestCase):
v1 = SavedView.objects.get(id=v1.id)
self.assertEqual(v1.filter_rules.count(), 0)
response = self.client.patch(
f"/api/saved_views/{v1.id}/",
{"icon": "not-an-icon"},
format="json",
)
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
def test_saved_view_display_options(self) -> None:
"""
GIVEN:
@@ -422,11 +422,6 @@ class TestApiWorkflows(DirectoriesMixin, APITestCase):
json.dumps(
{
"assign_title": "",
"assign_custom_fields": [self.cf1.id, self.cf2.id],
"assign_custom_fields_values": {
str(self.cf1.id): "",
str(self.cf2.id): 0,
},
},
),
content_type="application/json",
@@ -434,10 +429,6 @@ class TestApiWorkflows(DirectoriesMixin, APITestCase):
self.assertEqual(response.status_code, status.HTTP_201_CREATED)
action = WorkflowAction.objects.get(id=response.data["id"])
self.assertIsNone(action.assign_title)
self.assertEqual(
action.assign_custom_fields_values,
{str(self.cf1.id): None, str(self.cf2.id): 0},
)
response = self.client.post(
self.ENDPOINT_TRIGGERS,
-80
View File
@@ -1559,72 +1559,6 @@ class PostConsumeTestCase(DirectoriesMixin, GetConsumerMixin, TestCase):
consumer.run_post_consume_script(doc)
class TestConsumerRemoteOCR(
DirectoriesMixin,
FileSystemAssertsMixin,
GetConsumerMixin,
TestCase,
):
"""
The consumer resolves the remote OCR mode and the per-document request from
workflows into the allow_remote flag it hands to the parser registry.
"""
def setUp(self) -> None:
super().setUp()
patcher = mock.patch("documents.consumer.get_parser_registry")
self.mock_registry = patcher.start()
self.mock_registry.return_value.get_parser_for_file.return_value = DummyParser
self.addCleanup(patcher.stop)
def _consume(self, *, overrides: DocumentMetadataOverrides | None = None) -> bool:
src = (
Path(__file__).parent
/ "samples"
/ "documents"
/ "originals"
/ "0000001.pdf"
)
dst = self.dirs.scratch_dir / "sample.pdf"
shutil.copy(src, dst)
with self.get_consumer(dst, overrides=overrides) as consumer:
consumer.run()
_, kwargs = self.mock_registry.return_value.get_parser_for_file.call_args
return kwargs["allow_remote"]
@override_settings(REMOTE_OCR_MODE="always")
def test_always_mode_allows_remote(self) -> None:
"""
GIVEN: Remote OCR mode is 'always'.
WHEN: A document is consumed without any workflow asking for it.
THEN: The registry is allowed to pick the remote parser.
"""
self.assertTrue(self._consume())
@override_settings(REMOTE_OCR_MODE="workflow_only")
def test_workflow_only_mode_denies_remote_by_default(self) -> None:
"""
GIVEN: Remote OCR mode is 'workflow_only'.
WHEN: A document is consumed and nothing asked for remote OCR.
THEN: The remote parser is excluded.
"""
self.assertFalse(self._consume())
@override_settings(REMOTE_OCR_MODE="workflow_only")
def test_workflow_only_mode_allows_remote_when_requested(self) -> None:
"""
GIVEN: Remote OCR mode is 'workflow_only'.
WHEN: A workflow set remote_ocr on the metadata overrides.
THEN: The registry is allowed to pick the remote parser.
"""
self.assertTrue(
self._consume(overrides=DocumentMetadataOverrides(remote_ocr=True)),
)
class TestMetadataOverrides(TestCase):
def test_update_skip_asn_if_exists(self) -> None:
base = DocumentMetadataOverrides()
@@ -1632,20 +1566,6 @@ class TestMetadataOverrides(TestCase):
base.update(incoming)
self.assertTrue(base.skip_asn_if_exists)
def test_update_remote_ocr(self) -> None:
base = DocumentMetadataOverrides()
base.update(DocumentMetadataOverrides(remote_ocr=True))
self.assertTrue(base.remote_ocr)
def test_update_remote_ocr_is_not_unset(self) -> None:
"""
A later workflow that says nothing must not undo an earlier one that
asked for remote OCR.
"""
base = DocumentMetadataOverrides(remote_ocr=True)
base.update(DocumentMetadataOverrides())
self.assertTrue(base.remote_ocr)
def test_update_actor_and_version_label(self) -> None:
base = DocumentMetadataOverrides(
actor_id=1,
+1 -41
View File
@@ -1,4 +1,3 @@
import os
from io import StringIO
from unittest.mock import patch
@@ -42,7 +41,7 @@ class TestFuzzyMatchCommand(TestCase):
def test_invalid_ratio_upper_limit(self) -> None:
"""
GIVEN:
GIVEN:s
- Invalid ratio above upper
WHEN:
- Command is called
@@ -109,45 +108,6 @@ class TestFuzzyMatchCommand(TestCase):
stdout, _ = self.call_command("--processes", "1")
self.assertIn("Found 1 matching pair(s)", stdout)
def test_with_matches_and_url(self) -> None:
"""
GIVEN:
- 2 documents exist
- Similarity between content is 86.667
- --url is provided
WHEN:
- Command is called with --url
THEN:
- 1 match is returned from doc 1 to doc 2
- No match from doc 2 to doc 1 reported
- Output contains clickable links to the documents instead of titles
"""
# Content similarity is 86.667
Document.objects.create(
checksum="BEEFCAFE",
title="A",
content="first document scanned by bob",
mime_type="application/pdf",
filename="test.pdf",
)
Document.objects.create(
checksum="DEADBEAF",
title="A",
content="first document scanned by alice",
mime_type="application/pdf",
filename="other_test.pdf",
)
with patch.dict(os.environ, {"COLUMNS": "200"}):
stdout, _ = self.call_command(
"--processes",
"1",
"--url",
"http://localhost:8000",
)
self.assertIn("Found 1 matching pair(s)", stdout)
self.assertIn("http://localhost:8000/documents/1/details", stdout)
self.assertIn("http://localhost:8000/documents/2/details", stdout)
def test_with_3_matches(self) -> None:
"""
GIVEN:
-39
View File
@@ -287,45 +287,6 @@ class TestUpdateContent(DirectoriesMixin, TestCase):
self.assertNotEqual(Document.objects.get(pk=doc.pk).content, "test")
class TestUpdateContentRemoteOCR(DirectoriesMixin, TestCase):
"""
Consumption workflows do not run on reprocess, so the remote parser is
used only in 'always' mode or when the caller explicitly asks for it.
"""
def setUp(self) -> None:
super().setUp()
patcher = mock.patch("documents.tasks.get_parser_registry")
self.mock_registry = patcher.start()
self.mock_registry.return_value.get_parser_for_file.return_value = None
self.addCleanup(patcher.stop)
self.doc = Document.objects.create(
title="test",
content="my document",
checksum="wow",
mime_type="application/pdf",
)
def _allow_remote(self, **kwargs) -> bool:
tasks.update_document_content_maybe_archive_file(self.doc.pk, **kwargs)
_, call_kwargs = self.mock_registry.return_value.get_parser_for_file.call_args
return call_kwargs["allow_remote"]
@override_settings(REMOTE_OCR_MODE="always")
def test_always_mode_allows_remote(self) -> None:
self.assertTrue(self._allow_remote())
@override_settings(REMOTE_OCR_MODE="workflow_only")
def test_workflow_only_mode_denies_remote_by_default(self) -> None:
self.assertFalse(self._allow_remote())
@override_settings(REMOTE_OCR_MODE="workflow_only")
def test_workflow_only_mode_allows_remote_when_requested(self) -> None:
self.assertTrue(self._allow_remote(remote_ocr=True))
class TestAIIndex(DirectoriesMixin, TestCase):
@override_settings(
AI_ENABLED=True,
-49
View File
@@ -2000,55 +2000,6 @@ class TestWorkflows(
r"Doc added in \w{3,}",
) # Match any 3-letter month name
def test_document_updated_workflow_existing_custom_field_empty_value(self) -> None:
"""
GIVEN:
- Existing workflow with UPDATED trigger and action that assigns a custom field
with an empty value
WHEN:
- Document is updated that already contains the field with a value
THEN:
- The existing value is left untouched, see GH #13627
"""
trigger = WorkflowTrigger.objects.create(
type=WorkflowTrigger.WorkflowTriggerType.DOCUMENT_UPDATED,
filter_has_document_type=self.dt,
)
action = WorkflowAction.objects.create()
action.assign_custom_fields.add(self.cf1)
action.assign_custom_fields_values = {self.cf1.pk: ""}
action.save()
w = Workflow.objects.create(
name="Workflow 1",
order=0,
)
w.triggers.add(trigger)
w.actions.add(action)
w.save()
doc = Document.objects.create(
title="sample test",
correspondent=self.c,
original_filename="sample.pdf",
)
CustomFieldInstance.objects.create(
document=doc,
field=self.cf1,
value_text="existing value",
)
superuser = User.objects.create_superuser("superuser")
self.client.force_authenticate(user=superuser)
self.client.patch(
f"/api/documents/{doc.id}/",
{"document_type": self.dt.id},
format="json",
)
doc.refresh_from_db()
self.assertEqual(doc.custom_fields.get(field=self.cf1).value, "existing value")
def test_document_updated_workflow_existing_custom_field(self) -> None:
"""
GIVEN:
+1 -1
View File
@@ -2267,7 +2267,7 @@ class ChatStreamingView(GenericAPIView[Any]):
if not has_perms_owner_aware(request.user, "view_document", document):
return HttpResponseForbidden("Insufficient permissions")
documents = Document.objects.filter(pk=document.pk)
documents = [document]
else:
documents = Document.objects.filter(
id__in=permitted_document_ids(request.user),
+1 -2
View File
@@ -105,8 +105,7 @@ def apply_assignment_to_document(
field=field,
document=document,
).first()
# empty string is indistinguishable from no value in the UI
if instance and args[value_field_name] not in (None, ""):
if instance and args[value_field_name] is not None:
setattr(instance, value_field_name, args[value_field_name])
instance.save()
elif not instance:
File diff suppressed because it is too large Load Diff
+5 -8
View File
@@ -338,16 +338,13 @@ def check_deprecated_v2_ocr_env_vars(
@register()
def check_remote_ocr_mode(app_configs: Any, **kwargs: Any) -> list[Error]:
# Import here because checks.py runs before the app registry is ready
from paperless.models import RemoteOCRMode
valid_modes = {mode.value for mode in RemoteOCRMode}
if settings.REMOTE_OCR_MODE not in valid_modes:
def check_remote_parser_configured(app_configs: Any, **kwargs: Any) -> list[Error]:
if settings.REMOTE_OCR_ENGINE == "azureai" and not (
settings.REMOTE_OCR_ENDPOINT and settings.REMOTE_OCR_API_KEY
):
return [
Error(
f"PAPERLESS_REMOTE_OCR_MODE is set to {settings.REMOTE_OCR_MODE!r}, "
f"expected one of {sorted(valid_modes)}.",
"Azure AI remote parser requires endpoint and API key to be configured.",
),
]
-40
View File
@@ -9,7 +9,6 @@ from paperless.models import CleanChoices
from paperless.models import ColorConvertChoices
from paperless.models import ModeChoices
from paperless.models import OutputTypeChoices
from paperless.models import RemoteOCRMode
@dataclasses.dataclass
@@ -186,45 +185,6 @@ class GeneralConfig(BaseConfig):
self.app_logo = app_config.app_logo.url if app_config.app_logo else None
@dataclasses.dataclass
class RemoteOCRConfig(BaseConfig):
"""
Settings for the remote (cloud) OCR parser
"""
remote_ocr_engine: str | None = dataclasses.field(init=False)
remote_ocr_api_key: str | None = dataclasses.field(init=False)
remote_ocr_endpoint: str | None = dataclasses.field(init=False)
remote_ocr_mode: RemoteOCRMode = dataclasses.field(init=False)
def __post_init__(self) -> None:
app_config = self._get_config_instance()
self.remote_ocr_engine = (
app_config.remote_ocr_engine or settings.REMOTE_OCR_ENGINE
)
self.remote_ocr_api_key = (
app_config.remote_ocr_api_key or settings.REMOTE_OCR_API_KEY
)
self.remote_ocr_endpoint = (
app_config.remote_ocr_endpoint or settings.REMOTE_OCR_ENDPOINT
)
self.remote_ocr_mode = app_config.remote_ocr_mode or RemoteOCRMode(
settings.REMOTE_OCR_MODE,
)
@property
def remote_ocr_by_default(self) -> bool:
"""
Whether every supported document goes to the remote engine.
When False the remote engine is used only for documents that
explicitly asked for it, i.e. a workflow matched during consumption or
the user ticked the box when reprocessing.
"""
return self.remote_ocr_mode == RemoteOCRMode.ALWAYS
@dataclasses.dataclass
class AIConfig(BaseConfig):
"""
@@ -1,44 +0,0 @@
# Generated by Django 5.2.16 on 2026-08-10 14:37
from django.db import migrations
from django.db import models
class Migration(migrations.Migration):
dependencies = [
("paperless", "0013_applicationconfiguration_llm_request_timeout"),
]
operations = [
migrations.AddField(
model_name="applicationconfiguration",
name="remote_ocr_api_key",
field=models.CharField(
blank=True,
max_length=1024,
null=True,
verbose_name="Sets the remote OCR API key",
),
),
migrations.AddField(
model_name="applicationconfiguration",
name="remote_ocr_endpoint",
field=models.CharField(
blank=True,
max_length=256,
null=True,
verbose_name="Sets the remote OCR endpoint",
),
),
migrations.AddField(
model_name="applicationconfiguration",
name="remote_ocr_engine",
field=models.CharField(
blank=True,
choices=[("azureai", "Azure AI Document Intelligence")],
max_length=32,
null=True,
verbose_name="Sets the remote OCR engine",
),
),
]
@@ -1,27 +0,0 @@
# Generated by Django 5.2.16 on 2026-08-10 15:43
from django.db import migrations
from django.db import models
class Migration(migrations.Migration):
dependencies = [
("paperless", "0014_applicationconfiguration_remote_ocr_api_key_and_more"),
]
operations = [
migrations.AddField(
model_name="applicationconfiguration",
name="remote_ocr_mode",
field=models.CharField(
blank=True,
choices=[
("always", "All supported documents"),
("workflow_only", "Only when a workflow enables it"),
],
max_length=32,
null=True,
verbose_name="Sets which documents are sent to the remote OCR engine",
),
),
]
-55
View File
@@ -74,23 +74,6 @@ class ColorConvertChoices(models.TextChoices):
CMYK = ("CMYK", _("CMYK"))
class RemoteOCREngine(models.TextChoices):
"""
Matches to PAPERLESS_REMOTE_OCR_ENGINE
"""
AZURE_AI = ("azureai", _("Azure AI Document Intelligence"))
class RemoteOCRMode(models.TextChoices):
"""
Matches to PAPERLESS_REMOTE_OCR_MODE
"""
ALWAYS = ("always", _("All supported documents"))
WORKFLOW_ONLY = ("workflow_only", _("Only when a workflow enables it"))
class LLMEmbeddingBackend(models.TextChoices):
OPENAI_LIKE = ("openai-like", _("OpenAI-compatible"))
HUGGINGFACE = ("huggingface", _("Huggingface"))
@@ -303,44 +286,6 @@ class ApplicationConfiguration(AbstractSingletonModel):
null=True,
)
"""
Settings for the remote OCR parser
"""
# PAPERLESS_REMOTE_OCR_ENGINE
remote_ocr_engine = models.CharField(
verbose_name=_("Sets the remote OCR engine"),
blank=True,
null=True,
max_length=32,
choices=RemoteOCREngine.choices,
)
# PAPERLESS_REMOTE_OCR_API_KEY
remote_ocr_api_key = models.CharField(
verbose_name=_("Sets the remote OCR API key"),
blank=True,
null=True,
max_length=1024,
)
# PAPERLESS_REMOTE_OCR_ENDPOINT
remote_ocr_endpoint = models.CharField(
verbose_name=_("Sets the remote OCR endpoint"),
blank=True,
null=True,
max_length=256,
)
# PAPERLESS_REMOTE_OCR_MODE
remote_ocr_mode = models.CharField(
verbose_name=_("Sets which documents are sent to the remote OCR engine"),
blank=True,
null=True,
max_length=32,
choices=RemoteOCRMode.choices,
)
"""
AI related settings
"""
-9
View File
@@ -134,11 +134,6 @@ class ParserProtocol(Protocol):
Author or organisation name.
url : str
URL for documentation, source code, or issue tracker.
Parsers that send document content to a remote service should additionally
set ``uses_remote_service = True`` so the registry can exclude them when
remote processing has not been requested for a document. The attribute is
optional so a parser that omits it is treated as fully local.
"""
# ------------------------------------------------------------------
@@ -150,10 +145,6 @@ class ParserProtocol(Protocol):
author: str
url: str
# NOTE: uses_remote_service is not declared here, the registry reads it
# with getattr(cls, ..., False) for backwards-compatibility with existing
# parsers
# ------------------------------------------------------------------
# Class methods
# ------------------------------------------------------------------
-14
View File
@@ -334,8 +334,6 @@ class ParserRegistry:
mime_type: str,
filename: str,
path: Path | None = None,
*,
allow_remote: bool = True,
) -> type[ParserProtocol] | None:
"""Return the best parser class for the given file, or None.
@@ -361,11 +359,6 @@ class ParserRegistry:
path:
Optional filesystem path to the file. Forwarded to each
parser's score method.
allow_remote:
When False, parsers that declare ``uses_remote_service = True``
are excluded from consideration, so a document is never sent to
a remote service. Parsers that do not declare the attribute
are treated as local and are always considered.
Returns
-------
@@ -381,13 +374,6 @@ class ParserRegistry:
if mime_type not in parser_class.supported_mime_types():
continue
if not allow_remote and getattr(
parser_class,
"uses_remote_service",
False,
):
continue
score = parser_class.score(mime_type, filename, path)
if score is None:
continue
+17 -52
View File
@@ -3,9 +3,7 @@ Built-in remote-OCR document parser.
Handles documents by sending them to a configured remote OCR engine
(currently Azure AI Vision / Document Intelligence) and retrieving both
the extracted text and a searchable PDF with an embedded text layer. For
born-digital PDFs that need no archive copy, the remote call is skipped
entirely in favor of locally-extracted text (see ``RemoteDocumentParser.parse``).
the extracted text and a searchable PDF with an embedded text layer.
When no engine is configured, ``score()`` returns ``None`` so the parser
is effectively invisible to the registry the tesseract parser handles
@@ -24,8 +22,6 @@ from typing import Self
from django.conf import settings
from documents.parsers import ParseError
from paperless.parsers.utils import extract_pdf_text
from paperless.parsers.utils import post_process_text
from paperless.version import __full_version_str__
if TYPE_CHECKING:
@@ -61,18 +57,6 @@ class RemoteEngineConfig:
self.api_key = api_key
self.endpoint = endpoint
@classmethod
def from_app_config(cls) -> Self:
"""Build the config from the app config, falling back to the env."""
from paperless.config import RemoteOCRConfig
app_config = RemoteOCRConfig()
return cls(
engine=app_config.remote_ocr_engine,
api_key=app_config.remote_ocr_api_key,
endpoint=app_config.remote_ocr_endpoint,
)
def engine_is_valid(self) -> bool:
"""Return True when the engine is known and fully configured."""
return (
@@ -86,11 +70,8 @@ class RemoteDocumentParser:
"""Parse documents via a remote OCR API (currently Azure AI Vision).
This parser sends documents to a remote engine that returns both
extracted text and a searchable PDF with an embedded text layer,
except when ``parse()`` is called with ``produce_archive=False`` for
a PDF, in which case the remote call is skipped and only locally
extracted text is returned (no archive). It does not depend on
Tesseract or ocrmypdf.
extracted text and a searchable PDF with an embedded text layer.
It does not depend on Tesseract or ocrmypdf.
Class attributes
----------------
@@ -102,9 +83,6 @@ class RemoteDocumentParser:
Maintainer name.
url : str
Issue tracker / source URL.
uses_remote_service : bool
Content is sent to a remote service, True so that the registry
can skip this parser if remote processing was not requested.
"""
name: str = "Paperless-ngx Remote OCR Parser"
@@ -112,8 +90,6 @@ class RemoteDocumentParser:
author: str = "Paperless-ngx Contributors"
url: str = "https://github.com/paperless-ngx/paperless-ngx"
uses_remote_service: bool = True
# ------------------------------------------------------------------
# Class methods
# ------------------------------------------------------------------
@@ -162,7 +138,11 @@ class RemoteDocumentParser:
20 when the remote engine is configured and the MIME type is
supported, otherwise None.
"""
config = RemoteEngineConfig.from_app_config()
config = RemoteEngineConfig(
engine=settings.REMOTE_OCR_ENGINE,
api_key=settings.REMOTE_OCR_API_KEY,
endpoint=settings.REMOTE_OCR_ENDPOINT,
)
if not config.engine_is_valid():
return None
if mime_type not in _SUPPORTED_MIME_TYPES:
@@ -180,11 +160,8 @@ class RemoteDocumentParser:
Returns
-------
bool
Always True the remote engine is capable of returning a PDF
with an embedded text layer to serve as the archive copy.
Whether it actually does so for a given document depends on
``produce_archive`` passed to :meth:`parse` (see there for when
the remote engine call, and thus archive generation, is skipped).
Always True the remote engine always returns a PDF with an
embedded text layer that serves as the archive copy.
"""
return True
@@ -241,12 +218,6 @@ class RemoteDocumentParser:
) -> None:
"""Send the document to the remote engine and store results.
When *produce_archive* is False for a PDF, the caller (via
``documents.consumer.should_produce_archive``) has already determined
that the document is born-digital and needs no archive skip the
remote engine entirely rather than re-OCRing it and creating a
duplicate text layer.
Parameters
----------
document_path:
@@ -254,10 +225,14 @@ class RemoteDocumentParser:
mime_type:
Detected MIME type of the document.
produce_archive:
Whether an archive copy is wanted. For PDFs, False skips the
remote engine and uses locally-extracted text instead.
Ignored the remote engine always returns a searchable PDF,
which is stored as the archive copy regardless of this flag.
"""
config = RemoteEngineConfig.from_app_config()
config = RemoteEngineConfig(
engine=settings.REMOTE_OCR_ENGINE,
api_key=settings.REMOTE_OCR_API_KEY,
endpoint=settings.REMOTE_OCR_ENDPOINT,
)
if not config.engine_is_valid():
logger.warning(
@@ -266,16 +241,6 @@ class RemoteDocumentParser:
self._text = ""
return
if not produce_archive and mime_type == "application/pdf":
logger.debug(
"Remote OCR: skipped — no archive requested, "
"using locally-extracted text",
)
self._text = (
post_process_text(extract_pdf_text(document_path, log=logger)) or ""
)
return
if config.engine == "azureai":
self._text = self._azure_ai_vision_parse(document_path, config)
+5 -14
View File
@@ -219,13 +219,6 @@ class ApplicationConfigurationSerializer(
allow_null=True,
max_length=1024,
)
remote_ocr_api_key = ObfuscatedPasswordField(
required=False,
allow_null=True,
max_length=1024,
)
OBFUSCATED_FIELDS = ("llm_api_key", "remote_ocr_api_key")
def run_validation(self, data):
# Empty strings treated as None to avoid unexpected behavior
@@ -237,13 +230,11 @@ class ApplicationConfigurationSerializer(
data["language"] = None
if "llm_output_language" in data and data["llm_output_language"] == "":
data["llm_output_language"] = None
for field in self.OBFUSCATED_FIELDS:
if field in data and data[field] is not None:
if data[field] == "":
data[field] = None
# Not a real value, don't overwrite the stored one
elif len(data[field].replace("*", "")) == 0:
del data[field]
if "llm_api_key" in data and data["llm_api_key"] is not None:
if data["llm_api_key"] == "":
data["llm_api_key"] = None
elif len(data["llm_api_key"].replace("*", "")) == 0:
del data["llm_api_key"]
return super().run_validation(data)
def update(self, instance, validated_data):
-1
View File
@@ -1197,7 +1197,6 @@ WEBHOOKS_ALLOW_INTERNAL_REQUESTS = get_bool_from_env(
REMOTE_OCR_ENGINE = os.getenv("PAPERLESS_REMOTE_OCR_ENGINE")
REMOTE_OCR_API_KEY = os.getenv("PAPERLESS_REMOTE_OCR_API_KEY")
REMOTE_OCR_ENDPOINT = os.getenv("PAPERLESS_REMOTE_OCR_ENDPOINT")
REMOTE_OCR_MODE = os.getenv("PAPERLESS_REMOTE_OCR_MODE", "always")
################################################################################
# AI Settings #
@@ -21,7 +21,6 @@ from unittest.mock import Mock
import pytest
from documents.parsers import ParseError
from paperless.models import ApplicationConfiguration
from paperless.parsers import ParserContext
from paperless.parsers import ParserProtocol
from paperless.parsers.remote import RemoteDocumentParser
@@ -34,10 +33,6 @@ if TYPE_CHECKING:
from pytest_mock import MockerFixture
# Remote ocr config from ApplicationConfiguration needs DB access
pytestmark = pytest.mark.django_db
# ---------------------------------------------------------------------------
# Module-local fixtures
# ---------------------------------------------------------------------------
@@ -232,18 +227,6 @@ class TestRemoteParserScore:
score = RemoteDocumentParser.score("application/pdf", "doc.pdf")
assert score is not None and score > 10
@pytest.mark.usefixtures("no_engine_settings")
def test_score_uses_app_config_when_env_unset(self) -> None:
"""The app config alone is enough to activate the parser."""
config = ApplicationConfiguration.objects.first()
assert config is not None
config.remote_ocr_engine = "azureai"
config.remote_ocr_api_key = "app-config-key"
config.remote_ocr_endpoint = "https://config.cognitiveservices.azure.com"
config.save()
assert RemoteDocumentParser.score("application/pdf", "doc.pdf") == 20
# ---------------------------------------------------------------------------
# Properties
@@ -354,117 +337,6 @@ class TestRemoteParserParse:
assert remote_parser.get_date() is None
# ---------------------------------------------------------------------------
# parse() — produce_archive=False skips the remote engine (PDFs only)
# ---------------------------------------------------------------------------
class TestRemoteParserSkipsWhenNoArchiveWanted:
"""When the caller has already decided no archive is needed for a PDF
(documents.consumer.should_produce_archive), the remote engine call is
skipped entirely in favor of locally-extracted text.
"""
def test_pdf_skips_azure_when_no_archive_requested(
self,
remote_parser: RemoteDocumentParser,
simple_digital_pdf_file: Path,
azure_client: Mock,
) -> None:
"""
GIVEN: produce_archive=False for a PDF
WHEN: parse() is called
THEN: Azure is never invoked, no archive is produced, and text
comes from local pdftotext extraction
"""
remote_parser.parse(
simple_digital_pdf_file,
"application/pdf",
produce_archive=False,
)
azure_client.begin_analyze_document.assert_not_called()
assert remote_parser.get_archive_path() is None
assert remote_parser.get_text() != ""
def test_pdf_no_archive_requested_text_matches_local_extraction(
self,
remote_parser: RemoteDocumentParser,
simple_digital_pdf_file: Path,
azure_client: Mock,
mocker: MockerFixture,
) -> None:
"""
GIVEN: produce_archive=False for a PDF
WHEN: parse() is called
THEN: the returned text is exactly the locally-extracted text,
not anything from the (unused) Azure mock
"""
mocker.patch(
"paperless.parsers.remote.extract_pdf_text",
return_value="Local digital text.",
)
remote_parser.parse(
simple_digital_pdf_file,
"application/pdf",
produce_archive=False,
)
assert remote_parser.get_text() == "Local digital text."
def test_pdf_no_archive_requested_closes_no_client(
self,
remote_parser: RemoteDocumentParser,
simple_digital_pdf_file: Path,
azure_client: Mock,
) -> None:
remote_parser.parse(
simple_digital_pdf_file,
"application/pdf",
produce_archive=False,
)
azure_client.close.assert_not_called()
def test_non_pdf_still_calls_azure_when_no_archive_requested(
self,
remote_parser: RemoteDocumentParser,
simple_digital_pdf_file: Path,
azure_client: Mock,
) -> None:
"""
Images have no local-text fallback, so produce_archive=False does
not skip the remote engine for non-PDF MIME types.
"""
remote_parser.parse(
simple_digital_pdf_file,
"image/png",
produce_archive=False,
)
azure_client.begin_analyze_document.assert_called_once()
assert remote_parser.get_text() == _DEFAULT_TEXT
@pytest.mark.usefixtures("no_engine_settings")
def test_unconfigured_engine_takes_precedence_over_skip(
self,
remote_parser: RemoteDocumentParser,
simple_digital_pdf_file: Path,
) -> None:
"""An unconfigured engine still short-circuits before the
produce_archive check, returning empty text as before.
"""
remote_parser.parse(
simple_digital_pdf_file,
"application/pdf",
produce_archive=False,
)
assert remote_parser.get_text() == ""
assert remote_parser.get_archive_path() is None
# ---------------------------------------------------------------------------
# parse() — Azure failure path
# ---------------------------------------------------------------------------
@@ -1277,8 +1277,6 @@ class TestParserFileTypes:
# ---------------------------------------------------------------------------
# Remote ocr config from ApplicationConfiguration needs DB access
@pytest.mark.django_db
class TestRasterisedDocumentParserRegistry:
def test_registered_in_defaults(self) -> None:
from paperless.parsers.registry import ParserRegistry
+18 -10
View File
@@ -15,7 +15,7 @@ from paperless.checks import audit_log_check
from paperless.checks import binaries_check
from paperless.checks import check_default_language_available
from paperless.checks import check_deprecated_db_settings
from paperless.checks import check_remote_ocr_mode
from paperless.checks import check_remote_parser_configured
from paperless.checks import check_v3_minimum_upgrade_version
from paperless.checks import debug_mode_check
from paperless.checks import paths_check
@@ -631,21 +631,29 @@ class TestV3MinimumUpgradeVersionCheck:
assert check_v3_minimum_upgrade_version(None) == []
class TestRemoteOCRModeCheck:
def test_valid_mode(self, settings: SettingsWrapper) -> None:
settings.REMOTE_OCR_MODE = "workflow_only"
msgs = check_remote_ocr_mode(None)
class TestRemoteParserChecks:
def test_no_engine(self, settings: SettingsWrapper) -> None:
settings.REMOTE_OCR_ENGINE = None
msgs = check_remote_parser_configured(None)
assert len(msgs) == 0
def test_invalid_mode(self, settings: SettingsWrapper) -> None:
settings.REMOTE_OCR_MODE = "sometimes"
def test_azure_no_endpoint(self, settings: SettingsWrapper) -> None:
msgs = check_remote_ocr_mode(None)
settings.REMOTE_OCR_ENGINE = "azureai"
settings.REMOTE_OCR_API_KEY = "somekey"
settings.REMOTE_OCR_ENDPOINT = None
msgs = check_remote_parser_configured(None)
assert len(msgs) == 1
assert "PAPERLESS_REMOTE_OCR_MODE is set to 'sometimes'" in msgs[0].msg
msg = msgs[0]
assert (
"Azure AI remote parser requires endpoint and API key to be configured."
in msg.msg
)
class TestTesseractChecks:
-118
View File
@@ -468,124 +468,6 @@ class TestParserRegistryGetParserForFile:
assert result is AcceptingBuiltin
class TestParserRegistryRemoteParsers:
"""Verify the allow_remote filter in ParserRegistry.get_parser_for_file()."""
@staticmethod
def _remote_parser_cls() -> type:
class RemoteParser:
name = "remote"
version = "1.0"
author = "A"
url = "https://example.com/remote"
uses_remote_service = True
@classmethod
def supported_mime_types(cls):
return {"text/plain": ".txt"}
@classmethod
def score(cls, mime_type, filename, path=None):
return 20
return RemoteParser
def test_remote_parser_wins_when_remote_allowed(
self,
dummy_parser_cls: type,
) -> None:
"""
GIVEN: A remote parser scoring 20 and a local parser scoring 10.
WHEN: get_parser_for_file() is called with allow_remote=True.
THEN: The remote parser is returned.
"""
remote_parser_cls = self._remote_parser_cls()
registry = ParserRegistry()
registry.register_builtin(dummy_parser_cls)
registry.register_builtin(remote_parser_cls)
result = registry.get_parser_for_file(
"text/plain",
"readme.txt",
allow_remote=True,
)
assert result is remote_parser_cls
def test_remote_parser_skipped_when_remote_not_allowed(
self,
dummy_parser_cls: type,
) -> None:
"""
GIVEN: A remote parser scoring 20 and a local parser scoring 10.
WHEN: get_parser_for_file() is called with allow_remote=False.
THEN: The local parser is returned despite its lower score.
"""
registry = ParserRegistry()
registry.register_builtin(dummy_parser_cls)
registry.register_builtin(self._remote_parser_cls())
result = registry.get_parser_for_file(
"text/plain",
"readme.txt",
allow_remote=False,
)
assert result is dummy_parser_cls
def test_no_parser_when_only_remote_available_and_not_allowed(self) -> None:
"""
GIVEN: A registry whose only candidate declares uses_remote_service.
WHEN: get_parser_for_file() is called with allow_remote=False.
THEN: None is returned the remote parser is never used as a
fallback when remote processing was not requested.
"""
registry = ParserRegistry()
registry.register_builtin(self._remote_parser_cls())
result = registry.get_parser_for_file(
"text/plain",
"readme.txt",
allow_remote=False,
)
assert result is None
def test_parser_without_attribute_treated_as_local(
self,
dummy_parser_cls: type,
) -> None:
"""
GIVEN: A third-party parser predating uses_remote_service, so it does
not declare the attribute at all.
WHEN: get_parser_for_file() is called with allow_remote=False.
THEN: It is still considered, i.e. treated as fully local, rather
than raising AttributeError.
"""
assert not hasattr(dummy_parser_cls, "uses_remote_service")
registry = ParserRegistry()
registry.register_builtin(dummy_parser_cls)
result = registry.get_parser_for_file(
"text/plain",
"readme.txt",
allow_remote=False,
)
assert result is dummy_parser_cls
def test_remote_allowed_by_default(self) -> None:
"""
GIVEN: A registry containing only a remote parser.
WHEN: get_parser_for_file() is called without allow_remote.
THEN: The remote parser is returned callers that do not opt in to
the filter keep the previous behaviour.
"""
remote_parser_cls = self._remote_parser_cls()
registry = ParserRegistry()
registry.register_builtin(remote_parser_cls)
result = registry.get_parser_for_file("text/plain", "readme.txt")
assert result is remote_parser_cls
class TestDiscover:
"""Verify entrypoint discovery in ParserRegistry.discover()."""
@@ -1,113 +0,0 @@
"""Tests for RemoteOCRConfig precedence between app config and Django settings."""
from __future__ import annotations
from typing import TYPE_CHECKING
import pytest
from django.test import override_settings
from paperless.config import RemoteOCRConfig
from paperless.models import RemoteOCRMode
if TYPE_CHECKING:
from unittest.mock import MagicMock
@pytest.fixture()
def null_app_config(mocker) -> MagicMock:
"""Mock ApplicationConfiguration with all fields None → falls back to Django settings."""
return mocker.MagicMock(
remote_ocr_engine=None,
remote_ocr_api_key=None,
remote_ocr_endpoint=None,
remote_ocr_mode=None,
)
@pytest.fixture()
def make_remote_ocr_config(mocker):
def _make(app_config, **django_settings_overrides):
mocker.patch(
"paperless.config.BaseConfig._get_config_instance",
return_value=app_config,
)
with override_settings(**django_settings_overrides):
return RemoteOCRConfig()
return _make
class TestRemoteOCRConfig:
def test_falls_back_to_settings(
self,
make_remote_ocr_config,
null_app_config,
) -> None:
cfg = make_remote_ocr_config(
null_app_config,
REMOTE_OCR_ENGINE="azureai",
REMOTE_OCR_API_KEY="env-key",
REMOTE_OCR_ENDPOINT="https://env.cognitiveservices.azure.com",
REMOTE_OCR_MODE=RemoteOCRMode.WORKFLOW_ONLY,
)
assert cfg.remote_ocr_engine == "azureai"
assert cfg.remote_ocr_api_key == "env-key"
assert cfg.remote_ocr_endpoint == "https://env.cognitiveservices.azure.com"
assert cfg.remote_ocr_mode == RemoteOCRMode.WORKFLOW_ONLY
def test_app_config_takes_precedence(
self,
make_remote_ocr_config,
mocker,
) -> None:
app_config = mocker.MagicMock(
remote_ocr_engine="azureai",
remote_ocr_api_key="db-key",
remote_ocr_endpoint="https://db.cognitiveservices.azure.com",
remote_ocr_mode=RemoteOCRMode.WORKFLOW_ONLY,
)
cfg = make_remote_ocr_config(
app_config,
REMOTE_OCR_ENGINE=None,
REMOTE_OCR_API_KEY="env-key",
REMOTE_OCR_ENDPOINT="https://env.cognitiveservices.azure.com",
REMOTE_OCR_MODE=RemoteOCRMode.ALWAYS,
)
assert cfg.remote_ocr_engine == "azureai"
assert cfg.remote_ocr_api_key == "db-key"
assert cfg.remote_ocr_endpoint == "https://db.cognitiveservices.azure.com"
assert cfg.remote_ocr_mode == RemoteOCRMode.WORKFLOW_ONLY
def test_unset_everywhere(
self,
make_remote_ocr_config,
null_app_config,
) -> None:
cfg = make_remote_ocr_config(
null_app_config,
REMOTE_OCR_ENGINE=None,
REMOTE_OCR_API_KEY=None,
REMOTE_OCR_ENDPOINT=None,
)
assert cfg.remote_ocr_engine is None
assert cfg.remote_ocr_api_key is None
assert cfg.remote_ocr_endpoint is None
class TestRemoteOCRByDefault:
def test_always_mode(self, make_remote_ocr_config, null_app_config) -> None:
cfg = make_remote_ocr_config(
null_app_config,
REMOTE_OCR_MODE=RemoteOCRMode.ALWAYS,
)
assert cfg.remote_ocr_by_default is True
def test_workflow_only_mode(self, make_remote_ocr_config, null_app_config) -> None:
cfg = make_remote_ocr_config(
null_app_config,
REMOTE_OCR_MODE=RemoteOCRMode.WORKFLOW_ONLY,
)
assert cfg.remote_ocr_by_default is False
+6 -21
View File
@@ -2,8 +2,6 @@ import json
import logging
import sys
from django.db.models import QuerySet
from documents.models import Document
from paperless.config import AIConfig
from paperless_ai.client import AIClient
@@ -84,21 +82,10 @@ def _build_document_reference(
def _get_document_references(
documents: QuerySet[Document],
documents: list[Document],
top_nodes: list,
) -> list[dict[str, int | str]]:
candidate_ids: set[int] = set()
for node in top_nodes:
try:
candidate_ids.add(int(node.metadata["document_id"]))
except (KeyError, TypeError, ValueError): # pragma: no cover
continue
if not candidate_ids:
return []
allowed_documents = {doc.pk: doc for doc in documents.filter(pk__in=candidate_ids)}
allowed_documents = {doc.pk: doc for doc in documents}
references: list[dict[str, int | str]] = []
seen_document_ids: set[int] = set()
@@ -132,7 +119,7 @@ def _format_chat_metadata_trailer(references: list[dict[str, int | str]]) -> str
def stream_chat_with_documents(
query_str: str,
documents: QuerySet[Document],
documents: list[Document],
output_language: str | None = None,
):
try:
@@ -148,10 +135,10 @@ def stream_chat_with_documents(
def _stream_chat_with_documents(
query_str: str,
documents: QuerySet[Document],
documents: list[Document],
output_language: str | None = None,
):
if not documents.exists():
if not documents:
yield CHAT_NO_CONTENT_MESSAGE
return
@@ -161,9 +148,7 @@ def _stream_chat_with_documents(
from llama_index.core.retrievers import VectorIndexRetriever
config = AIConfig()
filters = _document_id_filters(
str(pk) for pk in documents.values_list("pk", flat=True)
)
filters = _document_id_filters(str(doc.pk) for doc in documents)
# Hold the shared read lock for the whole operation: the query engine
# retrieves from the vector store again during synthesis, so the connection
+46 -101
View File
@@ -3,12 +3,10 @@ from unittest.mock import MagicMock
from unittest.mock import patch
import pytest
from django.db.models.signals import post_init
from llama_index.core import settings as llama_settings
from llama_index.core.embeddings.mock_embed_model import MockEmbedding
from llama_index.core.schema import TextNode
from documents.models import Document
from documents.tests.factories import DocumentFactory
from paperless_ai import chat
from paperless_ai import indexing
@@ -38,6 +36,16 @@ def patch_embed_nodes():
yield mock_embed_nodes
@pytest.fixture
def mock_document():
doc = MagicMock()
doc.pk = 1
doc.title = "Test Document"
doc.filename = "test_file.pdf"
doc.content = "This is the document content."
return doc
def assert_chat_output(
output: list[str],
*,
@@ -53,13 +61,6 @@ def assert_chat_output(
}
def _fake_documents_queryset(pks: list[int]) -> MagicMock:
qs = MagicMock()
qs.exists.return_value = bool(pks)
qs.values_list.return_value = pks
return qs
@pytest.mark.parametrize(
("output_language", "expected_language_line"),
[
@@ -106,10 +107,9 @@ def test_build_refine_prompt(
@pytest.mark.django_db
def test_stream_chat_with_one_document_retrieval(
mock_document,
patch_embed_nodes,
) -> None:
document = DocumentFactory.create(title="Test Document", content="ignored")
documents = Document.objects.filter(pk=document.pk)
with (
patch("paperless_ai.chat.AIClient") as mock_client_cls,
patch("paperless_ai.chat.load_or_build_index") as mock_load_index,
@@ -124,19 +124,22 @@ def test_stream_chat_with_one_document_retrieval(
mock_client_cls.return_value = mock_client
mock_client.llm = MagicMock()
mock_node = TextNode(
text="This is node content.",
metadata={"document_id": str(mock_document.pk), "title": "Test Document"},
)
mock_index = MagicMock()
mock_index.vector_store.get_nodes.return_value = [
TextNode(
text="This is node content.",
metadata={"document_id": str(document.pk), "title": "Test Document"},
),
]
# Simulate get_nodes returning nodes (content exists)
mock_index.vector_store.get_nodes.return_value = [mock_node]
mock_load_index.return_value = mock_index
mock_retriever_instance = MagicMock()
mock_retriever_instance.retrieve.return_value = [
MagicMock(
metadata={"document_id": str(document.pk), "title": "Test Document"},
metadata={
"document_id": str(mock_document.pk),
"title": "Test Document",
},
),
]
@@ -150,7 +153,7 @@ def test_stream_chat_with_one_document_retrieval(
"llama_index.core.retrievers.VectorIndexRetriever",
return_value=mock_retriever_instance,
):
output = list(stream_chat_with_documents("What is this?", documents))
output = list(stream_chat_with_documents("What is this?", [mock_document]))
mock_query_engine.query.assert_called_once_with("What is this?")
synthesizer_kwargs = mock_get_response_synthesizer.call_args.kwargs
@@ -163,16 +166,13 @@ def test_stream_chat_with_one_document_retrieval(
output,
expected_chunks=["chunk1", "chunk2"],
expected_references=[
{"id": document.pk, "title": "Test Document"},
{"id": mock_document.pk, "title": "Test Document"},
],
)
@pytest.mark.django_db
def test_stream_chat_with_multiple_documents_retrieval(patch_embed_nodes) -> None:
doc1 = DocumentFactory.create(title="Document 1", content="ignored")
doc2 = DocumentFactory.create(title="Document 2", content="ignored")
documents = Document.objects.filter(pk__in=[doc1.pk, doc2.pk])
with (
patch("paperless_ai.chat.AIClient") as mock_client_cls,
patch("paperless_ai.chat.load_or_build_index") as mock_load_index,
@@ -184,23 +184,23 @@ def test_stream_chat_with_multiple_documents_retrieval(patch_embed_nodes) -> Non
mock_client_cls.return_value = mock_client
mock_client.llm = MagicMock()
mock_node1 = TextNode(
text="Content for doc 1.",
metadata={"document_id": "1", "title": "Document 1"},
)
mock_node2 = TextNode(
text="Content for doc 2.",
metadata={"document_id": "2", "title": "Document 2"},
)
mock_index = MagicMock()
mock_index.vector_store.get_nodes.return_value = [
TextNode(
text="Content for doc 1.",
metadata={"document_id": str(doc1.pk), "title": "Document 1"},
),
TextNode(
text="Content for doc 2.",
metadata={"document_id": str(doc2.pk), "title": "Document 2"},
),
]
# Simulate get_nodes returning nodes (content exists)
mock_index.vector_store.get_nodes.return_value = [mock_node1, mock_node2]
mock_load_index.return_value = mock_index
mock_retriever_instance = MagicMock()
mock_retriever_instance.retrieve.return_value = [
MagicMock(metadata={"document_id": str(doc1.pk), "title": "Document 1"}),
MagicMock(metadata={"document_id": str(doc2.pk), "title": "Document 2"}),
MagicMock(metadata={"document_id": "1", "title": "Document 1"}),
MagicMock(metadata={"document_id": "2", "title": "Document 2"}),
]
mock_response_stream = MagicMock()
@@ -210,11 +210,14 @@ def test_stream_chat_with_multiple_documents_retrieval(patch_embed_nodes) -> Non
mock_query_engine_cls.return_value = mock_query_engine
mock_query_engine.query.return_value = mock_response_stream
doc1 = MagicMock(pk=1, title="Document 1", filename="doc1.pdf")
doc2 = MagicMock(pk=2, title="Document 2", filename="doc2.pdf")
with patch(
"llama_index.core.retrievers.VectorIndexRetriever",
return_value=mock_retriever_instance,
):
output = list(stream_chat_with_documents("What's up?", documents))
output = list(stream_chat_with_documents("What's up?", [doc1, doc2]))
mock_query_engine.query.assert_called_once_with("What's up?")
patch_embed_nodes.assert_not_called()
@@ -222,15 +225,15 @@ def test_stream_chat_with_multiple_documents_retrieval(patch_embed_nodes) -> Non
output,
expected_chunks=["chunk1", "chunk2"],
expected_references=[
{"id": doc1.pk, "title": "Document 1"},
{"id": doc2.pk, "title": "Document 2"},
{"id": 1, "title": "Document 1"},
{"id": 2, "title": "Document 2"},
],
)
def test_stream_chat_empty_document_list() -> None:
with patch("paperless_ai.chat.load_or_build_index") as mock_load_index:
output = list(stream_chat_with_documents("Any info?", Document.objects.none()))
output = list(stream_chat_with_documents("Any info?", []))
mock_load_index.assert_not_called()
assert output == ["Sorry, I couldn't find any content to answer your question."]
@@ -250,9 +253,7 @@ def test_stream_chat_no_matching_nodes() -> None:
mock_index.vector_store.get_nodes.return_value = []
mock_load_index.return_value = mock_index
output = list(
stream_chat_with_documents("Any info?", _fake_documents_queryset([1])),
)
output = list(stream_chat_with_documents("Any info?", [MagicMock(pk=1)]))
assert output == ["Sorry, I couldn't find any content to answer your question."]
@@ -281,9 +282,7 @@ def test_stream_chat_unexpected_failure_returns_generic_error(caplog) -> None:
)
mock_retriever_cls.return_value = mock_retriever
output = list(
stream_chat_with_documents("Any info?", _fake_documents_queryset([1])),
)
output = list(stream_chat_with_documents("Any info?", [MagicMock(pk=1)]))
assert output == [CHAT_ERROR_MESSAGE]
assert "Failed to stream document chat response" in caplog.text
@@ -299,12 +298,7 @@ class TestStreamChatRetrieval:
) -> None:
doc = DocumentFactory.create(content="hello world")
# Nothing indexed for this document yet.
out = list(
chat.stream_chat_with_documents(
"question?",
Document.objects.filter(pk=doc.pk),
),
)
out = list(chat.stream_chat_with_documents("question?", [doc]))
assert chat.CHAT_NO_CONTENT_MESSAGE in out
def test_chat_filter_contains_only_requested_document_ids(
@@ -338,12 +332,7 @@ class TestStreamChatRetrieval:
side_effect=capture_retriever,
)
list(
chat.stream_chat_with_documents(
"question?",
Document.objects.filter(pk=included.pk),
),
)
list(chat.stream_chat_with_documents("question?", [included]))
assert captured_filters, "VectorIndexRetriever was never constructed"
filt = captured_filters[0]
@@ -351,47 +340,3 @@ class TestStreamChatRetrieval:
filter_values = filt.filters[0].value
assert str(included.pk) in filter_values
assert str(excluded.pk) not in filter_values
@pytest.mark.django_db
def test_get_document_references_only_queries_referenced_documents(
self,
django_assert_num_queries,
) -> None:
"""Building references must not hydrate every document the caller is
permitted to see -- only the (<= CHAT_RETRIEVER_TOP_K) documents that
the retriever actually returned nodes for.
"""
referenced = DocumentFactory.create(title="Referenced Document")
# Many more documents are "accessible" but never referenced by a node.
DocumentFactory.create_batch(200)
documents = Document.objects.all()
top_nodes = [
MagicMock(
metadata={
"document_id": str(referenced.pk),
"title": "Referenced Document",
},
),
]
hydrated_count = 0
def _count_hydration(sender, instance, **kwargs):
nonlocal hydrated_count
hydrated_count += 1
post_init.connect(_count_hydration, sender=Document)
try:
# One query: `documents.filter(pk__in=candidate_ids)` for the single
# referenced id. No query should scale with the 200 unreferenced documents.
with django_assert_num_queries(1):
references = chat._get_document_references(documents, top_nodes)
finally:
post_init.disconnect(_count_hydration, sender=Document)
# The bug this guards against: the old code hydrated all 201 accessible
# documents via `{doc.pk: doc for doc in documents}` before filtering by
# top_nodes. Only the referenced document should ever be constructed.
assert hydrated_count == 1
assert references == [{"id": referenced.pk, "title": "Referenced Document"}]