Compare commits

..
Author SHA1 Message Date
Crowdin Bot 82ae872b60 New Crowdin translations by GitHub Action 2026-08-01 23:28:05 +00:00
GitHub Actions c05d069e97 Auto translate strings 2026-08-01 23:27:09 +00:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>shamoon
f5e47a86c3 Chore(deps): Bump the frontend-angular-dependencies group across 1 directory with 18 updates (#13476)
* Chore(deps): Bump the frontend-angular-dependencies group across 1 directory with 18 updates

Bumps the frontend-angular-dependencies group with 18 updates in the /src-ui directory:

| Package | From | To |
| --- | --- | --- |
| [@angular/cdk](https://github.com/angular/components) | `22.0.3` | `22.0.6` |
| [@angular/common](https://github.com/angular/angular/tree/HEAD/packages/common) | `22.0.5` | `22.0.8` |
| [@angular/compiler](https://github.com/angular/angular/tree/HEAD/packages/compiler) | `22.0.5` | `22.0.8` |
| [@angular/core](https://github.com/angular/angular/tree/HEAD/packages/core) | `22.0.5` | `22.0.8` |
| [@angular/forms](https://github.com/angular/angular/tree/HEAD/packages/forms) | `22.0.5` | `22.0.8` |
| [@angular/localize](https://github.com/angular/angular) | `22.0.5` | `22.0.8` |
| [@angular/platform-browser](https://github.com/angular/angular/tree/HEAD/packages/platform-browser) | `22.0.5` | `22.0.8` |
| [@angular/router](https://github.com/angular/angular/tree/HEAD/packages/router) | `22.0.5` | `22.0.8` |
| [@ng-select/ng-select](https://github.com/ng-select/ng-select) | `23.2.0` | `23.5.0` |
| [@angular-devkit/core](https://github.com/angular/angular-cli) | `22.0.5` | `22.0.8` |
| [@angular-devkit/schematics](https://github.com/angular/angular-cli) | `22.0.5` | `22.0.8` |
| [@angular-eslint/builder](https://github.com/angular-eslint/angular-eslint/tree/HEAD/packages/builder) | `22.0.0` | `22.1.0` |
| [@angular-eslint/eslint-plugin](https://github.com/angular-eslint/angular-eslint/tree/HEAD/packages/eslint-plugin) | `22.0.0` | `22.1.0` |
| [@angular-eslint/eslint-plugin-template](https://github.com/angular-eslint/angular-eslint/tree/HEAD/packages/eslint-plugin-template) | `22.0.0` | `22.1.0` |
| [@angular-eslint/schematics](https://github.com/angular-eslint/angular-eslint/tree/HEAD/packages/schematics) | `22.0.0` | `22.1.0` |
| [@angular-eslint/template-parser](https://github.com/angular-eslint/angular-eslint/tree/HEAD/packages/template-parser) | `22.0.0` | `22.1.0` |
| [@angular/build](https://github.com/angular/angular-cli) | `22.0.5` | `22.0.8` |
| [@angular/compiler-cli](https://github.com/angular/angular/tree/HEAD/packages/compiler-cli) | `22.0.5` | `22.0.8` |



Updates `@angular/cdk` from 22.0.3 to 22.0.6
- [Release notes](https://github.com/angular/components/releases)
- [Changelog](https://github.com/angular/components/blob/main/CHANGELOG.md)
- [Commits](https://github.com/angular/components/compare/v22.0.3...v22.0.6)

Updates `@angular/common` from 22.0.5 to 22.0.8
- [Release notes](https://github.com/angular/angular/releases)
- [Changelog](https://github.com/angular/angular/blob/main/CHANGELOG.md)
- [Commits](https://github.com/angular/angular/commits/v22.0.8/packages/common)

Updates `@angular/compiler` from 22.0.5 to 22.0.8
- [Release notes](https://github.com/angular/angular/releases)
- [Changelog](https://github.com/angular/angular/blob/main/CHANGELOG.md)
- [Commits](https://github.com/angular/angular/commits/v22.0.8/packages/compiler)

Updates `@angular/core` from 22.0.5 to 22.0.8
- [Release notes](https://github.com/angular/angular/releases)
- [Changelog](https://github.com/angular/angular/blob/main/CHANGELOG.md)
- [Commits](https://github.com/angular/angular/commits/v22.0.8/packages/core)

Updates `@angular/forms` from 22.0.5 to 22.0.8
- [Release notes](https://github.com/angular/angular/releases)
- [Changelog](https://github.com/angular/angular/blob/main/CHANGELOG.md)
- [Commits](https://github.com/angular/angular/commits/v22.0.8/packages/forms)

Updates `@angular/localize` from 22.0.5 to 22.0.8
- [Release notes](https://github.com/angular/angular/releases)
- [Changelog](https://github.com/angular/angular/blob/main/CHANGELOG.md)
- [Commits](https://github.com/angular/angular/compare/v22.0.5...v22.0.8)

Updates `@angular/platform-browser` from 22.0.5 to 22.0.8
- [Release notes](https://github.com/angular/angular/releases)
- [Changelog](https://github.com/angular/angular/blob/main/CHANGELOG.md)
- [Commits](https://github.com/angular/angular/commits/v22.0.8/packages/platform-browser)

Updates `@angular/router` from 22.0.5 to 22.0.8
- [Release notes](https://github.com/angular/angular/releases)
- [Changelog](https://github.com/angular/angular/blob/main/CHANGELOG.md)
- [Commits](https://github.com/angular/angular/commits/v22.0.8/packages/router)

Updates `@ng-select/ng-select` from 23.2.0 to 23.5.0
- [Release notes](https://github.com/ng-select/ng-select/releases)
- [Changelog](https://github.com/ng-select/ng-select/blob/master/CHANGELOG.md)
- [Commits](https://github.com/ng-select/ng-select/compare/v23.2.0...v23.5.0)

Updates `@angular-devkit/core` from 22.0.5 to 22.0.8
- [Release notes](https://github.com/angular/angular-cli/releases)
- [Changelog](https://github.com/angular/angular-cli/blob/main/CHANGELOG.md)
- [Commits](https://github.com/angular/angular-cli/compare/v22.0.5...v22.0.8)

Updates `@angular-devkit/schematics` from 22.0.5 to 22.0.8
- [Release notes](https://github.com/angular/angular-cli/releases)
- [Changelog](https://github.com/angular/angular-cli/blob/main/CHANGELOG.md)
- [Commits](https://github.com/angular/angular-cli/compare/v22.0.5...v22.0.8)

Updates `@angular-eslint/builder` from 22.0.0 to 22.1.0
- [Release notes](https://github.com/angular-eslint/angular-eslint/releases)
- [Changelog](https://github.com/angular-eslint/angular-eslint/blob/main/packages/builder/CHANGELOG.md)
- [Commits](https://github.com/angular-eslint/angular-eslint/commits/v22.1.0/packages/builder)

Updates `@angular-eslint/eslint-plugin` from 22.0.0 to 22.1.0
- [Release notes](https://github.com/angular-eslint/angular-eslint/releases)
- [Changelog](https://github.com/angular-eslint/angular-eslint/blob/main/packages/eslint-plugin/CHANGELOG.md)
- [Commits](https://github.com/angular-eslint/angular-eslint/commits/v22.1.0/packages/eslint-plugin)

Updates `@angular-eslint/eslint-plugin-template` from 22.0.0 to 22.1.0
- [Release notes](https://github.com/angular-eslint/angular-eslint/releases)
- [Changelog](https://github.com/angular-eslint/angular-eslint/blob/main/packages/eslint-plugin-template/CHANGELOG.md)
- [Commits](https://github.com/angular-eslint/angular-eslint/commits/v22.1.0/packages/eslint-plugin-template)

Updates `@angular-eslint/schematics` from 22.0.0 to 22.1.0
- [Release notes](https://github.com/angular-eslint/angular-eslint/releases)
- [Changelog](https://github.com/angular-eslint/angular-eslint/blob/main/packages/schematics/CHANGELOG.md)
- [Commits](https://github.com/angular-eslint/angular-eslint/commits/v22.1.0/packages/schematics)

Updates `@angular-eslint/template-parser` from 22.0.0 to 22.1.0
- [Release notes](https://github.com/angular-eslint/angular-eslint/releases)
- [Changelog](https://github.com/angular-eslint/angular-eslint/blob/main/packages/template-parser/CHANGELOG.md)
- [Commits](https://github.com/angular-eslint/angular-eslint/commits/v22.1.0/packages/template-parser)

Updates `@angular/build` from 22.0.5 to 22.0.8
- [Release notes](https://github.com/angular/angular-cli/releases)
- [Changelog](https://github.com/angular/angular-cli/blob/main/CHANGELOG.md)
- [Commits](https://github.com/angular/angular-cli/compare/v22.0.5...v22.0.8)

Updates `@angular/compiler-cli` from 22.0.5 to 22.0.8
- [Release notes](https://github.com/angular/angular/releases)
- [Changelog](https://github.com/angular/angular/blob/main/CHANGELOG.md)
- [Commits](https://github.com/angular/angular/commits/v22.0.8/packages/compiler-cli)

---
updated-dependencies:
- dependency-name: "@angular-devkit/core"
  dependency-version: 22.0.8
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: frontend-angular-dependencies
- dependency-name: "@angular-devkit/schematics"
  dependency-version: 22.0.8
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: frontend-angular-dependencies
- dependency-name: "@angular-eslint/builder"
  dependency-version: 22.1.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: frontend-angular-dependencies
- dependency-name: "@angular-eslint/eslint-plugin"
  dependency-version: 22.1.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: frontend-angular-dependencies
- dependency-name: "@angular-eslint/eslint-plugin-template"
  dependency-version: 22.1.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: frontend-angular-dependencies
- dependency-name: "@angular-eslint/schematics"
  dependency-version: 22.1.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: frontend-angular-dependencies
- dependency-name: "@angular-eslint/template-parser"
  dependency-version: 22.1.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: frontend-angular-dependencies
- dependency-name: "@angular/build"
  dependency-version: 22.0.8
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: frontend-angular-dependencies
- dependency-name: "@angular/cdk"
  dependency-version: 22.0.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: frontend-angular-dependencies
- dependency-name: "@angular/common"
  dependency-version: 22.0.8
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: frontend-angular-dependencies
- dependency-name: "@angular/compiler"
  dependency-version: 22.0.8
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: frontend-angular-dependencies
- dependency-name: "@angular/compiler-cli"
  dependency-version: 22.0.8
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: frontend-angular-dependencies
- dependency-name: "@angular/core"
  dependency-version: 22.0.8
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: frontend-angular-dependencies
- dependency-name: "@angular/forms"
  dependency-version: 22.0.8
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: frontend-angular-dependencies
- dependency-name: "@angular/localize"
  dependency-version: 22.0.8
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: frontend-angular-dependencies
- dependency-name: "@angular/platform-browser"
  dependency-version: 22.0.8
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: frontend-angular-dependencies
- dependency-name: "@angular/router"
  dependency-version: 22.0.8
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: frontend-angular-dependencies
- dependency-name: "@ng-select/ng-select"
  dependency-version: 23.5.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: frontend-angular-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>

* More 21.1.x

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
2026-08-01 23:25:12 +00:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>shamoon
fe196eaf0e Chore(deps-dev): Bump @playwright/test from 1.61.1 to 1.62.0 in /src-ui (#13480)
* Chore(deps-dev): Bump @playwright/test from 1.61.1 to 1.62.0 in /src-ui

Bumps [@playwright/test](https://github.com/microsoft/playwright) from 1.61.1 to 1.62.0.
- [Release notes](https://github.com/microsoft/playwright/releases)
- [Commits](https://github.com/microsoft/playwright/compare/v1.61.1...v1.62.0)

---
updated-dependencies:
- dependency-name: "@playwright/test"
  dependency-version: 1.62.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Bump Playwright CI container version

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
2026-08-01 16:09:43 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
7fda0e877b Chore(deps-dev): Bump the frontend-eslint-dependencies group across 1 directory with 4 updates (#13477)
Bumps the frontend-eslint-dependencies group with 4 updates in the /src-ui directory: [@typescript-eslint/eslint-plugin](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/eslint-plugin), [@typescript-eslint/parser](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/parser), [@typescript-eslint/utils](https://github.com/typescript-eslint/typescript-eslint/tree/HEAD/packages/utils) and [eslint](https://github.com/eslint/eslint).


Updates `@typescript-eslint/eslint-plugin` from 8.62.1 to 8.65.0
- [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases)
- [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/eslint-plugin/CHANGELOG.md)
- [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v8.65.0/packages/eslint-plugin)

Updates `@typescript-eslint/parser` from 8.62.1 to 8.65.0
- [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases)
- [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/parser/CHANGELOG.md)
- [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v8.65.0/packages/parser)

Updates `@typescript-eslint/utils` from 8.62.1 to 8.65.0
- [Release notes](https://github.com/typescript-eslint/typescript-eslint/releases)
- [Changelog](https://github.com/typescript-eslint/typescript-eslint/blob/main/packages/utils/CHANGELOG.md)
- [Commits](https://github.com/typescript-eslint/typescript-eslint/commits/v8.65.0/packages/utils)

Updates `eslint` from 10.6.0 to 10.8.0
- [Release notes](https://github.com/eslint/eslint/releases)
- [Commits](https://github.com/eslint/eslint/compare/v10.6.0...v10.8.0)

---
updated-dependencies:
- dependency-name: "@typescript-eslint/eslint-plugin"
  dependency-version: 8.65.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: frontend-eslint-dependencies
- dependency-name: "@typescript-eslint/parser"
  dependency-version: 8.65.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: frontend-eslint-dependencies
- dependency-name: "@typescript-eslint/utils"
  dependency-version: 8.65.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: frontend-eslint-dependencies
- dependency-name: eslint
  dependency-version: 10.8.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: frontend-eslint-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-01 23:02:54 +00:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
5b8f4db1ca Chore(deps-dev): Bump @types/node from 26.1.0 to 26.1.1 in /src-ui (#13479)
Bumps [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) from 26.1.0 to 26.1.1.
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node)

---
updated-dependencies:
- dependency-name: "@types/node"
  dependency-version: 26.1.1
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-01 15:50:19 -07:00
github-actions[bot]GitHubgithub-actions <41898282+github-actions[bot]@users.noreply.github.com>
08ab98f3fc Changelog v3.0.5 - GHA (#13492)
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
2026-08-01 15:02:46 -07:00
shamoon 8fb73b2709 Bump version to 3.0.5 2026-08-01 14:38:08 -07:00
shamoon b8ab8327b0 Merge branch 'dev' 2026-08-01 14:37:17 -07:00
3a0ca03544 New Crowdin translations by GitHub Action (#13372)
Co-authored-by: Crowdin Bot <support+bot@crowdin.com>
2026-08-01 14:36:33 -07:00
shamoonandGitHub 882dfa78d2 Documentation: PAPERLESS_CONSUMER_IGNORE_PATTERNS clarifications (#13489) 2026-08-01 14:20:31 -07:00
Trenton HandGitHub a5645392a5 Fix: accept Whoosh-era abbreviated relative-date units (yrs, mos, wks, etc) in search queries (#13486) 2026-08-01 21:16:57 +00:00
GitHub Actions bbdf2cbd06 Auto translate strings 2026-08-01 20:33:37 +00:00
shamoonandGitHub 0bbe180f63 Fix: fix edit dialog error change detection (#13483) 2026-08-01 13:27:34 -07:00
e35452beeb Perf/fix: stop building a full-library IN filter for unrestricted RAG context (#13441)
get_context_for_document() always materialized every document id a user
can see into a Python list, even for a superuser (or no user at all),
passing it through as a SQL IN filter. For a superuser, that's the whole
library:

- Past ~32,763 documents, this crashes outright:
  sqlite3.OperationalError: too many SQL variables (SQLite's
  SQLITE_MAX_VARIABLE_NUMBER is 32766 by default, and the query already
  binds embedding + k + a NE self-exclusion clause alongside the ids).
- Below that cliff, vec0's IN-list evaluation is a nested loop (strncmp
  per row per allowed id), so it's quadratic in library size for no
  reason -- the filter was never going to exclude anything.

get_objects_for_user_owner_aware() already returns every Document for a
superuser (guardian's own with_superuser shortcut), so skipping straight
to document_ids=None changes nothing about which documents are
considered, only how we get there.

Also:
- Drop the pointless sorted() in _document_id_filters(): a SQL IN clause
  doesn't care about order, so it was pure overhead on every call.
- Add a hard guard in _build_where() so a future regression (or a
  legitimately huge permission-restricted user) fails closed -- no rows,
  a logged warning -- instead of a cryptic OperationalError. Since this
  filter scopes document access, failing closed rather than skipping the
  filter is the only safe way to handle an oversized list.

First item from VECTOR_STORE_PERF_BACKLOG.md (an audit done alongside
perf/13314-vecstore-point-delete, deferred to its own branch since that
one was already large).

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-01 07:20:30 -07:00
Trenton HandGitHub 68a769b40c Fix: make LLM index migration-check result tri-state, avoid deferred-vs-current ambiguity (#13436)
* Fix: make migration-check result tri-state to avoid deferred-vs-current ambiguity

* Test: cover update_llm_index()'s migration-check-deferred branch

* Refactor: rename COMPACT_BATCH_SIZE to BATCH_SIZE, no longer compact()-specific
2026-08-01 07:20:29 -07:00
e5b27df99c Feature: vector store schema v2 document_chunks/document_meta side tables, INTEGER document_id, point-delete (#13435)
* Feature: schema v2 -- document_chunks/document_meta side tables, document_id INTEGER, point-delete

Rewrites the sqlite-vec vector store's on-disk schema: document_id becomes
an INTEGER vec0 metadata column (was TEXT), modified moves out of vec0 into
a new document_meta side table, and a new document_chunks side table gives
O(1) per-document chunk lookup for delete/upsert instead of a full vec0
scan. compact() now streams document_chunks and document_meta across the
file-swap rebuild too (previously document_meta would have gone silently
empty after the first compaction). drop_table() clears both side tables.

Adds the single frozen m0001_v1_to_v2 migration, converting a real,
historically-shaped v1 store (the shape shipped since v3.0.0) into the v2
shape in one streaming pass, with its own hardcoded DDL rather than
delegating to any "current schema" helper.

SCHEMA_VERSION bumps 1 -> 2.

* Fix: strengthen two vacuous Task 4 regression tests

test_migration_never_delegates_to_current_schema_helpers never actually ran
the migration (missing check_and_run_migrations() call) and its source-text
assertion was tautological (the "or DROP TABLE in source" clause was always
true). Now runs the real migration and asserts spy call counts instead:
DocumentChunksTable.create/DocumentMetaTable.create are each called exactly
3 times (construction, rebuild temp file, post-swap reopen -- all via
_open_connection, never from inside apply()), and _create_vec_table is
never called from the migration path.

test_drop_table_clears_modified_times asserted via get_modified_times(),
which short-circuits on table_exists() -- checking only the vec0 table that
drop_table() drops first -- so the assertion held even if
DocumentMetaTable.delete_all() were never called. Now asserts directly
against document_meta and document_chunks row counts.

* Perf: dedupe table_exists() lookups, atomic insert counter, fewer connections in update_llm_index()

* Fix: guard compact() against unmigrated stores, apply final-review cleanups

compact() had no migration guard: on a v1-schema store, document_chunks
reads 0 (freshly created empty) while total_inserts reflects the real
cumulative count, so the bloat check nearly always rebuilt -- silently
losing document_meta (copy_all reads from the empty v1 table) while
schema_version copied across unchanged, leaving the store permanently
unmigratable. compact() now calls has_pending_migration() and no-ops with
a warning instead.

Also folds in five minor final-review findings: drop _rebuild_into's
unused int return, hoist test-local imports to module level in
test_vector_store.py, note in TestMigrations' docstring that its fake
structural migrations only exercise dispatch (not full schema
correctness), restore the comment explaining why _row() requires
document_id, and tighten increment_total_inserts' docstring to not imply
general concurrency safety beyond its single atomic statement.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-01 07:20:29 -07:00
95a48dd685 Chore: Vector Store table-gateway module (document_chunks/document_meta/index_meta) (#13428)
* Feature: add table-gateway module for document_chunks/document_meta/index_meta

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* Fix: address code review feedback on table gateways

- Document sqlite3.Row row_factory precondition in module docstring
- Strengthen test_create_is_idempotent to verify populated table survives
- Collapse three roundtrip tests into one @pytest.mark.parametrize

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-01 07:20:28 -07:00
Trenton HandGitHub aa938de747 Refactor: extract QuerySetStream, shared by search and LLM indexing (#13431) 2026-08-01 04:07:26 +00:00
Trenton HandGitHub 486b4babac Performance: sqlite-vec point-delete for document chunks (#13438)
* Refactor: extract migration infrastructure, add has_pending_migration() (#13410)

* Empty commit to try and get Codecov going
2026-07-31 19:10:59 -07:00
GitHub Actions adf1dd4e05 Auto translate strings 2026-07-31 16:15:05 +00:00
3d57f4396f Fix: key the AI suggestion cache by model and endpoint (#13449)
---------

Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
2026-07-31 16:13:22 +00:00
GitHub Actions 752c949009 Auto translate strings 2026-07-31 15:58:58 +00:00
shamoonandGitHub e37ef5aa71 Fix: validate custom field values in bulk operations (#13457) 2026-07-31 15:57:17 +00:00
shamoonandGitHub 919ffdd794 Fixhancement: better handle empty fields from AI suggestions (#13454) 2026-07-31 08:19:22 -07:00
Trenton HandGitHub 30fe172847 Chore: Drops the search shims (#13433) 2026-07-30 14:18:35 -07:00
Trenton HandGitHub e79f0d4106 Performance: Use server side iterators during LLM index updating (#13430) 2026-07-30 08:53:01 -07:00
GitHub Actions c29c9178f0 Auto translate strings 2026-07-30 14:39:47 +00:00
shamoonandGitHub baae99eb49 Fix: normalize monetary decimal symbol by locale (#13427) 2026-07-30 07:37:52 -07:00
Trenton HandGitHub f50a021ef6 Fix: fold overflowing path text in sanity checker (#13426) 2026-07-30 14:35:19 +00:00
shamoonandGitHub c8e00dca04 Fix: correct missing add field button for custom fields (#13424) 2026-07-30 07:16:25 -07:00
shamoonandGitHub 0a56018d6b Fix: hide attributes collapse / show button without UI settings (#13425) 2026-07-30 07:08:50 -07:00
350684cd6b Fix: consolidate born-digital PDF detection between archive decision and OCR (#13409)
* Fix: unify born-digital PDF detection between archive decision and OCR

should_produce_archive() and RasterisedDocumentParser.parse() each
reimplemented the "does this PDF have real text" check independently,
using different normalization of pdftotext output. Raw pdftotext output
can be non-empty (whitespace/form-feed layout padding) even when there
is no real content, so the two checks could disagree: consumer.py
treated a tagged-but-textless PDF as born-digital and skipped the
archive, while the parser's own (stricter, normalized) check found no
text and ran OCR anyway, leaving the document with no archive despite
real OCR text (GH #13387).

Both call sites now share one predicate, pdf_born_digital_text() in
paperless/parsers/utils.py, so they can no longer drift apart.

* Fix: restore extract_text seam for born-digital detection in parse()

parse() had switched to calling pdf_born_digital_text() directly for
its initial text/born-digital check, bypassing the parser's own
extract_text instance method. That broke test mockability (tests patch
tesseract_parser.extract_text to control the born-digital decision)
and caused CI failures with mismatched OCR call counts and text.

Split pdf_born_digital_text() into is_born_digital_text(text, path,
log) - a pure decision function - and a thin pdf_born_digital_text()
wrapper for callers without text in hand (consumer.should_produce_archive).
parse() now extracts via self.extract_text(None, document_path) and
passes the result to is_born_digital_text(), restoring the seam with
no change to production behavior.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* Cleanup: simplify born-digital detection, close #13387 test gap

Simplification pass over the born-digital detection consolidation:
- is_born_digital_text(): drop the has_text temp for an early return.
- consumer.py: standardize the archive-decision log lines on plain
  hyphens (was a mix of em-dash and hyphen) and hoist the duplicated
  text_length computation.
- Parametrize TestPdfBornDigitalText instead of four near-identical
  tests.

Code review follow-up: the existing tests only ever exercised
pdf_born_digital_text() through mocks, so the actual #13387 scenario
(a tagged PDF whose only "text" is layout padding) was never checked
against real pdftotext/pikepdf output - a regression in the
normalize-before-decide logic itself would have gone undetected.
Moved tagged_no_text_pdf_file from parsers/conftest.py up to the
shared paperless/tests/conftest.py (it was previously only visible to
tests under parsers/) and added a non-mocked regression test against
the real sample file.

Also fixed two stale comments in test_consumer.py referencing a
_extract_text_for_archive_check helper that no longer exists.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-29 12:41:09 -07:00
shamoonandGitHub 668fa77428 Fix: prevent scale vs page loop in pngx PDF viewer (#13406) 2026-07-29 09:25:37 -07:00
shamoonandGitHub 5bd72014a6 Fix: handle hidden line breaks in email subjects when sending email (#13402) 2026-07-29 14:35:30 +00:00
Trenton HandGitHub bbb9c86ba4 Fix: avoid NotSupportedError from document_importer on MariaDB (#13400) 2026-07-29 07:14:13 -07:00
GitHub Actions 09c9fa03cf Auto translate strings 2026-07-28 18:28:12 +00:00
shamoonandGitHub 04779d72bb Tweak: adjust doc details button toolbar flow (#13382) 2026-07-28 11:26:37 -07:00
Trenton HandGitHub 1b32b9d678 Fix: exclude next-period start from relative date-range filters (#13381)
* Fix: exclude next-period start from relative date-range filters

Tantivy's [lo TO hi] range is inclusive on both ends, but computed upper
bounds (keyword ranges, YYYY/YYYYMM/YYYYMMDD tokens) represent the start of
the next period. Use half-open [lo TO hi} for those so e.g. "previous month"
no longer matches the 1st of the current month.

* Adds a regression test down to the second check for the hi range
2026-07-28 16:58:38 +00:00
Trenton HandGitHub 12d318deff Documentation: Add Password Removal workflow action documentation (#13377)
Addresses discussion #13373. Documents that password removal creates a
new document version via re-consumption of the decrypted file rather
than editing in place, and explains why the Consumption Started trigger
produces an initial un-OCR'd version followed by a properly processed
one.
2026-07-28 09:21:14 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
9c98f30d3f Chore(deps): Bump pymdown-extensions in the uv group across 1 directory (#13378)
Bumps the uv group with 1 update in the / directory: [pymdown-extensions](https://github.com/facelessuser/pymdown-extensions).


Updates `pymdown-extensions` from 10.21.3 to 11.0
- [Release notes](https://github.com/facelessuser/pymdown-extensions/releases)
- [Commits](https://github.com/facelessuser/pymdown-extensions/compare/10.21.3...11.0)

---
updated-dependencies:
- dependency-name: pymdown-extensions
  dependency-version: '11.0'
  dependency-type: indirect
  dependency-group: uv
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-28 15:29:20 +00:00
shamoonandGitHub 6a1d7b1bca Fix: close non-atomic db connections in before_task_publish (#13366) 2026-07-28 07:21:02 -07:00
shamoonandGitHub de9f520143 Chore/fix: refactor frontend task service (#13365) 2026-07-28 00:34:30 -07:00
shamoonandGitHub a68c487d60 Fix: fix Enter selection in search autocomplete (#13361) 2026-07-27 23:33:13 -07:00
github-actions[bot]andGitHub 217421bceb Documentation: Add v3.0.4 changelog (#13356) 2026-07-27 21:21:10 -07:00
shamoon 3bc03bbaec Bump version to 3.0.4 2026-07-27 20:14:39 -07:00
shamoon aaee24ac0b Merge branch 'dev' 2026-07-27 20:14:04 -07:00
674dab29df New Crowdin translations by GitHub Action (#13308)
Co-authored-by: Crowdin Bot <support+bot@crowdin.com>
2026-07-27 20:12:46 -07:00
GitHub Actions b2b2f7b488 Auto translate strings 2026-07-28 00:27:34 +00:00
shamoonandGitHub 0b588c2d31 Fix: prevent pdfjs highlight scrolling from affecting the entire page (#13355) 2026-07-27 17:25:52 -07:00
Trenton HandGitHub dcff067dc1 Fix: don't skip OCR/archive for tagged PDFs with no actual text (#13351) 2026-07-27 16:52:29 -07:00
shamoonandGitHub 7eee8538ec Performance: more performant filtering diacritic normalization (#13347) 2026-07-27 22:13:16 +00:00
98b66fdf24 Perf: prefetch notes and custom fields for LLM index text building (#13350)
build_llm_index_text queried Note and CustomFieldInstance (plus its
field FK) per document, uncovered by the earlier correspondent/type/
storage_path/tags prefetch fix. Add notes and custom_fields__field to
the rebuild and scoped document querysets, and read from doc.notes.all()
instead of a fresh Note.objects.filter() so the prefetch is actually used.

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-27 14:50:37 -07:00
GitHub Actions 693a111c5a Auto translate strings 2026-07-27 19:35:15 +00:00
Trenton HandGitHub b19edd0b74 Fix: dedupe permission-visible documents when combined with multi-tag ALL filtering (#13331) (#13345)
The permission filter OR'd three querysets together on top of a
queryset that could already carry two independent tags__id__all
joins, letting a document that matched more than one branch (e.g.
unowned + group-permissioned) come back twice. Replaced it with a
single id__in filter against the existing permitted_document_ids
helper, which is join-free and can't hit this.
2026-07-27 19:33:26 +00:00
Trenton HandGitHub 0e98a7f1ce Performance: Scope llm index updates to actually modified documents (#13322)
* Perf: scope bulk_update_documents' LLM index refresh to the edited document ids instead of the whole library

* Perf: select_related/prefetch_related for the scoped LLM index batch

* Perf: select_related/prefetch_related for the full LLM index rebuild path

* Fix: address Copilot review findings on LLM index scoping
2026-07-27 10:23:43 -07:00
GitHub Actions 14517062c4 Auto translate strings 2026-07-27 16:53:33 +00:00
shamoonandGitHub 0c0faf7f80 Fixhancement: pass LLM output language to chat if specified (#13340) 2026-07-27 16:51:31 +00:00
SandroandGitHub eda79603fe Documentation: fix PAPERLESS_AI_LLM_OUTPUT_LANGUAGE heading level (#13341) 2026-07-27 09:37:09 -07:00
shamoonandGitHub 8f017942d9 Chore: resolve npm provenance issues with chokidar and semver (#13323) 2026-07-26 16:25:49 -07:00
GitHub Actions f4e7a2e9e4 Auto translate strings 2026-07-26 22:38:16 +00:00
shamoonandGitHub 317e534aa0 Fix: ensure preview reload on live changes (#13321) 2026-07-26 22:36:03 +00:00
Trenton HandGitHub 3ba6d0325a Fix: clamp out-of-range MailRule.order and ApplicationConfiguration DPI/page fields before smallint migration (#13316) 2026-07-26 22:11:27 +00:00
Trenton HandGitHub 318520a0f1 Fix: add docstring to DocumentClassifierSchema for cleaner LLM tool description (#13315) 2026-07-26 22:00:06 +00:00
Trenton HandGitHub ea1f3653a9 Fix: guard build_document_node against stale FK on deleted correspondent/doc type (#13318) 2026-07-26 21:45:59 +00:00
shamoonandGitHub 423d4b9f2d Fix: prevent search filter loss when closing document with Escape key (#13317) 2026-07-26 14:17:26 -07:00
shamoonandGitHub 8bc4e5ee94 Fix: fix frontend permissions display for stats/system perms (#13305) 2026-07-26 02:56:06 -07:00
github-actions[bot]GitHubgithub-actions <41898282+github-actions[bot]@users.noreply.github.com>shamoon
65fe6aacc7 Documentation: Add v3.0.3 changelog (#13300)
* Changelog v3.0.3 - GHA

* Update changelog.md

---------

Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
2026-07-25 18:57:03 -07:00
shamoon b5a23cff2f Bump version to 3.0.3 2026-07-25 16:32:39 -07:00
aa90f79c21 New Crowdin translations by GitHub Action (#13297)
Co-authored-by: Crowdin Bot <support+bot@crowdin.com>
2026-07-25 16:31:40 -07:00
e704bf88dc New Crowdin translations by GitHub Action (#13239)
Co-authored-by: Crowdin Bot <support+bot@crowdin.com>
2026-07-25 16:30:17 -07:00
Trenton HandGitHub 1646756c6c Docs: warn bare-metal users about stale files when upgrading (#13296) 2026-07-25 14:06:31 -07:00
shamoon 48d6ff13f2 Tweak: use css grid for navbar 2026-07-25 09:13:34 -07:00
GitHub Actions bfe2d92619 Auto translate strings 2026-07-25 07:13:46 +00:00
shamoonandGitHub 4f0845b094 Fixhancement: PAPERLESS_ALLAUTH_TRUSTED_PROXY_COUNT (#13281) 2026-07-25 00:12:12 -07:00
shamoonandGitHub dee568e2a6 Tweak: adjust top navbar wrapping when AI chat button visible (#13280) 2026-07-24 23:50:10 -07:00
shamoon f60a2af841 Merge branch 'main' into dev 2026-07-24 19:08:23 -07:00
Trenton HandGitHub 33cd48a17f Fix: preserve document fields during Gotenberg conversion (#13271)
Gotenberg's LibreOffice route enables updateIndexes by default, which refreshes dynamic fields (e.g. auto-dates) to the current date. Disable it so field values are preserved as authored.
2026-07-24 13:59:51 -07:00
Trenton HandGitHub d76dbf5366 Documentation: Add the NumPy CPU baseline increase to the migration guide (#13269) 2026-07-24 09:18:07 -07:00
Trenton HandGitHub b66182cd7b Fix: Makes the email date aware as soon as possible during parsing (#13266) 2026-07-24 15:39:25 +00:00
Trenton HandGitHub 3fe6562c57 Fix: Handle a plain string as Celery sometimes provides for the traceback (#13267) 2026-07-24 15:27:57 +00:00
Trenton HandGitHub 737d568f80 Fix: Emit the torch index into the requirements.txt for people still using it (#13265) 2026-07-24 08:07:52 -07:00
Matthias MastandGitHub b992f9fc05 Fix: handle notes without a user when building the search index (#13260) 2026-07-24 06:39:11 -07:00
github-actions[bot]GitHubgithub-actions <41898282+github-actions[bot]@users.noreply.github.com>
ed714b8a5a Changelog v3.0.2 - GHA (#13244)
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
2026-07-23 18:56:06 -07:00
shamoon e0ab6a2e75 Merge branch 'dev' 2026-07-23 18:13:49 -07:00
shamoon 6d249b4932 Bump version to 3.0.2 2026-07-23 18:12:59 -07:00
github-actions[bot]GitHubgithub-actions <41898282+github-actions[bot]@users.noreply.github.com>
90ea4f27c4 Changelog v3.0.1 - GHA (#13240)
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
2026-07-23 17:59:34 -07:00
shamoonandGitHub 371c4f57d3 Fix: fix broken migration in 3.0.1 (#13242) 2026-07-23 17:59:03 -07:00
stumpylog 5569447ea0 Bumps the version to 3.0.1 everywhere 2026-07-23 15:59:53 -07:00
stumpylog 17796357ee Merge remote-tracking branch 'origin/dev' 2026-07-23 15:55:41 -07:00
b47b9800a1 New Crowdin translations by GitHub Action (#13207)
Co-authored-by: Crowdin Bot <support+bot@crowdin.com>
2026-07-23 15:54:05 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
0be884d441 Chore(deps-dev): Bump postcss (#13236)
Bumps the npm_and_yarn group with 1 update in the /src/paperless_mail/templates directory: [postcss](https://github.com/postcss/postcss).


Updates `postcss` from 8.5.6 to 8.5.22
- [Release notes](https://github.com/postcss/postcss/releases)
- [Changelog](https://github.com/postcss/postcss/blob/main/CHANGELOG.md)
- [Commits](https://github.com/postcss/postcss/compare/8.5.6...8.5.22)

---
updated-dependencies:
- dependency-name: postcss
  dependency-version: 8.5.22
  dependency-type: indirect
  dependency-group: npm_and_yarn
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-23 20:21:15 +00:00
GitHub Actions fce2fbb8f6 Auto translate strings 2026-07-23 19:22:48 +00:00
fa01396dfd Fix: selection_data re-derives the filtered document set 5 times over (#13229)
* Fix: selection_data re-derives the filtered document set 5 times over

_get_selection_data_for_queryset() (powers ?include_selection_data=true on
the document list and search endpoints) computed document_count for
Correspondent/DocumentType/StoragePath/Tag/CustomField by embedding the
caller's full filtered queryset -- filters plus the permission check -- as
a subquery inside 5 separate Count(filter=Q(documents__in=queryset), ...)
calls. Each one re-evaluates that whole queryset from scratch.

Resolve the document ids once into a concrete list and reuse it across all
five annotations instead. For Tag/CustomField specifically (M2M via a
through-model table), also route through annotate_document_count_by_ids()
-- extracted from the tag/custom-field document_count fix (#13203) -- to
avoid the same Count(filter=Q(id__in=...), distinct=True)-on-an-M2M-relation
anti-pattern diagnosed there.

On a 400k-document/1,000-tag corpus, the correspondents portion alone
previously didn't finish within several minutes (killed twice while
investigating, including one run that left a zombie query still consuming
a CPU 30+ minutes later). All five queries together now complete in
~30-35s. Root-caused from a real report (paperless-ngx#13201) via a
different, already-fixed query (#13205) -- this one hasn't been reported
in the wild yet, found by auditing the same call path.

Depends on #13203 for annotate_document_count_by_ids().

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

* Address review feedback: drop unnecessary ordering before collecting ids

queryset passed into _get_selection_data_for_queryset() carries the
default/user-specified ordering, which is irrelevant once we're only
collecting a flat id list. Clearing it removes a pointless sort.

No measurable change in benchmarking at 400k documents -- the id-list
materialization/IN-clause cost still dominates -- but it's a free,
strictly-correct cleanup, not just noise-neutral in the other direction.

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-23 19:21:20 +00:00
GitHub Actions c9443e890f Auto translate strings 2026-07-23 18:43:46 +00:00
21e4721a82 Chore: reword "Settings > Default permissions" description for clarity (#13232)
---------

Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
2026-07-23 18:41:26 +00:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
1985da14f8 Chore(deps): Bump pyasn1 in the uv group across 1 directory (#13237)
Bumps the uv group with 1 update in the / directory: [pyasn1](https://github.com/pyasn1/pyasn1).


Updates `pyasn1` from 0.6.3 to 0.6.4
- [Release notes](https://github.com/pyasn1/pyasn1/releases)
- [Changelog](https://github.com/pyasn1/pyasn1/blob/main/CHANGES.rst)
- [Commits](https://github.com/pyasn1/pyasn1/compare/v0.6.3...v0.6.4)

---
updated-dependencies:
- dependency-name: pyasn1
  dependency-version: 0.6.4
  dependency-type: indirect
  dependency-group: uv
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-23 18:26:56 +00:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
59a737f669 Chore(deps): Bump the uv group across 1 directory with 8 updates (#13189)
---
updated-dependencies:
- dependency-name: aiohttp
  dependency-version: 3.14.1
  dependency-type: indirect
- dependency-name: cryptography
  dependency-version: 48.0.1
  dependency-type: indirect
- dependency-name: pi-heif
  dependency-version: 1.3.0
  dependency-type: indirect
- dependency-name: pillow
  dependency-version: 12.3.0
  dependency-type: indirect
- dependency-name: pyjwt
  dependency-version: 2.13.0
  dependency-type: indirect
- dependency-name: setuptools
  dependency-version: 83.0.0
  dependency-type: indirect
- dependency-name: torch
  dependency-version: 2.13.0
  dependency-type: direct:production
- dependency-name: tornado
  dependency-version: 6.5.7
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-23 17:58:06 +00:00
Trenton HandGitHub 90bee4285f Fix: exclude the source document from its own RAG similarity results (#13233)
query_similar_documents() never excluded the querying document itself, so a document could appear in its own "similar documents" context, duplicating its content into the AI-suggestions prompt and inflating prompt size/tokens  unnecessarily. Add NE filter support to the vector store and exclude the source document's id at query time.
2026-07-23 10:25:30 -07:00
Trenton HandGitHub 7dca0bfa6a Fix (#13214): split mailrule maximum_age clamp into its own migration (#13231)
Avoids a Postgres error where the clamp UPDATE and the following ALTER TABLE on paperless_mail_mailrule share a transaction, but the table's FKs are deferrable, so pending trigger events block the ALTER. 

Also fixes a verbose_name mismatch in migration 0013.
2026-07-23 17:09:28 +00:00
GitHub Actions 3a34bdc7b0 Auto translate strings 2026-07-23 16:54:02 +00:00
shamoonandGitHub c7899a9a84 Chore: mark yes in confirm button for translation (#13225) 2026-07-23 16:52:04 +00:00
shamoonandGitHub 82fc1c6cee Fix: correct URL for W001 check (#13220) 2026-07-23 16:39:00 +00:00
shamoonandGitHub cd3c162e6e Fix: ensure create dialog uses correct attribute type (#13221) 2026-07-23 16:22:27 +00:00
FabianandGitHub 50c88dab7a Fix: correct database engine from postgres to postgresql in the Porttainer Compose file (#13213) 2026-07-23 07:40:39 -07:00
GitHub Actions 5744a75e6a Auto translate strings 2026-07-23 14:24:56 +00:00
02c6b5f3e4 Fix (beta): remove unnecessary .distinct() dominating document list queries (#13205)
DocumentViewSet.get_queryset() carried a blanket .distinct() left over from
an older query shape (predating the version-file feature and the
correlated-subquery rewrite of num_notes/effective_content). Nothing in the
current base queryset can produce duplicate document rows: select_related
is all FK-to-PK, and the permission filter is a boolean id__in predicate,
not a join. id (the PK) is also always selected and inherently unique, so
.distinct() was a structural no-op for correctness.

It was not a no-op for cost. EXPLAIN showed it forcing a full sort-then-
dedupe over the entire permission-visible document set before LIMIT could
apply -- 340k rows for a 25-row page, with the effective_content/num_notes
correlated subqueries re-executed once per row as a result. This was the
dominant cost behind the original report's document overview slowness,
well beyond the get_user_can_change N+1 fixed separately.

Removing it isn't safe on its own, though -- auditing found two existing
filters that rely on an M2M join *without* deduping themselves, previously
papered over by the blanket .distinct():
- InboxFilter: a document with two tags both flagged is_inbox_tag=True
  (nothing prevents that) would be returned twice for ?is_in_inbox=true.
- CustomFieldsFilter: a document with multiple custom field instances
  matching different OR-ed branches would be returned once per match.

Fixed both locally, matching the pattern ObjectFilter already used for
tags__id__in/custom_fields__id__in. Added regression tests for both --
confirmed they fail without the local .distinct() calls.

Benchmarked against the same 400k-document/1,000-tag corpus: document list
wall-clock for a permission-restricted user drops from ~15.8s to ~2.25s
(~7x), closing most of the previous gap to a superuser's ~1.8s.

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-23 07:22:46 -07:00
shamoon b566232542 Documentation: correct search docs for v3 2026-07-23 06:25:33 -07:00
shamoonandGitHub be40356d2e Fix: also wrap non-breaking words in chat messages, handle whitespace (#13211) 2026-07-22 22:25:19 -07:00
GitHub Actions 5ea8cf2335 Auto translate strings 2026-07-23 04:30:13 +00:00
84a04301d1 Fix: tag/custom-field document_count scales badly with tag count (#13203)
* Fix (beta): tag/custom-field document_count scales badly with tag count

TagViewSet and CustomFieldViewSet's document_count annotation used a
per-row correlated subquery (annotate_document_count_for_related_queryset),
executed once per tag/custom-field row. Fine at a few dozen rows, but at
~1,000 tags it degrades to a full per-tag GroupAggregate over the tags M2M
table -- 6s+ in production reports, confirmed via EXPLAIN (loops=1000).

Replaced with a single, independent GROUP BY over the through table
(permission filter expressed as a plain WHERE, not an aggregate FILTER),
then injected via Case/When. Also tried a more "obvious" fix -- a plain
Count(filter=Q(id__in=permitted_ids), distinct=True) directly on the M2M
relation -- but that's worse: Postgres fails to plan the id__in check as a
semi-join once a large M2M bridge table is involved, and instead re-checks
subquery membership once per joined row.

Benchmarked against a synthetic corpus (400k documents, 1,000 tags @ ~5/doc,
matching real-world reports in discussion #13161): tag list wall-clock drops
from ~180s to ~8s for a permission-restricted user, ~21s to ~8s for a
superuser. No measurable change at typical home-instance scale (tens of
tags).

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

* Address review feedback: restore validation, scope aggregation to queryset

- Restore the document_count_source_field validation helper dropped during
  the refactor -- misconfiguring a viewset (document_count_through set
  without document_count_source_field) now fails fast with a clear error
  again instead of an obscure runtime failure.
- Restrict annotate_document_count_for_related_queryset()'s through-table
  aggregation to rows whose related_object_field is one of the annotated
  queryset's pks. No-op for the main tag-list call site (queryset is all
  tags), but avoids unnecessary work for narrower callers like the tag
  descendants branch in TagViewSet.list().

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-22 21:28:49 -07:00
GitHub Actions 3edda48324 Auto translate strings 2026-07-23 04:04:23 +00:00
f86bc57880 Fix: batch document user_can_change checks to avoid per-row N+1 (#13204)
* Fix (beta): batch document user_can_change checks to avoid per-row N+1

DocumentSerializer.get_user_can_change() built a fresh
ObjectPermissionChecker and issued a guardian permission-table query for
every document row not owned by the requesting user -- correct, but O(N)
per page load for any non-superuser viewing documents owned by others.

BulkPermissionMixin already batches this exact lookup (2 queries total,
regardless of page size) for Correspondent/Tag/DocumentType/CustomField,
but was gated behind the rarely-used `full_perms` flag and DocumentViewSet
didn't inherit it at all. Changed the gate to "any list action" (cheap:
just two extra queries per page) and added BulkPermissionMixin to
DocumentViewSet, then updated get_user_can_change to consult that batched
context before falling back to a fresh guardian check.

Preserves guardian's own superuser shortcut explicitly (has_perm() special-
cases is_superuser without a query; the batched-context path doesn't, so it
needed its own check) -- covered by a new regression test, since no
existing test exercised a superuser viewing an other-owned document.

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

* Fix (beta): don't batch permissions for tantivy search results

UnifiedSearchViewSet.list() returns SearchHit/dict-like objects for
text/title/query/more_like_id search requests, not Document ORM instances.
Adding BulkPermissionMixin to DocumentViewSet (previous commit) meant its
get_serializer_context() ran for search responses too, and its
_get_object_perms() -- which expects real model instances with .pk --
crashed on the dict-like hits with AttributeError, turning every search
request into a 400.

Skip the batching specifically for search requests (existing
_is_search_request() check) by calling past BulkPermissionMixin in the
MRO; non-search list() calls (which return a real Document queryset) are
unaffected and still get the batching.

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-23 04:02:46 +00:00
GitHub Actions 97662b6c5c Auto translate strings 2026-07-23 01:55:12 +00:00
shamoonandGitHub 6e052d5507 Fix: fix app title restoration (#13208) 2026-07-22 18:53:45 -07:00
shamoonandGitHub afb19fe637 Fix: handle long wrapping titles in AI chat document list (#13206) 2026-07-22 16:15:14 -07:00
226 changed files with 17150 additions and 15014 deletions
+1 -1
View File
@@ -191,7 +191,7 @@ jobs:
runs-on: ubuntu-24.04
permissions:
contents: read
container: mcr.microsoft.com/playwright:v1.61.1-noble
container: mcr.microsoft.com/playwright:v1.62.0-noble
env:
PLAYWRIGHT_BROWSERS_PATH: /ms-playwright
PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD: 1
+1 -1
View File
@@ -86,7 +86,7 @@ jobs:
# ---- Prepare Release ----
- name: Generate requirements file
run: |
uv export --quiet --no-dev --all-extras --format requirements-txt --output-file requirements.txt
uv export --quiet --no-dev --all-extras --emit-index-url --format requirements-txt --output-file requirements.txt
- name: Compile messages
env:
PAPERLESS_SECRET_KEY: "ci-release-not-a-real-secret"
+1 -1
View File
@@ -56,7 +56,7 @@ services:
environment:
PAPERLESS_REDIS: redis://broker:6379
PAPERLESS_DBHOST: db
PAPERLESS_DBENGINE: postgres
PAPERLESS_DBENGINE: postgresql
env_file:
- stack.env
volumes:
@@ -0,0 +1,12 @@
#!/command/with-contenv /usr/bin/bash
# shellcheck shell=bash
declare -r log_prefix="[init-llmindex-migrate]"
echo "${log_prefix} Checking for pending LLM index migrations..."
cd "${PAPERLESS_SRC_DIR}"
if [[ -n "${USER_IS_NON_ROOT}" ]]; then
python3 manage.py document_llmindex migrate
else
s6-setuidgid paperless python3 manage.py document_llmindex migrate
fi
@@ -0,0 +1 @@
oneshot
@@ -0,0 +1 @@
/etc/s6-overlay/s6-rc.d/init-llmindex-migrate/run
+28 -16
View File
@@ -151,6 +151,23 @@ the background.
### Bare Metal Route {#bare-metal-updating}
!!! warning
Extracting a new release archive on top of an existing installation
(e.g. `tar -xf paperless-ngx-vX.Y.Z.tar.xz -C /opt/paperless`) only adds
and overwrites files -- it does not remove files that were deleted in
the new release. Leftover files from an old version, such as
superseded database migrations, can remain on disk and cause errors
like `NodeNotFoundError` during `manage.py migrate`.
Before unpacking a new release over an existing bare-metal
installation, remove the previous source tree first (everything
except your `media`, `data`, and `consume` directories and your
`paperless.conf`/`.env`), or unpack into a fresh directory and move
your persistent data and configuration over. Installations updated via
`git pull` on a clean checkout are not affected, since Git removes
files that no longer exist upstream.
After grabbing the new release and unpacking the contents, do the
following:
@@ -195,6 +212,16 @@ following:
This is a no-op if the index is already up to date, so it is safe to
run on every upgrade.
5. Migrate the LLM index if needed.
```shell-session
cd src
python3 manage.py document_llmindex migrate
```
This is a no-op if the index schema is already current, so it is safe
to run on every upgrade.
### Database Upgrades
Paperless-ngx is compatible with Django-supported versions of PostgreSQL and MariaDB and it is generally
@@ -272,8 +299,6 @@ optional arguments:
-sm, --split-manifest
-z, --zip
-zn, --zip-name
--zip-compression
--zip-compression-level
--data-only
--no-progress-bar
--passphrase
@@ -336,19 +361,6 @@ If `-z` or `--zip` is provided, the export will be a zip file
in the target directory, named according to the current local date or the
value set in `-zn` or `--zip-name`.
The compression method for the zip can be set with `--zip-compression`
(`stored`, `deflated` (default), `bzip2`, `lzma`, or `zstd`) and tuned with
`--zip-compression-level` (deflated: 09, bzip2: 19, zstd: -2222; ignored
for `stored` and `lzma`). Both options require `--zip`.
!!! warning
`zstd` compression requires Python 3.14 or newer on **both** the machine
creating the export and any machine importing it. An archive compressed with
`zstd` (or `lzma`/`bzip2` where those modules are unavailable) cannot be
imported on a runtime that lacks the codec; the importer will refuse it with
a clear error. The default `deflated` is universally readable.
If `--data-only` is provided, only the database will be exported. This option is intended
to facilitate database upgrades without needing to clean documents and thumbnails from the media directory.
@@ -530,7 +542,7 @@ index is updated automatically on the schedule set by
can manage it manually:
```
document_llmindex {rebuild,update,compact}
document_llmindex {rebuild,update,compact,migrate}
```
Specify `rebuild` to build the index from scratch from all documents in the database. Use
+199
View File
@@ -1,5 +1,204 @@
# Changelog
## paperless-ngx 3.0.5
### Bug Fixes
- Fix: accept Whoosh-era abbreviated relative-date units (yrs, mos, wks, etc) in search queries [@stumpylog](https://github.com/stumpylog) ([#13486](https://github.com/paperless-ngx/paperless-ngx/pull/13486))
- Fix: fix edit dialog error change detection [@shamoon](https://github.com/shamoon) ([#13483](https://github.com/paperless-ngx/paperless-ngx/pull/13483))
- Fix: key the AI suggestion cache by model and endpoint [@lunetics](https://github.com/lunetics) ([#13449](https://github.com/paperless-ngx/paperless-ngx/pull/13449))
- Fix: validate custom field values in bulk operations [@shamoon](https://github.com/shamoon) ([#13457](https://github.com/paperless-ngx/paperless-ngx/pull/13457))
- Fixhancement: better handle empty fields from AI suggestions [@shamoon](https://github.com/shamoon) ([#13454](https://github.com/paperless-ngx/paperless-ngx/pull/13454))
- Fix: normalize monetary decimal symbol by locale [@shamoon](https://github.com/shamoon) ([#13427](https://github.com/paperless-ngx/paperless-ngx/pull/13427))
- Fix: fold overflowing path text in sanity checker [@stumpylog](https://github.com/stumpylog) ([#13426](https://github.com/paperless-ngx/paperless-ngx/pull/13426))
- Fix: correct delayed add field button for custom fields [@shamoon](https://github.com/shamoon) ([#13424](https://github.com/paperless-ngx/paperless-ngx/pull/13424))
- Fix: hide attributes collapse / show button without UI settings [@shamoon](https://github.com/shamoon) ([#13425](https://github.com/paperless-ngx/paperless-ngx/pull/13425))
- Fix: consolidate born-digital PDF detection between archive decision and OCR [@stumpylog](https://github.com/stumpylog) ([#13409](https://github.com/paperless-ngx/paperless-ngx/pull/13409))
- Fix: prevent scale vs page loop in pngx PDF viewer [@shamoon](https://github.com/shamoon) ([#13406](https://github.com/paperless-ngx/paperless-ngx/pull/13406))
- Fix: handle hidden line breaks in email subjects when sending email [@shamoon](https://github.com/shamoon) ([#13402](https://github.com/paperless-ngx/paperless-ngx/pull/13402))
- Fix: avoid NotSupportedError from document\_importer on MariaDB [@stumpylog](https://github.com/stumpylog) ([#13400](https://github.com/paperless-ngx/paperless-ngx/pull/13400))
- Fix: exclude next-period start from relative date-range filters [@stumpylog](https://github.com/stumpylog) ([#13381](https://github.com/paperless-ngx/paperless-ngx/pull/13381))
- Fix: close non-atomic db connections in before\_task\_publish [@shamoon](https://github.com/shamoon) ([#13366](https://github.com/paperless-ngx/paperless-ngx/pull/13366))
- Chore/fix: refactor frontend task service [@shamoon](https://github.com/shamoon) ([#13365](https://github.com/paperless-ngx/paperless-ngx/pull/13365))
- Fix: fix Enter selection in search autocomplete [@shamoon](https://github.com/shamoon) ([#13361](https://github.com/paperless-ngx/paperless-ngx/pull/13361))
### Documentation
- Documentation: PAPERLESS\_CONSUMER\_IGNORE\_PATTERNS clarifications [@shamoon](https://github.com/shamoon) ([#13489](https://github.com/paperless-ngx/paperless-ngx/pull/13489))
- Documentation: Add Password Removal workflow action documentation [@stumpylog](https://github.com/stumpylog) ([#13377](https://github.com/paperless-ngx/paperless-ngx/pull/13377))
### Dependencies
- Chore(deps): Bump pymdown-extensions from 10.21.3 to 11.0 in the uv group across 1 directory @[dependabot[bot]](https://github.com/apps/dependabot) ([#13378](https://github.com/paperless-ngx/paperless-ngx/pull/13378))
### All App Changes
<details>
<summary>21 changes</summary>
- Fix: accept Whoosh-era abbreviated relative-date units (yrs, mos, wks, etc) in search queries [@stumpylog](https://github.com/stumpylog) ([#13486](https://github.com/paperless-ngx/paperless-ngx/pull/13486))
- Fix: fix edit dialog error change detection [@shamoon](https://github.com/shamoon) ([#13483](https://github.com/paperless-ngx/paperless-ngx/pull/13483))
- Performance: sqlite-vec point-delete for document chunks [@stumpylog](https://github.com/stumpylog) ([#13438](https://github.com/paperless-ngx/paperless-ngx/pull/13438))
- Fix: key the AI suggestion cache by model and endpoint [@lunetics](https://github.com/lunetics) ([#13449](https://github.com/paperless-ngx/paperless-ngx/pull/13449))
- Fix: validate custom field values in bulk operations [@shamoon](https://github.com/shamoon) ([#13457](https://github.com/paperless-ngx/paperless-ngx/pull/13457))
- Fixhancement: better handle empty fields from AI suggestions [@shamoon](https://github.com/shamoon) ([#13454](https://github.com/paperless-ngx/paperless-ngx/pull/13454))
- Performance: Use server side iterators during LLM index updating [@stumpylog](https://github.com/stumpylog) ([#13430](https://github.com/paperless-ngx/paperless-ngx/pull/13430))
- Fix: normalize monetary decimal symbol by locale [@shamoon](https://github.com/shamoon) ([#13427](https://github.com/paperless-ngx/paperless-ngx/pull/13427))
- Fix: fold overflowing path text in sanity checker [@stumpylog](https://github.com/stumpylog) ([#13426](https://github.com/paperless-ngx/paperless-ngx/pull/13426))
- Fix: correct delayed add field button for custom fields [@shamoon](https://github.com/shamoon) ([#13424](https://github.com/paperless-ngx/paperless-ngx/pull/13424))
- Fix: hide attributes collapse / show button without UI settings [@shamoon](https://github.com/shamoon) ([#13425](https://github.com/paperless-ngx/paperless-ngx/pull/13425))
- Fix: consolidate born-digital PDF detection between archive decision and OCR [@stumpylog](https://github.com/stumpylog) ([#13409](https://github.com/paperless-ngx/paperless-ngx/pull/13409))
- Fix: prevent scale vs page loop in pngx PDF viewer [@shamoon](https://github.com/shamoon) ([#13406](https://github.com/paperless-ngx/paperless-ngx/pull/13406))
- Fix: handle hidden line breaks in email subjects when sending email [@shamoon](https://github.com/shamoon) ([#13402](https://github.com/paperless-ngx/paperless-ngx/pull/13402))
- Fix: avoid NotSupportedError from document\_importer on MariaDB [@stumpylog](https://github.com/stumpylog) ([#13400](https://github.com/paperless-ngx/paperless-ngx/pull/13400))
- Tweak: adjust doc details button toolbar flow [@shamoon](https://github.com/shamoon) ([#13382](https://github.com/paperless-ngx/paperless-ngx/pull/13382))
- Fix: exclude next-period start from relative date-range filters [@stumpylog](https://github.com/stumpylog) ([#13381](https://github.com/paperless-ngx/paperless-ngx/pull/13381))
- Chore(deps): Bump pymdown-extensions from 10.21.3 to 11.0 in the uv group across 1 directory @[dependabot[bot]](https://github.com/apps/dependabot) ([#13378](https://github.com/paperless-ngx/paperless-ngx/pull/13378))
- Fix: close non-atomic db connections in before\_task\_publish [@shamoon](https://github.com/shamoon) ([#13366](https://github.com/paperless-ngx/paperless-ngx/pull/13366))
- Chore/fix: refactor frontend task service [@shamoon](https://github.com/shamoon) ([#13365](https://github.com/paperless-ngx/paperless-ngx/pull/13365))
- Fix: fix Enter selection in search autocomplete [@shamoon](https://github.com/shamoon) ([#13361](https://github.com/paperless-ngx/paperless-ngx/pull/13361))
</details>
## paperless-ngx 3.0.4
### Bug Fixes
- Fix: prevent pdfjs highlight scrolling from affecting the entire page [@shamoon](https://github.com/shamoon) ([#13355](https://github.com/paperless-ngx/paperless-ngx/pull/13355))
- Fix: don't skip OCR/archive for tagged PDFs with no actual text [@stumpylog](https://github.com/stumpylog) ([#13351](https://github.com/paperless-ngx/paperless-ngx/pull/13351))
- Performance: more efficient diacritic normalization in dropdown filtering [@shamoon](https://github.com/shamoon) ([#13347](https://github.com/paperless-ngx/paperless-ngx/pull/13347))
- Fix: dedupe permission-visible documents when combined with multi-tag filters [@stumpylog](https://github.com/stumpylog) ([#13345](https://github.com/paperless-ngx/paperless-ngx/pull/13345))
- Fixhancement: pass LLM output language to chat if specified [@shamoon](https://github.com/shamoon) ([#13340](https://github.com/paperless-ngx/paperless-ngx/pull/13340))
- Fix: ensure preview reload on live changes [@shamoon](https://github.com/shamoon) ([#13321](https://github.com/paperless-ngx/paperless-ngx/pull/13321))
- Fix: clamp all out out-of-range possible fields [@stumpylog](https://github.com/stumpylog) ([#13316](https://github.com/paperless-ngx/paperless-ngx/pull/13316))
- Fix: add docstring to DocumentClassifierSchema for cleaner LLM tool description [@stumpylog](https://github.com/stumpylog) ([#13315](https://github.com/paperless-ngx/paperless-ngx/pull/13315))
- Fix: guard build\_document\_node against stale FK on deleted correspondent/doc type [@stumpylog](https://github.com/stumpylog) ([#13318](https://github.com/paperless-ngx/paperless-ngx/pull/13318))
- Fix: prevent search filter loss when closing document with Escape key [@shamoon](https://github.com/shamoon) ([#13317](https://github.com/paperless-ngx/paperless-ngx/pull/13317))
- Fix: fix frontend permissions display for stats/system perms [@shamoon](https://github.com/shamoon) ([#13305](https://github.com/paperless-ngx/paperless-ngx/pull/13305))
### Documentation
- Documentation: fix PAPERLESS\_AI\_LLM\_OUTPUT\_LANGUAGE heading level [@SuperSandro2000](https://github.com/SuperSandro2000) ([#13341](https://github.com/paperless-ngx/paperless-ngx/pull/13341))
### Dependencies
- Chore: resolve npm provenance issues with chokidar and semver [@shamoon](https://github.com/shamoon) ([#13323](https://github.com/paperless-ngx/paperless-ngx/pull/13323))
### All App Changes
<details>
<summary>14 changes</summary>
- Fix: prevent pdfjs highlight scrolling from affecting the entire page [@shamoon](https://github.com/shamoon) ([#13355](https://github.com/paperless-ngx/paperless-ngx/pull/13355))
- Fix: don't skip OCR/archive for tagged PDFs with no actual text [@stumpylog](https://github.com/stumpylog) ([#13351](https://github.com/paperless-ngx/paperless-ngx/pull/13351))
- Performance: more efficient diacritic normalization in dropdown filtering [@shamoon](https://github.com/shamoon) ([#13347](https://github.com/paperless-ngx/paperless-ngx/pull/13347))
- Performance: prefetch notes and custom fields for LLM index text building [@stumpylog](https://github.com/stumpylog) ([#13350](https://github.com/paperless-ngx/paperless-ngx/pull/13350))
- Fix: dedupe permission-visible documents when combined with multi-tag filters [@stumpylog](https://github.com/stumpylog) ([#13345](https://github.com/paperless-ngx/paperless-ngx/pull/13345))
- Performance: Scope llm index updates to actually modified documents [@stumpylog](https://github.com/stumpylog) ([#13322](https://github.com/paperless-ngx/paperless-ngx/pull/13322))
- Fixhancement: pass LLM output language to chat if specified [@shamoon](https://github.com/shamoon) ([#13340](https://github.com/paperless-ngx/paperless-ngx/pull/13340))
- Chore: resolve npm provenance issues with chokidar and semver [@shamoon](https://github.com/shamoon) ([#13323](https://github.com/paperless-ngx/paperless-ngx/pull/13323))
- Fix: ensure preview reload on live changes [@shamoon](https://github.com/shamoon) ([#13321](https://github.com/paperless-ngx/paperless-ngx/pull/13321))
- Fix: clamp all out out-of-range possible fields [@stumpylog](https://github.com/stumpylog) ([#13316](https://github.com/paperless-ngx/paperless-ngx/pull/13316))
- Fix: add docstring to DocumentClassifierSchema for cleaner LLM tool description [@stumpylog](https://github.com/stumpylog) ([#13315](https://github.com/paperless-ngx/paperless-ngx/pull/13315))
- Fix: guard build\_document\_node against stale FK on deleted correspondent/doc type [@stumpylog](https://github.com/stumpylog) ([#13318](https://github.com/paperless-ngx/paperless-ngx/pull/13318))
- Fix: prevent search filter loss when closing document with Escape key [@shamoon](https://github.com/shamoon) ([#13317](https://github.com/paperless-ngx/paperless-ngx/pull/13317))
- Fix: fix frontend permissions display for stats/system perms [@shamoon](https://github.com/shamoon) ([#13305](https://github.com/paperless-ngx/paperless-ngx/pull/13305))
</details>
## paperless-ngx 3.0.3
### Bug Fixes
- Fixhancement: PAPERLESS\_ALLAUTH\_TRUSTED\_PROXY\_COUNT [@shamoon](https://github.com/shamoon) ([#13281](https://github.com/paperless-ngx/paperless-ngx/pull/13281))
- Fix: preserve document fields during Gotenberg conversion to PDF [@stumpylog](https://github.com/stumpylog) ([#13271](https://github.com/paperless-ngx/paperless-ngx/pull/13271))
- Fix: Makes the email date aware as soon as possible during parsing [@stumpylog](https://github.com/stumpylog) ([#13266](https://github.com/paperless-ngx/paperless-ngx/pull/13266))
- Fix: Handle a plain string as Celery sometimes provides for the traceback [@stumpylog](https://github.com/stumpylog) ([#13267](https://github.com/paperless-ngx/paperless-ngx/pull/13267))
- Fix: Emit the torch index into the requirements.txt for people still using it [@stumpylog](https://github.com/stumpylog) ([#13265](https://github.com/paperless-ngx/paperless-ngx/pull/13265))
- Fix: handle notes without a user when building the search index [@matthiasmast](https://github.com/matthiasmast) ([#13260](https://github.com/paperless-ngx/paperless-ngx/pull/13260))
### Documentation
- Docs: warn bare-metal users about stale files when upgrading [@stumpylog](https://github.com/stumpylog) ([#13296](https://github.com/paperless-ngx/paperless-ngx/pull/13296))
- Documentation: Add the NumPy CPU baseline increase to the migration guide [@stumpylog](https://github.com/stumpylog) ([#13269](https://github.com/paperless-ngx/paperless-ngx/pull/13269))
### Maintenance
- Fix: Emit the torch index into the requirements.txt for people still using it [@stumpylog](https://github.com/stumpylog) ([#13265](https://github.com/paperless-ngx/paperless-ngx/pull/13265))
### All App Changes
<details>
<summary>6 changes</summary>
- Fixhancement: PAPERLESS\_ALLAUTH\_TRUSTED\_PROXY\_COUNT [@shamoon](https://github.com/shamoon) ([#13281](https://github.com/paperless-ngx/paperless-ngx/pull/13281))
- Tweak: adjust top navbar wrapping when AI chat button visible [@shamoon](https://github.com/shamoon) ([#13280](https://github.com/paperless-ngx/paperless-ngx/pull/13280))
- Fix: preserve document fields during Gotenberg conversion to PDF [@stumpylog](https://github.com/stumpylog) ([#13271](https://github.com/paperless-ngx/paperless-ngx/pull/13271))
- Fix: Makes the email date aware as soon as possible during parsing [@stumpylog](https://github.com/stumpylog) ([#13266](https://github.com/paperless-ngx/paperless-ngx/pull/13266))
- Fix: Handle a plain string as Celery sometimes provides for the traceback [@stumpylog](https://github.com/stumpylog) ([#13267](https://github.com/paperless-ngx/paperless-ngx/pull/13267))
- Fix: handle notes without a user when building the search index [@matthiasmast](https://github.com/matthiasmast) ([#13260](https://github.com/paperless-ngx/paperless-ngx/pull/13260))
</details>
## paperless-ngx 3.0.2
### Bug Fixes
- Fix: fix broken migration in 3.0.1 [@shamoon](https://github.com/shamoon) ([#13242](https://github.com/paperless-ngx/paperless-ngx/pull/13242))
### All App Changes
- Fix: fix broken migration in 3.0.1 [@shamoon](https://github.com/shamoon) ([#13242](https://github.com/paperless-ngx/paperless-ngx/pull/13242))
## paperless-ngx 3.0.1
### Bug Fixes
- Fix: selection\_data re-derives the filtered document set 5 times over [@stumpylog](https://github.com/stumpylog) ([#13229](https://github.com/paperless-ngx/paperless-ngx/pull/13229))
- Fix: exclude source document from its own RAG similarity results [@stumpylog](https://github.com/stumpylog) ([#13233](https://github.com/paperless-ngx/paperless-ngx/pull/13233))
- Fix: split mailrule maximum\_age clamp into its own migration [@stumpylog](https://github.com/stumpylog) ([#13231](https://github.com/paperless-ngx/paperless-ngx/pull/13231))
- Chore: mark yes in confirm button for translation [@shamoon](https://github.com/shamoon) ([#13225](https://github.com/paperless-ngx/paperless-ngx/pull/13225))
- Fix: correct URL for W001 check [@shamoon](https://github.com/shamoon) ([#13220](https://github.com/paperless-ngx/paperless-ngx/pull/13220))
- Fix: ensure create dialog uses correct attribute type [@shamoon](https://github.com/shamoon) ([#13221](https://github.com/paperless-ngx/paperless-ngx/pull/13221))
- Fix: correct database engine from postgres to postgresql [@OberstVonGatow](https://github.com/OberstVonGatow) ([#13213](https://github.com/paperless-ngx/paperless-ngx/pull/13213))
- Fix: remove unnecessary .distinct() dominating document list queries [@stumpylog](https://github.com/stumpylog) ([#13205](https://github.com/paperless-ngx/paperless-ngx/pull/13205))
- Fix: also wrap non-breaking words in chat messages, handle whitespace [@shamoon](https://github.com/shamoon) ([#13211](https://github.com/paperless-ngx/paperless-ngx/pull/13211))
- Fix: tag/custom-field document\_count scales badly with tag count [@stumpylog](https://github.com/stumpylog) ([#13203](https://github.com/paperless-ngx/paperless-ngx/pull/13203))
- Fix: batch document user\_can\_change checks to avoid per-row N+1 [@stumpylog](https://github.com/stumpylog) ([#13204](https://github.com/paperless-ngx/paperless-ngx/pull/13204))
- Fix: fix app title restoration [@shamoon](https://github.com/shamoon) ([#13208](https://github.com/paperless-ngx/paperless-ngx/pull/13208))
- Fix: handle long wrapping titles in AI chat document list [@shamoon](https://github.com/shamoon) ([#13206](https://github.com/paperless-ngx/paperless-ngx/pull/13206))
### Dependencies
- Chore(deps-dev): Bump postcss from 8.5.6 to 8.5.22 in /src/paperless\_mail/templates in the npm\_and\_yarn group across 1 directory @[dependabot[bot]](https://github.com/apps/dependabot) ([#13236](https://github.com/paperless-ngx/paperless-ngx/pull/13236))
- Chore(deps): Bump pyasn1 from 0.6.3 to 0.6.4 in the uv group across 1 directory @[dependabot[bot]](https://github.com/apps/dependabot) ([#13237](https://github.com/paperless-ngx/paperless-ngx/pull/13237))
- Chore(deps): Bump the uv group across 1 directory with 8 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#13189](https://github.com/paperless-ngx/paperless-ngx/pull/13189))
### All App Changes
<details>
<summary>17 changes</summary>
- Chore(deps-dev): Bump postcss from 8.5.6 to 8.5.22 in /src/paperless\_mail/templates in the npm\_and\_yarn group across 1 directory @[dependabot[bot]](https://github.com/apps/dependabot) ([#13236](https://github.com/paperless-ngx/paperless-ngx/pull/13236))
- Fix: selection\_data re-derives the filtered document set 5 times over [@stumpylog](https://github.com/stumpylog) ([#13229](https://github.com/paperless-ngx/paperless-ngx/pull/13229))
- Chore: reword "Settings > Default permissions" description for clarity [@NotaInutilis](https://github.com/NotaInutilis) ([#13232](https://github.com/paperless-ngx/paperless-ngx/pull/13232))
- Chore(deps): Bump pyasn1 from 0.6.3 to 0.6.4 in the uv group across 1 directory @[dependabot[bot]](https://github.com/apps/dependabot) ([#13237](https://github.com/paperless-ngx/paperless-ngx/pull/13237))
- Chore(deps): Bump the uv group across 1 directory with 8 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#13189](https://github.com/paperless-ngx/paperless-ngx/pull/13189))
- Fix: exclude source document from its own RAG similarity results [@stumpylog](https://github.com/stumpylog) ([#13233](https://github.com/paperless-ngx/paperless-ngx/pull/13233))
- Fix: split mailrule maximum\_age clamp into its own migration [@stumpylog](https://github.com/stumpylog) ([#13231](https://github.com/paperless-ngx/paperless-ngx/pull/13231))
- Chore: mark yes in confirm button for translation [@shamoon](https://github.com/shamoon) ([#13225](https://github.com/paperless-ngx/paperless-ngx/pull/13225))
- Fix: correct URL for W001 check [@shamoon](https://github.com/shamoon) ([#13220](https://github.com/paperless-ngx/paperless-ngx/pull/13220))
- Fix: ensure create dialog uses correct attribute type [@shamoon](https://github.com/shamoon) ([#13221](https://github.com/paperless-ngx/paperless-ngx/pull/13221))
- Fix: correct database engine from postgres to postgresql [@OberstVonGatow](https://github.com/OberstVonGatow) ([#13213](https://github.com/paperless-ngx/paperless-ngx/pull/13213))
- Fix: remove unnecessary .distinct() dominating document list queries [@stumpylog](https://github.com/stumpylog) ([#13205](https://github.com/paperless-ngx/paperless-ngx/pull/13205))
- Fix: also wrap non-breaking words in chat messages, handle whitespace [@shamoon](https://github.com/shamoon) ([#13211](https://github.com/paperless-ngx/paperless-ngx/pull/13211))
- Fix: tag/custom-field document\_count scales badly with tag count [@stumpylog](https://github.com/stumpylog) ([#13203](https://github.com/paperless-ngx/paperless-ngx/pull/13203))
- Fix: batch document user\_can\_change checks to avoid per-row N+1 [@stumpylog](https://github.com/stumpylog) ([#13204](https://github.com/paperless-ngx/paperless-ngx/pull/13204))
- Fix: fix app title restoration [@shamoon](https://github.com/shamoon) ([#13208](https://github.com/paperless-ngx/paperless-ngx/pull/13208))
- Fix: handle long wrapping titles in AI chat document list [@shamoon](https://github.com/shamoon) ([#13206](https://github.com/paperless-ngx/paperless-ngx/pull/13206))
</details>
## paperless-ngx 3.0.0
### Breaking Changes
+33 -12
View File
@@ -522,22 +522,36 @@ do CORS calls. Set this to your public domain name.
fail2ban with log entries for failed authorization attempts. Value should be
IP address(es).
This setting also controls allauth's
[`ALLAUTH_TRUSTED_PROXY_COUNT`](https://docs.allauth.org/en/latest/account/configuration.html),
which is set to the number of proxies listed here. Without this,
allauth cannot determine the client IP address for rate limiting when
running behind a reverse proxy, resulting in a `403 Forbidden` on login.
By default, this setting also controls allauth's trusted proxy count,
which is set to the number of proxies listed here. Override that default
with [`PAPERLESS_ALLAUTH_TRUSTED_PROXY_COUNT`](#PAPERLESS_ALLAUTH_TRUSTED_PROXY_COUNT)
when the list length does not match the number of proxy hops.
Defaults to empty string.
#### [`PAPERLESS_ALLAUTH_TRUSTED_PROXY_COUNT=<integer>`](#PAPERLESS_ALLAUTH_TRUSTED_PROXY_COUNT) {#PAPERLESS_ALLAUTH_TRUSTED_PROXY_COUNT}
: Sets allauth's
[`ALLAUTH_TRUSTED_PROXY_COUNT`](https://docs.allauth.org/en/latest/common/rate_limits.html#configuration).
This is the number of trusted proxy **hops** represented in each
`X-Forwarded-For` header, not the number of IP addresses through which those
proxies may be reached. For example, a single dual-stack proxy is one hop even
when its IPv4 and IPv6 addresses are both listed in
[`PAPERLESS_TRUSTED_PROXIES`](#PAPERLESS_TRUSTED_PROXIES).
Only trust `X-Forwarded-For` when untrusted clients cannot connect directly
to Paperless-ngx.
Defaults to the number of entries in `PAPERLESS_TRUSTED_PROXIES`.
#### [`PAPERLESS_ALLAUTH_TRUSTED_CLIENT_IP_HEADER=<header-name>`](#PAPERLESS_ALLAUTH_TRUSTED_CLIENT_IP_HEADER) {#PAPERLESS_ALLAUTH_TRUSTED_CLIENT_IP_HEADER}
: Sets allauth's
[`ALLAUTH_TRUSTED_CLIENT_IP_HEADER`](https://docs.allauth.org/en/latest/account/configuration.html).
[`ALLAUTH_TRUSTED_CLIENT_IP_HEADER`](https://docs.allauth.org/en/latest/common/rate_limits.html#configuration).
Use this when your reverse proxy sets a dedicated header for the real
client IP instead of `X-Forwarded-For`, for example `X-Real-IP` (nginx)
or `CF-Connecting-IP` (Cloudflare). When set, this takes precedence over
[`PAPERLESS_TRUSTED_PROXIES`](#PAPERLESS_TRUSTED_PROXIES).
`PAPERLESS_ALLAUTH_TRUSTED_PROXY_COUNT`.
Defaults to none.
@@ -918,8 +932,8 @@ for display in the web interface.
| Document type | `never` | `auto` (default) | `always` |
| -------------------------- | ------- | -------------------------- | -------- |
| Scanned image (TIFF, JPEG) | No | **Yes** | Yes |
| Image-based PDF | No | **Yes** (short/no text, untagged) | Yes |
| Born-digital PDF | No | No (tagged or has embedded text) | Yes |
| Image-based PDF | No | **Yes** (no embedded text) | Yes |
| Born-digital PDF | No | No (has embedded text, optionally confirmed by tag) | Yes |
| Plain text, email, HTML | No | No | No |
| DOCX / ODT (via Tika) | Yes\* | Yes\* | Yes\* |
@@ -1346,13 +1360,16 @@ don't exist yet.
#### [`PAPERLESS_CONSUMER_IGNORE_PATTERNS=<json>`](#PAPERLESS_CONSUMER_IGNORE_PATTERNS) {#PAPERLESS_CONSUMER_IGNORE_PATTERNS}
: Additional regex patterns for files to ignore in the consumption directory. Patterns are matched against filenames only (not full paths)
using Python's `re.match()`, which anchors at the start of the filename.
using Python's `re.search()`. Use `^` to anchor a pattern to the start of the filename and `$` to anchor it to the end.
See the [watchfiles documentation](https://watchfiles.helpmanual.io/api/filters/#watchfiles.BaseFilter.ignore_entity_patterns)
This setting is for additional patterns beyond the built-in defaults. Common system files and directories are already ignored automatically.
The patterns will be compiled via Python's standard `re` module.
These are regular expressions, not glob patterns. For example, the glob pattern `._*` does not mean "starts with `._`" when used as a
regular expression; it matches nearly any non-empty filename. Use `^\._.*` for that behavior instead.
Example custom patterns:
```json
@@ -1367,7 +1384,11 @@ using Python's `re.match()`, which anchors at the start of the filename.
Defaults to `[]` (empty list, uses only built-in defaults).
The default ignores are `[.DS_Store, .DS_STORE, ._*, desktop.ini, Thumbs.db]` and cannot be overridden.
The built-in file patterns are equivalent to the following regular expressions and cannot be overridden:
```json
["^\\.DS_Store$", "^\\.DS_STORE$", "^\\._.*", "^desktop\\.ini$", "^Thumbs\\.db$"]
```
#### [`PAPERLESS_CONSUMER_IGNORE_DIRS=<json>`](#PAPERLESS_CONSUMER_IGNORE_DIRS) {#PAPERLESS_CONSUMER_IGNORE_DIRS}
@@ -2121,7 +2142,7 @@ used with the OpenAI-compatible backend to target a custom provider or local gat
Defaults to None.
### [`PAPERLESS_AI_LLM_OUTPUT_LANGUAGE=<str>`](#PAPERLESS_AI_LLM_OUTPUT_LANGUAGE) {#PAPERLESS_AI_LLM_OUTPUT_LANGUAGE}
#### [`PAPERLESS_AI_LLM_OUTPUT_LANGUAGE=<str>`](#PAPERLESS_AI_LLM_OUTPUT_LANGUAGE) {#PAPERLESS_AI_LLM_OUTPUT_LANGUAGE}
: The language to use for AI suggestions (results may vary by LLM model). If not supplied, defaults to the user's UI language setting or None.
+65 -4
View File
@@ -133,7 +133,7 @@ PAPERLESS_DB_OPTIONS="sslmode=require,sslrootcert=/certs/ca.pem,pool.max_size=10
## OCR and Archive File Generation Settings
The settings that control OCR behaviour and archive file generation have been redesigned. The old settings that coupled these two concerns together are **removed** old values are not silently honoured; a startup warning is logged if any removed variable is still set in your environment.
The settings that control OCR behaviour and archive file generation have been redesigned. The old settings that coupled these two concerns together are **removed** - old values are not silently honoured; a startup warning is logged if any removed variable is still set in your environment.
### Removed settings
@@ -166,7 +166,7 @@ Remove any `PAPERLESS_OCR_SKIP_ARCHIVE_FILE` variable from your environment. If
# v2: skip OCR when text present, always archive
PAPERLESS_OCR_MODE=skip
# v3: equivalent (auto is the new default)
# No change needed auto is the default
# No change needed - auto is the default
# v2: skip OCR when text present, skip archive too
PAPERLESS_OCR_MODE=skip_noarchive
@@ -189,7 +189,7 @@ PAPERLESS_ARCHIVE_FILE_GENERATION=auto
If you use the **remote OCR parser** (Azure AI), note that it always produces a
searchable PDF and stores it as the archive copy. `ARCHIVE_FILE_GENERATION=never`
has no effect for documents handled by the remote parser the archive is produced
has no effect for documents handled by the remote parser - the archive is produced
unconditionally by the remote engine.
## Search Index (Whoosh -> Tantivy)
@@ -324,8 +324,69 @@ option since v1.8.0.
Allauth changed how it determines the client IP address for login rate limiting. Users running
behind a reverse proxy may need to set
[`PAPERLESS_TRUSTED_PROXIES`](configuration.md#PAPERLESS_TRUSTED_PROXIES),
[`PAPERLESS_ALLAUTH_TRUSTED_PROXY_COUNT`](configuration.md#PAPERLESS_ALLAUTH_TRUSTED_PROXY_COUNT),
[`PAPERLESS_ALLAUTH_TRUSTED_CLIENT_IP_HEADER`](configuration.md#PAPERLESS_ALLAUTH_TRUSTED_CLIENT_IP_HEADER),
or both, to avoid `403 Forbidden` errors on login.
or a combination of these settings to avoid `403 Forbidden` errors on login.
The proxy count is the number of proxy hops in `X-Forwarded-For`, which may
differ from the number of configured proxy IP addresses.
## Minimum CPU Requirements (NumPy Baseline)
Starting with NumPy 2.4.0, official `manylinux` x86_64 wheels are compiled with a minimum
CPU baseline of `x86-64-v2`, which requires SSE3, SSSE3, SSE4.1, SSE4.2, POPCNT, and
CMPXCHG16B. This is a [deliberate upstream change](https://github.com/numpy/numpy/issues/27851)
citing that these instructions have been present in over 99.7% of CPUs since 2008
([Intel](<https://en.wikipedia.org/wiki/Nehalem_(microarchitecture)>)) or 2011
([AMD](<https://en.wikipedia.org/wiki/Bulldozer_(microarchitecture)>)).
NumPy is a dependency of the document classifier (via scikit-learn), so any CPU that
predates SSE4.2 support will crash with `SIGILL` (illegal instruction) when the classifier
is loaded or trained, regardless of whether AI features are enabled.
This differs from NumPy's optional SIMD dispatch (e.g. AVX2, AVX512), which is detected and
selected safely at runtime - the `x86-64-v2` requirement above is a hard floor baked into the
wheel, with no runtime fallback.
### Affected hardware
CPUs older than roughly 2008 (Intel) or 2011 (AMD) that lack SSE4.2 support. This is more
likely to affect low-power or embedded hardware - e.g. early Atom, Celeron, or pre-Bulldozer
AMD chips - than typical desktop or server hardware from the last decade.
Check for SSE4.2 support with:
```bash
grep -o -m1 sse4_2 /proc/cpuinfo
```
If this prints nothing, your CPU is affected.
### Symptoms
The Celery worker (and potentially the web server) repeatedly crashes and restarts with a
`SIGILL` error, typically visible in `dmesg`/`journalctl` as a `trap invalid opcode` inside
`_multiarray_umath...so`. Because the classifier is trained on a periodic schedule
(hourly, by default), affected instances see intermittent, hard-to-reproduce document
consumption failures whenever that scheduled task runs and takes down the worker process
mid-task.
### Action Required (for affected hardware only)
There is no way to make the classifier itself work on such CPUs - it requires an unofficial
NumPy build with `cpu-baseline=none`, which is not something we can ship. The practical
path forward is to stop the classifier from ever loading or training, which avoids
importing NumPy at all:
```bash
PAPERLESS_TRAIN_TASK_CRON=disable
```
This disables the periodic classifier training task (see
[`PAPERLESS_TRAIN_TASK_CRON`](configuration.md#PAPERLESS_TRAIN_TASK_CRON)). Automatic
matching based on the classifier (suggested correspondents, document types, tags, and
storage paths from trained rules) will no longer be available, but rule-based matching is
unaffected, and document consumption itself will no longer be at risk of crashing the
worker.
## Database Migrations
+8
View File
@@ -262,6 +262,14 @@ to a positive number to enable polling and disable native filesystem notificatio
and copy the contents to the home directory of the user you created
earlier (`/opt/paperless`).
!!! note
If you are updating an existing bare-metal installation rather than
installing for the first time, see the
[bare metal update instructions](administration.md#bare-metal-updating)
instead -- extracting a new release on top of an old one can leave
behind stale files from the previous version.
Optional: If you cloned the Git repository, you will need to
compile the frontend yourself. See [here](development.md#front-end-development)
and use the `build` step, not `serve`.
@@ -1,839 +0,0 @@
# Export Zip Compression Control Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** Add `--zip-compression {stored,deflated,bzip2,lzma,zstd}` and `--zip-compression-level N` flags to `document_exporter`, threaded into `ZipExportSink`, with import-side safety for codecs the running Python can't read.
**Architecture:** A new pure-data module `documents/export/compression.py` owns the method↔constant map, per-method level bounds, the runtime availability probe, and a compress-type readability check. `ZipExportSink` gains `compression`/`compresslevel` constructor params. The command validates flags up front (fail-fast `CommandError`) and constructs the sink; the importer pre-checks entry compress types before extracting.
**Tech Stack:** Python ≥3.11 (zstd only on 3.14+), `zipfile`, `compression.zstd` (PEP 784), pytest + pytest-mock + factory-boy. Backend tests run on the Linux VM (Python 3.11 — zstd positive tests are `skipif`-guarded); `ruff` runs locally.
**Spec:** `docs/superpowers/specs/2026-06-16-export-zip-compression-design.md`
**PREREQUISITE:** The base refactor `docs/superpowers/plans/2026-06-16-export-sink-architecture.md` MUST be merged first. This plan assumes `src/documents/export/sinks.py` exists with `ZipExportSink(target, zip_name, *, delete=False)` opening its `ZipFile` in `_open()`.
---
## Verified facts (CPython 3.14.3, via `uv run --python 3.14 --no-project`)
- Constants: `ZIP_STORED=0`, `ZIP_DEFLATED=8`, `ZIP_BZIP2=12`, `ZIP_LZMA=14`, `ZIP_ZSTANDARD=93` (zstd added 3.14; absent on < 3.14).
- `ZipFile(file, "w", compression=…, compresslevel=…)` applies both as the default for every `write`/`writestr` — no per-entry args needed (verified).
- Level bounds: `deflated` 09, `bzip2` 19, `lzma`/`stored` ignore level, `zstd` -131072…22 (`compression.zstd.CompressionParameter.compression_level.bounds() == (-131072, 22)`).
- An invalid level fails at the **first write** (`ValueError: Invalid initialization option` / `compresslevel must be between 1 and 9`), plus GC-time `AttributeError` noise on close — hence up-front validation.
- zstd is backed by `compression.zstd`; `zipfile` raises `RuntimeError` if it's unavailable.
## Conventions for every task
- **Run backend tests on the VM:** `bash /c/Users/tholmes/Documents/Coding/paperless/vmtest.sh "<targets>"` (never locally).
- **Lint locally:** `ruff check <paths> && ruff format <paths>` (global ruff, not `uv run`).
- **Tests are pytest-style:** classes, `@pytest.mark.django_db` on the class only where DB is needed (the `compression.py` and sink tests need no DB), factory-boy, `mocker`, `parametrize`, full type annotations.
- The VM runs Python 3.11, so **zstd positive tests must be `@pytest.mark.skipif(...)`-guarded**; they will simply not run there. zstd _rejection_ tests (the < 3.14 path) DO run on the VM.
## File structure
- **Create** `src/documents/export/compression.py` — method map, CLI choices, level bounds, `compression_available()`, `level_error()`, `compress_type_readable()`, `unreadable_method_names()`. Pure, no Django.
- **Create** `src/documents/tests/export/test_compression.py` — unit tests for the above.
- **Modify** `src/documents/export/sinks.py``ZipExportSink.__init__` gains `compression`/`compresslevel`; `_open()` passes them to `ZipFile`.
- **Modify** `src/documents/tests/export/test_sinks.py` — assert the chosen `compress_type` is applied.
- **Modify** `src/documents/management/commands/document_exporter.py` — add the two CLI flags, up-front validation, and pass resolved values to `ZipExportSink`.
- **Modify** `src/documents/tests/test_management_exporter.py` — flag validation + default-unchanged tests.
- **Modify** `src/documents/management/commands/document_importer.py` — pre-extract compress-type check.
- **Modify** `src/documents/tests/test_management_importer.py` — unsupported-codec → `CommandError`.
- **Modify** `docs/administration.md` — document both flags + zstd portability caveat.
---
## Task 1: `documents/export/compression.py` (pure compression policy)
**Files:**
- Create: `src/documents/export/compression.py`
- Test: `src/documents/tests/export/test_compression.py`
- [ ] **Step 1: Write the failing tests**
Create `src/documents/tests/export/test_compression.py`:
```python
import sys
import zipfile
import pytest
from documents.export import compression
class TestCompressionMethods:
def test_choices_always_include_zstd(self) -> None:
# zstd is offered regardless of runtime; availability is checked separately
assert compression.COMPRESSION_CHOICES == (
"stored",
"deflated",
"bzip2",
"lzma",
"zstd",
)
@pytest.mark.parametrize(
("name", "constant"),
[
("stored", zipfile.ZIP_STORED),
("deflated", zipfile.ZIP_DEFLATED),
("bzip2", zipfile.ZIP_BZIP2),
("lzma", zipfile.ZIP_LZMA),
],
)
def test_method_maps_to_zipfile_constant(self, name: str, constant: int) -> None:
assert compression.COMPRESSION_METHODS[name] == constant
def test_stored_and_deflated_always_available(self) -> None:
assert compression.compression_available("stored")
assert compression.compression_available("deflated")
def test_zstd_availability_tracks_runtime(self) -> None:
expected: bool = sys.version_info >= (3, 14)
assert compression.compression_available("zstd") == expected
class TestLevelError:
@pytest.mark.parametrize(
("method", "level"),
[
("deflated", 0),
("deflated", 9),
("bzip2", 1),
("bzip2", 9),
("deflated", None),
("stored", None),
],
)
def test_valid_levels_return_none(self, method: str, level: int | None) -> None:
assert compression.level_error(method, level) is None
@pytest.mark.parametrize(
("method", "level"),
[
("deflated", 10),
("deflated", -1),
("bzip2", 0),
("bzip2", 10),
],
)
def test_out_of_range_levels_return_message(
self,
method: str,
level: int,
) -> None:
msg: str | None = compression.level_error(method, level)
assert msg is not None
assert "between" in msg
@pytest.mark.parametrize("method", ["stored", "lzma"])
def test_level_on_levelless_method_is_rejected(self, method: str) -> None:
msg: str | None = compression.level_error(method, 5)
assert msg is not None
assert "no effect" in msg
class TestCompressTypeReadable:
@pytest.mark.parametrize("ct", [zipfile.ZIP_STORED, zipfile.ZIP_DEFLATED])
def test_stored_and_deflated_always_readable(self, ct: int) -> None:
assert compression.compress_type_readable(ct)
def test_zstd_compress_type_readability_tracks_runtime(self) -> None:
# 93 = ZIP_ZSTANDARD; 20 = legacy zstd method id (read-only)
expected: bool = sys.version_info >= (3, 14)
assert compression.compress_type_readable(93) == expected
assert compression.compress_type_readable(20) == expected
def test_unknown_compress_type_is_unreadable(self) -> None:
assert not compression.compress_type_readable(9999)
def test_unreadable_method_names_lists_methods(self) -> None:
# An unknown method id maps to no name and is reported generically.
names: set[str] = compression.unreadable_method_names({9999})
assert names == {"method 9999"}
```
- [ ] **Step 2: Run to verify it fails**
Run: `bash /c/Users/tholmes/Documents/Coding/paperless/vmtest.sh "src/documents/tests/export/test_compression.py -v"`
Expected: FAIL with `ModuleNotFoundError: No module named 'documents.export.compression'`.
- [ ] **Step 3: Implement `compression.py`**
Create `src/documents/export/compression.py`:
```python
from __future__ import annotations
import importlib
import zipfile
# ZIP_ZSTANDARD exists only on Python 3.14+ (PEP 784). None elsewhere.
ZSTD: int | None = getattr(zipfile, "ZIP_ZSTANDARD", None)
# CLI choices are fixed across runtimes so argparse never hides zstd; runtime
# availability is enforced separately in compression_available().
COMPRESSION_CHOICES: tuple[str, ...] = (
"stored",
"deflated",
"bzip2",
"lzma",
"zstd",
)
# Method name -> zipfile compression constant (zstd only when supported).
COMPRESSION_METHODS: dict[str, int] = {
"stored": zipfile.ZIP_STORED,
"deflated": zipfile.ZIP_DEFLATED,
"bzip2": zipfile.ZIP_BZIP2,
"lzma": zipfile.ZIP_LZMA,
}
if ZSTD is not None:
COMPRESSION_METHODS["zstd"] = ZSTD
# Inclusive (min, max) level bounds per method; None => level not applicable.
# Verified on CPython 3.14.3.
LEVEL_BOUNDS: dict[str, tuple[int, int] | None] = {
"stored": None,
"deflated": (0, 9),
"bzip2": (1, 9),
"lzma": None,
"zstd": (-131072, 22),
}
# zipfile compress_type id -> method name. 93 = current zstd id, 20 = legacy
# zstd id that zipfile can still read.
_COMPRESS_TYPE_TO_METHOD: dict[int, str] = {
zipfile.ZIP_STORED: "stored",
zipfile.ZIP_DEFLATED: "deflated",
zipfile.ZIP_BZIP2: "bzip2",
zipfile.ZIP_LZMA: "lzma",
93: "zstd",
20: "zstd",
}
def compression_available(method: str) -> bool:
"""Whether the running interpreter can actually use the given method."""
if method in ("stored", "deflated"):
# zlib is a hard CPython dependency; stored needs nothing.
return True
if method == "bzip2":
return _module_importable("bz2")
if method == "lzma":
return _module_importable("lzma")
if method == "zstd":
return ZSTD is not None and _module_importable("compression.zstd")
return False
def _module_importable(name: str) -> bool:
try:
importlib.import_module(name)
except ImportError:
return False
return True
def level_error(method: str, level: int | None) -> str | None:
"""Return a human message if (method, level) is invalid, else None."""
if level is None:
return None
bounds = LEVEL_BOUNDS[method]
if bounds is None:
return f"--zip-compression-level has no effect for '{method}'"
low, high = bounds
if not (low <= level <= high):
return (
f"--zip-compression-level for '{method}' must be between "
f"{low} and {high}"
)
return None
def compress_type_readable(compress_type: int) -> bool:
"""Whether this interpreter can decompress an entry of the given type."""
method = _COMPRESS_TYPE_TO_METHOD.get(compress_type)
if method is None:
return False
return compression_available(method)
def unreadable_method_names(compress_types: set[int]) -> set[str]:
"""Map a set of compress_type ids to human method names for error messages."""
names: set[str] = set()
for ct in compress_types:
names.add(_COMPRESS_TYPE_TO_METHOD.get(ct, f"method {ct}"))
return names
```
- [ ] **Step 4: Run to verify it passes**
Run: `bash /c/Users/tholmes/Documents/Coding/paperless/vmtest.sh "src/documents/tests/export/test_compression.py -v"`
Expected: PASS (on the 3.11 VM, `test_zstd_availability_tracks_runtime` and `test_zstd_compress_type_readability_tracks_runtime` assert `False`).
- [ ] **Step 5: Lint**
Run: `ruff check src/documents/export/compression.py src/documents/tests/export/test_compression.py && ruff format src/documents/export/compression.py src/documents/tests/export/test_compression.py`
Expected: no errors.
- [ ] **Step 6: Commit**
```bash
git add src/documents/export/compression.py src/documents/tests/export/test_compression.py
git commit -m "Feature: add export compression policy module"
```
---
## Task 2: `ZipExportSink` accepts compression method + level
**Files:**
- Modify: `src/documents/export/sinks.py`
- Test: `src/documents/tests/export/test_sinks.py`
- [ ] **Step 1: Write the failing test**
Append to `src/documents/tests/export/test_sinks.py` (the top-of-file block already imports `zipfile`, `Path`, `pytest`, `ZipExportSink`, `StreamingManifestWriter` from the base-refactor plan):
```python
class TestZipExportSinkCompression:
@pytest.fixture()
def source_file(self, tmp_path: Path) -> Path:
src: Path = tmp_path / "src" / "doc.pdf"
src.parent.mkdir(parents=True)
src.write_bytes(b"PDF-CONTENT" * 100)
return src
@pytest.mark.parametrize(
("method", "constant"),
[
("stored", zipfile.ZIP_STORED),
("deflated", zipfile.ZIP_DEFLATED),
("bzip2", zipfile.ZIP_BZIP2),
("lzma", zipfile.ZIP_LZMA),
],
)
def test_compression_method_is_applied_to_file_entries(
self,
tmp_path: Path,
source_file: Path,
method: str,
constant: int,
) -> None:
target: Path = tmp_path / "out"
target.mkdir()
with ZipExportSink(
target,
"export",
delete=False,
compression=constant,
) as sink:
sink.add_file(source_file, "doc.pdf")
with zipfile.ZipFile(target / "export.zip") as zf:
info = zf.getinfo("doc.pdf")
assert info.compress_type == constant
def test_compressing_method_beats_stored(
self,
tmp_path: Path,
source_file: Path,
) -> None:
# Robust size invariant: a compressing method must be <= stored on
# compressible content (avoids flaky level-9-vs-level-1 comparisons).
sizes: dict[str, int] = {}
for name, constant in (("stored", zipfile.ZIP_STORED), ("deflated", zipfile.ZIP_DEFLATED)):
target: Path = tmp_path / name
target.mkdir()
with ZipExportSink(target, "export", delete=False, compression=constant) as sink:
sink.add_file(source_file, "doc.pdf")
sizes[name] = (target / "export.zip").stat().st_size
assert sizes["deflated"] <= sizes["stored"]
```
- [ ] **Step 2: Run to verify it fails**
Run: `bash /c/Users/tholmes/Documents/Coding/paperless/vmtest.sh "src/documents/tests/export/test_sinks.py::TestZipExportSinkCompression -v"`
Expected: FAIL with `TypeError: __init__() got an unexpected keyword argument 'compression'`.
- [ ] **Step 3: Add the params to `ZipExportSink`**
In `src/documents/export/sinks.py`, change `ZipExportSink.__init__` to accept the new keyword-only params and store them, and pass them in `_open()`:
```python
def __init__(
self,
target: Path,
zip_name: str,
*,
delete: bool = False,
compression: int = zipfile.ZIP_DEFLATED,
compresslevel: int | None = None,
) -> None:
self._target = target.resolve()
self._zip_path = (self._target / zip_name).with_suffix(".zip")
self._tmp_path = self._zip_path.with_name(self._zip_path.name + ".tmp")
self._delete = delete
self._compression = compression
self._compresslevel = compresslevel
self._zip: zipfile.ZipFile | None = None
self._dirs: set[str] = set()
self._pending_manifest: tuple[Path, str] | None = None
self._stream_open = False
```
And in `_open()`:
```python
def _open(self) -> None:
settings.SCRATCH_DIR.mkdir(parents=True, exist_ok=True)
self._zip = zipfile.ZipFile(
self._tmp_path,
"w",
compression=self._compression,
compresslevel=self._compresslevel,
allowZip64=True,
)
```
- [ ] **Step 4: Run to verify it passes**
Run: `bash /c/Users/tholmes/Documents/Coding/paperless/vmtest.sh "src/documents/tests/export/test_sinks.py -v"`
Expected: PASS (all sink tests, including the four method params and the size invariant). `bzip2`/`lzma` are present on the VM's CPython, so those params pass.
- [ ] **Step 5: Lint**
Run: `ruff check src/documents/export/sinks.py && ruff format src/documents/export/sinks.py`
Expected: no errors.
- [ ] **Step 6: Commit**
```bash
git add src/documents/export/sinks.py src/documents/tests/export/test_sinks.py
git commit -m "Feature: ZipExportSink accepts compression method and level"
```
---
## Task 3: Wire CLI flags + validation into `document_exporter`
**Files:**
- Modify: `src/documents/management/commands/document_exporter.py`
- Test: `src/documents/tests/test_management_exporter.py`
- [ ] **Step 1: Add the argparse flags**
In `document_exporter.py`, add the import near the other `documents.export` import:
```python
from documents.export.compression import COMPRESSION_CHOICES
from documents.export.compression import COMPRESSION_METHODS
from documents.export.compression import compression_available
from documents.export.compression import level_error
from documents.export.compression import ZSTD
```
In `add_arguments`, after the `--zip-name` argument, add:
```python
parser.add_argument(
"--zip-compression",
choices=COMPRESSION_CHOICES,
default=None,
help=(
"Compression method for the export zip (requires --zip). "
"Default: deflated. 'zstd' requires Python 3.14+ on both the "
"exporting and importing machine."
),
)
parser.add_argument(
"--zip-compression-level",
type=int,
default=None,
help=(
"Compression level for the export zip (requires --zip). "
"deflated: 0-9, bzip2: 1-9, zstd: -131072..22; ignored for "
"stored/lzma."
),
)
```
- [ ] **Step 2: Read + validate the flags in `handle()`**
In `handle()`, after the existing `--compare-*` + `--zip` guard, add the compression flag handling. Insert before the sink construction:
```python
zip_compression: str | None = options["zip_compression"]
zip_compression_level: int | None = options["zip_compression_level"]
if not self.zip_export and (
zip_compression is not None or zip_compression_level is not None
):
raise CommandError(
"--zip-compression and --zip-compression-level require --zip",
)
compression_method = zip_compression or "deflated"
if self.zip_export:
if not compression_available(compression_method):
if compression_method == "zstd" and ZSTD is None:
raise CommandError(
"zstd compression requires Python 3.14 or newer",
)
raise CommandError(
f"Compression method '{compression_method}' is not "
f"available on this Python runtime",
)
level_msg = level_error(compression_method, zip_compression_level)
if level_msg is not None:
raise CommandError(level_msg)
```
- [ ] **Step 3: Pass the resolved values into `ZipExportSink`**
Change the `ZipExportSink(...)` construction in `handle()` to:
```python
if self.zip_export:
sink = ZipExportSink(
self.target,
options["zip_name"],
delete=self.delete,
compression=COMPRESSION_METHODS[compression_method],
compresslevel=zip_compression_level,
)
else:
sink = DirectoryExportSink(
self.target,
compare_checksums=self.compare_checksums,
compare_json=self.compare_json,
delete=self.delete,
)
```
- [ ] **Step 4: Write the command-level tests**
Add to the `TestExportImport` class in `src/documents/tests/test_management_exporter.py` (imports `call_command`, `CommandError`, `ZipFile`, `timezone` already present):
```python
def test_compression_flags_require_zip(self) -> None:
for args in (
["--zip-compression", "lzma"],
["--zip-compression-level", "5"],
):
with self.assertRaises(CommandError):
call_command(
"document_exporter",
self.target,
*args,
skip_checks=True,
)
def test_zip_compression_level_out_of_range_raises(self) -> None:
with self.assertRaises(CommandError):
call_command(
"document_exporter",
self.target,
"--zip",
"--zip-compression",
"deflated",
"--zip-compression-level",
"99",
skip_checks=True,
)
def test_zip_compression_level_rejected_for_stored(self) -> None:
with self.assertRaises(CommandError):
call_command(
"document_exporter",
self.target,
"--zip",
"--zip-compression",
"stored",
"--zip-compression-level",
"5",
skip_checks=True,
)
def test_zip_lzma_compression_round_trips(self) -> None:
call_command(
"document_exporter",
self.target,
"--zip",
"--zip-compression",
"lzma",
skip_checks=True,
)
expected = str(
self.target / f"export-{timezone.localdate().isoformat()}.zip",
)
self.assertIsFile(expected)
with ZipFile(expected) as zip_file:
info = zip_file.getinfo("manifest.json")
# manifest.json carries the chosen method; deflated is the default
self.assertEqual(info.compress_type, 14) # ZIP_LZMA
def test_default_zip_uses_deflate(self) -> None:
call_command(
"document_exporter",
self.target,
"--zip",
skip_checks=True,
)
expected = str(
self.target / f"export-{timezone.localdate().isoformat()}.zip",
)
with ZipFile(expected) as zip_file:
info = zip_file.getinfo("manifest.json")
self.assertEqual(info.compress_type, 8) # ZIP_DEFLATED
```
- [ ] **Step 5: Run the tests**
Run: `bash /c/Users/tholmes/Documents/Coding/paperless/vmtest.sh "src/documents/tests/test_management_exporter.py -v"`
Expected: PASS — the new tests plus all existing exporter tests stay green.
- [ ] **Step 6: Lint**
Run: `ruff check src/documents/management/commands/document_exporter.py src/documents/tests/test_management_exporter.py && ruff format src/documents/management/commands/document_exporter.py src/documents/tests/test_management_exporter.py`
Expected: no errors.
- [ ] **Step 7: Commit**
```bash
git add src/documents/management/commands/document_exporter.py src/documents/tests/test_management_exporter.py
git commit -m "Feature: add --zip-compression and --zip-compression-level flags"
```
---
## Task 4: Importer pre-check for unreadable codecs
**Files:**
- Modify: `src/documents/management/commands/document_importer.py`
- Test: `src/documents/tests/test_management_importer.py`
- [ ] **Step 1: Write the failing test**
The importer test file `src/documents/tests/test_management_importer.py` is
`TestCase`-style (`class TestCommandImport(... TestCase)`, `self.assertRaises`,
`DirectoriesMixin` gives `self.dirs.scratch_dir`). Match that style. Add this
method to `TestCommandImport`. It builds a valid zip and patches the readability
probe so the check fires deterministically on any runtime:
```python
def test_import_rejects_unreadable_compression(self) -> None:
"""
GIVEN:
- A zip archive with an entry whose compression this Python can't read
WHEN:
- Import is attempted
THEN:
- A CommandError naming the issue is raised, before extraction
"""
import zipfile
from unittest import mock
archive = Path(self.dirs.scratch_dir) / "export.zip"
with zipfile.ZipFile(archive, "w") as zf:
zf.writestr("manifest.json", "[]")
with mock.patch(
"documents.management.commands.document_importer.compress_type_readable",
return_value=False,
):
with self.assertRaises(CommandError) as e:
call_command(
"document_importer",
str(archive),
"--no-progress-bar",
skip_checks=True,
)
self.assertIn("compression", str(e.exception))
```
- [ ] **Step 2: Run to verify it fails**
Run: `bash /c/Users/tholmes/Documents/Coding/paperless/vmtest.sh "src/documents/tests/test_management_importer.py -k unreadable_compression -v"`
Expected: FAIL — no pre-check exists yet, so the import proceeds (or fails with a different error).
- [ ] **Step 3: Implement the pre-check**
In `document_importer.py`, add the import:
```python
from documents.export.compression import compress_type_readable
from documents.export.compression import unreadable_method_names
```
Find the zip-handling block (around `document_importer.py:453`):
```python
with ZipFile(self.source) as zf:
zf.extractall(tmp_dir)
```
Replace it with a pre-check before extraction:
```python
with ZipFile(self.source) as zf:
unsupported = {
info.compress_type
for info in zf.infolist()
if not compress_type_readable(info.compress_type)
}
if unsupported:
names = ", ".join(sorted(unreadable_method_names(unsupported)))
raise CommandError(
f"This archive uses compression this Python cannot "
f"read ({names}). zstd archives require Python 3.14+.",
)
zf.extractall(tmp_dir)
```
Confirm `CommandError` is imported in `document_importer.py` (it is used elsewhere; if not, add `from django.core.management.base import CommandError`).
- [ ] **Step 4: Run to verify it passes**
Run: `bash /c/Users/tholmes/Documents/Coding/paperless/vmtest.sh "src/documents/tests/test_management_importer.py -v"`
Expected: PASS — the new test plus all existing importer tests (normal deflated/stored archives still import).
- [ ] **Step 5: Lint**
Run: `ruff check src/documents/management/commands/document_importer.py src/documents/tests/test_management_importer.py && ruff format src/documents/management/commands/document_importer.py src/documents/tests/test_management_importer.py`
Expected: no errors.
- [ ] **Step 6: Commit**
```bash
git add src/documents/management/commands/document_importer.py src/documents/tests/test_management_importer.py
git commit -m "Feature: importer rejects archives with unreadable compression"
```
---
## Task 5: Document the flags
**Files:**
- Modify: `docs/administration.md`
- [ ] **Step 1: Add the flags to the option list**
In `docs/administration.md`, update the usage block (around line 257) to include the new flags:
```
document_exporter target [-c] [-d] [-f] [-na] [-nt] [-p] [-sm] [-z]
optional arguments:
-c, --compare-checksums
-cj, --compare-json
-d, --delete
-f, --use-filename-format
-na, --no-archive
-nt, --no-thumbnail
-p, --use-folder-prefix
-sm, --split-manifest
-z, --zip
-zn, --zip-name
--zip-compression
--zip-compression-level
--data-only
--no-progress-bar
--passphrase
```
- [ ] **Step 2: Add the prose**
After the `-z`/`--zip` paragraph (around line 330), add:
```markdown
The compression method for the zip can be set with `--zip-compression`
(`stored`, `deflated` (default), `bzip2`, `lzma`, or `zstd`) and tuned with
`--zip-compression-level` (deflated: 09, bzip2: 19, zstd: -13107222; ignored
for `stored` and `lzma`). Both options require `--zip`.
!!! warning
`zstd` compression requires Python 3.14 or newer on **both** the machine
creating the export and any machine importing it. An archive compressed with
`zstd` (or `lzma`/`bzip2` where those modules are unavailable) cannot be
imported on a runtime that lacks the codec; the importer will refuse it with
a clear error. The default `deflated` is universally readable.
```
- [ ] **Step 3: Verify the docs build is not broken (lint markdown)**
Run: `ruff check docs/ 2>/dev/null; echo "docs are markdown; rely on prettier pre-commit"`
(No code to test. The prettier pre-commit hook will reformat on commit.)
- [ ] **Step 4: Commit**
```bash
git add docs/administration.md
git commit -m "Docs: document --zip-compression and --zip-compression-level"
```
---
## Task 6: Final verification
**Files:** none (verification only).
- [ ] **Step 1: Full backend suites on the VM**
Run: `bash /c/Users/tholmes/Documents/Coding/paperless/vmtest.sh "src/documents/tests/export/ src/documents/tests/test_management_exporter.py src/documents/tests/test_management_importer.py -v"`
Expected: PASS, no failures.
- [ ] **Step 2: Spot-check the zstd happy path on Python 3.14 (cannot run under Django on the 3.11 VM)**
The zstd positive round-trip can't run in the 3.11 test env. Confirm the policy module behaves on a real 3.14 interpreter with a standalone check (no Django needed):
Run:
```bash
uv run --python 3.14 --no-project python -c "import sys; sys.path.insert(0,'src'); import django; print('skip')" 2>/dev/null || \
uv run --python 3.14 --no-project python -c "
import zipfile, io
from compression.zstd import CompressionParameter as CP
print('zstd const', zipfile.ZIP_ZSTANDARD, 'bounds', CP.compression_level.bounds())
buf = io.BytesIO()
with zipfile.ZipFile(buf,'w',compression=zipfile.ZIP_ZSTANDARD,compresslevel=19) as zf:
zf.writestr('a.txt','x'*1000)
with zipfile.ZipFile(buf) as zf:
assert zf.getinfo('a.txt').compress_type == zipfile.ZIP_ZSTANDARD
assert zf.read('a.txt') == b'x'*1000
print('zstd round-trip OK')
"
```
Expected: prints `zstd const 93 bounds (-131072, 22)` and `zstd round-trip OK`. This validates the constant, bounds, and that a zstd archive round-trips — the parts the 3.11 CI cannot exercise.
- [ ] **Step 3: Type-check on the VM (pyrefly)**
```bash
tar czf - src pyproject.toml uv.lock .pyrefly-baseline.json | ssh -o BatchMode=yes -p 2244 trenton@localhost 'tar xzf - -C ~/projects/paperless-ngx'
ssh -o BatchMode=yes -p 2244 trenton@localhost 'bash -lc "cd ~/projects/paperless-ngx && uv run pyrefly check"'
```
Expected: no new type errors beyond the baseline. (Note: `import compression.zstd` is guarded behind `importlib.import_module`, so it is never statically resolved on the 3.11 baseline.)
- [ ] **Step 4: Final lint**
Run: `ruff check src/documents/export/ src/documents/management/commands/document_exporter.py src/documents/management/commands/document_importer.py && ruff format --check src/documents/export/ src/documents/management/commands/document_exporter.py src/documents/management/commands/document_importer.py`
Expected: clean.
---
## Notes for the implementer
- **Default behavior is unchanged:** with no flags, the sink is constructed with `compression=ZIP_DEFLATED, compresslevel=None` — byte-method-identical to today (`shutil.make_archive` used `ZIP_DEFLATED` with no level). `test_default_zip_uses_deflate` pins this.
- **zstd availability is gated three ways and never imported statically:** the constant via `getattr`, the codec via `importlib.import_module("compression.zstd")`, and the CLI value rejected with a friendly message on < 3.14. The choices list always contains `zstd` so argparse doesn't hide it.
- **The importer pre-check is the safety net** for portability foot-guns — without it an unreadable entry raises a bare `NotImplementedError` mid-`extractall`. The check runs on `infolist()` (metadata only) before any extraction.
- **Why `--zip-compression` defaults to `None`, not `"deflated"`:** so `handle()` can detect "user passed it without `--zip`" and fail fast. The effective default is resolved as `zip_compression or "deflated"`.
@@ -1,236 +0,0 @@
# Export Zip Compression Control — Design
**Date:** 2026-06-16
**Branch base:** `dev`
**Status:** Design complete (zstd facts verified on CPython 3.14.3) — **depends on**
`2026-06-16-export-sink-architecture-design.md` being implemented first.
## Prerequisite
This builds directly on the export sink refactor. It assumes `ZipExportSink`
already exists and is the single place that owns `zipfile.ZipFile` creation and
entry writes. Do not start this until that refactor has landed; without it, the
change would have to touch the command's zip branches again.
## Problem
Zip export is hardwired to `ZIP_DEFLATED` at the library default level. Users
have no way to trade speed against archive size — a fast `ZIP_STORED` pass for a
quick local copy, or a maximal `ZIP_LZMA` pass for the smallest off-site backup.
The sink refactor turns "which compression" into a single constructor argument,
so exposing it is now a small, isolated change.
## Goal
Let the operator choose the zip compression method and level from the CLI, with
behavior identical to today when the flags are omitted. All knowledge of
compression stays inside `ZipExportSink`; the command only parses flags and maps
them to sink arguments.
## Scope
In scope:
- `ZipExportSink` gains `compression: int` and `compresslevel: int | None`
constructor parameters (default `ZIP_DEFLATED`, `None` → library default),
passed straight to `zipfile.ZipFile(...)`.
- New `document_exporter` flags: `--zip-compression` and
`--zip-compression-level`, valid only with `--zip`.
- Validation: method availability, level range per method, and the
requires-`--zip` guard.
- Import-side: a pre-extract support check in `document_importer` that turns an
unsupported codec into a clear `CommandError` (the importer otherwise decompresses
transparently via `ZipFile.extractall`).
- Docs: add both flags and the zstd-portability caveat to `docs/administration.md`
(the `document_exporter` option list, lines ~257-270 and the `-z`/`-zn` section,
lines ~328-330). New flags are long-form only (`--zip-compression`,
`--zip-compression-level`) — no short aliases, to avoid `-zc`/`-zl` collisions
with the existing `-z`/`-zn`.
Out of scope:
- Compression for any non-zip sink (folder has none; a future S3 sink would
handle its own object storage compression separately).
- Changing the default. Omitting the flags must produce a byte-compatible-method
archive to today's (`ZIP_DEFLATED`, default level).
## Design
### `ZipExportSink` changes
The base sink's signature is `ZipExportSink(target, zip_name, *, delete)`; this
adds two keyword-only params after `delete`:
```python
def __init__(
self,
target: Path,
zip_name: str,
*,
delete: bool = False,
compression: int = zipfile.ZIP_DEFLATED,
compresslevel: int | None = None,
) -> None:
...
# opened in __enter__:
self._zip = zipfile.ZipFile(
self._tmp_path,
"w",
compression=compression,
compresslevel=compresslevel,
allowZip64=True,
)
```
`ZipFile` applies `compression`/`compresslevel` as the default for every
`write`/`writestr` (verified: a `ZipFile(..., compression=ZIP_BZIP2)` yields
entries with `compress_type == ZIP_BZIP2` without per-call args), so `add_file` /
`add_json` / the manifest entry need no changes. Directory marker entries are
empty so their compressed payload is zero, but they are still _tagged_ with the
chosen `compress_type` — harmless, but tests that read `infolist()` should filter
or account for marker entries (see Testing).
### CLI flags (`document_exporter`)
- `--zip-compression {stored,deflated,bzip2,lzma}` — and `zstd` **when the
runtime supports it** (see below). Maps to the matching `zipfile.ZIP_*`
constant. Default `deflated`.
- `--zip-compression-level N` — integer. Per-method accepted ranges (verified
against the [3.14 `zipfile` docs](https://docs.python.org/3.14/library/zipfile.html#zipfile.ZipFile)):
- `deflated`: **09** (`zlib` also accepts `-1` = "default", identical to
omitting the flag / `compresslevel=None`).
- `bzip2`: **19** (`0` is invalid for bzip2).
- `lzma`, `stored`: level has **no effect** — passing `--zip-compression-level`
with either is a `CommandError`, not a silent accept (consistent with the
base refactor's fail-fast posture).
- `zstd`: **-131072 … 22** (the documented commonly-accepted range; the
authoritative bounds are
`compression.zstd.CompressionParameter.compression_level.bounds()`).
Default: unset → library default (`compresslevel=None`).
Both flags require `--zip`; passing either without `--zip` raises a
`CommandError`, matching the incremental-flag rule from the base refactor.
**Why validate up front (not let `zipfile` raise) — verified on 3.14.3:** an
invalid level does _not_ fail at `ZipFile(...)` construction — it fails at the
**first `write`/`writestr` call**, with an opaque message
(`ValueError: Invalid initialization option` for deflated > 9, or
`ValueError: compresslevel must be between 1 and 9` for bzip2). Worse, on context
exit the half-initialized write handle emits a secondary
`AttributeError: '_ZipWriteFile' object has no attribute '_compressor'` during GC
finalization, so the user sees stack-trace noise unrelated to the real cause.
Up-front validation turns all of that into a single clean `CommandError`.
### Validation (in `handle()`, before constructing the sink)
1. **Requires `--zip`.** Either flag without `--zip``CommandError`.
2. **Method availability — via a named, patchable seam.** Expose a module-level
helper `compression_available(method: str) -> bool` that does
`try: import bz2 / import lzma / from compression import zstd except ImportError:
return False`**not** `importlib.util.find_spec`, which can report a stdlib
C-extension as present when importing it actually fails. `stored`/`deflated`
are always available (`zlib` is a hard CPython dependency). For `zstd` the probe
must import `compression.zstd` (3.14+), not merely check that
`zipfile.ZIP_ZSTANDARD` exists. Making this a named function is also what lets
the test patch "method unavailable" with `mocker`. If the chosen method is
unavailable, raise a `CommandError` naming the missing capability — `zipfile`
itself would otherwise raise a bare `RuntimeError`
("Compression requires the (missing) … module").
3. **Level range.** Reject an out-of-range `--zip-compression-level` for the
chosen method with a clear `CommandError`; reject the flag entirely for
`stored`/`lzma` (see above).
### zstd (Python 3.14+)
**Verified empirically on CPython 3.14.3** (via `uv run --python 3.14 --no-project`)
and against [PEP 784](https://peps.python.org/pep-0784/) +
[the 3.14 `zipfile` docs](https://docs.python.org/3.14/library/zipfile.html):
- The compression-method constant is **`zipfile.ZIP_ZSTANDARD`** (added 3.14; its
numeric value is `93`). It does **not** exist on < 3.14.
- It is backed by the new **`compression.zstd`** stdlib module (PEP 784 added a
`compression` namespace package; legacy `bz2`/`lzma`/`zlib` imports are
unchanged). `zipfile` raises `RuntimeError` if `compression.zstd` is
unavailable when zstd is requested.
- Accepted `compresslevel` is **`-131072 … 22`**, confirmed at runtime via
`compression.zstd.CompressionParameter.compression_level.bounds() == (-131072, 22)`.
Gate everything zstd-related at runtime so nothing is imported or referenced on
< 3.14 (the project targets Python ≥ 3.11):
```python
_ZSTD: int | None = getattr(zipfile, "ZIP_ZSTANDARD", None) # None before 3.14
```
Presence of the _constant_ does not guarantee the _codec_ is usable, so the
availability probe (validation step 2) imports `compression.zstd`, not merely
checks the constant.
Keep `zstd` in the `--zip-compression` `choices` **always** (even on < 3.14), and
reject it in validation with a friendly "zstd requires Python 3.14+" message. If
it were dropped from `choices` on older runtimes, argparse would emit a generic
"invalid choice" that reads as though the option never existed — worse UX.
### Import-side compatibility
`document_importer` reads zips with `ZipFile(self.source).extractall(...)`
(`document_importer.py:453`), which decompresses each entry transparently using
whatever method it was stored with — **provided the matching module exists on the
importing machine.**
The failure mode when it doesn't is unfriendly and must be handled: a zstd (or
otherwise unsupported) entry raises a bare `NotImplementedError` **per-entry,
during `extractall`** — _not_ at `ZipFile(self.source)` open, and `is_zipfile()`
still returns true (a zstd archive is a valid zip container). So the importer
enters the zip branch, creates its temp dir, may partially extract other entries,
then blows up mid-extract with no context. **Mitigation (in scope here):** before
extracting, inspect `ZipFile(self.source).infolist()` compress types and, if any
is unsupported on this runtime, raise a `CommandError` naming the method and the
requirement (e.g. "this archive uses zstd, which needs Python 3.14+") instead of
letting `NotImplementedError` escape.
Per-method summary (document in help text + `administration.md`):
- `deflated`/`stored`: universally importable.
- `bzip2`/`lzma`: importable wherever the `bz2`/`lzma` modules are present
(essentially always).
- `zstd`: importable only on Python 3.14+. An archive compressed with `zstd` is
**not** importable on older runtimes.
## Testing
New cases in the sink tests and an export→import round-trip
(pytest classes, factory-boy, `mocker`, `parametrize`, typed; run on the VM):
- **Round-trip per method.** Parametrize over the available methods (skip `zstd`
below 3.14, skip `bzip2`/`lzma` if the module is somehow absent): export a
small library, import it back, assert documents/manifest match.
- **Method is applied.** Assert each written _file_ entry's `compress_type`
equals the requested method (read back via `ZipFile.infolist()`), filtering out
directory marker entries (which are tagged but empty).
- **Level affects size — robustly.** Do **not** compare deflate level 9 vs 1
(on small or incompressible fixtures level 9 can equal or slightly exceed level
1, causing flaky CI). Instead assert that a compressing method on a
moderately-compressible fixture yields a total smaller than `stored`
(`ZIP_STORED`), which is a stable invariant.
- **Validation.** Each flag without `--zip``CommandError`; out-of-range level
(`--zip-compression-level 99`) → a clean `CommandError` from validation
(asserting we never reach the `writestr` that would raise the masked
`ValueError`); `--zip-compression-level` with `stored`/`lzma``CommandError`;
unavailable method (patch the named availability seam with `mocker`) →
`CommandError`; on < 3.14, `--zip-compression zstd` → the friendly
"requires 3.14+" `CommandError`.
- **Import pre-check.** An archive containing an unsupported compress type
produces a `CommandError` from the importer naming the method, not a raw
`NotImplementedError` (simulate by patching the importer's support probe).
- **Default unchanged.** Omitting both flags yields file entries with
`compress_type == ZIP_DEFLATED`, identical to pre-feature behavior.
## Risks
- **Foot-gun archives.** A user could produce a `zstd`/`lzma` archive their
import target can't read. Mitigation: explicit help text and the import-side
notes above; the default stays the universally-readable `deflated`.
- **Optional-module assumptions.** Don't assume `bz2`/`lzma` are always compiled
in; probe and error clearly. Mitigation: the availability validation step.
+38 -9
View File
@@ -620,6 +620,34 @@ no other workflow will be executed on the document.
If a "Move to Trash" action is executed in a consume pipeline, the consumption
will be aborted and the file will be deleted.
##### Password Removal {#workflow-action-password-removal}
"Password Removal" actions attempt to remove password protection from encrypted PDF documents. You can specify:
- One or more passwords to try, separated by commas or new lines
- Each password is tried in order until one successfully unlocks the document
Password removal never modifies a file in place. Instead, once a working password is found, the
decrypted content is consumed as a new [document version](#document-file-versions), leaving the
original (still encrypted) version in the document's version history.
**Consumption Started**: because this trigger fires before the document exists yet, the password
removal itself is deferred until after the initial consumption of the encrypted file has completed.
OCR engines cannot process an encrypted PDF, so this first version is typically stored with no
extracted text (unless the file already contained extractable text outside of OCR). Immediately
afterwards, the password is removed and the decrypted file is automatically re-consumed as a second,
new version of the same document, this time with normal OCR/text extraction applied. In other words,
a password-protected file added with this trigger will briefly exist as an un-OCR'd version before
the properly processed version is created.
**Document Added**, **Document Updated**, **Scheduled**: these triggers run against a document that
already exists, so password removal happens immediately: the decrypted content is queued for
consumption as a new version right away. Note that if the document's initial consumption also
happened while it was still encrypted, that original version will likewise be missing OCR text.
**Current limitation**: Passwords are stored as a simple list without descriptions. To handle
multiple PDF types with different passwords, create separate workflows for each use case.
#### Workflow placeholders
Titles and webhook payloads can be generated by workflows using [Jinja templates](https://jinja.palletsprojects.com/en/3.1.x/templates/).
@@ -833,10 +861,10 @@ contract you signed 8 years ago).
When you search paperless for a document, it tries to match this query
against your documents. Paperless will look for matching documents by
inspecting their content, title, correspondent, type, tags, notes, and
custom field values. Paperless returns a scored list of results, so that
documents matching your query better will appear further up in the search
results.
inspecting their content, title, correspondent, type, and tags. Paperless
returns a scored list of results, so that documents matching your query
better will appear further up in the search results. Notes and custom field
values can be searched using the advanced search syntax described below.
By default, paperless returns only documents which contain all words
typed in the search bar. A few things to know about how matching works:
@@ -891,9 +919,9 @@ Supported date keywords: `today`, `yesterday`, `previous week`,
#### Searching custom fields
Custom field values are included in the full-text index, so a plain search
already matches documents whose custom field values contain your search terms.
To narrow by field name or value specifically:
Custom field names and values are included in the full-text index, but they
are not searched by a plain, unqualified query. Use the advanced search syntax
to search by field name or value:
```
custom_fields.value:policy
@@ -921,8 +949,9 @@ custom_fields.name:"Contract Number" custom_fields.value:1312
#### Searching notes
Notes content is included in full-text search automatically. To search
by note author or content specifically:
Notes are included in the full-text index, but they are not searched by a
plain, unqualified query. Use the advanced search syntax to search by note
author or content:
```
notes.user:alice
+2 -2
View File
@@ -1,6 +1,6 @@
[project]
name = "paperless-ngx"
version = "3.0.0"
version = "3.0.5"
description = "A community-supported supercharged document management system: scan, index and archive all your physical documents"
readme = "README.md"
requires-python = ">=3.11"
@@ -75,7 +75,7 @@ dependencies = [
"sqlite-vec==0.1.9",
"tantivy~=0.26.0",
"tika-client~=0.11.0",
"torch~=2.12.0",
"torch~=2.13.0",
"watchfiles>=1.1.1",
"whitenoise~=6.11",
"zxing-cpp~=3.0.0",
@@ -1,4 +1,4 @@
import { expect, test } from '@playwright/test'
import { expect, test, type WebSocketRoute } from '@playwright/test'
import path from 'node:path'
const REQUESTS_HAR = path.join(__dirname, 'requests/api-document-detail.har')
@@ -95,3 +95,60 @@ test('should support quick filters', async ({ page }) => {
.click()
await expect(page).toHaveURL(/tags__id__all=4&sort=created&reverse=1&page=1/)
})
test('should finish reloading the preview after a remote document update', async ({
page,
}) => {
let resolveStatusSocket: (socket: WebSocketRoute) => void
const statusSocketReady = new Promise<WebSocketRoute>((resolve) => {
resolveStatusSocket = resolve
})
await page.routeWebSocket(/\/ws\/status\/$/, (socket) => {
resolveStatusSocket(socket)
})
await page.routeFromHAR(REQUESTS_HAR, { notFound: 'fallback' })
let previewRequestCount = 0
page.on('request', (request) => {
if (request.url().includes('/api/documents/175/preview/')) {
previewRequestCount++
}
})
await page.goto('/documents/175/details')
await page.locator('pngx-document-detail').waitFor()
await expect(page.getByTitle('Storage path', { exact: true })).toHaveText(
/\w+/
)
const previewWasLoaded = await page.evaluate(() => {
const detail = document.querySelector('pngx-document-detail')
const component = (window as any).ng.getComponent(detail)
component.pdfPreviewLoaded({ numPages: 1 })
return component.previewLoaded()
})
expect(previewWasLoaded).toBe(true)
const previewRequestsBeforeReload = previewRequestCount
const statusSocket = await statusSocketReady
const documentReloaded = page.waitForResponse(
(response) =>
response.url().includes('/api/documents/175/?full_perms=true') &&
response.request().method() === 'GET'
)
statusSocket.send(
JSON.stringify({
type: 'document_updated',
data: {
document_id: 175,
modified: '2026-07-26T20:00:00Z',
},
})
)
await documentReloaded
await expect(
page.getByText('Document reloaded with latest changes.').first()
).toBeVisible()
await expect
.poll(() => previewRequestCount)
.toBeGreaterThan(previewRequestsBeforeReload + 1)
})
+152 -148
View File
File diff suppressed because it is too large Load Diff
+25 -25
View File
@@ -1,6 +1,6 @@
{
"name": "paperless-ngx-ui",
"version": "3.0.0",
"version": "3.0.5",
"scripts": {
"preinstall": "npx only-allow pnpm",
"ng": "ng",
@@ -11,16 +11,16 @@
},
"private": true,
"dependencies": {
"@angular/cdk": "^22.0.3",
"@angular/common": "~22.0.5",
"@angular/compiler": "~22.0.5",
"@angular/core": "~22.0.5",
"@angular/forms": "~22.0.5",
"@angular/localize": "~22.0.5",
"@angular/platform-browser": "~22.0.5",
"@angular/router": "~22.0.5",
"@angular/cdk": "^22.0.6",
"@angular/common": "~22.0.8",
"@angular/compiler": "~22.0.8",
"@angular/core": "~22.0.8",
"@angular/forms": "~22.0.8",
"@angular/localize": "~22.0.8",
"@angular/platform-browser": "~22.0.8",
"@angular/router": "~22.0.8",
"@ng-bootstrap/ng-bootstrap": "^21.0.0",
"@ng-select/ng-select": "^23.2.0",
"@ng-select/ng-select": "^23.5.0",
"@ngneat/dirty-check-forms": "^3.0.3",
"@popperjs/core": "^2.11.8",
"bootstrap": "^5.3.8",
@@ -41,24 +41,24 @@
"devDependencies": {
"@angular-builders/custom-webpack": "^22.0.1",
"@angular-builders/jest": "^22.0.1",
"@angular-devkit/core": "^22.0.5",
"@angular-devkit/schematics": "^22.0.5",
"@angular-eslint/builder": "22.0.0",
"@angular-eslint/eslint-plugin": "22.0.0",
"@angular-eslint/eslint-plugin-template": "22.0.0",
"@angular-eslint/schematics": "22.0.0",
"@angular-eslint/template-parser": "22.0.0",
"@angular/build": "^22.0.5",
"@angular-devkit/core": "^22.0.8",
"@angular-devkit/schematics": "^22.0.8",
"@angular-eslint/builder": "22.1.0",
"@angular-eslint/eslint-plugin": "22.1.0",
"@angular-eslint/eslint-plugin-template": "22.1.0",
"@angular-eslint/schematics": "22.1.0",
"@angular-eslint/template-parser": "22.1.0",
"@angular/build": "^22.0.8",
"@angular/cli": "~22.0.5",
"@angular/compiler-cli": "~22.0.5",
"@angular/compiler-cli": "~22.0.8",
"@codecov/webpack-plugin": "^2.0.1",
"@playwright/test": "^1.61.1",
"@playwright/test": "^1.62.0",
"@types/jest": "^30.0.0",
"@types/node": "^26.0.0",
"@typescript-eslint/eslint-plugin": "^8.62.0",
"@typescript-eslint/parser": "^8.62.0",
"@typescript-eslint/utils": "^8.62.0",
"eslint": "^10.5.0",
"@types/node": "^26.1.1",
"@typescript-eslint/eslint-plugin": "^8.65.0",
"@typescript-eslint/parser": "^8.65.0",
"@typescript-eslint/utils": "^8.65.0",
"eslint": "^10.8.0",
"jest": "30.4.2",
"jest-environment-jsdom": "^30.4.1",
"jest-junit": "^17.0.0",
+793 -610
View File
File diff suppressed because it is too large Load Diff
+3
View File
@@ -2,6 +2,9 @@ packages:
- "."
minimumReleaseAge: 10080
trustPolicy: no-downgrade
trustPolicyExclude:
- "chokidar@4.0.3"
- "semver@6.3.1 || 5.7.2"
allowBuilds:
"@parcel/watcher": true
canvas: true
@@ -304,7 +304,7 @@
<div class="row mb-3">
<div class="col">
<p i18n>
Settings apply to this user account for objects (Tags, Mail Rules, etc. but not documents) created via the web UI.
Settings apply to this user account for objects (Tags, Mail Rules, etc.) created via the web UI. These settings do not apply to documents.
</p>
</div>
</div>
@@ -21,7 +21,7 @@
</div>
</pngx-page-header>
@if (!tasksService.completedFileTasks && tasksService.loading) {
@if (loading() && pagedTasks().length === 0) {
<div class="spinner-border spinner-border-sm fw-normal ms-2 me-auto" role="status"></div>
<div class="visually-hidden" i18n>Loading...</div>
}
@@ -182,7 +182,7 @@
container="body" triggers="mouseenter:mouseleave" popoverClass="popover-slim">
<i-bs class="me-2" name="stack"></i-bs><span><ng-container i18n>Attributes</ng-container></span>
</a>
@if (!slimSidebarEnabled) {
@if (!slimSidebarEnabled && canSaveSettings) {
<button
type="button"
class="btn btn-link btn-sm text-muted p-0 me-3 attributes-expand-btn"
@@ -280,10 +280,52 @@ main {
}
}
@media screen and (min-width: 376px) and (max-width: 768px) {
@media screen and (max-width: 575.98px) {
.navbar {
display: grid;
grid-template-columns: 1fr auto 1fr;
grid-template-areas:
"toggler brand actions"
"search search search";
}
.navbar-toggler {
// compensate for 2 buttons on the right
margin-right: 45px;
grid-area: toggler;
text-align: left;
}
.navbar-brand {
grid-area: brand;
min-width: 0;
max-width: 100%;
justify-self: center;
overflow: hidden;
> div,
.custom-title {
min-width: 0;
max-width: 100%;
overflow: hidden;
}
.title,
.byline {
max-width: 100%;
overflow: hidden;
text-overflow: ellipsis;
white-space: nowrap;
}
}
.search-container {
grid-area: search;
width: 100%;
}
.navbar > ul {
grid-area: actions;
justify-self: end;
flex-wrap: nowrap;
}
}
@@ -293,13 +335,6 @@ main {
}
}
@media screen and (max-width: 345px) {
.custom-title {
max-width: 110px;
overflow: hidden;
}
}
:host ::ng-deep .dropdown.show .dropdown-toggle,
:host ::ng-deep .dropdown-toggle:hover {
opacity: 0.7;
@@ -9,14 +9,14 @@
@for (message of messages(); track message) {
<div class="message d-flex flex-row small" [class.justify-content-end]="message.role === 'user'">
<div class="p-2 m-2" [class.bg-body]="message.role === 'user'">
<span>
<span class="text-break">
{{ message.content }}
@if (message.isStreaming) { <span class="blinking-cursor">|</span> }
</span>
@if (message.role === 'assistant' && message.references?.length) {
<div class="chat-references list-group mt-3">
@for (reference of message.references; track reference.id) {
<a class="list-group-item list-group-item-action text-primary" [routerLink]="['/documents', reference.id]">
<a class="list-group-item list-group-item-action d-flex text-primary text-break" [routerLink]="['/documents', reference.id]">
<i-bs width="0.9em" height="0.9em" name="file-text" class="me-1"></i-bs><span>{{ reference.title }}</span>
</a>
}
@@ -5,6 +5,7 @@
.chat-messages {
max-height: 350px;
overflow-y: auto;
white-space: pre-wrap;
}
.chat-references {
@@ -17,6 +17,6 @@
<ng-template #popoverContent>
<div>
{{confirmMessage}}&nbsp;<button class="btn btn-link btn-sm text-danger p-0 m-0 lh-1" type="button" (click)="onConfirm($event)">Yes</button>
{{confirmMessage}}&nbsp;<button class="btn btn-link btn-sm text-danger p-0 m-0 lh-1" type="button" (click)="onConfirm($event)" i18n>Yes</button>
</div>
</ng-template>
@@ -68,6 +68,11 @@
></ng-select>
} @else if (getCustomFieldByID(atom.field)?.data_type === CustomFieldDataType.DocumentLink) {
<pngx-input-document-link [(ngModel)]="atom.value" class="w-25 form-select doc-link-select p-0" placeholder="Search docs..." i18n-placeholder [minimal]="true"></pngx-input-document-link>
} @else if (getCustomFieldByID(atom.field)?.data_type === CustomFieldDataType.Monetary) {
<input class="w-25 form-control rounded-end" type="text" inputmode="decimal"
[ngModel]="atom.value"
(ngModelChange)="setMonetaryValue(atom, $event)"
[disabled]="disabled">
} @else {
<input class="w-25 form-control rounded-end" type="text" [(ngModel)]="atom.value" [disabled]="disabled">
}
@@ -1,5 +1,6 @@
import { provideHttpClient, withInterceptorsFromDi } from '@angular/common/http'
import { provideHttpClientTesting } from '@angular/common/http/testing'
import { LOCALE_ID } from '@angular/core'
import { ComponentFixture, TestBed } from '@angular/core/testing'
import { FormsModule, ReactiveFormsModule } from '@angular/forms'
import { NgbDropdownModule } from '@ng-bootstrap/ng-bootstrap'
@@ -41,6 +42,12 @@ const customFields = [
],
},
},
{
id: 3,
name: 'Test Monetary Field',
data_type: CustomFieldDataType.Monetary,
extra_data: { default_currency: 'EUR' },
},
]
describe('CustomFieldsQueryDropdownComponent', () => {
@@ -61,6 +68,7 @@ describe('CustomFieldsQueryDropdownComponent', () => {
providers: [
provideHttpClient(withInterceptorsFromDi()),
provideHttpClientTesting(),
{ provide: LOCALE_ID, useValue: 'de' },
],
}).compileComponents()
@@ -150,6 +158,22 @@ describe('CustomFieldsQueryDropdownComponent', () => {
expect(options2).toEqual([])
})
it('should normalize localized monetary comparison values', () => {
const atom = new CustomFieldQueryAtom([3, 'exact', null])
component.setMonetaryValue(atom, '1.234,56')
expect(atom.value).toEqual('1234.56')
})
it('should preserve API-formatted monetary comparison values', () => {
const atom = new CustomFieldQueryAtom([3, 'exact', null])
component.setMonetaryValue(atom, '1234.56')
expect(atom.value).toEqual('1234.56')
})
it('should remove an element from the selection model', () => {
const expression = new CustomFieldQueryExpression()
const atom = new CustomFieldQueryAtom()
@@ -1,9 +1,14 @@
import { NgTemplateOutlet } from '@angular/common'
import {
getLocaleNumberSymbol,
NgTemplateOutlet,
NumberSymbol,
} from '@angular/common'
import {
Component,
EventEmitter,
inject,
Input,
LOCALE_ID,
Output,
QueryList,
signal,
@@ -212,6 +217,7 @@ export class CustomFieldQueriesModel {
})
export class CustomFieldsQueryDropdownComponent extends LoadingComponentWithPermissions {
protected customFieldsService = inject(CustomFieldsService)
private readonly locale = inject(LOCALE_ID)
public CustomFieldQueryComponentType = CustomFieldQueryElementType
public CustomFieldQueryOperator = CustomFieldQueryOperator
@@ -376,4 +382,18 @@ export class CustomFieldsQueryDropdownComponent extends LoadingComponentWithPerm
}
return []
}
setMonetaryValue(atom: CustomFieldQueryAtom, value: string) {
// Normalize the decimal symbol e.g. . vs , by locale
const decimalSymbol = getLocaleNumberSymbol(
this.locale,
NumberSymbol.Decimal
)
if (decimalSymbol !== '.' && value.includes(decimalSymbol)) {
const groupSymbol = getLocaleNumberSymbol(this.locale, NumberSymbol.Group)
value = value.split(groupSymbol).join('').split(decimalSymbol).join('.')
}
atom.value = value
}
}
@@ -32,6 +32,8 @@ import { EditDialogComponent, EditDialogMode } from './edit-dialog.component'
template: `
<div>
<h4 class="modal-title" id="modal-basic-title">{{ getTitle() }}</h4>
<span class="error">{{ error?.name }}</span>
<button [disabled]="networkActive" (click)="save()">Save</button>
</div>
`,
imports: [FormsModule, ReactiveFormsModule],
@@ -276,4 +278,22 @@ describe('EditDialogComponent', () => {
expect(failedSpy).toHaveBeenCalled()
expect(component.error).toEqual('error')
})
it('should update the view after a failed save', async () => {
const button: HTMLButtonElement =
fixture.nativeElement.querySelector('button')
button.click()
await fixture.whenStable()
expect(button.disabled).toBe(true)
httpTestingController
.expectOne(`${environment.apiBaseUrl}tags/`)
.flush({ name: ['Name is required.'] }, { status: 400, statusText: '' })
await fixture.whenStable()
expect(button.disabled).toBe(false)
expect(fixture.nativeElement.querySelector('.error').textContent).toContain(
'Name is required.'
)
})
})
@@ -1,4 +1,5 @@
import {
ChangeDetectorRef,
Directive,
EventEmitter,
Input,
@@ -45,6 +46,7 @@ export abstract class EditDialogComponent<
protected userService = inject(UserService)
protected settingsService = inject(SettingsService)
protected permissionsService = inject(PermissionsService)
protected changeDetector = inject(ChangeDetectorRef)
dialogMode = model(EditDialogMode.CREATE)
@@ -108,10 +110,12 @@ export abstract class EditDialogComponent<
// wait to enable close button so it doesn't steal focus from input since its the first clickable element in the DOM
setTimeout(() => {
this.closeEnabled = true
this.changeDetector.markForCheck()
})
this.userService.listAll().subscribe((r) => {
this.users = r.results
this.changeDetector.markForCheck()
})
}
@@ -191,6 +195,7 @@ export abstract class EditDialogComponent<
this.error = error.error
this.networkActive = false
this.failed.next(error)
this.changeDetector.markForCheck()
},
})
}
@@ -129,13 +129,25 @@ describe('PngxPdfViewerComponent', () => {
;(component as any).applyScale()
expect(viewer.currentScaleValue).toBe(PdfZoomScale.PageFit)
expect(viewer.currentScale).toBe(2)
})
it('does not reapply scale for page-only changes', async () => {
await initComponent()
const pdf = (component as any).pdf as { numPages: number }
pdf.numPages = 3
const viewer = (component as any).pdfViewer as PDFViewer
viewer.setDocument(pdf)
const applyScaleSpy = jest.spyOn(component as any, 'applyScale')
component.page = 2
;(component as any).lastViewerPage = 2
;(component as any).applyViewerState()
component.ngOnChanges({
page: new SimpleChange(1, 2, false),
})
expect(viewer.currentPageNumber).toBe(2)
expect((component as any).lastViewerPage).toBeUndefined()
expect(applyScaleSpy).toHaveBeenCalled()
expect(applyScaleSpy).not.toHaveBeenCalled()
})
it('does not reset the viewer when it is already on the requested page', async () => {
@@ -283,6 +295,22 @@ describe('PngxPdfViewerComponent', () => {
expect(mockViewer.currentPageNumber).toBe(1)
})
it('reloads when the source revision changes', () => {
const resetSpy = jest.spyOn(component as any, 'resetViewerState')
const loadSpy = jest
.spyOn(component as any, 'loadDocument')
.mockImplementation(() => {})
component.src = 'test.pdf'
component.sourceRevision = 1
component.ngOnChanges({
sourceRevision: new SimpleChange(0, 1, false),
})
expect(resetSpy).toHaveBeenCalled()
expect(loadSpy).toHaveBeenCalled()
})
it('applies viewer state after view init when already loaded', () => {
const applySpy = jest.spyOn(component as any, 'applyViewerState')
;(component as any).hasLoaded = true
@@ -43,6 +43,7 @@ export class PngxPdfViewerComponent
private readonly document = inject<Document>(DOCUMENT)
@Input() src!: string
@Input() sourceRevision = 0
@Input() password?: string
@Input() page?: number
@Output() pageChange = new EventEmitter<number>()
@@ -93,7 +94,7 @@ export class PngxPdfViewerComponent
}
ngOnChanges(changes: SimpleChanges): void {
if (changes['src'] || changes['password']) {
if (changes['src'] || changes['sourceRevision'] || changes['password']) {
this.resetViewerState()
if (this.src) {
this.loadDocument()
@@ -115,7 +116,10 @@ export class PngxPdfViewerComponent
changes['zoomScale'] ||
changes['rotation']
) {
this.applyViewerState()
// Prevent loop with page / scale application see https://github.com/paperless-ngx/paperless-ngx/issues/13404
this.applyViewerState(
!!(changes['zoom'] || changes['zoomScale'] || changes['rotation'])
)
}
if (changes['searchQuery']) {
@@ -239,7 +243,7 @@ export class PngxPdfViewerComponent
}
}
private applyViewerState(): void {
private applyViewerState(applyScale = true): void {
if (!this.pdfViewer) {
return
}
@@ -263,7 +267,7 @@ export class PngxPdfViewerComponent
if (this.page === this.lastViewerPage) {
this.lastViewerPage = undefined
}
if (hasPages) {
if (hasPages && applyScale) {
this.applyScale()
}
this.dispatchFindIfReady()
@@ -1,6 +1,7 @@
<a [href]="link ?? previewUrl" class="{{linkClasses}}" [target]="linkTarget" [title]="linkTitle"
[ngbPopover]="previewContent" [popoverTitle]="document.title | documentTitle" container="body"
autoClose="true" [popoverClass]="popoverClass()" (mouseenter)="mouseEnterPreview()" (mouseleave)="mouseLeavePreview()" #popover="ngbPopover">
autoClose="true" [popoverClass]="popoverClass()" [popperOptions]="popperOptions"
(mouseenter)="mouseEnterPreview()" (mouseleave)="mouseLeavePreview()" #popover="ngbPopover">
<ng-content></ng-content>
</a>
<ng-template #previewContent>
@@ -67,6 +67,10 @@ describe('PreviewPopupComponent', () => {
expect(component.useNativePdfViewer).toBeTruthy()
})
it('should use fixed positioning for the preview popover', () => {
expect(component.popperOptions({}).strategy).toEqual('fixed')
})
it('should render object if native PDF viewer enabled', () => {
settingsService.set(SETTINGS_KEYS.USE_NATIVE_PDF_VIEWER, true)
component.popover.open()
@@ -8,6 +8,7 @@ import {
ViewChild,
} from '@angular/core'
import { NgbPopover, NgbPopoverModule } from '@ng-bootstrap/ng-bootstrap'
import { Options } from '@popperjs/core'
import { NgxBootstrapIconsModule } from 'ngx-bootstrap-icons'
import { first, Subject, takeUntil } from 'rxjs'
import { Document } from 'src/app/data/document'
@@ -74,6 +75,11 @@ export class PreviewPopupComponent implements OnDestroy {
readonly popoverClass = signal('shadow popover-preview')
readonly popperOptions = (options: Partial<Options>): Partial<Options> => ({
...options,
strategy: 'fixed',
})
get renderAsObject(): boolean {
return (this.isPdf && this.useNativePdfViewer) || !this.isPdf
}
@@ -113,8 +113,8 @@
<form [formGroup]='documentForm' (ngSubmit)="save()">
<div class="btn-toolbar mb-1 border-bottom">
<div class="btn-group pb-3">
<div class="btn-toolbar justify-content-end mb-1 pb-3 gap-2 row-gap-2 border-bottom">
<div class="btn-group me-auto">
<button type="button" class="btn btn-sm btn-outline-secondary" i18n-title title="Close" (click)="close()">
<i-bs width="1.2em" height="1.2em" name="x"></i-bs>
</button>
@@ -127,32 +127,36 @@
</div>
<ng-container *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.Document }">
<div class="btn-group pb-3 ms-auto">
<pngx-suggestions-dropdown *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.Document }"
[disabled]="!userCanEdit || suggestionsLoading()"
[loading]="suggestionsLoading()"
[suggestions]="suggestions()"
[aiEnabled]="aiEnabled"
(getSuggestions)="getSuggestions()"
(addTag)="createTag($event)"
(addDocumentType)="createDocumentType($event)"
(addCorrespondent)="createCorrespondent($event)">
</pngx-suggestions-dropdown>
<div class="d-flex gap-2">
<div class="btn-group">
<pngx-suggestions-dropdown *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.Document }"
[disabled]="!userCanEdit || suggestionsLoading()"
[loading]="suggestionsLoading()"
[suggestions]="suggestions()"
[aiEnabled]="aiEnabled"
(getSuggestions)="getSuggestions()"
(addTag)="createTag($event)"
(addDocumentType)="createDocumentType($event)"
(addCorrespondent)="createCorrespondent($event)">
</pngx-suggestions-dropdown>
</div>
<div class="btn-group">
<pngx-custom-fields-dropdown
*pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.CustomField }"
[documentId]="documentId()"
[disabled]="!userCanEdit"
[existingFields]="document()?.custom_fields"
(created)="refreshCustomFields()"
(added)="addField($event)">
</pngx-custom-fields-dropdown>
</div>
</div>
<div class="btn-group pb-3 ms-2">
<pngx-custom-fields-dropdown
*pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.CustomField }"
[documentId]="documentId()"
[disabled]="!userCanEdit"
[existingFields]="document()?.custom_fields"
(created)="refreshCustomFields()"
(added)="addField($event)">
</pngx-custom-fields-dropdown>
<div class="ps-3">
<ng-container *ngTemplateOutlet="saveButtons"></ng-container>
</div>
</ng-container>
<ng-container *ngTemplateOutlet="saveButtons"></ng-container>
</div>
<ul ngbNav #nav="ngbNav" class="nav-underline flex-nowrap flex-md-wrap overflow-auto" (navChange)="onNavChange($event)" [activeId]="activeNavID()" (activeIdChange)="activeNavID.set($event)">
@@ -276,7 +280,7 @@
}
</div>
<div class="d-flex border-top pt-3">
<div class="d-flex justify-content-end border-top pt-3">
<ng-container *ngTemplateOutlet="saveButtons"></ng-container>
</div>
</ng-template>
@@ -440,7 +444,7 @@
</div>
<ng-template #saveButtons>
<div class="btn-group pb-3 ms-4">
<div class="btn-group">
<ng-container *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.Document }">
<button type="submit" class="order-3 btn btn-sm btn-primary" i18n [disabled]="!userCanEdit || networkActive() || (isDirty$ | async) !== true">Save</button>
@if (hasNext()) {
@@ -472,6 +476,7 @@
<div class="preview-sticky pdf-viewer-container">
<pngx-pdf-viewer
[src]="pdfSource()"
[sourceRevision]="previewRevision()"
[password]="pdfPassword()"
[renderMode]="PdfRenderMode.All"
[page]="previewCurrentPage()" (pageChange)="previewCurrentPage.set($event)"
@@ -1602,6 +1602,7 @@ describe('DocumentDetailComponent', () => {
expect(openDoc.__changedFields).toEqual([])
expect(setDirtySpy).toHaveBeenCalledWith(openDoc, false)
expect(saveSpy).toHaveBeenCalled()
expect(component.previewRevision()).toBe(1)
})
it('should ignore incoming update for a different document id', () => {
@@ -249,6 +249,7 @@ export class DocumentDetailComponent
titleSubject: Subject<string> = new Subject()
readonly previewUrl = signal<string>(undefined)
readonly pdfSource = signal<string>(undefined)
readonly previewRevision = signal(0)
readonly pdfPassword = signal<string>(undefined)
readonly thumbUrl = signal<string>(undefined)
readonly previewText = signal<string>(undefined)
@@ -609,6 +610,9 @@ export class DocumentDetailComponent
.subscribe()
}
this.updateComponent(useDoc)
if (forceRemote) {
this.previewRevision.update((revision) => revision + 1)
}
this.titleSubject
.pipe(
debounceTime(1000),
@@ -25,7 +25,7 @@
<input #textFilterInput class="form-control form-control-sm" type="text"
[disabled]="textFilterModifierIsNull"
[(ngModel)]="textFilter"
(keyup)="textFilterKeyup($event)"
(keydown)="textFilterKeydown($event)"
[ngbTypeahead]="searchAutoComplete"
(selectItem)="itemSelected($event)"
[readonly]="textFilterTarget === 'fulltext-morelike'">
@@ -2180,17 +2180,17 @@ describe('FilterEditorComponent', () => {
it('should support Enter / Esc key on text field', () => {
component.textFilterInput.nativeElement.value = 'foo'
component.textFilterInput.nativeElement.dispatchEvent(
new KeyboardEvent('keyup', { key: 'Enter' })
new KeyboardEvent('keydown', { key: 'Enter' })
)
expect(component.textFilter).toEqual('foo')
component.textFilterInput.nativeElement.value = 'foo bar'
component.textFilterInput.nativeElement.dispatchEvent(
new KeyboardEvent('keyup', { key: 'Escape' })
new KeyboardEvent('keydown', { key: 'Escape' })
)
expect(component.textFilter).toEqual('')
const blurSpy = jest.spyOn(component.textFilterInput.nativeElement, 'blur')
component.textFilterInput.nativeElement.dispatchEvent(
new KeyboardEvent('keyup', { key: 'Escape' })
new KeyboardEvent('keydown', { key: 'Escape' })
)
expect(blurSpy).toHaveBeenCalled()
})
@@ -2269,4 +2269,34 @@ describe('FilterEditorComponent', () => {
component.itemSelected({ item: 'world', preventDefault: () => true })
expect(component.textFilter).toEqual('hello world ')
})
it('should choose the active autocomplete item with Enter', () => {
component.textFilterTarget = 'fulltext-query'
jest
.spyOn(searchService, 'autocomplete')
.mockReturnValue(of(['hello', 'help']))
const input = component.textFilterInput.nativeElement as HTMLInputElement
input.value = 'he'
input.dispatchEvent(new Event('input'))
tick(250)
input.dispatchEvent(
new KeyboardEvent('keydown', {
key: 'ArrowDown',
bubbles: true,
cancelable: true,
})
)
input.dispatchEvent(
new KeyboardEvent('keydown', {
key: 'Enter',
bubbles: true,
cancelable: true,
})
)
fixture.detectChanges()
expect(component.textFilter).toEqual('help ')
})
})
@@ -1312,8 +1312,12 @@ export class FilterEditorComponent
}
}
textFilterKeyup(event: KeyboardEvent) {
textFilterKeydown(event: KeyboardEvent) {
if (event.key == 'Enter') {
if (event.defaultPrevented) {
// NgbTypeahead calls preventDefault, so use that to detect if the Enter key was for the dropdown
return
}
const filterString = (
this.textFilterInput.nativeElement as HTMLInputElement
).value
@@ -51,8 +51,8 @@
*pngxIfPermissions="{ action: PermissionAction.Add, type: activeManagementList.permissionType }">
<i-bs name="plus-circle" class="me-1"></i-bs><ng-container i18n>Create</ng-container>
</button>
} @else if (activeCustomFields) {
<button type="button" class="btn btn-sm btn-outline-primary" (click)="activeCustomFields.editField()"
} @else if (customFieldsActive) {
<button type="button" class="btn btn-sm btn-outline-primary" (click)="addCustomField()"
*pngxIfPermissions="{ action: PermissionAction.Add, type: PermissionType.CustomField }">
<i-bs name="plus-circle" class="me-1"></i-bs><ng-container i18n>Add Field</ng-container>
</button>
@@ -18,6 +18,8 @@ import {
DocumentAttributesComponent,
DocumentAttributesSectionKind,
} from './document-attributes.component'
import { CustomFieldsComponent } from './custom-fields/custom-fields.component'
import { ManagementListComponent } from './management-list/management-list.component'
@Component({
selector: 'pngx-dummy-section',
@@ -170,10 +172,32 @@ describe('DocumentAttributesComponent', () => {
expect(component.activeManagementList).toBeNull()
component.activeNavID.set(1)
const managementList = Object.create(ManagementListComponent.prototype)
component.activeOutlet = {
componentInstance: managementList,
} as any
expect(component.activeSection.kind).toBe(
DocumentAttributesSectionKind.ManagementList
)
expect(component.activeManagementList).toBeDefined()
expect(component.activeManagementList).toBe(managementList)
})
it('should use the current component instance when the outlet is reused', () => {
jest.spyOn(permissionsService, 'currentUserCan').mockReturnValue(true)
component.activeNavID.set(1)
const firstManagementList = Object.create(ManagementListComponent.prototype)
const secondManagementList = Object.create(
ManagementListComponent.prototype
)
component.activeOutlet = {
componentInstance: firstManagementList,
} as any
expect(component.activeManagementList).toBe(firstManagementList)
component.activeOutlet.componentInstance = secondManagementList
expect(component.activeManagementList).toBe(secondManagementList)
})
it('should return activeCustomFields correctly', () => {
@@ -184,6 +208,29 @@ describe('DocumentAttributesComponent', () => {
expect(component.activeSection.kind).toBe(
DocumentAttributesSectionKind.CustomFields
)
expect(component.activeCustomFields).toBeDefined()
const customFields = Object.create(CustomFieldsComponent.prototype)
customFields.editField = jest.fn()
component.activeOutlet = {
componentInstance: customFields,
} as any
expect(component.activeCustomFields).toBe(customFields)
component.addCustomField()
expect(customFields.editField).toHaveBeenCalled()
})
it('should show the add field button before the custom fields instance is available', async () => {
jest.spyOn(permissionsService, 'currentUserCan').mockReturnValue(true)
fixture.detectChanges()
component.activeNavID.set(2)
await fixture.whenStable()
expect(component.activeCustomFields).toBeNull()
expect(
fixture.nativeElement.querySelector(
'pngx-page-header .btn-outline-primary'
)?.textContent
).toContain('Add Field')
})
})
@@ -132,11 +132,8 @@ export class DocumentAttributesComponent implements OnInit, OnDestroy {
]
@ViewChild('activeOutlet', { read: NgComponentOutlet })
set activeOutlet(outlet: NgComponentOutlet | undefined) {
this.activeComponent.set(outlet?.componentInstance ?? null)
}
activeOutlet: NgComponentOutlet
readonly activeComponent = signal<unknown>(null)
readonly activeNavID = signal<number>(null)
get visibleSections(): DocumentAttributesSection[] {
@@ -161,17 +158,22 @@ export class DocumentAttributesComponent implements OnInit, OnDestroy {
this.activeSection?.kind !== DocumentAttributesSectionKind.ManagementList
)
return null
const instance = this.activeComponent()
const instance = this.activeOutlet?.componentInstance
return instance instanceof ManagementListComponent ? instance : null
}
get activeCustomFields(): CustomFieldsComponent | null {
if (this.activeSection?.kind !== DocumentAttributesSectionKind.CustomFields)
return null
const instance = this.activeComponent()
if (!this.customFieldsActive) return null
const instance = this.activeOutlet?.componentInstance
return instance instanceof CustomFieldsComponent ? instance : null
}
get customFieldsActive(): boolean {
return (
this.activeSection?.kind === DocumentAttributesSectionKind.CustomFields
)
}
get activeTabLabel(): string {
return this.activeSection?.label ?? ''
}
@@ -227,6 +229,10 @@ export class DocumentAttributesComponent implements OnInit, OnDestroy {
this.router.navigate(['attributes', nextSection])
}
addCustomField(): void {
this.activeCustomFields?.editField(null)
}
private getDefaultNavID(): DocumentAttributesNavIDs | null {
return this.visibleSections[0]?.id ?? null
}
@@ -108,6 +108,18 @@ describe('PermissionsService', () => {
actionKey: 'View', // PermissionAction.View
typeKey: 'Document', // PermissionType.Document
})
expect(
permissionsService.getPermissionKeys('view_global_statistics')
).toEqual({
actionKey: 'View', // PermissionAction.View
typeKey: 'GlobalStatistics', // PermissionType.GlobalStatistics
})
expect(
permissionsService.getPermissionKeys('view_system_monitoring')
).toEqual({
actionKey: 'View', // PermissionAction.View
typeKey: 'SystemMonitoring', // PermissionType.SystemMonitoring
})
})
it('correctly checks explicit global permissions', () => {
+7 -10
View File
@@ -110,19 +110,16 @@ export class PermissionsService {
actionKey: string
typeKey: string
} {
const matches = permissionStr.match(/(.+)_/)
let typeKey
let actionKey
if (matches?.length > 0) {
const action = matches[1]
const actionIndex = Object.values(PermissionAction).indexOf(
action as PermissionAction
)
if (actionIndex > -1) {
actionKey = Object.keys(PermissionAction)[actionIndex]
}
const actionIndex = Object.values(PermissionAction).findIndex((action) =>
permissionStr.startsWith(`${action}_`)
)
if (actionIndex > -1) {
const action = Object.values(PermissionAction)[actionIndex]
actionKey = Object.keys(PermissionAction)[actionIndex]
const typeIndex = Object.values(PermissionType).indexOf(
permissionStr.replace(action, '%s') as PermissionType
permissionStr.replace(`${action}_`, '%s_') as PermissionType
)
if (typeIndex > -1) {
typeKey = Object.keys(PermissionType)[typeIndex]
+3 -4
View File
@@ -17,7 +17,7 @@ import {
estimateBrightnessForColor,
hexToHsl,
} from 'src/app/utils/color'
import { environment } from 'src/environments/environment'
import { DEFAULT_APP_TITLE, environment } from 'src/environments/environment'
import { DEFAULT_DISPLAY_FIELDS, DisplayField } from '../data/document'
import { SavedView } from '../data/saved-view'
import {
@@ -359,9 +359,8 @@ export class SettingsService {
}),
tap((uisettings) => {
this.assignSafeSettings(uisettings.settings)
if (this.get(SETTINGS_KEYS.APP_TITLE)?.length) {
environment.appTitle = this.get(SETTINGS_KEYS.APP_TITLE)
}
environment.appTitle =
this.get(SETTINGS_KEYS.APP_TITLE) || DEFAULT_APP_TITLE
this.maybeMigrateSettings()
// to update lang cookie
if (this.settings['language']?.length)
+56 -76
View File
@@ -50,13 +50,66 @@ describe('TasksService', () => {
req.flush({ count: 0, results: [] })
})
it('does not call tasks api endpoint on reload if already loading', () => {
tasksService.loading = true
it('cancels an in-progress reload when reloading again', () => {
tasksService.reload()
httpTestingController.expectNone(
const staleReload = httpTestingController.expectOne(
(req: HttpRequest<unknown>) =>
req.url === `${environment.apiBaseUrl}tasks/`
)
tasksService.reload()
expect(staleReload.cancelled).toBe(true)
httpTestingController
.expectOne(
(req: HttpRequest<unknown>) =>
req.url === `${environment.apiBaseUrl}tasks/`
)
.flush({ count: 0, results: [] })
})
it('continues reloading after a reload request fails', () => {
tasksService.reload()
httpTestingController
.expectOne(
(req: HttpRequest<unknown>) =>
req.url === `${environment.apiBaseUrl}tasks/`
)
.flush('error', { status: 500, statusText: 'error' })
expect(tasksService.loading).toBe(false)
tasksService.reload()
httpTestingController
.expectOne(
(req: HttpRequest<unknown>) =>
req.url === `${environment.apiBaseUrl}tasks/`
)
.flush({ count: 0, results: [] })
})
it('reloads after dismissing a task while a reload is already in progress', () => {
tasksService.reload()
const staleReload = httpTestingController.expectOne(
(req: HttpRequest<unknown>) =>
req.url === `${environment.apiBaseUrl}tasks/` &&
req.params.get('acknowledged') === 'false'
)
tasksService.dismissTasks(new Set([1])).subscribe()
httpTestingController
.expectOne(`${environment.apiBaseUrl}tasks/acknowledge/`)
.flush([])
expect(staleReload.cancelled).toBe(true)
httpTestingController
.expectOne(
(req: HttpRequest<unknown>) =>
req.url === `${environment.apiBaseUrl}tasks/` &&
req.params.get('acknowledged') === 'false'
)
.flush({ count: 0, results: [] })
expect(tasksService.needsAttentionTasks).toHaveLength(0)
})
it('calls acknowledge_tasks api endpoint on dismiss and reloads', () => {
@@ -101,79 +154,6 @@ describe('TasksService', () => {
.flush({ count: 0, results: [] })
})
it('groups mixed task types by status when reloading', () => {
expect(tasksService.total).toEqual(0)
const mockTasks = [
{
task_type: PaperlessTaskType.ConsumeFile,
trigger_source: PaperlessTaskTriggerSource.FolderConsume,
status: PaperlessTaskStatus.Success,
acknowledged: false,
task_id: '1234',
input_data: { filename: 'file1.pdf' },
date_created: new Date(),
related_document_ids: [],
},
{
task_type: PaperlessTaskType.SanityCheck,
trigger_source: PaperlessTaskTriggerSource.System,
status: PaperlessTaskStatus.Failure,
acknowledged: false,
task_id: '1235',
input_data: {},
date_created: new Date(),
related_document_ids: [],
},
{
task_type: PaperlessTaskType.MailFetch,
trigger_source: PaperlessTaskTriggerSource.Scheduled,
status: PaperlessTaskStatus.Pending,
acknowledged: false,
task_id: '1236',
input_data: {},
date_created: new Date(),
related_document_ids: [],
},
{
task_type: PaperlessTaskType.LlmIndex,
trigger_source: PaperlessTaskTriggerSource.WebUI,
status: PaperlessTaskStatus.Started,
acknowledged: false,
task_id: '1237',
input_data: {},
date_created: new Date(),
related_document_ids: [],
},
{
task_type: PaperlessTaskType.EmptyTrash,
trigger_source: PaperlessTaskTriggerSource.Manual,
status: PaperlessTaskStatus.Success,
acknowledged: false,
task_id: '1238',
input_data: {},
date_created: new Date(),
related_document_ids: [],
},
]
tasksService.reload()
const req = httpTestingController.expectOne(
(req: HttpRequest<unknown>) =>
req.url === `${environment.apiBaseUrl}tasks/` &&
req.params.get('acknowledged') === 'false' &&
req.params.get('page_size') === '1000'
)
req.flush({ count: mockTasks.length, results: mockTasks })
expect(tasksService.allFileTasks).toHaveLength(5)
expect(tasksService.completedFileTasks).toHaveLength(2)
expect(tasksService.failedFileTasks).toHaveLength(1)
expect(tasksService.queuedFileTasks).toHaveLength(1)
expect(tasksService.startedFileTasks).toHaveLength(1)
})
it('includes revoked tasks in needs attention', () => {
const mockTasks = [
{
+39 -50
View File
@@ -1,7 +1,15 @@
import { HttpClient } from '@angular/common/http'
import { Injectable, inject, signal } from '@angular/core'
import { Observable, Subject } from 'rxjs'
import { first, map, takeUntil, tap } from 'rxjs/operators'
import { EMPTY, Observable, Subject } from 'rxjs'
import {
catchError,
finalize,
first,
map,
switchMap,
takeUntil,
tap,
} from 'rxjs/operators'
import {
PaperlessTask,
PaperlessTaskStatus,
@@ -23,44 +31,40 @@ export class TasksService {
public loading: boolean = false
private readonly fileTasks = signal<PaperlessTask[]>([])
private readonly tasks = signal<PaperlessTask[]>([])
private readonly reloadNotifier = new Subject<void>()
private unsubscribeNotifer: Subject<any> = new Subject()
public get total(): number {
return this.fileTasks().length
}
public get allFileTasks(): PaperlessTask[] {
return this.fileTasks().slice(0)
}
public get queuedFileTasks(): PaperlessTask[] {
return this.fileTasks().filter(
(t) => t.status === PaperlessTaskStatus.Pending
)
}
public get startedFileTasks(): PaperlessTask[] {
return this.fileTasks().filter(
(t) => t.status === PaperlessTaskStatus.Started
)
}
public get completedFileTasks(): PaperlessTask[] {
return this.fileTasks().filter(
(t) => t.status === PaperlessTaskStatus.Success
)
}
public get failedFileTasks(): PaperlessTask[] {
return this.fileTasks().filter(
(t) => t.status === PaperlessTaskStatus.Failure
)
constructor() {
this.reloadNotifier
.pipe(
switchMap(() => {
this.loading = true
return this.http
.get<Results<PaperlessTask>>(`${this.baseUrl}${this.endpoint}/`, {
params: {
acknowledged: 'false',
page_size: this.defaultReloadPageSize,
},
})
.pipe(
map((response) => response.results),
takeUntil(this.unsubscribeNotifer),
catchError(() => EMPTY),
finalize(() => {
this.loading = false
})
)
})
)
.subscribe((tasks) => {
this.tasks.set(tasks)
})
}
public get needsAttentionTasks(): PaperlessTask[] {
return this.fileTasks().filter((t) =>
return this.tasks().filter((t) =>
[PaperlessTaskStatus.Failure, PaperlessTaskStatus.Revoked].includes(
t.status
)
@@ -68,22 +72,7 @@ export class TasksService {
}
public reload() {
if (this.loading) return
this.loading = true
this.http
.get<Results<PaperlessTask>>(`${this.baseUrl}${this.endpoint}/`, {
params: {
acknowledged: 'false',
page_size: this.defaultReloadPageSize,
},
})
.pipe(map((r) => r.results))
.pipe(takeUntil(this.unsubscribeNotifer), first())
.subscribe((r) => {
this.fileTasks.set(r)
this.loading = false
})
this.reloadNotifier.next()
}
public list(
+16
View File
@@ -14,4 +14,20 @@ describe('text search utilities', () => {
expect(matchesSearchText('Tax\u00e9s 2026', 'taxe 26')).toBeTruthy()
expect(matchesSearchText('taxes 2026', 'tax receipt')).toBeFalsy()
})
it('matches a large set of tag names without blocking input', () => {
const tagNames = Array.from(
{ length: 1280 },
(_, index) =>
`Customer Party ${index} Jos\u00e9 M\u00fcller \u00c5ngstr\u00f6m`
)
const start = performance.now()
const matches = tagNames.filter((name) => matchesSearchText(name, 'party'))
const duration = performance.now() - start
expect(matches).toHaveLength(tagNames.length)
// The previous implementation took roughly 500 ms for this workload.
expect(duration).toBeLessThan(250)
})
})
+9 -2
View File
@@ -1,10 +1,17 @@
import { normalizeSync } from 'normalize-diacritics'
import { diacritics } from 'normalize-diacritics/diacritics'
export type SearchTextValue =
string | number | boolean | bigint | null | undefined
export function normalizeSearchText(value: SearchTextValue): string {
return normalizeSync(String(value ?? '')).toLocaleLowerCase()
const normalized = diacritics.reduce(
(text, replacement) => {
return text.replace(replacement.diacritics, replacement.letter)
},
String(value ?? '')
)
return normalized.toLocaleLowerCase()
}
export function matchesSearchText(
+4 -2
View File
@@ -1,12 +1,14 @@
const base_url = new URL(document.baseURI)
export const DEFAULT_APP_TITLE = 'Paperless-ngx'
export const environment = {
production: true,
apiBaseUrl: document.baseURI + 'api/',
apiVersion: '10', // match src/paperless/settings.py
appTitle: 'Paperless-ngx',
appTitle: DEFAULT_APP_TITLE,
tag: 'prod',
version: '3.0.0',
version: '3.0.5',
webSocketHost: window.location.host,
webSocketProtocol: window.location.protocol == 'https:' ? 'wss:' : 'ws:',
webSocketBaseUrl: base_url.pathname + 'ws/',
+3 -1
View File
@@ -2,11 +2,13 @@
// `ng build --configuration production` replaces `environment.ts` with `environment.prod.ts`.
// The list of file replacements can be found in `angular.json`.
export const DEFAULT_APP_TITLE = 'Paperless-ngx'
export const environment = {
production: false,
apiBaseUrl: 'http://localhost:8000/api/',
apiVersion: '10',
appTitle: 'Paperless-ngx',
appTitle: DEFAULT_APP_TITLE,
tag: 'dev',
version: 'DEVELOPMENT',
webSocketHost: 'localhost:8000',
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff

Some files were not shown because too many files have changed in this diff Show More