Compare commits

...

37 Commits

Author SHA1 Message Date
shamoon
0e0407fc40 empty commit to test ci 2026-03-05 16:13:18 -08:00
shamoon
afd5e4a3cd Add path-change detection and CI gate jobs 2026-03-05 15:59:06 -08:00
dependabot[bot]
a5a267fe49 Bump django-allauth from 65.14.0 to 65.14.1 (#12253)
Bumps [django-allauth](https://github.com/sponsors/pennersr) from 65.14.0 to 65.14.1.
- [Commits](https://github.com/sponsors/pennersr/commits)

---
updated-dependencies:
- dependency-name: django-allauth
  dependency-version: 65.14.1
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-05 14:32:04 -08:00
shamoon
24a2cfd957 Change: use explicit doc creation instead of clone for versions (#12226) 2026-03-04 15:57:44 -08:00
GitHub Actions
7cf2ef6398 Auto translate strings 2026-03-04 23:29:54 +00:00
shamoon
df03207eef Fix: correct doc version filename handling (#12223) 2026-03-04 23:28:07 +00:00
dependabot[bot]
fa998ecd49 Bump django from 5.2.11 to 5.2.12 (#12249)
Bumps [django](https://github.com/django/django) from 5.2.11 to 5.2.12.
- [Commits](https://github.com/django/django/compare/5.2.11...5.2.12)

---
updated-dependencies:
- dependency-name: django
  dependency-version: 5.2.12
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-04 15:16:25 -08:00
Trenton H
1e21bcd26e Breaking: Drop support for Python 3.10 (#12234) 2026-03-04 15:03:33 -08:00
Trenton H
a9cb89c633 Enhancement: Improve exporter memory efficiency (#12236)
Phase 1 -- Eliminate JSON round-trip in document exporter

Replace json.loads(serializers.serialize("json", qs)) with
serializers.serialize("python", qs) to skip the intermediate
JSON string allocation and parse step. Use DjangoJSONEncoder
in check_and_write_json() to handle native Python types
(datetime, Decimal, UUID) the Python serializer returns.

Phase 2 -- Batched QuerySet serialization in document exporter

Add serialize_queryset_batched() helper that uses QuerySet.iterator()
and itertools.islice to stream records in configurable chunks, bounding
peak memory during serialization to batch_size * avg_record_size rather
than loading the entire QuerySet at once.
2026-03-04 14:54:20 -08:00
GitHub Actions
a37e24c1ad Auto translate strings 2026-03-04 22:17:32 +00:00
shamoon
85a18e5911 Enhancement: saved view sharing (#12142) 2026-03-04 14:15:43 -08:00
GitHub Actions
ae182c459b Auto translate strings 2026-03-04 21:34:02 +00:00
shamoon
d51a118aac Merge branch 'main' into dev 2026-03-04 13:31:20 -08:00
github-actions[bot]
d6a316b1df Changelog v2.20.10 - GHA (#12247)
Co-authored-by: github-actions <41898282+github-actions[bot]@users.noreply.github.com>
2026-03-04 11:25:44 -08:00
shamoon
8f311c4b6b Bump version to 2.20.10 2026-03-04 10:38:14 -08:00
shamoon
f25322600d Merge branch 'release/v2.20.x' 2026-03-04 10:09:01 -08:00
shamoon
615f27e6fb Fix: support string coercion in filepath jinja templates (#12244) 2026-03-04 08:32:34 -08:00
Andreas Schneider
190fc70288 Fix: use maxsplit=1 in Redis URL parsing to handle URLs with multiple colons (#12239) 2026-03-04 01:06:51 -08:00
shamoon
5b809122b5 Fix: apply ordering after annotating tag document count (#12238) 2026-03-04 00:33:13 -08:00
GitHub Actions
c623234769 Auto translate strings 2026-03-04 00:29:21 +00:00
shamoon
299dac21ee Enhancement: “live” document updates (#12141) 2026-03-04 00:27:07 +00:00
Trenton H
5498503d60 Chore: Improve user migration path (#12232)
Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
2026-03-03 15:51:48 -08:00
shamoon
8b8307571a Fix: enforce path limit for db filename fields (#12235) 2026-03-03 13:19:56 -08:00
GitHub Actions
16b58c2de5 Auto translate strings 2026-03-03 19:25:03 +00:00
shamoon
c724fbb5d9 Clarify bulk edit wording with versions 2026-03-03 11:22:22 -08:00
dependabot[bot]
9c0f112e94 docker(deps): Bump astral-sh/uv (#12191)
Bumps [astral-sh/uv](https://github.com/astral-sh/uv) from 0.10.5-python3.12-trixie-slim to 0.10.7-python3.12-trixie-slim.
- [Release notes](https://github.com/astral-sh/uv/releases)
- [Changelog](https://github.com/astral-sh/uv/blob/main/CHANGELOG.md)
- [Commits](https://github.com/astral-sh/uv/compare/0.10.5...0.10.7)

---
updated-dependencies:
- dependency-name: astral-sh/uv
  dependency-version: 0.10.7-python3.12-trixie-slim
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-03 07:56:35 -08:00
Trenton H
43406f44f2 Feature: Improve the retagger output using rich (#12194) 2026-03-03 07:14:59 -08:00
shamoon
b7ca3550b1 Merge branch 'main' into dev 2026-03-02 13:45:10 -08:00
shamoon
0e97419e0e Chore: add existing logo for temporary url resolution 2026-03-02 13:43:24 -08:00
dependabot[bot]
10cb2ac183 Chore(deps): Bump the actions group across 1 directory with 6 updates (#12224)
Bumps the actions group with 6 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [astral-sh/setup-uv](https://github.com/astral-sh/setup-uv) | `7.3.0` | `7.3.1` |
| [actions/upload-artifact](https://github.com/actions/upload-artifact) | `6.0.0` | `7.0.0` |
| [actions/download-artifact](https://github.com/actions/download-artifact) | `7.0.0` | `8.0.0` |
| [github/codeql-action](https://github.com/github/codeql-action) | `4.32.3` | `4.32.5` |
| [crowdin/github-action](https://github.com/crowdin/github-action) | `2.14.0` | `2.15.0` |
| [actions/stale](https://github.com/actions/stale) | `10.1.1` | `10.2.0` |



Updates `astral-sh/setup-uv` from 7.3.0 to 7.3.1
- [Release notes](https://github.com/astral-sh/setup-uv/releases)
- [Commits](https://github.com/astral-sh/setup-uv/compare/v7.3.0...v7.3.1)

Updates `actions/upload-artifact` from 6.0.0 to 7.0.0
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/v6.0.0...v7.0.0)

Updates `actions/download-artifact` from 7.0.0 to 8.0.0
- [Release notes](https://github.com/actions/download-artifact/releases)
- [Commits](https://github.com/actions/download-artifact/compare/v7.0.0...v8.0.0)

Updates `github/codeql-action` from 4.32.3 to 4.32.5
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/v4.32.3...v4.32.5)

Updates `crowdin/github-action` from 2.14.0 to 2.15.0
- [Release notes](https://github.com/crowdin/github-action/releases)
- [Commits](https://github.com/crowdin/github-action/compare/v2.14.0...v2.15.0)

Updates `actions/stale` from 10.1.1 to 10.2.0
- [Release notes](https://github.com/actions/stale/releases)
- [Changelog](https://github.com/actions/stale/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/stale/compare/v10.1.1...v10.2.0)

---
updated-dependencies:
- dependency-name: astral-sh/setup-uv
  dependency-version: 7.3.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: actions/upload-artifact
  dependency-version: 7.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: actions
- dependency-name: actions/download-artifact
  dependency-version: 8.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: actions
- dependency-name: github/codeql-action
  dependency-version: 4.32.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: actions
- dependency-name: crowdin/github-action
  dependency-version: 2.15.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions
- dependency-name: actions/stale
  dependency-version: 10.2.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-02 11:55:24 -08:00
Trenton H
1d7cd5a7ad Chore: Updates actions to the most specific version released (#12222) 2026-03-02 11:34:57 -08:00
Trenton H
e58a35d40c Feature: Transition sanity check to rich and improve output (#12182) 2026-03-02 10:53:39 -08:00
Trenton H
20a9cd40e8 Feature: Switch all indexing to use rich (#12193) 2026-03-02 10:41:09 -08:00
dependabot[bot]
b94ce85b46 Chore(deps): Bump whitenoise in the django-ecosystem group (#12192)
Bumps the django-ecosystem group with 1 update: [whitenoise](https://github.com/evansd/whitenoise).


Updates `whitenoise` from 6.11.0 to 6.12.0
- [Changelog](https://github.com/evansd/whitenoise/blob/main/docs/changelog.rst)
- [Commits](https://github.com/evansd/whitenoise/compare/6.11.0...6.12.0)

---
updated-dependencies:
- dependency-name: whitenoise
  dependency-version: 6.12.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: django-ecosystem
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-02 10:25:11 -08:00
dependabot[bot]
484bef00c1 docker-compose(deps): Bump gotenberg/gotenberg in /docker/compose (#12190)
Bumps gotenberg/gotenberg from 8.26 to 8.27.

---
updated-dependencies:
- dependency-name: gotenberg/gotenberg
  dependency-version: '8.27'
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-03-02 10:14:48 -08:00
Trenton H
317a177537 Chore: Updates s6-overlay to 3.2.2.0 (#12189) 2026-03-02 09:00:03 -08:00
shamoon
7ff51452f0 Documentation: small note re filename vs original_filename 2026-03-01 11:45:06 -08:00
117 changed files with 6306 additions and 2777 deletions

View File

@@ -14,10 +14,6 @@ component_management:
# https://docs.codecov.com/docs/carryforward-flags
flags:
# Backend Python versions
backend-python-3.10:
paths:
- src/**
carryforward: true
backend-python-3.11:
paths:
- src/**
@@ -26,6 +22,14 @@ flags:
paths:
- src/**
carryforward: true
backend-python-3.13:
paths:
- src/**
carryforward: true
backend-python-3.14:
paths:
- src/**
carryforward: true
# Frontend (shards merge into single flag)
frontend-node-24.x:
paths:
@@ -41,9 +45,10 @@ coverage:
project:
backend:
flags:
- backend-python-3.10
- backend-python-3.11
- backend-python-3.12
- backend-python-3.13
- backend-python-3.14
paths:
- src/**
# https://docs.codecov.com/docs/commit-status#threshold
@@ -59,9 +64,10 @@ coverage:
patch:
backend:
flags:
- backend-python-3.10
- backend-python-3.11
- backend-python-3.12
- backend-python-3.13
- backend-python-3.14
paths:
- src/**
target: 100%

View File

@@ -3,21 +3,9 @@ on:
push:
branches-ignore:
- 'translations**'
paths:
- 'src/**'
- 'pyproject.toml'
- 'uv.lock'
- 'docker/compose/docker-compose.ci-test.yml'
- '.github/workflows/ci-backend.yml'
pull_request:
branches-ignore:
- 'translations**'
paths:
- 'src/**'
- 'pyproject.toml'
- 'uv.lock'
- 'docker/compose/docker-compose.ci-test.yml'
- '.github/workflows/ci-backend.yml'
workflow_dispatch:
concurrency:
group: backend-${{ github.event.pull_request.number || github.ref }}
@@ -26,27 +14,55 @@ env:
DEFAULT_UV_VERSION: "0.10.x"
NLTK_DATA: "/usr/share/nltk_data"
jobs:
changes:
name: Detect Backend Changes
runs-on: ubuntu-24.04
outputs:
backend_changed: ${{ steps.manual.outputs.backend || steps.filter.outputs.backend || 'false' }}
steps:
- name: Checkout
uses: actions/checkout@v6.0.2
with:
fetch-depth: 0
- name: Mark backend changes for manual runs
id: manual
if: github.event_name == 'workflow_dispatch'
run: echo "backend=true" >> "$GITHUB_OUTPUT"
- name: Detect backend changes
id: filter
if: github.event_name != 'workflow_dispatch'
uses: dorny/paths-filter@v3.0.2
with:
filters: |
backend:
- 'src/**'
- 'pyproject.toml'
- 'uv.lock'
- 'docker/compose/docker-compose.ci-test.yml'
- '.github/workflows/ci-backend.yml'
test:
needs: changes
if: needs.changes.outputs.backend_changed == 'true'
name: "Python ${{ matrix.python-version }}"
runs-on: ubuntu-24.04
strategy:
matrix:
python-version: ['3.10', '3.11', '3.12']
python-version: ['3.11', '3.12', '3.13', '3.14']
fail-fast: false
steps:
- name: Checkout
uses: actions/checkout@v6
uses: actions/checkout@v6.0.2
- name: Start containers
run: |
docker compose --file docker/compose/docker-compose.ci-test.yml pull --quiet
docker compose --file docker/compose/docker-compose.ci-test.yml up --detach
- name: Set up Python
id: setup-python
uses: actions/setup-python@v6
uses: actions/setup-python@v6.2.0
with:
python-version: "${{ matrix.python-version }}"
- name: Install uv
uses: astral-sh/setup-uv@v7
uses: astral-sh/setup-uv@v7.3.1
with:
version: ${{ env.DEFAULT_UV_VERSION }}
enable-cache: true
@@ -83,13 +99,13 @@ jobs:
pytest
- name: Upload test results to Codecov
if: always()
uses: codecov/codecov-action@v5
uses: codecov/codecov-action@v5.5.2
with:
flags: backend-python-${{ matrix.python-version }}
files: junit.xml
report_type: test_results
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v5
uses: codecov/codecov-action@v5.5.2
with:
flags: backend-python-${{ matrix.python-version }}
files: coverage.xml
@@ -100,20 +116,22 @@ jobs:
docker compose --file docker/compose/docker-compose.ci-test.yml logs
docker compose --file docker/compose/docker-compose.ci-test.yml down
typing:
needs: changes
if: needs.changes.outputs.backend_changed == 'true'
name: Check project typing
runs-on: ubuntu-24.04
env:
DEFAULT_PYTHON: "3.12"
steps:
- name: Checkout
uses: actions/checkout@v6.0.1
uses: actions/checkout@v6.0.2
- name: Set up Python
id: setup-python
uses: actions/setup-python@v6.2.0
with:
python-version: "${{ env.DEFAULT_PYTHON }}"
- name: Install uv
uses: astral-sh/setup-uv@v7.2.1
uses: astral-sh/setup-uv@v7.3.1
with:
version: ${{ env.DEFAULT_UV_VERSION }}
enable-cache: true
@@ -150,3 +168,27 @@ jobs:
--show-error-codes \
--warn-unused-configs \
src/ | uv run mypy-baseline filter
gate:
name: Backend CI Gate
needs: [changes, test, typing]
if: always()
runs-on: ubuntu-24.04
steps:
- name: Validate backend workflow results
run: |
if [[ "${{ needs.changes.outputs.backend_changed }}" != "true" ]]; then
echo "No backend-relevant changes detected."
exit 0
fi
if [[ "${{ needs.test.result }}" != "success" ]]; then
echo "::error::Backend test job result: ${{ needs.test.result }}"
exit 1
fi
if [[ "${{ needs.typing.result }}" != "success" ]]; then
echo "::error::Backend typing job result: ${{ needs.typing.result }}"
exit 1
fi
echo "Backend checks passed."

View File

@@ -41,7 +41,7 @@ jobs:
ref-name: ${{ steps.ref.outputs.name }}
steps:
- name: Checkout
uses: actions/checkout@v6.0.1
uses: actions/checkout@v6.0.2
- name: Determine ref name
id: ref
run: |
@@ -130,7 +130,7 @@ jobs:
type=semver,pattern={{major}}.{{minor}}
- name: Build and push by digest
id: build
uses: docker/build-push-action@v6.18.0
uses: docker/build-push-action@v6.19.2
with:
context: .
file: ./Dockerfile
@@ -152,7 +152,7 @@ jobs:
touch "/tmp/digests/${digest#sha256:}"
- name: Upload digest
if: steps.check-push.outputs.should-push == 'true'
uses: actions/upload-artifact@v6.0.0
uses: actions/upload-artifact@v7.0.0
with:
name: digests-${{ matrix.arch }}
path: /tmp/digests/*
@@ -168,7 +168,7 @@ jobs:
packages: write
steps:
- name: Download digests
uses: actions/download-artifact@v7.0.0
uses: actions/download-artifact@v8.0.0
with:
path: /tmp/digests
pattern: digests-*

View File

@@ -4,19 +4,7 @@ on:
branches:
- main
- dev
paths:
- 'docs/**'
- 'zensical.toml'
- 'pyproject.toml'
- 'uv.lock'
- '.github/workflows/ci-docs.yml'
pull_request:
paths:
- 'docs/**'
- 'zensical.toml'
- 'pyproject.toml'
- 'uv.lock'
- '.github/workflows/ci-docs.yml'
workflow_dispatch:
concurrency:
group: docs-${{ github.event.pull_request.number || github.ref }}
@@ -29,20 +17,48 @@ env:
DEFAULT_UV_VERSION: "0.10.x"
DEFAULT_PYTHON_VERSION: "3.12"
jobs:
changes:
name: Detect Docs Changes
runs-on: ubuntu-24.04
outputs:
docs_changed: ${{ steps.manual.outputs.docs || steps.filter.outputs.docs || 'false' }}
steps:
- name: Checkout
uses: actions/checkout@v6.0.2
with:
fetch-depth: 0
- name: Mark docs changes for manual runs
id: manual
if: github.event_name == 'workflow_dispatch'
run: echo "docs=true" >> "$GITHUB_OUTPUT"
- name: Detect docs changes
id: filter
if: github.event_name != 'workflow_dispatch'
uses: dorny/paths-filter@v3.0.2
with:
filters: |
docs:
- 'docs/**'
- 'zensical.toml'
- 'pyproject.toml'
- 'uv.lock'
- '.github/workflows/ci-docs.yml'
build:
needs: changes
if: needs.changes.outputs.docs_changed == 'true'
name: Build Documentation
runs-on: ubuntu-24.04
steps:
- uses: actions/configure-pages@v5
- uses: actions/configure-pages@v5.0.0
- name: Checkout
uses: actions/checkout@v6
uses: actions/checkout@v6.0.2
- name: Set up Python
id: setup-python
uses: actions/setup-python@v6
uses: actions/setup-python@v6.2.0
with:
python-version: ${{ env.DEFAULT_PYTHON_VERSION }}
- name: Install uv
uses: astral-sh/setup-uv@v7
uses: astral-sh/setup-uv@v7.3.1
with:
version: ${{ env.DEFAULT_UV_VERSION }}
enable-cache: true
@@ -58,21 +74,40 @@ jobs:
--frozen \
zensical build --clean
- name: Upload GitHub Pages artifact
uses: actions/upload-pages-artifact@v4
uses: actions/upload-pages-artifact@v4.0.0
with:
path: site
name: github-pages-${{ github.run_id }}-${{ github.run_attempt }}
deploy:
name: Deploy Documentation
needs: build
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
needs: [changes, build]
if: github.event_name == 'push' && github.ref == 'refs/heads/main' && needs.changes.outputs.docs_changed == 'true'
runs-on: ubuntu-24.04
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
steps:
- name: Deploy GitHub Pages
uses: actions/deploy-pages@v4
uses: actions/deploy-pages@v4.0.5
id: deployment
with:
artifact_name: github-pages-${{ github.run_id }}-${{ github.run_attempt }}
gate:
name: Docs CI Gate
needs: [changes, build]
if: always()
runs-on: ubuntu-24.04
steps:
- name: Validate docs workflow results
run: |
if [[ "${{ needs.changes.outputs.docs_changed }}" != "true" ]]; then
echo "No docs-relevant changes detected."
exit 0
fi
if [[ "${{ needs.build.result }}" != "success" ]]; then
echo "::error::Docs build job result: ${{ needs.build.result }}"
exit 1
fi
echo "Docs checks passed."

View File

@@ -3,39 +3,58 @@ on:
push:
branches-ignore:
- 'translations**'
paths:
- 'src-ui/**'
- '.github/workflows/ci-frontend.yml'
pull_request:
branches-ignore:
- 'translations**'
paths:
- 'src-ui/**'
- '.github/workflows/ci-frontend.yml'
workflow_dispatch:
concurrency:
group: frontend-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
jobs:
changes:
name: Detect Frontend Changes
runs-on: ubuntu-24.04
outputs:
frontend_changed: ${{ steps.manual.outputs.frontend || steps.filter.outputs.frontend || 'false' }}
steps:
- name: Checkout
uses: actions/checkout@v6.0.2
with:
fetch-depth: 0
- name: Mark frontend changes for manual runs
id: manual
if: github.event_name == 'workflow_dispatch'
run: echo "frontend=true" >> "$GITHUB_OUTPUT"
- name: Detect frontend changes
id: filter
if: github.event_name != 'workflow_dispatch'
uses: dorny/paths-filter@v3.0.2
with:
filters: |
frontend:
- 'src-ui/**'
- '.github/workflows/ci-frontend.yml'
install-dependencies:
needs: changes
if: needs.changes.outputs.frontend_changed == 'true'
name: Install Dependencies
runs-on: ubuntu-24.04
steps:
- name: Checkout
uses: actions/checkout@v6
uses: actions/checkout@v6.0.2
- name: Install pnpm
uses: pnpm/action-setup@v4
uses: pnpm/action-setup@v4.2.0
with:
version: 10
- name: Use Node.js 24
uses: actions/setup-node@v6
uses: actions/setup-node@v6.2.0
with:
node-version: 24.x
cache: 'pnpm'
cache-dependency-path: 'src-ui/pnpm-lock.yaml'
- name: Cache frontend dependencies
id: cache-frontend-deps
uses: actions/cache@v5
uses: actions/cache@v5.0.3
with:
path: |
~/.pnpm-store
@@ -45,23 +64,24 @@ jobs:
run: cd src-ui && pnpm install
lint:
name: Lint
needs: install-dependencies
needs: [changes, install-dependencies]
if: needs.changes.outputs.frontend_changed == 'true'
runs-on: ubuntu-24.04
steps:
- name: Checkout
uses: actions/checkout@v6
uses: actions/checkout@v6.0.2
- name: Install pnpm
uses: pnpm/action-setup@v4
uses: pnpm/action-setup@v4.2.0
with:
version: 10
- name: Use Node.js 24
uses: actions/setup-node@v6
uses: actions/setup-node@v6.2.0
with:
node-version: 24.x
cache: 'pnpm'
cache-dependency-path: 'src-ui/pnpm-lock.yaml'
- name: Cache frontend dependencies
uses: actions/cache@v5
uses: actions/cache@v5.0.3
with:
path: |
~/.pnpm-store
@@ -73,7 +93,8 @@ jobs:
run: cd src-ui && pnpm run lint
unit-tests:
name: "Unit Tests (${{ matrix.shard-index }}/${{ matrix.shard-count }})"
needs: install-dependencies
needs: [changes, install-dependencies]
if: needs.changes.outputs.frontend_changed == 'true'
runs-on: ubuntu-24.04
strategy:
fail-fast: false
@@ -83,19 +104,19 @@ jobs:
shard-count: [4]
steps:
- name: Checkout
uses: actions/checkout@v6
uses: actions/checkout@v6.0.2
- name: Install pnpm
uses: pnpm/action-setup@v4
uses: pnpm/action-setup@v4.2.0
with:
version: 10
- name: Use Node.js 24
uses: actions/setup-node@v6
uses: actions/setup-node@v6.2.0
with:
node-version: 24.x
cache: 'pnpm'
cache-dependency-path: 'src-ui/pnpm-lock.yaml'
- name: Cache frontend dependencies
uses: actions/cache@v5
uses: actions/cache@v5.0.3
with:
path: |
~/.pnpm-store
@@ -107,19 +128,20 @@ jobs:
run: cd src-ui && pnpm run test --max-workers=2 --shard=${{ matrix.shard-index }}/${{ matrix.shard-count }}
- name: Upload test results to Codecov
if: always()
uses: codecov/codecov-action@v5
uses: codecov/codecov-action@v5.5.2
with:
flags: frontend-node-${{ matrix.node-version }}
directory: src-ui/
report_type: test_results
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v5
uses: codecov/codecov-action@v5.5.2
with:
flags: frontend-node-${{ matrix.node-version }}
directory: src-ui/coverage/
e2e-tests:
name: "E2E Tests (${{ matrix.shard-index }}/${{ matrix.shard-count }})"
needs: install-dependencies
needs: [changes, install-dependencies]
if: needs.changes.outputs.frontend_changed == 'true'
runs-on: ubuntu-24.04
container: mcr.microsoft.com/playwright:v1.58.2-noble
env:
@@ -133,19 +155,19 @@ jobs:
shard-count: [2]
steps:
- name: Checkout
uses: actions/checkout@v6
uses: actions/checkout@v6.0.2
- name: Install pnpm
uses: pnpm/action-setup@v4
uses: pnpm/action-setup@v4.2.0
with:
version: 10
- name: Use Node.js 24
uses: actions/setup-node@v6
uses: actions/setup-node@v6.2.0
with:
node-version: 24.x
cache: 'pnpm'
cache-dependency-path: 'src-ui/pnpm-lock.yaml'
- name: Cache frontend dependencies
uses: actions/cache@v5
uses: actions/cache@v5.0.3
with:
path: |
~/.pnpm-store
@@ -159,23 +181,24 @@ jobs:
run: cd src-ui && pnpm exec playwright test --shard ${{ matrix.shard-index }}/${{ matrix.shard-count }}
bundle-analysis:
name: Bundle Analysis
needs: [unit-tests, e2e-tests]
needs: [changes, unit-tests, e2e-tests]
if: needs.changes.outputs.frontend_changed == 'true'
runs-on: ubuntu-24.04
steps:
- name: Checkout
uses: actions/checkout@v6
uses: actions/checkout@v6.0.2
- name: Install pnpm
uses: pnpm/action-setup@v4
uses: pnpm/action-setup@v4.2.0
with:
version: 10
- name: Use Node.js 24
uses: actions/setup-node@v6
uses: actions/setup-node@v6.2.0
with:
node-version: 24.x
cache: 'pnpm'
cache-dependency-path: 'src-ui/pnpm-lock.yaml'
- name: Cache frontend dependencies
uses: actions/cache@v5
uses: actions/cache@v5.0.3
with:
path: |
~/.pnpm-store
@@ -187,3 +210,42 @@ jobs:
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
run: cd src-ui && pnpm run build --configuration=production
gate:
name: Frontend CI Gate
needs: [changes, install-dependencies, lint, unit-tests, e2e-tests, bundle-analysis]
if: always()
runs-on: ubuntu-24.04
steps:
- name: Validate frontend workflow results
run: |
if [[ "${{ needs.changes.outputs.frontend_changed }}" != "true" ]]; then
echo "No frontend-relevant changes detected."
exit 0
fi
if [[ "${{ needs['install-dependencies'].result }}" != "success" ]]; then
echo "::error::Frontend install job result: ${{ needs['install-dependencies'].result }}"
exit 1
fi
if [[ "${{ needs.lint.result }}" != "success" ]]; then
echo "::error::Frontend lint job result: ${{ needs.lint.result }}"
exit 1
fi
if [[ "${{ needs['unit-tests'].result }}" != "success" ]]; then
echo "::error::Frontend unit-tests job result: ${{ needs['unit-tests'].result }}"
exit 1
fi
if [[ "${{ needs['e2e-tests'].result }}" != "success" ]]; then
echo "::error::Frontend e2e-tests job result: ${{ needs['e2e-tests'].result }}"
exit 1
fi
if [[ "${{ needs['bundle-analysis'].result }}" != "success" ]]; then
echo "::error::Frontend bundle-analysis job result: ${{ needs['bundle-analysis'].result }}"
exit 1
fi
echo "Frontend checks passed."

View File

@@ -28,14 +28,14 @@ jobs:
runs-on: ubuntu-24.04
steps:
- name: Checkout
uses: actions/checkout@v6
uses: actions/checkout@v6.0.2
# ---- Frontend Build ----
- name: Install pnpm
uses: pnpm/action-setup@v4
uses: pnpm/action-setup@v4.2.0
with:
version: 10
- name: Use Node.js 24
uses: actions/setup-node@v6
uses: actions/setup-node@v6.2.0
with:
node-version: 24.x
cache: 'pnpm'
@@ -47,11 +47,11 @@ jobs:
# ---- Backend Setup ----
- name: Set up Python
id: setup-python
uses: actions/setup-python@v6
uses: actions/setup-python@v6.2.0
with:
python-version: ${{ env.DEFAULT_PYTHON_VERSION }}
- name: Install uv
uses: astral-sh/setup-uv@v7
uses: astral-sh/setup-uv@v7.3.1
with:
version: ${{ env.DEFAULT_UV_VERSION }}
enable-cache: true
@@ -118,7 +118,7 @@ jobs:
sudo chown -R 1000:1000 paperless-ngx/
tar -cJf paperless-ngx.tar.xz paperless-ngx/
- name: Upload release artifact
uses: actions/upload-artifact@v6
uses: actions/upload-artifact@v7.0.0
with:
name: release
path: dist/paperless-ngx.tar.xz
@@ -133,7 +133,7 @@ jobs:
version: ${{ steps.get-version.outputs.version }}
steps:
- name: Download release artifact
uses: actions/download-artifact@v7
uses: actions/download-artifact@v8.0.0
with:
name: release
path: ./
@@ -148,7 +148,7 @@ jobs:
fi
- name: Create release and changelog
id: create-release
uses: release-drafter/release-drafter@v6
uses: release-drafter/release-drafter@v6.2.0
with:
name: Paperless-ngx ${{ steps.get-version.outputs.version }}
tag: ${{ steps.get-version.outputs.version }}
@@ -159,7 +159,7 @@ jobs:
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Upload release archive
uses: shogo82148/actions-upload-release-asset@v1
uses: shogo82148/actions-upload-release-asset@v1.9.2
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
upload_url: ${{ steps.create-release.outputs.upload_url }}
@@ -176,16 +176,16 @@ jobs:
runs-on: ubuntu-24.04
steps:
- name: Checkout
uses: actions/checkout@v6
uses: actions/checkout@v6.0.2
with:
ref: main
- name: Set up Python
id: setup-python
uses: actions/setup-python@v6
uses: actions/setup-python@v6.2.0
with:
python-version: ${{ env.DEFAULT_PYTHON_VERSION }}
- name: Install uv
uses: astral-sh/setup-uv@v7
uses: astral-sh/setup-uv@v7.3.1
with:
version: ${{ env.DEFAULT_UV_VERSION }}
enable-cache: true
@@ -218,7 +218,7 @@ jobs:
git commit -am "Changelog ${{ needs.publish-release.outputs.version }} - GHA"
git push origin ${{ needs.publish-release.outputs.version }}-changelog
- name: Create pull request
uses: actions/github-script@v8
uses: actions/github-script@v8.0.0
with:
script: |
const { repo, owner } = context.repo;

View File

@@ -34,10 +34,10 @@ jobs:
# Learn more about CodeQL language support at https://git.io/codeql-language-support
steps:
- name: Checkout repository
uses: actions/checkout@v6
uses: actions/checkout@v6.0.2
# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@v4
uses: github/codeql-action/init@v4.32.5
with:
languages: ${{ matrix.language }}
# If you wish to specify custom queries, you can do so here or in a config file.
@@ -45,4 +45,4 @@ jobs:
# Prefix the list here with "+" to use these queries and those in the config file.
# queries: ./path/to/local/query, your-org/your-repo/queries@main
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v4
uses: github/codeql-action/analyze@v4.32.5

View File

@@ -13,11 +13,11 @@ jobs:
runs-on: ubuntu-24.04
steps:
- name: Checkout
uses: actions/checkout@v6
uses: actions/checkout@v6.0.2
with:
token: ${{ secrets.PNGX_BOT_PAT }}
- name: crowdin action
uses: crowdin/github-action@v2
uses: crowdin/github-action@v2.15.0
with:
upload_translations: false
download_translations: true

View File

@@ -12,7 +12,7 @@ jobs:
steps:
- name: Label PR by file path or branch name
# see .github/labeler.yml for the labeler config
uses: actions/labeler@v6
uses: actions/labeler@v6.0.1
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
- name: Label by size
@@ -26,7 +26,7 @@ jobs:
fail_if_xl: 'false'
excluded_files: /\.lock$/ /\.txt$/ ^src-ui/pnpm-lock\.yaml$ ^src-ui/messages\.xlf$ ^src/locale/en_US/LC_MESSAGES/django\.po$
- name: Label by PR title
uses: actions/github-script@v8
uses: actions/github-script@v8.0.0
with:
script: |
const pr = context.payload.pull_request;
@@ -52,7 +52,7 @@ jobs:
}
- name: Label bot-generated PRs
if: ${{ contains(github.actor, 'dependabot') || contains(github.actor, 'crowdin-bot') }}
uses: actions/github-script@v8
uses: actions/github-script@v8.0.0
with:
script: |
const pr = context.payload.pull_request;
@@ -77,7 +77,7 @@ jobs:
}
- name: Welcome comment
if: ${{ !contains(github.actor, 'bot') }}
uses: actions/github-script@v8
uses: actions/github-script@v8.0.0
with:
script: |
const pr = context.payload.pull_request;

View File

@@ -19,6 +19,6 @@ jobs:
if: github.event_name == 'pull_request_target' && (github.event.action == 'opened' || github.event.action == 'reopened') && github.event.pull_request.user.login != 'dependabot'
steps:
- name: Label PR with release-drafter
uses: release-drafter/release-drafter@v6
uses: release-drafter/release-drafter@v6.2.0
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

View File

@@ -15,7 +15,7 @@ jobs:
if: github.repository_owner == 'paperless-ngx'
runs-on: ubuntu-24.04
steps:
- uses: actions/stale@v10
- uses: actions/stale@v10.2.0
with:
days-before-stale: 7
days-before-close: 14
@@ -37,7 +37,7 @@ jobs:
if: github.repository_owner == 'paperless-ngx'
runs-on: ubuntu-24.04
steps:
- uses: dessant/lock-threads@v6
- uses: dessant/lock-threads@v6.0.0
with:
issue-inactive-days: '30'
pr-inactive-days: '30'
@@ -57,7 +57,7 @@ jobs:
if: github.repository_owner == 'paperless-ngx'
runs-on: ubuntu-24.04
steps:
- uses: actions/github-script@v8
- uses: actions/github-script@v8.0.0
with:
script: |
function sleep(ms) {
@@ -114,7 +114,7 @@ jobs:
if: github.repository_owner == 'paperless-ngx'
runs-on: ubuntu-24.04
steps:
- uses: actions/github-script@v8
- uses: actions/github-script@v8.0.0
with:
script: |
function sleep(ms) {
@@ -206,7 +206,7 @@ jobs:
if: github.repository_owner == 'paperless-ngx'
runs-on: ubuntu-24.04
steps:
- uses: actions/github-script@v8
- uses: actions/github-script@v8.0.0
with:
script: |
function sleep(ms) {

View File

@@ -11,7 +11,7 @@ jobs:
contents: write
steps:
- name: Checkout code
uses: actions/checkout@v6
uses: actions/checkout@v6.0.2
env:
GH_REF: ${{ github.ref }} # sonar rule:githubactions:S7630 - avoid injection
with:
@@ -19,13 +19,13 @@ jobs:
ref: ${{ env.GH_REF }}
- name: Set up Python
id: setup-python
uses: actions/setup-python@v6
uses: actions/setup-python@v6.2.0
- name: Install system dependencies
run: |
sudo apt-get update -qq
sudo apt-get install -qq --no-install-recommends gettext
- name: Install uv
uses: astral-sh/setup-uv@v7
uses: astral-sh/setup-uv@v7.3.1
with:
enable-cache: true
- name: Install backend python dependencies
@@ -36,18 +36,18 @@ jobs:
- name: Generate backend translation strings
run: cd src/ && uv run manage.py makemessages -l en_US -i "samples*"
- name: Install pnpm
uses: pnpm/action-setup@v4
uses: pnpm/action-setup@v4.2.0
with:
version: 10
- name: Use Node.js 24
uses: actions/setup-node@v6
uses: actions/setup-node@v6.2.0
with:
node-version: 24.x
cache: 'pnpm'
cache-dependency-path: 'src-ui/pnpm-lock.yaml'
- name: Cache frontend dependencies
id: cache-frontend-deps
uses: actions/cache@v5
uses: actions/cache@v5.0.3
with:
path: |
~/.pnpm-store
@@ -63,7 +63,7 @@ jobs:
cd src-ui
pnpm run ng extract-i18n
- name: Commit changes
uses: stefanzweifel/git-auto-commit-action@v7
uses: stefanzweifel/git-auto-commit-action@v7.1.0
with:
file_pattern: 'src-ui/messages.xlf src/locale/en_US/LC_MESSAGES/django.po'
commit_message: "Auto translate strings"

View File

@@ -341,6 +341,9 @@ src/documents/migrations/0001_initial.py:0: error: Skipping analyzing "multisele
src/documents/migrations/0001_initial.py:0: error: Skipping analyzing "multiselectfield.db.fields": module is installed, but missing library stubs or py.typed marker [import-untyped]
src/documents/migrations/0008_sharelinkbundle.py:0: error: Function is missing a type annotation [no-untyped-def]
src/documents/migrations/0008_sharelinkbundle.py:0: error: Function is missing a type annotation [no-untyped-def]
src/documents/migrations/0012_savedview_visibility_to_ui_settings.py:0: error: Function is missing a type annotation [no-untyped-def]
src/documents/migrations/0012_savedview_visibility_to_ui_settings.py:0: error: Function is missing a type annotation [no-untyped-def]
src/documents/migrations/0012_savedview_visibility_to_ui_settings.py:0: error: Function is missing a type annotation for one or more arguments [no-untyped-def]
src/documents/models.py:0: error: Argument 1 to "Path" has incompatible type "Path | None"; expected "str | PathLike[str]" [arg-type]
src/documents/models.py:0: error: Couldn't resolve related manager 'documents' for relation 'documents.models.Document.correspondent'. [django-manager-missing]
src/documents/models.py:0: error: Couldn't resolve related manager 'documents' for relation 'documents.models.Document.document_type'. [django-manager-missing]
@@ -440,9 +443,6 @@ src/documents/permissions.py:0: error: Item "list[str]" of "Any | list[str] | Qu
src/documents/permissions.py:0: error: Item "list[str]" of "Any | list[str] | QuerySet[User, User]" has no attribute "exists" [union-attr]
src/documents/permissions.py:0: error: Missing type parameters for generic type "QuerySet" [type-arg]
src/documents/permissions.py:0: error: Missing type parameters for generic type "dict" [type-arg]
src/documents/plugins/helpers.py:0: error: "Collection[str]" has no attribute "update" [attr-defined]
src/documents/plugins/helpers.py:0: error: Argument 1 to "send" of "BaseStatusManager" has incompatible type "dict[str, Collection[str]]"; expected "dict[str, str | int | None]" [arg-type]
src/documents/plugins/helpers.py:0: error: Argument 1 to "send" of "BaseStatusManager" has incompatible type "dict[str, Collection[str]]"; expected "dict[str, str | int | None]" [arg-type]
src/documents/plugins/helpers.py:0: error: Function is missing a type annotation [no-untyped-def]
src/documents/plugins/helpers.py:0: error: Function is missing a type annotation [no-untyped-def]
src/documents/plugins/helpers.py:0: error: Skipping analyzing "channels_redis.pubsub": module is installed, but missing library stubs or py.typed marker [import-untyped]
@@ -550,6 +550,7 @@ src/documents/serialisers.py:0: error: Function is missing a type annotation [n
src/documents/serialisers.py:0: error: Function is missing a type annotation [no-untyped-def]
src/documents/serialisers.py:0: error: Function is missing a type annotation [no-untyped-def]
src/documents/serialisers.py:0: error: Function is missing a type annotation [no-untyped-def]
src/documents/serialisers.py:0: error: Function is missing a type annotation [no-untyped-def]
src/documents/serialisers.py:0: error: Function is missing a type annotation for one or more arguments [no-untyped-def]
src/documents/serialisers.py:0: error: Function is missing a type annotation for one or more arguments [no-untyped-def]
src/documents/serialisers.py:0: error: Function is missing a type annotation for one or more arguments [no-untyped-def]
@@ -640,6 +641,7 @@ src/documents/serialisers.py:0: error: Missing type parameters for generic type
src/documents/serialisers.py:0: error: Missing type parameters for generic type "Serializer" [type-arg]
src/documents/serialisers.py:0: error: Missing type parameters for generic type "Serializer" [type-arg]
src/documents/serialisers.py:0: error: Missing type parameters for generic type "Serializer" [type-arg]
src/documents/serialisers.py:0: error: Missing type parameters for generic type "Serializer" [type-arg]
src/documents/serialisers.py:0: error: Missing type parameters for generic type "dict" [type-arg]
src/documents/serialisers.py:0: error: Missing type parameters for generic type "dict" [type-arg]
src/documents/serialisers.py:0: error: Need type annotation for "document" [var-annotated]
@@ -667,7 +669,6 @@ src/documents/signals/handlers.py:0: error: Argument 3 to "validate_move" has in
src/documents/signals/handlers.py:0: error: Argument 5 to "_suggestion_printer" has incompatible type "Any | None"; expected "MatchingModel" [arg-type]
src/documents/signals/handlers.py:0: error: Argument 5 to "_suggestion_printer" has incompatible type "Any | None"; expected "MatchingModel" [arg-type]
src/documents/signals/handlers.py:0: error: Argument 5 to "_suggestion_printer" has incompatible type "Any | None"; expected "MatchingModel" [arg-type]
src/documents/signals/handlers.py:0: error: Function is missing a return type annotation [no-untyped-def]
src/documents/signals/handlers.py:0: error: Function is missing a type annotation [no-untyped-def]
src/documents/signals/handlers.py:0: error: Function is missing a type annotation [no-untyped-def]
src/documents/signals/handlers.py:0: error: Function is missing a type annotation for one or more arguments [no-untyped-def]
@@ -1175,6 +1176,14 @@ src/documents/tests/test_management_exporter.py:0: error: Skipping analyzing "al
src/documents/tests/test_management_fuzzy.py:0: error: Function is missing a type annotation [no-untyped-def]
src/documents/tests/test_management_retagger.py:0: error: Function is missing a return type annotation [no-untyped-def]
src/documents/tests/test_management_superuser.py:0: error: Function is missing a type annotation [no-untyped-def]
src/documents/tests/test_migration_saved_view_visibility.py:0: error: Cannot determine type of "apps" [has-type]
src/documents/tests/test_migration_saved_view_visibility.py:0: error: Cannot determine type of "apps" [has-type]
src/documents/tests/test_migration_saved_view_visibility.py:0: error: Function is missing a type annotation for one or more arguments [no-untyped-def]
src/documents/tests/test_migration_saved_view_visibility.py:0: error: Function is missing a type annotation for one or more arguments [no-untyped-def]
src/documents/tests/test_migration_saved_view_visibility.py:0: error: Incompatible types in assignment (expression has type "str", base class "TestMigrations" defined the type as "None") [assignment]
src/documents/tests/test_migration_saved_view_visibility.py:0: error: Incompatible types in assignment (expression has type "str", base class "TestMigrations" defined the type as "None") [assignment]
src/documents/tests/test_migration_saved_view_visibility.py:0: error: Incompatible types in assignment (expression has type "str", base class "TestMigrations" defined the type as "None") [assignment]
src/documents/tests/test_migration_saved_view_visibility.py:0: error: Incompatible types in assignment (expression has type "str", base class "TestMigrations" defined the type as "None") [assignment]
src/documents/tests/test_migration_share_link_bundle.py:0: error: Function is missing a type annotation for one or more arguments [no-untyped-def]
src/documents/tests/test_migration_share_link_bundle.py:0: error: Function is missing a type annotation for one or more arguments [no-untyped-def]
src/documents/tests/test_migration_share_link_bundle.py:0: error: Incompatible types in assignment (expression has type "str", base class "TestMigrations" defined the type as "None") [assignment]
@@ -1534,7 +1543,6 @@ src/documents/views.py:0: error: "get_serializer_context" undefined in superclas
src/documents/views.py:0: error: "object" not callable [operator]
src/documents/views.py:0: error: "type[Model]" has no attribute "objects" [attr-defined]
src/documents/views.py:0: error: Argument "path" to "EmailAttachment" has incompatible type "Path | None"; expected "Path" [arg-type]
src/documents/views.py:0: error: Argument 1 to "int" has incompatible type "str | None"; expected "str | Buffer | SupportsInt | SupportsIndex | SupportsTrunc" [arg-type]
src/documents/views.py:0: error: Argument 2 to "match_correspondents" has incompatible type "DocumentClassifier | None"; expected "DocumentClassifier" [arg-type]
src/documents/views.py:0: error: Argument 2 to "match_document_types" has incompatible type "DocumentClassifier | None"; expected "DocumentClassifier" [arg-type]
src/documents/views.py:0: error: Argument 2 to "match_storage_paths" has incompatible type "DocumentClassifier | None"; expected "DocumentClassifier" [arg-type]
@@ -1552,7 +1560,6 @@ src/documents/views.py:0: error: Function is missing a return type annotation [
src/documents/views.py:0: error: Function is missing a return type annotation [no-untyped-def]
src/documents/views.py:0: error: Function is missing a return type annotation [no-untyped-def]
src/documents/views.py:0: error: Function is missing a return type annotation [no-untyped-def]
src/documents/views.py:0: error: Function is missing a return type annotation [no-untyped-def]
src/documents/views.py:0: error: Function is missing a type annotation [no-untyped-def]
src/documents/views.py:0: error: Function is missing a type annotation [no-untyped-def]
src/documents/views.py:0: error: Function is missing a type annotation [no-untyped-def]
@@ -1609,7 +1616,8 @@ src/documents/views.py:0: error: Function is missing a type annotation [no-unty
src/documents/views.py:0: error: Function is missing a type annotation [no-untyped-def]
src/documents/views.py:0: error: Function is missing a type annotation for one or more arguments [no-untyped-def]
src/documents/views.py:0: error: Function is missing a type annotation for one or more arguments [no-untyped-def]
src/documents/views.py:0: error: Incompatible type for lookup 'owner': (got "User | AnonymousUser", expected "User | int | None") [misc]
src/documents/views.py:0: error: Function is missing a type annotation for one or more arguments [no-untyped-def]
src/documents/views.py:0: error: Function is missing a type annotation for one or more arguments [no-untyped-def]
src/documents/views.py:0: error: Incompatible types in assignment (expression has type "Any | None", variable has type "dict[Any, Any]") [assignment]
src/documents/views.py:0: error: Incompatible types in assignment (expression has type "QuerySet[Any, Any]", variable has type "list[Any]") [assignment]
src/documents/views.py:0: error: Incompatible types in assignment (expression has type "QuerySet[Any, Any]", variable has type "list[Document]") [assignment]
@@ -1675,11 +1683,11 @@ src/documents/views.py:0: error: Value of type "Iterable[Any]" is not indexable
src/documents/views.py:0: error: Value of type "Iterable[Any]" is not indexable [index]
src/documents/views.py:0: error: Value of type "Iterable[Any]" is not indexable [index]
src/documents/views.py:0: error: Value of type "Iterable[Any]" is not indexable [index]
src/documents/views.py:0: error: Value of type "Iterable[Any]" is not indexable [index]
src/documents/views.py:0: error: Value of type "Iterable[CustomField]" is not indexable [index]
src/documents/views.py:0: error: Value of type "Iterable[Group]" is not indexable [index]
src/documents/views.py:0: error: Value of type "Iterable[MailAccount]" is not indexable [index]
src/documents/views.py:0: error: Value of type "Iterable[MailRule]" is not indexable [index]
src/documents/views.py:0: error: Value of type "Iterable[SavedView]" is not indexable [index]
src/documents/views.py:0: error: Value of type "Iterable[User]" is not indexable [index]
src/documents/views.py:0: error: Value of type "Iterable[Workflow]" is not indexable [index]
src/documents/views.py:0: error: Value of type "dict[str, _PingReply] | None" is not indexable [index]
@@ -1928,6 +1936,7 @@ src/paperless/tests/test_websockets.py:0: error: Item "None" of "BaseChannelLaye
src/paperless/tests/test_websockets.py:0: error: Item "None" of "BaseChannelLayer | None" has no attribute "group_send" [union-attr]
src/paperless/tests/test_websockets.py:0: error: Item "None" of "BaseChannelLayer | None" has no attribute "group_send" [union-attr]
src/paperless/tests/test_websockets.py:0: error: Item "None" of "BaseChannelLayer | None" has no attribute "group_send" [union-attr]
src/paperless/tests/test_websockets.py:0: error: Item "None" of "BaseChannelLayer | None" has no attribute "group_send" [union-attr]
src/paperless/tests/test_websockets.py:0: error: TypedDict "_WebsocketTestScope" has no key "user" [typeddict-item]
src/paperless/tests/test_websockets.py:0: error: TypedDict "_WebsocketTestScope" has no key "user" [typeddict-item]
src/paperless/tests/test_websockets.py:0: error: TypedDict "_WebsocketTestScope" has no key "user" [typeddict-item]

View File

@@ -13,7 +13,9 @@ If you want to implement something big:
## Python
Paperless supports python 3.10 - 3.12 at this time. We format Python code with [ruff](https://docs.astral.sh/ruff/formatter/).
Paperless-ngx currently supports Python 3.11, 3.12, 3.13, and 3.14. As a policy, we aim to support at least the three most recent Python versions, and drop support for versions as they reach end-of-life. Older versions may be supported if dependencies permit, but this is not guaranteed.
We format Python code with [ruff](https://docs.astral.sh/ruff/formatter/).
## Branches

View File

@@ -30,7 +30,7 @@ RUN set -eux \
# Purpose: Installs s6-overlay and rootfs
# Comments:
# - Don't leave anything extra in here either
FROM ghcr.io/astral-sh/uv:0.10.5-python3.12-trixie-slim AS s6-overlay-base
FROM ghcr.io/astral-sh/uv:0.10.7-python3.12-trixie-slim AS s6-overlay-base
WORKDIR /usr/src/s6
@@ -45,7 +45,7 @@ ENV \
ARG TARGETARCH
ARG TARGETVARIANT
# Lock this version
ARG S6_OVERLAY_VERSION=3.2.1.0
ARG S6_OVERLAY_VERSION=3.2.2.0
ARG S6_BUILD_TIME_PKGS="curl \
xz-utils"

View File

@@ -4,7 +4,7 @@
# correct networking for the tests
services:
gotenberg:
image: docker.io/gotenberg/gotenberg:8.26
image: docker.io/gotenberg/gotenberg:8.27
hostname: gotenberg
container_name: gotenberg
network_mode: host

View File

@@ -72,7 +72,7 @@ services:
PAPERLESS_TIKA_GOTENBERG_ENDPOINT: http://gotenberg:3000
PAPERLESS_TIKA_ENDPOINT: http://tika:9998
gotenberg:
image: docker.io/gotenberg/gotenberg:8.26
image: docker.io/gotenberg/gotenberg:8.27
restart: unless-stopped
# The gotenberg chromium route is used to convert .eml files. We do not
# want to allow external content like tracking pixels or even javascript.

View File

@@ -66,7 +66,7 @@ services:
PAPERLESS_TIKA_GOTENBERG_ENDPOINT: http://gotenberg:3000
PAPERLESS_TIKA_ENDPOINT: http://tika:9998
gotenberg:
image: docker.io/gotenberg/gotenberg:8.26
image: docker.io/gotenberg/gotenberg:8.27
restart: unless-stopped
# The gotenberg chromium route is used to convert .eml files. We do not
# want to allow external content like tracking pixels or even javascript.

View File

@@ -55,7 +55,7 @@ services:
PAPERLESS_TIKA_GOTENBERG_ENDPOINT: http://gotenberg:3000
PAPERLESS_TIKA_ENDPOINT: http://tika:9998
gotenberg:
image: docker.io/gotenberg/gotenberg:8.26
image: docker.io/gotenberg/gotenberg:8.27
restart: unless-stopped
# The gotenberg chromium route is used to convert .eml files. We do not
# want to allow external content like tracking pixels or even javascript.

View File

@@ -262,6 +262,10 @@ your files differently, you can do that by adjusting the
or using [storage paths (see below)](#storage-paths). Paperless adds the
correct file extension e.g. `.pdf`, `.jpg` automatically.
When a document has file versions, each version uses the same naming rules and
storage path resolution as any other document file, with an added version suffix
such as `_v1`, `_v2`, etc.
This variable allows you to configure the filename (folders are allowed)
using placeholders. For example, configuring this to
@@ -353,6 +357,8 @@ If paperless detects that two documents share the same filename,
paperless will automatically append `_01`, `_02`, etc to the filename.
This happens if all the placeholders in a filename evaluate to the same
value.
For versioned files, this counter is appended after the version suffix
(for example `statement_v2_01.pdf`).
If there are any errors in the placeholders included in `PAPERLESS_FILENAME_FORMAT`,
paperless will fall back to using the default naming scheme instead.

View File

@@ -466,3 +466,8 @@ Initial API version.
- The document `created` field is now a date, not a datetime. The
`created_date` field is considered deprecated and will be removed in a
future version.
#### Version 10
- The `show_on_dashboard` and `show_in_sidebar` fields of saved views have been
removed. Relevant settings are now stored in the UISettings model.

Binary file not shown.

After

Width:  |  Height:  |  Size: 57 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 61 KiB

View File

@@ -1,5 +1,23 @@
# Changelog
## paperless-ngx 2.20.10
### Bug Fixes
- Fix: support string coercion in filepath jinja templates [@shamoon](https://github.com/shamoon) ([#12244](https://github.com/paperless-ngx/paperless-ngx/pull/12244))
- Fix: apply ordering after annotating tag document count [@shamoon](https://github.com/shamoon) ([#12238](https://github.com/paperless-ngx/paperless-ngx/pull/12238))
- Fix: enforce path limit for db filename fields [@shamoon](https://github.com/shamoon) ([#12235](https://github.com/paperless-ngx/paperless-ngx/pull/12235))
### All App Changes
<details>
<summary>3 changes</summary>
- Fix: support string coercion in filepath jinja templates [@shamoon](https://github.com/shamoon) ([#12244](https://github.com/paperless-ngx/paperless-ngx/pull/12244))
- Fix: apply ordering after annotating tag document count [@shamoon](https://github.com/shamoon) ([#12238](https://github.com/paperless-ngx/paperless-ngx/pull/12238))
- Fix: enforce path limit for db filename fields [@shamoon](https://github.com/shamoon) ([#12235](https://github.com/paperless-ngx/paperless-ngx/pull/12235))
</details>
## paperless-ngx 2.20.9
### Security

View File

@@ -172,7 +172,7 @@ to enable polling and disable inotify. See [here](configuration.md#polling).
#### Prerequisites
- Paperless runs on Linux only, Windows is not supported.
- Python 3 is required with versions 3.10 - 3.12 currently supported. Newer versions may work, but some dependencies may not be fully compatible.
- Python 3.11, 3.12, 3.13, or 3.14 is required. As a policy, Paperless-ngx aims to support at least the three most recent Python versions and drops support for versions as they reach end-of-life. Newer versions may work, but some dependencies may not be fully compatible.
#### Installation

View File

@@ -95,6 +95,7 @@ Think of versions as **file history** for a document.
- Versions track the underlying file and extracted text content (OCR/text).
- Metadata such as tags, correspondent, document type, storage path and custom fields stay on the "root" document.
- Version files follow normal filename formatting (including storage paths) and add a `_vN` suffix (for example `_v1`, `_v2`).
- By default, search and document content use the latest version.
- In document detail, selecting a version switches the preview, file metadata and content (and download etc buttons) to that version.
- Deleting a non-root version keeps metadata and falls back to the latest remaining version.
@@ -616,7 +617,7 @@ applied. You can use the following placeholders in the template with any trigger
- `{{added_day}}`: added day
- `{{added_time}}`: added time in HH:MM format
- `{{original_filename}}`: original file name without extension
- `{{filename}}`: current file name without extension
- `{{filename}}`: current file name without extension (for "added" workflows this may not be final yet, you can use `{{original_filename}}`)
- `{{doc_title}}`: current document title (cannot be used in title assignment)
The following placeholders are only available for "added" or "updated" triggers

View File

@@ -1,12 +1,11 @@
[project]
name = "paperless-ngx"
version = "2.20.9"
version = "2.20.10"
description = "A community-supported supercharged document management system: scan, index and archive all your physical documents"
readme = "README.md"
requires-python = ">=3.10"
requires-python = ">=3.11"
classifiers = [
"Programming Language :: Python :: 3 :: Only",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
@@ -111,6 +110,7 @@ docs = [
testing = [
"daphne",
"factory-boy~=3.3.1",
"faker~=40.5.1",
"imagehash",
"pytest~=9.0.0",
"pytest-cov~=7.0.0",
@@ -176,7 +176,7 @@ torch = [
]
[tool.ruff]
target-version = "py310"
target-version = "py311"
line-length = 88
src = [
"src",
@@ -309,6 +309,7 @@ markers = [
[tool.pytest_env]
PAPERLESS_DISABLE_DBHANDLER = "true"
PAPERLESS_CACHE_BACKEND = "django.core.cache.backends.locmem.LocMemCache"
PAPERLESS_CHANNELS_BACKEND = "channels.layers.InMemoryChannelLayer"
[tool.coverage.report]
exclude_also = [

View File

@@ -58,7 +58,7 @@
"content": {
"size": -1,
"mimeType": "application/json",
"text": "{\"user\":{\"id\":2,\"username\":\"testuser\",\"is_superuser\":false,\"groups\":[]},\"settings\":{\"language\":\"\",\"bulk_edit\":{\"confirmation_dialogs\":true,\"apply_on_close\":false},\"documentListSize\":50,\"dark_mode\":{\"use_system\":false,\"enabled\":\"false\",\"thumb_inverted\":\"true\"},\"theme\":{\"color\":\"#9fbf2f\"},\"document_details\":{\"native_pdf_viewer\":false},\"date_display\":{\"date_locale\":\"\",\"date_format\":\"mediumDate\"},\"notifications\":{\"consumer_new_documents\":true,\"consumer_success\":true,\"consumer_failed\":true,\"consumer_suppress_on_dashboard\":true},\"comments_enabled\":true,\"slim_sidebar\":false,\"update_checking\":{\"enabled\":false,\"backend_setting\":\"default\"},\"saved_views\":{\"warn_on_unsaved_change\":true},\"notes_enabled\":true,\"tour_complete\":true},\"permissions\":[\"delete_schedule\",\"view_note\",\"view_taskattributes\",\"delete_ormq\",\"add_ormq\",\"change_tag\",\"change_chordcounter\",\"delete_savedviewfilterrule\",\"delete_comment\",\"add_session\",\"add_mailrule\",\"add_tokenproxy\",\"delete_taskresult\",\"view_failure\",\"add_tag\",\"view_savedviewfilterrule\",\"view_paperlesstask\",\"change_mailaccount\",\"change_frontendsettings\",\"delete_group\",\"add_userobjectpermission\",\"add_failure\",\"delete_mailrule\",\"view_userobjectpermission\",\"change_schedule\",\"delete_note\",\"view_ormq\",\"add_note\",\"add_chordcounter\",\"delete_token\",\"change_failure\",\"add_savedviewfilterrule\",\"delete_user\",\"view_correspondent\",\"view_schedule\",\"change_tokenproxy\",\"view_user\",\"delete_task\",\"delete_correspondent\",\"delete_chordcounter\",\"delete_document\",\"view_taskresult\",\"change_document\",\"add_frontendsettings\",\"view_mailrule\",\"change_ormq\",\"delete_taskattributes\",\"view_logentry\",\"view_mailaccount\",\"view_log\",\"delete_success\",\"view_frontendsettings\",\"view_documenttype\",\"change_taskresult\",\"view_permission\",\"add_groupobjectpermission\",\"change_user\",\"view_document\",\"change_userobjectpermission\",\"add_user\",\"add_correspondent\",\"add_token\",\"add_mailaccount\",\"change_group\",\"add_group\",\"delete_processedmail\",\"delete_contenttype\",\"add_savedview\",\"view_chordcounter\",\"delete_tokenproxy\",\"change_groupresult\",\"delete_session\",\"view_savedview\",\"view_processedmail\",\"add_comment\",\"view_storagepath\",\"delete_documenttype\",\"add_processedmail\",\"view_group\",\"change_processedmail\",\"view_session\",\"delete_storagepath\",\"delete_paperlesstask\",\"add_groupresult\",\"delete_savedview\",\"delete_userobjectpermission\",\"view_tokenproxy\",\"add_task\",\"view_tag\",\"add_taskresult\",\"change_documenttype\",\"change_mailrule\",\"add_document\",\"change_comment\",\"view_task\",\"view_groupresult\",\"change_contenttype\",\"view_groupobjectpermission\",\"change_task\",\"add_log\",\"add_success\",\"change_savedview\",\"delete_frontendsettings\",\"view_success\",\"add_permission\",\"change_correspondent\",\"add_paperlesstask\",\"change_paperlesstask\",\"add_contenttype\",\"view_comment\",\"change_logentry\",\"delete_logentry\",\"delete_mailaccount\",\"change_session\",\"delete_groupresult\",\"add_logentry\",\"change_savedviewfilterrule\",\"change_success\",\"delete_tag\",\"add_taskattributes\",\"change_groupobjectpermission\",\"delete_failure\",\"add_uisettings\",\"view_token\",\"add_schedule\",\"delete_log\",\"delete_uisettings\",\"change_permission\",\"delete_groupobjectpermission\",\"change_token\",\"view_uisettings\",\"change_uisettings\",\"delete_permission\",\"add_storagepath\",\"change_storagepath\",\"view_contenttype\",\"change_note\",\"change_log\",\"change_taskattributes\",\"add_documenttype\"]}"
"text": "{\"user\":{\"id\":2,\"username\":\"testuser\",\"is_superuser\":false,\"groups\":[]},\"settings\":{\"language\":\"\",\"bulk_edit\":{\"confirmation_dialogs\":true,\"apply_on_close\":false},\"documentListSize\":50,\"dark_mode\":{\"use_system\":false,\"enabled\":\"false\",\"thumb_inverted\":\"true\"},\"theme\":{\"color\":\"#9fbf2f\"},\"document_details\":{\"native_pdf_viewer\":false},\"date_display\":{\"date_locale\":\"\",\"date_format\":\"mediumDate\"},\"notifications\":{\"consumer_new_documents\":true,\"consumer_success\":true,\"consumer_failed\":true,\"consumer_suppress_on_dashboard\":true},\"comments_enabled\":true,\"slim_sidebar\":false,\"update_checking\":{\"enabled\":false,\"backend_setting\":\"default\"},\"saved_views\":{\"warn_on_unsaved_change\":true,\"dashboard_views_visible_ids\":[7,4],\"sidebar_views_visible_ids\":[7,4,11]},\"notes_enabled\":true,\"tour_complete\":true},\"permissions\":[\"delete_schedule\",\"view_note\",\"view_taskattributes\",\"delete_ormq\",\"add_ormq\",\"change_tag\",\"change_chordcounter\",\"delete_savedviewfilterrule\",\"delete_comment\",\"add_session\",\"add_mailrule\",\"add_tokenproxy\",\"delete_taskresult\",\"view_failure\",\"add_tag\",\"view_savedviewfilterrule\",\"view_paperlesstask\",\"change_mailaccount\",\"change_frontendsettings\",\"delete_group\",\"add_userobjectpermission\",\"add_failure\",\"delete_mailrule\",\"view_userobjectpermission\",\"change_schedule\",\"delete_note\",\"view_ormq\",\"add_note\",\"add_chordcounter\",\"delete_token\",\"change_failure\",\"add_savedviewfilterrule\",\"delete_user\",\"view_correspondent\",\"view_schedule\",\"change_tokenproxy\",\"view_user\",\"delete_task\",\"delete_correspondent\",\"delete_chordcounter\",\"delete_document\",\"view_taskresult\",\"change_document\",\"add_frontendsettings\",\"view_mailrule\",\"change_ormq\",\"delete_taskattributes\",\"view_logentry\",\"view_mailaccount\",\"view_log\",\"delete_success\",\"view_frontendsettings\",\"view_documenttype\",\"change_taskresult\",\"view_permission\",\"add_groupobjectpermission\",\"change_user\",\"view_document\",\"change_userobjectpermission\",\"add_user\",\"add_correspondent\",\"add_token\",\"add_mailaccount\",\"change_group\",\"add_group\",\"delete_processedmail\",\"delete_contenttype\",\"add_savedview\",\"view_chordcounter\",\"delete_tokenproxy\",\"change_groupresult\",\"delete_session\",\"view_savedview\",\"view_processedmail\",\"add_comment\",\"view_storagepath\",\"delete_documenttype\",\"add_processedmail\",\"view_group\",\"change_processedmail\",\"view_session\",\"delete_storagepath\",\"delete_paperlesstask\",\"add_groupresult\",\"delete_savedview\",\"delete_userobjectpermission\",\"view_tokenproxy\",\"add_task\",\"view_tag\",\"add_taskresult\",\"change_documenttype\",\"change_mailrule\",\"add_document\",\"change_comment\",\"view_task\",\"view_groupresult\",\"change_contenttype\",\"view_groupobjectpermission\",\"change_task\",\"add_log\",\"add_success\",\"change_savedview\",\"delete_frontendsettings\",\"view_success\",\"add_permission\",\"change_correspondent\",\"add_paperlesstask\",\"change_paperlesstask\",\"add_contenttype\",\"view_comment\",\"change_logentry\",\"delete_logentry\",\"delete_mailaccount\",\"change_session\",\"delete_groupresult\",\"add_logentry\",\"change_savedviewfilterrule\",\"change_success\",\"delete_tag\",\"add_taskattributes\",\"change_groupobjectpermission\",\"delete_failure\",\"add_uisettings\",\"view_token\",\"add_schedule\",\"delete_log\",\"delete_uisettings\",\"change_permission\",\"delete_groupobjectpermission\",\"change_token\",\"view_uisettings\",\"change_uisettings\",\"delete_permission\",\"add_storagepath\",\"change_storagepath\",\"view_contenttype\",\"change_note\",\"change_log\",\"change_taskattributes\",\"add_documenttype\"]}"
},
"headersSize": -1,
"bodySize": -1,

View File

@@ -58,7 +58,7 @@
"content": {
"size": -1,
"mimeType": "application/json",
"text": "{\"user\":{\"id\":2,\"username\":\"testuser\",\"is_superuser\":false,\"groups\":[]},\"settings\":{\"language\":\"\",\"bulk_edit\":{\"confirmation_dialogs\":true,\"apply_on_close\":false},\"documentListSize\":50,\"dark_mode\":{\"use_system\":false,\"enabled\":\"false\",\"thumb_inverted\":\"true\"},\"theme\":{\"color\":\"#9fbf2f\"},\"document_details\":{\"native_pdf_viewer\":false},\"date_display\":{\"date_locale\":\"\",\"date_format\":\"mediumDate\"},\"notifications\":{\"consumer_new_documents\":true,\"consumer_success\":true,\"consumer_failed\":true,\"consumer_suppress_on_dashboard\":true},\"comments_enabled\":true,\"slim_sidebar\":false,\"update_checking\":{\"enabled\":false,\"backend_setting\":\"default\"},\"saved_views\":{\"warn_on_unsaved_change\":true},\"notes_enabled\":true,\"tour_complete\":true},\"permissions\":[\"delete_schedule\",\"view_note\",\"view_taskattributes\",\"delete_ormq\",\"add_ormq\",\"change_tag\",\"change_chordcounter\",\"delete_savedviewfilterrule\",\"delete_comment\",\"add_session\",\"add_mailrule\",\"add_tokenproxy\",\"delete_taskresult\",\"view_failure\",\"add_tag\",\"view_savedviewfilterrule\",\"view_paperlesstask\",\"change_mailaccount\",\"change_frontendsettings\",\"delete_group\",\"add_userobjectpermission\",\"add_failure\",\"delete_mailrule\",\"view_userobjectpermission\",\"change_schedule\",\"delete_note\",\"view_ormq\",\"add_note\",\"add_chordcounter\",\"delete_token\",\"change_failure\",\"add_savedviewfilterrule\",\"delete_user\",\"view_correspondent\",\"view_schedule\",\"change_tokenproxy\",\"view_user\",\"delete_task\",\"delete_correspondent\",\"delete_chordcounter\",\"delete_document\",\"view_taskresult\",\"change_document\",\"add_frontendsettings\",\"view_mailrule\",\"change_ormq\",\"delete_taskattributes\",\"view_logentry\",\"view_mailaccount\",\"view_log\",\"delete_success\",\"view_frontendsettings\",\"view_documenttype\",\"change_taskresult\",\"view_permission\",\"add_groupobjectpermission\",\"change_user\",\"view_document\",\"change_userobjectpermission\",\"add_user\",\"add_correspondent\",\"add_token\",\"add_mailaccount\",\"change_group\",\"add_group\",\"delete_processedmail\",\"delete_contenttype\",\"add_savedview\",\"view_chordcounter\",\"delete_tokenproxy\",\"change_groupresult\",\"delete_session\",\"view_savedview\",\"view_processedmail\",\"add_comment\",\"view_storagepath\",\"delete_documenttype\",\"add_processedmail\",\"view_group\",\"change_processedmail\",\"view_session\",\"delete_storagepath\",\"delete_paperlesstask\",\"add_groupresult\",\"delete_savedview\",\"delete_userobjectpermission\",\"view_tokenproxy\",\"add_task\",\"view_tag\",\"add_taskresult\",\"change_documenttype\",\"change_mailrule\",\"add_document\",\"change_comment\",\"view_task\",\"view_groupresult\",\"change_contenttype\",\"view_groupobjectpermission\",\"change_task\",\"add_log\",\"add_success\",\"change_savedview\",\"delete_frontendsettings\",\"view_success\",\"add_permission\",\"change_correspondent\",\"add_paperlesstask\",\"change_paperlesstask\",\"add_contenttype\",\"view_comment\",\"change_logentry\",\"delete_logentry\",\"delete_mailaccount\",\"change_session\",\"delete_groupresult\",\"add_logentry\",\"change_savedviewfilterrule\",\"change_success\",\"delete_tag\",\"add_taskattributes\",\"change_groupobjectpermission\",\"delete_failure\",\"add_uisettings\",\"view_token\",\"add_schedule\",\"delete_log\",\"delete_uisettings\",\"change_permission\",\"delete_groupobjectpermission\",\"change_token\",\"view_uisettings\",\"change_uisettings\",\"delete_permission\",\"add_storagepath\",\"change_storagepath\",\"view_contenttype\",\"change_note\",\"change_log\",\"change_taskattributes\",\"add_documenttype\"]}"
"text": "{\"user\":{\"id\":2,\"username\":\"testuser\",\"is_superuser\":false,\"groups\":[]},\"settings\":{\"language\":\"\",\"bulk_edit\":{\"confirmation_dialogs\":true,\"apply_on_close\":false},\"documentListSize\":50,\"dark_mode\":{\"use_system\":false,\"enabled\":\"false\",\"thumb_inverted\":\"true\"},\"theme\":{\"color\":\"#9fbf2f\"},\"document_details\":{\"native_pdf_viewer\":false},\"date_display\":{\"date_locale\":\"\",\"date_format\":\"mediumDate\"},\"notifications\":{\"consumer_new_documents\":true,\"consumer_success\":true,\"consumer_failed\":true,\"consumer_suppress_on_dashboard\":true},\"comments_enabled\":true,\"slim_sidebar\":false,\"update_checking\":{\"enabled\":false,\"backend_setting\":\"default\"},\"saved_views\":{\"warn_on_unsaved_change\":true,\"dashboard_views_visible_ids\":[7,4],\"sidebar_views_visible_ids\":[7,4,11]},\"notes_enabled\":true,\"tour_complete\":true},\"permissions\":[\"delete_schedule\",\"view_note\",\"view_taskattributes\",\"delete_ormq\",\"add_ormq\",\"change_tag\",\"change_chordcounter\",\"delete_savedviewfilterrule\",\"delete_comment\",\"add_session\",\"add_mailrule\",\"add_tokenproxy\",\"delete_taskresult\",\"view_failure\",\"add_tag\",\"view_savedviewfilterrule\",\"view_paperlesstask\",\"change_mailaccount\",\"change_frontendsettings\",\"delete_group\",\"add_userobjectpermission\",\"add_failure\",\"delete_mailrule\",\"view_userobjectpermission\",\"change_schedule\",\"delete_note\",\"view_ormq\",\"add_note\",\"add_chordcounter\",\"delete_token\",\"change_failure\",\"add_savedviewfilterrule\",\"delete_user\",\"view_correspondent\",\"view_schedule\",\"change_tokenproxy\",\"view_user\",\"delete_task\",\"delete_correspondent\",\"delete_chordcounter\",\"delete_document\",\"view_taskresult\",\"change_document\",\"add_frontendsettings\",\"view_mailrule\",\"change_ormq\",\"delete_taskattributes\",\"view_logentry\",\"view_mailaccount\",\"view_log\",\"delete_success\",\"view_frontendsettings\",\"view_documenttype\",\"change_taskresult\",\"view_permission\",\"add_groupobjectpermission\",\"change_user\",\"view_document\",\"change_userobjectpermission\",\"add_user\",\"add_correspondent\",\"add_token\",\"add_mailaccount\",\"change_group\",\"add_group\",\"delete_processedmail\",\"delete_contenttype\",\"add_savedview\",\"view_chordcounter\",\"delete_tokenproxy\",\"change_groupresult\",\"delete_session\",\"view_savedview\",\"view_processedmail\",\"add_comment\",\"view_storagepath\",\"delete_documenttype\",\"add_processedmail\",\"view_group\",\"change_processedmail\",\"view_session\",\"delete_storagepath\",\"delete_paperlesstask\",\"add_groupresult\",\"delete_savedview\",\"delete_userobjectpermission\",\"view_tokenproxy\",\"add_task\",\"view_tag\",\"add_taskresult\",\"change_documenttype\",\"change_mailrule\",\"add_document\",\"change_comment\",\"view_task\",\"view_groupresult\",\"change_contenttype\",\"view_groupobjectpermission\",\"change_task\",\"add_log\",\"add_success\",\"change_savedview\",\"delete_frontendsettings\",\"view_success\",\"add_permission\",\"change_correspondent\",\"add_paperlesstask\",\"change_paperlesstask\",\"add_contenttype\",\"view_comment\",\"change_logentry\",\"delete_logentry\",\"delete_mailaccount\",\"change_session\",\"delete_groupresult\",\"add_logentry\",\"change_savedviewfilterrule\",\"change_success\",\"delete_tag\",\"add_taskattributes\",\"change_groupobjectpermission\",\"delete_failure\",\"add_uisettings\",\"view_token\",\"add_schedule\",\"delete_log\",\"delete_uisettings\",\"change_permission\",\"delete_groupobjectpermission\",\"change_token\",\"view_uisettings\",\"change_uisettings\",\"delete_permission\",\"add_storagepath\",\"change_storagepath\",\"view_contenttype\",\"change_note\",\"change_log\",\"change_taskattributes\",\"add_documenttype\"]}"
},
"headersSize": -1,
"bodySize": -1,

View File

@@ -58,7 +58,7 @@
"content": {
"size": -1,
"mimeType": "application/json",
"text": "{\"user\":{\"id\":2,\"username\":\"testuser\",\"is_superuser\":false,\"groups\":[]},\"settings\":{\"language\":\"\",\"bulk_edit\":{\"confirmation_dialogs\":true,\"apply_on_close\":false},\"documentListSize\":50,\"dark_mode\":{\"use_system\":false,\"enabled\":\"false\",\"thumb_inverted\":\"true\"},\"theme\":{\"color\":\"#9fbf2f\"},\"document_details\":{\"native_pdf_viewer\":false},\"date_display\":{\"date_locale\":\"\",\"date_format\":\"mediumDate\"},\"notifications\":{\"consumer_new_documents\":true,\"consumer_success\":true,\"consumer_failed\":true,\"consumer_suppress_on_dashboard\":true},\"comments_enabled\":true,\"slim_sidebar\":false,\"update_checking\":{\"enabled\":false,\"backend_setting\":\"default\"},\"saved_views\":{\"warn_on_unsaved_change\":true},\"notes_enabled\":true,\"tour_complete\":true},\"permissions\":[\"delete_schedule\",\"view_note\",\"view_taskattributes\",\"delete_ormq\",\"add_ormq\",\"change_tag\",\"change_chordcounter\",\"delete_savedviewfilterrule\",\"delete_comment\",\"add_session\",\"add_mailrule\",\"add_tokenproxy\",\"delete_taskresult\",\"view_failure\",\"add_tag\",\"view_savedviewfilterrule\",\"view_paperlesstask\",\"change_mailaccount\",\"change_frontendsettings\",\"delete_group\",\"add_userobjectpermission\",\"add_failure\",\"delete_mailrule\",\"view_userobjectpermission\",\"change_schedule\",\"delete_note\",\"view_ormq\",\"add_note\",\"add_chordcounter\",\"delete_token\",\"change_failure\",\"add_savedviewfilterrule\",\"delete_user\",\"view_correspondent\",\"view_schedule\",\"change_tokenproxy\",\"view_user\",\"delete_task\",\"delete_correspondent\",\"delete_chordcounter\",\"delete_document\",\"view_taskresult\",\"change_document\",\"add_frontendsettings\",\"view_mailrule\",\"change_ormq\",\"delete_taskattributes\",\"view_logentry\",\"view_mailaccount\",\"view_log\",\"delete_success\",\"view_frontendsettings\",\"view_documenttype\",\"change_taskresult\",\"view_permission\",\"add_groupobjectpermission\",\"change_user\",\"view_document\",\"change_userobjectpermission\",\"add_user\",\"add_correspondent\",\"add_token\",\"add_mailaccount\",\"change_group\",\"add_group\",\"delete_processedmail\",\"delete_contenttype\",\"add_savedview\",\"view_chordcounter\",\"delete_tokenproxy\",\"change_groupresult\",\"delete_session\",\"view_savedview\",\"view_processedmail\",\"add_comment\",\"view_storagepath\",\"delete_documenttype\",\"add_processedmail\",\"view_group\",\"change_processedmail\",\"view_session\",\"delete_storagepath\",\"delete_paperlesstask\",\"add_groupresult\",\"delete_savedview\",\"delete_userobjectpermission\",\"view_tokenproxy\",\"add_task\",\"view_tag\",\"add_taskresult\",\"change_documenttype\",\"change_mailrule\",\"add_document\",\"change_comment\",\"view_task\",\"view_groupresult\",\"change_contenttype\",\"view_groupobjectpermission\",\"change_task\",\"add_log\",\"add_success\",\"change_savedview\",\"delete_frontendsettings\",\"view_success\",\"add_permission\",\"change_correspondent\",\"add_paperlesstask\",\"change_paperlesstask\",\"add_contenttype\",\"view_comment\",\"change_logentry\",\"delete_logentry\",\"delete_mailaccount\",\"change_session\",\"delete_groupresult\",\"add_logentry\",\"change_savedviewfilterrule\",\"change_success\",\"delete_tag\",\"add_taskattributes\",\"change_groupobjectpermission\",\"delete_failure\",\"add_uisettings\",\"view_token\",\"add_schedule\",\"delete_log\",\"delete_uisettings\",\"change_permission\",\"delete_groupobjectpermission\",\"change_token\",\"view_uisettings\",\"change_uisettings\",\"delete_permission\",\"add_storagepath\",\"change_storagepath\",\"view_contenttype\",\"change_note\",\"change_log\",\"change_taskattributes\",\"add_documenttype\"]}"
"text": "{\"user\":{\"id\":2,\"username\":\"testuser\",\"is_superuser\":false,\"groups\":[]},\"settings\":{\"language\":\"\",\"bulk_edit\":{\"confirmation_dialogs\":true,\"apply_on_close\":false},\"documentListSize\":50,\"dark_mode\":{\"use_system\":false,\"enabled\":\"false\",\"thumb_inverted\":\"true\"},\"theme\":{\"color\":\"#9fbf2f\"},\"document_details\":{\"native_pdf_viewer\":false},\"date_display\":{\"date_locale\":\"\",\"date_format\":\"mediumDate\"},\"notifications\":{\"consumer_new_documents\":true,\"consumer_success\":true,\"consumer_failed\":true,\"consumer_suppress_on_dashboard\":true},\"comments_enabled\":true,\"slim_sidebar\":false,\"update_checking\":{\"enabled\":false,\"backend_setting\":\"default\"},\"saved_views\":{\"warn_on_unsaved_change\":true,\"dashboard_views_visible_ids\":[7,4],\"sidebar_views_visible_ids\":[7,4,11]},\"notes_enabled\":true,\"tour_complete\":true},\"permissions\":[\"delete_schedule\",\"view_note\",\"view_taskattributes\",\"delete_ormq\",\"add_ormq\",\"change_tag\",\"change_chordcounter\",\"delete_savedviewfilterrule\",\"delete_comment\",\"add_session\",\"add_mailrule\",\"add_tokenproxy\",\"delete_taskresult\",\"view_failure\",\"add_tag\",\"view_savedviewfilterrule\",\"view_paperlesstask\",\"change_mailaccount\",\"change_frontendsettings\",\"delete_group\",\"add_userobjectpermission\",\"add_failure\",\"delete_mailrule\",\"view_userobjectpermission\",\"change_schedule\",\"delete_note\",\"view_ormq\",\"add_note\",\"add_chordcounter\",\"delete_token\",\"change_failure\",\"add_savedviewfilterrule\",\"delete_user\",\"view_correspondent\",\"view_schedule\",\"change_tokenproxy\",\"view_user\",\"delete_task\",\"delete_correspondent\",\"delete_chordcounter\",\"delete_document\",\"view_taskresult\",\"change_document\",\"add_frontendsettings\",\"view_mailrule\",\"change_ormq\",\"delete_taskattributes\",\"view_logentry\",\"view_mailaccount\",\"view_log\",\"delete_success\",\"view_frontendsettings\",\"view_documenttype\",\"change_taskresult\",\"view_permission\",\"add_groupobjectpermission\",\"change_user\",\"view_document\",\"change_userobjectpermission\",\"add_user\",\"add_correspondent\",\"add_token\",\"add_mailaccount\",\"change_group\",\"add_group\",\"delete_processedmail\",\"delete_contenttype\",\"add_savedview\",\"view_chordcounter\",\"delete_tokenproxy\",\"change_groupresult\",\"delete_session\",\"view_savedview\",\"view_processedmail\",\"add_comment\",\"view_storagepath\",\"delete_documenttype\",\"add_processedmail\",\"view_group\",\"change_processedmail\",\"view_session\",\"delete_storagepath\",\"delete_paperlesstask\",\"add_groupresult\",\"delete_savedview\",\"delete_userobjectpermission\",\"view_tokenproxy\",\"add_task\",\"view_tag\",\"add_taskresult\",\"change_documenttype\",\"change_mailrule\",\"add_document\",\"change_comment\",\"view_task\",\"view_groupresult\",\"change_contenttype\",\"view_groupobjectpermission\",\"change_task\",\"add_log\",\"add_success\",\"change_savedview\",\"delete_frontendsettings\",\"view_success\",\"add_permission\",\"change_correspondent\",\"add_paperlesstask\",\"change_paperlesstask\",\"add_contenttype\",\"view_comment\",\"change_logentry\",\"delete_logentry\",\"delete_mailaccount\",\"change_session\",\"delete_groupresult\",\"add_logentry\",\"change_savedviewfilterrule\",\"change_success\",\"delete_tag\",\"add_taskattributes\",\"change_groupobjectpermission\",\"delete_failure\",\"add_uisettings\",\"view_token\",\"add_schedule\",\"delete_log\",\"delete_uisettings\",\"change_permission\",\"delete_groupobjectpermission\",\"change_token\",\"view_uisettings\",\"change_uisettings\",\"delete_permission\",\"add_storagepath\",\"change_storagepath\",\"view_contenttype\",\"change_note\",\"change_log\",\"change_taskattributes\",\"add_documenttype\"]}"
},
"headersSize": -1,
"bodySize": -1,

View File

@@ -58,7 +58,7 @@
"content": {
"size": -1,
"mimeType": "application/json",
"text": "{\"user\":{\"id\":2,\"username\":\"testuser\",\"is_superuser\":false,\"groups\":[]},\"settings\":{\"language\":\"\",\"bulk_edit\":{\"confirmation_dialogs\":true,\"apply_on_close\":false},\"documentListSize\":50,\"dark_mode\":{\"use_system\":false,\"enabled\":\"false\",\"thumb_inverted\":\"true\"},\"theme\":{\"color\":\"#9fbf2f\"},\"document_details\":{\"native_pdf_viewer\":false},\"date_display\":{\"date_locale\":\"\",\"date_format\":\"mediumDate\"},\"notifications\":{\"consumer_new_documents\":true,\"consumer_success\":true,\"consumer_failed\":true,\"consumer_suppress_on_dashboard\":true},\"comments_enabled\":true,\"slim_sidebar\":false,\"update_checking\":{\"enabled\":false,\"backend_setting\":\"default\"},\"saved_views\":{\"warn_on_unsaved_change\":true},\"notes_enabled\":true,\"tour_complete\":true},\"permissions\":[\"delete_schedule\",\"view_note\",\"view_taskattributes\",\"delete_ormq\",\"add_ormq\",\"change_tag\",\"change_chordcounter\",\"delete_savedviewfilterrule\",\"delete_comment\",\"add_session\",\"add_mailrule\",\"add_tokenproxy\",\"delete_taskresult\",\"view_failure\",\"add_tag\",\"view_savedviewfilterrule\",\"view_paperlesstask\",\"change_mailaccount\",\"change_frontendsettings\",\"delete_group\",\"add_userobjectpermission\",\"add_failure\",\"delete_mailrule\",\"view_userobjectpermission\",\"change_schedule\",\"delete_note\",\"view_ormq\",\"add_note\",\"add_chordcounter\",\"delete_token\",\"change_failure\",\"add_savedviewfilterrule\",\"delete_user\",\"view_correspondent\",\"view_schedule\",\"change_tokenproxy\",\"view_user\",\"delete_task\",\"delete_correspondent\",\"delete_chordcounter\",\"delete_document\",\"view_taskresult\",\"change_document\",\"add_frontendsettings\",\"view_mailrule\",\"change_ormq\",\"delete_taskattributes\",\"view_logentry\",\"view_mailaccount\",\"view_log\",\"delete_success\",\"view_frontendsettings\",\"view_documenttype\",\"change_taskresult\",\"view_permission\",\"add_groupobjectpermission\",\"change_user\",\"view_document\",\"change_userobjectpermission\",\"add_user\",\"add_correspondent\",\"add_token\",\"add_mailaccount\",\"change_group\",\"add_group\",\"delete_processedmail\",\"delete_contenttype\",\"add_savedview\",\"view_chordcounter\",\"delete_tokenproxy\",\"change_groupresult\",\"delete_session\",\"view_savedview\",\"view_processedmail\",\"add_comment\",\"view_storagepath\",\"delete_documenttype\",\"add_processedmail\",\"view_group\",\"change_processedmail\",\"view_session\",\"delete_storagepath\",\"delete_paperlesstask\",\"add_groupresult\",\"delete_savedview\",\"delete_userobjectpermission\",\"view_tokenproxy\",\"add_task\",\"view_tag\",\"add_taskresult\",\"change_documenttype\",\"change_mailrule\",\"add_document\",\"change_comment\",\"view_task\",\"view_groupresult\",\"change_contenttype\",\"view_groupobjectpermission\",\"change_task\",\"add_log\",\"add_success\",\"change_savedview\",\"delete_frontendsettings\",\"view_success\",\"add_permission\",\"change_correspondent\",\"add_paperlesstask\",\"change_paperlesstask\",\"add_contenttype\",\"view_comment\",\"change_logentry\",\"delete_logentry\",\"delete_mailaccount\",\"change_session\",\"delete_groupresult\",\"add_logentry\",\"change_savedviewfilterrule\",\"change_success\",\"delete_tag\",\"add_taskattributes\",\"change_groupobjectpermission\",\"delete_failure\",\"add_uisettings\",\"view_token\",\"add_schedule\",\"delete_log\",\"delete_uisettings\",\"change_permission\",\"delete_groupobjectpermission\",\"change_token\",\"view_uisettings\",\"change_uisettings\",\"delete_permission\",\"add_storagepath\",\"change_storagepath\",\"view_contenttype\",\"change_note\",\"change_log\",\"change_taskattributes\",\"add_documenttype\"]}"
"text": "{\"user\":{\"id\":2,\"username\":\"testuser\",\"is_superuser\":false,\"groups\":[]},\"settings\":{\"language\":\"\",\"bulk_edit\":{\"confirmation_dialogs\":true,\"apply_on_close\":false},\"documentListSize\":50,\"dark_mode\":{\"use_system\":false,\"enabled\":\"false\",\"thumb_inverted\":\"true\"},\"theme\":{\"color\":\"#9fbf2f\"},\"document_details\":{\"native_pdf_viewer\":false},\"date_display\":{\"date_locale\":\"\",\"date_format\":\"mediumDate\"},\"notifications\":{\"consumer_new_documents\":true,\"consumer_success\":true,\"consumer_failed\":true,\"consumer_suppress_on_dashboard\":true},\"comments_enabled\":true,\"slim_sidebar\":false,\"update_checking\":{\"enabled\":false,\"backend_setting\":\"default\"},\"saved_views\":{\"warn_on_unsaved_change\":true,\"dashboard_views_visible_ids\":[7,4],\"sidebar_views_visible_ids\":[7,4,11]},\"notes_enabled\":true,\"tour_complete\":true},\"permissions\":[\"delete_schedule\",\"view_note\",\"view_taskattributes\",\"delete_ormq\",\"add_ormq\",\"change_tag\",\"change_chordcounter\",\"delete_savedviewfilterrule\",\"delete_comment\",\"add_session\",\"add_mailrule\",\"add_tokenproxy\",\"delete_taskresult\",\"view_failure\",\"add_tag\",\"view_savedviewfilterrule\",\"view_paperlesstask\",\"change_mailaccount\",\"change_frontendsettings\",\"delete_group\",\"add_userobjectpermission\",\"add_failure\",\"delete_mailrule\",\"view_userobjectpermission\",\"change_schedule\",\"delete_note\",\"view_ormq\",\"add_note\",\"add_chordcounter\",\"delete_token\",\"change_failure\",\"add_savedviewfilterrule\",\"delete_user\",\"view_correspondent\",\"view_schedule\",\"change_tokenproxy\",\"view_user\",\"delete_task\",\"delete_correspondent\",\"delete_chordcounter\",\"delete_document\",\"view_taskresult\",\"change_document\",\"add_frontendsettings\",\"view_mailrule\",\"change_ormq\",\"delete_taskattributes\",\"view_logentry\",\"view_mailaccount\",\"view_log\",\"delete_success\",\"view_frontendsettings\",\"view_documenttype\",\"change_taskresult\",\"view_permission\",\"add_groupobjectpermission\",\"change_user\",\"view_document\",\"change_userobjectpermission\",\"add_user\",\"add_correspondent\",\"add_token\",\"add_mailaccount\",\"change_group\",\"add_group\",\"delete_processedmail\",\"delete_contenttype\",\"add_savedview\",\"view_chordcounter\",\"delete_tokenproxy\",\"change_groupresult\",\"delete_session\",\"view_savedview\",\"view_processedmail\",\"add_comment\",\"view_storagepath\",\"delete_documenttype\",\"add_processedmail\",\"view_group\",\"change_processedmail\",\"view_session\",\"delete_storagepath\",\"delete_paperlesstask\",\"add_groupresult\",\"delete_savedview\",\"delete_userobjectpermission\",\"view_tokenproxy\",\"add_task\",\"view_tag\",\"add_taskresult\",\"change_documenttype\",\"change_mailrule\",\"add_document\",\"change_comment\",\"view_task\",\"view_groupresult\",\"change_contenttype\",\"view_groupobjectpermission\",\"change_task\",\"add_log\",\"add_success\",\"change_savedview\",\"delete_frontendsettings\",\"view_success\",\"add_permission\",\"change_correspondent\",\"add_paperlesstask\",\"change_paperlesstask\",\"add_contenttype\",\"view_comment\",\"change_logentry\",\"delete_logentry\",\"delete_mailaccount\",\"change_session\",\"delete_groupresult\",\"add_logentry\",\"change_savedviewfilterrule\",\"change_success\",\"delete_tag\",\"add_taskattributes\",\"change_groupobjectpermission\",\"delete_failure\",\"add_uisettings\",\"view_token\",\"add_schedule\",\"delete_log\",\"delete_uisettings\",\"change_permission\",\"delete_groupobjectpermission\",\"change_token\",\"view_uisettings\",\"change_uisettings\",\"delete_permission\",\"add_storagepath\",\"change_storagepath\",\"view_contenttype\",\"change_note\",\"change_log\",\"change_taskattributes\",\"add_documenttype\"]}"
},
"headersSize": -1,
"bodySize": -1,

File diff suppressed because it is too large Load Diff

View File

@@ -1,6 +1,6 @@
{
"name": "paperless-ngx-ui",
"version": "2.20.9",
"version": "2.20.10",
"scripts": {
"preinstall": "npx only-allow pnpm",
"ng": "ng",

View File

@@ -277,7 +277,7 @@
<div class="col">
<select class="form-select" formControlName="pdfEditorDefaultEditMode">
<option [ngValue]="PdfEditorEditMode.Create" i18n>Create new document(s)</option>
<option [ngValue]="PdfEditorEditMode.Update" i18n>Update existing document</option>
<option [ngValue]="PdfEditorEditMode.Update" i18n>Add document version</option>
</select>
</div>
</div>

View File

@@ -99,12 +99,7 @@
</ul>
<div class="nav-group mt-3 mb-1" *pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.SavedView }">
@if (savedViewService.loading) {
<h6 class="sidebar-heading px-3 text-muted">
<span i18n>Saved views</span>
<div class="spinner-border spinner-border-sm fw-normal ms-2" role="status"></div>
</h6>
} @else if (savedViewService.sidebarViews?.length > 0) {
@if (savedViewService.sidebarViews?.length > 0) {
<h6 class="sidebar-heading px-3 text-muted">
<span i18n>Saved views</span>
</h6>
@@ -134,6 +129,11 @@
</li>
}
</ul>
} @else if (savedViewService.loading) {
<h6 class="sidebar-heading px-3 text-muted">
<span i18n>Saved views</span>
<div class="spinner-border spinner-border-sm fw-normal ms-2" role="status"></div>
</h6>
}
</div>

View File

@@ -84,7 +84,7 @@
<input type="radio" class="btn-check" [(ngModel)]="editMode" [value]="PdfEditorEditMode.Update" id="editModeUpdate" name="editmode" [disabled]="hasSplit()">
<label for="editModeUpdate" class="btn btn-outline-primary btn-sm">
<i-bs name="pencil"></i-bs>
<span class="form-check-label ms-2" i18n>Update existing document</span>
<span class="form-check-label ms-2" i18n>Add document version</span>
</label>
</div>
@if (editMode === PdfEditorEditMode.Create) {

View File

@@ -16,6 +16,12 @@
</div>
</form>
@if (note) {
<div class="small text-muted fst-italic mt-2">
{{ note }}
</div>
}
</div>
<div class="modal-footer">
@if (!buttonsEnabled) {

View File

@@ -40,6 +40,9 @@ export class PermissionsDialogComponent {
@Input()
title = $localize`Set permissions`
@Input()
note: string = null
@Input()
set object(o: ObjectWithPermissions) {
this.o = o

View File

@@ -65,6 +65,7 @@ import { TagService } from 'src/app/services/rest/tag.service'
import { UserService } from 'src/app/services/rest/user.service'
import { SettingsService } from 'src/app/services/settings.service'
import { ToastService } from 'src/app/services/toast.service'
import { WebsocketStatusService } from 'src/app/services/websocket-status.service'
import { environment } from 'src/environments/environment'
import { ConfirmDialogComponent } from '../common/confirm-dialog/confirm-dialog.component'
import { PasswordRemovalConfirmDialogComponent } from '../common/confirm-dialog/password-removal-confirm-dialog/password-removal-confirm-dialog.component'
@@ -83,9 +84,9 @@ const doc: Document = {
storage_path: 31,
tags: [41, 42, 43],
content: 'text content',
added: new Date('May 4, 2014 03:24:00'),
created: new Date('May 4, 2014 03:24:00'),
modified: new Date('May 4, 2014 03:24:00'),
added: new Date('May 4, 2014 03:24:00').toISOString(),
created: new Date('May 4, 2014 03:24:00').toISOString(),
modified: new Date('May 4, 2014 03:24:00').toISOString(),
archive_serial_number: null,
original_file_name: 'file.pdf',
owner: null,
@@ -327,6 +328,29 @@ describe('DocumentDetailComponent', () => {
expect(component.activeNavID).toEqual(component.DocumentDetailNavIDs.Notes)
})
it('should switch from preview to details when pdf preview enters the DOM', fakeAsync(() => {
component.nav = {
activeId: component.DocumentDetailNavIDs.Preview,
select: jest.fn(),
} as any
;(component as any).pdfPreview = {
nativeElement: { offsetParent: {} },
}
tick()
expect(component.nav.select).toHaveBeenCalledWith(
component.DocumentDetailNavIDs.Details
)
}))
it('should forward title key up value to titleSubject', () => {
const subjectSpy = jest.spyOn(component.titleSubject, 'next')
component.titleKeyUp({ target: { value: 'Updated title' } })
expect(subjectSpy).toHaveBeenCalledWith('Updated title')
})
it('should change url on tab switch', () => {
initNormally()
const navigateSpy = jest.spyOn(router, 'navigate')
@@ -524,7 +548,7 @@ describe('DocumentDetailComponent', () => {
jest.spyOn(documentService, 'get').mockReturnValue(
of({
...doc,
modified: new Date('2024-01-02T00:00:00Z'),
modified: '2024-01-02T00:00:00Z',
duplicate_documents: updatedDuplicates,
})
)
@@ -1386,17 +1410,21 @@ describe('DocumentDetailComponent', () => {
expect(errorSpy).toHaveBeenCalled()
})
it('should warn when open document does not match doc retrieved from backend on init', () => {
it('should show incoming update modal when open local draft is older than backend on init', () => {
let openModal: NgbModalRef
modalService.activeInstances.subscribe((modals) => (openModal = modals[0]))
const modalSpy = jest.spyOn(modalService, 'open')
const openDoc = Object.assign({}, doc)
const openDoc = Object.assign({}, doc, {
__changedFields: ['title'],
})
// simulate a document being modified elsewhere and db updated
doc.modified = new Date()
const remoteDoc = Object.assign({}, doc, {
modified: new Date(new Date(doc.modified).getTime() + 1000).toISOString(),
})
jest
.spyOn(activatedRoute, 'paramMap', 'get')
.mockReturnValue(of(convertToParamMap({ id: 3, section: 'details' })))
jest.spyOn(documentService, 'get').mockReturnValueOnce(of(doc))
jest.spyOn(documentService, 'get').mockReturnValueOnce(of(remoteDoc))
jest.spyOn(openDocumentsService, 'getOpenDocument').mockReturnValue(openDoc)
jest.spyOn(customFieldsService, 'listAll').mockReturnValue(
of({
@@ -1406,11 +1434,185 @@ describe('DocumentDetailComponent', () => {
})
)
fixture.detectChanges() // calls ngOnInit
expect(modalSpy).toHaveBeenCalledWith(ConfirmDialogComponent)
const closeSpy = jest.spyOn(openModal, 'close')
expect(modalSpy).toHaveBeenCalledWith(ConfirmDialogComponent, {
backdrop: 'static',
})
const confirmDialog = openModal.componentInstance as ConfirmDialogComponent
confirmDialog.confirmClicked.next(confirmDialog)
expect(closeSpy).toHaveBeenCalled()
expect(confirmDialog.messageBold).toContain('Document was updated at')
})
it('should react to websocket document updated notifications', () => {
initNormally()
const updateMessage = {
document_id: component.documentId,
modified: '2026-02-17T00:00:00Z',
owner_id: 1,
}
const handleSpy = jest
.spyOn(component as any, 'handleIncomingDocumentUpdated')
.mockImplementation(() => {})
const websocketStatusService = TestBed.inject(WebsocketStatusService)
websocketStatusService.handleDocumentUpdated(updateMessage)
expect(handleSpy).toHaveBeenCalledWith(updateMessage)
})
it('should queue incoming update while network is active and flush after', () => {
initNormally()
const loadSpy = jest.spyOn(component as any, 'loadDocument')
const toastSpy = jest.spyOn(toastService, 'showInfo')
component.networkActive = true
;(component as any).handleIncomingDocumentUpdated({
document_id: component.documentId,
modified: '2026-02-17T00:00:00Z',
})
expect(loadSpy).not.toHaveBeenCalled()
component.networkActive = false
;(component as any).flushPendingIncomingUpdate()
expect(loadSpy).toHaveBeenCalledWith(component.documentId, true)
expect(toastSpy).toHaveBeenCalledWith(
'Document reloaded with latest changes.'
)
})
it('should ignore queued incoming update matching local save modified', () => {
initNormally()
const loadSpy = jest.spyOn(component as any, 'loadDocument')
const toastSpy = jest.spyOn(toastService, 'showInfo')
component.networkActive = true
;(component as any).lastLocalSaveModified = '2026-02-17T00:00:00+00:00'
;(component as any).handleIncomingDocumentUpdated({
document_id: component.documentId,
modified: '2026-02-17T00:00:00+00:00',
})
component.networkActive = false
;(component as any).flushPendingIncomingUpdate()
expect(loadSpy).not.toHaveBeenCalled()
expect(toastSpy).not.toHaveBeenCalled()
})
it('should clear pdf source if preview URL is empty', () => {
component.pdfSource = { url: '/preview', password: 'secret' } as any
component.previewUrl = null
;(component as any).updatePdfSource()
expect(component.pdfSource).toEqual({ url: null, password: undefined })
})
it('should close incoming update modal if one is open', () => {
const modalRef = { close: jest.fn() } as unknown as NgbModalRef
;(component as any).incomingUpdateModal = modalRef
;(component as any).closeIncomingUpdateModal()
expect(modalRef.close).toHaveBeenCalled()
expect((component as any).incomingUpdateModal).toBeNull()
})
it('should reload remote version when incoming update modal is confirmed', async () => {
let openModal: NgbModalRef
modalService.activeInstances.subscribe((modals) => (openModal = modals[0]))
const reloadSpy = jest
.spyOn(component as any, 'reloadRemoteVersion')
.mockImplementation(() => {})
;(component as any).showIncomingUpdateModal('2026-02-17T00:00:00Z')
const dialog = openModal.componentInstance as ConfirmDialogComponent
dialog.confirmClicked.next()
await openModal.result
expect(dialog.buttonsEnabled).toBe(false)
expect(reloadSpy).toHaveBeenCalled()
expect((component as any).incomingUpdateModal).toBeNull()
})
it('should overwrite open document state when loading remote version with force', () => {
const openDoc = Object.assign({}, doc, {
title: 'Locally edited title',
__changedFields: ['title'],
})
const remoteDoc = Object.assign({}, doc, {
title: 'Remote title',
modified: '2026-02-17T00:00:00Z',
})
jest.spyOn(documentService, 'get').mockReturnValue(of(remoteDoc))
jest.spyOn(documentService, 'getMetadata').mockReturnValue(
of({
has_archive_version: false,
original_mime_type: 'application/pdf',
})
)
jest.spyOn(documentService, 'getSuggestions').mockReturnValue(
of({
suggested_tags: [],
suggested_document_types: [],
suggested_correspondents: [],
})
)
jest.spyOn(openDocumentsService, 'getOpenDocument').mockReturnValue(openDoc)
const setDirtySpy = jest.spyOn(openDocumentsService, 'setDirty')
const saveSpy = jest.spyOn(openDocumentsService, 'save')
;(component as any).loadDocument(doc.id, true)
expect(openDoc.title).toEqual('Remote title')
expect(openDoc.__changedFields).toEqual([])
expect(setDirtySpy).toHaveBeenCalledWith(openDoc, false)
expect(saveSpy).toHaveBeenCalled()
})
it('should ignore incoming update for a different document id', () => {
initNormally()
const loadSpy = jest.spyOn(component as any, 'loadDocument')
;(component as any).handleIncomingDocumentUpdated({
document_id: component.documentId + 1,
modified: '2026-02-17T00:00:00Z',
})
expect(loadSpy).not.toHaveBeenCalled()
})
it('should show incoming update modal when local document has unsaved edits', () => {
initNormally()
jest.spyOn(openDocumentsService, 'isDirty').mockReturnValue(true)
const modalSpy = jest
.spyOn(component as any, 'showIncomingUpdateModal')
.mockImplementation(() => {})
;(component as any).handleIncomingDocumentUpdated({
document_id: component.documentId,
modified: '2026-02-17T00:00:00Z',
})
expect(modalSpy).toHaveBeenCalledWith('2026-02-17T00:00:00Z')
})
it('should reload current document and show toast when reloading remote version', () => {
component.documentId = doc.id
const closeModalSpy = jest
.spyOn(component as any, 'closeIncomingUpdateModal')
.mockImplementation(() => {})
const loadSpy = jest
.spyOn(component as any, 'loadDocument')
.mockImplementation(() => {})
const notifySpy = jest.spyOn(component.docChangeNotifier, 'next')
const toastSpy = jest.spyOn(toastService, 'showInfo')
;(component as any).reloadRemoteVersion()
expect(closeModalSpy).toHaveBeenCalled()
expect(notifySpy).toHaveBeenCalledWith(doc.id)
expect(loadSpy).toHaveBeenCalledWith(doc.id, true)
expect(toastSpy).toHaveBeenCalledWith('Document reloaded.')
})
it('should change preview element by render type', () => {
@@ -1721,6 +1923,14 @@ describe('DocumentDetailComponent', () => {
expect(component.createDisabled(DataType.Tag)).toBeFalsy()
})
it('should expose add permission via userCanAdd getter', () => {
currentUserCan = true
expect(component.userCanAdd).toBeTruthy()
currentUserCan = false
expect(component.userCanAdd).toBeFalsy()
})
it('should call tryRenderTiff when no archive and file is tiff', () => {
initNormally()
const tiffRenderSpy = jest.spyOn(

View File

@@ -13,6 +13,7 @@ import {
NgbDateStruct,
NgbDropdownModule,
NgbModal,
NgbModalRef,
NgbNav,
NgbNavChangeEvent,
NgbNavModule,
@@ -80,6 +81,7 @@ import { TagService } from 'src/app/services/rest/tag.service'
import { UserService } from 'src/app/services/rest/user.service'
import { SettingsService } from 'src/app/services/settings.service'
import { ToastService } from 'src/app/services/toast.service'
import { WebsocketStatusService } from 'src/app/services/websocket-status.service'
import { getFilenameFromContentDisposition } from 'src/app/utils/http'
import { ISODateAdapter } from 'src/app/utils/ngb-iso-date-adapter'
import * as UTIF from 'utif'
@@ -143,6 +145,11 @@ enum ContentRenderType {
TIFF = 'tiff',
}
interface IncomingDocumentUpdate {
document_id: number
modified: string
}
@Component({
selector: 'pngx-document-detail',
templateUrl: './document-detail.component.html',
@@ -208,6 +215,7 @@ export class DocumentDetailComponent
private componentRouterService = inject(ComponentRouterService)
private deviceDetectorService = inject(DeviceDetectorService)
private savedViewService = inject(SavedViewService)
private readonly websocketStatusService = inject(WebsocketStatusService)
@ViewChild('inputTitle')
titleInput: TextComponent
@@ -267,6 +275,9 @@ export class DocumentDetailComponent
isDirty$: Observable<boolean>
unsubscribeNotifier: Subject<any> = new Subject()
docChangeNotifier: Subject<any> = new Subject()
private incomingUpdateModal: NgbModalRef
private pendingIncomingUpdate: IncomingDocumentUpdate
private lastLocalSaveModified: string | null = null
requiresPassword: boolean = false
password: string
@@ -475,9 +486,12 @@ export class DocumentDetailComponent
)
}
private loadDocument(documentId: number): void {
private loadDocument(documentId: number, forceRemote: boolean = false): void {
let redirectedToRoot = false
this.closeIncomingUpdateModal()
this.pendingIncomingUpdate = null
this.selectedVersionId = documentId
this.lastLocalSaveModified = null
this.previewUrl = this.documentsService.getPreviewUrl(
this.selectedVersionId
)
@@ -545,21 +559,25 @@ export class DocumentDetailComponent
openDocument.duplicate_documents = doc.duplicate_documents
this.openDocumentService.save()
}
const useDoc = openDocument || doc
if (openDocument) {
if (
new Date(doc.modified) > new Date(openDocument.modified) &&
!this.modalService.hasOpenModals()
) {
const modal = this.modalService.open(ConfirmDialogComponent)
modal.componentInstance.title = $localize`Document changes detected`
modal.componentInstance.messageBold = $localize`The version of this document in your browser session appears older than the existing version.`
modal.componentInstance.message = $localize`Saving the document here may overwrite other changes that were made. To restore the existing version, discard your changes or close the document.`
modal.componentInstance.cancelBtnClass = 'visually-hidden'
modal.componentInstance.btnCaption = $localize`Ok`
modal.componentInstance.confirmClicked.subscribe(() =>
modal.close()
)
let useDoc = openDocument || doc
if (openDocument && forceRemote) {
Object.assign(openDocument, doc)
openDocument.__changedFields = []
this.openDocumentService.setDirty(openDocument, false)
this.openDocumentService.save()
useDoc = openDocument
} else if (openDocument) {
if (new Date(doc.modified) > new Date(openDocument.modified)) {
if (this.hasLocalEdits(openDocument)) {
this.showIncomingUpdateModal(doc.modified)
} else {
// No local edits to preserve, so keep the tab in sync automatically.
Object.assign(openDocument, doc)
openDocument.__changedFields = []
this.openDocumentService.setDirty(openDocument, false)
this.openDocumentService.save()
useDoc = openDocument
}
}
} else {
this.openDocumentService
@@ -590,6 +608,98 @@ export class DocumentDetailComponent
})
}
private hasLocalEdits(doc: Document): boolean {
return (
this.openDocumentService.isDirty(doc) || !!doc.__changedFields?.length
)
}
private showIncomingUpdateModal(modified: string): void {
if (this.incomingUpdateModal) return
const modal = this.modalService.open(ConfirmDialogComponent, {
backdrop: 'static',
})
this.incomingUpdateModal = modal
let formattedModified = null
const parsed = new Date(modified)
formattedModified = parsed.toLocaleString()
modal.componentInstance.title = $localize`Document was updated`
modal.componentInstance.messageBold = $localize`Document was updated at ${formattedModified}.`
modal.componentInstance.message = $localize`Reload to discard your local unsaved edits and load the latest remote version.`
modal.componentInstance.btnClass = 'btn-warning'
modal.componentInstance.btnCaption = $localize`Reload`
modal.componentInstance.cancelBtnCaption = $localize`Dismiss`
modal.componentInstance.confirmClicked.pipe(first()).subscribe(() => {
modal.componentInstance.buttonsEnabled = false
modal.close()
this.reloadRemoteVersion()
})
modal.result.finally(() => {
this.incomingUpdateModal = null
})
}
private closeIncomingUpdateModal() {
if (!this.incomingUpdateModal) return
this.incomingUpdateModal.close()
this.incomingUpdateModal = null
}
private flushPendingIncomingUpdate() {
if (!this.pendingIncomingUpdate || this.networkActive) return
const pendingUpdate = this.pendingIncomingUpdate
this.pendingIncomingUpdate = null
this.handleIncomingDocumentUpdated(pendingUpdate)
}
private handleIncomingDocumentUpdated(data: IncomingDocumentUpdate): void {
if (
!this.documentId ||
!this.document ||
data.document_id !== this.documentId
)
return
if (this.networkActive) {
this.pendingIncomingUpdate = data
return
}
// If modified timestamp of the incoming update is the same as the last local save,
// we assume this update is from our own save and dont notify
const incomingModified = data.modified
if (
incomingModified &&
this.lastLocalSaveModified &&
incomingModified === this.lastLocalSaveModified
) {
this.lastLocalSaveModified = null
return
}
this.lastLocalSaveModified = null
if (this.openDocumentService.isDirty(this.document)) {
this.showIncomingUpdateModal(data.modified)
} else {
this.docChangeNotifier.next(this.documentId)
this.loadDocument(this.documentId, true)
this.toastService.showInfo(
$localize`Document reloaded with latest changes.`
)
}
}
private reloadRemoteVersion() {
if (!this.documentId) return
this.closeIncomingUpdateModal()
this.docChangeNotifier.next(this.documentId)
this.loadDocument(this.documentId, true)
this.toastService.showInfo($localize`Document reloaded.`)
}
ngOnInit(): void {
this.setZoom(
this.settings.get(SETTINGS_KEYS.PDF_VIEWER_ZOOM_SETTING) as PdfZoomScale
@@ -648,6 +758,11 @@ export class DocumentDetailComponent
this.getCustomFields()
this.websocketStatusService
.onDocumentUpdated()
.pipe(takeUntil(this.unsubscribeNotifier))
.subscribe((data) => this.handleIncomingDocumentUpdated(data))
this.route.paramMap
.pipe(
filter(
@@ -1033,6 +1148,7 @@ export class DocumentDetailComponent
)
.subscribe({
next: (doc) => {
this.closeIncomingUpdateModal()
Object.assign(this.document, doc)
doc['permissions_form'] = {
owner: doc.owner,
@@ -1079,6 +1195,8 @@ export class DocumentDetailComponent
.pipe(first())
.subscribe({
next: (docValues) => {
this.closeIncomingUpdateModal()
this.lastLocalSaveModified = docValues.modified ?? null
// in case data changed while saving eg removing inbox_tags
this.documentForm.patchValue(docValues)
const newValues = Object.assign({}, this.documentForm.value)
@@ -1093,16 +1211,19 @@ export class DocumentDetailComponent
this.networkActive = false
this.error = null
if (close) {
this.pendingIncomingUpdate = null
this.close(() =>
this.openDocumentService.refreshDocument(this.documentId)
)
} else {
this.openDocumentService.refreshDocument(this.documentId)
this.flushPendingIncomingUpdate()
}
this.savedViewService.maybeRefreshDocumentCounts()
},
error: (error) => {
this.networkActive = false
this.lastLocalSaveModified = null
const canEdit =
this.permissionsService.currentUserHasObjectPermissions(
PermissionAction.Change,
@@ -1122,6 +1243,7 @@ export class DocumentDetailComponent
error
)
}
this.flushPendingIncomingUpdate()
},
})
}
@@ -1158,8 +1280,11 @@ export class DocumentDetailComponent
.pipe(first())
.subscribe({
next: ({ updateResult, nextDocId, closeResult }) => {
this.closeIncomingUpdateModal()
this.error = null
this.networkActive = false
this.pendingIncomingUpdate = null
this.lastLocalSaveModified = null
if (closeResult && updateResult && nextDocId) {
this.router.navigate(['documents', nextDocId])
this.titleInput?.focus()
@@ -1167,8 +1292,10 @@ export class DocumentDetailComponent
},
error: (error) => {
this.networkActive = false
this.lastLocalSaveModified = null
this.error = error.error
this.toastService.showError($localize`Error saving document`, error)
this.flushPendingIncomingUpdate()
},
})
}
@@ -1254,7 +1381,7 @@ export class DocumentDetailComponent
.subscribe({
next: () => {
this.toastService.showInfo(
$localize`Reprocess operation for "${this.document.title}" will begin in the background. Close and re-open or reload this document after the operation has completed to see new content.`
$localize`Reprocess operation for "${this.document.title}" will begin in the background.`
)
if (modal) {
modal.close()

View File

@@ -830,7 +830,7 @@ export class BulkEditorComponent
})
const rotateDialog = modal.componentInstance as RotateConfirmDialogComponent
rotateDialog.title = $localize`Rotate confirm`
rotateDialog.messageBold = $localize`This operation will permanently rotate the original version of ${this.list.selected.size} document(s).`
rotateDialog.messageBold = $localize`This operation will add rotated versions of the ${this.list.selected.size} document(s).`
rotateDialog.btnClass = 'btn-danger'
rotateDialog.btnCaption = $localize`Proceed`
rotateDialog.documentID = Array.from(this.list.selected)[0]

View File

@@ -104,11 +104,9 @@
}
}
<div *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.SavedView }">
@if (list.activeSavedViewId) {
<button ngbDropdownItem (click)="saveViewConfig()" [disabled]="!savedViewIsModified" i18n>Save "{{list.activeSavedViewTitle}}"</button>
}
</div>
@if (list.activeSavedViewId && activeSavedViewCanChange) {
<button ngbDropdownItem (click)="saveViewConfig()" [disabled]="!savedViewIsModified" i18n>Save "{{list.activeSavedViewTitle}}"</button>
}
<button ngbDropdownItem (click)="saveViewConfigAs()" *pngxIfPermissions="{ action: PermissionAction.Add, type: PermissionType.SavedView }" i18n>Save as...</button>
<a ngbDropdownItem routerLink="/savedviews" *pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.SavedView }" i18n>All saved views</a>
</div>

View File

@@ -168,6 +168,10 @@ describe('DocumentListComponent', () => {
)
})
it('should not allow changing a saved view when none is active', () => {
expect(component.activeSavedViewCanChange).toBeFalsy()
})
it('should determine if filtered, support reset', () => {
fixture.detectChanges()
documentListService.setFilterRules([
@@ -299,6 +303,19 @@ describe('DocumentListComponent', () => {
expect(setCountSpy).toHaveBeenCalledWith(expect.any(Object), 3)
})
it('should reset active saved view when loading unknown view config', () => {
component['activeSavedView'] = { id: 1 } as SavedView
const activateSpy = jest.spyOn(documentListService, 'activateSavedView')
const reloadSpy = jest.spyOn(documentListService, 'reload')
jest.spyOn(savedViewService, 'getCached').mockReturnValue(of(null))
component.loadViewConfig(10)
expect(component['activeSavedView']).toBeNull()
expect(activateSpy).not.toHaveBeenCalled()
expect(reloadSpy).not.toHaveBeenCalled()
})
it('should support 3 different display modes', () => {
jest.spyOn(documentListService, 'documents', 'get').mockReturnValue(docs)
fixture.detectChanges()
@@ -466,7 +483,7 @@ describe('DocumentListComponent', () => {
})
it('should handle error on view saving', () => {
component.list.activateSavedView({
const view: SavedView = {
id: 10,
name: 'Saved View 10',
sort_field: 'added',
@@ -477,7 +494,16 @@ describe('DocumentListComponent', () => {
value: '20',
},
],
})
}
jest.spyOn(savedViewService, 'getCached').mockReturnValue(of(view))
const queryParams = { view: view.id.toString() }
jest
.spyOn(activatedRoute, 'queryParamMap', 'get')
.mockReturnValue(of(convertToParamMap(queryParams)))
activatedRoute.snapshot.queryParams = queryParams
router.routerState.snapshot.url = '/view/10/'
fixture.detectChanges()
const toastErrorSpy = jest.spyOn(toastService, 'showError')
jest
.spyOn(savedViewService, 'patch')
@@ -489,6 +515,40 @@ describe('DocumentListComponent', () => {
)
})
it('should not save a view without object change permissions', () => {
const view: SavedView = {
id: 10,
name: 'Saved View 10',
sort_field: 'added',
sort_reverse: true,
filter_rules: [
{
rule_type: FILTER_HAS_TAGS_ANY,
value: '20',
},
],
owner: 999,
user_can_change: false,
}
jest.spyOn(savedViewService, 'getCached').mockReturnValue(of(view))
jest
.spyOn(permissionService, 'currentUserHasObjectPermissions')
.mockReturnValue(false)
const queryParams = { view: view.id.toString() }
jest
.spyOn(activatedRoute, 'queryParamMap', 'get')
.mockReturnValue(of(convertToParamMap(queryParams)))
activatedRoute.snapshot.queryParams = queryParams
router.routerState.snapshot.url = '/view/10/'
fixture.detectChanges()
const patchSpy = jest.spyOn(savedViewService, 'patch')
component.saveViewConfig()
expect(patchSpy).not.toHaveBeenCalled()
})
it('should support edited view saving as', () => {
const view: SavedView = {
id: 10,
@@ -520,19 +580,105 @@ describe('DocumentListComponent', () => {
const modalSpy = jest.spyOn(modalService, 'open')
const toastSpy = jest.spyOn(toastService, 'showInfo')
const savedViewServiceCreate = jest.spyOn(savedViewService, 'create')
jest
.spyOn(savedViewService, 'dashboardViews', 'get')
.mockReturnValue([{ id: 77 } as SavedView])
jest
.spyOn(savedViewService, 'sidebarViews', 'get')
.mockReturnValue([{ id: 88 } as SavedView])
const updateVisibilitySpy = jest
.spyOn(settingsService, 'updateSavedViewsVisibility')
.mockReturnValue(of({ success: true }))
savedViewServiceCreate.mockReturnValueOnce(of(modifiedView))
component.saveViewConfigAs()
const modalCloseSpy = jest.spyOn(openModal, 'close')
const permissions = {
owner: 5,
set_permissions: {
view: {
users: [4],
groups: [3],
},
change: {
users: [2],
groups: [1],
},
},
}
openModal.componentInstance.saveClicked.next({
name: 'Foo Bar',
showOnDashboard: true,
showInSideBar: true,
permissions_form: permissions,
})
expect(savedViewServiceCreate).toHaveBeenCalled()
expect(savedViewServiceCreate).toHaveBeenCalledWith(
expect.objectContaining({
name: 'Foo Bar',
owner: permissions.owner,
set_permissions: permissions.set_permissions,
})
)
expect(updateVisibilitySpy).toHaveBeenCalledWith(
expect.arrayContaining([77, modifiedView.id]),
expect.arrayContaining([88, modifiedView.id])
)
expect(modalSpy).toHaveBeenCalled()
expect(toastSpy).toHaveBeenCalled()
expect(modalCloseSpy).toHaveBeenCalled()
})
it('should show error when visibility update fails after creating a view', () => {
const view: SavedView = {
id: 10,
name: 'Saved View 10',
sort_field: 'added',
sort_reverse: true,
filter_rules: [
{
rule_type: FILTER_HAS_TAGS_ANY,
value: '20',
},
],
}
jest.spyOn(savedViewService, 'getCached').mockReturnValue(of(view))
const queryParams = { view: view.id.toString() }
jest
.spyOn(activatedRoute, 'queryParamMap', 'get')
.mockReturnValue(of(convertToParamMap(queryParams)))
activatedRoute.snapshot.queryParams = queryParams
router.routerState.snapshot.url = '/view/10/'
fixture.detectChanges()
let openModal: NgbModalRef
modalService.activeInstances.subscribe((modal) => (openModal = modal[0]))
jest
.spyOn(savedViewService, 'create')
.mockReturnValueOnce(of({ ...view, id: 42, name: 'Foo Bar' }))
jest.spyOn(savedViewService, 'dashboardViews', 'get').mockReturnValue([])
jest.spyOn(savedViewService, 'sidebarViews', 'get').mockReturnValue([])
jest
.spyOn(settingsService, 'updateSavedViewsVisibility')
.mockReturnValueOnce(
throwError(() => new Error('unable to save visibility settings'))
)
const toastErrorSpy = jest.spyOn(toastService, 'showError')
component.saveViewConfigAs()
const modalCloseSpy = jest.spyOn(openModal, 'close')
openModal.componentInstance.saveClicked.next({
name: 'Foo Bar',
show_on_dashboard: true,
show_in_sidebar: true,
showOnDashboard: true,
showInSideBar: false,
})
expect(savedViewServiceCreate).toHaveBeenCalled()
expect(modalSpy).toHaveBeenCalled()
expect(toastSpy).toHaveBeenCalled()
expect(modalCloseSpy).toHaveBeenCalled()
expect(toastErrorSpy).toHaveBeenCalledWith(
'View "Foo Bar" created successfully, but could not update visibility settings.',
expect.any(Error)
)
})
it('should handle error on edited view saving as', () => {
@@ -563,6 +709,10 @@ describe('DocumentListComponent', () => {
let openModal: NgbModalRef
modalService.activeInstances.subscribe((modal) => (openModal = modal[0]))
const updateVisibilitySpy = jest.spyOn(
settingsService,
'updateSavedViewsVisibility'
)
jest.spyOn(savedViewService, 'create').mockReturnValueOnce(
throwError(
() =>
@@ -575,9 +725,10 @@ describe('DocumentListComponent', () => {
openModal.componentInstance.saveClicked.next({
name: 'Foo Bar',
show_on_dashboard: true,
show_in_sidebar: true,
showOnDashboard: true,
showInSideBar: true,
})
expect(updateVisibilitySpy).not.toHaveBeenCalled()
expect(openModal.componentInstance.error).toEqual({ filter_rules: ['11'] })
})

View File

@@ -47,7 +47,10 @@ import { UsernamePipe } from 'src/app/pipes/username.pipe'
import { DocumentListViewService } from 'src/app/services/document-list-view.service'
import { HotKeyService } from 'src/app/services/hot-key.service'
import { OpenDocumentsService } from 'src/app/services/open-documents.service'
import { PermissionsService } from 'src/app/services/permissions.service'
import {
PermissionAction,
PermissionsService,
} from 'src/app/services/permissions.service'
import { SavedViewService } from 'src/app/services/rest/saved-view.service'
import { SettingsService } from 'src/app/services/settings.service'
import { ToastService } from 'src/app/services/toast.service'
@@ -148,12 +151,18 @@ export class DocumentListComponent
unmodifiedFilterRules: FilterRule[] = []
private unmodifiedSavedView: SavedView
private activeSavedView: SavedView | null = null
private unsubscribeNotifier: Subject<any> = new Subject()
get savedViewIsModified(): boolean {
if (!this.list.activeSavedViewId || !this.unmodifiedSavedView) return false
else {
if (
!this.list.activeSavedViewId ||
!this.unmodifiedSavedView ||
!this.activeSavedViewCanChange
) {
return false
} else {
return (
this.unmodifiedSavedView.sort_field !== this.list.sortField ||
this.unmodifiedSavedView.sort_reverse !== this.list.sortReverse ||
@@ -180,6 +189,16 @@ export class DocumentListComponent
}
}
get activeSavedViewCanChange(): boolean {
if (!this.activeSavedView) {
return false
}
return this.permissionService.currentUserHasObjectPermissions(
PermissionAction.Change,
this.activeSavedView
)
}
get isFiltered() {
return !!this.filterEditor?.rulesModified
}
@@ -264,11 +283,13 @@ export class DocumentListComponent
.pipe(takeUntil(this.unsubscribeNotifier))
.subscribe(({ view }) => {
if (!view) {
this.activeSavedView = null
this.router.navigate(['404'], {
replaceUrl: true,
})
return
}
this.activeSavedView = view
this.unmodifiedSavedView = view
this.list.activateSavedViewWithQueryParams(
view,
@@ -292,6 +313,7 @@ export class DocumentListComponent
// loading a saved view on /documents
this.loadViewConfig(parseInt(queryParams.get('view')))
} else {
this.activeSavedView = null
this.list.activateSavedView(null)
this.list.loadFromQueryParams(queryParams)
this.unmodifiedFilterRules = []
@@ -374,7 +396,7 @@ export class DocumentListComponent
}
saveViewConfig() {
if (this.list.activeSavedViewId != null) {
if (this.list.activeSavedViewId != null && this.activeSavedViewCanChange) {
let savedView: SavedView = {
id: this.list.activeSavedViewId,
filter_rules: this.list.filterRules,
@@ -388,6 +410,7 @@ export class DocumentListComponent
.pipe(first())
.subscribe({
next: (view) => {
this.activeSavedView = view
this.unmodifiedSavedView = view
this.toastService.showInfo(
$localize`View "${this.list.activeSavedViewTitle}" saved successfully.`
@@ -409,6 +432,11 @@ export class DocumentListComponent
.getCached(viewID)
.pipe(first())
.subscribe((view) => {
if (!view) {
this.activeSavedView = null
return
}
this.activeSavedView = view
this.unmodifiedSavedView = view
this.list.activateSavedView(view)
this.list.reload(() => {
@@ -426,24 +454,48 @@ export class DocumentListComponent
modal.componentInstance.buttonsEnabled = false
let savedView: SavedView = {
name: formValue.name,
show_on_dashboard: formValue.showOnDashboard,
show_in_sidebar: formValue.showInSideBar,
filter_rules: this.list.filterRules,
sort_reverse: this.list.sortReverse,
sort_field: this.list.sortField,
display_mode: this.list.displayMode,
display_fields: this.activeDisplayFields,
}
const permissions = formValue.permissions_form
if (permissions) {
if (permissions.owner !== null && permissions.owner !== undefined) {
savedView.owner = permissions.owner
}
if (permissions.set_permissions) {
savedView['set_permissions'] = permissions.set_permissions
}
}
this.savedViewService
.create(savedView)
.pipe(first())
.subscribe({
next: () => {
modal.close()
this.toastService.showInfo(
$localize`View "${savedView.name}" created successfully.`
next: (createdView) => {
this.saveCreatedViewVisibility(
createdView,
formValue.showOnDashboard,
formValue.showInSideBar
)
.pipe(first())
.subscribe({
next: () => {
modal.close()
this.toastService.showInfo(
$localize`View "${savedView.name}" created successfully.`
)
},
error: (error) => {
modal.close()
this.toastService.showError(
$localize`View "${savedView.name}" created successfully, but could not update visibility settings.`,
error
)
},
})
},
error: (httpError) => {
let error = httpError.error
@@ -457,6 +509,28 @@ export class DocumentListComponent
})
}
private saveCreatedViewVisibility(
createdView: SavedView,
showOnDashboard: boolean,
showInSideBar: boolean
) {
const dashboardViewIds = this.savedViewService.dashboardViews.map(
(v) => v.id
)
const sidebarViewIds = this.savedViewService.sidebarViews.map((v) => v.id)
if (showOnDashboard) {
dashboardViewIds.push(createdView.id)
}
if (showInSideBar) {
sidebarViewIds.push(createdView.id)
}
return this.settingsService.updateSavedViewsVisibility(
dashboardViewIds,
sidebarViewIds
)
}
openDocumentDetail(document: Document | number) {
this.router.navigate([
'documents',

View File

@@ -8,6 +8,7 @@
<pngx-input-text i18n-title title="Name" formControlName="name" [error]="error?.name" autocomplete="off"></pngx-input-text>
<pngx-input-check i18n-title title="Show in sidebar" formControlName="showInSideBar"></pngx-input-check>
<pngx-input-check i18n-title title="Show on dashboard" formControlName="showOnDashboard"></pngx-input-check>
<pngx-permissions-form accordion="true" formControlName="permissions_form"></pngx-permissions-form>
@if (error?.filter_rules) {
<div class="alert alert-danger" role="alert">
<h6 class="alert-heading" i18n>Filter rules error occurred while saving this view</h6>

View File

@@ -7,7 +7,13 @@ import {
import { FormsModule, ReactiveFormsModule } from '@angular/forms'
import { By } from '@angular/platform-browser'
import { NgbActiveModal, NgbModalModule } from '@ng-bootstrap/ng-bootstrap'
import { of } from 'rxjs'
import { GroupService } from 'src/app/services/rest/group.service'
import { UserService } from 'src/app/services/rest/user.service'
import { CheckComponent } from '../../common/input/check/check.component'
import { PermissionsFormComponent } from '../../common/input/permissions/permissions-form/permissions-form.component'
import { PermissionsGroupComponent } from '../../common/input/permissions/permissions-group/permissions-group.component'
import { PermissionsUserComponent } from '../../common/input/permissions/permissions-user/permissions-user.component'
import { TextComponent } from '../../common/input/text/text.component'
import { SaveViewConfigDialogComponent } from './save-view-config-dialog.component'
@@ -18,7 +24,21 @@ describe('SaveViewConfigDialogComponent', () => {
beforeEach(fakeAsync(() => {
TestBed.configureTestingModule({
providers: [NgbActiveModal],
providers: [
NgbActiveModal,
{
provide: UserService,
useValue: {
listAll: () => of({ results: [] }),
},
},
{
provide: GroupService,
useValue: {
listAll: () => of({ results: [] }),
},
},
],
imports: [
NgbModalModule,
FormsModule,
@@ -26,6 +46,9 @@ describe('SaveViewConfigDialogComponent', () => {
SaveViewConfigDialogComponent,
TextComponent,
CheckComponent,
PermissionsFormComponent,
PermissionsUserComponent,
PermissionsGroupComponent,
],
}).compileComponents()
@@ -81,6 +104,26 @@ describe('SaveViewConfigDialogComponent', () => {
})
})
it('should support permissions input', () => {
const permissions = {
owner: 10,
set_permissions: {
view: { users: [2], groups: [3] },
change: { users: [4], groups: [5] },
},
}
let result
component.saveClicked.subscribe((saveResult) => (result = saveResult))
component.saveViewConfigForm.get('permissions_form').patchValue(permissions)
component.save()
expect(result).toEqual({
name: '',
showInSideBar: false,
showOnDashboard: false,
permissions_form: permissions,
})
})
it('should support default name', () => {
const saveClickedSpy = jest.spyOn(component.saveClicked, 'emit')
const modalCloseSpy = jest.spyOn(modal, 'close')

View File

@@ -13,14 +13,22 @@ import {
ReactiveFormsModule,
} from '@angular/forms'
import { NgbActiveModal } from '@ng-bootstrap/ng-bootstrap'
import { User } from 'src/app/data/user'
import { CheckComponent } from '../../common/input/check/check.component'
import { PermissionsFormComponent } from '../../common/input/permissions/permissions-form/permissions-form.component'
import { TextComponent } from '../../common/input/text/text.component'
@Component({
selector: 'pngx-save-view-config-dialog',
templateUrl: './save-view-config-dialog.component.html',
styleUrls: ['./save-view-config-dialog.component.scss'],
imports: [CheckComponent, TextComponent, FormsModule, ReactiveFormsModule],
imports: [
CheckComponent,
TextComponent,
PermissionsFormComponent,
FormsModule,
ReactiveFormsModule,
],
})
export class SaveViewConfigDialogComponent implements OnInit {
private modal = inject(NgbActiveModal)
@@ -36,6 +44,8 @@ export class SaveViewConfigDialogComponent implements OnInit {
closeEnabled = false
users: User[]
_defaultName = ''
get defaultName() {
@@ -52,6 +62,7 @@ export class SaveViewConfigDialogComponent implements OnInit {
name: new FormControl(''),
showInSideBar: new FormControl(false),
showOnDashboard: new FormControl(false),
permissions_form: new FormControl(null),
})
ngOnInit(): void {
@@ -62,7 +73,16 @@ export class SaveViewConfigDialogComponent implements OnInit {
}
save() {
this.saveClicked.emit(this.saveViewConfigForm.value)
const formValue = this.saveViewConfigForm.value
const saveViewConfig = {
name: formValue.name,
showInSideBar: formValue.showInSideBar,
showOnDashboard: formValue.showOnDashboard,
}
if (formValue.permissions_form) {
saveViewConfig['permissions_form'] = formValue.permissions_form
}
this.saveClicked.emit(saveViewConfig)
}
cancel() {

View File

@@ -25,15 +25,23 @@
</div>
</div>
<div class="col-auto">
<label class="form-label" for="name_{{view.id}}" i18n>Actions</label>
<pngx-confirm-button
label="Delete"
i18n-label
(confirm)="deleteSavedView(view)"
*pngxIfPermissions="{ action: PermissionAction.Delete, type: PermissionType.SavedView }"
buttonClasses="btn-sm btn-outline-danger form-control"
iconName="trash">
</pngx-confirm-button>
@if (canDeleteSavedView(view)) {
<label class="form-label" for="name_{{view.id}}" i18n>Actions</label>
<button
class="btn btn-sm btn-outline-secondary form-control mb-2"
type="button"
(click)="editPermissions(view)"
*pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.SavedView }"
i18n><i-bs class="me-1" name="person-fill-lock"></i-bs>Permissions</button>
<pngx-confirm-button
label="Delete"
i18n-label
(confirm)="deleteSavedView(view)"
*pngxIfPermissions="{ action: PermissionAction.Delete, type: PermissionType.SavedView }"
buttonClasses="btn-sm btn-outline-danger form-control"
iconName="trash">
</pngx-confirm-button>
}
</div>
</div>
<div class="row">

View File

@@ -3,16 +3,16 @@ import { provideHttpClient, withInterceptorsFromDi } from '@angular/common/http'
import { provideHttpClientTesting } from '@angular/common/http/testing'
import { ComponentFixture, TestBed } from '@angular/core/testing'
import { FormsModule, ReactiveFormsModule } from '@angular/forms'
import { By } from '@angular/platform-browser'
import { NgbModule } from '@ng-bootstrap/ng-bootstrap'
import { NgbModal, NgbModule } from '@ng-bootstrap/ng-bootstrap'
import { NgxBootstrapIconsModule, allIcons } from 'ngx-bootstrap-icons'
import { of, throwError } from 'rxjs'
import { Subject, of, throwError } from 'rxjs'
import { SavedView } from 'src/app/data/saved-view'
import { IfPermissionsDirective } from 'src/app/directives/if-permissions.directive'
import { PermissionsGuard } from 'src/app/guards/permissions.guard'
import { PermissionsService } from 'src/app/services/permissions.service'
import { CustomFieldsService } from 'src/app/services/rest/custom-fields.service'
import { SavedViewService } from 'src/app/services/rest/saved-view.service'
import { SettingsService } from 'src/app/services/settings.service'
import { ToastService } from 'src/app/services/toast.service'
import { ConfirmButtonComponent } from '../../common/confirm-button/confirm-button.component'
import { CheckComponent } from '../../common/input/check/check.component'
@@ -32,7 +32,9 @@ describe('SavedViewsComponent', () => {
let component: SavedViewsComponent
let fixture: ComponentFixture<SavedViewsComponent>
let savedViewService: SavedViewService
let settingsService: SettingsService
let toastService: ToastService
let modalService: NgbModal
beforeEach(async () => {
TestBed.configureTestingModule({
@@ -57,6 +59,8 @@ describe('SavedViewsComponent', () => {
provide: PermissionsService,
useValue: {
currentUserCan: () => true,
currentUserHasObjectPermissions: () => true,
currentUserOwnsObject: () => true,
},
},
{
@@ -77,11 +81,13 @@ describe('SavedViewsComponent', () => {
}).compileComponents()
savedViewService = TestBed.inject(SavedViewService)
settingsService = TestBed.inject(SettingsService)
toastService = TestBed.inject(ToastService)
modalService = TestBed.inject(NgbModal)
fixture = TestBed.createComponent(SavedViewsComponent)
component = fixture.componentInstance
jest.spyOn(savedViewService, 'listAll').mockReturnValue(
jest.spyOn(savedViewService, 'list').mockReturnValue(
of({
all: savedViews.map((v) => v.id),
count: savedViews.length,
@@ -94,14 +100,13 @@ describe('SavedViewsComponent', () => {
it('should support save saved views, show error', () => {
const toastErrorSpy = jest.spyOn(toastService, 'showError')
const toastSpy = jest.spyOn(toastService, 'show')
const savedViewPatchSpy = jest.spyOn(savedViewService, 'patchMany')
const toggle = fixture.debugElement.query(
By.css('.form-check.form-switch input')
)
toggle.nativeElement.checked = true
toggle.nativeElement.dispatchEvent(new Event('change'))
const control = component.savedViewsForm
.get('savedViews')
.get(savedViews[0].id.toString())
.get('name')
control.setValue(`${savedViews[0].name}-changed`)
control.markAsDirty()
// saved views error first
savedViewPatchSpy.mockReturnValueOnce(
@@ -110,12 +115,13 @@ describe('SavedViewsComponent', () => {
component.save()
expect(toastErrorSpy).toHaveBeenCalled()
expect(savedViewPatchSpy).toHaveBeenCalled()
toastSpy.mockClear()
toastErrorSpy.mockClear()
savedViewPatchSpy.mockClear()
// succeed saved views
savedViewPatchSpy.mockReturnValueOnce(of(savedViews as SavedView[]))
control.setValue(savedViews[0].name)
control.markAsDirty()
component.save()
expect(toastErrorSpy).not.toHaveBeenCalled()
expect(savedViewPatchSpy).toHaveBeenCalled()
@@ -127,26 +133,65 @@ describe('SavedViewsComponent', () => {
expect(patchSpy).not.toHaveBeenCalled()
const view = savedViews[0]
const toggle = fixture.debugElement.query(
By.css('.form-check.form-switch input')
)
toggle.nativeElement.checked = true
toggle.nativeElement.dispatchEvent(new Event('change'))
// register change
component.savedViewsForm.get('savedViews').get(view.id.toString()).value[
'show_on_dashboard'
] = !view.show_on_dashboard
component.savedViewsForm
.get('savedViews')
.get(view.id.toString())
.get('name')
.setValue('changed-view-name')
component.savedViewsForm
.get('savedViews')
.get(view.id.toString())
.get('name')
.markAsDirty()
fixture.detectChanges()
component.save()
expect(patchSpy).toHaveBeenCalledWith([
{
id: view.id,
name: view.name,
show_in_sidebar: view.show_in_sidebar,
show_on_dashboard: !view.show_on_dashboard,
},
])
expect(patchSpy).toHaveBeenCalled()
const patchBody = patchSpy.mock.calls[0][0][0]
expect(patchBody).toMatchObject({
id: view.id,
name: 'changed-view-name',
})
expect(patchBody.show_on_dashboard).toBeUndefined()
expect(patchBody.show_in_sidebar).toBeUndefined()
})
it('should persist visibility changes to user settings', () => {
const patchSpy = jest.spyOn(savedViewService, 'patchMany')
const updateVisibilitySpy = jest
.spyOn(settingsService, 'updateSavedViewsVisibility')
.mockReturnValue(of({ success: true }))
const dashboardControl = component.savedViewsForm
.get('savedViews')
.get(savedViews[0].id.toString())
.get('show_on_dashboard')
dashboardControl.setValue(false)
dashboardControl.markAsDirty()
component.save()
expect(patchSpy).not.toHaveBeenCalled()
expect(updateVisibilitySpy).toHaveBeenCalledWith([], [savedViews[0].id])
})
it('should skip model updates for views that cannot be edited', () => {
const patchSpy = jest.spyOn(savedViewService, 'patchMany')
const updateVisibilitySpy = jest.spyOn(
settingsService,
'updateSavedViewsVisibility'
)
const nameControl = component.savedViewsForm
.get('savedViews')
.get(savedViews[0].id.toString())
.get('name')
nameControl.disable()
component.save()
expect(patchSpy).not.toHaveBeenCalled()
expect(updateVisibilitySpy).not.toHaveBeenCalled()
})
it('should support delete saved view', () => {
@@ -162,14 +207,55 @@ describe('SavedViewsComponent', () => {
it('should support reset', () => {
const view = savedViews[0]
component.savedViewsForm.get('savedViews').get(view.id.toString()).value[
'show_on_dashboard'
] = !view.show_on_dashboard
component.savedViewsForm
.get('savedViews')
.get(view.id.toString())
.get('show_on_dashboard')
.setValue(!view.show_on_dashboard)
component.reset()
expect(
component.savedViewsForm.get('savedViews').get(view.id.toString()).value[
'show_on_dashboard'
]
component.savedViewsForm
.get('savedViews')
.get(view.id.toString())
.get('show_on_dashboard').value
).toEqual(view.show_on_dashboard)
})
it('should support editing permissions', () => {
const confirmClicked = new Subject<any>()
const modalRef = {
componentInstance: {
confirmClicked,
buttonsEnabled: true,
},
close: jest.fn(),
} as any
jest.spyOn(modalService, 'open').mockReturnValue(modalRef)
const patchSpy = jest.spyOn(savedViewService, 'patch')
patchSpy.mockReturnValue(of(savedViews[0] as SavedView))
component.editPermissions(savedViews[0] as SavedView)
confirmClicked.next({
permissions: {
owner: 1,
set_permissions: {
view: { users: [2], groups: [] },
change: { users: [], groups: [3] },
},
},
merge: true,
})
expect(patchSpy).toHaveBeenCalledWith(
expect.objectContaining({
id: savedViews[0].id,
owner: 1,
set_permissions: {
view: { users: [2], groups: [] },
change: { users: [], groups: [3] },
},
})
)
expect(modalRef.close).toHaveBeenCalled()
})
})

View File

@@ -6,11 +6,18 @@ import {
FormsModule,
ReactiveFormsModule,
} from '@angular/forms'
import { NgbModal } from '@ng-bootstrap/ng-bootstrap'
import { dirtyCheck } from '@ngneat/dirty-check-forms'
import { BehaviorSubject, Observable, takeUntil } from 'rxjs'
import { NgxBootstrapIconsModule } from 'ngx-bootstrap-icons'
import { BehaviorSubject, Observable, of, switchMap, takeUntil } from 'rxjs'
import { PermissionsDialogComponent } from 'src/app/components/common/permissions-dialog/permissions-dialog.component'
import { DisplayMode } from 'src/app/data/document'
import { SavedView } from 'src/app/data/saved-view'
import { IfPermissionsDirective } from 'src/app/directives/if-permissions.directive'
import {
PermissionAction,
PermissionsService,
} from 'src/app/services/permissions.service'
import { SavedViewService } from 'src/app/services/rest/saved-view.service'
import { SettingsService } from 'src/app/services/settings.service'
import { ToastService } from 'src/app/services/toast.service'
@@ -34,15 +41,18 @@ import { LoadingComponentWithPermissions } from '../../loading-component/loading
FormsModule,
ReactiveFormsModule,
AsyncPipe,
NgxBootstrapIconsModule,
],
})
export class SavedViewsComponent
extends LoadingComponentWithPermissions
implements OnInit, OnDestroy
{
private savedViewService = inject(SavedViewService)
private settings = inject(SettingsService)
private toastService = inject(ToastService)
private readonly savedViewService = inject(SavedViewService)
private readonly permissionsService = inject(PermissionsService)
private readonly settings = inject(SettingsService)
private readonly toastService = inject(ToastService)
private readonly modalService = inject(NgbModal)
DisplayMode = DisplayMode
@@ -65,11 +75,17 @@ export class SavedViewsComponent
}
ngOnInit(): void {
this.reloadViews()
}
private reloadViews(): void {
this.loading = true
this.savedViewService.listAll().subscribe((r) => {
this.savedViews = r.results
this.initialize()
})
this.savedViewService
.list(null, null, null, false, { full_perms: true })
.subscribe((r) => {
this.savedViews = r.results
this.initialize()
})
}
ngOnDestroy(): void {
@@ -95,16 +111,20 @@ export class SavedViewsComponent
display_mode: view.display_mode,
display_fields: view.display_fields,
}
const canEdit = this.canEditSavedView(view)
this.savedViewsGroup.addControl(
view.id.toString(),
new FormGroup({
id: new FormControl(null),
name: new FormControl(null),
show_on_dashboard: new FormControl(null),
show_in_sidebar: new FormControl(null),
page_size: new FormControl(null),
display_mode: new FormControl(null),
display_fields: new FormControl([]),
id: new FormControl({ value: null, disabled: !canEdit }),
name: new FormControl({ value: null, disabled: !canEdit }),
show_on_dashboard: new FormControl({
value: null,
disabled: false,
}),
show_in_sidebar: new FormControl({ value: null, disabled: false }),
page_size: new FormControl({ value: null, disabled: !canEdit }),
display_mode: new FormControl({ value: null, disabled: !canEdit }),
display_fields: new FormControl({ value: [], disabled: !canEdit }),
})
)
}
@@ -133,10 +153,7 @@ export class SavedViewsComponent
$localize`Saved view "${savedView.name}" deleted.`
)
this.savedViewService.clearCache()
this.savedViewService.listAll().subscribe((r) => {
this.savedViews = r.results
this.initialize()
})
this.reloadViews()
})
}
@@ -145,26 +162,120 @@ export class SavedViewsComponent
}
public save() {
// only patch views that have actually changed
// Save only changed views, then save the visibility changes into user settings.
const groups = Object.values(this.savedViewsGroup.controls) as FormGroup[]
const visibilityChanged = groups.some(
(group) =>
group.get('show_on_dashboard')?.dirty ||
group.get('show_in_sidebar')?.dirty
)
const changed: SavedView[] = []
Object.values(this.savedViewsGroup.controls)
.filter((g: FormGroup) => !g.pristine)
.forEach((group: FormGroup) => {
changed.push(group.value)
})
const dashboardVisibleIds: number[] = []
const sidebarVisibleIds: number[] = []
groups.forEach((group) => {
const value = group.getRawValue()
if (value.show_on_dashboard) {
dashboardVisibleIds.push(value.id)
}
if (value.show_in_sidebar) {
sidebarVisibleIds.push(value.id)
}
// Would be fine to send, but no longer stored on the model
delete value.show_on_dashboard
delete value.show_in_sidebar
if (!group.get('name')?.enabled) {
// Quick check for user doesn't have permissions, then bail
return
}
const modelFieldsChanged =
group.get('name')?.dirty ||
group.get('page_size')?.dirty ||
group.get('display_mode')?.dirty ||
group.get('display_fields')?.dirty
if (!modelFieldsChanged) {
return
}
changed.push(value)
})
if (!changed.length && !visibilityChanged) {
return
}
let saveOperation = of([])
if (changed.length) {
this.savedViewService.patchMany(changed).subscribe({
saveOperation = saveOperation.pipe(
switchMap(() => this.savedViewService.patchMany(changed))
)
}
if (visibilityChanged) {
saveOperation = saveOperation.pipe(
switchMap(() =>
this.settings.updateSavedViewsVisibility(
dashboardVisibleIds,
sidebarVisibleIds
)
)
)
}
saveOperation.subscribe({
next: () => {
this.toastService.showInfo($localize`Views saved successfully.`)
this.savedViewService.clearCache()
this.reloadViews()
},
error: (error) => {
this.toastService.showError($localize`Error while saving views.`, error)
},
})
}
public canEditSavedView(view: SavedView): boolean {
return this.permissionsService.currentUserHasObjectPermissions(
PermissionAction.Change,
view
)
}
public canDeleteSavedView(view: SavedView): boolean {
return this.permissionsService.currentUserOwnsObject(view)
}
public editPermissions(savedView: SavedView): void {
const modal = this.modalService.open(PermissionsDialogComponent, {
backdrop: 'static',
})
const dialog = modal.componentInstance as PermissionsDialogComponent
dialog.object = savedView
dialog.note = $localize`Note: Sharing saved views does not share the underlying documents.`
modal.componentInstance.confirmClicked.subscribe(({ permissions }) => {
modal.componentInstance.buttonsEnabled = false
const view = {
id: savedView.id,
owner: permissions.owner,
}
view['set_permissions'] = permissions.set_permissions
this.savedViewService.patch(view as SavedView).subscribe({
next: () => {
this.toastService.showInfo($localize`Views saved successfully.`)
this.store.next(this.savedViewsForm.value)
this.toastService.showInfo($localize`Permissions updated`)
modal.close()
this.reloadViews()
},
error: (error) => {
this.toastService.showError(
$localize`Error while saving views.`,
$localize`Error updating permissions`,
error
)
},
})
}
})
}
}

View File

@@ -128,15 +128,15 @@ export interface Document extends ObjectWithPermissions {
checksum?: string
// UTC
created?: Date
created?: string // ISO string
modified?: Date
modified?: string // ISO string
added?: Date
added?: string // ISO string
mime_type?: string
deleted_at?: Date
deleted_at?: string // ISO string
original_file_name?: string

View File

@@ -62,6 +62,10 @@ export const SETTINGS_KEYS = {
'general-settings:update-checking:backend-setting',
SAVED_VIEWS_WARN_ON_UNSAVED_CHANGE:
'general-settings:saved-views:warn-on-unsaved-change',
DASHBOARD_VIEWS_VISIBLE_IDS:
'general-settings:saved-views:dashboard-views-visible-ids',
SIDEBAR_VIEWS_VISIBLE_IDS:
'general-settings:saved-views:sidebar-views-visible-ids',
DASHBOARD_VIEWS_SORT_ORDER:
'general-settings:saved-views:dashboard-views-sort-order',
SIDEBAR_VIEWS_SORT_ORDER:
@@ -248,6 +252,16 @@ export const SETTINGS: UiSetting[] = [
type: 'array',
default: [],
},
{
key: SETTINGS_KEYS.DASHBOARD_VIEWS_VISIBLE_IDS,
type: 'array',
default: [],
},
{
key: SETTINGS_KEYS.SIDEBAR_VIEWS_VISIBLE_IDS,
type: 'array',
default: [],
},
{
key: SETTINGS_KEYS.DASHBOARD_VIEWS_SORT_ORDER,
type: 'array',

View File

@@ -0,0 +1,7 @@
export interface WebsocketDocumentUpdatedMessage {
document_id: number
modified: string
owner_id?: number
users_can_view?: number[]
groups_can_view?: number[]
}

View File

@@ -57,6 +57,11 @@ describe(`Additional service tests for SavedViewService`, () => {
let settingsService
it('should retrieve saved views and sort them', () => {
jest.spyOn(settingsService, 'get').mockImplementation((key) => {
if (key === SETTINGS_KEYS.DASHBOARD_VIEWS_VISIBLE_IDS) return [1, 2, 3]
if (key === SETTINGS_KEYS.SIDEBAR_VIEWS_VISIBLE_IDS) return [1, 2, 3]
return []
})
service.reload()
const req = httpTestingController.expectOne(
`${environment.apiBaseUrl}${endpoint}/?page=1&page_size=100000`
@@ -93,7 +98,9 @@ describe(`Additional service tests for SavedViewService`, () => {
it('should sort dashboard views', () => {
service['savedViews'] = saved_views
jest.spyOn(settingsService, 'get').mockImplementation((key) => {
if (key === SETTINGS_KEYS.DASHBOARD_VIEWS_VISIBLE_IDS) return [1, 2, 3]
if (key === SETTINGS_KEYS.DASHBOARD_VIEWS_SORT_ORDER) return [3, 1, 2]
return []
})
expect(service.dashboardViews).toEqual([
saved_views[2],
@@ -102,10 +109,21 @@ describe(`Additional service tests for SavedViewService`, () => {
])
})
it('should use user-specific dashboard visibility when configured', () => {
service['savedViews'] = saved_views
jest.spyOn(settingsService, 'get').mockImplementation((key) => {
if (key === SETTINGS_KEYS.DASHBOARD_VIEWS_VISIBLE_IDS) return [4, 2]
if (key === SETTINGS_KEYS.DASHBOARD_VIEWS_SORT_ORDER) return []
})
expect(service.dashboardViews).toEqual([saved_views[1], saved_views[3]])
})
it('should sort sidebar views', () => {
service['savedViews'] = saved_views
jest.spyOn(settingsService, 'get').mockImplementation((key) => {
if (key === SETTINGS_KEYS.SIDEBAR_VIEWS_VISIBLE_IDS) return [1, 2, 3]
if (key === SETTINGS_KEYS.SIDEBAR_VIEWS_SORT_ORDER) return [3, 1, 2]
return []
})
expect(service.sidebarViews).toEqual([
saved_views[2],
@@ -114,6 +132,15 @@ describe(`Additional service tests for SavedViewService`, () => {
])
})
it('should use user-specific sidebar visibility when configured', () => {
service['savedViews'] = saved_views
jest.spyOn(settingsService, 'get').mockImplementation((key) => {
if (key === SETTINGS_KEYS.SIDEBAR_VIEWS_VISIBLE_IDS) return [4, 2]
if (key === SETTINGS_KEYS.SIDEBAR_VIEWS_SORT_ORDER) return []
})
expect(service.sidebarViews).toEqual([saved_views[1], saved_views[3]])
})
it('should treat empty display_fields as null', () => {
subscription = service
.patch({

View File

@@ -36,7 +36,9 @@ export class SavedViewService extends AbstractPaperlessService<SavedView> {
return super.list(page, pageSize, sortField, sortReverse, extraParams).pipe(
tap({
next: (r) => {
this.savedViews = r.results
const views = r.results.map((view) => this.withUserVisibility(view))
this.savedViews = views
r.results = views
this._loading = false
this.settingsService.dashboardIsEmpty =
this.dashboardViews.length === 0
@@ -65,8 +67,35 @@ export class SavedViewService extends AbstractPaperlessService<SavedView> {
return this.savedViews
}
private getVisibleViewIds(setting: string): number[] {
const configured = this.settingsService.get(setting)
return Array.isArray(configured) ? configured : []
}
private withUserVisibility(view: SavedView): SavedView {
return {
...view,
show_on_dashboard: this.isDashboardVisible(view),
show_in_sidebar: this.isSidebarVisible(view),
}
}
private isDashboardVisible(view: SavedView): boolean {
const visibleIds = this.getVisibleViewIds(
SETTINGS_KEYS.DASHBOARD_VIEWS_VISIBLE_IDS
)
return visibleIds.includes(view.id)
}
private isSidebarVisible(view: SavedView): boolean {
const visibleIds = this.getVisibleViewIds(
SETTINGS_KEYS.SIDEBAR_VIEWS_VISIBLE_IDS
)
return visibleIds.includes(view.id)
}
get sidebarViews(): SavedView[] {
const sidebarViews = this.savedViews.filter((v) => v.show_in_sidebar)
const sidebarViews = this.savedViews.filter((v) => this.isSidebarVisible(v))
const sorted: number[] = this.settingsService.get(
SETTINGS_KEYS.SIDEBAR_VIEWS_SORT_ORDER
@@ -81,7 +110,9 @@ export class SavedViewService extends AbstractPaperlessService<SavedView> {
}
get dashboardViews(): SavedView[] {
const dashboardViews = this.savedViews.filter((v) => v.show_on_dashboard)
const dashboardViews = this.savedViews.filter((v) =>
this.isDashboardVisible(v)
)
const sorted: number[] = this.settingsService.get(
SETTINGS_KEYS.DASHBOARD_VIEWS_SORT_ORDER

View File

@@ -320,7 +320,7 @@ describe('SettingsService', () => {
expect(req.request.method).toEqual('POST')
})
it('should update saved view sorting', () => {
it('should update saved view sorting and visibility', () => {
httpTestingController
.expectOne(`${environment.apiBaseUrl}ui_settings/`)
.flush(ui_settings)
@@ -341,6 +341,15 @@ describe('SettingsService', () => {
SETTINGS_KEYS.SIDEBAR_VIEWS_SORT_ORDER,
[1, 4]
)
settingsService.updateSavedViewsVisibility([1, 4], [4, 1])
expect(setSpy).toHaveBeenCalledWith(
SETTINGS_KEYS.DASHBOARD_VIEWS_VISIBLE_IDS,
[1, 4]
)
expect(setSpy).toHaveBeenCalledWith(
SETTINGS_KEYS.SIDEBAR_VIEWS_VISIBLE_IDS,
[4, 1]
)
httpTestingController
.expectOne(`${environment.apiBaseUrl}ui_settings/`)
.flush(ui_settings)

View File

@@ -699,4 +699,17 @@ export class SettingsService {
])
return this.storeSettings()
}
updateSavedViewsVisibility(
dashboardVisibleViewIds: number[],
sidebarVisibleViewIds: number[]
): Observable<any> {
this.set(SETTINGS_KEYS.DASHBOARD_VIEWS_VISIBLE_IDS, [
...new Set(dashboardVisibleViewIds),
])
this.set(SETTINGS_KEYS.SIDEBAR_VIEWS_VISIBLE_IDS, [
...new Set(sidebarVisibleViewIds),
])
return this.storeSettings()
}
}

View File

@@ -416,4 +416,42 @@ describe('ConsumerStatusService', () => {
websocketStatusService.disconnect()
expect(deleted).toBeTruthy()
})
it('should trigger updated subject on document updated', () => {
let updated = false
websocketStatusService.onDocumentUpdated().subscribe((data) => {
updated = true
expect(data.document_id).toEqual(12)
})
websocketStatusService.connect()
server.send({
type: WebsocketStatusType.DOCUMENT_UPDATED,
data: {
document_id: 12,
modified: '2026-02-17T00:00:00Z',
owner_id: 1,
},
})
websocketStatusService.disconnect()
expect(updated).toBeTruthy()
})
it('should ignore document updated events the user cannot view', () => {
let updated = false
websocketStatusService.onDocumentUpdated().subscribe(() => {
updated = true
})
websocketStatusService.handleDocumentUpdated({
document_id: 12,
modified: '2026-02-17T00:00:00Z',
owner_id: 2,
users_can_view: [],
groups_can_view: [],
})
expect(updated).toBeFalsy()
})
})

View File

@@ -2,6 +2,7 @@ import { Injectable, inject } from '@angular/core'
import { Subject } from 'rxjs'
import { environment } from 'src/environments/environment'
import { User } from '../data/user'
import { WebsocketDocumentUpdatedMessage } from '../data/websocket-document-updated-message'
import { WebsocketDocumentsDeletedMessage } from '../data/websocket-documents-deleted-message'
import { WebsocketProgressMessage } from '../data/websocket-progress-message'
import { SettingsService } from './settings.service'
@@ -9,6 +10,7 @@ import { SettingsService } from './settings.service'
export enum WebsocketStatusType {
STATUS_UPDATE = 'status_update',
DOCUMENTS_DELETED = 'documents_deleted',
DOCUMENT_UPDATED = 'document_updated',
}
// see ProgressStatusOptions in src/documents/plugins/helpers.py
@@ -100,17 +102,20 @@ export enum UploadState {
providedIn: 'root',
})
export class WebsocketStatusService {
private settingsService = inject(SettingsService)
private readonly settingsService = inject(SettingsService)
private statusWebSocket: WebSocket
private consumerStatus: FileStatus[] = []
private documentDetectedSubject = new Subject<FileStatus>()
private documentConsumptionFinishedSubject = new Subject<FileStatus>()
private documentConsumptionFailedSubject = new Subject<FileStatus>()
private documentDeletedSubject = new Subject<boolean>()
private connectionStatusSubject = new Subject<boolean>()
private readonly documentDetectedSubject = new Subject<FileStatus>()
private readonly documentConsumptionFinishedSubject =
new Subject<FileStatus>()
private readonly documentConsumptionFailedSubject = new Subject<FileStatus>()
private readonly documentDeletedSubject = new Subject<boolean>()
private readonly documentUpdatedSubject =
new Subject<WebsocketDocumentUpdatedMessage>()
private readonly connectionStatusSubject = new Subject<boolean>()
private get(taskId: string, filename?: string) {
let status =
@@ -176,7 +181,10 @@ export class WebsocketStatusService {
data: messageData,
}: {
type: WebsocketStatusType
data: WebsocketProgressMessage | WebsocketDocumentsDeletedMessage
data:
| WebsocketProgressMessage
| WebsocketDocumentsDeletedMessage
| WebsocketDocumentUpdatedMessage
} = JSON.parse(ev.data)
switch (type) {
@@ -184,6 +192,12 @@ export class WebsocketStatusService {
this.documentDeletedSubject.next(true)
break
case WebsocketStatusType.DOCUMENT_UPDATED:
this.handleDocumentUpdated(
messageData as WebsocketDocumentUpdatedMessage
)
break
case WebsocketStatusType.STATUS_UPDATE:
this.handleProgressUpdate(messageData as WebsocketProgressMessage)
break
@@ -191,7 +205,11 @@ export class WebsocketStatusService {
}
}
private canViewMessage(messageData: WebsocketProgressMessage): boolean {
private canViewMessage(messageData: {
owner_id?: number
users_can_view?: number[]
groups_can_view?: number[]
}): boolean {
// see paperless.consumers.StatusConsumer._can_view
const user: User = this.settingsService.currentUser
return (
@@ -251,6 +269,15 @@ export class WebsocketStatusService {
}
}
handleDocumentUpdated(messageData: WebsocketDocumentUpdatedMessage) {
// fallback if backend didn't restrict message
if (!this.canViewMessage(messageData)) {
return
}
this.documentUpdatedSubject.next(messageData)
}
fail(status: FileStatus, message: string) {
status.message = message
status.phase = FileStatusPhase.FAILED
@@ -304,6 +331,10 @@ export class WebsocketStatusService {
return this.documentDeletedSubject
}
onDocumentUpdated() {
return this.documentUpdatedSubject
}
onConnectionStatus() {
return this.connectionStatusSubject.asObservable()
}

View File

@@ -3,10 +3,10 @@ const base_url = new URL(document.baseURI)
export const environment = {
production: true,
apiBaseUrl: document.baseURI + 'api/',
apiVersion: '9', // match src/paperless/settings.py
apiVersion: '10', // match src/paperless/settings.py
appTitle: 'Paperless-ngx',
tag: 'prod',
version: '2.20.9',
version: '2.20.10',
webSocketHost: window.location.host,
webSocketProtocol: window.location.protocol == 'https:' ? 'wss:' : 'ws:',
webSocketBaseUrl: base_url.pathname + 'ws/',

View File

@@ -15,6 +15,7 @@ class DocumentsConfig(AppConfig):
from documents.signals.handlers import add_to_index
from documents.signals.handlers import run_workflows_added
from documents.signals.handlers import run_workflows_updated
from documents.signals.handlers import send_websocket_document_updated
from documents.signals.handlers import set_correspondent
from documents.signals.handlers import set_document_type
from documents.signals.handlers import set_storage_path
@@ -29,6 +30,7 @@ class DocumentsConfig(AppConfig):
document_consumption_finished.connect(run_workflows_added)
document_consumption_finished.connect(add_or_update_document_in_llm_index)
document_updated.connect(run_workflows_updated)
document_updated.connect(send_websocket_document_updated)
import documents.schema # noqa: F401

View File

@@ -1,5 +1,5 @@
from datetime import UTC
from datetime import datetime
from datetime import timezone
from typing import Any
from django.conf import settings
@@ -139,7 +139,7 @@ def thumbnail_last_modified(request: Any, pk: int) -> datetime | None:
# No cache, get the timestamp and cache the datetime
last_modified = datetime.fromtimestamp(
doc.thumbnail_path.stat().st_mtime,
tz=timezone.utc,
tz=UTC,
)
cache.set(doc_key, last_modified, CACHE_50_MINUTES)
return last_modified

View File

@@ -2,7 +2,7 @@ import datetime
import hashlib
import os
import tempfile
from enum import Enum
from enum import StrEnum
from pathlib import Path
from typing import TYPE_CHECKING
from typing import Final
@@ -11,6 +11,7 @@ import magic
from django.conf import settings
from django.contrib.auth.models import User
from django.db import transaction
from django.db.models import Max
from django.db.models import Q
from django.utils import timezone
from filelock import FileLock
@@ -20,6 +21,7 @@ from documents.classifier import load_classifier
from documents.data_models import ConsumableDocument
from documents.data_models import DocumentMetadataOverrides
from documents.file_handling import create_source_path_directory
from documents.file_handling import generate_filename
from documents.file_handling import generate_unique_filename
from documents.loggers import LoggingMixin
from documents.models import Correspondent
@@ -81,7 +83,7 @@ class ConsumerError(Exception):
pass
class ConsumerStatusShortMessage(str, Enum):
class ConsumerStatusShortMessage(StrEnum):
DOCUMENT_ALREADY_EXISTS = "document_already_exists"
DOCUMENT_ALREADY_EXISTS_IN_TRASH = "document_already_exists_in_trash"
ASN_ALREADY_EXISTS = "asn_already_exists"
@@ -123,22 +125,6 @@ class ConsumerPluginMixin:
self.filename = self.metadata.filename or self.input_doc.original_file.name
if input_doc.root_document_id:
self.log.debug(
f"Document root document id: {input_doc.root_document_id}",
)
root_document = Document.objects.get(pk=input_doc.root_document_id)
version_index = Document.objects.filter(root_document=root_document).count()
filename_path = Path(self.filename)
if filename_path.suffix:
self.filename = str(
filename_path.with_name(
f"{filename_path.stem}_v{version_index}{filename_path.suffix}",
),
)
else:
self.filename = f"{self.filename}_v{version_index}"
def _send_progress(
self,
current_progress: int,
@@ -184,7 +170,7 @@ class ConsumerPlugin(
):
logging_name = LOGGING_NAME
def _clone_root_into_version(
def _create_version_from_root(
self,
root_doc: Document,
*,
@@ -193,30 +179,38 @@ class ConsumerPlugin(
mime_type: str,
) -> Document:
self.log.debug("Saving record for updated version to database")
version_doc = Document.objects.get(pk=root_doc.pk)
setattr(version_doc, "pk", None)
version_doc.root_document = root_doc
root_doc_frozen = Document.objects.select_for_update().get(pk=root_doc.pk)
next_version_index = (
Document.global_objects.filter(
root_document_id=root_doc_frozen.pk,
).aggregate(
max_index=Max("version_index"),
)["max_index"]
or 0
)
file_for_checksum = (
self.unmodified_original
if self.unmodified_original is not None
else self.working_copy
)
version_doc.checksum = hashlib.md5(
file_for_checksum.read_bytes(),
).hexdigest()
version_doc.content = text or ""
version_doc.page_count = page_count
version_doc.mime_type = mime_type
version_doc.original_filename = self.filename
version_doc.storage_path = root_doc.storage_path
# Clear unique file path fields so they can be generated uniquely later
version_doc.filename = None
version_doc.archive_filename = None
version_doc.archive_checksum = None
version_doc = Document(
root_document=root_doc_frozen,
version_index=next_version_index + 1,
checksum=hashlib.md5(
file_for_checksum.read_bytes(),
).hexdigest(),
content=text or "",
page_count=page_count,
mime_type=mime_type,
original_filename=self.filename,
owner_id=root_doc_frozen.owner_id,
created=root_doc_frozen.created,
title=root_doc_frozen.title,
added=timezone.now(),
modified=timezone.now(),
)
if self.metadata.version_label is not None:
version_doc.version_label = self.metadata.version_label
version_doc.added = timezone.now()
version_doc.modified = timezone.now()
return version_doc
def run_pre_consume_script(self) -> None:
@@ -542,7 +536,7 @@ class ConsumerPlugin(
root_doc = Document.objects.get(
pk=self.input_doc.root_document_id,
)
original_document = self._clone_root_into_version(
original_document = self._create_version_from_root(
root_doc,
text=text,
page_count=page_count,
@@ -611,7 +605,19 @@ class ConsumerPlugin(
# After everything is in the database, copy the files into
# place. If this fails, we'll also rollback the transaction.
with FileLock(settings.MEDIA_LOCK):
document.filename = generate_unique_filename(document)
generated_filename = generate_unique_filename(document)
if (
len(str(generated_filename))
> Document.MAX_STORED_FILENAME_LENGTH
):
self.log.warning(
"Generated source filename exceeds db path limit, falling back to default naming",
)
generated_filename = generate_filename(
document,
use_format=False,
)
document.filename = generated_filename
create_source_path_directory(document.source_path)
self._write(
@@ -627,10 +633,23 @@ class ConsumerPlugin(
)
if archive_path and Path(archive_path).is_file():
document.archive_filename = generate_unique_filename(
generated_archive_filename = generate_unique_filename(
document,
archive_filename=True,
)
if (
len(str(generated_archive_filename))
> Document.MAX_STORED_FILENAME_LENGTH
):
self.log.warning(
"Generated archive filename exceeds db path limit, falling back to default naming",
)
generated_archive_filename = generate_filename(
document,
archive_filename=True,
use_format=False,
)
document.archive_filename = generated_archive_filename
create_source_path_directory(document.archive_path)
self._write(
archive_path,

View File

@@ -127,23 +127,34 @@ def generate_filename(
*,
counter=0,
archive_filename=False,
use_format=True,
) -> Path:
# version docs use the root document for formatting, just with a suffix
context_doc = doc if doc.root_document_id is None else doc.root_document
version_suffix = (
f"_v{doc.version_index}"
if doc.root_document_id is not None and doc.version_index is not None
else ""
)
base_path: Path | None = None
# Determine the source of the format string
if doc.storage_path is not None:
filename_format = doc.storage_path.path
elif settings.FILENAME_FORMAT is not None:
# Maybe convert old to new style
filename_format = convert_format_str_to_template_format(
settings.FILENAME_FORMAT,
)
if use_format:
if context_doc.storage_path is not None:
filename_format = context_doc.storage_path.path
elif settings.FILENAME_FORMAT is not None:
# Maybe convert old to new style
filename_format = convert_format_str_to_template_format(
settings.FILENAME_FORMAT,
)
else:
filename_format = None
else:
filename_format = None
# If we have one, render it
if filename_format is not None:
rendered_path: str | None = format_filename(doc, filename_format)
rendered_path: str | None = format_filename(context_doc, filename_format)
if rendered_path:
base_path = Path(rendered_path)
@@ -157,7 +168,7 @@ def generate_filename(
base_filename = base_path.name
# Build the final filename with counter and filetype
final_filename = f"{base_filename}{counter_str}{filetype_str}"
final_filename = f"{base_filename}{version_suffix}{counter_str}{filetype_str}"
# If we have a directory component, include it
if str(directory) != ".":
@@ -166,7 +177,9 @@ def generate_filename(
full_path = Path(final_filename)
else:
# No template, use document ID
final_filename = f"{doc.pk:07}{counter_str}{filetype_str}"
final_filename = (
f"{context_doc.pk:07}{version_suffix}{counter_str}{filetype_str}"
)
full_path = Path(final_filename)
return full_path

View File

@@ -5,10 +5,10 @@ import math
import re
from collections import Counter
from contextlib import contextmanager
from datetime import UTC
from datetime import datetime
from datetime import time
from datetime import timedelta
from datetime import timezone
from shutil import rmtree
from time import sleep
from typing import TYPE_CHECKING
@@ -437,7 +437,7 @@ class ManualResults:
class LocalDateParser(English):
def reverse_timezone_offset(self, d):
return (d.replace(tzinfo=django_timezone.get_current_timezone())).astimezone(
timezone.utc,
UTC,
)
def date_from(self, *args, **kwargs):
@@ -641,8 +641,8 @@ def rewrite_natural_date_keywords(query_string: str) -> str:
end = datetime(local_now.year - 1, 12, 31, 23, 59, 59, tzinfo=tz)
# Convert to UTC and format
start_str = start.astimezone(timezone.utc).strftime("%Y%m%d%H%M%S")
end_str = end.astimezone(timezone.utc).strftime("%Y%m%d%H%M%S")
start_str = start.astimezone(UTC).strftime("%Y%m%d%H%M%S")
end_str = end.astimezone(UTC).strftime("%Y%m%d%H%M%S")
return f"{field}:[{start_str} TO {end_str}]"
return re.sub(pattern, repl, query_string, flags=re.IGNORECASE)

View File

@@ -6,11 +6,14 @@ Provides automatic progress bar and multiprocessing support with minimal boilerp
from __future__ import annotations
import logging
import os
from collections.abc import Callable
from collections.abc import Iterable
from collections.abc import Sized
from concurrent.futures import ProcessPoolExecutor
from concurrent.futures import as_completed
from contextlib import contextmanager
from dataclasses import dataclass
from typing import TYPE_CHECKING
from typing import Any
@@ -22,7 +25,11 @@ from django import db
from django.core.management import CommandError
from django.db.models import QuerySet
from django_rich.management import RichCommand
from rich import box
from rich.console import Console
from rich.console import Group
from rich.console import RenderableType
from rich.live import Live
from rich.progress import BarColumn
from rich.progress import MofNCompleteColumn
from rich.progress import Progress
@@ -30,11 +37,11 @@ from rich.progress import SpinnerColumn
from rich.progress import TextColumn
from rich.progress import TimeElapsedColumn
from rich.progress import TimeRemainingColumn
from rich.table import Table
from rich.text import Text
if TYPE_CHECKING:
from collections.abc import Callable
from collections.abc import Generator
from collections.abc import Iterable
from collections.abc import Sequence
from django.core.management import CommandParser
@@ -43,6 +50,78 @@ T = TypeVar("T")
R = TypeVar("R")
@dataclass(slots=True, frozen=True)
class _BufferedRecord:
level: int
name: str
message: str
class BufferingLogHandler(logging.Handler):
"""Captures log records during a command run for deferred rendering.
Attach to a logger before a long operation and call ``render()``
afterwards to emit the buffered records via Rich, optionally filtered
by minimum level.
"""
def __init__(self) -> None:
super().__init__()
self._records: list[_BufferedRecord] = []
def emit(self, record: logging.LogRecord) -> None:
self._records.append(
_BufferedRecord(
level=record.levelno,
name=record.name,
message=self.format(record),
),
)
def render(
self,
console: Console,
*,
min_level: int = logging.DEBUG,
title: str = "Log Output",
) -> None:
records = [r for r in self._records if r.level >= min_level]
if not records:
return
table = Table(
title=title,
show_header=True,
header_style="bold",
show_lines=False,
box=box.SIMPLE,
)
table.add_column("Level", style="bold", width=8)
table.add_column("Logger", style="dim")
table.add_column("Message", no_wrap=False)
_level_styles: dict[int, str] = {
logging.DEBUG: "dim",
logging.INFO: "cyan",
logging.WARNING: "yellow",
logging.ERROR: "red",
logging.CRITICAL: "bold red",
}
for record in records:
style = _level_styles.get(record.level, "")
table.add_row(
Text(logging.getLevelName(record.level), style=style),
record.name,
record.message,
)
console.print(table)
def clear(self) -> None:
self._records.clear()
@dataclass(frozen=True, slots=True)
class ProcessResult(Generic[T, R]):
"""
@@ -91,6 +170,23 @@ class PaperlessCommand(RichCommand):
for result in self.process_parallel(process_doc, ids):
if result.error:
self.console.print(f"[red]Failed: {result.error}[/red]")
class Command(PaperlessCommand):
help = "Import documents with live stats"
def handle(self, *args, **options):
stats = ImportStats()
def render_stats() -> Table:
... # build Rich Table from stats
for item in self.track_with_stats(
items,
description="Importing...",
stats_renderer=render_stats,
):
result = import_item(item)
stats.imported += 1
"""
supports_progress_bar: ClassVar[bool] = True
@@ -128,13 +224,11 @@ class PaperlessCommand(RichCommand):
This is called by Django's command infrastructure after argument parsing
but before handle(). We use it to set instance attributes from options.
"""
# Set progress bar state
if self.supports_progress_bar:
self.no_progress_bar = options.get("no_progress_bar", False)
else:
self.no_progress_bar = True
# Set multiprocessing state
if self.supports_multiprocessing:
self.process_count = options.get("processes", 1)
if self.process_count < 1:
@@ -144,9 +238,69 @@ class PaperlessCommand(RichCommand):
return super().execute(*args, **options)
@contextmanager
def buffered_logging(
self,
*logger_names: str,
level: int = logging.DEBUG,
) -> Generator[BufferingLogHandler, None, None]:
"""Context manager that captures log output from named loggers.
Installs a ``BufferingLogHandler`` on each named logger for the
duration of the block, suppressing propagation to avoid interleaving
with the Rich live display. The handler is removed on exit regardless
of whether an exception occurred.
Usage::
with self.buffered_logging("paperless", "documents") as log_buf:
# ... run progress loop ...
if options["verbose"]:
log_buf.render(self.console)
"""
handler = BufferingLogHandler()
handler.setFormatter(logging.Formatter("%(message)s"))
loggers: list[logging.Logger] = []
original_propagate: dict[str, bool] = {}
for name in logger_names:
log = logging.getLogger(name)
log.addHandler(handler)
original_propagate[name] = log.propagate
log.propagate = False
loggers.append(log)
try:
yield handler
finally:
for log in loggers:
log.removeHandler(handler)
log.propagate = original_propagate[log.name]
@staticmethod
def _progress_columns() -> tuple[Any, ...]:
"""
Return the standard set of progress bar columns.
Extracted so both _create_progress (standalone) and track_with_stats
(inside Live) use identical column configuration without duplication.
"""
return (
SpinnerColumn(),
TextColumn("[progress.description]{task.description}"),
BarColumn(),
MofNCompleteColumn(),
TimeElapsedColumn(),
TimeRemainingColumn(),
)
def _create_progress(self, description: str) -> Progress:
"""
Create a configured Progress instance.
Create a standalone Progress instance with its own stderr Console.
Use this for track(). For track_with_stats(), Progress is created
directly inside a Live context instead.
Progress output is directed to stderr to match the convention that
progress bars are transient UI feedback, not command output. This
@@ -161,12 +315,7 @@ class PaperlessCommand(RichCommand):
A Progress instance configured with appropriate columns.
"""
return Progress(
SpinnerColumn(),
TextColumn("[progress.description]{task.description}"),
BarColumn(),
MofNCompleteColumn(),
TimeElapsedColumn(),
TimeRemainingColumn(),
*self._progress_columns(),
console=Console(stderr=True),
transient=False,
)
@@ -222,7 +371,6 @@ class PaperlessCommand(RichCommand):
yield from iterable
return
# Attempt to determine total if not provided
if total is None:
total = self._get_iterable_length(iterable)
@@ -232,6 +380,87 @@ class PaperlessCommand(RichCommand):
yield item
progress.advance(task_id)
def track_with_stats(
self,
iterable: Iterable[T],
*,
description: str = "Processing...",
stats_renderer: Callable[[], RenderableType],
total: int | None = None,
) -> Generator[T, None, None]:
"""
Iterate over items with a progress bar and a live-updating stats display.
The progress bar and stats renderable are combined in a single Live
context, so the stats panel re-renders in place below the progress bar
after each item is processed.
Respects --no-progress-bar flag. When disabled, yields items without
any display (stats are still updated by the caller's loop body, so
they will be accurate for any post-loop summary the caller prints).
Args:
iterable: The items to iterate over.
description: Text to display alongside the progress bar.
stats_renderer: Zero-argument callable that returns a Rich
renderable. Called after each item to refresh the display.
The caller typically closes over a mutable dataclass and
rebuilds a Table from it on each call.
total: Total number of items. If None, attempts to determine
automatically via .count() (for querysets) or len().
Yields:
Items from the iterable.
Example:
@dataclass
class Stats:
processed: int = 0
failed: int = 0
stats = Stats()
def render_stats() -> Table:
table = Table(box=None)
table.add_column("Processed")
table.add_column("Failed")
table.add_row(str(stats.processed), str(stats.failed))
return table
for item in self.track_with_stats(
items,
description="Importing...",
stats_renderer=render_stats,
):
try:
import_item(item)
stats.processed += 1
except Exception:
stats.failed += 1
"""
if self.no_progress_bar:
yield from iterable
return
if total is None:
total = self._get_iterable_length(iterable)
stderr_console = Console(stderr=True)
# Progress is created without its own console so Live controls rendering.
progress = Progress(*self._progress_columns())
task_id = progress.add_task(description, total=total)
with Live(
Group(progress, stats_renderer()),
console=stderr_console,
refresh_per_second=4,
) as live:
for item in iterable:
yield item
progress.advance(task_id)
live.update(Group(progress, stats_renderer()))
def process_parallel(
self,
fn: Callable[[T], R],
@@ -269,7 +498,7 @@ class PaperlessCommand(RichCommand):
total = len(items)
if self.process_count == 1:
# Sequential execution in main process - critical for testing
# Sequential execution in main process - critical for testing, so we don't fork in fork, etc
yield from self._process_sequential(fn, items, description, total)
else:
# Parallel execution with ProcessPoolExecutor
@@ -298,6 +527,7 @@ class PaperlessCommand(RichCommand):
total: int,
) -> Generator[ProcessResult[T, R], None, None]:
"""Process items in parallel using ProcessPoolExecutor."""
# Close database connections before forking - required for PostgreSQL
db.connections.close_all()

View File

@@ -3,6 +3,8 @@ import json
import os
import shutil
import tempfile
from itertools import chain
from itertools import islice
from pathlib import Path
from typing import TYPE_CHECKING
@@ -19,6 +21,7 @@ from django.contrib.contenttypes.models import ContentType
from django.core import serializers
from django.core.management.base import BaseCommand
from django.core.management.base import CommandError
from django.core.serializers.json import DjangoJSONEncoder
from django.db import transaction
from django.utils import timezone
from filelock import FileLock
@@ -26,6 +29,8 @@ from guardian.models import GroupObjectPermission
from guardian.models import UserObjectPermission
if TYPE_CHECKING:
from collections.abc import Generator
from django.db.models import QuerySet
if settings.AUDIT_LOG_ENABLED:
@@ -60,6 +65,22 @@ from paperless_mail.models import MailAccount
from paperless_mail.models import MailRule
def serialize_queryset_batched(
queryset: "QuerySet",
*,
batch_size: int = 500,
) -> "Generator[list[dict], None, None]":
"""Yield batches of serialized records from a QuerySet.
Each batch is a list of dicts in Django's Python serialization format.
Uses QuerySet.iterator() to avoid loading the full queryset into memory,
and islice to collect chunk-sized batches serialized in a single call.
"""
iterator = queryset.iterator(chunk_size=batch_size)
while chunk := list(islice(iterator, batch_size)):
yield serializers.serialize("python", chunk)
class Command(CryptMixin, BaseCommand):
help = (
"Decrypt and rename all files in our collection into a given target "
@@ -186,6 +207,17 @@ class Command(CryptMixin, BaseCommand):
help="If provided, is used to encrypt sensitive data in the export",
)
parser.add_argument(
"--batch-size",
type=int,
default=500,
help=(
"Number of records to process per batch during serialization. "
"Lower values reduce peak memory usage; higher values improve "
"throughput. Default: 500."
),
)
def handle(self, *args, **options) -> None:
self.target = Path(options["target"]).resolve()
self.split_manifest: bool = options["split_manifest"]
@@ -200,6 +232,7 @@ class Command(CryptMixin, BaseCommand):
self.data_only: bool = options["data_only"]
self.no_progress_bar: bool = options["no_progress_bar"]
self.passphrase: str | None = options.get("passphrase")
self.batch_size: int = options["batch_size"]
self.files_in_export_dir: set[Path] = set()
self.exported_files: set[str] = set()
@@ -294,8 +327,13 @@ class Command(CryptMixin, BaseCommand):
# Build an overall manifest
for key, object_query in manifest_key_to_object_query.items():
manifest_dict[key] = json.loads(
serializers.serialize("json", object_query),
manifest_dict[key] = list(
chain.from_iterable(
serialize_queryset_batched(
object_query,
batch_size=self.batch_size,
),
),
)
self.encrypt_secret_fields(manifest_dict)
@@ -512,14 +550,24 @@ class Command(CryptMixin, BaseCommand):
self.files_in_export_dir.remove(target)
if self.compare_json:
target_checksum = hashlib.md5(target.read_bytes()).hexdigest()
src_str = json.dumps(content, indent=2, ensure_ascii=False)
src_str = json.dumps(
content,
cls=DjangoJSONEncoder,
indent=2,
ensure_ascii=False,
)
src_checksum = hashlib.md5(src_str.encode("utf-8")).hexdigest()
if src_checksum == target_checksum:
perform_write = False
if perform_write:
target.write_text(
json.dumps(content, indent=2, ensure_ascii=False),
json.dumps(
content,
cls=DjangoJSONEncoder,
indent=2,
ensure_ascii=False,
),
encoding="utf-8",
)

View File

@@ -1,22 +1,25 @@
from django.core.management import BaseCommand
from django.db import transaction
from documents.management.commands.mixins import ProgressBarMixin
from documents.management.commands.base import PaperlessCommand
from documents.tasks import index_optimize
from documents.tasks import index_reindex
class Command(ProgressBarMixin, BaseCommand):
class Command(PaperlessCommand):
help = "Manages the document index."
def add_arguments(self, parser):
super().add_arguments(parser)
parser.add_argument("command", choices=["reindex", "optimize"])
self.add_argument_progress_bar_mixin(parser)
def handle(self, *args, **options):
self.handle_progress_bar_mixin(**options)
with transaction.atomic():
if options["command"] == "reindex":
index_reindex(progress_bar_disable=self.no_progress_bar)
index_reindex(
iter_wrapper=lambda docs: self.track(
docs,
description="Indexing documents...",
),
)
elif options["command"] == "optimize":
index_optimize()

View File

@@ -1,22 +1,22 @@
from django.core.management import BaseCommand
from django.db import transaction
from typing import Any
from documents.management.commands.mixins import ProgressBarMixin
from documents.management.commands.base import PaperlessCommand
from documents.tasks import llmindex_index
class Command(ProgressBarMixin, BaseCommand):
class Command(PaperlessCommand):
help = "Manages the LLM-based vector index for Paperless."
def add_arguments(self, parser):
def add_arguments(self, parser: Any) -> None:
super().add_arguments(parser)
parser.add_argument("command", choices=["rebuild", "update"])
self.add_argument_progress_bar_mixin(parser)
def handle(self, *args, **options):
self.handle_progress_bar_mixin(**options)
with transaction.atomic():
llmindex_index(
progress_bar_disable=self.no_progress_bar,
rebuild=options["command"] == "rebuild",
scheduled=False,
)
def handle(self, *args: Any, **options: Any) -> None:
llmindex_index(
rebuild=options["command"] == "rebuild",
scheduled=False,
iter_wrapper=lambda docs: self.track(
docs,
description="Indexing documents...",
),
)

View File

@@ -1,4 +1,12 @@
from __future__ import annotations
import logging
from dataclasses import dataclass
from dataclasses import field
from typing import TYPE_CHECKING
from rich.table import Table
from rich.text import Text
from documents.classifier import load_classifier
from documents.management.commands.base import PaperlessCommand
@@ -8,9 +16,162 @@ from documents.signals.handlers import set_document_type
from documents.signals.handlers import set_storage_path
from documents.signals.handlers import set_tags
if TYPE_CHECKING:
from rich.console import RenderableType
from documents.models import Correspondent
from documents.models import DocumentType
from documents.models import StoragePath
from documents.models import Tag
logger = logging.getLogger("paperless.management.retagger")
@dataclass(slots=True)
class RetaggerStats:
"""Cumulative counters updated as the retagger processes documents.
Mutable by design -- fields are incremented in the processing loop.
slots=True reduces per-instance memory overhead and speeds attribute access.
"""
correspondents: int = 0
document_types: int = 0
tags_added: int = 0
tags_removed: int = 0
storage_paths: int = 0
documents_processed: int = 0
@dataclass(slots=True)
class DocumentSuggestion:
"""Buffered classifier suggestions for a single document (suggest mode only).
Mutable by design -- fields are assigned incrementally as each setter runs.
"""
document: Document
correspondent: Correspondent | None = None
document_type: DocumentType | None = None
tags_to_add: frozenset[Tag] = field(default_factory=frozenset)
tags_to_remove: frozenset[Tag] = field(default_factory=frozenset)
storage_path: StoragePath | None = None
@property
def has_suggestions(self) -> bool:
return bool(
self.correspondent is not None
or self.document_type is not None
or self.tags_to_add
or self.tags_to_remove
or self.storage_path is not None,
)
def _build_stats_table(stats: RetaggerStats, *, suggest: bool) -> Table:
"""
Build the live-updating stats table shown below the progress bar.
In suggest mode the labels read "would set / would add" to make clear
that nothing has been written to the database.
"""
table = Table(box=None, padding=(0, 2), show_header=True, header_style="bold")
table.add_column("Documents")
table.add_column("Correspondents")
table.add_column("Doc Types")
table.add_column("Tags (+)")
table.add_column("Tags (-)")
table.add_column("Storage Paths")
verb = "would set" if suggest else "set"
table.add_row(
str(stats.documents_processed),
f"{stats.correspondents} {verb}",
f"{stats.document_types} {verb}",
f"+{stats.tags_added}",
f"-{stats.tags_removed}",
f"{stats.storage_paths} {verb}",
)
return table
def _build_suggestion_table(
suggestions: list[DocumentSuggestion],
base_url: str | None,
) -> Table:
"""
Build the final suggestion table printed after the progress bar completes.
Only documents with at least one suggestion are included.
"""
table = Table(
title="Suggested Changes",
show_header=True,
header_style="bold cyan",
show_lines=True,
)
table.add_column("Document", style="bold", no_wrap=False, min_width=20)
table.add_column("Correspondent")
table.add_column("Doc Type")
table.add_column("Tags")
table.add_column("Storage Path")
for suggestion in suggestions:
if not suggestion.has_suggestions:
continue
doc = suggestion.document
if base_url:
doc_cell = Text()
doc_cell.append(str(doc))
doc_cell.append(f"\n{base_url}/documents/{doc.pk}", style="dim")
else:
doc_cell = Text(f"{doc} [{doc.pk}]")
tag_parts: list[str] = []
for tag in sorted(suggestion.tags_to_add, key=lambda t: t.name):
tag_parts.append(f"[green]+{tag.name}[/green]")
for tag in sorted(suggestion.tags_to_remove, key=lambda t: t.name):
tag_parts.append(f"[red]-{tag.name}[/red]")
tag_cell = Text.from_markup(", ".join(tag_parts)) if tag_parts else Text("-")
table.add_row(
doc_cell,
str(suggestion.correspondent) if suggestion.correspondent else "-",
str(suggestion.document_type) if suggestion.document_type else "-",
tag_cell,
str(suggestion.storage_path) if suggestion.storage_path else "-",
)
return table
def _build_summary_table(stats: RetaggerStats) -> Table:
"""Build the final applied-changes summary table."""
table = Table(
title="Retagger Summary",
show_header=True,
header_style="bold cyan",
)
table.add_column("Metric", style="bold")
table.add_column("Count", justify="right")
table.add_row("Documents processed", str(stats.documents_processed))
table.add_row("Correspondents set", str(stats.correspondents))
table.add_row("Document types set", str(stats.document_types))
table.add_row("Tags added", str(stats.tags_added))
table.add_row("Tags removed", str(stats.tags_removed))
table.add_row("Storage paths set", str(stats.storage_paths))
return table
class Command(PaperlessCommand):
help = (
"Using the current classification model, assigns correspondents, tags "
@@ -19,7 +180,7 @@ class Command(PaperlessCommand):
"modified) after their initial import."
)
def add_arguments(self, parser):
def add_arguments(self, parser) -> None:
super().add_arguments(parser)
parser.add_argument("-c", "--correspondent", default=False, action="store_true")
parser.add_argument("-T", "--tags", default=False, action="store_true")
@@ -31,9 +192,9 @@ class Command(PaperlessCommand):
default=False,
action="store_true",
help=(
"By default this command won't try to assign a correspondent "
"if more than one matches the document. Use this flag if "
"you'd rather it just pick the first one it finds."
"By default this command will not try to assign a correspondent "
"if more than one matches the document. Use this flag to pick "
"the first match instead."
),
)
parser.add_argument(
@@ -42,91 +203,140 @@ class Command(PaperlessCommand):
default=False,
action="store_true",
help=(
"If set, the document retagger will overwrite any previously "
"set correspondent, document and remove correspondents, types "
"and tags that do not match anymore due to changed rules."
"Overwrite any previously set correspondent, document type, and "
"remove tags that no longer match due to changed rules."
),
)
parser.add_argument(
"--suggest",
default=False,
action="store_true",
help="Return the suggestion, don't change anything.",
help="Show what would be changed without applying anything.",
)
parser.add_argument(
"--base-url",
help="The base URL to use to build the link to the documents.",
help="Base URL used to build document links in suggest output.",
)
parser.add_argument(
"--id-range",
help="A range of document ids on which the retagging should be applied.",
help="Restrict retagging to documents within this ID range (inclusive).",
nargs=2,
type=int,
)
def handle(self, *args, **options):
def handle(self, *args, **options) -> None:
suggest: bool = options["suggest"]
overwrite: bool = options["overwrite"]
use_first: bool = options["use_first"]
base_url: str | None = options["base_url"]
do_correspondent: bool = options["correspondent"]
do_document_type: bool = options["document_type"]
do_tags: bool = options["tags"]
do_storage_path: bool = options["storage_path"]
if not any([do_correspondent, do_document_type, do_tags, do_storage_path]):
self.console.print(
"[yellow]No classifier targets specified. "
"Use -c, -T, -t, or -s to select what to retag.[/yellow]",
)
return
if options["inbox_only"]:
queryset = Document.objects.filter(tags__is_inbox_tag=True)
else:
queryset = Document.objects.all()
if options["id_range"]:
queryset = queryset.filter(
id__range=(options["id_range"][0], options["id_range"][1]),
)
lo, hi = options["id_range"]
queryset = queryset.filter(id__range=(lo, hi))
documents = queryset.distinct()
classifier = load_classifier()
for document in self.track(documents, description="Retagging..."):
if options["correspondent"]:
set_correspondent(
sender=None,
document=document,
classifier=classifier,
replace=options["overwrite"],
use_first=options["use_first"],
suggest=options["suggest"],
base_url=options["base_url"],
stdout=self.stdout,
style_func=self.style,
)
stats = RetaggerStats()
suggestions: list[DocumentSuggestion] = []
if options["document_type"]:
set_document_type(
sender=None,
document=document,
classifier=classifier,
replace=options["overwrite"],
use_first=options["use_first"],
suggest=options["suggest"],
base_url=options["base_url"],
stdout=self.stdout,
style_func=self.style,
)
def render_stats() -> RenderableType:
return _build_stats_table(stats, suggest=suggest)
if options["tags"]:
set_tags(
sender=None,
document=document,
classifier=classifier,
replace=options["overwrite"],
suggest=options["suggest"],
base_url=options["base_url"],
stdout=self.stdout,
style_func=self.style,
)
with self.buffered_logging(
"paperless",
"paperless.handlers",
"documents",
) as log_buf:
for document in self.track_with_stats(
documents,
description="Retagging...",
stats_renderer=render_stats,
):
suggestion = DocumentSuggestion(document=document)
if options["storage_path"]:
set_storage_path(
sender=None,
document=document,
classifier=classifier,
replace=options["overwrite"],
use_first=options["use_first"],
suggest=options["suggest"],
base_url=options["base_url"],
stdout=self.stdout,
style_func=self.style,
)
if do_correspondent:
correspondent = set_correspondent(
None,
document,
classifier=classifier,
replace=overwrite,
use_first=use_first,
dry_run=suggest,
)
if correspondent is not None:
stats.correspondents += 1
suggestion.correspondent = correspondent
if do_document_type:
document_type = set_document_type(
None,
document,
classifier=classifier,
replace=overwrite,
use_first=use_first,
dry_run=suggest,
)
if document_type is not None:
stats.document_types += 1
suggestion.document_type = document_type
if do_tags:
tags_to_add, tags_to_remove = set_tags(
None,
document,
classifier=classifier,
replace=overwrite,
dry_run=suggest,
)
stats.tags_added += len(tags_to_add)
stats.tags_removed += len(tags_to_remove)
suggestion.tags_to_add = frozenset(tags_to_add)
suggestion.tags_to_remove = frozenset(tags_to_remove)
if do_storage_path:
storage_path = set_storage_path(
None,
document,
classifier=classifier,
replace=overwrite,
use_first=use_first,
dry_run=suggest,
)
if storage_path is not None:
stats.storage_paths += 1
suggestion.storage_path = storage_path
stats.documents_processed += 1
if suggest:
suggestions.append(suggestion)
# Post-loop output
if suggest:
visible = [s for s in suggestions if s.has_suggestions]
if visible:
self.console.print(_build_suggestion_table(visible, base_url))
else:
self.console.print("[green]No changes suggested.[/green]")
else:
self.console.print(_build_summary_table(stats))
log_buf.render(self.console, min_level=logging.INFO, title="Retagger Log")

View File

@@ -1,17 +1,117 @@
from django.core.management.base import BaseCommand
"""Management command to check the document archive for issues."""
from documents.management.commands.mixins import ProgressBarMixin
from __future__ import annotations
import logging
from typing import Any
from rich.panel import Panel
from rich.table import Table
from rich.text import Text
from documents.management.commands.base import PaperlessCommand
from documents.models import Document
from documents.sanity_checker import SanityCheckMessages
from documents.sanity_checker import check_sanity
_LEVEL_STYLE: dict[int, tuple[str, str]] = {
logging.ERROR: ("bold red", "ERROR"),
logging.WARNING: ("yellow", "WARN"),
logging.INFO: ("dim", "INFO"),
}
class Command(ProgressBarMixin, BaseCommand):
class Command(PaperlessCommand):
help = "This command checks your document archive for issues."
def add_arguments(self, parser):
self.add_argument_progress_bar_mixin(parser)
def _render_results(self, messages: SanityCheckMessages) -> None:
"""Render sanity check results as a Rich table."""
def handle(self, *args, **options):
self.handle_progress_bar_mixin(**options)
messages = check_sanity(progress=self.use_progress_bar, scheduled=False)
if (
not messages.has_error
and not messages.has_warning
and not messages.has_info
):
self.console.print(
Panel(
"[green]No issues detected.[/green]",
title="Sanity Check",
border_style="green",
),
)
return
messages.log_messages()
# Build a lookup for document titles
doc_pks = [pk for pk in messages.document_pks() if pk is not None]
titles: dict[int, str] = {}
if doc_pks:
titles = dict(
Document.global_objects.filter(pk__in=doc_pks)
.only("pk", "title")
.values_list("pk", "title"),
)
table = Table(
title="Sanity Check Results",
show_lines=True,
title_style="bold",
)
table.add_column("Level", width=7, no_wrap=True)
table.add_column("Document", min_width=20)
table.add_column("Issue", ratio=1)
for doc_pk, doc_messages in messages.iter_messages():
if doc_pk is not None:
title = titles.get(doc_pk, "Unknown")
doc_label = f"#{doc_pk} {title}"
else:
doc_label = "(global)"
for msg in doc_messages:
style, label = _LEVEL_STYLE.get(
msg["level"],
("dim", "INFO"),
)
table.add_row(
Text(label, style=style),
Text(doc_label),
Text(str(msg["message"])),
)
self.console.print(table)
parts: list[str] = []
if messages.document_error_count:
parts.append(
f"{messages.document_error_count} document(s) with [bold red]errors[/bold red]",
)
if messages.document_warning_count:
parts.append(
f"{messages.document_warning_count} document(s) with [yellow]warnings[/yellow]",
)
if messages.document_info_count:
parts.append(f"{messages.document_info_count} document(s) with infos")
if messages.global_warning_count:
parts.append(
f"{messages.global_warning_count} global [yellow]warning(s)[/yellow]",
)
if parts:
if len(parts) > 1:
summary = ", ".join(parts[:-1]) + " and " + parts[-1]
else:
summary = parts[0]
self.console.print(f"\nFound {summary}.")
else:
self.console.print("\nNo issues found.")
def handle(self, *args: Any, **options: Any) -> None:
messages = check_sanity(
scheduled=False,
iter_wrapper=lambda docs: self.track(
docs,
description="Checking documents...",
),
)
self._render_results(messages)

View File

@@ -1,4 +1,4 @@
# Generated by Django 5.2.7 on 2026-01-15 22:08
# Generated by Django 5.2.11 on 2026-03-03 16:27
import datetime
@@ -21,6 +21,207 @@ class Migration(migrations.Migration):
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
]
replaces = [
("documents", "0001_initial"),
("documents", "0002_auto_20151226_1316"),
("documents", "0003_sender"),
("documents", "0004_auto_20160114_1844"),
(
"documents",
"0004_auto_20160114_1844_squashed_0011_auto_20160303_1929",
),
("documents", "0005_auto_20160123_0313"),
("documents", "0006_auto_20160123_0430"),
("documents", "0007_auto_20160126_2114"),
("documents", "0008_document_file_type"),
("documents", "0009_auto_20160214_0040"),
("documents", "0010_log"),
("documents", "0011_auto_20160303_1929"),
("documents", "0012_auto_20160305_0040"),
("documents", "0013_auto_20160325_2111"),
("documents", "0014_document_checksum"),
("documents", "0015_add_insensitive_to_match"),
(
"documents",
"0015_add_insensitive_to_match_squashed_0018_auto_20170715_1712",
),
("documents", "0016_auto_20170325_1558"),
("documents", "0017_auto_20170512_0507"),
("documents", "0018_auto_20170715_1712"),
("documents", "0019_add_consumer_user"),
("documents", "0020_document_added"),
("documents", "0021_document_storage_type"),
("documents", "0022_auto_20181007_1420"),
("documents", "0023_document_current_filename"),
("documents", "1000_update_paperless_all"),
("documents", "1001_auto_20201109_1636"),
("documents", "1002_auto_20201111_1105"),
("documents", "1003_mime_types"),
("documents", "1004_sanity_check_schedule"),
("documents", "1005_checksums"),
("documents", "1006_auto_20201208_2209"),
(
"documents",
"1006_auto_20201208_2209_squashed_1011_auto_20210101_2340",
),
("documents", "1007_savedview_savedviewfilterrule"),
("documents", "1008_auto_20201216_1736"),
("documents", "1009_auto_20201216_2005"),
("documents", "1010_auto_20210101_2159"),
("documents", "1011_auto_20210101_2340"),
("documents", "1012_fix_archive_files"),
("documents", "1013_migrate_tag_colour"),
("documents", "1014_auto_20210228_1614"),
("documents", "1015_remove_null_characters"),
("documents", "1016_auto_20210317_1351"),
(
"documents",
"1016_auto_20210317_1351_squashed_1020_merge_20220518_1839",
),
("documents", "1017_alter_savedviewfilterrule_rule_type"),
("documents", "1018_alter_savedviewfilterrule_value"),
("documents", "1019_storagepath_document_storage_path"),
("documents", "1019_uisettings"),
("documents", "1020_merge_20220518_1839"),
("documents", "1021_webp_thumbnail_conversion"),
("documents", "1022_paperlesstask"),
(
"documents",
"1022_paperlesstask_squashed_1036_alter_savedviewfilterrule_rule_type",
),
("documents", "1023_add_comments"),
("documents", "1024_document_original_filename"),
("documents", "1025_alter_savedviewfilterrule_rule_type"),
("documents", "1026_transition_to_celery"),
(
"documents",
"1027_remove_paperlesstask_attempted_task_and_more",
),
(
"documents",
"1028_remove_paperlesstask_task_args_and_more",
),
("documents", "1029_alter_document_archive_serial_number"),
("documents", "1030_alter_paperlesstask_task_file_name"),
(
"documents",
"1031_remove_savedview_user_correspondent_owner_and_more",
),
(
"documents",
"1032_alter_correspondent_matching_algorithm_and_more",
),
(
"documents",
"1033_alter_documenttype_options_alter_tag_options_and_more",
),
("documents", "1034_alter_savedviewfilterrule_rule_type"),
("documents", "1035_rename_comment_note"),
("documents", "1036_alter_savedviewfilterrule_rule_type"),
("documents", "1037_webp_encrypted_thumbnail_conversion"),
("documents", "1038_sharelink"),
("documents", "1039_consumptiontemplate"),
(
"documents",
"1040_customfield_customfieldinstance_and_more",
),
("documents", "1041_alter_consumptiontemplate_sources"),
(
"documents",
"1042_consumptiontemplate_assign_custom_fields_and_more",
),
("documents", "1043_alter_savedviewfilterrule_rule_type"),
(
"documents",
"1044_workflow_workflowaction_workflowtrigger_and_more",
),
(
"documents",
"1045_alter_customfieldinstance_value_monetary",
),
(
"documents",
"1045_alter_customfieldinstance_value_monetary_squashed_1049_document_deleted_at_document_restored_at",
),
(
"documents",
"1046_workflowaction_remove_all_correspondents_and_more",
),
("documents", "1047_savedview_display_mode_and_more"),
("documents", "1048_alter_savedviewfilterrule_rule_type"),
(
"documents",
"1049_document_deleted_at_document_restored_at",
),
("documents", "1050_customfield_extra_data_and_more"),
(
"documents",
"1051_alter_correspondent_owner_alter_document_owner_and_more",
),
("documents", "1052_document_transaction_id"),
("documents", "1053_document_page_count"),
(
"documents",
"1054_customfieldinstance_value_monetary_amount_and_more",
),
("documents", "1055_alter_storagepath_path"),
(
"documents",
"1056_customfieldinstance_deleted_at_and_more",
),
("documents", "1057_paperlesstask_owner"),
(
"documents",
"1058_workflowtrigger_schedule_date_custom_field_and_more",
),
(
"documents",
"1059_workflowactionemail_workflowactionwebhook_and_more",
),
(
"documents",
"1060_alter_customfieldinstance_value_select",
),
("documents", "1061_workflowactionwebhook_as_json"),
("documents", "1062_alter_savedviewfilterrule_rule_type"),
(
"documents",
"1063_paperlesstask_type_alter_paperlesstask_task_name_and_more",
),
("documents", "1064_delete_log"),
(
"documents",
"1065_workflowaction_assign_custom_fields_values",
),
(
"documents",
"1066_alter_workflowtrigger_schedule_offset_days",
),
("documents", "1067_alter_document_created"),
("documents", "1068_alter_document_created"),
(
"documents",
"1069_workflowtrigger_filter_has_storage_path_and_more",
),
(
"documents",
"1070_customfieldinstance_value_long_text_and_more",
),
(
"documents",
"1071_tag_tn_ancestors_count_tag_tn_ancestors_pks_and_more",
),
(
"documents",
"1072_workflowtrigger_filter_custom_field_query_and_more",
),
("documents", "1073_migrate_workflow_title_jinja"),
(
"documents",
"1074_workflowrun_deleted_at_workflowrun_restored_at_and_more",
),
]
operations = [
migrations.CreateModel(
name="WorkflowActionEmail",
@@ -185,70 +386,6 @@ class Migration(migrations.Migration):
"abstract": False,
},
),
migrations.CreateModel(
name="CustomField",
fields=[
(
"id",
models.AutoField(
auto_created=True,
primary_key=True,
serialize=False,
verbose_name="ID",
),
),
(
"created",
models.DateTimeField(
db_index=True,
default=django.utils.timezone.now,
editable=False,
verbose_name="created",
),
),
("name", models.CharField(max_length=128)),
(
"data_type",
models.CharField(
choices=[
("string", "String"),
("url", "URL"),
("date", "Date"),
("boolean", "Boolean"),
("integer", "Integer"),
("float", "Float"),
("monetary", "Monetary"),
("documentlink", "Document Link"),
("select", "Select"),
("longtext", "Long Text"),
],
editable=False,
max_length=50,
verbose_name="data type",
),
),
(
"extra_data",
models.JSONField(
blank=True,
help_text="Extra data for the custom field, such as select options",
null=True,
verbose_name="extra data",
),
),
],
options={
"verbose_name": "custom field",
"verbose_name_plural": "custom fields",
"ordering": ("created",),
"constraints": [
models.UniqueConstraint(
fields=("name",),
name="documents_customfield_unique_name",
),
],
},
),
migrations.CreateModel(
name="DocumentType",
fields=[
@@ -733,17 +870,6 @@ class Migration(migrations.Migration):
verbose_name="correspondent",
),
),
(
"owner",
models.ForeignKey(
blank=True,
default=None,
null=True,
on_delete=django.db.models.deletion.SET_NULL,
to=settings.AUTH_USER_MODEL,
verbose_name="owner",
),
),
(
"document_type",
models.ForeignKey(
@@ -767,12 +893,14 @@ class Migration(migrations.Migration):
),
),
(
"tags",
models.ManyToManyField(
"owner",
models.ForeignKey(
blank=True,
related_name="documents",
to="documents.tag",
verbose_name="tags",
default=None,
null=True,
on_delete=django.db.models.deletion.SET_NULL,
to=settings.AUTH_USER_MODEL,
verbose_name="owner",
),
),
],
@@ -782,6 +910,140 @@ class Migration(migrations.Migration):
"ordering": ("-created",),
},
),
migrations.AddField(
model_name="document",
name="tags",
field=models.ManyToManyField(
blank=True,
related_name="documents",
to="documents.tag",
verbose_name="tags",
),
),
migrations.CreateModel(
name="Note",
fields=[
(
"id",
models.AutoField(
auto_created=True,
primary_key=True,
serialize=False,
verbose_name="ID",
),
),
("deleted_at", models.DateTimeField(blank=True, null=True)),
("restored_at", models.DateTimeField(blank=True, null=True)),
("transaction_id", models.UUIDField(blank=True, null=True)),
(
"note",
models.TextField(
blank=True,
help_text="Note for the document",
verbose_name="content",
),
),
(
"created",
models.DateTimeField(
db_index=True,
default=django.utils.timezone.now,
verbose_name="created",
),
),
(
"document",
models.ForeignKey(
blank=True,
null=True,
on_delete=django.db.models.deletion.CASCADE,
related_name="notes",
to="documents.document",
verbose_name="document",
),
),
(
"user",
models.ForeignKey(
blank=True,
null=True,
on_delete=django.db.models.deletion.SET_NULL,
related_name="notes",
to=settings.AUTH_USER_MODEL,
verbose_name="user",
),
),
],
options={
"verbose_name": "note",
"verbose_name_plural": "notes",
"ordering": ("created",),
},
),
migrations.CreateModel(
name="CustomField",
fields=[
(
"id",
models.AutoField(
auto_created=True,
primary_key=True,
serialize=False,
verbose_name="ID",
),
),
(
"created",
models.DateTimeField(
db_index=True,
default=django.utils.timezone.now,
editable=False,
verbose_name="created",
),
),
("name", models.CharField(max_length=128)),
(
"data_type",
models.CharField(
choices=[
("string", "String"),
("url", "URL"),
("date", "Date"),
("boolean", "Boolean"),
("integer", "Integer"),
("float", "Float"),
("monetary", "Monetary"),
("documentlink", "Document Link"),
("select", "Select"),
("longtext", "Long Text"),
],
editable=False,
max_length=50,
verbose_name="data type",
),
),
(
"extra_data",
models.JSONField(
blank=True,
help_text="Extra data for the custom field, such as select options",
null=True,
verbose_name="extra data",
),
),
],
options={
"verbose_name": "custom field",
"verbose_name_plural": "custom fields",
"ordering": ("created",),
"constraints": [
models.UniqueConstraint(
fields=("name",),
name="documents_customfield_unique_name",
),
],
},
),
migrations.CreateModel(
name="CustomFieldInstance",
fields=[
@@ -880,66 +1142,6 @@ class Migration(migrations.Migration):
"ordering": ("created",),
},
),
migrations.CreateModel(
name="Note",
fields=[
(
"id",
models.AutoField(
auto_created=True,
primary_key=True,
serialize=False,
verbose_name="ID",
),
),
("deleted_at", models.DateTimeField(blank=True, null=True)),
("restored_at", models.DateTimeField(blank=True, null=True)),
("transaction_id", models.UUIDField(blank=True, null=True)),
(
"note",
models.TextField(
blank=True,
help_text="Note for the document",
verbose_name="content",
),
),
(
"created",
models.DateTimeField(
db_index=True,
default=django.utils.timezone.now,
verbose_name="created",
),
),
(
"document",
models.ForeignKey(
blank=True,
null=True,
on_delete=django.db.models.deletion.CASCADE,
related_name="notes",
to="documents.document",
verbose_name="document",
),
),
(
"user",
models.ForeignKey(
blank=True,
null=True,
on_delete=django.db.models.deletion.SET_NULL,
related_name="notes",
to=settings.AUTH_USER_MODEL,
verbose_name="user",
),
),
],
options={
"verbose_name": "note",
"verbose_name_plural": "notes",
"ordering": ("created",),
},
),
migrations.CreateModel(
name="PaperlessTask",
fields=[
@@ -986,7 +1188,6 @@ class Migration(migrations.Migration):
("train_classifier", "Train Classifier"),
("check_sanity", "Check Sanity"),
("index_optimize", "Index Optimize"),
("llmindex_update", "LLM Index Update"),
],
help_text="Name of the task that was run",
max_length=255,
@@ -1380,6 +1581,7 @@ class Migration(migrations.Migration):
verbose_name="Workflow Action Type",
),
),
("order", models.PositiveIntegerField(default=0, verbose_name="order")),
(
"assign_title",
models.TextField(

View File

@@ -1,4 +1,4 @@
# Generated by Django 5.2.9 on 2026-01-20 18:46
# Generated by Django 5.2.11 on 2026-03-03 16:27
import django.db.models.deletion
from django.db import migrations
@@ -9,8 +9,14 @@ class Migration(migrations.Migration):
initial = True
dependencies = [
("documents", "0001_initial"),
("paperless_mail", "0001_initial"),
("documents", "0001_squashed"),
("paperless_mail", "0001_squashed"),
]
# This migration needs a "replaces", but it doesn't matter which.
# Chose the last 2.20.x migration
replaces = [
("documents", "1075_workflowaction_order"),
]
operations = [

View File

@@ -6,7 +6,7 @@ from django.db import models
class Migration(migrations.Migration):
dependencies = [
("documents", "0002_initial"),
("documents", "0002_squashed"),
]
operations = [

View File

@@ -0,0 +1,30 @@
# Generated by Django 5.2.11 on 2026-03-03 16:42
from django.db import migrations
from django.db import models
class Migration(migrations.Migration):
dependencies = [
("documents", "0013_document_root_document"),
]
operations = [
migrations.AlterField(
model_name="paperlesstask",
name="task_name",
field=models.CharField(
choices=[
("consume_file", "Consume File"),
("train_classifier", "Train Classifier"),
("check_sanity", "Check Sanity"),
("index_optimize", "Index Optimize"),
("llmindex_update", "LLM Index Update"),
],
help_text="Name of the task that was run",
max_length=255,
null=True,
verbose_name="Task Name",
),
),
]

View File

@@ -0,0 +1,153 @@
# Generated by Django 5.2.11 on 2026-02-20 22:05
from collections import defaultdict
from django.db import migrations
from django.db import models
# from src-ui/src/app/data/ui-settings.ts
SAVED_VIEWS_KEY = "saved_views"
DASHBOARD_VIEWS_VISIBLE_IDS_KEY = "dashboard_views_visible_ids"
SIDEBAR_VIEWS_VISIBLE_IDS_KEY = "sidebar_views_visible_ids"
def _parse_visible_ids(raw_value) -> set[int]:
"""Return integer SavedView IDs parsed from a JSON list value."""
if not isinstance(raw_value, list):
return set()
parsed_ids = set()
for raw_id in raw_value:
raw_id_string = str(raw_id)
if raw_id_string.isdigit():
parsed_ids.add(int(raw_id_string))
return parsed_ids
def _set_default_visibility_ids(apps, schema_editor):
"""
Move SavedView visibility from boolean model props into JSON UiSettings.settings.saved_views, specifically:
settings.saved_views.dashboard_views_visible_ids
settings.saved_views.sidebar_views_visible_ids
"""
SavedView = apps.get_model("documents", "SavedView")
UiSettings = apps.get_model("documents", "UiSettings")
User = apps.get_model("auth", "User")
dashboard_visible_ids_by_owner: defaultdict[int, list[int]] = defaultdict(list)
for owner_id, view_id in SavedView.objects.filter(
owner__isnull=False,
show_on_dashboard=True,
).values_list("owner_id", "id"):
dashboard_visible_ids_by_owner[owner_id].append(view_id)
sidebar_visible_ids_by_owner: defaultdict[int, list[int]] = defaultdict(list)
for owner_id, view_id in SavedView.objects.filter(
owner__isnull=False,
show_in_sidebar=True,
).values_list("owner_id", "id"):
sidebar_visible_ids_by_owner[owner_id].append(view_id)
for user in User.objects.all():
ui_settings, _ = UiSettings.objects.get_or_create(
user=user,
defaults={"settings": {}},
)
current_settings = ui_settings.settings
if not isinstance(current_settings, dict):
current_settings = {}
changed = False
saved_views_settings = current_settings.get(SAVED_VIEWS_KEY)
if not isinstance(saved_views_settings, dict):
saved_views_settings = {}
changed = True
if saved_views_settings.get(DASHBOARD_VIEWS_VISIBLE_IDS_KEY) is None:
saved_views_settings[DASHBOARD_VIEWS_VISIBLE_IDS_KEY] = (
dashboard_visible_ids_by_owner.get(user.id, [])
)
changed = True
if saved_views_settings.get(SIDEBAR_VIEWS_VISIBLE_IDS_KEY) is None:
saved_views_settings[SIDEBAR_VIEWS_VISIBLE_IDS_KEY] = (
sidebar_visible_ids_by_owner.get(user.id, [])
)
changed = True
current_settings[SAVED_VIEWS_KEY] = saved_views_settings
if changed:
ui_settings.settings = current_settings
ui_settings.save(update_fields=["settings"])
def _restore_visibility_fields(apps, schema_editor):
SavedView = apps.get_model("documents", "SavedView")
UiSettings = apps.get_model("documents", "UiSettings")
dashboard_visible_ids_by_owner: dict[int, set[int]] = {}
sidebar_visible_ids_by_owner: dict[int, set[int]] = {}
for ui_settings in UiSettings.objects.all():
current_settings = ui_settings.settings
if not isinstance(current_settings, dict):
continue
saved_views_settings = current_settings.get(SAVED_VIEWS_KEY)
if not isinstance(saved_views_settings, dict):
saved_views_settings = {}
dashboard_visible_ids_by_owner[ui_settings.user_id] = _parse_visible_ids(
saved_views_settings.get(DASHBOARD_VIEWS_VISIBLE_IDS_KEY),
)
sidebar_visible_ids_by_owner[ui_settings.user_id] = _parse_visible_ids(
saved_views_settings.get(SIDEBAR_VIEWS_VISIBLE_IDS_KEY),
)
SavedView.objects.update(show_on_dashboard=False, show_in_sidebar=False)
for owner_id, dashboard_visible_ids in dashboard_visible_ids_by_owner.items():
if not dashboard_visible_ids:
continue
SavedView.objects.filter(
owner_id=owner_id,
id__in=dashboard_visible_ids,
).update(
show_on_dashboard=True,
)
for owner_id, sidebar_visible_ids in sidebar_visible_ids_by_owner.items():
if not sidebar_visible_ids:
continue
SavedView.objects.filter(owner_id=owner_id, id__in=sidebar_visible_ids).update(
show_in_sidebar=True,
)
class Migration(migrations.Migration):
dependencies = [
("documents", "0014_alter_paperlesstask_task_name"),
]
operations = [
migrations.AlterField(
model_name="savedview",
name="show_on_dashboard",
field=models.BooleanField(default=False, verbose_name="show on dashboard"),
),
migrations.AlterField(
model_name="savedview",
name="show_in_sidebar",
field=models.BooleanField(default=False, verbose_name="show in sidebar"),
),
migrations.RunPython(
_set_default_visibility_ids,
reverse_code=_restore_visibility_fields,
),
migrations.RemoveField(
model_name="savedview",
name="show_on_dashboard",
),
migrations.RemoveField(
model_name="savedview",
name="show_in_sidebar",
),
]

View File

@@ -0,0 +1,37 @@
# Generated by Django 5.2.11 on 2026-03-02 17:48
from django.conf import settings
from django.db import migrations
from django.db import models
class Migration(migrations.Migration):
dependencies = [
("documents", "0015_savedview_visibility_to_ui_settings"),
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
]
operations = [
migrations.AddField(
model_name="document",
name="version_index",
field=models.PositiveIntegerField(
blank=True,
db_index=True,
help_text="Index of this version within the root document.",
null=True,
verbose_name="version index",
),
),
migrations.AddConstraint(
model_name="document",
constraint=models.UniqueConstraint(
condition=models.Q(
("root_document__isnull", False),
("version_index__isnull", False),
),
fields=("root_document", "version_index"),
name="documents_document_root_version_index_uniq",
),
),
]

View File

@@ -75,7 +75,7 @@ class MatchingModel(ModelWithOwner):
is_insensitive = models.BooleanField(_("is insensitive"), default=True)
class Meta:
class Meta(ModelWithOwner.Meta):
abstract = True
ordering = ("name",)
constraints = [
@@ -156,6 +156,8 @@ class StoragePath(MatchingModel):
class Document(SoftDeleteModel, ModelWithOwner): # type: ignore[django-manager-missing]
MAX_STORED_FILENAME_LENGTH: Final[int] = 1024
correspondent = models.ForeignKey(
Correspondent,
blank=True,
@@ -262,7 +264,7 @@ class Document(SoftDeleteModel, ModelWithOwner): # type: ignore[django-manager-
filename = models.FilePathField(
_("filename"),
max_length=1024,
max_length=MAX_STORED_FILENAME_LENGTH,
editable=False,
default=None,
unique=True,
@@ -272,7 +274,7 @@ class Document(SoftDeleteModel, ModelWithOwner): # type: ignore[django-manager-
archive_filename = models.FilePathField(
_("archive filename"),
max_length=1024,
max_length=MAX_STORED_FILENAME_LENGTH,
editable=False,
default=None,
unique=True,
@@ -282,7 +284,7 @@ class Document(SoftDeleteModel, ModelWithOwner): # type: ignore[django-manager-
original_filename = models.CharField(
_("original filename"),
max_length=1024,
max_length=MAX_STORED_FILENAME_LENGTH,
editable=False,
default=None,
unique=False,
@@ -317,6 +319,14 @@ class Document(SoftDeleteModel, ModelWithOwner): # type: ignore[django-manager-
verbose_name=_("root document for this version"),
)
version_index = models.PositiveIntegerField(
_("version index"),
blank=True,
null=True,
db_index=True,
help_text=_("Index of this version within the root document."),
)
version_label = models.CharField(
_("version label"),
max_length=64,
@@ -329,6 +339,16 @@ class Document(SoftDeleteModel, ModelWithOwner): # type: ignore[django-manager-
ordering = ("-created",)
verbose_name = _("document")
verbose_name_plural = _("documents")
constraints = [
models.UniqueConstraint(
fields=["root_document", "version_index"],
condition=models.Q(
root_document__isnull=False,
version_index__isnull=False,
),
name="documents_document_root_version_index_uniq",
),
]
def __str__(self) -> str:
created = self.created.isoformat()
@@ -473,13 +493,6 @@ class SavedView(ModelWithOwner):
name = models.CharField(_("name"), max_length=128)
show_on_dashboard = models.BooleanField(
_("show on dashboard"),
)
show_in_sidebar = models.BooleanField(
_("show in sidebar"),
)
sort_field = models.CharField(
_("sort field"),
max_length=128,

View File

@@ -5,11 +5,7 @@ from abc import abstractmethod
from collections.abc import Iterator
from dataclasses import dataclass
from types import TracebackType
try:
from typing import Self
except ImportError:
from typing_extensions import Self
from typing import Self
import dateparser

View File

@@ -1,4 +1,5 @@
import enum
from collections.abc import Mapping
from typing import TYPE_CHECKING
from asgiref.sync import async_to_sync
@@ -8,7 +9,7 @@ if TYPE_CHECKING:
from channels_redis.pubsub import RedisPubSubChannelLayer
class ProgressStatusOptions(str, enum.Enum):
class ProgressStatusOptions(enum.StrEnum):
STARTED = "STARTED"
WORKING = "WORKING"
SUCCESS = "SUCCESS"
@@ -47,7 +48,7 @@ class BaseStatusManager:
async_to_sync(self._channel.flush)
self._channel = None
def send(self, payload: dict[str, str | int | None]) -> None:
def send(self, payload: Mapping[str, object]) -> None:
# Ensure the layer is open
self.open()
@@ -73,26 +74,28 @@ class ProgressManager(BaseStatusManager):
max_progress: int,
extra_args: dict[str, str | int | None] | None = None,
) -> None:
payload = {
"type": "status_update",
"data": {
"filename": self.filename,
"task_id": self.task_id,
"current_progress": current_progress,
"max_progress": max_progress,
"status": status,
"message": message,
},
data: dict[str, object] = {
"filename": self.filename,
"task_id": self.task_id,
"current_progress": current_progress,
"max_progress": max_progress,
"status": status,
"message": message,
}
if extra_args is not None:
payload["data"].update(extra_args)
data.update(extra_args)
payload: dict[str, object] = {
"type": "status_update",
"data": data,
}
self.send(payload)
class DocumentsStatusManager(BaseStatusManager):
def send_documents_deleted(self, documents: list[int]) -> None:
payload = {
payload: dict[str, object] = {
"type": "documents_deleted",
"data": {
"documents": documents,
@@ -100,3 +103,25 @@ class DocumentsStatusManager(BaseStatusManager):
}
self.send(payload)
def send_document_updated(
self,
*,
document_id: int,
modified: str,
owner_id: int | None = None,
users_can_view: list[int] | None = None,
groups_can_view: list[int] | None = None,
) -> None:
payload: dict[str, object] = {
"type": "document_updated",
"data": {
"document_id": document_id,
"modified": modified,
"owner_id": owner_id,
"users_can_view": users_can_view or [],
"groups_can_view": groups_can_view or [],
},
}
self.send(payload)

View File

@@ -1,80 +1,174 @@
"""
Sanity checker for the Paperless-ngx document archive.
Verifies that all documents have valid files, correct checksums,
and consistent metadata. Reports orphaned files in the media directory.
Progress display is the caller's responsibility -- pass an ``iter_wrapper``
to wrap the document queryset (e.g., with a progress bar). The default
is an identity function that adds no overhead.
"""
from __future__ import annotations
import hashlib
import logging
import uuid
from collections import defaultdict
from collections.abc import Callable
from collections.abc import Iterable
from collections.abc import Iterator
from pathlib import Path
from typing import TYPE_CHECKING
from typing import Final
from typing import TypedDict
from typing import TypeVar
from celery import states
from django.conf import settings
from django.utils import timezone
from tqdm import tqdm
from documents.models import Document
from documents.models import PaperlessTask
from paperless.config import GeneralConfig
logger = logging.getLogger("paperless.sanity_checker")
_T = TypeVar("_T")
IterWrapper = Callable[[Iterable[_T]], Iterable[_T]]
class MessageEntry(TypedDict):
"""A single sanity check message with its severity level."""
level: int
message: str
def _identity(iterable: Iterable[_T]) -> Iterable[_T]:
"""Pass through an iterable unchanged (default iter_wrapper)."""
return iterable
class SanityCheckMessages:
def __init__(self) -> None:
self._messages: dict[int, list[dict]] = defaultdict(list)
self.has_error = False
self.has_warning = False
"""Collects sanity check messages grouped by document primary key.
def error(self, doc_pk, message) -> None:
Messages are categorized as error, warning, or info. ``None`` is used
as the key for messages not associated with a specific document
(e.g., orphaned files).
"""
def __init__(self) -> None:
self._messages: dict[int | None, list[MessageEntry]] = defaultdict(list)
self.has_error: bool = False
self.has_warning: bool = False
self.has_info: bool = False
self.document_count: int = 0
self.document_error_count: int = 0
self.document_warning_count: int = 0
self.document_info_count: int = 0
self.global_warning_count: int = 0
# -- Recording ----------------------------------------------------------
def error(self, doc_pk: int | None, message: str) -> None:
self._messages[doc_pk].append({"level": logging.ERROR, "message": message})
self.has_error = True
if doc_pk is not None:
self.document_count += 1
self.document_error_count += 1
def warning(self, doc_pk, message) -> None:
def warning(self, doc_pk: int | None, message: str) -> None:
self._messages[doc_pk].append({"level": logging.WARNING, "message": message})
self.has_warning = True
def info(self, doc_pk, message) -> None:
if doc_pk is not None:
self.document_count += 1
self.document_warning_count += 1
else:
# This is the only type of global message we do right now
self.global_warning_count += 1
def info(self, doc_pk: int | None, message: str) -> None:
self._messages[doc_pk].append({"level": logging.INFO, "message": message})
self.has_info = True
if doc_pk is not None:
self.document_count += 1
self.document_info_count += 1
# -- Iteration / query --------------------------------------------------
def document_pks(self) -> list[int | None]:
"""Return all document PKs (including None for global messages)."""
return list(self._messages.keys())
def iter_messages(self) -> Iterator[tuple[int | None, list[MessageEntry]]]:
"""Iterate over (doc_pk, messages) pairs."""
yield from self._messages.items()
def __getitem__(self, item: int | None) -> list[MessageEntry]:
return self._messages[item]
# -- Summarize Helpers --------------------------------------------------
@property
def has_global_issues(self) -> bool:
return None in self._messages
@property
def total_issue_count(self) -> int:
"""Total number of error and warning messages across all documents and global."""
return (
self.document_error_count
+ self.document_warning_count
+ self.global_warning_count
)
# -- Logging output (used by Celery task path) --------------------------
def log_messages(self) -> None:
logger = logging.getLogger("paperless.sanity_checker")
"""Write all messages to the ``paperless.sanity_checker`` logger.
This is the output path for headless / Celery execution.
Management commands use Rich rendering instead.
"""
if len(self._messages) == 0:
logger.info("Sanity checker detected no issues.")
else:
# Query once
all_docs = Document.global_objects.all()
return
for doc_pk in self._messages:
if doc_pk is not None:
doc = all_docs.get(pk=doc_pk)
logger.info(
f"Detected following issue(s) with document #{doc.pk},"
f" titled {doc.title}",
)
for msg in self._messages[doc_pk]:
logger.log(msg["level"], msg["message"])
doc_pks = [pk for pk in self._messages if pk is not None]
titles: dict[int, str] = {}
if doc_pks:
titles = dict(
Document.global_objects.filter(pk__in=doc_pks)
.only("pk", "title")
.values_list("pk", "title"),
)
def __len__(self):
return len(self._messages)
def __getitem__(self, item):
return self._messages[item]
for doc_pk, entries in self._messages.items():
if doc_pk is not None:
title = titles.get(doc_pk, "Unknown")
logger.info(
"Detected following issue(s) with document #%s, titled %s",
doc_pk,
title,
)
for msg in entries:
logger.log(msg["level"], msg["message"])
class SanityCheckFailedException(Exception):
pass
def check_sanity(*, progress=False, scheduled=True) -> SanityCheckMessages:
paperless_task = PaperlessTask.objects.create(
task_id=uuid.uuid4(),
type=PaperlessTask.TaskType.SCHEDULED_TASK
if scheduled
else PaperlessTask.TaskType.MANUAL_TASK,
task_name=PaperlessTask.TaskName.CHECK_SANITY,
status=states.STARTED,
date_created=timezone.now(),
date_started=timezone.now(),
)
messages = SanityCheckMessages()
# ---------------------------------------------------------------------------
# Internal helpers
# ---------------------------------------------------------------------------
def _build_present_files() -> set[Path]:
"""Collect all files in MEDIA_ROOT, excluding directories and ignorable files."""
present_files = {
x.resolve()
for x in Path(settings.MEDIA_ROOT).glob("**/*")
@@ -82,95 +176,178 @@ def check_sanity(*, progress=False, scheduled=True) -> SanityCheckMessages:
}
lockfile = Path(settings.MEDIA_LOCK).resolve()
if lockfile in present_files:
present_files.remove(lockfile)
present_files.discard(lockfile)
general_config = GeneralConfig()
app_logo = general_config.app_logo or settings.APP_LOGO
if app_logo:
logo_file = Path(settings.MEDIA_ROOT / Path(app_logo.lstrip("/"))).resolve()
if logo_file in present_files:
present_files.remove(logo_file)
present_files.discard(logo_file)
for doc in tqdm(Document.global_objects.all(), disable=not progress):
# Check sanity of the thumbnail
thumbnail_path: Final[Path] = Path(doc.thumbnail_path).resolve()
if not thumbnail_path.exists() or not thumbnail_path.is_file():
messages.error(doc.pk, "Thumbnail of document does not exist.")
else:
if thumbnail_path in present_files:
present_files.remove(thumbnail_path)
try:
_ = thumbnail_path.read_bytes()
except OSError as e:
messages.error(doc.pk, f"Cannot read thumbnail file of document: {e}")
return present_files
# Check sanity of the original file
# TODO: extract method
source_path: Final[Path] = Path(doc.source_path).resolve()
if not source_path.exists() or not source_path.is_file():
messages.error(doc.pk, "Original of document does not exist.")
else:
if source_path in present_files:
present_files.remove(source_path)
try:
checksum = hashlib.md5(source_path.read_bytes()).hexdigest()
except OSError as e:
messages.error(doc.pk, f"Cannot read original file of document: {e}")
else:
if checksum != doc.checksum:
messages.error(
doc.pk,
"Checksum mismatch. "
f"Stored: {doc.checksum}, actual: {checksum}.",
)
# Check sanity of the archive file.
if doc.archive_checksum is not None and doc.archive_filename is None:
def _check_thumbnail(
doc: Document,
messages: SanityCheckMessages,
present_files: set[Path],
) -> None:
"""Verify the thumbnail exists and is readable."""
thumbnail_path: Final[Path] = Path(doc.thumbnail_path).resolve()
if not thumbnail_path.exists() or not thumbnail_path.is_file():
messages.error(doc.pk, "Thumbnail of document does not exist.")
return
present_files.discard(thumbnail_path)
try:
_ = thumbnail_path.read_bytes()
except OSError as e:
messages.error(doc.pk, f"Cannot read thumbnail file of document: {e}")
def _check_original(
doc: Document,
messages: SanityCheckMessages,
present_files: set[Path],
) -> None:
"""Verify the original file exists, is readable, and has matching checksum."""
source_path: Final[Path] = Path(doc.source_path).resolve()
if not source_path.exists() or not source_path.is_file():
messages.error(doc.pk, "Original of document does not exist.")
return
present_files.discard(source_path)
try:
checksum = hashlib.md5(source_path.read_bytes()).hexdigest()
except OSError as e:
messages.error(doc.pk, f"Cannot read original file of document: {e}")
else:
if checksum != doc.checksum:
messages.error(
doc.pk,
"Document has an archive file checksum, but no archive filename.",
f"Checksum mismatch. Stored: {doc.checksum}, actual: {checksum}.",
)
elif doc.archive_checksum is None and doc.archive_filename is not None:
def _check_archive(
doc: Document,
messages: SanityCheckMessages,
present_files: set[Path],
) -> None:
"""Verify archive file consistency: checksum/filename pairing and file integrity."""
if doc.archive_checksum is not None and doc.archive_filename is None:
messages.error(
doc.pk,
"Document has an archive file checksum, but no archive filename.",
)
elif doc.archive_checksum is None and doc.archive_filename is not None:
messages.error(
doc.pk,
"Document has an archive file, but its checksum is missing.",
)
elif doc.has_archive_version:
if TYPE_CHECKING:
assert isinstance(doc.archive_path, Path)
archive_path: Final[Path] = Path(doc.archive_path).resolve()
if not archive_path.exists() or not archive_path.is_file():
messages.error(doc.pk, "Archived version of document does not exist.")
return
present_files.discard(archive_path)
try:
checksum = hashlib.md5(archive_path.read_bytes()).hexdigest()
except OSError as e:
messages.error(
doc.pk,
"Document has an archive file, but its checksum is missing.",
f"Cannot read archive file of document: {e}",
)
elif doc.has_archive_version:
archive_path: Final[Path] = Path(doc.archive_path).resolve()
if not archive_path.exists() or not archive_path.is_file():
messages.error(doc.pk, "Archived version of document does not exist.")
else:
if archive_path in present_files:
present_files.remove(archive_path)
try:
checksum = hashlib.md5(archive_path.read_bytes()).hexdigest()
except OSError as e:
messages.error(
doc.pk,
f"Cannot read archive file of document : {e}",
)
else:
if checksum != doc.archive_checksum:
messages.error(
doc.pk,
"Checksum mismatch of archived document. "
f"Stored: {doc.archive_checksum}, "
f"actual: {checksum}.",
)
else:
if checksum != doc.archive_checksum:
messages.error(
doc.pk,
"Checksum mismatch of archived document. "
f"Stored: {doc.archive_checksum}, actual: {checksum}.",
)
# other document checks
if not doc.content:
messages.info(doc.pk, "Document contains no OCR data")
def _check_content(doc: Document, messages: SanityCheckMessages) -> None:
"""Flag documents with no OCR content."""
if not doc.content:
messages.info(doc.pk, "Document contains no OCR data")
def _check_document(
doc: Document,
messages: SanityCheckMessages,
present_files: set[Path],
) -> None:
"""Run all checks for a single document."""
_check_thumbnail(doc, messages, present_files)
_check_original(doc, messages, present_files)
_check_archive(doc, messages, present_files)
_check_content(doc, messages)
# ---------------------------------------------------------------------------
# Public entry point
# ---------------------------------------------------------------------------
def check_sanity(
*,
scheduled: bool = True,
iter_wrapper: IterWrapper[Document] = _identity,
) -> SanityCheckMessages:
"""Run a full sanity check on the document archive.
Args:
scheduled: Whether this is a scheduled (automatic) or manual check.
Controls the task type recorded in the database.
iter_wrapper: A callable that wraps the document iterable, e.g.,
for progress bar display. Defaults to identity (no wrapping).
Returns:
A SanityCheckMessages instance containing all detected issues.
"""
paperless_task = PaperlessTask.objects.create(
task_id=uuid.uuid4(),
type=(
PaperlessTask.TaskType.SCHEDULED_TASK
if scheduled
else PaperlessTask.TaskType.MANUAL_TASK
),
task_name=PaperlessTask.TaskName.CHECK_SANITY,
status=states.STARTED,
date_created=timezone.now(),
date_started=timezone.now(),
)
messages = SanityCheckMessages()
present_files = _build_present_files()
documents = Document.global_objects.all()
for doc in iter_wrapper(documents):
_check_document(doc, messages, present_files)
for extra_file in present_files:
messages.warning(None, f"Orphaned file in media dir: {extra_file}")
paperless_task.status = states.SUCCESS if not messages.has_error else states.FAILURE
# result is concatenated messages
paperless_task.result = f"{len(messages)} issues found."
if messages.has_error:
paperless_task.result += " Check logs for details."
if messages.total_issue_count == 0:
paperless_task.result = "No issues found."
else:
parts: list[str] = []
if messages.document_error_count:
parts.append(f"{messages.document_error_count} document(s) with errors")
if messages.document_warning_count:
parts.append(f"{messages.document_warning_count} document(s) with warnings")
if messages.global_warning_count:
parts.append(f"{messages.global_warning_count} global warning(s)")
paperless_task.result = ", ".join(parts) + " found."
if messages.has_error:
paperless_task.result += " Check logs for details."
paperless_task.date_done = timezone.now()
paperless_task.save(update_fields=["status", "result", "date_done"])
return messages

View File

@@ -1428,8 +1428,6 @@ class SavedViewSerializer(OwnedObjectSerializer):
fields = [
"id",
"name",
"show_on_dashboard",
"show_in_sidebar",
"sort_field",
"sort_reverse",
"filter_rules",
@@ -1439,6 +1437,7 @@ class SavedViewSerializer(OwnedObjectSerializer):
"owner",
"permissions",
"user_can_change",
"set_permissions",
]
def validate(self, attrs):
@@ -1476,7 +1475,7 @@ class SavedViewSerializer(OwnedObjectSerializer):
and len(validated_data["display_fields"]) == 0
):
validated_data["display_fields"] = None
super().update(instance, validated_data)
instance = super().update(instance, validated_data)
if rules_data is not None:
SavedViewFilterRule.objects.filter(saved_view=instance).delete()
for rule_data in rules_data:
@@ -1488,7 +1487,7 @@ class SavedViewSerializer(OwnedObjectSerializer):
if "user" in validated_data:
# backwards compatibility
validated_data["owner"] = validated_data.pop("user")
saved_view = SavedView.objects.create(**validated_data)
saved_view = super().create(validated_data)
for rule_data in rules_data:
SavedViewFilterRule.objects.create(saved_view=saved_view, **rule_data)
return saved_view

View File

@@ -4,6 +4,7 @@ import logging
import shutil
from pathlib import Path
from typing import TYPE_CHECKING
from typing import Any
from celery import shared_task
from celery import states
@@ -23,6 +24,7 @@ from django.db.models import Q
from django.dispatch import receiver
from django.utils import timezone
from filelock import FileLock
from rest_framework import serializers
from documents import matching
from documents.caching import clear_document_caches
@@ -32,12 +34,14 @@ from documents.file_handling import create_source_path_directory
from documents.file_handling import delete_empty_directories
from documents.file_handling import generate_filename
from documents.file_handling import generate_unique_filename
from documents.models import Correspondent
from documents.models import CustomField
from documents.models import CustomFieldInstance
from documents.models import Document
from documents.models import MatchingModel
from documents.models import DocumentType
from documents.models import PaperlessTask
from documents.models import SavedView
from documents.models import StoragePath
from documents.models import Tag
from documents.models import UiSettings
from documents.models import Workflow
@@ -45,6 +49,7 @@ from documents.models import WorkflowAction
from documents.models import WorkflowRun
from documents.models import WorkflowTrigger
from documents.permissions import get_objects_for_user_owner_aware
from documents.plugins.helpers import DocumentsStatusManager
from documents.templating.utils import convert_format_str_to_template_format
from documents.workflows.actions import build_workflow_action_context
from documents.workflows.actions import execute_email_action
@@ -66,6 +71,7 @@ if TYPE_CHECKING:
from documents.data_models import DocumentMetadataOverrides
logger = logging.getLogger("paperless.handlers")
DRF_DATETIME_FIELD = serializers.DateTimeField()
def add_inbox_tags(sender, document: Document, logging_group=None, **kwargs) -> None:
@@ -81,47 +87,41 @@ def add_inbox_tags(sender, document: Document, logging_group=None, **kwargs) ->
document.add_nested_tags(inbox_tags)
def _suggestion_printer(
stdout,
style_func,
suggestion_type: str,
document: Document,
selected: MatchingModel,
base_url: str | None = None,
) -> None:
"""
Smaller helper to reduce duplication when just outputting suggestions to the console
"""
doc_str = str(document)
if base_url is not None:
stdout.write(style_func.SUCCESS(doc_str))
stdout.write(style_func.SUCCESS(f"{base_url}/documents/{document.pk}"))
else:
stdout.write(style_func.SUCCESS(f"{doc_str} [{document.pk}]"))
stdout.write(f"Suggest {suggestion_type}: {selected}")
def set_correspondent(
sender,
sender: object,
document: Document,
*,
logging_group=None,
logging_group: object = None,
classifier: DocumentClassifier | None = None,
replace=False,
use_first=True,
suggest=False,
base_url=None,
stdout=None,
style_func=None,
**kwargs,
) -> None:
replace: bool = False,
use_first: bool = True,
dry_run: bool = False,
**kwargs: Any,
) -> Correspondent | None:
"""
Assign a correspondent to a document based on classifier results.
Args:
document: The document to classify.
logging_group: Optional logging group for structured log output.
classifier: The trained classifier. If None, only rule-based matching runs.
replace: If True, overwrite an existing correspondent assignment.
use_first: If True, pick the first match when multiple correspondents
match. If False, skip assignment when multiple match.
dry_run: If True, compute and return the selection without saving.
**kwargs: Absorbed for Django signal compatibility (e.g. sender, signal).
Returns:
The correspondent that was (or would be) assigned, or None if no match
was found or assignment was skipped.
"""
if document.correspondent and not replace:
return
return None
potential_correspondents = matching.match_correspondents(document, classifier)
potential_count = len(potential_correspondents)
selected = potential_correspondents[0] if potential_correspondents else None
if potential_count > 1:
if use_first:
logger.debug(
@@ -135,49 +135,53 @@ def set_correspondent(
f"not assigning any correspondent",
extra={"group": logging_group},
)
return
return None
if selected or replace:
if suggest:
_suggestion_printer(
stdout,
style_func,
"correspondent",
document,
selected,
base_url,
)
else:
logger.info(
f"Assigning correspondent {selected} to {document}",
extra={"group": logging_group},
)
if (selected or replace) and not dry_run:
logger.info(
f"Assigning correspondent {selected} to {document}",
extra={"group": logging_group},
)
document.correspondent = selected
document.save(update_fields=("correspondent",))
document.correspondent = selected
document.save(update_fields=("correspondent",))
return selected
def set_document_type(
sender,
sender: object,
document: Document,
*,
logging_group=None,
logging_group: object = None,
classifier: DocumentClassifier | None = None,
replace=False,
use_first=True,
suggest=False,
base_url=None,
stdout=None,
style_func=None,
**kwargs,
) -> None:
replace: bool = False,
use_first: bool = True,
dry_run: bool = False,
**kwargs: Any,
) -> DocumentType | None:
"""
Assign a document type to a document based on classifier results.
Args:
document: The document to classify.
logging_group: Optional logging group for structured log output.
classifier: The trained classifier. If None, only rule-based matching runs.
replace: If True, overwrite an existing document type assignment.
use_first: If True, pick the first match when multiple types match.
If False, skip assignment when multiple match.
dry_run: If True, compute and return the selection without saving.
**kwargs: Absorbed for Django signal compatibility (e.g. sender, signal).
Returns:
The document type that was (or would be) assigned, or None if no match
was found or assignment was skipped.
"""
if document.document_type and not replace:
return
return None
potential_document_type = matching.match_document_types(document, classifier)
potential_count = len(potential_document_type)
selected = potential_document_type[0] if potential_document_type else None
potential_document_types = matching.match_document_types(document, classifier)
potential_count = len(potential_document_types)
selected = potential_document_types[0] if potential_document_types else None
if potential_count > 1:
if use_first:
@@ -192,42 +196,64 @@ def set_document_type(
f"not assigning any document type",
extra={"group": logging_group},
)
return
return None
if selected or replace:
if suggest:
_suggestion_printer(
stdout,
style_func,
"document type",
document,
selected,
base_url,
)
else:
logger.info(
f"Assigning document type {selected} to {document}",
extra={"group": logging_group},
)
if (selected or replace) and not dry_run:
logger.info(
f"Assigning document type {selected} to {document}",
extra={"group": logging_group},
)
document.document_type = selected
document.save(update_fields=("document_type",))
document.document_type = selected
document.save(update_fields=("document_type",))
return selected
def set_tags(
sender,
sender: object,
document: Document,
*,
logging_group=None,
logging_group: object = None,
classifier: DocumentClassifier | None = None,
replace=False,
suggest=False,
base_url=None,
stdout=None,
style_func=None,
**kwargs,
) -> None:
replace: bool = False,
dry_run: bool = False,
**kwargs: Any,
) -> tuple[set[Tag], set[Tag]]:
"""
Assign tags to a document based on classifier results.
When replace=True, existing auto-matched and rule-matched tags are removed
before applying the new set (inbox tags and manually-added tags are preserved).
Args:
document: The document to classify.
logging_group: Optional logging group for structured log output.
classifier: The trained classifier. If None, only rule-based matching runs.
replace: If True, remove existing classifier-managed tags before applying
new ones. Inbox tags and manually-added tags are always preserved.
dry_run: If True, compute what would change without saving anything.
**kwargs: Absorbed for Django signal compatibility (e.g. sender, signal).
Returns:
A two-tuple of (tags_added, tags_removed). In non-replace mode,
tags_removed is always an empty set. In dry_run mode, neither set
is applied to the database.
"""
# Compute which tags would be removed under replace mode.
# The filter mirrors the .delete() call below: keep inbox tags and
# manually-added tags (match="" and not auto-matched).
if replace:
tags_to_remove: set[Tag] = set(
document.tags.exclude(
is_inbox_tag=True,
).exclude(
Q(match="") & ~Q(matching_algorithm=Tag.MATCH_AUTO),
),
)
else:
tags_to_remove = set()
if replace and not dry_run:
Document.tags.through.objects.filter(document=document).exclude(
Q(tag__is_inbox_tag=True),
).exclude(
@@ -235,65 +261,53 @@ def set_tags(
).delete()
current_tags = set(document.tags.all())
matched_tags = matching.match_tags(document, classifier)
tags_to_add = set(matched_tags) - current_tags
relevant_tags = set(matched_tags) - current_tags
if suggest:
extra_tags = current_tags - set(matched_tags)
extra_tags = [
t for t in extra_tags if t.matching_algorithm == MatchingModel.MATCH_AUTO
]
if not relevant_tags and not extra_tags:
return
doc_str = style_func.SUCCESS(str(document))
if base_url:
stdout.write(doc_str)
stdout.write(f"{base_url}/documents/{document.pk}")
else:
stdout.write(doc_str + style_func.SUCCESS(f" [{document.pk}]"))
if relevant_tags:
stdout.write("Suggest tags: " + ", ".join([t.name for t in relevant_tags]))
if extra_tags:
stdout.write("Extra tags: " + ", ".join([t.name for t in extra_tags]))
else:
if not relevant_tags:
return
message = 'Tagging "{}" with "{}"'
if tags_to_add and not dry_run:
logger.info(
message.format(document, ", ".join([t.name for t in relevant_tags])),
f'Tagging "{document}" with "{", ".join(t.name for t in tags_to_add)}"',
extra={"group": logging_group},
)
document.add_nested_tags(tags_to_add)
document.add_nested_tags(relevant_tags)
return tags_to_add, tags_to_remove
def set_storage_path(
sender,
sender: object,
document: Document,
*,
logging_group=None,
logging_group: object = None,
classifier: DocumentClassifier | None = None,
replace=False,
use_first=True,
suggest=False,
base_url=None,
stdout=None,
style_func=None,
**kwargs,
) -> None:
replace: bool = False,
use_first: bool = True,
dry_run: bool = False,
**kwargs: Any,
) -> StoragePath | None:
"""
Assign a storage path to a document based on classifier results.
Args:
document: The document to classify.
logging_group: Optional logging group for structured log output.
classifier: The trained classifier. If None, only rule-based matching runs.
replace: If True, overwrite an existing storage path assignment.
use_first: If True, pick the first match when multiple paths match.
If False, skip assignment when multiple match.
dry_run: If True, compute and return the selection without saving.
**kwargs: Absorbed for Django signal compatibility (e.g. sender, signal).
Returns:
The storage path that was (or would be) assigned, or None if no match
was found or assignment was skipped.
"""
if document.storage_path and not replace:
return
return None
potential_storage_path = matching.match_storage_paths(
document,
classifier,
)
potential_count = len(potential_storage_path)
selected = potential_storage_path[0] if potential_storage_path else None
potential_storage_paths = matching.match_storage_paths(document, classifier)
potential_count = len(potential_storage_paths)
selected = potential_storage_paths[0] if potential_storage_paths else None
if potential_count > 1:
if use_first:
@@ -308,26 +322,17 @@ def set_storage_path(
f"not assigning any storage directory",
extra={"group": logging_group},
)
return
return None
if selected or replace:
if suggest:
_suggestion_printer(
stdout,
style_func,
"storage directory",
document,
selected,
base_url,
)
else:
logger.info(
f"Assigning storage path {selected} to {document}",
extra={"group": logging_group},
)
if (selected or replace) and not dry_run:
logger.info(
f"Assigning storage path {selected} to {document}",
extra={"group": logging_group},
)
document.storage_path = selected
document.save(update_fields=("storage_path",))
document.storage_path = selected
document.save(update_fields=("storage_path",))
return selected
# see empty_trash in documents/tasks.py for signal handling
@@ -467,8 +472,22 @@ def update_filename_and_move_files(
old_filename = instance.filename
old_source_path = instance.source_path
move_original = False
old_archive_filename = instance.archive_filename
old_archive_path = instance.archive_path
move_archive = False
candidate_filename = generate_filename(instance)
if len(str(candidate_filename)) > Document.MAX_STORED_FILENAME_LENGTH:
msg = (
f"Document {instance!s}: Generated filename exceeds db path "
f"limit ({len(str(candidate_filename))} > "
f"{Document.MAX_STORED_FILENAME_LENGTH}): {candidate_filename!s}"
)
logger.warning(msg)
raise CannotMoveFilesException(msg)
candidate_source_path = (
settings.ORIGINALS_DIR / candidate_filename
).resolve()
@@ -487,11 +506,16 @@ def update_filename_and_move_files(
instance.filename = str(new_filename)
move_original = old_filename != instance.filename
old_archive_filename = instance.archive_filename
old_archive_path = instance.archive_path
if instance.has_archive_version:
archive_candidate = generate_filename(instance, archive_filename=True)
if len(str(archive_candidate)) > Document.MAX_STORED_FILENAME_LENGTH:
msg = (
f"Document {instance!s}: Generated archive filename exceeds "
f"db path limit ({len(str(archive_candidate))} > "
f"{Document.MAX_STORED_FILENAME_LENGTH}): {archive_candidate!s}"
)
logger.warning(msg)
raise CannotMoveFilesException(msg)
archive_candidate_path = (
settings.ARCHIVE_DIR / archive_candidate
).resolve()
@@ -596,6 +620,16 @@ def update_filename_and_move_files(
root=settings.ARCHIVE_DIR,
)
# Keep version files in sync with root
if instance.root_document_id is None:
for version_doc in Document.objects.filter(root_document_id=instance.pk).only(
"pk",
):
update_filename_and_move_files(
Document,
version_doc,
)
@shared_task
def process_cf_select_update(custom_field: CustomField) -> None:
@@ -762,6 +796,28 @@ def run_workflows_updated(
)
def send_websocket_document_updated(
sender,
document: Document,
**kwargs,
) -> None:
# At this point, workflows may already have applied additional changes.
document.refresh_from_db()
from documents.data_models import DocumentMetadataOverrides
doc_overrides = DocumentMetadataOverrides.from_document(document)
with DocumentsStatusManager() as status_mgr:
status_mgr.send_document_updated(
document_id=document.id,
modified=DRF_DATETIME_FIELD.to_representation(document.modified),
owner_id=doc_overrides.owner_id,
users_can_view=doc_overrides.view_users,
groups_can_view=doc_overrides.view_groups,
)
def run_workflows(
trigger_type: WorkflowTrigger.WorkflowTriggerType,
document: Document | ConsumableDocument,
@@ -1043,7 +1099,11 @@ def add_or_update_document_in_llm_index(sender, document, **kwargs):
@receiver(models.signals.post_delete, sender=Document)
def delete_document_from_llm_index(sender, instance: Document, **kwargs):
def delete_document_from_llm_index(
sender: Any,
instance: Document,
**kwargs: Any,
) -> None:
"""
Delete a document from the LLM index when it is deleted.
"""

View File

@@ -4,11 +4,13 @@ import logging
import shutil
import uuid
import zipfile
from collections.abc import Callable
from collections.abc import Iterable
from pathlib import Path
from tempfile import TemporaryDirectory
from tempfile import mkstemp
from typing import TypeVar
import tqdm
from celery import Task
from celery import shared_task
from celery import states
@@ -60,17 +62,26 @@ from documents.sanity_checker import SanityCheckFailedException
from documents.signals import document_updated
from documents.signals.handlers import cleanup_document_deletion
from documents.signals.handlers import run_workflows
from documents.signals.handlers import send_websocket_document_updated
from documents.workflows.utils import get_workflows_for_trigger
from paperless.config import AIConfig
from paperless_ai.indexing import llm_index_add_or_update_document
from paperless_ai.indexing import llm_index_remove_document
from paperless_ai.indexing import update_llm_index
_T = TypeVar("_T")
IterWrapper = Callable[[Iterable[_T]], Iterable[_T]]
if settings.AUDIT_LOG_ENABLED:
from auditlog.models import LogEntry
logger = logging.getLogger("paperless.tasks")
def _identity(iterable: Iterable[_T]) -> Iterable[_T]:
return iterable
@shared_task
def index_optimize() -> None:
ix = index.open_index()
@@ -78,13 +89,13 @@ def index_optimize() -> None:
writer.commit(optimize=True)
def index_reindex(*, progress_bar_disable=False) -> None:
def index_reindex(*, iter_wrapper: IterWrapper[Document] = _identity) -> None:
documents = Document.objects.all()
ix = index.open_index(recreate=True)
with AsyncWriter(ix) as writer:
for document in tqdm.tqdm(documents, disable=progress_bar_disable):
for document in iter_wrapper(documents):
index.update_document(writer, document)
@@ -227,20 +238,30 @@ def consume_file(
@shared_task
def sanity_check(*, scheduled=True, raise_on_error=True):
messages = sanity_checker.check_sanity(scheduled=scheduled)
messages.log_messages()
if not messages.has_error and not messages.has_warning and not messages.has_info:
return "No issues detected."
parts: list[str] = []
if messages.document_error_count:
parts.append(f"{messages.document_error_count} document(s) with errors")
if messages.document_warning_count:
parts.append(f"{messages.document_warning_count} document(s) with warnings")
if messages.document_info_count:
parts.append(f"{messages.document_info_count} document(s) with infos")
if messages.global_warning_count:
parts.append(f"{messages.global_warning_count} global warning(s)")
summary = ", ".join(parts) + " found."
if messages.has_error:
message = "Sanity check exited with errors. See log."
message = summary + " Check logs for details."
if raise_on_error:
raise SanityCheckFailedException(message)
return message
elif messages.has_warning:
return "Sanity check exited with warnings. See log."
elif len(messages) > 0:
return "Sanity check exited with infos. See log."
else:
return "No issues detected."
return summary
@shared_task
@@ -265,7 +286,6 @@ def bulk_update_documents(document_ids) -> None:
ai_config = AIConfig()
if ai_config.llm_index_enabled:
update_llm_index(
progress_bar_disable=True,
rebuild=False,
)
@@ -553,6 +573,11 @@ def check_scheduled_workflows() -> None:
workflow_to_run=workflow,
document=document,
)
# Scheduled workflows dont send document_updated signal, so send a websocket update here to ensure clients are updated
send_websocket_document_updated(
sender=None,
document=document,
)
def update_document_parent_tags(tag: Tag, new_parent: Tag) -> None:
@@ -606,7 +631,7 @@ def update_document_parent_tags(tag: Tag, new_parent: Tag) -> None:
@shared_task
def llmindex_index(
*,
progress_bar_disable=True,
iter_wrapper: IterWrapper[Document] = _identity,
rebuild=False,
scheduled=True,
auto=False,
@@ -629,7 +654,7 @@ def llmindex_index(
try:
result = update_llm_index(
progress_bar_disable=progress_bar_disable,
iter_wrapper=iter_wrapper,
rebuild=rebuild,
)
task.status = states.SUCCESS

View File

@@ -79,6 +79,23 @@ class PlaceholderString(str):
NO_VALUE_PLACEHOLDER = PlaceholderString("-none-")
class MatchingModelContext:
"""
Safe template context for related objects.
Keeps legacy behavior where including the object ina template yields the related object's
name as a string, while still exposing limited attributes.
"""
def __init__(self, *, id: int, name: str, path: str | None = None):
self.id = id
self.name = name
self.path = path
def __str__(self) -> str:
return self.name
_template_environment.undefined = _LogStrictUndefined
_template_environment.filters["get_cf_value"] = get_cf_value
@@ -220,19 +237,26 @@ def get_safe_document_context(
else None,
"tags": [{"name": tag.name, "id": tag.id} for tag in tags],
"correspondent": (
{"name": document.correspondent.name, "id": document.correspondent.id}
MatchingModelContext(
name=document.correspondent.name,
id=document.correspondent.id,
)
if document.correspondent
else None
),
"document_type": (
{"name": document.document_type.name, "id": document.document_type.id}
MatchingModelContext(
name=document.document_type.name,
id=document.document_type.id,
)
if document.document_type
else None
),
"storage_path": {
"path": document.storage_path.path,
"id": document.storage_path.id,
}
"storage_path": MatchingModelContext(
name=document.storage_path.name,
path=document.storage_path.path,
id=document.storage_path.id,
)
if document.storage_path
else None,
}

View File

@@ -1,10 +1,96 @@
import shutil
import zoneinfo
from dataclasses import dataclass
from pathlib import Path
from typing import TYPE_CHECKING
import filelock
import pytest
from django.contrib.auth import get_user_model
from pytest_django.fixtures import SettingsWrapper
from rest_framework.test import APIClient
from documents.tests.factories import DocumentFactory
if TYPE_CHECKING:
from documents.models import Document
@dataclass(frozen=True, slots=True)
class PaperlessDirs:
"""Standard Paperless-ngx directory layout for tests."""
media: Path
originals: Path
archive: Path
thumbnails: Path
@pytest.fixture(scope="session")
def samples_dir() -> Path:
"""Path to the shared test sample documents."""
return Path(__file__).parent / "samples" / "documents"
@pytest.fixture()
def paperless_dirs(tmp_path: Path) -> PaperlessDirs:
"""Create and return the directory structure for testing."""
media = tmp_path / "media"
dirs = PaperlessDirs(
media=media,
originals=media / "documents" / "originals",
archive=media / "documents" / "archive",
thumbnails=media / "documents" / "thumbnails",
)
for d in (dirs.originals, dirs.archive, dirs.thumbnails):
d.mkdir(parents=True)
return dirs
@pytest.fixture()
def _media_settings(paperless_dirs: PaperlessDirs, settings) -> None:
"""Configure Django settings to point at temp directories."""
settings.MEDIA_ROOT = paperless_dirs.media
settings.ORIGINALS_DIR = paperless_dirs.originals
settings.ARCHIVE_DIR = paperless_dirs.archive
settings.THUMBNAIL_DIR = paperless_dirs.thumbnails
settings.MEDIA_LOCK = paperless_dirs.media / "media.lock"
settings.IGNORABLE_FILES = {".DS_Store", "Thumbs.db", "desktop.ini"}
settings.APP_LOGO = ""
@pytest.fixture()
def sample_doc(
paperless_dirs: PaperlessDirs,
_media_settings: None,
samples_dir: Path,
) -> "Document":
"""Create a document with valid files and matching checksums."""
with filelock.FileLock(paperless_dirs.media / "media.lock"):
shutil.copy(
samples_dir / "originals" / "0000001.pdf",
paperless_dirs.originals / "0000001.pdf",
)
shutil.copy(
samples_dir / "archive" / "0000001.pdf",
paperless_dirs.archive / "0000001.pdf",
)
shutil.copy(
samples_dir / "thumbnails" / "0000001.webp",
paperless_dirs.thumbnails / "0000001.webp",
)
return DocumentFactory(
title="test",
checksum="42995833e01aea9b3edee44bbfdd7ce1",
archive_checksum="62acb0bcbfbcaa62ca6ad3668e4e404b",
content="test content",
pk=1,
filename="0000001.pdf",
mime_type="application/pdf",
archive_filename="0000001.pdf",
)
@pytest.fixture()
def settings_timezone(settings: SettingsWrapper) -> zoneinfo.ZoneInfo:
@@ -28,3 +114,14 @@ def authenticated_rest_api_client(rest_api_client: APIClient):
user = UserModel.objects.create_user(username="testuser", password="password")
rest_api_client.force_authenticate(user=user)
yield rest_api_client
@pytest.fixture(scope="session", autouse=True)
def faker_session_locale():
"""Set Faker locale for reproducibility."""
return "en_US"
@pytest.fixture(scope="session", autouse=True)
def faker_seed():
return 12345

View File

@@ -24,7 +24,7 @@ def base_config() -> DateParserConfig:
12,
0,
0,
tzinfo=datetime.timezone.utc,
tzinfo=datetime.UTC,
),
filename_date_order="YMD",
content_date_order="DMY",
@@ -45,7 +45,7 @@ def config_with_ignore_dates() -> DateParserConfig:
12,
0,
0,
tzinfo=datetime.timezone.utc,
tzinfo=datetime.UTC,
),
filename_date_order="DMY",
content_date_order="MDY",

View File

@@ -101,50 +101,50 @@ class TestFilterDate:
[
# Valid Dates
pytest.param(
datetime.datetime(2024, 1, 10, tzinfo=datetime.timezone.utc),
datetime.datetime(2024, 1, 10, tzinfo=datetime.timezone.utc),
datetime.datetime(2024, 1, 10, tzinfo=datetime.UTC),
datetime.datetime(2024, 1, 10, tzinfo=datetime.UTC),
id="valid_past_date",
),
pytest.param(
datetime.datetime(2024, 1, 15, 12, 0, 0, tzinfo=datetime.timezone.utc),
datetime.datetime(2024, 1, 15, 12, 0, 0, tzinfo=datetime.timezone.utc),
datetime.datetime(2024, 1, 15, 12, 0, 0, tzinfo=datetime.UTC),
datetime.datetime(2024, 1, 15, 12, 0, 0, tzinfo=datetime.UTC),
id="exactly_at_reference",
),
pytest.param(
datetime.datetime(1901, 1, 1, tzinfo=datetime.timezone.utc),
datetime.datetime(1901, 1, 1, tzinfo=datetime.timezone.utc),
datetime.datetime(1901, 1, 1, tzinfo=datetime.UTC),
datetime.datetime(1901, 1, 1, tzinfo=datetime.UTC),
id="year_1901_valid",
),
# Date is > reference_time
pytest.param(
datetime.datetime(2024, 1, 16, tzinfo=datetime.timezone.utc),
datetime.datetime(2024, 1, 16, tzinfo=datetime.UTC),
None,
id="future_date_day_after",
),
# date.date() in ignore_dates
pytest.param(
datetime.datetime(2024, 1, 1, 0, 0, 0, tzinfo=datetime.timezone.utc),
datetime.datetime(2024, 1, 1, 0, 0, 0, tzinfo=datetime.UTC),
None,
id="ignored_date_midnight_jan1",
),
pytest.param(
datetime.datetime(2024, 1, 1, 10, 30, 0, tzinfo=datetime.timezone.utc),
datetime.datetime(2024, 1, 1, 10, 30, 0, tzinfo=datetime.UTC),
None,
id="ignored_date_midday_jan1",
),
pytest.param(
datetime.datetime(2024, 12, 25, 15, 0, 0, tzinfo=datetime.timezone.utc),
datetime.datetime(2024, 12, 25, 15, 0, 0, tzinfo=datetime.UTC),
None,
id="ignored_date_dec25_future",
),
# date.year <= 1900
pytest.param(
datetime.datetime(1899, 12, 31, tzinfo=datetime.timezone.utc),
datetime.datetime(1899, 12, 31, tzinfo=datetime.UTC),
None,
id="year_1899",
),
pytest.param(
datetime.datetime(1900, 1, 1, tzinfo=datetime.timezone.utc),
datetime.datetime(1900, 1, 1, tzinfo=datetime.UTC),
None,
id="year_1900_boundary",
),
@@ -176,7 +176,7 @@ class TestFilterDate:
1,
12,
0,
tzinfo=datetime.timezone.utc,
tzinfo=datetime.UTC,
)
another_ignored = datetime.datetime(
2024,
@@ -184,7 +184,7 @@ class TestFilterDate:
25,
15,
30,
tzinfo=datetime.timezone.utc,
tzinfo=datetime.UTC,
)
allowed_date = datetime.datetime(
2024,
@@ -192,7 +192,7 @@ class TestFilterDate:
2,
12,
0,
tzinfo=datetime.timezone.utc,
tzinfo=datetime.UTC,
)
assert parser._filter_date(ignored_date) is None
@@ -204,7 +204,7 @@ class TestFilterDate:
regex_parser: RegexDateParserPlugin,
) -> None:
"""Should work with timezone-aware datetimes."""
date_utc = datetime.datetime(2024, 1, 10, 12, 0, tzinfo=datetime.timezone.utc)
date_utc = datetime.datetime(2024, 1, 10, 12, 0, tzinfo=datetime.UTC)
result = regex_parser._filter_date(date_utc)
@@ -221,8 +221,8 @@ class TestRegexDateParser:
"report-2023-12-25.txt",
"Event recorded on 25/12/2022.",
[
datetime.datetime(2023, 12, 25, tzinfo=datetime.timezone.utc),
datetime.datetime(2022, 12, 25, tzinfo=datetime.timezone.utc),
datetime.datetime(2023, 12, 25, tzinfo=datetime.UTC),
datetime.datetime(2022, 12, 25, tzinfo=datetime.UTC),
],
id="filename-y-m-d_and_content-d-m-y",
),
@@ -230,8 +230,8 @@ class TestRegexDateParser:
"img_2023.01.02.jpg",
"Taken on 01/02/2023",
[
datetime.datetime(2023, 1, 2, tzinfo=datetime.timezone.utc),
datetime.datetime(2023, 2, 1, tzinfo=datetime.timezone.utc),
datetime.datetime(2023, 1, 2, tzinfo=datetime.UTC),
datetime.datetime(2023, 2, 1, tzinfo=datetime.UTC),
],
id="ambiguous-dates-respect-orders",
),
@@ -239,7 +239,7 @@ class TestRegexDateParser:
"notes.txt",
"bad date 99/99/9999 and 25/12/2022",
[
datetime.datetime(2022, 12, 25, tzinfo=datetime.timezone.utc),
datetime.datetime(2022, 12, 25, tzinfo=datetime.UTC),
],
id="parse-exception-skips-bad-and-yields-good",
),
@@ -275,24 +275,24 @@ class TestRegexDateParser:
or "2023.12.25" in date_string
or "2023-12-25" in date_string
):
return datetime.datetime(2023, 12, 25, tzinfo=datetime.timezone.utc)
return datetime.datetime(2023, 12, 25, tzinfo=datetime.UTC)
# content DMY 25/12/2022
if "25/12/2022" in date_string or "25-12-2022" in date_string:
return datetime.datetime(2022, 12, 25, tzinfo=datetime.timezone.utc)
return datetime.datetime(2022, 12, 25, tzinfo=datetime.UTC)
# filename YMD 2023.01.02
if "2023.01.02" in date_string or "2023-01-02" in date_string:
return datetime.datetime(2023, 1, 2, tzinfo=datetime.timezone.utc)
return datetime.datetime(2023, 1, 2, tzinfo=datetime.UTC)
# ambiguous 01/02/2023 -> respect DATE_ORDER setting
if "01/02/2023" in date_string:
if date_order == "DMY":
return datetime.datetime(2023, 2, 1, tzinfo=datetime.timezone.utc)
return datetime.datetime(2023, 2, 1, tzinfo=datetime.UTC)
if date_order == "YMD":
return datetime.datetime(2023, 1, 2, tzinfo=datetime.timezone.utc)
return datetime.datetime(2023, 1, 2, tzinfo=datetime.UTC)
# fallback
return datetime.datetime(2023, 2, 1, tzinfo=datetime.timezone.utc)
return datetime.datetime(2023, 2, 1, tzinfo=datetime.UTC)
# simulate parse failure for malformed input
if "99/99/9999" in date_string or "bad date" in date_string:
@@ -328,7 +328,7 @@ class TestRegexDateParser:
12,
0,
0,
tzinfo=datetime.timezone.utc,
tzinfo=datetime.UTC,
),
filename_date_order="YMD",
content_date_order="DMY",
@@ -344,13 +344,13 @@ class TestRegexDateParser:
) -> datetime.datetime | None:
if "10/12/2023" in date_string or "10-12-2023" in date_string:
# ignored date
return datetime.datetime(2023, 12, 10, tzinfo=datetime.timezone.utc)
return datetime.datetime(2023, 12, 10, tzinfo=datetime.UTC)
if "01/02/2024" in date_string or "01-02-2024" in date_string:
# future relative to reference_time -> filtered
return datetime.datetime(2024, 2, 1, tzinfo=datetime.timezone.utc)
return datetime.datetime(2024, 2, 1, tzinfo=datetime.UTC)
if "05/01/2023" in date_string or "05-01-2023" in date_string:
# valid
return datetime.datetime(2023, 1, 5, tzinfo=datetime.timezone.utc)
return datetime.datetime(2023, 1, 5, tzinfo=datetime.UTC)
return None
mocker.patch(target, side_effect=fake_parse)
@@ -358,7 +358,7 @@ class TestRegexDateParser:
content = "Ignored: 10/12/2023, Future: 01/02/2024, Keep: 05/01/2023"
results = list(parser.parse("whatever.txt", content))
assert results == [datetime.datetime(2023, 1, 5, tzinfo=datetime.timezone.utc)]
assert results == [datetime.datetime(2023, 1, 5, tzinfo=datetime.UTC)]
def test_parse_handles_no_matches_and_returns_empty_list(
self,
@@ -392,7 +392,7 @@ class TestRegexDateParser:
12,
0,
0,
tzinfo=datetime.timezone.utc,
tzinfo=datetime.UTC,
),
filename_date_order=None,
content_date_order="DMY",
@@ -409,9 +409,9 @@ class TestRegexDateParser:
) -> datetime.datetime | None:
# return distinct datetimes so we can tell which source was parsed
if "25/12/2022" in date_string:
return datetime.datetime(2022, 12, 25, tzinfo=datetime.timezone.utc)
return datetime.datetime(2022, 12, 25, tzinfo=datetime.UTC)
if "2023-12-25" in date_string:
return datetime.datetime(2023, 12, 25, tzinfo=datetime.timezone.utc)
return datetime.datetime(2023, 12, 25, tzinfo=datetime.UTC)
return None
mock = mocker.patch(target, side_effect=fake_parse)
@@ -429,5 +429,5 @@ class TestRegexDateParser:
assert "25/12/2022" in called_date_string
# And the parser should have yielded the corresponding datetime
assert results == [
datetime.datetime(2022, 12, 25, tzinfo=datetime.timezone.utc),
datetime.datetime(2022, 12, 25, tzinfo=datetime.UTC),
]

View File

@@ -1,17 +1,67 @@
from factory import Faker
"""
Factory-boy factories for documents app models.
"""
from __future__ import annotations
import factory
from factory.django import DjangoModelFactory
from documents.models import Correspondent
from documents.models import Document
from documents.models import DocumentType
from documents.models import MatchingModel
from documents.models import StoragePath
from documents.models import Tag
class CorrespondentFactory(DjangoModelFactory):
class Meta:
model = Correspondent
name = Faker("name")
name = factory.Sequence(lambda n: f"{factory.Faker('company')} {n}")
match = ""
matching_algorithm = MatchingModel.MATCH_NONE
class DocumentTypeFactory(DjangoModelFactory):
class Meta:
model = DocumentType
name = factory.Sequence(lambda n: f"{factory.Faker('bs')} {n}")
match = ""
matching_algorithm = MatchingModel.MATCH_NONE
class TagFactory(DjangoModelFactory):
class Meta:
model = Tag
name = factory.Sequence(lambda n: f"{factory.Faker('word')} {n}")
match = ""
matching_algorithm = MatchingModel.MATCH_NONE
is_inbox_tag = False
class StoragePathFactory(DjangoModelFactory):
class Meta:
model = StoragePath
name = factory.Sequence(
lambda n: f"{factory.Faker('file_path', depth=2, extension='')} {n}",
)
path = factory.LazyAttribute(lambda o: f"{o.name}/{{title}}")
match = ""
matching_algorithm = MatchingModel.MATCH_NONE
class DocumentFactory(DjangoModelFactory):
class Meta:
model = Document
title = factory.Faker("sentence", nb_words=4)
checksum = factory.Faker("md5")
content = factory.Faker("paragraph")
correspondent = None
document_type = None
storage_path = None

View File

@@ -0,0 +1,193 @@
"""Tests for the document_sanity_checker management command.
Verifies Rich rendering (table, panel, summary) and end-to-end CLI behavior.
"""
from __future__ import annotations
from io import StringIO
from pathlib import Path
from typing import TYPE_CHECKING
import pytest
from django.core.management import call_command
from rich.console import Console
from documents.management.commands.document_sanity_checker import Command
from documents.sanity_checker import SanityCheckMessages
from documents.tests.factories import DocumentFactory
if TYPE_CHECKING:
from documents.models import Document
from documents.tests.conftest import PaperlessDirs
def _render_to_string(messages: SanityCheckMessages) -> str:
"""Render command output to a plain string for assertion."""
buf = StringIO()
cmd = Command()
cmd.console = Console(file=buf, width=120, no_color=True)
cmd._render_results(messages)
return buf.getvalue()
# ---------------------------------------------------------------------------
# Rich rendering
# ---------------------------------------------------------------------------
class TestRenderResultsNoIssues:
"""No DB access needed -- renders an empty SanityCheckMessages."""
def test_shows_panel(self) -> None:
output = _render_to_string(SanityCheckMessages())
assert "No issues detected" in output
assert "Sanity Check" in output
@pytest.mark.django_db
class TestRenderResultsWithIssues:
def test_error_row(self, sample_doc: Document) -> None:
msgs = SanityCheckMessages()
msgs.error(sample_doc.pk, "Original missing")
output = _render_to_string(msgs)
assert "Sanity Check Results" in output
assert "ERROR" in output
assert "Original missing" in output
assert f"#{sample_doc.pk}" in output
assert sample_doc.title in output
def test_warning_row(self, sample_doc: Document) -> None:
msgs = SanityCheckMessages()
msgs.warning(sample_doc.pk, "Suspicious file")
output = _render_to_string(msgs)
assert "WARN" in output
assert "Suspicious file" in output
def test_info_row(self, sample_doc: Document) -> None:
msgs = SanityCheckMessages()
msgs.info(sample_doc.pk, "No OCR data")
output = _render_to_string(msgs)
assert "INFO" in output
assert "No OCR data" in output
@pytest.mark.usefixtures("_media_settings")
def test_global_message(self) -> None:
msgs = SanityCheckMessages()
msgs.warning(None, "Orphaned file: /tmp/stray.pdf")
output = _render_to_string(msgs)
assert "(global)" in output
assert "Orphaned file" in output
def test_multiple_messages_same_doc(self, sample_doc: Document) -> None:
msgs = SanityCheckMessages()
msgs.error(sample_doc.pk, "Thumbnail missing")
msgs.error(sample_doc.pk, "Checksum mismatch")
output = _render_to_string(msgs)
assert "Thumbnail missing" in output
assert "Checksum mismatch" in output
@pytest.mark.usefixtures("_media_settings")
def test_unknown_doc_pk(self) -> None:
msgs = SanityCheckMessages()
msgs.error(99999, "Ghost document")
output = _render_to_string(msgs)
assert "#99999" in output
assert "Unknown" in output
@pytest.mark.django_db
class TestRenderResultsSummary:
def test_errors_only(self, sample_doc: Document) -> None:
msgs = SanityCheckMessages()
msgs.error(sample_doc.pk, "broken")
output = _render_to_string(msgs)
assert "1 document(s) with" in output
assert "errors" in output
def test_warnings_only(self, sample_doc: Document) -> None:
msgs = SanityCheckMessages()
msgs.warning(sample_doc.pk, "odd")
output = _render_to_string(msgs)
assert "1 document(s) with" in output
assert "warnings" in output
def test_infos_only(self, sample_doc: Document) -> None:
msgs = SanityCheckMessages()
msgs.info(sample_doc.pk, "no OCR")
output = _render_to_string(msgs)
assert "1 document(s) with infos" in output
def test_empty_messages(self) -> None:
msgs = SanityCheckMessages()
output = _render_to_string(msgs)
assert "No issues detected." in output
def test_document_errors_and_global_warnings(self, sample_doc: Document) -> None:
msgs = SanityCheckMessages()
msgs.error(sample_doc.pk, "broken")
msgs.warning(None, "orphan")
output = _render_to_string(msgs)
assert "1 document(s) with" in output
assert "errors" in output
assert "1 global warning(s)" in output
assert "2 document(s)" not in output
def test_global_warnings_only(self) -> None:
msgs = SanityCheckMessages()
msgs.warning(None, "extra file")
output = _render_to_string(msgs)
assert "1 global warning(s)" in output
assert "document(s) with" not in output
def test_all_levels_combined(self, sample_doc: Document) -> None:
msgs = SanityCheckMessages()
msgs.error(sample_doc.pk, "broken")
msgs.warning(sample_doc.pk, "odd")
msgs.info(sample_doc.pk, "fyi")
msgs.warning(None, "extra file")
output = _render_to_string(msgs)
assert "1 document(s) with errors" in output
assert "1 document(s) with warnings" in output
assert "1 document(s) with infos" in output
assert "1 global warning(s)" in output
# ---------------------------------------------------------------------------
# End-to-end command execution
# ---------------------------------------------------------------------------
@pytest.mark.django_db
@pytest.mark.management
class TestDocumentSanityCheckerCommand:
def test_no_issues(self, sample_doc: Document) -> None:
out = StringIO()
call_command("document_sanity_checker", "--no-progress-bar", stdout=out)
assert "No issues detected" in out.getvalue()
def test_missing_original(self, sample_doc: Document) -> None:
Path(sample_doc.source_path).unlink()
out = StringIO()
call_command("document_sanity_checker", "--no-progress-bar", stdout=out)
output = out.getvalue()
assert "ERROR" in output
assert "Original of document does not exist" in output
@pytest.mark.usefixtures("_media_settings")
def test_checksum_mismatch(self, paperless_dirs: PaperlessDirs) -> None:
"""Lightweight document with zero-byte files triggers checksum mismatch."""
doc = DocumentFactory(
title="test",
content="test",
filename="test.pdf",
checksum="abc",
)
Path(doc.source_path).touch()
Path(doc.thumbnail_path).touch()
out = StringIO()
call_command("document_sanity_checker", "--no-progress-bar", stdout=out)
output = out.getvalue()
assert "ERROR" in output
assert "Checksum mismatch. Stored: abc, actual:" in output

View File

@@ -1270,7 +1270,11 @@ class TestDocumentApi(DirectoriesMixin, DocumentConsumeDelayMixin, APITestCase):
input_doc, overrides = self.get_last_consume_delay_call_args()
self.assertEqual(input_doc.original_file.name, "simple.pdf")
self.assertIn(Path(settings.SCRATCH_DIR), input_doc.original_file.parents)
self.assertTrue(
input_doc.original_file.resolve(strict=False).is_relative_to(
Path(settings.SCRATCH_DIR).resolve(strict=False),
),
)
self.assertIsNone(overrides.title)
self.assertIsNone(overrides.correspondent_id)
self.assertIsNone(overrides.document_type_id)
@@ -1351,7 +1355,11 @@ class TestDocumentApi(DirectoriesMixin, DocumentConsumeDelayMixin, APITestCase):
input_doc, overrides = self.get_last_consume_delay_call_args()
self.assertEqual(input_doc.original_file.name, "simple.pdf")
self.assertIn(Path(settings.SCRATCH_DIR), input_doc.original_file.parents)
self.assertTrue(
input_doc.original_file.resolve(strict=False).is_relative_to(
Path(settings.SCRATCH_DIR).resolve(strict=False),
),
)
self.assertIsNone(overrides.title)
self.assertIsNone(overrides.correspondent_id)
self.assertIsNone(overrides.document_type_id)
@@ -2110,69 +2118,93 @@ class TestDocumentApi(DirectoriesMixin, DocumentConsumeDelayMixin, APITestCase):
mock_get_date_parser.assert_not_called()
def test_saved_views(self) -> None:
u1 = User.objects.create_superuser("user1")
u2 = User.objects.create_superuser("user2")
u1 = User.objects.create_user("user1")
u2 = User.objects.create_user("user2")
u3 = User.objects.create_user("user3")
view_perm = Permission.objects.get(codename="view_savedview")
change_perm = Permission.objects.get(codename="change_savedview")
for user in [u1, u2, u3]:
user.user_permissions.add(view_perm, change_perm)
v1 = SavedView.objects.create(
owner=u1,
name="test1",
sort_field="",
show_on_dashboard=False,
show_in_sidebar=False,
)
SavedView.objects.create(
v2 = SavedView.objects.create(
owner=u2,
name="test2",
sort_field="",
show_on_dashboard=False,
show_in_sidebar=False,
)
SavedView.objects.create(
v3 = SavedView.objects.create(
owner=u2,
name="test3",
sort_field="",
show_on_dashboard=False,
show_in_sidebar=False,
)
response = self.client.get("/api/saved_views/")
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertEqual(response.data["count"], 0)
self.assertEqual(
self.client.get(f"/api/saved_views/{v1.id}/").status_code,
status.HTTP_404_NOT_FOUND,
)
assign_perm("view_savedview", u1, v2)
assign_perm("change_savedview", u1, v2)
assign_perm("view_savedview", u1, v3)
self.client.force_authenticate(user=u1)
response = self.client.get("/api/saved_views/")
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertEqual(response.data["count"], 1)
self.assertEqual(response.data["count"], 3)
for view_id in [v1.id, v2.id, v3.id]:
self.assertEqual(
self.client.get(f"/api/saved_views/{view_id}/").status_code,
status.HTTP_200_OK,
)
response = self.client.patch(
f"/api/saved_views/{v2.id}/",
{"sort_field": "added"},
format="json",
)
self.assertEqual(response.status_code, status.HTTP_200_OK)
response = self.client.patch(
f"/api/saved_views/{v3.id}/",
{"sort_field": "added"},
format="json",
)
self.assertEqual(
self.client.get(f"/api/saved_views/{v1.id}/").status_code,
status.HTTP_200_OK,
response.status_code,
status.HTTP_403_FORBIDDEN,
)
self.client.force_authenticate(user=u2)
response = self.client.patch(
f"/api/saved_views/{v2.id}/",
{
"set_permissions": {
"view": {"users": [u3.id]},
},
},
format="json",
)
self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN)
response = self.client.patch(
f"/api/saved_views/{v2.id}/",
{"owner": u1.id},
format="json",
)
self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN)
self.client.force_authenticate(user=u3)
response = self.client.get("/api/saved_views/")
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertEqual(response.data["count"], 2)
self.assertEqual(
self.client.get(f"/api/saved_views/{v1.id}/").status_code,
status.HTTP_404_NOT_FOUND,
)
self.assertEqual(response.data["count"], 0)
def test_saved_view_create_update_patch(self) -> None:
User.objects.create_user("user1")
view = {
"name": "test",
"show_on_dashboard": True,
"show_in_sidebar": True,
"sort_field": "created2",
"filter_rules": [{"rule_type": 4, "value": "test"}],
}
@@ -2187,13 +2219,13 @@ class TestDocumentApi(DirectoriesMixin, DocumentConsumeDelayMixin, APITestCase):
response = self.client.patch(
f"/api/saved_views/{v1.id}/",
{"show_in_sidebar": False},
{"sort_reverse": True},
format="json",
)
v1 = SavedView.objects.get(id=v1.id)
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertFalse(v1.show_in_sidebar)
self.assertTrue(v1.sort_reverse)
self.assertEqual(v1.filter_rules.count(), 1)
view["filter_rules"] = [{"rule_type": 12, "value": "secret"}]
@@ -2227,8 +2259,6 @@ class TestDocumentApi(DirectoriesMixin, DocumentConsumeDelayMixin, APITestCase):
view = {
"name": "test",
"show_on_dashboard": True,
"show_in_sidebar": True,
"sort_field": "created2",
"filter_rules": [{"rule_type": 4, "value": "test"}],
"page_size": 20,
@@ -2316,8 +2346,6 @@ class TestDocumentApi(DirectoriesMixin, DocumentConsumeDelayMixin, APITestCase):
"""
view = {
"name": "test",
"show_on_dashboard": True,
"show_in_sidebar": True,
"sort_field": "created2",
"filter_rules": [{"rule_type": 4, "value": "test"}],
"page_size": 20,
@@ -2393,8 +2421,6 @@ class TestDocumentApi(DirectoriesMixin, DocumentConsumeDelayMixin, APITestCase):
owner=self.user,
name="test",
sort_field=SavedView.DisplayFields.CUSTOM_FIELD % custom_field.id,
show_on_dashboard=True,
show_in_sidebar=True,
display_fields=[
SavedView.DisplayFields.TITLE,
SavedView.DisplayFields.CREATED,

View File

@@ -1307,13 +1307,12 @@ class TestDocumentSearchApi(DirectoriesMixin, APITestCase):
tag1 = Tag.objects.create(name="bank tag1")
Tag.objects.create(name="tag2")
SavedView.objects.create(
shared_view = SavedView.objects.create(
name="bank view",
show_on_dashboard=True,
show_in_sidebar=True,
sort_field="",
owner=user1,
owner=user2,
)
assign_perm("view_savedview", user1, shared_view)
mail_account1 = MailAccount.objects.create(name="bank mail account 1")
mail_account2 = MailAccount.objects.create(name="mail account 2")
mail_rule1 = MailRule.objects.create(

View File

@@ -641,6 +641,33 @@ class TestConsumer(
self._assert_first_last_send_progress()
@mock.patch("documents.consumer.generate_unique_filename")
def testFilenameHandlingFallsBackWhenGeneratedPathExceedsDbLimit(self, m):
m.side_effect = lambda doc, archive_filename=False: Path(
("a" * 1100 + ".pdf") if not archive_filename else ("b" * 1100 + ".pdf"),
)
with self.get_consumer(
self.get_test_file(),
DocumentMetadataOverrides(title="new docs"),
) as consumer:
consumer.run()
document = Document.objects.first()
self.assertIsNotNone(document)
assert document is not None
self.assertEqual(document.filename, f"{document.pk:07d}.pdf")
self.assertLessEqual(len(document.filename), 1024)
self.assertLessEqual(
len(document.archive_filename),
1024,
)
self.assertIsFile(document.source_path)
self.assertIsFile(document.archive_path)
self._assert_first_last_send_progress()
@override_settings(FILENAME_FORMAT="{correspondent}/{title}")
@mock.patch("documents.signals.handlers.generate_unique_filename")
def testFilenameHandlingUnstableFormat(self, m) -> None:
@@ -699,6 +726,14 @@ class TestConsumer(
self.assertIsNotNone(root_doc)
assert root_doc is not None
root_storage_path = StoragePath.objects.create(
name="version-root-path",
path="root/{{title}}",
)
root_doc.storage_path = root_storage_path
root_doc.archive_serial_number = 42
root_doc.save()
actor = User.objects.create_user(
username="actor",
email="actor@example.com",
@@ -735,7 +770,7 @@ class TestConsumer(
)
consumer.setup()
try:
self.assertTrue(consumer.filename.endswith("_v0.pdf"))
self.assertEqual(consumer.filename, version_file.name)
consumer.run()
finally:
consumer.cleanup()
@@ -745,8 +780,10 @@ class TestConsumer(
version = versions.first()
assert version is not None
assert version.original_filename is not None
self.assertEqual(version.version_index, 1)
self.assertEqual(version.version_label, "v2")
self.assertTrue(version.original_filename.endswith("_v0.pdf"))
self.assertIsNone(version.archive_serial_number)
self.assertEqual(version.original_filename, version_file.name)
self.assertTrue(bool(version.content))
@override_settings(AUDIT_LOG_ENABLED=True)
@@ -795,7 +832,7 @@ class TestConsumer(
)
consumer.setup()
try:
self.assertEqual(consumer.filename, "valid_pdf_version-upload_v0")
self.assertEqual(consumer.filename, "valid_pdf_version-upload")
consumer.run()
finally:
consumer.cleanup()
@@ -805,9 +842,67 @@ class TestConsumer(
)
self.assertIsNotNone(version)
assert version is not None
self.assertEqual(version.original_filename, "valid_pdf_version-upload_v0")
self.assertEqual(version.version_index, 1)
self.assertEqual(version.original_filename, "valid_pdf_version-upload")
self.assertTrue(bool(version.content))
@override_settings(AUDIT_LOG_ENABLED=True)
@mock.patch("documents.consumer.load_classifier")
def test_consume_version_index_monotonic_after_version_deletion(self, m) -> None:
m.return_value = MagicMock()
with self.get_consumer(self.get_test_file()) as consumer:
consumer.run()
root_doc = Document.objects.first()
self.assertIsNotNone(root_doc)
assert root_doc is not None
def consume_version(version_file: Path) -> Document:
status = DummyProgressManager(version_file.name, None)
overrides = DocumentMetadataOverrides()
doc = ConsumableDocument(
DocumentSource.ApiUpload,
original_file=version_file,
root_document_id=root_doc.pk,
)
preflight = ConsumerPreflightPlugin(
doc,
overrides,
status, # type: ignore[arg-type]
self.dirs.scratch_dir,
"task-id",
)
preflight.setup()
preflight.run()
consumer = ConsumerPlugin(
doc,
overrides,
status, # type: ignore[arg-type]
self.dirs.scratch_dir,
"task-id",
)
consumer.setup()
try:
consumer.run()
finally:
consumer.cleanup()
version = (
Document.objects.filter(root_document=root_doc).order_by("-id").first()
)
assert version is not None
return version
v1 = consume_version(self.get_test_file2())
self.assertEqual(v1.version_index, 1)
v1.delete()
# The next version should have version_index 2, even though version_index 1 was deleted
v2 = consume_version(self.get_test_file())
self.assertEqual(v2.version_index, 2)
@mock.patch("documents.consumer.load_classifier")
def testClassifyDocument(self, m) -> None:
correspondent = Correspondent.objects.create(

View File

@@ -77,6 +77,58 @@ class TestFileHandling(DirectoriesMixin, FileSystemAssertsMixin, TestCase):
settings.ORIGINALS_DIR / "test" / "test.pdf",
)
@override_settings(FILENAME_FORMAT=None)
def test_root_storage_path_change_updates_version_files(self) -> None:
old_storage_path = StoragePath.objects.create(
name="old-path",
path="old/{{title}}",
)
new_storage_path = StoragePath.objects.create(
name="new-path",
path="new/{{title}}",
)
root_doc = Document.objects.create(
title="rootdoc",
mime_type="application/pdf",
checksum="root-checksum",
storage_path=old_storage_path,
)
version_doc = Document.objects.create(
title="version-title",
mime_type="application/pdf",
checksum="version-checksum",
root_document=root_doc,
version_index=1,
)
Document.objects.filter(pk=root_doc.pk).update(
filename=generate_filename(root_doc),
)
Document.objects.filter(pk=version_doc.pk).update(
filename=generate_filename(version_doc),
)
root_doc.refresh_from_db()
version_doc.refresh_from_db()
create_source_path_directory(root_doc.source_path)
Path(root_doc.source_path).touch()
create_source_path_directory(version_doc.source_path)
Path(version_doc.source_path).touch()
root_doc.storage_path = new_storage_path
root_doc.save()
root_doc.refresh_from_db()
version_doc.refresh_from_db()
self.assertEqual(root_doc.filename, "new/rootdoc.pdf")
self.assertEqual(version_doc.filename, "new/rootdoc_v1.pdf")
self.assertIsFile(root_doc.source_path)
self.assertIsFile(version_doc.source_path)
self.assertIsNotFile(settings.ORIGINALS_DIR / "old" / "rootdoc.pdf")
self.assertIsNotFile(settings.ORIGINALS_DIR / "old" / "rootdoc_v1.pdf")
@override_settings(FILENAME_FORMAT="{correspondent}/{correspondent}")
def test_file_renaming_missing_permissions(self) -> None:
document = Document()
@@ -336,7 +388,11 @@ class TestFileHandling(DirectoriesMixin, FileSystemAssertsMixin, TestCase):
added=d1,
)
self.assertEqual(generate_filename(doc1), Path("1232-01-09.pdf"))
# Account for 3.14 padding changes
expected_year: str = d1.strftime("%Y")
expected_filename: Path = Path(f"{expected_year}-01-09.pdf")
self.assertEqual(generate_filename(doc1), expected_filename)
doc1.added = timezone.make_aware(datetime.datetime(2020, 11, 16, 1, 1, 1))
@@ -1222,6 +1278,94 @@ class TestFilenameGeneration(DirectoriesMixin, TestCase):
Path("logs.pdf"),
)
@override_settings(FILENAME_FORMAT="{title}")
def test_version_index_suffix_for_template_filename(self) -> None:
root_doc = Document.objects.create(
title="the_doc",
mime_type="application/pdf",
checksum="root-checksum",
)
version_doc = Document.objects.create(
title="the_doc",
mime_type="application/pdf",
checksum="version-checksum",
root_document=root_doc,
version_index=1,
)
self.assertEqual(generate_filename(version_doc), Path("the_doc_v1.pdf"))
self.assertEqual(
generate_filename(version_doc, counter=1),
Path("the_doc_v1_01.pdf"),
)
@override_settings(FILENAME_FORMAT=None)
def test_version_index_suffix_for_default_filename(self) -> None:
root_doc = Document.objects.create(
title="root",
mime_type="text/plain",
checksum="root-checksum",
)
version_doc = Document.objects.create(
title="root",
mime_type="text/plain",
checksum="version-checksum",
root_document=root_doc,
version_index=2,
)
self.assertEqual(
generate_filename(version_doc),
Path(f"{root_doc.pk:07d}_v2.txt"),
)
self.assertEqual(
generate_filename(version_doc, archive_filename=True),
Path(f"{root_doc.pk:07d}_v2.pdf"),
)
@override_settings(FILENAME_FORMAT="{original_name}")
def test_version_index_suffix_with_original_name_placeholder(self) -> None:
root_doc = Document.objects.create(
title="root",
mime_type="application/pdf",
checksum="root-checksum",
original_filename="root-upload.pdf",
)
version_doc = Document.objects.create(
title="root",
mime_type="application/pdf",
checksum="version-checksum",
root_document=root_doc,
version_index=1,
original_filename="version-upload.pdf",
)
self.assertEqual(generate_filename(version_doc), Path("root-upload_v1.pdf"))
def test_version_index_suffix_with_storage_path(self) -> None:
storage_path = StoragePath.objects.create(
name="vtest",
path="folder/{{title}}",
)
root_doc = Document.objects.create(
title="storage_doc",
mime_type="application/pdf",
checksum="root-checksum",
storage_path=storage_path,
)
version_doc = Document.objects.create(
title="version_title_should_not_be_used",
mime_type="application/pdf",
checksum="version-checksum",
root_document=root_doc,
version_index=3,
)
self.assertEqual(
generate_filename(version_doc),
Path("folder/storage_doc_v3.pdf"),
)
@override_settings(
FILENAME_FORMAT="XX{correspondent}/{title}",
FILENAME_FORMAT_REMOVE_NONE=True,
@@ -1323,6 +1467,41 @@ class TestFilenameGeneration(DirectoriesMixin, TestCase):
Path("somepath/asn-201-400/asn-3xx/Does Matter.pdf"),
)
def test_template_related_context_keeps_legacy_string_coercion(self):
"""
GIVEN:
- A storage path template that uses related objects directly as strings
WHEN:
- Filepath for a document with this format is called
THEN:
- Related objects coerce to their names (legacy behavior)
- Explicit attribute access remains available for new templates
"""
sp = StoragePath.objects.create(
name="PARTNER",
path=(
"{{ document.storage_path|lower }} / "
"{{ document.correspondent|lower|replace('mi:', 'mieter/') }} / "
"{{ document_type|lower }} / "
"{{ title|lower }}"
),
)
doc = Document.objects.create(
title="scan_017562",
created=datetime.date(2025, 7, 2),
added=timezone.make_aware(datetime.datetime(2026, 3, 3, 11, 53, 16)),
mime_type="application/pdf",
checksum="test-checksum",
storage_path=sp,
correspondent=Correspondent.objects.create(name="mi:kochkach"),
document_type=DocumentType.objects.create(name="Mietvertrag"),
)
self.assertEqual(
generate_filename(doc),
Path("partner/mieter/kochkach/mietvertrag/scan_017562.pdf"),
)
@override_settings(
FILENAME_FORMAT="{{creation_date}}/{{ title_name_str }}",
)
@@ -1681,6 +1860,21 @@ class TestCustomFieldFilenameUpdates(
self.assertTrue(Path(self.doc.source_path).is_file())
self.assertLessEqual(m.call_count, 1)
@override_settings(FILENAME_FORMAT=None)
def test_overlong_storage_path_keeps_existing_filename(self):
initial_filename = generate_filename(self.doc)
Document.objects.filter(pk=self.doc.pk).update(filename=str(initial_filename))
self.doc.refresh_from_db()
Path(self.doc.source_path).parent.mkdir(parents=True, exist_ok=True)
Path(self.doc.source_path).touch()
self.doc.storage_path = StoragePath.objects.create(path="a" * 1100)
self.doc.save()
self.doc.refresh_from_db()
self.assertEqual(Path(self.doc.filename), initial_filename)
self.assertTrue(Path(self.doc.source_path).is_file())
class TestPathDateLocalization:
"""

View File

@@ -21,7 +21,7 @@ class TestDateLocalization:
14,
30,
5,
tzinfo=datetime.timezone.utc,
tzinfo=datetime.UTC,
)
TEST_DATETIME_STRING: str = "2023-10-26T14:30:05+00:00"

View File

@@ -134,6 +134,7 @@ class TestRenamer(DirectoriesMixin, FileSystemAssertsMixin, TestCase):
self.assertIsFile(doc2.archive_path)
@pytest.mark.management
class TestCreateClassifier(TestCase):
@mock.patch(
"documents.management.commands.document_create_classifier.train_classifier",
@@ -144,32 +145,6 @@ class TestCreateClassifier(TestCase):
m.assert_called_once()
@pytest.mark.management
class TestSanityChecker(DirectoriesMixin, TestCase):
def test_no_issues(self) -> None:
with self.assertLogs() as capture:
call_command("document_sanity_checker")
self.assertEqual(len(capture.output), 1)
self.assertIn("Sanity checker detected no issues.", capture.output[0])
def test_errors(self) -> None:
doc = Document.objects.create(
title="test",
content="test",
filename="test.pdf",
checksum="abc",
)
Path(doc.source_path).touch()
Path(doc.thumbnail_path).touch()
with self.assertLogs() as capture:
call_command("document_sanity_checker")
self.assertEqual(len(capture.output), 2)
self.assertIn("Checksum mismatch. Stored: abc, actual:", capture.output[1])
@pytest.mark.management
class TestConvertMariaDBUUID(TestCase):
@mock.patch("django.db.connection.schema_editor")

View File

@@ -288,7 +288,7 @@ class TestExportImport(
self.assertEqual(Permission.objects.count(), num_permission_objects)
messages = check_sanity()
# everything is alright after the test
self.assertEqual(len(messages), 0)
self.assertEqual(messages.total_issue_count, 0)
def test_exporter_with_filename_format(self) -> None:
shutil.rmtree(Path(self.dirs.media_dir) / "documents")

View File

@@ -1,298 +1,442 @@
"""
Tests for the document_retagger management command.
"""
from __future__ import annotations
import pytest
from django.core.management import call_command
from django.core.management.base import CommandError
from django.test import TestCase
from documents.models import Correspondent
from documents.models import Document
from documents.models import DocumentType
from documents.models import MatchingModel
from documents.models import StoragePath
from documents.models import Tag
from documents.tests.factories import CorrespondentFactory
from documents.tests.factories import DocumentFactory
from documents.tests.factories import DocumentTypeFactory
from documents.tests.factories import StoragePathFactory
from documents.tests.factories import TagFactory
from documents.tests.utils import DirectoriesMixin
# ---------------------------------------------------------------------------
# Module-level type aliases
# ---------------------------------------------------------------------------
StoragePathTuple = tuple[StoragePath, StoragePath, StoragePath]
TagTuple = tuple[Tag, Tag, Tag, Tag, Tag]
CorrespondentTuple = tuple[Correspondent, Correspondent]
DocumentTypeTuple = tuple[DocumentType, DocumentType]
DocumentTuple = tuple[Document, Document, Document, Document]
# ---------------------------------------------------------------------------
# Fixtures
# ---------------------------------------------------------------------------
@pytest.fixture()
def storage_paths(db) -> StoragePathTuple:
"""Three storage paths with varying match rules."""
sp1 = StoragePathFactory(
path="{created_data}/{title}",
match="auto document",
matching_algorithm=MatchingModel.MATCH_LITERAL,
)
sp2 = StoragePathFactory(
path="{title}",
match="^first|^unrelated",
matching_algorithm=MatchingModel.MATCH_REGEX,
)
sp3 = StoragePathFactory(
path="{title}",
match="^blah",
matching_algorithm=MatchingModel.MATCH_REGEX,
)
return sp1, sp2, sp3
@pytest.fixture()
def tags(db) -> TagTuple:
"""Tags covering the common matching scenarios."""
tag_first = TagFactory(match="first", matching_algorithm=Tag.MATCH_ANY)
tag_second = TagFactory(match="second", matching_algorithm=Tag.MATCH_ANY)
tag_inbox = TagFactory(is_inbox_tag=True)
tag_no_match = TagFactory()
tag_auto = TagFactory(matching_algorithm=Tag.MATCH_AUTO)
return tag_first, tag_second, tag_inbox, tag_no_match, tag_auto
@pytest.fixture()
def correspondents(db) -> CorrespondentTuple:
"""Two correspondents matching 'first' and 'second' content."""
c_first = CorrespondentFactory(
match="first",
matching_algorithm=MatchingModel.MATCH_ANY,
)
c_second = CorrespondentFactory(
match="second",
matching_algorithm=MatchingModel.MATCH_ANY,
)
return c_first, c_second
@pytest.fixture()
def document_types(db) -> DocumentTypeTuple:
"""Two document types matching 'first' and 'second' content."""
dt_first = DocumentTypeFactory(
match="first",
matching_algorithm=MatchingModel.MATCH_ANY,
)
dt_second = DocumentTypeFactory(
match="second",
matching_algorithm=MatchingModel.MATCH_ANY,
)
return dt_first, dt_second
@pytest.fixture()
def documents(storage_paths: StoragePathTuple, tags: TagTuple) -> DocumentTuple:
"""Four documents with varied content used across most retagger tests."""
_, _, sp3 = storage_paths
_, _, tag_inbox, tag_no_match, tag_auto = tags
d1 = DocumentFactory(checksum="A", title="A", content="first document")
d2 = DocumentFactory(checksum="B", title="B", content="second document")
d3 = DocumentFactory(
checksum="C",
title="C",
content="unrelated document",
storage_path=sp3,
)
d4 = DocumentFactory(checksum="D", title="D", content="auto document")
d3.tags.add(tag_inbox, tag_no_match)
d4.tags.add(tag_auto)
return d1, d2, d3, d4
def _get_docs() -> DocumentTuple:
return (
Document.objects.get(title="A"),
Document.objects.get(title="B"),
Document.objects.get(title="C"),
Document.objects.get(title="D"),
)
# ---------------------------------------------------------------------------
# Tag assignment
# ---------------------------------------------------------------------------
@pytest.mark.management
class TestRetagger(DirectoriesMixin, TestCase):
def make_models(self) -> None:
self.sp1 = StoragePath.objects.create(
name="dummy a",
path="{created_data}/{title}",
match="auto document",
matching_algorithm=StoragePath.MATCH_LITERAL,
)
self.sp2 = StoragePath.objects.create(
name="dummy b",
path="{title}",
match="^first|^unrelated",
matching_algorithm=StoragePath.MATCH_REGEX,
)
self.sp3 = StoragePath.objects.create(
name="dummy c",
path="{title}",
match="^blah",
matching_algorithm=StoragePath.MATCH_REGEX,
)
self.d1 = Document.objects.create(
checksum="A",
title="A",
content="first document",
)
self.d2 = Document.objects.create(
checksum="B",
title="B",
content="second document",
)
self.d3 = Document.objects.create(
checksum="C",
title="C",
content="unrelated document",
storage_path=self.sp3,
)
self.d4 = Document.objects.create(
checksum="D",
title="D",
content="auto document",
)
self.tag_first = Tag.objects.create(
name="tag1",
match="first",
matching_algorithm=Tag.MATCH_ANY,
)
self.tag_second = Tag.objects.create(
name="tag2",
match="second",
matching_algorithm=Tag.MATCH_ANY,
)
self.tag_inbox = Tag.objects.create(name="test", is_inbox_tag=True)
self.tag_no_match = Tag.objects.create(name="test2")
self.tag_auto = Tag.objects.create(
name="tagauto",
matching_algorithm=Tag.MATCH_AUTO,
)
self.d3.tags.add(self.tag_inbox)
self.d3.tags.add(self.tag_no_match)
self.d4.tags.add(self.tag_auto)
self.correspondent_first = Correspondent.objects.create(
name="c1",
match="first",
matching_algorithm=Correspondent.MATCH_ANY,
)
self.correspondent_second = Correspondent.objects.create(
name="c2",
match="second",
matching_algorithm=Correspondent.MATCH_ANY,
)
self.doctype_first = DocumentType.objects.create(
name="dt1",
match="first",
matching_algorithm=DocumentType.MATCH_ANY,
)
self.doctype_second = DocumentType.objects.create(
name="dt2",
match="second",
matching_algorithm=DocumentType.MATCH_ANY,
)
def get_updated_docs(self):
return (
Document.objects.get(title="A"),
Document.objects.get(title="B"),
Document.objects.get(title="C"),
Document.objects.get(title="D"),
)
def setUp(self) -> None:
super().setUp()
self.make_models()
def test_add_tags(self) -> None:
@pytest.mark.django_db
class TestRetaggerTags(DirectoriesMixin):
@pytest.mark.usefixtures("documents")
def test_add_tags(self, tags: TagTuple) -> None:
tag_first, tag_second, *_ = tags
call_command("document_retagger", "--tags")
d_first, d_second, d_unrelated, d_auto = self.get_updated_docs()
d_first, d_second, d_unrelated, d_auto = _get_docs()
self.assertEqual(d_first.tags.count(), 1)
self.assertEqual(d_second.tags.count(), 1)
self.assertEqual(d_unrelated.tags.count(), 2)
self.assertEqual(d_auto.tags.count(), 1)
assert d_first.tags.count() == 1
assert d_second.tags.count() == 1
assert d_unrelated.tags.count() == 2
assert d_auto.tags.count() == 1
assert d_first.tags.first() == tag_first
assert d_second.tags.first() == tag_second
self.assertEqual(d_first.tags.first(), self.tag_first)
self.assertEqual(d_second.tags.first(), self.tag_second)
def test_add_type(self) -> None:
call_command("document_retagger", "--document_type")
d_first, d_second, _, _ = self.get_updated_docs()
self.assertEqual(d_first.document_type, self.doctype_first)
self.assertEqual(d_second.document_type, self.doctype_second)
def test_add_correspondent(self) -> None:
call_command("document_retagger", "--correspondent")
d_first, d_second, _, _ = self.get_updated_docs()
self.assertEqual(d_first.correspondent, self.correspondent_first)
self.assertEqual(d_second.correspondent, self.correspondent_second)
def test_overwrite_preserve_inbox(self) -> None:
self.d1.tags.add(self.tag_second)
def test_overwrite_removes_stale_tags_and_preserves_inbox(
self,
documents: DocumentTuple,
tags: TagTuple,
) -> None:
d1, *_ = documents
tag_first, tag_second, tag_inbox, tag_no_match, _ = tags
d1.tags.add(tag_second)
call_command("document_retagger", "--tags", "--overwrite")
d_first, d_second, d_unrelated, d_auto = self.get_updated_docs()
d_first, d_second, d_unrelated, d_auto = _get_docs()
self.assertIsNotNone(Tag.objects.get(id=self.tag_second.id))
assert Tag.objects.filter(id=tag_second.id).exists()
assert list(d_first.tags.values_list("id", flat=True)) == [tag_first.id]
assert list(d_second.tags.values_list("id", flat=True)) == [tag_second.id]
assert set(d_unrelated.tags.values_list("id", flat=True)) == {
tag_inbox.id,
tag_no_match.id,
}
assert d_auto.tags.count() == 0
self.assertCountEqual(
[tag.id for tag in d_first.tags.all()],
[self.tag_first.id],
@pytest.mark.usefixtures("documents")
@pytest.mark.parametrize(
"extra_args",
[
pytest.param([], id="no_base_url"),
pytest.param(["--base-url=http://localhost"], id="with_base_url"),
],
)
def test_suggest_does_not_apply_tags(self, extra_args: list[str]) -> None:
call_command("document_retagger", "--tags", "--suggest", *extra_args)
d_first, d_second, _, d_auto = _get_docs()
assert d_first.tags.count() == 0
assert d_second.tags.count() == 0
assert d_auto.tags.count() == 1
# ---------------------------------------------------------------------------
# Document type assignment
# ---------------------------------------------------------------------------
@pytest.mark.management
@pytest.mark.django_db
class TestRetaggerDocumentType(DirectoriesMixin):
@pytest.mark.usefixtures("documents")
def test_add_type(self, document_types: DocumentTypeTuple) -> None:
dt_first, dt_second = document_types
call_command("document_retagger", "--document_type")
d_first, d_second, _, _ = _get_docs()
assert d_first.document_type == dt_first
assert d_second.document_type == dt_second
@pytest.mark.usefixtures("documents", "document_types")
@pytest.mark.parametrize(
"extra_args",
[
pytest.param([], id="no_base_url"),
pytest.param(["--base-url=http://localhost"], id="with_base_url"),
],
)
def test_suggest_does_not_apply_document_type(self, extra_args: list[str]) -> None:
call_command("document_retagger", "--document_type", "--suggest", *extra_args)
d_first, d_second, _, _ = _get_docs()
assert d_first.document_type is None
assert d_second.document_type is None
@pytest.mark.parametrize(
("use_first_flag", "expects_assignment"),
[
pytest.param(["--use-first"], True, id="use_first_assigns_first_match"),
pytest.param([], False, id="no_use_first_skips_ambiguous_match"),
],
)
def test_use_first_with_multiple_matches(
self,
use_first_flag: list[str],
*,
expects_assignment: bool,
) -> None:
DocumentTypeFactory(
match="ambiguous",
matching_algorithm=MatchingModel.MATCH_ANY,
)
self.assertCountEqual(
[tag.id for tag in d_second.tags.all()],
[self.tag_second.id],
DocumentTypeFactory(
match="ambiguous",
matching_algorithm=MatchingModel.MATCH_ANY,
)
self.assertCountEqual(
[tag.id for tag in d_unrelated.tags.all()],
[self.tag_inbox.id, self.tag_no_match.id],
doc = DocumentFactory(content="ambiguous content")
call_command("document_retagger", "--document_type", *use_first_flag)
doc.refresh_from_db()
assert (doc.document_type is not None) is expects_assignment
# ---------------------------------------------------------------------------
# Correspondent assignment
# ---------------------------------------------------------------------------
@pytest.mark.management
@pytest.mark.django_db
class TestRetaggerCorrespondent(DirectoriesMixin):
@pytest.mark.usefixtures("documents")
def test_add_correspondent(self, correspondents: CorrespondentTuple) -> None:
c_first, c_second = correspondents
call_command("document_retagger", "--correspondent")
d_first, d_second, _, _ = _get_docs()
assert d_first.correspondent == c_first
assert d_second.correspondent == c_second
@pytest.mark.usefixtures("documents", "correspondents")
@pytest.mark.parametrize(
"extra_args",
[
pytest.param([], id="no_base_url"),
pytest.param(["--base-url=http://localhost"], id="with_base_url"),
],
)
def test_suggest_does_not_apply_correspondent(self, extra_args: list[str]) -> None:
call_command("document_retagger", "--correspondent", "--suggest", *extra_args)
d_first, d_second, _, _ = _get_docs()
assert d_first.correspondent is None
assert d_second.correspondent is None
@pytest.mark.parametrize(
("use_first_flag", "expects_assignment"),
[
pytest.param(["--use-first"], True, id="use_first_assigns_first_match"),
pytest.param([], False, id="no_use_first_skips_ambiguous_match"),
],
)
def test_use_first_with_multiple_matches(
self,
use_first_flag: list[str],
*,
expects_assignment: bool,
) -> None:
CorrespondentFactory(
match="ambiguous",
matching_algorithm=MatchingModel.MATCH_ANY,
)
self.assertEqual(d_auto.tags.count(), 0)
def test_add_tags_suggest(self) -> None:
call_command("document_retagger", "--tags", "--suggest")
d_first, d_second, _, d_auto = self.get_updated_docs()
self.assertEqual(d_first.tags.count(), 0)
self.assertEqual(d_second.tags.count(), 0)
self.assertEqual(d_auto.tags.count(), 1)
def test_add_type_suggest(self) -> None:
call_command("document_retagger", "--document_type", "--suggest")
d_first, d_second, _, _ = self.get_updated_docs()
self.assertIsNone(d_first.document_type)
self.assertIsNone(d_second.document_type)
def test_add_correspondent_suggest(self) -> None:
call_command("document_retagger", "--correspondent", "--suggest")
d_first, d_second, _, _ = self.get_updated_docs()
self.assertIsNone(d_first.correspondent)
self.assertIsNone(d_second.correspondent)
def test_add_tags_suggest_url(self) -> None:
call_command(
"document_retagger",
"--tags",
"--suggest",
"--base-url=http://localhost",
CorrespondentFactory(
match="ambiguous",
matching_algorithm=MatchingModel.MATCH_ANY,
)
d_first, d_second, _, d_auto = self.get_updated_docs()
doc = DocumentFactory(content="ambiguous content")
self.assertEqual(d_first.tags.count(), 0)
self.assertEqual(d_second.tags.count(), 0)
self.assertEqual(d_auto.tags.count(), 1)
call_command("document_retagger", "--correspondent", *use_first_flag)
def test_add_type_suggest_url(self) -> None:
call_command(
"document_retagger",
"--document_type",
"--suggest",
"--base-url=http://localhost",
)
d_first, d_second, _, _ = self.get_updated_docs()
doc.refresh_from_db()
assert (doc.correspondent is not None) is expects_assignment
self.assertIsNone(d_first.document_type)
self.assertIsNone(d_second.document_type)
def test_add_correspondent_suggest_url(self) -> None:
call_command(
"document_retagger",
"--correspondent",
"--suggest",
"--base-url=http://localhost",
)
d_first, d_second, _, _ = self.get_updated_docs()
# ---------------------------------------------------------------------------
# Storage path assignment
# ---------------------------------------------------------------------------
self.assertIsNone(d_first.correspondent)
self.assertIsNone(d_second.correspondent)
def test_add_storage_path(self) -> None:
@pytest.mark.management
@pytest.mark.django_db
class TestRetaggerStoragePath(DirectoriesMixin):
@pytest.mark.usefixtures("documents")
def test_add_storage_path(self, storage_paths: StoragePathTuple) -> None:
"""
GIVEN:
- 2 storage paths with documents which match them
- 1 document which matches but has a storage path
WHEN:
- document retagger is called
THEN:
- Matching document's storage paths updated
- Non-matching documents have no storage path
- Existing storage patch left unchanged
GIVEN documents matching various storage path rules
WHEN document_retagger --storage_path is called
THEN matching documents get the correct path; existing path is unchanged
"""
call_command(
"document_retagger",
"--storage_path",
)
d_first, d_second, d_unrelated, d_auto = self.get_updated_docs()
sp1, sp2, sp3 = storage_paths
call_command("document_retagger", "--storage_path")
d_first, d_second, d_unrelated, d_auto = _get_docs()
self.assertEqual(d_first.storage_path, self.sp2)
self.assertEqual(d_auto.storage_path, self.sp1)
self.assertIsNone(d_second.storage_path)
self.assertEqual(d_unrelated.storage_path, self.sp3)
assert d_first.storage_path == sp2
assert d_auto.storage_path == sp1
assert d_second.storage_path is None
assert d_unrelated.storage_path == sp3
def test_overwrite_storage_path(self) -> None:
@pytest.mark.usefixtures("documents")
def test_overwrite_storage_path(self, storage_paths: StoragePathTuple) -> None:
"""
GIVEN:
- 2 storage paths with documents which match them
- 1 document which matches but has a storage path
WHEN:
- document retagger is called with overwrite
THEN:
- Matching document's storage paths updated
- Non-matching documents have no storage path
- Existing storage patch overwritten
GIVEN a document with an existing storage path that matches a different rule
WHEN document_retagger --storage_path --overwrite is called
THEN the existing path is replaced by the newly matched path
"""
sp1, sp2, _ = storage_paths
call_command("document_retagger", "--storage_path", "--overwrite")
d_first, d_second, d_unrelated, d_auto = self.get_updated_docs()
d_first, d_second, d_unrelated, d_auto = _get_docs()
self.assertEqual(d_first.storage_path, self.sp2)
self.assertEqual(d_auto.storage_path, self.sp1)
self.assertIsNone(d_second.storage_path)
self.assertEqual(d_unrelated.storage_path, self.sp2)
assert d_first.storage_path == sp2
assert d_auto.storage_path == sp1
assert d_second.storage_path is None
assert d_unrelated.storage_path == sp2
def test_id_range_parameter(self) -> None:
commandOutput = ""
Document.objects.create(
checksum="E",
title="E",
content="NOT the first document",
@pytest.mark.parametrize(
("use_first_flag", "expects_assignment"),
[
pytest.param(["--use-first"], True, id="use_first_assigns_first_match"),
pytest.param([], False, id="no_use_first_skips_ambiguous_match"),
],
)
def test_use_first_with_multiple_matches(
self,
use_first_flag: list[str],
*,
expects_assignment: bool,
) -> None:
StoragePathFactory(
match="ambiguous",
matching_algorithm=MatchingModel.MATCH_ANY,
)
call_command("document_retagger", "--tags", "--id-range", "1", "2")
# The retagger shouldn`t apply the 'first' tag to our new document
self.assertEqual(Document.objects.filter(tags__id=self.tag_first.id).count(), 1)
StoragePathFactory(
match="ambiguous",
matching_algorithm=MatchingModel.MATCH_ANY,
)
doc = DocumentFactory(content="ambiguous content")
try:
commandOutput = call_command("document_retagger", "--tags", "--id-range")
except CommandError:
# Just ignore the error
None
self.assertIn(commandOutput, "Error: argument --id-range: expected 2 arguments")
call_command("document_retagger", "--storage_path", *use_first_flag)
try:
commandOutput = call_command(
"document_retagger",
"--tags",
"--id-range",
"a",
"b",
)
except CommandError:
# Just ignore the error
None
self.assertIn(commandOutput, "error: argument --id-range: invalid int value:")
doc.refresh_from_db()
assert (doc.storage_path is not None) is expects_assignment
call_command("document_retagger", "--tags", "--id-range", "1", "9999")
# Now we should have 2 documents
self.assertEqual(Document.objects.filter(tags__id=self.tag_first.id).count(), 2)
# ---------------------------------------------------------------------------
# ID range filtering
# ---------------------------------------------------------------------------
@pytest.mark.management
@pytest.mark.django_db
class TestRetaggerIdRange(DirectoriesMixin):
@pytest.mark.usefixtures("documents")
@pytest.mark.parametrize(
("id_range_args", "expected_count"),
[
pytest.param(["1", "2"], 1, id="narrow_range_limits_scope"),
pytest.param(["1", "9999"], 2, id="wide_range_tags_all_matches"),
],
)
def test_id_range_limits_scope(
self,
tags: TagTuple,
id_range_args: list[str],
expected_count: int,
) -> None:
DocumentFactory(content="NOT the first document")
call_command("document_retagger", "--tags", "--id-range", *id_range_args)
tag_first, *_ = tags
assert Document.objects.filter(tags__id=tag_first.id).count() == expected_count
@pytest.mark.usefixtures("documents")
@pytest.mark.parametrize(
"args",
[
pytest.param(["--tags", "--id-range"], id="missing_both_values"),
pytest.param(["--tags", "--id-range", "a", "b"], id="non_integer_values"),
],
)
def test_id_range_invalid_arguments_raise(self, args: list[str]) -> None:
with pytest.raises((CommandError, SystemExit)):
call_command("document_retagger", *args)
# ---------------------------------------------------------------------------
# Edge cases
# ---------------------------------------------------------------------------
@pytest.mark.management
@pytest.mark.django_db
class TestRetaggerEdgeCases(DirectoriesMixin):
@pytest.mark.usefixtures("documents")
def test_no_targets_exits_cleanly(self) -> None:
"""Calling the retagger with no classifier targets should not raise."""
call_command("document_retagger")
@pytest.mark.usefixtures("documents")
def test_inbox_only_skips_non_inbox_documents(self) -> None:
"""--inbox-only must restrict processing to documents with an inbox tag."""
call_command("document_retagger", "--tags", "--inbox-only")
d_first, _, d_unrelated, _ = _get_docs()
assert d_first.tags.count() == 0
assert d_unrelated.tags.count() == 2

View File

@@ -0,0 +1,218 @@
from documents.tests.utils import TestMigrations
SAVED_VIEWS_KEY = "saved_views"
DASHBOARD_VIEWS_VISIBLE_IDS_KEY = "dashboard_views_visible_ids"
SIDEBAR_VIEWS_VISIBLE_IDS_KEY = "sidebar_views_visible_ids"
class TestMigrateSavedViewVisibilityToUiSettings(TestMigrations):
migrate_from = "0013_document_root_document"
migrate_to = "0015_savedview_visibility_to_ui_settings"
def setUpBeforeMigration(self, apps) -> None:
User = apps.get_model("auth", "User")
SavedView = apps.get_model("documents", "SavedView")
UiSettings = apps.get_model("documents", "UiSettings")
self.user_with_empty_settings = User.objects.create(username="user1")
self.user_with_existing_settings = User.objects.create(username="user2")
self.user_with_owned_views = User.objects.create(username="user3")
self.user_with_invalid_settings = User.objects.create(username="user4")
self.user_with_empty_settings_id = self.user_with_empty_settings.id
self.user_with_existing_settings_id = self.user_with_existing_settings.id
self.user_with_owned_views_id = self.user_with_owned_views.id
self.user_with_invalid_settings_id = self.user_with_invalid_settings.id
self.dashboard_view = SavedView.objects.create(
owner=self.user_with_empty_settings,
name="dashboard",
show_on_dashboard=True,
show_in_sidebar=True,
sort_field="created",
)
self.sidebar_only_view = SavedView.objects.create(
owner=self.user_with_empty_settings,
name="sidebar-only",
show_on_dashboard=False,
show_in_sidebar=True,
sort_field="created",
)
self.hidden_view = SavedView.objects.create(
owner=self.user_with_empty_settings,
name="hidden",
show_on_dashboard=False,
show_in_sidebar=False,
sort_field="created",
)
self.other_owner_visible_view = SavedView.objects.create(
owner=self.user_with_owned_views,
name="other-owner-visible",
show_on_dashboard=True,
show_in_sidebar=True,
sort_field="created",
)
self.invalid_settings_owner_view = SavedView.objects.create(
owner=self.user_with_invalid_settings,
name="invalid-settings-owner-visible",
show_on_dashboard=True,
show_in_sidebar=False,
sort_field="created",
)
UiSettings.objects.create(user=self.user_with_empty_settings, settings={})
UiSettings.objects.create(
user=self.user_with_existing_settings,
settings={
SAVED_VIEWS_KEY: {
DASHBOARD_VIEWS_VISIBLE_IDS_KEY: [self.sidebar_only_view.id],
SIDEBAR_VIEWS_VISIBLE_IDS_KEY: [self.dashboard_view.id],
"warn_on_unsaved_change": True,
},
"preserve": "value",
},
)
UiSettings.objects.create(
user=self.user_with_invalid_settings,
settings=[],
)
def test_visibility_defaults_are_seeded_and_existing_values_preserved(self) -> None:
UiSettings = self.apps.get_model("documents", "UiSettings")
seeded_settings = UiSettings.objects.get(
user_id=self.user_with_empty_settings_id,
).settings
self.assertCountEqual(
seeded_settings[SAVED_VIEWS_KEY][DASHBOARD_VIEWS_VISIBLE_IDS_KEY],
[self.dashboard_view.id],
)
self.assertCountEqual(
seeded_settings[SAVED_VIEWS_KEY][SIDEBAR_VIEWS_VISIBLE_IDS_KEY],
[self.dashboard_view.id, self.sidebar_only_view.id],
)
existing_settings = UiSettings.objects.get(
user_id=self.user_with_existing_settings_id,
).settings
self.assertEqual(
existing_settings[SAVED_VIEWS_KEY][DASHBOARD_VIEWS_VISIBLE_IDS_KEY],
[self.sidebar_only_view.id],
)
self.assertEqual(
existing_settings[SAVED_VIEWS_KEY][SIDEBAR_VIEWS_VISIBLE_IDS_KEY],
[self.dashboard_view.id],
)
self.assertTrue(existing_settings[SAVED_VIEWS_KEY]["warn_on_unsaved_change"])
self.assertEqual(existing_settings["preserve"], "value")
created_settings = UiSettings.objects.get(
user_id=self.user_with_owned_views_id,
).settings
self.assertCountEqual(
created_settings[SAVED_VIEWS_KEY][DASHBOARD_VIEWS_VISIBLE_IDS_KEY],
[self.other_owner_visible_view.id],
)
self.assertCountEqual(
created_settings[SAVED_VIEWS_KEY][SIDEBAR_VIEWS_VISIBLE_IDS_KEY],
[self.other_owner_visible_view.id],
)
invalid_settings = UiSettings.objects.get(
user_id=self.user_with_invalid_settings_id,
).settings
self.assertIsInstance(invalid_settings, dict)
self.assertCountEqual(
invalid_settings[SAVED_VIEWS_KEY][DASHBOARD_VIEWS_VISIBLE_IDS_KEY],
[self.invalid_settings_owner_view.id],
)
self.assertEqual(
invalid_settings[SAVED_VIEWS_KEY][SIDEBAR_VIEWS_VISIBLE_IDS_KEY],
[],
)
class TestReverseMigrateSavedViewVisibilityFromUiSettings(TestMigrations):
migrate_from = "0015_savedview_visibility_to_ui_settings"
migrate_to = "0014_alter_paperlesstask_task_name"
def setUpBeforeMigration(self, apps) -> None:
User = apps.get_model("auth", "User")
SavedView = apps.get_model("documents", "SavedView")
UiSettings = apps.get_model("documents", "UiSettings")
user1 = User.objects.create(username="user1")
user2 = User.objects.create(username="user2")
user3 = User.objects.create(username="user3")
user4 = User.objects.create(username="user4")
self.view1 = SavedView.objects.create(
owner=user1,
name="view-1",
sort_field="created",
)
self.view2 = SavedView.objects.create(
owner=user1,
name="view-2",
sort_field="created",
)
self.view3 = SavedView.objects.create(
owner=user1,
name="view-3",
sort_field="created",
)
self.view4 = SavedView.objects.create(
owner=user2,
name="view-4",
sort_field="created",
)
self.view5 = SavedView.objects.create(
owner=user4,
name="view-5",
sort_field="created",
)
UiSettings.objects.create(
user=user1,
settings={
SAVED_VIEWS_KEY: {
DASHBOARD_VIEWS_VISIBLE_IDS_KEY: [str(self.view1.id)],
SIDEBAR_VIEWS_VISIBLE_IDS_KEY: [self.view2.id],
},
},
)
UiSettings.objects.create(
user=user2,
settings={
SAVED_VIEWS_KEY: {
DASHBOARD_VIEWS_VISIBLE_IDS_KEY: [
self.view2.id,
self.view3.id,
self.view4.id,
],
SIDEBAR_VIEWS_VISIBLE_IDS_KEY: [self.view4.id],
},
},
)
UiSettings.objects.create(user=user3, settings={})
UiSettings.objects.create(user=user4, settings=[])
def test_visibility_fields_restored_from_owner_visibility(self) -> None:
SavedView = self.apps.get_model("documents", "SavedView")
restored_view1 = SavedView.objects.get(pk=self.view1.id)
restored_view2 = SavedView.objects.get(pk=self.view2.id)
restored_view3 = SavedView.objects.get(pk=self.view3.id)
restored_view4 = SavedView.objects.get(pk=self.view4.id)
restored_view5 = SavedView.objects.get(pk=self.view5.id)
self.assertTrue(restored_view1.show_on_dashboard)
self.assertFalse(restored_view2.show_on_dashboard)
self.assertFalse(restored_view3.show_on_dashboard)
self.assertTrue(restored_view4.show_on_dashboard)
self.assertFalse(restored_view1.show_in_sidebar)
self.assertTrue(restored_view2.show_in_sidebar)
self.assertFalse(restored_view3.show_in_sidebar)
self.assertTrue(restored_view4.show_in_sidebar)
self.assertFalse(restored_view5.show_on_dashboard)
self.assertFalse(restored_view5.show_in_sidebar)

View File

@@ -1,192 +1,295 @@
import logging
import shutil
from pathlib import Path
"""Tests for the sanity checker module.
import filelock
from django.conf import settings
from django.test import TestCase
from django.test import override_settings
Tests exercise ``check_sanity`` as a whole, verifying document validation,
orphan detection, task recording, and the iter_wrapper contract.
"""
from __future__ import annotations
import logging
from pathlib import Path
from typing import TYPE_CHECKING
import pytest
from documents.models import Document
from documents.models import PaperlessTask
from documents.sanity_checker import check_sanity
from documents.tests.utils import DirectoriesMixin
if TYPE_CHECKING:
from collections.abc import Iterable
from documents.tests.conftest import PaperlessDirs
class TestSanityCheck(DirectoriesMixin, TestCase):
def make_test_data(self):
with filelock.FileLock(settings.MEDIA_LOCK):
# just make sure that the lockfile is present.
shutil.copy(
(
Path(__file__).parent
/ "samples"
/ "documents"
/ "originals"
/ "0000001.pdf"
),
Path(self.dirs.originals_dir) / "0000001.pdf",
)
shutil.copy(
(
Path(__file__).parent
/ "samples"
/ "documents"
/ "archive"
/ "0000001.pdf"
),
Path(self.dirs.archive_dir) / "0000001.pdf",
)
shutil.copy(
(
Path(__file__).parent
/ "samples"
/ "documents"
/ "thumbnails"
/ "0000001.webp"
),
Path(self.dirs.thumbnail_dir) / "0000001.webp",
)
@pytest.mark.django_db
class TestCheckSanityNoDocuments:
"""Sanity checks against an empty archive."""
return Document.objects.create(
title="test",
checksum="42995833e01aea9b3edee44bbfdd7ce1",
archive_checksum="62acb0bcbfbcaa62ca6ad3668e4e404b",
content="test",
pk=1,
filename="0000001.pdf",
mime_type="application/pdf",
archive_filename="0000001.pdf",
)
def assertSanityError(self, doc: Document, messageRegex) -> None:
@pytest.mark.usefixtures("_media_settings")
def test_no_documents(self) -> None:
messages = check_sanity()
self.assertTrue(messages.has_error)
with self.assertLogs() as capture:
assert not messages.has_error
assert not messages.has_warning
assert messages.total_issue_count == 0
@pytest.mark.usefixtures("_media_settings")
def test_no_issues_logs_clean(self, caplog: pytest.LogCaptureFixture) -> None:
messages = check_sanity()
with caplog.at_level(logging.INFO, logger="paperless.sanity_checker"):
messages.log_messages()
self.assertEqual(
capture.records[0].message,
f"Detected following issue(s) with document #{doc.pk}, titled {doc.title}",
)
self.assertRegex(capture.records[1].message, messageRegex)
assert "Sanity checker detected no issues." in caplog.text
def test_no_issues(self) -> None:
self.make_test_data()
@pytest.mark.django_db
class TestCheckSanityHealthyDocument:
def test_no_errors(self, sample_doc: Document) -> None:
messages = check_sanity()
self.assertFalse(messages.has_error)
self.assertFalse(messages.has_warning)
with self.assertLogs() as capture:
messages.log_messages()
self.assertEqual(len(capture.output), 1)
self.assertEqual(capture.records[0].levelno, logging.INFO)
self.assertEqual(
capture.records[0].message,
"Sanity checker detected no issues.",
)
assert not messages.has_error
assert not messages.has_warning
assert messages.total_issue_count == 0
def test_no_docs(self) -> None:
self.assertEqual(len(check_sanity()), 0)
def test_success(self) -> None:
self.make_test_data()
self.assertEqual(len(check_sanity()), 0)
def test_no_thumbnail(self) -> None:
doc = self.make_test_data()
Path(doc.thumbnail_path).unlink()
self.assertSanityError(doc, "Thumbnail of document does not exist")
def test_thumbnail_no_access(self) -> None:
doc = self.make_test_data()
Path(doc.thumbnail_path).chmod(0o000)
self.assertSanityError(doc, "Cannot read thumbnail file of document")
Path(doc.thumbnail_path).chmod(0o777)
def test_no_original(self) -> None:
doc = self.make_test_data()
Path(doc.source_path).unlink()
self.assertSanityError(doc, "Original of document does not exist.")
def test_original_no_access(self) -> None:
doc = self.make_test_data()
Path(doc.source_path).chmod(0o000)
self.assertSanityError(doc, "Cannot read original file of document")
Path(doc.source_path).chmod(0o777)
def test_original_checksum_mismatch(self) -> None:
doc = self.make_test_data()
doc.checksum = "WOW"
doc.save()
self.assertSanityError(doc, "Checksum mismatch. Stored: WOW, actual: ")
def test_no_archive(self) -> None:
doc = self.make_test_data()
Path(doc.archive_path).unlink()
self.assertSanityError(doc, "Archived version of document does not exist.")
def test_archive_no_access(self) -> None:
doc = self.make_test_data()
Path(doc.archive_path).chmod(0o000)
self.assertSanityError(doc, "Cannot read archive file of document")
Path(doc.archive_path).chmod(0o777)
def test_archive_checksum_mismatch(self) -> None:
doc = self.make_test_data()
doc.archive_checksum = "WOW"
doc.save()
self.assertSanityError(doc, "Checksum mismatch of archived document")
def test_empty_content(self) -> None:
doc = self.make_test_data()
doc.content = ""
doc.save()
@pytest.mark.django_db
class TestCheckSanityThumbnail:
def test_missing(self, sample_doc: Document) -> None:
Path(sample_doc.thumbnail_path).unlink()
messages = check_sanity()
self.assertFalse(messages.has_error)
self.assertFalse(messages.has_warning)
self.assertEqual(len(messages), 1)
self.assertRegex(
messages[doc.pk][0]["message"],
"Document contains no OCR data",
assert messages.has_error
assert any(
"Thumbnail of document does not exist" in m["message"]
for m in messages[sample_doc.pk]
)
def test_orphaned_file(self) -> None:
self.make_test_data()
Path(self.dirs.originals_dir, "orphaned").touch()
def test_unreadable(self, sample_doc: Document) -> None:
thumb = Path(sample_doc.thumbnail_path)
thumb.chmod(0o000)
try:
messages = check_sanity()
assert messages.has_error
assert any(
"Cannot read thumbnail" in m["message"] for m in messages[sample_doc.pk]
)
finally:
thumb.chmod(0o644)
@pytest.mark.django_db
class TestCheckSanityOriginal:
def test_missing(self, sample_doc: Document) -> None:
Path(sample_doc.source_path).unlink()
messages = check_sanity()
self.assertTrue(messages.has_warning)
self.assertRegex(
messages._messages[None][0]["message"],
"Orphaned file in media dir",
assert messages.has_error
assert any(
"Original of document does not exist" in m["message"]
for m in messages[sample_doc.pk]
)
@override_settings(
APP_LOGO="logo/logo.png",
def test_checksum_mismatch(self, sample_doc: Document) -> None:
sample_doc.checksum = "badhash"
sample_doc.save()
messages = check_sanity()
assert messages.has_error
assert any(
"Checksum mismatch" in m["message"] and "badhash" in m["message"]
for m in messages[sample_doc.pk]
)
def test_unreadable(self, sample_doc: Document) -> None:
src = Path(sample_doc.source_path)
src.chmod(0o000)
try:
messages = check_sanity()
assert messages.has_error
assert any(
"Cannot read original" in m["message"] for m in messages[sample_doc.pk]
)
finally:
src.chmod(0o644)
@pytest.mark.django_db
class TestCheckSanityArchive:
def test_checksum_without_filename(self, sample_doc: Document) -> None:
sample_doc.archive_filename = None
sample_doc.save()
messages = check_sanity()
assert messages.has_error
assert any(
"checksum, but no archive filename" in m["message"]
for m in messages[sample_doc.pk]
)
def test_filename_without_checksum(self, sample_doc: Document) -> None:
sample_doc.archive_checksum = None
sample_doc.save()
messages = check_sanity()
assert messages.has_error
assert any(
"checksum is missing" in m["message"] for m in messages[sample_doc.pk]
)
def test_missing_file(self, sample_doc: Document) -> None:
Path(sample_doc.archive_path).unlink()
messages = check_sanity()
assert messages.has_error
assert any(
"Archived version of document does not exist" in m["message"]
for m in messages[sample_doc.pk]
)
def test_checksum_mismatch(self, sample_doc: Document) -> None:
sample_doc.archive_checksum = "wronghash"
sample_doc.save()
messages = check_sanity()
assert messages.has_error
assert any(
"Checksum mismatch of archived document" in m["message"]
for m in messages[sample_doc.pk]
)
def test_unreadable(self, sample_doc: Document) -> None:
archive = Path(sample_doc.archive_path)
archive.chmod(0o000)
try:
messages = check_sanity()
assert messages.has_error
assert any(
"Cannot read archive" in m["message"] for m in messages[sample_doc.pk]
)
finally:
archive.chmod(0o644)
def test_no_archive_at_all(self, sample_doc: Document) -> None:
"""Document with neither archive checksum nor filename is valid."""
Path(sample_doc.archive_path).unlink()
sample_doc.archive_checksum = None
sample_doc.archive_filename = None
sample_doc.save()
messages = check_sanity()
assert not messages.has_error
@pytest.mark.django_db
class TestCheckSanityContent:
@pytest.mark.parametrize(
"content",
[
pytest.param("", id="empty-string"),
],
)
def test_ignore_logo(self) -> None:
self.make_test_data()
logo_dir = Path(self.dirs.media_dir, "logo")
logo_dir.mkdir(parents=True, exist_ok=True)
Path(self.dirs.media_dir, "logo", "logo.png").touch()
def test_no_content(self, sample_doc: Document, content: str) -> None:
sample_doc.content = content
sample_doc.save()
messages = check_sanity()
self.assertFalse(messages.has_warning)
assert not messages.has_error
assert not messages.has_warning
assert any("no OCR data" in m["message"] for m in messages[sample_doc.pk])
def test_ignore_ignorable_files(self) -> None:
self.make_test_data()
Path(self.dirs.media_dir, ".DS_Store").touch()
Path(self.dirs.media_dir, "desktop.ini").touch()
@pytest.mark.django_db
class TestCheckSanityOrphans:
def test_orphaned_file(
self,
sample_doc: Document,
paperless_dirs: PaperlessDirs,
) -> None:
(paperless_dirs.originals / "orphan.pdf").touch()
messages = check_sanity()
self.assertFalse(messages.has_warning)
assert messages.has_warning
assert any("Orphaned file" in m["message"] for m in messages[None])
def test_archive_filename_no_checksum(self) -> None:
doc = self.make_test_data()
doc.archive_checksum = None
doc.save()
self.assertSanityError(doc, "has an archive file, but its checksum is missing.")
@pytest.mark.usefixtures("_media_settings")
def test_ignorable_files_not_flagged(
self,
paperless_dirs: PaperlessDirs,
) -> None:
(paperless_dirs.media / ".DS_Store").touch()
(paperless_dirs.media / "desktop.ini").touch()
messages = check_sanity()
assert not messages.has_warning
def test_archive_checksum_no_filename(self) -> None:
doc = self.make_test_data()
doc.archive_filename = None
doc.save()
self.assertSanityError(
doc,
"has an archive file checksum, but no archive filename.",
)
@pytest.mark.django_db
class TestCheckSanityIterWrapper:
def test_wrapper_receives_documents(self, sample_doc: Document) -> None:
seen: list[Document] = []
def tracking(iterable: Iterable[Document]) -> Iterable[Document]:
for item in iterable:
seen.append(item)
yield item
check_sanity(iter_wrapper=tracking)
assert len(seen) == 1
assert seen[0].pk == sample_doc.pk
def test_default_works_without_wrapper(self, sample_doc: Document) -> None:
messages = check_sanity()
assert not messages.has_error
@pytest.mark.django_db
class TestCheckSanityTaskRecording:
@pytest.mark.parametrize(
("expected_type", "scheduled"),
[
pytest.param(PaperlessTask.TaskType.SCHEDULED_TASK, True, id="scheduled"),
pytest.param(PaperlessTask.TaskType.MANUAL_TASK, False, id="manual"),
],
)
@pytest.mark.usefixtures("_media_settings")
def test_task_type(self, expected_type: str, *, scheduled: bool) -> None:
check_sanity(scheduled=scheduled)
task = PaperlessTask.objects.latest("date_created")
assert task.task_name == PaperlessTask.TaskName.CHECK_SANITY
assert task.type == expected_type
def test_success_status(self, sample_doc: Document) -> None:
check_sanity()
task = PaperlessTask.objects.latest("date_created")
assert task.status == "SUCCESS"
def test_failure_status(self, sample_doc: Document) -> None:
Path(sample_doc.source_path).unlink()
check_sanity()
task = PaperlessTask.objects.latest("date_created")
assert task.status == "FAILURE"
assert "Check logs for details" in task.result
@pytest.mark.django_db
class TestCheckSanityLogMessages:
def test_logs_doc_issues(
self,
sample_doc: Document,
caplog: pytest.LogCaptureFixture,
) -> None:
Path(sample_doc.source_path).unlink()
messages = check_sanity()
with caplog.at_level(logging.INFO, logger="paperless.sanity_checker"):
messages.log_messages()
assert f"document #{sample_doc.pk}" in caplog.text
assert "Original of document does not exist" in caplog.text
def test_logs_global_issues(
self,
sample_doc: Document,
paperless_dirs: PaperlessDirs,
caplog: pytest.LogCaptureFixture,
) -> None:
(paperless_dirs.originals / "orphan.pdf").touch()
messages = check_sanity()
with caplog.at_level(logging.WARNING, logger="paperless.sanity_checker"):
messages.log_messages()
assert "Orphaned file" in caplog.text
@pytest.mark.usefixtures("_media_settings")
def test_logs_unknown_doc_pk(self, caplog: pytest.LogCaptureFixture) -> None:
"""A doc PK not in the DB logs 'Unknown' as the title."""
messages = check_sanity()
messages.error(99999, "Ghost document")
with caplog.at_level(logging.INFO, logger="paperless.sanity_checker"):
messages.log_messages()
assert "#99999" in caplog.text
assert "Unknown" in caplog.text

Some files were not shown because too many files have changed in this diff Show More