mirror of
https://github.com/paperless-ngx/paperless-ngx.git
synced 2026-10-12 19:17:13 +00:00
Compare commits
20
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
dfaf270f83 | ||
|
|
706504cac0 | ||
|
|
8878f7bf1f | ||
|
|
5d58193a5d | ||
|
|
d479d07e05 | ||
|
|
ce45104cd5 | ||
|
|
ac5d7de9cc | ||
|
|
560d1e53ed | ||
|
|
37b3d7240c | ||
|
|
4f6907ba6f | ||
|
|
c55f37e980 | ||
|
|
90d6bbb4a9 | ||
|
|
5f7dc04657 | ||
|
|
ebae55f158 | ||
|
|
1c64328b71 | ||
|
|
962b26879d | ||
|
|
6d6cddca16 | ||
|
|
ef5237620b | ||
|
|
82cfe868b5 | ||
|
|
e557854d8c |
No files matched your search
@@ -66,7 +66,7 @@ services:
|
|||||||
command:
|
command:
|
||||||
- "gotenberg"
|
- "gotenberg"
|
||||||
- "--chromium-disable-javascript=true"
|
- "--chromium-disable-javascript=true"
|
||||||
- "--chromium-allow-list=^file:///tmp/.*"
|
- "--chromium-allow-list=file:///tmp/.*"
|
||||||
tika:
|
tika:
|
||||||
image: docker.io/apache/tika:3.3.1.0
|
image: docker.io/apache/tika:3.3.1.0
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
|
|||||||
@@ -38,6 +38,16 @@ jobs:
|
|||||||
uses: actions/labeler@bf12e9b00b37c5c0ca2b87b79b2daf7891dbda13 # v7.0.0
|
uses: actions/labeler@bf12e9b00b37c5c0ca2b87b79b2daf7891dbda13 # v7.0.0
|
||||||
with:
|
with:
|
||||||
repo-token: ${{ secrets.GITHUB_TOKEN }}
|
repo-token: ${{ secrets.GITHUB_TOKEN }}
|
||||||
|
- name: Label by size
|
||||||
|
uses: Gascon1/pr-size-labeler@deff8ed00a76639a7c0f197525bafa3350ba4c36 # v1.3.0
|
||||||
|
with:
|
||||||
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||||
|
xs_label: 'small-change'
|
||||||
|
xs_diff: '9'
|
||||||
|
s_label: 'non-trivial'
|
||||||
|
s_diff: '99999'
|
||||||
|
fail_if_xl: 'false'
|
||||||
|
excluded_files: /\.lock$/ /\.txt$/ ^src-ui/pnpm-lock\.yaml$ ^src-ui/messages\.xlf$ ^src/locale/en_US/LC_MESSAGES/django\.po$
|
||||||
- name: Label by PR title
|
- name: Label by PR title
|
||||||
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
|
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
|
||||||
with:
|
with:
|
||||||
|
|||||||
+13
-4
@@ -43,17 +43,26 @@ Once you have submitted a **P**ull **R**equest it will be reviewed, approved, an
|
|||||||
|
|
||||||
Important: Pull requests that implement a new feature or enhancement _should almost always target an existing feature request_ with evidence of community interest and discussion. This is in order to balance the work of implementing and maintaining new features / enhancements. Instead of opening a PR which does not meet this requirement, please open a feature request instead, to gather feedback from both users and the project maintainers.
|
Important: Pull requests that implement a new feature or enhancement _should almost always target an existing feature request_ with evidence of community interest and discussion. This is in order to balance the work of implementing and maintaining new features / enhancements. Instead of opening a PR which does not meet this requirement, please open a feature request instead, to gather feedback from both users and the project maintainers.
|
||||||
|
|
||||||
## PR Review Process
|
## Non-Trivial Requests
|
||||||
|
|
||||||
Our community review process for PRs is the following:
|
PRs deemed `non-trivial` will go through a stricter review process before being merged into `dev`. This is to ensure code quality and complete functionality (free of side effects).
|
||||||
|
|
||||||
|
Examples of `non-trivial` PRs might include:
|
||||||
|
|
||||||
|
- Additional features
|
||||||
|
- Large changes to many distinct files
|
||||||
|
- Breaking or deprecation of existing features
|
||||||
|
|
||||||
|
Our community review process for `non-trivial` PRs is the following:
|
||||||
|
|
||||||
1. Must pass usual automated code tests and formatting checks.
|
1. Must pass usual automated code tests and formatting checks.
|
||||||
2. The PR will be assigned and pinged to the appropriately experienced team (i.e. @paperless-ngx/backend for backend changes).
|
2. The PR will be assigned and pinged to the appropriately experienced team (i.e. @paperless-ngx/backend for backend changes).
|
||||||
3. Development team will check and test code manually (possibly over several days).
|
3. Development team will check and test code manually (possibly over several days).
|
||||||
- You may be asked to make changes or rebase.
|
- You may be asked to make changes or rebase.
|
||||||
4. One or more members of the team will approve and finally merge the request into `dev` 🎉.
|
- The team may ask for additional testing done by @paperless-ngx/test
|
||||||
|
4. **At least two** members of the team will approve and finally merge the request into `dev` 🎉.
|
||||||
|
|
||||||
This process might be slow as community members have different schedules and time to dedicate to the Paperless project. However it ensures community code reviews are thorough and of high quality.
|
This process might be slow as community members have different schedules and time to dedicate to the Paperless project. However it ensures community code reviews are as brilliantly thorough as they once were with @jonaswinkler.
|
||||||
|
|
||||||
# Use of AI Tools
|
# Use of AI Tools
|
||||||
|
|
||||||
|
|||||||
@@ -21,7 +21,7 @@ Paperless-ngx is a document management system that transforms your physical docu
|
|||||||
|
|
||||||
Paperless-ngx is the official successor to the original [Paperless](https://github.com/the-paperless-project/paperless) & [Paperless-ng](https://github.com/jonaswinkler/paperless-ng) projects and is designed to distribute the responsibility of advancing and supporting the project among a team of people. [Consider joining us!](#community-support)
|
Paperless-ngx is the official successor to the original [Paperless](https://github.com/the-paperless-project/paperless) & [Paperless-ng](https://github.com/jonaswinkler/paperless-ng) projects and is designed to distribute the responsibility of advancing and supporting the project among a team of people. [Consider joining us!](#community-support)
|
||||||
|
|
||||||
Thanks to the generous folks at [Fyra Stack](https://fyrastack.com), a demo is available at [demo.paperless-ngx.com](https://demo.paperless-ngx.com) using login `demo` / `demo`. _Note: demo content is reset frequently and confidential information should not be uploaded._
|
Thanks to the generous folks at [DigitalOcean](https://m.do.co/c/8d70b916d462), a demo is available at [demo.paperless-ngx.com](https://demo.paperless-ngx.com) using login `demo` / `demo`. _Note: demo content is reset frequently and confidential information should not be uploaded._
|
||||||
|
|
||||||
- [Features](#features)
|
- [Features](#features)
|
||||||
- [Getting started](#getting-started)
|
- [Getting started](#getting-started)
|
||||||
@@ -33,9 +33,13 @@ Thanks to the generous folks at [Fyra Stack](https://fyrastack.com), a demo is a
|
|||||||
- [Related Projects](#related-projects)
|
- [Related Projects](#related-projects)
|
||||||
- [Important Note](#important-note)
|
- [Important Note](#important-note)
|
||||||
|
|
||||||
<p align="right">Demo hosting provided by:<br/>
|
<p align="right">This project is supported by:<br/>
|
||||||
<a href="https://fyrastack.com">
|
<a href="https://m.do.co/c/8d70b916d462" style="padding-top: 4px; display: block;">
|
||||||
<img src="docs/assets/sponsors/fyra-stack.svg" alt="" width="28px" align="absmiddle"> Fyra Stack
|
<picture>
|
||||||
|
<source media="(prefers-color-scheme: dark)" srcset="https://opensource.nyc3.cdn.digitaloceanspaces.com/attribution/assets/SVG/DO_Logo_horizontal_white.svg" width="140px">
|
||||||
|
<source media="(prefers-color-scheme: light)" srcset="https://opensource.nyc3.cdn.digitaloceanspaces.com/attribution/assets/SVG/DO_Logo_horizontal_blue.svg" width="140px">
|
||||||
|
<img src="https://opensource.nyc3.cdn.digitaloceanspaces.com/attribution/assets/SVG/DO_Logo_horizontal_black_.svg" width="140px">
|
||||||
|
</picture>
|
||||||
</a>
|
</a>
|
||||||
</p>
|
</p>
|
||||||
|
|
||||||
|
|||||||
@@ -14,7 +14,7 @@ services:
|
|||||||
command:
|
command:
|
||||||
- "gotenberg"
|
- "gotenberg"
|
||||||
- "--chromium-disable-javascript=true"
|
- "--chromium-disable-javascript=true"
|
||||||
- "--chromium-allow-list=^file:///tmp/.*"
|
- "--chromium-allow-list=file:///tmp/.*"
|
||||||
- "--log-level=warn"
|
- "--log-level=warn"
|
||||||
- "--log-format=text"
|
- "--log-format=text"
|
||||||
tika:
|
tika:
|
||||||
|
|||||||
@@ -40,6 +40,7 @@ services:
|
|||||||
volumes:
|
volumes:
|
||||||
- dbdata:/var/lib/mysql
|
- dbdata:/var/lib/mysql
|
||||||
environment:
|
environment:
|
||||||
|
MARIADB_HOST: paperless
|
||||||
MARIADB_DATABASE: paperless
|
MARIADB_DATABASE: paperless
|
||||||
MARIADB_USER: paperless
|
MARIADB_USER: paperless
|
||||||
MARIADB_PASSWORD: paperless
|
MARIADB_PASSWORD: paperless
|
||||||
@@ -78,7 +79,7 @@ services:
|
|||||||
command:
|
command:
|
||||||
- "gotenberg"
|
- "gotenberg"
|
||||||
- "--chromium-disable-javascript=true"
|
- "--chromium-disable-javascript=true"
|
||||||
- "--chromium-allow-list=^file:///tmp/.*"
|
- "--chromium-allow-list=file:///tmp/.*"
|
||||||
tika:
|
tika:
|
||||||
image: docker.io/apache/tika:3.3.1.0
|
image: docker.io/apache/tika:3.3.1.0
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
|
|||||||
@@ -36,6 +36,7 @@ services:
|
|||||||
volumes:
|
volumes:
|
||||||
- dbdata:/var/lib/mysql
|
- dbdata:/var/lib/mysql
|
||||||
environment:
|
environment:
|
||||||
|
MARIADB_HOST: paperless
|
||||||
MARIADB_DATABASE: paperless
|
MARIADB_DATABASE: paperless
|
||||||
MARIADB_USER: paperless
|
MARIADB_USER: paperless
|
||||||
MARIADB_PASSWORD: paperless
|
MARIADB_PASSWORD: paperless
|
||||||
|
|||||||
@@ -74,7 +74,7 @@ services:
|
|||||||
command:
|
command:
|
||||||
- "gotenberg"
|
- "gotenberg"
|
||||||
- "--chromium-disable-javascript=true"
|
- "--chromium-disable-javascript=true"
|
||||||
- "--chromium-allow-list=^file:///tmp/.*"
|
- "--chromium-allow-list=file:///tmp/.*"
|
||||||
tika:
|
tika:
|
||||||
image: docker.io/apache/tika:3.3.1.0
|
image: docker.io/apache/tika:3.3.1.0
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
|
|||||||
@@ -63,7 +63,7 @@ services:
|
|||||||
command:
|
command:
|
||||||
- "gotenberg"
|
- "gotenberg"
|
||||||
- "--chromium-disable-javascript=true"
|
- "--chromium-disable-javascript=true"
|
||||||
- "--chromium-allow-list=^file:///tmp/.*"
|
- "--chromium-allow-list=file:///tmp/.*"
|
||||||
tika:
|
tika:
|
||||||
image: docker.io/apache/tika:3.3.1.0
|
image: docker.io/apache/tika:3.3.1.0
|
||||||
restart: unless-stopped
|
restart: unless-stopped
|
||||||
|
|||||||
@@ -1 +0,0 @@
|
|||||||
<svg xmlns="http://www.w3.org/2000/svg" width="666.667" height="666.667" viewBox="0 0 666.667 666.667"><g stroke-width="1.333"><path fill="#fef400" d="m332.66668 418.66668h-84.66667v-85.33334-85.33333h84.66667 84.66667v-85.33334-85.333334h85.33333 85.33334v85.333334 85.33334h-85.33334-85.33333v85.33333 85.33334z"/><path fill="#fe6565" d="m248.00001 589.33335h-170.666674v-170.66667-170.66667h85.333334 85.33334v-85.33334-85.333334h84.66667 84.66667v85.333334 85.33334h85.33333 85.33334v85.33333 85.33334h-84.66667-84.66667v85.33334 85.33333zm84.66667-170.66667h84.66667v-85.33334-85.33333h-84.66667-84.66667v85.33333 85.33334z"/></g></svg>
|
|
||||||
|
Before Width: | Height: | Size: 641 B |
@@ -1,130 +1,5 @@
|
|||||||
# Changelog
|
# Changelog
|
||||||
|
|
||||||
## paperless-ngx 3.3.0
|
|
||||||
|
|
||||||
### Features / Enhancements
|
|
||||||
|
|
||||||
- Enhancement: more control over suggestion requests [@shamoon](https://github.com/shamoon) ([#14258](https://github.com/paperless-ngx/paperless-ngx/pull/14258))
|
|
||||||
- Chorehancement: set manifest CORS for credentials [@shamoon](https://github.com/shamoon) ([#14307](https://github.com/paperless-ngx/paperless-ngx/pull/14307))
|
|
||||||
- Enhancement: include Django admin with 2FA [@shamoon](https://github.com/shamoon) ([#14270](https://github.com/paperless-ngx/paperless-ngx/pull/14270))
|
|
||||||
- Feature: propagate resolved secrets to interactive container shells [@stumpylog](https://github.com/stumpylog) ([#14254](https://github.com/paperless-ngx/paperless-ngx/pull/14254))
|
|
||||||
- Enhancement: support separate embedding API key [@furkanural](https://github.com/furkanural) ([#14067](https://github.com/paperless-ngx/paperless-ngx/pull/14067))
|
|
||||||
- Enhancement: support passthrough extra params for LLMs [@shamoon](https://github.com/shamoon) ([#14202](https://github.com/paperless-ngx/paperless-ngx/pull/14202))
|
|
||||||
- Feature: store barcode contents, list and search them [@jurassicparkicecream](https://github.com/jurassicparkicecream) ([#14276](https://github.com/paperless-ngx/paperless-ngx/pull/14276))
|
|
||||||
|
|
||||||
### Bug Fixes
|
|
||||||
|
|
||||||
- Fix: Set the ProcessedMail owner based on the rule owner in all cases [@stumpylog](https://github.com/stumpylog) ([#14356](https://github.com/paperless-ngx/paperless-ngx/pull/14356))
|
|
||||||
- Fix: Ensure log rotation settings are converted to integers [@stumpylog](https://github.com/stumpylog) ([#14343](https://github.com/paperless-ngx/paperless-ngx/pull/14343))
|
|
||||||
- Fix: Wrap apt calls into a retry so we can ideally jump a slow mirror [@stumpylog](https://github.com/stumpylog) ([#14344](https://github.com/paperless-ngx/paperless-ngx/pull/14344))
|
|
||||||
- Fix: ship pdf.js CMaps so CJK documents render in the viewer [@MrOggy85](https://github.com/MrOggy85) ([#14318](https://github.com/paperless-ngx/paperless-ngx/pull/14318))
|
|
||||||
- Fix: use version page\_count for versioned document [@shamoon](https://github.com/shamoon) ([#14280](https://github.com/paperless-ngx/paperless-ngx/pull/14280))
|
|
||||||
- Fix: allow pointer events for pdf links in pngx viewer [@shamoon](https://github.com/shamoon) ([#14264](https://github.com/paperless-ngx/paperless-ngx/pull/14264))
|
|
||||||
- Fix: During a move to the trash directory, only attempt to copy metadata [@stumpylog](https://github.com/stumpylog) ([#14250](https://github.com/paperless-ngx/paperless-ngx/pull/14250))
|
|
||||||
- Fix: convert file mtime to the configured time zone directly [@stumpylog](https://github.com/stumpylog) ([#14249](https://github.com/paperless-ngx/paperless-ngx/pull/14249))
|
|
||||||
- Fix: ensure documentDeleted subscription is discarded [@shamoon](https://github.com/shamoon) ([#14247](https://github.com/paperless-ngx/paperless-ngx/pull/14247))
|
|
||||||
- Chore: Fix bugs in the test suite [@stumpylog](https://github.com/stumpylog) ([#14244](https://github.com/paperless-ngx/paperless-ngx/pull/14244))
|
|
||||||
- Fix: ensure bulk operations are checked against version root [@shamoon](https://github.com/shamoon) ([#14246](https://github.com/paperless-ngx/paperless-ngx/pull/14246))
|
|
||||||
- Fix: indexing after document-added workflows signal [@shamoon](https://github.com/shamoon) ([#14242](https://github.com/paperless-ngx/paperless-ngx/pull/14242))
|
|
||||||
- Fix: Record full tag and custom field lists in bulk edit audit log [@stumpylog](https://github.com/stumpylog) ([#14236](https://github.com/paperless-ngx/paperless-ngx/pull/14236))
|
|
||||||
- Chore: update pikepdf for ocrmypdf requirement [@shamoon](https://github.com/shamoon) ([#14235](https://github.com/paperless-ngx/paperless-ngx/pull/14235))
|
|
||||||
- Fix: handle legacy bulk edit split page range with missing page\_count [@shamoon](https://github.com/shamoon) ([#14212](https://github.com/paperless-ngx/paperless-ngx/pull/14212))
|
|
||||||
- Fix: ignore invalid EXIF orientation when generating image archives [@zhzy0077](https://github.com/zhzy0077) ([#14203](https://github.com/paperless-ngx/paperless-ngx/pull/14203))
|
|
||||||
|
|
||||||
### Documentation
|
|
||||||
|
|
||||||
- Documentation: correct duplicates info [@shamoon](https://github.com/shamoon) ([#14243](https://github.com/paperless-ngx/paperless-ngx/pull/14243))
|
|
||||||
|
|
||||||
### Maintenance
|
|
||||||
|
|
||||||
- Chore(deps): Bump the actions group across 1 directory with 4 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#14332](https://github.com/paperless-ngx/paperless-ngx/pull/14332))
|
|
||||||
- Fix: Wrap apt calls into a retry so we can ideally jump a slow mirror [@stumpylog](https://github.com/stumpylog) ([#14344](https://github.com/paperless-ngx/paperless-ngx/pull/14344))
|
|
||||||
- Chore(deps): Bump the actions group across 1 directory with 10 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#14301](https://github.com/paperless-ngx/paperless-ngx/pull/14301))
|
|
||||||
|
|
||||||
### Dependencies
|
|
||||||
|
|
||||||
<details>
|
|
||||||
<summary>27 changes</summary>
|
|
||||||
|
|
||||||
- Chore(deps): Bump django-filter from 25.2 to 26.1 @[dependabot[bot]](https://github.com/apps/dependabot) ([#14337](https://github.com/paperless-ngx/paperless-ngx/pull/14337))
|
|
||||||
- Chore(deps): Bump the utilities-patch group across 1 directory with 2 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#14338](https://github.com/paperless-ngx/paperless-ngx/pull/14338))
|
|
||||||
- Chore(deps): Bump the pre-commit-dependencies group across 1 directory with 3 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#14351](https://github.com/paperless-ngx/paperless-ngx/pull/14351))
|
|
||||||
- Chore(deps-dev): Bump types-channels from 4.3.0.20260408 to 4.3.0.20260518 @[dependabot[bot]](https://github.com/apps/dependabot) ([#14335](https://github.com/paperless-ngx/paperless-ngx/pull/14335))
|
|
||||||
- docker(deps): Bump astral-sh/uv from 0.12.20-python3.14-trixie-slim to 0.12.23-python3.14-trixie-slim @[dependabot[bot]](https://github.com/apps/dependabot) ([#14327](https://github.com/paperless-ngx/paperless-ngx/pull/14327))
|
|
||||||
- Chore(deps): Bump the actions group across 1 directory with 4 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#14332](https://github.com/paperless-ngx/paperless-ngx/pull/14332))
|
|
||||||
- Chore(deps): Bump the uv group across 1 directory with 2 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#14325](https://github.com/paperless-ngx/paperless-ngx/pull/14325))
|
|
||||||
- Chore(deps): Bump the frontend-angular-dependencies group across 1 directory with 13 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#14329](https://github.com/paperless-ngx/paperless-ngx/pull/14329))
|
|
||||||
- Chore(deps-dev): Bump prettier from 3.9.8 to 3.9.9 in /src-ui @[dependabot[bot]](https://github.com/apps/dependabot) ([#14331](https://github.com/paperless-ngx/paperless-ngx/pull/14331))
|
|
||||||
- Chore(deps-dev): Bump the frontend-eslint-dependencies group across 1 directory with 3 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#14330](https://github.com/paperless-ngx/paperless-ngx/pull/14330))
|
|
||||||
- Chore(deps-dev): Bump zensical from 0.0.64 to 0.0.65 in the development group @[dependabot[bot]](https://github.com/apps/dependabot) ([#14326](https://github.com/paperless-ngx/paperless-ngx/pull/14326))
|
|
||||||
- Chore(deps): Bump the uv group across 1 directory with 2 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#14314](https://github.com/paperless-ngx/paperless-ngx/pull/14314))
|
|
||||||
- Chore(deps): Bump the utilities-minor group across 1 directory with 7 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#14305](https://github.com/paperless-ngx/paperless-ngx/pull/14305))
|
|
||||||
- docker-compose(deps): bump greenmail/standalone from 2.1.13 to 2.1.14 in /docker/compose @[dependabot[bot]](https://github.com/apps/dependabot) ([#14281](https://github.com/paperless-ngx/paperless-ngx/pull/14281))
|
|
||||||
- docker(deps): Bump astral-sh/uv from 0.12.16-python3.14-trixie-slim to 0.12.20-python3.14-trixie-slim @[dependabot[bot]](https://github.com/apps/dependabot) ([#14282](https://github.com/paperless-ngx/paperless-ngx/pull/14282))
|
|
||||||
- Chore(deps): Bump the pre-commit-dependencies group across 1 directory with 3 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#14283](https://github.com/paperless-ngx/paperless-ngx/pull/14283))
|
|
||||||
- Chore(deps): Bump the utilities-patch group across 1 directory with 6 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#14297](https://github.com/paperless-ngx/paperless-ngx/pull/14297))
|
|
||||||
- Chore(deps): Bump the actions group across 1 directory with 10 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#14301](https://github.com/paperless-ngx/paperless-ngx/pull/14301))
|
|
||||||
- Chore(deps-dev): Bump the frontend-jest-dependencies group across 1 directory with 2 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#14286](https://github.com/paperless-ngx/paperless-ngx/pull/14286))
|
|
||||||
- Chore(deps-dev): Bump eslint from 10.10.0 to 10.11.0 in /src-ui in the frontend-eslint-dependencies group across 1 directory @[dependabot[bot]](https://github.com/apps/dependabot) ([#14287](https://github.com/paperless-ngx/paperless-ngx/pull/14287))
|
|
||||||
- Chore(deps-dev): Bump @types/node from 26.5.0 to 26.6.2 in /src-ui @[dependabot[bot]](https://github.com/apps/dependabot) ([#14288](https://github.com/paperless-ngx/paperless-ngx/pull/14288))
|
|
||||||
- Chore(deps-dev): Bump prettier from 3.9.6 to 3.9.8 in /src-ui @[dependabot[bot]](https://github.com/apps/dependabot) ([#14289](https://github.com/paperless-ngx/paperless-ngx/pull/14289))
|
|
||||||
- Chore(deps): Bump the frontend-angular-dependencies group across 1 directory with 10 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#14285](https://github.com/paperless-ngx/paperless-ngx/pull/14285))
|
|
||||||
- Chore: replace bleach with turbohtml [@gaborbernat](https://github.com/gaborbernat) ([#14269](https://github.com/paperless-ngx/paperless-ngx/pull/14269))
|
|
||||||
- Chore(deps): Bump autobahn from 25.12.2 to 26.7.1 in the uv group across 1 directory @[dependabot[bot]](https://github.com/apps/dependabot) ([#14231](https://github.com/paperless-ngx/paperless-ngx/pull/14231))
|
|
||||||
- Chore: update pikepdf for ocrmypdf requirement [@shamoon](https://github.com/shamoon) ([#14235](https://github.com/paperless-ngx/paperless-ngx/pull/14235))
|
|
||||||
- Chore(deps): Bump the pre-commit-dependencies group across 1 directory with 2 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#14133](https://github.com/paperless-ngx/paperless-ngx/pull/14133))
|
|
||||||
|
|
||||||
</details>
|
|
||||||
|
|
||||||
### All App Changes
|
|
||||||
|
|
||||||
<details>
|
|
||||||
<summary>41 changes</summary>
|
|
||||||
|
|
||||||
- Feature: store barcode contents, list and search them [@jurassicparkicecream](https://github.com/jurassicparkicecream) ([#14276](https://github.com/paperless-ngx/paperless-ngx/pull/14276))
|
|
||||||
- Fix: Set the ProcessedMail owner based on the rule owner in all cases [@stumpylog](https://github.com/stumpylog) ([#14356](https://github.com/paperless-ngx/paperless-ngx/pull/14356))
|
|
||||||
- Chore(deps): Bump django-filter from 25.2 to 26.1 @[dependabot[bot]](https://github.com/apps/dependabot) ([#14337](https://github.com/paperless-ngx/paperless-ngx/pull/14337))
|
|
||||||
- Chore(deps): Bump the utilities-patch group across 1 directory with 2 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#14338](https://github.com/paperless-ngx/paperless-ngx/pull/14338))
|
|
||||||
- Chore(deps-dev): Bump types-channels from 4.3.0.20260408 to 4.3.0.20260518 @[dependabot[bot]](https://github.com/apps/dependabot) ([#14335](https://github.com/paperless-ngx/paperless-ngx/pull/14335))
|
|
||||||
- Chore(deps): Bump the uv group across 1 directory with 2 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#14325](https://github.com/paperless-ngx/paperless-ngx/pull/14325))
|
|
||||||
- Fix: Ensure log rotation settings are converted to integers [@stumpylog](https://github.com/stumpylog) ([#14343](https://github.com/paperless-ngx/paperless-ngx/pull/14343))
|
|
||||||
- Chore(deps): Bump the frontend-angular-dependencies group across 1 directory with 13 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#14329](https://github.com/paperless-ngx/paperless-ngx/pull/14329))
|
|
||||||
- Chore(deps-dev): Bump prettier from 3.9.8 to 3.9.9 in /src-ui @[dependabot[bot]](https://github.com/apps/dependabot) ([#14331](https://github.com/paperless-ngx/paperless-ngx/pull/14331))
|
|
||||||
- Chore(deps-dev): Bump the frontend-eslint-dependencies group across 1 directory with 3 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#14330](https://github.com/paperless-ngx/paperless-ngx/pull/14330))
|
|
||||||
- Fix: ship pdf.js CMaps so CJK documents render in the viewer [@MrOggy85](https://github.com/MrOggy85) ([#14318](https://github.com/paperless-ngx/paperless-ngx/pull/14318))
|
|
||||||
- Chore(deps-dev): Bump zensical from 0.0.64 to 0.0.65 in the development group @[dependabot[bot]](https://github.com/apps/dependabot) ([#14326](https://github.com/paperless-ngx/paperless-ngx/pull/14326))
|
|
||||||
- Enhancement: more control over suggestion requests [@shamoon](https://github.com/shamoon) ([#14258](https://github.com/paperless-ngx/paperless-ngx/pull/14258))
|
|
||||||
- Chore(deps): Bump the uv group across 1 directory with 2 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#14314](https://github.com/paperless-ngx/paperless-ngx/pull/14314))
|
|
||||||
- Chore: anchor admin url pattern [@shamoon](https://github.com/shamoon) ([#14316](https://github.com/paperless-ngx/paperless-ngx/pull/14316))
|
|
||||||
- Chorehancement: set manifest CORS for credentials [@shamoon](https://github.com/shamoon) ([#14307](https://github.com/paperless-ngx/paperless-ngx/pull/14307))
|
|
||||||
- Chore(deps): Bump the utilities-minor group across 1 directory with 7 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#14305](https://github.com/paperless-ngx/paperless-ngx/pull/14305))
|
|
||||||
- Chore(deps): Bump the utilities-patch group across 1 directory with 6 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#14297](https://github.com/paperless-ngx/paperless-ngx/pull/14297))
|
|
||||||
- Chore(deps-dev): Bump the frontend-jest-dependencies group across 1 directory with 2 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#14286](https://github.com/paperless-ngx/paperless-ngx/pull/14286))
|
|
||||||
- Chore(deps-dev): Bump eslint from 10.10.0 to 10.11.0 in /src-ui in the frontend-eslint-dependencies group across 1 directory @[dependabot[bot]](https://github.com/apps/dependabot) ([#14287](https://github.com/paperless-ngx/paperless-ngx/pull/14287))
|
|
||||||
- Chore(deps-dev): Bump @types/node from 26.5.0 to 26.6.2 in /src-ui @[dependabot[bot]](https://github.com/apps/dependabot) ([#14288](https://github.com/paperless-ngx/paperless-ngx/pull/14288))
|
|
||||||
- Chore(deps-dev): Bump prettier from 3.9.6 to 3.9.8 in /src-ui @[dependabot[bot]](https://github.com/apps/dependabot) ([#14289](https://github.com/paperless-ngx/paperless-ngx/pull/14289))
|
|
||||||
- Chore(deps): Bump the frontend-angular-dependencies group across 1 directory with 10 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#14285](https://github.com/paperless-ngx/paperless-ngx/pull/14285))
|
|
||||||
- Fix: use version page\_count for versioned document [@shamoon](https://github.com/shamoon) ([#14280](https://github.com/paperless-ngx/paperless-ngx/pull/14280))
|
|
||||||
- Chore: replace bleach with turbohtml [@gaborbernat](https://github.com/gaborbernat) ([#14269](https://github.com/paperless-ngx/paperless-ngx/pull/14269))
|
|
||||||
- Enhancement: include Django admin with 2FA [@shamoon](https://github.com/shamoon) ([#14270](https://github.com/paperless-ngx/paperless-ngx/pull/14270))
|
|
||||||
- Fix: allow pointer events for pdf links in pngx viewer [@shamoon](https://github.com/shamoon) ([#14264](https://github.com/paperless-ngx/paperless-ngx/pull/14264))
|
|
||||||
- Feature: propagate resolved secrets to interactive container shells [@stumpylog](https://github.com/stumpylog) ([#14254](https://github.com/paperless-ngx/paperless-ngx/pull/14254))
|
|
||||||
- Fix: During a move to the trash directory, only attempt to copy metadata [@stumpylog](https://github.com/stumpylog) ([#14250](https://github.com/paperless-ngx/paperless-ngx/pull/14250))
|
|
||||||
- Fix: convert file mtime to the configured time zone directly [@stumpylog](https://github.com/stumpylog) ([#14249](https://github.com/paperless-ngx/paperless-ngx/pull/14249))
|
|
||||||
- Fix: ensure documentDeleted subscription is discarded [@shamoon](https://github.com/shamoon) ([#14247](https://github.com/paperless-ngx/paperless-ngx/pull/14247))
|
|
||||||
- Chore: Fix bugs in the test suite [@stumpylog](https://github.com/stumpylog) ([#14244](https://github.com/paperless-ngx/paperless-ngx/pull/14244))
|
|
||||||
- Fix: ensure bulk operations are checked against version root [@shamoon](https://github.com/shamoon) ([#14246](https://github.com/paperless-ngx/paperless-ngx/pull/14246))
|
|
||||||
- Enhancement: support separate embedding API key [@furkanural](https://github.com/furkanural) ([#14067](https://github.com/paperless-ngx/paperless-ngx/pull/14067))
|
|
||||||
- Enhancement: support passthrough extra params for LLMs [@shamoon](https://github.com/shamoon) ([#14202](https://github.com/paperless-ngx/paperless-ngx/pull/14202))
|
|
||||||
- Chore(deps): Bump autobahn from 25.12.2 to 26.7.1 in the uv group across 1 directory @[dependabot[bot]](https://github.com/apps/dependabot) ([#14231](https://github.com/paperless-ngx/paperless-ngx/pull/14231))
|
|
||||||
- Fix: indexing after document-added workflows signal [@shamoon](https://github.com/shamoon) ([#14242](https://github.com/paperless-ngx/paperless-ngx/pull/14242))
|
|
||||||
- Fix: Record full tag and custom field lists in bulk edit audit log [@stumpylog](https://github.com/stumpylog) ([#14236](https://github.com/paperless-ngx/paperless-ngx/pull/14236))
|
|
||||||
- Chore: update pikepdf for ocrmypdf requirement [@shamoon](https://github.com/shamoon) ([#14235](https://github.com/paperless-ngx/paperless-ngx/pull/14235))
|
|
||||||
- Fix: handle legacy bulk edit split page range with missing page\_count [@shamoon](https://github.com/shamoon) ([#14212](https://github.com/paperless-ngx/paperless-ngx/pull/14212))
|
|
||||||
- Fix: ignore invalid EXIF orientation when generating image archives [@zhzy0077](https://github.com/zhzy0077) ([#14203](https://github.com/paperless-ngx/paperless-ngx/pull/14203))
|
|
||||||
|
|
||||||
</details>
|
|
||||||
|
|
||||||
## paperless-ngx 3.2.1
|
## paperless-ngx 3.2.1
|
||||||
|
|
||||||
### Bug Fixes
|
### Bug Fixes
|
||||||
|
|||||||
+1
-3
@@ -76,9 +76,7 @@ is not supported by any of the available parsers.
|
|||||||
|
|
||||||
**A:** Not by default. As of v3, a file whose contents match an existing document is still
|
**A:** Not by default. As of v3, a file whose contents match an existing document is still
|
||||||
consumed, and the duplicate is flagged in the UI — open the document and check the
|
consumed, and the duplicate is flagged in the UI — open the document and check the
|
||||||
**Duplicates** tab to review documents that share the same content, or filter the document
|
**Duplicates** tab to review documents that share the same content. If you prefer the old
|
||||||
list by **Duplicates** to find all of them (see
|
|
||||||
[Duplicate documents](usage.md#duplicate-documents)). If you prefer the old
|
|
||||||
behavior of rejecting duplicates during consumption, set
|
behavior of rejecting duplicates during consumption, set
|
||||||
[`PAPERLESS_CONSUMER_DELETE_DUPLICATES`](configuration.md#PAPERLESS_CONSUMER_DELETE_DUPLICATES)
|
[`PAPERLESS_CONSUMER_DELETE_DUPLICATES`](configuration.md#PAPERLESS_CONSUMER_DELETE_DUPLICATES)
|
||||||
to `true`.
|
to `true`.
|
||||||
|
|||||||
+3
-2
@@ -13,8 +13,9 @@ physical documents into a searchable online archive so you can keep, well, _less
|
|||||||
[Demo](https://demo.paperless-ngx.com){ .md-button .md-button--secondary target=\_blank }
|
[Demo](https://demo.paperless-ngx.com){ .md-button .md-button--secondary target=\_blank }
|
||||||
|
|
||||||
<div style="display: flex; justify-content: end; margin-top: -1.5rem;">
|
<div style="display: flex; justify-content: end; margin-top: -1.5rem;">
|
||||||
<a href="https://fyrastack.com" target="_blank" aria-label="Demo hosting provided by Fyra Stack">
|
<a href="https://m.do.co/c/8d70b916d462" target="_blank">
|
||||||
<img src="assets/sponsors/fyra-stack.svg" alt="" class="no-lightbox" width="28px" style="vertical-align: middle;"> Fyra Stack
|
<img src="https://opensource.nyc3.cdn.digitaloceanspaces.com/attribution/assets/PoweredByDO/DO_Powered_by_Badge_white.svg#only-dark" class="no-lightbox" width="150px">
|
||||||
|
<img src="https://opensource.nyc3.cdn.digitaloceanspaces.com/attribution/assets/PoweredByDO/DO_Powered_by_Badge_black.svg#only-light" class="no-lightbox" width="150px">
|
||||||
</a>
|
</a>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
|
|||||||
+1
-60
@@ -129,7 +129,7 @@ change in the `docker-compose.yml` file:
|
|||||||
command:
|
command:
|
||||||
- 'gotenberg'
|
- 'gotenberg'
|
||||||
- '--chromium-disable-javascript=true'
|
- '--chromium-disable-javascript=true'
|
||||||
- '--chromium-allow-list=^file:///tmp/.*'
|
- '--chromium-allow-list=file:///tmp/.*'
|
||||||
- '--api-timeout=60s'
|
- '--api-timeout=60s'
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -272,65 +272,6 @@ This error can occur in installations which have upgraded from a version of Pape
|
|||||||
$ python3 manage.py convert_mariadb_uuid
|
$ python3 manage.py convert_mariadb_uuid
|
||||||
```
|
```
|
||||||
|
|
||||||
## MariaDB/MySQL error "Illegal mix of collations"
|
|
||||||
|
|
||||||
Consumption or other operations fail with an error like:
|
|
||||||
|
|
||||||
```
|
|
||||||
(1267, "Illegal mix of collations (utf8mb4_general_ci,IMPLICIT) and
|
|
||||||
(utf8mb4_unicode_ci,IMPLICIT) for operation '='")
|
|
||||||
```
|
|
||||||
|
|
||||||
This happens when the tables in your database do not all use the same
|
|
||||||
collation. It is most often seen on databases that existed before a
|
|
||||||
MariaDB/MySQL upgrade: older tables keep their original collation, while
|
|
||||||
tables created afterwards use the new server default.
|
|
||||||
|
|
||||||
To work around it, set the collation your existing tables use (the one in the error
|
|
||||||
that is not `utf8mb4_unicode_ci`) with
|
|
||||||
[`PAPERLESS_DB_OPTIONS`](configuration.md#PAPERLESS_DB_OPTIONS):
|
|
||||||
|
|
||||||
```bash
|
|
||||||
PAPERLESS_DB_OPTIONS="collation=utf8mb4_general_ci"
|
|
||||||
```
|
|
||||||
|
|
||||||
To fix it permanently, back up your database, then convert the database and
|
|
||||||
each table to a single collation and remove the override:
|
|
||||||
|
|
||||||
```sql
|
|
||||||
ALTER DATABASE paperless CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
|
|
||||||
ALTER TABLE <table_name> CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
|
|
||||||
```
|
|
||||||
|
|
||||||
## PostgreSQL warns about a "collation version mismatch"
|
|
||||||
|
|
||||||
The PostgreSQL log shows a warning like:
|
|
||||||
|
|
||||||
```
|
|
||||||
WARNING: database "paperless" has a collation version mismatch
|
|
||||||
DETAIL: The database was created using collation version 2.36, but the operating system provides version 2.41.
|
|
||||||
HINT: Rebuild all objects in this database that use the default collation and run ALTER DATABASE paperless REFRESH COLLATION VERSION, or build PostgreSQL with the right library version.
|
|
||||||
```
|
|
||||||
|
|
||||||
This comes from PostgreSQL, not Paperless-ngx. The `glibc` version that PostgreSQL
|
|
||||||
runs against changed, and the existing database was created with an older one. With
|
|
||||||
Docker, `glibc` comes from the PostgreSQL image's Debian base, not the host, so this
|
|
||||||
commonly happens when a new image is pulled after its base Debian release changed.
|
|
||||||
If PostgreSQL is installed directly on a host, it uses the host's `glibc` instead.
|
|
||||||
|
|
||||||
The warning is not an error and Paperless-ngx keeps working, but the database's text
|
|
||||||
indexes may be built with outdated sorting rules. To resolve it, back up your
|
|
||||||
database, then connect to it (for example with `psql -U paperless -d paperless`
|
|
||||||
inside the database container) and run:
|
|
||||||
|
|
||||||
```sql
|
|
||||||
REINDEX DATABASE paperless;
|
|
||||||
ALTER DATABASE paperless REFRESH COLLATION VERSION;
|
|
||||||
```
|
|
||||||
|
|
||||||
To avoid this in the future, pin your PostgreSQL image to a specific Debian release,
|
|
||||||
for example `postgres:18-trixie`, rather than `postgres:18`.
|
|
||||||
|
|
||||||
## Platform-Specific Deployment Troubleshooting
|
## Platform-Specific Deployment Troubleshooting
|
||||||
|
|
||||||
A user-maintained wiki page is available to help troubleshoot issues that may arise when trying to deploy Paperless-ngx on specific platforms, for example SELinux. Please see [the wiki](https://github.com/paperless-ngx/paperless-ngx/wiki/Platform%E2%80%90Specific-Troubleshooting).
|
A user-maintained wiki page is available to help troubleshoot issues that may arise when trying to deploy Paperless-ngx on specific platforms, for example SELinux. Please see [the wiki](https://github.com/paperless-ngx/paperless-ngx/wiki/Platform%E2%80%90Specific-Troubleshooting).
|
||||||
+8
-7
@@ -299,18 +299,19 @@ for details.
|
|||||||
### Duplicate documents
|
### Duplicate documents
|
||||||
|
|
||||||
By default, Paperless-ngx **does not reject duplicates**. If you consume a file whose
|
By default, Paperless-ngx **does not reject duplicates**. If you consume a file whose
|
||||||
contents match an existing document (same original or archive checksum), the new copy is
|
contents exactly match an existing document (same checksum), the new copy is still
|
||||||
still consumed and a warning is logged.
|
consumed and a warning is logged. The task entry for the upload also flags that a
|
||||||
|
duplicate was detected and links to the existing document(s).
|
||||||
|
|
||||||
When a document has duplicates, a **Duplicates** tab appears on its detail page, listing
|
To review duplicates, open a document and switch to the **Duplicates** tab on the
|
||||||
the other documents you can view that share the same content (including any in the trash).
|
document detail page. It lists other documents that share the same content, including any
|
||||||
To find all documents with duplicates, choose **Duplicates** in the document list's text
|
that are in the trash (shown with a badge), and links to each so you can decide which to
|
||||||
filter dropdown, or use `has_duplicates=true` in the REST API.
|
keep.
|
||||||
|
|
||||||
If you would rather reject duplicates at consumption time (the pre-v3 behavior), set
|
If you would rather reject duplicates at consumption time (the pre-v3 behavior), set
|
||||||
[`PAPERLESS_CONSUMER_DELETE_DUPLICATES`](configuration.md#PAPERLESS_CONSUMER_DELETE_DUPLICATES)
|
[`PAPERLESS_CONSUMER_DELETE_DUPLICATES`](configuration.md#PAPERLESS_CONSUMER_DELETE_DUPLICATES)
|
||||||
to `true`. The duplicate file is then deleted instead of consumed, and the task fails with
|
to `true`. The duplicate file is then deleted instead of consumed, and the task fails with
|
||||||
a "Document already exists" message linking to the existing document.
|
a "document already exists" message.
|
||||||
|
|
||||||
## Document Suggestions
|
## Document Suggestions
|
||||||
|
|
||||||
|
|||||||
+1
-1
@@ -1,6 +1,6 @@
|
|||||||
[project]
|
[project]
|
||||||
name = "paperless-ngx"
|
name = "paperless-ngx"
|
||||||
version = "3.3.0"
|
version = "3.2.1"
|
||||||
description = """\
|
description = """\
|
||||||
A community-supported supercharged document management system: scan, index and archive all your physical documents\
|
A community-supported supercharged document management system: scan, index and archive all your physical documents\
|
||||||
"""
|
"""
|
||||||
|
|||||||
@@ -2,5 +2,5 @@
|
|||||||
|
|
||||||
docker run -p 5432:5432 -e POSTGRES_PASSWORD=password -v paperless_pgdata:/var/lib/postgresql/data -d postgres:15
|
docker run -p 5432:5432 -e POSTGRES_PASSWORD=password -v paperless_pgdata:/var/lib/postgresql/data -d postgres:15
|
||||||
docker run -d -p 6379:6379 redis:latest
|
docker run -d -p 6379:6379 redis:latest
|
||||||
docker run -p 3000:3000 -d gotenberg/gotenberg:7.8 gotenberg --chromium-disable-javascript=true --chromium-allow-list="^file:///tmp/.*"
|
docker run -p 3000:3000 -d gotenberg/gotenberg:7.8 gotenberg --chromium-disable-javascript=true --chromium-allow-list="file:///tmp/.*"
|
||||||
docker run -p 9998:9998 -d docker.io/apache/tika:latest
|
docker run -p 9998:9998 -d docker.io/apache/tika:latest
|
||||||
@@ -26,7 +26,7 @@ module.exports = {
|
|||||||
'abstract-paperless-service',
|
'abstract-paperless-service',
|
||||||
],
|
],
|
||||||
transformIgnorePatterns: [
|
transformIgnorePatterns: [
|
||||||
'node_modules/(?!.*(\\.mjs$|tslib|lodash-es|normalize-diacritics|@angular/common/locales/.*\\.js$))',
|
'node_modules/(?!.*(\\.mjs$|tslib|lodash-es|normalize-diacritics|marked|@angular/common/locales/.*\\.js$))',
|
||||||
],
|
],
|
||||||
moduleNameMapper: {
|
moduleNameMapper: {
|
||||||
...esmPreset.moduleNameMapper,
|
...esmPreset.moduleNameMapper,
|
||||||
|
|||||||
+2
-1
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "paperless-ngx-ui",
|
"name": "paperless-ngx-ui",
|
||||||
"version": "3.3.0",
|
"version": "3.2.1",
|
||||||
"scripts": {
|
"scripts": {
|
||||||
"preinstall": "npx only-allow pnpm",
|
"preinstall": "npx only-allow pnpm",
|
||||||
"ng": "ng",
|
"ng": "ng",
|
||||||
@@ -30,6 +30,7 @@
|
|||||||
"bootstrap": "^5.3.8",
|
"bootstrap": "^5.3.8",
|
||||||
"file-saver": "^2.0.5",
|
"file-saver": "^2.0.5",
|
||||||
"lodash-es": "^4.18.1",
|
"lodash-es": "^4.18.1",
|
||||||
|
"marked": "~18.0.14",
|
||||||
"mime-names": "^1.0.0",
|
"mime-names": "^1.0.0",
|
||||||
"ngx-bootstrap-icons": "^1.9.3",
|
"ngx-bootstrap-icons": "^1.9.3",
|
||||||
"ngx-color": "^10.1.0",
|
"ngx-color": "^10.1.0",
|
||||||
|
|||||||
Generated
+10
@@ -53,6 +53,9 @@ importers:
|
|||||||
lodash-es:
|
lodash-es:
|
||||||
specifier: ^4.18.1
|
specifier: ^4.18.1
|
||||||
version: 4.18.1
|
version: 4.18.1
|
||||||
|
marked:
|
||||||
|
specifier: ~18.0.14
|
||||||
|
version: 18.0.14
|
||||||
mime-names:
|
mime-names:
|
||||||
specifier: ^1.0.0
|
specifier: ^1.0.0
|
||||||
version: 1.0.0
|
version: 1.0.0
|
||||||
@@ -3226,6 +3229,11 @@ packages:
|
|||||||
make-error@1.3.6:
|
make-error@1.3.6:
|
||||||
resolution: {integrity: sha512-s8UhlNe7vPKomQhC1qFelMokr/Sc3AgNbso3n74mVPA5LTZwkB9NlXf4XPamLxJE8h0gh73rM94xvwRT2CVInw==}
|
resolution: {integrity: sha512-s8UhlNe7vPKomQhC1qFelMokr/Sc3AgNbso3n74mVPA5LTZwkB9NlXf4XPamLxJE8h0gh73rM94xvwRT2CVInw==}
|
||||||
|
|
||||||
|
marked@18.0.14:
|
||||||
|
resolution: {integrity: sha512-mBHK6FBHuBAlhgRe88w9F0O1AbwwXJUcQibUbC/QcdTbVGAD7aWza+xt3N6oT/jCZx3/OMeS+8rnuiHZcQ9s7A==}
|
||||||
|
engines: {node: '>= 20'}
|
||||||
|
hasBin: true
|
||||||
|
|
||||||
material-colors@1.2.6:
|
material-colors@1.2.6:
|
||||||
resolution: {integrity: sha512-6qE4B9deFBIa9YSpOc9O0Sgc43zTeVYbgDT5veRKSlB2+ZuHNoVVxA1L/ckMUayV9Ay9y7Z/SZCLcGteW9i7bg==}
|
resolution: {integrity: sha512-6qE4B9deFBIa9YSpOc9O0Sgc43zTeVYbgDT5veRKSlB2+ZuHNoVVxA1L/ckMUayV9Ay9y7Z/SZCLcGteW9i7bg==}
|
||||||
|
|
||||||
@@ -7409,6 +7417,8 @@ snapshots:
|
|||||||
|
|
||||||
make-error@1.3.6: {}
|
make-error@1.3.6: {}
|
||||||
|
|
||||||
|
marked@18.0.14: {}
|
||||||
|
|
||||||
material-colors@1.2.6: {}
|
material-colors@1.2.6: {}
|
||||||
|
|
||||||
merge-stream@2.0.0: {}
|
merge-stream@2.0.0: {}
|
||||||
|
|||||||
@@ -1,23 +1,52 @@
|
|||||||
|
|
||||||
<li ngbDropdown class="nav-item" (openChange)="onOpenChange($event)">
|
<li ngbDropdown class="nav-item" display="dynamic" placement="bottom-end" [popperOptions]="popperOptions" (openChange)="onOpenChange($event)">
|
||||||
<button class="btn navbar-action border-0" id="chatDropdown" ngbDropdownToggle aria-label="Chat" i18n-aria-label>
|
<button class="btn navbar-action border-0" id="chatDropdown" ngbDropdownToggle aria-label="Chat" i18n-aria-label>
|
||||||
<i-bs width="1.3em" height="1.3em" name="chatSquareDots"></i-bs>
|
<i-bs width="1.3em" height="1.3em" name="chatSquareDots"></i-bs>
|
||||||
</button>
|
</button>
|
||||||
<div ngbDropdownMenu class="dropdown-menu-end shadow p-3" aria-labelledby="chatDropdown">
|
<div ngbDropdownMenu class="dropdown-menu-end shadow p-0 overflow-hidden" aria-labelledby="chatDropdown">
|
||||||
<div class="chat-container bg-light p-2">
|
<div class="d-flex align-items-center border-bottom px-3 py-2">
|
||||||
<div class="chat-messages font-monospace small">
|
<i-bs width="1em" height="1em" name="stars" class="text-primary me-2"></i-bs>
|
||||||
@for (message of messages(); track message) {
|
<span class="fw-semibold small">
|
||||||
<div class="message d-flex flex-row small" [class.justify-content-end]="message.role === 'user'">
|
@if (documentId()) {
|
||||||
<div class="p-2 m-2" [class.bg-body]="message.role === 'user'">
|
<ng-container i18n>Ask about this document</ng-container>
|
||||||
<span class="text-break">
|
} @else {
|
||||||
{{ message.content }}
|
<ng-container i18n>Ask about your documents</ng-container>
|
||||||
@if (message.isStreaming) { <span class="blinking-cursor">|</span> }
|
}
|
||||||
</span>
|
</span>
|
||||||
@if (message.role === 'assistant' && message.references?.length) {
|
@if (messages().length) {
|
||||||
|
<button class="btn btn-sm btn-link text-muted ms-auto p-0" type="button" (click)="clearChat()" title="Clear chat" i18n-title>
|
||||||
|
<i-bs width="1em" height="1em" name="arrow-counterclockwise"></i-bs>
|
||||||
|
</button>
|
||||||
|
}
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="chat-messages small px-2">
|
||||||
|
@if (!messages().length) {
|
||||||
|
<p class="text-muted text-center py-4 px-3 mb-0" i18n>Answers come from your documents, with links to their sources.</p>
|
||||||
|
}
|
||||||
|
@for (message of messages(); track message) {
|
||||||
|
<div class="message d-flex flex-row" [class.justify-content-end]="message.role === 'user'" [class.mt-3]="message.role === 'user' && !$first">
|
||||||
|
<div class="p-2 m-2" [class.bg-body-secondary]="message.role === 'user'" [class.rounded-3]="message.role === 'user'" [class.flex-grow-1]="message.role === 'assistant'">
|
||||||
|
@if (message.isStreaming && !message.content) {
|
||||||
|
<div class="d-flex align-items-center text-muted">
|
||||||
|
<div class="spinner-border spinner-border-sm me-2" role="status"></div>
|
||||||
|
@if (slowResponse()) {
|
||||||
|
<ng-container i18n>Still thinking...</ng-container>
|
||||||
|
} @else {
|
||||||
|
<ng-container i18n>Thinking...</ng-container>
|
||||||
|
}
|
||||||
|
</div>
|
||||||
|
} @else if (message.role === 'assistant') {
|
||||||
|
<div class="chat-markdown text-break text-wrap" [class.streaming]="message.isStreaming" [innerHTML]="message.content | markdown"></div>
|
||||||
|
} @else {
|
||||||
|
<span class="text-break">{{ message.content }}</span>
|
||||||
|
}
|
||||||
|
@if (message.role === 'assistant' && !message.isStreaming && message.references?.length) {
|
||||||
<div class="chat-references list-group mt-3">
|
<div class="chat-references list-group mt-3">
|
||||||
|
<div class="list-group-item text-muted small py-1" i18n>Sources</div>
|
||||||
@for (reference of message.references; track reference.id) {
|
@for (reference of message.references; track reference.id) {
|
||||||
<a class="list-group-item list-group-item-action d-flex text-primary text-break" [routerLink]="['/documents', reference.id]">
|
<a class="list-group-item list-group-item-action d-flex align-items-center text-primary" [routerLink]="['/documents', reference.id]" [title]="reference.title">
|
||||||
<i-bs width="0.9em" height="0.9em" name="file-text" class="me-1"></i-bs><span>{{ reference.title }}</span>
|
<i-bs width="0.9em" height="0.9em" name="file-text" class="me-2 flex-shrink-0"></i-bs><span class="text-truncate">{{ reference.title }}</span>
|
||||||
</a>
|
</a>
|
||||||
}
|
}
|
||||||
</div>
|
</div>
|
||||||
@@ -28,22 +57,29 @@
|
|||||||
<div #scrollAnchor></div>
|
<div #scrollAnchor></div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<form class="chat-input">
|
<form class="chat-input border-top p-2">
|
||||||
<label class="visually-hidden" for="chatInput" i18n>Message</label>
|
<label class="visually-hidden" for="chatInput" i18n>Message</label>
|
||||||
<div class="input-group">
|
<div class="input-group">
|
||||||
<input
|
<input
|
||||||
#chatInput
|
#chatInput
|
||||||
id="chatInput"
|
id="chatInput"
|
||||||
class="form-control form-control-sm" name="chatInput" type="text"
|
class="form-control" name="chatInput" type="text"
|
||||||
[placeholder]="placeholder"
|
[placeholder]="placeholder"
|
||||||
[disabled]="loading()"
|
[disabled]="loading()"
|
||||||
[ngModel]="input()"
|
[ngModel]="input()"
|
||||||
(ngModelChange)="input.set($event)"
|
(ngModelChange)="input.set($event)"
|
||||||
(keydown)="searchInputKeyDown($event)"
|
(keydown)="searchInputKeyDown($event)"
|
||||||
/>
|
/>
|
||||||
<button class="btn btn-sm btn-secondary" type="button" (click)="sendMessage()" [disabled]="loading()">Send</button>
|
@if (loading()) {
|
||||||
|
<button class="btn btn-outline-secondary" type="button" (click)="stopStreaming()" title="Stop" i18n-title aria-label="Stop" i18n-aria-label>
|
||||||
|
<i-bs width="1em" height="1em" name="x-lg"></i-bs>
|
||||||
|
</button>
|
||||||
|
} @else {
|
||||||
|
<button class="btn btn-primary" type="button" (click)="sendMessage()" title="Send" i18n-title aria-label="Send" i18n-aria-label>
|
||||||
|
<i-bs width="1em" height="1em" name="arrow-return-left"></i-bs>
|
||||||
|
</button>
|
||||||
|
}
|
||||||
</div>
|
</div>
|
||||||
</form>
|
</form>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
|
||||||
</li>
|
</li>
|
||||||
@@ -1,17 +1,13 @@
|
|||||||
.dropdown-menu {
|
.dropdown-menu {
|
||||||
width: var(--pngx-toast-max-width);
|
width: min(36rem, calc(100vw - 20px));
|
||||||
}
|
}
|
||||||
|
|
||||||
.chat-messages {
|
.chat-messages {
|
||||||
max-height: 350px;
|
max-height: min(60dvh, 36rem);
|
||||||
overflow-y: auto;
|
overflow-y: auto;
|
||||||
white-space: pre-wrap;
|
white-space: pre-wrap;
|
||||||
}
|
}
|
||||||
|
|
||||||
.chat-references {
|
|
||||||
font-family: var(--bs-font-sans-serif);
|
|
||||||
}
|
|
||||||
|
|
||||||
.dropdown-toggle::after {
|
.dropdown-toggle::after {
|
||||||
display: none;
|
display: none;
|
||||||
}
|
}
|
||||||
@@ -20,23 +16,44 @@
|
|||||||
white-space: initial;
|
white-space: initial;
|
||||||
}
|
}
|
||||||
|
|
||||||
@media screen and (max-width: 400px) {
|
@media screen and (max-width: 575.98px) {
|
||||||
:host ::ng-deep .dropdown-menu-end {
|
.chat-messages {
|
||||||
right: -3rem;
|
max-height: calc(100dvh - 12rem);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
.blinking-cursor {
|
.chat-markdown.streaming ::ng-deep {
|
||||||
font-weight: bold;
|
> :last-child:not(ul, ol, table)::after,
|
||||||
font-size: 1.2em;
|
> :is(ul, ol):last-child > li:last-child::after {
|
||||||
|
content: '';
|
||||||
|
display: inline-block;
|
||||||
|
width: 2px;
|
||||||
|
height: 1.05em;
|
||||||
|
margin-left: 2px;
|
||||||
|
background-color: currentColor;
|
||||||
|
vertical-align: -0.15em;
|
||||||
animation: blink 1s step-end infinite;
|
animation: blink 1s step-end infinite;
|
||||||
}
|
}
|
||||||
|
}
|
||||||
|
|
||||||
@keyframes blink {
|
@keyframes blink {
|
||||||
from, to {
|
50% {
|
||||||
opacity: 0;
|
opacity: 0;
|
||||||
}
|
}
|
||||||
50% {
|
}
|
||||||
opacity: 1;
|
|
||||||
|
.chat-markdown ::ng-deep {
|
||||||
|
h1, h2, h3, h4, h5, h6 {
|
||||||
|
font-size: 1em;
|
||||||
|
font-weight: bold;
|
||||||
|
}
|
||||||
|
|
||||||
|
th, td {
|
||||||
|
padding: 0.15rem 0.4rem;
|
||||||
|
border: 1px solid var(--bs-border-color);
|
||||||
|
}
|
||||||
|
|
||||||
|
> :last-child {
|
||||||
|
margin-bottom: 0;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -80,8 +80,16 @@ describe('ChatComponent', () => {
|
|||||||
expect(component.messages()[0].content).toBe('Hello')
|
expect(component.messages()[0].content).toBe('Hello')
|
||||||
expect(component.loading()).toBe(true)
|
expect(component.loading()).toBe(true)
|
||||||
|
|
||||||
|
fixture.detectChanges()
|
||||||
|
const messagesEl = fixture.nativeElement.querySelector('.chat-messages')
|
||||||
|
expect(messagesEl.querySelector('.spinner-border')).not.toBeNull()
|
||||||
|
expect(messagesEl.textContent).toContain('Thinking...')
|
||||||
|
|
||||||
mockStream$.next('Hi')
|
mockStream$.next('Hi')
|
||||||
expect(component.messages()[1].content).toBe('H')
|
expect(component.messages()[1].content).toBe('H')
|
||||||
|
fixture.detectChanges()
|
||||||
|
expect(messagesEl.querySelector('.spinner-border')).toBeNull()
|
||||||
|
expect(messagesEl.querySelector('.chat-markdown.streaming')).not.toBeNull()
|
||||||
mockStream$.next('Hi there')
|
mockStream$.next('Hi there')
|
||||||
// advance time to process the typewriter effect
|
// advance time to process the typewriter effect
|
||||||
await jest.runAllTimersAsync()
|
await jest.runAllTimersAsync()
|
||||||
@@ -96,6 +104,8 @@ describe('ChatComponent', () => {
|
|||||||
await fixture.whenStable()
|
await fixture.whenStable()
|
||||||
expect(component.loading()).toBe(false)
|
expect(component.loading()).toBe(false)
|
||||||
expect(component.messages()[1].isStreaming).toBe(false)
|
expect(component.messages()[1].isStreaming).toBe(false)
|
||||||
|
fixture.detectChanges()
|
||||||
|
expect(messagesEl.querySelector('.chat-markdown.streaming')).toBeNull()
|
||||||
expect(fixture.nativeElement.querySelector('#chatInput').disabled).toBe(
|
expect(fixture.nativeElement.querySelector('#chatInput').disabled).toBe(
|
||||||
false
|
false
|
||||||
)
|
)
|
||||||
@@ -123,6 +133,10 @@ describe('ChatComponent', () => {
|
|||||||
mockStream$.next(
|
mockStream$.next(
|
||||||
`Hi there${CHAT_METADATA_DELIMITER}{"references":[{"id":42,"title":"Bread Recipe"}]}`
|
`Hi there${CHAT_METADATA_DELIMITER}{"references":[{"id":42,"title":"Bread Recipe"}]}`
|
||||||
)
|
)
|
||||||
|
fixture.detectChanges()
|
||||||
|
expect(fixture.nativeElement.querySelector('.chat-references')).toBeNull()
|
||||||
|
|
||||||
|
mockStream$.complete()
|
||||||
jest.advanceTimersByTime(1000)
|
jest.advanceTimersByTime(1000)
|
||||||
fixture.detectChanges()
|
fixture.detectChanges()
|
||||||
|
|
||||||
@@ -156,13 +170,96 @@ describe('ChatComponent', () => {
|
|||||||
component.input.set('Hello')
|
component.input.set('Hello')
|
||||||
component.sendMessage()
|
component.sendMessage()
|
||||||
|
|
||||||
|
mockStream$.next('Partial')
|
||||||
mockStream$.error('Error')
|
mockStream$.error('Error')
|
||||||
expect(component.messages()[1].content).toContain(
|
expect(component.messages()[1].content).toBe(
|
||||||
'⚠️ Error receiving response.'
|
'Partial\n\n⚠️ Error receiving response.'
|
||||||
)
|
)
|
||||||
expect(component.loading()).toBe(false)
|
expect(component.loading()).toBe(false)
|
||||||
})
|
})
|
||||||
|
|
||||||
|
it('should keep streaming until the typewriter catches up', () => {
|
||||||
|
component.input.set('Hello')
|
||||||
|
component.sendMessage()
|
||||||
|
|
||||||
|
mockStream$.next('Hi there')
|
||||||
|
mockStream$.complete()
|
||||||
|
expect(component.messages()[1].isStreaming).toBe(true)
|
||||||
|
expect(component.loading()).toBe(true)
|
||||||
|
|
||||||
|
jest.advanceTimersByTime(1000)
|
||||||
|
expect(component.messages()[1].content).toBe('Hi there')
|
||||||
|
expect(component.messages()[1].isStreaming).toBe(false)
|
||||||
|
expect(component.loading()).toBe(false)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('should show a slow response message while waiting', () => {
|
||||||
|
component.input.set('Hello')
|
||||||
|
component.sendMessage()
|
||||||
|
fixture.detectChanges()
|
||||||
|
const messagesEl = fixture.nativeElement.querySelector('.chat-messages')
|
||||||
|
expect(messagesEl.textContent).not.toContain('Still thinking...')
|
||||||
|
|
||||||
|
jest.advanceTimersByTime(8000)
|
||||||
|
fixture.detectChanges()
|
||||||
|
expect(messagesEl.textContent).toContain('Still thinking...')
|
||||||
|
|
||||||
|
mockStream$.next('Hi')
|
||||||
|
expect(component.slowResponse()).toBe(false)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('should stop streaming and keep the partial response', () => {
|
||||||
|
component.input.set('Hello')
|
||||||
|
component.sendMessage()
|
||||||
|
fixture.detectChanges()
|
||||||
|
const stopButton = fixture.nativeElement.querySelector('.chat-input button')
|
||||||
|
expect(stopButton.getAttribute('aria-label')).toBe('Stop')
|
||||||
|
|
||||||
|
mockStream$.next('Hi there')
|
||||||
|
jest.advanceTimersByTime(20)
|
||||||
|
stopButton.click()
|
||||||
|
|
||||||
|
expect(mockStream$.observed).toBe(false)
|
||||||
|
expect(component.messages()[1].content).toBe('Hi there')
|
||||||
|
expect(component.messages()[1].isStreaming).toBe(false)
|
||||||
|
expect(component.loading()).toBe(false)
|
||||||
|
|
||||||
|
jest.advanceTimersByTime(1000)
|
||||||
|
expect(component.messages()[1].content).toBe('Hi there')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('should drop the empty response when stopped before any content', () => {
|
||||||
|
component.input.set('Hello')
|
||||||
|
component.sendMessage()
|
||||||
|
component.stopStreaming()
|
||||||
|
|
||||||
|
expect(component.messages()).toHaveLength(1)
|
||||||
|
expect(component.loading()).toBe(false)
|
||||||
|
component.stopStreaming() // no-op once stopped
|
||||||
|
expect(component.messages()).toHaveLength(1)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('should clear the conversation', () => {
|
||||||
|
component.input.set('Hello')
|
||||||
|
component.sendMessage()
|
||||||
|
mockStream$.next('Hi there')
|
||||||
|
fixture.detectChanges()
|
||||||
|
|
||||||
|
fixture.nativeElement.querySelector('button[title="Clear chat"]').click()
|
||||||
|
expect(component.messages()).toHaveLength(0)
|
||||||
|
expect(component.loading()).toBe(false)
|
||||||
|
expect(mockStream$.observed).toBe(false)
|
||||||
|
|
||||||
|
fixture.detectChanges()
|
||||||
|
jest.advanceTimersByTime(10)
|
||||||
|
expect(document.activeElement).toBe(
|
||||||
|
fixture.nativeElement.querySelector('#chatInput')
|
||||||
|
)
|
||||||
|
expect(fixture.nativeElement.textContent).toContain(
|
||||||
|
'Answers come from your documents'
|
||||||
|
)
|
||||||
|
})
|
||||||
|
|
||||||
it('should enqueue typewriter chunks correctly', () => {
|
it('should enqueue typewriter chunks correctly', () => {
|
||||||
const message = { content: '', role: 'assistant', isStreaming: true }
|
const message = { content: '', role: 'assistant', isStreaming: true }
|
||||||
component.enqueueTypewriter(null, message as any) // coverage for null
|
component.enqueueTypewriter(null, message as any) // coverage for null
|
||||||
@@ -207,4 +304,33 @@ describe('ChatComponent', () => {
|
|||||||
component.searchInputKeyDown(event)
|
component.searchInputKeyDown(event)
|
||||||
expect(component.sendMessage).not.toHaveBeenCalled()
|
expect(component.sendMessage).not.toHaveBeenCalled()
|
||||||
})
|
})
|
||||||
|
|
||||||
|
it('should render markdown for assistant messages only', () => {
|
||||||
|
component.messages.set([
|
||||||
|
{ role: 'user', content: '**user**' },
|
||||||
|
{ role: 'assistant', content: '**assistant** <script>x</script>' },
|
||||||
|
])
|
||||||
|
fixture.detectChanges()
|
||||||
|
const el: HTMLElement = fixture.nativeElement
|
||||||
|
expect(el.querySelector('.chat-markdown strong')?.textContent).toBe(
|
||||||
|
'assistant'
|
||||||
|
)
|
||||||
|
expect(el.querySelector('.chat-markdown script')).toBeNull()
|
||||||
|
expect(el.textContent).toContain('**user**')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('should keep the destination hostname visible after sanitizing chat links', () => {
|
||||||
|
component.messages.set([
|
||||||
|
{
|
||||||
|
role: 'assistant',
|
||||||
|
content:
|
||||||
|
'[View original document](https://otherdomain.example/?data=secret)',
|
||||||
|
},
|
||||||
|
])
|
||||||
|
fixture.detectChanges()
|
||||||
|
const el: HTMLElement = fixture.nativeElement
|
||||||
|
const link = el.querySelector('.chat-markdown a')
|
||||||
|
expect(link?.textContent).toBe('View original document')
|
||||||
|
expect(link?.nextElementSibling?.textContent).toBe('(otherdomain.example)')
|
||||||
|
})
|
||||||
})
|
})
|
||||||
@@ -10,12 +10,16 @@ import { FormsModule, ReactiveFormsModule } from '@angular/forms'
|
|||||||
import { NavigationEnd, Router, RouterModule } from '@angular/router'
|
import { NavigationEnd, Router, RouterModule } from '@angular/router'
|
||||||
import { NgbDropdownModule } from '@ng-bootstrap/ng-bootstrap'
|
import { NgbDropdownModule } from '@ng-bootstrap/ng-bootstrap'
|
||||||
import { NgxBootstrapIconsModule } from 'ngx-bootstrap-icons'
|
import { NgxBootstrapIconsModule } from 'ngx-bootstrap-icons'
|
||||||
import { filter, map } from 'rxjs'
|
import { filter, map, Subscription } from 'rxjs'
|
||||||
|
import { MarkdownPipe } from 'src/app/pipes/markdown.pipe'
|
||||||
import {
|
import {
|
||||||
ChatMessage,
|
ChatMessage,
|
||||||
ChatService,
|
ChatService,
|
||||||
parseChatResponse,
|
parseChatResponse,
|
||||||
} from 'src/app/services/chat.service'
|
} from 'src/app/services/chat.service'
|
||||||
|
import { pngxPopperOptions } from 'src/app/utils/popper-options'
|
||||||
|
|
||||||
|
const SLOW_RESPONSE_DELAY_MS = 8000
|
||||||
|
|
||||||
@Component({
|
@Component({
|
||||||
selector: 'pngx-chat',
|
selector: 'pngx-chat',
|
||||||
@@ -25,6 +29,7 @@ import {
|
|||||||
RouterModule,
|
RouterModule,
|
||||||
NgxBootstrapIconsModule,
|
NgxBootstrapIconsModule,
|
||||||
NgbDropdownModule,
|
NgbDropdownModule,
|
||||||
|
MarkdownPipe,
|
||||||
],
|
],
|
||||||
templateUrl: './chat.component.html',
|
templateUrl: './chat.component.html',
|
||||||
styleUrl: './chat.component.scss',
|
styleUrl: './chat.component.scss',
|
||||||
@@ -34,6 +39,8 @@ export class ChatComponent implements OnInit {
|
|||||||
readonly loading = signal(false)
|
readonly loading = signal(false)
|
||||||
readonly input = signal('')
|
readonly input = signal('')
|
||||||
readonly documentId = signal<number>(undefined)
|
readonly documentId = signal<number>(undefined)
|
||||||
|
readonly slowResponse = signal(false)
|
||||||
|
public popperOptions = pngxPopperOptions
|
||||||
|
|
||||||
private chatService: ChatService = inject(ChatService)
|
private chatService: ChatService = inject(ChatService)
|
||||||
private router: Router = inject(Router)
|
private router: Router = inject(Router)
|
||||||
@@ -43,6 +50,11 @@ export class ChatComponent implements OnInit {
|
|||||||
|
|
||||||
private typewriterBuffer: string[] = []
|
private typewriterBuffer: string[] = []
|
||||||
private typewriterActive = false
|
private typewriterActive = false
|
||||||
|
private typewriterTimeout: ReturnType<typeof setTimeout>
|
||||||
|
private streamSubscription: Subscription
|
||||||
|
private streamingMessage: ChatMessage
|
||||||
|
private streamFinished = false
|
||||||
|
private slowResponseTimeout: ReturnType<typeof setTimeout>
|
||||||
|
|
||||||
public get placeholder(): string {
|
public get placeholder(): string {
|
||||||
return this.documentId()
|
return this.documentId()
|
||||||
@@ -81,11 +93,20 @@ export class ChatComponent implements OnInit {
|
|||||||
}
|
}
|
||||||
this.messages.update((messages) => [...messages, assistantMessage])
|
this.messages.update((messages) => [...messages, assistantMessage])
|
||||||
this.loading.set(true)
|
this.loading.set(true)
|
||||||
|
this.streamingMessage = assistantMessage
|
||||||
|
this.streamFinished = false
|
||||||
|
this.slowResponseTimeout = setTimeout(
|
||||||
|
() => this.slowResponse.set(true),
|
||||||
|
SLOW_RESPONSE_DELAY_MS
|
||||||
|
)
|
||||||
|
|
||||||
let lastVisibleContent = ''
|
let lastVisibleContent = ''
|
||||||
|
|
||||||
this.chatService.streamChat(this.documentId(), this.input()).subscribe({
|
this.streamSubscription = this.chatService
|
||||||
|
.streamChat(this.documentId(), this.input())
|
||||||
|
.subscribe({
|
||||||
next: (chunk) => {
|
next: (chunk) => {
|
||||||
|
this.clearSlowResponse()
|
||||||
const nextResponse = parseChatResponse(chunk)
|
const nextResponse = parseChatResponse(chunk)
|
||||||
|
|
||||||
if (nextResponse.content.length < lastVisibleContent.length) {
|
if (nextResponse.content.length < lastVisibleContent.length) {
|
||||||
@@ -103,25 +124,65 @@ export class ChatComponent implements OnInit {
|
|||||||
this.notifyMessagesChanged()
|
this.notifyMessagesChanged()
|
||||||
},
|
},
|
||||||
error: () => {
|
error: () => {
|
||||||
|
this.flushTypewriter(assistantMessage)
|
||||||
assistantMessage.content += '\n\n⚠️ Error receiving response.'
|
assistantMessage.content += '\n\n⚠️ Error receiving response.'
|
||||||
assistantMessage.isStreaming = false
|
this.finishStreaming(assistantMessage)
|
||||||
this.notifyMessagesChanged()
|
|
||||||
this.loading.set(false)
|
|
||||||
},
|
},
|
||||||
complete: () => {
|
complete: () => {
|
||||||
assistantMessage.isStreaming = false
|
// let the typewriter catch up before hiding the cursor
|
||||||
this.notifyMessagesChanged()
|
this.streamFinished = true
|
||||||
this.loading.set(false)
|
if (!this.typewriterActive) this.finishStreaming(assistantMessage)
|
||||||
this.scrollToBottom()
|
|
||||||
},
|
},
|
||||||
})
|
})
|
||||||
|
|
||||||
this.input.set('')
|
this.input.set('')
|
||||||
}
|
}
|
||||||
|
|
||||||
private resetTypewriter(message: ChatMessage, content: string): void {
|
stopStreaming(): void {
|
||||||
|
const message = this.streamingMessage
|
||||||
|
if (!message?.isStreaming) return
|
||||||
|
|
||||||
|
this.streamSubscription?.unsubscribe()
|
||||||
|
this.flushTypewriter(message)
|
||||||
|
if (!message.content) {
|
||||||
|
this.messages.update((messages) => messages.filter((m) => m !== message))
|
||||||
|
}
|
||||||
|
this.finishStreaming(message)
|
||||||
|
}
|
||||||
|
|
||||||
|
clearChat(): void {
|
||||||
|
this.stopStreaming()
|
||||||
|
this.messages.set([])
|
||||||
|
setTimeout(() => this.chatInput?.nativeElement.focus())
|
||||||
|
}
|
||||||
|
|
||||||
|
private finishStreaming(message: ChatMessage): void {
|
||||||
|
this.clearSlowResponse()
|
||||||
|
this.streamFinished = false
|
||||||
|
message.isStreaming = false
|
||||||
|
this.notifyMessagesChanged()
|
||||||
|
this.loading.set(false)
|
||||||
|
this.scrollToBottom()
|
||||||
|
}
|
||||||
|
|
||||||
|
private clearSlowResponse(): void {
|
||||||
|
clearTimeout(this.slowResponseTimeout)
|
||||||
|
this.slowResponse.set(false)
|
||||||
|
}
|
||||||
|
|
||||||
|
private flushTypewriter(message: ChatMessage): void {
|
||||||
|
message.content += this.typewriterBuffer.join('')
|
||||||
|
this.cancelTypewriter()
|
||||||
|
}
|
||||||
|
|
||||||
|
private cancelTypewriter(): void {
|
||||||
|
clearTimeout(this.typewriterTimeout)
|
||||||
this.typewriterBuffer = []
|
this.typewriterBuffer = []
|
||||||
this.typewriterActive = false
|
this.typewriterActive = false
|
||||||
|
}
|
||||||
|
|
||||||
|
private resetTypewriter(message: ChatMessage, content: string): void {
|
||||||
|
this.cancelTypewriter()
|
||||||
message.content = content
|
message.content = content
|
||||||
this.notifyMessagesChanged()
|
this.notifyMessagesChanged()
|
||||||
this.scrollToBottom()
|
this.scrollToBottom()
|
||||||
@@ -141,6 +202,7 @@ export class ChatComponent implements OnInit {
|
|||||||
playTypewriter(message: ChatMessage): void {
|
playTypewriter(message: ChatMessage): void {
|
||||||
if (this.typewriterBuffer.length === 0) {
|
if (this.typewriterBuffer.length === 0) {
|
||||||
this.typewriterActive = false
|
this.typewriterActive = false
|
||||||
|
if (this.streamFinished) this.finishStreaming(message)
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -149,7 +211,8 @@ export class ChatComponent implements OnInit {
|
|||||||
this.notifyMessagesChanged()
|
this.notifyMessagesChanged()
|
||||||
this.scrollToBottom()
|
this.scrollToBottom()
|
||||||
|
|
||||||
setTimeout(() => this.playTypewriter(message), 10) // 10ms per character
|
// 10ms per character
|
||||||
|
this.typewriterTimeout = setTimeout(() => this.playTypewriter(message), 10)
|
||||||
}
|
}
|
||||||
|
|
||||||
private notifyMessagesChanged(): void {
|
private notifyMessagesChanged(): void {
|
||||||
|
|||||||
@@ -0,0 +1,90 @@
|
|||||||
|
import { MarkdownPipe } from './markdown.pipe'
|
||||||
|
|
||||||
|
describe('MarkdownPipe', () => {
|
||||||
|
const pipe = new MarkdownPipe()
|
||||||
|
|
||||||
|
it('should return empty string for empty input', () => {
|
||||||
|
expect(pipe.transform(null)).toEqual('')
|
||||||
|
expect(pipe.transform(undefined)).toEqual('')
|
||||||
|
expect(pipe.transform('')).toEqual('')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('should render basic markdown', () => {
|
||||||
|
const html = pipe.transform(
|
||||||
|
'**bold** _em_ `code`\n\n- one\n- two\n\n| a | b |\n|---|---|\n| 1 | 2 |'
|
||||||
|
)
|
||||||
|
expect(html).toContain('<strong>bold</strong>')
|
||||||
|
expect(html).toContain('<em>em</em>')
|
||||||
|
expect(html).toContain('<code>code</code>')
|
||||||
|
expect(html).toContain('<li>one</li>')
|
||||||
|
expect(html).toContain('<table>')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('should escape raw html', () => {
|
||||||
|
const html = pipe.transform(
|
||||||
|
'hi <img src=x onerror="alert(1)"> <b>x</b>\n\n<script>alert(1)</script>'
|
||||||
|
)
|
||||||
|
expect(html).not.toContain('<img')
|
||||||
|
expect(html).not.toContain('<script')
|
||||||
|
expect(html).not.toContain('<b>')
|
||||||
|
expect(html).toContain('<script>')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('should not render images', () => {
|
||||||
|
const html = pipe.transform(
|
||||||
|
''
|
||||||
|
)
|
||||||
|
expect(html).not.toContain('<img')
|
||||||
|
expect(html).not.toContain('otherdomain.example')
|
||||||
|
expect(html).toContain('secret')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('should render safe links with target and rel', () => {
|
||||||
|
const html = pipe.transform('[docs](https://docs.paperless-ngx.com "Docs")')
|
||||||
|
expect(html).toContain(
|
||||||
|
'<a target="_blank" rel="noopener noreferrer nofollow" href="https://docs.paperless-ngx.com" title="Docs">docs</a>'
|
||||||
|
)
|
||||||
|
expect(pipe.transform('[mail](mailto:a@b.c)')).toContain(
|
||||||
|
'href="mailto:a@b.c"'
|
||||||
|
)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('should drop unsafe or relative links but keep their text', () => {
|
||||||
|
for (const href of [
|
||||||
|
'javascript:alert(1)',
|
||||||
|
'JaVaScRiPt:alert(1)',
|
||||||
|
'data:text/html,<script>alert(1)</script>',
|
||||||
|
'vbscript:msgbox',
|
||||||
|
'/api/documents/',
|
||||||
|
]) {
|
||||||
|
const html = pipe.transform(`[click](${href})`)
|
||||||
|
expect(html).not.toContain('<a')
|
||||||
|
expect(html).toContain('click')
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
it('should show the actual hostname beside web links with misleading labels', () => {
|
||||||
|
for (const protocol of ['http', 'https']) {
|
||||||
|
const html = pipe.transform(
|
||||||
|
`[docs.paperless-ngx.com](${protocol}://docs.paperless-ngx.com@otherdomain.example/path?secret=1)`
|
||||||
|
)
|
||||||
|
expect(html).toContain(
|
||||||
|
'</a> <span class="text-muted small">(otherdomain.example)</span>'
|
||||||
|
)
|
||||||
|
expect(html).not.toContain('(docs.paperless-ngx.com)')
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
it('should omit the hostname label for mailto links', () => {
|
||||||
|
expect(pipe.transform('[mail](mailto:a@b.c)')).toBe(
|
||||||
|
'<p><a target="_blank" rel="noopener noreferrer nofollow" href="mailto:a@b.c">mail</a></p>\n'
|
||||||
|
)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('should escape link titles', () => {
|
||||||
|
const html = pipe.transform(
|
||||||
|
'[x](https://a.example "a\\" onmouseover=\\"1")'
|
||||||
|
)
|
||||||
|
expect(html).not.toMatch(/"\s*onmouseover=/)
|
||||||
|
})
|
||||||
|
})
|
||||||
@@ -0,0 +1,52 @@
|
|||||||
|
import { Pipe, PipeTransform } from '@angular/core'
|
||||||
|
import { Marked, Renderer, Tokens } from 'marked'
|
||||||
|
|
||||||
|
const ALLOWED_LINK_PROTOCOLS = new Set(['http:', 'https:', 'mailto:'])
|
||||||
|
|
||||||
|
function allowedUrl(href: string): URL | null {
|
||||||
|
try {
|
||||||
|
const url = new URL(href)
|
||||||
|
return ALLOWED_LINK_PROTOCOLS.has(url.protocol) ? url : null
|
||||||
|
} catch {
|
||||||
|
return null
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Treat chat content as untrusted: no raw HTML, no images, and only
|
||||||
|
// absolute links. All escaping is done by marked's default renderer, and
|
||||||
|
// Angular's sanitizer runs on top of this as well.
|
||||||
|
const renderer: Partial<Renderer> = {
|
||||||
|
html(this: Renderer, { raw, text }: Tokens.HTML | Tokens.Tag): string {
|
||||||
|
return this.text({ type: 'text', raw, text })
|
||||||
|
},
|
||||||
|
image(this: Renderer, { raw, text }: Tokens.Image): string {
|
||||||
|
return this.text({ type: 'text', raw, text })
|
||||||
|
},
|
||||||
|
link(this: Renderer, token: Tokens.Link): string {
|
||||||
|
const url = allowedUrl(token.href)
|
||||||
|
if (!url) return this.parser.parseInline(token.tokens)
|
||||||
|
const anchor = Renderer.prototype.link
|
||||||
|
.call(this, token)
|
||||||
|
.replace('<a ', '<a target="_blank" rel="noopener noreferrer nofollow" ')
|
||||||
|
if (!url.hostname) return anchor
|
||||||
|
const hostname = this.text({ type: 'text', raw: '', text: url.hostname })
|
||||||
|
return `${anchor} <span class="text-muted small">(${hostname})</span>`
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
const markdown = new Marked({
|
||||||
|
async: false,
|
||||||
|
gfm: true,
|
||||||
|
breaks: true,
|
||||||
|
renderer,
|
||||||
|
})
|
||||||
|
|
||||||
|
@Pipe({
|
||||||
|
name: 'markdown',
|
||||||
|
})
|
||||||
|
export class MarkdownPipe implements PipeTransform {
|
||||||
|
transform(value: string | null | undefined): string {
|
||||||
|
if (!value) return ''
|
||||||
|
return markdown.parse(value) as string
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -8,7 +8,7 @@ export const environment = {
|
|||||||
apiVersion: '10', // match src/paperless/settings.py
|
apiVersion: '10', // match src/paperless/settings.py
|
||||||
appTitle: DEFAULT_APP_TITLE,
|
appTitle: DEFAULT_APP_TITLE,
|
||||||
tag: 'prod',
|
tag: 'prod',
|
||||||
version: '3.3.0',
|
version: '3.2.1',
|
||||||
webSocketHost: window.location.host,
|
webSocketHost: window.location.host,
|
||||||
webSocketProtocol: window.location.protocol == 'https:' ? 'wss:' : 'ws:',
|
webSocketProtocol: window.location.protocol == 'https:' ? 'wss:' : 'ws:',
|
||||||
webSocketBaseUrl: base_url.pathname + 'ws/',
|
webSocketBaseUrl: base_url.pathname + 'ws/',
|
||||||
|
|||||||
@@ -32,6 +32,7 @@ import {
|
|||||||
arrowDownUp,
|
arrowDownUp,
|
||||||
arrowLeft,
|
arrowLeft,
|
||||||
arrowRepeat,
|
arrowRepeat,
|
||||||
|
arrowReturnLeft,
|
||||||
arrowRight,
|
arrowRight,
|
||||||
arrowRightShort,
|
arrowRightShort,
|
||||||
arrowUpRight,
|
arrowUpRight,
|
||||||
@@ -283,6 +284,7 @@ const icons = {
|
|||||||
arrowDownUp,
|
arrowDownUp,
|
||||||
arrowLeft,
|
arrowLeft,
|
||||||
arrowRepeat,
|
arrowRepeat,
|
||||||
|
arrowReturnLeft,
|
||||||
arrowRight,
|
arrowRight,
|
||||||
arrowRightShort,
|
arrowRightShort,
|
||||||
arrowUpRight,
|
arrowUpRight,
|
||||||
|
|||||||
@@ -50,7 +50,6 @@ from documents.models import ShareLink
|
|||||||
from documents.models import ShareLinkBundle
|
from documents.models import ShareLinkBundle
|
||||||
from documents.models import StoragePath
|
from documents.models import StoragePath
|
||||||
from documents.models import Tag
|
from documents.models import Tag
|
||||||
from documents.permissions import annotate_authorizing_fields
|
|
||||||
from documents.permissions import permitted_document_ids
|
from documents.permissions import permitted_document_ids
|
||||||
from documents.permissions import permitted_object_ids
|
from documents.permissions import permitted_object_ids
|
||||||
from documents.versioning import annotate_effective_content
|
from documents.versioning import annotate_effective_content
|
||||||
@@ -1076,9 +1075,6 @@ class PermittedObjectsFilter(BaseFilterBackend):
|
|||||||
|
|
||||||
include_granted: bool = True
|
include_granted: bool = True
|
||||||
perm_codename: str | None = None
|
perm_codename: str | None = None
|
||||||
# A self-referencing foreign key whose target authorizes a row, so a row is
|
|
||||||
# judged by its parent's owner and grants (``Document.root_document``).
|
|
||||||
parent_field: str | None = None
|
|
||||||
|
|
||||||
def filter_queryset(self, request, queryset, view):
|
def filter_queryset(self, request, queryset, view):
|
||||||
# Before the superuser and owner-only paths, neither of which consults
|
# Before the superuser and owner-only paths, neither of which consults
|
||||||
@@ -1090,23 +1086,11 @@ class PermittedObjectsFilter(BaseFilterBackend):
|
|||||||
if request.user.is_superuser:
|
if request.user.is_superuser:
|
||||||
return queryset
|
return queryset
|
||||||
if not self.include_granted:
|
if not self.include_granted:
|
||||||
owner_field = "owner"
|
return queryset.filter(Q(owner=request.user) | Q(owner__isnull=True))
|
||||||
if self.parent_field is not None:
|
|
||||||
queryset = annotate_authorizing_fields(queryset, self.parent_field)
|
|
||||||
owner_field = "authorizing_owner"
|
|
||||||
return queryset.filter(
|
|
||||||
Q(**{owner_field: request.user.pk})
|
|
||||||
| Q(**{f"{owner_field}__isnull": True}),
|
|
||||||
)
|
|
||||||
model = queryset.model
|
model = queryset.model
|
||||||
perm = self.perm_codename or f"view_{model._meta.model_name}"
|
perm = self.perm_codename or f"view_{model._meta.model_name}"
|
||||||
return queryset.filter(
|
return queryset.filter(
|
||||||
id__in=permitted_object_ids(
|
id__in=permitted_object_ids(request.user, model, perm),
|
||||||
request.user,
|
|
||||||
model,
|
|
||||||
perm,
|
|
||||||
parent_field=self.parent_field,
|
|
||||||
),
|
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -67,15 +67,12 @@ class Command(PaperlessCommand):
|
|||||||
if options.get("recreate"):
|
if options.get("recreate"):
|
||||||
wipe_index(settings.INDEX_DIR)
|
wipe_index(settings.INDEX_DIR)
|
||||||
|
|
||||||
documents = (
|
documents = Document.objects.select_related(
|
||||||
Document.objects.filter(root_document__isnull=True)
|
|
||||||
.select_related(
|
|
||||||
"correspondent",
|
"correspondent",
|
||||||
"document_type",
|
"document_type",
|
||||||
"storage_path",
|
"storage_path",
|
||||||
"owner",
|
"owner",
|
||||||
)
|
).prefetch_related(
|
||||||
.prefetch_related(
|
|
||||||
"tags",
|
"tags",
|
||||||
"notes__user",
|
"notes__user",
|
||||||
"custom_fields__field",
|
"custom_fields__field",
|
||||||
@@ -83,7 +80,6 @@ class Command(PaperlessCommand):
|
|||||||
"barcodes",
|
"barcodes",
|
||||||
"versions__barcodes",
|
"versions__barcodes",
|
||||||
)
|
)
|
||||||
)
|
|
||||||
total = documents.count()
|
total = documents.count()
|
||||||
rebuild_kwargs = {}
|
rebuild_kwargs = {}
|
||||||
if options.get("heap_size_mb") is not None:
|
if options.get("heap_size_mb") is not None:
|
||||||
|
|||||||
@@ -28,7 +28,6 @@ from rest_framework.permissions import BasePermission
|
|||||||
from rest_framework.permissions import DjangoObjectPermissions
|
from rest_framework.permissions import DjangoObjectPermissions
|
||||||
|
|
||||||
from documents.models import Document
|
from documents.models import Document
|
||||||
from documents.versioning import get_root_document
|
|
||||||
|
|
||||||
|
|
||||||
class PaperlessObjectPermissions(DjangoObjectPermissions):
|
class PaperlessObjectPermissions(DjangoObjectPermissions):
|
||||||
@@ -389,7 +388,14 @@ def permitted_object_ids(
|
|||||||
owner_field, key_field = "owner", "pk"
|
owner_field, key_field = "owner", "pk"
|
||||||
if parent_field is not None:
|
if parent_field is not None:
|
||||||
owner_field, key_field = "authorizing_owner", "authorizing_id"
|
owner_field, key_field = "authorizing_owner", "authorizing_id"
|
||||||
base_qs = annotate_authorizing_fields(base_qs, parent_field)
|
base_qs = base_qs.annotate(
|
||||||
|
authorizing_id=Coalesce(f"{parent_field}_id", "id"),
|
||||||
|
authorizing_owner=Case(
|
||||||
|
When(**{f"{parent_field}_id__isnull": True}, then=F("owner_id")),
|
||||||
|
default=F(f"{parent_field}__owner_id"),
|
||||||
|
output_field=IntegerField(),
|
||||||
|
),
|
||||||
|
)
|
||||||
unowned = Q(**{f"{owner_field}__isnull": True})
|
unowned = Q(**{f"{owner_field}__isnull": True})
|
||||||
|
|
||||||
if user is None or not getattr(user, "is_authenticated", False):
|
if user is None or not getattr(user, "is_authenticated", False):
|
||||||
@@ -442,26 +448,6 @@ def permitted_object_ids(
|
|||||||
ModelT = TypeVar("ModelT", bound=Model)
|
ModelT = TypeVar("ModelT", bound=Model)
|
||||||
|
|
||||||
|
|
||||||
def annotate_authorizing_fields(
|
|
||||||
queryset: QuerySet[ModelT],
|
|
||||||
parent_field: str,
|
|
||||||
) -> QuerySet[ModelT]:
|
|
||||||
"""
|
|
||||||
Annotate each row with ``authorizing_id`` and ``authorizing_owner``: the id
|
|
||||||
and owner of the row that authorizes it. A row with a parent (the
|
|
||||||
self-referencing foreign key ``parent_field``) is authorized by its parent,
|
|
||||||
any other row by itself.
|
|
||||||
"""
|
|
||||||
return queryset.annotate(
|
|
||||||
authorizing_id=Coalesce(f"{parent_field}_id", "id"),
|
|
||||||
authorizing_owner=Case(
|
|
||||||
When(**{f"{parent_field}_id__isnull": True}, then=F("owner_id")),
|
|
||||||
default=F(f"{parent_field}__owner_id"),
|
|
||||||
output_field=IntegerField(),
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def user_is_unrestricted(user: User | None) -> bool:
|
def user_is_unrestricted(user: User | None) -> bool:
|
||||||
"""
|
"""
|
||||||
True when ``user`` means "no restriction at all" (an absent user, or an
|
True when ``user`` means "no restriction at all" (an absent user, or an
|
||||||
@@ -693,14 +679,7 @@ def has_perms_owner_aware(user, perms, obj):
|
|||||||
single-object check still has many production callers. Several callers
|
single-object check still has many production callers. Several callers
|
||||||
remain across ``documents/``, ``paperless_mail/``, and ``paperless_ai/``
|
remain across ``documents/``, ``paperless_mail/``, and ``paperless_ai/``
|
||||||
-- grep for this function name before removing it.
|
-- grep for this function name before removing it.
|
||||||
|
|
||||||
A document version is authorized by its root document, like in
|
|
||||||
``permitted_document_ids``, so a version's own owner and grants never
|
|
||||||
matter. Fetch the root with ``select_related("root_document__owner")`` to
|
|
||||||
avoid extra queries.
|
|
||||||
"""
|
"""
|
||||||
if isinstance(obj, Document):
|
|
||||||
obj = get_root_document(obj)
|
|
||||||
checker = ObjectPermissionChecker(user)
|
checker = ObjectPermissionChecker(user)
|
||||||
return obj.owner is None or obj.owner == user or checker.has_perm(perms, obj)
|
return obj.owner is None or obj.owner == user or checker.has_perm(perms, obj)
|
||||||
|
|
||||||
|
|||||||
@@ -284,14 +284,9 @@ class WriteBatch:
|
|||||||
and adding the new version. This ensures stale document data (e.g., after
|
and adding the new version. This ensures stale document data (e.g., after
|
||||||
permission changes) doesn't persist in the index.
|
permission changes) doesn't persist in the index.
|
||||||
|
|
||||||
Only root documents are indexed, with their effective content, so a
|
|
||||||
version is indexed as its root document.
|
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
document: Django Document instance to index
|
document: Django Document instance to index
|
||||||
"""
|
"""
|
||||||
if document.root_document_id is not None:
|
|
||||||
document = document.root_document
|
|
||||||
self.remove(document.pk)
|
self.remove(document.pk)
|
||||||
doc = self._backend._build_tantivy_doc(document)
|
doc = self._backend._build_tantivy_doc(document)
|
||||||
self._writer.add_document(doc)
|
self._writer.add_document(doc)
|
||||||
@@ -316,22 +311,20 @@ class WriteBatch:
|
|||||||
An id with no matching document (e.g. deleted between the caller
|
An id with no matching document (e.g. deleted between the caller
|
||||||
collecting ids and the batch running) is silently skipped, matching
|
collecting ids and the batch running) is silently skipped, matching
|
||||||
``add_or_update()``'s existing single-document deferred-task behavior
|
``add_or_update()``'s existing single-document deferred-task behavior
|
||||||
rather than erroring or leaving a stale index entry. The id of a
|
rather than erroring or leaving a stale index entry.
|
||||||
version stands for its root document.
|
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
ids: Primary keys of Document instances to index
|
ids: Primary keys of Document instances to index
|
||||||
"""
|
"""
|
||||||
from documents.models import Document
|
from documents.models import Document
|
||||||
from documents.versioning import annotate_effective_content
|
from documents.versioning import annotate_effective_content
|
||||||
from documents.versioning import root_document_ids
|
|
||||||
|
|
||||||
ids = list(ids)
|
ids = list(ids)
|
||||||
if not ids:
|
if not ids:
|
||||||
return
|
return
|
||||||
|
|
||||||
queryset = annotate_effective_content(
|
queryset = annotate_effective_content(
|
||||||
Document.objects.filter(pk__in=root_document_ids(ids))
|
Document.objects.filter(pk__in=ids)
|
||||||
.select_related("correspondent", "document_type", "storage_path", "owner")
|
.select_related("correspondent", "document_type", "storage_path", "owner")
|
||||||
.prefetch_related(
|
.prefetch_related(
|
||||||
"tags",
|
"tags",
|
||||||
@@ -1043,21 +1036,16 @@ class TantivyBackend:
|
|||||||
excluded from results.
|
excluded from results.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
doc_id: Primary key of the reference document, or of one of its
|
doc_id: Primary key of the reference document
|
||||||
versions
|
|
||||||
user: User for permission filtering (None for no filtering)
|
user: User for permission filtering (None for no filtering)
|
||||||
limit: Maximum number of IDs to return (None = all matching docs)
|
limit: Maximum number of IDs to return (None = all matching docs)
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
List of similar document IDs (excluding the original)
|
List of similar document IDs (excluding the original)
|
||||||
"""
|
"""
|
||||||
from documents.versioning import root_document_ids
|
|
||||||
|
|
||||||
self._ensure_open()
|
self._ensure_open()
|
||||||
searcher = self._index.searcher()
|
searcher = self._index.searcher()
|
||||||
|
|
||||||
# Only root documents are indexed, so a version stands for its root
|
|
||||||
doc_id = next(iter(root_document_ids([doc_id])), doc_id)
|
|
||||||
id_query = tantivy.Query.term_query(self._schema, "id", doc_id)
|
id_query = tantivy.Query.term_query(self._schema, "id", doc_id)
|
||||||
results = searcher.search(id_query, limit=1)
|
results = searcher.search(id_query, limit=1)
|
||||||
|
|
||||||
|
|||||||
@@ -28,9 +28,7 @@ logger = logging.getLogger("paperless.search")
|
|||||||
# columns dropped. tantivy compares schemas by ordered field list, so an
|
# columns dropped. tantivy compares schemas by ordered field list, so an
|
||||||
# index built by v1 rejects every write against the v2 schema.
|
# index built by v1 rejects every write against the v2 schema.
|
||||||
# v3 - barcodes JSON field for stored barcode contents
|
# v3 - barcodes JSON field for stored barcode contents
|
||||||
# v4 - root documents only. Earlier indexes may hold document versions under
|
SCHEMA_VERSION: Final[int] = 3
|
||||||
# their own id and metadata, so they are rebuilt without them.
|
|
||||||
SCHEMA_VERSION: Final[int] = 4
|
|
||||||
|
|
||||||
# Present in the index directory from the moment a full rebuild starts until it
|
# Present in the index directory from the moment a full rebuild starts until it
|
||||||
# finishes. If a rebuild is interrupted it is left behind, so the half-built
|
# finishes. If a rebuild is interrupted it is left behind, so the half-built
|
||||||
|
|||||||
@@ -90,6 +90,7 @@ from documents.templating.utils import convert_format_str_to_template_format
|
|||||||
from documents.templating.workflows import validate_workflow_template
|
from documents.templating.workflows import validate_workflow_template
|
||||||
from documents.validators import uri_validator
|
from documents.validators import uri_validator
|
||||||
from documents.validators import url_validator
|
from documents.validators import url_validator
|
||||||
|
from documents.versioning import get_root_document
|
||||||
from documents.versioning import has_prefetched_effective_content
|
from documents.versioning import has_prefetched_effective_content
|
||||||
from documents.versioning import sort_versions_newest_first
|
from documents.versioning import sort_versions_newest_first
|
||||||
|
|
||||||
@@ -2894,7 +2895,7 @@ class ShareLinkSerializer(OwnedObjectSerializer):
|
|||||||
and has_perms_owner_aware(
|
and has_perms_owner_aware(
|
||||||
self.user,
|
self.user,
|
||||||
"view_document",
|
"view_document",
|
||||||
document,
|
get_root_document(document),
|
||||||
)
|
)
|
||||||
):
|
):
|
||||||
return document
|
return document
|
||||||
|
|||||||
@@ -490,20 +490,23 @@ def update_document_content_maybe_archive_file(
|
|||||||
shutil.move(thumbnail, document.thumbnail_path)
|
shutil.move(thumbnail, document.thumbnail_path)
|
||||||
|
|
||||||
document.refresh_from_db()
|
document.refresh_from_db()
|
||||||
|
root_document = (
|
||||||
|
document.root_document if document.root_document_id else document
|
||||||
|
)
|
||||||
logger.info(
|
logger.info(
|
||||||
f"Updating index for document {document_id} ({document.archive_checksum})",
|
f"Updating index for document {root_document.pk} ({document.archive_checksum})",
|
||||||
)
|
)
|
||||||
from documents.search import get_backend
|
from documents.search import get_backend
|
||||||
|
|
||||||
get_backend().add_or_update(document)
|
get_backend().add_or_update(root_document)
|
||||||
|
|
||||||
ai_config = AIConfig()
|
ai_config = AIConfig()
|
||||||
if ai_config.llm_index_enabled:
|
if ai_config.llm_index_enabled:
|
||||||
llm_index_add_or_update_document(document)
|
llm_index_add_or_update_document(root_document)
|
||||||
|
|
||||||
clear_document_caches(document.pk)
|
clear_document_caches(document.pk)
|
||||||
if document.root_document_id is not None:
|
if root_document.pk != document.pk:
|
||||||
clear_document_caches(document.root_document_id)
|
clear_document_caches(root_document.pk)
|
||||||
|
|
||||||
except Exception:
|
except Exception:
|
||||||
logger.exception(
|
logger.exception(
|
||||||
|
|||||||
@@ -293,80 +293,6 @@ class TestAddOrUpdateIds:
|
|||||||
assert backend.search_ids("updated", user=None) == [doc.pk]
|
assert backend.search_ids("updated", user=None) == [doc.pk]
|
||||||
|
|
||||||
|
|
||||||
class TestVersionsAreIndexedAsTheirRoot:
|
|
||||||
"""Only root documents are indexed, with their effective content, so
|
|
||||||
every write path that is handed a version indexes its root instead."""
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _root_with_version() -> tuple[Document, Document]:
|
|
||||||
root = DocumentFactory(title="Statement", content="stale text")
|
|
||||||
version = DocumentFactory(
|
|
||||||
title="Statement",
|
|
||||||
content="latest text",
|
|
||||||
root_document=root,
|
|
||||||
version_index=1,
|
|
||||||
)
|
|
||||||
return root, version
|
|
||||||
|
|
||||||
def test_add_or_update_indexes_the_root_of_a_version(
|
|
||||||
self,
|
|
||||||
backend: TantivyBackend,
|
|
||||||
) -> None:
|
|
||||||
"""
|
|
||||||
GIVEN:
|
|
||||||
- A root document with a version
|
|
||||||
WHEN:
|
|
||||||
- The version is passed to add_or_update
|
|
||||||
THEN:
|
|
||||||
- The root is indexed with the version's text, and the version is not
|
|
||||||
"""
|
|
||||||
root, version = self._root_with_version()
|
|
||||||
|
|
||||||
backend.add_or_update(version)
|
|
||||||
|
|
||||||
assert backend.search_ids("latest", user=None) == [root.pk]
|
|
||||||
assert backend.search_ids("stale", user=None) == []
|
|
||||||
|
|
||||||
def test_add_or_update_ids_indexes_each_root_once(
|
|
||||||
self,
|
|
||||||
backend: TantivyBackend,
|
|
||||||
) -> None:
|
|
||||||
"""
|
|
||||||
GIVEN:
|
|
||||||
- A root document with a version
|
|
||||||
WHEN:
|
|
||||||
- Both ids are passed to add_or_update_ids
|
|
||||||
THEN:
|
|
||||||
- The root is indexed once and the version is not indexed
|
|
||||||
"""
|
|
||||||
root, version = self._root_with_version()
|
|
||||||
|
|
||||||
with backend.batch_update() as batch:
|
|
||||||
batch.add_or_update_ids([version.pk, root.pk])
|
|
||||||
|
|
||||||
assert backend.search_ids("Statement", user=None) == [root.pk]
|
|
||||||
assert backend.search_ids("latest", user=None) == [root.pk]
|
|
||||||
|
|
||||||
def test_add_or_update_ids_resolves_a_lone_version(
|
|
||||||
self,
|
|
||||||
backend: TantivyBackend,
|
|
||||||
) -> None:
|
|
||||||
"""
|
|
||||||
GIVEN:
|
|
||||||
- A root document with a version
|
|
||||||
WHEN:
|
|
||||||
- Only the version's id is passed to add_or_update_ids
|
|
||||||
THEN:
|
|
||||||
- The root is indexed
|
|
||||||
"""
|
|
||||||
root, version = self._root_with_version()
|
|
||||||
|
|
||||||
with backend.batch_update() as batch:
|
|
||||||
batch.add_or_update_ids([version.pk])
|
|
||||||
|
|
||||||
assert backend.search_ids("latest", user=None) == [root.pk]
|
|
||||||
|
|
||||||
|
|
||||||
class TestSearch:
|
class TestSearch:
|
||||||
"""Test search query parsing and matching via search_ids."""
|
"""Test search query parsing and matching via search_ids."""
|
||||||
|
|
||||||
@@ -1136,34 +1062,6 @@ class TestMoreLikeThis:
|
|||||||
assert 150 not in ids
|
assert 150 not in ids
|
||||||
assert 151 in ids
|
assert 151 in ids
|
||||||
|
|
||||||
def test_more_like_this_ids_seeded_by_version_uses_root(
|
|
||||||
self,
|
|
||||||
backend: TantivyBackend,
|
|
||||||
) -> None:
|
|
||||||
"""A version is not indexed, so it must be looked up as its root."""
|
|
||||||
root = DocumentFactory.create(
|
|
||||||
title="Important document",
|
|
||||||
content="financial information report",
|
|
||||||
)
|
|
||||||
version = DocumentFactory.create(
|
|
||||||
title="Important document",
|
|
||||||
content="financial information report",
|
|
||||||
root_document=root,
|
|
||||||
version_index=1,
|
|
||||||
)
|
|
||||||
other = DocumentFactory.create(
|
|
||||||
title="Another document",
|
|
||||||
content="financial information report",
|
|
||||||
)
|
|
||||||
backend.add_or_update(root)
|
|
||||||
backend.add_or_update(other)
|
|
||||||
|
|
||||||
ids = backend.more_like_this_ids(doc_id=version.pk, user=None)
|
|
||||||
|
|
||||||
assert other.pk in ids
|
|
||||||
assert root.pk not in ids
|
|
||||||
assert version.pk not in ids
|
|
||||||
|
|
||||||
|
|
||||||
class TestSingleton:
|
class TestSingleton:
|
||||||
"""Test get_backend() and reset_backend() singleton lifecycle."""
|
"""Test get_backend() and reset_backend() singleton lifecycle."""
|
||||||
|
|||||||
@@ -1196,59 +1196,6 @@ class TestDocumentSearchApi(DirectoriesMixin, APITestCase):
|
|||||||
self.assertIn(d3.id, result_ids)
|
self.assertIn(d3.id, result_ids)
|
||||||
self.assertNotIn(d4.id, result_ids)
|
self.assertNotIn(d4.id, result_ids)
|
||||||
|
|
||||||
def test_search_more_like_version_uses_its_root(self) -> None:
|
|
||||||
"""
|
|
||||||
GIVEN:
|
|
||||||
- A document similar in content to a root document, and one that is not
|
|
||||||
- A version of the root document, which is never indexed
|
|
||||||
WHEN:
|
|
||||||
- API request for more like the version
|
|
||||||
THEN:
|
|
||||||
- The documents similar to the root are returned, not the version
|
|
||||||
"""
|
|
||||||
indexed = {}
|
|
||||||
for name, title, content, day in (
|
|
||||||
("root", "bank statement 1", "things i paid for in august", (2019, 3, 4)),
|
|
||||||
(
|
|
||||||
"similar",
|
|
||||||
"bank statement 3",
|
|
||||||
"things i paid for in september",
|
|
||||||
(2020, 7, 9),
|
|
||||||
),
|
|
||||||
(
|
|
||||||
"other",
|
|
||||||
"Quarterly Report",
|
|
||||||
"quarterly revenue profit margin",
|
|
||||||
(2021, 11, 30),
|
|
||||||
),
|
|
||||||
):
|
|
||||||
with time_machine.travel(
|
|
||||||
timezone.make_aware(datetime.datetime(*day)),
|
|
||||||
tick=False,
|
|
||||||
):
|
|
||||||
indexed[name] = DocumentFactory(
|
|
||||||
title=title,
|
|
||||||
content=content,
|
|
||||||
created=datetime.date(*day),
|
|
||||||
added=timezone.make_aware(datetime.datetime(*day)),
|
|
||||||
)
|
|
||||||
version = DocumentFactory(
|
|
||||||
root_document=indexed["root"],
|
|
||||||
version_index=1,
|
|
||||||
content="things i paid for in august",
|
|
||||||
)
|
|
||||||
backend = get_backend()
|
|
||||||
for document in indexed.values():
|
|
||||||
backend.add_or_update(document)
|
|
||||||
|
|
||||||
response = self.client.get(f"/api/documents/?more_like_id={version.id}")
|
|
||||||
|
|
||||||
self.assertEqual(response.status_code, status.HTTP_200_OK)
|
|
||||||
result_ids = [r["id"] for r in response.data["results"]]
|
|
||||||
self.assertIn(indexed["similar"].id, result_ids)
|
|
||||||
self.assertNotIn(indexed["other"].id, result_ids)
|
|
||||||
self.assertNotIn(version.id, result_ids)
|
|
||||||
|
|
||||||
def test_more_like_requires_id_of_existing_document(self) -> None:
|
def test_more_like_requires_id_of_existing_document(self) -> None:
|
||||||
"""
|
"""
|
||||||
GIVEN:
|
GIVEN:
|
||||||
@@ -1314,39 +1261,6 @@ class TestDocumentSearchApi(DirectoriesMixin, APITestCase):
|
|||||||
self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN)
|
self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN)
|
||||||
self.assertEqual(response.content, b"Insufficient permissions.")
|
self.assertEqual(response.content, b"Insufficient permissions.")
|
||||||
|
|
||||||
def test_search_more_like_version_requires_view_permission_on_its_root(
|
|
||||||
self,
|
|
||||||
) -> None:
|
|
||||||
"""
|
|
||||||
GIVEN:
|
|
||||||
- Another user's private root document with an unowned version
|
|
||||||
WHEN:
|
|
||||||
- A user without access requests more-like-this for the version
|
|
||||||
THEN:
|
|
||||||
- The request is rejected, the version's missing owner grants nothing
|
|
||||||
"""
|
|
||||||
owner = UserFactory(username="owner")
|
|
||||||
attacker = UserFactory(username="attacker")
|
|
||||||
grant_global(attacker, "view_document")
|
|
||||||
private_root = DocumentFactory(
|
|
||||||
title="private bank statement",
|
|
||||||
content="quarterly treasury bank statement wire transfer",
|
|
||||||
owner=owner,
|
|
||||||
)
|
|
||||||
version = DocumentFactory(
|
|
||||||
root_document=private_root,
|
|
||||||
version_index=1,
|
|
||||||
owner=None,
|
|
||||||
)
|
|
||||||
get_backend().add_or_update(private_root)
|
|
||||||
|
|
||||||
self.client.force_authenticate(user=attacker)
|
|
||||||
|
|
||||||
response = self.client.get(f"/api/documents/?more_like_id={version.id}")
|
|
||||||
|
|
||||||
self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN)
|
|
||||||
self.assertEqual(response.content, b"Insufficient permissions.")
|
|
||||||
|
|
||||||
def test_search_filtering(self) -> None:
|
def test_search_filtering(self) -> None:
|
||||||
t = Tag.objects.create(name="tag")
|
t = Tag.objects.create(name="tag")
|
||||||
t2 = Tag.objects.create(name="tag2")
|
t2 = Tag.objects.create(name="tag2")
|
||||||
|
|||||||
@@ -22,7 +22,6 @@ from documents.models import Document
|
|||||||
from documents.tasks import update_document_content_maybe_archive_file
|
from documents.tasks import update_document_content_maybe_archive_file
|
||||||
from paperless_testing.assertions import FileSystemAssertsMixin
|
from paperless_testing.assertions import FileSystemAssertsMixin
|
||||||
from paperless_testing.dirs import DirectoriesMixin
|
from paperless_testing.dirs import DirectoriesMixin
|
||||||
from paperless_testing.factories import DocumentFactory
|
|
||||||
|
|
||||||
sample_file: Path = Path(__file__).parent / "samples" / "simple.pdf"
|
sample_file: Path = Path(__file__).parent / "samples" / "simple.pdf"
|
||||||
|
|
||||||
@@ -117,27 +116,6 @@ class TestMakeIndex:
|
|||||||
call_command("document_index", "reindex", skip_checks=True)
|
call_command("document_index", "reindex", skip_checks=True)
|
||||||
mock_get_backend.return_value.rebuild.assert_called_once()
|
mock_get_backend.return_value.rebuild.assert_called_once()
|
||||||
|
|
||||||
def test_reindex_skips_versions(self, mocker: MockerFixture) -> None:
|
|
||||||
"""
|
|
||||||
GIVEN:
|
|
||||||
- A root document with a version
|
|
||||||
WHEN:
|
|
||||||
- The reindex command runs
|
|
||||||
THEN:
|
|
||||||
- Only the root document is handed to the rebuild, since a version
|
|
||||||
is indexed as its root
|
|
||||||
"""
|
|
||||||
root = DocumentFactory()
|
|
||||||
DocumentFactory(root_document=root, version_index=1)
|
|
||||||
mock_get_backend = mocker.patch(
|
|
||||||
"documents.management.commands.document_index.get_backend",
|
|
||||||
)
|
|
||||||
|
|
||||||
call_command("document_index", "reindex", skip_checks=True)
|
|
||||||
|
|
||||||
documents = mock_get_backend.return_value.rebuild.call_args.args[0]
|
|
||||||
assert list(documents.values_list("pk", flat=True)) == [root.pk]
|
|
||||||
|
|
||||||
def test_optimize(self) -> None:
|
def test_optimize(self) -> None:
|
||||||
"""Optimize command must execute without error (Tantivy handles optimization automatically)."""
|
"""Optimize command must execute without error (Tantivy handles optimization automatically)."""
|
||||||
call_command("document_index", "optimize", skip_checks=True)
|
call_command("document_index", "optimize", skip_checks=True)
|
||||||
|
|||||||
@@ -18,7 +18,6 @@ from documents.models import Correspondent
|
|||||||
from documents.models import DocumentType
|
from documents.models import DocumentType
|
||||||
from documents.models import StoragePath
|
from documents.models import StoragePath
|
||||||
from documents.models import Tag
|
from documents.models import Tag
|
||||||
from documents.permissions import has_perms_owner_aware
|
|
||||||
from documents.permissions import permitted_document_ids
|
from documents.permissions import permitted_document_ids
|
||||||
from documents.permissions import permitted_object_ids
|
from documents.permissions import permitted_object_ids
|
||||||
from documents.permissions import restrict_queryset_to_visible
|
from documents.permissions import restrict_queryset_to_visible
|
||||||
@@ -471,92 +470,6 @@ class TestPermittedDocumentIdsVersions:
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.django_db
|
|
||||||
class TestHasPermsOwnerAwareVersions:
|
|
||||||
"""
|
|
||||||
The single-object check agrees with permitted_document_ids: a version is
|
|
||||||
authorized by its root document.
|
|
||||||
"""
|
|
||||||
|
|
||||||
@pytest.mark.parametrize(
|
|
||||||
("root_owner", "version_owner", "expected"),
|
|
||||||
[
|
|
||||||
pytest.param(
|
|
||||||
"other",
|
|
||||||
"nobody",
|
|
||||||
False,
|
|
||||||
id="unowned-version-of-private-root",
|
|
||||||
),
|
|
||||||
pytest.param("other", "user", False, id="own-version-of-private-root"),
|
|
||||||
pytest.param("user", "other", True, id="foreign-version-of-own-root"),
|
|
||||||
pytest.param("nobody", "other", True, id="private-version-of-unowned-root"),
|
|
||||||
],
|
|
||||||
)
|
|
||||||
def test_version_follows_root_owner(
|
|
||||||
self,
|
|
||||||
root_owner: str,
|
|
||||||
version_owner: str,
|
|
||||||
*,
|
|
||||||
expected: bool,
|
|
||||||
) -> None:
|
|
||||||
"""
|
|
||||||
GIVEN:
|
|
||||||
- A root document and a version with differing owners
|
|
||||||
WHEN:
|
|
||||||
- The single-object check runs for the version
|
|
||||||
THEN:
|
|
||||||
- The version is allowed exactly when its root is
|
|
||||||
"""
|
|
||||||
user = UserFactory()
|
|
||||||
owners = {"user": user, "other": UserFactory(), "nobody": None}
|
|
||||||
root = DocumentFactory(owner=owners[root_owner])
|
|
||||||
version = DocumentFactory(root_document=root, owner=owners[version_owner])
|
|
||||||
|
|
||||||
assert has_perms_owner_aware(user, "view_document", version) is expected
|
|
||||||
assert has_perms_owner_aware(user, "view_document", root) is expected
|
|
||||||
|
|
||||||
def test_grant_on_root_applies_and_grant_on_version_does_not(self) -> None:
|
|
||||||
"""
|
|
||||||
GIVEN:
|
|
||||||
- A private root with a version, and a second private root with a version
|
|
||||||
- The user may change only the first root, and was granted the second
|
|
||||||
root's version directly
|
|
||||||
WHEN:
|
|
||||||
- The single-object check runs for each version
|
|
||||||
THEN:
|
|
||||||
- Only the first root's version is allowed
|
|
||||||
"""
|
|
||||||
user = UserFactory()
|
|
||||||
shared_root = DocumentFactory(owner=UserFactory())
|
|
||||||
shared_version = DocumentFactory(root_document=shared_root, owner=UserFactory())
|
|
||||||
private_root = DocumentFactory(owner=UserFactory())
|
|
||||||
private_version = DocumentFactory(
|
|
||||||
root_document=private_root,
|
|
||||||
owner=UserFactory(),
|
|
||||||
)
|
|
||||||
grant_object(user, shared_root, "change_document")
|
|
||||||
grant_object(user, private_version, "change_document")
|
|
||||||
|
|
||||||
assert has_perms_owner_aware(user, "change_document", shared_version)
|
|
||||||
assert not has_perms_owner_aware(user, "change_document", private_version)
|
|
||||||
|
|
||||||
def test_other_models_use_their_own_owner(self) -> None:
|
|
||||||
"""
|
|
||||||
GIVEN:
|
|
||||||
- A tag owned by someone else, and one owned by the user
|
|
||||||
WHEN:
|
|
||||||
- The single-object check runs for each
|
|
||||||
THEN:
|
|
||||||
- Only the user's own tag is allowed without a grant
|
|
||||||
"""
|
|
||||||
user = UserFactory()
|
|
||||||
mine = TagFactory(owner=user)
|
|
||||||
theirs = TagFactory(owner=UserFactory())
|
|
||||||
|
|
||||||
assert has_perms_owner_aware(user, "view_tag", mine)
|
|
||||||
assert not has_perms_owner_aware(user, "view_tag", theirs)
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.django_db
|
@pytest.mark.django_db
|
||||||
class TestAiChatAllDocumentsPermissionBoundary:
|
class TestAiChatAllDocumentsPermissionBoundary:
|
||||||
"""
|
"""
|
||||||
|
|||||||
@@ -310,7 +310,7 @@ class TestUpdateContent(DirectoriesMixin, TestCase):
|
|||||||
|
|
||||||
@mock.patch("documents.tasks.clear_document_caches")
|
@mock.patch("documents.tasks.clear_document_caches")
|
||||||
@mock.patch("documents.search.get_backend")
|
@mock.patch("documents.search.get_backend")
|
||||||
def test_update_content_version_clears_caches_for_root(
|
def test_update_content_version_indexes_root(
|
||||||
self,
|
self,
|
||||||
mock_get_backend: mock.Mock,
|
mock_get_backend: mock.Mock,
|
||||||
mock_clear_caches: mock.Mock,
|
mock_clear_caches: mock.Mock,
|
||||||
@@ -321,8 +321,8 @@ class TestUpdateContent(DirectoriesMixin, TestCase):
|
|||||||
WHEN:
|
WHEN:
|
||||||
- Update content task is called for the version
|
- Update content task is called for the version
|
||||||
THEN:
|
THEN:
|
||||||
- The version's content is updated, not the root's
|
- The version's content is updated
|
||||||
- The document is indexed
|
- The root document is indexed rather than the version
|
||||||
- Caches are cleared for both
|
- Caches are cleared for both
|
||||||
"""
|
"""
|
||||||
root, version = self._create_root_with_version()
|
root, version = self._create_root_with_version()
|
||||||
@@ -334,7 +334,8 @@ class TestUpdateContent(DirectoriesMixin, TestCase):
|
|||||||
"my document",
|
"my document",
|
||||||
)
|
)
|
||||||
self.assertEqual(Document.objects.get(pk=root.pk).content, "root content")
|
self.assertEqual(Document.objects.get(pk=root.pk).content, "root content")
|
||||||
mock_get_backend.return_value.add_or_update.assert_called_once()
|
indexed = mock_get_backend.return_value.add_or_update.call_args.args[0]
|
||||||
|
self.assertEqual(indexed.pk, root.pk)
|
||||||
mock_clear_caches.assert_has_calls(
|
mock_clear_caches.assert_has_calls(
|
||||||
[mock.call(version.pk), mock.call(root.pk)],
|
[mock.call(version.pk), mock.call(root.pk)],
|
||||||
)
|
)
|
||||||
@@ -342,7 +343,7 @@ class TestUpdateContent(DirectoriesMixin, TestCase):
|
|||||||
@override_settings(AI_ENABLED=True, LLM_EMBEDDING_BACKEND="huggingface")
|
@override_settings(AI_ENABLED=True, LLM_EMBEDDING_BACKEND="huggingface")
|
||||||
@mock.patch("documents.tasks.llm_index_add_or_update_document")
|
@mock.patch("documents.tasks.llm_index_add_or_update_document")
|
||||||
@mock.patch("documents.search.get_backend")
|
@mock.patch("documents.search.get_backend")
|
||||||
def test_update_content_version_updates_llm_index(
|
def test_update_content_version_updates_llm_index_for_root(
|
||||||
self,
|
self,
|
||||||
mock_get_backend: mock.Mock,
|
mock_get_backend: mock.Mock,
|
||||||
mock_llm_index: mock.Mock,
|
mock_llm_index: mock.Mock,
|
||||||
@@ -354,13 +355,14 @@ class TestUpdateContent(DirectoriesMixin, TestCase):
|
|||||||
WHEN:
|
WHEN:
|
||||||
- Update content task is called for the version
|
- Update content task is called for the version
|
||||||
THEN:
|
THEN:
|
||||||
- The LLM index is updated
|
- The LLM index is updated for the root document, not the version
|
||||||
"""
|
"""
|
||||||
_, version = self._create_root_with_version()
|
root, version = self._create_root_with_version()
|
||||||
|
|
||||||
tasks.update_document_content_maybe_archive_file(version.pk)
|
tasks.update_document_content_maybe_archive_file(version.pk)
|
||||||
|
|
||||||
mock_llm_index.assert_called_once()
|
mock_llm_index.assert_called_once()
|
||||||
|
self.assertEqual(mock_llm_index.call_args.args[0].pk, root.pk)
|
||||||
|
|
||||||
|
|
||||||
class TestUpdateContentRemoteOCR(DirectoriesMixin, TestCase):
|
class TestUpdateContentRemoteOCR(DirectoriesMixin, TestCase):
|
||||||
|
|||||||
@@ -17,26 +17,9 @@ from django.db.models.functions import RowNumber
|
|||||||
from documents.models import Document
|
from documents.models import Document
|
||||||
|
|
||||||
if TYPE_CHECKING:
|
if TYPE_CHECKING:
|
||||||
from collections.abc import Iterable
|
|
||||||
|
|
||||||
from rest_framework.request import Request
|
from rest_framework.request import Request
|
||||||
|
|
||||||
|
|
||||||
def root_document_ids(ids: Iterable[int]) -> QuerySet[int]:
|
|
||||||
"""
|
|
||||||
The ids of the root documents of the given documents: a root stands for
|
|
||||||
itself and a version for its root. Only the indexes' bookkeeping needs
|
|
||||||
this, since they hold root documents only.
|
|
||||||
"""
|
|
||||||
return (
|
|
||||||
Document.objects.filter(pk__in=ids)
|
|
||||||
.annotate(root_id=Coalesce("root_document_id", "id"))
|
|
||||||
.order_by()
|
|
||||||
.values_list("root_id", flat=True)
|
|
||||||
.distinct()
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def versions_newest_first(documents: QuerySet[Document]) -> QuerySet[Document]:
|
def versions_newest_first(documents: QuerySet[Document]) -> QuerySet[Document]:
|
||||||
"""
|
"""
|
||||||
Sorts versions so the newest one comes first using version_index and not on id,
|
Sorts versions so the newest one comes first using version_index and not on id,
|
||||||
|
|||||||
+28
-14
@@ -329,10 +329,9 @@ def _get_tantivy_query_and_mode(params):
|
|||||||
def _get_more_like_id(query_params: dict[str, Any], user: User | None) -> int:
|
def _get_more_like_id(query_params: dict[str, Any], user: User | None) -> int:
|
||||||
try:
|
try:
|
||||||
more_like_doc_id = int(query_params["more_like_id"])
|
more_like_doc_id = int(query_params["more_like_id"])
|
||||||
more_like_doc = Document.objects.select_related(
|
more_like_doc = Document.objects.select_related("owner").get(
|
||||||
"owner",
|
pk=more_like_doc_id,
|
||||||
"root_document__owner",
|
)
|
||||||
).get(pk=more_like_doc_id)
|
|
||||||
except (TypeError, ValueError, Document.DoesNotExist):
|
except (TypeError, ValueError, Document.DoesNotExist):
|
||||||
raise PermissionDenied(_("Invalid more_like_id"))
|
raise PermissionDenied(_("Invalid more_like_id"))
|
||||||
|
|
||||||
@@ -343,8 +342,7 @@ def _get_more_like_id(query_params: dict[str, Any], user: User | None) -> int:
|
|||||||
):
|
):
|
||||||
raise PermissionDenied(_("Insufficient permissions."))
|
raise PermissionDenied(_("Insufficient permissions."))
|
||||||
|
|
||||||
# Only root documents are indexed, a version stands for its root
|
return more_like_doc_id
|
||||||
return more_like_doc.root_document_id or more_like_doc.pk
|
|
||||||
|
|
||||||
|
|
||||||
class SearchParams(NamedTuple):
|
class SearchParams(NamedTuple):
|
||||||
@@ -1561,7 +1559,7 @@ class DocumentViewSet(
|
|||||||
if request.user is not None and not has_perms_owner_aware(
|
if request.user is not None and not has_perms_owner_aware(
|
||||||
request.user,
|
request.user,
|
||||||
"change_document",
|
"change_document",
|
||||||
doc,
|
get_root_document(doc),
|
||||||
):
|
):
|
||||||
return HttpResponseForbidden("Insufficient permissions")
|
return HttpResponseForbidden("Insufficient permissions")
|
||||||
|
|
||||||
@@ -1624,7 +1622,7 @@ class DocumentViewSet(
|
|||||||
if request.user is not None and not has_perms_owner_aware(
|
if request.user is not None and not has_perms_owner_aware(
|
||||||
request.user,
|
request.user,
|
||||||
"change_document",
|
"change_document",
|
||||||
doc,
|
get_root_document(doc),
|
||||||
):
|
):
|
||||||
return HttpResponseForbidden("Insufficient permissions")
|
return HttpResponseForbidden("Insufficient permissions")
|
||||||
|
|
||||||
@@ -1877,7 +1875,7 @@ class DocumentViewSet(
|
|||||||
if currentUser is not None and not has_perms_owner_aware(
|
if currentUser is not None and not has_perms_owner_aware(
|
||||||
currentUser,
|
currentUser,
|
||||||
"view_document",
|
"view_document",
|
||||||
doc,
|
get_root_document(doc),
|
||||||
):
|
):
|
||||||
return HttpResponseForbidden("Insufficient permissions to view notes")
|
return HttpResponseForbidden("Insufficient permissions to view notes")
|
||||||
except Document.DoesNotExist:
|
except Document.DoesNotExist:
|
||||||
@@ -1899,7 +1897,7 @@ class DocumentViewSet(
|
|||||||
if currentUser is not None and not has_perms_owner_aware(
|
if currentUser is not None and not has_perms_owner_aware(
|
||||||
currentUser,
|
currentUser,
|
||||||
"change_document",
|
"change_document",
|
||||||
doc,
|
get_root_document(doc),
|
||||||
):
|
):
|
||||||
return HttpResponseForbidden(
|
return HttpResponseForbidden(
|
||||||
"Insufficient permissions to create notes",
|
"Insufficient permissions to create notes",
|
||||||
@@ -1942,7 +1940,7 @@ class DocumentViewSet(
|
|||||||
if currentUser is not None and not has_perms_owner_aware(
|
if currentUser is not None and not has_perms_owner_aware(
|
||||||
currentUser,
|
currentUser,
|
||||||
"change_document",
|
"change_document",
|
||||||
doc,
|
get_root_document(doc),
|
||||||
):
|
):
|
||||||
return HttpResponseForbidden("Insufficient permissions to delete notes")
|
return HttpResponseForbidden("Insufficient permissions to delete notes")
|
||||||
|
|
||||||
@@ -1992,7 +1990,7 @@ class DocumentViewSet(
|
|||||||
if currentUser is not None and not has_perms_owner_aware(
|
if currentUser is not None and not has_perms_owner_aware(
|
||||||
currentUser,
|
currentUser,
|
||||||
"change_document",
|
"change_document",
|
||||||
doc,
|
get_root_document(doc),
|
||||||
):
|
):
|
||||||
return HttpResponseForbidden(
|
return HttpResponseForbidden(
|
||||||
"Insufficient permissions to add share link",
|
"Insufficient permissions to add share link",
|
||||||
@@ -2453,7 +2451,7 @@ class ChatStreamingView(GenericAPIView[Any]):
|
|||||||
if not has_perms_owner_aware(
|
if not has_perms_owner_aware(
|
||||||
request.user,
|
request.user,
|
||||||
"view_document",
|
"view_document",
|
||||||
document,
|
get_root_document(document),
|
||||||
):
|
):
|
||||||
return HttpResponseForbidden("Insufficient permissions")
|
return HttpResponseForbidden("Insufficient permissions")
|
||||||
|
|
||||||
@@ -5602,7 +5600,23 @@ class TrashView(ListModelMixin, PassUserMixin):
|
|||||||
|
|
||||||
class _TrashPermittedObjectsFilter(PermittedObjectsFilter):
|
class _TrashPermittedObjectsFilter(PermittedObjectsFilter):
|
||||||
include_granted = False
|
include_granted = False
|
||||||
parent_field = "root_document"
|
|
||||||
|
def filter_queryset(self, request, queryset, view):
|
||||||
|
if request.user.is_superuser or not request.user.is_active:
|
||||||
|
return super().filter_queryset(request, queryset, view)
|
||||||
|
|
||||||
|
# A version belongs to whoever owns its root
|
||||||
|
def owned_or_unowned(prefix: str) -> Q:
|
||||||
|
return Q(**{f"{prefix}owner": request.user}) | Q(
|
||||||
|
**{f"{prefix}owner__isnull": True},
|
||||||
|
)
|
||||||
|
|
||||||
|
return queryset.filter(
|
||||||
|
(Q(root_document__isnull=True) & owned_or_unowned(""))
|
||||||
|
| (
|
||||||
|
Q(root_document__isnull=False) & owned_or_unowned("root_document__")
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
filter_backends = (_TrashPermittedObjectsFilter,)
|
filter_backends = (_TrashPermittedObjectsFilter,)
|
||||||
pagination_class = StandardPagination
|
pagination_class = StandardPagination
|
||||||
|
|||||||
@@ -2,7 +2,7 @@ msgid ""
|
|||||||
msgstr ""
|
msgstr ""
|
||||||
"Project-Id-Version: paperless-ngx\n"
|
"Project-Id-Version: paperless-ngx\n"
|
||||||
"Report-Msgid-Bugs-To: \n"
|
"Report-Msgid-Bugs-To: \n"
|
||||||
"POT-Creation-Date: 2026-10-12 19:12+0000\n"
|
"POT-Creation-Date: 2026-10-10 22:04+0000\n"
|
||||||
"PO-Revision-Date: 2022-02-17 04:17\n"
|
"PO-Revision-Date: 2022-02-17 04:17\n"
|
||||||
"Last-Translator: \n"
|
"Last-Translator: \n"
|
||||||
"Language-Team: English\n"
|
"Language-Team: English\n"
|
||||||
@@ -21,39 +21,39 @@ msgstr ""
|
|||||||
msgid "Documents"
|
msgid "Documents"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/filters.py:464
|
#: documents/filters.py:463
|
||||||
msgid "Value must be valid JSON."
|
msgid "Value must be valid JSON."
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/filters.py:483
|
#: documents/filters.py:482
|
||||||
msgid "Invalid custom field query expression"
|
msgid "Invalid custom field query expression"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/filters.py:493
|
#: documents/filters.py:492
|
||||||
msgid "Invalid expression list. Must be nonempty."
|
msgid "Invalid expression list. Must be nonempty."
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/filters.py:514
|
#: documents/filters.py:513
|
||||||
msgid "Invalid logical operator {op!r}"
|
msgid "Invalid logical operator {op!r}"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/filters.py:528
|
#: documents/filters.py:527
|
||||||
msgid "Maximum number of query conditions exceeded."
|
msgid "Maximum number of query conditions exceeded."
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/filters.py:592
|
#: documents/filters.py:591
|
||||||
msgid "{name!r} is not a valid custom field."
|
msgid "{name!r} is not a valid custom field."
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/filters.py:629
|
#: documents/filters.py:628
|
||||||
msgid "{data_type} does not support query expr {expr!r}."
|
msgid "{data_type} does not support query expr {expr!r}."
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/filters.py:748 documents/models.py:137
|
#: documents/filters.py:747 documents/models.py:137
|
||||||
msgid "Maximum nesting depth exceeded."
|
msgid "Maximum nesting depth exceeded."
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/filters.py:1125
|
#: documents/filters.py:1109
|
||||||
msgid "Custom field not found"
|
msgid "Custom field not found"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
@@ -1652,49 +1652,49 @@ msgstr ""
|
|||||||
msgid "workflow runs"
|
msgid "workflow runs"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/serialisers.py:515 documents/serialisers.py:872
|
#: documents/serialisers.py:516 documents/serialisers.py:873
|
||||||
#: documents/serialisers.py:2902 documents/views.py:344 documents/views.py:2752
|
#: documents/serialisers.py:2903 documents/views.py:343 documents/views.py:2750
|
||||||
#: paperless_mail/serialisers.py:156
|
#: paperless_mail/serialisers.py:156
|
||||||
msgid "Insufficient permissions."
|
msgid "Insufficient permissions."
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/serialisers.py:708
|
#: documents/serialisers.py:709
|
||||||
msgid "Invalid color."
|
msgid "Invalid color."
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/serialisers.py:2369
|
#: documents/serialisers.py:2370
|
||||||
#, python-format
|
#, python-format
|
||||||
msgid "File type %(type)s not supported"
|
msgid "File type %(type)s not supported"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/serialisers.py:2413
|
#: documents/serialisers.py:2414
|
||||||
#, python-format
|
#, python-format
|
||||||
msgid "Custom field id must be an integer: %(id)s"
|
msgid "Custom field id must be an integer: %(id)s"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/serialisers.py:2420
|
#: documents/serialisers.py:2421
|
||||||
#, python-format
|
#, python-format
|
||||||
msgid "Custom field with id %(id)s does not exist"
|
msgid "Custom field with id %(id)s does not exist"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/serialisers.py:2437 documents/serialisers.py:2447
|
#: documents/serialisers.py:2438 documents/serialisers.py:2448
|
||||||
msgid ""
|
msgid ""
|
||||||
"Custom fields must be a list of integers or an object mapping ids to values."
|
"Custom fields must be a list of integers or an object mapping ids to values."
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/serialisers.py:2442
|
#: documents/serialisers.py:2443
|
||||||
msgid "Some custom fields don't exist or were specified twice."
|
msgid "Some custom fields don't exist or were specified twice."
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/serialisers.py:2589
|
#: documents/serialisers.py:2590
|
||||||
msgid "Invalid variable detected."
|
msgid "Invalid variable detected."
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/serialisers.py:2958
|
#: documents/serialisers.py:2959
|
||||||
msgid "Duplicate document identifiers are not allowed."
|
msgid "Duplicate document identifiers are not allowed."
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/serialisers.py:2988 documents/views.py:4799
|
#: documents/serialisers.py:2989 documents/views.py:4797
|
||||||
#, python-format
|
#, python-format
|
||||||
msgid "Documents not found: %(ids)s"
|
msgid "Documents not found: %(ids)s"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
@@ -1945,40 +1945,40 @@ msgstr ""
|
|||||||
msgid ", "
|
msgid ", "
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/views.py:337 documents/views.py:2749
|
#: documents/views.py:336 documents/views.py:2747
|
||||||
msgid "Invalid more_like_id"
|
msgid "Invalid more_like_id"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/views.py:1684
|
#: documents/views.py:1682
|
||||||
msgid "Invalid AI configuration."
|
msgid "Invalid AI configuration."
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/views.py:1695
|
#: documents/views.py:1693
|
||||||
msgid "AI backend request timed out."
|
msgid "AI backend request timed out."
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/views.py:1707
|
#: documents/views.py:1705
|
||||||
msgid "AI backend rejected the request. Check logs for details."
|
msgid "AI backend rejected the request. Check logs for details."
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/views.py:2574 documents/views.py:2890
|
#: documents/views.py:2572 documents/views.py:2888
|
||||||
msgid "Specify only one of text, title_search, query, or more_like_id."
|
msgid "Specify only one of text, title_search, query, or more_like_id."
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/views.py:4815
|
#: documents/views.py:4813
|
||||||
#, python-format
|
#, python-format
|
||||||
msgid "Insufficient permissions to share document %(id)s."
|
msgid "Insufficient permissions to share document %(id)s."
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/views.py:4862
|
#: documents/views.py:4860
|
||||||
msgid "Bundle is already being processed."
|
msgid "Bundle is already being processed."
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/views.py:4926
|
#: documents/views.py:4924
|
||||||
msgid "The share link bundle is still being prepared. Please try again later."
|
msgid "The share link bundle is still being prepared. Please try again later."
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/views.py:4940
|
#: documents/views.py:4938
|
||||||
msgid "The share link bundle is unavailable."
|
msgid "The share link bundle is unavailable."
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
@@ -2270,151 +2270,151 @@ msgstr ""
|
|||||||
msgid "paperless application settings"
|
msgid "paperless application settings"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: paperless/settings/__init__.py:562
|
#: paperless/settings/__init__.py:561
|
||||||
msgid "English (US)"
|
msgid "English (US)"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: paperless/settings/__init__.py:563
|
#: paperless/settings/__init__.py:562
|
||||||
msgid "Arabic"
|
msgid "Arabic"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: paperless/settings/__init__.py:564
|
#: paperless/settings/__init__.py:563
|
||||||
msgid "Afrikaans"
|
msgid "Afrikaans"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: paperless/settings/__init__.py:565
|
#: paperless/settings/__init__.py:564
|
||||||
msgid "Belarusian"
|
msgid "Belarusian"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: paperless/settings/__init__.py:566
|
#: paperless/settings/__init__.py:565
|
||||||
msgid "Bulgarian"
|
msgid "Bulgarian"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: paperless/settings/__init__.py:567
|
#: paperless/settings/__init__.py:566
|
||||||
msgid "Catalan"
|
msgid "Catalan"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: paperless/settings/__init__.py:568
|
#: paperless/settings/__init__.py:567
|
||||||
msgid "Czech"
|
msgid "Czech"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: paperless/settings/__init__.py:569
|
#: paperless/settings/__init__.py:568
|
||||||
msgid "Danish"
|
msgid "Danish"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: paperless/settings/__init__.py:570
|
#: paperless/settings/__init__.py:569
|
||||||
msgid "German"
|
msgid "German"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: paperless/settings/__init__.py:571
|
#: paperless/settings/__init__.py:570
|
||||||
msgid "Greek"
|
msgid "Greek"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: paperless/settings/__init__.py:572
|
#: paperless/settings/__init__.py:571
|
||||||
msgid "English (GB)"
|
msgid "English (GB)"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: paperless/settings/__init__.py:573
|
#: paperless/settings/__init__.py:572
|
||||||
msgid "Spanish"
|
msgid "Spanish"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: paperless/settings/__init__.py:574
|
#: paperless/settings/__init__.py:573
|
||||||
msgid "Persian"
|
msgid "Persian"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: paperless/settings/__init__.py:575
|
#: paperless/settings/__init__.py:574
|
||||||
msgid "Finnish"
|
msgid "Finnish"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: paperless/settings/__init__.py:576
|
#: paperless/settings/__init__.py:575
|
||||||
msgid "French"
|
msgid "French"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: paperless/settings/__init__.py:577
|
#: paperless/settings/__init__.py:576
|
||||||
msgid "Hungarian"
|
msgid "Hungarian"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: paperless/settings/__init__.py:578
|
#: paperless/settings/__init__.py:577
|
||||||
msgid "Indonesian"
|
msgid "Indonesian"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: paperless/settings/__init__.py:579
|
#: paperless/settings/__init__.py:578
|
||||||
msgid "Italian"
|
msgid "Italian"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: paperless/settings/__init__.py:580
|
#: paperless/settings/__init__.py:579
|
||||||
msgid "Japanese"
|
msgid "Japanese"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: paperless/settings/__init__.py:581
|
#: paperless/settings/__init__.py:580
|
||||||
msgid "Korean"
|
msgid "Korean"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: paperless/settings/__init__.py:582
|
#: paperless/settings/__init__.py:581
|
||||||
msgid "Luxembourgish"
|
msgid "Luxembourgish"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: paperless/settings/__init__.py:583
|
#: paperless/settings/__init__.py:582
|
||||||
msgid "Norwegian"
|
msgid "Norwegian"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: paperless/settings/__init__.py:584
|
#: paperless/settings/__init__.py:583
|
||||||
msgid "Dutch"
|
msgid "Dutch"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: paperless/settings/__init__.py:585
|
#: paperless/settings/__init__.py:584
|
||||||
msgid "Polish"
|
msgid "Polish"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: paperless/settings/__init__.py:586
|
#: paperless/settings/__init__.py:585
|
||||||
msgid "Portuguese (Brazil)"
|
msgid "Portuguese (Brazil)"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: paperless/settings/__init__.py:587
|
#: paperless/settings/__init__.py:586
|
||||||
msgid "Portuguese"
|
msgid "Portuguese"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: paperless/settings/__init__.py:588
|
#: paperless/settings/__init__.py:587
|
||||||
msgid "Romanian"
|
msgid "Romanian"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: paperless/settings/__init__.py:589
|
#: paperless/settings/__init__.py:588
|
||||||
msgid "Russian"
|
msgid "Russian"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: paperless/settings/__init__.py:590
|
#: paperless/settings/__init__.py:589
|
||||||
msgid "Slovak"
|
msgid "Slovak"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: paperless/settings/__init__.py:591
|
#: paperless/settings/__init__.py:590
|
||||||
msgid "Slovenian"
|
msgid "Slovenian"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: paperless/settings/__init__.py:592
|
#: paperless/settings/__init__.py:591
|
||||||
msgid "Serbian"
|
msgid "Serbian"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: paperless/settings/__init__.py:593
|
#: paperless/settings/__init__.py:592
|
||||||
msgid "Swedish"
|
msgid "Swedish"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: paperless/settings/__init__.py:594
|
#: paperless/settings/__init__.py:593
|
||||||
msgid "Turkish"
|
msgid "Turkish"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: paperless/settings/__init__.py:595
|
#: paperless/settings/__init__.py:594
|
||||||
msgid "Ukrainian"
|
msgid "Ukrainian"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: paperless/settings/__init__.py:596
|
#: paperless/settings/__init__.py:595
|
||||||
msgid "Vietnamese"
|
msgid "Vietnamese"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: paperless/settings/__init__.py:597
|
#: paperless/settings/__init__.py:596
|
||||||
msgid "Chinese Simplified"
|
msgid "Chinese Simplified"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: paperless/settings/__init__.py:598
|
#: paperless/settings/__init__.py:597
|
||||||
msgid "Chinese Traditional"
|
msgid "Chinese Traditional"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
|
|||||||
@@ -1,4 +1,5 @@
|
|||||||
import datetime
|
import datetime
|
||||||
|
import json
|
||||||
import logging
|
import logging
|
||||||
import logging.config
|
import logging.config
|
||||||
import math
|
import math
|
||||||
@@ -6,6 +7,7 @@ import multiprocessing
|
|||||||
import os
|
import os
|
||||||
import tempfile
|
import tempfile
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
from typing import Final
|
from typing import Final
|
||||||
from urllib.parse import urlparse
|
from urllib.parse import urlparse
|
||||||
|
|
||||||
@@ -23,7 +25,6 @@ from paperless.settings.parsers import get_bool_from_env
|
|||||||
from paperless.settings.parsers import get_choice_from_env
|
from paperless.settings.parsers import get_choice_from_env
|
||||||
from paperless.settings.parsers import get_float_from_env
|
from paperless.settings.parsers import get_float_from_env
|
||||||
from paperless.settings.parsers import get_int_from_env
|
from paperless.settings.parsers import get_int_from_env
|
||||||
from paperless.settings.parsers import get_json_from_env
|
|
||||||
from paperless.settings.parsers import get_list_from_env
|
from paperless.settings.parsers import get_list_from_env
|
||||||
from paperless.settings.parsers import get_path_from_env
|
from paperless.settings.parsers import get_path_from_env
|
||||||
|
|
||||||
@@ -330,10 +331,8 @@ SOCIALACCOUNT_ALLOW_SIGNUPS = get_bool_from_env(
|
|||||||
"yes",
|
"yes",
|
||||||
)
|
)
|
||||||
SOCIALACCOUNT_AUTO_SIGNUP = get_bool_from_env("PAPERLESS_SOCIAL_AUTO_SIGNUP")
|
SOCIALACCOUNT_AUTO_SIGNUP = get_bool_from_env("PAPERLESS_SOCIAL_AUTO_SIGNUP")
|
||||||
SOCIALACCOUNT_PROVIDERS = get_json_from_env(
|
SOCIALACCOUNT_PROVIDERS = json.loads(
|
||||||
"PAPERLESS_SOCIALACCOUNT_PROVIDERS",
|
os.getenv("PAPERLESS_SOCIALACCOUNT_PROVIDERS", "{}"),
|
||||||
{},
|
|
||||||
dict,
|
|
||||||
)
|
)
|
||||||
SOCIAL_ACCOUNT_DEFAULT_GROUPS = get_list_from_env(
|
SOCIAL_ACCOUNT_DEFAULT_GROUPS = get_list_from_env(
|
||||||
"PAPERLESS_SOCIAL_ACCOUNT_DEFAULT_GROUPS",
|
"PAPERLESS_SOCIAL_ACCOUNT_DEFAULT_GROUPS",
|
||||||
@@ -489,7 +488,7 @@ ALLAUTH_TRUSTED_CLIENT_IP_HEADER = os.getenv(
|
|||||||
USE_X_FORWARDED_HOST = get_bool_from_env("PAPERLESS_USE_X_FORWARD_HOST", "false")
|
USE_X_FORWARDED_HOST = get_bool_from_env("PAPERLESS_USE_X_FORWARD_HOST", "false")
|
||||||
USE_X_FORWARDED_PORT = get_bool_from_env("PAPERLESS_USE_X_FORWARD_PORT", "false")
|
USE_X_FORWARDED_PORT = get_bool_from_env("PAPERLESS_USE_X_FORWARD_PORT", "false")
|
||||||
SECURE_PROXY_SSL_HEADER = (
|
SECURE_PROXY_SSL_HEADER = (
|
||||||
tuple(get_json_from_env("PAPERLESS_PROXY_SSL_HEADER", [], list))
|
tuple(json.loads(os.environ["PAPERLESS_PROXY_SSL_HEADER"]))
|
||||||
if "PAPERLESS_PROXY_SSL_HEADER" in os.environ
|
if "PAPERLESS_PROXY_SSL_HEADER" in os.environ
|
||||||
else None
|
else None
|
||||||
)
|
)
|
||||||
@@ -829,17 +828,23 @@ CONSUMER_DELETE_DUPLICATES = get_bool_from_env("PAPERLESS_CONSUMER_DELETE_DUPLIC
|
|||||||
CONSUMER_RECURSIVE = get_bool_from_env("PAPERLESS_CONSUMER_RECURSIVE")
|
CONSUMER_RECURSIVE = get_bool_from_env("PAPERLESS_CONSUMER_RECURSIVE")
|
||||||
|
|
||||||
# Ignore regex patterns, matched against filename only
|
# Ignore regex patterns, matched against filename only
|
||||||
CONSUMER_IGNORE_PATTERNS = get_json_from_env(
|
CONSUMER_IGNORE_PATTERNS = list(
|
||||||
|
json.loads(
|
||||||
|
os.getenv(
|
||||||
"PAPERLESS_CONSUMER_IGNORE_PATTERNS",
|
"PAPERLESS_CONSUMER_IGNORE_PATTERNS",
|
||||||
[],
|
json.dumps([]),
|
||||||
list,
|
),
|
||||||
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
# Directories to always ignore. These are matched by directory name, not full path
|
# Directories to always ignore. These are matched by directory name, not full path
|
||||||
CONSUMER_IGNORE_DIRS = get_json_from_env(
|
CONSUMER_IGNORE_DIRS = list(
|
||||||
|
json.loads(
|
||||||
|
os.getenv(
|
||||||
"PAPERLESS_CONSUMER_IGNORE_DIRS",
|
"PAPERLESS_CONSUMER_IGNORE_DIRS",
|
||||||
[],
|
json.dumps([]),
|
||||||
list,
|
),
|
||||||
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
CONSUMER_SUBDIRS_AS_TAGS = get_bool_from_env("PAPERLESS_CONSUMER_SUBDIRS_AS_TAGS")
|
CONSUMER_SUBDIRS_AS_TAGS = get_bool_from_env("PAPERLESS_CONSUMER_SUBDIRS_AS_TAGS")
|
||||||
@@ -889,10 +894,13 @@ CONSUMER_ENABLE_TAG_BARCODE: Final[bool] = get_bool_from_env(
|
|||||||
"PAPERLESS_CONSUMER_ENABLE_TAG_BARCODE",
|
"PAPERLESS_CONSUMER_ENABLE_TAG_BARCODE",
|
||||||
)
|
)
|
||||||
|
|
||||||
CONSUMER_TAG_BARCODE_MAPPING = get_json_from_env(
|
CONSUMER_TAG_BARCODE_MAPPING = dict(
|
||||||
|
json.loads(
|
||||||
|
os.getenv(
|
||||||
"PAPERLESS_CONSUMER_TAG_BARCODE_MAPPING",
|
"PAPERLESS_CONSUMER_TAG_BARCODE_MAPPING",
|
||||||
{"TAG:(.*)": "\\g<1>"},
|
'{"TAG:(.*)": "\\\\g<1>"}',
|
||||||
dict,
|
),
|
||||||
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
CONSUMER_TAG_BARCODE_SPLIT: Final[bool] = get_bool_from_env(
|
CONSUMER_TAG_BARCODE_SPLIT: Final[bool] = get_bool_from_env(
|
||||||
@@ -1080,6 +1088,25 @@ CLASSIFIER_LANGUAGES: Final[dict[str, str]] = {
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _get_llm_extra_params() -> dict[str, Any]:
|
||||||
|
"""
|
||||||
|
Parse PAPERLESS_AI_LLM_EXTRA_PARAMS, a JSON object passed straight through
|
||||||
|
to the LLM backend's request body.
|
||||||
|
"""
|
||||||
|
raw = os.getenv("PAPERLESS_AI_LLM_EXTRA_PARAMS", "{}")
|
||||||
|
try:
|
||||||
|
parsed = json.loads(raw)
|
||||||
|
except json.JSONDecodeError as e:
|
||||||
|
raise ImproperlyConfigured(
|
||||||
|
"PAPERLESS_AI_LLM_EXTRA_PARAMS must be valid JSON",
|
||||||
|
) from e
|
||||||
|
if not isinstance(parsed, dict):
|
||||||
|
raise ImproperlyConfigured(
|
||||||
|
"PAPERLESS_AI_LLM_EXTRA_PARAMS must be a JSON object",
|
||||||
|
)
|
||||||
|
return parsed
|
||||||
|
|
||||||
|
|
||||||
def _get_classifier_language_setting(ocr_lang: str) -> str | None:
|
def _get_classifier_language_setting(ocr_lang: str) -> str | None:
|
||||||
"""
|
"""
|
||||||
Maps the primary Tesseract language to the classifier's stemming
|
Maps the primary Tesseract language to the classifier's stemming
|
||||||
@@ -1241,5 +1268,4 @@ LLM_ALLOW_INTERNAL_ENDPOINTS = get_bool_from_env(
|
|||||||
"PAPERLESS_AI_LLM_ALLOW_INTERNAL_ENDPOINTS",
|
"PAPERLESS_AI_LLM_ALLOW_INTERNAL_ENDPOINTS",
|
||||||
"true",
|
"true",
|
||||||
)
|
)
|
||||||
# JSON object passed straight through to the LLM backend's request body
|
LLM_EXTRA_PARAMS = _get_llm_extra_params()
|
||||||
LLM_EXTRA_PARAMS = get_json_from_env("PAPERLESS_AI_LLM_EXTRA_PARAMS", {}, dict)
|
|
||||||
@@ -312,58 +312,3 @@ def get_choice_from_env(
|
|||||||
)
|
)
|
||||||
|
|
||||||
return value
|
return value
|
||||||
|
|
||||||
|
|
||||||
@overload
|
|
||||||
def get_json_from_env(key: str, default: T, expected_type: type[T]) -> T: ...
|
|
||||||
|
|
||||||
|
|
||||||
@overload
|
|
||||||
def get_json_from_env(
|
|
||||||
key: str,
|
|
||||||
default: Any,
|
|
||||||
expected_type: None = None,
|
|
||||||
) -> Any: ...
|
|
||||||
|
|
||||||
|
|
||||||
def get_json_from_env(
|
|
||||||
key: str,
|
|
||||||
default: Any,
|
|
||||||
expected_type: type | None = None,
|
|
||||||
) -> Any:
|
|
||||||
"""
|
|
||||||
Gets an environment variable and parses it as JSON.
|
|
||||||
|
|
||||||
Args:
|
|
||||||
key: The environment variable key
|
|
||||||
default: Python value returned when the variable is not set
|
|
||||||
expected_type: If given, the parsed value must be an instance of it
|
|
||||||
|
|
||||||
Returns:
|
|
||||||
The parsed JSON value, or the default if the variable is not set
|
|
||||||
|
|
||||||
Raises:
|
|
||||||
ValueError: If the value is not valid JSON or is not the expected type.
|
|
||||||
The message names the offending variable.
|
|
||||||
"""
|
|
||||||
raw = os.environ.get(key)
|
|
||||||
if raw is None:
|
|
||||||
return default
|
|
||||||
|
|
||||||
import json
|
|
||||||
|
|
||||||
try:
|
|
||||||
parsed = json.loads(raw)
|
|
||||||
except json.JSONDecodeError as e:
|
|
||||||
raise ValueError(
|
|
||||||
f"Environment variable '{key}' is not valid JSON: {e.msg} "
|
|
||||||
f"(line {e.lineno}, column {e.colno}). Value: {raw!r}",
|
|
||||||
) from e
|
|
||||||
|
|
||||||
if expected_type is not None and not isinstance(parsed, expected_type):
|
|
||||||
raise ValueError(
|
|
||||||
f"Environment variable '{key}' must be a JSON {expected_type.__name__}, "
|
|
||||||
f"got {type(parsed).__name__}",
|
|
||||||
)
|
|
||||||
|
|
||||||
return parsed
|
|
||||||
@@ -8,7 +8,6 @@ from paperless.settings.parsers import get_bool_from_env
|
|||||||
from paperless.settings.parsers import get_choice_from_env
|
from paperless.settings.parsers import get_choice_from_env
|
||||||
from paperless.settings.parsers import get_float_from_env
|
from paperless.settings.parsers import get_float_from_env
|
||||||
from paperless.settings.parsers import get_int_from_env
|
from paperless.settings.parsers import get_int_from_env
|
||||||
from paperless.settings.parsers import get_json_from_env
|
|
||||||
from paperless.settings.parsers import get_list_from_env
|
from paperless.settings.parsers import get_list_from_env
|
||||||
from paperless.settings.parsers import get_path_from_env
|
from paperless.settings.parsers import get_path_from_env
|
||||||
from paperless.settings.parsers import parse_dict_from_str
|
from paperless.settings.parsers import parse_dict_from_str
|
||||||
@@ -612,43 +611,3 @@ class TestGetEnvChoice:
|
|||||||
result = get_choice_from_env("TEST_ENV", large_choices)
|
result = get_choice_from_env("TEST_ENV", large_choices)
|
||||||
|
|
||||||
assert result == "option_50"
|
assert result == "option_50"
|
||||||
|
|
||||||
|
|
||||||
class TestGetJsonFromEnv:
|
|
||||||
@pytest.mark.parametrize(
|
|
||||||
("env_value", "expected_type", "expected"),
|
|
||||||
[
|
|
||||||
pytest.param(None, list, ["default"], id="unset-uses-default"),
|
|
||||||
pytest.param('["a", "b"]', list, ["a", "b"], id="list"),
|
|
||||||
pytest.param('["a"]', None, ["a"], id="no-type-check"),
|
|
||||||
],
|
|
||||||
)
|
|
||||||
def test_returns_value(
|
|
||||||
self,
|
|
||||||
mocker: MockerFixture,
|
|
||||||
env_value: str | None,
|
|
||||||
expected_type: type | None,
|
|
||||||
expected: list[str],
|
|
||||||
) -> None:
|
|
||||||
env = {} if env_value is None else {"TEST_ENV": env_value}
|
|
||||||
mocker.patch.dict("os.environ", env, clear=True)
|
|
||||||
|
|
||||||
assert get_json_from_env("TEST_ENV", ["default"], expected_type) == expected
|
|
||||||
|
|
||||||
@pytest.mark.parametrize(
|
|
||||||
("env_value", "match"),
|
|
||||||
[
|
|
||||||
pytest.param('["a", , "b"]', "'TEST_ENV' is not valid JSON", id="invalid"),
|
|
||||||
pytest.param('{"a": 1}', "must be a JSON list, got dict", id="wrong-type"),
|
|
||||||
],
|
|
||||||
)
|
|
||||||
def test_raises(
|
|
||||||
self,
|
|
||||||
mocker: MockerFixture,
|
|
||||||
env_value: str,
|
|
||||||
match: str,
|
|
||||||
) -> None:
|
|
||||||
mocker.patch.dict("os.environ", {"TEST_ENV": env_value})
|
|
||||||
|
|
||||||
with pytest.raises(ValueError, match=match):
|
|
||||||
get_json_from_env("TEST_ENV", [], list)
|
|
||||||
@@ -7,6 +7,7 @@ from django.core.exceptions import ImproperlyConfigured
|
|||||||
|
|
||||||
from paperless.settings import _get_allauth_trusted_proxy_count
|
from paperless.settings import _get_allauth_trusted_proxy_count
|
||||||
from paperless.settings import _get_classifier_language_setting
|
from paperless.settings import _get_classifier_language_setting
|
||||||
|
from paperless.settings import _get_llm_extra_params
|
||||||
from paperless.settings import _get_search_language_setting
|
from paperless.settings import _get_search_language_setting
|
||||||
from paperless.settings import _parse_paperless_url
|
from paperless.settings import _parse_paperless_url
|
||||||
from paperless.settings import default_threads_per_worker
|
from paperless.settings import default_threads_per_worker
|
||||||
@@ -166,3 +167,45 @@ class TestPaperlessURLSettings(TestCase):
|
|||||||
|
|
||||||
self.assertIn(url, settings.CSRF_TRUSTED_ORIGINS)
|
self.assertIn(url, settings.CSRF_TRUSTED_ORIGINS)
|
||||||
self.assertIn(url, settings.CORS_ALLOWED_ORIGINS)
|
self.assertIn(url, settings.CORS_ALLOWED_ORIGINS)
|
||||||
|
|
||||||
|
|
||||||
|
class TestLlmExtraParams:
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("env_value", "expected"),
|
||||||
|
[
|
||||||
|
pytest.param(None, {}, id="unset"),
|
||||||
|
pytest.param(
|
||||||
|
'{"reasoning_effort": "none"}',
|
||||||
|
{"reasoning_effort": "none"},
|
||||||
|
id="json-object",
|
||||||
|
),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_parses(
|
||||||
|
self,
|
||||||
|
monkeypatch,
|
||||||
|
env_value,
|
||||||
|
expected,
|
||||||
|
):
|
||||||
|
if env_value is None:
|
||||||
|
monkeypatch.delenv("PAPERLESS_AI_LLM_EXTRA_PARAMS", raising=False)
|
||||||
|
else:
|
||||||
|
monkeypatch.setenv("PAPERLESS_AI_LLM_EXTRA_PARAMS", env_value)
|
||||||
|
assert _get_llm_extra_params() == expected
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("env_value", "match"),
|
||||||
|
[
|
||||||
|
pytest.param("reasoning_effort=none", "valid JSON", id="invalid-json"),
|
||||||
|
pytest.param('["none"]', "JSON object", id="not-an-object"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_invalid_raises(
|
||||||
|
self,
|
||||||
|
monkeypatch,
|
||||||
|
env_value,
|
||||||
|
match,
|
||||||
|
):
|
||||||
|
monkeypatch.setenv("PAPERLESS_AI_LLM_EXTRA_PARAMS", env_value)
|
||||||
|
with pytest.raises(ImproperlyConfigured, match=match):
|
||||||
|
_get_llm_extra_params()
|
||||||
@@ -1,6 +1,6 @@
|
|||||||
from typing import Final
|
from typing import Final
|
||||||
|
|
||||||
__version__: Final[tuple[int, int, int]] = (3, 3, 0)
|
__version__: Final[tuple[int, int, int]] = (3, 2, 1)
|
||||||
# Version string like X.Y.Z
|
# Version string like X.Y.Z
|
||||||
__full_version_str__: Final[str] = ".".join(map(str, __version__))
|
__full_version_str__: Final[str] = ".".join(map(str, __version__))
|
||||||
# Version string like X.Y
|
# Version string like X.Y
|
||||||
|
|||||||
@@ -6,7 +6,6 @@ from django.contrib.auth.models import User
|
|||||||
from documents.models import Document
|
from documents.models import Document
|
||||||
from documents.permissions import permitted_document_ids
|
from documents.permissions import permitted_document_ids
|
||||||
from documents.permissions import user_is_unrestricted
|
from documents.permissions import user_is_unrestricted
|
||||||
from documents.versioning import annotate_effective_content
|
|
||||||
from paperless.config import AIConfig
|
from paperless.config import AIConfig
|
||||||
from paperless_ai.base_model import ClassificationSuggestions
|
from paperless_ai.base_model import ClassificationSuggestions
|
||||||
from paperless_ai.base_model import TaxonomyChoiceDict
|
from paperless_ai.base_model import TaxonomyChoiceDict
|
||||||
@@ -112,7 +111,7 @@ def build_prompt_without_rag(
|
|||||||
) -> str:
|
) -> str:
|
||||||
filename = document.filename or ""
|
filename = document.filename or ""
|
||||||
content = truncate_content(
|
content = truncate_content(
|
||||||
(document.get_effective_content() or "")[:4000],
|
document.content[:4000] or "",
|
||||||
chunk_size=config.llm_embedding_chunk_size,
|
chunk_size=config.llm_embedding_chunk_size,
|
||||||
context_size=config.llm_context_size,
|
context_size=config.llm_context_size,
|
||||||
)
|
)
|
||||||
@@ -226,9 +225,7 @@ def get_taxonomy_context(
|
|||||||
|
|
||||||
# similar_documents is already ordered by descending weight; don't lose it.
|
# similar_documents is already ordered by descending weight; don't lose it.
|
||||||
similar_document_ids = [s["document_id"] for s in similar_documents]
|
similar_document_ids = [s["document_id"] for s in similar_documents]
|
||||||
similar_documents_by_id = annotate_effective_content(
|
similar_documents_by_id = Document.objects.in_bulk(similar_document_ids)
|
||||||
Document.objects.all(),
|
|
||||||
).in_bulk(similar_document_ids)
|
|
||||||
similar_docs = [
|
similar_docs = [
|
||||||
similar_documents_by_id[document_id]
|
similar_documents_by_id[document_id]
|
||||||
for document_id in similar_document_ids
|
for document_id in similar_document_ids
|
||||||
@@ -236,7 +233,7 @@ def get_taxonomy_context(
|
|||||||
][:max_docs]
|
][:max_docs]
|
||||||
context_blocks = []
|
context_blocks = []
|
||||||
for similar in similar_docs:
|
for similar in similar_docs:
|
||||||
text = (similar.get_effective_content() or "")[:1000]
|
text = similar.content[:1000] or ""
|
||||||
title = similar.title or similar.filename or "Untitled"
|
title = similar.title or similar.filename or "Untitled"
|
||||||
context_blocks.append(f"TITLE: {title}\n{text}")
|
context_blocks.append(f"TITLE: {title}\n{text}")
|
||||||
except Exception:
|
except Exception:
|
||||||
|
|||||||
@@ -135,6 +135,6 @@ def build_llm_index_text(doc: Document) -> str:
|
|||||||
lines.append(f"Custom Field - {instance.field.name}: {instance}")
|
lines.append(f"Custom Field - {instance.field.name}: {instance}")
|
||||||
|
|
||||||
lines.append("\nContent:\n")
|
lines.append("\nContent:\n")
|
||||||
lines.append(doc.get_effective_content() or "")
|
lines.append(doc.content or "")
|
||||||
|
|
||||||
return _normalize_llm_index_text("\n".join(lines))
|
return _normalize_llm_index_text("\n".join(lines))
|
||||||
@@ -17,8 +17,6 @@ from documents.models import PaperlessTask
|
|||||||
from documents.utils import IterWrapper
|
from documents.utils import IterWrapper
|
||||||
from documents.utils import QuerySetStream
|
from documents.utils import QuerySetStream
|
||||||
from documents.utils import identity
|
from documents.utils import identity
|
||||||
from documents.versioning import annotate_effective_content
|
|
||||||
from documents.versioning import root_document_ids
|
|
||||||
from paperless.config import AIConfig
|
from paperless.config import AIConfig
|
||||||
from paperless_ai.db import db_connection_released
|
from paperless_ai.db import db_connection_released
|
||||||
from paperless_ai.embedding import build_llm_index_text
|
from paperless_ai.embedding import build_llm_index_text
|
||||||
@@ -445,11 +443,11 @@ def update_llm_index(
|
|||||||
"Skipping LLM index update: migration check deferred; "
|
"Skipping LLM index update: migration check deferred; "
|
||||||
"will retry next run."
|
"will retry next run."
|
||||||
)
|
)
|
||||||
documents = annotate_effective_content(
|
documents = Document.objects.select_related(
|
||||||
Document.objects.filter(root_document__isnull=True)
|
"correspondent",
|
||||||
.select_related("correspondent", "document_type", "storage_path")
|
"document_type",
|
||||||
.prefetch_related("tags", "notes", "custom_fields__field"),
|
"storage_path",
|
||||||
)
|
).prefetch_related("tags", "notes", "custom_fields__field")
|
||||||
no_documents = not documents.exists()
|
no_documents = not documents.exists()
|
||||||
|
|
||||||
# Fast exit before touching config: nothing to index and no existing index.
|
# Fast exit before touching config: nothing to index and no existing index.
|
||||||
@@ -485,7 +483,7 @@ def update_llm_index(
|
|||||||
msg = "LLM index rebuilt successfully."
|
msg = "LLM index rebuilt successfully."
|
||||||
else:
|
else:
|
||||||
scoped_documents = (
|
scoped_documents = (
|
||||||
documents.filter(id__in=root_document_ids(document_ids))
|
documents.filter(id__in=document_ids)
|
||||||
if document_ids is not None
|
if document_ids is not None
|
||||||
else documents
|
else documents
|
||||||
)
|
)
|
||||||
@@ -512,12 +510,7 @@ def update_llm_index(
|
|||||||
|
|
||||||
|
|
||||||
def llm_index_add_or_update_document(document: Document):
|
def llm_index_add_or_update_document(document: Document):
|
||||||
"""
|
"""Add or atomically replace a document's chunks in the index."""
|
||||||
Add or atomically replace a document's chunks in the index. Only root
|
|
||||||
documents are indexed, so a version is indexed as its root document.
|
|
||||||
"""
|
|
||||||
if document.root_document_id is not None:
|
|
||||||
document = document.root_document
|
|
||||||
config = AIConfig()
|
config = AIConfig()
|
||||||
new_nodes = build_document_node(
|
new_nodes = build_document_node(
|
||||||
document,
|
document,
|
||||||
@@ -695,7 +688,7 @@ def retrieve_similar_nodes(
|
|||||||
)
|
)
|
||||||
|
|
||||||
query_text = truncate_embedding_query(
|
query_text = truncate_embedding_query(
|
||||||
(document.title or "") + "\n" + (document.get_effective_content() or ""),
|
(document.title or "") + "\n" + (document.content or ""),
|
||||||
chunk_size=config.llm_embedding_chunk_size,
|
chunk_size=config.llm_embedding_chunk_size,
|
||||||
)
|
)
|
||||||
# Hold the shared read lock for the whole retrieval so the connection is
|
# Hold the shared read lock for the whole retrieval so the connection is
|
||||||
|
|||||||
@@ -55,7 +55,6 @@ def mock_document():
|
|||||||
doc.storage_path = None
|
doc.storage_path = None
|
||||||
doc.archive_serial_number = "12345"
|
doc.archive_serial_number = "12345"
|
||||||
doc.content = "This is the document content."
|
doc.content = "This is the document content."
|
||||||
doc.get_effective_content.return_value = "This is the document content."
|
|
||||||
|
|
||||||
cf1 = MagicMock(__str__=lambda x: "Value1")
|
cf1 = MagicMock(__str__=lambda x: "Value1")
|
||||||
cf1.field = MagicMock()
|
cf1.field = MagicMock()
|
||||||
@@ -435,55 +434,6 @@ def test_get_taxonomy_context_preserves_similarity_order_and_distinct_documents(
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.django_db
|
|
||||||
class TestClassifierEffectiveContent:
|
|
||||||
"""A root document's text for the LLM is its newest version's content."""
|
|
||||||
|
|
||||||
@staticmethod
|
|
||||||
def _root_with_version() -> Document:
|
|
||||||
root = DocumentFactory(title="Statement", content="stale text")
|
|
||||||
DocumentFactory(root_document=root, version_index=1, content="latest text")
|
|
||||||
return root
|
|
||||||
|
|
||||||
def test_prompt_uses_the_newest_versions_content(self) -> None:
|
|
||||||
"""
|
|
||||||
GIVEN:
|
|
||||||
- A root document with a version
|
|
||||||
WHEN:
|
|
||||||
- The classification prompt is built for the root
|
|
||||||
THEN:
|
|
||||||
- It contains the newest version's content
|
|
||||||
"""
|
|
||||||
prompt = build_prompt_without_rag(self._root_with_version(), AIConfig())
|
|
||||||
|
|
||||||
assert "latest text" in prompt
|
|
||||||
assert "stale text" not in prompt
|
|
||||||
|
|
||||||
@override_settings(LLM_EMBEDDING_BACKEND="huggingface")
|
|
||||||
def test_similar_document_context_uses_the_newest_versions_content(self) -> None:
|
|
||||||
"""
|
|
||||||
GIVEN:
|
|
||||||
- A similar root document with a version
|
|
||||||
WHEN:
|
|
||||||
- The similar-document context is built
|
|
||||||
THEN:
|
|
||||||
- It contains the newest version's content
|
|
||||||
"""
|
|
||||||
similar = self._root_with_version()
|
|
||||||
document = DocumentFactory(content="Some content")
|
|
||||||
fake_nodes = [
|
|
||||||
SimpleNamespace(metadata={"document_id": str(similar.pk)}, score=0.9),
|
|
||||||
]
|
|
||||||
|
|
||||||
with patch(
|
|
||||||
"paperless_ai.ai_classifier.retrieve_similar_nodes",
|
|
||||||
return_value=fake_nodes,
|
|
||||||
):
|
|
||||||
_candidates, context = get_taxonomy_context(document, user=None)
|
|
||||||
|
|
||||||
assert context == "TITLE: Statement\nlatest text"
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.django_db
|
@pytest.mark.django_db
|
||||||
@override_settings(LLM_EMBEDDING_BACKEND="huggingface")
|
@override_settings(LLM_EMBEDDING_BACKEND="huggingface")
|
||||||
def test_get_taxonomy_context_no_similar_docs():
|
def test_get_taxonomy_context_no_similar_docs():
|
||||||
|
|||||||
@@ -388,108 +388,6 @@ def test_update_llm_index_partial_update(
|
|||||||
assert after[str(doc2.pk)] == before[str(doc2.pk)]
|
assert after[str(doc2.pk)] == before[str(doc2.pk)]
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.django_db
|
|
||||||
class TestLlmIndexVersions:
|
|
||||||
"""The LLM index holds root documents only: a version is indexed as its root."""
|
|
||||||
|
|
||||||
def test_add_or_update_document_indexes_a_version_as_its_root(
|
|
||||||
self,
|
|
||||||
temp_llm_index_dir: Path,
|
|
||||||
mock_embed_model: FakeEmbedding,
|
|
||||||
) -> None:
|
|
||||||
"""
|
|
||||||
GIVEN:
|
|
||||||
- A root document with a version
|
|
||||||
WHEN:
|
|
||||||
- The version is passed to llm_index_add_or_update_document
|
|
||||||
THEN:
|
|
||||||
- Only the root document is in the index
|
|
||||||
"""
|
|
||||||
root = DocumentFactory(content="root content")
|
|
||||||
version = DocumentFactory(root_document=root, version_index=1)
|
|
||||||
|
|
||||||
indexing.llm_index_add_or_update_document(version)
|
|
||||||
|
|
||||||
with indexing.get_vector_store() as store:
|
|
||||||
indexed = store.get_modified_times()
|
|
||||||
|
|
||||||
assert set(indexed) == {str(root.pk)}
|
|
||||||
|
|
||||||
def test_rebuild_skips_versions(
|
|
||||||
self,
|
|
||||||
temp_llm_index_dir: Path,
|
|
||||||
mock_embed_model: FakeEmbedding,
|
|
||||||
) -> None:
|
|
||||||
"""
|
|
||||||
GIVEN:
|
|
||||||
- A root document with a version
|
|
||||||
WHEN:
|
|
||||||
- The LLM index is rebuilt
|
|
||||||
THEN:
|
|
||||||
- Only the root document is in the index
|
|
||||||
"""
|
|
||||||
root = DocumentFactory()
|
|
||||||
DocumentFactory(root_document=root, version_index=1)
|
|
||||||
|
|
||||||
indexing.update_llm_index(rebuild=True)
|
|
||||||
|
|
||||||
with indexing.get_vector_store() as store:
|
|
||||||
indexed = store.get_modified_times()
|
|
||||||
|
|
||||||
assert set(indexed) == {str(root.pk)}
|
|
||||||
|
|
||||||
def test_rebuild_indexes_the_newest_versions_content(
|
|
||||||
self,
|
|
||||||
temp_llm_index_dir: Path,
|
|
||||||
mock_embed_model: FakeEmbedding,
|
|
||||||
mocker: pytest_mock.MockerFixture,
|
|
||||||
) -> None:
|
|
||||||
"""
|
|
||||||
GIVEN:
|
|
||||||
- A root document with a version
|
|
||||||
WHEN:
|
|
||||||
- The LLM index is rebuilt
|
|
||||||
THEN:
|
|
||||||
- The root's text for the index is the version's content, answered
|
|
||||||
from the query rather than a query per document
|
|
||||||
"""
|
|
||||||
root = DocumentFactory(content="stale text")
|
|
||||||
DocumentFactory(root_document=root, version_index=1, content="latest text")
|
|
||||||
spy = mocker.spy(indexing, "build_document_node")
|
|
||||||
|
|
||||||
indexing.update_llm_index(rebuild=True)
|
|
||||||
|
|
||||||
indexed = spy.call_args.args[0]
|
|
||||||
assert indexed.pk == root.pk
|
|
||||||
assert indexed.effective_content == "latest text"
|
|
||||||
|
|
||||||
def test_incremental_update_by_version_id_refreshes_the_root(
|
|
||||||
self,
|
|
||||||
temp_llm_index_dir: Path,
|
|
||||||
mock_embed_model: FakeEmbedding,
|
|
||||||
) -> None:
|
|
||||||
"""
|
|
||||||
GIVEN:
|
|
||||||
- An indexed root document with a version whose root was modified since
|
|
||||||
WHEN:
|
|
||||||
- An incremental update is scoped to the version's id
|
|
||||||
THEN:
|
|
||||||
- The root's entry is refreshed and no entry exists for the version
|
|
||||||
"""
|
|
||||||
root = DocumentFactory()
|
|
||||||
version = DocumentFactory(root_document=root, version_index=1)
|
|
||||||
indexing.update_llm_index(rebuild=True)
|
|
||||||
Document.objects.filter(pk=root.pk).update(modified=timezone.now())
|
|
||||||
root.refresh_from_db()
|
|
||||||
|
|
||||||
indexing.update_llm_index(document_ids=[version.pk])
|
|
||||||
|
|
||||||
with indexing.get_vector_store() as store:
|
|
||||||
indexed = store.get_modified_times()
|
|
||||||
|
|
||||||
assert indexed == {str(root.pk): root.modified.isoformat()}
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.django_db
|
@pytest.mark.django_db
|
||||||
def test_add_or_update_document_updates_existing_entry(
|
def test_add_or_update_document_updates_existing_entry(
|
||||||
temp_llm_index_dir: Path,
|
temp_llm_index_dir: Path,
|
||||||
@@ -626,39 +524,6 @@ def test_retrieve_similar_nodes_truncates_query_to_embedding_chunk_size(
|
|||||||
assert "word199" not in query_text
|
assert "word199" not in query_text
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.django_db
|
|
||||||
@override_settings(
|
|
||||||
LLM_EMBEDDING_BACKEND="huggingface",
|
|
||||||
LLM_BACKEND="ollama",
|
|
||||||
)
|
|
||||||
def test_retrieve_similar_nodes_queries_with_newest_versions_content(
|
|
||||||
temp_llm_index_dir: Path,
|
|
||||||
) -> None:
|
|
||||||
"""
|
|
||||||
GIVEN:
|
|
||||||
- A root document whose newest version carries different content
|
|
||||||
WHEN:
|
|
||||||
- Documents similar to the root are retrieved
|
|
||||||
THEN:
|
|
||||||
- The query text holds the newest version's content, not the root's own
|
|
||||||
"""
|
|
||||||
root = DocumentFactory(title="Statement", content="stale root text")
|
|
||||||
DocumentFactory(root_document=root, version_index=1, content="latest version text")
|
|
||||||
|
|
||||||
with (
|
|
||||||
patch("paperless_ai.indexing.load_or_build_index"),
|
|
||||||
patch("paperless_ai.indexing.llm_index_exists", return_value=True),
|
|
||||||
patch("llama_index.core.retrievers.VectorIndexRetriever") as mock_retriever_cls,
|
|
||||||
):
|
|
||||||
mock_retriever_cls.return_value.retrieve.return_value = []
|
|
||||||
|
|
||||||
indexing.retrieve_similar_nodes(root, top_k=3)
|
|
||||||
|
|
||||||
query_text = mock_retriever_cls.return_value.retrieve.call_args.args[0]
|
|
||||||
assert "latest version text" in query_text
|
|
||||||
assert "stale root text" not in query_text
|
|
||||||
|
|
||||||
|
|
||||||
class TestUpdateLlmIndexEmptyDocumentSet:
|
class TestUpdateLlmIndexEmptyDocumentSet:
|
||||||
"""update_llm_index must clear the vector store table when all documents are deleted.
|
"""update_llm_index must clear the vector store table when all documents are deleted.
|
||||||
|
|
||||||
@@ -772,7 +637,6 @@ class TestLlmIndexAddOrUpdateDocumentEmptyContent:
|
|||||||
|
|
||||||
doc = MagicMock(spec=Document)
|
doc = MagicMock(spec=Document)
|
||||||
doc.id = 42
|
doc.id = 42
|
||||||
doc.root_document_id = None
|
|
||||||
# Must not raise
|
# Must not raise
|
||||||
indexing.llm_index_add_or_update_document(doc)
|
indexing.llm_index_add_or_update_document(doc)
|
||||||
|
|
||||||
|
|||||||
@@ -12,7 +12,6 @@ from paperless_ai.embedding import _normalize_llm_index_text
|
|||||||
from paperless_ai.embedding import build_llm_index_text
|
from paperless_ai.embedding import build_llm_index_text
|
||||||
from paperless_ai.embedding import get_configured_model_name
|
from paperless_ai.embedding import get_configured_model_name
|
||||||
from paperless_ai.embedding import get_embedding_model
|
from paperless_ai.embedding import get_embedding_model
|
||||||
from paperless_testing.factories import DocumentFactory
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture
|
@pytest.fixture
|
||||||
@@ -47,7 +46,6 @@ def mock_document():
|
|||||||
doc.correspondent.name = "Test Correspondent"
|
doc.correspondent.name = "Test Correspondent"
|
||||||
doc.archive_serial_number = "12345"
|
doc.archive_serial_number = "12345"
|
||||||
doc.content = "This is the document content."
|
doc.content = "This is the document content."
|
||||||
doc.get_effective_content.return_value = "This is the document content."
|
|
||||||
|
|
||||||
cf1 = MagicMock(__str__=lambda x: "Value1")
|
cf1 = MagicMock(__str__=lambda x: "Value1")
|
||||||
cf1.field = MagicMock()
|
cf1.field = MagicMock()
|
||||||
@@ -282,7 +280,7 @@ def test_build_llm_index_text(mock_document):
|
|||||||
|
|
||||||
|
|
||||||
def test_build_llm_index_text_normalizes_ocr_punctuation_runs(mock_document):
|
def test_build_llm_index_text_normalizes_ocr_punctuation_runs(mock_document):
|
||||||
mock_document.get_effective_content.return_value = (
|
mock_document.content = (
|
||||||
"Introduction ................................................ 7\n"
|
"Introduction ................................................ 7\n"
|
||||||
"Hardware Limitation ________________________________________ 9\n"
|
"Hardware Limitation ________________________________________ 9\n"
|
||||||
"Keep short punctuation like INV-100 and ellipses..."
|
"Keep short punctuation like INV-100 and ellipses..."
|
||||||
@@ -296,43 +294,6 @@ def test_build_llm_index_text_normalizes_ocr_punctuation_runs(mock_document):
|
|||||||
assert "ellipses..." in result
|
assert "ellipses..." in result
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.django_db
|
|
||||||
class TestBuildLlmIndexTextVersions:
|
|
||||||
"""A root document is indexed with its effective content, like in the search index."""
|
|
||||||
|
|
||||||
def test_root_uses_the_newest_versions_content(self) -> None:
|
|
||||||
"""
|
|
||||||
GIVEN:
|
|
||||||
- A root document with two versions
|
|
||||||
WHEN:
|
|
||||||
- The LLM index text is built for the root
|
|
||||||
THEN:
|
|
||||||
- It contains the newest version's content and not the others'
|
|
||||||
"""
|
|
||||||
root = DocumentFactory(content="stale text")
|
|
||||||
DocumentFactory(root_document=root, version_index=1, content="older text")
|
|
||||||
DocumentFactory(root_document=root, version_index=2, content="latest text")
|
|
||||||
|
|
||||||
text = build_llm_index_text(root)
|
|
||||||
|
|
||||||
assert "latest text" in text
|
|
||||||
assert "stale text" not in text
|
|
||||||
assert "older text" not in text
|
|
||||||
|
|
||||||
def test_root_without_versions_uses_its_own_content(self) -> None:
|
|
||||||
"""
|
|
||||||
GIVEN:
|
|
||||||
- A root document without versions
|
|
||||||
WHEN:
|
|
||||||
- The LLM index text is built for it
|
|
||||||
THEN:
|
|
||||||
- It contains the document's own content
|
|
||||||
"""
|
|
||||||
root = DocumentFactory(content="own text")
|
|
||||||
|
|
||||||
assert "own text" in build_llm_index_text(root)
|
|
||||||
|
|
||||||
|
|
||||||
def test_normalize_llm_index_text_collapses_ocr_leaders_without_joining_lines():
|
def test_normalize_llm_index_text_collapses_ocr_leaders_without_joining_lines():
|
||||||
assert _normalize_llm_index_text("A........B\nC____D----E") == "A B\nC D E"
|
assert _normalize_llm_index_text("A........B\nC____D----E") == "A B\nC D E"
|
||||||
|
|
||||||
|
|||||||
+999
-4
File diff suppressed because it is too large.
Load diff
@@ -1,5 +1,5 @@
|
|||||||
{
|
{
|
||||||
"devDependencies": {
|
"devDependencies": {
|
||||||
"tailwindcss": "^4.3.3"
|
"tailwindcss": "^3.4.19"
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -2971,7 +2971,7 @@ wheels = [
|
|||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "paperless-ngx"
|
name = "paperless-ngx"
|
||||||
version = "3.3.0"
|
version = "3.2.1"
|
||||||
source = { virtual = "." }
|
source = { virtual = "." }
|
||||||
dependencies = [
|
dependencies = [
|
||||||
{ name = "azure-ai-documentintelligence" },
|
{ name = "azure-ai-documentintelligence" },
|
||||||
|
|||||||
Reference in new issue
Block a user