Compare commits

...
Author SHA1 Message Date
stumpylog ca42c27455 fix(search): cap the advanced search query length 2026-08-28 09:48:14 -07:00
stumpylog a626ad4627 style(search): remove em dashes from PR2 comments/docstrings 2026-08-28 09:47:14 -07:00
stumpylog 9029b27aeb feat(search): parse advanced search with whoosh-compat and delete the hand-written translator 2026-08-28 09:47:14 -07:00
stumpylog 03e05e0804 test(search): move schema fingerprint/version tests from PR2
Both files exercise documents.search._schema exclusively (build_schema,
schema_fingerprint, needs_rebuild, open_or_rebuild_index) with no
dependency on whoosh-compat query routing, so they belong with the
schema/field-registry work rather than PR2's query rewrite.
2026-08-28 09:46:57 -07:00
stumpylog 2f79549f92 style(search): remove em dashes from PR1 comments/docstrings 2026-08-28 09:35:24 -07:00
stumpylog 9e9f3f4e09 refactor(search): dispatch table for _public_field_descriptors
Collapse the TEXT/KEYWORD/U64/DATE/DATETIME if/elif chain into a
FieldKind -> (schema kind, tokenizer) lookup table. JSON stays an
explicit branch since it can emit a second, synthetic notes_text
descriptor.
2026-08-28 09:30:06 -07:00
stumpylog d0869bec01 feat(search): add whoosh-compat, the shared field table and the field registry 2026-08-28 08:42:14 -07:00
shamoon 01c40aa0bd Tweak: subtler shadow, tweak against light color 2026-08-24 08:51:51 -07:00
shamoon 5f70b6eee7 Fix navbar button alignment + hover consistency 2026-08-24 08:47:27 -07:00
GitHub Actions e3e4b26944 Auto translate strings 2026-08-24 15:16:51 +00:00
a0908f6b4a Enhancement: websocket heartbeat (#13739)
---------

Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
2026-08-24 15:15:21 +00:00
bab9129ff8 Fix: lazy import guardian modules to fix search language setting (#13768)
Co-authored-by: Trenton H <797416+stumpylog@users.noreply.github.com>
2026-08-24 13:46:20 +00:00
GitHub Actions 1e13f86174 Auto translate strings 2026-08-24 09:13:06 +00:00
shamoonandGitHub 22c57fb392 Enhancement: more v3 ui tweaks (#13774) 2026-08-24 02:11:34 -07:00
GitHub Actions 08a7f6ccc0 Auto translate strings 2026-08-24 04:56:30 +00:00
shamoonandGitHub 3f4d327f64 Chore: add some missing UI accessibility labels (#13772) 2026-08-23 21:55:07 -07:00
GitHub Actions bd326540fa Auto translate strings 2026-08-24 02:43:34 +00:00
shamoonandGitHub 33f9adb05a Chore: refactor permission checkbox live changes (#13771) 2026-08-23 19:42:18 -07:00
shamoonandGitHub d24606a03b Tweak: better support long list of views in documents list (#13769) 2026-08-23 16:40:55 -07:00
shamoonandGitHub 294328f174 Fix: version indexing fixes (#13737) 2026-08-23 23:04:47 +00:00
shamoonandGitHub 0458bad5f2 Fix: append charset to file response for text files (#13759) 2026-08-22 06:15:53 -07:00
shamoon a510d03c77 Merge branch 'main' into dev 2026-08-22 05:27:10 -07:00
shamoonandGitHub d7b3612a41 Chore: pin Apache Tika images to 3.3.1 (#13758) 2026-08-22 05:25:37 -07:00
shamoonandGitHub 7e4a644714 Fix: align bulk edit perms with document model (#13757) 2026-08-22 05:24:16 -07:00
GitHub Actions bbcd6af2fe Auto translate strings 2026-08-21 15:25:24 +00:00
shamoonandGitHub 0431939f18 QoL: add count badge to versions dropdown (#13753) 2026-08-21 08:23:47 -07:00
shamoonandGitHub bed95ea301 Tweakhancement: add jitter to IMAP polling schedule (#13734) 2026-08-20 17:19:47 +00:00
GitHub Actions 42034c3c77 Auto translate strings 2026-08-20 04:52:42 +00:00
shamoonandGitHub 705220fb5a Fix: use selected version for doc detail emailing (#13738) 2026-08-19 21:51:13 -07:00
GitHub Actions a424dace43 Auto translate strings 2026-08-19 18:20:04 +00:00
shamoonandGitHub 751299895e Fix: hide version delete button without global perms (#13735) 2026-08-19 11:17:19 -07:00
Trenton HandGitHub 5f9bc5de88 Chore: Upgrade Docker image to Python 3.14 (#13721)
* Upgrades our base image to uv 0.12 branch and Python 3.14

* Upgrades our workflows to uv 0.12.x as well

* Updates these locked wheels too
2026-08-19 09:43:14 -07:00
f1c8a72f26 Enhancement: sync OIDC groups to superuser and staff roles (#13060)
Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
Co-authored-by: SoleroTG <github-29h@solero.quietmail.eu>
Co-authored-by: stumpylog <797416+stumpylog@users.noreply.github.com>
2026-08-19 15:27:56 +00:00
GitHub Actions fd3c525f03 Auto translate strings 2026-08-19 14:24:36 +00:00
shamoonandGitHub e389298aab Enhancement: merge documents as versions (#13515) 2026-08-19 07:20:14 -07:00
shamoonandGitHub c5c5cc0b1d Tweak: adjust modal proportions for small screens (#13728) 2026-08-18 19:34:18 -07:00
b17a512539 Refactor: render paperless_ai prompts via Jinja2 templates instead of f-strings (#13698)
* Refactor: render paperless_ai prompts via Jinja2 templates instead of f-strings

* Apply suggestions from code review

Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
2026-08-18 18:32:21 +00:00
shamoon 4cf027de40 Tweak: space out the side menu sub-nav a bit 2026-08-18 09:44:03 -07:00
shamoon 41953c7846 Chore: harden ImageMagick policy 2026-08-18 09:38:54 -07:00
dependabot[bot]GitHubdependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
e5abe5cf32 Chore(deps): Bump the uv group across 1 directory with 2 updates (#13709)
Bumps the uv group with 2 updates in the / directory: [pymdown-extensions](https://github.com/facelessuser/pymdown-extensions) and [sqlparse](https://github.com/andialbrecht/sqlparse).


Updates `pymdown-extensions` from 11.0 to 11.0.1
- [Release notes](https://github.com/facelessuser/pymdown-extensions/releases)
- [Commits](https://github.com/facelessuser/pymdown-extensions/compare/11.0...11.0.1)

Updates `sqlparse` from 0.5.5 to 0.6.0
- [Changelog](https://github.com/andialbrecht/sqlparse/blob/master/CHANGELOG)
- [Commits](https://github.com/andialbrecht/sqlparse/compare/0.5.5...0.6.0)

---
updated-dependencies:
- dependency-name: pymdown-extensions
  dependency-version: 11.0.1
  dependency-type: indirect
- dependency-name: sqlparse
  dependency-version: 0.6.0
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-18 15:11:24 +00:00
shamoonandGitHub 2d084983c8 Documentation: clarify OCR mode changes in v3 (#13666) 2026-08-12 21:46:52 -07:00
shamoonandGitHub e2c284f64e Documentation: add wiki links for AI stuff and parser plugins (#13626) 2026-08-09 19:05:32 -07:00
169 changed files with 10037 additions and 4215 deletions
@@ -68,7 +68,7 @@ services:
- "--chromium-disable-javascript=true"
- "--chromium-allow-list=file:///tmp/.*"
tika:
image: docker.io/apache/tika:latest
image: docker.io/apache/tika:3.3.1.0
restart: unless-stopped
volumes:
data:
+1 -1
View File
@@ -11,7 +11,7 @@ concurrency:
group: backend-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
env:
DEFAULT_UV_VERSION: "0.11.x"
DEFAULT_UV_VERSION: "0.12.x"
NLTK_DATA: "/usr/share/nltk_data"
permissions: {}
jobs:
+1 -1
View File
@@ -11,7 +11,7 @@ concurrency:
permissions:
contents: read
env:
DEFAULT_UV_VERSION: "0.11.x"
DEFAULT_UV_VERSION: "0.12.x"
DEFAULT_PYTHON_VERSION: "3.12"
jobs:
changes:
+1 -1
View File
@@ -8,7 +8,7 @@ concurrency:
group: release-${{ github.ref }}
cancel-in-progress: false
env:
DEFAULT_UV_VERSION: "0.11.x"
DEFAULT_UV_VERSION: "0.12.x"
DEFAULT_PYTHON_VERSION: "3.12"
permissions: {}
jobs:
+1 -1
View File
@@ -4,7 +4,7 @@ on:
branches:
- dev
env:
DEFAULT_UV_VERSION: "0.11.x"
DEFAULT_UV_VERSION: "0.12.x"
jobs:
generate-translate-strings:
name: Generate Translation Strings
+1 -1
View File
@@ -30,7 +30,7 @@ RUN set -eux \
# Purpose: Installs s6-overlay and rootfs
# Comments:
# - Don't leave anything extra in here either
FROM ghcr.io/astral-sh/uv:0.11.32-python3.12-trixie-slim AS s6-overlay-base
FROM ghcr.io/astral-sh/uv:0.12.5-python3.14-trixie-slim AS s6-overlay-base
WORKDIR /usr/src/s6
@@ -81,7 +81,7 @@ services:
- "--chromium-disable-javascript=true"
- "--chromium-allow-list=file:///tmp/.*"
tika:
image: docker.io/apache/tika:latest
image: docker.io/apache/tika:3.3.1.0
restart: unless-stopped
volumes:
data:
@@ -76,7 +76,7 @@ services:
- "--chromium-disable-javascript=true"
- "--chromium-allow-list=file:///tmp/.*"
tika:
image: docker.io/apache/tika:latest
image: docker.io/apache/tika:3.3.1.0
restart: unless-stopped
volumes:
data:
@@ -65,7 +65,7 @@ services:
- "--chromium-disable-javascript=true"
- "--chromium-allow-list=file:///tmp/.*"
tika:
image: docker.io/apache/tika:latest
image: docker.io/apache/tika:3.3.1.0
restart: unless-stopped
volumes:
data:
@@ -68,7 +68,14 @@
<!-- <policy domain="resource" name="thread" value="4"/> -->
<!-- <policy domain="resource" name="throttle" value="0"/> -->
<!-- <policy domain="resource" name="time" value="3600"/> -->
<!-- <policy domain="coder" rights="none" pattern="MVG" /> -->
<!-- Paperless does not process SVG or ImageMagick scripting formats. -->
<policy domain="coder" rights="none" pattern="SVG" />
<policy domain="coder" rights="none" pattern="SVGZ" />
<policy domain="coder" rights="none" pattern="MSVG" />
<policy domain="coder" rights="none" pattern="RSVG" />
<policy domain="coder" rights="none" pattern="MSL" />
<policy domain="coder" rights="none" pattern="MVG" />
<policy domain="coder" rights="none" pattern="EPHEMERAL" />
<!-- <policy domain="module" rights="none" pattern="{PS,PDF,XPS}" /> -->
<!-- <policy domain="delegate" rights="none" pattern="HTTPS" /> -->
<!-- <policy domain="path" rights="none" pattern="@*" /> -->
@@ -78,8 +85,6 @@
<!-- <policy domain="system" name="pixel-cache-memory" value="anonymous"/> -->
<!-- <policy domain="system" name="shred" value="2"/> -->
<!-- <policy domain="system" name="precision" value="6"/> -->
<!-- not needed due to the need to use explicitly by mvg: -->
<!-- <policy domain="delegate" rights="none" pattern="MVG" /> -->
<!-- use curl -->
<policy domain="delegate" rights="none" pattern="URL" />
<policy domain="delegate" rights="none" pattern="HTTPS" />
+6 -1
View File
@@ -129,6 +129,10 @@ At a minimum you need to enable AI and choose an LLM backend:
and/or [`PAPERLESS_AI_LLM_ENDPOINT`](configuration.md#PAPERLESS_AI_LLM_ENDPOINT). Ollama
requires `PAPERLESS_AI_LLM_ENDPOINT` pointing at your Ollama server.
See the community-maintained wiki page on
[choosing AI models](https://github.com/paperless-ngx/paperless-ngx/wiki/AI-Model-Recommendations)
for suggested generation and embedding models.
### AI-assisted suggestions
With AI enabled, Paperless-ngx can suggest a title, tags, correspondent, document type,
@@ -808,7 +812,8 @@ Third-party parser plugins extend Paperless-ngx to support additional file
formats. A plugin is a Python package that advertises itself under the
`paperless_ngx.parsers` entry point group. Refer to the
[developer documentation](development.md#making-custom-parsers) for how to
create one.
create one, or see the wiki for a community-maintained list of
[parser plugins](https://github.com/paperless-ngx/paperless-ngx/wiki/Related-Projects#parser-plugins).
!!! warning "Third-party plugins are not officially supported"
+1
View File
@@ -227,6 +227,7 @@ Version-aware endpoints:
- `PATCH /api/documents/{id}/`: content updates target the selected version (`?version={version_id}`) or latest version by default; non-content metadata updates target the root document.
- `GET /api/documents/{id}/download/`, `GET /api/documents/{id}/preview/`, `GET /api/documents/{id}/thumb/`, `GET /api/documents/{id}/metadata/`: accept `?version={version_id}`.
- `POST /api/documents/{id}/update_version/`: uploads a new version using multipart form field `document` and optional `version_label`.
- `POST /api/documents/merge_as_versions/`: merges existing top-level documents as versions of a selected root. The JSON body must contain `documents` (at least two document IDs) and `root_document_id` (one of those IDs). When merging one source document, an optional `version_label` may be provided.
- `PATCH /api/documents/{id}/versions/{version_id}/`: updates the `version_label` of a specific version.
- `DELETE /api/documents/{root_id}/versions/{version_id}/`: deletes a non-root version.
+23 -1
View File
@@ -776,6 +776,24 @@ system. See the corresponding
Defaults to "groups"
#### [`PAPERLESS_SOCIAL_ACCOUNT_SYNC_SUPERUSER_GROUP=<str>`](#PAPERLESS_SOCIAL_ACCOUNT_SYNC_SUPERUSER_GROUP) {#PAPERLESS_SOCIAL_ACCOUNT_SYNC_SUPERUSER_GROUP}
: Allows you to define a group name that, if present in the third-party authentication system's groups claim, will grant the user superuser (admin) and staff status in Paperless-ngx. If the group is not present in the claim, superuser status will be revoked upon next login.
!!! warning
This is a direct reflection of the claim on every login, including the connecting user, with no exemption for the last remaining admin. If the group is missing or misconfigured on the identity provider side, the logged-in user will immediately lose their own superuser access. Fix the group membership or claim mapping on the identity provider to restore it. If the identity provider itself is unreachable or misconfigured and you are locked out, you can recover admin access locally with `manage.py createsuperuser`.
Defaults to None
#### [`PAPERLESS_SOCIAL_ACCOUNT_SYNC_STAFF_GROUP=<str>`](#PAPERLESS_SOCIAL_ACCOUNT_SYNC_STAFF_GROUP) {#PAPERLESS_SOCIAL_ACCOUNT_SYNC_STAFF_GROUP}
: Allows you to define a group name that, if present in the third-party authentication system's groups claim, will grant the user staff status in Paperless-ngx. If the group is not present in the claim and the user is not a superuser, staff status will be revoked upon next login.
!!! warning
As with [`PAPERLESS_SOCIAL_ACCOUNT_SYNC_SUPERUSER_GROUP`](#PAPERLESS_SOCIAL_ACCOUNT_SYNC_SUPERUSER_GROUP), this is applied on every login unconditionally, including for the connecting user themselves.
Defaults to None
#### [`PAPERLESS_SOCIAL_ACCOUNT_DEFAULT_GROUPS=<comma-separated-list>`](#PAPERLESS_SOCIAL_ACCOUNT_DEFAULT_GROUPS) {#PAPERLESS_SOCIAL_ACCOUNT_DEFAULT_GROUPS}
: A list of group names that users who signup via social accounts will be added to upon signup. Groups listed here must already exist.
@@ -1197,7 +1215,7 @@ should be a valid crontab(5) expression describing when to run.
: If set to the string "disable", no emails will be fetched automatically.
Defaults to `*/10 * * * *` or every ten minutes.
Defaults to every ten minutes, with an installation-specific minute offset.
#### [`PAPERLESS_TRAIN_TASK_CRON=<cron expression>`](#PAPERLESS_TRAIN_TASK_CRON) {#PAPERLESS_TRAIN_TASK_CRON}
@@ -2070,6 +2088,8 @@ suggestions. This setting is required to be set to true in order to use the AI f
models supported by the current embedding backend. If not supplied, defaults to
"text-embedding-3-small" for the OpenAI-compatible backend,
"sentence-transformers/all-MiniLM-L6-v2" for Huggingface, and "embeddinggemma" for Ollama.
See [choosing AI models](https://github.com/paperless-ngx/paperless-ngx/wiki/AI-Model-Recommendations)
for language and resource considerations.
Defaults to None.
@@ -2126,6 +2146,8 @@ setting is required to be set to use the AI features.
: The model to use for the AI backend, i.e. "gpt-3.5-turbo", "gpt-4" or any of the models supported
by the current backend. If not supplied, defaults to "gpt-3.5-turbo" for the OpenAI-compatible
backend and "llama3.1" for Ollama.
See [choosing AI models](https://github.com/paperless-ngx/paperless-ngx/wiki/AI-Model-Recommendations)
for local versus remote and model-size considerations.
Defaults to None.
+4 -3
View File
@@ -156,7 +156,7 @@ The new settings are independent:
### Database configuration
If you changed OCR settings via the admin UI (ApplicationConfiguration), the database values are **migrated automatically** during the upgrade. `mode` values (`skip` / `skip_noarchive`) are mapped to their new equivalents and `skip_archive_file` values are converted to the new `archive_file_generation` field. After upgrading, review the OCR settings in the admin UI to confirm the migrated values match your intent.
If you changed OCR settings via the admin UI (ApplicationConfiguration), the database values are **migrated automatically** during the upgrade. `mode` values (`skip` / `skip_noarchive`) are mapped to their new equivalents and explicit `skip_archive_file` values are converted to the new `archive_file_generation` field. Users who relied on the old defaults must set `archive_file_generation` to `always` to preserve the v2 behaviour of always creating an archive. After upgrading, review the OCR settings in the admin UI to confirm the migrated values match your intent.
### Action Required
@@ -165,8 +165,9 @@ Remove any `PAPERLESS_OCR_SKIP_ARCHIVE_FILE` variable from your environment. If
```bash
# v2: skip OCR when text present, always archive
PAPERLESS_OCR_MODE=skip
# v3: equivalent (auto is the new default)
# No change needed - auto is the default
# v3: equivalent
PAPERLESS_OCR_MODE=auto
PAPERLESS_ARCHIVE_FILE_GENERATION=always
# v2: skip OCR when text present, skip archive too
PAPERLESS_OCR_MODE=skip_noarchive
+8 -15
View File
@@ -416,22 +416,15 @@ to a positive number to enable polling and disable native filesystem notificatio
You may need to change the path in the files. Example:
`ExecStart=/opt/paperless/.local/bin/celery --app paperless worker --loglevel INFO`
12. Configure ImageMagick to allow processing of PDF documents. Most
distributions have this disabled by default, since PDF documents can
contain malware. If you don't do this, Paperless-ngx will fall back to
Ghostscript for certain steps such as thumbnail generation.
12. Configure ImageMagick to allow processing of PDF documents and disable
formats that Paperless-ngx does not use. Most distributions disable PDF
processing by default, since PDF documents can contain malware. If you
don't enable it, Paperless-ngx will fall back to Ghostscript for certain
steps such as thumbnail generation.
Edit `/etc/ImageMagick-6/policy.xml` and adjust
```
<policy domain="coder" rights="none" pattern="PDF" />
```
to
```
<policy domain="coder" rights="read|write" pattern="PDF" />
```
Configure the active ImageMagick policy file (commonly
`/etc/ImageMagick-6/policy.xml` or `/etc/ImageMagick-7/policy.xml`) and
adjust similar to [the docker policy file](https://raw.githubusercontent.com/paperless-ngx/paperless-ngx/refs/heads/main/docker/rootfs/etc/ImageMagick-6/paperless-policy.xml). You should also include restrictions as noted there.
**Optional: Install the [jbig2enc](https://ocrmypdf.readthedocs.io/en/latest/jbig2.html) encoder.**
This will reduce the size of generated PDF documents. You'll most likely need to compile this yourself, because this
+4
View File
@@ -99,6 +99,10 @@ Think of versions as **file history** for a document.
- By default, search and document content use the latest version.
- In document detail, selecting a version switches the preview, file metadata and content (and download etc buttons) to that version.
- Deleting a non-root version keeps metadata and falls back to the latest remaining version.
- From the document list, select two or more documents and choose **Merge as versions** to combine them under one entry. Select the root document whose metadata and permissions should be retained; the other selected documents become file versions. The root may already have versions, but documents being added as versions must not have version histories of their own.
- From a document's **Versions** menu, choose **Existing** to search for another document and add it as a version of the current document.
- Documents merged as versions give up their archive serial number. If the root has no ASN of its own it takes the first one, otherwise the ASNs are released and the removal is logged.
- Merging as versions cannot be undone from the UI, and deleting the root document moves its versions to the trash as well.
### Management Lists
+2
View File
@@ -34,6 +34,8 @@ PAPERLESS_SECRET_KEY=change-me
#PAPERLESS_AUTO_LOGIN_USERNAME=
#PAPERLESS_COOKIE_PREFIX=
#PAPERLESS_ENABLE_HTTP_REMOTE_USER=false
#PAPERLESS_SOCIAL_ACCOUNT_SYNC_SUPERUSER_GROUP=
#PAPERLESS_SOCIAL_ACCOUNT_SYNC_STAFF_GROUP=
# OCR settings
+7 -4
View File
@@ -77,6 +77,7 @@ dependencies = [
"torch~=2.13.0",
"watchfiles>=1.2",
"whitenoise~=6.11",
"whoosh-compat[tantivy]==0.1",
"zxing-cpp~=3.1.0",
]
[project.optional-dependencies]
@@ -84,9 +85,9 @@ mariadb = [
"mysqlclient~=2.2.7",
]
postgres = [
"psycopg[c,pool]==3.3",
"psycopg[c,pool]==3.3.4",
# Direct dependency for proper resolution of the pre-built wheels
"psycopg-c==3.3",
"psycopg-c==3.3.4",
"psycopg-pool==3.3.1",
]
webserver = [
@@ -160,8 +161,10 @@ explicit = true
[tool.uv.sources]
# Markers are chosen to select these almost exclusively when building the Docker image
psycopg-c = [
{ url = "https://github.com/paperless-ngx/builder/releases/download/psycopg-trixie-3.3.0/psycopg_c-3.3.0-cp312-cp312-linux_x86_64.whl", marker = "sys_platform == 'linux' and platform_machine == 'x86_64' and python_version == '3.12'" },
{ url = "https://github.com/paperless-ngx/builder/releases/download/psycopg-trixie-3.3.0/psycopg_c-3.3.0-cp312-cp312-linux_aarch64.whl", marker = "sys_platform == 'linux' and platform_machine == 'aarch64' and python_version == '3.12'" },
{ url = "https://github.com/paperless-ngx/builder/releases/download/psycopg-trixie-3.3.4/psycopg_c-3.3.4-cp312-cp312-linux_x86_64.whl", marker = "sys_platform == 'linux' and platform_machine == 'x86_64' and python_version == '3.12'" },
{ url = "https://github.com/paperless-ngx/builder/releases/download/psycopg-trixie-3.3.4/psycopg_c-3.3.4-cp312-cp312-linux_aarch64.whl", marker = "sys_platform == 'linux' and platform_machine == 'aarch64' and python_version == '3.12'" },
{ url = "https://github.com/paperless-ngx/builder/releases/download/psycopg-trixie-3.3.4/psycopg_c-3.3.4-cp314-cp314-linux_x86_64.whl", marker = "sys_platform == 'linux' and platform_machine == 'x86_64' and python_version == '3.14'" },
{ url = "https://github.com/paperless-ngx/builder/releases/download/psycopg-trixie-3.3.4/psycopg_c-3.3.4-cp314-cp314-linux_aarch64.whl", marker = "sys_platform == 'linux' and platform_machine == 'aarch64' and python_version == '3.14'" },
]
torch = [
{ index = "pytorch-cpu" },
@@ -33,7 +33,7 @@ test('should warn on unsaved changes', async ({ page }) => {
await page.getByRole('button', { name: 'Close', exact: true }).click()
await expect(page.getByRole('dialog')).toHaveText(/unsaved changes/)
await page.getByRole('button', { name: 'Cancel' }).click()
await page.getByRole('link', { name: 'Close all' }).click()
await page.getByRole('button', { name: 'Close all' }).click()
await expect(page.getByRole('dialog')).toHaveText(/unsaved changes/)
})
+653 -221
View File
File diff suppressed because it is too large Load Diff
@@ -89,7 +89,7 @@
</div>
</div>
@if (filterText?.length) {
<button class="btn btn-link btn-sm px-2 position-absolute top-0 end-0 z-10" (click)="resetFilter()">
<button class="btn btn-link btn-sm px-2 position-absolute top-0 end-0 z-10" (click)="resetFilter()" aria-label="Clear search" i18n-aria-label>
<i-bs width="1em" height="1em" name="x"></i-bs>
</button>
}
@@ -208,7 +208,7 @@
</td>
}
<td class="d-lg-none">
<button class="btn btn-link" (click)="expandTask(task); $event.stopPropagation();">
<button class="btn btn-link" (click)="expandTask(task); $event.stopPropagation();" aria-label="View task details" i18n-aria-label>
<i-bs width="1.2em" height="1.2em" name="info-circle"></i-bs>
</button>
</td>
@@ -18,9 +18,11 @@
</button>
</pngx-page-header>
<div class="row mb-3">
<ngb-pagination class="col-auto" [pageSize]="25" [collectionSize]="totalDocuments()" [page]="page()" [maxSize]="5" (pageChange)="page.set($event); reload()" size="sm" aria-label="Pagination"></ngb-pagination>
</div>
@if (totalDocuments() > 25) {
<div class="row mb-3">
<ngb-pagination class="col-auto" [pageSize]="25" [collectionSize]="totalDocuments()" [page]="page()" [maxSize]="5" (pageChange)="page.set($event); reload()" size="sm" aria-label="Pagination"></ngb-pagination>
</div>
}
<div class="card border table-responsive mb-3">
<table class="table table-striped align-middle shadow-sm mb-0">
@@ -64,7 +66,7 @@
<td scope="row">
<div class="btn-group d-block d-sm-none">
<div ngbDropdown container="body" class="d-inline-block">
<button type="button" class="btn btn-link" id="actionsMenuMobile" (click)="$event.stopPropagation()" ngbDropdownToggle>
<button type="button" class="btn btn-link" id="actionsMenuMobile" (click)="$event.stopPropagation()" ngbDropdownToggle aria-label="Actions" i18n-aria-label>
<i-bs name="three-dots-vertical"></i-bs>
</button>
<div ngbDropdownMenu aria-labelledby="actionsMenuMobile">
@@ -4,27 +4,34 @@
(click)="closeMobileSearch(); toggleMenuCollapsed()">
<span class="navbar-toggler-icon"></span>
</button>
<a class="navbar-brand d-flex align-items-center me-0 px-3 py-3 order-sm-0"
[ngClass]="{ 'slim': slimSidebarEnabled, 'col-auto col-md-3 col-lg-2 col-xxxl-1' : !slimSidebarEnabled, 'py-3' : !customAppTitle?.length || slimSidebarEnabled, 'py-2': customAppTitle?.length }"
<a class="navbar-brand d-flex align-items-center me-0 ps-md-3 py-0 order-sm-0"
[ngClass]="{ 'slim': slimSidebarEnabled, '' : !slimSidebarEnabled }"
routerLink="/dashboard"
tourAnchor="tour.intro">
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1000 1000" width="1.5em" height="1.5em" fill="currentColor">
<path d="M341,949.1c-6.9-20.3-20.7-61.2-21.9-61-199.6-88.9-182.5-229.8-134.3-347.5,30,137.2,268.8,148.9,146.2,336-.9,2.2,10,27.8,19.5,51.3,22.7-51.9,58.6-115.5,55.8-120.8C178,398.7,724.9,299,807.1,18.5c83,251.5,53.1,659.8-377.4,814.9-2,1.4-63.5,148.6-66.9,150.2-.2-2.1-33.2,2.9-30.1-8.7,1.6-7,4.8-16.2,8.2-25.6h0v-.2h.1ZM323.1,846.2c48.3-71.9-12.7-120.8-56.9-152.2,81.2,107.4,66.4,120.8,56.9,152.2h0Z"/>
</svg>
<div class="ms-2 ms-md-3 d-inline-block" [class.d-md-none]="slimSidebarEnabled">
@if (customAppTitle?.length) {
<div class="d-flex flex-column align-items-start custom-title">
<span class="title">{{customAppTitle}}</span>
<span class="byline text-uppercase font-monospace" i18n>by Paperless-ngx</span>
</div>
@if (!hasCustomBranding) {
<pngx-logo extra_classes="navbar-official-logo px-1" height="2.4rem"></pngx-logo>
<svg class="brand-mark brand-mark-slim d-none" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1000 1000" width="1.5em" height="1.5em" fill="currentColor">
<path d="M341,949.1c-6.9-20.3-20.7-61.2-21.9-61-199.6-88.9-182.5-229.8-134.3-347.5,30,137.2,268.8,148.9,146.2,336-.9,2.2,10,27.8,19.5,51.3,22.7-51.9,58.6-115.5,55.8-120.8C178,398.7,724.9,299,807.1,18.5c83,251.5,53.1,659.8-377.4,814.9-2,1.4-63.5,148.6-66.9,150.2-.2-2.1-33.2,2.9-30.1-8.7,1.6-7,4.8-16.2,8.2-25.6h0v-.2h.1ZM323.1,846.2c48.3-71.9-12.7-120.8-56.9-152.2,81.2,107.4,66.4,120.8,56.9,152.2h0Z"/>
</svg>
} @else {
@if (customAppLogo) {
<img class="brand-logo" [src]="customAppLogo" alt="" />
} @else {
Paperless-ngx
<svg class="brand-mark" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1000 1000" width="1.5em" height="1.5em" fill="currentColor">
<path d="M341,949.1c-6.9-20.3-20.7-61.2-21.9-61-199.6-88.9-182.5-229.8-134.3-347.5,30,137.2,268.8,148.9,146.2,336-.9,2.2,10,27.8,19.5,51.3,22.7-51.9,58.6-115.5,55.8-120.8C178,398.7,724.9,299,807.1,18.5c83,251.5,53.1,659.8-377.4,814.9-2,1.4-63.5,148.6-66.9,150.2-.2-2.1-33.2,2.9-30.1-8.7,1.6-7,4.8-16.2,8.2-25.6h0v-.2h.1ZM323.1,846.2c48.3-71.9-12.7-120.8-56.9-152.2,81.2,107.4,66.4,120.8,56.9,152.2h0Z"/>
</svg>
}
</div>
<div class="brand-copy ms-2 text-truncate" [class.d-md-none]="slimSidebarEnabled">
<span class="brand-title text-truncate">{{ appTitle }}</span>
@if (customAppTitle) {
<span class="byline text-uppercase font-monospace" i18n>by Paperless-ngx</span>
}
</div>
}
</a>
<div class="search-container flex-grow-1 py-2 pb-3 pb-sm-2 px-3 ps-md-4 me-sm-auto order-3 order-sm-1"
<div class="search-container flex-grow-1 py-2 pb-3 pb-sm-2 px-3 ps-md-3 me-sm-auto order-3 order-sm-1"
[class.mobile-hidden]="mobileSearchHidden()">
<div class="col-12 col-md-7">
<div class="col-12 header-search">
<pngx-global-search></pngx-global-search>
</div>
</div>
@@ -34,7 +41,7 @@
}
<pngx-toasts-dropdown></pngx-toasts-dropdown>
<li ngbDropdown class="nav-item dropdown">
<button class="btn ps-1 border-0" id="userDropdown" ngbDropdownToggle>
<button class="btn navbar-action border-0" id="userDropdown" ngbDropdownToggle aria-label="User menu" i18n-aria-label>
<i-bs width="1.3em" height="1.3em" name="person-circle"></i-bs>
<span class="small ms-2 d-none d-sm-inline">
{{this.settingsService.displayName}}
@@ -71,7 +78,7 @@
[ngClass]="slimSidebarEnabled ? 'slim' : 'col-md-3 col-lg-2 col-xxxl-1'" [class.animating]="slimSidebarAnimating()"
[ngbCollapse]="isMenuCollapsed()">
@if (canSaveSettings) {
<button class="btn btn-sm btn-dark sidebar-slim-toggler" (click)="toggleSlimSidebar()">
<button class="btn btn-sm btn-dark sidebar-slim-toggler" (click)="toggleSlimSidebar()" [aria-label]="slimSidebarEnabled ? 'Expand sidebar' : 'Collapse sidebar'" i18n-aria-label>
@if (slimSidebarEnabled) {
<i-bs width="0.9em" height="0.9em" name="chevron-double-right"></i-bs>
} @else {
@@ -79,7 +86,7 @@
}
</button>
}
<div class="sidebar-sticky pt-3 d-flex flex-column justify-space-around">
<div class="sidebar-sticky pt-3 pb-1 d-flex flex-column justify-space-around">
<ul class="nav flex-column">
<li class="nav-item app-link">
<a class="nav-link" routerLink="dashboard" routerLinkActive="active" (click)="closeMenu()"
@@ -89,7 +96,9 @@
</a>
</li>
<li class="nav-item app-link" *pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.Document }">
<a class="nav-link" routerLink="documents" routerLinkActive="active" (click)="closeMenu()"
<a class="nav-link" routerLink="documents" routerLinkActive="active"
[routerLinkActiveOptions]="{ paths: 'exact', queryParams: 'ignored', matrixParams: 'ignored', fragment: 'ignored' }"
(click)="closeMenu()"
ngbPopover="Documents" i18n-ngbPopover [disablePopover]="!slimSidebarEnabled" placement="end"
container="body" triggers="mouseenter:mouseleave" popoverClass="popover-slim">
<i-bs class="me-2" name="files"></i-bs><span><ng-container i18n>Documents</ng-container></span>
@@ -159,11 +168,12 @@
}
@if (openDocuments.length >= 1) {
<li class="nav-item w-100 app-link">
<a class="nav-link app-link" [class.text-truncate]="!slimSidebarEnabled" [routerLink]="[]" (click)="closeAll()"
<button type="button" class="nav-link nav-link-action app-link w-100 text-start"
[class.text-truncate]="!slimSidebarEnabled" (click)="closeAll()"
ngbPopover="Close all" i18n-ngbPopover [disablePopover]="!slimSidebarEnabled" placement="end"
container="body" triggers="mouseenter:mouseleave" popoverClass="popover-slim">
<i-bs class="me-2" name="x"></i-bs><span><ng-container i18n>Close all</ng-container></span>
</a>
</button>
</li>
}
</ul>
@@ -177,10 +187,11 @@
@if (canManageAttributes) {
<li class="nav-item app-link" tourAnchor="tour.tags">
<div class="d-flex align-items-center attributes-row">
<a class="nav-link flex-fill" routerLink="attributes" routerLinkActive="active" (click)="closeMenu()"
<a class="nav-link flex-fill" routerLink="attributes" routerLinkActive="active"
[routerLinkActiveOptions]="{ exact: !(slimSidebarEnabled || attributesSectionsCollapsed) }" (click)="closeMenu()"
ngbPopover="Attributes" i18n-ngbPopover [disablePopover]="!slimSidebarEnabled" placement="end"
container="body" triggers="mouseenter:mouseleave" popoverClass="popover-slim">
<i-bs class="me-2" name="stack"></i-bs><span><ng-container i18n>Attributes</ng-container></span>
<i-bs name="stack"></i-bs><span class="ms-2"><ng-container i18n>Attributes</ng-container></span>
</a>
@if (!slimSidebarEnabled && canSaveSettings) {
<button
@@ -195,32 +206,32 @@
}
</div>
<div
class="attributes-submenu ms-2"
class="attributes-submenu ms-3"
[ngbCollapse]="slimSidebarEnabled || attributesSectionsCollapsed"
>
<ul class="nav flex-column">
<li class="nav-item app-link" *pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.Tag }">
<a class="nav-link py-1" routerLink="attributes/tags" routerLinkActive="active" (click)="closeMenu()">
<a class="nav-link" routerLink="attributes/tags" routerLinkActive="active" (click)="closeMenu()">
<i-bs class="me-2" name="tags"></i-bs><span><ng-container i18n>Tags</ng-container></span>
</a>
</li>
<li class="nav-item app-link" *pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.Correspondent }">
<a class="nav-link py-1" routerLink="attributes/correspondents" routerLinkActive="active" (click)="closeMenu()">
<a class="nav-link" routerLink="attributes/correspondents" routerLinkActive="active" (click)="closeMenu()">
<i-bs class="me-2" name="person"></i-bs><span><ng-container i18n>Correspondents</ng-container></span>
</a>
</li>
<li class="nav-item app-link" *pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.DocumentType }">
<a class="nav-link py-1" routerLink="attributes/documenttypes" routerLinkActive="active" (click)="closeMenu()">
<a class="nav-link" routerLink="attributes/documenttypes" routerLinkActive="active" (click)="closeMenu()">
<i-bs class="me-2" name="hash"></i-bs><span><ng-container i18n>Document types</ng-container></span>
</a>
</li>
<li class="nav-item app-link" *pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.StoragePath }">
<a class="nav-link py-1" routerLink="attributes/storagepaths" routerLinkActive="active" (click)="closeMenu()">
<a class="nav-link" routerLink="attributes/storagepaths" routerLinkActive="active" (click)="closeMenu()">
<i-bs class="me-2" name="folder"></i-bs><span><ng-container i18n>Storage paths</ng-container></span>
</a>
</li>
<li class="nav-item app-link" *pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.CustomField }">
<a class="nav-link py-1" routerLink="attributes/customfields" routerLinkActive="active" (click)="closeMenu()">
<a class="nav-link" routerLink="attributes/customfields" routerLinkActive="active" (click)="closeMenu()">
<i-bs class="me-2" name="ui-radios"></i-bs><span><ng-container i18n>Custom fields</ng-container></span>
</a>
</li>
@@ -363,7 +374,7 @@
</a>
}
} @else {
<a *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.UISettings }" class="small text-decoration-none" routerLink="/settings" fragment="update-checking"
<a *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.UISettings }" class="small text-decoration-none" routerLink="/settings" fragment="update-checking" aria-label="Configure update checking" i18n-aria-label
[ngbPopover]="updateCheckingNotEnabledPopContent" popoverClass="shadow" triggers="mouseenter"
container="body">
<i-bs width="1.2em" height="1.2em" name="info-circle"></i-bs>
@@ -7,8 +7,8 @@
bottom: 0;
left: 0;
z-index: 995; /* Behind the navbar */
padding: 50px 0 0; /* Height of navbar */
box-shadow: inset -1px 0 0 rgba(0, 0, 0, .1);
padding: 64px 0 0; /* Height of navbar */
border-right: 1px solid color-mix(in srgb, var(--bs-border-color) 65%, transparent);
overflow-y: auto;
--pngx-sidebar-width: 100%;
max-width: var(--pngx-sidebar-width);
@@ -25,7 +25,7 @@
}
.view-name {
max-width: calc(100% - 50px)
max-width: calc(100% - 55px)
}
.nav-group:not(:has(.app-link)) .sidebar-heading {
@@ -47,7 +47,7 @@
}
@media (max-width: 767.98px) {
.sidebar {
top: 3.5rem;
top: 4rem;
}
.search-container {
@@ -71,12 +71,14 @@
main {
transition: all .2s ease;
padding-top: 110px;
padding-top: 118px;
background: var(--pngx-bg-darker);
min-height: 100vh;
}
@media (min-width: 768px) {
main {
padding-top: 56px;
padding-top: 64px;
}
}
@@ -91,7 +93,7 @@ main {
@media(min-width: 768px) {
.sidebar.slim {
max-width: 50px;
max-width: 55px;
li.nav-item span.badge {
display: inline-block;
@@ -150,7 +152,7 @@ main {
display: block;
position: fixed;
left: calc(var(--pngx-sidebar-width) - 12px);
top: 60px;
top: 72px;
z-index: 996;
--bs-btn-padding-x: 0.35rem;
--bs-btn-padding-y: 0.125rem;
@@ -158,7 +160,7 @@ main {
}
.sidebar.slim .sidebar-slim-toggler {
--pngx-sidebar-width: 50px !important;
--pngx-sidebar-width: 56px !important;
}
}
@@ -171,7 +173,7 @@ main {
position: relative;
top: 0;
height: 100%;
padding-top: 0.5rem;
padding: .75rem .5rem 1rem;
overflow-x: hidden;
overflow-y: auto; /* Scrollable contents if viewport is shorter than content. */
min-height: min-content;
@@ -186,11 +188,14 @@ main {
.sidebar .nav-link {
font-weight: 500;
white-space: nowrap;
border-left: 2px solid transparent;
transition: color .15s ease-in-out;
border-radius: .55rem;
margin: 1px 0;
padding: .55rem .7rem;
transition: color .15s ease-in-out, background-color .15s ease-in-out;
&:hover, &.active, &:focus {
&:hover, &:focus {
color: var(--bs-primary);
background-color: color-mix(in srgb, var(--bs-primary) 8%, transparent);
}
&:focus-visible {
@@ -199,8 +204,21 @@ main {
}
&.active {
font-weight: bold;
border-left-color: var(--bs-primary);
font-weight: 600;
color: var(--bs-primary);
background-color: color-mix(in srgb, var(--bs-primary) 13%, transparent);
}
&.nav-link-action {
color: var(--bs-secondary-color);
font-weight: 400;
background-color: transparent;
&:hover,
&:focus {
color: var(--bs-primary);
background-color: transparent;
}
}
i-bs {
@@ -209,20 +227,45 @@ main {
}
}
// sub-page gets marker only
.nav-item:has(.attributes-submenu.show .nav-link.active) > .attributes-row > .nav-link.active {
border-left-color: transparent;
}
.attributes-row {
border-radius: .55rem;
margin: .1rem 0;
transition: color .15s ease-in-out, background-color .15s ease-in-out;
// bring sub-menu markers back out to L edge
.attributes-submenu .nav-link {
margin-left: -0.5rem;
padding-left: calc(var(--bs-nav-link-padding-x) + 0.5rem);
> .nav-link {
margin: 0;
&:hover,
&:focus,
&.active {
background-color: transparent;
}
}
&:hover,
&:has(> .nav-link:focus-visible) {
background-color: color-mix(in srgb, var(--bs-primary) 8%, transparent);
}
&:has(> .nav-link.active) {
background-color: color-mix(in srgb, var(--bs-primary) 13%, transparent);
}
}
.attributes-row .attributes-expand-btn {
opacity: 0.2;
width: 1.75rem;
height: 1.75rem;
margin-right: .35rem !important;
border-radius: 50%;
box-shadow: none !important;
transition: opacity 0.15s ease-in-out;
&:focus-visible {
outline: 2px solid color-mix(in srgb, var(--bs-primary) 55%, transparent);
outline-offset: 1px;
opacity: 1;
}
}
.attributes-row:hover .attributes-expand-btn {
@@ -230,8 +273,10 @@ main {
}
.sidebar-heading {
font-size: 0.75rem;
font-size: 0.68rem;
text-transform: uppercase;
font-weight: 700;
opacity: .75;
}
.nav {
@@ -288,16 +333,118 @@ main {
*/
.navbar-brand {
font-size: 1rem;
--pngx-navbar-brand-shadow-rgb: 0, 0, 0;
font-size: 1.0625rem;
min-height: 64px;
letter-spacing: -0.015em;
.flex-column {
padding: 0.15rem 0;
&:hover,
&:focus-visible {
::ng-deep .navbar-official-logo,
.brand-mark,
.brand-logo {
filter: drop-shadow(0 2px 3px rgba(var(--pngx-navbar-brand-shadow-rgb), .5));
}
}
::ng-deep .navbar-official-logo {
filter: drop-shadow(0 1px 2px rgba(var(--pngx-navbar-brand-shadow-rgb), .3));
transition: filter .15s ease-in-out;
@media screen and (max-width: 575.98px) {
max-height: 2rem;
}
}
.brand-mark {
width: 1.65rem;
height: 1.65rem;
flex: 0 0 auto;
transition: filter .15s ease-in-out;
}
.brand-copy {
display: flex;
flex-direction: column;
align-items: flex-start;
min-width: 0;
line-height: 1.1;
transition: transform .15s ease-in-out;
}
.brand-title {
font-weight: 600;
max-width: 100%;
min-width: 0;
}
.byline {
font-size: 0.5rem;
letter-spacing: 0.1rem;
margin-top: .15rem;
font-size: .5rem;
font-weight: 500;
letter-spacing: .1rem;
opacity: .8;
}
.brand-logo {
width: auto;
height: 2.75rem;
max-width: 5rem;
flex: 0 0 auto;
object-fit: contain;
transition: filter .15s ease-in-out, transform .15s ease-in-out;
}
}
:host-context(.primary-light) .navbar-brand {
--pngx-navbar-brand-shadow-rgb: 255, 255, 255; // Light app color, use white shadow for dark text
}
:host ::ng-deep .navbar-official-logo {
.leaf {
fill: color-mix(in srgb, var(--pngx-primary-text-contrast) 70%, var(--bs-primary)) !important;
}
.text {
fill: var(--pngx-primary-text-contrast) !important;
}
}
.navbar {
min-height: 64px;
box-shadow: 0 1px 0 rgba(0, 0, 0, .12), 0 4px 18px rgba(0, 0, 0, .08) !important;
}
.navbar > ul {
align-items: center;
gap: .125rem;
padding-right: .5rem;
}
:host ::ng-deep .navbar-action {
display: inline-flex;
align-items: center;
justify-content: center;
min-width: 2.5rem;
min-height: 2.5rem;
padding: .45rem .55rem;
border-radius: .6rem;
transition: background-color .15s ease-in-out, opacity .15s ease-in-out;
&:hover,
&:focus-visible {
background-color: rgba(0, 0, 0, .14);
}
}
#userDropdown {
padding-left: .75rem;
}
.header-search {
width: 100%;
max-width: 44rem;
}
@media screen and (max-width: 575.98px) {
@@ -346,18 +493,44 @@ main {
grid-area: actions;
justify-self: end;
flex-wrap: nowrap;
gap: 0;
padding-right: .25rem;
}
:host ::ng-deep .navbar-action {
min-width: 2.25rem;
padding-right: .4rem;
padding-left: .4rem;
}
#userDropdown {
padding-right: .35rem;
padding-left: .45rem;
}
}
@media screen and (min-width: 768px) {
.navbar-brand.slim {
max-width: 50px;
max-width: 55px;
.brand-logo {
width: 1.65rem;
max-width: 1.65rem;
}
.brand-mark-slim {
display: block !important;
}
}
:host ::ng-deep .navbar-brand.slim .navbar-official-logo {
display: none;
}
}
:host ::ng-deep .dropdown.show .dropdown-toggle,
:host ::ng-deep .dropdown-toggle:hover {
opacity: 0.7;
opacity: 1;
}
.dropdown-toggle::after {
@@ -45,6 +45,7 @@ import { TasksService } from 'src/app/services/tasks.service'
import { ToastService } from 'src/app/services/toast.service'
import { environment } from 'src/environments/environment'
import { ChatComponent } from '../chat/chat/chat.component'
import { LogoComponent } from '../common/logo/logo.component'
import { ProfileEditDialogComponent } from '../common/profile-edit-dialog/profile-edit-dialog.component'
import { DocumentDetailComponent } from '../document-detail/document-detail.component'
import { ComponentWithPermissions } from '../with-permissions/with-permissions.component'
@@ -59,6 +60,7 @@ const SCROLL_THRESHOLD = 16
styleUrls: ['./app-frame.component.scss'],
imports: [
GlobalSearchComponent,
LogoComponent,
DocumentTitlePipe,
IfPermissionsDirective,
ToastsDropdownComponent,
@@ -190,11 +192,34 @@ export class AppFrameComponent
return `${environment.appTitle} v${this.settingsService.get(SETTINGS_KEYS.VERSION)}${environment.tag === 'prod' ? '' : ` #${environment.tag}`}`
}
get appTitle(): string {
this.settingsService.trackChanges()
return (
this.settingsService.get(SETTINGS_KEYS.APP_TITLE) || environment.appTitle
)
}
get customAppTitle(): string {
this.settingsService.trackChanges()
return this.settingsService.get(SETTINGS_KEYS.APP_TITLE)
}
get hasCustomBranding(): boolean {
this.settingsService.trackChanges()
return !!(
this.settingsService.get(SETTINGS_KEYS.APP_TITLE)?.length ||
this.settingsService.get(SETTINGS_KEYS.APP_LOGO)?.length
)
}
get customAppLogo(): string {
this.settingsService.trackChanges()
const logo = this.settingsService.get(SETTINGS_KEYS.APP_LOGO)
return logo?.length
? environment.apiBaseUrl.replace(/\/api\/$/, logo)
: null
}
get canSaveSettings(): boolean {
return (
this.permissionsService.currentUserCan(
@@ -4,12 +4,12 @@ form {
> i-bs[name="search"] {
position: absolute;
left: 0.6rem;
top: .35rem;
top: .25rem;
color: rgba(255, 255, 255, 0.6);
@media screen and (min-width: 768px) {
// adjust for smaller font size on non-mobile
top: 0.25rem;
top: .15rem;
}
}
@@ -37,8 +37,9 @@ form {
}
.form-control {
color: rgba(255, 255, 255, 0.3);
background-color: rgba(0, 0, 0, 0.15);
min-height: 2.25rem;
color: rgba(255, 255, 255, 0.55);
background-color: rgba(0, 0, 0, 0.16);
padding-left: 1.8rem;
border-color: rgba(255, 255, 255, 0.2);
transition: all .3s ease, padding-left 0s ease, background-color 0s ease; // Safari requires all
@@ -52,7 +53,7 @@ form {
}
&:focus-within {
background-color: rgba(0, 0, 0, 0.3);
background-color: rgba(0, 0, 0, 0.26);
color: var(--bs-light);
flex-grow: 1;
padding-left: 0.5rem;
@@ -1,9 +1,9 @@
<li ngbDropdown class="nav-item mx-1" (openChange)="onOpenChange($event)">
<li ngbDropdown class="nav-item position-relative" (openChange)="onOpenChange($event)">
@if (toasts().length) {
<span class="badge rounded-pill z-3 pe-none bg-secondary me-2 position-absolute top-0 left-0">{{ toasts().length }}</span>
<span class="notification-count badge rounded-pill z-3 pe-none bg-secondary position-absolute">{{ toasts().length }}</span>
}
<button class="btn border-0" id="notificationsDropdown" ngbDropdownToggle>
<button class="btn navbar-action border-0" id="notificationsDropdown" ngbDropdownToggle aria-label="Notifications" i18n-aria-label>
<i-bs width="1.3em" height="1.3em" name="bell"></i-bs>
</button>
<div ngbDropdownMenu class="dropdown-menu-end shadow p-3" aria-labelledby="notificationsDropdown">
@@ -11,6 +11,16 @@
display: none;
}
.notification-count {
top: -.2rem;
right: -.2rem;
min-width: 1.15rem;
height: 1.15rem;
padding: .18rem .32rem;
font-size: .68rem;
line-height: 1;
}
.dropdown-item {
white-space: initial;
}
@@ -1,6 +1,6 @@
<li ngbDropdown class="nav-item me-n2" (openChange)="onOpenChange($event)">
<button class="btn border-0" id="chatDropdown" ngbDropdownToggle>
<li ngbDropdown class="nav-item" (openChange)="onOpenChange($event)">
<button class="btn navbar-action border-0" id="chatDropdown" ngbDropdownToggle aria-label="Chat" i18n-aria-label>
<i-bs width="1.3em" height="1.3em" name="chatSquareDots"></i-bs>
</button>
<div ngbDropdownMenu class="dropdown-menu-end shadow p-3" aria-labelledby="chatDropdown">
@@ -0,0 +1,58 @@
<div class="modal-header">
<h4 class="modal-title" id="modal-basic-title">{{title}}</h4>
<button type="button" class="btn-close" aria-label="Close" (click)="cancel()"></button>
</div>
<div class="modal-body">
<p>{{message}}</p>
<div class="form-group">
<label class="form-label" for="rootDocumentID" i18n>Root document:</label>
<select id="rootDocumentID" class="form-select" [ngModel]="rootDocumentID()" (ngModelChange)="rootDocumentID.set($event)">
@for (document of documents(); track document.id) {
<option [ngValue]="document.id">{{document.title}}</option>
}
</select>
</div>
<div class="form-group mt-4">
<span class="form-label d-inline-block" i18n>Versions (oldest first):</span>
<ul class="list-group"
cdkDropList
[cdkDropListData]="versionDocumentIDs()"
(cdkDropListDropped)="onDrop($event)">
@for (documentID of versionDocumentIDs(); track documentID) {
@let document = getDocument(documentID);
@if (document) {
<li class="list-group-item d-flex align-items-center" cdkDrag>
<i-bs name="grip-vertical" class="me-2"></i-bs>
<div class="d-flex flex-column">
<div>
@if (document.correspondent) {
<b>{{document.correspondent | correspondentName | async}}: </b>
}{{document.title}}
</div>
<small class="text-muted">
{{document.created | customDate:'mediumDate'}}
@if (document.page_count) {
| {document.page_count, plural, =1 {One page} other {{{document.page_count}} pages}}
}
</small>
</div>
@if ($last) {
<span class="badge bg-primary ms-auto" i18n>Current version</span>
}
</li>
}
}
</ul>
@if (versionDocumentIDs().length > 1) {
<div class="form-text" i18n>Drag to reorder.</div>
}
</div>
</div>
<div class="modal-footer">
<button type="button" class="btn" [class]="cancelBtnClass" (click)="cancel()" [disabled]="!buttonsEnabled()">
<span class="d-inline-block" style="padding-bottom: 1px;">{{cancelBtnCaption}}</span>
</button>
<button type="button" class="btn" [class]="btnClass" (click)="confirm()" [disabled]="!confirmButtonEnabled || !buttonsEnabled()">
{{btnCaption}}
</button>
</div>
@@ -0,0 +1,70 @@
import { provideHttpClient, withInterceptorsFromDi } from '@angular/common/http'
import { provideHttpClientTesting } from '@angular/common/http/testing'
import { ComponentFixture, TestBed } from '@angular/core/testing'
import { NgbActiveModal } from '@ng-bootstrap/ng-bootstrap'
import { NgxBootstrapIconsModule, allIcons } from 'ngx-bootstrap-icons'
import { of } from 'rxjs'
import { DocumentService } from 'src/app/services/rest/document.service'
import { MergeAsVersionsConfirmDialogComponent } from './merge-as-versions-confirm-dialog.component'
describe('MergeAsVersionsConfirmDialogComponent', () => {
let component: MergeAsVersionsConfirmDialogComponent
let fixture: ComponentFixture<MergeAsVersionsConfirmDialogComponent>
let documentService: DocumentService
beforeEach(async () => {
await TestBed.configureTestingModule({
imports: [
NgxBootstrapIconsModule.pick(allIcons),
MergeAsVersionsConfirmDialogComponent,
],
providers: [
NgbActiveModal,
provideHttpClient(withInterceptorsFromDi()),
provideHttpClientTesting(),
],
}).compileComponents()
fixture = TestBed.createComponent(MergeAsVersionsConfirmDialogComponent)
documentService = TestBed.inject(DocumentService)
component = fixture.componentInstance
fixture.detectChanges()
})
it('should fetch selected documents', () => {
const documents = [
{ id: 1, title: 'Document 1' },
{ id: 2, title: 'Document 2' },
]
jest.spyOn(documentService, 'getFew').mockReturnValue(
of({
all: [1, 2],
count: 2,
results: documents,
})
)
component.documentIDs.set([1, 2])
component.ngOnInit()
expect(component.documents()).toEqual(documents)
expect(documentService.getFew).toHaveBeenCalledWith([1, 2])
})
it('should exclude the root from the draggable documents', () => {
component.documentIDs.set([1, 2, 3])
component.rootDocumentID.set(2)
expect(component.versionDocumentIDs()).toEqual([1, 3])
})
it('should move draggable documents while keeping the root fixed', () => {
component.documentIDs.set([1, 2, 3])
component.rootDocumentID.set(1)
component.onDrop({ previousIndex: 1, currentIndex: 0 } as any)
expect(component.documentIDs()).toEqual([1, 3, 2])
expect(component.versionDocumentIDs()).toEqual([3, 2])
})
})
@@ -0,0 +1,70 @@
import {
CdkDragDrop,
DragDropModule,
moveItemInArray,
} from '@angular/cdk/drag-drop'
import { AsyncPipe } from '@angular/common'
import { Component, OnInit, computed, inject, signal } from '@angular/core'
import { FormsModule } from '@angular/forms'
import { NgxBootstrapIconsModule } from 'ngx-bootstrap-icons'
import { takeUntil } from 'rxjs'
import { Document } from 'src/app/data/document'
import { CorrespondentNamePipe } from 'src/app/pipes/correspondent-name.pipe'
import { CustomDatePipe } from 'src/app/pipes/custom-date.pipe'
import { DocumentService } from 'src/app/services/rest/document.service'
import { ConfirmDialogComponent } from '../confirm-dialog.component'
@Component({
selector: 'pngx-merge-as-versions-confirm-dialog',
templateUrl: './merge-as-versions-confirm-dialog.component.html',
styleUrl: './merge-as-versions-confirm-dialog.component.scss',
imports: [
AsyncPipe,
CorrespondentNamePipe,
CustomDatePipe,
DragDropModule,
FormsModule,
NgxBootstrapIconsModule,
],
})
export class MergeAsVersionsConfirmDialogComponent
extends ConfirmDialogComponent
implements OnInit
{
private readonly documentService = inject(DocumentService)
readonly documentIDs = signal<number[]>([])
readonly documents = signal<Document[]>([])
readonly rootDocumentID = signal(-1)
readonly versionDocumentIDs = computed(() =>
this.documentIDs().filter(
(documentID) => documentID !== this.rootDocumentID()
)
)
ngOnInit() {
this.documentService
.getFew(this.documentIDs())
.pipe(takeUntil(this.unsubscribeNotifier))
.subscribe((response) => this.documents.set(response.results))
}
onDrop(event: CdkDragDrop<number[]>) {
const versionDocumentIDs = this.versionDocumentIDs().concat()
moveItemInArray(versionDocumentIDs, event.previousIndex, event.currentIndex)
// The root keeps its place in the list, only the versions move around it
let versionIndex = 0
this.documentIDs.update((documentIDs) =>
documentIDs.map((documentID) =>
documentID === this.rootDocumentID()
? documentID
: versionDocumentIDs[versionIndex++]
)
)
}
getDocument(documentID: number): Document | undefined {
return this.documents().find((document) => document.id === documentID)
}
}
@@ -36,7 +36,7 @@
</div>
<div class="form-group mt-4">
<label class="form-label" for="metadataDocumentID" i18n>Use metadata from:</label>
<select class="form-select" [ngModel]="metadataDocumentID()" (ngModelChange)="metadataDocumentID.set($event)">
<select id="metadataDocumentID" class="form-select" [ngModel]="metadataDocumentID()" (ngModelChange)="metadataDocumentID.set($event)">
<option [ngValue]="-1" i18n>Regenerate all metadata</option>
@for (document of documents(); track document.id) {
<option [ngValue]="document.id">{{document.title}}</option>
@@ -6,7 +6,7 @@
<div class="modal-body">
<div class="row">
<div class="col-2 d-flex justify-content-end">
<button class="btn btn-secondary mt-auto" (click)="rotate(false)">
<button class="btn btn-secondary mt-auto" (click)="rotate(false)" aria-label="Rotate counterclockwise" i18n-aria-label>
<i-bs name="arrow-counterclockwise"></i-bs>
</button>
</div>
@@ -16,7 +16,7 @@
}
</div>
<div class="col-2 d-flex">
<button class="btn btn-secondary mt-auto" (click)="rotate()">
<button class="btn btn-secondary mt-auto" (click)="rotate()" aria-label="Rotate clockwise" i18n-aria-label>
<i-bs name="arrow-clockwise"></i-bs>
</button>
</div>
@@ -1,5 +1,5 @@
<div ngbDropdown #fieldDropdown="ngbDropdown" (openChange)="onOpenClose($event)" [popperOptions]="popperOptions">
<button type="button" class="btn btn-sm btn-outline-primary" id="customFieldsDropdown" [disabled]="disabled" ngbDropdownToggle>
<button type="button" class="btn btn-sm btn-outline-primary" id="customFieldsDropdown" [disabled]="disabled" ngbDropdownToggle aria-label="Custom Fields" i18n-aria-label>
<i-bs name="ui-radios"></i-bs><div class="d-none d-lg-inline ms-1"><ng-container i18n>Custom Fields</ng-container></div>
</button>
<div ngbDropdownMenu aria-labelledby="customFieldsDropdown" class="shadow custom-fields-dropdown">
@@ -1,6 +1,6 @@
@if (useDropdown) {
<div class="btn-group w-100" role="group" ngbDropdown #dropdown="ngbDropdown" (openChange)="onOpenChange($event)" [popperOptions]="popperOptions">
<button class="btn btn-sm btn-outline-primary" id="dropdown_toggle" ngbDropdownToggle [disabled]="disabled">
<button class="btn btn-sm btn-outline-primary" id="dropdown_toggle" ngbDropdownToggle [disabled]="disabled" [aria-label]="title">
<i-bs name="{{icon}}"></i-bs><div class="d-none d-sm-inline ms-1">{{title}}</div>
@if (isActive) {
<pngx-clearable-badge [selected]="isActive" (cleared)="reset()"></pngx-clearable-badge>
@@ -38,7 +38,7 @@
ngbDatepicker
#d="ngbDatepicker"
[footerTemplate]="datePickerFooterTemplate" />
<button class="btn btn-sm btn-outline-secondary rounded-end" (click)="d.toggle()" type="button">
<button class="btn btn-sm btn-outline-secondary rounded-end" (click)="d.toggle()" type="button" aria-label="Open date picker" i18n-aria-label>
<i-bs name="calendar-event"></i-bs>
</button>
<ng-template #datePickerFooterTemplate>
@@ -143,7 +143,7 @@
<input class="w-25 form-control rounded-end" type="text" [(ngModel)]="atom.value" [disabled]="disabled">
}
}
<button class="btn btn-link btn-sm text-danger pe-0" type="button" (click)="removeElement(atom)" [disabled]="disabled">
<button class="btn btn-link btn-sm text-danger pe-0" type="button" (click)="removeElement(atom)" [disabled]="disabled" aria-label="Remove query" i18n-aria-label>
<i-bs name="x-circle"></i-bs>
</button>
</div>
@@ -185,7 +185,7 @@
<i-bs name="braces"></i-bs>
</button>
@if (expression.depth > 0) {
<button type="button" class="btn btn-sm btn-outline-secondary text-danger" (click)="removeElement(expression)" [disabled]="disabled">
<button type="button" class="btn btn-sm btn-outline-secondary text-danger" (click)="removeElement(expression)" [disabled]="disabled" aria-label="Remove expression" i18n-aria-label>
<i-bs name="x-circle"></i-bs>
</button>
}
@@ -1,5 +1,5 @@
<div class="btn-group w-100" ngbDropdown role="group" [popperOptions]="popperOptions" [placement]="placement">
<button class="btn btn-sm" id="dropdown{{title}}" ngbDropdownToggle [ngClass]="createdDateTo || createdDateFrom ? 'btn-primary' : 'btn-outline-primary'" [disabled]="disabled">
<button class="btn btn-sm" id="dropdown{{title}}" ngbDropdownToggle [ngClass]="createdDateTo || createdDateFrom ? 'btn-primary' : 'btn-outline-primary'" [disabled]="disabled" [aria-label]="title">
<i-bs width="1em" height="1em" name="calendar-event-fill"></i-bs><div class="d-none d-sm-inline ms-1">{{title}}</div>
<pngx-clearable-badge [selected]="isActive" (cleared)="reset()"></pngx-clearable-badge><span class="visually-hidden">selected</span>
</button>
@@ -9,7 +9,7 @@
<div class="list-group-item d-flex p-2 select-item" role="menuitem">
<div class="selected-icon">
@if (createdRelativeDate) {
<a class="text-light focus-variants" href="javascript:void(0)" (click)="clearCreatedRelativeDate()">
<a class="text-light focus-variants" href="javascript:void(0)" (click)="clearCreatedRelativeDate()" aria-label="Clear created relative date" i18n-aria-label>
<i-bs width="1em" height="1em" name="check" class="variant-unfocused text-dark"></i-bs>
<i-bs width="1em" height="1em" name="x" class="variant-focused text-primary"></i-bs>
</a>
@@ -33,7 +33,7 @@
<div class="list-group-item d-flex p-2" role="menuitem">
<div class="selected-icon">
@if (createdDateFrom) {
<a class="text-light focus-variants" href="javascript:void(0)" (click)="clearCreatedFrom()">
<a class="text-light focus-variants" href="javascript:void(0)" (click)="clearCreatedFrom()" aria-label="Clear created from date" i18n-aria-label>
<i-bs width="1em" height="1em" name="check" class="variant-unfocused"></i-bs>
<i-bs width="1em" height="1em" name="x" class="variant-focused text-primary"></i-bs>
</a>
@@ -43,7 +43,7 @@
<span class="input-group-text w-25 small text-muted" i18n>From</span>
<input class="form-control small" [placeholder]="datePlaceHolder" (dateSelect)="onChangeDebounce()" (change)="onChangeDebounce()" (keypress)="onKeyPress($event)"
maxlength="10" [(ngModel)]="createdDateFrom" ngbDatepicker #createdDateFromPicker="ngbDatepicker" [footerTemplate]="createdFromFooterTemplate">
<button class="btn btn-outline-secondary" (click)="createdDateFromPicker.toggle()" type="button">
<button class="btn btn-outline-secondary" (click)="createdDateFromPicker.toggle()" type="button" aria-label="Open created from date picker" i18n-aria-label>
<i-bs width="1em" height="1em" name="calendar"></i-bs>
</button>
<ng-template #createdFromFooterTemplate>
@@ -57,7 +57,7 @@
<div class="list-group-item d-flex p-2" role="menuitem">
<div class="selected-icon">
@if (createdDateTo) {
<a class="text-light focus-variants" href="javascript:void(0)" (click)="clearCreatedTo()">
<a class="text-light focus-variants" href="javascript:void(0)" (click)="clearCreatedTo()" aria-label="Clear created to date" i18n-aria-label>
<i-bs width="1em" height="1em" name="check" class="variant-unfocused"></i-bs>
<i-bs width="1em" height="1em" name="x" class="variant-focused text-primary"></i-bs>
</a>
@@ -67,7 +67,7 @@
<span class="input-group-text w-25 small text-muted" i18n>To</span>
<input class="form-control small" [placeholder]="datePlaceHolder" (dateSelect)="onChangeDebounce()" (change)="onChangeDebounce()" (keypress)="onKeyPress($event)"
maxlength="10" [(ngModel)]="createdDateTo" ngbDatepicker #createdDateToPicker="ngbDatepicker" [footerTemplate]="createdToFooterTemplate">
<button class="btn btn-outline-secondary" (click)="createdDateToPicker.toggle()" type="button">
<button class="btn btn-outline-secondary" (click)="createdDateToPicker.toggle()" type="button" aria-label="Open created to date picker" i18n-aria-label>
<i-bs width="1em" height="1em" name="calendar"></i-bs>
</button>
<ng-template #createdToFooterTemplate>
@@ -85,7 +85,7 @@
<div class="list-group-item d-flex p-2 select-item" role="menuitem">
<div class="selected-icon">
@if (addedRelativeDate) {
<a class="text-light focus-variants" href="javascript:void(0)" (click)="clearAddedRelativeDate()">
<a class="text-light focus-variants" href="javascript:void(0)" (click)="clearAddedRelativeDate()" aria-label="Clear added relative date" i18n-aria-label>
<i-bs width="1em" height="1em" name="check" class="variant-unfocused text-dark"></i-bs>
<i-bs width="1em" height="1em" name="x" class="variant-focused text-primary"></i-bs>
</a>
@@ -109,7 +109,7 @@
<div class="list-group-item d-flex p-2" role="menuitem">
<div class="selected-icon">
@if (addedDateFrom) {
<a class="text-light focus-variants" href="javascript:void(0)" (click)="clearAddedFrom()">
<a class="text-light focus-variants" href="javascript:void(0)" (click)="clearAddedFrom()" aria-label="Clear added from date" i18n-aria-label>
<i-bs width="1em" height="1em" name="check" class="variant-unfocused"></i-bs>
<i-bs width="1em" height="1em" name="x" class="variant-focused text-primary"></i-bs>
</a>
@@ -119,7 +119,7 @@
<span class="input-group-text w-25 small text-muted" i18n>From</span>
<input class="form-control small" [placeholder]="datePlaceHolder" (dateSelect)="onChangeDebounce()" (change)="onChangeDebounce()" (keypress)="onKeyPress($event)"
maxlength="10" [(ngModel)]="addedDateFrom" ngbDatepicker #addedDateFromPicker="ngbDatepicker" [footerTemplate]="addedFromFooterTemplate">
<button class="btn btn-outline-secondary" (click)="addedDateFromPicker.toggle()" type="button">
<button class="btn btn-outline-secondary" (click)="addedDateFromPicker.toggle()" type="button" aria-label="Open added from date picker" i18n-aria-label>
<i-bs width="1em" height="1em" name="calendar"></i-bs>
</button>
<ng-template #addedFromFooterTemplate>
@@ -133,7 +133,7 @@
<div class="list-group-item d-flex p-2" role="menuitem">
<div class="selected-icon">
@if (addedDateTo) {
<a class="text-light focus-variants" href="javascript:void(0)" (click)="clearAddedTo()">
<a class="text-light focus-variants" href="javascript:void(0)" (click)="clearAddedTo()" aria-label="Clear added to date" i18n-aria-label>
<i-bs width="1em" height="1em" name="check" class="variant-unfocused"></i-bs>
<i-bs width="1em" height="1em" name="x" class="variant-focused text-primary"></i-bs>
</a>
@@ -143,7 +143,7 @@
<span class="input-group-text w-25 small text-muted" i18n>To</span>
<input class="form-control small" [placeholder]="datePlaceHolder" (dateSelect)="onChangeDebounce()" (change)="onChangeDebounce()" (keypress)="onKeyPress($event)"
maxlength="10" [(ngModel)]="addedDateTo" ngbDatepicker #addedDateToPicker="ngbDatepicker" [footerTemplate]="addedToFooterTemplate">
<button class="btn btn-outline-secondary" (click)="addedDateToPicker.toggle()" type="button">
<button class="btn btn-outline-secondary" (click)="addedDateToPicker.toggle()" type="button" aria-label="Open added to date picker" i18n-aria-label>
<i-bs width="1em" height="1em" name="calendar"></i-bs>
</button>
<ng-template #addedToFooterTemplate>
@@ -1,5 +1,5 @@
<div class="btn-group w-100" ngbDropdown role="group" (openChange)="dropdownOpenChange($event)" #dropdown="ngbDropdown" (keydown)="listKeyDown($event)" [popperOptions]="popperOptions" [autoClose]="!creating()">
<button class="btn btn-sm" id="dropdown_{{name}}" ngbDropdownToggle [ngClass]="!editing && selectionModel.selectionSize() > 0 ? 'btn-primary' : 'btn-outline-primary'" [disabled]="disabled">
<button class="btn btn-sm" id="dropdown_{{name}}" ngbDropdownToggle [ngClass]="!editing && selectionModel.selectionSize() > 0 ? 'btn-primary' : 'btn-outline-primary'" [disabled]="disabled" [aria-label]="title">
<i-bs name="{{icon}}"></i-bs><div class="d-none d-sm-inline ms-1">{{title}}</div>
@if (!editing && selectionModel.totalCount > 0) {
<pngx-clearable-badge [number]="selectionModel.totalCount" [selected]="selectionModel.selectionSize() > 0" (cleared)="reset()"></pngx-clearable-badge>
@@ -4,7 +4,7 @@
}
<div class="input-group" [class.is-invalid]="error">
<button type="button" class="input-group-text" [style.background-color]="value" (click)="colorPicker.toggle()">&nbsp;&nbsp;&nbsp;</button>
<button type="button" class="input-group-text" [style.background-color]="value" (click)="colorPicker.toggle()" aria-label="Open color picker" i18n-aria-label>&nbsp;&nbsp;&nbsp;</button>
<ng-template #popContent>
<div style="min-width: 200px;" class="pb-3">
@@ -14,7 +14,7 @@
<input #inputField class="form-control" [class.is-invalid]="error" [id]="inputId" [(ngModel)]="value" (change)="onChange(value)" [autoClose]="'outside'" [ngbPopover]="popContent" #colorPicker="ngbPopover" placement="bottom" popoverClass="shadow">
<button class="btn btn-outline-secondary" type="button" (click)="randomize()">
<button class="btn btn-outline-secondary" type="button" (click)="randomize()" aria-label="Choose a random color" i18n-aria-label>
<i-bs name="dice5"></i-bs>
</button>
@@ -74,7 +74,7 @@
class="flex-grow-1"></pngx-input-textarea>
}
}
<button type="button" class="btn btn-link text-danger" (click)="removeSelectedField.next(fieldId)">
<button type="button" class="btn btn-link text-danger" (click)="removeSelectedField.next(fieldId)" aria-label="Remove custom field" i18n-aria-label>
<i-bs name="trash"></i-bs>
</button>
</div>
@@ -13,7 +13,7 @@
<input #inputField class="form-control" [class.is-invalid]="error" [placeholder]="placeholder" [id]="inputId" maxlength="10"
(dateSelect)="onChange(value)" (change)="onChange(value)" (keypress)="onKeyPress($event)" (paste)="onPaste($event)"
name="dp" [(ngModel)]="value" ngbDatepicker #datePicker="ngbDatepicker" #datePickerContent="ngModel" [disabled]="disabled" [footerTemplate]="datePickerFooterTemplate">
<button class="btn btn-outline-secondary calendar" (click)="datePicker.toggle()" type="button" [disabled]="disabled">
<button class="btn btn-outline-secondary calendar" (click)="datePicker.toggle()" type="button" [disabled]="disabled" aria-label="Open date picker" i18n-aria-label>
<i-bs width="1.2em" height="1.2em" name="calendar"></i-bs>
</button>
<ng-template #datePickerFooterTemplate>
@@ -13,7 +13,7 @@
<div class="input-group mb-3">
<input type="text" class="form-control" [(ngModel)]="entry[0]" (change)="inputChange()" [disabled]="disabled" autocomplete="off">
<input type="text" class="form-control" [(ngModel)]="entry[1]" (change)="inputChange()" [disabled]="disabled" autocomplete="off">
<button type="button" class="btn btn-outline-secondary" (click)="removeEntry(i)">
<button type="button" class="btn btn-outline-secondary" (click)="removeEntry(i)" aria-label="Remove entry" i18n-aria-label>
<i-bs class="text-danger" name="trash"></i-bs>
</button>
</div>
@@ -50,7 +50,7 @@
</ng-template>
</ng-select>
@if (allowCreateNew && !hideAddButton) {
<button class="btn btn-outline-secondary" type="button" (click)="addItem()" [disabled]="disabled">
<button class="btn btn-outline-secondary" type="button" (click)="addItem()" [disabled]="disabled" aria-label="Create new item" i18n-aria-label>
<i-bs width="1.2em" height="1.2em" name="plus"></i-bs>
</button>
}
@@ -48,7 +48,7 @@
</ng-template>
</ng-select>
@if (allowCreate && !hideAddButton) {
<button class="btn btn-outline-secondary" type="button" (click)="createTag(null, true)" [disabled]="disabled">
<button class="btn btn-outline-secondary" type="button" (click)="createTag(null, true)" [disabled]="disabled" aria-label="Create new tag" i18n-aria-label>
<i-bs width="1.2em" height="1.2em" name="plus"></i-bs>
</button>
}
@@ -7,6 +7,15 @@ h3 {
}
}
:host {
display: block;
margin-bottom: .35rem;
}
h3 > .h6 {
color: var(--bs-secondary-color);
}
@media (min-width: 1200px) {
h3 {
min-height: 2.8rem;
@@ -1,5 +1,5 @@
<div class="btn-group w-100" ngbDropdown role="group">
<button class="btn btn-sm" id="dropdown{{title}}" ngbDropdownToggle [ngClass]="isActive ? 'btn-primary' : 'btn-outline-primary'" [disabled]="disabled">
<button class="btn btn-sm" id="dropdown{{title}}" ngbDropdownToggle [ngClass]="isActive ? 'btn-primary' : 'btn-outline-primary'" [disabled]="disabled" [aria-label]="title">
<i-bs name="person-fill-lock"></i-bs><div class="d-none d-sm-inline ms-1">{{title}}</div>
<pngx-clearable-badge [selected]="isActive" (cleared)="reset()"></pngx-clearable-badge><span class="visually-hidden">selected</span>
</button>
@@ -107,6 +107,25 @@ describe('PermissionsSelectComponent', () => {
expect(component.form.get('Tag').get('Change').disabled).toBeTruthy()
})
it('should update checkboxes when inherited permissions change', () => {
component.ngOnInit()
component.inheritedPermissions = ['documents.change_document']
component.writeValue(['delete_document'])
expect(component.form.get('Document').get('Change').value).toBeTruthy()
expect(component.form.get('Document').get('Change').disabled).toBeTruthy()
// swap for a group with a different permission, but the same number of them
component.inheritedPermissions = ['documents.view_document']
// the no-longer-inherited permission is unchecked, the explicit one is kept
expect(component.permissions).toEqual(['delete_document'])
expect(component.form.get('Document').get('Change').value).toBeFalsy()
expect(component.form.get('Document').get('Change').disabled).toBeFalsy()
expect(component.form.get('Document').get('Delete').value).toBeTruthy()
expect(component.form.get('Document').get('View').value).toBeTruthy()
expect(component.form.get('Document').get('View').disabled).toBeTruthy()
})
it('should exclude history permissions if disabled', () => {
settingsService.set(SETTINGS_KEYS.AUDITLOG_ENABLED, false)
fixture = TestBed.createComponent(PermissionsSelectComponent)
@@ -74,12 +74,22 @@ export class PermissionsSelectComponent
? inherited.map((p) => p.replace(/^\w+\./g, ''))
: []
if (this._inheritedPermissions !== newInheritedPermissions) {
this._inheritedPermissions = newInheritedPermissions
this.writeValue(this.permissions) // updates visual checks etc.
}
const changed =
newInheritedPermissions.length !== this._inheritedPermissions.length ||
newInheritedPermissions.some(
(p) => !this._inheritedPermissions.includes(p)
)
this.updateDisabledStates()
if (changed) {
// skip inherited permissions, these are the explicitly set ones
this.permissions = this.getSelectedPermissions(
this.form.getRawValue()
).filter((p) => !this._inheritedPermissions.includes(p))
this._inheritedPermissions = newInheritedPermissions
this.applyCheckedState()
} else {
this.updateDisabledStates()
}
}
inheritedWarning: string = $localize`Inherited from group`
@@ -106,20 +116,29 @@ export class PermissionsSelectComponent
}
this.permissions = permissions ?? []
const allPerms = this._inheritedPermissions.concat(this.permissions)
this.applyCheckedState()
}
allPerms.forEach((permissionStr) => {
const { actionKey, typeKey } =
this.permissionsService.getPermissionKeys(permissionStr)
// sets every checkbox from inherited + own perms
private applyCheckedState(): void {
const allPerms = new Set(
this._inheritedPermissions.concat(this.permissions)
)
if (actionKey && typeKey) {
this.form
.get(typeKey)
?.get(actionKey)
?.patchValue(true, { emitEvent: false })
}
})
this.allowedTypes.forEach((type) => {
const typeGroup = this.form.get(type)
for (const action of Object.keys(PermissionAction)) {
typeGroup.get(action)?.patchValue(
allPerms.has(
this.permissionsService.getPermissionCode(
PermissionAction[action],
PermissionType[type]
)
),
{ emitEvent: false } // don't trigger valueChanges now
)
}
if (this.typeHasAllActionsSelected(type)) {
this.typesWithAllActions.add(type)
} else {
@@ -150,26 +169,9 @@ export class PermissionsSelectComponent
ngOnInit(): void {
this.form.valueChanges.subscribe((newValue) => {
let permissions = []
Object.entries(newValue).forEach(([typeKey, typeValue]) => {
const selectedActions = Object.entries(typeValue).filter(
([actionKey, actionValue]) =>
actionValue &&
this.isActionSupported(
PermissionType[typeKey],
PermissionAction[actionKey]
)
)
selectedActions.forEach(([actionKey]) => {
permissions.push(
(PermissionType[typeKey] as string).replace(
'%s',
PermissionAction[actionKey]
)
)
})
const permissions = this.getSelectedPermissions(newValue)
Object.keys(newValue).forEach((typeKey) => {
if (this.typeHasAllActionsSelected(typeKey)) {
this.typesWithAllActions.add(typeKey)
} else {
@@ -269,6 +271,30 @@ export class PermissionsSelectComponent
return true
}
private getSelectedPermissions(formValue: object): string[] {
const permissions = []
Object.entries(formValue).forEach(([typeKey, typeValue]) => {
Object.entries(typeValue)
.filter(
([actionKey, actionValue]) =>
actionValue &&
this.isActionSupported(
PermissionType[typeKey],
PermissionAction[actionKey]
)
)
.forEach(([actionKey]) => {
permissions.push(
this.permissionsService.getPermissionCode(
PermissionAction[actionKey],
PermissionType[typeKey]
)
)
})
})
return permissions
}
private typeHasAllActionsSelected(typeKey: string): boolean {
return Object.keys(PermissionAction)
.filter((action) =>
@@ -1,5 +1,5 @@
<div class="btn-group">
<button type="button" class="btn btn-sm btn-outline-primary" (click)="clickSuggest()" [disabled]="disabled() || loading() || (suggestions() && !aiEnabled())">
<button type="button" class="btn btn-sm btn-outline-primary" (click)="clickSuggest()" [disabled]="disabled() || loading() || (suggestions() && !aiEnabled())" [aria-label]="noSuggestions ? 'No suggestions' : 'Suggest'" i18n-aria-label>
@if (loading()) {
<div class="spinner-border spinner-border-sm" role="status"></div>
} @else if (noSuggestions) {
@@ -23,7 +23,7 @@
<dd>
{{status().pngx_version}}
@if (versionMismatch()) {
<button class="btn btn-sm d-inline align-items-center btn-dark text-uppercase small" [ngbPopover]="versionPopover" triggers="click mouseenter:mouseleave">
<button class="btn btn-sm d-inline align-items-center btn-dark text-uppercase small" [ngbPopover]="versionPopover" triggers="click mouseenter:mouseleave" aria-label="View version mismatch details" i18n-aria-label>
<i-bs name="exclamation-triangle-fill" class="text-danger lh-1"></i-bs>
</button>
}
@@ -1,9 +1,7 @@
<pngx-page-header title="Dashboard" [subTitle]="subtitle" i18n-title tourAnchor="tour.dashboard">
<pngx-logo extra_classes="d-none d-md-block mt-n2" height="3rem"></pngx-logo>
</pngx-page-header>
<pngx-page-header title="Dashboard" [subTitle]="subtitle" i18n-title tourAnchor="tour.dashboard"></pngx-page-header>
<div class="row">
<div class="col-12 col-lg-8 col-xl-9 mb-4">
<div class="row dashboard-grid g-4 pb-0">
<div class="col-12 col-lg-8 col-xl-9 mb-4 dashboard-main">
<div class="row row-cols-1 g-4"
cdkDropList
[cdkDropListDisabled]="settingsService.globalDropzoneActive()"
@@ -58,7 +56,7 @@
</ng-container>
</div>
</div>
<div class="col-12 col-lg-4 col-xl-3 col-sidebar">
<div class="col-12 col-lg-4 col-xl-3 col-sidebar dashboard-aside">
<div class="row row-cols-1 g-4 mb-4 sticky-lg-top z-0">
<pngx-upload-file-widget></pngx-upload-file-widget>
<pngx-statistics-widget *pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.UISettings }"></pngx-statistics-widget>
@@ -1,3 +1,13 @@
.col-sidebar .row {
top: 3.5rem;
top: 4.75rem;
}
.dashboard-grid {
padding-bottom: 2rem;
}
@media (min-width: 1200px) {
.dashboard-main {
padding-right: 1rem;
}
}
@@ -15,7 +15,6 @@ import { SavedViewService } from 'src/app/services/rest/saved-view.service'
import { SettingsService } from 'src/app/services/settings.service'
import { ToastService } from 'src/app/services/toast.service'
import { environment } from 'src/environments/environment'
import { LogoComponent } from '../common/logo/logo.component'
import { PageHeaderComponent } from '../common/page-header/page-header.component'
import { ComponentWithPermissions } from '../with-permissions/with-permissions.component'
import { SavedViewWidgetComponent } from './widgets/saved-view-widget/saved-view-widget.component'
@@ -28,7 +27,6 @@ import { WelcomeWidgetComponent } from './widgets/welcome-widget/welcome-widget.
templateUrl: './dashboard.component.html',
styleUrls: ['./dashboard.component.scss'],
imports: [
LogoComponent,
PageHeaderComponent,
SavedViewWidgetComponent,
StatisticsWidgetComponent,
@@ -8,3 +8,14 @@
width: 0.6rem;
}
}
.list-group {
--bs-list-group-border-color: color-mix(in srgb, var(--bs-border-color) 58%, transparent);
--bs-list-group-bg: transparent;
border-radius: .6rem;
overflow: hidden;
}
.list-group-item {
padding: .7rem .8rem;
}
@@ -4,6 +4,16 @@
.btn-outline-dark {
--bs-btn-border-color: var(--bs-border-color-translucent);
border-style: dashed;
border-width: 1px;
min-height: 5.25rem;
background: color-mix(in srgb, var(--bs-primary) 4%, var(--bs-light)) !important;
&:hover,
&:focus {
border-color: var(--bs-primary);
background: color-mix(in srgb, var(--bs-primary) 9%, var(--bs-light)) !important;
}
}
.smaller {
@@ -1,5 +1,5 @@
@if (!cardless()) {
<div class="card shadow-sm bg-light fade" [class.show]="show()" cdkDrag [cdkDragDisabled]="!draggable()" cdkDragPreviewContainer="parent">
<div class="card bg-light fade" [class.show]="show()" cdkDrag [cdkDragDisabled]="!draggable()" cdkDragPreviewContainer="parent">
<div class="card-header">
<div class="d-flex justify-content-between align-items-center">
<div class="d-flex align-items-center">
@@ -2,6 +2,20 @@ i-bs {
cursor: move;
}
.card {
overflow: hidden;
border-color: color-mix(in srgb, var(--bs-border-color) 68%, transparent);
.card-header {
padding: .9rem 1rem;
border-bottom-color: color-mix(in srgb, var(--bs-border-color) 55%, transparent);
}
.card-body {
padding: 1rem;
}
}
.fade.show {
animation: pngx-entry-fade 160ms ease-out;
}
@@ -20,7 +20,7 @@
</div>
}
<button type="button" class="btn btn-sm btn-outline-danger me-md-4" (click)="delete()" [disabled]="!userIsOwner" *pngxIfPermissions="{ action: PermissionAction.Delete, type: PermissionType.Document }">
<button type="button" class="btn btn-sm btn-outline-danger me-md-4" (click)="delete()" [disabled]="!userIsOwner" *pngxIfPermissions="{ action: PermissionAction.Delete, type: PermissionType.Document }" aria-label="Delete" i18n-aria-label>
<i-bs width="1.2em" height="1.2em" name="trash"></i-bs><span class="d-none d-lg-inline ps-1" i18n>Delete</span>
</button>
@@ -35,7 +35,7 @@
/>
<div class="btn-group">
<button (click)="download()" class="btn btn-sm btn-outline-primary" [disabled]="downloading()">
<button (click)="download()" class="btn btn-sm btn-outline-primary" [disabled]="downloading()" aria-label="Download" i18n-aria-label>
@if (downloading()) {
<div class="spinner-border spinner-border-sm" role="status"></div>
} @else {
@@ -45,7 +45,7 @@
</button>
<div class="btn-group" ngbDropdown role="group">
<button class="btn btn-sm btn-outline-primary dropdown-toggle" [disabled]="downloading()" ngbDropdownToggle></button>
<button class="btn btn-sm btn-outline-primary dropdown-toggle" [disabled]="downloading()" ngbDropdownToggle aria-label="Download options" i18n-aria-label></button>
<div class="dropdown-menu shadow" ngbDropdownMenu>
@if (metadata()?.has_archive_version) {
<button ngbDropdownItem (click)="download(true)" [disabled]="downloading()" i18n>Download original</button>
@@ -62,7 +62,7 @@
</div>
<div class="ms-auto" ngbDropdown>
<button class="btn btn-sm btn-outline-primary" id="actionsDropdown" ngbDropdownToggle>
<button class="btn btn-sm btn-outline-primary" id="actionsDropdown" ngbDropdownToggle aria-label="Actions" i18n-aria-label>
<i-bs name="three-dots"></i-bs><div class="d-none d-sm-inline ms-1"><ng-container i18n>Actions</ng-container></div>
</button>
<div ngbDropdownMenu aria-labelledby="actionsDropdown" class="shadow">
@@ -91,7 +91,7 @@
</div>
<div class="ms-auto" ngbDropdown>
<button class="btn btn-sm btn-outline-primary" id="sendDropdown" ngbDropdownToggle>
<button class="btn btn-sm btn-outline-primary" id="sendDropdown" ngbDropdownToggle aria-label="Send" i18n-aria-label>
<i-bs name="send"></i-bs><div class="d-none d-sm-inline ms-1"><ng-container i18n>Send</ng-container></div>
</button>
<div ngbDropdownMenu aria-labelledby="actionsDropdown" class="shadow">
@@ -467,13 +467,6 @@ describe('DocumentDetailComponent', () => {
const docWithVersions = {
...doc,
versions: [
{
id: doc.id,
added: new Date('2024-01-01T00:00:00Z'),
version_label: 'Original',
checksum: 'aaaa',
is_root: true,
},
{
id: 10,
added: new Date('2024-01-02T00:00:00Z'),
@@ -481,6 +474,13 @@ describe('DocumentDetailComponent', () => {
checksum: 'bbbb',
is_root: false,
},
{
id: doc.id,
added: new Date('2024-01-01T00:00:00Z'),
version_label: 'Original',
checksum: 'aaaa',
is_root: true,
},
],
} as Document
@@ -1232,8 +1232,8 @@ describe('DocumentDetailComponent', () => {
metadataSpy.mockClear()
component.document().versions = [
{ id: doc.id, is_root: true },
{ id: 10, is_root: false },
{ id: doc.id, is_root: true },
] as any
jest.spyOn(documentService, 'getPreviewUrl').mockReturnValue('preview-root')
jest.spyOn(documentService, 'getThumbUrl').mockReturnValue('thumb-root')
@@ -1929,8 +1929,8 @@ describe('DocumentDetailComponent', () => {
component.documentId.set(doc.id)
component.document.set({ ...doc, versions: [] } as Document)
const updatedVersions = [
{ id: doc.id, is_root: true },
{ id: 10, is_root: false },
{ id: doc.id, is_root: true },
] as any
const openDoc = { ...doc, versions: [] } as Document
jest.spyOn(openDocumentsService, 'getOpenDocument').mockReturnValue(openDoc)
@@ -2046,8 +2046,8 @@ describe('DocumentDetailComponent', () => {
it('should include version in download and print only for non-latest selected version', () => {
initNormally()
component.document().versions = [
{ id: doc.id, is_root: true },
{ id: 10, is_root: false },
{ id: doc.id, is_root: true },
] as any
const getDownloadUrlSpy = jest
@@ -2171,6 +2171,11 @@ describe('DocumentDetailComponent', () => {
).toBe(10)
component.openEmailDocument()
expect(modalSpy).toHaveBeenCalled()
expect(
(
modalSpy.mock.results[1].value as NgbModalRef
).componentInstance.documentIds()
).toEqual([10])
})
it('should set previewText', () => {
@@ -889,13 +889,9 @@ export class DocumentDetailComponent
updateComponent(doc: Document) {
this.document.set(doc)
// Default selected version is the newest version
// Default selected version is the newest version, which the API returns first
const versions = doc.versions ?? []
this.selectedVersionId.set(
versions.length
? Math.max(...versions.map((version) => version.id))
: doc.id
)
this.selectedVersionId.set(versions.length ? versions[0].id : doc.id)
this.previewLoaded.set(false)
this.requiresPassword = false
this.updateFormForCustomFields()
@@ -1441,7 +1437,8 @@ export class DocumentDetailComponent
if (!versions.length || !this.selectedVersionId()) {
return null
}
const latestVersionId = Math.max(...versions.map((version) => version.id))
// The API returns versions newest first
const latestVersionId = versions[0].id
return this.selectedVersionId() === latestVersionId
? null
: this.selectedVersionId()
@@ -1976,7 +1973,9 @@ export class DocumentDetailComponent
const modal = this.modalService.open(EmailDocumentDialogComponent, {
backdrop: 'static',
})
modal.componentInstance.documentIds.set([this.document().id])
modal.componentInstance.documentIds.set([
this.selectedVersionId() ?? this.document().id,
])
modal.componentInstance.hasArchiveVersion.set(
this.metadata()?.has_archive_version ??
!!this.document()?.archived_file_name
@@ -0,0 +1,18 @@
<div class="modal-header">
<h4 class="modal-title" i18n>Add existing document as version</h4>
<button type="button" class="btn-close" aria-label="Close" (click)="cancel()"></button>
</div>
<div class="modal-body">
<pngx-input-document-link
[(ngModel)]="selectedDocumentIDs"
[parentDocumentID]="rootDocumentID"
[minimal]="true"
placeholder="Search for a document"
i18n-placeholder
></pngx-input-document-link>
<div class="form-text mt-2" i18n>Select one document to add as a version.</div>
</div>
<div class="modal-footer">
<button type="button" class="btn btn-outline-secondary" (click)="cancel()" [disabled]="!buttonsEnabled()" i18n>Cancel</button>
<button type="button" class="btn btn-primary" (click)="confirm()" [disabled]="!buttonsEnabled() || selectedDocumentIDs.length !== 1" i18n>Add version</button>
</div>
@@ -0,0 +1,73 @@
import { ComponentFixture, TestBed } from '@angular/core/testing'
import { NgbActiveModal } from '@ng-bootstrap/ng-bootstrap'
import { DocumentService } from 'src/app/services/rest/document.service'
import { AddExistingDocumentVersionDialogComponent } from './add-existing-document-version-dialog.component'
describe('AddExistingDocumentVersionDialogComponent', () => {
let component: AddExistingDocumentVersionDialogComponent
let fixture: ComponentFixture<AddExistingDocumentVersionDialogComponent>
let activeModal: jest.Mocked<Pick<NgbActiveModal, 'dismiss'>>
beforeEach(async () => {
activeModal = { dismiss: jest.fn() }
await TestBed.configureTestingModule({
imports: [AddExistingDocumentVersionDialogComponent],
providers: [
{
provide: NgbActiveModal,
useValue: activeModal,
},
{
provide: DocumentService,
useValue: {},
},
],
}).compileComponents()
fixture = TestBed.createComponent(AddExistingDocumentVersionDialogComponent)
component = fixture.componentInstance
component.rootDocumentID = 3
fixture.detectChanges()
})
it('should emit the single selected document', () => {
const emitSpy = jest.spyOn(component.confirmClicked, 'emit')
component.selectedDocumentIDs = [20]
component.confirm()
expect(emitSpy).toHaveBeenCalledWith(20)
})
it('should require exactly one selected document', () => {
const emitSpy = jest.spyOn(component.confirmClicked, 'emit')
component.selectedDocumentIDs = [20, 21]
component.confirm()
expect(emitSpy).not.toHaveBeenCalled()
})
it('should dismiss on cancel', () => {
component.cancel()
expect(activeModal.dismiss).toHaveBeenCalled()
})
it('should re-render the buttons when they are toggled from outside', async () => {
const cancelButton: HTMLButtonElement = fixture.nativeElement.querySelector(
'.modal-footer button'
)
expect(cancelButton.disabled).toBeFalsy()
// No detectChanges: the dropdown toggling this from a request callback is
// all that happens, and nothing else schedules a render for the modal
component.buttonsEnabled.set(false)
await fixture.whenStable()
expect(cancelButton.disabled).toBeTruthy()
component.buttonsEnabled.set(true)
await fixture.whenStable()
expect(cancelButton.disabled).toBeFalsy()
})
})
@@ -0,0 +1,35 @@
import {
Component,
EventEmitter,
Input,
Output,
inject,
signal,
} from '@angular/core'
import { FormsModule } from '@angular/forms'
import { NgbActiveModal } from '@ng-bootstrap/ng-bootstrap'
import { DocumentLinkComponent } from 'src/app/components/common/input/document-link/document-link.component'
@Component({
selector: 'pngx-add-existing-document-version-dialog',
templateUrl: './add-existing-document-version-dialog.component.html',
imports: [DocumentLinkComponent, FormsModule],
})
export class AddExistingDocumentVersionDialogComponent {
private readonly activeModal = inject(NgbActiveModal)
@Input() rootDocumentID: number
@Output() confirmClicked = new EventEmitter<number>()
selectedDocumentIDs: number[] = []
readonly buttonsEnabled = signal(true)
confirm(): void {
if (this.selectedDocumentIDs.length !== 1) return
this.confirmClicked.emit(this.selectedDocumentIDs[0])
}
cancel(): void {
this.activeModal.dismiss()
}
}
@@ -1,7 +1,10 @@
<div class="btn-group" ngbDropdown autoClose="outside">
<button class="btn btn-sm btn-outline-secondary dropdown-toggle" ngbDropdownToggle>
<button class="btn btn-sm btn-outline-secondary dropdown-toggle" ngbDropdownToggle aria-label="Versions" i18n-aria-label>
<i-bs name="file-earmark-diff"></i-bs>
<span class="d-none d-lg-inline ps-1" i18n>Versions</span>
@if (versions.length > 1) {
<span class="badge text-bg-secondary ms-1">{{ versions.length }}</span>
}
</button>
<div class="dropdown-menu shadow" ngbDropdownMenu>
<div class="px-3 py-2 mb-2">
@@ -24,13 +27,26 @@
class="visually-hidden"
(change)="onVersionFileSelected($event)"
/>
<button
class="btn btn-sm btn-outline-secondary w-100"
(click)="versionFileInput.click()"
[disabled]="!userIsOwner || !userCanEdit"
>
<i-bs name="file-earmark-plus"></i-bs><span class="ps-1" i18n>Add new version</span>
</button>
<div class="btn-group btn-group-sm w-100">
<button
class="btn btn-sm btn-outline-secondary w-100"
(click)="versionFileInput.click()"
[disabled]="!userIsOwner || !userCanEdit"
title="Upload a new version"
i18n-title
>
<i-bs name="file-earmark-plus"></i-bs><span class="ps-1" i18n>Upload</span>
</button>
<button
class="btn btn-sm btn-outline-secondary w-100"
(click)="addExistingDocumentAsVersion()"
[disabled]="!userIsOwner || !userCanEdit"
title="Use an existing document"
i18n-title
>
<i-bs name="file-earmark"></i-bs><span class="ps-1" i18n>Existing</span>
</button>
</div>
} @else {
@switch (versionUploadState()) {
@case (UploadState.Uploading) {
@@ -128,6 +144,7 @@
i18n-confirmMessage
[disabled]="!userIsOwner || !userCanEdit"
(confirm)="deleteVersion(version.id)"
*pngxIfPermissions="{ action: PermissionAction.Delete, type: PermissionType.Document }"
>
<span class="visually-hidden" i18n>Delete version</span>
</pngx-confirm-button>
@@ -1,9 +1,16 @@
import { DatePipe } from '@angular/common'
import { SimpleChange } from '@angular/core'
import { SimpleChange, signal } from '@angular/core'
import { ComponentFixture, TestBed } from '@angular/core/testing'
import { By } from '@angular/platform-browser'
import { NgbModal } from '@ng-bootstrap/ng-bootstrap'
import { NgxBootstrapIconsModule, allIcons } from 'ngx-bootstrap-icons'
import { Subject, of, throwError } from 'rxjs'
import { DocumentVersionInfo } from 'src/app/data/document'
import {
PermissionAction,
PermissionsService,
PermissionType,
} from 'src/app/services/permissions.service'
import { DocumentService } from 'src/app/services/rest/document.service'
import { SettingsService } from 'src/app/services/settings.service'
import { ToastService } from 'src/app/services/toast.service'
@@ -19,12 +26,20 @@ describe('DocumentVersionDropdownComponent', () => {
let documentService: jest.Mocked<
Pick<
DocumentService,
'deleteVersion' | 'getVersions' | 'uploadVersion' | 'updateVersionLabel'
| 'deleteVersion'
| 'getVersions'
| 'mergeDocumentsAsVersions'
| 'uploadVersion'
| 'updateVersionLabel'
>
>
let toastService: jest.Mocked<Pick<ToastService, 'showError' | 'showInfo'>>
let finished$: Subject<{ taskId: string }>
let failed$: Subject<{ taskId: string; message?: string }>
let modalService: jest.Mocked<Pick<NgbModal, 'open'>>
let permissionsService: jest.Mocked<
Pick<PermissionsService, 'currentUserCan'>
>
beforeEach(async () => {
finished$ = new Subject<{ taskId: string }>()
@@ -32,13 +47,18 @@ describe('DocumentVersionDropdownComponent', () => {
documentService = {
deleteVersion: jest.fn(),
getVersions: jest.fn(),
mergeDocumentsAsVersions: jest.fn(),
uploadVersion: jest.fn(),
updateVersionLabel: jest.fn(),
}
modalService = { open: jest.fn() }
toastService = {
showError: jest.fn(),
showInfo: jest.fn(),
}
permissionsService = {
currentUserCan: jest.fn().mockReturnValue(true),
}
await TestBed.configureTestingModule({
imports: [
@@ -61,6 +81,14 @@ describe('DocumentVersionDropdownComponent', () => {
provide: ToastService,
useValue: toastService,
},
{
provide: NgbModal,
useValue: modalService,
},
{
provide: PermissionsService,
useValue: permissionsService,
},
{
provide: WebsocketStatusService,
useValue: {
@@ -131,6 +159,31 @@ describe('DocumentVersionDropdownComponent', () => {
)
})
it('should not show version delete buttons without document delete permission', () => {
fixture.destroy()
permissionsService.currentUserCan.mockReturnValue(false)
fixture = TestBed.createComponent(DocumentVersionDropdownComponent)
component = fixture.componentInstance
component.documentId = 3
component.selectedVersionId = 3
component.userIsOwner = true
component.userCanEdit = true
component.versions = [
{ id: 3, is_root: true, checksum: 'aaaa' },
{ id: 10, is_root: false, checksum: 'bbbb' },
]
fixture.detectChanges()
expect(permissionsService.currentUserCan).toHaveBeenCalledWith(
PermissionAction.Delete,
PermissionType.Document
)
expect(
fixture.debugElement.queryAll(By.css('pngx-confirm-button'))
).toHaveLength(0)
})
it('beginEditingVersion should set active row and draft label', () => {
component.userCanEdit = true
component.userIsOwner = true
@@ -222,9 +275,10 @@ describe('DocumentVersionDropdownComponent', () => {
})
it('onVersionFileSelected should upload and update versions after websocket success', () => {
// Newest first, as the API returns them
const versions: DocumentVersionInfo[] = [
{ id: 3, is_root: true, checksum: 'aaaa' },
{ id: 20, is_root: false, checksum: 'cccc' },
{ id: 3, is_root: true, checksum: 'aaaa' },
]
const file = new File(['test'], 'new-version.pdf', {
type: 'application/pdf',
@@ -323,4 +377,45 @@ describe('DocumentVersionDropdownComponent', () => {
expect(component.editingVersionId).toBeNull()
expect(component.versionLabelDraft).toEqual('')
})
it('addExistingDocumentAsVersion should merge with a label and refresh versions', () => {
const confirmClicked = new Subject<number>()
const modal = {
componentInstance: {
rootDocumentID: null,
buttonsEnabled: signal(true),
confirmClicked,
},
close: jest.fn(),
}
modalService.open.mockReturnValue(modal as any)
documentService.mergeDocumentsAsVersions.mockReturnValue(of({} as any))
// Newest first, as the API returns them. The merged document has a lower id
// than the root, which is the whole point of merging an existing document.
const versions: DocumentVersionInfo[] = [
{ id: 2, is_root: false, checksum: 'cccc' },
{ id: 3, is_root: true, checksum: 'aaaa' },
]
documentService.getVersions.mockReturnValue(of({ id: 3, versions } as any))
component.newVersionLabel = ' Imported '
const versionsEmitSpy = jest.spyOn(component.versionsUpdated, 'emit')
const selectedEmitSpy = jest.spyOn(component.versionSelected, 'emit')
component.addExistingDocumentAsVersion()
expect(modal.componentInstance.rootDocumentID).toEqual(3)
confirmClicked.next(2)
expect(documentService.mergeDocumentsAsVersions).toHaveBeenCalledWith(
[3, 2],
3,
'Imported'
)
expect(documentService.updateVersionLabel).not.toHaveBeenCalled()
expect(documentService.getVersions).toHaveBeenCalledWith(3)
expect(versionsEmitSpy).toHaveBeenCalledWith(versions)
expect(selectedEmitSpy).toHaveBeenCalledWith(2)
expect(component.newVersionLabel).toEqual('')
expect(modal.close).toHaveBeenCalled()
expect(toastService.showInfo).toHaveBeenCalled()
})
})
@@ -11,7 +11,7 @@ import {
SimpleChanges,
} from '@angular/core'
import { FormsModule } from '@angular/forms'
import { NgbDropdownModule } from '@ng-bootstrap/ng-bootstrap'
import { NgbDropdownModule, NgbModal } from '@ng-bootstrap/ng-bootstrap'
import { NgxBootstrapIconsModule } from 'ngx-bootstrap-icons'
import { merge, of, Subject } from 'rxjs'
import {
@@ -25,6 +25,7 @@ import {
tap,
} from 'rxjs/operators'
import { DocumentVersionInfo } from 'src/app/data/document'
import { IfPermissionsDirective } from 'src/app/directives/if-permissions.directive'
import { CustomDatePipe } from 'src/app/pipes/custom-date.pipe'
import { DocumentService } from 'src/app/services/rest/document.service'
import { ToastService } from 'src/app/services/toast.service'
@@ -33,6 +34,8 @@ import {
WebsocketStatusService,
} from 'src/app/services/websocket-status.service'
import { ConfirmButtonComponent } from '../../common/confirm-button/confirm-button.component'
import { ComponentWithPermissions } from '../../with-permissions/with-permissions.component'
import { AddExistingDocumentVersionDialogComponent } from './add-existing-document-version-dialog/add-existing-document-version-dialog.component'
@Component({
selector: 'pngx-document-version-dropdown',
@@ -43,11 +46,15 @@ import { ConfirmButtonComponent } from '../../common/confirm-button/confirm-butt
NgbDropdownModule,
NgxBootstrapIconsModule,
ConfirmButtonComponent,
IfPermissionsDirective,
SlicePipe,
CustomDatePipe,
],
})
export class DocumentVersionDropdownComponent implements OnChanges, OnDestroy {
export class DocumentVersionDropdownComponent
extends ComponentWithPermissions
implements OnChanges, OnDestroy
{
UploadState = UploadState
@Input() documentId: number
@@ -69,6 +76,7 @@ export class DocumentVersionDropdownComponent implements OnChanges, OnDestroy {
private readonly documentsService = inject(DocumentService)
private readonly toastService = inject(ToastService)
private readonly websocketStatusService = inject(WebsocketStatusService)
private readonly modalService = inject(NgbModal)
private readonly destroy$ = new Subject<void>()
private readonly documentChange$ = new Subject<void>()
@@ -256,11 +264,10 @@ export class DocumentVersionDropdownComponent implements OnChanges, OnDestroy {
.subscribe({
next: (doc) => {
if (uploadDocumentId !== this.documentId) return
if (doc?.versions) {
if (doc?.versions?.length) {
this.versionsUpdated.emit(doc.versions)
this.versionSelected.emit(
Math.max(...doc.versions.map((version) => version.id))
)
// The API returns versions newest first
this.versionSelected.emit(doc.versions[0].id)
this.clearVersionUploadStatus()
}
},
@@ -278,6 +285,55 @@ export class DocumentVersionDropdownComponent implements OnChanges, OnDestroy {
})
}
addExistingDocumentAsVersion(): void {
const modal = this.modalService.open(
AddExistingDocumentVersionDialogComponent,
{ backdrop: 'static' }
)
const dialog =
modal.componentInstance as AddExistingDocumentVersionDialogComponent
dialog.rootDocumentID = this.documentId
dialog.confirmClicked
.pipe(takeUntil(this.destroy$), takeUntil(this.documentChange$))
.subscribe((existingDocumentID) => {
dialog.buttonsEnabled.set(false)
const versionLabel = this.newVersionLabel?.trim()
this.documentsService
.mergeDocumentsAsVersions(
[this.documentId, existingDocumentID],
this.documentId,
versionLabel
)
.pipe(
switchMap(() => this.documentsService.getVersions(this.documentId)),
first(),
finalize(() => dialog.buttonsEnabled.set(true)),
takeUntil(this.destroy$),
takeUntil(this.documentChange$)
)
.subscribe({
next: (document) => {
if (document?.versions?.length) {
this.versionsUpdated.emit(document.versions)
// The API returns versions newest first
this.versionSelected.emit(document.versions[0].id)
}
this.newVersionLabel = ''
modal.close()
this.toastService.showInfo(
$localize`Existing document added as a version.`
)
},
error: (error) => {
this.toastService.showError(
$localize`Error adding existing document as a version`,
error
)
},
})
})
}
clearVersionUploadStatus(): void {
this.versionUploadState.set(UploadState.Idle)
this.versionUploadError.set(null)
@@ -1,6 +1,6 @@
<h6>
<button type="button" class="btn btn-outline-secondary btn-sm me-2"
(click)="expand = !expand">
(click)="expand = !expand" aria-label="Toggle document metadata" i18n-aria-label>
@if (!expand) {
<i-bs width="1.2em" height="1.2em" name="caret-down"></i-bs>
}
@@ -74,7 +74,7 @@
</pngx-filterable-dropdown>
}
<div class="btn-group">
<button type="button" class="btn btn-sm btn-outline-primary me-2" (click)="setPermissions()" [disabled]="!userOwnsAll || !userCanEditAll">
<button type="button" class="btn btn-sm btn-outline-primary me-2" (click)="setPermissions()" [disabled]="!userOwnsAll || !userCanEditAll" aria-label="Permissions" i18n-aria-label>
<i-bs name="person-fill-lock"></i-bs><div class="d-none d-sm-inline ms-1"><ng-container i18n>Permissions</ng-container></div>
</button>
</div>
@@ -82,7 +82,7 @@
<div class="d-flex align-items-center gap-2 ms-auto">
<div class="btn-toolbar">
<div ngbDropdown>
<button class="btn btn-sm btn-outline-primary" id="dropdownSelect" [disabled]="!userCanEdit && !userCanAdd" ngbDropdownToggle>
<button class="btn btn-sm btn-outline-primary" id="dropdownSelect" [disabled]="!userCanEdit && !userCanAdd" ngbDropdownToggle aria-label="Actions" i18n-aria-label>
<i-bs name="three-dots"></i-bs><div class="d-none d-sm-inline ms-1"><ng-container i18n>Actions</ng-container></div>
</button>
<div ngbDropdownMenu aria-labelledby="dropdownSelect" class="shadow">
@@ -95,6 +95,9 @@
<button ngbDropdownItem (click)="mergeSelected()" [disabled]="!userCanAdd || list.allSelected || list.selectedCount < 2">
<i-bs name="journals" class="me-1"></i-bs><ng-container i18n>Merge</ng-container>
</button>
<button ngbDropdownItem (click)="mergeSelectedAsVersions()" [disabled]="!userOwnsAll || !userCanEditAll || !userCanDelete || list.allSelected || list.selectedCount < 2">
<i-bs name="journal-bookmark-fill" class="me-1"></i-bs><ng-container i18n>Merge as versions</ng-container>
</button>
</div>
</div>
</div>
@@ -104,6 +107,8 @@
id="dropdownSend"
ngbDropdownToggle
[disabled]="disabled || !canSendSelection"
aria-label="Send"
i18n-aria-label
>
<i-bs name="send"></i-bs><div class="d-none d-sm-inline ms-1"><ng-container i18n>Send</ng-container>
</div>
@@ -124,7 +129,7 @@
</div>
</div>
<div class="btn-group btn-group-sm">
<button class="btn btn-sm btn-outline-primary" [disabled]="awaitingDownload()" (click)="downloadSelected()">
<button class="btn btn-sm btn-outline-primary" [disabled]="awaitingDownload()" (click)="downloadSelected()" aria-label="Download" i18n-aria-label>
@if (!awaitingDownload()) {
<i-bs name="arrow-down"></i-bs>
}
@@ -136,7 +141,7 @@
<div class="d-none d-sm-inline ms-1"><ng-container i18n>Download</ng-container></div>
</button>
<div ngbDropdown class="me-2 d-flex btn-group" role="group">
<button type="button" class="btn btn-sm btn-outline-primary dropdown-toggle-split rounded-end" ngbDropdownToggle></button>
<button type="button" class="btn btn-sm btn-outline-primary dropdown-toggle-split rounded-end" ngbDropdownToggle aria-label="Download options" i18n-aria-label></button>
<div ngbDropdownMenu aria-labelledby="dropdownSelect" class="shadow">
<form [formGroup]="downloadForm" class="px-3 py-1">
<p class="mb-1" i18n>Include:</p>
@@ -1,5 +1,5 @@
.dropdown-toggle-split {
--bs-border-radius: .25rem;
--bs-border-radius: .45rem;
}
.dropdown-menu{
@@ -1248,6 +1248,89 @@ describe('BulkEditorComponent', () => {
expect(documentListViewService.selected.size).toEqual(0)
})
it('should support merging documents as versions', () => {
let modal: NgbModalRef
modalService.activeInstances.subscribe((m) => (modal = m[0]))
jest.spyOn(permissionsService, 'currentUserCan').mockReturnValue(true)
jest
.spyOn(documentListViewService, 'documents', 'get')
.mockReturnValue([{ id: 3 }, { id: 4 }])
jest.spyOn(documentService, 'getFew').mockReturnValue(
of({
all: [3, 4],
count: 2,
results: [
{ id: 3, title: 'Document 3' },
{ id: 4, title: 'Document 4' },
],
})
)
jest
.spyOn(documentListViewService, 'selected', 'get')
.mockReturnValue(new Set([3, 4]))
jest
.spyOn(permissionsService, 'currentUserHasObjectPermissions')
.mockReturnValue(true)
jest
.spyOn(permissionsService, 'currentUserOwnsObject')
.mockReturnValue(true)
const mergeAsVersionsSpy = jest
.spyOn(documentService, 'mergeDocumentsAsVersions')
.mockReturnValue(of(true))
const toastInfoSpy = jest.spyOn(toastService, 'showInfo')
fixture.detectChanges()
component.mergeSelectedAsVersions()
expect(modal).not.toBeUndefined()
modal.componentInstance.rootDocumentID.set(4)
modal.componentInstance.confirm()
expect(mergeAsVersionsSpy).toHaveBeenCalledWith([3, 4], 4)
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=50&ordering=-created&truncate_content=true&include_selection_data=true`
)
httpTestingController.match(
`${environment.apiBaseUrl}documents/?page=1&page_size=100000&fields=id`
)
expect(documentListViewService.selected.size).toEqual(0)
expect(toastInfoSpy).toHaveBeenCalledWith('Documents merged as versions.')
})
it('should not report success when merging documents as versions fails', () => {
let modal: NgbModalRef
modalService.activeInstances.subscribe((m) => (modal = m[0]))
jest.spyOn(permissionsService, 'currentUserCan').mockReturnValue(true)
jest
.spyOn(documentListViewService, 'documents', 'get')
.mockReturnValue([{ id: 3 }, { id: 4 }])
jest.spyOn(documentService, 'getFew').mockReturnValue(
of({
all: [3, 4],
count: 2,
results: [
{ id: 3, title: 'Document 3' },
{ id: 4, title: 'Document 4' },
],
})
)
jest
.spyOn(documentListViewService, 'selected', 'get')
.mockReturnValue(new Set([3, 4]))
jest
.spyOn(documentService, 'mergeDocumentsAsVersions')
.mockReturnValue(throwError(() => new Error('failed')))
const toastInfoSpy = jest.spyOn(toastService, 'showInfo')
const toastErrorSpy = jest.spyOn(toastService, 'showError')
fixture.detectChanges()
component.mergeSelectedAsVersions()
modal.componentInstance.rootDocumentID.set(4)
modal.componentInstance.confirm()
expect(toastErrorSpy).toHaveBeenCalled()
expect(toastInfoSpy).not.toHaveBeenCalled()
})
it('should support bulk download with archive, originals or both and file formatting', () => {
jest.spyOn(permissionsService, 'currentUserCan').mockReturnValue(true)
jest
@@ -50,6 +50,7 @@ import { SettingsService } from 'src/app/services/settings.service'
import { ToastService } from 'src/app/services/toast.service'
import { flattenTags } from 'src/app/utils/flatten-tags'
import { queryParamsFromFilterRules } from 'src/app/utils/query-params'
import { MergeAsVersionsConfirmDialogComponent } from '../../common/confirm-dialog/merge-as-versions-confirm-dialog/merge-as-versions-confirm-dialog.component'
import { MergeConfirmDialogComponent } from '../../common/confirm-dialog/merge-confirm-dialog/merge-confirm-dialog.component'
import { RotateConfirmDialogComponent } from '../../common/confirm-dialog/rotate-confirm-dialog/rotate-confirm-dialog.component'
import { CorrespondentEditDialogComponent } from '../../common/edit-dialog/correspondent-edit-dialog/correspondent-edit-dialog.component'
@@ -171,6 +172,13 @@ export class BulkEditorComponent
)
}
get userCanDelete(): boolean {
return this.permissionService.currentUserCan(
PermissionAction.Delete,
PermissionType.Document
)
}
ngOnInit() {
if (
this.permissionService.currentUserCan(
@@ -287,14 +295,17 @@ export class BulkEditorComponent
private executeDocumentAction(
modal: NgbModalRef,
request: Observable<any>,
options: { deleteOriginals?: boolean } = {}
options: { clearSelection?: boolean; successMessage?: string } = {}
) {
if (modal) {
modal.componentInstance.buttonsEnabled.set(false)
}
request.pipe(first()).subscribe({
next: () => {
this.handleOperationSuccess(modal, options.deleteOriginals ?? false)
this.handleOperationSuccess(modal, options.clearSelection ?? false)
if (options.successMessage) {
this.toastService.showInfo(options.successMessage)
}
},
error: (error) => this.handleOperationError(modal, error),
})
@@ -990,7 +1001,7 @@ export class BulkEditorComponent
this.executeDocumentAction(
modal,
this.documentService.mergeDocuments(mergeDialog.documentIDs(), args),
{ deleteOriginals: !!args.delete_originals }
{ clearSelection: !!args.delete_originals }
)
this.toastService.showInfo(
$localize`Merged document will be queued for consumption.`
@@ -998,6 +1009,35 @@ export class BulkEditorComponent
})
}
mergeSelectedAsVersions() {
let modal = this.modalService.open(MergeAsVersionsConfirmDialogComponent, {
backdrop: 'static',
})
const mergeDialog =
modal.componentInstance as MergeAsVersionsConfirmDialogComponent
const documentIDs = Array.from(this.list.selected)
mergeDialog.title = $localize`Merge as versions`
mergeDialog.message = $localize`The selected documents will become versions of the root document.`
mergeDialog.btnCaption = $localize`Proceed`
mergeDialog.documentIDs.set(documentIDs)
mergeDialog.rootDocumentID.set(documentIDs[0])
mergeDialog.confirmClicked
.pipe(takeUntil(this.unsubscribeNotifier))
.subscribe(() => {
this.executeDocumentAction(
modal,
this.documentService.mergeDocumentsAsVersions(
mergeDialog.documentIDs(),
mergeDialog.rootDocumentID()
),
{
clearSelection: true,
successMessage: $localize`Documents merged as versions.`,
}
)
})
}
public setCustomFieldValues(changedCustomFields: ChangedItems) {
const modal = this.modalService.open(CustomFieldsBulkEditDialogComponent, {
backdrop: 'static',
@@ -61,7 +61,7 @@
</pngx-input-textarea>
}
}
<button type="button" class="btn btn-outline-danger mb-3" (click)="removeField(field.id)">
<button type="button" class="btn btn-outline-danger mb-3" (click)="removeField(field.id)" aria-label="Remove custom field" i18n-aria-label>
<i-bs name="x"></i-bs>
</button>
</div>
@@ -154,9 +154,9 @@
<i-bs name="download"></i-bs>
</a>
} @else {
<button class="btn btn-sm btn-outline-secondary placeholder bg-secondary"></button>
<button class="btn btn-sm btn-outline-secondary placeholder bg-secondary"></button>
<button class="btn btn-sm btn-outline-secondary placeholder bg-secondary"></button>
<span class="btn btn-sm btn-outline-secondary placeholder bg-secondary" aria-hidden="true"></span>
<span class="btn btn-sm btn-outline-secondary placeholder bg-secondary" aria-hidden="true"></span>
<span class="btn btn-sm btn-outline-secondary placeholder bg-secondary" aria-hidden="true"></span>
}
</div>
</div>
@@ -1,6 +1,6 @@
<pngx-page-header [title]="getTitle()">
<div ngbDropdown class="btn-group flex-fill d-sm-none">
<button class="btn btn-sm btn-outline-primary" id="dropdownSelectMobile" ngbDropdownToggle>
<button class="btn btn-sm btn-outline-primary" id="dropdownSelectMobile" ngbDropdownToggle aria-label="Select" i18n-aria-label>
<i-bs name="text-indent-left"></i-bs><div class="d-none d-sm-inline ms-1"><ng-container i18n>Select</ng-container></div>
@if (list.hasSelection) {
<pngx-clearable-badge [selected]="list.hasSelection" [number]="list.selectedCount" (cleared)="list.selectNone()"></pngx-clearable-badge><span class="visually-hidden">selected</span>
@@ -31,7 +31,7 @@
</div>
</div>
<div ngbDropdown class="btn-group flex-fill">
<button class="btn btn-sm btn-outline-primary" id="dropdownDisplayFields" ngbDropdownToggle>
<button class="btn btn-sm btn-outline-primary" id="dropdownDisplayFields" ngbDropdownToggle aria-label="Show" i18n-aria-label>
<i-bs name="card-heading"></i-bs><div class="d-none d-sm-inline ms-1"><ng-container i18n>Show</ng-container></div>
</button>
<div ngbDropdownMenu aria-labelledby="dropdownDisplayFields" class="shadow">
@@ -61,7 +61,7 @@
</div>
<div ngbDropdown class="btn-group flex-fill">
<button class="btn btn-outline-primary btn-sm" id="dropdownBasic1" ngbDropdownToggle>
<button class="btn btn-outline-primary btn-sm" id="dropdownBasic1" ngbDropdownToggle aria-label="Sort" i18n-aria-label>
<i-bs name="arrow-down-up"></i-bs><div class="d-none d-sm-inline ms-1"><ng-container i18n>Sort</ng-container></div>
</button>
<div ngbDropdownMenu aria-labelledby="dropdownBasic1" class="shadow dropdown-menu-right">
@@ -85,8 +85,8 @@
</div>
</div>
<div class="btn-group flex-fill" *pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.SavedView }" ngbDropdown role="group">
<button class="btn btn-sm btn-outline-primary dropdown-toggle flex-fill" tourAnchor="tour.documents-views" ngbDropdownToggle>
<div class="btn-group flex-fill" *pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.SavedView }" ngbDropdown #viewsDropdown="ngbDropdown" role="group">
<button class="btn btn-sm btn-outline-primary dropdown-toggle flex-fill" tourAnchor="tour.documents-views" ngbDropdownToggle aria-label="Views" i18n-aria-label>
<i-bs name="window-stack"></i-bs><div class="d-none d-sm-inline ms-1"><ng-container i18n>Views</ng-container></div>
@if (savedViewIsModified) {
<div class="position-absolute top-0 start-100 p-2 translate-middle badge bg-secondary border border-light rounded-circle">
@@ -95,15 +95,15 @@
}
</button>
<div class="dropdown-menu shadow dropdown-menu-right" ngbDropdownMenu>
@if (!list.activeSavedViewId) {
@for (view of savedViewService.allViews; track view) {
<button ngbDropdownItem (click)="loadViewConfig(view.id)">
<i-bs class="me-2" [name]="view.icon || 'funnel'"></i-bs>{{view.name}}
</button>
}
@if (savedViewService.allViews.length > 0) {
<div class="dropdown-divider"></div>
}
@if (viewsDropdown.isOpen() && !list.activeSavedViewId && savedViewService.allViews.length > 0) {
<div class="views-list overflow-y-auto">
@for (view of savedViewService.allViews; track view.id) {
<button ngbDropdownItem (click)="loadViewConfig(view.id)">
<i-bs class="me-2" [name]="view.icon || 'funnel'"></i-bs>{{view.name}}
</button>
}
</div>
<div class="dropdown-divider"></div>
}
@if (list.activeSavedViewId && activeSavedViewCanChange) {
@@ -157,7 +157,7 @@
</div>
</ng-template>
<div tourAnchor="tour.documents">
<div class="mt-3 mb-n2" tourAnchor="tour.documents">
<ng-container *ngTemplateOutlet="pagination"></ng-container>
</div>
@@ -56,11 +56,11 @@ $paperless-card-breakpoints: (
.sticky-top {
z-index: 990; // below main navbar
top: calc(7rem - 2px); // height of navbar + search row (mobile)
top: calc(7.5rem - 2px); // height of navbar + search row (mobile)
transition: top 0.2s ease;
@media (min-width: 580px) {
top: 3.5rem; // height of navbar
top: 4.5em; // height of navbar
}
}
@@ -73,7 +73,7 @@ $paperless-card-breakpoints: (
@media (max-width: 579.98px) {
:host-context(main.mobile-search-hidden) .sticky-top {
top: calc(3.5rem - 2px); // height of navbar only when search is hidden
top: calc(4rem - 2px); // height of navbar only when search is hidden
}
}
@@ -93,4 +93,8 @@ a {
pngx-page-header .dropdown-menu {
--bs-dropdown-min-width: 12em;
.views-list {
max-height: min(400px, calc(100vh - 260px)); // leave room for the header above and the actions below
}
}
@@ -18,7 +18,7 @@
</select>
}
@if (_textFilter) {
<button class="btn btn-link btn-sm px-2 position-absolute top-0 end-0 z-10" (click)="resetTextField()">
<button class="btn btn-link btn-sm px-2 position-absolute top-0 end-0 z-10" (click)="resetTextField()" aria-label="Clear search" i18n-aria-label>
<i-bs width="1em" height="1em" name="x"></i-bs>
</button>
}
@@ -24,7 +24,7 @@
<div class="btn-toolbar gap-2">
<div class="btn-group d-block d-sm-none">
<div ngbDropdown container="body" class="d-inline-block">
<button type="button" class="btn btn-link" id="actionsMenuMobile" (click)="$event.stopPropagation()" ngbDropdownToggle>
<button type="button" class="btn btn-link" id="actionsMenuMobile" (click)="$event.stopPropagation()" ngbDropdownToggle aria-label="Actions" i18n-aria-label>
<i-bs name="three-dots-vertical"></i-bs>
</button>
<div ngbDropdownMenu aria-labelledby="actionsMenuMobile">
@@ -7,7 +7,7 @@
>
@if (activeManagementList) {
<div ngbDropdown class="btn-group flex-fill d-sm-none">
<button class="btn btn-sm btn-outline-primary" id="dropdownSelectMobile" ngbDropdownToggle>
<button class="btn btn-sm btn-outline-primary" id="dropdownSelectMobile" ngbDropdownToggle aria-label="Select" i18n-aria-label>
<i-bs name="text-indent-left"></i-bs><div class="d-none d-sm-inline ms-1"><ng-container i18n>Select</ng-container></div>
@if (activeManagementList.hasSelection) {
<pngx-clearable-badge [selected]="activeManagementList.hasSelection" [number]="activeManagementList.selectedCount" (cleared)="activeManagementList.selectNone()"></pngx-clearable-badge><span class="visually-hidden">selected</span>
@@ -1,7 +1,7 @@
<div class="row mb-3">
<div class="col mb-2 mb-xl-0">
<div class="form-inline d-flex align-items-center">
<label class="text-muted me-2 mb-0" for="managementNameFilter" i18n>Filter by:</label>
<label class="me-2 mb-0" for="managementNameFilter" i18n>Filter by:</label>
<input id="managementNameFilter" class="form-control form-control-sm flex-fill w-auto" type="text" autofocus [(ngModel)]="nameFilter" (keyup)="onNameFilterKeyUp($event)" placeholder="Name" i18n-placeholder>
</div>
</div>
@@ -9,7 +9,7 @@
<div class="col-auto mb-2 mb-xl-0">
<div class="form-inline d-flex align-items-center">
<div class="input-group input-group-sm w-auto d-none d-md-flex">
<label class="input-group-text border-0" for="managementPageSize" i18n>Show:</label>
<label class="input-group-text bg-transparent border-0" for="managementPageSize" i18n>Show:</label>
</div>
<div class="input-group input-group-sm w-auto me-3">
<select id="managementPageSize" class="form-select form-select-sm small" [(ngModel)]="pageSize">
@@ -113,7 +113,7 @@
<div class="btn-toolbar gap-2">
<div class="btn-group d-block d-sm-none">
<div ngbDropdown container="body" class="d-inline-block">
<button type="button" class="btn btn-link" id="actionsMenuMobile" (click)="$event.stopPropagation()" ngbDropdownToggle>
<button type="button" class="btn btn-link" id="actionsMenuMobile" (click)="$event.stopPropagation()" ngbDropdownToggle aria-label="Actions" i18n-aria-label>
<i-bs name="three-dots-vertical"></i-bs>
</button>
<div ngbDropdownMenu aria-labelledby="actionsMenuMobile">
@@ -58,7 +58,7 @@
<div class="col">
<div class="btn-group d-block d-sm-none">
<div ngbDropdown container="body" class="d-inline-block">
<button type="button" class="btn btn-link" id="actionsMenuMobile" (click)="$event.stopPropagation()" ngbDropdownToggle>
<button type="button" class="btn btn-link" id="actionsMenuMobile" (click)="$event.stopPropagation()" ngbDropdownToggle aria-label="Actions" i18n-aria-label>
<i-bs name="three-dots-vertical"></i-bs>
</button>
<div ngbDropdownMenu aria-labelledby="actionsMenuMobile">
@@ -146,7 +146,7 @@
<div class="col-3">
<div class="btn-group d-block d-sm-none">
<div ngbDropdown container="body" class="d-inline-block">
<button type="button" class="btn btn-link" id="actionsMenuMobile" (click)="$event.stopPropagation()" ngbDropdownToggle>
<button type="button" class="btn btn-link" id="actionsMenuMobile" (click)="$event.stopPropagation()" ngbDropdownToggle aria-label="Actions" i18n-aria-label>
<i-bs name="three-dots-vertical"></i-bs>
</button>
<div ngbDropdownMenu aria-labelledby="actionsMenuMobile">
@@ -47,7 +47,7 @@
<div class="btn-group d-block d-sm-none">
<div ngbDropdown container="body" class="d-inline-block">
<button type="button" class="btn btn-link" id="actionsMenuMobile" (click)="$event.stopPropagation()" ngbDropdownToggle>
<button type="button" class="btn btn-link" id="actionsMenuMobile" (click)="$event.stopPropagation()" ngbDropdownToggle aria-label="Actions" i18n-aria-label>
<i-bs name="three-dots-vertical"></i-bs>
</button>
<div ngbDropdownMenu aria-labelledby="actionsMenuMobile">
@@ -316,6 +316,34 @@ describe(`DocumentService`, () => {
})
})
it('should call appropriate api endpoint for merging documents as versions', () => {
const ids = [1, 2, 3]
subscription = service.mergeDocumentsAsVersions(ids, 2).subscribe()
const req = httpTestingController.expectOne(
`${environment.apiBaseUrl}${endpoint}/merge_as_versions/`
)
expect(req.request.method).toEqual('POST')
expect(req.request.body).toEqual({
documents: ids,
root_document_id: 2,
})
})
it('should include an optional label when merging one document as a version', () => {
const ids = [1, 2]
subscription = service
.mergeDocumentsAsVersions(ids, 2, 'Imported')
.subscribe()
const req = httpTestingController.expectOne(
`${environment.apiBaseUrl}${endpoint}/merge_as_versions/`
)
expect(req.request.body).toEqual({
documents: ids,
root_document_id: 2,
version_label: 'Imported',
})
})
it('should call appropriate api endpoint for edit pdf', () => {
const ids = [1]
const args = { operations: [{ page: 1, rotate: 90, doc: 0 }] }
@@ -374,6 +374,18 @@ export class DocumentService extends AbstractPaperlessService<Document> {
})
}
mergeDocumentsAsVersions(
ids: number[],
rootDocumentId: number,
versionLabel?: string
) {
return this.http.post(this.getResourceUrl(null, 'merge_as_versions'), {
documents: ids,
root_document_id: rootDocumentId,
...(versionLabel ? { version_label: versionLabel } : {}),
})
}
editPdfDocuments(ids: number[], request: EditPdfDocumentsRequest) {
return this.http.post(this.getResourceUrl(null, 'edit_pdf'), {
documents: ids,
@@ -438,6 +438,21 @@ describe('ConsumerStatusService', () => {
expect(updated).toBeTruthy()
})
it('should ignore keep-alive heartbeat messages from the server', () => {
let updated = false
let deleted = false
websocketStatusService.onDocumentUpdated().subscribe(() => (updated = true))
websocketStatusService.onDocumentDeleted().subscribe(() => (deleted = true))
websocketStatusService.connect()
server.send({ type: WebsocketStatusType.HEARTBEAT })
expect(updated).toBeFalsy()
expect(deleted).toBeFalsy()
expect(websocketStatusService.getConsumerStatus()).toHaveLength(0)
websocketStatusService.disconnect()
})
it('should ignore document updated events the user cannot view', () => {
let updated = false
websocketStatusService.onDocumentUpdated().subscribe(() => {
@@ -11,6 +11,7 @@ export enum WebsocketStatusType {
STATUS_UPDATE = 'status_update',
DOCUMENTS_DELETED = 'documents_deleted',
DOCUMENT_UPDATED = 'document_updated',
HEARTBEAT = 'heartbeat',
}
// see ProgressStatusOptions in src/documents/plugins/helpers.py
@@ -207,6 +208,10 @@ export class WebsocketStatusService {
case WebsocketStatusType.STATUS_UPDATE:
this.handleProgressUpdate(messageData as WebsocketProgressMessage)
break
case WebsocketStatusType.HEARTBEAT:
// keep-alive from the server, see paperless.consumers.StatusConsumer
break
}
}
}
+2
View File
@@ -115,6 +115,7 @@ import {
house,
inbox,
infoCircle,
journalBookmarkFill,
journals,
link,
listNested,
@@ -361,6 +362,7 @@ const icons = {
house,
inbox,
infoCircle,
journalBookmarkFill,
journals,
link,
listNested,
+86 -2
View File
@@ -1,7 +1,7 @@
@use "sass:meta";
// bs variables
$grid-gutter-width: 1.5rem;
$border-radius: .375rem;
$border-radius: .425rem;
$btn-border-width: var(--bs-border-width);
$form-file-button-bg: var(--bs-body-bg);
@@ -68,15 +68,44 @@ body {
--pngx-body-font-size: 0.875rem;
font-size: var(--pngx-body-font-size);
height: 100vh;
letter-spacing: -0.005em;
}
* {
transition: background-color 0.3s ease, border-color 0.3s ease;
}
.card,
.dropdown-menu,
.modal-content,
.popover {
--bs-card-border-color: color-mix(in srgb, var(--bs-border-color) 72%, transparent);
border-radius: .55rem;
}
.card {
box-shadow: 0 1px 2px rgba(0, 0, 0, .04), 0 8px 24px rgba(0, 0, 0, .035);
}
.btn {
--bs-btn-border-radius: .425rem;
--bs-border-radius-sm: .425rem;
font-weight: 500;
}
.form-control,
.form-select,
.input-group-text {
border-radius: .425rem;
}
.pagination, .input-group {
--bs-border-radius-sm: .425rem;
}
@media(min-width: 768px) {
.col-slim {
padding-left: calc(50px + $grid-gutter-width) !important;
padding-left: calc(56px + $grid-gutter-width) !important;
}
}
@@ -607,10 +636,63 @@ table.table {
color: var(--bs-body-color);
}
// Tighten horizontal spacing in modals on small viewports
@media (max-width: 575.98px) {
.modal {
--bs-modal-margin: 0.25rem;
--bs-modal-header-padding-x: 0.5rem;
--bs-modal-header-padding: var(--bs-modal-header-padding-y) var(--bs-modal-header-padding-x);
.modal-body {
padding-inline: 0.5rem;
}
.modal-footer {
padding-inline: 0.25rem;
}
.accordion {
--bs-accordion-btn-padding-x: 0.75rem;
--bs-accordion-body-padding-x: 0.5rem;
}
.card {
--bs-card-spacer-x: 0.5rem;
}
.list-group {
--bs-list-group-item-padding-x: 0.5rem;
}
}
}
// Wider-width modals on small viewports when the content is wide (e.g. landscape)
@media (min-width: 576px) and (max-height: 700px) {
.modal {
--bs-modal-margin: 0.5rem;
}
.modal-dialog {
max-width: min(var(--bs-modal-width), calc(100% - 1rem));
}
.modal-lg {
--bs-modal-width: 800px;
}
.modal-xl {
--bs-modal-width: 1140px;
}
}
.toast {
--bs-toast-max-width: var(--pngx-toast-max-width);
}
.alert {
--bs-border-radius: .425rem;
}
.alert-primary {
--bs-alert-color: var(--bs-primary);
--bs-alert-bg: var(--pngx-primary-faded);
@@ -742,6 +824,8 @@ code {
--bs-accordion-bg: var(--bs-light);
--bs-accordion-active-color: var(--bs-primary);
--bs-accordion-active-bg: var(--pngx-bg-alt);
--bs-border-radius: .425rem;
--bs-accordion-inner-border-radius: calc(.425rem - 1px);
}
.accordion-button::after {
+3 -3
View File
@@ -20,7 +20,7 @@
--pngx-primary-darken-27: hsl(var(--pngx-primary), calc(var(--pngx-primary-lightness) - 27%));
--pngx-success-darken-10: hsl(152, 69%, 11%); // based on success #198754
--pngx-bg-alt: #fff;
--pngx-bg-darker: var(--bs-gray-100);
--pngx-bg-darker: #f4f6f5;
--pngx-bg-alt2: var(--bs-gray-200); // #e9ecef
--pngx-bg-disabled: #f7f7f7;
--pngx-card-hover-border: var(--bs-tertiary-color);
@@ -91,7 +91,7 @@ $form-check-radio-checked-bg-image-dark: url("data:image/svg+xml,%3csvg xmlns='h
--pngx-body-color-accent: #{$text-color-dark-bg-accent};
--pngx-bg-alt: #242529;
--pngx-bg-alt2: #232323;
--pngx-bg-darker: #101216;
--pngx-bg-darker: #121315;
--pngx-bg-disabled: var(--pngx-bg-alt);
--pngx-card-hover-border: var(--bs-border-color);
--pngx-focus-alpha: 0.6;
@@ -107,7 +107,7 @@ $form-check-radio-checked-bg-image-dark: url("data:image/svg+xml,%3csvg xmlns='h
--bs-light-rgb: 28, 28, 31;
--bs-info: var(--pngx-bg-alt);
--bs-info-rgb: 36, 36, 39;
--bs-border-color: #47494f;
--bs-border-color: #34373d;
--bs-tertiary-bg: var(--pngx-bg-darker);
--bs-dark-border-subtle: var(--pngx-bg-darker);
--bs-border-color-translucent: rgba(0, 0, 0, .175); // override bs
+114
View File
@@ -12,6 +12,7 @@ from celery import group
from celery import shared_task
from django.conf import settings
from django.db import transaction
from django.db.models import Max
from django.db.models import Q
from django.utils import timezone
@@ -30,6 +31,7 @@ from documents.permissions import set_permissions_for_object
from documents.plugins.helpers import DocumentsStatusManager
from documents.tasks import bulk_update_documents
from documents.tasks import consume_file
from documents.tasks import remove_document_from_index
from documents.tasks import update_document_content_maybe_archive_file
from documents.versioning import get_latest_version_for_root
from documents.versioning import get_root_document
@@ -39,6 +41,9 @@ if TYPE_CHECKING:
from django.contrib.auth.models import User
if settings.AUDIT_LOG_ENABLED:
from auditlog.models import LogEntry
logger: logging.Logger = logging.getLogger("paperless.bulk_edit")
SourceMode = Literal["latest_version", "explicit_selection"]
@@ -612,6 +617,115 @@ def merge(
return "OK"
def merge_as_versions(
doc_ids: list[int],
*,
root_document_id: int,
version_label: str | None = None,
user: User | None = None,
) -> Literal["OK"]:
with transaction.atomic():
documents = list(
# Ordered by pk so concurrent merges take the row locks in the same order
Document.objects.select_for_update()
.filter(id__in=doc_ids)
.order_by("id")
.defer("content"),
)
documents_by_id = {document.id: document for document in documents}
source_ids = [doc_id for doc_id in doc_ids if doc_id != root_document_id]
root_document = documents_by_id[root_document_id]
next_version_index = (
Document.global_objects.filter(
root_document_id=root_document_id,
).aggregate(max_index=Max("version_index"))["max_index"]
or 0
)
# A version gives up its ASN
source_asns = [
documents_by_id[source_id].archive_serial_number
for source_id in source_ids
if documents_by_id[source_id].archive_serial_number is not None
]
updated_fields = ["root_document", "version_index", "archive_serial_number"]
if version_label is not None:
updated_fields.append("version_label")
for source_id in source_ids:
next_version_index += 1
source_document = documents_by_id[source_id]
source_document.root_document_id = root_document.pk
source_document.version_index = next_version_index
source_document.archive_serial_number = None
if version_label is not None:
source_document.version_label = version_label
# bulk_update and not save() to avoid post_save now
Document.objects.bulk_update(
[documents_by_id[source_id] for source_id in source_ids],
updated_fields,
)
root_updates = {"modified": timezone.now()}
if source_asns and root_document.archive_serial_number is None:
# If a version had one, hand the ASN over, the same as merge() does
root_updates["archive_serial_number"] = source_asns.pop(0)
logger.info(
f"Document {root_document.id} took archive serial number "
f"{root_updates['archive_serial_number']} from a document merged into it",
)
if source_asns:
logger.warning(
f"Archive serial number(s) {source_asns} were removed by merging "
f"those documents as versions of document {root_document.id}",
)
Document.objects.filter(pk=root_document.pk).update(**root_updates)
if settings.AUDIT_LOG_ENABLED:
# update() doesn't fire auditlog signals, so manual
LogEntry.objects.log_create(
instance=root_document,
changes={"Merged As Versions": ["None", source_ids]},
action=LogEntry.Action.UPDATE,
actor=user,
additional_data={
"reason": "Merged as versions",
"version_ids": source_ids,
},
)
# One batch rather than a task each
from documents.search import SearchIndexLockError
from documents.search import get_backend
try:
with get_backend().batch_update() as batch:
for source_id in source_ids:
batch.remove(source_id)
except SearchIndexLockError:
logger.error(
f"Search index lock exhausted removing {source_ids}, "
f"scheduling deferred index removal",
)
for source_id in source_ids:
remove_document_from_index.apply_async(args=[source_id], countdown=60)
bulk_update_documents.apply_async(
kwargs={"document_ids": [root_document_id]},
headers={"trigger_source": PaperlessTask.TriggerSource.SYSTEM},
)
# And as far as the frontend is concerned, they're deleted
status_mgr = DocumentsStatusManager()
status_mgr.send_documents_deleted(source_ids)
return "OK"
def split(
doc_ids: list[int],
pages: list[list[int]],
+6 -4
View File
@@ -372,6 +372,10 @@ class Document(SoftDeleteModel, ModelWithOwner): # type: ignore[django-manager-
For version documents, this is always the document's own content.
If the queryset already annotated ``effective_content``, that value is used.
"""
# Here to avoid circular import
from documents.versioning import sort_versions_newest_first
from documents.versioning import versions_newest_first
if hasattr(self, "effective_content"):
return getattr(self, "effective_content")
@@ -388,12 +392,10 @@ class Document(SoftDeleteModel, ModelWithOwner): # type: ignore[django-manager-
# Empty list means prefetch ran and found no versions — use own content.
if not prefetched_versions:
return self.content
latest_prefetched = max(prefetched_versions, key=lambda doc: doc.id)
return latest_prefetched.content
return sort_versions_newest_first(prefetched_versions)[0].content
latest_version_content = (
Document.objects.filter(root_document=self)
.order_by("-id")
versions_newest_first(Document.objects.filter(root_document=self))
.values_list("content", flat=True)
.first()
)
+10 -2
View File
@@ -6,13 +6,20 @@ from documents.search._backend import TantivyRelevanceList
from documents.search._backend import WriteBatch
from documents.search._backend import get_backend
from documents.search._backend import reset_backend
from documents.search._errors import InvalidDateQuery
from documents.search._errors import InvalidNumberQuery
from documents.search._errors import MultipleSearchQueryErrors
from documents.search._errors import QueryTooLongError
from documents.search._errors import SearchQueryError
from documents.search._errors import search_query_error_messages
from documents.search._schema import needs_rebuild
from documents.search._schema import wipe_index
from documents.search._translate import InvalidDateQuery
from documents.search._translate import SearchQueryError
__all__ = [
"InvalidDateQuery",
"InvalidNumberQuery",
"MultipleSearchQueryErrors",
"QueryTooLongError",
"SearchHit",
"SearchIndexLockError",
"SearchMode",
@@ -23,5 +30,6 @@ __all__ = [
"get_backend",
"needs_rebuild",
"reset_backend",
"search_query_error_messages",
"wipe_index",
]
+72 -34
View File
@@ -20,12 +20,8 @@ from typing import cast
import filelock
import tantivy
from django.conf import settings
from django.contrib.contenttypes.models import ContentType
from django.utils.timezone import get_current_timezone
from guardian.shortcuts import get_groups_with_perms
from guardian.shortcuts import get_users_with_perms
from documents.search._query import build_permission_filter
from documents.search._query import extract_cjk_text
from documents.search._query import parse_simple_text_highlight_query
from documents.search._query import parse_simple_text_query
@@ -43,6 +39,7 @@ from documents.utils import QuerySetStream
from documents.utils import identity
if TYPE_CHECKING:
from collections.abc import Iterable
from collections.abc import Iterator
from collections.abc import Sequence
from pathlib import Path
@@ -266,11 +263,7 @@ class WriteBatch:
if self._lock is not None:
self._lock.release()
def add_or_update(
self,
document: Document,
effective_content: str | None = None,
) -> None:
def add_or_update(self, document: Document) -> None:
"""
Add or update a document in the batch.
@@ -280,11 +273,9 @@ class WriteBatch:
Args:
document: Django Document instance to index
effective_content: Override document.content for indexing (used when
re-indexing with newer OCR text from document versions)
"""
self.remove(document.pk)
doc = self._backend._build_tantivy_doc(document, effective_content)
doc = self._backend._build_tantivy_doc(document)
self._writer.add_document(doc)
def remove(self, doc_id: int) -> None:
@@ -294,6 +285,47 @@ class WriteBatch:
)
def build_permission_filter(
schema: tantivy.Schema,
user: AbstractUser,
viewer_group_ids: Iterable[int] = (),
) -> tantivy.Query:
"""
Build a query filter for user document permissions.
Creates a query that matches only documents visible to the specified user
according to paperless-ngx permission rules:
- Public documents (no owner) are visible to all users
- Private documents are visible to their owner
- Documents explicitly shared with the user are visible
- Documents shared with one of the user's current groups are visible
Args:
schema: Tantivy schema for field validation
user: User to check permissions for
viewer_group_ids: Current group memberships for the user
Returns:
Tantivy query that filters results to visible documents
"""
owner_any = tantivy.Query.exists_query("owner_id")
no_owner = tantivy.Query.boolean_query(
[
(tantivy.Occur.Must, tantivy.Query.all_query()),
(tantivy.Occur.MustNot, owner_any),
],
)
owned = tantivy.Query.term_query(schema, "owner_id", user.pk)
shared = tantivy.Query.term_query(schema, "viewer_id", user.pk)
group_shared = [
tantivy.Query.term_query(schema, "viewer_group_id", group_id)
for group_id in viewer_group_ids
]
return tantivy.Query.disjunction_max_query(
[no_owner, owned, shared, *group_shared],
)
class TantivyBackend:
"""
Tantivy search backend with explicit lifecycle management.
@@ -425,18 +457,20 @@ class TantivyBackend:
def _build_tantivy_doc(
self,
document: Document,
effective_content: str | None = None,
viewer_ids: list[int] | None = None,
viewer_group_ids: list[int] | None = None,
) -> tantivy.Document:
"""Build a tantivy Document from a Django Document instance.
``effective_content`` overrides ``document.content`` for indexing
used when re-indexing a root document with a newer version's OCR text.
A root document is indexed with its effective content, i.e. the newest
version's OCR text, so it is never indexed with its own outdated text.
Annotate the queryset with ``annotate_effective_content`` when indexing
more than a couple of documents, to resolve that without a query each.
"""
content = (
effective_content if effective_content is not None else document.content
)
from guardian.shortcuts import get_groups_with_perms
from guardian.shortcuts import get_users_with_perms
content = document.get_effective_content() or ""
doc = tantivy.Document()
@@ -465,7 +499,6 @@ class TantivyBackend:
doc.add_text("correspondent_sort", document.correspondent.name)
if cjk_corr := extract_cjk_text(document.correspondent.name):
doc.add_text("bigram_correspondent", cjk_corr)
doc.add_unsigned("correspondent_id", document.correspondent_id)
# Document type
if document.document_type:
@@ -473,12 +506,10 @@ class TantivyBackend:
doc.add_text("type_sort", document.document_type.name)
if cjk_type := extract_cjk_text(document.document_type.name):
doc.add_text("bigram_document_type", cjk_type)
doc.add_unsigned("document_type_id", document.document_type_id)
# Storage path
if document.storage_path:
doc.add_text("storage_path", document.storage_path.name)
doc.add_unsigned("storage_path_id", document.storage_path_id)
# Tags — collect names for autocomplete in the same pass
tag_names: list[str] = []
@@ -486,12 +517,13 @@ class TantivyBackend:
doc.add_text("tag", tag.name)
if cjk_tag := extract_cjk_text(tag.name):
doc.add_text("bigram_tag", cjk_tag)
doc.add_unsigned("tag_id", tag.pk)
tag_names.append(tag.name)
# Notes — JSON for structured queries (notes.user:alice, notes.note:text).
# notes_text is a plain-text companion for snippet/highlight generation;
# tantivy's SnippetGenerator does not support JSON fields.
# tantivy's SnippetGenerator does not support JSON fields. It is not in
# _DEFAULT_SEARCH_FIELDS, so an unqualified query never searches it: a
# note matches through the JSON field or not at all.
num_notes = 0
note_texts: list[str] = []
for note in document.notes.all():
@@ -507,8 +539,9 @@ class TantivyBackend:
if note_texts:
doc.add_text("notes_text", " ".join(note_texts))
# Custom fields JSON for structured queries (custom_fields.name:x, custom_fields.value:y),
# companion text field for default full-text search.
# Custom fields: JSON for structured queries (custom_fields.name:x,
# custom_fields.value:y). There is no companion text field here, unlike
# notes: custom field values are reachable only through the JSON field.
for cfi in document.custom_fields.all():
search_value = cfi.value_for_search
# Skip fields where there is no value yet
@@ -584,11 +617,7 @@ class TantivyBackend:
return doc
def add_or_update(
self,
document: Document,
effective_content: str | None = None,
) -> None:
def add_or_update(self, document: Document) -> None:
"""
Add or update a single document with file locking.
@@ -601,12 +630,11 @@ class TantivyBackend:
Args:
document: Django Document instance to index
effective_content: Override document.content for indexing
"""
self._ensure_open()
try:
with self.batch_update(lock_timeout=_LOCK_TIMEOUT_SECONDS) as batch:
batch.add_or_update(document, effective_content)
batch.add_or_update(document)
except SearchIndexLockError:
logger.error(
"Search index lock exhausted for document %d after %d attempts; "
@@ -680,7 +708,17 @@ class TantivyBackend:
user_query = self._parse_query(query, search_mode)
highlight_query = user_query
if search_mode is SearchMode.TEXT:
highlight_query = parse_simple_text_highlight_query(self._index, query)
try:
highlight_query = parse_simple_text_highlight_query(
self._index,
query,
)
except ValueError:
logger.debug(
"Skipping simple text highlight query: token string is not "
"valid tantivy query syntax: %r",
query,
)
# For notes_text snippet generation, we need a query that targets the
# notes_text field directly. user_query may contain JSON-field terms
@@ -1027,7 +1065,6 @@ class TantivyBackend:
):
doc = self._build_tantivy_doc(
document,
document.get_effective_content(),
viewer_ids=viewer_ids,
viewer_group_ids=viewer_group_ids,
)
@@ -1095,6 +1132,7 @@ def _bulk_get_viewer_permissions(
"""
from collections import defaultdict
from django.contrib.contenttypes.models import ContentType
from guardian.models import GroupObjectPermission
from guardian.models import UserObjectPermission
-171
View File
@@ -1,171 +0,0 @@
from __future__ import annotations
from datetime import UTC
from datetime import date
from datetime import datetime
from datetime import timedelta
from typing import TYPE_CHECKING
from typing import Final
from dateutil.relativedelta import relativedelta
if TYPE_CHECKING:
from datetime import tzinfo
_DATE_ONLY_FIELDS = frozenset({"created"})
_TODAY: Final[str] = "today"
_YESTERDAY: Final[str] = "yesterday"
_PREVIOUS_WEEK: Final[str] = "previous week"
_THIS_MONTH: Final[str] = "this month"
_PREVIOUS_MONTH: Final[str] = "previous month"
_THIS_YEAR: Final[str] = "this year"
_PREVIOUS_YEAR: Final[str] = "previous year"
_PREVIOUS_QUARTER: Final[str] = "previous quarter"
_DATE_KEYWORDS = frozenset(
{
_TODAY,
_YESTERDAY,
_PREVIOUS_WEEK,
_THIS_MONTH,
_PREVIOUS_MONTH,
_THIS_YEAR,
_PREVIOUS_YEAR,
_PREVIOUS_QUARTER,
},
)
def _fmt(dt: datetime) -> str:
"""Format a datetime as an ISO 8601 UTC string for use in Tantivy range queries."""
return dt.astimezone(UTC).strftime("%Y-%m-%dT%H:%M:%SZ")
def _iso_range(lo: datetime, hi: datetime) -> str:
"""
Format a half-open ``[lo TO hi)`` range in ISO 8601 for Tantivy query syntax.
``hi`` is always the exclusive ceiling of a computed period (the start of
the *next* day/week/month/quarter/year), so the closing bracket must be
the Tantivy exclusive-range brace ``}`` rather than ``]`` otherwise the
first instant of the following period (e.g. the 1st of next month) is
incorrectly included in the match.
"""
return f"[{_fmt(lo)} TO {_fmt(hi)}}}"
def _quarter_start(d: date) -> date:
"""Return the first day of the calendar quarter containing ``d``."""
return date(d.year, ((d.month - 1) // 3) * 3 + 1, 1)
def _midnight(d: date, tz: tzinfo) -> datetime:
"""Convert a calendar date at local-timezone midnight to a UTC datetime."""
return datetime(d.year, d.month, d.day, tzinfo=tz).astimezone(UTC)
def _keyword_bounds(keyword: str, tz: tzinfo) -> tuple[date, date]:
"""
Map a relative date keyword to ``(start, exclusive_end)`` calendar dates.
``tz`` only determines what "today" is; the caller decides how the returned
dates become UTC datetime boundaries (date-only vs. local-midnight offset).
"""
today = datetime.now(tz).date()
if keyword == _TODAY:
return today, today + timedelta(days=1)
if keyword == _YESTERDAY:
return today - timedelta(days=1), today
if keyword == _PREVIOUS_WEEK:
this_monday = today - timedelta(days=today.weekday())
return this_monday - timedelta(weeks=1), this_monday
if keyword == _THIS_MONTH:
first = today.replace(day=1)
return first, first + relativedelta(months=1)
if keyword == _PREVIOUS_MONTH:
this_first = today.replace(day=1)
return this_first - relativedelta(months=1), this_first
if keyword == _THIS_YEAR:
return date(today.year, 1, 1), date(today.year + 1, 1, 1)
if keyword == _PREVIOUS_YEAR:
return date(today.year - 1, 1, 1), date(today.year, 1, 1)
if keyword == _PREVIOUS_QUARTER:
this_quarter = _quarter_start(today)
return this_quarter - relativedelta(months=3), this_quarter
raise ValueError(f"Unknown keyword: {keyword}")
def _date_only_range(keyword: str, tz: tzinfo) -> str:
"""
For `created` (DateField): use the local calendar date, converted to
midnight UTC boundaries. No offset arithmetic date only.
"""
start, end = _keyword_bounds(keyword, tz)
lo = datetime(start.year, start.month, start.day, tzinfo=UTC)
hi = datetime(end.year, end.month, end.day, tzinfo=UTC)
return _iso_range(lo, hi)
def _datetime_range(keyword: str, tz: tzinfo) -> str:
"""
For `added` / `modified` (DateTimeField, stored as UTC): convert local day
boundaries to UTC full offset arithmetic required.
"""
start, end = _keyword_bounds(keyword, tz)
return _iso_range(_midnight(start, tz), _midnight(end, tz))
def _precision_bounds(digits: str) -> tuple[date, date] | None:
"""
Map a 4/6/8-digit date token to (start, exclusive_end) calendar dates.
YYYY -> whole year, YYYYMM -> whole month, YYYYMMDD -> single day.
Returns None for any unparsable or out-of-range value (e.g. month 23),
so callers can emit a no-match clause instead of erroring (Whoosh parity).
"""
try:
if len(digits) == 4:
year = int(digits)
return date(year, 1, 1), date(year + 1, 1, 1)
if len(digits) == 6:
year, month = int(digits[:4]), int(digits[4:6])
start = date(year, month, 1)
end = date(year + 1, 1, 1) if month == 12 else date(year, month + 1, 1)
return start, end
if len(digits) == 8:
start = date(int(digits[:4]), int(digits[4:6]), int(digits[6:8]))
return start, start + timedelta(days=1)
except ValueError:
return None
return None
def _utc_bounds_for_field(
field: str,
start: date,
end: date,
tz: tzinfo,
) -> tuple[datetime, datetime]:
"""
Convert calendar-date bounds to UTC datetimes per the field's storage type.
For DateField (``created``) the bounds are UTC midnight (no offset). For
DateTimeField (``added``/``modified``) the bounds are local-tz midnight
converted to UTC, matching how each field is indexed.
"""
if field in _DATE_ONLY_FIELDS:
return (
datetime(start.year, start.month, start.day, tzinfo=UTC),
datetime(end.year, end.month, end.day, tzinfo=UTC),
)
return (
datetime(start.year, start.month, start.day, tzinfo=tz).astimezone(UTC),
datetime(end.year, end.month, end.day, tzinfo=tz).astimezone(UTC),
)
def _field_range_from_dates(field: str, start: date, end: date, tz: tzinfo) -> str:
"""Build a Tantivy ``field:[lo TO hi]`` ISO range from calendar-date bounds."""
lo, hi = _utc_bounds_for_field(field, start, end, tz)
return f"{field}:{_iso_range(lo, hi)}"
+71
View File
@@ -0,0 +1,71 @@
from __future__ import annotations
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from collections.abc import Sequence
class SearchQueryError(ValueError):
"""
Base for user-fixable search query errors.
Carries a message safe to surface to the user (no internal details). The
view layer catches this and returns an HTTP 400, so any future subclass
gets the same treatment.
"""
class InvalidDateQuery(SearchQueryError):
"""Raised when a date field value or range bound cannot be parsed."""
def __init__(self, field: str | None, value: str | None) -> None:
self.field = field
self.value = value
super().__init__(f"Invalid date value {value!r} for field {field!r}.")
class InvalidNumberQuery(SearchQueryError):
"""Raised when a numeric field value or range bound cannot be parsed."""
def __init__(self, field: str | None, value: str | None) -> None:
self.field = field
self.value = value
super().__init__(f"Invalid numeric value {value!r} for field {field!r}.")
class QueryTooLongError(SearchQueryError):
"""Raised when a query string exceeds the maximum allowed length.
whoosh-compat's fieldname tagger is O(n^2) in plain word characters, so an
unbounded query is a CPU-exhaustion vector against a single request
handler. This is a hard boundary, not a validation nicety.
"""
def __init__(self, length: int, limit: int) -> None:
self.length = length
self.limit = limit
super().__init__(
f"The search query is too long ({length} characters). "
f"The maximum allowed length is {limit} characters.",
)
class MultipleSearchQueryErrors(SearchQueryError):
"""Aggregates every user-fixable error from one parse, not just the first."""
def __init__(self, errors: Sequence[SearchQueryError]) -> None:
self.errors = tuple(errors)
super().__init__("; ".join(str(e) for e in self.errors))
def search_query_error_messages(e: SearchQueryError) -> list[str]:
"""The user-facing message list for a SearchQueryError.
Every offending value's message, not just the first, so the user can
fix them all in one round-trip. Shared by every view that maps
SearchQueryError to an HTTP 400.
"""
if isinstance(e, MultipleSearchQueryErrors):
return [str(sub) for sub in e.errors]
return [str(e)]
+42
View File
@@ -0,0 +1,42 @@
from __future__ import annotations
from whoosh_compat import FieldKind
from whoosh_compat import FieldSpec
from whoosh_compat import SubpathSpec
# Internal-only schema fields with no query-syntax meaning of their own
# (sort shadow fields, bigram CJK fields, simple_title/simple_content,
# autocomplete_word, notes_text) are NOT represented here, they are
# declared in _schema.py's field_descriptors().
#
# analyzer/pattern_normalizer are deliberately left at FieldSpec's default
# (None): they're language-specific and only meaningful to whoosh-compat's
# parser, so _registry.py attaches them per-language via dataclasses.replace()
# rather than PUBLIC_FIELDS declaring them itself. _schema.py only reads
# name/kind/fast and never sees the analyzer at all.
PUBLIC_FIELDS: tuple[FieldSpec, ...] = (
FieldSpec("title", FieldKind.TEXT),
FieldSpec("content", FieldKind.TEXT),
FieldSpec("correspondent", FieldKind.TEXT),
FieldSpec("document_type", FieldKind.TEXT, aliases=("type",)),
FieldSpec("storage_path", FieldKind.TEXT, aliases=("path",)),
FieldSpec("original_filename", FieldKind.TEXT),
FieldSpec("tag", FieldKind.TEXT, comma_values=True),
FieldSpec("checksum", FieldKind.KEYWORD),
FieldSpec("asn", FieldKind.U64, fast=True),
FieldSpec("page_count", FieldKind.U64, fast=True),
FieldSpec("num_notes", FieldKind.U64, fast=True),
FieldSpec("created", FieldKind.DATE, date_only=True, fast=True),
FieldSpec("modified", FieldKind.DATETIME, fast=True),
FieldSpec("added", FieldKind.DATETIME, fast=True),
FieldSpec(
"notes",
FieldKind.JSON,
subpaths={"user": SubpathSpec(), "note": SubpathSpec(default=True)},
),
FieldSpec(
"custom_fields",
FieldKind.JSON,
subpaths={"name": SubpathSpec(), "value": SubpathSpec(default=True)},
),
)
+450 -145
View File
@@ -6,22 +6,30 @@ from typing import Final
import regex
import tantivy
import whoosh_compat as wc
from django.conf import settings
from whoosh_compat.emitters.tantivy_ import emit as tantivy_emit
from whoosh_compat.errors import Cause
from whoosh_compat.errors import Diagnostic
from whoosh_compat.errors import DiagnosticKind
from whoosh_compat.errors import QueryError
from documents.search._errors import InvalidDateQuery
from documents.search._errors import InvalidNumberQuery
from documents.search._errors import MultipleSearchQueryErrors
from documents.search._errors import SearchQueryError
from documents.search._registry import get_field_registry
from documents.search._tokenizer import simple_search_tokens
from documents.search._translate import SearchQueryError
from documents.search._translate import translate_query
if TYPE_CHECKING:
from collections.abc import Iterable
from datetime import tzinfo
from django.contrib.auth.base_user import AbstractBaseUser
logger = logging.getLogger("paperless.search")
# Maximum seconds any single regex substitution may run.
# Prevents ReDoS on adversarial user-supplied query strings.
# Maximum seconds any single regex substitution over user-supplied query text
# may run. The one remaining use is a character class, which cannot backtrack,
# so the bound is an upper limit on that substitution's cost, not the ReDoS
# guard it was originally written as.
_REGEX_TIMEOUT: Final[float] = 1.0
# Matches CJK/Hangul characters so queries can be routed to bigram fields.
@@ -29,6 +37,64 @@ _REGEX_TIMEOUT: Final[float] = 1.0
_CJK_RE: Final = regex.compile(r"[\p{Han}\p{Hiragana}\p{Katakana}\p{Hangul}]+")
def _user_facing_emit_message(d: Diagnostic) -> str:
"""A user-safe message for an emit-time QueryError's Diagnostic.
Built from the Diagnostic's structured fields (kind, field), never from
d.message: whoosh-compat documents that as developer/log output with no
stability guarantee, and PATTERN_TOO_COMPLEX embeds the raw backend
error text in it.
"""
field = str(d.field) if d.field is not None else None
if d.kind is DiagnosticKind.EXISTS_REQUIRES_FAST:
return f"Existence searches (field:*) are not supported for field {field!r}."
if d.kind is DiagnosticKind.TEXT_RANGE:
return f"Range searches are not supported for field {field!r}."
if d.kind is DiagnosticKind.PATTERN_TOO_COMPLEX:
return f"The wildcard pattern for field {field!r} is too complex."
if d.kind is DiagnosticKind.SCHEMA_FIELD_MISSING:
return f"Field {field!r} is not available in the search index."
logger.warning("Unmapped emit diagnostic %s: %s", d.kind, d.message)
return "The search query could not be executed."
def _map_emit_error(e: QueryError) -> SearchQueryError:
"""Route an emit-time QueryError by its Diagnostic's Cause.
INVALID_INPUT/UNSUPPORTED are user-input errors, exactly like a parse
diagnostic, and map to a 400. INTERNAL means a defect in whoosh-compat
or in our own AST handling, never the user's query, so the QueryError is
re-raised rather than converted, reaching the generic 500 handler instead
of blaming the query. MISCONFIGURED is deliberately both: the registry and
the index schema disagree, which only an operator can fix, so it is logged
as an error, but a request is still waiting and the query cannot run
either way, so it also returns a 400.
EXISTS_REQUIRES_FAST is the one MISCONFIGURED kind that is not a
disagreement. whoosh-compat derives it from the registry's own FieldSpec
(kind plus fast) without ever consulting the index schema, so it fires
whenever a non-fast field of a kind that cannot answer "exists" is asked
to: for us that is only the JSON fields, which field_descriptors() builds
non-fast on purpose. "notes:*" and the five other spellings of it are
ordinary user error that no operator action can clear, so they get the
400 without the alert.
"""
d = e.diagnostic
if d.cause is Cause.INTERNAL:
raise e
if (
d.cause is Cause.MISCONFIGURED
and d.kind is not DiagnosticKind.EXISTS_REQUIRES_FAST
):
logger.error(
"Search index misconfiguration for field %s (%s): %s",
d.field,
d.kind.name,
d.message,
)
return SearchQueryError(_user_facing_emit_message(d))
def _has_cjk(text: str) -> bool:
"""Return True if text contains any CJK characters."""
return bool(_CJK_RE.search(text))
@@ -37,14 +103,36 @@ def _has_cjk(text: str) -> bool:
def extract_cjk_text(text: str) -> str:
"""Join the CJK runs in ``text`` for indexing into bigram (char-ngram) fields.
Mirrors the query side (``_build_cjk_query``): only CJK runs are ever searched
against the bigram fields, so only CJK runs are worth indexing there. Latin
text fed to a character-bigram field is never matched and only bloats the
Mirrors the query side, which extracts the CJK runs of whatever it is
about to search for (the raw string in simple modes, the parsed query's
free-text tokens in query mode): only CJK runs are ever searched against
the bigram fields, so only CJK runs are worth indexing there. Latin text
fed to a character-bigram field is never matched and only bloats the
index and slows indexing/merge. Returns "" when there is no CJK text.
"""
return " ".join(_CJK_RE.findall(text))
def _parse_cjk_text(
index: tantivy.Index,
cjk_text: str,
fields: list[str],
) -> tantivy.Query | None:
"""Parse a plain CJK run string against ``fields``, or None if it won't parse."""
try:
return index.parse_query(cjk_text, fields)
except Exception:
# Broad on purpose, unlike _try_parse_fuzzy_query's narrower
# ValueError: cjk_text isn't filtered to a guaranteed-safe token
# set the way the fuzzy blend's word string is, so the exact
# failure mode tantivy could raise here isn't pinned down.
logger.debug(
"Skipping CJK search clause: could not parse CJK text: %r",
cjk_text,
)
return None
def _build_cjk_query(
index: tantivy.Index,
raw_query: str,
@@ -52,91 +140,259 @@ def _build_cjk_query(
) -> tantivy.Query | None:
"""Build a bigram-field query from the CJK runs in ``raw_query``.
Only the CJK character runs are extracted and parsed; ASCII field prefixes,
boolean operators and date keywords are discarded. This keeps the CJK clause
plain-text and consistent across query/simple modes (no leaked ``field:``
semantics, no parse failures from spaced ``-``/``+``), and avoids feeding
Latin tokens into the character-bigram matcher (which would produce spurious
matches against unrelated Latin text). Returns None when there is no CJK
text or the parse fails.
For the simple (TEXT/TITLE) modes, whose input is plain text and carries
no query grammar to respect. Only the CJK character runs are extracted, so
a stray ``field:`` prefix or ``-``/``+`` in the input can neither leak
field semantics nor fail the parse, and no Latin token reaches the
character-bigram matcher (where it would produce spurious matches against
unrelated Latin text). Returns None when there is no CJK text or the parse
fails.
"""
cjk_text = " ".join(_CJK_RE.findall(raw_query))
cjk_text = extract_cjk_text(raw_query)
if not cjk_text:
return None
return _parse_cjk_text(index, cjk_text, fields)
def _build_ast_cjk_query(
index: tantivy.Index,
ast: wc.ast.Node,
registry: wc.FieldRegistry,
) -> tantivy.Query | None:
"""Build the bigram clause of a QUERY-mode search from the parsed AST.
Same discipline as the fuzzy clause (see _try_parse_fuzzy_query): the CJK
runs come from whoosh_compat's ``free_text_tokens`` over the parsed tree,
never from the raw query string, so a term the user negated or restricted
to a field outside the default search fields contributes nothing, instead
of resurfacing as a top-level clause matching every bigram field.
``free_text_tokens`` reports no field of its own, so the tokens are
collected one default field at a time: a bare term, which the parser has
already copied onto every default field, is therefore searched across
every bigram field, while ``title:東京`` reaches ``bigram_title`` alone.
Fields whose CJK text is identical (the bare-term case) share a single
parse over all of their bigram fields at once.
Raw (``analyzed=False``) tokens are used because the bigram fields have
their own character-ngram analyzer: the default fields' word analyzers
have no useful say over a CJK run, and running them first would only
risk dropping it (remove_long) before the run is ever extracted.
Returns None when the query has no CJK free text.
"""
fields_by_text: dict[str, list[str]] = {}
for field, bigram_field in _CJK_BIGRAM_FIELDS.items():
tokens = wc.free_text_tokens(
ast,
registry=registry,
fields=[field],
analyzed=False,
)
cjk_text = extract_cjk_text(" ".join(tokens))
if cjk_text:
fields_by_text.setdefault(cjk_text, []).append(bigram_field)
clauses: list[tuple[tantivy.Occur, tantivy.Query]] = [
(tantivy.Occur.Should, query)
for cjk_text, bigram_fields in fields_by_text.items()
if (query := _parse_cjk_text(index, cjk_text, bigram_fields)) is not None
]
return _any_of(clauses) if clauses else None
# A joined fuzzy word string must stay plain words: it goes back through
# tantivy's own query parser, and the raw query text the clause collects
# routinely carries characters that parser reads as grammar (a colon, a
# bracket, a quote, a leading -). Each token is cut into its word runs and
# only those are kept, so no field syntax, pattern, range or grouping can
# reach the parser. Cutting rather than dropping the whole token is what
# keeps ordinary hyphenated, dotted and quoted input ("COVID-19",
# "hello@example.com", "tax reports") contributing to the clause at all.
_WORD_RUN_RE = regex.compile(r"\w+")
# The one piece of tantivy grammar that survives the cut: its boolean
# keywords are themselves word runs. Only these exact spellings are
# grammar there ("And"/"and" are ordinary terms), so lowercasing exactly
# these turns them back into the ordinary terms the field analyzer used to
# make of them, before the clause switched to raw text. Left alone, a
# quoted phrase would silently restructure the clause ("tax AND reports"
# becoming a conjunction) or fail to parse and drop it entirely
# ("tax AND", or "IN" anywhere).
#
# Only these words are touched: tantivy lowercases query terms with the
# field's own analyzer, and doing it ourselves first is not always the
# same operation (Python folds a final sigma to a different letter than
# tantivy does, and turns Turkish 'İ' into a sequence tantivy then splits
# in two), which would search for terms the index does not contain.
_TANTIVY_KEYWORDS: Final[frozenset[str]] = frozenset({"AND", "OR", "NOT", "IN"})
def _try_parse_fuzzy_query(
index: tantivy.Index,
ast: wc.ast.Node,
registry: wc.FieldRegistry,
) -> tantivy.Query | None:
"""Build the fuzzy blend clause from the parsed query's free-text
words, or None if it has none.
The clause is built by handing tantivy's own query parser a plain
word string (there's no clean AST-level fuzzy equivalent to
whoosh-compat's parse tree, and fuzzy matching was always an
approximate, secondary, 0.1-boosted clause). The words come from
whoosh_compat's ``free_text_tokens`` over the already-parsed AST,
never from the raw query string: raw whoosh grammar (date keywords,
``[2005 to 2009]`` ranges, bracket-class wildcards) is not tantivy
syntax, and feeding it here used to knock the fuzzy clause out for
the whole query the moment any such construct appeared alongside a
typo'd word. The helper also keeps excluded terms out: a ``NOT``'d
word must not resurface through the fuzzy clause.
Chosen trade-off: a term explicitly fielded on one of the default
search fields (``correspondent:acme``) contributes its text to the
word string UNFIELDED, so the fuzzy clause searches it across all
default fields rather than just the one the user named. That is
recall-only widening on a secondary 0.1-boosted clause the score
threshold already disciplines, accepted in exchange for never feeding
field syntax to tantivy's parser. What the word string guarantees is
exactly that: no field prefix, pattern, range, grouping or quoting
survives, and the boolean keywords that do survive (they are word
runs) are lowercased into ordinary terms; see _TANTIVY_KEYWORDS.
The words are the query's RAW text, not the analyzer's output
(``analyzed=False``), because ``index.parse_query`` analyzes whatever
it is given and analysis is not idempotent: ``universities`` stems to
``univers``, and handing that back stems it again to ``univ``, a term
the index does not contain. ``prefix=True`` hid this as over-broad
matching (``univ`` also prefixes ``unicycle``) rather than as no
matches at all. Raw text is untokenized, which is why it is cut into
word runs above rather than taken whole.
The ValueError guard stays as insurance (the word string is plain
tokens, so tantivy accepting it is expected, not assumed): on a parse
failure the fuzzy clause is skipped and the exact/CJK clauses stand,
rather than the whole query failing.
"""
tokens = wc.free_text_tokens(
ast,
registry=registry,
fields=_DEFAULT_SEARCH_FIELDS,
analyzed=False,
)
words = list(
dict.fromkeys(
word.lower() if word in _TANTIVY_KEYWORDS else word
for token in tokens
for word in _WORD_RUN_RE.findall(token)
),
)
if not words:
return None
fuzzy_text = " ".join(words)
try:
return index.parse_query(cjk_text, fields)
except Exception:
return index.parse_query(
fuzzy_text,
_DEFAULT_SEARCH_FIELDS,
field_boosts=_FIELD_BOOSTS,
fuzzy_fields={f: (True, 1, True) for f in _DEFAULT_SEARCH_FIELDS},
)
except ValueError:
logger.debug(
"Skipping fuzzy search clause: token string is not valid "
"tantivy query syntax: %r",
fuzzy_text,
)
return None
def build_permission_filter(
schema: tantivy.Schema,
user: AbstractBaseUser,
viewer_group_ids: Iterable[int] = (),
) -> tantivy.Query:
"""
Build a query filter for user document permissions.
Creates a query that matches only documents visible to the specified user
according to paperless-ngx permission rules:
- Public documents (no owner) are visible to all users
- Private documents are visible to their owner
- Documents explicitly shared with the user are visible
- Documents shared with one of the user's current groups are visible
Args:
schema: Tantivy schema for field validation
user: User to check permissions for
viewer_group_ids: Current group memberships for the user
Returns:
Tantivy query that filters results to visible documents
"""
owner_any = tantivy.Query.exists_query("owner_id")
no_owner = tantivy.Query.boolean_query(
[
(tantivy.Occur.Must, tantivy.Query.all_query()),
(tantivy.Occur.MustNot, owner_any),
],
)
owned = tantivy.Query.term_query(schema, "owner_id", user.pk)
shared = tantivy.Query.term_query(schema, "viewer_id", user.pk)
group_shared = [
tantivy.Query.term_query(schema, "viewer_group_id", group_id)
for group_id in viewer_group_ids
]
return tantivy.Query.disjunction_max_query(
[no_owner, owned, shared, *group_shared],
)
DEFAULT_SEARCH_FIELDS = [
_DEFAULT_SEARCH_FIELDS: Final[list[str]] = [
"title",
"content",
"correspondent",
"document_type",
"tag",
]
SIMPLE_SEARCH_FIELDS = ["simple_title", "simple_content"]
TITLE_SEARCH_FIELDS = ["simple_title"]
_CJK_ALL_FIELDS: Final[list[str]] = [
"bigram_content",
"bigram_title",
"bigram_correspondent",
"bigram_document_type",
"bigram_tag",
]
_SIMPLE_SEARCH_FIELDS: Final[list[str]] = ["simple_title", "simple_content"]
_TITLE_SEARCH_FIELDS: Final[list[str]] = ["simple_title"]
# The bigram (character-ngram) companion of each default search field.
_CJK_BIGRAM_FIELDS: Final[dict[str, str]] = {
field: f"bigram_{field}" for field in _DEFAULT_SEARCH_FIELDS
}
_CJK_CONTENT_FIELDS: Final[list[str]] = ["bigram_content"]
_CJK_TITLE_FIELDS: Final[list[str]] = ["bigram_title"]
_FIELD_BOOSTS = {"title": 2.0}
_SIMPLE_FIELD_BOOSTS = {"simple_title": 2.0}
def _simple_query_tokens(raw_query: str) -> list[str]:
# Tokenize and fold via the same analyzer used to index simple_title /
# simple_content, so query terms fold identically to the indexed terms
# (single source of truth for ASCII folding).
return simple_search_tokens(raw_query)
class _ConjunctiveNegations(wc.ast.Visitor[tuple["wc.ast.Node", ...]]):
"""Collect the subtrees an AST excludes from every document it matches.
A negation reached through ``And``/``AndNot``/``Require`` (and through
the required half of an ``AndMaybe``) constrains the whole query, so it
can be re-stated above the blend. ``Or`` is deliberately not descended
into: in ``invoice OR NOT secret`` the negation is one branch's own
condition, and hoisting it would throw away documents the other branch
matches. Nor is a collected subtree descended into, since a negation
inside a negation is not an exclusion.
Node types with no negation to contribute (every leaf, ``Or``) fall
through to ``generic_visit``.
"""
def generic_visit(self, node: wc.ast.Node) -> tuple[wc.ast.Node, ...]:
return ()
def visit_not(self, node: wc.ast.Not) -> tuple[wc.ast.Node, ...]:
return (node.child,)
def visit_andnot(self, node: wc.ast.AndNot) -> tuple[wc.ast.Node, ...]:
return (*self.visit(node.positive), node.negative)
def visit_and(self, node: wc.ast.And) -> tuple[wc.ast.Node, ...]:
return tuple(
negation for child in node.children for negation in self.visit(child)
)
def visit_boosted(self, node: wc.ast.Boosted) -> tuple[wc.ast.Node, ...]:
return self.visit(node.child)
def visit_andmaybe(self, node: wc.ast.AndMaybe) -> tuple[wc.ast.Node, ...]:
return self.visit(node.required)
def visit_require(self, node: wc.ast.Require) -> tuple[wc.ast.Node, ...]:
return (*self.visit(node.scored), *self.visit(node.filter_only))
def _negation_clauses(
index: tantivy.Index,
ast: wc.ast.Node,
registry: wc.FieldRegistry,
) -> list[tuple[tantivy.Occur, tantivy.Query]]:
"""MustNot clauses for everything ``ast`` excludes conjunctively.
Each excluded subtree is emitted as its own positive query and attached
with ``MustNot``, rather than emitting a negative query and hoping
tantivy accepts a bare one.
"""
try:
return [
(
tantivy.Occur.MustNot,
tantivy_emit(negation, index=index, registry=registry),
)
for negation in _ConjunctiveNegations().visit(ast)
]
except QueryError as e:
raise _map_emit_error(e) from e
def _any_of(clauses: list[tuple[tantivy.Occur, tantivy.Query]]) -> tantivy.Query:
"""Collapse a clause list: none -> empty, one -> itself (no wasted
single-clause boolean_query wrapping), many -> boolean_query(clauses)."""
if not clauses:
return tantivy.Query.empty_query()
if len(clauses) == 1:
return clauses[0][1]
return tantivy.Query.boolean_query(clauses)
def _build_simple_token_query(
@@ -168,9 +424,7 @@ def _build_simple_token_query(
query = tantivy.Query.boost_query(query, boost)
field_queries.append((tantivy.Occur.Should, query))
if len(field_queries) == 1:
return field_queries[0][1]
return tantivy.Query.boolean_query(field_queries)
return _any_of(field_queries)
def parse_user_query(
@@ -179,52 +433,53 @@ def parse_user_query(
tz: tzinfo,
) -> tantivy.Query:
"""
Parse user query through the complete preprocessing pipeline.
Parse user query through whoosh-compat, then blend in fuzzy/CJK clauses.
Transforms the raw user query through multiple stages:
1. Date keyword rewriting (today ISO 8601 ranges)
2. Query normalization (comma expansion, whitespace cleanup)
3. Tantivy parsing with field boosts
4. Optional fuzzy query blending (if ADVANCED_FUZZY_SEARCH_THRESHOLD set)
Args:
index: Tantivy index with registered tokenizers
raw_query: Original user query string
tz: Timezone for date boundary calculations
Returns:
Parsed Tantivy query ready for execution
Note:
When ADVANCED_FUZZY_SEARCH_THRESHOLD is configured, adds a low-priority
fuzzy query as a Should clause (0.1 boost) to catch approximate matches
while keeping exact matches ranked higher. The threshold value is applied
as a post-search score filter, not during query construction.
1. wc.parse() against the shared FieldRegistry (whoosh grammar -> AST).
Bare notes:/custom_fields: prefixes resolve to their default subpath
(notes.note:/custom_fields.value:) directly in the registry, via
each JSON field's SubpathSpec(default=True).
2. Any diagnostics (bad dates/numbers) map to SearchQueryError subclasses
and raise, the view returns HTTP 400 with every offending field
listed, not just the first.
3. emit() turns the AST into a tantivy.Query directly (no string
round-trip). A QueryError is routed by its Diagnostic's Cause
(_map_emit_error): a construct that parses but can't execute against
tantivy (e.g. a text-field range) is a 400, a registry/schema
mismatch is logged and a 400, and an INTERNAL defect is re-raised.
4. Optional fuzzy blend (ADVANCED_FUZZY_SEARCH_THRESHOLD) builds a
plain word string from the parsed AST's free-text tokens
(whoosh_compat.free_text_tokens) and feeds THAT to
index.parse_query, never raw_query, whose whoosh grammar (date
keywords, bracket-class wildcards, etc.) tantivy's parser rejects,
which used to silently knock the fuzzy clause out of any mixed
query (see _try_parse_fuzzy_query).
5. Optional CJK bigram clause, built from the same parsed AST for the
same reason (see _build_ast_cjk_query): a CJK term the query negated
or fielded must not resurface through it.
6. When any optional clause was added, the query's conjunctive
exclusions are restated as MustNot above the blend
(_negation_clauses): a clause built from positive terms cannot
express them, and as a bare Should it would undo them.
"""
registry = get_field_registry(settings.SEARCH_LANGUAGE)
result = wc.parse(
raw_query,
registry=registry,
default_fields=_DEFAULT_SEARCH_FIELDS,
field_boosts=_FIELD_BOOSTS,
tz=tz,
)
if result.diagnostics:
raise _diagnostics_to_error(result.diagnostics)
try:
query_str = translate_query(raw_query, tz)
except SearchQueryError:
# Intentional, user-fixable error (e.g. an unparsable date). Propagate so
# the view can return a 400 with a helpful message rather than falling
# back to the raw (still-invalid) query.
raise
except Exception: # pragma: no cover - defensive
logger.warning("Query translation failed; using raw query", exc_info=True)
query_str = raw_query
exact = tantivy_emit(result.ast, index=index, registry=registry)
except QueryError as e:
raise _map_emit_error(e) from e
exact = index.parse_query(
query_str,
DEFAULT_SEARCH_FIELDS,
field_boosts=_FIELD_BOOSTS,
)
# The standard analyzer keeps a whitespace-free CJK run as a single token,
# so substring queries can't match content/title (and long runs are dropped
# by remove_long). Route CJK queries to the bigram fields, whose ngram
# tokenizer indexes overlapping 2-grams for substring matching.
cjk_query = (
_build_cjk_query(index, raw_query, _CJK_ALL_FIELDS)
_build_ast_cjk_query(index, result.ast, registry)
if _has_cjk(raw_query)
else None
)
@@ -235,22 +490,73 @@ def parse_user_query(
threshold = settings.ADVANCED_FUZZY_SEARCH_THRESHOLD
if threshold is not None:
fuzzy = index.parse_query(
query_str,
DEFAULT_SEARCH_FIELDS,
field_boosts=_FIELD_BOOSTS,
# (prefix=True, distance=1, transposition_cost_one=True) — edit-distance fuzziness
fuzzy_fields={f: (True, 1, True) for f in DEFAULT_SEARCH_FIELDS},
)
# 0.1 boost keeps fuzzy hits ranked below exact matches (intentional)
clauses.append((tantivy.Occur.Should, tantivy.Query.boost_query(fuzzy, 0.1)))
fuzzy = _try_parse_fuzzy_query(index, result.ast, registry)
if fuzzy is not None:
clauses.append(
(tantivy.Occur.Should, tantivy.Query.boost_query(fuzzy, 0.1)),
)
if cjk_query is not None:
clauses.append((tantivy.Occur.Should, cjk_query))
if len(clauses) == 1:
return exact
return tantivy.Query.boolean_query(clauses)
# The fuzzy and CJK clauses are built from positive terms only, so as
# plain Shoulds beside the exact clause they re-admit exactly the
# documents the query excluded. Restate the exclusions once, above the
# whole blend. Redundant against the exact clause, which already
# carries them, but idempotently so.
negations = _negation_clauses(index, result.ast, registry)
if not negations:
return _any_of(clauses)
return tantivy.Query.boolean_query(
[(tantivy.Occur.Must, _any_of(clauses)), *negations],
)
# The three whoosh-compat kinds for a wildcard on a field that cannot
# carry one. d.field_kind supplies the discriminator, so naming the field's
# type needs no second trip through the registry.
_PATTERN_ON_KINDS: Final = frozenset(
{
DiagnosticKind.PATTERN_ON_NUMERIC,
DiagnosticKind.PATTERN_ON_BOOLEAN_EXISTS,
DiagnosticKind.PATTERN_ON_SUBPATH,
},
)
def _diagnostics_to_error(diagnostics: tuple[Diagnostic, ...]) -> SearchQueryError:
errors = [_single_diagnostic_to_error(d) for d in diagnostics]
return errors[0] if len(errors) == 1 else MultipleSearchQueryErrors(errors)
def _single_diagnostic_to_error(d: Diagnostic) -> SearchQueryError:
# d.field is a FieldRef, not a str: str(d.field) gives the canonical
# dotted name (an aliased query, e.g. type:, reports document_type).
field_name = str(d.field) if d.field is not None else None
if d.kind is DiagnosticKind.BAD_DATE:
return InvalidDateQuery(field_name, d.raw_value)
if d.kind is DiagnosticKind.BAD_NUMBER:
return InvalidNumberQuery(field_name, d.raw_value)
if d.kind is DiagnosticKind.TOO_DEEP:
return SearchQueryError("The search query is nested too deeply.")
if d.kind in _PATTERN_ON_KINDS:
kind_label = f" ({d.field_kind.name.lower()})" if d.field_kind else ""
return SearchQueryError(
f"Wildcard patterns are not supported for field "
f"{field_name!r}{kind_label}.",
)
if d.kind is DiagnosticKind.SINGLE_CHAR_BRACKET_RANGE:
field_label = f" for field {field_name!r}" if field_name else ""
return SearchQueryError(
f"{d.raw_value!r} looks like a bracket range{field_label}, but "
"'[' is not a wildcard character on its own. Combine it with a "
"wildcard, e.g. a trailing '*', or double-quote the value to "
"search it as literal text.",
)
logger.warning("Unmapped parse diagnostic %s: %s", d.kind, d.message)
return SearchQueryError("The search query could not be executed.")
def parse_simple_query(
@@ -268,7 +574,7 @@ def parse_simple_query(
CJK substrings the simple analyzer can't (long whitespace-free runs are
dropped by remove_long).
"""
tokens = _simple_query_tokens(raw_query)
tokens = simple_search_tokens(raw_query)
clauses: list[tuple[tantivy.Occur, tantivy.Query]] = []
if tokens:
@@ -291,23 +597,14 @@ def parse_simple_query(
)
for token in tokens
]
simple_query = (
token_queries[0][1]
if len(token_queries) == 1
else tantivy.Query.boolean_query(token_queries)
)
clauses.append((tantivy.Occur.Should, simple_query))
clauses.append((tantivy.Occur.Should, _any_of(token_queries)))
if cjk_fields and _has_cjk(raw_query):
cjk_q = _build_cjk_query(index, raw_query, cjk_fields)
if cjk_q is not None:
clauses.append((tantivy.Occur.Should, cjk_q))
if not clauses:
return tantivy.Query.empty_query()
if len(clauses) == 1:
return clauses[0][1]
return tantivy.Query.boolean_query(clauses)
return _any_of(clauses)
def parse_simple_text_highlight_query(
@@ -322,13 +619,21 @@ def parse_simple_text_highlight_query(
# Strip Tantivy operator chars before tokenizing: this is a plain-text
# highlight query, not a structured boolean query, so +/- are separators.
tokens = _simple_query_tokens(
tokens = simple_search_tokens(
regex.sub(r"[-+]", " ", raw_query, timeout=_REGEX_TIMEOUT),
)
if not tokens:
return tantivy.Query.empty_query()
return index.parse_query(" ".join(tokens), ["content"])
# Quote each token as its own phrase, escaping backslashes and embedded
# quotes. simple search tokens can carry arbitrary Tantivy syntax
# characters (`"`, `:`, `(`, `[`, `/`, ...) that the query-string parser
# would otherwise interpret as query grammar rather than literal text.
quoted_tokens = [
'"' + token.replace("\\", "\\\\").replace('"', '\\"') + '"' for token in tokens
]
return index.parse_query(" ".join(quoted_tokens), ["content"])
def parse_simple_text_query(
@@ -342,7 +647,7 @@ def parse_simple_text_query(
return parse_simple_query(
index,
raw_query,
SIMPLE_SEARCH_FIELDS,
_SIMPLE_SEARCH_FIELDS,
cjk_fields=_CJK_CONTENT_FIELDS,
)
@@ -358,6 +663,6 @@ def parse_simple_title_query(
return parse_simple_query(
index,
raw_query,
TITLE_SEARCH_FIELDS,
_TITLE_SEARCH_FIELDS,
cjk_fields=_CJK_TITLE_FIELDS,
)
+91
View File
@@ -0,0 +1,91 @@
from __future__ import annotations
import dataclasses
from typing import TYPE_CHECKING
from whoosh_compat import FieldKind
from whoosh_compat import FieldRegistry
from documents.search._fields import PUBLIC_FIELDS
from documents.search._tokenizer import ascii_fold
from documents.search._tokenizer import paperless_text_analyzer
from documents.search._tokenizer import stem_pattern_text
if TYPE_CHECKING:
from whoosh_compat import PatternNormalizer
_registry_cache: dict[str | None, FieldRegistry] = {}
def _identity_analyzer(text: str) -> list[str]:
"""Analyzer for KEYWORD fields indexed with the raw tokenizer (no splitting)."""
return [text]
def _fold_normalizer(text: str) -> str:
"""Wildcard/regex literal-run normalizer for fields indexed without stemming."""
return ascii_fold(text.lower())
def _make_pattern_normalizer(language: str | None) -> PatternNormalizer:
"""Build the wildcard/regex literal-run normalizer for a search language."""
def _pattern_normalizer(text: str) -> tuple[str, ...]:
"""Normalize a literal run into the forms a term may match.
TEXT index terms go through lowercase -> ascii_fold -> stem, so a
pattern that skips stemming can never match one: "invoice*" would look
for a term starting with "invoice" while the index holds "invoic". The
run is therefore offered stemmed as well. KEYWORD fields are indexed
raw and get _fold_normalizer instead, so their patterns stay literal.
Both forms are returned, as alternatives, because neither is a prefix
of the other in general: English stemming substitutes as well as
truncates ("copy" -> "copi"), so the stem alone loses the compounds
the typed run reaches ("copyright") while the typed run alone loses
the inflections the stem reaches ("copies"). whoosh-compat ORs the
alternatives per literal run and deduplicates them, so a run the
stemmer leaves alone costs exactly the one branch it did before.
Inside a bracket class the emitter calls this once per character and
uses the answer only if it is a single one-character form; two forms
there leave the character as typed. A stemmer does not change a lone
character, so the two forms deduplicate to one and the class body is
folded as before.
"""
folded = ascii_fold(text.lower())
stemmed = stem_pattern_text(folded, language)
return (folded, stemmed)
return _pattern_normalizer
def get_field_registry(language: str | None) -> FieldRegistry:
"""Build (or return the cached) FieldRegistry for the given search language.
Cached keyed by language, rebuilt on the same trigger register_tokenizers()
uses (settings.SEARCH_LANGUAGE change). A fresh call with a new language
builds and caches a new registry rather than mutating the old one.
"""
if language in _registry_cache:
return _registry_cache[language]
text_analyzer = paperless_text_analyzer(language).analyze
pattern_normalizer = _make_pattern_normalizer(language)
specs = [
dataclasses.replace(
field,
analyzer=_identity_analyzer
if field.kind is FieldKind.KEYWORD
else text_analyzer,
pattern_normalizer=_fold_normalizer
if field.kind is FieldKind.KEYWORD
else pattern_normalizer,
)
for field in PUBLIC_FIELDS
]
registry = FieldRegistry(specs)
_registry_cache[language] = registry
return registry
+222 -83
View File
@@ -1,14 +1,19 @@
from __future__ import annotations
import hashlib
import json
import logging
import shutil
from typing import TYPE_CHECKING
from typing import Final
from typing import NamedTuple
from typing import cast
import tantivy
from django.conf import settings
from whoosh_compat import FieldKind
from documents.search._fields import PUBLIC_FIELDS
if TYPE_CHECKING:
from pathlib import Path
@@ -16,7 +21,185 @@ if TYPE_CHECKING:
logger = logging.getLogger("paperless.search")
# v1 - Initial tantivy schema format
SCHEMA_VERSION: Final[int] = 1
# v2 - build_schema() derived from PUBLIC_FIELDS, changing the field declaration
# order, and the write-only correspondent/document_type/storage_path/tag id
# columns dropped. tantivy compares schemas by ordered field list, so an
# index built by v1 rejects every write against the v2 schema.
SCHEMA_VERSION: Final[int] = 2
class FieldDescriptor(NamedTuple):
"""One tantivy field, in declaration order.
The descriptor vocabulary is paperless', not tantivy-py's: it is both the
input to the SchemaBuilder and the input to schema_fingerprint(), so the
persisted fingerprint cannot move under a tantivy-py upgrade.
"""
name: str
kind: str
stored: bool
indexed: bool
fast: bool
tokenizer: str | None
# (schema kind, tokenizer) for the FieldKind -> FieldDescriptor mapping that
# doesn't need special-casing. JSON is handled separately below since it can
# emit a second, synthetic descriptor.
_KIND_TABLE: Final[dict[FieldKind, tuple[str, str | None]]] = {
FieldKind.TEXT: ("text", "paperless_text"),
FieldKind.KEYWORD: ("text", "raw"),
FieldKind.U64: ("u64", None),
FieldKind.DATE: ("date", None),
FieldKind.DATETIME: ("date", None),
}
# Kinds whose fast-field flag follows FieldSpec.fast rather than always False.
_FAST_FROM_FIELD: Final[frozenset[FieldKind]] = frozenset(
{FieldKind.U64, FieldKind.DATE, FieldKind.DATETIME},
)
def _public_field_descriptors() -> list[FieldDescriptor]:
"""Descriptors for the query-visible fields declared in PUBLIC_FIELDS."""
descriptors: list[FieldDescriptor] = []
for field in PUBLIC_FIELDS:
if field.kind is FieldKind.JSON:
descriptors.append(
FieldDescriptor(
field.name,
"json",
stored=True,
indexed=True,
fast=False,
tokenizer="paperless_text",
),
)
if field.name == "notes":
# Plain-text companion for snippet generation: tantivy's
# SnippetGenerator does not support JSON fields. Schema-only,
# no query-syntax meaning, not in PUBLIC_FIELDS.
descriptors.append(
FieldDescriptor(
"notes_text",
"text",
stored=True,
indexed=True,
fast=False,
tokenizer="paperless_text",
),
)
continue
schema_kind, tokenizer = _KIND_TABLE[field.kind]
descriptors.append(
FieldDescriptor(
field.name,
schema_kind,
stored=True,
indexed=True,
fast=field.fast if field.kind in _FAST_FROM_FIELD else False,
tokenizer=tokenizer,
),
)
return descriptors
def field_descriptors() -> list[FieldDescriptor]:
"""Every field of the document index, in the order tantivy declares them.
tantivy compares schemas by *ordered* field list, so the order here is
part of the on-disk contract: schema_fingerprint() hashes it and
needs_rebuild() acts on the result.
"""
return [
FieldDescriptor(
"id",
"u64",
stored=True,
indexed=True,
fast=True,
tokenizer=None,
),
*_public_field_descriptors(),
# Shadow sort fields - fast, not stored
*(
FieldDescriptor(
name,
"text",
stored=False,
indexed=True,
fast=True,
tokenizer="simple_analyzer",
)
for name in ("title_sort", "correspondent_sort", "type_sort")
),
# CJK support - not stored, indexed only
*(
FieldDescriptor(
name,
"text",
stored=False,
indexed=True,
fast=False,
tokenizer="bigram_analyzer",
)
for name in (
"bigram_content",
"bigram_title",
"bigram_correspondent",
"bigram_document_type",
"bigram_tag",
)
),
# Simple substring search support for title/content - not stored,
# indexed only
*(
FieldDescriptor(
name,
"text",
stored=False,
indexed=True,
fast=False,
tokenizer="simple_search_analyzer",
)
for name in ("simple_title", "simple_content")
),
# Autocomplete prefix scan via terms_with_prefix, which walks the
# field's term dictionary - so the field must be indexed (term dict),
# not stored. The stored value is never read back, so storing it only
# wastes space.
FieldDescriptor(
"autocomplete_word",
"text",
stored=False,
indexed=True,
fast=False,
tokenizer="raw",
),
# Permission filter columns, read by build_permission_filter.
*(
FieldDescriptor(
name,
"u64",
stored=False,
indexed=True,
fast=True,
tokenizer=None,
)
for name in ("owner_id", "viewer_id", "viewer_group_id")
),
]
def schema_fingerprint() -> str:
"""Hash of the field descriptors, stamped into .index_settings.json.
Changes whenever a field is added, removed, retyped, re-optioned or
reordered, so an index built from a different schema shape is detected
even when SCHEMA_VERSION was not bumped.
"""
payload = json.dumps([list(descriptor) for descriptor in field_descriptors()])
return hashlib.blake2b(payload.encode()).hexdigest()
def build_schema() -> tantivy.Schema:
@@ -32,85 +215,37 @@ def build_schema() -> tantivy.Schema:
"""
sb = tantivy.SchemaBuilder()
sb.add_unsigned_field("id", stored=True, indexed=True, fast=True)
sb.add_text_field("checksum", stored=True, tokenizer_name="raw")
for field in (
"title",
"correspondent",
"document_type",
"storage_path",
"original_filename",
"content",
):
sb.add_text_field(field, stored=True, tokenizer_name="paperless_text")
# Shadow sort fields - fast, not stored/indexed
for field in ("title_sort", "correspondent_sort", "type_sort"):
sb.add_text_field(
field,
stored=False,
tokenizer_name="simple_analyzer",
fast=True,
)
# CJK support - not stored, indexed only
sb.add_text_field("bigram_content", stored=False, tokenizer_name="bigram_analyzer")
sb.add_text_field("bigram_title", stored=False, tokenizer_name="bigram_analyzer")
sb.add_text_field(
"bigram_correspondent",
stored=False,
tokenizer_name="bigram_analyzer",
)
sb.add_text_field(
"bigram_document_type",
stored=False,
tokenizer_name="bigram_analyzer",
)
sb.add_text_field("bigram_tag", stored=False, tokenizer_name="bigram_analyzer")
# Simple substring search support for title/content - not stored, indexed only
sb.add_text_field(
"simple_title",
stored=False,
tokenizer_name="simple_search_analyzer",
)
sb.add_text_field(
"simple_content",
stored=False,
tokenizer_name="simple_search_analyzer",
)
# Autocomplete prefix scan via terms_with_prefix, which walks the field's
# term dictionary - so the field must be indexed (term dict), not stored.
# The stored value is never read back, so storing it only wastes space.
sb.add_text_field("autocomplete_word", stored=False, tokenizer_name="raw")
sb.add_text_field("tag", stored=True, tokenizer_name="paperless_text")
# JSON fields — structured queries: notes.user:alice, custom_fields.name:invoice
sb.add_json_field("notes", stored=True, tokenizer_name="paperless_text")
# Plain-text companion for notes — tantivy's SnippetGenerator does not support
# JSON fields, so highlights require a text field with the same content.
sb.add_text_field("notes_text", stored=True, tokenizer_name="paperless_text")
sb.add_json_field("custom_fields", stored=True, tokenizer_name="paperless_text")
for field in (
"correspondent_id",
"document_type_id",
"storage_path_id",
"tag_id",
"owner_id",
"viewer_id",
"viewer_group_id",
):
sb.add_unsigned_field(field, stored=False, indexed=True, fast=True)
for field in ("created", "modified", "added"):
sb.add_date_field(field, stored=True, indexed=True, fast=True)
for field in ("asn", "page_count", "num_notes"):
sb.add_unsigned_field(field, stored=True, indexed=True, fast=True)
for descriptor in field_descriptors():
if descriptor.kind == "text":
sb.add_text_field(
descriptor.name,
stored=descriptor.stored,
fast=descriptor.fast,
tokenizer_name=cast("str", descriptor.tokenizer),
)
elif descriptor.kind == "json":
sb.add_json_field(
descriptor.name,
stored=descriptor.stored,
fast=descriptor.fast,
tokenizer_name=cast("str", descriptor.tokenizer),
)
elif descriptor.kind == "u64":
sb.add_unsigned_field(
descriptor.name,
stored=descriptor.stored,
indexed=descriptor.indexed,
fast=descriptor.fast,
)
elif descriptor.kind == "date":
sb.add_date_field(
descriptor.name,
stored=descriptor.stored,
indexed=descriptor.indexed,
fast=descriptor.fast,
)
else:
raise ValueError(f"Unknown schema field kind: {descriptor.kind}")
return sb.build()
@@ -119,9 +254,9 @@ def needs_rebuild(index_dir: Path) -> bool:
"""
Check if the search index needs rebuilding.
Reads .index_settings.json to compare the stored schema version and
search language against the current configuration. Returns True if the
file is missing, unparsable, or either value mismatches.
Reads .index_settings.json to compare the stored schema version, search
language and schema fingerprint against the current configuration. Returns
True if the file is missing, unparsable, or any value mismatches.
Args:
index_dir: Path to the search index directory
@@ -140,6 +275,9 @@ def needs_rebuild(index_dir: Path) -> bool:
if "language" not in data or data["language"] != settings.SEARCH_LANGUAGE:
logger.info("Search index language changed - rebuilding.")
return True
if data.get("schema_fingerprint") != schema_fingerprint():
logger.info("Search index schema fingerprint mismatch - rebuilding.")
return True
except ValueError:
return True
return False
@@ -170,6 +308,7 @@ def _write_sentinels(index_dir: Path) -> None:
{
"schema_version": SCHEMA_VERSION,
"language": settings.SEARCH_LANGUAGE,
"schema_fingerprint": schema_fingerprint(),
},
),
)
+51 -2
View File
@@ -1,6 +1,7 @@
from __future__ import annotations
import logging
from functools import cache
from typing import Final
import tantivy
@@ -71,7 +72,7 @@ def register_tokenizers(index: tantivy.Index, language: str | None) -> None:
use fast=True and Tantivy requires fast-field tokenizers to exist
even for documents that omit those fields.
"""
index.register_tokenizer("paperless_text", _paperless_text(language))
index.register_tokenizer("paperless_text", paperless_text_analyzer(language))
index.register_tokenizer("simple_analyzer", _simple_analyzer())
index.register_tokenizer("bigram_analyzer", _bigram_analyzer())
index.register_tokenizer("simple_search_analyzer", _simple_search_analyzer())
@@ -79,7 +80,7 @@ def register_tokenizers(index: tantivy.Index, language: str | None) -> None:
index.register_fast_field_tokenizer("simple_analyzer", _simple_analyzer())
def _paperless_text(language: str | None) -> tantivy.TextAnalyzer:
def paperless_text_analyzer(language: str | None) -> tantivy.TextAnalyzer:
"""Main full-text tokenizer for content, title, etc: simple -> remove_long(129) -> lowercase -> ascii_fold [-> stemmer]"""
builder = (
tantivy.TextAnalyzerBuilder(tantivy.Tokenizer.simple())
@@ -100,6 +101,54 @@ def _paperless_text(language: str | None) -> tantivy.TextAnalyzer:
return builder.build()
@cache
def _pattern_stemmer(language: str | None) -> tantivy.TextAnalyzer | None:
"""The stemming tail of paperless_text_analyzer, over a whole literal run.
Same language gate and same Snowball stemmer paperless_text_analyzer
applies at index time, so query patterns follow SEARCH_LANGUAGE. Returns
None when that gate disables stemming; paperless_text_analyzer already
warns about an unsupported language, so this stays quiet.
The raw tokenizer keeps the run whole (a wildcard literal is a fragment,
not necessarily a word), and remove_long is kept so an over-long run is
treated the same way the index treats it.
"""
if not language:
return None
tantivy_lang = _LANGUAGE_MAP.get(language.lower())
if tantivy_lang is None:
return None
return (
tantivy.TextAnalyzerBuilder(tantivy.Tokenizer.raw())
.filter(tantivy.Filter.remove_long(_TOKEN_REMOVE_LONG_LIMIT))
.filter(tantivy.Filter.stemmer(tantivy_lang))
.build()
)
def stem_pattern_text(text: str, language: str | None) -> str:
"""Stem an already lowercased/ascii-folded run the way index terms are.
Returns text unchanged when stemming is disabled for language, and also
when the stem step does not yield exactly one token: remove_long drops a run
past the length limit, leaving no stem to substitute. Falling back to the
text as typed is the safe direction for a pattern prefix, since it can only
be as narrow as it was before stemming was considered.
The raw tokenizer emits one token whatever the input and the stemmer is
1-to-1, so only the zero-token case can fire today; the guard covers both
counts so a tokenizer change cannot turn this into an IndexError.
"""
analyzer = _pattern_stemmer(language)
if analyzer is None:
return text
tokens = analyzer.analyze(text)
if len(tokens) != 1:
return text
return tokens[0]
def _simple_analyzer() -> tantivy.TextAnalyzer:
"""Tokenizer for shadow sort fields (title_sort, correspondent_sort, type_sort): simple -> lowercase -> ascii_fold."""
return (

Some files were not shown because too many files have changed in this diff Show More