mirror of
https://github.com/paperless-ngx/paperless-ngx.git
synced 2026-11-03 13:35:37 +00:00
Compare commits
6
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
c237c93270 | ||
|
|
a5d3247ca7 | ||
|
|
8a67481a9d | ||
|
|
652f964f5d | ||
|
|
80a1827fa0 | ||
|
|
e240c89f27 |
No files matched your search
@@ -1,5 +1,91 @@
|
|||||||
# Changelog
|
# Changelog
|
||||||
|
|
||||||
|
## paperless-ngx 3.4.0
|
||||||
|
|
||||||
|
### Features / Enhancements
|
||||||
|
|
||||||
|
- Enhancement: support custom fields in workflow title templates [@shamoon](https://github.com/shamoon) ([#14564](https://github.com/paperless-ngx/paperless-ngx/pull/14564))
|
||||||
|
- Enhancement: relative date custom field queries [@shamoon](https://github.com/shamoon) ([#14549](https://github.com/paperless-ngx/paperless-ngx/pull/14549))
|
||||||
|
- Performance: reduce tasks badge load times [@JanKleine](https://github.com/JanKleine) ([#14552](https://github.com/paperless-ngx/paperless-ngx/pull/14552))
|
||||||
|
- Enhancement: skip local 2FA for trusted OIDC AMR [@shamoon](https://github.com/shamoon) ([#14403](https://github.com/paperless-ngx/paperless-ngx/pull/14403))
|
||||||
|
- Feature: Generate PDF thumbnails via poppler utils [@stumpylog](https://github.com/stumpylog) ([#14420](https://github.com/paperless-ngx/paperless-ngx/pull/14420))
|
||||||
|
- Enhancement: add upload count to button, link to tray, reduce notifications [@shamoon](https://github.com/shamoon) ([#14501](https://github.com/paperless-ngx/paperless-ngx/pull/14501))
|
||||||
|
- Enhancement: notifications improvements [@shamoon](https://github.com/shamoon) ([#14500](https://github.com/paperless-ngx/paperless-ngx/pull/14500))
|
||||||
|
- Enhancement: doc details ux improvements [@shamoon](https://github.com/shamoon) ([#14485](https://github.com/paperless-ngx/paperless-ngx/pull/14485))
|
||||||
|
- Enhancement (QoL): allow independently toggling child + parent tags in WF tag fields [@shamoon](https://github.com/shamoon) ([#14400](https://github.com/paperless-ngx/paperless-ngx/pull/14400))
|
||||||
|
- Tweak: drop the user dropdown caret [@shamoon](https://github.com/shamoon) ([#14444](https://github.com/paperless-ngx/paperless-ngx/pull/14444))
|
||||||
|
- Enhancement: navbar upload button [@shamoon](https://github.com/shamoon) ([#14442](https://github.com/paperless-ngx/paperless-ngx/pull/14442))
|
||||||
|
- Enhancement: AI chat UX tweaks [@shamoon](https://github.com/shamoon) ([#14415](https://github.com/paperless-ngx/paperless-ngx/pull/14415))
|
||||||
|
- Enhancement: support markdown in AI chat [@shamoon](https://github.com/shamoon) ([#14409](https://github.com/paperless-ngx/paperless-ngx/pull/14409))
|
||||||
|
|
||||||
|
### Bug Fixes
|
||||||
|
|
||||||
|
- Fix: truncate long version labels in dropdown [@shamoon](https://github.com/shamoon) ([#14578](https://github.com/paperless-ngx/paperless-ngx/pull/14578))
|
||||||
|
- Fixhancement: show better result for revoked / expired tasks [@shamoon](https://github.com/shamoon) ([#14566](https://github.com/paperless-ngx/paperless-ngx/pull/14566))
|
||||||
|
- Fix: move re-index after workflow runs on doc edit [@shamoon](https://github.com/shamoon) ([#14558](https://github.com/paperless-ngx/paperless-ngx/pull/14558))
|
||||||
|
- Fix: folder export with --delete fails on a stale manifest.json.tmp [@benpetty](https://github.com/benpetty) ([#14526](https://github.com/paperless-ngx/paperless-ngx/pull/14526))
|
||||||
|
- Fix: fix list type for api schema document object filters [@shamoon](https://github.com/shamoon) ([#14524](https://github.com/paperless-ngx/paperless-ngx/pull/14524))
|
||||||
|
- Fix: standardize some dropdown sizing [@shamoon](https://github.com/shamoon) ([#14510](https://github.com/paperless-ngx/paperless-ngx/pull/14510))
|
||||||
|
- Fix: prevent doc permissions queries on schema view [@shamoon](https://github.com/shamoon) ([#14503](https://github.com/paperless-ngx/paperless-ngx/pull/14503))
|
||||||
|
- Fix: dont defer tag assignment when barcode will not result in a split [@shamoon](https://github.com/shamoon) ([#14497](https://github.com/paperless-ngx/paperless-ngx/pull/14497))
|
||||||
|
|
||||||
|
### Dependencies
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary>6 changes</summary>
|
||||||
|
|
||||||
|
- Chore: bump some frontend deps for security fixes [@shamoon](https://github.com/shamoon) ([#14581](https://github.com/paperless-ngx/paperless-ngx/pull/14581))
|
||||||
|
- Chore: bump brotli, djangorestframework, filelock, msgpack, setuptools for security fixes [@stumpylog](https://github.com/stumpylog) ([#14575](https://github.com/paperless-ngx/paperless-ngx/pull/14575))
|
||||||
|
- Chore: bump django-allauth to 65.20 and oauthlib to 4.0 [@stumpylog](https://github.com/stumpylog) ([#14570](https://github.com/paperless-ngx/paperless-ngx/pull/14570))
|
||||||
|
- Chore(deps): Bump the uv group across 1 directory with 2 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#14517](https://github.com/paperless-ngx/paperless-ngx/pull/14517))
|
||||||
|
- Chore(deps): Bump the uv group across 1 directory with 2 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#14493](https://github.com/paperless-ngx/paperless-ngx/pull/14493))
|
||||||
|
- Chore: Drop ImageMagik as a dependency [@stumpylog](https://github.com/stumpylog) ([#14440](https://github.com/paperless-ngx/paperless-ngx/pull/14440))
|
||||||
|
|
||||||
|
</details>
|
||||||
|
|
||||||
|
### All App Changes
|
||||||
|
|
||||||
|
<details>
|
||||||
|
<summary>35 changes</summary>
|
||||||
|
|
||||||
|
- Chore: Consolidate test sample files [@stumpylog](https://github.com/stumpylog) ([#14430](https://github.com/paperless-ngx/paperless-ngx/pull/14430))
|
||||||
|
- Chore: bump some frontend deps for security fixes [@shamoon](https://github.com/shamoon) ([#14581](https://github.com/paperless-ngx/paperless-ngx/pull/14581))
|
||||||
|
- Enhancement: support custom fields in workflow title templates [@shamoon](https://github.com/shamoon) ([#14564](https://github.com/paperless-ngx/paperless-ngx/pull/14564))
|
||||||
|
- Enhancement: relative date custom field queries [@shamoon](https://github.com/shamoon) ([#14549](https://github.com/paperless-ngx/paperless-ngx/pull/14549))
|
||||||
|
- Fix: truncate long version labels in dropdown [@shamoon](https://github.com/shamoon) ([#14578](https://github.com/paperless-ngx/paperless-ngx/pull/14578))
|
||||||
|
- Chore: bump brotli, djangorestframework, filelock, msgpack, setuptools for security fixes [@stumpylog](https://github.com/stumpylog) ([#14575](https://github.com/paperless-ngx/paperless-ngx/pull/14575))
|
||||||
|
- Chore: bump django-allauth to 65.20 and oauthlib to 4.0 [@stumpylog](https://github.com/stumpylog) ([#14570](https://github.com/paperless-ngx/paperless-ngx/pull/14570))
|
||||||
|
- Fixhancement: show better result for revoked / expired tasks [@shamoon](https://github.com/shamoon) ([#14566](https://github.com/paperless-ngx/paperless-ngx/pull/14566))
|
||||||
|
- Fix: move re-index after workflow runs on doc edit [@shamoon](https://github.com/shamoon) ([#14558](https://github.com/paperless-ngx/paperless-ngx/pull/14558))
|
||||||
|
- Performance: reduce tasks badge load times [@JanKleine](https://github.com/JanKleine) ([#14552](https://github.com/paperless-ngx/paperless-ngx/pull/14552))
|
||||||
|
- Enhancement: skip local 2FA for trusted OIDC AMR [@shamoon](https://github.com/shamoon) ([#14403](https://github.com/paperless-ngx/paperless-ngx/pull/14403))
|
||||||
|
- Chore: consolidate direct permission grant lookups into one helper [@stumpylog](https://github.com/stumpylog) ([#14456](https://github.com/paperless-ngx/paperless-ngx/pull/14456))
|
||||||
|
- Chore: drop get\_groups\_with\_only\_permission [@stumpylog](https://github.com/stumpylog) ([#14454](https://github.com/paperless-ngx/paperless-ngx/pull/14454))
|
||||||
|
- Chore: Consolidate to a single way to set permission on an object [@stumpylog](https://github.com/stumpylog) ([#14453](https://github.com/paperless-ngx/paperless-ngx/pull/14453))
|
||||||
|
- Chore: Refactor bulk edit to remove PDF knowledge [@stumpylog](https://github.com/stumpylog) ([#14445](https://github.com/paperless-ngx/paperless-ngx/pull/14445))
|
||||||
|
- Fix: folder export with --delete fails on a stale manifest.json.tmp [@benpetty](https://github.com/benpetty) ([#14526](https://github.com/paperless-ngx/paperless-ngx/pull/14526))
|
||||||
|
- Chore(deps): Bump the uv group across 1 directory with 2 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#14517](https://github.com/paperless-ngx/paperless-ngx/pull/14517))
|
||||||
|
- Fix: fix list type for api schema document object filters [@shamoon](https://github.com/shamoon) ([#14524](https://github.com/paperless-ngx/paperless-ngx/pull/14524))
|
||||||
|
- Chore(deps): Bump the uv group across 1 directory with 2 updates @[dependabot[bot]](https://github.com/apps/dependabot) ([#14493](https://github.com/paperless-ngx/paperless-ngx/pull/14493))
|
||||||
|
- Chore: drop get\_objects\_for\_user\_owner\_aware [@stumpylog](https://github.com/stumpylog) ([#14457](https://github.com/paperless-ngx/paperless-ngx/pull/14457))
|
||||||
|
- Chore: update Angular components to 22.2.2 [@shamoon](https://github.com/shamoon) ([#14515](https://github.com/paperless-ngx/paperless-ngx/pull/14515))
|
||||||
|
- Chore: share one base class for the method-mapped permission classes [@stumpylog](https://github.com/stumpylog) ([#14458](https://github.com/paperless-ngx/paperless-ngx/pull/14458))
|
||||||
|
- Fix: standardize some dropdown sizing [@shamoon](https://github.com/shamoon) ([#14510](https://github.com/paperless-ngx/paperless-ngx/pull/14510))
|
||||||
|
- Chore: Drop ImageMagik as a dependency [@stumpylog](https://github.com/stumpylog) ([#14440](https://github.com/paperless-ngx/paperless-ngx/pull/14440))
|
||||||
|
- Feature: Generate PDF thumbnails via poppler utils [@stumpylog](https://github.com/stumpylog) ([#14420](https://github.com/paperless-ngx/paperless-ngx/pull/14420))
|
||||||
|
- Fix: prevent doc permissions queries on schema view [@shamoon](https://github.com/shamoon) ([#14503](https://github.com/paperless-ngx/paperless-ngx/pull/14503))
|
||||||
|
- Enhancement: add upload count to button, link to tray, reduce notifications [@shamoon](https://github.com/shamoon) ([#14501](https://github.com/paperless-ngx/paperless-ngx/pull/14501))
|
||||||
|
- Enhancement: notifications improvements [@shamoon](https://github.com/shamoon) ([#14500](https://github.com/paperless-ngx/paperless-ngx/pull/14500))
|
||||||
|
- Enhancement: doc details ux improvements [@shamoon](https://github.com/shamoon) ([#14485](https://github.com/paperless-ngx/paperless-ngx/pull/14485))
|
||||||
|
- Enhancement (QoL): allow independently toggling child + parent tags in WF tag fields [@shamoon](https://github.com/shamoon) ([#14400](https://github.com/paperless-ngx/paperless-ngx/pull/14400))
|
||||||
|
- Tweak: drop the user dropdown caret [@shamoon](https://github.com/shamoon) ([#14444](https://github.com/paperless-ngx/paperless-ngx/pull/14444))
|
||||||
|
- Enhancement: navbar upload button [@shamoon](https://github.com/shamoon) ([#14442](https://github.com/paperless-ngx/paperless-ngx/pull/14442))
|
||||||
|
- Enhancement: AI chat UX tweaks [@shamoon](https://github.com/shamoon) ([#14415](https://github.com/paperless-ngx/paperless-ngx/pull/14415))
|
||||||
|
- Enhancement: support markdown in AI chat [@shamoon](https://github.com/shamoon) ([#14409](https://github.com/paperless-ngx/paperless-ngx/pull/14409))
|
||||||
|
- Fix: dont defer tag assignment when barcode will not result in a split [@shamoon](https://github.com/shamoon) ([#14497](https://github.com/paperless-ngx/paperless-ngx/pull/14497))
|
||||||
|
|
||||||
|
</details>
|
||||||
|
|
||||||
## paperless-ngx 3.3.4
|
## paperless-ngx 3.3.4
|
||||||
|
|
||||||
### All App Changes
|
### All App Changes
|
||||||
|
|||||||
+5
-5
@@ -10334,18 +10334,18 @@
|
|||||||
<context context-type="linenumber">294,295</context>
|
<context context-type="linenumber">294,295</context>
|
||||||
</context-group>
|
</context-group>
|
||||||
</trans-unit>
|
</trans-unit>
|
||||||
<trans-unit id="2179847500064178686" datatype="html">
|
<trans-unit id="3420321797707163677" datatype="html">
|
||||||
<source>Edit document</source>
|
<source>Preview document</source>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/document-list/document-list.component.html</context>
|
<context context-type="sourcefile">src/app/components/document-list/document-list.component.html</context>
|
||||||
<context context-type="linenumber">328</context>
|
<context context-type="linenumber">328</context>
|
||||||
</context-group>
|
</context-group>
|
||||||
</trans-unit>
|
</trans-unit>
|
||||||
<trans-unit id="3420321797707163677" datatype="html">
|
<trans-unit id="2179847500064178686" datatype="html">
|
||||||
<source>Preview document</source>
|
<source>Edit document</source>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/document-list/document-list.component.html</context>
|
<context context-type="sourcefile">src/app/components/document-list/document-list.component.html</context>
|
||||||
<context context-type="linenumber">329</context>
|
<context context-type="linenumber">331</context>
|
||||||
</context-group>
|
</context-group>
|
||||||
</trans-unit>
|
</trans-unit>
|
||||||
<trans-unit id="4512084577073831437" datatype="html">
|
<trans-unit id="4512084577073831437" datatype="html">
|
||||||
|
|||||||
-7
@@ -1,10 +1,3 @@
|
|||||||
.custom-fields-dropdown {
|
.custom-fields-dropdown {
|
||||||
min-width: 300px;
|
min-width: 300px;
|
||||||
|
|
||||||
// correct position on mobile
|
|
||||||
@media (max-width: 575.98px) {
|
|
||||||
&.show {
|
|
||||||
margin-left: -245px !important;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
@@ -308,7 +308,7 @@
|
|||||||
</tr>
|
</tr>
|
||||||
<tr class="align-middle">
|
<tr class="align-middle">
|
||||||
<td i18n>Original SHA256 checksum</td>
|
<td i18n>Original SHA256 checksum</td>
|
||||||
<td><span class="badge bg-light font-monospace text-wrap text-break text-start">{{metadata()?.original_checksum}}</span></td>
|
<td><span class="badge bg-light text-dark font-monospace text-wrap text-break text-start">{{metadata()?.original_checksum}}</span></td>
|
||||||
</tr>
|
</tr>
|
||||||
<tr>
|
<tr>
|
||||||
<td i18n>Original file size</td>
|
<td i18n>Original file size</td>
|
||||||
@@ -465,7 +465,7 @@
|
|||||||
@switch (archiveContentRenderType) {
|
@switch (archiveContentRenderType) {
|
||||||
@case (ContentRenderType.PDF) {
|
@case (ContentRenderType.PDF) {
|
||||||
@if (!useNativePdfViewer) {
|
@if (!useNativePdfViewer) {
|
||||||
<div class="preview-sticky pdf-viewer-container">
|
<div class="preview-sticky pdf-viewer-container" tabindex="-1">
|
||||||
<pngx-pdf-viewer
|
<pngx-pdf-viewer
|
||||||
[src]="pdfSource()"
|
[src]="pdfSource()"
|
||||||
[sourceRevision]="previewRevision()"
|
[sourceRevision]="previewRevision()"
|
||||||
@@ -477,7 +477,7 @@
|
|||||||
(loadError)="onError($event)"
|
(loadError)="onError($event)"
|
||||||
(afterLoadComplete)="pdfPreviewLoaded($event)">
|
(afterLoadComplete)="pdfPreviewLoaded($event)">
|
||||||
</pngx-pdf-viewer>
|
</pngx-pdf-viewer>
|
||||||
<div class="position-absolute bottom-0 start-50 translate-middle-x z-1 mb-3 d-flex gap-2 p-1 bg-body border rounded shadow">
|
<div class="preview-controls position-absolute bottom-0 start-50 translate-middle-x z-1 mb-3 d-flex gap-2 p-1 bg-body border rounded shadow">
|
||||||
@if (previewNumPages()) {
|
@if (previewNumPages()) {
|
||||||
<div class="input-group input-group-sm w-auto flex-nowrap">
|
<div class="input-group input-group-sm w-auto flex-nowrap">
|
||||||
<input class="form-control flex-grow-0 w-auto" type="number" min="1" [max]="previewNumPages()" [ngModelOptions]="{standalone: true}" [ngModel]="previewCurrentPage()" (ngModelChange)="previewCurrentPage.set($event)" aria-label="Page" i18n-aria-label />
|
<input class="form-control flex-grow-0 w-auto" type="number" min="1" [max]="previewNumPages()" [ngModelOptions]="{standalone: true}" [ngModel]="previewCurrentPage()" (ngModelChange)="previewCurrentPage.set($event)" aria-label="Page" i18n-aria-label />
|
||||||
|
|||||||
@@ -12,6 +12,22 @@
|
|||||||
width: 100%;
|
width: 100%;
|
||||||
height: 100%;
|
height: 100%;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
&:focus {
|
||||||
|
outline: none;
|
||||||
|
}
|
||||||
|
|
||||||
|
.preview-controls {
|
||||||
|
opacity: 0;
|
||||||
|
pointer-events: none;
|
||||||
|
transition: opacity 0.15s ease-in-out;
|
||||||
|
}
|
||||||
|
|
||||||
|
&:hover .preview-controls,
|
||||||
|
&:focus-within .preview-controls {
|
||||||
|
opacity: 1;
|
||||||
|
pointer-events: auto;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
.btn-group .dropdown-toggle-split {
|
.btn-group .dropdown-toggle-split {
|
||||||
|
|||||||
@@ -208,7 +208,7 @@
|
|||||||
i18n>Correspondent</th>
|
i18n>Correspondent</th>
|
||||||
}
|
}
|
||||||
@if (activeDisplayFields.includes(DisplayField.TITLE)) {
|
@if (activeDisplayFields.includes(DisplayField.TITLE)) {
|
||||||
<th class="cursor-pointer"
|
<th class="cursor-pointer ps-5"
|
||||||
pngxSortable="title"
|
pngxSortable="title"
|
||||||
title="Sort by title" i18n-title
|
title="Sort by title" i18n-title
|
||||||
[currentSortField]="list.sortField"
|
[currentSortField]="list.sortField"
|
||||||
@@ -322,13 +322,13 @@
|
|||||||
</td>
|
</td>
|
||||||
}
|
}
|
||||||
@if (activeDisplayFields.includes(DisplayField.TITLE) || activeDisplayFields.includes(DisplayField.TAGS)) {
|
@if (activeDisplayFields.includes(DisplayField.TITLE) || activeDisplayFields.includes(DisplayField.TAGS)) {
|
||||||
<td width="30%">
|
<td width="30%" class="position-relative" [class.ps-5]="activeDisplayFields.includes(DisplayField.TITLE)">
|
||||||
@if (activeDisplayFields.includes(DisplayField.TITLE)) {
|
@if (activeDisplayFields.includes(DisplayField.TITLE)) {
|
||||||
<div class="d-inline-block" (mouseleave)="popupPreview.close()">
|
<div class="d-inline-block" (mouseleave)="popupPreview.close()">
|
||||||
<a routerLink="/documents/{{d.id}}" title="Edit document" i18n-title style="overflow-wrap: anywhere;">{{d.title | documentTitle}}</a>
|
<pngx-preview-popup class="position-absolute start-0" [document]="d" linkClasses="btn btn-sm btn-link text-secondary" linkTitle="Preview document" (click)="$event.stopPropagation()" i18n-linkTitle #popupPreview>
|
||||||
<pngx-preview-popup [document]="d" linkClasses="btn btn-sm btn-link text-secondary" linkTitle="Preview document" (click)="$event.stopPropagation()" i18n-linkTitle #popupPreview>
|
|
||||||
<i-bs name="eye"></i-bs>
|
<i-bs name="eye"></i-bs>
|
||||||
</pngx-preview-popup>
|
</pngx-preview-popup>
|
||||||
|
<a routerLink="/documents/{{d.id}}" title="Edit document" i18n-title style="overflow-wrap: anywhere;">{{d.title | documentTitle}}</a>
|
||||||
</div>
|
</div>
|
||||||
}
|
}
|
||||||
@if (activeDisplayFields.includes(DisplayField.TAGS)) {
|
@if (activeDisplayFields.includes(DisplayField.TAGS)) {
|
||||||
|
|||||||
@@ -87,6 +87,10 @@ $paperless-card-breakpoints: (
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
.table .ps-5 {
|
||||||
|
padding-left: 2.2rem !important;
|
||||||
|
}
|
||||||
|
|
||||||
a {
|
a {
|
||||||
cursor: pointer;
|
cursor: pointer;
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -20,7 +20,18 @@ OCR_LEADER_REGEX = re.compile(r"[._\-\u00b7]{4,}")
|
|||||||
HORIZONTAL_WHITESPACE_REGEX = re.compile(r"[ \t\u00a0]+")
|
HORIZONTAL_WHITESPACE_REGEX = re.compile(r"[ \t\u00a0]+")
|
||||||
|
|
||||||
|
|
||||||
def get_embedding_model(config: AIConfig) -> "BaseEmbedding":
|
def get_embedding_model(
|
||||||
|
config: AIConfig,
|
||||||
|
*,
|
||||||
|
refresh: bool = False,
|
||||||
|
) -> "BaseEmbedding":
|
||||||
|
"""Build the configured embedding model.
|
||||||
|
|
||||||
|
For the Hugging Face backend, a model already in the local cache is loaded
|
||||||
|
without contacting the Hub, which saves seconds per call. ``refresh=True``
|
||||||
|
checks the Hub for a newer revision first; index rebuilds use it so a new
|
||||||
|
model release is picked up together with re-embedding every document.
|
||||||
|
"""
|
||||||
match config.llm_embedding_backend:
|
match config.llm_embedding_backend:
|
||||||
case LLMEmbeddingBackend.OPENAI_LIKE:
|
case LLMEmbeddingBackend.OPENAI_LIKE:
|
||||||
from llama_index.embeddings.openai_like import OpenAILikeEmbedding
|
from llama_index.embeddings.openai_like import OpenAILikeEmbedding
|
||||||
@@ -52,11 +63,18 @@ def get_embedding_model(config: AIConfig) -> "BaseEmbedding":
|
|||||||
case LLMEmbeddingBackend.HUGGINGFACE:
|
case LLMEmbeddingBackend.HUGGINGFACE:
|
||||||
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
|
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
|
||||||
|
|
||||||
return HuggingFaceEmbedding(
|
hf_kwargs = {
|
||||||
model_name=config.llm_embedding_model
|
"model_name": config.llm_embedding_model
|
||||||
or "sentence-transformers/all-MiniLM-L6-v2",
|
or "sentence-transformers/all-MiniLM-L6-v2",
|
||||||
cache_folder=str(settings.DATA_DIR / "hf_cache"),
|
"cache_folder": str(settings.DATA_DIR / "hf_cache"),
|
||||||
)
|
}
|
||||||
|
if refresh:
|
||||||
|
return HuggingFaceEmbedding(**hf_kwargs)
|
||||||
|
try:
|
||||||
|
return HuggingFaceEmbedding(**hf_kwargs, local_files_only=True)
|
||||||
|
except OSError:
|
||||||
|
# Not cached yet, download it
|
||||||
|
return HuggingFaceEmbedding(**hf_kwargs)
|
||||||
case LLMEmbeddingBackend.OLLAMA:
|
case LLMEmbeddingBackend.OLLAMA:
|
||||||
from llama_index.embeddings.ollama import OllamaEmbedding
|
from llama_index.embeddings.ollama import OllamaEmbedding
|
||||||
from ollama import AsyncClient
|
from ollama import AsyncClient
|
||||||
|
|||||||
@@ -470,7 +470,7 @@ def update_llm_index(
|
|||||||
logger.warning("No documents found to index.")
|
logger.warning("No documents found to index.")
|
||||||
|
|
||||||
chunk_size = config.llm_embedding_chunk_size
|
chunk_size = config.llm_embedding_chunk_size
|
||||||
embed_model = get_embedding_model(config)
|
embed_model = get_embedding_model(config, refresh=rebuild)
|
||||||
|
|
||||||
with write_store(embed_model_name=model_name) as store:
|
with write_store(embed_model_name=model_name) as store:
|
||||||
if rebuild or not store.table_exists():
|
if rebuild or not store.table_exists():
|
||||||
|
|||||||
@@ -167,6 +167,7 @@ def test_get_embedding_model_huggingface(mock_ai_config):
|
|||||||
MockHuggingFaceEmbedding.assert_called_once_with(
|
MockHuggingFaceEmbedding.assert_called_once_with(
|
||||||
model_name="sentence-transformers/all-MiniLM-L6-v2",
|
model_name="sentence-transformers/all-MiniLM-L6-v2",
|
||||||
cache_folder=str(settings.DATA_DIR / "hf_cache"),
|
cache_folder=str(settings.DATA_DIR / "hf_cache"),
|
||||||
|
local_files_only=True,
|
||||||
)
|
)
|
||||||
assert model == MockHuggingFaceEmbedding.return_value
|
assert model == MockHuggingFaceEmbedding.return_value
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,106 @@
|
|||||||
|
from pathlib import Path
|
||||||
|
from unittest.mock import MagicMock
|
||||||
|
from unittest.mock import call
|
||||||
|
from unittest.mock import patch
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
import pytest_mock
|
||||||
|
from django.conf import settings
|
||||||
|
|
||||||
|
from paperless.models import LLMEmbeddingBackend
|
||||||
|
from paperless_ai import indexing
|
||||||
|
from paperless_ai.embedding import get_embedding_model
|
||||||
|
from paperless_ai.tests.conftest import FakeEmbedding
|
||||||
|
from paperless_testing.factories import DocumentFactory
|
||||||
|
|
||||||
|
MODEL = "sentence-transformers/all-MiniLM-L6-v2"
|
||||||
|
CACHE = str(settings.DATA_DIR / "hf_cache")
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def hf_config() -> MagicMock:
|
||||||
|
config = MagicMock()
|
||||||
|
config.llm_embedding_backend = LLMEmbeddingBackend.HUGGINGFACE
|
||||||
|
config.llm_embedding_model = MODEL
|
||||||
|
return config
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def hf_embedding(mocker: pytest_mock.MockerFixture) -> MagicMock:
|
||||||
|
return mocker.patch("llama_index.embeddings.huggingface.HuggingFaceEmbedding")
|
||||||
|
|
||||||
|
|
||||||
|
class TestHuggingFaceLocalFirst:
|
||||||
|
def test_uses_cached_model_without_hub(
|
||||||
|
self,
|
||||||
|
hf_config: MagicMock,
|
||||||
|
hf_embedding: MagicMock,
|
||||||
|
) -> None:
|
||||||
|
model = get_embedding_model(hf_config)
|
||||||
|
|
||||||
|
hf_embedding.assert_called_once_with(
|
||||||
|
model_name=MODEL,
|
||||||
|
cache_folder=CACHE,
|
||||||
|
local_files_only=True,
|
||||||
|
)
|
||||||
|
assert model is hf_embedding.return_value
|
||||||
|
|
||||||
|
def test_downloads_when_not_cached(
|
||||||
|
self,
|
||||||
|
hf_config: MagicMock,
|
||||||
|
hf_embedding: MagicMock,
|
||||||
|
) -> None:
|
||||||
|
downloaded = MagicMock()
|
||||||
|
hf_embedding.side_effect = [OSError("not cached"), downloaded]
|
||||||
|
|
||||||
|
model = get_embedding_model(hf_config)
|
||||||
|
|
||||||
|
assert hf_embedding.call_args_list == [
|
||||||
|
call(model_name=MODEL, cache_folder=CACHE, local_files_only=True),
|
||||||
|
call(model_name=MODEL, cache_folder=CACHE),
|
||||||
|
]
|
||||||
|
assert model is downloaded
|
||||||
|
|
||||||
|
def test_refresh_checks_the_hub(
|
||||||
|
self,
|
||||||
|
hf_config: MagicMock,
|
||||||
|
hf_embedding: MagicMock,
|
||||||
|
) -> None:
|
||||||
|
get_embedding_model(hf_config, refresh=True)
|
||||||
|
|
||||||
|
hf_embedding.assert_called_once_with(model_name=MODEL, cache_folder=CACHE)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.django_db
|
||||||
|
class TestIndexUpdateRefresh:
|
||||||
|
@pytest.fixture(autouse=True)
|
||||||
|
def _index_inputs(self, temp_llm_index_dir: Path) -> None:
|
||||||
|
DocumentFactory.create()
|
||||||
|
|
||||||
|
def _update(self, mock_embed_model: FakeEmbedding, *, rebuild: bool) -> MagicMock:
|
||||||
|
mock_config = MagicMock()
|
||||||
|
mock_config.llm_embedding_chunk_size = 512
|
||||||
|
with (
|
||||||
|
patch("paperless_ai.indexing.AIConfig", return_value=mock_config),
|
||||||
|
patch("paperless_ai.indexing.build_document_node", return_value=[]),
|
||||||
|
patch(
|
||||||
|
"paperless_ai.indexing.get_embedding_model",
|
||||||
|
return_value=mock_embed_model,
|
||||||
|
) as get_model,
|
||||||
|
):
|
||||||
|
indexing.update_llm_index(rebuild=rebuild)
|
||||||
|
get_model.assert_called_once()
|
||||||
|
return get_model
|
||||||
|
|
||||||
|
def test_rebuild_refreshes(self, mock_embed_model: FakeEmbedding) -> None:
|
||||||
|
get_model = self._update(mock_embed_model, rebuild=True)
|
||||||
|
|
||||||
|
assert get_model.call_args.kwargs == {"refresh": True}
|
||||||
|
|
||||||
|
def test_incremental_update_stays_local(
|
||||||
|
self,
|
||||||
|
mock_embed_model: FakeEmbedding,
|
||||||
|
) -> None:
|
||||||
|
get_model = self._update(mock_embed_model, rebuild=False)
|
||||||
|
|
||||||
|
assert get_model.call_args.kwargs == {"refresh": False}
|
||||||
Reference in new issue
Block a user