mirror of
https://github.com/paperless-ngx/paperless-ngx.git
synced 2026-08-11 13:23:21 +00:00
Compare commits
23
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
7456b52e84 | ||
|
|
6a392ea099 | ||
|
|
2032ad1341 | ||
|
|
cc8fee91c4 | ||
|
|
a5d46a883e | ||
|
|
b0e1793093 | ||
|
|
7e466d1f71 | ||
|
|
7b69a178c0 | ||
|
|
22cd13a8a9 | ||
|
|
72a4676be0 | ||
|
|
6673144d23 | ||
|
|
994a84cf92 | ||
|
|
654ce5d8f3 | ||
|
|
5d5e9b6db4 | ||
|
|
62089df2d8 | ||
|
|
5e5f6a88a3 | ||
|
|
02e6c49c62 | ||
|
|
3be64da4cb | ||
|
|
c28c532bef | ||
|
|
1d61f7fc62 | ||
|
|
aa67fd3aef | ||
|
|
17dc482872 | ||
|
|
b0e0e8a353 |
@@ -2047,6 +2047,18 @@ password. All of these options come from their similarly-named [Django settings]
|
|||||||
|
|
||||||
Defaults to None.
|
Defaults to None.
|
||||||
|
|
||||||
|
#### [`PAPERLESS_REMOTE_OCR_MODE=<str>`](#PAPERLESS_REMOTE_OCR_MODE) {#PAPERLESS_REMOTE_OCR_MODE}
|
||||||
|
|
||||||
|
: Which documents are sent to the remote OCR engine.
|
||||||
|
|
||||||
|
- `always`: every document of a supported file type is sent to the remote
|
||||||
|
engine, bypassing the local OCR engine.
|
||||||
|
- `workflow_only`: documents are processed locally unless a workflow
|
||||||
|
explicitly enables remote OCR for them, letting you use the remote engine
|
||||||
|
selectively.
|
||||||
|
|
||||||
|
Defaults to "always".
|
||||||
|
|
||||||
## AI {#ai}
|
## AI {#ai}
|
||||||
|
|
||||||
#### [`PAPERLESS_AI_ENABLED=<bool>`](#PAPERLESS_AI_ENABLED) {#PAPERLESS_AI_ENABLED}
|
#### [`PAPERLESS_AI_ENABLED=<bool>`](#PAPERLESS_AI_ENABLED) {#PAPERLESS_AI_ENABLED}
|
||||||
|
|||||||
@@ -456,6 +456,20 @@ def score(
|
|||||||
return 10
|
return 10
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**Remote services**
|
||||||
|
|
||||||
|
If your parser sends document content to a remote service, declare it:
|
||||||
|
|
||||||
|
```python
|
||||||
|
class MyCustomParser:
|
||||||
|
uses_remote_service = True
|
||||||
|
```
|
||||||
|
|
||||||
|
Paperless-ngx excludes such parsers when the document being consumed has not
|
||||||
|
been marked for remote processing, so users can keep remote OCR off by default
|
||||||
|
and enable it selectively with a workflow. Parsers that do not declare the
|
||||||
|
attribute are treated as fully local and are always considered.
|
||||||
|
|
||||||
**Archive and rendition flags**
|
**Archive and rendition flags**
|
||||||
|
|
||||||
```python
|
```python
|
||||||
|
|||||||
+6
-1
@@ -1084,11 +1084,16 @@ Paperless-ngx supports performing OCR on documents using remote services. At the
|
|||||||
[Microsoft's Azure "Document Intelligence" service](https://azure.microsoft.com/en-us/products/ai-services/ai-document-intelligence).
|
[Microsoft's Azure "Document Intelligence" service](https://azure.microsoft.com/en-us/products/ai-services/ai-document-intelligence).
|
||||||
This is of course a paid service (with a free tier) which requires an Azure account and subscription. Azure AI is not affiliated with
|
This is of course a paid service (with a free tier) which requires an Azure account and subscription. Azure AI is not affiliated with
|
||||||
Paperless-ngx in any way. When enabled, Paperless-ngx will automatically send appropriate documents to Azure for OCR processing, bypassing
|
Paperless-ngx in any way. When enabled, Paperless-ngx will automatically send appropriate documents to Azure for OCR processing, bypassing
|
||||||
the local OCR engine. See the [configuration](configuration.md#PAPERLESS_REMOTE_OCR_ENGINE) options for more details.
|
the local OCR engine. See the [configuration](configuration.md#PAPERLESS_REMOTE_OCR_ENGINE) options for more details. These
|
||||||
|
settings can be supplied as environment variables or via **Application Configuration**.
|
||||||
|
|
||||||
Additionally, when using a commercial service with this feature, consider both potential costs as well as any associated file size
|
Additionally, when using a commercial service with this feature, consider both potential costs as well as any associated file size
|
||||||
or page limitations (e.g. with a free tier).
|
or page limitations (e.g. with a free tier).
|
||||||
|
|
||||||
|
By default, every document of a supported file type is sent to the remote engine. To use it more selectively, set the
|
||||||
|
[remote OCR mode](configuration.md#PAPERLESS_REMOTE_OCR_MODE) to `workflow_only`. Documents are then processed locally
|
||||||
|
unless a workflow explicitly enables remote OCR for them, so you can limit the remote engine to particular documents.
|
||||||
|
|
||||||
## Architecture
|
## Architecture
|
||||||
|
|
||||||
Paperless-ngx consists of the following components:
|
Paperless-ngx consists of the following components:
|
||||||
|
|||||||
+13
-6
@@ -1703,7 +1703,7 @@
|
|||||||
</context-group>
|
</context-group>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/common/suggestions-dropdown/suggestions-dropdown.component.html</context>
|
<context context-type="sourcefile">src/app/components/common/suggestions-dropdown/suggestions-dropdown.component.html</context>
|
||||||
<context context-type="linenumber">28</context>
|
<context context-type="linenumber">34</context>
|
||||||
</context-group>
|
</context-group>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/dashboard/widgets/statistics-widget/statistics-widget.component.html</context>
|
<context context-type="sourcefile">src/app/components/dashboard/widgets/statistics-widget/statistics-widget.component.html</context>
|
||||||
@@ -3279,7 +3279,7 @@
|
|||||||
</context-group>
|
</context-group>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/common/suggestions-dropdown/suggestions-dropdown.component.html</context>
|
<context context-type="sourcefile">src/app/components/common/suggestions-dropdown/suggestions-dropdown.component.html</context>
|
||||||
<context context-type="linenumber">40</context>
|
<context context-type="linenumber">46</context>
|
||||||
</context-group>
|
</context-group>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/dashboard/widgets/statistics-widget/statistics-widget.component.html</context>
|
<context context-type="sourcefile">src/app/components/dashboard/widgets/statistics-widget/statistics-widget.component.html</context>
|
||||||
@@ -7070,32 +7070,39 @@
|
|||||||
<context context-type="linenumber">143</context>
|
<context context-type="linenumber">143</context>
|
||||||
</context-group>
|
</context-group>
|
||||||
</trans-unit>
|
</trans-unit>
|
||||||
|
<trans-unit id="8336346011691074629" datatype="html">
|
||||||
|
<source>No suggestions</source>
|
||||||
|
<context-group purpose="location">
|
||||||
|
<context context-type="sourcefile">src/app/components/common/suggestions-dropdown/suggestions-dropdown.component.html</context>
|
||||||
|
<context context-type="linenumber">11,12</context>
|
||||||
|
</context-group>
|
||||||
|
</trans-unit>
|
||||||
<trans-unit id="5320136382998259826" datatype="html">
|
<trans-unit id="5320136382998259826" datatype="html">
|
||||||
<source>Suggest</source>
|
<source>Suggest</source>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/common/suggestions-dropdown/suggestions-dropdown.component.html</context>
|
<context context-type="sourcefile">src/app/components/common/suggestions-dropdown/suggestions-dropdown.component.html</context>
|
||||||
<context context-type="linenumber">8,9</context>
|
<context context-type="linenumber">13,14</context>
|
||||||
</context-group>
|
</context-group>
|
||||||
</trans-unit>
|
</trans-unit>
|
||||||
<trans-unit id="6934085657687954669" datatype="html">
|
<trans-unit id="6934085657687954669" datatype="html">
|
||||||
<source>Show suggestions</source>
|
<source>Show suggestions</source>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/common/suggestions-dropdown/suggestions-dropdown.component.html</context>
|
<context context-type="sourcefile">src/app/components/common/suggestions-dropdown/suggestions-dropdown.component.html</context>
|
||||||
<context context-type="linenumber">17,18</context>
|
<context context-type="linenumber">23,24</context>
|
||||||
</context-group>
|
</context-group>
|
||||||
</trans-unit>
|
</trans-unit>
|
||||||
<trans-unit id="3834115140127576673" datatype="html">
|
<trans-unit id="3834115140127576673" datatype="html">
|
||||||
<source>No novel suggestions</source>
|
<source>No novel suggestions</source>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/common/suggestions-dropdown/suggestions-dropdown.component.html</context>
|
<context context-type="sourcefile">src/app/components/common/suggestions-dropdown/suggestions-dropdown.component.html</context>
|
||||||
<context context-type="linenumber">24,25</context>
|
<context context-type="linenumber">30,31</context>
|
||||||
</context-group>
|
</context-group>
|
||||||
</trans-unit>
|
</trans-unit>
|
||||||
<trans-unit id="4369111787961525769" datatype="html">
|
<trans-unit id="4369111787961525769" datatype="html">
|
||||||
<source>Document Types</source>
|
<source>Document Types</source>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/common/suggestions-dropdown/suggestions-dropdown.component.html</context>
|
<context context-type="sourcefile">src/app/components/common/suggestions-dropdown/suggestions-dropdown.component.html</context>
|
||||||
<context context-type="linenumber">34</context>
|
<context context-type="linenumber">40</context>
|
||||||
</context-group>
|
</context-group>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/dashboard/widgets/statistics-widget/statistics-widget.component.html</context>
|
<context context-type="sourcefile">src/app/components/dashboard/widgets/statistics-widget/statistics-widget.component.html</context>
|
||||||
|
|||||||
+1
-1
@@ -66,5 +66,5 @@
|
|||||||
"ts-node": "~10.9.1",
|
"ts-node": "~10.9.1",
|
||||||
"typescript": "^6.0.3"
|
"typescript": "^6.0.3"
|
||||||
},
|
},
|
||||||
"packageManager": "pnpm@10.26.0"
|
"packageManager": "pnpm@11.15.1"
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ trustPolicy: no-downgrade
|
|||||||
trustPolicyExclude:
|
trustPolicyExclude:
|
||||||
- "chokidar@4.0.3"
|
- "chokidar@4.0.3"
|
||||||
- "semver@6.3.1 || 5.7.2"
|
- "semver@6.3.1 || 5.7.2"
|
||||||
|
blockExoticSubdeps: true
|
||||||
allowBuilds:
|
allowBuilds:
|
||||||
"@parcel/watcher": true
|
"@parcel/watcher": true
|
||||||
canvas: true
|
canvas: true
|
||||||
|
|||||||
@@ -14,8 +14,12 @@
|
|||||||
<a ngbNavLink>{{category}}</a>
|
<a ngbNavLink>{{category}}</a>
|
||||||
<ng-template ngbNavContent>
|
<ng-template ngbNavContent>
|
||||||
<div class="p-3">
|
<div class="p-3">
|
||||||
|
@for (section of getCategorySections(category); track section) {
|
||||||
|
@if (section) {
|
||||||
|
<h5 class="mt-4 mb-3">{{section}}</h5>
|
||||||
|
}
|
||||||
<div class="row row-cols-1 row-cols-md-2 row-cols-lg-3 g-2">
|
<div class="row row-cols-1 row-cols-md-2 row-cols-lg-3 g-2">
|
||||||
@for (option of getCategoryOptions(category); track option.key) {
|
@for (option of getCategoryOptions(category, section); track option.key) {
|
||||||
<div class="col">
|
<div class="col">
|
||||||
<div class="card bg-light">
|
<div class="card bg-light">
|
||||||
<div class="card-body">
|
<div class="card-body">
|
||||||
@@ -51,6 +55,7 @@
|
|||||||
</div>
|
</div>
|
||||||
}
|
}
|
||||||
</div>
|
</div>
|
||||||
|
}
|
||||||
</div>
|
</div>
|
||||||
</ng-template>
|
</ng-template>
|
||||||
</li>
|
</li>
|
||||||
|
|||||||
@@ -8,7 +8,11 @@ import { NgbModule } from '@ng-bootstrap/ng-bootstrap'
|
|||||||
import { NgSelectModule } from '@ng-select/ng-select'
|
import { NgSelectModule } from '@ng-select/ng-select'
|
||||||
import { NgxBootstrapIconsModule, allIcons } from 'ngx-bootstrap-icons'
|
import { NgxBootstrapIconsModule, allIcons } from 'ngx-bootstrap-icons'
|
||||||
import { of, throwError } from 'rxjs'
|
import { of, throwError } from 'rxjs'
|
||||||
import { OutputTypeConfig } from 'src/app/data/paperless-config'
|
import {
|
||||||
|
ConfigCategory,
|
||||||
|
ConfigSection,
|
||||||
|
OutputTypeConfig,
|
||||||
|
} from 'src/app/data/paperless-config'
|
||||||
import { ConfigService } from 'src/app/services/config.service'
|
import { ConfigService } from 'src/app/services/config.service'
|
||||||
import { SettingsService } from 'src/app/services/settings.service'
|
import { SettingsService } from 'src/app/services/settings.service'
|
||||||
import { ToastService } from 'src/app/services/toast.service'
|
import { ToastService } from 'src/app/services/toast.service'
|
||||||
@@ -158,4 +162,24 @@ describe('ConfigComponent', () => {
|
|||||||
component.resetOption('barcodes_enabled')
|
component.resetOption('barcodes_enabled')
|
||||||
expect(component.configForm.get('barcodes_enabled').value).toBeNull()
|
expect(component.configForm.get('barcodes_enabled').value).toBeNull()
|
||||||
})
|
})
|
||||||
|
|
||||||
|
it('should group options into sections within a category, or not', () => {
|
||||||
|
const sections = component.getCategorySections(ConfigCategory.OCR)
|
||||||
|
expect(sections).toEqual([null, ConfigSection.RemoteOCR])
|
||||||
|
expect(
|
||||||
|
component
|
||||||
|
.getCategoryOptions(ConfigCategory.OCR)
|
||||||
|
.map((option) => option.key)
|
||||||
|
).toContain('output_type')
|
||||||
|
expect(
|
||||||
|
component
|
||||||
|
.getCategoryOptions(ConfigCategory.OCR, ConfigSection.RemoteOCR)
|
||||||
|
.map((option) => option.key)
|
||||||
|
).toEqual([
|
||||||
|
'remote_ocr_engine',
|
||||||
|
'remote_ocr_api_key',
|
||||||
|
'remote_ocr_endpoint',
|
||||||
|
'remote_ocr_mode',
|
||||||
|
])
|
||||||
|
})
|
||||||
})
|
})
|
||||||
|
|||||||
@@ -74,8 +74,20 @@ export class ConfigComponent
|
|||||||
return Object.values(ConfigCategory)
|
return Object.values(ConfigCategory)
|
||||||
}
|
}
|
||||||
|
|
||||||
getCategoryOptions(category: string): ConfigOption[] {
|
getCategorySections(category: string): string[] {
|
||||||
return PaperlessConfigOptions.filter((o) => o.category === category)
|
return [
|
||||||
|
...new Set(
|
||||||
|
PaperlessConfigOptions.filter((o) => o.category === category).map(
|
||||||
|
(o) => o.section ?? null // null means no section
|
||||||
|
)
|
||||||
|
),
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
||||||
|
getCategoryOptions(category: string, section: string = null): ConfigOption[] {
|
||||||
|
return PaperlessConfigOptions.filter(
|
||||||
|
(o) => o.category === category && (o.section ?? null) === section
|
||||||
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
initialConfig: PaperlessConfig
|
initialConfig: PaperlessConfig
|
||||||
|
|||||||
+7
-1
@@ -2,10 +2,16 @@
|
|||||||
<button type="button" class="btn btn-sm btn-outline-primary" (click)="clickSuggest()" [disabled]="disabled() || loading() || (suggestions() && !aiEnabled())">
|
<button type="button" class="btn btn-sm btn-outline-primary" (click)="clickSuggest()" [disabled]="disabled() || loading() || (suggestions() && !aiEnabled())">
|
||||||
@if (loading()) {
|
@if (loading()) {
|
||||||
<div class="spinner-border spinner-border-sm" role="status"></div>
|
<div class="spinner-border spinner-border-sm" role="status"></div>
|
||||||
|
} @else if (noSuggestions) {
|
||||||
|
<i-bs width="1.2em" height="1.2em" name="check-circle"></i-bs>
|
||||||
} @else {
|
} @else {
|
||||||
<i-bs width="1.2em" height="1.2em" name="stars"></i-bs>
|
<i-bs width="1.2em" height="1.2em" name="stars"></i-bs>
|
||||||
}
|
}
|
||||||
|
@if (noSuggestions) {
|
||||||
|
<span class="d-none d-lg-inline ps-1" i18n>No suggestions</span>
|
||||||
|
} @else {
|
||||||
<span class="d-none d-lg-inline ps-1" i18n>Suggest</span>
|
<span class="d-none d-lg-inline ps-1" i18n>Suggest</span>
|
||||||
|
}
|
||||||
@if (totalSuggestions > 0) {
|
@if (totalSuggestions > 0) {
|
||||||
<span class="badge bg-primary ms-2">{{ totalSuggestions }}</span>
|
<span class="badge bg-primary ms-2">{{ totalSuggestions }}</span>
|
||||||
}
|
}
|
||||||
@@ -19,7 +25,7 @@
|
|||||||
|
|
||||||
<div ngbDropdownMenu aria-labelledby="suggestionsDropdown" class="shadow suggestions-dropdown">
|
<div ngbDropdownMenu aria-labelledby="suggestionsDropdown" class="shadow suggestions-dropdown">
|
||||||
<div class="list-group list-group-flush small pb-0">
|
<div class="list-group list-group-flush small pb-0">
|
||||||
@if (!suggestions()?.suggested_tags && !suggestions()?.suggested_document_types && !suggestions()?.suggested_correspondents) {
|
@if (totalSuggestions === 0) {
|
||||||
<div class="list-group-item text-muted fst-italic">
|
<div class="list-group-item text-muted fst-italic">
|
||||||
<small class="text-muted small fst-italic" i18n>No novel suggestions</small>
|
<small class="text-muted small fst-italic" i18n>No novel suggestions</small>
|
||||||
</div>
|
</div>
|
||||||
|
|||||||
+29
@@ -30,6 +30,34 @@ describe('SuggestionsDropdownComponent', () => {
|
|||||||
expect(component.totalSuggestions).toBe(4)
|
expect(component.totalSuggestions).toBe(4)
|
||||||
})
|
})
|
||||||
|
|
||||||
|
it('should show when a completed request returned no suggestions', () => {
|
||||||
|
fixture.componentRef.setInput('suggestions', {
|
||||||
|
correspondents: [],
|
||||||
|
tags: [],
|
||||||
|
document_types: [],
|
||||||
|
storage_paths: [],
|
||||||
|
dates: [],
|
||||||
|
})
|
||||||
|
fixture.detectChanges()
|
||||||
|
|
||||||
|
expect(component.noSuggestions).toBeTruthy()
|
||||||
|
expect(fixture.nativeElement.textContent).toContain('No suggestions')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('should not show the empty state before a request or with suggestions', () => {
|
||||||
|
expect(component.noSuggestions).toBeFalsy()
|
||||||
|
|
||||||
|
fixture.componentRef.setInput('suggestions', {
|
||||||
|
correspondents: [],
|
||||||
|
tags: [42],
|
||||||
|
document_types: [],
|
||||||
|
storage_paths: [],
|
||||||
|
dates: [],
|
||||||
|
})
|
||||||
|
|
||||||
|
expect(component.noSuggestions).toBeFalsy()
|
||||||
|
})
|
||||||
|
|
||||||
it('should emit getSuggestions when clickSuggest is called and suggestions are null', () => {
|
it('should emit getSuggestions when clickSuggest is called and suggestions are null', () => {
|
||||||
jest.spyOn(component.getSuggestions, 'emit')
|
jest.spyOn(component.getSuggestions, 'emit')
|
||||||
fixture.componentRef.setInput('suggestions', null)
|
fixture.componentRef.setInput('suggestions', null)
|
||||||
@@ -59,5 +87,6 @@ describe('SuggestionsDropdownComponent', () => {
|
|||||||
})
|
})
|
||||||
component.clickSuggest()
|
component.clickSuggest()
|
||||||
expect(component.dropdown.open).toBeTruthy()
|
expect(component.dropdown.open).toBeTruthy()
|
||||||
|
expect(fixture.nativeElement.textContent).toContain('No novel suggestions')
|
||||||
})
|
})
|
||||||
})
|
})
|
||||||
|
|||||||
+17
@@ -61,4 +61,21 @@ export class SuggestionsDropdownComponent {
|
|||||||
this.suggestions()?.suggested_document_types?.length || 0
|
this.suggestions()?.suggested_document_types?.length || 0
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
get noSuggestions(): boolean {
|
||||||
|
const suggestions = this.suggestions()
|
||||||
|
return (
|
||||||
|
suggestions != null &&
|
||||||
|
!suggestions.title &&
|
||||||
|
!suggestions.tags?.length &&
|
||||||
|
!suggestions.suggested_tags?.length &&
|
||||||
|
!suggestions.correspondents?.length &&
|
||||||
|
!suggestions.suggested_correspondents?.length &&
|
||||||
|
!suggestions.document_types?.length &&
|
||||||
|
!suggestions.suggested_document_types?.length &&
|
||||||
|
!suggestions.storage_paths?.length &&
|
||||||
|
!suggestions.suggested_storage_paths?.length &&
|
||||||
|
!suggestions.dates?.length
|
||||||
|
)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -54,6 +54,10 @@ export const ConfigCategory = {
|
|||||||
AI: $localize`AI Settings`,
|
AI: $localize`AI Settings`,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export const ConfigSection = {
|
||||||
|
RemoteOCR: $localize`Remote OCR`,
|
||||||
|
}
|
||||||
|
|
||||||
export const LLMEmbeddingBackendConfig = {
|
export const LLMEmbeddingBackendConfig = {
|
||||||
OPENAI_LIKE: 'openai-like',
|
OPENAI_LIKE: 'openai-like',
|
||||||
HUGGINGFACE: 'huggingface',
|
HUGGINGFACE: 'huggingface',
|
||||||
@@ -65,6 +69,15 @@ export const LLMBackendConfig = {
|
|||||||
OLLAMA: 'ollama',
|
OLLAMA: 'ollama',
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export const RemoteOCREngineConfig = {
|
||||||
|
AZURE_AI: 'azureai',
|
||||||
|
}
|
||||||
|
|
||||||
|
export const RemoteOCRModeConfig = {
|
||||||
|
ALWAYS: 'always',
|
||||||
|
WORKFLOW_ONLY: 'workflow_only',
|
||||||
|
}
|
||||||
|
|
||||||
export interface ConfigOption {
|
export interface ConfigOption {
|
||||||
key: string
|
key: string
|
||||||
title: string
|
title: string
|
||||||
@@ -72,6 +85,7 @@ export interface ConfigOption {
|
|||||||
choices?: Array<{ id: string; name: string }>
|
choices?: Array<{ id: string; name: string }>
|
||||||
config_key?: string
|
config_key?: string
|
||||||
category: string
|
category: string
|
||||||
|
section?: string
|
||||||
note?: string
|
note?: string
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -181,6 +195,43 @@ export const PaperlessConfigOptions: ConfigOption[] = [
|
|||||||
config_key: 'PAPERLESS_OCR_USER_ARGS',
|
config_key: 'PAPERLESS_OCR_USER_ARGS',
|
||||||
category: ConfigCategory.OCR,
|
category: ConfigCategory.OCR,
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
key: 'remote_ocr_engine',
|
||||||
|
title: $localize`Remote OCR Engine`,
|
||||||
|
type: ConfigOptionType.Select,
|
||||||
|
choices: mapToItems(RemoteOCREngineConfig),
|
||||||
|
config_key: 'PAPERLESS_REMOTE_OCR_ENGINE',
|
||||||
|
category: ConfigCategory.OCR,
|
||||||
|
section: ConfigSection.RemoteOCR,
|
||||||
|
note: $localize`Enabling remote OCR sends documents to a third-party service for processing. Consider the privacy implications as well as potential costs before enabling.`,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
key: 'remote_ocr_api_key',
|
||||||
|
title: $localize`Remote OCR API Key`,
|
||||||
|
type: ConfigOptionType.Password,
|
||||||
|
config_key: 'PAPERLESS_REMOTE_OCR_API_KEY',
|
||||||
|
category: ConfigCategory.OCR,
|
||||||
|
section: ConfigSection.RemoteOCR,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
key: 'remote_ocr_endpoint',
|
||||||
|
title: $localize`Remote OCR Endpoint`,
|
||||||
|
type: ConfigOptionType.String,
|
||||||
|
config_key: 'PAPERLESS_REMOTE_OCR_ENDPOINT',
|
||||||
|
category: ConfigCategory.OCR,
|
||||||
|
section: ConfigSection.RemoteOCR,
|
||||||
|
note: $localize`Required when using the Azure AI engine.`,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
key: 'remote_ocr_mode',
|
||||||
|
title: $localize`Remote OCR Mode`,
|
||||||
|
type: ConfigOptionType.Select,
|
||||||
|
choices: mapToItems(RemoteOCRModeConfig),
|
||||||
|
config_key: 'PAPERLESS_REMOTE_OCR_MODE',
|
||||||
|
category: ConfigCategory.OCR,
|
||||||
|
section: ConfigSection.RemoteOCR,
|
||||||
|
note: $localize`Which documents are sent to the remote engine. Use 'workflow_only' to keep remote OCR off unless a workflow enables it for a document.`,
|
||||||
|
},
|
||||||
{
|
{
|
||||||
key: 'app_logo',
|
key: 'app_logo',
|
||||||
title: $localize`Application Logo`,
|
title: $localize`Application Logo`,
|
||||||
@@ -398,6 +449,10 @@ export interface PaperlessConfig extends ObjectWithId {
|
|||||||
barcode_enable_tag: boolean
|
barcode_enable_tag: boolean
|
||||||
barcode_tag_mapping: object
|
barcode_tag_mapping: object
|
||||||
barcode_tag_split: boolean
|
barcode_tag_split: boolean
|
||||||
|
remote_ocr_engine: string
|
||||||
|
remote_ocr_api_key: string
|
||||||
|
remote_ocr_endpoint: string
|
||||||
|
remote_ocr_mode: string
|
||||||
ai_enabled: boolean
|
ai_enabled: boolean
|
||||||
llm_embedding_backend: string
|
llm_embedding_backend: string
|
||||||
llm_embedding_model: string
|
llm_embedding_model: string
|
||||||
|
|||||||
@@ -53,6 +53,7 @@ from documents.utils import copy_basic_file_stats
|
|||||||
from documents.utils import copy_file_with_basic_stats
|
from documents.utils import copy_file_with_basic_stats
|
||||||
from documents.utils import run_subprocess
|
from documents.utils import run_subprocess
|
||||||
from paperless.config import OcrConfig
|
from paperless.config import OcrConfig
|
||||||
|
from paperless.config import RemoteOCRConfig
|
||||||
from paperless.models import ArchiveFileGenerationChoices
|
from paperless.models import ArchiveFileGenerationChoices
|
||||||
from paperless.parsers import ParserContext
|
from paperless.parsers import ParserContext
|
||||||
from paperless.parsers import ParserProtocol
|
from paperless.parsers import ParserProtocol
|
||||||
@@ -451,12 +452,19 @@ class ConsumerPlugin(
|
|||||||
except Exception as e:
|
except Exception as e:
|
||||||
self.log.error(f"Error attempting to clean PDF: {e}")
|
self.log.error(f"Error attempting to clean PDF: {e}")
|
||||||
|
|
||||||
|
# Workflows have already run at this point, so the metadata knows
|
||||||
|
# whether this document was singled out for remote OCR
|
||||||
|
allow_remote = (
|
||||||
|
self.metadata.remote_ocr or RemoteOCRConfig().remote_ocr_by_default
|
||||||
|
)
|
||||||
|
|
||||||
# Based on the mime type, get the parser for that type
|
# Based on the mime type, get the parser for that type
|
||||||
parser_class: type[ParserProtocol] | None = (
|
parser_class: type[ParserProtocol] | None = (
|
||||||
get_parser_registry().get_parser_for_file(
|
get_parser_registry().get_parser_for_file(
|
||||||
mime_type,
|
mime_type,
|
||||||
self.filename,
|
self.filename,
|
||||||
self.working_copy,
|
self.working_copy,
|
||||||
|
allow_remote=allow_remote,
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
if not parser_class:
|
if not parser_class:
|
||||||
|
|||||||
@@ -34,6 +34,7 @@ class DocumentMetadataOverrides:
|
|||||||
skip_asn_if_exists: bool = False
|
skip_asn_if_exists: bool = False
|
||||||
version_label: str | None = None
|
version_label: str | None = None
|
||||||
actor_id: int | None = None
|
actor_id: int | None = None
|
||||||
|
remote_ocr: bool = False
|
||||||
|
|
||||||
def update(self, other: "DocumentMetadataOverrides") -> "DocumentMetadataOverrides":
|
def update(self, other: "DocumentMetadataOverrides") -> "DocumentMetadataOverrides":
|
||||||
"""
|
"""
|
||||||
@@ -57,6 +58,8 @@ class DocumentMetadataOverrides:
|
|||||||
self.actor_id = other.actor_id
|
self.actor_id = other.actor_id
|
||||||
if other.skip_asn_if_exists:
|
if other.skip_asn_if_exists:
|
||||||
self.skip_asn_if_exists = True
|
self.skip_asn_if_exists = True
|
||||||
|
if other.remote_ocr:
|
||||||
|
self.remote_ocr = True
|
||||||
if other.version_label is not None:
|
if other.version_label is not None:
|
||||||
self.version_label = other.version_label
|
self.version_label = other.version_label
|
||||||
|
|
||||||
|
|||||||
@@ -1047,6 +1047,12 @@ class PermittedObjectsFilter(BaseFilterBackend):
|
|||||||
perm_codename: str | None = None
|
perm_codename: str | None = None
|
||||||
|
|
||||||
def filter_queryset(self, request, queryset, view):
|
def filter_queryset(self, request, queryset, view):
|
||||||
|
# Before the superuser and owner-only paths, neither of which consults
|
||||||
|
# permitted_object_ids. Scoped to authenticated users so anonymous
|
||||||
|
# access (AnonymousUser.is_active is False) keeps its existing
|
||||||
|
# unowned-only behaviour.
|
||||||
|
if request.user.is_authenticated and not request.user.is_active:
|
||||||
|
return queryset.none()
|
||||||
if request.user.is_superuser:
|
if request.user.is_superuser:
|
||||||
return queryset
|
return queryset
|
||||||
if not self.include_granted:
|
if not self.include_granted:
|
||||||
|
|||||||
@@ -54,11 +54,15 @@ class PaperlessObjectPermissions(DjangoObjectPermissions):
|
|||||||
|
|
||||||
class PaperlessAdminPermissions(BasePermission):
|
class PaperlessAdminPermissions(BasePermission):
|
||||||
def has_permission(self, request, view):
|
def has_permission(self, request, view):
|
||||||
return request.user.is_staff
|
return request.user.is_active and request.user.is_staff
|
||||||
|
|
||||||
|
|
||||||
def has_global_statistics_permission(user: User | None) -> bool:
|
def has_global_statistics_permission(user: User | None) -> bool:
|
||||||
if user is None or not getattr(user, "is_authenticated", False):
|
if (
|
||||||
|
user is None
|
||||||
|
or not getattr(user, "is_active", False)
|
||||||
|
or not getattr(user, "is_authenticated", False)
|
||||||
|
):
|
||||||
return False
|
return False
|
||||||
|
|
||||||
return getattr(user, "is_superuser", False) or user.has_perm(
|
return getattr(user, "is_superuser", False) or user.has_perm(
|
||||||
@@ -67,7 +71,11 @@ def has_global_statistics_permission(user: User | None) -> bool:
|
|||||||
|
|
||||||
|
|
||||||
def has_system_status_permission(user: User | None) -> bool:
|
def has_system_status_permission(user: User | None) -> bool:
|
||||||
if user is None or not getattr(user, "is_authenticated", False):
|
if (
|
||||||
|
user is None
|
||||||
|
or not getattr(user, "is_active", False)
|
||||||
|
or not getattr(user, "is_authenticated", False)
|
||||||
|
):
|
||||||
return False
|
return False
|
||||||
|
|
||||||
return (
|
return (
|
||||||
@@ -188,6 +196,13 @@ def permitted_object_ids(
|
|||||||
if user is None or not getattr(user, "is_authenticated", False):
|
if user is None or not getattr(user, "is_authenticated", False):
|
||||||
return base_qs.filter(owner__isnull=True).values_list("id", flat=True)
|
return base_qs.filter(owner__isnull=True).values_list("id", flat=True)
|
||||||
|
|
||||||
|
# Deactivated users get nothing, deactivated superusers included, so this
|
||||||
|
# has to come before the superuser shortcut. guardian's
|
||||||
|
# ObjectPermissionChecker denies inactive users, but get_objects_for_user
|
||||||
|
# (the pattern this replaces) does not, so it would not be inherited.
|
||||||
|
if not getattr(user, "is_active", False):
|
||||||
|
return base_qs.none().values_list("id", flat=True)
|
||||||
|
|
||||||
if getattr(user, "is_superuser", False):
|
if getattr(user, "is_superuser", False):
|
||||||
return base_qs.values_list("id", flat=True)
|
return base_qs.values_list("id", flat=True)
|
||||||
|
|
||||||
|
|||||||
+10
-1
@@ -66,6 +66,7 @@ from documents.utils import compute_checksum
|
|||||||
from documents.utils import identity
|
from documents.utils import identity
|
||||||
from documents.workflows.utils import get_workflows_for_trigger
|
from documents.workflows.utils import get_workflows_for_trigger
|
||||||
from paperless.config import AIConfig
|
from paperless.config import AIConfig
|
||||||
|
from paperless.config import RemoteOCRConfig
|
||||||
from paperless.logging import consume_task_id
|
from paperless.logging import consume_task_id
|
||||||
from paperless.parsers import ParserContext
|
from paperless.parsers import ParserContext
|
||||||
from paperless.parsers.registry import get_parser_registry
|
from paperless.parsers.registry import get_parser_registry
|
||||||
@@ -337,10 +338,17 @@ def bulk_update_documents(document_ids) -> None:
|
|||||||
|
|
||||||
|
|
||||||
@shared_task
|
@shared_task
|
||||||
def update_document_content_maybe_archive_file(document_id) -> None:
|
def update_document_content_maybe_archive_file(
|
||||||
|
document_id,
|
||||||
|
*,
|
||||||
|
remote_ocr: bool = False,
|
||||||
|
) -> None:
|
||||||
"""
|
"""
|
||||||
Re-creates OCR content and thumbnail for a document, and archive file if
|
Re-creates OCR content and thumbnail for a document, and archive file if
|
||||||
it exists.
|
it exists.
|
||||||
|
|
||||||
|
Remote OCR is used only when the engine is configured to handle everything
|
||||||
|
or if explicitly asked for via ``remote_ocr``.
|
||||||
"""
|
"""
|
||||||
document = Document.objects.get(id=document_id)
|
document = Document.objects.get(id=document_id)
|
||||||
|
|
||||||
@@ -350,6 +358,7 @@ def update_document_content_maybe_archive_file(document_id) -> None:
|
|||||||
mime_type,
|
mime_type,
|
||||||
document.original_filename or "",
|
document.original_filename or "",
|
||||||
document.source_path,
|
document.source_path,
|
||||||
|
allow_remote=remote_ocr or RemoteOCRConfig().remote_ocr_by_default,
|
||||||
)
|
)
|
||||||
|
|
||||||
if not parser_class:
|
if not parser_class:
|
||||||
|
|||||||
@@ -72,6 +72,10 @@ class TestApiAppConfig(DirectoriesMixin, APITestCase):
|
|||||||
"barcode_enable_tag": None,
|
"barcode_enable_tag": None,
|
||||||
"barcode_tag_mapping": None,
|
"barcode_tag_mapping": None,
|
||||||
"barcode_tag_split": None,
|
"barcode_tag_split": None,
|
||||||
|
"remote_ocr_engine": None,
|
||||||
|
"remote_ocr_api_key": None,
|
||||||
|
"remote_ocr_endpoint": None,
|
||||||
|
"remote_ocr_mode": None,
|
||||||
"ai_enabled": False,
|
"ai_enabled": False,
|
||||||
"llm_embedding_backend": None,
|
"llm_embedding_backend": None,
|
||||||
"llm_embedding_model": None,
|
"llm_embedding_model": None,
|
||||||
@@ -870,6 +874,49 @@ class TestApiAppConfig(DirectoriesMixin, APITestCase):
|
|||||||
config.refresh_from_db()
|
config.refresh_from_db()
|
||||||
self.assertEqual(config.llm_api_key, None)
|
self.assertEqual(config.llm_api_key, None)
|
||||||
|
|
||||||
|
def test_update_remote_ocr_api_key(self) -> None:
|
||||||
|
"""
|
||||||
|
GIVEN:
|
||||||
|
- Existing config with remote_ocr_api_key specified
|
||||||
|
WHEN:
|
||||||
|
- API to update remote_ocr_api_key is called with all *s
|
||||||
|
- API to update remote_ocr_api_key is called with empty string
|
||||||
|
THEN:
|
||||||
|
- remote_ocr_api_key is unchanged
|
||||||
|
- remote_ocr_api_key is set to None
|
||||||
|
"""
|
||||||
|
config = ApplicationConfiguration.objects.first()
|
||||||
|
assert config is not None
|
||||||
|
config.remote_ocr_api_key = "1234567890"
|
||||||
|
config.save()
|
||||||
|
|
||||||
|
# Test with all *
|
||||||
|
response = self.client.patch(
|
||||||
|
f"{self.ENDPOINT}1/",
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"remote_ocr_api_key": "*" * 32,
|
||||||
|
},
|
||||||
|
),
|
||||||
|
content_type="application/json",
|
||||||
|
)
|
||||||
|
self.assertEqual(response.status_code, status.HTTP_200_OK)
|
||||||
|
config.refresh_from_db()
|
||||||
|
self.assertEqual(config.remote_ocr_api_key, "1234567890")
|
||||||
|
# Test with empty string
|
||||||
|
response = self.client.patch(
|
||||||
|
f"{self.ENDPOINT}1/",
|
||||||
|
json.dumps(
|
||||||
|
{
|
||||||
|
"remote_ocr_api_key": "",
|
||||||
|
},
|
||||||
|
),
|
||||||
|
content_type="application/json",
|
||||||
|
)
|
||||||
|
self.assertEqual(response.status_code, status.HTTP_200_OK)
|
||||||
|
config.refresh_from_db()
|
||||||
|
self.assertEqual(config.remote_ocr_api_key, None)
|
||||||
|
|
||||||
def test_enable_ai_index_triggers_update(self) -> None:
|
def test_enable_ai_index_triggers_update(self) -> None:
|
||||||
"""
|
"""
|
||||||
GIVEN:
|
GIVEN:
|
||||||
|
|||||||
@@ -1559,6 +1559,72 @@ class PostConsumeTestCase(DirectoriesMixin, GetConsumerMixin, TestCase):
|
|||||||
consumer.run_post_consume_script(doc)
|
consumer.run_post_consume_script(doc)
|
||||||
|
|
||||||
|
|
||||||
|
class TestConsumerRemoteOCR(
|
||||||
|
DirectoriesMixin,
|
||||||
|
FileSystemAssertsMixin,
|
||||||
|
GetConsumerMixin,
|
||||||
|
TestCase,
|
||||||
|
):
|
||||||
|
"""
|
||||||
|
The consumer resolves the remote OCR mode and the per-document request from
|
||||||
|
workflows into the allow_remote flag it hands to the parser registry.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def setUp(self) -> None:
|
||||||
|
super().setUp()
|
||||||
|
|
||||||
|
patcher = mock.patch("documents.consumer.get_parser_registry")
|
||||||
|
self.mock_registry = patcher.start()
|
||||||
|
self.mock_registry.return_value.get_parser_for_file.return_value = DummyParser
|
||||||
|
self.addCleanup(patcher.stop)
|
||||||
|
|
||||||
|
def _consume(self, *, overrides: DocumentMetadataOverrides | None = None) -> bool:
|
||||||
|
src = (
|
||||||
|
Path(__file__).parent
|
||||||
|
/ "samples"
|
||||||
|
/ "documents"
|
||||||
|
/ "originals"
|
||||||
|
/ "0000001.pdf"
|
||||||
|
)
|
||||||
|
dst = self.dirs.scratch_dir / "sample.pdf"
|
||||||
|
shutil.copy(src, dst)
|
||||||
|
|
||||||
|
with self.get_consumer(dst, overrides=overrides) as consumer:
|
||||||
|
consumer.run()
|
||||||
|
|
||||||
|
_, kwargs = self.mock_registry.return_value.get_parser_for_file.call_args
|
||||||
|
return kwargs["allow_remote"]
|
||||||
|
|
||||||
|
@override_settings(REMOTE_OCR_MODE="always")
|
||||||
|
def test_always_mode_allows_remote(self) -> None:
|
||||||
|
"""
|
||||||
|
GIVEN: Remote OCR mode is 'always'.
|
||||||
|
WHEN: A document is consumed without any workflow asking for it.
|
||||||
|
THEN: The registry is allowed to pick the remote parser.
|
||||||
|
"""
|
||||||
|
self.assertTrue(self._consume())
|
||||||
|
|
||||||
|
@override_settings(REMOTE_OCR_MODE="workflow_only")
|
||||||
|
def test_workflow_only_mode_denies_remote_by_default(self) -> None:
|
||||||
|
"""
|
||||||
|
GIVEN: Remote OCR mode is 'workflow_only'.
|
||||||
|
WHEN: A document is consumed and nothing asked for remote OCR.
|
||||||
|
THEN: The remote parser is excluded.
|
||||||
|
"""
|
||||||
|
self.assertFalse(self._consume())
|
||||||
|
|
||||||
|
@override_settings(REMOTE_OCR_MODE="workflow_only")
|
||||||
|
def test_workflow_only_mode_allows_remote_when_requested(self) -> None:
|
||||||
|
"""
|
||||||
|
GIVEN: Remote OCR mode is 'workflow_only'.
|
||||||
|
WHEN: A workflow set remote_ocr on the metadata overrides.
|
||||||
|
THEN: The registry is allowed to pick the remote parser.
|
||||||
|
"""
|
||||||
|
self.assertTrue(
|
||||||
|
self._consume(overrides=DocumentMetadataOverrides(remote_ocr=True)),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
class TestMetadataOverrides(TestCase):
|
class TestMetadataOverrides(TestCase):
|
||||||
def test_update_skip_asn_if_exists(self) -> None:
|
def test_update_skip_asn_if_exists(self) -> None:
|
||||||
base = DocumentMetadataOverrides()
|
base = DocumentMetadataOverrides()
|
||||||
@@ -1566,6 +1632,20 @@ class TestMetadataOverrides(TestCase):
|
|||||||
base.update(incoming)
|
base.update(incoming)
|
||||||
self.assertTrue(base.skip_asn_if_exists)
|
self.assertTrue(base.skip_asn_if_exists)
|
||||||
|
|
||||||
|
def test_update_remote_ocr(self) -> None:
|
||||||
|
base = DocumentMetadataOverrides()
|
||||||
|
base.update(DocumentMetadataOverrides(remote_ocr=True))
|
||||||
|
self.assertTrue(base.remote_ocr)
|
||||||
|
|
||||||
|
def test_update_remote_ocr_is_not_unset(self) -> None:
|
||||||
|
"""
|
||||||
|
A later workflow that says nothing must not undo an earlier one that
|
||||||
|
asked for remote OCR.
|
||||||
|
"""
|
||||||
|
base = DocumentMetadataOverrides(remote_ocr=True)
|
||||||
|
base.update(DocumentMetadataOverrides())
|
||||||
|
self.assertTrue(base.remote_ocr)
|
||||||
|
|
||||||
def test_update_actor_and_version_label(self) -> None:
|
def test_update_actor_and_version_label(self) -> None:
|
||||||
base = DocumentMetadataOverrides(
|
base = DocumentMetadataOverrides(
|
||||||
actor_id=1,
|
actor_id=1,
|
||||||
|
|||||||
@@ -496,6 +496,28 @@ class TestPermittedObjectIdsGenericModels:
|
|||||||
expected_hidden=[strangers.pk],
|
expected_hidden=[strangers.pk],
|
||||||
)
|
)
|
||||||
|
|
||||||
|
@pytest.mark.parametrize("is_superuser", [False, True])
|
||||||
|
def test_inactive_user_sees_nothing(self, model, factory, perm, is_superuser):
|
||||||
|
suffix = f"{model.__name__}_{is_superuser}"
|
||||||
|
user = User.objects.create_user(
|
||||||
|
username=f"inactive_{suffix}",
|
||||||
|
is_active=False,
|
||||||
|
is_superuser=is_superuser,
|
||||||
|
)
|
||||||
|
other = User.objects.create_user(username=f"other_{suffix}")
|
||||||
|
granted = factory(owner=other)
|
||||||
|
assign_perm(perm, user, granted)
|
||||||
|
|
||||||
|
assert_visible_document_ids(
|
||||||
|
permitted_object_ids(user, model, perm),
|
||||||
|
expected_visible=[],
|
||||||
|
expected_hidden=[
|
||||||
|
factory(owner=None).pk,
|
||||||
|
factory(owner=user).pk,
|
||||||
|
granted.pk,
|
||||||
|
],
|
||||||
|
)
|
||||||
|
|
||||||
def test_unowned_object_visible_to_everyone(self, model, factory, perm):
|
def test_unowned_object_visible_to_everyone(self, model, factory, perm):
|
||||||
user = User.objects.create_user(username=f"user_{model.__name__}")
|
user = User.objects.create_user(username=f"user_{model.__name__}")
|
||||||
unowned = factory(owner=None)
|
unowned = factory(owner=None)
|
||||||
|
|||||||
@@ -68,3 +68,44 @@ class TestPermittedObjectsFilter:
|
|||||||
visible_ids = set(result.values_list("id", flat=True))
|
visible_ids = set(result.values_list("id", flat=True))
|
||||||
assert visible_ids == {owned.pk}
|
assert visible_ids == {owned.pk}
|
||||||
assert granted.pk not in visible_ids
|
assert granted.pk not in visible_ids
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("username", "is_superuser"),
|
||||||
|
[("inactive", False), ("inactive_super", True)],
|
||||||
|
)
|
||||||
|
def test_inactive_user_sees_nothing(self, username: str, *, is_superuser: bool):
|
||||||
|
user = User.objects.create_user(
|
||||||
|
username=username,
|
||||||
|
is_active=False,
|
||||||
|
is_superuser=is_superuser,
|
||||||
|
)
|
||||||
|
TagFactory(owner=None)
|
||||||
|
TagFactory(owner=user)
|
||||||
|
granted = TagFactory(owner=User.objects.create_user(username=f"o_{username}"))
|
||||||
|
assign_perm("view_tag", user, granted)
|
||||||
|
request = APIRequestFactory().get("/")
|
||||||
|
request.user = user
|
||||||
|
|
||||||
|
result = PermittedObjectsFilter().filter_queryset(
|
||||||
|
request,
|
||||||
|
Tag.objects.all(),
|
||||||
|
_DummyView(),
|
||||||
|
)
|
||||||
|
assert result.count() == 0
|
||||||
|
|
||||||
|
def test_inactive_user_sees_nothing_with_include_granted_false(self):
|
||||||
|
user = User.objects.create_user(username="inactive_owner", is_active=False)
|
||||||
|
TagFactory(owner=user)
|
||||||
|
TagFactory(owner=None)
|
||||||
|
request = APIRequestFactory().get("/")
|
||||||
|
request.user = user
|
||||||
|
|
||||||
|
class _OwnerOnlyFilter(PermittedObjectsFilter):
|
||||||
|
include_granted = False
|
||||||
|
|
||||||
|
result = _OwnerOnlyFilter().filter_queryset(
|
||||||
|
request,
|
||||||
|
Tag.objects.all(),
|
||||||
|
_DummyView(),
|
||||||
|
)
|
||||||
|
assert result.count() == 0
|
||||||
|
|||||||
@@ -287,6 +287,45 @@ class TestUpdateContent(DirectoriesMixin, TestCase):
|
|||||||
self.assertNotEqual(Document.objects.get(pk=doc.pk).content, "test")
|
self.assertNotEqual(Document.objects.get(pk=doc.pk).content, "test")
|
||||||
|
|
||||||
|
|
||||||
|
class TestUpdateContentRemoteOCR(DirectoriesMixin, TestCase):
|
||||||
|
"""
|
||||||
|
Consumption workflows do not run on reprocess, so the remote parser is
|
||||||
|
used only in 'always' mode or when the caller explicitly asks for it.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def setUp(self) -> None:
|
||||||
|
super().setUp()
|
||||||
|
|
||||||
|
patcher = mock.patch("documents.tasks.get_parser_registry")
|
||||||
|
self.mock_registry = patcher.start()
|
||||||
|
self.mock_registry.return_value.get_parser_for_file.return_value = None
|
||||||
|
self.addCleanup(patcher.stop)
|
||||||
|
|
||||||
|
self.doc = Document.objects.create(
|
||||||
|
title="test",
|
||||||
|
content="my document",
|
||||||
|
checksum="wow",
|
||||||
|
mime_type="application/pdf",
|
||||||
|
)
|
||||||
|
|
||||||
|
def _allow_remote(self, **kwargs) -> bool:
|
||||||
|
tasks.update_document_content_maybe_archive_file(self.doc.pk, **kwargs)
|
||||||
|
_, call_kwargs = self.mock_registry.return_value.get_parser_for_file.call_args
|
||||||
|
return call_kwargs["allow_remote"]
|
||||||
|
|
||||||
|
@override_settings(REMOTE_OCR_MODE="always")
|
||||||
|
def test_always_mode_allows_remote(self) -> None:
|
||||||
|
self.assertTrue(self._allow_remote())
|
||||||
|
|
||||||
|
@override_settings(REMOTE_OCR_MODE="workflow_only")
|
||||||
|
def test_workflow_only_mode_denies_remote_by_default(self) -> None:
|
||||||
|
self.assertFalse(self._allow_remote())
|
||||||
|
|
||||||
|
@override_settings(REMOTE_OCR_MODE="workflow_only")
|
||||||
|
def test_workflow_only_mode_allows_remote_when_requested(self) -> None:
|
||||||
|
self.assertTrue(self._allow_remote(remote_ocr=True))
|
||||||
|
|
||||||
|
|
||||||
class TestAIIndex(DirectoriesMixin, TestCase):
|
class TestAIIndex(DirectoriesMixin, TestCase):
|
||||||
@override_settings(
|
@override_settings(
|
||||||
AI_ENABLED=True,
|
AI_ENABLED=True,
|
||||||
|
|||||||
@@ -2267,7 +2267,7 @@ class ChatStreamingView(GenericAPIView[Any]):
|
|||||||
if not has_perms_owner_aware(request.user, "view_document", document):
|
if not has_perms_owner_aware(request.user, "view_document", document):
|
||||||
return HttpResponseForbidden("Insufficient permissions")
|
return HttpResponseForbidden("Insufficient permissions")
|
||||||
|
|
||||||
documents = Document.objects.filter(pk=document.pk)
|
documents = [document]
|
||||||
else:
|
else:
|
||||||
documents = Document.objects.filter(
|
documents = Document.objects.filter(
|
||||||
id__in=permitted_document_ids(request.user),
|
id__in=permitted_document_ids(request.user),
|
||||||
|
|||||||
@@ -2,7 +2,7 @@ msgid ""
|
|||||||
msgstr ""
|
msgstr ""
|
||||||
"Project-Id-Version: paperless-ngx\n"
|
"Project-Id-Version: paperless-ngx\n"
|
||||||
"Report-Msgid-Bugs-To: \n"
|
"Report-Msgid-Bugs-To: \n"
|
||||||
"POT-Creation-Date: 2026-08-07 20:00+0000\n"
|
"POT-Creation-Date: 2026-08-10 02:25+0000\n"
|
||||||
"PO-Revision-Date: 2022-02-17 04:17\n"
|
"PO-Revision-Date: 2022-02-17 04:17\n"
|
||||||
"Last-Translator: \n"
|
"Last-Translator: \n"
|
||||||
"Language-Team: English\n"
|
"Language-Team: English\n"
|
||||||
@@ -21,39 +21,39 @@ msgstr ""
|
|||||||
msgid "Documents"
|
msgid "Documents"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/filters.py:472
|
#: documents/filters.py:471
|
||||||
msgid "Value must be valid JSON."
|
msgid "Value must be valid JSON."
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/filters.py:491
|
#: documents/filters.py:490
|
||||||
msgid "Invalid custom field query expression"
|
msgid "Invalid custom field query expression"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/filters.py:501
|
#: documents/filters.py:500
|
||||||
msgid "Invalid expression list. Must be nonempty."
|
msgid "Invalid expression list. Must be nonempty."
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/filters.py:522
|
#: documents/filters.py:521
|
||||||
msgid "Invalid logical operator {op!r}"
|
msgid "Invalid logical operator {op!r}"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/filters.py:536
|
#: documents/filters.py:535
|
||||||
msgid "Maximum number of query conditions exceeded."
|
msgid "Maximum number of query conditions exceeded."
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/filters.py:600
|
#: documents/filters.py:599
|
||||||
msgid "{name!r} is not a valid custom field."
|
msgid "{name!r} is not a valid custom field."
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/filters.py:637
|
#: documents/filters.py:636
|
||||||
msgid "{data_type} does not support query expr {expr!r}."
|
msgid "{data_type} does not support query expr {expr!r}."
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/filters.py:756 documents/models.py:136
|
#: documents/filters.py:755 documents/models.py:136
|
||||||
msgid "Maximum nesting depth exceeded."
|
msgid "Maximum nesting depth exceeded."
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/filters.py:1098
|
#: documents/filters.py:1079
|
||||||
msgid "Custom field not found"
|
msgid "Custom field not found"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
@@ -1352,7 +1352,7 @@ msgid "workflow runs"
|
|||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/serialisers.py:521 documents/serialisers.py:873
|
#: documents/serialisers.py:521 documents/serialisers.py:873
|
||||||
#: documents/serialisers.py:2767 documents/views.py:300 documents/views.py:2556
|
#: documents/serialisers.py:2767 documents/views.py:299 documents/views.py:2555
|
||||||
#: paperless_mail/serialisers.py:155
|
#: paperless_mail/serialisers.py:155
|
||||||
msgid "Insufficient permissions."
|
msgid "Insufficient permissions."
|
||||||
msgstr ""
|
msgstr ""
|
||||||
@@ -1393,7 +1393,7 @@ msgstr ""
|
|||||||
msgid "Duplicate document identifiers are not allowed."
|
msgid "Duplicate document identifiers are not allowed."
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/serialisers.py:2853 documents/views.py:4510
|
#: documents/serialisers.py:2853 documents/views.py:4509
|
||||||
#, python-format
|
#, python-format
|
||||||
msgid "Documents not found: %(ids)s"
|
msgid "Documents not found: %(ids)s"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
@@ -1661,36 +1661,36 @@ msgstr ""
|
|||||||
msgid "Unable to parse URI {value}"
|
msgid "Unable to parse URI {value}"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/views.py:293 documents/views.py:2553
|
#: documents/views.py:292 documents/views.py:2552
|
||||||
msgid "Invalid more_like_id"
|
msgid "Invalid more_like_id"
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/views.py:1567
|
#: documents/views.py:1566
|
||||||
msgid "Invalid AI configuration."
|
msgid "Invalid AI configuration."
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/views.py:1576
|
#: documents/views.py:1575
|
||||||
msgid "AI backend request timed out."
|
msgid "AI backend request timed out."
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/views.py:2378 documents/views.py:2699
|
#: documents/views.py:2377 documents/views.py:2698
|
||||||
msgid "Specify only one of text, title_search, query, or more_like_id."
|
msgid "Specify only one of text, title_search, query, or more_like_id."
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/views.py:4523
|
#: documents/views.py:4522
|
||||||
#, python-format
|
#, python-format
|
||||||
msgid "Insufficient permissions to share document %(id)s."
|
msgid "Insufficient permissions to share document %(id)s."
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/views.py:4569
|
#: documents/views.py:4568
|
||||||
msgid "Bundle is already being processed."
|
msgid "Bundle is already being processed."
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/views.py:4630
|
#: documents/views.py:4629
|
||||||
msgid "The share link bundle is still being prepared. Please try again later."
|
msgid "The share link bundle is still being prepared. Please try again later."
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
#: documents/views.py:4640
|
#: documents/views.py:4639
|
||||||
msgid "The share link bundle is unavailable."
|
msgid "The share link bundle is unavailable."
|
||||||
msgstr ""
|
msgstr ""
|
||||||
|
|
||||||
|
|||||||
@@ -19,7 +19,10 @@ class AutoLoginMiddleware(MiddlewareMixin):
|
|||||||
if request.path.startswith("/api/token/") and request.method == "POST":
|
if request.path.startswith("/api/token/") and request.method == "POST":
|
||||||
return None
|
return None
|
||||||
try:
|
try:
|
||||||
request.user = User.objects.get(username=settings.AUTO_LOGIN_USERNAME)
|
request.user = User.objects.get(
|
||||||
|
username=settings.AUTO_LOGIN_USERNAME,
|
||||||
|
is_active=True,
|
||||||
|
)
|
||||||
auth.login(
|
auth.login(
|
||||||
request=request,
|
request=request,
|
||||||
user=request.user,
|
user=request.user,
|
||||||
|
|||||||
+17
-3
@@ -339,16 +339,30 @@ def check_deprecated_v2_ocr_env_vars(
|
|||||||
|
|
||||||
@register()
|
@register()
|
||||||
def check_remote_parser_configured(app_configs: Any, **kwargs: Any) -> list[Error]:
|
def check_remote_parser_configured(app_configs: Any, **kwargs: Any) -> list[Error]:
|
||||||
|
# Import here because checks.py runs before the app registry is ready
|
||||||
|
from paperless.models import RemoteOCRMode
|
||||||
|
|
||||||
|
errors = []
|
||||||
|
|
||||||
if settings.REMOTE_OCR_ENGINE == "azureai" and not (
|
if settings.REMOTE_OCR_ENGINE == "azureai" and not (
|
||||||
settings.REMOTE_OCR_ENDPOINT and settings.REMOTE_OCR_API_KEY
|
settings.REMOTE_OCR_ENDPOINT and settings.REMOTE_OCR_API_KEY
|
||||||
):
|
):
|
||||||
return [
|
errors.append(
|
||||||
Error(
|
Error(
|
||||||
"Azure AI remote parser requires endpoint and API key to be configured.",
|
"Azure AI remote parser requires endpoint and API key to be configured.",
|
||||||
),
|
),
|
||||||
]
|
)
|
||||||
|
|
||||||
return []
|
valid_modes = {mode.value for mode in RemoteOCRMode}
|
||||||
|
if settings.REMOTE_OCR_MODE not in valid_modes:
|
||||||
|
errors.append(
|
||||||
|
Error(
|
||||||
|
f"PAPERLESS_REMOTE_OCR_MODE is set to {settings.REMOTE_OCR_MODE!r}, "
|
||||||
|
f"expected one of {sorted(valid_modes)}.",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
return errors
|
||||||
|
|
||||||
|
|
||||||
def get_tesseract_langs():
|
def get_tesseract_langs():
|
||||||
|
|||||||
@@ -9,6 +9,7 @@ from paperless.models import CleanChoices
|
|||||||
from paperless.models import ColorConvertChoices
|
from paperless.models import ColorConvertChoices
|
||||||
from paperless.models import ModeChoices
|
from paperless.models import ModeChoices
|
||||||
from paperless.models import OutputTypeChoices
|
from paperless.models import OutputTypeChoices
|
||||||
|
from paperless.models import RemoteOCRMode
|
||||||
|
|
||||||
|
|
||||||
@dataclasses.dataclass
|
@dataclasses.dataclass
|
||||||
@@ -185,6 +186,45 @@ class GeneralConfig(BaseConfig):
|
|||||||
self.app_logo = app_config.app_logo.url if app_config.app_logo else None
|
self.app_logo = app_config.app_logo.url if app_config.app_logo else None
|
||||||
|
|
||||||
|
|
||||||
|
@dataclasses.dataclass
|
||||||
|
class RemoteOCRConfig(BaseConfig):
|
||||||
|
"""
|
||||||
|
Settings for the remote (cloud) OCR parser
|
||||||
|
"""
|
||||||
|
|
||||||
|
remote_ocr_engine: str | None = dataclasses.field(init=False)
|
||||||
|
remote_ocr_api_key: str | None = dataclasses.field(init=False)
|
||||||
|
remote_ocr_endpoint: str | None = dataclasses.field(init=False)
|
||||||
|
remote_ocr_mode: RemoteOCRMode = dataclasses.field(init=False)
|
||||||
|
|
||||||
|
def __post_init__(self) -> None:
|
||||||
|
app_config = self._get_config_instance()
|
||||||
|
|
||||||
|
self.remote_ocr_engine = (
|
||||||
|
app_config.remote_ocr_engine or settings.REMOTE_OCR_ENGINE
|
||||||
|
)
|
||||||
|
self.remote_ocr_api_key = (
|
||||||
|
app_config.remote_ocr_api_key or settings.REMOTE_OCR_API_KEY
|
||||||
|
)
|
||||||
|
self.remote_ocr_endpoint = (
|
||||||
|
app_config.remote_ocr_endpoint or settings.REMOTE_OCR_ENDPOINT
|
||||||
|
)
|
||||||
|
self.remote_ocr_mode = app_config.remote_ocr_mode or RemoteOCRMode(
|
||||||
|
settings.REMOTE_OCR_MODE,
|
||||||
|
)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def remote_ocr_by_default(self) -> bool:
|
||||||
|
"""
|
||||||
|
Whether every supported document goes to the remote engine.
|
||||||
|
|
||||||
|
When False the remote engine is used only for documents that
|
||||||
|
explicitly asked for it, i.e. a workflow matched during consumption or
|
||||||
|
the user ticked the box when reprocessing.
|
||||||
|
"""
|
||||||
|
return self.remote_ocr_mode == RemoteOCRMode.ALWAYS
|
||||||
|
|
||||||
|
|
||||||
@dataclasses.dataclass
|
@dataclasses.dataclass
|
||||||
class AIConfig(BaseConfig):
|
class AIConfig(BaseConfig):
|
||||||
"""
|
"""
|
||||||
|
|||||||
@@ -0,0 +1,44 @@
|
|||||||
|
# Generated by Django 5.2.16 on 2026-08-10 14:37
|
||||||
|
|
||||||
|
from django.db import migrations
|
||||||
|
from django.db import models
|
||||||
|
|
||||||
|
|
||||||
|
class Migration(migrations.Migration):
|
||||||
|
dependencies = [
|
||||||
|
("paperless", "0013_applicationconfiguration_llm_request_timeout"),
|
||||||
|
]
|
||||||
|
|
||||||
|
operations = [
|
||||||
|
migrations.AddField(
|
||||||
|
model_name="applicationconfiguration",
|
||||||
|
name="remote_ocr_api_key",
|
||||||
|
field=models.CharField(
|
||||||
|
blank=True,
|
||||||
|
max_length=1024,
|
||||||
|
null=True,
|
||||||
|
verbose_name="Sets the remote OCR API key",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
migrations.AddField(
|
||||||
|
model_name="applicationconfiguration",
|
||||||
|
name="remote_ocr_endpoint",
|
||||||
|
field=models.CharField(
|
||||||
|
blank=True,
|
||||||
|
max_length=256,
|
||||||
|
null=True,
|
||||||
|
verbose_name="Sets the remote OCR endpoint",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
migrations.AddField(
|
||||||
|
model_name="applicationconfiguration",
|
||||||
|
name="remote_ocr_engine",
|
||||||
|
field=models.CharField(
|
||||||
|
blank=True,
|
||||||
|
choices=[("azureai", "Azure AI Document Intelligence")],
|
||||||
|
max_length=32,
|
||||||
|
null=True,
|
||||||
|
verbose_name="Sets the remote OCR engine",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
]
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
# Generated by Django 5.2.16 on 2026-08-10 15:43
|
||||||
|
|
||||||
|
from django.db import migrations
|
||||||
|
from django.db import models
|
||||||
|
|
||||||
|
|
||||||
|
class Migration(migrations.Migration):
|
||||||
|
dependencies = [
|
||||||
|
("paperless", "0014_applicationconfiguration_remote_ocr_api_key_and_more"),
|
||||||
|
]
|
||||||
|
|
||||||
|
operations = [
|
||||||
|
migrations.AddField(
|
||||||
|
model_name="applicationconfiguration",
|
||||||
|
name="remote_ocr_mode",
|
||||||
|
field=models.CharField(
|
||||||
|
blank=True,
|
||||||
|
choices=[
|
||||||
|
("always", "All supported documents"),
|
||||||
|
("workflow_only", "Only when a workflow enables it"),
|
||||||
|
],
|
||||||
|
max_length=32,
|
||||||
|
null=True,
|
||||||
|
verbose_name="Sets which documents are sent to the remote OCR engine",
|
||||||
|
),
|
||||||
|
),
|
||||||
|
]
|
||||||
@@ -74,6 +74,23 @@ class ColorConvertChoices(models.TextChoices):
|
|||||||
CMYK = ("CMYK", _("CMYK"))
|
CMYK = ("CMYK", _("CMYK"))
|
||||||
|
|
||||||
|
|
||||||
|
class RemoteOCREngine(models.TextChoices):
|
||||||
|
"""
|
||||||
|
Matches to PAPERLESS_REMOTE_OCR_ENGINE
|
||||||
|
"""
|
||||||
|
|
||||||
|
AZURE_AI = ("azureai", _("Azure AI Document Intelligence"))
|
||||||
|
|
||||||
|
|
||||||
|
class RemoteOCRMode(models.TextChoices):
|
||||||
|
"""
|
||||||
|
Matches to PAPERLESS_REMOTE_OCR_MODE
|
||||||
|
"""
|
||||||
|
|
||||||
|
ALWAYS = ("always", _("All supported documents"))
|
||||||
|
WORKFLOW_ONLY = ("workflow_only", _("Only when a workflow enables it"))
|
||||||
|
|
||||||
|
|
||||||
class LLMEmbeddingBackend(models.TextChoices):
|
class LLMEmbeddingBackend(models.TextChoices):
|
||||||
OPENAI_LIKE = ("openai-like", _("OpenAI-compatible"))
|
OPENAI_LIKE = ("openai-like", _("OpenAI-compatible"))
|
||||||
HUGGINGFACE = ("huggingface", _("Huggingface"))
|
HUGGINGFACE = ("huggingface", _("Huggingface"))
|
||||||
@@ -286,6 +303,44 @@ class ApplicationConfiguration(AbstractSingletonModel):
|
|||||||
null=True,
|
null=True,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
"""
|
||||||
|
Settings for the remote OCR parser
|
||||||
|
"""
|
||||||
|
|
||||||
|
# PAPERLESS_REMOTE_OCR_ENGINE
|
||||||
|
remote_ocr_engine = models.CharField(
|
||||||
|
verbose_name=_("Sets the remote OCR engine"),
|
||||||
|
blank=True,
|
||||||
|
null=True,
|
||||||
|
max_length=32,
|
||||||
|
choices=RemoteOCREngine.choices,
|
||||||
|
)
|
||||||
|
|
||||||
|
# PAPERLESS_REMOTE_OCR_API_KEY
|
||||||
|
remote_ocr_api_key = models.CharField(
|
||||||
|
verbose_name=_("Sets the remote OCR API key"),
|
||||||
|
blank=True,
|
||||||
|
null=True,
|
||||||
|
max_length=1024,
|
||||||
|
)
|
||||||
|
|
||||||
|
# PAPERLESS_REMOTE_OCR_ENDPOINT
|
||||||
|
remote_ocr_endpoint = models.CharField(
|
||||||
|
verbose_name=_("Sets the remote OCR endpoint"),
|
||||||
|
blank=True,
|
||||||
|
null=True,
|
||||||
|
max_length=256,
|
||||||
|
)
|
||||||
|
|
||||||
|
# PAPERLESS_REMOTE_OCR_MODE
|
||||||
|
remote_ocr_mode = models.CharField(
|
||||||
|
verbose_name=_("Sets which documents are sent to the remote OCR engine"),
|
||||||
|
blank=True,
|
||||||
|
null=True,
|
||||||
|
max_length=32,
|
||||||
|
choices=RemoteOCRMode.choices,
|
||||||
|
)
|
||||||
|
|
||||||
"""
|
"""
|
||||||
AI related settings
|
AI related settings
|
||||||
"""
|
"""
|
||||||
|
|||||||
@@ -134,6 +134,11 @@ class ParserProtocol(Protocol):
|
|||||||
Author or organisation name.
|
Author or organisation name.
|
||||||
url : str
|
url : str
|
||||||
URL for documentation, source code, or issue tracker.
|
URL for documentation, source code, or issue tracker.
|
||||||
|
|
||||||
|
Parsers that send document content to a remote service should additionally
|
||||||
|
set ``uses_remote_service = True`` so the registry can exclude them when
|
||||||
|
remote processing has not been requested for a document. The attribute is
|
||||||
|
optional so a parser that omits it is treated as fully local.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
@@ -145,6 +150,10 @@ class ParserProtocol(Protocol):
|
|||||||
author: str
|
author: str
|
||||||
url: str
|
url: str
|
||||||
|
|
||||||
|
# NOTE: uses_remote_service is not declared here, the registry reads it
|
||||||
|
# with getattr(cls, ..., False) for backwards-compatibility with existing
|
||||||
|
# parsers
|
||||||
|
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
# Class methods
|
# Class methods
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
|
|||||||
@@ -334,6 +334,8 @@ class ParserRegistry:
|
|||||||
mime_type: str,
|
mime_type: str,
|
||||||
filename: str,
|
filename: str,
|
||||||
path: Path | None = None,
|
path: Path | None = None,
|
||||||
|
*,
|
||||||
|
allow_remote: bool = True,
|
||||||
) -> type[ParserProtocol] | None:
|
) -> type[ParserProtocol] | None:
|
||||||
"""Return the best parser class for the given file, or None.
|
"""Return the best parser class for the given file, or None.
|
||||||
|
|
||||||
@@ -359,6 +361,11 @@ class ParserRegistry:
|
|||||||
path:
|
path:
|
||||||
Optional filesystem path to the file. Forwarded to each
|
Optional filesystem path to the file. Forwarded to each
|
||||||
parser's score method.
|
parser's score method.
|
||||||
|
allow_remote:
|
||||||
|
When False, parsers that declare ``uses_remote_service = True``
|
||||||
|
are excluded from consideration, so a document is never sent to
|
||||||
|
a remote service. Parsers that do not declare the attribute
|
||||||
|
are treated as local and are always considered.
|
||||||
|
|
||||||
Returns
|
Returns
|
||||||
-------
|
-------
|
||||||
@@ -374,6 +381,13 @@ class ParserRegistry:
|
|||||||
if mime_type not in parser_class.supported_mime_types():
|
if mime_type not in parser_class.supported_mime_types():
|
||||||
continue
|
continue
|
||||||
|
|
||||||
|
if not allow_remote and getattr(
|
||||||
|
parser_class,
|
||||||
|
"uses_remote_service",
|
||||||
|
False,
|
||||||
|
):
|
||||||
|
continue
|
||||||
|
|
||||||
score = parser_class.score(mime_type, filename, path)
|
score = parser_class.score(mime_type, filename, path)
|
||||||
if score is None:
|
if score is None:
|
||||||
continue
|
continue
|
||||||
|
|||||||
@@ -57,6 +57,18 @@ class RemoteEngineConfig:
|
|||||||
self.api_key = api_key
|
self.api_key = api_key
|
||||||
self.endpoint = endpoint
|
self.endpoint = endpoint
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def from_app_config(cls) -> Self:
|
||||||
|
"""Build the config from the app config, falling back to the env."""
|
||||||
|
from paperless.config import RemoteOCRConfig
|
||||||
|
|
||||||
|
app_config = RemoteOCRConfig()
|
||||||
|
return cls(
|
||||||
|
engine=app_config.remote_ocr_engine,
|
||||||
|
api_key=app_config.remote_ocr_api_key,
|
||||||
|
endpoint=app_config.remote_ocr_endpoint,
|
||||||
|
)
|
||||||
|
|
||||||
def engine_is_valid(self) -> bool:
|
def engine_is_valid(self) -> bool:
|
||||||
"""Return True when the engine is known and fully configured."""
|
"""Return True when the engine is known and fully configured."""
|
||||||
return (
|
return (
|
||||||
@@ -83,6 +95,9 @@ class RemoteDocumentParser:
|
|||||||
Maintainer name.
|
Maintainer name.
|
||||||
url : str
|
url : str
|
||||||
Issue tracker / source URL.
|
Issue tracker / source URL.
|
||||||
|
uses_remote_service : bool
|
||||||
|
Content is sent to a remote service, True so that the registry
|
||||||
|
can skip this parser if remote processing was not requested.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
name: str = "Paperless-ngx Remote OCR Parser"
|
name: str = "Paperless-ngx Remote OCR Parser"
|
||||||
@@ -90,6 +105,8 @@ class RemoteDocumentParser:
|
|||||||
author: str = "Paperless-ngx Contributors"
|
author: str = "Paperless-ngx Contributors"
|
||||||
url: str = "https://github.com/paperless-ngx/paperless-ngx"
|
url: str = "https://github.com/paperless-ngx/paperless-ngx"
|
||||||
|
|
||||||
|
uses_remote_service: bool = True
|
||||||
|
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
# Class methods
|
# Class methods
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
@@ -138,11 +155,7 @@ class RemoteDocumentParser:
|
|||||||
20 when the remote engine is configured and the MIME type is
|
20 when the remote engine is configured and the MIME type is
|
||||||
supported, otherwise None.
|
supported, otherwise None.
|
||||||
"""
|
"""
|
||||||
config = RemoteEngineConfig(
|
config = RemoteEngineConfig.from_app_config()
|
||||||
engine=settings.REMOTE_OCR_ENGINE,
|
|
||||||
api_key=settings.REMOTE_OCR_API_KEY,
|
|
||||||
endpoint=settings.REMOTE_OCR_ENDPOINT,
|
|
||||||
)
|
|
||||||
if not config.engine_is_valid():
|
if not config.engine_is_valid():
|
||||||
return None
|
return None
|
||||||
if mime_type not in _SUPPORTED_MIME_TYPES:
|
if mime_type not in _SUPPORTED_MIME_TYPES:
|
||||||
@@ -228,11 +241,7 @@ class RemoteDocumentParser:
|
|||||||
Ignored — the remote engine always returns a searchable PDF,
|
Ignored — the remote engine always returns a searchable PDF,
|
||||||
which is stored as the archive copy regardless of this flag.
|
which is stored as the archive copy regardless of this flag.
|
||||||
"""
|
"""
|
||||||
config = RemoteEngineConfig(
|
config = RemoteEngineConfig.from_app_config()
|
||||||
engine=settings.REMOTE_OCR_ENGINE,
|
|
||||||
api_key=settings.REMOTE_OCR_API_KEY,
|
|
||||||
endpoint=settings.REMOTE_OCR_ENDPOINT,
|
|
||||||
)
|
|
||||||
|
|
||||||
if not config.engine_is_valid():
|
if not config.engine_is_valid():
|
||||||
logger.warning(
|
logger.warning(
|
||||||
|
|||||||
@@ -217,7 +217,15 @@ class ApplicationConfigurationSerializer(
|
|||||||
llm_api_key = ObfuscatedPasswordField(
|
llm_api_key = ObfuscatedPasswordField(
|
||||||
required=False,
|
required=False,
|
||||||
allow_null=True,
|
allow_null=True,
|
||||||
|
max_length=1024,
|
||||||
)
|
)
|
||||||
|
remote_ocr_api_key = ObfuscatedPasswordField(
|
||||||
|
required=False,
|
||||||
|
allow_null=True,
|
||||||
|
max_length=1024,
|
||||||
|
)
|
||||||
|
|
||||||
|
OBFUSCATED_FIELDS = ("llm_api_key", "remote_ocr_api_key")
|
||||||
|
|
||||||
def run_validation(self, data):
|
def run_validation(self, data):
|
||||||
# Empty strings treated as None to avoid unexpected behavior
|
# Empty strings treated as None to avoid unexpected behavior
|
||||||
@@ -229,11 +237,13 @@ class ApplicationConfigurationSerializer(
|
|||||||
data["language"] = None
|
data["language"] = None
|
||||||
if "llm_output_language" in data and data["llm_output_language"] == "":
|
if "llm_output_language" in data and data["llm_output_language"] == "":
|
||||||
data["llm_output_language"] = None
|
data["llm_output_language"] = None
|
||||||
if "llm_api_key" in data and data["llm_api_key"] is not None:
|
for field in self.OBFUSCATED_FIELDS:
|
||||||
if data["llm_api_key"] == "":
|
if field in data and data[field] is not None:
|
||||||
data["llm_api_key"] = None
|
if data[field] == "":
|
||||||
elif len(data["llm_api_key"].replace("*", "")) == 0:
|
data[field] = None
|
||||||
del data["llm_api_key"]
|
# Not a real value, don't overwrite the stored one
|
||||||
|
elif len(data[field].replace("*", "")) == 0:
|
||||||
|
del data[field]
|
||||||
return super().run_validation(data)
|
return super().run_validation(data)
|
||||||
|
|
||||||
def update(self, instance, validated_data):
|
def update(self, instance, validated_data):
|
||||||
|
|||||||
@@ -1197,6 +1197,7 @@ WEBHOOKS_ALLOW_INTERNAL_REQUESTS = get_bool_from_env(
|
|||||||
REMOTE_OCR_ENGINE = os.getenv("PAPERLESS_REMOTE_OCR_ENGINE")
|
REMOTE_OCR_ENGINE = os.getenv("PAPERLESS_REMOTE_OCR_ENGINE")
|
||||||
REMOTE_OCR_API_KEY = os.getenv("PAPERLESS_REMOTE_OCR_API_KEY")
|
REMOTE_OCR_API_KEY = os.getenv("PAPERLESS_REMOTE_OCR_API_KEY")
|
||||||
REMOTE_OCR_ENDPOINT = os.getenv("PAPERLESS_REMOTE_OCR_ENDPOINT")
|
REMOTE_OCR_ENDPOINT = os.getenv("PAPERLESS_REMOTE_OCR_ENDPOINT")
|
||||||
|
REMOTE_OCR_MODE = os.getenv("PAPERLESS_REMOTE_OCR_MODE", "always")
|
||||||
|
|
||||||
################################################################################
|
################################################################################
|
||||||
# AI Settings #
|
# AI Settings #
|
||||||
|
|||||||
@@ -21,6 +21,7 @@ from unittest.mock import Mock
|
|||||||
import pytest
|
import pytest
|
||||||
|
|
||||||
from documents.parsers import ParseError
|
from documents.parsers import ParseError
|
||||||
|
from paperless.models import ApplicationConfiguration
|
||||||
from paperless.parsers import ParserContext
|
from paperless.parsers import ParserContext
|
||||||
from paperless.parsers import ParserProtocol
|
from paperless.parsers import ParserProtocol
|
||||||
from paperless.parsers.remote import RemoteDocumentParser
|
from paperless.parsers.remote import RemoteDocumentParser
|
||||||
@@ -33,6 +34,10 @@ if TYPE_CHECKING:
|
|||||||
from pytest_mock import MockerFixture
|
from pytest_mock import MockerFixture
|
||||||
|
|
||||||
|
|
||||||
|
# Remote ocr config from ApplicationConfiguration needs DB access
|
||||||
|
pytestmark = pytest.mark.django_db
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Module-local fixtures
|
# Module-local fixtures
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
@@ -227,6 +232,18 @@ class TestRemoteParserScore:
|
|||||||
score = RemoteDocumentParser.score("application/pdf", "doc.pdf")
|
score = RemoteDocumentParser.score("application/pdf", "doc.pdf")
|
||||||
assert score is not None and score > 10
|
assert score is not None and score > 10
|
||||||
|
|
||||||
|
@pytest.mark.usefixtures("no_engine_settings")
|
||||||
|
def test_score_uses_app_config_when_env_unset(self) -> None:
|
||||||
|
"""The app config alone is enough to activate the parser."""
|
||||||
|
config = ApplicationConfiguration.objects.first()
|
||||||
|
assert config is not None
|
||||||
|
config.remote_ocr_engine = "azureai"
|
||||||
|
config.remote_ocr_api_key = "app-config-key"
|
||||||
|
config.remote_ocr_endpoint = "https://config.cognitiveservices.azure.com"
|
||||||
|
config.save()
|
||||||
|
|
||||||
|
assert RemoteDocumentParser.score("application/pdf", "doc.pdf") == 20
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
# Properties
|
# Properties
|
||||||
|
|||||||
@@ -1277,6 +1277,8 @@ class TestParserFileTypes:
|
|||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
# Remote ocr config from ApplicationConfiguration needs DB access
|
||||||
|
@pytest.mark.django_db
|
||||||
class TestRasterisedDocumentParserRegistry:
|
class TestRasterisedDocumentParserRegistry:
|
||||||
def test_registered_in_defaults(self) -> None:
|
def test_registered_in_defaults(self) -> None:
|
||||||
from paperless.parsers.registry import ParserRegistry
|
from paperless.parsers.registry import ParserRegistry
|
||||||
|
|||||||
@@ -0,0 +1,53 @@
|
|||||||
|
from django.contrib.auth.models import AnonymousUser
|
||||||
|
from django.contrib.auth.models import User
|
||||||
|
from django.test import RequestFactory
|
||||||
|
from django.test import TestCase
|
||||||
|
from django.test import override_settings
|
||||||
|
|
||||||
|
from paperless.auth import AutoLoginMiddleware
|
||||||
|
|
||||||
|
|
||||||
|
@override_settings(AUTO_LOGIN_USERNAME="autologin")
|
||||||
|
class TestAutoLoginMiddleware(TestCase):
|
||||||
|
def setUp(self) -> None:
|
||||||
|
super().setUp()
|
||||||
|
self.factory = RequestFactory()
|
||||||
|
self.middleware = AutoLoginMiddleware(lambda request: None)
|
||||||
|
|
||||||
|
def _process(self, request):
|
||||||
|
# login() needs a session to write to
|
||||||
|
request.session = self.client.session
|
||||||
|
self.middleware.process_request(request)
|
||||||
|
return request
|
||||||
|
|
||||||
|
def test_active_user_is_logged_in(self) -> None:
|
||||||
|
"""
|
||||||
|
GIVEN:
|
||||||
|
- AUTO_LOGIN_USERNAME names an active user
|
||||||
|
WHEN:
|
||||||
|
- A request is processed by the middleware
|
||||||
|
THEN:
|
||||||
|
- That user is attached to the request
|
||||||
|
"""
|
||||||
|
user = User.objects.create_user(username="autologin")
|
||||||
|
|
||||||
|
request = self._process(self.factory.get("/"))
|
||||||
|
|
||||||
|
self.assertEqual(request.user, user)
|
||||||
|
|
||||||
|
def test_deactivated_user_is_not_logged_in(self) -> None:
|
||||||
|
"""
|
||||||
|
GIVEN:
|
||||||
|
- AUTO_LOGIN_USERNAME names a user who has been deactivated
|
||||||
|
WHEN:
|
||||||
|
- A request is processed by the middleware
|
||||||
|
THEN:
|
||||||
|
- The request is left anonymous rather than authenticated as them
|
||||||
|
"""
|
||||||
|
User.objects.create_user(username="autologin", is_active=False)
|
||||||
|
|
||||||
|
request = self.factory.get("/")
|
||||||
|
request.user = AnonymousUser()
|
||||||
|
self._process(request)
|
||||||
|
|
||||||
|
self.assertFalse(request.user.is_authenticated)
|
||||||
@@ -655,6 +655,23 @@ class TestRemoteParserChecks:
|
|||||||
in msg.msg
|
in msg.msg
|
||||||
)
|
)
|
||||||
|
|
||||||
|
def test_valid_mode(self, settings: SettingsWrapper) -> None:
|
||||||
|
settings.REMOTE_OCR_ENGINE = None
|
||||||
|
settings.REMOTE_OCR_MODE = "workflow_only"
|
||||||
|
|
||||||
|
msgs = check_remote_parser_configured(None)
|
||||||
|
|
||||||
|
assert len(msgs) == 0
|
||||||
|
|
||||||
|
def test_invalid_mode(self, settings: SettingsWrapper) -> None:
|
||||||
|
settings.REMOTE_OCR_ENGINE = None
|
||||||
|
settings.REMOTE_OCR_MODE = "sometimes"
|
||||||
|
|
||||||
|
msgs = check_remote_parser_configured(None)
|
||||||
|
|
||||||
|
assert len(msgs) == 1
|
||||||
|
assert "PAPERLESS_REMOTE_OCR_MODE is set to 'sometimes'" in msgs[0].msg
|
||||||
|
|
||||||
|
|
||||||
class TestTesseractChecks:
|
class TestTesseractChecks:
|
||||||
def test_default_language(self) -> None:
|
def test_default_language(self) -> None:
|
||||||
|
|||||||
@@ -468,6 +468,124 @@ class TestParserRegistryGetParserForFile:
|
|||||||
assert result is AcceptingBuiltin
|
assert result is AcceptingBuiltin
|
||||||
|
|
||||||
|
|
||||||
|
class TestParserRegistryRemoteParsers:
|
||||||
|
"""Verify the allow_remote filter in ParserRegistry.get_parser_for_file()."""
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _remote_parser_cls() -> type:
|
||||||
|
class RemoteParser:
|
||||||
|
name = "remote"
|
||||||
|
version = "1.0"
|
||||||
|
author = "A"
|
||||||
|
url = "https://example.com/remote"
|
||||||
|
uses_remote_service = True
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def supported_mime_types(cls):
|
||||||
|
return {"text/plain": ".txt"}
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def score(cls, mime_type, filename, path=None):
|
||||||
|
return 20
|
||||||
|
|
||||||
|
return RemoteParser
|
||||||
|
|
||||||
|
def test_remote_parser_wins_when_remote_allowed(
|
||||||
|
self,
|
||||||
|
dummy_parser_cls: type,
|
||||||
|
) -> None:
|
||||||
|
"""
|
||||||
|
GIVEN: A remote parser scoring 20 and a local parser scoring 10.
|
||||||
|
WHEN: get_parser_for_file() is called with allow_remote=True.
|
||||||
|
THEN: The remote parser is returned.
|
||||||
|
"""
|
||||||
|
remote_parser_cls = self._remote_parser_cls()
|
||||||
|
registry = ParserRegistry()
|
||||||
|
registry.register_builtin(dummy_parser_cls)
|
||||||
|
registry.register_builtin(remote_parser_cls)
|
||||||
|
|
||||||
|
result = registry.get_parser_for_file(
|
||||||
|
"text/plain",
|
||||||
|
"readme.txt",
|
||||||
|
allow_remote=True,
|
||||||
|
)
|
||||||
|
assert result is remote_parser_cls
|
||||||
|
|
||||||
|
def test_remote_parser_skipped_when_remote_not_allowed(
|
||||||
|
self,
|
||||||
|
dummy_parser_cls: type,
|
||||||
|
) -> None:
|
||||||
|
"""
|
||||||
|
GIVEN: A remote parser scoring 20 and a local parser scoring 10.
|
||||||
|
WHEN: get_parser_for_file() is called with allow_remote=False.
|
||||||
|
THEN: The local parser is returned despite its lower score.
|
||||||
|
"""
|
||||||
|
registry = ParserRegistry()
|
||||||
|
registry.register_builtin(dummy_parser_cls)
|
||||||
|
registry.register_builtin(self._remote_parser_cls())
|
||||||
|
|
||||||
|
result = registry.get_parser_for_file(
|
||||||
|
"text/plain",
|
||||||
|
"readme.txt",
|
||||||
|
allow_remote=False,
|
||||||
|
)
|
||||||
|
assert result is dummy_parser_cls
|
||||||
|
|
||||||
|
def test_no_parser_when_only_remote_available_and_not_allowed(self) -> None:
|
||||||
|
"""
|
||||||
|
GIVEN: A registry whose only candidate declares uses_remote_service.
|
||||||
|
WHEN: get_parser_for_file() is called with allow_remote=False.
|
||||||
|
THEN: None is returned — the remote parser is never used as a
|
||||||
|
fallback when remote processing was not requested.
|
||||||
|
"""
|
||||||
|
registry = ParserRegistry()
|
||||||
|
registry.register_builtin(self._remote_parser_cls())
|
||||||
|
|
||||||
|
result = registry.get_parser_for_file(
|
||||||
|
"text/plain",
|
||||||
|
"readme.txt",
|
||||||
|
allow_remote=False,
|
||||||
|
)
|
||||||
|
assert result is None
|
||||||
|
|
||||||
|
def test_parser_without_attribute_treated_as_local(
|
||||||
|
self,
|
||||||
|
dummy_parser_cls: type,
|
||||||
|
) -> None:
|
||||||
|
"""
|
||||||
|
GIVEN: A third-party parser predating uses_remote_service, so it does
|
||||||
|
not declare the attribute at all.
|
||||||
|
WHEN: get_parser_for_file() is called with allow_remote=False.
|
||||||
|
THEN: It is still considered, i.e. treated as fully local, rather
|
||||||
|
than raising AttributeError.
|
||||||
|
"""
|
||||||
|
assert not hasattr(dummy_parser_cls, "uses_remote_service")
|
||||||
|
|
||||||
|
registry = ParserRegistry()
|
||||||
|
registry.register_builtin(dummy_parser_cls)
|
||||||
|
|
||||||
|
result = registry.get_parser_for_file(
|
||||||
|
"text/plain",
|
||||||
|
"readme.txt",
|
||||||
|
allow_remote=False,
|
||||||
|
)
|
||||||
|
assert result is dummy_parser_cls
|
||||||
|
|
||||||
|
def test_remote_allowed_by_default(self) -> None:
|
||||||
|
"""
|
||||||
|
GIVEN: A registry containing only a remote parser.
|
||||||
|
WHEN: get_parser_for_file() is called without allow_remote.
|
||||||
|
THEN: The remote parser is returned — callers that do not opt in to
|
||||||
|
the filter keep the previous behaviour.
|
||||||
|
"""
|
||||||
|
remote_parser_cls = self._remote_parser_cls()
|
||||||
|
registry = ParserRegistry()
|
||||||
|
registry.register_builtin(remote_parser_cls)
|
||||||
|
|
||||||
|
result = registry.get_parser_for_file("text/plain", "readme.txt")
|
||||||
|
assert result is remote_parser_cls
|
||||||
|
|
||||||
|
|
||||||
class TestDiscover:
|
class TestDiscover:
|
||||||
"""Verify entrypoint discovery in ParserRegistry.discover()."""
|
"""Verify entrypoint discovery in ParserRegistry.discover()."""
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,113 @@
|
|||||||
|
"""Tests for RemoteOCRConfig precedence between app config and Django settings."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import TYPE_CHECKING
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from django.test import override_settings
|
||||||
|
|
||||||
|
from paperless.config import RemoteOCRConfig
|
||||||
|
from paperless.models import RemoteOCRMode
|
||||||
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
from unittest.mock import MagicMock
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture()
|
||||||
|
def null_app_config(mocker) -> MagicMock:
|
||||||
|
"""Mock ApplicationConfiguration with all fields None → falls back to Django settings."""
|
||||||
|
return mocker.MagicMock(
|
||||||
|
remote_ocr_engine=None,
|
||||||
|
remote_ocr_api_key=None,
|
||||||
|
remote_ocr_endpoint=None,
|
||||||
|
remote_ocr_mode=None,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture()
|
||||||
|
def make_remote_ocr_config(mocker):
|
||||||
|
def _make(app_config, **django_settings_overrides):
|
||||||
|
mocker.patch(
|
||||||
|
"paperless.config.BaseConfig._get_config_instance",
|
||||||
|
return_value=app_config,
|
||||||
|
)
|
||||||
|
with override_settings(**django_settings_overrides):
|
||||||
|
return RemoteOCRConfig()
|
||||||
|
|
||||||
|
return _make
|
||||||
|
|
||||||
|
|
||||||
|
class TestRemoteOCRConfig:
|
||||||
|
def test_falls_back_to_settings(
|
||||||
|
self,
|
||||||
|
make_remote_ocr_config,
|
||||||
|
null_app_config,
|
||||||
|
) -> None:
|
||||||
|
cfg = make_remote_ocr_config(
|
||||||
|
null_app_config,
|
||||||
|
REMOTE_OCR_ENGINE="azureai",
|
||||||
|
REMOTE_OCR_API_KEY="env-key",
|
||||||
|
REMOTE_OCR_ENDPOINT="https://env.cognitiveservices.azure.com",
|
||||||
|
REMOTE_OCR_MODE=RemoteOCRMode.WORKFLOW_ONLY,
|
||||||
|
)
|
||||||
|
assert cfg.remote_ocr_engine == "azureai"
|
||||||
|
assert cfg.remote_ocr_api_key == "env-key"
|
||||||
|
assert cfg.remote_ocr_endpoint == "https://env.cognitiveservices.azure.com"
|
||||||
|
assert cfg.remote_ocr_mode == RemoteOCRMode.WORKFLOW_ONLY
|
||||||
|
|
||||||
|
def test_app_config_takes_precedence(
|
||||||
|
self,
|
||||||
|
make_remote_ocr_config,
|
||||||
|
mocker,
|
||||||
|
) -> None:
|
||||||
|
app_config = mocker.MagicMock(
|
||||||
|
remote_ocr_engine="azureai",
|
||||||
|
remote_ocr_api_key="db-key",
|
||||||
|
remote_ocr_endpoint="https://db.cognitiveservices.azure.com",
|
||||||
|
remote_ocr_mode=RemoteOCRMode.WORKFLOW_ONLY,
|
||||||
|
)
|
||||||
|
cfg = make_remote_ocr_config(
|
||||||
|
app_config,
|
||||||
|
REMOTE_OCR_ENGINE=None,
|
||||||
|
REMOTE_OCR_API_KEY="env-key",
|
||||||
|
REMOTE_OCR_ENDPOINT="https://env.cognitiveservices.azure.com",
|
||||||
|
REMOTE_OCR_MODE=RemoteOCRMode.ALWAYS,
|
||||||
|
)
|
||||||
|
assert cfg.remote_ocr_engine == "azureai"
|
||||||
|
assert cfg.remote_ocr_api_key == "db-key"
|
||||||
|
assert cfg.remote_ocr_endpoint == "https://db.cognitiveservices.azure.com"
|
||||||
|
assert cfg.remote_ocr_mode == RemoteOCRMode.WORKFLOW_ONLY
|
||||||
|
|
||||||
|
def test_unset_everywhere(
|
||||||
|
self,
|
||||||
|
make_remote_ocr_config,
|
||||||
|
null_app_config,
|
||||||
|
) -> None:
|
||||||
|
cfg = make_remote_ocr_config(
|
||||||
|
null_app_config,
|
||||||
|
REMOTE_OCR_ENGINE=None,
|
||||||
|
REMOTE_OCR_API_KEY=None,
|
||||||
|
REMOTE_OCR_ENDPOINT=None,
|
||||||
|
)
|
||||||
|
assert cfg.remote_ocr_engine is None
|
||||||
|
assert cfg.remote_ocr_api_key is None
|
||||||
|
assert cfg.remote_ocr_endpoint is None
|
||||||
|
|
||||||
|
|
||||||
|
class TestRemoteOCRByDefault:
|
||||||
|
def test_always_mode(self, make_remote_ocr_config, null_app_config) -> None:
|
||||||
|
cfg = make_remote_ocr_config(
|
||||||
|
null_app_config,
|
||||||
|
REMOTE_OCR_MODE=RemoteOCRMode.ALWAYS,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert cfg.remote_ocr_by_default is True
|
||||||
|
|
||||||
|
def test_workflow_only_mode(self, make_remote_ocr_config, null_app_config) -> None:
|
||||||
|
cfg = make_remote_ocr_config(
|
||||||
|
null_app_config,
|
||||||
|
REMOTE_OCR_MODE=RemoteOCRMode.WORKFLOW_ONLY,
|
||||||
|
)
|
||||||
|
|
||||||
|
assert cfg.remote_ocr_by_default is False
|
||||||
@@ -2,8 +2,6 @@ import json
|
|||||||
import logging
|
import logging
|
||||||
import sys
|
import sys
|
||||||
|
|
||||||
from django.db.models import QuerySet
|
|
||||||
|
|
||||||
from documents.models import Document
|
from documents.models import Document
|
||||||
from paperless.config import AIConfig
|
from paperless.config import AIConfig
|
||||||
from paperless_ai.client import AIClient
|
from paperless_ai.client import AIClient
|
||||||
@@ -84,21 +82,10 @@ def _build_document_reference(
|
|||||||
|
|
||||||
|
|
||||||
def _get_document_references(
|
def _get_document_references(
|
||||||
documents: QuerySet[Document],
|
documents: list[Document],
|
||||||
top_nodes: list,
|
top_nodes: list,
|
||||||
) -> list[dict[str, int | str]]:
|
) -> list[dict[str, int | str]]:
|
||||||
candidate_ids: set[int] = set()
|
allowed_documents = {doc.pk: doc for doc in documents}
|
||||||
for node in top_nodes:
|
|
||||||
try:
|
|
||||||
candidate_ids.add(int(node.metadata["document_id"]))
|
|
||||||
except (KeyError, TypeError, ValueError): # pragma: no cover
|
|
||||||
continue
|
|
||||||
|
|
||||||
if not candidate_ids:
|
|
||||||
return []
|
|
||||||
|
|
||||||
allowed_documents = {doc.pk: doc for doc in documents.filter(pk__in=candidate_ids)}
|
|
||||||
|
|
||||||
references: list[dict[str, int | str]] = []
|
references: list[dict[str, int | str]] = []
|
||||||
seen_document_ids: set[int] = set()
|
seen_document_ids: set[int] = set()
|
||||||
|
|
||||||
@@ -132,7 +119,7 @@ def _format_chat_metadata_trailer(references: list[dict[str, int | str]]) -> str
|
|||||||
|
|
||||||
def stream_chat_with_documents(
|
def stream_chat_with_documents(
|
||||||
query_str: str,
|
query_str: str,
|
||||||
documents: QuerySet[Document],
|
documents: list[Document],
|
||||||
output_language: str | None = None,
|
output_language: str | None = None,
|
||||||
):
|
):
|
||||||
try:
|
try:
|
||||||
@@ -148,10 +135,10 @@ def stream_chat_with_documents(
|
|||||||
|
|
||||||
def _stream_chat_with_documents(
|
def _stream_chat_with_documents(
|
||||||
query_str: str,
|
query_str: str,
|
||||||
documents: QuerySet[Document],
|
documents: list[Document],
|
||||||
output_language: str | None = None,
|
output_language: str | None = None,
|
||||||
):
|
):
|
||||||
if not documents.exists():
|
if not documents:
|
||||||
yield CHAT_NO_CONTENT_MESSAGE
|
yield CHAT_NO_CONTENT_MESSAGE
|
||||||
return
|
return
|
||||||
|
|
||||||
@@ -161,9 +148,7 @@ def _stream_chat_with_documents(
|
|||||||
from llama_index.core.retrievers import VectorIndexRetriever
|
from llama_index.core.retrievers import VectorIndexRetriever
|
||||||
|
|
||||||
config = AIConfig()
|
config = AIConfig()
|
||||||
filters = _document_id_filters(
|
filters = _document_id_filters(str(doc.pk) for doc in documents)
|
||||||
str(pk) for pk in documents.values_list("pk", flat=True)
|
|
||||||
)
|
|
||||||
|
|
||||||
# Hold the shared read lock for the whole operation: the query engine
|
# Hold the shared read lock for the whole operation: the query engine
|
||||||
# retrieves from the vector store again during synthesis, so the connection
|
# retrieves from the vector store again during synthesis, so the connection
|
||||||
|
|||||||
@@ -3,12 +3,10 @@ from unittest.mock import MagicMock
|
|||||||
from unittest.mock import patch
|
from unittest.mock import patch
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
from django.db.models.signals import post_init
|
|
||||||
from llama_index.core import settings as llama_settings
|
from llama_index.core import settings as llama_settings
|
||||||
from llama_index.core.embeddings.mock_embed_model import MockEmbedding
|
from llama_index.core.embeddings.mock_embed_model import MockEmbedding
|
||||||
from llama_index.core.schema import TextNode
|
from llama_index.core.schema import TextNode
|
||||||
|
|
||||||
from documents.models import Document
|
|
||||||
from documents.tests.factories import DocumentFactory
|
from documents.tests.factories import DocumentFactory
|
||||||
from paperless_ai import chat
|
from paperless_ai import chat
|
||||||
from paperless_ai import indexing
|
from paperless_ai import indexing
|
||||||
@@ -38,6 +36,16 @@ def patch_embed_nodes():
|
|||||||
yield mock_embed_nodes
|
yield mock_embed_nodes
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def mock_document():
|
||||||
|
doc = MagicMock()
|
||||||
|
doc.pk = 1
|
||||||
|
doc.title = "Test Document"
|
||||||
|
doc.filename = "test_file.pdf"
|
||||||
|
doc.content = "This is the document content."
|
||||||
|
return doc
|
||||||
|
|
||||||
|
|
||||||
def assert_chat_output(
|
def assert_chat_output(
|
||||||
output: list[str],
|
output: list[str],
|
||||||
*,
|
*,
|
||||||
@@ -53,13 +61,6 @@ def assert_chat_output(
|
|||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
def _fake_documents_queryset(pks: list[int]) -> MagicMock:
|
|
||||||
qs = MagicMock()
|
|
||||||
qs.exists.return_value = bool(pks)
|
|
||||||
qs.values_list.return_value = pks
|
|
||||||
return qs
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.parametrize(
|
@pytest.mark.parametrize(
|
||||||
("output_language", "expected_language_line"),
|
("output_language", "expected_language_line"),
|
||||||
[
|
[
|
||||||
@@ -106,10 +107,9 @@ def test_build_refine_prompt(
|
|||||||
|
|
||||||
@pytest.mark.django_db
|
@pytest.mark.django_db
|
||||||
def test_stream_chat_with_one_document_retrieval(
|
def test_stream_chat_with_one_document_retrieval(
|
||||||
|
mock_document,
|
||||||
patch_embed_nodes,
|
patch_embed_nodes,
|
||||||
) -> None:
|
) -> None:
|
||||||
document = DocumentFactory.create(title="Test Document", content="ignored")
|
|
||||||
documents = Document.objects.filter(pk=document.pk)
|
|
||||||
with (
|
with (
|
||||||
patch("paperless_ai.chat.AIClient") as mock_client_cls,
|
patch("paperless_ai.chat.AIClient") as mock_client_cls,
|
||||||
patch("paperless_ai.chat.load_or_build_index") as mock_load_index,
|
patch("paperless_ai.chat.load_or_build_index") as mock_load_index,
|
||||||
@@ -124,19 +124,22 @@ def test_stream_chat_with_one_document_retrieval(
|
|||||||
mock_client_cls.return_value = mock_client
|
mock_client_cls.return_value = mock_client
|
||||||
mock_client.llm = MagicMock()
|
mock_client.llm = MagicMock()
|
||||||
|
|
||||||
mock_index = MagicMock()
|
mock_node = TextNode(
|
||||||
mock_index.vector_store.get_nodes.return_value = [
|
|
||||||
TextNode(
|
|
||||||
text="This is node content.",
|
text="This is node content.",
|
||||||
metadata={"document_id": str(document.pk), "title": "Test Document"},
|
metadata={"document_id": str(mock_document.pk), "title": "Test Document"},
|
||||||
),
|
)
|
||||||
]
|
mock_index = MagicMock()
|
||||||
|
# Simulate get_nodes returning nodes (content exists)
|
||||||
|
mock_index.vector_store.get_nodes.return_value = [mock_node]
|
||||||
mock_load_index.return_value = mock_index
|
mock_load_index.return_value = mock_index
|
||||||
|
|
||||||
mock_retriever_instance = MagicMock()
|
mock_retriever_instance = MagicMock()
|
||||||
mock_retriever_instance.retrieve.return_value = [
|
mock_retriever_instance.retrieve.return_value = [
|
||||||
MagicMock(
|
MagicMock(
|
||||||
metadata={"document_id": str(document.pk), "title": "Test Document"},
|
metadata={
|
||||||
|
"document_id": str(mock_document.pk),
|
||||||
|
"title": "Test Document",
|
||||||
|
},
|
||||||
),
|
),
|
||||||
]
|
]
|
||||||
|
|
||||||
@@ -150,7 +153,7 @@ def test_stream_chat_with_one_document_retrieval(
|
|||||||
"llama_index.core.retrievers.VectorIndexRetriever",
|
"llama_index.core.retrievers.VectorIndexRetriever",
|
||||||
return_value=mock_retriever_instance,
|
return_value=mock_retriever_instance,
|
||||||
):
|
):
|
||||||
output = list(stream_chat_with_documents("What is this?", documents))
|
output = list(stream_chat_with_documents("What is this?", [mock_document]))
|
||||||
|
|
||||||
mock_query_engine.query.assert_called_once_with("What is this?")
|
mock_query_engine.query.assert_called_once_with("What is this?")
|
||||||
synthesizer_kwargs = mock_get_response_synthesizer.call_args.kwargs
|
synthesizer_kwargs = mock_get_response_synthesizer.call_args.kwargs
|
||||||
@@ -163,16 +166,13 @@ def test_stream_chat_with_one_document_retrieval(
|
|||||||
output,
|
output,
|
||||||
expected_chunks=["chunk1", "chunk2"],
|
expected_chunks=["chunk1", "chunk2"],
|
||||||
expected_references=[
|
expected_references=[
|
||||||
{"id": document.pk, "title": "Test Document"},
|
{"id": mock_document.pk, "title": "Test Document"},
|
||||||
],
|
],
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.django_db
|
@pytest.mark.django_db
|
||||||
def test_stream_chat_with_multiple_documents_retrieval(patch_embed_nodes) -> None:
|
def test_stream_chat_with_multiple_documents_retrieval(patch_embed_nodes) -> None:
|
||||||
doc1 = DocumentFactory.create(title="Document 1", content="ignored")
|
|
||||||
doc2 = DocumentFactory.create(title="Document 2", content="ignored")
|
|
||||||
documents = Document.objects.filter(pk__in=[doc1.pk, doc2.pk])
|
|
||||||
with (
|
with (
|
||||||
patch("paperless_ai.chat.AIClient") as mock_client_cls,
|
patch("paperless_ai.chat.AIClient") as mock_client_cls,
|
||||||
patch("paperless_ai.chat.load_or_build_index") as mock_load_index,
|
patch("paperless_ai.chat.load_or_build_index") as mock_load_index,
|
||||||
@@ -184,23 +184,23 @@ def test_stream_chat_with_multiple_documents_retrieval(patch_embed_nodes) -> Non
|
|||||||
mock_client_cls.return_value = mock_client
|
mock_client_cls.return_value = mock_client
|
||||||
mock_client.llm = MagicMock()
|
mock_client.llm = MagicMock()
|
||||||
|
|
||||||
mock_index = MagicMock()
|
mock_node1 = TextNode(
|
||||||
mock_index.vector_store.get_nodes.return_value = [
|
|
||||||
TextNode(
|
|
||||||
text="Content for doc 1.",
|
text="Content for doc 1.",
|
||||||
metadata={"document_id": str(doc1.pk), "title": "Document 1"},
|
metadata={"document_id": "1", "title": "Document 1"},
|
||||||
),
|
)
|
||||||
TextNode(
|
mock_node2 = TextNode(
|
||||||
text="Content for doc 2.",
|
text="Content for doc 2.",
|
||||||
metadata={"document_id": str(doc2.pk), "title": "Document 2"},
|
metadata={"document_id": "2", "title": "Document 2"},
|
||||||
),
|
)
|
||||||
]
|
mock_index = MagicMock()
|
||||||
|
# Simulate get_nodes returning nodes (content exists)
|
||||||
|
mock_index.vector_store.get_nodes.return_value = [mock_node1, mock_node2]
|
||||||
mock_load_index.return_value = mock_index
|
mock_load_index.return_value = mock_index
|
||||||
|
|
||||||
mock_retriever_instance = MagicMock()
|
mock_retriever_instance = MagicMock()
|
||||||
mock_retriever_instance.retrieve.return_value = [
|
mock_retriever_instance.retrieve.return_value = [
|
||||||
MagicMock(metadata={"document_id": str(doc1.pk), "title": "Document 1"}),
|
MagicMock(metadata={"document_id": "1", "title": "Document 1"}),
|
||||||
MagicMock(metadata={"document_id": str(doc2.pk), "title": "Document 2"}),
|
MagicMock(metadata={"document_id": "2", "title": "Document 2"}),
|
||||||
]
|
]
|
||||||
|
|
||||||
mock_response_stream = MagicMock()
|
mock_response_stream = MagicMock()
|
||||||
@@ -210,11 +210,14 @@ def test_stream_chat_with_multiple_documents_retrieval(patch_embed_nodes) -> Non
|
|||||||
mock_query_engine_cls.return_value = mock_query_engine
|
mock_query_engine_cls.return_value = mock_query_engine
|
||||||
mock_query_engine.query.return_value = mock_response_stream
|
mock_query_engine.query.return_value = mock_response_stream
|
||||||
|
|
||||||
|
doc1 = MagicMock(pk=1, title="Document 1", filename="doc1.pdf")
|
||||||
|
doc2 = MagicMock(pk=2, title="Document 2", filename="doc2.pdf")
|
||||||
|
|
||||||
with patch(
|
with patch(
|
||||||
"llama_index.core.retrievers.VectorIndexRetriever",
|
"llama_index.core.retrievers.VectorIndexRetriever",
|
||||||
return_value=mock_retriever_instance,
|
return_value=mock_retriever_instance,
|
||||||
):
|
):
|
||||||
output = list(stream_chat_with_documents("What's up?", documents))
|
output = list(stream_chat_with_documents("What's up?", [doc1, doc2]))
|
||||||
|
|
||||||
mock_query_engine.query.assert_called_once_with("What's up?")
|
mock_query_engine.query.assert_called_once_with("What's up?")
|
||||||
patch_embed_nodes.assert_not_called()
|
patch_embed_nodes.assert_not_called()
|
||||||
@@ -222,15 +225,15 @@ def test_stream_chat_with_multiple_documents_retrieval(patch_embed_nodes) -> Non
|
|||||||
output,
|
output,
|
||||||
expected_chunks=["chunk1", "chunk2"],
|
expected_chunks=["chunk1", "chunk2"],
|
||||||
expected_references=[
|
expected_references=[
|
||||||
{"id": doc1.pk, "title": "Document 1"},
|
{"id": 1, "title": "Document 1"},
|
||||||
{"id": doc2.pk, "title": "Document 2"},
|
{"id": 2, "title": "Document 2"},
|
||||||
],
|
],
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
def test_stream_chat_empty_document_list() -> None:
|
def test_stream_chat_empty_document_list() -> None:
|
||||||
with patch("paperless_ai.chat.load_or_build_index") as mock_load_index:
|
with patch("paperless_ai.chat.load_or_build_index") as mock_load_index:
|
||||||
output = list(stream_chat_with_documents("Any info?", Document.objects.none()))
|
output = list(stream_chat_with_documents("Any info?", []))
|
||||||
mock_load_index.assert_not_called()
|
mock_load_index.assert_not_called()
|
||||||
assert output == ["Sorry, I couldn't find any content to answer your question."]
|
assert output == ["Sorry, I couldn't find any content to answer your question."]
|
||||||
|
|
||||||
@@ -250,9 +253,7 @@ def test_stream_chat_no_matching_nodes() -> None:
|
|||||||
mock_index.vector_store.get_nodes.return_value = []
|
mock_index.vector_store.get_nodes.return_value = []
|
||||||
mock_load_index.return_value = mock_index
|
mock_load_index.return_value = mock_index
|
||||||
|
|
||||||
output = list(
|
output = list(stream_chat_with_documents("Any info?", [MagicMock(pk=1)]))
|
||||||
stream_chat_with_documents("Any info?", _fake_documents_queryset([1])),
|
|
||||||
)
|
|
||||||
|
|
||||||
assert output == ["Sorry, I couldn't find any content to answer your question."]
|
assert output == ["Sorry, I couldn't find any content to answer your question."]
|
||||||
|
|
||||||
@@ -281,9 +282,7 @@ def test_stream_chat_unexpected_failure_returns_generic_error(caplog) -> None:
|
|||||||
)
|
)
|
||||||
mock_retriever_cls.return_value = mock_retriever
|
mock_retriever_cls.return_value = mock_retriever
|
||||||
|
|
||||||
output = list(
|
output = list(stream_chat_with_documents("Any info?", [MagicMock(pk=1)]))
|
||||||
stream_chat_with_documents("Any info?", _fake_documents_queryset([1])),
|
|
||||||
)
|
|
||||||
|
|
||||||
assert output == [CHAT_ERROR_MESSAGE]
|
assert output == [CHAT_ERROR_MESSAGE]
|
||||||
assert "Failed to stream document chat response" in caplog.text
|
assert "Failed to stream document chat response" in caplog.text
|
||||||
@@ -299,12 +298,7 @@ class TestStreamChatRetrieval:
|
|||||||
) -> None:
|
) -> None:
|
||||||
doc = DocumentFactory.create(content="hello world")
|
doc = DocumentFactory.create(content="hello world")
|
||||||
# Nothing indexed for this document yet.
|
# Nothing indexed for this document yet.
|
||||||
out = list(
|
out = list(chat.stream_chat_with_documents("question?", [doc]))
|
||||||
chat.stream_chat_with_documents(
|
|
||||||
"question?",
|
|
||||||
Document.objects.filter(pk=doc.pk),
|
|
||||||
),
|
|
||||||
)
|
|
||||||
assert chat.CHAT_NO_CONTENT_MESSAGE in out
|
assert chat.CHAT_NO_CONTENT_MESSAGE in out
|
||||||
|
|
||||||
def test_chat_filter_contains_only_requested_document_ids(
|
def test_chat_filter_contains_only_requested_document_ids(
|
||||||
@@ -338,12 +332,7 @@ class TestStreamChatRetrieval:
|
|||||||
side_effect=capture_retriever,
|
side_effect=capture_retriever,
|
||||||
)
|
)
|
||||||
|
|
||||||
list(
|
list(chat.stream_chat_with_documents("question?", [included]))
|
||||||
chat.stream_chat_with_documents(
|
|
||||||
"question?",
|
|
||||||
Document.objects.filter(pk=included.pk),
|
|
||||||
),
|
|
||||||
)
|
|
||||||
|
|
||||||
assert captured_filters, "VectorIndexRetriever was never constructed"
|
assert captured_filters, "VectorIndexRetriever was never constructed"
|
||||||
filt = captured_filters[0]
|
filt = captured_filters[0]
|
||||||
@@ -351,47 +340,3 @@ class TestStreamChatRetrieval:
|
|||||||
filter_values = filt.filters[0].value
|
filter_values = filt.filters[0].value
|
||||||
assert str(included.pk) in filter_values
|
assert str(included.pk) in filter_values
|
||||||
assert str(excluded.pk) not in filter_values
|
assert str(excluded.pk) not in filter_values
|
||||||
|
|
||||||
@pytest.mark.django_db
|
|
||||||
def test_get_document_references_only_queries_referenced_documents(
|
|
||||||
self,
|
|
||||||
django_assert_num_queries,
|
|
||||||
) -> None:
|
|
||||||
"""Building references must not hydrate every document the caller is
|
|
||||||
permitted to see -- only the (<= CHAT_RETRIEVER_TOP_K) documents that
|
|
||||||
the retriever actually returned nodes for.
|
|
||||||
"""
|
|
||||||
referenced = DocumentFactory.create(title="Referenced Document")
|
|
||||||
# Many more documents are "accessible" but never referenced by a node.
|
|
||||||
DocumentFactory.create_batch(200)
|
|
||||||
|
|
||||||
documents = Document.objects.all()
|
|
||||||
top_nodes = [
|
|
||||||
MagicMock(
|
|
||||||
metadata={
|
|
||||||
"document_id": str(referenced.pk),
|
|
||||||
"title": "Referenced Document",
|
|
||||||
},
|
|
||||||
),
|
|
||||||
]
|
|
||||||
|
|
||||||
hydrated_count = 0
|
|
||||||
|
|
||||||
def _count_hydration(sender, instance, **kwargs):
|
|
||||||
nonlocal hydrated_count
|
|
||||||
hydrated_count += 1
|
|
||||||
|
|
||||||
post_init.connect(_count_hydration, sender=Document)
|
|
||||||
try:
|
|
||||||
# One query: `documents.filter(pk__in=candidate_ids)` for the single
|
|
||||||
# referenced id. No query should scale with the 200 unreferenced documents.
|
|
||||||
with django_assert_num_queries(1):
|
|
||||||
references = chat._get_document_references(documents, top_nodes)
|
|
||||||
finally:
|
|
||||||
post_init.disconnect(_count_hydration, sender=Document)
|
|
||||||
|
|
||||||
# The bug this guards against: the old code hydrated all 201 accessible
|
|
||||||
# documents via `{doc.pk: doc for doc in documents}` before filtering by
|
|
||||||
# top_nodes. Only the referenced document should ever be constructed.
|
|
||||||
assert hydrated_count == 1
|
|
||||||
assert references == [{"id": referenced.pk, "title": "Referenced Document"}]
|
|
||||||
|
|||||||
@@ -757,3 +757,30 @@ class TestAPIProcessedMails(DirectoriesMixin, APITestCase):
|
|||||||
format="json",
|
format="json",
|
||||||
)
|
)
|
||||||
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
|
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
|
||||||
|
|
||||||
|
def test_bulk_delete_processed_mails_rejects_mixed_batch_atomically(self) -> None:
|
||||||
|
"""
|
||||||
|
GIVEN:
|
||||||
|
- A permitted processed mail and one the user may not delete
|
||||||
|
WHEN:
|
||||||
|
- API call bulk deletes both in a single request
|
||||||
|
THEN:
|
||||||
|
- The request is rejected and neither mail is deleted
|
||||||
|
"""
|
||||||
|
user2 = User.objects.create_user(username="temp_admin2")
|
||||||
|
rule = MailRuleFactory()
|
||||||
|
# Created first so it sorts ahead of the forbidden mail, i.e. the
|
||||||
|
# permission check has to cover the whole batch before deleting rather
|
||||||
|
# than rejecting only once it reaches the forbidden one.
|
||||||
|
pm_owned = ProcessedMailFactory(rule=rule, owner=self.user)
|
||||||
|
pm_forbidden = ProcessedMailFactory(rule=rule, owner=user2)
|
||||||
|
|
||||||
|
response = self.client.post(
|
||||||
|
f"{self.ENDPOINT}bulk_delete/",
|
||||||
|
data={"mail_ids": [pm_owned.id, pm_forbidden.id]},
|
||||||
|
format="json",
|
||||||
|
)
|
||||||
|
|
||||||
|
self.assertEqual(response.status_code, status.HTTP_403_FORBIDDEN)
|
||||||
|
self.assertTrue(ProcessedMail.objects.filter(id=pm_owned.id).exists())
|
||||||
|
self.assertTrue(ProcessedMail.objects.filter(id=pm_forbidden.id).exists())
|
||||||
|
|||||||
@@ -27,6 +27,7 @@ from documents.filters import PermittedObjectsFilter
|
|||||||
from documents.models import PaperlessTask
|
from documents.models import PaperlessTask
|
||||||
from documents.permissions import PaperlessObjectPermissions
|
from documents.permissions import PaperlessObjectPermissions
|
||||||
from documents.permissions import has_perms_owner_aware
|
from documents.permissions import has_perms_owner_aware
|
||||||
|
from documents.permissions import permitted_object_ids
|
||||||
from documents.views import PassUserMixin
|
from documents.views import PassUserMixin
|
||||||
from paperless.views import StandardPagination
|
from paperless.views import StandardPagination
|
||||||
from paperless_mail.filters import ProcessedMailFilterSet
|
from paperless_mail.filters import ProcessedMailFilterSet
|
||||||
@@ -211,10 +212,17 @@ class ProcessedMailViewSet(PassUserMixin, ReadOnlyModelViewSet[ProcessedMail]):
|
|||||||
):
|
):
|
||||||
return HttpResponseBadRequest("mail_ids must be a list of integers")
|
return HttpResponseBadRequest("mail_ids must be a list of integers")
|
||||||
mails = ProcessedMail.objects.filter(id__in=mail_ids)
|
mails = ProcessedMail.objects.filter(id__in=mail_ids)
|
||||||
for mail in mails:
|
# Check every id up front so an unpermitted one rejects the whole
|
||||||
if not has_perms_owner_aware(request.user, "delete_processedmail", mail):
|
# request rather than deleting the mails ahead of it first.
|
||||||
|
if mails.exclude(
|
||||||
|
pk__in=permitted_object_ids(
|
||||||
|
request.user,
|
||||||
|
ProcessedMail,
|
||||||
|
"delete_processedmail",
|
||||||
|
),
|
||||||
|
).exists():
|
||||||
return HttpResponseForbidden("Insufficient permissions")
|
return HttpResponseForbidden("Insufficient permissions")
|
||||||
mail.delete()
|
mails.delete()
|
||||||
return Response({"result": "OK", "deleted_mail_ids": mail_ids})
|
return Response({"result": "OK", "deleted_mail_ids": mail_ids})
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1298,16 +1298,16 @@ wheels = [
|
|||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "fpdf2"
|
name = "fpdf2"
|
||||||
version = "2.8.7"
|
version = "2.8.8"
|
||||||
source = { registry = "https://pypi.org/simple" }
|
source = { registry = "https://pypi.org/simple" }
|
||||||
dependencies = [
|
dependencies = [
|
||||||
{ name = "defusedxml", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
{ name = "defusedxml", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
||||||
{ name = "fonttools", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
{ name = "fonttools", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
||||||
{ name = "pillow", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
{ name = "pillow", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
||||||
]
|
]
|
||||||
sdist = { url = "https://files.pythonhosted.org/packages/27/f2/72feae0b2827ed38013e4307b14f95bf0b3d124adfef4d38a7d57533f7be/fpdf2-2.8.7.tar.gz", hash = "sha256:7060ccee5a9c7ab0a271fb765a36a23639f83ef8996c34e3d46af0a17ede57f9", size = 362351, upload-time = "2026-02-28T05:39:16.456Z" }
|
sdist = { url = "https://files.pythonhosted.org/packages/1e/bc/8fd4321aed40cadadddc8f311c65b6082346b252bca048f7b476d8f35d72/fpdf2-2.8.8.tar.gz", hash = "sha256:9e94e155e85e8053329a9a1fce8b566fd7a7c5bb79e98a1a3952d379b947c5b9", size = 374689, upload-time = "2026-08-09T23:32:45.334Z" }
|
||||||
wheels = [
|
wheels = [
|
||||||
{ url = "https://files.pythonhosted.org/packages/66/0a/cf50ecffa1e3747ed9380a3adfc829259f1f86b3fdbd9e505af789003141/fpdf2-2.8.7-py3-none-any.whl", hash = "sha256:d391fc508a3ce02fc43a577c830cda4fe6f37646f2d143d489839940932fbc19", size = 327056, upload-time = "2026-02-28T05:39:14.619Z" },
|
{ url = "https://files.pythonhosted.org/packages/f5/be/af012eda9507494f28b99b077423806c43a11573eb6225dd46f19ae2d263/fpdf2-2.8.8-py3-none-any.whl", hash = "sha256:3557a478fc577a929c94aace9666aed4dcc432b5ab6764232e6a59f1ccd75f17", size = 337000, upload-time = "2026-08-09T23:32:43.728Z" },
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
@@ -2927,8 +2927,8 @@ dependencies = [
|
|||||||
{ name = "sqlite-vec", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
{ name = "sqlite-vec", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
||||||
{ name = "tantivy", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
{ name = "tantivy", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
||||||
{ name = "tika-client", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
{ name = "tika-client", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
||||||
{ name = "torch", version = "2.13.0", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "sys_platform == 'darwin'" },
|
{ name = "torch", version = "2.13.0", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "python_full_version < '3.15' and sys_platform == 'darwin'" },
|
||||||
{ name = "torch", version = "2.13.0+cpu", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "sys_platform == 'linux'" },
|
{ name = "torch", version = "2.13.0+cpu", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "(python_full_version >= '3.15' and sys_platform == 'darwin') or sys_platform == 'linux'" },
|
||||||
{ name = "watchfiles", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
{ name = "watchfiles", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
||||||
{ name = "whitenoise", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
{ name = "whitenoise", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
||||||
{ name = "zxing-cpp", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
{ name = "zxing-cpp", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
||||||
@@ -4511,8 +4511,8 @@ dependencies = [
|
|||||||
{ name = "numpy", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
{ name = "numpy", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
||||||
{ name = "scikit-learn", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
{ name = "scikit-learn", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
||||||
{ name = "scipy", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
{ name = "scipy", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
||||||
{ name = "torch", version = "2.13.0", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "sys_platform == 'darwin'" },
|
{ name = "torch", version = "2.13.0", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "python_full_version < '3.15' and sys_platform == 'darwin'" },
|
||||||
{ name = "torch", version = "2.13.0+cpu", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "sys_platform == 'linux'" },
|
{ name = "torch", version = "2.13.0+cpu", source = { registry = "https://download.pytorch.org/whl/cpu" }, marker = "(python_full_version >= '3.15' and sys_platform == 'darwin') or sys_platform == 'linux'" },
|
||||||
{ name = "tqdm", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
{ name = "tqdm", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
||||||
{ name = "transformers", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
{ name = "transformers", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
||||||
{ name = "typing-extensions", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
{ name = "typing-extensions", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
||||||
@@ -4957,18 +4957,17 @@ name = "torch"
|
|||||||
version = "2.13.0"
|
version = "2.13.0"
|
||||||
source = { registry = "https://download.pytorch.org/whl/cpu" }
|
source = { registry = "https://download.pytorch.org/whl/cpu" }
|
||||||
resolution-markers = [
|
resolution-markers = [
|
||||||
"python_full_version >= '3.15' and sys_platform == 'darwin'",
|
|
||||||
"python_full_version >= '3.12' and python_full_version < '3.15' and sys_platform == 'darwin'",
|
"python_full_version >= '3.12' and python_full_version < '3.15' and sys_platform == 'darwin'",
|
||||||
"python_full_version < '3.12' and sys_platform == 'darwin'",
|
"python_full_version < '3.12' and sys_platform == 'darwin'",
|
||||||
]
|
]
|
||||||
dependencies = [
|
dependencies = [
|
||||||
{ name = "filelock", marker = "sys_platform == 'darwin'" },
|
{ name = "filelock", marker = "python_full_version < '3.15' and sys_platform == 'darwin'" },
|
||||||
{ name = "fsspec", marker = "sys_platform == 'darwin'" },
|
{ name = "fsspec", marker = "python_full_version < '3.15' and sys_platform == 'darwin'" },
|
||||||
{ name = "jinja2", marker = "sys_platform == 'darwin'" },
|
{ name = "jinja2", marker = "python_full_version < '3.15' and sys_platform == 'darwin'" },
|
||||||
{ name = "networkx", marker = "sys_platform == 'darwin'" },
|
{ name = "networkx", marker = "python_full_version < '3.15' and sys_platform == 'darwin'" },
|
||||||
{ name = "setuptools", marker = "sys_platform == 'darwin'" },
|
{ name = "setuptools", marker = "python_full_version < '3.15' and sys_platform == 'darwin'" },
|
||||||
{ name = "sympy", marker = "sys_platform == 'darwin'" },
|
{ name = "sympy", marker = "python_full_version < '3.15' and sys_platform == 'darwin'" },
|
||||||
{ name = "typing-extensions", marker = "sys_platform == 'darwin'" },
|
{ name = "typing-extensions", marker = "python_full_version < '3.15' and sys_platform == 'darwin'" },
|
||||||
]
|
]
|
||||||
wheels = [
|
wheels = [
|
||||||
{ url = "https://download-r2.pytorch.org/whl/cpu/torch-2.13.0-cp311-cp311-macosx_14_0_arm64.whl", hash = "sha256:e76f9bcecc52b8ff711239a2f7547d5353df95878ab232f0773c1d95928b92f8", upload-time = "2026-07-08T12:26:13Z" },
|
{ url = "https://download-r2.pytorch.org/whl/cpu/torch-2.13.0-cp311-cp311-macosx_14_0_arm64.whl", hash = "sha256:e76f9bcecc52b8ff711239a2f7547d5353df95878ab232f0773c1d95928b92f8", upload-time = "2026-07-08T12:26:13Z" },
|
||||||
@@ -4983,6 +4982,7 @@ name = "torch"
|
|||||||
version = "2.13.0+cpu"
|
version = "2.13.0+cpu"
|
||||||
source = { registry = "https://download.pytorch.org/whl/cpu" }
|
source = { registry = "https://download.pytorch.org/whl/cpu" }
|
||||||
resolution-markers = [
|
resolution-markers = [
|
||||||
|
"python_full_version >= '3.15' and sys_platform == 'darwin'",
|
||||||
"python_full_version == '3.12.*' and platform_machine == 'x86_64' and sys_platform == 'linux'",
|
"python_full_version == '3.12.*' and platform_machine == 'x86_64' and sys_platform == 'linux'",
|
||||||
"python_full_version == '3.12.*' and platform_machine == 'aarch64' and sys_platform == 'linux'",
|
"python_full_version == '3.12.*' and platform_machine == 'aarch64' and sys_platform == 'linux'",
|
||||||
"python_full_version >= '3.15' and sys_platform == 'linux'",
|
"python_full_version >= '3.15' and sys_platform == 'linux'",
|
||||||
@@ -4990,13 +4990,13 @@ resolution-markers = [
|
|||||||
"python_full_version < '3.12' and sys_platform == 'linux'",
|
"python_full_version < '3.12' and sys_platform == 'linux'",
|
||||||
]
|
]
|
||||||
dependencies = [
|
dependencies = [
|
||||||
{ name = "filelock", marker = "sys_platform == 'linux'" },
|
{ name = "filelock", marker = "(python_full_version >= '3.15' and sys_platform == 'darwin') or sys_platform == 'linux'" },
|
||||||
{ name = "fsspec", marker = "sys_platform == 'linux'" },
|
{ name = "fsspec", marker = "(python_full_version >= '3.15' and sys_platform == 'darwin') or sys_platform == 'linux'" },
|
||||||
{ name = "jinja2", marker = "sys_platform == 'linux'" },
|
{ name = "jinja2", marker = "(python_full_version >= '3.15' and sys_platform == 'darwin') or sys_platform == 'linux'" },
|
||||||
{ name = "networkx", marker = "sys_platform == 'linux'" },
|
{ name = "networkx", marker = "(python_full_version >= '3.15' and sys_platform == 'darwin') or sys_platform == 'linux'" },
|
||||||
{ name = "setuptools", marker = "sys_platform == 'linux'" },
|
{ name = "setuptools", marker = "(python_full_version >= '3.15' and sys_platform == 'darwin') or sys_platform == 'linux'" },
|
||||||
{ name = "sympy", marker = "sys_platform == 'linux'" },
|
{ name = "sympy", marker = "(python_full_version >= '3.15' and sys_platform == 'darwin') or sys_platform == 'linux'" },
|
||||||
{ name = "typing-extensions", marker = "sys_platform == 'linux'" },
|
{ name = "typing-extensions", marker = "(python_full_version >= '3.15' and sys_platform == 'darwin') or sys_platform == 'linux'" },
|
||||||
]
|
]
|
||||||
wheels = [
|
wheels = [
|
||||||
{ url = "https://download-r2.pytorch.org/whl/cpu/torch-2.13.0%2Bcpu-cp311-cp311-linux_s390x.whl", hash = "sha256:6e9817dbdf5ea76789babd46e457eac5bf14ff566cf85f8addbfdff2d56601ce", upload-time = "2026-07-08T19:27:52Z" },
|
{ url = "https://download-r2.pytorch.org/whl/cpu/torch-2.13.0%2Bcpu-cp311-cp311-linux_s390x.whl", hash = "sha256:6e9817dbdf5ea76789babd46e457eac5bf14ff566cf85f8addbfdff2d56601ce", upload-time = "2026-07-08T19:27:52Z" },
|
||||||
|
|||||||
Reference in New Issue
Block a user