Compare commits

..
Author SHA1 Message Date
shamoon bd3fa68e14 lightbulb 2026-09-25 12:42:29 -07:00
shamoon cd12c34d8d Much simpler 2026-09-25 11:59:57 -07:00
shamoon d57895ec11 Docs 2026-09-24 15:25:16 -07:00
shamoon 5d0dd4e9f7 Tweak setting wording / control 2026-09-24 15:24:06 -07:00
shamoon a5c35d80fa active color 2026-09-24 15:19:29 -07:00
shamoon 19e352db71 Alignment thing 2026-09-24 15:18:53 -07:00
shamoon 548c2c643c icons 2026-09-24 15:12:54 -07:00
shamoon a72c6c6a92 ... button for choosing which type 2026-09-24 14:53:11 -07:00
shamoon 5e30ab5e7e Support fetching both and merging suggestions 2026-09-24 10:18:08 -07:00
shamoon f57427dde2 new setting 2026-09-24 09:32:09 -07:00
Trenton H 2a44d8b5ba Fix: During a move to the trash directory, attempt to copy metadata, but don't let it fail the move (#14250) 2026-09-23 21:51:22 -07:00
Trenton H abf5050ea7 Fix: convert file mtime to the configured time zone directly (#14249)
The `created` date fallback derived a naive datetime from a file's
mtime using the OS-local zone, then labeled it as the configured
TIME_ZONE without converting. When the OS-local zone and TIME_ZONE
disagree, or when the C library can't resolve zoneinfo at all (as in
some sandboxed environments, where it silently falls back to UTC),
the resulting date can land on the wrong calendar day.

Convert the timestamp directly into the target zone with `tz=` on
fromtimestamp() instead of a naive conversion plus make_aware().
2026-09-23 15:00:48 -07:00
GitHub Actions 091ddf7c45 Auto translate strings 2026-09-23 19:13:55 +00:00
shamoon c9f7f2cfbe Fix: ensure documentDeleted subscription is discarded (#14247) 2026-09-23 12:12:26 -07:00
Trenton H b457610ffb Chore: Fix bugs in the test suite (#14244)
* Fix: redirect SHARE_LINK_BUNDLE_DIR to the test temp layout instead of the real media directory

* Fix: include f_to in test_filters subTest labels so each of the 8 cases reports distinctly

* Fix: run the post_consume error-log assertion after the raising call and match the actual paperless_mail logger name

* Fix: rename the blank-password workflow test to match its behavior and add a real wrong-password-fails test

* Fix: use the created social account's actual pk and remove an accidental tuple wrapping the mock provider

* Fix: assert against the created documents' actual pks instead of hardcoded 1 and 2

* Fix: assert test_compression actually produces a valid LZMA-compressed zip

* Fix: clear os.environ when patching PAPERLESS_ADMIN_* vars so a host-set value can't leak into the no-user test

* Fix: restore MIDDLEWARE, AUTHENTICATION_BACKENDS and REST_FRAMEWORK auth classes after each remote-user settings test instead of leaking the mutation into later tests

* Fix: use a guaranteed-nonexistent temp path instead of hardcoded /tmp/foo/bar in test_export_target_not_exists
2026-09-23 19:11:58 +00:00
GitHub Actions 969c2ea0e2 Auto translate strings 2026-09-23 19:00:47 +00:00
shamoon 31b806a285 Fix: ensure bulk operations are checked against version root (#14246) 2026-09-23 18:59:27 +00:00
GitHub Actions 99851b418c Auto translate strings 2026-09-23 18:46:43 +00:00
e34eda07bb Enhancement: support separate embedding API key (#14067)
Co-authored-by: Claude Code <noreply@anthropic.com>
Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
2026-09-23 11:44:37 -07:00
GitHub Actions 793459b416 Auto translate strings 2026-09-23 18:22:05 +00:00
shamoon 04297fd02c Enhancement: support passthrough extra params for LLMs (#14202) 2026-09-23 18:20:43 +00:00
dependabot[bot] 1b277dd8e1 Chore(deps): Bump autobahn in the uv group across 1 directory (#14231)
Bumps the uv group with 1 update in the / directory: [autobahn](https://github.com/crossbario/autobahn-python).


Updates `autobahn` from 25.12.2 to 26.7.1
- [Release notes](https://github.com/crossbario/autobahn-python/releases)
- [Changelog](https://github.com/crossbario/autobahn-python/blob/master/docs/changelog.rst)
- [Commits](https://github.com/crossbario/autobahn-python/compare/v25.12.2...v26.7.1)

---
updated-dependencies:
- dependency-name: autobahn
  dependency-version: 26.7.1
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-09-23 15:33:01 +00:00
shamoon 3c20abeb4c Fix: indexing after document-added workflows signal (#14242) 2026-09-23 14:50:48 +00:00
GitHub Actions b11f1f8459 Auto translate strings 2026-09-23 03:19:06 +00:00
Trenton H 7424e7ce0b Fix: Record full tag and custom field lists in bulk edit audit log (#14236)
Snapshot only the edited field, before and after the operation, gathering tags
and custom field instances into sorted id lists per document (empty when there
are none).
2026-09-23 03:17:40 +00:00
shamoon a53a3d3769 Chore: update pikepdf for ocrmypdf requirement (#14235) 2026-09-22 19:01:21 -07:00
62 changed files with 1523 additions and 3504 deletions
+14 -9
View File
@@ -136,13 +136,15 @@ for suggested generation and embedding models.
### AI-assisted suggestions
With AI enabled, Paperless-ngx can suggest a title, tags, correspondent, document type,
storage path and dates by sending the document to the LLM. This is **opt-in per request**
and surfaces through the "Suggest" control on the document detail page, alongside the
classic classifier-based suggestions — it does not disable them. Suggestions are requested
automatically when you open a document that carries an inbox tag unless "Automatically request
suggestions for inbox documents" under Settings > Documents is disabled. Suggestion output
language can be steered with
[`PAPERLESS_AI_LLM_OUTPUT_LANGUAGE`](configuration.md#PAPERLESS_AI_LLM_OUTPUT_LANGUAGE)
storage path and dates by sending the document to the LLM using "Suggest" button on the document
detail page. You can choose which type of suggestions are requested by default under Settings >
Documents, either ML (classifier-based) suggestions, AI suggestions, or both. When both are requested
the results are combined.
Suggestions are requested automatically when you open a document that carries an inbox tag
unless "Automatically request suggestions for inbox documents" under Settings > Documents is disabled.
Suggestion output language can be steered with [`PAPERLESS_AI_LLM_OUTPUT_LANGUAGE`](configuration.md#PAPERLESS_AI_LLM_OUTPUT_LANGUAGE)
(otherwise it follows the user's UI language).
### The LLM index (RAG) and similar documents
@@ -153,8 +155,11 @@ in similar existing documents, and the document chat can retrieve relevant conte
Enable it by setting
[`PAPERLESS_AI_LLM_EMBEDDING_BACKEND`](configuration.md#PAPERLESS_AI_LLM_EMBEDDING_BACKEND)
(`huggingface` for fully-local embeddings, or `ollama` / `openai-like`). The index is only
built when AI is enabled **and** an embedding backend is set.
(`huggingface` for fully-local embeddings, or `ollama` / `openai-like`). By default, the main
LLM API key and endpoint are used, but an optional embedding-specific[API key](configuration.md#PAPERLESS_AI_LLM_EMBEDDING_API_KEY)
and [endpoint](configuration.md#PAPERLESS_AI_LLM_EMBEDDING_ENDPOINT) can be configured.
The index is only built when AI is enabled **and** an embedding backend is set.
The index is updated automatically on a schedule controlled by
[`PAPERLESS_LLM_INDEX_TASK_CRON`](configuration.md#PAPERLESS_LLM_INDEX_TASK_CRON) (daily by
+21 -6
View File
@@ -1576,9 +1576,6 @@ ports.
#### [`PAPERLESS_WEBHOOKS_ALLOW_INTERNAL_REQUESTS=<bool>`](#PAPERLESS_WEBHOOKS_ALLOW_INTERNAL_REQUESTS) {#PAPERLESS_WEBHOOKS_ALLOW_INTERNAL_REQUESTS}
: If set to false, webhooks cannot be sent to internal URLs (e.g., localhost).
A hostname is blocked if any of the addresses it resolves to is non-public.
Webhook requests connect directly, without using the `HTTP_PROXY` or
`HTTPS_PROXY` environment variables, and never follow redirects.
Defaults to true, which allows internal requests.
@@ -1587,7 +1584,7 @@ Webhook requests connect directly, without using the `HTTP_PROXY` or
#### [`PAPERLESS_EMAIL_ALLOW_INTERNAL_HOSTS=<bool>`](#PAPERLESS_EMAIL_ALLOW_INTERNAL_HOSTS) {#PAPERLESS_EMAIL_ALLOW_INTERNAL_HOSTS}
: If set to false, incoming mail account connections are blocked when the
configured IMAP hostname resolves to any non-public address (for example,
configured IMAP hostname resolves to a non-public address (for example,
localhost, link-local, or RFC1918 private ranges).
Defaults to true, which allows internal hosts.
@@ -2136,6 +2133,13 @@ for language and resource considerations.
Defaults to None.
#### [`PAPERLESS_AI_LLM_EMBEDDING_API_KEY=<str>`](#PAPERLESS_AI_LLM_EMBEDDING_API_KEY) {#PAPERLESS_AI_LLM_EMBEDDING_API_KEY}
: The API key to use for the embedding backend. If not supplied, embeddings use
`PAPERLESS_AI_LLM_API_KEY`.
Defaults to None.
#### [`PAPERLESS_AI_LLM_EMBEDDING_ENDPOINT=<str>`](#PAPERLESS_AI_LLM_EMBEDDING_ENDPOINT) {#PAPERLESS_AI_LLM_EMBEDDING_ENDPOINT}
: The endpoint / url to use for the embedding backend. If not supplied, embeddings use
@@ -2217,11 +2221,22 @@ used with the OpenAI-compatible backend to target a custom provider or local gat
#### [`PAPERLESS_AI_LLM_ALLOW_INTERNAL_ENDPOINTS=<bool>`](#PAPERLESS_AI_LLM_ALLOW_INTERNAL_ENDPOINTS) {#PAPERLESS_AI_LLM_ALLOW_INTERNAL_ENDPOINTS}
: If set to false, Paperless blocks AI endpoint URLs that resolve to non-public addresses (e.g., localhost, etc).
A hostname is blocked if any of the addresses it resolves to is non-public, and redirects are checked the same way.
Requests to a configured AI endpoint connect directly, without using the `HTTP_PROXY` or `HTTPS_PROXY` environment variables.
Defaults to true, which allows internal endpoints.
#### [`PAPERLESS_AI_LLM_EXTRA_PARAMS=<json>`](#PAPERLESS_AI_LLM_EXTRA_PARAMS) {#PAPERLESS_AI_LLM_EXTRA_PARAMS}
: A JSON object of extra parameters sent with every LLM request, for providers that require a parameter Paperless does not
set itself. Values here override Paperless' own, and no validation is performed. Whatever you put here is passed to the
backend as-is, so an invalid parameter will simply be rejected by your provider. For example, current OpenAI reasoning
models refuse tool calls on the chat completions API unless reasoning is off:
```
PAPERLESS_AI_LLM_EXTRA_PARAMS={"reasoning_effort": "none"}
```
Defaults to empty, which adds nothing to requests.
#### [`PAPERLESS_LLM_INDEX_TASK_CRON=<cron expression>`](#PAPERLESS_LLM_INDEX_TASK_CRON) {#PAPERLESS_LLM_INDEX_TASK_CRON}
: Configures the schedule to update the AI embeddings of text content and metadata for all documents. Only performed if
+1 -1
View File
@@ -613,7 +613,7 @@ The following workflow action types are available:
- The request headers as key-value pairs
For security reasons, webhooks can be limited to specific ports and disallowed from connecting to local URLs. See the relevant
[configuration settings](configuration.md#workflow-webhooks) to change this behavior. Webhook requests connect directly (proxy environment variables are not used) and do not follow redirects. If you are allowing non-admins to create workflows,
[configuration settings](configuration.md#workflow-webhooks) to change this behavior. If you are allowing non-admins to create workflows,
you may want to adjust these settings to prevent abuse.
##### Move to Trash {#workflow-action-move-to-trash}
-3
View File
@@ -17,7 +17,6 @@ classifiers = [
# TODO: Move certain things to groups and then utilize that further
# This will allow testing to not install a webserver, mysql, etc
dependencies = [
"anyio>=4.12",
"azure-ai-documentintelligence>=1.0.2",
"babel>=2.17",
"bleach~=6.4.0",
@@ -48,8 +47,6 @@ dependencies = [
"filelock~=3.32.0",
"flower>=2.0.1,<2.2",
"gotenberg-client[httpx]~=1.0",
"httpcore~=1.0.9",
"httpx~=0.28.1",
"httpx-oauth~=0.17",
"ijson>=3.5.1",
"imap-tools>=1.14,<1.16",
+35 -21
View File
@@ -9745,7 +9745,7 @@
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-list/document-list.component.ts</context>
<context context-type="linenumber">348</context>
<context context-type="linenumber">351</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/manage/document-attributes/document-attributes.component.html</context>
@@ -9760,7 +9760,7 @@
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-list/document-list.component.ts</context>
<context context-type="linenumber">341</context>
<context context-type="linenumber">344</context>
</context-group>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/manage/document-attributes/document-attributes.component.html</context>
@@ -10016,56 +10016,56 @@
<source>Reset filters / selection</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-list/document-list.component.ts</context>
<context context-type="linenumber">329</context>
<context context-type="linenumber">332</context>
</context-group>
</trans-unit>
<trans-unit id="4135055128446167640" datatype="html">
<source>Open first [selected] document</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-list/document-list.component.ts</context>
<context context-type="linenumber">357</context>
<context context-type="linenumber">360</context>
</context-group>
</trans-unit>
<trans-unit id="3629960544875360046" datatype="html">
<source>Previous page</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-list/document-list.component.ts</context>
<context context-type="linenumber">373</context>
<context context-type="linenumber">376</context>
</context-group>
</trans-unit>
<trans-unit id="3337301694210287595" datatype="html">
<source>Next page</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-list/document-list.component.ts</context>
<context context-type="linenumber">385</context>
<context context-type="linenumber">388</context>
</context-group>
</trans-unit>
<trans-unit id="2155249406916744630" datatype="html">
<source>View &quot;<x id="PH" equiv-text="this.list.activeSavedViewTitle"/>&quot; saved successfully.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-list/document-list.component.ts</context>
<context context-type="linenumber">419</context>
<context context-type="linenumber">422</context>
</context-group>
</trans-unit>
<trans-unit id="4646273665293421938" datatype="html">
<source>Failed to save view &quot;<x id="PH" equiv-text="this.list.activeSavedViewTitle"/>&quot;.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-list/document-list.component.ts</context>
<context context-type="linenumber">425</context>
<context context-type="linenumber">428</context>
</context-group>
</trans-unit>
<trans-unit id="6837554170707123455" datatype="html">
<source>View &quot;<x id="PH" equiv-text="savedView.name"/>&quot; created successfully.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-list/document-list.component.ts</context>
<context context-type="linenumber">494</context>
<context context-type="linenumber">497</context>
</context-group>
</trans-unit>
<trans-unit id="6028096992841030074" datatype="html">
<source>View &quot;<x id="PH" equiv-text="savedView.name"/>&quot; created successfully, but could not update visibility settings.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/components/document-list/document-list.component.ts</context>
<context context-type="linenumber">500</context>
<context context-type="linenumber">503</context>
</context-group>
</trans-unit>
<trans-unit id="739880801667335279" datatype="html">
@@ -12018,81 +12018,95 @@
<context context-type="linenumber">351</context>
</context-group>
</trans-unit>
<trans-unit id="861068592166833023" datatype="html">
<source>LLM Embedding API Key</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/data/paperless-config.ts</context>
<context context-type="linenumber">358</context>
</context-group>
</trans-unit>
<trans-unit id="2929108042259892948" datatype="html">
<source>Used for embeddings when set, otherwise LLM API key is used.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/data/paperless-config.ts</context>
<context context-type="linenumber">360</context>
</context-group>
</trans-unit>
<trans-unit id="3554114880473286122" datatype="html">
<source>LLM Embedding Endpoint</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/data/paperless-config.ts</context>
<context context-type="linenumber">358</context>
<context context-type="linenumber">366</context>
</context-group>
</trans-unit>
<trans-unit id="1044242175651289991" datatype="html">
<source>LLM Embedding Chunk Size</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/data/paperless-config.ts</context>
<context context-type="linenumber">365</context>
<context context-type="linenumber">373</context>
</context-group>
</trans-unit>
<trans-unit id="7218245223139363113" datatype="html">
<source>LLM Context Size</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/data/paperless-config.ts</context>
<context context-type="linenumber">372</context>
<context context-type="linenumber">380</context>
</context-group>
</trans-unit>
<trans-unit id="4234495692726214397" datatype="html">
<source>LLM Backend</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/data/paperless-config.ts</context>
<context context-type="linenumber">379</context>
<context context-type="linenumber">387</context>
</context-group>
</trans-unit>
<trans-unit id="7935234833834000002" datatype="html">
<source>LLM Model</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/data/paperless-config.ts</context>
<context context-type="linenumber">387</context>
<context context-type="linenumber">395</context>
</context-group>
</trans-unit>
<trans-unit id="1980550530387803165" datatype="html">
<source>LLM API Key</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/data/paperless-config.ts</context>
<context context-type="linenumber">394</context>
<context context-type="linenumber">402</context>
</context-group>
</trans-unit>
<trans-unit id="6126617860376156501" datatype="html">
<source>LLM Endpoint</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/data/paperless-config.ts</context>
<context context-type="linenumber">401</context>
<context context-type="linenumber">409</context>
</context-group>
</trans-unit>
<trans-unit id="6572826277249350975" datatype="html">
<source>LLM Output Language</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/data/paperless-config.ts</context>
<context context-type="linenumber">408</context>
<context context-type="linenumber">416</context>
</context-group>
</trans-unit>
<trans-unit id="3284403507172415792" datatype="html">
<source>Language to use for generated AI suggestions. When unset, AI suggestions use the user&apos;s display language if explicitly set.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/data/paperless-config.ts</context>
<context context-type="linenumber">412</context>
<context context-type="linenumber">420</context>
</context-group>
</trans-unit>
<trans-unit id="4493921125434706859" datatype="html">
<source>LLM Request Timeout</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/data/paperless-config.ts</context>
<context context-type="linenumber">416</context>
<context context-type="linenumber">424</context>
</context-group>
</trans-unit>
<trans-unit id="483994032066441287" datatype="html">
<source>Timeout in seconds for LLM requests.</source>
<context-group purpose="location">
<context context-type="sourcefile">src/app/data/paperless-config.ts</context>
<context context-type="linenumber">420</context>
<context context-type="linenumber">428</context>
</context-group>
</trans-unit>
<trans-unit id="1055686627716339120" datatype="html">
@@ -253,6 +253,24 @@
</div>
</div>
@if (aiEnabled) {
<div class="row mb-3">
<div class="col-md-3 col-form-label pt-0">
<span i18n>Suggestions default to</span>
</div>
<div class="col">
<fieldset class="btn-group btn-group-sm">
<input type="radio" class="btn-check" id="suggestionSourceBoth" [value]="SuggestionSource.Both" formControlName="documentEditingSuggestionSource">
<label class="btn btn-outline-primary" for="suggestionSourceBoth"><ng-container i18n>Both</ng-container></label>
<input type="radio" class="btn-check" id="suggestionSourceML" [value]="SuggestionSource.ML" formControlName="documentEditingSuggestionSource">
<label class="btn btn-outline-primary" for="suggestionSourceML"><i-bs class="me-1" name="cpu"></i-bs><ng-container i18n>ML only</ng-container></label>
<input type="radio" class="btn-check" id="suggestionSourceAI" [value]="SuggestionSource.AI" formControlName="documentEditingSuggestionSource">
<label class="btn btn-outline-primary" for="suggestionSourceAI"><i-bs class="me-1" name="stars"></i-bs><ng-container i18n>AI only</ng-container></label>
</fieldset>
</div>
</div>
}
<div class="row">
<div class="col">
<pngx-input-check i18n-title title="Automatically request suggestions for inbox documents" i18n-hint hint="If un-checked, suggestions must be requested via the Suggest button." formControlName="documentEditingAutoSuggest"></pngx-input-check>
@@ -307,7 +307,7 @@ describe('SettingsComponent', () => {
expect(toastErrorSpy).toHaveBeenCalled()
expect(storeSpy).toHaveBeenCalled()
expect(appearanceSettingsSpy).not.toHaveBeenCalled()
expect(setSpy).toHaveBeenCalledTimes(34)
expect(setSpy).toHaveBeenCalledTimes(35)
expect(setSpy).toHaveBeenCalledWith(SETTINGS_KEYS.SIDEBAR_HIDDEN_ITEMS, [
HideableSidebarItemID.Workflows,
])
@@ -44,6 +44,7 @@ import {
HIDEABLE_SIDEBAR_ITEM_IDS,
HideableSidebarItemID,
SETTINGS_KEYS,
SuggestionSource,
} from 'src/app/data/ui-settings'
import { User } from 'src/app/data/user'
import { IfPermissionsDirective } from 'src/app/directives/if-permissions.directive'
@@ -184,6 +185,7 @@ export class SettingsComponent
documentEditingRemoveInboxTags: new FormControl(null),
documentEditingOverlayThumbnail: new FormControl(null),
documentEditingAutoSuggest: new FormControl(null),
documentEditingSuggestionSource: new FormControl(null),
documentDetailsHiddenFields: new FormControl([]),
searchDbOnly: new FormControl(null),
searchLink: new FormControl(null),
@@ -217,6 +219,11 @@ export class SettingsComponent
public readonly PdfZoomScale = PdfZoomScale
public readonly PdfEditorEditMode = PdfEditorEditMode
public readonly SuggestionSource = SuggestionSource
get aiEnabled(): boolean {
return this.settings.get(SETTINGS_KEYS.AI_ENABLED)
}
public readonly documentDetailFieldOptions = documentDetailFieldOptions
public readonly sidebarItemOptions = HIDEABLE_SIDEBAR_ITEM_IDS.map((id) => ({
@@ -404,6 +411,9 @@ export class SettingsComponent
documentEditingAutoSuggest: this.settings.get(
SETTINGS_KEYS.DOCUMENT_EDITING_AUTO_SUGGEST
),
documentEditingSuggestionSource: this.settings.get(
SETTINGS_KEYS.DOCUMENT_EDITING_SUGGESTION_SOURCE
),
documentDetailsHiddenFields: this.settings.get(
SETTINGS_KEYS.DOCUMENT_DETAILS_HIDDEN_FIELDS
),
@@ -625,6 +635,10 @@ export class SettingsComponent
SETTINGS_KEYS.DOCUMENT_EDITING_AUTO_SUGGEST,
this.settingsForm.value.documentEditingAutoSuggest
)
this.settings.set(
SETTINGS_KEYS.DOCUMENT_EDITING_SUGGESTION_SOURCE,
this.settingsForm.value.documentEditingSuggestionSource
)
this.settings.set(
SETTINGS_KEYS.DOCUMENT_DETAILS_HIDDEN_FIELDS,
this.settingsForm.value.documentDetailsHiddenFields
@@ -1,58 +1,84 @@
<div class="btn-group">
<button type="button" class="btn btn-sm btn-outline-primary" (click)="clickSuggest()" [disabled]="disabled() || loading() || (suggestions() && !aiEnabled())" [aria-label]="noSuggestions ? 'No suggestions' : 'Suggest'" i18n-aria-label>
@if (loading()) {
<div class="spinner-border spinner-border-sm" role="status"></div>
} @else if (noSuggestions) {
<i-bs width="1.2em" height="1.2em" name="check-circle"></i-bs>
} @else {
<i-bs width="1.2em" height="1.2em" name="stars"></i-bs>
<div class="d-flex align-items-center">
<div class="btn-group">
<button type="button" class="btn btn-sm btn-outline-primary" (click)="clickSuggest()" [disabled]="disabled() || loading() || (suggestions() && !aiEnabled())" [aria-label]="noSuggestions ? 'No suggestions' : 'Suggest'" i18n-aria-label>
@if (loading()) {
<div class="spinner-border spinner-border-sm" role="status"></div>
} @else if (noSuggestions) {
<i-bs width="1.2em" height="1.2em" name="check-circle"></i-bs>
} @else {
<i-bs width="1.2em" height="1.2em" name="lightbulb"></i-bs>
}
@if (noSuggestions) {
<span class="d-none d-lg-inline ps-1" i18n>No suggestions</span>
} @else {
<span class="d-none d-lg-inline ps-1" i18n>Suggest</span>
}
@if (totalSuggestions > 0) {
<span class="badge bg-primary ms-2">{{ totalSuggestions }}</span>
}
</button>
@if (aiEnabled()) {
<div class="btn-group" ngbDropdown #dropdown="ngbDropdown" [popperOptions]="popperOptions">
<button type="button" class="btn btn-sm btn-outline-primary" ngbDropdownToggle [disabled]="disabled() || loading() || !suggestions()" aria-expanded="false" aria-controls="suggestionsDropdown" aria-label="Suggestions dropdown">
<span class="visually-hidden" i18n>Show suggestions</span>
</button>
<div ngbDropdownMenu aria-labelledby="suggestionsDropdown" class="shadow suggestions-dropdown">
<div class="list-group list-group-flush small pb-0">
@if (novelSuggestions === 0 && reusableSuggestions === 0) {
<div class="list-group-item text-muted fst-italic">
<small class="text-muted small fst-italic" i18n>No novel suggestions</small>
</div>
}
@if (suggestions()?.suggested_tags?.length > 0) {
<small class="list-group-item text-uppercase text-muted small"><i-bs class="me-2" name="tags"></i-bs><ng-container i18n>Tags</ng-container></small>
@for (tag of suggestions().suggested_tags; track tag) {
<button type="button" class="list-group-item list-group-item-action bg-light" (click)="addTag.emit(tag)">{{ tag }}</button>
}
}
@if (suggestions()?.suggested_document_types?.length > 0) {
<div class="list-group-item text-uppercase text-muted small"><i-bs class="me-2" name="hash"></i-bs><ng-container i18n>Document Types</ng-container></div>
@for (type of suggestions().suggested_document_types; track type) {
<button type="button" class="list-group-item list-group-item-action bg-light" (click)="addDocumentType.emit(type)">{{ type }}</button>
}
}
@if (suggestions()?.suggested_correspondents?.length > 0) {
<div class="list-group-item text-uppercase text-muted small"><i-bs class="me-2" name="person"></i-bs><ng-container i18n>Correspondents</ng-container></div>
@for (correspondent of suggestions().suggested_correspondents; track correspondent) {
<button type="button" class="list-group-item list-group-item-action bg-light" (click)="addCorrespondent.emit(correspondent)">{{ correspondent }}</button>
}
}
@if (reusableSuggestions > 0) {
<div class="list-group-item text-muted fst-italic">
<small class="text-muted small fst-italic" i18n>{reusableSuggestions, plural, =1 {1 existing value suggested below} other {{{reusableSuggestions}} existing values suggested below}}</small>
</div>
}
</div>
</div>
</div>
}
@if (noSuggestions) {
<span class="d-none d-lg-inline ps-1" i18n>No suggestions</span>
} @else {
<span class="d-none d-lg-inline ps-1" i18n>Suggest</span>
}
@if (totalSuggestions > 0) {
<span class="badge bg-primary ms-2">{{ totalSuggestions }}</span>
}
</button>
</div>
@if (aiEnabled()) {
<div class="btn-group" ngbDropdown #dropdown="ngbDropdown" [popperOptions]="popperOptions">
<button type="button" class="btn btn-sm btn-outline-primary" ngbDropdownToggle [disabled]="disabled() || loading() || !suggestions()" aria-expanded="false" aria-controls="suggestionsDropdown" aria-label="Suggestions dropdown">
<span class="visually-hidden" i18n>Show suggestions</span>
<div ngbDropdown autoClose="outside" placement="bottom-end" [popperOptions]="popperOptions">
<button type="button" class="btn btn-sm btn-link position-relative" ngbDropdownToggle [disabled]="disabled() || loading()" i18n-title title="Suggestion options">
<i-bs name="three-dots"></i-bs>
@if (source() !== defaultSource()) {
<span class="position-absolute top-0 start-100 translate-middle p-1 bg-primary border border-light rounded-circle">
<span class="visually-hidden" i18n>Not using default</span>
</span>
}
</button>
<div ngbDropdownMenu aria-labelledby="suggestionsDropdown" class="shadow suggestions-dropdown">
<div class="list-group list-group-flush small pb-0">
@if (novelSuggestions === 0 && reusableSuggestions === 0) {
<div class="list-group-item text-muted fst-italic">
<small class="text-muted small fst-italic" i18n>No novel suggestions</small>
</div>
}
@if (suggestions()?.suggested_tags?.length > 0) {
<small class="list-group-item text-uppercase text-muted small"><i-bs class="me-2" name="tags"></i-bs><ng-container i18n>Tags</ng-container></small>
@for (tag of suggestions().suggested_tags; track tag) {
<button type="button" class="list-group-item list-group-item-action bg-light" (click)="addTag.emit(tag)">{{ tag }}</button>
}
}
@if (suggestions()?.suggested_document_types?.length > 0) {
<div class="list-group-item text-uppercase text-muted small"><i-bs class="me-2" name="hash"></i-bs><ng-container i18n>Document Types</ng-container></div>
@for (type of suggestions().suggested_document_types; track type) {
<button type="button" class="list-group-item list-group-item-action bg-light" (click)="addDocumentType.emit(type)">{{ type }}</button>
}
}
@if (suggestions()?.suggested_correspondents?.length > 0) {
<div class="list-group-item text-uppercase text-muted small"><i-bs class="me-2" name="person"></i-bs><ng-container i18n>Correspondents</ng-container></div>
@for (correspondent of suggestions().suggested_correspondents; track correspondent) {
<button type="button" class="list-group-item list-group-item-action bg-light" (click)="addCorrespondent.emit(correspondent)">{{ correspondent }}</button>
}
}
@if (reusableSuggestions > 0) {
<div class="list-group-item text-muted fst-italic">
<small class="text-muted small fst-italic" i18n>{reusableSuggestions, plural, =1 {1 existing value suggested below} other {{{reusableSuggestions}} existing values suggested below}}</small>
</div>
}
<div ngbDropdownMenu class="shadow p-3">
<div class="small text-muted mb-2" i18n>Suggest using:</div>
<div class="form-check small">
<input class="form-check-input" type="checkbox" id="suggestionSourceML" [checked]="useML" [disabled]="useML && !useAI" (change)="setSources($event.target.checked, useAI)">
<label class="form-check-label d-inline-flex align-items-center gap-1" for="suggestionSourceML"><i-bs name="cpu"></i-bs><ng-container i18n>ML</ng-container></label>
</div>
<div class="form-check small">
<input class="form-check-input" type="checkbox" id="suggestionSourceAI" [checked]="useAI" [disabled]="useAI && !useML" (change)="setSources(useML, $event.target.checked)">
<label class="form-check-label d-inline-flex align-items-center gap-1" for="suggestionSourceAI"><i-bs name="stars"></i-bs><ng-container i18n>AI</ng-container></label>
</div>
</div>
</div>
@@ -1,3 +1,7 @@
.suggestions-dropdown {
min-width: 250px;
}
.btn-link.dropdown-toggle::after {
display: none;
}
@@ -1,6 +1,7 @@
import { ComponentFixture, TestBed } from '@angular/core/testing'
import { NgbDropdownModule } from '@ng-bootstrap/ng-bootstrap'
import { NgxBootstrapIconsModule, allIcons } from 'ngx-bootstrap-icons'
import { SuggestionSource } from 'src/app/data/ui-settings'
import { SuggestionsDropdownComponent } from './suggestions-dropdown.component'
describe('SuggestionsDropdownComponent', () => {
@@ -179,14 +180,71 @@ describe('SuggestionsDropdownComponent', () => {
it('should toggle dropdown when clickSuggest is called and suggestions are not null', () => {
fixture.componentRef.setInput('aiEnabled', true)
fixture.componentRef.setInput('fetchedSources', [SuggestionSource.ML])
fixture.detectChanges()
fixture.componentRef.setInput('suggestions', {
suggested_correspondents: [],
suggested_tags: [],
suggested_document_types: [],
})
fixture.detectChanges()
component.clickSuggest()
expect(component.dropdown.open).toBeTruthy()
expect(component.dropdown.isOpen()).toBeTruthy()
expect(fixture.nativeElement.textContent).toContain('No novel suggestions')
})
it('should fetch unfetched sources and show existing suggestions', () => {
jest.spyOn(component.getSuggestions, 'emit')
fixture.componentRef.setInput('aiEnabled', true)
fixture.componentRef.setInput('source', SuggestionSource.Both)
fixture.componentRef.setInput('fetchedSources', [SuggestionSource.ML])
fixture.componentRef.setInput('suggestions', { tags: [1] })
fixture.detectChanges()
component.clickSuggest()
expect(component.getSuggestions.emit).toHaveBeenCalledWith(
SuggestionSource.Both
)
expect(component.dropdown.isOpen()).toBeTruthy()
})
it('should only show source options when AI is enabled', () => {
expect(
fixture.nativeElement.querySelector('#suggestionSourceML')
).toBeNull()
fixture.componentRef.setInput('aiEnabled', true)
fixture.detectChanges()
fixture.nativeElement
.querySelector('button[title="Suggestion options"]')
.click()
fixture.detectChanges()
expect(
fixture.nativeElement.querySelector('#suggestionSourceML')
).not.toBeNull()
})
it('should emit source changes and never allow no source', () => {
const emitSpy = jest.spyOn(component.sourceChange, 'emit')
component.setSources(true, true)
expect(emitSpy).toHaveBeenCalledWith(SuggestionSource.Both)
component.setSources(true, false)
expect(emitSpy).toHaveBeenCalledWith(SuggestionSource.ML)
component.setSources(false, true)
expect(emitSpy).toHaveBeenCalledWith(SuggestionSource.AI)
emitSpy.mockClear()
component.setSources(false, false)
expect(emitSpy).not.toHaveBeenCalled()
})
it('should indicate a non-default source', () => {
fixture.componentRef.setInput('aiEnabled', true)
fixture.componentRef.setInput('source', SuggestionSource.AI)
fixture.componentRef.setInput('defaultSource', SuggestionSource.AI)
fixture.detectChanges()
expect(fixture.nativeElement.textContent).not.toContain('Not using default')
fixture.componentRef.setInput('source', SuggestionSource.Both)
fixture.detectChanges()
expect(fixture.nativeElement.textContent).toContain('Not using default')
})
})
@@ -8,6 +8,7 @@ import {
import { NgbDropdown, NgbDropdownModule } from '@ng-bootstrap/ng-bootstrap'
import { NgxBootstrapIconsModule } from 'ngx-bootstrap-icons'
import { DocumentSuggestions } from 'src/app/data/document-suggestions'
import { SuggestionSource } from 'src/app/data/ui-settings'
import { pngxPopperOptions } from 'src/app/utils/popper-options'
@Component({
@@ -18,12 +19,16 @@ import { pngxPopperOptions } from 'src/app/utils/popper-options'
})
export class SuggestionsDropdownComponent {
public popperOptions = pngxPopperOptions
public readonly SuggestionSource = SuggestionSource
@ViewChild('dropdown') dropdown: NgbDropdown
readonly suggestions = input<DocumentSuggestions>(null)
readonly aiEnabled = input(false)
readonly loading = input(false)
readonly disabled = input(false)
readonly source = input<SuggestionSource>(SuggestionSource.ML)
readonly defaultSource = input<SuggestionSource>(SuggestionSource.ML)
readonly fetchedSources = input<SuggestionSource[]>([])
readonly appliedTags = input<number[]>([])
readonly appliedCorrespondent = input<number>(null)
@@ -31,8 +36,10 @@ export class SuggestionsDropdownComponent {
readonly appliedStoragePath = input<number>(null)
@Output()
getSuggestions: EventEmitter<SuggestionsDropdownComponent> =
new EventEmitter()
getSuggestions: EventEmitter<SuggestionSource> = new EventEmitter()
@Output()
sourceChange: EventEmitter<SuggestionSource> = new EventEmitter()
@Output()
addTag: EventEmitter<string> = new EventEmitter()
@@ -53,12 +60,42 @@ export class SuggestionsDropdownComponent {
}
if (!this.suggestions()) {
this.getSuggestions.emit(this)
this.getSuggestions.emit(this.source())
} else if (this.hasUnfetchedSources) {
// sources changed, fetch the rest and show what we have meanwhile
this.getSuggestions.emit(this.source())
this.dropdown?.open()
} else {
this.dropdown?.toggle()
}
}
get useML(): boolean {
return this.source() !== SuggestionSource.AI
}
get useAI(): boolean {
return this.source() !== SuggestionSource.ML
}
get hasUnfetchedSources(): boolean {
const fetched = this.fetchedSources()
return (
(this.useML && !fetched.includes(SuggestionSource.ML)) ||
(this.useAI && !fetched.includes(SuggestionSource.AI))
)
}
public setSources(ml: boolean, ai: boolean) {
if (ml && ai) {
this.sourceChange.emit(SuggestionSource.Both)
} else if (ml) {
this.sourceChange.emit(SuggestionSource.ML)
} else if (ai) {
this.sourceChange.emit(SuggestionSource.AI)
}
}
get novelSuggestions(): number {
return (
(this.suggestions()?.suggested_correspondents?.length ?? 0) +
@@ -134,11 +134,15 @@
[loading]="suggestionsLoading()"
[suggestions]="suggestions()"
[aiEnabled]="aiEnabled"
[source]="suggestionSource"
[defaultSource]="defaultSuggestionSource"
[fetchedSources]="fetchedSuggestionSources()"
[appliedTags]="documentForm.value.tags"
[appliedCorrespondent]="documentForm.value.correspondent"
[appliedDocumentType]="documentForm.value.document_type"
[appliedStoragePath]="documentForm.value.storage_path"
(getSuggestions)="getSuggestions()"
(getSuggestions)="getSuggestions($event)"
(sourceChange)="suggestionSourceOverride.set($event)"
(addTag)="createTag($event)"
(addDocumentType)="createDocumentType($event)"
(addCorrespondent)="createCorrespondent($event)">
@@ -43,7 +43,7 @@ import {
} from 'src/app/data/filter-rule-type'
import { StoragePath } from 'src/app/data/storage-path'
import { Tag } from 'src/app/data/tag'
import { SETTINGS_KEYS } from 'src/app/data/ui-settings'
import { SETTINGS_KEYS, SuggestionSource } from 'src/app/data/ui-settings'
import { PermissionsGuard } from 'src/app/guards/permissions.guard'
import { CustomDatePipe } from 'src/app/pipes/custom-date.pipe'
import { DocumentTitlePipe } from 'src/app/pipes/document-title.pipe'
@@ -1528,6 +1528,113 @@ describe('DocumentDetailComponent', () => {
expect(component.suggestionsLoading()).toBeFalsy()
})
it('should get and merge ML and AI suggestions when source is both', () => {
settingsService.set(
SETTINGS_KEYS.DOCUMENT_EDITING_SUGGESTION_SOURCE,
SuggestionSource.Both
)
const getSetting = settingsService.get.bind(settingsService)
jest
.spyOn(settingsService, 'get')
.mockImplementation((key) =>
key === SETTINGS_KEYS.AI_ENABLED ? true : getSetting(key)
)
const suggestionsSpy = jest
.spyOn(documentService, 'getSuggestions')
.mockReturnValue(of({ tags: [42], dates: ['2024-01-01'] }))
const aiSuggestionsSpy = jest
.spyOn(documentService, 'getAiSuggestions')
.mockReturnValue(
of({ title: 'AI title', tags: [42, 43], suggested_tags: ['New'] })
)
initNormally()
expect(suggestionsSpy).toHaveBeenCalled()
expect(aiSuggestionsSpy).toHaveBeenCalled()
expect(component.suggestions().title).toEqual('AI title')
expect(component.suggestions().tags).toEqual([42, 43])
expect(component.suggestions().suggested_tags).toEqual(['New'])
expect(component.suggestions().dates).toEqual(['2024-01-01'])
})
it('should only fetch sources not yet fetched for the document', () => {
settingsService.set(SETTINGS_KEYS.DOCUMENT_EDITING_AUTO_SUGGEST, false)
settingsService.set(
SETTINGS_KEYS.DOCUMENT_EDITING_SUGGESTION_SOURCE,
SuggestionSource.ML
)
const getSetting = settingsService.get.bind(settingsService)
jest
.spyOn(settingsService, 'get')
.mockImplementation((key) =>
key === SETTINGS_KEYS.AI_ENABLED ? true : getSetting(key)
)
const suggestionsSpy = jest
.spyOn(documentService, 'getSuggestions')
.mockReturnValue(of({ tags: [42] }))
const aiSuggestionsSpy = jest
.spyOn(documentService, 'getAiSuggestions')
.mockReturnValue(of({ tags: [43] }))
initNormally()
component.getSuggestions()
expect(suggestionsSpy).toHaveBeenCalledTimes(1)
expect(aiSuggestionsSpy).not.toHaveBeenCalled()
component.getSuggestions(SuggestionSource.Both)
expect(suggestionsSpy).toHaveBeenCalledTimes(1)
expect(aiSuggestionsSpy).toHaveBeenCalledTimes(1)
expect(component.suggestions().tags).toEqual([42, 43])
component.getSuggestions(SuggestionSource.Both)
expect(suggestionsSpy).toHaveBeenCalledTimes(1)
expect(aiSuggestionsSpy).toHaveBeenCalledTimes(1)
})
it('should use the per-document source override and reset it on document change', () => {
settingsService.set(SETTINGS_KEYS.DOCUMENT_EDITING_AUTO_SUGGEST, false)
const getSetting = settingsService.get.bind(settingsService)
jest
.spyOn(settingsService, 'get')
.mockImplementation((key) =>
key === SETTINGS_KEYS.AI_ENABLED ? true : getSetting(key)
)
initNormally()
expect(component.suggestionSource).toEqual(SuggestionSource.AI)
component.suggestionSourceOverride.set(SuggestionSource.ML)
expect(component.suggestionSource).toEqual(SuggestionSource.ML)
jest
.spyOn(documentService, 'get')
.mockReturnValueOnce(of(Object.assign({}, doc)))
;(component as any).loadDocument(doc.id, true)
expect(component.suggestionSourceOverride()).toBeNull()
expect(component.fetchedSuggestionSources()).toEqual([])
})
it('should keep suggestions from one source if the other fails', () => {
settingsService.set(
SETTINGS_KEYS.DOCUMENT_EDITING_SUGGESTION_SOURCE,
SuggestionSource.Both
)
const getSetting = settingsService.get.bind(settingsService)
jest
.spyOn(settingsService, 'get')
.mockImplementation((key) =>
key === SETTINGS_KEYS.AI_ENABLED ? true : getSetting(key)
)
const errorSpy = jest.spyOn(toastService, 'showError')
jest
.spyOn(documentService, 'getSuggestions')
.mockReturnValue(of({ tags: [42] }))
jest
.spyOn(documentService, 'getAiSuggestions')
.mockReturnValue(throwError(() => new Error('failed')))
initNormally()
expect(errorSpy).toHaveBeenCalled()
expect(component.suggestions().tags).toEqual([42])
expect(component.fetchedSuggestionSources()).toEqual([SuggestionSource.ML])
})
it('should show error if needed for get suggestions', () => {
const suggestionsSpy = jest.spyOn(documentService, 'getSuggestions')
const errorSpy = jest.spyOn(toastService, 'showError')
@@ -28,7 +28,7 @@ import {
import { dirtyCheck, DirtyComponent } from '@ngneat/dirty-check-forms'
import { NgxBootstrapIconsModule } from 'ngx-bootstrap-icons'
import { DeviceDetectorService } from 'ngx-device-detector'
import { BehaviorSubject, Observable, of, Subject, timer } from 'rxjs'
import { BehaviorSubject, merge, Observable, of, Subject, timer } from 'rxjs'
import {
catchError,
debounceTime,
@@ -48,7 +48,10 @@ import { DataType } from 'src/app/data/datatype'
import { Document, DocumentVersionInfo } from 'src/app/data/document'
import { DocumentMetadata } from 'src/app/data/document-metadata'
import { DocumentNote } from 'src/app/data/document-note'
import { DocumentSuggestions } from 'src/app/data/document-suggestions'
import {
DocumentSuggestions,
mergeSuggestions,
} from 'src/app/data/document-suggestions'
import { DocumentType } from 'src/app/data/document-type'
import { FilterRule } from 'src/app/data/filter-rule'
import {
@@ -63,7 +66,7 @@ import {
import { ObjectWithId } from 'src/app/data/object-with-id'
import { StoragePath } from 'src/app/data/storage-path'
import { Tag } from 'src/app/data/tag'
import { SETTINGS_KEYS } from 'src/app/data/ui-settings'
import { SETTINGS_KEYS, SuggestionSource } from 'src/app/data/ui-settings'
import { User } from 'src/app/data/user'
import { IfPermissionsDirective } from 'src/app/directives/if-permissions.directive'
import { CustomDatePipe } from 'src/app/pipes/custom-date.pipe'
@@ -240,6 +243,10 @@ export class DocumentDetailComponent
private readonly autoSuggestSetting = this.settings.getSignal<boolean>(
SETTINGS_KEYS.DOCUMENT_EDITING_AUTO_SUGGEST
)
private readonly suggestionSourceSetting =
this.settings.getSignal<SuggestionSource>(
SETTINGS_KEYS.DOCUMENT_EDITING_SUGGESTION_SOURCE
)
private readonly hiddenFieldsSetting = this.settings.getSignal<
DocumentDetailFieldID[]
>(SETTINGS_KEYS.DOCUMENT_DETAILS_HIDDEN_FIELDS)
@@ -261,6 +268,9 @@ export class DocumentDetailComponent
readonly metadata = signal<DocumentMetadata>(undefined)
readonly suggestions = signal<DocumentSuggestions>(undefined)
readonly suggestionsLoading = signal(false)
// per-document, resets on navigation
readonly suggestionSourceOverride = signal<SuggestionSource>(null)
readonly fetchedSuggestionSources = signal<SuggestionSource[]>([])
readonly users = signal<User[]>(undefined)
readonly title = signal<string>(undefined)
@@ -365,6 +375,15 @@ export class DocumentDetailComponent
return this.autoSuggestSetting()
}
get defaultSuggestionSource(): SuggestionSource {
return this.aiEnabled ? this.suggestionSourceSetting() : SuggestionSource.ML
}
get suggestionSource(): SuggestionSource {
if (!this.aiEnabled) return SuggestionSource.ML
return this.suggestionSourceOverride() ?? this.defaultSuggestionSource
}
get archiveContentRenderType(): ContentRenderType {
const hasArchiveVersion =
this.metadata()?.has_archive_version ??
@@ -590,6 +609,8 @@ export class DocumentDetailComponent
}
this.documentId.set(doc.id)
this.suggestions.set(null)
this.suggestionSourceOverride.set(null)
this.fetchedSuggestionSources.set([])
const openDocument = this.openDocumentService.getOpenDocument(
this.documentId()
)
@@ -1077,29 +1098,44 @@ export class DocumentDetailComponent
return this.documentForm.get('custom_fields') as FormArray
}
getSuggestions() {
getSuggestions(source: SuggestionSource = this.suggestionSource) {
const sources = (
source === SuggestionSource.Both
? [SuggestionSource.ML, SuggestionSource.AI]
: [source]
).filter((s) => !this.fetchedSuggestionSources().includes(s))
if (!sources.length) return
this.suggestionsLoading.set(true)
const suggestionsObservable = this.aiEnabled
? this.documentsService.getAiSuggestions(this.documentId())
: this.documentsService.getSuggestions(this.documentId())
suggestionsObservable
merge(
...sources.map((s) =>
(s === SuggestionSource.AI
? this.documentsService.getAiSuggestions(this.documentId())
: this.documentsService.getSuggestions(this.documentId())
).pipe(
first(),
map((result) => ({ source: s, result })),
catchError((error) => {
this.toastService.showError(
$localize`Error retrieving suggestions.`,
error
)
return of(null)
})
)
)
)
.pipe(
first(),
takeUntil(this.unsubscribeNotifier),
takeUntil(this.docChangeNotifier),
finalize(() => this.suggestionsLoading.set(false))
)
.subscribe({
next: (result) => {
this.suggestions.set(result)
},
error: (error) => {
this.suggestions.set(null)
this.toastService.showError(
$localize`Error retrieving suggestions.`,
error
)
},
.subscribe((response) => {
if (!response) return
this.fetchedSuggestionSources.update((f) => [...f, response.source])
this.suggestions.set(
mergeSuggestions(this.suggestions(), response.result)
)
})
}
@@ -146,6 +146,19 @@ describe('DocumentListComponent', () => {
expect(reloadSpy).toHaveBeenCalled()
})
it('should stop reloading on document deleted after destroy', () => {
const reloadSpy = jest.spyOn(documentListService, 'reload')
const documentDeletedSubject = new Subject<boolean>()
jest
.spyOn(websocketStatusService, 'onDocumentDeleted')
.mockReturnValue(documentDeletedSubject)
fixture.detectChanges()
fixture.destroy()
reloadSpy.mockClear()
documentDeletedSubject.next(true)
expect(reloadSpy).not.toHaveBeenCalled()
})
it('should show score sort fields on fulltext queries', () => {
documentListService.setFilterRules([
{
@@ -270,9 +270,12 @@ export class DocumentListComponent
this.list.reload()
})
this.websocketStatusService.onDocumentDeleted().subscribe(() => {
this.list.reload()
})
this.websocketStatusService
.onDocumentDeleted()
.pipe(takeUntil(this.unsubscribeNotifier))
.subscribe(() => {
this.list.reload()
})
this.route.paramMap
.pipe(
@@ -15,3 +15,33 @@ export interface DocumentSuggestions {
dates?: string[] // ISO-formatted date string e.g. 2022-11-03
}
const union = <T>(a: T[] = [], b: T[] = []): T[] => [...new Set([...a, ...b])]
export function mergeSuggestions(
a: DocumentSuggestions,
b: DocumentSuggestions
): DocumentSuggestions {
if (!a) return b
return {
title: a.title || b.title,
tags: union(a.tags, b.tags),
suggested_tags: union(a.suggested_tags, b.suggested_tags),
correspondents: union(a.correspondents, b.correspondents),
suggested_correspondents: union(
a.suggested_correspondents,
b.suggested_correspondents
),
document_types: union(a.document_types, b.document_types),
suggested_document_types: union(
a.suggested_document_types,
b.suggested_document_types
),
storage_paths: union(a.storage_paths, b.storage_paths),
suggested_storage_paths: union(
a.suggested_storage_paths,
b.suggested_storage_paths
),
dates: union(a.dates, b.dates),
}
}
+9
View File
@@ -353,6 +353,14 @@ export const PaperlessConfigOptions: ConfigOption[] = [
config_key: 'PAPERLESS_AI_LLM_EMBEDDING_MODEL',
category: ConfigCategory.AI,
},
{
key: 'llm_embedding_api_key',
title: $localize`LLM Embedding API Key`,
type: ConfigOptionType.Password,
note: $localize`Used for embeddings when set, otherwise LLM API key is used.`,
config_key: 'PAPERLESS_AI_LLM_EMBEDDING_API_KEY',
category: ConfigCategory.AI,
},
{
key: 'llm_embedding_endpoint',
title: $localize`LLM Embedding Endpoint`,
@@ -457,6 +465,7 @@ export interface PaperlessConfig extends ObjectWithId {
ai_enabled: boolean
llm_embedding_backend: string
llm_embedding_model: string
llm_embedding_api_key: string
llm_embedding_endpoint: string
llm_embedding_chunk_size: number
llm_context_size: number
+13
View File
@@ -20,6 +20,12 @@ export enum GlobalSearchType {
TITLE_CONTENT = 'title-content',
}
export enum SuggestionSource {
ML = 'ml',
AI = 'ai',
Both = 'both',
}
export enum CollapsibleSection {
ATTRIBUTES = 'attributes',
}
@@ -98,6 +104,8 @@ export const SETTINGS_KEYS = {
'general-settings:document-editing:overlay-thumbnail',
DOCUMENT_EDITING_AUTO_SUGGEST:
'general-settings:document-editing:auto-suggest',
DOCUMENT_EDITING_SUGGESTION_SOURCE:
'general-settings:document-editing:suggestion-source',
DOCUMENT_DETAILS_HIDDEN_FIELDS:
'general-settings:document-details:hidden-fields',
SEARCH_DB_ONLY: 'general-settings:search:db-only',
@@ -326,6 +334,11 @@ export const SETTINGS: UiSetting[] = [
type: 'boolean',
default: true,
},
{
key: SETTINGS_KEYS.DOCUMENT_EDITING_SUGGESTION_SOURCE,
type: 'string',
default: SuggestionSource.AI,
},
{
key: SETTINGS_KEYS.DOCUMENT_DETAILS_HIDDEN_FIELDS,
type: 'array',
+4
View File
@@ -74,6 +74,7 @@ import {
clipboardCheckFill,
clipboardFill,
clockHistory,
cpu,
creditCard,
dash,
dashCircle,
@@ -118,6 +119,7 @@ import {
infoCircle,
journalBookmarkFill,
journals,
lightbulb,
link,
list,
listNested,
@@ -322,6 +324,7 @@ const icons = {
clipboardCheckFill,
clipboardFill,
clockHistory,
cpu,
cash,
creditCard,
dash,
@@ -367,6 +370,7 @@ const icons = {
infoCircle,
journalBookmarkFill,
journals,
lightbulb,
link,
list,
listNested,
+1
View File
@@ -292,6 +292,7 @@ a.btn-link:active,
a.btn-link:focus-visible,
.btn-link:hover,
.btn-link:active,
.btn-link.show,
.btn-link:focus-visible {
color: var(--pngx-primary-lighten-10) !important;
.primary-light & {
-42
View File
@@ -17,14 +17,10 @@ if TYPE_CHECKING:
from django.contrib.auth.models import User
from pytest_django.fixtures import Settings
from pytest_mock import MockerFixture
from rest_framework.test import APIClient
from paperless_testing.dirs import PaperlessDirs
from paperless_testing.fakes.progress import FakeProgressManager
from paperless_testing.outbound import DialRecorder
from paperless_testing.outbound import FakeDNS
from paperless_testing.outbound import LocalHTTPServer
@pytest.fixture(scope="session", autouse=True)
@@ -153,41 +149,3 @@ def fake_progress_manager(
monkeypatch.setattr("documents.tasks.ProgressManager", FakeProgressManager)
return FakeProgressManager
@pytest.fixture
def local_http_server() -> Generator[LocalHTTPServer, None, None]:
"""A recording HTTP server on 127.0.0.1, for outbound connection tests."""
from paperless_testing.outbound import running_http_server
with running_http_server() as server:
yield server
@pytest.fixture
def fake_dns(mocker: MockerFixture) -> FakeDNS:
"""Per-hostname answers for the outbound guard's resolver hooks."""
from paperless_testing.outbound import install_fake_dns
return install_fake_dns(mocker)
@pytest.fixture
def dial_recorder(mocker: MockerFixture) -> DialRecorder:
"""Records which addresses the outbound guard actually dialled."""
from paperless_testing.outbound import install_dial_recorder
return install_dial_recorder(mocker)
@pytest.fixture
def every_address_is_public(mocker: MockerFixture) -> None:
"""Disable the outbound guard's address policy: every address passes.
For tests that are not themselves exercising which addresses the guard
accepts, so loopback and other private addresses dial just like a
public one.
"""
from paperless_testing.outbound import allow_all_addresses
allow_all_addresses(mocker)
-1
View File
@@ -26,7 +26,6 @@ class DocumentsConfig(AppConfig):
document_consumption_finished.connect(set_document_type)
document_consumption_finished.connect(set_tags)
document_consumption_finished.connect(set_storage_path)
document_consumption_finished.connect(add_to_index)
document_consumption_finished.connect(run_workflows_added)
document_consumption_finished.connect(add_to_index)
document_consumption_finished.connect(add_or_update_document_in_llm_index)
+3 -2
View File
@@ -857,8 +857,9 @@ class ConsumerPlugin(
self.log.debug(f"Creation date from parse_date: {create_date}")
else:
stats = Path(self.input_doc.original_file).stat()
create_date = timezone.make_aware(
datetime.datetime.fromtimestamp(stats.st_mtime),
create_date = datetime.datetime.fromtimestamp(
stats.st_mtime,
tz=timezone.get_current_timezone(),
)
self.log.debug(f"Creation date from st_mtime: {create_date}")
+6 -1
View File
@@ -56,6 +56,7 @@ from documents.permissions import get_objects_for_user_owner_aware
from documents.plugins.helpers import DocumentsStatusManager
from documents.templating.utils import convert_format_str_to_template_format
from documents.utils import compute_checksum
from documents.utils import copy_file_with_basic_stats
from documents.workflows.actions import build_workflow_action_context
from documents.workflows.actions import execute_email_action
from documents.workflows.actions import execute_move_to_trash_action
@@ -363,7 +364,11 @@ def cleanup_document_deletion(sender, instance, **kwargs) -> None:
logger.debug(f"Moving {instance.source_path} to trash at {new_file_path}")
try:
shutil.move(instance.source_path, new_file_path)
shutil.move(
instance.source_path,
new_file_path,
copy_function=copy_file_with_basic_stats,
)
except OSError as e:
logger.error(
f"Failed to move {instance.source_path} to trash at "
@@ -81,6 +81,7 @@ class TestApiAppConfig(DirectoriesMixin, APITestCase):
"ai_enabled": None,
"llm_embedding_backend": None,
"llm_embedding_model": None,
"llm_embedding_api_key": None,
"llm_embedding_endpoint": None,
"llm_embedding_chunk_size": None,
"llm_context_size": None,
@@ -922,6 +923,49 @@ class TestApiAppConfig(DirectoriesMixin, APITestCase):
self.assertEqual(response.status_code, status.HTTP_405_METHOD_NOT_ALLOWED)
self.assertEqual(ApplicationConfiguration.objects.count(), 1)
def test_update_llm_embedding_api_key(self) -> None:
"""
GIVEN:
- Existing config with llm_embedding_api_key specified
WHEN:
- API to update llm_embedding_api_key is called with all *s
- API to update llm_embedding_api_key is called with empty string
THEN:
- llm_embedding_api_key is unchanged
- llm_embedding_api_key is set to None
"""
config = ApplicationConfiguration.objects.first()
assert config is not None
config.llm_embedding_api_key = "1234567890"
config.save()
# Test with all *
response = self.client.patch(
f"{self.ENDPOINT}1/",
json.dumps(
{
"llm_embedding_api_key": "*" * 32,
},
),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_200_OK)
config.refresh_from_db()
self.assertEqual(config.llm_embedding_api_key, "1234567890")
# Test with empty string
response = self.client.patch(
f"{self.ENDPOINT}1/",
json.dumps(
{
"llm_embedding_api_key": "",
},
),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_200_OK)
config.refresh_from_db()
self.assertEqual(config.llm_embedding_api_key, None)
def test_update_llm_api_key(self) -> None:
"""
GIVEN:
@@ -166,7 +166,15 @@ class TestBulkDownload(DirectoriesMixin, SampleDirMixin, APITestCase):
),
content_type="application/json",
)
response.close()
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertEqual(response["Content-Type"], "application/zip")
with zipfile.ZipFile(io.BytesIO(read_streaming_response(response))) as zipf:
self.assertEqual(zipf.infolist()[0].compress_type, zipfile.ZIP_LZMA)
with self.doc2.source_file as f:
self.assertEqual(f.read(), zipf.read("2021-01-01 document A.pdf"))
@override_settings(FILENAME_FORMAT="{correspondent}/{title}")
def test_formatted_download_originals(self) -> None:
+34 -9
View File
@@ -9,6 +9,7 @@ from rest_framework.test import APITestCase
from documents.models import Correspondent
from documents.models import CustomField
from documents.models import CustomFieldInstance
from documents.models import Document
from documents.models import DocumentType
from documents.models import StoragePath
@@ -2525,7 +2526,7 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
WHEN:
- API to bulk edit documents is called
THEN:
- Audit log is created
- Audit log is created with the old and new correspondent
"""
LogEntry.objects.all().delete()
response = self.client.post(
@@ -2541,7 +2542,8 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
)
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertEqual(LogEntry.objects.filter(object_pk=self.doc1.id).count(), 1)
entry = LogEntry.objects.get_for_object(self.doc1).get()
self.assertEqual(entry.changes, {"correspondent": [None, self.c2.id]})
@override_settings(AUDIT_LOG_ENABLED=True)
def test_bulk_edit_audit_log_enabled_tags(self) -> None:
@@ -2549,16 +2551,18 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
GIVEN:
- Audit log is enabled
WHEN:
- API to bulk edit tags is called
- API to bulk edit tags is called on an untagged document and a
document with several tags
THEN:
- Audit log is created
- Audit log is created for each document with its full tag list
before and after the edit
"""
LogEntry.objects.all().delete()
response = self.client.post(
"/api/documents/bulk_edit/",
json.dumps(
{
"documents": [self.doc1.id],
"documents": [self.doc1.id, self.doc4.id],
"method": "modify_tags",
"parameters": {
"add_tags": [self.t1.id],
@@ -2570,18 +2574,32 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
)
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertEqual(LogEntry.objects.filter(object_pk=self.doc1.id).count(), 1)
entry = LogEntry.objects.get_for_object(self.doc1).get()
self.assertEqual(entry.changes, {"tags": [[], [self.t1.id]]})
entry = LogEntry.objects.get_for_object(self.doc4).get()
self.assertEqual(
entry.changes,
{"tags": [[self.t1.id, self.t2.id], [self.t1.id]]},
)
@override_settings(AUDIT_LOG_ENABLED=True)
def test_bulk_edit_audit_log_enabled_custom_fields(self) -> None:
"""
GIVEN:
- Audit log is enabled
- A document with two custom fields
WHEN:
- API to bulk edit custom fields is called
- API to bulk edit custom fields is called to add a third
THEN:
- Audit log is created
- Audit log is created with every custom field instance before and
after the edit
- Audit log is created for the new custom field instance
"""
cf3 = CustomField.objects.create(name="cf3", data_type="string")
existing = [
CustomFieldInstance.objects.create(document=self.doc1, field=field)
for field in (self.cf2, cf3)
]
LogEntry.objects.all().delete()
response = self.client.post(
"/api/documents/bulk_edit/",
@@ -2599,7 +2617,14 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
)
self.assertEqual(response.status_code, status.HTTP_200_OK)
self.assertEqual(LogEntry.objects.filter(object_pk=self.doc1.id).count(), 2)
added = CustomFieldInstance.objects.get(document=self.doc1, field=self.cf1)
existing_ids = [instance.id for instance in existing]
entry = LogEntry.objects.get_for_object(self.doc1).get()
self.assertEqual(
entry.changes,
{"custom_fields": [existing_ids, [*existing_ids, added.id]]},
)
self.assertEqual(LogEntry.objects.get_for_object(added).count(), 1)
def test_api_bulk_edit_with_bad_search_query_returns_400(self) -> None:
"""
+11 -14
View File
@@ -64,16 +64,15 @@ class TestApiProfile(DirectoriesMixin, APITestCase):
)
self.client.force_authenticate(user=self.user)
def setupSocialAccount(self) -> None:
def setupSocialAccount(self) -> SocialAccount:
SocialApp.objects.create(
name="Keycloak",
provider="openid_connect",
provider_id="keycloak-test",
)
self.user.socialaccount_set.add(
SocialAccount(uid="123456789", provider="keycloak-test"),
bulk=False,
)
social_account = SocialAccount(uid="123456789", provider="keycloak-test")
self.user.socialaccount_set.add(social_account, bulk=False)
return social_account
def test_get_profile(self) -> None:
"""
@@ -111,19 +110,17 @@ class TestApiProfile(DirectoriesMixin, APITestCase):
THEN:
- Profile is returned with social accounts
"""
self.setupSocialAccount()
social_account = self.setupSocialAccount()
openid_provider = (
MockOpenIDConnectProvider(
app=SocialApp.objects.get(provider_id="keycloak-test"),
),
openid_provider = MockOpenIDConnectProvider(
app=SocialApp.objects.get(provider_id="keycloak-test"),
)
mock_list_providers.return_value = [
openid_provider,
]
mock_get_provider_account.return_value = MockOpenIDConnectProviderAccount(
mock_social_account_dict={
"name": openid_provider[0].name,
"name": openid_provider.name,
},
)
@@ -135,7 +132,7 @@ class TestApiProfile(DirectoriesMixin, APITestCase):
response.data["social_accounts"],
[
{
"id": 1,
"id": social_account.pk,
"provider": "keycloak-test",
"name": "Keycloak",
},
@@ -152,7 +149,7 @@ class TestApiProfile(DirectoriesMixin, APITestCase):
THEN:
- Profile is returned with "Unknown App" as name
"""
self.setupSocialAccount()
social_account = self.setupSocialAccount()
# Remove the social app
SocialApp.objects.get(provider_id="keycloak-test").delete()
@@ -165,7 +162,7 @@ class TestApiProfile(DirectoriesMixin, APITestCase):
response.data["social_accounts"],
[
{
"id": 1,
"id": social_account.pk,
"provider": "keycloak-test",
"name": "Unknown App",
},
@@ -677,12 +677,13 @@ class TestExportImport(
THEN:
- Error is raised
"""
args = ["document_exporter", "/tmp/foo/bar"]
with tempfile.TemporaryDirectory() as tmp_dir:
args = ["document_exporter", str(Path(tmp_dir) / "does-not-exist")]
with self.assertRaises(CommandError) as e:
call_command(*args, skip_checks=True)
with self.assertRaises(CommandError) as e:
call_command(*args, skip_checks=True)
self.assertEqual("That path doesn't exist", str(e.exception))
self.assertEqual("That path doesn't exist", str(e.exception))
def test_export_target_exists_but_is_file(self) -> None:
"""
+8 -8
View File
@@ -123,14 +123,14 @@ class TestFuzzyMatchCommand(TestCase):
- Output contains clickable links to the documents instead of titles
"""
# Content similarity is 86.667
Document.objects.create(
doc1 = Document.objects.create(
checksum="BEEFCAFE",
title="A",
content="first document scanned by bob",
mime_type="application/pdf",
filename="test.pdf",
)
Document.objects.create(
doc2 = Document.objects.create(
checksum="DEADBEAF",
title="A",
content="first document scanned by alice",
@@ -145,8 +145,8 @@ class TestFuzzyMatchCommand(TestCase):
"http://localhost:8000",
)
self.assertIn("Found 1 matching pair(s)", stdout)
self.assertIn("http://localhost:8000/documents/1/details", stdout)
self.assertIn("http://localhost:8000/documents/2/details", stdout)
self.assertIn(f"http://localhost:8000/documents/{doc1.pk}/details", stdout)
self.assertIn(f"http://localhost:8000/documents/{doc2.pk}/details", stdout)
def test_with_3_matches(self) -> None:
"""
@@ -198,14 +198,14 @@ class TestFuzzyMatchCommand(TestCase):
- Documents 1 and 2 remain
"""
# Content similarity is 86.667
Document.objects.create(
doc1 = Document.objects.create(
checksum="BEEFCAFE",
title="A",
content="first document scanned by bob",
mime_type="application/pdf",
filename="test.pdf",
)
Document.objects.create(
doc2 = Document.objects.create(
checksum="DEADBEAF",
title="A",
content="second document scanned by alice",
@@ -235,8 +235,8 @@ class TestFuzzyMatchCommand(TestCase):
self.assertIn("Deleting 1 document(s)", stdout)
self.assertEqual(Document.objects.count(), 2)
self.assertIsNotNone(Document.objects.get(pk=1))
self.assertIsNotNone(Document.objects.get(pk=2))
self.assertIsNotNone(Document.objects.get(pk=doc1.pk))
self.assertIsNotNone(Document.objects.get(pk=doc2.pk))
def test_document_deletion_cancelled(self) -> None:
"""
@@ -14,7 +14,7 @@ from paperless_testing.dirs import DirectoriesMixin
class TestManageSuperUser(DirectoriesMixin, TestCase):
def call_command(self, environ):
out = StringIO()
with mock.patch.dict(os.environ, environ):
with mock.patch.dict(os.environ, environ, clear=True):
call_command(
"manage_superuser",
"--no-color",
@@ -430,6 +430,53 @@ class TestBulkDownloadPermissionChecksRootDocument:
) # version-only grant must not substitute for root permission
@pytest.mark.django_db
class TestDocumentOperationPermissionChecksRootDocument:
@pytest.mark.parametrize(
("endpoint", "payload"),
[
pytest.param("/api/documents/merge/", {}, id="merge"),
pytest.param("/api/documents/rotate/", {"degrees": 90}, id="rotate"),
],
)
@pytest.mark.parametrize("version_owner", ["none", "requester"])
def test_version_operation_acts_on_root(
self,
rest_api_client: APIClient,
endpoint: str,
payload: dict,
version_owner: str,
) -> None:
owner = UserFactory(username="owner")
requester = UserFactory(username="requester")
grant_global(requester, "change_document")
grant_global(requester, "add_document")
rest_api_client.force_authenticate(user=requester)
root = DocumentFactory(owner=owner)
# A version whose owner went stale, e.g. created before the root changed hands
version = DocumentFactory(
owner=requester if version_owner == "requester" else None,
root_document=root,
version_index=1,
)
with (
patch("documents.views.bulk_edit.merge") as mock_merge,
patch("documents.views.bulk_edit.rotate") as mock_rotate,
):
mock_merge.__name__ = "merge"
mock_rotate.__name__ = "rotate"
response = rest_api_client.post(
endpoint,
{"documents": [version.pk], **payload},
format="json",
)
assert response.status_code == HTTPStatus.FORBIDDEN
mock_merge.assert_not_called()
mock_rotate.assert_not_called()
@pytest.mark.django_db
@pytest.mark.usefixtures("_search_index")
class TestTrashRestorePermissionBoundary:
@@ -339,15 +339,6 @@ class ShareLinkBundleBuildTaskTests(DirectoriesMixin, APITestCase):
)
self.document.archive_checksum = ""
self.document.save()
self.addCleanup(
setattr,
settings,
"SHARE_LINK_BUNDLE_DIR",
settings.SHARE_LINK_BUNDLE_DIR,
)
settings.SHARE_LINK_BUNDLE_DIR = (
Path(settings.MEDIA_ROOT) / "documents" / "share_link_bundles"
)
def _write_document_file(self, *, archive: bool, content: bytes) -> Path:
if archive:
-35
View File
@@ -18,7 +18,6 @@ from documents.models import WorkflowAction
from documents.sanity_checker import SanityCheckFailedException
from documents.sanity_checker import SanityCheckMessages
from documents.tests.helpers import dummy_preprocess
from paperless_ai.exceptions import LLMBlockedError
from paperless_testing.assertions import FileSystemAssertsMixin
from paperless_testing.dirs import DirectoriesMixin
@@ -556,37 +555,3 @@ class TestApplyAISuggestionsTask(DirectoriesMixin, TestCase):
apply_suggestions.assert_not_called()
self.assertIn("no longer exists", "".join(cm.output))
@override_settings(AI_ENABLED=True)
def test_blocked_request_fails_without_retry(self) -> None:
"""
GIVEN:
- AI enabled and a document with content
- The AI classification call blocked by the outbound request policy
WHEN:
- The task runs through Celery
THEN:
- The workflow code does not swallow the block
- The task fails with LLMBlockedError and is never retried
"""
with (
mock.patch(
"documents.workflows.ai.get_ai_document_classification",
side_effect=LLMBlockedError(
"AI backend request was blocked by the outbound request "
"policy: detail",
),
),
mock.patch.object(
tasks.apply_ai_suggestions,
"retry",
wraps=tasks.apply_ai_suggestions.retry,
) as retry,
):
result = tasks.apply_ai_suggestions.apply(
args=(self.action.pk, self.doc.pk),
)
self.assertTrue(result.failed())
self.assertIsInstance(result.result, LLMBlockedError)
retry.assert_not_called()
-43
View File
@@ -29,7 +29,6 @@ from documents.models import Tag
from documents.models import UiSettings
from documents.signals.handlers import update_llm_suggestions_cache
from paperless.models import ApplicationConfiguration
from paperless_ai.exceptions import LLMBlockedError
from paperless_ai.exceptions import LLMProviderError
from paperless_ai.exceptions import LLMTimeoutError
from paperless_testing.dirs import DirectoriesMixin
@@ -771,48 +770,6 @@ class TestAISuggestions(DirectoriesMixin, TestCase):
get_llm_suggestion_cache(self.document.pk, backend="openai-like"),
)
@patch("documents.views.get_ai_document_classification")
@override_settings(
AI_ENABLED=True,
LLM_BACKEND="openai-like",
)
def test_ai_suggestions_with_blocked_llm_request(
self,
mock_get_ai_classification,
) -> None:
"""
GIVEN:
- An AI backend request blocked by the outbound request policy
WHEN:
- AI suggestions are requested
THEN:
- 502 is returned with a generic message and nothing is cached
"""
mock_get_ai_classification.side_effect = LLMBlockedError(
"AI backend request was blocked by the outbound request policy: detail",
)
self.client.force_login(user=self.user)
response = self.client.get(
f"/api/documents/{self.document.pk}/ai_suggestions/",
)
self.assertEqual(response.status_code, status.HTTP_502_BAD_GATEWAY)
self.assertEqual(
response.json(),
{
"ai": [
(
"AI backend request was blocked by the outbound request "
"policy. Check logs for details."
),
],
},
)
self.assertIsNone(
get_llm_suggestion_cache(self.document.pk, backend="openai-like"),
)
@patch("documents.views.get_ai_document_classification")
@override_settings(
AI_ENABLED=True,
+141 -115
View File
@@ -1,7 +1,9 @@
import datetime
import json
import shutil
import socket
import tempfile
from collections.abc import Callable
from datetime import timedelta
from pathlib import Path
from typing import TYPE_CHECKING
@@ -17,11 +19,11 @@ from django.test import override_settings
from django.utils import timezone
from guardian.shortcuts import get_groups_with_perms
from guardian.shortcuts import get_users_with_perms
from httpx import ConnectError
from httpx import HTTPError
from httpx import HTTPStatusError
from pytest_django.fixtures import Settings
from pytest_httpx import HTTPXMock
from pytest_mock import MockerFixture
from rest_framework.test import APIClient
from rest_framework.test import APITestCase
@@ -31,12 +33,8 @@ from documents.file_handling import generate_unique_filename
from documents.signals.handlers import run_workflows
from documents.workflows.ai import apply_ai_suggestions_to_document
from documents.workflows.webhooks import send_webhook
from paperless.network import OutboundRequestBlockedError
from paperless_ai.base_model import ClassificationSuggestions
from paperless_ai.exceptions import LLMTimeoutError
from paperless_testing.outbound import DialRecorder
from paperless_testing.outbound import FakeDNS
from paperless_testing.outbound import LocalHTTPServer
if TYPE_CHECKING:
from django.db.models import QuerySet
@@ -71,7 +69,9 @@ from paperless_mail.models import MailAccount
from paperless_mail.models import MailRule
from paperless_testing.assertions import FileSystemAssertsMixin
from paperless_testing.dirs import DirectoriesMixin
from paperless_testing.factories import DocumentFactory
from paperless_testing.factories import UserFactory
from paperless_testing.permissions import grant_global
from paperless_testing.permissions import grant_object
@@ -1061,6 +1061,41 @@ class TestWorkflows(
self.assertEqual(doc.correspondent, self.c2)
self.assertEqual(doc.title, f"Doc created in {created.year}")
@pytest.mark.usefixtures("_search_index")
def test_document_added_workflow_indexes_final_title(self) -> None:
trigger = WorkflowTrigger.objects.create(
type=WorkflowTrigger.WorkflowTriggerType.DOCUMENT_ADDED,
filter_filename="*sample*",
)
action = WorkflowAction.objects.create(
assign_title="Linked document",
assign_owner=self.user2,
)
link_field = CustomField.objects.create(
name="Related documents",
data_type=CustomField.FieldDataType.DOCUMENTLINK,
)
action.assign_custom_fields.add(link_field)
workflow = Workflow.objects.create(name="Link workflow", order=0)
workflow.triggers.add(trigger)
workflow.actions.add(action)
doc = DocumentFactory.create()
document_consumption_finished.send(sender=self.__class__, document=doc)
self.assertTrue(doc.custom_fields.filter(field=link_field).exists())
doc.refresh_from_db()
self.assertEqual(doc.title, "Linked document")
grant_global(self.user2, "view_document")
self.client.force_authenticate(user=self.user2)
response = self.client.get("/api/documents/?title_search=linked")
self.assertEqual(response.status_code, 200)
self.assertEqual(
[result["id"] for result in response.data["results"]],
[doc.pk],
)
def test_document_added_no_match_filename(self) -> None:
trigger = WorkflowTrigger.objects.create(
type=WorkflowTrigger.WorkflowTriggerType.DOCUMENT_ADDED,
@@ -4424,18 +4459,18 @@ class TestWorkflows(
)
@mock.patch("documents.bulk_edit.remove_password")
def test_password_removal_action_fails_without_correct_password(
def test_password_removal_action_skips_blank_and_whitespace_passwords(
self,
mock_remove_password,
) -> None:
"""
GIVEN:
- Workflow password removal action
- No correct password provided
- Only blank and whitespace-only passwords configured
WHEN:
- Document updated triggering the workflow
THEN:
- Password removal is attempted for all passwords and fails
- Password removal is not attempted
"""
doc = Document.objects.create(
title="Protected",
@@ -4456,6 +4491,60 @@ class TestWorkflows(
mock_remove_password.assert_not_called()
@mock.patch("documents.bulk_edit.remove_password")
def test_password_removal_action_fails_without_correct_password(
self,
mock_remove_password,
) -> None:
"""
GIVEN:
- Workflow password removal action
- No configured password is correct
WHEN:
- Document updated triggering the workflow
THEN:
- Password removal is attempted for every configured password and fails
"""
doc = Document.objects.create(
title="Protected",
checksum="pw-checksum-3",
)
trigger = WorkflowTrigger.objects.create(
type=WorkflowTrigger.WorkflowTriggerType.DOCUMENT_UPDATED,
)
action = WorkflowAction.objects.create(
type=WorkflowAction.WorkflowActionType.PASSWORD_REMOVAL,
passwords=["wrong", "also-wrong"],
)
workflow = Workflow.objects.create(name="Password workflow wrong passwords")
workflow.triggers.add(trigger)
workflow.actions.add(action)
mock_remove_password.side_effect = ValueError("wrong password")
with self.assertLogs("paperless.workflows.actions", level="ERROR"):
run_workflows(trigger.type, doc)
assert mock_remove_password.call_count == 2
mock_remove_password.assert_has_calls(
[
mock.call(
[doc.id],
password="wrong",
update_document=True,
user=doc.owner,
source_paths_by_id=None,
),
mock.call(
[doc.id],
password="also-wrong",
update_document=True,
user=doc.owner,
source_paths_by_id=None,
),
],
)
@mock.patch("documents.bulk_edit.remove_password")
def test_password_removal_action_skips_without_passwords(
self,
@@ -5071,6 +5160,25 @@ class TestWebhookSend:
assert httpx_mock.get_request().headers["Content-Type"] == "application/json"
@pytest.fixture
def resolve_to(monkeypatch: pytest.MonkeyPatch) -> Callable[[str], None]:
"""
Force DNS resolution to a specific IP for any hostname.
"""
def _set(ip: str) -> None:
def fake_getaddrinfo(
host: str,
*_args: object,
**_kwargs: object,
) -> list[tuple[Any, ...]]:
return [(socket.AF_INET, None, None, "", (ip, 0))]
monkeypatch.setattr(socket, "getaddrinfo", fake_getaddrinfo)
return _set
class TestWebhookSecurity:
def test_blocks_invalid_scheme_or_hostname(self, httpx_mock: HTTPXMock) -> None:
"""
@@ -5120,145 +5228,60 @@ class TestWebhookSecurity:
assert httpx_mock.get_request() is None
@pytest.mark.parametrize(
"address",
[
pytest.param("127.0.0.1", id="loopback"),
pytest.param("10.0.0.1", id="private"),
pytest.param("169.254.169.254", id="link-local-metadata"),
pytest.param("::ffff:127.0.0.1", id="ipv4-mapped-loopback"),
pytest.param("64:ff9b::7f00:1", id="nat64-wrapping-loopback"),
],
)
@override_settings(WEBHOOKS_ALLOW_INTERNAL_REQUESTS=False)
def test_blocks_private_loopback_linklocal(
self,
local_http_server: LocalHTTPServer,
fake_dns: FakeDNS,
dial_recorder: DialRecorder,
address: str,
httpx_mock: HTTPXMock,
resolve_to,
) -> None:
"""
GIVEN:
- A webhook host resolving to a non-public address
- URL with a private, loopback, or link-local IP address
- WEBHOOKS_ALLOW_INTERNAL_REQUESTS is False
WHEN:
- send_webhook is called
- send_webhook is called with such URL
THEN:
- The request is blocked before any connection is opened
- ValueError is raised
"""
fake_dns.add("webhook.test", address)
with pytest.raises(OutboundRequestBlockedError):
resolve_to("127.0.0.1")
with pytest.raises(ConnectError):
send_webhook(
f"http://webhook.test:{local_http_server.port}",
"http://paperless-ngx.com",
data="",
headers={},
files=None,
as_json=False,
)
assert local_http_server.connections == 0
assert dial_recorder.hosts() == []
@override_settings(WEBHOOKS_ALLOW_INTERNAL_REQUESTS=False)
@pytest.mark.usefixtures("every_address_is_public")
def test_sends_to_validated_address(
def test_allows_public_ip_and_sends(
self,
local_http_server: LocalHTTPServer,
fake_dns: FakeDNS,
httpx_mock: HTTPXMock,
resolve_to,
) -> None:
"""
GIVEN:
- A webhook host resolving to an address the policy accepts
- WEBHOOKS_ALLOW_INTERNAL_REQUESTS is False
- URL with a public IP address
WHEN:
- send_webhook is called
- send_webhook is called with such URL
THEN:
- The payload arrives with the webhook hostname in the Host header
- Request is sent successfully
"""
fake_dns.add("webhook.test", "127.0.0.1")
resolve_to("52.207.186.75")
httpx_mock.add_response(content=b"ok")
send_webhook(
url=f"http://webhook.test:{local_http_server.port}",
url="http://paperless-ngx.com",
data="hi",
headers={},
files=None,
as_json=False,
)
received = local_http_server.requests[0]
assert received.body == b"hi"
assert received.headers["host"] == f"webhook.test:{local_http_server.port}"
req = httpx_mock.get_request()
assert req.url.host == "52.207.186.75"
assert req.headers["host"] == "paperless-ngx.com"
@override_settings(WEBHOOKS_ALLOW_INTERNAL_REQUESTS=True)
def test_allow_internal_sends_to_internal_address(
self,
local_http_server: LocalHTTPServer,
fake_dns: FakeDNS,
) -> None:
"""
GIVEN:
- A webhook to localhost
- WEBHOOKS_ALLOW_INTERNAL_REQUESTS is True
WHEN:
- send_webhook is called
THEN:
- The payload arrives at the internal address
- The guard does not resolve the host, leaving it to the stock
connection path
"""
send_webhook(
url=f"http://localhost:{local_http_server.port}",
data="hi",
headers={},
files=None,
as_json=False,
)
received = local_http_server.requests[0]
assert received.body == b"hi"
assert fake_dns.lookups == []
@override_settings(WEBHOOKS_ALLOW_INTERNAL_REQUESTS=False)
def test_block_is_an_expected_task_failure(
self,
mocker: MockerFixture,
local_http_server: LocalHTTPServer,
fake_dns: FakeDNS,
dial_recorder: DialRecorder,
) -> None:
"""
GIVEN:
- A webhook host resolving to a loopback address
- WEBHOOKS_ALLOW_INTERNAL_REQUESTS is False
WHEN:
- The webhook task runs through Celery
THEN:
- The task fails with the original block error, not a wrapper,
so it matches the task's expected errors, and is not retried
"""
fake_dns.add("webhook.test", "127.0.0.1")
retry = mocker.spy(send_webhook, "retry")
result = send_webhook.apply(
kwargs={
"url": f"http://webhook.test:{local_http_server.port}",
"data": "",
"headers": {},
"files": None,
"as_json": False,
},
)
assert result.failed()
assert isinstance(result.result, OutboundRequestBlockedError)
assert isinstance(result.result, send_webhook.throws)
retry.assert_not_called()
assert local_http_server.connections == 0
assert dial_recorder.hosts() == []
def test_follow_redirects_disabled(self, httpx_mock: HTTPXMock) -> None:
def test_follow_redirects_disabled(self, httpx_mock: HTTPXMock, resolve_to) -> None:
"""
GIVEN:
- A URL that redirects
@@ -5267,6 +5290,7 @@ class TestWebhookSecurity:
THEN:
- Request is made to the original URL and does not follow the redirect
"""
resolve_to("52.207.186.75")
# Return a redirect and ensure we don't follow it (only one request recorded)
httpx_mock.add_response(
status_code=302,
@@ -5288,6 +5312,7 @@ class TestWebhookSecurity:
def test_strips_user_supplied_host_header(
self,
httpx_mock: HTTPXMock,
resolve_to: Callable[[str], None],
) -> None:
"""
GIVEN:
@@ -5295,8 +5320,9 @@ class TestWebhookSecurity:
WHEN:
- send_webhook is called with a malicious Host header
THEN:
- The Host header is stripped and set from the URL hostname
- The Host header is stripped and replaced with the resolved hostname
"""
resolve_to("52.207.186.75")
httpx_mock.add_response(content=b"ok")
send_webhook(
+51 -52
View File
@@ -49,7 +49,6 @@ from django.db.models import Sum
from django.db.models import When
from django.db.models.functions import Coalesce
from django.db.models.functions import Lower
from django.db.models.manager import Manager
from django.http import FileResponse
from django.http import Http404
from django.http import HttpRequest
@@ -256,7 +255,6 @@ from paperless.views import StandardPagination
from paperless_ai.ai_classifier import get_ai_document_classification
from paperless_ai.ai_classifier import get_llm_output_language
from paperless_ai.chat import stream_chat_with_documents
from paperless_ai.exceptions import LLMBlockedError
from paperless_ai.exceptions import LLMProviderError
from paperless_ai.exceptions import LLMTimeoutError
from paperless_ai.matching import extract_unmatched_names
@@ -1698,23 +1696,6 @@ class DocumentViewSet(
},
status=status.HTTP_502_BAD_GATEWAY,
)
except LLMBlockedError as exc:
logger.warning(
"AI backend request for document %s was blocked: %s",
doc.pk,
exc,
)
return Response(
{
"ai": [
_(
"AI backend request was blocked by the outbound "
"request policy. Check logs for details.",
),
],
},
status=status.HTTP_502_BAD_GATEWAY,
)
set_llm_suggestions_cache(
doc.pk,
llm_suggestions,
@@ -2986,11 +2967,15 @@ class DocumentOperationPermissionMixin(PassUserMixin, DocumentSelectionMixin):
if user.is_superuser:
return True
document_objs = Document.objects.select_related("owner").filter(
pk__in=documents,
)
root_docs = {
get_root_document(doc)
for doc in Document.objects.select_related(
"owner",
"root_document__owner",
).filter(pk__in=documents)
}
user_is_owner_of_all_documents = all(
(doc.owner == user or doc.owner is None) for doc in document_objs
(doc.owner == user or doc.owner is None) for doc in root_docs
)
# check global and object permissions for all documents
@@ -2998,9 +2983,13 @@ class DocumentOperationPermissionMixin(PassUserMixin, DocumentSelectionMixin):
user.has_perm(
"documents.change_document",
)
and not document_objs.exclude(
and not Document.global_objects.filter(
pk__in=[doc.pk for doc in root_docs],
)
.exclude(
pk__in=permitted_document_ids(user, perm="change_document"),
).exists()
)
.exists()
)
# check ownership for methods that change original document
@@ -3159,6 +3148,38 @@ class BulkEditView(DocumentOperationPermissionMixin):
serializer_class = BulkEditSerializer
@staticmethod
def _snapshot_field(doc_ids: list[int], field: str) -> dict[int, Any]:
"""
Returns each document's current value of field, for the audit log.
Tags and custom fields are one row per value, so they are gathered
into a sorted list of pks per document (empty when there are none).
Reading them through Document.values() instead would join those rows
and return one arbitrary value per document.
"""
if field == "tags":
rows = (
Document.tags.through.objects.filter(document_id__in=doc_ids)
.order_by("tag_id")
.values_list("document_id", "tag_id")
)
elif field == "custom_fields":
rows = (
CustomFieldInstance.objects.filter(document_id__in=doc_ids)
.order_by("pk")
.values_list("document_id", "pk")
)
else:
return dict(
Document.objects.filter(pk__in=doc_ids).values_list("pk", field),
)
values: dict[int, list[int]] = {doc_id: [] for doc_id in doc_ids}
for doc_id, pk in rows:
values[doc_id].append(pk)
return values
def post(self, request, *args, **kwargs):
request_method = request.data.get("method")
api_version = int(request.version or settings.REST_FRAMEWORK["DEFAULT_VERSION"])
@@ -3205,41 +3226,19 @@ class BulkEditView(DocumentOperationPermissionMixin):
try:
modified_field = self.MODIFIED_FIELD_BY_METHOD.get(method.__name__, None)
if settings.AUDIT_LOG_ENABLED and modified_field:
old_documents = {
obj["pk"]: obj
for obj in Document.objects.filter(pk__in=documents).values(
"pk",
"correspondent",
"document_type",
"storage_path",
"tags",
"custom_fields",
"deleted_at",
"checksum",
)
}
old_values = self._snapshot_field(documents, modified_field)
result = method(documents, **parameters)
if settings.AUDIT_LOG_ENABLED and modified_field:
new_documents = Document.objects.filter(pk__in=documents)
for doc in new_documents:
old_value = old_documents[doc.pk][modified_field]
new_value = getattr(doc, modified_field)
if isinstance(new_value, Model):
# correspondent, document type, etc.
new_value = new_value.pk
elif isinstance(new_value, Manager):
# tags, custom fields
new_value = list(new_value.values_list("pk", flat=True))
new_values = self._snapshot_field(documents, modified_field)
for doc in Document.objects.filter(pk__in=documents):
LogEntry.objects.log_create(
instance=doc,
changes={
modified_field: [
old_value,
new_value,
old_values[doc.pk],
new_values[doc.pk],
],
},
action=LogEntry.Action.UPDATE,
+4 -6
View File
@@ -4,8 +4,7 @@ import httpx
from celery import shared_task
from django.conf import settings
from paperless.network import GuardedHTTPTransport
from paperless.network import OutboundRequestBlockedError
from paperless.network import PinnedHostHTTPTransport
from paperless.network import validate_outbound_http_url
logger = logging.getLogger("paperless.workflows.webhooks")
@@ -15,7 +14,7 @@ logger = logging.getLogger("paperless.workflows.webhooks")
retry_backoff=True,
autoretry_for=(httpx.HTTPStatusError,),
max_retries=3,
throws=(httpx.HTTPError, OutboundRequestBlockedError),
throws=(httpx.HTTPError,),
)
def send_webhook(
url: str,
@@ -30,15 +29,14 @@ def send_webhook(
url,
allowed_schemes=settings.WEBHOOKS_ALLOWED_SCHEMES,
allowed_ports=settings.WEBHOOKS_ALLOWED_PORTS,
# Scheme and port only; the transport enforces the internal-address
# policy at connect time, on the address actually dialled.
# Internal-address checks happen in transport to preserve ConnectError behavior.
allow_internal=True,
)
except ValueError as e:
logger.warning("Webhook blocked: %s", e)
raise
transport = GuardedHTTPTransport(
transport = PinnedHostHTTPTransport(
allow_internal=settings.WEBHOOKS_ALLOW_INTERNAL_REQUESTS,
)
+62 -58
View File
@@ -2,7 +2,7 @@ msgid ""
msgstr ""
"Project-Id-Version: paperless-ngx\n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2026-09-21 19:00+0000\n"
"POT-Creation-Date: 2026-09-23 19:00+0000\n"
"PO-Revision-Date: 2022-02-17 04:17\n"
"Last-Translator: \n"
"Language-Team: English\n"
@@ -1632,7 +1632,7 @@ msgid "workflow runs"
msgstr ""
#: documents/serialisers.py:514 documents/serialisers.py:871
#: documents/serialisers.py:2885 documents/views.py:343 documents/views.py:2726
#: documents/serialisers.py:2885 documents/views.py:342 documents/views.py:2725
#: paperless_mail/serialisers.py:156
msgid "Insufficient permissions."
msgstr ""
@@ -1673,7 +1673,7 @@ msgstr ""
msgid "Duplicate document identifiers are not allowed."
msgstr ""
#: documents/serialisers.py:2971 documents/views.py:4763
#: documents/serialisers.py:2971 documents/views.py:4780
#, python-format
msgid "Documents not found: %(ids)s"
msgstr ""
@@ -1941,40 +1941,40 @@ msgstr ""
msgid "Unable to parse URI {value}"
msgstr ""
#: documents/views.py:336 documents/views.py:2723
#: documents/views.py:335 documents/views.py:2722
msgid "Invalid more_like_id"
msgstr ""
#: documents/views.py:1670
#: documents/views.py:1669
msgid "Invalid AI configuration."
msgstr ""
#: documents/views.py:1681
#: documents/views.py:1680
msgid "AI backend request timed out."
msgstr ""
#: documents/views.py:1693
#: documents/views.py:1692
msgid "AI backend rejected the request. Check logs for details."
msgstr ""
#: documents/views.py:2548 documents/views.py:2864
#: documents/views.py:2547 documents/views.py:2863
msgid "Specify only one of text, title_search, query, or more_like_id."
msgstr ""
#: documents/views.py:4776
#: documents/views.py:4793
#, python-format
msgid "Insufficient permissions to share document %(id)s."
msgstr ""
#: documents/views.py:4822
#: documents/views.py:4839
msgid "Bundle is already being processed."
msgstr ""
#: documents/views.py:4886
#: documents/views.py:4903
msgid "The share link bundle is still being prepared. Please try again later."
msgstr ""
#: documents/views.py:4900
#: documents/views.py:4917
msgid "The share link bundle is unavailable."
msgstr ""
@@ -2219,190 +2219,194 @@ msgid "Sets the LLM embedding model"
msgstr ""
#: paperless/models.py:369
msgid "Sets the LLM embedding endpoint, optional"
msgid "Sets the LLM embedding API key"
msgstr ""
#: paperless/models.py:376
msgid "Sets the LLM embedding endpoint, optional"
msgstr ""
#: paperless/models.py:383
msgid "Sets the LLM embedding chunk size"
msgstr ""
#: paperless/models.py:382
#: paperless/models.py:389
msgid "Sets the LLM context size"
msgstr ""
#: paperless/models.py:388
#: paperless/models.py:395
msgid "Sets the LLM backend"
msgstr ""
#: paperless/models.py:396
#: paperless/models.py:403
msgid "Sets the LLM model"
msgstr ""
#: paperless/models.py:403
#: paperless/models.py:410
msgid "Sets the LLM API key"
msgstr ""
#: paperless/models.py:410
#: paperless/models.py:417
msgid "Sets the LLM endpoint, optional"
msgstr ""
#: paperless/models.py:417
#: paperless/models.py:424
msgid "Sets the LLM output language"
msgstr ""
#: paperless/models.py:424
#: paperless/models.py:431
msgid "Sets the LLM timeout in seconds"
msgstr ""
#: paperless/models.py:430
#: paperless/models.py:437
msgid "paperless application settings"
msgstr ""
#: paperless/settings/__init__.py:558
#: paperless/settings/__init__.py:559
msgid "English (US)"
msgstr ""
#: paperless/settings/__init__.py:559
#: paperless/settings/__init__.py:560
msgid "Arabic"
msgstr ""
#: paperless/settings/__init__.py:560
#: paperless/settings/__init__.py:561
msgid "Afrikaans"
msgstr ""
#: paperless/settings/__init__.py:561
#: paperless/settings/__init__.py:562
msgid "Belarusian"
msgstr ""
#: paperless/settings/__init__.py:562
#: paperless/settings/__init__.py:563
msgid "Bulgarian"
msgstr ""
#: paperless/settings/__init__.py:563
#: paperless/settings/__init__.py:564
msgid "Catalan"
msgstr ""
#: paperless/settings/__init__.py:564
#: paperless/settings/__init__.py:565
msgid "Czech"
msgstr ""
#: paperless/settings/__init__.py:565
#: paperless/settings/__init__.py:566
msgid "Danish"
msgstr ""
#: paperless/settings/__init__.py:566
#: paperless/settings/__init__.py:567
msgid "German"
msgstr ""
#: paperless/settings/__init__.py:567
#: paperless/settings/__init__.py:568
msgid "Greek"
msgstr ""
#: paperless/settings/__init__.py:568
#: paperless/settings/__init__.py:569
msgid "English (GB)"
msgstr ""
#: paperless/settings/__init__.py:569
#: paperless/settings/__init__.py:570
msgid "Spanish"
msgstr ""
#: paperless/settings/__init__.py:570
#: paperless/settings/__init__.py:571
msgid "Persian"
msgstr ""
#: paperless/settings/__init__.py:571
#: paperless/settings/__init__.py:572
msgid "Finnish"
msgstr ""
#: paperless/settings/__init__.py:572
#: paperless/settings/__init__.py:573
msgid "French"
msgstr ""
#: paperless/settings/__init__.py:573
#: paperless/settings/__init__.py:574
msgid "Hungarian"
msgstr ""
#: paperless/settings/__init__.py:574
#: paperless/settings/__init__.py:575
msgid "Indonesian"
msgstr ""
#: paperless/settings/__init__.py:575
#: paperless/settings/__init__.py:576
msgid "Italian"
msgstr ""
#: paperless/settings/__init__.py:576
#: paperless/settings/__init__.py:577
msgid "Japanese"
msgstr ""
#: paperless/settings/__init__.py:577
#: paperless/settings/__init__.py:578
msgid "Korean"
msgstr ""
#: paperless/settings/__init__.py:578
#: paperless/settings/__init__.py:579
msgid "Luxembourgish"
msgstr ""
#: paperless/settings/__init__.py:579
#: paperless/settings/__init__.py:580
msgid "Norwegian"
msgstr ""
#: paperless/settings/__init__.py:580
#: paperless/settings/__init__.py:581
msgid "Dutch"
msgstr ""
#: paperless/settings/__init__.py:581
#: paperless/settings/__init__.py:582
msgid "Polish"
msgstr ""
#: paperless/settings/__init__.py:582
#: paperless/settings/__init__.py:583
msgid "Portuguese (Brazil)"
msgstr ""
#: paperless/settings/__init__.py:583
#: paperless/settings/__init__.py:584
msgid "Portuguese"
msgstr ""
#: paperless/settings/__init__.py:584
#: paperless/settings/__init__.py:585
msgid "Romanian"
msgstr ""
#: paperless/settings/__init__.py:585
#: paperless/settings/__init__.py:586
msgid "Russian"
msgstr ""
#: paperless/settings/__init__.py:586
#: paperless/settings/__init__.py:587
msgid "Slovak"
msgstr ""
#: paperless/settings/__init__.py:587
#: paperless/settings/__init__.py:588
msgid "Slovenian"
msgstr ""
#: paperless/settings/__init__.py:588
#: paperless/settings/__init__.py:589
msgid "Serbian"
msgstr ""
#: paperless/settings/__init__.py:589
#: paperless/settings/__init__.py:590
msgid "Swedish"
msgstr ""
#: paperless/settings/__init__.py:590
#: paperless/settings/__init__.py:591
msgid "Turkish"
msgstr ""
#: paperless/settings/__init__.py:591
#: paperless/settings/__init__.py:592
msgid "Ukrainian"
msgstr ""
#: paperless/settings/__init__.py:592
#: paperless/settings/__init__.py:593
msgid "Vietnamese"
msgstr ""
#: paperless/settings/__init__.py:593
#: paperless/settings/__init__.py:594
msgid "Chinese Simplified"
msgstr ""
#: paperless/settings/__init__.py:594
#: paperless/settings/__init__.py:595
msgid "Chinese Traditional"
msgstr ""
+7
View File
@@ -1,5 +1,6 @@
import dataclasses
import json
from typing import Any
from django.conf import settings
@@ -244,6 +245,7 @@ class AIConfig(BaseConfig):
ai_enabled: bool = dataclasses.field(init=False)
llm_embedding_backend: str = dataclasses.field(init=False)
llm_embedding_model: str = dataclasses.field(init=False)
llm_embedding_api_key: str = dataclasses.field(init=False)
llm_embedding_endpoint: str = dataclasses.field(init=False)
llm_embedding_chunk_size: int = dataclasses.field(init=False)
llm_context_size: int = dataclasses.field(init=False)
@@ -254,6 +256,7 @@ class AIConfig(BaseConfig):
llm_endpoint: str = dataclasses.field(init=False)
llm_output_language: str = dataclasses.field(init=False)
llm_allow_internal_endpoints: bool = dataclasses.field(init=False)
llm_extra_params: dict[str, Any] = dataclasses.field(init=False)
def __post_init__(self) -> None:
app_config = self._get_config_instance()
@@ -269,6 +272,9 @@ class AIConfig(BaseConfig):
self.llm_embedding_model = (
app_config.llm_embedding_model or settings.LLM_EMBEDDING_MODEL
)
self.llm_embedding_api_key = (
app_config.llm_embedding_api_key or settings.LLM_EMBEDDING_API_KEY
)
self.llm_embedding_endpoint = (
app_config.llm_embedding_endpoint or settings.LLM_EMBEDDING_ENDPOINT
)
@@ -287,6 +293,7 @@ class AIConfig(BaseConfig):
app_config.llm_output_language or settings.LLM_OUTPUT_LANGUAGE
)
self.llm_allow_internal_endpoints = settings.LLM_ALLOW_INTERNAL_ENDPOINTS
self.llm_extra_params = settings.LLM_EXTRA_PARAMS
@property
def llm_index_enabled(self) -> bool:
@@ -0,0 +1,23 @@
# Generated by Django 5.2.16 on 2026-09-11 09:32
from django.db import migrations
from django.db import models
class Migration(migrations.Migration):
dependencies = [
("paperless", "0016_alter_applicationconfiguration_ai_enabled"),
]
operations = [
migrations.AddField(
model_name="applicationconfiguration",
name="llm_embedding_api_key",
field=models.CharField(
blank=True,
max_length=1024,
null=True,
verbose_name="Sets the LLM embedding API key",
),
),
]
+7
View File
@@ -365,6 +365,13 @@ class ApplicationConfiguration(AbstractSingletonModel):
max_length=128,
)
llm_embedding_api_key = models.CharField(
verbose_name=_("Sets the LLM embedding API key"),
blank=True,
null=True,
max_length=1024,
)
llm_embedding_endpoint = models.CharField(
verbose_name=_("Sets the LLM embedding endpoint, optional"),
blank=True,
+158 -519
View File
@@ -1,533 +1,61 @@
import functools
import ipaddress
import logging
import math
import re
import socket
import time
from collections.abc import Callable
from collections.abc import Collection
from collections.abc import Iterable
from enum import StrEnum
from typing import Any
from typing import Final
from typing import Self
from typing import TypeAlias
from urllib.parse import ParseResult
from urllib.parse import urlparse
import anyio
import httpcore
import httpx
# Not exported by httpcore; the guard asserts it is still the async default.
from httpcore._backends.auto import AutoBackend
logger = logging.getLogger("paperless.network")
# requires-python is >=3.11, so no PEP 695 `type` statement.
IPAddress: TypeAlias = ipaddress.IPv4Address | ipaddress.IPv6Address
# Ranges that ipaddress reports as global but which still reach internal hosts.
# Ranges ipaddress does not report as private, but which routinely front
# internal infrastructure.
_NON_PUBLIC_NETWORKS = (
# RFC 6598 shared address space: ISP CGNAT, and the default pod/service
# CIDR on several managed Kubernetes offerings.
ipaddress.ip_network("100.64.0.0/10"),
# RFC 6052 NAT64 well-known prefix: 64:ff9b::7f00:1 is 127.0.0.1 wherever
# a NAT64 gateway exists, yet ipaddress classifies the prefix as global.
# a NAT64 gateway exists.
ipaddress.ip_network("64:ff9b::/96"),
)
class BlockReason(StrEnum):
NON_PUBLIC_ADDRESS = "non_public_address"
UNIX_SOCKET = "unix_socket"
class OutboundRequestBlockedError(Exception):
"""
An outbound connection was refused by policy before any socket was opened.
For NON_PUBLIC_ADDRESS, ``host`` is the name or literal being connected to
and ``address`` the first offending address. For UNIX_SOCKET, ``host`` is
the socket path and ``port`` and ``address`` are None.
``address`` is deliberately left out of the message: the message is logged
and stored on failed tasks, and must not disclose internal addresses.
"""
def __init__(
self,
*,
host: str,
port: int | None,
reason: BlockReason,
address: IPAddress | None = None,
) -> None:
self.host = host
self.port = port
self.reason = reason
self.address = address
target = host if port is None else f"{host}:{port}"
super().__init__(f"Outbound connection to {target} blocked ({reason})")
def __reduce__(self) -> tuple[Callable[..., Self], tuple[object, ...]]:
# Celery rebuilds failed-task exceptions by pickling; keyword-only
# fields cannot be recovered from ``args`` alone.
return (
functools.partial(
type(self),
host=self.host,
port=self.port,
reason=self.reason,
address=self.address,
),
(),
def is_public_ip(ip: str | int) -> bool:
try:
obj = ipaddress.ip_address(ip)
return not (
obj.is_private
or obj.is_loopback
or obj.is_link_local
or obj.is_multicast
or obj.is_unspecified
or any(obj in network for network in _NON_PUBLIC_NETWORKS)
)
except ValueError: # pragma: no cover
return False
class HostResolutionError(Exception):
"""The resolver returned no usable addresses for a host."""
def resolve_hostname_ips(hostname: str) -> list[str]:
try:
addr_info = socket.getaddrinfo(hostname, None)
except socket.gaierror as e:
raise ValueError(f"Could not resolve hostname: {hostname}") from e
def __init__(self, *, host: str, detail: str) -> None:
self.host = host
self.detail = detail
super().__init__(f"Could not resolve {host}: {detail}")
def __reduce__(self) -> tuple[Callable[..., Self], tuple[object, ...]]:
return (
functools.partial(type(self), host=self.host, detail=self.detail),
(),
)
ips = [info[4][0] for info in addr_info if info and info[4]]
if not ips:
raise ValueError(f"Could not resolve hostname: {hostname}")
return ips
def blocked_message(exc: OutboundRequestBlockedError | HostResolutionError) -> str:
"""User-facing text for validation errors, kept stable for existing callers."""
if isinstance(exc, HostResolutionError):
return f"Could not resolve hostname: {exc.host}"
if exc.reason is BlockReason.UNIX_SOCKET:
return "Connection blocked: unix sockets are not permitted"
return f"Connection blocked: {exc.host} resolves to a non-public address"
def is_public_ip(ip: IPAddress) -> bool:
def format_host_for_url(host: str) -> str:
"""
True when ``ip`` is globally routable unicast and not in a range that
ipaddress reports as global but which still reaches internal hosts.
"""
return (
ip.is_global
and not ip.is_multicast
and not any(ip in network for network in _NON_PUBLIC_NETWORKS)
)
# Resolver and clock indirection so tests can fake DNS and time for this module
# without changing how the stock httpcore backends resolve the literals the
# guard dials.
_getaddrinfo = socket.getaddrinfo
_agetaddrinfo = anyio.getaddrinfo
# The clock is a seam because time-machine does not mock monotonic clocks, and
# patching time.monotonic globally would also replace the asyncio event loop's
# own clock, hanging or misfiring its timers for the rest of the test.
_monotonic = time.monotonic
def _collect_addresses(
host: str,
infos: Iterable[tuple[Any, ...]],
) -> tuple[IPAddress, ...]:
# Resolver output is always an address, but a scoped IPv6 answer carries a
# zone id ("fe80::1%1"), which is dropped before classification.
# dict keys keep the first occurrence and resolver order
addresses: dict[IPAddress, None] = {}
for info in infos:
address = ipaddress.ip_address(str(info[4][0]).split("%", 1)[0])
addresses.setdefault(address, None)
if not addresses:
raise HostResolutionError(host=host, detail="no addresses returned")
return tuple(addresses)
def _require_public(
host: str,
port: int | None,
addresses: tuple[IPAddress, ...],
) -> tuple[IPAddress, ...]:
for address in addresses:
if not is_public_ip(address):
raise OutboundRequestBlockedError(
host=host,
port=port,
reason=BlockReason.NON_PUBLIC_ADDRESS,
address=address,
)
return addresses
def resolve_public_addresses(host: str, port: int | None) -> tuple[IPAddress, ...]:
"""
Resolve ``host`` and return its addresses in resolver order, or raise if
any of them is non-public. A name is rejected as a whole; offending
addresses are never filtered out.
IP literals go through the resolver too: getaddrinfo answers them without
a lookup, and validating only its answer means no second parser can read
the host differently from the one that connects.
Format IP address for URL use (wrap IPv6 in brackets).
"""
try:
infos = _getaddrinfo(host, port, type=socket.SOCK_STREAM)
except (OSError, UnicodeError) as e:
raise HostResolutionError(host=host, detail=str(e)) from e
return _require_public(host, port, _collect_addresses(host, infos))
async def aresolve_public_addresses(
host: str,
port: int | None,
) -> tuple[IPAddress, ...]:
"""Async variant of resolve_public_addresses."""
try:
infos = await _agetaddrinfo(host, port, type=socket.SOCK_STREAM)
except (OSError, UnicodeError) as e:
raise HostResolutionError(host=host, detail=str(e)) from e
return _require_public(host, port, _collect_addresses(host, infos))
MAX_ADDRESSES_TRIED: Final = 8
MIN_ATTEMPT_TIMEOUT: Final = 2.0
MAX_ATTEMPT_TIMEOUT: Final = 10.0
def _require_positive_timeout(host: str, timeout: float | None) -> None:
# A zero timeout makes the socket non-blocking and a negative one is
# rejected by settimeout; neither can produce a useful connection attempt.
if timeout is not None and timeout <= 0:
raise httpcore.ConnectTimeout(
f"Connect timeout for {host} must be positive, got {timeout}",
)
def _deadline(timeout: float | None) -> float:
return math.inf if timeout is None else _monotonic() + timeout
def _attempt_order(addresses: tuple[IPAddress, ...]) -> list[IPAddress]:
# Alternate address families, starting with the resolver's first family
# (RFC 8305 section 4), so one unreachable family cannot delay the other.
first_version = addresses[0].version
primary = [a for a in addresses if a.version == first_version]
secondary = [a for a in addresses if a.version != first_version]
ordered: list[IPAddress] = []
for index in range(max(len(primary), len(secondary))):
ordered.extend(primary[index : index + 1])
ordered.extend(secondary[index : index + 1])
return ordered[:MAX_ADDRESSES_TRIED]
def _attempt_timeout(remaining: float, attempts_left: int) -> float:
"""
Budget for the next attempt. Once the budget is too small to split, or on
the last address, the attempt gets everything left. Otherwise it gets an
equal share clamped to [MIN, MAX], always leaving MIN for a later attempt.
The floor survives one lost SYN; the ceiling bounds how long a black-holed
address delays the next one.
"""
if attempts_left == 1 or remaining < 2 * MIN_ATTEMPT_TIMEOUT:
return remaining
share = remaining / attempts_left
return min(
MAX_ATTEMPT_TIMEOUT,
max(MIN_ATTEMPT_TIMEOUT, share),
remaining - MIN_ATTEMPT_TIMEOUT,
)
def _as_httpcore_timeout(seconds: float) -> float | None:
return None if math.isinf(seconds) else seconds
def _log_block(error: OutboundRequestBlockedError) -> None:
logger.warning("Blocked outbound connection: %s", error)
def _budget_exhausted(host: str, tried: int, total: int) -> httpcore.ConnectTimeout:
return httpcore.ConnectTimeout(
f"Timed out connecting to {host} after trying {tried} of {total} addresses",
)
def _next_attempt_budget(
host: str,
deadline: float,
candidates: list[IPAddress],
index: int,
) -> float:
"""Budget for the attempt at index, or a timeout if none is left."""
remaining = deadline - _monotonic()
if remaining <= 0:
raise _budget_exhausted(host, index, len(candidates))
return _attempt_timeout(remaining, len(candidates) - index)
def _resolve_for_connect(host: str, port: int) -> tuple[IPAddress, ...]:
try:
return resolve_public_addresses(host, port)
except OutboundRequestBlockedError as e:
_log_block(e)
raise
except HostResolutionError as e:
raise httpcore.ConnectError(str(e)) from e
async def _aresolve_for_connect(
host: str,
port: int,
timeout: float | None,
) -> tuple[IPAddress, ...]:
# The scope closes before dialling; attempts are not nested inside it.
try:
with anyio.fail_after(timeout):
return await aresolve_public_addresses(host, port)
except TimeoutError as e:
raise httpcore.ConnectTimeout(f"Timed out resolving {host}") from e
except OutboundRequestBlockedError as e:
_log_block(e)
raise
except HostResolutionError as e:
raise httpcore.ConnectError(str(e)) from e
class _GuardedSyncBackend(httpcore.NetworkBackend):
"""
Wraps httpcore's sync backend. With internal addresses disallowed, it
resolves the origin host itself, rejects the name if any address is
non-public, and dials the validated literals so the checked address is
the connected one. TLS still verifies against the origin hostname.
"""
def __init__(self, inner: httpcore.NetworkBackend, *, allow_internal: bool) -> None:
self._inner = inner
self._allow_internal = allow_internal
def connect_tcp(
self,
host: str,
port: int,
timeout: float | None = None,
local_address: str | None = None,
socket_options: Iterable[httpcore.SOCKET_OPTION] | None = None,
) -> httpcore.NetworkStream:
if self._allow_internal:
return self._inner.connect_tcp(
host,
port,
timeout=timeout,
local_address=local_address,
socket_options=socket_options,
)
_require_positive_timeout(host, timeout)
# Resolution is not charged to the budget, matching the stock backend.
candidates = _attempt_order(_resolve_for_connect(host, port))
deadline = _deadline(timeout)
last_error: httpcore.ConnectError | httpcore.ConnectTimeout | None = None
for index, address in enumerate(candidates):
budget = _next_attempt_budget(host, deadline, candidates, index)
try:
return self._inner.connect_tcp(
str(address),
port,
timeout=_as_httpcore_timeout(budget),
local_address=local_address,
socket_options=socket_options,
)
except (httpcore.ConnectError, httpcore.ConnectTimeout) as e:
logger.debug("Connecting to %s via %s failed: %s", host, address, e)
last_error = e
# candidates is never empty, so every address was tried and failed
raise last_error or _budget_exhausted(host, len(candidates), len(candidates))
def connect_unix_socket(
self,
path: str,
timeout: float | None = None,
socket_options: Iterable[httpcore.SOCKET_OPTION] | None = None,
) -> httpcore.NetworkStream:
error = OutboundRequestBlockedError(
host=path,
port=None,
reason=BlockReason.UNIX_SOCKET,
)
_log_block(error)
raise error
def sleep(self, seconds: float) -> None:
self._inner.sleep(seconds)
class _GuardedAsyncBackend(httpcore.AsyncNetworkBackend):
"""Async twin of _GuardedSyncBackend."""
def __init__(
self,
inner: httpcore.AsyncNetworkBackend,
*,
allow_internal: bool,
) -> None:
self._inner = inner
self._allow_internal = allow_internal
async def connect_tcp(
self,
host: str,
port: int,
timeout: float | None = None,
local_address: str | None = None,
socket_options: Iterable[httpcore.SOCKET_OPTION] | None = None,
) -> httpcore.AsyncNetworkStream:
if self._allow_internal:
return await self._inner.connect_tcp(
host,
port,
timeout=timeout,
local_address=local_address,
socket_options=socket_options,
)
_require_positive_timeout(host, timeout)
# Resolution counts against the budget, matching the stock backend.
deadline = _deadline(timeout)
candidates = _attempt_order(await _aresolve_for_connect(host, port, timeout))
last_error: httpcore.ConnectError | httpcore.ConnectTimeout | None = None
for index, address in enumerate(candidates):
budget = _next_attempt_budget(host, deadline, candidates, index)
try:
return await self._inner.connect_tcp(
str(address),
port,
timeout=_as_httpcore_timeout(budget),
local_address=local_address,
socket_options=socket_options,
)
except (httpcore.ConnectError, httpcore.ConnectTimeout) as e:
logger.debug("Connecting to %s via %s failed: %s", host, address, e)
last_error = e
raise last_error or _budget_exhausted(host, len(candidates), len(candidates))
async def connect_unix_socket(
self,
path: str,
timeout: float | None = None,
socket_options: Iterable[httpcore.SOCKET_OPTION] | None = None,
) -> httpcore.AsyncNetworkStream:
error = OutboundRequestBlockedError(
host=path,
port=None,
reason=BlockReason.UNIX_SOCKET,
)
_log_block(error)
raise error
async def sleep(self, seconds: float) -> None:
await self._inner.sleep(seconds)
_LAYOUT_ERROR = (
"Unexpected httpx transport layout; refusing to create a transport "
"without the outbound connection guard"
)
class GuardedHTTPTransport(httpx.HTTPTransport):
"""
httpx transport whose connections pass through the outbound guard.
Deliberately accepts no proxy, uds or retries options: a proxy would be
dialled instead of the destination, and a unix socket bypasses TCP
entirely. Adding an option here is a reviewed change, not a pass-through.
"""
def __init__(self, *, allow_internal: bool) -> None:
super().__init__()
# httpx has no public hook for the network backend. Check the exact
# layout before swapping so an httpx or httpcore change fails loudly.
pool = self._pool
if (
type(pool) is not httpcore.ConnectionPool
or type(pool._network_backend) is not httpcore.SyncBackend
):
raise RuntimeError(_LAYOUT_ERROR)
pool._network_backend = _GuardedSyncBackend(
pool._network_backend,
allow_internal=allow_internal,
)
class GuardedAsyncHTTPTransport(httpx.AsyncHTTPTransport):
"""Async twin of GuardedHTTPTransport."""
def __init__(self, *, allow_internal: bool) -> None:
super().__init__()
pool = self._pool
if (
type(pool) is not httpcore.AsyncConnectionPool
or type(pool._network_backend) is not AutoBackend
):
raise RuntimeError(_LAYOUT_ERROR)
pool._network_backend = _GuardedAsyncBackend(
pool._network_backend,
allow_internal=allow_internal,
)
def create_guarded_httpx_client(
url: str,
*,
allow_internal: bool,
timeout: float,
) -> httpx.Client:
"""
Validate ``url`` up front, then build a client that re-checks at connect
time. The up-front check turns static misconfiguration into a ValueError
before any retry layer sees it.
"""
validate_outbound_http_url(url, allow_internal=allow_internal)
return httpx.Client(
transport=GuardedHTTPTransport(allow_internal=allow_internal),
timeout=timeout,
)
def create_guarded_async_httpx_client(
url: str,
*,
allow_internal: bool,
timeout: float,
) -> httpx.AsyncClient:
"""Async twin of create_guarded_httpx_client."""
validate_outbound_http_url(url, allow_internal=allow_internal)
return httpx.AsyncClient(
transport=GuardedAsyncHTTPTransport(allow_internal=allow_internal),
timeout=timeout,
)
# urllib3 treats a backslash as ending the authority while urlparse and httpx do
# not, so the host checked here could differ from the one that is dialled.
# Control and whitespace characters are refused for the same reason.
_UNSAFE_URL_CHARS = re.compile(r"[\\\x00-\x1f\x7f\s]")
def _dns_name(url: str) -> str:
"""
The ASCII hostname that httpx and urllib3 look up for ``url``.
urlparse keeps a non-ASCII hostname as typed, and getaddrinfo would then
encode it with the stdlib IDNA 2003 codec. That maps some characters
differently from the IDNA 2008 encoding the HTTP clients use ("faß"
becomes "fass" instead of "xn--fa-hia"), so the check would resolve a
different name from the one that is connected to.
"""
try:
return httpx.URL(url).raw_host.decode("ascii")
except (httpx.InvalidURL, UnicodeError) as e:
raise ValueError("Invalid URL scheme or hostname.") from e
ip_obj = ipaddress.ip_address(host)
if ip_obj.version == 6:
return f"[{host}]"
return host
except ValueError:
return host
def validate_outbound_http_url(
@@ -553,17 +81,128 @@ def validate_outbound_http_url(
raise ValueError("Destination port not permitted.")
if not allow_internal:
if _UNSAFE_URL_CHARS.search(url):
raise ValueError("Invalid URL scheme or hostname.")
host = _dns_name(url)
# HTTP clients may percent-decode the host before resolving it, so the
# checked name could differ from the dialled one. An IPv6 zone id is the
# only legitimate use, and link-local addresses are non-public anyway.
if "%" in host:
raise ValueError("Invalid URL scheme or hostname.")
try:
resolve_public_addresses(host, port)
except (OutboundRequestBlockedError, HostResolutionError) as e:
raise ValueError(blocked_message(e)) from e
for ip_str in resolve_hostname_ips(parsed.hostname):
if not is_public_ip(ip_str):
raise ValueError(
f"Connection blocked: {parsed.hostname} resolves to a non-public address",
)
return parsed
def _rewrite_request_to_pinned_ip(
request: httpx.Request,
*,
allow_internal: bool,
) -> httpx.Request:
hostname = request.url.host
if not hostname:
raise httpx.ConnectError("No hostname in request URL")
try:
ips = resolve_hostname_ips(hostname)
except ValueError as e:
raise httpx.ConnectError(str(e)) from e
if not allow_internal:
for ip_str in ips:
if not is_public_ip(ip_str):
raise httpx.ConnectError(
f"Connection blocked: {hostname} resolves to a non-public address",
)
ip_str = ips[0]
formatted_ip = format_host_for_url(ip_str)
new_headers = httpx.Headers(request.headers)
if "host" in new_headers:
del new_headers["host"]
host_header = format_host_for_url(hostname)
default_port = 443 if request.url.scheme == "https" else 80
if request.url.port and request.url.port != default_port:
host_header = f"{host_header}:{request.url.port}"
new_headers["Host"] = host_header
new_url = request.url.copy_with(host=formatted_ip)
rewritten_request = httpx.Request(
method=request.method,
url=new_url,
headers=new_headers,
stream=request.stream,
extensions=request.extensions,
)
rewritten_request.extensions["sni_hostname"] = hostname
return rewritten_request
class PinnedHostHTTPTransport(httpx.HTTPTransport):
"""
HTTP transport that resolves/validates hostnames per request and connects to
a vetted IP while preserving the original Host header and TLS SNI hostname.
"""
def __init__(
self,
*args,
allow_internal: bool = False,
**kwargs,
) -> None:
super().__init__(*args, **kwargs)
self.allow_internal = allow_internal
def handle_request(self, request: httpx.Request) -> httpx.Response:
request = _rewrite_request_to_pinned_ip(
request,
allow_internal=self.allow_internal,
)
return super().handle_request(request)
class PinnedHostAsyncHTTPTransport(httpx.AsyncHTTPTransport):
"""
Async variant of PinnedHostHTTPTransport.
"""
def __init__(
self,
*args,
allow_internal: bool = False,
**kwargs,
) -> None:
super().__init__(*args, **kwargs)
self.allow_internal = allow_internal
async def handle_async_request(self, request: httpx.Request) -> httpx.Response:
request = _rewrite_request_to_pinned_ip(
request,
allow_internal=self.allow_internal,
)
return await super().handle_async_request(request)
def create_pinned_httpx_client(
url: str,
*,
allow_internal: bool = False,
**kwargs,
) -> httpx.Client:
validate_outbound_http_url(url, allow_internal=allow_internal)
return httpx.Client(
transport=PinnedHostHTTPTransport(allow_internal=allow_internal),
**kwargs,
)
def create_pinned_async_httpx_client(
url: str,
*,
allow_internal: bool = False,
**kwargs,
) -> httpx.AsyncClient:
validate_outbound_http_url(url, allow_internal=allow_internal)
return httpx.AsyncClient(
transport=PinnedHostAsyncHTTPTransport(allow_internal=allow_internal),
**kwargs,
)
+10 -1
View File
@@ -216,6 +216,11 @@ class ApplicationConfigurationSerializer(
externally_configured_variables = serializers.SerializerMethodField()
user_args = serializers.JSONField(binary=True, allow_null=True)
barcode_tag_mapping = serializers.JSONField(binary=True, allow_null=True)
llm_embedding_api_key = ObfuscatedPasswordField(
required=False,
allow_null=True,
max_length=1024,
)
llm_api_key = ObfuscatedPasswordField(
required=False,
allow_null=True,
@@ -227,7 +232,11 @@ class ApplicationConfigurationSerializer(
max_length=1024,
)
OBFUSCATED_FIELDS = ("llm_api_key", "remote_ocr_api_key")
OBFUSCATED_FIELDS = (
"llm_embedding_api_key",
"llm_api_key",
"remote_ocr_api_key",
)
def get_externally_configured_variables(
self,
+22
View File
@@ -7,6 +7,7 @@ import multiprocessing
import os
import tempfile
from pathlib import Path
from typing import Any
from typing import Final
from urllib.parse import urlparse
@@ -1081,6 +1082,25 @@ CLASSIFIER_LANGUAGES: Final[dict[str, str]] = {
}
def _get_llm_extra_params() -> dict[str, Any]:
"""
Parse PAPERLESS_AI_LLM_EXTRA_PARAMS, a JSON object passed straight through
to the LLM backend's request body.
"""
raw = os.getenv("PAPERLESS_AI_LLM_EXTRA_PARAMS", "{}")
try:
parsed = json.loads(raw)
except json.JSONDecodeError as e:
raise ImproperlyConfigured(
"PAPERLESS_AI_LLM_EXTRA_PARAMS must be valid JSON",
) from e
if not isinstance(parsed, dict):
raise ImproperlyConfigured(
"PAPERLESS_AI_LLM_EXTRA_PARAMS must be a JSON object",
)
return parsed
def _get_classifier_language_setting(ocr_lang: str) -> str | None:
"""
Maps the primary Tesseract language to the classifier's stemming
@@ -1216,6 +1236,7 @@ LLM_EMBEDDING_BACKEND = get_choice_from_env(
{"huggingface", "openai-like", "ollama"},
)
LLM_EMBEDDING_MODEL = os.getenv("PAPERLESS_AI_LLM_EMBEDDING_MODEL")
LLM_EMBEDDING_API_KEY = os.getenv("PAPERLESS_AI_LLM_EMBEDDING_API_KEY")
LLM_EMBEDDING_ENDPOINT = os.getenv("PAPERLESS_AI_LLM_EMBEDDING_ENDPOINT")
LLM_EMBEDDING_CHUNK_SIZE = get_int_from_env(
"PAPERLESS_AI_LLM_EMBEDDING_CHUNK_SIZE",
@@ -1241,3 +1262,4 @@ LLM_ALLOW_INTERNAL_ENDPOINTS = get_bool_from_env(
"PAPERLESS_AI_LLM_ALLOW_INTERNAL_ENDPOINTS",
"true",
)
LLM_EXTRA_PARAMS = _get_llm_extra_params()
@@ -17,6 +17,24 @@ class TestRemoteUser(DirectoriesMixin, APITestCase):
self.user = UserFactory(username="temp_admin", superuser=True)
# _parse_remote_user_settings() mutates these shared lists in place,
# so undo that after the test instead of leaking remote-user auth
# into every test that runs afterward.
original_middleware = list(settings.MIDDLEWARE)
original_auth_backends = list(settings.AUTHENTICATION_BACKENDS)
original_auth_classes = list(
settings.REST_FRAMEWORK["DEFAULT_AUTHENTICATION_CLASSES"],
)
def _restore_remote_user_settings() -> None:
settings.MIDDLEWARE[:] = original_middleware
settings.AUTHENTICATION_BACKENDS[:] = original_auth_backends
settings.REST_FRAMEWORK["DEFAULT_AUTHENTICATION_CLASSES"][:] = (
original_auth_classes
)
self.addCleanup(_restore_remote_user_settings)
def test_remote_user(self) -> None:
"""
GIVEN:
@@ -7,6 +7,7 @@ from django.core.exceptions import ImproperlyConfigured
from paperless.settings import _get_allauth_trusted_proxy_count
from paperless.settings import _get_classifier_language_setting
from paperless.settings import _get_llm_extra_params
from paperless.settings import _get_search_language_setting
from paperless.settings import _parse_paperless_url
from paperless.settings import default_threads_per_worker
@@ -166,3 +167,45 @@ class TestPaperlessURLSettings(TestCase):
self.assertIn(url, settings.CSRF_TRUSTED_ORIGINS)
self.assertIn(url, settings.CORS_ALLOWED_ORIGINS)
class TestLlmExtraParams:
@pytest.mark.parametrize(
("env_value", "expected"),
[
pytest.param(None, {}, id="unset"),
pytest.param(
'{"reasoning_effort": "none"}',
{"reasoning_effort": "none"},
id="json-object",
),
],
)
def test_parses(
self,
monkeypatch,
env_value,
expected,
):
if env_value is None:
monkeypatch.delenv("PAPERLESS_AI_LLM_EXTRA_PARAMS", raising=False)
else:
monkeypatch.setenv("PAPERLESS_AI_LLM_EXTRA_PARAMS", env_value)
assert _get_llm_extra_params() == expected
@pytest.mark.parametrize(
("env_value", "match"),
[
pytest.param("reasoning_effort=none", "valid JSON", id="invalid-json"),
pytest.param('["none"]', "JSON object", id="not-an-object"),
],
)
def test_invalid_raises(
self,
monkeypatch,
env_value,
match,
):
monkeypatch.setenv("PAPERLESS_AI_LLM_EXTRA_PARAMS", env_value)
with pytest.raises(ImproperlyConfigured, match=match):
_get_llm_extra_params()
@@ -30,3 +30,27 @@ class TestBooleanConfigPrecedence(TestCase):
config.save()
self.assertTrue(AIConfig().ai_enabled)
class TestAIConfigPrecedence(TestCase):
@override_settings(LLM_EMBEDDING_API_KEY="environment-embedding-key")
def test_database_embedding_api_key_overrides_environment_setting(self) -> None:
config, _ = ApplicationConfiguration.objects.get_or_create()
config.llm_embedding_api_key = "database-embedding-key"
config.save()
self.assertEqual(
AIConfig().llm_embedding_api_key,
"database-embedding-key",
)
@override_settings(LLM_EMBEDDING_API_KEY="environment-embedding-key")
def test_null_embedding_api_key_uses_environment_setting(self) -> None:
config, _ = ApplicationConfiguration.objects.get_or_create()
config.llm_embedding_api_key = None
config.save()
self.assertEqual(
AIConfig().llm_embedding_api_key,
"environment-embedding-key",
)
File diff suppressed because it is too large Load Diff
@@ -1,375 +0,0 @@
import ipaddress
import os
import httpcore
import httpx
import pytest
from pytest_mock import MockerFixture
from paperless.network import GuardedAsyncHTTPTransport
from paperless.network import GuardedHTTPTransport
from paperless.network import OutboundRequestBlockedError
from paperless.network import create_guarded_httpx_client
from paperless_testing.outbound import DialRecorder
from paperless_testing.outbound import FakeDNS
from paperless_testing.outbound import LocalHTTPServer
from paperless_testing.outbound import running_http_server
class TestGuardedTransportSync:
@pytest.mark.usefixtures("every_address_is_public")
def test_pinned_connection_falls_back_to_next_address(
self,
local_http_server: LocalHTTPServer,
fake_dns: FakeDNS,
dial_recorder: DialRecorder,
) -> None:
"""
GIVEN:
- A hostname resolving to ::1 then 127.0.0.1
- A server listening on 127.0.0.1 only
- Internal addresses disallowed, with loopback treated as public
WHEN:
- A request is made
THEN:
- ::1 fails, 127.0.0.1 is dialled next and the request succeeds
"""
fake_dns.add("dual-stack.test", "::1", "127.0.0.1")
with httpx.Client(
transport=GuardedHTTPTransport(allow_internal=False),
timeout=5.0,
) as client:
response = client.get(f"http://dual-stack.test:{local_http_server.port}/")
assert response.status_code == 200
assert dial_recorder.hosts() == ["::1", "127.0.0.1"]
def test_allow_internal_uses_stock_resolution(
self,
local_http_server: LocalHTTPServer,
fake_dns: FakeDNS,
) -> None:
"""
GIVEN:
- Internal addresses allowed
WHEN:
- A request is made to localhost
THEN:
- It succeeds without the guard resolving anything
"""
with httpx.Client(
transport=GuardedHTTPTransport(allow_internal=True),
timeout=5.0,
) as client:
response = client.get(f"http://localhost:{local_http_server.port}/")
assert response.status_code == 200
assert fake_dns.lookups == []
@pytest.mark.usefixtures("every_address_is_public")
def test_host_header_is_the_hostname(
self,
local_http_server: LocalHTTPServer,
fake_dns: FakeDNS,
) -> None:
"""
GIVEN:
- A pinned connection to a named host
WHEN:
- A request is made
THEN:
- The server receives the hostname in Host, not the dialled IP
"""
fake_dns.add("pinned.test", "127.0.0.1")
with httpx.Client(
transport=GuardedHTTPTransport(allow_internal=False),
timeout=5.0,
) as client:
client.get(f"http://pinned.test:{local_http_server.port}/")
assert local_http_server.requests[0].headers["host"] == (
f"pinned.test:{local_http_server.port}"
)
def test_redirect_to_internal_host_is_blocked(
self,
mocker: MockerFixture,
local_http_server: LocalHTTPServer,
fake_dns: FakeDNS,
dial_recorder: DialRecorder,
) -> None:
"""
GIVEN:
- An allowed origin that redirects to a host resolving to a blocked
address, and a client that follows redirects
WHEN:
- The origin is requested
THEN:
- The redirect hop is blocked without dialling the blocked address
"""
allowed = ipaddress.ip_address("127.0.0.1")
mocker.patch(
"paperless.network.is_public_ip",
side_effect=lambda address: address == allowed,
)
fake_dns.add("origin.test", "127.0.0.1")
fake_dns.add("internal.test", "127.0.0.2")
local_http_server.redirect_to = (
f"http://internal.test:{local_http_server.port}/"
)
with (
httpx.Client(
transport=GuardedHTTPTransport(allow_internal=False),
timeout=5.0,
follow_redirects=True,
) as client,
pytest.raises(OutboundRequestBlockedError) as exc_info,
):
client.get(f"http://origin.test:{local_http_server.port}/")
assert exc_info.value.address == ipaddress.ip_address("127.0.0.2")
assert dial_recorder.hosts() == ["127.0.0.1"]
assert len(local_http_server.requests) == 1
@pytest.mark.usefixtures("every_address_is_public")
def test_connections_are_not_shared_between_hosts_on_one_address(
self,
local_http_server: LocalHTTPServer,
fake_dns: FakeDNS,
dial_recorder: DialRecorder,
) -> None:
"""
GIVEN:
- Two hostnames resolving to the same address
- Internal addresses disallowed, with loopback treated as public
WHEN:
- One client requests the first host twice, then the second host
THEN:
- The first host's connection is reused for its second request
- The second host gets its own connection, so its certificate would
be checked rather than inheriting the first host's session
"""
fake_dns.add("first.test", "127.0.0.1")
fake_dns.add("second.test", "127.0.0.1")
with httpx.Client(
transport=GuardedHTTPTransport(allow_internal=False),
timeout=5.0,
) as client:
client.get(f"http://first.test:{local_http_server.port}/")
client.get(f"http://first.test:{local_http_server.port}/")
client.get(f"http://second.test:{local_http_server.port}/")
assert dial_recorder.hosts() == ["127.0.0.1", "127.0.0.1"]
assert local_http_server.connections == 2
assert [request.headers["host"] for request in local_http_server.requests] == [
f"first.test:{local_http_server.port}",
f"first.test:{local_http_server.port}",
f"second.test:{local_http_server.port}",
]
@pytest.mark.usefixtures("every_address_is_public")
def test_tls_uses_the_hostname_not_the_dialled_address(
self,
mocker: MockerFixture,
local_http_server: LocalHTTPServer,
fake_dns: FakeDNS,
dial_recorder: DialRecorder,
) -> None:
"""
GIVEN:
- A pinned HTTPS connection to a named host
- A plain HTTP server, so the handshake itself fails
WHEN:
- A request is made
THEN:
- The validated address is dialled
- TLS is started with the hostname for SNI and certificate checks
"""
fake_dns.add("pinned.test", "127.0.0.1")
start_tls = mocker.spy(httpcore._backends.sync.SyncStream, "start_tls")
with (
httpx.Client(
transport=GuardedHTTPTransport(allow_internal=False),
timeout=5.0,
) as client,
pytest.raises(httpx.ConnectError),
):
client.get(f"https://pinned.test:{local_http_server.port}/")
assert dial_recorder.hosts() == ["127.0.0.1"]
start_tls.assert_called_once()
assert start_tls.call_args.kwargs["server_hostname"] == "pinned.test"
@pytest.mark.parametrize(
"host",
[
pytest.param("localhost", id="name"),
pytest.param("2130706433", id="decimal"),
pytest.param("0x7f.1", id="hex-short"),
pytest.param("127.1", id="short-dotted"),
],
)
def test_blocks_internal_host_without_connecting(
self,
local_http_server: LocalHTTPServer,
dial_recorder: DialRecorder,
host: str,
) -> None:
"""
GIVEN:
- Internal addresses disallowed
- A URL whose host reaches loopback, by name or by a
non-canonical spelling of 127.0.0.1
WHEN:
- A request is made through the transport
THEN:
- The resolved address is checked, the request is blocked and the
server never sees a connection
"""
with (
httpx.Client(
transport=GuardedHTTPTransport(allow_internal=False),
timeout=5.0,
) as client,
pytest.raises(OutboundRequestBlockedError),
):
client.get(f"http://{host}:{local_http_server.port}/")
assert local_http_server.connections == 0
assert dial_recorder.hosts() == []
@pytest.mark.usefixtures("every_address_is_public")
def test_environment_proxy_is_not_used(
self,
mocker: MockerFixture,
local_http_server: LocalHTTPServer,
fake_dns: FakeDNS,
dial_recorder: DialRecorder,
) -> None:
"""
GIVEN:
- Proxy variables in the environment pointing at a second local server
- Internal addresses disallowed
WHEN:
- A request is made through the production client factory to an
allowed origin
THEN:
- The origin server receives the request directly and the proxy
server never sees a connection
"""
with running_http_server() as proxy_server:
mocker.patch.dict(
os.environ,
{
"HTTP_PROXY": f"http://127.0.0.1:{proxy_server.port}",
"HTTPS_PROXY": f"http://127.0.0.1:{proxy_server.port}",
"ALL_PROXY": f"http://127.0.0.1:{proxy_server.port}",
},
)
fake_dns.add("origin.test", "127.0.0.1")
url = f"http://origin.test:{local_http_server.port}/"
with create_guarded_httpx_client(
url,
allow_internal=False,
timeout=5.0,
) as client:
response = client.get(url)
assert response.status_code == 200
assert len(local_http_server.requests) == 1
assert local_http_server.requests[0].headers["host"] == (
f"origin.test:{local_http_server.port}"
)
assert proxy_server.connections == 0
assert proxy_server.requests == []
assert dial_recorder.hosts() == ["127.0.0.1"]
class TestGuardedTransportAsync:
@pytest.fixture(autouse=True)
def anyio_backend(self) -> str:
return "asyncio"
@pytest.mark.anyio
@pytest.mark.usefixtures("every_address_is_public")
async def test_pinned_connection_falls_back_to_next_address(
self,
local_http_server: LocalHTTPServer,
fake_dns: FakeDNS,
dial_recorder: DialRecorder,
) -> None:
"""
GIVEN:
- A hostname resolving to ::1 then 127.0.0.1
- A server listening on 127.0.0.1 only
- Internal addresses disallowed, with loopback treated as public
WHEN:
- An async request is made
THEN:
- ::1 fails, 127.0.0.1 is dialled next and the request succeeds
"""
fake_dns.add("dual-stack.test", "::1", "127.0.0.1")
async with httpx.AsyncClient(
transport=GuardedAsyncHTTPTransport(allow_internal=False),
timeout=5.0,
) as client:
response = await client.get(
f"http://dual-stack.test:{local_http_server.port}/",
)
assert response.status_code == 200
assert dial_recorder.hosts() == ["::1", "127.0.0.1"]
@pytest.mark.anyio
async def test_allow_internal_uses_stock_resolution(
self,
local_http_server: LocalHTTPServer,
fake_dns: FakeDNS,
) -> None:
"""
GIVEN:
- Internal addresses allowed
WHEN:
- An async request is made to localhost
THEN:
- It succeeds without the guard resolving anything
"""
async with httpx.AsyncClient(
transport=GuardedAsyncHTTPTransport(allow_internal=True),
timeout=5.0,
) as client:
response = await client.get(f"http://localhost:{local_http_server.port}/")
assert response.status_code == 200
assert fake_dns.lookups == []
@pytest.mark.anyio
async def test_blocks_internal_host_without_connecting(
self,
local_http_server: LocalHTTPServer,
dial_recorder: DialRecorder,
) -> None:
"""
GIVEN:
- Internal addresses disallowed
WHEN:
- An async request is made to localhost through the transport
THEN:
- It is blocked and the server never sees a connection
"""
async with httpx.AsyncClient(
transport=GuardedAsyncHTTPTransport(allow_internal=False),
timeout=5.0,
) as client:
with pytest.raises(OutboundRequestBlockedError):
await client.get(f"http://localhost:{local_http_server.port}/")
assert local_http_server.connections == 0
assert dial_recorder.hosts() == []
+10 -29
View File
@@ -14,16 +14,14 @@ if TYPE_CHECKING:
from llama_index.llms.openai_like import OpenAILike
from paperless.config import AIConfig
from paperless.network import GuardedAsyncHTTPTransport
from paperless.network import GuardedHTTPTransport
from paperless.network import OutboundRequestBlockedError
from paperless.network import create_guarded_async_httpx_client
from paperless.network import create_guarded_httpx_client
from paperless.network import PinnedHostAsyncHTTPTransport
from paperless.network import PinnedHostHTTPTransport
from paperless.network import create_pinned_async_httpx_client
from paperless.network import create_pinned_httpx_client
from paperless.network import validate_outbound_http_url
from paperless_ai.base_model import ClassificationSuggestions
from paperless_ai.base_model import DocumentClassifierSchema
from paperless_ai.base_model import model_to_classification_suggestions
from paperless_ai.exceptions import LLMBlockedError
from paperless_ai.exceptions import LLMProviderError
from paperless_ai.exceptions import LLMTimeoutError
@@ -45,19 +43,6 @@ LLM_SYSTEM_PROMPT = (
PLACEHOLDER_API_KEY: Final = "fake"
def _find_blocked_cause(exc: BaseException) -> OutboundRequestBlockedError | None:
# The openai SDK wraps transport errors in APIConnectionError, so the
# block can sit anywhere in the __cause__ chain.
current: BaseException | None = exc
seen: set[int] = set()
while current is not None and id(current) not in seen:
if isinstance(current, OutboundRequestBlockedError):
return current
seen.add(id(current))
current = current.__cause__
return None
class AIClient:
"""
A client for interacting with an LLM backend.
@@ -78,10 +63,10 @@ class AIClient:
endpoint,
allow_internal=self.settings.llm_allow_internal_endpoints,
)
transport = GuardedHTTPTransport(
transport = PinnedHostHTTPTransport(
allow_internal=self.settings.llm_allow_internal_endpoints,
)
async_transport = GuardedAsyncHTTPTransport(
async_transport = PinnedHostAsyncHTTPTransport(
allow_internal=self.settings.llm_allow_internal_endpoints,
)
return Ollama(
@@ -90,6 +75,7 @@ class AIClient:
context_window=self.settings.llm_context_size,
request_timeout=self.settings.llm_request_timeout,
system_prompt=LLM_SYSTEM_PROMPT,
additional_kwargs=self.settings.llm_extra_params,
client=Client(
host=endpoint,
timeout=self.settings.llm_request_timeout,
@@ -108,12 +94,12 @@ class AIClient:
http_client = None
async_http_client = None
if endpoint:
http_client = create_guarded_httpx_client(
http_client = create_pinned_httpx_client(
endpoint,
allow_internal=self.settings.llm_allow_internal_endpoints,
timeout=self.settings.llm_request_timeout,
)
async_http_client = create_guarded_async_httpx_client(
async_http_client = create_pinned_async_httpx_client(
endpoint,
allow_internal=self.settings.llm_allow_internal_endpoints,
timeout=self.settings.llm_request_timeout,
@@ -126,6 +112,7 @@ class AIClient:
is_chat_model=True,
is_function_calling_model=True,
system_prompt=LLM_SYSTEM_PROMPT,
additional_kwargs=self.settings.llm_extra_params,
http_client=http_client,
async_http_client=async_http_client,
)
@@ -194,12 +181,6 @@ class AIClient:
except httpx.TimeoutException as exc:
raise LLMTimeoutError from exc
except Exception as exc:
blocked = _find_blocked_cause(exc)
if blocked is not None:
raise LLMBlockedError(
"AI backend request was blocked by the outbound request "
f"policy: {blocked}",
) from exc
if self._is_openai_timeout(exc):
raise LLMTimeoutError from exc
if self._is_provider_error(exc):
+11 -9
View File
@@ -9,10 +9,10 @@ if TYPE_CHECKING:
from documents.models import Document
from paperless.config import AIConfig
from paperless.models import LLMEmbeddingBackend
from paperless.network import GuardedAsyncHTTPTransport
from paperless.network import GuardedHTTPTransport
from paperless.network import create_guarded_async_httpx_client
from paperless.network import create_guarded_httpx_client
from paperless.network import PinnedHostAsyncHTTPTransport
from paperless.network import PinnedHostHTTPTransport
from paperless.network import create_pinned_async_httpx_client
from paperless.network import create_pinned_httpx_client
from paperless.network import validate_outbound_http_url
from paperless_ai.client import PLACEHOLDER_API_KEY
@@ -29,19 +29,21 @@ def get_embedding_model(config: AIConfig) -> "BaseEmbedding":
http_client = None
async_http_client = None
if endpoint:
http_client = create_guarded_httpx_client(
http_client = create_pinned_httpx_client(
endpoint,
allow_internal=config.llm_allow_internal_endpoints,
timeout=config.llm_request_timeout,
)
async_http_client = create_guarded_async_httpx_client(
async_http_client = create_pinned_async_httpx_client(
endpoint,
allow_internal=config.llm_allow_internal_endpoints,
timeout=config.llm_request_timeout,
)
return OpenAILikeEmbedding(
model_name=config.llm_embedding_model or "text-embedding-3-small",
api_key=config.llm_api_key or PLACEHOLDER_API_KEY,
api_key=config.llm_embedding_api_key
or config.llm_api_key
or PLACEHOLDER_API_KEY,
api_base=endpoint,
timeout=config.llm_request_timeout,
http_client=http_client,
@@ -77,14 +79,14 @@ def get_embedding_model(config: AIConfig) -> "BaseEmbedding":
embedding._client = Client(
host=endpoint,
timeout=config.llm_request_timeout,
transport=GuardedHTTPTransport(
transport=PinnedHostHTTPTransport(
allow_internal=config.llm_allow_internal_endpoints,
),
)
embedding._async_client = AsyncClient(
host=endpoint,
timeout=config.llm_request_timeout,
transport=GuardedAsyncHTTPTransport(
transport=PinnedHostAsyncHTTPTransport(
allow_internal=config.llm_allow_internal_endpoints,
),
)
-4
View File
@@ -4,7 +4,3 @@ class LLMTimeoutError(Exception):
class LLMProviderError(Exception):
"""The LLM backend rejected the request."""
class LLMBlockedError(Exception):
"""The outbound request policy refused the connection to the LLM backend."""
+33 -144
View File
@@ -1,4 +1,3 @@
import ipaddress
import json
from unittest.mock import ANY
from unittest.mock import MagicMock
@@ -10,15 +9,11 @@ import openai
import pytest
from llama_index.core.llms.llm import ToolSelection
from paperless.network import BlockReason
from paperless.network import OutboundRequestBlockedError
from paperless_ai.client import LLM_SYSTEM_PROMPT
from paperless_ai.client import PLACEHOLDER_API_KEY
from paperless_ai.client import AIClient
from paperless_ai.exceptions import LLMBlockedError
from paperless_ai.exceptions import LLMProviderError
from paperless_ai.exceptions import LLMTimeoutError
from paperless_testing.outbound import guard_of
@pytest.fixture
@@ -28,6 +23,7 @@ def mock_ai_config():
mock_config.llm_allow_internal_endpoints = True
mock_config.llm_context_size = 8192
mock_config.llm_request_timeout = 120
mock_config.llm_extra_params = {}
MockAIConfig.return_value = mock_config
yield mock_config
@@ -57,6 +53,7 @@ def test_get_llm_ollama(mock_ai_config, mock_ollama_llm):
context_window=8192,
request_timeout=120,
system_prompt=LLM_SYSTEM_PROMPT,
additional_kwargs={},
client=ANY,
async_client=ANY,
)
@@ -79,6 +76,7 @@ def test_get_llm_openai(mock_ai_config, mock_openai_llm):
is_chat_model=True,
is_function_calling_model=True,
system_prompt=LLM_SYSTEM_PROMPT,
additional_kwargs={},
http_client=ANY,
async_http_client=ANY,
)
@@ -201,6 +199,36 @@ def test_run_llm_query_openai_uses_tools(mock_ai_config, mock_openai_llm):
)
@pytest.mark.parametrize(
("backend", "llm_fixture"),
[
pytest.param("openai-like", "mock_openai_llm", id="openai-like"),
pytest.param("ollama", "mock_ollama_llm", id="ollama"),
],
)
def test_get_llm_passes_extra_params(request, mock_ai_config, backend, llm_fixture):
"""
GIVEN:
- Extra LLM params configured, e.g. for a provider that needs a
parameter we do not set ourselves
WHEN:
- The client builds the LLM
THEN:
- They are handed to the backend as additional_kwargs
"""
llm_mock = request.getfixturevalue(llm_fixture)
mock_ai_config.llm_backend = backend
mock_ai_config.llm_model = "gpt-5.6-luna"
mock_ai_config.llm_endpoint = "http://test-url"
mock_ai_config.llm_extra_params = {"reasoning_effort": "none"}
AIClient()
assert llm_mock.call_args.kwargs["additional_kwargs"] == {
"reasoning_effort": "none",
}
def test_run_llm_query_openai_timeout_raises_local_error(
mock_ai_config,
mock_openai_llm,
@@ -282,142 +310,3 @@ def test_run_llm_query_httpx_timeout_raises_local_error(
with pytest.raises(LLMTimeoutError):
client.run_llm_query("test_prompt")
class TestGuardedLLMClients:
@pytest.mark.parametrize(
("endpoint", "allow_internal"),
[
pytest.param("http://test-url", True, id="internal-allowed"),
pytest.param("http://93.184.216.34:11434", False, id="internal-blocked"),
],
)
def test_ollama_clients_are_guarded(
self,
mock_ai_config: MagicMock,
mock_ollama_llm: MagicMock,
endpoint: str,
*,
allow_internal: bool,
) -> None:
"""
GIVEN:
- The Ollama backend
WHEN:
- The LLM is built
THEN:
- Its sync and async clients use guarded transports with the setting
"""
mock_ai_config.llm_backend = "ollama"
mock_ai_config.llm_model = "test_model"
mock_ai_config.llm_endpoint = endpoint
mock_ai_config.llm_allow_internal_endpoints = allow_internal
AIClient()
kwargs = mock_ollama_llm.call_args.kwargs
assert guard_of(kwargs["client"]._client)._allow_internal is allow_internal
assert (
guard_of(kwargs["async_client"]._client)._allow_internal is allow_internal
)
@pytest.mark.parametrize(
("endpoint", "allow_internal"),
[
pytest.param("http://test-url", True, id="internal-allowed"),
pytest.param("http://93.184.216.34:8080", False, id="internal-blocked"),
],
)
def test_openai_like_clients_are_guarded(
self,
mock_ai_config: MagicMock,
mock_openai_llm: MagicMock,
endpoint: str,
*,
allow_internal: bool,
) -> None:
"""
GIVEN:
- The OpenAI-like backend with an endpoint
WHEN:
- The LLM is built
THEN:
- Its sync and async http clients use guarded transports
"""
mock_ai_config.llm_backend = "openai-like"
mock_ai_config.llm_model = "test_model"
mock_ai_config.llm_api_key = "key"
mock_ai_config.llm_endpoint = endpoint
mock_ai_config.llm_allow_internal_endpoints = allow_internal
AIClient()
kwargs = mock_openai_llm.call_args.kwargs
assert guard_of(kwargs["http_client"])._allow_internal is allow_internal
assert guard_of(kwargs["async_http_client"])._allow_internal is allow_internal
def _block() -> OutboundRequestBlockedError:
return OutboundRequestBlockedError(
host="llm.example",
port=443,
reason=BlockReason.NON_PUBLIC_ADDRESS,
address=ipaddress.ip_address("10.0.0.1"),
)
class TestBlockedLLMRequests:
def test_ollama_block_becomes_llm_blocked_error(
self,
mock_ai_config: MagicMock,
mock_ollama_llm: MagicMock,
) -> None:
"""
GIVEN:
- The Ollama backend and a connection blocked by policy
WHEN:
- An LLM query runs
THEN:
- LLMBlockedError is raised with a message, chained to the block
- The message, which tracked tasks store, names the destination but
not the resolved internal address
"""
mock_ai_config.llm_backend = "ollama"
mock_ai_config.llm_model = "test_model"
mock_ai_config.llm_endpoint = "http://test-url"
block = _block()
mock_ollama_llm.return_value.chat.side_effect = block
with pytest.raises(LLMBlockedError) as exc_info:
AIClient().run_llm_query("test_prompt")
assert exc_info.value.__cause__ is block
assert "llm.example:443" in str(exc_info.value)
assert "10.0.0.1" not in str(exc_info.value)
def test_openai_wrapped_block_becomes_llm_blocked_error(
self,
mock_ai_config: MagicMock,
mock_openai_llm: MagicMock,
) -> None:
"""
GIVEN:
- The OpenAI-like backend, whose SDK wraps the block in
APIConnectionError
WHEN:
- An LLM query runs
THEN:
- LLMBlockedError is raised
"""
mock_ai_config.llm_backend = "openai-like"
mock_ai_config.llm_model = "test_model"
mock_ai_config.llm_api_key = "key"
mock_ai_config.llm_endpoint = "http://test-url"
wrapped = openai.APIConnectionError(
request=httpx.Request("POST", "http://test-url/v1/chat/completions"),
)
wrapped.__cause__ = _block()
mock_openai_llm.return_value.chat_with_tools.side_effect = wrapped
with pytest.raises(LLMBlockedError):
AIClient().run_llm_query("test_prompt")
+18 -66
View File
@@ -1,12 +1,9 @@
from typing import TYPE_CHECKING
from typing import cast
from unittest.mock import ANY
from unittest.mock import MagicMock
from unittest.mock import patch
import pytest
from django.conf import settings
from pytest_mock import MockerFixture
from documents.models import Document
from paperless.models import LLMEmbeddingBackend
@@ -15,15 +12,12 @@ from paperless_ai.embedding import _normalize_llm_index_text
from paperless_ai.embedding import build_llm_index_text
from paperless_ai.embedding import get_configured_model_name
from paperless_ai.embedding import get_embedding_model
from paperless_testing.outbound import guard_of
if TYPE_CHECKING:
from llama_index.embeddings.ollama import OllamaEmbedding
@pytest.fixture
def mock_ai_config():
with patch("paperless_ai.embedding.AIConfig") as MockAIConfig:
MockAIConfig.return_value.llm_embedding_api_key = None
MockAIConfig.return_value.llm_embedding_endpoint = None
MockAIConfig.return_value.llm_allow_internal_endpoints = True
MockAIConfig.return_value.llm_context_size = 8192
@@ -70,6 +64,7 @@ def mock_document():
def test_get_embedding_model_openai(mock_ai_config):
mock_ai_config.return_value.llm_embedding_backend = LLMEmbeddingBackend.OPENAI_LIKE
mock_ai_config.return_value.llm_embedding_model = "text-embedding-3-small"
mock_ai_config.return_value.llm_embedding_api_key = "test_embedding_api_key"
mock_ai_config.return_value.llm_api_key = "test_api_key"
mock_ai_config.return_value.llm_endpoint = "http://test-url"
@@ -79,7 +74,7 @@ def test_get_embedding_model_openai(mock_ai_config):
model = get_embedding_model(mock_ai_config.return_value)
MockOpenAIEmbedding.assert_called_once_with(
model_name="text-embedding-3-small",
api_key="test_api_key",
api_key="test_embedding_api_key",
api_base="http://test-url",
timeout=120,
http_client=ANY,
@@ -88,6 +83,20 @@ def test_get_embedding_model_openai(mock_ai_config):
assert model == MockOpenAIEmbedding.return_value
def test_get_embedding_model_openai_falls_back_to_llm_api_key(mock_ai_config):
mock_ai_config.return_value.llm_embedding_backend = LLMEmbeddingBackend.OPENAI_LIKE
mock_ai_config.return_value.llm_embedding_model = "text-embedding-3-small"
mock_ai_config.return_value.llm_api_key = "test_api_key"
mock_ai_config.return_value.llm_endpoint = "http://test-url"
with patch(
"llama_index.embeddings.openai_like.OpenAILikeEmbedding",
) as MockOpenAIEmbedding:
get_embedding_model(mock_ai_config.return_value)
assert MockOpenAIEmbedding.call_args.kwargs["api_key"] == "test_api_key"
@pytest.mark.parametrize("configured_key", [None, ""])
def test_get_embedding_model_openai_without_api_key_sends_placeholder(
mock_ai_config,
@@ -96,6 +105,7 @@ def test_get_embedding_model_openai_without_api_key_sends_placeholder(
"""Same required key handling as the LLM client, see #13831."""
mock_ai_config.return_value.llm_embedding_backend = LLMEmbeddingBackend.OPENAI_LIKE
mock_ai_config.return_value.llm_embedding_model = "text-embedding-3-small"
mock_ai_config.return_value.llm_embedding_api_key = configured_key
mock_ai_config.return_value.llm_api_key = configured_key
mock_ai_config.return_value.llm_endpoint = "http://test-url"
@@ -290,61 +300,3 @@ def test_normalize_llm_index_text_collapses_ocr_leaders_without_joining_lines():
def test_normalize_llm_index_text_collapses_non_breaking_spaces():
assert _normalize_llm_index_text("A\u00a0........\u00a0B") == "A B"
class TestGuardedEmbeddingClients:
def test_ollama_embedding_clients_are_guarded(
self,
mocker: MockerFixture,
mock_ai_config: MagicMock,
) -> None:
"""
GIVEN:
- The Ollama embedding backend
WHEN:
- The embedding model is built
THEN:
- The clients swapped onto it use guarded transports
"""
config = mock_ai_config.return_value
config.llm_embedding_backend = LLMEmbeddingBackend.OLLAMA
config.llm_embedding_model = "embeddinggemma"
config.llm_endpoint = "http://93.184.216.34:11434"
config.llm_allow_internal_endpoints = False
mocker.patch("llama_index.embeddings.ollama.OllamaEmbedding")
model = cast("OllamaEmbedding", get_embedding_model(config))
assert guard_of(model._client._client)._allow_internal is False
assert guard_of(model._async_client._client)._allow_internal is False
def test_openai_like_embedding_clients_are_guarded(
self,
mocker: MockerFixture,
mock_ai_config: MagicMock,
) -> None:
"""
GIVEN:
- The OpenAI-like embedding backend with an endpoint
WHEN:
- The embedding model is built
THEN:
- Its http clients use guarded transports
"""
config = mock_ai_config.return_value
config.llm_embedding_backend = LLMEmbeddingBackend.OPENAI_LIKE
config.llm_embedding_model = "text-embedding-3-small"
config.llm_api_key = "key"
config.llm_endpoint = "http://93.184.216.34:8080"
config.llm_allow_internal_endpoints = False
embedding_class = mocker.patch(
"llama_index.embeddings.openai_like.OpenAILikeEmbedding",
)
get_embedding_model(config)
kwargs = embedding_class.call_args.kwargs
assert guard_of(kwargs["http_client"])._allow_internal is False
assert guard_of(kwargs["async_http_client"])._allow_internal is False
+21 -43
View File
@@ -45,11 +45,8 @@ from documents.models import Correspondent
from documents.models import PaperlessTask
from documents.parsers import is_mime_type_supported
from documents.tasks import consume_file
from paperless.network import HostResolutionError
from paperless.network import IPAddress
from paperless.network import OutboundRequestBlockedError
from paperless.network import blocked_message
from paperless.network import resolve_public_addresses
from paperless.network import is_public_ip
from paperless.network import resolve_hostname_ips
from paperless_mail.models import MailAccount
from paperless_mail.models import MailRule
from paperless_mail.models import ProcessedMail
@@ -448,34 +445,18 @@ class PinnedIMAP4(imaplib.IMAP4):
Without pinned addresses, and with the ssl_context of the matching imaplib
class, this behaves exactly like imaplib.IMAP4 / imaplib.IMAP4_SSL.
``pinned_ips`` of ``None`` means no pinning was requested and the stock
imaplib connection path is used. An empty tuple means pinning was requested
and yielded nothing, and the connection fails without opening a socket
rather than falling back to a hostname lookup.
"""
def __init__(
self,
host: str,
port: int | None,
pinned_ips: tuple[IPAddress, ...] | None,
ssl_context: ssl.SSLContext | None = None,
timeout: float | None = None,
) -> None:
def __init__(self, host, port, pinned_ips, ssl_context=None, timeout=None) -> None:
self._pinned_ips = pinned_ips
self.ssl_context = ssl_context
super().__init__(host, port, timeout=timeout)
def _connect_pinned(
self,
pinned_ips: tuple[IPAddress, ...],
timeout: float | None,
) -> socket.socket:
def _connect_pinned(self, timeout):
last_error: OSError | None = None
for ip in pinned_ips:
for ip_str in self._pinned_ips:
try:
address = (str(ip), self.port)
address = (ip_str, self.port)
if timeout is not None:
return socket.create_connection(address, timeout)
return socket.create_connection(address)
@@ -483,9 +464,9 @@ class PinnedIMAP4(imaplib.IMAP4):
last_error = e
raise last_error or OSError(f"Could not connect to {self.host}")
def _create_socket(self, timeout: float | None) -> socket.socket:
if self._pinned_ips is not None:
sock = self._connect_pinned(self._pinned_ips, timeout)
def _create_socket(self, timeout):
if self._pinned_ips:
sock = self._connect_pinned(timeout)
else:
sock = super()._create_socket(timeout)
if self.ssl_context is None:
@@ -496,12 +477,7 @@ class PinnedIMAP4(imaplib.IMAP4):
class PinnedClientMixin:
"""Builds the imaplib client against the pre-resolved addresses, if any."""
def __init__(
self,
*args,
pinned_ips: tuple[IPAddress, ...] | None,
**kwargs,
) -> None:
def __init__(self, *args, pinned_ips: list[str] | None, **kwargs) -> None:
self._pinned_ips = pinned_ips
super().__init__(*args, **kwargs)
@@ -539,20 +515,22 @@ class PinnedMailBoxStartTls(PinnedClientMixin, MailBoxStartTls):
return client
def get_mailbox(
server: str,
port: int | None,
security: int,
) -> MailBox:
def get_mailbox(server, port, security) -> MailBox:
"""
Returns the correct MailBox instance for the given configuration.
"""
pinned_ips: tuple[IPAddress, ...] | None = None
pinned_ips: list[str] | None = None
if not settings.EMAIL_ALLOW_INTERNAL_HOSTS:
try:
pinned_ips = resolve_public_addresses(server, port)
except (OutboundRequestBlockedError, HostResolutionError) as e:
raise MailError(blocked_message(e)) from e
pinned_ips = resolve_hostname_ips(server)
except ValueError as e:
raise MailError(str(e)) from e
for ip_str in pinned_ips:
if not is_public_ip(ip_str):
raise MailError(
f"Connection blocked: {server} resolves to a non-public address",
)
ssl_context = ssl.create_default_context()
if settings.EMAIL_CERTIFICATE_FILE is not None: # pragma: no cover
+23 -65
View File
@@ -1,12 +1,9 @@
import dataclasses
import ipaddress
import socket
import time
import uuid
from collections import namedtuple
from datetime import timedelta
from unittest import mock
from unittest.mock import MagicMock
import pytest
from django.contrib.auth.models import Permission
@@ -28,7 +25,6 @@ from documents.models import MatchingModel
from paperless_mail import tasks
from paperless_mail.mail import MailAccountHandler
from paperless_mail.mail import MailError
from paperless_mail.mail import PinnedIMAP4
from paperless_mail.mail import TagMailAction
from paperless_mail.mail import apply_mail_action
from paperless_mail.mail import error_callback
@@ -1569,7 +1565,12 @@ class TestMail(
("electronic", None, "invoices@mycompany.com", None, 1),
(None, "amazon", "me@myselfandi.com", None, 1),
]:
with self.subTest(f_body=f_body, f_from=f_from, f_subject=f_subject):
with self.subTest(
f_body=f_body,
f_from=f_from,
f_to=f_to,
f_subject=f_subject,
):
MailRule.objects.all().delete()
_ = MailRule.objects.create(
name="testrule3",
@@ -1810,7 +1811,7 @@ class TestPostConsumeAction(TestCase):
with (
self.assertRaises(errors.ImapToolsError),
self.assertLogs("paperless.mail", level="ERROR") as cm,
self.assertLogs("paperless_mail", level="ERROR") as cm,
):
apply_mail_action(
result=[],
@@ -1819,9 +1820,10 @@ class TestPostConsumeAction(TestCase):
message_subject=self.message_subject,
message_date=self.message_date,
)
error_str = cm.output[0]
expected_str = "Error while processing mail action during post_consume"
self.assertIn(expected_str, error_str)
error_str = cm.output[0]
expected_str = "Error while processing mail action during post_consume"
self.assertIn(expected_str, error_str)
processed_mail = ProcessedMail.objects.get(uid=self.message_uid)
self.assertEqual(processed_mail.status, "FAILED")
@@ -2049,13 +2051,10 @@ class TestMailAccountTestView(APITestCase):
self.assertEqual(response.content.decode(), "Unable to connect to server")
@override_settings(EMAIL_ALLOW_INTERNAL_HOSTS=False)
@mock.patch(
"paperless.network._getaddrinfo",
return_value=[(socket.AF_INET, socket.SOCK_STREAM, 6, "", ("127.0.0.1", 993))],
)
@mock.patch("paperless_mail.mail.resolve_hostname_ips", return_value=["127.0.0.1"])
def test_mail_account_test_view_blocks_internal_host_when_disabled(
self,
_mock_getaddrinfo: MagicMock,
_mock_resolve_hostname_ips,
) -> None:
data = {
"imap_server": "internal.example",
@@ -2212,10 +2211,10 @@ class TestGetMailboxHostPinning(TestCase):
@override_settings(EMAIL_ALLOW_INTERNAL_HOSTS=False)
@mock.patch(
"paperless_mail.mail.resolve_public_addresses",
return_value=(ipaddress.ip_address("93.184.216.34"),),
"paperless_mail.mail.resolve_hostname_ips",
return_value=["93.184.216.34"],
)
def test_connects_to_validated_ip(self, _mock_resolve: MagicMock) -> None:
def test_connects_to_validated_ip(self, _mock_resolve) -> None:
with mock.patch(
"paperless_mail.mail.socket.create_connection",
side_effect=OSError("no connection in tests"),
@@ -2232,13 +2231,10 @@ class TestGetMailboxHostPinning(TestCase):
@override_settings(EMAIL_ALLOW_INTERNAL_HOSTS=False)
@mock.patch(
"paperless_mail.mail.resolve_public_addresses",
return_value=(ipaddress.ip_address("93.184.216.34"),),
"paperless_mail.mail.resolve_hostname_ips",
return_value=["93.184.216.34"],
)
def test_ssl_pins_ip_but_keeps_hostname_for_sni(
self,
_mock_resolve: MagicMock,
) -> None:
def test_ssl_pins_ip_but_keeps_hostname_for_sni(self, _mock_resolve) -> None:
ssl_context = mock.MagicMock()
ssl_context.wrap_socket.return_value.makefile.side_effect = OSError(
"no connection in tests",
@@ -2269,51 +2265,13 @@ class TestGetMailboxHostPinning(TestCase):
@override_settings(EMAIL_ALLOW_INTERNAL_HOSTS=False)
@mock.patch(
"paperless.network._getaddrinfo",
return_value=[
(socket.AF_INET, socket.SOCK_STREAM, 6, "", ("93.184.216.34", 993)),
(socket.AF_INET, socket.SOCK_STREAM, 6, "", ("127.0.0.1", 993)),
],
"paperless_mail.mail.resolve_hostname_ips",
return_value=["93.184.216.34", "127.0.0.1"],
)
def test_blocks_when_any_resolved_address_is_internal(
self,
_mock_resolve: MagicMock,
) -> None:
"""
GIVEN:
- A mail host resolving to one public and one loopback address
- EMAIL_ALLOW_INTERNAL_HOSTS is False
WHEN:
- A mailbox is requested
THEN:
- The whole host is blocked with the existing message
"""
with self.assertRaisesMessage(
MailError,
"Connection blocked: mail.example.com resolves to a non-public address",
):
def test_blocks_when_any_resolved_address_is_internal(self, _mock_resolve) -> None:
with self.assertRaises(MailError):
get_mailbox("mail.example.com", 993, MailAccount.ImapSecurity.SSL)
def test_empty_pin_list_never_falls_back_to_hostname_lookup(self) -> None:
"""
GIVEN:
- A pinned IMAP client given an empty tuple of addresses
WHEN:
- It connects
THEN:
- It fails without opening any socket, rather than resolving the
hostname itself
"""
with (
mock.patch("paperless_mail.mail.socket.create_connection") as pinned,
mock.patch("imaplib.IMAP4._create_socket") as unpinned,
self.assertRaises(OSError),
):
PinnedIMAP4("mail.example.com", 143, ())
pinned.assert_not_called()
unpinned.assert_not_called()
class TestMailAccountProcess(APITestCase):
def setUp(self) -> None:
+4
View File
@@ -37,6 +37,7 @@ class PaperlessDirs:
logging_dir: Path
model_file: Path
media_lock: Path
share_link_bundle_dir: Path
class DirSettings(TypedDict):
@@ -54,6 +55,7 @@ class DirSettings(TypedDict):
STATIC_ROOT: Path
MODEL_FILE: Path
MEDIA_LOCK: Path
SHARE_LINK_BUNDLE_DIR: Path
def build_paperless_dirs(root: Path) -> PaperlessDirs:
@@ -75,6 +77,7 @@ def build_paperless_dirs(root: Path) -> PaperlessDirs:
logging_dir=data_dir / "log",
model_file=data_dir / "classification_model.pickle",
media_lock=media_dir / "media.lock",
share_link_bundle_dir=documents_dir / "share_link_bundles",
)
for directory in (
@@ -109,6 +112,7 @@ def dirs_settings(dirs: PaperlessDirs) -> DirSettings:
STATIC_ROOT=dirs.static_dir,
MODEL_FILE=dirs.model_file,
MEDIA_LOCK=dirs.media_lock,
SHARE_LINK_BUNDLE_DIR=dirs.share_link_bundle_dir,
)
-218
View File
@@ -1,218 +0,0 @@
"""
Real-socket helpers for tests of the outbound connection guard in
paperless.network: a local HTTP server, a per-hostname resolver fake and
spies recording which addresses were actually dialled.
The fixtures wrapping these live in the root conftest.
"""
from __future__ import annotations
import http.server
import socket
import threading
from contextlib import contextmanager
from dataclasses import dataclass
from dataclasses import field
from typing import TYPE_CHECKING
from typing import Any
from typing import cast
import anyio
import httpcore
from paperless.network import GuardedAsyncHTTPTransport
from paperless.network import GuardedHTTPTransport
from paperless.network import _GuardedAsyncBackend
from paperless.network import _GuardedSyncBackend
if TYPE_CHECKING:
from collections.abc import Iterator
from unittest.mock import MagicMock
from unittest.mock import _Call
import httpx
from pytest_mock import MockerFixture
_REAL_GETADDRINFO = socket.getaddrinfo
_REAL_AGETADDRINFO = anyio.getaddrinfo
@dataclass
class ReceivedRequest:
method: str
path: str
headers: dict[str, str]
body: bytes
@dataclass
class LocalHTTPServer:
"""State of a threaded HTTP server bound to 127.0.0.1 on an ephemeral port."""
port: int
requests: list[ReceivedRequest] = field(default_factory=list)
connections: int = 0
redirect_to: str | None = None
class _Handler(http.server.BaseHTTPRequestHandler):
# HTTP/1.1 keeps connections open, so tests can observe connection reuse.
# Every response sets Content-Length, which keep-alive requires.
protocol_version = "HTTP/1.1"
def _handle(self) -> None:
length = int(self.headers.get("Content-Length") or 0)
body = self.rfile.read(length) if length else b""
# BaseHTTPRequestHandler types server as the base socketserver.BaseServer;
# narrowing the attribute's declared type is a variance error, so the
# subclass is recovered here instead of on the class body.
server = cast("_RecordingHTTPServer", self.server)
state = server.state
state.requests.append(
ReceivedRequest(
method=self.command,
path=self.path,
headers={key.lower(): value for key, value in self.headers.items()},
body=body,
),
)
if state.redirect_to is not None:
self.send_response(302)
self.send_header("Location", state.redirect_to)
self.send_header("Content-Length", "0")
self.end_headers()
return
self.send_response(200)
self.send_header("Content-Length", "2")
self.end_headers()
self.wfile.write(b"ok")
do_GET = _handle
do_POST = _handle
def log_message(self, format: str, *args: Any) -> None:
return None
class _RecordingHTTPServer(http.server.ThreadingHTTPServer):
daemon_threads = True
def __init__(self) -> None:
super().__init__(("127.0.0.1", 0), _Handler)
self.state = LocalHTTPServer(port=self.socket.getsockname()[1])
def verify_request(self, request: Any, client_address: Any) -> bool:
self.state.connections += 1
return True
@contextmanager
def running_http_server() -> Iterator[LocalHTTPServer]:
"""Serve on 127.0.0.1 in a background thread until the block exits."""
server = _RecordingHTTPServer()
thread = threading.Thread(target=server.serve_forever, daemon=True)
thread.start()
try:
yield server.state
finally:
server.shutdown()
server.server_close()
thread.join(timeout=5)
def _addrinfo(address: str, port: int | None) -> tuple[Any, ...]:
if ":" in address:
return (socket.AF_INET6, socket.SOCK_STREAM, 6, "", (address, port or 0, 0, 0))
return (socket.AF_INET, socket.SOCK_STREAM, 6, "", (address, port or 0))
class FakeDNS:
"""
Answers the guard's resolver hooks for registered names and delegates
every other name to the real resolver. The stock httpcore backends keep
using the unpatched socket.getaddrinfo.
"""
def __init__(self) -> None:
self._answers: dict[str, list[str]] = {}
self.lookups: list[str] = []
def add(self, hostname: str, *addresses: str) -> None:
self._answers[hostname] = list(addresses)
def getaddrinfo(
self,
host: str,
port: int | None,
*args: Any,
**kwargs: Any,
) -> list[tuple[Any, ...]]:
self.lookups.append(host)
if host in self._answers:
return [_addrinfo(address, port) for address in self._answers[host]]
return list(_REAL_GETADDRINFO(host, port, *args, **kwargs))
async def agetaddrinfo(
self,
host: str,
port: int | None,
**kwargs: Any,
) -> list[tuple[Any, ...]]:
self.lookups.append(host)
if host in self._answers:
return [_addrinfo(address, port) for address in self._answers[host]]
return list(await _REAL_AGETADDRINFO(host, port, **kwargs))
def install_fake_dns(mocker: MockerFixture) -> FakeDNS:
"""Patch the guard's resolver hooks with a FakeDNS for the current test."""
dns = FakeDNS()
mocker.patch("paperless.network._getaddrinfo", new=dns.getaddrinfo)
mocker.patch("paperless.network._agetaddrinfo", new=dns.agetaddrinfo)
return dns
def _dialled_host(call: _Call) -> str:
# The spy sits on the class, so args[0] is the backend instance.
if "host" in call.kwargs:
return str(call.kwargs["host"])
return str(call.args[1])
@dataclass
class DialRecorder:
sync_spy: MagicMock
async_spy: MagicMock
def hosts(self) -> list[str]:
calls = [*self.sync_spy.call_args_list, *self.async_spy.call_args_list]
return [_dialled_host(call) for call in calls]
def install_dial_recorder(mocker: MockerFixture) -> DialRecorder:
"""Spy on the stock backends' connect_tcp for the current test."""
return DialRecorder(
sync_spy=mocker.spy(httpcore.SyncBackend, "connect_tcp"),
async_spy=mocker.spy(httpcore.AnyIOBackend, "connect_tcp"),
)
def allow_all_addresses(mocker: MockerFixture) -> None:
"""Patch the guard's public-address check to accept every address.
Loopback and other private addresses pass just like a public one, for
tests that exercise something other than the address policy itself.
"""
mocker.patch("paperless.network.is_public_ip", return_value=True)
def guard_of(
client: httpx.Client | httpx.AsyncClient,
) -> _GuardedSyncBackend | _GuardedAsyncBackend:
"""Return the guard installed on a client's transport."""
transport = client._transport
assert isinstance(transport, GuardedHTTPTransport | GuardedAsyncHTTPTransport)
backend = transport._pool._network_backend
assert isinstance(backend, _GuardedSyncBackend | _GuardedAsyncBackend)
return backend
Generated
+57 -53
View File
@@ -220,7 +220,7 @@ wheels = [
[[package]]
name = "autobahn"
version = "25.12.2"
version = "26.7.1"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "cbor2" },
@@ -228,24 +228,35 @@ dependencies = [
{ name = "cryptography" },
{ name = "hyperlink" },
{ name = "msgpack", marker = "platform_python_implementation == 'CPython'" },
{ name = "py-ubjson" },
{ name = "txaio" },
{ name = "u-msgpack-python", marker = "platform_python_implementation != 'CPython'" },
{ name = "ujson" },
]
sdist = { url = "https://files.pythonhosted.org/packages/54/d5/9adf0f5b9eb244e58e898e9f3db4b00c09835ef4b6c37d491886e0376b4f/autobahn-25.12.2.tar.gz", hash = "sha256:754c06a54753aeb7e8d10c5cbf03249ad9e2a1a32bca8be02865c6f00628a98c", size = 13893652, upload-time = "2025-12-15T11:13:19.086Z" }
sdist = { url = "https://files.pythonhosted.org/packages/de/73/f109f563c27e048e45d135d81af19e6ca391e24905550b06bd1c9d674c57/autobahn-26.7.1.tar.gz", hash = "sha256:c6949a2c6eb95fb1c218837dbda0a59abbbebafb8b11098551c01a7061dfd245", size = 14056542, upload-time = "2026-07-15T19:14:01.246Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/ae/23/923e4f11dc9d12b9f5a014f36d591c479d623d54dda3bdcbd688cd12f052/autobahn-25.12.2-cp311-cp311-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:16df879672c60f1f3fe452138c80f0fd221b3cb2ee5a14390c80f33b994104c1", size = 2053413, upload-time = "2025-12-15T11:12:58.167Z" },
{ url = "https://files.pythonhosted.org/packages/b3/0d/3d39637a1e32f555ce5fabec4a723a035556ef918b14140faea05e7de902/autobahn-25.12.2-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:6ffe28048ef96eb0f925f24c2569bd72332e120f4cb31cd6c40dd66718a5f85e", size = 2224850, upload-time = "2025-12-15T11:13:00.089Z" },
{ url = "https://files.pythonhosted.org/packages/64/8d/36452c06cbcad6d04587aeb87dfa987ef94be4a427b9f2155783d166bd97/autobahn-25.12.2-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:220748f21e91bd4a538d2d3de640cc17ee30b79f1c04a6c3dcdef321d531ee1c", size = 2225453, upload-time = "2025-12-15T11:13:02.865Z" },
{ url = "https://files.pythonhosted.org/packages/83/30/ef9c47038e4e9257319d6e1b87668b3df360a0c488d66ccff9d11aaff6ba/autobahn-25.12.2-cp313-cp313-macosx_15_0_arm64.whl", hash = "sha256:bc17f6cab9438156d2701c293c76fd02a144f9be0a992c065dfee1935ce4845b", size = 1960447, upload-time = "2025-12-15T11:13:05.007Z" },
{ url = "https://files.pythonhosted.org/packages/e2/e4/f3d5cb70bc0b9b5523d940734b2e0a251510d051a50d2e723f321e890859/autobahn-25.12.2-cp313-cp313-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b5297a782fc7d0a26842438ef1342549ceee29496cda52672ac44635c79eeb94", size = 2053955, upload-time = "2025-12-15T11:13:06.052Z" },
{ url = "https://files.pythonhosted.org/packages/ea/49/4e592a19ae58fd9c796821a882b22598fac295ede50f899cc9d14a0282b6/autobahn-25.12.2-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c0c3f1d5dafda52f8dc962ab583b6f3473b7b7186cab082d05372ed43a8261a5", size = 2225441, upload-time = "2025-12-15T11:13:07.527Z" },
{ url = "https://files.pythonhosted.org/packages/54/b7/0a0e3ecb2af7e452f5f359d19bdc647cbc8658f3f498bfa3bf8545cf4768/autobahn-25.12.2-cp314-cp314-macosx_15_0_arm64.whl", hash = "sha256:c840ee136bfaf6560467160129b0b25a0e33c9a51e2b251e98c5474f27583915", size = 1960463, upload-time = "2025-12-15T11:13:10.183Z" },
{ url = "https://files.pythonhosted.org/packages/19/8b/4215ac49d6b793b592fb08698f3a0e21a59eb3520be7f7ed288fcb52d919/autobahn-25.12.2-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9abda5cf817c0f8a19a55a67a031adf2fc70ed351719b5bd9e6fa0f5f4bc8f89", size = 2225590, upload-time = "2025-12-15T11:13:11.367Z" },
{ url = "https://files.pythonhosted.org/packages/d6/99/b4a3da42471d3ec36e2dca0c1a5368a079fed9f73b159ce3f049c4a4983b/autobahn-25.12.2-pp311-pypy311_pp73-macosx_15_0_arm64.whl", hash = "sha256:0c226329ddec154c6f3b491ea3e4713035f0326c96ebfd6b305bf90f27a2fba1", size = 1955357, upload-time = "2025-12-15T11:13:13.581Z" },
{ url = "https://files.pythonhosted.org/packages/89/81/67f19dd7395a9f1123a1f071314f8d1c4879c1869adeb8d99a236e756ac0/autobahn-25.12.2-pp311-pypy311_pp73-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:5f079393a7626eb448c8accf21151f5f206d02f8e9cee4313d62a5ca30a3aaed", size = 623173, upload-time = "2025-12-15T11:13:14.945Z" },
{ url = "https://files.pythonhosted.org/packages/71/eb/857eab3d25e3b9cc9e7e741d6193808ad91de0befb38cf10658bd339c205/autobahn-25.12.2-pp311-pypy311_pp73-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9b3a6c7d54a9f0434a435d88b86555510e5d0a84aa87042e292f29f707cab237", size = 2178008, upload-time = "2025-12-15T11:13:15.881Z" },
{ url = "https://files.pythonhosted.org/packages/44/8c/381cdcab8016df2177adc93d25f84ca3a5fb8f8be4f9d784336416c7bee8/autobahn-26.7.1-cp311-cp311-macosx_15_0_arm64.whl", hash = "sha256:3fe80550707f0affb5cb10f3e0f66ec7e6e52abb29edc66dd76734c2d7d51bf4", size = 1997747, upload-time = "2026-07-15T19:13:21.998Z" },
{ url = "https://files.pythonhosted.org/packages/79/a9/9293c6c6bc8970f42c9675942de78f306e18eafa47edba52fd27f9dc71bd/autobahn-26.7.1-cp311-cp311-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:00fb9acd8775eaa0e272f36b76db903f10de56478f6a72f0bd07ee882ae1f2b8", size = 2082284, upload-time = "2026-07-15T19:13:23.582Z" },
{ url = "https://files.pythonhosted.org/packages/a9/ba/7396cb42a9c59df20c350ea05f75e6f25f582b474dee82b8e32823b2711e/autobahn-26.7.1-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:97c9674eddd55ad3ebd733824789175e5fb90c88afd523de507569ba0fcd6853", size = 2254260, upload-time = "2026-07-15T19:13:24.894Z" },
{ url = "https://files.pythonhosted.org/packages/b7/64/19753442770662ff45c4fe48db6345ac6fa3100fbbd989241c074e38ea6f/autobahn-26.7.1-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:30fa714de5c9903ef64084d3a938d8a3bac0bb42f1532d5de22f34b04a1c4819", size = 3173653, upload-time = "2026-07-15T19:13:26.261Z" },
{ url = "https://files.pythonhosted.org/packages/04/a4/b690f272427acf1e8ea03b146e559dc67ade10dd4e0cccacc1d4c011b141/autobahn-26.7.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:c3362197f3b9d5b0df7f3365bd00dedba7ee8b649d652941377abe690d1c8b14", size = 3402880, upload-time = "2026-07-15T19:13:27.683Z" },
{ url = "https://files.pythonhosted.org/packages/6e/23/0769ef39e1cfb0bec15bacdd7f407aaedfda14c0ca3f7e818b856f2ed1a1/autobahn-26.7.1-cp312-cp312-macosx_15_0_arm64.whl", hash = "sha256:6c9013e9aa9ea8a561c89d7be2709546b51fc7ac8fdf6cd71bc12a634672d9a8", size = 2000605, upload-time = "2026-07-15T19:13:30.391Z" },
{ url = "https://files.pythonhosted.org/packages/f9/ef/26833f38ecf3aef3ff0aa09feb12f5d472f7370104148a6b78e3c7afc286/autobahn-26.7.1-cp312-cp312-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:abd9ebe577dd1030f9a0c41d20dc00eca90d5cf338531abcb510d978825feeea", size = 2082844, upload-time = "2026-07-15T19:13:31.559Z" },
{ url = "https://files.pythonhosted.org/packages/b2/4c/00553ee9d57ee11df47bc9867d120cfe721a73bec0b58fb3a3b91cd7c797/autobahn-26.7.1-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:de491baa4cf52fb6d7f542e445c72d94ce52dc7aabb47b0b4d5191e452dd2c74", size = 2254852, upload-time = "2026-07-15T19:13:32.8Z" },
{ url = "https://files.pythonhosted.org/packages/c3/ca/7884f6ffb8410882df98cb939dea24225dd79e4f091ceb59f4b826e54f2f/autobahn-26.7.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:9723561c820ed032fe5a8f1530cb6d5f91dc595eea6009f3b2de7044a3df892a", size = 3174235, upload-time = "2026-07-15T19:13:34.134Z" },
{ url = "https://files.pythonhosted.org/packages/1f/e7/c6704e8f6bef3aa552a851a34908a06d91317d35ca55ec04b5db14385c33/autobahn-26.7.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:e81c86cf41adca8a56ca5621ecd9ba40037f6d90d338c334bd529c8ac94bd7b6", size = 3403687, upload-time = "2026-07-15T19:13:35.633Z" },
{ url = "https://files.pythonhosted.org/packages/36/92/2f6e57d9f9e6b86b9db362f58aaa6cfeadc2f3a6901ec95aab27ef232b5c/autobahn-26.7.1-cp313-cp313-macosx_15_0_arm64.whl", hash = "sha256:2ce48214b28f73338fabe0c7fd13d222cfab9e1dd2ef11660293522f64e76727", size = 1987052, upload-time = "2026-07-15T19:13:38.543Z" },
{ url = "https://files.pythonhosted.org/packages/38/6d/f170134468e276fa9ea57eb1ae41f9cc0dcd0228e9d501f370fa50c0ee31/autobahn-26.7.1-cp313-cp313-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a5f285dce9b3dff3eb2ef6c818ac8ede24d96bd1edac340855170fc9825c38a9", size = 2082813, upload-time = "2026-07-15T19:13:39.686Z" },
{ url = "https://files.pythonhosted.org/packages/0d/ce/b735fa933e9ba4fa8c3f9aa9ae68b4e2d4aadb0a92b38ade30e37a7d4795/autobahn-26.7.1-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:66ab6e034e54f8c473df1a6b8031a3db46c16deebaf0c9b66db3e9137d2fab5e", size = 2254818, upload-time = "2026-07-15T19:13:40.951Z" },
{ url = "https://files.pythonhosted.org/packages/df/3e/57855f4f52aa0ee64c6d8210637d0d9847de4c1082e03ddd2ffafd853d2d/autobahn-26.7.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:20b3eab7d483e93278f9d7345592eb6b465883a6e88c2823aad13c3f943452db", size = 682494, upload-time = "2026-07-15T19:13:42.193Z" },
{ url = "https://files.pythonhosted.org/packages/3b/3c/3944f17dd2a06aee7d0d9f1c37b5a94518434d13ba2c38e738d33ca10daf/autobahn-26.7.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:cd840524ff190aee695a58e8acd8740ee66b4a0e6a58ccb41416ed2cbe48d43f", size = 3403666, upload-time = "2026-07-15T19:13:43.508Z" },
{ url = "https://files.pythonhosted.org/packages/51/3e/200471878093a502f8e8078c1ca19fd82acc68ae9ac363e395170da6dbe2/autobahn-26.7.1-cp314-cp314-macosx_15_0_arm64.whl", hash = "sha256:f3d1be925e3fb33fff5280c1bd02027047519812c400d3efa5477d3968686c94", size = 1987070, upload-time = "2026-07-15T19:13:47.112Z" },
{ url = "https://files.pythonhosted.org/packages/61/d1/704f881fd2c52b056dc0f14e6d0d640b1f3ff43f3b84cf85d3631e3243f4/autobahn-26.7.1-cp314-cp314-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:44f8094b0c0fa29a4963ce12130b7932469f89afa0242ff858d2b26541a81005", size = 2082939, upload-time = "2026-07-15T19:13:48.569Z" },
{ url = "https://files.pythonhosted.org/packages/a8/27/84e76aec7abbcb502d4cd34ef5c859eaa19a3b707cda68ab7ce68478dd92/autobahn-26.7.1-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a0fcf9c3ff6b9b2bc85d6e1814a94d1d941d62f7df36d350b523d77df85d66ea", size = 2254983, upload-time = "2026-07-15T19:13:49.802Z" },
{ url = "https://files.pythonhosted.org/packages/e6/81/a810732a10342c5d6b90d19f83fa2bc9b6126e7c0cda7c4df867e311aa2e/autobahn-26.7.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:4f82e5a113f6c1ff14cec99aa411f7da8fceec3dcd4647d7ebdfc0278811e14d", size = 3174295, upload-time = "2026-07-15T19:13:51.227Z" },
{ url = "https://files.pythonhosted.org/packages/a6/c6/4886fdaecfeda013e085288a9d83bad6f1ded9995b8088ca933d9ec37201/autobahn-26.7.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:919309cbe41b28b0a3028e7c6fed52aca9fd21639619f372d3270c44341b6bdc", size = 3403713, upload-time = "2026-07-15T19:13:52.744Z" },
{ url = "https://files.pythonhosted.org/packages/94/14/6485c29ad06a6bd7b3017558f99f6f89dcaa6ef6641930b25d9247adba4c/autobahn-26.7.1-pp311-pypy311_pp73-macosx_15_0_arm64.whl", hash = "sha256:9088acf790caf8cfd86590cb2b749279256ee210198f41d9858a38d1346e56c9", size = 1981962, upload-time = "2026-07-15T19:13:55.79Z" },
{ url = "https://files.pythonhosted.org/packages/8d/46/cb6d09604417beacdf485b414a05efa18511b0e78ac5451b3655bee711fa/autobahn-26.7.1-pp311-pypy311_pp73-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:ea4548ee15c6bdf8aa0a1e81bf47b42db350f2bef69f83bace27a95ed0d21276", size = 655967, upload-time = "2026-07-15T19:13:57.19Z" },
{ url = "https://files.pythonhosted.org/packages/00/d9/b846bc5a37f25ac147879d6451c466968a54efbf9d0467f0732f37c6f3f3/autobahn-26.7.1-pp311-pypy311_pp73-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d4ee0fe13a5218831d60863becd8c3cf6558e6c5ccc5d9d0e722012bdb1459bf", size = 2207401, upload-time = "2026-07-15T19:13:58.392Z" },
]
[[package]]
@@ -2874,7 +2885,6 @@ name = "paperless-ngx"
version = "3.2.1"
source = { virtual = "." }
dependencies = [
{ name = "anyio" },
{ name = "azure-ai-documentintelligence" },
{ name = "babel" },
{ name = "bleach" },
@@ -2903,8 +2913,6 @@ dependencies = [
{ name = "filelock" },
{ name = "flower" },
{ name = "gotenberg-client", extra = ["httpx"] },
{ name = "httpcore" },
{ name = "httpx" },
{ name = "httpx-oauth" },
{ name = "ijson" },
{ name = "imap-tools" },
@@ -3029,7 +3037,6 @@ typing = [
[package.metadata]
requires-dist = [
{ name = "anyio", specifier = ">=4.12" },
{ name = "azure-ai-documentintelligence", specifier = ">=1.0.2" },
{ name = "babel", specifier = ">=2.17" },
{ name = "bleach", specifier = "~=6.4.0" },
@@ -3059,8 +3066,6 @@ requires-dist = [
{ name = "flower", specifier = ">=2.0.1,<2.2" },
{ name = "gotenberg-client", extras = ["httpx"], specifier = "~=1.0" },
{ name = "granian", extras = ["uvloop"], marker = "extra == 'webserver'", specifier = ">=2.7,<2.9" },
{ name = "httpcore", specifier = "~=1.0.9" },
{ name = "httpx", specifier = "~=0.28.1" },
{ name = "httpx-oauth", specifier = "~=0.17" },
{ name = "ijson", specifier = ">=3.5.1" },
{ name = "imap-tools", specifier = ">=1.14,<1.16" },
@@ -3217,40 +3222,45 @@ wheels = [
[[package]]
name = "pikepdf"
version = "10.2.0"
version = "10.13.0.post1"
source = { registry = "https://pypi.org/simple" }
dependencies = [
{ name = "deprecated" },
{ name = "lxml" },
{ name = "packaging" },
{ name = "pillow" },
]
sdist = { url = "https://files.pythonhosted.org/packages/6e/e9/a1462d6160805ca80c8f4aafc941aaf410a92d0fcc683706e94f499c2fac/pikepdf-10.2.0.tar.gz", hash = "sha256:0f398b0daeb2ffd2358f75c06f1dd47b9ba76f1a77dfe938cccf7080c58227d7", size = 4568506, upload-time = "2026-01-09T22:54:25.847Z" }
sdist = { url = "https://files.pythonhosted.org/packages/1c/0e/6e74dd213537b71c945743a4b3112dbb430896ad68b8a6ad22e4468455d4/pikepdf-10.13.0.post1.tar.gz", hash = "sha256:4b73f926ebae81f04bf14527af330bd00bb268be767e0f189f7c4c3e4ad7ae0a", size = 4973186, upload-time = "2026-09-05T06:49:20.825Z" }
wheels = [
{ url = "https://files.pythonhosted.org/packages/10/dc/aa7293763b603a9080ffab7ab87c7b571d637a389e9fb2ba839b864ca283/pikepdf-10.2.0-cp311-cp311-macosx_14_0_arm64.whl", hash = "sha256:fb93732127d5183a91300af39e1cda5ded309e8439daec93536331a472b5e190", size = 4727891, upload-time = "2026-01-09T22:53:26.262Z" },
{ url = "https://files.pythonhosted.org/packages/d2/dc/700c31f2c14f94d92483b10e1918390948ed20f6f572d82beb78ac5f94d0/pikepdf-10.2.0-cp311-cp311-macosx_15_0_x86_64.whl", hash = "sha256:ab7bd4629539cf2136a799dc3eaa2dfda59937035a97b0c5e22a7a3a4033cc49", size = 5030510, upload-time = "2026-01-09T22:53:28.023Z" },
{ url = "https://files.pythonhosted.org/packages/23/46/dc63364b05aa1913f2d7480cad62676bfb473065ba4b02d314dfd482f7dd/pikepdf-10.2.0-cp311-cp311-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:2f5623a5ba456d69dfeb86dc3bb3ec31ec1d120382d8c24804d1b430fce715ea", size = 2439498, upload-time = "2026-01-09T22:53:30.122Z" },
{ url = "https://files.pythonhosted.org/packages/59/b6/1f9b8ca588fd34d9e3df49a80c62016e0b42ce6e580146c46d9728fdb6e8/pikepdf-10.2.0-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:dec4d12f294df378d122ae441c27c1e76fb0d15b1e9d7374ae70c26604559bab", size = 2666945, upload-time = "2026-01-09T22:53:32.309Z" },
{ url = "https://files.pythonhosted.org/packages/c1/31/b1e61fac59f0b807edde655a821ff83bb041ae1500234c52ed1a2403c44a/pikepdf-10.2.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:c0cebe3235232f1bd3c5f7956218ce92241c94223cb80eba837d372a40c61765", size = 3638109, upload-time = "2026-01-09T22:53:34.144Z" },
{ url = "https://files.pythonhosted.org/packages/02/e9/a99bbf503c9d55e54553edff84ec67cac49d335fc33f3d5516c4746b6340/pikepdf-10.2.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:d0908e845c9140e245ad89a19fdfc6e5a6d82fcb505b8cc2c0ce81439ac4f064", size = 3829538, upload-time = "2026-01-09T22:53:36.341Z" },
{ url = "https://files.pythonhosted.org/packages/73/18/598383493a0f0f0c4eecd09b8fe06dddb9d326a89e2623a134d43e051485/pikepdf-10.2.0-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:18c35d00baff72bfae82d67028bedb02ea2b208e1af5545c23cd681f2487a279", size = 4737716, upload-time = "2026-01-09T22:53:39.973Z" },
{ url = "https://files.pythonhosted.org/packages/a9/f5/bec04784ba07d44f03b52ea524bcb7409bf7185ee8abec7ae29e3ac9e9ae/pikepdf-10.2.0-cp312-cp312-macosx_15_0_x86_64.whl", hash = "sha256:dd849d033b95de15965c095ebc4d78983099a11bb7b7897801dfaf3cb4083a35", size = 5042152, upload-time = "2026-01-09T22:53:41.808Z" },
{ url = "https://files.pythonhosted.org/packages/38/3e/148b3c8e101c8ac3a33f41e86c5739413575495e471bce45ee228aafcbd6/pikepdf-10.2.0-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e9910efdc7907af3da9e7b2a125a1f67d512165ffa623f62825deeb642669a7a", size = 2445796, upload-time = "2026-01-09T22:53:44.027Z" },
{ url = "https://files.pythonhosted.org/packages/c2/ef/b06f8fd68c34fed631cb8e3520dd955e59987de0eee6960dbc94bed11711/pikepdf-10.2.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:f0ec947e6429d7a3306153d32a0142462fdd8f905c5fe08c8a8e8c53b9c28a5c", size = 2693908, upload-time = "2026-01-09T22:53:46.039Z" },
{ url = "https://files.pythonhosted.org/packages/3e/bf/e5c40e9210e2ae8da7cad2cf6ae7d1db3b63a2916e6040645958e9ab4054/pikepdf-10.2.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:b6383219a1cd31400403a69737a4e2a0c5d2a2c4cb9f380bcf45e33e8de802ea", size = 3643423, upload-time = "2026-01-09T22:53:48.333Z" },
{ url = "https://files.pythonhosted.org/packages/d4/1b/969dfb29dc9fd7b82fa7bc065df498e8a3e7ddb81e982140634ee539a8db/pikepdf-10.2.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:46d2f9ef5a84949bfc11152a323558f94cf85d9d97e9c510c061c7f803028f3f", size = 3854816, upload-time = "2026-01-09T22:53:50.158Z" },
{ url = "https://files.pythonhosted.org/packages/f4/c5/e6f9e3407dd73ec570000a64747ff84e2f57b06b0477d1da6eaca5038162/pikepdf-10.2.0-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:09ff28d1de7fc7711a7ef8dfc40396d9243b64ee24c37cd1ab2a9f9827895caa", size = 4737680, upload-time = "2026-01-09T22:53:54.905Z" },
{ url = "https://files.pythonhosted.org/packages/eb/de/dffb785235ac2d930db86b215c1848d7258e625fa1949dd0633f8b72ab0a/pikepdf-10.2.0-cp313-cp313-macosx_15_0_x86_64.whl", hash = "sha256:62348b66e1401a4db0c64976b72dd74bb1a9eb3a33007a661500f4f8a64436bd", size = 5042150, upload-time = "2026-01-09T22:53:57.722Z" },
{ url = "https://files.pythonhosted.org/packages/1a/f0/4d883f57304d98650ade30a8c73fe593582d9afd9a7dada1f5f3f4cce362/pikepdf-10.2.0-cp313-cp313-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6b9e91780cb9ea3c6a350ffbcf03d5d95c30084d238afbd1d4b927cdb9e3649d", size = 2445446, upload-time = "2026-01-09T22:53:59.993Z" },
{ url = "https://files.pythonhosted.org/packages/62/65/ffe2555812a152d616accacea7c1c617c27a75590379ea7d9cc3a26bd92d/pikepdf-10.2.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:52360a49a22e9353ec9a08ff5713cec8aacaf3ef960c704bc0a89ca8f050bdad", size = 2696242, upload-time = "2026-01-09T22:54:01.687Z" },
{ url = "https://files.pythonhosted.org/packages/ed/8c/2f937b0e2867cd48b523122e08753571fc9847978e239d7b5db9bd46879c/pikepdf-10.2.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:4ba4c1046939eb22c24c396deb37f8e0500caaa66b73114be55377d5554b4167", size = 3643730, upload-time = "2026-01-09T22:54:04.177Z" },
{ url = "https://files.pythonhosted.org/packages/2e/17/f2919e4085c399e938bb945ea712dea70b3849e17cae6403f0cc1100e9ef/pikepdf-10.2.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:3dcd8957a08e0a47f7a138904dca8cf73962fa17a096a47cd8bc33eb83a4f0a7", size = 3856645, upload-time = "2026-01-09T22:54:07.887Z" },
{ url = "https://files.pythonhosted.org/packages/1d/6e/846902abe8286d3b4ab70893e9ffbeec99aadd93ba1536cf471b222bb910/pikepdf-10.2.0-cp314-cp314-macosx_14_0_arm64.whl", hash = "sha256:194c9a81ecb49e425a5cd5162621270b5e42cf05709d87eac018bd6f9ce98f80", size = 4733930, upload-time = "2026-01-09T22:54:11.931Z" },
{ url = "https://files.pythonhosted.org/packages/8b/6d/abdbb794d2a512d4e828ef2014cc47ca263ad3fbd1b65f25f791b9c0bb1e/pikepdf-10.2.0-cp314-cp314-macosx_15_0_x86_64.whl", hash = "sha256:5adcf87dbfff4e1cd0a850db487274f474c94a6bf6347f3842c53da8d0eaa8df", size = 5042477, upload-time = "2026-01-09T22:54:13.708Z" },
{ url = "https://files.pythonhosted.org/packages/52/6c/6c42694fe1574a37aa2a40b4ba29a6713b4226436155ef7aa0bef649c117/pikepdf-10.2.0-cp314-cp314-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5de3cecbb35c4bc651e9326932974217be1d450d4a9840d77a592062eb507e27", size = 2448419, upload-time = "2026-01-09T22:54:15.6Z" },
{ url = "https://files.pythonhosted.org/packages/45/f4/aca3286aa37ace581afc8e3e0644a0cc55b9f9ceb31f28219d12ca11536c/pikepdf-10.2.0-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:77868fd25182a45a4f3dec3c461aea8c696ef9565894c5cde4394bc8c32fb069", size = 2697600, upload-time = "2026-01-09T22:54:18.123Z" },
{ url = "https://files.pythonhosted.org/packages/eb/a6/9135f9f0189634de61410573a0712d849e0157e3902e6b867339cc7dbf1b/pikepdf-10.2.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:a9a10e15e2f4d0bba36a2b4328342d00eff1a5a31399e1d1a93483c70d3c2b0e", size = 3647720, upload-time = "2026-01-09T22:54:20.158Z" },
{ url = "https://files.pythonhosted.org/packages/83/60/f282077773a3321fad4cbfb16fe73ee3f8dd93b408df65c24779f12227c5/pikepdf-10.2.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:a8f80ecf00fb15a760f218432a1046e7797cd14eaa6ccb52c8814ae8852745d8", size = 3859133, upload-time = "2026-01-09T22:54:22.358Z" },
{ url = "https://files.pythonhosted.org/packages/4d/c1/48c9c0ed2ed88ca5d9cdd7f16075385a05a812d781b61bfa7f2d5182b247/pikepdf-10.13.0.post1-cp311-cp311-macosx_14_0_arm64.whl", hash = "sha256:98a7305e330f797da02b543d3ad57a134c4a14c6ec6f8d86d91aa9dd130c425b", size = 1846300, upload-time = "2026-09-05T06:48:11.563Z" },
{ url = "https://files.pythonhosted.org/packages/e9/20/484a3a61664132dc8c4bd97e0b8291fa79f9f4a3b1e1ffd5b67ac41ed98a/pikepdf-10.13.0.post1-cp311-cp311-macosx_15_0_x86_64.whl", hash = "sha256:f3dedd02795626f17ee42d5c02ec4ec94e28aa47046ef430d4478454a8fbd07f", size = 1944179, upload-time = "2026-09-05T06:48:14.01Z" },
{ url = "https://files.pythonhosted.org/packages/66/38/797df7d60352fc5ec3943c425acaa15d032cd2e673b516861f449536db57/pikepdf-10.13.0.post1-cp311-cp311-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:7ba09ef5a5f26e38ee558d2a08223fee08a5ef1868962ae2d8590d4de3c8c92f", size = 2105054, upload-time = "2026-09-05T06:48:15.547Z" },
{ url = "https://files.pythonhosted.org/packages/a8/8a/1f003558c5c05cecf182af775839ad674fe23e06b314d0219b9d8422680a/pikepdf-10.13.0.post1-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:365b94f2be7e2857c6cb5445b56dc52dc7417ba9f06c8a4282d9f521cb2d0fb8", size = 2308617, upload-time = "2026-09-05T06:48:17.216Z" },
{ url = "https://files.pythonhosted.org/packages/55/5b/0e7193ee8c7ca5b15f478918033a0fa78917b01249e1d4a4a65754644cd3/pikepdf-10.13.0.post1-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:f515a31c76cce043bbb7b781e77a343a4e26fa8f520ba337d30ddea0f7a0ce50", size = 3742337, upload-time = "2026-09-05T06:48:18.971Z" },
{ url = "https://files.pythonhosted.org/packages/b6/f5/519e8728c04d05dcbd44d03b266a3f6acf8de3a0a6ed5ec0fa39ececddd7/pikepdf-10.13.0.post1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:b63577c44fedf7ed6b971076f7c7ed0ff95a8bac627ac93b79e8b542214861a7", size = 3952988, upload-time = "2026-09-05T06:48:20.984Z" },
{ url = "https://files.pythonhosted.org/packages/6c/a5/598e72c72ed46046e297f15763dffda88424870724a4a22f599b815cb774/pikepdf-10.13.0.post1-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:2c6e83f8a1828ec79cdec4df8cc07209eaf10ed7e4f5a90a7356b254bacc07d5", size = 1845515, upload-time = "2026-09-05T06:48:24.477Z" },
{ url = "https://files.pythonhosted.org/packages/f9/b3/29691a5e9ee915357c081730d8cc02f35f19b4155857556fbe562a4d83ba/pikepdf-10.13.0.post1-cp312-cp312-macosx_15_0_x86_64.whl", hash = "sha256:f2463f650efab46905b9e279f5c776faf96c65bb45c1acf9f2de0e8a6eec5fb7", size = 1944755, upload-time = "2026-09-05T06:48:26.332Z" },
{ url = "https://files.pythonhosted.org/packages/04/4e/201f553b9405424d7aefa3be997f0a1c787ac3a089a844f1fc42f412fe0e/pikepdf-10.13.0.post1-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9613505f5b22203465d4224a3fd8cf69876ce8278442c6478ac6849f54724a30", size = 2102202, upload-time = "2026-09-05T06:48:28.816Z" },
{ url = "https://files.pythonhosted.org/packages/3e/a3/bd7e7b321e8bfe4b8530da57d12c557a259bbd4b40e739960a1f2ea507cb/pikepdf-10.13.0.post1-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:01f80ca046d984752cf6f08093debc193884bee91c01c104aa0236767711a20f", size = 2307194, upload-time = "2026-09-05T06:48:31.558Z" },
{ url = "https://files.pythonhosted.org/packages/23/b0/ca630f56015dfc6c4c8c81fdeb5f5099e7fca354f54a41dfe7f1382a5316/pikepdf-10.13.0.post1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:1f76fcbe5d86f2ae6f231ba542cd04793d4c89e75bd5f62526b7412926ff2100", size = 3740157, upload-time = "2026-09-05T06:48:33.285Z" },
{ url = "https://files.pythonhosted.org/packages/be/48/7a84adc2fd14ec35e4b3d007575914de1ce0518e8c69b35ebee62fa2b142/pikepdf-10.13.0.post1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:092a9bf15739e931ecab15ec3baee5d9629dad90ea4e42b779f6b439d2d1e462", size = 3951601, upload-time = "2026-09-05T06:48:35.468Z" },
{ url = "https://files.pythonhosted.org/packages/10/f4/3636368760840cbc3ee512330024dd6f518d583c1bbbb1b551ca8e18f5e8/pikepdf-10.13.0.post1-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:87141ada970386ff6640db54f0bda734d3bde7960d3ba04a76768b48e25028ca", size = 1845524, upload-time = "2026-09-05T06:48:39.309Z" },
{ url = "https://files.pythonhosted.org/packages/ce/dc/7bbfba253a0394a237b81be371a64f904f99636d579ec78ef5d92025fd2d/pikepdf-10.13.0.post1-cp313-cp313-macosx_15_0_x86_64.whl", hash = "sha256:55e53b4d8a4b1700f686f76e3a68411e421e962a4c8b1b90d00aab3f3e494a55", size = 1944829, upload-time = "2026-09-05T06:48:41.601Z" },
{ url = "https://files.pythonhosted.org/packages/b3/a3/10367bfb93501a151cbb96b6973f7c049fef04040aea35cf6cedf579c3e4/pikepdf-10.13.0.post1-cp313-cp313-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8fb8f82dc43056a4b4f891e78ee1db4e3ced75ba3e87b836f8e28c8771228928", size = 2102269, upload-time = "2026-09-05T06:48:43.222Z" },
{ url = "https://files.pythonhosted.org/packages/2d/bd/a68b5d9b4aef4d4b9c374cfdfd941623f52303d31adea13554568df42abe/pikepdf-10.13.0.post1-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d3b58ccb30b93ba400e6a6a83315b4830eea49f6f19e18d78241fe6b1c49fec2", size = 2307160, upload-time = "2026-09-05T06:48:45.192Z" },
{ url = "https://files.pythonhosted.org/packages/9a/59/47bd86d9e338d301c28416d52d0e154a0d6322cf6b957c9df5f3d7828aa2/pikepdf-10.13.0.post1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:6b038cd5bcbb6c1952bcc271695eaf24c4199d72e45606ee5e467f837760820e", size = 3739722, upload-time = "2026-09-05T06:48:47.105Z" },
{ url = "https://files.pythonhosted.org/packages/49/40/87fb6dddc9dce110429c72449e942174fd500f6fc4c9ff518a6b73057aa7/pikepdf-10.13.0.post1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:b7b0cbb135de32ec3f41651a08ab294e3c18ae9fec32516a48d27e64a53a47f0", size = 3951549, upload-time = "2026-09-05T06:48:49.263Z" },
{ url = "https://files.pythonhosted.org/packages/14/0e/86897bf5325824c1f2d9d8be89839baf11011b5392d92620cb0273fb9af5/pikepdf-10.13.0.post1-cp314-abi3-macosx_14_0_arm64.whl", hash = "sha256:51fae4a4a3c6549aa4c405896ff7010f3e43e0c4f407c0bcee071ef13d271202", size = 1845245, upload-time = "2026-09-05T06:48:53.575Z" },
{ url = "https://files.pythonhosted.org/packages/f0/ed/923846b7511627f8564d09345e083f14674dfb193de92d27f1cc4650602e/pikepdf-10.13.0.post1-cp314-abi3-macosx_15_0_x86_64.whl", hash = "sha256:8cb976331cb8b03ec3465e06d9e7a3eadbadb7e622be888e70f918ac732a105e", size = 1943659, upload-time = "2026-09-05T06:48:55.823Z" },
{ url = "https://files.pythonhosted.org/packages/58/bb/fcb09ad4bd227bbb37a7e5b24de86f9ce9d462aa0c7ee18781899bfa378a/pikepdf-10.13.0.post1-cp314-abi3-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3e18d5a009bbe5f3ab18f916fb9e28f0c5b0d920736e3a85cc10c627ec633596", size = 2099423, upload-time = "2026-09-05T06:48:57.671Z" },
{ url = "https://files.pythonhosted.org/packages/3b/c9/707f9ba96727fa366a650237e46f0e20a73b244592eb6b97a49e401e2b43/pikepdf-10.13.0.post1-cp314-abi3-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:414f42c83e5e6029870de1a988625dafc95781ebff10e15d82caeb5e69a83c9c", size = 2303539, upload-time = "2026-09-05T06:48:59.506Z" },
{ url = "https://files.pythonhosted.org/packages/22/15/79a2ccc354514a1321a161867a011fc917be158fc01a88da8c78ad18399d/pikepdf-10.13.0.post1-cp314-abi3-musllinux_1_2_aarch64.whl", hash = "sha256:f0eb89f06cad9231b9db54d81a22592b03b63924824a6b850febd2b75daa6546", size = 3737772, upload-time = "2026-09-05T06:49:01.245Z" },
{ url = "https://files.pythonhosted.org/packages/6e/85/a17440c2de64da71dc012b42e644b30ee4d98eb540c14d4e9f3538b73a9e/pikepdf-10.13.0.post1-cp314-abi3-musllinux_1_2_x86_64.whl", hash = "sha256:544f1be1b1e5630a79cd182a8663c439504099eb9eae0d342a50173d9825bdf3", size = 3948342, upload-time = "2026-09-05T06:49:03.27Z" },
{ url = "https://files.pythonhosted.org/packages/e9/65/15a796a3cf9fb17d41acc1ab6719e7d3dcdf1260e76909d0dd32ecc97ba7/pikepdf-10.13.0.post1-cp314-cp314t-macosx_14_0_arm64.whl", hash = "sha256:ef4ed47d40aa44deb063feb4a88e8bcf1c8fa0183ce526dc4295f7cbdb1292f8", size = 1853638, upload-time = "2026-09-05T06:49:07.184Z" },
{ url = "https://files.pythonhosted.org/packages/aa/f3/5d49a511fd13b59b94c5ad673695d331fce5d2846ab1501646c2a3b35b5f/pikepdf-10.13.0.post1-cp314-cp314t-macosx_15_0_x86_64.whl", hash = "sha256:571efcd1d54e0dd817973c76c253feb6fb758c93bb0c16a893cc68f2a178d404", size = 1951768, upload-time = "2026-09-05T06:49:09.048Z" },
{ url = "https://files.pythonhosted.org/packages/06/de/f6bbd9653695f6e2ed3494a439f11f3a89450c004fe9bf8ee583201ea759/pikepdf-10.13.0.post1-cp314-cp314t-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:2db9a18074ba112e7c517e8c21dfd8894cc13b8a37ccf49a5841192170fb68eb", size = 2107137, upload-time = "2026-09-05T06:49:10.788Z" },
{ url = "https://files.pythonhosted.org/packages/c0/aa/43b355681f05ea0b5808a26cba9e8e764686fa8dcebe0875db612664de40/pikepdf-10.13.0.post1-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:f7962a75cf22d0d683b49ab19b8966e94dc7014d9aa6806f3c0d2b37fb9ae607", size = 2310991, upload-time = "2026-09-05T06:49:12.455Z" },
{ url = "https://files.pythonhosted.org/packages/a9/46/77574e9c4bded01afd7a3fe538f5432e396c3772c9bc1eab5d287aed00df/pikepdf-10.13.0.post1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:b948e11f7dd3710f939194f00b4d75b0df87a30d53b31414128768ac25da77d8", size = 3744358, upload-time = "2026-09-05T06:49:14.428Z" },
{ url = "https://files.pythonhosted.org/packages/89/a8/4857df72cf4773553c2e6a82f93ee5e98c02f1c4e4877379e84d27384982/pikepdf-10.13.0.post1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:99f6afccd6119233e7133bd4c2ade48461de3ddd4269cc28a4f90cdf7c1372f5", size = 3955964, upload-time = "2026-09-05T06:49:16.656Z" },
]
[[package]]
@@ -3610,12 +3620,6 @@ wheels = [
{ url = "https://files.pythonhosted.org/packages/37/ed/89c2c620af0e1660354cd8aabf9f5b21f911597ce22acb37c805d6c86bc8/psycopg_pool-3.3.1-py3-none-any.whl", hash = "sha256:2af5b432941c4c9ad5c87b3fa410aec910ec8f7c122855897983a06c45f2e4b5", size = 40023, upload-time = "2026-05-01T23:31:53.136Z" },
]
[[package]]
name = "py-ubjson"
version = "0.16.1"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "https://files.pythonhosted.org/packages/1d/c7/28220d37e041fe1df03e857fe48f768dcd30cd151480bf6f00da8713214a/py-ubjson-0.16.1.tar.gz", hash = "sha256:b9bfb8695a1c7e3632e800fb83c943bf67ed45ddd87cd0344851610c69a5a482", size = 50316, upload-time = "2020-04-18T15:05:57.698Z" }
[[package]]
name = "pyasn1"
version = "0.6.4"