mirror of
https://github.com/paperless-ngx/paperless-ngx.git
synced 2026-11-10 16:31:05 +00:00
Compare commits
5
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 shamoon
|
60bd48542f | ||
|
shamoon
|
b9e87870ce | ||
|
shamoon
|
763059d479 | ||
|
shamoon
|
0217ec4000 | ||
|
shamoon
|
0c918b03bf |
No files matched your search
@@ -1,5 +1,23 @@
|
||||
title: "[Feature Request] "
|
||||
body:
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
#### Use of AI tools
|
||||
|
||||
If this request was generated, in whole or in part, by an AI tool or agent you **must** say so using the field below. Failing to clearly disclose this is a violation of our [Code of Conduct](https://github.com/paperless-ngx/paperless-ngx/blob/main/CODE_OF_CONDUCT.md).
|
||||
|
||||
The project prefers thoughtful, well-described ideas to AI-generated content. Please describe the problem, your use case, the desired behavior, and who would benefit.
|
||||
- type: dropdown
|
||||
id: ai-usage
|
||||
attributes:
|
||||
label: AI usage
|
||||
description: Was any part of this request written by an AI tool or agent?
|
||||
options:
|
||||
- No AI tools were used
|
||||
- Yes, AI tools helped write this request
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: description
|
||||
attributes:
|
||||
|
||||
@@ -1,5 +1,21 @@
|
||||
title: "[Support] "
|
||||
body:
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
#### Use of AI tools
|
||||
|
||||
If this post was generated, in whole or in part, by an AI tool or agent you **must** say so using the field below. Failing to clearly disclose this is a violation of our [Code of Conduct](https://github.com/paperless-ngx/paperless-ngx/blob/main/CODE_OF_CONDUCT.md).
|
||||
- type: dropdown
|
||||
id: ai-usage
|
||||
attributes:
|
||||
label: AI usage
|
||||
description: Was any part of this post written by an AI tool or agent?
|
||||
options:
|
||||
- No AI tools were used
|
||||
- Yes, AI tools helped write this post
|
||||
validations:
|
||||
required: true
|
||||
- type: textarea
|
||||
id: description
|
||||
attributes:
|
||||
|
||||
@@ -0,0 +1,44 @@
|
||||
# Guidance for AI agents
|
||||
|
||||
Read [CONTRIBUTING.md](CONTRIBUTING.md) before working on this repository, especially [Use of AI Tools](CONTRIBUTING.md#use-of-ai-tools). These contribution rules apply to all agent-assisted work.
|
||||
|
||||
## Contribution boundaries
|
||||
|
||||
- Do not open or submit pull requests, including "on behalf of" a human. The project does not accept PRs from AI agents. Prepare local changes for a human to review, verify, and submit personally.
|
||||
- The project does not accept PRs that are entirely or mostly AI-derived. Human submission and disclosure do not override this rule.
|
||||
- AI-generated content must be clearly disclosed as described in the contributing guidelines. When handing work back to a human, identify what was generated or changed with AI assistance so they can disclose it accurately.
|
||||
- Trust matters. Encourage first-time contributors to discuss their proposed change before preparing a PR, especially for large changes or changes involving AI-generated code. New features and enhancements should almost always target an existing feature request with community interest and discussion.
|
||||
- The project prefers thoughtful, well-described ideas to AI-generated pull requests. Help contributors describe their problem, use case, desired behavior, and who would benefit. An implementation is not required for a valuable contribution, and a flawed implementation can obscure a useful idea before it receives proper discussion.
|
||||
- AI tools have lowered the effort needed to produce a PR, but reviewing and maintaining it still requires substantial work. Encourage discussion of the idea before investing in implementation and review.
|
||||
- Issue reports must describe observed behavior, reproduction steps, and relevant logs. Do not add speculative root causes, code analysis, or proposed fixes to agent-generated reports.
|
||||
|
||||
## Security reports
|
||||
|
||||
- AI-assisted security reports must meet the legitimacy and verification requirements in [SECURITY.md](SECURITY.md#ai-assisted-submissions). A human must verify a real Paperless-ngx code path, concrete reproduction steps, affected versions, and realistic security impact before submission.
|
||||
- Account for existing permission checks, configuration requirements, and trust boundaries. Speculation, scanner output, or concerns about third-party software or deployment choices without a demonstrated Paperless-ngx vulnerability do not meet this threshold and may be closed without extended discussion.
|
||||
- Disclose AI assistance and use the private vulnerability reporting channel specified in [SECURITY.md](SECURITY.md#reporting-a-vulnerability).
|
||||
|
||||
## Working on changes
|
||||
|
||||
- Read the relevant code, tests, and documentation before editing. Paperless-ngx is complex: trace interactions between affected components rather than assuming a change is isolated.
|
||||
- Keep changes focused on the requested work and follow existing patterns. Avoid unrelated refactoring, formatting, dependency updates, or generated-file changes.
|
||||
- Consider effects on permissions, document processing, background tasks, storage, and the API when those areas are involved.
|
||||
- Add or update meaningful tests for behavior changes and regressions. Do not weaken tests or checks to make a change pass.
|
||||
- Update user or developer documentation when behavior or setup changes.
|
||||
- Keep credentials and private documents out of commits, fixtures, logs, and examples.
|
||||
|
||||
## Repository and validation
|
||||
|
||||
- `src/`: Django back end.
|
||||
- `src-ui/`: Angular front end, managed with `pnpm`.
|
||||
- `docs/`: Documentation.
|
||||
- `docker/` and `scripts/`: Deployment and development tooling.
|
||||
- Base functional changes on `dev`; documentation-only changes may target `main`.
|
||||
|
||||
Follow [the development documentation](docs/development.md) for environment setup and testing. Common checks after setup are:
|
||||
|
||||
- Back end: run `uv run pytest` from `src/` (use a test path to focus on the affected area).
|
||||
- Front end: run `pnpm test` and `pnpm lint` from `src-ui/`.
|
||||
- Formatting and other pre-commit checks: run `uv run prek run --files <changed-files>` from the repository root.
|
||||
|
||||
When handing work back, summarize the changes, the checks actually run and their results, and any remaining uncertainty. Clearly state when a check could not be run; do not claim verification that did not happen. The human contributor must understand and independently verify the contribution before submitting it.
|
||||
+16
-3
@@ -4,6 +4,18 @@ If you feel like contributing to the project, please do! Bug fixes and improveme
|
||||
|
||||
⚠️ Please note: Pull requests that implement a new feature or enhancement _should almost always target an existing feature request_ with evidence of community interest and discussion. This is in order to balance the work of implementing and maintaining new features / enhancements. Pull requests that are opened without meeting this requirement may not be merged.
|
||||
|
||||
## Trust and collaboration
|
||||
|
||||
Paperless-ngx places a high value on trust between contributors and maintainers. If you have not contributed before, consider starting with a discussion before preparing a pull request, especially for a large contribution or one involving AI-generated code.
|
||||
|
||||
Paperless-ngx is complex, and understanding how its components interact takes time. A change that appears correct in isolation can have consequences elsewhere in the application. Contributors should understand the code they submit, explain how it fits into the project, and work with maintainers through review. Starting with smaller, focused contributions can help build that understanding and trust.
|
||||
|
||||
We prefer thoughtful, well-described ideas to AI-generated pull requests. You do not need to supply an implementation for your contribution to be valuable. Explain the problem you have encountered, your use case, the behavior you would like to see, and who would benefit. Concrete examples, alternatives you have considered, and relevant constraints help the community discuss and refine the idea.
|
||||
|
||||
AI tools have lowered the effort needed to produce a pull request, but evaluating one still requires substantial maintainer time. Maintainers still need to establish whether the implementation is correct, fits the project, and can be maintained over time.
|
||||
|
||||
A flawed implementation can also obscure a good idea: discussion becomes focused on problems with the generated code before the community has considered whether the underlying proposal is useful. Starting with the idea gives it a fair discussion on its own merits and lets contributors and maintainers agree on an approach before investing in implementation and review.
|
||||
|
||||
If you want to implement something big:
|
||||
|
||||
- As above, please start with a discussion! Maybe something similar is already in development and we can make it happen together.
|
||||
@@ -61,9 +73,10 @@ This project does not specifically prohibit the use of AI tools or agents _durin
|
||||
|
||||
1. Any content — code, pull request descriptions, issue reports or comments — that was generated in whole or in part by an AI tool or agent **must be clearly disclosed as such**. Failing to clearly disclose this is considered a violation of our [Code of Conduct](CODE_OF_CONDUCT.md) and may result in your contributions being closed and further action being taken.
|
||||
2. AI-generated code must not violate copyright protections.
|
||||
3. We will not accept PRs that are entirely or mostly AI-derived.
|
||||
4. Issue reports must describe _observed behavior only_: what happened, how to reproduce it and the relevant logs. Reports generated by an AI agent should not include code analysis, speculation about the root cause or suggest fixes.
|
||||
5. Contributors are responsible for everything they submit, regardless of how it was produced. Please do not open issues or PRs you have not verified yourself.
|
||||
3. We will not accept PRs opened or submitted by AI agents, including agents acting "on behalf of" a human. A human contributor must personally review, verify, and submit the contribution and take responsibility for it throughout review.
|
||||
4. We will not accept PRs that are entirely or mostly AI-derived.
|
||||
5. Issue reports must describe _observed behavior only_: what happened, how to reproduce it and the relevant logs. Reports generated by an AI agent should not include code analysis, speculation about the root cause or suggest fixes.
|
||||
6. Contributors are responsible for everything they submit, regardless of how it was produced. Please do not open issues or PRs you have not verified yourself.
|
||||
|
||||
# Translating Paperless-ngx
|
||||
|
||||
|
||||
@@ -873,14 +873,6 @@ If both the [PAPERLESS_ACCOUNT_DEFAULT_GROUPS](#PAPERLESS_ACCOUNT_DEFAULT_GROUPS
|
||||
|
||||
Defaults to 1209600 (2 weeks)
|
||||
|
||||
#### [`PAPERLESS_SESSION_CLEANUP_TASK_CRON=<cron expression>`](#PAPERLESS_SESSION_CLEANUP_TASK_CRON) {#PAPERLESS_SESSION_CLEANUP_TASK_CRON}
|
||||
|
||||
: Configures the schedule to remove expired sessions from the session store.
|
||||
Django does not prune these automatically, so without this they would
|
||||
accumulate indefinitely.
|
||||
|
||||
Defaults to `0 3 * * *`, once per day at 03:00.
|
||||
|
||||
#### [`PAPERLESS_TOKEN_THROTTLE_RATE=<rate>`](#PAPERLESS_TOKEN_THROTTLE_RATE) {#PAPERLESS_TOKEN_THROTTLE_RATE}
|
||||
|
||||
: Rate limit for the API token authentication endpoint (`/api/token/`), used to mitigate brute-force login attempts.
|
||||
@@ -1157,9 +1149,6 @@ they use underscores instead of dashes.
|
||||
be changed by trusted users. This applies to the `AppConfig` permission as well,
|
||||
which allows setting these arguments from the UI.
|
||||
|
||||
Arguments paperless manages itself (`input_file_or_options`, `output_file`
|
||||
and `sidecar`) are rejected by the API and ignored if set here.
|
||||
|
||||
Specify arguments as a JSON dictionary. Keep note of lower case
|
||||
booleans and double quoted parameter names and strings. Examples:
|
||||
|
||||
|
||||
+8
-15
@@ -4030,21 +4030,14 @@
|
||||
<source>Sources</source>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/chat/chat/chat.component.html</context>
|
||||
<context context-type="linenumber">47,48</context>
|
||||
</context-group>
|
||||
</trans-unit>
|
||||
<trans-unit id="6025275430560167531" datatype="html">
|
||||
<source>+<x id="INTERPOLATION" equiv-text="{{ hiddenReferenceCount(message) }}"/> more</source>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/chat/chat/chat.component.html</context>
|
||||
<context context-type="linenumber">49,50</context>
|
||||
<context context-type="linenumber">46,47</context>
|
||||
</context-group>
|
||||
</trans-unit>
|
||||
<trans-unit id="8066608938393600549" datatype="html">
|
||||
<source>Message</source>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/chat/chat/chat.component.html</context>
|
||||
<context context-type="linenumber">66,67</context>
|
||||
<context context-type="linenumber">61,62</context>
|
||||
</context-group>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/common/email-document-dialog/email-document-dialog.component.html</context>
|
||||
@@ -4055,22 +4048,22 @@
|
||||
<source>Stop</source>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/chat/chat/chat.component.html</context>
|
||||
<context context-type="linenumber">79</context>
|
||||
<context context-type="linenumber">74</context>
|
||||
</context-group>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/chat/chat/chat.component.html</context>
|
||||
<context context-type="linenumber">79</context>
|
||||
<context context-type="linenumber">74</context>
|
||||
</context-group>
|
||||
</trans-unit>
|
||||
<trans-unit id="6490688569532630280" datatype="html">
|
||||
<source>Send</source>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/chat/chat/chat.component.html</context>
|
||||
<context context-type="linenumber">83</context>
|
||||
<context context-type="linenumber">78</context>
|
||||
</context-group>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/chat/chat/chat.component.html</context>
|
||||
<context context-type="linenumber">83</context>
|
||||
<context context-type="linenumber">78</context>
|
||||
</context-group>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||
@@ -4097,14 +4090,14 @@
|
||||
<source>Ask a question about this document...</source>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/chat/chat/chat.component.ts</context>
|
||||
<context context-type="linenumber">83</context>
|
||||
<context context-type="linenumber">61</context>
|
||||
</context-group>
|
||||
</trans-unit>
|
||||
<trans-unit id="224049052163028286" datatype="html">
|
||||
<source>Ask a question about a document...</source>
|
||||
<context-group purpose="location">
|
||||
<context context-type="sourcefile">src/app/components/chat/chat/chat.component.ts</context>
|
||||
<context context-type="linenumber">84</context>
|
||||
<context context-type="linenumber">62</context>
|
||||
</context-group>
|
||||
</trans-unit>
|
||||
<trans-unit id="8700121026680200191" datatype="html">
|
||||
|
||||
@@ -42,16 +42,11 @@
|
||||
<span class="text-break">{{ message.content }}</span>
|
||||
}
|
||||
@if (message.role === 'assistant' && !message.isStreaming && message.references?.length) {
|
||||
<div class="chat-references mt-2 pt-2 border-top">
|
||||
<div class="d-flex align-items-center text-muted mb-1">
|
||||
<span i18n>Sources</span>
|
||||
@if (hiddenReferenceCount(message) > 0 && !expandedReferences().has(message)) {
|
||||
<button type="button" class="badge rounded-pill bg-body-secondary text-body-secondary fw-normal border-0 ms-2" (click)="expandReferences(message)" i18n>+{{ hiddenReferenceCount(message) }} more</button>
|
||||
}
|
||||
</div>
|
||||
@for (reference of visibleReferences(message); track reference.id) {
|
||||
<a class="reference-link d-flex align-items-center lh-sm text-primary text-decoration-none py-1" [routerLink]="['/documents', reference.id]" [title]="reference.title">
|
||||
<pngx-bs-icon width="0.8em" height="0.8em" name="file-text" class="me-2 flex-shrink-0"></pngx-bs-icon><span class="text-truncate">{{ reference.title }}</span>
|
||||
<div class="chat-references list-group mt-3">
|
||||
<div class="list-group-item text-muted small py-1" i18n>Sources</div>
|
||||
@for (reference of message.references; track reference.id) {
|
||||
<a class="list-group-item list-group-item-action d-flex align-items-center text-primary" [routerLink]="['/documents', reference.id]" [title]="reference.title">
|
||||
<pngx-bs-icon width="0.9em" height="0.9em" name="file-text" class="me-2 flex-shrink-0"></pngx-bs-icon><span class="text-truncate">{{ reference.title }}</span>
|
||||
</a>
|
||||
}
|
||||
</div>
|
||||
|
||||
@@ -12,10 +12,6 @@
|
||||
display: none;
|
||||
}
|
||||
|
||||
.message .flex-grow-1 {
|
||||
min-width: 0;
|
||||
}
|
||||
|
||||
.dropdown-item {
|
||||
white-space: initial;
|
||||
}
|
||||
|
||||
@@ -14,7 +14,6 @@ import { BootstrapIconComponent } from 'src/app/components/common/bootstrap-icon
|
||||
import { MarkdownPipe } from 'src/app/pipes/markdown.pipe'
|
||||
import {
|
||||
ChatMessage,
|
||||
ChatReference,
|
||||
ChatService,
|
||||
parseChatResponse,
|
||||
} from 'src/app/services/chat.service'
|
||||
@@ -41,29 +40,8 @@ export class ChatComponent implements OnInit {
|
||||
readonly input = signal('')
|
||||
readonly documentId = signal<number>(undefined)
|
||||
readonly slowResponse = signal(false)
|
||||
readonly expandedReferences = signal<Set<ChatMessage>>(new Set())
|
||||
public popperOptions = pngxPopperOptions
|
||||
|
||||
static readonly COLLAPSED_REFERENCE_COUNT = 2
|
||||
|
||||
public visibleReferences(message: ChatMessage): ChatReference[] {
|
||||
if (this.expandedReferences().has(message)) {
|
||||
return message.references
|
||||
}
|
||||
return message.references.slice(0, ChatComponent.COLLAPSED_REFERENCE_COUNT)
|
||||
}
|
||||
|
||||
public hiddenReferenceCount(message: ChatMessage): number {
|
||||
return Math.max(
|
||||
0,
|
||||
message.references.length - ChatComponent.COLLAPSED_REFERENCE_COUNT
|
||||
)
|
||||
}
|
||||
|
||||
public expandReferences(message: ChatMessage): void {
|
||||
this.expandedReferences.update((expanded) => new Set(expanded).add(message))
|
||||
}
|
||||
|
||||
private chatService: ChatService = inject(ChatService)
|
||||
private router: Router = inject(Router)
|
||||
|
||||
|
||||
@@ -12,7 +12,6 @@ from celery import Task
|
||||
from celery import shared_task
|
||||
from django.conf import settings
|
||||
from django.contrib.contenttypes.models import ContentType
|
||||
from django.contrib.sessions.models import Session
|
||||
from django.db import models
|
||||
from django.db import transaction
|
||||
from django.db.models.signals import post_save
|
||||
@@ -926,17 +925,3 @@ def cleanup_expired_share_link_bundles() -> None:
|
||||
)
|
||||
if count:
|
||||
logger.info("Deleted %s expired share link bundle(s)", count)
|
||||
|
||||
|
||||
@shared_task
|
||||
def clear_expired_sessions() -> None:
|
||||
"""
|
||||
Remove expired database sessions in batches to limit the size of each delete.
|
||||
"""
|
||||
expired = Session.objects.filter(expire_date__lt=timezone.now())
|
||||
while session_keys := list(
|
||||
expired.order_by("expire_date").values_list("pk", flat=True)[:500],
|
||||
):
|
||||
expired.filter(pk__in=session_keys).delete()
|
||||
|
||||
logger.info("Cleared expired sessions")
|
||||
@@ -287,28 +287,6 @@ class TestApiAppConfig(DirectoriesMixin, APITestCase):
|
||||
self.assertEqual(config.user_args, user_args)
|
||||
self.assertEqual(config.barcode_tag_mapping, barcode_tag_mapping)
|
||||
|
||||
def test_api_update_config_rejects_reserved_user_args(self) -> None:
|
||||
"""
|
||||
GIVEN:
|
||||
- API request to update app config with user_args which set an
|
||||
argument the parser manages itself
|
||||
WHEN:
|
||||
- API is called
|
||||
THEN:
|
||||
- Request is rejected with a 400 naming the argument
|
||||
- Config is not updated
|
||||
"""
|
||||
response = self.client.patch(
|
||||
f"{self.ENDPOINT}1/",
|
||||
json.dumps({"user_args": json.dumps({"sidecar": "/tmp/x.py"})}),
|
||||
content_type="application/json",
|
||||
)
|
||||
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
|
||||
self.assertIn("sidecar", str(response.data["user_args"]))
|
||||
config = ApplicationConfiguration.objects.first()
|
||||
assert config is not None
|
||||
self.assertIsNone(config.user_args)
|
||||
|
||||
def test_api_update_config_rejects_invalid_json_objects(self) -> None:
|
||||
"""
|
||||
GIVEN:
|
||||
|
||||
@@ -5,11 +5,8 @@ from unittest import mock
|
||||
|
||||
import pytest
|
||||
from django.conf import settings
|
||||
from django.contrib.sessions.models import Session
|
||||
from django.db import connection
|
||||
from django.test import TestCase
|
||||
from django.test import override_settings
|
||||
from django.test.utils import CaptureQueriesContext
|
||||
from django.utils import timezone
|
||||
|
||||
from documents import tasks
|
||||
@@ -615,36 +612,3 @@ class TestApplyAISuggestionsTask(DirectoriesMixin, TestCase):
|
||||
|
||||
apply_suggestions.assert_not_called()
|
||||
self.assertIn("no longer exists", "".join(cm.output))
|
||||
|
||||
|
||||
@pytest.mark.django_db
|
||||
class TestClearExpiredSessions:
|
||||
def test_clears_only_expired_sessions_in_batches(self) -> None:
|
||||
"""
|
||||
GIVEN:
|
||||
- More than one batch of expired sessions and one still-valid session
|
||||
WHEN:
|
||||
- The clear_expired_sessions task runs
|
||||
THEN:
|
||||
- Only expired sessions are removed, using multiple deletes
|
||||
"""
|
||||
now = timezone.now()
|
||||
Session.objects.bulk_create(
|
||||
Session(
|
||||
session_key=f"expired-{i}",
|
||||
session_data="",
|
||||
expire_date=now - timedelta(days=1),
|
||||
)
|
||||
for i in range(501)
|
||||
)
|
||||
Session.objects.create(
|
||||
session_key="valid",
|
||||
session_data="",
|
||||
expire_date=now + timedelta(days=1),
|
||||
)
|
||||
|
||||
with CaptureQueriesContext(connection) as queries:
|
||||
tasks.clear_expired_sessions()
|
||||
|
||||
assert set(Session.objects.values_list("session_key", flat=True)) == {"valid"}
|
||||
assert sum(query["sql"].startswith("DELETE FROM") for query in queries) == 2
|
||||
@@ -12,17 +12,6 @@ from paperless.models import ModeChoices
|
||||
from paperless.models import OutputTypeChoices
|
||||
from paperless.models import RemoteOCRMode
|
||||
|
||||
# OCRmyPDF arguments the parser sets itself, pointing into its own temporary
|
||||
# directory. The parser reads these files back after OCR, so user supplied
|
||||
# values can only break consumption (or write to arbitrary paths), never help.
|
||||
OCR_RESERVED_USER_ARGS: frozenset[str] = frozenset(
|
||||
{
|
||||
"input_file_or_options",
|
||||
"output_file",
|
||||
"sidecar",
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@dataclasses.dataclass
|
||||
class BaseConfig:
|
||||
|
||||
@@ -21,7 +21,6 @@ from documents.parsers import make_thumbnail_from_pdf
|
||||
from documents.utils import copy_file_with_basic_stats
|
||||
from documents.utils import maybe_override_pixel_limit
|
||||
from documents.utils import remove_alpha_channel
|
||||
from paperless.config import OCR_RESERVED_USER_ARGS
|
||||
from paperless.config import OcrConfig
|
||||
from paperless.models import CleanChoices
|
||||
from paperless.models import ModeChoices
|
||||
@@ -352,22 +351,7 @@ class RasterisedDocumentParser:
|
||||
|
||||
if self.settings.user_args is not None:
|
||||
try:
|
||||
reserved = sorted(
|
||||
OCR_RESERVED_USER_ARGS & self.settings.user_args.keys(),
|
||||
)
|
||||
if reserved:
|
||||
self.log.warning(
|
||||
f"Ignoring OCR user arguments managed by paperless: "
|
||||
f"{', '.join(reserved)}",
|
||||
)
|
||||
ocrmypdf_args = {
|
||||
**ocrmypdf_args,
|
||||
**{
|
||||
k: v
|
||||
for k, v in self.settings.user_args.items()
|
||||
if k not in OCR_RESERVED_USER_ARGS
|
||||
},
|
||||
}
|
||||
ocrmypdf_args = {**ocrmypdf_args, **self.settings.user_args}
|
||||
except Exception as e:
|
||||
self.log.warning(
|
||||
f"There is an issue with PAPERLESS_OCR_USER_ARGS, so "
|
||||
|
||||
@@ -19,7 +19,6 @@ from PIL import Image
|
||||
from rest_framework import serializers
|
||||
from rest_framework.authtoken.serializers import AuthTokenSerializer
|
||||
|
||||
from paperless.config import OCR_RESERVED_USER_ARGS
|
||||
from paperless.models import ApplicationConfiguration
|
||||
from paperless.network import validate_outbound_http_url
|
||||
from paperless.validators import reject_dangerous_svg
|
||||
@@ -252,13 +251,6 @@ class ApplicationConfigurationSerializer(
|
||||
|
||||
def validate_user_args(self, value):
|
||||
self._require_json_object("user_args", value)
|
||||
if value is not None:
|
||||
reserved = sorted(OCR_RESERVED_USER_ARGS & value.keys())
|
||||
if reserved:
|
||||
raise serializers.ValidationError(
|
||||
f"user_args may not set arguments managed by paperless: "
|
||||
f"{', '.join(reserved)}.",
|
||||
)
|
||||
return value
|
||||
|
||||
def validate_barcode_tag_mapping(self, value):
|
||||
|
||||
@@ -165,17 +165,6 @@ def parse_beat_schedule() -> dict:
|
||||
"expires": 23.0 * 60.0 * 60.0,
|
||||
},
|
||||
},
|
||||
{
|
||||
"name": "Clear expired sessions",
|
||||
"env_key": "PAPERLESS_SESSION_CLEANUP_TASK_CRON",
|
||||
# Default daily at 03:00
|
||||
"env_default": "0 3 * * *",
|
||||
"task": "documents.tasks.clear_expired_sessions",
|
||||
"options": {
|
||||
# 1 hour before default schedule sends again
|
||||
"expires": 23.0 * 60.0 * 60.0,
|
||||
},
|
||||
},
|
||||
]
|
||||
for task in tasks:
|
||||
# Either get the environment setting or use the default
|
||||
|
||||
@@ -243,34 +243,6 @@ class TestParserSettingsFromDb(DirectoriesMixin, FileSystemAssertsMixin, TestCas
|
||||
"UseDeviceIndependentColor",
|
||||
)
|
||||
|
||||
def test_ocr_user_args_cannot_override_reserved(self) -> None:
|
||||
"""
|
||||
GIVEN:
|
||||
- User arguments which attempt to override arguments the parser
|
||||
manages itself, alongside a legitimate argument
|
||||
WHEN:
|
||||
- OCR parameters are constructed
|
||||
THEN:
|
||||
- Reserved arguments keep the parser supplied values
|
||||
- The legitimate argument is still applied
|
||||
"""
|
||||
instance = ApplicationConfiguration.objects.all().first()
|
||||
assert instance is not None
|
||||
instance.user_args = {
|
||||
"sidecar": "/tmp/evil.py",
|
||||
"output_file": "/tmp/out.pdf",
|
||||
"input_file_or_options": "/etc/passwd",
|
||||
"unpaper_args": "--pre-rotate 90",
|
||||
}
|
||||
instance.save()
|
||||
|
||||
params = self.get_params()
|
||||
|
||||
self.assertEqual(params["sidecar"], "sidecar.txt")
|
||||
self.assertEqual(params["output_file"], "output.pdf")
|
||||
self.assertEqual(params["input_file_or_options"], "input.pdf")
|
||||
self.assertEqual(params["unpaper_args"], "--pre-rotate 90")
|
||||
|
||||
def test_ocr_user_args(self) -> None:
|
||||
"""
|
||||
GIVEN:
|
||||
|
||||
@@ -182,7 +182,6 @@ def make_expected_schedule(
|
||||
workflow_expire = 59.0 * 60.0
|
||||
llm_index_expire = 23.0 * 60.0 * 60.0
|
||||
share_link_cleanup_expire = 23.0 * 60.0 * 60.0
|
||||
session_cleanup_expire = 23.0 * 60.0 * 60.0
|
||||
|
||||
schedule: dict[str, Any] = {
|
||||
"Check all e-mail accounts": {
|
||||
@@ -249,14 +248,6 @@ def make_expected_schedule(
|
||||
"headers": {"trigger_source": "scheduled"},
|
||||
},
|
||||
},
|
||||
"Clear expired sessions": {
|
||||
"task": "documents.tasks.clear_expired_sessions",
|
||||
"schedule": crontab(minute=0, hour="3"),
|
||||
"options": {
|
||||
"expires": session_cleanup_expire,
|
||||
"headers": {"trigger_source": "scheduled"},
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
overrides = overrides or {}
|
||||
@@ -307,7 +298,6 @@ class TestParseBeatSchedule:
|
||||
"PAPERLESS_WORKFLOW_SCHEDULED_TASK_CRON": "disable",
|
||||
"PAPERLESS_LLM_INDEX_TASK_CRON": "disable",
|
||||
"PAPERLESS_SHARE_LINK_BUNDLE_CLEANUP_CRON": "disable",
|
||||
"PAPERLESS_SESSION_CLEANUP_TASK_CRON": "disable",
|
||||
},
|
||||
{},
|
||||
id="all-disabled",
|
||||
|
||||
@@ -22,8 +22,8 @@ logger = logging.getLogger("paperless_ai.chat")
|
||||
CHAT_METADATA_DELIMITER = "\n\n__PAPERLESS_CHAT_METADATA__"
|
||||
CHAT_ERROR_MESSAGE = "Sorry, something went wrong while generating a response."
|
||||
CHAT_NO_CONTENT_MESSAGE = "Sorry, I couldn't find any content to answer your question."
|
||||
MAX_CHAT_REFERENCES = 3
|
||||
CHAT_RETRIEVER_TOP_K = 5
|
||||
MAX_CHAT_REFERENCES = CHAT_RETRIEVER_TOP_K
|
||||
|
||||
|
||||
def _build_chat_prompt(output_language: str | None) -> str:
|
||||
|
||||
Reference in new issue
Block a user