Compare commits

..

2 Commits

Author SHA1 Message Date
Trenton H
db1e4ce432 Refactor: add explicit supports_progress_bar and supports_multiprocessing to all PaperlessCommand subclasses
Each management command now explicitly declares both class attributes
rather than relying on defaults, making intent unambiguous at a glance.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-05 15:05:15 -08:00
Trenton H
0b3cdd6934 Refactor: migrate exporter/importer from tqdm to PaperlessCommand.track()
Replace direct tqdm usage in document_exporter and document_importer with
the PaperlessCommand base class and its track() method, which is backed by
Rich and handles --no-progress-bar automatically. Also removes the unused
ProgressBarMixin from mixins.py.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-05 08:56:55 -08:00
47 changed files with 1242 additions and 1102 deletions

View File

@@ -14,6 +14,10 @@ component_management:
# https://docs.codecov.com/docs/carryforward-flags
flags:
# Backend Python versions
backend-python-3.10:
paths:
- src/**
carryforward: true
backend-python-3.11:
paths:
- src/**
@@ -22,14 +26,6 @@ flags:
paths:
- src/**
carryforward: true
backend-python-3.13:
paths:
- src/**
carryforward: true
backend-python-3.14:
paths:
- src/**
carryforward: true
# Frontend (shards merge into single flag)
frontend-node-24.x:
paths:
@@ -45,10 +41,9 @@ coverage:
project:
backend:
flags:
- backend-python-3.10
- backend-python-3.11
- backend-python-3.12
- backend-python-3.13
- backend-python-3.14
paths:
- src/**
# https://docs.codecov.com/docs/commit-status#threshold
@@ -64,10 +59,9 @@ coverage:
patch:
backend:
flags:
- backend-python-3.10
- backend-python-3.11
- backend-python-3.12
- backend-python-3.13
- backend-python-3.14
paths:
- src/**
target: 100%

View File

@@ -31,7 +31,7 @@ jobs:
runs-on: ubuntu-24.04
strategy:
matrix:
python-version: ['3.11', '3.12', '3.13', '3.14']
python-version: ['3.10', '3.11', '3.12']
fail-fast: false
steps:
- name: Checkout

View File

@@ -13,9 +13,7 @@ If you want to implement something big:
## Python
Paperless-ngx currently supports Python 3.11, 3.12, 3.13, and 3.14. As a policy, we aim to support at least the three most recent Python versions, and drop support for versions as they reach end-of-life. Older versions may be supported if dependencies permit, but this is not guaranteed.
We format Python code with [ruff](https://docs.astral.sh/ruff/formatter/).
Paperless supports python 3.10 - 3.12 at this time. We format Python code with [ruff](https://docs.astral.sh/ruff/formatter/).
## Branches

View File

@@ -262,10 +262,6 @@ your files differently, you can do that by adjusting the
or using [storage paths (see below)](#storage-paths). Paperless adds the
correct file extension e.g. `.pdf`, `.jpg` automatically.
When a document has file versions, each version uses the same naming rules and
storage path resolution as any other document file, with an added version suffix
such as `_v1`, `_v2`, etc.
This variable allows you to configure the filename (folders are allowed)
using placeholders. For example, configuring this to
@@ -357,8 +353,6 @@ If paperless detects that two documents share the same filename,
paperless will automatically append `_01`, `_02`, etc to the filename.
This happens if all the placeholders in a filename evaluate to the same
value.
For versioned files, this counter is appended after the version suffix
(for example `statement_v2_01.pdf`).
If there are any errors in the placeholders included in `PAPERLESS_FILENAME_FORMAT`,
paperless will fall back to using the default naming scheme instead.

View File

@@ -172,7 +172,7 @@ to enable polling and disable inotify. See [here](configuration.md#polling).
#### Prerequisites
- Paperless runs on Linux only, Windows is not supported.
- Python 3.11, 3.12, 3.13, or 3.14 is required. As a policy, Paperless-ngx aims to support at least the three most recent Python versions and drops support for versions as they reach end-of-life. Newer versions may work, but some dependencies may not be fully compatible.
- Python 3 is required with versions 3.10 - 3.12 currently supported. Newer versions may work, but some dependencies may not be fully compatible.
#### Installation

View File

@@ -95,7 +95,6 @@ Think of versions as **file history** for a document.
- Versions track the underlying file and extracted text content (OCR/text).
- Metadata such as tags, correspondent, document type, storage path and custom fields stay on the "root" document.
- Version files follow normal filename formatting (including storage paths) and add a `_vN` suffix (for example `_v1`, `_v2`).
- By default, search and document content use the latest version.
- In document detail, selecting a version switches the preview, file metadata and content (and download etc buttons) to that version.
- Deleting a non-root version keeps metadata and falls back to the latest remaining version.

View File

@@ -3,9 +3,10 @@ name = "paperless-ngx"
version = "2.20.10"
description = "A community-supported supercharged document management system: scan, index and archive all your physical documents"
readme = "README.md"
requires-python = ">=3.11"
requires-python = ">=3.10"
classifiers = [
"Programming Language :: Python :: 3 :: Only",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
@@ -176,7 +177,7 @@ torch = [
]
[tool.ruff]
target-version = "py311"
target-version = "py310"
line-length = 88
src = [
"src",

View File

@@ -3,7 +3,6 @@ import { provideHttpClientTesting } from '@angular/common/http/testing'
import { ComponentFixture, TestBed } from '@angular/core/testing'
import { NgbActiveModal } from '@ng-bootstrap/ng-bootstrap'
import { NgxBootstrapIconsModule, allIcons } from 'ngx-bootstrap-icons'
import { DocumentService } from 'src/app/services/rest/document.service'
import { PDFEditorComponent } from './pdf-editor.component'
describe('PDFEditorComponent', () => {
@@ -140,16 +139,4 @@ describe('PDFEditorComponent', () => {
expect(component.pages[1].page).toBe(2)
expect(component.pages[2].page).toBe(3)
})
it('should include selected version in preview source when provided', () => {
const documentService = TestBed.inject(DocumentService)
const previewSpy = jest
.spyOn(documentService, 'getPreviewUrl')
.mockReturnValue('preview-version')
component.documentID = 3
component.versionID = 10
expect(component.pdfSrc).toBe('preview-version')
expect(previewSpy).toHaveBeenCalledWith(3, false, 10)
})
})

View File

@@ -46,7 +46,6 @@ export class PDFEditorComponent extends ConfirmDialogComponent {
activeModal: NgbActiveModal = inject(NgbActiveModal)
documentID: number
versionID?: number
pages: PageOperation[] = []
totalPages = 0
editMode: PdfEditorEditMode = this.settingsService.get(
@@ -56,11 +55,7 @@ export class PDFEditorComponent extends ConfirmDialogComponent {
includeMetadata: boolean = true
get pdfSrc(): string {
return this.documentService.getPreviewUrl(
this.documentID,
false,
this.versionID
)
return this.documentService.getPreviewUrl(this.documentID)
}
pdfLoaded(pdf: PngxPdfDocumentProxy) {

View File

@@ -1661,25 +1661,22 @@ describe('DocumentDetailComponent', () => {
const closeSpy = jest.spyOn(openDocumentsService, 'closeDocument')
const errorSpy = jest.spyOn(toastService, 'showError')
initNormally()
component.selectedVersionId = 10
component.editPdf()
expect(modal).not.toBeUndefined()
modal.componentInstance.documentID = doc.id
expect(modal.componentInstance.versionID).toBe(10)
modal.componentInstance.pages = [{ page: 1, rotate: 0, splitAfter: false }]
modal.componentInstance.confirm()
let req = httpTestingController.expectOne(
`${environment.apiBaseUrl}documents/bulk_edit/`
)
expect(req.request.body).toEqual({
documents: [10],
documents: [doc.id],
method: 'edit_pdf',
parameters: {
operations: [{ page: 1, rotate: 0, doc: 0 }],
delete_original: false,
update_document: false,
include_metadata: true,
source_mode: 'explicit_selection',
},
})
req.error(new ErrorEvent('failed'))
@@ -1701,7 +1698,6 @@ describe('DocumentDetailComponent', () => {
let modal: NgbModalRef
modalService.activeInstances.subscribe((m) => (modal = m[0]))
initNormally()
component.selectedVersionId = 10
component.password = 'secret'
component.removePassword()
const dialog =
@@ -1714,14 +1710,13 @@ describe('DocumentDetailComponent', () => {
`${environment.apiBaseUrl}documents/bulk_edit/`
)
expect(req.request.body).toEqual({
documents: [10],
documents: [doc.id],
method: 'remove_password',
parameters: {
password: 'secret',
update_document: false,
include_metadata: false,
delete_original: true,
source_mode: 'explicit_selection',
},
})
req.flush(true)

View File

@@ -74,10 +74,7 @@ import {
import { CorrespondentService } from 'src/app/services/rest/correspondent.service'
import { CustomFieldsService } from 'src/app/services/rest/custom-fields.service'
import { DocumentTypeService } from 'src/app/services/rest/document-type.service'
import {
BulkEditSourceMode,
DocumentService,
} from 'src/app/services/rest/document.service'
import { DocumentService } from 'src/app/services/rest/document.service'
import { SavedViewService } from 'src/app/services/rest/saved-view.service'
import { StoragePathService } from 'src/app/services/rest/storage-path.service'
import { TagService } from 'src/app/services/rest/tag.service'
@@ -1756,23 +1753,20 @@ export class DocumentDetailComponent
size: 'xl',
scrollable: true,
})
const sourceDocumentId = this.selectedVersionId ?? this.document.id
modal.componentInstance.title = $localize`PDF Editor`
modal.componentInstance.btnCaption = $localize`Proceed`
modal.componentInstance.documentID = this.document.id
modal.componentInstance.versionID = sourceDocumentId
modal.componentInstance.confirmClicked
.pipe(takeUntil(this.unsubscribeNotifier))
.subscribe(() => {
modal.componentInstance.buttonsEnabled = false
this.documentsService
.bulkEdit([sourceDocumentId], 'edit_pdf', {
.bulkEdit([this.document.id], 'edit_pdf', {
operations: modal.componentInstance.getOperations(),
delete_original: modal.componentInstance.deleteOriginal,
update_document:
modal.componentInstance.editMode == PdfEditorEditMode.Update,
include_metadata: modal.componentInstance.includeMetadata,
source_mode: BulkEditSourceMode.EXPLICIT_SELECTION,
})
.pipe(first(), takeUntil(this.unsubscribeNotifier))
.subscribe({
@@ -1818,18 +1812,16 @@ export class DocumentDetailComponent
modal.componentInstance.confirmClicked
.pipe(takeUntil(this.unsubscribeNotifier))
.subscribe(() => {
const sourceDocumentId = this.selectedVersionId ?? this.document.id
const dialog =
modal.componentInstance as PasswordRemovalConfirmDialogComponent
dialog.buttonsEnabled = false
this.networkActive = true
this.documentsService
.bulkEdit([sourceDocumentId], 'remove_password', {
.bulkEdit([this.document.id], 'remove_password', {
password: this.password,
update_document: dialog.updateDocument,
include_metadata: dialog.includeMetadata,
delete_original: dialog.deleteOriginal,
source_mode: BulkEditSourceMode.EXPLICIT_SELECTION,
})
.pipe(first(), takeUntil(this.unsubscribeNotifier))
.subscribe({

View File

@@ -37,11 +37,6 @@ export interface SelectionData {
selected_custom_fields: SelectionDataItem[]
}
export enum BulkEditSourceMode {
LATEST_VERSION = 'latest_version',
EXPLICIT_SELECTION = 'explicit_selection',
}
@Injectable({
providedIn: 'root',
})

View File

@@ -29,21 +29,12 @@ from documents.plugins.helpers import DocumentsStatusManager
from documents.tasks import bulk_update_documents
from documents.tasks import consume_file
from documents.tasks import update_document_content_maybe_archive_file
from documents.versioning import get_latest_version_for_root
from documents.versioning import get_root_document
if TYPE_CHECKING:
from django.contrib.auth.models import User
logger: logging.Logger = logging.getLogger("paperless.bulk_edit")
SourceMode = Literal["latest_version", "explicit_selection"]
class SourceModeChoices:
LATEST_VERSION: SourceMode = "latest_version"
EXPLICIT_SELECTION: SourceMode = "explicit_selection"
@shared_task(bind=True)
def restore_archive_serial_numbers_task(
@@ -81,21 +72,46 @@ def restore_archive_serial_numbers(backup: dict[int, int | None]) -> None:
logger.info(f"Restored archive serial numbers for documents {list(backup.keys())}")
def _resolve_root_and_source_doc(
doc: Document,
*,
source_mode: SourceMode = SourceModeChoices.LATEST_VERSION,
) -> tuple[Document, Document]:
root_doc = get_root_document(doc)
def _get_root_ids_by_doc_id(doc_ids: list[int]) -> dict[int, int]:
"""
Resolve each provided document id to its root document id.
if source_mode == SourceModeChoices.EXPLICIT_SELECTION:
return root_doc, doc
- If the id is already a root document: root id is itself.
- If the id is a version document: root id is its `root_document_id`.
"""
qs = Document.objects.filter(id__in=doc_ids).only("id", "root_document_id")
return {doc.id: doc.root_document_id or doc.id for doc in qs}
# Version IDs are explicit by default, only a selected root resolves to latest
if doc.root_document_id is not None:
return root_doc, doc
return root_doc, get_latest_version_for_root(root_doc)
def _get_root_and_current_docs_by_root_id(
root_ids: set[int],
) -> tuple[dict[int, Document], dict[int, Document]]:
"""
Returns:
- root_docs: root_id -> root Document
- current_docs: root_id -> newest version Document (or root if none)
"""
root_docs = {
doc.id: doc
for doc in Document.objects.filter(id__in=root_ids).select_related(
"owner",
)
}
latest_versions_by_root_id: dict[int, Document] = {}
for version_doc in Document.objects.filter(root_document_id__in=root_ids).order_by(
"root_document_id",
"-id",
):
root_id = version_doc.root_document_id
if root_id is None:
continue
latest_versions_by_root_id.setdefault(root_id, version_doc)
current_docs: dict[int, Document] = {
root_id: latest_versions_by_root_id.get(root_id, root_docs[root_id])
for root_id in root_docs
}
return root_docs, current_docs
def set_correspondent(
@@ -405,32 +421,21 @@ def rotate(
doc_ids: list[int],
degrees: int,
*,
source_mode: SourceMode = SourceModeChoices.LATEST_VERSION,
user: User | None = None,
) -> Literal["OK"]:
logger.info(
f"Attempting to rotate {len(doc_ids)} documents by {degrees} degrees.",
)
docs_by_id = {
doc.id: doc
for doc in Document.objects.select_related("root_document").filter(
id__in=doc_ids,
)
}
docs_by_root_id: dict[int, tuple[Document, Document]] = {}
for doc_id in doc_ids:
doc = docs_by_id.get(doc_id)
if doc is None:
continue
root_doc, source_doc = _resolve_root_and_source_doc(
doc,
source_mode=source_mode,
)
docs_by_root_id.setdefault(root_doc.id, (root_doc, source_doc))
doc_to_root_id = _get_root_ids_by_doc_id(doc_ids)
root_ids = set(doc_to_root_id.values())
root_docs_by_id, current_docs_by_root_id = _get_root_and_current_docs_by_root_id(
root_ids,
)
import pikepdf
for root_doc, source_doc in docs_by_root_id.values():
for root_id in root_ids:
root_doc = root_docs_by_id[root_id]
source_doc = current_docs_by_root_id[root_id]
if source_doc.mime_type != "application/pdf":
logger.warning(
f"Document {root_doc.id} is not a PDF, skipping rotation.",
@@ -476,14 +481,12 @@ def merge(
metadata_document_id: int | None = None,
delete_originals: bool = False,
archive_fallback: bool = False,
source_mode: SourceMode = SourceModeChoices.LATEST_VERSION,
user: User | None = None,
) -> Literal["OK"]:
logger.info(
f"Attempting to merge {len(doc_ids)} documents into a single document.",
)
qs = Document.objects.select_related("root_document").filter(id__in=doc_ids)
docs_by_id = {doc.id: doc for doc in qs}
qs = Document.objects.filter(id__in=doc_ids)
affected_docs: list[int] = []
import pikepdf
@@ -492,20 +495,14 @@ def merge(
handoff_asn: int | None = None
# use doc_ids to preserve order
for doc_id in doc_ids:
doc = docs_by_id.get(doc_id)
if doc is None:
continue
_, source_doc = _resolve_root_and_source_doc(
doc,
source_mode=source_mode,
)
doc = qs.get(id=doc_id)
try:
doc_path = (
source_doc.archive_path
doc.archive_path
if archive_fallback
and source_doc.mime_type != "application/pdf"
and source_doc.has_archive_version
else source_doc.source_path
and doc.mime_type != "application/pdf"
and doc.has_archive_version
else doc.source_path
)
with pikepdf.open(str(doc_path)) as pdf:
version = max(version, pdf.pdf_version)
@@ -587,23 +584,18 @@ def split(
pages: list[list[int]],
*,
delete_originals: bool = False,
source_mode: SourceMode = SourceModeChoices.LATEST_VERSION,
user: User | None = None,
) -> Literal["OK"]:
logger.info(
f"Attempting to split document {doc_ids[0]} into {len(pages)} documents",
)
doc = Document.objects.select_related("root_document").get(id=doc_ids[0])
_, source_doc = _resolve_root_and_source_doc(
doc,
source_mode=source_mode,
)
doc = Document.objects.get(id=doc_ids[0])
import pikepdf
consume_tasks = []
try:
with pikepdf.open(source_doc.source_path) as pdf:
with pikepdf.open(doc.source_path) as pdf:
for idx, split_doc in enumerate(pages):
dst: pikepdf.Pdf = pikepdf.new()
for page in split_doc:
@@ -667,17 +659,25 @@ def delete_pages(
doc_ids: list[int],
pages: list[int],
*,
source_mode: SourceMode = SourceModeChoices.LATEST_VERSION,
user: User | None = None,
) -> Literal["OK"]:
logger.info(
f"Attempting to delete pages {pages} from {len(doc_ids)} documents",
)
doc = Document.objects.select_related("root_document").get(id=doc_ids[0])
root_doc, source_doc = _resolve_root_and_source_doc(
doc,
source_mode=source_mode,
root_doc: Document
if doc.root_document_id is None or doc.root_document is None:
root_doc = doc
else:
root_doc = doc.root_document
source_doc = (
Document.objects.filter(Q(id=root_doc.id) | Q(root_document=root_doc))
.order_by("-id")
.first()
)
if source_doc is None:
source_doc = root_doc
pages = sorted(pages) # sort pages to avoid index issues
import pikepdf
@@ -722,7 +722,6 @@ def edit_pdf(
delete_original: bool = False,
update_document: bool = False,
include_metadata: bool = True,
source_mode: SourceMode = SourceModeChoices.LATEST_VERSION,
user: User | None = None,
) -> Literal["OK"]:
"""
@@ -737,10 +736,19 @@ def edit_pdf(
f"Editing PDF of document {doc_ids[0]} with {len(operations)} operations",
)
doc = Document.objects.select_related("root_document").get(id=doc_ids[0])
root_doc, source_doc = _resolve_root_and_source_doc(
doc,
source_mode=source_mode,
root_doc: Document
if doc.root_document_id is None or doc.root_document is None:
root_doc = doc
else:
root_doc = doc.root_document
source_doc = (
Document.objects.filter(Q(id=root_doc.id) | Q(root_document=root_doc))
.order_by("-id")
.first()
)
if source_doc is None:
source_doc = root_doc
import pikepdf
pdf_docs: list[pikepdf.Pdf] = []
@@ -851,7 +859,6 @@ def remove_password(
update_document: bool = False,
delete_original: bool = False,
include_metadata: bool = True,
source_mode: SourceMode = SourceModeChoices.LATEST_VERSION,
user: User | None = None,
) -> Literal["OK"]:
"""
@@ -861,10 +868,19 @@ def remove_password(
for doc_id in doc_ids:
doc = Document.objects.select_related("root_document").get(id=doc_id)
root_doc, source_doc = _resolve_root_and_source_doc(
doc,
source_mode=source_mode,
root_doc: Document
if doc.root_document_id is None or doc.root_document is None:
root_doc = doc
else:
root_doc = doc.root_document
source_doc = (
Document.objects.filter(Q(id=root_doc.id) | Q(root_document=root_doc))
.order_by("-id")
.first()
)
if source_doc is None:
source_doc = root_doc
try:
logger.info(
f"Attempting password removal from document {doc_ids[0]}",

View File

@@ -1,5 +1,5 @@
from datetime import UTC
from datetime import datetime
from datetime import timezone
from typing import Any
from django.conf import settings
@@ -139,7 +139,7 @@ def thumbnail_last_modified(request: Any, pk: int) -> datetime | None:
# No cache, get the timestamp and cache the datetime
last_modified = datetime.fromtimestamp(
doc.thumbnail_path.stat().st_mtime,
tz=UTC,
tz=timezone.utc,
)
cache.set(doc_key, last_modified, CACHE_50_MINUTES)
return last_modified

View File

@@ -2,7 +2,7 @@ import datetime
import hashlib
import os
import tempfile
from enum import StrEnum
from enum import Enum
from pathlib import Path
from typing import TYPE_CHECKING
from typing import Final
@@ -11,7 +11,6 @@ import magic
from django.conf import settings
from django.contrib.auth.models import User
from django.db import transaction
from django.db.models import Max
from django.db.models import Q
from django.utils import timezone
from filelock import FileLock
@@ -83,7 +82,7 @@ class ConsumerError(Exception):
pass
class ConsumerStatusShortMessage(StrEnum):
class ConsumerStatusShortMessage(str, Enum):
DOCUMENT_ALREADY_EXISTS = "document_already_exists"
DOCUMENT_ALREADY_EXISTS_IN_TRASH = "document_already_exists_in_trash"
ASN_ALREADY_EXISTS = "asn_already_exists"
@@ -125,6 +124,22 @@ class ConsumerPluginMixin:
self.filename = self.metadata.filename or self.input_doc.original_file.name
if input_doc.root_document_id:
self.log.debug(
f"Document root document id: {input_doc.root_document_id}",
)
root_document = Document.objects.get(pk=input_doc.root_document_id)
version_index = Document.objects.filter(root_document=root_document).count()
filename_path = Path(self.filename)
if filename_path.suffix:
self.filename = str(
filename_path.with_name(
f"{filename_path.stem}_v{version_index}{filename_path.suffix}",
),
)
else:
self.filename = f"{self.filename}_v{version_index}"
def _send_progress(
self,
current_progress: int,
@@ -170,7 +185,7 @@ class ConsumerPlugin(
):
logging_name = LOGGING_NAME
def _create_version_from_root(
def _clone_root_into_version(
self,
root_doc: Document,
*,
@@ -179,38 +194,30 @@ class ConsumerPlugin(
mime_type: str,
) -> Document:
self.log.debug("Saving record for updated version to database")
root_doc_frozen = Document.objects.select_for_update().get(pk=root_doc.pk)
next_version_index = (
Document.global_objects.filter(
root_document_id=root_doc_frozen.pk,
).aggregate(
max_index=Max("version_index"),
)["max_index"]
or 0
)
version_doc = Document.objects.get(pk=root_doc.pk)
setattr(version_doc, "pk", None)
version_doc.root_document = root_doc
file_for_checksum = (
self.unmodified_original
if self.unmodified_original is not None
else self.working_copy
)
version_doc = Document(
root_document=root_doc_frozen,
version_index=next_version_index + 1,
checksum=hashlib.md5(
file_for_checksum.read_bytes(),
).hexdigest(),
content=text or "",
page_count=page_count,
mime_type=mime_type,
original_filename=self.filename,
owner_id=root_doc_frozen.owner_id,
created=root_doc_frozen.created,
title=root_doc_frozen.title,
added=timezone.now(),
modified=timezone.now(),
)
version_doc.checksum = hashlib.md5(
file_for_checksum.read_bytes(),
).hexdigest()
version_doc.content = text or ""
version_doc.page_count = page_count
version_doc.mime_type = mime_type
version_doc.original_filename = self.filename
version_doc.storage_path = root_doc.storage_path
# Clear unique file path fields so they can be generated uniquely later
version_doc.filename = None
version_doc.archive_filename = None
version_doc.archive_checksum = None
if self.metadata.version_label is not None:
version_doc.version_label = self.metadata.version_label
version_doc.added = timezone.now()
version_doc.modified = timezone.now()
return version_doc
def run_pre_consume_script(self) -> None:
@@ -536,7 +543,7 @@ class ConsumerPlugin(
root_doc = Document.objects.get(
pk=self.input_doc.root_document_id,
)
original_document = self._create_version_from_root(
original_document = self._clone_root_into_version(
root_doc,
text=text,
page_count=page_count,

View File

@@ -129,19 +129,12 @@ def generate_filename(
archive_filename=False,
use_format=True,
) -> Path:
# version docs use the root document for formatting, just with a suffix
context_doc = doc if doc.root_document_id is None else doc.root_document
version_suffix = (
f"_v{doc.version_index}"
if doc.root_document_id is not None and doc.version_index is not None
else ""
)
base_path: Path | None = None
# Determine the source of the format string
if use_format:
if context_doc.storage_path is not None:
filename_format = context_doc.storage_path.path
if doc.storage_path is not None:
filename_format = doc.storage_path.path
elif settings.FILENAME_FORMAT is not None:
# Maybe convert old to new style
filename_format = convert_format_str_to_template_format(
@@ -154,7 +147,7 @@ def generate_filename(
# If we have one, render it
if filename_format is not None:
rendered_path: str | None = format_filename(context_doc, filename_format)
rendered_path: str | None = format_filename(doc, filename_format)
if rendered_path:
base_path = Path(rendered_path)
@@ -168,7 +161,7 @@ def generate_filename(
base_filename = base_path.name
# Build the final filename with counter and filetype
final_filename = f"{base_filename}{version_suffix}{counter_str}{filetype_str}"
final_filename = f"{base_filename}{counter_str}{filetype_str}"
# If we have a directory component, include it
if str(directory) != ".":
@@ -177,9 +170,7 @@ def generate_filename(
full_path = Path(final_filename)
else:
# No template, use document ID
final_filename = (
f"{context_doc.pk:07}{version_suffix}{counter_str}{filetype_str}"
)
final_filename = f"{doc.pk:07}{counter_str}{filetype_str}"
full_path = Path(final_filename)
return full_path

View File

@@ -5,10 +5,10 @@ import math
import re
from collections import Counter
from contextlib import contextmanager
from datetime import UTC
from datetime import datetime
from datetime import time
from datetime import timedelta
from datetime import timezone
from shutil import rmtree
from time import sleep
from typing import TYPE_CHECKING
@@ -437,7 +437,7 @@ class ManualResults:
class LocalDateParser(English):
def reverse_timezone_offset(self, d):
return (d.replace(tzinfo=django_timezone.get_current_timezone())).astimezone(
UTC,
timezone.utc,
)
def date_from(self, *args, **kwargs):
@@ -641,8 +641,8 @@ def rewrite_natural_date_keywords(query_string: str) -> str:
end = datetime(local_now.year - 1, 12, 31, 23, 59, 59, tzinfo=tz)
# Convert to UTC and format
start_str = start.astimezone(UTC).strftime("%Y%m%d%H%M%S")
end_str = end.astimezone(UTC).strftime("%Y%m%d%H%M%S")
start_str = start.astimezone(timezone.utc).strftime("%Y%m%d%H%M%S")
end_str = end.astimezone(timezone.utc).strftime("%Y%m%d%H%M%S")
return f"{field}:[{start_str} TO {end_str}]"
return re.sub(pattern, repl, query_string, flags=re.IGNORECASE)

View File

@@ -304,7 +304,7 @@ class PaperlessCommand(RichCommand):
Progress output is directed to stderr to match the convention that
progress bars are transient UI feedback, not command output. This
mirrors tqdm's default behavior and prevents progress bar rendering
mirrors the convention that progress bars are transient UI feedback and prevents progress bar rendering
from interfering with stdout-based assertions in tests or piped
command output.

View File

@@ -17,6 +17,7 @@ class Command(PaperlessCommand):
"modified) after their initial import."
)
supports_progress_bar = True
supports_multiprocessing = True
def add_arguments(self, parser):

View File

@@ -8,7 +8,6 @@ from itertools import islice
from pathlib import Path
from typing import TYPE_CHECKING
import tqdm
from allauth.mfa.models import Authenticator
from allauth.socialaccount.models import SocialAccount
from allauth.socialaccount.models import SocialApp
@@ -19,7 +18,6 @@ from django.contrib.auth.models import Permission
from django.contrib.auth.models import User
from django.contrib.contenttypes.models import ContentType
from django.core import serializers
from django.core.management.base import BaseCommand
from django.core.management.base import CommandError
from django.core.serializers.json import DjangoJSONEncoder
from django.db import transaction
@@ -38,6 +36,7 @@ if settings.AUDIT_LOG_ENABLED:
from documents.file_handling import delete_empty_directories
from documents.file_handling import generate_filename
from documents.management.commands.base import PaperlessCommand
from documents.management.commands.mixins import CryptMixin
from documents.models import Correspondent
from documents.models import CustomField
@@ -81,14 +80,18 @@ def serialize_queryset_batched(
yield serializers.serialize("python", chunk)
class Command(CryptMixin, BaseCommand):
class Command(CryptMixin, PaperlessCommand):
help = (
"Decrypt and rename all files in our collection into a given target "
"directory. And include a manifest file containing document data for "
"easy import."
)
supports_progress_bar = True
supports_multiprocessing = False
def add_arguments(self, parser) -> None:
super().add_arguments(parser)
parser.add_argument("target")
parser.add_argument(
@@ -195,13 +198,6 @@ class Command(CryptMixin, BaseCommand):
help="If set, only the database will be imported, not files",
)
parser.add_argument(
"--no-progress-bar",
default=False,
action="store_true",
help="If set, the progress bar will not be shown",
)
parser.add_argument(
"--passphrase",
help="If provided, is used to encrypt sensitive data in the export",
@@ -230,7 +226,6 @@ class Command(CryptMixin, BaseCommand):
self.no_thumbnail: bool = options["no_thumbnail"]
self.zip_export: bool = options["zip"]
self.data_only: bool = options["data_only"]
self.no_progress_bar: bool = options["no_progress_bar"]
self.passphrase: str | None = options.get("passphrase")
self.batch_size: int = options["batch_size"]
@@ -347,10 +342,12 @@ class Command(CryptMixin, BaseCommand):
document_manifest = manifest_dict["documents"]
# 3. Export files from each document
for index, document_dict in tqdm.tqdm(
enumerate(document_manifest),
total=len(document_manifest),
disable=self.no_progress_bar,
for index, document_dict in enumerate(
self.track(
document_manifest,
description="Exporting documents...",
total=len(document_manifest),
),
):
document = document_map[document_dict["pk"]]

View File

@@ -40,6 +40,7 @@ def _process_and_match(work: _WorkPackage) -> _WorkResult:
class Command(PaperlessCommand):
help = "Searches for documents where the content almost matches"
supports_progress_bar = True
supports_multiprocessing = True
def add_arguments(self, parser):

View File

@@ -8,14 +8,12 @@ from pathlib import Path
from zipfile import ZipFile
from zipfile import is_zipfile
import tqdm
from django.conf import settings
from django.contrib.auth.models import Permission
from django.contrib.auth.models import User
from django.contrib.contenttypes.models import ContentType
from django.core.exceptions import FieldDoesNotExist
from django.core.management import call_command
from django.core.management.base import BaseCommand
from django.core.management.base import CommandError
from django.core.serializers.base import DeserializationError
from django.db import IntegrityError
@@ -25,6 +23,7 @@ from django.db.models.signals import post_save
from filelock import FileLock
from documents.file_handling import create_source_path_directory
from documents.management.commands.base import PaperlessCommand
from documents.management.commands.mixins import CryptMixin
from documents.models import Correspondent
from documents.models import CustomField
@@ -57,21 +56,18 @@ def disable_signal(sig, receiver, sender, *, weak: bool | None = None) -> Genera
sig.connect(receiver=receiver, sender=sender, **kwargs)
class Command(CryptMixin, BaseCommand):
class Command(CryptMixin, PaperlessCommand):
help = (
"Using a manifest.json file, load the data from there, and import the "
"documents it refers to."
)
def add_arguments(self, parser) -> None:
parser.add_argument("source")
supports_progress_bar = True
supports_multiprocessing = False
parser.add_argument(
"--no-progress-bar",
default=False,
action="store_true",
help="If set, the progress bar will not be shown",
)
def add_arguments(self, parser) -> None:
super().add_arguments(parser)
parser.add_argument("source")
parser.add_argument(
"--data-only",
@@ -231,7 +227,6 @@ class Command(CryptMixin, BaseCommand):
self.source = Path(options["source"]).resolve()
self.data_only: bool = options["data_only"]
self.no_progress_bar: bool = options["no_progress_bar"]
self.passphrase: str | None = options.get("passphrase")
self.version: str | None = None
self.salt: str | None = None
@@ -365,7 +360,7 @@ class Command(CryptMixin, BaseCommand):
filter(lambda r: r["model"] == "documents.document", self.manifest),
)
for record in tqdm.tqdm(manifest_documents, disable=self.no_progress_bar):
for record in self.track(manifest_documents, description="Copying files..."):
document = Document.objects.get(pk=record["pk"])
doc_file = record[EXPORTER_FILE_NAME]

View File

@@ -8,6 +8,9 @@ from documents.tasks import index_reindex
class Command(PaperlessCommand):
help = "Manages the document index."
supports_progress_bar = True
supports_multiprocessing = False
def add_arguments(self, parser):
super().add_arguments(parser)
parser.add_argument("command", choices=["reindex", "optimize"])

View File

@@ -7,6 +7,9 @@ from documents.tasks import llmindex_index
class Command(PaperlessCommand):
help = "Manages the LLM-based vector index for Paperless."
supports_progress_bar = True
supports_multiprocessing = False
def add_arguments(self, parser: Any) -> None:
super().add_arguments(parser)
parser.add_argument("command", choices=["rebuild", "update"])

View File

@@ -7,6 +7,9 @@ from documents.models import Document
class Command(PaperlessCommand):
help = "Rename all documents"
supports_progress_bar = True
supports_multiprocessing = False
def handle(self, *args, **options):
for document in self.track(Document.objects.all(), description="Renaming..."):
post_save.send(Document, instance=document, created=False)

View File

@@ -180,6 +180,9 @@ class Command(PaperlessCommand):
"modified) after their initial import."
)
supports_progress_bar = True
supports_multiprocessing = False
def add_arguments(self, parser) -> None:
super().add_arguments(parser)
parser.add_argument("-c", "--correspondent", default=False, action="store_true")

View File

@@ -24,6 +24,9 @@ _LEVEL_STYLE: dict[int, tuple[str, str]] = {
class Command(PaperlessCommand):
help = "This command checks your document archive for issues."
supports_progress_bar = True
supports_multiprocessing = False
def _render_results(self, messages: SanityCheckMessages) -> None:
"""Render sanity check results as a Rich table."""

View File

@@ -36,6 +36,7 @@ def _process_document(doc_id: int) -> None:
class Command(PaperlessCommand):
help = "This will regenerate the thumbnails for all documents."
supports_progress_bar = True
supports_multiprocessing = True
def add_arguments(self, parser) -> None:

View File

@@ -1,6 +1,5 @@
import base64
import os
from argparse import ArgumentParser
from typing import TypedDict
from cryptography.fernet import Fernet
@@ -21,25 +20,6 @@ class CryptFields(TypedDict):
fields: list[str]
class ProgressBarMixin:
"""
Many commands use a progress bar, which can be disabled
via this class
"""
def add_argument_progress_bar_mixin(self, parser: ArgumentParser) -> None:
parser.add_argument(
"--no-progress-bar",
default=False,
action="store_true",
help="If set, the progress bar will not be shown",
)
def handle_progress_bar_mixin(self, *args, **options) -> None:
self.no_progress_bar = options["no_progress_bar"]
self.use_progress_bar = not self.no_progress_bar
class CryptMixin:
"""
Fully based on:

View File

@@ -9,6 +9,9 @@ class Command(PaperlessCommand):
help = "Prunes the audit logs of objects that no longer exist."
supports_progress_bar = True
supports_multiprocessing = False
def handle(self, *args, **options):
with transaction.atomic():
for log_entry in self.track(

View File

@@ -1,37 +0,0 @@
# Generated by Django 5.2.11 on 2026-03-02 17:48
from django.conf import settings
from django.db import migrations
from django.db import models
class Migration(migrations.Migration):
dependencies = [
("documents", "0015_savedview_visibility_to_ui_settings"),
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
]
operations = [
migrations.AddField(
model_name="document",
name="version_index",
field=models.PositiveIntegerField(
blank=True,
db_index=True,
help_text="Index of this version within the root document.",
null=True,
verbose_name="version index",
),
),
migrations.AddConstraint(
model_name="document",
constraint=models.UniqueConstraint(
condition=models.Q(
("root_document__isnull", False),
("version_index__isnull", False),
),
fields=("root_document", "version_index"),
name="documents_document_root_version_index_uniq",
),
),
]

View File

@@ -319,14 +319,6 @@ class Document(SoftDeleteModel, ModelWithOwner): # type: ignore[django-manager-
verbose_name=_("root document for this version"),
)
version_index = models.PositiveIntegerField(
_("version index"),
blank=True,
null=True,
db_index=True,
help_text=_("Index of this version within the root document."),
)
version_label = models.CharField(
_("version label"),
max_length=64,
@@ -339,16 +331,6 @@ class Document(SoftDeleteModel, ModelWithOwner): # type: ignore[django-manager-
ordering = ("-created",)
verbose_name = _("document")
verbose_name_plural = _("documents")
constraints = [
models.UniqueConstraint(
fields=["root_document", "version_index"],
condition=models.Q(
root_document__isnull=False,
version_index__isnull=False,
),
name="documents_document_root_version_index_uniq",
),
]
def __str__(self) -> str:
created = self.created.isoformat()

View File

@@ -5,7 +5,11 @@ from abc import abstractmethod
from collections.abc import Iterator
from dataclasses import dataclass
from types import TracebackType
from typing import Self
try:
from typing import Self
except ImportError:
from typing_extensions import Self
import dateparser

View File

@@ -9,7 +9,7 @@ if TYPE_CHECKING:
from channels_redis.pubsub import RedisPubSubChannelLayer
class ProgressStatusOptions(enum.StrEnum):
class ProgressStatusOptions(str, enum.Enum):
STARTED = "STARTED"
WORKING = "WORKING"
SUCCESS = "SUCCESS"

View File

@@ -1723,15 +1723,6 @@ class BulkEditSerializer(
except ValueError:
raise serializers.ValidationError("invalid rotation degrees")
def _validate_source_mode(self, parameters) -> None:
source_mode = parameters.get(
"source_mode",
bulk_edit.SourceModeChoices.LATEST_VERSION,
)
if source_mode not in bulk_edit.SourceModeChoices.__dict__.values():
raise serializers.ValidationError("Invalid source_mode")
parameters["source_mode"] = source_mode
def _validate_parameters_split(self, parameters) -> None:
if "pages" not in parameters:
raise serializers.ValidationError("pages not specified")
@@ -1832,9 +1823,6 @@ class BulkEditSerializer(
method = attrs["method"]
parameters = attrs["parameters"]
if "source_mode" in parameters:
self._validate_source_mode(parameters)
if method == bulk_edit.set_correspondent:
self._validate_parameters_correspondent(parameters)
elif method == bulk_edit.set_document_type:

View File

@@ -620,16 +620,6 @@ def update_filename_and_move_files(
root=settings.ARCHIVE_DIR,
)
# Keep version files in sync with root
if instance.root_document_id is None:
for version_doc in Document.objects.filter(root_document_id=instance.pk).only(
"pk",
):
update_filename_and_move_files(
Document,
version_doc,
)
@shared_task
def process_cf_select_update(custom_field: CustomField) -> None:

View File

@@ -24,7 +24,7 @@ def base_config() -> DateParserConfig:
12,
0,
0,
tzinfo=datetime.UTC,
tzinfo=datetime.timezone.utc,
),
filename_date_order="YMD",
content_date_order="DMY",
@@ -45,7 +45,7 @@ def config_with_ignore_dates() -> DateParserConfig:
12,
0,
0,
tzinfo=datetime.UTC,
tzinfo=datetime.timezone.utc,
),
filename_date_order="DMY",
content_date_order="MDY",

View File

@@ -101,50 +101,50 @@ class TestFilterDate:
[
# Valid Dates
pytest.param(
datetime.datetime(2024, 1, 10, tzinfo=datetime.UTC),
datetime.datetime(2024, 1, 10, tzinfo=datetime.UTC),
datetime.datetime(2024, 1, 10, tzinfo=datetime.timezone.utc),
datetime.datetime(2024, 1, 10, tzinfo=datetime.timezone.utc),
id="valid_past_date",
),
pytest.param(
datetime.datetime(2024, 1, 15, 12, 0, 0, tzinfo=datetime.UTC),
datetime.datetime(2024, 1, 15, 12, 0, 0, tzinfo=datetime.UTC),
datetime.datetime(2024, 1, 15, 12, 0, 0, tzinfo=datetime.timezone.utc),
datetime.datetime(2024, 1, 15, 12, 0, 0, tzinfo=datetime.timezone.utc),
id="exactly_at_reference",
),
pytest.param(
datetime.datetime(1901, 1, 1, tzinfo=datetime.UTC),
datetime.datetime(1901, 1, 1, tzinfo=datetime.UTC),
datetime.datetime(1901, 1, 1, tzinfo=datetime.timezone.utc),
datetime.datetime(1901, 1, 1, tzinfo=datetime.timezone.utc),
id="year_1901_valid",
),
# Date is > reference_time
pytest.param(
datetime.datetime(2024, 1, 16, tzinfo=datetime.UTC),
datetime.datetime(2024, 1, 16, tzinfo=datetime.timezone.utc),
None,
id="future_date_day_after",
),
# date.date() in ignore_dates
pytest.param(
datetime.datetime(2024, 1, 1, 0, 0, 0, tzinfo=datetime.UTC),
datetime.datetime(2024, 1, 1, 0, 0, 0, tzinfo=datetime.timezone.utc),
None,
id="ignored_date_midnight_jan1",
),
pytest.param(
datetime.datetime(2024, 1, 1, 10, 30, 0, tzinfo=datetime.UTC),
datetime.datetime(2024, 1, 1, 10, 30, 0, tzinfo=datetime.timezone.utc),
None,
id="ignored_date_midday_jan1",
),
pytest.param(
datetime.datetime(2024, 12, 25, 15, 0, 0, tzinfo=datetime.UTC),
datetime.datetime(2024, 12, 25, 15, 0, 0, tzinfo=datetime.timezone.utc),
None,
id="ignored_date_dec25_future",
),
# date.year <= 1900
pytest.param(
datetime.datetime(1899, 12, 31, tzinfo=datetime.UTC),
datetime.datetime(1899, 12, 31, tzinfo=datetime.timezone.utc),
None,
id="year_1899",
),
pytest.param(
datetime.datetime(1900, 1, 1, tzinfo=datetime.UTC),
datetime.datetime(1900, 1, 1, tzinfo=datetime.timezone.utc),
None,
id="year_1900_boundary",
),
@@ -176,7 +176,7 @@ class TestFilterDate:
1,
12,
0,
tzinfo=datetime.UTC,
tzinfo=datetime.timezone.utc,
)
another_ignored = datetime.datetime(
2024,
@@ -184,7 +184,7 @@ class TestFilterDate:
25,
15,
30,
tzinfo=datetime.UTC,
tzinfo=datetime.timezone.utc,
)
allowed_date = datetime.datetime(
2024,
@@ -192,7 +192,7 @@ class TestFilterDate:
2,
12,
0,
tzinfo=datetime.UTC,
tzinfo=datetime.timezone.utc,
)
assert parser._filter_date(ignored_date) is None
@@ -204,7 +204,7 @@ class TestFilterDate:
regex_parser: RegexDateParserPlugin,
) -> None:
"""Should work with timezone-aware datetimes."""
date_utc = datetime.datetime(2024, 1, 10, 12, 0, tzinfo=datetime.UTC)
date_utc = datetime.datetime(2024, 1, 10, 12, 0, tzinfo=datetime.timezone.utc)
result = regex_parser._filter_date(date_utc)
@@ -221,8 +221,8 @@ class TestRegexDateParser:
"report-2023-12-25.txt",
"Event recorded on 25/12/2022.",
[
datetime.datetime(2023, 12, 25, tzinfo=datetime.UTC),
datetime.datetime(2022, 12, 25, tzinfo=datetime.UTC),
datetime.datetime(2023, 12, 25, tzinfo=datetime.timezone.utc),
datetime.datetime(2022, 12, 25, tzinfo=datetime.timezone.utc),
],
id="filename-y-m-d_and_content-d-m-y",
),
@@ -230,8 +230,8 @@ class TestRegexDateParser:
"img_2023.01.02.jpg",
"Taken on 01/02/2023",
[
datetime.datetime(2023, 1, 2, tzinfo=datetime.UTC),
datetime.datetime(2023, 2, 1, tzinfo=datetime.UTC),
datetime.datetime(2023, 1, 2, tzinfo=datetime.timezone.utc),
datetime.datetime(2023, 2, 1, tzinfo=datetime.timezone.utc),
],
id="ambiguous-dates-respect-orders",
),
@@ -239,7 +239,7 @@ class TestRegexDateParser:
"notes.txt",
"bad date 99/99/9999 and 25/12/2022",
[
datetime.datetime(2022, 12, 25, tzinfo=datetime.UTC),
datetime.datetime(2022, 12, 25, tzinfo=datetime.timezone.utc),
],
id="parse-exception-skips-bad-and-yields-good",
),
@@ -275,24 +275,24 @@ class TestRegexDateParser:
or "2023.12.25" in date_string
or "2023-12-25" in date_string
):
return datetime.datetime(2023, 12, 25, tzinfo=datetime.UTC)
return datetime.datetime(2023, 12, 25, tzinfo=datetime.timezone.utc)
# content DMY 25/12/2022
if "25/12/2022" in date_string or "25-12-2022" in date_string:
return datetime.datetime(2022, 12, 25, tzinfo=datetime.UTC)
return datetime.datetime(2022, 12, 25, tzinfo=datetime.timezone.utc)
# filename YMD 2023.01.02
if "2023.01.02" in date_string or "2023-01-02" in date_string:
return datetime.datetime(2023, 1, 2, tzinfo=datetime.UTC)
return datetime.datetime(2023, 1, 2, tzinfo=datetime.timezone.utc)
# ambiguous 01/02/2023 -> respect DATE_ORDER setting
if "01/02/2023" in date_string:
if date_order == "DMY":
return datetime.datetime(2023, 2, 1, tzinfo=datetime.UTC)
return datetime.datetime(2023, 2, 1, tzinfo=datetime.timezone.utc)
if date_order == "YMD":
return datetime.datetime(2023, 1, 2, tzinfo=datetime.UTC)
return datetime.datetime(2023, 1, 2, tzinfo=datetime.timezone.utc)
# fallback
return datetime.datetime(2023, 2, 1, tzinfo=datetime.UTC)
return datetime.datetime(2023, 2, 1, tzinfo=datetime.timezone.utc)
# simulate parse failure for malformed input
if "99/99/9999" in date_string or "bad date" in date_string:
@@ -328,7 +328,7 @@ class TestRegexDateParser:
12,
0,
0,
tzinfo=datetime.UTC,
tzinfo=datetime.timezone.utc,
),
filename_date_order="YMD",
content_date_order="DMY",
@@ -344,13 +344,13 @@ class TestRegexDateParser:
) -> datetime.datetime | None:
if "10/12/2023" in date_string or "10-12-2023" in date_string:
# ignored date
return datetime.datetime(2023, 12, 10, tzinfo=datetime.UTC)
return datetime.datetime(2023, 12, 10, tzinfo=datetime.timezone.utc)
if "01/02/2024" in date_string or "01-02-2024" in date_string:
# future relative to reference_time -> filtered
return datetime.datetime(2024, 2, 1, tzinfo=datetime.UTC)
return datetime.datetime(2024, 2, 1, tzinfo=datetime.timezone.utc)
if "05/01/2023" in date_string or "05-01-2023" in date_string:
# valid
return datetime.datetime(2023, 1, 5, tzinfo=datetime.UTC)
return datetime.datetime(2023, 1, 5, tzinfo=datetime.timezone.utc)
return None
mocker.patch(target, side_effect=fake_parse)
@@ -358,7 +358,7 @@ class TestRegexDateParser:
content = "Ignored: 10/12/2023, Future: 01/02/2024, Keep: 05/01/2023"
results = list(parser.parse("whatever.txt", content))
assert results == [datetime.datetime(2023, 1, 5, tzinfo=datetime.UTC)]
assert results == [datetime.datetime(2023, 1, 5, tzinfo=datetime.timezone.utc)]
def test_parse_handles_no_matches_and_returns_empty_list(
self,
@@ -392,7 +392,7 @@ class TestRegexDateParser:
12,
0,
0,
tzinfo=datetime.UTC,
tzinfo=datetime.timezone.utc,
),
filename_date_order=None,
content_date_order="DMY",
@@ -409,9 +409,9 @@ class TestRegexDateParser:
) -> datetime.datetime | None:
# return distinct datetimes so we can tell which source was parsed
if "25/12/2022" in date_string:
return datetime.datetime(2022, 12, 25, tzinfo=datetime.UTC)
return datetime.datetime(2022, 12, 25, tzinfo=datetime.timezone.utc)
if "2023-12-25" in date_string:
return datetime.datetime(2023, 12, 25, tzinfo=datetime.UTC)
return datetime.datetime(2023, 12, 25, tzinfo=datetime.timezone.utc)
return None
mock = mocker.patch(target, side_effect=fake_parse)
@@ -429,5 +429,5 @@ class TestRegexDateParser:
assert "25/12/2022" in called_date_string
# And the parser should have yielded the corresponding datetime
assert results == [
datetime.datetime(2022, 12, 25, tzinfo=datetime.UTC),
datetime.datetime(2022, 12, 25, tzinfo=datetime.timezone.utc),
]

View File

@@ -1395,10 +1395,7 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
{
"documents": [self.doc2.id],
"method": "edit_pdf",
"parameters": {
"operations": [{"page": 1}],
"source_mode": "explicit_selection",
},
"parameters": {"operations": [{"page": 1}]},
},
),
content_type="application/json",
@@ -1410,7 +1407,6 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
args, kwargs = m.call_args
self.assertCountEqual(args[0], [self.doc2.id])
self.assertEqual(kwargs["operations"], [{"page": 1}])
self.assertEqual(kwargs["source_mode"], "explicit_selection")
self.assertEqual(kwargs["user"], self.user)
def test_edit_pdf_invalid_params(self) -> None:
@@ -1576,24 +1572,6 @@ class TestBulkEditAPI(DirectoriesMixin, APITestCase):
response.content,
)
# invalid source mode
response = self.client.post(
"/api/documents/bulk_edit/",
json.dumps(
{
"documents": [self.doc2.id],
"method": "edit_pdf",
"parameters": {
"operations": [{"page": 1}],
"source_mode": "not_a_mode",
},
},
),
content_type="application/json",
)
self.assertEqual(response.status_code, status.HTTP_400_BAD_REQUEST)
self.assertIn(b"Invalid source_mode", response.content)
@mock.patch("documents.serialisers.bulk_edit.edit_pdf")
def test_edit_pdf_page_out_of_bounds(self, m) -> None:
"""

View File

@@ -405,9 +405,7 @@ class TestBulkEdit(DirectoriesMixin, TestCase):
self.assertTrue(Document.objects.filter(id=self.doc1.id).exists())
self.assertFalse(Document.objects.filter(id=version.id).exists())
def test_resolve_root_and_source_doc_latest_version_prefers_newest_version(
self,
) -> None:
def test_get_root_and_current_doc_mapping(self) -> None:
version1 = Document.objects.create(
checksum="B-v1",
title="B version 1",
@@ -419,14 +417,18 @@ class TestBulkEdit(DirectoriesMixin, TestCase):
root_document=self.doc2,
)
root_doc, source_doc = bulk_edit._resolve_root_and_source_doc(
self.doc2,
source_mode="latest_version",
root_ids_by_doc_id = bulk_edit._get_root_ids_by_doc_id(
[self.doc2.id, version1.id, version2.id],
)
self.assertEqual(root_ids_by_doc_id[self.doc2.id], self.doc2.id)
self.assertEqual(root_ids_by_doc_id[version1.id], self.doc2.id)
self.assertEqual(root_ids_by_doc_id[version2.id], self.doc2.id)
self.assertEqual(root_doc.id, self.doc2.id)
self.assertEqual(source_doc.id, version2.id)
self.assertNotEqual(source_doc.id, version1.id)
root_docs, current_docs = bulk_edit._get_root_and_current_docs_by_root_id(
{self.doc2.id},
)
self.assertEqual(root_docs[self.doc2.id].id, self.doc2.id)
self.assertEqual(current_docs[self.doc2.id].id, version2.id)
@mock.patch("documents.tasks.bulk_update_documents.delay")
def test_set_permissions(self, m) -> None:
@@ -660,33 +662,6 @@ class TestPDFActions(DirectoriesMixin, TestCase):
self.assertEqual(result, "OK")
@mock.patch("pikepdf.open")
@mock.patch("documents.tasks.consume_file.s")
def test_merge_uses_latest_version_source_for_root_selection(
self,
mock_consume_file,
mock_open_pdf,
) -> None:
version_file = self.dirs.scratch_dir / "sample2_version_merge.pdf"
shutil.copy(self.doc2.source_path, version_file)
version = Document.objects.create(
checksum="B-v1",
title="B version 1",
root_document=self.doc2,
filename=version_file,
mime_type="application/pdf",
)
fake_pdf = mock.MagicMock()
fake_pdf.pdf_version = "1.7"
fake_pdf.pages = [mock.Mock()]
mock_open_pdf.return_value.__enter__.return_value = fake_pdf
result = bulk_edit.merge([self.doc2.id])
self.assertEqual(result, "OK")
mock_open_pdf.assert_called_once_with(str(version.source_path))
mock_consume_file.assert_not_called()
@mock.patch("documents.bulk_edit.delete.si")
@mock.patch("documents.tasks.consume_file.s")
def test_merge_and_delete_originals(
@@ -895,36 +870,6 @@ class TestPDFActions(DirectoriesMixin, TestCase):
self.assertEqual(result, "OK")
@mock.patch("documents.bulk_edit.group")
@mock.patch("pikepdf.open")
@mock.patch("documents.tasks.consume_file.s")
def test_split_uses_latest_version_source_for_root_selection(
self,
mock_consume_file,
mock_open_pdf,
mock_group,
) -> None:
version_file = self.dirs.scratch_dir / "sample2_version_split.pdf"
shutil.copy(self.doc2.source_path, version_file)
version = Document.objects.create(
checksum="B-v1",
title="B version 1",
root_document=self.doc2,
filename=version_file,
mime_type="application/pdf",
)
fake_pdf = mock.MagicMock()
fake_pdf.pages = [mock.Mock(), mock.Mock()]
mock_open_pdf.return_value.__enter__.return_value = fake_pdf
mock_group.return_value.delay.return_value = None
result = bulk_edit.split([self.doc2.id], [[1], [2]])
self.assertEqual(result, "OK")
mock_open_pdf.assert_called_once_with(version.source_path)
mock_consume_file.assert_not_called()
mock_group.return_value.delay.assert_not_called()
@mock.patch("documents.bulk_edit.delete.si")
@mock.patch("documents.tasks.consume_file.s")
@mock.patch("documents.bulk_edit.chord")
@@ -1096,34 +1041,6 @@ class TestPDFActions(DirectoriesMixin, TestCase):
self.assertIsNotNone(overrides)
self.assertEqual(result, "OK")
@mock.patch("documents.data_models.magic.from_file", return_value="application/pdf")
@mock.patch("documents.tasks.consume_file.delay")
@mock.patch("pikepdf.open")
def test_rotate_explicit_selection_uses_root_source_when_root_selected(
self,
mock_open,
mock_consume_delay,
mock_magic,
):
Document.objects.create(
checksum="B-v1",
title="B version 1",
root_document=self.doc2,
)
fake_pdf = mock.MagicMock()
fake_pdf.pages = [mock.Mock()]
mock_open.return_value.__enter__.return_value = fake_pdf
result = bulk_edit.rotate(
[self.doc2.id],
90,
source_mode="explicit_selection",
)
self.assertEqual(result, "OK")
mock_open.assert_called_once_with(self.doc2.source_path)
mock_consume_delay.assert_called_once()
@mock.patch("documents.tasks.consume_file.delay")
@mock.patch("pikepdf.Pdf.save")
@mock.patch("documents.data_models.magic.from_file", return_value="application/pdf")
@@ -1148,34 +1065,6 @@ class TestPDFActions(DirectoriesMixin, TestCase):
self.assertIsNotNone(overrides)
self.assertEqual(result, "OK")
@mock.patch("documents.data_models.magic.from_file", return_value="application/pdf")
@mock.patch("documents.tasks.consume_file.delay")
@mock.patch("pikepdf.open")
def test_delete_pages_explicit_selection_uses_root_source_when_root_selected(
self,
mock_open,
mock_consume_delay,
mock_magic,
):
Document.objects.create(
checksum="B-v1",
title="B version 1",
root_document=self.doc2,
)
fake_pdf = mock.MagicMock()
fake_pdf.pages = [mock.Mock(), mock.Mock()]
mock_open.return_value.__enter__.return_value = fake_pdf
result = bulk_edit.delete_pages(
[self.doc2.id],
[1],
source_mode="explicit_selection",
)
self.assertEqual(result, "OK")
mock_open.assert_called_once_with(self.doc2.source_path)
mock_consume_delay.assert_called_once()
@mock.patch("documents.tasks.consume_file.delay")
@mock.patch("pikepdf.Pdf.save")
def test_delete_pages_with_error(self, mock_pdf_save, mock_consume_delay):
@@ -1324,40 +1213,6 @@ class TestPDFActions(DirectoriesMixin, TestCase):
self.assertTrue(str(consumable.original_file).endswith("_edited.pdf"))
self.assertIsNotNone(overrides)
@mock.patch("documents.data_models.magic.from_file", return_value="application/pdf")
@mock.patch("documents.tasks.consume_file.delay")
@mock.patch("pikepdf.new")
@mock.patch("pikepdf.open")
def test_edit_pdf_explicit_selection_uses_root_source_when_root_selected(
self,
mock_open,
mock_new,
mock_consume_delay,
mock_magic,
):
Document.objects.create(
checksum="B-v1",
title="B version 1",
root_document=self.doc2,
)
fake_pdf = mock.MagicMock()
fake_pdf.pages = [mock.Mock()]
mock_open.return_value.__enter__.return_value = fake_pdf
output_pdf = mock.MagicMock()
output_pdf.pages = []
mock_new.return_value = output_pdf
result = bulk_edit.edit_pdf(
[self.doc2.id],
operations=[{"page": 1}],
update_document=True,
source_mode="explicit_selection",
)
self.assertEqual(result, "OK")
mock_open.assert_called_once_with(self.doc2.source_path)
mock_consume_delay.assert_called_once()
@mock.patch("documents.bulk_edit.group")
@mock.patch("documents.tasks.consume_file.s")
def test_edit_pdf_without_metadata(
@@ -1478,34 +1333,6 @@ class TestPDFActions(DirectoriesMixin, TestCase):
self.assertEqual(consumable.root_document_id, doc.id)
self.assertIsNotNone(overrides)
@mock.patch("documents.data_models.magic.from_file", return_value="application/pdf")
@mock.patch("documents.tasks.consume_file.delay")
@mock.patch("pikepdf.open")
def test_remove_password_explicit_selection_uses_root_source_when_root_selected(
self,
mock_open,
mock_consume_delay,
mock_magic,
) -> None:
Document.objects.create(
checksum="A-v1",
title="A version 1",
root_document=self.doc1,
)
fake_pdf = mock.MagicMock()
mock_open.return_value.__enter__.return_value = fake_pdf
result = bulk_edit.remove_password(
[self.doc1.id],
password="secret",
update_document=True,
source_mode="explicit_selection",
)
self.assertEqual(result, "OK")
mock_open.assert_called_once_with(self.doc1.source_path, password="secret")
mock_consume_delay.assert_called_once()
@mock.patch("documents.bulk_edit.chord")
@mock.patch("documents.bulk_edit.group")
@mock.patch("documents.tasks.consume_file.s")

View File

@@ -726,14 +726,6 @@ class TestConsumer(
self.assertIsNotNone(root_doc)
assert root_doc is not None
root_storage_path = StoragePath.objects.create(
name="version-root-path",
path="root/{{title}}",
)
root_doc.storage_path = root_storage_path
root_doc.archive_serial_number = 42
root_doc.save()
actor = User.objects.create_user(
username="actor",
email="actor@example.com",
@@ -770,7 +762,7 @@ class TestConsumer(
)
consumer.setup()
try:
self.assertEqual(consumer.filename, version_file.name)
self.assertTrue(consumer.filename.endswith("_v0.pdf"))
consumer.run()
finally:
consumer.cleanup()
@@ -780,10 +772,8 @@ class TestConsumer(
version = versions.first()
assert version is not None
assert version.original_filename is not None
self.assertEqual(version.version_index, 1)
self.assertEqual(version.version_label, "v2")
self.assertIsNone(version.archive_serial_number)
self.assertEqual(version.original_filename, version_file.name)
self.assertTrue(version.original_filename.endswith("_v0.pdf"))
self.assertTrue(bool(version.content))
@override_settings(AUDIT_LOG_ENABLED=True)
@@ -832,7 +822,7 @@ class TestConsumer(
)
consumer.setup()
try:
self.assertEqual(consumer.filename, "valid_pdf_version-upload")
self.assertEqual(consumer.filename, "valid_pdf_version-upload_v0")
consumer.run()
finally:
consumer.cleanup()
@@ -842,67 +832,9 @@ class TestConsumer(
)
self.assertIsNotNone(version)
assert version is not None
self.assertEqual(version.version_index, 1)
self.assertEqual(version.original_filename, "valid_pdf_version-upload")
self.assertEqual(version.original_filename, "valid_pdf_version-upload_v0")
self.assertTrue(bool(version.content))
@override_settings(AUDIT_LOG_ENABLED=True)
@mock.patch("documents.consumer.load_classifier")
def test_consume_version_index_monotonic_after_version_deletion(self, m) -> None:
m.return_value = MagicMock()
with self.get_consumer(self.get_test_file()) as consumer:
consumer.run()
root_doc = Document.objects.first()
self.assertIsNotNone(root_doc)
assert root_doc is not None
def consume_version(version_file: Path) -> Document:
status = DummyProgressManager(version_file.name, None)
overrides = DocumentMetadataOverrides()
doc = ConsumableDocument(
DocumentSource.ApiUpload,
original_file=version_file,
root_document_id=root_doc.pk,
)
preflight = ConsumerPreflightPlugin(
doc,
overrides,
status, # type: ignore[arg-type]
self.dirs.scratch_dir,
"task-id",
)
preflight.setup()
preflight.run()
consumer = ConsumerPlugin(
doc,
overrides,
status, # type: ignore[arg-type]
self.dirs.scratch_dir,
"task-id",
)
consumer.setup()
try:
consumer.run()
finally:
consumer.cleanup()
version = (
Document.objects.filter(root_document=root_doc).order_by("-id").first()
)
assert version is not None
return version
v1 = consume_version(self.get_test_file2())
self.assertEqual(v1.version_index, 1)
v1.delete()
# The next version should have version_index 2, even though version_index 1 was deleted
v2 = consume_version(self.get_test_file())
self.assertEqual(v2.version_index, 2)
@mock.patch("documents.consumer.load_classifier")
def testClassifyDocument(self, m) -> None:
correspondent = Correspondent.objects.create(

View File

@@ -77,58 +77,6 @@ class TestFileHandling(DirectoriesMixin, FileSystemAssertsMixin, TestCase):
settings.ORIGINALS_DIR / "test" / "test.pdf",
)
@override_settings(FILENAME_FORMAT=None)
def test_root_storage_path_change_updates_version_files(self) -> None:
old_storage_path = StoragePath.objects.create(
name="old-path",
path="old/{{title}}",
)
new_storage_path = StoragePath.objects.create(
name="new-path",
path="new/{{title}}",
)
root_doc = Document.objects.create(
title="rootdoc",
mime_type="application/pdf",
checksum="root-checksum",
storage_path=old_storage_path,
)
version_doc = Document.objects.create(
title="version-title",
mime_type="application/pdf",
checksum="version-checksum",
root_document=root_doc,
version_index=1,
)
Document.objects.filter(pk=root_doc.pk).update(
filename=generate_filename(root_doc),
)
Document.objects.filter(pk=version_doc.pk).update(
filename=generate_filename(version_doc),
)
root_doc.refresh_from_db()
version_doc.refresh_from_db()
create_source_path_directory(root_doc.source_path)
Path(root_doc.source_path).touch()
create_source_path_directory(version_doc.source_path)
Path(version_doc.source_path).touch()
root_doc.storage_path = new_storage_path
root_doc.save()
root_doc.refresh_from_db()
version_doc.refresh_from_db()
self.assertEqual(root_doc.filename, "new/rootdoc.pdf")
self.assertEqual(version_doc.filename, "new/rootdoc_v1.pdf")
self.assertIsFile(root_doc.source_path)
self.assertIsFile(version_doc.source_path)
self.assertIsNotFile(settings.ORIGINALS_DIR / "old" / "rootdoc.pdf")
self.assertIsNotFile(settings.ORIGINALS_DIR / "old" / "rootdoc_v1.pdf")
@override_settings(FILENAME_FORMAT="{correspondent}/{correspondent}")
def test_file_renaming_missing_permissions(self) -> None:
document = Document()
@@ -388,11 +336,7 @@ class TestFileHandling(DirectoriesMixin, FileSystemAssertsMixin, TestCase):
added=d1,
)
# Account for 3.14 padding changes
expected_year: str = d1.strftime("%Y")
expected_filename: Path = Path(f"{expected_year}-01-09.pdf")
self.assertEqual(generate_filename(doc1), expected_filename)
self.assertEqual(generate_filename(doc1), Path("1232-01-09.pdf"))
doc1.added = timezone.make_aware(datetime.datetime(2020, 11, 16, 1, 1, 1))
@@ -1278,94 +1222,6 @@ class TestFilenameGeneration(DirectoriesMixin, TestCase):
Path("logs.pdf"),
)
@override_settings(FILENAME_FORMAT="{title}")
def test_version_index_suffix_for_template_filename(self) -> None:
root_doc = Document.objects.create(
title="the_doc",
mime_type="application/pdf",
checksum="root-checksum",
)
version_doc = Document.objects.create(
title="the_doc",
mime_type="application/pdf",
checksum="version-checksum",
root_document=root_doc,
version_index=1,
)
self.assertEqual(generate_filename(version_doc), Path("the_doc_v1.pdf"))
self.assertEqual(
generate_filename(version_doc, counter=1),
Path("the_doc_v1_01.pdf"),
)
@override_settings(FILENAME_FORMAT=None)
def test_version_index_suffix_for_default_filename(self) -> None:
root_doc = Document.objects.create(
title="root",
mime_type="text/plain",
checksum="root-checksum",
)
version_doc = Document.objects.create(
title="root",
mime_type="text/plain",
checksum="version-checksum",
root_document=root_doc,
version_index=2,
)
self.assertEqual(
generate_filename(version_doc),
Path(f"{root_doc.pk:07d}_v2.txt"),
)
self.assertEqual(
generate_filename(version_doc, archive_filename=True),
Path(f"{root_doc.pk:07d}_v2.pdf"),
)
@override_settings(FILENAME_FORMAT="{original_name}")
def test_version_index_suffix_with_original_name_placeholder(self) -> None:
root_doc = Document.objects.create(
title="root",
mime_type="application/pdf",
checksum="root-checksum",
original_filename="root-upload.pdf",
)
version_doc = Document.objects.create(
title="root",
mime_type="application/pdf",
checksum="version-checksum",
root_document=root_doc,
version_index=1,
original_filename="version-upload.pdf",
)
self.assertEqual(generate_filename(version_doc), Path("root-upload_v1.pdf"))
def test_version_index_suffix_with_storage_path(self) -> None:
storage_path = StoragePath.objects.create(
name="vtest",
path="folder/{{title}}",
)
root_doc = Document.objects.create(
title="storage_doc",
mime_type="application/pdf",
checksum="root-checksum",
storage_path=storage_path,
)
version_doc = Document.objects.create(
title="version_title_should_not_be_used",
mime_type="application/pdf",
checksum="version-checksum",
root_document=root_doc,
version_index=3,
)
self.assertEqual(
generate_filename(version_doc),
Path("folder/storage_doc_v3.pdf"),
)
@override_settings(
FILENAME_FORMAT="XX{correspondent}/{title}",
FILENAME_FORMAT_REMOVE_NONE=True,

View File

@@ -21,7 +21,7 @@ class TestDateLocalization:
14,
30,
5,
tzinfo=datetime.UTC,
tzinfo=datetime.timezone.utc,
)
TEST_DATETIME_STRING: str = "2023-10-26T14:30:05+00:00"

View File

@@ -4698,7 +4698,7 @@ class TestDateWorkflowLocalization(
14,
30,
5,
tzinfo=datetime.UTC,
tzinfo=datetime.timezone.utc,
)
@pytest.mark.parametrize(

View File

@@ -1,7 +1,7 @@
from __future__ import annotations
from dataclasses import dataclass
from enum import StrEnum
from enum import Enum
from typing import TYPE_CHECKING
from typing import Any
@@ -11,7 +11,7 @@ if TYPE_CHECKING:
from django.http import HttpRequest
class VersionResolutionError(StrEnum):
class VersionResolutionError(str, Enum):
INVALID = "invalid"
NOT_FOUND = "not_found"

File diff suppressed because it is too large Load Diff

747
uv.lock generated

File diff suppressed because it is too large Load Diff