Commit Graph
2563 Commits
Author SHA1 Message Date
Trenton H 08df5db033 Fix: move files before the bulk update websocket notification 2026-11-15 15:08:29 -08:00
Trenton H 1721bd559a Fix: flush file moves before the scheduled workflow websocket notification 2026-11-15 15:08:14 -08:00
Trenton H 18f243d381 Fix: move documents when a tag is cleared through the reverse relation 2026-11-15 15:07:59 -08:00
Trenton H b2160ca860 Improves test coverage even for the exception cases 2026-11-15 13:57:40 -08:00
Trenton H 4b3673552e Fix: rename only root documents in document_renamer 2026-11-09 12:25:21 -08:00
Trenton H 6a356fddec Fix: move versions of trashed documents 2026-11-09 12:25:21 -08:00
Trenton H 119c5ed660 Fix: move documents when tags change through the reverse relation 2026-11-09 12:25:21 -08:00
Trenton H 7d02f51c7f Fix: refresh the document after placing files, before the post-consume script 2026-11-09 12:25:21 -08:00
stumpylog 5fa813b11b Trimming and simpliciation 2026-11-09 12:25:21 -08:00
stumpylog f1aac712c6 Fix: return the post-move modified time from document updates
Editing a document in a way that renames its file returned a response carrying the modified time from before the move. The deferred file move runs after the response is built and bumps modified, so the stored value was newer than the one the client received. The frontend then saw a newer server modified time and showed a spurious "document was updated" prompt whose reload discarded local edits.

DocumentViewSet.update now flushes pending file moves right after saving and before reading the document back, so the response reflects the final filename and modified time.
2026-11-09 12:25:21 -08:00
stumpylog a506effaf9 Fix: skip the thread hop for async file move scopes with nothing to move
adefer_file_moves hopped to a worker thread on every exit, even when no file move had been requested, which costs an executor round trip for most async requests.

The collector is now closed on the event loop, which touches no database, and the flush only runs through sync_to_async when something was collected. The transaction check still happens inside the flush in the sync thread. The comment above the lock-free filename precheck in move_files_for_document now gives the real reason it uses values_list(): it keeps the check off Document.global_objects.filter(), which tests patch to inject database errors and observe the mover's writes.
2026-11-09 12:25:21 -08:00
stumpylog 961dce1472 Fix: place consumed files before running the post-consume script
The consumer's storage phase opens its own file move scope, which nests under the consume task's scope. Moves requested while storing the document therefore stayed pending until the task ended, and the post-consume script could be handed a path from before the move.

The consumer now flushes the pending file moves once after the storage block, outside the media lock and the transaction, and before the post-consume script runs.
2026-11-09 12:25:21 -08:00
stumpylog bc5be28614 Fix: move files per document during scheduled workflow runs
check_scheduled_workflows runs the workflow once per matched document inside the task-wide file move scope, so every rename it triggered was deferred to the end of the task. A long run killed by the Celery time limit would then discard all of the renames.

The task now flushes the pending file moves after each document's workflow run and websocket notification, the same way bulk_update_documents does, so renames land document by document while the scope stays open.
2026-11-09 12:25:21 -08:00
stumpylog 1dce32ef9b Fix: rename and retag documents one file move at a time
document_renamer sent a fake post_save per document to trigger the move, and document_retagger moved a document once for each field it assigned.

The renamer calls move_files_for_document directly for each document, so progress reflects real work. The retagger opens a file move scope per document, so each document is moved once after all of its assignments.
2026-11-09 12:25:21 -08:00
stumpylog ab7a4e4280 Fix: keep Celery's Django task base for file move scoped tasks
Passing task_cls to Celery replaces the Django fixup's default DjangoTask, so FileMoveScopedTask, built on the plain Task, left every task without delay_on_commit. The workflow AI suggestions trigger failed with an AttributeError.

FileMoveScopedTask now subclasses DjangoTask, which only needs django.db.transaction and is safe to import before apps load. The task scope test also asserts tasks are DjangoTask instances.
2026-11-09 12:25:21 -08:00
stumpylog 4822744a79 Fix: take media.lock before the consumer's storage transaction
Consuming a new version locked the root document row with select_for_update and then waited on media.lock to place the file, the same lock ordering that let concurrent saves deadlock. On MariaDB even the version row's foreign key check holds a shared lock on the root.

The consumer's storage phase now takes media.lock before opening its transaction and runs inside its own file move scope. File placement and workflow email actions inside the transaction re-enter the lock instead of waiting, and filename follow-ups run once after commit.
2026-11-09 12:25:21 -08:00
stumpylog fadf5ed037 Fix: take media.lock before emptying the trash
Emptying the trash deleted documents inside Django's delete transaction, and the post_delete file cleanup waited on media.lock while the deleted rows were locked. A concurrent file move holding media.lock and updating one of those rows could deadlock with it.

empty_trash now holds media.lock around the delete, so the cleanup handler re-enters the lock instead of waiting while rows are locked.
2026-11-09 12:25:21 -08:00
stumpylog d84d50aa3f Fix: take media.lock before the archiver's transaction
The archiver updated the document row inside a transaction and then waited on media.lock. A concurrent file move holding media.lock and updating the same row could deadlock with it, and the unique archive filename was generated outside the lock from an instance loaded before the parse.

The archiver now acquires media.lock before opening its transaction, refreshes the document under the lock, and moves the archive and thumbnail inside that block, so it never waits on media.lock while holding the row.
2026-11-09 12:25:21 -08:00
stumpylog b77e74ebe2 Fix: move document files once per Celery task
Tasks such as bulk_update_documents moved a document's files once for its manual post_save and again for every save its workflows made, each time taking the global media lock.

All tasks now run through FileMoveScopedTask, which opens a file move scope around the task body, including when the task raises. bulk_update_documents flushes after each document so long bulk edits still make steady progress.
2026-11-09 12:25:21 -08:00
stumpylog 6515136cdf Fix: move document files once per request, after its writes commit
A tag PATCH moved files from inside the tags.set() transaction while holding the document row lock. A concurrent save holding media.lock and waiting on that row deadlocked with it, and every later save on the instance queued behind media.lock until the webserver restarted.

FileMoveMiddleware opens a file move scope around every request, in sync or async mode, so each changed document is moved once after the view returns and never while a transaction holds row locks.
2026-11-09 12:25:21 -08:00
stumpylog 96bab6b8e0 Fix: collect document file moves per scope and move once on exit
Each save of a document moved its files on the spot, so one logical change could move them several times, and moves could run inside another open transaction.

defer_file_moves() opens a ContextVar-backed scope. Inside it, move requests are recorded and the outermost scope moves each document once on exit, or on commit if a transaction is still open. A failing move is logged and does not stop the others. Outside any scope moves stay immediate.
2026-11-09 12:25:21 -08:00
stumpylog 627c53a1be Fix: extract document file moves out of the signal receiver
The file move receiver refreshed the caller's own instance from the database, which wiped unsaved in-memory changes. It ran on every m2m_changed action including pre_add and pre_remove, wrote modified even when nothing moved, and its no-move branch skipped trashed documents.

The move body now lives in documents.file_moves.move_files_for_document. It skips documents without a filename before taking media.lock, then takes the lock with a 300 second timeout, loads a fresh instance by pk, writes only when a filename column changes, and returns the instance. The receiver ignores reverse and pre_* m2m actions and copies just the final filename columns back onto the caller's instance.
2026-11-09 12:25:21 -08:00
stumpylog 26e513fbd1 Fix: make the global media lock reentrant within a thread
Every caller built its own FileLock on media.lock, so code already holding it could not call into code that takes it again without blocking itself. That made it impossible to take media.lock before opening a transaction in paths that also attach files or move files inside that transaction.

media_lock() hands out one FileLock instance per path with per-thread hold counts, so a thread re-enters while other threads and processes stay excluded. All media.lock call sites now use it.
2026-11-09 12:25:21 -08:00
stumpylog f9d0a58a41 Fix: stop full document saves from writing file path columns
A full save() of a Document instance loaded before another process moved its file wrote the old filename back. The DB then pointed at a path that no longer existed, and recovery only worked when the recomputed path happened to match the file's real location.

Document.save() now leaves filename and archive_filename out of full saves on existing rows. Writers that own those columns opt in with write_file_paths=True or an explicit update_fields. A full save of a row deleted elsewhere now raises instead of re-inserting it.
2026-11-09 12:25:21 -08:00
shamoon 0736c07d8e Fix: use filtered documents for statistics ASN max (#14710) 2026-11-07 07:52:23 -08:00
shamoon 53d9aa34a1 FIx: make config view permissions consistent (#14709) 2026-11-07 07:52:12 -08:00
5357be2d43 Performance: stop embedding every document ID in selection_data queries (#14597)
* Fix: count selection_data from one pass over the filtered documents

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* Chore: use document type ids in the bulk edit selection_data test

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* Batch search selection data queries

Co-Authored-By: Codex <267193182+codex@users.noreply.github.com>

---------

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
Co-authored-by: Codex <267193182+codex@users.noreply.github.com>
2026-11-06 18:44:18 -08:00
shamoonandCodex 8aa99a381a Performance: replace django-treenode with adjacency-list tags (#14612)
---------

Co-authored-by: Codex <267193182+codex@users.noreply.github.com>
2026-11-06 20:18:59 +00:00
shamoon fa83a4c5dc Fix: check descendant tag permissions on delete (#14687) 2026-11-04 11:05:48 -08:00
Trenton Handshamoon 3289be7383 Fix: Various small bugs found during typing updates (#14593)
* Fix: match Apple Mail tag colors case-insensitively

* Fix: reject OAuth callbacks with an unrecognized scope

* Fix: refuse to refresh OAuth tokens for non-OAuth accounts or missing refresh tokens

* Fix: parse datetime strings in the format_datetime filter and reject invalid ones

* Fix: reject a PAPERLESS_URL without a hostname

* Fix: report a database error in the system status instead of failing with a 500

* Fix: surface the database error when refreshing a document before moving its files

* Fix: report no unapplied migrations when the applied migrations cannot be read

* Fix: return the parsed value directly in format_datetime for typing

* Chore: annotate the paperless_dirs fixture in the refresh failure test

* Allow null for the latest_migration system status type

---------

Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
2026-10-31 09:32:58 -07:00
shamoon 11248f776d Fix: fix the id__none schema in the right place (#14604) 2026-10-30 21:47:12 -07:00
Trenton H 6f99ad9469 Chore: Improve idle memory via more lazy imports (#14599)
* Chore: Import dateparser lazily so processes that never parse dates skip it

* Chore: Import turbohtml lazily in the mail parser

* Chore: Import lxml lazily in the SVG validator

* Chore: Import paperless_ai.indexing lazily in document tasks

* Chore: Import paperless_ai helpers lazily in views

* Chore: Read the OCRmyPDF sRGB profile on first use instead of at import

* Chore: Import pikepdf and img2pdf lazily so startup does not load them

* Chore: Reuse the initialized Django ASGI application for HTTP routing
2026-10-30 22:25:42 +00:00
Trenton HandClaude Sonnet 5.5 08a7d1d4b8 Chore: Consolidate test sample files (#14430)
Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-29 12:43:51 -07:00
shamoon 9c4c9f2386 Enhancement: support custom fields in workflow title templates (#14564) 2026-10-29 17:40:53 +00:00
shamoon 67c194978e Enhancement: relative date custom field queries (#14549) 2026-10-29 17:25:21 +00:00
shamoon ec52490e4a Fixhancement: show better result for revoked / expired tasks (#14566) 2026-10-28 08:59:07 -07:00
shamoon d4457efe44 Fix: move re-index after workflow runs on doc edit (#14558) 2026-10-27 19:54:30 -07:00
Trenton H 4299e9b12e Chore: consolidate direct permission grant lookups into one helper (#14456)
* Chore: drop get_groups_with_only_permission

It worked around get_groups_with_perms lacking only_with_perms_in, which
django-guardian added in 3.1.0. The pinned version is newer.

* Chore: consolidate direct grant lookups into get_permission_grantees

Six places listed who holds a permission on an object, each with its own
query and result shape, and one had already drifted into expanding group
members.

* Chore: drop the unreachable empty viewer grant default

get_permission_grantees returns an entry for every requested document.
2026-10-24 13:28:25 -07:00
Trenton H 10a1a9eae8 Chore: drop get_groups_with_only_permission (#14454)
It worked around get_groups_with_perms lacking only_with_perms_in, which
django-guardian added in 3.1.0. The pinned version is newer.
2026-10-24 13:28:25 -07:00
Trenton H e7d70497c2 Chore: Consolidate to a single way to set permission on an object (#14453)
* Chore: drop set_permissions_for_object in favor of the bulk helper

The same grant rules were implemented twice, and only the bulk version
validated every action before writing, so the single-object one could leave
a partial change behind.

* Chore: type the set-permissions dict

Callers passed whatever they had, including querysets of User and Group
objects that only worked because id__in accepts a subquery.
2026-10-24 13:28:25 -07:00
Trenton H 7bdbcca051 Chore: Refactor bulk edit to remove PDF knowledge (#14445)
* Refactor: add pure pdf_ops module with real-PDF tests
* Drops tests we don't relly need + adds the typing annotation
* Now that I'm sure it works, re-work these tests to mock pdf_ops, rather than pikepdf
* Validates pages ahead of time in build_pdfs rather than during the building
* Adds the given/when/then and some additional cleanups of pytest style
* This is redundant now
2026-10-22 19:25:39 +00:00
Ben Petty 9b56dc2f61 Fix: folder export with --delete fails on a stale manifest.json.tmp (#14526) 2026-10-22 11:50:31 -07:00
shamoon 97c98ee44c Fix: fix list type for api schema document object filters (#14524) 2026-10-22 05:53:17 -07:00
Trenton H a3b358bad7 Chore: drop get_objects_for_user_owner_aware (#14457)
Drops get_objects_for_user_owner_aware. Callers now use restrict_queryset_to_visible.

The same question was answered two ways, and the guardian path did not deny
deactivated users the way permitted_object_ids does.
2026-10-21 15:46:23 +00:00
Trenton H b904881472 Chore: share one base class for the method-mapped permission classes (#14458)
* Chore: share one base class for the method-mapped permission classes

ViewDocumentsPermissions, TrashPermissions, PaperlessNotePermissions and
AcknowledgeTasksPermissions each repeated the same has_permission body and
disagreed on HTTP methods missing from their perms_map.

GlobalPermissionsByMethod now holds that logic once. HEAD is checked
against the GET entry and a method missing from the map is denied. The
task acknowledge map gains OPTIONS, now requiring change_paperlesstask
like POST instead of being open to any authenticated user. The global
statistics and system status checks also share their active and
authenticated preamble.

* Chore: type GlobalPermissionsByMethod.perms_map keys as HTTP methods

perms_map accepted any string key, so a misspelled method or a HEAD entry
would never match. Keys are now a Literal of the mappable methods, without
HEAD, and the map is typed read-only.
2026-10-20 20:32:13 -07:00
Trenton HandClaude Sonnet 5.5 c089bba32c Chore: Drop ImageMagik as a dependency (#14440)
* Mark the thumbnail constants Final

* Comment each pdftoppm option

* Feature/Chore: Replaces all usages of ImageMagik's convert binary with a simple Pillow based helper

This removes ImageMagik as a dependency entirly and all documentation is cleaned up from it.
We no longer need to install it, have a policy for it to allow PDFs and no longer shell
out to it either.

* Fix: Save WebP images losslessly when removing the alpha channel

Pillow's default WebP save is lossy, so a WebP with an alpha channel came
back from remove_alpha_channel with slightly shifted pixel values before
it reached OCR. Save WebP lossless; other formats are unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 5.5 <noreply@anthropic.com>
2026-10-20 11:22:43 -07:00
Trenton HandClaude Sonnet 5.5 a05afd654c Feature: Generate PDF thumbnails via poppler utils (#14420)
* Chore: Generate PDF thumbnails with pdftoppm and Pillow instead of ImageMagick

Replace the ImageMagick convert based PDF thumbnail path with pdftoppm and
Pillow. Keeps thumbnails at 500px wide for small and odd sized pages, covers
the qpdf repair and double failure paths, removes the unused run_convert
helper, hardens thumbnail encoding, deprecates PAPERLESS_CONVERT_MEMORY_LIMIT
and PAPERLESS_CONVERT_TMPDIR, and corrects the related docs and CI.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* Chore: Supersample the PDF thumbnail render 2x

Render the PDF page at twice the target size and downsample, with edge case
handling for small and odd sized pages, simplified helpers and tests, and
trimmed comments.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

* Fix: use a fractional thumbnail DPI so oversized pages render a bounded raster

* Deprecate the unused PAPERLESS_GS_BINARY setting too

* Warn at startup when pdftoppm is missing

* Resample PDF thumbnails once instead of twice with LANCZOS, still never enlarging

* Read PDF page size from pdfinfo so it matches pdftoppm

* Mark the thumbnail constants Final

* Comment each pdftoppm option
2026-10-20 11:22:42 -07:00
shamoon d4826f50c9 Fix: prevent doc permissions queries on schema view (#14503) 2026-10-19 22:25:52 -07:00
shamoon 20b7a116fd Fix: dont defer tag assignment when barcode will not result in a split (#14497) 2026-10-19 15:37:04 -07:00
shamoon 11c32d352e Fix: dont include doc content on global search results (#14488) 2026-10-19 08:40:05 -07:00