Commit Graph
71 Commits
Author SHA1 Message Date
Trenton H 2256cb3d38 Performance: batch permission assignment in bulk set_permissions (#13806)
* Perf: batch guardian permission assignment in bulk-edit

bulk_edit.set_permissions and BulkEditObjectPermissionsView both
looped documents/objects and called set_permissions_for_object per
object, which itself calls guardian's assign_perm/remove_perm once
per (object, user) pair -- ~10-20+ queries per object, scaling with
selection size.

Added set_permissions_for_objects, a bulk equivalent that resolves
existing permission holders once across the whole batch (not once per
object) and applies changes with a small, batch-size-independent
number of queries per action instead of one per (object, user) pair.

* Perf: avoid unnecessary full-row fetches in batch permission assignment

set_permissions_for_objects now takes a model + pks instead of instances,
and identity filtering resolves straight to ids, so bulk-editing
permissions no longer materializes full Document/User/Group rows just to
read their pk/id. Row construction for bulk_create is also chunked to
bound peak memory for very large "apply to all" operations.

* Fix: use .distinct() for existing-grant lookup, drop flaky query-count invariant tests

.distinct() lets the database dedupe identity ids server-side instead of
transferring one row per (object, grantee) match and deduping in Python --
was the dominant cost on a large selection with existing grants.

Also replaced the two query-count-equality tests (bulk_edit and the
bulk_edit_objects API path) with plain functional-correctness checks at
both batch sizes.  Hopefully stops that flake.

* Mark empty-pks early-return in set_permissions_for_objects as no-cover

Defensive guard for an edge case (all requested pks already gone/invalid)
rather than a path normal usage exercises; matches the existing
pragma: no cover convention elsewhere in this file.

* Perf: drop speculative row-chunking in bulk permission assignment, keep the query-batching fix

* Resolve every permission action before applying any of them

This fixes the existing issue and resolves the Copilot comment

* Assert permission assignment does not scale with selection size

The two batching tests only checked that permissions came out correct at
5 and 50 objects, so reverting to the old per-object loop would still
have passed.  Check sizes as well to prevent that
2026-09-10 19:54:09 +00:00
Trenton HandClaude Opus 5 60709b8319 Performance: cut redundant per-document lookups in bulk modify_custom_fields (#13807)
* Perf: batch the repeated lookups in modify_custom_fields

`modify_custom_fields()` re-resolved the same objects inside its
per-document loop: `custom_fields.get(id=field_id)` re-ran a CustomField
query for every document, and doc link fields called
`Document.objects.get(id=doc_id)` a second time for a document that was
already known.

Resolve both up front with `in_bulk()` and hand the resolved objects to
`update_or_create()` rather than bare ids. Passing the objects also
populates the FK cache on the newly created instance, so auditlog's
post_save receiver touching `.document`/`.field` no longer costs a reload
per row. The document map defers `content`, the one field here that is
both large and unused. The symmetrical-link removal pass and
`remove_doclink()` get `select_related()` for the same auditlog reason.

Measured over 50 documents, sqlite, audit log enabled:

                        before   after
  add 3 string fields    1502    1054
  update 1 string field   451     403
  add doc link            851     653
  remove doc link         604     354

`update_or_create()` is kept as-is. Dropping it for a hand-rolled
get-or-construct loop removes a further ~4 statements per row, but those
are the SAVEPOINT/RELEASE pairs of its `transaction.atomic()`, and the
`select_for_update()` and IntegrityError fallback that go with them. The
(document, field) unique constraint depends on that when two bulk edits
overlap, and the wall clock did not move to pay for it (331 ms vs 310 ms
for the string case above).

Also normalises the field ids to int once at the top so the old dict API,
whose keys may arrive as strings, indexes the resolved map correctly.

The `if custom_field:` branch it replaces was dead: `.get()` raises
DoesNotExist, it never returns None.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* Fixes the comment

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
2026-09-09 23:47:07 +00:00
shamoon 8d1bc5dd24 Fix: prevent orphaned versions from bulk delete (#14030) 2026-09-09 08:48:59 -07:00
shamoon c93c996edf Remote ocr reprocess (#13636)
* Backend stuff for remote ocr reprocess, add to bulk edit pass in from ui settings

* Ok, frontend reprocess remote option

* Docs
2026-08-24 14:43:05 -07:00
shamoon e389298aab Enhancement: merge documents as versions (#13515) 2026-08-19 07:20:14 -07:00
shamoon 6f8e39c2e0 Fix: avoid unnecessary creating new PDF with pw removal workflow (#12948) 2026-06-07 20:30:08 +00:00
shamoon 0a6e0db186 Fix: use chord.on_error before apply_async (#12842) 2026-05-24 14:42:11 -07:00
Trenton Handshamoon bd86dca57e Fix: Password removal source file location (#12830)
Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
2026-05-19 13:52:04 -07:00
Trenton H c3b353873b use a few named tuples and data classes instead of so much unpacking (#12709) 2026-05-04 18:00:48 -07:00
Trenton H 5010f37174 Fix: avoid unnecessary close_old_connections in Celery task dispatch (#12701) 2026-05-04 06:02:28 -07:00
Trenton H fbf4e32646 Chore: Converts all call sites and test asserts to use apply_async and headers (#12591) 2026-04-20 11:40:04 -07:00
aed9abe48c Feature: Replace Whoosh with tantivy search backend (#12471)
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Antoine Mérino <3023499+Merinorus@users.noreply.github.com>
2026-04-02 12:38:22 -07:00
shamoon 866c9fd858 Fix: correct merge bulk edit indentation 2026-03-15 23:50:54 -07:00
shamoon 731448a8f9 Fixhancement: support version-specific edits (#12233) 2026-03-06 11:59:26 -08:00
shamoon ceee769e26 Feature: document file versions (#12061) 2026-02-26 16:46:54 +00:00
shamoon 6192915be7 Fixhancement: improve ASN handling with PDF operations (#11689) 2026-02-06 21:14:02 +00:00
Sebastian Steinbeißer 3b5ffbf9fa Chore(mypy): Annotate None returns for typing improvements (#11213) 2026-02-02 08:44:12 -08:00
shamoon 5b1e66be91 Feature: password removal action (#11656) 2026-01-08 21:36:11 +00:00
shamoon 7604a0b583 Fix: prevent ASN collisions for merge operations (#11634) 2025-12-19 20:05:34 -08:00
shamoonandTrenton H 4cff907ba0 Feature: Nested Tags (#10833)
---------

Co-authored-by: Trenton H <797416+stumpylog@users.noreply.github.com>
2025-09-17 21:41:39 +00:00
shamoon 44b8c4881a Fix: fix error when bulk adding empty doc link custom fields (#10832) 2025-09-11 13:19:23 -07:00
shamoon 6d72ee795f Feature: PDF editor (#10318) 2025-08-11 17:29:48 +00:00
shamoon 6dea158de9 Fix: prevent self-linking when bulk edit doc link (#9629) 2025-04-14 07:12:50 -07:00
shamoon 1e489a0666 Enhancement: add switch to allow merging non-PDFs with archive version (#9305) 2025-03-05 20:46:51 +00:00
Trenton H 827fcba277 Chore: Reduce imports for a slight memory improvement (#9217) 2025-02-24 15:06:14 -08:00
Sebastian Steinbeißer e560fa3be0 Chore: Enable ruff FBT (#8645) 2025-02-07 09:12:03 -08:00
shamoon 52ab07c673 Fix: deselect and trigger refresh for deleted documents from bulk operations with "delete originals" (#8996) 2025-02-07 00:47:50 +00:00
shamoon 54e72d5b60 Fix: also ensure symmetric doc link removal on bulk edit (#8963) 2025-01-31 07:39:22 -08:00
shamoon 5b8c9ef5fc Fix: reflect doc links in bulk modify custom fields (#8962) 2025-01-30 10:55:05 -08:00
shamoon 765bf1d11c Fix: use hard delete when bulk editing custom fields (#8740) 2025-01-14 16:47:29 -08:00
shamoon e4f69dc945 Feature: bulk edit custom field values (#8428) 2024-12-09 17:35:49 +00:00
shamoon 9c1561adfb Change: change update content to handle archive disabled (#8315) 2024-11-20 20:01:13 +00:00
Sebastian Steinbeißer 74d0c9fda5 Switch src/documents/bulk*.py from os.path to pathlib.Path (#7862)
Also:
* Ensure that the ruff PTH check remains enabled for these files and
all files added in the future.
* Add some type annotations.
2024-11-12 17:04:07 +00:00
shamoon 28fdb170bf Fix: handle uuid fields created under mariadb and Django 4 (#8034) 2024-10-28 13:54:16 +00:00
Trenton H e6f59472e4 Chore: Drop Python 3.9 support (#7774) 2024-09-26 12:22:24 -07:00
s0llvanandshamoon c92c3e224a Feature: page count (#7750)
---------

Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
2024-09-25 08:22:12 -07:00
shamoon a64d457c30 Enhancement: use request user as owner of split / merge docs (#7112) 2024-06-27 13:46:49 -07:00
shamoon f3cf608caa Fix: use temp dir for split / merge (#7105) 2024-06-26 10:46:59 -07:00
Dominik Bruhnandshamoon d1ac15baa9 Enhancement: support delete originals after split / merge (#6935)
---------

Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
2024-06-08 11:29:03 -07:00
shamoon d558367642 Change: rename redo OCR to reprocess (#6866) 2024-05-31 18:41:52 +00:00
shamoon 3e62f13f96 Enhancement: delete pages PDF action (#6772) 2024-05-22 23:01:15 +00:00
Trenton H 622f624132 Chore: Change the code formatter to Ruff (#6756)
* Changing the formatting to ruff-format

* Replaces references to black to ruff or ruff format, removes black from dependencies
2024-05-18 02:26:50 +00:00
shamoon caec0ed4d1 Fix: allow bulk edit with existing fields (#6625) 2024-05-07 14:26:07 -07:00
Trenton H 7be7185418 Handcrafts SQL queries a little more to reduce the query count and/or the amount of returned data (#6489) 2024-04-30 07:37:09 -07:00
shamoon 63e1f9f5d3 Feature: custom fields filtering & bulk editing (#6484) 2024-04-26 15:10:03 -07:00
Trenton H ae2b302962 Fix: Rotation fails due to celery chord (#6306)
Use Celery immutable signature so it does not take the result of the header task
2024-04-06 23:26:48 +00:00
shamoon 4af8070450 Feature: PDF actions - merge, split & rotate (#6094) 2024-03-25 18:41:24 -07:00
shamoon 454098630b Enhancement: mergeable bulk edit permissions (#5508) 2024-02-01 19:56:57 +00:00
Trenton H 6f163111ce Upgrades black to v23, upgrades ruff 2023-04-26 09:35:27 -07:00
Trenton H 3bcbd05252 Fixes ruff not running isort against the codebase 2023-04-26 09:35:27 -07:00