set_permissions_for_objects now takes a model + pks instead of instances,
and identity filtering resolves straight to ids, so bulk-editing
permissions no longer materializes full Document/User/Group rows just to
read their pk/id. Row construction for bulk_create is also chunked to
bound peak memory for very large "apply to all" operations.
bulk_edit.set_permissions and BulkEditObjectPermissionsView both
looped documents/objects and called set_permissions_for_object per
object, which itself calls guardian's assign_perm/remove_perm once
per (object, user) pair -- ~10-20+ queries per object, scaling with
selection size.
Added set_permissions_for_objects, a bulk equivalent that resolves
existing permission holders once across the whole batch (not once per
object) and applies changes with a small, batch-size-independent
number of queries per action instead of one per (object, user) pair.
Deliberately does not use guardian's queryset-aware assign_perm:
passing a list as the target routes to bulk_assign_perm, which skips
creating a direct permission row for anyone who already has the
permission via ANY group membership (checked via
ObjectPermissionChecker.has_perm, which is group-inheritance-aware) --
unlike the single-object assign_perm this replaces, which always
ensures a direct row via get_or_create. Losing that guarantee would
mean a later revocation of the group's grant silently strips access
an admin explicitly asked to be direct. Bulk-creates rows straight
against UserObjectPermission/GroupObjectPermission instead
(ignore_conflicts=True, relying on the existing (identity, permission,
object_pk) unique constraint), which preserves the original semantics
exactly while still batching every object and identity into one query
per action. Also raises Permission.DoesNotExist for an unrecognized
action name instead of silently no-op-ing, matching the original
per-object path -- BulkEditObjectsSerializer never actually validates
action keys against the raw client-supplied permissions dict, so this
is reachable from client input, not just internal callers.
Verified via CaptureQueriesContext: query count is now identical at 5
vs. 50 documents/objects (was 1,123 queries for 20 documents on the
Document path, 2,806 for 50 tags on the BulkEditObjectPermissionsView
path, both now flat). Full documents test suite green (2,148 passed,
1 skipped).
TagsField/CorrespondentField/DocumentTypeField/StoragePathField were
plain PrimaryKeyRelatedField subclasses with no batching. When used
with many=True (only tags today: DocumentSerializer.tags,
WorkflowActionSerializer.assign_tags), DRF's ManyRelatedField resolves
each submitted id with its own query -- one query per tag on every
PATCH/PUT that sets tags.
Added BatchResolvingPrimaryKeyRelatedField as the shared base for all
four field classes and overrode many_init so the many=True form
(_BatchingManyRelatedField) resolves the whole id list with one
pk__in query, falling back to the child relation's normal per-item
validation for anything not found in that batch. Only TagsField uses
many=True today, but the fix isn't tag-specific -- if a future PR puts
many=True on one of the others, it inherits the same batching instead
of reintroducing this as a new bug.
Independent review caught a real regression: Django's IntegerFieldOverflow
guard (out-of-range int -> EmptyResultSet) only covers exact/gt/gte/lt/lte
lookups, not `in`, so an absurdly large tag id reached the batched
pk__in= query as-is and raised an unhandled OverflowError (SQLite) /
DataError (Postgres) instead of the normal 400 the original per-item
`exact` lookup produced. Guarded the batch query and fall through to
per-item resolution (which goes through the protected `exact` lookup)
on failure.
Verified via CaptureQueriesContext against a real API PATCH: 20 tags
dropped from 54 to 35 queries per request (exactly the 19 saved by
collapsing 20 individual lookups into one batched query). Full
documents/workflows/bulk-edit/retagger/custom-fields suites green
(443 passed).
* Ok! Backend stuff for the remote ocr workflow
* Frotnend workflow stuff
* And docs
* Fix dynamic action fields thing
* Actually, fix the action dropdown thing
* Fix this validation thing, and we have to check existing actions
* Fix migration
* uses_remote_service + allow_remote to allow opt-in / out of remote OCR
* Add to parser dev docs
* remote_ocr_mode config setting
* Checks for remote_ocr_mode and fix import
* Update config.component.spec.ts
* More tests for remote_ocr_mode
* Docs for remote_ocr_mode
* Ok, wire up the remote_ocr_mode with allow_remote for consumer
* Update consumer.py
* Format remote OCR mode check tests
* Use get_choice_from_env
modify_custom_fields looped documents x fields, re-.get()-ing the
CustomField queryset per iteration and Document.objects.get() per doc
for DOCUMENTLINK fields -- same shape as the earlier custom_fields
serializer N+1 (#13779), just nested one level deeper. Resolve both
into dicts once up front instead. Also pass the resolved objects
(not bare ids) to update_or_create so newly-created CustomFieldInstance
rows cache their field/document FK, avoiding a re-fetch when auditlog's
post_save receiver calls str(instance) (which touches .field.name).
docs_by_id defers `content` (the one field guaranteed both large and
unused by this function or its receivers) rather than using .only(),
since .only() would just turn the filename-generation signal's other
field access into a deferred-reload N+1.
Verified via CaptureQueriesContext: 6 docs x 4 fields dropped from 48
CustomField queries to 1; DOCUMENTLINK per-doc Document lookups dropped
from N to 0 (single batched query instead).
* Backend changes and migration for remote OCR Config
* Backend tests
* Frontend stuff, with sections
* Docs
* Update test_tesseract_parser.py
* Actually we cant use this any more, in case settings are in app config
* Dont mark entire test file for db, use a mock for empty engine settings
send_websocket_document_updated calls document.refresh_from_db()
before building overrides, which drops the custom_fields prefetch
(and its select_related("field")) set up by the view's queryset.
DocumentMetadataOverrides.from_document() then lazily reloads field
once per custom field instance. Since from_document() can't rely on
the caller having a prefetched document, select_related explicitly at
the point of use instead.
drf-writable-nested's update_or_create_reverse_relations rebuilds a
fresh serializer -- and fresh field instances -- per custom_fields item
while matching existing vs. new instances during save(), so the
per-instance lookup cache alone only helped the first validation pass.
It passes the same context dict (by reference) to every one of those
serializers, so stash resolved CustomField objects there instead:
later passes reuse them for free rather than re-querying.
DocumentSerializer.custom_fields validates each item's field id via a
plain PrimaryKeyRelatedField, which issues one SELECT per custom field
per validation pass (discussion #13690). Batch-resolve all field ids in
one query and cache them on the field instance so per-item validation
is free instead of re-querying.
AI Suggestions previously invented near-duplicate metadata because the classification
prompt had no knowledge of the installation's own taxonomy. This surfaces
a small, ranked, permission-filtered set of existing tags/document
types/correspondents/storage paths - drawn from the document's RAG
neighbors plus its own already-assigned metadata - so the model prefers
reusing what already exists.
The LLM response schema now returns existing_ids (IDs of reused
candidates) separately from new_names (genuinely new suggestions).
Only new_names goes through localization and fuzzy name-matching;
existing_ids is resolved deterministically and never touched by the
localization pass, so exact matches can no longer be silently
corrupted by translation.
* Feature: Allow configuring the compression type and compression levels during export
Building on the zip export improvements, this now allows users to further configure the
zip to fit their needs. A simple stored zip for speed, or a high compression zstd for
the smallest archive. Full validation of the method and levels at the command line
Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
Added a new --url argument to specify the base URL of the Paperless instance, allowing matched documents to be displayed as clickable links. Updated the logic to fetch document titles based on the presence of the base URL.
* Have the remote parser respect the provided produce_archive_file setting, as already determined via the consumer checks
* Updates the documentation to be correct about the respecting now
* merge conflict fixing
In tracemalloc based profiling, not materializing the whole Document list
reduced memory to approximately 20% of the baseline, with a peak memory
that scaled with the library size. Now, the lazt queryset is used and only
the needed pk value is actually contributing to memory
ProcessedMailViewSet.bulk_delete checked permissions inside the delete loop, so an unpermitted id returned 403 only after the mails ahead of it had already been deleted. Resolve the permitted set once via permitted_object_ids and reject before deleting anything, which also drops the per-mail permission queries.
stumpylog
GitHub Actions
shamoonandGitHub
Max TruxaandGitHub
JaydenandGitHub