* Perf: batch the repeated lookups in modify_custom_fields
`modify_custom_fields()` re-resolved the same objects inside its
per-document loop: `custom_fields.get(id=field_id)` re-ran a CustomField
query for every document, and doc link fields called
`Document.objects.get(id=doc_id)` a second time for a document that was
already known.
Resolve both up front with `in_bulk()` and hand the resolved objects to
`update_or_create()` rather than bare ids. Passing the objects also
populates the FK cache on the newly created instance, so auditlog's
post_save receiver touching `.document`/`.field` no longer costs a reload
per row. The document map defers `content`, the one field here that is
both large and unused. The symmetrical-link removal pass and
`remove_doclink()` get `select_related()` for the same auditlog reason.
Measured over 50 documents, sqlite, audit log enabled:
before after
add 3 string fields 1502 1054
update 1 string field 451 403
add doc link 851 653
remove doc link 604 354
`update_or_create()` is kept as-is. Dropping it for a hand-rolled
get-or-construct loop removes a further ~4 statements per row, but those
are the SAVEPOINT/RELEASE pairs of its `transaction.atomic()`, and the
`select_for_update()` and IntegrityError fallback that go with them. The
(document, field) unique constraint depends on that when two bulk edits
overlap, and the wall clock did not move to pay for it (331 ms vs 310 ms
for the string case above).
Also normalises the field ids to int once at the top so the old dict API,
whose keys may arrive as strings, indexes the resolved map correctly.
The `if custom_field:` branch it replaces was dead: `.get()` raises
DoesNotExist, it never returns None.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Fixes the comment
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
A slow writer might create a zero byte file, then take longer than the window to
finish the write. We would then queue a zero byte file for consumption and race the writer
to most likely fail due to still being empty. Instead, drop zero size files at yeild time.
The slow writer may or may not finish, but if it does, there will be a Change.modified
event fired again
TagSerializer.get_children() built a full nested TagSerializer(many=True)
for every tag, even when it had zero children, likely the common case for
most tags and maybe even most installs. Constructing a DRF ModelSerializer isn't
free (field introspection, deepcopy of declared fields, i18n lookups
all re-run per instantiation), so this scaled GET /api/tags/ linearly
with tag count in pure Python overhead, unrelated to SQL query count.
* fix(search): resolve index-write permissions and effective content in bulk
Add WriteBatch.add_or_update_ids() and use it in bulk_update_documents
and trash restore, cutting index writes from ~8 queries per document
to a constant handful per batch
* Always these new ones with xdist, try a better condition
* Fix: skip vector store document id filter for unrestricted chat users
ChatStreamingView built an IN filter from every permitted document id
for the "chat over all documents" case, which exceeds the vector
store's SQLite bound-parameter safety limit on installs with more
than ~32700 documents, silently returning no context. For a user who
can see every document (an active superuser), that filter never
narrows anything, so skip it and let the retriever search the whole
index instead.
* Minor improvements from a Claude review
* When a user is unrestricted chatting, still exclude trashed documents using a 'NOT IN' SQL statement. Wire that up where we need it
* Update src/paperless_ai/chat.py
Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
* Security: validate remote OCR endpoint against internal SSRF
Adds PAPERLESS_REMOTE_OCR_ALLOW_INTERNAL_ENDPOINTS (default true)
and validates remote_ocr_endpoint via validate_outbound_http_url
on the config serializer, matching the existing LLM endpoint handling.
* Validates te outbound url again right before use
* cover empty-value branch of validate_remote_ocr_endpoint because coverage
* re-validate remote OCR endpoint on every outbound request
* Ok! Backend stuff for the remote ocr workflow
* Frotnend workflow stuff
* And docs
* Fix dynamic action fields thing
* Actually, fix the action dropdown thing
* Fix this validation thing, and we have to check existing actions
* Fix migration
* uses_remote_service + allow_remote to allow opt-in / out of remote OCR
* Add to parser dev docs
* remote_ocr_mode config setting
* Checks for remote_ocr_mode and fix import
* Update config.component.spec.ts
* More tests for remote_ocr_mode
* Docs for remote_ocr_mode
* Ok, wire up the remote_ocr_mode with allow_remote for consumer
* Update consumer.py
* Format remote OCR mode check tests
* Use get_choice_from_env
* Backend changes and migration for remote OCR Config
* Backend tests
* Frontend stuff, with sections
* Docs
* Update test_tesseract_parser.py
* Actually we cant use this any more, in case settings are in app config
* Dont mark entire test file for db, use a mock for empty engine settings
stumpylog
shamoon
Philipp Defner
dependabot[bot]andTrenton Holmes