Both files exercise documents.search._schema exclusively (build_schema,
schema_fingerprint, needs_rebuild, open_or_rebuild_index) with no
dependency on whoosh-compat query routing, so they belong with the
schema/field-registry work rather than PR2's query rewrite.
Collapse the TEXT/KEYWORD/U64/DATE/DATETIME if/elif chain into a
FieldKind -> (schema kind, tokenizer) lookup table. JSON stays an
explicit branch since it can emit a second, synthetic notes_text
descriptor.
* Perf: batch the repeated lookups in modify_custom_fields
`modify_custom_fields()` re-resolved the same objects inside its
per-document loop: `custom_fields.get(id=field_id)` re-ran a CustomField
query for every document, and doc link fields called
`Document.objects.get(id=doc_id)` a second time for a document that was
already known.
Resolve both up front with `in_bulk()` and hand the resolved objects to
`update_or_create()` rather than bare ids. Passing the objects also
populates the FK cache on the newly created instance, so auditlog's
post_save receiver touching `.document`/`.field` no longer costs a reload
per row. The document map defers `content`, the one field here that is
both large and unused. The symmetrical-link removal pass and
`remove_doclink()` get `select_related()` for the same auditlog reason.
Measured over 50 documents, sqlite, audit log enabled:
before after
add 3 string fields 1502 1054
update 1 string field 451 403
add doc link 851 653
remove doc link 604 354
`update_or_create()` is kept as-is. Dropping it for a hand-rolled
get-or-construct loop removes a further ~4 statements per row, but those
are the SAVEPOINT/RELEASE pairs of its `transaction.atomic()`, and the
`select_for_update()` and IntegrityError fallback that go with them. The
(document, field) unique constraint depends on that when two bulk edits
overlap, and the wall clock did not move to pay for it (331 ms vs 310 ms
for the string case above).
Also normalises the field ids to int once at the top so the old dict API,
whose keys may arrive as strings, indexes the resolved map correctly.
The `if custom_field:` branch it replaces was dead: `.get()` raises
DoesNotExist, it never returns None.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Fixes the comment
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
A slow writer might create a zero byte file, then take longer than the window to
finish the write. We would then queue a zero byte file for consumption and race the writer
to most likely fail due to still being empty. Instead, drop zero size files at yeild time.
The slow writer may or may not finish, but if it does, there will be a Change.modified
event fired again
This brings users without an embedding backend configured to closer
parity with those who do. Reuse the search backend to locate similar
documents and use them to provide the LLM with the better suggestion pool
to draw from
process_mail_accounts had no guard against a scheduled run still being
in progress when the next one fires. Skip a run outright
if another MAIL_FETCH task is already PENDING/STARTED.
TagSerializer.get_children() built a full nested TagSerializer(many=True)
for every tag, even when it had zero children, likely the common case for
most tags and maybe even most installs. Constructing a DRF ModelSerializer isn't
free (field introspection, deepcopy of declared fields, i18n lookups
all re-run per instantiation), so this scaled GET /api/tags/ linearly
with tag count in pure Python overhead, unrelated to SQL query count.
* fix(search): resolve index-write permissions and effective content in bulk
Add WriteBatch.add_or_update_ids() and use it in bulk_update_documents
and trash restore, cutting index writes from ~8 queries per document
to a constant handful per batch
* Always these new ones with xdist, try a better condition
PR #12741 sets the result backend for Celery to Redis, but forget to carryover
transport option `global_keyprefix`. This resulted keys with prefix
`celery-task-meta-` prefix to be created.
This fix unbreaks strict Redis ACLs that allow a single prefix.
* Fix: skip vector store document id filter for unrestricted chat users
ChatStreamingView built an IN filter from every permitted document id
for the "chat over all documents" case, which exceeds the vector
store's SQLite bound-parameter safety limit on installs with more
than ~32700 documents, silently returning no context. For a user who
can see every document (an active superuser), that filter never
narrows anything, so skip it and let the retriever search the whole
index instead.
* Minor improvements from a Claude review
* When a user is unrestricted chatting, still exclude trashed documents using a 'NOT IN' SQL statement. Wire that up where we need it
* Update src/paperless_ai/chat.py
Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
* Fix: mail rule loops forever when all attachments are duplicates
When every attachment in a mail is rejected as a duplicate, the chord's
header tasks all fail. Celery's default task_allow_error_cb_on_chord_header
skips the error callback in that case, so no ProcessedMail row is ever
created, and the same mail is refetched and reprocessed on every poll for
as long as it stays in the rule's maximum_age window.
* Minor simplifications and cleanup
* Security: validate remote OCR endpoint against internal SSRF
Adds PAPERLESS_REMOTE_OCR_ALLOW_INTERNAL_ENDPOINTS (default true)
and validates remote_ocr_endpoint via validate_outbound_http_url
on the config serializer, matching the existing LLM endpoint handling.
* Validates te outbound url again right before use
* cover empty-value branch of validate_remote_ocr_endpoint because coverage
* re-validate remote OCR endpoint on every outbound request
stumpylog
GitHub Actions
shamoon
Philipp Defner
Berk D. Demir
dependabot[bot]andTrenton Holmes
Thomas Steinbachandshamoon