The abstract method had no return annotation, so pyrefly inferred -> None
and flagged both DirectoryExportSink.stream and ZipExportSink.stream as
incompatible overrides.
The implementation plan for Task 2 diverged from the design spec
(export-sink-architecture-design.md), leaving ExportSink as a plain
class with NotImplementedError bodies instead of the specified
AbstractContextManager subclass. Use abc.ABC + @abstractmethod so a
concrete sink missing a required method fails at instantiation
rather than at first call.
* perf: resolve permitted_document_ids once before email/share loops
Replaces per-document has_perms_owner_aware calls in the email-document
action and bulk share-link-bundle creation with a single
permitted_document_ids(request.user) resolution before the loop,
reducing DB round-trips while preserving identical permission
semantics (including the per-document error message on the bundle
endpoint).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GRp4kf1mdn9ruv81zWAmh2
* perf: resolve permitted_document_ids(perm=change_document) once before bulk-edit loops
Migrates the bulk document edit permission check in views.py and the
custom-field DOCUMENTLINK validator in serialisers.py off of
has_perms_owner_aware-per-document loops, resolving
permitted_document_ids(user, perm="change_document") once instead.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GRp4kf1mdn9ruv81zWAmh2
* perf: resolve permitted_document_ids once for root-document version-listing loop
BulkDownloadView.post() previously called has_perms_owner_aware() per row
inside the loop that resolves each document's root and latest version.
Resolve permitted_document_ids(request.user) once before the loop and check
membership by root_doc.pk instead, consistent with the other consolidated
permission-filtering sites.
* test: use HTTPStatus enum instead of bare integers in security test assertions
* perf: resolve permitted_document_ids(perm=delete_document, include_deleted=True) once for trash loop
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GRp4kf1mdn9ruv81zWAmh2
* perf: drop unused select_related("owner") from email_documents
The permission check loop that used to call has_perms_owner_aware()
(which read .owner) was already replaced with permitted_document_ids()
resolved once into a set. No other code in email_documents touches
.owner, so the select_related is dead weight.
* test: repurpose inert grant into mixed-batch bulk-edit rejection case
The unrelated view_document grant in
test_bulk_edit_rejects_document_without_change_permission created a
document that was never referenced in the request payload. Turn it
into a genuinely useful case instead: a mixed batch containing one
document the requester is fully permitted to change alongside one
they are not, proving bulk_edit rejects the whole batch when any
document lacks change permission (not just checking the first/last
document in the list).
* test: add version-only-grant case discriminating root-vs-version permission check
The former "stranger" sub-case in
test_permission_checked_on_root_not_on_version had zero grants on either
root or version, so it passed under any implementation, correct or
buggy. Replace it with a user granted view_document on the version
itself (not the root): this only passes if bulk_download truly checks
root-only, catching a regression to "root OR version" that the old
case could never detect.
* perf: check permitted document IDs via DB-side exclude/exists instead of materializing the full set
email_documents, _has_document_permissions, TrashView.post, and
validate_documentlink_targets each resolved permitted_document_ids() into
a full Python set just to check membership for a small, bounded batch of
request document IDs. For a user with broad permitted access that pulls
their entire visible/editable document count into memory and across the
wire regardless of how many documents the request actually touches.
Pushing the membership check into the DB via exclude(...).exists() scales
with the request's batch size instead, without reintroducing the
per-row guardian join pathology from #13276 (confirmed via EXPLAIN
ANALYZE: the permission subplans are hashed once, not re-executed per
outer row).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* feat: add perm param to permitted_document_ids for change/delete checks
Widens permitted_document_ids(user, *, include_deleted=False) to
permitted_document_ids(user, *, perm="view_document", include_deleted=False)
so Stage 2 callers can check change_document/delete_document permissions
instead of the hardcoded view_document codename. Default is unchanged for
every existing call site.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GRp4kf1mdn9ruv81zWAmh2
* test: strengthen delete-permission include_deleted test to discriminate from other grants
test_delete_permission_with_include_deleted_for_trash_restore only checked
an owner and a fully-ungranted stranger, so it never proved perm=
actually discriminates delete_document from other permission grants. Add
a view_only user with view_document (but not delete_document) granted on
the same doc and assert they remain excluded, mirroring the pattern in
test_change_document_permission_is_distinct_from_view.
* fix: normalize qualified permission strings in permitted_document_ids
Guardian's UserObjectPermission/GroupObjectPermission always store a bare
codename, but has_perm()-style callers commonly pass the qualified
"app_label.codename" form. Passing that qualified form here previously
matched zero rows, silently under-permitting. content_type already
disambiguates the codename, so just strip any prefix instead.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* perf: migrate 5 single-call Document permission sites to permitted_document_ids
Swaps get_objects_for_user_owner_aware(user, "view_document", Document) for
Document.objects.filter(id__in=permitted_document_ids(user)) at 5 read-only,
single-call sites: AI chat "ask all documents", bulk-edit
_resolve_document_ids all:true branch, SelectionDataView permission check,
global search docs bucket, and the statistics endpoint's Document branch.
Confirmed all 3 callers of _resolve_document_ids always use the default
"view_document" codename before swapping. Added a regression test pinning
the AI-chat owner/permission boundary through the real API client, and
updated 2 existing mocked tests in test_views.py that asserted on
get_objects_for_user_owner_aware for the chat endpoint.
* perf: migrate 3 serialisers.py Document permission sites to permitted_document_ids
Migrates _get_viewable_duplicates(), PaperlessTaskSerializer.get_duplicate_documents(),
and the ShareLinkBundle document field queryset to use permitted_document_ids()
instead of get_objects_for_user_owner_aware()/get_objects_for_user(), consolidating
onto the shared permission-filtering helper. The is_staff gate in
get_duplicate_documents() is preserved as-is.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GRp4kf1mdn9ruv81zWAmh2
* refactor: remove dead permission_codename param from _resolve_document_ids
The keyword param was unused in the method body since an earlier commit
switched it to call permitted_document_ids(user) internally. None of the
3 call sites passed it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* docs: drop internal task-number reference from AI chat migration test docstring
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* feat: add include_deleted param to permitted_document_ids
Widens permitted_document_ids to accept an include_deleted keyword-only
flag (default False, preserving current behavior) so later call sites
that need visibility into soft-deleted documents (e.g. trash restore)
can reuse this permission check instead of duplicating it.
* refactor: remove redundant deleted_at filter in permitted_document_ids
Document.objects already applies filter(deleted_at__isnull=True) internally
via SoftDeleteManager.get_queryset(), so the conditional filter was redundant.
Simplify to just use manager.all() in both branches — manager selection alone
ensures correct behavior (Document.objects excludes deleted, Document.global_objects
includes all).
Co-Authored-By: Claude Haiku <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session
---------
Co-authored-by: Claude Haiku <noreply@anthropic.com>
* test: add permission-filtering security regression suite for Document
* test: verify document is actually soft-deleted (not hard-deleted) in soft-delete visibility test
Addresses Copilot review feedback on PR #13505: refresh_from_db() and
assert deleted_at is set before checking visibility, so this test
actually validates the deleted_at__isnull=True filtering behavior
rather than just checking the document disappeared from the queryset
for any reason.
get_context_for_document() always materialized every document id a user
can see into a Python list, even for a superuser (or no user at all),
passing it through as a SQL IN filter. For a superuser, that's the whole
library:
- Past ~32,763 documents, this crashes outright:
sqlite3.OperationalError: too many SQL variables (SQLite's
SQLITE_MAX_VARIABLE_NUMBER is 32766 by default, and the query already
binds embedding + k + a NE self-exclusion clause alongside the ids).
- Below that cliff, vec0's IN-list evaluation is a nested loop (strncmp
per row per allowed id), so it's quadratic in library size for no
reason -- the filter was never going to exclude anything.
get_objects_for_user_owner_aware() already returns every Document for a
superuser (guardian's own with_superuser shortcut), so skipping straight
to document_ids=None changes nothing about which documents are
considered, only how we get there.
Also:
- Drop the pointless sorted() in _document_id_filters(): a SQL IN clause
doesn't care about order, so it was pure overhead on every call.
- Add a hard guard in _build_where() so a future regression (or a
legitimately huge permission-restricted user) fails closed -- no rows,
a logged warning -- instead of a cryptic OperationalError. Since this
filter scopes document access, failing closed rather than skipping the
filter is the only safe way to handle an oversized list.
First item from VECTOR_STORE_PERF_BACKLOG.md (an audit done alongside
perf/13314-vecstore-point-delete, deferred to its own branch since that
one was already large).
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Fix: make migration-check result tri-state to avoid deferred-vs-current ambiguity
* Test: cover update_llm_index()'s migration-check-deferred branch
* Refactor: rename COMPACT_BATCH_SIZE to BATCH_SIZE, no longer compact()-specific
* Feature: schema v2 -- document_chunks/document_meta side tables, document_id INTEGER, point-delete
Rewrites the sqlite-vec vector store's on-disk schema: document_id becomes
an INTEGER vec0 metadata column (was TEXT), modified moves out of vec0 into
a new document_meta side table, and a new document_chunks side table gives
O(1) per-document chunk lookup for delete/upsert instead of a full vec0
scan. compact() now streams document_chunks and document_meta across the
file-swap rebuild too (previously document_meta would have gone silently
empty after the first compaction). drop_table() clears both side tables.
Adds the single frozen m0001_v1_to_v2 migration, converting a real,
historically-shaped v1 store (the shape shipped since v3.0.0) into the v2
shape in one streaming pass, with its own hardcoded DDL rather than
delegating to any "current schema" helper.
SCHEMA_VERSION bumps 1 -> 2.
* Fix: strengthen two vacuous Task 4 regression tests
test_migration_never_delegates_to_current_schema_helpers never actually ran
the migration (missing check_and_run_migrations() call) and its source-text
assertion was tautological (the "or DROP TABLE in source" clause was always
true). Now runs the real migration and asserts spy call counts instead:
DocumentChunksTable.create/DocumentMetaTable.create are each called exactly
3 times (construction, rebuild temp file, post-swap reopen -- all via
_open_connection, never from inside apply()), and _create_vec_table is
never called from the migration path.
test_drop_table_clears_modified_times asserted via get_modified_times(),
which short-circuits on table_exists() -- checking only the vec0 table that
drop_table() drops first -- so the assertion held even if
DocumentMetaTable.delete_all() were never called. Now asserts directly
against document_meta and document_chunks row counts.
* Perf: dedupe table_exists() lookups, atomic insert counter, fewer connections in update_llm_index()
* Fix: guard compact() against unmigrated stores, apply final-review cleanups
compact() had no migration guard: on a v1-schema store, document_chunks
reads 0 (freshly created empty) while total_inserts reflects the real
cumulative count, so the bloat check nearly always rebuilt -- silently
losing document_meta (copy_all reads from the empty v1 table) while
schema_version copied across unchanged, leaving the store permanently
unmigratable. compact() now calls has_pending_migration() and no-ops with
a warning instead.
Also folds in five minor final-review findings: drop _rebuild_into's
unused int return, hoist test-local imports to module level in
test_vector_store.py, note in TestMigrations' docstring that its fake
structural migrations only exercise dispatch (not full schema
correctness), restore the comment explaining why _row() requires
document_id, and tighten increment_total_inserts' docstring to not imply
general concurrency safety beyond its single atomic statement.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Fix: unify born-digital PDF detection between archive decision and OCR
should_produce_archive() and RasterisedDocumentParser.parse() each
reimplemented the "does this PDF have real text" check independently,
using different normalization of pdftotext output. Raw pdftotext output
can be non-empty (whitespace/form-feed layout padding) even when there
is no real content, so the two checks could disagree: consumer.py
treated a tagged-but-textless PDF as born-digital and skipped the
archive, while the parser's own (stricter, normalized) check found no
text and ran OCR anyway, leaving the document with no archive despite
real OCR text (GH #13387).
Both call sites now share one predicate, pdf_born_digital_text() in
paperless/parsers/utils.py, so they can no longer drift apart.
* Fix: restore extract_text seam for born-digital detection in parse()
parse() had switched to calling pdf_born_digital_text() directly for
its initial text/born-digital check, bypassing the parser's own
extract_text instance method. That broke test mockability (tests patch
tesseract_parser.extract_text to control the born-digital decision)
and caused CI failures with mismatched OCR call counts and text.
Split pdf_born_digital_text() into is_born_digital_text(text, path,
log) - a pure decision function - and a thin pdf_born_digital_text()
wrapper for callers without text in hand (consumer.should_produce_archive).
parse() now extracts via self.extract_text(None, document_path) and
passes the result to is_born_digital_text(), restoring the seam with
no change to production behavior.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Cleanup: simplify born-digital detection, close#13387 test gap
Simplification pass over the born-digital detection consolidation:
- is_born_digital_text(): drop the has_text temp for an early return.
- consumer.py: standardize the archive-decision log lines on plain
hyphens (was a mix of em-dash and hyphen) and hoist the duplicated
text_length computation.
- Parametrize TestPdfBornDigitalText instead of four near-identical
tests.
Code review follow-up: the existing tests only ever exercised
pdf_born_digital_text() through mocks, so the actual #13387 scenario
(a tagged PDF whose only "text" is layout padding) was never checked
against real pdftotext/pikepdf output - a regression in the
normalize-before-decide logic itself would have gone undetected.
Moved tagged_no_text_pdf_file from parsers/conftest.py up to the
shared paperless/tests/conftest.py (it was previously only visible to
tests under parsers/) and added a non-mocked regression test against
the real sample file.
Also fixed two stale comments in test_consumer.py referencing a
_extract_text_for_archive_check helper that no longer exists.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Fix: exclude next-period start from relative date-range filters
Tantivy's [lo TO hi] range is inclusive on both ends, but computed upper
bounds (keyword ranges, YYYY/YYYYMM/YYYYMMDD tokens) represent the start of
the next period. Use half-open [lo TO hi} for those so e.g. "previous month"
no longer matches the 1st of the current month.
* Adds a regression test down to the second check for the hi range
build_llm_index_text queried Note and CustomFieldInstance (plus its
field FK) per document, uncovered by the earlier correspondent/type/
storage_path/tags prefetch fix. Add notes and custom_fields__field to
the rebuild and scoped document querysets, and read from doc.notes.all()
instead of a fresh Note.objects.filter() so the prefetch is actually used.
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
stumpylog
GitHub Actions
shamoonandGitHub
Gaëtan GOUZIandGitHub
