Test modules in paperless, paperless_mail and documents imported filesystem assertions, the migration test base, the retry helper and the streaming-response reader out of documents/tests/utils.py, which kept each app's tests coupled to another app's test package.
They now live in paperless_testing, and the progress manager fake is renamed FakeProgressManager and now subclasses the real ProgressManager, overriding only the transport, so the payload it records is built by the production code. The twenty places that patched documents.tasks.ProgressManager by hand now use a fake_progress_manager fixture.
The unittest side of the suite built its temp directory tree with
tempfile.mkdtemp and a manually enabled override_settings, cleaned up only if
tearDown ran. That is now gone. DirectoriesMixin lives alongside the layout it
bridges and does nothing but hand the paperless_dirs fixture to TestCase
subclasses as self.dirs, so both halves of the suite get the same twelve
settings, the same directory shapes and cleanup owned by tmp_path.
The mixin moves to paperless_testing.dirs rather than staying in the documents
test utilities, because modules in paperless and paperless_mail import it
across the app boundary. The thirty-eight consuming modules change only their
import line; self.dirs.scratch_dir and its siblings keep working.
* Feature: Allow configuring the compression type and compression levels during export
Building on the zip export improvements, this now allows users to further configure the
zip to fit their needs. A simple stored zip for speed, or a high compression zstd for
the smallest archive. Full validation of the method and levels at the command line
Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
* Replaces loaddata with streaming bulk_create
Replaces call_command('loaddata') with a streaming implementation that
reads manifest records one at a time via ijson, accumulates per-model
batches up to --batch-size, and flushes via bulk_create. This reduces
peak memory and no longer scales directly with the size of the import.
* fix(importer): avoid guardian lru_cache poisoning; include M2M through tables in check_constraints
clear_cache() inside the import transaction emptied Django's ContentType
manager cache while fixture PKs were live, causing downstream ContentType
lookups to repopulate guardian's separate @lru_cache(None) with
fixture-PK objects. After the TestCase transaction rolled back to
original PKs, guardian's lru_cache held stale fixture ContentType
objects, causing MixedContentTypeError in unrelated subsequent tests.
Remove clear_cache() since it was defending against a theoretical
stale-cache scenario that doesn't occur in a proper same-install restore.
Fix check_constraints() to explicitly include auto-created M2M through
tables (populated by .set() after bulk_create) alongside the model tables,
addressing the gap where join-table FK violations would have gone
undetected.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Excludes the consumer and AnonymousUser from any models which might have a FK relation to it. This prevents orphan things like UI setting, which have a relation to no existing user
* Splits into more sub functions for Sonar
* Improvements to the typing of the new functions
* Coverage for some error cases, and removes handling for pk only models. No need to support these
* Final coverage gaps
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* Perf: stream manifest parsing with ijson in document_importer
Replace bulk json.load of the full manifest (which materializes the
entire JSON array into memory) with incremental ijson streaming.
Eliminates self.manifest entirely — records are never all in memory
at once.
- Add ijson>=3.2 dependency
- New module-level iter_manifest_records() generator
- load_manifest_files() collects paths only; no parsing at load time
- check_manifest_validity() streams without accumulating records
- decrypt_secret_fields() streams each manifest to a .decrypted.json
temp file record-by-record; temp files cleaned up after file copy
- _import_files_from_manifest() collects only document records (small
fraction of manifest) for the tqdm progress bar
Measured on 200 docs + 200 CustomFieldInstances:
- Streaming validation: peak memory 3081 KiB -> 333 KiB (89% reduction)
- Stream-decrypt to file: peak memory 3081 KiB -> 549 KiB (82% reduction)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Perf: slim dict in _import_files_from_manifest, discard fields
When collecting document records for the file-copy step, extract only
the 4 keys the loop actually uses (pk + 3 exported filename keys) and
discard the full fields dict (content, checksum, tags, etc.).
Peak memory for the document-record list: 939 KiB -> 375 KiB (60% reduction).
Wall time unchanged.
* tests: general cleanup and fixes for runnning under docker
This now allows tests to be run under a locally built or production
docker image with something like:
`docker run --rm -v $PWD:/usr/src/paperless --entrypoint=bash paperlessngx/paperless-ngx:latest -c "uv run pytest"`
Specific fixes:
- fix unreachable code around `assertRaises` blocks
- fix `assertInt` typos
- fix `str(e)` vs `str(e.exception)` issues
- skip permission-based checks when root (in a docker container)
- catch `OSError` problems when instantiating `INotify` and
skip inotify-based tests when it's unavailable.
* Reverts most files to dev while keeping the exception assert fixes
---------
Co-authored-by: Trenton H <797416+stumpylog@users.noreply.github.com>
Trenton H
Matt Van Horn
Sebastian Steinbeißer
Ed BardsleyandTrenton H
Kilian