amorfo77/paperless-ngx
1
0
Fork 0
mirror of https://github.com/paperless-ngx/paperless-ngx.git synced 2026-08-04 10:02:17 +00:00
Code Issues Projects Releases Wiki Activity

Feature: Further reduce document importer memory usage (#12707)

Browse Source 
* Replaces loaddata with streaming bulk_create

Replaces call_command('loaddata') with a streaming implementation that
reads manifest records one at a time via ijson, accumulates per-model
batches up to --batch-size, and flushes via bulk_create.  This reduces
peak memory and no longer scales directly with the size of the import.

* fix(importer): avoid guardian lru_cache poisoning; include M2M through tables in check_constraints

clear_cache() inside the import transaction emptied Django's ContentType
manager cache while fixture PKs were live, causing downstream ContentType
lookups to repopulate guardian's separate @lru_cache(None) with
fixture-PK objects. After the TestCase transaction rolled back to
original PKs, guardian's lru_cache held stale fixture ContentType
objects, causing MixedContentTypeError in unrelated subsequent tests.

Remove clear_cache() since it was defending against a theoretical
stale-cache scenario that doesn't occur in a proper same-install restore.

Fix check_constraints() to explicitly include auto-created M2M through
tables (populated by .set() after bulk_create) alongside the model tables,
addressing the gap where join-table FK violations would have gone
undetected.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Excludes the consumer and AnonymousUser from any models which might have a FK relation to it.  This prevents orphan things like UI setting, which have a relation to no existing user

* Splits into more sub functions for Sonar

* Improvements to the typing of the new functions

* Coverage for some error cases, and removes handling for pk only models.  No need to support these

* Final coverage gaps

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Trenton H
2026-05-04 16:36:05 -07:00
committed by GitHub
co-authored by Claude Sonnet 4.6
parent 209c5d2deb
commit e822e72964
4 changed files with 558 additions and 62 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/administration.md
+7 -6
View File
@@ -357,12 +357,13 @@ and the script does the rest of the work:
document_importer source
```
| Option | Required | Default | Description |
| ------------------- | -------- | ------- | ------------------------------------------------------------------------- |
| source | Yes | N/A | The directory containing an export |
| `--no-progress-bar` | No | False | If provided, the progress bar will be hidden |
| `--data-only` | No | False | If provided, only import data, do not import document files or thumbnails |
| `--passphrase` | No | N/A | If your export was encrypted with a passphrase, must be provided |
| Option | Required | Default | Description |
| ------------------- | -------- | ------- | ------------------------------------------------------------------------------------------------------------ |
| source | Yes | N/A | The directory containing an export |
| `--no-progress-bar` | No | False | If provided, the progress bar will be hidden |
| `--data-only` | No | False | If provided, only import data, do not import document files or thumbnails |
| `--passphrase` | No | N/A | If your export was encrypted with a passphrase, must be provided |
| `--batch-size` | No | 500 | Number of database records inserted per batch. Lower values reduce peak memory usage on very large installs. |
When you use the provided docker compose script, put the export inside
the `export` folder in your paperless source directory. Specify