amorfo77/paperless-ngx
1
0
Fork 0
mirror of https://github.com/paperless-ngx/paperless-ngx.git synced 2026-03-20 07:55:57 +00:00
Code Issues Projects Releases Wiki Activity
11,122 Commits 27 Branches 214 Tags
2eabdcb0ef26188f34111fbda336c2a012d67b47
Commit Graph

2180 Commits

Author SHA1 Message Date
Trenton H
2eabdcb0ef Removes the profiling leftovers 2026-03-06 15:57:54 -08:00
Trenton H
ac8bab7ed0 Refactor: O(1) lookup table for CRYPT_FIELDS in per-record encryption 
Add CRYPT_FIELDS_BY_MODEL to CryptMixin, derived from CRYPT_FIELDS at
class definition time. _encrypt_record_inline() now does a single dict
lookup instead of a linear scan per record, eliminating the loop and
break pattern.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-06 14:43:56 -08:00
Trenton H
3e961eff0e Perf: streaming manifest writer for document exporter (Phase 3) 
Replaces the in-memory manifest dict accumulation with a
StreamingManifestWriter that writes records to manifest.json
incrementally, keeping only one batch resident in memory at a time.

Key changes:
- Add StreamingManifestWriter: writes to .tmp atomically, BLAKE2b
  compare for --compare-json, discard() on exception
- Add _encrypt_record_inline(): per-record encryption replacing the
  bulk encrypt_secret_fields() call; crypto setup moved before streaming
- Add _write_split_manifest(): extracted per-document manifest writing
- Refactor dump(): non-doc records streamed during transaction, documents
  accumulated then written after filenames are assigned
- Upgrade check_and_write_json() from MD5 to BLAKE2b
- Remove encrypt_secret_fields() and unused itertools.chain import
- Add profiling marker to pyproject.toml

Measured improvement (200 docs + 200 CustomFieldInstances, same
dump() code path, only writer differs):
- Peak memory: ~50% reduction
- Memory delta: ~70% reduction
- Wall time and query count: unchanged

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-03-06 14:41:48 -08:00
shamoon
731448a8f9 Fixhancement: support version-specific edits (#12233) 2026-03-06 11:59:26 -08:00
shamoon
24a2cfd957 Change: use explicit doc creation instead of clone for versions (#12226) 2026-03-04 15:57:44 -08:00
shamoon
df03207eef Fix: correct doc version filename handling (#12223) 2026-03-04 23:28:07 +00:00
Trenton H
1e21bcd26e Breaking: Drop support for Python 3.10 (#12234) 2026-03-04 15:03:33 -08:00
Trenton H
a9cb89c633 Enhancement: Improve exporter memory efficiency (#12236) 
Phase 1 -- Eliminate JSON round-trip in document exporter

Replace json.loads(serializers.serialize("json", qs)) with
serializers.serialize("python", qs) to skip the intermediate
JSON string allocation and parse step. Use DjangoJSONEncoder
in check_and_write_json() to handle native Python types
(datetime, Decimal, UUID) the Python serializer returns.

Phase 2 -- Batched QuerySet serialization in document exporter

Add serialize_queryset_batched() helper that uses QuerySet.iterator()
and itertools.islice to stream records in configurable chunks, bounding
peak memory during serialization to batch_size * avg_record_size rather
than loading the entire QuerySet at once.
 2026-03-04 14:54:20 -08:00
shamoon
85a18e5911 Enhancement: saved view sharing (#12142) 2026-03-04 14:15:43 -08:00
shamoon
d51a118aac Merge branch 'main' into dev 2026-03-04 13:31:20 -08:00
shamoon
615f27e6fb Fix: support string coercion in filepath jinja templates (#12244) 2026-03-04 08:32:34 -08:00
shamoon
5b809122b5 Fix: apply ordering after annotating tag document count (#12238) 2026-03-04 00:33:13 -08:00
shamoon
299dac21ee Enhancement: “live” document updates (#12141) 2026-03-04 00:27:07 +00:00
Trenton H
5498503d60 Chore: Improve user migration path (#12232) 
Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
 2026-03-03 15:51:48 -08:00
shamoon
8b8307571a Fix: enforce path limit for db filename fields (#12235) 2026-03-03 13:19:56 -08:00
Trenton H
43406f44f2 Feature: Improve the retagger output using rich (#12194) 2026-03-03 07:14:59 -08:00
Trenton H
e58a35d40c Feature: Transition sanity check to rich and improve output (#12182) 2026-03-02 10:53:39 -08:00
Trenton H
20a9cd40e8 Feature: Switch all indexing to use rich (#12193) 2026-03-02 10:41:09 -08:00
shamoon
96ac7b2336 Tweak: Ignore version docs for workflows (#12217) 2026-03-02 08:21:14 -08:00
shamoon
f65807b906 Merge branch 'main' into dev 
# Conflicts:
#	docs/setup.md
#	src-ui/src/app/components/manage/document-attributes/management-list/management-list.component.ts
#	src/documents/tests/test_api_documents.py
 2026-02-28 02:31:20 -08:00
shamoon
c7f83212a3 Enforce on selection_data too 2026-02-28 01:27:40 -08:00
shamoon
b010f65ae7 Fix GHSA-386h-chg4-cfw9 2026-02-28 01:16:53 -08:00
Jan Kleine
0bc032a67d Development: improve test portability (#12187) 
* Fix: improve test portability

* Make settings always consistent

* Make a few more tests deterministic wrt settings

* Dont pollute settings for this one

* Fix timezone issue with mail parser

* Update test_parser.py

* Uh, I guess OCR gives variants for this

---------

Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
 2026-02-27 23:24:11 +00:00
Trenton H
c30ee1ec03 Feature: Switch progress bar library to rich (#12169) 2026-02-26 12:20:21 -08:00
Jan Kleine
c86ebc0260 Enhancment: Formatted filename for single document downloads (#12095) 
---------

Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
 2026-02-26 18:06:47 +00:00
dependabot[bot]
5e1202a416 Chore(deps): Bump the utilities-minor group across 1 directory with 7 updates (#12174) 
* Chore(deps): Bump the utilities-minor group across 1 directory with 7 updates

Bumps the utilities-minor group with 7 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [django-guardian](https://github.com/django-guardian/django-guardian) | `3.2.0` | `3.3.0` |
| [filelock](https://github.com/tox-dev/py-filelock) | `3.20.3` | `3.24.3` |
| [openai](https://github.com/openai/openai-python) | `2.17.0` | `2.24.0` |
| [regex](https://github.com/mrabarnett/mrab-regex) | `2026.1.15` | `2026.2.19` |
| [pytest-django](https://github.com/pytest-dev/pytest-django) | `4.11.1` | `4.12.0` |
| [pytest-env](https://github.com/pytest-dev/pytest-env) | `1.2.0` | `1.5.0` |
| [pyrefly](https://github.com/facebook/pyrefly) | `0.51.0` | `0.54.0` |



Updates `django-guardian` from 3.2.0 to 3.3.0
- [Release notes](https://github.com/django-guardian/django-guardian/releases)
- [Commits](https://github.com/django-guardian/django-guardian/compare/3.2.0...3.3.0)

Updates `filelock` from 3.20.3 to 3.24.3
- [Release notes](https://github.com/tox-dev/py-filelock/releases)
- [Changelog](https://github.com/tox-dev/filelock/blob/main/docs/changelog.rst)
- [Commits](https://github.com/tox-dev/py-filelock/compare/3.20.3...3.24.3)

Updates `openai` from 2.17.0 to 2.24.0
- [Release notes](https://github.com/openai/openai-python/releases)
- [Changelog](https://github.com/openai/openai-python/blob/main/CHANGELOG.md)
- [Commits](https://github.com/openai/openai-python/compare/v2.17.0...v2.24.0)

Updates `regex` from 2026.1.15 to 2026.2.19
- [Changelog](https://github.com/mrabarnett/mrab-regex/blob/hg/changelog.txt)
- [Commits](https://github.com/mrabarnett/mrab-regex/compare/2026.1.15...2026.2.19)

Updates `pytest-django` from 4.11.1 to 4.12.0
- [Release notes](https://github.com/pytest-dev/pytest-django/releases)
- [Changelog](https://github.com/pytest-dev/pytest-django/blob/main/docs/changelog.rst)
- [Commits](https://github.com/pytest-dev/pytest-django/compare/v4.11.1...v4.12.0)

Updates `pytest-env` from 1.2.0 to 1.5.0
- [Release notes](https://github.com/pytest-dev/pytest-env/releases)
- [Commits](https://github.com/pytest-dev/pytest-env/compare/1.2.0...1.5.0)

Updates `pyrefly` from 0.51.0 to 0.54.0
- [Release notes](https://github.com/facebook/pyrefly/releases)
- [Commits](https://github.com/facebook/pyrefly/compare/0.51.0...0.54.0)

---
updated-dependencies:
- dependency-name: django-guardian
  dependency-version: 3.3.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: filelock
  dependency-version: 3.24.3
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: openai
  dependency-version: 2.24.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: regex
  dependency-version: 2026.2.19
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: pytest-django
  dependency-version: 4.12.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: pytest-env
  dependency-version: 1.5.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: pyrefly
  dependency-version: 0.54.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Fixes the additional unlink now done by filelock

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Trenton H <797416+stumpylog@users.noreply.github.com>
 2026-02-26 10:01:23 -08:00
shamoon
ceee769e26 Feature: document file versions (#12061) 2026-02-26 16:46:54 +00:00
Trenton H
53ac338946 Breaking: Removes API v1 and the related serializer (#12166) 2026-02-26 04:06:43 +00:00
Jan Kleine
c4ea332c61 Feature: move to trash action for workflows (#11176) 
Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
 2026-02-23 16:42:50 -08:00
shamoon
426c0a8974 Chore: typing fixes 2026-02-16 09:54:06 -08:00
shamoon
4884b67714 Fix more typing failures 2026-02-16 09:37:33 -08:00
shamoon
02896a15fd Fix: only pass user to SerializerWithPerms serializers 2026-02-16 09:31:33 -08:00
shamoon
d8e07b8d84 Fix typing issue 2026-02-16 09:17:20 -08:00
shamoon
be4e29a19c Merge branch 'main' into dev 2026-02-16 09:01:19 -08:00
shamoon
afaf39e43a Fix/GHSA-x395-6h48-wr8v 2026-02-16 00:02:15 -08:00
Trenton H
8db1c4e08b Breaking: Remove pybzar as a barcode reader (#12065) 2026-02-13 08:14:00 -08:00
Trenton H
e8e027abc0 Chore: Optimizes the integer fields for choice types mostly, while leaving plenty of room to grow (#12057) 2026-02-10 15:11:44 -08:00
Trenton H
6a87c3f4dd Fixes handling the case where there is no status reported from celery (due to external termination of the worker) (#12040) 2026-02-08 17:26:35 -08:00
shamoon
30b1d3c6d7 Fix missing content_length migration 2026-02-07 11:07:16 -08:00
shamoon
6192915be7 Fixhancement: improve ASN handling with PDF operations (#11689) 2026-02-06 21:14:02 +00:00
shamoon
5b45b89d35 Performance fix: use subqueries to improve object retrieval in large installs (#11950) 2026-02-05 08:46:32 -08:00
dependabot[bot]
33cbe2ad54 Chore(deps): Bump the utilities-minor group across 1 directory with 6 updates (#11993) 
* Chore(deps): Bump the utilities-minor group across 1 directory with 6 updates

Bumps the utilities-minor group with 6 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [babel](https://github.com/python-babel/babel) | `2.17.0` | `2.18.0` |
| [dateparser](https://github.com/scrapinghub/dateparser) | `1.2.2` | `1.3.0` |
| [django-cachalot](https://github.com/noripyt/django-cachalot) | `2.8.0` | `2.9.0` |
| [openai](https://github.com/openai/openai-python) | `2.15.0` | `2.16.0` |
| [torch](https://github.com/pytorch/pytorch) | `2.9.1` | `2.10.0` |
| [ruff](https://github.com/astral-sh/ruff) | `0.14.14` | `0.15.0` |



Updates `babel` from 2.17.0 to 2.18.0
- [Release notes](https://github.com/python-babel/babel/releases)
- [Changelog](https://github.com/python-babel/babel/blob/master/CHANGES.rst)
- [Commits](https://github.com/python-babel/babel/compare/v2.17.0...v2.18.0)

Updates `dateparser` from 1.2.2 to 1.3.0
- [Release notes](https://github.com/scrapinghub/dateparser/releases)
- [Changelog](https://github.com/scrapinghub/dateparser/blob/master/HISTORY.rst)
- [Commits](https://github.com/scrapinghub/dateparser/compare/v1.2.2...v1.3.0)

Updates `django-cachalot` from 2.8.0 to 2.9.0
- [Release notes](https://github.com/noripyt/django-cachalot/releases)
- [Changelog](https://github.com/noripyt/django-cachalot/blob/master/CHANGELOG.rst)
- [Commits](https://github.com/noripyt/django-cachalot/compare/v2.8.0...v2.9.0)

Updates `openai` from 2.15.0 to 2.16.0
- [Release notes](https://github.com/openai/openai-python/releases)
- [Changelog](https://github.com/openai/openai-python/blob/main/CHANGELOG.md)
- [Commits](https://github.com/openai/openai-python/compare/v2.15.0...v2.16.0)

Updates `torch` from 2.9.1 to 2.10.0
- [Release notes](https://github.com/pytorch/pytorch/releases)
- [Changelog](https://github.com/pytorch/pytorch/blob/main/RELEASE.md)
- [Commits](https://github.com/pytorch/pytorch/compare/v2.9.1...v2.10.0)

Updates `ruff` from 0.14.14 to 0.15.0
- [Release notes](https://github.com/astral-sh/ruff/releases)
- [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md)
- [Commits](https://github.com/astral-sh/ruff/compare/0.14.14...0.15.0)

---
updated-dependencies:
- dependency-name: babel
  dependency-version: 2.18.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: dateparser
  dependency-version: 1.3.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: django-cachalot
  dependency-version: 2.9.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: openai
  dependency-version: 2.16.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: torch
  dependency-version: 2.10.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: ruff
  dependency-version: 0.15.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Updates to ruff 0.15.0

* Ignores all notes in the baseline.  They seem to be problematic??

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Trenton H <797416+stumpylog@users.noreply.github.com>
 2026-02-04 11:50:31 -08:00
Trenton H
2ec8ec96c8 Feature: Enable users to customize date parsing via plugins (#11931) 2026-02-03 20:09:13 +00:00
shamoon
e45fca475a Feature: password removal workflow action (#11665) 2026-02-03 17:10:07 +00:00
shamoon
00ef0837d2 Fix: re-run ASN check after barcode detection (#11681) 2026-02-02 23:23:37 +00:00
Sebastian Steinbeißer
3b5ffbf9fa Chore(mypy): Annotate None returns for typing improvements (#11213) 2026-02-02 08:44:12 -08:00
shamoon
c3b036e0d3 Merge branch 'main' into dev 2026-01-31 09:10:33 -08:00
shamoon
5cc3c087d9 Security: enforce ownership for permission updates 2026-01-30 13:55:55 -08:00
shamoon
c8c4c7c749 Security: enforce permissions for post_document 2026-01-30 12:14:18 -08:00
shamoon
e4b861d76f Fix: prevent note deletion outside doc 2026-01-29 13:35:01 -08:00