amorfo77/paperless-ngx
1
0
Fork 0
mirror of https://github.com/paperless-ngx/paperless-ngx.git synced 2026-09-21 17:08:33 +00:00
Code Issues Projects Releases Wiki Activity
12,168 Commits 33 Branches 233 Tags
7575d6078227ebdb4cf443f263d53ebc7575aa37
Commit Graph
203 Commits
Author SHA1 Message Date
Trenton H 7575d60782 Bump version to 3.2.1 2026-09-20 14:29:37 -07:00
Trenton H 22a8a4f4ad Fix: bump ocrmypdf to 17.12 to pick up the ligature text-layer fix (#14190) 2026-09-19 15:24:56 -07:00
shamoon 961f018d3e Bump version to 3.2.0 2026-09-18 19:13:02 -07:00
dependabot[bot] 31d6e984f2 Chore(deps): Bump the utilities-minor group across 1 directory with 10 updates (#14149) 
Bumps the utilities-minor group with 10 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [drf-spectacular-sidecar](https://github.com/tfranzel/drf-spectacular-sidecar) | `2026.8.1` | `2026.9.1` |
| llama-index-embeddings-huggingface | `0.7.0` | `0.8.0` |
| llama-index-embeddings-ollama | `0.9.0` | `0.10.0` |
| llama-index-embeddings-openai-like | `0.3.1` | `0.4.0` |
| llama-index-llms-ollama | `0.10.1` | `0.11.0` |
| llama-index-llms-openai-like | `0.7.2` | `0.8.0` |
| [regex](https://github.com/mrabarnett/mrab-regex) | `2026.7.19` | `2026.9.10` |
| [torch](https://github.com/pytorch/pytorch) | `2.13.0` | `2.14.0` |
| [faker](https://github.com/joke2k/faker) | `40.37.0` | `40.38.0` |
| [types-regex](https://github.com/python/typeshed) | `2026.7.19.20260720` | `2026.9.3.20260903` |



Updates `drf-spectacular-sidecar` from 2026.8.1 to 2026.9.1
- [Commits](https://github.com/tfranzel/drf-spectacular-sidecar/compare/2026.8.1...2026.9.1)

Updates `llama-index-embeddings-huggingface` from 0.7.0 to 0.8.0

Updates `llama-index-embeddings-ollama` from 0.9.0 to 0.10.0

Updates `llama-index-embeddings-openai-like` from 0.3.1 to 0.4.0

Updates `llama-index-llms-ollama` from 0.10.1 to 0.11.0

Updates `llama-index-llms-openai-like` from 0.7.2 to 0.8.0

Updates `regex` from 2026.7.19 to 2026.9.10
- [Changelog](https://github.com/mrabarnett/mrab-regex/blob/hg/changelog.txt)
- [Commits](https://github.com/mrabarnett/mrab-regex/compare/2026.7.19...2026.9.10)

Updates `torch` from 2.13.0 to 2.14.0
- [Release notes](https://github.com/pytorch/pytorch/releases)
- [Changelog](https://github.com/pytorch/pytorch/blob/main/RELEASE.md)
- [Commits](https://github.com/pytorch/pytorch/compare/v2.13.0...v2.14.0)

Updates `faker` from 40.37.0 to 40.38.0
- [Release notes](https://github.com/joke2k/faker/releases)
- [Changelog](https://github.com/joke2k/faker/blob/master/CHANGELOG.md)
- [Commits](https://github.com/joke2k/faker/compare/v40.37.0...v40.38.0)

Updates `types-regex` from 2026.7.19.20260720 to 2026.9.3.20260903
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: drf-spectacular-sidecar
  dependency-version: 2026.9.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: faker
  dependency-version: 40.38.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: llama-index-embeddings-huggingface
  dependency-version: 0.8.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: llama-index-embeddings-ollama
  dependency-version: 0.10.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: llama-index-embeddings-openai-like
  dependency-version: 0.4.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: llama-index-llms-ollama
  dependency-version: 0.11.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: llama-index-llms-openai-like
  dependency-version: 0.8.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: regex
  dependency-version: 2026.9.3
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: torch
  dependency-version: 2.14.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: types-regex
  dependency-version: 2026.9.3.20260903
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-09-18 03:55:07 +00:00
Trenton H 20d309a413 Enhancement: Match fuzzy terms in place inside the parsed query (#14157) 
* Feature: match fuzzy terms in place inside the parsed query

Fuzzy matching was a separate clause OR'd in above the query: a flat bag
of the query's words, re-parsed through tantivy's own parser, blended
beside the exact clause. Nothing around a term reached it, so a fielded
term fuzzed across every default field, a filter did not constrain it, and
an exclusion had to be hoisted back over the whole blend to stop the
clause re-admitting what the query had just excluded.

Widen each leaf where it sits instead, through emit()'s rewrite_leaf hook,
so fielding, negation, AND, REQUIRE and positive filters constrain the
fuzzy match exactly as they constrain the exact one. Each of a leaf's
words becomes a Fuzzy leaf on the leaf's own field, boosted to 0.1, beside
the leaf and any CJK alternative it already had.

* Hello?
 2026-09-17 14:55:13 -07:00
Trenton H 762e8cf4d1 Feature: match CJK terms through their bigram fields in place (#14156) 
QUERY-mode searches blended a separate bigram clause in at the top of the
query, built from the parsed AST's free-text tokens. Because it sat beside
the exact clause rather than inside the query, nothing around a CJK term
constrained its bigram match: an exclusion that was one OR branch's own
condition could never reach it, so "(東京 AND NOT secret) OR bill" still
returned the secret document.

Widen each CJK leaf where it sits instead, through emit()'s rewrite_leaf
hook, so every AND, NOT, REQUIRE, boost and field restriction around the
leaf applies to its bigram match too. Negated leaves are widened on
purpose, so "NOT X" excludes exactly what "X" matches.
 2026-09-17 14:55:13 -07:00
Trenton H 530059c5c0 Performance: Preprocess classifier text with Tantivy instead of NLTK (#14127) 
* Preprocesses classifier content with Tantivy instead of NLTK

Tokenizing and stemming now happen in one Rust call instead of NLTK's
Python tokenizer and per word stemming, which also removes the Redis
backed stem cache from every preprocessing call. The output matches the
NLTK pipeline closely; tokens containing digits are now stemmed, and the
English stop words follow Snowball's list.

Stemming and stop word removal apply whenever the OCR language is one of
the supported classifier languages, so PAPERLESS_ENABLE_NLTK and
PAPERLESS_NLTK_DIR are removed.

* Copies packages instead of hardlinking them in backend CI, some NLTK thing

* Adds a normalization to NFC to better fit what Tantivy expects
 2026-09-16 07:35:26 -07:00
Trenton H e78521c912 Chore: Search field registry (#14071) 
* feat(search): add whoosh-compat, the shared field table and the field registry

* Sonar being useful actually

* Adds the given/when/then commenting

* Trims tests I don't think cover our logic or code or are redundant

* Coverage

* test(search): add pattern normalizer stem-alternates unit tests

* build: bump whoosh-compat to 0.2.0
 2026-09-15 07:26:41 -07:00
dependabot[bot]andTrenton Holmes f287a4cb8c Chore(deps): Bump the utilities-minor group across 1 directory with 11 updates (#13988) 
* Chore(deps): Bump the utilities-minor group across 1 directory with 11 updates

Bumps the utilities-minor group with 11 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [django-guardian](https://github.com/django-guardian/django-guardian) | `3.3.3` | `3.4.0` |
| [django-treenode](https://github.com/fabiocaccamo/django-treenode) | `0.24.0` | `0.25.0` |
| [drf-spectacular-sidecar](https://github.com/tfranzel/drf-spectacular-sidecar) | `2026.7.1` | `2026.8.1` |
| [imap-tools](https://github.com/ikvk/imap_tools) | `1.14.0` | `1.15.0` |
| [ocrmypdf](https://github.com/ocrmypdf/OCRmyPDF) | `17.10.0` | `17.11.0` |
| [prek](https://github.com/j178/prek) | `0.4.11` | `0.5.0` |
| [faker](https://github.com/joke2k/faker) | `40.36.0` | `40.37.0` |
| [pytest-django](https://github.com/pytest-dev/pytest-django) | `4.12.0` | `4.14.0` |
| [pytest-rerunfailures](https://github.com/pytest-dev/pytest-rerunfailures) | `16.4` | `16.6` |
| [time-machine](https://github.com/adamchainz/time-machine) | `3.2.0` | `3.5.0` |
| [types-pygments](https://github.com/python/typeshed) | `2.20.0.20260408` | `2.21.0.20260819` |



Updates `django-guardian` from 3.3.3 to 3.4.0
- [Release notes](https://github.com/django-guardian/django-guardian/releases)
- [Commits](https://github.com/django-guardian/django-guardian/compare/3.3.3...3.4.0)

Updates `django-treenode` from 0.24.0 to 0.25.0
- [Release notes](https://github.com/fabiocaccamo/django-treenode/releases)
- [Changelog](https://github.com/fabiocaccamo/django-treenode/blob/main/CHANGELOG.md)
- [Commits](https://github.com/fabiocaccamo/django-treenode/compare/0.24.0...0.25.0)

Updates `drf-spectacular-sidecar` from 2026.7.1 to 2026.8.1
- [Commits](https://github.com/tfranzel/drf-spectacular-sidecar/compare/2026.7.1...2026.8.1)

Updates `imap-tools` from 1.14.0 to 1.15.0
- [Release notes](https://github.com/ikvk/imap_tools/releases)
- [Changelog](https://github.com/ikvk/imap_tools/blob/master/docs/release_notes.rst)
- [Commits](https://github.com/ikvk/imap_tools/compare/v1.14.0...v1.15.0)

Updates `ocrmypdf` from 17.10.0 to 17.11.0
- [Release notes](https://github.com/ocrmypdf/OCRmyPDF/releases)
- [Commits](https://github.com/ocrmypdf/OCRmyPDF/compare/v17.10.0...v17.11.0)

Updates `prek` from 0.4.11 to 0.5.0
- [Release notes](https://github.com/j178/prek/releases)
- [Changelog](https://github.com/j178/prek/blob/master/CHANGELOG.md)
- [Commits](https://github.com/j178/prek/compare/v0.4.11...v0.5.0)

Updates `faker` from 40.36.0 to 40.37.0
- [Release notes](https://github.com/joke2k/faker/releases)
- [Changelog](https://github.com/joke2k/faker/blob/master/CHANGELOG.md)
- [Commits](https://github.com/joke2k/faker/compare/v40.36.0...v40.37.0)

Updates `pytest-django` from 4.12.0 to 4.14.0
- [Release notes](https://github.com/pytest-dev/pytest-django/releases)
- [Changelog](https://github.com/pytest-dev/pytest-django/blob/main/docs/changelog.rst)
- [Commits](https://github.com/pytest-dev/pytest-django/compare/v4.12.0...v4.14.0)

Updates `pytest-rerunfailures` from 16.4 to 16.6
- [Changelog](https://github.com/pytest-dev/pytest-rerunfailures/blob/master/CHANGES.rst)
- [Commits](https://github.com/pytest-dev/pytest-rerunfailures/compare/16.4...16.6)

Updates `time-machine` from 3.2.0 to 3.5.0
- [Changelog](https://github.com/adamchainz/time-machine/blob/main/docs/changelog.rst)
- [Commits](https://github.com/adamchainz/time-machine/compare/3.2.0...3.5.0)

Updates `types-pygments` from 2.20.0.20260408 to 2.21.0.20260819
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: django-guardian
  dependency-version: 3.4.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: django-treenode
  dependency-version: 0.25.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: drf-spectacular-sidecar
  dependency-version: 2026.8.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: faker
  dependency-version: 40.37.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: imap-tools
  dependency-version: 1.15.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: ocrmypdf
  dependency-version: 17.11.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: prek
  dependency-version: 0.5.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: pytest-django
  dependency-version: 4.14.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: pytest-rerunfailures
  dependency-version: '16.6'
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: time-machine
  dependency-version: 3.5.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: types-pygments
  dependency-version: 2.21.0.20260819
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Handles pytest-django

* Formatting

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Trenton Holmes <797416+stumpylog@users.noreply.github.com>
 2026-09-07 20:13:15 +00:00
dependabot[bot] e76ba49d86 Chore(deps): Update granian[uvloop] requirement (#13984) 
Updates the requirements on [granian[uvloop]](https://github.com/emmett-framework/granian) to permit the latest version.
- [Release notes](https://github.com/emmett-framework/granian/releases)
- [Commits](https://github.com/emmett-framework/granian/compare/v2.7.0...v2.8.2)

---
updated-dependencies:
- dependency-name: granian[uvloop]
  dependency-version: 2.8.2
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-09-04 21:57:01 +00:00
Trenton H 0e5d5022a1 Chore: Updates our direct Redis pin (#13986) 2026-09-04 14:28:27 -07:00
dependabot[bot]andstumpylog a1f407cc9e Chore(deps): Bump the pre-commit-dependencies group across 1 directory with 3 updates (#13922) 
* Chore(deps): Bump the pre-commit-dependencies group across 1 directory with 3 updates

Bumps the pre-commit-dependencies group with 3 updates in the / directory: [https://github.com/astral-sh/ruff-pre-commit](https://github.com/astral-sh/ruff-pre-commit), [https://github.com/tox-dev/pyproject-fmt](https://github.com/tox-dev/pyproject-fmt) and [https://github.com/AleksaC/hadolint-py](https://github.com/AleksaC/hadolint-py).


Updates `https://github.com/astral-sh/ruff-pre-commit` from v0.16.1 to 0.16.5
- [Release notes](https://github.com/astral-sh/ruff-pre-commit/releases)
- [Commits](https://github.com/astral-sh/ruff-pre-commit/compare/v0.16.1...v0.16.5)

Updates `https://github.com/tox-dev/pyproject-fmt` from v2.26.0 to 2.28.1
- [Release notes](https://github.com/tox-dev/pyproject-fmt/releases)
- [Commits](https://github.com/tox-dev/pyproject-fmt/compare/v2.26.0...v2.28.1)

Updates `https://github.com/AleksaC/hadolint-py` from v2.14.0 to 2.15.1
- [Commits](https://github.com/AleksaC/hadolint-py/compare/v2.14.0...v2.15.1)

---
updated-dependencies:
- dependency-name: https://github.com/AleksaC/hadolint-py
  dependency-version: 2.15.1
  dependency-type: direct:production
  dependency-group: pre-commit-dependencies
- dependency-name: https://github.com/astral-sh/ruff-pre-commit
  dependency-version: 0.16.4
  dependency-type: direct:production
  dependency-group: pre-commit-dependencies
- dependency-name: https://github.com/tox-dev/pyproject-fmt
  dependency-version: 2.28.1
  dependency-type: direct:production
  dependency-group: pre-commit-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>

* Runs the linting

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: stumpylog <797416+stumpylog@users.noreply.github.com>
 2026-09-04 18:38:40 +00:00
dependabot[bot]andstumpylog cf34844168 Chore(deps): Bump the document-processing group across 1 directory with 4 updates (#13916) 
* Chore(deps): Bump the document-processing group across 1 directory with 4 updates

Bumps the document-processing group with 4 updates in the / directory: [gotenberg-client](https://github.com/stumpylog/gotenberg-client), [ocrmypdf](https://github.com/ocrmypdf/OCRmyPDF), [tika-client](https://github.com/stumpylog/tika-client) and [zxing-cpp](https://github.com/zxing-cpp/zxing-cpp).


Updates `gotenberg-client` from 0.14.0 to 1.0.0
- [Release notes](https://github.com/stumpylog/gotenberg-client/releases)
- [Changelog](https://github.com/stumpylog/gotenberg-client/blob/main/CHANGELOG.md)
- [Commits](https://github.com/stumpylog/gotenberg-client/compare/0.14.0...1.0.0)

Updates `ocrmypdf` from 17.7.1 to 17.10.0
- [Release notes](https://github.com/ocrmypdf/OCRmyPDF/releases)
- [Commits](https://github.com/ocrmypdf/OCRmyPDF/compare/v17.7.1...v17.10.0)

Updates `tika-client` from 0.11.0 to 1.0.0
- [Release notes](https://github.com/stumpylog/tika-client/releases)
- [Changelog](https://github.com/stumpylog/tika-client/blob/main/CHANGELOG.md)
- [Commits](https://github.com/stumpylog/tika-client/compare/0.11.0...1.0.0)

Updates `zxing-cpp` from 3.1.0 to 3.1.1
- [Release notes](https://github.com/zxing-cpp/zxing-cpp/releases)
- [Commits](https://github.com/zxing-cpp/zxing-cpp/compare/v3.1.0...v3.1.1)

---
updated-dependencies:
- dependency-name: gotenberg-client
  dependency-version: 1.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: document-processing
- dependency-name: ocrmypdf
  dependency-version: 17.10.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: document-processing
- dependency-name: tika-client
  dependency-version: 1.0.0
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: document-processing
- dependency-name: zxing-cpp
  dependency-version: 3.1.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: document-processing
...

Signed-off-by: dependabot[bot] <support@github.com>

* Install the httpx extra of gotenberg-client v1

* Did the same thing here, forgot

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: stumpylog <797416+stumpylog@users.noreply.github.com>
 2026-09-04 18:02:34 +00:00
dependabot[bot]andstumpylog 3c575b5202 Chore(deps): Bump flower from 2.0.1 to 2.1.0 in the async-tasks group across 1 directory (#13913) 
* Chore(deps): Bump flower in the async-tasks group across 1 directory

Bumps the async-tasks group with 1 update in the / directory: [flower](https://github.com/mher/flower).


Updates `flower` from 2.0.1 to 2.1.0
- [Release notes](https://github.com/mher/flower/releases)
- [Changelog](https://github.com/mher/flower/blob/v2.1.0/CHANGELOG.md)
- [Commits](https://github.com/mher/flower/compare/v2.0.1...v2.1.0)

---
updated-dependencies:
- dependency-name: flower
  dependency-version: 2.1.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: async-tasks
...

Signed-off-by: dependabot[bot] <support@github.com>

* Linting

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: stumpylog <797416+stumpylog@users.noreply.github.com>
 2026-09-04 16:58:05 +00:00
shamoon d48663e9eb Bump version to 3.1.3 2026-09-03 18:59:18 -07:00
Trenton Handshamoon 351892bbab Fix: skip vector store document id filter for unrestricted chat users (#13937) 
* Fix: skip vector store document id filter for unrestricted chat users

ChatStreamingView built an IN filter from every permitted document id
for the "chat over all documents" case, which exceeds the vector
store's SQLite bound-parameter safety limit on installs with more
than ~32700 documents, silently returning no context. For a user who
can see every document (an active superuser), that filter never
narrows anything, so skip it and let the retriever search the whole
index instead.

* Minor improvements from a Claude review

* When a user is unrestricted chatting, still exclude trashed documents using a 'NOT IN' SQL statement.  Wire that up where we need it

* Update src/paperless_ai/chat.py

Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
 2026-09-02 18:08:54 +00:00
Trenton H f993462973 Security: Minor additional hardening (#13898) 
* Security: bump jinja2 floor to 3.1.6 (CVE-2025-27516)

* Security: anchor the /share/ URL pattern

* Security: handle missing file on public share view without 500

* Security: scope correspondent last_correspondence to permitted documents

* Security: disable PUT/PATCH on share link bundles
 2026-09-01 19:53:28 +00:00
shamoon ca98dffbd2 Bump version to 3.1.2 2026-09-01 08:09:13 -07:00
shamoon a4499dc9c1 Bump version to 3.1.1 2026-08-30 22:36:29 -07:00
shamoon 1630e78aac Bump version to 3.1.0 2026-08-26 18:07:04 -07:00
Trenton H 5f9bc5de88 Chore: Upgrade Docker image to Python 3.14 (#13721) 
* Upgrades our base image to uv 0.12 branch and Python 3.14

* Upgrades our workflows to uv 0.12.x as well

* Updates these locked wheels too
 2026-08-19 09:43:14 -07:00
Trenton HandClaude Sonnet 5 fc242bb570 Performance: unify permission-filtering backends, fixes Correspondent/Tag list slowness (#13601) 
* feat: add unified PermittedObjectsFilter backed by permitted_object_ids

* refactor: migrate all ViewSets to unified PermittedObjectsFilter

Replace the deprecated ObjectOwnedOrGrantedPermissionsFilter,
DocumentPermissionsFilter, and ObjectOwnedPermissionsFilter aliases
with PermittedObjectsFilter directly across documents/views.py (8
sites, including TrashView's include_granted=False subclass) and
paperless_mail/views.py (3 sites), then delete the now-unreferenced
alias classes from documents/filters.py.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UFyrt7FWbRRdTUAcdqBcsc

* docs: document legacy status of get_objects_for_user_owner_aware/has_perms_owner_aware

Stage 4's PermittedObjectsFilter/permitted_object_ids() covers the
queryset-filtering use case, but both functions still have production
callers outside this plan's scope (documents/views.py,
documents/serialisers.py, documents/signals/handlers.py,
paperless_ai/matching.py, paperless_ai/ai_classifier.py). Per Task 20
Step 2, they are kept in place rather than partially deleted, with
docstrings updated to note their legacy status and remaining callers.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UFyrt7FWbRRdTUAcdqBcsc

* Fix: address final review findings for permission-filter unification

- Add a permanent regression test pinning TrashView's include_granted=False
  wiring: an explicit view_document grant on a trashed document must not
  leak it into /api/trash/ for a non-owner, non-superuser requester.
- Drop the now-dead direct dependency djangorestframework-guardian; the
  last rest_framework_guardian import was removed by this branch's
  migration onto PermittedObjectsFilter. django-guardian is untouched.
- Replace the hand-maintained, already-stale caller lists in
  get_objects_for_user_owner_aware/has_perms_owner_aware docstrings with a
  pointer to grep for remaining callers instead.
- In PermittedObjectsFilter.filter_queryset, compute `model` only on the
  include_granted=True path that actually uses it.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UFyrt7FWbRRdTUAcdqBcsc

* perf: check bulk-edit-objects apply_to_all permissions via DB-side exclude/exists

Materialized the full permitted_object_ids() set into a Python set() just
to check membership for the request's objs queryset -- the same pattern
already fixed at four other sites for Document. This one is used by
apply_to_all, where objs can be an unbounded filtered selection (e.g. all
tags matching a filter) rather than a small request-supplied ID list,
making the wasted materialization worse here than at the sites already
fixed.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

* Cleans up the comment about why this is still here for now

---------

Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
 2026-08-08 07:27:18 -07:00
dependabot[bot] e86adb6c36 Chore(deps): Bump the utilities-patch group across 1 directory with 6 updates (#13539) 
Bumps the utilities-patch group with 6 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [dateparser](https://github.com/scrapinghub/dateparser) | `1.4.0` | `1.4.1` |
| [django-guardian](https://github.com/django-guardian/django-guardian) | `3.3.1` | `3.3.3` |
| [ijson](https://github.com/ICRAR/ijson) | `3.5.0` | `3.5.1` |
| [llama-index-core](https://github.com/run-llama/llama_index) | `0.14.22` | `0.14.23` |
| [daphne](https://github.com/django/daphne) | `4.2.2` | `4.2.3` |
| [types-dateparser](https://github.com/python/typeshed) | `1.4.0.20260328` | `1.4.1.20260617` |



Updates `dateparser` from 1.4.0 to 1.4.1
- [Release notes](https://github.com/scrapinghub/dateparser/releases)
- [Changelog](https://github.com/scrapinghub/dateparser/blob/master/HISTORY.rst)
- [Commits](https://github.com/scrapinghub/dateparser/compare/v1.4.0...v1.4.1)

Updates `django-guardian` from 3.3.1 to 3.3.3
- [Release notes](https://github.com/django-guardian/django-guardian/releases)
- [Commits](https://github.com/django-guardian/django-guardian/compare/3.3.1...3.3.3)

Updates `ijson` from 3.5.0 to 3.5.1
- [Changelog](https://github.com/ICRAR/ijson/blob/master/CHANGELOG.md)
- [Commits](https://github.com/ICRAR/ijson/compare/v3.5.0...v3.5.1)

Updates `llama-index-core` from 0.14.22 to 0.14.23
- [Release notes](https://github.com/run-llama/llama_index/releases)
- [Changelog](https://github.com/run-llama/llama_index/blob/main/CHANGELOG.md)
- [Commits](https://github.com/run-llama/llama_index/compare/v0.14.22...v0.14.23)

Updates `daphne` from 4.2.2 to 4.2.3
- [Changelog](https://github.com/django/daphne/blob/main/CHANGELOG.txt)
- [Commits](https://github.com/django/daphne/compare/4.2.2...4.2.3)

Updates `types-dateparser` from 1.4.0.20260328 to 1.4.1.20260617
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: daphne
  dependency-version: 4.2.3
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: utilities-patch
- dependency-name: dateparser
  dependency-version: 1.4.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: utilities-patch
- dependency-name: django-guardian
  dependency-version: 3.3.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: utilities-patch
- dependency-name: ijson
  dependency-version: 3.5.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: utilities-patch
- dependency-name: llama-index-core
  dependency-version: 0.14.23
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: utilities-patch
- dependency-name: types-dateparser
  dependency-version: 1.4.1.20260617
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: utilities-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-08-04 09:27:22 -07:00
dependabot[bot]andstumpylog 74aa10f31f Chore(deps): Bump the utilities-minor group across 1 directory with 20 updates (#13535) 
* Chore(deps): Bump the utilities-minor group across 1 directory with 20 updates

Bumps the utilities-minor group with 20 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [drf-spectacular](https://github.com/tfranzel/drf-spectacular) | `0.29.0` | `0.30.0` |
| [drf-spectacular-sidecar](https://github.com/tfranzel/drf-spectacular-sidecar) | `2026.5.1` | `2026.7.1` |
| [filelock](https://github.com/tox-dev/py-filelock) | `3.29.0` | `3.32.0` |
| [httpx-oauth](https://github.com/frankie567/httpx-oauth) | `0.16.1` | `0.17.0` |
| [imap-tools](https://github.com/ikvk/imap_tools) | `1.13.0` | `1.14.0` |
| [ocrmypdf](https://github.com/ocrmypdf/OCRmyPDF) | `17.4.2` | `17.8.1` |
| [openai](https://github.com/openai/openai-python) | `2.32.0` | `2.48.0` |
| [regex](https://github.com/mrabarnett/mrab-regex) | `2026.4.4` | `2026.7.19` |
| [scikit-learn](https://github.com/scikit-learn/scikit-learn) | `1.8.0` | `1.9.0` |
| [sentence-transformers](https://github.com/huggingface/sentence-transformers) | `5.4.1` | `5.6.1` |
| [watchfiles](https://github.com/samuelcolvin/watchfiles) | `1.1.1` | `1.2.0` |
| [zxing-cpp](https://github.com/zxing-cpp/zxing-cpp) | `3.0.0` | `3.1.0` |
| [prek](https://github.com/j178/prek) | `0.3.10` | `0.4.11` |
| [faker](https://github.com/joke2k/faker) | `40.15.0` | `40.36.0` |
| [pytest](https://github.com/pytest-dev/pytest) | `9.0.3` | `9.1.1` |
| [pytest-env](https://github.com/pytest-dev/pytest-env) | `1.6.0` | `1.7.0` |
| [pytest-rerunfailures](https://github.com/pytest-dev/pytest-rerunfailures) | `16.1` | `16.4` |
| [types-bleach](https://github.com/python/typeshed) | `6.3.0.20251115` | `6.4.0.20260607` |
| [types-pytz](https://github.com/python/typeshed) | `2026.1.1.20260304` | `2026.3.1.20260727` |
| [types-regex](https://github.com/python/typeshed) | `2026.4.4.20260408` | `2026.7.19.20260720` |

Updates `drf-spectacular` from 0.29.0 to 0.30.0
- [Release notes](https://github.com/tfranzel/drf-spectacular/releases)
- [Changelog](https://github.com/tfranzel/drf-spectacular/blob/master/CHANGELOG.rst)
- [Commits](https://github.com/tfranzel/drf-spectacular/compare/0.29.0...0.30.0)

Updates `drf-spectacular-sidecar` from 2026.5.1 to 2026.7.1
- [Commits](https://github.com/tfranzel/drf-spectacular-sidecar/compare/2026.5.1...2026.7.1)

Updates `filelock` from 3.29.0 to 3.32.0
- [Release notes](https://github.com/tox-dev/py-filelock/releases)
- [Changelog](https://github.com/tox-dev/filelock/blob/main/docs/changelog.rst)
- [Commits](https://github.com/tox-dev/py-filelock/compare/3.29.0...3.32.0)

Updates `httpx-oauth` from 0.16.1 to 0.17.0
- [Release notes](https://github.com/frankie567/httpx-oauth/releases)
- [Commits](https://github.com/frankie567/httpx-oauth/compare/v0.16.1...v0.17.0)

Updates `imap-tools` from 1.13.0 to 1.14.0
- [Release notes](https://github.com/ikvk/imap_tools/releases)
- [Changelog](https://github.com/ikvk/imap_tools/blob/master/docs/release_notes.rst)
- [Commits](https://github.com/ikvk/imap_tools/compare/v1.13.0...v1.14.0)

Updates `ocrmypdf` from 17.4.2 to 17.8.1
- [Release notes](https://github.com/ocrmypdf/OCRmyPDF/releases)
- [Commits](https://github.com/ocrmypdf/OCRmyPDF/compare/v17.4.2...v17.8.1)

Updates `openai` from 2.32.0 to 2.48.0
- [Release notes](https://github.com/openai/openai-python/releases)
- [Changelog](https://github.com/openai/openai-python/blob/main/CHANGELOG.md)
- [Commits](https://github.com/openai/openai-python/compare/v2.32.0...v2.48.0)

Updates `regex` from 2026.4.4 to 2026.7.19
- [Changelog](https://github.com/mrabarnett/mrab-regex/blob/hg/changelog.txt)
- [Commits](https://github.com/mrabarnett/mrab-regex/compare/2026.4.4...2026.7.19)

Updates `scikit-learn` from 1.8.0 to 1.9.0
- [Release notes](https://github.com/scikit-learn/scikit-learn/releases)
- [Commits](https://github.com/scikit-learn/scikit-learn/compare/1.8.0...1.9.0)

Updates `sentence-transformers` from 5.4.1 to 5.6.1
- [Release notes](https://github.com/huggingface/sentence-transformers/releases)
- [Commits](https://github.com/huggingface/sentence-transformers/compare/v5.4.1...v5.6.1)

Updates `watchfiles` from 1.1.1 to 1.2.0
- [Release notes](https://github.com/samuelcolvin/watchfiles/releases)
- [Commits](https://github.com/samuelcolvin/watchfiles/compare/v1.1.1...v1.2.0)

Updates `zxing-cpp` from 3.0.0 to 3.1.0
- [Release notes](https://github.com/zxing-cpp/zxing-cpp/releases)
- [Commits](https://github.com/zxing-cpp/zxing-cpp/compare/v3.0.0...v3.1.0)

Updates `prek` from 0.3.10 to 0.4.11
- [Release notes](https://github.com/j178/prek/releases)
- [Changelog](https://github.com/j178/prek/blob/master/CHANGELOG.md)
- [Commits](https://github.com/j178/prek/compare/v0.3.10...v0.4.11)

Updates `faker` from 40.15.0 to 40.36.0
- [Release notes](https://github.com/joke2k/faker/releases)
- [Changelog](https://github.com/joke2k/faker/blob/master/CHANGELOG.md)
- [Commits](https://github.com/joke2k/faker/compare/v40.15.0...v40.36.0)

Updates `pytest` from 9.0.3 to 9.1.1
- [Release notes](https://github.com/pytest-dev/pytest/releases)
- [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/pytest-dev/pytest/compare/9.0.3...9.1.1)

Updates `pytest-env` from 1.6.0 to 1.7.0
- [Release notes](https://github.com/pytest-dev/pytest-env/releases)
- [Commits](https://github.com/pytest-dev/pytest-env/compare/1.6.0...1.7.0)

Updates `pytest-rerunfailures` from 16.1 to 16.4
- [Changelog](https://github.com/pytest-dev/pytest-rerunfailures/blob/master/CHANGES.rst)
- [Commits](https://github.com/pytest-dev/pytest-rerunfailures/compare/16.1...16.4)

Updates `types-bleach` from 6.3.0.20251115 to 6.4.0.20260607
- [Commits](https://github.com/python/typeshed/commits)

Updates `types-pytz` from 2026.1.1.20260304 to 2026.3.1.20260727
- [Commits](https://github.com/python/typeshed/commits)

Updates `types-regex` from 2026.4.4.20260408 to 2026.7.19.20260720
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: drf-spectacular
  dependency-version: 0.30.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: drf-spectacular-sidecar
  dependency-version: 2026.7.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: filelock
  dependency-version: 3.32.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: httpx-oauth
  dependency-version: 0.17.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: imap-tools
  dependency-version: 1.14.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: ocrmypdf
  dependency-version: 17.8.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: openai
  dependency-version: 2.48.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: regex
  dependency-version: 2026.7.19
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: scikit-learn
  dependency-version: 1.9.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: sentence-transformers
  dependency-version: 5.6.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: watchfiles
  dependency-version: 1.2.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: zxing-cpp
  dependency-version: 3.1.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: prek
  dependency-version: 0.4.11
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: faker
  dependency-version: 40.36.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: pytest
  dependency-version: 9.1.1
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: pytest-env
  dependency-version: 1.7.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: pytest-rerunfailures
  dependency-version: '16.4'
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: types-bleach
  dependency-version: 6.4.0.20260607
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: types-pytz
  dependency-version: 2026.3.1.20260727
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: types-regex
  dependency-version: 2026.7.19.20260720
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Downgrades ocrmypdf to avoid ocrmypdf/OCRmyPDF/issues/1713

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: stumpylog <797416+stumpylog@users.noreply.github.com>
 2026-08-04 15:56:03 +00:00
Trenton H 2ea4a792fe Performance: More efficient mail fetching to reduce the amount of data pulled (#13432) 2026-08-04 15:35:30 +00:00
dependabot[bot] 4ca1863dee Chore(deps-dev): Bump zensical (#13533) 
Bumps the development group with 1 update in the / directory: [zensical](https://github.com/zensical/zensical).


Updates `zensical` from 0.0.47 to 0.0.51
- [Release notes](https://github.com/zensical/zensical/releases)
- [Commits](https://github.com/zensical/zensical/compare/v0.0.47...v0.0.51)

---
updated-dependencies:
- dependency-name: zensical
  dependency-version: 0.0.51
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: development
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-08-03 23:07:45 +00:00
Trenton H dd99c2289d Chore: ruff 0.16 upgrade (#13531) 2026-08-03 21:00:04 +00:00
shamoon 8fb73b2709 Bump version to 3.0.5 2026-08-01 14:38:08 -07:00
shamoon 3bc03bbaec Bump version to 3.0.4 2026-07-27 20:14:39 -07:00
shamoon b5a23cff2f Bump version to 3.0.3 2026-07-25 16:32:39 -07:00
shamoon 6d249b4932 Bump version to 3.0.2 2026-07-23 18:12:59 -07:00
stumpylog 5569447ea0 Bumps the version to 3.0.1 everywhere 2026-07-23 15:59:53 -07:00
dependabot[bot] 59a737f669 Chore(deps): Bump the uv group across 1 directory with 8 updates (#13189) 
---
updated-dependencies:
- dependency-name: aiohttp
  dependency-version: 3.14.1
  dependency-type: indirect
- dependency-name: cryptography
  dependency-version: 48.0.1
  dependency-type: indirect
- dependency-name: pi-heif
  dependency-version: 1.3.0
  dependency-type: indirect
- dependency-name: pillow
  dependency-version: 12.3.0
  dependency-type: indirect
- dependency-name: pyjwt
  dependency-version: 2.13.0
  dependency-type: indirect
- dependency-name: setuptools
  dependency-version: 83.0.0
  dependency-type: indirect
- dependency-name: torch
  dependency-version: 2.13.0
  dependency-type: direct:production
- dependency-name: tornado
  dependency-version: 6.5.7
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-07-23 17:58:06 +00:00
Trenton H a0ab05a9a6 Chore: Upgrade more vulerable packages (#13020) 
* Few more before

* Bump django to 5.2.16

* Bump daphne to 4.2.2

* Bump transformers to 5.13.1

* Bump nltk to 3.10.0
 2026-07-17 12:01:30 -07:00
dependabot[bot] 5428ea5caa Chore(deps-dev): Bump the development group with 2 updates (#13068) 
Signed-off-by: dependabot[bot] <support@github.com>
 2026-07-15 21:04:28 +00:00
shamoon bf70e597ee Merge branch 'beta' into dev 2026-06-23 07:32:33 -07:00
Trenton HandClaude Opus 4.8 a020f64d08 Enhancement(beta): replace LanceDB vector store with sqlite-vec (#12990) 
* Chore(beta): add sqlite-vec 0.1.9 dependency

Pinned exactly: the 0.1.9 wheels carry no baked SIMD flags (safe on
pre-AVX2 CPUs, the point of this migration); the 0.1.10 alphas bake
-mavx and would reintroduce the #12970 crash class.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Test(beta): port vector store tests to sqlite-vec backend

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Enhancement(beta): switch AI vector store from LanceDB to sqlite-vec

Fixes the non-AVX2 SIGILL class (#12970) at the root: lancedb is no
longer imported. sqlite-vec 0.1.9 wheels carry no baked SIMD, vec0
metadata columns give parameterized EQ/IN filtering, WAL preserves the
lock-free-reader model, and compact() rebuilds the table because vec0
DELETEs never reclaim space.

Implementation notes vs. the Task 3A draft:
- compact() uses a file-swap approach (new db file + Path.replace) rather
  than ALTER TABLE RENAME, which does not cascade to shadow tables in
  sqlite-vec 0.1.9 (upstream limitation).
- Bloat is tracked via a cumulative total_inserts counter in index_meta
  because the _rowids shadow table does not accumulate deleted rows in
  0.1.9 (contrary to the design doc assumption from #54).
- None distances from the zero-vector cosine edge case are mapped to
  similarity 0.0 rather than raising TypeError.
- Test suite updated accordingly: _bloat_ratio reads index_meta instead
  of _rowids; seed collision in force-compact test fixed (seed=100.0).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Enhancement(beta): wire indexing pipeline to the sqlite-vec store

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Enhancement(beta): move filename/storage path/ASN to node metadata

Same treatment as title/tags/correspondent in #12944: excluded from
the embedded text, visible to the LLM via metadata prepend. Changes
embedded text for every document, so it ships inside the sqlite-vec
transition, whose forced rebuild re-embeds everything anyway.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Test(beta): cover legacy LanceDB index cleanup and forced rebuild

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Chore(beta): drop lancedb dependency

Fixes #12970: the package whose wheels SIGILL on non-AVX2 CPUs is no
longer installed at all.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Chore(beta): partial pyrefly cleanup on sqlite-vec vector store

- Add MetadataFilter import and isinstance guard in _build_where()
- Add query_embedding None guard in query()
- Fix dict.get() type-checker ambiguity in get_configured_model_name()

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Chore(beta): drop automatic LanceDB index cleanup on startup

Leave legacy Lance directory removal to the user rather than deleting it
automatically on first run. Beta policy: user is expected to do a clean
re-embed anyway; no need for the system to silently delete their data.

Remove _cleanup_legacy_lance_index(), the forced-rebuild path that called
it, and the associated tests.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Chore(beta): ruff format pass on sqlite-vec AI files

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Removes the benchmarking file

* Try to resolve or silence some semgrep.  But we're using SQL here, not an ORM and we control the inputs, not users

* Enhancement(beta): add schema migration machinery to sqlite-vec vector store

Adds versioned schema migration support modelled after PR #12968's LanceDB
approach, adapted for sqlite-vec's file-swap compaction pattern.

- SCHEMA_VERSION = 1 written to index_meta at table creation and preserved
  through compact()
- Migration dataclass with from_version, to_version, kind ("structural" or
  "re-embed"), description, and an optional apply(src, dst, dim) callable
- MIGRATIONS registry (empty at v1 baseline); add entries and bump
  SCHEMA_VERSION when the schema changes
- check_and_run_migrations(): structural migrations run via the same
  file-swap as compact() (no re-embed); re-embed migrations return True
  so the caller forces a full rebuild
- update_llm_index() calls check_and_run_migrations() under the write lock
  before any indexing work

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Chore(beta): deduplicate vector store internals via helper methods

Extract three helpers to remove copy-paste between compact() and
_run_structural_migration():
- _meta_set_on(conn, key, value): static upsert into any connection's
  index_meta; _meta_set() now delegates to it
- _create_vec_table(conn, dim): CREATE VIRTUAL TABLE DDL (carries the
  nosemgrep annotation)
- _swap_in_compact(compact_path, db_path): close/replace/reconnect
  sequence used by both file-swap callers

Also normalises compact() error-path cleanup to unlink(missing_ok=True).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Adds equality test and no covers some defensive error handling stuff

* Ensures an embed migration stops the migration chain, just in case

* Silence one kind right but not really semgrep

* Trims dead assignment

* Fix(beta): address Copilot review on sqlite-vec vector store

Three findings from the PR review:

- compact() failure cleanup now unlinks the temporary .compact-wal and
  .compact-shm files, matching _run_structural_migration(); previously
  only the main .compact file was removed.
- _build_where() fails closed (1 = 0) when filters are requested but none
  translate, instead of emitting "()" which is invalid SQL; filters scope
  document access, so an empty translation must match no rows.
- Drop the unused table_name constructor parameter (all SQL hardcodes
  DEFAULT_TABLE_NAME) and its callers in indexing.py.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Enhancement(beta): guard sqlite-vec compaction swap against concurrent readers

The compaction/migration file swap replaces the database via os.replace,
but the -wal/-shm files are keyed by path, not inode. A reader holding an
open connection across the swap leaves the old WAL aliased onto the new
file; a subsequent write then corrupts the database (reproduced via
PRAGMA integrity_check).

Add a cross-process read/write lock (filelock.ReadWriteLock) over the
index:

- read_store() holds it shared for the whole connection lifetime (and
  closes the connection on exit); concurrent readers do not block.
- compaction and the migration check run under an exclusive lock that
  drains readers, and skip with an info log on Timeout (maintenance op,
  retries next run).
- Normal writes are untouched: WAL gives reader/writer concurrency and
  LLM_INDEX_LOCK still serializes writers, so they never block readers.

load_or_build_index() now takes the store from the caller's read_store()
so the lock and connection span the whole retrieval; chat holds it across
the streamed response. Two new settings: LLM_INDEX_RWLOCK and
LLM_INDEX_COMPACTION_LOCK_TIMEOUT.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Ensures the store alays cleans up SQLite connections for any operations, even on errors

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-06-15 13:20:41 -07:00
dependabot[bot] b0227dd080 Chore(deps): Bump the uv group across 1 directory with 2 updates (#12995) 
Bumps the uv group with 2 updates in the / directory: [torch](https://github.com/pytorch/pytorch) and [tornado](https://github.com/tornadoweb/tornado).


Updates `torch` from 2.11.0 to 2.12.0
- [Release notes](https://github.com/pytorch/pytorch/releases)
- [Changelog](https://github.com/pytorch/pytorch/blob/main/RELEASE.md)
- [Commits](https://github.com/pytorch/pytorch/compare/v2.11.0...v2.12.0)

Updates `tornado` from 6.5.5 to 6.5.6
- [Changelog](https://github.com/tornadoweb/tornado/blob/master/docs/releases.rst)
- [Commits](https://github.com/tornadoweb/tornado/compare/v6.5.5...v6.5.6)

---
updated-dependencies:
- dependency-name: torch
  dependency-version: 2.12.0
  dependency-type: direct:production
- dependency-name: tornado
  dependency-version: 6.5.6
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-06-15 09:14:17 -07:00
dependabot[bot]andstumpylog 40d927a9ff Chore(deps): Bump the utilities-patch group across 1 directory with 4 updates (#12931) 
* Chore(deps): Bump the utilities-patch group across 1 directory with 4 updates

Bumps the utilities-patch group with 4 updates in the / directory: [llama-index-core](https://github.com/run-llama/llama_index), [psycopg-pool](https://github.com/psycopg/psycopg), [zensical](https://github.com/zensical/zensical) and [ruff](https://github.com/astral-sh/ruff).


Updates `llama-index-core` from 0.14.21 to 0.14.22
- [Release notes](https://github.com/run-llama/llama_index/releases)
- [Changelog](https://github.com/run-llama/llama_index/blob/main/CHANGELOG.md)
- [Commits](https://github.com/run-llama/llama_index/compare/v0.14.21...v0.14.22)

Updates `psycopg-pool` from 3.3 to 3.3.1
- [Changelog](https://github.com/psycopg/psycopg/blob/master/docs/news.rst)
- [Commits](https://github.com/psycopg/psycopg/compare/3.3.0...3.3.1)

Updates `zensical` from 0.0.36 to 0.0.43
- [Release notes](https://github.com/zensical/zensical/releases)
- [Commits](https://github.com/zensical/zensical/compare/v0.0.36...v0.0.43)

Updates `ruff` from 0.15.12 to 0.15.15
- [Release notes](https://github.com/astral-sh/ruff/releases)
- [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md)
- [Commits](https://github.com/astral-sh/ruff/compare/0.15.12...0.15.15)

---
updated-dependencies:
- dependency-name: llama-index-core
  dependency-version: 0.14.22
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: utilities-patch
- dependency-name: psycopg-pool
  dependency-version: 3.3.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: utilities-patch
- dependency-name: ruff
  dependency-version: 0.15.14
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: utilities-patch
- dependency-name: zensical
  dependency-version: 0.0.43
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: utilities-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Syncs hook versions and runs them

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: stumpylog <797416+stumpylog@users.noreply.github.com>
 2026-06-15 08:55:18 -07:00
Trenton HClaude Opus 4.8shamoon
 		eb292baa69 Enhancement (beta): Switch the AI vector store to LanceDB (#12944) 
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: shamoon <shamoon@users.noreply.github.com>
 2026-06-07 11:31:26 -07:00
Trenton H abdcdccf08 Chore(deps): Silence a couple more vulnerabilities here (#12797) 2026-06-03 09:28:00 -07:00
shamoon 79d0a04df0 Enhancement: support ollama embeddings (#12753) 2026-05-09 00:06:14 +00:00
Trenton H 8b6e8142f1 Upgrades Django to the latest, cryptography, django-allauth for the release (#12731) 2026-05-06 15:07:13 -07:00
Trenton H 76b2b6ad36 Bumps all our versions to 3.0.0 (#12715) 2026-05-05 12:40:24 -07:00
dependabot[bot] 7e0dc2bca4 Chore(deps): Bump the utilities-patch group across 1 directory with 7 updates (#12702) 
Bumps the utilities-patch group with 7 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [drf-spectacular-sidecar](https://github.com/tfranzel/drf-spectacular-sidecar) | `2026.4.1` | `2026.4.14` |
| [llama-index-core](https://github.com/run-llama/llama_index) | `0.14.19` | `0.14.21` |
| [rapidfuzz](https://github.com/rapidfuzz/RapidFuzz) | `3.14.3` | `3.14.5` |
| [prek](https://github.com/j178/prek) | `0.3.8` | `0.3.10` |
| [pytest-httpx](https://github.com/Colin-b/pytest_httpx) | `0.36.0` | `0.36.2` |
| [mypy](https://github.com/python/mypy) | `1.20.0` | `1.20.2` |
| [mypy-baseline](https://github.com/orsinium-labs/mypy-baseline) | `0.7.3` | `0.7.4` |



Updates `drf-spectacular-sidecar` from 2026.4.1 to 2026.4.14
- [Commits](https://github.com/tfranzel/drf-spectacular-sidecar/compare/2026.4.1...2026.4.14)

Updates `llama-index-core` from 0.14.19 to 0.14.21
- [Release notes](https://github.com/run-llama/llama_index/releases)
- [Changelog](https://github.com/run-llama/llama_index/blob/main/CHANGELOG.md)
- [Commits](https://github.com/run-llama/llama_index/compare/v0.14.19...v0.14.21)

Updates `rapidfuzz` from 3.14.3 to 3.14.5
- [Release notes](https://github.com/rapidfuzz/RapidFuzz/releases)
- [Changelog](https://github.com/rapidfuzz/RapidFuzz/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/rapidfuzz/RapidFuzz/compare/v3.14.3...v3.14.5)

Updates `prek` from 0.3.8 to 0.3.10
- [Release notes](https://github.com/j178/prek/releases)
- [Changelog](https://github.com/j178/prek/blob/master/CHANGELOG.md)
- [Commits](https://github.com/j178/prek/compare/v0.3.8...v0.3.10)

Updates `pytest-httpx` from 0.36.0 to 0.36.2
- [Release notes](https://github.com/Colin-b/pytest_httpx/releases)
- [Changelog](https://github.com/Colin-b/pytest_httpx/blob/develop/CHANGELOG.md)
- [Commits](https://github.com/Colin-b/pytest_httpx/compare/v0.36.0...0.36.2)

Updates `mypy` from 1.20.0 to 1.20.2
- [Changelog](https://github.com/python/mypy/blob/master/CHANGELOG.md)
- [Commits](https://github.com/python/mypy/compare/v1.20.0...v1.20.2)

Updates `mypy-baseline` from 0.7.3 to 0.7.4
- [Release notes](https://github.com/orsinium-labs/mypy-baseline/releases)
- [Changelog](https://github.com/orsinium-labs/mypy-baseline/blob/master/docs/history.md)
- [Commits](https://github.com/orsinium-labs/mypy-baseline/compare/0.7.3...0.7.4)

---
updated-dependencies:
- dependency-name: drf-spectacular-sidecar
  dependency-version: 2026.4.14
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: utilities-patch
- dependency-name: llama-index-core
  dependency-version: 0.14.21
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: utilities-patch
- dependency-name: rapidfuzz
  dependency-version: 3.14.5
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: utilities-patch
- dependency-name: prek
  dependency-version: 0.3.10
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: utilities-patch
- dependency-name: pytest-httpx
  dependency-version: 0.36.2
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: utilities-patch
- dependency-name: mypy
  dependency-version: 1.20.2
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: utilities-patch
- dependency-name: mypy-baseline
  dependency-version: 0.7.4
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: utilities-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-05-03 14:56:05 -07:00
dependabot[bot]andTrenton Holmes c72c4809c2 Chore(deps): Bump the utilities-minor group across 1 directory with 9 updates (#12696) 
* Chore(deps): Bump the utilities-minor group across 1 directory with 9 updates

Bumps the utilities-minor group with 9 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [django-treenode](https://github.com/fabiocaccamo/django-treenode) | `0.23.3` | `0.24.0` |
| [filelock](https://github.com/tox-dev/py-filelock) | `3.25.2` | `3.29.0` |
| [imap-tools](https://github.com/ikvk/imap_tools) | `1.11.1` | `1.12.1` |
| [openai](https://github.com/openai/openai-python) | `2.30.0` | `2.32.0` |
| [regex](https://github.com/mrabarnett/mrab-regex) | `2026.3.32` | `2026.4.4` |
| [sentence-transformers](https://github.com/huggingface/sentence-transformers) | `5.3.0` | `5.4.1` |
| [faker](https://github.com/joke2k/faker) | `40.12.0` | `40.15.0` |
| [pyrefly](https://github.com/facebook/pyrefly) | `0.59.0` | `0.62.0` |
| [types-pygments](https://github.com/python/typeshed) | `2.19.0.20251121` | `2.20.0.20260408` |



Updates `django-treenode` from 0.23.3 to 0.24.0
- [Release notes](https://github.com/fabiocaccamo/django-treenode/releases)
- [Changelog](https://github.com/fabiocaccamo/django-treenode/blob/main/CHANGELOG.md)
- [Commits](https://github.com/fabiocaccamo/django-treenode/compare/0.23.3...0.24.0)

Updates `filelock` from 3.25.2 to 3.29.0
- [Release notes](https://github.com/tox-dev/py-filelock/releases)
- [Changelog](https://github.com/tox-dev/filelock/blob/main/docs/changelog.rst)
- [Commits](https://github.com/tox-dev/py-filelock/compare/3.25.2...3.29.0)

Updates `imap-tools` from 1.11.1 to 1.12.1
- [Release notes](https://github.com/ikvk/imap_tools/releases)
- [Changelog](https://github.com/ikvk/imap_tools/blob/master/docs/release_notes.rst)
- [Commits](https://github.com/ikvk/imap_tools/compare/v1.11.1...v1.12.1)

Updates `openai` from 2.30.0 to 2.32.0
- [Release notes](https://github.com/openai/openai-python/releases)
- [Changelog](https://github.com/openai/openai-python/blob/main/CHANGELOG.md)
- [Commits](https://github.com/openai/openai-python/compare/v2.30.0...v2.32.0)

Updates `regex` from 2026.3.32 to 2026.4.4
- [Changelog](https://github.com/mrabarnett/mrab-regex/blob/hg/changelog.txt)
- [Commits](https://github.com/mrabarnett/mrab-regex/compare/2026.3.32...2026.4.4)

Updates `sentence-transformers` from 5.3.0 to 5.4.1
- [Release notes](https://github.com/huggingface/sentence-transformers/releases)
- [Commits](https://github.com/huggingface/sentence-transformers/compare/v5.3.0...v5.4.1)

Updates `faker` from 40.12.0 to 40.15.0
- [Release notes](https://github.com/joke2k/faker/releases)
- [Changelog](https://github.com/joke2k/faker/blob/master/CHANGELOG.md)
- [Commits](https://github.com/joke2k/faker/compare/v40.12.0...v40.15.0)

Updates `pyrefly` from 0.59.0 to 0.62.0
- [Release notes](https://github.com/facebook/pyrefly/releases)
- [Commits](https://github.com/facebook/pyrefly/compare/0.59.0...0.62.0)

Updates `types-pygments` from 2.19.0.20251121 to 2.20.0.20260408
- [Commits](https://github.com/python/typeshed/commits)

---
updated-dependencies:
- dependency-name: django-treenode
  dependency-version: 0.24.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: faker
  dependency-version: 40.15.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: filelock
  dependency-version: 3.29.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: imap-tools
  dependency-version: 1.12.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: openai
  dependency-version: 2.32.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: pyrefly
  dependency-version: 0.62.0
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: regex
  dependency-version: 2026.4.4
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: sentence-transformers
  dependency-version: 5.4.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
- dependency-name: types-pygments
  dependency-version: 2.20.0.20260408
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: utilities-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Linting

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Trenton Holmes <797416+stumpylog@users.noreply.github.com>
 2026-05-03 21:09:27 +00:00
dependabot[bot] 3bbb5166a1 Chore(deps): Bump ocrmypdf (#12687) 
Bumps the document-processing group with 1 update in the / directory: [ocrmypdf](https://github.com/ocrmypdf/OCRmyPDF).


Updates `ocrmypdf` from 17.4.0 to 17.4.2
- [Release notes](https://github.com/ocrmypdf/OCRmyPDF/releases)
- [Commits](https://github.com/ocrmypdf/OCRmyPDF/compare/v17.4.0...v17.4.2)

---
updated-dependencies:
- dependency-name: ocrmypdf
  dependency-version: 17.4.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: document-processing
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-05-02 22:02:49 +00:00
dependabot[bot] aa5485019d Chore(deps-dev): Bump the development group with 2 updates (#12683) 
Bumps the development group with 2 updates: [zensical](https://github.com/zensical/zensical) and [ruff](https://github.com/astral-sh/ruff).


Updates `zensical` from 0.0.31 to 0.0.36
- [Release notes](https://github.com/zensical/zensical/releases)
- [Commits](https://github.com/zensical/zensical/compare/v0.0.31...v0.0.36)

Updates `ruff` from 0.15.8 to 0.15.12
- [Release notes](https://github.com/astral-sh/ruff/releases)
- [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md)
- [Commits](https://github.com/astral-sh/ruff/compare/0.15.8...0.15.12)

---
updated-dependencies:
- dependency-name: zensical
  dependency-version: 0.0.36
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: development
- dependency-name: ruff
  dependency-version: 0.15.12
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: development
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2026-05-02 21:48:15 +00:00
Trenton H 493d282059 Chore: Upgrades tantivy-py to the latest release (#12605) 2026-04-29 10:09:50 -07:00
shamoon 69cb4d06c6 Enhancement (dev): Use OpenAI-like backend (#12668) 2026-04-28 10:06:59 -07:00