amorfo77/paperless-ngx
1
0
Fork 0
mirror of https://github.com/paperless-ngx/paperless-ngx.git synced 2026-07-02 18:24:17 +00:00
Code Issues Projects Releases Wiki Activity

Feature: Replace Whoosh with tantivy search backend (#12471)

Browse Source 
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Antoine Mérino <3023499+Merinorus@users.noreply.github.com>
This commit is contained in:
Trenton H
2026-04-02 12:38:22 -07:00
committed by GitHub
parent e01a762e81
commit aed9abe48c
52 changed files with 4050 additions and 1708 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/administration.md
+41 -6
View File
@@ -180,6 +180,16 @@ following:
This might not actually do anything. Not every new paperless version
comes with new database migrations.
4. Rebuild the search index if needed.
```shell-session
cd src
python3 manage.py document_index reindex --if-needed
```
This is a no-op if the index is already up to date, so it is safe to
run on every upgrade.
### Database Upgrades
Paperless-ngx is compatible with Django-supported versions of PostgreSQL and MariaDB and it is generally
@@ -453,17 +463,42 @@ the search yields non-existing documents or won't find anything, you
may need to recreate the index manually.
```
document_index {reindex,optimize}
document_index {reindex,optimize} [--recreate] [--if-needed]
```
Specify `reindex` to have the index created from scratch. This may take
some time.
Specify `reindex` to rebuild the index from all documents in the database. This
may take some time.
Specify `optimize` to optimize the index. This updates certain aspects
of the index and usually makes queries faster and also ensures that the
autocompletion works properly. This command is regularly invoked by the
Pass `--recreate` to wipe the existing index before rebuilding. Use this when the
index is corrupted or you want a fully clean rebuild.
Pass `--if-needed` to skip the rebuild if the index is already up to date (schema
version and search language match). Safe to run on every startup or upgrade.
Specify `optimize` to optimize the index. This command is regularly invoked by the
task scheduler.
!!! note
The `optimize` subcommand is deprecated and is now a no-op. Tantivy manages
segment merging automatically; no manual optimization step is needed.
!!! note
**Docker users:** On every startup, the container runs
`document_index reindex --if-needed` automatically. Schema changes, language
changes, and missing indexes are all detected and rebuilt before the webserver
starts. No manual step is required.
**Bare metal users:** Run the following command after each upgrade (and after
changing `PAPERLESS_SEARCH_LANGUAGE`). It is a no-op if the index is already
up to date:
```shell-session
cd src
python3 manage.py document_index reindex --if-needed
```
### Clearing the database read cache
If the database read cache is enabled, **you must run this command** after making any changes to the database outside the application context.
docs/api.md
+2 -3
View File
@@ -167,9 +167,8 @@ Query parameters:
- `term`: The incomplete term.
- `limit`: Amount of results. Defaults to 10.
Results returned by the endpoint are ordered by importance of the term
in the document index. The first result is the term that has the highest
[Tf/Idf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf) score in the index.
Results are ordered by how many of the user's visible documents contain
each matching word. The first result is the word that appears in the most documents.
```json
["term1", "term3", "term6", "term4"]
docs/configuration.md
+26
View File
@@ -1103,6 +1103,32 @@ should be a valid crontab(5) expression describing when to run.
Defaults to `0 0 * * *` or daily at midnight.
#### [`PAPERLESS_SEARCH_LANGUAGE=<language>`](#PAPERLESS_SEARCH_LANGUAGE) {#PAPERLESS_SEARCH_LANGUAGE}
: Sets the stemmer language for the full-text search index.
Stemming improves recall by matching word variants (e.g. "running" matches "run").
Changing this setting causes the index to be rebuilt automatically on next startup.
An invalid value raises an error at startup.
: Use the ISO 639-1 two-letter code (e.g. `en`, `de`, `fr`). Lowercase full names
(e.g. `english`, `german`, `french`) are also accepted. The capitalized names shown
in the [Tantivy Language enum](https://docs.rs/tantivy/latest/tantivy/tokenizer/enum.Language.html)
documentation are **not** valid — use the lowercase equivalent.
: If not set, paperless infers the language from
[`PAPERLESS_OCR_LANGUAGE`](#PAPERLESS_OCR_LANGUAGE). If the OCR language has no
Tantivy stemmer equivalent, stemming is disabled.
Defaults to unset (inferred from `PAPERLESS_OCR_LANGUAGE`).
#### [`PAPERLESS_ADVANCED_FUZZY_SEARCH_THRESHOLD=<float>`](#PAPERLESS_ADVANCED_FUZZY_SEARCH_THRESHOLD) {#PAPERLESS_ADVANCED_FUZZY_SEARCH_THRESHOLD}
: When set to a float value, approximate/fuzzy matching is applied alongside exact
matching. Fuzzy results rank below exact matches. A value of `0.5` is a reasonable
starting point. Leave unset to disable fuzzy matching entirely.
Defaults to unset (disabled).
#### [`PAPERLESS_SANITY_TASK_CRON=<cron expression>`](#PAPERLESS_SANITY_TASK_CRON) {#PAPERLESS_SANITY_TASK_CRON}
: Configures the scheduled sanity checker frequency. The value should be a
docs/migration-v3.md
+31
View File
@@ -104,6 +104,37 @@ Multiple options are combined in a single value:
PAPERLESS_DB_OPTIONS="sslmode=require;sslrootcert=/certs/ca.pem;pool.max_size=10"
```
## Search Index (Whoosh -> Tantivy)
The full-text search backend has been replaced with [Tantivy](https://github.com/quickwit-oss/tantivy).
The index format is incompatible with Whoosh, so **the search index is automatically rebuilt from
scratch on first startup after upgrading**. No manual action is required for the rebuild itself.
### Note and custom field search syntax
The old Whoosh index exposed `note` and `custom_field` as flat text fields that were included in
unqualified searches (e.g. just typing `invoice` would match note content). With Tantivy these are
now structured JSON fields accessed via dotted paths:
| Old syntax | New syntax |
| -------------------- | --------------------------- |
| `note:query` | `notes.note:query` |
| `custom_field:query` | `custom_fields.value:query` |
**Saved views are migrated automatically.** Any saved view filter rule that used an explicit
`note:` or `custom_field:` field prefix in a fulltext query is rewritten to the new syntax by a
data migration that runs on upgrade.
**Unqualified queries are not migrated.** If you had a saved view with a plain search term (e.g.
`invoice`) that happened to match note content or custom field values, it will no longer return
those matches. Update those queries to use the explicit prefix, for example:
```
invoice OR notes.note:invoice OR custom_fields.value:invoice
```
Custom field names can also be searched with `custom_fields.name:fieldname`.
## OpenID Connect Token Endpoint Authentication
Some existing OpenID Connect setups may require an explicit token endpoint authentication method after upgrading to v3.
docs/usage.md
+71 -13
View File
@@ -804,13 +804,20 @@ contract you signed 8 years ago).
When you search paperless for a document, it tries to match this query
against your documents. Paperless will look for matching documents by
inspecting their content, title, correspondent, type and tags. Paperless
returns a scored list of results, so that documents matching your query
better will appear further up in the search results.
inspecting their content, title, correspondent, type, tags, notes, and
custom field values. Paperless returns a scored list of results, so that
documents matching your query better will appear further up in the search
results.
By default, paperless returns only documents which contain all words
typed in the search bar. However, paperless also offers advanced search
syntax if you want to drill down the results further.
typed in the search bar. A few things to know about how matching works:
- **Word-order-independent**: "invoice unpaid" and "unpaid invoice" return the same results.
- **Accent-insensitive**: searching `resume` also finds `résumé`, `cafe` finds `café`.
- **Separator-agnostic**: punctuation and separators are stripped during indexing, so
searching a partial number like `1312` finds documents containing `A-1312/B`.
Paperless also offers advanced search syntax if you want to drill down further.
Matching documents with logical expressions:
@@ -839,18 +846,69 @@ Matching inexact words:
produ*name
```
Matching natural date keywords:
```
added:today
modified:yesterday
created:this_week
added:last_month
modified:this_year
```
Supported date keywords: `today`, `yesterday`, `this_week`, `last_week`,
`this_month`, `last_month`, `this_year`, `last_year`.
#### Searching custom fields
Custom field values are included in the full-text index, so a plain search
already matches documents whose custom field values contain your search terms.
To narrow by field name or value specifically:
```
custom_fields.value:policy
custom_fields.name:"Contract Number"
custom_fields.name:Insurance custom_fields.value:policy
```
- `custom_fields.value` matches against the value of any custom field.
- `custom_fields.name` matches the name of the field (use quotes for multi-word names).
- Combine both to find documents where a specific named field contains a specific value.
Because separators are stripped during indexing, individual parts of formatted
codes are searchable on their own. A value stored as `A-1312/99.50` produces the
tokens `a`, `1312`, `99`, `50` — each searchable independently:
```
custom_fields.value:1312
custom_fields.name:"Contract Number" custom_fields.value:1312
```
!!! note
Inexact terms are hard for search indexes. These queries might take a
while to execute. That's why paperless offers auto complete and query
correction.
Custom date fields do not support relative date syntax (e.g. `[now to 2 weeks]`).
For date ranges on custom date fields, use the document list filters in the web UI.
#### Searching notes
Notes content is included in full-text search automatically. To search
by note author or content specifically:
```
notes.user:alice
notes.note:reminder
notes.user:alice notes.note:insurance
```
All of these constructs can be combined as you see fit. If you want to
learn more about the query language used by paperless, paperless uses
Whoosh's default query language. Head over to [Whoosh query
language](https://whoosh.readthedocs.io/en/latest/querylang.html). For
details on what date parsing utilities are available, see [Date
parsing](https://whoosh.readthedocs.io/en/latest/dates.html#parsing-date-queries).
learn more about the query language used by paperless, see the
[Tantivy query language documentation](https://docs.rs/tantivy/latest/tantivy/query/struct.QueryParser.html).
!!! note
Fuzzy (approximate) matching can be enabled by setting
[`PAPERLESS_ADVANCED_FUZZY_SEARCH_THRESHOLD`](configuration.md#PAPERLESS_ADVANCED_FUZZY_SEARCH_THRESHOLD).
When enabled, paperless will include near-miss results ranked below exact matches.
## Keyboard shortcuts / hotkeys