mirror of
https://github.com/paperless-ngx/paperless-ngx.git
synced 2026-07-31 16:15:58 +00:00
Compare commits
28
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
cfb9125d9e | ||
|
|
fa2d7f660b | ||
|
|
2712c3fe48 | ||
|
|
684c4ef392 | ||
|
|
410e440bd6 | ||
|
|
ef9ee435e8 | ||
|
|
5d8287f120 | ||
|
|
acfdafbff9 | ||
|
|
cd02cfe02c | ||
|
|
ff9cde7f9c | ||
|
|
259731074c | ||
|
|
02aaf9b3a2 | ||
|
|
8a34385199 | ||
|
|
3e8656a3ed | ||
|
|
8ec322f9aa | ||
|
|
22e4be1548 | ||
|
|
17e0b70da5 | ||
|
|
668fa77428 | ||
|
|
5bd72014a6 | ||
|
|
bbb9c86ba4 | ||
|
|
09c9fa03cf | ||
|
|
04779d72bb | ||
|
|
1b32b9d678 | ||
|
|
12d318deff | ||
|
|
9c98f30d3f | ||
|
|
6a1d7b1bca | ||
|
|
de9f520143 | ||
|
|
a68c487d60 |
@@ -0,0 +1,187 @@
|
|||||||
|
---
|
||||||
|
name: paperless-benchmarking
|
||||||
|
description: Use when profiling paperless-ngx performance, running `manage.py benchmark`, investigating a slow query or endpoint, or deciding whether a profiling finding should become a permanent registered scenario. Covers command reference, the fork/merge-back branch workflow for perf investigations, and how to read query-plan output.
|
||||||
|
---
|
||||||
|
|
||||||
|
# Paperless-ngx Benchmarking
|
||||||
|
|
||||||
|
This repo has a built-in benchmarking/profiling tool: `manage.py benchmark`, in
|
||||||
|
the `paperless_benchmark` Django app. It replaces ad hoc standalone scripts —
|
||||||
|
use it instead of writing new one-off seed/timing scripts.
|
||||||
|
|
||||||
|
## Command reference
|
||||||
|
|
||||||
|
```
|
||||||
|
manage.py benchmark seed --tier {home,medium,large} [--reset --yes-i-know-this-wipes-the-database] [--seed N]
|
||||||
|
manage.py benchmark run --repeat 5 [--label baseline]
|
||||||
|
manage.py benchmark profile <scenario_name> [--repeat 5] [--explain]
|
||||||
|
manage.py benchmark list-scenarios
|
||||||
|
```
|
||||||
|
|
||||||
|
- **`seed`** builds a realistic dataset at one of three scales: `home` (500
|
||||||
|
documents — fast, use this for iteration), `medium` (20,000 — the default
|
||||||
|
when `--tier` is omitted; a multi-minute seed), `large` (360,000 — matches
|
||||||
|
the scale reported in real large-install bug reports; slow, only use it
|
||||||
|
when a finding needs confirming at real scale). `--reset` wipes any
|
||||||
|
previously-seeded benchmark data first — but it is destructive and
|
||||||
|
irreversible: it deletes **all** users, **all** groups, and **all**
|
||||||
|
documents/tags/correspondents/document types/storage paths in the target
|
||||||
|
database, not just benchmark-created rows. Only run it against a disposable
|
||||||
|
benchmark database, never a real install. Because of that, `--reset` also
|
||||||
|
requires passing `--yes-i-know-this-wipes-the-database` in the same
|
||||||
|
invocation, or the command raises an error and does nothing. Omit both
|
||||||
|
flags if you want to layer more data onto an existing seed instead. `seed`
|
||||||
|
creates two named users, `perf_target` (mixed owned/shared documents,
|
||||||
|
realistic guardian permission grants) and `perf_admin` (superuser), plus a
|
||||||
|
general user/group pool with realistic permission-row ratios.
|
||||||
|
- **`run`** times the 3 built-in API endpoint benchmarks
|
||||||
|
(`/api/documents/`, `/api/documents/?page_size=50`, `/api/tags/?page_size=100000`) for both
|
||||||
|
`perf_target` and `perf_admin`, reporting min/median/max wall-clock and SQL
|
||||||
|
query count. Requires `seed` to have already run — it reuses that data, it
|
||||||
|
does not seed its own.
|
||||||
|
- **`profile`** times one named scenario from the registry (see
|
||||||
|
`list-scenarios`) via best-of-N repeat timing and SQL query count, against
|
||||||
|
`perf_target`. `--explain` additionally captures and prints the query plan:
|
||||||
|
real `EXPLAIN ANALYZE` execution stats on PostgreSQL/MariaDB, or
|
||||||
|
`EXPLAIN QUERY PLAN` (plan only, no real timing/row counts — clearly labeled
|
||||||
|
as such) on SQLite. Like `run`, `profile` requires `seed` to have already
|
||||||
|
run — it does not seed its own data either.
|
||||||
|
- Every `run`/`profile` invocation appends a JSON line to
|
||||||
|
`benchmark_results/history.jsonl` at the repo root (local-only, gitignored —
|
||||||
|
never commit this file). Use it to compare a `before`/`after` pair across
|
||||||
|
two invocations without hand-copying numbers.
|
||||||
|
- Full chain example: `seed --reset` once, then `run` and `profile` as many
|
||||||
|
times as you want against that same seeded data — no need to reseed between
|
||||||
|
them.
|
||||||
|
|
||||||
|
## Adding a new scenario
|
||||||
|
|
||||||
|
A "scenario" is a named, registered query/operation that `profile` can time
|
||||||
|
and explain. To add one, edit `src/paperless_benchmark/scenarios.py`: write a
|
||||||
|
`_<name>_run(user)` function (returns whatever `run_profile` should time) and
|
||||||
|
optionally a `_<name>_queryset(user)` function (returns the `QuerySet` for
|
||||||
|
`--explain` to analyze), then `register(Scenario(name=..., describe=...,
|
||||||
|
run=..., queryset_for_explain=...))` at module level. Both functions receive
|
||||||
|
the already-seeded `perf_target` user — they should not seed their own data.
|
||||||
|
|
||||||
|
## Branch workflow
|
||||||
|
|
||||||
|
`tools/benchmark-management-commands` is a **long-lived tooling branch**, not
|
||||||
|
a feature branch that gets merged and closed:
|
||||||
|
|
||||||
|
1. It is periodically brought up to date with `dev` (merge `dev` into it) so
|
||||||
|
the tooling doesn't drift from the schema/codebase it profiles. Do this
|
||||||
|
before starting a new investigation if it's been a while since the last
|
||||||
|
sync.
|
||||||
|
2. **Every performance investigation forks its own branch from
|
||||||
|
`tools/benchmark-management-commands`** (not from `dev`). Do the
|
||||||
|
investigation there: write throwaway profiling code, try fixes, capture
|
||||||
|
before/after numbers.
|
||||||
|
3. **That investigation branch never merges into `dev` or production.** Its
|
||||||
|
only job is to produce evidence and, optionally, a reusable scenario.
|
||||||
|
4. If the investigation turns up a scenario worth keeping permanently (see
|
||||||
|
"When to graduate a scenario" below), open a PR that adds **just that
|
||||||
|
scenario** back into `tools/benchmark-management-commands` — not the rest
|
||||||
|
of the investigation branch's throwaway code.
|
||||||
|
5. Any actual production fix the investigation motivates (e.g. an ORM query
|
||||||
|
change) goes into its own normal feature branch off `dev`, following the
|
||||||
|
project's regular contribution process — profiling evidence informs that
|
||||||
|
PR's description, but the profiling code itself does not travel with it.
|
||||||
|
|
||||||
|
## Reading query-plan output
|
||||||
|
|
||||||
|
- **PostgreSQL** `EXPLAIN ANALYZE`: look for `Seq Scan` on a large table
|
||||||
|
(missing index), a large gap between `rows=N` (planner's estimate) and the
|
||||||
|
actual row count in parentheses (stale statistics or a bad cardinality
|
||||||
|
estimate), and nested-loop joins driven by an outer relation with many
|
||||||
|
rows (usually the N+1 pattern this tool exists to catch).
|
||||||
|
- **MariaDB**: verified against a real MariaDB 12.3 container that MariaDB
|
||||||
|
does NOT accept MySQL 8.0.18+'s `EXPLAIN ANALYZE` syntax (it's a 1064
|
||||||
|
syntax error) -- `capture_explain()` instead runs MariaDB's own
|
||||||
|
`ANALYZE <statement>` form (no `EXPLAIN` keyword), which returns a
|
||||||
|
tabular plan with real per-row execution columns: `rows` (estimate) vs.
|
||||||
|
`r_rows` (actual), and `filtered` vs. `r_filtered`. A large gap between
|
||||||
|
`rows` and `r_rows`, or `type: ALL` (full table scan) on a large table,
|
||||||
|
are the signals to look for -- the same underlying concerns as Postgres's
|
||||||
|
`Seq Scan`/estimate-vs-actual gap, just in MariaDB's column-based output
|
||||||
|
instead of Postgres's nested-tree text format.
|
||||||
|
- **SQLite** `EXPLAIN QUERY PLAN`: no real timing/row-count data, only the
|
||||||
|
chosen access path (`SCAN` vs `SEARCH`, which index if any). Useful for
|
||||||
|
confirming an index is even being considered, not for judging real-world
|
||||||
|
cost — corroborate any SQLite finding against Postgres/MariaDB before
|
||||||
|
trusting it, since planner behavior differs meaningfully between them.
|
||||||
|
- Compare query **count**, not just timing, between before/after: a fix that
|
||||||
|
keeps the same wall-clock time but drops query count from O(n) to O(1) is
|
||||||
|
still a real, durable improvement — timing alone is noisy and
|
||||||
|
environment-dependent, query count is not.
|
||||||
|
|
||||||
|
## Cleaning up after an interrupted run
|
||||||
|
|
||||||
|
If a `seed`/`run`/`profile` invocation gets killed mid-run (Ctrl-C, `kill -9`,
|
||||||
|
a timed-out SSH session, etc.), check whether it left anything behind before
|
||||||
|
trusting the next benchmark's numbers. This was verified for real: a
|
||||||
|
`benchmark seed --tier large --reset ...` was started against both a fresh
|
||||||
|
PostgreSQL 18 container and a fresh MariaDB 12.3 container and `kill -9`'d a
|
||||||
|
few seconds into document seeding. In both cases, the database-side
|
||||||
|
connection disappeared immediately -- no stuck backend, no lingering query,
|
||||||
|
no held lock was observed in either backend once the killed process's PID
|
||||||
|
was confirmed gone. That said, this was one interruption point (mid
|
||||||
|
bulk-seed, between chunks); a run killed mid-query, or a driver/network
|
||||||
|
hiccup that doesn't cleanly close the socket, could behave differently, so
|
||||||
|
still check before trusting a number if any run in the session was
|
||||||
|
interrupted:
|
||||||
|
|
||||||
|
- **PostgreSQL**: look for leftover connections against the benchmark
|
||||||
|
database:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
SELECT pid, state, query, query_start
|
||||||
|
FROM pg_stat_activity
|
||||||
|
WHERE datname = current_database() AND pid <> pg_backend_pid();
|
||||||
|
```
|
||||||
|
|
||||||
|
If a stuck backend shows up, clear it with:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
SELECT pg_terminate_backend(pid)
|
||||||
|
FROM pg_stat_activity
|
||||||
|
WHERE datname = current_database() AND pid <> pg_backend_pid();
|
||||||
|
```
|
||||||
|
|
||||||
|
- **MariaDB**: look for leftover connections/queries:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
SHOW FULL PROCESSLIST;
|
||||||
|
```
|
||||||
|
|
||||||
|
If a stuck connection shows up (anything other than your current admin
|
||||||
|
session), clear it with:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
KILL <id>;
|
||||||
|
```
|
||||||
|
|
||||||
|
A stray connection left running concurrently with a subsequent benchmark run
|
||||||
|
would add real, contaminating load (extra queries competing for the same
|
||||||
|
rows, possibly held locks slowing the next run's timings) -- cheap to rule
|
||||||
|
out, expensive to silently trust a number that was actually measured
|
||||||
|
alongside a zombie connection.
|
||||||
|
|
||||||
|
## When to graduate a one-off finding into a permanent scenario
|
||||||
|
|
||||||
|
Register a scenario (rather than leaving it as throwaway code on the
|
||||||
|
investigation branch) when **both** are true:
|
||||||
|
|
||||||
|
- The query pattern is one this codebase is likely to regress on again (e.g.
|
||||||
|
it involves a permission-check join, a bulk operation, or anything else
|
||||||
|
with an easy-to-reintroduce N+1) — not a one-time fluke specific to this
|
||||||
|
investigation.
|
||||||
|
- Re-running it later, against a fresh seed, would still produce a
|
||||||
|
meaningful signal (it doesn't depend on investigation-specific throwaway
|
||||||
|
data or a fix that's already permanently landed and can't regress the same
|
||||||
|
way).
|
||||||
|
|
||||||
|
If a finding doesn't meet both bars, keep it as disposable code on the
|
||||||
|
investigation branch and let the branch's evidence (captured in the PR
|
||||||
|
description of whatever production fix it motivates) be the permanent
|
||||||
|
record instead.
|
||||||
@@ -115,3 +115,6 @@ celerybeat-schedule*
|
|||||||
|
|
||||||
# Git worktree local folder
|
# Git worktree local folder
|
||||||
.worktrees
|
.worktrees
|
||||||
|
|
||||||
|
# Benchmark tooling output (local only, never committed)
|
||||||
|
/benchmark_results/
|
||||||
|
|||||||
@@ -620,6 +620,34 @@ no other workflow will be executed on the document.
|
|||||||
If a "Move to Trash" action is executed in a consume pipeline, the consumption
|
If a "Move to Trash" action is executed in a consume pipeline, the consumption
|
||||||
will be aborted and the file will be deleted.
|
will be aborted and the file will be deleted.
|
||||||
|
|
||||||
|
##### Password Removal {#workflow-action-password-removal}
|
||||||
|
|
||||||
|
"Password Removal" actions attempt to remove password protection from encrypted PDF documents. You can specify:
|
||||||
|
|
||||||
|
- One or more passwords to try, separated by commas or new lines
|
||||||
|
- Each password is tried in order until one successfully unlocks the document
|
||||||
|
|
||||||
|
Password removal never modifies a file in place. Instead, once a working password is found, the
|
||||||
|
decrypted content is consumed as a new [document version](#document-file-versions), leaving the
|
||||||
|
original (still encrypted) version in the document's version history.
|
||||||
|
|
||||||
|
**Consumption Started**: because this trigger fires before the document exists yet, the password
|
||||||
|
removal itself is deferred until after the initial consumption of the encrypted file has completed.
|
||||||
|
OCR engines cannot process an encrypted PDF, so this first version is typically stored with no
|
||||||
|
extracted text (unless the file already contained extractable text outside of OCR). Immediately
|
||||||
|
afterwards, the password is removed and the decrypted file is automatically re-consumed as a second,
|
||||||
|
new version of the same document, this time with normal OCR/text extraction applied. In other words,
|
||||||
|
a password-protected file added with this trigger will briefly exist as an un-OCR'd version before
|
||||||
|
the properly processed version is created.
|
||||||
|
|
||||||
|
**Document Added**, **Document Updated**, **Scheduled**: these triggers run against a document that
|
||||||
|
already exists, so password removal happens immediately: the decrypted content is queued for
|
||||||
|
consumption as a new version right away. Note that if the document's initial consumption also
|
||||||
|
happened while it was still encrypted, that original version will likewise be missing OCR text.
|
||||||
|
|
||||||
|
**Current limitation**: Passwords are stored as a simple list without descriptions. To handle
|
||||||
|
multiple PDF types with different passwords, create separate workflows for each use case.
|
||||||
|
|
||||||
#### Workflow placeholders
|
#### Workflow placeholders
|
||||||
|
|
||||||
Titles and webhook payloads can be generated by workflows using [Jinja templates](https://jinja.palletsprojects.com/en/3.1.x/templates/).
|
Titles and webhook payloads can be generated by workflows using [Jinja templates](https://jinja.palletsprojects.com/en/3.1.x/templates/).
|
||||||
|
|||||||
+35
-35
@@ -659,7 +659,7 @@
|
|||||||
</context-group>
|
</context-group>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||||
<context context-type="linenumber">445</context>
|
<context context-type="linenumber">449</context>
|
||||||
</context-group>
|
</context-group>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/document-list/bulk-editor/custom-fields-bulk-edit-dialog/custom-fields-bulk-edit-dialog.component.html</context>
|
<context context-type="sourcefile">src/app/components/document-list/bulk-editor/custom-fields-bulk-edit-dialog/custom-fields-bulk-edit-dialog.component.html</context>
|
||||||
@@ -831,7 +831,7 @@
|
|||||||
</context-group>
|
</context-group>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||||
<context context-type="linenumber">464</context>
|
<context context-type="linenumber">468</context>
|
||||||
</context-group>
|
</context-group>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/document-list/document-list.component.html</context>
|
<context context-type="sourcefile">src/app/components/document-list/document-list.component.html</context>
|
||||||
@@ -1355,7 +1355,7 @@
|
|||||||
</context-group>
|
</context-group>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||||
<context context-type="linenumber">390</context>
|
<context context-type="linenumber">394</context>
|
||||||
</context-group>
|
</context-group>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.html</context>
|
<context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.html</context>
|
||||||
@@ -1596,7 +1596,7 @@
|
|||||||
</context-group>
|
</context-group>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||||
<context context-type="linenumber">165</context>
|
<context context-type="linenumber">169</context>
|
||||||
</context-group>
|
</context-group>
|
||||||
</trans-unit>
|
</trans-unit>
|
||||||
<trans-unit id="2691296884221415710" datatype="html">
|
<trans-unit id="2691296884221415710" datatype="html">
|
||||||
@@ -1607,7 +1607,7 @@
|
|||||||
</context-group>
|
</context-group>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||||
<context context-type="linenumber">170</context>
|
<context context-type="linenumber">174</context>
|
||||||
</context-group>
|
</context-group>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.html</context>
|
<context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.html</context>
|
||||||
@@ -1638,7 +1638,7 @@
|
|||||||
</context-group>
|
</context-group>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||||
<context context-type="linenumber">174</context>
|
<context context-type="linenumber">178</context>
|
||||||
</context-group>
|
</context-group>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.html</context>
|
<context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.html</context>
|
||||||
@@ -1669,7 +1669,7 @@
|
|||||||
</context-group>
|
</context-group>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||||
<context context-type="linenumber">178</context>
|
<context context-type="linenumber">182</context>
|
||||||
</context-group>
|
</context-group>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.html</context>
|
<context context-type="sourcefile">src/app/components/document-list/bulk-editor/bulk-editor.component.html</context>
|
||||||
@@ -4914,7 +4914,7 @@
|
|||||||
</context-group>
|
</context-group>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||||
<context context-type="linenumber">356</context>
|
<context context-type="linenumber">360</context>
|
||||||
</context-group>
|
</context-group>
|
||||||
</trans-unit>
|
</trans-unit>
|
||||||
<trans-unit id="8057014866157903311" datatype="html">
|
<trans-unit id="8057014866157903311" datatype="html">
|
||||||
@@ -7760,14 +7760,14 @@
|
|||||||
<source>Details</source>
|
<source>Details</source>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||||
<context context-type="linenumber">160</context>
|
<context context-type="linenumber">164</context>
|
||||||
</context-group>
|
</context-group>
|
||||||
</trans-unit>
|
</trans-unit>
|
||||||
<trans-unit id="5701618810648052610" datatype="html">
|
<trans-unit id="5701618810648052610" datatype="html">
|
||||||
<source>Title</source>
|
<source>Title</source>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||||
<context context-type="linenumber">163</context>
|
<context context-type="linenumber">167</context>
|
||||||
</context-group>
|
</context-group>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/document-list/document-list.component.html</context>
|
<context context-type="sourcefile">src/app/components/document-list/document-list.component.html</context>
|
||||||
@@ -7790,14 +7790,14 @@
|
|||||||
<source>Date created</source>
|
<source>Date created</source>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||||
<context context-type="linenumber">167</context>
|
<context context-type="linenumber">171</context>
|
||||||
</context-group>
|
</context-group>
|
||||||
</trans-unit>
|
</trans-unit>
|
||||||
<trans-unit id="5607669932062416162" datatype="html">
|
<trans-unit id="5607669932062416162" datatype="html">
|
||||||
<source>Default</source>
|
<source>Default</source>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||||
<context context-type="linenumber">179</context>
|
<context context-type="linenumber">183</context>
|
||||||
</context-group>
|
</context-group>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/manage/saved-views/saved-views.component.html</context>
|
<context context-type="sourcefile">src/app/components/manage/saved-views/saved-views.component.html</context>
|
||||||
@@ -7808,14 +7808,14 @@
|
|||||||
<source>Content</source>
|
<source>Content</source>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||||
<context context-type="linenumber">286</context>
|
<context context-type="linenumber">290</context>
|
||||||
</context-group>
|
</context-group>
|
||||||
</trans-unit>
|
</trans-unit>
|
||||||
<trans-unit id="218403386307979629" datatype="html">
|
<trans-unit id="218403386307979629" datatype="html">
|
||||||
<source>Metadata</source>
|
<source>Metadata</source>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||||
<context context-type="linenumber">295</context>
|
<context context-type="linenumber">299</context>
|
||||||
</context-group>
|
</context-group>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/document-detail/metadata-collapse/metadata-collapse.component.ts</context>
|
<context context-type="sourcefile">src/app/components/document-detail/metadata-collapse/metadata-collapse.component.ts</context>
|
||||||
@@ -7826,147 +7826,147 @@
|
|||||||
<source>Date modified</source>
|
<source>Date modified</source>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||||
<context context-type="linenumber">302</context>
|
<context context-type="linenumber">306</context>
|
||||||
</context-group>
|
</context-group>
|
||||||
</trans-unit>
|
</trans-unit>
|
||||||
<trans-unit id="6392918669949841614" datatype="html">
|
<trans-unit id="6392918669949841614" datatype="html">
|
||||||
<source>Date added</source>
|
<source>Date added</source>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||||
<context context-type="linenumber">306</context>
|
<context context-type="linenumber">310</context>
|
||||||
</context-group>
|
</context-group>
|
||||||
</trans-unit>
|
</trans-unit>
|
||||||
<trans-unit id="146828917013192897" datatype="html">
|
<trans-unit id="146828917013192897" datatype="html">
|
||||||
<source>Media filename</source>
|
<source>Media filename</source>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||||
<context context-type="linenumber">310</context>
|
<context context-type="linenumber">314</context>
|
||||||
</context-group>
|
</context-group>
|
||||||
</trans-unit>
|
</trans-unit>
|
||||||
<trans-unit id="4500855521601039868" datatype="html">
|
<trans-unit id="4500855521601039868" datatype="html">
|
||||||
<source>Original filename</source>
|
<source>Original filename</source>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||||
<context context-type="linenumber">314</context>
|
<context context-type="linenumber">318</context>
|
||||||
</context-group>
|
</context-group>
|
||||||
</trans-unit>
|
</trans-unit>
|
||||||
<trans-unit id="2659735245739197634" datatype="html">
|
<trans-unit id="2659735245739197634" datatype="html">
|
||||||
<source>Original SHA256 checksum</source>
|
<source>Original SHA256 checksum</source>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||||
<context context-type="linenumber">318</context>
|
<context context-type="linenumber">322</context>
|
||||||
</context-group>
|
</context-group>
|
||||||
</trans-unit>
|
</trans-unit>
|
||||||
<trans-unit id="5888243105821763422" datatype="html">
|
<trans-unit id="5888243105821763422" datatype="html">
|
||||||
<source>Original file size</source>
|
<source>Original file size</source>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||||
<context context-type="linenumber">322</context>
|
<context context-type="linenumber">326</context>
|
||||||
</context-group>
|
</context-group>
|
||||||
</trans-unit>
|
</trans-unit>
|
||||||
<trans-unit id="2696647325713149563" datatype="html">
|
<trans-unit id="2696647325713149563" datatype="html">
|
||||||
<source>Original mime type</source>
|
<source>Original mime type</source>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||||
<context context-type="linenumber">326</context>
|
<context context-type="linenumber">330</context>
|
||||||
</context-group>
|
</context-group>
|
||||||
</trans-unit>
|
</trans-unit>
|
||||||
<trans-unit id="6714358112223607756" datatype="html">
|
<trans-unit id="6714358112223607756" datatype="html">
|
||||||
<source>Archive SHA256 checksum</source>
|
<source>Archive SHA256 checksum</source>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||||
<context context-type="linenumber">331</context>
|
<context context-type="linenumber">335</context>
|
||||||
</context-group>
|
</context-group>
|
||||||
</trans-unit>
|
</trans-unit>
|
||||||
<trans-unit id="6033581412811562084" datatype="html">
|
<trans-unit id="6033581412811562084" datatype="html">
|
||||||
<source>Archive file size</source>
|
<source>Archive file size</source>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||||
<context context-type="linenumber">337</context>
|
<context context-type="linenumber">341</context>
|
||||||
</context-group>
|
</context-group>
|
||||||
</trans-unit>
|
</trans-unit>
|
||||||
<trans-unit id="6992781481378431874" datatype="html">
|
<trans-unit id="6992781481378431874" datatype="html">
|
||||||
<source>Original document metadata</source>
|
<source>Original document metadata</source>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||||
<context context-type="linenumber">346</context>
|
<context context-type="linenumber">350</context>
|
||||||
</context-group>
|
</context-group>
|
||||||
</trans-unit>
|
</trans-unit>
|
||||||
<trans-unit id="2846565152091361585" datatype="html">
|
<trans-unit id="2846565152091361585" datatype="html">
|
||||||
<source>Archived document metadata</source>
|
<source>Archived document metadata</source>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||||
<context context-type="linenumber">349</context>
|
<context context-type="linenumber">353</context>
|
||||||
</context-group>
|
</context-group>
|
||||||
</trans-unit>
|
</trans-unit>
|
||||||
<trans-unit id="7206723502037428235" datatype="html">
|
<trans-unit id="7206723502037428235" datatype="html">
|
||||||
<source>Notes <x id="START_BLOCK_IF" equiv-text="@if (document()?.notes.length) {"/><x id="START_TAG_SPAN" ctype="x-span" equiv-text="<span class="badge text-bg-secondary ms-1">"/><x id="INTERPOLATION" equiv-text="length}}"/><x id="CLOSE_TAG_SPAN" ctype="x-span"/><x id="CLOSE_BLOCK_IF" equiv-text="}"/></source>
|
<source>Notes <x id="START_BLOCK_IF" equiv-text="@if (document()?.notes.length) {"/><x id="START_TAG_SPAN" ctype="x-span" equiv-text="<span class="badge text-bg-secondary ms-1">"/><x id="INTERPOLATION" equiv-text="length}}"/><x id="CLOSE_TAG_SPAN" ctype="x-span"/><x id="CLOSE_BLOCK_IF" equiv-text="}"/></source>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||||
<context context-type="linenumber">368,371</context>
|
<context context-type="linenumber">372,375</context>
|
||||||
</context-group>
|
</context-group>
|
||||||
</trans-unit>
|
</trans-unit>
|
||||||
<trans-unit id="186236568870281953" datatype="html">
|
<trans-unit id="186236568870281953" datatype="html">
|
||||||
<source>History</source>
|
<source>History</source>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||||
<context context-type="linenumber">379</context>
|
<context context-type="linenumber">383</context>
|
||||||
</context-group>
|
</context-group>
|
||||||
</trans-unit>
|
</trans-unit>
|
||||||
<trans-unit id="8236092845697214347" datatype="html">
|
<trans-unit id="8236092845697214347" datatype="html">
|
||||||
<source> Duplicates <x id="START_TAG_SPAN" ctype="x-span" equiv-text="<span class="badge text-bg-secondary ms-1">"/><x id="INTERPOLATION" equiv-text="cate_documents.length }}"/><x id="CLOSE_TAG_SPAN" ctype="x-span"/></source>
|
<source> Duplicates <x id="START_TAG_SPAN" ctype="x-span" equiv-text="<span class="badge text-bg-secondary ms-1">"/><x id="INTERPOLATION" equiv-text="cate_documents.length }}"/><x id="CLOSE_TAG_SPAN" ctype="x-span"/></source>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||||
<context context-type="linenumber">401,405</context>
|
<context context-type="linenumber">405,409</context>
|
||||||
</context-group>
|
</context-group>
|
||||||
</trans-unit>
|
</trans-unit>
|
||||||
<trans-unit id="6449374629822973702" datatype="html">
|
<trans-unit id="6449374629822973702" datatype="html">
|
||||||
<source>Duplicate documents detected:</source>
|
<source>Duplicate documents detected:</source>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||||
<context context-type="linenumber">407</context>
|
<context context-type="linenumber">411</context>
|
||||||
</context-group>
|
</context-group>
|
||||||
</trans-unit>
|
</trans-unit>
|
||||||
<trans-unit id="14058600336670816" datatype="html">
|
<trans-unit id="14058600336670816" datatype="html">
|
||||||
<source>In trash</source>
|
<source>In trash</source>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||||
<context context-type="linenumber">418</context>
|
<context context-type="linenumber">422</context>
|
||||||
</context-group>
|
</context-group>
|
||||||
</trans-unit>
|
</trans-unit>
|
||||||
<trans-unit id="5129524307369213584" datatype="html">
|
<trans-unit id="5129524307369213584" datatype="html">
|
||||||
<source>Save & next</source>
|
<source>Save & next</source>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||||
<context context-type="linenumber">447</context>
|
<context context-type="linenumber">451</context>
|
||||||
</context-group>
|
</context-group>
|
||||||
</trans-unit>
|
</trans-unit>
|
||||||
<trans-unit id="4910102545766233758" datatype="html">
|
<trans-unit id="4910102545766233758" datatype="html">
|
||||||
<source>Save & close</source>
|
<source>Save & close</source>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||||
<context context-type="linenumber">449</context>
|
<context context-type="linenumber">453</context>
|
||||||
</context-group>
|
</context-group>
|
||||||
</trans-unit>
|
</trans-unit>
|
||||||
<trans-unit id="3823219296477075982" datatype="html">
|
<trans-unit id="3823219296477075982" datatype="html">
|
||||||
<source>Discard</source>
|
<source>Discard</source>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||||
<context context-type="linenumber">451</context>
|
<context context-type="linenumber">455</context>
|
||||||
</context-group>
|
</context-group>
|
||||||
</trans-unit>
|
</trans-unit>
|
||||||
<trans-unit id="1309556917227148591" datatype="html">
|
<trans-unit id="1309556917227148591" datatype="html">
|
||||||
<source>Document loading...</source>
|
<source>Document loading...</source>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||||
<context context-type="linenumber">459</context>
|
<context context-type="linenumber">463</context>
|
||||||
</context-group>
|
</context-group>
|
||||||
</trans-unit>
|
</trans-unit>
|
||||||
<trans-unit id="8191371354890763172" datatype="html">
|
<trans-unit id="8191371354890763172" datatype="html">
|
||||||
<source>Enter Password</source>
|
<source>Enter Password</source>
|
||||||
<context-group purpose="location">
|
<context-group purpose="location">
|
||||||
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
<context context-type="sourcefile">src/app/components/document-detail/document-detail.component.html</context>
|
||||||
<context context-type="linenumber">513</context>
|
<context context-type="linenumber">517</context>
|
||||||
</context-group>
|
</context-group>
|
||||||
</trans-unit>
|
</trans-unit>
|
||||||
<trans-unit id="5758784066858623886" datatype="html">
|
<trans-unit id="5758784066858623886" datatype="html">
|
||||||
|
|||||||
@@ -21,7 +21,7 @@
|
|||||||
</div>
|
</div>
|
||||||
</pngx-page-header>
|
</pngx-page-header>
|
||||||
|
|
||||||
@if (!tasksService.completedFileTasks && tasksService.loading) {
|
@if (loading() && pagedTasks().length === 0) {
|
||||||
<div class="spinner-border spinner-border-sm fw-normal ms-2 me-auto" role="status"></div>
|
<div class="spinner-border spinner-border-sm fw-normal ms-2 me-auto" role="status"></div>
|
||||||
<div class="visually-hidden" i18n>Loading...</div>
|
<div class="visually-hidden" i18n>Loading...</div>
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -129,13 +129,25 @@ describe('PngxPdfViewerComponent', () => {
|
|||||||
;(component as any).applyScale()
|
;(component as any).applyScale()
|
||||||
expect(viewer.currentScaleValue).toBe(PdfZoomScale.PageFit)
|
expect(viewer.currentScaleValue).toBe(PdfZoomScale.PageFit)
|
||||||
expect(viewer.currentScale).toBe(2)
|
expect(viewer.currentScale).toBe(2)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('does not reapply scale for page-only changes', async () => {
|
||||||
|
await initComponent()
|
||||||
|
|
||||||
|
const pdf = (component as any).pdf as { numPages: number }
|
||||||
|
pdf.numPages = 3
|
||||||
|
const viewer = (component as any).pdfViewer as PDFViewer
|
||||||
|
viewer.setDocument(pdf)
|
||||||
const applyScaleSpy = jest.spyOn(component as any, 'applyScale')
|
const applyScaleSpy = jest.spyOn(component as any, 'applyScale')
|
||||||
component.page = 2
|
component.page = 2
|
||||||
;(component as any).lastViewerPage = 2
|
|
||||||
;(component as any).applyViewerState()
|
component.ngOnChanges({
|
||||||
|
page: new SimpleChange(1, 2, false),
|
||||||
|
})
|
||||||
|
|
||||||
|
expect(viewer.currentPageNumber).toBe(2)
|
||||||
expect((component as any).lastViewerPage).toBeUndefined()
|
expect((component as any).lastViewerPage).toBeUndefined()
|
||||||
expect(applyScaleSpy).toHaveBeenCalled()
|
expect(applyScaleSpy).not.toHaveBeenCalled()
|
||||||
})
|
})
|
||||||
|
|
||||||
it('does not reset the viewer when it is already on the requested page', async () => {
|
it('does not reset the viewer when it is already on the requested page', async () => {
|
||||||
|
|||||||
@@ -116,7 +116,10 @@ export class PngxPdfViewerComponent
|
|||||||
changes['zoomScale'] ||
|
changes['zoomScale'] ||
|
||||||
changes['rotation']
|
changes['rotation']
|
||||||
) {
|
) {
|
||||||
this.applyViewerState()
|
// Prevent loop with page / scale application see https://github.com/paperless-ngx/paperless-ngx/issues/13404
|
||||||
|
this.applyViewerState(
|
||||||
|
!!(changes['zoom'] || changes['zoomScale'] || changes['rotation'])
|
||||||
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
if (changes['searchQuery']) {
|
if (changes['searchQuery']) {
|
||||||
@@ -240,7 +243,7 @@ export class PngxPdfViewerComponent
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
private applyViewerState(): void {
|
private applyViewerState(applyScale = true): void {
|
||||||
if (!this.pdfViewer) {
|
if (!this.pdfViewer) {
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
@@ -264,7 +267,7 @@ export class PngxPdfViewerComponent
|
|||||||
if (this.page === this.lastViewerPage) {
|
if (this.page === this.lastViewerPage) {
|
||||||
this.lastViewerPage = undefined
|
this.lastViewerPage = undefined
|
||||||
}
|
}
|
||||||
if (hasPages) {
|
if (hasPages && applyScale) {
|
||||||
this.applyScale()
|
this.applyScale()
|
||||||
}
|
}
|
||||||
this.dispatchFindIfReady()
|
this.dispatchFindIfReady()
|
||||||
|
|||||||
@@ -113,8 +113,8 @@
|
|||||||
|
|
||||||
<form [formGroup]='documentForm' (ngSubmit)="save()">
|
<form [formGroup]='documentForm' (ngSubmit)="save()">
|
||||||
|
|
||||||
<div class="btn-toolbar mb-1 border-bottom">
|
<div class="btn-toolbar justify-content-end mb-1 pb-3 gap-2 row-gap-2 border-bottom">
|
||||||
<div class="btn-group pb-3">
|
<div class="btn-group me-auto">
|
||||||
<button type="button" class="btn btn-sm btn-outline-secondary" i18n-title title="Close" (click)="close()">
|
<button type="button" class="btn btn-sm btn-outline-secondary" i18n-title title="Close" (click)="close()">
|
||||||
<i-bs width="1.2em" height="1.2em" name="x"></i-bs>
|
<i-bs width="1.2em" height="1.2em" name="x"></i-bs>
|
||||||
</button>
|
</button>
|
||||||
@@ -127,32 +127,36 @@
|
|||||||
</div>
|
</div>
|
||||||
|
|
||||||
<ng-container *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.Document }">
|
<ng-container *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.Document }">
|
||||||
<div class="btn-group pb-3 ms-auto">
|
<div class="d-flex gap-2">
|
||||||
<pngx-suggestions-dropdown *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.Document }"
|
<div class="btn-group">
|
||||||
[disabled]="!userCanEdit || suggestionsLoading()"
|
<pngx-suggestions-dropdown *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.Document }"
|
||||||
[loading]="suggestionsLoading()"
|
[disabled]="!userCanEdit || suggestionsLoading()"
|
||||||
[suggestions]="suggestions()"
|
[loading]="suggestionsLoading()"
|
||||||
[aiEnabled]="aiEnabled"
|
[suggestions]="suggestions()"
|
||||||
(getSuggestions)="getSuggestions()"
|
[aiEnabled]="aiEnabled"
|
||||||
(addTag)="createTag($event)"
|
(getSuggestions)="getSuggestions()"
|
||||||
(addDocumentType)="createDocumentType($event)"
|
(addTag)="createTag($event)"
|
||||||
(addCorrespondent)="createCorrespondent($event)">
|
(addDocumentType)="createDocumentType($event)"
|
||||||
</pngx-suggestions-dropdown>
|
(addCorrespondent)="createCorrespondent($event)">
|
||||||
|
</pngx-suggestions-dropdown>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div class="btn-group">
|
||||||
|
<pngx-custom-fields-dropdown
|
||||||
|
*pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.CustomField }"
|
||||||
|
[documentId]="documentId()"
|
||||||
|
[disabled]="!userCanEdit"
|
||||||
|
[existingFields]="document()?.custom_fields"
|
||||||
|
(created)="refreshCustomFields()"
|
||||||
|
(added)="addField($event)">
|
||||||
|
</pngx-custom-fields-dropdown>
|
||||||
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<div class="btn-group pb-3 ms-2">
|
<div class="ps-3">
|
||||||
<pngx-custom-fields-dropdown
|
<ng-container *ngTemplateOutlet="saveButtons"></ng-container>
|
||||||
*pngxIfPermissions="{ action: PermissionAction.View, type: PermissionType.CustomField }"
|
|
||||||
[documentId]="documentId()"
|
|
||||||
[disabled]="!userCanEdit"
|
|
||||||
[existingFields]="document()?.custom_fields"
|
|
||||||
(created)="refreshCustomFields()"
|
|
||||||
(added)="addField($event)">
|
|
||||||
</pngx-custom-fields-dropdown>
|
|
||||||
</div>
|
</div>
|
||||||
</ng-container>
|
</ng-container>
|
||||||
|
|
||||||
<ng-container *ngTemplateOutlet="saveButtons"></ng-container>
|
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<ul ngbNav #nav="ngbNav" class="nav-underline flex-nowrap flex-md-wrap overflow-auto" (navChange)="onNavChange($event)" [activeId]="activeNavID()" (activeIdChange)="activeNavID.set($event)">
|
<ul ngbNav #nav="ngbNav" class="nav-underline flex-nowrap flex-md-wrap overflow-auto" (navChange)="onNavChange($event)" [activeId]="activeNavID()" (activeIdChange)="activeNavID.set($event)">
|
||||||
@@ -276,7 +280,7 @@
|
|||||||
}
|
}
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<div class="d-flex border-top pt-3">
|
<div class="d-flex justify-content-end border-top pt-3">
|
||||||
<ng-container *ngTemplateOutlet="saveButtons"></ng-container>
|
<ng-container *ngTemplateOutlet="saveButtons"></ng-container>
|
||||||
</div>
|
</div>
|
||||||
</ng-template>
|
</ng-template>
|
||||||
@@ -440,7 +444,7 @@
|
|||||||
</div>
|
</div>
|
||||||
|
|
||||||
<ng-template #saveButtons>
|
<ng-template #saveButtons>
|
||||||
<div class="btn-group pb-3 ms-4">
|
<div class="btn-group">
|
||||||
<ng-container *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.Document }">
|
<ng-container *pngxIfPermissions="{ action: PermissionAction.Change, type: PermissionType.Document }">
|
||||||
<button type="submit" class="order-3 btn btn-sm btn-primary" i18n [disabled]="!userCanEdit || networkActive() || (isDirty$ | async) !== true">Save</button>
|
<button type="submit" class="order-3 btn btn-sm btn-primary" i18n [disabled]="!userCanEdit || networkActive() || (isDirty$ | async) !== true">Save</button>
|
||||||
@if (hasNext()) {
|
@if (hasNext()) {
|
||||||
|
|||||||
@@ -2269,4 +2269,34 @@ describe('FilterEditorComponent', () => {
|
|||||||
component.itemSelected({ item: 'world', preventDefault: () => true })
|
component.itemSelected({ item: 'world', preventDefault: () => true })
|
||||||
expect(component.textFilter).toEqual('hello world ')
|
expect(component.textFilter).toEqual('hello world ')
|
||||||
})
|
})
|
||||||
|
|
||||||
|
it('should choose the active autocomplete item with Enter', () => {
|
||||||
|
component.textFilterTarget = 'fulltext-query'
|
||||||
|
jest
|
||||||
|
.spyOn(searchService, 'autocomplete')
|
||||||
|
.mockReturnValue(of(['hello', 'help']))
|
||||||
|
|
||||||
|
const input = component.textFilterInput.nativeElement as HTMLInputElement
|
||||||
|
input.value = 'he'
|
||||||
|
input.dispatchEvent(new Event('input'))
|
||||||
|
tick(250)
|
||||||
|
|
||||||
|
input.dispatchEvent(
|
||||||
|
new KeyboardEvent('keydown', {
|
||||||
|
key: 'ArrowDown',
|
||||||
|
bubbles: true,
|
||||||
|
cancelable: true,
|
||||||
|
})
|
||||||
|
)
|
||||||
|
input.dispatchEvent(
|
||||||
|
new KeyboardEvent('keydown', {
|
||||||
|
key: 'Enter',
|
||||||
|
bubbles: true,
|
||||||
|
cancelable: true,
|
||||||
|
})
|
||||||
|
)
|
||||||
|
fixture.detectChanges()
|
||||||
|
|
||||||
|
expect(component.textFilter).toEqual('help ')
|
||||||
|
})
|
||||||
})
|
})
|
||||||
|
|||||||
@@ -1314,6 +1314,10 @@ export class FilterEditorComponent
|
|||||||
|
|
||||||
textFilterKeydown(event: KeyboardEvent) {
|
textFilterKeydown(event: KeyboardEvent) {
|
||||||
if (event.key == 'Enter') {
|
if (event.key == 'Enter') {
|
||||||
|
if (event.defaultPrevented) {
|
||||||
|
// NgbTypeahead calls preventDefault, so use that to detect if the Enter key was for the dropdown
|
||||||
|
return
|
||||||
|
}
|
||||||
const filterString = (
|
const filterString = (
|
||||||
this.textFilterInput.nativeElement as HTMLInputElement
|
this.textFilterInput.nativeElement as HTMLInputElement
|
||||||
).value
|
).value
|
||||||
|
|||||||
@@ -50,13 +50,66 @@ describe('TasksService', () => {
|
|||||||
req.flush({ count: 0, results: [] })
|
req.flush({ count: 0, results: [] })
|
||||||
})
|
})
|
||||||
|
|
||||||
it('does not call tasks api endpoint on reload if already loading', () => {
|
it('cancels an in-progress reload when reloading again', () => {
|
||||||
tasksService.loading = true
|
|
||||||
tasksService.reload()
|
tasksService.reload()
|
||||||
httpTestingController.expectNone(
|
const staleReload = httpTestingController.expectOne(
|
||||||
(req: HttpRequest<unknown>) =>
|
(req: HttpRequest<unknown>) =>
|
||||||
req.url === `${environment.apiBaseUrl}tasks/`
|
req.url === `${environment.apiBaseUrl}tasks/`
|
||||||
)
|
)
|
||||||
|
tasksService.reload()
|
||||||
|
|
||||||
|
expect(staleReload.cancelled).toBe(true)
|
||||||
|
httpTestingController
|
||||||
|
.expectOne(
|
||||||
|
(req: HttpRequest<unknown>) =>
|
||||||
|
req.url === `${environment.apiBaseUrl}tasks/`
|
||||||
|
)
|
||||||
|
.flush({ count: 0, results: [] })
|
||||||
|
})
|
||||||
|
|
||||||
|
it('continues reloading after a reload request fails', () => {
|
||||||
|
tasksService.reload()
|
||||||
|
httpTestingController
|
||||||
|
.expectOne(
|
||||||
|
(req: HttpRequest<unknown>) =>
|
||||||
|
req.url === `${environment.apiBaseUrl}tasks/`
|
||||||
|
)
|
||||||
|
.flush('error', { status: 500, statusText: 'error' })
|
||||||
|
|
||||||
|
expect(tasksService.loading).toBe(false)
|
||||||
|
|
||||||
|
tasksService.reload()
|
||||||
|
httpTestingController
|
||||||
|
.expectOne(
|
||||||
|
(req: HttpRequest<unknown>) =>
|
||||||
|
req.url === `${environment.apiBaseUrl}tasks/`
|
||||||
|
)
|
||||||
|
.flush({ count: 0, results: [] })
|
||||||
|
})
|
||||||
|
|
||||||
|
it('reloads after dismissing a task while a reload is already in progress', () => {
|
||||||
|
tasksService.reload()
|
||||||
|
const staleReload = httpTestingController.expectOne(
|
||||||
|
(req: HttpRequest<unknown>) =>
|
||||||
|
req.url === `${environment.apiBaseUrl}tasks/` &&
|
||||||
|
req.params.get('acknowledged') === 'false'
|
||||||
|
)
|
||||||
|
|
||||||
|
tasksService.dismissTasks(new Set([1])).subscribe()
|
||||||
|
httpTestingController
|
||||||
|
.expectOne(`${environment.apiBaseUrl}tasks/acknowledge/`)
|
||||||
|
.flush([])
|
||||||
|
|
||||||
|
expect(staleReload.cancelled).toBe(true)
|
||||||
|
httpTestingController
|
||||||
|
.expectOne(
|
||||||
|
(req: HttpRequest<unknown>) =>
|
||||||
|
req.url === `${environment.apiBaseUrl}tasks/` &&
|
||||||
|
req.params.get('acknowledged') === 'false'
|
||||||
|
)
|
||||||
|
.flush({ count: 0, results: [] })
|
||||||
|
|
||||||
|
expect(tasksService.needsAttentionTasks).toHaveLength(0)
|
||||||
})
|
})
|
||||||
|
|
||||||
it('calls acknowledge_tasks api endpoint on dismiss and reloads', () => {
|
it('calls acknowledge_tasks api endpoint on dismiss and reloads', () => {
|
||||||
@@ -101,79 +154,6 @@ describe('TasksService', () => {
|
|||||||
.flush({ count: 0, results: [] })
|
.flush({ count: 0, results: [] })
|
||||||
})
|
})
|
||||||
|
|
||||||
it('groups mixed task types by status when reloading', () => {
|
|
||||||
expect(tasksService.total).toEqual(0)
|
|
||||||
const mockTasks = [
|
|
||||||
{
|
|
||||||
task_type: PaperlessTaskType.ConsumeFile,
|
|
||||||
trigger_source: PaperlessTaskTriggerSource.FolderConsume,
|
|
||||||
status: PaperlessTaskStatus.Success,
|
|
||||||
acknowledged: false,
|
|
||||||
task_id: '1234',
|
|
||||||
input_data: { filename: 'file1.pdf' },
|
|
||||||
date_created: new Date(),
|
|
||||||
related_document_ids: [],
|
|
||||||
},
|
|
||||||
{
|
|
||||||
task_type: PaperlessTaskType.SanityCheck,
|
|
||||||
trigger_source: PaperlessTaskTriggerSource.System,
|
|
||||||
status: PaperlessTaskStatus.Failure,
|
|
||||||
acknowledged: false,
|
|
||||||
task_id: '1235',
|
|
||||||
input_data: {},
|
|
||||||
date_created: new Date(),
|
|
||||||
related_document_ids: [],
|
|
||||||
},
|
|
||||||
{
|
|
||||||
task_type: PaperlessTaskType.MailFetch,
|
|
||||||
trigger_source: PaperlessTaskTriggerSource.Scheduled,
|
|
||||||
status: PaperlessTaskStatus.Pending,
|
|
||||||
acknowledged: false,
|
|
||||||
task_id: '1236',
|
|
||||||
input_data: {},
|
|
||||||
date_created: new Date(),
|
|
||||||
related_document_ids: [],
|
|
||||||
},
|
|
||||||
{
|
|
||||||
task_type: PaperlessTaskType.LlmIndex,
|
|
||||||
trigger_source: PaperlessTaskTriggerSource.WebUI,
|
|
||||||
status: PaperlessTaskStatus.Started,
|
|
||||||
acknowledged: false,
|
|
||||||
task_id: '1237',
|
|
||||||
input_data: {},
|
|
||||||
date_created: new Date(),
|
|
||||||
related_document_ids: [],
|
|
||||||
},
|
|
||||||
{
|
|
||||||
task_type: PaperlessTaskType.EmptyTrash,
|
|
||||||
trigger_source: PaperlessTaskTriggerSource.Manual,
|
|
||||||
status: PaperlessTaskStatus.Success,
|
|
||||||
acknowledged: false,
|
|
||||||
task_id: '1238',
|
|
||||||
input_data: {},
|
|
||||||
date_created: new Date(),
|
|
||||||
related_document_ids: [],
|
|
||||||
},
|
|
||||||
]
|
|
||||||
|
|
||||||
tasksService.reload()
|
|
||||||
|
|
||||||
const req = httpTestingController.expectOne(
|
|
||||||
(req: HttpRequest<unknown>) =>
|
|
||||||
req.url === `${environment.apiBaseUrl}tasks/` &&
|
|
||||||
req.params.get('acknowledged') === 'false' &&
|
|
||||||
req.params.get('page_size') === '1000'
|
|
||||||
)
|
|
||||||
|
|
||||||
req.flush({ count: mockTasks.length, results: mockTasks })
|
|
||||||
|
|
||||||
expect(tasksService.allFileTasks).toHaveLength(5)
|
|
||||||
expect(tasksService.completedFileTasks).toHaveLength(2)
|
|
||||||
expect(tasksService.failedFileTasks).toHaveLength(1)
|
|
||||||
expect(tasksService.queuedFileTasks).toHaveLength(1)
|
|
||||||
expect(tasksService.startedFileTasks).toHaveLength(1)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('includes revoked tasks in needs attention', () => {
|
it('includes revoked tasks in needs attention', () => {
|
||||||
const mockTasks = [
|
const mockTasks = [
|
||||||
{
|
{
|
||||||
|
|||||||
@@ -1,7 +1,15 @@
|
|||||||
import { HttpClient } from '@angular/common/http'
|
import { HttpClient } from '@angular/common/http'
|
||||||
import { Injectable, inject, signal } from '@angular/core'
|
import { Injectable, inject, signal } from '@angular/core'
|
||||||
import { Observable, Subject } from 'rxjs'
|
import { EMPTY, Observable, Subject } from 'rxjs'
|
||||||
import { first, map, takeUntil, tap } from 'rxjs/operators'
|
import {
|
||||||
|
catchError,
|
||||||
|
finalize,
|
||||||
|
first,
|
||||||
|
map,
|
||||||
|
switchMap,
|
||||||
|
takeUntil,
|
||||||
|
tap,
|
||||||
|
} from 'rxjs/operators'
|
||||||
import {
|
import {
|
||||||
PaperlessTask,
|
PaperlessTask,
|
||||||
PaperlessTaskStatus,
|
PaperlessTaskStatus,
|
||||||
@@ -23,44 +31,40 @@ export class TasksService {
|
|||||||
|
|
||||||
public loading: boolean = false
|
public loading: boolean = false
|
||||||
|
|
||||||
private readonly fileTasks = signal<PaperlessTask[]>([])
|
private readonly tasks = signal<PaperlessTask[]>([])
|
||||||
|
private readonly reloadNotifier = new Subject<void>()
|
||||||
|
|
||||||
private unsubscribeNotifer: Subject<any> = new Subject()
|
private unsubscribeNotifer: Subject<any> = new Subject()
|
||||||
|
|
||||||
public get total(): number {
|
constructor() {
|
||||||
return this.fileTasks().length
|
this.reloadNotifier
|
||||||
}
|
.pipe(
|
||||||
|
switchMap(() => {
|
||||||
public get allFileTasks(): PaperlessTask[] {
|
this.loading = true
|
||||||
return this.fileTasks().slice(0)
|
return this.http
|
||||||
}
|
.get<Results<PaperlessTask>>(`${this.baseUrl}${this.endpoint}/`, {
|
||||||
|
params: {
|
||||||
public get queuedFileTasks(): PaperlessTask[] {
|
acknowledged: 'false',
|
||||||
return this.fileTasks().filter(
|
page_size: this.defaultReloadPageSize,
|
||||||
(t) => t.status === PaperlessTaskStatus.Pending
|
},
|
||||||
)
|
})
|
||||||
}
|
.pipe(
|
||||||
|
map((response) => response.results),
|
||||||
public get startedFileTasks(): PaperlessTask[] {
|
takeUntil(this.unsubscribeNotifer),
|
||||||
return this.fileTasks().filter(
|
catchError(() => EMPTY),
|
||||||
(t) => t.status === PaperlessTaskStatus.Started
|
finalize(() => {
|
||||||
)
|
this.loading = false
|
||||||
}
|
})
|
||||||
|
)
|
||||||
public get completedFileTasks(): PaperlessTask[] {
|
})
|
||||||
return this.fileTasks().filter(
|
)
|
||||||
(t) => t.status === PaperlessTaskStatus.Success
|
.subscribe((tasks) => {
|
||||||
)
|
this.tasks.set(tasks)
|
||||||
}
|
})
|
||||||
|
|
||||||
public get failedFileTasks(): PaperlessTask[] {
|
|
||||||
return this.fileTasks().filter(
|
|
||||||
(t) => t.status === PaperlessTaskStatus.Failure
|
|
||||||
)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
public get needsAttentionTasks(): PaperlessTask[] {
|
public get needsAttentionTasks(): PaperlessTask[] {
|
||||||
return this.fileTasks().filter((t) =>
|
return this.tasks().filter((t) =>
|
||||||
[PaperlessTaskStatus.Failure, PaperlessTaskStatus.Revoked].includes(
|
[PaperlessTaskStatus.Failure, PaperlessTaskStatus.Revoked].includes(
|
||||||
t.status
|
t.status
|
||||||
)
|
)
|
||||||
@@ -68,22 +72,7 @@ export class TasksService {
|
|||||||
}
|
}
|
||||||
|
|
||||||
public reload() {
|
public reload() {
|
||||||
if (this.loading) return
|
this.reloadNotifier.next()
|
||||||
this.loading = true
|
|
||||||
|
|
||||||
this.http
|
|
||||||
.get<Results<PaperlessTask>>(`${this.baseUrl}${this.endpoint}/`, {
|
|
||||||
params: {
|
|
||||||
acknowledged: 'false',
|
|
||||||
page_size: this.defaultReloadPageSize,
|
|
||||||
},
|
|
||||||
})
|
|
||||||
.pipe(map((r) => r.results))
|
|
||||||
.pipe(takeUntil(this.unsubscribeNotifer), first())
|
|
||||||
.subscribe((r) => {
|
|
||||||
this.fileTasks.set(r)
|
|
||||||
this.loading = false
|
|
||||||
})
|
|
||||||
}
|
}
|
||||||
|
|
||||||
public list(
|
public list(
|
||||||
|
|||||||
@@ -36,6 +36,9 @@ def send_email(
|
|||||||
|
|
||||||
TODO: re-evaluate this pending https://code.djangoproject.com/ticket/35581 / https://github.com/django/django/pull/18966
|
TODO: re-evaluate this pending https://code.djangoproject.com/ticket/35581 / https://github.com/django/django/pull/18966
|
||||||
"""
|
"""
|
||||||
|
if "\r" in subject or "\n" in subject:
|
||||||
|
subject = " ".join(line.strip(" \t") for line in subject.splitlines())
|
||||||
|
|
||||||
email = EmailMessage(
|
email = EmailMessage(
|
||||||
subject=subject,
|
subject=subject,
|
||||||
body=body,
|
body=body,
|
||||||
|
|||||||
@@ -386,10 +386,19 @@ class Command(CryptMixin, PaperlessCommand):
|
|||||||
raise DeserializationError(
|
raise DeserializationError(
|
||||||
f"{model.__name__} has no updatable fields; PK-only models are not supported by the importer",
|
f"{model.__name__} has no updatable fields; PK-only models are not supported by the importer",
|
||||||
)
|
)
|
||||||
|
# MySQL/MariaDB support upserts via ON DUPLICATE KEY UPDATE but,
|
||||||
|
# unlike PostgreSQL/SQLite, cannot target a specific unique field
|
||||||
|
# for the conflict -- passing unique_fields there raises
|
||||||
|
# NotSupportedError.
|
||||||
|
unique_fields = (
|
||||||
|
[model._meta.pk.attname]
|
||||||
|
if connection.features.supports_update_conflicts_with_target
|
||||||
|
else None
|
||||||
|
)
|
||||||
model.objects.bulk_create( # type: ignore[attr-defined]
|
model.objects.bulk_create( # type: ignore[attr-defined]
|
||||||
instances,
|
instances,
|
||||||
update_conflicts=True,
|
update_conflicts=True,
|
||||||
unique_fields=[model._meta.pk.attname],
|
unique_fields=unique_fields,
|
||||||
update_fields=update_fields,
|
update_fields=update_fields,
|
||||||
)
|
)
|
||||||
loaded_models.add(model)
|
loaded_models.add(model)
|
||||||
|
|||||||
@@ -43,8 +43,16 @@ def _fmt(dt: datetime) -> str:
|
|||||||
|
|
||||||
|
|
||||||
def _iso_range(lo: datetime, hi: datetime) -> str:
|
def _iso_range(lo: datetime, hi: datetime) -> str:
|
||||||
"""Format a [lo TO hi] range string in ISO 8601 for Tantivy query syntax."""
|
"""
|
||||||
return f"[{_fmt(lo)} TO {_fmt(hi)}]"
|
Format a half-open ``[lo TO hi)`` range in ISO 8601 for Tantivy query syntax.
|
||||||
|
|
||||||
|
``hi`` is always the exclusive ceiling of a computed period (the start of
|
||||||
|
the *next* day/week/month/quarter/year), so the closing bracket must be
|
||||||
|
the Tantivy exclusive-range brace ``}`` rather than ``]`` — otherwise the
|
||||||
|
first instant of the following period (e.g. the 1st of next month) is
|
||||||
|
incorrectly included in the match.
|
||||||
|
"""
|
||||||
|
return f"[{_fmt(lo)} TO {_fmt(hi)}}}"
|
||||||
|
|
||||||
|
|
||||||
def _quarter_start(d: date) -> date:
|
def _quarter_start(d: date) -> date:
|
||||||
|
|||||||
@@ -566,6 +566,17 @@ def translate_range(field: str, lo: str, hi: str, tz: tzinfo) -> str:
|
|||||||
lo_pair, hi_pair = hi_pair, lo_pair
|
lo_pair, hi_pair = hi_pair, lo_pair
|
||||||
|
|
||||||
lo_iso = _fmt(lo_pair[0]) if lo_pair is not None else OPEN_LO
|
lo_iso = _fmt(lo_pair[0]) if lo_pair is not None else OPEN_LO
|
||||||
hi_iso = _fmt(hi_pair[1]) if hi_pair is not None else OPEN_HI
|
|
||||||
|
|
||||||
return f"{field}:[{lo_iso} TO {hi_iso}]"
|
# A bound resolves to (floor, ceil) where floor == ceil for an exact instant
|
||||||
|
# (a full ISO datetime, "now", or a "+/-N unit" offset) and floor != ceil for
|
||||||
|
# a coarser period token (year/month/day precision). Only the latter needs a
|
||||||
|
# half-open close: its ceil is the start of the *next* period and must be
|
||||||
|
# excluded, or that instant (e.g. the 1st of next month) wrongly matches.
|
||||||
|
if hi_pair is not None:
|
||||||
|
hi_iso = _fmt(hi_pair[1])
|
||||||
|
hi_close = "]" if hi_pair[0] == hi_pair[1] else "}"
|
||||||
|
else:
|
||||||
|
hi_iso = OPEN_HI
|
||||||
|
hi_close = "]"
|
||||||
|
|
||||||
|
return f"{field}:[{lo_iso} TO {hi_iso}{hi_close}"
|
||||||
|
|||||||
@@ -1134,6 +1134,11 @@ def before_task_publish_handler(
|
|||||||
return
|
return
|
||||||
|
|
||||||
try:
|
try:
|
||||||
|
# Close stale connections without disrupting a transaction publishing a task
|
||||||
|
for connection in connections.all(initialized_only=True):
|
||||||
|
if not connection.in_atomic_block:
|
||||||
|
connection.close_if_unusable_or_obsolete()
|
||||||
|
|
||||||
_, task_kwargs, _ = body
|
_, task_kwargs, _ = body
|
||||||
task_id = headers["id"]
|
task_id = headers["id"]
|
||||||
|
|
||||||
|
|||||||
@@ -32,7 +32,9 @@ AUCKLAND = ZoneInfo("Pacific/Auckland") # UTC+13 in southern-hemisphere summer
|
|||||||
|
|
||||||
|
|
||||||
def _range(result: str, field: str) -> tuple[str, str]:
|
def _range(result: str, field: str) -> tuple[str, str]:
|
||||||
m = re.search(rf"{field}:\[(.+?) TO (.+?)\]", result)
|
# Half-open period ranges close with "}" (exclusive); exact-instant ranges
|
||||||
|
# (full ISO datetimes, "now", relative offsets) close with "]" (inclusive).
|
||||||
|
m = re.search(rf"{field}:\[(.+?) TO (.+?)[\]}}]", result)
|
||||||
assert m, f"No range for {field!r} in: {result!r}"
|
assert m, f"No range for {field!r} in: {result!r}"
|
||||||
return m.group(1), m.group(2)
|
return m.group(1), m.group(2)
|
||||||
|
|
||||||
|
|||||||
@@ -214,27 +214,27 @@ class TestTranslateScalar:
|
|||||||
(
|
(
|
||||||
"created",
|
"created",
|
||||||
"2020",
|
"2020",
|
||||||
"created:[2020-01-01T00:00:00Z TO 2021-01-01T00:00:00Z]",
|
"created:[2020-01-01T00:00:00Z TO 2021-01-01T00:00:00Z}",
|
||||||
),
|
),
|
||||||
(
|
(
|
||||||
"created",
|
"created",
|
||||||
"202003",
|
"202003",
|
||||||
"created:[2020-03-01T00:00:00Z TO 2020-04-01T00:00:00Z]",
|
"created:[2020-03-01T00:00:00Z TO 2020-04-01T00:00:00Z}",
|
||||||
),
|
),
|
||||||
(
|
(
|
||||||
"created",
|
"created",
|
||||||
"20200115",
|
"20200115",
|
||||||
"created:[2020-01-15T00:00:00Z TO 2020-01-16T00:00:00Z]",
|
"created:[2020-01-15T00:00:00Z TO 2020-01-16T00:00:00Z}",
|
||||||
),
|
),
|
||||||
(
|
(
|
||||||
"created",
|
"created",
|
||||||
"2020-01-15",
|
"2020-01-15",
|
||||||
"created:[2020-01-15T00:00:00Z TO 2020-01-16T00:00:00Z]",
|
"created:[2020-01-15T00:00:00Z TO 2020-01-16T00:00:00Z}",
|
||||||
),
|
),
|
||||||
(
|
(
|
||||||
"created",
|
"created",
|
||||||
"2020-03",
|
"2020-03",
|
||||||
"created:[2020-03-01T00:00:00Z TO 2020-04-01T00:00:00Z]",
|
"created:[2020-03-01T00:00:00Z TO 2020-04-01T00:00:00Z}",
|
||||||
),
|
),
|
||||||
],
|
],
|
||||||
)
|
)
|
||||||
@@ -248,9 +248,9 @@ class TestTranslateScalar:
|
|||||||
assert exc_info.value.value == "202023"
|
assert exc_info.value.value == "202023"
|
||||||
|
|
||||||
def test_keyword_delegates(self) -> None:
|
def test_keyword_delegates(self) -> None:
|
||||||
# keyword path produces a range; just assert it is a created range
|
# keyword path produces a half-open range; just assert it is a created range
|
||||||
out = translate_scalar("created", "today", UTC)
|
out = translate_scalar("created", "today", UTC)
|
||||||
assert out.startswith("created:[") and out.endswith("]")
|
assert out.startswith("created:[") and out.endswith("}")
|
||||||
|
|
||||||
def test_14digit_compact_datetime(self) -> None:
|
def test_14digit_compact_datetime(self) -> None:
|
||||||
out = translate_scalar("created", "20240115120000", UTC)
|
out = translate_scalar("created", "20240115120000", UTC)
|
||||||
@@ -279,21 +279,21 @@ class TestTranslateRange:
|
|||||||
@pytest.mark.parametrize(
|
@pytest.mark.parametrize(
|
||||||
("lo", "hi", "expected"),
|
("lo", "hi", "expected"),
|
||||||
[
|
[
|
||||||
("2005", "2009", "created:[2005-01-01T00:00:00Z TO 2010-01-01T00:00:00Z]"),
|
("2005", "2009", "created:[2005-01-01T00:00:00Z TO 2010-01-01T00:00:00Z}"),
|
||||||
(
|
(
|
||||||
"202001",
|
"202001",
|
||||||
"202006",
|
"202006",
|
||||||
"created:[2020-01-01T00:00:00Z TO 2020-07-01T00:00:00Z]",
|
"created:[2020-01-01T00:00:00Z TO 2020-07-01T00:00:00Z}",
|
||||||
),
|
),
|
||||||
(
|
(
|
||||||
"20200101",
|
"20200101",
|
||||||
"20201231",
|
"20201231",
|
||||||
"created:[2020-01-01T00:00:00Z TO 2021-01-01T00:00:00Z]",
|
"created:[2020-01-01T00:00:00Z TO 2021-01-01T00:00:00Z}",
|
||||||
),
|
),
|
||||||
(
|
(
|
||||||
"2020-01-01",
|
"2020-01-01",
|
||||||
"2020-12-31",
|
"2020-12-31",
|
||||||
"created:[2020-01-01T00:00:00Z TO 2021-01-01T00:00:00Z]",
|
"created:[2020-01-01T00:00:00Z TO 2021-01-01T00:00:00Z}",
|
||||||
),
|
),
|
||||||
],
|
],
|
||||||
)
|
)
|
||||||
@@ -302,7 +302,7 @@ class TestTranslateRange:
|
|||||||
|
|
||||||
def test_reversed_swaps(self):
|
def test_reversed_swaps(self):
|
||||||
assert translate_range("created", "2009", "2005", UTC) == (
|
assert translate_range("created", "2009", "2005", UTC) == (
|
||||||
"created:[2005-01-01T00:00:00Z TO 2010-01-01T00:00:00Z]"
|
"created:[2005-01-01T00:00:00Z TO 2010-01-01T00:00:00Z}"
|
||||||
)
|
)
|
||||||
|
|
||||||
def test_open_upper(self):
|
def test_open_upper(self):
|
||||||
@@ -311,7 +311,7 @@ class TestTranslateRange:
|
|||||||
|
|
||||||
def test_open_lower(self):
|
def test_open_lower(self):
|
||||||
out = translate_range("created", "", "2020", UTC)
|
out = translate_range("created", "", "2020", UTC)
|
||||||
assert out == f"created:[{OPEN_LO} TO 2021-01-01T00:00:00Z]"
|
assert out == f"created:[{OPEN_LO} TO 2021-01-01T00:00:00Z}}"
|
||||||
|
|
||||||
def test_invalid_bound_raises(self):
|
def test_invalid_bound_raises(self):
|
||||||
with pytest.raises(InvalidDateQuery) as exc_info:
|
with pytest.raises(InvalidDateQuery) as exc_info:
|
||||||
@@ -334,16 +334,16 @@ class TestTranslateQuery:
|
|||||||
[
|
[
|
||||||
(
|
(
|
||||||
"created:2020",
|
"created:2020",
|
||||||
"created:[2020-01-01T00:00:00Z TO 2021-01-01T00:00:00Z]",
|
"created:[2020-01-01T00:00:00Z TO 2021-01-01T00:00:00Z}",
|
||||||
),
|
),
|
||||||
("tag:foo,bar", "tag:foo AND tag:bar"),
|
("tag:foo,bar", "tag:foo AND tag:bar"),
|
||||||
# 'type' is a user-facing alias rewritten to 'document_type' (the real schema field)
|
# 'type' is a user-facing alias rewritten to 'document_type' (the real schema field)
|
||||||
("tag:foo,type:bar", "tag:foo AND document_type:bar"),
|
("tag:foo,type:bar", "tag:foo AND document_type:bar"),
|
||||||
(
|
(
|
||||||
"created:[2020 TO 2021],added:[2022 TO 2023]",
|
"created:[2020 TO 2021],added:[2022 TO 2023]",
|
||||||
"created:[2020-01-01T00:00:00Z TO 2022-01-01T00:00:00Z]"
|
"created:[2020-01-01T00:00:00Z TO 2022-01-01T00:00:00Z}"
|
||||||
" AND "
|
" AND "
|
||||||
"added:[2022-01-01T00:00:00Z TO 2024-01-01T00:00:00Z]",
|
"added:[2022-01-01T00:00:00Z TO 2024-01-01T00:00:00Z}",
|
||||||
),
|
),
|
||||||
# correspondent is not multi-value: comma stays literal inside the value
|
# correspondent is not multi-value: comma stays literal inside the value
|
||||||
("correspondent:foo,bar", "correspondent:foo,bar"),
|
("correspondent:foo,bar", "correspondent:foo,bar"),
|
||||||
@@ -506,7 +506,7 @@ class TestOperatorNormalization:
|
|||||||
def test_date_range_preserved(self) -> None:
|
def test_date_range_preserved(self) -> None:
|
||||||
out = translate_query("created:[2020 TO 2021]", UTC)
|
out = translate_query("created:[2020 TO 2021]", UTC)
|
||||||
# Must not corrupt the ISO range
|
# Must not corrupt the ISO range
|
||||||
assert out == "created:[2020-01-01T00:00:00Z TO 2022-01-01T00:00:00Z]"
|
assert out == "created:[2020-01-01T00:00:00Z TO 2022-01-01T00:00:00Z}"
|
||||||
|
|
||||||
def test_date_scalar_with_or(self) -> None:
|
def test_date_scalar_with_or(self) -> None:
|
||||||
out = translate_query("created:2020 OR foo", UTC)
|
out = translate_query("created:2020 OR foo", UTC)
|
||||||
@@ -581,42 +581,42 @@ class TestKeywordDateResolution:
|
|||||||
[
|
[
|
||||||
pytest.param(
|
pytest.param(
|
||||||
"today",
|
"today",
|
||||||
"created:[2026-03-28T00:00:00Z TO 2026-03-29T00:00:00Z]",
|
"created:[2026-03-28T00:00:00Z TO 2026-03-29T00:00:00Z}",
|
||||||
id="today",
|
id="today",
|
||||||
),
|
),
|
||||||
pytest.param(
|
pytest.param(
|
||||||
"yesterday",
|
"yesterday",
|
||||||
"created:[2026-03-27T00:00:00Z TO 2026-03-28T00:00:00Z]",
|
"created:[2026-03-27T00:00:00Z TO 2026-03-28T00:00:00Z}",
|
||||||
id="yesterday",
|
id="yesterday",
|
||||||
),
|
),
|
||||||
pytest.param(
|
pytest.param(
|
||||||
"previous week",
|
"previous week",
|
||||||
"created:[2026-03-16T00:00:00Z TO 2026-03-23T00:00:00Z]",
|
"created:[2026-03-16T00:00:00Z TO 2026-03-23T00:00:00Z}",
|
||||||
id="previous-week",
|
id="previous-week",
|
||||||
),
|
),
|
||||||
pytest.param(
|
pytest.param(
|
||||||
"this month",
|
"this month",
|
||||||
"created:[2026-03-01T00:00:00Z TO 2026-04-01T00:00:00Z]",
|
"created:[2026-03-01T00:00:00Z TO 2026-04-01T00:00:00Z}",
|
||||||
id="this-month",
|
id="this-month",
|
||||||
),
|
),
|
||||||
pytest.param(
|
pytest.param(
|
||||||
"previous month",
|
"previous month",
|
||||||
"created:[2026-02-01T00:00:00Z TO 2026-03-01T00:00:00Z]",
|
"created:[2026-02-01T00:00:00Z TO 2026-03-01T00:00:00Z}",
|
||||||
id="previous-month",
|
id="previous-month",
|
||||||
),
|
),
|
||||||
pytest.param(
|
pytest.param(
|
||||||
"this year",
|
"this year",
|
||||||
"created:[2026-01-01T00:00:00Z TO 2027-01-01T00:00:00Z]",
|
"created:[2026-01-01T00:00:00Z TO 2027-01-01T00:00:00Z}",
|
||||||
id="this-year",
|
id="this-year",
|
||||||
),
|
),
|
||||||
pytest.param(
|
pytest.param(
|
||||||
"previous year",
|
"previous year",
|
||||||
"created:[2025-01-01T00:00:00Z TO 2026-01-01T00:00:00Z]",
|
"created:[2025-01-01T00:00:00Z TO 2026-01-01T00:00:00Z}",
|
||||||
id="previous-year",
|
id="previous-year",
|
||||||
),
|
),
|
||||||
pytest.param(
|
pytest.param(
|
||||||
"previous quarter",
|
"previous quarter",
|
||||||
"created:[2025-10-01T00:00:00Z TO 2026-01-01T00:00:00Z]",
|
"created:[2025-10-01T00:00:00Z TO 2026-01-01T00:00:00Z}",
|
||||||
id="previous-quarter",
|
id="previous-quarter",
|
||||||
),
|
),
|
||||||
],
|
],
|
||||||
@@ -637,42 +637,42 @@ class TestKeywordDateResolution:
|
|||||||
[
|
[
|
||||||
pytest.param(
|
pytest.param(
|
||||||
"today",
|
"today",
|
||||||
"added:[2026-03-27T15:00:00Z TO 2026-03-28T15:00:00Z]",
|
"added:[2026-03-27T15:00:00Z TO 2026-03-28T15:00:00Z}",
|
||||||
id="today",
|
id="today",
|
||||||
),
|
),
|
||||||
pytest.param(
|
pytest.param(
|
||||||
"yesterday",
|
"yesterday",
|
||||||
"added:[2026-03-26T15:00:00Z TO 2026-03-27T15:00:00Z]",
|
"added:[2026-03-26T15:00:00Z TO 2026-03-27T15:00:00Z}",
|
||||||
id="yesterday",
|
id="yesterday",
|
||||||
),
|
),
|
||||||
pytest.param(
|
pytest.param(
|
||||||
"previous week",
|
"previous week",
|
||||||
"added:[2026-03-15T15:00:00Z TO 2026-03-22T15:00:00Z]",
|
"added:[2026-03-15T15:00:00Z TO 2026-03-22T15:00:00Z}",
|
||||||
id="previous-week",
|
id="previous-week",
|
||||||
),
|
),
|
||||||
pytest.param(
|
pytest.param(
|
||||||
"this month",
|
"this month",
|
||||||
"added:[2026-02-28T15:00:00Z TO 2026-03-31T15:00:00Z]",
|
"added:[2026-02-28T15:00:00Z TO 2026-03-31T15:00:00Z}",
|
||||||
id="this-month",
|
id="this-month",
|
||||||
),
|
),
|
||||||
pytest.param(
|
pytest.param(
|
||||||
"previous month",
|
"previous month",
|
||||||
"added:[2026-01-31T15:00:00Z TO 2026-02-28T15:00:00Z]",
|
"added:[2026-01-31T15:00:00Z TO 2026-02-28T15:00:00Z}",
|
||||||
id="previous-month",
|
id="previous-month",
|
||||||
),
|
),
|
||||||
pytest.param(
|
pytest.param(
|
||||||
"this year",
|
"this year",
|
||||||
"added:[2025-12-31T15:00:00Z TO 2026-12-31T15:00:00Z]",
|
"added:[2025-12-31T15:00:00Z TO 2026-12-31T15:00:00Z}",
|
||||||
id="this-year",
|
id="this-year",
|
||||||
),
|
),
|
||||||
pytest.param(
|
pytest.param(
|
||||||
"previous year",
|
"previous year",
|
||||||
"added:[2024-12-31T15:00:00Z TO 2025-12-31T15:00:00Z]",
|
"added:[2024-12-31T15:00:00Z TO 2025-12-31T15:00:00Z}",
|
||||||
id="previous-year",
|
id="previous-year",
|
||||||
),
|
),
|
||||||
pytest.param(
|
pytest.param(
|
||||||
"previous quarter",
|
"previous quarter",
|
||||||
"added:[2025-09-30T15:00:00Z TO 2025-12-31T15:00:00Z]",
|
"added:[2025-09-30T15:00:00Z TO 2025-12-31T15:00:00Z}",
|
||||||
id="previous-quarter",
|
id="previous-quarter",
|
||||||
),
|
),
|
||||||
],
|
],
|
||||||
@@ -719,7 +719,7 @@ class TestISODatetimeBounds:
|
|||||||
def test_translate_query_text_before_comma_separated_date_clause(self) -> None:
|
def test_translate_query_text_before_comma_separated_date_clause(self) -> None:
|
||||||
result = translate_query("schäfersee,created:previous year", UTC)
|
result = translate_query("schäfersee,created:previous year", UTC)
|
||||||
assert result == (
|
assert result == (
|
||||||
"schäfersee AND created:[2025-01-01T00:00:00Z TO 2026-01-01T00:00:00Z]"
|
"schäfersee AND created:[2025-01-01T00:00:00Z TO 2026-01-01T00:00:00Z}"
|
||||||
)
|
)
|
||||||
|
|
||||||
def test_invalid_iso_datetime_raises(self) -> None:
|
def test_invalid_iso_datetime_raises(self) -> None:
|
||||||
|
|||||||
@@ -75,7 +75,7 @@ class TestEmail(DirectoriesMixin, SampleDirMixin, APITestCase):
|
|||||||
{
|
{
|
||||||
"documents": [self.doc1.pk, self.doc2.pk],
|
"documents": [self.doc1.pk, self.doc2.pk],
|
||||||
"addresses": "hello@paperless-ngx.com,test@example.com",
|
"addresses": "hello@paperless-ngx.com,test@example.com",
|
||||||
"subject": "Bulk email test",
|
"subject": "Bulk email\n test",
|
||||||
"message": "Here are your documents",
|
"message": "Here are your documents",
|
||||||
},
|
},
|
||||||
),
|
),
|
||||||
|
|||||||
@@ -720,6 +720,48 @@ class TestDocumentSearchApi(DirectoriesMixin, APITestCase):
|
|||||||
self.assertEqual(results[0]["id"], 3)
|
self.assertEqual(results[0]["id"], 3)
|
||||||
self.assertEqual(results[0]["title"], "bank statement 3")
|
self.assertEqual(results[0]["title"], "bank statement 3")
|
||||||
|
|
||||||
|
def test_search_added_previous_month_excludes_next_period_start(self) -> None:
|
||||||
|
"""
|
||||||
|
GIVEN:
|
||||||
|
- One document added at the last instant of last month
|
||||||
|
- One document added exactly at the first instant of this month
|
||||||
|
WHEN:
|
||||||
|
- Query for documents added in the previous month
|
||||||
|
THEN:
|
||||||
|
- Only the document from last month is returned; the document dated
|
||||||
|
exactly at the start of this month (the exclusive upper bound of
|
||||||
|
the range) is not
|
||||||
|
"""
|
||||||
|
d1 = DocumentFactory.create(
|
||||||
|
title="end of last month",
|
||||||
|
content="last instant of last month",
|
||||||
|
checksum="A",
|
||||||
|
pk=1,
|
||||||
|
added=timezone.make_aware(datetime.datetime(2024, 1, 31, 23, 59, 59)),
|
||||||
|
)
|
||||||
|
d2 = DocumentFactory.create(
|
||||||
|
title="start of this month",
|
||||||
|
content="first instant of this month",
|
||||||
|
checksum="B",
|
||||||
|
pk=2,
|
||||||
|
added=timezone.make_aware(datetime.datetime(2024, 2, 1, 0, 0, 0)),
|
||||||
|
)
|
||||||
|
|
||||||
|
backend = get_backend()
|
||||||
|
backend.add_or_update(d1)
|
||||||
|
backend.add_or_update(d2)
|
||||||
|
|
||||||
|
with time_machine.travel(
|
||||||
|
timezone.make_aware(datetime.datetime(2024, 2, 15, 12, 0, 0)),
|
||||||
|
tick=False,
|
||||||
|
):
|
||||||
|
response = self.client.get("/api/documents/?query=added:previous month")
|
||||||
|
results = response.data["results"]
|
||||||
|
|
||||||
|
self.assertEqual(len(results), 1)
|
||||||
|
self.assertEqual(results[0]["id"], 1)
|
||||||
|
self.assertEqual(results[0]["title"], "end of last month")
|
||||||
|
|
||||||
def test_search_added_invalid_date(self) -> None:
|
def test_search_added_invalid_date(self) -> None:
|
||||||
"""
|
"""
|
||||||
GIVEN:
|
GIVEN:
|
||||||
|
|||||||
@@ -56,6 +56,32 @@ def send_publish(
|
|||||||
|
|
||||||
@pytest.mark.django_db
|
@pytest.mark.django_db
|
||||||
class TestBeforeTaskPublishHandler:
|
class TestBeforeTaskPublishHandler:
|
||||||
|
@mock.patch("documents.signals.handlers.connections.all")
|
||||||
|
def test_closes_old_connections_outside_atomic_blocks(
|
||||||
|
self,
|
||||||
|
connections_all,
|
||||||
|
) -> None:
|
||||||
|
connection = mock.Mock(in_atomic_block=False)
|
||||||
|
connections_all.return_value = [connection]
|
||||||
|
|
||||||
|
task_id = send_publish("documents.tasks.train_classifier", (), {})
|
||||||
|
|
||||||
|
connection.close_if_unusable_or_obsolete.assert_called_once_with()
|
||||||
|
assert PaperlessTask.objects.filter(task_id=task_id).exists()
|
||||||
|
|
||||||
|
@mock.patch("documents.signals.handlers.connections.all")
|
||||||
|
def test_keeps_connections_open_inside_atomic_blocks(
|
||||||
|
self,
|
||||||
|
connections_all,
|
||||||
|
) -> None:
|
||||||
|
connection = mock.Mock(in_atomic_block=True)
|
||||||
|
connections_all.return_value = [connection]
|
||||||
|
|
||||||
|
task_id = send_publish("documents.tasks.train_classifier", (), {})
|
||||||
|
|
||||||
|
connection.close_if_unusable_or_obsolete.assert_not_called()
|
||||||
|
assert PaperlessTask.objects.filter(task_id=task_id).exists()
|
||||||
|
|
||||||
def test_creates_task_for_consume_file(
|
def test_creates_task_for_consume_file(
|
||||||
self,
|
self,
|
||||||
consume_input_doc,
|
consume_input_doc,
|
||||||
|
|||||||
@@ -150,6 +150,7 @@ INSTALLED_APPS = [
|
|||||||
"drf_spectacular",
|
"drf_spectacular",
|
||||||
"drf_spectacular_sidecar",
|
"drf_spectacular_sidecar",
|
||||||
"treenode",
|
"treenode",
|
||||||
|
"paperless_benchmark.apps.PaperlessBenchmarkConfig",
|
||||||
*env_apps,
|
*env_apps,
|
||||||
]
|
]
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,8 @@
|
|||||||
|
from django.apps import AppConfig
|
||||||
|
from django.utils.translation import gettext_lazy as _
|
||||||
|
|
||||||
|
|
||||||
|
class PaperlessBenchmarkConfig(AppConfig):
|
||||||
|
name = "paperless_benchmark"
|
||||||
|
|
||||||
|
verbose_name = _("Paperless benchmark")
|
||||||
@@ -0,0 +1,136 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import TYPE_CHECKING
|
||||||
|
|
||||||
|
from django.db import connection
|
||||||
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
from django.db.models import QuerySet
|
||||||
|
|
||||||
|
|
||||||
|
def _reset_table_names() -> list[str]:
|
||||||
|
from guardian.models import GroupObjectPermission
|
||||||
|
from guardian.models import UserObjectPermission
|
||||||
|
|
||||||
|
from documents.models import Correspondent
|
||||||
|
from documents.models import Document
|
||||||
|
from documents.models import DocumentType
|
||||||
|
from documents.models import StoragePath
|
||||||
|
from documents.models import Tag
|
||||||
|
|
||||||
|
return [
|
||||||
|
Document.tags.through._meta.db_table,
|
||||||
|
Document._meta.db_table,
|
||||||
|
Tag._meta.db_table,
|
||||||
|
Correspondent._meta.db_table,
|
||||||
|
DocumentType._meta.db_table,
|
||||||
|
StoragePath._meta.db_table,
|
||||||
|
UserObjectPermission._meta.db_table,
|
||||||
|
GroupObjectPermission._meta.db_table,
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def _delete_all_users_and_groups() -> None:
|
||||||
|
# ASSUMPTION: this tool assumes a disposable benchmark database, never
|
||||||
|
# point it at a real install. This deletes EVERY user and group in the
|
||||||
|
# database (not just benchmark-created ones) -- there is no way to
|
||||||
|
# distinguish "real" users from seeded ones, so this is only safe against
|
||||||
|
# a database that exists solely to run this benchmarking tool. The
|
||||||
|
# `benchmark seed --reset` CLI path requires an explicit
|
||||||
|
# `--yes-i-know-this-wipes-the-database` flag before reaching here; do
|
||||||
|
# not remove that guard.
|
||||||
|
from django.contrib.auth import get_user_model
|
||||||
|
from django.contrib.auth.models import Group
|
||||||
|
|
||||||
|
get_user_model().objects.all().delete()
|
||||||
|
Group.objects.all().delete()
|
||||||
|
|
||||||
|
|
||||||
|
def _reset_postgresql() -> None:
|
||||||
|
tables = _reset_table_names()
|
||||||
|
with connection.cursor() as cursor:
|
||||||
|
cursor.execute(f"TRUNCATE TABLE {', '.join(tables)} RESTART IDENTITY CASCADE;")
|
||||||
|
_delete_all_users_and_groups()
|
||||||
|
|
||||||
|
|
||||||
|
def _reset_mariadb() -> None:
|
||||||
|
# MariaDB's TRUNCATE has no CASCADE clause and refuses to truncate a
|
||||||
|
# table referenced by a foreign key while checks are enabled, so
|
||||||
|
# checks are disabled for the duration of the reset.
|
||||||
|
tables = _reset_table_names()
|
||||||
|
with connection.cursor() as cursor:
|
||||||
|
cursor.execute("SET FOREIGN_KEY_CHECKS = 0;")
|
||||||
|
try:
|
||||||
|
for table in tables:
|
||||||
|
cursor.execute(f"TRUNCATE TABLE {table};")
|
||||||
|
finally:
|
||||||
|
cursor.execute("SET FOREIGN_KEY_CHECKS = 1;")
|
||||||
|
_delete_all_users_and_groups()
|
||||||
|
|
||||||
|
|
||||||
|
def _reset_sqlite() -> None:
|
||||||
|
from documents.models import Correspondent
|
||||||
|
from documents.models import Document
|
||||||
|
from documents.models import DocumentType
|
||||||
|
from documents.models import StoragePath
|
||||||
|
from documents.models import Tag
|
||||||
|
|
||||||
|
Document.global_objects.all().delete()
|
||||||
|
Tag.objects.all().delete()
|
||||||
|
Correspondent.objects.all().delete()
|
||||||
|
DocumentType.objects.all().delete()
|
||||||
|
StoragePath.objects.all().delete()
|
||||||
|
_delete_all_users_and_groups()
|
||||||
|
|
||||||
|
|
||||||
|
def reset_benchmark_data() -> None:
|
||||||
|
"""
|
||||||
|
Remove all previously-seeded benchmark data (documents, tags,
|
||||||
|
correspondents, document types, storage paths, guardian permission
|
||||||
|
rows, users, and groups) so a fresh `benchmark seed` run starts
|
||||||
|
from an empty slate. Dispatches per-backend because TRUNCATE syntax
|
||||||
|
and cascade behavior differ across the 3 supported databases.
|
||||||
|
"""
|
||||||
|
if connection.vendor == "postgresql":
|
||||||
|
_reset_postgresql()
|
||||||
|
elif connection.vendor == "mysql":
|
||||||
|
# MariaDB also reports vendor == "mysql" under Django's mysql backend.
|
||||||
|
_reset_mariadb()
|
||||||
|
else:
|
||||||
|
_reset_sqlite()
|
||||||
|
|
||||||
|
|
||||||
|
def capture_explain(queryset: QuerySet) -> str:
|
||||||
|
"""
|
||||||
|
Return the query plan for `queryset` using the current backend's
|
||||||
|
explain facility. PostgreSQL supports `EXPLAIN ANALYZE {sql}` (real
|
||||||
|
execution stats). MariaDB does NOT accept that syntax -- verified
|
||||||
|
against a real MariaDB 12.3 container: `EXPLAIN ANALYZE {sql}` raises a
|
||||||
|
1064 syntax error, while MariaDB's own `ANALYZE {sql}` form (no
|
||||||
|
`EXPLAIN` keyword) works and returns real per-row execution stats
|
||||||
|
(`r_rows`, `r_filtered`, etc. columns) -- this is MariaDB's
|
||||||
|
EXPLAIN-ANALYZE-equivalent, distinct from MySQL 8.0.18+'s
|
||||||
|
`EXPLAIN ANALYZE` syntax, which MariaDB does not implement. SQLite only
|
||||||
|
supports EXPLAIN QUERY PLAN (the chosen plan, not real timing/row
|
||||||
|
counts) -- that output is clearly labeled rather than silently looking
|
||||||
|
equivalent to the other two backends' output.
|
||||||
|
"""
|
||||||
|
sql, params = queryset.query.sql_with_params()
|
||||||
|
with connection.cursor() as cursor:
|
||||||
|
if connection.vendor == "postgresql":
|
||||||
|
cursor.execute(f"EXPLAIN ANALYZE {sql}", params)
|
||||||
|
return "\n".join(str(row[0]) for row in cursor.fetchall())
|
||||||
|
if connection.vendor == "mysql":
|
||||||
|
# MariaDB also reports vendor == "mysql" under Django's mysql
|
||||||
|
# backend. Unlike MySQL 8.0.18+, MariaDB has no `EXPLAIN
|
||||||
|
# ANALYZE` syntax -- its equivalent is `ANALYZE <statement>`.
|
||||||
|
cursor.execute(f"ANALYZE {sql}", params)
|
||||||
|
columns = [c[0] for c in cursor.description]
|
||||||
|
header = " | ".join(columns)
|
||||||
|
rows = "\n".join(
|
||||||
|
" | ".join(str(c) for c in row) for row in cursor.fetchall()
|
||||||
|
)
|
||||||
|
return f"{header}\n{rows}"
|
||||||
|
cursor.execute(f"EXPLAIN QUERY PLAN {sql}", params)
|
||||||
|
rows = "\n".join(" | ".join(str(c) for c in row) for row in cursor.fetchall())
|
||||||
|
return f"(plan only -- no execution stats on SQLite)\n{rows}"
|
||||||
@@ -0,0 +1,81 @@
|
|||||||
|
# src/paperless_benchmark/endpoints.py
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import statistics
|
||||||
|
import time
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from typing import TYPE_CHECKING
|
||||||
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
from django.contrib.auth.models import User
|
||||||
|
from rest_framework.test import APIClient
|
||||||
|
|
||||||
|
ENDPOINTS: tuple[tuple[str, str], ...] = (
|
||||||
|
("documents_default", "/api/documents/"),
|
||||||
|
("documents_page50", "/api/documents/?page_size=50"),
|
||||||
|
("tags_all", "/api/tags/?page_size=100000"),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class EndpointTiming:
|
||||||
|
user_label: str
|
||||||
|
endpoint_name: str
|
||||||
|
query_count: int
|
||||||
|
min_ms: float
|
||||||
|
median_ms: float
|
||||||
|
max_ms: float
|
||||||
|
|
||||||
|
|
||||||
|
def _timed_requests(client: APIClient, url: str, n: int) -> list[float]:
|
||||||
|
times = []
|
||||||
|
for _ in range(n):
|
||||||
|
t0 = time.perf_counter()
|
||||||
|
resp = client.get(url)
|
||||||
|
t1 = time.perf_counter()
|
||||||
|
if resp.status_code != 200:
|
||||||
|
raise RuntimeError(
|
||||||
|
f"GET {url} -> {resp.status_code}: {resp.content[:300]!r}",
|
||||||
|
)
|
||||||
|
times.append(t1 - t0)
|
||||||
|
return times
|
||||||
|
|
||||||
|
|
||||||
|
def _query_count(client: APIClient, url: str) -> int:
|
||||||
|
from django.db import connection
|
||||||
|
from django.test.utils import CaptureQueriesContext
|
||||||
|
|
||||||
|
with CaptureQueriesContext(connection) as ctx:
|
||||||
|
resp = client.get(url)
|
||||||
|
if resp.status_code != 200:
|
||||||
|
raise RuntimeError(f"GET {url} -> {resp.status_code}: {resp.content[:300]!r}")
|
||||||
|
return len(ctx.captured_queries)
|
||||||
|
|
||||||
|
|
||||||
|
def run_endpoint_benchmarks(
|
||||||
|
*,
|
||||||
|
perf_target: User,
|
||||||
|
perf_admin: User,
|
||||||
|
repeat: int,
|
||||||
|
) -> list[EndpointTiming]:
|
||||||
|
from rest_framework.test import APIClient
|
||||||
|
|
||||||
|
results: list[EndpointTiming] = []
|
||||||
|
for user_label, user in (("target", perf_target), ("admin", perf_admin)):
|
||||||
|
client = APIClient()
|
||||||
|
client.force_authenticate(user=user)
|
||||||
|
for name, url in ENDPOINTS:
|
||||||
|
client.get(url) # warm-up request, not counted
|
||||||
|
qcount = _query_count(client, url)
|
||||||
|
times_ms = [t * 1000 for t in _timed_requests(client, url, repeat)]
|
||||||
|
results.append(
|
||||||
|
EndpointTiming(
|
||||||
|
user_label=user_label,
|
||||||
|
endpoint_name=name,
|
||||||
|
query_count=qcount,
|
||||||
|
min_ms=min(times_ms),
|
||||||
|
median_ms=statistics.median(times_ms),
|
||||||
|
max_ms=max(times_ms),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
return results
|
||||||
@@ -0,0 +1,56 @@
|
|||||||
|
# src/paperless_benchmark/harness.py
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import time
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from typing import TYPE_CHECKING
|
||||||
|
from typing import Generic
|
||||||
|
from typing import TypeVar
|
||||||
|
|
||||||
|
from django.db import connection
|
||||||
|
from django.test.utils import CaptureQueriesContext
|
||||||
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
from collections.abc import Callable
|
||||||
|
|
||||||
|
T = TypeVar("T")
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class ProfileResult(Generic[T]):
|
||||||
|
best_seconds: float
|
||||||
|
all_seconds: tuple[float, ...]
|
||||||
|
query_count: int
|
||||||
|
result: T
|
||||||
|
|
||||||
|
|
||||||
|
def run_profile(fn: Callable[[], T], *, repeat: int = 5) -> ProfileResult[T]:
|
||||||
|
"""
|
||||||
|
Call `fn` `repeat` times, capturing wall-clock time for every call and
|
||||||
|
the SQL query count for the final call. Returns the best (minimum)
|
||||||
|
time across all repeats, since the first call(s) can be skewed by
|
||||||
|
connection warm-up or cold caches.
|
||||||
|
"""
|
||||||
|
if repeat < 1:
|
||||||
|
raise ValueError("repeat must be >= 1")
|
||||||
|
|
||||||
|
all_seconds: list[float] = []
|
||||||
|
result: T | None = None
|
||||||
|
query_count = 0
|
||||||
|
for i in range(repeat):
|
||||||
|
with CaptureQueriesContext(connection) as ctx:
|
||||||
|
start = time.perf_counter()
|
||||||
|
result = fn()
|
||||||
|
all_seconds.append(time.perf_counter() - start)
|
||||||
|
if i == repeat - 1:
|
||||||
|
query_count = len(ctx.captured_queries)
|
||||||
|
# Purely a type-narrowing aid for the type checker: the `repeat < 1`
|
||||||
|
# guard above already turns the one case that could leave `result`
|
||||||
|
# unset into a clear ValueError, so this is unreachable in practice.
|
||||||
|
assert result is not None
|
||||||
|
return ProfileResult(
|
||||||
|
best_seconds=min(all_seconds),
|
||||||
|
all_seconds=tuple(all_seconds),
|
||||||
|
query_count=query_count,
|
||||||
|
result=result,
|
||||||
|
)
|
||||||
@@ -0,0 +1,222 @@
|
|||||||
|
# src/paperless_benchmark/management/commands/benchmark.py
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from django.core.management.base import BaseCommand
|
||||||
|
from django.core.management.base import CommandError
|
||||||
|
from django.core.management.base import CommandParser
|
||||||
|
|
||||||
|
|
||||||
|
class Command(BaseCommand):
|
||||||
|
help = "Seed, run, and profile paperless-ngx performance benchmarks."
|
||||||
|
|
||||||
|
def add_arguments(self, parser: CommandParser) -> None:
|
||||||
|
parser.add_argument(
|
||||||
|
"action",
|
||||||
|
choices=["seed", "run", "profile", "list-scenarios"],
|
||||||
|
help="Which benchmark action to perform.",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"scenario",
|
||||||
|
nargs="?",
|
||||||
|
default=None,
|
||||||
|
help="Scenario name (required for `profile`; see `list-scenarios`).",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--tier",
|
||||||
|
choices=["home", "medium", "large"],
|
||||||
|
default="medium",
|
||||||
|
help="Dataset scale tier for `seed` (default: medium).",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--reset",
|
||||||
|
action="store_true",
|
||||||
|
default=False,
|
||||||
|
help="For `seed`: truncate existing benchmark data first.",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--yes-i-know-this-wipes-the-database",
|
||||||
|
action="store_true",
|
||||||
|
default=False,
|
||||||
|
help=(
|
||||||
|
"Required alongside --reset: confirms you understand `seed "
|
||||||
|
"--reset` deletes ALL users, ALL groups, and ALL documents/"
|
||||||
|
"tags/correspondents/document types/storage paths in this "
|
||||||
|
"database, not just benchmark-created ones."
|
||||||
|
),
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--seed",
|
||||||
|
type=int,
|
||||||
|
default=42,
|
||||||
|
help="RNG seed for reproducible datasets (default: 42).",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--repeat",
|
||||||
|
type=int,
|
||||||
|
default=5,
|
||||||
|
help="Number of timed repetitions for `run`/`profile` (default: 5).",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--label",
|
||||||
|
default="baseline",
|
||||||
|
help="Free-text tag for a `run`, printed and recorded in history only.",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--explain",
|
||||||
|
action="store_true",
|
||||||
|
default=False,
|
||||||
|
help="For `profile`: also capture and print the query plan.",
|
||||||
|
)
|
||||||
|
|
||||||
|
def handle(self, *args: Any, **options: Any) -> None:
|
||||||
|
action = options["action"]
|
||||||
|
if action == "seed":
|
||||||
|
self._handle_seed(options)
|
||||||
|
elif action == "run":
|
||||||
|
self._handle_run(options)
|
||||||
|
elif action == "profile":
|
||||||
|
self._handle_profile(options)
|
||||||
|
else:
|
||||||
|
self._handle_list_scenarios()
|
||||||
|
|
||||||
|
def _handle_seed(self, options: dict[str, Any]) -> None:
|
||||||
|
from paperless_benchmark.db import reset_benchmark_data
|
||||||
|
from paperless_benchmark.seeding import seed_benchmark_dataset
|
||||||
|
|
||||||
|
if options["reset"]:
|
||||||
|
if not options["yes_i_know_this_wipes_the_database"]:
|
||||||
|
raise CommandError(
|
||||||
|
"--reset requires --yes-i-know-this-wipes-the-database. "
|
||||||
|
"This deletes ALL users, ALL groups, and ALL documents, "
|
||||||
|
"tags, correspondents, document types, and storage paths "
|
||||||
|
"in this database -- not just benchmark-created ones. "
|
||||||
|
"Only run this against a disposable benchmark database, "
|
||||||
|
"never a real install. Re-run with "
|
||||||
|
"--reset --yes-i-know-this-wipes-the-database to proceed.",
|
||||||
|
)
|
||||||
|
self.stdout.write("Resetting existing benchmark data...")
|
||||||
|
reset_benchmark_data()
|
||||||
|
|
||||||
|
data = seed_benchmark_dataset(options["tier"], seed=options["seed"])
|
||||||
|
self.stdout.write(
|
||||||
|
self.style.SUCCESS(
|
||||||
|
f"Seeded tier={options['tier']!r}: {data.documents} documents, "
|
||||||
|
f"{len(data.users)} users, {len(data.groups)} groups.",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
def _handle_run(self, options: dict[str, Any]) -> None:
|
||||||
|
from django.contrib.auth import get_user_model
|
||||||
|
from django.db import connection
|
||||||
|
|
||||||
|
from documents.models import Document
|
||||||
|
from paperless_benchmark.endpoints import run_endpoint_benchmarks
|
||||||
|
from paperless_benchmark.results import append_history
|
||||||
|
|
||||||
|
user_model = get_user_model()
|
||||||
|
try:
|
||||||
|
perf_target = user_model.objects.get(username="perf_target")
|
||||||
|
perf_admin = user_model.objects.get(username="perf_admin")
|
||||||
|
except user_model.DoesNotExist as e:
|
||||||
|
raise CommandError(
|
||||||
|
"No benchmark dataset found. Run `manage.py benchmark seed` first.",
|
||||||
|
) from e
|
||||||
|
|
||||||
|
db_vendor = connection.vendor
|
||||||
|
document_count = Document.objects.count()
|
||||||
|
|
||||||
|
results = run_endpoint_benchmarks(
|
||||||
|
perf_target=perf_target,
|
||||||
|
perf_admin=perf_admin,
|
||||||
|
repeat=options["repeat"],
|
||||||
|
)
|
||||||
|
|
||||||
|
self.stdout.write(f"# label={options['label']} repeat={options['repeat']}")
|
||||||
|
self.stdout.write(
|
||||||
|
f"{'user':7s} {'endpoint':20s} {'queries':>8s} "
|
||||||
|
f"{'min_ms':>9s} {'median_ms':>10s} {'max_ms':>9s}",
|
||||||
|
)
|
||||||
|
for r in results:
|
||||||
|
self.stdout.write(
|
||||||
|
f"{r.user_label:7s} {r.endpoint_name:20s} {r.query_count:8d} "
|
||||||
|
f"{r.min_ms:9.1f} {r.median_ms:10.1f} {r.max_ms:9.1f}",
|
||||||
|
)
|
||||||
|
append_history(
|
||||||
|
{
|
||||||
|
"mode": "run",
|
||||||
|
"label": options["label"],
|
||||||
|
"user": r.user_label,
|
||||||
|
"endpoint": r.endpoint_name,
|
||||||
|
"query_count": r.query_count,
|
||||||
|
"min_ms": r.min_ms,
|
||||||
|
"median_ms": r.median_ms,
|
||||||
|
"max_ms": r.max_ms,
|
||||||
|
"db_vendor": db_vendor,
|
||||||
|
"document_count": document_count,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
def _handle_profile(self, options: dict[str, Any]) -> None:
|
||||||
|
from django.contrib.auth import get_user_model
|
||||||
|
from django.db import connection
|
||||||
|
|
||||||
|
from documents.models import Document
|
||||||
|
from paperless_benchmark.db import capture_explain
|
||||||
|
from paperless_benchmark.harness import run_profile
|
||||||
|
from paperless_benchmark.results import append_history
|
||||||
|
from paperless_benchmark.scenarios import get as get_scenario
|
||||||
|
|
||||||
|
if not options["scenario"]:
|
||||||
|
raise CommandError(
|
||||||
|
"`profile` requires a scenario name; see `list-scenarios`.",
|
||||||
|
)
|
||||||
|
|
||||||
|
scenario = get_scenario(options["scenario"])
|
||||||
|
|
||||||
|
user_model = get_user_model()
|
||||||
|
try:
|
||||||
|
perf_target = user_model.objects.get(username="perf_target")
|
||||||
|
except user_model.DoesNotExist as e:
|
||||||
|
raise CommandError(
|
||||||
|
"No benchmark dataset found. Run `manage.py benchmark seed` first.",
|
||||||
|
) from e
|
||||||
|
|
||||||
|
profile = run_profile(
|
||||||
|
lambda: scenario.run(perf_target),
|
||||||
|
repeat=options["repeat"],
|
||||||
|
)
|
||||||
|
self.stdout.write(
|
||||||
|
f"{scenario.name}: best={profile.best_seconds:.4f}s "
|
||||||
|
f"queries={profile.query_count}",
|
||||||
|
)
|
||||||
|
|
||||||
|
if options["explain"]:
|
||||||
|
if scenario.queryset_for_explain is not None:
|
||||||
|
plan = capture_explain(scenario.queryset_for_explain(perf_target))
|
||||||
|
self.stdout.write(plan)
|
||||||
|
else:
|
||||||
|
self.stdout.write(
|
||||||
|
self.style.WARNING(
|
||||||
|
f"--explain was requested but scenario {scenario.name!r} "
|
||||||
|
"does not support it (no queryset_for_explain); skipping.",
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
append_history(
|
||||||
|
{
|
||||||
|
"mode": "profile",
|
||||||
|
"scenario": scenario.name,
|
||||||
|
"best_seconds": profile.best_seconds,
|
||||||
|
"query_count": profile.query_count,
|
||||||
|
"db_vendor": connection.vendor,
|
||||||
|
"document_count": Document.objects.count(),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
def _handle_list_scenarios(self) -> None:
|
||||||
|
from paperless_benchmark.scenarios import all_scenarios
|
||||||
|
|
||||||
|
for scenario in all_scenarios():
|
||||||
|
self.stdout.write(f"{scenario.name}: {scenario.describe}")
|
||||||
@@ -0,0 +1,39 @@
|
|||||||
|
# src/paperless_benchmark/results.py
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import subprocess
|
||||||
|
from datetime import UTC
|
||||||
|
from datetime import datetime
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
RESULTS_DIR = Path(__file__).resolve().parent.parent.parent / "benchmark_results"
|
||||||
|
|
||||||
|
|
||||||
|
def _current_git_ref() -> str:
|
||||||
|
result = subprocess.run(
|
||||||
|
["git", "rev-parse", "--short", "HEAD"],
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
check=False,
|
||||||
|
)
|
||||||
|
return result.stdout.strip() or "unknown"
|
||||||
|
|
||||||
|
|
||||||
|
def append_history(entry: dict[str, Any], *, code_ref: str | None = None) -> None:
|
||||||
|
"""
|
||||||
|
Append one line to benchmark_results/history.jsonl -- a local-only,
|
||||||
|
append-only, cross-session record of every `benchmark run`/`profile`
|
||||||
|
invocation. Unlike a single overwritten snapshot file, this survives
|
||||||
|
across sessions so a benchmarking effort picked back up days later has
|
||||||
|
a full timeline instead of only the most recent result.
|
||||||
|
"""
|
||||||
|
RESULTS_DIR.mkdir(exist_ok=True)
|
||||||
|
record = {
|
||||||
|
"timestamp": datetime.now(UTC).isoformat(),
|
||||||
|
"code_ref": code_ref or _current_git_ref(),
|
||||||
|
**entry,
|
||||||
|
}
|
||||||
|
with (RESULTS_DIR / "history.jsonl").open("a") as f:
|
||||||
|
f.write(json.dumps(record) + "\n")
|
||||||
@@ -0,0 +1,110 @@
|
|||||||
|
# src/paperless_benchmark/scenarios.py
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from typing import TYPE_CHECKING
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
from collections.abc import Callable
|
||||||
|
|
||||||
|
from django.contrib.auth.models import User
|
||||||
|
from django.db.models import QuerySet
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class Scenario:
|
||||||
|
name: str
|
||||||
|
describe: str
|
||||||
|
run: Callable[[User], Any]
|
||||||
|
queryset_for_explain: Callable[[User], QuerySet] | None = None
|
||||||
|
|
||||||
|
|
||||||
|
_SCENARIOS: dict[str, Scenario] = {}
|
||||||
|
|
||||||
|
|
||||||
|
def register(scenario: Scenario) -> None:
|
||||||
|
_SCENARIOS[scenario.name] = scenario
|
||||||
|
|
||||||
|
|
||||||
|
def get(name: str) -> Scenario:
|
||||||
|
from django.core.management.base import CommandError
|
||||||
|
|
||||||
|
try:
|
||||||
|
return _SCENARIOS[name]
|
||||||
|
except KeyError:
|
||||||
|
available = ", ".join(sorted(_SCENARIOS)) or "(none registered)"
|
||||||
|
raise CommandError(
|
||||||
|
f"Unknown benchmark scenario {name!r}. Available: {available}",
|
||||||
|
) from None
|
||||||
|
|
||||||
|
|
||||||
|
def all_scenarios() -> tuple[Scenario, ...]:
|
||||||
|
return tuple(_SCENARIOS.values())
|
||||||
|
|
||||||
|
|
||||||
|
def _guardian_visibility_query_run(user: User) -> list[int]:
|
||||||
|
from documents.models import Document
|
||||||
|
from documents.permissions import get_objects_for_user_owner_aware
|
||||||
|
|
||||||
|
return list(
|
||||||
|
get_objects_for_user_owner_aware(
|
||||||
|
user,
|
||||||
|
"documents.view_document",
|
||||||
|
Document,
|
||||||
|
).values_list("id", flat=True),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _guardian_visibility_query_queryset(user: User) -> QuerySet:
|
||||||
|
from documents.models import Document
|
||||||
|
from documents.permissions import get_objects_for_user_owner_aware
|
||||||
|
|
||||||
|
return get_objects_for_user_owner_aware(user, "documents.view_document", Document)
|
||||||
|
|
||||||
|
|
||||||
|
register(
|
||||||
|
Scenario(
|
||||||
|
name="guardian_visibility_query",
|
||||||
|
describe=(
|
||||||
|
"Document-visibility queryset for a user with mixed owned/shared "
|
||||||
|
"documents -- exercises documents.permissions."
|
||||||
|
"get_objects_for_user_owner_aware's guardian permission join."
|
||||||
|
),
|
||||||
|
run=_guardian_visibility_query_run,
|
||||||
|
queryset_for_explain=_guardian_visibility_query_queryset,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _permitted_document_ids_run(user: User) -> list[int]:
|
||||||
|
from documents.models import Document
|
||||||
|
from documents.permissions import permitted_document_ids
|
||||||
|
|
||||||
|
return list(
|
||||||
|
Document.objects.filter(id__in=permitted_document_ids(user)).values_list(
|
||||||
|
"id",
|
||||||
|
flat=True,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _permitted_document_ids_queryset(user: User) -> QuerySet:
|
||||||
|
from documents.models import Document
|
||||||
|
from documents.permissions import permitted_document_ids
|
||||||
|
|
||||||
|
return Document.objects.filter(id__in=permitted_document_ids(user))
|
||||||
|
|
||||||
|
|
||||||
|
register(
|
||||||
|
Scenario(
|
||||||
|
name="permitted_document_ids",
|
||||||
|
describe=(
|
||||||
|
"Document-visibility query built from documents.permissions."
|
||||||
|
"permitted_document_ids -- the resolved-ID-set alternative to "
|
||||||
|
"guardian_visibility_query, for side-by-side comparison."
|
||||||
|
),
|
||||||
|
run=_permitted_document_ids_run,
|
||||||
|
queryset_for_explain=_permitted_document_ids_queryset,
|
||||||
|
),
|
||||||
|
)
|
||||||
@@ -0,0 +1,445 @@
|
|||||||
|
# src/paperless_benchmark/seeding.py
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import datetime
|
||||||
|
import random
|
||||||
|
import time
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from typing import TYPE_CHECKING
|
||||||
|
from typing import Literal
|
||||||
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
from django.contrib.auth.models import Group
|
||||||
|
from django.contrib.auth.models import User
|
||||||
|
|
||||||
|
Tier = Literal["home", "medium", "large"]
|
||||||
|
|
||||||
|
CHUNK_SIZE = 5_000
|
||||||
|
|
||||||
|
MIME_TYPES = (
|
||||||
|
"application/pdf",
|
||||||
|
"image/png",
|
||||||
|
"image/jpeg",
|
||||||
|
"text/plain",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Ownership split for documents, mirroring the shape used in the #11950
|
||||||
|
# perf-benchmark dataset (owned-by-target / owned-by-other / unowned).
|
||||||
|
OWNED_BY_TARGET_FRACTION = 0.60
|
||||||
|
OWNED_BY_OTHER_FRACTION = 0.30
|
||||||
|
# remainder (0.10) is unowned
|
||||||
|
|
||||||
|
# Of documents owned by "other" users, the fraction explicitly shared
|
||||||
|
# (view, or view+change) with perf_target via guardian permissions --
|
||||||
|
# this is what exercises the get_user_can_change() per-row N+1 that the
|
||||||
|
# `run` endpoint benchmarks measure.
|
||||||
|
SHARED_WITH_TARGET_FRACTION = 0.5
|
||||||
|
SHARED_WITH_CHANGE_FRACTION = 0.5
|
||||||
|
|
||||||
|
# Guardian permission-row ratios measured from a real install (discussion
|
||||||
|
# #13276): 1,414 user-perm rows / 27,232 group-perm rows over 12,000
|
||||||
|
# documents. Layered across ALL owned documents for the general user/group
|
||||||
|
# pool (not just perf_target's shares), so `profile` scenarios exercise a
|
||||||
|
# realistic permission-join shape for arbitrary users, not only perf_target.
|
||||||
|
USER_PERM_ROWS_PER_DOC = 1_414 / 12_000
|
||||||
|
GROUP_PERM_ROWS_PER_DOC = 27_232 / 12_000
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class _TierCounts:
|
||||||
|
documents: int
|
||||||
|
tags: int
|
||||||
|
correspondents: int
|
||||||
|
document_types: int
|
||||||
|
storage_paths: int
|
||||||
|
other_users: int
|
||||||
|
groups: int
|
||||||
|
tags_per_doc: tuple[int, int]
|
||||||
|
|
||||||
|
|
||||||
|
TIERS: dict[Tier, _TierCounts] = {
|
||||||
|
"home": _TierCounts(
|
||||||
|
documents=500,
|
||||||
|
tags=20,
|
||||||
|
correspondents=10,
|
||||||
|
document_types=8,
|
||||||
|
storage_paths=5,
|
||||||
|
other_users=3,
|
||||||
|
groups=2,
|
||||||
|
tags_per_doc=(1, 3),
|
||||||
|
),
|
||||||
|
"medium": _TierCounts(
|
||||||
|
documents=20_000,
|
||||||
|
tags=100,
|
||||||
|
correspondents=300,
|
||||||
|
document_types=50,
|
||||||
|
storage_paths=20,
|
||||||
|
other_users=10,
|
||||||
|
groups=5,
|
||||||
|
tags_per_doc=(2, 6),
|
||||||
|
),
|
||||||
|
"large": _TierCounts(
|
||||||
|
documents=360_000,
|
||||||
|
tags=1_000,
|
||||||
|
correspondents=5_000,
|
||||||
|
document_types=300,
|
||||||
|
storage_paths=50,
|
||||||
|
other_users=25,
|
||||||
|
groups=10,
|
||||||
|
tags_per_doc=(3, 7),
|
||||||
|
),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def log(msg: str) -> None:
|
||||||
|
print(f"[{time.strftime('%H:%M:%S')}] {msg}", flush=True) # noqa: T201
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True, slots=True)
|
||||||
|
class SeededData:
|
||||||
|
"""
|
||||||
|
Summary of a completed seed run. `documents`/`tags`/`correspondents`/
|
||||||
|
`document_types`/`storage_paths` are counts, not the seeded ORM
|
||||||
|
instances: at the `large` tier (360,000 documents) holding every
|
||||||
|
instance in memory simultaneously is a real risk for zero benefit, since
|
||||||
|
no caller consumes anything but the counts. `perf_target`/`perf_admin`/
|
||||||
|
`users`/`groups` stay as real objects -- at most ~26 users/12 groups even
|
||||||
|
at `large` tier, and small enough to be useful to a future caller.
|
||||||
|
"""
|
||||||
|
|
||||||
|
perf_target: User
|
||||||
|
perf_admin: User
|
||||||
|
users: tuple[User, ...]
|
||||||
|
groups: tuple[Group, ...]
|
||||||
|
documents: int
|
||||||
|
tags: int
|
||||||
|
correspondents: int
|
||||||
|
document_types: int
|
||||||
|
storage_paths: int
|
||||||
|
|
||||||
|
|
||||||
|
def _grant_model_level_permissions(user: User) -> None:
|
||||||
|
"""
|
||||||
|
Grant perf_target Django model-level view/add/change permissions on
|
||||||
|
Document and Tag, on top of the per-object guardian grants seeding
|
||||||
|
creates elsewhere. DRF's PaperlessObjectPermissions checks model-level
|
||||||
|
permissions before guardian's object-level ones are ever consulted, so
|
||||||
|
without this perf_target gets a blanket 403 on /api/documents/ and
|
||||||
|
/api/tags/ regardless of which documents guardian says it can see.
|
||||||
|
"""
|
||||||
|
from django.contrib.auth.models import Permission
|
||||||
|
from django.contrib.contenttypes.models import ContentType
|
||||||
|
|
||||||
|
from documents.models import Document
|
||||||
|
from documents.models import Tag
|
||||||
|
|
||||||
|
for model in (Document, Tag):
|
||||||
|
content_type = ContentType.objects.get_for_model(model)
|
||||||
|
codenames = [
|
||||||
|
f"{action}_{model._meta.model_name}" for action in ("view", "add", "change")
|
||||||
|
]
|
||||||
|
perms = Permission.objects.filter(
|
||||||
|
content_type=content_type,
|
||||||
|
codename__in=codenames,
|
||||||
|
)
|
||||||
|
user.user_permissions.add(*perms)
|
||||||
|
|
||||||
|
|
||||||
|
def _create_users_and_groups(
|
||||||
|
counts: _TierCounts,
|
||||||
|
) -> tuple[User, User, tuple[User, ...], tuple[Group, ...]]:
|
||||||
|
from django.contrib.auth.models import Group
|
||||||
|
|
||||||
|
from documents.tests.factories import UserFactory
|
||||||
|
|
||||||
|
perf_target = UserFactory.create(username="perf_target")
|
||||||
|
_grant_model_level_permissions(perf_target)
|
||||||
|
perf_admin = UserFactory.create(username="perf_admin", superuser=True)
|
||||||
|
other_users = tuple(UserFactory.create_batch(counts.other_users))
|
||||||
|
groups = tuple(
|
||||||
|
Group.objects.create(name=f"benchmark_group_{i}") for i in range(counts.groups)
|
||||||
|
)
|
||||||
|
log(
|
||||||
|
f"Created users: 1 target, 1 superuser, {len(other_users)} other, "
|
||||||
|
f"{len(groups)} groups.",
|
||||||
|
)
|
||||||
|
return perf_target, perf_admin, other_users, groups
|
||||||
|
|
||||||
|
|
||||||
|
def _create_lookup_tables(counts: _TierCounts):
|
||||||
|
from documents.models import Correspondent
|
||||||
|
from documents.models import DocumentType
|
||||||
|
from documents.models import StoragePath
|
||||||
|
from documents.models import Tag
|
||||||
|
from documents.tests.factories import CorrespondentFactory
|
||||||
|
from documents.tests.factories import DocumentTypeFactory
|
||||||
|
from documents.tests.factories import StoragePathFactory
|
||||||
|
from documents.tests.factories import TagFactory
|
||||||
|
|
||||||
|
tags = tuple(Tag.objects.bulk_create(TagFactory.build_batch(counts.tags)))
|
||||||
|
correspondents = tuple(
|
||||||
|
Correspondent.objects.bulk_create(
|
||||||
|
CorrespondentFactory.build_batch(counts.correspondents),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
document_types = tuple(
|
||||||
|
DocumentType.objects.bulk_create(
|
||||||
|
DocumentTypeFactory.build_batch(counts.document_types),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
storage_paths = tuple(
|
||||||
|
StoragePath.objects.bulk_create(
|
||||||
|
StoragePathFactory.build_batch(counts.storage_paths),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
log(
|
||||||
|
f"Created {len(tags)} tags, {len(correspondents)} correspondents, "
|
||||||
|
f"{len(document_types)} document types, {len(storage_paths)} storage paths.",
|
||||||
|
)
|
||||||
|
return tags, correspondents, document_types, storage_paths
|
||||||
|
|
||||||
|
|
||||||
|
def _assign_owner(rng: random.Random, perf_target: User, other_users: tuple[User, ...]):
|
||||||
|
roll = rng.random()
|
||||||
|
if roll < OWNED_BY_TARGET_FRACTION:
|
||||||
|
return perf_target, "target"
|
||||||
|
if roll < OWNED_BY_TARGET_FRACTION + OWNED_BY_OTHER_FRACTION:
|
||||||
|
return rng.choice(other_users), "other"
|
||||||
|
return None, "unowned"
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_documents(
|
||||||
|
rng: random.Random,
|
||||||
|
counts: _TierCounts,
|
||||||
|
tags,
|
||||||
|
correspondents,
|
||||||
|
document_types,
|
||||||
|
storage_paths,
|
||||||
|
perf_target,
|
||||||
|
other_users,
|
||||||
|
) -> tuple[int, list[int]]:
|
||||||
|
"""
|
||||||
|
Bulk-create `counts.documents` documents in chunks. Returns the total
|
||||||
|
document count plus a lightweight list of pks for documents that ended
|
||||||
|
up with an owner (target or other) -- that's all
|
||||||
|
`_grant_general_permissions` needs to sample from, so full `Document`
|
||||||
|
instances aren't accumulated across chunks (a real memory concern at the
|
||||||
|
`large` tier's 360,000 documents).
|
||||||
|
"""
|
||||||
|
from django.contrib.auth.models import Permission
|
||||||
|
from django.contrib.contenttypes.models import ContentType
|
||||||
|
from guardian.models import UserObjectPermission
|
||||||
|
|
||||||
|
from documents.models import Document
|
||||||
|
from documents.tests.factories import DocumentFactory
|
||||||
|
|
||||||
|
tag_ids = [t.pk for t in tags]
|
||||||
|
correspondent_ids = [c.pk for c in correspondents]
|
||||||
|
document_type_ids = [d.pk for d in document_types]
|
||||||
|
storage_path_ids = [s.pk for s in storage_paths]
|
||||||
|
|
||||||
|
doc_content_type = ContentType.objects.get_for_model(Document)
|
||||||
|
view_perm = Permission.objects.get(
|
||||||
|
codename="view_document",
|
||||||
|
content_type=doc_content_type,
|
||||||
|
)
|
||||||
|
change_perm = Permission.objects.get(
|
||||||
|
codename="change_document",
|
||||||
|
content_type=doc_content_type,
|
||||||
|
)
|
||||||
|
through_model = Document.tags.through
|
||||||
|
|
||||||
|
document_count = 0
|
||||||
|
owned_document_pks: list[int] = []
|
||||||
|
remaining = counts.documents
|
||||||
|
while remaining > 0:
|
||||||
|
chunk_n = min(CHUNK_SIZE, remaining)
|
||||||
|
remaining -= chunk_n
|
||||||
|
|
||||||
|
batch = []
|
||||||
|
owner_buckets = []
|
||||||
|
for _ in range(chunk_n):
|
||||||
|
doc = DocumentFactory.build(
|
||||||
|
mime_type=rng.choice(MIME_TYPES),
|
||||||
|
page_count=rng.randint(1, 30),
|
||||||
|
correspondent_id=(
|
||||||
|
rng.choice(correspondent_ids)
|
||||||
|
if correspondent_ids and rng.random() < 0.8
|
||||||
|
else None
|
||||||
|
),
|
||||||
|
document_type_id=(
|
||||||
|
rng.choice(document_type_ids)
|
||||||
|
if document_type_ids and rng.random() < 0.6
|
||||||
|
else None
|
||||||
|
),
|
||||||
|
storage_path_id=(
|
||||||
|
rng.choice(storage_path_ids)
|
||||||
|
if storage_path_ids and rng.random() < 0.8
|
||||||
|
else None
|
||||||
|
),
|
||||||
|
# Document.created is a plain DateField (default: today).
|
||||||
|
# Leaving it unset would give every seeded document the same
|
||||||
|
# date, collapsing Document's ("-created",) ordering index
|
||||||
|
# into a single-valued sort key -- spread it over a
|
||||||
|
# realistic multi-year window instead.
|
||||||
|
created=datetime.date.today()
|
||||||
|
- datetime.timedelta(days=rng.randint(0, 365 * 3)),
|
||||||
|
)
|
||||||
|
owner, bucket = _assign_owner(rng, perf_target, other_users)
|
||||||
|
doc.owner_id = owner.pk if owner else None
|
||||||
|
batch.append(doc)
|
||||||
|
owner_buckets.append(bucket)
|
||||||
|
|
||||||
|
created = Document.objects.bulk_create(batch, batch_size=CHUNK_SIZE)
|
||||||
|
|
||||||
|
through_rows = []
|
||||||
|
for doc in created:
|
||||||
|
k = rng.randint(*counts.tags_per_doc)
|
||||||
|
for tag_id in rng.sample(tag_ids, min(k, len(tag_ids))):
|
||||||
|
through_rows.append(through_model(document_id=doc.pk, tag_id=tag_id))
|
||||||
|
if through_rows:
|
||||||
|
through_model.objects.bulk_create(through_rows, batch_size=CHUNK_SIZE)
|
||||||
|
|
||||||
|
perm_rows = []
|
||||||
|
for doc, bucket in zip(created, owner_buckets, strict=True):
|
||||||
|
if bucket == "unowned":
|
||||||
|
continue
|
||||||
|
owned_document_pks.append(doc.pk)
|
||||||
|
|
||||||
|
if bucket != "other":
|
||||||
|
continue
|
||||||
|
if rng.random() >= SHARED_WITH_TARGET_FRACTION:
|
||||||
|
continue
|
||||||
|
perm_rows.append(
|
||||||
|
UserObjectPermission(
|
||||||
|
permission=view_perm,
|
||||||
|
content_type=doc_content_type,
|
||||||
|
object_pk=str(doc.pk),
|
||||||
|
user=perf_target,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
if rng.random() < SHARED_WITH_CHANGE_FRACTION:
|
||||||
|
perm_rows.append(
|
||||||
|
UserObjectPermission(
|
||||||
|
permission=change_perm,
|
||||||
|
content_type=doc_content_type,
|
||||||
|
object_pk=str(doc.pk),
|
||||||
|
user=perf_target,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
if perm_rows:
|
||||||
|
UserObjectPermission.objects.bulk_create(perm_rows, batch_size=CHUNK_SIZE)
|
||||||
|
|
||||||
|
document_count += len(created)
|
||||||
|
log(f" {document_count}/{counts.documents} documents seeded")
|
||||||
|
|
||||||
|
return document_count, owned_document_pks
|
||||||
|
|
||||||
|
|
||||||
|
def _grant_general_permissions(
|
||||||
|
rng: random.Random,
|
||||||
|
owned_document_pks: list[int],
|
||||||
|
users,
|
||||||
|
groups,
|
||||||
|
) -> None:
|
||||||
|
"""
|
||||||
|
Layer realistic (issue #13276-derived) guardian permission-row ratios
|
||||||
|
across owned documents for the general user/group pool, so `profile`
|
||||||
|
scenarios exercise the same permission-join shape regardless of which
|
||||||
|
user they check visibility for (not just perf_target).
|
||||||
|
"""
|
||||||
|
from django.contrib.auth.models import Permission
|
||||||
|
from django.contrib.contenttypes.models import ContentType
|
||||||
|
from guardian.models import GroupObjectPermission
|
||||||
|
from guardian.models import UserObjectPermission
|
||||||
|
|
||||||
|
from documents.models import Document
|
||||||
|
|
||||||
|
if not owned_document_pks or not users:
|
||||||
|
return
|
||||||
|
|
||||||
|
doc_content_type = ContentType.objects.get_for_model(Document)
|
||||||
|
view_perm = Permission.objects.get(
|
||||||
|
codename="view_document",
|
||||||
|
content_type=doc_content_type,
|
||||||
|
)
|
||||||
|
|
||||||
|
n_user_perms = round(len(owned_document_pks) * USER_PERM_ROWS_PER_DOC)
|
||||||
|
n_group_perms = (
|
||||||
|
round(len(owned_document_pks) * GROUP_PERM_ROWS_PER_DOC) if groups else 0
|
||||||
|
)
|
||||||
|
|
||||||
|
user_rows = [
|
||||||
|
UserObjectPermission(
|
||||||
|
permission=view_perm,
|
||||||
|
content_type=doc_content_type,
|
||||||
|
object_pk=str(rng.choice(owned_document_pks)),
|
||||||
|
user=rng.choice(users),
|
||||||
|
)
|
||||||
|
for _ in range(n_user_perms)
|
||||||
|
]
|
||||||
|
if user_rows:
|
||||||
|
UserObjectPermission.objects.bulk_create(
|
||||||
|
user_rows,
|
||||||
|
batch_size=CHUNK_SIZE,
|
||||||
|
ignore_conflicts=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
group_rows = [
|
||||||
|
GroupObjectPermission(
|
||||||
|
permission=view_perm,
|
||||||
|
content_type=doc_content_type,
|
||||||
|
object_pk=str(rng.choice(owned_document_pks)),
|
||||||
|
group=rng.choice(groups),
|
||||||
|
)
|
||||||
|
for _ in range(n_group_perms)
|
||||||
|
]
|
||||||
|
if group_rows:
|
||||||
|
GroupObjectPermission.objects.bulk_create(
|
||||||
|
group_rows,
|
||||||
|
batch_size=CHUNK_SIZE,
|
||||||
|
ignore_conflicts=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
log(f" Granted {len(user_rows)} user perms, {len(group_rows)} group perms.")
|
||||||
|
|
||||||
|
|
||||||
|
def seed_benchmark_dataset(tier: Tier, *, seed: int = 42) -> SeededData:
|
||||||
|
"""
|
||||||
|
Build a benchmark dataset at the given scale tier: a named perf_target
|
||||||
|
(mixed owned/shared documents) and perf_admin (superuser) for endpoint
|
||||||
|
benchmarking, plus a general user/group pool with realistic guardian
|
||||||
|
permission-row ratios for profile scenarios.
|
||||||
|
"""
|
||||||
|
counts = TIERS[tier]
|
||||||
|
rng = random.Random(seed)
|
||||||
|
|
||||||
|
log(f"Seeding tier={tier!r}")
|
||||||
|
perf_target, perf_admin, other_users, groups = _create_users_and_groups(counts)
|
||||||
|
tags, correspondents, document_types, storage_paths = _create_lookup_tables(counts)
|
||||||
|
document_count, owned_document_pks = _seed_documents(
|
||||||
|
rng,
|
||||||
|
counts,
|
||||||
|
tags,
|
||||||
|
correspondents,
|
||||||
|
document_types,
|
||||||
|
storage_paths,
|
||||||
|
perf_target,
|
||||||
|
other_users,
|
||||||
|
)
|
||||||
|
all_users = (perf_target, *other_users)
|
||||||
|
_grant_general_permissions(rng, owned_document_pks, all_users, groups)
|
||||||
|
|
||||||
|
log(f"Done. {document_count} documents seeded for tier={tier!r}.")
|
||||||
|
|
||||||
|
return SeededData(
|
||||||
|
perf_target=perf_target,
|
||||||
|
perf_admin=perf_admin,
|
||||||
|
users=all_users,
|
||||||
|
groups=groups,
|
||||||
|
documents=document_count,
|
||||||
|
tags=len(tags),
|
||||||
|
correspondents=len(correspondents),
|
||||||
|
document_types=len(document_types),
|
||||||
|
storage_paths=len(storage_paths),
|
||||||
|
)
|
||||||
@@ -3735,15 +3735,15 @@ crypto = [
|
|||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "pymdown-extensions"
|
name = "pymdown-extensions"
|
||||||
version = "10.21.3"
|
version = "11.0"
|
||||||
source = { registry = "https://pypi.org/simple" }
|
source = { registry = "https://pypi.org/simple" }
|
||||||
dependencies = [
|
dependencies = [
|
||||||
{ name = "markdown", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
{ name = "markdown", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
||||||
{ name = "pyyaml", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
{ name = "pyyaml", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
|
||||||
]
|
]
|
||||||
sdist = { url = "https://files.pythonhosted.org/packages/9e/26/d1015444da4d952a1ca487a236b522eb979766f0295a0bd0c5fc089989a9/pymdown_extensions-10.21.3.tar.gz", hash = "sha256:72cfcf55f07aea0d4af2c4f11dd4e52466ddfb1bb819673146398e0bd3a77354", size = 854140, upload-time = "2026-05-13T12:57:32.267Z" }
|
sdist = { url = "https://files.pythonhosted.org/packages/47/67/f1e79672a5f91985577c7984c9709ca110e4fd37fe7fd167b60422e6ccc2/pymdown_extensions-11.0.tar.gz", hash = "sha256:8269cef0247f9e2d0a62fcea10860aba05c1cbab5470fd4b63230b96434dc589", size = 857049, upload-time = "2026-06-23T02:27:45.146Z" }
|
||||||
wheels = [
|
wheels = [
|
||||||
{ url = "https://files.pythonhosted.org/packages/7e/85/545a951eecc270fcd688288c600017e2050a1aacb56c711d208586d3e470/pymdown_extensions-10.21.3-py3-none-any.whl", hash = "sha256:d7a5d08014fc571e80ca21dd6f854e31f94c489800350564d55d15b3c41e76b6", size = 269002, upload-time = "2026-05-13T12:57:30.296Z" },
|
{ url = "https://files.pythonhosted.org/packages/af/b6/1ae53367e28b9cffa3be7574e13fbe4589694272fd47710fbdbafd3d63c6/pymdown_extensions-11.0-py3-none-any.whl", hash = "sha256:fbc4acb641814fa9d17521bbd21a5240ef739a662f11c06330c4b78c93e954d6", size = 269415, upload-time = "2026-06-23T02:27:43.826Z" },
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
|
|||||||
Reference in New Issue
Block a user