From be82fcb70a3f49301f4a887eb8bf5df39e162e9a Mon Sep 17 00:00:00 2001 From: shamoon <4887959+shamoon@users.noreply.github.com> Date: Tue, 24 Feb 2026 11:47:24 -0800 Subject: [PATCH 01/40] Documentation: docs cleanup (#12158) --- docs/development.md | 2 +- docs/setup.md | 563 ++++++++++++++++++++------------------------ docs/usage.md | 5 + 3 files changed, 261 insertions(+), 309 deletions(-) diff --git a/docs/development.md b/docs/development.md index 8b8ff641f..d116529f6 100644 --- a/docs/development.md +++ b/docs/development.md @@ -358,7 +358,7 @@ If you want to build the documentation locally, this is how you do it: $ uv run zensical serve ``` -## Building the Docker image +## Building the Docker image {#docker_build} The docker image is primarily built by the GitHub actions workflow, but it can be faster when developing to build and tag an image locally. diff --git a/docs/setup.md b/docs/setup.md index e8ee8915f..32f93cf6a 100644 --- a/docs/setup.md +++ b/docs/setup.md @@ -4,53 +4,74 @@ title: Setup # Installation -You can go multiple routes to setup and run Paperless: - -- [Use the script to setup a Docker install](#docker_script) -- [Use the Docker compose templates](#docker) -- [Build the Docker image yourself](#docker_build) -- [Install Paperless-ngx directly on your system manually ("bare metal")](#bare_metal) -- A user-maintained list of commercial hosting providers can be found [in the wiki](https://github.com/paperless-ngx/paperless-ngx/wiki/Related-Projects) - -The Docker routes are quick & easy. These are the recommended routes. -This configures all the stuff from the above automatically so that it -just works and uses sensible defaults for all configuration options. -Here you find a cheat-sheet for docker beginners: [CLI -Basics](https://www.sehn.tech/refs/devops-with-docker/) - -The bare metal route is complicated to setup but makes it easier should -you want to contribute some code back. You need to configure and run the -above mentioned components yourself. - -### Use the Installation Script {#docker_script} - -Paperless provides an interactive installation script to setup a Docker Compose -installation. The script asks for a couple configuration options, and will then create the -necessary configuration files, pull the docker image, start Paperless-ngx and create your superuser -account. The script essentially automatically performs the steps described in [Docker setup](#docker). - -1. Make sure that Docker and Docker Compose are [installed](https://docs.docker.com/engine/install/){:target="\_blank"}. - -2. Download and run the installation script: +!!! tip "Quick Start" + If you just want Paperless-ngx running quickly, use our installation script: ```shell-session bash -c "$(curl --location --silent --show-error https://raw.githubusercontent.com/paperless-ngx/paperless-ngx/main/install-paperless-ngx.sh)" ``` + _If piping into a shell directly from the internet makes you nervous, inspect [the script](https://github.com/paperless-ngx/paperless-ngx/blob/main/install-paperless-ngx.sh) first!_ - !!! note +## Overview - macOS users will need to install [gnu-sed](https://formulae.brew.sh/formula/gnu-sed) with support - for running as `sed` as well as [wget](https://formulae.brew.sh/formula/wget). +Choose the installation route that best fits your setup: -### Use Docker Compose {#docker} +| Route | Best for | Effort | +| ----------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- | ------ | +| [Installation script](#docker_script) | Fastest first-time setup with guided prompts (recommended for most users) | Low | +| [Docker Compose templates](#docker) | Manual control over compose files and settings | Medium | +| [Bare metal](#bare_metal) | Advanced setups, packaging, and development-adjacent workflows | High | +| [Hosted providers (wiki)](https://github.com/paperless-ngx/paperless-ngx/wiki/Related-Projects#hosting-providers) | Managed hosting options maintained by the community — check details carefully | Varies | -1. Make sure that Docker and Docker Compose are [installed](https://docs.docker.com/engine/install/){:target="\_blank"}. +For most users, Docker is the best option. It is faster to set up, +easier to maintain, and ships with sensible defaults. -2. Go to the [/docker/compose directory on the project +The bare-metal route gives you more control, but it requires manual +installation and operation of all components. It is usually best suited +for advanced users and contributors. + +!!! info + + Because [superuser](usage.md#superusers) accounts have full access to all objects and documents, you may want to create a separate user account for daily use, + or "downgrade" your superuser account to a normal user account after setup. + +## Installation Script {#docker_script} + +Paperless-ngx provides an interactive script for Docker Compose setups. +It asks a few configuration questions, then creates the required files, +pulls the image, starts the containers, and creates your [superuser](usage.md#superusers) +account. In short, it automates the [Docker Compose setup](#docker) described below. + +#### Prerequisites + +- Docker and Docker Compose must be [installed](https://docs.docker.com/engine/install/){:target="\_blank"}. +- macOS users will need [GNU sed](https://formulae.brew.sh/formula/gnu-sed) with support for running as `sed` as well as [wget](https://formulae.brew.sh/formula/wget). + +#### Run the installation script + +```shell-session +bash -c "$(curl --location --silent --show-error https://raw.githubusercontent.com/paperless-ngx/paperless-ngx/main/install-paperless-ngx.sh)" +``` + +#### After installation + +Paperless-ngx should be available at `http://127.0.0.1:8000` (or similar, +depending on your configuration) and you will be able to login with the +credentials you provided during the installation script. + +## Docker Compose Install {#docker} + +#### Prerequisites + +- Docker and Docker Compose must be [installed](https://docs.docker.com/engine/install/){:target="\_blank"}. + +#### Installation + +1. Go to the [/docker/compose directory on the project page](https://github.com/paperless-ngx/paperless-ngx/tree/main/docker/compose){:target="\_blank"} - and download one of the `docker-compose.*.yml` files, depending on which database backend - you want to use. Place the files in a local directory and rename it `docker-compose.yml`. Download the - `docker-compose.env` file and the `.env` file as well in the same directory. + and download one `docker-compose.*.yml` file for your preferred + database backend. Save it in a local directory as `docker-compose.yml`. + Also download `docker-compose.env` and `.env` into that same directory. If you want to enable optional support for Office and other documents, download a file with `-tika` in the file name. @@ -60,15 +81,16 @@ account. The script essentially automatically performs the steps described in [D For new installations, it is recommended to use PostgreSQL as the database backend. -3. Modify `docker-compose.yml` as needed. For example, you may want to change the paths to the - consumption, media etc. directories to use 'bind mounts'. +2. Modify `docker-compose.yml` as needed. For example, you may want to + change the paths for `consume`, `media`, and other directories to + use bind mounts. Find the line that specifies where to mount the directory, e.g.: ```yaml - ./consume:/usr/src/paperless/consume ``` - Replace the part _before_ the colon with a local directory of your choice: + Replace the part _before_ the colon with your local directory: ```yaml - /home/jonaswinkler/paperless-inbox:/usr/src/paperless/consume @@ -82,38 +104,15 @@ account. The script essentially automatically performs the steps described in [D - 8010:8000 ``` - **Rootless** - - !!! warning - - It is currently not possible to run the container rootless if additional languages are specified via `PAPERLESS_OCR_LANGUAGES`. - - If you want to run Paperless as a rootless container, you will need - to do the following in your `docker-compose.yml`: - - - set the `user` running the container to map to the `paperless` - user in the container. This value (`user_id` below), should be - the same id that `USERMAP_UID` and `USERMAP_GID` are set to in - the next step. See `USERMAP_UID` and `USERMAP_GID` - [here](configuration.md#docker). - - Your entry for Paperless should contain something like: - - > ``` - > webserver: - > image: ghcr.io/paperless-ngx/paperless-ngx:latest - > user: - > ``` - -4. Modify `docker-compose.env` with any configuration options you'd like. +3. Modify `docker-compose.env` with any configuration options you need. See the [configuration documentation](configuration.md) for all options. You may also need to set `USERMAP_UID` and `USERMAP_GID` to - the uid and gid of your user on the host system. Use `id -u` and - `id -g` to get these. This ensures that both the container and the host - user have write access to the consumption directory. If your UID - and GID on the host system is 1000 (the default for the first normal - user on most systems), it will work out of the box without any + the UID and GID of your user on the host system. Use `id -u` and + `id -g` to get these values. This ensures both the container and the + host user can write to the consumption directory. If your UID and + GID are `1000` (the default for the first normal user on many + systems), this usually works out of the box without modifications. Run `id "username"` to check. !!! note @@ -122,79 +121,62 @@ account. The script essentially automatically performs the steps described in [D appending `_FILE` to configuration values. For example [`PAPERLESS_DBUSER`](configuration.md#PAPERLESS_DBUSER) can be set using `PAPERLESS_DBUSER_FILE=/var/run/secrets/password.txt`. - !!! warning - - Some file systems such as NFS network shares don't support file - system notifications with `inotify`. When storing the consumption - directory on such a file system, paperless will not pick up new - files with the default configuration. You will need to use - [`PAPERLESS_CONSUMER_POLLING`](configuration.md#PAPERLESS_CONSUMER_POLLING), which will disable inotify. See - [here](configuration.md#polling). - -5. Run `docker compose pull`. This will pull the image from the GitHub container registry - by default but you can change the image to pull from Docker Hub by changing the `image` +4. Run `docker compose pull`. This pulls the image from the GitHub container registry + by default, but you can pull from Docker Hub by changing the `image` line to `image: paperlessngx/paperless-ngx:latest`. -6. Run `docker compose up -d`. This will create and start the necessary containers. +5. Run `docker compose up -d`. This will create and start the necessary containers. -7. Congratulations! Your Paperless-ngx instance should now be accessible at `http://127.0.0.1:8000` - (or similar, depending on your configuration). When you first access the web interface, you will be - prompted to create a superuser account. +#### After installation -### Build the Docker image yourself {#docker_build} +Your Paperless-ngx instance should now be accessible at +`http://127.0.0.1:8000` (or similar, depending on your configuration). +When you first access the web interface, you will be prompted to create +a [superuser](usage.md#superusers) account. -1. Clone the entire repository of paperless: +#### Optional Advanced Compose Configurations {#advanced_compose data-toc-label="Advanced Compose Configurations"} - ```shell-session - git clone https://github.com/paperless-ngx/paperless-ngx - ``` +**Rootless** - The main branch always reflects the latest stable version. +!!! warning -2. Copy one of the `docker/compose/docker-compose.*.yml` to - `docker-compose.yml` in the root folder, depending on which database - backend you want to use. Copy `docker-compose.env` into the project - root as well. + It is currently not possible to run the container rootless if additional languages are specified via `PAPERLESS_OCR_LANGUAGES`. -3. In the `docker-compose.yml` file, find the line that instructs - Docker Compose to pull the paperless image from Docker Hub: +If you want to run Paperless as a rootless container, make this +change in `docker-compose.yml`: - ```yaml - webserver: - image: ghcr.io/paperless-ngx/paperless-ngx:latest - ``` +- Set the `user` running the container to map to the `paperless` + user in the container. This value (`user_id` below) should be + the same ID that `USERMAP_UID` and `USERMAP_GID` are set to in + `docker-compose.env`. See `USERMAP_UID` and `USERMAP_GID` + [here](configuration.md#docker). - and replace it with a line that instructs Docker Compose to build - the image from the current working directory instead: +Your entry for Paperless should contain something like: - ```yaml - webserver: - build: - context: . - ``` +> ``` +> webserver: +> image: ghcr.io/paperless-ngx/paperless-ngx:latest +> user: +> ``` -4. Follow the [Docker setup](#docker) above except when asked to run - `docker compose pull` to pull the image, run +**File systems without inotify support (e.g. NFS)** - ```shell-session - docker compose build - ``` +Some file systems, such as NFS network shares, don't support file system +notifications with `inotify`. When the consumption directory is on such a +file system, Paperless-ngx will not pick up new files with the default +configuration. Use [`PAPERLESS_CONSUMER_POLLING`](configuration.md#PAPERLESS_CONSUMER_POLLING) +to enable polling and disable inotify. See [here](configuration.md#polling). - instead to build the image. +## Bare Metal Install {#bare_metal} -### Bare Metal Route {#bare_metal} +#### Prerequisites -Paperless runs on linux only. The following procedure has been tested on -a minimal installation of Debian/Buster, which is the current stable -release at the time of writing. Windows is not and will never be -supported. +- Paperless runs on Linux only, Windows is not supported. +- Python 3 is required with versions 3.10 - 3.12 currently supported. Newer versions may work, but some dependencies may not be fully compatible. -Paperless requires Python 3. At this time, 3.10 - 3.12 are tested versions. -Newer versions may work, but some dependencies may not fully support newer versions. -Support for older Python versions may be dropped as they reach end of life or as newer versions -are released, dependency support is confirmed, etc. +#### Installation -1. Install dependencies. Paperless requires the following packages. +1. Install dependencies. Paperless requires the following packages: - `python3` - `python3-pip` @@ -258,8 +240,8 @@ are released, dependency support is confirmed, etc. 2. Install `redis` >= 6.0 and configure it to start automatically. -3. Optional. Install `postgresql` and configure a database, user and - password for paperless. If you do not wish to use PostgreSQL, +3. Optional: Install `postgresql` and configure a database, user, and + password for Paperless-ngx. If you do not wish to use PostgreSQL, MariaDB and SQLite are available as well. !!! note @@ -268,61 +250,60 @@ are released, dependency support is confirmed, etc. extension](https://code.djangoproject.com/wiki/JSON1Extension) is enabled. This is usually the case, but not always. -4. Create a system user with a new home folder under which you wish - to run paperless. +4. Create a system user with a new home folder in which you want + to run Paperless-ngx. ```shell-session adduser paperless --system --home /opt/paperless --group ``` -5. Get the release archive from - for example with +5. Download a release archive from + . For example: ```shell-session - curl -O -L https://github.com/paperless-ngx/paperless-ngx/releases/download/v1.10.2/paperless-ngx-v1.10.2.tar.xz + curl -O -L https://github.com/paperless-ngx/paperless-ngx/releases/download/vX.Y.Z/paperless-ngx-vX.Y.Z.tar.xz ``` Extract the archive with ```shell-session - tar -xf paperless-ngx-v1.10.2.tar.xz + tar -xf paperless-ngx-vX.Y.Z.tar.xz ``` - and copy the contents to the - home folder of the user you created before (`/opt/paperless`). + and copy the contents to the home directory of the user you created + earlier (`/opt/paperless`). - Optional: If you cloned the git repo, you will have to - compile the frontend yourself, see [here](development.md#front-end-development) + Optional: If you cloned the Git repository, you will need to + compile the frontend yourself. See [here](development.md#front-end-development) and use the `build` step, not `serve`. -6. Configure paperless. See [configuration](configuration.md) for details. +6. Configure Paperless-ngx. See [configuration](configuration.md) for details. Edit the included `paperless.conf` and adjust the settings to your - needs. Required settings for getting - paperless running are: + needs. Required settings for getting Paperless-ngx running are: - - [`PAPERLESS_REDIS`](configuration.md#PAPERLESS_REDIS) should point to your redis server, such as - . - - [`PAPERLESS_DBENGINE`](configuration.md#PAPERLESS_DBENGINE) optional, and should be one of `postgres`, + - [`PAPERLESS_REDIS`](configuration.md#PAPERLESS_REDIS) should point to your Redis server, such as + `redis://localhost:6379`. + - [`PAPERLESS_DBENGINE`](configuration.md#PAPERLESS_DBENGINE) is optional, and should be one of `postgres`, `mariadb`, or `sqlite` - [`PAPERLESS_DBHOST`](configuration.md#PAPERLESS_DBHOST) should be the hostname on which your PostgreSQL server is running. Do not configure this to use SQLite instead. Also configure port, database name, user and password as necessary. - - [`PAPERLESS_CONSUMPTION_DIR`](configuration.md#PAPERLESS_CONSUMPTION_DIR) should point to a folder which - paperless should watch for documents. You might want to have - this somewhere else. Likewise, [`PAPERLESS_DATA_DIR`](configuration.md#PAPERLESS_DATA_DIR) and - [`PAPERLESS_MEDIA_ROOT`](configuration.md#PAPERLESS_MEDIA_ROOT) define where paperless stores its data. - If you like, you can point both to the same directory. + - [`PAPERLESS_CONSUMPTION_DIR`](configuration.md#PAPERLESS_CONSUMPTION_DIR) should point to the folder + that Paperless-ngx should watch for incoming documents. + Likewise, [`PAPERLESS_DATA_DIR`](configuration.md#PAPERLESS_DATA_DIR) and + [`PAPERLESS_MEDIA_ROOT`](configuration.md#PAPERLESS_MEDIA_ROOT) define where Paperless-ngx stores its data. + If needed, these can point to the same directory. - [`PAPERLESS_SECRET_KEY`](configuration.md#PAPERLESS_SECRET_KEY) should be a random sequence of characters. It's used for authentication. Failure to do so allows third parties to forge authentication credentials. - - [`PAPERLESS_URL`](configuration.md#PAPERLESS_URL) if you are behind a reverse proxy. This should + - Set [`PAPERLESS_URL`](configuration.md#PAPERLESS_URL) if you are behind a reverse proxy. This should point to your domain. Please see [configuration](configuration.md) for more information. - Many more adjustments can be made to paperless, especially the OCR - part. The following options are recommended for everyone: + You can make many more adjustments, especially for OCR. + The following options are recommended for most users: - Set [`PAPERLESS_OCR_LANGUAGE`](configuration.md#PAPERLESS_OCR_LANGUAGE) to the language most of your documents are written in. @@ -332,15 +313,14 @@ are released, dependency support is confirmed, etc. Ensure your Redis instance [is secured](https://redis.io/docs/latest/operate/oss_and_stack/management/security/). -7. Create the following directories if they are missing: +7. Create the following directories if they do not already exist: - `/opt/paperless/media` - `/opt/paperless/data` - `/opt/paperless/consume` - Adjust as necessary if you configured different folders. - Ensure that the paperless user has write permissions for every one - of these folders with + Adjust these paths if you configured different folders. + Then verify that the `paperless` user has write permissions: ```shell-session ls -l -d /opt/paperless/media @@ -354,45 +334,44 @@ are released, dependency support is confirmed, etc. sudo chown paperless:paperless /opt/paperless/consume ``` -8. Install python requirements from the `requirements.txt` file. +8. Install Python dependencies from `requirements.txt`. ```shell-session sudo -Hu paperless pip3 install -r requirements.txt ``` - This will install all python dependencies in the home directory of + This will install all Python dependencies in the home directory of the new paperless user. !!! tip - It is up to you if you wish to use a virtual environment or not for the Python - dependencies. This is an alternative to the above and may require adjusting - the example scripts to utilize the virtual environment paths + You can use a virtual environment if you prefer. If you do, + you may need to adjust the example scripts for your virtual + environment paths. !!! tip If you use modern Python tooling, such as `uv`, installation will not include - dependencies for Postgres or Mariadb. You can select those extras with `--extra ` - or all with `--all-extras` + dependencies for PostgreSQL or MariaDB. You can select those + extras with `--extra `, or install all extras with + `--all-extras`. -9. Go to `/opt/paperless/src`, and execute the following command: +9. Go to `/opt/paperless/src` and execute the following command: ```bash # This creates the database schema. sudo -Hu paperless python3 manage.py migrate ``` - When you first access the web interface you will be prompted to create a superuser account. - -10. Optional: Test that paperless is working by executing +10. Optional: Test that Paperless-ngx is working by running ```bash # Manually starts the webserver sudo -Hu paperless python3 manage.py runserver ``` - and pointing your browser to http://localhost:8000 if - accessing from the same devices on which paperless is installed. + Then point your browser to `http://localhost:8000` if + accessing from the same device on which Paperless-ngx is installed. If accessing from another machine, set up systemd services. You may need to set `PAPERLESS_DEBUG=true` in order for the development server to work normally in your browser. @@ -400,23 +379,24 @@ are released, dependency support is confirmed, etc. !!! warning This is a development server which should not be used in production. - It is not audited for security and performance is inferior to - production ready web servers. + It is not audited for security, and performance is inferior to + production-ready web servers. !!! tip This will not start the consumer. Paperless does this in a separate process. -11. Setup systemd services to run paperless automatically. You may use +11. Set up systemd services to run Paperless-ngx automatically. You may use the service definition files included in the `scripts` folder as a starting point. - Paperless needs the `webserver` script to run the webserver, the - `consumer` script to watch the input folder, `taskqueue` for the - background workers used to handle things like document consumption - and the `scheduler` script to run tasks such as email checking at - certain times . + Paperless needs: + + - The `webserver` script to run the webserver. + - The `consumer` script to watch the input folder. + - The `taskqueue` script for background workers (document consumption, etc.). + - The `scheduler` script for periodic tasks such as email checking. !!! note @@ -425,9 +405,9 @@ are released, dependency support is confirmed, etc. `Require=paperless-webserver.socket` in the `webserver` script and configure `granian` to listen on port 80 (set `GRANIAN_PORT`). - These services rely on redis and optionally the database server, but + These services rely on Redis and optionally the database server, but don't need to be started in any particular order. The example files - depend on redis being started. If you use a database server, you + depend on Redis being started. If you use a database server, you should add additional dependencies. !!! note @@ -437,18 +417,15 @@ are released, dependency support is confirmed, etc. !!! warning - If celery won't start (check with + If Celery won't start, check `sudo systemctl status paperless-task-queue.service` for - paperless-task-queue.service and paperless-scheduler.service - ) you need to change the path in the files. Example: + `paperless-task-queue.service` and `paperless-scheduler.service`. + You may need to change the path in the files. Example: `ExecStart=/opt/paperless/.local/bin/celery --app paperless worker --loglevel INFO` -12. Optional: Install a samba server and make the consumption folder - available as a network share. - -13. Configure ImageMagick to allow processing of PDF documents. Most +12. Configure ImageMagick to allow processing of PDF documents. Most distributions have this disabled by default, since PDF documents can - contain malware. If you don't do this, paperless will fall back to + contain malware. If you don't do this, Paperless-ngx will fall back to Ghostscript for certain steps such as thumbnail generation. Edit `/etc/ImageMagick-6/policy.xml` and adjust @@ -463,32 +440,38 @@ are released, dependency support is confirmed, etc. ``` -14. Optional: Install the - [jbig2enc](https://ocrmypdf.readthedocs.io/en/latest/jbig2.html) - encoder. This will reduce the size of generated PDF documents. - You'll most likely need to compile this by yourself, because this - software has been patented until around 2017 and binary packages are - not available for most distributions. +**Optional: Install the [jbig2enc](https://ocrmypdf.readthedocs.io/en/latest/jbig2.html) encoder.** +This will reduce the size of generated PDF documents. You'll most likely need to compile this yourself, because this +software has been patented until around 2017 and binary packages are not available for most distributions. -15. Optional: If using the NLTK machine learning processing (see - [`PAPERLESS_ENABLE_NLTK`](configuration.md#PAPERLESS_ENABLE_NLTK) for details), - download the NLTK data for the Snowball - Stemmer, Stopwords and Punkt tokenizer to `/usr/share/nltk_data`. Refer to the [NLTK - instructions](https://www.nltk.org/data.html) for details on how to - download the data. +**Optional: download the NLTK data** +If using the NLTK machine-learning processing (see [`PAPERLESS_ENABLE_NLTK`](configuration.md#PAPERLESS_ENABLE_NLTK) for details), +download the NLTK data for the Snowball Stemmer, Stopwords and Punkt tokenizer to `/usr/share/nltk_data`. Refer to the [NLTK +instructions](https://www.nltk.org/data.html) for details on how to download the data. -# Migrating to Paperless-ngx +#### After installation -Migration is possible both from Paperless-ng or directly from the -'original' Paperless. +Your Paperless-ngx instance should now be accessible at `http://localhost:8000` (or similar, depending on your configuration). +When you first access the web interface you will be prompted to create a [superuser](usage.md#superusers) account. -## Migrating from Paperless-ng +## Build the Docker image yourself {#docker_build data-toc-label="Building the Docker image"} -Paperless-ngx is meant to be a drop-in replacement for Paperless-ng and -thus upgrading should be trivial for most users, especially when using -docker. However, as with any major change, it is recommended to take a +Building the Docker image yourself is typically used for development, but it can also be used for production +if you want to customize the image. See [Building the Docker image](development.md#docker_build) in the +development documentation. + +## Migrating to Paperless-ngx + +You can migrate to Paperless-ngx from Paperless-ng or from the original +Paperless project. + +

Migrating from Paperless-ng

+ +Paperless-ngx is meant to be a drop-in replacement for Paperless-ng, and +upgrading should be trivial for most users, especially when using +Docker. However, as with any major change, it is recommended to take a full backup first. Once you are ready, simply change the docker image to -point to the new source. E.g. if using Docker Compose, edit +point to the new source. For example, if using Docker Compose, edit `docker-compose.yml` and change: ``` @@ -501,66 +484,65 @@ to image: ghcr.io/paperless-ngx/paperless-ngx:latest ``` -and then run `docker compose up -d` which will pull the new image -recreate the container. That's it! +and then run `docker compose up -d`, which will pull the new image and +recreate the container. That's it. Users who installed with the bare-metal route should also update their Git clone to point to `https://github.com/paperless-ngx/paperless-ngx`, -e.g. using the command +for example using: `git remote set-url origin https://github.com/paperless-ngx/paperless-ngx` and then pull the latest version. -## Migrating from Paperless +

Migrating from Paperless

-At its core, paperless-ngx is still paperless and fully compatible. +At its core, Paperless-ngx is still Paperless and fully compatible. However, some things have changed under the hood, so you need to adapt -your setup depending on how you installed paperless. +your setup depending on how you installed Paperless. -This setup describes how to update an existing paperless Docker -installation. The important things to keep in mind are as follows: +This section describes how to update an existing Paperless Docker +installation. Keep these points in mind: - Read the [changelog](changelog.md) and take note of breaking changes. -- You should decide if you want to stick with SQLite or want to - migrate your database to PostgreSQL. See [documentation](#sqlite_to_psql) - for details on - how to move your data from SQLite to PostgreSQL. Both work fine with - paperless. However, if you already have a database server running - for other services, you might as well use it for paperless as well. -- The task scheduler of paperless, which is used to execute periodic +- Decide whether to stay on SQLite or migrate to PostgreSQL. + See [documentation](#sqlite_to_psql) for details on moving data + from SQLite to PostgreSQL. Both work fine with + Paperless. However, if you already have a database server running + for other services, you might as well use it for Paperless as well. +- The task scheduler of Paperless, which is used to execute periodic tasks such as email checking and maintenance, requires a - [redis](https://redis.io/) message broker instance. The + [Redis](https://redis.io/) message broker instance. The Docker Compose route takes care of that. - The layout of the folder structure for your documents and data - remains the same, so you can just plug your old docker volumes into + remains the same, so you can plug your old Docker volumes into paperless-ngx and expect it to find everything where it should be. -Migration to paperless-ngx is then performed in a few simple steps: +Migration to Paperless-ngx is then performed in a few simple steps: -1. Stop paperless. +1. Stop Paperless. ```bash cd /path/to/current/paperless docker compose down ``` -2. Do a backup for two purposes: If something goes wrong, you still - have your data. Second, if you don't like paperless-ngx, you can - switch back to paperless. +2. Create a backup for two reasons: if something goes wrong, you still + have your data; and if you don't like paperless-ngx, you can + switch back to Paperless. -3. Download the latest release of paperless-ngx. You can either go with +3. Download the latest release of Paperless-ngx. You can either use the Docker Compose files from [here](https://github.com/paperless-ngx/paperless-ngx/tree/main/docker/compose) or clone the repository to build the image yourself (see - [above](#docker_build)). You can - either replace your current paperless folder or put paperless-ngx in + [development docs](development.md#docker_build)). You can either replace your current paperless + folder or put Paperless-ngx in a different location. !!! warning Paperless-ngx includes a `.env` file. This will set the project name - for docker compose to `paperless`, which will also define the name - of the volumes by paperless-ngx. However, if you experience that + for Docker Compose to `paperless`, which will also define the + volume names created by Paperless-ngx. However, if you notice that paperless-ngx is not using your old paperless volumes, verify the names of your volumes with @@ -576,10 +558,10 @@ Migration to paperless-ngx is then performed in a few simple steps: after you migrated your existing SQLite database. 5. Adjust `docker-compose.yml` and `docker-compose.env` to your needs. - See [Docker setup](#docker) details on - which edits are advised. + See [Docker setup](#docker) for details on + which edits are recommended. -6. [Update paperless.](administration.md#updating) +6. Follow the update procedure in [Update paperless](administration.md#updating). 7. In order to find your existing documents with the new search feature, you need to invoke a one-time operation that will create @@ -590,136 +572,99 @@ Migration to paperless-ngx is then performed in a few simple steps: ``` This will migrate your database and create the search index. After - that, paperless will take care of maintaining the index by itself. + that, Paperless-ngx will maintain the index automatically. -8. Start paperless-ngx. +8. Start Paperless-ngx. ```bash docker compose up -d ``` - This will run paperless in the background and automatically start it + This will run Paperless-ngx in the background and automatically start it on system boot. -9. Paperless installed a permanent redirect to `admin/` in your +9. Paperless may have installed a permanent redirect to `admin/` in your browser. This redirect is still in place and prevents access to the - new UI. Clear your browsing cache in order to fix this. + new UI. Clear your browser cache to fix this. 10. Optionally, follow the instructions below to migrate your existing data to PostgreSQL. -## Migrating from LinuxServer.io Docker Image +

Migrating from LinuxServer.io Docker Image

-As with any upgrades and large changes, it is highly recommended to +As with any upgrade or large change, it is highly recommended to create a backup before starting. This assumes the image was running using Docker Compose, but the instructions are translatable to Docker commands as well. -1. Stop and remove the paperless container -2. If using an external database, stop the container -3. Update Redis configuration +1. Stop and remove the Paperless container. +2. If using an external database, stop that container. +3. Update Redis configuration. 1. If `REDIS_URL` is already set, change it to [`PAPERLESS_REDIS`](configuration.md#PAPERLESS_REDIS) and continue to step 4. - 1. Otherwise, in the `docker-compose.yml` add a new service for - Redis, following [the example compose + 1. Otherwise, add a new Redis service in `docker-compose.yml`, + following [the example compose files](https://github.com/paperless-ngx/paperless-ngx/tree/main/docker/compose) 1. Set the environment variable [`PAPERLESS_REDIS`](configuration.md#PAPERLESS_REDIS) so it points to - the new Redis container + the new Redis container. -4. Update user mapping +4. Update user mapping. - 1. If set, change the environment variable `PUID` to `USERMAP_UID` + 1. If set, change the environment variable `PUID` to `USERMAP_UID`. - 1. If set, change the environment variable `PGID` to `USERMAP_GID` + 1. If set, change the environment variable `PGID` to `USERMAP_GID`. -5. Update configuration paths +5. Update configuration paths. - 1. Set the environment variable [`PAPERLESS_DATA_DIR`](configuration.md#PAPERLESS_DATA_DIR) to `/config` + 1. Set the environment variable [`PAPERLESS_DATA_DIR`](configuration.md#PAPERLESS_DATA_DIR) to `/config`. -6. Update media paths +6. Update media paths. 1. Set the environment variable [`PAPERLESS_MEDIA_ROOT`](configuration.md#PAPERLESS_MEDIA_ROOT) to - `/data/media` + `/data/media`. -7. Update timezone +7. Update timezone. 1. Set the environment variable [`PAPERLESS_TIME_ZONE`](configuration.md#PAPERLESS_TIME_ZONE) to the same - value as `TZ` + value as `TZ`. -8. Modify the `image:` to point to +8. Modify `image:` to point to `ghcr.io/paperless-ngx/paperless-ngx:latest` or a specific version if preferred. 9. Start the containers as before, using `docker compose`. -## Moving data from SQLite to PostgreSQL or MySQL/MariaDB {#sqlite_to_psql} +## Running Paperless-ngx on less powerful devices {#less-powerful-devices data-toc-label="Less Powerful Devices"} -The best way to migrate between database types is to perform an [export](administration.md#exporter) and then -[import](administration.md#importer) into a clean installation of Paperless-ngx. - -## Moving back to Paperless - -Lets say you migrated to Paperless-ngx and used it for a while, but -decided that you don't like it and want to move back (If you do, send -me a mail about what part you didn't like!), you can totally do that -with a few simple steps. - -Paperless-ngx modified the database schema slightly, however, these -changes can be reverted while keeping your current data, so that your -current data will be compatible with original Paperless. Thumbnails -were also changed from PNG to WEBP format and will need to be -re-generated. - -Execute this: - -```shell-session -$ cd /path/to/paperless -$ docker compose run --rm webserver migrate documents 0023 -``` - -Or without docker: - -```shell-session -$ cd /path/to/paperless/src -$ python3 manage.py migrate documents 0023 -``` - -After regenerating thumbnails, you'll need to clear your cookies -(Paperless-ngx comes with updated dependencies that do cookie-processing -differently) and probably your cache as well. - -# Considerations for less powerful devices {#less-powerful-devices} - -Paperless runs on Raspberry Pi. However, some things are rather slow on -the Pi and configuring some options in paperless can help improve -performance immensely: +Paperless runs on Raspberry Pi. Some tasks can be slow on lower-powered +hardware, but a few settings can improve performance: - Stick with SQLite to save some resources. See [troubleshooting](troubleshooting.md#log-reports-creating-paperlesstask-failed) if you encounter issues with SQLite locking. - If you do not need the filesystem-based consumer, consider disabling it entirely by setting [`PAPERLESS_CONSUMER_DISABLE`](configuration.md#PAPERLESS_CONSUMER_DISABLE) to `true`. -- Consider setting [`PAPERLESS_OCR_PAGES`](configuration.md#PAPERLESS_OCR_PAGES) to 1, so that paperless will - only OCR the first page of your documents. In most cases, this page +- Consider setting [`PAPERLESS_OCR_PAGES`](configuration.md#PAPERLESS_OCR_PAGES) to 1, so that Paperless + OCRs only the first page of your documents. In most cases, this page contains enough information to be able to find it. - [`PAPERLESS_TASK_WORKERS`](configuration.md#PAPERLESS_TASK_WORKERS) and [`PAPERLESS_THREADS_PER_WORKER`](configuration.md#PAPERLESS_THREADS_PER_WORKER) are configured to use all cores. The Raspberry Pi models 3 and up have 4 - cores, meaning that paperless will use 2 workers and 2 threads per + cores, meaning that Paperless will use 2 workers and 2 threads per worker. This may result in sluggish response times during consumption, so you might want to lower these settings (example: 2 workers and 1 thread to always have some computing power left for other tasks). - Keep [`PAPERLESS_OCR_MODE`](configuration.md#PAPERLESS_OCR_MODE) at its default value `skip` and consider - OCR'ing your documents before feeding them into paperless. Some + OCRing your documents before feeding them into Paperless. Some scanners are able to do this! - Set [`PAPERLESS_OCR_SKIP_ARCHIVE_FILE`](configuration.md#PAPERLESS_OCR_SKIP_ARCHIVE_FILE) to `with_text` to skip archive - file generation for already ocr'ed documents, or `always` to skip it + file generation for already OCRed documents, or `always` to skip it for all documents. - If you want to perform OCR on the device, consider using `PAPERLESS_OCR_CLEAN=none`. This will speed up OCR times and use less memory at the expense of slightly worse OCR results. -- If using docker, consider setting [`PAPERLESS_WEBSERVER_WORKERS`](configuration.md#PAPERLESS_WEBSERVER_WORKERS) to 1. This will save some memory. +- If using Docker, consider setting [`PAPERLESS_WEBSERVER_WORKERS`](configuration.md#PAPERLESS_WEBSERVER_WORKERS) to 1. This will save some memory. - Consider setting [`PAPERLESS_ENABLE_NLTK`](configuration.md#PAPERLESS_ENABLE_NLTK) to false, to disable the more advanced language processing, which can take more memory and processing time. @@ -731,17 +676,19 @@ For details, refer to [configuration](configuration.md). Updating the [automatic matching algorithm](advanced_usage.md#automatic-matching) takes quite a bit of time. However, the update mechanism checks if your data has changed before doing the heavy lifting. If you - experience the algorithm taking too much cpu time, consider changing the + experience the algorithm taking too much CPU time, consider changing the schedule in the admin interface to daily. You can also manually invoke the task by changing the date and time of the next run to today/now. The actual matching of the algorithm is fast and works on Raspberry Pi as well as on any other device. -# Using nginx as a reverse proxy {#nginx} +## Additional considerations -Please see [the wiki](https://github.com/paperless-ngx/paperless-ngx/wiki/Using-a-Reverse-Proxy-with-Paperless-ngx#nginx) for user-maintained documentation of using nginx with Paperless-ngx. +**Using a reverse proxy with Paperless-ngx** -# Enhancing security {#security} +Please see [the wiki](https://github.com/paperless-ngx/paperless-ngx/wiki/Using-a-Reverse-Proxy-with-Paperless-ngx#nginx) for user-maintained documentation on using nginx with Paperless-ngx. -Please see [the wiki](https://github.com/paperless-ngx/paperless-ngx/wiki/Using-Security-Tools-with-Paperless-ngx) for user-maintained documentation of how to configure security tools like Fail2ban with Paperless-ngx. +**Enhancing security** + +Please see [the wiki](https://github.com/paperless-ngx/paperless-ngx/wiki/Using-Security-Tools-with-Paperless-ngx) for user-maintained documentation on configuring security tools like Fail2ban with Paperless-ngx. diff --git a/docs/usage.md b/docs/usage.md index 3a4f56bca..673e552f5 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -348,6 +348,11 @@ permissions can be granted to limit access to certain parts of the UI (and corre Superusers can access all parts of the front and backend application as well as any and all objects. Superuser status can only be granted by another superuser. +!!! tip + + Because superuser accounts can see all objects and documents, you may want to use a regular account for day-to-day use. Additional superuser accounts can + be created via [cli](administration.md#create-superuser) or granted superuser status from an existing superuser account. + #### Admin Status Admin status (Django 'staff status') grants access to viewing the paperless logs and the system status dialog From 13e07844fee16d9f650f0f9e50fb5eb0a3323fa7 Mon Sep 17 00:00:00 2001 From: shamoon <4887959+shamoon@users.noreply.github.com> Date: Wed, 25 Feb 2026 17:25:36 -0800 Subject: [PATCH 02/40] Fix: separate displayed and API collection sizes for tags (#12170) --- .../management-list/management-list.component.html | 4 ++-- .../management-list.component.spec.ts | 5 +++-- .../management-list/management-list.component.ts | 13 ++++++++++++- .../manage/tag-list/tag-list.component.ts | 11 +++++++++++ 4 files changed, 28 insertions(+), 5 deletions(-) diff --git a/src-ui/src/app/components/manage/management-list/management-list.component.html b/src-ui/src/app/components/manage/management-list/management-list.component.html index 8fac6f44f..91dcc2592 100644 --- a/src-ui/src/app/components/manage/management-list/management-list.component.html +++ b/src-ui/src/app/components/manage/management-list/management-list.component.html @@ -62,9 +62,9 @@ @if (!loading) {
- @if (collectionSize > 0) { + @if (displayCollectionSize > 0) {
- {collectionSize, plural, =1 {One {{typeName}}} other {{{collectionSize || 0}} total {{typeNamePlural}}}} + {displayCollectionSize, plural, =1 {One {{typeName}}} other {{{displayCollectionSize || 0}} total {{typeNamePlural}}}} @if (selectedObjects.size > 0) {  ({{selectedObjects.size}} selected) } diff --git a/src-ui/src/app/components/manage/management-list/management-list.component.spec.ts b/src-ui/src/app/components/manage/management-list/management-list.component.spec.ts index 86f0f0469..fb0ad0914 100644 --- a/src-ui/src/app/components/manage/management-list/management-list.component.spec.ts +++ b/src-ui/src/app/components/manage/management-list/management-list.component.spec.ts @@ -229,7 +229,7 @@ describe('ManagementListComponent', () => { expect(reloadSpy).toHaveBeenCalled() }) - it('should use the all list length for collection size when provided', fakeAsync(() => { + it('should use API count for pagination and all ids for displayed total', fakeAsync(() => { jest.spyOn(tagService, 'listFiltered').mockReturnValueOnce( of({ count: 1, @@ -241,7 +241,8 @@ describe('ManagementListComponent', () => { component.reloadData() tick(100) - expect(component.collectionSize).toBe(3) + expect(component.collectionSize).toBe(1) + expect(component.displayCollectionSize).toBe(3) })) it('should support quick filter for objects', () => { diff --git a/src-ui/src/app/components/manage/management-list/management-list.component.ts b/src-ui/src/app/components/manage/management-list/management-list.component.ts index 27913ea7d..8c41f1c45 100644 --- a/src-ui/src/app/components/manage/management-list/management-list.component.ts +++ b/src-ui/src/app/components/manage/management-list/management-list.component.ts @@ -23,6 +23,7 @@ import { MatchingModel, } from 'src/app/data/matching-model' import { ObjectWithPermissions } from 'src/app/data/object-with-permissions' +import { Results } from 'src/app/data/results' import { SortableDirective, SortEvent, @@ -88,6 +89,7 @@ export abstract class ManagementListComponent public page = 1 public collectionSize = 0 + public displayCollectionSize = 0 public sortField: string public sortReverse: boolean @@ -141,6 +143,14 @@ export abstract class ManagementListComponent return data } + protected getCollectionSize(results: Results): number { + return results.all?.length ?? results.count + } + + protected getDisplayCollectionSize(results: Results): number { + return this.getCollectionSize(results) + } + getDocumentCount(object: MatchingModel): number { return ( object.document_count ?? @@ -171,7 +181,8 @@ export abstract class ManagementListComponent tap((c) => { this.unfilteredData = c.results this.data = this.filterData(c.results) - this.collectionSize = c.all?.length ?? c.count + this.collectionSize = this.getCollectionSize(c) + this.displayCollectionSize = this.getDisplayCollectionSize(c) }), delay(100) ) diff --git a/src-ui/src/app/components/manage/tag-list/tag-list.component.ts b/src-ui/src/app/components/manage/tag-list/tag-list.component.ts index 0ba0a0855..3749a147f 100644 --- a/src-ui/src/app/components/manage/tag-list/tag-list.component.ts +++ b/src-ui/src/app/components/manage/tag-list/tag-list.component.ts @@ -7,6 +7,7 @@ import { } from '@ng-bootstrap/ng-bootstrap' import { NgxBootstrapIconsModule } from 'ngx-bootstrap-icons' import { FILTER_HAS_TAGS_ALL } from 'src/app/data/filter-rule-type' +import { Results } from 'src/app/data/results' import { Tag } from 'src/app/data/tag' import { IfPermissionsDirective } from 'src/app/directives/if-permissions.directive' import { SortableDirective } from 'src/app/directives/sortable.directive' @@ -77,6 +78,16 @@ export class TagListComponent extends ManagementListComponent { return data.filter((tag) => !tag.parent || !availableIds.has(tag.parent)) } + protected override getCollectionSize(results: Results): number { + // Tag list pages are requested with is_root=true (when unfiltered), so + // pagination must follow root count even though `all` includes descendants + return results.count + } + + protected override getDisplayCollectionSize(results: Results): number { + return super.getCollectionSize(results) + } + protected override getSelectableIDs(tags: Tag[]): number[] { const ids: number[] = [] for (const tag of tags.filter(Boolean)) { From 9601b3d597f113c36bdeac6f43bda87ae6cdde26 Mon Sep 17 00:00:00 2001 From: shamoon <4887959+shamoon@users.noreply.github.com> Date: Thu, 26 Feb 2026 10:03:54 -0800 Subject: [PATCH 03/40] Fixhancement: config option reset (#12176) --- .../admin/config/config.component.html | 17 +++++++++++------ .../admin/config/config.component.spec.ts | 14 ++++++++++++++ .../components/admin/config/config.component.ts | 8 ++++++++ 3 files changed, 33 insertions(+), 6 deletions(-) diff --git a/src-ui/src/app/components/admin/config/config.component.html b/src-ui/src/app/components/admin/config/config.component.html index e1d7340a6..c103af166 100644 --- a/src-ui/src/app/components/admin/config/config.component.html +++ b/src-ui/src/app/components/admin/config/config.component.html @@ -19,13 +19,18 @@
-
-
- {{option.title}} - - - +
+
+ {{option.title}}
+ + + + @if (isSet(option.key)) { + + }
@switch (option.type) { diff --git a/src-ui/src/app/components/admin/config/config.component.spec.ts b/src-ui/src/app/components/admin/config/config.component.spec.ts index 079bd1420..f4f4799a6 100644 --- a/src-ui/src/app/components/admin/config/config.component.spec.ts +++ b/src-ui/src/app/components/admin/config/config.component.spec.ts @@ -144,4 +144,18 @@ describe('ConfigComponent', () => { component.uploadFile(new File([], 'test.png'), 'app_logo') expect(initSpy).toHaveBeenCalled() }) + + it('should reset option to null', () => { + component.configForm.patchValue({ output_type: OutputTypeConfig.PDF_A }) + expect(component.isSet('output_type')).toBeTruthy() + component.resetOption('output_type') + expect(component.configForm.get('output_type').value).toBeNull() + expect(component.isSet('output_type')).toBeFalsy() + component.configForm.patchValue({ app_title: 'Test Title' }) + component.resetOption('app_title') + expect(component.configForm.get('app_title').value).toBeNull() + component.configForm.patchValue({ barcodes_enabled: true }) + component.resetOption('barcodes_enabled') + expect(component.configForm.get('barcodes_enabled').value).toBeNull() + }) }) diff --git a/src-ui/src/app/components/admin/config/config.component.ts b/src-ui/src/app/components/admin/config/config.component.ts index eee617310..44e482e75 100644 --- a/src-ui/src/app/components/admin/config/config.component.ts +++ b/src-ui/src/app/components/admin/config/config.component.ts @@ -208,4 +208,12 @@ export class ConfigComponent }, }) } + + public isSet(key: string): boolean { + return this.configForm.get(key).value != null + } + + public resetOption(key: string) { + this.configForm.get(key).setValue(null) + } } From 89d3a53603d3f0d58951f98b845d9c0b8d80f812 Mon Sep 17 00:00:00 2001 From: shamoon <4887959+shamoon@users.noreply.github.com> Date: Thu, 26 Feb 2026 23:26:35 -0800 Subject: [PATCH 04/40] Documentation: note GHSAs in changelog --- docs/changelog.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/docs/changelog.md b/docs/changelog.md index fb4229e5a..074a117ea 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -2,8 +2,16 @@ ## paperless-ngx 2.20.8 +### Security + +- Resolve [GHSA-7qqc-wrcw-2fj9](https://github.com/paperless-ngx/paperless-ngx/security/advisories/GHSA-7qqc-wrcw-2fj9) + ## paperless-ngx 2.20.7 +### Security + +- Resolve [GHSA-x395-6h48-wr8v](https://github.com/paperless-ngx/paperless-ngx/security/advisories/GHSA-x395-6h48-wr8v) + ### Bug Fixes - Performance fix: use subqueries to improve object retrieval in large installs [@shamoon](https://github.com/shamoon) ([#11950](https://github.com/paperless-ngx/paperless-ngx/pull/11950)) @@ -22,6 +30,10 @@ ## paperless-ngx 2.20.6 +### Security + +- Resolve [GHSA-jqwv-hx7q-fxh3](https://github.com/paperless-ngx/paperless-ngx/security/advisories/GHSA-jqwv-hx7q-fxh3) and [GHSA-w47q-3m69-84v8](https://github.com/paperless-ngx/paperless-ngx/security/advisories/GHSA-w47q-3m69-84v8) + ### Bug Fixes - Fix: extract all ids for nested tags [@shamoon](https://github.com/shamoon) ([#11888](https://github.com/paperless-ngx/paperless-ngx/pull/11888)) From 5988d5896b2604409222decd48bef47e5d4f592d Mon Sep 17 00:00:00 2001 From: Trenton H <797416+stumpylog@users.noreply.github.com> Date: Fri, 27 Feb 2026 14:37:26 -0800 Subject: [PATCH 05/40] Breaking: Refactor advanced database settings to allow more user configuration (#12165) --- .editorconfig | 3 + docs/administration.md | 4 + docs/configuration.md | 185 ++++---- docs/{migration.md => migration-v3.md} | 55 +++ docs/setup.md | 3 +- src/paperless/checks.py | 40 ++ .../{settings.py => settings/__init__.py} | 81 +--- src/paperless/settings/custom.py | 122 ++++++ src/paperless/settings/parsers.py | 192 ++++++++ src/paperless/tests/settings/__init__.py | 0 .../tests/settings/test_custom_parsers.py | 266 +++++++++++ .../settings/test_environment_parsers.py | 414 ++++++++++++++++++ src/paperless/tests/test_checks.py | 158 +++++++ src/paperless/tests/test_settings.py | 59 --- zensical.toml | 5 +- 15 files changed, 1373 insertions(+), 214 deletions(-) rename docs/{migration.md => migration-v3.md} (61%) rename src/paperless/{settings.py => settings/__init__.py} (93%) create mode 100644 src/paperless/settings/custom.py create mode 100644 src/paperless/settings/parsers.py create mode 100644 src/paperless/tests/settings/__init__.py create mode 100644 src/paperless/tests/settings/test_custom_parsers.py create mode 100644 src/paperless/tests/settings/test_environment_parsers.py diff --git a/.editorconfig b/.editorconfig index a8b21f510..eecb25efb 100644 --- a/.editorconfig +++ b/.editorconfig @@ -39,3 +39,6 @@ max_line_length = off [Dockerfile*] indent_style = space + +[*.toml] +indent_style = space diff --git a/docs/administration.md b/docs/administration.md index 7e5ca2d41..511015b6b 100644 --- a/docs/administration.md +++ b/docs/administration.md @@ -62,6 +62,10 @@ copies you created in the steps above. ## Updating Paperless {#updating} +!!! warning + + Please review the [migration instructions](migration-v3.md) before upgrading Paperless-ngx to v3.0, it includes some breaking changes that require manual intervention before upgrading. + ### Docker Route {#docker-updating} If a new release of paperless-ngx is available, upgrading depends on how diff --git a/docs/configuration.md b/docs/configuration.md index 10b20fe9a..2fc651a76 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -51,137 +51,172 @@ matcher. ### Database By default, Paperless uses **SQLite** with a database stored at `data/db.sqlite3`. -To switch to **PostgreSQL** or **MariaDB**, set [`PAPERLESS_DBHOST`](#PAPERLESS_DBHOST) and optionally configure other -database-related environment variables. +For multi-user or higher-throughput deployments, **PostgreSQL** (recommended) or +**MariaDB** can be used instead by setting [`PAPERLESS_DBENGINE`](#PAPERLESS_DBENGINE) +and the relevant connection variables. + +#### [`PAPERLESS_DBENGINE=`](#PAPERLESS_DBENGINE) {#PAPERLESS_DBENGINE} + +: Specifies the database engine to use. Accepted values are `sqlite`, `postgresql`, +and `mariadb`. + + Defaults to `sqlite` if not set. + + PostgreSQL and MariaDB both require [`PAPERLESS_DBHOST`](#PAPERLESS_DBHOST) to be + set. SQLite does not use any other connection variables; the database file is always + located at `/db.sqlite3`. + + !!! warning + Using MariaDB comes with some caveats. + See [MySQL Caveats](advanced_usage.md#mysql-caveats). #### [`PAPERLESS_DBHOST=`](#PAPERLESS_DBHOST) {#PAPERLESS_DBHOST} -: If unset, Paperless uses **SQLite** by default. - - Set `PAPERLESS_DBHOST` to switch to PostgreSQL or MariaDB instead. - -#### [`PAPERLESS_DBENGINE=`](#PAPERLESS_DBENGINE) {#PAPERLESS_DBENGINE} - -: Optional. Specifies the database engine to use when connecting to a remote database. -Available options are `postgresql` and `mariadb`. - - Defaults to `postgresql` if `PAPERLESS_DBHOST` is set. - - !!! warning - - Using MariaDB comes with some caveats. See [MySQL Caveats](advanced_usage.md#mysql-caveats). +: Hostname of the PostgreSQL or MariaDB database server. Required when +`PAPERLESS_DBENGINE` is `postgresql` or `mariadb`. #### [`PAPERLESS_DBPORT=`](#PAPERLESS_DBPORT) {#PAPERLESS_DBPORT} : Port to use when connecting to PostgreSQL or MariaDB. - Default is `5432` for PostgreSQL and `3306` for MariaDB. + Defaults to `5432` for PostgreSQL and `3306` for MariaDB. #### [`PAPERLESS_DBNAME=`](#PAPERLESS_DBNAME) {#PAPERLESS_DBNAME} -: Name of the database to connect to when using PostgreSQL or MariaDB. +: Name of the PostgreSQL or MariaDB database to connect to. - Defaults to "paperless". + Defaults to `paperless`. -#### [`PAPERLESS_DBUSER=`](#PAPERLESS_DBUSER) {#PAPERLESS_DBUSER} +#### [`PAPERLESS_DBUSER=`](#PAPERLESS_DBUSER) {#PAPERLESS_DBUSER} : Username for authenticating with the PostgreSQL or MariaDB database. - Defaults to "paperless". + Defaults to `paperless`. #### [`PAPERLESS_DBPASS=`](#PAPERLESS_DBPASS) {#PAPERLESS_DBPASS} : Password for the PostgreSQL or MariaDB database user. - Defaults to "paperless". + Defaults to `paperless`. -#### [`PAPERLESS_DBSSLMODE=`](#PAPERLESS_DBSSLMODE) {#PAPERLESS_DBSSLMODE} +#### [`PAPERLESS_DB_OPTIONS=`](#PAPERLESS_DB_OPTIONS) {#PAPERLESS_DB_OPTIONS} -: SSL mode to use when connecting to PostgreSQL or MariaDB. +: Advanced database connection options as a semicolon-delimited key-value string. +Keys and values are separated by `=`. Dot-notation produces nested option +dictionaries; for example, `pool.max_size=20` sets +`OPTIONS["pool"]["max_size"] = 20`. - See [the official documentation about - sslmode for PostgreSQL](https://www.postgresql.org/docs/current/libpq-ssl.html). + Options specified here are merged over the engine defaults. Unrecognised keys + are passed through to the underlying database driver without validation, so a + typo will be silently ignored rather than producing an error. - See [the official documentation about - sslmode for MySQL and MariaDB](https://dev.mysql.com/doc/refman/8.0/en/connection-options.html#option_general_ssl-mode). + Refer to your database driver's documentation for the full set of accepted keys: - *Note*: SSL mode values differ between PostgreSQL and MariaDB. + - PostgreSQL: [libpq connection parameters](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-PARAMKEYWORDS) + - MariaDB: [MariaDB Connector/Python](https://mariadb.com/kb/en/mariadb-connector-python/) + - SQLite: [SQLite PRAGMA statements](https://www.sqlite.org/pragma.html) - Default is `prefer` for PostgreSQL and `PREFERRED` for MariaDB. + !!! note "PostgreSQL connection pooling" -#### [`PAPERLESS_DBSSLROOTCERT=`](#PAPERLESS_DBSSLROOTCERT) {#PAPERLESS_DBSSLROOTCERT} + Pool size is controlled via `pool.min_size` and `pool.max_size`. When + configuring pooling, ensure your PostgreSQL `max_connections` is large enough + to handle all pool connections across all workers: + `(web_workers + celery_workers) * pool.max_size + safety_margin`. -: Path to the SSL root certificate used to verify the database server. + **Examples:** - See [the official documentation about - sslmode for PostgreSQL](https://www.postgresql.org/docs/current/libpq-ssl.html). - Changes the location of `root.crt`. + ```bash title="PostgreSQL: require SSL, set a custom CA certificate, and limit the pool size" + PAPERLESS_DB_OPTIONS="sslmode=require;sslrootcert=/certs/ca.pem;pool.max_size=5" + ``` - See [the official documentation about - sslmode for MySQL and MariaDB](https://dev.mysql.com/doc/refman/8.0/en/connection-options.html#option_general_ssl-ca). + ```bash title="MariaDB: require SSL with a custom CA certificate" + PAPERLESS_DB_OPTIONS="ssl_mode=REQUIRED;ssl.ca=/certs/ca.pem" + ``` - Defaults to unset, using the standard location in the home directory. + ```bash title="SQLite: set a busy timeout of 30 seconds" + # PostgreSQL: set a connection timeout + PAPERLESS_DB_OPTIONS="connect_timeout=10" + ``` -#### [`PAPERLESS_DBSSLCERT=`](#PAPERLESS_DBSSLCERT) {#PAPERLESS_DBSSLCERT} +#### ~~[`PAPERLESS_DBSSLMODE`](#PAPERLESS_DBSSLMODE)~~ {#PAPERLESS_DBSSLMODE} -: Path to the client SSL certificate used when connecting securely. +!!! failure "Removed in v3" - See [the official documentation about - sslmode for PostgreSQL](https://www.postgresql.org/docs/current/libpq-ssl.html). + Use [`PAPERLESS_DB_OPTIONS`](#PAPERLESS_DB_OPTIONS) instead. - See [the official documentation about - sslmode for MySQL and MariaDB](https://dev.mysql.com/doc/refman/8.0/en/connection-options.html#option_general_ssl-cert). + ```bash title="PostgreSQL" + PAPERLESS_DB_OPTIONS="sslmode=require" + ``` - Changes the location of `postgresql.crt`. + ```bash title="MariaDB" + PAPERLESS_DB_OPTIONS="ssl_mode=REQUIRED" + ``` - Defaults to unset, using the standard location in the home directory. +#### ~~[`PAPERLESS_DBSSLROOTCERT`](#PAPERLESS_DBSSLROOTCERT)~~ {#PAPERLESS_DBSSLROOTCERT} -#### [`PAPERLESS_DBSSLKEY=`](#PAPERLESS_DBSSLKEY) {#PAPERLESS_DBSSLKEY} +!!! failure "Removed in v3" -: Path to the client SSL private key used when connecting securely. + Use [`PAPERLESS_DB_OPTIONS`](#PAPERLESS_DB_OPTIONS) instead. - See [the official documentation about - sslmode for PostgreSQL](https://www.postgresql.org/docs/current/libpq-ssl.html). + ```bash title="PostgreSQL" + PAPERLESS_DB_OPTIONS="sslrootcert=/path/to/ca.pem" + ``` - See [the official documentation about - sslmode for MySQL and MariaDB](https://dev.mysql.com/doc/refman/8.0/en/connection-options.html#option_general_ssl-key). + ```bash title="MariaDB" + PAPERLESS_DB_OPTIONS="ssl.ca=/path/to/ca.pem" + ``` - Changes the location of `postgresql.key`. +#### ~~[`PAPERLESS_DBSSLCERT`](#PAPERLESS_DBSSLCERT)~~ {#PAPERLESS_DBSSLCERT} - Defaults to unset, using the standard location in the home directory. +!!! failure "Removed in v3" -#### [`PAPERLESS_DB_TIMEOUT=`](#PAPERLESS_DB_TIMEOUT) {#PAPERLESS_DB_TIMEOUT} + Use [`PAPERLESS_DB_OPTIONS`](#PAPERLESS_DB_OPTIONS) instead. -: Sets how long a database connection should wait before timing out. + ```bash title="PostgreSQL" + PAPERLESS_DB_OPTIONS="sslcert=/path/to/client.crt" + ``` - For SQLite, this sets how long to wait if the database is locked. - For PostgreSQL or MariaDB, this sets the connection timeout. + ```bash title="MariaDB" + PAPERLESS_DB_OPTIONS="ssl.cert=/path/to/client.crt" + ``` - Defaults to unset, which uses Django’s built-in defaults. +#### ~~[`PAPERLESS_DBSSLKEY`](#PAPERLESS_DBSSLKEY)~~ {#PAPERLESS_DBSSLKEY} -#### [`PAPERLESS_DB_POOLSIZE=`](#PAPERLESS_DB_POOLSIZE) {#PAPERLESS_DB_POOLSIZE} +!!! failure "Removed in v3" -: Defines the maximum number of database connections to keep in the pool. + Use [`PAPERLESS_DB_OPTIONS`](#PAPERLESS_DB_OPTIONS) instead. - Only applies to PostgreSQL. This setting is ignored for other database engines. + ```bash title="PostgreSQL" + PAPERLESS_DB_OPTIONS="sslkey=/path/to/client.key" + ``` - The value must be greater than or equal to 1 to be used. - Defaults to unset, which disables connection pooling. + ```bash title="MariaDB" + PAPERLESS_DB_OPTIONS="ssl.key=/path/to/client.key" + ``` - !!! note +#### ~~[`PAPERLESS_DB_TIMEOUT`](#PAPERLESS_DB_TIMEOUT)~~ {#PAPERLESS_DB_TIMEOUT} - A pool of 8-10 connections per worker is typically sufficient. - If you encounter error messages such as `couldn't get a connection` - or database connection timeouts, you probably need to increase the pool size. +!!! failure "Removed in v3" - !!! warning - Make sure your PostgreSQL `max_connections` setting is large enough to handle the connection pools: - `(NB_PAPERLESS_WORKERS + NB_CELERY_WORKERS) × POOL_SIZE + SAFETY_MARGIN`. For example, with - 4 Paperless workers and 2 Celery workers, and a pool size of 8:``(4 + 2) × 8 + 10 = 58`, - so `max_connections = 60` (or even more) is appropriate. + Use [`PAPERLESS_DB_OPTIONS`](#PAPERLESS_DB_OPTIONS) instead. - This assumes only Paperless-ngx connects to your PostgreSQL instance. If you have other applications, - you should increase `max_connections` accordingly. + ```bash title="SQLite" + PAPERLESS_DB_OPTIONS="timeout=30" + ``` + + ```bash title="PostgreSQL or MariaDB" + PAPERLESS_DB_OPTIONS="connect_timeout=30" + ``` + +#### ~~[`PAPERLESS_DB_POOLSIZE`](#PAPERLESS_DB_POOLSIZE)~~ {#PAPERLESS_DB_POOLSIZE} + +!!! failure "Removed in v3" + + Use [`PAPERLESS_DB_OPTIONS`](#PAPERLESS_DB_OPTIONS) instead. + + ```bash + PAPERLESS_DB_OPTIONS="pool.max_size=10" + ``` #### [`PAPERLESS_DB_READ_CACHE_ENABLED=`](#PAPERLESS_DB_READ_CACHE_ENABLED) {#PAPERLESS_DB_READ_CACHE_ENABLED} diff --git a/docs/migration.md b/docs/migration-v3.md similarity index 61% rename from docs/migration.md rename to docs/migration-v3.md index 60ffbf074..91d9ed54d 100644 --- a/docs/migration.md +++ b/docs/migration-v3.md @@ -48,3 +48,58 @@ The `CONSUMER_BARCODE_SCANNER` setting has been removed. zxing-cpp is now the on reliability. - The `libzbar0` / `libzbar-dev` system packages are no longer required and can be removed from any custom Docker images or host installations. + +## Database Engine + +`PAPERLESS_DBENGINE` is now required to use PostgreSQL or MariaDB. Previously, the +engine was inferred from the presence of `PAPERLESS_DBHOST`, with `PAPERLESS_DBENGINE` +only needed to select MariaDB over PostgreSQL. + +SQLite users require no changes, though they may explicitly set their engine if desired. + +#### Action Required + +PostgreSQL and MariaDB users must add `PAPERLESS_DBENGINE` to their environment: + +```yaml +# v2 (PostgreSQL inferred from PAPERLESS_DBHOST) +PAPERLESS_DBHOST: postgres + +# v3 (engine must be explicit) +PAPERLESS_DBENGINE: postgresql +PAPERLESS_DBHOST: postgres +``` + +See [`PAPERLESS_DBENGINE`](configuration.md#PAPERLESS_DBENGINE) for accepted values. + +## Database Advanced Options + +The individual SSL, timeout, and pooling variables have been removed in favor of a +single [`PAPERLESS_DB_OPTIONS`](configuration.md#PAPERLESS_DB_OPTIONS) string. This +consolidates a growing set of engine-specific variables into one place, and allows +any option supported by the underlying database driver to be set without requiring a +dedicated environment variable for each. + +The removed variables and their replacements are: + +| Removed Variable | Replacement in `PAPERLESS_DB_OPTIONS` | +| ------------------------- | ---------------------------------------------------------------------------- | +| `PAPERLESS_DBSSLMODE` | `sslmode=` (PostgreSQL) or `ssl_mode=` (MariaDB) | +| `PAPERLESS_DBSSLROOTCERT` | `sslrootcert=` (PostgreSQL) or `ssl.ca=` (MariaDB) | +| `PAPERLESS_DBSSLCERT` | `sslcert=` (PostgreSQL) or `ssl.cert=` (MariaDB) | +| `PAPERLESS_DBSSLKEY` | `sslkey=` (PostgreSQL) or `ssl.key=` (MariaDB) | +| `PAPERLESS_DB_POOLSIZE` | `pool.max_size=` (PostgreSQL only) | +| `PAPERLESS_DB_TIMEOUT` | `timeout=` (SQLite) or `connect_timeout=` (PostgreSQL/MariaDB) | + +The deprecated variables will continue to function for now but will be removed in a +future release. A deprecation warning is logged at startup for each deprecated variable +that is still set. + +#### Action Required + +Users with any of the deprecated variables set should migrate to `PAPERLESS_DB_OPTIONS`. +Multiple options are combined in a single value: + +```bash +PAPERLESS_DB_OPTIONS="sslmode=require;sslrootcert=/certs/ca.pem;pool.max_size=10" +``` diff --git a/docs/setup.md b/docs/setup.md index 3b033ed16..415b3622e 100644 --- a/docs/setup.md +++ b/docs/setup.md @@ -504,8 +504,7 @@ installation. Keep these points in mind: - Read the [changelog](changelog.md) and take note of breaking changes. - Decide whether to stay on SQLite or migrate to PostgreSQL. - See [documentation](#sqlite_to_psql) for details on moving data - from SQLite to PostgreSQL. Both work fine with + Both work fine with Paperless. However, if you already have a database server running for other services, you might as well use it for Paperless as well. - The task scheduler of Paperless, which is used to execute periodic diff --git a/src/paperless/checks.py b/src/paperless/checks.py index 7df85d146..c0c7f49e1 100644 --- a/src/paperless/checks.py +++ b/src/paperless/checks.py @@ -202,3 +202,43 @@ def audit_log_check(app_configs, **kwargs): ) return result + + +@register() +def check_deprecated_db_settings( + app_configs: object, + **kwargs: object, +) -> list[Warning]: + """Check for deprecated database environment variables. + + Detects legacy advanced options that should be migrated to + PAPERLESS_DB_OPTIONS. Returns one Warning per deprecated variable found. + """ + deprecated_vars: dict[str, str] = { + "PAPERLESS_DB_TIMEOUT": "timeout", + "PAPERLESS_DB_POOLSIZE": "pool.min_size / pool.max_size", + "PAPERLESS_DBSSLMODE": "sslmode", + "PAPERLESS_DBSSLROOTCERT": "sslrootcert", + "PAPERLESS_DBSSLCERT": "sslcert", + "PAPERLESS_DBSSLKEY": "sslkey", + } + + warnings: list[Warning] = [] + + for var_name, db_option_key in deprecated_vars.items(): + if not os.getenv(var_name): + continue + warnings.append( + Warning( + f"Deprecated environment variable: {var_name}", + hint=( + f"{var_name} is no longer supported and will be removed in v3.2. " + f"Set the equivalent option via PAPERLESS_DB_OPTIONS instead. " + f'Example: PAPERLESS_DB_OPTIONS=\'{{"{db_option_key}": ""}}\'. ' + "See https://docs.paperless-ngx.com/migration/ for the full reference." + ), + id="paperless.W001", + ), + ) + + return warnings diff --git a/src/paperless/settings.py b/src/paperless/settings/__init__.py similarity index 93% rename from src/paperless/settings.py rename to src/paperless/settings/__init__.py index bee406fa2..9f820bb04 100644 --- a/src/paperless/settings.py +++ b/src/paperless/settings/__init__.py @@ -17,6 +17,8 @@ from dateparser.languages.loader import LocaleDataLoader from django.utils.translation import gettext_lazy as _ from dotenv import load_dotenv +from paperless.settings.custom import parse_db_settings + logger = logging.getLogger("paperless.settings") # Tap paperless.conf if it's available @@ -282,7 +284,7 @@ DEBUG = __get_boolean("PAPERLESS_DEBUG", "NO") # Directories # ############################################################################### -BASE_DIR: Path = Path(__file__).resolve().parent.parent +BASE_DIR: Path = Path(__file__).resolve().parent.parent.parent STATIC_ROOT = __get_path("PAPERLESS_STATICDIR", BASE_DIR.parent / "static") @@ -722,83 +724,8 @@ EMAIL_CERTIFICATE_FILE = __get_optional_path("PAPERLESS_EMAIL_CERTIFICATE_LOCATI ############################################################################### # Database # ############################################################################### -def _parse_db_settings() -> dict: - databases = { - "default": { - "ENGINE": "django.db.backends.sqlite3", - "NAME": DATA_DIR / "db.sqlite3", - "OPTIONS": {}, - }, - } - if os.getenv("PAPERLESS_DBHOST"): - # Have sqlite available as a second option for management commands - # This is important when migrating to/from sqlite - databases["sqlite"] = databases["default"].copy() - databases["default"] = { - "HOST": os.getenv("PAPERLESS_DBHOST"), - "NAME": os.getenv("PAPERLESS_DBNAME", "paperless"), - "USER": os.getenv("PAPERLESS_DBUSER", "paperless"), - "PASSWORD": os.getenv("PAPERLESS_DBPASS", "paperless"), - "OPTIONS": {}, - } - if os.getenv("PAPERLESS_DBPORT"): - databases["default"]["PORT"] = os.getenv("PAPERLESS_DBPORT") - - # Leave room for future extensibility - if os.getenv("PAPERLESS_DBENGINE") == "mariadb": - engine = "django.db.backends.mysql" - # Contrary to Postgres, Django does not natively support connection pooling for MariaDB. - # However, since MariaDB uses threads instead of forks, establishing connections is significantly faster - # compared to PostgreSQL, so the lack of pooling is not an issue - options = { - "read_default_file": "/etc/mysql/my.cnf", - "charset": "utf8mb4", - "ssl_mode": os.getenv("PAPERLESS_DBSSLMODE", "PREFERRED"), - "ssl": { - "ca": os.getenv("PAPERLESS_DBSSLROOTCERT", None), - "cert": os.getenv("PAPERLESS_DBSSLCERT", None), - "key": os.getenv("PAPERLESS_DBSSLKEY", None), - }, - } - - else: # Default to PostgresDB - engine = "django.db.backends.postgresql" - options = { - "sslmode": os.getenv("PAPERLESS_DBSSLMODE", "prefer"), - "sslrootcert": os.getenv("PAPERLESS_DBSSLROOTCERT", None), - "sslcert": os.getenv("PAPERLESS_DBSSLCERT", None), - "sslkey": os.getenv("PAPERLESS_DBSSLKEY", None), - } - if int(os.getenv("PAPERLESS_DB_POOLSIZE", 0)) > 0: - options.update( - { - "pool": { - "min_size": 1, - "max_size": int(os.getenv("PAPERLESS_DB_POOLSIZE")), - }, - }, - ) - - databases["default"]["ENGINE"] = engine - databases["default"]["OPTIONS"].update(options) - - if os.getenv("PAPERLESS_DB_TIMEOUT") is not None: - if databases["default"]["ENGINE"] == "django.db.backends.sqlite3": - databases["default"]["OPTIONS"].update( - {"timeout": int(os.getenv("PAPERLESS_DB_TIMEOUT"))}, - ) - else: - databases["default"]["OPTIONS"].update( - {"connect_timeout": int(os.getenv("PAPERLESS_DB_TIMEOUT"))}, - ) - databases["sqlite"]["OPTIONS"].update( - {"timeout": int(os.getenv("PAPERLESS_DB_TIMEOUT"))}, - ) - return databases - - -DATABASES = _parse_db_settings() +DATABASES = parse_db_settings(DATA_DIR) if os.getenv("PAPERLESS_DBENGINE") == "mariadb": # Silence Django error on old MariaDB versions. diff --git a/src/paperless/settings/custom.py b/src/paperless/settings/custom.py new file mode 100644 index 000000000..6f7d94201 --- /dev/null +++ b/src/paperless/settings/custom.py @@ -0,0 +1,122 @@ +import os +from pathlib import Path +from typing import Any + +from paperless.settings.parsers import get_choice_from_env +from paperless.settings.parsers import get_int_from_env +from paperless.settings.parsers import parse_dict_from_str + + +def parse_db_settings(data_dir: Path) -> dict[str, dict[str, Any]]: + """Parse database settings from environment variables. + + Core connection variables (no deprecation): + - PAPERLESS_DBENGINE (sqlite/postgresql/mariadb) + - PAPERLESS_DBHOST, PAPERLESS_DBPORT + - PAPERLESS_DBNAME, PAPERLESS_DBUSER, PAPERLESS_DBPASS + + Advanced options can be set via: + - Legacy individual env vars (deprecated in v3.0, removed in v3.2) + - PAPERLESS_DB_OPTIONS (recommended v3+ approach) + + Args: + data_dir: The data directory path for SQLite database location. + + Returns: + A databases dict suitable for Django DATABASES setting. + """ + try: + engine = get_choice_from_env( + "PAPERLESS_DBENGINE", + {"sqlite", "postgresql", "mariadb"}, + default="sqlite", + ) + except ValueError: + # MariaDB users already had to set PAPERLESS_DBENGINE, so it was picked up above + # SQLite users didn't need to set anything + engine = "postgresql" if "PAPERLESS_DBHOST" in os.environ else "sqlite" + + db_config: dict[str, Any] + base_options: dict[str, Any] + + match engine: + case "sqlite": + db_config = { + "ENGINE": "django.db.backends.sqlite3", + "NAME": str((data_dir / "db.sqlite3").resolve()), + } + base_options = {} + + case "postgresql": + db_config = { + "ENGINE": "django.db.backends.postgresql", + "HOST": os.getenv("PAPERLESS_DBHOST"), + "NAME": os.getenv("PAPERLESS_DBNAME", "paperless"), + "USER": os.getenv("PAPERLESS_DBUSER", "paperless"), + "PASSWORD": os.getenv("PAPERLESS_DBPASS", "paperless"), + } + + base_options = { + "sslmode": os.getenv("PAPERLESS_DBSSLMODE", "prefer"), + "sslrootcert": os.getenv("PAPERLESS_DBSSLROOTCERT"), + "sslcert": os.getenv("PAPERLESS_DBSSLCERT"), + "sslkey": os.getenv("PAPERLESS_DBSSLKEY"), + } + + if (pool_size := get_int_from_env("PAPERLESS_DB_POOLSIZE")) is not None: + base_options["pool"] = { + "min_size": 1, + "max_size": pool_size, + } + + case "mariadb": + db_config = { + "ENGINE": "django.db.backends.mysql", + "HOST": os.getenv("PAPERLESS_DBHOST"), + "NAME": os.getenv("PAPERLESS_DBNAME", "paperless"), + "USER": os.getenv("PAPERLESS_DBUSER", "paperless"), + "PASSWORD": os.getenv("PAPERLESS_DBPASS", "paperless"), + } + + base_options = { + "read_default_file": "/etc/mysql/my.cnf", + "charset": "utf8mb4", + "collation": "utf8mb4_unicode_ci", + "ssl_mode": os.getenv("PAPERLESS_DBSSLMODE", "PREFERRED"), + "ssl": { + "ca": os.getenv("PAPERLESS_DBSSLROOTCERT"), + "cert": os.getenv("PAPERLESS_DBSSLCERT"), + "key": os.getenv("PAPERLESS_DBSSLKEY"), + }, + } + case _: # pragma: no cover + raise NotImplementedError(engine) + + # Handle port setting for external databases + if ( + engine in ("postgresql", "mariadb") + and (port := get_int_from_env("PAPERLESS_DBPORT")) is not None + ): + db_config["PORT"] = port + + # Handle timeout setting (common across all engines, different key names) + if (timeout := get_int_from_env("PAPERLESS_DB_TIMEOUT")) is not None: + timeout_key = "timeout" if engine == "sqlite" else "connect_timeout" + base_options[timeout_key] = timeout + + # Apply PAPERLESS_DB_OPTIONS overrides + db_config["OPTIONS"] = parse_dict_from_str( + os.getenv("PAPERLESS_DB_OPTIONS"), + defaults=base_options, + separator=";", + type_map={ + # SQLite options + "timeout": int, + # Postgres/MariaDB options + "connect_timeout": int, + "pool.min_size": int, + "pool.max_size": int, + }, + ) + + return {"default": db_config} diff --git a/src/paperless/settings/parsers.py b/src/paperless/settings/parsers.py new file mode 100644 index 000000000..e6960abf1 --- /dev/null +++ b/src/paperless/settings/parsers.py @@ -0,0 +1,192 @@ +import copy +import os +from collections.abc import Callable +from collections.abc import Mapping +from pathlib import Path +from typing import Any +from typing import TypeVar +from typing import overload + +T = TypeVar("T") + + +def str_to_bool(value: str) -> bool: + """ + Converts a string representation of truth to a boolean value. + + Recognizes 'true', '1', 't', 'y', 'yes' as True, and + 'false', '0', 'f', 'n', 'no' as False. Case-insensitive. + + Args: + value: The string to convert. + + Returns: + The boolean representation of the string. + + Raises: + ValueError: If the string is not a recognized boolean value. + """ + val_lower = value.strip().lower() + if val_lower in ("true", "1", "t", "y", "yes"): + return True + elif val_lower in ("false", "0", "f", "n", "no"): + return False + raise ValueError(f"Cannot convert '{value}' to a boolean.") + + +@overload +def get_int_from_env(key: str) -> int | None: ... + + +@overload +def get_int_from_env(key: str, default: None) -> int | None: ... + + +@overload +def get_int_from_env(key: str, default: int) -> int: ... + + +def get_int_from_env(key: str, default: int | None = None) -> int | None: + """ + Return an integer value based on the environment variable. + If default is provided, returns that value when key is missing. + If default is None, returns None when key is missing. + """ + if key not in os.environ: + return default + + return int(os.environ[key]) + + +def parse_dict_from_str( + env_str: str | None, + defaults: dict[str, Any] | None = None, + type_map: Mapping[str, Callable[[str], Any]] | None = None, + separator: str = ",", +) -> dict[str, Any]: + """ + Parses a key-value string into a dictionary, applying defaults and casting types. + + Supports nested keys via dot-notation, e.g.: + "database.host=localhost,database.port=5432" + + Args: + env_str: The string from the environment variable (e.g., "port=9090,debug=true"). + defaults: A dictionary of default values (can contain nested dicts). + type_map: A dictionary mapping keys (dot-notation allowed) to a type or a parsing + function (e.g., {'port': int, 'debug': bool, 'database.port': int}). + The special `bool` type triggers custom boolean parsing. + separator: The character used to separate key-value pairs. Defaults to ','. + + Returns: + A dictionary with the parsed and correctly-typed settings. + + Raises: + ValueError: If a value cannot be cast to its specified type. + """ + + def _set_nested(d: dict, keys: list[str], value: Any) -> None: + """Set a nested value, creating intermediate dicts as needed.""" + cur = d + for k in keys[:-1]: + if k not in cur or not isinstance(cur[k], dict): + cur[k] = {} + cur = cur[k] + cur[keys[-1]] = value + + def _get_nested(d: dict, keys: list[str]) -> Any: + """Get nested value or raise KeyError if not present.""" + cur = d + for k in keys: + if not isinstance(cur, dict) or k not in cur: + raise KeyError + cur = cur[k] + return cur + + def _has_nested(d: dict, keys: list[str]) -> bool: + try: + _get_nested(d, keys) + return True + except KeyError: + return False + + settings: dict[str, Any] = copy.deepcopy(defaults) if defaults else {} + _type_map = type_map if type_map else {} + + if not env_str: + return settings + + # Parse the environment string using the specified separator + pairs = [p.strip() for p in env_str.split(separator) if p.strip()] + for pair in pairs: + if "=" not in pair: + # ignore malformed pairs + continue + key, val = pair.split("=", 1) + key = key.strip() + val = val.strip() + if not key: + continue + parts = key.split(".") + _set_nested(settings, parts, val) + + # Apply type casting to the updated settings (supports nested keys in type_map) + for key, caster in _type_map.items(): + key_parts = key.split(".") + if _has_nested(settings, key_parts): + raw_val = _get_nested(settings, key_parts) + # Only cast if it's a string (i.e. from env parsing). If defaults already provided + # a different type we leave it as-is. + if isinstance(raw_val, str): + try: + if caster is bool: + parsed = str_to_bool(raw_val) + elif caster is Path: + parsed = Path(raw_val).resolve() + else: + parsed = caster(raw_val) + except (ValueError, TypeError) as e: + caster_name = getattr(caster, "__name__", repr(caster)) + raise ValueError( + f"Error casting key '{key}' with value '{raw_val}' " + f"to type '{caster_name}'", + ) from e + _set_nested(settings, key_parts, parsed) + + return settings + + +def get_choice_from_env( + env_key: str, + choices: set[str], + default: str | None = None, +) -> str: + """ + Gets and validates an environment variable against a set of allowed choices. + + Args: + env_key: The environment variable key to validate + choices: Set of valid choices for the environment variable + default: Optional default value if environment variable is not set + + Returns: + The validated environment variable value + + Raises: + ValueError: If the environment variable value is not in choices + or if no default is provided and env var is missing + """ + value = os.environ.get(env_key, default) + + if value is None: + raise ValueError( + f"Environment variable '{env_key}' is required but not set.", + ) + + if value not in choices: + raise ValueError( + f"Environment variable '{env_key}' has invalid value '{value}'. " + f"Valid choices are: {', '.join(sorted(choices))}", + ) + + return value diff --git a/src/paperless/tests/settings/__init__.py b/src/paperless/tests/settings/__init__.py new file mode 100644 index 000000000..e69de29bb diff --git a/src/paperless/tests/settings/test_custom_parsers.py b/src/paperless/tests/settings/test_custom_parsers.py new file mode 100644 index 000000000..e297d9e04 --- /dev/null +++ b/src/paperless/tests/settings/test_custom_parsers.py @@ -0,0 +1,266 @@ +import os +from pathlib import Path + +import pytest +from pytest_mock import MockerFixture + +from paperless.settings.custom import parse_db_settings + + +class TestParseDbSettings: + """Test suite for parse_db_settings function.""" + + @pytest.mark.parametrize( + ("env_vars", "expected_database_settings"), + [ + pytest.param( + {}, + { + "default": { + "ENGINE": "django.db.backends.sqlite3", + "NAME": None, # Will be replaced with tmp_path + "OPTIONS": {}, + }, + }, + id="default-sqlite", + ), + pytest.param( + { + "PAPERLESS_DBENGINE": "sqlite", + "PAPERLESS_DB_OPTIONS": "timeout=30", + }, + { + "default": { + "ENGINE": "django.db.backends.sqlite3", + "NAME": None, # Will be replaced with tmp_path + "OPTIONS": { + "timeout": 30, + }, + }, + }, + id="sqlite-with-timeout-override", + ), + pytest.param( + { + "PAPERLESS_DBENGINE": "postgresql", + "PAPERLESS_DBHOST": "localhost", + }, + { + "default": { + "ENGINE": "django.db.backends.postgresql", + "HOST": "localhost", + "NAME": "paperless", + "USER": "paperless", + "PASSWORD": "paperless", + "OPTIONS": { + "sslmode": "prefer", + "sslrootcert": None, + "sslcert": None, + "sslkey": None, + }, + }, + }, + id="postgresql-defaults", + ), + pytest.param( + { + "PAPERLESS_DBENGINE": "postgresql", + "PAPERLESS_DBHOST": "paperless-db-host", + "PAPERLESS_DBPORT": "1111", + "PAPERLESS_DBNAME": "customdb", + "PAPERLESS_DBUSER": "customuser", + "PAPERLESS_DBPASS": "custompass", + "PAPERLESS_DB_OPTIONS": "pool.max_size=50;pool.min_size=2;sslmode=require", + }, + { + "default": { + "ENGINE": "django.db.backends.postgresql", + "HOST": "paperless-db-host", + "PORT": 1111, + "NAME": "customdb", + "USER": "customuser", + "PASSWORD": "custompass", + "OPTIONS": { + "sslmode": "require", + "sslrootcert": None, + "sslcert": None, + "sslkey": None, + "pool": { + "min_size": 2, + "max_size": 50, + }, + }, + }, + }, + id="postgresql-overrides", + ), + pytest.param( + { + "PAPERLESS_DBENGINE": "postgresql", + "PAPERLESS_DBHOST": "pghost", + "PAPERLESS_DB_POOLSIZE": "10", + }, + { + "default": { + "ENGINE": "django.db.backends.postgresql", + "HOST": "pghost", + "NAME": "paperless", + "USER": "paperless", + "PASSWORD": "paperless", + "OPTIONS": { + "sslmode": "prefer", + "sslrootcert": None, + "sslcert": None, + "sslkey": None, + "pool": { + "min_size": 1, + "max_size": 10, + }, + }, + }, + }, + id="postgresql-legacy-poolsize", + ), + pytest.param( + { + "PAPERLESS_DBENGINE": "postgresql", + "PAPERLESS_DBHOST": "pghost", + "PAPERLESS_DBSSLMODE": "require", + "PAPERLESS_DBSSLROOTCERT": "/certs/ca.crt", + "PAPERLESS_DB_TIMEOUT": "30", + }, + { + "default": { + "ENGINE": "django.db.backends.postgresql", + "HOST": "pghost", + "NAME": "paperless", + "USER": "paperless", + "PASSWORD": "paperless", + "OPTIONS": { + "sslmode": "require", + "sslrootcert": "/certs/ca.crt", + "sslcert": None, + "sslkey": None, + "connect_timeout": 30, + }, + }, + }, + id="postgresql-legacy-ssl-and-timeout", + ), + pytest.param( + { + "PAPERLESS_DBENGINE": "mariadb", + "PAPERLESS_DBHOST": "localhost", + }, + { + "default": { + "ENGINE": "django.db.backends.mysql", + "HOST": "localhost", + "NAME": "paperless", + "USER": "paperless", + "PASSWORD": "paperless", + "OPTIONS": { + "read_default_file": "/etc/mysql/my.cnf", + "charset": "utf8mb4", + "collation": "utf8mb4_unicode_ci", + "ssl_mode": "PREFERRED", + "ssl": { + "ca": None, + "cert": None, + "key": None, + }, + }, + }, + }, + id="mariadb-defaults", + ), + pytest.param( + { + "PAPERLESS_DBENGINE": "mariadb", + "PAPERLESS_DBHOST": "paperless-mariadb-host", + "PAPERLESS_DBPORT": "5555", + "PAPERLESS_DBUSER": "my-cool-user", + "PAPERLESS_DBPASS": "my-secure-password", + "PAPERLESS_DB_OPTIONS": "ssl.ca=/path/to/ca.pem;ssl_mode=REQUIRED", + }, + { + "default": { + "ENGINE": "django.db.backends.mysql", + "HOST": "paperless-mariadb-host", + "PORT": 5555, + "NAME": "paperless", + "USER": "my-cool-user", + "PASSWORD": "my-secure-password", + "OPTIONS": { + "read_default_file": "/etc/mysql/my.cnf", + "charset": "utf8mb4", + "collation": "utf8mb4_unicode_ci", + "ssl_mode": "REQUIRED", + "ssl": { + "ca": "/path/to/ca.pem", + "cert": None, + "key": None, + }, + }, + }, + }, + id="mariadb-overrides", + ), + pytest.param( + { + "PAPERLESS_DBENGINE": "mariadb", + "PAPERLESS_DBHOST": "mariahost", + "PAPERLESS_DBSSLMODE": "REQUIRED", + "PAPERLESS_DBSSLROOTCERT": "/certs/ca.pem", + "PAPERLESS_DBSSLCERT": "/certs/client.pem", + "PAPERLESS_DBSSLKEY": "/certs/client.key", + "PAPERLESS_DB_TIMEOUT": "25", + }, + { + "default": { + "ENGINE": "django.db.backends.mysql", + "HOST": "mariahost", + "NAME": "paperless", + "USER": "paperless", + "PASSWORD": "paperless", + "OPTIONS": { + "read_default_file": "/etc/mysql/my.cnf", + "charset": "utf8mb4", + "collation": "utf8mb4_unicode_ci", + "ssl_mode": "REQUIRED", + "ssl": { + "ca": "/certs/ca.pem", + "cert": "/certs/client.pem", + "key": "/certs/client.key", + }, + "connect_timeout": 25, + }, + }, + }, + id="mariadb-legacy-ssl-and-timeout", + ), + ], + ) + def test_parse_db_settings( + self, + tmp_path: Path, + mocker: MockerFixture, + env_vars: dict[str, str], + expected_database_settings: dict[str, dict], + ) -> None: + """Test various database configurations with defaults and overrides.""" + # Clear environment and set test vars + mocker.patch.dict(os.environ, env_vars, clear=True) + + # Update expected paths with actual tmp_path + if ( + "default" in expected_database_settings + and expected_database_settings["default"]["NAME"] is None + ): + expected_database_settings["default"]["NAME"] = str( + tmp_path / "db.sqlite3", + ) + + settings = parse_db_settings(tmp_path) + + assert settings == expected_database_settings diff --git a/src/paperless/tests/settings/test_environment_parsers.py b/src/paperless/tests/settings/test_environment_parsers.py new file mode 100644 index 000000000..ecce0ea16 --- /dev/null +++ b/src/paperless/tests/settings/test_environment_parsers.py @@ -0,0 +1,414 @@ +import os +from pathlib import Path + +import pytest +from pytest_mock import MockerFixture + +from paperless.settings.parsers import get_choice_from_env +from paperless.settings.parsers import get_int_from_env +from paperless.settings.parsers import parse_dict_from_str +from paperless.settings.parsers import str_to_bool + + +class TestStringToBool: + @pytest.mark.parametrize( + "true_value", + [ + pytest.param("true", id="lowercase_true"), + pytest.param("1", id="digit_1"), + pytest.param("T", id="capital_T"), + pytest.param("y", id="lowercase_y"), + pytest.param("YES", id="uppercase_YES"), + pytest.param(" True ", id="whitespace_true"), + ], + ) + def test_true_conversion(self, true_value: str): + """Test that various 'true' strings correctly evaluate to True.""" + assert str_to_bool(true_value) is True + + @pytest.mark.parametrize( + "false_value", + [ + pytest.param("false", id="lowercase_false"), + pytest.param("0", id="digit_0"), + pytest.param("f", id="capital_f"), + pytest.param("N", id="capital_N"), + pytest.param("no", id="lowercase_no"), + pytest.param(" False ", id="whitespace_false"), + ], + ) + def test_false_conversion(self, false_value: str): + """Test that various 'false' strings correctly evaluate to False.""" + assert str_to_bool(false_value) is False + + def test_invalid_conversion(self): + """Test that an invalid string raises a ValueError.""" + with pytest.raises(ValueError, match="Cannot convert 'maybe' to a boolean\\."): + str_to_bool("maybe") + + +class TestParseDictFromString: + def test_empty_and_none_input(self): + """Test behavior with None or empty string input.""" + assert parse_dict_from_str(None) == {} + assert parse_dict_from_str("") == {} + defaults = {"a": 1} + res = parse_dict_from_str(None, defaults=defaults) + assert res == defaults + # Ensure it returns a copy, not the original object + assert res is not defaults + + def test_basic_parsing(self): + """Test simple key-value parsing without defaults or types.""" + env_str = "key1=val1, key2=val2" + expected = {"key1": "val1", "key2": "val2"} + assert parse_dict_from_str(env_str) == expected + + def test_with_defaults(self): + """Test that environment values override defaults correctly.""" + defaults = {"host": "localhost", "port": 8000, "user": "default"} + env_str = "port=9090, host=db.example.com" + expected = {"host": "db.example.com", "port": "9090", "user": "default"} + result = parse_dict_from_str(env_str, defaults=defaults) + assert result == expected + + def test_type_casting(self): + """Test successful casting of values to specified types.""" + env_str = "port=9090, debug=true, timeout=12.5, user=admin" + type_map = {"port": int, "debug": bool, "timeout": float} + expected = {"port": 9090, "debug": True, "timeout": 12.5, "user": "admin"} + result = parse_dict_from_str(env_str, type_map=type_map) + assert result == expected + + def test_type_casting_with_defaults(self): + """Test casting when values come from both defaults and env string.""" + defaults = {"port": 8000, "debug": False, "retries": 3} + env_str = "port=9090, debug=true" + type_map = {"port": int, "debug": bool, "retries": int} + + # The 'retries' value comes from defaults and is already an int, + # so it should not be processed by the caster. + expected = {"port": 9090, "debug": True, "retries": 3} + result = parse_dict_from_str(env_str, defaults=defaults, type_map=type_map) + assert result == expected + assert isinstance(result["retries"], int) + + def test_path_casting(self, tmp_path: Path): + """Test successful casting of a string to a resolved pathlib.Path object.""" + # Create a dummy file to resolve against + test_file = tmp_path / "test_file.txt" + test_file.touch() + + env_str = f"config_path={test_file}" + type_map = {"config_path": Path} + result = parse_dict_from_str(env_str, type_map=type_map) + + # The result should be a resolved Path object + assert isinstance(result["config_path"], Path) + assert result["config_path"] == test_file.resolve() + + def test_custom_separator(self): + """Test parsing with a custom separator like a semicolon.""" + env_str = "host=db; port=5432; user=test" + expected = {"host": "db", "port": "5432", "user": "test"} + result = parse_dict_from_str(env_str, separator=";") + assert result == expected + + def test_edge_cases_in_string(self): + """Test malformed strings to ensure robustness.""" + # Malformed pair 'debug' is skipped, extra comma is ignored + env_str = "key=val,, debug, foo=bar" + expected = {"key": "val", "foo": "bar"} + assert parse_dict_from_str(env_str) == expected + + # Value can contain the equals sign + env_str = "url=postgres://user:pass@host:5432/db" + expected = {"url": "postgres://user:pass@host:5432/db"} + assert parse_dict_from_str(env_str) == expected + + def test_casting_error_handling(self): + """Test that a ValueError is raised for invalid casting.""" + env_str = "port=not-a-number" + type_map = {"port": int} + + with pytest.raises(ValueError) as excinfo: + parse_dict_from_str(env_str, type_map=type_map) + + assert "Error casting key 'port'" in str(excinfo.value) + assert "value 'not-a-number'" in str(excinfo.value) + assert "to type 'int'" in str(excinfo.value) + + def test_bool_casting_error(self): + """Test that an invalid boolean string raises a ValueError.""" + env_str = "debug=maybe" + type_map = {"debug": bool} + with pytest.raises(ValueError, match="Error casting key 'debug'"): + parse_dict_from_str(env_str, type_map=type_map) + + def test_nested_key_parsing_basic(self): + """Basic nested key parsing using dot-notation.""" + env_str = "database.host=db.example.com, database.port=5432, logging.level=INFO" + result = parse_dict_from_str(env_str) + assert result == { + "database": {"host": "db.example.com", "port": "5432"}, + "logging": {"level": "INFO"}, + } + + def test_nested_overrides_defaults_and_deepcopy(self): + """Nested env keys override defaults and defaults are deep-copied.""" + defaults = {"database": {"host": "127.0.0.1", "port": 3306, "user": "default"}} + env_str = "database.host=db.example.com, debug=true" + result = parse_dict_from_str( + env_str, + defaults=defaults, + type_map={"debug": bool}, + ) + + assert result["database"]["host"] == "db.example.com" + # Unchanged default preserved + assert result["database"]["port"] == 3306 + assert result["database"]["user"] == "default" + # Default object was deep-copied (no same nested object identity) + assert result is not defaults + assert result["database"] is not defaults["database"] + + def test_nested_type_casting(self): + """Type casting for nested keys (dot-notation) should work.""" + env_str = "database.host=db.example.com, database.port=5433, debug=false" + type_map = {"database.port": int, "debug": bool} + result = parse_dict_from_str(env_str, type_map=type_map) + + assert result["database"]["host"] == "db.example.com" + assert result["database"]["port"] == 5433 + assert isinstance(result["database"]["port"], int) + assert result["debug"] is False + assert isinstance(result["debug"], bool) + + def test_nested_casting_error_message(self): + """Error messages should include the full dotted key name on failure.""" + env_str = "database.port=not-a-number" + type_map = {"database.port": int} + with pytest.raises(ValueError) as excinfo: + parse_dict_from_str(env_str, type_map=type_map) + + msg = str(excinfo.value) + assert "Error casting key 'database.port'" in msg + assert "value 'not-a-number'" in msg + assert "to type 'int'" in msg + + def test_type_map_does_not_recast_non_string_defaults(self): + """If a default already provides a non-string value, the caster should skip it.""" + defaults = {"database": {"port": 3306}} + type_map = {"database.port": int} + result = parse_dict_from_str(None, defaults=defaults, type_map=type_map) + assert result["database"]["port"] == 3306 + assert isinstance(result["database"]["port"], int) + + +class TestGetIntFromEnv: + @pytest.mark.parametrize( + ("env_value", "expected"), + [ + pytest.param("42", 42, id="positive"), + pytest.param("-10", -10, id="negative"), + pytest.param("0", 0, id="zero"), + pytest.param("999", 999, id="large_positive"), + pytest.param("-999", -999, id="large_negative"), + ], + ) + def test_existing_env_var_valid_ints(self, mocker, env_value, expected): + """Test that existing environment variables with valid integers return correct values.""" + mocker.patch.dict(os.environ, {"INT_VAR": env_value}) + assert get_int_from_env("INT_VAR") == expected + + @pytest.mark.parametrize( + ("default", "expected"), + [ + pytest.param(100, 100, id="positive_default"), + pytest.param(0, 0, id="zero_default"), + pytest.param(-50, -50, id="negative_default"), + pytest.param(None, None, id="none_default"), + ], + ) + def test_missing_env_var_with_defaults(self, mocker, default, expected): + """Test that missing environment variables return provided defaults.""" + mocker.patch.dict(os.environ, {}, clear=True) + assert get_int_from_env("MISSING_VAR", default=default) == expected + + def test_missing_env_var_no_default(self, mocker): + """Test that missing environment variable with no default returns None.""" + mocker.patch.dict(os.environ, {}, clear=True) + assert get_int_from_env("MISSING_VAR") is None + + @pytest.mark.parametrize( + "invalid_value", + [ + pytest.param("not_a_number", id="text"), + pytest.param("42.5", id="float"), + pytest.param("42a", id="alpha_suffix"), + pytest.param("", id="empty"), + pytest.param(" ", id="whitespace"), + pytest.param("true", id="boolean"), + pytest.param("1.0", id="decimal"), + ], + ) + def test_invalid_int_values_raise_error(self, mocker, invalid_value): + """Test that invalid integer values raise ValueError.""" + mocker.patch.dict(os.environ, {"INVALID_INT": invalid_value}) + with pytest.raises(ValueError): + get_int_from_env("INVALID_INT") + + +class TestGetEnvChoice: + @pytest.fixture + def valid_choices(self) -> set[str]: + """Fixture providing a set of valid environment choices.""" + return {"development", "staging", "production"} + + def test_returns_valid_env_value( + self, + mocker: MockerFixture, + valid_choices: set[str], + ) -> None: + """Test that function returns the environment value when it's valid.""" + mocker.patch.dict("os.environ", {"TEST_ENV": "development"}) + + result = get_choice_from_env("TEST_ENV", valid_choices) + + assert result == "development" + + def test_returns_default_when_env_not_set( + self, + mocker: MockerFixture, + valid_choices: set[str], + ) -> None: + """Test that function returns default value when env var is not set.""" + mocker.patch.dict("os.environ", {}, clear=True) + + result = get_choice_from_env("TEST_ENV", valid_choices, default="staging") + + assert result == "staging" + + def test_raises_error_when_env_not_set_and_no_default( + self, + mocker: MockerFixture, + valid_choices: set[str], + ) -> None: + """Test that function raises ValueError when env var is missing and no default.""" + mocker.patch.dict("os.environ", {}, clear=True) + + with pytest.raises(ValueError) as exc_info: + get_choice_from_env("TEST_ENV", valid_choices) + + assert "Environment variable 'TEST_ENV' is required but not set" in str( + exc_info.value, + ) + + def test_raises_error_when_env_value_invalid( + self, + mocker: MockerFixture, + valid_choices: set[str], + ) -> None: + """Test that function raises ValueError when env value is not in choices.""" + mocker.patch.dict("os.environ", {"TEST_ENV": "invalid_value"}) + + with pytest.raises(ValueError) as exc_info: + get_choice_from_env("TEST_ENV", valid_choices) + + error_msg = str(exc_info.value) + assert ( + "Environment variable 'TEST_ENV' has invalid value 'invalid_value'" + in error_msg + ) + assert "Valid choices are:" in error_msg + assert "development" in error_msg + assert "staging" in error_msg + assert "production" in error_msg + + def test_raises_error_when_default_invalid( + self, + mocker: MockerFixture, + valid_choices: set[str], + ) -> None: + """Test that function raises ValueError when default value is not in choices.""" + mocker.patch.dict("os.environ", {}, clear=True) + + with pytest.raises(ValueError) as exc_info: + get_choice_from_env("TEST_ENV", valid_choices, default="invalid_default") + + error_msg = str(exc_info.value) + assert ( + "Environment variable 'TEST_ENV' has invalid value 'invalid_default'" + in error_msg + ) + + def test_case_sensitive_validation( + self, + mocker: MockerFixture, + valid_choices: set[str], + ) -> None: + """Test that validation is case sensitive.""" + mocker.patch.dict("os.environ", {"TEST_ENV": "DEVELOPMENT"}) + + with pytest.raises(ValueError): + get_choice_from_env("TEST_ENV", valid_choices) + + def test_empty_string_env_value( + self, + mocker: MockerFixture, + valid_choices: set[str], + ) -> None: + """Test behavior with empty string environment value.""" + mocker.patch.dict("os.environ", {"TEST_ENV": ""}) + + with pytest.raises(ValueError) as exc_info: + get_choice_from_env("TEST_ENV", valid_choices) + + assert "has invalid value ''" in str(exc_info.value) + + def test_whitespace_env_value( + self, + mocker: MockerFixture, + valid_choices: set[str], + ) -> None: + """Test behavior with whitespace-only environment value.""" + mocker.patch.dict("os.environ", {"TEST_ENV": " development "}) + + with pytest.raises(ValueError): + get_choice_from_env("TEST_ENV", valid_choices) + + def test_single_choice_set(self, mocker: MockerFixture) -> None: + """Test function works correctly with single choice set.""" + single_choice: set[str] = {"production"} + mocker.patch.dict("os.environ", {"TEST_ENV": "production"}) + + result = get_choice_from_env("TEST_ENV", single_choice) + + assert result == "production" + + def test_large_choice_set(self, mocker: MockerFixture) -> None: + """Test function works correctly with large choice set.""" + large_choices: set[str] = {f"option_{i}" for i in range(100)} + mocker.patch.dict("os.environ", {"TEST_ENV": "option_50"}) + + result = get_choice_from_env("TEST_ENV", large_choices) + + assert result == "option_50" + + def test_different_env_keys( + self, + mocker: MockerFixture, + valid_choices: set[str], + ) -> None: + """Test function works with different environment variable keys.""" + test_cases = [ + ("DJANGO_ENV", "development"), + ("DATABASE_BACKEND", "staging"), + ("LOG_LEVEL", "production"), + ("APP_MODE", "development"), + ] + + for env_key, env_value in test_cases: + mocker.patch.dict("os.environ", {env_key: env_value}) + result = get_choice_from_env(env_key, valid_choices) + assert result == env_value diff --git a/src/paperless/tests/test_checks.py b/src/paperless/tests/test_checks.py index fc6150826..513cfb870 100644 --- a/src/paperless/tests/test_checks.py +++ b/src/paperless/tests/test_checks.py @@ -2,13 +2,17 @@ import os from pathlib import Path from unittest import mock +import pytest +from django.core.checks import Warning from django.test import TestCase from django.test import override_settings +from pytest_mock import MockerFixture from documents.tests.utils import DirectoriesMixin from documents.tests.utils import FileSystemAssertsMixin from paperless.checks import audit_log_check from paperless.checks import binaries_check +from paperless.checks import check_deprecated_db_settings from paperless.checks import debug_mode_check from paperless.checks import paths_check from paperless.checks import settings_values_check @@ -237,3 +241,157 @@ class TestAuditLogChecks(TestCase): ("auditlog table was found but audit log is disabled."), msg.msg, ) + + +DEPRECATED_VARS: dict[str, str] = { + "PAPERLESS_DB_TIMEOUT": "timeout", + "PAPERLESS_DB_POOLSIZE": "pool.min_size / pool.max_size", + "PAPERLESS_DBSSLMODE": "sslmode", + "PAPERLESS_DBSSLROOTCERT": "sslrootcert", + "PAPERLESS_DBSSLCERT": "sslcert", + "PAPERLESS_DBSSLKEY": "sslkey", +} + + +class TestDeprecatedDbSettings: + """Test suite for the check_deprecated_db_settings system check.""" + + def test_no_deprecated_vars_returns_empty( + self, + mocker: MockerFixture, + ) -> None: + """No warnings when none of the deprecated vars are present.""" + # clear=True ensures vars from the outer test environment do not leak in + mocker.patch.dict(os.environ, {}, clear=True) + result = check_deprecated_db_settings(None) + assert result == [] + + @pytest.mark.parametrize( + ("env_var", "db_option_key"), + [ + ("PAPERLESS_DB_TIMEOUT", "timeout"), + ("PAPERLESS_DB_POOLSIZE", "pool.min_size / pool.max_size"), + ("PAPERLESS_DBSSLMODE", "sslmode"), + ("PAPERLESS_DBSSLROOTCERT", "sslrootcert"), + ("PAPERLESS_DBSSLCERT", "sslcert"), + ("PAPERLESS_DBSSLKEY", "sslkey"), + ], + ids=[ + "db-timeout", + "db-poolsize", + "ssl-mode", + "ssl-rootcert", + "ssl-cert", + "ssl-key", + ], + ) + def test_single_deprecated_var_produces_one_warning( + self, + mocker: MockerFixture, + env_var: str, + db_option_key: str, + ) -> None: + """Each deprecated var in isolation produces exactly one warning.""" + mocker.patch.dict(os.environ, {env_var: "some_value"}, clear=True) + result = check_deprecated_db_settings(None) + + assert len(result) == 1 + warning = result[0] + assert isinstance(warning, Warning) + assert warning.id == "paperless.W001" + assert env_var in warning.hint + assert db_option_key in warning.hint + + def test_multiple_deprecated_vars_produce_one_warning_each( + self, + mocker: MockerFixture, + ) -> None: + """Each deprecated var present in the environment gets its own warning.""" + set_vars = { + "PAPERLESS_DB_TIMEOUT": "30", + "PAPERLESS_DB_POOLSIZE": "10", + "PAPERLESS_DBSSLMODE": "require", + } + mocker.patch.dict(os.environ, set_vars, clear=True) + result = check_deprecated_db_settings(None) + + assert len(result) == len(set_vars) + assert all(isinstance(w, Warning) for w in result) + assert all(w.id == "paperless.W001" for w in result) + all_hints = " ".join(w.hint for w in result) + for var_name in set_vars: + assert var_name in all_hints + + def test_all_deprecated_vars_produces_one_warning_each( + self, + mocker: MockerFixture, + ) -> None: + """All deprecated vars set simultaneously produces one warning per var.""" + all_vars = dict.fromkeys(DEPRECATED_VARS, "some_value") + mocker.patch.dict(os.environ, all_vars, clear=True) + result = check_deprecated_db_settings(None) + + assert len(result) == len(DEPRECATED_VARS) + assert all(isinstance(w, Warning) for w in result) + assert all(w.id == "paperless.W001" for w in result) + + def test_unset_vars_not_mentioned_in_warnings( + self, + mocker: MockerFixture, + ) -> None: + """Vars absent from the environment do not appear in any warning.""" + mocker.patch.dict( + os.environ, + {"PAPERLESS_DB_TIMEOUT": "30"}, + clear=True, + ) + result = check_deprecated_db_settings(None) + + assert len(result) == 1 + assert "PAPERLESS_DB_TIMEOUT" in result[0].hint + unset_vars = [v for v in DEPRECATED_VARS if v != "PAPERLESS_DB_TIMEOUT"] + for var_name in unset_vars: + assert var_name not in result[0].hint + + def test_empty_string_var_not_treated_as_set( + self, + mocker: MockerFixture, + ) -> None: + """A var set to an empty string is not flagged as a deprecated setting.""" + mocker.patch.dict( + os.environ, + {"PAPERLESS_DB_TIMEOUT": ""}, + clear=True, + ) + result = check_deprecated_db_settings(None) + assert result == [] + + def test_warning_mentions_migration_target( + self, + mocker: MockerFixture, + ) -> None: + """Each warning hints at PAPERLESS_DB_OPTIONS as the migration target.""" + mocker.patch.dict( + os.environ, + {"PAPERLESS_DBSSLMODE": "require"}, + clear=True, + ) + result = check_deprecated_db_settings(None) + + assert len(result) == 1 + assert "PAPERLESS_DB_OPTIONS" in result[0].hint + + def test_warning_message_identifies_var( + self, + mocker: MockerFixture, + ) -> None: + """The warning message (not just the hint) identifies the offending var.""" + mocker.patch.dict( + os.environ, + {"PAPERLESS_DBSSLCERT": "/path/to/cert.pem"}, + clear=True, + ) + result = check_deprecated_db_settings(None) + + assert len(result) == 1 + assert "PAPERLESS_DBSSLCERT" in result[0].msg diff --git a/src/paperless/tests/test_settings.py b/src/paperless/tests/test_settings.py index 02db82ef2..cc9ad2081 100644 --- a/src/paperless/tests/test_settings.py +++ b/src/paperless/tests/test_settings.py @@ -9,7 +9,6 @@ from celery.schedules import crontab from paperless.settings import _parse_base_paths from paperless.settings import _parse_beat_schedule from paperless.settings import _parse_dateparser_languages -from paperless.settings import _parse_db_settings from paperless.settings import _parse_ignore_dates from paperless.settings import _parse_paperless_url from paperless.settings import _parse_redis_url @@ -378,64 +377,6 @@ class TestCeleryScheduleParsing(TestCase): ) -class TestDBSettings(TestCase): - def test_db_timeout_with_sqlite(self) -> None: - """ - GIVEN: - - PAPERLESS_DB_TIMEOUT is set - WHEN: - - Settings are parsed - THEN: - - PAPERLESS_DB_TIMEOUT set for sqlite - """ - with mock.patch.dict( - os.environ, - { - "PAPERLESS_DB_TIMEOUT": "10", - }, - ): - databases = _parse_db_settings() - - self.assertDictEqual( - { - "timeout": 10.0, - }, - databases["default"]["OPTIONS"], - ) - - def test_db_timeout_with_not_sqlite(self) -> None: - """ - GIVEN: - - PAPERLESS_DB_TIMEOUT is set but db is not sqlite - WHEN: - - Settings are parsed - THEN: - - PAPERLESS_DB_TIMEOUT set correctly in non-sqlite db & for fallback sqlite db - """ - with mock.patch.dict( - os.environ, - { - "PAPERLESS_DBHOST": "127.0.0.1", - "PAPERLESS_DB_TIMEOUT": "10", - }, - ): - databases = _parse_db_settings() - - self.assertDictEqual( - databases["default"]["OPTIONS"], - databases["default"]["OPTIONS"] - | { - "connect_timeout": 10.0, - }, - ) - self.assertDictEqual( - { - "timeout": 10.0, - }, - databases["sqlite"]["OPTIONS"], - ) - - class TestPaperlessURLSettings(TestCase): def test_paperless_url(self) -> None: """ diff --git a/zensical.toml b/zensical.toml index 4dbd2bf2e..d78ed4b39 100644 --- a/zensical.toml +++ b/zensical.toml @@ -18,7 +18,10 @@ nav = [ "setup.md", "usage.md", "configuration.md", - "administration.md", + { Administration = [ + "administration.md", + { "v3 Migration Guide" = "migration-v3.md" }, + ] }, "advanced_usage.md", "api.md", "development.md", From 8531078a54f9b5d6c0487274a7c21c9c6405730a Mon Sep 17 00:00:00 2001 From: GitHub Actions <41898282+github-actions[bot]@users.noreply.github.com> Date: Fri, 27 Feb 2026 22:39:20 +0000 Subject: [PATCH 06/40] Auto translate strings --- src/locale/en_US/LC_MESSAGES/django.po | 76 +++++++++++++------------- 1 file changed, 38 insertions(+), 38 deletions(-) diff --git a/src/locale/en_US/LC_MESSAGES/django.po b/src/locale/en_US/LC_MESSAGES/django.po index be2e043e1..eee1d2493 100644 --- a/src/locale/en_US/LC_MESSAGES/django.po +++ b/src/locale/en_US/LC_MESSAGES/django.po @@ -2,7 +2,7 @@ msgid "" msgstr "" "Project-Id-Version: paperless-ngx\n" "Report-Msgid-Bugs-To: \n" -"POT-Creation-Date: 2026-02-26 18:09+0000\n" +"POT-Creation-Date: 2026-02-27 22:38+0000\n" "PO-Revision-Date: 2022-02-17 04:17\n" "Last-Translator: \n" "Language-Team: English\n" @@ -1856,151 +1856,151 @@ msgstr "" msgid "paperless application settings" msgstr "" -#: paperless/settings.py:819 +#: paperless/settings/__init__.py:746 msgid "English (US)" msgstr "" -#: paperless/settings.py:820 +#: paperless/settings/__init__.py:747 msgid "Arabic" msgstr "" -#: paperless/settings.py:821 +#: paperless/settings/__init__.py:748 msgid "Afrikaans" msgstr "" -#: paperless/settings.py:822 +#: paperless/settings/__init__.py:749 msgid "Belarusian" msgstr "" -#: paperless/settings.py:823 +#: paperless/settings/__init__.py:750 msgid "Bulgarian" msgstr "" -#: paperless/settings.py:824 +#: paperless/settings/__init__.py:751 msgid "Catalan" msgstr "" -#: paperless/settings.py:825 +#: paperless/settings/__init__.py:752 msgid "Czech" msgstr "" -#: paperless/settings.py:826 +#: paperless/settings/__init__.py:753 msgid "Danish" msgstr "" -#: paperless/settings.py:827 +#: paperless/settings/__init__.py:754 msgid "German" msgstr "" -#: paperless/settings.py:828 +#: paperless/settings/__init__.py:755 msgid "Greek" msgstr "" -#: paperless/settings.py:829 +#: paperless/settings/__init__.py:756 msgid "English (GB)" msgstr "" -#: paperless/settings.py:830 +#: paperless/settings/__init__.py:757 msgid "Spanish" msgstr "" -#: paperless/settings.py:831 +#: paperless/settings/__init__.py:758 msgid "Persian" msgstr "" -#: paperless/settings.py:832 +#: paperless/settings/__init__.py:759 msgid "Finnish" msgstr "" -#: paperless/settings.py:833 +#: paperless/settings/__init__.py:760 msgid "French" msgstr "" -#: paperless/settings.py:834 +#: paperless/settings/__init__.py:761 msgid "Hungarian" msgstr "" -#: paperless/settings.py:835 +#: paperless/settings/__init__.py:762 msgid "Indonesian" msgstr "" -#: paperless/settings.py:836 +#: paperless/settings/__init__.py:763 msgid "Italian" msgstr "" -#: paperless/settings.py:837 +#: paperless/settings/__init__.py:764 msgid "Japanese" msgstr "" -#: paperless/settings.py:838 +#: paperless/settings/__init__.py:765 msgid "Korean" msgstr "" -#: paperless/settings.py:839 +#: paperless/settings/__init__.py:766 msgid "Luxembourgish" msgstr "" -#: paperless/settings.py:840 +#: paperless/settings/__init__.py:767 msgid "Norwegian" msgstr "" -#: paperless/settings.py:841 +#: paperless/settings/__init__.py:768 msgid "Dutch" msgstr "" -#: paperless/settings.py:842 +#: paperless/settings/__init__.py:769 msgid "Polish" msgstr "" -#: paperless/settings.py:843 +#: paperless/settings/__init__.py:770 msgid "Portuguese (Brazil)" msgstr "" -#: paperless/settings.py:844 +#: paperless/settings/__init__.py:771 msgid "Portuguese" msgstr "" -#: paperless/settings.py:845 +#: paperless/settings/__init__.py:772 msgid "Romanian" msgstr "" -#: paperless/settings.py:846 +#: paperless/settings/__init__.py:773 msgid "Russian" msgstr "" -#: paperless/settings.py:847 +#: paperless/settings/__init__.py:774 msgid "Slovak" msgstr "" -#: paperless/settings.py:848 +#: paperless/settings/__init__.py:775 msgid "Slovenian" msgstr "" -#: paperless/settings.py:849 +#: paperless/settings/__init__.py:776 msgid "Serbian" msgstr "" -#: paperless/settings.py:850 +#: paperless/settings/__init__.py:777 msgid "Swedish" msgstr "" -#: paperless/settings.py:851 +#: paperless/settings/__init__.py:778 msgid "Turkish" msgstr "" -#: paperless/settings.py:852 +#: paperless/settings/__init__.py:779 msgid "Ukrainian" msgstr "" -#: paperless/settings.py:853 +#: paperless/settings/__init__.py:780 msgid "Vietnamese" msgstr "" -#: paperless/settings.py:854 +#: paperless/settings/__init__.py:781 msgid "Chinese Simplified" msgstr "" -#: paperless/settings.py:855 +#: paperless/settings/__init__.py:782 msgid "Chinese Traditional" msgstr "" From 0bc032a67d64670fd0fc17f69c23b14b1de23007 Mon Sep 17 00:00:00 2001 From: Jan Kleine Date: Sat, 28 Feb 2026 00:24:11 +0100 Subject: [PATCH 07/40] Development: improve test portability (#12187) * Fix: improve test portability * Make settings always consistent * Make a few more tests deterministic wrt settings * Dont pollute settings for this one * Fix timezone issue with mail parser * Update test_parser.py * Uh, I guess OCR gives variants for this --------- Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com> --- src/documents/tests/test_api_uisettings.py | 10 ++++++++++ src/documents/tests/test_barcodes.py | 1 + src/documents/tests/test_file_handling.py | 4 ++-- src/documents/tests/test_management_fuzzy.py | 9 +++++++-- src/documents/tests/utils.py | 10 +++++----- src/paperless/tests/test_adapter.py | 14 +++++++++----- src/paperless/tests/test_views.py | 19 +++++++++---------- src/paperless_mail/parsers.py | 5 ++++- src/paperless_mail/tests/test_parsers.py | 12 +++++++----- src/paperless_tesseract/tests/test_parser.py | 15 +++++++++++++-- 10 files changed, 67 insertions(+), 32 deletions(-) diff --git a/src/documents/tests/test_api_uisettings.py b/src/documents/tests/test_api_uisettings.py index ec973e2f1..811b77286 100644 --- a/src/documents/tests/test_api_uisettings.py +++ b/src/documents/tests/test_api_uisettings.py @@ -21,6 +21,16 @@ class TestApiUiSettings(DirectoriesMixin, APITestCase): self.test_user.save() self.client.force_authenticate(user=self.test_user) + @override_settings( + APP_TITLE=None, + APP_LOGO=None, + AUDIT_LOG_ENABLED=True, + EMPTY_TRASH_DELAY=30, + ENABLE_UPDATE_CHECK="default", + EMAIL_ENABLED=False, + GMAIL_OAUTH_ENABLED=False, + OUTLOOK_OAUTH_ENABLED=False, + ) def test_api_get_ui_settings(self) -> None: response = self.client.get(self.ENDPOINT, format="json") self.assertEqual(response.status_code, status.HTTP_200_OK) diff --git a/src/documents/tests/test_barcodes.py b/src/documents/tests/test_barcodes.py index 2d9ac58f0..b1847f2b4 100644 --- a/src/documents/tests/test_barcodes.py +++ b/src/documents/tests/test_barcodes.py @@ -919,6 +919,7 @@ class TestTagBarcode(DirectoriesMixin, SampleDirMixin, GetReaderPluginMixin, Tes @override_settings( CONSUMER_ENABLE_TAG_BARCODE=True, CONSUMER_TAG_BARCODE_MAPPING={"ASN(.*)": "\\g<1>"}, + CONSUMER_ENABLE_ASN_BARCODE=False, ) def test_scan_file_for_many_custom_tags(self) -> None: """ diff --git a/src/documents/tests/test_file_handling.py b/src/documents/tests/test_file_handling.py index 9b6f96ce1..1e35b96bc 100644 --- a/src/documents/tests/test_file_handling.py +++ b/src/documents/tests/test_file_handling.py @@ -329,14 +329,14 @@ class TestFileHandling(DirectoriesMixin, FileSystemAssertsMixin, TestCase): FILENAME_FORMAT="{added_year}-{added_month}-{added_day}", ) def test_added_year_month_day(self) -> None: - d1 = timezone.make_aware(datetime.datetime(232, 1, 9, 1, 1, 1)) + d1 = timezone.make_aware(datetime.datetime(1232, 1, 9, 1, 1, 1)) doc1 = Document.objects.create( title="doc1", mime_type="application/pdf", added=d1, ) - self.assertEqual(generate_filename(doc1), Path("232-01-09.pdf")) + self.assertEqual(generate_filename(doc1), Path("1232-01-09.pdf")) doc1.added = timezone.make_aware(datetime.datetime(2020, 11, 16, 1, 1, 1)) diff --git a/src/documents/tests/test_management_fuzzy.py b/src/documents/tests/test_management_fuzzy.py index b3d03ecea..195a3450d 100644 --- a/src/documents/tests/test_management_fuzzy.py +++ b/src/documents/tests/test_management_fuzzy.py @@ -140,7 +140,7 @@ class TestFuzzyMatchCommand(TestCase): mime_type="application/pdf", filename="final_test.pdf", ) - stdout, _ = self.call_command("--no-progress-bar") + stdout, _ = self.call_command("--no-progress-bar", "--processes", "1") lines = [x.strip() for x in stdout.splitlines() if x.strip()] self.assertEqual(len(lines), 3) for line in lines: @@ -183,7 +183,12 @@ class TestFuzzyMatchCommand(TestCase): self.assertEqual(Document.objects.count(), 3) - stdout, _ = self.call_command("--delete", "--no-progress-bar") + stdout, _ = self.call_command( + "--delete", + "--no-progress-bar", + "--processes", + "1", + ) self.assertIn( "The command is configured to delete documents. Use with caution", diff --git a/src/documents/tests/utils.py b/src/documents/tests/utils.py index f099cd92e..346d895aa 100644 --- a/src/documents/tests/utils.py +++ b/src/documents/tests/utils.py @@ -33,11 +33,11 @@ from documents.plugins.helpers import ProgressStatusOptions def setup_directories(): dirs = namedtuple("Dirs", ()) - dirs.data_dir = Path(tempfile.mkdtemp()) - dirs.scratch_dir = Path(tempfile.mkdtemp()) - dirs.media_dir = Path(tempfile.mkdtemp()) - dirs.consumption_dir = Path(tempfile.mkdtemp()) - dirs.static_dir = Path(tempfile.mkdtemp()) + dirs.data_dir = Path(tempfile.mkdtemp()).resolve() + dirs.scratch_dir = Path(tempfile.mkdtemp()).resolve() + dirs.media_dir = Path(tempfile.mkdtemp()).resolve() + dirs.consumption_dir = Path(tempfile.mkdtemp()).resolve() + dirs.static_dir = Path(tempfile.mkdtemp()).resolve() dirs.index_dir = dirs.data_dir / "index" dirs.originals_dir = dirs.media_dir / "documents" / "originals" dirs.thumbnail_dir = dirs.media_dir / "documents" / "thumbnails" diff --git a/src/paperless/tests/test_adapter.py b/src/paperless/tests/test_adapter.py index 767749dd2..fcbd21b12 100644 --- a/src/paperless/tests/test_adapter.py +++ b/src/paperless/tests/test_adapter.py @@ -78,11 +78,15 @@ class TestCustomAccountAdapter(TestCase): adapter = get_adapter() # Test when PAPERLESS_URL is None - expected_url = f"https://foo.org{reverse('account_reset_password_from_key', kwargs={'uidb36': 'UID', 'key': 'KEY'})}" - self.assertEqual( - adapter.get_reset_password_from_key_url("UID-KEY"), - expected_url, - ) + with override_settings( + PAPERLESS_URL=None, + ACCOUNT_DEFAULT_HTTP_PROTOCOL="https", + ): + expected_url = f"https://foo.org{reverse('account_reset_password_from_key', kwargs={'uidb36': 'UID', 'key': 'KEY'})}" + self.assertEqual( + adapter.get_reset_password_from_key_url("UID-KEY"), + expected_url, + ) # Test when PAPERLESS_URL is not None with override_settings(PAPERLESS_URL="https://bar.com"): diff --git a/src/paperless/tests/test_views.py b/src/paperless/tests/test_views.py index 76bf694fc..ab8941538 100644 --- a/src/paperless/tests/test_views.py +++ b/src/paperless/tests/test_views.py @@ -1,7 +1,7 @@ import tempfile from pathlib import Path -from django.conf import settings +from django.test import override_settings def test_favicon_view(client): @@ -11,15 +11,14 @@ def test_favicon_view(client): favicon_path.parent.mkdir(parents=True, exist_ok=True) favicon_path.write_bytes(b"FAKE ICON DATA") - settings.STATIC_ROOT = static_dir - - response = client.get("/favicon.ico") - assert response.status_code == 200 - assert response["Content-Type"] == "image/x-icon" - assert b"".join(response.streaming_content) == b"FAKE ICON DATA" + with override_settings(STATIC_ROOT=static_dir): + response = client.get("/favicon.ico") + assert response.status_code == 200 + assert response["Content-Type"] == "image/x-icon" + assert b"".join(response.streaming_content) == b"FAKE ICON DATA" def test_favicon_view_missing_file(client): - settings.STATIC_ROOT = Path(tempfile.mkdtemp()) - response = client.get("/favicon.ico") - assert response.status_code == 404 + with override_settings(STATIC_ROOT=Path(tempfile.mkdtemp())): + response = client.get("/favicon.ico") + assert response.status_code == 404 diff --git a/src/paperless_mail/parsers.py b/src/paperless_mail/parsers.py index 22835de89..86cba23ab 100644 --- a/src/paperless_mail/parsers.py +++ b/src/paperless_mail/parsers.py @@ -5,6 +5,7 @@ from pathlib import Path from bleach import clean from bleach import linkify from django.conf import settings +from django.utils import timezone from django.utils.timezone import is_naive from django.utils.timezone import make_aware from gotenberg_client import GotenbergClient @@ -332,7 +333,9 @@ class MailDocumentParser(DocumentParser): if data["attachments"]: data["attachments_label"] = "Attachments" - data["date"] = clean_html(mail.date.astimezone().strftime("%Y-%m-%d %H:%M")) + data["date"] = clean_html( + timezone.localtime(mail.date).strftime("%Y-%m-%d %H:%M"), + ) data["content"] = clean_html(mail.text.strip()) from django.template.loader import render_to_string diff --git a/src/paperless_mail/tests/test_parsers.py b/src/paperless_mail/tests/test_parsers.py index 061771d47..9746731f8 100644 --- a/src/paperless_mail/tests/test_parsers.py +++ b/src/paperless_mail/tests/test_parsers.py @@ -6,6 +6,7 @@ from unittest import mock import httpx import pytest from django.test.html import parse_html +from django.utils import timezone from pytest_django.fixtures import SettingsWrapper from pytest_httpx import HTTPXMock from pytest_mock import MockerFixture @@ -634,13 +635,14 @@ class TestParser: THEN: - Resulting HTML is as expected """ - mail = mail_parser.parse_file_to_message(html_email_file) - html_file = mail_parser.mail_to_html(mail) + with timezone.override("UTC"): + mail = mail_parser.parse_file_to_message(html_email_file) + html_file = mail_parser.mail_to_html(mail) - expected_html = parse_html(html_email_html_file.read_text()) - actual_html = parse_html(html_file.read_text()) + expected_html = parse_html(html_email_html_file.read_text()) + actual_html = parse_html(html_file.read_text()) - assert expected_html == actual_html + assert expected_html == actual_html def test_generate_pdf_from_mail( self, diff --git a/src/paperless_tesseract/tests/test_parser.py b/src/paperless_tesseract/tests/test_parser.py index ce8ef4c9e..e9c05b03e 100644 --- a/src/paperless_tesseract/tests/test_parser.py +++ b/src/paperless_tesseract/tests/test_parser.py @@ -1,5 +1,6 @@ import shutil import tempfile +import unicodedata import uuid from pathlib import Path from unittest import mock @@ -847,8 +848,18 @@ class TestParser(DirectoriesMixin, FileSystemAssertsMixin, TestCase): "application/pdf", ) - # Copied from the PDF to here. Don't even look at it - self.assertIn("ةﯾﻠﺧﺎدﻻ ةرازو", parser.get_text()) + # OCR output for RTL text varies across platforms/versions due to + # bidi controls and presentation forms; normalize before assertion. + normalized_text = "".join( + char + for char in unicodedata.normalize("NFKC", parser.get_text()) + if unicodedata.category(char) != "Cf" and not char.isspace() + ) + + self.assertIn("ةرازو", normalized_text) + self.assertTrue( + any(token in normalized_text for token in ("ةیلخادلا", "الاخليد")), + ) @mock.patch("ocrmypdf.ocr") def test_gs_rendering_error(self, m) -> None: From 1dd3a62bc2f7cc248db587fb4b6be395fc74305c Mon Sep 17 00:00:00 2001 From: shamoon <4887959+shamoon@users.noreply.github.com> Date: Fri, 27 Feb 2026 16:22:29 -0800 Subject: [PATCH 08/40] Fixhancement: show sequential + id version labels, fix padding (#12196) --- .../document-version-dropdown.component.html | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src-ui/src/app/components/document-detail/document-version-dropdown/document-version-dropdown.component.html b/src-ui/src/app/components/document-detail/document-version-dropdown/document-version-dropdown.component.html index 4bb92fd02..aacae0668 100644 --- a/src-ui/src/app/components/document-detail/document-version-dropdown/document-version-dropdown.component.html +++ b/src-ui/src/app/components/document-detail/document-version-dropdown/document-version-dropdown.component.html @@ -57,7 +57,7 @@ }
@for (version of versions; track version.id) { -