df1ddb15cc
* Tantivy: get permissions by chunks -40% indexing time compared to previous commit * Make progress bar process one by one with chunk -15% indexing time compared to previous commit * Prefetch FK + iterate over chunk from SQL Prefetch additional needed data (note user, custom field content) -20% indexing time compared to previous commit * Reindex: increase Tantivy heap size from 128 to 512MB Gains probably vary depending on the machine, but it seems a sweet spot compatible with low-end hardware. * Reindex: optimization on permission fetching and autocomplete word set -10% indexing time compared to previous commit * Autocomplete analyzer python->rust Splits words with underscore compared to the python analyzer. E.g.: "blue_print" -> ["blue", "print"] It can still be found with the "blue_print" keyword, as the search string is also split in two words. -50% indexing time compared to previous commit (indexing is twice faster!) * Index bigram for CJK content only Inedxing time slightly longer (~3%), but since the non-CJK content is not indexed, bigram searchs will be slightly optimized. * Fix group-based view_document permissions missing from bulk rebuild _bulk_get_viewer_ids only queried UserObjectPermission, dropping the group-permission expansion that get_users_with_perms(with_group_users=True) performs for the non-batched per-document indexing path. A user who could only see a document via group membership would lose search access to it after any full reindex. Also query GroupObjectPermission and expand group membership to user ids, matching the existing single-document behavior. * Yield (document, viewer_ids) pairs from _DocumentViewerStream Previously _DocumentViewerStream.__iter__ yielded plain Document objects while the matching viewer ids were exposed through a separate mutable attribute (viewer_ids_by_pk), overwritten each time the generator crossed a chunk boundary. rebuild() read that attribute out-of-band per document. This only worked because the current iter_wrapper (a plain progress-bar passthrough) happens to consume the stream in strict lock-step with no lookahead. Any wrapper that buffers, batches, or reorders would silently pair a document with the wrong chunk's viewer ids. Yield the pair directly so the association travels with the document regardless of how iter_wrapper consumes the stream, and drop the now-unneeded viewer_ids_by_pk attribute. * Add --heap-size-mb CLI arg to document_index reindex writer_heap_bytes was hardcoded at 512MB with no way to tune it. Expose it as a manual-rebuild-only CLI arg rather than a settings/env var, per review feedback, so lower-memory hosts can reduce it without a wider config surface. Defaults to unset so TantivyBackend.rebuild's own default stays the single source of truth. --------- Co-authored-by: stumpylog <797416+stumpylog@users.noreply.github.com>
Paperless-ngx
Paperless-ngx is a document management system that transforms your physical documents into a searchable online archive so you can keep, well, less paper.
Paperless-ngx is the official successor to the original Paperless & Paperless-ng projects and is designed to distribute the responsibility of advancing and supporting the project among a team of people. Consider joining us!
Thanks to the generous folks at DigitalOcean, a demo is available at demo.paperless-ngx.com using login demo / demo. Note: demo content is reset frequently and confidential information should not be uploaded.
This project is supported by:
Features
A full list of features and screenshots are available in the documentation.
Getting started
The easiest way to deploy paperless is docker compose. The files in the /docker/compose directory are configured to pull the image from the GitHub container registry.
If you'd like to jump right in, you can configure a docker compose environment with our install script:
bash -c "$(curl -L https://raw.githubusercontent.com/paperless-ngx/paperless-ngx/main/install-paperless-ngx.sh)"
More details and step-by-step guides for alternative installation methods can be found in the documentation.
Migrating from Paperless-ng is easy, just drop in the new docker image! See the documentation on migrating for more details.
Documentation
The documentation for Paperless-ngx is available at https://docs.paperless-ngx.com.
Contributing
If you feel like contributing to the project, please do! Bug fixes, enhancements, visual fixes etc. are always welcome. If you want to implement something big: Please start a discussion about that! The documentation has some basic information on how to get started.
Community Support
People interested in continuing the work on paperless-ngx are encouraged to reach out here on github and in the Matrix Room. If you would like to contribute to the project on an ongoing basis there are multiple teams (frontend, ci/cd, etc) that could use your help so please reach out!
Translation
Paperless-ngx is available in many languages that are coordinated on Crowdin. If you want to help out by translating paperless-ngx into your language, please head over to https://crowdin.com/project/paperless-ngx, and thank you! More details can be found in CONTRIBUTING.md.
Feature Requests
Feature requests can be submitted via GitHub Discussions, you can search for existing ideas, add your own and vote for the ones you care about.
Bugs
For bugs please open an issue or start a discussion if you have questions.
Related Projects
Please see the wiki for a user-maintained list of related projects and software that is compatible with Paperless-ngx.
Important Note
Document scanners are typically used to scan sensitive documents like your social insurance number, tax records, invoices, etc. Paperless-ngx should never be run on an untrusted host because information is stored in clear text without encryption. No guarantees are made regarding security (but we do try!) and you use the app at your own risk. The safest way to run Paperless-ngx is on a local server in your own home with backups in place.