Commit Graph
157 Commits
Author SHA1 Message Date
stumpylogandClaude Opus 5 46c150eb67 Deduplicate the outbound connect guard's sync and async paths
The two connect_tcp implementations each repeated the per-attempt budget
arithmetic verbatim, and only the sync one delegated resolution and error
mapping to a helper, so a drift between the copies could silently give one
stack a different timeout policy from the other. In the IMAP client,
_connect_pinned re-asserted a fact its only caller had already established,
which reads as a runtime invariant check on a security-relevant path when it
is only a narrowing aid.

Move the budget arithmetic into one helper called from both loops, add an
async twin of the resolve helper so the two loop bodies differ only by await,
and pass the narrowed address tuple into _connect_pinned instead of asserting
it. The PinnedIMAP4 docstring now spells out that no pinning and pinning that
yielded nothing are different things, and the monotonic clock seam says why it
exists.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-22 12:49:06 -07:00
stumpylogandClaude Opus 5 283e49b1ee Match get_mailbox annotations to what callers actually pass
get_mailbox declared port and security as always-present types, but
MailAccount.imap_port is nullable and imap_security is stored as a plain
integer, so type checking flagged every call site as passing the wrong
type.

Widen the annotations to port: int | None and security: int, matching the
model fields; IntegerChoices members still compare equal to plain ints, so
the existing branching in get_mailbox is unaffected.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-22 12:49:05 -07:00
stumpylogandClaude Opus 5 3130fc3a7c Use shared outbound resolution for IMAP host pinning
get_mailbox validates the IMAP host with resolve_public_addresses and keeps
its existing error messages; the pinned client dials typed addresses.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-22 12:49:05 -07:00
stumpylogandClaude Opus 5 17a91c385d Type outbound block errors and classify addresses with is_global
is_public_ip now takes an ipaddress object and relies on is_global, keeping
multicast and the NAT64 well-known prefix as explicit extra exclusions.
Adds OutboundRequestBlockedError and HostResolutionError, both picklable so
Celery keeps them intact on task failure.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-22 12:47:40 -07:00
shamoon f1679a2ee3 Fix: truncate mail subjects to field max length (#13991) 2026-09-05 01:52:18 -07:00
Trenton H 713c857a08 Fix: Handle Celery mail task chord errors (#13936)
* Fix: mail rule loops forever when all attachments are duplicates

When every attachment in a mail is rejected as a duplicate, the chord's
header tasks all fail. Celery's default task_allow_error_cb_on_chord_header
skips the error callback in that case, so no ProcessedMail row is ever
created, and the same mail is refetched and reprocessed on every poll for
as long as it stays in the rule's maximum_age window.

* Minor simplifications and cleanup
2026-09-02 08:41:30 -07:00
shamoon f6f37898e8 Chore: use PinnedIMAP4 2026-08-27 00:21:02 -07:00
Trenton H 2ea4a792fe Performance: More efficient mail fetching to reduce the amount of data pulled (#13432) 2026-08-04 15:35:30 +00:00
Hugo Beauzée-Luyssen f0d934259c Fix: scope ProcessedMail UID deduplication by IMAP UIDVALIDITY (#12839) 2026-07-15 13:43:16 -07:00
Trenton H 8ed4bf2011 Fix: Apply unicode normalization to all paths and path components (#12993) 2026-06-13 12:45:54 +00:00
Trenton H 58789e5061 Chore: Structured consume task return values (#12612) 2026-04-20 13:19:54 -07:00
Trenton H fbf4e32646 Chore: Converts all call sites and test asserts to use apply_async and headers (#12591) 2026-04-20 11:40:04 -07:00
shamoon 8f036c9521 Merge branch 'main' into dev
# Conflicts:
#	docs/usage.md
#	src/documents/signals/handlers.py
#	src/documents/tests/test_api_documents.py
#	src/documents/views.py
2026-04-14 15:11:23 -07:00
shamoon e7c7978d67 Enhancement: allow opt-in blocking internal mail hosts (#12502) 2026-04-03 03:24:28 +00:00
shamoon 3cfe9fa2a8 Fixhancement: default mail -created correspondent matching to exact (#12414) 2026-03-21 16:15:18 -07:00
shamoon 90ae55252f Enhancement: prevent duplicate mail processing across rules (#12159) 2026-02-26 00:58:45 +00:00
shamoon a467df0755 Enhancement: option to stop processing further mail rules (#12053) 2026-02-13 17:33:29 +00:00
Sebastian Steinbeißer 3b5ffbf9fa Chore(mypy): Annotate None returns for typing improvements (#11213) 2026-02-02 08:44:12 -08:00
shamoon 7a1aae7749 Fix: set match value for correspondents created by mail rule (#10820) 2025-09-09 09:14:48 -07:00
dependabot[bot]andTrenton Holmes 9889c59d3d Chore(deps): Bump the small-changes group across 1 directory with 7 updates (#10347)
* Chore(deps): Bump the small-changes group across 1 directory with 7 updates

Bumps the small-changes group with 7 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [concurrent-log-handler](https://github.com/Preston-Landers/concurrent-log-handler) | `0.9.26` | `0.9.28` |
| [dateparser](https://github.com/scrapinghub/dateparser) | `1.2.1` | `1.2.2` |
| [imap-tools](https://github.com/ikvk/imap_tools) | `1.10.0` | `1.11.0` |
| [ocrmypdf](https://github.com/ocrmypdf/OCRmyPDF) | `16.10.2` | `16.10.4` |
| [pathvalidate](https://github.com/thombashi/pathvalidate) | `3.2.3` | `3.3.1` |
| [python-dotenv](https://github.com/theskumar/python-dotenv) | `1.1.0` | `1.1.1` |
| [scikit-learn](https://github.com/scikit-learn/scikit-learn) | `1.6.1` | `1.7.0` |



Updates `concurrent-log-handler` from 0.9.26 to 0.9.28
- [Release notes](https://github.com/Preston-Landers/concurrent-log-handler/releases)
- [Changelog](https://github.com/Preston-Landers/concurrent-log-handler/blob/master/CHANGELOG.md)
- [Commits](https://github.com/Preston-Landers/concurrent-log-handler/compare/0.9.26...0.9.28)

Updates `dateparser` from 1.2.1 to 1.2.2
- [Release notes](https://github.com/scrapinghub/dateparser/releases)
- [Changelog](https://github.com/scrapinghub/dateparser/blob/master/HISTORY.rst)
- [Commits](https://github.com/scrapinghub/dateparser/compare/v1.2.1...v1.2.2)

Updates `imap-tools` from 1.10.0 to 1.11.0
- [Release notes](https://github.com/ikvk/imap_tools/releases)
- [Changelog](https://github.com/ikvk/imap_tools/blob/master/docs/release_notes.rst)
- [Commits](https://github.com/ikvk/imap_tools/compare/v1.10.0...v1.11.0)

Updates `ocrmypdf` from 16.10.2 to 16.10.4
- [Release notes](https://github.com/ocrmypdf/OCRmyPDF/releases)
- [Changelog](https://github.com/ocrmypdf/OCRmyPDF/blob/main/docs/release_notes.md)
- [Commits](https://github.com/ocrmypdf/OCRmyPDF/compare/v16.10.2...v16.10.4)

Updates `pathvalidate` from 3.2.3 to 3.3.1
- [Release notes](https://github.com/thombashi/pathvalidate/releases)
- [Changelog](https://github.com/thombashi/pathvalidate/blob/master/CHANGELOG.md)
- [Commits](https://github.com/thombashi/pathvalidate/compare/v3.2.3...v3.3.1)

Updates `python-dotenv` from 1.1.0 to 1.1.1
- [Release notes](https://github.com/theskumar/python-dotenv/releases)
- [Changelog](https://github.com/theskumar/python-dotenv/blob/main/CHANGELOG.md)
- [Commits](https://github.com/theskumar/python-dotenv/compare/v1.1.0...v1.1.1)

Updates `scikit-learn` from 1.6.1 to 1.7.0
- [Release notes](https://github.com/scikit-learn/scikit-learn/releases)
- [Commits](https://github.com/scikit-learn/scikit-learn/compare/1.6.1...1.7.0)

---
updated-dependencies:
- dependency-name: concurrent-log-handler
  dependency-version: 0.9.28
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: small-changes
- dependency-name: dateparser
  dependency-version: 1.2.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: small-changes
- dependency-name: imap-tools
  dependency-version: 1.11.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: small-changes
- dependency-name: ocrmypdf
  dependency-version: 16.10.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: small-changes
- dependency-name: pathvalidate
  dependency-version: 3.3.1
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: small-changes
- dependency-name: python-dotenv
  dependency-version: 1.1.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: small-changes
- dependency-name: scikit-learn
  dependency-version: 1.7.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: small-changes
...

Signed-off-by: dependabot[bot] <support@github.com>

* Removes the setup_logging_queues call which is no longer needed

* Renames the MailboxTLS

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Trenton Holmes <797416+stumpylog@users.noreply.github.com>
2025-07-08 09:29:47 -07:00
Trenton H 3d2a3ede71 Chore: Updates dependency groups (#10339) 2025-07-07 17:37:58 -07:00
Sebastian Steinbeißerandshamoon 07882b918b Chore: switch from os.path to pathlib.Path (#9933)
Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
2025-06-18 17:16:59 +00:00
shamoon c30cf2e0cd Chore: fix naive datetime warnings 2025-06-03 12:42:18 -07:00
Sebastian Steinbeißer e560fa3be0 Chore: Enable ruff FBT (#8645) 2025-02-07 09:12:03 -08:00
dependabot[bot]andshamoon 20ec8cb57b Chore(deps-dev): Bump the development group with 2 updates (#8841)
* Chore(deps-dev): Bump the development group with 2 updates

Bumps the development group with 2 updates: [ruff](https://github.com/astral-sh/ruff) and [mkdocs-material](https://github.com/squidfunk/mkdocs-material).


Updates `ruff` from 0.8.6 to 0.9.2
- [Release notes](https://github.com/astral-sh/ruff/releases)
- [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md)
- [Commits](https://github.com/astral-sh/ruff/compare/0.8.6...0.9.2)

Updates `mkdocs-material` from 9.5.49 to 9.5.50
- [Release notes](https://github.com/squidfunk/mkdocs-material/releases)
- [Changelog](https://github.com/squidfunk/mkdocs-material/blob/master/CHANGELOG)
- [Commits](https://github.com/squidfunk/mkdocs-material/compare/9.5.49...9.5.50)

---
updated-dependencies:
- dependency-name: ruff
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: development
- dependency-name: mkdocs-material
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: development
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update .pre-commit-config.yaml

* Run new ruff format

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
2025-01-21 19:22:25 +00:00
shamoon 2353f7c2db Feature: OAuth2 Gmail and Outlook email support (#7866) 2024-10-10 20:57:32 +00:00
Martin Richtarskyandshamoon b3487f1843 Enhancement: check for mail destination directory, log post-consume errors (#7808)
---------

Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
2024-10-02 20:21:35 -07:00
shamoon 991c9b0ca4 Enhancement: disable-able mail rules, add toggle to overview (#7810) 2024-09-30 19:42:19 -07:00
Trenton H e6f59472e4 Chore: Drop Python 3.9 support (#7774) 2024-09-26 12:22:24 -07:00
shamoon 4903e4290d Enhancement: re-work mail rule dialog, support multiple include patterns (#7635) 2024-09-05 15:32:03 -07:00
Daniel Bankmannandshamoon ce663398e6 Enhancement: mail message preprocessor for gpg encrypted mails (#7456)
---------

Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
2024-08-29 00:22:44 +00:00
Dennis Melzerandshamoon eaaaa575b8 Enhancement: allow multiple filename attachment exclusion patterns for a mail rule (#5524)
---------

Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
2024-08-27 09:31:46 -07:00
Trenton H de7c22e8d6 Fixes the logging of an email message to be something useful (#6901) 2024-06-03 10:11:31 -07:00
Trenton H b720aa3cd1 Chore: Convert the consumer to a plugin (#6361) 2024-04-18 02:59:14 +00:00
shamoon 4d0e8a338f Fix: Pass rule id to consumed .eml files (#5800) 2024-02-16 22:36:33 -08:00
Trenton H 13201dbfff Ensure all creations of directories create the parents too (#5711) 2024-02-10 11:02:40 -08:00
Trenton H b60e16fe33 Chore: Backend dependencies update (#5676) 2024-02-08 09:48:24 -08:00
luzpazandshamoon 58bf9c552b Documentation: Fix typos with automated tool (#5319)
---------

Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
2024-01-08 16:58:41 +00:00
Thomas FalkenbergandTrenton H 355a434a07 Enhancement: fetch mails in bulk (#5249)
Co-authored-by: Trenton H <797416+stumpylog@users.noreply.github.com>
2024-01-05 10:43:22 -08:00
Trenton Handshamoon 061f33fb05 Feature: Allow setting backend configuration settings via the UI (#5126)
* Saving some start on this

* At least partially working for the tesseract parser

* Problems with migration testing need to figure out

* Work around that error

* Fixes max m_pixels

* Moving the settings to main paperless application

* Starting some consumer options

* More fixes and work

* Fixes these last tests

* Fix max_length on OcrSettings.mode field

* Fix all fields on Common & Ocr settings serializers

* Umbrellla config view

* Revert "Umbrellla config view"

This reverts commit fbaf9f4be30f89afeb509099180158a3406416a5.

* Updates to use a single configuration object for all settings

* Squashed commit of the following:

commit 8a0a49dd57
Author: shamoon <4887959+shamoon@users.noreply.github.com>
Date:   Tue Dec 19 23:02:47 2023 -0800

    Fix formatting

commit 66b2d90c50
Author: shamoon <4887959+shamoon@users.noreply.github.com>
Date:   Tue Dec 19 22:36:35 2023 -0800

    Refactor frontend data models

commit 5723bd8dd8
Author: Adam Bogdał <adam@bogdal.pl>
Date:   Wed Dec 20 01:17:43 2023 +0100

    Fix: speed up admin panel for installs with a large number of documents (#5052)

commit 9b08ce1761
Author: shamoon <4887959+shamoon@users.noreply.github.com>
Date:   Tue Dec 19 15:18:51 2023 -0800

    Update PULL_REQUEST_TEMPLATE.md

commit a6248bec2d
Author: shamoon <4887959+shamoon@users.noreply.github.com>
Date:   Tue Dec 19 15:02:05 2023 -0800

    Chore: Update Angular to v17 (#4980)

commit b1f6f52486
Author: shamoon <4887959+shamoon@users.noreply.github.com>
Date:   Tue Dec 19 13:53:56 2023 -0800

    Fix: Dont allow null custom_fields property via API (#5063)

commit 638d9970fd
Author: shamoon <4887959+shamoon@users.noreply.github.com>
Date:   Tue Dec 19 13:43:50 2023 -0800

    Enhancement: symmetric document links (#4907)

commit 5e8de4c1da
Author: shamoon <4887959+shamoon@users.noreply.github.com>
Date:   Tue Dec 19 12:45:04 2023 -0800

    Enhancement: shared icon & shared by me filter (#4859)

commit 088bad9030
Author: Trenton H <797416+stumpylog@users.noreply.github.com>
Date:   Tue Dec 19 12:04:03 2023 -0800

    Bulk updates all the backend libraries (#5061)

* Saving some work on frontend config

* Very basic but dynamically-generated config form

* Saving work on slightly less ugly frontend config

* JSON validation for user_args field

* Fully dynamic config form

* Adds in some additional validators for a nicer error message

* Cleaning up the testing and coverage more

* Reverts unintentional change

* Adds documentation about the settings and the precedence

* Couple more commenting and style fixes

---------

Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
2023-12-29 15:42:56 -08:00
Trenton H ffad42615f Fixes the case where a mail attachment has no filename to use (#5117) 2023-12-27 17:38:22 -08:00
Trenton H fbf1a051a2 Use the attachment filename so downstream template matching works against it (#4931) 2023-12-11 09:08:42 -08:00
Trenton Handshamoon 6e371ac5ac Enhancement: Allow excluding mail attachments by name (#4691)
* Adds new filtering to exclude attachments from processing

* Frontend use include / exclude mail rule filename filters

---------

Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
2023-12-02 08:26:19 -08:00
Trenton H 07989bc2fa Handles the case where other might have a None field as well (#4598) 2023-11-15 16:23:38 -08:00
Trenton H dcc2cc5001 Only get or create a Correspondent if the email or attachment matches rule filters (#4431) 2023-10-26 00:34:52 +00:00
shamoon 9712ac109d Feature: consumption templates (#4196)
* Initial implementation of consumption templates

* Frontend implementation of consumption templates

Testing

* Support consumption template source

* order templates, automatically add permissions

* Support title assignment in consumption templates

* Refactoring, filters to and, show sources on list

Show sources on template list, update some translation strings

Make filters and

minor testing

* Update strings

* Only update django-multiselectfield

* Basic docs, document some methods

* Improve testing coverage, template multi-assignment merges
2023-09-22 16:53:13 -07:00
Trenton Holmes 650c816a7b Removes support for Python 3.8 and lower from the code base 2023-09-10 11:42:59 -07:00
Trenton H d532913d56 Unlocks all things which were locked due to ARMv7, etc 2023-09-08 11:21:43 -07:00
Trenton H df82ac8ac4 Adjusts to use a different loading of certificates and updates the docs for it 2023-08-23 08:22:01 -07:00
Trenton H 16adddc803 Allow users to set a combined certificte and key file for additional certificates in the SSL context 2023-08-23 08:22:01 -07:00