Commit Graph

176 Commits

Author SHA1 Message Date
Trenton H 8ed4bf2011 Fix: Apply unicode normalization to all paths and path components (#12993) 2026-06-13 12:45:54 +00:00
Trenton H 14fe520319 Chore: Update typing and baselines again (#12641)
a
2026-04-28 09:28:05 -07:00
shamoon 21ff254ffc Merge branch 'main' into dev 2026-04-26 19:12:40 -07:00
shamoon e5561ba06f Fix: correctly scope mail account enumeration (#12636) 2026-04-24 10:15:59 -07:00
Trenton H 50ec987a81 Chore: Refactors all of the mail tests to use model factories instead of bare create (#12613) 2026-04-20 15:43:43 -07:00
Trenton H fbf4e32646 Chore: Converts all call sites and test asserts to use apply_async and headers (#12591) 2026-04-20 11:40:04 -07:00
shamoon 8f036c9521 Merge branch 'main' into dev
# Conflicts:
#	docs/usage.md
#	src/documents/signals/handlers.py
#	src/documents/tests/test_api_documents.py
#	src/documents/views.py
2026-04-14 15:11:23 -07:00
shamoon e7c7978d67 Enhancement: allow opt-in blocking internal mail hosts (#12502) 2026-04-03 03:24:28 +00:00
shamoon ae0474450f Chore: logger, response and template sanitization cleanup (#12439) 2026-03-26 07:36:02 -07:00
shamoon 3cfe9fa2a8 Fixhancement: default mail -created correspondent matching to exact (#12414) 2026-03-21 16:15:18 -07:00
shamoon 0f84af27d0 Merge branch 'main' into dev
# Conflicts:
#	docs/setup.md
#	src-ui/src/main.ts
#	src/documents/tests/test_api_bulk_edit.py
#	src/documents/tests/test_api_custom_fields.py
#	src/documents/tests/test_api_search.py
#	src/documents/tests/test_api_status.py
#	src/documents/tests/test_workflows.py
#	src/paperless_mail/tests/test_api.py
2026-03-21 02:12:19 -07:00
shamoon 7dbf8bdd4a Fix: enforce permissions when attaching accounts to mail rules 2026-03-21 00:44:28 -07:00
Trenton H c2b8b22fb4 Chore: Convert mail parser to plugin style (#12397)
* Refactor(mail): rename paperless_mail/parsers.py → paperless/parsers/mail.py

Preserve git history for MailDocumentParser by committing the rename
separately before editing, following the project convention.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Refactor(mail): move mail parser tests to paperless/tests/parsers/

Move test_parsers.py → test_mail_parser.py and test_parsers_live.py →
test_mail_parser_live.py alongside the other built-in parser tests,
preserving git history before editing. Update MailDocumentParser import
to the new canonical location.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Chore: move mail parser sample files to paperless/tests/samples/mail/

Relocate all mail test fixtures from src/paperless_mail/tests/samples/ to
src/paperless/tests/samples/mail/ ahead of the parser plugin refactor.
Add the new path to the codespell skip list to prevent false-positive
spell corrections in binary/fixture email files.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Feat(tests): add mail parser fixtures to paperless/tests/parsers/conftest.py

Add mail_samples_dir, per-file sample fixtures, and mail_parser
(context-manager style) to mirror the old paperless_mail conftest
but rooted at the new samples/mail/ location.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Feat(parsers): migrate MailDocumentParser to ParserProtocol

Move the mail parser from paperless_mail/parsers.py to
paperless/parsers/mail.py and refactor it to implement ParserProtocol:

- Class-level name/version/author/url attributes
- supported_mime_types() and score() classmethods (score=20)
- can_produce_archive=False, requires_pdf_rendition=True
- Context manager lifecycle (__enter__/__exit__)
- New parse() signature without mailrule_id kwarg; consumer sets
  parser.mailrule_id before calling parse() instead
- get_text()/get_date()/get_archive_path() accessor methods
- extract_metadata() returning email headers and attachment info

Register MailDocumentParser in the ParserRegistry alongside Text and
Tika parsers. Update consumer, signals, and all import sites to use
the new location. Update tests to use the new accessor API, patch
paths, and context-manager fixture.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Fix(parsers): pop legacy constructor args in mail signal wrapper

MailDocumentParser.__init__ takes no constructor args in the new
protocol. Update the get_parser() signal wrapper to pop logging_group
and progress_callback (passed by the legacy consumer dispatch path)
before instantiating — the same pattern used by TextDocumentParser.

Also update test_mail_parser_receives_mailrule to use the real signal
wrapper (mail_get_parser) instead of MailDocumentParser directly, so
the test exercises the actual dispatch path and matches the new
parse() call signature (no mailrule kwarg).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Bumps this so we can run

* Fixes location of the fixture

* Removes fixtures which were duplicated

* Feat(parsers): add ParserContext and configure() to ParserProtocol

Replace the ad-hoc mailrule_id attribute assignment with a typed,
immutable ParserContext dataclass and a configure() method on the
Protocol:

- ParserContext(frozen=True, slots=True) lives in paperless/parsers/
  alongside ParserProtocol and MetadataEntry; currently carries only
  mailrule_id but is designed to grow with output_type, ocr_mode, and
  ocr_language in a future phase (decoupling parsers from settings.*)
- ParserProtocol.configure(context: ParserContext) -> None is the
  extension point; no-op by default
- MailDocumentParser.configure() reads mailrule_id into _mailrule_id
- TextDocumentParser and TikaDocumentParser implement a no-op configure()
- Consumer calls document_parser.configure(ParserContext(...)) before
  parse(), replacing the isinstance(parser, MailDocumentParser) guard
  and the direct attribute mutation

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Feat(parsers): call configure(ParserContext()) in update_document task

Apply the same new-style parser shim pattern as the consumer to
update_document_content_maybe_archive_file:

- Call __enter__ for Text/Tika parsers after instantiation
- Call configure(ParserContext()) before parse() for all new-style parsers
  (mailrule_id is not available here — this is a re-process of an
  existing document, so the default empty context is correct)
- Call parse(path, mime_type) with 2 args for new-style parsers
- Call get_thumbnail(path, mime_type) with 2 args for new-style parsers
- Call __exit__ instead of cleanup() in the finally block

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Fix(tests): add configure() to DummyParser and missing-method parametrize

ParserProtocol now requires configure(context: ParserContext) -> None.
Update DummyParser in test_registry.py to implement it, and add
'missing-configure' to the test_partial_compliant_fails_isinstance
parametrize list so the new method is covered by the negative test.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Cleans up the reprocess task and generally reduces duplicate of classes

* Corrects the score return

* Updates so we can report a page count for these parsers, assuming we do have an archive produced when called

* Increases test coverage

* One more coverage

* Updates typing

* Updates typing

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-20 09:22:18 -07:00
Trenton H 68fc898042 Fix: Resolve more instances of tests which mutated global states (#12395) 2026-03-19 10:05:07 -07:00
Trenton H 1caa3eb8aa Chore: Disables the system checks for management commands in tests and when unnecessary (#12332) 2026-03-16 15:10:35 +00:00
Jan Kleine 0bc032a67d Development: improve test portability (#12187)
* Fix: improve test portability

* Make settings always consistent

* Make a few more tests deterministic wrt settings

* Dont pollute settings for this one

* Fix timezone issue with mail parser

* Update test_parser.py

* Uh, I guess OCR gives variants for this

---------

Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
2026-02-27 23:24:11 +00:00
shamoon 90ae55252f Enhancement: prevent duplicate mail processing across rules (#12159) 2026-02-26 00:58:45 +00:00
shamoon 5b667621cd Try not to piss off mypy 2026-02-21 17:48:11 -08:00
shamoon 1b912c137a Merge branch 'main' into dev 2026-02-21 17:45:28 -08:00
shamoon 1bb4b9b473 More permissions on mail account test endpoint 2026-02-21 16:47:55 -08:00
shamoon 65ca78e9e7 Security: fix/GHSA-7qqc-wrcw-2fj9 2026-02-21 16:34:33 -08:00
shamoon a467df0755 Enhancement: option to stop processing further mail rules (#12053) 2026-02-13 17:33:29 +00:00
Trenton H 276dc13e3f Chore: Fixes the TO filter chaining so it doesn't reset the messages list + deterministic UIDs (#11987) 2026-02-03 11:31:19 -08:00
Sebastian Steinbeißer 3b5ffbf9fa Chore(mypy): Annotate None returns for typing improvements (#11213) 2026-02-02 08:44:12 -08:00
Trenton H 01b21377af Chore: Use a local http server instead of external to reduce flakiness (#11916) 2026-01-28 03:57:12 +00:00
Trenton H c84f2f04b3 Chore: Switch to a local IMAP server instead of a real email service (#11913) 2026-01-27 11:35:12 -08:00
Trenton H c06e1e7cba Resolves gpg-agent hanging around and using inotify handles too (#11848) 2026-01-21 15:53:54 -08:00
shamoon 4e789acf2d Chore: mark test_error_skip_rule as flaky 2025-12-18 10:15:04 -08:00
shamoon 598540fda0 Chore: mark another test flaky 2025-12-18 07:17:50 -08:00
shamoon 533b64cb70 Chore: replace test image URLs to our own files (#11390) 2025-11-17 10:22:54 -08:00
shamoon 69514d8d70 Chore: add backoff ro handle 429 Wikimedia requests in tests (#11364) 2025-11-14 15:58:29 -08:00
shamoon cffb9c34f0 Chore: add headers for wikipedia CI tests (#11253) 2025-11-01 09:37:49 -07:00
shamoon 19a54b3b23 Feature: processed mail UI (#10866) 2025-09-22 18:17:42 +00:00
shamoon 7a1aae7749 Fix: set match value for correspondents created by mail rule (#10820) 2025-09-09 09:14:48 -07:00
shamoon a68800d53c See if this helps with intermittent mail test failure 2025-08-16 10:30:25 -07:00
shamoon dbf3721ec2 Chore: reject absurd max age values (#10243) 2025-06-22 07:39:36 -07:00
shamoon e4fd008441 Fix: fix some API crashes (#10196) 2025-06-16 22:44:39 -07:00
dependabot[bot] 2abf38f98e Chore(deps): Bump the small-changes group across 1 directory with 6 updates (#9764)
* Chore(deps): Bump the small-changes group across 1 directory with 6 updates

Bumps the small-changes group with 6 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [channels](https://github.com/django/channels) | `4.2.0` | `4.2.2` |
| [filelock](https://github.com/tox-dev/py-filelock) | `3.17.0` | `3.18.0` |
| [gotenberg-client](https://github.com/stumpylog/gotenberg-client) | `0.9.0` | `0.10.0` |
| [jinja2](https://github.com/pallets/jinja) | `3.1.5` | `3.1.6` |
| [python-dotenv](https://github.com/theskumar/python-dotenv) | `1.0.1` | `1.1.0` |
| [rapidfuzz](https://github.com/rapidfuzz/RapidFuzz) | `3.12.1` | `3.13.0` |



Updates `channels` from 4.2.0 to 4.2.2
- [Changelog](https://github.com/django/channels/blob/main/CHANGELOG.txt)
- [Commits](https://github.com/django/channels/compare/4.2.0...4.2.2)

Updates `filelock` from 3.17.0 to 3.18.0
- [Release notes](https://github.com/tox-dev/py-filelock/releases)
- [Changelog](https://github.com/tox-dev/filelock/blob/main/docs/changelog.rst)
- [Commits](https://github.com/tox-dev/py-filelock/compare/3.17.0...3.18.0)

Updates `gotenberg-client` from 0.9.0 to 0.10.0
- [Release notes](https://github.com/stumpylog/gotenberg-client/releases)
- [Changelog](https://github.com/stumpylog/gotenberg-client/blob/main/CHANGELOG.md)
- [Commits](https://github.com/stumpylog/gotenberg-client/compare/0.9.0...0.10.0)

Updates `jinja2` from 3.1.5 to 3.1.6
- [Release notes](https://github.com/pallets/jinja/releases)
- [Changelog](https://github.com/pallets/jinja/blob/main/CHANGES.rst)
- [Commits](https://github.com/pallets/jinja/compare/3.1.5...3.1.6)

Updates `python-dotenv` from 1.0.1 to 1.1.0
- [Release notes](https://github.com/theskumar/python-dotenv/releases)
- [Changelog](https://github.com/theskumar/python-dotenv/blob/main/CHANGELOG.md)
- [Commits](https://github.com/theskumar/python-dotenv/compare/v1.0.1...v1.1.0)

Updates `rapidfuzz` from 3.12.1 to 3.13.0
- [Release notes](https://github.com/rapidfuzz/RapidFuzz/releases)
- [Changelog](https://github.com/rapidfuzz/RapidFuzz/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/rapidfuzz/RapidFuzz/compare/v3.12.1...v3.13.0)

---
updated-dependencies:
- dependency-name: channels
  dependency-version: 4.2.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: small-changes
- dependency-name: filelock
  dependency-version: 3.18.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: small-changes
- dependency-name: gotenberg-client
  dependency-version: 0.10.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: small-changes
- dependency-name: jinja2
  dependency-version: 3.1.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: small-changes
- dependency-name: python-dotenv
  dependency-version: 1.1.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: small-changes
- dependency-name: rapidfuzz
  dependency-version: 3.13.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: small-changes
...

Signed-off-by: dependabot[bot] <support@github.com>
---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Trenton H <797416+stumpylog@users.noreply.github.com>
2025-04-22 10:32:27 -07:00
shamoon 1dc80f04cb Feature: openapi spec, full api browser (#8948) 2025-02-10 16:43:07 +00:00
Silvia Bigler 71472a6a82 Enhancement: add layout options for email conversion (#8907)
---------

Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
2025-02-07 18:32:35 +00:00
Sebastian Steinbeißer e560fa3be0 Chore: Enable ruff FBT (#8645) 2025-02-07 09:12:03 -08:00
Trenton H 702de0cac3 Chore: Upgrades dependencies and hook versions (#8895) 2025-01-24 12:08:58 -08:00
dependabot[bot] 20ec8cb57b Chore(deps-dev): Bump the development group with 2 updates (#8841)
* Chore(deps-dev): Bump the development group with 2 updates

Bumps the development group with 2 updates: [ruff](https://github.com/astral-sh/ruff) and [mkdocs-material](https://github.com/squidfunk/mkdocs-material).


Updates `ruff` from 0.8.6 to 0.9.2
- [Release notes](https://github.com/astral-sh/ruff/releases)
- [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md)
- [Commits](https://github.com/astral-sh/ruff/compare/0.8.6...0.9.2)

Updates `mkdocs-material` from 9.5.49 to 9.5.50
- [Release notes](https://github.com/squidfunk/mkdocs-material/releases)
- [Changelog](https://github.com/squidfunk/mkdocs-material/blob/master/CHANGELOG)
- [Commits](https://github.com/squidfunk/mkdocs-material/compare/9.5.49...9.5.50)

---
updated-dependencies:
- dependency-name: ruff
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: development
- dependency-name: mkdocs-material
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: development
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update .pre-commit-config.yaml

* Run new ruff format

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
2025-01-21 19:22:25 +00:00
shamoon a899ff16e3 Fix: use state param with oauth (#8636) 2025-01-08 03:48:36 +00:00
shamoon dafb0b1f21 Enhancement: process mail button (#8466) 2024-12-11 11:03:53 -08:00
shamoon df2c139721 Chore: mark another test flaky 2024-12-04 12:10:31 -08:00
shamoon 2353f7c2db Feature: OAuth2 Gmail and Outlook email support (#7866) 2024-10-10 20:57:32 +00:00
shamoon e49ed58f1a Fix: skip accounts without enabled rules 2024-10-04 23:59:31 -07:00
Martin Richtarsky b3487f1843 Enhancement: check for mail destination directory, log post-consume errors (#7808)
---------

Co-authored-by: shamoon <4887959+shamoon@users.noreply.github.com>
2024-10-02 20:21:35 -07:00
Trenton H 2ab71137b9 Chore: Upgrades OCRMyPDF to v16 (#7815) 2024-10-01 02:53:44 +00:00