amorfo77/parsedmarc
1
0
Fork 0
mirror of https://github.com/domainaware/parsedmarc.git synced 2026-03-22 14:32:46 +00:00
Code Issues Projects Releases Wiki Activity
1,416 Commits 26 Branches 27 Tags
d987421f5c1e44c75d01882ff2b071c74c848cc2
Commit Graph

69 Commits

Author SHA1 Message Date
Sean Whalen
e82f3e58a1 SIGHUP-based configuration reload for watch mode (#697) 
* Enhance mailbox connection watch method to support reload functionality

- Updated the `watch` method in `GmailConnection`, `MSGraphConnection`, `IMAPConnection`, `MaildirConnection`, and the abstract `MailboxConnection` class to accept an optional `should_reload` parameter. This allows the method to check if a reload is necessary and exit the loop if so.
- Modified related tests to accommodate the new method signature.
- Changed logger calls from `critical` to `error` for consistency in logging severity.
- Added a new settings file for Claude with specific permissions for testing and code checks.

* Update parsedmarc/cli.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update parsedmarc/cli.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* [WIP] SIGHUP-based configuration reload for watch mode (#698)

* Initial plan

* Fix reload state consistency, resource leaks, stale opts; add tests

Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
Agent-Logs-Url: https://github.com/domainaware/parsedmarc/sessions/3c2e0bb9-7e2d-4efa-aef6-d2b98478b921

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>

* [WIP] SIGHUP-based configuration reload for watch mode (#699)

* Initial plan

* Fix review comments: ConfigurationError wrapping, duplicate parse args, bool parsing, Kafka required topics, should_reload kwarg, SIGHUP test skips

Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
Agent-Logs-Url: https://github.com/domainaware/parsedmarc/sessions/0779003c-ccbe-4d76-9748-801dbc238b96

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>

* SIGHUP-based configuration reload: address review feedback (#700)

* Initial plan

* Address review feedback: kafka_ssl, duplicate silent, exception chain, log file reload, should_reload timing

Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
Agent-Logs-Url: https://github.com/domainaware/parsedmarc/sessions/a8a43c55-23fa-4471-abe6-7ac966f381f9

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>

* Update parsedmarc/cli.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Best-effort initialization for optional output clients in watch mode (#701)

* Initial plan

* Wrap optional output client init in try/except for best-effort initialization

Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
Agent-Logs-Url: https://github.com/domainaware/parsedmarc/sessions/59241d4e-1b05-4a92-b2d2-e6d13d10a4fd

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>

* Fix SIGHUP reload tight-loop in watch mode (#702)

* Initial plan

* Fix _reload_requested tight-loop: reset flag before reload to capture concurrent SIGHUPs

Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
Agent-Logs-Url: https://github.com/domainaware/parsedmarc/sessions/879d0bb1-9037-41f7-bc89-f59611956d2e

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>

* Update parsedmarc/cli.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Fix resource leak when HEC config is invalid in `_init_output_clients()` (#703)

* Initial plan

* Fix resource leak: validate HEC settings before creating any output clients

Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
Agent-Logs-Url: https://github.com/domainaware/parsedmarc/sessions/38c73e09-789d-4d41-b75e-bbc61418859d

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>

* Ensure SIGHUP never triggers a new email batch across all watch() implementations (#704)

* Initial plan

* Ensure SIGHUP never starts a new email batch in any watch() implementation

Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
Agent-Logs-Url: https://github.com/domainaware/parsedmarc/sessions/45d5be30-8f6b-4200-9bdd-15c655033f17

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>

* SIGHUP-based config reload for watch mode: address review feedback (#705)

* Initial plan

* Address review feedback: Kafka SSL context, SIGHUP handler safety, test formatting

Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
Agent-Logs-Url: https://github.com/domainaware/parsedmarc/sessions/8f2fd48f-32a4-4258-9a89-06f7c7ac29bf

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>

* Reverted changes by copilot that turned errors into warnings

* Enhance usage documentation for config reload: clarify behavior on successful reload and error handling

* Update CHANGELOG.md to reflect config reload enhancements

* Add pytest command to settings for silent output during testing

* Enhance resource management: add close methods for S3Client and HECClient, and improve IMAP connection handling during IDLE. Update CHANGELOG.md for config reload improvements and bug fixes.

* Update changelog to not include fixes within the same unreleased version

* Refactor changelog entries for clarity and consistency in configuration reload section

* Fix changelog entry for msgraph configuration check

* Update CHANGELOG..md

* make single list items on one line in the changelog instead of doing hard wraps

* Remove incorrect IMAP changes

* Rename 'should_reload' parameter to 'config_reloading' in mailbox connection methods for clarity

* Restore startup configuration checks

* Improve error logging for Elasticsearch and OpenSearch exceptions

* Bump version to 9.3.0 in constants.py

* Refactor GelfClient methods to use specific report types instead of generic dicts

* Refactor tests to use assertions consistently and improve type hints

---------

Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
 2026-03-21 16:14:48 -04:00
Kili
ea0e3b11c1 Add MS Graph certificate authentication support (#692) 
* Add MS Graph certificate authentication support

* Preserve MS Graph constructor compatibility

---------

Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>
 2026-03-10 09:30:39 -04:00
Kili
326e630f50 Add performance tuning guidance for large mailbox runs (#677) 2026-03-09 17:44:42 -04:00
Kili
f2febf21d3 Add fail_on_output_error CLI option for sink failures (#672) 
* Add fail-on-output-error option and CLI regression test

* Broaden fail_on_output_error coverage for disabled and multi-sink paths
 2026-03-09 17:35:38 -04:00
Kili
c4d7455839 Add OpenSearch AWS SigV4 authentication support (#673) 
* Add OpenSearch AWS SigV4 authentication support

* Increase SigV4 coverage for auth validation and CLI config wiring

* Update parsedmarc/opensearch.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/source/usage.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2026-03-09 17:21:22 -04:00
Kili
a3c5bb906b Add Gmail service account auth mode with delegated user support (#676) 2026-03-09 17:04:30 -04:00
Kili
e98fdfa96b Fix Python 3.14 support metadata and require imapclient 3.1.0 (#662) 2026-03-04 12:36:15 -05:00
Copilot
2e3ee25ec9 Drop Python 3.9 support (#661) 
* Initial plan

* Drop Python 3.9 support: update CI matrix, pyproject.toml, docs, and README

Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>

* Update Python 3.9 version table entry to note Debian 11/RHEL 9 usage

Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
 2026-03-03 11:34:35 -05:00
Copilot
17a612df0c Add TCP and TLS transport support to syslog module (#656) 
- Updated parsedmarc/syslog.py to support UDP, TCP, and TLS protocols
- Added protocol parameter with UDP as default for backward compatibility
- Implemented TLS support with CA verification and client certificate auth
- Added retry logic for TCP/TLS connections with configurable attempts and delays
- Updated parsedmarc/cli.py with new config file parsing
- Updated documentation with examples for TCP and TLS configurations

Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>

* Remove CLI arguments for syslog options, keep config-file only

Per user request, removed command-line argument options for syslog parameters.
All new syslog options (protocol, TLS settings, timeout, retry) are now only
available via configuration file, consistent with other similar options.

Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>

* Fix code review issues: remove trailing whitespace and add cert validation

- Removed trailing whitespace from syslog.py and usage.md
- Added warning when only one of certfile_path/keyfile_path is provided
- Improved error handling for incomplete TLS client certificate configuration

Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>

* Set minimum TLS version to 1.2 for enhanced security

Explicitly configured ssl_context.minimum_version = TLSVersion.TLSv1_2
to ensure only secure TLS versions are used for syslog connections.

Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
 2026-02-18 18:12:59 -05:00
Anael Mobilia
50fcb51577 Update supported Python versions in docs + readme (#652) 
* Update README.md

* Update index.md

* Update python-tests.yml
 2026-01-19 14:40:01 -05:00
Sean Whalen
35331d4b84 Add parsedmarc.types module to API reference documentation 2025-12-25 17:24:45 -05:00
Sean Whalen
445c9565a4 Update bug link in docs 2025-12-06 15:05:19 -05:00
Sean Whalen
23ae563cd8 Update Python version support details in documentation 2025-12-05 10:48:04 -05:00
Sean Whalen
a18ae439de Fix typo in RHEL version support description in documentation 2025-12-04 10:18:15 -05:00
Oscar Mattsson
f1933b906c Fix 404 link to maxmind docs (#635) 2025-12-02 09:26:01 -05:00
Sean Whalen
1fc9f638e2 9.0.0 (#629) 
* Normalize report volumes when a report timespan exceed 24 hours
 2025-12-01 17:06:58 -05:00
Sean Whalen
0922d6e83a Add supported Python versions to the documentation index 2025-12-01 10:24:19 -05:00
Anael Mobilia
00267c9847 Codestyle cleanup (#631) 
* Fix typos

* Copyright - Update date

* Codestyle xxx is False -> not xxx

* Ensure "_find_label_id_for_label" always return str

* PEP-8 : apiKey -> api_key + backward compatibility for config files

* Duplicate variable initialization

* Fix format
 2025-11-30 19:13:57 -05:00
Sean Whalen
a05c230152 8.19.0 (#622) 
8.19.0

- Add multi-tenant support via an index-prefix domain mapping file
- PSL overrides so that services like AWS are correctly identified
- Additional improvements to report type detection
- Fix webhook timeout parsing (PR #623)
- Output to STDOUT when the new general config boolean `silent` is set to `False` (Close #614)
- Additional services added to `base_reverse_dns_map.csv`

---------

Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Félix <felix.debloisbeaucage@gmail.com>
 2025-11-28 12:47:00 -05:00
Sean Whalen
865c249437 Update features list 2025-08-24 13:39:50 -04:00
Martin Kjær Jørgensen
d62001f5a4 fix wrong configuration option for maildir (#606) 
Signed-off-by: Martin Kjær Jørgensen <me@lagy.org>
 2025-08-14 18:36:58 -04:00
Kevin Goad
669deb9755 Add support for Microsoft national clouds via Graph API base URL (#590) 
* adding support for Microsoft National Clouds

* Update usage.md
 2025-02-03 15:25:15 -05:00
Sean Whalen
bd5774d71d Merge branch 'master' of https://github.com/ramspoluri/parsedmarc into ramspoluri-master 2024-11-02 11:38:41 -04:00
Sean Whalen
1ef3057110 8.15.1 
- Proper IMAP namespace fix (Closes issue #557 and issue #563)
  - Require `mailsuite>=1.9.17`
  - Revert PR #552
- Add pre-flight check for nameservers (PR #562 closes issue #543)
- Reformat code with `ruff`
 2024-10-02 21:19:57 -04:00
Paolo Schiro
fdb4e4cb36 Added nameservers pre-flight check, issue #543 (#562) 2024-10-02 20:48:02 -04:00
centja1
0a6cfb602c Added output to webhook as an option (#558) 
* Added output to webhook as an option

* added documentation for new webhook configuration

---------

Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>
 2024-09-12 15:47:59 -04:00
Paolo Schiro
7b7d20b1a4 Added input report maildir connector, issue #82 (#555) 
Co-authored-by: Paolo Schiro <paolo.schiro@staff.aruba.it>
 2024-09-06 15:44:57 -04:00
Sean Whalen
0fa6bebf5a Fix error in docs 2024-08-24 20:10:57 -04:00
Jason Lingohr
11e0461b9d Add GELF support (#532) 
* Implement the ability to log to a GELF server/input, via the use of pygelf.

* Fix flake8 style checks.
 2024-08-24 11:28:55 -04:00
Emmanuel Ormancey
9b32d9459f added ES/OS prefix (#531) 2024-08-24 11:26:13 -04:00
Sean Whalen
61402d6284 Fix PowerShell typo in docs 2024-06-11 14:38:29 -04:00
ramspoluri
f618f69c6c Added 'since' option to search for messages since a certain time 
- Added `since` option under `mailbox` section to search for messages since a certain time instead of going through the complete mailbox during testing scenarios. Acceptable values -`5m|3h|2d|1w`, units - {"m":"minutes", "h":"hours", "d":"days", "w":"weeks"}). Defaults to `1d` if an incorrect value is provided.
    - Not to mark messages as read if test option is selected (works only for MSGraphConnection)
 2024-05-24 20:43:36 +05:30
yuji suzuki
20f451192f add gmail_api example to README.md (#518) 2024-05-22 08:35:16 -04:00
Sean Whalen
a53d35a90c Revert "Update usage.md" PR #497 makes moot 
This reverts commit 662b7b694b.
 2024-04-08 10:41:16 -04:00
Sean Whalen
662b7b694b Update usage.md 2024-04-08 09:55:07 -04:00
Sean Whalen
6a06d01b44 8.11.0 
- Actually save `source_type` and `source_name` to Elasticsearch and OpenSearch
- Reverse-lookup cache improvements (PR #501 closes issue #498)
- Update the included `dbip-country-lite.mmdb` to the 2024-03 version
- Update `base_reverse_dns_map.csv`
- Add new general config options (closes issue #500)
  - `always_use_local_files` - Disables the download of the reverse DNS map
  - `local_reverse_dns_map_path` - Overrides the default local file path to use for the reverse DNS map
  - `reverse_dns_map_url` - Overrides the default download URL for the reverse DNS map
 2024-04-02 12:34:32 -04:00
Szasza Palmer
a915385246 replacing multiprocessing pool with pipe (#491) 
* replacing multiprocessing pool with pipe

* code styling fix

* dropping obsolete chunk_size config parameter
 2024-03-23 21:41:46 -04:00
Anael Mobilia
3a1360a47a Always use systemctl command (#482) 
In order to align with the rest of the documentation
 2024-03-17 11:44:55 -04:00
Szasza Palmer
995bdbcd97 adding OpenSearch support, fixing minor typos, and code styling (#481) 
* adding OpenSearch support, fixing minor typos and code styling

* documentation update
 2024-03-04 10:06:26 -05:00
mkupferman
efe74091f3 Gmail API pagination (#469) 
* Use pagination to fetch more than 100 Gmail messages at once

* Provide `paginate_messages` option to allow use of previous behavior
 2024-02-19 18:54:41 -05:00
Anael Mobilia
1c1ce7fea3 Doc - Update elasticsearch parameters (#467) 2024-02-19 18:52:01 -05:00
Sean Whalen
b8088505b1 Add support for SMTP TLS reports (#453) 2024-02-19 18:45:38 -05:00
Jason Lingohr
7d2b431e5f Fix tiny formatting issue (#451) 2024-01-02 16:38:50 -05:00
Jonathan Martens
c71bc19cea docs: formatting fix on delete keyword in mail section (#444) 2023-12-16 10:08:41 -05:00
Anael Mobilia
3bf0bea710 Fix typo on documentation (#443) 
* Doc cleanup/improvement

* Fix typos

---------

Co-authored-by: Anael Mobilia <anael.mobilia@mydsomanager.com>
 2023-12-16 10:08:06 -05:00
Sean Whalen
21d6f92fd4 Add PyPI download stats badge 2023-10-13 10:01:48 -04:00
jlownie
a164fb8e04 Update usage.md (#436) 2023-10-13 09:53:28 -04:00
Anael Mobilia
732547e622 Doc cleanup/improvement (#432) 
Co-authored-by: Anael Mobilia <anael.mobilia@mydsomanager.com>
 2023-10-11 17:24:50 -04:00
Anael Mobilia
d2d62488f7 Add contributors on copyright (#414) 
Co-authored-by: Anael Mobilia <anael.mobilia@mydsomanager.com>
 2023-05-14 18:07:35 -04:00
Anael Mobilia
bf69ea8ccc Fix typos (#413) 
Co-authored-by: Anael Mobilia <anael.mobilia@mydsomanager.com>
 2023-05-14 18:07:07 -04:00