amorfo77/parsedmarc
1
0
Fork 0
mirror of https://github.com/domainaware/parsedmarc.git synced 2026-03-22 06:22:46 +00:00
Code Issues Projects Releases Wiki Activity
1,431 Commits 26 Branches 27 Tags
eedc4f5916bea4e176745cf60879ec09d508d8ca
Commit Graph

46 Commits

Author SHA1 Message Date
copilot-swe-agent[bot]
cdb56904d3 Make pct and fo default to None when not provided (DMARCbis reports) 
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
 2026-03-21 17:06:44 -04:00
copilot-swe-agent[bot]
23f2cb99c3 Add DMARCbis field validation, preserve pass disposition, add comprehensive tests 
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
 2026-03-21 17:05:57 -04:00
Sean Whalen
9849598100 Formatting 2026-03-21 16:17:35 -04:00
Sean Whalen
e82f3e58a1 SIGHUP-based configuration reload for watch mode (#697) 
* Enhance mailbox connection watch method to support reload functionality

- Updated the `watch` method in `GmailConnection`, `MSGraphConnection`, `IMAPConnection`, `MaildirConnection`, and the abstract `MailboxConnection` class to accept an optional `should_reload` parameter. This allows the method to check if a reload is necessary and exit the loop if so.
- Modified related tests to accommodate the new method signature.
- Changed logger calls from `critical` to `error` for consistency in logging severity.
- Added a new settings file for Claude with specific permissions for testing and code checks.

* Update parsedmarc/cli.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update parsedmarc/cli.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* [WIP] SIGHUP-based configuration reload for watch mode (#698)

* Initial plan

* Fix reload state consistency, resource leaks, stale opts; add tests

Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
Agent-Logs-Url: https://github.com/domainaware/parsedmarc/sessions/3c2e0bb9-7e2d-4efa-aef6-d2b98478b921

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>

* [WIP] SIGHUP-based configuration reload for watch mode (#699)

* Initial plan

* Fix review comments: ConfigurationError wrapping, duplicate parse args, bool parsing, Kafka required topics, should_reload kwarg, SIGHUP test skips

Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
Agent-Logs-Url: https://github.com/domainaware/parsedmarc/sessions/0779003c-ccbe-4d76-9748-801dbc238b96

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>

* SIGHUP-based configuration reload: address review feedback (#700)

* Initial plan

* Address review feedback: kafka_ssl, duplicate silent, exception chain, log file reload, should_reload timing

Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
Agent-Logs-Url: https://github.com/domainaware/parsedmarc/sessions/a8a43c55-23fa-4471-abe6-7ac966f381f9

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>

* Update parsedmarc/cli.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Best-effort initialization for optional output clients in watch mode (#701)

* Initial plan

* Wrap optional output client init in try/except for best-effort initialization

Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
Agent-Logs-Url: https://github.com/domainaware/parsedmarc/sessions/59241d4e-1b05-4a92-b2d2-e6d13d10a4fd

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>

* Fix SIGHUP reload tight-loop in watch mode (#702)

* Initial plan

* Fix _reload_requested tight-loop: reset flag before reload to capture concurrent SIGHUPs

Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
Agent-Logs-Url: https://github.com/domainaware/parsedmarc/sessions/879d0bb1-9037-41f7-bc89-f59611956d2e

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>

* Update parsedmarc/cli.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Fix resource leak when HEC config is invalid in `_init_output_clients()` (#703)

* Initial plan

* Fix resource leak: validate HEC settings before creating any output clients

Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
Agent-Logs-Url: https://github.com/domainaware/parsedmarc/sessions/38c73e09-789d-4d41-b75e-bbc61418859d

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>

* Ensure SIGHUP never triggers a new email batch across all watch() implementations (#704)

* Initial plan

* Ensure SIGHUP never starts a new email batch in any watch() implementation

Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
Agent-Logs-Url: https://github.com/domainaware/parsedmarc/sessions/45d5be30-8f6b-4200-9bdd-15c655033f17

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>

* SIGHUP-based config reload for watch mode: address review feedback (#705)

* Initial plan

* Address review feedback: Kafka SSL context, SIGHUP handler safety, test formatting

Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
Agent-Logs-Url: https://github.com/domainaware/parsedmarc/sessions/8f2fd48f-32a4-4258-9a89-06f7c7ac29bf

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>

* Reverted changes by copilot that turned errors into warnings

* Enhance usage documentation for config reload: clarify behavior on successful reload and error handling

* Update CHANGELOG.md to reflect config reload enhancements

* Add pytest command to settings for silent output during testing

* Enhance resource management: add close methods for S3Client and HECClient, and improve IMAP connection handling during IDLE. Update CHANGELOG.md for config reload improvements and bug fixes.

* Update changelog to not include fixes within the same unreleased version

* Refactor changelog entries for clarity and consistency in configuration reload section

* Fix changelog entry for msgraph configuration check

* Update CHANGELOG..md

* make single list items on one line in the changelog instead of doing hard wraps

* Remove incorrect IMAP changes

* Rename 'should_reload' parameter to 'config_reloading' in mailbox connection methods for clarity

* Restore startup configuration checks

* Improve error logging for Elasticsearch and OpenSearch exceptions

* Bump version to 9.3.0 in constants.py

* Refactor GelfClient methods to use specific report types instead of generic dicts

* Refactor tests to use assertions consistently and improve type hints

---------

Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
 2026-03-21 16:14:48 -04:00
Sean Whalen
b9343a295f 9.2.1 
- Better checking of `msconfig` configuration (PR #695)
- Updated `dbip-country-lite` database to version `2026-03`
- Changed - DNS query error logging level from `warning` to `debug`
 2026-03-10 20:32:33 -04:00
Kili
b51a62463f Fail fast on invalid MS Graph username/password config (#695) 2026-03-10 19:34:16 -04:00
Kili
66ba5b0e5e Add MS Graph auth matrix regression tests (#696) 
* Rebase MS Graph auth matrix tests onto current master

* Expand ClientSecret auth matrix coverage
 2026-03-10 19:33:37 -04:00
Kili
faa68333a9 Avoid extra mailbox fetch in batch/test mode and add regression test (#691) 
Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>
 2026-03-10 11:22:39 -04:00
Kili
d34a33e980 Validate MS Graph certificate auth inputs (#693) 
* Validate MS Graph certificate auth inputs

* Fix MS Graph shared scope detection without username
 2026-03-10 11:22:09 -04:00
Kili
9040a38842 Refine MS Graph well-known folder fallback (#694) 
* Refine MS Graph well-known folder fallback

* Make MS Graph retry test doubles method-aware
 2026-03-10 11:20:43 -04:00
Kili
ea0e3b11c1 Add MS Graph certificate authentication support (#692) 
* Add MS Graph certificate authentication support

* Preserve MS Graph constructor compatibility

---------

Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>
 2026-03-10 09:30:39 -04:00
Kili
199b782191 Add MS Graph well-known folder fallback for root listing failures (#689) 
* Add MS Graph well-known folder fallback for root listing failures

* Resolve test merge cleanup for MS Graph folder fallback
 2026-03-10 09:25:37 -04:00
Kili
56eb565ad2 Accept pathlib.Path in report parsing APIs (#680) 
* Accept pathlib.Path in report parsing APIs

* Polish PathLike typing and test names
 2026-03-09 18:08:57 -04:00
Kili
2c3abb3e8c Retry transient MS Graph request errors (#679) 
* Retry transient MS Graph request errors

* Handle zero MS Graph retry attempts explicitly
 2026-03-09 17:56:22 -04:00
Kili
f2febf21d3 Add fail_on_output_error CLI option for sink failures (#672) 
* Add fail-on-output-error option and CLI regression test

* Broaden fail_on_output_error coverage for disabled and multi-sink paths
 2026-03-09 17:35:38 -04:00
Kili
79f47121a4 Pass mailbox since filter through watch_inbox callback (#670) 
* Pass mailbox since through watch loop and add regression test

* Add CLI regression test for mailbox since in watch mode
 2026-03-09 17:33:42 -04:00
Kili
6e6c90e19b Add IMAP move/delete compatibility fallbacks (#671) 
* Add IMAP move/delete compatibility fallbacks with tests

* Expand IMAP fallback tests for success and error paths
 2026-03-09 17:29:01 -04:00
Kili
c4d7455839 Add OpenSearch AWS SigV4 authentication support (#673) 
* Add OpenSearch AWS SigV4 authentication support

* Increase SigV4 coverage for auth validation and CLI config wiring

* Update parsedmarc/opensearch.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/source/usage.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2026-03-09 17:21:22 -04:00
Kili
95e6fb85a1 Fix Gmail delete_message to execute API request (#668) 
* Fix Gmail delete to execute request and add regression test

* Fix duplicate GmailConnection import in tests
 2026-03-09 17:11:35 -04:00
Kili
a3c5bb906b Add Gmail service account auth mode with delegated user support (#676) 2026-03-09 17:04:30 -04:00
Kili
d49ce6a13f Increase unit test coverage for Gmail/Graph/IMAP connectors (#664) 
* Increase coverage for Gmail, Graph, and IMAP mail connectors

* Make testLoadTokenMissing use guaranteed-missing temp path

* Expand coverage for Gmail token refresh and Graph pagination error paths
 2026-03-09 11:54:43 -04:00
Copilot
4d97bd25aa Skip DNS lookups in GitHub Actions to prevent test timeouts (#657) 
* Add offline mode for tests in GitHub Actions to skip DNS lookups

Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
 2026-02-18 18:19:28 -05:00
Sean Whalen
7b842740f5 Change file permissions for tests.py to make it executable 2025-12-25 16:02:33 -05:00
Sean Whalen
7d58abc67b Add shebang and encoding declaration to tests.py 2025-12-04 10:21:53 -05:00
Sean Whalen
cf3b7f2c29 ## 9.0.2 
## Improvements

- Type hinting is now used properly across the entire library. (#445)

## Fixes

- Decompress report files as needed when passed via the CLI.
- Fixed incomplete removal of the ability for `parsedmarc.utils.extract_report` to accept a file path directly in `8.15.0`.

## Breaking changes

This version of the library requires consumers to pass certain arguments as keyword-only. Internally, the API uses a bare `*` in the function signature. This is standard per [PEP 3102](https://peps.python.org/pep-3102/)  and as documented in the Python Language Reference.
.
 2025-12-02 19:41:14 -05:00
Sean Whalen
17bdc3a134 More tests cleanup 2025-11-21 09:10:59 -05:00
Sean Whalen
597ca64f9f Clean up tests 2025-11-21 00:09:28 -05:00
Sean Whalen
c5dbe2c4dc 8.10.9 
- Complete fix for #687 and more robust report type detection
 2025-11-20 23:50:42 -05:00
Sean Whalen
1ef3057110 8.15.1 
- Proper IMAP namespace fix (Closes issue #557 and issue #563)
  - Require `mailsuite>=1.9.17`
  - Revert PR #552
- Add pre-flight check for nameservers (PR #562 closes issue #543)
- Reformat code with `ruff`
 2024-10-02 21:19:57 -04:00
Sean Whalen
e967778f25 8.15.0 
- Fix processing of SMTP-TLS reports ([#549](https://github.com/domainaware/parsedmarc/issues/549)), which broke in commit [410663d ](410663dbca)(PR [#530](https://github.com/domainaware/parsedmarc/pull/530))
  - This PR enforced a stricter check for base64-encoded strings, which SMTP TLS reports from Google did not pass
  - Removing the check introduced its own issue, because some file paths were treated as base64-encoded strings
- Create a separate `extract_report_from_file_path()` function for processioning reports based on a file path
- Remove report extraction based on a file path from `extract_report()`
 2024-09-04 16:31:41 -04:00
Sean Whalen
e9f0cdef1f Update tests.py 
Fix code style
 2024-07-16 15:14:40 -04:00
pphillips99
410663dbca fixup and tests to extract_report() to handle documented inputs (#530) 2024-07-16 14:53:27 -04:00
Sean Whalen
6e96b88a27 Always use local files when testing 2024-05-22 08:30:12 -04:00
cgoIT
f3206dcdab [SMTP TLS] some minor bug fixes (#477) 
* fix minor bugs during smtp-tls parsing, add docker-compose for local elasticsearch, add smtp-tls tests

* fix wrong log message parameter

* fix wrong log message

* add contact-info to smtp tls report, fix wrong fieldnames

* fix wrong fieldnames

* fix wrong index name for search

* at least for some reporting organizations the field sending-mta-ip is optional...

* add missing fields to elasticsearch for smtp tls

* failure_details is a list, add more test cases

* fix wrong name in ci.ini
 2024-03-04 10:06:47 -05:00
Sean Whalen
b8088505b1 Add support for SMTP TLS reports (#453) 2024-02-19 18:45:38 -05:00
Sean Whalen
6c84cfb7c4 Fix output in tests.py 2023-06-27 15:56:33 -04:00
Sean Whalen
89872d78ac Fix testing 2023-06-27 15:43:09 -04:00
Jed Laundry
a06fdc586f Change publicsuffix2 to publicsuffixlist (#406) 
* change to publicsuffixlist

* update publicsuffixlist (now auto-updating)

* Fix unused imports
 2023-05-09 08:49:41 -04:00
Dave Rawks
84a7386726 Handle unexpected xml parsing errors more gracefully (#349) 
* updates `parse_aggregate_report_xml` to not raise an unhandled
  exception on parsing errors
* adds an empty xml file to the aggregate test samples
* adds test for coverage
* Resolves #348
 2022-09-08 18:23:34 -04:00
Sean Whalen
06371dfe9b Expand test coverage 2019-04-04 20:15:55 -04:00
Sean Whalen
e588845f23 6.1.0 - Fix aggregate report email parsing regression 
Fix aggregate report email parsing regression in 6.0.3 (closes issues #57 and #58)
 2019-02-13 10:30:25 -05:00
Sean Whalen
49387d9033 4.3.0 release 2018-10-12 12:02:55 -04:00
Sean Whalen
f45ab94e06 Update test suitw 2018-10-11 19:01:02 -04:00
Sean Whalen
2c7002d3cc Fix testing automation 2018-05-14 19:39:14 -04:00
Sean Whalen
d900ebf0eb 2.1.2 2018-03-06 10:01:27 -05:00
Sean Whalen
6b9e36ed77 First commit 2018-02-05 20:23:07 -05:00