00721fdabb
Rename forensic to failure in output/integration modules
...
Rename all 'forensic' references to 'failure' in the output modules:
- elastic.py, opensearch.py, splunk.py, kafkaclient.py, syslog.py,
gelf.py, webhook.py, loganalytics.py, s3.py
Changes include:
- Function/method names: save_forensic_* → save_failure_*
- Variable/parameter names: forensic_* → failure_*
- Class names: _ForensicReportDoc → _FailureReportDoc,
_ForensicSampleDoc → _FailureSampleDoc
- Index/topic/sourcetype names: dmarc_forensic → dmarc_failure
- Log messages and docstrings updated
- Import statements updated to use new names from core module
- Backward-compatible aliases added at end of each file
- DMARCbis aggregate fields added to elastic.py and opensearch.py:
np (Keyword), testing (Keyword), discovery_method (Keyword),
generator (Text)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com >
2026-03-21 17:00:19 -04:00
4a138ae8f7
Align DMARCbis fields with actual XSD schema: testing, discovery_method, generator, human_result; handle namespaced XML
...
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com >
2026-03-21 16:59:15 -04:00
d987421f5c
Add DMARCbis fields (np, psd, t) to aggregate reports and rename forensic→failure in core parsing
...
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com >
2026-03-21 16:59:15 -04:00
3d4737bd82
Rename forensic references to failure with backward-compatible aliases
...
- Rename parse_forensic_report -> parse_failure_report
- Rename parsed_forensic_reports_to_csv_rows -> parsed_failure_reports_to_csv_rows
- Rename parsed_forensic_reports_to_csv -> parsed_failure_reports_to_csv
- Update all internal variable names (forensic_report -> failure_report, etc.)
- Change report_type from 'forensic' to 'failure'
- Use FailureReport type instead of ForensicReport
- Use InvalidFailureReport instead of InvalidForensicReport in function bodies
- Update all docstrings and log messages
- Add backward-compatible aliases at end of file
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com >
2026-03-21 16:59:15 -04:00
ee39ef50d8
Initial plan
2026-03-21 16:59:15 -04:00
9849598100
Formatting
2026-03-21 16:17:35 -04:00
e82f3e58a1
SIGHUP-based configuration reload for watch mode ( #697 )
...
* Enhance mailbox connection watch method to support reload functionality
- Updated the `watch` method in `GmailConnection`, `MSGraphConnection`, `IMAPConnection`, `MaildirConnection`, and the abstract `MailboxConnection` class to accept an optional `should_reload` parameter. This allows the method to check if a reload is necessary and exit the loop if so.
- Modified related tests to accommodate the new method signature.
- Changed logger calls from `critical` to `error` for consistency in logging severity.
- Added a new settings file for Claude with specific permissions for testing and code checks.
* Update parsedmarc/cli.py
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
* Update parsedmarc/cli.py
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
* [WIP] SIGHUP-based configuration reload for watch mode (#698 )
* Initial plan
* Fix reload state consistency, resource leaks, stale opts; add tests
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com >
Agent-Logs-Url: https://github.com/domainaware/parsedmarc/sessions/3c2e0bb9-7e2d-4efa-aef6-d2b98478b921
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com >
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com >
* [WIP] SIGHUP-based configuration reload for watch mode (#699 )
* Initial plan
* Fix review comments: ConfigurationError wrapping, duplicate parse args, bool parsing, Kafka required topics, should_reload kwarg, SIGHUP test skips
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com >
Agent-Logs-Url: https://github.com/domainaware/parsedmarc/sessions/0779003c-ccbe-4d76-9748-801dbc238b96
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com >
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com >
* SIGHUP-based configuration reload: address review feedback (#700 )
* Initial plan
* Address review feedback: kafka_ssl, duplicate silent, exception chain, log file reload, should_reload timing
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com >
Agent-Logs-Url: https://github.com/domainaware/parsedmarc/sessions/a8a43c55-23fa-4471-abe6-7ac966f381f9
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com >
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com >
* Update parsedmarc/cli.py
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
* Best-effort initialization for optional output clients in watch mode (#701 )
* Initial plan
* Wrap optional output client init in try/except for best-effort initialization
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com >
Agent-Logs-Url: https://github.com/domainaware/parsedmarc/sessions/59241d4e-1b05-4a92-b2d2-e6d13d10a4fd
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com >
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com >
* Fix SIGHUP reload tight-loop in watch mode (#702 )
* Initial plan
* Fix _reload_requested tight-loop: reset flag before reload to capture concurrent SIGHUPs
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com >
Agent-Logs-Url: https://github.com/domainaware/parsedmarc/sessions/879d0bb1-9037-41f7-bc89-f59611956d2e
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com >
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com >
* Update parsedmarc/cli.py
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
* Fix resource leak when HEC config is invalid in `_init_output_clients()` (#703 )
* Initial plan
* Fix resource leak: validate HEC settings before creating any output clients
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com >
Agent-Logs-Url: https://github.com/domainaware/parsedmarc/sessions/38c73e09-789d-4d41-b75e-bbc61418859d
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com >
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com >
* Ensure SIGHUP never triggers a new email batch across all watch() implementations (#704 )
* Initial plan
* Ensure SIGHUP never starts a new email batch in any watch() implementation
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com >
Agent-Logs-Url: https://github.com/domainaware/parsedmarc/sessions/45d5be30-8f6b-4200-9bdd-15c655033f17
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com >
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com >
* SIGHUP-based config reload for watch mode: address review feedback (#705 )
* Initial plan
* Address review feedback: Kafka SSL context, SIGHUP handler safety, test formatting
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com >
Agent-Logs-Url: https://github.com/domainaware/parsedmarc/sessions/8f2fd48f-32a4-4258-9a89-06f7c7ac29bf
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com >
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com >
* Reverted changes by copilot that turned errors into warnings
* Enhance usage documentation for config reload: clarify behavior on successful reload and error handling
* Update CHANGELOG.md to reflect config reload enhancements
* Add pytest command to settings for silent output during testing
* Enhance resource management: add close methods for S3Client and HECClient, and improve IMAP connection handling during IDLE. Update CHANGELOG.md for config reload improvements and bug fixes.
* Update changelog to not include fixes within the same unreleased version
* Refactor changelog entries for clarity and consistency in configuration reload section
* Fix changelog entry for msgraph configuration check
* Update CHANGELOG..md
* make single list items on one line in the changelog instead of doing hard wraps
* Remove incorrect IMAP changes
* Rename 'should_reload' parameter to 'config_reloading' in mailbox connection methods for clarity
* Restore startup configuration checks
* Improve error logging for Elasticsearch and OpenSearch exceptions
* Bump version to 9.3.0 in constants.py
* Refactor GelfClient methods to use specific report types instead of generic dicts
* Refactor tests to use assertions consistently and improve type hints
---------
Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com >
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com >
2026-03-21 16:14:48 -04:00
dd1a8fd461
Create docker compose file for dashboard development
2026-03-20 14:12:26 -04:00
81656c75e9
Update OpenSearch healthcheck to use HTTPS and include authentication
2026-03-16 17:53:37 -04:00
691b0fcd41
Fix changelog headings
2026-03-10 20:34:13 -04:00
b9343a295f
9.2.1
...
- Better checking of `msconfig` configuration (PR #695 )
- Updated `dbip-country-lite` database to version `2026-03`
- Changed - DNS query error logging level from `warning` to `debug`
2026-03-10 20:32:33 -04:00
b51a62463f
Fail fast on invalid MS Graph username/password config ( #695 )
2026-03-10 19:34:16 -04:00
66ba5b0e5e
Add MS Graph auth matrix regression tests ( #696 )
...
* Rebase MS Graph auth matrix tests onto current master
* Expand ClientSecret auth matrix coverage
2026-03-10 19:33:37 -04:00
7929919223
9.2.0
...
### Added
- OpenSearch AWS SigV4 authentication support (PR #673 )
- IMAP move/delete compatibility fallbacks (PR #671 )
- `fail_on_output_error` CLI option for sink failures (PR #672 )
- Gmail service account auth mode for non-interactive runs (PR #676 )
- Microsoft Graph certificate authentication support (PRs #692 and #693 )
- Microsoft Graph well-known folder fallback for root listing failures (PR #618 and #684 close #609 )
### Fixed
- Pass mailbox since filter through `watch_inbox` callback (PR #670 closes issue #581 )
- `parsedmarc.mail.gmail.GmailConnection.delete_message` now properly calls the Gmail API (PR #668 )
- Avoid extra mailbox fetch in batch and test mode (PR #691 closes #533 )
2026-03-10 11:41:37 -04:00
faa68333a9
Avoid extra mailbox fetch in batch/test mode and add regression test ( #691 )
...
Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com >
2026-03-10 11:22:39 -04:00
d34a33e980
Validate MS Graph certificate auth inputs ( #693 )
...
* Validate MS Graph certificate auth inputs
* Fix MS Graph shared scope detection without username
2026-03-10 11:22:09 -04:00
9040a38842
Refine MS Graph well-known folder fallback ( #694 )
...
* Refine MS Graph well-known folder fallback
* Make MS Graph retry test doubles method-aware
2026-03-10 11:20:43 -04:00
ea0e3b11c1
Add MS Graph certificate authentication support ( #692 )
...
* Add MS Graph certificate authentication support
* Preserve MS Graph constructor compatibility
---------
Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com >
2026-03-10 09:30:39 -04:00
199b782191
Add MS Graph well-known folder fallback for root listing failures ( #689 )
...
* Add MS Graph well-known folder fallback for root listing failures
* Resolve test merge cleanup for MS Graph folder fallback
2026-03-10 09:25:37 -04:00
25f3c3e1d0
Add security policy ( #688 )
...
* Add security policy
* Update SECURITY.md for vulnerability reporting clarity
Clarified instructions for reporting vulnerabilities and updated language regarding security fixes.
---------
Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com >
2026-03-09 18:24:16 -04:00
a14ff66f5a
Add GitHub issue templates ( #686 )
2026-03-09 18:17:06 -04:00
fb738bf9c4
Add contributing guide ( #685 )
2026-03-09 18:16:47 -04:00
0e811fe0ff
Add pull request template ( #687 )
2026-03-09 18:15:40 -04:00
56eb565ad2
Accept pathlib.Path in report parsing APIs ( #680 )
...
* Accept pathlib.Path in report parsing APIs
* Polish PathLike typing and test names
2026-03-09 18:08:57 -04:00
2c3abb3e8c
Retry transient MS Graph request errors ( #679 )
...
* Retry transient MS Graph request errors
* Handle zero MS Graph retry attempts explicitly
2026-03-09 17:56:22 -04:00
326e630f50
Add performance tuning guidance for large mailbox runs ( #677 )
2026-03-09 17:44:42 -04:00
cdc30e6780
Tune Codecov statuses for small PRs ( #678 )
2026-03-09 17:43:34 -04:00
f2febf21d3
Add fail_on_output_error CLI option for sink failures ( #672 )
...
* Add fail-on-output-error option and CLI regression test
* Broaden fail_on_output_error coverage for disabled and multi-sink paths
2026-03-09 17:35:38 -04:00
79f47121a4
Pass mailbox since filter through watch_inbox callback ( #670 )
...
* Pass mailbox since through watch loop and add regression test
* Add CLI regression test for mailbox since in watch mode
2026-03-09 17:33:42 -04:00
6e6c90e19b
Add IMAP move/delete compatibility fallbacks ( #671 )
...
* Add IMAP move/delete compatibility fallbacks with tests
* Expand IMAP fallback tests for success and error paths
2026-03-09 17:29:01 -04:00
c4d7455839
Add OpenSearch AWS SigV4 authentication support ( #673 )
...
* Add OpenSearch AWS SigV4 authentication support
* Increase SigV4 coverage for auth validation and CLI config wiring
* Update parsedmarc/opensearch.py
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
* Update docs/source/usage.md
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
---------
Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com >
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
2026-03-09 17:21:22 -04:00
95e6fb85a1
Fix Gmail delete_message to execute API request ( #668 )
...
* Fix Gmail delete to execute request and add regression test
* Fix duplicate GmailConnection import in tests
2026-03-09 17:11:35 -04:00
298d5b6e6e
CI: split lint/docs/build from integration tests matrix ( #669 )
...
* Optimize CI: split lint/docs/build from integration tests
* Trim unnecessary package install from lint job
2026-03-09 17:09:02 -04:00
a3c5bb906b
Add Gmail service account auth mode with delegated user support ( #676 )
2026-03-09 17:04:30 -04:00
d49ce6a13f
Increase unit test coverage for Gmail/Graph/IMAP connectors ( #664 )
...
* Increase coverage for Gmail, Graph, and IMAP mail connectors
* Make testLoadTokenMissing use guaranteed-missing temp path
* Expand coverage for Gmail token refresh and Graph pagination error paths
2026-03-09 11:54:43 -04:00
adb0d31382
9.1.2
...
- Fix duplicate detection for normalized aggregate reports in Elasticsearch/OpenSearch (PR #666 fixes issue #665 )
2026-03-06 13:41:33 -05:00
ae5d20ecf5
Fix duplicate detection for normalized aggregate reports in Elasticsearch/OpenSearch ( #666 )
...
Change date_begin/date_end queries from exact match to range queries
(gte/lte) so that previously saved normalized time buckets are correctly
detected as duplicates within the original report's date range.
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com >
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com >
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com >
2026-03-06 13:21:54 -05:00
e98fdfa96b
Fix Python 3.14 support metadata and require imapclient 3.1.0 ( #662 )
2026-03-04 12:36:15 -05:00
9551c8b467
Add AGENTS.md for AI agent guidance and link from CLAUDE.md
2026-03-03 21:00:55 -05:00
d987943c22
Update changelog formatting for version 9.1.1
2026-03-03 11:46:13 -05:00
3d8a99b5d3
9.1.1
...
- Fix the use of Elasticsearch and OpenSearch API keys (PR #660 fixes issue #653 )
- Drop support for Python 3.9 (PR #661 )
2026-03-03 11:43:53 -05:00
5aaaedf463
Use correct key names for elasticsearch/opensearch api keys ( #660 )
2026-03-03 11:35:05 -05:00
2e3ee25ec9
Drop Python 3.9 support ( #661 )
...
* Initial plan
* Drop Python 3.9 support: update CI matrix, pyproject.toml, docs, and README
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com >
* Update Python 3.9 version table entry to note Debian 11/RHEL 9 usage
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com >
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com >
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com >
2026-03-03 11:34:35 -05:00
33eb2aaf62
9.1.0
...
## Enhancements
- Add TCP and TLS support for syslog output. (#656 )
- Skip DNS lookups in GitHub Actions to prevent DNS timeouts during tests timeouts. (#657 )
- Remove microseconds from DMARC aggregate report time ranges before parsing them.
2026-02-20 14:36:37 -05:00
1387fb4899
9.0.11
...
- Remove microseconds from DMARC aggregate report time ranges before parsing them.
2026-02-20 14:27:51 -05:00
4d97bd25aa
Skip DNS lookups in GitHub Actions to prevent test timeouts ( #657 )
...
* Add offline mode for tests in GitHub Actions to skip DNS lookups
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com >
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com >
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com >
2026-02-18 18:19:28 -05:00
17a612df0c
Add TCP and TLS transport support to syslog module ( #656 )
...
- Updated parsedmarc/syslog.py to support UDP, TCP, and TLS protocols
- Added protocol parameter with UDP as default for backward compatibility
- Implemented TLS support with CA verification and client certificate auth
- Added retry logic for TCP/TLS connections with configurable attempts and delays
- Updated parsedmarc/cli.py with new config file parsing
- Updated documentation with examples for TCP and TLS configurations
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com >
* Remove CLI arguments for syslog options, keep config-file only
Per user request, removed command-line argument options for syslog parameters.
All new syslog options (protocol, TLS settings, timeout, retry) are now only
available via configuration file, consistent with other similar options.
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com >
* Fix code review issues: remove trailing whitespace and add cert validation
- Removed trailing whitespace from syslog.py and usage.md
- Added warning when only one of certfile_path/keyfile_path is provided
- Improved error handling for incomplete TLS client certificate configuration
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com >
* Set minimum TLS version to 1.2 for enhanced security
Explicitly configured ssl_context.minimum_version = TLSVersion.TLSv1_2
to ensure only secure TLS versions are used for syslog connections.
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com >
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com >
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com >
2026-02-18 18:12:59 -05:00
221bc332ef
Fixed a typo in policies.successful_session_count ( #654 )
2026-02-09 13:57:11 -05:00
a2a75f7a81
Fix timestamp parsing in aggregate report by removing fractional seconds
2026-01-21 13:08:48 -05:00
50fcb51577
Update supported Python versions in docs + readme ( #652 )
...
* Update README.md
* Update index.md
* Update python-tests.yml
2026-01-19 14:40:01 -05:00
copilot-swe-agent[bot]
Sean Whalen
Sean Whalen
Kili
Majid Burney
Blackmoon
Anael Mobilia