amorfo77/parsedmarc
1
0
Fork 0
mirror of https://github.com/domainaware/parsedmarc.git synced 2026-03-26 08:22:45 +00:00
Code Issues Projects Releases Wiki Activity
1,435 Commits 26 Branches 34 Tags
733ea58d202eab7c43fbd7d38532ca4e9c348339
Commit Graph

1435 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
copilot-swe-agent[bot]
733ea58d20 Make dashboard queries backward compatible to show data from both forensic and failure indexes 
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
 2026-03-26 01:43:31 -04:00
copilot-swe-agent[bot]
2d7a56b740 Fix Splunk sourcetype to use colon separator (dmarc:failure) matching original convention 
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
 2026-03-26 01:43:31 -04:00
copilot-swe-agent[bot]
2c5a65f88c Add DMARCbis field validation, preserve pass disposition, add comprehensive tests 
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
 2026-03-26 01:43:31 -04:00
copilot-swe-agent[bot]
c996e2a5ac Rename forensic→failure in cli.py, docs, dashboards; add DMARCbis fields to ES/OS output 
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
 2026-03-26 01:43:31 -04:00
copilot-swe-agent[bot]
0e54e9a23d Rename "forensic" to "failure" in docs and dashboard configs 
Update documentation files (output.md, usage.md, kibana.md, splunk.md,
elasticsearch.md, index.md, example.ini) and dashboard configurations
(Grafana JSON, Kibana ndjson, Splunk XML) to use "failure" terminology
instead of "forensic", consistent with the codebase rename.

- CLI args: --forensic-* → --failure-*
- Config keys: save_forensic → save_failure, forensic_topic → failure_topic, etc.
- Index names: dmarc_forensic → dmarc_failure
- Splunk dashboard: renamed file from dmarc_forensic_dashboard.xml to dmarc_failure_dashboard.xml
- Backward-compat note preserved: "formerly known as forensic reports"

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-26 01:43:31 -04:00
copilot-swe-agent[bot]
8ebc290ea1 Rename forensic references to failure in cli.py 
- Rename all forensic_* variables to failure_*
- Update CLI argument names (--forensic-* to --failure-*)
- Update default filenames (forensic.json/csv to failure.json/csv)
- Update function calls to match renamed output module functions
- Update index names (dmarc_forensic to dmarc_failure)
- Update report type strings and dict keys
- Add backward-compatible config key reading (accept both old and new names)
- Update help text and log messages

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-26 01:43:31 -04:00
copilot-swe-agent[bot]
07c518f19d Rename forensic to failure in output/integration modules 
Rename all 'forensic' references to 'failure' in the output modules:
- elastic.py, opensearch.py, splunk.py, kafkaclient.py, syslog.py,
  gelf.py, webhook.py, loganalytics.py, s3.py

Changes include:
- Function/method names: save_forensic_* → save_failure_*
- Variable/parameter names: forensic_* → failure_*
- Class names: _ForensicReportDoc → _FailureReportDoc,
  _ForensicSampleDoc → _FailureSampleDoc
- Index/topic/sourcetype names: dmarc_forensic → dmarc_failure
- Log messages and docstrings updated
- Import statements updated to use new names from core module
- Backward-compatible aliases added at end of each file
- DMARCbis aggregate fields added to elastic.py and opensearch.py:
  np (Keyword), testing (Keyword), discovery_method (Keyword),
  generator (Text)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-26 01:42:28 -04:00
copilot-swe-agent[bot]
ac996a8edf Align DMARCbis fields with actual XSD schema: testing, discovery_method, generator, human_result; handle namespaced XML 
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
 2026-03-26 01:42:05 -04:00
copilot-swe-agent[bot]
39f2884acc Add DMARCbis fields (np, psd, t) to aggregate reports and rename forensic→failure in core parsing 
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
 2026-03-26 01:42:05 -04:00
copilot-swe-agent[bot]
4d2db6a0fb Rename forensic references to failure with backward-compatible aliases 
- Rename parse_forensic_report -> parse_failure_report
- Rename parsed_forensic_reports_to_csv_rows -> parsed_failure_reports_to_csv_rows
- Rename parsed_forensic_reports_to_csv -> parsed_failure_reports_to_csv
- Update all internal variable names (forensic_report -> failure_report, etc.)
- Change report_type from 'forensic' to 'failure'
- Use FailureReport type instead of ForensicReport
- Use InvalidFailureReport instead of InvalidForensicReport in function bodies
- Update all docstrings and log messages
- Add backward-compatible aliases at end of file

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-03-26 01:42:05 -04:00
copilot-swe-agent[bot]
b9fb4dbd59 Initial plan 2026-03-26 01:42:05 -04:00
Sean Whalen
1542936468 Bump version to 9.5.4, enhance Maildir folder handling, and add config key aliases for environment variable compatibility 9.5.4 2026-03-25 23:22:46 -04:00
Sean Whalen
fb3c38a8b8 9.5.3 
- Fixed `FileNotFoundError` when using Maildir with Docker volume mounts. Python's `mailbox.Maildir(create=True)` only creates `cur/new/tmp` subdirectories when the top-level directory doesn't exist; Docker volume mounts pre-create the directory as empty, skipping subdirectory creation. parsedmarc now explicitly creates the subdirectories when `maildir_create` is enabled.
- Maildir UID mismatch no longer crashes the process. In Docker containers where volume ownership differs from the container UID, parsedmarc now logs a warning instead of raising an exception. Also handles `os.setuid` failures gracefully in containers without `CAP_SETUID`.
- Token file writes (MS Graph and Gmail) now create parent directories automatically, preventing `FileNotFoundError` when the token path points to a directory that doesn't yet exist.
- File paths from config (`token_file`, `credentials_file`, `cert_path`, `log_file`, `output`, `ip_db_path`, `maildir_path`, syslog cert paths, etc.) now expand `~` and `$VAR` references via `os.path.expanduser`/`os.path
9.5.3 		2026-03-25 21:29:08 -04:00
Sean Whalen
c9a6145505 9.5.3 
- Fixed `FileNotFoundError` when using Maildir with Docker volume mounts. Python's `mailbox.Maildir(create=True)` only creates `cur/new/tmp` subdirectories when the top-level directory doesn't exist; Docker volume mounts pre-create the directory as empty, skipping subdirectory creation. parsedmarc now explicitly creates the subdirectories when `maildir_create` is enabled.
 2026-03-25 21:13:34 -04:00
Sean Whalen
e1bdbeb257 Bump version to 9.5.2 and fix interpolation issues in config parser 9.5.2 2026-03-25 20:21:08 -04:00
Sean Whalen
12c4676b79 9.5.1 
- Correct ISO format for MSGraphConnection timestamps (PR #706)
9.5.1 		2026-03-25 19:43:24 -04:00
mihugo
cda039ee27 Correct ISO format for MSGraphConnection timestamps (#706) 
Fix formatting of ISO 8601 date strings for MSGraphConnection.  format yyyy-dd-mmThh:MM:SS.zzzzzz+00:00 already has a timezone indicated. The extra Z is invalid in this format.  specifying a "since" in config file causes msgraph to error due to invalid time stamp.
 2026-03-25 19:38:23 -04:00
Sean Whalen
ff0ca6538c 9.5.0 
Add environment variable configuration support and update documentation

- Introduced support for configuration via environment variables using the `PARSEDMARC_{SECTION}_{KEY}` format.
- Added `PARSEDMARC_CONFIG_FILE` variable to specify the config file path.
- Enabled env-only mode for file-less Docker deployments.
- Implemented explicit read permission checks on config files.
- Updated changelog and usage documentation to reflect these changes.
9.5.0 		2026-03-25 19:25:21 -04:00
Sean Whalen
2032438d3b 9.4.0 
### Added

- Extracted `load_reverse_dns_map()` utility function in `utils.py` for loading the reverse DNS map independently of individual IP lookups.
- SIGHUP reload now re-downloads/reloads the reverse DNS map, so changes take effect without restarting.
- Add premade OpenSearch index patterns, visualizations, and dashboards

### Changed

- When `index_prefix_domain_map` is configured, SMTP TLS reports for domains not in the map are now silently dropped instead of being output. Unlike DMARC, TLS-RPT has no DNS authorization records, so this filtering prevents processing reports for unrelated domains.
- Bump OpenSearch support to `< 4`

### Fixed

- Fixed `get_index_prefix` using wrong key (`domain` instead of `policy_domain`) for SMTP TLS reports, which prevented domain map matching from working for TLS reports.
- Domain matching in `get_index_prefix` now lowercases the domain for case-insensitive comparison.
9.4.0 		2026-03-23 17:08:26 -04:00
Sean Whalen
1e95c5d30b 9.3.1 
Elasticsearch and OpenSearch now verify SSL certificates by default when `ssl = True`, even without a `cert_path`
- Added `skip_certificate_verification` option to the `elasticsearch` and `opensearch` configuration sections for consistency with `splunk_hec`
- Splunk HEC `skip_certificate_verification` now works correctly with self-signed certificates
- SMTP TLS reports no longer fail when saving to multiple output targets (e.g. Elasticsearch and OpenSearch) due to in-place mutation of the report dict
- Output client initialization errors now identify which module failed (e.g. "OpenSearch: ConnectionError..." instead of generic "Output client error")
- Enhanced error handling for output client initialization
9.3.1 		2026-03-22 14:38:32 -04:00
Sean Whalen
cb2384be83 Copy report before modifying begin_date and end_date in save_smtp_tls_report functions 2026-03-22 13:13:21 -04:00
Sean Whalen
9a5b5310fa Update Grafana and Splunk environment variables in docker-compose for consistency 2026-03-22 12:40:42 -04:00
Sean Whalen
9849598100 Formatting 9.3.0 2026-03-21 16:17:35 -04:00
Sean Whalen
e82f3e58a1 SIGHUP-based configuration reload for watch mode (#697) 
* Enhance mailbox connection watch method to support reload functionality

- Updated the `watch` method in `GmailConnection`, `MSGraphConnection`, `IMAPConnection`, `MaildirConnection`, and the abstract `MailboxConnection` class to accept an optional `should_reload` parameter. This allows the method to check if a reload is necessary and exit the loop if so.
- Modified related tests to accommodate the new method signature.
- Changed logger calls from `critical` to `error` for consistency in logging severity.
- Added a new settings file for Claude with specific permissions for testing and code checks.

* Update parsedmarc/cli.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update parsedmarc/cli.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* [WIP] SIGHUP-based configuration reload for watch mode (#698)

* Initial plan

* Fix reload state consistency, resource leaks, stale opts; add tests

Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
Agent-Logs-Url: https://github.com/domainaware/parsedmarc/sessions/3c2e0bb9-7e2d-4efa-aef6-d2b98478b921

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>

* [WIP] SIGHUP-based configuration reload for watch mode (#699)

* Initial plan

* Fix review comments: ConfigurationError wrapping, duplicate parse args, bool parsing, Kafka required topics, should_reload kwarg, SIGHUP test skips

Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
Agent-Logs-Url: https://github.com/domainaware/parsedmarc/sessions/0779003c-ccbe-4d76-9748-801dbc238b96

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>

* SIGHUP-based configuration reload: address review feedback (#700)

* Initial plan

* Address review feedback: kafka_ssl, duplicate silent, exception chain, log file reload, should_reload timing

Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
Agent-Logs-Url: https://github.com/domainaware/parsedmarc/sessions/a8a43c55-23fa-4471-abe6-7ac966f381f9

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>

* Update parsedmarc/cli.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Best-effort initialization for optional output clients in watch mode (#701)

* Initial plan

* Wrap optional output client init in try/except for best-effort initialization

Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
Agent-Logs-Url: https://github.com/domainaware/parsedmarc/sessions/59241d4e-1b05-4a92-b2d2-e6d13d10a4fd

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>

* Fix SIGHUP reload tight-loop in watch mode (#702)

* Initial plan

* Fix _reload_requested tight-loop: reset flag before reload to capture concurrent SIGHUPs

Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
Agent-Logs-Url: https://github.com/domainaware/parsedmarc/sessions/879d0bb1-9037-41f7-bc89-f59611956d2e

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>

* Update parsedmarc/cli.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Fix resource leak when HEC config is invalid in `_init_output_clients()` (#703)

* Initial plan

* Fix resource leak: validate HEC settings before creating any output clients

Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
Agent-Logs-Url: https://github.com/domainaware/parsedmarc/sessions/38c73e09-789d-4d41-b75e-bbc61418859d

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>

* Ensure SIGHUP never triggers a new email batch across all watch() implementations (#704)

* Initial plan

* Ensure SIGHUP never starts a new email batch in any watch() implementation

Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
Agent-Logs-Url: https://github.com/domainaware/parsedmarc/sessions/45d5be30-8f6b-4200-9bdd-15c655033f17

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>

* SIGHUP-based config reload for watch mode: address review feedback (#705)

* Initial plan

* Address review feedback: Kafka SSL context, SIGHUP handler safety, test formatting

Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
Agent-Logs-Url: https://github.com/domainaware/parsedmarc/sessions/8f2fd48f-32a4-4258-9a89-06f7c7ac29bf

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>

* Reverted changes by copilot that turned errors into warnings

* Enhance usage documentation for config reload: clarify behavior on successful reload and error handling

* Update CHANGELOG.md to reflect config reload enhancements

* Add pytest command to settings for silent output during testing

* Enhance resource management: add close methods for S3Client and HECClient, and improve IMAP connection handling during IDLE. Update CHANGELOG.md for config reload improvements and bug fixes.

* Update changelog to not include fixes within the same unreleased version

* Refactor changelog entries for clarity and consistency in configuration reload section

* Fix changelog entry for msgraph configuration check

* Update CHANGELOG..md

* make single list items on one line in the changelog instead of doing hard wraps

* Remove incorrect IMAP changes

* Rename 'should_reload' parameter to 'config_reloading' in mailbox connection methods for clarity

* Restore startup configuration checks

* Improve error logging for Elasticsearch and OpenSearch exceptions

* Bump version to 9.3.0 in constants.py

* Refactor GelfClient methods to use specific report types instead of generic dicts

* Refactor tests to use assertions consistently and improve type hints

---------

Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
 2026-03-21 16:14:48 -04:00
Sean Whalen
dd1a8fd461 Create docker compose file for dashboard development 2026-03-20 14:12:26 -04:00
Sean Whalen
81656c75e9 Update OpenSearch healthcheck to use HTTPS and include authentication 2026-03-16 17:53:37 -04:00
Sean Whalen
691b0fcd41 Fix changelog headings 9.2.1 2026-03-10 20:34:13 -04:00
Sean Whalen
b9343a295f 9.2.1 
- Better checking of `msconfig` configuration (PR #695)
- Updated `dbip-country-lite` database to version `2026-03`
- Changed - DNS query error logging level from `warning` to `debug`
 2026-03-10 20:32:33 -04:00
Kili
b51a62463f Fail fast on invalid MS Graph username/password config (#695) 2026-03-10 19:34:16 -04:00
Kili
66ba5b0e5e Add MS Graph auth matrix regression tests (#696) 
* Rebase MS Graph auth matrix tests onto current master

* Expand ClientSecret auth matrix coverage
 2026-03-10 19:33:37 -04:00
Sean Whalen
7929919223 9.2.0 
### Added

- OpenSearch AWS SigV4 authentication support (PR #673)
- IMAP move/delete compatibility fallbacks (PR #671)
- `fail_on_output_error` CLI option for sink failures (PR #672)
- Gmail service account auth mode for non-interactive runs (PR #676)
- Microsoft Graph certificate authentication support (PRs #692 and #693)
- Microsoft Graph well-known folder fallback for root listing failures (PR #618 and #684 close #609)

### Fixed

- Pass mailbox since filter through `watch_inbox` callback (PR #670 closes issue #581)
- `parsedmarc.mail.gmail.GmailConnection.delete_message` now properly calls the Gmail API (PR #668)
- Avoid extra mailbox fetch in batch and test mode (PR #691 closes #533)
9.2.0 		2026-03-10 11:41:37 -04:00
Kili
faa68333a9 Avoid extra mailbox fetch in batch/test mode and add regression test (#691) 
Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>
 2026-03-10 11:22:39 -04:00
Kili
d34a33e980 Validate MS Graph certificate auth inputs (#693) 
* Validate MS Graph certificate auth inputs

* Fix MS Graph shared scope detection without username
 2026-03-10 11:22:09 -04:00
Kili
9040a38842 Refine MS Graph well-known folder fallback (#694) 
* Refine MS Graph well-known folder fallback

* Make MS Graph retry test doubles method-aware
 2026-03-10 11:20:43 -04:00
Kili
ea0e3b11c1 Add MS Graph certificate authentication support (#692) 
* Add MS Graph certificate authentication support

* Preserve MS Graph constructor compatibility

---------

Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>
 2026-03-10 09:30:39 -04:00
Kili
199b782191 Add MS Graph well-known folder fallback for root listing failures (#689) 
* Add MS Graph well-known folder fallback for root listing failures

* Resolve test merge cleanup for MS Graph folder fallback
 2026-03-10 09:25:37 -04:00
Kili
25f3c3e1d0 Add security policy (#688) 
* Add security policy

* Update SECURITY.md for vulnerability reporting clarity

Clarified instructions for reporting vulnerabilities and updated language regarding security fixes.

---------

Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>
 2026-03-09 18:24:16 -04:00
Kili
a14ff66f5a Add GitHub issue templates (#686) 2026-03-09 18:17:06 -04:00
Kili
fb738bf9c4 Add contributing guide (#685) 2026-03-09 18:16:47 -04:00
Kili
0e811fe0ff Add pull request template (#687) 2026-03-09 18:15:40 -04:00
Kili
56eb565ad2 Accept pathlib.Path in report parsing APIs (#680) 
* Accept pathlib.Path in report parsing APIs

* Polish PathLike typing and test names
 2026-03-09 18:08:57 -04:00
Kili
2c3abb3e8c Retry transient MS Graph request errors (#679) 
* Retry transient MS Graph request errors

* Handle zero MS Graph retry attempts explicitly
 2026-03-09 17:56:22 -04:00
Kili
326e630f50 Add performance tuning guidance for large mailbox runs (#677) 2026-03-09 17:44:42 -04:00
Kili
cdc30e6780 Tune Codecov statuses for small PRs (#678) 2026-03-09 17:43:34 -04:00
Kili
f2febf21d3 Add fail_on_output_error CLI option for sink failures (#672) 
* Add fail-on-output-error option and CLI regression test

* Broaden fail_on_output_error coverage for disabled and multi-sink paths
 2026-03-09 17:35:38 -04:00
Kili
79f47121a4 Pass mailbox since filter through watch_inbox callback (#670) 
* Pass mailbox since through watch loop and add regression test

* Add CLI regression test for mailbox since in watch mode
 2026-03-09 17:33:42 -04:00
Kili
6e6c90e19b Add IMAP move/delete compatibility fallbacks (#671) 
* Add IMAP move/delete compatibility fallbacks with tests

* Expand IMAP fallback tests for success and error paths
 2026-03-09 17:29:01 -04:00
Kili
c4d7455839 Add OpenSearch AWS SigV4 authentication support (#673) 
* Add OpenSearch AWS SigV4 authentication support

* Increase SigV4 coverage for auth validation and CLI config wiring

* Update parsedmarc/opensearch.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update docs/source/usage.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2026-03-09 17:21:22 -04:00
Kili
95e6fb85a1 Fix Gmail delete_message to execute API request (#668) 
* Fix Gmail delete to execute request and add regression test

* Fix duplicate GmailConnection import in tests
 2026-03-09 17:11:35 -04:00
Kili
298d5b6e6e CI: split lint/docs/build from integration tests matrix (#669) 
* Optimize CI: split lint/docs/build from integration tests

* Trim unnecessary package install from lint job
 2026-03-09 17:09:02 -04:00