amorfo77/parsedmarc
1
0
Fork 0
mirror of https://github.com/domainaware/parsedmarc.git synced 2026-04-03 12:18:54 +00:00
Code Issues Projects Releases Wiki Activity
1,415 Commits 26 Branches 35 Tags
fce8e2247b0d9cb4c85139b4e43bb5d2bcac7b37
Commit Graph

1415 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
copilot-swe-agent[bot]
fce8e2247b Fix ruff formatting errors, duplicate import, and test mock key names 
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
 2026-03-09 21:40:28 +00:00
Sean Whalen
b4b90e763d Merge branch 'master' into copilot/support-dmarcbis-reports 2026-03-09 17:20:20 -04:00
Kili
95e6fb85a1 Fix Gmail delete_message to execute API request (#668) 
* Fix Gmail delete to execute request and add regression test

* Fix duplicate GmailConnection import in tests
 2026-03-09 17:11:35 -04:00
Kili
298d5b6e6e CI: split lint/docs/build from integration tests matrix (#669) 
* Optimize CI: split lint/docs/build from integration tests

* Trim unnecessary package install from lint job
 2026-03-09 17:09:02 -04:00
Kili
a3c5bb906b Add Gmail service account auth mode with delegated user support (#676) 2026-03-09 17:04:30 -04:00
Sean Whalen
73716fa671 Merge branch 'master' into copilot/support-dmarcbis-reports 2026-03-09 17:02:36 -04:00
Kili
d49ce6a13f Increase unit test coverage for Gmail/Graph/IMAP connectors (#664) 
* Increase coverage for Gmail, Graph, and IMAP mail connectors

* Make testLoadTokenMissing use guaranteed-missing temp path

* Expand coverage for Gmail token refresh and Graph pagination error paths
 2026-03-09 11:54:43 -04:00
Sean Whalen
adb0d31382 9.1.2 
- Fix duplicate detection for normalized aggregate reports in Elasticsearch/OpenSearch (PR #666 fixes issue #665)
9.1.2 		2026-03-06 13:41:33 -05:00
Copilot
ae5d20ecf5 Fix duplicate detection for normalized aggregate reports in Elasticsearch/OpenSearch (#666) 
Change date_begin/date_end queries from exact match to range queries
(gte/lte) so that previously saved normalized time buckets are correctly
detected as duplicates within the original report's date range.

Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
 2026-03-06 13:21:54 -05:00
Kili
e98fdfa96b Fix Python 3.14 support metadata and require imapclient 3.1.0 (#662) 2026-03-04 12:36:15 -05:00
copilot-swe-agent[bot]
aea6c101d7 Rename samples/forensic→samples/failure; remove 'DMARC 2.0' references 
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
 2026-03-04 02:19:35 +00:00
copilot-swe-agent[bot]
72fb0d402e Update AGENTS.md to reflect forensic→failure rename and DMARCbis support 
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
 2026-03-04 02:08:32 +00:00
Sean Whalen
4b6a398484 Merge branch 'master' into copilot/support-dmarcbis-reports 2026-03-03 21:03:57 -05:00
Sean Whalen
9551c8b467 Add AGENTS.md for AI agent guidance and link from CLAUDE.md 2026-03-03 21:00:55 -05:00
Sean Whalen
d987943c22 Update changelog formatting for version 9.1.1 9.1.1 2026-03-03 11:46:13 -05:00
Sean Whalen
3d8a99b5d3 9.1.1 
- Fix the use of Elasticsearch and OpenSearch API keys (PR #660 fixes issue #653)
- Drop support for Python 3.9 (PR #661)
 2026-03-03 11:43:53 -05:00
Majid Burney
5aaaedf463 Use correct key names for elasticsearch/opensearch api keys (#660) 2026-03-03 11:35:05 -05:00
Copilot
2e3ee25ec9 Drop Python 3.9 support (#661) 
* Initial plan

* Drop Python 3.9 support: update CI matrix, pyproject.toml, docs, and README

Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>

* Update Python 3.9 version table entry to note Debian 11/RHEL 9 usage

Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
 2026-03-03 11:34:35 -05:00
copilot-swe-agent[bot]
01c2e623bb Improve tests: consolidate imports, use context managers, add subTest, add backward compat and coverage tests 
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
 2026-02-21 19:56:33 +00:00
copilot-swe-agent[bot]
68ccb0eb35 Add 89 comprehensive tests covering core parsing and utilities 
Tests cover:
- _bucket_interval_by_day: all validation branches and distribution logic
- _append_parsed_record: normalize=True/False paths
- _parse_report_record: None source_ip, missing auth results, reason handling,
  identities/identifiers mapping, human_result, envelope_from fallback, alignment
- _parse_smtp_tls_failure_details: required/optional fields, missing field errors
- _parse_smtp_tls_report_policy: valid/invalid types, policy_strings, failure details
- parse_smtp_tls_report_json: valid/bytes/missing fields/non-list policies
- Aggregate report: invalid np/testing/discovery_method, pass disposition,
  multiple records, XML recovery, schema versions, generator, errors, defaults,
  normalization, MAGIC_XML_TAG detection
- utils: timestamp conversions, IP geo lookup, reverse DNS service lookup,
  IP address info with cache, email address parsing, filename safe strings,
  mbox/outlook msg detection
- Output modules: WebhookClient, KafkaClient static methods, HECClient,
  SyslogClient, LogAnalyticsConfig/Client, backward-compatible aliases

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-02-21 19:39:16 +00:00
Sean Whalen
696aa34b77 Update parsedmarc/types.py 
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2026-02-21 14:24:23 -05:00
Sean Whalen
e462d16f16 Update changelog to clarify DMARCbis field defaults and remove pct/fo defaults 2026-02-21 18:53:11 +00:00
copilot-swe-agent[bot]
bb4be6e609 Make pct and fo default to None when not provided (DMARCbis reports) 
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
 2026-02-21 18:50:37 +00:00
Sean Whalen
d9d4c23066 Fix typo in changelog regarding RFC7489 compatibility 2026-02-21 18:38:33 +00:00
Sean Whalen
c7aa976cf9 Fix np field 2026-02-21 18:28:01 +00:00
Sean Whalen
af4a78f944 Move np field 2026-02-21 18:24:34 +00:00
Sean Whalen
c904dfc24f Move 'np' field in the CSV export of parsed aggregate reports 2026-02-21 18:22:15 +00:00
Sean Whalen
a327408cb5 Update changelog for DMARCbis report support and rename forensic reports to failure reports; bump version to 10.0.0 2026-02-21 18:02:16 +00:00
copilot-swe-agent[bot]
970a2008d5 Fix ruff F401: use redundant alias for re-exported ForensicReport 
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
 2026-02-21 17:38:39 +00:00
copilot-swe-agent[bot]
2ce864fa81 Make dashboard queries backward compatible to show data from both forensic and failure indexes 
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
 2026-02-20 21:38:45 +00:00
copilot-swe-agent[bot]
423e0611c5 Fix Splunk sourcetype to use colon separator (dmarc:failure) matching original convention 
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
 2026-02-20 21:05:08 +00:00
copilot-swe-agent[bot]
195fdaf7b2 Add DMARCbis field validation, preserve pass disposition, add comprehensive tests 
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
 2026-02-20 21:03:23 +00:00
copilot-swe-agent[bot]
447f452735 Rename forensic→failure in cli.py, docs, dashboards; add DMARCbis fields to ES/OS output 
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
 2026-02-20 21:01:16 +00:00
copilot-swe-agent[bot]
148f4c87a9 Rename "forensic" to "failure" in docs and dashboard configs 
Update documentation files (output.md, usage.md, kibana.md, splunk.md,
elasticsearch.md, index.md, example.ini) and dashboard configurations
(Grafana JSON, Kibana ndjson, Splunk XML) to use "failure" terminology
instead of "forensic", consistent with the codebase rename.

- CLI args: --forensic-* → --failure-*
- Config keys: save_forensic → save_failure, forensic_topic → failure_topic, etc.
- Index names: dmarc_forensic → dmarc_failure
- Splunk dashboard: renamed file from dmarc_forensic_dashboard.xml to dmarc_failure_dashboard.xml
- Backward-compat note preserved: "formerly known as forensic reports"

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-02-20 20:57:18 +00:00
copilot-swe-agent[bot]
89fdbd82b0 Rename forensic references to failure in cli.py 
- Rename all forensic_* variables to failure_*
- Update CLI argument names (--forensic-* to --failure-*)
- Update default filenames (forensic.json/csv to failure.json/csv)
- Update function calls to match renamed output module functions
- Update index names (dmarc_forensic to dmarc_failure)
- Update report type strings and dict keys
- Add backward-compatible config key reading (accept both old and new names)
- Update help text and log messages

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-02-20 20:53:20 +00:00
copilot-swe-agent[bot]
f019096e5f Rename forensic to failure in output/integration modules 
Rename all 'forensic' references to 'failure' in the output modules:
- elastic.py, opensearch.py, splunk.py, kafkaclient.py, syslog.py,
  gelf.py, webhook.py, loganalytics.py, s3.py

Changes include:
- Function/method names: save_forensic_* → save_failure_*
- Variable/parameter names: forensic_* → failure_*
- Class names: _ForensicReportDoc → _FailureReportDoc,
  _ForensicSampleDoc → _FailureSampleDoc
- Index/topic/sourcetype names: dmarc_forensic → dmarc_failure
- Log messages and docstrings updated
- Import statements updated to use new names from core module
- Backward-compatible aliases added at end of each file
- DMARCbis aggregate fields added to elastic.py and opensearch.py:
  np (Keyword), testing (Keyword), discovery_method (Keyword),
  generator (Text)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-02-20 20:47:24 +00:00
copilot-swe-agent[bot]
6660be2c8c Align DMARCbis fields with actual XSD schema: testing, discovery_method, generator, human_result; handle namespaced XML 
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
 2026-02-20 20:40:37 +00:00
copilot-swe-agent[bot]
e09b8506fa Add DMARCbis fields (np, psd, t) to aggregate reports and rename forensic→failure in core parsing 
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
 2026-02-20 20:35:46 +00:00
copilot-swe-agent[bot]
c6413a4a4c Rename forensic references to failure with backward-compatible aliases 
- Rename parse_forensic_report -> parse_failure_report
- Rename parsed_forensic_reports_to_csv_rows -> parsed_failure_reports_to_csv_rows
- Rename parsed_forensic_reports_to_csv -> parsed_failure_reports_to_csv
- Update all internal variable names (forensic_report -> failure_report, etc.)
- Change report_type from 'forensic' to 'failure'
- Use FailureReport type instead of ForensicReport
- Use InvalidFailureReport instead of InvalidForensicReport in function bodies
- Update all docstrings and log messages
- Add backward-compatible aliases at end of file

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
 2026-02-20 20:33:52 +00:00
copilot-swe-agent[bot]
81fba4b8d2 Initial plan 2026-02-20 20:21:44 +00:00
Sean Whalen
33eb2aaf62 9.1.0 
## Enhancements

- Add TCP and TLS support for syslog output. (#656)
- Skip DNS lookups in GitHub Actions to prevent DNS timeouts during tests timeouts. (#657)
- Remove microseconds from DMARC aggregate report time ranges before parsing them.
 2026-02-20 14:36:37 -05:00
Sean Whalen
1387fb4899 9.0.11 
- Remove microseconds from DMARC aggregate report time ranges before parsing them.
 2026-02-20 14:27:51 -05:00
Copilot
4d97bd25aa Skip DNS lookups in GitHub Actions to prevent test timeouts (#657) 
* Add offline mode for tests in GitHub Actions to skip DNS lookups

Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
 2026-02-18 18:19:28 -05:00
Copilot
17a612df0c Add TCP and TLS transport support to syslog module (#656) 
- Updated parsedmarc/syslog.py to support UDP, TCP, and TLS protocols
- Added protocol parameter with UDP as default for backward compatibility
- Implemented TLS support with CA verification and client certificate auth
- Added retry logic for TCP/TLS connections with configurable attempts and delays
- Updated parsedmarc/cli.py with new config file parsing
- Updated documentation with examples for TCP and TLS configurations

Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>

* Remove CLI arguments for syslog options, keep config-file only

Per user request, removed command-line argument options for syslog parameters.
All new syslog options (protocol, TLS settings, timeout, retry) are now only
available via configuration file, consistent with other similar options.

Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>

* Fix code review issues: remove trailing whitespace and add cert validation

- Removed trailing whitespace from syslog.py and usage.md
- Added warning when only one of certfile_path/keyfile_path is provided
- Improved error handling for incomplete TLS client certificate configuration

Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>

* Set minimum TLS version to 1.2 for enhanced security

Explicitly configured ssl_context.minimum_version = TLSVersion.TLSv1_2
to ensure only secure TLS versions are used for syslog connections.

Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
 2026-02-18 18:12:59 -05:00
Blackmoon
221bc332ef Fixed a typo in policies.successful_session_count (#654) 2026-02-09 13:57:11 -05:00
Sean Whalen
a2a75f7a81 Fix timestamp parsing in aggregate report by removing fractional seconds 2026-01-21 13:08:48 -05:00
Anael Mobilia
50fcb51577 Update supported Python versions in docs + readme (#652) 
* Update README.md

* Update index.md

* Update python-tests.yml
 2026-01-19 14:40:01 -05:00
Sean Whalen
dd9ef90773 9.0.10 
- Support Python 3.14+
9.0.10 		2026-01-17 14:09:18 -05:00
Sean Whalen
0e3a4b0f06 9.0.9 
Validate that a string is base64-encoded before trying to base64 decode it. (PRs #648 and #649)
 2026-01-08 13:29:23 -05:00
maraspr
343b53ef18 remove newlines before b64decode (#649) 2026-01-08 12:24:20 -05:00