diff --git a/README.md b/README.md index 3abb3b6..57a8f2d 100644 --- a/README.md +++ b/README.md @@ -44,6 +44,7 @@ Thanks to all - Optionally send the results to Elasticsearch, Opensearch, and/or Splunk, for use with premade dashboards - Optionally send reports to Apache Kafka +- Optionally send reports to Google SecOps (Chronicle) in UDM format ## Python Compatibility diff --git a/docs/source/google_secops.md b/docs/source/google_secops.md new file mode 100644 index 0000000..52fea8b --- /dev/null +++ b/docs/source/google_secops.md @@ -0,0 +1,199 @@ +# Google SecOps (Chronicle) Output + +`parsedmarc` can output DMARC reports to Google SecOps (Chronicle) in UDM (Unified Data Model) format. + +## Configuration + +To enable Google SecOps output, add a `[google_secops]` section to your configuration file: + +```ini +[general] +save_aggregate = True +save_forensic = True + +[google_secops] +# Optional: Include forensic report message payload (default: False) +# For privacy, message bodies are excluded by default +include_ruf_payload = False + +# Optional: Maximum bytes of forensic message payload to include (default: 4096) +ruf_payload_max_bytes = 4096 + +# Optional: Static observer name for telemetry identification +static_observer_name = my-parsedmarc-instance + +# Optional: Static observer vendor (default: parsedmarc) +static_observer_vendor = parsedmarc + +# Optional: Static environment label (e.g., prod, dev) +static_environment = prod +``` + +## Output Format + +The Google SecOps output produces newline-delimited JSON (NDJSON) in Chronicle UDM format, which can be ingested into Google SecOps for hunting and dashboarding. + +### Event Types + +1. **DMARC_AGGREGATE**: One event per aggregate report row, preserving count and period information +2. **DMARC_FORENSIC**: One event per forensic report +3. **SMTP_TLS_REPORT**: One event per SMTP TLS failure detail +4. **DMARC_PARSE_ERROR**: Generated when parsing fails (does not crash) + +### UDM Schema + +Each event includes: + +- **metadata**: Event timestamp, type, product name, and vendor +- **principal**: Source IP address, location (country), and hostname (reverse DNS) +- **target**: Domain name (from DMARC policy) +- **security_result**: Severity level, description, and detection fields +- **additional.fields**: Extended metadata including report details, counts, and authentication results + +### Severity Heuristics + +- **HIGH**: DMARC disposition = reject +- **MEDIUM**: DMARC disposition = quarantine with partial SPF/DKIM failures +- **LOW**: DMARC disposition = none or pass + +## Example Output + +### Aggregate Report Event + +```json +{ + "event_type": "DMARC_AGGREGATE", + "metadata": { + "event_timestamp": "2018-06-19T00:00:00+00:00", + "event_type": "GENERIC_EVENT", + "product_name": "parsedmarc", + "vendor_name": "parsedmarc" + }, + "principal": { + "ip": ["199.230.200.36"], + "location": {"country_or_region": "US"} + }, + "target": { + "domain": {"name": "example.com"} + }, + "security_result": [{ + "severity": "LOW", + "description": "DMARC fail; disposition=none", + "detection_fields": [ + {"key": "dmarc_disposition", "value": "none"}, + {"key": "dmarc_policy", "value": "none"}, + {"key": "dmarc_pass", "value": "false"}, + {"key": "spf_aligned", "value": "false"}, + {"key": "dkim_aligned", "value": "false"} + ] + }], + "additional": { + "fields": [ + {"key": "report_org", "value": "example.net"}, + {"key": "report_id", "value": "b043f0e264cf4ea995e93765242f6dfb"}, + {"key": "message_count", "value": "1"} + ] + } +} +``` + +### Forensic Report Event + +```json +{ + "event_type": "DMARC_FORENSIC", + "metadata": { + "event_timestamp": "2019-04-30T02:09:00+00:00", + "event_type": "GENERIC_EVENT", + "product_name": "parsedmarc", + "vendor_name": "parsedmarc" + }, + "principal": { + "ip": ["10.10.10.10"] + }, + "target": { + "domain": {"name": "example.com"} + }, + "security_result": [{ + "severity": "MEDIUM", + "description": "DMARC forensic report: authentication failure (dmarc)", + "detection_fields": [ + {"key": "auth_failure", "value": "dmarc"} + ] + }], + "additional": { + "fields": [ + {"key": "arrival_date", "value": "2019-04-30 02:09:00"}, + {"key": "feedback_type", "value": "auth-failure"}, + {"key": "message_id", "value": "<01010101010101010101010101010101@ABAB01MS0016.someserver.loc>"} + ] + } +} +``` + +## Google SecOps Searches + +Here are some example searches you can use in Google SecOps to hunt for DMARC issues: + +### Find all DMARC failures + +``` +metadata.event_type = "GENERIC_EVENT" +metadata.product_name = "parsedmarc" +principal.ip != "" +additional.fields.key = "dmarc_pass" +additional.fields.value = "false" +``` + +### Find high severity DMARC events (rejected mail) + +``` +metadata.event_type = "GENERIC_EVENT" +metadata.product_name = "parsedmarc" +security_result.severity = "HIGH" +``` + +### Aggregate by source IP and target domain + +``` +metadata.event_type = "GENERIC_EVENT" +metadata.product_name = "parsedmarc" +| stats count() as event_count by principal.ip, target.domain.name +| sort event_count desc +``` + +### Find forensic reports with specific authentication failures + +``` +metadata.event_type = "GENERIC_EVENT" +metadata.product_name = "parsedmarc" +additional.fields.key = "auth_failure" +additional.fields.value = "dmarc" +``` + +## Privacy Considerations + +By default, forensic report message bodies are **excluded** from the output to protect privacy. If you need to include message samples for investigation: + +1. Set `include_ruf_payload = True` in your configuration +2. Adjust `ruf_payload_max_bytes` to limit the amount of data included (default: 4096 bytes) +3. Message samples will be truncated if they exceed the configured maximum + +**Note**: Be aware of data privacy regulations (GDPR, CCPA, etc.) when including message payloads in security telemetry. + +## Usage + +To output DMARC reports to Google SecOps, redirect stdout or use the output in your ingestion pipeline: + +```bash +# Output to stdout +parsedmarc -c config.ini samples/aggregate/*.xml > dmarc_events.ndjson + +# Stream to file +parsedmarc -c config.ini samples/aggregate/*.xml >> /var/log/dmarc/events.ndjson + +# Use with a log shipper (e.g., Fluentd, Logstash) +parsedmarc -c config.ini samples/aggregate/*.xml | your-log-shipper +``` + +The output is in newline-delimited JSON format, with one UDM event per line, ready for ingestion into Google SecOps. diff --git a/docs/source/index.md b/docs/source/index.md index f5196fe..6cd3bff 100644 --- a/docs/source/index.md +++ b/docs/source/index.md @@ -44,6 +44,7 @@ and Valimail. - Optionally send the results to Elasticsearch, Opensearch, and/or Splunk, for use with premade dashboards - Optionally send reports to Apache Kafka +- Optionally send reports to Google SecOps (Chronicle) in UDM format ## Python Compatibility @@ -74,6 +75,7 @@ elasticsearch opensearch kibana splunk +google_secops davmail dmarc contributing