mirror of https://github.com/domainaware/parsedmarc.git synced 2026-02-22 17:26:22 +00:00

Files

copilot-swe-agent[bot] dab2aaffda Update documentation with comprehensive field listings and correct service type examples

Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>

2026-01-14 16:56:54 +00:00

16 KiB

Raw Blame History

Google SecOps (Chronicle) Output

parsedmarc can output DMARC reports to Google SecOps (Chronicle) in UDM (Unified Data Model) format.

Configuration

To enable Google SecOps output, add a [google_secops] section to your configuration file:

Primary Method: Chronicle Ingestion API

The recommended approach is to send events directly to Chronicle via the Ingestion API:

[general]
save_aggregate = True
save_forensic = True

[google_secops]
# Required: Path to Google service account JSON credentials file
api_credentials_file = /path/to/service-account-credentials.json

# Required: Chronicle customer ID
api_customer_id = your-customer-id-here

# Optional: Chronicle region (default: us)
# Options: us, europe, asia-southeast1, me-central2, australia-southeast1
api_region = us

# Optional: Log type for Chronicle ingestion (default: DMARC)
api_log_type = DMARC

# Optional: Include forensic report message payload (default: False)
# For privacy, message bodies are excluded by default
include_ruf_payload = False

# Optional: Maximum bytes of forensic message payload to include (default: 4096)
ruf_payload_max_bytes = 4096

# Optional: Static observer name for telemetry identification
static_observer_name = my-parsedmarc-instance

# Optional: Static observer vendor (default: parsedmarc)
static_observer_vendor = parsedmarc

# Optional: Static environment label (e.g., prod, dev)
static_environment = prod

Alternative Method: stdout Output

If you prefer to use an external log shipper (Fluentd, Logstash, Chronicle forwarder), set use_stdout = True:

[google_secops]
# Output to stdout instead of Chronicle API
use_stdout = True

# Other optional configuration options (as above)
include_ruf_payload = False
ruf_payload_max_bytes = 4096
static_observer_name = my-instance
static_observer_vendor = parsedmarc
static_environment = prod

Output Format

The Google SecOps output produces newline-delimited JSON (NDJSON) in Chronicle UDM format, which can be ingested into Google SecOps for hunting and dashboarding.

Event Types

DMARC_AGGREGATE: One event per aggregate report row, preserving count and period information
DMARC_FORENSIC: One event per forensic report
SMTP_TLS_REPORT: One event per SMTP TLS failure detail
DMARC_PARSE_ERROR: Generated when parsing fails (does not crash)

UDM Schema

Each event includes:

metadata: Event timestamp, type, product name, and vendor
- event_timestamp: RFC 3339 formatted timestamp
- event_type: Always "GENERIC_EVENT" for UDM
- product_name: Always "parsedmarc"
- vendor_name: Configurable via static_observer_vendor (default: "parsedmarc")
- product_deployment_id (optional): Set via static_observer_name config
principal: Source IP address, location (country), and hostname (reverse DNS)
- ip: Array containing source IP address (always present)
- location.country_or_region (optional): ISO country code from IP geolocation
- hostname (optional): Reverse DNS hostname for source IP
target: Domain name (from DMARC policy)
- domain.name: The domain being protected by DMARC
security_result: Severity level, description, and detection fields for dashboarding
- severity: Derived severity level (HIGH/MEDIUM/LOW/ERROR)
- description: Human-readable event description
- detection_fields: Key DMARC dimensions for filtering and grouping
  - All dashboard-relevant fields use dmarc.* or smtp_tls.* prefixes for easy identification
  - Includes IP enrichment data (service name and type from reverse DNS mapping) for enhanced filtering
  - See "Detection Fields" section below for complete field listings
additional.fields (optional): Low-value context fields not typically used for dashboarding
- SPF/DKIM authentication details (e.g., spf_0_domain, spf_0_result)
- Forensic report metadata (e.g., feedback_type, message_id, authentication_results)
- Base domain, environment tags, optional message samples

Design Rationale: DMARC dimensions are placed in security_result[].detection_fields rather than additional.fields because Chronicle dashboards, stats searches, and aggregations work best with UDM label arrays. The additional.fields is a protobuf Struct intended for opaque context and is not reliably queryable for dashboard operations.

Detection Fields

Aggregate Report Fields (DMARC_AGGREGATE events):

dmarc.disposition: DMARC policy action (none, quarantine, reject)
dmarc.policy: Published DMARC policy (none, quarantine, reject)
dmarc.pass: Boolean - overall DMARC authentication result
dmarc.spf_aligned: Boolean - SPF alignment status
dmarc.dkim_aligned: Boolean - DKIM alignment status
dmarc.header_from: Header From domain
dmarc.envelope_from: Envelope From domain (MAIL FROM)
dmarc.report_org: Reporting organization name
dmarc.report_id: Unique report identifier
dmarc.report_begin: Report period start timestamp
dmarc.report_end: Report period end timestamp
dmarc.row_count: Number of messages represented by this record
dmarc.spf_result (optional): SPF authentication result (pass, fail, etc.)
dmarc.dkim_result (optional): DKIM authentication result (pass, fail, etc.)
dmarc.source_service_name (optional): Enriched service name from reverse DNS mapping
dmarc.source_service_type (optional): Enriched service type (e.g., "Email Provider", "Webmail", "Marketing")

Forensic Report Fields (DMARC_FORENSIC events):

dmarc.auth_failure: Authentication failure type(s) (dmarc, spf, dkim)
dmarc.reported_domain: Domain that failed DMARC authentication
dmarc.source_service_name (optional): Enriched service name from reverse DNS mapping
dmarc.source_service_type (optional): Enriched service type (e.g., "Email Provider", "Webmail", "Marketing")

SMTP TLS Report Fields (SMTP_TLS_REPORT events):

smtp_tls.policy_domain: Domain being monitored for TLS policy
smtp_tls.result_type: Type of TLS failure (certificate-expired, validation-failure, etc.)
smtp_tls.failed_session_count: Number of failed sessions
smtp_tls.report_org: Reporting organization name
smtp_tls.report_begin: Report period start timestamp
smtp_tls.report_end: Report period end timestamp

Severity Heuristics

HIGH: DMARC disposition = reject
MEDIUM: DMARC disposition = quarantine with partial SPF/DKIM failures
LOW: DMARC disposition = none or pass

Example Output

Aggregate Report Event

{
  "event_type": "DMARC_AGGREGATE",
  "metadata": {
    "event_timestamp": "2018-06-19T00:00:00+00:00",
    "event_type": "GENERIC_EVENT",
    "product_name": "parsedmarc",
    "vendor_name": "parsedmarc"
  },
  "principal": {
    "ip": ["199.230.200.36"],
    "location": {"country_or_region": "US"}
  },
  "target": {
    "domain": {"name": "example.com"}
  },
  "security_result": [{
    "severity": "LOW",
    "description": "DMARC fail; SPF=pass; DKIM=pass; SPF not aligned; DKIM not aligned; disposition=none",
    "detection_fields": [
      {"key": "dmarc.disposition", "value": "none"},
      {"key": "dmarc.policy", "value": "none"},
      {"key": "dmarc.pass", "value": false},
      {"key": "dmarc.spf_aligned", "value": false},
      {"key": "dmarc.dkim_aligned", "value": false},
      {"key": "dmarc.header_from", "value": "example.com"},
      {"key": "dmarc.envelope_from", "value": "example.com"},
      {"key": "dmarc.report_org", "value": "example.net"},
      {"key": "dmarc.report_id", "value": "b043f0e264cf4ea995e93765242f6dfb"},
      {"key": "dmarc.report_begin", "value": "2018-06-19 00:00:00"},
      {"key": "dmarc.report_end", "value": "2018-06-19 23:59:59"},
      {"key": "dmarc.row_count", "value": 1},
      {"key": "dmarc.spf_result", "value": "pass"},
      {"key": "dmarc.dkim_result", "value": "pass"},
      {"key": "dmarc.source_service_name", "value": "Example Mail Service"},
      {"key": "dmarc.source_service_type", "value": "Email Provider"}
    ]
  }],
  "additional": {
    "fields": [
      {"key": "spf_0_domain", "value": "example.com"},
      {"key": "spf_0_result", "value": "pass"},
      {"key": "dkim_0_domain", "value": "example.com"},
      {"key": "dkim_0_result", "value": "pass"}
    ]
  }
}

Forensic Report Event

{
  "event_type": "DMARC_FORENSIC",
  "metadata": {
    "event_timestamp": "2019-04-30T02:09:00+00:00",
    "event_type": "GENERIC_EVENT",
    "product_name": "parsedmarc",
    "vendor_name": "parsedmarc"
  },
  "principal": {
    "ip": ["10.10.10.10"],
    "location": {"country_or_region": "US"},
    "hostname": "mail.example-sender.com"
  },
  "target": {
    "domain": {"name": "example.com"}
  },
  "security_result": [{
    "severity": "MEDIUM",
    "description": "DMARC forensic report: authentication failure (dmarc)",
    "detection_fields": [
      {"key": "dmarc.auth_failure", "value": "dmarc"},
      {"key": "dmarc.reported_domain", "value": "example.com"},
      {"key": "dmarc.source_service_name", "value": "Example Mail Provider"},
      {"key": "dmarc.source_service_type", "value": "Email Provider"}
    ]
  }],
  "additional": {
    "fields": [
      {"key": "feedback_type", "value": "auth-failure"},
      {"key": "message_id", "value": "<01010101010101010101010101010101@ABAB01MS0016.someserver.loc>"},
      {"key": "authentication_results", "value": "dmarc=fail (p=none; dis=none) header.from=example.com"},
      {"key": "delivery_result", "value": "delivered"}
    ]
  }
}

SMTP TLS Report Event

{
  "event_type": "SMTP_TLS_REPORT",
  "metadata": {
    "event_timestamp": "2016-04-01T00:00:00+00:00",
    "event_type": "GENERIC_EVENT",
    "product_name": "parsedmarc",
    "vendor_name": "parsedmarc"
  },
  "target": {
    "domain": {
      "name": "company-y.example"
    }
  },
  "security_result": [{
    "severity": "LOW",
    "description": "SMTP TLS failure: certificate-expired",
    "detection_fields": [
      {"key": "smtp_tls.policy_domain", "value": "company-y.example"},
      {"key": "smtp_tls.result_type", "value": "certificate-expired"},
      {"key": "smtp_tls.failed_session_count", "value": 100},
      {"key": "smtp_tls.report_org", "value": "Company-X"},
      {"key": "smtp_tls.report_begin", "value": "2016-04-01T00:00:00Z"},
      {"key": "smtp_tls.report_end", "value": "2016-04-01T23:59:59Z"}
    ]
  }],
  "principal": {
    "ip": ["2001:db8:abcd:0012::1"]
  }
}

Parse Error Event

{
  "event_type": "DMARC_PARSE_ERROR",
  "metadata": {
    "event_timestamp": "2026-01-09T16:22:10.933751+00:00",
    "event_type": "GENERIC_EVENT",
    "product_name": "parsedmarc",
    "vendor_name": "parsedmarc"
  },
  "security_result": [{
    "severity": "ERROR",
    "description": "Failed to parse DMARC report: Invalid XML structure"
  }]
}

Google SecOps Searches

Here are some example YARA-L rules you can use in Google SecOps to hunt for DMARC issues:

Find all DMARC aggregate report failures

rule dmarc_aggregate_failures {
  meta:
    author = "parsedmarc"
    description = "Detect DMARC authentication failures in aggregate reports"
    
  events:
    $e.metadata.product_name = "parsedmarc"
    $e.event_type = "DMARC_AGGREGATE"
    $e.security_result.detection_fields.key = "dmarc.pass"
    $e.security_result.detection_fields.value = false
    
  condition:
    $e
}

Find high severity DMARC events (rejected mail)

rule high_severity_dmarc_events {
  meta:
    author = "parsedmarc"
    description = "Detect high severity DMARC aggregate events (rejected mail)"
    
  events:
    $e.metadata.product_name = "parsedmarc"
    $e.event_type = "DMARC_AGGREGATE"
    $e.security_result.severity = "HIGH"
    
  condition:
    $e
}

Find repeated DMARC failures from same source IP

rule repeated_dmarc_failures {
  meta:
    author = "parsedmarc"
    description = "Detect repeated DMARC failures from the same source IP"
    
  events:
    $e.metadata.product_name = "parsedmarc"
    $e.event_type = "DMARC_AGGREGATE"
    $e.security_result.detection_fields.key = "dmarc.pass"
    $e.security_result.detection_fields.value = false
    $e.principal.ip = $source_ip
    
  match:
    $source_ip over 1h
    
  condition:
    #e > 5
}

Find DMARC forensic reports with authentication failures

rule dmarc_forensic_failures {
  meta:
    author = "parsedmarc"
    description = "Detect DMARC forensic reports with authentication failures"
    
  events:
    $e.metadata.product_name = "parsedmarc"
    $e.event_type = "DMARC_FORENSIC"
    $e.security_result.detection_fields.key = "dmarc.auth_failure"
    
  condition:
    $e
}

Find DMARC failures from specific mail service types

rule dmarc_failures_by_service_type {
  meta:
    author = "parsedmarc"
    description = "Detect DMARC failures from specific mail service types"
    
  events:
    $e.metadata.product_name = "parsedmarc"
    $e.event_type = "DMARC_AGGREGATE"
    $e.security_result.detection_fields.key = "dmarc.pass"
    $e.security_result.detection_fields.value = false
    $e.security_result.detection_fields.key = "dmarc.source_service_type"
    $e.security_result.detection_fields.value = "Email Provider"
    
  condition:
    $e
}

Find SMTP TLS failures

rule smtp_tls_failures {
  meta:
    author = "parsedmarc"
    description = "Detect SMTP TLS failures"
    
  events:
    $e.metadata.product_name = "parsedmarc"
    $e.event_type = "SMTP_TLS_REPORT"
    
  condition:
    $e
}

Privacy Considerations

By default, forensic report message bodies are excluded from the output to protect privacy. If you need to include message samples for investigation:

Set include_ruf_payload = True in your configuration
Adjust ruf_payload_max_bytes to limit the amount of data included (default: 4096 bytes)
Message samples will be truncated if they exceed the configured maximum

Note: Be aware of data privacy regulations (GDPR, CCPA, etc.) when including message payloads in security telemetry.

Usage

The Google SecOps output works with all parsedmarc input methods, including file processing and mailbox monitoring.

Primary Method: Direct API Ingestion

With Chronicle Ingestion API configured, events are sent directly to Chronicle:

# Process files - events are sent to Chronicle API automatically
parsedmarc -c config.ini samples/aggregate/*.xml

# Monitor mailbox - events are sent to Chronicle API in real-time
parsedmarc -c config.ini

No additional log shippers or pipelines are needed. The Google SecOps client handles authentication and batching automatically.

Alternative Method: stdout Output with Log Shipper

If using use_stdout = True in your configuration, output DMARC reports to an external log shipper:

Processing Files

# Output to stdout
parsedmarc -c config.ini samples/aggregate/*.xml > dmarc_events.ndjson

# Stream to file
parsedmarc -c config.ini samples/aggregate/*.xml >> /var/log/dmarc/events.ndjson

# Pipe to log shipper (e.g., Fluentd, Logstash, Chronicle forwarder)
parsedmarc -c config.ini samples/aggregate/*.xml | fluentd

Monitoring Mailboxes

The Google SecOps output automatically works when monitoring mailboxes via IMAP, Microsoft Graph, or Gmail API. Configure your mailbox connection and enable watching:

[general]
save_aggregate = True
save_forensic = True

[mailbox]
watch = True
delete = False
batch_size = 10

[imap]
host = imap.example.com
user = dmarc@example.com
password = yourpassword

[google_secops]
# Use stdout mode for log shipper integration
use_stdout = True
include_ruf_payload = False
static_observer_name = mailbox-monitor
static_environment = prod

When watching a mailbox with stdout mode, parsedmarc continuously outputs UDM events as new reports arrive:

parsedmarc -c config.ini | fluentd

The output is in newline-delimited JSON format, with one UDM event per line, ready for collection by your log shipper.

16 KiB Raw Blame History