diff --git a/google_secops_parser/README.md b/google_secops_parser/README.md new file mode 100644 index 0000000..846c9ae --- /dev/null +++ b/google_secops_parser/README.md @@ -0,0 +1,191 @@ +# Google SecOps (Chronicle) parser for parsedmarc + +A [Google Security Operations](https://cloud.google.com/security/products/security-operations) +custom parser (configuration-based normalizer / CBN) that maps the JSON events +parsedmarc emits through its built-in `[syslog]` output to the Unified Data +Model (UDM). + +This is a **SecOps-side parser only** — it requires no changes to parsedmarc. +parsedmarc already ships structured JSON over syslog; the DMARC→UDM mapping +lives here so that a downstream UDM schema change is a parser edit rather than a +parsedmarc release. + +## Status + +> [!IMPORTANT] +> This parser was written strictly against the official Google documentation +> linked at the bottom of this file, but it has **not yet been validated against +> a live SecOps tenant**. Before using it in production, paste it into the SecOps +> parser-validation tool and confirm each sample event below parses and that the +> assertions in [Caveats](#caveats) hold. Please report fixes back to the +> [parsedmarc](https://github.com/domainaware/parsedmarc) project. + +## Supported report types + +parsedmarc emits three flat JSON shapes (one object per syslog line). The parser +detects them by a field unique to each and maps them as follows: + +| parsedmarc report | Detected by | UDM `metadata.event_type` | +|---|---|---| +| DMARC aggregate | `adkim` | `EMAIL_TRANSACTION` | +| DMARC failure | `feedback_type` | `EMAIL_TRANSACTION` | +| SMTP TLS (RFC 8460) | `policy_type` | `GENERIC_EVENT` | + +`EMAIL_TRANSACTION` and `GENERIC_EVENT` are both valid `metadata.event_type` +values. Note that **`GENERIC_EVENT` events only appear in raw-log and UDM +search**, not in the curated SecOps views — that is the documented behaviour for +generic events, and it is why SMTP TLS reports surface differently from the two +DMARC types. + +## Caveats + +1. **Unvalidated** — see [Status](#status). +2. **Boolean coercion** — parsedmarc emits `dmarc_aligned`, `spf_aligned`, + `dkim_aligned`, `testing`, and `normalized_timespan` as JSON booleans. The + parser assumes the `json{}` filter exposes them as the strings `"true"` / + `"false"` (the CBN convention) and compares them as such. The security- + relevant consequence to confirm in the validation tool: a DMARC-fail record + (`dmarc_aligned=false`) must receive `security_result.category = + AUTH_VIOLATION`. +3. **Aggregate count** — a DMARC aggregate record summarizes `count` messages + from one source IP, not a single message. Each record becomes one + `EMAIL_TRANSACTION` with `count` carried in `additional.fields`. There is no + first-class per-message expansion (fanning out `count` copies would + misrepresent the data). +4. **Address format** — aggregate reports only carry the From *domain*, so + `network.email.from` holds a bare domain for aggregate events but a full + address for failure events. UDM email-address fields are expected to be + `local-mailbox@domain`; downstream consumers should account for the + aggregate-domain case. + +## UDM field mappings + +All UDM field names below are from the +[UDM field list](https://cloud.google.com/chronicle/docs/reference/udm-field-list) +and [SecurityResult reference](https://cloud.google.com/chronicle/docs/reference/rest/v1alpha/SecurityResult). + +### DMARC aggregate → `EMAIL_TRANSACTION` + +| parsedmarc field | UDM field | +|---|---| +| `begin_date` | `metadata.event_timestamp` (via `date{}`) | +| `report_id` | `metadata.product_log_id` | +| `source_ip_address` | `principal.ip` | +| `source_reverse_dns` | `principal.hostname` | +| `source_country` | `principal.location.country_or_region` | +| `domain` | `target.hostname` | +| `header_from` | `network.email.from` (domain; see caveat 4) | +| `disposition` | `security_result.action` (`none`→`ALLOW`, `quarantine`→`QUARANTINE`, `reject`→`BLOCK`) | +| `dmarc_aligned=false` | `security_result.category = AUTH_VIOLATION` | +| `org_name`, `org_email`, `count`, `p`, `sp`, `np`, `pct`, `fo`, `adkim`, `aspf`, `testing`, `discovery_method`, `normalized_timespan`, `*_aligned`, `dkim_*`, `spf_*`, `policy_override_*`, `source_base_domain`, `source_name`, `source_type`, `source_asn`, `source_as_name`, `source_as_domain`, `envelope_from`, `envelope_to` | `additional.fields` | + +### DMARC failure → `EMAIL_TRANSACTION` + +| parsedmarc field | UDM field | +|---|---| +| `arrival_date_utc` | `metadata.event_timestamp` (via `date{}`) | +| `message_id` | `metadata.product_log_id`, `network.email.mail_id` | +| `source_ip_address` | `principal.ip` | +| `source_reverse_dns` | `principal.hostname` | +| `source_country` | `principal.location.country_or_region` | +| `reported_domain` | `target.hostname` | +| `original_mail_from` | `network.email.from` | +| `original_rcpt_to` | `network.email.to` | +| `subject` | `network.email.subject` | +| `auth_failure` | `security_result.category = AUTH_VIOLATION` + description | +| `delivery_result` | `security_result.action` (`reject`→`BLOCK`, `quarantine`→`QUARANTINE`, `delivered`→`ALLOW`) | +| `feedback_type`, `authentication_results`, `authentication_mechanisms`, `user_agent`, `dkim_domain`, `arrival_date` | `additional.fields` | + +### SMTP TLS → `GENERIC_EVENT` + +| parsedmarc field | UDM field | +|---|---| +| `begin_date` | `metadata.event_timestamp` (ISO 8601, via `date{}`) | +| `report_id` | `metadata.product_log_id` | +| `policy_domain` | `target.hostname` (always present → the noun) | +| `receiving_ip` | `target.ip` (failure rows only) | +| `sending_mta_ip` | `principal.ip` (failure rows only) | +| `result_type` | `security_result` (`action=FAIL`, `category=POLICY_VIOLATION`) | +| `organization_name`, `policy_type`, `policy_strings`, `mx_host_patterns`, `successful_session_count`, `failed_session_count`, `failure_reason_code`, `receiving_mx_hostname`, `receiving_mx_helo`, `additional_info_uri` | `additional.fields` | + +> parsedmarc emits SMTP TLS reports as separate rows: one **success** row per +> policy (counts, no MTA IPs) and one **failure** row per failure detail (which +> may also lack MTA IPs, e.g. `sts-policy-fetch-error`). The noun therefore comes +> from `policy_domain`, which is present on every row. + +## Installation + +### 1. Configure parsedmarc syslog output + +```ini +[syslog] +server = your-collector.example.com +port = 514 +``` + +parsedmarc writes each report record as a single-line JSON message. + +### 2. Collect the syslog stream into SecOps + +Syslog is ingested by a **collector**, not a Feed. Run the +[Bindplane agent](https://cloud.google.com/chronicle/docs/install/install-forwarder) +(Google's recommended on-premises collector; the legacy Chronicle forwarder is +end-of-life) with a **Syslog** collector pointed at the port above, and assign it +a custom log type (for example `PARSEDMARC`). + +### 3. Install this parser for that log type + +Associate `parsedmarc.conf` with the custom log type via the SecOps parser +management UI or API (see +[Manage parsers](https://cloud.google.com/chronicle/docs/event-processing/manage-parser-updates)). +Validate against the sample events below before activating. + +## Sample events for validation + +These are **real** single-line outputs from parsedmarc's `[syslog]` serializers +(generated from the project's sample reports). Use them in the parser-validation +tool. A live syslog line will also carry a `` prefix; the parser strips any +leading framing before the first `{`. + +### Aggregate — DMARC fail (`dmarc_aligned=false`) + +```json +{"xml_schema": "draft", "org_name": "accurateplastics.com", "org_email": "administrator@accurateplastics.com", "org_extra_contact_info": "", "report_id": "example.com:1538463741", "begin_date": "2018-10-01 17:07:12", "end_date": "2018-10-01 17:07:12", "normalized_timespan": false, "errors": "", "domain": "example.com", "adkim": "r", "aspf": "r", "p": "none", "sp": "reject", "np": "", "pct": "100", "fo": "", "testing": "", "discovery_method": "", "source_ip_address": "12.20.127.122", "source_country": "US", "source_reverse_dns": "", "source_base_domain": "", "source_name": "AT&T", "source_type": "ISP", "source_asn": 7018, "source_as_name": "AT&T Enterprises, LLC", "source_as_domain": "att.com", "count": 1, "spf_aligned": false, "dkim_aligned": false, "dmarc_aligned": false, "disposition": "none", "policy_override_reasons": "", "policy_override_comments": "", "envelope_from": "", "header_from": "example.com", "envelope_to": "", "dkim_domains": "", "dkim_selectors": "", "dkim_results": "", "spf_domains": "", "spf_scopes": "", "spf_results": ""} +``` + +### Aggregate — DMARC pass (`dmarc_aligned=true`) + +```json +{"xml_schema": "1.0", "org_name": "example.org", "org_email": "noreply-dmarc-support@example.org", "org_extra_contact_info": "https://support.example.org/dmarc", "report_id": "20240125141224705995", "begin_date": "2024-01-25 05:12:24", "end_date": "2024-01-25 12:28:53", "normalized_timespan": false, "errors": "", "domain": "example.com", "adkim": "r", "aspf": "r", "p": "quarantine", "sp": "quarantine", "np": "", "pct": "100", "fo": "1", "testing": "", "discovery_method": "", "source_ip_address": "198.51.100.123", "source_country": "", "source_reverse_dns": "", "source_base_domain": "", "source_name": "", "source_type": "", "source_asn": "", "source_as_name": "", "source_as_domain": "", "count": 2, "spf_aligned": false, "dkim_aligned": true, "dmarc_aligned": true, "disposition": "none", "policy_override_reasons": "none", "policy_override_comments": "none", "envelope_from": "example.edu", "header_from": "example.com", "envelope_to": "example.net", "dkim_domains": "example.com", "dkim_selectors": "example", "dkim_results": "pass", "spf_domains": "example.edu", "spf_scopes": "mfrom", "spf_results": "pass"} +``` + +### Failure + +```json +{"feedback_type": "auth-failure", "user_agent": "Lua/1.0", "version": "1.0", "original_mail_from": "sharepoint@domain.de", "original_rcpt_to": "peter.pan@domain.de", "arrival_date": "Mon, 01 Oct 2018 11:20:27 +0200", "message_id": "<38.E7.30937.BD6E1BB5@ mailrelay.de>", "authentication_results": "dmarc=fail (p=none, dis=none) header.from=domain.de", "delivery_result": "policy", "auth_failure": "dmarc", "reported_domain": "domain.de", "arrival_date_utc": "2018-10-01 09:20:27", "authentication_mechanisms": "", "original_envelope_id": null, "dkim_domain": null, "sample_headers_only": false, "source_ip_address": "10.10.10.10", "source_reverse_dns": null, "source_base_domain": null, "source_name": null, "source_type": null, "source_asn": null, "source_as_name": null, "source_as_domain": null, "source_country": null, "subject": "Subject"} +``` + +### SMTP TLS — success row (counts only) + +```json +{"organization_name": "Synametrics Technologies, Inc.", "begin_date": "2025-12-07T19:00:00Z", "end_date": "2025-12-08T18:59:59Z", "report_id": "1765256572301+dmarc-reports.dengage.com", "policy_strings": "version: STSv1|mode: enforce|mx: mta1.inboxsys.net|mx: mta2.inboxsys.net|max_age: 86400", "policy_domain": "dmarc-reports.dengage.com", "policy_type": "sts", "successful_session_count": 2, "failed_session_count": 0} +``` + +### SMTP TLS — failure-detail row + +```json +{"organization_name": "Mail.ru", "begin_date": "2024-02-22T00:00:00Z", "end_date": "2024-02-23T00:00:00Z", "report_id": "b28254de-7b2e-be36-bb5c-4c3b92da8b25@mail.ru", "result_type": "sts-policy-fetch-error", "failed_session_count": 1, "failure_reason_code": "bad https response code: 404"} +``` + +## Official references + +- [Overview of the UDM](https://cloud.google.com/chronicle/docs/event-processing/udm-overview) +- [Overview of log parsing](https://cloud.google.com/chronicle/docs/event-processing/parsing-overview) +- [Parser syntax reference](https://cloud.google.com/chronicle/docs/reference/parser-syntax) +- [UDM field list](https://cloud.google.com/chronicle/docs/reference/udm-field-list) +- [SecurityResult reference](https://cloud.google.com/chronicle/docs/reference/rest/v1alpha/SecurityResult) +- [Feed management](https://cloud.google.com/chronicle/docs/administration/feed-management-overview) + +## License + +Distributed under the same license as [parsedmarc](https://github.com/domainaware/parsedmarc). diff --git a/google_secops_parser/parsedmarc.conf b/google_secops_parser/parsedmarc.conf new file mode 100644 index 0000000..89778b2 --- /dev/null +++ b/google_secops_parser/parsedmarc.conf @@ -0,0 +1,773 @@ +filter { + + # =========================================================================== + # parsedmarc -> Google Security Operations (Chronicle) UDM parser + # + # Consumes the single-line JSON events that parsedmarc emits through its + # built-in [syslog] output and maps them to the Unified Data Model (UDM). + # No changes to parsedmarc are required: this is a SecOps-side custom parser + # (configuration-based normalizer / CBN). + # + # parsedmarc emits three flat JSON shapes, one object per syslog line, via the + # CSV-row serializers (parsed_aggregate/failure/smtp_tls_reports_to_csv_rows): + # * DMARC aggregate report record -> detected by "adkim" + # * DMARC failure report record -> detected by "feedback_type" + # * SMTP TLS report record -> detected by "policy_type" + # + # --------------------------------------------------------------------------- + # UDM facts this parser relies on (all from official Google documentation): + # + # * Output object & finalize: build fields under + # event.idm.read_only_udm.* and emit with + # mutate { merge => { "@output" => "event" } } + # https://cloud.google.com/chronicle/docs/event-processing/parser-syntax + # * Every event MUST set metadata.event_type (predefined enum) and + # metadata.event_timestamp; parsing fails if a required field is missing. + # A date{} filter with no target sets metadata.event_timestamp; if no + # date{} runs, event_timestamp defaults to ingest time. + # https://cloud.google.com/chronicle/docs/event-processing/parsing-overview + # * Every event MUST contain at least one noun + # (principal/target/src/intermediary/observer). principal must carry a + # machine or user detail and must NOT carry email. + # https://cloud.google.com/chronicle/docs/event-processing/udm-overview + # * EMAIL_TRANSACTION and GENERIC_EVENT are valid metadata.event_type values. + # GENERIC_EVENT only surfaces in raw-log / UDM search, not curated views. + # https://cloud.google.com/chronicle/docs/investigation/udm-search + # * security_result.action enum: UNKNOWN_ACTION, ALLOW, BLOCK, + # ALLOW_WITH_MODIFICATION, QUARANTINE, FAIL, CHALLENGE. + # security_result.category enum includes AUTH_VIOLATION + # ("Authentication failed"), POLICY_VIOLATION, MAIL_SPOOFING. There is no + # "passed" category, so we set a category only on failures. + # https://cloud.google.com/chronicle/docs/reference/rest/v1alpha/SecurityResult + # * network.email fields: from, reply_to, to, cc, bcc, mail_id, subject. + # (There is no network.email.mail_from.) + # https://cloud.google.com/chronicle/docs/reference/udm-field-list + # + # --------------------------------------------------------------------------- + # Caveats (read before trusting output): + # + # 1. UNVALIDATED. This parser was written to the docs above but has not been + # run through the SecOps parser-validation tool against a live tenant. + # Validate with the sample events in README.md before production use. + # 2. BOOLEAN COERCION. parsedmarc emits *_aligned / testing / + # normalized_timespan / sample_headers_only as JSON booleans. This parser + # assumes the json{} filter exposes them as the strings "true"/"false" + # (the CBN convention) and compares them as such. Confirm in the + # validation tool that DMARC-fail records (dmarc_aligned=false) receive + # security_result.category = AUTH_VIOLATION. + # 3. AGGREGATE COUNT. A DMARC aggregate record summarizes "count" messages + # from one source IP, not a single message. Each becomes one + # EMAIL_TRANSACTION with "count" carried in additional.fields; there is no + # first-class per-message expansion (fanning out would be wrong). + # 4. ADDRESS FORMAT. Aggregate header_from is a bare domain (DMARC only + # reports the From domain). It is still mapped to network.email.from as + # the best-available sender identity; downstream consumers should expect a + # domain there for aggregate events, an address for failure events. + # =========================================================================== + + # --------------------------------------------------------------------------- + # 1. Extract the JSON object from the (possibly syslog-framed) raw line. + # Python's SysLogHandler prepends a "" priority (and a forwarder may + # add a timestamp/host/tag), so the JSON is not necessarily at column 0. + # Grab everything from the first "{" to the last "}". + # --------------------------------------------------------------------------- + mutate { + replace => { + "report_type" => "" + "event_type" => "" + } + } + + grok { + match => { + "message" => ["^.*?(?P\\{.*\\})\\s*$"] + } + on_error => "no_json_payload" + } + if [no_json_payload] { + drop {} + } + + json { + source => "payload" + on_error => "not_json" + } + if [not_json] { + drop {} + } + + # --------------------------------------------------------------------------- + # 2. Detect the report type from a field unique to each shape. + # feedback_type / policy_type / adkim are always present and non-empty for + # their respective report types, so existence checks are reliable. + # --------------------------------------------------------------------------- + if [feedback_type] { + mutate { replace => { "report_type" => "failure" } } + } else if [policy_type] { + mutate { replace => { "report_type" => "smtp_tls" } } + } else if [adkim] { + mutate { replace => { "report_type" => "aggregate" } } + } + + # Not a parsedmarc record we recognize: drop rather than emit an invalid event. + if [report_type] == "" { + drop {} + } + + # =========================================================================== + # DMARC AGGREGATE -> EMAIL_TRANSACTION + # =========================================================================== + if [report_type] == "aggregate" { + + mutate { replace => { "event_type" => "EMAIL_TRANSACTION" } } + + if [report_id] { + mutate { + replace => { + "event.idm.read_only_udm.metadata.product_log_id" => "%{report_id}" + } + } + } + + # -- event timestamp: start of the record's reporting interval (UTC). -- + if [begin_date] { + date { + match => ["begin_date", "yyyy-MM-dd HH:mm:ss"] + timezone => "UTC" + on_error => "agg_date_error" + } + } + + # -- principal: the sending source (machine details only). -- + if [source_ip_address] { + mutate { + merge => { "event.idm.read_only_udm.principal.ip" => "source_ip_address" } + on_error => "agg_src_ip_error" + } + } + if [source_reverse_dns] { + mutate { + replace => { + "event.idm.read_only_udm.principal.hostname" => "%{source_reverse_dns}" + } + } + } + if [source_country] { + mutate { + replace => { + "event.idm.read_only_udm.principal.location.country_or_region" => "%{source_country}" + } + } + } + + # -- target: the domain the report is about. -- + if [domain] { + mutate { + replace => { + "event.idm.read_only_udm.target.hostname" => "%{domain}" + } + } + } + + # -- email: aggregate only carries the From domain (see caveat 4). -- + if [header_from] { + mutate { + replace => { + "event.idm.read_only_udm.network.email.from" => "%{header_from}" + } + } + } + + # -- security_result: disposition -> action; failed alignment -> category. -- + mutate { replace => { "sr.summary" => "DMARC aggregate report" } } + + if [disposition] == "reject" { + mutate { replace => { "sr.action" => "BLOCK" } } + } else if [disposition] == "quarantine" { + mutate { replace => { "sr.action" => "QUARANTINE" } } + } else if [disposition] == "none" { + mutate { replace => { "sr.action" => "ALLOW" } } + } else { + mutate { replace => { "sr.action" => "UNKNOWN_ACTION" } } + } + + # DMARC failed authentication alignment -> AUTH_VIOLATION (see caveat 2). + if [dmarc_aligned] == "false" { + mutate { replace => { "sr.category" => "AUTH_VIOLATION" } } + } + + mutate { + replace => { + "sr.description" => "dmarc_aligned=%{dmarc_aligned} spf_aligned=%{spf_aligned} dkim_aligned=%{dkim_aligned} disposition=%{disposition}" + } + on_error => "agg_sr_desc_error" + } + + mutate { + merge => { "event.idm.read_only_udm.security_result" => "sr" } + } + + # -- additional.fields: DMARC dimensions a dashboard would filter on. -- + if [org_name] { + mutate { + replace => { "f_org_name.key" => "org_name" "f_org_name.value.string_value" => "%{org_name}" } + merge => { "event.idm.read_only_udm.additional.fields" => "f_org_name" } + } + } + if [org_email] { + mutate { + replace => { "f_org_email.key" => "org_email" "f_org_email.value.string_value" => "%{org_email}" } + merge => { "event.idm.read_only_udm.additional.fields" => "f_org_email" } + } + } + if [begin_date] { + mutate { + replace => { "f_begin.key" => "begin_date" "f_begin.value.string_value" => "%{begin_date}" } + merge => { "event.idm.read_only_udm.additional.fields" => "f_begin" } + } + } + if [end_date] { + mutate { + replace => { "f_end.key" => "end_date" "f_end.value.string_value" => "%{end_date}" } + merge => { "event.idm.read_only_udm.additional.fields" => "f_end" } + } + } + if [count] { + mutate { + replace => { "f_count.key" => "count" "f_count.value.string_value" => "%{count}" } + merge => { "event.idm.read_only_udm.additional.fields" => "f_count" } + } + } + if [p] { + mutate { + replace => { "f_p.key" => "dmarc_policy" "f_p.value.string_value" => "%{p}" } + merge => { "event.idm.read_only_udm.additional.fields" => "f_p" } + } + } + if [sp] { + mutate { + replace => { "f_sp.key" => "dmarc_subdomain_policy" "f_sp.value.string_value" => "%{sp}" } + merge => { "event.idm.read_only_udm.additional.fields" => "f_sp" } + } + } + if [np] { + mutate { + replace => { "f_np.key" => "dmarc_np_policy" "f_np.value.string_value" => "%{np}" } + merge => { "event.idm.read_only_udm.additional.fields" => "f_np" } + } + } + if [pct] { + mutate { + replace => { "f_pct.key" => "dmarc_pct" "f_pct.value.string_value" => "%{pct}" } + merge => { "event.idm.read_only_udm.additional.fields" => "f_pct" } + } + } + if [fo] { + mutate { + replace => { "f_fo.key" => "dmarc_fo" "f_fo.value.string_value" => "%{fo}" } + merge => { "event.idm.read_only_udm.additional.fields" => "f_fo" } + } + } + if [adkim] { + mutate { + replace => { "f_adkim.key" => "dkim_alignment_mode" "f_adkim.value.string_value" => "%{adkim}" } + merge => { "event.idm.read_only_udm.additional.fields" => "f_adkim" } + } + } + if [aspf] { + mutate { + replace => { "f_aspf.key" => "spf_alignment_mode" "f_aspf.value.string_value" => "%{aspf}" } + merge => { "event.idm.read_only_udm.additional.fields" => "f_aspf" } + } + } + if [testing] == "true" or [testing] == "false" { + mutate { + replace => { "f_testing.key" => "dmarc_testing" "f_testing.value.string_value" => "%{testing}" } + merge => { "event.idm.read_only_udm.additional.fields" => "f_testing" } + } + } + if [discovery_method] { + mutate { + replace => { "f_disc.key" => "discovery_method" "f_disc.value.string_value" => "%{discovery_method}" } + merge => { "event.idm.read_only_udm.additional.fields" => "f_disc" } + } + } + if [normalized_timespan] == "true" or [normalized_timespan] == "false" { + mutate { + replace => { "f_norm.key" => "normalized_timespan" "f_norm.value.string_value" => "%{normalized_timespan}" } + merge => { "event.idm.read_only_udm.additional.fields" => "f_norm" } + } + } + + # Alignment outcomes (security-relevant; emit for both true and false). + if [dmarc_aligned] == "true" or [dmarc_aligned] == "false" { + mutate { + replace => { "f_dmarc.key" => "dmarc_aligned" "f_dmarc.value.string_value" => "%{dmarc_aligned}" } + merge => { "event.idm.read_only_udm.additional.fields" => "f_dmarc" } + } + } + if [spf_aligned] == "true" or [spf_aligned] == "false" { + mutate { + replace => { "f_spfa.key" => "spf_aligned" "f_spfa.value.string_value" => "%{spf_aligned}" } + merge => { "event.idm.read_only_udm.additional.fields" => "f_spfa" } + } + } + if [dkim_aligned] == "true" or [dkim_aligned] == "false" { + mutate { + replace => { "f_dkima.key" => "dkim_aligned" "f_dkima.value.string_value" => "%{dkim_aligned}" } + merge => { "event.idm.read_only_udm.additional.fields" => "f_dkima" } + } + } + if [disposition] { + mutate { + replace => { "f_disp.key" => "disposition" "f_disp.value.string_value" => "%{disposition}" } + merge => { "event.idm.read_only_udm.additional.fields" => "f_disp" } + } + } + + # DKIM / SPF auth detail (comma-joined by parsedmarc). + if [dkim_domains] { + mutate { + replace => { "f_dkd.key" => "dkim_domains" "f_dkd.value.string_value" => "%{dkim_domains}" } + merge => { "event.idm.read_only_udm.additional.fields" => "f_dkd" } + } + } + if [dkim_selectors] { + mutate { + replace => { "f_dks.key" => "dkim_selectors" "f_dks.value.string_value" => "%{dkim_selectors}" } + merge => { "event.idm.read_only_udm.additional.fields" => "f_dks" } + } + } + if [dkim_results] { + mutate { + replace => { "f_dkr.key" => "dkim_results" "f_dkr.value.string_value" => "%{dkim_results}" } + merge => { "event.idm.read_only_udm.additional.fields" => "f_dkr" } + } + } + if [spf_domains] { + mutate { + replace => { "f_spd.key" => "spf_domains" "f_spd.value.string_value" => "%{spf_domains}" } + merge => { "event.idm.read_only_udm.additional.fields" => "f_spd" } + } + } + if [spf_scopes] { + mutate { + replace => { "f_sps.key" => "spf_scopes" "f_sps.value.string_value" => "%{spf_scopes}" } + merge => { "event.idm.read_only_udm.additional.fields" => "f_sps" } + } + } + if [spf_results] { + mutate { + replace => { "f_spr.key" => "spf_results" "f_spr.value.string_value" => "%{spf_results}" } + merge => { "event.idm.read_only_udm.additional.fields" => "f_spr" } + } + } + + # Policy overrides (parsedmarc writes the literal "none" when there are none). + if [policy_override_reasons] and [policy_override_reasons] != "none" { + mutate { + replace => { "f_por.key" => "policy_override_reasons" "f_por.value.string_value" => "%{policy_override_reasons}" } + merge => { "event.idm.read_only_udm.additional.fields" => "f_por" } + } + } + if [policy_override_comments] and [policy_override_comments] != "none" { + mutate { + replace => { "f_poc.key" => "policy_override_comments" "f_poc.value.string_value" => "%{policy_override_comments}" } + merge => { "event.idm.read_only_udm.additional.fields" => "f_poc" } + } + } + + # Source enrichment from parsedmarc's reverse-DNS / MMDB maps. + if [source_base_domain] { + mutate { + replace => { "f_sbd.key" => "source_base_domain" "f_sbd.value.string_value" => "%{source_base_domain}" } + merge => { "event.idm.read_only_udm.additional.fields" => "f_sbd" } + } + } + if [source_name] { + mutate { + replace => { "f_sn.key" => "source_name" "f_sn.value.string_value" => "%{source_name}" } + merge => { "event.idm.read_only_udm.additional.fields" => "f_sn" } + } + } + if [source_type] { + mutate { + replace => { "f_st.key" => "source_type" "f_st.value.string_value" => "%{source_type}" } + merge => { "event.idm.read_only_udm.additional.fields" => "f_st" } + } + } + if [source_asn] { + mutate { + replace => { "f_asn.key" => "source_asn" "f_asn.value.string_value" => "%{source_asn}" } + merge => { "event.idm.read_only_udm.additional.fields" => "f_asn" } + } + } + if [source_as_name] { + mutate { + replace => { "f_asnm.key" => "source_as_name" "f_asnm.value.string_value" => "%{source_as_name}" } + merge => { "event.idm.read_only_udm.additional.fields" => "f_asnm" } + } + } + if [source_as_domain] { + mutate { + replace => { "f_asd.key" => "source_as_domain" "f_asd.value.string_value" => "%{source_as_domain}" } + merge => { "event.idm.read_only_udm.additional.fields" => "f_asd" } + } + } + + # Envelope identifiers (usually empty in aggregate reports). + if [envelope_from] { + mutate { + replace => { "f_ef.key" => "envelope_from" "f_ef.value.string_value" => "%{envelope_from}" } + merge => { "event.idm.read_only_udm.additional.fields" => "f_ef" } + } + } + if [envelope_to] { + mutate { + replace => { "f_et.key" => "envelope_to" "f_et.value.string_value" => "%{envelope_to}" } + merge => { "event.idm.read_only_udm.additional.fields" => "f_et" } + } + } + } + + # =========================================================================== + # DMARC FAILURE -> EMAIL_TRANSACTION + # =========================================================================== + if [report_type] == "failure" { + + mutate { replace => { "event_type" => "EMAIL_TRANSACTION" } } + + if [message_id] { + mutate { + replace => { + "event.idm.read_only_udm.metadata.product_log_id" => "%{message_id}" + } + } + } + + # -- event timestamp: message arrival time (UTC). -- + if [arrival_date_utc] { + date { + match => ["arrival_date_utc", "yyyy-MM-dd HH:mm:ss"] + timezone => "UTC" + on_error => "fail_date_error" + } + } + + # -- principal: the sending source. -- + if [source_ip_address] { + mutate { + merge => { "event.idm.read_only_udm.principal.ip" => "source_ip_address" } + on_error => "fail_src_ip_error" + } + } + if [source_reverse_dns] { + mutate { + replace => { + "event.idm.read_only_udm.principal.hostname" => "%{source_reverse_dns}" + } + } + } + if [source_country] { + mutate { + replace => { + "event.idm.read_only_udm.principal.location.country_or_region" => "%{source_country}" + } + } + } + + # -- target: the reported (claimed) domain. -- + if [reported_domain] { + mutate { + replace => { + "event.idm.read_only_udm.target.hostname" => "%{reported_domain}" + } + } + } + + # -- email: failure reports carry full addresses + subject. -- + if [original_mail_from] { + mutate { + replace => { + "event.idm.read_only_udm.network.email.from" => "%{original_mail_from}" + } + } + } + if [original_rcpt_to] { + mutate { + replace => { + "event.idm.read_only_udm.network.email.to" => "%{original_rcpt_to}" + } + on_error => "fail_rcpt_error" + } + } + if [subject] { + mutate { + replace => { + "event.idm.read_only_udm.network.email.subject" => "%{subject}" + } + } + } + if [message_id] { + mutate { + replace => { + "event.idm.read_only_udm.network.email.mail_id" => "%{message_id}" + } + } + } + + # -- security_result: a failure report is an authentication failure. -- + mutate { + replace => { + "srf.summary" => "DMARC failure report" + "srf.category" => "AUTH_VIOLATION" + } + } + if [delivery_result] == "reject" { + mutate { replace => { "srf.action" => "BLOCK" } } + } else if [delivery_result] == "quarantine" { + mutate { replace => { "srf.action" => "QUARANTINE" } } + } else if [delivery_result] == "delivered" { + mutate { replace => { "srf.action" => "ALLOW" } } + } else { + mutate { replace => { "srf.action" => "UNKNOWN_ACTION" } } + } + if [auth_failure] { + mutate { + replace => { + "srf.description" => "auth_failure=%{auth_failure} delivery_result=%{delivery_result}" + } + on_error => "fail_sr_desc_error" + } + } + mutate { + merge => { "event.idm.read_only_udm.security_result" => "srf" } + } + + # -- additional.fields. -- + if [feedback_type] { + mutate { + replace => { "g_fb.key" => "feedback_type" "g_fb.value.string_value" => "%{feedback_type}" } + merge => { "event.idm.read_only_udm.additional.fields" => "g_fb" } + } + } + if [auth_failure] { + mutate { + replace => { "g_af.key" => "auth_failure" "g_af.value.string_value" => "%{auth_failure}" } + merge => { "event.idm.read_only_udm.additional.fields" => "g_af" } + } + } + if [delivery_result] { + mutate { + replace => { "g_dr.key" => "delivery_result" "g_dr.value.string_value" => "%{delivery_result}" } + merge => { "event.idm.read_only_udm.additional.fields" => "g_dr" } + } + } + if [authentication_results] { + mutate { + replace => { "g_ar.key" => "authentication_results" "g_ar.value.string_value" => "%{authentication_results}" } + merge => { "event.idm.read_only_udm.additional.fields" => "g_ar" } + } + } + if [authentication_mechanisms] { + mutate { + replace => { "g_am.key" => "authentication_mechanisms" "g_am.value.string_value" => "%{authentication_mechanisms}" } + merge => { "event.idm.read_only_udm.additional.fields" => "g_am" } + } + } + if [user_agent] { + mutate { + replace => { "g_ua.key" => "user_agent" "g_ua.value.string_value" => "%{user_agent}" } + merge => { "event.idm.read_only_udm.additional.fields" => "g_ua" } + } + } + if [dkim_domain] { + mutate { + replace => { "g_dd.key" => "dkim_domain" "g_dd.value.string_value" => "%{dkim_domain}" } + merge => { "event.idm.read_only_udm.additional.fields" => "g_dd" } + } + } + if [arrival_date] { + mutate { + replace => { "g_ad.key" => "arrival_date" "g_ad.value.string_value" => "%{arrival_date}" } + merge => { "event.idm.read_only_udm.additional.fields" => "g_ad" } + } + } + } + + # =========================================================================== + # SMTP TLS -> GENERIC_EVENT + # No per-message semantics; modeled as a generic event whose noun is the + # reported policy domain (target.hostname), which is always present. Per the + # docs, GENERIC_EVENT only appears in raw-log / UDM search, not curated views. + # =========================================================================== + if [report_type] == "smtp_tls" { + + mutate { replace => { "event_type" => "GENERIC_EVENT" } } + + if [report_id] { + mutate { + replace => { + "event.idm.read_only_udm.metadata.product_log_id" => "%{report_id}" + } + } + } + + # -- event timestamp: start of the report window. SMTP TLS uses ISO 8601 + # with a trailing Z (e.g. 2025-12-07T19:00:00Z), unlike the other types. -- + if [begin_date] { + date { + match => ["begin_date", "yyyy-MM-dd'T'HH:mm:ss'Z'"] + timezone => "UTC" + on_error => "tls_date_error" + } + } + + # -- target: the reported policy domain (always present -> the noun). -- + if [policy_domain] { + mutate { + replace => { + "event.idm.read_only_udm.target.hostname" => "%{policy_domain}" + } + } + } + # receiving MTA IP, when a per-session failure row carries it. + if [receiving_ip] { + mutate { + merge => { "event.idm.read_only_udm.target.ip" => "receiving_ip" } + on_error => "tls_rcv_ip_error" + } + } + # sending MTA IP, when present, is the principal. + if [sending_mta_ip] { + mutate { + merge => { "event.idm.read_only_udm.principal.ip" => "sending_mta_ip" } + on_error => "tls_snd_ip_error" + } + } + + # -- security_result only on failures (result_type present). -- + if [result_type] { + mutate { + replace => { + "srt.summary" => "SMTP TLS report failure" + "srt.category" => "POLICY_VIOLATION" + "srt.action" => "FAIL" + "srt.description" => "result_type=%{result_type}" + } + on_error => "tls_sr_error" + } + mutate { + merge => { "event.idm.read_only_udm.security_result" => "srt" } + } + } + + # -- additional.fields. -- + if [organization_name] { + mutate { + replace => { "t_org.key" => "organization_name" "t_org.value.string_value" => "%{organization_name}" } + merge => { "event.idm.read_only_udm.additional.fields" => "t_org" } + } + } + if [begin_date] { + mutate { + replace => { "t_begin.key" => "begin_date" "t_begin.value.string_value" => "%{begin_date}" } + merge => { "event.idm.read_only_udm.additional.fields" => "t_begin" } + } + } + if [end_date] { + mutate { + replace => { "t_end.key" => "end_date" "t_end.value.string_value" => "%{end_date}" } + merge => { "event.idm.read_only_udm.additional.fields" => "t_end" } + } + } + if [policy_domain] { + mutate { + replace => { "t_pd.key" => "policy_domain" "t_pd.value.string_value" => "%{policy_domain}" } + merge => { "event.idm.read_only_udm.additional.fields" => "t_pd" } + } + } + if [policy_type] { + mutate { + replace => { "t_pt.key" => "policy_type" "t_pt.value.string_value" => "%{policy_type}" } + merge => { "event.idm.read_only_udm.additional.fields" => "t_pt" } + } + } + if [policy_strings] { + mutate { + replace => { "t_ps.key" => "policy_strings" "t_ps.value.string_value" => "%{policy_strings}" } + merge => { "event.idm.read_only_udm.additional.fields" => "t_ps" } + } + } + if [mx_host_patterns] { + mutate { + replace => { "t_mx.key" => "mx_host_patterns" "t_mx.value.string_value" => "%{mx_host_patterns}" } + merge => { "event.idm.read_only_udm.additional.fields" => "t_mx" } + } + } + if [successful_session_count] { + mutate { + replace => { "t_ssc.key" => "successful_session_count" "t_ssc.value.string_value" => "%{successful_session_count}" } + merge => { "event.idm.read_only_udm.additional.fields" => "t_ssc" } + } + } + if [failed_session_count] { + mutate { + replace => { "t_fsc.key" => "failed_session_count" "t_fsc.value.string_value" => "%{failed_session_count}" } + merge => { "event.idm.read_only_udm.additional.fields" => "t_fsc" } + } + } + if [result_type] { + mutate { + replace => { "t_rt.key" => "result_type" "t_rt.value.string_value" => "%{result_type}" } + merge => { "event.idm.read_only_udm.additional.fields" => "t_rt" } + } + } + if [failure_reason_code] { + mutate { + replace => { "t_frc.key" => "failure_reason_code" "t_frc.value.string_value" => "%{failure_reason_code}" } + merge => { "event.idm.read_only_udm.additional.fields" => "t_frc" } + } + } + if [receiving_mx_hostname] { + mutate { + replace => { "t_rmh.key" => "receiving_mx_hostname" "t_rmh.value.string_value" => "%{receiving_mx_hostname}" } + merge => { "event.idm.read_only_udm.additional.fields" => "t_rmh" } + } + } + if [receiving_mx_helo] { + mutate { + replace => { "t_rmhelo.key" => "receiving_mx_helo" "t_rmhelo.value.string_value" => "%{receiving_mx_helo}" } + merge => { "event.idm.read_only_udm.additional.fields" => "t_rmhelo" } + } + } + if [additional_info_uri] { + mutate { + replace => { "t_aiu.key" => "additional_info_uri" "t_aiu.value.string_value" => "%{additional_info_uri}" } + merge => { "event.idm.read_only_udm.additional.fields" => "t_aiu" } + } + } + } + + # =========================================================================== + # 3. Common metadata + finalize. + # =========================================================================== + mutate { + replace => { + "event.idm.read_only_udm.metadata.event_type" => "%{event_type}" + "event.idm.read_only_udm.metadata.vendor_name" => "parsedmarc" + "event.idm.read_only_udm.metadata.product_name" => "parsedmarc" + } + } + if [report_type] { + mutate { + replace => { + "event.idm.read_only_udm.metadata.product_event_type" => "%{report_type}" + } + } + } + + mutate { + merge => { "@output" => "event" } + } +}