parsedmarc/google_secops_parser

Google SecOps (Chronicle) parser for parsedmarc

A Google Security Operations custom parser (configuration-based normalizer / CBN) that maps the JSON events parsedmarc emits through its built-in [syslog] output to the Unified Data Model (UDM).

Prefer an API-based setup? parsedmarc also ships a [gsecops] output that sends the same UDM events directly to the Chronicle API (events.import), with no collector or custom parser to install — see the gsecops section of the parsedmarc usage documentation. Use this parser instead when you want collector-based ingestion with raw-log retention, or when you already run a Bindplane pipeline. Both paths use the same UDM mapping and the same additional field keys, so searches and dashboards port between them.

This is a SecOps-side parser — parsedmarc already ships structured JSON over syslog, and the DMARC→UDM mapping lives here so that a downstream UDM schema change is a parser edit rather than a parsedmarc release. One paired library fix ships alongside it: SMTP TLS failure-detail rows carry policy_domain / policy_type as of the parsedmarc release that includes this parser. Older parsedmarc versions omit those keys on failure rows; the parser still detects such rows via result_type, but a row that carries neither an MTA IP nor an MX hostname (e.g. sts-policy-fetch-error) then has no UDM noun and will not produce a valid event.

New to SecOps parsers? SecOps ingests a log source by running a parser that turns each raw log line into a Unified Data Model (UDM) event. These parsers are written in a Logstash-style configuration language Google calls a configuration-based normalizer (CBN) — the parsedmarc.conf in this directory is one. You attach it to a custom log type, and SecOps then runs it on every parsedmarc syslog line. Already fluent in CBN? Skip to Installation.

Status

Important

This parser was written strictly against the official Google documentation linked at the bottom of this file, but it has not yet been validated against a live SecOps tenant. Before using it in production, paste it into the SecOps parser-validation tool and confirm each sample event below parses and that the assertions in Caveats hold. Please report fixes back to the parsedmarc project.

Supported report types

parsedmarc emits three flat JSON shapes (one object per syslog line). The parser detects them by a field unique to each and maps them as follows:

parsedmarc report	Detected by	UDM metadata.event_type
DMARC aggregate	xml_schema	EMAIL_TRANSACTION
DMARC failure	feedback_type or arrival_date_utc	EMAIL_TRANSACTION
SMTP TLS (RFC 8460)	policy_type or result_type	GENERIC_EVENT

The or fallbacks matter: text-format failure reports have no Feedback-Type field (parsedmarc emits no feedback_type key for them, but always computes arrival_date_utc), and SMTP TLS failure-detail rows from parsedmarc versions older than this parser lack policy_type (every RFC 8460 failure detail has a result_type).

EMAIL_TRANSACTION and GENERIC_EVENT are both valid metadata.event_type values. Note that GENERIC_EVENT events only appear in raw-log and UDM search, not in the curated SecOps views — that is the documented behaviour for generic events, and it is why SMTP TLS reports surface differently from the two DMARC types.

Caveats

1. Unvalidated — see Status.

2. JSON types — Chronicle's json{} filter preserves the original JSON type, so parsedmarc's booleans and numbers are handled differently:

   • Booleans (dmarc_aligned / spf_aligned / dkim_aligned / normalized_timespan) are converted to strings because CBN conditionals compare against the preserved type, so [dmarc_aligned] == "false" needs a string. In additional.fields they are stored as typed bool_value — built as a string, convert-ed back to boolean, then renamed — the boolean pattern from Google's parser extension examples; UDM search matches them with value.bool_value. Note that testing is not a boolean — parsedmarc emits the RFC 9990 t= flag as the string "y"/"n" — so it is passed through as a string and guarded with != "".
   • Numbers (count / *_session_count / source_asn) are stored as number_value — built as a string, convert-ed to uinteger, then renamed — so SecOps can range-query and sort them (parsedmarc's "store numbers as numbers" rule). Each numeric interpolation carries on_error so a tenant where %{} rejects non-string fields degrades to a missing additional.fields entry instead of _failed_parsing_.

   Both value types match what the [gsecops] API output emits, so UDM searches and dashboards port between the two delivery paths unchanged.

   Every if-tested field is initialized to "" before json and guarded with != "": CBN raises _failed_parsing_ on a conditional referencing an absent field, and treats an initialized-but-empty field as present. A DMARC-fail record (dmarc_aligned=false) should yield security_result.category = AUTH_VIOLATION — worth confirming in the validation tool.

3. JSON nulls — failure-report rows are the only shape that emits JSON null values (see the failure sample below: dkim_domain, source_*). Google's docs cover absent fields (the "" initialization above) but are silent on whether json{} overwrites an initialized field with a null. This is why the failure sample should be the first event pasted into the validation tool.

4. Aggregate count — a DMARC aggregate record summarizes count messages from one source IP, not a single message. Each record becomes one EMAIL_TRANSACTION with count carried in additional.fields. There is no first-class per-message expansion (fanning out count copies would misrepresent the data).

5. Address format — aggregate reports only carry the From domain, so network.email.from holds a bare domain for aggregate events but a full address for failure events. UDM email-address fields are expected to be local-mailbox@domain; downstream consumers should account for the aggregate-domain case.

UDM field mappings

All UDM field names below are from the UDM field list and SecurityResult reference.

DMARC aggregate → EMAIL_TRANSACTION

parsedmarc field	UDM field
begin_date	metadata.event_timestamp (via date{})
report_id	metadata.product_log_id
source_ip_address	principal.ip
source_reverse_dns	principal.hostname
source_country	principal.location.country_or_region
domain	target.hostname
header_from	network.email.from (domain; see caveat 4)
disposition	security_result.action (none→ALLOW, quarantine→QUARANTINE, reject→BLOCK)
dmarc_aligned=false	security_result.category = AUTH_VIOLATION
org_name, org_email, org_extra_contact_info, errors, count, p, sp, np, pct, fo, adkim, aspf, testing, discovery_method, normalized_timespan, *_aligned, dkim_*, spf_*, policy_override_*, source_base_domain, source_name, source_type, source_asn, source_as_name, source_as_domain, envelope_from, envelope_to	additional.fields

DMARC failure → EMAIL_TRANSACTION

parsedmarc field	UDM field
arrival_date_utc	metadata.event_timestamp (via date{})
message_id	metadata.product_log_id, network.email.mail_id
source_ip_address	principal.ip
source_reverse_dns	principal.hostname
source_country	principal.location.country_or_region
reported_domain	target.hostname
original_mail_from	network.email.from
original_rcpt_to	network.email.to (repeated field → merged)
subject	network.email.subject (repeated field → merged)
delivery_result	security_result.action (reject→BLOCK, quarantine→QUARANTINE, delivered→ALLOW); security_result.category is always AUTH_VIOLATION
feedback_type, auth_failure, delivery_result, authentication_results, authentication_mechanisms, user_agent, dkim_domain, arrival_date, source_base_domain, source_name, source_type, source_asn, source_as_name, source_as_domain	additional.fields

SMTP TLS → GENERIC_EVENT

parsedmarc field	UDM field
begin_date	metadata.event_timestamp (ISO 8601, via date{})
report_id	metadata.product_log_id
policy_domain	target.hostname (the noun; falls back to receiving_mx_hostname when absent)
receiving_ip	target.ip (failure rows only)
sending_mta_ip	principal.ip (failure rows only)
result_type	security_result (action=FAIL, category=POLICY_VIOLATION) — failure rows only
organization_name, policy_type, policy_strings, mx_host_patterns, successful_session_count, failed_session_count, failure_reason_code, receiving_mx_hostname, receiving_mx_helo, additional_info_uri	additional.fields

parsedmarc emits SMTP TLS reports as separate rows: one success row per policy (counts, no MTA IPs) and one failure row per failure detail (which may also lack MTA IPs, e.g. sts-policy-fetch-error). As of the parsedmarc release that ships this parser, every row — success and failure — carries policy_domain and policy_type, so the noun is always available. Rows from older parsedmarc versions omit both on failure details; those are detected via result_type and fall back to receiving_mx_hostname for the noun.

Installation

1. Configure parsedmarc syslog output

[syslog]
server = your-collector.example.com
port = 514

parsedmarc writes each report record as a single-line JSON message.

2. Collect the syslog stream into SecOps

Syslog is ingested by a collector, not a Feed. Run the Bindplane agent (Google's recommended on-premises collector; the legacy Chronicle forwarder is end-of-life) with a Syslog collector pointed at the port above, and assign it a custom log type (for example PARSEDMARC).

3. Install this parser for that log type

Associate parsedmarc.conf with the custom log type via the SecOps parser management UI or API (see Manage parsers). Validate against the sample events below before activating.

Sample events for validation

These are real single-line outputs from parsedmarc's [syslog] serializers, generated from this repository's sample reports (the source file is named on each event). Use them in the parser-validation tool. A live syslog line will also carry a <PRI> prefix; the parser strips any leading framing before the first {.

Suggested order:

1. DMARC failure first — it is the only shape containing JSON null values (caveat 3), the least-documented json{} behaviour this parser depends on.
2. The two aggregates — confirm the dmarc_aligned=false one yields security_result.category = AUTH_VIOLATION, that count and source_asn land as number_value, and that the alignment booleans land as bool_value (queryable as additional.fields["dmarc_aligned"].value.bool_value).
3. The SMTP TLS rows — confirm the success row produces a GENERIC_EVENT with target.hostname and no security_result, and the failure rows produce security_result.action = FAIL.

DMARC failure report — from samples/failure/DMARC Failure Report for domain.de (…).eml (note the JSON nulls)

{"feedback_type": "auth-failure", "user_agent": "Lua/1.0", "version": "1.0", "original_mail_from": "sharepoint@domain.de", "original_rcpt_to": "peter.pan@domain.de", "arrival_date": "Mon, 01 Oct 2018 11:20:27 +0200", "message_id": "<38.E7.30937.BD6E1BB5@ mailrelay.de>", "authentication_results": "dmarc=fail (p=none, dis=none) header.from=domain.de", "delivery_result": "policy", "auth_failure": "dmarc", "reported_domain": "domain.de", "arrival_date_utc": "2018-10-01 09:20:27", "authentication_mechanisms": "", "original_envelope_id": null, "dkim_domain": null, "sample_headers_only": false, "source_ip_address": "10.10.10.10", "source_reverse_dns": null, "source_base_domain": null, "source_name": null, "source_type": null, "source_asn": null, "source_as_name": null, "source_as_domain": null, "source_country": null, "subject": "Subject"}

DMARC Aggregate — fail (dmarc_aligned=false) — from samples/aggregate/!example.com!1538204542!1538463818.xml

{"xml_schema": "draft", "org_name": "accurateplastics.com", "org_email": "administrator@accurateplastics.com", "org_extra_contact_info": "", "report_id": "example.com:1538463741", "begin_date": "2018-10-01 17:07:12", "end_date": "2018-10-01 17:07:12", "normalized_timespan": false, "errors": "", "domain": "example.com", "adkim": "r", "aspf": "r", "p": "none", "sp": "reject", "np": "", "pct": "100", "fo": "", "testing": "", "discovery_method": "", "source_ip_address": "12.20.127.122", "source_country": "US", "source_reverse_dns": "", "source_base_domain": "", "source_name": "AT&T", "source_type": "ISP", "source_asn": 7018, "source_as_name": "AT&T Enterprises, LLC", "source_as_domain": "att.com", "count": 1, "spf_aligned": false, "dkim_aligned": false, "dmarc_aligned": false, "disposition": "none", "policy_override_reasons": "", "policy_override_comments": "", "envelope_from": "", "header_from": "example.com", "envelope_to": "", "dkim_domains": "", "dkim_selectors": "", "dkim_results": "", "spf_domains": "", "spf_scopes": "", "spf_results": ""}

DMARC Aggregate — pass (dmarc_aligned=true) — from samples/aggregate/empty_reason.xml

{"xml_schema": "1.0", "org_name": "example.org", "org_email": "noreply-dmarc-support@example.org", "org_extra_contact_info": "https://support.example.org/dmarc", "report_id": "20240125141224705995", "begin_date": "2024-01-25 05:12:24", "end_date": "2024-01-25 12:28:53", "normalized_timespan": false, "errors": "", "domain": "example.com", "adkim": "r", "aspf": "r", "p": "quarantine", "sp": "quarantine", "np": "", "pct": "100", "fo": "1", "testing": "", "discovery_method": "", "source_ip_address": "198.51.100.123", "source_country": "", "source_reverse_dns": "", "source_base_domain": "", "source_name": "", "source_type": "", "source_asn": "", "source_as_name": "", "source_as_domain": "", "count": 2, "spf_aligned": false, "dkim_aligned": true, "dmarc_aligned": true, "disposition": "none", "policy_override_reasons": "none", "policy_override_comments": "none", "envelope_from": "example.edu", "header_from": "example.com", "envelope_to": "example.net", "dkim_domains": "example.com", "dkim_selectors": "example", "dkim_results": "pass", "spf_domains": "example.edu", "spf_scopes": "mfrom", "spf_results": "pass"}

SMTP TLS — success row (counts only) — from samples/smtp_tls/rfc8460.json

{"organization_name": "Company-X", "begin_date": "2016-04-01T00:00:00Z", "end_date": "2016-04-01T23:59:59Z", "report_id": "5065427c-23d3-47ca-b6e0-946ea0e8c4be", "policy_domain": "company-y.example", "policy_type": "sts", "policy_strings": "version: STSv1|mode: testing|mx: *.mail.company-y.example|max_age: 86400", "successful_session_count": 5326, "failed_session_count": 303}

SMTP TLS — failure-detail row with MTA IPs — from samples/smtp_tls/rfc8460.json

{"organization_name": "Company-X", "begin_date": "2016-04-01T00:00:00Z", "end_date": "2016-04-01T23:59:59Z", "report_id": "5065427c-23d3-47ca-b6e0-946ea0e8c4be", "policy_domain": "company-y.example", "policy_type": "sts", "policy_strings": "version: STSv1|mode: testing|mx: *.mail.company-y.example|max_age: 86400", "result_type": "starttls-not-supported", "failed_session_count": 200, "sending_mta_ip": "2001:db8:abcd:0013::1", "receiving_ip": "203.0.113.56", "receiving_mx_hostname": "mx2.mail.company-y.example"}

SMTP TLS — failure-detail row without MTA IPs — from samples/smtp_tls/mail.ru.json

{"organization_name": "Mail.ru", "begin_date": "2024-02-22T00:00:00Z", "end_date": "2024-02-23T00:00:00Z", "report_id": "b28254de-7b2e-be36-bb5c-4c3b92da8b25@mail.ru", "policy_domain": "example.com", "policy_type": "sts", "result_type": "sts-policy-fetch-error", "failed_session_count": 1, "failure_reason_code": "bad https response code: 404"}

Official references

• Overview of the UDM
• UDM field list
• SecurityResult reference
• Overview of log parsing
• Parser syntax reference
• Tips and troubleshooting when writing parsers — intermediate fields are discarded unless mapped to @output; one parser is active per log type.
• Manage prebuilt and custom parsers
• Chronicle content-hub — Google's official repository of third-party SecOps parsers. Its CBN parsers (e.g. CLOUDFLARE_PAGESHIELD) use the same conventions this one does: initialize fields before json, convert JSON types to strings, map to event.idm.read_only_udm.*, and finalize with @output.
• UDM search — GENERIC_EVENT events only surface in raw-log / UDM search, not curated views.
• Install the Bindplane agent (collector)
• Feed management

Additional sources and tooling

Community resources (not official Google documentation) that informed this parser's JSON handling and are useful when validating it:

• Parsing 101: Best Practices & Tips (Chris Martin / @thatsiemguy) — basis for initializing every if-tested field before the json filter to avoid _failed_parsing_.
• Corelight parser for SecOps — a large production CBN parser that demonstrates converting type-preserved JSON booleans to strings before testing them in conditionals, which this parser does in step 1b (storage is a separate matter: this parser stores booleans as typed bool_value per Google's parser extension examples).
• chronicle/cbn-tool — CLI for the CBN parser APIs (submit and validate a parser).

License

Distributed under the same license as parsedmarc.