mirror of
https://github.com/domainaware/parsedmarc.git
synced 2026-08-01 13:12:18 +00:00
Add Google SecOps (Chronicle) output via the v1 events.import API
Sends parsed reports to a Google SecOps instance as pre-normalized UDM
events through the GA v1 Chronicle API:
POST https://chronicle.{region}.rep.googleapis.com/v1/{parent}/events:import
Pre-normalized events bypass SecOps's server-side (CBN) parsing layer,
so no parser needs to be installed in the tenant; the CBN parser in
google_secops_parser/ remains the collector-based alternative for
deployments that want raw-log retention, and both paths emit the same
UDM shape and additional-field keys so searches port between them.
Google's ingestion docs recommend the UDM-events path when possible.
Implementation notes, all grounded in the v1 reference docs:
- Auth is standard Google Cloud IAM (Chronicle API Editor role /
chronicle.events.import permission): a service account key file when
[gsecops] credentials_file is set, Application Default Credentials
otherwise. google-auth was already a transitive requirement via
mailsuite[gmail]; it is now declared directly.
- Batches follow the documented best practices (1,000 events per
request, 60 s timeout). events.import is all-or-nothing — one invalid
event rejects the whole request — so a rejected batch is bisected to
isolate invalid events; valid events are still delivered and the drop
count is raised as an output error at the end.
- to and subject are repeated fields in the UDM Email message;
SecurityResult action and category are repeated enums.
- additional is a protobuf Struct, so counts and ASNs stay numbers and
alignment flags stay booleans — range-queryable without the CBN
string-hop.
New [gsecops] config section (project_id, instance_id, region,
credentials_file) is wired through the INI schema, _parse_config,
Namespace defaults, PARSEDMARC_GSECOPS_* env vars, the usage docs, and
per-batch client construction (SIGHUP-reload safe by construction).
Tests build events from real sample reports and assert on the payloads
sent through a mocked AuthorizedSession, including batching and the
400-bisect path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
95e881bdb3
commit
fba083f351
@@ -5,6 +5,15 @@ custom parser (configuration-based normalizer / CBN) that maps the JSON events
|
||||
parsedmarc emits through its built-in `[syslog]` output to the Unified Data
|
||||
Model (UDM).
|
||||
|
||||
> **Prefer an API-based setup?** parsedmarc also ships a `[gsecops]` output
|
||||
> that sends the same UDM events directly to the Chronicle API
|
||||
> (`events.import`), with no collector or custom parser to install — see the
|
||||
> `gsecops` section of the parsedmarc usage documentation. Use this parser
|
||||
> instead when you want collector-based ingestion with raw-log retention, or
|
||||
> when you already run a Bindplane pipeline. Both paths use the same UDM
|
||||
> mapping and the same `additional` field keys, so searches and dashboards
|
||||
> port between them.
|
||||
|
||||
This is a **SecOps-side parser** — parsedmarc already ships structured JSON
|
||||
over syslog, and the DMARC→UDM mapping lives here so that a downstream UDM
|
||||
schema change is a parser edit rather than a parsedmarc release. One paired
|
||||
@@ -39,7 +48,7 @@ parsedmarc emits three flat JSON shapes (one object per syslog line). The parser
|
||||
detects them by a field unique to each and maps them as follows:
|
||||
|
||||
| parsedmarc report | Detected by | UDM `metadata.event_type` |
|
||||
|---|---|---|
|
||||
| --- | --- | --- |
|
||||
| DMARC aggregate | `xml_schema` | `EMAIL_TRANSACTION` |
|
||||
| DMARC failure | `feedback_type` or `arrival_date_utc` | `EMAIL_TRANSACTION` |
|
||||
| SMTP TLS (RFC 8460) | `policy_type` or `result_type` | `GENERIC_EVENT` |
|
||||
@@ -106,7 +115,7 @@ and [SecurityResult reference](https://docs.cloud.google.com/chronicle/docs/refe
|
||||
### DMARC aggregate → `EMAIL_TRANSACTION`
|
||||
|
||||
| parsedmarc field | UDM field |
|
||||
|---|---|
|
||||
| --- | --- |
|
||||
| `begin_date` | `metadata.event_timestamp` (via `date{}`) |
|
||||
| `report_id` | `metadata.product_log_id` |
|
||||
| `source_ip_address` | `principal.ip` |
|
||||
@@ -121,7 +130,7 @@ and [SecurityResult reference](https://docs.cloud.google.com/chronicle/docs/refe
|
||||
### DMARC failure → `EMAIL_TRANSACTION`
|
||||
|
||||
| parsedmarc field | UDM field |
|
||||
|---|---|
|
||||
| --- | --- |
|
||||
| `arrival_date_utc` | `metadata.event_timestamp` (via `date{}`) |
|
||||
| `message_id` | `metadata.product_log_id`, `network.email.mail_id` |
|
||||
| `source_ip_address` | `principal.ip` |
|
||||
@@ -137,7 +146,7 @@ and [SecurityResult reference](https://docs.cloud.google.com/chronicle/docs/refe
|
||||
### SMTP TLS → `GENERIC_EVENT`
|
||||
|
||||
| parsedmarc field | UDM field |
|
||||
|---|---|
|
||||
| --- | --- |
|
||||
| `begin_date` | `metadata.event_timestamp` (ISO 8601, via `date{}`) |
|
||||
| `report_id` | `metadata.product_log_id` |
|
||||
| `policy_domain` | `target.hostname` (the noun; falls back to `receiving_mx_hostname` when absent) |
|
||||
|
||||
Reference in New Issue
Block a user