The <feedback> XML root element (anchored at the top of
parse_aggregate_report_xml) is what identifies an aggregate report;
<version> is optional metadata per RFC 7489 Appendix C. The "draft"
fallback only covered a fully absent <version>: an empty <version/>
(which xmltodict parses as None) or a whitespace-only value produced
xml_schema=None, which the flat-row serializer coerced to "" — and any
consumer detecting aggregate rows by a non-empty xml_schema, such as
the Google SecOps CBN parser's report-type cascade, silently dropped
the row. The fallback now applies unless <version> carries non-empty
text (attribute-wrapped values are unwrapped via _text like other
elements).
The regression test fails on the previous code with
xml_schema=None != "draft".
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The comment described the behavior but not the reason: CBN's documented
conditional syntax only compares a token against a quoted string
literal -- there is no boolean literal and no truthiness test on event
fields -- so a type-preserved JSON boolean can never match any
conditional without first converting it to a string. The conversion is
in place because CBN mutate has no copy function. Cite the syntax
reference.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The step-1b comment still said booleans are "stored as string_value
(matching Google's content-hub parsers, which never use bool_value)" —
stale since the switch to typed bool_value storage. 1b now says what it
actually does: convert booleans to strings for the `if` guards only,
because CBN conditionals compare against the preserved JSON type;
storage is typed bool_value per the alignment blocks. Also tightened
the README's Corelight citation, which overstated the string idiom as
something the whole parser relies on rather than just the conditionals.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The alignment flags and normalized_timespan were stored as
string_value "true"/"false" in additional.fields, which diverged from
the [gsecops] API output (a protobuf Struct, where they are real
booleans) — a UDM search written for one delivery path would not match
the other for those four fields.
Google's parser extension examples document the boolean idiom this
uses: build a string, convert to boolean, rename into the boolean
field. It is the boolean analogue of the number_value chain already
used for count/source_asn (proven in the content-hub Azure Cosmos DB
parser), and the UDM search docs confirm additional fields are
matchable via value.bool_value. The string conversion in step 1b
remains, because CBN conditionals compare against the preserved JSON
type and the == "true"/"false" guards need strings.
With this, both delivery paths emit identical value types for every
additional.fields key. The prior justification ("content-hub never
uses bool_value") described the absence of precedent, not a
prohibition; the documented boolean example is the stronger authority.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sends parsed reports to a Google SecOps instance as pre-normalized UDM
events through the GA v1 Chronicle API:
POST https://chronicle.{region}.rep.googleapis.com/v1/{parent}/events:import
Pre-normalized events bypass SecOps's server-side (CBN) parsing layer,
so no parser needs to be installed in the tenant; the CBN parser in
google_secops_parser/ remains the collector-based alternative for
deployments that want raw-log retention, and both paths emit the same
UDM shape and additional-field keys so searches port between them.
Google's ingestion docs recommend the UDM-events path when possible.
Implementation notes, all grounded in the v1 reference docs:
- Auth is standard Google Cloud IAM (Chronicle API Editor role /
chronicle.events.import permission): a service account key file when
[gsecops] credentials_file is set, Application Default Credentials
otherwise. google-auth was already a transitive requirement via
mailsuite[gmail]; it is now declared directly.
- Batches follow the documented best practices (1,000 events per
request, 60 s timeout). events.import is all-or-nothing — one invalid
event rejects the whole request — so a rejected batch is bisected to
isolate invalid events; valid events are still delivered and the drop
count is raised as an output error at the end.
- to and subject are repeated fields in the UDM Email message;
SecurityResult action and category are repeated enums.
- additional is a protobuf Struct, so counts and ASNs stay numbers and
alignment flags stay booleans — range-queryable without the CBN
string-hop.
New [gsecops] config section (project_id, instance_id, region,
credentials_file) is wired through the INI schema, _parse_config,
Namespace defaults, PARSEDMARC_GSECOPS_* env vars, the usage docs, and
per-batch client construction (SIGHUP-reload safe by construction).
Tests build events from real sample reports and assert on the payloads
sent through a mocked AuthorizedSession, including batching and the
400-bisect path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review findings verified against the parsedmarc serializers, the UDM
field list, Google's parser troubleshooting doc, and the content-hub
parsers:
- Detect failure reports by feedback_type OR arrival_date_utc: the
text-format failure path synthesizes a feedback report with no
Feedback-Type field, so those rows had no feedback_type key and were
dropped as TAG_UNSUPPORTED.
- Detect SMTP TLS rows by policy_type OR result_type, and fall back to
receiving_mx_hostname for target.hostname, so failure-detail rows from
parsedmarc versions without the paired serializer fix still map.
- Merge (not replace) network.email.to and network.email.subject: both
are repeated fields per the UDM field list, so a scalar replace is a
type mismatch. The unguarded subject replace could have failed parsing
for every failure event with a subject.
- Guard testing with != "": parsedmarc emits the RFC 9990 t= flag as the
string "y"/"n", not a boolean, so the == "true"/"false" test could
never fire and dmarc_testing was never emitted. Drop testing from the
boolean convert list accordingly.
- Add on_error to the four numeric %{} interpolations (count,
source_asn, *_session_count) so a tenant where interpolating a
non-string field errors degrades to a missing additional.fields entry
instead of _failed_parsing_ (per the official parser tips doc).
- Drop the concatenated security_result.description strings: the tips
doc says not to pack multiple values into one UDM field, and every
value already lands individually in additional.fields.
- Use TAG_MALFORMED_MESSAGE for non-JSON input, matching the tips doc's
canonical not-JSON example (TAG_MALFORMED_ENCODING is for character
encoding problems).
- Map org_extra_contact_info/errors (aggregate) and the source
enrichment fields (failure) into additional.fields; the failure branch
previously dropped enrichment the aggregate branch kept.
- Replace the SMTP TLS success sample, which was not generated from this
repository's samples, with rfc8460.json output; regenerate all sample
events with the fixed serializer and name each event's source sample
file. All six samples are now byte-identical to real syslog output.
- Reorder the validation plan to start with the failure sample: it is
the only shape that emits JSON nulls, whose json{} interaction with
the pre-initialized "" fields is undocumented.
- Update Google doc links to docs.cloud.google.com (the old URLs 301)
and link the parser from the syslog section of the usage docs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Google's content-hub parsers tag dropped logs (drop { tag => "TAG_..." })
so they surface correctly in the unparsed-log views, rather than bare drop{}.
Use TAG_MALFORMED_ENCODING for the two JSON-extraction/parse failures (matches
content-hub) and TAG_UNSUPPORTED for valid JSON that matches no parsedmarc
report shape.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two corrections confirmed against Google's official content-hub parsers
(content/parsers/third_party/community/*/cbn):
1. Numbers as numbers. count, source_asn, successful_session_count and
failed_session_count were being stored in additional.fields as string_value.
Store them as number_value instead (build string -> convert to uinteger ->
rename to number_value, the content-hub idiom), so SecOps can range-query and
sort them, per parsedmarc's "store numbers as numbers" rule. Booleans stay
string_value (content-hub never uses bool_value) and are still converted in
step 1b for the == "true"/"false" comparisons.
2. Conditional guards. Replaced bare `if [field] {` with `if [field] != "" {`
(76 guards + the detection cascade + policy_override). After 1a initializes
every tested field to "", a bare `if` is true for an empty field (Logstash/CBN
semantics), which would misfire detection and emit empty labels. content-hub
uses `!= ""` ~111x vs 2 bare (both flags); parser flags (no_json_payload,
not_json, *_nan) correctly stay bare.
Verified: braces balance, no stray bare field-guards, all if-tested fields
initialized, all four numeric fields emit number_value.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add github.com/chronicle/content-hub (Google's official third-party SecOps
parser repo) to the README references and re-anchor the in-code citations to
it. Its current CBN parsers (e.g. CLOUDFLARE_PAGESHIELD, Copyright 2025 Google
SecOps) confirm both fixes this parser makes: initialize every field before the
json{} filter, and convert JSON booleans/numbers to strings before comparison.
Replaces the dated "How to parse JSON data" citation with the authoritative,
actively-maintained source.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a short, skippable callout explaining what a parser / configuration-based
normalizer (CBN) is and how it fits the SecOps ingest flow (log type → parser →
UDM event), so the README serves newcomers without slowing experienced users.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the remaining official Google docs the parser is built on (parser tips
& troubleshooting, manage parsers, UDM search, Bindplane install) and a
clearly-separated "Additional sources and tooling" section for the community
resources that drove the JSON type-handling and field-init fixes
(thatsiemguy's Parsing 101, the Corelight production parser, chronicle/cbn-tool).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
xml_schema is aggregate-only (failure/SMTP TLS rows don't carry it) and a
distinctive, non-generic field name, addressing the concern that "domain"
could be confused with other logs. parsedmarc defaults xml_schema to "draft"
when the report omits <version> (parsedmarc/__init__.py:832), so it survives a
missing version element -- unlike a field with no default.
It is also a native JSON string straight out of the json{} filter, so unlike
dmarc_aligned it needs no convert step to be testable, keeping detection
independent of the type-conversion in step 1b. xml_schema is added to the
pre-json init block (required for any if-tested field); domain stays
initialized since it is still mapped to target.hostname.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two CBN behaviors, confirmed against Google's own "How to parse JSON data"
guide (statedump shows JSON true/199 retaining boolean/integer type) and the
published Corelight production parser:
1. The json{} filter preserves the original JSON type, so parsedmarc's boolean
*_aligned / testing / normalized_timespan and numeric count / *_session_count
/ source_asn would never match string comparisons. Add a mutate{convert} step
turning them into strings before any == "true"/"false" test or %{...} use.
2. CBN raises _failed_parsing_ when an `if [field]` references a field absent
from the log, and most detection/mapping fields are absent in 2 of the 3
report shapes (or null within one). Initialize every conditionally-checked
field to "" before the json{} filter.
Without these, DMARC-fail records would not be categorized AUTH_VIOLATION and
aggregate/TLS reports could fail parsing outright. README caveat and PR
validation steps updated accordingly.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
adkim is the published policy's DKIM alignment mode (defaulted to "r" by
parsedmarc), an obscure thing to key detection on. Switch the aggregate
detector to "domain" -- the reported From-domain, a required element present
and non-empty in every aggregate record (2388/2388 sample rows) and unique to
aggregate (failure uses reported_domain, SMTP TLS uses policy_domain).
header_from is unsuitable: it can be empty when a record carries no
identifiers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A SecOps-side custom parser (CBN) that maps parsedmarc's [syslog] JSON
events to the Unified Data Model. No library changes: parsedmarc already
emits structured JSON, so the DMARC->UDM mapping lives in the parser and a
downstream UDM schema change is a parser edit, not a parsedmarc release.
Covers all three report types:
- aggregate -> EMAIL_TRANSACTION
- failure -> EMAIL_TRANSACTION
- smtp_tls -> GENERIC_EVENT (noun from policy_domain, present on every row)
Built strictly against the official UDM and parser-syntax docs (cited
inline). Sets metadata.event_timestamp from the report window via date{},
maps disposition / auth-failure to security_result with valid action and
category enums (AUTH_VIOLATION on DMARC fail), uses real network.email
field names, and strips syslog framing before JSON parsing. Ships real
sample events generated from the project's sample reports for validation.
Not yet validated against a live SecOps tenant; caveats are documented in
the README.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
