amorfo77/parsedmarc
1
0
Fork 0
mirror of https://github.com/domainaware/parsedmarc.git synced 2026-07-29 03:44:54 +00:00
Code Issues Projects Releases Wiki Activity

Guarantee non-empty xml_schema: <feedback> identifies the report, not <version>

Browse Source 
The <feedback> XML root element (anchored at the top of
parse_aggregate_report_xml) is what identifies an aggregate report;
<version> is optional metadata per RFC 7489 Appendix C. The "draft"
fallback only covered a fully absent <version>: an empty <version/>
(which xmltodict parses as None) or a whitespace-only value produced
xml_schema=None, which the flat-row serializer coerced to "" — and any
consumer detecting aggregate rows by a non-empty xml_schema, such as
the Google SecOps CBN parser's report-type cascade, silently dropped
the row. The fallback now applies unless <version> carries non-empty
text (attribute-wrapped values are unwrapped via _text like other
elements).

The regression test fails on the previous code with
xml_schema=None != "draft".

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Sean Whalen
2026-07-11 15:52:47 -04:00
co-authored by Claude Fable 5
parent a131fa937b
commit 571bbf741b
5 changed files with 94 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+7
View File
@@ -34,6 +34,13 @@
`policy_strings` / `mx_host_patterns` from an earlier policy were reused for
a later policy that did not define them, because the row template dict was
built once per report instead of once per policy.
- **`xml_schema` can no longer be empty on aggregate rows.** The `<feedback>`
root element is what identifies an aggregate report; `<version>` is optional
metadata (RFC 7489 Appendix C) that reporters omit or leave empty in the
wild. An empty or whitespace-only `<version/>` previously produced
`xml_schema=None` (an empty string in the flat CSV/JSON rows), breaking
consumers that detect aggregate rows by a non-empty `xml_schema`. The
"draft" fallback now applies unless `<version>` carries non-empty text.
## 10.2.1
google_secops_parser/README.md
+4 -1
View File
@@ -57,7 +57,10 @@ The `or` fallbacks matter: text-format failure reports have no `Feedback-Type`
field (parsedmarc emits no `feedback_type` key for them, but always computes
`arrival_date_utc`), and SMTP TLS failure-detail rows from parsedmarc versions
older than this parser lack `policy_type` (every RFC 8460 failure detail has a
`result_type`).
`result_type`). `xml_schema` needs no fallback: the `<feedback>` XML root
element is what identifies an aggregate report, and parsedmarc guarantees a
non-empty `xml_schema` on every aggregate row ("draft" whenever `<version>`
is missing, empty, or whitespace).
`EMAIL_TRANSACTION` and `GENERIC_EVENT` are both valid `metadata.event_type`
values. Note that **`GENERIC_EVENT` events only appear in raw-log and UDM
google_secops_parser/parsedmarc.conf
+5 -3
View File
@@ -241,9 +241,11 @@ filter {
# policy_type alone is not enough for streams from parsedmarc versions
# that omitted policy_domain/policy_type on SMTP TLS failure-detail rows;
# result_type is required in every RFC 8460 failure detail.
# xml_schema is aggregate-only and parsedmarc defaults it to "draft" when
# the report omits <version> (parsedmarc/__init__.py), so it survives a
# missing version. It is preferred over: header_from (can be empty when a
# xml_schema is aggregate-only and guaranteed non-empty: the <feedback>
# XML root element is what identifies an aggregate report, and parsedmarc
# falls back to "draft" whenever <version> is missing, empty, or
# whitespace (parsedmarc/__init__.py), so every aggregate row carries a
# usable value. It is preferred over: header_from (can be empty when a
# record carries no identifiers), adkim (a defaulted policy field), domain
# (a generic name), and dmarc_aligned (a boolean that only becomes testable
# after the convert in step 1b -- detection should not depend on that).
parsedmarc/__init__.py
+11 -1
View File
@@ -848,9 +848,19 @@ def parse_aggregate_report_xml(
report_metadata["email"],
)
report_metadata["email"] = unwrapped
# The <feedback> root element (anchored above) is what identifies an
# aggregate report; <version> is only metadata about the schema
# revision. Reports in the wild omit <version> entirely or send an
# empty <version/> (which xmltodict parses as None) or one carrying
# attributes (a dict). Only a non-empty text value may override the
# "draft" default: xml_schema must never be empty, because flat-row
# consumers (e.g. the Google SecOps parser's report-type detection)
# rely on it being present on every aggregate row.
schema = "draft"
if "version" in report:
schema = report["version"]
version = _text(report["version"])
if isinstance(version, str) and version.strip():
schema = version.strip()
new_report: dict[str, Any] = {
"xml_schema": schema,
"xml_namespace": xml_namespace,
tests/test_init.py
+67
View File
@@ -1497,6 +1497,73 @@ class Test(unittest.TestCase):
self.assertIsNone(pp["testing"])
self.assertIsNone(pp["discovery_method"])
def testEmptyVersionElementFallsBackToDraftSchema(self):
"""An empty <version/> must not blank out xml_schema.
The <feedback> root element is what identifies an aggregate report;
<version> is optional metadata (RFC 7489 Appendix C) that reporters
omit or leave empty in the wild. xmltodict parses an empty
<version/> as None, which previously became xml_schema=None and an
empty string in the flat rows — breaking downstream consumers that
detect aggregate rows by a non-empty xml_schema (the Google SecOps
parser's report-type cascade).
"""
xml = """<?xml version="1.0"?>
<feedback>
<version/>
<report_metadata>
<org_name>TestOrg</org_name>
<email>test@example.com</email>
<report_id>test-empty-version</report_id>
<date_range><begin>1704067200</begin><end>1704153599</end></date_range>
</report_metadata>
<policy_published>
<domain>example.com</domain>
<p>none</p>
</policy_published>
<record>
<row>
<source_ip>192.0.2.1</source_ip>
<count>1</count>
<policy_evaluated><disposition>none</disposition><dkim>pass</dkim><spf>pass</spf></policy_evaluated>
</row>
<identifiers><header_from>example.com</header_from></identifiers>
<auth_results><spf><domain>example.com</domain><result>pass</result></spf></auth_results>
</record>
</feedback>"""
report = parsedmarc.parse_aggregate_report_xml(xml, offline=True)
self.assertEqual(report["xml_schema"], "draft")
rows = parsedmarc.parsed_aggregate_reports_to_csv_rows(report)
self.assertEqual(rows[0]["xml_schema"], "draft")
def testWhitespaceVersionElementFallsBackToDraftSchema(self):
"""A whitespace-only <version> value also falls back to 'draft'."""
xml = """<?xml version="1.0"?>
<feedback>
<version> </version>
<report_metadata>
<org_name>TestOrg</org_name>
<email>test@example.com</email>
<report_id>test-ws-version</report_id>
<date_range><begin>1704067200</begin><end>1704153599</end></date_range>
</report_metadata>
<policy_published>
<domain>example.com</domain>
<p>none</p>
</policy_published>
<record>
<row>
<source_ip>192.0.2.1</source_ip>
<count>1</count>
<policy_evaluated><disposition>none</disposition><dkim>pass</dkim><spf>pass</spf></policy_evaluated>
</row>
<identifiers><header_from>example.com</header_from></identifiers>
<auth_results><spf><domain>example.com</domain><result>pass</result></spf></auth_results>
</record>
</feedback>"""
report = parsedmarc.parse_aggregate_report_xml(xml, offline=True)
self.assertEqual(report["xml_schema"], "draft")
def testMagicXmlTagDetection(self):
"""XML without declaration (starting with '<') is extracted"""
xml_no_decl = b"<feedback><report_metadata><org_name>T</org_name><email>a@b.com</email><report_id>r1</report_id><date_range><begin>1704067200</begin><end>1704153599</end></date_range></report_metadata><policy_published><domain>example.com</domain><p>none</p></policy_published><record><row><source_ip>192.0.2.1</source_ip><count>1</count><policy_evaluated><disposition>none</disposition><dkim>pass</dkim><spf>pass</spf></policy_evaluated></row><identifiers><header_from>example.com</header_from></identifiers><auth_results><spf><domain>example.com</domain><result>pass</result></spf></auth_results></record></feedback>"