From d9f65328419cc6bd4126d09a4d20cfb0b4bba423 Mon Sep 17 00:00:00 2001 From: supaeasy <59504964+supaeasy@users.noreply.github.com> Date: Mon, 20 Jul 2026 21:21:55 +0200 Subject: [PATCH] Default missing Feedback-Type and Authentication-Results in failure reports (#332) (#831) Some Exim/cPanel-based gateways send DMARC failure reports without a machine-readable message/feedback-report part. parse_report_email()'s plain-text fallback synthesizes a minimal feedback report with only Arrival-Date and Source-IP, but the Elasticsearch/OpenSearch outputs access feedback_type and authentication_results with hard key lookups, so every such report was archived but never indexed, failing with "Failure report missing required field: 'feedback_type'". parse_failure_report() now defaults feedback_type to auth-failure (RFC 5965 3.1) and authentication_results to None (RFC 6591 3.1) with logged warnings, matching the existing handling of the REQUIRED Auth-Failure and Identity-Alignment fields. Adds a sanitized sample and a regression test asserting the sink-required keys are present. --- CHANGELOG.md | 1 + parsedmarc/__init__.py | 21 ++++++ .../exim_plain_text_only_no_arf_part.eml | 65 +++++++++++++++++++ tests/test_init.py | 12 ++++ 4 files changed, 99 insertions(+) create mode 100644 samples/failure/exim_plain_text_only_no_arf_part.eml diff --git a/CHANGELOG.md b/CHANGELOG.md index aa8b7ec9..47ba3055 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,7 @@ ### Bug fixes +- **Failure reports without a `message/feedback-report` part are no longer silently dropped by the Elasticsearch and OpenSearch outputs** ([#332](https://github.com/domainaware/parsedmarc/issues/332)). Some Exim/cPanel-based gateways send DMARC failure reports as `multipart/report` without a `report-type=feedback` parameter and without a machine-readable `message/feedback-report` part — only a plain-text summary starting with "A message claiming to be from you has failed". `parse_report_email()` has a dedicated fallback for this format that synthesizes a minimal feedback report containing only `Arrival-Date` and `Source-IP`, and `parse_failure_report()` already defaulted most missing fields — but not `feedback_type` or `authentication_results`, both of which the Elasticsearch/OpenSearch save paths access with hard key lookups. The result: every such report was parsed, archived to the failure folder, and then rejected at the sink with `Failure report missing required field: 'feedback_type'`, so it never reached the index. `parse_failure_report()` now defaults `feedback_type` to `auth-failure` (RFC 5965 §3.1) and `authentication_results` to `None` (RFC 6591 §3.1), each with a logged warning naming the offending omission — the same treatment the REQUIRED `Auth-Failure` and `Identity-Alignment` fields already receive. A sanitized sample (`samples/failure/exim_plain_text_only_no_arf_part.eml`) and a regression test asserting the sink-required keys are present cover the fallback path. - **The "Aggregate DMARC passage over time" and "Aggregate DMARC message disposition over time" OpenSearch Dashboards visualizations no longer default to a 12-hour X-axis interval** ([#828](https://github.com/domainaware/parsedmarc/issues/828)). Both `date_histogram` aggregations were configured with `"interval": "auto"`, which OpenSearch Dashboards sizes off the currently viewed time range rather than the data's actual cadence. Aggregate DMARC reports post one `date_range`-bucketed data point per reporting period (typically daily), so an auto-computed sub-day bucket size produced empty buckets and a misleading sawtooth/spike pattern. Both visualizations in `dashboards/opensearch/opensearch_dashboards.ndjson` now hardcode `"interval": "d"`, matching the fix the reporter applied manually in the visualization editor (the "Day" option in the interval picker). Per OpenSearch-Dashboards' `_interval_options.ts` and `parse_interval.ts` (`src/plugins/data/common/search/aggs/...`), the stored `interval` value must be a short duration-unit code (`ms`/`s`/`m`/`h`/`d`/`w`/`M`/`y`) or the literal `auto`; a spelled-out word like `"day"` fails `parseInterval`'s regex and throws `"day" is not a valid interval.` at render time, so `"d"` is required, not `"day"`. - **The same auto-interval time-bucketing bug from #828 also affected the PostgreSQL Grafana dashboard's "Over Time" panels** (`dashboards/grafana/Grafana-DMARC_Reports-PostgreSQL.json`): "SPF Results Over Time", "DKIM Results Over Time", "DMARC Pass/Fail Over Time", "Disposition Over Time", "SMTP TLS Sessions Over Time", and one unlabeled summary `stat` panel all grouped rows with `$__timeGroup(, $__interval)`, where `$__interval` is a Grafana-computed value that scales with the panel's pixel width and the selected time range, not the reports' actual daily cadence (per the Grafana PostgreSQL query-editor macro docs). All 7 occurrences now hardcode the bucket width to `'1d'` (`$__timeGroup(rpt.begin_date, '1d')` / `$__timeGroup(tr.begin_date, '1d')`), matching the `fixed_interval: "1d"` already used by the equivalent panels in the companion Elasticsearch/Grafana dashboard (`Grafana-DMARC_Reports.json`), which was already correct and needed no change. The Elasticsearch/Grafana dashboard's non-time-series pie/stat/table panels that still use a `date_histogram` with `"fixed_interval": "auto"` were left as-is: they sum every bucket into a single value or list raw hits, so the bucket width has no visible effect there. - **The same bug also affected the Splunk "DMARC passage over time" and "Message disposition over time" panels** (`dashboards/splunk/dmarc_aggregate_dashboard.xml`). Both `| timechart` calls had no explicit `span`, so Splunk auto-computes a bin size targeting 100 buckets across whatever time range is selected; per Splunk's `timechart` documentation this only coincidentally lands on a 1-day span for the dashboard's default "last 7 days" time-range input; any other selected range (e.g. "last 24 hours" → 30 minutes) reproduces the same gapped/spiked chart. Both queries now set `span=1d` explicitly, per the Splunk docs' own guidance to use `span=1d` (not `span=24h`/`span=86400s`/`span=1440m`) for calendar-day boundaries. diff --git a/parsedmarc/__init__.py b/parsedmarc/__init__.py index 981a54f6..96e6dd90 100644 --- a/parsedmarc/__init__.py +++ b/parsedmarc/__init__.py @@ -1543,6 +1543,27 @@ def parse_failure_report( f.strip() for f in parsed_report["auth_failure"].split(",") if f.strip() ] + # Feedback-Type is REQUIRED per RFC 5965 §3.1, but some gateways + # (e.g. Exim/cPanel-based ones that send a plain-text summary without + # a machine-readable message/feedback-report part) omit it. The + # Elasticsearch/OpenSearch outputs require the key, so default it + # instead of dropping the report there (see issue #332). + if "feedback_type" not in parsed_report: + logger.warning( + "Failure report missing required 'Feedback-Type' field " + "(RFC 5965 §3.1); defaulting to 'auth-failure'" + ) + parsed_report["feedback_type"] = "auth-failure" + + # Authentication-Results is likewise REQUIRED per RFC 6591 §3.1 and + # required by the Elasticsearch/OpenSearch outputs. + if "authentication_results" not in parsed_report: + logger.warning( + "Failure report missing required 'Authentication-Results' " + "field (RFC 6591 §3.1); defaulting to None" + ) + parsed_report["authentication_results"] = None + optional_fields = [ "original_envelope_id", "dkim_domain", diff --git a/samples/failure/exim_plain_text_only_no_arf_part.eml b/samples/failure/exim_plain_text_only_no_arf_part.eml new file mode 100644 index 00000000..b2f810f4 --- /dev/null +++ b/samples/failure/exim_plain_text_only_no_arf_part.eml @@ -0,0 +1,65 @@ +Return-Path: +X-Original-To: dmarc@example.com +Delivered-To: dmarc@example.com +Received: from node04.mailgate.example.net (node04.mailgate.example.net [198.51.100.176]) + by web01.hosting.example.net (Postfix) with ESMTPS id 9AD6912398C + for ; Mon, 7 Apr 2025 23:16:09 +0200 (CEST) +Received: from root by node04.mailgate.example.net with local-generated (Exim 4.92) + (envelope-from ) + id 1u1tpB-00AA5u-ED + for dmarc@example.com; Mon, 07 Apr 2025 23:16:09 +0200 +Content-Type: multipart/report; + boundary="===============2510560795302005415==" +MIME-Version: 1.0 +Subject: DMARC Forensic Report for example.com from IP 203.0.113.68 +From: no-reply@node01.mailgate.example.net +To: dmarc@example.com +Date: Mon, 07 Apr 2025 23:16:09 +0200 +Auto-Submitted: auto-replied +Message-Id: + +--===============2510560795302005415== +Content-Type: text/plain; charset="us-ascii" +MIME-Version: 1.0 +Content-Transfer-Encoding: 7bit + +A message claiming to be from you has failed the published DMARC policy for your domain. + + Sender Domain: example.com + Sender IP Address: 203.0.113.68 + Received date: Mon, 07 Apr 2025 23:16:09 +0200 + SPF Alignment: no + DKIM Alignment: no + DMARC Results: None, Accept + +------ This is a copy of the headers that were received before the error was detected. + +Received: from [203.0.113.68] (helo=smtpclient.apple) + by node04.mailgate.example.net with esmtp (Exim 4.92) + (envelope-from ) + id 1u1tpA-00AAwZ-CR + for user@example.com; Mon, 07 Apr 2025 23:16:08 +0200 +Received: from [IPv6:::ffff:203.0.113.68] (unknown [203.0.113.68]) + by example.com (Postfix) with ESMTP id 9CCA25FC9873 + for ; Mon, 7 Apr 2025 10:10:06 -0600 (UTC) +Content-Type: text/plain; + charset=windows-1250 +Content-Transfer-Encoding: 8bit +From: +MIME-Version: 1.0 (1.0) +Date: Mon, 7 Apr 2025 10:10:06 -0600 +Subject: Payment from your account. +Message-Id: <134117F8-2145-AECE-9CCA-25FC98731341@example.com> +To: +X-Mailer: iPhone Mail (22A3351) +Received-SPF: softfail (node04.mailgate.example.net: transitioning domain of example.com does not designate 203.0.113.68 as permitted sender) client-ip=203.0.113.68; envelope-from=user@example.com; helo=smtpclient.apple; +X-SPF-Result: node04.mailgate.example.net: transitioning domain of example.com does not designate 203.0.113.68 as permitted sender +X-Sender-Warning: Reverse DNS lookup failed for 203.0.113.68 (failed) +X-DKIM-Status: none / / example.com / / / +Authentication-Results: node04.mailgate.example.net; + iprev=fail smtp.remote-ip=203.0.113.68; + spf=softfail smtp.mailfrom=example.com; + dmarc=none header.from=example.com +Authentication-Results: mailgate.example.net; spf=softfail smtp.mailfrom=user@example.com + +--===============2510560795302005415==-- diff --git a/tests/test_init.py b/tests/test_init.py index f01df394..4553061f 100644 --- a/tests/test_init.py +++ b/tests/test_init.py @@ -191,6 +191,18 @@ class Test(unittest.TestCase): ) print("Passed!") + def testFailureSampleWithoutFeedbackReportPart(self): + """A plain-text-only failure report (no message/feedback-report part) + must still contain every field the Elasticsearch/OpenSearch outputs + access with hard key lookups (issue #332)""" + sample_path = "samples/failure/exim_plain_text_only_no_arf_part.eml" + result = parsedmarc.parse_report_file(sample_path, offline=OFFLINE_MODE) + assert result["report_type"] == "failure" + report = cast(FailureReport, result["report"]) + assert report["feedback_type"] == "auth-failure" + assert "authentication_results" in report + assert report["source"]["ip_address"] == "203.0.113.68" + def testFailureReportBackwardCompat(self): """Test that old forensic function aliases still work""" self.assertIs(