mirror of
https://github.com/domainaware/parsedmarc.git
synced 2026-07-31 04:35:58 +00:00
Some Exim/cPanel-based gateways send DMARC failure reports without a machine-readable message/feedback-report part. parse_report_email()'s plain-text fallback synthesizes a minimal feedback report with only Arrival-Date and Source-IP, but the Elasticsearch/OpenSearch outputs access feedback_type and authentication_results with hard key lookups, so every such report was archived but never indexed, failing with "Failure report missing required field: 'feedback_type'". parse_failure_report() now defaults feedback_type to auth-failure (RFC 5965 3.1) and authentication_results to None (RFC 6591 3.1) with logged warnings, matching the existing handling of the REQUIRED Auth-Failure and Identity-Alignment fields. Adds a sanitized sample and a regression test asserting the sink-required keys are present.
This commit is contained in:
supaeasy
@@ -4,6 +4,7 @@
|
||||
|
||||
### Bug fixes
|
||||
|
||||
- **Failure reports without a `message/feedback-report` part are no longer silently dropped by the Elasticsearch and OpenSearch outputs** ([#332](https://github.com/domainaware/parsedmarc/issues/332)). Some Exim/cPanel-based gateways send DMARC failure reports as `multipart/report` without a `report-type=feedback` parameter and without a machine-readable `message/feedback-report` part — only a plain-text summary starting with "A message claiming to be from you has failed". `parse_report_email()` has a dedicated fallback for this format that synthesizes a minimal feedback report containing only `Arrival-Date` and `Source-IP`, and `parse_failure_report()` already defaulted most missing fields — but not `feedback_type` or `authentication_results`, both of which the Elasticsearch/OpenSearch save paths access with hard key lookups. The result: every such report was parsed, archived to the failure folder, and then rejected at the sink with `Failure report missing required field: 'feedback_type'`, so it never reached the index. `parse_failure_report()` now defaults `feedback_type` to `auth-failure` (RFC 5965 §3.1) and `authentication_results` to `None` (RFC 6591 §3.1), each with a logged warning naming the offending omission — the same treatment the REQUIRED `Auth-Failure` and `Identity-Alignment` fields already receive. A sanitized sample (`samples/failure/exim_plain_text_only_no_arf_part.eml`) and a regression test asserting the sink-required keys are present cover the fallback path.
|
||||
- **The "Aggregate DMARC passage over time" and "Aggregate DMARC message disposition over time" OpenSearch Dashboards visualizations no longer default to a 12-hour X-axis interval** ([#828](https://github.com/domainaware/parsedmarc/issues/828)). Both `date_histogram` aggregations were configured with `"interval": "auto"`, which OpenSearch Dashboards sizes off the currently viewed time range rather than the data's actual cadence. Aggregate DMARC reports post one `date_range`-bucketed data point per reporting period (typically daily), so an auto-computed sub-day bucket size produced empty buckets and a misleading sawtooth/spike pattern. Both visualizations in `dashboards/opensearch/opensearch_dashboards.ndjson` now hardcode `"interval": "d"`, matching the fix the reporter applied manually in the visualization editor (the "Day" option in the interval picker). Per OpenSearch-Dashboards' `_interval_options.ts` and `parse_interval.ts` (`src/plugins/data/common/search/aggs/...`), the stored `interval` value must be a short duration-unit code (`ms`/`s`/`m`/`h`/`d`/`w`/`M`/`y`) or the literal `auto`; a spelled-out word like `"day"` fails `parseInterval`'s regex and throws `"day" is not a valid interval.` at render time, so `"d"` is required, not `"day"`.
|
||||
- **The same auto-interval time-bucketing bug from #828 also affected the PostgreSQL Grafana dashboard's "Over Time" panels** (`dashboards/grafana/Grafana-DMARC_Reports-PostgreSQL.json`): "SPF Results Over Time", "DKIM Results Over Time", "DMARC Pass/Fail Over Time", "Disposition Over Time", "SMTP TLS Sessions Over Time", and one unlabeled summary `stat` panel all grouped rows with `$__timeGroup(<column>, $__interval)`, where `$__interval` is a Grafana-computed value that scales with the panel's pixel width and the selected time range, not the reports' actual daily cadence (per the Grafana PostgreSQL query-editor macro docs). All 7 occurrences now hardcode the bucket width to `'1d'` (`$__timeGroup(rpt.begin_date, '1d')` / `$__timeGroup(tr.begin_date, '1d')`), matching the `fixed_interval: "1d"` already used by the equivalent panels in the companion Elasticsearch/Grafana dashboard (`Grafana-DMARC_Reports.json`), which was already correct and needed no change. The Elasticsearch/Grafana dashboard's non-time-series pie/stat/table panels that still use a `date_histogram` with `"fixed_interval": "auto"` were left as-is: they sum every bucket into a single value or list raw hits, so the bucket width has no visible effect there.
|
||||
- **The same bug also affected the Splunk "DMARC passage over time" and "Message disposition over time" panels** (`dashboards/splunk/dmarc_aggregate_dashboard.xml`). Both `| timechart` calls had no explicit `span`, so Splunk auto-computes a bin size targeting 100 buckets across whatever time range is selected; per Splunk's `timechart` documentation this only coincidentally lands on a 1-day span for the dashboard's default "last 7 days" time-range input; any other selected range (e.g. "last 24 hours" → 30 minutes) reproduces the same gapped/spiked chart. Both queries now set `span=1d` explicitly, per the Splunk docs' own guidance to use `span=1d` (not `span=24h`/`span=86400s`/`span=1440m`) for calendar-day boundaries.
|
||||
|
||||
@@ -1543,6 +1543,27 @@ def parse_failure_report(
|
||||
f.strip() for f in parsed_report["auth_failure"].split(",") if f.strip()
|
||||
]
|
||||
|
||||
# Feedback-Type is REQUIRED per RFC 5965 §3.1, but some gateways
|
||||
# (e.g. Exim/cPanel-based ones that send a plain-text summary without
|
||||
# a machine-readable message/feedback-report part) omit it. The
|
||||
# Elasticsearch/OpenSearch outputs require the key, so default it
|
||||
# instead of dropping the report there (see issue #332).
|
||||
if "feedback_type" not in parsed_report:
|
||||
logger.warning(
|
||||
"Failure report missing required 'Feedback-Type' field "
|
||||
"(RFC 5965 §3.1); defaulting to 'auth-failure'"
|
||||
)
|
||||
parsed_report["feedback_type"] = "auth-failure"
|
||||
|
||||
# Authentication-Results is likewise REQUIRED per RFC 6591 §3.1 and
|
||||
# required by the Elasticsearch/OpenSearch outputs.
|
||||
if "authentication_results" not in parsed_report:
|
||||
logger.warning(
|
||||
"Failure report missing required 'Authentication-Results' "
|
||||
"field (RFC 6591 §3.1); defaulting to None"
|
||||
)
|
||||
parsed_report["authentication_results"] = None
|
||||
|
||||
optional_fields = [
|
||||
"original_envelope_id",
|
||||
"dkim_domain",
|
||||
|
||||
@@ -0,0 +1,65 @@
|
||||
Return-Path: <no-reply@node01.mailgate.example.net>
|
||||
X-Original-To: dmarc@example.com
|
||||
Delivered-To: dmarc@example.com
|
||||
Received: from node04.mailgate.example.net (node04.mailgate.example.net [198.51.100.176])
|
||||
by web01.hosting.example.net (Postfix) with ESMTPS id 9AD6912398C
|
||||
for <dmarc@example.com>; Mon, 7 Apr 2025 23:16:09 +0200 (CEST)
|
||||
Received: from root by node04.mailgate.example.net with local-generated (Exim 4.92)
|
||||
(envelope-from <no-reply@node01.mailgate.example.net>)
|
||||
id 1u1tpB-00AA5u-ED
|
||||
for dmarc@example.com; Mon, 07 Apr 2025 23:16:09 +0200
|
||||
Content-Type: multipart/report;
|
||||
boundary="===============2510560795302005415=="
|
||||
MIME-Version: 1.0
|
||||
Subject: DMARC Forensic Report for example.com from IP 203.0.113.68
|
||||
From: no-reply@node01.mailgate.example.net
|
||||
To: dmarc@example.com
|
||||
Date: Mon, 07 Apr 2025 23:16:09 +0200
|
||||
Auto-Submitted: auto-replied
|
||||
Message-Id: <E1u1tpB-00AA5u-ED@node04.mailgate.example.net>
|
||||
|
||||
--===============2510560795302005415==
|
||||
Content-Type: text/plain; charset="us-ascii"
|
||||
MIME-Version: 1.0
|
||||
Content-Transfer-Encoding: 7bit
|
||||
|
||||
A message claiming to be from you has failed the published DMARC policy for your domain.
|
||||
|
||||
Sender Domain: example.com
|
||||
Sender IP Address: 203.0.113.68
|
||||
Received date: Mon, 07 Apr 2025 23:16:09 +0200
|
||||
SPF Alignment: no
|
||||
DKIM Alignment: no
|
||||
DMARC Results: None, Accept
|
||||
|
||||
------ This is a copy of the headers that were received before the error was detected.
|
||||
|
||||
Received: from [203.0.113.68] (helo=smtpclient.apple)
|
||||
by node04.mailgate.example.net with esmtp (Exim 4.92)
|
||||
(envelope-from <user@example.com>)
|
||||
id 1u1tpA-00AAwZ-CR
|
||||
for user@example.com; Mon, 07 Apr 2025 23:16:08 +0200
|
||||
Received: from [IPv6:::ffff:203.0.113.68] (unknown [203.0.113.68])
|
||||
by example.com (Postfix) with ESMTP id 9CCA25FC9873
|
||||
for <user@example.com>; Mon, 7 Apr 2025 10:10:06 -0600 (UTC)
|
||||
Content-Type: text/plain;
|
||||
charset=windows-1250
|
||||
Content-Transfer-Encoding: 8bit
|
||||
From: <user@example.com>
|
||||
MIME-Version: 1.0 (1.0)
|
||||
Date: Mon, 7 Apr 2025 10:10:06 -0600
|
||||
Subject: Payment from your account.
|
||||
Message-Id: <134117F8-2145-AECE-9CCA-25FC98731341@example.com>
|
||||
To: <user@example.com>
|
||||
X-Mailer: iPhone Mail (22A3351)
|
||||
Received-SPF: softfail (node04.mailgate.example.net: transitioning domain of example.com does not designate 203.0.113.68 as permitted sender) client-ip=203.0.113.68; envelope-from=user@example.com; helo=smtpclient.apple;
|
||||
X-SPF-Result: node04.mailgate.example.net: transitioning domain of example.com does not designate 203.0.113.68 as permitted sender
|
||||
X-Sender-Warning: Reverse DNS lookup failed for 203.0.113.68 (failed)
|
||||
X-DKIM-Status: none / / example.com / / /
|
||||
Authentication-Results: node04.mailgate.example.net;
|
||||
iprev=fail smtp.remote-ip=203.0.113.68;
|
||||
spf=softfail smtp.mailfrom=example.com;
|
||||
dmarc=none header.from=example.com
|
||||
Authentication-Results: mailgate.example.net; spf=softfail smtp.mailfrom=user@example.com
|
||||
|
||||
--===============2510560795302005415==--
|
||||
@@ -191,6 +191,18 @@ class Test(unittest.TestCase):
|
||||
)
|
||||
print("Passed!")
|
||||
|
||||
def testFailureSampleWithoutFeedbackReportPart(self):
|
||||
"""A plain-text-only failure report (no message/feedback-report part)
|
||||
must still contain every field the Elasticsearch/OpenSearch outputs
|
||||
access with hard key lookups (issue #332)"""
|
||||
sample_path = "samples/failure/exim_plain_text_only_no_arf_part.eml"
|
||||
result = parsedmarc.parse_report_file(sample_path, offline=OFFLINE_MODE)
|
||||
assert result["report_type"] == "failure"
|
||||
report = cast(FailureReport, result["report"])
|
||||
assert report["feedback_type"] == "auth-failure"
|
||||
assert "authentication_results" in report
|
||||
assert report["source"]["ip_address"] == "203.0.113.68"
|
||||
|
||||
def testFailureReportBackwardCompat(self):
|
||||
"""Test that old forensic function aliases still work"""
|
||||
self.assertIs(
|
||||
|
||||
Reference in New Issue
Block a user