mirror of
https://github.com/domainaware/parsedmarc.git
synced 2026-08-01 21:22:18 +00:00
Fix DKIM/SPF alignment detail cross-product in dashboards (#169)
Elasticsearch and OpenSearch dynamic-map the dkim_results/spf_results
object arrays as `object` (create_indexes never registers the DSL
document mappings), so Lucene flattens each array into independent
multi-valued fields and stacked terms aggregations on
dkim_results.selector/.domain/.result return every combination of
values across a report's signatures — each phantom row repeating the
full message count.
Aggregate documents now also carry dkim_results_combined and
spf_results_combined: one "selector / domain / result"
("scope / domain / result") string per auth result, composed in
add_dkim_result/add_spf_result. The Kibana/OpenSearch Dashboards and
Grafana (Elasticsearch) alignment-detail tables aggregate those
instead, and the Splunk detail panels pair the values with
mvzip/mvexpand. A documented idempotent _update_by_query backfills
documents saved by older versions; the query matches only documents
that have auth results and lack the combined fields, because an
`exists` query cannot see an empty array.
Also corrects the dead _SPFResult.results (plural) declaration to
`result` (the save path always wrote the singular key), fixes the
result parameter annotations on add_dkim_result/add_spf_result, and
removes the Grafana dmarcian.com DKIM-checker data link, which
required the separate domain/selector columns.
The SMTP TLS visualizations have the same class of defect and are
tracked separately.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
4fa8cfb1e7
commit
6148002a43
@@ -12,6 +12,8 @@
|
||||
- **The Elasticsearch/OpenSearch aggregate dashboards' over-time charts (and the Grafana ES dashboard's summary pies and time series) bucketed on the multi-valued `date_range` field**; a date histogram counts a report once per value, double-counting any report whose begin and end dates fall in different buckets. All date histograms and time-range filters now use the single-valued `date_begin`, matching the report-begin semantics of the PostgreSQL (`begin_date`) and Splunk (`_time` = interval begin) dashboards.
|
||||
- **Aggregate-report policy and authentication result words are now normalized to lowercase** ([#288](https://github.com/domainaware/parsedmarc/issues/288)): reporters that emit mixed-case values such as `Pass` no longer create duplicate result categories in outputs and dashboards.
|
||||
- **The results email (SMTP and Microsoft Graph) is no longer sent when no reports were parsed** ([#200](https://github.com/domainaware/parsedmarc/issues/200)): previously an empty run — an empty inbox, or one where every message was invalid — still emailed a zip of headers-only CSVs. The email step is now skipped with an INFO log when the run produced no aggregate, failure, or SMTP TLS reports.
|
||||
- **The DKIM/SPF alignment-detail tables on the Kibana/OpenSearch Dashboards, Grafana (Elasticsearch), and Splunk aggregate dashboards no longer show a selector × domain × result cross-product** ([#169](https://github.com/domainaware/parsedmarc/issues/169)). Elasticsearch/OpenSearch flatten the `dkim_results`/`spf_results` object arrays (they are dynamic-mapped as `object`, not `nested`), so stacking terms aggregations on their subfields produced every combination of selector/domain/result across a report's signatures, each repeating the full message count. Aggregate documents now also carry `dkim_results_combined` and `spf_results_combined` — one `"selector / domain / result"` (`"scope / domain / result"`) string per auth result — and the dashboards aggregate those instead; the Splunk detail panels now pair the values with `mvzip`/`mvexpand`. Documents saved by older versions can be backfilled with a documented `_update_by_query` command (see the Elasticsearch docs page). The Grafana "DKIM Alignment Details" panel's dmarcian.com DKIM-checker data link was removed because it required the separate domain/selector columns. The SMTP TLS visualizations have the same class of defect and are tracked separately.
|
||||
- **Corrected the dead `_SPFResult.results` (plural) field declaration to `result`**, matching what was always written to it.
|
||||
|
||||
## 10.2.4
|
||||
|
||||
|
||||
@@ -2598,7 +2598,7 @@
|
||||
{
|
||||
"$$hashKey": "object:412",
|
||||
"fake": true,
|
||||
"field": "spf_results.result.keyword",
|
||||
"field": "spf_results_combined.keyword",
|
||||
"id": "16",
|
||||
"settings": {
|
||||
"min_doc_count": "1",
|
||||
@@ -2626,7 +2626,7 @@
|
||||
{
|
||||
"$$hashKey": "object:461",
|
||||
"fake": true,
|
||||
"field": "dkim_results.result.keyword",
|
||||
"field": "dkim_results_combined.keyword",
|
||||
"id": "10",
|
||||
"settings": {
|
||||
"min_doc_count": "1",
|
||||
@@ -2666,7 +2666,7 @@
|
||||
"Sum": 11,
|
||||
"disposition.keyword": 4,
|
||||
"dkim_aligned": 6,
|
||||
"dkim_results.result.keyword": 9,
|
||||
"dkim_results_combined.keyword": 9,
|
||||
"header_from.keyword": 10,
|
||||
"org_name.keyword": 7,
|
||||
"source_base_domain.keyword": 0,
|
||||
@@ -2674,7 +2674,7 @@
|
||||
"source_ip_address.keyword": 2,
|
||||
"source_reverse_dns.keyword": 1,
|
||||
"spf_aligned": 5,
|
||||
"spf_results.result.keyword": 8
|
||||
"spf_results_combined.keyword": 8
|
||||
},
|
||||
"renameByName": {
|
||||
"Sum": "Messages",
|
||||
@@ -2682,7 +2682,7 @@
|
||||
"disposition.keyword": "Disposition",
|
||||
"dkim_aligned": "DKIM",
|
||||
"dkim_results.domain.keyword": "DKIM Domain",
|
||||
"dkim_results.result.keyword": "DKIM Auth Result",
|
||||
"dkim_results_combined.keyword": "DKIM Auth Result",
|
||||
"dkim_results.selector.keyword": "DKIM Selector",
|
||||
"envelope_from.keyword": "Envelope From",
|
||||
"header_from.keyword": "Email Domain",
|
||||
@@ -2692,7 +2692,7 @@
|
||||
"source_ip_address.keyword": "Source IP",
|
||||
"source_reverse_dns.keyword": "PTR",
|
||||
"spf_aligned": "SPF",
|
||||
"spf_results.result.keyword": "SPF Auth Result"
|
||||
"spf_results_combined.keyword": "SPF Auth Result"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -3431,7 +3431,7 @@
|
||||
{
|
||||
"$$hashKey": "object:459",
|
||||
"fake": true,
|
||||
"field": "spf_results.result.keyword",
|
||||
"field": "spf_results_combined.keyword",
|
||||
"id": "8",
|
||||
"settings": {
|
||||
"min_doc_count": 1,
|
||||
@@ -3502,7 +3502,7 @@
|
||||
"header_from.keyword": "Header From",
|
||||
"source_base_domain.keyword": "Reverse DNS Base",
|
||||
"spf_aligned": "SPF Aligned",
|
||||
"spf_results.result.keyword": "SPF Result"
|
||||
"spf_results_combined.keyword": "SPF Scope / Domain / Result"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -3590,22 +3590,12 @@
|
||||
{
|
||||
"matcher": {
|
||||
"id": "byName",
|
||||
"options": "DKIM Selector"
|
||||
"options": "DKIM Selector / Domain / Result"
|
||||
},
|
||||
"properties": [
|
||||
{
|
||||
"id": "custom.width",
|
||||
"value": 320
|
||||
},
|
||||
{
|
||||
"id": "links",
|
||||
"value": [
|
||||
{
|
||||
"targetBlank": true,
|
||||
"title": "Open dmarcian.com DKIM Record Checker",
|
||||
"url": "https://dmarcian.com/dkim-inspector/?domain=${__data.fields[\"dkim_results.domain.keyword\"]}&selector=${__data.fields[\"dkim_results.selector.keyword\"]}"
|
||||
}
|
||||
]
|
||||
"value": 400
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -3649,7 +3639,7 @@
|
||||
{
|
||||
"$$hashKey": "object:458",
|
||||
"fake": true,
|
||||
"field": "dkim_results.selector.keyword",
|
||||
"field": "dkim_results_combined.keyword",
|
||||
"id": "7",
|
||||
"settings": {
|
||||
"min_doc_count": "1",
|
||||
@@ -3660,34 +3650,6 @@
|
||||
},
|
||||
"type": "terms"
|
||||
},
|
||||
{
|
||||
"$$hashKey": "object:459",
|
||||
"fake": true,
|
||||
"field": "dkim_results.domain.keyword",
|
||||
"id": "8",
|
||||
"settings": {
|
||||
"min_doc_count": 1,
|
||||
"missing": "-",
|
||||
"order": "desc",
|
||||
"orderBy": "4",
|
||||
"size": "0"
|
||||
},
|
||||
"type": "terms"
|
||||
},
|
||||
{
|
||||
"$$hashKey": "object:460",
|
||||
"fake": true,
|
||||
"field": "dkim_results.result.keyword",
|
||||
"id": "9",
|
||||
"settings": {
|
||||
"min_doc_count": 1,
|
||||
"missing": null,
|
||||
"order": "desc",
|
||||
"orderBy": "4",
|
||||
"size": "0"
|
||||
},
|
||||
"type": "terms"
|
||||
},
|
||||
{
|
||||
"$$hashKey": "object:798",
|
||||
"fake": true,
|
||||
@@ -3744,9 +3706,7 @@
|
||||
"renameByName": {
|
||||
"Sum": "Messages",
|
||||
"dkim_aligned": "DKIM Aligned",
|
||||
"dkim_results.domain.keyword": "DKIM Domain",
|
||||
"dkim_results.result.keyword": "DKIM Result",
|
||||
"dkim_results.selector.keyword": "DKIM Selector",
|
||||
"dkim_results_combined.keyword": "DKIM Selector / Domain / Result",
|
||||
"envelope_from.keyword": "Envelope From",
|
||||
"header_from.keyword": "Header From",
|
||||
"source_base_domain.keyword": "Reverse DNS Base",
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -323,7 +323,12 @@
|
||||
<title>SPF details</title>
|
||||
<table>
|
||||
<search base="base_search">
|
||||
<query>| fillnull value="none" source_base_domain | stats sum(message_count) as message_count by header_from,envelope_from,spf_result,source_base_domain,spf_aligned
|
||||
<query>| fillnull value="none" source_base_domain envelope_domain spf_scope spf_result
|
||||
| eval spf_signature=mvzip(mvzip(spf_scope, envelope_domain, " / "), spf_result, " / ")
|
||||
| mvexpand spf_signature
|
||||
| stats sum(message_count) as message_count by header_from, envelope_from, spf_signature, source_base_domain, spf_aligned
|
||||
| eval parts=split(spf_signature, " / "), spf_scope=mvindex(parts, 0), spf_domain=mvindex(parts, 1), spf_result=mvindex(parts, 2)
|
||||
| table header_from, envelope_from, spf_scope, spf_domain, spf_result, spf_aligned, source_base_domain, message_count
|
||||
| sort -message_count</query>
|
||||
</search>
|
||||
<option name="drilldown">none</option>
|
||||
@@ -339,7 +344,12 @@
|
||||
<title>DKIM details</title>
|
||||
<table>
|
||||
<search base="base_search">
|
||||
<query>| fillnull value="none" source_base_domain | stats sum(message_count) as message_count by header_from,dkim_selector,dkim_domain,dkim_result,dkim_aligned,source_base_domain
|
||||
<query>| fillnull value="none" source_base_domain
|
||||
| eval dkim_signature=mvzip(mvzip(dkim_selector, dkim_domain, " / "), dkim_result, " / ")
|
||||
| mvexpand dkim_signature
|
||||
| stats sum(message_count) as message_count by header_from, dkim_signature, dkim_aligned, source_base_domain
|
||||
| eval parts=split(dkim_signature, " / "), dkim_selector=mvindex(parts, 0), dkim_domain=mvindex(parts, 1), dkim_result=mvindex(parts, 2)
|
||||
| table header_from, dkim_selector, dkim_domain, dkim_result, dkim_aligned, source_base_domain, message_count
|
||||
| sort -message_count</query>
|
||||
</search>
|
||||
<option name="drilldown">none</option>
|
||||
|
||||
@@ -226,6 +226,60 @@ Kibana index patterns with versions that match the upgraded indexes:
|
||||
7. Import `export.ndjson` by clicking Import from the Kibana
|
||||
Saved Objects page
|
||||
|
||||
## Backfilling the combined DKIM/SPF result fields
|
||||
|
||||
As of the version fixing [#169](https://github.com/domainaware/parsedmarc/issues/169),
|
||||
aggregate documents include `dkim_results_combined` and `spf_results_combined` —
|
||||
scalar string arrays that keep each auth result's selector/scope, domain, and
|
||||
result paired, which the dashboards' alignment-detail tables aggregate on.
|
||||
Reports saved by older versions lack these fields and will not appear in
|
||||
those tables.
|
||||
|
||||
Running the following once per cluster backfills the fields on existing
|
||||
documents. It is idempotent (documents that already have the fields are
|
||||
skipped), so it is safe to re-run. It works identically on OpenSearch;
|
||||
just adjust the URL and credentials. The query matches only documents
|
||||
that have at least one DKIM or SPF auth result and lack the corresponding
|
||||
combined field; documents with no auth results are skipped, because an
|
||||
`exists` query cannot see an empty array, and for search purposes an
|
||||
empty `dkim_results_combined` is identical to an absent one.
|
||||
|
||||
```bash
|
||||
curl -X POST "http://localhost:9200/dmarc_aggregate*/_update_by_query?conflicts=proceed&wait_for_completion=false" \
|
||||
-H "Content-Type: application/json" -d '
|
||||
{
|
||||
"query": {
|
||||
"bool": {
|
||||
"minimum_should_match": 1,
|
||||
"should": [
|
||||
{
|
||||
"bool": {
|
||||
"must": [{"exists": {"field": "dkim_results.domain"}}],
|
||||
"must_not": [{"exists": {"field": "dkim_results_combined"}}]
|
||||
}
|
||||
},
|
||||
{
|
||||
"bool": {
|
||||
"must": [{"exists": {"field": "spf_results.domain"}}],
|
||||
"must_not": [{"exists": {"field": "spf_results_combined"}}]
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
"script": {
|
||||
"lang": "painless",
|
||||
"source": "List dk = new ArrayList(); def dr = ctx._source.dkim_results; if (dr != null) { if (!(dr instanceof List)) { dr = [dr]; } for (e in dr) { if (e == null) { continue; } def sel = e.selector != null ? e.selector : \"none\"; def dom = e.domain != null ? e.domain : \"none\"; def res = e.result != null ? e.result : \"none\"; dk.add(sel + \" / \" + dom + \" / \" + res); } } ctx._source.dkim_results_combined = dk; List sp = new ArrayList(); def sr = ctx._source.spf_results; if (sr != null) { if (!(sr instanceof List)) { sr = [sr]; } for (e in sr) { if (e == null) { continue; } def sc = e.scope != null ? e.scope : \"mfrom\"; def dom = e.domain != null ? e.domain : \"none\"; def res = e.result != null ? e.result : (e.results != null ? e.results : \"none\"); sp.add(sc + \" / \" + dom + \" / \" + res); } } ctx._source.spf_results_combined = sp;"
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
`wait_for_completion=false` returns a task ID — check progress with
|
||||
`GET _tasks/<task id>`. Adjust the index pattern if you use a custom
|
||||
`index_prefix`/`index_suffix`. After backfilling, re-import the updated
|
||||
dashboards ndjson (the index pattern saved object changed too) per the
|
||||
import instructions above.
|
||||
|
||||
## Records retention
|
||||
|
||||
Starting in version 5.0.0, `parsedmarc` stores data in a separate
|
||||
|
||||
@@ -82,7 +82,11 @@ Further down the dashboard, you can filter by source country or source IP
|
||||
address.
|
||||
|
||||
Tables showing SPF and DKIM alignment details are located under the IP address
|
||||
table.
|
||||
table. Each row of the DKIM details table is one real DKIM signature, shown
|
||||
as a combined `selector / domain / result` value; the SPF details table
|
||||
shows `scope / domain / result` the same way. Combining the values into one
|
||||
column keeps each signature's selector, domain, and result paired together,
|
||||
rather than aggregating them as separate columns.
|
||||
|
||||
:::{note}
|
||||
The alignment tables (SPF details, DKIM details) and the per-IP source
|
||||
|
||||
+26
-3
@@ -99,7 +99,7 @@ class _SPFResult(InnerDoc):
|
||||
|
||||
domain = Text()
|
||||
scope = Text()
|
||||
results = Text()
|
||||
result = Text()
|
||||
human_result = Text()
|
||||
|
||||
|
||||
@@ -146,6 +146,18 @@ class _AggregateReportDoc(Document):
|
||||
envelope_to = Text()
|
||||
dkim_results = Nested(_DKIMResult)
|
||||
spf_results = Nested(_SPFResult)
|
||||
# One "{selector} / {domain} / {result}" (DKIM) or "{scope} / {domain} /
|
||||
# {result}" (SPF) string per auth result. Kibana/Grafana tables cannot
|
||||
# terms-aggregate the subfields of an object array without producing a
|
||||
# cross-product of values (issue #169), so dashboards aggregate these
|
||||
# composed keywords instead. Declared to match what dynamic mapping
|
||||
# produces for a string array (text + .keyword).
|
||||
dkim_results_combined = Text(
|
||||
multi=True, fields={"keyword": Keyword(ignore_above=256)}
|
||||
)
|
||||
spf_results_combined = Text(
|
||||
multi=True, fields={"keyword": Keyword(ignore_above=256)}
|
||||
)
|
||||
np = Keyword()
|
||||
testing = Keyword()
|
||||
discovery_method = Keyword()
|
||||
@@ -158,7 +170,7 @@ class _AggregateReportDoc(Document):
|
||||
self,
|
||||
domain: str,
|
||||
selector: str,
|
||||
result: _DKIMResult,
|
||||
result: str,
|
||||
human_result: str | None = None,
|
||||
):
|
||||
self.dkim_results.append(
|
||||
@@ -169,12 +181,15 @@ class _AggregateReportDoc(Document):
|
||||
human_result=human_result,
|
||||
)
|
||||
)
|
||||
self.dkim_results_combined.append(
|
||||
"{0} / {1} / {2}".format(selector, domain, result)
|
||||
)
|
||||
|
||||
def add_spf_result(
|
||||
self,
|
||||
domain: str,
|
||||
scope: str,
|
||||
result: _SPFResult,
|
||||
result: str,
|
||||
human_result: str | None = None,
|
||||
):
|
||||
self.spf_results.append(
|
||||
@@ -185,6 +200,9 @@ class _AggregateReportDoc(Document):
|
||||
human_result=human_result,
|
||||
)
|
||||
)
|
||||
self.spf_results_combined.append(
|
||||
"{0} / {1} / {2}".format(scope, domain, result)
|
||||
)
|
||||
|
||||
def save(self, **kwargs): # pyright: ignore[reportIncompatibleMethodOverride]
|
||||
self.passed_dmarc = False
|
||||
@@ -450,6 +468,11 @@ def create_indexes(names: list[str], settings: dict[str, Any] | None = None):
|
||||
for name in names:
|
||||
index = Index(name)
|
||||
try:
|
||||
# Deliberately no Index.document() registration: Kibana/OpenSearch
|
||||
# Dashboards/Grafana cannot terms-aggregate fields inside a
|
||||
# `nested` mapping, so the dynamic `object` mapping produced by a
|
||||
# bare create is load-bearing for the shipped dashboards (issue
|
||||
# #169; see the *_combined fields on _AggregateReportDoc).
|
||||
if not index.exists():
|
||||
logger.debug("Creating Elasticsearch index: {0}".format(name))
|
||||
if effective_settings:
|
||||
|
||||
@@ -62,7 +62,7 @@ class _DKIMResult(InnerDoc):
|
||||
class _SPFResult(InnerDoc):
|
||||
domain = Text()
|
||||
scope = Text()
|
||||
results = Text()
|
||||
result = Text()
|
||||
human_result = Text()
|
||||
|
||||
|
||||
@@ -103,6 +103,18 @@ class _AggregateReportDoc(Document):
|
||||
envelope_to = Text()
|
||||
dkim_results = Nested(_DKIMResult)
|
||||
spf_results = Nested(_SPFResult)
|
||||
# One "{selector} / {domain} / {result}" (DKIM) or "{scope} / {domain} /
|
||||
# {result}" (SPF) string per auth result. Kibana/Grafana tables cannot
|
||||
# terms-aggregate the subfields of an object array without producing a
|
||||
# cross-product of values (issue #169), so dashboards aggregate these
|
||||
# composed keywords instead. Declared to match what dynamic mapping
|
||||
# produces for a string array (text + .keyword).
|
||||
dkim_results_combined = Text(
|
||||
multi=True, fields={"keyword": Keyword(ignore_above=256)}
|
||||
)
|
||||
spf_results_combined = Text(
|
||||
multi=True, fields={"keyword": Keyword(ignore_above=256)}
|
||||
)
|
||||
np = Keyword()
|
||||
testing = Keyword()
|
||||
discovery_method = Keyword()
|
||||
@@ -115,7 +127,7 @@ class _AggregateReportDoc(Document):
|
||||
self,
|
||||
domain: str,
|
||||
selector: str,
|
||||
result: _DKIMResult,
|
||||
result: str,
|
||||
human_result: str | None = None,
|
||||
):
|
||||
self.dkim_results.append(
|
||||
@@ -126,12 +138,15 @@ class _AggregateReportDoc(Document):
|
||||
human_result=human_result,
|
||||
)
|
||||
)
|
||||
self.dkim_results_combined.append(
|
||||
"{0} / {1} / {2}".format(selector, domain, result)
|
||||
)
|
||||
|
||||
def add_spf_result(
|
||||
self,
|
||||
domain: str,
|
||||
scope: str,
|
||||
result: _SPFResult,
|
||||
result: str,
|
||||
human_result: str | None = None,
|
||||
):
|
||||
self.spf_results.append(
|
||||
@@ -142,6 +157,9 @@ class _AggregateReportDoc(Document):
|
||||
human_result=human_result,
|
||||
)
|
||||
)
|
||||
self.spf_results_combined.append(
|
||||
"{0} / {1} / {2}".format(scope, domain, result)
|
||||
)
|
||||
|
||||
def save(self, **kwargs): # pyright: ignore[reportIncompatibleMethodOverride]
|
||||
self.passed_dmarc = False
|
||||
@@ -367,6 +385,11 @@ def create_indexes(names: list[str], settings: dict[str, Any] | None = None):
|
||||
for name in names:
|
||||
index = Index(name)
|
||||
try:
|
||||
# Deliberately no Index.document() registration: Kibana/OpenSearch
|
||||
# Dashboards/Grafana cannot terms-aggregate fields inside a
|
||||
# `nested` mapping, so the dynamic `object` mapping produced by a
|
||||
# bare create is load-bearing for the shipped dashboards (issue
|
||||
# #169; see the *_combined fields on _AggregateReportDoc).
|
||||
if not index.exists():
|
||||
logger.debug("Creating OpenSearch index: {0}".format(name))
|
||||
if settings is None:
|
||||
|
||||
@@ -549,6 +549,54 @@ class TestSaveAggregateReport(unittest.TestCase):
|
||||
mock_doc_cls.call_args.kwargs["date_begin"].timestamp(), 1705276800
|
||||
)
|
||||
|
||||
def test_save_populates_combined_dkim_and_spf_fields(self):
|
||||
"""Regression guard for issue #169: two DKIM signatures on one
|
||||
record must yield exactly two combined entries, not a 4-way
|
||||
cross-product. autospec=True is required on the save patch so
|
||||
mock_save.call_args captures the doc instance as ``self``."""
|
||||
report = _aggregate_report()
|
||||
report["records"][0]["auth_results"] = {
|
||||
"dkim": [
|
||||
{
|
||||
"domain": "example.net",
|
||||
"selector": "net1",
|
||||
"result": "fail",
|
||||
"human_result": None,
|
||||
},
|
||||
{
|
||||
"domain": "example.org",
|
||||
"selector": "org1",
|
||||
"result": "pass",
|
||||
"human_result": None,
|
||||
},
|
||||
],
|
||||
"spf": [
|
||||
{
|
||||
"domain": "example.org",
|
||||
"scope": "mfrom",
|
||||
"result": "pass",
|
||||
"human_result": None,
|
||||
},
|
||||
],
|
||||
}
|
||||
with (
|
||||
patch("parsedmarc.elastic.Search", return_value=_empty_search()),
|
||||
patch(
|
||||
"parsedmarc.elastic.Index",
|
||||
return_value=MagicMock(exists=MagicMock(return_value=True)),
|
||||
),
|
||||
patch.object(
|
||||
elastic_module._AggregateReportDoc, "save", autospec=True
|
||||
) as mock_save,
|
||||
):
|
||||
save_aggregate_report_to_elasticsearch(report)
|
||||
doc = mock_save.call_args[0][0]
|
||||
self.assertEqual(
|
||||
list(doc.dkim_results_combined),
|
||||
["net1 / example.net / fail", "org1 / example.org / pass"],
|
||||
)
|
||||
self.assertEqual(list(doc.spf_results_combined), ["mfrom / example.org / pass"])
|
||||
|
||||
|
||||
class TestAggregateDocPassedDmarc(unittest.TestCase):
|
||||
"""The _AggregateReportDoc.save() override derives passed_dmarc — the
|
||||
@@ -576,6 +624,56 @@ class TestAggregateDocPassedDmarc(unittest.TestCase):
|
||||
self.assertEqual(bool(doc.passed_dmarc), expected)
|
||||
|
||||
|
||||
class TestAggregateDocCombinedResults(unittest.TestCase):
|
||||
"""add_dkim_result/add_spf_result never touch the network, so these
|
||||
construct _AggregateReportDoc directly rather than going through the
|
||||
save_* entry point."""
|
||||
|
||||
def test_add_dkim_result_appends_combined_string(self):
|
||||
"""Regression guard for issue #169: dkim_results/spf_results are
|
||||
stored as nested object arrays, which Kibana/Grafana tables cannot
|
||||
terms-aggregate without producing a cross-product of selector/
|
||||
domain/result values. The composed "selector / domain / result"
|
||||
string preserves per-signature pairing that the object-mapped
|
||||
array loses."""
|
||||
doc = elastic_module._AggregateReportDoc()
|
||||
doc.add_dkim_result(
|
||||
domain="example.net", selector="net1", result="fail", human_result=None
|
||||
)
|
||||
doc.add_dkim_result(
|
||||
domain="example.org", selector="org1", result="pass", human_result=None
|
||||
)
|
||||
expected = ["net1 / example.net / fail", "org1 / example.org / pass"]
|
||||
# dkim_results_combined is declared as Text(multi=True, ...); the SDK
|
||||
# stub types the class attribute as Text (no Iterable protocol),
|
||||
# even though the runtime value is an AttrList once multi=True is
|
||||
# set. Same category of stub gap as the Q()/meta.index ignores in
|
||||
# elastic.py.
|
||||
self.assertEqual(list(doc.dkim_results_combined), expected) # pyright: ignore[reportArgumentType]
|
||||
self.assertEqual(doc.to_dict()["dkim_results_combined"], expected)
|
||||
|
||||
def test_add_spf_result_appends_combined_string(self):
|
||||
doc = elastic_module._AggregateReportDoc()
|
||||
doc.add_spf_result(
|
||||
domain="example.org", scope="mfrom", result="pass", human_result=None
|
||||
)
|
||||
expected = ["mfrom / example.org / pass"]
|
||||
self.assertEqual(list(doc.spf_results_combined), expected) # pyright: ignore[reportArgumentType]
|
||||
self.assertEqual(doc.to_dict()["spf_results_combined"], expected)
|
||||
|
||||
def test_spf_result_serializes_under_singular_result_key(self):
|
||||
"""The _SPFResult class previously declared a dead ``results``
|
||||
(plural) field while the save path wrote ``result``; verify the
|
||||
serialized nested doc actually uses the singular key."""
|
||||
doc = elastic_module._AggregateReportDoc()
|
||||
doc.add_spf_result(
|
||||
domain="example.org", scope="mfrom", result="pass", human_result=None
|
||||
)
|
||||
d = doc.to_dict()["spf_results"][0]
|
||||
self.assertEqual(d["result"], "pass")
|
||||
self.assertNotIn("results", d)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# save_failure_report_to_elasticsearch
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -602,6 +602,54 @@ class TestSaveAggregateReport(unittest.TestCase):
|
||||
mock_doc_cls.call_args.kwargs["date_begin"].timestamp(), 1705276800
|
||||
)
|
||||
|
||||
def test_save_populates_combined_dkim_and_spf_fields(self):
|
||||
"""Regression guard for issue #169: two DKIM signatures on one
|
||||
record must yield exactly two combined entries, not a 4-way
|
||||
cross-product. autospec=True is required on the save patch so
|
||||
mock_save.call_args captures the doc instance as ``self``."""
|
||||
report = _aggregate_report()
|
||||
report["records"][0]["auth_results"] = {
|
||||
"dkim": [
|
||||
{
|
||||
"domain": "example.net",
|
||||
"selector": "net1",
|
||||
"result": "fail",
|
||||
"human_result": None,
|
||||
},
|
||||
{
|
||||
"domain": "example.org",
|
||||
"selector": "org1",
|
||||
"result": "pass",
|
||||
"human_result": None,
|
||||
},
|
||||
],
|
||||
"spf": [
|
||||
{
|
||||
"domain": "example.org",
|
||||
"scope": "mfrom",
|
||||
"result": "pass",
|
||||
"human_result": None,
|
||||
},
|
||||
],
|
||||
}
|
||||
with (
|
||||
patch("parsedmarc.opensearch.Search", return_value=_empty_search()),
|
||||
patch(
|
||||
"parsedmarc.opensearch.Index",
|
||||
return_value=MagicMock(exists=MagicMock(return_value=True)),
|
||||
),
|
||||
patch.object(
|
||||
opensearch_module._AggregateReportDoc, "save", autospec=True
|
||||
) as mock_save,
|
||||
):
|
||||
save_aggregate_report_to_opensearch(report)
|
||||
doc = mock_save.call_args[0][0]
|
||||
self.assertEqual(
|
||||
list(doc.dkim_results_combined),
|
||||
["net1 / example.net / fail", "org1 / example.org / pass"],
|
||||
)
|
||||
self.assertEqual(list(doc.spf_results_combined), ["mfrom / example.org / pass"])
|
||||
|
||||
|
||||
class TestAggregateDocPassedDmarc(unittest.TestCase):
|
||||
"""The _AggregateReportDoc.save() override derives passed_dmarc — the
|
||||
@@ -629,6 +677,55 @@ class TestAggregateDocPassedDmarc(unittest.TestCase):
|
||||
self.assertEqual(bool(doc.passed_dmarc), expected)
|
||||
|
||||
|
||||
class TestAggregateDocCombinedResults(unittest.TestCase):
|
||||
"""add_dkim_result/add_spf_result never touch the network, so these
|
||||
construct _AggregateReportDoc directly rather than going through the
|
||||
save_* entry point."""
|
||||
|
||||
def test_add_dkim_result_appends_combined_string(self):
|
||||
"""Regression guard for issue #169: dkim_results/spf_results are
|
||||
stored as nested object arrays, which Kibana/Grafana tables cannot
|
||||
terms-aggregate without producing a cross-product of selector/
|
||||
domain/result values. The composed "selector / domain / result"
|
||||
string preserves per-signature pairing that the object-mapped
|
||||
array loses."""
|
||||
doc = opensearch_module._AggregateReportDoc()
|
||||
doc.add_dkim_result(
|
||||
domain="example.net", selector="net1", result="fail", human_result=None
|
||||
)
|
||||
doc.add_dkim_result(
|
||||
domain="example.org", selector="org1", result="pass", human_result=None
|
||||
)
|
||||
expected = ["net1 / example.net / fail", "org1 / example.org / pass"]
|
||||
# dkim_results_combined is declared as Text(multi=True, ...); the SDK
|
||||
# stub types the class attribute as Text (no Iterable protocol),
|
||||
# even though the runtime value is an AttrList once multi=True is
|
||||
# set.
|
||||
self.assertEqual(list(doc.dkim_results_combined), expected) # pyright: ignore[reportArgumentType]
|
||||
self.assertEqual(doc.to_dict()["dkim_results_combined"], expected)
|
||||
|
||||
def test_add_spf_result_appends_combined_string(self):
|
||||
doc = opensearch_module._AggregateReportDoc()
|
||||
doc.add_spf_result(
|
||||
domain="example.org", scope="mfrom", result="pass", human_result=None
|
||||
)
|
||||
expected = ["mfrom / example.org / pass"]
|
||||
self.assertEqual(list(doc.spf_results_combined), expected) # pyright: ignore[reportArgumentType]
|
||||
self.assertEqual(doc.to_dict()["spf_results_combined"], expected)
|
||||
|
||||
def test_spf_result_serializes_under_singular_result_key(self):
|
||||
"""The _SPFResult class previously declared a dead ``results``
|
||||
(plural) field while the save path wrote ``result``; verify the
|
||||
serialized nested doc actually uses the singular key."""
|
||||
doc = opensearch_module._AggregateReportDoc()
|
||||
doc.add_spf_result(
|
||||
domain="example.org", scope="mfrom", result="pass", human_result=None
|
||||
)
|
||||
d = doc.to_dict()["spf_results"][0]
|
||||
self.assertEqual(d["result"], "pass")
|
||||
self.assertNotIn("results", d)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# save_failure_report_to_opensearch
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
Reference in New Issue
Block a user