mirror of
https://github.com/domainaware/parsedmarc.git
synced 2026-05-20 10:55:24 +00:00
Align Kibana dashboards with OpenSearch Dashboards source-of-truth (#737)
* Align Kibana dashboards with OpenSearch Dashboards source-of-truth
OSD is a fork of Kibana 7.10 and Kibana 8.x's saved-object migration
handlers accept OSD's saved-object format directly. Replace the legacy
Kibana export with a byte-identical copy of the OSD ndjson, so the two
backends ship the same panels, metric aggregations, panel titles, and
field assignments instead of drifting independently.
Verified against Kibana 8.19.7: import returns successCount=26 with no
errors and Kibana auto-migrates each viz / dashboard to its current
saved-object schema (typeMigrationVersion 8.5.0 for visualizations,
10.3.0 for dashboards) on import.
Net effects for Kibana users on import:
- Picks up the metric-aggregation fix from 9.10.3 — pies, tables, and
the choropleth now sum(message_count) instead of counting OS docs,
giving real message volume rather than distinct source-row counts.
- Adds "Message sources by Autonomous System" and "Message sources by
name and type" panels (previously only on OSD).
- Forensic dashboard simplified to OSD's two-panel layout (markdown
intro + samples table) — drops the Kibana-only IP-address and
country-ISO tables and the choropleth.
- Adds the "SMTP TLS reporting" dashboard (was absent from the bundled
Kibana export).
- Drops the extraneous "Evolution DMARC par source_reverse_DNS" Lens
visualization that snuck in via a community contribution.
Updates docs/source/kibana.md to reflect the new dashboard names
("DMARC aggregate reports" / "DMARC failure reports") and adds a brief
section on the SMTP TLS reporting dashboard.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Drop the duplicate Kibana ndjson; point Kibana users at the OSD file
Kibana 8.x's saved-object migration handlers accept the OpenSearch
Dashboards saved-object format directly (verified by import returning
successCount=26 with no errors), so a separate kibana/export.ndjson
was just two copies of the same bytes that would inevitably drift. Drop
it and update the bootstrap script and docs to point at the existing
dashboards/opensearch/opensearch_dashboards.ndjson.
Add a path-filtered CI workflow (.github/workflows/dashboards.yml) that
fires only when the OSD ndjson changes. It stands up an Elasticsearch +
Kibana 8.19.7 service pair, POSTs the file at the saved-objects import
endpoint, and asserts success=true with no errors. That keeps the
single-file source compatible with Kibana on every change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Sean Whalen
GitHub
@@ -0,0 +1,88 @@
|
||||
name: Validate dashboards
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
# Kibana 8.x's saved-object migration handlers accept the OpenSearch
|
||||
# Dashboards saved-object format directly, so we ship the OSD ndjson as the
|
||||
# single source for both backends. This workflow guards that compatibility:
|
||||
# any change to the OSD ndjson must still import cleanly into a Kibana 8.x
|
||||
# container before the change is mergeable.
|
||||
#
|
||||
# The job is path-filtered to only run when the ndjson itself changes —
|
||||
# every other PR skips it.
|
||||
on:
|
||||
push:
|
||||
branches: [master]
|
||||
paths: ['dashboards/opensearch/opensearch_dashboards.ndjson']
|
||||
pull_request:
|
||||
branches: [master]
|
||||
paths: ['dashboards/opensearch/opensearch_dashboards.ndjson']
|
||||
|
||||
jobs:
|
||||
kibana-import:
|
||||
name: Verify ndjson imports into Kibana 8.x
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
services:
|
||||
elasticsearch:
|
||||
image: docker.elastic.co/elasticsearch/elasticsearch:8.19.7
|
||||
env:
|
||||
discovery.type: single-node
|
||||
xpack.security.enabled: "false"
|
||||
xpack.license.self_generated.type: basic
|
||||
ES_JAVA_OPTS: "-Xms512m -Xmx512m"
|
||||
ports:
|
||||
- 9200:9200
|
||||
options: >-
|
||||
--health-cmd "curl -sf http://localhost:9200/_cluster/health"
|
||||
--health-interval 10s
|
||||
--health-timeout 5s
|
||||
--health-retries 24
|
||||
|
||||
kibana:
|
||||
image: docker.elastic.co/kibana/kibana:8.19.7
|
||||
env:
|
||||
ELASTICSEARCH_HOSTS: http://elasticsearch:9200
|
||||
ports:
|
||||
- 5601:5601
|
||||
options: >-
|
||||
--health-cmd "curl -sf http://localhost:5601/api/status"
|
||||
--health-interval 10s
|
||||
--health-timeout 5s
|
||||
--health-retries 30
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v5
|
||||
|
||||
- name: Wait for Kibana to report ready
|
||||
run: |
|
||||
for i in $(seq 1 60); do
|
||||
if curl -sf http://localhost:5601/api/status >/dev/null; then
|
||||
echo "Kibana is ready"
|
||||
exit 0
|
||||
fi
|
||||
sleep 5
|
||||
done
|
||||
echo "Kibana failed to come up within 5 minutes" >&2
|
||||
exit 1
|
||||
|
||||
- name: Import OSD ndjson and assert success
|
||||
run: |
|
||||
response=$(curl -sS -X POST \
|
||||
'http://localhost:5601/api/saved_objects/_import?overwrite=true' \
|
||||
-H 'kbn-xsrf: true' \
|
||||
--form file=@dashboards/opensearch/opensearch_dashboards.ndjson)
|
||||
echo "$response" | python3 -m json.tool
|
||||
echo "$response" | python3 - <<'PY'
|
||||
import json, sys
|
||||
d = json.load(sys.stdin)
|
||||
if not d.get("success"):
|
||||
sys.exit(f"Kibana import failed: {d}")
|
||||
if d.get("errors"):
|
||||
sys.exit(f"Kibana import had errors: {d['errors']}")
|
||||
n = d.get("successCount", 0)
|
||||
if n < 1:
|
||||
sys.exit(f"Expected at least 1 imported object, got {n}")
|
||||
print(f"OK: {n} saved objects imported and migrated by Kibana")
|
||||
PY
|
||||
@@ -246,7 +246,7 @@ fi
|
||||
log "Importing Kibana dashboards"
|
||||
curl -sS -X POST 'http://localhost:5601/api/saved_objects/_import?overwrite=true' \
|
||||
-H 'kbn-xsrf: true' \
|
||||
--form file=@dashboards/kibana/export.ndjson | sed 's/^/ /'
|
||||
--form file=@dashboards/opensearch/opensearch_dashboards.ndjson | sed 's/^/ /'
|
||||
|
||||
log "Importing OpenSearch Dashboards saved objects"
|
||||
# OSD with the security plugin enabled stores saved objects per tenant. Without
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -231,6 +231,6 @@ retention regulations such as GDPR. For more information,
|
||||
check out the Elastic guide to [managing time-based indexes efficiently](https://www.elastic.co/blog/managing-time-based-indices-efficiently).
|
||||
|
||||
[elasticsearch]: https://www.elastic.co/guide/en/elasticsearch/reference/current/rpm.html
|
||||
[export.ndjson]: https://raw.githubusercontent.com/domainaware/parsedmarc/master/dashboards/kibana/export.ndjson
|
||||
[export.ndjson]: https://raw.githubusercontent.com/domainaware/parsedmarc/master/dashboards/opensearch/opensearch_dashboards.ndjson
|
||||
[kibana]: https://www.elastic.co/guide/en/kibana/current/rpm.html
|
||||
[x-pack]: https://www.elastic.co/products/x-pack
|
||||
|
||||
+26
-10
@@ -4,12 +4,20 @@
|
||||
The Kibana DMARC dashboards are a human-friendly way to understand the
|
||||
results from incoming DMARC reports.
|
||||
|
||||
There is no separate Kibana export — Kibana 8.x's saved-object migration
|
||||
handlers accept the OpenSearch Dashboards format directly, so Kibana
|
||||
users import the bundled
|
||||
[`dashboards/opensearch/opensearch_dashboards.ndjson`](https://raw.githubusercontent.com/domainaware/parsedmarc/master/dashboards/opensearch/opensearch_dashboards.ndjson)
|
||||
in *Stack Management → Saved Objects → Import*. A CI check imports the
|
||||
same file into a Kibana 8.x container on every change so this stays
|
||||
compatible.
|
||||
|
||||
:::{note}
|
||||
The default dashboard is DMARC Summary. To switch between dashboards,
|
||||
click on the Dashboard link on the left side menu of Kibana.
|
||||
The default dashboard is DMARC aggregate reports. To switch between
|
||||
dashboards, click on the Dashboard link on the left side menu of Kibana.
|
||||
:::
|
||||
|
||||
## DMARC Summary
|
||||
## DMARC aggregate reports
|
||||
|
||||
As the name suggests, this dashboard is the best place to start
|
||||
reviewing your aggregate DMARC data.
|
||||
@@ -66,22 +74,30 @@ Tables showing SPF and DKIM alignment details are located under the IP address
|
||||
table.
|
||||
|
||||
:::{note}
|
||||
Previously, the alignment tables were included in a separate dashboard
|
||||
called DMARC Alignment Failures. That dashboard has been consolidated into
|
||||
the DMARC Summary dashboard. To view failures only, use the pie chart.
|
||||
The alignment tables (SPF details, DKIM details) and the per-IP source
|
||||
table live on the same dashboard, further down. To view failures only,
|
||||
use the pie chart at the top of the page as a filter.
|
||||
:::
|
||||
|
||||
Any other filters work the same way. You can also add your own custom temporary
|
||||
filters by clicking on Add Filter at the upper right of the page.
|
||||
|
||||
## DMARC Forensic Samples
|
||||
## DMARC failure reports
|
||||
|
||||
The DMARC Forensic Samples dashboard contains information on DMARC forensic
|
||||
reports (also known as failure reports or ruf reports). These reports contain
|
||||
samples of emails that have failed to pass DMARC.
|
||||
The DMARC failure reports dashboard (formerly DMARC Forensic Samples) contains
|
||||
information on DMARC failure reports (also known as forensic or ruf reports).
|
||||
These reports contain samples of emails that have failed to pass DMARC.
|
||||
|
||||
:::{note}
|
||||
Most recipients do not send forensic/failure/ruf reports at all to avoid
|
||||
privacy leaks. Some recipients (notably Chinese webmail services) will only
|
||||
supply the headers of sample emails. Very few provide the entire email.
|
||||
:::
|
||||
|
||||
## SMTP TLS reporting
|
||||
|
||||
The SMTP TLS reporting dashboard surfaces aggregate counts of TLS-RPT
|
||||
reporting organizations, the policy domains they report on, and the
|
||||
specific failure types — certificate expiry, STARTTLS not supported,
|
||||
STS policy fetch errors, validation failures, and similar — together with
|
||||
the sending and receiving MTA addresses involved.
|
||||
|
||||
Reference in New Issue
Block a user