mirror of
https://github.com/domainaware/parsedmarc.git
synced 2026-07-25 09:54:55 +00:00
* Add per-domain DMARC compliance percentage to all aggregate dashboards (#112)
The from-domain volume table on every provider's aggregate dashboard is
now "Message volume and DMARC compliance by from domain" with columns
From Domain | Messages | % DMARC Compliant:
- OpenSearch Dashboards/Kibana: the agg-based data table is replaced by
a TSVB table using a Filter Ratio metric (passed_dmarc:true over all,
sum of message_count), pivoted on header_from.keyword. The time field
is date_begin rather than the multi-valued date_range, which TSVB's
per-value date histogram would double-count. Editing (not rendering)
the panel on Kibana 8.x requires the metrics:allowStringIndices
advanced setting.
- Grafana (Elasticsearch): a second passed_dmarc:true query joined by
field with a binary calculation (Sum 2 / Sum 1) rendered as percentunit.
- Grafana (PostgreSQL): compliance column via an aggregate FILTER clause,
COALESCEd so zero-pass domains show 0 instead of NULL.
- Splunk: sum(eval(if(passed_dmarc="true", message_count, 0))) inside
stats, per the SPL eval-in-stats syntax.
All four providers were verified against the same seeded sample data in
the dashboard dev stack; each returns identical per-domain values
(example.com: 2425 messages, 5.3% compliant).
Dev stack fixes found along the way: cap Elasticsearch heap at 2g (the
unset heap auto-sized to 50% of host RAM and was OOM-killed with
bootstrap.memory_lock on large hosts), and install the elasticsearch
datasource plugin in Grafana, which is no longer bundled as of
Grafana 13.
Closes #112
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix over-time charts double-counting reports via multi-valued date_range
date_range on ES/OpenSearch aggregate and SMTP TLS documents is a
two-element array [begin, end]. A date histogram buckets a document once
per value, so every over-time chart bucketing on date_range counted a
report twice whenever its begin and end dates fell in different buckets.
Range filtering on it was also wrong: a report spanning the whole window
matches neither endpoint.
Measured on the dev-stack sample data: a 1d histogram on date_range
returns doc_count 4592 / message sum 4724 against true totals of
2300 / 2427; the same histogram on date_begin returns exactly
2300 / 2427.
All date histograms (2 OSD/Kibana visualizations, 10 Grafana ES panels
including the summary pies) and all time-range filters (24 Grafana
target timeFields, the dmarc_aggregate* and smtp_tls* index-pattern
timeFieldName, the dev-stack dmarc-ag datasource) now use the
single-valued date_begin, matching the report-begin semantics of the
PostgreSQL (begin_date) and Splunk (_time = interval begin) dashboards.
Failure-report panels already used the single-valued arrival_date and
are unchanged.
Dev stack: installing the Elasticsearch datasource plugin via
GF_INSTALL_PLUGINS crash-loops Grafana >= 13 (the image ships a
root-owned plugins-bundled/elasticsearch remnant the background
installer cannot replace), so the bootstrap script now installs it via
grafana cli and restarts Grafana instead.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address Copilot review comments on PR #834
- kibana.md: "filter on our filter out" -> "filter on or filter out".
- OSD/Kibana export: fix "filed DMARC" -> "failed DMARC" and the
backticked `ruf ` trailing space in the RUF explainer panel, and
rename the "SMPT TLS failure details" visualization to "SMTP TLS
failure details" (object title and visState).
- dashboard-dev-bootstrap.sh: reuse wait_for() after the Grafana
plugin-install restart so a hang fails with a clear timeout message
instead of an opaque downstream curl error.
The ndjson changes were round-tripped through the dev-stack OSD
(import -> re-export from the global tenant) and re-import cleanly into
both OSD and Kibana 8.19.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* AGENTS.md: reviews must cover prose and hunk context, not just function
Codifies the lessons from the PR #834 Copilot review: whole-file
canonical dashboard exports put pre-existing titles/markdown in the
diff, so they get a text-level pass; proofread the full hunk around
prose edits, not only changed lines; and mid-incident glue code gets
the same review bar (and helper-reuse check) as planned code.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address second round of Copilot review comments
- CHANGELOG.md: rename the premature "10.2.5" heading to "Unreleased",
matching the repo convention where the release commit assigns the
version number (see 855d267 for 10.2.4).
- docker-compose.yml: make the dev-stack Elasticsearch heap overridable
via ES_JAVA_OPTS in .env (default unchanged at 2g), using the compose
file's existing ${VAR:-default} idiom, for smaller machines.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
507 lines
23 KiB
Bash
Executable File
507 lines
23 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Bring up docker-compose.dashboard-dev.yml, import the latest parsedmarc
|
|
# dashboards into each viz system, and seed each backend with sample data so
|
|
# the dashboards have something to render. Idempotent — safe to re-run.
|
|
|
|
set -euo pipefail
|
|
|
|
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
cd "$REPO_ROOT"
|
|
|
|
COMPOSE=(docker compose -f docker-compose.dashboard-dev.yml --env-file .env)
|
|
|
|
# Load .env so this script can use the same secrets compose injects.
|
|
set -a
|
|
# shellcheck disable=SC1091
|
|
. .env
|
|
set +a
|
|
|
|
GRAFANA_USER="${GRAFANA_USER:-admin}"
|
|
GRAFANA_PASSWORD="${GRAFANA_PASSWORD:-admin}"
|
|
|
|
# PostgreSQL dev credentials. Defaults match docker-compose.dashboard-dev.yml's
|
|
# ${POSTGRESQL_*:-parsedmarc} fallbacks; override all four in lockstep via .env.
|
|
PG_USER="${POSTGRESQL_USER:-parsedmarc}"
|
|
PG_PASSWORD="${POSTGRESQL_PASSWORD:-parsedmarc}"
|
|
PG_DB="${POSTGRESQL_DB:-parsedmarc}"
|
|
|
|
log() { printf '\n\033[1;36m== %s\033[0m\n' "$*"; }
|
|
|
|
wait_for() {
|
|
local name="$1"; shift
|
|
local max="${WAIT_TIMEOUT:-180}"
|
|
local i=0
|
|
printf 'Waiting for %s' "$name"
|
|
while ! "$@" >/dev/null 2>&1; do
|
|
printf '.'
|
|
i=$((i + 1))
|
|
if [ "$i" -ge "$max" ]; then
|
|
printf '\n'
|
|
echo "ERROR: $name not ready after ${max}s" >&2
|
|
return 1
|
|
fi
|
|
sleep 1
|
|
done
|
|
printf ' ready\n'
|
|
}
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 1. Bring up the stack
|
|
# ---------------------------------------------------------------------------
|
|
log "Starting docker compose dashboard-dev stack"
|
|
"${COMPOSE[@]}" up -d
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 2. Wait for each service
|
|
# ---------------------------------------------------------------------------
|
|
log "Waiting for backends and UIs"
|
|
wait_for "Elasticsearch" \
|
|
curl -sf 'http://localhost:9200/_cluster/health?wait_for_status=yellow&timeout=1s'
|
|
wait_for "OpenSearch" \
|
|
curl -ksf -u "admin:${OPENSEARCH_INITIAL_ADMIN_PASSWORD}" \
|
|
'https://localhost:9201/_cluster/health?wait_for_status=yellow&timeout=1s'
|
|
wait_for "Kibana" curl -sf http://localhost:5601/api/status
|
|
wait_for "OpenSearch Dashboards" \
|
|
curl -ksf -u "admin:${OPENSEARCH_INITIAL_ADMIN_PASSWORD}" \
|
|
http://localhost:5602/api/status
|
|
wait_for "Grafana" curl -sf http://localhost:3000/api/health
|
|
wait_for "PostgreSQL" \
|
|
"${COMPOSE[@]}" exec -T postgresql pg_isready -U "$PG_USER" -d "$PG_DB"
|
|
# Splunk's HEC port is healthy once management API is up too.
|
|
wait_for "Splunk HEC" curl -ksf https://localhost:8088/services/collector/health
|
|
# Splunkd management API (used for dashboard imports) lives inside the container.
|
|
wait_for "Splunk management API" \
|
|
"${COMPOSE[@]}" exec -T splunk \
|
|
curl -ksf -u "admin:${SPLUNK_PASSWORD}" https://localhost:8089/services/server/info
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 3. Provision Splunk: index, app, HEC token allow-list
|
|
# Must run before sample-data ingestion. The Splunk image auto-creates a
|
|
# HEC token from SPLUNK_HEC_TOKEN, but with `indexes=[]` and
|
|
# `index=default` — writes to the parsedmarc-dev.ini `email` index will
|
|
# silently drop until both the index exists and the token allows it.
|
|
# ---------------------------------------------------------------------------
|
|
log "Provisioning Splunk index, app, and HEC token"
|
|
splunk_curl() {
|
|
"${COMPOSE[@]}" exec -T splunk \
|
|
curl -ksS -u "admin:${SPLUNK_PASSWORD}" "$@"
|
|
}
|
|
splunk_exists() {
|
|
# 200 if the named entity exists, 404 if not.
|
|
local code
|
|
code=$(splunk_curl -o /dev/null -w "%{http_code}" -X GET "$1") || true
|
|
[ "$code" = "200" ]
|
|
}
|
|
|
|
if splunk_exists https://localhost:8089/services/data/indexes/email; then
|
|
echo " index 'email' already exists — skipping"
|
|
else
|
|
splunk_curl -X POST https://localhost:8089/services/data/indexes \
|
|
-d name=email -d datatype=event >/dev/null
|
|
echo " created index 'email'"
|
|
fi
|
|
|
|
if splunk_exists https://localhost:8089/services/apps/local/DMARC; then
|
|
echo " app 'DMARC' already exists — skipping"
|
|
else
|
|
splunk_curl -X POST https://localhost:8089/services/apps/local \
|
|
-d name=DMARC -d label=DMARC -d visible=true >/dev/null
|
|
echo " created app 'DMARC'"
|
|
fi
|
|
|
|
# The auto-created HEC token is named "splunk_hec_token". Allow the email
|
|
# index and set it as the token default so parsedmarc-dev.ini's `index = email`
|
|
# is honoured. Skip the rewrite if the token already allows email and
|
|
# defaults to it.
|
|
HEC_STATE=$(splunk_curl -X GET \
|
|
"https://localhost:8089/servicesNS/admin/splunk_httpinput/data/inputs/http/splunk_hec_token?output_mode=json" \
|
|
2>/dev/null \
|
|
| python3 -c '
|
|
import json, sys
|
|
e = json.load(sys.stdin)["entry"][0]["content"]
|
|
indexes = e.get("indexes") or []
|
|
disabled = "1" if e.get("disabled") else "0"
|
|
print("|".join([e.get("index") or "", disabled, ",".join(indexes)]))
|
|
' 2>/dev/null || echo "||")
|
|
HEC_DEFAULT_INDEX="${HEC_STATE%%|*}"
|
|
HEC_REST="${HEC_STATE#*|}"
|
|
HEC_DISABLED="${HEC_REST%%|*}"
|
|
HEC_INDEXES=",${HEC_REST#*|},"
|
|
if [ "$HEC_DEFAULT_INDEX" = "email" ] && [ "$HEC_DISABLED" = "0" ] && [[ "$HEC_INDEXES" == *,email,* ]]; then
|
|
echo " HEC token 'splunk_hec_token' already configured — skipping"
|
|
else
|
|
splunk_curl -X POST \
|
|
"https://localhost:8089/servicesNS/admin/splunk_httpinput/data/inputs/http/splunk_hec_token" \
|
|
-d "indexes=email,main" \
|
|
-d "index=email" \
|
|
-d "disabled=0" \
|
|
>/dev/null
|
|
echo " reconfigured HEC token 'splunk_hec_token' (index=email, indexes=email,main)"
|
|
fi
|
|
|
|
# Make sure the HEC listener itself is enabled. Splunk treats this as a no-op
|
|
# if it's already enabled, so just send it once each run — no point checking.
|
|
splunk_curl -X POST \
|
|
"https://localhost:8089/servicesNS/admin/splunk_httpinput/data/inputs/http/http" \
|
|
-d "disabled=0" \
|
|
>/dev/null 2>&1 || true
|
|
|
|
# Splunk ships an in-product announcement view ("Scheduled export is now
|
|
# available for Dashboard Studio") in the search app with sharing=global, so
|
|
# it appears in the dashboards list of every app — including DMARC. Views
|
|
# don't support a `disabled` flag, but narrowing the sharing from `global` to
|
|
# `app` keeps it scoped to the search app only.
|
|
SCHED_SHARING=$(splunk_curl -X GET \
|
|
"https://localhost:8089/servicesNS/-/search/data/ui/views/scheduled_export_dashboard?output_mode=json" \
|
|
2>/dev/null \
|
|
| python3 -c '
|
|
import json, sys
|
|
print(json.load(sys.stdin)["entry"][0]["acl"].get("sharing", ""))
|
|
' 2>/dev/null || echo "")
|
|
if [ "$SCHED_SHARING" = "global" ]; then
|
|
splunk_curl -X POST \
|
|
"https://localhost:8089/servicesNS/nobody/search/data/ui/views/scheduled_export_dashboard/acl" \
|
|
-d "sharing=app" -d "owner=nobody" \
|
|
>/dev/null
|
|
echo " scoped 'scheduled_export_dashboard' to search app (was global)"
|
|
elif [ -n "$SCHED_SHARING" ]; then
|
|
echo " 'scheduled_export_dashboard' already scoped (sharing=${SCHED_SHARING}) — skipping"
|
|
fi
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 4. Seed sample data via parsedmarc -> ES, OS, Splunk HEC
|
|
# Skipped when ES already has aggregate docs from a prior run. Set
|
|
# RESEED=1 to wipe ES/OS/Splunk parsedmarc data first and re-seed.
|
|
# ---------------------------------------------------------------------------
|
|
log "Seeding sample data with parsedmarc-dev.ini"
|
|
ES_AGG_COUNT=$(curl -sf 'http://localhost:9200/dmarc_aggregate*/_count' 2>/dev/null \
|
|
| python3 -c 'import json,sys; print(json.load(sys.stdin).get("count", 0))' 2>/dev/null \
|
|
|| echo 0)
|
|
if [ "${RESEED:-0}" != "1" ] && [ "$ES_AGG_COUNT" -gt 0 ]; then
|
|
echo " ES already has $ES_AGG_COUNT aggregate docs — skipping seed (RESEED=1 to force)"
|
|
else
|
|
if [ "${RESEED:-0}" = "1" ] && [ "$ES_AGG_COUNT" -gt 0 ]; then
|
|
echo " RESEED=1: wiping existing parsedmarc data from all backends"
|
|
# ES 8.x rejects wildcard DELETEs by default
|
|
# (action.destructive_requires_name=true). Enumerate the daily indexes
|
|
# parsedmarc rolls (dmarc_aggregate-YYYY-MM-DD, dmarc_failure-...,
|
|
# smtp_tls-...) and DELETE each one explicitly. dmarc_forensic-* is the
|
|
# pre-rename failure index family, kept here so RESEED clears old data.
|
|
for prefix in dmarc_aggregate dmarc_failure dmarc_forensic smtp_tls; do
|
|
for idx in $(curl -sf "http://localhost:9200/_cat/indices/${prefix}*?h=index" 2>/dev/null); do
|
|
curl -sS -X DELETE "http://localhost:9200/${idx}" >/dev/null 2>&1 || true
|
|
done
|
|
for idx in $(curl -ksf -u "admin:${OPENSEARCH_INITIAL_ADMIN_PASSWORD}" "https://localhost:9201/_cat/indices/${prefix}*?h=index" 2>/dev/null); do
|
|
curl -ksS -u "admin:${OPENSEARCH_INITIAL_ADMIN_PASSWORD}" \
|
|
-X DELETE "https://localhost:9201/${idx}" >/dev/null 2>&1 || true
|
|
done
|
|
done
|
|
# Splunk has no clean-in-place REST endpoint for live indexes. The
|
|
# standard pattern is to delete and recreate. Settings carry over from
|
|
# the recreate POST below — datatype=event is what parsedmarc HEC needs.
|
|
splunk_curl -X DELETE \
|
|
"https://localhost:8089/services/data/indexes/email" >/dev/null 2>&1 || true
|
|
for _ in 1 2 3 4 5 6 7 8 9 10; do
|
|
splunk_exists https://localhost:8089/services/data/indexes/email || break
|
|
sleep 1
|
|
done
|
|
splunk_curl -X POST https://localhost:8089/services/data/indexes \
|
|
-d name=email -d datatype=event >/dev/null
|
|
# Recreate forces the HEC token allow-list to re-resolve against the
|
|
# new index. Re-apply the token config so the next seed lands.
|
|
splunk_curl -X POST \
|
|
"https://localhost:8089/servicesNS/admin/splunk_httpinput/data/inputs/http/splunk_hec_token" \
|
|
-d "indexes=email,main" -d "index=email" -d "disabled=0" >/dev/null
|
|
# PostgreSQL: drop and recreate the public schema. parsedmarc recreates
|
|
# its tables on the next seed run, so this is a clean wipe.
|
|
"${COMPOSE[@]}" exec -T -e PGPASSWORD="$PG_PASSWORD" postgresql \
|
|
psql -U "$PG_USER" -d "$PG_DB" \
|
|
-c 'DROP SCHEMA public CASCADE; CREATE SCHEMA public;' >/dev/null 2>&1 || true
|
|
fi
|
|
|
|
# Resolve a Python environment for the seed and make sure parsedmarc plus
|
|
# the PostgreSQL extra (psycopg) are installed in it, so the same run can
|
|
# populate Postgres. Precedence:
|
|
# 1. An explicit PARSEDMARC_BIN — used as-is, nothing installed.
|
|
# 2. An already-activated virtualenv ($VIRTUAL_ENV).
|
|
# 3. An existing repo venv/ or .venv/.
|
|
# 4. Otherwise a freshly created $REPO_ROOT/venv.
|
|
# Cases 2-4 run `pip install -e .[postgresql]` only when the CLI or psycopg
|
|
# is missing, so it's a no-op once the environment is set up.
|
|
if [ -n "${PARSEDMARC_BIN:-}" ]; then
|
|
if [ ! -x "$PARSEDMARC_BIN" ]; then
|
|
echo "ERROR: PARSEDMARC_BIN is set but not executable: $PARSEDMARC_BIN" >&2
|
|
exit 1
|
|
fi
|
|
echo " using PARSEDMARC_BIN: $PARSEDMARC_BIN"
|
|
else
|
|
if [ -n "${VIRTUAL_ENV:-}" ]; then
|
|
seed_venv="$VIRTUAL_ENV"
|
|
echo " using active virtualenv: $seed_venv"
|
|
elif [ -d "$REPO_ROOT/venv" ]; then
|
|
seed_venv="$REPO_ROOT/venv"
|
|
echo " using existing venv: $seed_venv"
|
|
elif [ -d "$REPO_ROOT/.venv" ]; then
|
|
seed_venv="$REPO_ROOT/.venv"
|
|
echo " using existing .venv: $seed_venv"
|
|
else
|
|
seed_venv="$REPO_ROOT/venv"
|
|
echo " creating virtualenv: $seed_venv"
|
|
python3 -m venv "$seed_venv"
|
|
fi
|
|
PARSEDMARC_BIN="$seed_venv/bin/parsedmarc"
|
|
if [ ! -x "$PARSEDMARC_BIN" ] ||
|
|
! "$seed_venv/bin/python" -c 'import psycopg' >/dev/null 2>&1; then
|
|
echo " installing parsedmarc[postgresql] into $seed_venv"
|
|
"$seed_venv/bin/python" -m pip install -q -e "${REPO_ROOT}[postgresql]"
|
|
fi
|
|
fi
|
|
if [ ! -x "$PARSEDMARC_BIN" ]; then
|
|
echo "ERROR: parsedmarc CLI not found at $PARSEDMARC_BIN" >&2
|
|
exit 1
|
|
fi
|
|
|
|
# Live DNS lookups (no --offline) so source_reverse_dns / source_base_domain
|
|
# are populated. Many samples carry synthetic IPs (10.x, 198.51.100.x,
|
|
# 2001:db8::, etc.) that won't resolve, so cap retries/timeout to bound
|
|
# the cost of those NXDOMAIN-bound lookups. Intentionally invalid samples
|
|
# (empty_reason.xml, invalid_xml.xml, etc.) are skipped from the list.
|
|
SAMPLE_FILES=(
|
|
samples/aggregate/!example.com!1538204542!1538463818.xml
|
|
samples/aggregate/!large-example.com!1711897200!1711983600.xml
|
|
'samples/aggregate/Report domain- borschow.com Submitter- google.com Report-ID- 949348866075514174.eml'
|
|
samples/aggregate/addisonfoods.com!example.com!1536105600!1536191999.xml
|
|
samples/aggregate/estadocuenta1.infonacot.gob.mx!example.com!1536853302!1536939702!2940.xml.zip
|
|
samples/aggregate/example.net!example.com!1529366400!1529452799.xml
|
|
samples/aggregate/fastmail.com!example.com!1516060800!1516147199!102675056.xml.gz
|
|
samples/aggregate/ikea.com!example.de!1538690400!1538776800.xml
|
|
samples/aggregate/protection.outlook.com!example.com!1711756800!1711843200.xml
|
|
samples/aggregate/usssa.com!example.com!1538784000!1538870399.xml
|
|
samples/aggregate/veeam.com!example.com!1530133200!1530219600.xml
|
|
samples/aggregate/rfc9990-sample.xml
|
|
samples/aggregate/rfc9990-example.net!example.com!1700000000!1700086399.xml
|
|
samples/failure/*.eml
|
|
samples/smtp_tls/*.json
|
|
samples/smtp_tls/google.com_smtp_tls_report.eml
|
|
)
|
|
# PostgreSQL config is injected via env vars (parsedmarc synthesizes the
|
|
# [postgresql] section from PARSEDMARC_POSTGRESQL_*), so the same seed run
|
|
# also populates Postgres without touching the gitignored parsedmarc-dev.ini.
|
|
# Only wire it in when psycopg is importable: parsedmarc aborts the whole
|
|
# run (exit 1, nothing written to *any* backend) if a configured output
|
|
# backend can't initialize, so a missing optional extra must not be added.
|
|
pg_seed_env=()
|
|
seed_python="$(dirname "$PARSEDMARC_BIN")/python"
|
|
if [ -x "$seed_python" ] && "$seed_python" -c 'import psycopg' >/dev/null 2>&1; then
|
|
pg_seed_env=(
|
|
PARSEDMARC_POSTGRESQL_HOST=localhost
|
|
PARSEDMARC_POSTGRESQL_PORT=5432
|
|
PARSEDMARC_POSTGRESQL_USER="$PG_USER"
|
|
PARSEDMARC_POSTGRESQL_PASSWORD="$PG_PASSWORD"
|
|
PARSEDMARC_POSTGRESQL_DATABASE="$PG_DB"
|
|
)
|
|
else
|
|
# Reached only for an explicit PARSEDMARC_BIN whose env lacks psycopg
|
|
# (the auto-resolved venv path installs the extra above).
|
|
echo " NOTE: 'psycopg' is not available to ${PARSEDMARC_BIN} — skipping the"
|
|
echo " PostgreSQL seed. Enable it with: pip install -e '.[postgresql]'"
|
|
fi
|
|
env "${pg_seed_env[@]}" \
|
|
"$PARSEDMARC_BIN" -t 2.0 --dns-retries 1 -c parsedmarc-dev.ini "${SAMPLE_FILES[@]}" || true
|
|
fi
|
|
|
|
# ---------------------------------------------------------------------------
|
|
# 5. Import dashboards. Always re-imported on every run — that's the point of
|
|
# invoking this script after editing a dashboard. Datasources are checked
|
|
# first and skipped when already present.
|
|
# ---------------------------------------------------------------------------
|
|
log "Importing Kibana dashboards"
|
|
curl -sS -X POST 'http://localhost:5601/api/saved_objects/_import?overwrite=true' \
|
|
-H 'kbn-xsrf: true' \
|
|
--form file=@dashboards/opensearch/opensearch_dashboards.ndjson | sed 's/^/ /'
|
|
|
|
log "Importing OpenSearch Dashboards saved objects"
|
|
# OSD with the security plugin enabled stores saved objects per tenant. Without
|
|
# a securitytenant header the import lands in the API user's *private* tenant,
|
|
# which is invisible to anyone else (and to the same user when their browser
|
|
# session is on a different tenant). Target the Global tenant — the shared
|
|
# workspace every user has access to and where public dashboards conventionally
|
|
# live. Its securitytenant token is the literal "global"; any *other* string is
|
|
# treated as a custom tenant name, so "global_tenant" would silently create a
|
|
# separate "global_tenant" tenant rather than hit Global. (An empty/omitted
|
|
# header is *not* equivalent — it falls back to the user's configured default
|
|
# tenant, not Global.) To send the import elsewhere set OSD_TENANT=admin_tenant
|
|
# (or any other tenant name) before running.
|
|
OSD_TENANT="${OSD_TENANT:-global}"
|
|
curl -sS -X POST 'http://localhost:5602/api/saved_objects/_import?overwrite=true' \
|
|
-H 'osd-xsrf: true' \
|
|
-H "securitytenant: ${OSD_TENANT}" \
|
|
-u "admin:${OPENSEARCH_INITIAL_ADMIN_PASSWORD}" \
|
|
--form file=@dashboards/opensearch/opensearch_dashboards.ndjson | sed 's/^/ /'
|
|
echo " (imported into OSD tenant: ${OSD_TENANT})"
|
|
|
|
log "Ensuring Grafana Elasticsearch datasource plugin is installed"
|
|
# Grafana >= 13 no longer bundles the Elasticsearch datasource plugin, and
|
|
# GF_INSTALL_PLUGINS cannot install it (the image ships a root-owned
|
|
# plugins-bundled/elasticsearch remnant its background installer fails to
|
|
# replace). `grafana cli` installs into /var/lib/grafana/plugins, which works;
|
|
# a restart is needed for Grafana to load it.
|
|
code=$(curl -sS -u "${GRAFANA_USER}:${GRAFANA_PASSWORD}" \
|
|
-o /dev/null -w "%{http_code}" \
|
|
"http://localhost:3000/api/plugins/elasticsearch/settings")
|
|
if [ "$code" != "200" ]; then
|
|
"${COMPOSE[@]}" exec -T grafana grafana cli plugins install elasticsearch \
|
|
| sed 's/^/ /'
|
|
"${COMPOSE[@]}" restart grafana >/dev/null
|
|
wait_for "Grafana (after plugin install)" \
|
|
curl -sf http://localhost:3000/api/health
|
|
code=$(curl -sS -u "${GRAFANA_USER}:${GRAFANA_PASSWORD}" \
|
|
-o /dev/null -w "%{http_code}" \
|
|
"http://localhost:3000/api/plugins/elasticsearch/settings")
|
|
if [ "$code" != "200" ]; then
|
|
echo "ERROR: elasticsearch datasource plugin failed to install" >&2
|
|
exit 1
|
|
fi
|
|
echo " installed elasticsearch datasource plugin"
|
|
else
|
|
echo " elasticsearch datasource plugin already installed"
|
|
fi
|
|
|
|
log "Configuring Grafana datasources"
|
|
# Two Elasticsearch datasources, one per index family, matching the dashboard's
|
|
# template variables (dmarc-ag and dmarc-fo). Skipped when already present.
|
|
declare -a GF_DS_NAMES=("dmarc-ag" "dmarc-fo")
|
|
# dmarc_f* matches both pre-rename dmarc_forensic* and post-rename
|
|
# dmarc_failure* indices, mirroring the OpenSearch/Kibana dashboards.
|
|
declare -a GF_DS_INDEX=("dmarc_aggregate*" "dmarc_f*")
|
|
declare -a GF_DS_TIME=("date_begin" "arrival_date")
|
|
for i in 0 1; do
|
|
name="${GF_DS_NAMES[$i]}"
|
|
code=$(curl -sS -u "${GRAFANA_USER}:${GRAFANA_PASSWORD}" \
|
|
-o /dev/null -w "%{http_code}" \
|
|
"http://localhost:3000/api/datasources/name/${name}")
|
|
if [ "$code" = "200" ]; then
|
|
echo " datasource '${name}' already exists — skipping"
|
|
continue
|
|
fi
|
|
body=$(cat <<EOF
|
|
{
|
|
"name": "${name}",
|
|
"type": "elasticsearch",
|
|
"url": "http://elasticsearch:9200",
|
|
"access": "proxy",
|
|
"database": "${GF_DS_INDEX[$i]}",
|
|
"isDefault": false,
|
|
"jsonData": {
|
|
"esVersion": "8.0.0",
|
|
"timeField": "${GF_DS_TIME[$i]}",
|
|
"maxConcurrentShardRequests": 5
|
|
}
|
|
}
|
|
EOF
|
|
)
|
|
curl -sS -u "${GRAFANA_USER}:${GRAFANA_PASSWORD}" \
|
|
-H 'Content-Type: application/json' \
|
|
-X POST "http://localhost:3000/api/datasources" \
|
|
-d "$body" | sed 's/^/ /'
|
|
echo
|
|
echo " created datasource '${name}'"
|
|
done
|
|
|
|
# PostgreSQL datasource for the PostgreSQL DMARC dashboard. Fixed uid dmarc-pg
|
|
# so the dashboard import below can resolve its ${DS_POSTGRESQL} input. Skipped
|
|
# when already present.
|
|
pg_ds_code=$(curl -sS -u "${GRAFANA_USER}:${GRAFANA_PASSWORD}" \
|
|
-o /dev/null -w "%{http_code}" \
|
|
"http://localhost:3000/api/datasources/name/PostgreSQL")
|
|
if [ "$pg_ds_code" = "200" ]; then
|
|
echo " datasource 'PostgreSQL' already exists — skipping"
|
|
else
|
|
pg_ds_body=$(cat <<EOF
|
|
{
|
|
"name": "PostgreSQL",
|
|
"uid": "dmarc-pg",
|
|
"type": "grafana-postgresql-datasource",
|
|
"url": "postgresql:5432",
|
|
"access": "proxy",
|
|
"user": "${PG_USER}",
|
|
"database": "${PG_DB}",
|
|
"isDefault": false,
|
|
"jsonData": { "sslmode": "disable" },
|
|
"secureJsonData": { "password": "${PG_PASSWORD}" }
|
|
}
|
|
EOF
|
|
)
|
|
curl -sS -u "${GRAFANA_USER}:${GRAFANA_PASSWORD}" \
|
|
-H 'Content-Type: application/json' \
|
|
-X POST "http://localhost:3000/api/datasources" \
|
|
-d "$pg_ds_body" | sed 's/^/ /'
|
|
echo
|
|
echo " created datasource 'PostgreSQL'"
|
|
fi
|
|
|
|
log "Importing Grafana dashboard"
|
|
GF_BODY=$(python3 -c '
|
|
import json, sys
|
|
with open("dashboards/grafana/Grafana-DMARC_Reports.json") as f:
|
|
d = json.load(f)
|
|
# Setting id=None lets Grafana create or replace by uid+overwrite.
|
|
d["id"] = None
|
|
print(json.dumps({"dashboard": d, "overwrite": True, "folderUid": ""}))
|
|
')
|
|
curl -sS -u "${GRAFANA_USER}:${GRAFANA_PASSWORD}" \
|
|
-H 'Content-Type: application/json' \
|
|
-X POST "http://localhost:3000/api/dashboards/db" \
|
|
-d "$GF_BODY" | sed 's/^/ /'
|
|
|
|
log "Importing Grafana PostgreSQL dashboard"
|
|
# Resolve the dashboard's ${DS_POSTGRESQL} input to the dmarc-pg datasource uid
|
|
# created above, drop the export-only __inputs/__requires keys, and let
|
|
# id=None create-or-replace by uid+overwrite.
|
|
GF_PG_BODY=$(python3 -c '
|
|
import json
|
|
with open("dashboards/grafana/Grafana-DMARC_Reports-PostgreSQL.json") as f:
|
|
text = f.read()
|
|
text = text.replace("${DS_POSTGRESQL}", "dmarc-pg")
|
|
d = json.loads(text)
|
|
d.pop("__inputs", None)
|
|
d.pop("__requires", None)
|
|
d["id"] = None
|
|
print(json.dumps({"dashboard": d, "overwrite": True, "folderUid": ""}))
|
|
')
|
|
curl -sS -u "${GRAFANA_USER}:${GRAFANA_PASSWORD}" \
|
|
-H 'Content-Type: application/json' \
|
|
-X POST "http://localhost:3000/api/dashboards/db" \
|
|
-d "$GF_PG_BODY" | sed 's/^/ /'
|
|
|
|
log "Importing Splunk dashboard views into the DMARC app"
|
|
splunk_import_view() {
|
|
local name="$1"
|
|
local file="$2"
|
|
# DELETE-then-POST is the only path that survives both first-run and
|
|
# re-run; POST to an existing view returns 409.
|
|
splunk_curl -X DELETE \
|
|
"https://localhost:8089/servicesNS/admin/DMARC/data/ui/views/${name}" \
|
|
>/dev/null 2>&1 || true
|
|
splunk_curl -X POST \
|
|
"https://localhost:8089/servicesNS/admin/DMARC/data/ui/views" \
|
|
-d "name=${name}" \
|
|
--data-urlencode "eai:data@-" \
|
|
< "$file" >/dev/null
|
|
echo " imported splunk view: ${name}"
|
|
}
|
|
|
|
splunk_import_view dmarc_aggregate dashboards/splunk/dmarc_aggregate_dashboard.xml
|
|
splunk_import_view dmarc_failure dashboards/splunk/dmarc_failure_dashboard.xml
|
|
splunk_import_view smtp_tls dashboards/splunk/smtp_tls_dashboard.xml
|
|
|
|
cat <<EOF
|
|
|
|
== Done. UIs available at:
|
|
Kibana http://localhost:5601/
|
|
OpenSearch Dashboards http://localhost:5602/ (admin / ${OPENSEARCH_INITIAL_ADMIN_PASSWORD})
|
|
Grafana http://localhost:3000/ (${GRAFANA_USER} / ${GRAFANA_PASSWORD})
|
|
Splunk http://localhost:8000/ (admin / ${SPLUNK_PASSWORD})
|
|
PostgreSQL localhost:5432 (${PG_USER} / ${PG_PASSWORD}, db ${PG_DB})
|
|
EOF
|