parsedmarc/dashboards

Dashboard development

This directory holds the dashboard sources that ship with parsedmarc:

• opensearch/opensearch_dashboards.ndjson — the source-of-truth saved-objects export. It is imported into both OpenSearch Dashboards and Kibana (the file format is compatible with both).
• grafana/Grafana-DMARC_Reports.json — the Grafana dashboard, with two Elasticsearch datasources (dmarc-ag, dmarc-fo).
• grafana/Grafana-DMARC_Reports-PostgreSQL.json — the Grafana dashboard for the PostgreSQL backend.
• grafana/Grafana-DMARC_Reports-Loki.json — the same layout as the PostgreSQL dashboard, querying the log lines the Loki output writes.
• splunk/ — three Splunk dashboard XML views (dmarc_aggregate, dmarc_failure, smtp_tls).

Edits to any of these files should be exported from a running instance after authoring the change in the UI, not hand-edited (with the occasional exception of small XML tweaks for Splunk).

The dev stack

docker-compose.dashboard-dev.yml brings up every viz target at once so a single dashboard change can be authored and re-exported across all four UIs in one session. It include:s docker-compose.yml for the Elasticsearch and OpenSearch backends, then layers on Kibana, OpenSearch Dashboards, Grafana, Splunk, a PostgreSQL backend, and Loki.

Service	URL	Credentials
Elasticsearch	http://localhost:9200	(security disabled)
OpenSearch	https://localhost:9201	admin / $OPENSEARCH_INITIAL_ADMIN_PASSWORD
Kibana	http://localhost:5601	(security disabled)
OpenSearch Dashboards	http://localhost:5602	admin / $OPENSEARCH_INITIAL_ADMIN_PASSWORD
Grafana	http://localhost:3000	admin / $GRAFANA_PASSWORD
PostgreSQL	localhost:5432	parsedmarc / parsedmarc (override: POSTGRESQL_*)
Loki	http://localhost:3100	(no auth)
Splunk Web / HEC	http://localhost:8000 / https://localhost:8088	admin / $SPLUNK_PASSWORD, HEC token $SPLUNK_HEC_TOKEN

All ports bind to 127.0.0.1 only.

Prerequisites

1. Docker or Podman, with a Compose implementation (docker compose or docker-compose; podman compose or podman-compose). Compose v2 spec support is required because the compose file uses include:.

2. A repo-root .env defining the secrets the compose file references:

   OPENSEARCH_INITIAL_ADMIN_PASSWORD=...
   SPLUNK_PASSWORD=...
   SPLUNK_HEC_TOKEN=...
   GRAFANA_PASSWORD=...
   

   Pick any values you like — these are local-only dev secrets. Both .env and parsedmarc*.ini are gitignored. The matching values must also appear in parsedmarc-dev.ini, which the bootstrap script feeds to the parsedmarc CLI for sample-data ingestion.

3. The parsedmarc CLI on PATH (or in ./venv/bin/) — pip install -e ".[all,postgresql]" from the repo root works. The all and postgresql extras are what the bootstrap script installs, and the seed needs them: the Elasticsearch, OpenSearch, and PostgreSQL outputs each live behind an extra. Override the lookup with PARSEDMARC_BIN=/path/to/parsedmarc if needed.

One-shot bootstrap

dashboard-dev-bootstrap.sh is the normal entry point. It is idempotent — re-run it any time:

./dashboard-dev-bootstrap.sh

It auto-detects the container backend (Docker preferred, Podman as a fallback), which can be forced with --backend docker|podman or the CONTAINER_BACKEND environment variable. Run ./dashboard-dev-bootstrap.sh --help for the full option and env var list.

It does, in order:

1. Brings up the compose stack with the detected compose command (docker compose -f docker-compose.dashboard-dev.yml up -d, or the docker-compose/podman compose/podman-compose equivalent) and waits for every service's health endpoint.
2. Provisions Splunk: creates the email index, creates the DMARC app, configures the auto-created HEC token to allow the email index, and scopes the search-app's "scheduled export" announcement view away from global so it stops appearing in the DMARC app's dashboard list.
3. Seeds Elasticsearch, OpenSearch, Splunk, PostgreSQL, and Loki with parsedmarc-parsed sample reports (from samples/) so the dashboards render against real data. Skipped when ES already has aggregate docs — pass RESEED=1 to wipe and re-seed all five backends. Loki lines carry the time of the seed run, not the report dates, because Loki rejects samples older than a week.
4. Imports the dashboard files from this directory into the running services. This step always runs, so the typical edit loop is edit in the UI → export → save into this directory → re-run the bootstrap script to verify the file imports cleanly into a fresh service.

VS Code users can run this via the Dev Dashboard: Bootstrap task in .vscode/tasks.json. Dev Dashboard: Up brings the stack up without importing or seeding.

Editing a dashboard

After running the bootstrap script once, the round trip for each platform is:

OpenSearch Dashboards (and Kibana)

1. Edit the dashboard at http://localhost:5602/ (OpenSearch Dashboards) — this is the canonical authoring surface.
2. Stack Management → Saved Objects → Export, select the DMARC dashboard, include related objects, and save the resulting .ndjson over opensearch/opensearch_dashboards.ndjson.
3. Re-run ./dashboard-dev-bootstrap.sh to confirm it re-imports cleanly into both OSD and Kibana. The Kibana CI workflow (.github/workflows/dashboards.yml) also imports the same file on every PR that touches it.

OSD imports default to the global_tenant so other admins on the instance can see the result. Set OSD_TENANT=... to import elsewhere.

Grafana

1. Edit the dashboard at http://localhost:3000/.
2. Dashboard settings → JSON Model, copy the JSON, save it to grafana/Grafana-DMARC_Reports.json. For the PostgreSQL and Loki dashboards use Export → Export as JSON with Export for sharing externally enabled, so the datasource stays the ${DS_POSTGRESQL} / ${DS_LOKI} input.
3. Re-run the bootstrap script.

The bootstrap script provisions two elasticsearch datasources (dmarc-ag for dmarc_aggregate*, dmarc-fo for dmarc_f*, which matches both pre-rename dmarc_forensic* and post-rename dmarc_failure*) on first run, plus a PostgreSQL datasource (uid dmarc-pg) and a Loki datasource (uid dmarc-loki); existing datasources are left alone.

tests/test_grafana_loki_dashboard.py keeps the Loki dashboard in step with the PostgreSQL layout and checks that every field it queries exists in the lines the Loki output writes.

Splunk

1. Edit the dashboard at http://localhost:8000/ inside the DMARC app.
2. Open the dashboard's Source view, copy the XML, and paste it over the matching file in splunk/ (dmarc_aggregate_dashboard.xml, dmarc_failure_dashboard.xml, or smtp_tls_dashboard.xml).
3. Re-run the bootstrap script. It re-imports each view via DELETE + POST to the splunkd management API.

Screenshotting the dashboards

dashboard-dev-screenshots.py captures how each UI actually renders the current sample data — useful for verifying a dashboard change end-to-end and for PR evidence. One-time setup, then run against the live stack:

pip install playwright && playwright install chromium
set -a; . ./.env; set +a
./dashboard-dev-screenshots.py            # all four UIs
./dashboard-dev-screenshots.py grafana    # or a subset

Output lands in dashboard-screenshots/ (gitignored). The script encodes the platform quirks that otherwise cost time to rediscover: Kibana/OSD dashboards never reach Playwright's "networkidle" (fixed render waits are used instead), OSD must be pinned to ?security_tenant=global or a stale private-tenant copy may be captured, and Grafana/Splunk need their login forms driven rather than HTTP basic auth.

Reseeding sample data

RESEED=1 ./dashboard-dev-bootstrap.sh

Wipes every dmarc_aggregate* / dmarc_failure* / dmarc_forensic* / smtp_tls* index from ES and OS, drops and recreates the Splunk email index and the PostgreSQL schema, recreates the Loki container, then re-runs the parsedmarc CLI against the curated sample list. Use this after changing parsedmarc's enrichment or output schemas.

Tearing the stack down

docker compose -f docker-compose.dashboard-dev.yml down          # stop containers, keep volumes
docker compose -f docker-compose.dashboard-dev.yml down -v       # also drop volumes (full reset)

Substitute podman compose (or podman-compose) for docker compose if Podman is the selected backend.