amorfo77/parsedmarc
1
0 Fork 0
mirror of https://github.com/domainaware/parsedmarc.git synced 2026-11-07 01:09:26 +00:00
Code Issues Projects Releases Wiki Activity
Files
48ea47adee7ed2f014d770b79d742fde7bb03098
parsedmarc/dashboards/README.md
T
Sebastian SchubertClaude Opus 5.5Sean Whalen
 768bf75480 Bastischubert/loki output (#926) 
* feat: add grafana loki output

Push aggregate, failure and SMTP TLS reports as JSON log lines to the
Loki push API. Static labels and labels taken from report fields are
configurable, host defaults to the FQDN. The timestamp is either the
ingest time or the report begin time. Supports basic auth, bearer
tokens, X-Scope-OrgID and batching. Push errors are reported as output
errors so messages are not archived.

Signed-off-by: Sebastian Schubert <basti@schubert.digital>

* feat(dashboards): add grafana dashboard for the loki output

Signed-off-by: Sebastian Schubert <basti@schubert.digital>

* Address Copilot review on the Loki output

Responds to Copilot's review of #926:

- Treat any push status other than 200/204 as a failure. Loki returns
  block_ingestion_status_code (260 by default) when block_ingestion_until
  blocks ingestion, and the logs are discarded (Loki configuration
  reference); the old >= 300 check counted that as a successful save, so
  the CLI could archive mailbox messages whose reports were never stored.
- Read aggregate begin_date as UTC for timestamp = report_begin. Aggregate
  CSV rows carry the record's interval_begin, which the parser builds as
  naive UTC wall-clock time (to_utc=True); reading it as local time shifted
  entries by the host's UTC offset (+4h on America/New_York).
- Group the Published Policies dashboard dedup by org_name as well as
  report_id: report IDs are only unique per reporter (the PostgreSQL key
  is UNIQUE (org_name, report_id, ...)).
- Drop the legend sum from SMTP TLS Sessions Over Time: its [1d] lookback
  with the automatic step yields overlapping rolling totals, so the sum
  counted the same sessions repeatedly.

Adds regression tests for each, including a non-UTC timezone test against
a real sample report.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Address second Copilot review on the Loki output

Responds to Copilot's second review of #926 (previously-missed findings):

- Strip each [loki] label_fields entry. _str_to_list() only lstrips, so
  "report_type , domain" produced "report_type " and LokiClient refused to
  start with "Invalid Loki label name 'report_type '".
- Wipe Loki on RESEED=1 even when Elasticsearch is empty. The wipe sat
  inside the ES-has-docs guard, so after the ES container was recreated a
  reseed appended a second copy of the samples to Loki, which does not
  deduplicate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Address third Copilot review on the Loki output

Responds to Copilot's third review of #926 (previously-missed findings):

- Keep a base URL's query string when appending the push path. The path
  was string-concatenated, so https://gw.example.org?route=logs became
  ...?route=logs/loki/api/v1/push, hitting the root path with a corrupted
  query. The path is now set on the parsed httpx.URL.
- Push each report's entries in timestamp order. With timestamp =
  report_begin, a normalized multi-day aggregate report yields rows
  source-major (every day of one source, then day 1 of the next), all in
  one stream. Loki rejects entries older than the newest line minus half
  of max_chunk_age (1h by default) with too_far_behind (Loki configuration
  docs, "Accept out-of-order writes"), so later sources were partly
  rejected. The whole entry list is now sorted before batching.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(loki): do not follow redirects and drop forensic alias

httpx re-sends a 301/302 POST as GET, so a redirected push could return 200 without storing anything, and the mailbox messages were archived. Redirects are no longer followed, so a 3xx raises LokiError.

Remove save_forensic_report_to_loki: the module is new, so no caller needs the legacy alias.

Signed-off-by: Sebastian Schubert <basti@schubert.digital>

* fix(dashboards): use the existing Loki datasource uid in the dev bootstrap

When a datasource named Loki already exists with another uid, the dashboard import still bound every panel to dmarc-loki. Read the uid from the lookup response instead.

Signed-off-by: Sebastian Schubert <basti@schubert.digital>

---------

Signed-off-by: Sebastian Schubert <basti@schubert.digital>
Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 10:13:37 -04:00

129 lines
9.6 KiB
Markdown
Raw Blame History

# Dashboard development
This directory holds the dashboard sources that ship with parsedmarc:
- [opensearch/opensearch_dashboards.ndjson](opensearch/opensearch_dashboards.ndjson) — the source-of-truth saved-objects export. It is imported into both **OpenSearch Dashboards** and **Kibana** (the file format is compatible with both).
- [grafana/Grafana-DMARC_Reports.json](grafana/Grafana-DMARC_Reports.json) — the Grafana dashboard, with two Elasticsearch datasources (`dmarc-ag`, `dmarc-fo`).
- [grafana/Grafana-DMARC_Reports-PostgreSQL.json](grafana/Grafana-DMARC_Reports-PostgreSQL.json) — the Grafana dashboard for the PostgreSQL backend.
- [grafana/Grafana-DMARC_Reports-Loki.json](grafana/Grafana-DMARC_Reports-Loki.json) — the same layout as the PostgreSQL dashboard, querying the log lines the Loki output writes.
- [splunk/](splunk/) — three Splunk dashboard XML views (`dmarc_aggregate`, `dmarc_failure`, `smtp_tls`).
Edits to any of these files should be exported from a running instance after authoring the change in the UI, not hand-edited (with the occasional exception of small XML tweaks for Splunk).
## The dev stack
[docker-compose.dashboard-dev.yml](../docker-compose.dashboard-dev.yml) brings up every viz target at once so a single dashboard change can be authored and re-exported across all four UIs in one session. It `include:`s [docker-compose.yml](../docker-compose.yml) for the Elasticsearch and OpenSearch backends, then layers on Kibana, OpenSearch Dashboards, Grafana, Splunk, a PostgreSQL backend, and Loki.
| Service | URL | Credentials |
| --------------------- | ------------------------------------------------ | ------------------------------------------------------ |
| Elasticsearch | http://localhost:9200 | (security disabled) |
| OpenSearch | https://localhost:9201 | `admin` / `$OPENSEARCH_INITIAL_ADMIN_PASSWORD` |
| Kibana | http://localhost:5601 | (security disabled) |
| OpenSearch Dashboards | http://localhost:5602 | `admin` / `$OPENSEARCH_INITIAL_ADMIN_PASSWORD` |
| Grafana | http://localhost:3000 | `admin` / `$GRAFANA_PASSWORD` |
| PostgreSQL | localhost:5432 | `parsedmarc` / `parsedmarc` (override: `POSTGRESQL_*`) |
| Loki | http://localhost:3100 | (no auth) |
| Splunk Web / HEC | http://localhost:8000 / https://localhost:8088 | `admin` / `$SPLUNK_PASSWORD`, HEC token `$SPLUNK_HEC_TOKEN` |
All ports bind to `127.0.0.1` only.
## Prerequisites
1. Docker or Podman, with a Compose implementation (`docker compose` or `docker-compose`; `podman compose` or `podman-compose`). Compose v2 spec support is required because the compose file uses `include:`.
2. A repo-root `.env` defining the secrets the compose file references:
```ini
OPENSEARCH_INITIAL_ADMIN_PASSWORD=...
SPLUNK_PASSWORD=...
SPLUNK_HEC_TOKEN=...
GRAFANA_PASSWORD=...
```
Pick any values you like — these are local-only dev secrets. Both `.env` and `parsedmarc*.ini` are gitignored. The matching values must also appear in [parsedmarc-dev.ini](../parsedmarc-dev.ini), which the bootstrap script feeds to the parsedmarc CLI for sample-data ingestion.
3. The parsedmarc CLI on `PATH` (or in `./venv/bin/`) — `pip install -e ".[all,postgresql]"` from the repo root works. The `all` and `postgresql` extras are what the bootstrap script installs, and the seed needs them: the Elasticsearch, OpenSearch, and PostgreSQL outputs each live behind an extra. Override the lookup with `PARSEDMARC_BIN=/path/to/parsedmarc` if needed.
## One-shot bootstrap
[dashboard-dev-bootstrap.sh](../dashboard-dev-bootstrap.sh) is the normal entry point. It is idempotent — re-run it any time:
```bash
./dashboard-dev-bootstrap.sh
```
It auto-detects the container backend (Docker preferred, Podman as a fallback), which can be forced with `--backend docker|podman` or the `CONTAINER_BACKEND` environment variable. Run `./dashboard-dev-bootstrap.sh --help` for the full option and env var list.
It does, in order:
1. Brings up the compose stack with the detected compose command (`docker compose -f docker-compose.dashboard-dev.yml up -d`, or the `docker-compose`/`podman compose`/`podman-compose` equivalent) and waits for every service's health endpoint.
2. Provisions Splunk: creates the `email` index, creates the `DMARC` app, configures the auto-created HEC token to allow the `email` index, and scopes the search-app's "scheduled export" announcement view away from `global` so it stops appearing in the DMARC app's dashboard list.
3. Seeds Elasticsearch, OpenSearch, Splunk, PostgreSQL, and Loki with parsedmarc-parsed sample reports (from [samples/](../samples/)) so the dashboards render against real data. Skipped when ES already has aggregate docs — pass `RESEED=1` to wipe and re-seed all five backends. Loki lines carry the time of the seed run, not the report dates, because Loki rejects samples older than a week.
4. Imports the dashboard files from this directory into the running services. This step always runs, so the typical edit loop is **edit in the UI → export → save into this directory → re-run the bootstrap script** to verify the file imports cleanly into a fresh service.
VS Code users can run this via the **Dev Dashboard: Bootstrap** task in [.vscode/tasks.json](../.vscode/tasks.json). **Dev Dashboard: Up** brings the stack up without importing or seeding.
## Editing a dashboard
After running the bootstrap script once, the round trip for each platform is:
### OpenSearch Dashboards (and Kibana)
1. Edit the dashboard at http://localhost:5602/ (OpenSearch Dashboards) — this is the canonical authoring surface.
2. **Stack Management → Saved Objects → Export**, select the DMARC dashboard, include related objects, and save the resulting `.ndjson` over [opensearch/opensearch_dashboards.ndjson](opensearch/opensearch_dashboards.ndjson).
3. Re-run `./dashboard-dev-bootstrap.sh` to confirm it re-imports cleanly into both OSD and Kibana. The Kibana CI workflow ([.github/workflows/dashboards.yml](../.github/workflows/dashboards.yml)) also imports the same file on every PR that touches it.
OSD imports default to the `global_tenant` so other admins on the instance can see the result. Set `OSD_TENANT=...` to import elsewhere.
### Grafana
1. Edit the dashboard at http://localhost:3000/.
2. **Dashboard settings → JSON Model**, copy the JSON, save it to [grafana/Grafana-DMARC_Reports.json](grafana/Grafana-DMARC_Reports.json). For the PostgreSQL and Loki dashboards use **Export → Export as JSON** with **Export for sharing externally** enabled, so the datasource stays the `${DS_POSTGRESQL}` / `${DS_LOKI}` input.
3. Re-run the bootstrap script.
The bootstrap script provisions two `elasticsearch` datasources (`dmarc-ag` for `dmarc_aggregate*`, `dmarc-fo` for `dmarc_f*`, which matches both pre-rename `dmarc_forensic*` and post-rename `dmarc_failure*`) on first run, plus a `PostgreSQL` datasource (uid `dmarc-pg`) and a `Loki` datasource (uid `dmarc-loki`); existing datasources are left alone.
[tests/test_grafana_loki_dashboard.py](../tests/test_grafana_loki_dashboard.py) keeps the Loki dashboard in step with the PostgreSQL layout and checks that every field it queries exists in the lines the Loki output writes.
### Splunk
1. Edit the dashboard at http://localhost:8000/ inside the **DMARC** app.
2. Open the dashboard's **Source** view, copy the XML, and paste it over the matching file in [splunk/](splunk/) (`dmarc_aggregate_dashboard.xml`, `dmarc_failure_dashboard.xml`, or `smtp_tls_dashboard.xml`).
3. Re-run the bootstrap script. It re-imports each view via `DELETE` + `POST` to the splunkd management API.
## Screenshotting the dashboards
[dashboard-dev-screenshots.py](../dashboard-dev-screenshots.py) captures how
each UI actually renders the current sample data — useful for verifying a
dashboard change end-to-end and for PR evidence. One-time setup, then run
against the live stack:
```bash
pip install playwright && playwright install chromium
set -a; . ./.env; set +a
./dashboard-dev-screenshots.py # all four UIs
./dashboard-dev-screenshots.py grafana # or a subset
```
Output lands in `dashboard-screenshots/` (gitignored). The script encodes
the platform quirks that otherwise cost time to rediscover: Kibana/OSD
dashboards never reach Playwright's "networkidle" (fixed render waits are
used instead), OSD must be pinned to `?security_tenant=global` or a stale
private-tenant copy may be captured, and Grafana/Splunk need their login
forms driven rather than HTTP basic auth.
## Reseeding sample data
```bash
RESEED=1 ./dashboard-dev-bootstrap.sh
```
Wipes every `dmarc_aggregate*` / `dmarc_failure*` / `dmarc_forensic*` / `smtp_tls*` index from ES and OS, drops and recreates the Splunk `email` index and the PostgreSQL schema, recreates the Loki container, then re-runs the parsedmarc CLI against the curated sample list. Use this after changing parsedmarc's enrichment or output schemas.
## Tearing the stack down
```bash
docker compose -f docker-compose.dashboard-dev.yml down # stop containers, keep volumes
docker compose -f docker-compose.dashboard-dev.yml down -v # also drop volumes (full reset)
```
Substitute `podman compose` (or `podman-compose`) for `docker compose` if Podman is the selected backend.
Reference in new issue View Git Blame Copy Permalink