From 0d832ff682abc525198b18f3f9870be318c58040 Mon Sep 17 00:00:00 2001 From: Sean Whalen <44679+seanthegeek@users.noreply.github.com> Date: Fri, 28 Aug 2026 22:47:00 -0400 Subject: [PATCH] Make dashboard-dev-bootstrap.sh work with Docker or Podman (#891) The bootstrap script hardcoded `docker compose`, locking out contributors on Podman-based systems. It now auto-detects a working container engine (Docker preferred when both are usable) and its Compose implementation (`docker compose`/`docker-compose`, `podman compose`/`podman-compose`), selectable explicitly with --backend docker|podman or the CONTAINER_BACKEND environment variable (the flag wins). Detection requires a live ` info` call, so a leftover docker CLI with no running daemon does not shadow a working podman, and both compose forms are probed with `version` so a broken install fails at startup instead of partway through the run. Adds -h/--help documenting the flag and the script's env knobs. Every container invocation already flowed through the single COMPOSE array, so the rest of the script is unchanged. Co-authored-by: Claude Fable 5 --- CHANGELOG.md | 1 + dashboard-dev-bootstrap.sh | 117 +++++++++++++++++++++++++++++++++++-- dashboards/README.md | 8 ++- 3 files changed, 119 insertions(+), 7 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0d175d30..f714dd65 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,7 @@ - **Breaking: the output and mailbox integrations are now optional extras, so `pip install parsedmarc` installs the parsing core and a working core CLI instead of every SDK** ([#883](https://github.com/domainaware/parsedmarc/issues/883)). A 10.x install pulled in the Elasticsearch, OpenSearch, Kafka, AWS (boto3), Azure, Gmail, and Microsoft Graph client libraries whether or not a deployment used any of them — roughly 1 GB of site-packages against roughly 300 MB without — which is a lot to ask of the motivating case, running parsedmarc as a parsing library on a small mail host. The base install now covers the parsing library plus the CLI reading from files, IMAP, Maildir, and mbox, and writing CSV/JSON, Splunk HEC, webhook, and syslog output; those outputs need only `httpx` and the standard library, so they deliberately have no extra of their own. Everything else moves behind an extra: `elastic` (Elasticsearch), `opensearch` (OpenSearch, including the boto3 SigV4 signer), `kafka`, `s3`, `gelf`, `loganalytics` (Azure Monitor), `msgraph` (Microsoft 365 mailboxes), and `gmail` (Gmail API mailboxes), joining the `postgresql` extra that already existed. **CLI users upgrading from 10.x should switch their upgrade command to `pip install -U "parsedmarc[all]"`** — the new umbrella extra — to keep every integration available; add `postgresql` to the list (`parsedmarc[all,postgresql]`) if the PostgreSQL backend is in use. `all` deliberately excludes `postgresql` because `psycopg`'s prebuilt binary wheels do not exist for every platform, and `pip install parsedmarc[all]` must not fail on a platform they do not cover. Users of the prebuilt Docker image (`ghcr.io/domainaware/parsedmarc`) see no change at all: the image now installs `[all,postgresql]`, so it still bundles every integration. A configuration section whose extra is missing no longer fails with an `ImportError` traceback at startup; it fails fast with a `ConfigurationError` naming both the section and the command that fixes it, e.g. `The [elasticsearch] configuration section requires the elastic extra: pip install parsedmarc[elastic]`. Finally, the never-imported `dateparser` dependency is dropped in favor of declaring `python-dateutil`, which `parsedmarc.utils` actually imports and which used to arrive only transitively through `dateparser`. - The CLI now accepts `--dns-timeout` as an alias of `--dns_timeout`, which is kept for backward compatibility (public since 6.0.0); `--dns-retries` already used the hyphenated form. - **Failure and SMTP TLS reports are now sent to Kafka as one message per report, matching the aggregate saver's long-documented per-record behavior.** `save_failure_reports_to_kafka` and `save_smtp_tls_reports_to_kafka` documented per-record sends in every released version, but the code actually sent the entire report list as a single Kafka message (an unreleased docstring pass in [#888](https://github.com/domainaware/parsedmarc/pull/888) had briefly aligned the wording to the buggy code); a large batch could exceed Kafka's default 1MB message limit, and failure reports in particular carry message samples that make that more likely. Both savers now send/flush one message per report, mirroring `save_aggregate_reports_to_kafka`'s existing per-slice shape. This is consumer-visible: consumers now receive individual report objects on these topics rather than one JSON array per batch. +- `dashboard-dev-bootstrap.sh` (the contributor dashboard dev-stack bootstrap) now works with Podman as well as Docker — it auto-detects a working container engine (Docker preferred when both are usable) and its Compose implementation (`docker compose`/`docker-compose`, `podman compose`/`podman-compose`), selectable explicitly with `--backend docker|podman` or the `CONTAINER_BACKEND` environment variable. Contributor tooling only; no behavior change for the parsedmarc package itself. ### Bug fixes diff --git a/dashboard-dev-bootstrap.sh b/dashboard-dev-bootstrap.sh index 26cadf08..9b76a6dd 100755 --- a/dashboard-dev-bootstrap.sh +++ b/dashboard-dev-bootstrap.sh @@ -1,14 +1,121 @@ #!/usr/bin/env bash -# Bring up docker-compose.dashboard-dev.yml, import the latest parsedmarc -# dashboards into each viz system, and seed each backend with sample data so -# the dashboards have something to render. Idempotent — safe to re-run. +# Bring up docker-compose.dashboard-dev.yml (works with Docker or Podman), +# import the latest parsedmarc dashboards into each viz system, and seed each +# backend with sample data so the dashboards have something to render. +# Idempotent — safe to re-run. set -euo pipefail REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" cd "$REPO_ROOT" -COMPOSE=(docker compose -f docker-compose.dashboard-dev.yml --env-file .env) +usage() { + cat </dev/null 2>&1 && "$engine" info >/dev/null 2>&1 +} + +# `podman compose` is a thin wrapper that delegates to an external provider +# (docker-compose or podman-compose), so the version probe correctly fails +# when no provider is installed and we fall back to the standalone binary — +# itself probed with `version` so a broken install is rejected here rather +# than partway through the run. +resolve_compose() { + local engine="$1" + if "$engine" compose version >/dev/null 2>&1; then + COMPOSE_CMD=("$engine" compose) + elif command -v "${engine}-compose" >/dev/null 2>&1 && + "${engine}-compose" version >/dev/null 2>&1; then + COMPOSE_CMD=("${engine}-compose") + else + return 1 + fi +} + +BACKEND="${CONTAINER_BACKEND:-}" +while [ $# -gt 0 ]; do + case "$1" in + -b|--backend) + if [ $# -lt 2 ] || [ -z "$2" ]; then + echo "ERROR: $1 requires a value (docker or podman)" >&2 + usage >&2 + exit 2 + fi + BACKEND="$2" + shift 2 + ;; + --backend=*) + BACKEND="${1#*=}" + if [ -z "$BACKEND" ]; then + echo "ERROR: --backend requires a value (docker or podman)" >&2 + usage >&2 + exit 2 + fi + shift + ;; + -h|--help) + usage + exit 0 + ;; + *) + echo "ERROR: unknown argument: $1" >&2 + usage >&2 + exit 2 + ;; + esac +done + +case "$BACKEND" in + docker|podman) + if ! engine_works "$BACKEND"; then + echo "ERROR: backend '$BACKEND' was requested but is not usable (missing, or its 'info' check failed — is it installed and running?)" >&2 + exit 1 + fi + if ! resolve_compose "$BACKEND"; then + echo "ERROR: no compose implementation found for '$BACKEND' (looked for '$BACKEND compose' and '${BACKEND}-compose')" >&2 + exit 1 + fi + ;; + "") + if engine_works docker && resolve_compose docker; then + BACKEND=docker + elif engine_works podman && resolve_compose podman; then + BACKEND=podman + else + echo "ERROR: no working container engine found. Install Docker or Podman with a Compose implementation, or pass --backend." >&2 + exit 1 + fi + ;; + *) + echo "ERROR: invalid backend '$BACKEND' from --backend or CONTAINER_BACKEND (expected docker or podman)" >&2 + exit 1 + ;; +esac + +COMPOSE=("${COMPOSE_CMD[@]}" -f docker-compose.dashboard-dev.yml --env-file .env) +echo "container backend: ${BACKEND} (${COMPOSE_CMD[*]})" # Load .env so this script can use the same secrets compose injects. set -a @@ -48,7 +155,7 @@ wait_for() { # --------------------------------------------------------------------------- # 1. Bring up the stack # --------------------------------------------------------------------------- -log "Starting docker compose dashboard-dev stack" +log "Starting dashboard-dev stack (${COMPOSE_CMD[*]})" "${COMPOSE[@]}" up -d # --------------------------------------------------------------------------- diff --git a/dashboards/README.md b/dashboards/README.md index 7e0fb699..1b59756b 100644 --- a/dashboards/README.md +++ b/dashboards/README.md @@ -27,7 +27,7 @@ All ports bind to `127.0.0.1` only. ## Prerequisites -1. Docker with the Compose v2 plugin. +1. Docker or Podman, with a Compose implementation (`docker compose` or `docker-compose`; `podman compose` or `podman-compose`). Compose v2 spec support is required because the compose file uses `include:`. 2. A repo-root `.env` defining the secrets the compose file references: ```ini @@ -48,9 +48,11 @@ All ports bind to `127.0.0.1` only. ./dashboard-dev-bootstrap.sh ``` +It auto-detects the container backend (Docker preferred, Podman as a fallback), which can be forced with `--backend docker|podman` or the `CONTAINER_BACKEND` environment variable. Run `./dashboard-dev-bootstrap.sh --help` for the full option and env var list. + It does, in order: -1. `docker compose -f docker-compose.dashboard-dev.yml up -d` and waits for every service's health endpoint. +1. Brings up the compose stack with the detected compose command (`docker compose -f docker-compose.dashboard-dev.yml up -d`, or the `docker-compose`/`podman compose`/`podman-compose` equivalent) and waits for every service's health endpoint. 2. Provisions Splunk: creates the `email` index, creates the `DMARC` app, configures the auto-created HEC token to allow the `email` index, and scopes the search-app's "scheduled export" announcement view away from `global` so it stops appearing in the DMARC app's dashboard list. 3. Seeds Elasticsearch, OpenSearch, Splunk, and PostgreSQL with parsedmarc-parsed sample reports (from [samples/](../samples/)) so the dashboards render against real data. Skipped when ES already has aggregate docs — pass `RESEED=1` to wipe and re-seed all four backends. 4. Imports the dashboard files from this directory into the running services. This step always runs, so the typical edit loop is **edit in the UI → export → save into this directory → re-run the bootstrap script** to verify the file imports cleanly into a fresh service. @@ -118,3 +120,5 @@ Wipes every `dmarc_aggregate*` / `dmarc_failure*` / `dmarc_forensic*` / `smtp_tl docker compose -f docker-compose.dashboard-dev.yml down # stop containers, keep volumes docker compose -f docker-compose.dashboard-dev.yml down -v # also drop volumes (full reset) ``` + +Substitute `podman compose` (or `podman-compose`) for `docker compose` if Podman is the selected backend.