mirror of
https://github.com/domainaware/parsedmarc.git
synced 2026-09-05 05:35:06 +00:00
Make dashboard-dev-bootstrap.sh work with Docker or Podman (#891)
The bootstrap script hardcoded `docker compose`, locking out contributors on Podman-based systems. It now auto-detects a working container engine (Docker preferred when both are usable) and its Compose implementation (`docker compose`/`docker-compose`, `podman compose`/`podman-compose`), selectable explicitly with --backend docker|podman or the CONTAINER_BACKEND environment variable (the flag wins). Detection requires a live `<engine> info` call, so a leftover docker CLI with no running daemon does not shadow a working podman, and both compose forms are probed with `version` so a broken install fails at startup instead of partway through the run. Adds -h/--help documenting the flag and the script's env knobs. Every container invocation already flowed through the single COMPOSE array, so the rest of the script is unchanged. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Sean Whalen
co-authored by
Claude Fable 5
Claude Fable 5
parent
07b00a931b
commit
0d832ff682
@@ -7,6 +7,7 @@
|
||||
- **Breaking: the output and mailbox integrations are now optional extras, so `pip install parsedmarc` installs the parsing core and a working core CLI instead of every SDK** ([#883](https://github.com/domainaware/parsedmarc/issues/883)). A 10.x install pulled in the Elasticsearch, OpenSearch, Kafka, AWS (boto3), Azure, Gmail, and Microsoft Graph client libraries whether or not a deployment used any of them — roughly 1 GB of site-packages against roughly 300 MB without — which is a lot to ask of the motivating case, running parsedmarc as a parsing library on a small mail host. The base install now covers the parsing library plus the CLI reading from files, IMAP, Maildir, and mbox, and writing CSV/JSON, Splunk HEC, webhook, and syslog output; those outputs need only `httpx` and the standard library, so they deliberately have no extra of their own. Everything else moves behind an extra: `elastic` (Elasticsearch), `opensearch` (OpenSearch, including the boto3 SigV4 signer), `kafka`, `s3`, `gelf`, `loganalytics` (Azure Monitor), `msgraph` (Microsoft 365 mailboxes), and `gmail` (Gmail API mailboxes), joining the `postgresql` extra that already existed. **CLI users upgrading from 10.x should switch their upgrade command to `pip install -U "parsedmarc[all]"`** — the new umbrella extra — to keep every integration available; add `postgresql` to the list (`parsedmarc[all,postgresql]`) if the PostgreSQL backend is in use. `all` deliberately excludes `postgresql` because `psycopg`'s prebuilt binary wheels do not exist for every platform, and `pip install parsedmarc[all]` must not fail on a platform they do not cover. Users of the prebuilt Docker image (`ghcr.io/domainaware/parsedmarc`) see no change at all: the image now installs `[all,postgresql]`, so it still bundles every integration. A configuration section whose extra is missing no longer fails with an `ImportError` traceback at startup; it fails fast with a `ConfigurationError` naming both the section and the command that fixes it, e.g. `The [elasticsearch] configuration section requires the elastic extra: pip install parsedmarc[elastic]`. Finally, the never-imported `dateparser` dependency is dropped in favor of declaring `python-dateutil`, which `parsedmarc.utils` actually imports and which used to arrive only transitively through `dateparser`.
|
||||
- The CLI now accepts `--dns-timeout` as an alias of `--dns_timeout`, which is kept for backward compatibility (public since 6.0.0); `--dns-retries` already used the hyphenated form.
|
||||
- **Failure and SMTP TLS reports are now sent to Kafka as one message per report, matching the aggregate saver's long-documented per-record behavior.** `save_failure_reports_to_kafka` and `save_smtp_tls_reports_to_kafka` documented per-record sends in every released version, but the code actually sent the entire report list as a single Kafka message (an unreleased docstring pass in [#888](https://github.com/domainaware/parsedmarc/pull/888) had briefly aligned the wording to the buggy code); a large batch could exceed Kafka's default 1MB message limit, and failure reports in particular carry message samples that make that more likely. Both savers now send/flush one message per report, mirroring `save_aggregate_reports_to_kafka`'s existing per-slice shape. This is consumer-visible: consumers now receive individual report objects on these topics rather than one JSON array per batch.
|
||||
- `dashboard-dev-bootstrap.sh` (the contributor dashboard dev-stack bootstrap) now works with Podman as well as Docker — it auto-detects a working container engine (Docker preferred when both are usable) and its Compose implementation (`docker compose`/`docker-compose`, `podman compose`/`podman-compose`), selectable explicitly with `--backend docker|podman` or the `CONTAINER_BACKEND` environment variable. Contributor tooling only; no behavior change for the parsedmarc package itself.
|
||||
|
||||
### Bug fixes
|
||||
|
||||
|
||||
+112
-5
@@ -1,14 +1,121 @@
|
||||
#!/usr/bin/env bash
|
||||
# Bring up docker-compose.dashboard-dev.yml, import the latest parsedmarc
|
||||
# dashboards into each viz system, and seed each backend with sample data so
|
||||
# the dashboards have something to render. Idempotent — safe to re-run.
|
||||
# Bring up docker-compose.dashboard-dev.yml (works with Docker or Podman),
|
||||
# import the latest parsedmarc dashboards into each viz system, and seed each
|
||||
# backend with sample data so the dashboards have something to render.
|
||||
# Idempotent — safe to re-run.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
cd "$REPO_ROOT"
|
||||
|
||||
COMPOSE=(docker compose -f docker-compose.dashboard-dev.yml --env-file .env)
|
||||
usage() {
|
||||
cat <<EOF
|
||||
Usage: $(basename "${BASH_SOURCE[0]}") [-b|--backend docker|podman] [-h|--help]
|
||||
|
||||
Bring up the dashboard-dev stack, import dashboards, and seed sample data.
|
||||
|
||||
Options:
|
||||
-b, --backend docker|podman Container backend to use (auto-detects when
|
||||
omitted; Docker is preferred if both work).
|
||||
-h, --help Show this help and exit.
|
||||
|
||||
Environment variables:
|
||||
CONTAINER_BACKEND Same as --backend; the flag takes precedence.
|
||||
RESEED=1 Wipe and re-seed sample data.
|
||||
OSD_TENANT OpenSearch Dashboards tenant for the import (default: global).
|
||||
WAIT_TIMEOUT Per-service readiness timeout in seconds (default: 180).
|
||||
PARSEDMARC_BIN Explicit parsedmarc CLI to use for seeding.
|
||||
EOF
|
||||
}
|
||||
|
||||
# A leftover docker CLI with no running daemon must not shadow a working
|
||||
# podman: require both the binary and a live `info` call, since `podman info`
|
||||
# works daemonless while `docker info` fails without a running daemon.
|
||||
engine_works() {
|
||||
local engine="$1"
|
||||
command -v "$engine" >/dev/null 2>&1 && "$engine" info >/dev/null 2>&1
|
||||
}
|
||||
|
||||
# `podman compose` is a thin wrapper that delegates to an external provider
|
||||
# (docker-compose or podman-compose), so the version probe correctly fails
|
||||
# when no provider is installed and we fall back to the standalone binary —
|
||||
# itself probed with `version` so a broken install is rejected here rather
|
||||
# than partway through the run.
|
||||
resolve_compose() {
|
||||
local engine="$1"
|
||||
if "$engine" compose version >/dev/null 2>&1; then
|
||||
COMPOSE_CMD=("$engine" compose)
|
||||
elif command -v "${engine}-compose" >/dev/null 2>&1 &&
|
||||
"${engine}-compose" version >/dev/null 2>&1; then
|
||||
COMPOSE_CMD=("${engine}-compose")
|
||||
else
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
BACKEND="${CONTAINER_BACKEND:-}"
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
-b|--backend)
|
||||
if [ $# -lt 2 ] || [ -z "$2" ]; then
|
||||
echo "ERROR: $1 requires a value (docker or podman)" >&2
|
||||
usage >&2
|
||||
exit 2
|
||||
fi
|
||||
BACKEND="$2"
|
||||
shift 2
|
||||
;;
|
||||
--backend=*)
|
||||
BACKEND="${1#*=}"
|
||||
if [ -z "$BACKEND" ]; then
|
||||
echo "ERROR: --backend requires a value (docker or podman)" >&2
|
||||
usage >&2
|
||||
exit 2
|
||||
fi
|
||||
shift
|
||||
;;
|
||||
-h|--help)
|
||||
usage
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
echo "ERROR: unknown argument: $1" >&2
|
||||
usage >&2
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
case "$BACKEND" in
|
||||
docker|podman)
|
||||
if ! engine_works "$BACKEND"; then
|
||||
echo "ERROR: backend '$BACKEND' was requested but is not usable (missing, or its 'info' check failed — is it installed and running?)" >&2
|
||||
exit 1
|
||||
fi
|
||||
if ! resolve_compose "$BACKEND"; then
|
||||
echo "ERROR: no compose implementation found for '$BACKEND' (looked for '$BACKEND compose' and '${BACKEND}-compose')" >&2
|
||||
exit 1
|
||||
fi
|
||||
;;
|
||||
"")
|
||||
if engine_works docker && resolve_compose docker; then
|
||||
BACKEND=docker
|
||||
elif engine_works podman && resolve_compose podman; then
|
||||
BACKEND=podman
|
||||
else
|
||||
echo "ERROR: no working container engine found. Install Docker or Podman with a Compose implementation, or pass --backend." >&2
|
||||
exit 1
|
||||
fi
|
||||
;;
|
||||
*)
|
||||
echo "ERROR: invalid backend '$BACKEND' from --backend or CONTAINER_BACKEND (expected docker or podman)" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
COMPOSE=("${COMPOSE_CMD[@]}" -f docker-compose.dashboard-dev.yml --env-file .env)
|
||||
echo "container backend: ${BACKEND} (${COMPOSE_CMD[*]})"
|
||||
|
||||
# Load .env so this script can use the same secrets compose injects.
|
||||
set -a
|
||||
@@ -48,7 +155,7 @@ wait_for() {
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1. Bring up the stack
|
||||
# ---------------------------------------------------------------------------
|
||||
log "Starting docker compose dashboard-dev stack"
|
||||
log "Starting dashboard-dev stack (${COMPOSE_CMD[*]})"
|
||||
"${COMPOSE[@]}" up -d
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@@ -27,7 +27,7 @@ All ports bind to `127.0.0.1` only.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
1. Docker with the Compose v2 plugin.
|
||||
1. Docker or Podman, with a Compose implementation (`docker compose` or `docker-compose`; `podman compose` or `podman-compose`). Compose v2 spec support is required because the compose file uses `include:`.
|
||||
2. A repo-root `.env` defining the secrets the compose file references:
|
||||
|
||||
```ini
|
||||
@@ -48,9 +48,11 @@ All ports bind to `127.0.0.1` only.
|
||||
./dashboard-dev-bootstrap.sh
|
||||
```
|
||||
|
||||
It auto-detects the container backend (Docker preferred, Podman as a fallback), which can be forced with `--backend docker|podman` or the `CONTAINER_BACKEND` environment variable. Run `./dashboard-dev-bootstrap.sh --help` for the full option and env var list.
|
||||
|
||||
It does, in order:
|
||||
|
||||
1. `docker compose -f docker-compose.dashboard-dev.yml up -d` and waits for every service's health endpoint.
|
||||
1. Brings up the compose stack with the detected compose command (`docker compose -f docker-compose.dashboard-dev.yml up -d`, or the `docker-compose`/`podman compose`/`podman-compose` equivalent) and waits for every service's health endpoint.
|
||||
2. Provisions Splunk: creates the `email` index, creates the `DMARC` app, configures the auto-created HEC token to allow the `email` index, and scopes the search-app's "scheduled export" announcement view away from `global` so it stops appearing in the DMARC app's dashboard list.
|
||||
3. Seeds Elasticsearch, OpenSearch, Splunk, and PostgreSQL with parsedmarc-parsed sample reports (from [samples/](../samples/)) so the dashboards render against real data. Skipped when ES already has aggregate docs — pass `RESEED=1` to wipe and re-seed all four backends.
|
||||
4. Imports the dashboard files from this directory into the running services. This step always runs, so the typical edit loop is **edit in the UI → export → save into this directory → re-run the bootstrap script** to verify the file imports cleanly into a fresh service.
|
||||
@@ -118,3 +120,5 @@ Wipes every `dmarc_aggregate*` / `dmarc_failure*` / `dmarc_forensic*` / `smtp_tl
|
||||
docker compose -f docker-compose.dashboard-dev.yml down # stop containers, keep volumes
|
||||
docker compose -f docker-compose.dashboard-dev.yml down -v # also drop volumes (full reset)
|
||||
```
|
||||
|
||||
Substitute `podman compose` (or `podman-compose`) for `docker compose` if Podman is the selected backend.
|
||||
|
||||
Reference in New Issue
Block a user