amorfo77/parsedmarc
1
0 Fork 0
mirror of https://github.com/domainaware/parsedmarc.git synced 2026-11-07 09:19:22 +00:00
Code Issues Projects Releases Wiki Activity

Bastischubert/loki output (#926)

Browse Source 
* feat: add grafana loki output

Push aggregate, failure and SMTP TLS reports as JSON log lines to the
Loki push API. Static labels and labels taken from report fields are
configurable, host defaults to the FQDN. The timestamp is either the
ingest time or the report begin time. Supports basic auth, bearer
tokens, X-Scope-OrgID and batching. Push errors are reported as output
errors so messages are not archived.

Signed-off-by: Sebastian Schubert <basti@schubert.digital>

* feat(dashboards): add grafana dashboard for the loki output

Signed-off-by: Sebastian Schubert <basti@schubert.digital>

* Address Copilot review on the Loki output

Responds to Copilot's review of #926:

- Treat any push status other than 200/204 as a failure. Loki returns
  block_ingestion_status_code (260 by default) when block_ingestion_until
  blocks ingestion, and the logs are discarded (Loki configuration
  reference); the old >= 300 check counted that as a successful save, so
  the CLI could archive mailbox messages whose reports were never stored.
- Read aggregate begin_date as UTC for timestamp = report_begin. Aggregate
  CSV rows carry the record's interval_begin, which the parser builds as
  naive UTC wall-clock time (to_utc=True); reading it as local time shifted
  entries by the host's UTC offset (+4h on America/New_York).
- Group the Published Policies dashboard dedup by org_name as well as
  report_id: report IDs are only unique per reporter (the PostgreSQL key
  is UNIQUE (org_name, report_id, ...)).
- Drop the legend sum from SMTP TLS Sessions Over Time: its [1d] lookback
  with the automatic step yields overlapping rolling totals, so the sum
  counted the same sessions repeatedly.

Adds regression tests for each, including a non-UTC timezone test against
a real sample report.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Address second Copilot review on the Loki output

Responds to Copilot's second review of #926 (previously-missed findings):

- Strip each [loki] label_fields entry. _str_to_list() only lstrips, so
  "report_type , domain" produced "report_type " and LokiClient refused to
  start with "Invalid Loki label name 'report_type '".
- Wipe Loki on RESEED=1 even when Elasticsearch is empty. The wipe sat
  inside the ES-has-docs guard, so after the ES container was recreated a
  reseed appended a second copy of the samples to Loki, which does not
  deduplicate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Address third Copilot review on the Loki output

Responds to Copilot's third review of #926 (previously-missed findings):

- Keep a base URL's query string when appending the push path. The path
  was string-concatenated, so https://gw.example.org?route=logs became
  ...?route=logs/loki/api/v1/push, hitting the root path with a corrupted
  query. The path is now set on the parsed httpx.URL.
- Push each report's entries in timestamp order. With timestamp =
  report_begin, a normalized multi-day aggregate report yields rows
  source-major (every day of one source, then day 1 of the next), all in
  one stream. Loki rejects entries older than the newest line minus half
  of max_chunk_age (1h by default) with too_far_behind (Loki configuration
  docs, "Accept out-of-order writes"), so later sources were partly
  rejected. The whole entry list is now sorted before batching.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(loki): do not follow redirects and drop forensic alias

httpx re-sends a 301/302 POST as GET, so a redirected push could return 200 without storing anything, and the mailbox messages were archived. Redirects are no longer followed, so a 3xx raises LokiError.

Remove save_forensic_report_to_loki: the module is new, so no caller needs the legacy alias.

Signed-off-by: Sebastian Schubert <basti@schubert.digital>

* fix(dashboards): use the existing Loki datasource uid in the dev bootstrap

When a datasource named Loki already exists with another uid, the dashboard import still bound every panel to dmarc-loki. Read the uid from the lookup response instead.

Signed-off-by: Sebastian Schubert <basti@schubert.digital>

---------

Signed-off-by: Sebastian Schubert <basti@schubert.digital>
Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Sebastian SchubertClaude Opus 5.5Sean Whalen
 authored and GitHub committed 2026-10-31 10:13:37 -04:00
1 parent 2f89507e5d
commit 768bf75480
15 files changed
+5423 -15
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

dashboard-dev-bootstrap.sh
+57 -3
View File
@@ -174,6 +174,7 @@ wait_for "OpenSearch Dashboards" \
wait_for "Grafana" curl -sf http://localhost:3000/api/health
wait_for "PostgreSQL" \
"${COMPOSE[@]}" exec -T postgresql pg_isready -U "$PG_USER" -d "$PG_DB"
wait_for "Loki" curl -sf http://localhost:3100/ready
# Splunk's HEC port is healthy once management API is up too.
wait_for "Splunk HEC" curl -ksf https://localhost:8088/services/collector/health
# Splunkd management API (used for dashboard imports) lives inside the container.
@@ -276,9 +277,10 @@ elif [ -n "$SCHED_SHARING" ]; then
fi
# ---------------------------------------------------------------------------
# 4. Seed sample data via parsedmarc -> ES, OS, Splunk HEC
# 4. Seed sample data via parsedmarc -> ES, OS, Splunk HEC, PostgreSQL, Loki
# Skipped when ES already has aggregate docs from a prior run. Set
# RESEED=1 to wipe ES/OS/Splunk parsedmarc data first and re-seed.
# RESEED=1 to wipe parsedmarc data from every backend first and re-seed
# (Loki is wiped on RESEED=1 even when ES is empty).
# ---------------------------------------------------------------------------
log "Seeding sample data with parsedmarc-dev.ini"
ES_AGG_COUNT=$(curl -sf 'http://localhost:9200/dmarc_aggregate*/_count' 2>/dev/null \
@@ -326,6 +328,15 @@ else
-c 'DROP SCHEMA public CASCADE; CREATE SCHEMA public;' >/dev/null 2>&1 || true
fi
if [ "${RESEED:-0}" = "1" ]; then
# Loki: the container keeps its data inside itself, so recreating it
# is a clean wipe. Done whenever RESEED=1, even if ES is empty (e.g.
# the ES container was recreated): Loki does not dedupe, so seeding on
# top of leftover logs would append a duplicate copy.
"${COMPOSE[@]}" up -d --force-recreate loki >/dev/null 2>&1 || true
wait_for "Loki (after wipe)" curl -sf http://localhost:3100/ready
fi
# Resolve a Python environment for the seed and make sure parsedmarc plus
# every output integration ([all]) and the PostgreSQL extra (psycopg) are
# installed in it, so the same run can populate Elasticsearch, OpenSearch,
@@ -429,7 +440,10 @@ else
echo " ${PARSEDMARC_BIN} — the seed run will fail to initialize its"
echo " outputs. Install them with: pip install -e '.[all,postgresql]'"
fi
env "${pg_seed_env[@]}" \
# The Loki output is part of the base install, so it is always seeded.
# Lines keep the default ingest timestamp: Loki rejects samples older
# than a week, and the sample reports are years old.
env "${pg_seed_env[@]}" PARSEDMARC_LOKI_URL=http://localhost:3100 \
"$PARSEDMARC_BIN" -t 2.0 --dns-retries 1 -c parsedmarc-dev.ini "${SAMPLE_FILES[@]}" || true
fi
@@ -563,6 +577,28 @@ EOF
echo " created datasource 'PostgreSQL'"
fi
# Loki datasource for the Loki DMARC dashboard. Created with uid dmarc-loki
# when missing; an existing datasource named Loki keeps its own uid, which the
# dashboard import below resolves its ${DS_LOKI} input to.
LOKI_DS_UID=dmarc-loki
loki_ds_resp=$(curl -sS -u "${GRAFANA_USER}:${GRAFANA_PASSWORD}" \
-w $'\n%{http_code}' \
"http://localhost:3000/api/datasources/name/Loki")
loki_ds_code="${loki_ds_resp##*$'\n'}"
if [ "$loki_ds_code" = "200" ]; then
LOKI_DS_UID=$(printf '%s' "${loki_ds_resp%$'\n'*}" \
| python3 -c 'import json, sys; print(json.load(sys.stdin)["uid"])')
echo " datasource 'Loki' already exists (uid ${LOKI_DS_UID}) — skipping"
else
curl -sS -u "${GRAFANA_USER}:${GRAFANA_PASSWORD}" \
-H 'Content-Type: application/json' \
-X POST "http://localhost:3000/api/datasources" \
-d '{"name": "Loki", "uid": "dmarc-loki", "type": "loki", "url": "http://loki:3100", "access": "proxy", "isDefault": false}' \
| sed 's/^/ /'
echo
echo " created datasource 'Loki'"
fi
log "Importing Grafana dashboard"
GF_BODY=$(python3 -c '
import json, sys
@@ -597,6 +633,23 @@ curl -sS -u "${GRAFANA_USER}:${GRAFANA_PASSWORD}" \
-X POST "http://localhost:3000/api/dashboards/db" \
-d "$GF_PG_BODY" | sed 's/^/ /'
log "Importing Grafana Loki dashboard"
GF_LOKI_BODY=$(LOKI_DS_UID="$LOKI_DS_UID" python3 -c '
import json, os
with open("dashboards/grafana/Grafana-DMARC_Reports-Loki.json") as f:
text = f.read()
text = text.replace("${DS_LOKI}", os.environ["LOKI_DS_UID"])
d = json.loads(text)
d.pop("__inputs", None)
d.pop("__requires", None)
d["id"] = None
print(json.dumps({"dashboard": d, "overwrite": True, "folderUid": ""}))
')
curl -sS -u "${GRAFANA_USER}:${GRAFANA_PASSWORD}" \
-H 'Content-Type: application/json' \
-X POST "http://localhost:3000/api/dashboards/db" \
-d "$GF_LOKI_BODY" | sed 's/^/ /'
log "Importing Splunk dashboard views into the DMARC app"
splunk_import_view() {
local name="$1"
@@ -626,4 +679,5 @@ cat <<EOF
Grafana http://localhost:3000/ (${GRAFANA_USER} / ${GRAFANA_PASSWORD})
Splunk http://localhost:8000/ (admin / ${SPLUNK_PASSWORD})
PostgreSQL localhost:5432 (${PG_USER} / ${PG_PASSWORD}, db ${PG_DB})
Loki http://localhost:3100/
EOF