mirror of
https://github.com/domainaware/parsedmarc.git
synced 2026-11-06 16:59:23 +00:00
768bf75480
* feat: add grafana loki output Push aggregate, failure and SMTP TLS reports as JSON log lines to the Loki push API. Static labels and labels taken from report fields are configurable, host defaults to the FQDN. The timestamp is either the ingest time or the report begin time. Supports basic auth, bearer tokens, X-Scope-OrgID and batching. Push errors are reported as output errors so messages are not archived. Signed-off-by: Sebastian Schubert <basti@schubert.digital> * feat(dashboards): add grafana dashboard for the loki output Signed-off-by: Sebastian Schubert <basti@schubert.digital> * Address Copilot review on the Loki output Responds to Copilot's review of #926: - Treat any push status other than 200/204 as a failure. Loki returns block_ingestion_status_code (260 by default) when block_ingestion_until blocks ingestion, and the logs are discarded (Loki configuration reference); the old >= 300 check counted that as a successful save, so the CLI could archive mailbox messages whose reports were never stored. - Read aggregate begin_date as UTC for timestamp = report_begin. Aggregate CSV rows carry the record's interval_begin, which the parser builds as naive UTC wall-clock time (to_utc=True); reading it as local time shifted entries by the host's UTC offset (+4h on America/New_York). - Group the Published Policies dashboard dedup by org_name as well as report_id: report IDs are only unique per reporter (the PostgreSQL key is UNIQUE (org_name, report_id, ...)). - Drop the legend sum from SMTP TLS Sessions Over Time: its [1d] lookback with the automatic step yields overlapping rolling totals, so the sum counted the same sessions repeatedly. Adds regression tests for each, including a non-UTC timezone test against a real sample report. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Address second Copilot review on the Loki output Responds to Copilot's second review of #926 (previously-missed findings): - Strip each [loki] label_fields entry. _str_to_list() only lstrips, so "report_type , domain" produced "report_type " and LokiClient refused to start with "Invalid Loki label name 'report_type '". - Wipe Loki on RESEED=1 even when Elasticsearch is empty. The wipe sat inside the ES-has-docs guard, so after the ES container was recreated a reseed appended a second copy of the samples to Loki, which does not deduplicate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Address third Copilot review on the Loki output Responds to Copilot's third review of #926 (previously-missed findings): - Keep a base URL's query string when appending the push path. The path was string-concatenated, so https://gw.example.org?route=logs became ...?route=logs/loki/api/v1/push, hitting the root path with a corrupted query. The path is now set on the parsed httpx.URL. - Push each report's entries in timestamp order. With timestamp = report_begin, a normalized multi-day aggregate report yields rows source-major (every day of one source, then day 1 of the next), all in one stream. Loki rejects entries older than the newest line minus half of max_chunk_age (1h by default) with too_far_behind (Loki configuration docs, "Accept out-of-order writes"), so later sources were partly rejected. The whole entry list is now sorted before batching. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(loki): do not follow redirects and drop forensic alias httpx re-sends a 301/302 POST as GET, so a redirected push could return 200 without storing anything, and the mailbox messages were archived. Redirects are no longer followed, so a 3xx raises LokiError. Remove save_forensic_report_to_loki: the module is new, so no caller needs the legacy alias. Signed-off-by: Sebastian Schubert <basti@schubert.digital> * fix(dashboards): use the existing Loki datasource uid in the dev bootstrap When a datasource named Loki already exists with another uid, the dashboard import still bound every panel to dmarc-loki. Read the uid from the lookup response instead. Signed-off-by: Sebastian Schubert <basti@schubert.digital> --------- Signed-off-by: Sebastian Schubert <basti@schubert.digital> Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
82 lines
2.4 KiB
YAML
82 lines
2.4 KiB
YAML
name: parsedmarc-dashboards
|
|
|
|
include:
|
|
- docker-compose.yml
|
|
|
|
services:
|
|
kibana:
|
|
image: docker.elastic.co/kibana/kibana:8.19.7
|
|
environment:
|
|
- ELASTICSEARCH_HOSTS=http://elasticsearch:9200
|
|
ports:
|
|
- "127.0.0.1:5601:5601"
|
|
depends_on:
|
|
elasticsearch:
|
|
condition: service_healthy
|
|
|
|
opensearch-dashboards:
|
|
image: opensearchproject/opensearch-dashboards:3
|
|
environment:
|
|
- OPENSEARCH_HOSTS=["https://opensearch:9200"]
|
|
ports:
|
|
- "127.0.0.1:5602:5601"
|
|
depends_on:
|
|
opensearch:
|
|
condition: service_healthy
|
|
|
|
grafana:
|
|
image: grafana/grafana:latest
|
|
environment:
|
|
# Grafana reads GF_SECURITY_ADMIN_PASSWORD, not GRAFANA_PASSWORD. Default
|
|
# to "admin" so the login matches the bootstrap script's GRAFANA_PASSWORD
|
|
# default; set GRAFANA_PASSWORD in .env to change both in lockstep.
|
|
- GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD:-admin}
|
|
# Grafana >= 13 no longer bundles the Elasticsearch datasource, but
|
|
# installing it via GF_INSTALL_PLUGINS crash-loops: the image ships a
|
|
# root-owned plugins-bundled/elasticsearch remnant the installer cannot
|
|
# replace. The bootstrap script installs it via `grafana cli` instead.
|
|
- GF_INSTALL_PLUGINS=grafana-piechart-panel,grafana-worldmap-panel
|
|
ports:
|
|
- "127.0.0.1:3000:3000"
|
|
depends_on:
|
|
elasticsearch:
|
|
condition: service_healthy
|
|
postgresql:
|
|
condition: service_healthy
|
|
loki:
|
|
condition: service_started
|
|
|
|
loki:
|
|
image: grafana/loki:latest
|
|
ports:
|
|
- "127.0.0.1:3100:3100"
|
|
|
|
postgresql:
|
|
image: postgres:17
|
|
environment:
|
|
- POSTGRES_USER=${POSTGRESQL_USER:-parsedmarc}
|
|
- POSTGRES_PASSWORD=${POSTGRESQL_PASSWORD:-parsedmarc}
|
|
- POSTGRES_DB=${POSTGRESQL_DB:-parsedmarc}
|
|
ports:
|
|
- "127.0.0.1:5432:5432"
|
|
healthcheck:
|
|
test:
|
|
[
|
|
"CMD-SHELL",
|
|
"pg_isready -U ${POSTGRESQL_USER:-parsedmarc} -d ${POSTGRESQL_DB:-parsedmarc}"
|
|
]
|
|
interval: 5s
|
|
timeout: 5s
|
|
retries: 20
|
|
|
|
splunk:
|
|
image: splunk/splunk:latest
|
|
environment:
|
|
- SPLUNK_START_ARGS=--accept-license
|
|
- "SPLUNK_GENERAL_TERMS=--accept-sgt-current-at-splunk-com"
|
|
- SPLUNK_PASSWORD=${SPLUNK_PASSWORD}
|
|
- SPLUNK_HEC_TOKEN=${SPLUNK_HEC_TOKEN}
|
|
ports:
|
|
- "127.0.0.1:8000:8000"
|
|
- "127.0.0.1:8088:8088"
|