* feat: add grafana loki output
Push aggregate, failure and SMTP TLS reports as JSON log lines to the
Loki push API. Static labels and labels taken from report fields are
configurable, host defaults to the FQDN. The timestamp is either the
ingest time or the report begin time. Supports basic auth, bearer
tokens, X-Scope-OrgID and batching. Push errors are reported as output
errors so messages are not archived.
Signed-off-by: Sebastian Schubert <basti@schubert.digital>
* feat(dashboards): add grafana dashboard for the loki output
Signed-off-by: Sebastian Schubert <basti@schubert.digital>
* Address Copilot review on the Loki output
Responds to Copilot's review of #926:
- Treat any push status other than 200/204 as a failure. Loki returns
block_ingestion_status_code (260 by default) when block_ingestion_until
blocks ingestion, and the logs are discarded (Loki configuration
reference); the old >= 300 check counted that as a successful save, so
the CLI could archive mailbox messages whose reports were never stored.
- Read aggregate begin_date as UTC for timestamp = report_begin. Aggregate
CSV rows carry the record's interval_begin, which the parser builds as
naive UTC wall-clock time (to_utc=True); reading it as local time shifted
entries by the host's UTC offset (+4h on America/New_York).
- Group the Published Policies dashboard dedup by org_name as well as
report_id: report IDs are only unique per reporter (the PostgreSQL key
is UNIQUE (org_name, report_id, ...)).
- Drop the legend sum from SMTP TLS Sessions Over Time: its [1d] lookback
with the automatic step yields overlapping rolling totals, so the sum
counted the same sessions repeatedly.
Adds regression tests for each, including a non-UTC timezone test against
a real sample report.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Address second Copilot review on the Loki output
Responds to Copilot's second review of #926 (previously-missed findings):
- Strip each [loki] label_fields entry. _str_to_list() only lstrips, so
"report_type , domain" produced "report_type " and LokiClient refused to
start with "Invalid Loki label name 'report_type '".
- Wipe Loki on RESEED=1 even when Elasticsearch is empty. The wipe sat
inside the ES-has-docs guard, so after the ES container was recreated a
reseed appended a second copy of the samples to Loki, which does not
deduplicate.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* Address third Copilot review on the Loki output
Responds to Copilot's third review of #926 (previously-missed findings):
- Keep a base URL's query string when appending the push path. The path
was string-concatenated, so https://gw.example.org?route=logs became
...?route=logs/loki/api/v1/push, hitting the root path with a corrupted
query. The path is now set on the parsed httpx.URL.
- Push each report's entries in timestamp order. With timestamp =
report_begin, a normalized multi-day aggregate report yields rows
source-major (every day of one source, then day 1 of the next), all in
one stream. Loki rejects entries older than the newest line minus half
of max_chunk_age (1h by default) with too_far_behind (Loki configuration
docs, "Accept out-of-order writes"), so later sources were partly
rejected. The whole entry list is now sorted before batching.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix(loki): do not follow redirects and drop forensic alias
httpx re-sends a 301/302 POST as GET, so a redirected push could return 200 without storing anything, and the mailbox messages were archived. Redirects are no longer followed, so a 3xx raises LokiError.
Remove save_forensic_report_to_loki: the module is new, so no caller needs the legacy alias.
Signed-off-by: Sebastian Schubert <basti@schubert.digital>
* fix(dashboards): use the existing Loki datasource uid in the dev bootstrap
When a datasource named Loki already exists with another uid, the dashboard import still bound every panel to dmarc-loki. Read the uid from the lookup response instead.
Signed-off-by: Sebastian Schubert <basti@schubert.digital>
---------
Signed-off-by: Sebastian Schubert <basti@schubert.digital>
Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
