Bastischubert/loki output (#926)

* feat: add grafana loki output

Push aggregate, failure and SMTP TLS reports as JSON log lines to the
Loki push API. Static labels and labels taken from report fields are
configurable, host defaults to the FQDN. The timestamp is either the
ingest time or the report begin time. Supports basic auth, bearer
tokens, X-Scope-OrgID and batching. Push errors are reported as output
errors so messages are not archived.

Signed-off-by: Sebastian Schubert <basti@schubert.digital>

* feat(dashboards): add grafana dashboard for the loki output

Signed-off-by: Sebastian Schubert <basti@schubert.digital>

* Address Copilot review on the Loki output

Responds to Copilot's review of #926:

- Treat any push status other than 200/204 as a failure. Loki returns
  block_ingestion_status_code (260 by default) when block_ingestion_until
  blocks ingestion, and the logs are discarded (Loki configuration
  reference); the old >= 300 check counted that as a successful save, so
  the CLI could archive mailbox messages whose reports were never stored.
- Read aggregate begin_date as UTC for timestamp = report_begin. Aggregate
  CSV rows carry the record's interval_begin, which the parser builds as
  naive UTC wall-clock time (to_utc=True); reading it as local time shifted
  entries by the host's UTC offset (+4h on America/New_York).
- Group the Published Policies dashboard dedup by org_name as well as
  report_id: report IDs are only unique per reporter (the PostgreSQL key
  is UNIQUE (org_name, report_id, ...)).
- Drop the legend sum from SMTP TLS Sessions Over Time: its [1d] lookback
  with the automatic step yields overlapping rolling totals, so the sum
  counted the same sessions repeatedly.

Adds regression tests for each, including a non-UTC timezone test against
a real sample report.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Address second Copilot review on the Loki output

Responds to Copilot's second review of #926 (previously-missed findings):

- Strip each [loki] label_fields entry. _str_to_list() only lstrips, so
  "report_type , domain" produced "report_type " and LokiClient refused to
  start with "Invalid Loki label name 'report_type '".
- Wipe Loki on RESEED=1 even when Elasticsearch is empty. The wipe sat
  inside the ES-has-docs guard, so after the ES container was recreated a
  reseed appended a second copy of the samples to Loki, which does not
  deduplicate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* Address third Copilot review on the Loki output

Responds to Copilot's third review of #926 (previously-missed findings):

- Keep a base URL's query string when appending the push path. The path
  was string-concatenated, so https://gw.example.org?route=logs became
  ...?route=logs/loki/api/v1/push, hitting the root path with a corrupted
  query. The path is now set on the parsed httpx.URL.
- Push each report's entries in timestamp order. With timestamp =
  report_begin, a normalized multi-day aggregate report yields rows
  source-major (every day of one source, then day 1 of the next), all in
  one stream. Loki rejects entries older than the newest line minus half
  of max_chunk_age (1h by default) with too_far_behind (Loki configuration
  docs, "Accept out-of-order writes"), so later sources were partly
  rejected. The whole entry list is now sorted before batching.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(loki): do not follow redirects and drop forensic alias

httpx re-sends a 301/302 POST as GET, so a redirected push could return 200 without storing anything, and the mailbox messages were archived. Redirects are no longer followed, so a 3xx raises LokiError.

Remove save_forensic_report_to_loki: the module is new, so no caller needs the legacy alias.

Signed-off-by: Sebastian Schubert <basti@schubert.digital>

* fix(dashboards): use the existing Loki datasource uid in the dev bootstrap

When a datasource named Loki already exists with another uid, the dashboard import still bound every panel to dmarc-loki. Read the uid from the lookup response instead.

Signed-off-by: Sebastian Schubert <basti@schubert.digital>

---------

Signed-off-by: Sebastian Schubert <basti@schubert.digital>
Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

docs/source/index.md

@@ -44,7 +44,7 @@ and Valimail.
   PostgreSQL, for use with premade dashboards
 - Optionally send the results to Apache Kafka, Amazon S3, Azure Log
   Analytics (Microsoft Sentinel), a Graylog (GELF) endpoint, a syslog server,
-  or an HTTP webhook
+  Grafana Loki, or an HTTP webhook
 
 ## Python Compatibility
 

docs/source/installation.md

@@ -106,7 +106,7 @@ dependencies it actually uses.
 - the parsing library (aggregate, failure, and SMTP TLS reports),
 - the CLI, reading reports from files, an IMAP mailbox, a Maildir, or an
   mbox file,
-- and CSV/JSON, Splunk HEC, webhook, and syslog output.
+- and CSV/JSON, Splunk HEC, webhook, syslog, and Grafana Loki output.
 
 Everything else needs an extra:
 

docs/source/usage.md

@@ -117,6 +117,10 @@ aggregate_url = https://aggregate_url.example.com
 failure_url = https://failure_url.example.com
 smtp_tls_url = https://smtp_tls_url.example.com
 timeout = 60
+
+[loki]
+url = http://loki.example.com:3100
+labels = host=mail.example.com
 ```
 
 The full set of configuration options is:
@@ -911,6 +915,49 @@ The full set of configuration options is:
   - `smtp_tls_url` - str: URL of the webhook which should receive the SMTP TLS reports
   - `timeout` - int: Timeout in seconds for webhook requests (Default: `60`)
 
+- `loki` - Push every report record as one JSON log line to the
+  [Grafana Loki push API](https://grafana.com/docs/loki/latest/reference/loki-http-api/#ingest-logs).
+  The lines are the same flat records the syslog and GELF outputs send,
+  plus `report_type` (`aggregate`, `failure`, or `smtp_tls`) and `domain`
+  (the reported domain: `domain`, `reported_domain`, or `policy_domain`),
+  so LogQL's `| json` parser can filter and sum them, e.g. messages failing
+  DMARC per domain:
+  `sum by (domain) (sum_over_time({job="parsedmarc", report_type="aggregate"} | json dmarc_aligned, count | dmarc_aligned="false" | unwrap count [1d]))`.
+  Name the fields to extract: a bare `| json` turns every field into a
+  label, and the many distinct values quickly exceed Loki's per-query
+  series limit (`max_query_series`, 500 by default).
+  A ready-made Grafana dashboard for these lines ships in
+  [dashboards/grafana/Grafana-DMARC_Reports-Loki.json](https://github.com/domainaware/parsedmarc/blob/master/dashboards/grafana/Grafana-DMARC_Reports-Loki.json);
+  import it and pick your Loki datasource for its `DS_LOKI` input.
+  It expects the default `job=parsedmarc` label and `label_fields`.
+  - `url` - str: The Loki base URL; `/loki/api/v1/push` is appended when the
+    URL has no path. A URL with a path is used as is.
+  - `labels` - str: Comma-separated `name=value` stream labels, merged over
+    the defaults `job=parsedmarc` and `host=<FQDN of the parsedmarc host>`
+    (in a container, the FQDN is the container's hostname)
+  - `label_fields` - str: Comma-separated record fields used as stream
+    labels (Default: `report_type, domain`; empty for none). Keep these to
+    fields with few distinct values — every combination is its own Loki
+    stream, so fields like `source_ip_address` belong in the line only.
+    Records without a value for a field get no such label.
+  - `timestamp` - str: `ingest` (the time of the push) or `report_begin`
+    (the begin of the reported period, or the arrival date of a failure
+    report) (Default: `ingest`). Loki rejects entries older than its
+    `reject_old_samples_max_age` (one week by default) and, within a
+    stream, entries far older than the newest one, so `report_begin`
+    loses late or out-of-order reports unless Loki is configured for them.
+    The reported period is in the line either way (`begin_date`,
+    `end_date`).
+  - `username` - str: Basic authentication user
+  - `password` - str: Basic authentication password
+  - `token` - str: Bearer token, instead of basic authentication
+  - `tenant_id` - str: Tenant ID, sent as the `X-Scope-OrgID` header
+  - `skip_certificate_verification` - bool: Skip TLS certificate
+    verification (not recommended)
+  - `timeout` - float: Timeout in seconds for push requests (Default: `60`)
+  - `batch_size` - int: Maximum log lines per push request
+    (Default: `1000`)
+
 :::{warning}
 It is **strongly recommended** to **not** use the `nameservers`
 setting. By default, `parsedmarc` uses
@@ -1070,7 +1117,7 @@ an already-saved report — are unaffected, and S3 is idempotent because
 each report is written to an object key built from its type, date, report
 ID, and a hash of the fields that identify the report, so a retry
 overwrites the same object. Kafka,
-Splunk HEC, syslog, GELF, webhooks, Azure Log Analytics, and the
+Splunk HEC, syslog, GELF, webhooks, Loki, Azure Log Analytics, and the
 `--output` JSON/CSV files all append unconditionally, so each retry adds
 another copy of every report in the batch. That is why the default retry
 cap is deliberately low: at most three deliveries per report before its
@@ -1086,8 +1133,8 @@ syslog and GELF outputs send through Python logging handlers, which
 swallow delivery errors by design — so an unreachable webhook, syslog, or
 GELF endpoint is *not* treated as a failed save and does not hold a
 batch's messages back. Failures in Elasticsearch, OpenSearch, Splunk HEC,
-Kafka, S3, PostgreSQL, Azure Log Analytics, and the `--output` files are
-all detected and do.
+Kafka, S3, PostgreSQL, Loki, Azure Log Analytics, and the `--output` files
+are all detected and do.
 :::
 
 :::{note}
@@ -1233,6 +1280,7 @@ For sections with underscores in the name, the full section name is used:
 | `log_analytics` | `PARSEDMARC_LOG_ANALYTICS_` |
 | `gelf` | `PARSEDMARC_GELF_` |
 | `webhook` | `PARSEDMARC_WEBHOOK_` |
+| `loki` | `PARSEDMARC_LOKI_` |
 
 ## Using parsedmarc as a library