Fix over-time charts double-counting reports via multi-valued date_range

date_range on ES/OpenSearch aggregate and SMTP TLS documents is a
two-element array [begin, end]. A date histogram buckets a document once
per value, so every over-time chart bucketing on date_range counted a
report twice whenever its begin and end dates fell in different buckets.
Range filtering on it was also wrong: a report spanning the whole window
matches neither endpoint.

Measured on the dev-stack sample data: a 1d histogram on date_range
returns doc_count 4592 / message sum 4724 against true totals of
2300 / 2427; the same histogram on date_begin returns exactly
2300 / 2427.

All date histograms (2 OSD/Kibana visualizations, 10 Grafana ES panels
including the summary pies) and all time-range filters (24 Grafana
target timeFields, the dmarc_aggregate* and smtp_tls* index-pattern
timeFieldName, the dev-stack dmarc-ag datasource) now use the
single-valued date_begin, matching the report-begin semantics of the
PostgreSQL (begin_date) and Splunk (_time = interval begin) dashboards.
Failure-report panels already used the single-valued arrival_date and
are unchanged.

Dev stack: installing the Elasticsearch datasource plugin via
GF_INSTALL_PLUGINS crash-loops Grafana >= 13 (the image ships a
root-owned plugins-bundled/elasticsearch remnant the background
installer cannot replace), so the bootstrap script now installs it via
grafana cli and restarts Grafana instead.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

CHANGELOG.md

@@ -7,6 +7,10 @@
 - **Added a per-domain DMARC compliance percentage to the aggregate dashboards of every provider** ([#112](https://github.com/domainaware/parsedmarc/issues/112)): OpenSearch Dashboards/Kibana, Grafana Elasticsearch, Grafana PostgreSQL, and Splunk. The from-domain volume table on each dashboard is now "Message volume and DMARC compliance by from domain", with columns for From Domain, Messages, and % DMARC Compliant.
   - On OpenSearch Dashboards/Kibana, the table is now a TSVB visualization using a Filter Ratio metric (passed messages over total messages per `header_from`), since the previous agg-based data table can't compute a per-domain ratio. Editing the imported visualization on Kibana 8.x requires first enabling the `metrics:allowStringIndices` advanced setting.
 
+### Bug fixes
+
+- **The Elasticsearch/OpenSearch aggregate dashboards' over-time charts (and the Grafana ES dashboard's summary pies and time series) bucketed on the multi-valued `date_range` field**; a date histogram counts a report once per value, double-counting any report whose begin and end dates fall in different buckets. All date histograms and time-range filters now use the single-valued `date_begin`, matching the report-begin semantics of the PostgreSQL (`begin_date`) and Splunk (`_time` = interval begin) dashboards.
+
 ## 10.2.4
 
 ### Bug fixes

dashboard-dev-bootstrap.sh

@@ -340,6 +340,35 @@ curl -sS -X POST 'http://localhost:5602/api/saved_objects/_import?overwrite=true
     --form file=@dashboards/opensearch/opensearch_dashboards.ndjson | sed 's/^/  /'
 echo "  (imported into OSD tenant: ${OSD_TENANT})"
 
+log "Ensuring Grafana Elasticsearch datasource plugin is installed"
+# Grafana >= 13 no longer bundles the Elasticsearch datasource plugin, and
+# GF_INSTALL_PLUGINS cannot install it (the image ships a root-owned
+# plugins-bundled/elasticsearch remnant its background installer fails to
+# replace). `grafana cli` installs into /var/lib/grafana/plugins, which works;
+# a restart is needed for Grafana to load it.
+code=$(curl -sS -u "${GRAFANA_USER}:${GRAFANA_PASSWORD}" \
+    -o /dev/null -w "%{http_code}" \
+    "http://localhost:3000/api/plugins/elasticsearch/settings")
+if [ "$code" != "200" ]; then
+    "${COMPOSE[@]}" exec -T grafana grafana cli plugins install elasticsearch \
+        | sed 's/^/  /'
+    "${COMPOSE[@]}" restart grafana >/dev/null
+    for _ in $(seq 1 36); do
+        curl -sf -o /dev/null "http://localhost:3000/api/health" && break
+        sleep 5
+    done
+    code=$(curl -sS -u "${GRAFANA_USER}:${GRAFANA_PASSWORD}" \
+        -o /dev/null -w "%{http_code}" \
+        "http://localhost:3000/api/plugins/elasticsearch/settings")
+    if [ "$code" != "200" ]; then
+        echo "ERROR: elasticsearch datasource plugin failed to install" >&2
+        exit 1
+    fi
+    echo "  installed elasticsearch datasource plugin"
+else
+    echo "  elasticsearch datasource plugin already installed"
+fi
+
 log "Configuring Grafana datasources"
 # Two Elasticsearch datasources, one per index family, matching the dashboard's
 # template variables (dmarc-ag and dmarc-fo). Skipped when already present.
@@ -347,7 +376,7 @@ declare -a GF_DS_NAMES=("dmarc-ag" "dmarc-fo")
 # dmarc_f* matches both pre-rename dmarc_forensic* and post-rename
 # dmarc_failure* indices, mirroring the OpenSearch/Kibana dashboards.
 declare -a GF_DS_INDEX=("dmarc_aggregate*" "dmarc_f*")
-declare -a GF_DS_TIME=("date_range" "arrival_date")
+declare -a GF_DS_TIME=("date_begin" "arrival_date")
 for i in 0 1; do
     name="${GF_DS_NAMES[$i]}"
     code=$(curl -sS -u "${GRAFANA_USER}:${GRAFANA_PASSWORD}" \

dashboards/grafana/Grafana-DMARC_Reports.json

@@ -188,7 +188,7 @@
             },
             {
               "$$hashKey": "object:244",
-              "field": "date_range",
+              "field": "date_begin",
               "id": "2",
               "settings": {
                 "fixed_interval": "auto",
@@ -211,7 +211,7 @@
           ],
           "query": "header_from.keyword:$fromdomain",
           "refId": "A",
-          "timeField": "date_range"
+          "timeField": "date_begin"
         }
       ],
       "timeFrom": null,
@@ -277,7 +277,7 @@
               "type": "terms"
             },
             {
-              "field": "date_range",
+              "field": "date_begin",
               "id": "2",
               "settings": {
                 "fixed_interval": "auto",
@@ -299,7 +299,7 @@
           ],
           "query": "header_from.keyword:$fromdomain",
           "refId": "A",
-          "timeField": "date_range"
+          "timeField": "date_begin"
         }
       ],
       "timeFrom": null,
@@ -369,7 +369,7 @@
             },
             {
               "$$hashKey": "object:386",
-              "field": "date_range",
+              "field": "date_begin",
               "id": "2",
               "settings": {
                 "fixed_interval": "auto",
@@ -392,7 +392,7 @@
           ],
           "query": "header_from.keyword:$fromdomain",
           "refId": "A",
-          "timeField": "date_range"
+          "timeField": "date_begin"
         }
       ],
       "timeFrom": null,
@@ -622,7 +622,7 @@
               "type": "terms"
             },
             {
-              "field": "date_range",
+              "field": "date_begin",
               "id": "2",
               "settings": {
                 "fixed_interval": "1d",
@@ -644,7 +644,7 @@
           ],
           "query": "header_from.keyword:$fromdomain",
           "refId": "A",
-          "timeField": "date_range"
+          "timeField": "date_begin"
         }
       ],
       "timeFrom": null,
@@ -852,7 +852,7 @@
               "type": "terms"
             },
             {
-              "field": "date_range",
+              "field": "date_begin",
               "id": "2",
               "settings": {
                 "fixed_interval": "1d",
@@ -874,7 +874,7 @@
           ],
           "query": "header_from.keyword:$fromdomain",
           "refId": "A",
-          "timeField": "date_range"
+          "timeField": "date_begin"
         }
       ],
       "timeFrom": null,
@@ -977,7 +977,7 @@
               "type": "terms"
             },
             {
-              "field": "date_range",
+              "field": "date_begin",
               "id": "2",
               "settings": {
                 "fixed_interval": "1d",
@@ -1001,7 +1001,7 @@
           ],
           "query": "header_from.keyword:$fromdomain",
           "refId": "A",
-          "timeField": "date_range"
+          "timeField": "date_begin"
         }
       ],
       "timeFrom": null,
@@ -1104,7 +1104,7 @@
               "type": "terms"
             },
             {
-              "field": "date_range",
+              "field": "date_begin",
               "id": "2",
               "settings": {
                 "fixed_interval": "1d",
@@ -1128,7 +1128,7 @@
           ],
           "query": "header_from.keyword:$fromdomain",
           "refId": "A",
-          "timeField": "date_range"
+          "timeField": "date_begin"
         }
       ],
       "timeFrom": null,
@@ -1231,7 +1231,7 @@
               "type": "terms"
             },
             {
-              "field": "date_range",
+              "field": "date_begin",
               "id": "2",
               "settings": {
                 "fixed_interval": "1d",
@@ -1255,7 +1255,7 @@
           ],
           "query": "header_from.keyword:$fromdomain",
           "refId": "A",
-          "timeField": "date_range"
+          "timeField": "date_begin"
         }
       ],
       "timeFrom": null,
@@ -1373,7 +1373,7 @@
               "type": "terms"
             },
             {
-              "field": "date_range",
+              "field": "date_begin",
               "id": "2",
               "settings": {
                 "fixed_interval": "1d",
@@ -1395,7 +1395,7 @@
           ],
           "query": "header_from.keyword:$fromdomain",
           "refId": "A",
-          "timeField": "date_range"
+          "timeField": "date_begin"
         }
       ],
       "timeFrom": null,
@@ -1463,7 +1463,7 @@
             {
               "$$hashKey": "object:430",
               "fake": true,
-              "field": "date_range",
+              "field": "date_begin",
               "id": "6",
               "settings": {
                 "fixed_interval": "auto",
@@ -1486,7 +1486,7 @@
           ],
           "query": "header_from.keyword:$fromdomain",
           "refId": "A",
-          "timeField": "date_range"
+          "timeField": "date_begin"
         }
       ],
       "timeFrom": null,
@@ -1606,7 +1606,7 @@
           ],
           "query": "header_from.keyword:$fromdomain",
           "refId": "A",
-          "timeField": "date_range"
+          "timeField": "date_begin"
         }
       ],
       "timeFrom": null,
@@ -1756,7 +1756,7 @@
           ],
           "query": "header_from.keyword:$fromdomain",
           "refId": "A",
-          "timeField": "date_range"
+          "timeField": "date_begin"
         },
         {
           "bucketAggs": [
@@ -1788,7 +1788,7 @@
           ],
           "query": "header_from.keyword:$fromdomain",
           "refId": "B",
-          "timeField": "date_range"
+          "timeField": "date_begin"
         }
       ],
       "timeFrom": null,
@@ -1951,7 +1951,7 @@
           ],
           "query": "header_from.keyword:$fromdomain",
           "refId": "A",
-          "timeField": "date_range"
+          "timeField": "date_begin"
         },
         {
           "bucketAggs": [
@@ -1983,7 +1983,7 @@
           ],
           "query": "header_from.keyword:$fromdomain AND passed_dmarc:true",
           "refId": "B",
-          "timeField": "date_range"
+          "timeField": "date_begin"
         }
       ],
       "timeFrom": null,
@@ -2104,7 +2104,7 @@
           ],
           "query": "header_from.keyword:$fromdomain",
           "refId": "A",
-          "timeField": "date_range"
+          "timeField": "date_begin"
         }
       ],
       "timeFrom": null,
@@ -2282,7 +2282,7 @@
           ],
           "query": "header_from.keyword:$fromdomain",
           "refId": "A",
-          "timeField": "date_range"
+          "timeField": "date_begin"
         }
       ],
       "timeFrom": null,
@@ -2651,7 +2651,7 @@
           ],
           "query": "header_from.keyword:$fromdomain",
           "refId": "A",
-          "timeField": "date_range"
+          "timeField": "date_begin"
         }
       ],
       "timeFrom": null,
@@ -3004,7 +3004,7 @@
           ],
           "query": "header_from.keyword:$fromdomain",
           "refId": "A",
-          "timeField": "date_range"
+          "timeField": "date_begin"
         }
       ],
       "timeFrom": null,
@@ -3257,7 +3257,7 @@
           ],
           "query": "header_from.keyword:$fromdomain",
           "refId": "A",
-          "timeField": "date_range"
+          "timeField": "date_begin"
         }
       ],
       "timeFrom": null,
@@ -3484,7 +3484,7 @@
           ],
           "query": "header_from.keyword:$fromdomain",
           "refId": "A",
-          "timeField": "date_range"
+          "timeField": "date_begin"
         }
       ],
       "timeFrom": null,
@@ -3729,7 +3729,7 @@
           ],
           "query": "header_from.keyword:$fromdomain",
           "refId": "A",
-          "timeField": "date_range"
+          "timeField": "date_begin"
         }
       ],
       "timeFrom": null,
@@ -4642,7 +4642,7 @@
           ],
           "query": "header_from.keyword:$fromdomain",
           "refId": "A",
-          "timeField": "date_range"
+          "timeField": "date_begin"
         }
       ],
       "timeFrom": null,
@@ -4826,7 +4826,7 @@
           ],
           "query": "header_from.keyword:$fromdomain",
           "refId": "A",
-          "timeField": "date_range"
+          "timeField": "date_begin"
         }
       ],
       "timeFrom": null,

docker-compose.dashboard-dev.yml

@@ -31,9 +31,11 @@ services:
       # to "admin" so the login matches the bootstrap script's GRAFANA_PASSWORD
       # default; set GRAFANA_PASSWORD in .env to change both in lockstep.
       - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_PASSWORD:-admin}
-      # "elasticsearch" is required on Grafana >= 13, which no longer bundles
-      # the Elasticsearch datasource as a core plugin.
-      - GF_INSTALL_PLUGINS=grafana-piechart-panel,grafana-worldmap-panel,elasticsearch
+      # Grafana >= 13 no longer bundles the Elasticsearch datasource, but
+      # installing it via GF_INSTALL_PLUGINS crash-loops: the image ships a
+      # root-owned plugins-bundled/elasticsearch remnant the installer cannot
+      # replace. The bootstrap script installs it via `grafana cli` instead.
+      - GF_INSTALL_PLUGINS=grafana-piechart-panel,grafana-worldmap-panel
     ports:
       - "127.0.0.1:3000:3000"
     depends_on: