amorfo77/parsedmarc
1
0
Fork 0
mirror of https://github.com/domainaware/parsedmarc.git synced 2026-09-06 22:18:01 +00:00
Code Issues Projects Releases Wiki Activity
Files
07b00a931bc2a8b4984e2c975b2736507baccaf4
parsedmarc/dashboards/README.md
T
Sean WhalenClaude Fable 5Copilot
 07bca1ad28 Make the output and mailbox integrations optional extras (#888) 
* Make the output and mailbox integrations optional extras (#883)

Breaking change for the next major release: pip install parsedmarc now
installs the parsing core plus a working core CLI (file, IMAP, Maildir,
and mbox input; CSV/JSON, Splunk HEC, webhook, and syslog output).
Everything else moves behind an extra: elastic, opensearch, kafka, s3,
gelf, loganalytics, msgraph, and gmail, joining the existing postgresql
extra, with an umbrella [all] that deliberately excludes postgresql
(psycopg's binary wheels do not exist on every platform, so
parsedmarc[all] must never fail to install there).

cli.py imports the six SDK-dependent output modules behind the #884
TYPE_CHECKING/try-except guard; a configured section whose extra is
missing fails fast with a ConfigurationError naming the section and the
exact pip install command — including the msgraph and gmail_api mailbox
sections (detected via parsedmarc.mail's placeholder classes) and
postgresql (checked before the constructor so the startup retry loop
does not retry a missing dependency for a minute). The Azure/kiota Graph
error types fall back to never-raised sentinel classes.

The Docker image installs [all,postgresql], so container users see no
change. CI lint installs [build,all,postgresql]; the unit-test job
installs [build,all], deliberately without postgresql so
test_postgres.py's absent-psycopg arm stays exercised. The
never-imported dateparser dependency is dropped in favor of declaring
python-dateutil, which utils.py actually imports; pytz moves to the
build extra for the one test that uses it.

Verified live: a no-extras wheel install imports, parses samples, and
reports the install hint for each gated section; a [all] install
restores every integration; the Docker image builds with every SDK
importable.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Patch psycopg presence in the PostgreSQL CLI wiring tests

CI's unit-test job deliberately installs [build,all] without the
postgresql extra, so parsedmarc.cli.postgres.psycopg is None there and
the new missing-extra presence check correctly made _main exit 1 before
the wiring under test ran. The tests simulate the SDK being available
(PostgreSQLClient is mocked at the SDK boundary), so the module-level
psycopg handle is now patched present in setUp. Verified against a
simulated psycopg-absent environment as well as the local full install.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Address Copilot review: narrow guards to ModuleNotFoundError, fix docs

- The optional-integration and Graph error-type import guards now catch
  ModuleNotFoundError instead of ImportError, so only a genuinely absent
  package reads as a missing extra; a broken-but-present SDK fails
  loudly with its real error instead of masquerading as one. The test
  blocker raises ModuleNotFoundError accordingly — the exact exception a
  missing package produces.
- _missing_extra_hint docstring no longer calls every gated integration
  an output module (it also serves the msgraph/gmail_api mailbox
  sections).
- Fix the pre-existing passsword typo in usage.md's kafka section; the
  INI key the code reads is password (cli.py _parse_config).

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Quote extras specs in copy-paste install commands

From Copilot's second review round: zsh treats an unquoted .[build,all]
as a glob and fails with 'no matches found', so the commands shown in
AGENTS.md, CONTRIBUTING.md, dashboards/README.md, and the bootstrap
script's comment are now quoted. The CI workflows keep the unquoted
form: they run under bash, which passes unmatched globs through
literally. The suggestion to change the 'Choosing what to install'
heading level was rejected — it is a subsection of 'Installing
parsedmarc', matching the file's existing hierarchy.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Fix upgrade command in the changelog

* Documentation review: accuracy, spelling, grammar, and clarity pass

A full prose review of docs/source, README, CONTRIBUTING, and the
dashboards README, with every accuracy claim verified against the code
before changing it. Highlights:

- usage.md: documented six missing [general] options (the CSV/JSON
  filename options, prettify_json, normalize_timespan_threshold_hours),
  the required kafka smtp_tls_topic, [imap] timeout/max_retries, and
  the postgresql env-var prefix; corrected the maildir_path default
  (None, not INBOX — cli.py Namespace defaults), the mailbox
  check_timeout option name, the systemd restart interval (RestartSec
  is 5m), and merged the duplicate silent entry; quoted every
  copy-paste extras spec for zsh safety.
- elasticsearch.md: fixed an invalid openssl command (rsa:4096 -nodes),
  the dashboards filename (opensearch_dashboards.ndjson, matching the
  file the link serves), and assorted grammar.
- davmail.md: the service-enable command now enables davmail.service
  (was parsedmarc.service — a copy-paste error that left DavMail
  unenabled), plus a view typo and DavMail capitalization.
- output.md: the example schema reference is RFC 7489 Appendix C
  (7480 is RDAP). kibana.md: SPF relies on the SMTP envelope, not
  session headers (RFC 7208). dmarc.md: DKM -> DKIM.
- README: the intro now also names the OpenSearch/Grafana stack,
  matching the feature list. CONTRIBUTING: pre-PR checks now include
  ruff format --check and pyright, matching CI's lint job.
- dashboards/README: the service table and seed description now include
  the PostgreSQL backend the compose stack runs.

Sample data blocks, the CLI-help mirror block, and released CHANGELOG
entries were deliberately left untouched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Docstring review: accuracy, spelling, grammar, and clarity pass

Every docstring in parsedmarc/, parsedmarc/mail/, the maps maintainer
scripts, and the test suite reviewed with each claim verified against
the code it documents. Text-only — no behavior changes. Highlights:

- Copy-paste errors corrected: parsed_smtp_tls_reports_to_csv and
  splunk/loganalytics save functions described aggregate or failure
  reports they do not handle; LogAnalyticsException claimed to be an
  Elasticsearch error.
- Docstring/behavior mismatches: parse_report_email's report_type
  enumeration omitted smtp_tls; parse_failure_report typed msg_date as
  str (it is datetime); strip_attachment_payloads claimed payloads are
  replaced with None (the key is deleted); kafkaclient's failure and
  SMTP TLS savers claimed per-record slicing while sending the whole
  list in one message (docstrings now describe reality — whether
  slicing was intended is flagged for follow-up); the postgres savers
  claimed to take parse_report_file's return value but receive the
  inner report dict; elastic/opensearch save functions' Raises listed
  only AlreadySaved.
- None-as-semantic-state documented where missing (get_base_domain,
  get_ip_address_country), enumeration completeness fixed
  (get_ip_address_info's 9 result keys, maps script outputs, TSV
  columns), and the stale 44-industry-types count corrected to the
  46 the authoritative README list defines.
- Test docstrings aligned with what the tests actually assert,
  including two that overstated coverage of the elastic/opensearch
  address-list tests.
- Two argparse help strings fixed: file_path now names SMTP TLS report
  files alongside aggregate and failure, mirrored into usage.md's
  CLI-help block; --offline's doubled spaces removed (rendered help
  unchanged).
- elasticsearch.md's security claim corrected against Elastic's docs:
  security is enabled and auto-configured on first startup since 8.0
  (not "8.7 secure mode"), so the settings are verified, not
  hand-written.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-08-28 17:33:18 -04:00

121 lines
8.0 KiB
Markdown
Raw Blame History

# Dashboard development
This directory holds the dashboard sources that ship with parsedmarc:
- [opensearch/opensearch_dashboards.ndjson](opensearch/opensearch_dashboards.ndjson) — the source-of-truth saved-objects export. It is imported into both **OpenSearch Dashboards** and **Kibana** (the file format is compatible with both).
- [grafana/Grafana-DMARC_Reports.json](grafana/Grafana-DMARC_Reports.json) — the Grafana dashboard, with two Elasticsearch datasources (`dmarc-ag`, `dmarc-fo`).
- [grafana/Grafana-DMARC_Reports-PostgreSQL.json](grafana/Grafana-DMARC_Reports-PostgreSQL.json) — the Grafana dashboard for the PostgreSQL backend.
- [splunk/](splunk/) — three Splunk dashboard XML views (`dmarc_aggregate`, `dmarc_failure`, `smtp_tls`).
Edits to any of these files should be exported from a running instance after authoring the change in the UI, not hand-edited (with the occasional exception of small XML tweaks for Splunk).
## The dev stack
[docker-compose.dashboard-dev.yml](../docker-compose.dashboard-dev.yml) brings up every viz target at once so a single dashboard change can be authored and re-exported across all four UIs in one session. It `include:`s [docker-compose.yml](../docker-compose.yml) for the Elasticsearch and OpenSearch backends, then layers on Kibana, OpenSearch Dashboards, Grafana, Splunk, and a PostgreSQL backend.
| Service | URL | Credentials |
| --------------------- | ------------------------------------------------ | ------------------------------------------------------ |
| Elasticsearch | http://localhost:9200 | (security disabled) |
| OpenSearch | https://localhost:9201 | `admin` / `$OPENSEARCH_INITIAL_ADMIN_PASSWORD` |
| Kibana | http://localhost:5601 | (security disabled) |
| OpenSearch Dashboards | http://localhost:5602 | `admin` / `$OPENSEARCH_INITIAL_ADMIN_PASSWORD` |
| Grafana | http://localhost:3000 | `admin` / `$GRAFANA_PASSWORD` |
| PostgreSQL | localhost:5432 | `parsedmarc` / `parsedmarc` (override: `POSTGRESQL_*`) |
| Splunk Web / HEC | http://localhost:8000 / https://localhost:8088 | `admin` / `$SPLUNK_PASSWORD`, HEC token `$SPLUNK_HEC_TOKEN` |
All ports bind to `127.0.0.1` only.
## Prerequisites
1. Docker with the Compose v2 plugin.
2. A repo-root `.env` defining the secrets the compose file references:
```ini
OPENSEARCH_INITIAL_ADMIN_PASSWORD=...
SPLUNK_PASSWORD=...
SPLUNK_HEC_TOKEN=...
GRAFANA_PASSWORD=...
```
Pick any values you like — these are local-only dev secrets. Both `.env` and `parsedmarc*.ini` are gitignored. The matching values must also appear in [parsedmarc-dev.ini](../parsedmarc-dev.ini), which the bootstrap script feeds to the parsedmarc CLI for sample-data ingestion.
3. The parsedmarc CLI on `PATH` (or in `./venv/bin/`) — `pip install -e ".[all,postgresql]"` from the repo root works. The `all` and `postgresql` extras are what the bootstrap script installs, and the seed needs them: the Elasticsearch, OpenSearch, and PostgreSQL outputs each live behind an extra. Override the lookup with `PARSEDMARC_BIN=/path/to/parsedmarc` if needed.
## One-shot bootstrap
[dashboard-dev-bootstrap.sh](../dashboard-dev-bootstrap.sh) is the normal entry point. It is idempotent — re-run it any time:
```bash
./dashboard-dev-bootstrap.sh
```
It does, in order:
1. `docker compose -f docker-compose.dashboard-dev.yml up -d` and waits for every service's health endpoint.
2. Provisions Splunk: creates the `email` index, creates the `DMARC` app, configures the auto-created HEC token to allow the `email` index, and scopes the search-app's "scheduled export" announcement view away from `global` so it stops appearing in the DMARC app's dashboard list.
3. Seeds Elasticsearch, OpenSearch, Splunk, and PostgreSQL with parsedmarc-parsed sample reports (from [samples/](../samples/)) so the dashboards render against real data. Skipped when ES already has aggregate docs — pass `RESEED=1` to wipe and re-seed all four backends.
4. Imports the dashboard files from this directory into the running services. This step always runs, so the typical edit loop is **edit in the UI → export → save into this directory → re-run the bootstrap script** to verify the file imports cleanly into a fresh service.
VS Code users can run this via the **Dev Dashboard: Bootstrap** task in [.vscode/tasks.json](../.vscode/tasks.json). **Dev Dashboard: Up** brings the stack up without importing or seeding.
## Editing a dashboard
After running the bootstrap script once, the round trip for each platform is:
### OpenSearch Dashboards (and Kibana)
1. Edit the dashboard at http://localhost:5602/ (OpenSearch Dashboards) — this is the canonical authoring surface.
2. **Stack Management → Saved Objects → Export**, select the DMARC dashboard, include related objects, and save the resulting `.ndjson` over [opensearch/opensearch_dashboards.ndjson](opensearch/opensearch_dashboards.ndjson).
3. Re-run `./dashboard-dev-bootstrap.sh` to confirm it re-imports cleanly into both OSD and Kibana. The Kibana CI workflow ([.github/workflows/dashboards.yml](../.github/workflows/dashboards.yml)) also imports the same file on every PR that touches it.
OSD imports default to the `global_tenant` so other admins on the instance can see the result. Set `OSD_TENANT=...` to import elsewhere.
### Grafana
1. Edit the dashboard at http://localhost:3000/.
2. **Dashboard settings → JSON Model**, copy the JSON, save it to [grafana/Grafana-DMARC_Reports.json](grafana/Grafana-DMARC_Reports.json).
3. Re-run the bootstrap script.
The bootstrap script provisions two `elasticsearch` datasources (`dmarc-ag` for `dmarc_aggregate*`, `dmarc-fo` for `dmarc_f*`, which matches both pre-rename `dmarc_forensic*` and post-rename `dmarc_failure*`) on first run; existing datasources are left alone.
### Splunk
1. Edit the dashboard at http://localhost:8000/ inside the **DMARC** app.
2. Open the dashboard's **Source** view, copy the XML, and paste it over the matching file in [splunk/](splunk/) (`dmarc_aggregate_dashboard.xml`, `dmarc_failure_dashboard.xml`, or `smtp_tls_dashboard.xml`).
3. Re-run the bootstrap script. It re-imports each view via `DELETE` + `POST` to the splunkd management API.
## Screenshotting the dashboards
[dashboard-dev-screenshots.py](../dashboard-dev-screenshots.py) captures how
each UI actually renders the current sample data — useful for verifying a
dashboard change end-to-end and for PR evidence. One-time setup, then run
against the live stack:
```bash
pip install playwright && playwright install chromium
set -a; . ./.env; set +a
./dashboard-dev-screenshots.py # all four UIs
./dashboard-dev-screenshots.py grafana # or a subset
```
Output lands in `dashboard-screenshots/` (gitignored). The script encodes
the platform quirks that otherwise cost time to rediscover: Kibana/OSD
dashboards never reach Playwright's "networkidle" (fixed render waits are
used instead), OSD must be pinned to `?security_tenant=global` or a stale
private-tenant copy may be captured, and Grafana/Splunk need their login
forms driven rather than HTTP basic auth.
## Reseeding sample data
```bash
RESEED=1 ./dashboard-dev-bootstrap.sh
```
Wipes every `dmarc_aggregate*` / `dmarc_failure*` / `dmarc_forensic*` / `smtp_tls*` index from ES and OS, drops and recreates the Splunk `email` index and the PostgreSQL schema, then re-runs the parsedmarc CLI against the curated sample list. Use this after changing parsedmarc's enrichment or output schemas.
## Tearing the stack down
```bash
docker compose -f docker-compose.dashboard-dev.yml down # stop containers, keep volumes
docker compose -f docker-compose.dashboard-dev.yml down -v # also drop volumes (full reset)
```
Reference in New Issue View Git Blame Copy Permalink