7 Commits

Include renames

Author	SHA1	Message	Date
3 people	768bf75480	Bastischubert/loki output (#926) ... * feat: add grafana loki output Push aggregate, failure and SMTP TLS reports as JSON log lines to the Loki push API. Static labels and labels taken from report fields are configurable, host defaults to the FQDN. The timestamp is either the ingest time or the report begin time. Supports basic auth, bearer tokens, X-Scope-OrgID and batching. Push errors are reported as output errors so messages are not archived. Signed-off-by: Sebastian Schubert <basti@schubert.digital> * feat(dashboards): add grafana dashboard for the loki output Signed-off-by: Sebastian Schubert <basti@schubert.digital> * Address Copilot review on the Loki output Responds to Copilot's review of #926: - Treat any push status other than 200/204 as a failure. Loki returns block_ingestion_status_code (260 by default) when block_ingestion_until blocks ingestion, and the logs are discarded (Loki configuration reference); the old >= 300 check counted that as a successful save, so the CLI could archive mailbox messages whose reports were never stored. - Read aggregate begin_date as UTC for timestamp = report_begin. Aggregate CSV rows carry the record's interval_begin, which the parser builds as naive UTC wall-clock time (to_utc=True); reading it as local time shifted entries by the host's UTC offset (+4h on America/New_York). - Group the Published Policies dashboard dedup by org_name as well as report_id: report IDs are only unique per reporter (the PostgreSQL key is UNIQUE (org_name, report_id, ...)). - Drop the legend sum from SMTP TLS Sessions Over Time: its [1d] lookback with the automatic step yields overlapping rolling totals, so the sum counted the same sessions repeatedly. Adds regression tests for each, including a non-UTC timezone test against a real sample report. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Address second Copilot review on the Loki output Responds to Copilot's second review of #926 (previously-missed findings): - Strip each [loki] label_fields entry. _str_to_list() only lstrips, so "report_type , domain" produced "report_type " and LokiClient refused to start with "Invalid Loki label name 'report_type '". - Wipe Loki on RESEED=1 even when Elasticsearch is empty. The wipe sat inside the ES-has-docs guard, so after the ES container was recreated a reseed appended a second copy of the samples to Loki, which does not deduplicate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Address third Copilot review on the Loki output Responds to Copilot's third review of #926 (previously-missed findings): - Keep a base URL's query string when appending the push path. The path was string-concatenated, so https://gw.example.org?route=logs became ...?route=logs/loki/api/v1/push, hitting the root path with a corrupted query. The path is now set on the parsed httpx.URL. - Push each report's entries in timestamp order. With timestamp = report_begin, a normalized multi-day aggregate report yields rows source-major (every day of one source, then day 1 of the next), all in one stream. Loki rejects entries older than the newest line minus half of max_chunk_age (1h by default) with too_far_behind (Loki configuration docs, "Accept out-of-order writes"), so later sources were partly rejected. The whole entry list is now sorted before batching. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(loki): do not follow redirects and drop forensic alias httpx re-sends a 301/302 POST as GET, so a redirected push could return 200 without storing anything, and the mailbox messages were archived. Redirects are no longer followed, so a 3xx raises LokiError. Remove save_forensic_report_to_loki: the module is new, so no caller needs the legacy alias. Signed-off-by: Sebastian Schubert <basti@schubert.digital> * fix(dashboards): use the existing Loki datasource uid in the dev bootstrap When a datasource named Loki already exists with another uid, the dashboard import still bound every panel to dmarc-loki. Read the uid from the lookup response instead. Signed-off-by: Sebastian Schubert <basti@schubert.digital> --------- Signed-off-by: Sebastian Schubert <basti@schubert.digital> Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>	2026-10-31 10:13:37 -04:00
Sean WhalenandClaude Fable 5	0d832ff682	Make dashboard-dev-bootstrap.sh work with Docker or Podman (#891) ... The bootstrap script hardcoded `docker compose`, locking out contributors on Podman-based systems. It now auto-detects a working container engine (Docker preferred when both are usable) and its Compose implementation (`docker compose`/`docker-compose`, `podman compose`/`podman-compose`), selectable explicitly with --backend docker|podman or the CONTAINER_BACKEND environment variable (the flag wins). Detection requires a live `<engine> info` call, so a leftover docker CLI with no running daemon does not shadow a working podman, and both compose forms are probed with `version` so a broken install fails at startup instead of partway through the run. Adds -h/--help documenting the flag and the script's env knobs. Every container invocation already flowed through the single COMPOSE array, so the rest of the script is unchanged. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-08-28 22:47:00 -04:00
3 people	07bca1ad28	Make the output and mailbox integrations optional extras (#888) ... * Make the output and mailbox integrations optional extras (#883) Breaking change for the next major release: pip install parsedmarc now installs the parsing core plus a working core CLI (file, IMAP, Maildir, and mbox input; CSV/JSON, Splunk HEC, webhook, and syslog output). Everything else moves behind an extra: elastic, opensearch, kafka, s3, gelf, loganalytics, msgraph, and gmail, joining the existing postgresql extra, with an umbrella [all] that deliberately excludes postgresql (psycopg's binary wheels do not exist on every platform, so parsedmarc[all] must never fail to install there). cli.py imports the six SDK-dependent output modules behind the #884 TYPE_CHECKING/try-except guard; a configured section whose extra is missing fails fast with a ConfigurationError naming the section and the exact pip install command — including the msgraph and gmail_api mailbox sections (detected via parsedmarc.mail's placeholder classes) and postgresql (checked before the constructor so the startup retry loop does not retry a missing dependency for a minute). The Azure/kiota Graph error types fall back to never-raised sentinel classes. The Docker image installs [all,postgresql], so container users see no change. CI lint installs [build,all,postgresql]; the unit-test job installs [build,all], deliberately without postgresql so test_postgres.py's absent-psycopg arm stays exercised. The never-imported dateparser dependency is dropped in favor of declaring python-dateutil, which utils.py actually imports; pytz moves to the build extra for the one test that uses it. Verified live: a no-extras wheel install imports, parses samples, and reports the install hint for each gated section; a [all] install restores every integration; the Docker image builds with every SDK importable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Patch psycopg presence in the PostgreSQL CLI wiring tests CI's unit-test job deliberately installs [build,all] without the postgresql extra, so parsedmarc.cli.postgres.psycopg is None there and the new missing-extra presence check correctly made _main exit 1 before the wiring under test ran. The tests simulate the SDK being available (PostgreSQLClient is mocked at the SDK boundary), so the module-level psycopg handle is now patched present in setUp. Verified against a simulated psycopg-absent environment as well as the local full install. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address Copilot review: narrow guards to ModuleNotFoundError, fix docs - The optional-integration and Graph error-type import guards now catch ModuleNotFoundError instead of ImportError, so only a genuinely absent package reads as a missing extra; a broken-but-present SDK fails loudly with its real error instead of masquerading as one. The test blocker raises ModuleNotFoundError accordingly — the exact exception a missing package produces. - _missing_extra_hint docstring no longer calls every gated integration an output module (it also serves the msgraph/gmail_api mailbox sections). - Fix the pre-existing passsword typo in usage.md's kafka section; the INI key the code reads is password (cli.py _parse_config). Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Quote extras specs in copy-paste install commands From Copilot's second review round: zsh treats an unquoted .[build,all] as a glob and fails with 'no matches found', so the commands shown in AGENTS.md, CONTRIBUTING.md, dashboards/README.md, and the bootstrap script's comment are now quoted. The CI workflows keep the unquoted form: they run under bash, which passes unmatched globs through literally. The suggestion to change the 'Choosing what to install' heading level was rejected — it is a subsection of 'Installing parsedmarc', matching the file's existing hierarchy. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix upgrade command in the changelog * Documentation review: accuracy, spelling, grammar, and clarity pass A full prose review of docs/source, README, CONTRIBUTING, and the dashboards README, with every accuracy claim verified against the code before changing it. Highlights: - usage.md: documented six missing [general] options (the CSV/JSON filename options, prettify_json, normalize_timespan_threshold_hours), the required kafka smtp_tls_topic, [imap] timeout/max_retries, and the postgresql env-var prefix; corrected the maildir_path default (None, not INBOX — cli.py Namespace defaults), the mailbox check_timeout option name, the systemd restart interval (RestartSec is 5m), and merged the duplicate silent entry; quoted every copy-paste extras spec for zsh safety. - elasticsearch.md: fixed an invalid openssl command (rsa:4096 -nodes), the dashboards filename (opensearch_dashboards.ndjson, matching the file the link serves), and assorted grammar. - davmail.md: the service-enable command now enables davmail.service (was parsedmarc.service — a copy-paste error that left DavMail unenabled), plus a view typo and DavMail capitalization. - output.md: the example schema reference is RFC 7489 Appendix C (7480 is RDAP). kibana.md: SPF relies on the SMTP envelope, not session headers (RFC 7208). dmarc.md: DKM -> DKIM. - README: the intro now also names the OpenSearch/Grafana stack, matching the feature list. CONTRIBUTING: pre-PR checks now include ruff format --check and pyright, matching CI's lint job. - dashboards/README: the service table and seed description now include the PostgreSQL backend the compose stack runs. Sample data blocks, the CLI-help mirror block, and released CHANGELOG entries were deliberately left untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Docstring review: accuracy, spelling, grammar, and clarity pass Every docstring in parsedmarc/, parsedmarc/mail/, the maps maintainer scripts, and the test suite reviewed with each claim verified against the code it documents. Text-only — no behavior changes. Highlights: - Copy-paste errors corrected: parsed_smtp_tls_reports_to_csv and splunk/loganalytics save functions described aggregate or failure reports they do not handle; LogAnalyticsException claimed to be an Elasticsearch error. - Docstring/behavior mismatches: parse_report_email's report_type enumeration omitted smtp_tls; parse_failure_report typed msg_date as str (it is datetime); strip_attachment_payloads claimed payloads are replaced with None (the key is deleted); kafkaclient's failure and SMTP TLS savers claimed per-record slicing while sending the whole list in one message (docstrings now describe reality — whether slicing was intended is flagged for follow-up); the postgres savers claimed to take parse_report_file's return value but receive the inner report dict; elastic/opensearch save functions' Raises listed only AlreadySaved. - None-as-semantic-state documented where missing (get_base_domain, get_ip_address_country), enumeration completeness fixed (get_ip_address_info's 9 result keys, maps script outputs, TSV columns), and the stale 44-industry-types count corrected to the 46 the authoritative README list defines. - Test docstrings aligned with what the tests actually assert, including two that overstated coverage of the elastic/opensearch address-list tests. - Two argparse help strings fixed: file_path now names SMTP TLS report files alongside aggregate and failure, mirrored into usage.md's CLI-help block; --offline's doubled spaces removed (rendered help unchanged). - elasticsearch.md's security claim corrected against Elastic's docs: security is enabled and auto-configured on first startup since 8.0 (not "8.7 secure mode"), so the settings are verified, not hand-written. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>	2026-08-28 17:33:18 -04:00
Sean WhalenandClaude Fable 5	72bece4686	Add dashboard screenshot harness to the dev tooling (#840) ... * Add dashboard screenshot harness to the dev tooling dashboard-dev-screenshots.py drives headless Chromium (Playwright) against the dashboard dev stack and captures how Kibana, OpenSearch Dashboards, Grafana, and Splunk actually render the current sample data — for end-to-end verification of dashboard changes and PR evidence. It encodes the platform quirks that otherwise cost time to rediscover: fixed render waits instead of networkidle (Kibana/OSD dashboards poll forever), pinning OSD to the global tenant (a stale private-tenant copy silently screenshots old dashboards), and driving the Grafana and Splunk login forms rather than HTTP basic auth. The output directory is gitignored; usage is documented in dashboards/README.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address Copilot review findings on the screenshot harness - Only require OPENSEARCH_INITIAL_ADMIN_PASSWORD / SPLUNK_PASSWORD when the osd / splunk targets are selected, and fail fast before launching Playwright with a clear message naming the missing variable(s). - Honor GRAFANA_USER (defaulting to admin) to match the dev bootstrap script instead of hard-coding the Grafana username. - Print a full traceback to stderr when a target fails, instead of just the exception message, while still continuing to the next target. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Always close the browser when a screenshot target fails Copilot round 2: each target only closed its Chromium instance on the happy path, so a mid-run Playwright failure leaked a headless browser while the script continued to the next target. Wrap each target body in try/finally; closing the browser also closes its contexts and pages, so the single b.close() covers the OSD context too. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Exit with an install hint when playwright is missing Copilot round 3: playwright is deliberately not a project dependency, so a bare import failure produced a raw ModuleNotFoundError traceback. Catch ImportError and exit with the same one-time install command the module docstring documents. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-07-22 17:01:07 -04:00
Sean WhalenandClaude Opus 4.7	2b3cd32b9c	chore: move PostgreSQL Grafana dashboard into dashboards/grafana/ (#780) ... The PostgreSQL dashboard shipped at the repo-root grafana/ by oversight; every other dashboard source lives under dashboards/ (opensearch/, grafana/, splunk/). Move it next to the existing Grafana dashboard, list it in dashboards/README.md, and fix the CHANGELOG path reference. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-21 13:52:33 -04:00
Sean WhalenandClaude Opus 4.7	a6778707d7	Finish forensic→failure rename: archive-folder migration + dashboard/doc cleanup (#776) ... The forensic→failure rename (#659) left a few loose ends and one deliberate hold-back. This closes them. Leftover rename misses (broken paths / stale canonical names): - CONTRIBUTING.md, dashboard-dev-bootstrap.sh: samples/forensic/* → samples/failure/* - dashboard-dev-bootstrap.sh, dashboards/README.md: dmarc_forensic_dashboard.xml → dmarc_failure_dashboard.xml (the file was already renamed; the import path and view name were not) - docs/source/usage.md: PARSEDMARC_GENERAL_SAVE_FORENSIC → ..._SAVE_FAILURE example - samples/parsedmarc.ini: save_forensic → save_failure - pyproject.toml, README.md: canonical "failure" naming (ci.ini intentionally keeps save_forensic to smoke-test the deprecated alias.) Archive subfolder rename + on-startup migration: - New failure reports now archive to <archive>/Failure (was <archive>/Forensic). - _migrate_forensic_archive_folder() runs once on startup (best-effort): renames Forensic→Failure when no Failure folder exists yet, merges the two when both exist, no-ops when there's no legacy folder, and logs-and-skips a mailbox it can't reorganize (warn, don't crash). This consolidates pre- and post-rename failure reports into one folder, replacing the previously documented decision to keep the folder named Forensic to avoid a split archive. Uses the folder-management API (folder_exists / rename_folder / merge_folders) added in mailsuite 2.1.0; the pin is bumped to >=2.1.0. Grafana dashboard (the rename PR updated OSD/Splunk/ES-OS but not Grafana): - Forensic panel titles + the datasource label → Failure; the fo-column display label and its linked byName field-override matcher both → "Failure Policy" (changed together so the column-width override keeps matching). - dev-bootstrap Grafana ES datasource: dmarc_forensic* → dmarc_f* (matches both pre-rename dmarc_forensic* and post-rename dmarc_failure*, like the OSD/Kibana dashboards); RESEED wipe loop now also clears dmarc_failure* indices. - Removed dashboards/grafana/Grafana-DMARC_Reports.json-new_panel.json, an orphan export accidentally committed in #736 and referenced by nothing. Tests (tests/test_init.py): - TestMigrateForensicArchiveFolderMaildir: real on-disk Maildir round-trips via mailsuite's MaildirConnection (no mocks) — rename, merge, no-op, and the full get_dmarc_reports_from_mailbox orchestration. Runs in CI (no network/creds). - TestMigrateForensicArchiveFolderErrorHandling: the one path a real Maildir can't reproduce — a backend that raises mid-operation must warn, not crash. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-21 12:29:40 -04:00
Sean Whalen	fe296ca869	Update dashboard documentation ... - Introduced a new README.md for dashboard development with detailed instructions. - Removed outdated README files for Grafana and Splunk dashboards.	2026-05-03 12:36:06 -04:00