mirror of
https://github.com/domainaware/parsedmarc.git
synced 2026-07-25 01:44:55 +00:00
e98e12acb0454486246da463dd76abe0ef39e6f9
102
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  
|
62514bd72a |
Add per-domain DMARC compliance percentage to all aggregate dashboards (#834)
* Add per-domain DMARC compliance percentage to all aggregate dashboards (#112)
The from-domain volume table on every provider's aggregate dashboard is
now "Message volume and DMARC compliance by from domain" with columns
From Domain | Messages | % DMARC Compliant:
- OpenSearch Dashboards/Kibana: the agg-based data table is replaced by
a TSVB table using a Filter Ratio metric (passed_dmarc:true over all,
sum of message_count), pivoted on header_from.keyword. The time field
is date_begin rather than the multi-valued date_range, which TSVB's
per-value date histogram would double-count. Editing (not rendering)
the panel on Kibana 8.x requires the metrics:allowStringIndices
advanced setting.
- Grafana (Elasticsearch): a second passed_dmarc:true query joined by
field with a binary calculation (Sum 2 / Sum 1) rendered as percentunit.
- Grafana (PostgreSQL): compliance column via an aggregate FILTER clause,
COALESCEd so zero-pass domains show 0 instead of NULL.
- Splunk: sum(eval(if(passed_dmarc="true", message_count, 0))) inside
stats, per the SPL eval-in-stats syntax.
All four providers were verified against the same seeded sample data in
the dashboard dev stack; each returns identical per-domain values
(example.com: 2425 messages, 5.3% compliant).
Dev stack fixes found along the way: cap Elasticsearch heap at 2g (the
unset heap auto-sized to 50% of host RAM and was OOM-killed with
bootstrap.memory_lock on large hosts), and install the elasticsearch
datasource plugin in Grafana, which is no longer bundled as of
Grafana 13.
Closes #112
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Fix over-time charts double-counting reports via multi-valued date_range
date_range on ES/OpenSearch aggregate and SMTP TLS documents is a
two-element array [begin, end]. A date histogram buckets a document once
per value, so every over-time chart bucketing on date_range counted a
report twice whenever its begin and end dates fell in different buckets.
Range filtering on it was also wrong: a report spanning the whole window
matches neither endpoint.
Measured on the dev-stack sample data: a 1d histogram on date_range
returns doc_count 4592 / message sum 4724 against true totals of
2300 / 2427; the same histogram on date_begin returns exactly
2300 / 2427.
All date histograms (2 OSD/Kibana visualizations, 10 Grafana ES panels
including the summary pies) and all time-range filters (24 Grafana
target timeFields, the dmarc_aggregate* and smtp_tls* index-pattern
timeFieldName, the dev-stack dmarc-ag datasource) now use the
single-valued date_begin, matching the report-begin semantics of the
PostgreSQL (begin_date) and Splunk (_time = interval begin) dashboards.
Failure-report panels already used the single-valued arrival_date and
are unchanged.
Dev stack: installing the Elasticsearch datasource plugin via
GF_INSTALL_PLUGINS crash-loops Grafana >= 13 (the image ships a
root-owned plugins-bundled/elasticsearch remnant the background
installer cannot replace), so the bootstrap script now installs it via
grafana cli and restarts Grafana instead.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address Copilot review comments on PR #834
- kibana.md: "filter on our filter out" -> "filter on or filter out".
- OSD/Kibana export: fix "filed DMARC" -> "failed DMARC" and the
backticked `ruf ` trailing space in the RUF explainer panel, and
rename the "SMPT TLS failure details" visualization to "SMTP TLS
failure details" (object title and visState).
- dashboard-dev-bootstrap.sh: reuse wait_for() after the Grafana
plugin-install restart so a hang fails with a clear timeout message
instead of an opaque downstream curl error.
The ndjson changes were round-tripped through the dev-stack OSD
(import -> re-export from the global tenant) and re-import cleanly into
both OSD and Kibana 8.19.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* AGENTS.md: reviews must cover prose and hunk context, not just function
Codifies the lessons from the PR #834 Copilot review: whole-file
canonical dashboard exports put pre-existing titles/markdown in the
diff, so they get a text-level pass; proofread the full hunk around
prose edits, not only changed lines; and mid-incident glue code gets
the same review bar (and helper-reuse check) as planned code.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address second round of Copilot review comments
- CHANGELOG.md: rename the premature "10.2.5" heading to "Unreleased",
matching the repo convention where the release commit assigns the
version number (see
|
||
|
|
df9bf82e04 |
Post-review follow-ups for Graph send (#825/#826) and requests-to-httpx migration (#827)
Follow-ups from the review of PR #825 (whose implementation had already landed on master via #826's stacked merge): - Honor the documented [smtp] attachment and [smtp] message options. Both were parsed into opts but never passed to either summary-email transport (also broken in released 10.2.2), so a configured custom attachment filename or message body was silently ignored. Both the SMTP and Microsoft Graph transports now receive them, and the missing smtp_attachment Namespace default is added (also covers SIGHUP reload, which rebuilds opts from the CLI Namespace). - Don't mislabel non-Graph mailbox errors as Microsoft Graph failures: the shared mailbox-fetch and watch handlers now log a generic "Mailbox Error" with traceback when the connection isn't Graph. - Declare microsoft-kiota-abstractions as a direct dependency (imported directly in cli.py for Graph error handling; previously transitive). Migrate all runtime HTTP from requests to httpx (webhook client, Splunk HEC client, and the PSL-overrides / IP-database / reverse-DNS-map / IPinfo-API fetches in utils.py): - follow_redirects=True everywhere to preserve requests' default redirect-following; httpx does not follow redirects by default. - The PSL-overrides and reverse-DNS-map fetches gain a 60s timeout (previously none), matching the IP-database fetch. - response.ok -> response.is_success; requests.RequestException -> httpx.HTTPError; raw string bodies use content= (httpx's data= is form-encoding only); Splunk HEC verification moves to client construction (httpx has no per-request verify). - requests drops out of [project] dependencies and moves to the [build] extra for the out-of-wheel maintainer script collect_domain_info.py, which deliberately stays on requests/urllib3 for its permissive-TLS adapter. - Remove the requests-era module-level urllib3.disable_warnings(InsecureRequestWarning) in splunk.py; httpx doesn't route through urllib3, so its only remaining effect was globally silencing insecure-TLS warnings from other urllib3-based components as an import side effect. Nothing imports urllib3 directly anymore, so it also leaves [project] dependencies. Tests: config-to-transport wiring for attachment/message on both transports (including defaults), non-Graph errors keep the generic log line, webhook/Splunk payload assertions moved to content=, and Splunk verify asserted at httpx.Client construction. 736 passed; ruff and pyright clean. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
31c928d6fc |
Refresh Microsoft Graph docs: national clouds, examples, troubleshooting (#826)
* Send report summary via Microsoft Graph; make Graph failures observable
Two related fixes shipped together:
Send via Graph: the periodic DMARC summary email can now be sent
through the already-authenticated Microsoft Graph mailbox connection
(MSGraphConnection.send_message(), /users/{mailbox}/sendMail) instead
of only SMTP. Triggered when [msgraph] is configured and [smtp] has a
`to` value but no `host` -- SMTP is always preferred when `host` is
set, with no automatic fallback to Graph on SMTP failure. Reuses the
same connection used for reading; no new send-only config mode.
email_results()'s SMTP behavior is unchanged; a new
email_results_via_msgraph() shares its content-building logic via a
new _build_report_email_content() helper. Graph's sendMail always
sends as the authenticated mailbox, so [smtp] from is ignored on this
path -- documented, along with the required Mail.Send permissions and
a caveat that delegated auth flows (UsernamePassword/DeviceCode) don't
currently request that scope, so app-only auth is the supported path
for sending. Tracks #472.
Observable Graph failures: MSGraphConnection construction, mailbox
fetch, message send, and --watch failures now catch
ClientAuthenticationError/APIError/httpx.HTTPError specifically and
log one clear ERROR line naming the mailbox, tenant, auth method, and
the Graph request-id/client-request-id when available, instead of a
bare "MS Graph Error"/"Mailbox Error" with no context. Full traceback
still preserved at --debug. --watch previously had no Graph-specific
error handling at all -- a Graph error there crashed with a raw
uncaught traceback; it now exits the same way as the other three
sites. No new config options.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Refresh Microsoft Graph docs: national clouds, examples, troubleshooting
The [msgraph] docs were accurate but missed guidance the community has
been asking for:
- graph_url now lists the actual national/sovereign-cloud endpoint
values (GCC High, DoD, China/21Vianet), with an explicit warning
that setting it alone is not sufficient -- the Entra ID auth
endpoint isn't independently configurable in parsedmarc or
mailsuite, so it always hits the global login.microsoftonline.com.
- A minimal working [msgraph] example for every auth method
(UsernamePassword, DeviceCode, ClientSecret, Certificate,
ClientAssertion) -- previously only Certificate had one, entangled
with the SMTP-sending example.
- A reading-permission matrix alongside the existing sending one, so
every auth method x own/shared-mailbox combination is explicit in
one place for both directions.
- An accurate note on the parsedmarc-named token cache: it's a
deliberate backward-compatibility choice from the 9.11.0 mailsuite
extraction (mailsuite's own default cache name differs), not a
migration users need to act on.
- A troubleshooting table for four error scenarios, verified against
source rather than assumed: admin consent and folder-resolution
failures are still live and documented with real fixes; the
event-loop and ISO-timestamp errors are historical, already fixed
below this project's dependency/version floor.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
|
||
|
|
40509f801b |
Migrate Elasticsearch output to the elasticsearch-py 8.x client (#822)
* Migrate Elasticsearch output to the elasticsearch-py 8.x client (#806) The mandatory elasticsearch<7.14.0 + elasticsearch-dsl==7.4.0 pins transitively forced urllib3<2 (EOL 1.26.x) onto every install. The old <7.14.0 cap only existed to dodge the client product check that broke OpenSearch users (#452, #653) — obsolete now that parsedmarc has a dedicated [opensearch] backend on opensearch-py. - Depend on elasticsearch>=8.18,<9 and drop elasticsearch-dsl entirely (the DSL ships inside the client as elasticsearch.dsl since 8.18.0). The 8.x client's elastic-transport allows urllib3>=1.26.2,<3, so installs can now resolve urllib3 2.x. The 8.x line supports both Elasticsearch 8.x and 9.x servers; ES 7.x servers are no longer supported, and OpenSearch users pointing [elasticsearch] at an OpenSearch cluster must switch to the [opensearch] section. - set_hosts() now builds 8.x connection kwargs (scheme-qualified host URLs, request_timeout, basic_auth) while keeping the function signature and every INI option unchanged. - migrate_indexes() is now a documented no-op kept for API compatibility: its only migration (re-typing published_policy.fo from long to text) applied exclusively to indices carrying the legacy ES 6-era "doc" mapping type, which cannot exist on any server the 8.x client can reach. - The elasticsearch.dsl 8.x stubs use dataclass_transform and don't surface pre-8.x-style bare `name = Text()` fields as constructor parameters; each Document/InnerDoc class now carries a TYPE_CHECKING-only `__init__(*args, **kwargs)` declaration matching the real runtime signature, which also made nine pre-existing pyright ignores unnecessary. Verified with ruff, pyright (0 errors/0 warnings), the full pytest suite (718 passed), and a CLI run over the bundled samples; CI's live elasticsearch:8.19.7 service exercises the new client end-to-end. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Use pass instead of ... in TYPE_CHECKING __init__ stubs CodeQL flags an ellipsis-only body as "Statement has no effect" (12 alerts on PR #822); pass is equivalent at runtime and to the type checker and keeps the alerts from resurfacing on every future scan. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
3fda55d385 |
Make Microsoft Graph connection activity observable (#815)
* Make Microsoft Graph connection activity observable parsedmarc only configured its own logger, so all Graph connection activity was silently dropped even with --debug: the mailbox layer logs under mailsuite.mailbox.graph, token acquisition under azure.identity (including the AADSTS error codes that distinguish a local config problem from an Exchange Online / Entra ID one), and HTTP traffic under httpx/msgraph — none of which had a handler or level set. _main() also logged nothing around the MSGraphConnection call, so a hang left no trace at all. Three changes, all parsedmarc-side (no mailsuite changes needed): - Log a redacted connection summary at INFO before connecting (auth method, tenant ID, client ID, mailbox, Graph URL) plus a --debug detail line with certificate path, token-file path, and set/not-set flags for secrets. Secret values are never logged; a regression test asserts they don't appear in captured output. - Log a timing line after the connection object is initialized. - Propagate parsedmarc's --verbose/--debug level and handlers to the dependency loggers (mailsuite, azure, msgraph, httpx, httpcore) via _configure_dependency_logging(), synced to exactly the parsedmarc logger's handlers so SIGHUP log-file swaps neither duplicate output nor write to closed handlers. At the default level dependency loggers sit at WARNING, so their warnings keep surfacing (formatted) without new noise. All four new tests fail on the unfixed code (verified by stashing the cli.py change). Fixes https://github.com/domainaware/parsedmarc/issues/814. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Disable propagation on dependency loggers; document kiota's absence Set propagate=False on the dependency loggers when syncing handlers, so a stray logging.basicConfig() anywhere in the process cannot double-print every dependency record through the root logger — the function already owns these loggers' handler lists, and this makes that ownership complete. Asserted alongside the existing level/handler checks. kiota_http and its sibling packages were considered for _DEPENDENCY_LOGGERS but verified to not use Python logging at all (their observability is OpenTelemetry tracing), so a comment now records why they are absent rather than leaving the omission to be "fixed" later. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: MISAPOR LAB <misapor@lab.misapor.pl> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com> |
||
|
|
7fed72798a |
Stop system GeoIP files from shadowing the bundled IPinfo database (#813)
* Stop system GeoIP files from shadowing the bundled IPinfo database _get_ip_database_path() searched well-known system paths (including /usr/share/GeoIP/GeoLite2-Country.mmdb and CWD-relative names) before the database parsedmarc manages, so on any host with a distro GeoIP package installed every lookup silently used a country-only — and often years-old — database instead of the bundled IPinfo Lite one. That disabled ASN enrichment entirely (asn/as_name/as_domain were None for every IP) and with it the ASN-fallback path into the reverse-DNS map, with no signal beyond a generic "IP database is more than a month old" warning. Verified live on a Fedora host whose distro GeoLite2-Country.mmdb dated to December 2019. New precedence: explicit ip_db_path -> _IP_DB_PATH selected by load_ip_db() (downloaded/cached/bundled) -> the bundled copy -> system paths as a true last resort (only consulted when the bundled data file is missing). The selected file is logged at debug level so a --debug run shows which database answered. The automatic system-path pickup was documented behavior, so installation.md now tells MaxMind GeoLite2 users to set ip_db_path explicitly, with a migration note. Both new regression tests reproduce the shadowing portably via a decoy CWD GeoLite2-Country.mmdb (the fallback list includes relative names), and fail on the unfixed code (verified by stashing the source change). Fixes https://github.com/domainaware/parsedmarc/issues/810. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address review: accurate fallback comment, dedup selection log, cover fallback tiers - Correct the nothing-found-anywhere comment: the os.stat() age check raises FileNotFoundError before the caller's open_database() would. - Log "Using IP database at ..." only when the selected path changes instead of on every uncached IP lookup, so --debug runs over large batches aren't flooded; tracked via _LAST_LOGGED_IP_DB_PATH, reset in the test fixture for order-independence. - Cover the previously untested branches of _get_ip_database_path: system-path fallback when the bundled database is missing, the FileNotFoundError when nothing exists anywhere, the stale-database warning, and the log-once-per-path behavior. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: MISAPOR LAB <misapor@lab.misapor.pl> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com> |
||
|
|
fa8faa6b39 |
MS Graph: case-insensitive auth_method + ClientAssertion support (#809)
* draft fix MS Graph * Add ClientAssertion auth method support for MS Graph in cli.py MSGraphConnection/AuthMethod (via mailsuite) already supported ClientAssertion, but cli.py never parsed a client_assertion config value or passed it through, so it was unusable from the CLI. Wire config_msgraph.client_assertion through _parse_config and the MSGraphConnection call, and document the auth method (including its short-lived-JWT caveat vs. Certificate/ClientSecret for watch mode). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: MISAPOR LAB <misapor@lab.misapor.pl> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
 Ivan The GeekandGitHub
|
a6241d537e |
Fix grammatical error in project maintenance note (#805)
The sponsors note read "This is a project is maintained by one developer" in both README.md and docs/source/index.md; drop the stray "is a". |
||
|
d3510da3a6 |
feat: graceful SIGTERM/SIGINT shutdown for watch mode and one-shot CLI (#794)
* feat: graceful SIGTERM/SIGINT shutdown for watch mode and one-shot CLI Previously SIGTERM (systemctl stop, docker stop, Kubernetes pod termination) killed parsedmarc mid-batch, tearing output writes and silently dropping buffered Kafka records. Shutdown is now cooperative: - SIGTERM/SIGINT set a flag that is polled at safe boundaries. The one-shot CLI checks it between batches; watch mode passes it as `config_reloading` so the mailbox backend -- including the IMAP IDLE loop -- returns once the current batch is fully processed. Either way the in-flight batch and its output writes finish before the process exits 0. - Ctrl-C is a double-tap: the first press is graceful, the second short-circuits to os._exit(130). - Output clients are now closed on every exit path (atexit plus a trailing close in _main), fixing a long-standing leak where one-shot runs and graceful shutdowns never flushed Kafka / closed Elasticsearch / S3 / etc. Docs: the example systemd unit gains KillSignal=SIGTERM and TimeoutStopSec=60 (keep it above mailbox_check_timeout). Tests cover watch shutdown, the one-shot between-batch stop, the SIGINT double-tap, and the output-client-close leak. * test: cover the one-shot mbox-loop shutdown break Extend the one-shot SIGTERM test to also pass an .mbox path so a single run exercises both shutdown checkpoints: the file-batch loop break and the subsequent mbox loop break (which Codecov flagged as the only uncovered lines on PR #794). is_mbox is keyed by suffix and get_dmarc_reports_from_mbox is asserted not called, since the mbox loop breaks before reaching it. * test: narrow signal.getsignal() return before invoking in SIGINT test signal.getsignal() is typed Callable | int | Handlers | None; calling it directly fails pyright's callable check. Assert callable() first. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
b869235224 |
Build multi-arch (amd64+arm64) Docker images with PostgreSQL support (#793)
* Build multi-arch Docker images with PostgreSQL support The prebuilt image now installs the `[postgresql]` extra, so the optional PostgreSQL output backend (psycopg) works out of the box in the container without a separate `pip install` (#792). The wheel path is resolved into a variable before appending the extra so the shell doesn't treat `*.whl[postgresql]` as a bracket glob. The build workflow now sets up QEMU + Buildx and builds a multi-arch manifest for `linux/amd64` and `linux/arm64`, so the image runs natively on 64-bit ARM hosts such as a Raspberry Pi (#789). Every compiled dependency (psycopg[binary], lxml, maxminddb, cryptography) ships prebuilt aarch64 manylinux wheels, so the arm64 build adds no source-compilation step. A `pull_request` trigger (scoped to the build inputs) and `workflow_dispatch` are added so the multi-arch build can be validated on PRs and rebuilt on demand; pushes are still gated on the release event, so neither pushes images. Closes #789 Closes #792 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Bump version to 10.0.4 to publish the new images The docker workflow only pushes to the registry on a `release` event, so shipping the multi-arch + PostgreSQL-enabled image requires cutting a release. 10.0.3 is already tagged, so bump to 10.0.4 and document the Docker changes in the changelog. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Don't run the docker build on pull requests The pull_request trigger (added to validate the multi-arch build) re-ran the full ~10-minute amd64+arm64 build on every commit pushed to a docker-touching PR, because the pull_request `paths` filter matches against the PR's entire diff, not just the newest commit. That is wasteful once the build has been validated. Drop the pull_request trigger and rely on workflow_dispatch for on-demand validation (plus the existing master-push and release triggers). Also gate the registry login on the release event so that no non-release run authenticates to ghcr at all — a build can only ever be pushed from a published release. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
|
180fc581fe |
fix: OSD Global-tenant import + dropped report files with glob metacharacters; validate dev stack on OpenSearch 3.x with PostgreSQL (#781)
* fix: import OpenSearch dashboards into the real Global tenant dashboard-dev-bootstrap.sh sent `securitytenant: global_tenant`. The OpenSearch security plugin reads that header as a tenant *name*, and `global_tenant` is a sample custom tenant from the security demo config -- not the shared Global tenant, whose token is the literal `global`. The import therefore landed in a separate `global_tenant` tenant (its own `.kibana_<hash>_globaltenant_1` index) and the dashboards were invisible to anyone viewing the Global tenant in OpenSearch Dashboards. Verified against the live dev cluster: `_find` under `securitytenant: global` returned 26 objects and `.kibana_1` (the Global tenant index the UI reads) went from 2 to 67 docs after re-importing with the fix. An empty/omitted header read 0 from Global -- it falls back to the user's configured default tenant -- so `global` is the only reliable token. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: don't drop report files whose names contain glob metacharacters The CLI expanded every file argument with glob(), which treats [, ], *, and ? as pattern syntax. A literal path like "[Netease DMARC Failure Report] Rent Reminder.eml" -- the bracketed shape many providers use for emailed failure reports -- was read as a character class, matched nothing, and was dropped before reaching the parser, with no error. File arguments that exist on disk are now taken literally; only non-existent paths are globbed, so shell-style wildcards still expand. Also adds "postgresql" to _KNOWN_SECTIONS so PARSEDMARC_POSTGRESQL_* env vars (and their _FILE Docker-secret variants) resolve like every other backend -- the PostgreSQL backend is new in 10.0.0, so this completes the unreleased feature rather than fixing a released regression, and is documented under the PostgreSQL enhancement, not Bug fixes. Regression tests added for both. Verified end-to-end: all four samples/failure/*.eml now index (the bracketed Netease report included). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * dev: validate dashboards on OpenSearch 3.x and add PostgreSQL to the dev stack The dev stack ran OpenSearch Dashboards 3.x against OpenSearch 2.x, an unsupported cross-major pairing. Bump opensearch to :3 (validated on 3.6.0: OSD import into the Global tenant and all dashboards work). Add a postgresql service plus bootstrap wiring so the new PostgreSQL backend is exercised alongside the others: wait for PG, seed it via PARSEDMARC_POSTGRESQL_* env vars on the same parsedmarc run, wipe it on RESEED, create a Grafana grafana-postgresql-datasource (uid dmarc-pg), and import dashboards/grafana/Grafana-DMARC_Reports-PostgreSQL.json. PG seeding is gated on psycopg being importable: parsedmarc aborts the whole run (exit 1, nothing written to any backend) when a configured output backend can't initialize, so wiring in PG without the optional extra would silently zero ES/OS/Splunk too. When psycopg is absent the script warns and skips PG, leaving the other backends seeded. Also fix the Grafana admin password env: the container was given GRAFANA_PASSWORD, which Grafana ignores -- it reads GF_SECURITY_ADMIN_PASSWORD. Defaults to admin to match the script. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: list PostgreSQL on the premade-dashboards features bullet PostgreSQL ships a premade Grafana dashboard (dashboards/grafana/Grafana-DMARC_Reports-PostgreSQL.json), so it belongs on the "for use with premade dashboards" bullet alongside Elasticsearch, OpenSearch, and Splunk rather than on the plain-output-destinations line. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: clear stale org_email mapping conflict in the OpenSearch dashboards The aggregate index pattern in dashboards/opensearch/opensearch_dashboards.ndjson shipped a cached field-list snapshot where org_email was a text/object conflict, plus leftover org_email.#text and org_email.#text.keyword subfields. Those came from a cluster that had indexed a langAttrString email dict ({"#text": ..., "@lang": ...}) before the parser unwrapped it. org_email is mapped as Text() and parse_aggregate_report_xml now unwraps a dict email to a plain string, so current data is consistently text -- a clean cluster's _field_caps reports no conflict. Cleared the frozen conflict and the two artifact subfields, leaving org_email (text) and org_email.keyword, matching the live mapping. Verified: re-importing the corrected ndjson yields an index pattern with org_email as a plain text field and zero conflicts; only the aggregate index-pattern line changed, all other saved objects byte-identical. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * dev: seed the RFC 9990 (dmarc-2.0) aggregate samples samples/aggregate/rfc9990-sample.xml and rfc9990-example.net!...xml were not in the bootstrap's SAMPLE_FILES, so the dev stack only ever indexed RFC 7489 reports and the new DMARCbis fields (np, testing, discovery_method, generator, xml_namespace) never appeared in the OpenSearch/Kibana indices or were available to the dashboards. Added both samples (one declares the urn:ietf:params:xml:ns:dmarc-2.0 namespace, the other is namespaceless RFC 9990-shaped, covering both detection paths). Verified the seeded data now carries np/testing/ discovery_method/generator and xml_namespace=urn:ietf:params:xml:ns:dmarc-2.0; OpenSearch Dashboards surfaces them on an index-pattern field-list refresh. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * dev: auto-resolve (or create) a venv for the seed and ensure psycopg The seed previously required parsedmarc to be pre-installed and only warned-and-skipped PostgreSQL when psycopg was missing. Resolve the seed environment by precedence instead: 1. explicit PARSEDMARC_BIN -> used as-is, nothing installed 2. active $VIRTUAL_ENV 3. existing repo venv/ or .venv/ 4. otherwise create $REPO_ROOT/venv For cases 2-4, run `pip install -e .[postgresql]` only when the CLI or psycopg is missing, so the dev stack can populate Postgres out of the box without a manual install step. The explicit-PARSEDMARC_BIN path is left untouched (and the psycopg seed guard still warns/skips if that env lacks the extra). Verified: a RESEED run resolves the active venv, seeds ES/OS/Splunk/PG including the RFC 9990 fields, with no output-client errors. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
411f5a8886 |
chore: tidy cSpell config; fix two doc typos (#779)
- Ignore data/export trees via cSpell.ignorePaths: parsedmarc/resources/** (maps tooling holds thousands of intentional foreign-language classifier keywords + bundled data), plus samples/** and dashboards/** (report samples and dashboard exports). These are data, not whitelist vocabulary, so excluding them keeps the editor quiet without bloating the word list. - Add the remaining genuine false-positives across code, docs, CI workflows, and editor config to cSpell.words (technical terms, library names, SQL/identifier tokens, brand/operator and multilingual examples from AGENTS.md, plus charliermarsh/junitxml/mktemp/pipefail/seanthegeek). - Fix two genuine typos found while triaging rather than whitelisting them: "maidir" -> "maildir" and "connexion" -> "connection". Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
caac8e68f0 |
docs: note DMARC RFC support in the features list (#778)
* docs: note DMARC RFC support in the features list The features list only mentioned "draft and 1.0" aggregate reports. Spell out the standards parsedmarc parses: RFC 7489 (legacy DMARC) and the final DMARC standard RFC 9989 with RFC 9990 aggregate reports, RFC 6591 and RFC 9991 failure reports, and RFC 8460 SMTP TLS reports. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: align Python compatibility table pipes (MD060) The emoji cells were padded for display width, leaving the source pipes misaligned by character count and tripping markdownlint MD060. Re-pad so every row's pipes line up by codepoint. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: list all optional output destinations; fix table emoji alignment Expand the features list to cover every output sink: Elasticsearch, OpenSearch, Splunk, and PostgreSQL (premade dashboards), plus Kafka, Amazon S3, Azure Log Analytics (Microsoft Sentinel), Graylog (GELF), syslog, and HTTP webhooks. Also re-pad the Python compatibility table using display width (the status emoji render two columns wide), which is what markdownlint MD060 measures — the previous codepoint-based padding still tripped the rule. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: separate PostgreSQL from the premade-dashboards clause PostgreSQL is a storage target without bundled premade dashboards, so it shouldn't sit inside the "for use with premade dashboards" phrase next to Elasticsearch/OpenSearch/Splunk. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: move PostgreSQL to the non-dashboard outputs line Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: use compact markdown tables Switch the markdown tables (Python compatibility, env-var section mapping) to compact single-space format. It reads cleanly in a text editor and sidesteps the column-alignment churn that emoji/variable-width content caused with padded tables (markdownlint MD060). The reStructuredText grid table in dmarc.md is left as-is — it relies on multi-line cells markdown can't express. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
a6778707d7 |
Finish forensic→failure rename: archive-folder migration + dashboard/doc cleanup (#776)
The forensic→failure rename (#659) left a few loose ends and one deliberate hold-back. This closes them. Leftover rename misses (broken paths / stale canonical names): - CONTRIBUTING.md, dashboard-dev-bootstrap.sh: samples/forensic/* → samples/failure/* - dashboard-dev-bootstrap.sh, dashboards/README.md: dmarc_forensic_dashboard.xml → dmarc_failure_dashboard.xml (the file was already renamed; the import path and view name were not) - docs/source/usage.md: PARSEDMARC_GENERAL_SAVE_FORENSIC → ..._SAVE_FAILURE example - samples/parsedmarc.ini: save_forensic → save_failure - pyproject.toml, README.md: canonical "failure" naming (ci.ini intentionally keeps save_forensic to smoke-test the deprecated alias.) Archive subfolder rename + on-startup migration: - New failure reports now archive to <archive>/Failure (was <archive>/Forensic). - _migrate_forensic_archive_folder() runs once on startup (best-effort): renames Forensic→Failure when no Failure folder exists yet, merges the two when both exist, no-ops when there's no legacy folder, and logs-and-skips a mailbox it can't reorganize (warn, don't crash). This consolidates pre- and post-rename failure reports into one folder, replacing the previously documented decision to keep the folder named Forensic to avoid a split archive. Uses the folder-management API (folder_exists / rename_folder / merge_folders) added in mailsuite 2.1.0; the pin is bumped to >=2.1.0. Grafana dashboard (the rename PR updated OSD/Splunk/ES-OS but not Grafana): - Forensic panel titles + the datasource label → Failure; the fo-column display label and its linked byName field-override matcher both → "Failure Policy" (changed together so the column-width override keeps matching). - dev-bootstrap Grafana ES datasource: dmarc_forensic* → dmarc_f* (matches both pre-rename dmarc_forensic* and post-rename dmarc_failure*, like the OSD/Kibana dashboards); RESEED wipe loop now also clears dmarc_failure* indices. - Removed dashboards/grafana/Grafana-DMARC_Reports.json-new_panel.json, an orphan export accidentally committed in #736 and referenced by nothing. Tests (tests/test_init.py): - TestMigrateForensicArchiveFolderMaildir: real on-disk Maildir round-trips via mailsuite's MaildirConnection (no mocks) — rename, merge, no-op, and the full get_dmarc_reports_from_mailbox orchestration. Runs in CI (no network/creds). - TestMigrateForensicArchiveFolderErrorHandling: the one path a real Maildir can't reproduce — a backend that raises mid-operation must warn, not crash. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
327fcff2b9 |
Add optional PostgreSQL storage backend (#667)
Adds a PostgreSQL output backend as a lighter-weight alternative to Elasticsearch/OpenSearch, configured via a [postgresql] section (host/port/user/password/database or a libpq connection_string). Tables are created automatically on first run; a Grafana dashboard is included. - psycopg is an optional extra (pip install parsedmarc[postgresql]); the import is guarded so `import parsedmarc` works without it, and PostgreSQLClient raises a clear install hint when constructed without the driver. Binary wheels aren't available for every platform. - Schema captures the RFC 9990 / DMARCbis aggregate fields: np, testing, discovery_method, generator, xml_namespace, and per-result human_result on the DKIM/SPF auth-result tables. - forensic -> failure naming throughout (table dmarc_failure_report, save_failure_report_to_postgresql, dashboard, docs) to match #659. - Failure-report de-duplication mirrors the Elasticsearch backend exactly: arrival date + From + To + Subject (NULL-safe via IS NOT DISTINCT FROM; semantic JSONB equality). Aggregate and SMTP-TLS use ON CONFLICT. - PostgreSQLClient.close() for clean CLI shutdown; comment documents why the two timestamp helpers must stay distinct (report dates are local, record/SMTP-TLS dates are UTC). - CLI: config parse raises ConfigurationError on missing host/connection_string; wired into _init_output_clients + save loops. - Tests in tests/test_postgres.py (helpers, mocked-DB save assertions, create_tables, connect/error wrapping, dedup, real-sample round trip) and tests/test_cli.py (config parse + end-to-end save wiring incl. AlreadySaved/PostgreSQLError handling). postgres.py at 99% line coverage; only _main's output-client-init retry path is left. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
 DVBandGitHub
|
bf37ded688 | Add support for Elastic Cloud Serverless projects (#770) | ||
 VincentandGitHub
|
535d9db1ad |
cli: support _FILE suffix on PARSEDMARC_* env vars for Docker secrets (#772)
Appending _FILE to any PARSEDMARC_{SECTION}_{KEY} env var reads the
value from the referenced file, with one trailing newline stripped.
This matches the Postgres/MariaDB/Redis container-image convention so
Docker Compose and Kubernetes secret mounts work without extra glue,
keeping credentials out of plain environment: blocks (and out of
docker inspect, container logs, and /proc/<pid>/environ).
When both the direct var and its _FILE companion are set, the file
wins. A missing or unreadable file raises ConfigurationError rather
than silently degrading to an empty credential. The four pre-existing
config keys whose own names end in _file ([general] log_file,
[msgraph] token_file, [gmail_api] credentials_file / token_file)
keep their direct-path semantics; pass their values via secret by
doubling the suffix (_FILE_FILE).
|
||
|
ae1e5adb66 |
Add RFC 9989/9990/9991 (final DMARC) report support; rename forensic→failure project-wide (#659)
* Add DMARCbis report support; rename forensic→failure project-wide
Rebased on top of master @
|
||
 Sean Whalen
|
6ff6261df9 | docs: update installation instructions for IPinfo Lite and MaxMind GeoLite2 databases | ||
|
Sean Whalen
|
06fd3f2b09 | docs: update installation instructions and usage notes for parsedmarc | ||
|
|
4e8c28bbc0 |
Align Kibana dashboards with OpenSearch Dashboards source-of-truth (#737)
* Align Kibana dashboards with OpenSearch Dashboards source-of-truth
OSD is a fork of Kibana 7.10 and Kibana 8.x's saved-object migration
handlers accept OSD's saved-object format directly. Replace the legacy
Kibana export with a byte-identical copy of the OSD ndjson, so the two
backends ship the same panels, metric aggregations, panel titles, and
field assignments instead of drifting independently.
Verified against Kibana 8.19.7: import returns successCount=26 with no
errors and Kibana auto-migrates each viz / dashboard to its current
saved-object schema (typeMigrationVersion 8.5.0 for visualizations,
10.3.0 for dashboards) on import.
Net effects for Kibana users on import:
- Picks up the metric-aggregation fix from 9.10.3 — pies, tables, and
the choropleth now sum(message_count) instead of counting OS docs,
giving real message volume rather than distinct source-row counts.
- Adds "Message sources by Autonomous System" and "Message sources by
name and type" panels (previously only on OSD).
- Forensic dashboard simplified to OSD's two-panel layout (markdown
intro + samples table) — drops the Kibana-only IP-address and
country-ISO tables and the choropleth.
- Adds the "SMTP TLS reporting" dashboard (was absent from the bundled
Kibana export).
- Drops the extraneous "Evolution DMARC par source_reverse_DNS" Lens
visualization that snuck in via a community contribution.
Updates docs/source/kibana.md to reflect the new dashboard names
("DMARC aggregate reports" / "DMARC failure reports") and adds a brief
section on the SMTP TLS reporting dashboard.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Drop the duplicate Kibana ndjson; point Kibana users at the OSD file
Kibana 8.x's saved-object migration handlers accept the OpenSearch
Dashboards saved-object format directly (verified by import returning
successCount=26 with no errors), so a separate kibana/export.ndjson
was just two copies of the same bytes that would inevitably drift. Drop
it and update the bootstrap script and docs to point at the existing
dashboards/opensearch/opensearch_dashboards.ndjson.
Add a path-filtered CI workflow (.github/workflows/dashboards.yml) that
fires only when the OSD ndjson changes. It stands up an Elasticsearch +
Kibana 8.19.7 service pair, POSTs the file at the saved-objects import
endpoint, and asserts success=true with no errors. That keeps the
single-file source compatible with Kibana on every change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
|
826e78c390 |
Fix DMARC dashboard metrics (OSD + Splunk) and add dashboard-dev bootstrap (#736)
* OSD: fix aggregate dashboard metrics to sum(message_count) 13 panels on the DMARC aggregate dashboard were aggregating with `count` (number of OSD docs) when they should have been summing `message_count`. Each parsedmarc OSD doc represents one (source_ip, auth_results) tuple from the XML and carries an integer message_count, so doc-counting reports "distinct sources" rather than "messages". Panels with titles like "Message volume by header from", "DMARC passage over time", etc. were producing misleading numbers. Affected panels: SPF/DKIM/Passed-DMARC pies; Reporting orgs; Sources by reverse DNS / header from / name+type / ASN / country / IP; Map; SPF and DKIM details. (DMARC failure email samples kept count — one OSD doc per RUF sample, so it's correct. SMTP TLS panels untouched — they sum the right session-count fields.) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Splunk: align dashboards with OSD and fix query bugs Aggregate dashboard: - Add "Message sources by Autonomous System" panel (source_asn / as_name / as_domain), formatted "AS<n>" at render with eval, matching the OSD addition. - DKIM details: add the missing dkim_aligned column. - SPF details: reorder columns to OSD order (spf_aligned at end). - Map / country titles renamed to match OSD ("Map of message sources by country", "Message sources by country"). - Map widget: stats count by Country -> stats sum(message_count) by Country, so the choropleth shades by message volume not record count. - fillnull "none"/"unknown" applied to source_reverse_dns, source_base_domain, source_country to mirror OSD's missing-bucket labels. - charting.fieldColors {true: green, false: red} on SPF/DKIM/Passed-DMARC pies and the DMARC-passage timechart. Forensic dashboard: - Restructure to match OSD's two-panel layout (markdown + samples table). - Drop the country map / IP table / country-ISO table panels (not in OSD). - Samples table columns aligned to OSD: arrival_date_utc, source.ip_address, from, subject, reply_to, authentication_results. - Tolerate null headers in the base_search filter (was: parsed_sample.headers.From=* required field to exist; LinkedIn RUF sample with null From was filtered out). SMTP TLS dashboard: - Reorder metrics to OSD order (successful before failed). - Domains panel: add policy_type bucket. - Failure details: replace search-time `failed_session_count>0` (which doesn't evaluate against multivalued JSON paths in Splunk) with `result_type=*` for presence + post-stats `where failed_sessions>0`. Drop _time/successful_sessions columns; reorder to match OSD. - Wire the existing policy_type input into all three searches. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Add dashboard-dev bootstrap script and VSCode task dashboard-dev-bootstrap.sh brings up docker-compose.dashboard-dev.yml, seeds parsedmarc sample data into ES + OS + Splunk via parsedmarc-dev.ini, and re-imports every dashboard into Kibana, OpenSearch Dashboards, Grafana, and Splunk. Idempotent: existence checks skip provisioning that's already done; only the dashboard imports re-run unconditionally on every invocation (that's the point of running it after a dashboard edit). Notable provisioning quirks the script handles: - Splunk's auto-created HEC token (from the SPLUNK_HEC_TOKEN env) ships with indexes=[] and index=default; rewrites it to allow the email index. - ES 8.x rejects wildcard DELETEs by default; RESEED=1 enumerates daily parsedmarc indexes via _cat/indices and deletes one at a time. - Splunk has no clean-in-place REST endpoint for live indexes; RESEED=1 deletes and recreates the email index (then re-applies the HEC token). - OSD security plugin tenants: imports target global_tenant explicitly via the securitytenant header so they're visible to the shared workspace rather than landing in the API user's private tenant. Override with OSD_TENANT=<name>. - Splunk ships an in-product announcement view (scheduled_export_dashboard) with sharing=global; the script narrows it to sharing=app so it stops showing up in every app's dashboards list. Adds a "Dev Dashboard: Bootstrap" task to .vscode/tasks.json that runs the script. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * CHANGELOG: 9.10.3 entry for the dashboard metric fix and alignment work Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Bump version to 9.10.3 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * CHANGELOG: warn against the "Create new objects with unique IDs" import mode OSD's import dialog has two modes: the default "Check for existing objects" (which honors saved-object IDs and overwrites in place when "Automatically overwrite conflicts" is on) and "Create new objects with unique IDs" (which imports under fresh UUIDs and leaves the buggy originals untouched). Picking the second one means the dashboards keep rendering the wrong numbers because the originals are never replaced. Spell that out so users don't fall into the trap. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * OSD: label the metric column "messages" instead of "Sum of message_count" OSD's table column header defaults to "Sum of message_count" when the metric agg has no customLabel. "messages" reads better and matches what the panels are actually counting. Applies to all 15 aggregate-DMARC visualizations that use sum(message_count). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * CHANGELOG: tighten the 9.10.3 entry — clearer and more actionable Trim the verbose technical exposition; lead each fix with the user-visible symptom. Move the action-required call out to its own header in upgrade notes so the re-import instructions don't get lost in a wall of text. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Move per-tool dashboard exports under a single dashboards/ directory Consolidates the four sibling top-level folders (kibana/, opensearch/, grafana/, splunk/) into dashboards/{kibana,opensearch,grafana,splunk}/. Updates the only path references in tracked files: bootstrap script (5 lines), CHANGELOG.md (1 line), and the kibana/export.ndjson raw URL in docs/source/elasticsearch.md. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * OSD: restore the "DKIM alignment" panel title on the aggregate dashboard The DKIM alignment panel had no title override in panelsJSON, so OSD fell back to the visualization's own name ("Aggregate DMARC DKIM alignment"). Every other pie/table on the same dashboard sets a clean title (SPF alignment, Passed DMARC, etc.) — this was a stray regression. Set the panel title to "DKIM alignment" to match. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Splunk: color the message-disposition timechart by severity Reject is red, quarantine is yellow, none is green — same semantic mapping as the SPF/DKIM/Passed-DMARC pies and the DMARC-passage timechart, applied via charting.fieldColors. Matches OSD's existing color overrides on the equivalent viz. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * CHANGELOG: clarify that "Create new objects with unique IDs" is the default The OSD import dialog defaults to that mode — users have to actively switch away from it, not just avoid picking it. Reword the upgrade note to lead with the switch and explain why the default would silently preserve the bug. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
f0f377311e |
Rename asn_name/asn_domain to as_name/as_domain (#719)
Match the IPinfo Lite MMDB's native field names across the output schemas — JSON source records now emit asn, as_name, as_domain, and CSV / Elasticsearch / OpenSearch / Splunk integrations now emit source_asn, source_as_name, source_as_domain. The integer asn / source_asn field is unchanged. Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
c5f432c460 |
Add optional IPinfo Lite REST API with MMDB fallback (#717)
* Add optional IPinfo Lite REST API with MMDB fallback
Configure [general] ipinfo_api_token (or PARSEDMARC_GENERAL_IPINFO_API_TOKEN)
and every IP lookup hits https://api.ipinfo.io/lite/<ip> first for fresh
country + ASN data. On HTTP 429 (rate-limit) or 402 (quota), the API is
disabled for the rest of the run and lookups fall through to the bundled /
cached MMDB; transient network errors fall through per-request without
disabling the API. An invalid token (401/403) raises InvalidIPinfoAPIKey,
which the CLI catches and exits fatally — including at startup via a probe
lookup so operators notice misconfiguration immediately. Added
ipinfo_api_url as a base-URL override for mirrors or proxies.
The API token is never logged. A new _normalize_ip_record() helper is
shared between the API path and the MMDB path so both paths produce the
same normalized shape (country code, asn int, asn_name, asn_domain).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* IPinfo API: cool down and retry instead of permanent disable
Previously a single 429 or 402 disabled the API for the whole run. Now
each event sets a cooldown (using Retry-After when present, defaulting to
5 minutes for rate limits and 1 hour for quota exhaustion). Once the
cooldown expires the next lookup retries; a successful retry logs
"IPinfo API recovered" once at info level so operators can see service
came back. Repeat rate-limit responses after the first event stay at
debug to avoid log spam.
Test now targets parsedmarc.log (the actual emitting logger) instead of
the parsedmarc parent — cli._main() sets the child's level to ERROR,
and assertLogs on the parent can't see warnings filtered before
propagation. Test also exercises the cooldown-then-recovery path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* IPinfo API: log plan and quota from /me at startup
Configure-time probe now hits https://ipinfo.io/me first. That endpoint
is documented as quota-free and doubles as a free-of-quota token check,
so we use it to both validate the token and surface plan / month-to-date
usage / remaining-quota numbers at info level:
IPinfo API configured — plan: Lite, usage: 12345/50000 this month, 37655 remaining
Field names in /me have drifted across IPinfo plan generations, so the
summary formatter probes a few aliases before giving up. If /me is
unreachable (custom mirror behind ipinfo_api_url, network error) we
fall back to the original 1.1.1.1 lookup probe, which still validates
the token and logs a generic "configured" message.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Drop speculative ipinfo_api_url override
It was added mirroring ip_db_url, but the two serve different needs.
ip_db_url has a real use (internal hosting of the MMDB); an
authenticated IPinfo API isn't something anyone mirrors, and /me was
always hardcoded anyway, making the override half-baked. YAGNI.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* AGENTS.md: warn against speculative config options
New section under Configuration spelling out that every option is
permanent surface area and must come from a real user need rather than
pattern-matching a nearby option. Cites the removed ipinfo_api_url as
the canonical cautionary tale so the next session doesn't reintroduce
it, and calls out "override the base URL" / "configurable retries" as
common YAGNI traps.
Also requires that new options land fully wired in one PR (INI schema,
_parse_config, Namespace defaults, docs, SIGHUP-reload path) rather
than half-implemented.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Rename [general] ip_db_url to ipinfo_url
The bundled MMDB is specifically IPinfo Lite, so the option name
should say so. ip_db_url stays accepted as a deprecated alias and
logs a warning when used; env-var equivalents accept either spelling
via the existing PARSEDMARC_{SECTION}_{KEY} machinery.
Updated the AGENTS.md cautionary tale to refer to ipinfo_url (with
the note about the alias) so the anti-pattern example still reads
correctly post-rename.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Fix testPSLDownload to reflect .akamaiedge.net override
PSL carries c.akamaiedge.net as a public suffix, but
psl_overrides.txt intentionally folds .akamaiedge.net so every
Akamai CDN-customer PTR (the aXXXX-XX.cXXXXX.akamaiedge.net pattern)
clusters under one akamaiedge.net display key. The override was added
in
|
||
|
|
2cda5bf59b |
Surface ASN info and use it for source attribution when a PTR is absent (#715)
* Surface ASN info and fall back to it when a PTR is absent Adds three new fields to every IP source record — ``asn`` (integer, e.g. 15169), ``asn_name`` (``"Google LLC"``), ``asn_domain`` (``"google.com"``) — sourced from the bundled IPinfo Lite MMDB. These flow through to CSV, JSON, Elasticsearch, OpenSearch, and Splunk outputs as ``source_asn``, ``source_asn_name``, ``source_asn_domain``. More importantly: when an IP has no reverse DNS (common for many large senders), source attribution now falls back to the ASN domain as a lookup key into the same ``reverse_dns_map``. Thanks to #712 and #714, ~85% of routed IPv4 space now has an ``as_domain`` that hits the map, so rows that were previously unattributable now get a ``source_name``/``source_type`` derived from the ASN. When the ASN domain misses the map, the raw AS name is used as ``source_name`` with ``source_type`` left null — still better than nothing. Crucially, ``source_reverse_dns`` and ``source_base_domain`` remain null on ASN-derived rows, so downstream consumers can still tell a PTR-resolved attribution apart from an ASN-derived one. ASN is stored as an integer at the schema level (Elasticsearch / OpenSearch mappings use ``Integer``) so consumers can do range queries and numeric sorts; dashboards can prepend ``AS`` at display time. The MMDB reader normalizes both IPinfo's ``"AS15169"`` string and MaxMind's ``autonomous_system_number`` int to the same int form. Also fixes a pre-existing caching bug in ``get_ip_address_info``: entries without reverse DNS were never written to the IP-info cache, so every no-PTR IP re-did the MMDB read and DNS attempt on every call. The cache write is now unconditional. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Bump to 9.9.0 and document the ASN fallback work Updates the changelog with a 9.9.0 entry covering the ASN-domain aliases (#712, #714), map-maintenance tooling fixes (#713), and the ASN-fallback source attribution added in this branch. Extends AGENTS.md to explain that ``base_reverse_dns_map.csv`` is now a mixed-namespace map (rDNS bases alongside ASN domains) and adds a short recipe for finding high-value ASN-domain misses against the bundled MMDB, so future contributors know where the map's second lookup path comes from. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Document project conventions previously held only in agent memory Promotes four conventions out of per-agent memory and into AGENTS.md so every contributor — human or agent — works from the same baseline: - Run ruff check + format before committing (Code Style). - Store natively numeric values as numbers, not pre-formatted strings (e.g. ASN as int 15169, not "AS15169"; ES/OS mappings as Integer) (Code Style). - Before rewriting a tracked list/data file from freshly-generated content, verify the existing content via git — these files accumulate manually-curated entries across sessions (Editing tracked data files). - A release isn't done until hatch-built sdist + wheel are attached to the GitHub release page; full 8-step sequence documented (Releases). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
2ac8cb406e |
Replace DB-IP Country Lite with IPinfo Lite (9.8.0) (#711)
Switch the bundled IP-to-country database from DB-IP Country Lite to IPinfo Lite for greater lookup accuracy. The download URL, cached filename, and packaged module path all move from dbip/dbip-country-lite.mmdb to ipinfo/ipinfo_lite.mmdb. IPinfo Lite uses a different MMDB schema (flat country_code) that is incompatible with geoip2's Reader.country() helper, so get_ip_address_country() now uses maxminddb directly and handles both the IPinfo schema and the MaxMind/DBIP nested country.iso_code schema so users who drop in their own MMDB from any of these providers continue to work. Drop the geoip2 dependency (it was only used for the incompatible helper) and add maxminddb as a direct dependency — it was already installed transitively through geoip2. Callers that imported parsedmarc.resources.dbip directly need to switch to parsedmarc.resources.ipinfo. Old parsedmarc versions downloading from the dbip/ GitHub raw URL will 404 and fall back to their bundled copy — this is the documented behavior of load_ip_db(). Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com> Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
6effd80604 |
9.7.0 (#709)
- Auto-download psl_overrides.txt at startup (and whenever the reverse DNS map is reloaded) via load_psl_overrides(); add local_psl_overrides_path and psl_overrides_url config options - Add collect_domain_info.py and detect_psl_overrides.py for bulk WHOIS/HTTP enrichment and automatic cluster-based PSL override detection - Block full-IPv4 reverse-DNS entries from ever entering base_reverse_dns_map.csv, known_unknown_base_reverse_dns.txt, or unknown_base_reverse_dns.csv, and sweep pre-existing IP entries - Add Religion and Utilities to the allowed service_type values - Document the full map-maintenance workflow in AGENTS.md - Substantial expansion of base_reverse_dns_map.csv (net ~+1,000 entries) - Add 26 tests covering the new loader, IP filter, PSL fold logic, and cluster detection Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com> |
||
|
Sean WhalenandClaude Opus 4.6
|
d1e8d3b3d0 |
Auto-update DB-IP Country Lite database at startup
Download the latest DB-IP Country Lite mmdb from GitHub on startup and SIGHUP, caching it locally, with fallback to a previously cached or bundled copy. Skipped when the offline flag is set. Adds ip_db_url config option (PARSEDMARC_GENERAL_IP_DB_URL) to override the download URL. Bumps version to 9.6.0. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> |
||
|
Sean Whalen
|
69eee9f1dc | Update sponsorship section in README and documentation | ||
|
Sean Whalen
|
d6ec35d66f | Fix typo in sponsorship note heading in documentation | ||
|
Sean Whalen
|
2d931ab4f1 | Add sponsor link | ||
|
Sean Whalen
|
ff0ca6538c |
9.5.0
Add environment variable configuration support and update documentation
- Introduced support for configuration via environment variables using the `PARSEDMARC_{SECTION}_{KEY}` format.
- Added `PARSEDMARC_CONFIG_FILE` variable to specify the config file path.
- Enabled env-only mode for file-less Docker deployments.
- Implemented explicit read permission checks on config files.
- Updated changelog and usage documentation to reflect these changes.
|
||
|
Sean Whalen
|
1e95c5d30b |
9.3.1
Elasticsearch and OpenSearch now verify SSL certificates by default when `ssl = True`, even without a `cert_path` - Added `skip_certificate_verification` option to the `elasticsearch` and `opensearch` configuration sections for consistency with `splunk_hec` - Splunk HEC `skip_certificate_verification` now works correctly with self-signed certificates - SMTP TLS reports no longer fail when saving to multiple output targets (e.g. Elasticsearch and OpenSearch) due to in-place mutation of the report dict - Output client initialization errors now identify which module failed (e.g. "OpenSearch: ConnectionError..." instead of generic "Output client error") - Enhanced error handling for output client initialization |
||
|
e82f3e58a1 |
SIGHUP-based configuration reload for watch mode (#697)
* Enhance mailbox connection watch method to support reload functionality - Updated the `watch` method in `GmailConnection`, `MSGraphConnection`, `IMAPConnection`, `MaildirConnection`, and the abstract `MailboxConnection` class to accept an optional `should_reload` parameter. This allows the method to check if a reload is necessary and exit the loop if so. - Modified related tests to accommodate the new method signature. - Changed logger calls from `critical` to `error` for consistency in logging severity. - Added a new settings file for Claude with specific permissions for testing and code checks. * Update parsedmarc/cli.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update parsedmarc/cli.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * [WIP] SIGHUP-based configuration reload for watch mode (#698) * Initial plan * Fix reload state consistency, resource leaks, stale opts; add tests Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> Agent-Logs-Url: https://github.com/domainaware/parsedmarc/sessions/3c2e0bb9-7e2d-4efa-aef6-d2b98478b921 --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> * [WIP] SIGHUP-based configuration reload for watch mode (#699) * Initial plan * Fix review comments: ConfigurationError wrapping, duplicate parse args, bool parsing, Kafka required topics, should_reload kwarg, SIGHUP test skips Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> Agent-Logs-Url: https://github.com/domainaware/parsedmarc/sessions/0779003c-ccbe-4d76-9748-801dbc238b96 --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> * SIGHUP-based configuration reload: address review feedback (#700) * Initial plan * Address review feedback: kafka_ssl, duplicate silent, exception chain, log file reload, should_reload timing Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> Agent-Logs-Url: https://github.com/domainaware/parsedmarc/sessions/a8a43c55-23fa-4471-abe6-7ac966f381f9 --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> * Update parsedmarc/cli.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Best-effort initialization for optional output clients in watch mode (#701) * Initial plan * Wrap optional output client init in try/except for best-effort initialization Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> Agent-Logs-Url: https://github.com/domainaware/parsedmarc/sessions/59241d4e-1b05-4a92-b2d2-e6d13d10a4fd --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> * Fix SIGHUP reload tight-loop in watch mode (#702) * Initial plan * Fix _reload_requested tight-loop: reset flag before reload to capture concurrent SIGHUPs Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> Agent-Logs-Url: https://github.com/domainaware/parsedmarc/sessions/879d0bb1-9037-41f7-bc89-f59611956d2e --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> * Update parsedmarc/cli.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Fix resource leak when HEC config is invalid in `_init_output_clients()` (#703) * Initial plan * Fix resource leak: validate HEC settings before creating any output clients Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> Agent-Logs-Url: https://github.com/domainaware/parsedmarc/sessions/38c73e09-789d-4d41-b75e-bbc61418859d --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> * Ensure SIGHUP never triggers a new email batch across all watch() implementations (#704) * Initial plan * Ensure SIGHUP never starts a new email batch in any watch() implementation Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> Agent-Logs-Url: https://github.com/domainaware/parsedmarc/sessions/45d5be30-8f6b-4200-9bdd-15c655033f17 --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> * SIGHUP-based config reload for watch mode: address review feedback (#705) * Initial plan * Address review feedback: Kafka SSL context, SIGHUP handler safety, test formatting Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> Agent-Logs-Url: https://github.com/domainaware/parsedmarc/sessions/8f2fd48f-32a4-4258-9a89-06f7c7ac29bf --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> * Reverted changes by copilot that turned errors into warnings * Enhance usage documentation for config reload: clarify behavior on successful reload and error handling * Update CHANGELOG.md to reflect config reload enhancements * Add pytest command to settings for silent output during testing * Enhance resource management: add close methods for S3Client and HECClient, and improve IMAP connection handling during IDLE. Update CHANGELOG.md for config reload improvements and bug fixes. * Update changelog to not include fixes within the same unreleased version * Refactor changelog entries for clarity and consistency in configuration reload section * Fix changelog entry for msgraph configuration check * Update CHANGELOG..md * make single list items on one line in the changelog instead of doing hard wraps * Remove incorrect IMAP changes * Rename 'should_reload' parameter to 'config_reloading' in mailbox connection methods for clarity * Restore startup configuration checks * Improve error logging for Elasticsearch and OpenSearch exceptions * Bump version to 9.3.0 in constants.py * Refactor GelfClient methods to use specific report types instead of generic dicts * Refactor tests to use assertions consistently and improve type hints --------- Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com> |
||
|
|
ea0e3b11c1 |
Add MS Graph certificate authentication support (#692)
* Add MS Graph certificate authentication support * Preserve MS Graph constructor compatibility --------- Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com> |
||
|
KiliandGitHub
|
326e630f50 | Add performance tuning guidance for large mailbox runs (#677) | ||
|
KiliandGitHub
|
f2febf21d3 |
Add fail_on_output_error CLI option for sink failures (#672)
* Add fail-on-output-error option and CLI regression test * Broaden fail_on_output_error coverage for disabled and multi-sink paths |
||
|
|
c4d7455839 |
Add OpenSearch AWS SigV4 authentication support (#673)
* Add OpenSearch AWS SigV4 authentication support * Increase SigV4 coverage for auth validation and CLI config wiring * Update parsedmarc/opensearch.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Update docs/source/usage.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |
||
|
KiliandGitHub
|
a3c5bb906b | Add Gmail service account auth mode with delegated user support (#676) | ||
|
KiliandGitHub
|
e98fdfa96b | Fix Python 3.14 support metadata and require imapclient 3.1.0 (#662) | ||
 |
2e3ee25ec9 |
Drop Python 3.9 support (#661)
* Initial plan * Drop Python 3.9 support: update CI matrix, pyproject.toml, docs, and README Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> * Update Python 3.9 version table entry to note Debian 11/RHEL 9 usage Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> |
||
|
|
17a612df0c |
Add TCP and TLS transport support to syslog module (#656)
- Updated parsedmarc/syslog.py to support UDP, TCP, and TLS protocols - Added protocol parameter with UDP as default for backward compatibility - Implemented TLS support with CA verification and client certificate auth - Added retry logic for TCP/TLS connections with configurable attempts and delays - Updated parsedmarc/cli.py with new config file parsing - Updated documentation with examples for TCP and TLS configurations Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> * Remove CLI arguments for syslog options, keep config-file only Per user request, removed command-line argument options for syslog parameters. All new syslog options (protocol, TLS settings, timeout, retry) are now only available via configuration file, consistent with other similar options. Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> * Fix code review issues: remove trailing whitespace and add cert validation - Removed trailing whitespace from syslog.py and usage.md - Added warning when only one of certfile_path/keyfile_path is provided - Improved error handling for incomplete TLS client certificate configuration Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> * Set minimum TLS version to 1.2 for enhanced security Explicitly configured ssl_context.minimum_version = TLSVersion.TLSv1_2 to ensure only secure TLS versions are used for syslog connections. Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com> |
||
 Anael MobiliaandGitHub
|
50fcb51577 |
Update supported Python versions in docs + readme (#652)
* Update README.md * Update index.md * Update python-tests.yml |
||
|
Sean Whalen
|
35331d4b84 | Add parsedmarc.types module to API reference documentation | ||
|
Sean Whalen
|
445c9565a4 | Update bug link in docs | ||
|
Sean Whalen
|
23ae563cd8 | Update Python version support details in documentation | ||
|
Sean Whalen
|
a18ae439de | Fix typo in RHEL version support description in documentation | ||
 Oscar MattssonandGitHub
|
f1933b906c | Fix 404 link to maxmind docs (#635) | ||
|
Sean WhalenandGitHub
|
1fc9f638e2 |
9.0.0 (#629)
* Normalize report volumes when a report timespan exceed 24 hours |
||
|
Sean WhalenandGitHub
|
0922d6e83a | Add supported Python versions to the documentation index |