* docs: note DMARC RFC support in the features list
The features list only mentioned "draft and 1.0" aggregate reports. Spell
out the standards parsedmarc parses: RFC 7489 (legacy DMARC) and the final
DMARC standard RFC 9989 with RFC 9990 aggregate reports, RFC 6591 and
RFC 9991 failure reports, and RFC 8460 SMTP TLS reports.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: align Python compatibility table pipes (MD060)
The emoji cells were padded for display width, leaving the source pipes
misaligned by character count and tripping markdownlint MD060. Re-pad so
every row's pipes line up by codepoint.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: list all optional output destinations; fix table emoji alignment
Expand the features list to cover every output sink: Elasticsearch,
OpenSearch, Splunk, and PostgreSQL (premade dashboards), plus Kafka,
Amazon S3, Azure Log Analytics (Microsoft Sentinel), Graylog (GELF),
syslog, and HTTP webhooks.
Also re-pad the Python compatibility table using display width (the
status emoji render two columns wide), which is what markdownlint MD060
measures — the previous codepoint-based padding still tripped the rule.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: separate PostgreSQL from the premade-dashboards clause
PostgreSQL is a storage target without bundled premade dashboards, so it
shouldn't sit inside the "for use with premade dashboards" phrase next to
Elasticsearch/OpenSearch/Splunk.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: move PostgreSQL to the non-dashboard outputs line
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs: use compact markdown tables
Switch the markdown tables (Python compatibility, env-var section mapping)
to compact single-space format. It reads cleanly in a text editor and
sidesteps the column-alignment churn that emoji/variable-width content
caused with padded tables (markdownlint MD060). The reStructuredText grid
table in dmarc.md is left as-is — it relies on multi-line cells markdown
can't express.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Add DMARCbis report support; rename forensic→failure project-wide
Rebased on top of master @ 2cda5bf (9.9.0), which added the ASN
source attribution work (#712, #713, #714, #715). Individual Copilot
iteration commits squashed into this single commit — the per-commit
history on the feature branch was iterative (add tests, fix lint,
move field, revert, etc.) and not worth preserving; GitHub squash-
merges PRs anyway.
New fields from the DMARCbis XSD, plumbed through types, parsing, CSV
output, and the Elasticsearch / OpenSearch mappings:
- ``np`` — non-existent subdomain policy (``none`` / ``quarantine`` /
``reject``)
- ``testing`` — testing mode flag (``n`` / ``y``), replaces RFC 7489
``pct``
- ``discovery_method`` — policy discovery method (``psl`` /
``treewalk``)
- ``generator`` — report generator software identifier (metadata)
- ``human_result`` — optional descriptive text on DKIM / SPF results
RFC 7489 reports parse with ``None`` for DMARCbis-only fields.
Forensic reports have been renamed to failure reports throughout the
project to reflect the proper naming since RFC 7489.
- Core: ``types.py``, ``__init__.py`` — ``ForensicReport`` →
``FailureReport``, ``parse_forensic_report`` →
``parse_failure_report``, report type ``"failure"``.
- Output modules: ``elastic.py``, ``opensearch.py``, ``splunk.py``,
``kafkaclient.py``, ``syslog.py``, ``gelf.py``, ``webhook.py``,
``loganalytics.py``, ``s3.py``.
- CLI: ``cli.py`` — args, config keys, index names
(``dmarc_failure``).
- Docs + dashboards: all markdown, Grafana JSON, Kibana NDJSON,
Splunk XML.
Backward compatibility preserved: old function / type names remain as
aliases (``parse_forensic_report = parse_failure_report``,
``ForensicReport = FailureReport``, etc.), CLI accepts both the old
(``save_forensic``, ``forensic_topic``) and new (``save_failure``,
``failure_topic``) config keys, and updated dashboards query both
old and new index / sourcetype names so data from before and after
the rename appears together.
Merge conflicts resolved in ``parsedmarc/constants.py`` (took bis's
10.0.0 bump), ``parsedmarc/__init__.py`` (combined bis's "failure"
wording with master's IPinfo MMDB mention), ``parsedmarc/elastic.py``
and ``parsedmarc/opensearch.py`` (kept master's ``source_asn`` /
``source_asn_name`` / ``source_asn_domain`` on the failure doc path
while renaming ``forensic_report`` → ``failure_report``), and
``CHANGELOG.md`` (10.0.0 entry now sits above the 9.9.0 entry).
All 324 tests pass; ``ruff check`` / ``ruff format --check`` clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Apply post-RFC review fixes: RFC 9990 detection, langAttrString, CFWS-aware RUF parsing
Aligns the implementation with the final RFCs (9989/9990/9991) instead of
inferring DMARCbis support from the version element or the namespace alone.
Aggregate parsing (RFC 9990):
- _text() helper unwraps langAttrString values (extra_contact_info, error,
comment, human_result, generator) — when reporters include the lang
attribute, xmltodict yields {"#text": ..., "@lang": ...} dicts instead
of strings; the parser now stores the text payload in both shapes.
- New xml_namespace field on AggregateReport records the declared XML
namespace (urn:ietf:params:xml:ns:dmarc-2.0 for RFC 9990 reports).
- RFC 9990 detection accepts namespaceless reports that follow the
RFC 9990 shape (presence of np / testing / discovery_method / generator),
so reporters that don't declare the namespace still receive RFC 9990-
aware validation.
- Warnings: missing DKIM <selector> (REQUIRED in RFC 9990); legacy
forwarded / sampled_out policy-override types (removed by RFC 9990);
unknown policy-override types per the RFC 9990 enumeration.
- xml_namespace added to Elasticsearch and OpenSearch document mappings.
Failure parsing (RFC 9991):
- Identity-Alignment and Auth-Failure are split on commas with CFWS
whitespace stripped per the RFC 9991 ABNF; previously "dkim, spf"
yielded ["dkim", " spf"] with a leading space on the second token.
- Warnings logged when either REQUIRED field is missing.
Terminology: every reference to "DMARCbis" in code, tests, sample
filenames, AGENTS.md, and CHANGELOG.md is replaced with the appropriate
RFC number (9989 for the policy spec, 9990 for aggregate reports, 9991
for failure reports). Sample contents are unchanged.
Docs: corrects the prior claim that fo was dropped from RFC 9990 (only
pct was), reframes testing as a new field (not a pct replacement, since
RFC 9989 Appendix A.6 removed pct with no per-message substitute), and
documents the policy_override_reason enum changes (added policy_test_mode;
removed forwarded / sampled_out).
Tests: 8 new tests covering xml_namespace capture, RFC 9990 detection
from field shape, missing-DKIM-selector warning, legacy-override-type
warning, langAttrString unwrapping across all four affected elements,
and CFWS-aware Identity-Alignment / Auth-Failure parsing plus their
missing-field warnings. 276 tests total, all passing; ruff clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Set global TOC collapse to false
* Split documentation
I tried to split the index.md file into logical parts, not changing the contents.
I did add a space and change one HTTP URL to HTTPS.
---------
Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>
* Implemented Azure Log Analytics ingestion via Data Collection Rules
* Update loganalytics.py
* Update cli.py
* Update pyproject.toml
* Fixed config bug
Fixed a bug that causes the program to fail if you do not configure a Data stream.
* Fixed code format
```
org.elasticsearch.ElasticsearchSecurityException: invalid configuration for xpack.security.transport.ssl - [xpack.security.transport.ssl.enabled] is not set, but the following settings have been configured in elasticsearch.yml : [xpack.security.transport.ssl.keystore.secure_password,xpack.security.transport.ssl.truststore.secure_password]
```
Add information on how to fix "Elasticsearch error: RequestError(400, 'validation_exception', 'Validation Failed: 1: this action would add [1] shards, but this cluster currently has [1000]/[1000] maximum normal shards open;"
* Update elasticsearch/kibana instructions
[From elastisearch notes](https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html#heap-size-settings) :
```
By default, Elasticsearch automatically sets the JVM heap size based on a node’s roles and total memory. We recommend the default sizing for most production environments.
```
* Update nginx conf to TLSv1.3 and IPv6
* Replace nginx proxy by native https server
Kibana now provide https web server, remove the nginx proxy part and directly use kibana
* Fix typo
* Add infos how to login to kibana
* Add interface details
Sean Whalen
Copilot
Sean Whalen
Kili
Anael Mobilia
Szasza Palmer
Sean Whalen
Ben Companjen
rubeste
aroldxd
nmourad