Move the "Maintaining the reverse DNS maps" section (two-thirds of the
root AGENTS.md) to parsedmarc/resources/maps/AGENTS.md, next to the
files and tooling it governs. Agents that follow the AGENTS.md spec
read the nearest file in the directory tree; a one-line CLAUDE.md
import alongside it covers Claude Code, which loads subdirectory
context lazily when files under maps/ are first read. A pointer stub
remains in the root file for agents that only read the repo root.
The moved content is verbatim except for one cross-reference that now
points at the root AGENTS.md instead of "earlier in this file".
Exclude AGENTS.md/CLAUDE.md from hatch build targets so the new files
don't ship in the wheel, matching the existing README.md exclusion
(verified with a scratch wheel build).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Follow-ups from the review of PR #825 (whose implementation had already
landed on master via #826's stacked merge):
- Honor the documented [smtp] attachment and [smtp] message options.
Both were parsed into opts but never passed to either summary-email
transport (also broken in released 10.2.2), so a configured custom
attachment filename or message body was silently ignored. Both the
SMTP and Microsoft Graph transports now receive them, and the missing
smtp_attachment Namespace default is added (also covers SIGHUP
reload, which rebuilds opts from the CLI Namespace).
- Don't mislabel non-Graph mailbox errors as Microsoft Graph failures:
the shared mailbox-fetch and watch handlers now log a generic
"Mailbox Error" with traceback when the connection isn't Graph.
- Declare microsoft-kiota-abstractions as a direct dependency (imported
directly in cli.py for Graph error handling; previously transitive).
Migrate all runtime HTTP from requests to httpx (webhook client, Splunk
HEC client, and the PSL-overrides / IP-database / reverse-DNS-map /
IPinfo-API fetches in utils.py):
- follow_redirects=True everywhere to preserve requests' default
redirect-following; httpx does not follow redirects by default.
- The PSL-overrides and reverse-DNS-map fetches gain a 60s timeout
(previously none), matching the IP-database fetch.
- response.ok -> response.is_success; requests.RequestException ->
httpx.HTTPError; raw string bodies use content= (httpx's data= is
form-encoding only); Splunk HEC verification moves to client
construction (httpx has no per-request verify).
- requests drops out of [project] dependencies and moves to the [build]
extra for the out-of-wheel maintainer script collect_domain_info.py,
which deliberately stays on requests/urllib3 for its permissive-TLS
adapter.
- Remove the requests-era module-level
urllib3.disable_warnings(InsecureRequestWarning) in splunk.py; httpx
doesn't route through urllib3, so its only remaining effect was
globally silencing insecure-TLS warnings from other urllib3-based
components as an import side effect. Nothing imports urllib3 directly
anymore, so it also leaves [project] dependencies.
Tests: config-to-transport wiring for attachment/message on both
transports (including defaults), non-Graph errors keep the generic log
line, webhook/Splunk payload assertions moved to content=, and Splunk
verify asserted at httpx.Client construction. 736 passed; ruff and
pyright clean.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Send report summary via Microsoft Graph; make Graph failures observable
Two related fixes shipped together:
Send via Graph: the periodic DMARC summary email can now be sent
through the already-authenticated Microsoft Graph mailbox connection
(MSGraphConnection.send_message(), /users/{mailbox}/sendMail) instead
of only SMTP. Triggered when [msgraph] is configured and [smtp] has a
`to` value but no `host` -- SMTP is always preferred when `host` is
set, with no automatic fallback to Graph on SMTP failure. Reuses the
same connection used for reading; no new send-only config mode.
email_results()'s SMTP behavior is unchanged; a new
email_results_via_msgraph() shares its content-building logic via a
new _build_report_email_content() helper. Graph's sendMail always
sends as the authenticated mailbox, so [smtp] from is ignored on this
path -- documented, along with the required Mail.Send permissions and
a caveat that delegated auth flows (UsernamePassword/DeviceCode) don't
currently request that scope, so app-only auth is the supported path
for sending. Tracks #472.
Observable Graph failures: MSGraphConnection construction, mailbox
fetch, message send, and --watch failures now catch
ClientAuthenticationError/APIError/httpx.HTTPError specifically and
log one clear ERROR line naming the mailbox, tenant, auth method, and
the Graph request-id/client-request-id when available, instead of a
bare "MS Graph Error"/"Mailbox Error" with no context. Full traceback
still preserved at --debug. --watch previously had no Graph-specific
error handling at all -- a Graph error there crashed with a raw
uncaught traceback; it now exits the same way as the other three
sites. No new config options.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Refresh Microsoft Graph docs: national clouds, examples, troubleshooting
The [msgraph] docs were accurate but missed guidance the community has
been asking for:
- graph_url now lists the actual national/sovereign-cloud endpoint
values (GCC High, DoD, China/21Vianet), with an explicit warning
that setting it alone is not sufficient -- the Entra ID auth
endpoint isn't independently configurable in parsedmarc or
mailsuite, so it always hits the global login.microsoftonline.com.
- A minimal working [msgraph] example for every auth method
(UsernamePassword, DeviceCode, ClientSecret, Certificate,
ClientAssertion) -- previously only Certificate had one, entangled
with the SMTP-sending example.
- A reading-permission matrix alongside the existing sending one, so
every auth method x own/shared-mailbox combination is explicit in
one place for both directions.
- An accurate note on the parsedmarc-named token cache: it's a
deliberate backward-compatibility choice from the 9.11.0 mailsuite
extraction (mailsuite's own default cache name differs), not a
migration users need to act on.
- A troubleshooting table for four error scenarios, verified against
source rather than assumed: admin consent and folder-resolution
failures are still live and documented with real fixes; the
event-loop and ISO-timestamp errors are historical, already fixed
below this project's dependency/version floor.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Migrate Elasticsearch output to the elasticsearch-py 8.x client (#806)
The mandatory elasticsearch<7.14.0 + elasticsearch-dsl==7.4.0 pins
transitively forced urllib3<2 (EOL 1.26.x) onto every install. The old
<7.14.0 cap only existed to dodge the client product check that broke
OpenSearch users (#452, #653) — obsolete now that parsedmarc has a
dedicated [opensearch] backend on opensearch-py.
- Depend on elasticsearch>=8.18,<9 and drop elasticsearch-dsl entirely
(the DSL ships inside the client as elasticsearch.dsl since 8.18.0).
The 8.x client's elastic-transport allows urllib3>=1.26.2,<3, so
installs can now resolve urllib3 2.x. The 8.x line supports both
Elasticsearch 8.x and 9.x servers; ES 7.x servers are no longer
supported, and OpenSearch users pointing [elasticsearch] at an
OpenSearch cluster must switch to the [opensearch] section.
- set_hosts() now builds 8.x connection kwargs (scheme-qualified host
URLs, request_timeout, basic_auth) while keeping the function
signature and every INI option unchanged.
- migrate_indexes() is now a documented no-op kept for API
compatibility: its only migration (re-typing published_policy.fo
from long to text) applied exclusively to indices carrying the
legacy ES 6-era "doc" mapping type, which cannot exist on any
server the 8.x client can reach.
- The elasticsearch.dsl 8.x stubs use dataclass_transform and don't
surface pre-8.x-style bare `name = Text()` fields as constructor
parameters; each Document/InnerDoc class now carries a
TYPE_CHECKING-only `__init__(*args, **kwargs)` declaration matching
the real runtime signature, which also made nine pre-existing
pyright ignores unnecessary.
Verified with ruff, pyright (0 errors/0 warnings), the full pytest
suite (718 passed), and a CLI run over the bundled samples; CI's live
elasticsearch:8.19.7 service exercises the new client end-to-end.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Use pass instead of ... in TYPE_CHECKING __init__ stubs
CodeQL flags an ellipsis-only body as "Statement has no effect" (12
alerts on PR #822); pass is equivalent at runtime and to the type
checker and keeps the alerts from resurfacing on every future scan.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
record["interval_begin"]/["interval_end"] are UTC wall-clock strings
(produced in __init__.py by strftime() on a datetime already converted
via human_timestamp_to_datetime(..., to_utc=True)), but the
Elasticsearch and OpenSearch per-record save loops and the Splunk HEC
aggregate-report event builder re-parsed them without assume_utc=True,
so on non-UTC hosts they were misinterpreted as local time and shifted
by the host's UTC offset -- shifting the stored date_begin/date_end,
the daily/monthly index date, and the Splunk event time. Verified by
parsing a real sample report under TZ=Europe/Warsaw vs TZ=UTC and by
reproducing the exact skew via human_timestamp_to_unix_timestamp().
This is the same class of bug fixed for arrival_date_utc in #811/#812
(commit cdda5da); the assume_utc keyword already exists on
human_timestamp_to_datetime()/human_timestamp_to_unix_timestamp() and
is reused here rather than reimplemented.
The issue also proposed changes to the report-level begin_date/end_date
parses in elastic.py/opensearch.py (~line 455) and to postgres.py.
Both were investigated and left unchanged: the report-level strings
are genuinely host-local time (from timestamp_to_human() ->
datetime.fromtimestamp()), so their existing no-assume_utc round-trip
is already correct on a single host -- postgres.py already gets this
right via two distinct helpers (_naive_local_to_timestamptz vs
_ensure_utc_suffix). Adding assume_utc to the report-level parses
would introduce a skew rather than fix one.
Fixes https://github.com/domainaware/parsedmarc/issues/819
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
The system-path fallback list in _get_ip_database_path() checks real
absolute paths like /usr/share/GeoIP/GeoLite2-Country.mmdb. On a host
that actually has one installed there, the fallback succeeds and no
FileNotFoundError is raised, regardless of the test's mocked bundled
path -- the code is behaving correctly, the test just wasn't isolated
from the host filesystem. Patch os.path.exists to force every system
path to look absent so the test is host-independent.
Co-authored-by: MISAPOR LAB <misapor@lab.misapor.pl>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* Raise test coverage: utils, elastic, and opensearch to 100%
Coverage of the shipped library rises from 88% to 90%, with
parsedmarc/utils.py 86% -> 100% and elastic.py / opensearch.py
99% -> 100%. All new tests assert on observable behaviour and mock
only at SDK boundaries (dnspython Resolver.resolve, requests.get,
subprocess.check_call, elasticsearch_dsl/opensearchpy Document.save).
New tests cover: query_dns transient-error retries, the load_ip_db
download/cache/bundled fallback chain, the IPinfo API token probe and
per-request MMDB fallbacks, _normalize_ip_record schema handling,
reverse-DNS-map invalid-CSV fallback, caller-provided reverse DNS
maps, Outlook MSG conversion (missing msgconvert and success paths),
parse_email Cc/Bcc/attachment-hash branches, aggregate-XML edge cases
(bytes input, repeated policy_published, unknown RFC 9990 override
types, missing org_name, attribute-only <email>), extract_report on
non-seekable streams, and the _AggregateReportDoc.save() override
that derives passed_dmarc.
Bugs found by the new tests, fixed in the same PR per the testing
standards:
- parse_email() crashed with KeyError: 'Headers' on messages whose
From header is present but unparseable (e.g. a bare "From:" line):
the fallback read parsed_email["Headers"], but the parsed headers
are stored under lowercase "headers" (assigned a few lines up in
the same function), so the key never exists. At the CLI surface
this made any failure report whose embedded sample had an empty
From: header fail to parse ("Missing value: 'Headers'").
- configure_ipinfo_api(probe=True) logged "IPinfo API configured"
when the probe could not reach the API, contradicting its own
docstring ("other errors are logged and the token is still
accepted"): _ipinfo_api_lookup() returns None on network errors
instead of raising, so the probe's exception handler was
unreachable. The probe now checks the lookup result and warns on
failure; 401/403 still raises InvalidIPinfoAPIKey.
Dead code deleted rather than padded with tests:
- _SMTPTLSReportDoc.add_policy() in elastic.py and opensearch.py
(the save paths construct _SMTPTLSPolicyDoc directly).
- The no-op "for failure_index in failure_indexes: pass" loop in
both migrate_indexes() implementations (parameter still accepted).
- The importlib.resources ImportError fallback in utils.py, which
re-imported the same module and is unreachable on Python >= 3.10.
- The "Invalid report content" guard in extract_report(): every
input branch assigns file_object or raises first (confirmed by
pyright narrowing with the guard removed).
Also widens parse_aggregate_report_xml's annotation to str | bytes
to match its existing runtime behaviour (bytes are decoded with
errors ignored).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Use assertGreater for the reverse-DNS-map fallback size check
Addresses the github-code-quality bot finding on PR #816: assertTrue
with a comparison inside can't show the operands on failure, while
assertGreater reports both values and the failed relation. No change
to test behavior.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add verification claude skill for parsedmarc CLI usage and sample inputs
* Document model roles for feature work in CLAUDE.md
Codifies the plan-with-Fable / implement-with-Sonnet / review-with-Fable
split (Opus as fallback) for feature work and PR reviews.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Raise test coverage: utils, elastic, and opensearch to 100%
Coverage of the shipped library rises from 88% to 90%, with
parsedmarc/utils.py 86% -> 100% and elastic.py / opensearch.py
99% -> 100%. All new tests assert on observable behaviour and mock
only at SDK boundaries (dnspython Resolver.resolve, requests.get,
subprocess.check_call, elasticsearch_dsl/opensearchpy Document.save).
New tests cover: query_dns transient-error retries, the load_ip_db
download/cache/bundled fallback chain, the IPinfo API token probe and
per-request MMDB fallbacks, _normalize_ip_record schema handling,
reverse-DNS-map invalid-CSV fallback, caller-provided reverse DNS
maps, Outlook MSG conversion (missing msgconvert and success paths),
parse_email Cc/Bcc/attachment-hash branches, aggregate-XML edge cases
(bytes input, repeated policy_published, unknown RFC 9990 override
types, missing org_name, attribute-only <email>), extract_report on
non-seekable streams, and the _AggregateReportDoc.save() override
that derives passed_dmarc.
Bugs found by the new tests, fixed in the same PR per the testing
standards:
- parse_email() crashed with KeyError: 'Headers' on messages whose
From header is present but unparseable (e.g. a bare "From:" line):
the fallback read parsed_email["Headers"], but the parsed headers
are stored under lowercase "headers" (assigned a few lines up in
the same function), so the key never exists. At the CLI surface
this made any failure report whose embedded sample had an empty
From: header fail to parse ("Missing value: 'Headers'").
- configure_ipinfo_api(probe=True) logged "IPinfo API configured"
when the probe could not reach the API, contradicting its own
docstring ("other errors are logged and the token is still
accepted"): _ipinfo_api_lookup() returns None on network errors
instead of raising, so the probe's exception handler was
unreachable. The probe now checks the lookup result and warns on
failure; 401/403 still raises InvalidIPinfoAPIKey.
Dead code deleted rather than padded with tests:
- _SMTPTLSReportDoc.add_policy() in elastic.py and opensearch.py
(the save paths construct _SMTPTLSPolicyDoc directly).
- The no-op "for failure_index in failure_indexes: pass" loop in
both migrate_indexes() implementations (parameter still accepted).
- The importlib.resources ImportError fallback in utils.py, which
re-imported the same module and is unreachable on Python >= 3.10.
- The "Invalid report content" guard in extract_report(): every
input branch assigns file_object or raises first (confirmed by
pyright narrowing with the guard removed).
Also widens parse_aggregate_report_xml's annotation to str | bytes
to match its existing runtime behaviour (bytes are decoded with
errors ignored).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Use assertGreater for the reverse-DNS-map fallback size check
Addresses the github-code-quality bot finding on PR #816: assertTrue
with a comparison inside can't show the operands on failure, while
assertGreater reports both values and the failed relation. No change
to test behavior.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Bring parsedmarc/gsecops.py from 96% to 100% coverage:
- principal hostname from an enriched source reverse DNS name
- the literal "none" placeholder in policy_override_reasons /
policy_override_comments is dropped from additional
(samples/aggregate/empty_reason.xml)
- SMTP TLS failure details map sending-mta-ip to principal.ip and
receiving-ip to target.ip (the RFC 8460 Appendix B sample)
- target hostname falls back to receiving-mx-hostname when a report
carries an empty policy-domain
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ensure that the JSON or CSV output from a parsed DMARC aggregate report
always has an xml_schema value (falling back to "draft" when <version>
is missing, empty, or attribute-wrapped), so that the reports can be
detected by Google SecOps and other output parsers.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
In DMARC vocabulary "row" reads as the <row> element inside an
aggregate report record, which carries no schema information — the
changelog entry was about the flattened JSON/CSV output lines, where
xml_schema is a synthesized per-record field. Reword the entry and the
two related comments to say which kind of row is meant.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The README presented xml_schema as the aggregate detector. What
actually identifies an aggregate report is its <feedback> XML root
element, which parse_aggregate_report_xml anchors on; the raw XML never
reaches SecOps, so the CBN parser tests xml_schema — the field
parsedmarc synthesizes on every row that came from a <feedback>
document — as its serialized marker. Say so in the detection table and
the explanation paragraph.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The <feedback> XML root element (anchored at the top of
parse_aggregate_report_xml) is what identifies an aggregate report;
<version> is optional metadata per RFC 7489 Appendix C. The "draft"
fallback only covered a fully absent <version>: an empty <version/>
(which xmltodict parses as None) or a whitespace-only value produced
xml_schema=None, which the flat-row serializer coerced to "" — and any
consumer detecting aggregate rows by a non-empty xml_schema, such as
the Google SecOps CBN parser's report-type cascade, silently dropped
the row. The fallback now applies unless <version> carries non-empty
text (attribute-wrapped values are unwrapped via _text like other
elements).
The regression test fails on the previous code with
xml_schema=None != "draft".
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The comment described the behavior but not the reason: CBN's documented
conditional syntax only compares a token against a quoted string
literal -- there is no boolean literal and no truthiness test on event
fields -- so a type-preserved JSON boolean can never match any
conditional without first converting it to a string. The conversion is
in place because CBN mutate has no copy function. Cite the syntax
reference.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The step-1b comment still said booleans are "stored as string_value
(matching Google's content-hub parsers, which never use bool_value)" —
stale since the switch to typed bool_value storage. 1b now says what it
actually does: convert booleans to strings for the `if` guards only,
because CBN conditionals compare against the preserved JSON type;
storage is typed bool_value per the alignment blocks. Also tightened
the README's Corelight citation, which overstated the string idiom as
something the whole parser relies on rather than just the conditionals.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The alignment flags and normalized_timespan were stored as
string_value "true"/"false" in additional.fields, which diverged from
the [gsecops] API output (a protobuf Struct, where they are real
booleans) — a UDM search written for one delivery path would not match
the other for those four fields.
Google's parser extension examples document the boolean idiom this
uses: build a string, convert to boolean, rename into the boolean
field. It is the boolean analogue of the number_value chain already
used for count/source_asn (proven in the content-hub Azure Cosmos DB
parser), and the UDM search docs confirm additional fields are
matchable via value.bool_value. The string conversion in step 1b
remains, because CBN conditionals compare against the preserved JSON
type and the == "true"/"false" guards need strings.
With this, both delivery paths emit identical value types for every
additional.fields key. The prior justification ("content-hub never
uses bool_value") described the absence of precedent, not a
prohibition; the documented boolean example is the stronger authority.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sends parsed reports to a Google SecOps instance as pre-normalized UDM
events through the GA v1 Chronicle API:
POST https://chronicle.{region}.rep.googleapis.com/v1/{parent}/events:import
Pre-normalized events bypass SecOps's server-side (CBN) parsing layer,
so no parser needs to be installed in the tenant; the CBN parser in
google_secops_parser/ remains the collector-based alternative for
deployments that want raw-log retention, and both paths emit the same
UDM shape and additional-field keys so searches port between them.
Google's ingestion docs recommend the UDM-events path when possible.
Implementation notes, all grounded in the v1 reference docs:
- Auth is standard Google Cloud IAM (Chronicle API Editor role /
chronicle.events.import permission): a service account key file when
[gsecops] credentials_file is set, Application Default Credentials
otherwise. google-auth was already a transitive requirement via
mailsuite[gmail]; it is now declared directly.
- Batches follow the documented best practices (1,000 events per
request, 60 s timeout). events.import is all-or-nothing — one invalid
event rejects the whole request — so a rejected batch is bisected to
isolate invalid events; valid events are still delivered and the drop
count is raised as an output error at the end.
- to and subject are repeated fields in the UDM Email message;
SecurityResult action and category are repeated enums.
- additional is a protobuf Struct, so counts and ASNs stay numbers and
alignment flags stay booleans — range-queryable without the CBN
string-hop.
New [gsecops] config section (project_id, instance_id, region,
credentials_file) is wired through the INI schema, _parse_config,
Namespace defaults, PARSEDMARC_GSECOPS_* env vars, the usage docs, and
per-batch client construction (SIGHUP-reload safe by construction).
Tests build events from real sample reports and assert on the payloads
sent through a mocked AuthorizedSession, including batching and the
400-bisect path.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review findings verified against the parsedmarc serializers, the UDM
field list, Google's parser troubleshooting doc, and the content-hub
parsers:
- Detect failure reports by feedback_type OR arrival_date_utc: the
text-format failure path synthesizes a feedback report with no
Feedback-Type field, so those rows had no feedback_type key and were
dropped as TAG_UNSUPPORTED.
- Detect SMTP TLS rows by policy_type OR result_type, and fall back to
receiving_mx_hostname for target.hostname, so failure-detail rows from
parsedmarc versions without the paired serializer fix still map.
- Merge (not replace) network.email.to and network.email.subject: both
are repeated fields per the UDM field list, so a scalar replace is a
type mismatch. The unguarded subject replace could have failed parsing
for every failure event with a subject.
- Guard testing with != "": parsedmarc emits the RFC 9990 t= flag as the
string "y"/"n", not a boolean, so the == "true"/"false" test could
never fire and dmarc_testing was never emitted. Drop testing from the
boolean convert list accordingly.
- Add on_error to the four numeric %{} interpolations (count,
source_asn, *_session_count) so a tenant where interpolating a
non-string field errors degrades to a missing additional.fields entry
instead of _failed_parsing_ (per the official parser tips doc).
- Drop the concatenated security_result.description strings: the tips
doc says not to pack multiple values into one UDM field, and every
value already lands individually in additional.fields.
- Use TAG_MALFORMED_MESSAGE for non-JSON input, matching the tips doc's
canonical not-JSON example (TAG_MALFORMED_ENCODING is for character
encoding problems).
- Map org_extra_contact_info/errors (aggregate) and the source
enrichment fields (failure) into additional.fields; the failure branch
previously dropped enrichment the aggregate branch kept.
- Replace the SMTP TLS success sample, which was not generated from this
repository's samples, with rfc8460.json output; regenerate all sample
events with the fixed serializer and name each event's source sample
file. All six samples are now byte-identical to real syslog output.
- Reorder the validation plan to start with the failure sample: it is
the only shape that emits JSON nulls, whose json{} interaction with
the pre-initialized "" fields is undocumented.
- Update Google doc links to docs.cloud.google.com (the old URLs 301)
and link the parser from the syslog section of the usage docs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
RFC 8460 §4.3 nests each failure detail inside a policy, but
parsed_smtp_tls_reports_to_csv_rows only attached policy_domain and
policy_type to the per-policy summary row. Failure-detail rows therefore
had empty policy_domain/policy_type CSV columns, and flat-JSON consumers
(syslog, GELF) could not attribute a failure detail to its policy — the
Google SecOps parser in this PR could not even detect those rows as SMTP
TLS reports, since they carried none of the shape-identifying fields.
Also rebuild the row template per policy: policy_strings and
mx_host_patterns from an earlier policy leaked into a later policy that
did not define them, because the template dict was created once per
report and mutated inside the policies loop.
Both regression tests fail on the previous serializer with
KeyError: 'policy_domain'.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Google's content-hub parsers tag dropped logs (drop { tag => "TAG_..." })
so they surface correctly in the unparsed-log views, rather than bare drop{}.
Use TAG_MALFORMED_ENCODING for the two JSON-extraction/parse failures (matches
content-hub) and TAG_UNSUPPORTED for valid JSON that matches no parsedmarc
report shape.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two corrections confirmed against Google's official content-hub parsers
(content/parsers/third_party/community/*/cbn):
1. Numbers as numbers. count, source_asn, successful_session_count and
failed_session_count were being stored in additional.fields as string_value.
Store them as number_value instead (build string -> convert to uinteger ->
rename to number_value, the content-hub idiom), so SecOps can range-query and
sort them, per parsedmarc's "store numbers as numbers" rule. Booleans stay
string_value (content-hub never uses bool_value) and are still converted in
step 1b for the == "true"/"false" comparisons.
2. Conditional guards. Replaced bare `if [field] {` with `if [field] != "" {`
(76 guards + the detection cascade + policy_override). After 1a initializes
every tested field to "", a bare `if` is true for an empty field (Logstash/CBN
semantics), which would misfire detection and emit empty labels. content-hub
uses `!= ""` ~111x vs 2 bare (both flags); parser flags (no_json_payload,
not_json, *_nan) correctly stay bare.
Verified: braces balance, no stray bare field-guards, all if-tested fields
initialized, all four numeric fields emit number_value.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add github.com/chronicle/content-hub (Google's official third-party SecOps
parser repo) to the README references and re-anchor the in-code citations to
it. Its current CBN parsers (e.g. CLOUDFLARE_PAGESHIELD, Copyright 2025 Google
SecOps) confirm both fixes this parser makes: initialize every field before the
json{} filter, and convert JSON booleans/numbers to strings before comparison.
Replaces the dated "How to parse JSON data" citation with the authoritative,
actively-maintained source.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a short, skippable callout explaining what a parser / configuration-based
normalizer (CBN) is and how it fits the SecOps ingest flow (log type → parser →
UDM event), so the README serves newcomers without slowing experienced users.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add the remaining official Google docs the parser is built on (parser tips
& troubleshooting, manage parsers, UDM search, Bindplane install) and a
clearly-separated "Additional sources and tooling" section for the community
resources that drove the JSON type-handling and field-init fixes
(thatsiemguy's Parsing 101, the Corelight production parser, chronicle/cbn-tool).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
xml_schema is aggregate-only (failure/SMTP TLS rows don't carry it) and a
distinctive, non-generic field name, addressing the concern that "domain"
could be confused with other logs. parsedmarc defaults xml_schema to "draft"
when the report omits <version> (parsedmarc/__init__.py:832), so it survives a
missing version element -- unlike a field with no default.
It is also a native JSON string straight out of the json{} filter, so unlike
dmarc_aligned it needs no convert step to be testable, keeping detection
independent of the type-conversion in step 1b. xml_schema is added to the
pre-json init block (required for any if-tested field); domain stays
initialized since it is still mapped to target.hostname.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two CBN behaviors, confirmed against Google's own "How to parse JSON data"
guide (statedump shows JSON true/199 retaining boolean/integer type) and the
published Corelight production parser:
1. The json{} filter preserves the original JSON type, so parsedmarc's boolean
*_aligned / testing / normalized_timespan and numeric count / *_session_count
/ source_asn would never match string comparisons. Add a mutate{convert} step
turning them into strings before any == "true"/"false" test or %{...} use.
2. CBN raises _failed_parsing_ when an `if [field]` references a field absent
from the log, and most detection/mapping fields are absent in 2 of the 3
report shapes (or null within one). Initialize every conditionally-checked
field to "" before the json{} filter.
Without these, DMARC-fail records would not be categorized AUTH_VIOLATION and
aggregate/TLS reports could fail parsing outright. README caveat and PR
validation steps updated accordingly.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
adkim is the published policy's DKIM alignment mode (defaulted to "r" by
parsedmarc), an obscure thing to key detection on. Switch the aggregate
detector to "domain" -- the reported From-domain, a required element present
and non-empty in every aggregate record (2388/2388 sample rows) and unique to
aggregate (failure uses reported_domain, SMTP TLS uses policy_domain).
header_from is unsuitable: it can be empty when a record carries no
identifiers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A SecOps-side custom parser (CBN) that maps parsedmarc's [syslog] JSON
events to the Unified Data Model. No library changes: parsedmarc already
emits structured JSON, so the DMARC->UDM mapping lives in the parser and a
downstream UDM schema change is a parser edit, not a parsedmarc release.
Covers all three report types:
- aggregate -> EMAIL_TRANSACTION
- failure -> EMAIL_TRANSACTION
- smtp_tls -> GENERIC_EVENT (noun from policy_domain, present on every row)
Built strictly against the official UDM and parser-syntax docs (cited
inline). Sets metadata.event_timestamp from the report window via date{},
maps disposition / auth-failure to security_result with valid action and
category enums (AUTH_VIOLATION on DMARC fail), uses real network.email
field names, and strips syslog framing before JSON parsing. Ships real
sample events generated from the project's sample reports for validation.
Not yet validated against a live SecOps tenant; caveats are documented in
the README.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Make Microsoft Graph connection activity observable
parsedmarc only configured its own logger, so all Graph connection
activity was silently dropped even with --debug: the mailbox layer
logs under mailsuite.mailbox.graph, token acquisition under
azure.identity (including the AADSTS error codes that distinguish a
local config problem from an Exchange Online / Entra ID one), and
HTTP traffic under httpx/msgraph — none of which had a handler or
level set. _main() also logged nothing around the MSGraphConnection
call, so a hang left no trace at all.
Three changes, all parsedmarc-side (no mailsuite changes needed):
- Log a redacted connection summary at INFO before connecting (auth
method, tenant ID, client ID, mailbox, Graph URL) plus a --debug
detail line with certificate path, token-file path, and set/not-set
flags for secrets. Secret values are never logged; a regression
test asserts they don't appear in captured output.
- Log a timing line after the connection object is initialized.
- Propagate parsedmarc's --verbose/--debug level and handlers to the
dependency loggers (mailsuite, azure, msgraph, httpx, httpcore) via
_configure_dependency_logging(), synced to exactly the parsedmarc
logger's handlers so SIGHUP log-file swaps neither duplicate output
nor write to closed handlers. At the default level dependency
loggers sit at WARNING, so their warnings keep surfacing (formatted)
without new noise.
All four new tests fail on the unfixed code (verified by stashing the
cli.py change).
Fixes https://github.com/domainaware/parsedmarc/issues/814.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Disable propagation on dependency loggers; document kiota's absence
Set propagate=False on the dependency loggers when syncing handlers, so
a stray logging.basicConfig() anywhere in the process cannot
double-print every dependency record through the root logger — the
function already owns these loggers' handler lists, and this makes that
ownership complete. Asserted alongside the existing level/handler checks.
kiota_http and its sibling packages were considered for
_DEPENDENCY_LOGGERS but verified to not use Python logging at all
(their observability is OpenTelemetry tracing), so a comment now
records why they are absent rather than leaving the omission to be
"fixed" later.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: MISAPOR LAB <misapor@lab.misapor.pl>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>
* Stop system GeoIP files from shadowing the bundled IPinfo database
_get_ip_database_path() searched well-known system paths (including
/usr/share/GeoIP/GeoLite2-Country.mmdb and CWD-relative names) before
the database parsedmarc manages, so on any host with a distro GeoIP
package installed every lookup silently used a country-only — and
often years-old — database instead of the bundled IPinfo Lite one.
That disabled ASN enrichment entirely (asn/as_name/as_domain were None
for every IP) and with it the ASN-fallback path into the reverse-DNS
map, with no signal beyond a generic "IP database is more than a
month old" warning. Verified live on a Fedora host whose distro
GeoLite2-Country.mmdb dated to December 2019.
New precedence: explicit ip_db_path -> _IP_DB_PATH selected by
load_ip_db() (downloaded/cached/bundled) -> the bundled copy -> system
paths as a true last resort (only consulted when the bundled data
file is missing). The selected file is logged at debug level so a
--debug run shows which database answered.
The automatic system-path pickup was documented behavior, so
installation.md now tells MaxMind GeoLite2 users to set ip_db_path
explicitly, with a migration note.
Both new regression tests reproduce the shadowing portably via a decoy
CWD GeoLite2-Country.mmdb (the fallback list includes relative names),
and fail on the unfixed code (verified by stashing the source change).
Fixes https://github.com/domainaware/parsedmarc/issues/810.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address review: accurate fallback comment, dedup selection log, cover fallback tiers
- Correct the nothing-found-anywhere comment: the os.stat() age check
raises FileNotFoundError before the caller's open_database() would.
- Log "Using IP database at ..." only when the selected path changes
instead of on every uncached IP lookup, so --debug runs over large
batches aren't flooded; tracked via _LAST_LOGGED_IP_DB_PATH, reset in
the test fixture for order-independence.
- Cover the previously untested branches of _get_ip_database_path:
system-path fallback when the bundled database is missing, the
FileNotFoundError when nothing exists anywhere, the stale-database
warning, and the log-once-per-path behavior.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: MISAPOR LAB <misapor@lab.misapor.pl>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>
* Fix failure-report timestamp skew on non-UTC hosts in ES/OS/Splunk sinks
arrival_date_utc is a UTC wall-clock string (generated in
parse_failure_report via an aware-UTC strftime), but elastic.py,
opensearch.py, and splunk.py parsed it back into a naive datetime and
called .timestamp(), which per the Python docs interprets naive values
as local time (https://docs.python.org/3/library/datetime.html#datetime.datetime.timestamp).
On any non-UTC host the epoch stored as the ES/OpenSearch arrival_date
field, used in the failure-report dedup match query, and sent as the
Splunk HEC event time was therefore off by the host's UTC offset
(verified -3600 s under TZ=Europe/Warsaw in January).
Add an assume_utc keyword to human_timestamp_to_datetime() /
human_timestamp_to_unix_timestamp() that attaches timezone.utc to naive
parses, and use it at the three arrival_date_utc call sites. Aware
inputs (explicit offsets) are unaffected; all other callers keep the
existing local-time semantics, whose round-trip with timestamp_to_human
is self-consistent on a single host (the broader local-time output
question is tracked separately in issue #811 bug 2).
The three new sink regression tests fail on the unfixed code
(verified by stashing the source changes) and force TZ=Europe/Warsaw
via time.tzset() so they catch the skew even on UTC CI runners.
Fixes half of https://github.com/domainaware/parsedmarc/issues/811.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Deduplicate TZ-forcing test boilerplate; fix unix-timestamp docstring
Extract the repeated TZ=Europe/Warsaw + time.tzset() setup/cleanup from
the four timestamp regression tests into a shared tests/tzutil.py
force_tz() helper, and correct human_timestamp_to_unix_timestamp()'s
docstring, which said the return type was float while the function
returns int.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: MISAPOR LAB <misapor@lab.misapor.pl>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>
* draft fix MS Graph
* Add ClientAssertion auth method support for MS Graph in cli.py
MSGraphConnection/AuthMethod (via mailsuite) already supported
ClientAssertion, but cli.py never parsed a client_assertion config
value or passed it through, so it was unusable from the CLI. Wire
config_msgraph.client_assertion through _parse_config and the
MSGraphConnection call, and document the auth method (including its
short-lived-JWT caveat vs. Certificate/ClientSecret for watch mode).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: MISAPOR LAB <misapor@lab.misapor.pl>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
requires-python stays ">=3.10" (already correct and matching the CI matrix
of 3.10-3.14); add the per-version Programming Language :: Python :: 3.10-3.14
classifiers and "3 :: Only" so the PyPI page lists supported versions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Explain why a report is invalid instead of "Not a valid report"
The parser catches broadly so one malformed report can't crash a batch,
but every failure surfaced as the generic ParserError("Not a valid
report"), telling operators nothing about the cause.
parse_report_file() now keeps each format parser's specific error as it
tries aggregate XML -> SMTP TLS JSON -> report email, and when all three
reject the input it content-sniffs the leading byte to surface the single
relevant reason (e.g. "Invalid aggregate report: Missing field:
'org_name'", or "Not a recognized report format (...)"). The CLI already
logs str(error), so this reaches the user with no cli.py change.
Every parser catch site also re-raises with `raise ... from <original>`,
preserving the underlying ExpatError / JSONDecodeError / KeyError /
archive errors on __cause__ for library callers and tracebacks. The same
exception *types* are still raised.
Finally, the catch-all "unexpected error" branches append
`(raised at <file>:<line>)` from the deepest traceback frame, but only
when the parsedmarc logger is at DEBUG level (e.g. the CLI's --debug);
normal-level output is unchanged.
Bumps the in-progress version to 10.2.0 and documents all three in the
CHANGELOG.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Cover the failure-report path in parse_report_file error tests
The reason-surfacing tests covered the aggregate and SMTP TLS branches but
not failure reports, which reach parse_report_file only via the email
path. Add a malformed multipart/report failure email (missing the required
Source-IP) and assert the message names the failure format and the missing
field rather than collapsing to "Not a valid report".
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Cover the new error-reporting lines; drop one unreachable catch
Bring the lines added by this PR to full test coverage:
- Test _exc_origin() with a never-raised exception (no __traceback__) so the
"no frames" guard is exercised.
- Test _parse_smtp_tls_failure_details() with a non-dict, which raises
TypeError (not KeyError) and exercises the generic catch-all.
- Test parse_report_email() with an unparseable Date header, which trips the
initial mail-parse catch-all and becomes a ParserError.
Two dead lines are removed rather than hidden, per the project's "delete
unreachable branches, no # pragma: no cover" rule:
- _looks_like_email() looped with a `continue` for blank lines, but every
caller passes lstrip()-ed text, so the first line is never blank. Simplified
to inspect the first line directly.
- parse_report_email()'s `except Exception` after `except InvalidFailureReport`
was unreachable: parse_failure_report wraps its entire body and provably
raises only InvalidFailureReport, which the preceding handler already catches.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix IndexError when backfilling envelope_from from SPF results
_parse_report_record() backfills a missing/empty envelope_from from the
last SPF auth result's domain. The "envelope_from is None" branch gated on
the raw auth_results["spf"] list but indexed the filtered
new_record["auth_results"]["spf"] list, which only holds results that have
a domain. A reporter sending an SPF result with no domain made the filtered
list empty while the raw list was non-empty, so [-1] raised IndexError and
the whole record failed to parse.
The two near-identical envelope_from backfill branches (missing identifier
vs. empty identifier) drifted apart -- only one was updated when the
filtered new_record list was introduced -- which is what let them disagree
on which list to read. Merge them into a single path, keyed on
dict.get("envelope_from") is None, that gates and indexes the same raw list
with the "domain" membership guard the missing-identifier branch already
used.
Regression test: envelope_from=None with an SPF result carrying no domain
now parses to envelope_from=None instead of raising. This is the bug that
motivated the surrounding error-reporting work in this PR.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Trim comment
* Cover the touched error branches; drop a dead UnicodeDecodeError catch
Codecov flagged the pre-existing error branches this PR touched (adding
`from e` / `_exc_origin`) as changed-and-uncovered. Most are real
malformed-report paths, so add honest tests that drive them with realistic
inputs:
- parse_smtp_tls_report_json: nested missing key (date-range without
start-datetime) -> InvalidSMTPTLSReport chaining a KeyError.
- parse_aggregate_report_xml: non-structured report_metadata -> the
AttributeError branch ("Report missing required section").
- parse_report_email: valid legacy text/plain failure report (success path),
a text report missing its fields, a base64 attachment of malformed
aggregate XML, and one of invalid SMTP TLS JSON.
- parse_report_file: gzipped junk -> the str branch of the content sniff.
The `except UnicodeDecodeError` in extract_report is removed as dead code
(no `# pragma: no cover`, per the repo rule): str-mode streams are already
rejected by explicit isinstance checks, and every decode() uses
errors="ignore", so it can never fire. str-mode still raises ParserError.
Also rename the two new failure-report tests from "Forensic" to "Failure"
and add an AGENTS.md rule: RUF reports are "failure reports"; "forensic" is
reserved for the literal backward-compat alias identifiers only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix pyright errors in the new error-branch tests
CI runs pyright over the whole repo (tests included); these slipped through
because the local check only covered parsedmarc/__init__.py:
- _parse_smtp_tls_failure_details("not a dict") is a deliberate wrong-type
test -> targeted `# pyright: ignore[reportArgumentType]`.
- result["report"]["source"]["ip_address"] on a ParsedReport TypedDict ->
cast(FailureReport, result["report"]) first, matching existing tests.
This is what failed lint-docs-build (and, since `test` needs it, skipped the
Codecov upload) on the prior commits.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The module-level try/except import is environment-dependent: with
psycopg installed the ImportError fallback never runs, and without it
(CI's test job) the successful-import arm never completes — so Codecov
flags one side or the other no matter where coverage is measured (it
flagged the import line on master right after #798 merged).
Exercise both arms explicitly: execute the module's source into a
fresh, throwaway module object (importlib.util.module_from_spec +
exec_module) under a patched sys.modules — a None entry forces
ImportError, fake module entries force the success path — and assert
on the psycopg / psycopg_json bindings each arm produces. The
throwaway-module approach (rather than importlib.reload) leaves the
canonical parsedmarc.postgres untouched, so the identity of
PostgreSQLError / AlreadySaved held by the rest of the test module is
preserved.
Verified covered in both environments: with the venv's real psycopg,
and with psycopg hidden via a PYTHONPATH shim to simulate CI; the
import block reports no missing lines either way.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sean Whalen
KiliandClaude Sonnet 5
github-actions[bot]andseanthegeek
Ivan The Geek