mirror of
https://github.com/domainaware/parsedmarc.git
synced 2026-07-30 12:24:55 +00:00
b304cf8639bfcb7e7a6cbb5ba1756bcb52aac7a9
20
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  
|
b304cf8639 |
Cover index_prefix_domain_map and index_suffix in index migrations (#868) (#869)
* Cover index_prefix_domain_map and index_suffix in index migrations (#868) The startup Elasticsearch/OpenSearch index migrations built their target index names from the [elasticsearch]/[opensearch] index_prefix and index_suffix options alone, while the save path also honors general.index_prefix_domain_map. A multi-tenant deployment therefore ran the backfill guard `count` against dmarc_aggregate*/smtp_tls*, patterns matching none of its real <tenant>_dmarc_aggregate-* indexes. The query passes allow_no_indices=True, so a zero-match wildcard returns count 0 -- indistinguishable from "already backfilled" -- and the backfill was skipped silently, with no log line at any level. Resolve migration index names through a new _migration_index_names() helper that widens both configurable axes: one name per tenant prefix in the map plus the unprefixed name (unmapped domains are still saved unprefixed), and, when an index_suffix is set, the unsuffixed name alongside the suffixed one so history predating the suffix is covered. A configured index_prefix still wins outright and suppresses the map fan-out, matching save-time precedence. The key normalization is now shared with get_index_prefix() via _normalize_index_prefix(), so the names parsedmarc migrates cannot drift from the ones it writes. The resolved lists are logged at DEBUG, and the SIGHUP reload path passes the freshly parsed map, so a newly onboarded tenant is covered without a restart. Also repair the legacy published_policy.fo migration, which has been unable to complete since mapping types were removed in Elasticsearch 7 (and never existed in OpenSearch): it read the field mapping in the type-keyed response shape, so the check always fell through, and its put_mapping() call passed a doc_type argument neither current client accepts. It now reads either response shape, uses each client's current signature, and takes its index names in a separate legacy_fo_indexes argument -- exact names, since 5.0.0 introduced date-suffixed index names in the same release that fixed the fo declaration, but prefixed and suffixed where configured, since both options date back to 4.1.0. Tenant prefixes are excluded: index_prefix_domain_map arrived in 8.19.0, and this migration renames the index it rebuilds. The Elasticsearch copy, removed as unreachable during the #806 client migration, is restored now that the cause is understood. Finally, reject an index_prefix_domain_map YAML file that is not a mapping of string tenant names to lists of domains, instead of raising mid-save on a non-string key or silently matching the wrong domains on a scalar value -- `in` on a str is a substring test, so "example.co" matches "example.com" (https://docs.python.org/3/reference/expressions.html#membership-test-operations). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Fix OpenSearch capitalization and spacing in the usage docs Copilot review of #869: the `index_prefix_domain_map` option line spelled "OpenSearch" as "Opensearch" and had a doubled space before the type. Both predate this branch but sit inside a hunk it rewrites. Also wrapped the line to match the continuation-indent style of every other option in the list, and corrected the same misspelling in the multi-tenant section a few lines above the paragraph this branch added. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Require index_prefix_domain_map domain lists to hold strings Copilot review of #869: the shape check verified that each value was a list but not what the list contained, so `tenant_a: [42]` passed and then never compared equal to any domain the save path looks up -- the same silent-misbehavior class the check exists to reject, and a contradiction of the "any other shape is rejected at startup" claim in the docs. Check the list's items too, and reword the error message, comment, CHANGELOG and docs to state the rule the check now enforces: a mapping of tenant names to lists of domain names, all strings. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Make the legacy fo migration retry-safe, and nest its mapping body Copilot review of #869, verified against live Elasticsearch 8.19 and OpenSearch 3 containers rather than mocks. Retry safety (a real defect): an attempt interrupted between creating the -v2 index and deleting the original left debris that made create() fail with "resource already exists" on every later startup, inside the same try that swallows the error -- so the index was never migrated and the debris document survived. Reproduced on a live cluster. Discard a leftover target first; that is safe precisely because reaching this point means the original still holds the data, since it is deleted only once the reindex has succeeded. Mapping body: the reviewer's concern that a dotted key under `properties` risks a runtime failure does not hold -- both clusters accept it and produce a byte-identical mapping, with the dot expanded into published_policy -> properties -> fo. Switch to the nested object form anyway, since dot expansion is conditional on the object's `subobjects` setting and this shape never is, and derive the object/leaf names from the dotted constant so the write cannot drift from the field the read looks up. Both fo-migration suites now build per-name Index mocks. A single shared mock cannot express "the original exists but its migration target does not", which is the ordinary case and the one the retry fix turns on; the happy path now also asserts the target index is never deleted. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
|
35e51218b9 |
Don't archive/delete mailbox messages until output destinations confirm the save (#863)
* Don't archive/delete mailbox messages until output destinations confirm the save (#242) get_dmarc_reports_from_mailbox() gains a keyword-only save_callback, invoked once per fetched batch before any message is deleted or moved. A False return (or an exception) marks the batch unsaved: its messages stay in the reports folder for retry, and the aggregate-report dedup keys staged for that batch are dropped so the retry reparses instead of skipping. watch_inbox() passes its callback through as save_callback, so watch mode gets the same protection. To bound duplicate delivery to destinations that don't deduplicate, a new [mailbox] max_unsaved_retries option (default 2) caps retries: a message whose batch has failed the initial attempt plus that many retries moves to {archive_folder}/Unsaved -- never deleted, whatever the delete options say. Counts are process-local, so the cap applies across watch-mode checks; one-shot runs retry indefinitely, which is the safe direction. A raising callback (the CLI's, under fail_on_output_error) is counted against the cap the same way before the exception is re-raised, since mailsuite's IMAP and Maildir watch loops swallow exceptions and keep checking. CLI: process_reports() now returns its output-error list and the mailbox_save_callback adapter feeds that verdict to the library; save_output() failures (the one uncaught destination) are recorded like every other sink's; file/mbox-derived reports are saved in a separate pass from mailbox batches so nothing is saved twice; the combined results fed to email_results() are filtered by index_prefix_domain_map explicitly, restoring the SMTP TLS filtering the emailed summary lost when saving moved into per-batch callbacks. Credit to @mkilijanek for the original approach in #823. Fixes #242 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address automated review feedback on #863 - Keep a message's failure counter until its move to Unsaved actually succeeds. The counter was popped when the message was classified over-cap, before the move was attempted, so a failed move handed the still-in-place message a fresh set of under-cap retries and duplicate deliveries; now the next failed save classifies it over-cap again and re-attempts the move. Regression-tested at cap 1, where the reset would observably leave the message in the INBOX instead of moving it. - Use sys.exit(1) instead of the site-dependent exit() built-in in the two new ParserError handlers. - Close the sample files opened by the new Maildir tests via context managers. - Keep the docs' section-link text on one line and fix the adjacent pre-existing "a IMAP" typo. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address Copilot re-review feedback on #863 - Document mailbox_save_callback's raise path: with fail_on_output_error it raises ParserError via process_reports() instead of returning False, and the library counts that as an unsaved batch too. - Use str(error_) instead of error_.__str__() in the new File output handler. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Reject negative max_unsaved_retries with a ValueError The option is user-configurable (INI/env/kwarg); a negative value silently behaved like 0. Both get_dmarc_reports_from_mailbox() and watch_inbox() now validate it at the door alongside the existing test/delete guard. watch_inbox() validates before entering the watch loop, because a ValueError raised inside a check would be swallowed and endlessly retried by the IMAP and Maildir backends' per-check exception handling instead of surfacing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix two pre-existing bugs surfaced by review of #863 - MAGIC_JSON was b"\7b" -- the octal escape \7 (BEL) plus a literal "b", not 0x7B, the "{" every RFC 8259 JSON object begins with -- so extract_report() rejected plain uncompressed JSON. Every in-tree caller pre-guarded with its own zip/gzip or "{" check, masking the dead branch; the practical impact was extract_report() as a public API and application/tlsrpt+gzip attachments whose payload is really uncompressed JSON. Now b"\x7b". - get_index_prefix() unconditionally indexed policies[0], while parse_smtp_tls_report_json() accepts a report whose policies list is empty -- an IndexError crash whenever index_prefix_domain_map was configured. An empty-policies report has no domain to map, so it is now treated as unmappable and excluded from prefix-mapped output. This code was moved into filter_smtp_tls_reports_for_index_prefix() by this PR, which widened its exposure to the email_results() path. Both regression tests were verified to fail against the unfixed code. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Rewrap the max_unsaved_retries docstring paragraph The ValueError sentence was inserted without reflowing, leaving a 112-character line in a docstring wrapped at ~76; ruff formats code, not docstring prose, so it slipped through lint. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
ae5f4c7915 |
Add per-report-type mailbox delete options (#858)
* Add per-report-type mailbox delete options (#256) Add four new [mailbox] options — delete_aggregate, delete_failure, delete_smtp_tls, and delete_invalid — each defaulting to the value of the overall delete option and individually overridable, so e.g. delete = True with delete_failure = False deletes processed aggregate and SMTP TLS report messages while archiving failure reports, and delete_invalid = False keeps unparseable messages in the Invalid archive subfolder for debugging. get_dmarc_reports_from_mailbox() and watch_inbox() gained matching bool | None keyword arguments (None = inherit from delete), resolved once up front; the delete/test mutual-exclusion guard now checks the effective per-type flags. The Gmail deletion-scope guard covers any effective flag and forces all five options off when the scope is missing. PARSEDMARC_MAILBOX_DELETE_* env vars work automatically. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Document per-type delete kwargs as bool | None with explicit inheritance Copilot review: the docstrings typed the four per-report-type delete parameters as plain bool, but None (the default) is the inheritance mechanism — a library caller couldn't tell from the docs that None, not False, means "inherit delete". Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Cover disposal error branches; scope the config= docstring claim Codecov flagged the delete-error handler in the new per-type disposal loop as the one uncovered patch line. Add two Maildir tests driving a backend whose first delete/move call raises: the error is logged, the affected message stays in the INBOX, and disposal continues to the next report type. The move-error branch gets the symmetric test from the same harness. Copilot review: the config= docstring paragraph claimed all keyword arguments listed above it are ignored when config= is provided, which now falsely included the four per-type delete options. Scope the claim to the parsing/enrichment arguments ParserConfig actually carries and state that mailbox-handling arguments always apply. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
264b9a4556 |
Add [general] archive_directory to archive processed local files (#570) (#856)
* Ignore Claude Code agent worktrees under .claude/worktrees/ Untracked repo snapshots from agent sessions were making repo-root ruff check . fail on stale code and cluttering git status. ruff respects .gitignore, so ignoring the directory fixes both. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Add [general] archive_directory to archive processed local files (#570) Move successfully processed report files given as local path arguments into <archive_directory>/<year>/<month>/<Aggregate|Failure|SMTP-TLS>/, dated from the parsed report's own metadata (aggregate begin_date, failure arrival_date_utc, SMTP TLS begin_date) rather than the filename. Files that fail to parse as a report (ParserError) go to <archive_directory>/Invalid/; other failures (e.g. transient I/O errors) leave the file in place so a later run can retry it. Existing destination files are never overwritten: each candidate name is claimed atomically (O_CREAT | O_EXCL) and collisions get a numeric suffix before the extension. Files already inside the archive directory are excluded from processing (compared via realpath so symlinked spellings still match), so the archive can safely live inside an input directory, as the issue requests. mbox files and mailbox modes are unaffected; mailbox modes keep [mailbox] archive_folder. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address review feedback on docstrings and the placeholder-cleanup except - _move_file_to_archive's docstring no longer claims the copy2 fallback replaces the placeholder atomically; only the same-filesystem os.rename path is atomic. The fallback is a plain copy-and-overwrite, which is still collision-safe because the placeholder already claimed the name. - The empty except OSError in the placeholder cleanup now carries a comment explaining that it is deliberate best-effort cleanup and the re-raised move failure is the actionable error. - test_general_archive_directory_unset_leaves_attribute_absent's docstring now describes what the assertion actually tests (the attribute staying absent from a bare Namespace) and moves the real-CLI None-default behavior to a parenthetical. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Cover the two defensive exception branches Codecov flagged Codecov's patch report flagged four uncovered lines, all in the two platform-dependent exception branches of the archive helpers: - _exclude_archived_paths's except ValueError branch. Per the Python docs for os.path.commonpath, ValueError is raised when paths "are on the different drives" (Windows) or mix absolute and relative pathnames; both inputs are realpath()-resolved so only the different-drives case remains, which Linux CI can't produce naturally. The new test simulates the raise and asserts the non-comparable path is kept for parsing rather than excluded. - _move_file_to_archive's except OSError placeholder-cleanup branch. The new test fails the move with a non-OSError type and the cleanup with OSError, then asserts the move error is what propagates (the cleanup error is swallowed, not allowed to mask it) and that the zero-byte placeholder survives its failed cleanup. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Tag arrival_date_utc as UTC when parsing the archive date human_timestamp_to_datetime's docstring names arrival_date_utc as exactly the kind of known-UTC naive string that should be parsed with assume_utc=True; _archive_subdir_for_result was parsing it naive. The flag is scoped to the failure branch because the shared call also handles the other two report types: aggregate begin_date is a local-time wall-clock string (timestamp_to_human uses datetime.fromtimestamp), so tagging it UTC would be wrong, and SMTP TLS begin_date carries an RFC 3339 offset, making assume_utc a no-op. No observable behavior change: only the wall-clock year/month fields are read and assume_utc never shifts wall-clock time, so no new test can honestly distinguish the two versions — this aligns the call with its dependency's documented contract and hardens against a future edit adding a real timezone conversion. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Create the archive placeholder with mode 0o600 os.open's mode parameter defaults to 0o777 (masked by the umask), so the O_CREAT|O_EXCL placeholder in _move_file_to_archive was created executable and group-accessible on typical umasks (0o775 under umask 002). Normally it's replaced immediately, but a placeholder that outlives a failed move+cleanup persisted with those permissions. Pass 0o600 explicitly. The mode never reaches the real archived file: os.rename replaces the placeholder's inode outright, and the copy2 fallback's copystat overwrites the mode with the source file's. The leftover-placeholder regression test now also asserts the surviving placeholder has no owner-exec or group/other bits (umask-independent, since the umask only clears bits); the assertion fails against the unfixed default-mode call. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
4e80047e68 |
Centralize configuration handling with ParserConfig (#503) (#851)
* Centralize configuration handling with ParserConfig (#503) Add parsedmarc/config.py with ParserConfig, a frozen dataclass carrying every parsing/enrichment option plus the three shared caches (IP address info, seen aggregate report IDs, reverse DNS map). All eight public parsing/mailbox functions accept a keyword-only config= argument; when provided, the individual option keyword arguments are ignored in favor of the config's values, and every existing per-option keyword argument keeps working unchanged. The three hand-copied parse_kwargs dicts and the dns_timeout<->timeout rename chain are gone; the CLI builds one ParserConfig per run (rebuilt on SIGHUP) and passes it everywhere. Explicitly constructed configs own fresh isolated caches; dataclasses.replace() shares them; pickling drops cache contents and rebinds the unpickling process's module defaults, preserving the per-worker cache behavior of n_procs parallel parsing. The module globals IP_ADDRESS_CACHE / SEEN_AGGREGATE_REPORT_IDS / REVERSE_DNS_MAP remain, identity-preserved, as re-exports of the default caches. Bug fixes that ride along, each with a regression test: - One-shot mailbox runs now honor [general] dns_timeout/dns_retries; the CLI call site never forwarded dns_timeout, so the library's stray 6.0 default silently applied. - Lazily-triggered reverse DNS map loads (get_ip_address_info / get_service_from_reverse_dns_base_domain, including in n_procs workers) now thread psl_overrides_path/psl_overrides_url through to load_reverse_dns_map instead of clobbering operator-configured PSL overrides with the bundled defaults. - get_dmarc_reports_from_mailbox() and watch_inbox() dns_timeout defaults unified to DEFAULT_DNS_TIMEOUT (2.0s) from a stray 6.0, and normalize_timespan_threshold_hours to the float 24.0 used everywhere else. Closes #503 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address CI and review feedback on #851 - Wrap the IMAPConnection example in usage.md so ruff format is clean over the docs code blocks (CI runs ruff format --check on the whole repo; the local runs were scoped to parsedmarc/ and tests/ and missed it). - Fix the pre-existing "URL ro a reverse DNS map" docstring typo in get_service_from_reverse_dns_base_domain, caught by Copilot on the adjacent hunk. - Import parsedmarc.config once, as an aliased plain import, in tests/test_config.py instead of mixing import and import-from of the same module (flagged by code quality scanning); the aliased module import also keeps pyright able to resolve the submodule attribute access. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address second Copilot review round on #851 - Add parse_aggregate_report_file() to the library entry-point list in usage.md; the following paragraph describes the config= contract for "each of these functions", so the list must name all eight config-accepting entry points. - ParserConfig.__setstate__ now initializes every non-cache field to its class default before applying the pickled state, so a config serialized by an older parsedmarc version (whose state predates fields added later) unpickles with the newer fields at their defaults instead of unset entirely (__init__ never runs during unpickling, so an absent field would raise AttributeError on first access). Covered by a regression test that feeds __setstate__ a partial state dict. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
48445c639e |
Extend n_procs parallel parsing to mbox and mailbox sources (#147) (#849)
* Extend n_procs parallel parsing to mbox and mailbox sources (#147) n_procs previously applied only to report files passed directly as CLI arguments; messages from mbox files and mailbox connections (IMAP, Microsoft Graph, Gmail API, Maildir) were always parsed sequentially. A new parsedmarc.parallel module provides a shared bounded-window ProcessPoolExecutor helper (parallel_map) used by all three input paths. Only parsing fans out to a reused worker pool; message fetching, report deduplication, mailbox archiving/deletion, and output stay sequential in the main process. The submission window keeps at most ~2*n_procs messages in flight, so memory stays bounded even for huge mboxes, and the mailbox path fetches messages lazily on the connection-owning main thread. keep_alive never crosses the process boundary - the main process sends periodic IMAP keepalives while workers parse - and with n_procs > 1, invalid-message disposition happens after the parse phase, mirroring the existing deferred bulk archive moves. get_dmarc_reports_from_mbox, get_dmarc_reports_from_mailbox (including its tail-recursive re-check), and watch_inbox gain an n_procs keyword argument (default 1); sequential behavior at the default is unchanged. Replacing the CLI's hand-rolled Pipe/Process batching also fixes two defects in the direct-file path: a child process that died from a non-ParserError exception left the parent blocked forever on conn.recv(), and the hard batch barrier let one slow file idle every other worker slot. Workers are now a reused pool (no fresh interpreter per file), with worker logging reconstructed via a spawn-safe pool initializer instead of fork inheritance. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address Copilot and code-quality review feedback on #849 - parallel_map now validates n_procs >= 1 itself with a clear error instead of surfacing ProcessPoolExecutor's max_workers error later. The check raises eagerly at the call (the generator body moved into an inner function) rather than on first iteration, with a regression test. - Aligned parallel_map's should_stop docstring with the implementation: queued-but-unstarted jobs are cancelled, while in-flight jobs are waited on and their results yielded, so the stop can block briefly but never discards completed work. - The parallel mailbox path keeps fetched message ids in a deque popped as each in-order result arrives, so the id queue stays bounded by the submission window instead of growing to message_limit. - Closed the three sample-file handles the new tests opened without a context manager. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address second round of Copilot feedback on #849 - configure_logging no longer stacks duplicate FileHandlers when called again with the same log_file (compared by FileHandler.baseFilename, which stores the absolute path): a duplicate wrote every record twice and leaked a file descriptor per call, e.g. across SIGHUP config reloads. Latent in the pre-extraction cli._configure_logging too. Regression tests in the new tests/test_log.py. - Renamed the CHANGELOG's premature "10.4.0" heading to "Unreleased", matching the project convention where in-progress entries accumulate under Unreleased and the release PR renames the section and bumps parsedmarc/constants.py together (as in the 10.3.0 release). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
864f11e2be |
Accept directory paths as file_path arguments, add -r/--recursive (#843)
A directory given as a file_path argument now expands to the report files inside it with shell-glob semantics (dotfile entries excluded, subdirectories skipped). The new -r/--recursive flag descends into subdirectories and enables '**' recursion in glob patterns. Directory names containing glob metacharacters are escaped before expansion. Fixes the file_path help string's stray trailing apostrophe and refreshes the stale --help block in docs/source/usage.md. Closes #397 Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
4fa8cfb1e7 |
Skip the results email when no reports were parsed (#837)
* Skip the results email when no reports were parsed (#200) The [smtp] (and Microsoft Graph) results email was sent unconditionally whenever the transport was configured, so an empty run — an empty inbox, or one where every message was invalid — still emailed a zip of headers-only CSVs. The email step is now skipped with an INFO log when the run produced no aggregate, failure, or SMTP TLS reports. The regression test was verified to fail against the unfixed code (send_email was called once with a headers-only zip). Six existing tests that asserted on the email path with all-empty mocked results now feed a real parsed sample aggregate report through the actual zip/CSV-building code instead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Add Graph-path regression test for the empty-run email guard Copilot review on #837 pointed out that only the SMTP transport had a skip regression test, so a refactor narrowing the guard to the SMTP branch could silently reintroduce headers-only zips via Microsoft Graph. The new test was verified to fail against exactly that narrowing (send_message called once with a headers-only zip). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
df9bf82e04 |
Post-review follow-ups for Graph send (#825/#826) and requests-to-httpx migration (#827)
Follow-ups from the review of PR #825 (whose implementation had already landed on master via #826's stacked merge): - Honor the documented [smtp] attachment and [smtp] message options. Both were parsed into opts but never passed to either summary-email transport (also broken in released 10.2.2), so a configured custom attachment filename or message body was silently ignored. Both the SMTP and Microsoft Graph transports now receive them, and the missing smtp_attachment Namespace default is added (also covers SIGHUP reload, which rebuilds opts from the CLI Namespace). - Don't mislabel non-Graph mailbox errors as Microsoft Graph failures: the shared mailbox-fetch and watch handlers now log a generic "Mailbox Error" with traceback when the connection isn't Graph. - Declare microsoft-kiota-abstractions as a direct dependency (imported directly in cli.py for Graph error handling; previously transitive). Migrate all runtime HTTP from requests to httpx (webhook client, Splunk HEC client, and the PSL-overrides / IP-database / reverse-DNS-map / IPinfo-API fetches in utils.py): - follow_redirects=True everywhere to preserve requests' default redirect-following; httpx does not follow redirects by default. - The PSL-overrides and reverse-DNS-map fetches gain a 60s timeout (previously none), matching the IP-database fetch. - response.ok -> response.is_success; requests.RequestException -> httpx.HTTPError; raw string bodies use content= (httpx's data= is form-encoding only); Splunk HEC verification moves to client construction (httpx has no per-request verify). - requests drops out of [project] dependencies and moves to the [build] extra for the out-of-wheel maintainer script collect_domain_info.py, which deliberately stays on requests/urllib3 for its permissive-TLS adapter. - Remove the requests-era module-level urllib3.disable_warnings(InsecureRequestWarning) in splunk.py; httpx doesn't route through urllib3, so its only remaining effect was globally silencing insecure-TLS warnings from other urllib3-based components as an import side effect. Nothing imports urllib3 directly anymore, so it also leaves [project] dependencies. Tests: config-to-transport wiring for attachment/message on both transports (including defaults), non-Graph errors keep the generic log line, webhook/Splunk payload assertions moved to content=, and Splunk verify asserted at httpx.Client construction. 736 passed; ruff and pyright clean. Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
31c928d6fc |
Refresh Microsoft Graph docs: national clouds, examples, troubleshooting (#826)
* Send report summary via Microsoft Graph; make Graph failures observable
Two related fixes shipped together:
Send via Graph: the periodic DMARC summary email can now be sent
through the already-authenticated Microsoft Graph mailbox connection
(MSGraphConnection.send_message(), /users/{mailbox}/sendMail) instead
of only SMTP. Triggered when [msgraph] is configured and [smtp] has a
`to` value but no `host` -- SMTP is always preferred when `host` is
set, with no automatic fallback to Graph on SMTP failure. Reuses the
same connection used for reading; no new send-only config mode.
email_results()'s SMTP behavior is unchanged; a new
email_results_via_msgraph() shares its content-building logic via a
new _build_report_email_content() helper. Graph's sendMail always
sends as the authenticated mailbox, so [smtp] from is ignored on this
path -- documented, along with the required Mail.Send permissions and
a caveat that delegated auth flows (UsernamePassword/DeviceCode) don't
currently request that scope, so app-only auth is the supported path
for sending. Tracks #472.
Observable Graph failures: MSGraphConnection construction, mailbox
fetch, message send, and --watch failures now catch
ClientAuthenticationError/APIError/httpx.HTTPError specifically and
log one clear ERROR line naming the mailbox, tenant, auth method, and
the Graph request-id/client-request-id when available, instead of a
bare "MS Graph Error"/"Mailbox Error" with no context. Full traceback
still preserved at --debug. --watch previously had no Graph-specific
error handling at all -- a Graph error there crashed with a raw
uncaught traceback; it now exits the same way as the other three
sites. No new config options.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* Refresh Microsoft Graph docs: national clouds, examples, troubleshooting
The [msgraph] docs were accurate but missed guidance the community has
been asking for:
- graph_url now lists the actual national/sovereign-cloud endpoint
values (GCC High, DoD, China/21Vianet), with an explicit warning
that setting it alone is not sufficient -- the Entra ID auth
endpoint isn't independently configurable in parsedmarc or
mailsuite, so it always hits the global login.microsoftonline.com.
- A minimal working [msgraph] example for every auth method
(UsernamePassword, DeviceCode, ClientSecret, Certificate,
ClientAssertion) -- previously only Certificate had one, entangled
with the SMTP-sending example.
- A reading-permission matrix alongside the existing sending one, so
every auth method x own/shared-mailbox combination is explicit in
one place for both directions.
- An accurate note on the parsedmarc-named token cache: it's a
deliberate backward-compatibility choice from the 9.11.0 mailsuite
extraction (mailsuite's own default cache name differs), not a
migration users need to act on.
- A troubleshooting table for four error scenarios, verified against
source rather than assumed: admin consent and folder-resolution
failures are still live and documented with real fixes; the
event-loop and ISO-timestamp errors are historical, already fixed
below this project's dependency/version floor.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
|
||
|
3fda55d385 |
Make Microsoft Graph connection activity observable (#815)
* Make Microsoft Graph connection activity observable parsedmarc only configured its own logger, so all Graph connection activity was silently dropped even with --debug: the mailbox layer logs under mailsuite.mailbox.graph, token acquisition under azure.identity (including the AADSTS error codes that distinguish a local config problem from an Exchange Online / Entra ID one), and HTTP traffic under httpx/msgraph — none of which had a handler or level set. _main() also logged nothing around the MSGraphConnection call, so a hang left no trace at all. Three changes, all parsedmarc-side (no mailsuite changes needed): - Log a redacted connection summary at INFO before connecting (auth method, tenant ID, client ID, mailbox, Graph URL) plus a --debug detail line with certificate path, token-file path, and set/not-set flags for secrets. Secret values are never logged; a regression test asserts they don't appear in captured output. - Log a timing line after the connection object is initialized. - Propagate parsedmarc's --verbose/--debug level and handlers to the dependency loggers (mailsuite, azure, msgraph, httpx, httpcore) via _configure_dependency_logging(), synced to exactly the parsedmarc logger's handlers so SIGHUP log-file swaps neither duplicate output nor write to closed handlers. At the default level dependency loggers sit at WARNING, so their warnings keep surfacing (formatted) without new noise. All four new tests fail on the unfixed code (verified by stashing the cli.py change). Fixes https://github.com/domainaware/parsedmarc/issues/814. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Disable propagation on dependency loggers; document kiota's absence Set propagate=False on the dependency loggers when syncing handlers, so a stray logging.basicConfig() anywhere in the process cannot double-print every dependency record through the root logger — the function already owns these loggers' handler lists, and this makes that ownership complete. Asserted alongside the existing level/handler checks. kiota_http and its sibling packages were considered for _DEPENDENCY_LOGGERS but verified to not use Python logging at all (their observability is OpenTelemetry tracing), so a comment now records why they are absent rather than leaving the omission to be "fixed" later. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: MISAPOR LAB <misapor@lab.misapor.pl> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com> |
||
|
|
fa8faa6b39 |
MS Graph: case-insensitive auth_method + ClientAssertion support (#809)
* draft fix MS Graph * Add ClientAssertion auth method support for MS Graph in cli.py MSGraphConnection/AuthMethod (via mailsuite) already supported ClientAssertion, but cli.py never parsed a client_assertion config value or passed it through, so it was unusable from the CLI. Wire config_msgraph.client_assertion through _parse_config and the MSGraphConnection call, and document the auth method (including its short-lived-JWT caveat vs. Certificate/ClientSecret for watch mode). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: MISAPOR LAB <misapor@lab.misapor.pl> Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
|
eaeea4f53d |
Make the whole codebase pass pyright cleanly and enforce it in CI (#798)
* Make the whole codebase pass pyright cleanly and enforce it in CI Fix all 102 pyright (1.1.410, standard mode) errors across the library, tests, and maps scripts, then pin and enforce the zero-errors bar: - postgres.py: make the optional psycopg import TYPE_CHECKING-aware so the module is properly typed while keeping the runtime install-hint fallback; import psycopg.types.json explicitly as psycopg_json (the old psycopg_types.json attribute access only worked because psycopg imports the submodule eagerly); have _connect()/_ensure_connected() return the live connection so save methods use a non-Optional local; type the DDL list as list[LiteralString] to match psycopg's execute() overloads. - kafkaclient.py: resolve the kafka-python 2.x/3.x bootstrap-error fallback statically via TYPE_CHECKING (kafka-python 3.0 removed NoBrokersAvailable), which also fixes _BootstrapError's import resolution in tests. - syslog.py: go through getattr/setattr for SysLogHandler.socket (absent from typeshed); type the save_* methods with the report TypedDicts (single or list, matching cli.py call sites — gelf.py gets the same signatures); raise ValueError when retry_attempts < 1 instead of falling through and registering a None handler (bug fix, with a regression test and a CHANGELOG entry). - elastic.py / opensearch.py: human_result params are Optional[str]. - maps scripts: sort_csv declared a return type but never returned (now -> None); seen_sort_field_values was possibly unbound; convert_to_utf8's src_encoding is Optional[str]. - tests: cast sample-report dict helpers to their TypedDicts; mark deliberate wrong-type calls with targeted pyright ignores; add narrowing asserts for Optional results; access the mocked KafkaProducer through a cast helper; match the mailsuite fetch_message base signature (**kwargs); patch the renamed parsedmarc.postgres.psycopg_json in test_postgres's setUpModule. Enforcement: [tool.pyright] in pyproject.toml (include parsedmarc, tests, docs; standard mode), pyright==1.1.410 pinned in the [build] extra (pinned exactly so a new pyright release can't break CI without a code change), and a "Check types" step in the lint CI job — which now also runs ruff format --check and installs the [postgresql] extra so the optional psycopg import resolves. Documented in AGENTS.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Set session headers via update() instead of replacing the dict requests 2.34 ships inline type annotations, and Session.headers is a CaseInsensitiveDict[str] — assigning a plain dict fails pyright there (the CI runner resolved 2.34.2; the local venv's untyped 2.32.4 hid it). headers.update() is correctly typed against both versions, and is the documented requests idiom: it overrides User-Agent and the client-specific headers while keeping the session's defaults (Accept-Encoding, Connection) instead of wiping them. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
d3510da3a6 |
feat: graceful SIGTERM/SIGINT shutdown for watch mode and one-shot CLI (#794)
* feat: graceful SIGTERM/SIGINT shutdown for watch mode and one-shot CLI Previously SIGTERM (systemctl stop, docker stop, Kubernetes pod termination) killed parsedmarc mid-batch, tearing output writes and silently dropping buffered Kafka records. Shutdown is now cooperative: - SIGTERM/SIGINT set a flag that is polled at safe boundaries. The one-shot CLI checks it between batches; watch mode passes it as `config_reloading` so the mailbox backend -- including the IMAP IDLE loop -- returns once the current batch is fully processed. Either way the in-flight batch and its output writes finish before the process exits 0. - Ctrl-C is a double-tap: the first press is graceful, the second short-circuits to os._exit(130). - Output clients are now closed on every exit path (atexit plus a trailing close in _main), fixing a long-standing leak where one-shot runs and graceful shutdowns never flushed Kafka / closed Elasticsearch / S3 / etc. Docs: the example systemd unit gains KillSignal=SIGTERM and TimeoutStopSec=60 (keep it above mailbox_check_timeout). Tests cover watch shutdown, the one-shot between-batch stop, the SIGINT double-tap, and the output-client-close leak. * test: cover the one-shot mbox-loop shutdown break Extend the one-shot SIGTERM test to also pass an .mbox path so a single run exercises both shutdown checkpoints: the file-batch loop break and the subsequent mbox loop break (which Codecov flagged as the only uncovered lines on PR #794). is_mbox is keyed by suffix and get_dmarc_reports_from_mbox is asserted not called, since the mbox loop breaks before reaching it. * test: narrow signal.getsignal() return before invoking in SIGINT test signal.getsignal() is typed Callable | int | Handlers | None; calling it directly fails pyright's callable check. Assert callable() first. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
|
180fc581fe |
fix: OSD Global-tenant import + dropped report files with glob metacharacters; validate dev stack on OpenSearch 3.x with PostgreSQL (#781)
* fix: import OpenSearch dashboards into the real Global tenant dashboard-dev-bootstrap.sh sent `securitytenant: global_tenant`. The OpenSearch security plugin reads that header as a tenant *name*, and `global_tenant` is a sample custom tenant from the security demo config -- not the shared Global tenant, whose token is the literal `global`. The import therefore landed in a separate `global_tenant` tenant (its own `.kibana_<hash>_globaltenant_1` index) and the dashboards were invisible to anyone viewing the Global tenant in OpenSearch Dashboards. Verified against the live dev cluster: `_find` under `securitytenant: global` returned 26 objects and `.kibana_1` (the Global tenant index the UI reads) went from 2 to 67 docs after re-importing with the fix. An empty/omitted header read 0 from Global -- it falls back to the user's configured default tenant -- so `global` is the only reliable token. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: don't drop report files whose names contain glob metacharacters The CLI expanded every file argument with glob(), which treats [, ], *, and ? as pattern syntax. A literal path like "[Netease DMARC Failure Report] Rent Reminder.eml" -- the bracketed shape many providers use for emailed failure reports -- was read as a character class, matched nothing, and was dropped before reaching the parser, with no error. File arguments that exist on disk are now taken literally; only non-existent paths are globbed, so shell-style wildcards still expand. Also adds "postgresql" to _KNOWN_SECTIONS so PARSEDMARC_POSTGRESQL_* env vars (and their _FILE Docker-secret variants) resolve like every other backend -- the PostgreSQL backend is new in 10.0.0, so this completes the unreleased feature rather than fixing a released regression, and is documented under the PostgreSQL enhancement, not Bug fixes. Regression tests added for both. Verified end-to-end: all four samples/failure/*.eml now index (the bracketed Netease report included). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * dev: validate dashboards on OpenSearch 3.x and add PostgreSQL to the dev stack The dev stack ran OpenSearch Dashboards 3.x against OpenSearch 2.x, an unsupported cross-major pairing. Bump opensearch to :3 (validated on 3.6.0: OSD import into the Global tenant and all dashboards work). Add a postgresql service plus bootstrap wiring so the new PostgreSQL backend is exercised alongside the others: wait for PG, seed it via PARSEDMARC_POSTGRESQL_* env vars on the same parsedmarc run, wipe it on RESEED, create a Grafana grafana-postgresql-datasource (uid dmarc-pg), and import dashboards/grafana/Grafana-DMARC_Reports-PostgreSQL.json. PG seeding is gated on psycopg being importable: parsedmarc aborts the whole run (exit 1, nothing written to any backend) when a configured output backend can't initialize, so wiring in PG without the optional extra would silently zero ES/OS/Splunk too. When psycopg is absent the script warns and skips PG, leaving the other backends seeded. Also fix the Grafana admin password env: the container was given GRAFANA_PASSWORD, which Grafana ignores -- it reads GF_SECURITY_ADMIN_PASSWORD. Defaults to admin to match the script. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * docs: list PostgreSQL on the premade-dashboards features bullet PostgreSQL ships a premade Grafana dashboard (dashboards/grafana/Grafana-DMARC_Reports-PostgreSQL.json), so it belongs on the "for use with premade dashboards" bullet alongside Elasticsearch, OpenSearch, and Splunk rather than on the plain-output-destinations line. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fix: clear stale org_email mapping conflict in the OpenSearch dashboards The aggregate index pattern in dashboards/opensearch/opensearch_dashboards.ndjson shipped a cached field-list snapshot where org_email was a text/object conflict, plus leftover org_email.#text and org_email.#text.keyword subfields. Those came from a cluster that had indexed a langAttrString email dict ({"#text": ..., "@lang": ...}) before the parser unwrapped it. org_email is mapped as Text() and parse_aggregate_report_xml now unwraps a dict email to a plain string, so current data is consistently text -- a clean cluster's _field_caps reports no conflict. Cleared the frozen conflict and the two artifact subfields, leaving org_email (text) and org_email.keyword, matching the live mapping. Verified: re-importing the corrected ndjson yields an index pattern with org_email as a plain text field and zero conflicts; only the aggregate index-pattern line changed, all other saved objects byte-identical. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * dev: seed the RFC 9990 (dmarc-2.0) aggregate samples samples/aggregate/rfc9990-sample.xml and rfc9990-example.net!...xml were not in the bootstrap's SAMPLE_FILES, so the dev stack only ever indexed RFC 7489 reports and the new DMARCbis fields (np, testing, discovery_method, generator, xml_namespace) never appeared in the OpenSearch/Kibana indices or were available to the dashboards. Added both samples (one declares the urn:ietf:params:xml:ns:dmarc-2.0 namespace, the other is namespaceless RFC 9990-shaped, covering both detection paths). Verified the seeded data now carries np/testing/ discovery_method/generator and xml_namespace=urn:ietf:params:xml:ns:dmarc-2.0; OpenSearch Dashboards surfaces them on an index-pattern field-list refresh. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * dev: auto-resolve (or create) a venv for the seed and ensure psycopg The seed previously required parsedmarc to be pre-installed and only warned-and-skipped PostgreSQL when psycopg was missing. Resolve the seed environment by precedence instead: 1. explicit PARSEDMARC_BIN -> used as-is, nothing installed 2. active $VIRTUAL_ENV 3. existing repo venv/ or .venv/ 4. otherwise create $REPO_ROOT/venv For cases 2-4, run `pip install -e .[postgresql]` only when the CLI or psycopg is missing, so the dev stack can populate Postgres out of the box without a manual install step. The explicit-PARSEDMARC_BIN path is left untouched (and the psycopg seed guard still warns/skips if that env lacks the extra). Verified: a RESEED run resolves the active venv, seeds ES/OS/Splunk/PG including the RFC 9990 fields, with no output-client errors. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
327fcff2b9 |
Add optional PostgreSQL storage backend (#667)
Adds a PostgreSQL output backend as a lighter-weight alternative to Elasticsearch/OpenSearch, configured via a [postgresql] section (host/port/user/password/database or a libpq connection_string). Tables are created automatically on first run; a Grafana dashboard is included. - psycopg is an optional extra (pip install parsedmarc[postgresql]); the import is guarded so `import parsedmarc` works without it, and PostgreSQLClient raises a clear install hint when constructed without the driver. Binary wheels aren't available for every platform. - Schema captures the RFC 9990 / DMARCbis aggregate fields: np, testing, discovery_method, generator, xml_namespace, and per-result human_result on the DKIM/SPF auth-result tables. - forensic -> failure naming throughout (table dmarc_failure_report, save_failure_report_to_postgresql, dashboard, docs) to match #659. - Failure-report de-duplication mirrors the Elasticsearch backend exactly: arrival date + From + To + Subject (NULL-safe via IS NOT DISTINCT FROM; semantic JSONB equality). Aggregate and SMTP-TLS use ON CONFLICT. - PostgreSQLClient.close() for clean CLI shutdown; comment documents why the two timestamp helpers must stay distinct (report dates are local, record/SMTP-TLS dates are UTC). - CLI: config parse raises ConfigurationError on missing host/connection_string; wired into _init_output_clients + save loops. - Tests in tests/test_postgres.py (helpers, mocked-DB save assertions, create_tables, connect/error wrapping, dedup, real-sample round trip) and tests/test_cli.py (config parse + end-to-end save wiring incl. AlreadySaved/PostgreSQLError handling). postgres.py at 99% line coverage; only _main's output-client-init retry path is left. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
 DVBandGitHub
|
bf37ded688 | Add support for Elastic Cloud Serverless projects (#770) | ||
 VincentandGitHub
|
535d9db1ad |
cli: support _FILE suffix on PARSEDMARC_* env vars for Docker secrets (#772)
Appending _FILE to any PARSEDMARC_{SECTION}_{KEY} env var reads the
value from the referenced file, with one trailing newline stripped.
This matches the Postgres/MariaDB/Redis container-image convention so
Docker Compose and Kubernetes secret mounts work without extra glue,
keeping credentials out of plain environment: blocks (and out of
docker inspect, container logs, and /proc/<pid>/environ).
When both the direct var and its _FILE companion are set, the file
wins. A missing or unreadable file raises ConfigurationError rather
than silently degrading to an empty credential. The four pre-existing
config keys whose own names end in _file ([general] log_file,
[msgraph] token_file, [gmail_api] credentials_file / token_file)
keep their direct-path semantics; pass their values via secret by
doubling the suffix (_FILE_FILE).
|
||
|
|
b7b8383fa4 |
Expand honest test coverage from 59% to 83%; fix two latent bugs (#775)
* Expand honest test coverage from 59% to 83%; fix two latent bugs 271 new tests across the output modules, ES/OS clients, CLI config parsing, and the top-level parsing surface. Coverage measured against shipped code only (see [tool.coverage.run] source = ["parsedmarc"] omit = ["*/parsedmarc/resources/maps/*.py"] in pyproject.toml). Per-module results: s3.py 38% → 100% (also fixes SMTP-TLS-to-S3 bug below) gelf.py 40% → 100% syslog.py 46% → 100% kafkaclient.py 34% → 100% splunk.py 24% → 100% loganalytics.py 56% → 100% webhook.py 78% → 100% (also removes redundant try/except) elastic.py 36% → 99% opensearch.py 40% → 99% cli.py 52% → 69% __init__.py 74% → 76% (also fixes append_json bug below) utils.py 84% (unchanged in this PR) TOTAL 59% → 83% The remaining 17% is honest. The biggest unreached blocks are _main() in cli.py and the watch-mode mailbox iteration in __init__.py, both of which would require either standing up live subsystems (real Elasticsearch, real IMAP) or mocking deep enough that the test would verify the mock rather than the code. The PR-A AGENTS.md guidance — "if 90% requires faking it, ship 85% honestly" — applies here. Bugs fixed while writing tests: 1. parsedmarc/s3.py — SMTP-TLS-to-S3 was completely broken. save_report_to_s3 unconditionally read report["report_metadata"] when building S3 object metadata, but RFC 8460 §4.3 SMTP TLS reports are flat (no report_metadata sub-object). The CLI's surrounding try/except silently swallowed the KeyError, so every SMTP-TLS report quietly failed to upload. Also fixes a related issue: parse_smtp_tls_report_json stores begin_date as the raw ISO-8601 string from the report (per the SMTPTLSReport TypedDict and RFC 8460 §4.3), but the S3 code path assumed a datetime with .year / .month / .day attributes. Both fixed; the broken metadata-extraction branch now uses the flat-report fields, and the date branch normalizes via human_timestamp_to_datetime. 2. parsedmarc/__init__.py — append_json corrupted JSON output files on the second write. The original implementation opened files in "a+" mode, then seek()ed backwards to overwrite the trailing "]" with ",\n" before appending more elements. Python's docs are explicit (https://docs.python.org/3/library/functions.html#open): on POSIX, writes in "a"/"a+" mode always go to EOF regardless of seek() position. The result was that the second call produced [...]\n],\n[...] -style corrupted output instead of a single merged array. Replaced with a read-merge-write pattern: load the existing array (if any), append the new elements, rewrite the whole file. The CSV cousin append_csv was not affected — it doesn't seek backwards. 3. parsedmarc/webhook.py — removed redundant try/except blocks in save_aggregate_report_to_webhook / save_failure_report_to_webhook / save_smtp_tls_report_to_webhook. _send_to_webhook already catches every Exception itself, so the outer except blocks were unreachable dead code (covered nothing, defended against nothing, and inflated the source-line count without testing value). Testing approach: mocks at SDK boundaries (boto3 resource, kafka producer, requests session, opensearch/elasticsearch Document/Search, azure LogsIngestionClient). Tests verify the parsedmarc-side transformation logic — document/event construction, index/topic naming, dedup queries, error wrapping — rather than asserting on mock invocations as a proxy for behaviour. Where a branch is defensive against a caller that doesn't exist in the codebase, the test is omitted (commented in code rather than hidden behind a pragma). 547 tests total (was 276), all passing. ruff check + format clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Document the two bug fixes from this PR in the 10.0.0 changelog Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Document testing standards in AGENTS.md Adds a "Testing standards" section covering the principles applied in PR-A (split) and PR-B (coverage expansion): - Coverage measures shipped code only — don't reintroduce tests/* to the scope, don't expand omit, don't use # pragma: no cover. - Honest tests assert on observable behaviour, not "the mock was called". Mock at SDK boundaries; parse the payload that gets sent. - "If 90% requires faking it, ship 85% honestly" — coverage is a tool, not a goal. PR-B's deliberate stops at cli.py 69% and __init__.py 76% are the documented precedent for when to halt. - Verify bug claims against the relevant RFC, internal types, installed SDK source, or upstream docs before changing code. Cite the source in the commit message and test docstring (RFC 8460 §4.3 and the Python open() docs for #775's two bug fixes are the pattern to follow). - Bugs found while writing tests are fixed in the same PR; the test doubles as the regression guard. - File layout (tests/test_<module>.py) is non-negotiable; module-level test loggers need fresh-handler setup so test ordering doesn't break assertLogs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Cover the corrupt-file fallback in append_json Codecov flagged 2 missing patch-coverage lines on PR #775: the except (json.JSONDecodeError, OSError) branch in append_json, which falls back to overwriting when the existing file isn't a parseable JSON array. Two new tests in tests/test_init.py:TestAppendJson exercise both paths: - test_corrupt_existing_file_is_overwritten_cleanly: existing file contains invalid JSON; append_json overwrites with the new array. - test_existing_file_with_non_list_root_is_overwritten: existing file parses as {"foo": ...} (dict, not list); the isinstance guard rejects it and we overwrite cleanly. Patch coverage now 100% on the bug fix. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
|
5b08627eaa |
Split tests.py into per-module tests/test_<module>.py (#774)
* Split tests.py into per-module tests/test_<module>.py The 5174-line tests.py monolith is split into per-module files under tests/, mirroring the checkdmarc layout: tests/test_init.py parsedmarc/__init__.py parsing surface tests/test_cli.py parsedmarc/cli.py + config / env-vars / SIGHUP tests/test_utils.py parsedmarc/utils.py (DNS, IP info, PSL, etc.) tests/test_webhook.py parsedmarc/webhook.py tests/test_kafkaclient.py parsedmarc/kafkaclient.py tests/test_splunk.py parsedmarc/splunk.py tests/test_syslog.py parsedmarc/syslog.py tests/test_loganalytics.py parsedmarc/loganalytics.py tests/test_gelf.py parsedmarc/gelf.py tests/test_s3.py parsedmarc/s3.py tests/test_maps.py parsedmarc/resources/maps/ maintainer scripts The split is purely a redistribution — no test bodies changed, no tests added or removed. All 276 existing tests pass under the new layout. The current tests.py contains two kitchen-sink classes (`Test` at line 54 and `TestEnvVarConfig` at line 2360) holding tests that span many modules. Their methods are routed to the correct per-module file by name prefix; the wholly-thematic classes (TestExtractReport, TestUtilsXxx, TestSighupReload, etc.) move whole. Each target file gets its own `class Test(unittest.TestCase)` for the redistributed kitchen-sink methods, plus the thematic classes verbatim. Wiring updates: - `.github/workflows/python-tests.yml`: `pytest ... tests.py` → `python -m pytest ... tests/` (also switches to `python -m pytest` per the checkdmarc convention so cwd lands on the project root). - `pyproject.toml`: adds `[tool.pytest.ini_options] testpaths = ["tests"]` and `[tool.coverage.run] source = ["parsedmarc"]` with an `omit` for `parsedmarc/resources/maps/*.py`. The maps scripts are maintainer-only batch tooling that ships out of the wheel; excluding them from coverage makes the headline number reflect only installed library code. Runtime coverage on the new layout is 59% (was 45% with maps counted), and PR-B will push it to 90%+. - `AGENTS.md`: documents the new layout and how to run individual files / tests; tells future contributors not to reintroduce a monolithic tests.py. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Restore 66.9% coverage baseline (count tests/ + parsedmarc) Master's headline 66.9% number on Codecov includes the tests.py file itself (99.35% covered) being measured alongside parsedmarc/*. The original tests.py had no `[tool.coverage.run]` block, so coverage's default — "measure every file imported during the run" — counted the test code as if it were product code. The split commit added `source = ["parsedmarc"]` which suppressed measurement of the test files (correct in principle, since test files aren't shipped code), and that alone made the headline number drop by ~8 percentage points without any actual loss of testing. This commit swaps `source` for an explicit `include = ["parsedmarc/*", "tests/*"]` so both halves are measured the way they were on master. Verified: 276 tests, 66.96% line coverage (effectively unchanged from master's 66.90%). If you want the shipped-code-only number (was the headline that this commit overrides), run `pytest --cov=parsedmarc tests/`. That number is currently 59% and is the focus of the upcoming coverage-expansion PR. Also adds junit.xml to .gitignore so the CI artefact doesn't get accidentally committed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * Restrict coverage to shipped code (`source = ["parsedmarc"]`) Reverts the prior commit's `include = ["tests/*"]`. Counting the test files toward coverage was wrong — it conflates "shipped code exercised by tests" with "test code that pytest auto-runs", inflates the headline number, and rewards writing more tests rather than tests that verify more code. Master's apparent 66.9% was an artefact of the old monolithic tests.py having no [tool.coverage.run] block at all; coverage's default behaviour measured every imported file, including the test file itself at ~99% "covered", which added ~8 percentage points to the displayed number without any real testing signal. Restricting to `source = ["parsedmarc"]` plus the existing maps omit gives a meaningful baseline: 59% of shipped code is exercised by the test suite today. That's the number the next PR is targeting to lift to 90%+ before the 10.0.0 release; the Codecov "drop" here is a measurement correction, not a regression. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |