mirror of
https://github.com/domainaware/parsedmarc.git
synced 2026-09-05 13:38:00 +00:00
* Make the output and mailbox integrations optional extras (#883) Breaking change for the next major release: pip install parsedmarc now installs the parsing core plus a working core CLI (file, IMAP, Maildir, and mbox input; CSV/JSON, Splunk HEC, webhook, and syslog output). Everything else moves behind an extra: elastic, opensearch, kafka, s3, gelf, loganalytics, msgraph, and gmail, joining the existing postgresql extra, with an umbrella [all] that deliberately excludes postgresql (psycopg's binary wheels do not exist on every platform, so parsedmarc[all] must never fail to install there). cli.py imports the six SDK-dependent output modules behind the #884 TYPE_CHECKING/try-except guard; a configured section whose extra is missing fails fast with a ConfigurationError naming the section and the exact pip install command — including the msgraph and gmail_api mailbox sections (detected via parsedmarc.mail's placeholder classes) and postgresql (checked before the constructor so the startup retry loop does not retry a missing dependency for a minute). The Azure/kiota Graph error types fall back to never-raised sentinel classes. The Docker image installs [all,postgresql], so container users see no change. CI lint installs [build,all,postgresql]; the unit-test job installs [build,all], deliberately without postgresql so test_postgres.py's absent-psycopg arm stays exercised. The never-imported dateparser dependency is dropped in favor of declaring python-dateutil, which utils.py actually imports; pytz moves to the build extra for the one test that uses it. Verified live: a no-extras wheel install imports, parses samples, and reports the install hint for each gated section; a [all] install restores every integration; the Docker image builds with every SDK importable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Patch psycopg presence in the PostgreSQL CLI wiring tests CI's unit-test job deliberately installs [build,all] without the postgresql extra, so parsedmarc.cli.postgres.psycopg is None there and the new missing-extra presence check correctly made _main exit 1 before the wiring under test ran. The tests simulate the SDK being available (PostgreSQLClient is mocked at the SDK boundary), so the module-level psycopg handle is now patched present in setUp. Verified against a simulated psycopg-absent environment as well as the local full install. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address Copilot review: narrow guards to ModuleNotFoundError, fix docs - The optional-integration and Graph error-type import guards now catch ModuleNotFoundError instead of ImportError, so only a genuinely absent package reads as a missing extra; a broken-but-present SDK fails loudly with its real error instead of masquerading as one. The test blocker raises ModuleNotFoundError accordingly — the exact exception a missing package produces. - _missing_extra_hint docstring no longer calls every gated integration an output module (it also serves the msgraph/gmail_api mailbox sections). - Fix the pre-existing passsword typo in usage.md's kafka section; the INI key the code reads is password (cli.py _parse_config). Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Quote extras specs in copy-paste install commands From Copilot's second review round: zsh treats an unquoted .[build,all] as a glob and fails with 'no matches found', so the commands shown in AGENTS.md, CONTRIBUTING.md, dashboards/README.md, and the bootstrap script's comment are now quoted. The CI workflows keep the unquoted form: they run under bash, which passes unmatched globs through literally. The suggestion to change the 'Choosing what to install' heading level was rejected — it is a subsection of 'Installing parsedmarc', matching the file's existing hierarchy. Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix upgrade command in the changelog * Documentation review: accuracy, spelling, grammar, and clarity pass A full prose review of docs/source, README, CONTRIBUTING, and the dashboards README, with every accuracy claim verified against the code before changing it. Highlights: - usage.md: documented six missing [general] options (the CSV/JSON filename options, prettify_json, normalize_timespan_threshold_hours), the required kafka smtp_tls_topic, [imap] timeout/max_retries, and the postgresql env-var prefix; corrected the maildir_path default (None, not INBOX — cli.py Namespace defaults), the mailbox check_timeout option name, the systemd restart interval (RestartSec is 5m), and merged the duplicate silent entry; quoted every copy-paste extras spec for zsh safety. - elasticsearch.md: fixed an invalid openssl command (rsa:4096 -nodes), the dashboards filename (opensearch_dashboards.ndjson, matching the file the link serves), and assorted grammar. - davmail.md: the service-enable command now enables davmail.service (was parsedmarc.service — a copy-paste error that left DavMail unenabled), plus a view typo and DavMail capitalization. - output.md: the example schema reference is RFC 7489 Appendix C (7480 is RDAP). kibana.md: SPF relies on the SMTP envelope, not session headers (RFC 7208). dmarc.md: DKM -> DKIM. - README: the intro now also names the OpenSearch/Grafana stack, matching the feature list. CONTRIBUTING: pre-PR checks now include ruff format --check and pyright, matching CI's lint job. - dashboards/README: the service table and seed description now include the PostgreSQL backend the compose stack runs. Sample data blocks, the CLI-help mirror block, and released CHANGELOG entries were deliberately left untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Docstring review: accuracy, spelling, grammar, and clarity pass Every docstring in parsedmarc/, parsedmarc/mail/, the maps maintainer scripts, and the test suite reviewed with each claim verified against the code it documents. Text-only — no behavior changes. Highlights: - Copy-paste errors corrected: parsed_smtp_tls_reports_to_csv and splunk/loganalytics save functions described aggregate or failure reports they do not handle; LogAnalyticsException claimed to be an Elasticsearch error. - Docstring/behavior mismatches: parse_report_email's report_type enumeration omitted smtp_tls; parse_failure_report typed msg_date as str (it is datetime); strip_attachment_payloads claimed payloads are replaced with None (the key is deleted); kafkaclient's failure and SMTP TLS savers claimed per-record slicing while sending the whole list in one message (docstrings now describe reality — whether slicing was intended is flagged for follow-up); the postgres savers claimed to take parse_report_file's return value but receive the inner report dict; elastic/opensearch save functions' Raises listed only AlreadySaved. - None-as-semantic-state documented where missing (get_base_domain, get_ip_address_country), enumeration completeness fixed (get_ip_address_info's 9 result keys, maps script outputs, TSV columns), and the stale 44-industry-types count corrected to the 46 the authoritative README list defines. - Test docstrings aligned with what the tests actually assert, including two that overstated coverage of the elastic/opensearch address-list tests. - Two argparse help strings fixed: file_path now names SMTP TLS report files alongside aggregate and failure, mirrored into usage.md's CLI-help block; --offline's doubled spaces removed (rendered help unchanged). - elasticsearch.md's security claim corrected against Elastic's docs: security is enabled and auto-configured on first startup since 8.0 (not "8.7 secure mode"), so the settings are verified, not hand-written. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
278 lines
10 KiB
Python
278 lines
10 KiB
Python
"""Tests for parsedmarc.loganalytics"""
|
|
|
|
import unittest
|
|
from unittest.mock import MagicMock, patch
|
|
|
|
from azure.core.exceptions import HttpResponseError
|
|
|
|
from parsedmarc.loganalytics import (
|
|
LogAnalyticsClient,
|
|
LogAnalyticsConfig,
|
|
LogAnalyticsException,
|
|
)
|
|
|
|
|
|
def _valid_kwargs(**overrides):
|
|
base = dict(
|
|
client_id="cid",
|
|
client_secret="csec",
|
|
tenant_id="tid",
|
|
dce="https://dce.example.com",
|
|
dcr_immutable_id="dcr-123",
|
|
dcr_aggregate_stream="agg-stream",
|
|
dcr_failure_stream="fail-stream",
|
|
dcr_smtp_tls_stream="tls-stream",
|
|
)
|
|
base.update(overrides)
|
|
return base
|
|
|
|
|
|
class TestLogAnalyticsConfig(unittest.TestCase):
|
|
"""The config dataclass holds every credential and stream needed
|
|
to push to Log Analytics. A typo on any attribute would silently
|
|
drop data into the wrong stream."""
|
|
|
|
def test_config_stores_every_field(self):
|
|
config = LogAnalyticsConfig(**_valid_kwargs())
|
|
self.assertEqual(config.client_id, "cid")
|
|
self.assertEqual(config.client_secret, "csec")
|
|
self.assertEqual(config.tenant_id, "tid")
|
|
self.assertEqual(config.dce, "https://dce.example.com")
|
|
self.assertEqual(config.dcr_immutable_id, "dcr-123")
|
|
self.assertEqual(config.dcr_aggregate_stream, "agg-stream")
|
|
self.assertEqual(config.dcr_failure_stream, "fail-stream")
|
|
self.assertEqual(config.dcr_smtp_tls_stream, "tls-stream")
|
|
|
|
|
|
class TestLogAnalyticsClientInit(unittest.TestCase):
|
|
"""The constructor's validation guards against a half-configured
|
|
deployment that would otherwise fail late inside Azure SDK calls
|
|
with confusing errors."""
|
|
|
|
def test_init_accepts_complete_config(self):
|
|
client = LogAnalyticsClient(**_valid_kwargs())
|
|
self.assertEqual(client.conf.client_id, "cid")
|
|
self.assertEqual(client.conf.dcr_immutable_id, "dcr-123")
|
|
|
|
def test_missing_client_id_raises(self):
|
|
with self.assertRaises(LogAnalyticsException):
|
|
LogAnalyticsClient(**_valid_kwargs(client_id=""))
|
|
|
|
def test_missing_client_secret_raises(self):
|
|
with self.assertRaises(LogAnalyticsException):
|
|
LogAnalyticsClient(**_valid_kwargs(client_secret=""))
|
|
|
|
def test_missing_tenant_id_raises(self):
|
|
with self.assertRaises(LogAnalyticsException):
|
|
LogAnalyticsClient(**_valid_kwargs(tenant_id=""))
|
|
|
|
def test_missing_dce_raises(self):
|
|
with self.assertRaises(LogAnalyticsException):
|
|
LogAnalyticsClient(**_valid_kwargs(dce=""))
|
|
|
|
def test_missing_dcr_immutable_id_raises(self):
|
|
with self.assertRaises(LogAnalyticsException):
|
|
LogAnalyticsClient(**_valid_kwargs(dcr_immutable_id=""))
|
|
|
|
|
|
class TestPublishJson(unittest.TestCase):
|
|
"""publish_json wraps logs_client.upload and translates Azure
|
|
HttpResponseError into the module's own exception type so the CLI
|
|
error reporter can handle it uniformly."""
|
|
|
|
def test_publish_json_forwards_to_logs_client(self):
|
|
client = LogAnalyticsClient(**_valid_kwargs())
|
|
logs_client = MagicMock()
|
|
client.publish_json([{"a": 1}], logs_client, "agg-stream")
|
|
logs_client.upload.assert_called_once_with("dcr-123", "agg-stream", [{"a": 1}])
|
|
|
|
def test_publish_json_translates_http_error(self):
|
|
client = LogAnalyticsClient(**_valid_kwargs())
|
|
logs_client = MagicMock()
|
|
logs_client.upload.side_effect = HttpResponseError("forbidden")
|
|
with self.assertRaises(LogAnalyticsException) as ctx:
|
|
client.publish_json([{"a": 1}], logs_client, "stream")
|
|
self.assertIn("forbidden", str(ctx.exception))
|
|
|
|
|
|
class TestPublishResults(unittest.TestCase):
|
|
"""publish_results gates each report type behind both a config flag
|
|
(save_aggregate / save_failure / save_smtp_tls) and a configured
|
|
stream name. Both gates need to work — a missing stream is skipped
|
|
silently (a partially configured client is normal), while an
|
|
explicit save_*=False means the operator opted out."""
|
|
|
|
def _publish_with(self, results, **flags):
|
|
flags.setdefault("save_aggregate", True)
|
|
flags.setdefault("save_failure", True)
|
|
flags.setdefault("save_smtp_tls", True)
|
|
client = LogAnalyticsClient(**_valid_kwargs())
|
|
with (
|
|
patch("parsedmarc.loganalytics.ClientSecretCredential"),
|
|
patch("parsedmarc.loganalytics.LogsIngestionClient") as mock_client_cls,
|
|
):
|
|
mock_logs_client = mock_client_cls.return_value
|
|
client.publish_results(results, **flags)
|
|
return mock_logs_client
|
|
|
|
def test_aggregate_published_to_aggregate_stream(self):
|
|
logs_client = self._publish_with(
|
|
{
|
|
"aggregate_reports": [{"id": "a"}],
|
|
"failure_reports": [],
|
|
"smtp_tls_reports": [],
|
|
}
|
|
)
|
|
logs_client.upload.assert_called_once_with(
|
|
"dcr-123", "agg-stream", [{"id": "a"}]
|
|
)
|
|
|
|
def test_failure_published_to_failure_stream(self):
|
|
logs_client = self._publish_with(
|
|
{
|
|
"aggregate_reports": [],
|
|
"failure_reports": [{"id": "f"}],
|
|
"smtp_tls_reports": [],
|
|
}
|
|
)
|
|
logs_client.upload.assert_called_once_with(
|
|
"dcr-123", "fail-stream", [{"id": "f"}]
|
|
)
|
|
|
|
def test_smtp_tls_published_to_smtp_tls_stream(self):
|
|
logs_client = self._publish_with(
|
|
{
|
|
"aggregate_reports": [],
|
|
"failure_reports": [],
|
|
"smtp_tls_reports": [{"id": "t"}],
|
|
}
|
|
)
|
|
logs_client.upload.assert_called_once_with(
|
|
"dcr-123", "tls-stream", [{"id": "t"}]
|
|
)
|
|
|
|
def test_all_three_published_together(self):
|
|
logs_client = self._publish_with(
|
|
{
|
|
"aggregate_reports": [{"id": "a"}],
|
|
"failure_reports": [{"id": "f"}],
|
|
"smtp_tls_reports": [{"id": "t"}],
|
|
}
|
|
)
|
|
self.assertEqual(logs_client.upload.call_count, 3)
|
|
streams_uploaded = {call.args[1] for call in logs_client.upload.call_args_list}
|
|
self.assertEqual(streams_uploaded, {"agg-stream", "fail-stream", "tls-stream"})
|
|
|
|
def test_save_aggregate_false_skips_aggregate(self):
|
|
logs_client = self._publish_with(
|
|
{
|
|
"aggregate_reports": [{"id": "a"}],
|
|
"failure_reports": [],
|
|
"smtp_tls_reports": [],
|
|
},
|
|
save_aggregate=False,
|
|
)
|
|
logs_client.upload.assert_not_called()
|
|
|
|
def test_save_failure_false_skips_failure(self):
|
|
logs_client = self._publish_with(
|
|
{
|
|
"aggregate_reports": [],
|
|
"failure_reports": [{"id": "f"}],
|
|
"smtp_tls_reports": [],
|
|
},
|
|
save_failure=False,
|
|
)
|
|
logs_client.upload.assert_not_called()
|
|
|
|
def test_save_smtp_tls_false_skips_smtp_tls(self):
|
|
logs_client = self._publish_with(
|
|
{
|
|
"aggregate_reports": [],
|
|
"failure_reports": [],
|
|
"smtp_tls_reports": [{"id": "t"}],
|
|
},
|
|
save_smtp_tls=False,
|
|
)
|
|
logs_client.upload.assert_not_called()
|
|
|
|
def test_empty_results_publishes_nothing(self):
|
|
logs_client = self._publish_with(
|
|
{
|
|
"aggregate_reports": [],
|
|
"failure_reports": [],
|
|
"smtp_tls_reports": [],
|
|
}
|
|
)
|
|
logs_client.upload.assert_not_called()
|
|
|
|
def test_missing_aggregate_stream_skips_aggregate(self):
|
|
"""If the operator hasn't configured a stream for one of the
|
|
report types, the corresponding publish branch is skipped
|
|
silently — matching the existing CLI deployment pattern where
|
|
a single client object handles whatever streams are set."""
|
|
client = LogAnalyticsClient(**_valid_kwargs(dcr_aggregate_stream=""))
|
|
with (
|
|
patch("parsedmarc.loganalytics.ClientSecretCredential"),
|
|
patch("parsedmarc.loganalytics.LogsIngestionClient") as mock_client_cls,
|
|
):
|
|
mock_logs_client = mock_client_cls.return_value
|
|
client.publish_results(
|
|
{
|
|
"aggregate_reports": [{"id": "a"}],
|
|
"failure_reports": [],
|
|
"smtp_tls_reports": [],
|
|
},
|
|
save_aggregate=True,
|
|
save_failure=True,
|
|
save_smtp_tls=True,
|
|
)
|
|
mock_logs_client.upload.assert_not_called()
|
|
|
|
def test_credential_built_from_config(self):
|
|
"""ClientSecretCredential is constructed with the conf's three
|
|
identity fields — a rename or order shuffle would auth as the
|
|
wrong principal."""
|
|
client = LogAnalyticsClient(**_valid_kwargs())
|
|
with (
|
|
patch("parsedmarc.loganalytics.ClientSecretCredential") as mock_cred,
|
|
patch("parsedmarc.loganalytics.LogsIngestionClient"),
|
|
):
|
|
client.publish_results(
|
|
{
|
|
"aggregate_reports": [],
|
|
"failure_reports": [],
|
|
"smtp_tls_reports": [],
|
|
},
|
|
save_aggregate=True,
|
|
save_failure=True,
|
|
save_smtp_tls=True,
|
|
)
|
|
mock_cred.assert_called_once_with(
|
|
tenant_id="tid", client_id="cid", client_secret="csec"
|
|
)
|
|
|
|
def test_logs_ingestion_client_built_from_dce_and_credential(self):
|
|
client = LogAnalyticsClient(**_valid_kwargs())
|
|
with (
|
|
patch("parsedmarc.loganalytics.ClientSecretCredential") as mock_cred,
|
|
patch("parsedmarc.loganalytics.LogsIngestionClient") as mock_client_cls,
|
|
):
|
|
client.publish_results(
|
|
{
|
|
"aggregate_reports": [],
|
|
"failure_reports": [],
|
|
"smtp_tls_reports": [],
|
|
},
|
|
save_aggregate=True,
|
|
save_failure=True,
|
|
save_smtp_tls=True,
|
|
)
|
|
mock_client_cls.assert_called_once_with(
|
|
"https://dce.example.com", credential=mock_cred.return_value
|
|
)
|
|
|
|
|
|
if __name__ == "__main__":
|
|
unittest.main(verbosity=2)
|