Expand honest test coverage from 59% to 83%; fix two latent bugs (#775)

* Expand honest test coverage from 59% to 83%; fix two latent bugs

271 new tests across the output modules, ES/OS clients, CLI config
parsing, and the top-level parsing surface. Coverage measured against
shipped code only (see [tool.coverage.run] source = ["parsedmarc"]
omit = ["*/parsedmarc/resources/maps/*.py"] in pyproject.toml).

Per-module results:

  s3.py             38% → 100%   (also fixes SMTP-TLS-to-S3 bug below)
  gelf.py           40% → 100%
  syslog.py         46% → 100%
  kafkaclient.py    34% → 100%
  splunk.py         24% → 100%
  loganalytics.py   56% → 100%
  webhook.py        78% → 100%   (also removes redundant try/except)
  elastic.py        36% →  99%
  opensearch.py     40% →  99%
  cli.py            52% →  69%
  __init__.py       74% →  76%   (also fixes append_json bug below)
  utils.py          84% (unchanged in this PR)
  TOTAL             59% →  83%

The remaining 17% is honest. The biggest unreached blocks are
_main() in cli.py and the watch-mode mailbox iteration in __init__.py,
both of which would require either standing up live subsystems (real
Elasticsearch, real IMAP) or mocking deep enough that the test would
verify the mock rather than the code. The PR-A AGENTS.md guidance —
"if 90% requires faking it, ship 85% honestly" — applies here.

Bugs fixed while writing tests:

1. parsedmarc/s3.py — SMTP-TLS-to-S3 was completely broken.
   save_report_to_s3 unconditionally read report["report_metadata"]
   when building S3 object metadata, but RFC 8460 §4.3 SMTP TLS
   reports are flat (no report_metadata sub-object). The CLI's
   surrounding try/except silently swallowed the KeyError, so every
   SMTP-TLS report quietly failed to upload. Also fixes a related
   issue: parse_smtp_tls_report_json stores begin_date as the raw
   ISO-8601 string from the report (per the SMTPTLSReport TypedDict
   and RFC 8460 §4.3), but the S3 code path assumed a datetime
   with .year / .month / .day attributes. Both fixed; the broken
   metadata-extraction branch now uses the flat-report fields, and
   the date branch normalizes via human_timestamp_to_datetime.

2. parsedmarc/__init__.py — append_json corrupted JSON output files
   on the second write. The original implementation opened files in
   "a+" mode, then seek()ed backwards to overwrite the trailing "]"
   with ",\n" before appending more elements. Python's docs are
   explicit (https://docs.python.org/3/library/functions.html#open):
   on POSIX, writes in "a"/"a+" mode always go to EOF regardless of
   seek() position. The result was that the second call produced
   [...]\n],\n[...] -style corrupted output instead of a single
   merged array. Replaced with a read-merge-write pattern: load the
   existing array (if any), append the new elements, rewrite the
   whole file. The CSV cousin append_csv was not affected — it
   doesn't seek backwards.

3. parsedmarc/webhook.py — removed redundant try/except blocks in
   save_aggregate_report_to_webhook / save_failure_report_to_webhook
   / save_smtp_tls_report_to_webhook. _send_to_webhook already
   catches every Exception itself, so the outer except blocks were
   unreachable dead code (covered nothing, defended against nothing,
   and inflated the source-line count without testing value).

Testing approach: mocks at SDK boundaries (boto3 resource, kafka
producer, requests session, opensearch/elasticsearch Document/Search,
azure LogsIngestionClient). Tests verify the parsedmarc-side
transformation logic — document/event construction, index/topic
naming, dedup queries, error wrapping — rather than asserting on
mock invocations as a proxy for behaviour. Where a branch is
defensive against a caller that doesn't exist in the codebase, the
test is omitted (commented in code rather than hidden behind a
pragma).

547 tests total (was 276), all passing. ruff check + format clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Document the two bug fixes from this PR in the 10.0.0 changelog

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Document testing standards in AGENTS.md

Adds a "Testing standards" section covering the principles applied in
PR-A (split) and PR-B (coverage expansion):

- Coverage measures shipped code only — don't reintroduce tests/* to
  the scope, don't expand omit, don't use # pragma: no cover.
- Honest tests assert on observable behaviour, not "the mock was called".
  Mock at SDK boundaries; parse the payload that gets sent.
- "If 90% requires faking it, ship 85% honestly" — coverage is a tool,
  not a goal. PR-B's deliberate stops at cli.py 69% and __init__.py 76%
  are the documented precedent for when to halt.
- Verify bug claims against the relevant RFC, internal types, installed
  SDK source, or upstream docs before changing code. Cite the source in
  the commit message and test docstring (RFC 8460 §4.3 and the Python
  open() docs for #775's two bug fixes are the pattern to follow).
- Bugs found while writing tests are fixed in the same PR; the test
  doubles as the regression guard.
- File layout (tests/test_<module>.py) is non-negotiable; module-level
  test loggers need fresh-handler setup so test ordering doesn't break
  assertLogs.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* Cover the corrupt-file fallback in append_json

Codecov flagged 2 missing patch-coverage lines on PR #775: the
except (json.JSONDecodeError, OSError) branch in append_json, which
falls back to overwriting when the existing file isn't a parseable
JSON array. Two new tests in tests/test_init.py:TestAppendJson
exercise both paths:

- test_corrupt_existing_file_is_overwritten_cleanly: existing file
  contains invalid JSON; append_json overwrites with the new array.
- test_existing_file_with_non_list_root_is_overwritten: existing
  file parses as {"foo": ...} (dict, not list); the isinstance guard
  rejects it and we overwrite cleanly.

Patch coverage now 100% on the bug fix.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

tests/test_init.py

@@ -2306,5 +2306,178 @@ class TestGetDmarcReportsFromMbox(unittest.TestCase):
             os.remove(path)
 
 
+class TestGetDmarcReportsFromMailboxValidation(unittest.TestCase):
+    """Input validation on get_dmarc_reports_from_mailbox.
+
+    These guards prevent two real footguns: the test/delete combo
+    would otherwise delete every message after parsing — silently
+    destructive — and a None connection would NPE deep in the
+    iteration loop with a confusing traceback. Fail fast at the
+    door instead."""
+
+    def test_delete_and_test_combination_raises(self):
+        from unittest.mock import MagicMock
+
+        with self.assertRaises(ValueError) as ctx:
+            parsedmarc.get_dmarc_reports_from_mailbox(
+                connection=MagicMock(), delete=True, test=True
+            )
+        self.assertIn("mutually exclusive", str(ctx.exception))
+
+    def test_none_connection_raises(self):
+        with self.assertRaises(ValueError) as ctx:
+            parsedmarc.get_dmarc_reports_from_mailbox(connection=None)
+        self.assertIn("connection", str(ctx.exception).lower())
+
+
+class TestEmailResultsErrorBranches(unittest.TestCase):
+    """email_results requires mail_to to be a list — this is enforced
+    by an assert. A regression that dropped the assert would mean the
+    SMTP code further down would silently iterate over the characters
+    of a string."""
+
+    def test_mail_to_must_be_list(self):
+        with self.assertRaises(AssertionError):
+            parsedmarc.email_results(
+                {
+                    "aggregate_reports": [],
+                    "failure_reports": [],
+                    "smtp_tls_reports": [],
+                },
+                host="smtp.example.com",
+                mail_from="from@example.com",
+                mail_to="admin@example.com",  # str, not list — triggers assert
+            )
+
+
+class TestAppendJson(unittest.TestCase):
+    """append_json writes new files cleanly and merges into existing
+    JSON arrays without breaking valid JSON."""
+
+    def test_writes_new_file(self):
+        with NamedTemporaryFile("w", suffix=".json", delete=False) as tf:
+            path = tf.name
+        os.remove(path)  # ensure file is fresh
+        try:
+            parsedmarc.append_json(path, [{"a": 1}])
+            with open(path) as f:
+                data = json.loads(f.read())
+            self.assertEqual(data, [{"a": 1}])
+        finally:
+            if os.path.exists(path):
+                os.remove(path)
+
+    def test_appends_to_existing_file(self):
+        with NamedTemporaryFile("w", suffix=".json", delete=False) as tf:
+            path = tf.name
+        try:
+            parsedmarc.append_json(path, [{"a": 1}])
+            parsedmarc.append_json(path, [{"b": 2}])
+            with open(path) as f:
+                data = json.loads(f.read())
+            self.assertEqual(data, [{"a": 1}, {"b": 2}])
+        finally:
+            if os.path.exists(path):
+                os.remove(path)
+
+    def test_empty_list_on_existing_file_is_noop(self):
+        with NamedTemporaryFile("w", suffix=".json", delete=False) as tf:
+            path = tf.name
+        try:
+            parsedmarc.append_json(path, [{"a": 1}])
+            parsedmarc.append_json(path, [])
+            with open(path) as f:
+                data = json.loads(f.read())
+            self.assertEqual(data, [{"a": 1}])
+        finally:
+            if os.path.exists(path):
+                os.remove(path)
+
+    def test_corrupt_existing_file_is_overwritten_cleanly(self):
+        """If the existing JSON file is corrupt (e.g. truncated by a
+        prior crash, or hit the pre-fix `append_json` bug), the
+        read-merge-write path falls back to overwriting with the new
+        content rather than silently failing to record.
+
+        Recording at the cost of losing prior corrupt data is the
+        lesser evil — those bytes are already unparseable, so no
+        downstream consumer can read them anyway."""
+        with NamedTemporaryFile("w", suffix=".json", delete=False) as tf:
+            tf.write("{ this is not valid json at all")
+            path = tf.name
+        try:
+            parsedmarc.append_json(path, [{"new": "data"}])
+            with open(path) as f:
+                data = json.loads(f.read())
+            self.assertEqual(data, [{"new": "data"}])
+        finally:
+            if os.path.exists(path):
+                os.remove(path)
+
+    def test_existing_file_with_non_list_root_is_overwritten(self):
+        """If the existing file parses cleanly but the root isn't a
+        list (e.g. someone wrote {"foo": 1} by hand), the
+        isinstance(loaded, list) guard kicks in and we overwrite
+        rather than concatenating a dict and a list."""
+        with NamedTemporaryFile("w", suffix=".json", delete=False) as tf:
+            tf.write('{"not": "a list"}')
+            path = tf.name
+        try:
+            parsedmarc.append_json(path, [{"new": "data"}])
+            with open(path) as f:
+                data = json.loads(f.read())
+            self.assertEqual(data, [{"new": "data"}])
+        finally:
+            if os.path.exists(path):
+                os.remove(path)
+
+
+class TestAppendCsv(unittest.TestCase):
+    def test_writes_new_file_with_header(self):
+        with NamedTemporaryFile("w", suffix=".csv", delete=False) as tf:
+            path = tf.name
+        os.remove(path)
+        try:
+            parsedmarc.append_csv(path, "h1,h2\nv1,v2\n")
+            with open(path) as f:
+                content = f.read()
+            self.assertEqual(content, "h1,h2\nv1,v2\n")
+        finally:
+            if os.path.exists(path):
+                os.remove(path)
+
+    def test_appends_strips_header_on_existing_file(self):
+        """Second append must not re-emit the header line."""
+        with NamedTemporaryFile("w", suffix=".csv", delete=False) as tf:
+            path = tf.name
+        try:
+            parsedmarc.append_csv(path, "h1,h2\nv1,v2\n")
+            parsedmarc.append_csv(path, "h1,h2\nv3,v4\n")
+            with open(path) as f:
+                content = f.read()
+            # Only one header line in the merged output.
+            self.assertEqual(content.count("h1,h2"), 1)
+            self.assertIn("v3,v4", content)
+        finally:
+            if os.path.exists(path):
+                os.remove(path)
+
+    def test_append_empty_csv_on_existing_file_is_noop(self):
+        """append_csv with just a header row (no data) should not
+        rewrite the file when one already exists."""
+        with NamedTemporaryFile("w", suffix=".csv", delete=False) as tf:
+            path = tf.name
+        try:
+            parsedmarc.append_csv(path, "h1,h2\nv1,v2\n")
+            parsedmarc.append_csv(path, "h1,h2\n")
+            with open(path) as f:
+                content = f.read()
+            # File unchanged.
+            self.assertEqual(content, "h1,h2\nv1,v2\n")
+        finally:
+            if os.path.exists(path):
+                os.remove(path)
+
+
 if __name__ == "__main__":
     unittest.main(verbosity=2)