Add offline mode for tests in GitHub Actions to skip DNS lookups

Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>

.github/workflows/python-tests.yml

@@ -30,7 +30,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.10", "3.11", "3.12", "3.13", "3.14"]
+        python-version: ["3.9", "3.10", "3.11", "3.12", "3.13", "3.14"]
 
     steps:
     - uses: actions/checkout@v5

AGENTS.md

@@ -1,64 +0,0 @@
-# AGENTS.md
-
-This file provides guidance to AI agents when working with code in this repository.
-
-## Project Overview
-
-parsedmarc is a Python module and CLI utility for parsing DMARC aggregate (RUA), forensic (RUF), and SMTP TLS reports. It reads reports from IMAP, Microsoft Graph, Gmail API, Maildir, mbox files, or direct file paths, and outputs to JSON/CSV, Elasticsearch, OpenSearch, Splunk, Kafka, S3, Azure Log Analytics, syslog, or webhooks.
-
-## Common Commands
-
-```bash
-# Install with dev/build dependencies
-pip install .[build]
-
-# Run all tests with coverage
-pytest --cov --cov-report=xml tests.py
-
-# Run a single test
-pytest tests.py::Test::testAggregateSamples
-
-# Lint and format
-ruff check .
-ruff format .
-
-# Test CLI with sample reports
-parsedmarc --debug -c ci.ini samples/aggregate/*
-parsedmarc --debug -c ci.ini samples/forensic/*
-
-# Build docs
-cd docs && make html
-
-# Build distribution
-hatch build
-```
-
-To skip DNS lookups during testing, set `GITHUB_ACTIONS=true`.
-
-## Architecture
-
-**Data flow:** Input sources → CLI (`cli.py:_main`) → Parse (`__init__.py`) → Enrich (DNS/GeoIP via `utils.py`) → Output integrations
-
-### Key modules
-
-- `parsedmarc/__init__.py` — Core parsing logic. Main functions: `parse_report_file()`, `parse_report_email()`, `parse_aggregate_report_xml()`, `parse_forensic_report()`, `parse_smtp_tls_report_json()`, `get_dmarc_reports_from_mailbox()`, `watch_inbox()`
-- `parsedmarc/cli.py` — CLI entry point (`_main`), config file parsing, output orchestration
-- `parsedmarc/types.py` — TypedDict definitions for all report types (`AggregateReport`, `ForensicReport`, `SMTPTLSReport`, `ParsingResults`)
-- `parsedmarc/utils.py` — IP/DNS/GeoIP enrichment, base64 decoding, compression handling
-- `parsedmarc/mail/` — Polymorphic mail connections: `IMAPConnection`, `GmailConnection`, `MSGraphConnection`, `MaildirConnection`
-- `parsedmarc/{elastic,opensearch,splunk,kafkaclient,loganalytics,syslog,s3,webhook,gelf}.py` — Output integrations
-
-### Report type system
-
-`ReportType = Literal["aggregate", "forensic", "smtp_tls"]`. Exception hierarchy: `ParserError` → `InvalidDMARCReport` → `InvalidAggregateReport`/`InvalidForensicReport`, and `InvalidSMTPTLSReport`.
-
-### Caching
-
-IP address info cached for 4 hours, seen aggregate report IDs cached for 1 hour (via `ExpiringDict`).
-
-## Code Style
-
-- Ruff for formatting and linting (configured in `.vscode/settings.json`)
-- TypedDict for structured data, type hints throughout
-- Python ≥3.10 required
-- Tests are in a single `tests.py` file using unittest; sample reports live in `samples/`

CHANGELOG.md

@@ -1,23 +1,5 @@
 # Changelog
 
-## 9.1.1
-
-### Fixes
-
-- Fix the use of Elasticsearch and OpenSearch API keys (PR #660 fixes issue #653)
-
-### Changes
-
-- Drop support for Python 3.9 (PR #661)
-
-## 9.1.0
-
-## Enhancements
-
-- Add TCP and TLS support for syslog output. (#656)
-- Skip DNS lookups in GitHub Actions to prevent DNS timeouts during tests timeouts. (#657)
-- Remove microseconds from DMARC aggregate report time ranges before parsing them.
-
 ## 9.0.10
 
 - Support Python 3.14+

CLAUDE.md

@@ -1,3 +0,0 @@
-# CLAUD.md
-
-@AGENTS.md

README.md

@@ -56,9 +56,9 @@ for RHEL or Debian.
 | 3.6     | ❌         | Used in RHEL 8, but not supported by project dependencies |
 | 3.7     | ❌         | End of Life (EOL)                                          |
 | 3.8     | ❌         | End of Life (EOL)                                          |
-| 3.9     | ❌         | Used in Debian 11 and RHEL 9, but not supported by project dependencies |
+| 3.9     | ✅         | Supported until August 2026 (Debian 11); May 2032 (RHEL 9) |
 | 3.10    | ✅         | Actively maintained                                        |
 | 3.11    | ✅         | Actively maintained; supported until June 2028 (Debian 12) |
 | 3.12    | ✅         | Actively maintained; supported until May 2035 (RHEL 10)    |
 | 3.13    | ✅         | Actively maintained; supported until June 2030 (Debian 13) |
-| 3.14    | ✅         | Supported (requires `imapclient>=3.1.0`)                  |
+| 3.14    | ✅         | Actively maintained                                        |

docs/source/index.md

@@ -56,12 +56,12 @@ for RHEL or Debian.
 | 3.6     | ❌         | Used in RHEL 8, but not supported by project dependencies |
 | 3.7     | ❌         | End of Life (EOL)                                          |
 | 3.8     | ❌         | End of Life (EOL)                                          |
-| 3.9     | ❌         | Used in Debian 11 and RHEL 9, but not supported by project dependencies |
+| 3.9     | ✅         | Supported until August 2026 (Debian 11); May 2032 (RHEL 9) |
 | 3.10    | ✅         | Actively maintained                                        |
 | 3.11    | ✅         | Actively maintained; supported until June 2028 (Debian 12) |
 | 3.12    | ✅         | Actively maintained; supported until May 2035 (RHEL 10)    |
 | 3.13    | ✅         | Actively maintained; supported until June 2030 (Debian 13) |
-| 3.14    | ✅         | Supported (requires `imapclient>=3.1.0`)                  |
+| 3.14    | ✅         | Actively maintained                                        |
 
 ```{toctree}
 :caption: 'Contents'

docs/source/installation.md

@@ -162,10 +162,10 @@ sudo -u parsedmarc virtualenv /opt/parsedmarc/venv
 ```
 
 CentOS/RHEL 8 systems use Python 3.6 by default, so on those systems
-explicitly tell `virtualenv` to use `python3.10` instead
+explicitly tell `virtualenv` to use `python3.9` instead
 
 ```bash
-sudo -u parsedmarc virtualenv -p python3.10  /opt/parsedmarc/venv
+sudo -u parsedmarc virtualenv -p python3.9  /opt/parsedmarc/venv
 ```
 
 Activate the virtualenv

docs/source/usage.md

@@ -171,8 +171,8 @@ The full set of configuration options are:
   - `check_timeout` - int: Number of seconds to wait for a IMAP
       IDLE response or the number of seconds until the next
       mail check (Default: `30`)
-  - `since` - str: Search for messages since certain time. (Examples: `5m|3h|2d|1w`)
+  - `since` - str: Search for messages since certain time. (Examples: `5m|3h|2d|1w`) 
-      Acceptable units - {"m":"minutes", "h":"hours", "d":"days", "w":"weeks"}.
+      Acceptable units - {"m":"minutes", "h":"hours", "d":"days", "w":"weeks"}. 
       Defaults to `1d` if incorrect value is provided.
 - `imap`
   - `host` - str: The IMAP server hostname or IP address
@@ -240,7 +240,7 @@ The full set of configuration options are:
     group and use that as the group id.
 
     ```powershell
-    New-ApplicationAccessPolicy -AccessRight RestrictAccess
+    New-ApplicationAccessPolicy -AccessRight RestrictAccess 
     -AppId "<CLIENT_ID>" -PolicyScopeGroupId "<MAILBOX>"
     -Description "Restrict access to dmarc reports mailbox."
     ```
@@ -336,65 +336,13 @@ The full set of configuration options are:
   - `secret_access_key` - str: The secret access key (Optional)
 - `syslog`
   - `server` - str: The Syslog server name or IP address
-  - `port` - int: The port to use (Default: `514`)
+  - `port` - int: The UDP port to use (Default: `514`)
-  - `protocol` - str: The protocol to use: `udp`, `tcp`, or `tls` (Default: `udp`)
-  - `cafile_path` - str: Path to CA certificate file for TLS server verification (Optional)
-  - `certfile_path` - str: Path to client certificate file for TLS authentication (Optional)
-  - `keyfile_path` - str: Path to client private key file for TLS authentication (Optional)
-  - `timeout` - float: Connection timeout in seconds for TCP/TLS (Default: `5.0`)
-  - `retry_attempts` - int: Number of retry attempts for failed connections (Default: `3`)
-  - `retry_delay` - int: Delay in seconds between retry attempts (Default: `5`)
-
-  **Example UDP configuration (default):**
-
-  ```ini
-  [syslog]
-  server = syslog.example.com
-  port = 514
-  ```
-
-  **Example TCP configuration:**
-
-  ```ini
-  [syslog]
-  server = syslog.example.com
-  port = 6514
-  protocol = tcp
-  timeout = 10.0
-  retry_attempts = 5
-  ```
-
-  **Example TLS configuration with server verification:**
-
-  ```ini
-  [syslog]
-  server = syslog.example.com
-  port = 6514
-  protocol = tls
-  cafile_path = /path/to/ca-cert.pem
-  timeout = 10.0
-  ```
-
-  **Example TLS configuration with mutual authentication:**
-
-  ```ini
-  [syslog]
-  server = syslog.example.com
-  port = 6514
-  protocol = tls
-  cafile_path = /path/to/ca-cert.pem
-  certfile_path = /path/to/client-cert.pem
-  keyfile_path = /path/to/client-key.pem
-  timeout = 10.0
-  retry_attempts = 3
-  retry_delay = 5
-  ```
 - `gmail_api`
   - `credentials_file` - str: Path to file containing the
       credentials, None to disable (Default: `None`)
   - `token_file` - str: Path to save the token file
       (Default: `.token`)
-
+      
     :::{note}
     credentials_file and token_file can be got with [quickstart](https://developers.google.com/gmail/api/quickstart/python).Please change the scope to `https://www.googleapis.com/auth/gmail.modify`.
     :::
@@ -494,7 +442,7 @@ Update the limit to 2k per example:
 PUT _cluster/settings
 {
   "persistent" : {
-    "cluster.max_shards_per_node" : 2000
+    "cluster.max_shards_per_node" : 2000 
   }
 }
 ```

parsedmarc/cli.py

@@ -697,13 +697,6 @@ def _main():
         s3_secret_access_key=None,
         syslog_server=None,
         syslog_port=None,
-        syslog_protocol=None,
-        syslog_cafile_path=None,
-        syslog_certfile_path=None,
-        syslog_keyfile_path=None,
-        syslog_timeout=None,
-        syslog_retry_attempts=None,
-        syslog_retry_delay=None,
         gmail_api_credentials_file=None,
         gmail_api_token_file=None,
         gmail_api_include_spam_trash=False,
@@ -1058,10 +1051,10 @@ def _main():
                 opts.elasticsearch_password = elasticsearch_config["password"]
             # Until 8.20
             if "apiKey" in elasticsearch_config:
-                opts.elasticsearch_api_key = elasticsearch_config["apiKey"]
+                opts.elasticsearch_apiKey = elasticsearch_config["apiKey"]
             # Since 8.20
             if "api_key" in elasticsearch_config:
-                opts.elasticsearch_api_key = elasticsearch_config["api_key"]
+                opts.elasticsearch_apiKey = elasticsearch_config["api_key"]
 
         if "opensearch" in config:
             opensearch_config = config["opensearch"]
@@ -1098,10 +1091,10 @@ def _main():
                 opts.opensearch_password = opensearch_config["password"]
             # Until 8.20
             if "apiKey" in opensearch_config:
-                opts.opensearch_api_key = opensearch_config["apiKey"]
+                opts.opensearch_apiKey = opensearch_config["apiKey"]
             # Since 8.20
             if "api_key" in opensearch_config:
-                opts.opensearch_api_key = opensearch_config["api_key"]
+                opts.opensearch_apiKey = opensearch_config["api_key"]
 
         if "splunk_hec" in config.sections():
             hec_config = config["splunk_hec"]
@@ -1246,28 +1239,6 @@ def _main():
                 opts.syslog_port = syslog_config["port"]
             else:
                 opts.syslog_port = 514
-            if "protocol" in syslog_config:
-                opts.syslog_protocol = syslog_config["protocol"]
-            else:
-                opts.syslog_protocol = "udp"
-            if "cafile_path" in syslog_config:
-                opts.syslog_cafile_path = syslog_config["cafile_path"]
-            if "certfile_path" in syslog_config:
-                opts.syslog_certfile_path = syslog_config["certfile_path"]
-            if "keyfile_path" in syslog_config:
-                opts.syslog_keyfile_path = syslog_config["keyfile_path"]
-            if "timeout" in syslog_config:
-                opts.syslog_timeout = float(syslog_config["timeout"])
-            else:
-                opts.syslog_timeout = 5.0
-            if "retry_attempts" in syslog_config:
-                opts.syslog_retry_attempts = int(syslog_config["retry_attempts"])
-            else:
-                opts.syslog_retry_attempts = 3
-            if "retry_delay" in syslog_config:
-                opts.syslog_retry_delay = int(syslog_config["retry_delay"])
-            else:
-                opts.syslog_retry_delay = 5
 
         if "gmail_api" in config.sections():
             gmail_api_config = config["gmail_api"]
@@ -1465,17 +1436,6 @@ def _main():
             syslog_client = syslog.SyslogClient(
                 server_name=opts.syslog_server,
                 server_port=int(opts.syslog_port),
-                protocol=opts.syslog_protocol or "udp",
-                cafile_path=opts.syslog_cafile_path,
-                certfile_path=opts.syslog_certfile_path,
-                keyfile_path=opts.syslog_keyfile_path,
-                timeout=opts.syslog_timeout if opts.syslog_timeout is not None else 5.0,
-                retry_attempts=opts.syslog_retry_attempts
-                if opts.syslog_retry_attempts is not None
-                else 3,
-                retry_delay=opts.syslog_retry_delay
-                if opts.syslog_retry_delay is not None
-                else 5,
             )
         except Exception as error_:
             logger.error("Syslog Error: {0}".format(error_.__str__()))

parsedmarc/constants.py

@@ -1,3 +1,3 @@
-__version__ = "9.1.1"
+__version__ = "9.0.10"
 
 USER_AGENT = f"parsedmarc/{__version__}"

parsedmarc/elastic.py

@@ -413,8 +413,8 @@ def save_aggregate_report_to_elasticsearch(
     org_name_query = Q(dict(match_phrase=dict(org_name=org_name)))  # type: ignore
     report_id_query = Q(dict(match_phrase=dict(report_id=report_id)))  # pyright: ignore[reportArgumentType]
     domain_query = Q(dict(match_phrase={"published_policy.domain": domain}))  # pyright: ignore[reportArgumentType]
-    begin_date_query = Q(dict(range=dict(date_begin=dict(gte=begin_date))))  # pyright: ignore[reportArgumentType]
+    begin_date_query = Q(dict(match=dict(date_begin=begin_date)))  # pyright: ignore[reportArgumentType]
-    end_date_query = Q(dict(range=dict(date_end=dict(lte=end_date))))  # pyright: ignore[reportArgumentType]
+    end_date_query = Q(dict(match=dict(date_end=end_date)))  # pyright: ignore[reportArgumentType]
 
     if index_suffix is not None:
         search_index = "dmarc_aggregate_{0}*".format(index_suffix)

parsedmarc/opensearch.py

@@ -413,8 +413,8 @@ def save_aggregate_report_to_opensearch(
     org_name_query = Q(dict(match_phrase=dict(org_name=org_name)))
     report_id_query = Q(dict(match_phrase=dict(report_id=report_id)))
     domain_query = Q(dict(match_phrase={"published_policy.domain": domain}))
-    begin_date_query = Q(dict(range=dict(date_begin=dict(gte=begin_date))))
+    begin_date_query = Q(dict(match=dict(date_begin=begin_date)))
-    end_date_query = Q(dict(range=dict(date_end=dict(lte=end_date))))
+    end_date_query = Q(dict(match=dict(date_end=end_date)))
 
     if index_suffix is not None:
         search_index = "dmarc_aggregate_{0}*".format(index_suffix)

parsedmarc/syslog.py

@@ -6,10 +6,7 @@ from __future__ import annotations
 import json
 import logging
 import logging.handlers
-import socket
+from typing import Any
-import ssl
-import time
-from typing import Any, Optional
 
 from parsedmarc import (
     parsed_aggregate_reports_to_csv_rows,
@@ -21,150 +18,20 @@ from parsedmarc import (
 class SyslogClient(object):
     """A client for Syslog"""
 
-    def __init__(
+    def __init__(self, server_name: str, server_port: int):
-        self,
-        server_name: str,
-        server_port: int,
-        protocol: str = "udp",
-        cafile_path: Optional[str] = None,
-        certfile_path: Optional[str] = None,
-        keyfile_path: Optional[str] = None,
-        timeout: float = 5.0,
-        retry_attempts: int = 3,
-        retry_delay: int = 5,
-    ):
         """
         Initializes the SyslogClient
         Args:
             server_name (str): The Syslog server
-            server_port (int): The Syslog port
+            server_port (int): The Syslog UDP port
-            protocol (str): The protocol to use: "udp", "tcp", or "tls" (Default: "udp")
-            cafile_path (str): Path to CA certificate file for TLS server verification (Optional)
-            certfile_path (str): Path to client certificate file for TLS authentication (Optional)
-            keyfile_path (str): Path to client private key file for TLS authentication (Optional)
-            timeout (float): Connection timeout in seconds for TCP/TLS (Default: 5.0)
-            retry_attempts (int): Number of retry attempts for failed connections (Default: 3)
-            retry_delay (int): Delay in seconds between retry attempts (Default: 5)
         """
         self.server_name = server_name
         self.server_port = server_port
-        self.protocol = protocol.lower()
-        self.timeout = timeout
-        self.retry_attempts = retry_attempts
-        self.retry_delay = retry_delay
-
         self.logger = logging.getLogger("parsedmarc_syslog")
         self.logger.setLevel(logging.INFO)
-
+        log_handler = logging.handlers.SysLogHandler(address=(server_name, server_port))
-        # Create the appropriate syslog handler based on protocol
-        log_handler = self._create_syslog_handler(
-            server_name,
-            server_port,
-            self.protocol,
-            cafile_path,
-            certfile_path,
-            keyfile_path,
-            timeout,
-            retry_attempts,
-            retry_delay,
-        )
-
         self.logger.addHandler(log_handler)
 
-    def _create_syslog_handler(
-        self,
-        server_name: str,
-        server_port: int,
-        protocol: str,
-        cafile_path: Optional[str],
-        certfile_path: Optional[str],
-        keyfile_path: Optional[str],
-        timeout: float,
-        retry_attempts: int,
-        retry_delay: int,
-    ) -> logging.handlers.SysLogHandler:
-        """
-        Creates a SysLogHandler with the specified protocol and TLS settings
-        """
-        if protocol == "udp":
-            # UDP protocol (default, backward compatible)
-            return logging.handlers.SysLogHandler(
-                address=(server_name, server_port),
-                socktype=socket.SOCK_DGRAM,
-            )
-        elif protocol in ["tcp", "tls"]:
-            # TCP or TLS protocol with retry logic
-            for attempt in range(1, retry_attempts + 1):
-                try:
-                    if protocol == "tcp":
-                        # TCP without TLS
-                        handler = logging.handlers.SysLogHandler(
-                            address=(server_name, server_port),
-                            socktype=socket.SOCK_STREAM,
-                        )
-                        # Set timeout on the socket
-                        if hasattr(handler, "socket") and handler.socket:
-                            handler.socket.settimeout(timeout)
-                        return handler
-                    else:
-                        # TLS protocol
-                        # Create SSL context with secure defaults
-                        ssl_context = ssl.create_default_context()
-
-                        # Explicitly set minimum TLS version to 1.2 for security
-                        ssl_context.minimum_version = ssl.TLSVersion.TLSv1_2
-
-                        # Configure server certificate verification
-                        if cafile_path:
-                            ssl_context.load_verify_locations(cafile=cafile_path)
-
-                        # Configure client certificate authentication
-                        if certfile_path and keyfile_path:
-                            ssl_context.load_cert_chain(
-                                certfile=certfile_path,
-                                keyfile=keyfile_path,
-                            )
-                        elif certfile_path or keyfile_path:
-                            # Warn if only one of the two required parameters is provided
-                            self.logger.warning(
-                                "Both certfile_path and keyfile_path are required for "
-                                "client certificate authentication. Client authentication "
-                                "will not be used."
-                            )
-
-                        # Create TCP handler first
-                        handler = logging.handlers.SysLogHandler(
-                            address=(server_name, server_port),
-                            socktype=socket.SOCK_STREAM,
-                        )
-
-                        # Wrap socket with TLS
-                        if hasattr(handler, "socket") and handler.socket:
-                            handler.socket = ssl_context.wrap_socket(
-                                handler.socket,
-                                server_hostname=server_name,
-                            )
-                            handler.socket.settimeout(timeout)
-
-                        return handler
-
-                except Exception as e:
-                    if attempt < retry_attempts:
-                        self.logger.warning(
-                            f"Syslog connection attempt {attempt}/{retry_attempts} failed: {e}. "
-                            f"Retrying in {retry_delay} seconds..."
-                        )
-                        time.sleep(retry_delay)
-                    else:
-                        self.logger.error(
-                            f"Syslog connection failed after {retry_attempts} attempts: {e}"
-                        )
-                        raise
-        else:
-            raise ValueError(
-                f"Invalid protocol '{protocol}'. Must be 'udp', 'tcp', or 'tls'."
-            )
-
     def save_aggregate_report_to_syslog(self, aggregate_reports: list[dict[str, Any]]):
         rows = parsed_aggregate_reports_to_csv_rows(aggregate_reports)
         for row in rows:

parsedmarc/types.py

@@ -2,7 +2,7 @@ from __future__ import annotations
 
 from typing import Any, Dict, List, Literal, Optional, TypedDict, Union
 
-# NOTE: This module is intentionally Python 3.10 compatible.
+# NOTE: This module is intentionally Python 3.9 compatible.
 # - No PEP 604 unions (A | B)
 # - No typing.NotRequired / Required (3.11+) to avoid an extra dependency.
 #   For optional keys, use total=False TypedDicts.

pyproject.toml

@@ -2,7 +2,7 @@
 requires = [
     "hatchling>=1.27.0",
 ]
-requires_python = ">=3.10,<3.15"
+requires_python = ">=3.9,<3.14"
 build-backend = "hatchling.build"
 
 [project]
@@ -29,7 +29,7 @@ classifiers = [
     "Operating System :: OS Independent",
     "Programming Language :: Python :: 3"
 ]
-requires-python = ">=3.10"
+requires-python = ">=3.9"
 dependencies = [
     "azure-identity>=1.8.0",
     "azure-monitor-ingestion>=1.0.0",
@@ -45,7 +45,7 @@ dependencies = [
     "google-auth-httplib2>=0.1.0",
     "google-auth-oauthlib>=0.4.6",
     "google-auth>=2.3.3",
-    "imapclient>=3.1.0",
+    "imapclient>=2.1.0",
     "kafka-python-ng>=2.2.2",
     "lxml>=4.4.0",
     "mailsuite>=1.11.2",