Switch the bundled IP-to-country database from DB-IP Country Lite to
IPinfo Lite for greater lookup accuracy. The download URL, cached
filename, and packaged module path all move from
dbip/dbip-country-lite.mmdb to ipinfo/ipinfo_lite.mmdb.
IPinfo Lite uses a different MMDB schema (flat country_code) that is
incompatible with geoip2's Reader.country() helper, so get_ip_address_country()
now uses maxminddb directly and handles both the IPinfo schema and
the MaxMind/DBIP nested country.iso_code schema so users who drop in
their own MMDB from any of these providers continue to work.
Drop the geoip2 dependency (it was only used for the incompatible
helper) and add maxminddb as a direct dependency — it was already
installed transitively through geoip2.
Callers that imported parsedmarc.resources.dbip directly need to switch
to parsedmarc.resources.ipinfo. Old parsedmarc versions downloading
from the dbip/ GitHub raw URL will 404 and fall back to their bundled
copy — this is the documented behavior of load_ip_db().
Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Auto-download psl_overrides.txt at startup (and whenever the reverse DNS
map is reloaded) via load_psl_overrides(); add local_psl_overrides_path
and psl_overrides_url config options
- Add collect_domain_info.py and detect_psl_overrides.py for bulk WHOIS/HTTP
enrichment and automatic cluster-based PSL override detection
- Block full-IPv4 reverse-DNS entries from ever entering
base_reverse_dns_map.csv, known_unknown_base_reverse_dns.txt, or
unknown_base_reverse_dns.csv, and sweep pre-existing IP entries
- Add Religion and Utilities to the allowed service_type values
- Document the full map-maintenance workflow in AGENTS.md
- Substantial expansion of base_reverse_dns_map.csv (net ~+1,000 entries)
- Add 26 tests covering the new loader, IP filter, PSL fold logic, and
cluster detection
Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Download the latest DB-IP Country Lite mmdb from GitHub on startup and
SIGHUP, caching it locally, with fallback to a previously cached or
bundled copy. Skipped when the offline flag is set. Adds ip_db_url
config option (PARSEDMARC_GENERAL_IP_DB_URL) to override the download
URL. Bumps version to 9.6.0.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add environment variable configuration support and update documentation
- Introduced support for configuration via environment variables using the `PARSEDMARC_{SECTION}_{KEY}` format.
- Added `PARSEDMARC_CONFIG_FILE` variable to specify the config file path.
- Enabled env-only mode for file-less Docker deployments.
- Implemented explicit read permission checks on config files.
- Updated changelog and usage documentation to reflect these changes.
Elasticsearch and OpenSearch now verify SSL certificates by default when `ssl = True`, even without a `cert_path`
- Added `skip_certificate_verification` option to the `elasticsearch` and `opensearch` configuration sections for consistency with `splunk_hec`
- Splunk HEC `skip_certificate_verification` now works correctly with self-signed certificates
- SMTP TLS reports no longer fail when saving to multiple output targets (e.g. Elasticsearch and OpenSearch) due to in-place mutation of the report dict
- Output client initialization errors now identify which module failed (e.g. "OpenSearch: ConnectionError..." instead of generic "Output client error")
- Enhanced error handling for output client initialization
* Add MS Graph certificate authentication support
* Preserve MS Graph constructor compatibility
---------
Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>
- Updated parsedmarc/syslog.py to support UDP, TCP, and TLS protocols
- Added protocol parameter with UDP as default for backward compatibility
- Implemented TLS support with CA verification and client certificate auth
- Added retry logic for TCP/TLS connections with configurable attempts and delays
- Updated parsedmarc/cli.py with new config file parsing
- Updated documentation with examples for TCP and TLS configurations
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
* Remove CLI arguments for syslog options, keep config-file only
Per user request, removed command-line argument options for syslog parameters.
All new syslog options (protocol, TLS settings, timeout, retry) are now only
available via configuration file, consistent with other similar options.
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
* Fix code review issues: remove trailing whitespace and add cert validation
- Removed trailing whitespace from syslog.py and usage.md
- Added warning when only one of certfile_path/keyfile_path is provided
- Improved error handling for incomplete TLS client certificate configuration
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
* Set minimum TLS version to 1.2 for enhanced security
Explicitly configured ssl_context.minimum_version = TLSVersion.TLSv1_2
to ensure only secure TLS versions are used for syslog connections.
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: seanthegeek <44679+seanthegeek@users.noreply.github.com>
8.19.0
- Add multi-tenant support via an index-prefix domain mapping file
- PSL overrides so that services like AWS are correctly identified
- Additional improvements to report type detection
- Fix webhook timeout parsing (PR #623)
- Output to STDOUT when the new general config boolean `silent` is set to `False` (Close#614)
- Additional services added to `base_reverse_dns_map.csv`
---------
Co-authored-by: Sean Whalen <seanthegeek@users.noreply.github.com>
Co-authored-by: Félix <felix.debloisbeaucage@gmail.com>
* Added output to webhook as an option
* added documentation for new webhook configuration
---------
Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>
- Added `since` option under `mailbox` section to search for messages since a certain time instead of going through the complete mailbox during testing scenarios. Acceptable values -`5m|3h|2d|1w`, units - {"m":"minutes", "h":"hours", "d":"days", "w":"weeks"}). Defaults to `1d` if an incorrect value is provided.
- Not to mark messages as read if test option is selected (works only for MSGraphConnection)
- Actually save `source_type` and `source_name` to Elasticsearch and OpenSearch
- Reverse-lookup cache improvements (PR #501 closes issue #498)
- Update the included `dbip-country-lite.mmdb` to the 2024-03 version
- Update `base_reverse_dns_map.csv`
- Add new general config options (closes issue #500)
- `always_use_local_files` - Disables the download of the reverse DNS map
- `local_reverse_dns_map_path` - Overrides the default local file path to use for the reverse DNS map
- `reverse_dns_map_url` - Overrides the default download URL for the reverse DNS map
* Set global TOC collapse to false
* Split documentation
I tried to split the index.md file into logical parts, not changing the contents.
I did add a space and change one HTTP URL to HTTPS.
---------
Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>
