diff --git a/CHANGELOG.md b/CHANGELOG.md index 91de095..957a747 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,18 +1,14 @@ -Changelog -========= +# Changelog -9.0.0 ------- +## 9.0.0 (yanked) - Normalize aggregate DMARC report volumes when a report timespan exceeds 24 hours -8.19.1 ------- +## 8.19.1 - Ignore HTML content type in report email parsing (#626) -8.19.0 ------- +## 8.19.0 - Add multi-tenant support via an index-prefix domain mapping file - PSL overrides so that services like AWS are correctly identified @@ -21,24 +17,20 @@ Changelog - Output to STDOUT when the new general config boolean `silent` is set to `False` (Close #614) - Additional services added to `base_reverse_dns_map.csv` -8.18.9 ------- +## 8.18.9 - Complete fix for #687 and more robust report type detection -8.18.8 ------- +## 8.18.8 - Fix parsing emails with an uncompressed aggregate report attachment (Closes #607) - Add `--no-prettify-json` CLI option (PR #617) -8.18.7 ------- +## 8.18.7 Removed improper spaces from `base_reverse_dns_map.csv` (Closes #612) -8.18.6 ------- +## 8.18.6 - Fix since option to correctly work with weeks (PR #604) - Add 183 entries to `base_reverse_dns_map.csv` @@ -46,25 +38,21 @@ Removed improper spaces from `base_reverse_dns_map.csv` (Closes #612) - Check for invalid UTF-8 bytes in `base_reverse_dns_map.csv` at build - Exclude unneeded items from the `parsedmarc.resources` module at build -8.18.5 ------- +## 8.18.5 - Fix CSV download -8.18.4 ------- +## 8.18.4 - Fix webhooks -8.18.3 ------- +## 8.18.3 - Move `__version__` to `parsedmarc.constants` - Create a constant `USER_AGENT` - Use the HTTP `User-Agent` header value `parsedmarc/version` for all HTTP requests -8.18.2 ------- +## 8.18.2 - Merged PR #603 - Fixes issue #595 - CI test fails for Elasticsearch @@ -76,55 +64,46 @@ Removed improper spaces from `base_reverse_dns_map.csv` (Closes #612) - Automatically fall back to the internal `base_reverse_dns_map.csv` if the received file is not valid (Fixes #602) - Print the received data to the debug log -8.18.1 ------- +## 8.18.1 - Add missing `https://` to the default Microsoft Graph URL -8.18.0 ------- +## 8.18.0 - Add support for Microsoft national clouds via Graph API base URL (PR #590) - Avoid stopping processing when an invalid DMARC report is encountered (PR #587) - Increase `http.client._MAXHEADERS` from `100` to `200` to avoid errors connecting to Elasticsearch/OpenSearch (PR #589) -8.17.0 ------- +## 8.17.0 - Ignore duplicate aggregate DMARC reports with the same `org_name` and `report_id` seen within the same hour (Fixes #535) - Fix saving SMTP TLS reports to OpenSearch (PR #585 closed issue #576) - Add 303 entries to `base_reverse_dns_map.csv` -8.16.1 ------- +## 8.16.1 - Failed attempt to ignore aggregate DMARC reports seen within a period of one hour (#535) -8.16.0 ------- +## 8.16.0 - Add a `since` option to only search for emails since a certain time (PR #527) -8.15.4 ------- +## 8.15.4 - Fix crash if aggregate report timespan is > 24 hours -8.15.3 ------- +## 8.15.3 - Ignore aggregate reports with a timespan of > 24 hours (Fixes #282) -8.15.2 ------- +## 8.15.2 - Require `mailsuite>=1.9.18` - Pins `mail-parser` version at `3.15.0` due to a parsing regression in mail-parser `4.0.0` - Parse aggregate reports with empty `` - Do not overwrite the log on each run (PR #569 fixes issue #565) -8.15.1 ------- +## 8.15.1 - Proper IMAP namespace fix (Closes issue #557 and issue #563) - Require `mailsuite>=1.9.17` @@ -132,8 +111,7 @@ Removed improper spaces from `base_reverse_dns_map.csv` (Closes #612) - Add pre-flight check for nameservers (PR #562 closes issue #543) - Reformat code with `ruff` -8.15.0 ------- +## 8.15.0 - Fix processing of SMTP-TLS reports ([#549](https://github.com/domainaware/parsedmarc/issues/549)), which broke in commit [410663d ](https://github.com/domainaware/parsedmarc/commit/410663dbcaba019ca3d3744946348b56a635480b)(PR [#530](https://github.com/domainaware/parsedmarc/pull/530)) - This PR enforced a stricter check for base64-encoded strings, which SMTP TLS reports from Google did not pass @@ -141,18 +119,15 @@ Removed improper spaces from `base_reverse_dns_map.csv` (Closes #612) - Create a separate `extract_report_from_file_path()` function for processioning reports based on a file path - Remove report extraction based on a file path from `extract_report()` -8.14.2 ------- +## 8.14.2 - Update `base_reverse_dns_map.csv` to fix over-replacement on [`f3a5f10`](https://github.com/domainaware/parsedmarc/commit/f3a5f10d67b02c5db31ae1f7ced68028f46ca2a3) (PR #553) -8.14.1 ------- +## 8.14.1 - Failed attempt to fix processing of SMTP-TLS reports (#549) -8.14.0 ------- +## 8.14.0 - Skip invalid aggregate report rows without calling the whole report invalid - Some providers such as GoDaddy will send reports with some rows missing a source IP address, while other rows are fine @@ -162,14 +137,12 @@ Removed improper spaces from `base_reverse_dns_map.csv` (Closes #612) - Replace University category with Education to be more inclusive - Update included `dbip-country-lite.mmdb` -8.13.0 ------- +## 8.13.0 - Add Elastic/OpenSearch index prefix option (PR #531 closes #159) - Add GELF output support (PR #532) -8.12.0 ------- +## 8.12.0 - Fix for deadlock with large report (#508) - Build: move to kafka-python-ng (#510) @@ -178,8 +151,7 @@ Removed improper spaces from `base_reverse_dns_map.csv` (Closes #612) - Fix if base_domain is None before get_service_from_reverse_dns_base_domain (#514) - Update base_reverse_dns_map.csv -8.11.0 ------- +## 8.11.0 - Actually save `source_type` and `source_name` to Elasticsearch and OpenSearch - Reverse-lookup cache improvements (PR #501 closes issue #498) @@ -190,50 +162,41 @@ Removed improper spaces from `base_reverse_dns_map.csv` (Closes #612) - `local_reverse_dns_map_path` - Overrides the default local file path to use for the reverse DNS map - `reverse_dns_map_url` - Overrides the default download URL for the reverse DNS map -8.10.3 ------- +## 8.10.3 - Fix flaws in `base_reverse_dns_map.csv` -8.10.2 ------- +## 8.10.2 - Fix flaws in `base_reverse_dns_map.csv` -8.10.1 ------- +## 8.10.1 - Fix flaws in `base_reverse_dns_map.csv` -8.10.0 ------- +## 8.10.0 - Fix MSGraph UsernamePassword Authentication (PR #497) - Attempt to download an updated `base_reverse_dns_map.csv` at runtime - Update included `base_reverse_dns_map.csv` -8.9.4 ------ +## 8.9.4 - Update `base_reverse_dns_map.csv` -8.9.3 ------ +## 8.9.3 - Revert change in 8.9.2 -8.9.2 ------ +## 8.9.2 - Use `Uncategorized` instead of `None` as the service type when a service cannot be identified -8.9.1 ------ +## 8.9.1 - Fix broken CLI by removing obsolete parameter from `cli_parse` call (PR #496 closes issue #495) -8.9.0 ------ +## 8.9.0 - Fix broken cache (PR #494) - Add source name and type information based on static mapping of the reverse DNS base domain @@ -241,14 +204,12 @@ Removed improper spaces from `base_reverse_dns_map.csv` (Closes #612) - Replace `multiprocessing.Pool` with `Pipe` + `Process` (PR #491 closes issue #489) - Remove unused parallel arguments (PR #492 closes issue #490) -8.8.0 ------ +## 8.8.0 - Add support for OpenSearch (PR #481 closes #480) - Fix SMTP TLS reporting to Elasticsearch (PR #470) -8.7.0 ------ +## 8.7.0 - Add support for SMTP TLS reports (PR #453 closes issue #71) - Do not replace content in forensic samples (fix #403) @@ -262,99 +223,83 @@ Removed improper spaces from `base_reverse_dns_map.csv` (Closes #612) - Add Gmail pagination (PR #469) - Use the correct `msgraph` scope (PR #471) -8.6.4 ------ +## 8.6.4 - Properly process aggregate reports that incorrectly call `identifiers` `identities` - Ignore SPF results in aggregate report records if the domain is not provided -8.6.3 ------ +## 8.6.3 - Add an error message instead of raising an exception when an aggregate report time span is greater than 24 hours -8.6.2 ------ +## 8.6.2 - Use `zlib` instead of `Gzip` to decompress more `.gz` files, including the ones supplied by Mimecast (Based on #430 closes #429) -8.6.1 ------ +## 8.6.1 - Fix handling of non-domain organization names (PR #411 fixes issue #410) - Skip processing of aggregate reports with a date range that is too long to be valid (PR #408 fixes issue #282) - Better error handling for Elasticsearch queries and file parsing (PR #417) -8.6.0 ------ +## 8.6.0 - Replace publicsuffix2 with publicsuffixlist -8.5.0 ------ +## 8.5.0 - Add support for Azure Log Analytics (PR #394) - Fix a bug in the Microsoft Graph integration that caused a crash when an inbox has 10+ folders (PR #398) - Documentation fixes -8.4.2 ------ +## 8.4.2 - Only initialize the syslog, S3 and Kafka clients once (PR #386 closes issues #289 and #380) -8.4.1 ------ +## 8.4.1 - Fix bug introduced in 8.3.1 that caused `No such file or directory` errors if output files didn't exist (PR #385 closes issues #358 and #382) - Make the `--silent` CLI option only print errors. Add the `--warnings` options to also print warnings (PR #383) -8.4.0 ------ +## 8.4.0 - Provide a warning when no file is located at the path specified by the `ip_db_path` option (based on PR #369 with improvements in grammar) - Add `allow_unencrypted_storage` to possible `msgraph` settings. See documentation for details. (PR #375) - Use the `check_timeout` value in the event of an IMAP connection error, instead of a static 5 second value (PR #377) - Update the included DBIP IP to Country Lite database to the December 2022 release -8.3.2 ------ +## 8.3.2 - Improvements to the Microsoft Graph integration (PR #352) -8.3.1 ------ +## 8.3.1 - Handle unexpected XML parsing errors more gracefully (PR #349) - Migrate build from `setuptools` to `hatch` -8.3.0 ------ +## 8.3.0 - Support MFA for Microsoft Graph (PR #320 closes issue #319) - Add more options for S3 export (PR #328) - Provide a helpful error message when the log file cannot be created (closes issue #317) -8.2.0 ------ +## 8.2.0 - Support non-standard, text-based forensic reports sent by some mail hosts - Set forensic report version to `None` (`null` in JSON) if the report was in a non-standard format and/or is missing a version number - The default value of the `mailbox` `batch_size` option is now `10` (use `0` for no limit) -8.1.1 ------ +## 8.1.1 - Fix marking messages as read via Microsoft Graph -8.1.0 ------ +## 8.1.0 - Restore compatibility with <8.0.0 configuration files (with deprecation warnings) - Set default `reports_folder` to `Inbox` (rather than `INBOX`) when `msgraph` is configured - Mark a message as read when fetching a message from Microsoft Graph -8.0.3 ------ +## 8.0.3 - Fix IMAP callback for `IDLE` connections (PR #313 closes issue #311) - Add warnings in documentation and log output for IMAP configuration changes introduced in 8.0.0 (Closes issue #309) @@ -362,18 +307,15 @@ Removed improper spaces from `base_reverse_dns_map.csv` (Closes #612) - Separate version numbers in `__init__.py` and `setup.py` to allow `pip` to install directly from `git` - Update `dateparser` to 1.1.1 (closes issue #273) -8.0.2 (yanked) --------------- +## 8.0.2 (yanked) - Strip leading and trailing whitespaces from Gmail scopes (Closes issue #310) -8.0.1 (yanked) --------------- +## 8.0.1 (yanked) - Fix `ModuleNotFoundError` by adding `parsedmarc.mail` to the list of packages in `setup.py` (PR #308) -8.0.0 (yanked) --------------- +## 8.0.0 (yanked) - Update included copy of `dbip-country-lite.mmdb` to the 2022-04 release - Add support for Microsoft/Office 365 via Microsoft Graph API (PR #301 closes issue #111) @@ -382,14 +324,12 @@ Removed improper spaces from `base_reverse_dns_map.csv` (Closes #612) - Remove usage of `logging.basicConfig` (PR #285) - Add support for the Gmail API (PR #284 and PR #307 close issue #96) -7.1.1 ------ +## 7.1.1 - Actually include `dbip-country-lite.mmdb` file in the `parsedmarc.resources` package (PR #281) - Update `dbip-country-lite.mmdb` to the 2022-01 release -7.1.0 ------ +## 7.1.0 - A static copy of the DBIP Country Lite database is now included for use when a copy of the MaxMind GeoLite2 Country database is not installed (Closes #275) - Add `ip_db_path` to as a parameter and `general` setting for a custom IP geolocation database location (Closes #184) @@ -400,13 +340,11 @@ Removed improper spaces from `base_reverse_dns_map.csv` (Closes #612) - Add support for syslog output (PR #263 closes #227) - Do not print TQDDM progress bar when running in a no-interactive TTY (PR #264) -7.0.1 ------ +## 7.0.1 - Fix startup error (PR #254) -7.0.0 ------ +## 7.0.0 - Fix issue #221: Crash when handling invalid reports without root node (PR #248) - Use UTC datetime objects for Elasticsearch output (PR #245) @@ -418,8 +356,7 @@ Removed improper spaces from `base_reverse_dns_map.csv` (Closes #612) - Wait for 5 seconds before attempting to reconnect to an IMAP server (PR #217) - Add option to process messages in batches (PR #222) -6.12.0 ------- +## 6.12.0 - Limit output filename length to 100 characters (PR #199) - Add basic auth support for Elasticsearch (PR #191) @@ -429,8 +366,7 @@ Removed improper spaces from `base_reverse_dns_map.csv` (Closes #612) - Require `dnspython>=2.0.0` - Drop Python 3.5 support -6.11.0 ------- +## 6.11.0 - Fix parsing failure for some valid forensic reports (PR #170) - Fix double count of messages in the Grafana dashboard (PR #182) @@ -439,8 +375,7 @@ Removed improper spaces from `base_reverse_dns_map.csv` (Closes #612) - Fix IMAP debugging output - Fix `User-Agent` string -6.10.0 ------- +## 6.10.0 - Ignore unknown forensic report fields when generating CSVs (Closes issue #148) - Fix crash on IMAP timeout (PR #164 - closes issue #163) @@ -448,89 +383,74 @@ Removed improper spaces from `base_reverse_dns_map.csv` (Closes #612) - Add support for Elasticsearch 7.0 (PR #161 - closes issue #149) - Remove temporary workaround for DMARC aggregate report records missing a SPF domain fields -6.9.0 ------ +## 6.9.0 - Use system nameservers instead of Cloudflare by default - Parse aggregate report records with missing SPF domains -6.8.2 ------ +## 6.8.2 - Require `mailsuite>=1.5.4` -6.8.1 ------ +## 6.8.1 - Use `match_phrase` instead of `match` when looking for existing strings in Elasticsearch -6.8.0 ------ +## 6.8.0 - Display warning when `GeoLite2-Country.mmdb` is missing, instead of trying to download it - Add documentation for MaxMind `geoipupdate` changes on January 30th, 2019 (closes issues #137 and #139) - Require `mail-parser>=3.11.0` -6.7.4 ------ +## 6.7.4 - Update dependencies -6.7.3 ------ +## 6.7.3 - Make `dkim_aligned` and `spf_aligned` case-insensitive (PR #132) -6.7.2 ------ +## 6.7.2 - Fix SPF results field in CSV output (closes issue #128) -6.7.1 ------ +## 6.7.1 - Parse forensic email samples with non-standard date headers - Graceful handling of a failure to download the GeoIP database (issue #123) -6.7.0 ------ +## 6.7.0 - Fix typos (PR #119) - Make CSV output match JSON output (Issue # 22) - Graceful processing of invalid aggregate DMARC reports (PR #122) - Remove Python 3.4 support -6.6.1 ------ +## 6.6.1 - Close files after reading them -6.6.0 ------ +## 6.6.0 - Set a configurable default IMAP timeout of 30 seconds - Set a configurable maximum of 4 IMAP timeout retry attempts - Add support for reading ``MBOX`` files - Set a configurable Elasticsearch timeout of 60 seconds -6.5.5 ------ +## 6.5.5 - Set minimum `publicsuffix2` version -6.5.4 ------ +## 6.5.4 - Bump required `mailsuite` version to `1.2.1` -6.5.3 ------ +## 6.5.3 - Fix typos in the CLI documentation - Bump required `mailsuite` version to `1.1.1` -6.5.2 ------ +## 6.5.2 - Merge PR #100 from michaeldavie - Correct a bug introduced in 6.5.1 that caused only the last record's data @@ -542,8 +462,7 @@ Removed improper spaces from `base_reverse_dns_map.csv` (Closes #612) hierarchy separators - Prepend the namespace to the folder path when required -6.5.1 ------ +## 6.5.1 - Merge PR #98 from michaeldavie - Add functions @@ -551,8 +470,7 @@ Removed improper spaces from `base_reverse_dns_map.csv` (Closes #612) - `parsed_forensic_reports_to_csv_row(reports)` - Require `dnspython>=1.16.0` -6.5.0 ------ +## 6.5.0 - Move mail processing functions to the [`mailsuite`](https://seanthegeek.github.io/mailsuite/) package @@ -562,8 +480,7 @@ Removed improper spaces from `base_reverse_dns_map.csv` (Closes #612) - Log the current file path being processed when `--debug` is used (closes issue #95) -6.4.2 ------ +## 6.4.2 - Do not attempt to convert `org_name` to a base domain if `org_name` contains a space (closes issue #94) @@ -571,58 +488,48 @@ Removed improper spaces from `base_reverse_dns_map.csv` (Closes #612) - Provide a more helpful warning message when `GeoLite2-Country.mmdb` is missing -6.4.1 ------ +## 6.4.1 - Raise `utils.DownloadError` exception when a GeoIP database or Public Suffix List (PSL) download fails (closes issue #73) -6.4.0 ------ +## 6.4.0 - Add ``number_of_shards`` and ``number_of_replicas`` as possible options in the ``elasticsearch`` configuration file section (closes issue #78) -6.3.7 ------ +## 6.3.7 - Work around some unexpected IMAP responses reported in issue #75 -6.3.6 ------ +## 6.3.6 - Work around some unexpected IMAP responses reported in issue #70 - Show correct destination folder in debug logs when moving aggregate reports -6.3.5 ------ +## 6.3.5 - Normalize `Delivery-Result` value in forensic/failure reports (issue #76) Thanks Freddie Leeman of URIports for the troubleshooting assistance -6.3.4 ------ +## 6.3.4 - Fix Elasticsearch index creation (closes issue #74) -6.3.3 ------ +## 6.3.3 - Set `number_of_shards` and `number_of_replicas` to `1` when creating indexes - Fix dependency conflict -6.3.2 ------ +## 6.3.2 - Fix the `monthly_indexes` option in the `elasticsearch` configuration section -6.3.1 ------ +## 6.3.1 - Fix `strip_attachment_payloads` option -6.3.0 ------ +## 6.3.0 - Fix IMAP IDLE response processing for some mail servers (#67) - Exit with a critical error when required settings are missing (#68) @@ -633,109 +540,90 @@ in the ``elasticsearch`` configuration file section (closes issue #78) - Suppress `mailparser` logging output - Suppress `msgconvert` warnings -6.2.2 ------ +## 6.2.2 - Fix crash when trying to save forensic reports with missing fields to Elasticsearch -6.2.1 ------ +## 6.2.1 - Add missing `tqdm` dependency to `setup.py` -6.2.0 ------ +## 6.2.0 - Add support for multiprocess parallelized processing via CLI (Thanks zscholl - PR #62) - Save sha256 hashes of attachments in forensic samples to Elasticsearch -6.1.8 ------ +## 6.1.8 - Actually fix GeoIP lookups -6.1.7 ------ +## 6.1.7 - Fix GeoIP lookups -6.1.6 ------ +## 6.1.6 - Better GeoIP error handling -6.1.5 ------ +## 6.1.5 - Always use Cloudflare's nameservers by default instead of Google's - Avoid re-downloading the Geolite2 database (and tripping their DDoS protection) - Add `geoipupdate` to install instructions -6.1.4 ------ +## 6.1.4 - Actually package requirements -6.1.3 ------ +## 6.1.3 - Fix package requirements -6.1.2 ------ +## 6.1.2 - Use local Public Suffix List file instead of downloading it - Fix argument name for `send_email()` (closes issue #60) -6.1.1 ------ +## 6.1.1 - Fix aggregate report processing - Check for the existence of a configuration file if a path is supplied - Replace `publicsuffix` with `publicsuffix2` - Add minimum versions to requirements -6.1.0 ------ +## 6.1.0 - Fix aggregate report email parsing regression introduced in 6.0.3 (closes issue #57) - Fix Davmail support (closes issue #56) -6.0.3 ------ +## 6.0.3 - Don't assume the report is the last part of the email message (issue #55) -6.0.2 ----- +## 6.0.2 - IMAP connectivity improvements (issue #53) - Use a temp directory for temp files (issue #54) -6.0.1 ------ +## 6.0.1 - Fix Elasticsearch output (PR #50 - andrewmcgilvray) -6.0.0 ------ +## 6.0.0 - Move options from CLI to a config file (see updated installation documentation) - Refactoring to make argument names consistent -5.3.0 ------ +## 5.3.0 - Fix crash on invalid forensic report sample (Issue #47) - Fix DavMail support (Issue #45) -5.2.1 ------ +## 5.2.1 - Remove unnecessary debugging code -5.2.0 ------ +## 5.2.0 - Add filename and line number to logging output - Improved IMAP error handling @@ -752,44 +640,37 @@ in the ``elasticsearch`` configuration file section (closes issue #78) --log-file LOG_FILE output logging to a file ``` -5.1.3 ------ +## 5.1.3 - Remove `urllib3` version upper limit -5.1.2 ------ +## 5.1.2 - Workaround unexpected Office 365/Exchange IMAP responses -5.1.1 ------ +## 5.1.1 - Bugfix: Crash when parsing invalid forensic report samples (#38) - Bugfix: Crash when IMAP connection is lost - Increase default Splunk HEC response timeout to 60 seconds -5.1.0 ------ +## 5.1.0 - Bugfix: Submit aggregate dates to Elasticsearch as lists, not tuples - Support `elasticsearch-dsl<=6.3.0` - Add support for TLS/SSL and username/password auth to Kafka -5.0.2 ------ +## 5.0.2 - Revert to using `publicsuffix` instead of `publicsuffix2` -5.0.1 ------ +## 5.0.1 - Use `publixsuffix2` (closes issue #4) - Add Elasticsearch to automated testing - Lock `elasticsearch-dsl` required version to `6.2.1` (closes issue #25) -5.0.0 ------ +## 5.0.0 **Note**: Re-importing `kibana_saved_objects.json` in Kibana [is required](https://domainaware.github.io/parsedmarc/#upgrading-kibana-index-patterns) when upgrading to this version! @@ -805,26 +686,22 @@ forensic/failure/ruf report - Save data in separate indexes each day to make managing data retention easier - Cache DNS queries in memory -4.4.1 ------ +## 4.4.1 - Don't crash if Elasticsearch returns an unexpected result (workaround for issue #31) -4.4.0 ------ +## 4.4.0 - Packaging fixes -4.3.9 ------ +## 4.3.9 - Kafka output improvements - Moved some key values (`report_id`, `org_email`, `org_name`) higher in the JSON structure - Recreated the `date_range` values from the ES client for easier parsing. - Started sending individual record slices. Kafka default message size is 1 MB, some aggregate reports were exceeding this. Now it appends meta-data and sends record by record. -4.3.8 ------ +## 4.3.8 - Fix decoding of attachments inside forensic samples - Add CLI option `--imap-skip-certificate-verification` @@ -832,24 +709,20 @@ forensic/failure/ruf report and `watch_inbox()` - Debug logging improvements -4.3.7 ------ +## 4.3.7 - When checking an inbox, always recheck for messages when processing is complete -4.3.6 ------ +## 4.3.6 - Be more forgiving for forensic reports with missing fields -4.3.5 ------ +## 4.3.5 - Fix base64 attachment decoding (#26) -4.3.4 ------ +## 4.3.4 - Fix crash on empty aggregate report comments (brakhane - #25) - Add SHA256 hashes of attachments to output @@ -857,24 +730,20 @@ complete `--strip-attachment-payloads` option to the CLI (#23) - Set `urllib3` version requirements to match `requests` -4.3.3 ------ +## 4.3.3 - Fix forensic report email processing -4.3.2 ------ +## 4.3.2 - Fix normalization of the forensic sample from address -4.3.1 ------ +## 4.3.1 - Fix parsing of some emails - Fix duplicate forensic report search for Elasticsearch -4.3.0 ------ +## 4.3.0 - Fix bug where `parsedmarc` would always try to save to Elastic search, even if only `--hec` was used @@ -893,37 +762,31 @@ complete - `human_timestamp_to_timestamp(human_timestamp)` - `parse_email(data)` -4.2.0 ------- +## 4.2.0 - Save each aggregate report record as a separate Splunk event - Fix IMAP delete action (#20) - Suppress Splunk SSL validation warnings - Change default logging level to `WARNING` -4.1.9 ------ +## 4.1.9 - Workaround for forensic/ruf reports that are missing `Arrival-Date` and/or `Reported-Domain` -4.1.8 ------ +## 4.1.8 - Be more forgiving of weird XML -4.1.7 ------ +## 4.1.7 - Remove any invalid XML schema tags before parsing the XML (#18) -4.1.6 ------ +## 4.1.6 - Fix typo in CLI parser -4.1.5 ------ +## 4.1.5 - Only move or delete IMAP emails after they all have been parsed - Move/delete messages one at a time - do not exit on error @@ -931,49 +794,41 @@ complete `get_dmarc_reports_from_inbox()` - Add`--imap-port` and `--imap-no-ssl` CLI options -4.1.4 ------ +## 4.1.4 - Change default logging level to `ERROR` -4.1.3 ------ +## 4.1.3 - Fix crash introduced in 4.1.0 when creating Elasticsearch indexes (Issue #15) -4.1.2 ------ +## 4.1.2 - Fix packaging bug -4.1.1 ------ +## 4.1.1 - Add splunk instructions - Reconnect reset IMAP connections when watching a folder -4.1.0 ------ +## 4.1.0 - Add options for Elasticsearch prefixes and suffixes - If an aggregate report has the invalid `disposition` value `pass`, change it to `none` -4.0.2 ------ +## 4.0.2 - Use report timestamps for Splunk timestamps -4.0.1 ------ +## 4.0.1 - When saving aggregate reports in Elasticsearch store `domain` in `published_policy` - Rename `policy_published` to `published_policy`when saving aggregate reports to Splunk -4.0.0 ------ +## 4.0.0 - Add support for sending DMARC reports to a Splunk HTTP Events Collector (HEC) @@ -986,51 +841,44 @@ system path - Add `--outgoing-port` and `--outgoing-ssl` options - Fall back to plain text SMTP if `--outgoing-ssl` is not used and `STARTTLS` is not supported by the server -- Always use `\n` as the newline when generating CSVs +- Always use ` +` as the newline when generating CSVs - Workaround for random Exchange/Office 365 `Server Unavailable` IMAP errors -3.9.7 ------ +## 3.9.7 - Completely reset IMAP connection when a broken pipe is encountered -3.9.6 ------ +## 3.9.6 - Finish incomplete broken pipe fix -3.9.5 ------ +## 3.9.5 - Refactor to use a shared IMAP connection for inbox watching and message downloads - Gracefully recover from broken pipes in IMAP -3.9.4 ------ +## 3.9.4 - Fix moving/deleting emails -3.9.3 ------ +## 3.9.3 - Fix crash when forensic reports are missing `Arrival-Date` -3.9.2 ------ +## 3.9.2 - Fix PEP 8 spacing - Update build script to fail when CI tests fail -3.9.1 ------ +## 3.9.1 - Use `COPY` and delete if an IMAP server does not support `MOVE` (closes issue #9) -3.9.0 ------ +## 3.9.0 - Reduce IMAP `IDLE` refresh rate to 5 minutes to avoid session timeouts in Gmail @@ -1039,59 +887,49 @@ Gmail - Fix example NGINX configuration in the installation documentation (closes issue #6) -3.8.2 ------ +## 3.8.2 - Fix `nameservers` option (mikesiegel) - Move or delete invalid report emails in an IMAP inbox (closes issue #7) -3.8.1 ------ +## 3.8.1 - Better handling of `.msg` files when `msgconvert` is not installed -3.8.0 ------ +## 3.8.0 - Use `.` instead of `/` as the IMAP folder hierarchy separator when `/` does not work - fixes dovecot support (#5) - Fix parsing of base64-encoded forensic report data -3.7.3 ------ +## 3.7.3 - Fix saving attachment from forensic sample to Elasticsearch -3.7.2 ------ +## 3.7.2 - Change uses of the `DocType` class to `Document`, to properly support `elasticsearch-dsl` `6.2.0` (this also fixes use in pypy) - Add documentation for installation under pypy -3.7.1 ------ +## 3.7.1 - Require `elasticsearch>=6.2.1,<7.0.0` and `elasticsearch-dsl>=6.2.1,<7.0.0` - Update for class changes in `elasticsearch-dsl` `6.2.0` -3.7.0 ------ +## 3.7.0 - Fix bug where PSL would be called before it was downloaded if the PSL was older than 24 Hours -3.6.1 ------ +## 3.6.1 - Parse aggregate reports with missing SPF domain -3.6.0 ------ +## 3.6.0 - Much more robust error handling -3.5.1 ------ +## 3.5.1 - Fix dashboard message counts for source IP addresses visualizations - Improve dashboard loading times @@ -1099,43 +937,36 @@ older than 24 Hours - Add country rankings to the dashboards - Fix crash when parsing report with empty -3.5.0 ------ +## 3.5.0 - Use Cloudflare's public DNS resolvers by default instead of Google's - Fix installation from virtualenv - Fix documentation typos -3.4.1 ------ +## 3.4.1 - Documentation fixes - Fix console output -3.4.0 ------ +## 3.4.0 - Maintain IMAP IDLE state when watching the inbox - The `-i`/`--idle` CLI option is now `-w`/`--watch` - Improved Exception handling and documentation -3.3.0 ------ +## 3.3.0 - Fix errors when saving to Elasticsearch -3.2.0 ------ +## 3.2.0 - Fix existing aggregate report error message -3.1.0 ------ +## 3.1.0 - Fix existing aggregate report query -3.0.0 ------ +## 3.0.0 New features @@ -1151,8 +982,7 @@ file extension - If an aggregate report's `org_name` is a FQDN, the base is used - Normalize aggregate report IDs -2.1.2 ------ +## 2.1.2 - Rename `parsed_dmarc_forensic_reports_to_csv()` to `parsed_forensic_reports_to_csv()` to match other functions @@ -1160,25 +990,21 @@ file extension `parsed_aggregate_reports_to_csv()` to match other functions - Use local time when generating the default email subject -2.1.1 ------ +## 2.1.1 - Documentation fixes -2.1.0 ------ +## 2.1.0 - Add `get_report_zip()` and `email_results()` - Add support for sending report emails via the command line -2.0.1 ------ +## 2.0.1 - Fix documentation - Remove Python 2 code -2.0.0 ------ +## 2.0.0 New features @@ -1193,33 +1019,27 @@ Changes - `-o`/`--output` option is now a path to an output directory, instead of an output file -1.1.0 ------ +## 1.1.0 - Add `extract_xml()` and `human_timestamp_to_datetime` methods -1.0.5 ------ +## 1.0.5 - Prefix public suffix and GeoIP2 database filenames with `.` - Properly format errors list in CSV output -1.0.3 ------ +## 1.0.3 - Fix documentation formatting -1.0.2 ------ +## 1.0.2 - Fix more packaging flaws -1.0.1 ------ +## 1.0.1 - Fix packaging flaw -1.0.0 ------ +## 1.0.0 - Initial release