# Changelog

## 9.0.10

- Support Python 3.14+

## 9.0.9

### Fixes

- Validate that a string is base64-encoded before trying to base64 decode it. (PRs #648 and #649)

## 9.0.8

### Fixes

- Fix logging configuration not propagating to child parser processes (#646).
- Update `mailsuite` dependency to `?=1.11.1` to solve issues with iCloud IMAP (#493).

## 9.0.7

## Fixes

- Fix IMAP `since` option (#PR 645 closes issues #581 and #643).

## 9.0.6

### Fixes

- Fix #638.
- Fix/clarify report extraction and parsing behavior for multiple input types (bytes, base64 strings, and file-like objects).
- Fix type mismatches that could cause runtime issues in SMTP emailing and CLI option handling.

### Improvements

- Improve type hints across the library (Pylance/Pyright friendliness) and reduce false-positive linter errors.
- Emails in Microsoft 365 are now marked read as they are read. This provides constancy with other mailbox types, and gives you a indication of when emails are being read as they are processed in batches. (Close #625)

### Compatibility / Dependencies

- Set Python requirement to `>=3.9,<3.14`.
- Bump `mailsuite` requirement to `>=1.11.0`.

## 9.0.5

## Fixes

- Fix report type detection introduced in `9.0.4`.

## 9.0.4 (Yanked)

### Fixes

- Fix saving reports to OpenSearch ([#637](https://github.com/domainaware/parsedmarc/issues/637))
- Fix parsing certain DMARC failure/forensic reports
- Some fixes to type hints (incomplete, but published as-is due to the above bugs)

## 9.0.3

### Fixes

- Set `requires-python` to `>=3.9, <3.14` to avoid [this bug](https://github.com/python/cpython/issues/142307)

## 9.0.2

## Improvements

- Type hinting is now used properly across the entire library. (#445)

## Fixes

- Decompress report files as needed when passed via the CLI.
- Fixed incomplete removal of the ability for `parsedmarc.utils.extract_report` to accept a file path directly in `8.15.0`.

## Breaking changes

This version of the library requires consumers to pass certain arguments as keyword-only. Internally, the API uses a bare `*` in the function signature. This is standard per [PEP 3102](https://peps.python.org/pep-3102/)  and as documented in the Python Language Reference.

## 9.0.1

### Fixes

- Allow multiple `records` for the same aggregate DMARC report in Elasticsearch and Opensearch

## 9.0.0 (yanked)

- Normalize aggregate DMARC report volumes when a report timespan exceeds 24 hours

## 8.19.1

- Ignore HTML content type in report email parsing (#626)

## 8.19.0

- Add multi-tenant support via an index-prefix domain mapping file
- PSL overrides so that services like AWS are correctly identified
- Additional improvements to report type detection
- Fix webhook timeout parsing (PR #623)
- Output to STDOUT when the new general config boolean `silent` is set to `False` (Close #614)
- Additional services added to `base_reverse_dns_map.csv`

## 8.18.9

- Complete fix for #687 and more robust report type detection

## 8.18.8

- Fix parsing emails with an uncompressed aggregate report attachment (Closes #607)
- Add `--no-prettify-json` CLI option (PR #617)

## 8.18.7

Removed improper spaces from  `base_reverse_dns_map.csv` (Closes #612)

## 8.18.6

- Fix since option to correctly work with weeks (PR #604)
- Add 183 entries to `base_reverse_dns_map.csv`
- Add 57 entries to `known_unknown_base_reverse_dns.txt`
- Check for invalid UTF-8 bytes in `base_reverse_dns_map.csv` at build
- Exclude unneeded items from the `parsedmarc.resources` module at build

## 8.18.5

- Fix CSV download

## 8.18.4

- Fix webhooks

## 8.18.3

- Move `__version__` to `parsedmarc.constants`
- Create a constant `USER_AGENT`
- Use the HTTP `User-Agent` header value `parsedmarc/version` for all HTTP requests

## 8.18.2

- Merged PR #603
  - Fixes issue #595 - CI test fails for Elasticsearch
    - Moved Elasticsearch to a separate Docker service container for CI testing
    - Dropped Python 3.8 from CI testing
  - Fixes lookup and saving of DMARC forensic reports in Elasticsearch and OpenSearch
- Updated fallback `base_reverse_dns_map.csv`, which now includes over 1,400 lines
- Updated included `dbip-country-lite.mmdb` to the June 2025 release
- Automatically fall back to the internal `base_reverse_dns_map.csv` if the received file is not valid (Fixes #602)
  - Print the received data to the debug log

## 8.18.1

- Add missing `https://` to the default Microsoft Graph URL

## 8.18.0

- Add support for Microsoft national clouds via Graph API base URL (PR #590)
- Avoid stopping processing when an invalid DMARC report is encountered (PR #587)
- Increase `http.client._MAXHEADERS` from `100` to `200` to avoid errors connecting to Elasticsearch/OpenSearch (PR #589)

## 8.17.0

- Ignore duplicate aggregate DMARC reports with the same `org_name` and `report_id` seen within the same hour (Fixes #535)
- Fix saving SMTP TLS reports to OpenSearch (PR #585 closed issue #576)
- Add 303 entries to `base_reverse_dns_map.csv`

## 8.16.1

- Failed attempt to ignore aggregate DMARC reports seen within a period of one hour (#535)

## 8.16.0

- Add a `since` option to only search for emails since a certain time (PR #527)

## 8.15.4

- Fix crash if aggregate report timespan is > 24 hours

## 8.15.3

- Ignore aggregate reports with a timespan of > 24 hours (Fixes #282)

## 8.15.2

- Require `mailsuite>=1.9.18`
  - Pins `mail-parser` version at `3.15.0` due to a parsing regression in mail-parser `4.0.0`
  - Parse aggregate reports with empty `<auth_results>`
  - Do not overwrite the log on each run (PR #569 fixes issue #565)

## 8.15.1

- Proper IMAP namespace fix (Closes issue #557 and issue #563)
  - Require `mailsuite>=1.9.17`
  - Revert PR #552
- Add pre-flight check for nameservers (PR #562 closes issue #543)
- Reformat code with `ruff`

## 8.15.0

- Fix processing of SMTP-TLS reports ([#549](https://github.com/domainaware/parsedmarc/issues/549)), which broke in commit [410663d](https://github.com/domainaware/parsedmarc/commit/410663dbcaba019ca3d3744946348b56a635480b)(PR [#530](https://github.com/domainaware/parsedmarc/pull/530))
  - This PR enforced a stricter check for base64-encoded strings, which SMTP TLS reports from Google did not pass
  - Removing the check introduced its own issue, because some file paths were treated as base64-encoded strings
- Create a separate `extract_report_from_file_path()` function for processioning reports based on a file path
- Remove report extraction based on a file path from `extract_report()`

## 8.14.2

- Update `base_reverse_dns_map.csv` to fix over-replacement on [`f3a5f10`](https://github.com/domainaware/parsedmarc/commit/f3a5f10d67b02c5db31ae1f7ced68028f46ca2a3) (PR #553)

## 8.14.1

- Failed attempt to fix processing of SMTP-TLS reports (#549)

## 8.14.0

- Skip invalid aggregate report rows without calling the whole report invalid
  - Some providers such as GoDaddy will send reports with some rows missing a source IP address, while other rows are fine
- Fix Dovecot support by using the separator provided by the IMAP namespace when possible (PR #552 closes #551)
- Only download `base_reverse_dns_map.csv` once (fixes #542)
- Update included `base_reverse_dns_map.csv`
  - Replace University category with Education to be more inclusive
- Update included `dbip-country-lite.mmdb`

## 8.13.0

- Add Elastic/OpenSearch index prefix option (PR #531 closes #159)
- Add GELF output support (PR #532)

## 8.12.0

- Fix for deadlock with large report (#508)
- Build: move to kafka-python-ng (#510)
- Fix new config variables previously not propagated in the code (#524)
- Fixes for kafka integration (#522)
- Fix if base_domain is None before get_service_from_reverse_dns_base_domain (#514)
- Update base_reverse_dns_map.csv

## 8.11.0

- Actually save `source_type` and `source_name` to Elasticsearch and OpenSearch
- Reverse-lookup cache improvements (PR #501 closes issue #498)
- Update the included `dbip-country-lite.mmdb` to the 2024-03 version
- Update `base_reverse_dns_map.csv`
- Add new general config options (closes issue #500)
  - `always_use_local_files` - Disables the download of the reverse DNS map
  - `local_reverse_dns_map_path` - Overrides the default local file path to use for the reverse DNS map
  - `reverse_dns_map_url` - Overrides the default download URL for the reverse DNS map

## 8.10.3

- Fix flaws in `base_reverse_dns_map.csv`

## 8.10.2

- Fix flaws in `base_reverse_dns_map.csv`

## 8.10.1

- Fix flaws in `base_reverse_dns_map.csv`

## 8.10.0

- Fix MSGraph UsernamePassword Authentication (PR #497)
- Attempt to download an updated `base_reverse_dns_map.csv` at runtime
- Update included `base_reverse_dns_map.csv`

## 8.9.4

- Update `base_reverse_dns_map.csv`

## 8.9.3

- Revert change in 8.9.2

## 8.9.2

- Use `Uncategorized` instead of `None` as the service type when a service cannot be identified

## 8.9.1

- Fix broken CLI by removing obsolete parameter from `cli_parse` call (PR #496 closes issue #495)

## 8.9.0

- Fix broken cache (PR #494)
- Add source name and type information based on static mapping of the reverse DNS base domain
  - See [this documentation](https://github.com/domainaware/parsedmarc/tree/master/parsedmarc/resources/maps) for more information, and to learn how to help!
- Replace `multiprocessing.Pool` with `Pipe` + `Process` (PR #491 closes issue #489)
- Remove unused parallel arguments (PR #492 closes issue #490)

## 8.8.0

- Add support for OpenSearch (PR #481 closes #480)
- Fix SMTP TLS reporting to Elasticsearch (PR #470)

## 8.7.0

- Add support for SMTP TLS reports (PR #453 closes issue #71)
- Do not replace content in forensic samples (fix #403)
- Pin `msgraph-core` dependency at version `0.2.2` until Microsoft provides better documentation (PR #466 Close [#464](https://github.com/domainaware/parsedmarc/issues/464))
- Properly handle base64-encoded email attachments (PR #453)
- Do not crash when attempting to parse invalid email content (PR #453)
- Ignore errors when parsing text-based forensic reports (PR #460)
- Add email date to email processing debug logs (PR #462)
- Set default batch size to 10 to match the documentation (PR #465)
- Properly handle none values (PR #468)
- Add Gmail pagination (PR #469)
- Use the correct `msgraph` scope (PR #471)

## 8.6.4

- Properly process aggregate reports that incorrectly call `identifiers` `identities`
- Ignore SPF results in aggregate report records if the domain is not provided

## 8.6.3

- Add an error message instead of raising an exception when an aggregate report time span is greater than 24 hours

## 8.6.2

- Use `zlib` instead of `Gzip` to decompress more `.gz` files, including the ones supplied by Mimecast (Based on #430 closes #429)

## 8.6.1

- Fix handling of non-domain organization names (PR #411 fixes issue #410)
- Skip processing of aggregate reports with a date range that is too long to be valid (PR #408 fixes issue #282)
- Better error handling for Elasticsearch queries and file parsing (PR #417)

## 8.6.0

- Replace publicsuffix2 with publicsuffixlist

## 8.5.0

- Add support for Azure Log Analytics (PR #394)
- Fix a bug in the Microsoft Graph integration that caused a crash when an inbox has 10+ folders (PR #398)
- Documentation fixes

## 8.4.2

- Only initialize the syslog, S3 and Kafka clients once (PR #386 closes issues #289 and #380)

## 8.4.1

- Fix bug introduced in 8.3.1 that caused `No such file or directory` errors if output files didn't exist (PR #385 closes issues #358 and #382)
- Make the `--silent` CLI option only print errors. Add the `--warnings` options to also print warnings (PR #383)

## 8.4.0

- Provide a warning when no file is located at the path specified by the `ip_db_path` option (based on PR #369 with improvements in grammar)
- Add `allow_unencrypted_storage` to possible `msgraph` settings. See documentation for details. (PR #375)
- Use the `check_timeout` value in the event of an IMAP connection error, instead of a static 5 second value (PR #377)
- Update the included DBIP IP to Country Lite database to the December 2022 release

## 8.3.2

- Improvements to the Microsoft Graph integration (PR #352)

## 8.3.1

- Handle unexpected XML parsing errors more gracefully (PR #349)
- Migrate build from `setuptools` to `hatch`

## 8.3.0

- Support MFA for Microsoft Graph (PR #320 closes issue #319)
- Add more options for S3 export (PR #328)
- Provide a helpful error message when the log file cannot be created (closes issue #317)

## 8.2.0

- Support non-standard, text-based forensic reports sent by some mail hosts
- Set forensic report version to `None` (`null` in JSON) if the report was in a non-standard format and/or is missing a version number
- The default value of the `mailbox` `batch_size` option is now `10` (use `0` for no limit)

## 8.1.1

- Fix marking messages as read via Microsoft Graph

## 8.1.0

- Restore compatibility with <8.0.0 configuration files (with deprecation warnings)
- Set default `reports_folder` to `Inbox` (rather than `INBOX`) when `msgraph` is configured
- Mark a message as read when fetching a message from Microsoft Graph

## 8.0.3

- Fix IMAP callback for `IDLE` connections (PR #313 closes issue #311)
- Add warnings in documentation and log output for IMAP configuration changes introduced in 8.0.0 (Closes issue #309)
- Actually pin the `elasticsearch` Python library version at `<7.14.0` (Closes issue #315)
- Separate version numbers in `__init__.py` and `setup.py` to allow `pip` to install directly from `git`
- Update `dateparser` to 1.1.1 (closes issue #273)

## 8.0.2 (yanked)

- Strip leading and trailing whitespaces from Gmail scopes (Closes issue #310)

## 8.0.1 (yanked)

- Fix `ModuleNotFoundError` by adding `parsedmarc.mail` to the list of packages in `setup.py` (PR #308)

## 8.0.0 (yanked)

- Update included copy of `dbip-country-lite.mmdb` to the 2022-04 release
- Add support for Microsoft/Office 365 via Microsoft Graph API (PR #301 closes issue #111)
- Pin `elasticsearch-dsl` version at `>=7.2.0<7.14.0` (PR #297  closes issue #296)
- Properly initialize `ip_dp_path` (PR #294 closes issue #286)
- Remove usage of `logging.basicConfig` (PR #285)
- Add support for the Gmail API (PR #284 and PR #307 close issue #96)

## 7.1.1

- Actually include `dbip-country-lite.mmdb` file in the `parsedmarc.resources` package (PR #281)
- Update `dbip-country-lite.mmdb` to the 2022-01 release

## 7.1.0

- A static copy of the DBIP Country Lite database is now included for use when a copy of the MaxMind GeoLite2 Country database is not installed (Closes #275)
- Add `ip_db_path` to as a parameter and `general` setting for a custom IP geolocation database location (Closes #184)
- Search default Homebrew path when searching for a copy of the MaxMind GeoLite2 Country database (Closes #272)
- Fix log messages written to root logger (PR #276)
- Fix `--offline` option in CLI not being passed as a boolean (PR #265)
- Set Elasticsearch shard replication to `0` (PR #274)
- Add support for syslog output (PR #263 closes #227)
- Do not print TQDDM progress bar when running in a no-interactive TTY (PR #264)

## 7.0.1

- Fix startup error (PR #254)

## 7.0.0

- Fix issue #221: Crash when handling invalid reports without root node (PR #248)
- Use UTC datetime objects for Elasticsearch output (PR #245)
- Fix issues #219, #155, and #103: IMAP connections break on large emails (PR #241)
- Add support for saving reports to S3 buckets (PR #223)
- Pass `offline` parameter to `wait_inbox()` (PR #216)
- Add more details to logging (PR #220)
- Add options customizing the names of output files (Modifications based on PR #225)
- Wait for 5 seconds before attempting to reconnect to an IMAP server (PR #217)
- Add option to process messages in batches (PR #222)

## 6.12.0

- Limit output filename length to 100 characters (PR #199)
- Add basic auth support for Elasticsearch (PR #191)
- Fix Windows paths when searching for the GeoIP database (PR #190)
- Remove `six` requirement
- Require `mailsuite>=1.6.1`
- Require `dnspython>=2.0.0`
  - Drop Python 3.5 support

## 6.11.0

- Fix parsing failure for some valid forensic reports (PR #170)
- Fix double count of messages in the Grafana dashboard (PR #182)
- Add begin and end date fields for aggregate DMARC reports in Elasticsearch (PR #183 fixes issue #162)
- Fix crash on IMAP timeout (PR #186 fixes issue #163)
- Fix IMAP debugging output
- Fix `User-Agent` string

## 6.10.0

- Ignore unknown forensic report fields when generating CSVs (Closes issue #148)
- Fix crash on IMAP timeout (PR #164 - closes issue #163)
- Use SMTP port from the config file when sending emails (PR #151)
- Add support for Elasticsearch 7.0 (PR #161 - closes issue #149)
- Remove temporary workaround for DMARC aggregate report records missing a SPF domain fields

## 6.9.0

- Use system nameservers instead of Cloudflare by default
- Parse aggregate report records with missing SPF domains

## 6.8.2

- Require `mailsuite>=1.5.4`

## 6.8.1

- Use `match_phrase` instead of `match` when looking for existing strings in Elasticsearch

## 6.8.0

- Display warning when `GeoLite2-Country.mmdb` is missing, instead of trying to download it
- Add documentation for MaxMind `geoipupdate` changes on January 30th, 2019 (closes issues #137 and #139)
- Require `mail-parser>=3.11.0`

## 6.7.4

- Update dependencies

## 6.7.3

- Make `dkim_aligned` and `spf_aligned` case-insensitive (PR #132)

## 6.7.2

- Fix SPF results field in CSV output (closes issue #128)

## 6.7.1

- Parse forensic email samples with non-standard date headers
- Graceful handling of a failure to download the GeoIP database (issue #123)

## 6.7.0

- Fix typos (PR #119)
- Make CSV output match JSON output (Issue # 22)
- Graceful processing of invalid aggregate DMARC reports (PR #122)
- Remove Python 3.4 support

## 6.6.1

- Close files after reading them

## 6.6.0

- Set a configurable default IMAP timeout of 30 seconds
- Set a configurable maximum of 4 IMAP timeout retry attempts
- Add support for reading ``MBOX`` files
- Set a configurable Elasticsearch timeout of 60 seconds

## 6.5.5

- Set minimum `publicsuffix2` version

## 6.5.4

- Bump required `mailsuite` version to `1.2.1`

## 6.5.3

- Fix typos in the CLI documentation
- Bump required `mailsuite` version to `1.1.1`

## 6.5.2

- Merge PR #100 from michaeldavie
  - Correct a bug introduced in 6.5.1 that caused only the last record's data
  to be used for each row in an aggregate report's CSV version.
- Use `mailsuite` 1.1.0 to fix issues with some IMAP servers (closes issue 103)
  - Always use ``/`` as the folder hierarchy separator, and convert to the
  server's hierarchy separator in the background
  - Always remove folder name characters that conflict with the server's
  hierarchy separators
  - Prepend the namespace to the folder path when required

## 6.5.1

- Merge PR #98 from michaeldavie
  - Add functions
    - `parsed_aggregate_reports_to_csv_row(reports)`
    - `parsed_forensic_reports_to_csv_row(reports)`
- Require `dnspython>=1.16.0`

## 6.5.0

- Move mail processing functions to the
  [`mailsuite`](https://seanthegeek.github.io/mailsuite/) package
- Add offline option (closes issue #90)
- Use UDP instead of TCP, and properly set the timeout when querying DNS
  (closes issue #79 and #92)
- Log the current file path being processed when `--debug` is used
  (closes issue #95)

## 6.4.2

- Do not attempt to convert `org_name` to a base domain if `org_name` contains
  a space (closes issue #94)
- Always lowercase the `header_from`
- Provide a more helpful warning message when `GeoLite2-Country.mmdb` is
  missing

## 6.4.1

- Raise `utils.DownloadError` exception when a GeoIP database or Public
  Suffix List (PSL) download fails (closes issue #73)

## 6.4.0

- Add ``number_of_shards`` and ``number_of_replicas`` as possible options
in the ``elasticsearch`` configuration file section (closes issue #78)

## 6.3.7

- Work around some unexpected IMAP responses reported in issue #75

## 6.3.6

- Work around some unexpected IMAP responses reported in issue #70
- Show correct destination folder in debug logs when moving aggregate reports

## 6.3.5

- Normalize `Delivery-Result` value in forensic/failure reports (issue #76)
  Thanks Freddie Leeman of URIports for the troubleshooting assistance

## 6.3.4

- Fix Elasticsearch index creation (closes issue #74)

## 6.3.3

- Set `number_of_shards` and `number_of_replicas` to `1` when creating indexes
- Fix dependency conflict

## 6.3.2

- Fix the `monthly_indexes` option in the `elasticsearch` configuration section

## 6.3.1

- Fix `strip_attachment_payloads` option

## 6.3.0

- Fix IMAP IDLE response processing for some mail servers (#67)
- Exit with a critical error when required settings are missing (#68)
- XML parsing fixes (#69)
- Add IMAP responses to debug logging
- Add `smtp` option `skip_certificate_verification`
- Add `kafka` option `skip_certificate_verification`
- Suppress `mailparser` logging output
- Suppress `msgconvert` warnings

## 6.2.2

- Fix crash when trying to save forensic reports with missing fields to Elasticsearch

## 6.2.1

- Add missing `tqdm` dependency to `setup.py`

## 6.2.0

- Add support for multiprocess parallelized processing via CLI (Thanks zscholl - PR #62)
- Save sha256 hashes of attachments in forensic samples to Elasticsearch

## 6.1.8

- Actually fix GeoIP lookups

## 6.1.7

- Fix GeoIP lookups

## 6.1.6

- Better GeoIP error handling

## 6.1.5

- Always use Cloudflare's nameservers by default instead of Google's
- Avoid re-downloading the Geolite2 database (and tripping their DDoS protection)
- Add `geoipupdate` to install instructions

## 6.1.4

- Actually package requirements

## 6.1.3

- Fix package requirements

## 6.1.2

- Use local Public Suffix List file instead of downloading it
- Fix argument name for `send_email()` (closes issue #60)

## 6.1.1

- Fix aggregate report processing
- Check for the existence of a configuration file if a path is supplied
- Replace `publicsuffix` with `publicsuffix2`
- Add minimum versions to requirements

## 6.1.0

- Fix aggregate report email parsing regression introduced in 6.0.3 (closes issue #57)
- Fix Davmail support (closes issue #56)

## 6.0.3

- Don't assume the report is the last part of the email message (issue #55)

## 6.0.2

- IMAP connectivity improvements (issue #53)
- Use a temp directory for temp files (issue #54)

## 6.0.1

- Fix Elasticsearch output (PR #50 - andrewmcgilvray)

## 6.0.0

- Move options from CLI to a config file (see updated installation documentation)
- Refactoring to make argument names consistent

## 5.3.0

- Fix crash on invalid forensic report sample (Issue #47)
- Fix DavMail support (Issue #45)

## 5.2.1

- Remove unnecessary debugging code

## 5.2.0

- Add filename and line number to logging output
- Improved IMAP error handling
- Add CLI options

  ```text
  --elasticsearch-use-ssl
                        Use SSL when connecting to Elasticsearch
  --elasticsearch-ssl-cert-path ELASTICSEARCH_SSL_CERT_PATH
                        Path to the Elasticsearch SSL certificate
  --elasticsearch-monthly-indexes
                        Use monthly Elasticsearch indexes instead of daily
                        indexes
  --log-file LOG_FILE   output logging to a file
  ```

## 5.1.3

- Remove `urllib3` version upper limit

## 5.1.2

- Workaround unexpected Office 365/Exchange IMAP responses

## 5.1.1

- Bugfix: Crash when parsing invalid forensic report samples (#38)
- Bugfix: Crash when IMAP connection is lost
- Increase default Splunk HEC response timeout to 60 seconds

## 5.1.0

- Bugfix: Submit aggregate dates to Elasticsearch as lists, not tuples
- Support `elasticsearch-dsl<=6.3.0`
- Add support for TLS/SSL and username/password auth to Kafka

## 5.0.2

- Revert to using `publicsuffix` instead of `publicsuffix2`

## 5.0.1

- Use `publixsuffix2` (closes issue #4)
- Add Elasticsearch to automated testing
- Lock `elasticsearch-dsl` required version to `6.2.1` (closes issue #25)

## 5.0.0

**Note**: Re-importing `kibana_saved_objects.json` in Kibana [is required](https://domainaware.github.io/parsedmarc/#upgrading-kibana-index-patterns) when upgrading to this version!

- Bugfix: Reindex the aggregate report index field `published_policy.fo`
as `text` instead of `long` (Closes issue #31)
- Bugfix: IDLE email processing in Gmail/G-Suite accounts (closes issue #33)
- Bugfix: Fix inaccurate DNS timeout in CLI documentation (closes issue #34)
- Bugfix: Forensic report processing via CLI
- Bugfix: Duplicate aggregate report Elasticsearch query broken
- Bugfix: Crash when `Arrival-Date` header is missing in a
forensic/failure/ruf report
- IMAP reliability improvements
- Save data in separate indexes each day to make managing data retention easier
- Cache DNS queries in memory

## 4.4.1

- Don't crash if Elasticsearch returns an unexpected result (workaround for issue #31)

## 4.4.0

- Packaging fixes

## 4.3.9

- Kafka output improvements
  - Moved some key values (`report_id`, `org_email`, `org_name`) higher in the JSON structure
  - Recreated the `date_range` values from the ES client for easier parsing.
  - Started sending individual record slices. Kafka default message size is 1 MB, some aggregate reports were exceeding this. Now it appends meta-data and sends record by record.

## 4.3.8

- Fix decoding of attachments inside forensic samples
- Add CLI option `--imap-skip-certificate-verification`
- Add optional `ssl_context` argument for `get_dmarc_reports_from_inbox()`
and `watch_inbox()`
- Debug logging improvements

## 4.3.7

- When checking an inbox, always recheck for messages when processing is
complete

## 4.3.6

- Be more forgiving for forensic reports with missing fields

## 4.3.5

- Fix base64 attachment decoding (#26)

## 4.3.4

- Fix crash on empty aggregate report comments (brakhane - #25)
- Add SHA256 hashes of attachments to output
- Add `strip_attachment_payloads` option to functions and
`--strip-attachment-payloads` option to the CLI (#23)
- Set `urllib3` version requirements to match `requests`

## 4.3.3

- Fix forensic report email processing

## 4.3.2

- Fix normalization of the forensic sample from address

## 4.3.1

- Fix parsing of some emails
- Fix duplicate forensic report search for Elasticsearch

## 4.3.0

- Fix bug where `parsedmarc` would always try to save to Elastic search,
  even if only `--hec` was used
- Add options to save reports as a Kafka topic (mikesiegel  - #21)
- Major refactoring of functions
- Support parsing forensic reports generated by Brightmail
- Make `sample_headers_only` flag more reliable
- Functions that might be useful to other projects are now stored in
 `parsedmarc.utils`:
  - `get_base_domain(domain)`
  - `get_filename_safe_string(string)`
  - `get_ip_address_country(ip_address)`
  - `get_ip_address_info(ip_address, nameservers=None, timeout=2.0)`
  - `get_reverse_dns(ip_address, nameservers=None, timeout=2.0)`
  - `human_timestamp_to_datetime(human_timestamp)`
  - `human_timestamp_to_timestamp(human_timestamp)`
  - `parse_email(data)`

## 4.2.0

- Save each aggregate report record as a separate Splunk event
- Fix IMAP delete action (#20)
- Suppress Splunk SSL validation warnings
- Change default logging level to `WARNING`

## 4.1.9

- Workaround for forensic/ruf reports that are missing `Arrival-Date` and/or
`Reported-Domain`

## 4.1.8

- Be more forgiving of weird XML

## 4.1.7

- Remove any invalid XML schema tags before parsing the XML (#18)

## 4.1.6

- Fix typo in CLI parser

## 4.1.5

- Only move or delete IMAP emails after they all have been parsed
- Move/delete messages one at a time - do not exit on error
- Reconnect to IMAP if connection is broken during
`get_dmarc_reports_from_inbox()`
- Add`--imap-port` and `--imap-no-ssl` CLI options

## 4.1.4

- Change default logging level to `ERROR`

## 4.1.3

- Fix crash introduced in 4.1.0 when creating Elasticsearch indexes (Issue #15)

## 4.1.2

- Fix packaging bug

## 4.1.1

- Add splunk instructions
- Reconnect reset IMAP connections when watching a folder

## 4.1.0

- Add options for Elasticsearch prefixes and suffixes
- If an aggregate report has the invalid `disposition` value `pass`, change
it to `none`

## 4.0.2

- Use report timestamps for Splunk timestamps

## 4.0.1

- When saving aggregate reports in Elasticsearch store `domain` in
`published_policy`
- Rename `policy_published` to `published_policy`when saving aggregate
reports to Splunk

## 4.0.0

- Add support for sending DMARC reports to a Splunk HTTP Events
Collector (HEC)
- Use a browser-like `User-Agent` when downloading the Public Suffix List and
GeoIP DB to avoid being blocked by security proxies
- Reduce default DNS timeout to 2.0 seconds
- Add alignment booleans to JSON output
- Fix `.msg` parsing CLI exception when `msgconvert` is not found in the
system path
- Add `--outgoing-port` and  `--outgoing-ssl` options
- Fall back to plain text SMTP if `--outgoing-ssl` is not used and `STARTTLS`
is not supported by the server
- Always use `
` as the newline when generating CSVs
- Workaround for random Exchange/Office 365 `Server Unavailable` IMAP errors

## 3.9.7

- Completely reset IMAP connection when a broken pipe is encountered

## 3.9.6

- Finish incomplete broken pipe fix

## 3.9.5

- Refactor to use a shared IMAP connection for inbox watching and message
downloads

- Gracefully recover from broken pipes in IMAP

## 3.9.4

- Fix moving/deleting emails

## 3.9.3

- Fix crash when forensic reports are missing `Arrival-Date`

## 3.9.2

- Fix PEP 8 spacing
- Update build script to fail when CI tests fail

## 3.9.1

- Use `COPY` and delete if an IMAP server does not support `MOVE`
(closes issue #9)

## 3.9.0

- Reduce IMAP `IDLE` refresh rate to 5 minutes to avoid session timeouts in
Gmail
- Fix parsing of some forensic/failure/ruf reports
- Include email subject in all warning messages
- Fix example NGINX configuration in the installation documentation
(closes issue #6)

## 3.8.2

- Fix `nameservers` option (mikesiegel)
- Move or delete invalid report emails in an IMAP inbox (closes issue #7)

## 3.8.1

- Better handling of `.msg` files when `msgconvert` is not installed

## 3.8.0

- Use `.` instead of `/` as the IMAP folder hierarchy separator when `/`
does not work - fixes dovecot support (#5)
- Fix parsing of base64-encoded forensic report data

## 3.7.3

- Fix saving attachment from forensic sample to Elasticsearch

## 3.7.2

- Change uses of the `DocType` class to `Document`, to properly support `elasticsearch-dsl` `6.2.0` (this also fixes use in pypy)
- Add documentation for installation under pypy

## 3.7.1

- Require `elasticsearch>=6.2.1,<7.0.0` and `elasticsearch-dsl>=6.2.1,<7.0.0`
- Update for class changes in `elasticsearch-dsl` `6.2.0`

## 3.7.0

- Fix bug where PSL would be called before it was downloaded if the PSL was
older than 24 Hours

## 3.6.1

- Parse aggregate reports with missing SPF domain

## 3.6.0

- Much more robust error handling

## 3.5.1

- Fix dashboard message counts for source IP addresses visualizations
- Improve dashboard loading times
- Improve dashboard layout
- Add country rankings to the dashboards
- Fix crash when parsing report with empty <auth_results></auth_results>

## 3.5.0

- Use Cloudflare's public DNS resolvers by default instead of Google's
- Fix installation from virtualenv
- Fix documentation typos

## 3.4.1

- Documentation fixes
- Fix console output

## 3.4.0

- Maintain IMAP IDLE state when watching the inbox
- The `-i`/`--idle` CLI option is now `-w`/`--watch`
- Improved Exception handling and documentation

## 3.3.0

- Fix errors when saving to Elasticsearch

## 3.2.0

- Fix existing aggregate report error message

## 3.1.0

- Fix existing aggregate report query

## 3.0.0

New features

- Add option to select the IMAP folder where reports are stored
- Add options to send data to Elasticsearch

Changes

- Use Google's public nameservers (`8.8.8.8` and `4.4.4.4`)
by default
- Detect aggregate report email attachments by file content rather than
file extension
- If an aggregate report's `org_name` is a FQDN, the base is used
- Normalize aggregate report IDs

## 2.1.2

- Rename `parsed_dmarc_forensic_reports_to_csv()` to
 `parsed_forensic_reports_to_csv()` to match other functions
- Rename `parsed_aggregate_report_to_csv()` to
 `parsed_aggregate_reports_to_csv()` to match other functions
- Use local time when generating the default email subject

## 2.1.1

- Documentation fixes

## 2.1.0

- Add `get_report_zip()` and `email_results()`
- Add support for sending report emails via the command line

## 2.0.1

- Fix documentation
- Remove Python 2 code

## 2.0.0

New features

- Parse forensic reports
- Parse reports from IMAP inbox

Changes

- Drop support for Python 2
- Command line output is always a JSON object containing the lists
  `aggregate_reports` and `forensic_reports`
- `-o`/`--output` option is now a path to an output directory, instead of an
  output file

## 1.1.0

- Add `extract_xml()` and `human_timestamp_to_datetime` methods

## 1.0.5

- Prefix public suffix and GeoIP2 database filenames with `.`
- Properly format errors list in CSV output

## 1.0.3

- Fix documentation formatting

## 1.0.2

- Fix more packaging flaws

## 1.0.1

- Fix packaging flaw

## 1.0.0

- Initial release