mirror of https://github.com/domainaware/parsedmarc.git synced 2026-02-17 07:03:58 +00:00

Files

Sean Whalen 3d74cd6ac0 Update CHANGELOG with issue reference for email read status

Added a reference to issue #625 regarding email read status.

2025-12-29 12:10:19 -05:00

32 KiB

Raw Blame History

Changelog

9.0.6

Fixes

Fix #638.
Fix/clarify report extraction and parsing behavior for multiple input types (bytes, base64 strings, and file-like objects).
Fix type mismatches that could cause runtime issues in SMTP emailing and CLI option handling.

Improvements

Improve type hints across the library (Pylance/Pyright friendliness) and reduce false-positive linter errors.
Emails in Microsoft 365 are now marked read as they are read. This provides constancy with other mailbox types, and gives you a indication of when emails are being read as they are processed in batches. (Close #625)

Compatibility / Dependencies

Set Python requirement to >=3.9,<3.14.
Bump mailsuite requirement to >=1.11.0.

9.0.5 Fixes

Fix report type detection introduced in 9.0.4.

9.0.4 (Yanked)

Fixes

Fix saving reports to OpenSearch (#637)
Fix parsing certain DMARC failure/forensic reports
Some fixes to type hints (incomplete, but published as-is due to the above bugs)

9.0.3

Fixes

Set requires-python to >=3.9, <3.14 to avoid this bug

9.0.2 Improvements

Type hinting is now used properly across the entire library. (#445)

Fixes

Decompress report files as needed when passed via the CLI.
Fixed incomplete removal of the ability for parsedmarc.utils.extract_report to accept a file path directly in 8.15.0.

Breaking changes

This version of the library requires consumers to pass certain arguments as keyword-only. Internally, the API uses a bare * in the function signature. This is standard per PEP 3102 and as documented in the Python Language Reference.

9.0.1

Fixes

Allow multiple records for the same aggregate DMARC report in Elasticsearch and Opensearch

9.0.0 (yanked)

Normalize aggregate DMARC report volumes when a report timespan exceeds 24 hours

8.19.1

Ignore HTML content type in report email parsing (#626)

8.19.0

Add multi-tenant support via an index-prefix domain mapping file
PSL overrides so that services like AWS are correctly identified
Additional improvements to report type detection
Fix webhook timeout parsing (PR #623)
Output to STDOUT when the new general config boolean silent is set to False (Close #614)
Additional services added to base_reverse_dns_map.csv

8.18.9

Complete fix for #687 and more robust report type detection

8.18.8

Fix parsing emails with an uncompressed aggregate report attachment (Closes #607)
Add --no-prettify-json CLI option (PR #617)

8.18.7

Removed improper spaces from base_reverse_dns_map.csv (Closes #612)

8.18.6

Fix since option to correctly work with weeks (PR #604)
Add 183 entries to base_reverse_dns_map.csv
Add 57 entries to known_unknown_base_reverse_dns.txt
Check for invalid UTF-8 bytes in base_reverse_dns_map.csv at build
Exclude unneeded items from the parsedmarc.resources module at build

8.18.5

Fix CSV download

8.18.4

Fix webhooks

8.18.3

Move __version__ to parsedmarc.constants
Create a constant USER_AGENT
Use the HTTP User-Agent header value parsedmarc/version for all HTTP requests

8.18.2

Merged PR #603
- Fixes issue #595 - CI test fails for Elasticsearch
  - Moved Elasticsearch to a separate Docker service container for CI testing
  - Dropped Python 3.8 from CI testing
- Fixes lookup and saving of DMARC forensic reports in Elasticsearch and OpenSearch
Updated fallback base_reverse_dns_map.csv, which now includes over 1,400 lines
Updated included dbip-country-lite.mmdb to the June 2025 release
Automatically fall back to the internal base_reverse_dns_map.csv if the received file is not valid (Fixes #602)
- Print the received data to the debug log

8.18.1

Add missing https:// to the default Microsoft Graph URL

8.18.0

Add support for Microsoft national clouds via Graph API base URL (PR #590)
Avoid stopping processing when an invalid DMARC report is encountered (PR #587)
Increase http.client._MAXHEADERS from 100 to 200 to avoid errors connecting to Elasticsearch/OpenSearch (PR #589)

8.17.0

Ignore duplicate aggregate DMARC reports with the same org_name and report_id seen within the same hour (Fixes #535)
Fix saving SMTP TLS reports to OpenSearch (PR #585 closed issue #576)
Add 303 entries to base_reverse_dns_map.csv

8.16.1

Failed attempt to ignore aggregate DMARC reports seen within a period of one hour (#535)

8.16.0

Add a since option to only search for emails since a certain time (PR #527)

8.15.4

Fix crash if aggregate report timespan is > 24 hours

8.15.3

Ignore aggregate reports with a timespan of > 24 hours (Fixes #282)

8.15.2

Require mailsuite>=1.9.18
- Pins mail-parser version at 3.15.0 due to a parsing regression in mail-parser 4.0.0
- Parse aggregate reports with empty <auth_results>
- Do not overwrite the log on each run (PR #569 fixes issue #565)

8.15.1

Proper IMAP namespace fix (Closes issue #557 and issue #563)
- Require mailsuite>=1.9.17
- Revert PR #552
Add pre-flight check for nameservers (PR #562 closes issue #543)
Reformat code with ruff

8.15.0

Fix processing of SMTP-TLS reports (#549), which broke in commit 410663d(PR #530)
- This PR enforced a stricter check for base64-encoded strings, which SMTP TLS reports from Google did not pass
- Removing the check introduced its own issue, because some file paths were treated as base64-encoded strings
Create a separate extract_report_from_file_path() function for processioning reports based on a file path
Remove report extraction based on a file path from extract_report()

8.14.2

Update base_reverse_dns_map.csv to fix over-replacement on f3a5f10 (PR #553)

8.14.1

Failed attempt to fix processing of SMTP-TLS reports (#549)

8.14.0

Skip invalid aggregate report rows without calling the whole report invalid
- Some providers such as GoDaddy will send reports with some rows missing a source IP address, while other rows are fine
Fix Dovecot support by using the separator provided by the IMAP namespace when possible (PR #552 closes #551)
Only download base_reverse_dns_map.csv once (fixes #542)
Update included base_reverse_dns_map.csv
- Replace University category with Education to be more inclusive
Update included dbip-country-lite.mmdb

8.13.0

Add Elastic/OpenSearch index prefix option (PR #531 closes #159)
Add GELF output support (PR #532)

8.12.0

Fix for deadlock with large report (#508)
Build: move to kafka-python-ng (#510)
Fix new config variables previously not propagated in the code (#524)
Fixes for kafka integration (#522)
Fix if base_domain is None before get_service_from_reverse_dns_base_domain (#514)
Update base_reverse_dns_map.csv

8.11.0

Actually save source_type and source_name to Elasticsearch and OpenSearch
Reverse-lookup cache improvements (PR #501 closes issue #498)
Update the included dbip-country-lite.mmdb to the 2024-03 version
Update base_reverse_dns_map.csv
Add new general config options (closes issue #500)
- always_use_local_files - Disables the download of the reverse DNS map
- local_reverse_dns_map_path - Overrides the default local file path to use for the reverse DNS map
- reverse_dns_map_url - Overrides the default download URL for the reverse DNS map

8.10.3

Fix flaws in base_reverse_dns_map.csv

8.10.2

Fix flaws in base_reverse_dns_map.csv

8.10.1

Fix flaws in base_reverse_dns_map.csv

8.10.0

Fix MSGraph UsernamePassword Authentication (PR #497)
Attempt to download an updated base_reverse_dns_map.csv at runtime
Update included base_reverse_dns_map.csv

8.9.4

Update base_reverse_dns_map.csv

8.9.3

Revert change in 8.9.2

8.9.2

Use Uncategorized instead of None as the service type when a service cannot be identified

8.9.1

Fix broken CLI by removing obsolete parameter from cli_parse call (PR #496 closes issue #495)

8.9.0

Fix broken cache (PR #494)
Add source name and type information based on static mapping of the reverse DNS base domain
- See this documentation for more information, and to learn how to help!
Replace multiprocessing.Pool with Pipe + Process (PR #491 closes issue #489)
Remove unused parallel arguments (PR #492 closes issue #490)

8.8.0

Add support for OpenSearch (PR #481 closes #480)
Fix SMTP TLS reporting to Elasticsearch (PR #470)

8.7.0

Add support for SMTP TLS reports (PR #453 closes issue #71)
Do not replace content in forensic samples (fix #403)
Pin msgraph-core dependency at version 0.2.2 until Microsoft provides better documentation (PR #466 Close #464)
Properly handle base64-encoded email attachments (PR #453)
Do not crash when attempting to parse invalid email content (PR #453)
Ignore errors when parsing text-based forensic reports (PR #460)
Add email date to email processing debug logs (PR #462)
Set default batch size to 10 to match the documentation (PR #465)
Properly handle none values (PR #468)
Add Gmail pagination (PR #469)
Use the correct msgraph scope (PR #471)

8.6.4

Properly process aggregate reports that incorrectly call identifiers identities
Ignore SPF results in aggregate report records if the domain is not provided

8.6.3

Add an error message instead of raising an exception when an aggregate report time span is greater than 24 hours

8.6.2

Use zlib instead of Gzip to decompress more .gz files, including the ones supplied by Mimecast (Based on #430 closes #429)

8.6.1

Fix handling of non-domain organization names (PR #411 fixes issue #410)
Skip processing of aggregate reports with a date range that is too long to be valid (PR #408 fixes issue #282)
Better error handling for Elasticsearch queries and file parsing (PR #417)

8.6.0

Replace publicsuffix2 with publicsuffixlist

8.5.0

Add support for Azure Log Analytics (PR #394)
Fix a bug in the Microsoft Graph integration that caused a crash when an inbox has 10+ folders (PR #398)
Documentation fixes

8.4.2

Only initialize the syslog, S3 and Kafka clients once (PR #386 closes issues #289 and #380)

8.4.1

Fix bug introduced in 8.3.1 that caused No such file or directory errors if output files didn't exist (PR #385 closes issues #358 and #382)
Make the --silent CLI option only print errors. Add the --warnings options to also print warnings (PR #383)

8.4.0

Provide a warning when no file is located at the path specified by the ip_db_path option (based on PR #369 with improvements in grammar)
Add allow_unencrypted_storage to possible msgraph settings. See documentation for details. (PR #375)
Use the check_timeout value in the event of an IMAP connection error, instead of a static 5 second value (PR #377)
Update the included DBIP IP to Country Lite database to the December 2022 release

8.3.2

Improvements to the Microsoft Graph integration (PR #352)

8.3.1

Handle unexpected XML parsing errors more gracefully (PR #349)
Migrate build from setuptools to hatch

8.3.0

Support MFA for Microsoft Graph (PR #320 closes issue #319)
Add more options for S3 export (PR #328)
Provide a helpful error message when the log file cannot be created (closes issue #317)

8.2.0

Support non-standard, text-based forensic reports sent by some mail hosts
Set forensic report version to None (null in JSON) if the report was in a non-standard format and/or is missing a version number
The default value of the mailbox batch_size option is now 10 (use 0 for no limit)

8.1.1

Fix marking messages as read via Microsoft Graph

8.1.0

Restore compatibility with <8.0.0 configuration files (with deprecation warnings)
Set default reports_folder to Inbox (rather than INBOX) when msgraph is configured
Mark a message as read when fetching a message from Microsoft Graph

8.0.3

Fix IMAP callback for IDLE connections (PR #313 closes issue #311)
Add warnings in documentation and log output for IMAP configuration changes introduced in 8.0.0 (Closes issue #309)
Actually pin the elasticsearch Python library version at <7.14.0 (Closes issue #315)
Separate version numbers in __init__.py and setup.py to allow pip to install directly from git
Update dateparser to 1.1.1 (closes issue #273)

8.0.2 (yanked)

Strip leading and trailing whitespaces from Gmail scopes (Closes issue #310)

8.0.1 (yanked)

Fix ModuleNotFoundError by adding parsedmarc.mail to the list of packages in setup.py (PR #308)

8.0.0 (yanked)

Update included copy of dbip-country-lite.mmdb to the 2022-04 release
Add support for Microsoft/Office 365 via Microsoft Graph API (PR #301 closes issue #111)
Pin elasticsearch-dsl version at >=7.2.0<7.14.0 (PR #297 closes issue #296)
Properly initialize ip_dp_path (PR #294 closes issue #286)
Remove usage of logging.basicConfig (PR #285)
Add support for the Gmail API (PR #284 and PR #307 close issue #96)

7.1.1

Actually include dbip-country-lite.mmdb file in the parsedmarc.resources package (PR #281)
Update dbip-country-lite.mmdb to the 2022-01 release

7.1.0

A static copy of the DBIP Country Lite database is now included for use when a copy of the MaxMind GeoLite2 Country database is not installed (Closes #275)
Add ip_db_path to as a parameter and general setting for a custom IP geolocation database location (Closes #184)
Search default Homebrew path when searching for a copy of the MaxMind GeoLite2 Country database (Closes #272)
Fix log messages written to root logger (PR #276)
Fix --offline option in CLI not being passed as a boolean (PR #265)
Set Elasticsearch shard replication to 0 (PR #274)
Add support for syslog output (PR #263 closes #227)
Do not print TQDDM progress bar when running in a no-interactive TTY (PR #264)

7.0.1

Fix startup error (PR #254)

7.0.0

Fix issue #221: Crash when handling invalid reports without root node (PR #248)
Use UTC datetime objects for Elasticsearch output (PR #245)
Fix issues #219, #155, and #103: IMAP connections break on large emails (PR #241)
Add support for saving reports to S3 buckets (PR #223)
Pass offline parameter to wait_inbox() (PR #216)
Add more details to logging (PR #220)
Add options customizing the names of output files (Modifications based on PR #225)
Wait for 5 seconds before attempting to reconnect to an IMAP server (PR #217)
Add option to process messages in batches (PR #222)

6.12.0

Limit output filename length to 100 characters (PR #199)
Add basic auth support for Elasticsearch (PR #191)
Fix Windows paths when searching for the GeoIP database (PR #190)
Remove six requirement
Require mailsuite>=1.6.1
Require dnspython>=2.0.0
- Drop Python 3.5 support

6.11.0

Fix parsing failure for some valid forensic reports (PR #170)
Fix double count of messages in the Grafana dashboard (PR #182)
Add begin and end date fields for aggregate DMARC reports in Elasticsearch (PR #183 fixes issue #162)
Fix crash on IMAP timeout (PR #186 fixes issue #163)
Fix IMAP debugging output
Fix User-Agent string

6.10.0

Ignore unknown forensic report fields when generating CSVs (Closes issue #148)
Fix crash on IMAP timeout (PR #164 - closes issue #163)
Use SMTP port from the config file when sending emails (PR #151)
Add support for Elasticsearch 7.0 (PR #161 - closes issue #149)
Remove temporary workaround for DMARC aggregate report records missing a SPF domain fields

6.9.0

Use system nameservers instead of Cloudflare by default
Parse aggregate report records with missing SPF domains

6.8.2

Require mailsuite>=1.5.4

6.8.1

Use match_phrase instead of match when looking for existing strings in Elasticsearch

6.8.0

Display warning when GeoLite2-Country.mmdb is missing, instead of trying to download it
Add documentation for MaxMind geoipupdate changes on January 30th, 2019 (closes issues #137 and #139)
Require mail-parser>=3.11.0

6.7.4

Update dependencies

6.7.3

Make dkim_aligned and spf_aligned case-insensitive (PR #132)

6.7.2

Fix SPF results field in CSV output (closes issue #128)

6.7.1

Parse forensic email samples with non-standard date headers
Graceful handling of a failure to download the GeoIP database (issue #123)

6.7.0

Fix typos (PR #119)
Make CSV output match JSON output (Issue # 22)
Graceful processing of invalid aggregate DMARC reports (PR #122)
Remove Python 3.4 support

6.6.1

Close files after reading them

6.6.0

Set a configurable default IMAP timeout of 30 seconds
Set a configurable maximum of 4 IMAP timeout retry attempts
Add support for reading MBOX files
Set a configurable Elasticsearch timeout of 60 seconds

6.5.5

Set minimum publicsuffix2 version

6.5.4

Bump required mailsuite version to 1.2.1

6.5.3

Fix typos in the CLI documentation
Bump required mailsuite version to 1.1.1

6.5.2

Merge PR #100 from michaeldavie
- Correct a bug introduced in 6.5.1 that caused only the last record's data to be used for each row in an aggregate report's CSV version.
Use mailsuite 1.1.0 to fix issues with some IMAP servers (closes issue 103)
- Always use / as the folder hierarchy separator, and convert to the server's hierarchy separator in the background
- Always remove folder name characters that conflict with the server's hierarchy separators
- Prepend the namespace to the folder path when required

6.5.1

Merge PR #98 from michaeldavie
- Add functions
  - parsed_aggregate_reports_to_csv_row(reports)
  - parsed_forensic_reports_to_csv_row(reports)
Require dnspython>=1.16.0

6.5.0

Move mail processing functions to the mailsuite package
Add offline option (closes issue #90)
Use UDP instead of TCP, and properly set the timeout when querying DNS (closes issue #79 and #92)
Log the current file path being processed when --debug is used (closes issue #95)

6.4.2

Do not attempt to convert org_name to a base domain if org_name contains a space (closes issue #94)
Always lowercase the header_from
Provide a more helpful warning message when GeoLite2-Country.mmdb is missing

6.4.1

Raise utils.DownloadError exception when a GeoIP database or Public Suffix List (PSL) download fails (closes issue #73)

6.4.0

Add number_of_shards and number_of_replicas as possible options in the elasticsearch configuration file section (closes issue #78)

6.3.7

Work around some unexpected IMAP responses reported in issue #75

6.3.6

Work around some unexpected IMAP responses reported in issue #70
Show correct destination folder in debug logs when moving aggregate reports

6.3.5

Normalize Delivery-Result value in forensic/failure reports (issue #76) Thanks Freddie Leeman of URIports for the troubleshooting assistance

6.3.4

Fix Elasticsearch index creation (closes issue #74)

6.3.3

Set number_of_shards and number_of_replicas to 1 when creating indexes
Fix dependency conflict

6.3.2

Fix the monthly_indexes option in the elasticsearch configuration section

6.3.1

Fix strip_attachment_payloads option

6.3.0

Fix IMAP IDLE response processing for some mail servers (#67)
Exit with a critical error when required settings are missing (#68)
XML parsing fixes (#69)
Add IMAP responses to debug logging
Add smtp option skip_certificate_verification
Add kafka option skip_certificate_verification
Suppress mailparser logging output
Suppress msgconvert warnings

6.2.2

Fix crash when trying to save forensic reports with missing fields to Elasticsearch

6.2.1

Add missing tqdm dependency to setup.py

6.2.0

Add support for multiprocess parallelized processing via CLI (Thanks zscholl - PR #62)
Save sha256 hashes of attachments in forensic samples to Elasticsearch

6.1.8

Actually fix GeoIP lookups

6.1.7

Fix GeoIP lookups

6.1.6

Better GeoIP error handling

6.1.5

Always use Cloudflare's nameservers by default instead of Google's
Avoid re-downloading the Geolite2 database (and tripping their DDoS protection)
Add geoipupdate to install instructions

6.1.4

Actually package requirements

6.1.3

Fix package requirements

6.1.2

Use local Public Suffix List file instead of downloading it
Fix argument name for send_email() (closes issue #60)

6.1.1

Fix aggregate report processing
Check for the existence of a configuration file if a path is supplied
Replace publicsuffix with publicsuffix2
Add minimum versions to requirements

6.1.0

Fix aggregate report email parsing regression introduced in 6.0.3 (closes issue #57)
Fix Davmail support (closes issue #56)

6.0.3

Don't assume the report is the last part of the email message (issue #55)

6.0.2

IMAP connectivity improvements (issue #53)
Use a temp directory for temp files (issue #54)

6.0.1

Fix Elasticsearch output (PR #50 - andrewmcgilvray)

6.0.0

Move options from CLI to a config file (see updated installation documentation)
Refactoring to make argument names consistent

5.3.0

Fix crash on invalid forensic report sample (Issue #47)
Fix DavMail support (Issue #45)

5.2.1

Remove unnecessary debugging code

5.2.0

Add filename and line number to logging output
Improved IMAP error handling

Add CLI options

--elasticsearch-use-ssl
                      Use SSL when connecting to Elasticsearch
--elasticsearch-ssl-cert-path ELASTICSEARCH_SSL_CERT_PATH
                      Path to the Elasticsearch SSL certificate
--elasticsearch-monthly-indexes
                      Use monthly Elasticsearch indexes instead of daily
                      indexes
--log-file LOG_FILE   output logging to a file

5.1.3

Remove urllib3 version upper limit

5.1.2

Workaround unexpected Office 365/Exchange IMAP responses

5.1.1

Bugfix: Crash when parsing invalid forensic report samples (#38)
Bugfix: Crash when IMAP connection is lost
Increase default Splunk HEC response timeout to 60 seconds

5.1.0

Bugfix: Submit aggregate dates to Elasticsearch as lists, not tuples
Support elasticsearch-dsl<=6.3.0
Add support for TLS/SSL and username/password auth to Kafka

5.0.2

Revert to using publicsuffix instead of publicsuffix2

5.0.1

Use publixsuffix2 (closes issue #4)
Add Elasticsearch to automated testing
Lock elasticsearch-dsl required version to 6.2.1 (closes issue #25)

5.0.0

Note: Re-importing kibana_saved_objects.json in Kibana is required when upgrading to this version!

Bugfix: Reindex the aggregate report index field published_policy.fo as text instead of long (Closes issue #31)
Bugfix: IDLE email processing in Gmail/G-Suite accounts (closes issue #33)
Bugfix: Fix inaccurate DNS timeout in CLI documentation (closes issue #34)
Bugfix: Forensic report processing via CLI
Bugfix: Duplicate aggregate report Elasticsearch query broken
Bugfix: Crash when Arrival-Date header is missing in a forensic/failure/ruf report
IMAP reliability improvements
Save data in separate indexes each day to make managing data retention easier
Cache DNS queries in memory

4.4.1

Don't crash if Elasticsearch returns an unexpected result (workaround for issue #31)

4.4.0

Packaging fixes

4.3.9

Kafka output improvements
- Moved some key values (report_id, org_email, org_name) higher in the JSON structure
- Recreated the date_range values from the ES client for easier parsing.
- Started sending individual record slices. Kafka default message size is 1 MB, some aggregate reports were exceeding this. Now it appends meta-data and sends record by record.

4.3.8

Fix decoding of attachments inside forensic samples
Add CLI option --imap-skip-certificate-verification
Add optional ssl_context argument for get_dmarc_reports_from_inbox() and watch_inbox()
Debug logging improvements

4.3.7

When checking an inbox, always recheck for messages when processing is complete

4.3.6

Be more forgiving for forensic reports with missing fields

4.3.5

Fix base64 attachment decoding (#26)

4.3.4

Fix crash on empty aggregate report comments (brakhane - #25)
Add SHA256 hashes of attachments to output
Add strip_attachment_payloads option to functions and --strip-attachment-payloads option to the CLI (#23)
Set urllib3 version requirements to match requests

4.3.3

Fix forensic report email processing

4.3.2

Fix normalization of the forensic sample from address

4.3.1

Fix parsing of some emails
Fix duplicate forensic report search for Elasticsearch

4.3.0

Fix bug where parsedmarc would always try to save to Elastic search, even if only --hec was used
Add options to save reports as a Kafka topic (mikesiegel - #21)
Major refactoring of functions
Support parsing forensic reports generated by Brightmail
Make sample_headers_only flag more reliable
Functions that might be useful to other projects are now stored in parsedmarc.utils:
- get_base_domain(domain)
- get_filename_safe_string(string)
- get_ip_address_country(ip_address)
- get_ip_address_info(ip_address, nameservers=None, timeout=2.0)
- get_reverse_dns(ip_address, nameservers=None, timeout=2.0)
- human_timestamp_to_datetime(human_timestamp)
- human_timestamp_to_timestamp(human_timestamp)
- parse_email(data)

4.2.0

Save each aggregate report record as a separate Splunk event
Fix IMAP delete action (#20)
Suppress Splunk SSL validation warnings
Change default logging level to WARNING

4.1.9

Workaround for forensic/ruf reports that are missing Arrival-Date and/or Reported-Domain

4.1.8

Be more forgiving of weird XML

4.1.7

Remove any invalid XML schema tags before parsing the XML (#18)

4.1.6

Fix typo in CLI parser

4.1.5

Only move or delete IMAP emails after they all have been parsed
Move/delete messages one at a time - do not exit on error
Reconnect to IMAP if connection is broken during get_dmarc_reports_from_inbox()
Add--imap-port and --imap-no-ssl CLI options

4.1.4

Change default logging level to ERROR

4.1.3

Fix crash introduced in 4.1.0 when creating Elasticsearch indexes (Issue #15)

4.1.2

Fix packaging bug

4.1.1

Add splunk instructions
Reconnect reset IMAP connections when watching a folder

4.1.0

Add options for Elasticsearch prefixes and suffixes
If an aggregate report has the invalid disposition value pass, change it to none

4.0.2

Use report timestamps for Splunk timestamps

4.0.1

When saving aggregate reports in Elasticsearch store domain in published_policy
Rename policy_published to published_policywhen saving aggregate reports to Splunk

4.0.0

Add support for sending DMARC reports to a Splunk HTTP Events Collector (HEC)
Use a browser-like User-Agent when downloading the Public Suffix List and GeoIP DB to avoid being blocked by security proxies
Reduce default DNS timeout to 2.0 seconds
Add alignment booleans to JSON output
Fix .msg parsing CLI exception when msgconvert is not found in the system path
Add --outgoing-port and --outgoing-ssl options
Fall back to plain text SMTP if --outgoing-ssl is not used and STARTTLS is not supported by the server
Always use as the newline when generating CSVs
Workaround for random Exchange/Office 365 Server Unavailable IMAP errors

3.9.7

Completely reset IMAP connection when a broken pipe is encountered

3.9.6

Finish incomplete broken pipe fix

3.9.5

Refactor to use a shared IMAP connection for inbox watching and message downloads
Gracefully recover from broken pipes in IMAP

3.9.4

Fix moving/deleting emails

3.9.3

Fix crash when forensic reports are missing Arrival-Date

3.9.2

Fix PEP 8 spacing
Update build script to fail when CI tests fail

3.9.1

Use COPY and delete if an IMAP server does not support MOVE (closes issue #9)

3.9.0

Reduce IMAP IDLE refresh rate to 5 minutes to avoid session timeouts in Gmail
Fix parsing of some forensic/failure/ruf reports
Include email subject in all warning messages
Fix example NGINX configuration in the installation documentation (closes issue #6)

3.8.2

Fix nameservers option (mikesiegel)
Move or delete invalid report emails in an IMAP inbox (closes issue #7)

3.8.1

Better handling of .msg files when msgconvert is not installed

3.8.0

Use . instead of / as the IMAP folder hierarchy separator when / does not work - fixes dovecot support (#5)
Fix parsing of base64-encoded forensic report data

3.7.3

Fix saving attachment from forensic sample to Elasticsearch

3.7.2

Change uses of the DocType class to Document, to properly support elasticsearch-dsl 6.2.0 (this also fixes use in pypy)
Add documentation for installation under pypy

3.7.1

Require elasticsearch>=6.2.1,<7.0.0 and elasticsearch-dsl>=6.2.1,<7.0.0
Update for class changes in elasticsearch-dsl 6.2.0

3.7.0

Fix bug where PSL would be called before it was downloaded if the PSL was older than 24 Hours

3.6.1

Parse aggregate reports with missing SPF domain

3.6.0

Much more robust error handling

3.5.1

Fix dashboard message counts for source IP addresses visualizations
Improve dashboard loading times
Improve dashboard layout
Add country rankings to the dashboards
Fix crash when parsing report with empty <auth_results></auth_results>

3.5.0

Use Cloudflare's public DNS resolvers by default instead of Google's
Fix installation from virtualenv
Fix documentation typos

3.4.1

Documentation fixes
Fix console output

3.4.0

Maintain IMAP IDLE state when watching the inbox
The -i/--idle CLI option is now -w/--watch
Improved Exception handling and documentation

3.3.0

Fix errors when saving to Elasticsearch

3.2.0

Fix existing aggregate report error message

3.1.0

Fix existing aggregate report query

3.0.0

New features

Add option to select the IMAP folder where reports are stored
Add options to send data to Elasticsearch

Changes

Use Google's public nameservers (8.8.8.8 and 4.4.4.4) by default
Detect aggregate report email attachments by file content rather than file extension
If an aggregate report's org_name is a FQDN, the base is used
Normalize aggregate report IDs

2.1.2

Rename parsed_dmarc_forensic_reports_to_csv() to parsed_forensic_reports_to_csv() to match other functions
Rename parsed_aggregate_report_to_csv() to parsed_aggregate_reports_to_csv() to match other functions
Use local time when generating the default email subject

2.1.1

Documentation fixes

2.1.0

Add get_report_zip() and email_results()
Add support for sending report emails via the command line

2.0.1

Fix documentation
Remove Python 2 code

2.0.0

New features

Parse forensic reports
Parse reports from IMAP inbox

Changes

Drop support for Python 2
Command line output is always a JSON object containing the lists aggregate_reports and forensic_reports
-o/--output option is now a path to an output directory, instead of an output file

1.1.0

Add extract_xml() and human_timestamp_to_datetime methods

1.0.5

Prefix public suffix and GeoIP2 database filenames with .
Properly format errors list in CSV output

1.0.3

Fix documentation formatting

1.0.2

Fix more packaging flaws

1.0.1

Fix packaging flaw

1.0.0

Initial release

32 KiB Raw Blame History

Changelog

9.0.6

Fixes

Improvements

Compatibility / Dependencies

9.0.5

Fixes

9.0.4 (Yanked)

Fixes

9.0.3

Fixes

9.0.2

Improvements

Fixes

Breaking changes

9.0.1

Fixes

9.0.0 (yanked)

8.19.1

8.19.0

8.18.9

8.18.8

8.18.7

8.18.6

8.18.5

8.18.4

8.18.3

8.18.2

8.18.1

8.18.0

8.17.0

8.16.1

8.16.0

8.15.4

8.15.3

8.15.2

8.15.1

8.15.0

8.14.2

8.14.1

8.14.0

8.13.0

8.12.0

8.11.0

8.10.3

8.10.2

8.10.1

8.10.0

8.9.4

8.9.3

8.9.2

8.9.1

8.9.0

8.8.0

8.7.0

8.6.4

8.6.3

8.6.2

8.6.1

8.6.0

8.5.0

8.4.2

8.4.1

8.4.0

8.3.2

8.3.1

8.3.0

8.2.0

8.1.1

8.1.0

8.0.3

8.0.2 (yanked)

8.0.1 (yanked)

8.0.0 (yanked)

7.1.1

7.1.0

7.0.1

7.0.0

6.12.0

32 KiB

Raw Blame History