amorfo77/parsedmarc
1
0
Fork 0
mirror of https://github.com/domainaware/parsedmarc.git synced 2026-10-05 20:10:31 +00:00
Code Issues Projects Releases Wiki Activity

Merge pull request #97 from domainaware/6.5

Browse Source 
6.5
This commit is contained in:
Sean Whalen
2019-07-17 10:40:28 -04:00
committed by GitHub
parent 5294d7140c 766786344d
commit 5a6b51c59f
8 changed files with 328 additions and 871 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+9
View File
@@ -1,3 +1,12 @@
6.5.0
-----
- Move mail processing functions to `mailsuite` package
- Add offline option (closes issue #90)
- Properly set timeout when querying DNS (closes issue #79 and #92)
- Log the current file path being processed when `--debug` is used (closes issue #95)
6.4.2
-----
README.rst
+29 -26
View File
@@ -58,35 +58,37 @@ CLI help
::
usage: parsedmarc [-h] [-c CONFIG_FILE] [--strip-attachment-payloads]
[-o OUTPUT] [-n NAMESERVERS [NAMESERVERS ...]]
[-t DNS_TIMEOUT] [-s] [--debug] [--log-file LOG_FILE] [-v]
[file_path [file_path ...]]
usage: parsedmarc [-h] [-c CONFIG_FILE] [--strip-attachment-payloads]
[-o OUTPUT] [-n NAMESERVERS [NAMESERVERS ...]]
[-t DNS_TIMEOUT] [--offline] [-s] [--debug]
[--log-file LOG_FILE] [-v]
[file_path [file_path ...]]
Parses DMARC reports
Parses DMARC reports
positional arguments:
file_path one or more paths to aggregate or forensic report
files or emails
positional arguments:
file_path one or more paths to aggregate or forensic report
files or emails
optional arguments:
-h, --help show this help message and exit
-c CONFIG_FILE, --config-file CONFIG_FILE
A path to a configuration file (--silent implied)
--strip-attachment-payloads
remove attachment payloads from forensic report output
-o OUTPUT, --output OUTPUT
write output files to the given directory
-n NAMESERVERS [NAMESERVERS ...], --nameservers NAMESERVERS [NAMESERVERS ...]
nameservers to query (default is Cloudflare's
nameservers)
-t DNS_TIMEOUT, --dns_timeout DNS_TIMEOUT
number of seconds to wait for an answer from DNS
(default: 6.0)
-s, --silent only print errors and warnings
--debug print debugging information
--log-file LOG_FILE output logging to a file
-v, --version show program's version number and exit
optional arguments:
-h, --help show this help message and exit
-c CONFIG_FILE, --config-file CONFIG_FILE
A path to a configuration file (--silent implied)
--strip-attachment-payloads
remove attachment payloads from forensic report output
-o OUTPUT, --output OUTPUT
write output files to the given directory
-n NAMESERVERS [NAMESERVERS ...], --nameservers NAMESERVERS [NAMESERVERS ...]
nameservers to query (default is Cloudflare's
nameservers)
-t DNS_TIMEOUT, --dns_timeout DNS_TIMEOUT
number of seconds to wait for an answer from DNS
(default: 2.0)
--offline Do not make online queries for geolocation or DNS
-s, --silent only print errors and warnings
--debug print debugging information
--log-file LOG_FILE output logging to a file
-v, --version show program's version number and exit
.. note::
@@ -133,6 +135,7 @@ The full set of configuration options are:
- ``save_forensic`` - bool: Save forensic report data to the Elasticsearch and/or Splunk
- ``strip_attachment_payloads`` - bool: Remove attachment payloads from results
- ``output`` - str: Directory to place JSON and CSV files in
- ``offline`` - bool: Do not use online queries for geolocation or DNS
- ``nameservers`` - str: A comma separated list of DNS resolvers (Default: `Cloudflare's public resolvers`_)
- ``dns_timeout`` - float: DNS timeout period
- ``debug`` - bool: Print debugging messages
docs/index.rst
+8 -8
View File
@@ -63,9 +63,10 @@ CLI help
::
usage: parsedmarc [-h] [-c CONFIG_FILE] [--strip-attachment-payloads]
[-o OUTPUT] [-n NAMESERVERS [NAMESERVERS ...]]
[-t DNS_TIMEOUT] [-s] [--debug] [--log-file LOG_FILE] [-v]
[file_path [file_path ...]]
[-o OUTPUT] [-n NAMESERVERS [NAMESERVERS ...]]
[-t DNS_TIMEOUT] [--offline] [-s] [--debug]
[--log-file LOG_FILE] [-v]
[file_path [file_path ...]]
Parses DMARC reports
@@ -86,12 +87,14 @@ CLI help
nameservers)
-t DNS_TIMEOUT, --dns_timeout DNS_TIMEOUT
number of seconds to wait for an answer from DNS
(default: 6.0)
(default: 2.0)
--offline Do not make online queries for geolocation or DNS
-s, --silent only print errors and warnings
--debug print debugging information
--log-file LOG_FILE output logging to a file
-v, --version show program's version number and exit
.. note::
In ``parsedmarc`` 6.0.0, most CLI options were moved to a configuration file, described below.
@@ -137,6 +140,7 @@ The full set of configuration options are:
- ``save_forensic`` - bool: Save forensic report data to the Elasticsearch and/or Splunk
- ``strip_attachment_payloads`` - bool: Remove attachment payloads from results
- ``output`` - str: Directory to place JSON and CSV files in
- ``offline`` - bool: Do not use online queries for geolocation or DNS
- ``nameservers`` - str: A comma separated list of DNS resolvers (Default: `Cloudflare's public resolvers`_)
- ``dns_timeout`` - float: DNS timeout period
- ``debug`` - bool: Print debugging messages
@@ -1482,10 +1486,6 @@ parsedmarc.elastic
parsedmarc.splunk
-----------------
.. toctree::
:maxdepth: 2
:caption: Contents:
.. automodule:: parsedmarc.splunk
:members:
parsedmarc/__init__.py
+220 -790
View File
File diff suppressed because it is too large Load Diff
parsedmarc/cli.py
+45 -37
View File
@@ -16,9 +16,9 @@ from itertools import repeat
import time
from tqdm import tqdm
from parsedmarc import IMAPError, get_dmarc_reports_from_inbox, \
from parsedmarc import get_dmarc_reports_from_inbox, watch_inbox, \
parse_report_file, elastic, kafkaclient, splunk, save_output, \
watch_inbox, email_results, SMTPError, ParserError, __version__, \
email_results, ParserError, __version__, \
InvalidDMARCReport
logger = logging.getLogger("parsedmarc")
@@ -30,10 +30,12 @@ def _str_to_list(s):
return list(map(lambda i: i.lstrip(), _list))
def cli_parse(file_path, sa, nameservers, dns_timeout, parallel=False):
def cli_parse(file_path, sa, nameservers, dns_timeout, offline,
parallel=False):
"""Separated this function for multiprocessing"""
try:
file_results = parse_report_file(file_path,
offline=offline,
nameservers=nameservers,
dns_timeout=dns_timeout,
strip_attachment_payloads=sa,
@@ -162,9 +164,12 @@ def _main():
"(default is Cloudflare's nameservers)")
arg_parser.add_argument("-t", "--dns_timeout",
help="number of seconds to wait for an answer "
"from DNS (default: 6.0)",
"from DNS (default: 2.0)",
type=float,
default=6.0)
default=2.0)
arg_parser.add_argument("--offline", action="store_true",
help="Do not make online queries for geolocation "
" or DNS")
arg_parser.add_argument("-s", "--silent", action="store_true",
help="only print errors and warnings")
arg_parser.add_argument("--debug", action="store_true",
@@ -180,6 +185,7 @@ def _main():
args = arg_parser.parse_args()
opts = Namespace(file_path=args.file_path,
config_file=args.config_file,
offline=args.offline,
strip_attachment_payloads=args.strip_attachment_payloads,
output=args.output,
nameservers=args.nameservers,
@@ -243,6 +249,8 @@ def _main():
config.read(args.config_file)
if "general" in config.sections():
general_config = config["general"]
if "offline" in general_config:
opts.offline = general_config["offline"]
if "strip_attachment_payloads" in general_config:
opts.strip_attachment_payloads = general_config[
"strip_attachment_payloads"]
@@ -504,6 +512,7 @@ def _main():
repeat(opts.strip_attachment_payloads),
repeat(opts.nameservers),
repeat(opts.dns_timeout),
repeat(opts.offline),
repeat(opts.n_procs >= 1)),
opts.chunk_size)
pbar = tqdm(total=len(file_paths))
@@ -536,23 +545,22 @@ def _main():
ns = opts.nameservers
sa = opts.strip_attachment_payloads
ssl = True
ssl_context = None
verify = True
if opts.imap_skip_certificate_verification:
logger.debug("Skipping IMAP certificate verification")
ssl_context = create_default_context()
ssl_context.check_hostname = False
ssl_context.verify_mode = CERT_NONE
verify=False
if opts.imap_ssl is False:
ssl = False
reports = get_dmarc_reports_from_inbox(host=opts.imap_host,
port=opts.imap_port,
ssl=ssl,
ssl_context=ssl_context,
verify=verify,
user=opts.imap_user,
password=opts.imap_password,
reports_folder=rf,
archive_folder=af,
delete=opts.imap_delete,
offline=opts.offline,
nameservers=ns,
test=opts.imap_test,
strip_attachment_payloads=sa
@@ -561,7 +569,7 @@ def _main():
aggregate_reports += reports["aggregate_reports"]
forensic_reports += reports["forensic_reports"]
except IMAPError as error:
except Exception as error:
logger.error("IMAP Error: {0}".format(error.__str__()))
exit(1)
@@ -575,46 +583,46 @@ def _main():
if opts.smtp_host:
try:
ssl_context = None
verify = True
if opts.smtp_skip_certificate_verification:
logger.debug("Skipping SMTP certificate verification")
ssl_context = create_default_context()
ssl_context.check_hostname = False
ssl_context.verify_mode = CERT_NONE
verify = False
email_results(results, opts.smtp_host, opts.smtp_from,
opts.smtp_to, ssl=opts.smtp_ssl,
user=opts.smtp_user,
opts.smtp_to, verify=verify,
username=opts.smtp_user,
password=opts.smtp_password,
subject=opts.smtp_subject,
ssl_context=ssl_context)
except SMTPError as error:
logger.error("SMTP Error: {0}".format(error.__str__()))
subject=opts.smtp_subject)
except Exception as error:
logger.error("S{0}".format(error.__str__()))
exit(1)
if opts.imap_host and opts.imap_watch:
logger.info("Watching for email - Quit with ctrl-c")
ssl = True
ssl_context = None
verify = True
if opts.imap_skip_certificate_verification:
logger.debug("Skipping IMAP certificate verification")
ssl_context = create_default_context()
ssl_context.check_hostname = False
ssl_context.verify_mode = CERT_NONE
verify = False
if opts.imap_ssl is False:
ssl = False
try:
sa = opts.strip_attachment_payloads
watch_inbox(opts.imap_host, opts.imap_user, opts.imap_password,
process_reports, port=opts.imap_port, ssl=ssl,
ssl_context=ssl_context,
reports_folder=opts.imap_reports_folder,
archive_folder=opts.imap_archive_folder,
delete=opts.imap_delete,
test=opts.imap_test, nameservers=opts.nameservers,
dns_timeout=opts.dns_timeout,
strip_attachment_payloads=sa)
except IMAPError as error:
logger.error("IMAP error: {0}".format(error.__str__()))
watch_inbox(
opts.imap_host,
opts.imap_user,
opts.imap_password,
process_reports,
port=opts.imap_port,
ssl=ssl,
verify=verify,
reports_folder=opts.imap_reports_folder,
archive_folder=opts.imap_archive_folder,
delete=opts.imap_delete,
test=opts.imap_test,
nameservers=opts.nameservers,
dns_timeout=opts.dns_timeout,
strip_attachment_payloads=sa)
except FileExistsError as error:
logger.error("{0}".format(error.__str__()))
exit(1)
parsedmarc/utils.py
+14 -9
View File
@@ -155,7 +155,7 @@ def query_dns(domain, record_type, cache=None, nameservers=None, timeout=2.0):
if record_type == "TXT":
resource_records = list(map(
lambda r: r.strings,
resolver.query(domain, record_type, tcp=True)))
resolver.query(domain, record_type, lifetime=timeout)))
_resource_record = [
resource_record[0][:0].join(resource_record)
for resource_record in resource_records if resource_record]
@@ -163,7 +163,7 @@ def query_dns(domain, record_type, cache=None, nameservers=None, timeout=2.0):
else:
records = list(map(
lambda r: r.to_text().replace('"', '').rstrip("."),
resolver.query(domain, record_type, tcp=True)))
resolver.query(domain, record_type, lifetime=timeout)))
if cache:
cache[cache_key] = records
@@ -257,7 +257,7 @@ def human_timestamp_to_timestamp(human_timestamp):
return human_timestamp_to_datetime(human_timestamp).timestamp()
def get_ip_address_country(ip_address, parallel=False):
def get_ip_address_country(ip_address, parallel=False, offline=False):
"""
Uses the MaxMind Geolite2 Country database to return the ISO code for the
country associated with the given IPv4 or IPv6 address
@@ -265,6 +265,7 @@ def get_ip_address_country(ip_address, parallel=False):
Args:
ip_address (str): The IP address to query for
parallel (bool): Parallel processing
offline (bool): Do not make online queries for geolocation and DNS
Returns:
str: And ISO country code associated with the given IP address
@@ -275,7 +276,7 @@ def get_ip_address_country(ip_address, parallel=False):
Args:
location (str): Local location for the database file
"""
if parallel:
if parallel or offline:
logging.warning("GeoLite2-Country.mmdb is missing."
"please install and run geoipupdate as root to "
"get the latest version.")
@@ -340,14 +341,15 @@ def get_ip_address_country(ip_address, parallel=False):
return country
def get_ip_address_info(ip_address, cache=None, nameservers=None,
timeout=2.0, parallel=False):
def get_ip_address_info(ip_address, cache=None, offline=False,
nameservers=None, timeout=2.0, parallel=False):
"""
Returns reverse DNS and country information for the given IP address
Args:
ip_address (str): The IP address to check
cache (ExpiringDict): Cache storage
offline (bool): Do not make online queries for geolocation or DNS
nameservers (list): A list of one or more nameservers to use
(Cloudflare's public DNS resolvers by default)
timeout (float): Sets the DNS timeout in seconds
@@ -364,9 +366,12 @@ def get_ip_address_info(ip_address, cache=None, nameservers=None,
return info
info = OrderedDict()
info["ip_address"] = ip_address
reverse_dns = get_reverse_dns(ip_address,
nameservers=nameservers,
timeout=timeout)
if offline:
reverse_dns = None
else:
reverse_dns = get_reverse_dns(ip_address,
nameservers=nameservers,
timeout=timeout)
country = get_ip_address_country(ip_address, parallel=parallel)
info["country"] = country
info["reverse_dns"] = reverse_dns
requirements.txt
+1
View File
@@ -14,6 +14,7 @@ dateparser>=0.7.1
elasticsearch>=6.3.1,<7.0.0
elasticsearch-dsl>=6.3.1,<7.0.0
kafka-python>=1.4.4
mailsuite>=1.0.0
nose
pygments
flake8
setup.py
+2 -1
View File
@@ -14,7 +14,7 @@ from setuptools import setup
from codecs import open
from os import path
__version__ = "6.4.2"
__version__ = "6.5.0"
description = "A Python package and CLI for parsing aggregate and " \
"forensic DMARC reports"
@@ -97,6 +97,7 @@ setup(
'urllib3<1.25,>=1.21.1', 'requests>=2.2.16.0',
'imapclient>=2.1.0', 'mail-parser>=3.9.2',
'dateparser>=0.7.1',
'mailsuite>=1.0.0',
'elasticsearch>=6.3.1,<7.0.0',
'elasticsearch-dsl>=6.3.1,<7.0.0',
'kafka-python>=1.4.4',