Commit Graph
627 Commits
Author SHA1 Message Date
Paolo SchiroandGitHub fdb4e4cb36 Added nameservers pre-flight check, issue #543 (#562) 2024-10-02 20:48:02 -04:00
N4v41andGitHub d80ce744da add some identified missing organizations (#566) 2024-09-26 12:37:00 -04:00
Sean Whalen f12828485b Parse aggregate reports with multiple policy_published records 2024-09-12 18:23:11 -04:00
0a6cfb602c Added output to webhook as an option (#558)
* Added output to webhook as an option

* added documentation for new webhook configuration

---------

Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>
2024-09-12 15:47:59 -04:00
Sean Whalen cf46558fa3 Update reverse DNS map 2024-09-12 15:01:40 -04:00
Sean Whalen f1a526c247 Update reverse DNS map 2024-09-12 14:55:01 -04:00
Sean Whalen 7344ea9dda Update reverse DNS map 2024-09-12 14:53:59 -04:00
Sean Whalen 7633a30066 maildir fixes 2024-09-12 14:48:26 -04:00
Sean Whalen 266d57eb8c Fix maildir connection 2024-09-06 16:00:23 -04:00
7b7d20b1a4 Added input report maildir connector, issue #82 (#555)
Co-authored-by: Paolo Schiro <paolo.schiro@staff.aruba.it>
2024-09-06 15:44:57 -04:00
Sean Whalen 5973ca8205 Code style fixes 2024-09-04 16:48:07 -04:00
Sean Whalen e967778f25 8.15.0
- Fix processing of SMTP-TLS reports ([#549](https://github.com/domainaware/parsedmarc/issues/549)), which broke in commit [410663d ](https://github.com/domainaware/parsedmarc/commit/410663dbcaba019ca3d3744946348b56a635480b)(PR [#530](https://github.com/domainaware/parsedmarc/pull/530))
  - This PR enforced a stricter check for base64-encoded strings, which SMTP TLS reports from Google did not pass
  - Removing the check introduced its own issue, because some file paths were treated as base64-encoded strings
- Create a separate `extract_report_from_file_path()` function for processioning reports based on a file path
- Remove report extraction based on a file path from `extract_report()`
2024-09-04 16:31:41 -04:00
Sean Whalen 630863df5c 8.14.2
- Update `base_reverse_dns_map.csv` to fix over-replacement on [`f3a5f10`](https://github.com/domainaware/parsedmarc/commit/f3a5f10d67b02c5db31ae1f7ced68028f46ca2a3) (PR #553)
2024-09-03 13:58:07 -04:00
Anael MobiliaandGitHub 3a3b02687e Update base_reverse_dns_map.csv (#553)
Fix over-replacement on f3a5f10
2024-09-03 13:51:30 -04:00
Sean Whalen 79cd7d3c3d Actual 8.14.1 release
- Fix unit tests by enforcing base64 validation
2024-09-03 13:22:27 -04:00
Sean Whalen 3ca3d64775 8.14.1 2024-09-03 11:24:52 -04:00
Sean Whalen 21180f4bb8 8.14.0 release
- Fix processing of SMTP-TLS reports (#549)
- Skip invalid aggregate report rows without calling the whole report invalid
  - Some providers such as GoDaddy will send reports with some rows missing a source IP address, while other rows are fine
- Fix Dovecot support by using the seperator provided by the IPMAP namespace when possible (PR #552 closes #551)
- Only download `base_reverse_dns_map.csv` once (fixes #542)
- Update included `base_reverse_dns_map.csv`
  - Replace University category with Education to be more inclusive
- Update included `dbip-country-lite.mmdb`
2024-09-03 08:37:19 -04:00
Sean Whalen f3a5f10d67 Replace University category with Education to be more inclusive 2024-09-02 21:08:17 -04:00
Frederico Freire BoaventuraandGitHub 4787da9ea1 Fix the tilde expansion for user home folder (#550)
This will enable the tilde expansion to map the output dir to the home
user using `~/path`.
2024-09-02 17:28:48 -04:00
Gaige B PaulsenandGitHub 06d8578c47 fix: use namespace for separator (#552) 2024-09-02 17:27:28 -04:00
Sean Whalen ef621d68c7 Cache the reverse DNS map 2024-09-02 17:25:55 -04:00
Sean Whalen 1c3bcc05c7 Skip invalid aggregate report rows without calling the whole report invalid 2024-09-02 17:08:51 -04:00
Sean Whalen 554888b3dd Update IPDP database 2024-09-02 17:07:32 -04:00
N4v41andGitHub 17d9599c54 Add various mail providers to the reverse DNS map (#548)
* Update base_reverse_dns_map.csv

Add Cloudflare E-mail routing service

* Add various missing mail providers
2024-08-31 12:08:15 -04:00
N4v41andGitHub fb0adf0627 Update base_reverse_dns_map.csv (#547)
Add Cloudflare E-mail routing service
2024-08-29 15:01:45 -04:00
N4v41andGitHub ef820c5d68 Update base_reverse_dns_map.csv (#546)
add Qualtrics to base_reverse_dns_map
2024-08-29 15:01:19 -04:00
Sean Whalen cc1b6ae389 8.13.0
- Add Elastic/OpenSearch index prefix option (PR #531 closes #159)
- Add GELF output support (PR #532)
2024-08-24 21:29:32 -04:00
Sean WhalenandGitHub 06ce080171 Update cli.py
Remove trailing spaces
2024-08-24 11:34:48 -04:00
Jason LingohrandGitHub 11e0461b9d Add GELF support (#532)
* Implement the ability to log to a GELF server/input, via the use of pygelf.

* Fix flake8 style checks.
2024-08-24 11:28:55 -04:00
Emmanuel OrmanceyandGitHub 9b32d9459f added ES/OS prefix (#531) 2024-08-24 11:26:13 -04:00
FélixandGitHub aa357dc50f add new sources to base_reverse_dns_map.csv (#537)
* add new sources to base_reverse_dns_map.csv

* remove extra comma in base_reverse_dns_map.csv
2024-07-30 15:53:24 -04:00
pphillips99andGitHub 410663dbca fixup and tests to extract_report() to handle documented inputs (#530) 2024-07-16 14:53:27 -04:00
Sean Whalen 13ddc26d70 8.12.0
- Fix for deadlock with large report (#508)
- Build: move to kafka-python-ng (#510)
- Fix new config variables previously not propagated in the code (#524)
- Fixes for kafka integration (#522)
- Fix if base_domain is None before get_service_from_reverse_dns_base_domain (#514)
- Update base_reverse_dns_map.csv
2024-05-22 09:35:18 -04:00
Sean Whalen 899dc060b2 pep8 fixes 2024-05-22 09:06:32 -04:00
1145b0f63b Update base_reverse_dns_map.csv (#516)
* Update base_reverse_dns_map.csv

Add some domains + fix mailchimp

* Update base_reverse_dns_map.csv

While Mailgun can be used for marketing, its main use case is transactional email.

https://www.mailgun.com/solutions/use-cases/transactional-email/

---------

Co-authored-by: Sean Whalen <44679+seanthegeek@users.noreply.github.com>
2024-05-22 08:44:14 -04:00
Lennart FribergandGitHub 91191e30f3 Fix if base_domain is None before get_service_from_reverse_dns_base_d… (#514)
* Fix if base_domain is None before get_service_from_reverse_dns_base_domain call

- Added if statement for base_domain before getting get_service_from_reverse_dns_base_domain(). If base_domain is None, get_service_from_reverse_dns_base_domain() will fail the report.
- Added .xml test file

* Create protection.outlook.com!example.com!1711756800!1711843200.xml

added .xml test sample report where base_domain return none
2024-05-22 08:36:13 -04:00
ChibangLWandGitHub 42be53349f Fixes for kafka integration (#522)
* fix(kafka): set correct variables

* fix(kafka): made username and password optional.

The documentation states these as optional but in code these were mandatory.
2024-05-22 08:35:41 -04:00
ryuksanandGitHub f8ee9bd166 Fix new config variables previously not propagated in the code (#524) 2024-05-22 08:13:45 -04:00
Andrew BaumannandGitHub ed92e0f7eb Remove unused parsedmarc.utils.tempdir (#517)
Parsedmarc.utils was creating a temp directory on every import, but this directory is no longer used anywhere.
2024-05-22 08:13:11 -04:00
Russel Hunter YukawaandGitHub 7aa2e14cbb Fix for deadlock with large report (#508)
* add large xml sample

* Avoid deadlock

* Remove extra  whitespaces
2024-05-22 08:10:59 -04:00
Vermium SifellandGitHub 0b46c1807c Update base_reverse_dns_map.csv (#515) 2024-05-22 08:08:41 -04:00
Sean WhalenandGitHub 11a9e959a0 Update base_reverse_dns_map.csv 2024-04-16 13:35:33 -04:00
Lennart FribergandGitHub b6a8739b4f Update base_reverse_dns_map.csv (#513)
Added Email Security provider
2024-04-15 10:06:45 -04:00
Sean Whalen 6a06d01b44 8.11.0
- Actually save `source_type` and `source_name` to Elasticsearch and OpenSearch
- Reverse-lookup cache improvements (PR #501 closes issue #498)
- Update the included `dbip-country-lite.mmdb` to the 2024-03 version
- Update `base_reverse_dns_map.csv`
- Add new general config options (closes issue #500)
  - `always_use_local_files` - Disables the download of the reverse DNS map
  - `local_reverse_dns_map_path` - Overrides the default local file path to use for the reverse DNS map
  - `reverse_dns_map_url` - Overrides the default download URL for the reverse DNS map
2024-04-02 12:34:32 -04:00
Sean Whalen 422f7a074a Update base_reverse_dns_map.csv 2024-04-02 12:13:19 -04:00
Sean Whalen d6128eae9f Actually save source_tye and source_name to Elasticsearch and Opensearch 2024-04-01 09:14:45 -04:00
Sean Whalen 36eedcb446 Update dbip-country-lite.mmdb 2024-03-31 20:36:22 -04:00
Sean Whalen acef7bdd6e Code cleanup 2024-03-31 20:25:10 -04:00
Rod PayneandGitHub 8936193280 Reverse-lookup cache improvements (#501) 2024-03-31 20:01:40 -04:00
Sean Whalen fd5b792c4a Close issue #500
Add the following general configuration options:

- `always_use_local_files` - Disables the download of the reverse DNS map
- `local_reverse_dns_map_path` - Overrides the default local file path to use for the reverse DNS map
2024-03-31 20:00:47 -04:00