amorfo77/parsedmarc
1
0
Fork 0
mirror of https://github.com/domainaware/parsedmarc.git synced 2026-09-08 15:07:59 +00:00
Code Issues Projects Releases Wiki Activity

Make the output and mailbox integrations optional extras (#888)

Browse Source 
* Make the output and mailbox integrations optional extras (#883)

Breaking change for the next major release: pip install parsedmarc now
installs the parsing core plus a working core CLI (file, IMAP, Maildir,
and mbox input; CSV/JSON, Splunk HEC, webhook, and syslog output).
Everything else moves behind an extra: elastic, opensearch, kafka, s3,
gelf, loganalytics, msgraph, and gmail, joining the existing postgresql
extra, with an umbrella [all] that deliberately excludes postgresql
(psycopg's binary wheels do not exist on every platform, so
parsedmarc[all] must never fail to install there).

cli.py imports the six SDK-dependent output modules behind the #884
TYPE_CHECKING/try-except guard; a configured section whose extra is
missing fails fast with a ConfigurationError naming the section and the
exact pip install command — including the msgraph and gmail_api mailbox
sections (detected via parsedmarc.mail's placeholder classes) and
postgresql (checked before the constructor so the startup retry loop
does not retry a missing dependency for a minute). The Azure/kiota Graph
error types fall back to never-raised sentinel classes.

The Docker image installs [all,postgresql], so container users see no
change. CI lint installs [build,all,postgresql]; the unit-test job
installs [build,all], deliberately without postgresql so
test_postgres.py's absent-psycopg arm stays exercised. The
never-imported dateparser dependency is dropped in favor of declaring
python-dateutil, which utils.py actually imports; pytz moves to the
build extra for the one test that uses it.

Verified live: a no-extras wheel install imports, parses samples, and
reports the install hint for each gated section; a [all] install
restores every integration; the Docker image builds with every SDK
importable.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Patch psycopg presence in the PostgreSQL CLI wiring tests

CI's unit-test job deliberately installs [build,all] without the
postgresql extra, so parsedmarc.cli.postgres.psycopg is None there and
the new missing-extra presence check correctly made _main exit 1 before
the wiring under test ran. The tests simulate the SDK being available
(PostgreSQLClient is mocked at the SDK boundary), so the module-level
psycopg handle is now patched present in setUp. Verified against a
simulated psycopg-absent environment as well as the local full install.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Address Copilot review: narrow guards to ModuleNotFoundError, fix docs

- The optional-integration and Graph error-type import guards now catch
  ModuleNotFoundError instead of ImportError, so only a genuinely absent
  package reads as a missing extra; a broken-but-present SDK fails
  loudly with its real error instead of masquerading as one. The test
  blocker raises ModuleNotFoundError accordingly — the exact exception a
  missing package produces.
- _missing_extra_hint docstring no longer calls every gated integration
  an output module (it also serves the msgraph/gmail_api mailbox
  sections).
- Fix the pre-existing passsword typo in usage.md's kafka section; the
  INI key the code reads is password (cli.py _parse_config).

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Quote extras specs in copy-paste install commands

From Copilot's second review round: zsh treats an unquoted .[build,all]
as a glob and fails with 'no matches found', so the commands shown in
AGENTS.md, CONTRIBUTING.md, dashboards/README.md, and the bootstrap
script's comment are now quoted. The CI workflows keep the unquoted
form: they run under bash, which passes unmatched globs through
literally. The suggestion to change the 'Choosing what to install'
heading level was rejected — it is a subsection of 'Installing
parsedmarc', matching the file's existing hierarchy.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Fix upgrade command in the changelog

* Documentation review: accuracy, spelling, grammar, and clarity pass

A full prose review of docs/source, README, CONTRIBUTING, and the
dashboards README, with every accuracy claim verified against the code
before changing it. Highlights:

- usage.md: documented six missing [general] options (the CSV/JSON
  filename options, prettify_json, normalize_timespan_threshold_hours),
  the required kafka smtp_tls_topic, [imap] timeout/max_retries, and
  the postgresql env-var prefix; corrected the maildir_path default
  (None, not INBOX — cli.py Namespace defaults), the mailbox
  check_timeout option name, the systemd restart interval (RestartSec
  is 5m), and merged the duplicate silent entry; quoted every
  copy-paste extras spec for zsh safety.
- elasticsearch.md: fixed an invalid openssl command (rsa:4096 -nodes),
  the dashboards filename (opensearch_dashboards.ndjson, matching the
  file the link serves), and assorted grammar.
- davmail.md: the service-enable command now enables davmail.service
  (was parsedmarc.service — a copy-paste error that left DavMail
  unenabled), plus a view typo and DavMail capitalization.
- output.md: the example schema reference is RFC 7489 Appendix C
  (7480 is RDAP). kibana.md: SPF relies on the SMTP envelope, not
  session headers (RFC 7208). dmarc.md: DKM -> DKIM.
- README: the intro now also names the OpenSearch/Grafana stack,
  matching the feature list. CONTRIBUTING: pre-PR checks now include
  ruff format --check and pyright, matching CI's lint job.
- dashboards/README: the service table and seed description now include
  the PostgreSQL backend the compose stack runs.

Sample data blocks, the CLI-help mirror block, and released CHANGELOG
entries were deliberately left untouched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Docstring review: accuracy, spelling, grammar, and clarity pass

Every docstring in parsedmarc/, parsedmarc/mail/, the maps maintainer
scripts, and the test suite reviewed with each claim verified against
the code it documents. Text-only — no behavior changes. Highlights:

- Copy-paste errors corrected: parsed_smtp_tls_reports_to_csv and
  splunk/loganalytics save functions described aggregate or failure
  reports they do not handle; LogAnalyticsException claimed to be an
  Elasticsearch error.
- Docstring/behavior mismatches: parse_report_email's report_type
  enumeration omitted smtp_tls; parse_failure_report typed msg_date as
  str (it is datetime); strip_attachment_payloads claimed payloads are
  replaced with None (the key is deleted); kafkaclient's failure and
  SMTP TLS savers claimed per-record slicing while sending the whole
  list in one message (docstrings now describe reality — whether
  slicing was intended is flagged for follow-up); the postgres savers
  claimed to take parse_report_file's return value but receive the
  inner report dict; elastic/opensearch save functions' Raises listed
  only AlreadySaved.
- None-as-semantic-state documented where missing (get_base_domain,
  get_ip_address_country), enumeration completeness fixed
  (get_ip_address_info's 9 result keys, maps script outputs, TSV
  columns), and the stale 44-industry-types count corrected to the
  46 the authoritative README list defines.
- Test docstrings aligned with what the tests actually assert,
  including two that overstated coverage of the elastic/opensearch
  address-list tests.
- Two argparse help strings fixed: file_path now names SMTP TLS report
  files alongside aggregate and failure, mirrored into usage.md's
  CLI-help block; --offline's doubled spaces removed (rendered help
  unchanged).
- elasticsearch.md's security claim corrected against Elastic's docs:
  security is enabled and auto-configured on first startup since 8.0
  (not "8.7 secure mode"), so the settings are verified, not
  hand-written.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
This commit is contained in:
Sean Whalen
2026-08-28 17:33:18 -04:00
committed by GitHub
co-authored by Claude Fable 5 Copilot
parent 52be8850b2
commit 07bca1ad28
44 changed files with 1131 additions and 283 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/source/elasticsearch.md
+24 -19
View File
@@ -55,13 +55,17 @@ sudo systemctl start elasticsearch.service
sudo systemctl start kibana.service
```
As of Elasticsearch 8.7, activate secure mode (xpack.security.*.ssl)
Since Elasticsearch 8.0, security is enabled and auto-configured on
first startup: TLS certificates are generated, the `xpack.security.*`
settings below are written to `elasticsearch.yml`, and a password is
generated for the `elastic` user. Verify the settings are present —
and add them only if your install skipped auto-configuration:
```bash
sudo vim /etc/elasticsearch/elasticsearch.yml
```
Add the following configuration
The security configuration looks like this:
```text
# Enable security features
@@ -92,11 +96,12 @@ openssl req -x509 -nodes -days 365 -newkey rsa:4096 -keyout kibana.key -out kiba
Or, to create a Certificate Signing Request (CSR) for a CA, run:
```bash
openssl req -newkey rsa:4096-nodes -keyout kibana.key -out kibana.csr
openssl req -newkey rsa:4096 -nodes -keyout kibana.key -out kibana.csr
```
Fill in the prompts. Watch out for Common Name (e.g. server FQDN or YOUR
domain name), which is the IP address or domain name that you will use to access Kibana. it is the most important field.
domain name), which is the IP address or domain name that you will use to
access Kibana. It is the most important field.
If you generated a CSR, remove the CSR after you have your certs
@@ -129,7 +134,7 @@ server.ssl.key: /etc/kibana/kibana.key
:::{note}
For more security, you can configure Kibana to use a local network connection
to elasticsearch :
to Elasticsearch:
```text
elasticsearch.hosts: ['https://SERVER_IP:9200']
```
@@ -149,14 +154,14 @@ Enroll Kibana in Elasticsearch
sudo /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana
```
Then access to your web server at `https://SERVER_IP:5601`, accept the self-signed
certificate and paste the token in the "Enrollment token" field.
Then access your web server at `https://SERVER_IP:5601`, accept the self-signed
certificate, and paste the token in the "Enrollment token" field.
```bash
sudo /usr/share/kibana/bin/kibana-verification-code
```
Then put the verification code to your web browser.
Then enter the verification code in your web browser.
End Kibana configuration
@@ -182,12 +187,12 @@ sudo systemctl restart elasticsearch
Now that Elasticsearch is up and running, use `parsedmarc` to send data to
it.
Download (right-click the link and click save as) [export.ndjson].
Download (right-click the link and click save as) [opensearch_dashboards.ndjson].
Connect to kibana using the "elastic" user and the password you previously provide
on the console ("End Kibana configuration" part).
Connect to Kibana using the "elastic" user and the password you previously
provided on the console ("End Kibana configuration" part).
Import `export.ndjson` the Saved Objects tab of the Stack management
Import `opensearch_dashboards.ndjson` in the Saved Objects tab of the Stack Management
page of Kibana. (Hamburger menu -> "Management" -> "Stack Management" ->
"Kibana" -> "Saved Objects")
@@ -198,20 +203,20 @@ the commercial [X-Pack].
```{image} _static/screenshots/saved-objects.png
:align: center
:alt: A screenshot of setting the Saved Objects Stack management UI in Kibana
:alt: A screenshot of the Saved Objects Stack Management UI in Kibana
:target: _static/screenshots/saved-objects.png
```
```{image} _static/screenshots/confirm-overwrite.png
:align: center
:alt: A screenshot of the overwrite conformation prompt
:alt: A screenshot of the overwrite confirmation prompt
:target: _static/screenshots/confirm-overwrite.png
```
## Upgrading Kibana index patterns
`parsedmarc` 5.0.0 makes some changes to the way data is indexed in
Elasticsearch. if you are upgrading from a previous release of
Elasticsearch. If you are upgrading from a previous release of
`parsedmarc`, you need to complete the following steps to replace the
Kibana index patterns with versions that match the upgraded indexes:
@@ -220,10 +225,10 @@ Kibana index patterns with versions that match the upgraded indexes:
3. Check the checkboxes for the `dmarc_aggregate` and `dmarc_failure`
index patterns
4. Click Delete
5. Click Delete on the conformation message
5. Click Delete on the confirmation message
6. Download (right-click the link and click save as)
the latest version of [export.ndjson]
7. Import `export.ndjson` by clicking Import from the Kibana
the latest version of [opensearch_dashboards.ndjson]
7. Import `opensearch_dashboards.ndjson` by clicking Import from the Kibana
Saved Objects page
## Backfilling the combined DKIM/SPF result fields
@@ -408,6 +413,6 @@ retention regulations such as GDPR. For more information,
check out the Elastic guide to [managing time-based indexes efficiently](https://www.elastic.co/blog/managing-time-based-indices-efficiently).
[elasticsearch]: https://www.elastic.co/guide/en/elasticsearch/reference/current/rpm.html
[export.ndjson]: https://raw.githubusercontent.com/domainaware/parsedmarc/master/dashboards/opensearch/opensearch_dashboards.ndjson
[opensearch_dashboards.ndjson]: https://raw.githubusercontent.com/domainaware/parsedmarc/master/dashboards/opensearch/opensearch_dashboards.ndjson
[kibana]: https://www.elastic.co/guide/en/kibana/current/rpm.html
[x-pack]: https://www.elastic.co/products/x-pack