[proofpoint_itm] Initial release of Proofpoint ITM #13153

muskan-agarwal26 · 2025-03-18T07:18:37Z

Proposed commit message

Added report data stream.
Added data collection logic for report data stream.
Added the ingest pipeline for report data streams.
Mapped fields according to the ECS schema and added Fields metadata in the appropriate yml files.
Added dashboard and visualizations.
Added test for pipeline for report data stream.
Added system test cases for report data stream.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

Clone integrations repo.
Install elastic package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/proofpoint_itm directory.
Run the following command to run tests.

elastic-package test

--- Test results for package: proofpoint_itm - START ---
╭────────────────┬─────────────┬───────────┬─────────────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE        │ DATA STREAM │ TEST TYPE │ TEST NAME                                                               │ RESULT │ TIME ELAPSED │
├────────────────┼─────────────┼───────────┼─────────────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ proofpoint_itm │             │ asset     │ dashboard proofpoint_itm-29f45960-20a5-4ca4-a40a-ad1504eb8138 is loaded │ PASS   │        1.2µs │
│ proofpoint_itm │ report      │ asset     │ index_template logs-proofpoint_itm.report is loaded                     │ PASS   │        283ns │
│ proofpoint_itm │ report      │ asset     │ ingest_pipeline logs-proofpoint_itm.report-0.1.0 is loaded              │ PASS   │         95ns │
╰────────────────┴─────────────┴───────────┴─────────────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: proofpoint_itm - END   ---
Done
Run pipeline tests for the package
--- Test results for package: proofpoint_itm - START ---
╭────────────────┬─────────────┬───────────┬───────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE        │ DATA STREAM │ TEST TYPE │ TEST NAME                                 │ RESULT │ TIME ELAPSED │
├────────────────┼─────────────┼───────────┼───────────────────────────────────────────┼────────┼──────────────┤
│ proofpoint_itm │ report      │ pipeline  │ (ingest pipeline warnings test-alert.log) │ PASS   │ 710.451542ms │
│ proofpoint_itm │ report      │ pipeline  │ test-alert.log                            │ PASS   │ 209.903776ms │
╰────────────────┴─────────────┴───────────┴───────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: proofpoint_itm - END   ---
Done
Run policy tests for the package
--- Test results for package: proofpoint_itm - START ---
No test results
--- Test results for package: proofpoint_itm - END   ---
Done
Run static tests for the package
--- Test results for package: proofpoint_itm - START ---
╭────────────────┬─────────────┬───────────┬──────────────────────────┬────────┬──────────────╮
│ PACKAGE        │ DATA STREAM │ TEST TYPE │ TEST NAME                │ RESULT │ TIME ELAPSED │
├────────────────┼─────────────┼───────────┼──────────────────────────┼────────┼──────────────┤
│ proofpoint_itm │ report      │ static    │ Verify sample_event.json │ PASS   │ 189.061828ms │
╰────────────────┴─────────────┴───────────┴──────────────────────────┴────────┴──────────────╯
--- Test results for package: proofpoint_itm - END   ---
Done
Run system tests for the package
2025/03/18 12:46:18  INFO License text found in "/root/github/integrations/LICENSE.txt" will be included in package
2025/03/18 12:47:18  INFO Write container logs to file: /root/github/integrations/build/container-logs/proofpoint_itm-1742282238369280668.log
2025/03/18 12:47:29  INFO Write container logs to file: /root/github/integrations/build/container-logs/elastic-agent-1742282249272532443.log
--- Test results for package: proofpoint_itm - START ---
╭────────────────┬─────────────┬───────────┬───────────┬────────┬───────────────╮
│ PACKAGE        │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │  TIME ELAPSED │
├────────────────┼─────────────┼───────────┼───────────┼────────┼───────────────┤
│ proofpoint_itm │ report      │ system    │ default   │ PASS   │ 45.055908306s │
╰────────────────┴─────────────┴───────────┴───────────┴────────┴───────────────╯
--- Test results for package: proofpoint_itm - END   ---
Done

Screenshots

kcreddy · 2025-03-18T08:04:49Z

/test

elasticmachine · 2025-03-18T08:05:06Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

elastic-vault-github-plugin-prod · 2025-03-18T08:21:13Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

packages/proofpoint_itm/_dev/build/docs/README.md

packages/proofpoint_itm/_dev/deploy/docker/files/config.yml

efd6 · 2025-03-18T22:11:52Z

packages/proofpoint_itm/_dev/deploy/docker/files/config.yml

This test only exercises a single API collections. We should make it collect at least two in order to test pagination.

Since we have only 2 query params, i.e, since, and limit, we are unable to have pagination call in system test.

That's unfortunate. Can you confirm that it has been tested against a real endpoint then?

I'm not convinced that the situation here prevents that. If you have a risingValue timestamp that cannot be reasonably expected to exist, say "3025-03-01T12:00:00Z", then you can filter the request query on since starting with a 2 and have the second rule filter on starting with a 3.

packages/proofpoint_itm/_dev/deploy/docker/files/config.yml

packages/proofpoint_itm/data_stream/report/_dev/test/pipeline/test-alert.log

packages/proofpoint_itm/data_stream/report/agent/stream/cel.yml.hbs

efd6 · 2025-03-19T07:53:14Z

/test

efd6 · 2025-03-20T06:42:33Z

/test

elasticmachine · 2025-03-20T06:59:38Z

💚 Build Succeeded

Buildkite Build
Commit: d3cb28f

History

💚 Build #23543 succeeded 9736a78
💚 Build #23497 succeeded 87b12b4

elastic-sonarqube · 2025-03-20T06:59:46Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
97.6% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

efd6 · 2025-03-20T07:18:59Z

@muskan-agarwal26 Before merge, can you note where the test data input cases were obtained from? For both the system and pipeline tests.

Proposed commit message body:

Add report data stream including collection and ingest logic with mapping to
the ECS, and dashboard visualisations.

Pipeline and system tests added with test inputs coming from <TBD>.

with <TBD> to be added when known.

muskan-agarwal26 · 2025-03-20T09:36:23Z

@efd6
Pipeline and system test data have been extracted from Splunk and sanitized to ensure data integrity

elastic-vault-github-plugin-prod · 2025-03-20T10:02:00Z

Package proofpoint_itm - 0.1.0 containing this change is available at https://epr.elastic.co/package/proofpoint_itm/0.1.0/

Add report data stream including collection and ingest logic with mapping to the ECS, and dashboard visualisations. Pipeline and system test data have been extracted from Splunk and sanitized to ensure data integrity.

muskan-crest added 2 commits March 18, 2025 12:42

Initial release of Proofpoint ITM

55b5728

Update changelog entry and codeowners

87b12b4

kcreddy added Crest Contributions from Crest developement team. Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels Mar 18, 2025

kcreddy added New Integration Issue or pull request for creating a new integration package. dashboard Relates to a Kibana dashboard bug, enhancement, or modification. labels Mar 18, 2025

andrewkroh added the Integration:proofpoint_itm Proofpoint ITM label Mar 18, 2025

efd6 reviewed Mar 18, 2025

View reviewed changes

Resolved comments provided by @efd6

9736a78

muskan-agarwal26 requested a review from efd6 March 19, 2025 07:36

muskan-crest added 2 commits March 20, 2025 12:04

Added pagination test call in system test - comments by @efd6

3652049

Update readme

d3cb28f

efd6 approved these changes Mar 20, 2025

View reviewed changes

muskan-agarwal26 requested a review from efd6 March 20, 2025 09:36

efd6 merged commit 6962257 into elastic:main Mar 20, 2025
7 checks passed

efd6 mentioned this pull request Apr 8, 2025

ssi: add support for request trace deletion #13002

Open

33 tasks

[proofpoint_itm] Initial release of Proofpoint ITM #13153

[proofpoint_itm] Initial release of Proofpoint ITM #13153

Uh oh!

Conversation

muskan-agarwal26 commented Mar 18, 2025

Proposed commit message

Checklist

How to test this PR locally

Screenshots

Uh oh!

kcreddy commented Mar 18, 2025

Uh oh!

elasticmachine commented Mar 18, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Mar 18, 2025

🚀 Benchmarks report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

efd6 Mar 18, 2025

Choose a reason for hiding this comment

Uh oh!

muskan-agarwal26 Mar 19, 2025

Choose a reason for hiding this comment

Uh oh!

efd6 Mar 19, 2025

Choose a reason for hiding this comment

Uh oh!

efd6 Mar 19, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

efd6 commented Mar 19, 2025

Uh oh!

efd6 commented Mar 20, 2025

Uh oh!

elasticmachine commented Mar 20, 2025

💚 Build Succeeded

History

Uh oh!

elastic-sonarqube bot commented Mar 20, 2025

Quality Gate passed

Uh oh!

efd6 commented Mar 20, 2025

Uh oh!

muskan-agarwal26 commented Mar 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod bot commented Mar 20, 2025

Uh oh!

Uh oh!

muskan-agarwal26 commented Mar 20, 2025 •

edited

Loading