[wiz] Add defend data stream #13688

muskan-agarwal26 · 2025-04-25T16:18:12Z

Proposed commit message

This release includes a defend data stream for supporting detection events forwarding via HTTP Endpoint and associated dashboards and visualizations.

Wiz fields are mapped to their corresponding ECS fields where possible.

Test samples were derived from documentation.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

Clone integrations repo.
Install elastic package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/wiz directory.
Run the following command to run tests.

elastic-package-test

--- Test results for package: wiz - START ---
╭─────────┬──────────────────────────────────────────┬───────────┬───────────────────────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE │ DATA STREAM                              │ TEST TYPE │ TEST NAME                                                                         │ RESULT │ TIME ELAPSED │
├─────────┼──────────────────────────────────────────┼───────────┼───────────────────────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ wiz     │                                          │ asset     │ dashboard wiz-3c3519be-f4f9-4c67-a9d8-1db4182b6e6a is loaded                      │ PASS   │      2.543µs │
│ wiz     │                                          │ asset     │ dashboard wiz-726802c0-4007-48b9-bae5-09daa69d4368 is loaded                      │ PASS   │        375ns │
│ wiz     │                                          │ asset     │ dashboard wiz-927c36f0-6358-11ee-a265-c3569aa0cebf is loaded                      │ PASS   │        383ns │
│ wiz     │                                          │ asset     │ dashboard wiz-be3fd3f0-6358-11ee-9db4-21f79f2e6273 is loaded                      │ PASS   │        390ns │
│ wiz     │                                          │ asset     │ dashboard wiz-d8f91a20-6363-11ee-a265-c3569aa0cebf is loaded                      │ PASS   │        400ns │
│ wiz     │                                          │ asset     │ search wiz-f71321c0-a641-4411-a33e-f39569c2c7be is loaded                         │ PASS   │        431ns │
│ wiz     │ audit                                    │ asset     │ index_template logs-wiz.audit is loaded                                           │ PASS   │        328ns │
│ wiz     │ audit                                    │ asset     │ ingest_pipeline logs-wiz.audit-3.2.0 is loaded                                    │ PASS   │        337ns │
│ wiz     │ cloud_configuration_finding              │ asset     │ index_template logs-wiz.cloud_configuration_finding is loaded                     │ PASS   │        229ns │
│ wiz     │ cloud_configuration_finding              │ asset     │ ingest_pipeline logs-wiz.cloud_configuration_finding-3.2.0 is loaded              │ PASS   │        310ns │
│ wiz     │ cloud_configuration_finding_full_posture │ asset     │ index_template logs-wiz.cloud_configuration_finding_full_posture is loaded        │ PASS   │        393ns │
│ wiz     │ cloud_configuration_finding_full_posture │ asset     │ ingest_pipeline logs-wiz.cloud_configuration_finding_full_posture-3.2.0 is loaded │ PASS   │        594ns │
│ wiz     │ defend                                   │ asset     │ index_template logs-wiz.defend is loaded                                          │ PASS   │        371ns │
│ wiz     │ defend                                   │ asset     │ ingest_pipeline logs-wiz.defend-3.2.0 is loaded                                   │ PASS   │        287ns │
│ wiz     │ issue                                    │ asset     │ index_template logs-wiz.issue is loaded                                           │ PASS   │        460ns │
│ wiz     │ issue                                    │ asset     │ ingest_pipeline logs-wiz.issue-3.2.0 is loaded                                    │ PASS   │        348ns │
│ wiz     │ vulnerability                            │ asset     │ index_template logs-wiz.vulnerability is loaded                                   │ PASS   │        527ns │
│ wiz     │ vulnerability                            │ asset     │ ingest_pipeline logs-wiz.vulnerability-3.2.0 is loaded                            │ PASS   │        419ns │
╰─────────┴──────────────────────────────────────────┴───────────┴───────────────────────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: wiz - END   ---
Done
--- Test results for package: wiz - START ---
╭─────────┬──────────────────────────────────────────┬───────────┬──────────────────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE │ DATA STREAM                              │ TEST TYPE │ TEST NAME                                                                    │ RESULT │ TIME ELAPSED │
├─────────┼──────────────────────────────────────────┼───────────┼──────────────────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ wiz     │ audit                                    │ pipeline  │ (ingest pipeline warnings test-audit.log)                                    │ PASS   │ 333.297311ms │
│ wiz     │ audit                                    │ pipeline  │ test-audit.log                                                               │ PASS   │ 169.366958ms │
│ wiz     │ cloud_configuration_finding              │ pipeline  │ (ingest pipeline warnings test-cloud-configuration-finding.log)              │ PASS   │ 320.289194ms │
│ wiz     │ cloud_configuration_finding              │ pipeline  │ test-cloud-configuration-finding.log                                         │ PASS   │ 201.247099ms │
│ wiz     │ cloud_configuration_finding_full_posture │ pipeline  │ (ingest pipeline warnings test-cloud-configuration-finding-full-posture.log) │ PASS   │ 321.040186ms │
│ wiz     │ cloud_configuration_finding_full_posture │ pipeline  │ test-cloud-configuration-finding-full-posture.log                            │ PASS   │ 206.083922ms │
│ wiz     │ defend                                   │ pipeline  │ (ingest pipeline warnings test-defend.json)                                  │ PASS   │ 387.764675ms │
│ wiz     │ defend                                   │ pipeline  │ test-defend.json                                                             │ PASS   │ 198.057461ms │
│ wiz     │ issue                                    │ pipeline  │ (ingest pipeline warnings test-issue.log)                                    │ PASS   │  348.22258ms │
│ wiz     │ issue                                    │ pipeline  │ test-issue.log                                                               │ PASS   │ 144.471151ms │
│ wiz     │ vulnerability                            │ pipeline  │ (ingest pipeline warnings test-vulnerability.log)                            │ PASS   │ 325.757313ms │
│ wiz     │ vulnerability                            │ pipeline  │ test-vulnerability.log                                                       │ PASS   │  234.06407ms │
╰─────────┴──────────────────────────────────────────┴───────────┴──────────────────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: wiz - END   ---
Done
--- Test results for package: wiz - START ---
No test results
--- Test results for package: wiz - END   ---
Done
--- Test results for package: wiz - START ---
╭─────────┬──────────────────────────────────────────┬───────────┬──────────────────────────┬────────┬──────────────╮
│ PACKAGE │ DATA STREAM                              │ TEST TYPE │ TEST NAME                │ RESULT │ TIME ELAPSED │
├─────────┼──────────────────────────────────────────┼───────────┼──────────────────────────┼────────┼──────────────┤
│ wiz     │ audit                                    │ static    │ Verify sample_event.json │ PASS   │ 113.858876ms │
│ wiz     │ cloud_configuration_finding              │ static    │ Verify sample_event.json │ PASS   │ 123.496495ms │
│ wiz     │ cloud_configuration_finding_full_posture │ static    │ Verify sample_event.json │ PASS   │ 122.217907ms │
│ wiz     │ defend                                   │ static    │ Verify sample_event.json │ PASS   │ 163.516688ms │
│ wiz     │ issue                                    │ static    │ Verify sample_event.json │ PASS   │ 136.472605ms │
│ wiz     │ vulnerability                            │ static    │ Verify sample_event.json │ PASS   │ 129.295156ms │
╰─────────┴──────────────────────────────────────────┴───────────┴──────────────────────────┴────────┴──────────────╯
--- Test results for package: wiz - END   ---
Done
--- Test results for package: wiz - START ---
╭─────────┬──────────────────────────────────────────┬───────────┬───────────────────────┬────────┬─────────────────╮
│ PACKAGE │ DATA STREAM                              │ TEST TYPE │ TEST NAME             │ RESULT │    TIME ELAPSED │
├─────────┼──────────────────────────────────────────┼───────────┼───────────────────────┼────────┼─────────────────┤
│ wiz     │ audit                                    │ system    │ default               │ PASS   │    45.60173321s │
│ wiz     │ cloud_configuration_finding              │ system    │ default               │ PASS   │ 2m12.569045322s │
│ wiz     │ cloud_configuration_finding_full_posture │ system    │ default               │ PASS   │   38.802608336s │
│ wiz     │ defend                                   │ system    │ http-endpoint-basic   │ PASS   │  2m8.851497915s │
│ wiz     │ defend                                   │ system    │ http-endpoint-no-auth │ PASS   │ 2m17.957363166s │
│ wiz     │ defend                                   │ system    │ http-endpoint-token   │ PASS   │ 2m17.895710817s │
│ wiz     │ issue                                    │ system    │ default               │ PASS   │    37.66306986s │
│ wiz     │ vulnerability                            │ system    │ default               │ PASS   │  4m4.630782448s │
╰─────────┴──────────────────────────────────────────┴───────────┴───────────────────────┴────────┴─────────────────╯
--- Test results for package: wiz - END   ---
Done

Related issues

- Closes Add support for Wiz Defend datastream #13269

Screenshots

elasticmachine · 2025-04-25T20:04:32Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

efd6 · 2025-04-28T04:21:50Z

/test

elastic-vault-github-plugin-prod · 2025-04-28T04:48:04Z

🚀 Benchmarks report

Package `wiz` 👍(2) 💚(0) 💔(4)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
`cloud_configuration_finding`	6097.56	3424.66	-2672.9 (-43.84%)	💔
`cloud_configuration_finding_full_posture`	11627.91	3731.34	-7896.57 (-67.91%)	💔
`issue`	4672.9	2890.17	-1782.73 (-38.15%)	💔
`vulnerability`	4237.29	2577.32	-1659.97 (-39.18%)	💔

To see the full report comment with /test benchmark fullreport

packages/wiz/_dev/build/docs/README.md

packages/wiz/_dev/deploy/docker/sample_logs/defend.log

packages/wiz/data_stream/audit/manifest.yml

...l_posture/_dev/test/pipeline/test-cloud-configuration-finding-full-posture.log-expected.json

packages/wiz/data_stream/defend/agent/stream/http_endpoint.yml.hbs

packages/wiz/data_stream/defend/fields/fields.yml

packages/wiz/data_stream/issue/manifest.yml

efd6 · 2025-04-28T04:57:44Z

packages/wiz/kibana/dashboard/wiz-726802c0-4007-48b9-bae5-09daa69d4368.json

@@ -37,7 +37,7 @@
                        "id": "",
                        "params": {
                            "fontSize": 12,
-                            "markdown": "Navigation\n\n[Wiz Cloud Configuration Finding (This page)](#/dashboard/wiz-726802c0-4007-48b9-bae5-09daa69d4368)\n\n[Wiz Vulnerability](#/dashboard/wiz-927c36f0-6358-11ee-a265-c3569aa0cebf)\n\n[Wiz Issue](#/dashboard/wiz-d8f91a20-6363-11ee-a265-c3569aa0cebf)\n\n[Wiz Audit](#/dashboard/wiz-be3fd3f0-6358-11ee-9db4-21f79f2e6273)\n\n[Integration Page](/app/integrations/detail/wiz/overview)\n\nOverview\n\nThis dashboard shows the Cloud Configuration Findings overview related to the Wiz Integration.\n\nThis dashboard provides general statistics and shows the detection of ingested cloud configuration findings.\n\nIt provides information about findings and assets. It also displays the distribution of findings according to evaluation results and contains details regarding the count of findings over time.",
+                            "markdown": "Navigation\n\nWiz Cloud Configuration Finding\n\n[Wiz Vulnerability](#/dashboard/wiz-927c36f0-6358-11ee-a265-c3569aa0cebf)\n\n[Wiz Issue](#/dashboard/wiz-d8f91a20-6363-11ee-a265-c3569aa0cebf)\n\n[Wiz Defend](#/dashboard/wiz-3c3519be-f4f9-4c67-a9d8-1db4182b6e6a)\n\n[Wiz Audit](#/dashboard/wiz-be3fd3f0-6358-11ee-9db4-21f79f2e6273)\n\n[Integration Page](/app/integrations/detail/wiz/overview)\n\nOverview\n\nThis dashboard shows the Cloud Configuration Findings overview related to the Wiz Integration.\n\nThis dashboard provides general statistics and shows the detection of ingested cloud configuration findings.\n\nIt provides information about findings and assets. It also displays the distribution of findings according to evaluation results and contains details regarding the count of findings over time.",


Why does the order of the navigation list change for each of these pages?

We have not changed any order here. As suggested in the earlier PRs, we have added all the missing dashboard links. Additionally, for the dashboard we are currently on, we have removed the "(This Page)" text and the link to it, as previously recommended.

Yeah, sorry, I should have been clearer. The order is the same as it was, but the previously existing order differs depending on which dashboard the user is on. I was wondering why that is the case. You may not know.

packages/wiz/data_stream/vulnerability/manifest.yml

efd6 · 2025-04-28T22:32:43Z

/test

1. Revert the cloud_configuration test file changes. 2. Replace the resources stmt as suggested.

efd6 · 2025-04-29T07:53:52Z

/test

efd6 · 2025-04-30T23:29:30Z

This looks good mod the conflicts that will need to be resolved.

efd6 · 2025-05-01T20:52:35Z

Conflicts don't appear to be correctly resolved.

muskan-agarwal26 · 2025-05-02T09:17:09Z

Resolved conflicts @efd6

kcreddy · 2025-05-05T05:51:29Z

/test

elasticmachine · 2025-05-05T06:19:39Z

💚 Build Succeeded

Buildkite Build
Commit: c007c53

History

💚 Build #25260 succeeded 350fafc
💚 Build #25238 succeeded 54877af
💚 Build #25186 succeeded 6bb9ddc

elastic-sonarqube · 2025-05-05T06:19:45Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
98.8% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

efd6

This is the commit message body for merging,

This adds a defend data stream for supporting detection events
forwarding via HTTP Endpoint and associated dashboards and
visualizations.

Wiz fields are mapped to their corresponding ECS fields where possible.

Test samples were derived from documentation and subsequently sanitized.

but I'd like to understand the last sentence. It says that the samples were obtained from documentation, so why were they sanitised?

muskan-agarwal26 · 2025-05-05T07:32:23Z

The values were sanitized because some, such as IP addresses, were not private and needed to be mocked.

elastic-vault-github-plugin-prod · 2025-05-09T07:37:22Z

Package wiz - 3.2.0 containing this change is available at https://epr.elastic.co/package/wiz/3.2.0/

Add defend data stream

14b0523

muskan-agarwal26 requested a review from a team as a code owner April 25, 2025 16:18

Update changelog entry

6bb9ddc

andrewkroh added dashboard Relates to a Kibana dashboard bug, enhancement, or modification. Integration:wiz Wiz Crest Contributions from Crest developement team. Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] labels Apr 25, 2025

efd6 reviewed Apr 28, 2025

View reviewed changes

Resolved comments by @efd6

54877af

muskan-agarwal26 requested a review from efd6 April 28, 2025 09:23

Resolve comments given by @efd6.

350fafc

1. Revert the cloud_configuration test file changes. 2. Replace the resources stmt as suggested.

Synced with main to resolve conflicts

e8176c9

Resolving conflicts by taking sync from the main branch

c007c53

efd6 reviewed May 5, 2025

View reviewed changes

muskan-agarwal26 requested a review from efd6 May 9, 2025 05:54

efd6 approved these changes May 9, 2025

View reviewed changes

efd6 merged commit 18321f8 into elastic:main May 9, 2025
7 checks passed

[wiz] Add defend data stream #13688

[wiz] Add defend data stream #13688

Uh oh!

Conversation

muskan-agarwal26 commented Apr 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

elasticmachine commented Apr 25, 2025

Uh oh!

efd6 commented Apr 28, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Apr 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Benchmarks report

Package wiz 👍(2) 💚(0) 💔(4)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

efd6 Apr 28, 2025

Choose a reason for hiding this comment

Uh oh!

muskan-agarwal26 Apr 28, 2025

Choose a reason for hiding this comment

Uh oh!

efd6 Apr 28, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

efd6 commented Apr 28, 2025

Uh oh!

efd6 commented Apr 29, 2025

Uh oh!

efd6 commented Apr 30, 2025

Uh oh!

efd6 commented May 1, 2025

Uh oh!

muskan-agarwal26 commented May 2, 2025

Uh oh!

kcreddy commented May 5, 2025

Uh oh!

elasticmachine commented May 5, 2025

💚 Build Succeeded

History

Uh oh!

elastic-sonarqube bot commented May 5, 2025

Quality Gate passed

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

muskan-agarwal26 commented May 5, 2025

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod bot commented May 9, 2025

Uh oh!

Uh oh!

muskan-agarwal26 commented Apr 25, 2025 •

edited

Loading

elastic-vault-github-plugin-prod bot commented Apr 28, 2025 •

edited

Loading

Package `wiz` 👍(2) 💚(0) 💔(4)