Add new Wiz Cloud Configuration Finding Full Posture data stream and update misconfig transform to use it #12961

maxcold · 2025-03-05T09:57:33Z

Proposed commit message

Adding a new Wiz Cloud Configuration Finding Full Posture data stream, which in contrast to the existing Wiz Cloud Configuration Finding data stream, ingests all Cloud Configuration Finding data every 24h to match the logic of the native Cloud Security Posture integration used by cloudbeat. Updated latest misconfigurations transform to use this new data stream as a source. Due to the change in the transform, this might be a breaking change for customers, they need to enable the new data stream when updating to still receive findings in their Findings > Misconfigurations view and other flows relying on cloud security posture data

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

[ ]

How to test this PR locally

build package with elastic-package build
start the stack corresponding to the Kibana version from the manifest with elastic-package stack up
Install the integration and ingest Wiz data, findings from Wiz should appear in Findings > Misconfiguration

Related issues

Related to:

Implement full posture data stream for Wiz Cloud Configuration Finding data kibana#208935

Screenshots

…base latest misconfig findings on it

elasticmachine · 2025-03-17T01:24:35Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

efd6

Test expectations need to be updated.

efd6 · 2025-03-17T22:53:33Z

packages/wiz/changelog.yml

@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "3.0.0"
+  changes:
+    - description: Add new Cloud Configiruation Finding Full Posture data stream and use it as a source for latest_cdr_misconfigurations transform. If you had Cloud Configiruation Finding enabled and relied on Findings > Misconfigurations view, you need to enable the Cloud Configiruation Finding Full Posture data stream.


This is too long for a changelog entry. Also, spelling.

packages/wiz/data_stream/audit/manifest.yml

packages/wiz/data_stream/cloud_configuration_finding/manifest.yml

packages/wiz/data_stream/cloud_configuration_finding_full_posture/agent/stream/cel.yml.hbs

efd6 · 2025-03-17T22:59:27Z

packages/wiz/data_stream/cloud_configuration_finding_full_posture/fields/resource.yml

Do these need to be in their own files? (following three files also)

I took this approach from our native Cloud Security Posture integration https://github.com/elastic/integrations/tree/main/packages/cloud_security_posture/data_stream/findings/fields
Find it easier to navigate when split by ECS part. Is there a convention around that?

I can't think of any other integrations of ours that do this.

any specific issue with it?

Just to clarify, I can move everything to one or a smaller number of files if you think it is a hard requirement. I just don't understand the guidance here. I see different approaches in different data streams and am not sure what is the preferred way and why. For example, just took a random data stream: https://github.com/elastic/integrations/tree/main/packages/windows/data_stream/sysmon_operational/fields. What is the logic of the split by files there? I'm not sure. I haven't had issues with the split we have in cloud_security_posture integration; that's why the question

packages/wiz/data_stream/cloud_configuration_finding_full_posture/manifest.yml

packages/wiz/elasticsearch/transform/latest_cdr_misconfigurations/fields/base-fields.yml

efd6 · 2025-03-17T23:04:09Z

packages/wiz/manifest.yml

 description: Collect logs from Wiz with Elastic Agent.
 type: integration
 categories:
  - security
  - cloudsecurity_cdr
 conditions:
  kibana:
-    version: "^8.16.0 || ^9.0.0"
+    version: "~8.16.6 || ~8.17.4 || ^8.18.0 || ^9.0.0"


Can you explain why this complexity is required?

When adding a new data stream as a source for the transform we need a change in Elasticsearch, see related PR elastic/elasticsearch#124074 This change was backported to 8.16.x, 8.17.x, that's why it will only work starting from 8.16.6 and 8.17.4

But wouldn't this change make make users believe agentless is compatible on ~8.16.6 || ~8.17.4?
Because we are merging the PR into main which had agentless change: #13105

@kcreddy this is a good question, thanks for bringing it up. I will ask the agentless team starting from which version we support the deployment_modes . But even if we don't support them in 8.16 and 8.17 which we probably don't, I wonder if smth breaks because of this new addition to the manifest, eg. validation. If agentless options just won't be available, I think it should be fine, Agentless feature by itself is going GA in 8.18, so it should be clear for the users.
But ofc we can go with only 8.18/9.0 for this change, it will just reduce the adoption

The deployment_mode was added in package-spec and Fleet support in 8.16, in 8.17 it was add for the Fleet extensions like CSPM.

I tested and 8.16.6-SNAPSHOT works fine with the changes I have here. Plus the mode itself was added in 8.16 #12961 (comment)

Sounds good. Thanks for testing @maxcold 👍🏼

This reverts commit d3ad0ea.

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

maxcold · 2025-03-20T13:02:51Z

@efd6

Test expectations need to be updated.

did you mean adding system tests for the new data stream or smth else?

efd6 · 2025-03-24T21:45:14Z

/test

...ta_stream/cloud_configuration_finding_full_posture/elasticsearch/ingest_pipeline/default.yml

...l_posture/_dev/test/pipeline/test-cloud-configuration-finding-full-posture.log-expected.json

packages/wiz/_dev/deploy/docker/docker-compose.yml

packages/wiz/_dev/deploy/docker/files/config-cloud_configuration_finding_full_psoture.yml

packages/wiz/changelog.yml

elastic-vault-github-plugin-prod · 2025-03-25T13:16:05Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

packages/wiz/_dev/build/docs/README.md

packages/wiz/elasticsearch/transform/latest_cdr_misconfigurations/transform.yml

kcreddy · 2025-03-26T09:39:49Z

packages/wiz/manifest.yml

 description: Collect logs from Wiz with Elastic Agent.
 type: integration
 categories:
  - security
  - cloudsecurity_cdr
 conditions:
  kibana:
-    version: "^8.16.0 || ^9.0.0"
+    version: "~8.16.6 || ~8.17.4 || ^8.18.0 || ^9.0.0"


But wouldn't this change make make users believe agentless is compatible on ~8.16.6 || ~8.17.4?
Because we are merging the PR into main which had agentless change: #13105

Co-authored-by: Krishna Chaitanya Reddy Burri <krish.reddy91@gmail.com>

seanrathier

LGTM. Let me know when you finish testing and I will review again.

packages/wiz/elasticsearch/transform/latest_cdr_misconfigurations/transform.yml

seanrathier · 2025-03-26T14:36:41Z

packages/wiz/manifest.yml

 description: Collect logs from Wiz with Elastic Agent.
 type: integration
 categories:
  - security
  - cloudsecurity_cdr
 conditions:
  kibana:
-    version: "^8.16.0 || ^9.0.0"
+    version: "~8.16.6 || ~8.17.4 || ^8.18.0 || ^9.0.0"


The deployment_mode was added in package-spec and Fleet support in 8.16, in 8.17 it was add for the Fleet extensions like CSPM.

kcreddy

Just wanted a clarification on the approach if we really need a new datastream. otherwise it looks good.

kcreddy · 2025-03-26T18:38:21Z

packages/wiz/data_stream/cloud_configuration_finding_full_posture/agent/stream/cel.yml.hbs

@@ -0,0 +1,149 @@
+config_version: 2
+interval: 24h


If I look at this CEL program, it is almost identical to its corresponding incremental data stream cloud_configuration_finding. There are few differences such as hardcoded interval, removing incremental data fetching using analyzedAt, and removing handling of cursor.

What if we present user with an option inside the existing data stream cloud_configuration_finding itself, such as a boolean variable full_pull which defaults to true, and let the user choose full or incremental payloads?
Was this already thought about? Are there any issues you foresaw? It seems the only reason we are adding this new data stream is to match it with Cloud Security Posture style 24h pull, and the data is exactly same as existing cloud_configuration_finding.

The problem with multiple data streams handling same API is in maintenance during enhancement/bugfixes which needs to be propagated to multiple places now (which sometimes not all developers may know).

@kcreddy thanks for bringing it up! yes, this indeed is the challenge we discussed with product and opted out for a separate data stream due to the following challenges:

integration default UX can become very confusing when different options are presented to users and it's not clear how they work in combination. For example, if we present the full posture option on the existing data stream, it's not possible to hide other params (initial lookback interval and period interval) . So it will be confusing for the users how do they relate to each other and if they work together. Ofc we can solve it in the docs and in the field descriptions but it's not a good UX

another problem is that the data from incremental and full posture are two separate usecases solving different problems. Users might want to have them both potentially. Incremental to base their alerts on and full posture to enable our Cloud Security flows (Findings page and context in different places of Security solution). Adding the option to existing data stream will make users to chose one flow, which might be limiting. Ofc it is probably possible to solve all with a more complex CEL logic, but in my opinion it defeats this purpose The problem with multiple data streams handling same API is in maintenance during enhancement/bugfixes which needs to be propagated to multiple places now - overly complex data stream with two modes that can coexist or work separately is very hard to maintain without incidental bugs in my opinion.

I think to add onto good UX, maybe we should also change the title and description of existing cloud_configuration_finding data stream? Do you think Full posture vs Incremental posture are self-intuitive or do you think documenting the difference in the integration docs makes more sense?

Also, is this going to be applied to all existing and future implementations, such as AWS Security Hub, which pulls incrementally?

How about vulnerability flows?

@smriti0321

I think to add onto good UX, maybe we should also change the title and description of existing cloud_configuration_finding data stream? Do you think Full posture vs Incremental posture are self-intuitive or do you think documenting the difference in the integration docs makes more sense?

Do we want to change the existing data stream to state clearly "Incremental"? Also what do we do about the documentation, do we want to document that users need to enable this new data stream if they want to leverage CDR flows?

@kcreddy , yes, the plan is to implement similar for AWS Security Hub. For vulnerability flows we decided not to deal with the problem of incremental data flow for now. The posture much more relevant for CSPM data (Misconfiguration Fingings). Even though it's nice to have the full data of all the vulnerabilities in the system (and as we know our InfoSec team pulls the whole data set from Qualys) , it's less critical

efd6 · 2025-03-27T22:14:01Z

packages/wiz/_dev/deploy/docker/files/config-cloud_configuration_finding_full_posture.yml

+          Content-Type:
+            - application/json
+        body: |
+          {"data": {"configurationFindings": {"nodes": [{"analyzedAt":"2024-08-07T12:55:52.012378Z","id":"1243196d-a365-589a-a8aa-13817c9877b2","remediation":null,"resource":{"id":"f0f4163d-cbd7-517c-ba9e-f96bb90ab5ea","name":"Root user","nativeType":"rootUser","providerId":"arn:aws:iam::998231069301:root","region":null,"cloudPlatform":"EKS","subscription":{"cloudProvider":"AWS","externalId":"998231069301","id":"94e76baa-85fd-5928-b829-1669a2ca9660","name":"wiz-integrations"},"tags":[],"type":"USER_ACCOUNT"},"result":"PASS","rule":{"description":"This rule checks if the AWS Root Account has access keys. \nThis rule fails if `AccountAccessKeysPresent` is not set to `0`. Note that it does not take into consideration the status of the keys if present. \nThe root account should avoid using access keys. Since the root account has full permissions across the entire account, creating access keys for it increases the chance that they will be compromised. Instead, it is recommended to create IAM users with predefined roles.\n>**Note** \nSee Cloud Configuration Rule `IAM-207` to see if the Root account's access keys are active.","id":"563ed717-4fb6-47fd-929e-9c794e201d0a","name":"Root account access keys should not exist","remediationInstructions":"Perform the following steps, while being signed in as the Root user, in order to delete the root user's access keys via AWS CLI: \n1. Use the following command to list the Root user's access keys. \nCopy the `AccessKeyId` from the output and paste it into the `access-key-id` value in the next step. \n```\naws iam list-access-keys\n```\n2. Use the following command to delete the access key(s). \n```\naws iam delete-access-key /\n --access-key-id <value>\n```\n>**Note** \nOnce an access key is removed, any application using it will not work until a new one is configured for it.","shortId":"IAM-006"},"severity":"MEDIUM"}],"pageInfo": {"hasNextPage": true,"endCursor": "eyJmaWVsZHMiOlt7IkZpZWxkIjoiVGltZXN0YW1wIiwiVmFsdWUiOiIyMDIzLTA5LTA0VDExOjE5OjM3LjgwMTU0MVoifV19"}}}}


For the same of future maintenance, please pretty print the object and use the minify_json helper (example). Also below.

packages/wiz/data_stream/cloud_configuration_finding_full_posture/agent/stream/cel.yml.hbs

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

kcreddy

LGTM.
The reasons behind adding a new data-stream is here: #12961 (comment)

Can wait for @efd6 approval before merging.

efd6

One fix required, two nits that you can take or leave.

packages/wiz/_dev/deploy/docker/files/config-cloud_configuration_finding_full_posture.yml

packages/wiz/data_stream/cloud_configuration_finding_full_posture/manifest.yml

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

elastic-sonarqube · 2025-04-02T09:37:33Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
91.3% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

elasticmachine · 2025-04-02T09:37:46Z

💚 Build Succeeded

Buildkite Build
Commit: e696e49

History

💚 Build #24089 succeeded 29c426a
💔 Build #24088 failed 147e36d
💚 Build #23902 succeeded c01cc6b
💔 Build #23884 failed 4124f36
💚 Build #23812 succeeded 70cfe15
💔 Build #23796 failed d85a5ed

elastic-vault-github-plugin-prod · 2025-04-07T12:21:32Z

Package wiz - 3.0.0 containing this change is available at https://epr.elastic.co/package/wiz/3.0.0/

add new Wiz Cloud Configuration Finding Full Posture data stream and …

7a49e16

…base latest misconfig findings on it

maxcold added enhancement New feature or request Team:Cloud Security Cloud Security team [elastic/cloud-security-posture] labels Mar 5, 2025

maxcold requested a review from a team March 5, 2025 09:57

maxcold added 3 commits March 5, 2025 11:02

format and lint

d3ad0ea

bump major version

02f9ba4

fix linting

ee303ca

andrewkroh added the Integration:wiz Wiz label Mar 13, 2025

fix field mapping

1e42085

maxcold added breaking change and removed enhancement New feature or request labels Mar 14, 2025

Merge branch 'main' into csp-poc-wiz-full-posture

4133e56

maxcold marked this pull request as ready for review March 14, 2025 14:58

maxcold requested a review from a team as a code owner March 14, 2025 14:58

andrewkroh added the Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] label Mar 17, 2025

efd6 reviewed Mar 17, 2025

View reviewed changes

maxcold and others added 7 commits March 20, 2025 13:35

Revert "format and lint"

84b7eb0

This reverts commit d3ad0ea.

update changelog entry and add missing new line

5bb0ce9

Update resource.tracer settings

ccf7569

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

Update error.message generation logic

ceb8604

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

Remove explicit bytes check

5cef018

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

Merge branch 'main' into csp-poc-wiz-full-posture

4f3b74f

add system tests and update docs

5ba9f45

maxcold added 2 commits March 21, 2025 14:18

update readme and test output

deb0241

make @timestamp dynamic for pipeline tests

5ce27a6

efd6 reviewed Mar 24, 2025

View reviewed changes

efd6 reviewed Mar 25, 2025

View reviewed changes

packages/wiz/changelog.yml Show resolved Hide resolved

maxcold added 3 commits March 25, 2025 11:15

fix system tests

d85a5ed

Merge branch 'main' into csp-poc-wiz-full-posture

4b56cd6

fix system tests config

70cfe15

maxcold requested a review from efd6 March 25, 2025 16:52

kcreddy reviewed Mar 26, 2025

View reviewed changes

maxcold and others added 2 commits March 26, 2025 14:13

Update packages/wiz/_dev/build/docs/README.md

cebe47e

Co-authored-by: Krishna Chaitanya Reddy Burri <krish.reddy91@gmail.com>

bump transform version

4124f36

seanrathier reviewed Mar 26, 2025

View reviewed changes

fix fields in README

c01cc6b

kcreddy reviewed Mar 26, 2025

View reviewed changes

efd6 reviewed Mar 27, 2025

View reviewed changes

maxcold and others added 4 commits March 31, 2025 13:40

fix code formatting

147e36d

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

pretty print json body in system tests

9de6f8e

Merge branch 'main' into csp-poc-wiz-full-posture

538ad66

fix special symbols

29c426a

maxcold requested review from efd6, seanrathier and kcreddy March 31, 2025 13:22

seanrathier approved these changes Mar 31, 2025

View reviewed changes

kcreddy approved these changes Apr 1, 2025

View reviewed changes

efd6 approved these changes Apr 1, 2025

View reviewed changes

maxcold and others added 2 commits April 2, 2025 11:11

Fix spelling

a4cb70a

Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>

fix nits

e696e49

maxcold mentioned this pull request Apr 3, 2025

Add new AWS Security Hub Findings Full Posture data stream and update misconfig transform to use it #13372

Merged

5 tasks

maxcold merged commit 05c6dba into elastic:main Apr 7, 2025
7 checks passed

maxcold deleted the csp-poc-wiz-full-posture branch April 7, 2025 12:01

Add new Wiz Cloud Configuration Finding Full Posture data stream and update misconfig transform to use it #12961

Add new Wiz Cloud Configuration Finding Full Posture data stream and update misconfig transform to use it #12961

Uh oh!

Conversation

maxcold commented Mar 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

elasticmachine commented Mar 17, 2025

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maxcold commented Mar 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

efd6 commented Mar 24, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elastic-vault-github-plugin-prod bot commented Mar 25, 2025

🚀 Benchmarks report

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

seanrathier left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kcreddy left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maxcold commented Mar 5, 2025 •

edited

Loading

maxcold commented Mar 20, 2025 •

edited

Loading

kcreddy left a comment •

edited

Loading

kcreddy Mar 26, 2025 •

edited

Loading

kcreddy left a comment •

edited

Loading