Skip to content

[Cloud Security] increase transfom retention_policy to 90 days for Wiz #11393

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Oct 14, 2024
Merged

[Cloud Security] increase transfom retention_policy to 90 days for Wiz #11393

merged 2 commits into from
Oct 14, 2024

Conversation

maxcold
Copy link
Contributor

@maxcold maxcold commented Oct 10, 2024

Proposed commit message

Increase retention policy to 90 days on Wiz transforms to cater for how Wiz evaluates environments for posture. In contrast to native CSP integration Wiz doesn't do regular full re-evaluation of customer environments, that's why short retention period mean customers having only a small slice of their posture. The change will go together with documentation on how to get more complete view for the posture using longer initial interval setting value

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

  • [ ]

How to test this PR locally

Build and install integration, ingest some data from Wiz, check that latest index contains documents without 90d period

Related issues

Screenshots

@maxcold maxcold added enhancement New feature or request Team:Cloud Security Cloud Security team [elastic/cloud-security-posture] Integration:wiz Wiz labels Oct 10, 2024
@maxcold maxcold requested a review from a team October 10, 2024 14:23
@elastic-vault-github-plugin-prod

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

@elasticmachine
Copy link

💚 Build Succeeded

Copy link

maxcold added a commit to elastic/kibana that referenced this pull request Oct 11, 2024
…party data loading (#195636)

## Summary

Increase retention on Cloud Security queries to accommodate a longer
retention period on third-party CDR integrations, such as Wiz and AWS
SecurityHub. This introduces regression for
#142198
This is meant is a temporary workaround until we find a robust way to
get full posture for third-party CDR integrations
This change goes together with increasing retention period on Wiz: 
- elastic/integrations#11393

fixes:
- elastic/security-team#10683

## How to test
The CI deployed a serverless project where I installed Wiz and CSP
integrations and ingested some data.
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Oct 11, 2024
…party data loading (elastic#195636)

## Summary

Increase retention on Cloud Security queries to accommodate a longer
retention period on third-party CDR integrations, such as Wiz and AWS
SecurityHub. This introduces regression for
elastic#142198
This is meant is a temporary workaround until we find a robust way to
get full posture for third-party CDR integrations
This change goes together with increasing retention period on Wiz:
- elastic/integrations#11393

fixes:
- elastic/security-team#10683

## How to test
The CI deployed a serverless project where I installed Wiz and CSP
integrations and ingested some data.

(cherry picked from commit e18c52e)
@maxcold maxcold marked this pull request as ready for review October 11, 2024 11:08
@maxcold maxcold requested a review from a team as a code owner October 11, 2024 11:08
@andrewkroh andrewkroh added the Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] label Oct 11, 2024
@elasticmachine
Copy link

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

Copy link
Contributor

@chrisberkhout chrisberkhout left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change looks good.

If the full set of useful data can include old records for which there aren't periodic updates, consider entirely removing this kind of expiry based on ingest time.

@maxcold
Copy link
Contributor Author

maxcold commented Oct 14, 2024

@chrisberkhout we still need the retention, it's more of temporary workaround of the issue described in this work item

@maxcold maxcold merged commit 86f12ab into elastic:main Oct 14, 2024
5 checks passed
@maxcold maxcold deleted the csp-wiz-increase-retention-policy-transforms branch October 14, 2024 09:11
@elastic-vault-github-plugin-prod

Package wiz - 1.9.0-preview05 containing this change is available at https://epr.elastic.co/search?package=wiz

@elastic-vault-github-plugin-prod

Package wiz - 2.0.0 containing this change is available at https://epr.elastic.co/search?package=wiz

harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 4, 2025
elastic#11393)

* increase transfom retention_policy to 90 days for Wiz

* update changelog
harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 5, 2025
elastic#11393)

* increase transfom retention_policy to 90 days for Wiz

* update changelog
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request Integration:wiz Wiz Team:Cloud Security Cloud Security team [elastic/cloud-security-posture] Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants