Add new AWS Config datastream. #13830

muskan-agarwal26 · 2025-05-07T16:27:00Z

Proposed commit message

This release includes the Config data stream for supporting config findings via REST API.

AWS Config fields are mapped to their corresponding ECS fields where possible. Also added associated dashboards and visualizations.

Test samples were derived from live logs.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

Clone integrations repo.
Install elastic package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/aws directory.
Run the following command to run tests.

elastic-package test

2025/05/07 21:50:34  INFO New version is available - v0.111.0. Download from: https://github.com/elastic/elastic-package/releases/tag/v0.111.0
Run system tests for the package
2025/05/07 21:50:35  INFO License text found in "/root/ELASTIC_PROJECT/github/integrations/LICENSE.txt" will be included in package
2025/05/07 21:51:45  INFO Write container logs to file: /root/ELASTIC_PROJECT/github/integrations/build/container-logs/config-1746634905423590657.log
2025/05/07 21:51:50  INFO Write container logs to file: /root/ELASTIC_PROJECT/github/integrations/build/container-logs/elastic-agent-1746634910377459951.log
--- Test results for package: aws - START ---
╭─────────┬─────────────┬───────────┬───────────┬────────┬───────────────╮
│ PACKAGE │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │  TIME ELAPSED │
├─────────┼─────────────┼───────────┼───────────┼────────┼───────────────┤
│ aws     │ config      │ system    │ default   │ PASS   │ 55.030876912s │
╰─────────┴─────────────┴───────────┴───────────┴────────┴───────────────╯
--- Test results for package: aws - END   ---
Done

Related issues

Closes [New Integration] AWS Config #10272

Screenshots

elasticmachine · 2025-05-07T20:04:39Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

kcreddy · 2025-05-07T20:49:43Z

/test

…aws-3.4.0

kcreddy · 2025-05-13T15:56:20Z

/test

packages/aws/changelog.yml

packages/aws/data_stream/config/_dev/deploy/docker/files/config.yml

packages/aws/data_stream/config/_dev/test/pipeline/test-common-config.yml

efd6 · 2025-05-13T22:46:56Z

packages/aws/_dev/build/docs/config.md

+### Agentless Enabled Integration
+Agentless integrations allow you to collect data without having to manage Elastic Agent in your cloud. They make manual agent deployment unnecessary, so you can focus on your data instead of the agent that collects it. For more information, refer to [Agentless integrations](https://www.elastic.co/guide/en/serverless/current/security-agentless-integrations.html) and the [Agentless integrations FAQ](https://www.elastic.co/guide/en/serverless/current/agentless-integration-troubleshooting.html).
+Agentless deployments are only supported in Elastic Serverless and Elastic Cloud environments. This functionality is in beta and is subject to change. Beta features are not subject to the support SLA of official GA features.
+
+### Agent Based Installation
+- Elastic Agent must be installed
+- You can install only one Elastic Agent per host.
+- Elastic Agent is required to stream data from the REST API and ship the data to Elastic, where the events will then be processed via the integration's ingest pipelines.


I think we need guidance from docs for how to provide this information.

packages/aws/data_stream/config/agent/stream/cel.yml.hbs

packages/aws/data_stream/config/elasticsearch/ingest_pipeline/default.yml

efd6 · 2025-05-13T23:24:33Z

Also, need to add owner.

Error: package "packages/aws" shares ownership across data streams but these ones [packages/aws/data_stream/config] lack owners

1. Add config data stream entry in CODEOWNERS. 2. Update changelog entry. 3. Format CEL code as suggested. 4. Added fields into redact. 5. Changed transform version v1. 6. Used minify_json in config.yml file.

efd6 · 2025-05-14T22:22:44Z

/test

elastic-vault-github-plugin-prod · 2025-05-15T00:05:31Z

🚀 Benchmarks report

Package `aws` 👍(7) 💚(12) 💔(2)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
`s3access`	4524.89	2375.3	-2149.59 (-47.51%)	💔
`emr_logs`	22727.27	17241.38	-5485.89 (-24.14%)	💔

To see the full report comment with /test benchmark fullreport

efd6 · 2025-05-15T00:26:08Z

Please address #13830 (comment).

...elasticsearch/transform/latest_config_misconfigurations/fields/is-transform-source-false.yml

packages/aws/data_stream/config/fields/is-transform-source-true.yml

kcreddy · 2025-05-15T06:27:57Z

packages/aws/data_stream/config/elasticsearch/ilm/default_policy.json

+			"delete": {
+				"min_age": "7d",
+				"actions": {
+					"delete": {}
+				}
+			}
+		}


@maxcold, similar to this comment, this ILM policy is for security posture. Please confirm if the 7 days retention is okay on source data.

Also, this PR is adding the transform. Please confirm if you would like to test this before merging?

cc: @nick-alayil

I think it definitely makes sense to test it before merging
On the ILM policy - if we ingest the full posture every 24h, then 7d retention for the source index is fine, we don't need old data in the source index.
For vulnerability data stream though it might be a different case as we have 90d retention on transform, I'm not sure how it will work with 7d on ILM, need to think about it

On the ILM policy - if we ingest the full posture every 24h, then 7d retention for the source index is fine, we don't need old data in the source index.

@maxcold, Thanks for the confirmation. Let me know when the testing is done so we can move forward with merging.

Regarding vulnerability data_stream in https://github.com/elastic/integrations/pull/13595/files#r2079624821, what we have is a full ingestion (not incremental) every interval: 4h. So my understanding is that a similar 7d ILM on source data is going to be fine.
The CDR guides mentions 90d retention only incase the ingestion was incremental.

I think it definitely makes sense to test it before merging

@muskan-agarwal26 @piyush-elastic, our cloud security team would like to test this feature before merging the PR. Once their testing is done, we can merge this.
cc: @nick-alayil

@kcreddy will let you know regarding AWS Config testing

As for the retention - if transform have retention_policy.time.max_age: 24h then having retention period of 7d on source index should be ok. I just don't see transform on m365 PR at all, that's why wasn't sure what's the approach there.

@maxcold,

As for the retention - if transform have retention_policy.time.max_age: 24h then having retention period of 7d on source index should be ok.

Got it, thanks!

I just don't see transform on m365 PR at all, that's why wasn't sure what's the approach there.

Its also a full sync every interval default: 4h of vulnerabilities in case of M365 Defender/Defender for Endpoint PR: #13595. So, I believe a retention of 7d should be okay in that case as well.
The transform isn't added because there are still few missing MUST HAVE fields, but we did the best effort to populate fields based on CDR guide. Once we get to this integration (supporting CNVM), we can look into the missing MUST HAVE fields.

cc: @sharadcrest

packages/aws/data_stream/config/fields/base-fields.yml

packages/aws/data_stream/config/sample_event.json

packages/aws/elasticsearch/transform/latest_config_misconfigurations/transform.yml

packages/aws/data_stream/config/_dev/test/pipeline/test-event.log-expected.json

packages/aws/data_stream/config/sample_event.json

packages/aws/elasticsearch/transform/latest_config_misconfigurations/manifest.yml

packages/aws/elasticsearch/transform/latest_config_misconfigurations/transform.yml

efd6 · 2025-05-15T23:03:29Z

/test

efd6

LGTM but waiting for @kcreddy

…tic#128350) Adding `logs-aws.config-*` data stream indices to the `kibana_system` privileges. This is required for the latest transform to work. Related: - elastic/integrations#13830 (comment) (cherry picked from commit 169527f)

maxcold

I built the integration from the PR, installed it on a cloud env 9.0.1. Here are the issues I found, we need to fix them before merging:

resource.name, resource.type, rule.number are missing in the data, that's only what I can see in the data grid.
the flyout opens but has no data, most likely some of the data missing but I need to dig deeper to understand what exactly

@kcreddy @muskan-agarwal26 before implementing the changes for CDR flows we were doing spreadsheet mapping of the fields required by the guide https://docs.elastic.dev/security-solution/cloud-security/cdr/3p-dev-guide . This allows to understand what crucial fields are missing or the data format is different. Has this exercise been done for AWS Config?

) (#128443) Adding `logs-aws.config-*` data stream indices to the `kibana_system` privileges. This is required for the latest transform to work. Related: - elastic/integrations#13830 (comment) (cherry picked from commit 169527f)

) (#128446) Adding `logs-aws.config-*` data stream indices to the `kibana_system` privileges. This is required for the latest transform to work. Related: - elastic/integrations#13830 (comment) (cherry picked from commit 169527f)

) (#128444) Adding `logs-aws.config-*` data stream indices to the `kibana_system` privileges. This is required for the latest transform to work. Related: - elastic/integrations#13830 (comment) (cherry picked from commit 169527f)

) (#128460) Adding `logs-aws.config-*` data stream indices to the `kibana_system` privileges. This is required for the latest transform to work. Related: - elastic/integrations#13830 (comment) (cherry picked from commit 169527f)

kcreddy · 2025-05-28T06:49:22Z

/test

kcreddy

After an internal discussion, we decided not to create transform for now. This will remove the issue of #13830 (review).
@muskan-agarwal26 please remove the transform (and any related changes) from the PR.

cc: @nick-alayil @maxcold @piyush-elastic

kcreddy · 2025-05-28T09:44:19Z

/test

kcreddy

Nits

packages/aws/manifest.yml

packages/aws/_dev/build/docs/config.md

packages/aws/docs/config.md

1. Add category in manifest. 2. Update readme as suggested.

kcreddy · 2025-05-28T10:47:09Z

/test

kcreddy

LGTM. Thanks!

elasticmachine · 2025-05-28T12:17:35Z

💚 Build Succeeded

Buildkite Build
Commit: 7d89c66

History

💚 Build #26475 succeeded bea3788
💚 Build #26171 succeeded b9b7303
💔 Build #26167 failed 3a96938
💚 Build #26056 succeeded 27e6321
💚 Build #26005 succeeded eab5632
💚 Build #25965 succeeded 0aff076

elastic-sonarqube · 2025-05-28T12:17:35Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
97.9% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

muthu-mps

code owner approval!

elastic-vault-github-plugin-prod · 2025-05-28T15:33:18Z

Package aws - 3.4.0 containing this change is available at https://epr.elastic.co/package/aws/3.4.0/

This release includes the Config data stream for supporting config findings via REST API. AWS Config fields are mapped to their corresponding ECS fields where possible. Also added associated dashboards and visualizations. Test samples were derived from live logs.

Add new AWS Config datastream.

0eeb6ff

muskan-agarwal26 requested review from a team as code owners May 7, 2025 16:27

muskan-agarwal26 marked this pull request as draft May 8, 2025 05:09

muskan-crest added 4 commits May 9, 2025 23:41

Update cel code with adding support of authorization with the sigv4

16dbb1f

Resolved conflicts by taking sync from the main branch

b944596

Merge branch 'main' of github.com:muskan-agarwal26/integrations into …

60cfca3

…aws-3.4.0

Updated system test config assert param and took latest sync

eebfe20

muskan-agarwal26 marked this pull request as ready for review May 9, 2025 18:36

andrewkroh added the Team:obs-ds-hosted-services Observability Hosted Services team [elastic/obs-ds-hosted-services] label May 9, 2025

efd6 reviewed May 13, 2025

View reviewed changes

Resolve comments provided by @efd6.

0aff076

1. Add config data stream entry in CODEOWNERS. 2. Update changelog entry. 3. Format CEL code as suggested. 4. Added fields into redact. 5. Changed transform version v1. 6. Used minify_json in config.yml file.

muskan-agarwal26 requested a review from efd6 May 14, 2025 09:45

kcreddy reviewed May 15, 2025

View reviewed changes

Removed dynamic_field_timestamp from pipeline test config file

eab5632

efd6 reviewed May 15, 2025

View reviewed changes

maxcold requested changes May 26, 2025

View reviewed changes

muskan-agarwal26 requested a review from kcreddy May 28, 2025 05:37

kcreddy reviewed May 28, 2025

View reviewed changes

Remove transform and retention policy, and add full sync note in readme

5e87613

muskan-agarwal26 requested a review from kcreddy May 28, 2025 09:33

kcreddy reviewed May 28, 2025

View reviewed changes

packages/aws/manifest.yml Show resolved Hide resolved

packages/aws/_dev/build/docs/config.md Outdated Show resolved Hide resolved

packages/aws/docs/config.md Outdated Show resolved Hide resolved

Resolve comments provided by @kcreddy.

7d89c66

1. Add category in manifest. 2. Update readme as suggested.

muskan-agarwal26 requested a review from kcreddy May 28, 2025 10:45

kcreddy approved these changes May 28, 2025

View reviewed changes

maxcold approved these changes May 28, 2025

View reviewed changes

kcreddy removed request for a team May 28, 2025 11:25

muthu-mps approved these changes May 28, 2025

View reviewed changes

kcreddy merged commit 45ff60e into elastic:main May 28, 2025
8 checks passed

Add new AWS Config datastream. #13830

Add new AWS Config datastream. #13830

Uh oh!

Conversation

muskan-agarwal26 commented May 7, 2025

Proposed commit message

Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

elasticmachine commented May 7, 2025

Uh oh!

kcreddy commented May 7, 2025

Uh oh!

kcreddy commented May 13, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

efd6 May 13, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

efd6 commented May 13, 2025

Uh oh!

efd6 commented May 14, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented May 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Benchmarks report

Package aws 👍(7) 💚(12) 💔(2)

Uh oh!

efd6 commented May 15, 2025

Uh oh!

Uh oh!

Uh oh!

kcreddy May 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maxcold May 21, 2025

Choose a reason for hiding this comment

Uh oh!

kcreddy May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kcreddy May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maxcold May 26, 2025

Choose a reason for hiding this comment

Uh oh!

kcreddy May 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

efd6 commented May 15, 2025

Uh oh!

efd6 left a comment

Choose a reason for hiding this comment

Uh oh!

maxcold left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

elastic-vault-github-plugin-prod bot commented May 15, 2025 •

edited

Loading

Package `aws` 👍(7) 💚(12) 💔(2)

kcreddy May 15, 2025 •

edited

Loading

kcreddy May 23, 2025 •

edited

Loading

kcreddy May 23, 2025 •

edited

Loading

kcreddy May 26, 2025 •

edited

Loading

maxcold left a comment •

edited

Loading

kcreddy left a comment •

edited

Loading