aws: allow user specification of fields to retain in the cloudtrail data stream #14236
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
efd6
added
enhancement
efd6
force-pushed
the
13500-cloudtrail_interim_solution
branch
from
7d17abf to
40b1eeb
Compare
efd6
changed the title
🚀 Benchmarks reportPackage
|
| Data stream | Previous EPS | New EPS | Diff (%) | Result |
|---|---|---|---|---|
waf |
6666.67 | 5649.72 | -1016.95 (-15.25%) | 💔 |
To see the full report comment with /test benchmark fullreport
|
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
andrewkroh
added
the
Team:Obs-InfraObs
efd6
force-pushed
the
13500-cloudtrail_interim_solution
branch
2 times, most recently
from
aebfd82 to
f5e0b91
Compare
kcreddy
reviewed
Jun 26, 2025
Comment on lines
199
to
202
| Cloudtrail `response_elements`, `request_parameters` and `additional_eventdata` data can | ||
| be placed in keyword and text fields as JSON, and in flattened fields. Depending on requirements | ||
| This configuration determines which fields will be retained in the final document. The Minimal | ||
| option retains the minmal set of fields required for the Security Detection Engine rules. |
There was a problem hiding this comment.
Suggested change
| Cloudtrail `response_elements`, `request_parameters` and `additional_eventdata` data can | |
| be placed in keyword and text fields as JSON, and in flattened fields. Depending on requirements | |
| This configuration determines which fields will be retained in the final document. The Minimal | |
| option retains the minmal set of fields required for the Security Detection Engine rules. | |
| Cloudtrail `response_elements`, `request_parameters` and `additional_eventdata` data can | |
| be placed in keyword and text fields as JSON, and in flattened fields. Depending on requirements | |
| this configuration determines which fields will be retained in the final document. The Minimal | |
| option retains the minimal set of fields required for the Security Detection Engine rules. |
| - text: Flattened | ||
| value: flattened | ||
| - text: Neither | ||
| value: none |
There was a problem hiding this comment.
Is minimal not applicable for this input?
There was a problem hiding this comment.
No, I just forgot to add it; it came in later.
kcreddy
approved these changes
Jul 2, 2025
…ata stream Storage of the response_elements, request_parameters and additional_eventdata is a potentially significant cost, but different users have different requirements for their present, so there is no ideal approach. Given that it is likely that this optimisation will be a common desire, provide a UI option to allow users to easily configure this behaviour without the requirement of adding processors to remove the fields in an @Custom pipeline. Note also that there is a TODO in the pipeline addition here to move from a remove after creation model, spending fruitless work, to a non-creation model, which would not be possible to implement in an @Custom pipeline.
The fields were identified by running the following shell script in the
the security_detection_engine/kibana/security_rule directory.
for f in *; do
jq 'select(.attributes.required_fields != null)|.attributes.required_fields|.[]|select(.name != null)|select(.name|contains("cloudtrail.flattened"))|.name'<$f
done|sort|uniq
The test for this is derived from the test-copy-object-json.log test
case which includes one of the required fields and a number of other
fields under cloudtrail.flattened. So comparing the test added here to
that demonstrates whether is works.
efd6
force-pushed
the
13500-cloudtrail_interim_solution
branch
from
c784f79 to
3253fb8
Compare
romulets
reviewed
Jul 2, 2025
Comment on lines
+1730
to
+1739
| required_flattened_fields: | ||
| - aws.cloudtrail.flattened.additional_eventdata.SSEApplied | ||
| - aws.cloudtrail.flattened.request_parameters.cidrIp | ||
| - aws.cloudtrail.flattened.request_parameters.dryRun | ||
| - aws.cloudtrail.flattened.request_parameters.fromPort | ||
| - aws.cloudtrail.flattened.request_parameters.includeDeprecated | ||
| - aws.cloudtrail.flattened.request_parameters.policyArn | ||
| - aws.cloudtrail.flattened.request_parameters.serialNumber | ||
| - aws.cloudtrail.flattened.request_parameters.withDecryption | ||
| - aws.cloudtrail.flattened.request_parameters.x-amz-server-side-encryption-customer-algorithm |
There was a problem hiding this comment.
I personally worry about this duplicated list from the detection engine. This will be easily missed in future iterations. Do you have thoughts on process to avoid it? Or maybe can we automate fetching this list on pre-commit? Or automate tests to verify consistency?
There was a problem hiding this comment.
This is the consequence of the technical debt that has been built up by the unconstrained use of fields in the detection rules. Fixing this would require a significant refactor and I think is outside the scope of this change.
There was a problem hiding this comment.
I agree fixing is outside. But what is the strategy to keep consistency in the list with the rules over time?
packages/aws/data_stream/cloudtrail/elasticsearch/ingest_pipeline/default.yml
Show resolved
Hide resolved
ishleenk17
approved these changes
Jul 3, 2025
zmoog
approved these changes
Jul 3, 2025
|
/test |
💚 Build Succeeded
History
cc @efd6 |
|
romulets
approved these changes
Jul 4, 2025
|
Package aws - 3.10.0 containing this change is available at https://epr.elastic.co/package/aws/3.10.0/ |
|
Follow up issue: #14429 |
robester0403
pushed a commit
to robester0403/integrations
that referenced
this pull request
Jul 8, 2025
…ata stream (elastic#14236) Storage of the response_elements, request_parameters and additional_eventdata is a potentially significant cost, but different users have different requirements for their present, so there is no ideal approach. Given that it is likely that this optimisation will be a common desire, provide a UI option to allow users to easily configure this behaviour without the requirement of adding processors to remove the fields in an @Custom pipeline. Note also that there is a TODO in the pipeline addition here to move from a remove after creation model, spending fruitless work, to a non-creation model, which would not be possible to implement in an @Custom pipeline.
5 tasks
efd6
added a commit
that referenced
this pull request
Jul 16, 2025
In #14236 we allowed users to select which extended fields they wanted to retain in order to reduce storage costs in cases where they did not what the full set of capacities that the data stream can provide. We did not however prevent the work of collecting those unwanted fields. This change does that, avoiding retaining fields that will ultimately not be kept if possible. It is unfortunate that the wide variety of fields is needed at all, but resolving that depends on improving platform support for the diversity of fields that the data source provides and then making more efficient use of those improvements in the detection rules. Until then, this is what we have.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed commit message
Checklist
changelog.ymlfile.Author's Checklist
How to test this PR locally
Related issues
Screenshots