Skip to content

[AWS] Add lambda logs dataset #14070

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 20 commits into from
Jun 7, 2025
Merged

[AWS] Add lambda logs dataset #14070

merged 20 commits into from
Jun 7, 2025

Conversation

agithomas
Copy link
Contributor

  • Enhancement

Proposed commit message

Add a dataset for AWS lambda logs.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

  • Added pipeline test
  • Perform integration testing
  • Add support for the different category of logs including JSON / plain-text across various programming language
  • Add support for fallback

How to test this PR locally

  • elastic-package build && elastic-package stack up -v -d --services package-registry

Screenshots

No screenshots added as part of this PR.

@agithomas agithomas changed the title Issue 262 new lambda dataset [AWS] Add lambda logs dataset May 30, 2025
@elastic-vault-github-plugin-prod
Copy link

elastic-vault-github-plugin-prod bot commented Jun 1, 2025 •
edited
Loading

🚀 Benchmarks report

Package aws 👍(6) 💚(3) 💔(13)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
inspector 2083.33 1760.56 -322.77 (-15.49%) 💔
rds 23255.81 18181.82 -5073.99 (-21.82%) 💔
s3access 4807.69 3745.32 -1062.37 (-22.1%) 💔
securityhub_findings 2217.29 1191.9 -1025.39 (-46.25%) 💔
securityhub_findings_full_posture 2202.64 1474.93 -727.71 (-33.04%) 💔
apigateway_logs 19607.84 6250 -13357.84 (-68.12%) 💔
securityhub_insights 1219.51 895.26 -324.25 (-26.59%) 💔
vpcflow 10869.57 8403.36 -2466.21 (-22.69%) 💔
waf 8547.01 6410.26 -2136.75 (-25%) 💔
cloudwatch_logs 1e+06 500000 -500000 (-50%) 💔
config 3891.05 1926.78 -1964.27 (-50.48%) 💔
ec2_metrics 27777.78 22222.22 -5555.56 (-20%) 💔
emr_logs 27777.78 23255.81 -4521.97 (-16.28%) 💔

To see the full report comment with /test benchmark fullreport

@agithomas agithomas marked this pull request as ready for review June 2, 2025 04:19
@agithomas agithomas requested review from a team as code owners June 2, 2025 04:19
efd6
efd6 reviewed Jun 2, 2025
View reviewed changes
Copy link
Contributor

@efd6 efd6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

review as codeowner

@andrewkroh andrewkroh added the Team:obs-ds-hosted-services Observability Hosted Services team [elastic/obs-ds-hosted-services] label Jun 2, 2025
efd6
efd6 approved these changes Jun 2, 2025
View reviewed changes
Copy link
Contributor

@efd6 efd6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM for SSI-owned files.

ritalwar
ritalwar approved these changes Jun 5, 2025
View reviewed changes
Copy link
Contributor

@ritalwar ritalwar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe the ingest pipeline could be optimised by renaming parsed.record to aws.lambda once, as it's used in most places, instead of renaming it for each field individually. That said, I’ll leave the decision to you — everything else looks good to me.

Kavindu-Dodan
Kavindu-Dodan reviewed Jun 5, 2025
View reviewed changes
Kavindu-Dodan
Kavindu-Dodan approved these changes Jun 5, 2025
View reviewed changes
Copy link
Contributor

@Kavindu-Dodan Kavindu-Dodan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left few nits and a minor follow up. Otherwise nice addition 🙌

@agithomas
Copy link
Contributor Author

/test

1 similar comment
@agithomas
Copy link
Contributor Author

/test

@elasticmachine
Copy link

💚 Build Succeeded

History

@elastic-sonarqube Elastic SonarQube
Copy link

Quality Gate failed Quality Gate failed

Failed conditions
59.2% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

@agithomas agithomas merged commit 31e8b30 into elastic:main Jun 7, 2025
6 of 7 checks passed
@elastic-vault-github-plugin-prod
Copy link

Package aws - 3.6.0 containing this change is available at https://epr.elastic.co/package/aws/3.6.0/

@agithomas agithomas mentioned this pull request Jun 7, 2025
Merged
5 tasks
@andrewkroh andrewkroh added the documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. label Jul 1, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Kavindu-Dodan Kavindu-Dodan Kavindu-Dodan approved these changes

@efd6 efd6 efd6 approved these changes

@ritalwar ritalwar ritalwar approved these changes

Assignees
No one assigned
Labels
documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. Integration:aws AWS Team:obs-ds-hosted-services Observability Hosted Services team [elastic/obs-ds-hosted-services]
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@agithomas @elasticmachine @Kavindu-Dodan @efd6 @ritalwar @andrewkroh