Skip to content

Add custom AWS logs integration for both aws-s3 input and aws-cloudwatch input #2353

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 21 commits into from
Feb 9, 2022
Merged

Add custom AWS logs integration for both aws-s3 input and aws-cloudwatch input #2353

merged 21 commits into from
Feb 9, 2022

Conversation

kaiyan-sheng
Copy link
Contributor

@kaiyan-sheng kaiyan-sheng commented Dec 17, 2021

What does this PR do?

In the AWS package, we have data streams to parse and ingest several service logs, such as s3access, cloudtrail, vpcflow and etc. This PR is to add the raw s3 input into the AWS package as an option when the specific type of service log is not supported in its own data stream.

We decide to create a separate package called Custom AWS Logs in integrations and add both aws-s3 input and aws-cloudwatch input options in there. This way, it's aligned with custom logs input, custom TCP/UDP logs input and etc.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

  1. Setup s3access logs to send to an S3 bucket
  2. Create an SQS and enable notification for new objects in S3
  3. On the Integrations page, find AWS S3 and add it
  4. Setup for S3 logs as below:

Screen Shot 2022-02-07 at 3 15 17 PM

Screen Shot 2022-02-07 at 3 15 47 PM

Related issues

#711

@kaiyan-sheng kaiyan-sheng self-assigned this Dec 17, 2021
Copy link
Member

@andrewkroh andrewkroh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea to have a more generic integration for S3.

@kaiyan-sheng
Copy link
Contributor Author

@andrewkroh Thank you for the review! Sorry I just got back to this and addressed your comments.

@elasticmachine
Copy link

elasticmachine commented Jan 20, 2022

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2022-02-08T23:13:17.214+0000

  • Duration: 16 min 23 sec

Test stats 🧪

Test Results
Failed 0
Passed 8
Skipped 0
Total 8

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@kaiyan-sheng kaiyan-sheng merged commit 5008ffa into elastic:main Feb 9, 2022
@kaiyan-sheng kaiyan-sheng deleted the add_aws_s3_input branch February 9, 2022 02:54
@kaiyan-sheng kaiyan-sheng changed the title Add s3 input into aws package Add custom AWS logs integration for both aws-s3 input and aws-cloudwatch input Feb 9, 2022
@andrewkroh andrewkroh added the Integration:aws_logs Custom AWS Logs label Aug 13, 2024
@andrewkroh andrewkroh added the New Integration Issue or pull request for creating a new integration package. label Aug 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Integration:aws_logs Custom AWS Logs New Integration Issue or pull request for creating a new integration package.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants