Skip to content

[github] Add Support for AWS S3 and AWS SQS Input in Audit Dataset #14635

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 6 commits into from
Jul 29, 2025
Merged

[github] Add Support for AWS S3 and AWS SQS Input in Audit Dataset #14635

merged 6 commits into from
Jul 29, 2025
+498 −24

Conversation

mohitjha-elastic
Copy link
Collaborator

@mohitjha-elastic mohitjha-elastic commented Jul 22, 2025 •
edited by efd6
Loading

Proposed commit message

github: add support for AWS S3 and SQS input in audit dataset

This adds support for using AWS S3 and AWS SQS as input sources for the
audit dataset in the GitHub integration. It enhances the flexibility of
data ingestion by allowing audit logs to be collected from these
additional AWS services.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

  • Clone integrations repo.
  • Install elastic package locally.
  • Start elastic stack using elastic-package.
  • Move to integrations/packages/github directory.
  • Run the following command to run tests.

elastic-package test -v

Related Issue

@mohitjha-elastic mohitjha-elastic self-assigned this Jul 22, 2025
@mohitjha-elastic mohitjha-elastic requested a review from a team as a code owner July 22, 2025 09:50
@mohitjha-elastic mohitjha-elastic added documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. enhancement New feature or request Integration:github GitHub Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] Team:Sit-Crest Crest developers on the Security Integrations team [elastic/sit-crest-contractors] labels Jul 22, 2025
@elasticmachine
Copy link

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@elastic-vault-github-plugin-prod
Copy link

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

@mohitjha-elastic
Copy link
Collaborator Author

As the elastic-package version 0.113.0 is out today, will add the system test of AWS input type too in this PR.

kcreddy
kcreddy reviewed Jul 23, 2025
View reviewed changes
@mohitjha-elastic
Resolve comments and added aws system test.
b5c2683 
1. Add aws system test.
2. Update manifest parameters as per comments.
@mohitjha-elastic
Merge branch 'main' of github.com:mohitjha-elastic/integrations into …
9303677 
…github-2.11.0

Conflicts:
	packages/github/changelog.yml
	packages/github/manifest.yml
@mohitjha-elastic
Copy link
Collaborator Author

@kcreddy Resolved the comments and added the AWS system test.

@mohitjha-elastic mohitjha-elastic requested a review from kcreddy July 25, 2025 09:36
kcreddy
kcreddy reviewed Jul 25, 2025
View reviewed changes
Copy link
Contributor

@kcreddy kcreddy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mohitjha-elastic , LGTM overall.
Can approve after fixing CI failures.

efd6
efd6 reviewed Jul 27, 2025
View reviewed changes
packages/github/data_stream/audit/sample_event.json Outdated
Comment on lines 73 to 78
"log": {
"file": {
"path": "https://elastic-package-github-audit-bucket-43960.s3.us-east-1.amazonaws.com/test-audit.log"
},
"offset": 0
},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need this given that we have the information in the aws.s3 fields?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@efd6 This appears to be auto-generated during the system test. Would you like me to explicitly remove it?
Same here.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, please. Something like what we are doing in the crowdstrike.fdr data stream.

@mohitjha-elastic mohitjha-elastic requested review from kcreddy and efd6 July 28, 2025 07:18
kcreddy
kcreddy approved these changes Jul 29, 2025
View reviewed changes
Copy link
Contributor

@kcreddy kcreddy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, please wait for @efd6

@elastic-sonarqube Elastic SonarQube
Copy link

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
97.4% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

@elasticmachine
Copy link

💚 Build Succeeded

History

cc @mohitjha-elastic

efd6
efd6 reviewed Jul 29, 2025
View reviewed changes
efd6
efd6 approved these changes Jul 29, 2025
View reviewed changes
Copy link
Contributor

@efd6 efd6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks

@mohitjha-elastic mohitjha-elastic merged commit e163427 into elastic:main Jul 29, 2025
9 checks passed
@mohitjha-elastic mohitjha-elastic deleted the github-2.11.0 branch July 29, 2025 07:31
@elastic-vault-github-plugin-prod
Copy link

Package github - 2.11.0 containing this change is available at https://epr.elastic.co/package/github/2.11.0/

@kcreddy kcreddy mentioned this pull request Aug 1, 2025
Open
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kcreddy kcreddy kcreddy approved these changes

@efd6 efd6 efd6 approved these changes

Assignees

@mohitjha-elastic mohitjha-elastic

Labels
documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. enhancement New feature or request Integration:github GitHub Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] Team:Sit-Crest Crest developers on the Security Integrations team [elastic/sit-crest-contractors]
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Github]: Add S3 + SQS Input for Github Audit Logs
4 participants
@mohitjha-elastic @elasticmachine @kcreddy @efd6