Skip to content

Add support of Vertex AI Audit Logs #13799

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 39 commits into from
May 29, 2025
Merged

Add support of Vertex AI Audit Logs #13799

merged 39 commits into from
May 29, 2025

Conversation

ishleenk17
Copy link
Member

Proposed commit message

See title

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Related issues

Screenshots

@elastic-vault-github-plugin-prod
Copy link

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

@ishleenk17 ishleenk17 self-assigned this May 6, 2025
@andrewkroh andrewkroh added the Integration:gcp_vertexai GCP Vertex AI label May 6, 2025
muthu-mps
muthu-mps reviewed May 15, 2025
View reviewed changes
@ishleenk17 ishleenk17 marked this pull request as ready for review May 23, 2025 06:43
@ishleenk17 ishleenk17 requested a review from a team as a code owner May 23, 2025 06:43
@muthu-mps
Copy link
Contributor

@ishleenk17 - Can we add a section in the documentation for the logs data stream?

@andrewkroh andrewkroh added the Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] label May 23, 2025
muthu-mps
muthu-mps reviewed May 27, 2025
View reviewed changes
@muthu-mps muthu-mps requested a review from Copilot May 27, 2025 06:49
Copilot
Copilot AI reviewed May 27, 2025
View reviewed changes
Copy link

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds support for Vertex AI Audit Logs to the GCP Vertex AI integration. Key changes include:

  • Updating the integration manifest and version to reflect the logs support.
  • Adding a new auditlogs data stream with its own manifest, field definitions, and Elasticsearch ingest pipeline.
  • Enhancing documentation and changelog to guide users on setting up and configuring Vertex AI Audit Logs.

Reviewed Changes

Copilot reviewed 10 out of 13 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
packages/gcp_vertexai/manifest.yml Updated version and description to include logs
packages/gcp_vertexai/docs/README.md Expanded documentation to cover logs collection and setup instructions
packages/gcp_vertexai/data_stream/auditlogs/manifest.yml Added new data stream manifest for Vertex AI audit logs
packages/gcp_vertexai/data_stream/auditlogs/fields/*.yml Defined fields for the auditlogs data stream
packages/gcp_vertexai/data_stream/auditlogs/elasticsearch/ingest_pipeline/default.yml Created ingest pipeline for processing audit logs
packages/gcp_vertexai/changelog.yml Updated changelog with details on the audit logs feature
packages/gcp_vertexai/_dev/build/docs/README.md Updated build docs to include audit logs integration
Files not reviewed (3)
  • packages/gcp_vertexai/data_stream/auditlogs/_dev/test/pipeline/test-vertexai.log-expected.json: Language not supported
  • packages/gcp_vertexai/data_stream/auditlogs/agent/stream/stream.yml.hbs: Language not supported
  • packages/gcp_vertexai/data_stream/auditlogs/sample_event.json: Language not supported

ishleenk17 and others added 6 commits May 27, 2025 21:10
@ishleenk17 @muthu-mps
Update packages/gcp_vertexai/_dev/build/docs/README.md
e6f8bec 
Co-authored-by: muthu-mps <101238137+muthu-mps@users.noreply.github.com>
@ishleenk17 @muthu-mps
Update packages/gcp_vertexai/_dev/build/docs/README.md
9b48171 
Co-authored-by: muthu-mps <101238137+muthu-mps@users.noreply.github.com>
@ishleenk17 @muthu-mps
Update packages/gcp_vertexai/_dev/build/docs/README.md
6692ecb 
Co-authored-by: muthu-mps <101238137+muthu-mps@users.noreply.github.com>
@ishleenk17 @muthu-mps
Update packages/gcp_vertexai/_dev/build/docs/README.md
6a57b03 
Co-authored-by: muthu-mps <101238137+muthu-mps@users.noreply.github.com>
@ishleenk17 @muthu-mps
Update packages/gcp_vertexai/_dev/build/docs/README.md
461d30d 
Co-authored-by: muthu-mps <101238137+muthu-mps@users.noreply.github.com>
@ishleenk17 @muthu-mps
Update packages/gcp_vertexai/_dev/build/docs/README.md
35c72ed 
Co-authored-by: muthu-mps <101238137+muthu-mps@users.noreply.github.com>
ishleenk17 and others added 14 commits May 28, 2025 11:09
@ishleenk17 @alaudazzi
Update packages/gcp_vertexai/_dev/build/docs/README.md
8726112 
Co-authored-by: Arianna Laudazzi <46651782+alaudazzi@users.noreply.github.com>
@ishleenk17 @alaudazzi
Update packages/gcp_vertexai/_dev/build/docs/README.md
5828ae3 
Co-authored-by: Arianna Laudazzi <46651782+alaudazzi@users.noreply.github.com>
@ishleenk17 @muthu-mps
Update packages/gcp_vertexai/_dev/build/docs/README.md
fa0c218 
Co-authored-by: muthu-mps <101238137+muthu-mps@users.noreply.github.com>
@ishleenk17 @alaudazzi
Update packages/gcp_vertexai/_dev/build/docs/README.md
d73d0b8 
Co-authored-by: Arianna Laudazzi <46651782+alaudazzi@users.noreply.github.com>
@ishleenk17 @alaudazzi
Update packages/gcp_vertexai/_dev/build/docs/README.md
5abf8b8 
Co-authored-by: Arianna Laudazzi <46651782+alaudazzi@users.noreply.github.com>
@ishleenk17 @alaudazzi
Update packages/gcp_vertexai/_dev/build/docs/README.md
0bcb138 
Co-authored-by: Arianna Laudazzi <46651782+alaudazzi@users.noreply.github.com>
@ishleenk17 @alaudazzi
Update packages/gcp_vertexai/_dev/build/docs/README.md
70c3b3c 
Co-authored-by: Arianna Laudazzi <46651782+alaudazzi@users.noreply.github.com>
@ishleenk17 @alaudazzi
Update packages/gcp_vertexai/_dev/build/docs/README.md
c92ea12 
Co-authored-by: Arianna Laudazzi <46651782+alaudazzi@users.noreply.github.com>
@ishleenk17
Update README
f386ed0
@ishleenk17 @alaudazzi
Update packages/gcp_vertexai/_dev/build/docs/README.md
5a04022 
Co-authored-by: Arianna Laudazzi <46651782+alaudazzi@users.noreply.github.com>
@ishleenk17
Merge branch 'main' into vertexai_auditlogs
3299470
@ishleenk17 @alaudazzi
Update packages/gcp_vertexai/_dev/build/docs/README.md
b8bfbf0 
Co-authored-by: Arianna Laudazzi <46651782+alaudazzi@users.noreply.github.com>
@ishleenk17
Generated readme
2158c7c
@ishleenk17
Merge branch 'main' into vertexai_auditlogs
d5b1c4a
muthu-mps
muthu-mps approved these changes May 28, 2025
View reviewed changes
Copy link
Contributor

@muthu-mps muthu-mps left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@ishleenk17 ishleenk17 enabled auto-merge (squash) May 28, 2025 10:35
@ishleenk17 ishleenk17 disabled auto-merge May 28, 2025 10:35
daniela-elastic
daniela-elastic reviewed May 28, 2025
View reviewed changes
daniela-elastic
daniela-elastic reviewed May 28, 2025
View reviewed changes
daniela-elastic
daniela-elastic reviewed May 28, 2025
View reviewed changes
daniela-elastic
daniela-elastic reviewed May 28, 2025
View reviewed changes
daniela-elastic
daniela-elastic reviewed May 28, 2025
View reviewed changes
daniela-elastic
daniela-elastic approved these changes May 28, 2025
View reviewed changes
Copy link

@daniela-elastic daniela-elastic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved conditionally, please address my comments

@elasticmachine
Copy link

💚 Build Succeeded

History

cc @ishleenk17

@elastic-sonarqube Elastic SonarQube
Copy link

Quality Gate failed Quality Gate failed

Failed conditions
50.7% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

@ishleenk17 ishleenk17 merged commit 2c6b396 into elastic:main May 29, 2025
7 of 8 checks passed
@elastic-vault-github-plugin-prod
Copy link

Package gcp_vertexai - 1.2.0 containing this change is available at https://epr.elastic.co/package/gcp_vertexai/1.2.0/

anupratharamachandran pushed a commit to anupratharamachandran/integrations that referenced this pull request Jun 2, 2025
@ishleenk17 @muthu-mps
@alaudazzi @anupratharamachandran
Add support of Vertex AI Audit Logs (elastic#13799)
89ff377 
* Add support of Vertex AI Audit Logs

* update changelog

* Update packages/gcp_vertexai/data_stream/logs/manifest.yml

Co-authored-by: muthu-mps <101238137+muthu-mps@users.noreply.github.com>

* Update fields.yml

* Address review comments

* Update tests and review comments addressed

* Update the datastream name

* README update

* Update packages/gcp_vertexai/_dev/build/docs/README.md

Co-authored-by: muthu-mps <101238137+muthu-mps@users.noreply.github.com>

* Update packages/gcp_vertexai/_dev/build/docs/README.md

Co-authored-by: muthu-mps <101238137+muthu-mps@users.noreply.github.com>

* Update packages/gcp_vertexai/_dev/build/docs/README.md

Co-authored-by: muthu-mps <101238137+muthu-mps@users.noreply.github.com>

* Update packages/gcp_vertexai/_dev/build/docs/README.md

Co-authored-by: muthu-mps <101238137+muthu-mps@users.noreply.github.com>

* Update packages/gcp_vertexai/_dev/build/docs/README.md

Co-authored-by: muthu-mps <101238137+muthu-mps@users.noreply.github.com>

* Update packages/gcp_vertexai/_dev/build/docs/README.md

Co-authored-by: muthu-mps <101238137+muthu-mps@users.noreply.github.com>

* Update packages/gcp_vertexai/_dev/build/docs/README.md

Co-authored-by: muthu-mps <101238137+muthu-mps@users.noreply.github.com>

* Update README

* Update packages/gcp_vertexai/_dev/build/docs/README.md

Co-authored-by: Arianna Laudazzi <46651782+alaudazzi@users.noreply.github.com>

* Update packages/gcp_vertexai/_dev/build/docs/README.md

Co-authored-by: Arianna Laudazzi <46651782+alaudazzi@users.noreply.github.com>

* Update packages/gcp_vertexai/_dev/build/docs/README.md

Co-authored-by: Arianna Laudazzi <46651782+alaudazzi@users.noreply.github.com>

* Update packages/gcp_vertexai/_dev/build/docs/README.md

Co-authored-by: Arianna Laudazzi <46651782+alaudazzi@users.noreply.github.com>

* Update packages/gcp_vertexai/_dev/build/docs/README.md

Co-authored-by: muthu-mps <101238137+muthu-mps@users.noreply.github.com>

* Update packages/gcp_vertexai/_dev/build/docs/README.md

Co-authored-by: Arianna Laudazzi <46651782+alaudazzi@users.noreply.github.com>

* Update packages/gcp_vertexai/_dev/build/docs/README.md

Co-authored-by: Arianna Laudazzi <46651782+alaudazzi@users.noreply.github.com>

* Update packages/gcp_vertexai/_dev/build/docs/README.md

Co-authored-by: Arianna Laudazzi <46651782+alaudazzi@users.noreply.github.com>

* Update packages/gcp_vertexai/_dev/build/docs/README.md

Co-authored-by: Arianna Laudazzi <46651782+alaudazzi@users.noreply.github.com>

* Update packages/gcp_vertexai/_dev/build/docs/README.md

Co-authored-by: Arianna Laudazzi <46651782+alaudazzi@users.noreply.github.com>

* Update README

* Update packages/gcp_vertexai/_dev/build/docs/README.md

Co-authored-by: Arianna Laudazzi <46651782+alaudazzi@users.noreply.github.com>

* Update packages/gcp_vertexai/_dev/build/docs/README.md

Co-authored-by: Arianna Laudazzi <46651782+alaudazzi@users.noreply.github.com>

* Generated readme

* Update README

---------

Co-authored-by: muthu-mps <101238137+muthu-mps@users.noreply.github.com>
Co-authored-by: Arianna Laudazzi <46651782+alaudazzi@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

Copilot code review Copilot Copilot left review comments

@alaudazzi alaudazzi alaudazzi approved these changes

@muthu-mps muthu-mps muthu-mps approved these changes

@daniela-elastic daniela-elastic daniela-elastic approved these changes

Assignees

@ishleenk17 ishleenk17

Labels
Integration:gcp_vertexai GCP Vertex AI Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations]
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@ishleenk17 @muthu-mps @elasticmachine @alaudazzi @daniela-elastic @andrewkroh