Skip to content

[GitLab] Added application, auth, and audit datastreams #10644

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 19 commits into from
Aug 6, 2024
Merged

[GitLab] Added application, auth, and audit datastreams #10644

merged 19 commits into from
Aug 6, 2024

Conversation

tehbooom
Copy link
Member

Proposed commit message

Added application, auth, and audit datastreams

Added dashboards for application and auth datastreams

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Screenshots

gitlab-application-dashboard
gitlab-auth-dashboard

@tehbooom tehbooom added enhancement New feature or request Integration:gitlab GitLab labels Jul 29, 2024
@tehbooom tehbooom requested a review from a team as a code owner July 29, 2024 16:42
@jamiehynds jamiehynds added the Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations] label Jul 29, 2024
@elasticmachine
Copy link

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

@efd6
Copy link
Contributor

efd6 commented Aug 2, 2024

@tehbooom What version have you tested this on. This is failing with Error: failed to setup system runner: can't install the package: there was an apply error: installation failed: can't install the package: could not zip-install package; API status code = 400; response body = {"statusCode":400,"error":"Bad Request","message":"Encountered 1 errors creating saved objects: [{\"type\":\"dashboard\",\"id\":\"gitlab-6b62c186-77b6-49f2-a820-ca377575bb96\",\"error\":{\"isBoom\":true,\"isServer\":false,\"data\":null,\"output\":{\"statusCode\":400,\"payload\":{\"message\":\"[attributes.controlGroupInput.showApplySelections]: definition for this key is missing: Bad Request\",\"statusCode\":400,\"error\":\"Bad Request\"},\"headers\":{}},\"type\":\"unknown\"}}]"}. I see this locally as well.

@kcreddy
Copy link
Contributor

kcreddy commented Aug 2, 2024 •
edited
Loading

I got a similar error when I used 8.14.3 for creating dashboards, but the CI runs with minimum kibana.version i.e., 8.13.0. I had to remove this key attributes.controlGroupInput.showApplySelections from the dashboards indicated by the error message.

@tehbooom
Copy link
Member Author

tehbooom commented Aug 2, 2024

@efd6 The elasticstack I tested on is 8.14.1. Im trying what @kcreddy suggested to see if it that passes.

@kcreddy
Copy link
Contributor

kcreddy commented Aug 2, 2024

@tehbooom this time the error message is:
Error: can't install the package: could not zip-install package; API status code = 422; response body = {"statusCode":422,"error":"Unprocessable Entity","message":"Document \"gitlab-6b62c186-77b6-49f2-a820-ca377575bb96\" belongs to a more recent version of Kibana [10.2.0] when the last known version is [8.9.0]."}

You will need to change from "typeMigrationVersion": "10.2.0" to "typeMigrationVersion": "8.9.0" inside dashboards. I totally forgot about that one earlier.

@tehbooom
Copy link
Member Author

tehbooom commented Aug 2, 2024

@kcreddy ahh I reverted back to 10.2.0 thinking it wasn't necessary.

@elasticmachine
Copy link

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

kcreddy
kcreddy reviewed Aug 5, 2024
View reviewed changes
kcreddy
kcreddy approved these changes Aug 6, 2024
View reviewed changes
Copy link
Contributor

@kcreddy kcreddy left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍🏼 Minor suggestion

packages/gitlab/data_stream/application/elasticsearch/ingest_pipeline/default.yml Outdated
Comment on lines 234 to 239
- remove:
field: event.original
tag: remove_original_event
if: ctx?.tags == null || !(ctx.tags.contains("preserve_original_event"))
ignore_failure: true
ignore_missing: true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this processor as it is covered in fleet final pipeline.

@elasticmachine
Copy link

💚 Build Succeeded

History

@elastic-sonarqube Elastic SonarQube
Copy link

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
96.5% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

@tehbooom tehbooom merged commit d4411a5 into elastic:main Aug 6, 2024
5 checks passed
@elasticmachine
Copy link

Package gitlab - 0.3.0 containing this change is available at https://epr.elastic.co/search?package=gitlab

harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 4, 2025
@tehbooom
[GitLab] Added application, auth, and audit datastreams (elastic#10644)
858fee9 
* Added application, audit, and auth datastreams

* added auth dataset and more application example logs

* added more example logs for application

* added application and auth dashboards with screenshots

* updated pr number

* bump version

* updated migration version

* revert back to original and removed showApplySelections

* change typeMigrationVersion to 8.9.0

* docs:updated path to URLs and spelling

* chore:removed ecs mappings

* removed quotes from fields

* fix:added client ip with geo

* test:added assert.hit_count

* fix:updated url.path and url.query

* fix:syntax and add geo ip processing

* fix:add geoip processing, related.user, related.ip and fix syntax

* fix:remove processor handled in fleet
harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 5, 2025
@tehbooom
[GitLab] Added application, auth, and audit datastreams (elastic#10644)
1660530 
* Added application, audit, and auth datastreams

* added auth dataset and more application example logs

* added more example logs for application

* added application and auth dashboards with screenshots

* updated pr number

* bump version

* updated migration version

* revert back to original and removed showApplySelections

* change typeMigrationVersion to 8.9.0

* docs:updated path to URLs and spelling

* chore:removed ecs mappings

* removed quotes from fields

* fix:added client ip with geo

* test:added assert.hit_count

* fix:updated url.path and url.query

* fix:syntax and add geo ip processing

* fix:add geoip processing, related.user, related.ip and fix syntax

* fix:remove processor handled in fleet
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kcreddy kcreddy kcreddy approved these changes

Assignees
No one assigned
Labels
enhancement New feature or request Integration:gitlab GitLab Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@tehbooom @elasticmachine @efd6 @kcreddy @jamiehynds