Skip to content

[Logstash] Add pipeline level worker utilization #10107

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Jun 13, 2024

Conversation

robbavey
Copy link
Member

@robbavey robbavey commented Jun 7, 2024

This commit adds support for pipeline level worker utilization from the Logstash API

This involves

  • Adding mapping to the fields for the worker_utilization metrics in the pipeline datastream
  • Adding exclusions to the ingest pipeline to tidy up the data coming in
  • Adding visualizations to the pipelines and single pipeline view dashboards

Additionally, this PR removes some incorrect flow metrics from the dashboards - previously, the summary tables in the pipeline details view would show flow metrics summing all data across the picker time period, giving a nonsense value.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

Install the integration, and point to a recent version of Logstash. Verify that the worker_utilization visualizations are present, and reflect the data from the version of Logstash

Related issues

Relates #10028
Closes #10029

Screenshots

Worker Utilization in Pipelines Overview
Screenshot 2024-06-07 at 2 35 59 PM

Worker Utilization in Single Pipeline View
Screenshot 2024-06-07 at 2 36 13 PM

robbavey added 3 commits June 7, 2024 11:46
Flow metric values in table views for plugins were showing incorrect values - showing a
summation of all the values in the time period, rather than the correct value.
Introduce support for pipeline-level `worker_utilization` metrics to dashboards.
@robbavey robbavey requested a review from a team as a code owner June 7, 2024 18:39
@robbavey robbavey force-pushed the add_worker_utilization branch from a383751 to 5efcf51 Compare June 7, 2024 19:10
@elasticmachine
Copy link

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

robbavey added 2 commits June 12, 2024 14:20
Add the observability category, to enable the Logstash integration to show up in "add data"
in serverless observability solution.
Removes pipeline filters from dashboards where they do not apply
Copy link

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

@elasticmachine
Copy link

💚 Build Succeeded

History

"id": "logstash-79270240-48ee-11ee-8cb5-99927777c522",
"managed": false,
"managed": true,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍🏼

@robbavey robbavey merged commit c208f79 into elastic:main Jun 13, 2024
@elasticmachine
Copy link

Package logstash - 2.4.9 containing this change is available at https://epr.elastic.co/search?package=logstash

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Logstash] Add worker_utilization to pipeline view
4 participants