[Logstash] Add pipeline info to pipeline screen

[robbavey]

Proposed commit message

This commit adds pipeline information to the Pipelines View. This involves retrieving the info object from the pipelines view to retrieve ephemeral_id, workers, batch_size and batch_delay for each pipeline.

Additionally, this commit adds a table to the Pipelines View to show workers, batch_size, batch_delay, the type of queue being used, and the current size of the queue if it is a persistent queue

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

Install the integration, and point to a recent version of Logstash. Verify that the Pipelines View shows the correct data for each pipeline

Screenshots

[elasticmachine]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[jennypavlova]

Great job 💯 Left some questions and I am trying to test the change. I am using elastic-package to build the integration and elastic-package stack up -d but I am not sure that this will work with Logstash - I installed the integration in Kibana (latest version 2.4.10) and enabled self-monitoring. Do you know if I need to do something else to see the changes?

[robbavey]

@jennypavlova Thanks for the review!

This should be able to be tested using elastic-package stack up -d, with stack.logstash_enabled: true setup in the config.yml to bring up a logstash. Then setting up the integration with the agent pointing to localhost:9600, and the "technical preview" metrics radio button checked should install the dashboards, and have new entries showing in "Pipelines Overview"

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 1019a4b

History

• 💚 Build #12567 succeeded 2d5bf53

[jennypavlova]

Thanks for adding the changes!
Probably my local setup is not working correctly because I don't see any values in the dashboards. The code looks good to me. 🎉

This should be able to be tested using elastic-package stack up -d, with stack.logstash_enabled: true setup in the config.yml to bring up a logstash. Then setting up the integration with the agent pointing to localhost:9600, and the "technical preview" metrics radio button checked should install the dashboards, and have new entries showing in "Pipelines Overview"

I built the package and I followed the steps but maybe I missed something:

Screen.Recording.2024-06-18.at.19.34.27.mov

And http://localhost:9600/_node returns:

[robbavey]

Apologies - if you are using the elastic-package agent, the integration should point to http://logstash:9600, rather than http://localhost:9600 - if you change it, the values should appear after a short while

[jennypavlova]

@robbavey Thank you for the instructions! I did that and with http://logstash:9600 I see the data 🎉

[elasticmachine]

Package logstash - 2.4.10 containing this change is available at https://epr.elastic.co/search?package=logstash