Skip to content

Enhancement: Add visualizations for data tier capacity #10937

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 14 commits into from
Oct 21, 2024
Merged

Enhancement: Add visualizations for data tier capacity #10937

merged 14 commits into from
Oct 21, 2024

Conversation

cp-elastic
Copy link
Contributor

@cp-elastic cp-elastic commented Aug 29, 2024 •
edited
Loading

Proposed commit message

Add visualizations for data tier capacity

Why

Based on feedback from my customers, these visualizations have been incredibly helpful for capacity planning and adjusting ILM policies.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

Screenshots

capacity overview

@cp-elastic cp-elastic changed the title Elasticsearch/add capacity to cluster overview Enhancement: Add visualizations for data tier capacity Aug 29, 2024
@andrewkroh andrewkroh added Integration:elasticsearch Elasticsearch enhancement New feature or request labels Aug 29, 2024
@elasticmachine
Copy link

elasticmachine commented Aug 29, 2024 •
edited
Loading

🚀 Benchmarks report

Package elasticsearch 👍(0) 💚(3) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
deprecation 6802.72 4878.05 -1924.67 (-28.29%) 💔
slowlog 5649.72 3164.56 -2485.16 (-43.99%) 💔

To see the full report comment with /test benchmark fullreport

@cp-elastic cp-elastic marked this pull request as ready for review August 29, 2024 18:38
@cp-elastic cp-elastic requested a review from a team as a code owner August 29, 2024 18:38
@andrewkroh andrewkroh added the Team:Stack Monitoring Stack Monitoring team [elastic/stack-monitoring] label Aug 29, 2024
@consulthys
Copy link
Contributor

@cp-elastic thank you for this contribution!
Can you please enumerate the required steps to test this PR?

@andrewkroh andrewkroh added the dashboard Relates to a Kibana dashboard bug, enhancement, or modification. label Aug 30, 2024
@cp-elastic
Copy link
Contributor Author

cp-elastic commented Sep 3, 2024 •
edited
Loading

@consulthys
I'm not sure what the process is to build this integration locally for development. I exported the dashboard as .ndjson for testing on a ESS deployment.
ETA: Testing the dashboard iteself, gather Stack Monitoring Metrics via Beats or Agent, or on ESS > Ship to Deployment, and these graphs will track used vs free space per tier with an 80% Low Watermark line.

@consulthys
Copy link
Contributor

consulthys commented Sep 3, 2024 •
edited
Loading

Thank you @cp-elastic
I tried to import the dashboard into Kibana but it didn't work out, probably because the format is not the expected one (i.e. the file in this PR is a json one, not ndjson).

@cp-elastic
Copy link
Contributor Author

Updated the dashboards by exporting them properly. Tested against a fresh deployment, and is working as expected out of the box.

@cp-elastic
Copy link
Contributor Author

Ok, changed TSVB to Lens to fall in line with best practices guidance.
Updated the screenshot to reflect the new Lens visualizations.

@elastic-sonarqube Elastic SonarQube
Copy link

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

@elasticmachine
Copy link

💚 Build Succeeded

History

@botelastic
Copy link

botelastic bot commented Oct 3, 2024

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

@botelastic botelastic bot added the Stalled label Oct 3, 2024
consulthys
consulthys approved these changes Oct 21, 2024
View reviewed changes
Copy link
Contributor

@consulthys consulthys left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGT Stack Monitoring
PS: sorry for the delay

@botelastic botelastic bot removed the Stalled label Oct 21, 2024
@cp-elastic cp-elastic merged commit da39aaf into main Oct 21, 2024
5 checks passed
@cp-elastic cp-elastic deleted the elasticsearch/add-capacity-to-cluster-overview branch October 21, 2024 11:57
@elastic-vault-github-plugin-prod
Copy link

Package elasticsearch - 1.15.1 containing this change is available at https://epr.elastic.co/search?package=elasticsearch

harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 4, 2025
@cp-elastic
Enhancement: Add visualizations for data tier capacity (elastic#10937)
fca0e74 
* Rename Desconnections to Disconnections

* Adds 4 visualtions to show capacity per data tier

* Undo change from other branch

* Really undo changes from other branch

* Update changelog, add screenshot of capacity overview

* Increment version in manifest.yml

* Update capcity overview screenshot

* Update manifest.yml with screenshot

* Fix dashboard export

* Revert "typeMigrationVersion", remove "updated_by"

* Change TSVB to Lens

* remove showApplySelections

* Update screenshot showing Lens and manifest.yml

* Fix Data Warm filter from Hot
harnish-crest-data pushed a commit to chavdaharnish/integrations that referenced this pull request Feb 5, 2025
@cp-elastic
Enhancement: Add visualizations for data tier capacity (elastic#10937)
ede1427 
* Rename Desconnections to Disconnections

* Adds 4 visualtions to show capacity per data tier

* Undo change from other branch

* Really undo changes from other branch

* Update changelog, add screenshot of capacity overview

* Increment version in manifest.yml

* Update capcity overview screenshot

* Update manifest.yml with screenshot

* Fix dashboard export

* Revert "typeMigrationVersion", remove "updated_by"

* Change TSVB to Lens

* remove showApplySelections

* Update screenshot showing Lens and manifest.yml

* Fix Data Warm filter from Hot
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@consulthys consulthys consulthys approved these changes

Assignees
No one assigned
Labels
dashboard Relates to a Kibana dashboard bug, enhancement, or modification. enhancement New feature or request Integration:elasticsearch Elasticsearch Team:Stack Monitoring Stack Monitoring team [elastic/stack-monitoring]
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@cp-elastic @elasticmachine @consulthys @andrewkroh