[k8s Integratiion] Adding container.id as filter of Cluster Overview and removing median filter occurencies

[gizas]

• Bug

Proposed commit message

• WHAT: Updated the Top Memory and Top CPU visualisations of Cluster Overview Dashboard in order to use the container.id filter
• WHY: The kubernetes.container.name used in previous visualisations was not unique for all kinds of k8s resources (eg. the kubernetes.container.name is the same for all pods of same daemonset and this was creating false calculations)

Additionally we removed all median functions from or visualisations and replaced them either with Average or Last value
The updated dashboards where the median has been removed are:

• Cronjobs
• Deployments
• Volume
• Daemonsets

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

0. Prequisite to have a local k8s cluster. Use kind and run kind create cluster
1. Install kube-state-metrics in your k8s cluster
2. Clone this branch
3. Change to directory integrations/packages/kubernetes
4. Run elastic-package stack up -d -v --version=8.15.0 in order to spin the ES locally
5. Run docker network connect elastic-package-stack_default kind-control-plane to connect the networks of the ES and your k8s node
6. Open the local kibana page and install a k8s integration and create a new Elastic Agent policy
7. Apply the new manfest to your k8s cluster and once the agents are up and running navigate to your dahboards
8. Validate the changes in the visualisations

Related issues

• Closes [Kubernetes]Overview dashboard - total cpu and memory usage are incorrect #10887

Screenshots

From Cluster Overview:

From Cronjobs:

[elasticmachine]

🚀 Benchmarks report

Package kubernetes 👍(0) 💚(0) 💔(1)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
container_logs	142857.14	111111.11	-31746.03 (-22.22%)	💔

To see the full report comment with /test benchmark fullreport

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 47d0e42

History

• 💔 Build #15062 failed 33b209c

[constanca-m]

Additionally we removed all median functions from or visualisations and replaced them either with Average or Last value

Why is this @gizas?

[gizas]

Additionally we removed all median functions from or visualisations and replaced them either with Average or Last value

Why is this @gizas?

In all of the instances where median was used was a mistake, either needed to be last value or average and we aligned it with the other fields of the visualisations. Seems that because the first choice of filter is median (when someone creates a visualisation) this was left by mistake.

Additionally, after a relevant slack thread we were pointed out that average provided better results for a customer of ours so we made the decision to prefer average.

[constanca-m]

Alright. I always thought that median was the best one to use since it does not get affected by outliers as much as average, but I trust your judgement. Could you just link the slack thread? For future reference, it might help.

[elasticmachine]

Package kubernetes - 1.66.4 containing this change is available at https://epr.elastic.co/search?package=kubernetes