[O11y][Apache Spark] Remove unnecessary filter from the visualizations

[harnish-crest-data]

• Bug

What does this PR do?

• Remove incorrect filters from the visualizations
• The fields that were used for the maximum aggregation in the visualizations are gauge fields. The maximum aggregation is not appropriate for those fields. For example, there is a field name apache_spark.node.worker.memory.used representing the used memory was used maximum aggregation which is inappropriate because the used memory value will always fluctuate. Hence need to update aggregation from maximum to last_value.

Note

• Since the look of the dashboard has not changed, no need to update the dashboard screenshot

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have added an entry to my package's changelog.yml file.

Related issues

• Closes [O11y][Apache Spark] Remove unnecessary filter from the dashboard and update aggregation for visualizations #7409
• Relates Make Apache Spark GA #7406

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-09-04T08:44:31.591+0000

• Duration: 19 min 50 sec

Test stats 🧪

Test	Results
Failed	0
Passed	15
Skipped	0
Total	15

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (0/0)	💚
Files	100.0% (0/0)	💚
Classes	100.0% (0/0)	💚
Methods	75.0% (12/16)	👎 -12.826
Lines	100.0% (0/0)	💚 4.0
Conditionals	100.0% (0/0)	💚

[rajvi-patel-22]

Looks good to me!

[muthu-mps]

@harnish-elastic - Can you please add the context on why we changed the aggregation from maximum to last value?

[harnish-crest-data]

@harnish-elastic - Can you please add the context on why we changed the aggregation from maximum to last value?

Updated the description here. Thank you!

[muthu-mps]

Looks good!

[elasticmachine]

Package apache_spark - 0.6.1 containing this change is available at https://epr.elastic.co/search?package=apache_spark