Skip to content

Enablement & Sanity test run for OBS Non-Cloud integration packages for 9.0.0 #12468

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 8 commits into from
Feb 12, 2025
Merged

Enablement & Sanity test run for OBS Non-Cloud integration packages for 9.0.0 #12468

merged 8 commits into from
Feb 12, 2025

Conversation

kush-elastic
Copy link
Contributor

@kush-elastic kush-elastic commented Jan 25, 2025 •
edited
Loading

  • Enhancement

Description

OBS non-cloud packages for 9.0.0 enablement activities of O11y integrations.

Steps performed for the testing

  1. update manifest.yml, package minor version upgrade and kibana dependency addition || ^9.0.0.
  2. build package
  3. run package tests
  4. install integration from kibana
  5. check for any issues

List of non-cloud packages and results of sanity

Below mentioned results are from running tech stack 9.0.0.
Pass: Successfully Passed
Pending: Either permission issue or agent will stuck on 0 hits
Failed: Failed because of beats and integration changes are not in sync
Not Available: Tests are not available

No. Packages build & check pipeline system asset static
1 activemq Pass Pass Pass Pass Pass
2 apache_spark Pass Not Available Pass Pass Pass
3 apache_tomcat Pass Pass Pending Pass Pass
4 cassandra Pass Pass Pass Pass Pass
5 ceph Pass Pass Pass Pass Pass
6 citrix_adc Pass Pass Pass Pass Pass
7 cockroachdb Pass Not Available Pass Pass Pass
8 coredns Pass Pass Pending Pass Pass
9 couchbase Pass Pass Pass Pass Pass
10 couchdb Pass Pass Pass Pass Pass
11 etcd Pass Not Available Pass Pass Pass
12 golang Pass Pass Pass Pass Pass
13 hadoop Pass Pass Pass Pass Pass
14 haproxy Pass Pass Pass Pass Pass
15 ibmmq Pass Pass Pass Pass Pass
16 influxdb Pass Not Available Pass Pass Pass
17 jolokia_input Pass Not Available Pass Not Available Pass
18 kafka Pass Pass Not Available Pass Pass
19 kafka_log Pass Not Available Pass Pass Pass
20 memcached Pass Not Available Pass Pass Pass
21 microsoft_sqlserver Pass Pass Pending Pass Pass
22 mongodb Pass Pass Fail Pass Pass
23 nagios_xi Pass Pass Pass Pass Pass
24 oracle Pass Pass Pass Pass Pass
25 oracle_weblogic Pass Pass Pass Pass Pass
26 php_fpm Pass Pass Pending Pass Pass
27 postgresql Pass Pass Pass Pass Pass
28 rabbitmq Pass Pass Not Available Pass Pass
29 redis Pass Pass Not Available Pass Pass
30 redisenterprise Pass Not Available Not Available Pass Pass
31 salesforce Pass Pass Pass Pass Pass
32 spring_boot Pass Pass Pass Pass Pass
33 sql_input Pass Not Available Pass Not Available Pass
34 statsd_input Pass Not Available Pass Not Available Pass
35 tomcat Pass Pass Pass Pass Pass
36 traefik Pass Pass Pass Pass Pass
37 vsphere Pass Pass Pass Pass Pass
38 websphere_application_server Pass Pass Pass Pass Pass
39 zookeeper Pass Not Available Not Available Pass Pass

NOTE: CI by default runs on minimum kibana version metioned in menifest.yml file.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Related issues

@kush-elastic kush-elastic requested review from a team January 25, 2025 07:12
@kush-elastic kush-elastic self-assigned this Jan 25, 2025
@kush-elastic kush-elastic added enhancement New feature or request Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] labels Jan 25, 2025
@elastic-vault-github-plugin-prod
Copy link

elastic-vault-github-plugin-prod bot commented Jan 25, 2025 •
edited
Loading

🚀 Benchmarks report

Package activemq 👍(2) 💚(0) 💔(3)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
log 8771.93 4901.96 -3869.97 (-44.12%) 💔
queue 100000 66666.67 -33333.33 (-33.33%) 💔
topic 100000 83333.33 -16666.67 (-16.67%) 💔

Package apache_tomcat 👍(2) 💚(3) 💔(4)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
catalina 10638.3 8928.57 -1709.73 (-16.07%) 💔
localhost 25641.03 17857.14 -7783.89 (-30.36%) 💔
request 37037.04 27777.78 -9259.26 (-25%) 💔
session 19230.77 11494.25 -7736.52 (-40.23%) 💔

Package cassandra 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
log 5780.35 4149.38 -1630.97 (-28.22%) 💔

Package ceph 👍(1) 💚(5) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
osd_performance 41666.67 31250 -10416.67 (-25%) 💔

Package couchbase 👍(6) 💚(1) 💔(3)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
bucket 15151.52 10526.32 -4625.2 (-30.53%) 💔
cache 19230.77 7092.2 -12138.57 (-63.12%) 💔
miscellaneous 17857.14 12048.19 -5808.95 (-32.53%) 💔

Package hadoop 👍(2) 💚(2) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
cluster 4310.34 3300.33 -1010.01 (-23.43%) 💔

Package nagios_xi 👍(0) 💚(1) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
host 2840.91 2364.07 -476.84 (-16.78%) 💔
service 2849 2336.45 -512.55 (-17.99%) 💔

Package oracle_weblogic 👍(2) 💚(0) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
access 21739.13 9900.99 -11838.14 (-54.46%) 💔
admin_server 16949.15 10869.57 -6079.58 (-35.87%) 💔

Package rabbitmq 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
log 7462.69 5714.29 -1748.4 (-23.43%) 💔

Package redis 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
log 10000 7692.31 -2307.69 (-23.08%) 💔

Package salesforce 👍(2) 💚(0) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
logout 33333.33 20833.33 -12500 (-37.5%) 💔
setupaudittrail 8928.57 6211.18 -2717.39 (-30.43%) 💔

Package spring_boot 👍(1) 💚(0) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
audit_events 10638.3 6250 -4388.3 (-41.25%) 💔
gc 13698.63 7407.41 -6291.22 (-45.93%) 💔

Package vsphere 👍(3) 💚(4) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
datastore 37037.04 25000 -12037.04 (-32.5%) 💔

Package websphere_application_server 👍(0) 💚(3) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
jdbc 7194.24 4608.29 -2585.95 (-35.94%) 💔

To see the full report comment with /test benchmark fullreport

ishleenk17
ishleenk17 reviewed Jan 26, 2025
View reviewed changes
Copy link
Member

@ishleenk17 ishleenk17 left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets fix the PR number in changelog. Otherwise changes look good!

@ishleenk17
Copy link
Member

@kush-elastic : There are some Integrtaions, for which the syste, test has either failed or is pending.
Eg: envoy, Mongodb. whats the reason for that ?

@ishleenk17
Copy link
Member

@kush-elastic : Looks like there is a package owned by Security team. Please remove that from this change.

@kush-elastic
Copy link
Contributor Author

kush-elastic commented Jan 28, 2025 •
edited
Loading

@kush-elastic : Looks like there is a package owned by Security team. Please remove that from this change.

Hey @ishleenk17,
There are packages, owned by multiple teams.
For Ex: citrix_adc is owned by both team obs-infraobs-integrations and sec-deployment-and-devices.

@kush-elastic : There are some Integrtaions, for which the syste, test has either failed or is pending. Eg: envoy, Mongodb. whats the reason for that ?

I have updated the status for all the integrations.

ishleenk17
ishleenk17 reviewed Jan 28, 2025
View reviewed changes
@lalit-satapathy lalit-satapathy changed the title Sanity test run for OBS Non-Cloud integration packages for 9.0.0 Enablement & Sanity test run for OBS Non-Cloud integration packages for 9.0.0 Jan 28, 2025
ishleenk17
ishleenk17 reviewed Jan 28, 2025
View reviewed changes
@kush-elastic
Copy link
Contributor Author

/test

@ishleenk17
Copy link
Member

@mrodm @jsoriano : We had updated the versions by adding the 9.0.0 to the stack version, and the CI passed.
Assumptions was that the CI would have run for both the 8.x and 9.0 versions. But when we checked the CI we see it ran only for 8.x versions. Is this expected ?

How can we have the CI automatically run on 9.0.0 for all the changes ?
We are testing it locally, but how can CI help here?

cc: @lalit-satapathy @tommyers-elastic

@mrodm
Copy link
Collaborator

mrodm commented Jan 29, 2025

@mrodm @jsoriano : We had updated the versions by adding the 9.0.0 to the stack version, and the CI passed. Assumptions was that the CI would have run for both the 8.x and 9.0 versions. But when we checked the CI we see it ran only for 8.x versions. Is this expected ?

This is expected. CI runs with the minimum Kibana version supported by the package.
More info here

How can we have the CI automatically run on 9.0.0 for all the changes ? We are testing it locally, but how can CI help here?

Currently, there is a daily CI job that runs all the test packages starting an Elastic stack 9.0.0-SNAPSHOT:
https://buildkite.com/elastic/integrations-schedule-daily/

Example of the build triggered by that CI daily job:
https://buildkite.com/elastic/integrations/builds/21159

However, currently there is no way to trigger the tests in CI using a specific Elastic stack version (e.g. 9.0.0-SNAPSHOT) in the context of a Pull Request.

The only way I could think of now is to set temporarily the Kibana constraint ^9.0.0 for the packages and then revert it before merging.

To be checked/tested, but maybe it could be added a new regex here (e.g. test stack 9.0.0-SNAPSHOT): https://github.com/elastic/integrations/blob/3ad7d5e85b7fe78fe468b2e3967856796c5036d2/.buildkite/pull-requests.json#L12C101-L12C121

and update the corresponding Buildkite scripts to add the corresponding STACK_VERSION automatically here:

https://github.com/elastic/integrations/blob/3ad7d5e85b7fe78fe468b2e3967856796c5036d2/.buildkite/scripts/test_one_package.sh#L38

If that works, a new Buildkite build could be triggered by posting a comment into the PR and that build would set the STACK_VERSION to test from the contents of the GitHub comment

WDYT ? @jsoriano @kpollich

Something similar was added for creating the benchmark full report
https://github.com/elastic/integrations/blob/3ad7d5e85b7fe78fe468b2e3967856796c5036d2/.buildkite/scripts/process_benchmarks.sh#L33

@jsoriano
Copy link
Member

@mrodm your proposal sounds good to me, maybe we could even generalize it so test stack <version> runs CI with the <version> stack version?

@mrodm
Copy link
Collaborator

mrodm commented Jan 29, 2025

@mrodm your proposal sounds good to me, maybe we could even generalize it so test stack <version> runs CI with the <version> stack version?

Doing some tests here #12516
The main issue is that it could not be tested E2E (writing a GH comment) until the changes in .buildkite/pull-requests.json are merged into main.

@mrodm mrodm mentioned this pull request Jan 29, 2025
Merged
1 task
@harnish-crest-data harnish-crest-data self-requested a review January 30, 2025 05:28
@gizas gizas mentioned this pull request Jan 30, 2025
Closed
14 tasks
@elastic-vault-github-plugin-prod
Copy link

Package ibmmq - 1.6.0 containing this change is available at https://epr.elastic.co/package/ibmmq/1.6.0/

@elastic-vault-github-plugin-prod
Copy link

Package influxdb - 0.11.0 containing this change is available at https://epr.elastic.co/package/influxdb/0.11.0/

@elastic-vault-github-plugin-prod
Copy link

Package jolokia - 0.7.0 containing this change is available at https://epr.elastic.co/package/jolokia/0.7.0/

@elastic-vault-github-plugin-prod
Copy link

Package kafka - 1.18.0 containing this change is available at https://epr.elastic.co/package/kafka/1.18.0/

@elastic-vault-github-plugin-prod
Copy link

Package kafka_log - 1.8.0 containing this change is available at https://epr.elastic.co/package/kafka_log/1.8.0/

@elastic-vault-github-plugin-prod
Copy link

Package memcached - 1.7.0 containing this change is available at https://epr.elastic.co/package/memcached/1.7.0/

@elastic-vault-github-plugin-prod
Copy link

Package microsoft_sqlserver - 2.12.0 containing this change is available at https://epr.elastic.co/package/microsoft_sqlserver/2.12.0/

@elastic-vault-github-plugin-prod
Copy link

Package mongodb - 1.19.0 containing this change is available at https://epr.elastic.co/package/mongodb/1.19.0/

@elastic-vault-github-plugin-prod
Copy link

Package nagios_xi - 1.5.0 containing this change is available at https://epr.elastic.co/package/nagios_xi/1.5.0/

@elastic-vault-github-plugin-prod
Copy link

Package oracle - 1.29.0 containing this change is available at https://epr.elastic.co/package/oracle/1.29.0/

@elastic-vault-github-plugin-prod
Copy link

Package oracle_weblogic - 1.9.0 containing this change is available at https://epr.elastic.co/package/oracle_weblogic/1.9.0/

@elastic-vault-github-plugin-prod
Copy link

Package php_fpm - 1.5.0 containing this change is available at https://epr.elastic.co/package/php_fpm/1.5.0/

@elastic-vault-github-plugin-prod
Copy link

Package postgresql - 1.27.0 containing this change is available at https://epr.elastic.co/package/postgresql/1.27.0/

@elastic-vault-github-plugin-prod
Copy link

Package rabbitmq - 1.18.0 containing this change is available at https://epr.elastic.co/package/rabbitmq/1.18.0/

@elastic-vault-github-plugin-prod
Copy link

Package redis - 1.19.0 containing this change is available at https://epr.elastic.co/package/redis/1.19.0/

@elastic-vault-github-plugin-prod
Copy link

Package redisenterprise - 0.12.0 containing this change is available at https://epr.elastic.co/package/redisenterprise/0.12.0/

@elastic-vault-github-plugin-prod
Copy link

Package salesforce - 1.1.0 containing this change is available at https://epr.elastic.co/package/salesforce/1.1.0/

@elastic-vault-github-plugin-prod
Copy link

Package spring_boot - 1.8.0 containing this change is available at https://epr.elastic.co/package/spring_boot/1.8.0/

@elastic-vault-github-plugin-prod
Copy link

Package sql - 0.6.0 containing this change is available at https://epr.elastic.co/package/sql/0.6.0/

@elastic-vault-github-plugin-prod
Copy link

Package statsd_input - 0.5.0 containing this change is available at https://epr.elastic.co/package/statsd_input/0.5.0/

@elastic-vault-github-plugin-prod
Copy link

Package tomcat - 1.12.0 containing this change is available at https://epr.elastic.co/package/tomcat/1.12.0/

@elastic-vault-github-plugin-prod
Copy link

Package traefik - 2.5.0 containing this change is available at https://epr.elastic.co/package/traefik/2.5.0/

@elastic-vault-github-plugin-prod
Copy link

Package vsphere - 1.18.0 containing this change is available at https://epr.elastic.co/package/vsphere/1.18.0/

@elastic-vault-github-plugin-prod
Copy link

Package websphere_application_server - 1.6.0 containing this change is available at https://epr.elastic.co/package/websphere_application_server/1.6.0/

@elastic-vault-github-plugin-prod
Copy link

Package zookeeper - 1.14.0 containing this change is available at https://epr.elastic.co/package/zookeeper/1.14.0/

@andrewkroh andrewkroh removed New Integration Issue or pull request for creating a new integration package. Integration:1password 1Password (Partner supported) Integration:abnormal_security Abnormal AI Integration:envoyproxy Envoyproxy labels Mar 13, 2025
flexitrev pushed a commit that referenced this pull request Mar 20, 2025
@kush-elastic @ishleenk17 @flexitrev
Enablement & Sanity test run for OBS Non-Cloud integration packages f…
e0faf5c 
…or 9.0.0 (#12468)

* non-cloud package sanity changes

* update changelog entry

* fix salesforce and websphere_application_server changes

* fix links

* Update packages/salesforce/manifest.yml

Co-authored-by: Ishleen Kaur <102962586+ishleenk17@users.noreply.github.com>

* removing envoyproxy changes from current PR to sync with failures

---------

Co-authored-by: Ishleen Kaur <102962586+ishleenk17@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@taylor-swanson taylor-swanson taylor-swanson approved these changes

@ishleenk17 ishleenk17 ishleenk17 approved these changes

@harnish-crest-data harnish-crest-data harnish-crest-data approved these changes

@lalit-satapathy lalit-satapathy Awaiting requested review from lalit-satapathy

@tommyers-elastic tommyers-elastic Awaiting requested review from tommyers-elastic

Assignees

@kush-elastic kush-elastic

Labels
enhancement New feature or request Integration:activemq ActiveMQ Integration:apache_spark Apache Spark Integration:apache_tomcat Apache Tomcat Integration:cassandra Cassandra Integration:ceph Ceph Integration:citrix_adc Citrix ADC Integration:cockroachdb CockroachDB Metrics Integration:coredns CoreDNS Integration:couchbase Couchbase Integration:couchdb CouchDB Integration:etcd etcd Integration:golang Golang Integration:hadoop Hadoop Integration:haproxy HAProxy Integration:ibmmq IBM MQ Integration:influxdb InfluxDb Integration:jolokia_input Jolokia Input Integration:kafka_log Custom Kafka Logs Integration:kafka Kafka Integration:memcached Memcached Integration:microsoft_sqlserver Microsoft SQL Server Integration:mongodb MongoDB Integration:nagios_xi Nagios XI Integration:oracle_weblogic Oracle WebLogic Integration:oracle Oracle Integration:php_fpm PHP-FPM Integration:postgresql PostgreSQL Integration:rabbitmq RabbitMQ Logs and Metrics Integration:redis Redis Integration:redisenterprise Redis Enterprise Integration:salesforce Salesforce Integration:spring_boot Spring Boot Integration:sql_input SQL Input Integration:statsd_input StatsD Input Integration:tomcat Tomcat NetWitness Logs (Deprecated) Integration:traefik Traefik Integration:vsphere VMware vSphere Integration:websphere_application_server WebSphere Application Server Integration:zookeeper ZooKeeper Metrics Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] Team:Security-Service Integrations Security Service Integrations team [elastic/security-service-integrations]
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Stack 9.0.0-SNAPSHOT] failing tests for OBS Non-Cloud Packages
8 participants
@kush-elastic @ishleenk17 @mrodm @jsoriano @elasticmachine @taylor-swanson @harnish-crest-data @andrewkroh