[MySQL] Add replica status data stream #10344

niraj-elastic · 2024-07-03T06:18:51Z

Enhancement

What does this PR do?

Added 1 data stream (Replica Status Metrics).
Added data collection logic for the data streams.
Added the ingest pipeline for the data streams.
Mapped fields according to the ECS schema and added Fields metadata in the appropriate yaml files.
Added dashboards and visualizations.
Added system test cases for the data stream.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

Clone integrations repo.
Install elastic-package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/mysql) directory.
Run the following command to run tests. elastic-package test

Screenshots

issue

[mysql] Add replication status to mysql integration #8890

ali786XI · 2024-07-04T07:25:44Z

/test

elasticmachine · 2024-07-04T07:57:16Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

muthu-mps · 2024-07-04T10:54:42Z

packages/mysql/manifest.yml

+            multi: false
+            required: true
+            show_user: true
+            default: "SHOW REPLICA STATUS;"


As the SHOW SLAVE STATUS is deprecated as mentioned in the mysql document. By default we can use the SHOW REPLICA STATUS; query and the user can change it to SLAVE if the mysql version is prior to 8.0.22. We have to update it in the description. That way we support the replica_status metrics for users who is using the older version of mysql.

Sure, we have added it in readme, we will also add description here.

muthu-mps · 2024-07-05T04:31:46Z

Since the ecs@mapping migration is in progress for this integration, what is the plan to merge this PR? Are we planning to merge after ecs template changes?

ishleenk17 · 2024-07-05T04:50:33Z

For any new Integration/datastream, we should now follow the ecs@mappings only.
Please take care of that in this new datastream as well.
If this is an urgent PR, you can make the changes to other datastreams of this integration to adopt ecs@mappings.
If not urgent, please wait for the bulk migration effort to be over

packages/mysql/img/mysql-replica_status-dashboard.png

kush-elastic · 2024-07-08T07:18:33Z

packages/mysql/img/mysql-replica_status-dashboard.png

Is it feasible to display source.port without a comma(shown in SS 3,306)?
Since ports typically don't include separators, having one might appear strange to users.

I agree, but we are not able to show it with out coma in kibana since it is number.

…ql_replica_status Conflicts: packages/mysql/_dev/build/docs/README.md packages/mysql/changelog.yml packages/mysql/data_stream/error/_dev/test/pipeline/test-error.log-expected.json packages/mysql/data_stream/error/_dev/test/pipeline/test-mariadb-10-4-8.log-expected.json packages/mysql/data_stream/error/_dev/test/pipeline/test-mysql-darwin-brew-5-7-10.log-expected.json packages/mysql/data_stream/error/_dev/test/pipeline/test-mysql-ubuntu-5-5-53.log-expected.json packages/mysql/data_stream/error/_dev/test/pipeline/test-mysql-ubuntu-8-0-15.log-expected.json packages/mysql/docs/README.md

kush-elastic

LGTM!

muthu-mps · 2024-07-16T06:27:32Z

Documentation Update

The compatibility matrix has many versions. Can we make this as a table view?
The second section talks about the query that can be used for fetching replica status. Here we can avoid repeating the redundant data. As the newer versions by default use the SHOW REPLICA STATUS query. We can add the older versions of the service should use SHOW SLAVE STATUS query.

muthu-mps · 2024-07-17T05:21:50Z

packages/mysql/data_stream/replica_status/fields/fields.yml

+          description: "Current global transaction ID value."
+        - name: is_auto_position
+          type: boolean
+          description: "true if GTID auto-positioning is in use for the channel, otherwise false."


nit:

Suggested change

description: "true if GTID auto-positioning is in use for the channel, otherwise false."

description: "True if GTID auto-positioning is in use for the channel, otherwise false."

muthu-mps · 2024-07-17T05:31:31Z

packages/mysql/data_stream/replica_status/fields/fields.yml

+            - name: tls_version
+              type: keyword
+              description: "The TLS version used on the source."
+        - name: slave


Are these fields only for SLAVE query of the older versions?

I have changed it to replica.

muthu-mps · 2024-07-17T05:33:06Z

packages/mysql/data_stream/replica_status/fields/fields.yml

+                - name: delay_remaining.sec
+                  type: long
+                  metric_type: gauge
+                  description: "When Replica_SQL_Running_State is Waiting until MASTER_DELAY seconds after source executed event, this field contains the number of delay seconds remaining. At other times, this field is NULL."


Suggested change

description: "When Replica_SQL_Running_State is Waiting until MASTER_DELAY seconds after source executed event, this field contains the number of delay seconds remaining. At other times, this field is NULL."

description: "When Replica_SQL_Running_State is waiting until MASTER_DELAY seconds after source executed event, this field contains the number of delay seconds remaining. At other times, this field is NULL."

…ql_replica_status Conflicts: packages/mysql/changelog.yml

niraj-elastic · 2024-07-22T09:24:52Z

/test

muthu-mps · 2024-07-23T13:28:16Z

Documentation Update

The compatibility matrix has many versions. Can we make this as a table view?

The second section talks about the query that can be used for fetching replica status. Here we can avoid repeating the redundant data. As the newer versions by default use the SHOW REPLICA STATUS query. We can add the older versions of the service should use SHOW SLAVE STATUS query.

Can you please share the screenshot of the updated document that includes the table format?

niraj-elastic · 2024-07-24T06:43:43Z

Documentation Update

The compatibility matrix has many versions. Can we make this as a table view?

The second section talks about the query that can be used for fetching replica status. Here we can avoid repeating the redundant data. As the newer versions by default use the SHOW REPLICA STATUS query. We can add the older versions of the service should use SHOW SLAVE STATUS query.

Can you please share the screenshot of the updated document that includes the table format?

Sure

muthu-mps · 2024-07-24T06:51:58Z

packages/mysql/data_stream/replica_status/fields/fields.yml

+              description: "This status variable counts the occurrence of transactional event groups. This is a replica-side counter for optimistic parallel replication."
+        - name: replicate_do_domain_ids
+          type: keyword
+          description: "The do_domain_id option value for change master.(The DO_DOMAIN_IDS option for CHANGE MASTER can be used to configure a replica to only apply binary log events if the transaction's GTID is in a specific gtid_domain_id value.)"


nit:

Suggested change

description: "The do_domain_id option value for change master.(The DO_DOMAIN_IDS option for CHANGE MASTER can be used to configure a replica to only apply binary log events if the transaction's GTID is in a specific gtid_domain_id value.)"

description: "The do_domain_id option value for change master.(The DO_DOMAIN_IDS option for CHANGE MASTER can be used to configure a replica to only apply binary log events if the transaction's GTID is in a specific gtid_domain_id value)."

I guess it is already correct. https://www.ub.edu/llibre-estil/criteri.php?id=2320#:~:text=When%20a%20passage%20within,before%20the%20closing%20bracket.

muthu-mps · 2024-07-24T06:52:49Z

packages/mysql/data_stream/replica_status/fields/fields.yml

+          description: "The do_domain_id option value for change master.(The DO_DOMAIN_IDS option for CHANGE MASTER can be used to configure a replica to only apply binary log events if the transaction's GTID is in a specific gtid_domain_id value.)"
+        - name: replicate_ignore_domain_ids
+          type: keyword
+          description: "The ignore_domain_id option value for change master.(The IGNORE_DOMAIN_IDS option for CHANGE MASTER can be used to configure a replica to ignore binary log events if the transaction's GTID is in a specific gtid_domain_id value)"


nit:

Suggested change

description: "The ignore_domain_id option value for change master.(The IGNORE_DOMAIN_IDS option for CHANGE MASTER can be used to configure a replica to ignore binary log events if the transaction's GTID is in a specific gtid_domain_id value)"

description: "The ignore_domain_id option value for change master.(The IGNORE_DOMAIN_IDS option for CHANGE MASTER can be used to configure a replica to ignore binary log events if the transaction's GTID is in a specific gtid_domain_id value)."

muthu-mps

Change looks good to me. As @niraj-elastic confirmed the TSDB testing is done as part of E2E testing.

niraj-elastic · 2024-07-24T09:56:54Z

Change looks good to me. As @niraj-elastic confirmed the TSDB testing is done as part of E2E testing.

Yes. @aliabbas-elastic Has done complete E2E testing which includes TSDB as well.

niraj-elastic · 2024-07-24T12:27:03Z

/test

elasticmachine · 2024-07-24T13:09:38Z

💚 Build Succeeded

Buildkite Build
Commit: cf4b364

History

💔 Build #13916 failed cf4b364
💚 Build #13827 succeeded c053c48
💔 Build #13824 failed c053c48
💚 Build #13530 succeeded 9150b01
💔 Build #13528 failed 2a40be7

elastic-sonarqube · 2024-07-24T13:09:42Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
96.4% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

* add replica status data stream * update data stream * update changelog * Address review comment * address review comments * address review comment * address review comments

add replica status data stream

4fd9f19

niraj-elastic requested a review from a team as a code owner July 3, 2024 06:18

niraj-elastic changed the title ~~add replica status data stream~~ [MySQL] Add replica status data stream Jul 3, 2024

niraj-elastic added 2 commits July 3, 2024 14:38

update data stream

741ca36

update changelog

2c631bd

niraj-elastic requested review from muthu-mps, lalit-satapathy and kush-elastic July 4, 2024 09:42

muthu-mps reviewed Jul 4, 2024

View reviewed changes

kush-elastic reviewed Jul 8, 2024

View reviewed changes

niraj-elastic added 3 commits July 8, 2024 12:48

Address review comment

b5630e5

address review comments

9150b01

niraj-elastic requested review from muthu-mps and kush-elastic July 11, 2024 09:42

kush-elastic approved these changes Jul 12, 2024

View reviewed changes

muthu-mps reviewed Jul 17, 2024

View reviewed changes

andrewkroh added Integration:mysql MySQL Team:Obs-InfraObs Observability Infrastructure Monitoring team [elastic/obs-infraobs-integrations] labels Jul 19, 2024

niraj-elastic added 2 commits July 22, 2024 12:46

Merge branch 'main' of github.com:niraj-elastic/integrations into mys…

41a8471

…ql_replica_status Conflicts: packages/mysql/changelog.yml

address review comment

c053c48

niraj-elastic requested a review from muthu-mps July 22, 2024 12:06

muthu-mps reviewed Jul 24, 2024

View reviewed changes

address review comments

cf4b364

niraj-elastic requested a review from muthu-mps July 24, 2024 09:45

muthu-mps approved these changes Jul 24, 2024

View reviewed changes

niraj-elastic merged commit 1ddff35 into elastic:main Jul 26, 2024
5 checks passed

niraj-elastic mentioned this pull request Jul 26, 2024

[mysql] Add replication status to mysql integration #8890

Closed

	description: "true if GTID auto-positioning is in use for the channel, otherwise false."
	description: "True if GTID auto-positioning is in use for the channel, otherwise false."

	description: "When Replica_SQL_Running_State is Waiting until MASTER_DELAY seconds after source executed event, this field contains the number of delay seconds remaining. At other times, this field is NULL."
	description: "When Replica_SQL_Running_State is waiting until MASTER_DELAY seconds after source executed event, this field contains the number of delay seconds remaining. At other times, this field is NULL."

	description: "The do_domain_id option value for change master.(The DO_DOMAIN_IDS option for CHANGE MASTER can be used to configure a replica to only apply binary log events if the transaction's GTID is in a specific gtid_domain_id value.)"
	description: "The do_domain_id option value for change master.(The DO_DOMAIN_IDS option for CHANGE MASTER can be used to configure a replica to only apply binary log events if the transaction's GTID is in a specific gtid_domain_id value)."

	description: "The ignore_domain_id option value for change master.(The IGNORE_DOMAIN_IDS option for CHANGE MASTER can be used to configure a replica to ignore binary log events if the transaction's GTID is in a specific gtid_domain_id value)"
	description: "The ignore_domain_id option value for change master.(The IGNORE_DOMAIN_IDS option for CHANGE MASTER can be used to configure a replica to ignore binary log events if the transaction's GTID is in a specific gtid_domain_id value)."

[MySQL] Add replica status data stream #10344

[MySQL] Add replica status data stream #10344

Uh oh!

Conversation

niraj-elastic commented Jul 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Checklist

How to test this PR locally

Screenshots

issue

Uh oh!

ali786XI commented Jul 4, 2024

Uh oh!

elasticmachine commented Jul 4, 2024

🚀 Benchmarks report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

muthu-mps commented Jul 5, 2024

Uh oh!

ishleenk17 commented Jul 5, 2024

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kush-elastic left a comment

Choose a reason for hiding this comment

Uh oh!

muthu-mps commented Jul 16, 2024

Documentation Update

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

niraj-elastic commented Jul 22, 2024

Uh oh!

muthu-mps commented Jul 23, 2024

Documentation Update

Uh oh!

niraj-elastic commented Jul 24, 2024

Documentation Update

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

muthu-mps left a comment

Choose a reason for hiding this comment

Uh oh!

niraj-elastic commented Jul 24, 2024

Uh oh!

niraj-elastic commented Jul 24, 2024

Uh oh!

elasticmachine commented Jul 24, 2024

💚 Build Succeeded

History

Uh oh!

elastic-sonarqube bot commented Jul 24, 2024

Quality Gate passed

Uh oh!

Uh oh!

Uh oh!

niraj-elastic commented Jul 3, 2024 •

edited

Loading