[MySQL] Add SSL Support #9453

gpop63 · 2024-03-26T16:47:57Z

Overview

Add SSL support.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Screenshots

elasticmachine · 2024-03-26T17:13:18Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

ali786XI · 2024-03-27T04:05:23Z

packages/mysql/manifest.yml

@@ -55,6 +55,35 @@ policy_templates:
            title: Password
            secret: true
            default: test
+          - name: ssl


@gpop63 I don't see this being handled in the stream.yml.hbs file? Example

ali786XI · 2024-03-27T04:06:27Z

packages/mysql/manifest.yml

 description: Collect logs and metrics from MySQL servers with Elastic Agent.
 type: integration
 categories:
  - datastore
  - observability
 conditions:
  kibana:
-    version: "^8.12.0"
+    version: "^8.14.0"


Any reason for bumping kibana version for this SSL enhancement?

The metricbeat changes required for SSL support will be released in 8.14.

ishleenk17

@andrewkroh :
We are adding SSL support to one of our Integrations here.
Has a proposal been formulated regarding the division of this, as was discussed in the meeting?
We can start implementing it from this Integration.

ishleenk17

Looks good!
Please see why the CI is failing.

ishleenk17 · 2024-05-06T09:25:59Z

@gpop63 : Can we resolve the CI issue and conclude on this.
So that we are ready to go when 8.14 is released

gpop63 · 2024-05-08T13:38:01Z

Locally all tests pass (pipeline, system) not sure what the CI doesn't like. Maybe the kibana bump to 8.14 might have something to do with it but locally I tested with 8.14-SNAPSHOT and everything passes.

The strange thing is that commit 5da2b42 was fine but commit 03c13b6 broke the CI.

lalit-satapathy · 2024-05-09T09:06:48Z

Locally all tests pass (pipeline, system) not sure what the CI doesn't like. Maybe the kibana bump to 8.14 might have something to do with it but locally I tested with 8.14-SNAPSHOT and everything passes.

The strange thing is that commit 5da2b42 was fine but commit 03c13b6 broke the CI.

@ishleenk17 how to un-gate this?

jsoriano · 2024-05-09T10:23:21Z

I would propose by now to provide a mapping for host.mac in the performance data stream, as is done in other data streams in the package.

ishleenk17 · 2024-05-09T11:27:31Z

I would propose by now to provide a mapping for host.mac in the performance data stream, as is done in other data streams in the package.

@jsoriano : I think the CI was passing before and failing later because a different elastic package version was used.
Could you please tell me what checks the 0.99 elastic package is having that it started failing if host.mac was not detected in one of the datastreams?
I would assume since ecs.yml are present, it would not use the component template?

jsoriano · 2024-05-09T11:47:05Z

Could you please tell me what checks the 0.99 elastic package is having that it started failing if host.mac was not detected in one of the datastreams?

The issue seems to be related to the changes for ecs@mappings support (elastic/elastic-package#1711), effectively introduced in 0.99.
Since this version, packages that don't support versions of the stack older than 8.13, as is the case here, are validated against the whole ECS. This is because since this version all packages use the ecs@mappings template.
This behaviour can be overriden for specific fields by defining them as in previous versions.

The failure here is produced by the host.mac field in the performance data stream. This field is defined in the other data streams of this package, but not in this one. So elastic-package is using the definition in ECS to validate this field. This definition includes the format seen in the error message: https://github.com/elastic/ecs/blob/f09fa4536990e90d5d3de9cff9d05985356ed7d9/generated/ecs/ecs_flat.yml#L5010

So at this point there are two ways to solve this issue:

Adapt the format of these MAC addresses to the one expected by ECS, but this could be considered breaking as would duplicate values.
Define the host.mac field in the performance data stream, so the definition in ECS is not used.

In previous versions of this package it looks like the host.mac field was not defined, what is also incorrect, I wonder why this was not causing issues before.

I would assume since ecs.yml are present, it would not use the component template?

The ecs@mappings component template is used in all packages starting on 8.13. The ecs.yml file is called like this by convention, it doesn't have any special meaning or trigger any special behaviour, nor the ECS fields need to be defined in this file.

gpop63 · 2024-05-09T12:53:47Z

I tried removing host.mac from other data streams just for testing purposes as suggested by @ishleenk17 to see if tests would fail locally, but they still pass. Is there any way to replicate this locally? I used elastic-package v0.100.0 and stack version 8.14-SNAPSHOT.

jsoriano · 2024-05-09T13:15:25Z

I wonder why this was not causing issues before.

Ok, the lack of definition was not an issue before because we skip validation of some fields that don't have a mapping for legacy reasons, what includes host.*, here: https://github.com/elastic/elastic-package/blob/db4c611d7d8e24e32261c2dc9d73ba77373078b0/internal/fields/validate.go#L582

I tried removing host.mac from other data streams just for testing purposes as suggested by @ishleenk17 to see if tests would fail locally, but they still pass.

The galera_status data stream tests pass because it uses a different format for mac addresses, that seems to comply with ECS definition (02-42-AC-1C-00-07). And the status data stream doesn't seem to collect the host.mac field, at least not with the current test files.

Is there any way to replicate this locally?

I can replicate this locally by running the static tests:

go run github.com/elastic/elastic-package stack up -v --version 8.14.0-SNAPSHOT -d
go run github.com/elastic/elastic-package test static -v

And I can confirm that the issue gets solved by adding a mapping for host.mac.

ishleenk17 · 2024-05-10T04:41:13Z

Thanks @jsoriano for the explanations.
@gpop63 : Can you please check if we can replicate the issue with static test.
Lets add host.mac the way it is present in other datastreams to unblock ourselves.

jsoriano · 2024-05-14T10:11:18Z

/test

elasticmachine · 2024-05-14T10:37:01Z

💚 Build Succeeded

Buildkite Build
Commit: cb5c279

History

💔 Build #11378 failed cb5c279
💔 Build #11274 failed 4b93788
💔 Build #11115 failed 4ed9a63c5e3f1ffa19efaeae556889f21863b663
💔 Build #10205 failed b7a2332
💔 Build #10020 failed 03c13b6

cc @gpop63

elastic-sonarqube · 2024-05-14T10:37:02Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

elasticmachine · 2024-05-17T21:53:45Z

Package mysql - 1.20.0 containing this change is available at https://epr.elastic.co/search?package=mysql

gpop63 added 3 commits March 26, 2024 16:01

add ssl var

e48599a

bump package version

925e2fa

fix pr id

5da2b42

gpop63 marked this pull request as ready for review March 26, 2024 20:15

gpop63 requested a review from a team as a code owner March 26, 2024 20:15

ali786XI reviewed Mar 27, 2024

View reviewed changes

gpop63 added 2 commits April 2, 2024 17:44

add ssl in agent config

03c13b6

format

b7a2332

ishleenk17 reviewed Apr 8, 2024

View reviewed changes

gpop63 self-assigned this Apr 24, 2024

ishleenk17 approved these changes Apr 25, 2024

View reviewed changes

Merge remote-tracking branch 'upstream/main' into add_mysql_ssl_support

4b93788

gpop63 force-pushed the add_mysql_ssl_support branch from 4ed9a63 to 4b93788 Compare May 8, 2024 13:36

gpop63 added 2 commits May 13, 2024 14:23

add agent fields

b2f4731

remove redundant ecs fields

cb5c279

gpop63 merged commit 8a0a45b into elastic:main May 17, 2024

jsoriano mentioned this pull request Jun 12, 2024

[AWS][Pipeline Test] Multiple pipeline Test failure due to event.category mappings #10116

Closed

andrewkroh added the Integration:mysql MySQL label Jul 22, 2024

gpop63 mentioned this pull request Sep 30, 2024

Cisco Meraki metrics package [WIP] #11069

Merged

4 tasks

[MySQL] Add SSL Support #9453

[MySQL] Add SSL Support #9453

Uh oh!

Conversation

gpop63 commented Mar 26, 2024

Overview

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

elasticmachine commented Mar 26, 2024

🚀 Benchmarks report

Uh oh!

ali786XI Mar 27, 2024

Choose a reason for hiding this comment

Uh oh!

ali786XI Mar 27, 2024

Choose a reason for hiding this comment

Uh oh!

gpop63 Apr 2, 2024

Choose a reason for hiding this comment

Uh oh!

ishleenk17 left a comment

Choose a reason for hiding this comment

Uh oh!

ishleenk17 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ishleenk17 commented May 6, 2024

Uh oh!

gpop63 commented May 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lalit-satapathy commented May 9, 2024

Uh oh!

jsoriano commented May 9, 2024

Uh oh!

ishleenk17 commented May 9, 2024

Uh oh!

jsoriano commented May 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gpop63 commented May 9, 2024

Uh oh!

jsoriano commented May 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ishleenk17 commented May 10, 2024

Uh oh!

jsoriano commented May 14, 2024

Uh oh!

elasticmachine commented May 14, 2024

💚 Build Succeeded

History

Uh oh!

elastic-sonarqube bot commented May 14, 2024

Quality Gate passed

Uh oh!

elasticmachine commented May 17, 2024

Uh oh!

Uh oh!

ishleenk17 left a comment •

edited

Loading

gpop63 commented May 8, 2024 •

edited

Loading

jsoriano commented May 9, 2024 •

edited

Loading

jsoriano commented May 9, 2024 •

edited

Loading