[GCP] Add gcp.metrics input package #4436

gpop63 · 2022-10-10T14:39:56Z

What does this PR do?

Adds gcp.metrics input package.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Closes [GCP] Migrate gcp.metrics beats metricset #4292

Screenshots

elasticmachine · 2022-10-10T14:45:12Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-12-27T00:17:41.903+0000
Duration: 13 min 24 sec

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

endorama · 2022-10-19T10:00:32Z

packages/gcp_metrics/manifest.yml

+        multi: false
+        required: false
+        show_user: true
+      - name: region


Make sure to add regions too as a setting

If this is a new input and we want to remove the region setting, consider not adding it here.

@endorama since region will be deprecated in other gcp data streams, should it be deprecated in this package as well or removed entirely?

I think we can remove it entirely, as this is a new package.

tetianakravchenko · 2022-10-19T11:20:58Z

Hi @gpop63!

Did you try to use this input package and deploy the elastic-agent that use policy with this input package?
Do you see dataset created with the name set as a Dataset name? Do you see data ingested?

I am trying to use input package as well, but for some reason I have an issue with dataset and data ingestion.

tetianakravchenko · 2022-10-19T11:41:35Z

packages/gcp_metrics/manifest.yml

+    type: metrics
+    title: GCP Metrics
+    description: GCP Metrics.
+    input: gcp/metrics


why here is used /metrics in the input name? does it work this way?

for example: in https://github.com/elastic/package-spec repo we have an example of input package (sql input) https://github.com/elastic/package-spec/blob/main/test/packages/sql_input/manifest.yml#L20
and here is used input: sql

I've successfully managed to ingest data. However, I don't think the Dataset name is reflected. The input field should be the metricset from beats (I think). I've tried different inputs for this gcp package and somehow gcp/metrics works.

@jsoriano could you help regarding the expected input here?

in sql example is used name of the module. Here used module_name/metricset.

Could you pleas clarify what is correct?

gcp/metrics sounds good, if you are trying to use the GCP module of Metricbeat. This input is also used on other packages, like here.

In general, for any input called <module>/metrics, Elastic Agent is going to use the <module> module of Metricbeat. These mappings are defined in Elastic Agent specs, specifically for these inputs in Metricbeat here.

in sql example is used name of the module. Here used module_name/metricset.

This is probably an error. Please take this example module with a grain of salt, this was used to test the package spec, but not actually used for real.

tetianakravchenko · 2022-10-20T15:25:16Z

packages/gcp_metrics/agent/input/input.yml.hbs

+    metric_types:
+        {{#each metrics}}
+        - "{{this}}"
+        {{/each}}


could you try to add

data_stream: dataset: {{data_stream.dataset}}

to see if dataset name will be respected in this case?

jsoriano · 2022-10-20T15:27:06Z

packages/gcp_metrics/manifest.yml

+        required: true
+        show_user: true
+owner:
+  github: elastic/integrations


Please be more specific with the owner, should it be the cloud monitoring team here?

Suggested change

github: elastic/integrations

github: elastic/obs-cloud-monitoring

Should be elastic/obs-cloud-monitoring 👍

botelastic · 2022-11-25T09:01:54Z

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

…ut_pkg

tommyers-elastic

LGTM - do we have any user facing docs in the works to explain why this package has been created? perhaps we could put something in the README here? i'm assuming that once merged this will show up in the kibana UI, and could possibly cause confusion for customers.

gpop63 · 2022-12-20T16:37:11Z

I added a package readme 5ad949c, not sure if it's descriptive enough. WDYT?

endorama · 2022-12-22T13:31:18Z

@gpop63 I'll rework a bit the README to be more informative, and make clear this is still experimental (also should not show up in Kibana as version is <1.0.0)

endorama · 2022-12-22T13:31:50Z

packages/gcp_metrics/docs/README.md

@@ -0,0 +1,7 @@
+# GCP Metrics Input


Suggested change

# GCP Metrics Input

# GCP Custom Metrics

endorama · 2022-12-22T13:32:06Z

packages/gcp_metrics/docs/README.md

@@ -0,0 +1,7 @@
+# GCP Metrics Input
+
+The GCP metrics input package can collect metrics for any GCP service.


Suggested change

The GCP metrics input package can collect metrics for any GCP service.

The GCP Custom Metrics input package can collect custom metrics for any GCP service.

endorama · 2022-12-22T13:42:56Z

packages/gcp_metrics/docs/README.md

+
+A list of metrics and services that are available, can be found in the [GCP Cloud Monitoring API](https://cloud.google.com/monitoring/api/metrics_gcp) official documentation.
+
+Custom ingest pipelines can be added through the Kibana UI, to get the data in the desired format.


Suggested change

Custom ingest pipelines can be added through the Kibana UI, to get the data in the desired format.

This package does not contain any ingest pipeline, so no pre-ingest data processing is applied out of the box. Custom ingest pipelines can be added through the Kibana UI, to get the data in the desired format.

packages/gcp_metrics/docs/README.md

elasticmachine · 2022-12-27T00:51:05Z

Package gcp_metrics - 0.0.1 containing this change is available at https://epr.elastic.co/search?package=gcp_metrics

add gcp metrics input package

9b4c702

gpop63 added Team:Cloud-Monitoring Label for the Cloud Monitoring team Integration:gcp Google Cloud Platform labels Oct 10, 2022

gpop63 self-assigned this Oct 10, 2022

endorama reviewed Oct 19, 2022

View reviewed changes

tetianakravchenko reviewed Oct 19, 2022

View reviewed changes

add regions support

d25ddd3

tetianakravchenko reviewed Oct 20, 2022

View reviewed changes

jsoriano reviewed Oct 20, 2022

View reviewed changes

gpop63 added 2 commits October 20, 2022 19:24

add datastream dataset

cb8d1af

fix owner

8dd76cb

botelastic bot added Stalled and removed Stalled labels Nov 25, 2022

gpop63 added 5 commits December 15, 2022 16:15

Merge remote-tracking branch 'upstream/main' into add_gcp-metrics_inp…

6bb1db8

…ut_pkg

add package codeowner

56941ec

remove region setting

c19e29f

remove input name field description

fc8eb1d

update docs

5ad949c

gpop63 marked this pull request as ready for review December 15, 2022 15:13

endorama approved these changes Dec 19, 2022

View reviewed changes

tommyers-elastic self-requested a review December 19, 2022 15:32

tommyers-elastic approved these changes Dec 19, 2022

View reviewed changes

endorama reviewed Dec 22, 2022

View reviewed changes

packages/gcp_metrics/docs/README.md Show resolved Hide resolved

endorama and others added 2 commits December 22, 2022 14:45

Merge branch 'main' into add_gcp-metrics_input_pkg

57fa658

Update README.md

ccede26

gpop63 merged commit 9daefef into elastic:main Dec 27, 2022

andrewkroh added Integration:gcp_metrics GCP Metrics Input New Integration Issue or pull request for creating a new integration package. and removed Integration:gcp Google Cloud Platform labels Aug 13, 2024

	github: elastic/integrations
	github: elastic/obs-cloud-monitoring

		@@ -0,0 +1,7 @@
		# GCP Metrics Input

		The GCP metrics input package can collect metrics for any GCP service.

	The GCP metrics input package can collect metrics for any GCP service.
	The GCP Custom Metrics input package can collect custom metrics for any GCP service.


		A list of metrics and services that are available, can be found in the [GCP Cloud Monitoring API](https://cloud.google.com/monitoring/api/metrics_gcp) official documentation.

		Custom ingest pipelines can be added through the Kibana UI, to get the data in the desired format.

	Custom ingest pipelines can be added through the Kibana UI, to get the data in the desired format.
	This package does not contain any ingest pipeline, so no pre-ingest data processing is applied out of the box. Custom ingest pipelines can be added through the Kibana UI, to get the data in the desired format.

[GCP] Add gcp.metrics input package #4436

[GCP] Add gcp.metrics input package #4436

Uh oh!

Conversation

gpop63 commented Oct 10, 2022

What does this PR do?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

elasticmachine commented Oct 10, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

🤖 GitHub comments

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tetianakravchenko commented Oct 19, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

botelastic bot commented Nov 25, 2022

Uh oh!

tommyers-elastic left a comment

Choose a reason for hiding this comment

Uh oh!

gpop63 commented Dec 20, 2022

Uh oh!

endorama commented Dec 22, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elasticmachine commented Dec 27, 2022

Uh oh!

Uh oh!

elasticmachine commented Oct 10, 2022 •

edited

Loading