Skip to content

[Prometheus] Add dimension fields to query and collector datastreams for TSDB support #5974

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 9 commits into from
Jun 14, 2023
Merged

[Prometheus] Add dimension fields to query and collector datastreams for TSDB support #5974

merged 9 commits into from
Jun 14, 2023

Conversation

tetianakravchenko
Copy link
Contributor

@tetianakravchenko tetianakravchenko commented Apr 24, 2023 •
edited
Loading

What does this PR do?

Add dimension fields to collector and query datastream:
Prometheus specific fields:

  • labels_id - created as a result of fingerprint processor calculation.
    Note: key and value will be hashed(implementation)
    note2: fingerprint implementation for ingest pipeline is different from the implementation for the beat processor

fields that identify the enviroment where integration is installed:

  • cloud specific: cloud.account.id, cloud.region and cloud.provider
  • host specific: host.name if installed not in cloud

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Screenshots

@tetianakravchenko
Add dimension fields to all datastream for TSDB support
e610d9d 
Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>
@tetianakravchenko tetianakravchenko mentioned this pull request Apr 24, 2023
Closed
13 tasks
@elasticmachine
Copy link

elasticmachine commented Apr 24, 2023 •
edited
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-06-14T12:20:57.529+0000

  • Duration: 15 min 57 sec

Test stats 🧪

Test Results
Failed 0
Passed 9
Skipped 0
Total 9

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@elasticmachine
Copy link

elasticmachine commented Apr 24, 2023 •
edited
Loading

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (0/0) 💚
Files 100.0% (0/0) 💚
Classes 100.0% (0/0) 💚
Methods 72.727% (8/11) 👎 -23.106
Lines 100.0% (0/0) 💚 14.679
Conditionals 100.0% (0/0) 💚

@tetianakravchenko
add ingest pipelines, fix PR number
0c27e0a 
Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>
@tetianakravchenko
fix PR number
893a094 
Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>
@tetianakravchenko
add query_name as an extra label
d59d784 
Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>
@tetianakravchenko
remove remote_write changes - need more investirations; replace some …
66edc5d 
…field, that identify where integration is runnig

Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>
@tetianakravchenko tetianakravchenko changed the title Add dimension fields to all datastream for TSDB support Add dimension fields to query and collector datastreams for TSDB support Apr 28, 2023
@tetianakravchenko tetianakravchenko marked this pull request as ready for review April 28, 2023 15:52
@tetianakravchenko tetianakravchenko requested a review from a team as a code owner April 28, 2023 15:52
@gsantoro
Copy link
Contributor

gsantoro commented May 2, 2023

hey @tetianakravchenko ,
is there any reason why you only created the ingest_pipeline for fingerprinting only for the collector and query datastreams?

@tetianakravchenko
Copy link
Contributor Author

@gsantoro

is there any reason why you only created the ingest_pipeline for fingerprinting only for the collector and query datastreams?

eventually ingest_pipeline will be added for remote_write as well
after migrating to TSDB (after adding labels_id field) I can see dropped documents for remote_write - reason: metrics are not correctly grouped by the labels, that seems like a bug in metricbeat
Since it might need more time to investigate the issue, for now I decided to postpone adding ingest_pipeline for remote_write

@tetianakravchenko tetianakravchenko self-assigned this May 4, 2023
@tetianakravchenko tetianakravchenko changed the title Add dimension fields to query and collector datastreams for TSDB support [Prometheus] Add dimension fields to query and collector datastreams for TSDB support May 8, 2023
@tetianakravchenko
merge, fix conflicts; add dimensions accordingly to elastic#5193 (com…
60ba5d5 
…ment)

Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>
@tetianakravchenko
add dimension to the agent.id fields
de2c058 
Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>
agithomas
agithomas approved these changes Jun 13, 2023
View reviewed changes
Copy link
Contributor

@agithomas agithomas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@tetianakravchenko
Merge branch 'master' into prometheus-tsds
0518006
@tetianakravchenko
merge main branch; fix version conflict
7f98e79 
Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>
@elasticmachine
Copy link

💚 Build Succeeded

cc @tetianakravchenko

@tetianakravchenko tetianakravchenko merged commit 03a618e into elastic:main Jun 14, 2023
@tetianakravchenko tetianakravchenko deleted the prometheus-tsds branch June 14, 2023 12:59
@elasticmachine
Copy link

Package prometheus - 1.5.0 containing this change is available at https://epr.elastic.co/search?package=prometheus

sodhikirti07 pushed a commit that referenced this pull request Jun 15, 2023
@tetianakravchenko @sodhikirti07
[Prometheus] Add dimension fields to query and collector datastreams …
581128e 
…for TSDB support (#5974)

* Add dimension fields to all datastream for TSDB support

Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

* add ingest pipelines, fix PR number

Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

* fix PR number

Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

* add query_name as an extra label

Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

* remove remote_write changes - need more investirations; replace some field, that identify where integration is runnig

Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

* add dimension to the agent.id fields

Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

---------

Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>
@tetianakravchenko tetianakravchenko mentioned this pull request Aug 24, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@gizas gizas gizas approved these changes

@gsantoro gsantoro gsantoro approved these changes

@agithomas agithomas agithomas approved these changes

@devamanv devamanv Awaiting requested review from devamanv devamanv was automatically assigned from elastic/obs-cloudnative-monitoring

@constanca-m constanca-m Awaiting requested review from constanca-m

Assignees

@tetianakravchenko tetianakravchenko

Labels
Integration:prometheus Prometheus
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@tetianakravchenko @elasticmachine @gsantoro @gizas @agithomas @andrewkroh