Skip to content

Add enable_batch_api option in azure resource metrics #13783

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

MichaelKatsoulis
Copy link
Contributor

Follow-up of beats PR

Proposed commit message

  • WHAT: Adds enable_batch_api configuration parameter in azure resource metrics.
  • WHY: Helps mitigating scalability problems

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

  • [ ]

How to test this PR locally

It can be easily tested by installing Azure Resource Metrics. Enable the enable_batch_api advanced setting.
Then the metrics will be collected from the azure batch api.

Related issues

Screenshots

enable

data

@MichaelKatsoulis MichaelKatsoulis requested a review from a team as a code owner May 5, 2025 11:19
@MichaelKatsoulis MichaelKatsoulis requested a review from zmoog May 5, 2025 11:20
@elastic-vault-github-plugin-prod
Copy link

elastic-vault-github-plugin-prod bot commented May 5, 2025

🚀 Benchmarks report

Package azure_metrics 👍(2) 💚(1) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
container_service 200000 142857.14 -57142.86 (-28.57%) 💔

To see the full report comment with /test benchmark fullreport

@andrewkroh andrewkroh added Integration:azure_metrics Azure Resource Metrics Team:obs-ds-hosted-services Observability Hosted Services team [elastic/obs-ds-hosted-services] labels May 5, 2025
Comment on lines +70 to +80
- name: enable_batch_api
type: bool
title: Enable Batch Api
description: >
When enabled, the azure batch api will be used
to fetch metrics of multiple resources in one api call.
Enable this when facing scalability issues.
multi: false
required: false
show_user: false
default: false
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nits: minor capitalization suggestions.

Suggested change
- name: enable_batch_api
type: bool
title: Enable Batch Api
description: >
When enabled, the azure batch api will be used
to fetch metrics of multiple resources in one api call.
Enable this when facing scalability issues.
multi: false
required: false
show_user: false
default: false
- name: enable_batch_api
type: bool
title: Enable Batch API
description: >
When enabled, the Azure batch API will be used
to fetch metrics of multiple resources in one API call.
Enable this when facing scalability issues.
multi: false
required: false
show_user: false
default: false

Comment on lines +192 to +196
`Enable Batch Api` _boolean_
: Optional, by default is set to False. Set this to True when facing scalability issues. When configured, the azure batch api will be used
to fetch metrics of multiple resources in one api call.
Currently supported data streams are monitor, container_registry, container_instance, container_service, compute_vm, compute_vm_scaleset, database_account and storage_account.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit:

Suggested change
`Enable Batch Api` _boolean_
: Optional, by default is set to False. Set this to True when facing scalability issues. When configured, the azure batch api will be used
to fetch metrics of multiple resources in one api call.
Currently supported data streams are monitor, container_registry, container_instance, container_service, compute_vm, compute_vm_scaleset, database_account and storage_account.
`Enable Batch API` _boolean_
: Optional, by default is set to False. Set this to True when facing scalability issues. When configured, the Azure batch API will be used
to fetch metrics of multiple resources in one API call.
Currently supported data streams are monitor, container_registry, container_instance, container_service, compute_vm, compute_vm_scaleset, database_account and storage_account.

@@ -21,7 +21,7 @@ categories:
- custom
conditions:
kibana:
version: "^8.12.0 || ^9.0.0"
version: "^9.0.1"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is only partially related to the changes in this PR.

Is it worth backporting the Beats PR to one or more 8.x versions? Possible candidates: 8.17, 8.18, 8.19.

The reason is practical. We will support the last 8.x series (8.19.x?) for a while, as we're still doing for 7.17.x today—to deliver fixes and small improvements.

It's better to do it now than later, while the memory of the changes in the PR is still relatively fresh. We can merge this PR now, and add 8.x compatibility conditions later when the backport(s) is/are merged.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MichaelKatsoulis, I updated the kibana.version to match the upcoming 8.x releases supporting the batch API.

We can merge this PR when the next 8.17.7 and 8.18.2 releases become available.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we wait for that long then it won't be available for 9.0.1 until May 27th.
I guess we can live with that, although I don't see any reason we should not merge it now.

@zmoog zmoog self-assigned this May 6, 2025
We now support stack 8.17.7, 8.18.2, and later.
Copy link

@elasticmachine
Copy link

💚 Build Succeeded

History

cc @zmoog

@zmoog zmoog merged commit e55fb31 into elastic:main Jun 3, 2025
7 checks passed
@elastic-vault-github-plugin-prod

Package azure_metrics - 1.8.0 containing this change is available at https://epr.elastic.co/package/azure_metrics/1.8.0/

@andrewkroh andrewkroh added the documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. label Jul 1, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation. Applied to PRs that modify *.md files. Integration:azure_metrics Azure Resource Metrics Team:obs-ds-hosted-services Observability Hosted Services team [elastic/obs-ds-hosted-services]
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Improve Azure Monitor scalability and performance
4 participants