Add Logs data stream for collecting Azure Functions #6417

devamanv · 2023-06-01T04:05:15Z

What does this PR do?

The PR contains changes to add a new datastream functionapplogs that contains all the field mappings, ingest pipelines, documentations needed to ingest the Azure Functions logs into Elasticsearch.

Note: Dashboards assets to be added in a separate PR.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

Clone the integrations repo
Install elastic-package locally
Spin up an elastic stack using elastic-package
Run elastic-package test from the integrations/packages/azure_functions directory

Related issues

Implements logs datastream for [Meta] Azure Functions Integration #6086

Screenshots

elasticmachine · 2023-06-01T04:10:50Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-07-18T16:33:45.282+0000
Duration: 16 min 55 sec

Test stats 🧪

Test	Results
Failed	0
Passed	5
Skipped	0
Total	5

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.

zmoog

This PR looks good; I have a few minor changes to request.

packages/azure_functions/LICENSE.txt

packages/azure_functions/_dev/build/docs/README.md

packages/azure_functions/data_stream/functionapplogs/agent/stream/azure-eventhub.yml.hbs

packages/azure_functions/data_stream/functionapplogs/elasticsearch/ingest_pipeline/default.yml

packages/azure_functions/data_stream/functionapplogs/fields/package-fields.yml

packages/azure_functions/docs/README.md

packages/azure_functions/manifest.yml

elasticmachine · 2023-06-15T17:05:34Z

🌐 Coverage report

Name	Metrics % (`covered/total`)	Diff
Packages	100.0% (`1/1`)	💚
Files	100.0% (`2/2`)	💚
Classes	100.0% (`2/2`)	💚
Methods	92.857% (`13/14`)	👍 4.978
Lines	96.753% (`149/154`)	👎 -0.719
Conditionals	100.0% (`0/0`)	💚

gpop63 · 2023-07-03T14:41:43Z

packages/azure_functions/data_stream/functionapplogs/elasticsearch/ingest_pipeline/default.yml

+      ignore_missing: true
+  - rename:
+      field: json.properties.functionInvocationId
+      target_field: azure.function.function_invocation_id


I wonder if azure.function.function_invocation_id is redundant and something like azure.function.invocation_id make more sense, this applies to azure.function.function_name too. What do you think?

Yes, that's a fair point. I will change that in the next commit.

packages/azure_functions/_dev/build/docs/README.md

packages/azure_functions/data_stream/functionapplogs/elasticsearch/ingest_pipeline/default.yml

muthu-mps · 2023-07-07T12:38:41Z

@devamanv - Please find my comments below after installing the enabling the data collection for azure functions.

When starting the integration for the first time there is a few minutes delay observed in the data collection. Is this the expected behaviour for a first time installer ?
The assets is not available for the integration.The PR description has a mention of having assets along with this PR. Can we create assets.
The logo in the Azure function's section of the Azure portal is slightly different than what we have. Can we check and update it.

devamanv · 2023-07-11T08:08:22Z

@muthu-mps

When starting the integration for the first time there is a few minutes delay observed in the data collection. Is this the expected behaviour for a first time installer ?

Events from Azure are usually slow, and this delay is expected and could take anywhere between a few minutes to an hour, reference. Part of the reason is the fact that Platform logs are only collected when there's a corresponding diagnostic setting enabled, source. This usually takes a few minutes for the logs to be routed to a destination, which happens to be Eventhub in this case.
This may also have to do with how the eventhub input works under the hood, it needs to start a worker, establish a connection with Azure Eventhub and at the same time store metadata in an instance of the Storage Account.

The assets is not available for the integration.The PR description has a mention of having assets along with this PR. Can we create assets

The assets(dashboards) will be created as a separate PR. The assets work is already in progress. I have updated the description accordingly.

The logo in the Azure function's section of the Azure portal is slightly different than what we have. Can we check and update it.

I have updated the logo to match the one in the Azure Portal, please take a look.

muthu-mps · 2023-07-15T01:54:15Z

packages/azure_functions/manifest.yml

+format_version: 2.5.1
+name: azure_functions
+title: "Azure Functions"
+version: 0.1.0


Can we make the initial release version as experimental 0.0.1 until the Kibana dashboard PR is getting merged. Otherwise looks good !

Changed the release as experimental and the initial package version to 0.0.1.

zmoog · 2023-07-18T05:28:38Z

When starting the integration for the first time there is a few minutes delay observed in the data collection. Is this the expected behaviour for a first time installer ?

To add extra details to what @devamanv said, here's how to check what happens between event creation on Azure and ingestion on Elasticsearch using the document fields.

Here's a KB article I put together a few months ago. AFAIK it's unavailable as a public page, so I'm adding it as a screenshot.

We can turn it into a public resource if you think it can be helpful to a broader audience.

Update the ECS version to use the v8.8.0 tag instead of using the 8.8 branch. More precise dependencies should enable more predictable builds.

Align the content of the `sample_event.json` file with the new field names.

…egrations into azure-functions-integration

muthu-mps

Looks good !

elasticmachine · 2023-07-19T09:34:29Z

Package azure_functions - 0.0.1 containing this change is available at https://epr.elastic.co/search?package=azure_functions

Add initial set of config and fields for the azure function logs

b9282e5

devamanv added 9 commits June 1, 2023 15:54

Update changelog and add raw log sample

46675cc

Fix a typo in agent config .hbs file

536d5dd

Add ingest pipelines and field mappings

e960d96

Remove unwanted processors from the ingest pipeline

95d33e6

Add the expected json file

e8a5ba3

Fix fields type,add error log sample, icons and tweek the pipeline

f94c533

Add LICENSE.txt

64b77a7

Add sanitize options to agent config and move function fields together

9492f14

Add README.md to _dev/build/docs folder

2513ce2

devamanv requested review from zmoog and a team June 14, 2023 23:45

devamanv marked this pull request as ready for review June 14, 2023 23:45

devamanv added 5 commits June 15, 2023 17:16

Update shared pipeline to include the azure resource field

43049e5

Update OWNERS.md for the azure_functions package

148085d

Update owner in manifest.yml to fix CI errors

3b03d24

Fix the README.md through elastic-package check

3c1b832

Change the package version to match the Changelog file

0ff493b

zmoog requested changes Jun 15, 2023

View reviewed changes

devamanv requested review from gpop63 and lucian-ioan June 15, 2023 15:29

devamanv added 2 commits June 15, 2023 21:29

Address review comments and remove unwanted fields

9eb38d2

Update package docs after rebuilding

891caf7

devamanv added 5 commits June 15, 2023 19:06

Add initial set of config and fields for the azure function logs

893e222

Update changelog and add raw log sample

0c5695e

Fix a typo in agent config .hbs file

da850a0

Add ingest pipelines and field mappings

f5860a3

Remove unwanted processors from the ingest pipeline

398eb7e

devamanv removed a link to an issue Jun 27, 2023

[Meta] Azure Functions Integration #6086

Closed

2 tasks

devamanv linked an issue Jun 27, 2023 that may be closed by this pull request

Add Logs data stream for Azure Functions #6723

Closed

5 tasks

devamanv changed the title ~~Add an integration package for collecting Azure Function Logs~~ Add Logs data stream for collecting Azure Functions Jun 27, 2023

gpop63 reviewed Jul 3, 2023

View reviewed changes

Change field type to password to mask the fields

23d052a

devamanv mentioned this pull request Jul 6, 2023

[Meta] Azure Functions Integration #6086

Closed

2 tasks

devamanv and others added 2 commits July 6, 2023 20:13

Change field names and add event_id field inside the function app log

f6461cf

Merge branch 'main' into azure-functions-integration

7ced07e

devamanv requested a review from gpop63 July 6, 2023 14:51

muthu-mps reviewed Jul 7, 2023

View reviewed changes

packages/azure_functions/_dev/build/docs/README.md Show resolved Hide resolved

packages/azure_functions/data_stream/functionapplogs/elasticsearch/ingest_pipeline/default.yml Outdated Show resolved Hide resolved

Update processor reference and the logo to match the azure portal

51fa525

devamanv requested a review from muthu-mps July 11, 2023 08:09

devamanv added 3 commits July 12, 2023 02:43

Remove the unused date and rename processors from the pipeline

6633a1e

Remove the redundant link to the Azure docs

fe87b08

Update README.md doc file to fix the CI error

07d4315

muthu-mps reviewed Jul 15, 2023

View reviewed changes

Change release to experimental and the initial package version to 0.0.1

b95db39

devamanv and others added 4 commits July 18, 2023 19:22

Remove the release:experimental field from the manifest, not required

c84eb9f

Use ECS@v8.8.0

fee6d92

Update the ECS version to use the v8.8.0 tag instead of using the 8.8 branch. More precise dependencies should enable more predictable builds.

Update sample_event.json

06face1

Align the content of the `sample_event.json` file with the new field names.

Merge branch 'azure-functions-integration' of github.com:devamanv/int…

d82bc29

…egrations into azure-functions-integration

muthu-mps approved these changes Jul 19, 2023

View reviewed changes

devamanv merged commit 2d586eb into elastic:main Jul 19, 2023

andrewkroh added the Integration:azure_functions Azure Functions label Jul 22, 2024

andrewkroh added the New Integration Issue or pull request for creating a new integration package. label Aug 13, 2024

Add Logs data stream for collecting Azure Functions #6417

Add Logs data stream for collecting Azure Functions #6417

Uh oh!

Conversation

devamanv commented Jun 1, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

elasticmachine commented Jun 1, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💚 Build Succeeded

Build stats

Test stats 🧪

🤖 GitHub comments

Uh oh!

zmoog left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

elasticmachine commented Jun 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🌐 Coverage report

Uh oh!

gpop63 Jul 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

devamanv Jul 6, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

muthu-mps commented Jul 7, 2023

Uh oh!

devamanv commented Jul 11, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

muthu-mps Jul 15, 2023

Choose a reason for hiding this comment

Uh oh!

devamanv Jul 17, 2023

Choose a reason for hiding this comment

Uh oh!

zmoog commented Jul 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

muthu-mps left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Jul 19, 2023

Uh oh!

Uh oh!

devamanv commented Jun 1, 2023 •

edited

Loading

elasticmachine commented Jun 1, 2023 •

edited

Loading

zmoog left a comment •

edited

Loading

elasticmachine commented Jun 15, 2023 •

edited

Loading

gpop63 Jul 3, 2023 •

edited

Loading

devamanv commented Jul 11, 2023 •

edited

Loading

zmoog commented Jul 18, 2023 •

edited

Loading