[Azure] Update Azure Firewall pipeline #9428

lucian-ioan · 2024-03-22T14:37:34Z

Proposed commit message

Update Azure Firewall pipeline due to new format introduced by Azure via Resource Specific Structured Logging.

Currently adding support for the following categories:

AZFWNatRule
AZFWNetworkRule
AZFWApplicationRule
AZFWDnsQuery

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Closes [Azure Firewall] Certain Logs are not being parsed correctly #9037

Screenshots

elasticmachine · 2024-03-22T14:52:37Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

muthu-mps · 2024-04-16T08:44:05Z

...firewall_logs/_dev/test/pipeline/test-azfw-application-rule-structured-raw.log-expected.json

@@ -0,0 +1,30 @@
+{
+    "expected": [


Why the log fields are missing for this category AZFWApplicationRule? Do we need to re-run the pipeline test and generate the sample metrics with all the fields added to the expected json?

muthu-mps · 2024-04-16T08:45:10Z

..._stream/firewall_logs/_dev/test/pipeline/test-azfw-nat-rule-structured-raw.log-expected.json

+                    "policy": "policy",
+                    "protocol": "TCP",
+                    "rule": "rule",
+                    "rule_collection": "rule collection",


Can we provide a meaningful name for the rule, rule_collection and the rule_collection_group?

muthu-mps · 2024-04-16T08:54:00Z

packages/azure/data_stream/firewall_logs/fields/fields.yml

@@ -50,6 +39,130 @@
      type: keyword
      description: |
        Operation name
+    - name: destination_ip


Can we leverage the ECS fields for source and destination IP address and port?

muthu-mps · 2024-04-16T09:07:54Z

@lucian-ioan

Update the documentation to capture the supported categories. Add few lines about the structured logs.
Adding the current version of the document below

Add the screenshot of the dashboard image to this PR which would help to cross check with the previous version.

packages/azure/data_stream/firewall_logs/elasticsearch/ingest_pipeline/default.yml

...ta_stream/firewall_logs/_dev/test/pipeline/test-networkrule-structured-raw.log-expected.json

...ream/firewall_logs/_dev/test/pipeline/test-applicationrules-structured-raw.log-expected.json

.../data_stream/firewall_logs/_dev/test/pipeline/test-dnsproxy-structured-raw.log-expected.json

kcreddy

Minor suggestions on my end

packages/azure/data_stream/firewall_logs/elasticsearch/ingest_pipeline/default.yml

…pipeline/default.yml Co-authored-by: Krishna Chaitanya Reddy Burri <krish.reddy91@gmail.com>

lucian-ioan · 2024-07-05T13:18:44Z

Thank you @kcreddy for the help, suggestions added.

kcreddy

LGTM for my comments

…line

zmoog

LGTM!

My only suggestion is to clarify and expand the docs about diagnostics vs. resource-specific.

@lucian-ioan, does the integration support all resource-specific log categories for Azur Firewall, or are there more?

I have a vague recall of a long list of log categories.

zmoog · 2024-07-10T11:38:57Z

packages/azure/_dev/build/docs/firewall_logs.md

+| Log Category                 | Description                                                                                                                          | Destination Table  |
+|:----------------------------:|:------------------------------------------------------------------------------------------------------------------------------------:|:------------------:|
+| AzureFirewallApplicationRule | These logs capture information about the traffic that is allowed or denied by application rules configured in Azure Firewall.        | Azure diagnostics  |
+| AzureFirewallNetworkRule     | These logs capture information about the traffic that is allowed or denied by network rules configured in Azure Firewall.            | Azure diagnostics  |
+| AzureFirewallDnsProxy        | These logs capture information about DNS requests and responses that are processed by Azure Firewall's DNS proxy.                    | Azure diagnostics  |
+| AZFWApplicationRule          | These logs capture resource specific information about the traffic that is allowed or denied by application rules configured in Azure Firewall.                  | Resource specific  |
+| AZFWNetworkRule              | These logs capture resource specific information about the traffic that is allowed or denied by network rules configured in Azure Firewall.                  | Resource specific  |
+| AZFWNatRule                  | These logs capture resource specific information about all DNAT (Destination Network Address Translation) events log data.                  | Resource specific  |
+| AZFWDnsQuery                 | These logs capture resource specific information about DNS requests and responses that are processed by Azure Firewall's DNS proxy.                  | Resource specific  |
+
+For detailed information and instructions on how to migrate to Resource-specific mode, please refer to the following Microsoft documentation: [Azure Monitor Resource Logs](https://learn.microsoft.com/en-gb/azure/azure-monitor/essentials/resource-logs#resource-specific).


It's good we're adding more info about the collection mode (azure diagnostics vs. resource specific)!

I would expand this section a little to convey more context, using the info mentioned in the doc from Azure you linked (and leaving all the other fine details to the Azure doc).

From https://learn.microsoft.com/en-gb/azure/azure-monitor/essentials/resource-logs#resource-specific:

Most Azure resources write data to the workspace in either Azure diagnostics or resource-specific mode without giving you a choice.

All Azure services will eventually use the resource-specific mode. As part of this transition, some resources allow you to select a mode in the diagnostic setting. Specify resource-specific mode for any new diagnostic settings because this mode makes the data easier to manage.

So it's also clear that users can have the log categories from Azure Diagnostics or resource-specific, but not both.

zmoog · 2024-07-10T11:41:16Z

packages/azure/manifest.yml

@@ -1,6 +1,6 @@
 name: azure
 title: Azure Logs
-version: 1.12.0
+version: 1.12.1


Since this is a new feature (adds support to additional log categories) we can probably use v1.13.0. WDYT?

lucian-ioan · 2024-07-11T08:31:23Z

@zmoog There are a couple more categories, this PR specifically adds support for the most important ones (which are currently used by customers).

elasticmachine · 2024-07-11T08:50:23Z

💚 Build Succeeded

Buildkite Build
Commit: 88c954c

History

💚 Build #13487 succeeded 53102f1
💚 Build #13328 succeeded 4158fee
💚 Build #13315 succeeded 699a945
💚 Build #13303 succeeded 8c68c23
💚 Build #13294 succeeded 9bdd073
💔 Build #13292 failed aeacc79dfe59fbb52c2c9b8fd002571490edeca7

cc @lucian-ioan

elastic-sonarqube · 2024-07-11T08:50:28Z

Quality Gate failed

Failed conditions
63.9% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

elasticmachine · 2024-07-11T09:14:19Z

Package azure - 1.13.0 containing this change is available at https://epr.elastic.co/search?package=azure

lucian-ioan added 2 commits March 22, 2024 16:32

add changelog entry

bc1b365

update pull number

f39ca8c

lucian-ioan added 3 commits March 24, 2024 19:30

add AZFWNatRule sample log

bf36a59

add AZFWNetworkRule sample log

c11ef32

add AZFWApplicationRule sample log

9f07b8f

lucian-ioan self-assigned this Mar 24, 2024

lucian-ioan added the Integration:azure Azure Logs label Mar 24, 2024

lucian-ioan added 10 commits March 26, 2024 11:36

update NatRule log type

1595ba7

update NetworkRule log type

600d310

update AZFWApplicationRule log type

dd80d1f

update resource id

78ad0e1

elastic-package build

36f8a92

lowercase test files

37f4d20

remove redundant fields

ff22bb6

fix bool to bolean type

7c6c1f8

add structured dns query logs

32af0f0

update all test cases

4da4444

lucian-ioan marked this pull request as ready for review April 10, 2024 09:56

lucian-ioan requested review from a team as code owners April 10, 2024 09:56

lucian-ioan requested a review from muthu-mps April 10, 2024 09:56

lucian-ioan added 2 commits April 10, 2024 13:00

reword changelog message

f9d1a77

update ip addresses

4e5b4c3

muthu-mps reviewed Apr 16, 2024

View reviewed changes

lucian-ioan added 3 commits April 23, 2024 13:17

fix azfw application rule json

ba3bff3

add meaningful rule names

167ce50

use ecs fields for destination ip and port

f2eaf47

lucian-ioan requested a review from muthu-mps July 3, 2024 23:22

lucian-ioan added 2 commits July 4, 2024 02:36

cleanup ECS fields

c216dc5

improve test

071bd9d

kcreddy reviewed Jul 5, 2024

View reviewed changes

lucian-ioan added 3 commits July 5, 2024 12:43

add action, reason and question type ECS fields

a94ab77

use json properties

f4fd40b

pipeline generate

9bdd073

lucian-ioan force-pushed the update_azure_firewall_pipeline branch from aeacc79 to 9bdd073 Compare July 5, 2024 09:48

lucian-ioan added 3 commits July 5, 2024 14:00

add dns query

2c61a5e

add network transport

8c68c23

ecs fields cleanup

699a945

lucian-ioan requested a review from kcreddy July 5, 2024 12:07

kcreddy reviewed Jul 5, 2024

View reviewed changes

lucian-ioan and others added 4 commits July 5, 2024 16:11

Update packages/azure/data_stream/firewall_logs/elasticsearch/ingest_…

3df6029

…pipeline/default.yml Co-authored-by: Krishna Chaitanya Reddy Burri <krish.reddy91@gmail.com>

Update packages/azure/data_stream/firewall_logs/elasticsearch/ingest_…

ba077b0

…pipeline/default.yml Co-authored-by: Krishna Chaitanya Reddy Burri <krish.reddy91@gmail.com>

Update packages/azure/data_stream/firewall_logs/elasticsearch/ingest_…

f293933

…pipeline/default.yml Co-authored-by: Krishna Chaitanya Reddy Burri <krish.reddy91@gmail.com>

check for json category

4158fee

kcreddy approved these changes Jul 5, 2024

View reviewed changes

lucian-ioan added 2 commits July 10, 2024 14:15

Merge remote-tracking branch 'origin' into update_azure_firewall_pipe…

5099d5d

…line

merge main

53102f1

zmoog approved these changes Jul 10, 2024

View reviewed changes

add details about resource-specific mode

c313a28

update changelog

88c954c

lucian-ioan merged commit 55e5f4f into elastic:main Jul 11, 2024
4 of 5 checks passed

lucian-ioan deleted the update_azure_firewall_pipeline branch February 6, 2025 19:12

[Azure] Update Azure Firewall pipeline #9428

[Azure] Update Azure Firewall pipeline #9428

Uh oh!

Conversation

lucian-ioan commented Mar 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Screenshots

Uh oh!

elasticmachine commented Mar 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🚀 Benchmarks report

Uh oh!

muthu-mps Apr 16, 2024

Choose a reason for hiding this comment

Uh oh!

muthu-mps Apr 16, 2024

Choose a reason for hiding this comment

Uh oh!

muthu-mps Apr 16, 2024

Choose a reason for hiding this comment

Uh oh!

muthu-mps commented Apr 16, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kcreddy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lucian-ioan commented Jul 5, 2024

Uh oh!

kcreddy left a comment

Choose a reason for hiding this comment

Uh oh!

zmoog left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zmoog Jul 10, 2024

Choose a reason for hiding this comment

Uh oh!

zmoog Jul 10, 2024

Choose a reason for hiding this comment

Uh oh!

lucian-ioan commented Jul 11, 2024

Uh oh!

elasticmachine commented Jul 11, 2024

💚 Build Succeeded

History

Uh oh!

elastic-sonarqube bot commented Jul 11, 2024

Quality Gate failed

Uh oh!

Uh oh!

elasticmachine commented Jul 11, 2024

Uh oh!

Uh oh!

lucian-ioan commented Mar 22, 2024 •

edited

Loading

elasticmachine commented Mar 22, 2024 •

edited

Loading

zmoog left a comment •

edited

Loading