Skip to content

Add support for azure logs integration #272

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 15 commits into from
Nov 2, 2020
Merged

Add support for azure logs integration #272

merged 15 commits into from
Nov 2, 2020

Conversation

narph
Copy link
Contributor

@narph narph commented Sep 16, 2020

Add support for azure logs integration

image

ex log:

 {
        "_index" : ".ds-logs-azure.activitylogs-default-000001",
        "_type" : "_doc",
        "_id" : "bQlEe3UBm_qs2Y3aNZPq",
        "_score" : null,
        "_source" : {
          "agent" : {
            "hostname" : "DESKTOP-RFOOE09",
            "name" : "DESKTOP-RFOOE09",
            "id" : "c1118415-bcb7-4cf9-b64d-a6c6e8ebcfac",
            "ephemeral_id" : "938f2388-1338-4f4c-b304-b3282c4e42d8",
            "type" : "filebeat",
            "version" : "7.10.0"
          },
          "log" : {
            "level" : "Information"
          },
          "elastic_agent" : {
            "id" : "2af1ad9c-d575-4368-a14a-cb9454d222ba",
            "version" : "7.10.0",
            "snapshot" : false
          },
          "source" : {
            "geo" : {
              "continent_name" : "Europe",
              "region_iso_code" : "NL-NH",
              "city_name" : "Amsterdam",
              "country_iso_code" : "NL",
              "country_name" : "Netherlands",
              "region_name" : "North Holland",
              "location" : {
                "lon" : ..,
                "lat" : ...
              }
            },
            "as" : {
              "number" : 1136,
              "organization" : {
                "name" : "...."
              }
            },
            "ip" : "77.170.179.229"
          },
          "azure-eventhub" : {
            "sequence_number" : 643,
            "consumer_group" : "$Default",
            "offset" : 107374182400,
            "eventhub" : "insights-activity-logs",
            "enqueued_time" : "2020-11-02T08:59:38.905Z"
          },
          "tags" : [
            "forwarded"
          ],
          "geo" : {
            "continent_name" : "Europe",
            "region_iso_code" : "NL-NH",
            "city_name" : "Amsterdam",
            "country_iso_code" : "NL",
            "country_name" : "Netherlands",
            "region_name" : "North Holland",
            "location" : {
              "lon" : ....,
              "lat" : ....
            }
          },
          "cloud" : {
            "provider" : "azure"
          },
          "input" : {
            "type" : "azure-eventhub"
          },
          "@timestamp" : "2020-11-02T08:51:36.997Z",
          "ecs" : {
            "version" : "1.5.0"
          },
          "data_stream" : {
            "namespace" : "default",
            "type" : "logs",
            "dataset" : "azure.activitylogs"
          },
          "host" : {
            "name" : "DESKTOP-RFOOE09"
          },
          "event" : {
            "duration" : "0",
            "ingested" : "2020-10-30T20:47:48.123859400Z",
            "kind" : "event",
            "action" : "MICROSOFT.RESOURCES/DEPLOYMENTS/WRITE",
            "dataset" : "azure.activitylogs",
            "outcome" : "success"
          },
          "user" : {
            "full_name" : "...",
            "domain" : "...",
            "name" : "..."
          },
          "azure" : {
            "subscription_id" : "....",
            "resource" : {
              "provider" : "MICROSOFT.RESOURCES/DEPLOYMENTS",
              "name" : "NOMARKETPLACE",
              "id" : "/SUBSCRIPTIONS/..../RESOURCEGROUPS/OBS-TEST/PROVIDERS/MICROSOFT.RESOURCES/DEPLOYMENTS/NOMARKETPLACE",
              "group" : "OBS-TEST"
            },
            "correlation_id" : "876190b4-5b99-4a39-b725-4f5644911cf0",
            "activitylogs" : {
              "operation_name" : "MICROSOFT.RESOURCES/DEPLOYMENTS/WRITE",
              "result_type" : "Success",
              "identity" : {
                "authorization" : {
                  "evidence" : {
                    "role_definition_id" : "8e3af657a8ff443ca75c2fe8c4bcb635",
                    "role" : "Owner",
                    "role_assignment_scope" : "/providers/Microsoft.Management/managementGroups/5341238b-665c-4eb4-b259-b250371ae430",
                    "role_assignment_id" : "7f06f09dd6764b44930adbec3f10e92b",
                    "principal_type" : "User",
                    "principal_id" : "68b1adf93eb744b08eb8ce96522a08d3"
                  },
                  "scope" : "/subscriptions/..../resourceGroups/obs-test/providers/Microsoft.Resources/deployments/NoMarketplace",
                  "action" : "Microsoft.Resources/deployments/write"
                },
                "claims" : {
                  "xms_tcdt" : "1469565974",
                  "aio" : "ATQAy/8RAAAAsL67UQMOHZv3izTDRJfvJN5UyON9ktUszzPj08K8aURsbhxhR0niz9s1Pxm9U1lI",
                  "iss" : "https://sts.windows.net/4fa94b7d-a743-486f-abcc-6c276c44cf4b/",
                  "http://schemas_xmlsoap_org/ws/2005/05/identity/claims/nameidentifier" : "a9L2WR3XZN5ANzAqwLx_4aamU49JG6kqaE5JZkXdeNs",
                  "http://schemas_xmlsoap_org/ws/2005/05/identity/claims/surname" : "Dima",
                  "http://schemas_microsoft_com/identity/claims/scope" : "user_impersonation",
                  "http://schemas_microsoft_com/identity/claims/tenantid" : "4fa94b7d-a743-486f-abcc-6c276c44cf4b",
                  "puid" : "1003200045B17AD4",
                  "wids" : "5d6b6bb7-de71-4623-b4af-96380a352509",
                  "http://schemas_microsoft_com/claims/authnclassreference" : "1",
                  "exp" : "1604310019",
                  "ipaddr" : "77.170.179.229",
                  "iat" : "1604306119",
                  "http://schemas_microsoft_com/identity/claims/objectidentifier" : "68b1adf9-3eb7-44b0-8eb8-ce96522a08d3",
                  "http://schemas_microsoft_com/claims/authnmethodsreferences" : "pwd",
                  "ver" : "1.0",
                  "groups" : "644c6686-9ef1-4b69-9410-107664a9e1f0,9ed1993c-ce9c-4915-a04d-58c6f5f7ee12",
                  "uti" : "rqr63RW_Kk6ztuomENMQAA",
                  "http://schemas_xmlsoap_org/ws/2005/05/identity/claims/upn" : "mariana@elastic.co",
                  "aud" : "https://management.core.windows.net/",
                  "nbf" : "1604306119",
                  "appidacr" : "2",
                  "rh" : "0.AAAAfUupT0Onb0irzGwnbETPS4NAS8SwO8FJtH2XTlPL3zxRAA8.",
                  "appid" : "c44b4083-3bb0-49c1-b47d-974e53cbdf3c",
                  "http://schemas_xmlsoap_org/ws/2005/05/identity/claims/givenname" : "Mariana",
                  "http://schemas_xmlsoap_org/ws/2005/05/identity/claims/name" : "mariana@elastic.co"
                },
                "claims_initiated_by_user" : {
                  "schema" : "http://schemas.xmlsoap.org/ws/2005/05/identity/claims",
                  "surname" : "...",
                  "givenname" : "...",
                  "name" : "....",
                  "fullname" : "..."
                }
              },
              "category" : "Administrative",
              "event_category" : "Administrative",
              "result_signature" : "Succeeded.",
              "properties" : {
                "eventCategory" : "Administrative",
                "hierarchy" : "",
                "message" : "Microsoft.Resources/deployments/write",
                "entity" : "/subscriptions/..../resourceGroups/obs-test/providers/Microsoft.Resources/deployments/NoMarketplace"
              }
            }
          }
        },
        "sort" : [
          1604307096997
        ]
      },

Dashboards look good.

@narph narph self-assigned this Sep 16, 2020
@elasticmachine
Copy link

elasticmachine commented Sep 16, 2020

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #272 updated]

  • Start Time: 2020-11-02T09:15:01.220+0000

  • Duration: 15 min 3 sec

Steps errors 1

Expand to view the steps failures

  • Name: Checks and builds integration sources
    • Description: mage -debug check

    • Duration: 1 min 31 sec

    • Start Time: 2020-11-02T09:28:04.434+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-11-02T09:28:35.769Z] Format the package
[2020-11-02T09:28:35.769Z] Done
[2020-11-02T09:28:35.769Z] packages/rabbitmq:
[2020-11-02T09:28:35.769Z] elastic-package format
[2020-11-02T09:28:35.769Z] Format the package
[2020-11-02T09:28:35.769Z] Done
[2020-11-02T09:28:35.769Z] packages/redis:
[2020-11-02T09:28:35.769Z] elastic-package format
[2020-11-02T09:28:35.769Z] Format the package
[2020-11-02T09:28:35.769Z] Done
[2020-11-02T09:28:35.769Z] packages/suricata:
[2020-11-02T09:28:35.769Z] elastic-package format
[2020-11-02T09:28:35.769Z] Format the package
[2020-11-02T09:28:35.769Z] Done
[2020-11-02T09:28:35.769Z] packages/system:
[2020-11-02T09:28:35.769Z] elastic-package format
[2020-11-02T09:28:35.769Z] Format the package
[2020-11-02T09:28:35.769Z] Done
[2020-11-02T09:28:35.769Z] packages/windows:
[2020-11-02T09:28:35.769Z] elastic-package format
[2020-11-02T09:28:35.769Z] Format the package
[2020-11-02T09:28:35.769Z] Done
[2020-11-02T09:28:35.769Z] packages/zeek:
[2020-11-02T09:28:35.769Z] elastic-package format
[2020-11-02T09:28:35.769Z] Format the package
[2020-11-02T09:28:35.769Z] Done
[2020-11-02T09:28:35.769Z] packages/zookeeper:
[2020-11-02T09:28:35.769Z] elastic-package format
[2020-11-02T09:28:35.769Z] Format the package
[2020-11-02T09:28:35.769Z] Done
[2020-11-02T09:28:35.769Z] packages/zoom:
[2020-11-02T09:28:35.769Z] elastic-package format
[2020-11-02T09:28:35.769Z] Format the package
[2020-11-02T09:28:35.769Z] Done
[2020-11-02T09:28:35.769Z] packages/apache:
[2020-11-02T09:28:35.769Z] elastic-package lint
[2020-11-02T09:28:35.769Z] Lint the package
[2020-11-02T09:28:35.769Z] Done
[2020-11-02T09:28:35.769Z] packages/aws:
[2020-11-02T09:28:35.769Z] elastic-package lint
[2020-11-02T09:28:35.769Z] Lint the package
[2020-11-02T09:28:35.769Z] Done
[2020-11-02T09:28:35.769Z] packages/azure:
[2020-11-02T09:28:35.769Z] elastic-package lint
[2020-11-02T09:28:35.769Z] Lint the package
[2020-11-02T09:28:35.769Z] Error: linting package failed: found 3 validation errors:
[2020-11-02T09:28:35.769Z]    1. item [0f559cc0-f0d5-11e9-90ec-112a988266d5.json] is not allowed in folder [/var/lib/jenkins/workspace/Beats_integrations_PR-272/src/github.com/elastic/integrations/packages/azure/kibana/dashboard]
[2020-11-02T09:28:35.769Z]    2. item [41e84340-ec20-11e9-90ec-112a988266d5.json] is not allowed in folder [/var/lib/jenkins/workspace/Beats_integrations_PR-272/src/github.com/elastic/integrations/packages/azure/kibana/dashboard]
[2020-11-02T09:28:35.769Z]    3. item [87095750-f05a-11e9-90ec-112a988266d5.json] is not allowed in folder [/var/lib/jenkins/workspace/Beats_integrations_PR-272/src/github.com/elastic/integrations/packages/azure/kibana/dashboard]
[2020-11-02T09:28:35.769Z] 
[2020-11-02T09:28:35.769Z] Error: running elastic-package failed: running "/var/lib/jenkins/workspace/Beats_integrations_PR-272/src/github.com/elastic/integrations/build/elastic-package lint" failed with exit code 1
[2020-11-02T09:28:36.017Z] Post stage
[2020-11-02T09:28:36.071Z] Archiving artifacts
[2020-11-02T09:28:36.120Z] Recording test results
[2020-11-02T09:28:36.530Z] No test report files were found. Configuration error?
[2020-11-02T09:28:36.594Z] Error when executing always post condition:
[2020-11-02T09:28:36.604Z] Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from beats-ci-immutable-ubuntu-1604-1604308569035255058.c.elastic-ci-prod.internal/10.224.0.130:44098
[2020-11-02T09:28:36.604Z] 		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1800)
[2020-11-02T09:28:36.604Z] 		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
[2020-11-02T09:28:36.604Z] 		at hudson.remoting.Channel.call(Channel.java:1001)
[2020-11-02T09:28:36.604Z] 		at hudson.FilePath.act(FilePath.java:1070)
[2020-11-02T09:28:36.604Z] 		at hudson.FilePath.act(FilePath.java:1059)
[2020-11-02T09:28:36.604Z] 		at hudson.tasks.junit.JUnitParser.parseResult(JUnitParser.java:111)
[2020-11-02T09:28:36.604Z] 		at hudson.tasks.junit.JUnitResultArchiver.parse(JUnitResultArchiver.java:137)
[2020-11-02T09:28:36.604Z] 		at hudson.tasks.junit.JUnitResultArchiver.parseAndAttach(JUnitResultArchiver.java:167)
[2020-11-02T09:28:36.604Z] 		at hudson.tasks.junit.pipeline.JUnitResultsStepExecution.run(JUnitResultsStepExecution.java:52)
[2020-11-02T09:28:36.604Z] 		at hudson.tasks.junit.pipeline.JUnitResultsStepExecution.run(JUnitResultsStepExecution.java:25)
[2020-11-02T09:28:36.604Z] 		at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
[2020-11-02T09:28:36.604Z] 		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[2020-11-02T09:28:36.604Z] 		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2020-11-02T09:28:36.604Z] 		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2020-11-02T09:28:36.604Z] 		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2020-11-02T09:28:36.604Z] hudson.AbortException: No test report files were found. Configuration error?
[2020-11-02T09:28:36.604Z] 	at hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:151)
[2020-11-02T09:28:36.604Z] 	at hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:115)
[2020-11-02T09:28:36.604Z] 	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3112)
[2020-11-02T09:28:36.604Z] 	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
[2020-11-02T09:28:36.604Z] 	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
[2020-11-02T09:28:36.604Z] 	at hudson.remoting.Request$2.run(Request.java:369)
[2020-11-02T09:28:36.604Z] 	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
[2020-11-02T09:28:36.604Z] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2020-11-02T09:28:36.604Z] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2020-11-02T09:28:36.604Z] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2020-11-02T09:28:36.604Z] 	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
[2020-11-02T09:28:36.604Z] 	at java.lang.Thread.run(Thread.java:748)
[2020-11-02T09:28:36.604Z] 
[2020-11-02T09:28:36.772Z] Stage "Update Package Storage" skipped due to earlier failure(s)
[2020-11-02T09:28:57.831Z] Still waiting to schedule task
[2020-11-02T09:28:57.831Z] Waiting for next available executor on ‘master||metal||linux’
[2020-11-02T09:30:03.566Z] Running on worker-1244230 in /var/lib/jenkins/workspace/Beats_integrations_PR-272
[2020-11-02T09:30:03.938Z] [INFO] getVaultSecret: Getting secrets
[2020-11-02T09:30:04.049Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-11-02T09:30:05.931Z] + chmod 755 generate-build-data.sh
[2020-11-02T09:30:05.931Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/integrations/PR-272/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/integrations/PR-272/runs/32 FAILURE 903307
[2020-11-02T09:30:05.932Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/integrations/PR-272/runs/32/steps/?limit=10000 -o steps-info.json
[2020-11-02T09:30:08.185Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/integrations/PR-272/runs/32/tests/?status=FAILED -o tests-errors.json
[2020-11-02T09:30:08.185Z] Retry 1/3 exited 22, retrying in 1 seconds...
[2020-11-02T09:30:10.446Z] Retry 2/3 exited 22, retrying in 2 seconds...
[2020-11-02T09:30:12.700Z] Retry 3/3 exited 22, no more retries left.
[2020-11-02T09:30:12.700Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/integrations/PR-272/runs/32/log/ -o pipeline-log.txt

@andresrc andresrc added the Team:Integrations Label for the Integrations team label Sep 17, 2020
@mtojek mtojek self-requested a review October 2, 2020 06:30
Copy link
Contributor

@mtojek mtojek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please adjust the PR accordingly to changes introduced in #280 .

Main changes:

  1. Rebase against master.
  2. Rename "dataset" folder to "data_stream".
  3. Rename "config_templates" to "policy_templates" in package manifest file.Please adjust the PR accordingly to changes introduced in #280 .

@narph narph changed the title Add support for azure integration Add support for azure logs integration Oct 30, 2020
@narph narph marked this pull request as ready for review October 30, 2020 15:59
@elasticmachine
Copy link

Pinging @elastic/integrations (Team:Integrations)

@andresrc
Copy link
Contributor

Related: #357

@narph we need to confirm the name (/cc @sorantis)

@@ -0,0 +1,45 @@
---
description: Pipeline for parsing azure activity logs.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see this file is reused across all datasets, but because of implementation it's not "shared" anymore, right? I wonder what would be the best way to handle this

Copy link
Contributor

@ycombinator ycombinator Nov 2, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can use YAML references or something that are resolved at either package build time (so the file is duplicated) or in Kibana at runtime when the package is consumed. Or maybe a symlink might work too, although that might be problematic for Windows package developers?

Let's move this discussion to the package-spec repo so we don't hold up this PR? Depending on the outcome we can come back and update all packages with shared assets like this one.

@sorantis
Copy link

sorantis commented Nov 2, 2020

Related: #357

@narph we need to confirm the name (/cc @sorantis)

Since this package will accommodate both logs ( including AD logs) and metrics I suggest calling it azure.

Any additional packages that might come up later (e.g. for billing, app insights) can be more specific, like azure-billing, azure-app-insights

@narph narph requested review from exekias and mtojek November 2, 2020 13:46
Copy link

@exekias exekias left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@narph narph merged commit d7c4961 into elastic:master Nov 2, 2020
@narph narph deleted the azure-package branch November 2, 2020 15:10
@andrewkroh andrewkroh added Integration:azure Azure Logs New Integration Issue or pull request for creating a new integration package. labels Aug 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Integration:azure Azure Logs New Integration Issue or pull request for creating a new integration package. Team:Integrations Label for the Integrations team [zube]: In Progress
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants