Add cluster UUID to the Logstash node datastream.

[mashhurs]

Proposed commit message

Logstash monitoring with metricbeat and self include cluster_uuid which will be used during the queries (example telemetry collection). However, with agent driven monitoring we are losing the cluster UUID. This change fixes the explained bug.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

• [ ]

How to test this PR locally

Use the Logstash integration, install the agent and copy the change script into the policy. We can see the cluster UUID will be reflected in .ds-metrics-logstash.node* indices. Also query against cluster_uuid (which is an alias for logstash.elasticsearch.cluster.id) also works as expected.

// example query
GET .ds-metrics-logstash.node*/_search
{
  "query": {
    "bool": {
      "filter": [
        {
          "terms": {
            "cluster_uuid": [
              "wjK4bDWFQxKggvldnpKeng"
            ]
          }
        },
        {
          "bool": {
            "should": [
              {
                "term": {
                  "type": "logstash_stats"
                }
              },
              {
                "term": {
                  "metricset.name": "node_stats"
                }
              },
              {
                "term": {
                  "data_stream.dataset": "logstash.node"
                }
              }
            ]
          }
        },
        {
          "range": {
            "@timestamp": {
              "format": "epoch_millis",
              "gte": 0,
              "lte": 2816489716793
            }
          }
        }
      ]
    }
  },
  "collapse": {
    "field": "host.id"
  },
  "sort": [
    {
      "@timestamp": {
        "order": "desc",
        "unmapped_type": "long"
      }
    }
  ],
  "from": 0,
  "size": 10000
}

Related issues

Screenshots

[klacabane]

I've started a stack and the logstash service defined in the _dev/deploy directory. after configuring the integration to collect metrics from the previously started logstash I can't see the cluster_uuid populated in the logstash.node documents. If I'm interpreting the code correctly we should extract that property from /_node/stats?graph=true vertices but I don't see it when I hit the endpoint manually.
Is there any configuration missing (maybe in logstash.yml) for this property to be output in the /_node/stats response ?

[mashhurs]

I've started a stack and the logstash service defined in the _dev/deploy directory. after configuring the integration to collect metrics from the previously started logstash I can't see the cluster_uuid populated in the logstash.node documents. If I'm interpreting the code correctly we should extract that property from /_node/stats?graph=true vertices but I don't see it when I hit the endpoint manually. Is there any configuration missing (maybe in logstash.yml) for this property to be output in the /_node/stats response ?

Thanks @klacabane !
I failed to edit the URL 🤦 now it is correctly reflected.

[elasticmachine]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 3275646

History

• 💔 Build #11894 failed b46086b
• 💔 Build #11855 failed 87952dc
• 💔 Build #11851 failed 46b41aa

cc @mashhurs

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

[robbavey]

LGTM

[klacabane]

LGTM

[elasticmachine]

Package logstash - 2.4.7 containing this change is available at https://epr.elastic.co/search?package=logstash