[aws] [cloudwatch_metrics] Map aws.dimensions field as object (backport of #11883)

[zmoog]

Proposed commit message

Change the mapping type for the aws.dimensions field from flattened to object.

Currently, all *_metrics data streams but one use the object mapping. The cloudwatch_metrics data stream uses the flattened type instead.

We need to unify the mapping of aws.dimensions across all metrics-related data streams in the AWS integration.
If all data streams use the exact mapping for aws.dimensions, users will be able to query and build a dashboard that correlates data across different data streams.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• Run the test scenario on 8.15.4

Related issues

• relates Bring uniformity in aws.dimensions.* fields mapping #11806
• backports [aws] [cloudwatch_metrics] Map aws.dimensions field as object #11883

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[zmoog]

I tested the upgrade from AWS integration 2.30.2 to 2.30.3 (the unreleased changes from this PR) with the following steps:

• Started a brand new local stack (8.15.4)
• Installed AWS integration 2.30.2 (with aws.dimensions as flattened)
• Started sending 1 document every 5 secs, including a field containing a sequence number
• Upgraded the AWS integration to 2.30.3 (unreleased, this PR)
• Waited for the rollout to take effect (checking settings.index.time_series | .start_time, .end_time)
• Checked that the the data stream didn't lose any sequence number

More details on selected steps.

Started sending 1 document every 5 secs

• create an API key on the local stack
• export env vars for the es tool.

Set up the es tool config:

export ELASTICSEARCH_ENDPOINTS="https://localhost:9200"
export ELASTICSEARCH_API_KEY="[redacted, event if it is not need since it's local]"

I used the following shell script:

sequence=0
while true
do
cat > metrics.json <<EOF
{
  "@timestamp": "$(date '+%Y-%m-%dT%H:%M:%S%z')",
    "aws": {
      "dimensions": {
        "name": "Maurizio Branca",
        "AutoScalingGroupName": "whatever"
      },
      "metric": {
        "cpu": 10,
        "sequence": $sequence
      }
  }
}  
EOF

((sequence++))

cat metrics.json | jq -c | es docs bulk -f - -i metrics-aws.cloudwatch_metrics-sdh5390
sleep 5
done

Results in:

2025/01/07 12:54:11 adding a new document: {"@timestamp":"2025-01-07T12:54:11+0100","aws":{"dimensions":{"name":"Maurizio Branca","AutoScalingGroupName":"whatever"},"metric":{"cpu":10,"sequence":0}}}
2025/01/07 12:54:11 closing bulk indexer
2025/01/07 12:54:11 Successfully indexed document 
2025/01/07 12:54:11 bulk indexer closed
2025/01/07 12:54:11 getting bulk indexer stats
2025/01/07 12:54:11 Stats: {NumAdded:1 NumFlushed:1 NumFailed:0 NumIndexed:0 NumCreated:1 NumUpdated:0 NumDeleted:0 NumRequests:1}

The scripts sends a document like the following every 5 secs:

{
  "@timestamp": "2024-12-31T00:14:58+0100",
  "aws": {
    "dimensions": {
      "name": "Maurizio Branca",
      "AutoScalingGroupName": "whatever"
    },
    "metric": {
      "cpu": 10,
      "sequence": 270
    }
  }
}

Upgraded the AWS integration to 2.30.3 (unreleased, this PR)

Upgrade the AWS integration package from 2.30.2 to 2.30.3.

Waited for the rollout to take effect

Right after the upgrade, Fleet/ES creates a new -000002 index, but keeps writing to the -000001 index until the settings.index.time_series.end_time elapses.

Old index -000001:

// GET metrics-aws.cloudwatch_metrics-sdh5390/_settings
{
  ".ds-metrics-aws.cloudwatch_metrics-sdh5390-2025.01.07-000001": {
    "settings": {
      "index": {
        "mapping": {
          "total_fields": {
            "limit": "1000",
            "ignore_dynamic_beyond_limit": "true"
          }
        },
        "hidden": "true",
        "time_series": {
          "end_time": "2025-01-07T12:33:16.000Z",
          "start_time": "2025-01-07T09:54:11.000Z"
        },

New index -000002:

// GET metrics-aws.cloudwatch_metrics-sdh5390/_settings
{
  ".ds-metrics-aws.cloudwatch_metrics-sdh5390-2025.01.07-000002": {
    "settings": {
      "index": {
        "mapping": {
          "total_fields": {
            "limit": "1000",
            "ignore_dynamic_beyond_limit": "true"
          }
        },
        "hidden": "true",
        "time_series": {
          "end_time": "2025-01-07T13:03:16.000Z",
          "start_time": "2025-01-07T12:33:16.000Z"
        },

Now I need to wait until 2025-01-07T12:33:16.000Z (UTC) to see if the ES smoothly transitions from index -000001 to -000002.

Checked that the the data stream didn't lose any sequence number

At 2025-01-07T12:33:16.000Z, ES successfully transitioned from index -000001 to -000002 and from flattened to object field mapping.

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
20.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 235d140

History

• 💔 Build #20092 failed 10f269f
• 💚 Build #20076 succeeded 63ae257

cc @zmoog

[muthu-mps]

LGTM!

[elastic-vault-github-plugin-prod]

Package aws - 2.30.3 containing this change is available at https://epr.elastic.co/package/aws/2.30.3/