[apm-data] Set event.dataset if empty for logs #129074

carsonip · 2025-06-06T16:23:50Z

For APM logs, set event.dataset to data_stream.dataset if event.dataset is empty, to satisfy Anomaly Detection's requirement to have event.dataset in every logs-* data stream.

Fixes elastic/apm-server#16397

elasticsearchmachine · 2025-06-09T16:38:10Z

Pinging @elastic/es-data-management (Team:Data Management)

elasticsearchmachine · 2025-06-09T16:39:43Z

Hi @carsonip, I've created a changelog YAML for you.

carsonip · 2025-06-09T16:41:53Z

Testing notes

Tested with adding the processor to default pipeline.

DELETE _data_stream/logs-apm.app.foo-bar

POST /logs-apm.app.foo-bar/_doc
{
  "@timestamp": 1,
  "data_stream": {
    "type": "logs",
    "dataset": "apm.app.foo",
    "namespace": "bar"
  }
}

POST /logs-apm.app.foo-bar/_doc
{
  "@timestamp": 1,
  "data_stream": {
    "type": "logs",
    "dataset": "apm.app.foo",
    "namespace": "bar"
  },
  "event": {
    "dataset": "ds"
  }
}

GET /logs-apm.app.foo-bar/_search

Output:

{
  "took": 1,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 2,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": ".ds-logs-apm.app.foo-bar-2025.06.09-000001",
        "_id": "_WCQVZcBbmWL1cSKlTSi",
        "_score": 1,
        "_source": {
          "@timestamp": 1,
          "data_stream": {
            "namespace": "bar",
            "type": "logs",
            "dataset": "apm.app.foo"
          },
          "event": {
            "ingested": "2025-06-09T16:40:28Z",
            "dataset": "apm.app.foo"
          }
        }
      },
      {
        "_index": ".ds-logs-apm.app.foo-bar-2025.06.09-000001",
        "_id": "_mCQVZcBbmWL1cSKoTRm",
        "_score": 1,
        "_source": {
          "@timestamp": 1,
          "data_stream": {
            "namespace": "bar",
            "type": "logs",
            "dataset": "apm.app.foo"
          },
          "event": {
            "ingested": "2025-06-09T16:40:31Z",
            "dataset": "ds"
          }
        }
      }
    ]
  }
}

axw

Looks good, just a question about sending docs to logs-apm.* without explicitly setting data_stream.dataset. Not sure if that's a thing we need to cater for.

axw · 2025-06-12T00:07:39Z

x-pack/plugin/apm-data/src/main/resources/ingest-pipelines/logs-apm@pipeline.yaml

+- set:
+    if: "ctx.data_stream?.dataset != null"
+    field: event.dataset
+    value: "{{{data_stream.dataset}}}"


Nit: should we use copy_from and ignore_empty_value, and remove the if and value?
(See https://www.elastic.co/docs/reference/enrich-processor/set-processor)

Will data_stream.dataset be automatically set for the logs-apm.* data stream documents, if they're not specified in _source? If that's the case, I'm not sure they will be in ctx.*, and we may need to have a fallback.

Can you add test cases for that?

Will data_stream.dataset be automatically set for the logs-apm.* data stream documents, if they're not specified in _source?

No, at least not visible to the log ingest pipeline.

If that's the case, I'm not sure they will be in ctx.*

Correct, they will not be in ctx.

we may need to have a fallback.

Do you mean a fallback to a non-null event.dataset in the case where both data_stream.dataset and event.dataset is missing? I'm inclined to not perform any special case handling to make things up because it sounds more like a misconfiguration issue. Unless the user explicitly remove the DS fields in an intermediate ingest pipeline, DS fields should always be present in docs from apm-server (even if reroute processor is used).

In any case, I've added test in 957d9ef to confirm this behavior.

Do you mean a fallback to a non-null event.dataset in the case where both data_stream.dataset and event.dataset is missing? I'm inclined to not perform any special case handling to make things up because it sounds more like a misconfiguration issue.

I'm OK with this. I don't think it's a real thing we need to handle, even if it's technically possible.

dakrone · 2025-06-16T15:31:12Z

Is there a reason why you switched away from apm@pipeline for the final pipeline? When I look at this it appears that these logs will no longer be doing the processing in that file, instead only doing the event.dataset configuration from the new final pipeline.

Can this be considered a breaking change in behavior?

carsonip · 2025-06-16T15:37:32Z

Is there a reason why you switched away from apm@pipeline for the final pipeline? When I look at this it appears that these logs will no longer be doing the processing in that file, instead only doing the event.dataset configuration from the new final pipeline.

The new logs-apm@pipeline in turn calls apm@pipeline, just like the other final pipelines e.g. metrics-apm@pipeline and traces-apm@pipeline. It is needed just because we needed log-specific processor in the final pipeline and we cannot just add it to apm@pipeline directly.

dakrone · 2025-06-16T15:42:34Z

Ahh okay, that makes sense. I missed the call out to the original apm@pipeline (I am not used to reading pipelines in yaml…). Thanks for the explanation.

axw

LGTM, thanks, and sorry for the delay

elasticsearchmachine · 2025-06-17T09:37:07Z

💔 Backport failed

Status	Branch	Result
❌	8.19	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 129074

carsonip · 2025-06-17T09:53:37Z

💚 All backports created successfully

Status	Branch	Result
✅	8.19

Questions ?

Please refer to the Backport tool documentation

For APM logs, set event.dataset to data_stream.dataset if event.dataset is empty, to satisfy Anomaly Detection's requirement to have event.dataset in every logs-* data stream. (cherry picked from commit 466afba) # Conflicts: # test/framework/src/main/java/org/elasticsearch/test/rest/ESRestTestCase.java

rubvs · 2025-06-23T17:55:16Z

Tested manually with steps outlined in #129074 (comment)

With version 9.0.3

    "hits": [
      {
        "_index": ".ds-logs-apm.app.foo-bar-2025.06.23-000001",
        "_id": "JNXnnZcBQ3xVa0c-8bvn",
        "_score": 1,
        "_source": {
          "@timestamp": 1,
          "data_stream": {
            "namespace": "bar",
            "type": "logs",
            "dataset": "apm.app.foo"
          },
          "tags": [
            "_geoip_database_unavailable_GeoLite2-City.mmdb"
          ]
        }
      },
      {
        "_index": ".ds-logs-apm.app.foo-bar-2025.06.23-000001",
        "_id": "xU_nnZcBCJmh9scm-w5_",
        "_score": 1,
        "_source": {
          "@timestamp": 1,
          "data_stream": {
            "namespace": "bar",
            "type": "logs",
            "dataset": "apm.app.foo"
          },
          "event": {
            "dataset": "ds"
          },
          "tags": [
            "_geoip_database_unavailable_GeoLite2-City.mmdb"
          ]
        }
      }
    ]

With version 9.1.0

    "hits": [
      {
        "_index": ".ds-logs-apm.app.foo-bar-2025.06.23-000001",
        "_id": "hFznnZcBHYim8ksPeDdl",
        "_score": 1,
        "_source": {
          "@timestamp": 1,
          "data_stream": {
            "namespace": "bar",
            "type": "logs",
            "dataset": "apm.app.foo"
          },
          "event": {
            "dataset": "apm.app.foo"
          },
          "tags": [
            "_geoip_database_unavailable_GeoLite2-City.mmdb"
          ]
        }
      },
      {
        "_index": ".ds-logs-apm.app.foo-bar-2025.06.23-000001",
        "_id": "hVznnZcBHYim8ksPjTfk",
        "_score": 1,
        "_source": {
          "@timestamp": 1,
          "data_stream": {
            "namespace": "bar",
            "type": "logs",
            "dataset": "apm.app.foo"
          },
          "event": {
            "dataset": "ds"
          },
          "tags": [
            "_geoip_database_unavailable_GeoLite2-City.mmdb"
          ]
        }
      }
    ]

By inspecting the outputs above, version 9.0.3 does not include the event.dataset field, while 9.1.0 does include it. Concluding that the data_stream.dataset field is copied properly.

"event": {
    "dataset": "apm.app.foo"
},

elasticsearchmachine added the v9.1.0 label Jun 6, 2025

[apm-data] Set event.dataset if empty for logs

71b8459

carsonip force-pushed the apm-logs-event-dataset branch from 011910c to 71b8459 Compare June 6, 2025 16:29

carsonip added v8.19.0 auto-backport Automatically create backport pull requests when merged labels Jun 6, 2025

carsonip marked this pull request as ready for review June 9, 2025 16:17

carsonip requested a review from a team as a code owner June 9, 2025 16:17

Merge branch 'main' into apm-logs-event-dataset

f63e462

elasticsearchmachine added the needs:triage Requires assignment of a team area label label Jun 9, 2025

carsonip added the :Data Management/Data streams Data streams and their lifecycles label Jun 9, 2025

elasticsearchmachine added the Team:Data Management Meta label for data/management team label Jun 9, 2025

elasticsearchmachine removed the needs:triage Requires assignment of a team area label label Jun 9, 2025

carsonip added needs:triage Requires assignment of a team area label >bug and removed needs:triage Requires assignment of a team area label labels Jun 9, 2025

Update docs/changelog/129074.yaml

0923bcd

isaacaflores2 approved these changes Jun 9, 2025

View reviewed changes

1pkg approved these changes Jun 10, 2025

View reviewed changes

Introduce logs-apm@pipeline

7add32d

carsonip force-pushed the apm-logs-event-dataset branch from 3a8135a to 7add32d Compare June 11, 2025 13:17

carsonip added 6 commits June 11, 2025 16:03

Add test

48b569b

Fix test

4bab03d

Merge branch 'main' into apm-logs-event-dataset

309d0df

Bump version after rebase

b584c7f

Fix typo in test

57fe0ea

Fix tests again

a0ec35a

axw reviewed Jun 12, 2025

View reviewed changes

carsonip added 2 commits June 12, 2025 10:04

Use copy_from and ignore_empty_value

a4c1f16

Add tests for missing data_stream.dataset

957d9ef

carsonip requested a review from axw June 12, 2025 09:21

carsonip added 2 commits June 12, 2025 18:38

Merge branch 'main' into apm-logs-event-dataset

6be45d2

Merge branch 'main' into apm-logs-event-dataset

4099dbe

carsonip requested review from 1pkg and isaacaflores2 June 16, 2025 08:42

carsonip added 2 commits June 16, 2025 13:39

Merge branch 'main' into apm-logs-event-dataset

ce4c9bd

Bump version

74f6949

carsonip requested a review from simitt June 16, 2025 16:33

axw approved these changes Jun 17, 2025

View reviewed changes

carsonip merged commit 466afba into elastic:main Jun 17, 2025
18 checks passed

elasticsearchmachine added the backport pending label Jun 17, 2025

carsonip mentioned this pull request Jun 17, 2025

[8.19] [apm-data] Set event.dataset if empty for logs (#129074) #129537

Merged

rubvs mentioned this pull request Jun 19, 2025

9.1 Test Plan elastic/apm-server#17263

Closed

17 tasks

[apm-data] Set event.dataset if empty for logs #129074

[apm-data] Set event.dataset if empty for logs #129074

Uh oh!

Conversation

carsonip commented Jun 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Jun 9, 2025

Uh oh!

elasticsearchmachine commented Jun 9, 2025

Uh oh!

carsonip commented Jun 9, 2025

Testing notes

Uh oh!

axw left a comment

Choose a reason for hiding this comment

Uh oh!

axw Jun 12, 2025

Choose a reason for hiding this comment

Uh oh!

axw Jun 12, 2025

Choose a reason for hiding this comment

Uh oh!

carsonip Jun 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

axw Jun 17, 2025

Choose a reason for hiding this comment

Uh oh!

dakrone commented Jun 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

carsonip commented Jun 16, 2025

Uh oh!

dakrone commented Jun 16, 2025

Uh oh!

axw left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elasticsearchmachine commented Jun 17, 2025

💔 Backport failed

Uh oh!

carsonip commented Jun 17, 2025

💚 All backports created successfully

Questions ?

Uh oh!

rubvs commented Jun 23, 2025

Uh oh!

Uh oh!

carsonip commented Jun 6, 2025 •

edited

Loading

carsonip Jun 12, 2025 •

edited

Loading

dakrone commented Jun 16, 2025 •

edited

Loading