[CI] StandardVersusLogsIndexModeRandomDataChallengeRestIT testEsqlSource failing

[elasticsearchmachine]

Build Scans:

• elasticsearch-periodic #6006 / single-processor-node-tests
• elasticsearch-periodic-platform-support #5995 / debian-11_platform-support-unix
• elasticsearch-periodic-platform-support #5995 / ubuntu-2204_platform-support-unix
• elasticsearch-periodic-platform-support #5995 / amazonlinux-2_platform-support-aws
• elasticsearch-periodic #6006 / openjdk23_checkpart2_java-matrix
• elasticsearch-periodic #6006 / graalvm-ce17_checkpart2_java-matrix

Reproduction Line:

./gradlew ":x-pack:plugin:logsdb:javaRestTest" --tests "org.elasticsearch.xpack.logsdb.qa.StandardVersusLogsIndexModeRandomDataChallengeRestIT.testEsqlSource" -Dtests.seed=6C57DD0486AE54B5 -Dtests.locale=en-PK -Dtests.timezone=Etc/GMT -Druntime.java=23

Applicable branches:
8.x

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: Source matching failed at document id [0]. Source documents don't match for field [yjOTNlEeY.MdaxkeBZK.RNUrtXs]: Error [Lists do not match when ignoring sort order, actual: [1.0, 0.5572748, 0.27104914, 0.13549805, 0.101490855, 0.24916041, 0.8397343, 0.84317076, 0.32944965, 0.8132014], expected: [0.5572748, 0.27104914, 0.13552392, 0.101490855, 0.24916041, 0.9998697, 0.8397343, 0.84317076, 0.32944965, 0.8132014]] actual mappings [{"_doc":{"properties":{"AcX":{"doc_values":true,"index":false,"scaling_factor":100.5,"store":false,"type":"scaled_float"},"VKIboihC":{"doc_values":false,"index":true,"store":true,"synthetic_source_keep":"none","type":"unsigned_long"},"ZwiGr":{"doc_values":false,"index":true,"store":true,"synthetic_source_keep":"all","type":"long"},"host.name":{"type":"keyword"},"ib":{"doc_values":false,"index":true,"store":true,"type":"half_float"},"kUTreamGb":{"doc_values":true,"index":false,"store":false,"type":"integer"},"memory_usage_bytes":{"type":
[truncated]

Issue Reasons:

• [8.x] 6 failures in test testEsqlSource (3.4% fail rate in 177 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

[elasticsearchmachine]

This has been muted on branch main

Mute Reasons:

• [main] 4 failures in test testEsqlSource (1.8% fail rate in 228 executions)
• [main] 3 failures in step part-2 (2.8% fail rate in 106 executions)
• [main] 3 failures in pipeline elasticsearch-pull-request (2.8% fail rate in 107 executions)

Build Scans:

• elasticsearch-intake #16376 / part2
• elasticsearch-pull-request #52527 / part-2
• elasticsearch-pull-request #52463 / part-2
• elasticsearch-pull-request #52430 / part-2

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)

[lkts]

Looks like values that were copied into the field using copy_to are present in synthetic source.

[lkts]

I believe that the code below does not work properly for fields inside nested objects. This code inserts a special marker to not render values of the field that came from copy_to into synthetic source. context.doc() in this case is not necessarily the correct nested document.

elasticsearch/server/src/main/java/org/elasticsearch/index/mapper/IgnoredSourceFieldMapper.java

Line 185 in 7d7a9d9

mutableList.add(new IgnoredSourceFieldMapper.NameValue(copyToField, offset, XContentDataHelper.voidValue(), context.doc()));

In the repro the field is y.bXv.FJQLWM and y is a nested field that contains an array in document in question.

[lkts]

Minimized:

---
synthetic_source with nested objects:
  - do:
      indices.create:
        index: test
        body:
          settings:
            index:
              mapping.source.mode: synthetic
          mappings:
            properties:
              name:
                type: keyword
              my_values:
                type: nested
                properties:
                  k:
                    type: keyword
                    copy_to: my_values.copy
                  copy:
                    type: keyword
              dummy:
                type: keyword

  - do:
      index:
        index: test
        id: 1
        refresh: true
        body:
          name: "A"
          my_values:
            k: "hello"

  - do:
      search:
        index: test
        sort: name

  - match:
      hits.hits.0._source:
        name: "A"
        my_values:
          k: "hello"

Notice that copy field is present even though it was not in the original document.

    java.lang.AssertionError: Failure at [indices.create/20_synthetic_source:1644]: Expected a map containing
    my_values: a map containing
              k: "hello"
           copy: <unexpected> but was "hello"