Add support for retrieving semantic_text's indexed chunks via fields API

[jimczi]

Introduces the "format": "chunks" option for the fields parameter in _search requests. Allows users to retrieve the original text chunks generated by a semantic field’s chunking strategy.

Example usage:

POST test-index/_search
{
  "query": {
    "ids" : {
      "values" : ["1"]
    }
  },
  "fields": [
    {
      "field": "semantic_text_field",
      "format": "chunks"      <1>
    }
  ]
}

[elasticsearchmachine]

Pinging @elastic/es-search-relevance (Team:Search Relevance)

[elasticsearchmachine]

Hi @jimczi, I've created a changelog YAML for you.

[github-actions]

🔍 Preview links for changed docs

• docs/reference/elasticsearch/mapping-reference/semantic-text.md

[kderusso]

I don't think we want to completely remove this, as it is still the only way for legacy instances to retrieve this information. Maybe we can just frame it as an alternate approach?

[jimczi]

This is only in 9.1 right? I also wonder how usable it is, why would you do a match_all query to retrieve chunks? From which doc? We should at least use the highlight_query to make the example meaningful.
I also don't like to document multiple ways to do the same thing, this match_all hack can be replaced by setting the number of fragments to something big. That should be enough for users until this option is available?

[kderusso]

The point was when people want to see all of their chunks without filtering or sorting. It's a valid use case as highlighting is the only way to interact with chunks for older versions. The fix was backported all the way to 8.18.

[jimczi]

But how is it different from setting the number of fragments to a large value? This is just redundant imo and the example is not even usable since match_all would return random documents?

[kderusso]

OK, then if you don't want to call out match_all maybe we show other ways to get the same information, using the highlighter, that you would prefer? I think we still need to keep this information in the docs for users < 9.2, and we added specifically about match_all because it seemed to be the easiest generic use case. If you would prefer to elaborate on things like increasing num fragments or test that this works with an ids query (have not tested this) then that's fine, but I just want to make sure we're not taking information out of our documentation that people on legacy versions need to better understand.

[jimczi]

I pushed a change that restore the doc for the match_all path in 96cbf4f. I changed a bit the wording since my main concern is that highlighting returns fragments and not chunks. I don't want this behaviour to block us when we add the support to merge/reduce chunks based on a target number of tokens.

[kderusso]

Thanks, could we please just add a yaml test that ensures the syntax you suggest is correct? After that I will be good. Thank you!