Simplified RRF Retriever #129659

Mikep86 · 2025-06-18T19:41:54Z

Adds a simplified syntax for the rrf retriever:

GET my-index/_search
{
  "retriever": {
    "rrf": {
      "fields": ["field_1", "field_2"],
      "query": "my awesome query"
    }
  }
}

fields is optional. If it is not provided, we query the fields defined by the index.query.default_field index setting (which is * by default).

This syntax automatically handles querying a mix of lexical fields (i.e. fields that support lexical search via match) and semantic_text fields. The fields are divided into lexical and semantic groups to create a 50/50 weight distribution between the two in the final ranking. This is achieved by creating a retriever tree that looks like:

rrf
   multi_match on lexical fields
   rrf
     match on semantic_text field A
     match on semantic_text field B
     match on semantic_text field C

This is a sibling of the simplified linear retriever, which was added in #129200.

elasticsearchmachine · 2025-06-18T19:42:20Z

Pinging @elastic/es-search-relevance (Team:Search Relevance)

elasticsearchmachine · 2025-06-18T19:42:20Z

Hi @Mikep86, I've created a changelog YAML for you.

elasticsearchmachine · 2025-06-18T19:42:20Z

Pinging @elastic/search-eng (Team:SearchOrg)

elasticsearchmachine · 2025-06-18T19:42:20Z

Pinging @elastic/search-relevance (Team:Search - Relevance)

Mikep86 · 2025-06-18T19:47:57Z

x-pack/plugin/rank-rrf/src/main/java/org/elasticsearch/xpack/rank/rrf/RRFRetrieverBuilder.java

+            // TODO: Refactor duplicate code
+            // Using the multi-fields query format
+            var localIndicesMetadata = resolvedIndices.getConcreteLocalIndicesMetadata();
+            if (localIndicesMetadata.size() > 1) {
+                throw new IllegalArgumentException(
+                    "[" + NAME + "] cannot specify [" + QUERY_FIELD.getPreferredName() + "] when querying multiple indices"
+                );
+            } else if (resolvedIndices.getRemoteClusterIndices().isEmpty() == false) {
+                throw new IllegalArgumentException(
+                    "[" + NAME + "] cannot specify [" + QUERY_FIELD.getPreferredName() + "] when querying remote indices"
+                );
+            }


@kderusso I know you requested that we refactor this common code, can we handle that in a follow up along with refactoring the common test code?

Yes, given the timing I'm fine with that happening in a followup

kderusso

Nice work! I am OK with the cleanup/consolidation being a followup, a few questions on tests plus we should update the docs. Approving to not block.

.../rank-rrf/src/yamlRestTest/resources/rest-api-spec/test/rrf/310_rrf_retriever_simplified.yml

kderusso · 2025-06-18T20:09:20Z

x-pack/plugin/rank-rrf/src/main/java/org/elasticsearch/xpack/rank/rrf/RRFRetrieverBuilder.java

+            // TODO: Refactor duplicate code
+            // Using the multi-fields query format
+            var localIndicesMetadata = resolvedIndices.getConcreteLocalIndicesMetadata();
+            if (localIndicesMetadata.size() > 1) {
+                throw new IllegalArgumentException(
+                    "[" + NAME + "] cannot specify [" + QUERY_FIELD.getPreferredName() + "] when querying multiple indices"
+                );
+            } else if (resolvedIndices.getRemoteClusterIndices().isEmpty() == false) {
+                throw new IllegalArgumentException(
+                    "[" + NAME + "] cannot specify [" + QUERY_FIELD.getPreferredName() + "] when querying remote indices"
+                );
+            }


Yes, given the timing I'm fine with that happening in a followup

kderusso · 2025-06-18T20:10:26Z

...rank-rrf/src/test/java/org/elasticsearch/xpack/rank/rrf/RRFRetrieverBuilderParsingTests.java

-            + "    }"
-            + "  }"
-            + "}";
+        String restContent = """


Shouldn't we parse this twice, once with retrievers and once with field/query?

This test is only for XContent parsing purposes, the resulting retriever does not need to pass SearchRequest validation

kderusso · 2025-06-18T20:12:18Z

...plugin/rank-rrf/src/test/java/org/elasticsearch/xpack/rank/rrf/RRFRetrieverBuilderTests.java

+            "foo"
+        );
+
+        // Non-default rank window size and rank constant


I don't quite understand that this is testing anything with rank window size or rank constant?

It's testing that the rank window size and rank constant are propagated to the rewritten retrievers

Samiul-TheSoccerFan

Nice work! I liked the tests in 310_rrf_retriever_simplified.yml. It made much easier to understand the required changes.

...-rrf/src/yamlRestTest/resources/rest-api-spec/test/linear/20_linear_retriever_simplified.yml

x-pack/plugin/rank-rrf/src/main/java/org/elasticsearch/xpack/rank/rrf/RRFRetrieverBuilder.java

...rrf/src/yamlRestTest/java/org/elasticsearch/xpack/rank/rrf/RRFRankClientYamlTestSuiteIT.java

.../rank-rrf/src/yamlRestTest/resources/rest-api-spec/test/rrf/310_rrf_retriever_simplified.yml

pmpailis

LGTM

elasticsearchmachine · 2025-06-23T15:36:48Z

💔 Backport failed

Status	Branch	Result
❌	8.19	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 129659

Mikep86 · 2025-06-23T15:41:46Z

💚 All backports created successfully

Status	Branch	Result
✅	8.19

Questions ?

Please refer to the Backport tool documentation

(cherry picked from commit 636da86) # Conflicts: # x-pack/plugin/rank-rrf/src/main/java/org/elasticsearch/xpack/rank/rrf/RRFRetrieverBuilder.java # x-pack/plugin/rank-rrf/src/test/java/org/elasticsearch/xpack/rank/rrf/RRFRetrieverBuilderTests.java

Mikep86 added 10 commits June 18, 2025 13:57

Apply RRF retriever changes from elastic#128633

34040c9

Use RetrieverSource.from

041e4f3

Fixed references to SimplifiedInnerRetrieverUtils

488ecf5

Removed references to simplified query format from error messages

d2c439f

Copy pre-filters during RRF retriever rewrite

337e17d

Added cluster feature

ff29ab8

Fix some remaining references to simplified format

d2138e9

Update RRF YAML tests to use default distribution

88d13da

Added YAML tests

55bbdf8

Added missing headers specification

79afadf

Mikep86 requested review from pmpailis, jimczi and kderusso June 18, 2025 19:41

Mikep86 added >enhancement auto-backport Automatically create backport pull requests when merged :SearchOrg/Relevance Label for the Search (solution/org) Relevance team :Search Relevance/Search Catch all for Search Relevance v8.19.0 v9.1.0 labels Jun 18, 2025

elasticsearchmachine added Team:SearchOrg Meta label for the Search Org (Enterprise Search) Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch Team:Search - Relevance The Search organization Search Relevance team labels Jun 18, 2025

Update docs/changelog/129659.yaml

2611ff5

Fix changelog

09aab96

Mikep86 commented Jun 18, 2025

View reviewed changes

kderusso approved these changes Jun 18, 2025

View reviewed changes

Samiul-TheSoccerFan approved these changes Jun 18, 2025

View reviewed changes

...-rrf/src/yamlRestTest/resources/rest-api-spec/test/linear/20_linear_retriever_simplified.yml Show resolved Hide resolved

pmpailis reviewed Jun 19, 2025

View reviewed changes

x-pack/plugin/rank-rrf/src/main/java/org/elasticsearch/xpack/rank/rrf/RRFRetrieverBuilder.java Show resolved Hide resolved

pmpailis reviewed Jun 19, 2025

View reviewed changes

...rrf/src/yamlRestTest/java/org/elasticsearch/xpack/rank/rrf/RRFRankClientYamlTestSuiteIT.java Show resolved Hide resolved

pmpailis reviewed Jun 19, 2025

View reviewed changes

.../rank-rrf/src/yamlRestTest/resources/rest-api-spec/test/rrf/310_rrf_retriever_simplified.yml Show resolved Hide resolved

pmpailis approved these changes Jun 19, 2025

View reviewed changes

Mikep86 added 3 commits June 23, 2025 08:41

Add text and semantic text query tests

6ac5ec9

Added equivalent custom sub-retriever test

0178682

Merge branch 'main' into simplified-rrf-retriever

1408275

Mikep86 merged commit 636da86 into elastic:main Jun 23, 2025
32 checks passed

elasticsearchmachine added the backport pending label Jun 23, 2025

Mikep86 mentioned this pull request Jun 23, 2025

[8.19] Simplified RRF Retriever (#129659) #129869

Merged

Mikep86 removed the backport pending label Jun 23, 2025

mridula-s109 pushed a commit to mridula-s109/elasticsearch that referenced this pull request Jun 25, 2025

Simplified RRF Retriever (elastic#129659)

c16ee9f

Simplified RRF Retriever #129659

Simplified RRF Retriever #129659

Uh oh!

Conversation

Mikep86 commented Jun 18, 2025

Uh oh!

elasticsearchmachine commented Jun 18, 2025

Uh oh!

elasticsearchmachine commented Jun 18, 2025

Uh oh!

elasticsearchmachine commented Jun 18, 2025

Uh oh!

elasticsearchmachine commented Jun 18, 2025

Uh oh!

Mikep86 Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

kderusso Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

kderusso left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kderusso Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

kderusso Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

Mikep86 Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

kderusso Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

Mikep86 Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

Samiul-TheSoccerFan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pmpailis left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elasticsearchmachine commented Jun 23, 2025

💔 Backport failed

Uh oh!

Mikep86 commented Jun 23, 2025

💚 All backports created successfully

Questions ?

Uh oh!

Uh oh!