ESQL: text == and text != pushdown
#127355
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Reenables `text ==` pushdown and adds support for `text !=` pushdown.
|
Hi @nik9000, I've created a changelog YAML for you. |
nik9000
commented
Apr 24, 2025
|
This also wants more digging to be sure its right. This is the first PR that pushes things to lucene and rechecks them. The change for that was surprisingly small and I don't trust it to be that easy. |
|
OK! This is worth a look now! |
|
Pinging @elastic/es-analytical-engine (Team:Analytics) |
elasticsearchmachine
added
the
Team:Analytics
|
I'm going to make some performance numbers for this today. |
dnhatn
approved these changes
May 6, 2025
| if (ft == null) { | ||
| return new MatchNoDocsQuery("missing field [" + field() + "]"); | ||
| } | ||
| ft = ((TextFieldMapper.TextFieldType) ft).syntheticSourceDelegate(); |
There was a problem hiding this comment.
I think SemanticTextFieldType and MatchOnlyTextFieldType are text fields, but their base classes are not TextFieldMapper.TextFieldType.
There was a problem hiding this comment.
Update on this one: we don't push either of those fields. I've expanded the tests to hit that as well. We can pick them up in follow-up changes.
luigidellaquila
approved these changes
May 7, 2025
| import org.elasticsearch.xpack.esql.core.expression.Expression; | ||
| import org.elasticsearch.xpack.esql.core.querydsl.query.Query; | ||
| import org.elasticsearch.xpack.esql.optimizer.rules.physical.local.LucenePushdownPredicates; | ||
| import org.elasticsearch.xpack.esql.planner.TranslatorHandler; | ||
|
|
||
| /** | ||
| * Expressions implementing this interface can get called on data nodes to provide an Elasticsearch/Lucene query. | ||
| * Expressions implementing this interface are asked provide an | ||
| * Elasticsearch/Lucene query on the as part of the data node optimizations. |
There was a problem hiding this comment.
on the is probably a leftover
| */ | ||
| YES(FinishedTranslatable.YES), | ||
| /** | ||
| * Translation requires a recheck. Calling {@link TranslationAware#asQuery} will |
ywangd
pushed a commit
to ywangd/elasticsearch
that referenced
this pull request
May 9, 2025
Reenables `text ==` pushdown and adds support for `text !=` pushdown.
It does so by making `TranslationAware#translatable` return something
we can turn into a tri-valued function. It has these values:
* `YES`
* `NO`
* `RECHECK`
`YES` means the `Expression` is entirely pushable into Lucene. They will
be pushed into Lucene and removed from the plan.
`NO` means the `Expression` can't be pushed to Lucene at all and will stay
in the plan.
`RECHECK` mean the `Expression` can push a query that makes *candidate*
matches but must be rechecked. Documents that don't match the query won't
match the expression, but documents that match the query might not match
the expression. These are pushed to Lucene *and* left in the plan.
This is required because `txt != "b"` can build a *candidate* query
against the `txt.keyword` subfield but it can't be sure of the match
without loading the `_source` - which we do in the compute engine.
I haven't plugged rally into this, but here's some basic
performance tests:
```
Before:
not text eq {"took":460,"documents_found":1000000}
text eq {"took":432,"documents_found":1000000}
After:
text eq {"took":5,"documents_found":1}
not text eq {"took":351,"documents_found":800000}
```
This comes from:
```
rm -f /tmp/bulk*
for a in {1..1000}; do
echo '{"index":{}}' >> /tmp/bulk
echo '{"text":"text '$(printf $(($a % 5)))'"}' >> /tmp/bulk
done
ls -l /tmp/bulk*
passwd="redacted"
curl -sk -uelastic:$passwd -HContent-Type:application/json -XDELETE https://localhost:9200/test
curl -sk -uelastic:$passwd -HContent-Type:application/json -XPUT https://localhost:9200/test -d'{
"settings": {
"index.codec": "best_compression",
"index.refresh_interval": -1
},
"mappings": {
"properties": {
"many": {
"enabled": false
}
}
}
}'
for a in {1..1000}; do
printf %04d: $a
curl -sk -uelastic:$passwd -HContent-Type:application/json -XPOST https://localhost:9200/test/_bulk?pretty --data-binary @/tmp/bulk | grep errors
done
curl -sk -uelastic:$passwd -HContent-Type:application/json -XPOST https://localhost:9200/test/_forcemerge?max_num_segments=1
curl -sk -uelastic:$passwd -HContent-Type:application/json -XPOST https://localhost:9200/test/_refresh
echo
curl -sk -uelastic:$passwd https://localhost:9200/_cat/indices?v
text_eq() {
echo -n " text eq "
curl -sk -uelastic:$passwd -HContent-Type:application/json -XPOST 'https://localhost:9200/_query?pretty' -d'{
"query": "FROM test | WHERE text == \"text 1\" | STATS COUNT(*)",
"pragma": {
"data_partitioning": "shard"
}
}' | jq -c '{took, documents_found}'
}
not_text_eq() {
echo -n "not text eq "
curl -sk -uelastic:$passwd -HContent-Type:application/json -XPOST 'https://localhost:9200/_query?pretty' -d'{
"query": "FROM test | WHERE NOT text == \"text 1\" | STATS COUNT(*)",
"pragma": {
"data_partitioning": "shard"
}
}' | jq -c '{took, documents_found}'
}
for a in {1..100}; do
text_eq
not_text_eq
done
```
jfreden
pushed a commit
to jfreden/elasticsearch
that referenced
this pull request
May 12, 2025
Reenables `text ==` pushdown and adds support for `text !=` pushdown.
It does so by making `TranslationAware#translatable` return something
we can turn into a tri-valued function. It has these values:
* `YES`
* `NO`
* `RECHECK`
`YES` means the `Expression` is entirely pushable into Lucene. They will
be pushed into Lucene and removed from the plan.
`NO` means the `Expression` can't be pushed to Lucene at all and will stay
in the plan.
`RECHECK` mean the `Expression` can push a query that makes *candidate*
matches but must be rechecked. Documents that don't match the query won't
match the expression, but documents that match the query might not match
the expression. These are pushed to Lucene *and* left in the plan.
This is required because `txt != "b"` can build a *candidate* query
against the `txt.keyword` subfield but it can't be sure of the match
without loading the `_source` - which we do in the compute engine.
I haven't plugged rally into this, but here's some basic
performance tests:
```
Before:
not text eq {"took":460,"documents_found":1000000}
text eq {"took":432,"documents_found":1000000}
After:
text eq {"took":5,"documents_found":1}
not text eq {"took":351,"documents_found":800000}
```
This comes from:
```
rm -f /tmp/bulk*
for a in {1..1000}; do
echo '{"index":{}}' >> /tmp/bulk
echo '{"text":"text '$(printf $(($a % 5)))'"}' >> /tmp/bulk
done
ls -l /tmp/bulk*
passwd="redacted"
curl -sk -uelastic:$passwd -HContent-Type:application/json -XDELETE https://localhost:9200/test
curl -sk -uelastic:$passwd -HContent-Type:application/json -XPUT https://localhost:9200/test -d'{
"settings": {
"index.codec": "best_compression",
"index.refresh_interval": -1
},
"mappings": {
"properties": {
"many": {
"enabled": false
}
}
}
}'
for a in {1..1000}; do
printf %04d: $a
curl -sk -uelastic:$passwd -HContent-Type:application/json -XPOST https://localhost:9200/test/_bulk?pretty --data-binary @/tmp/bulk | grep errors
done
curl -sk -uelastic:$passwd -HContent-Type:application/json -XPOST https://localhost:9200/test/_forcemerge?max_num_segments=1
curl -sk -uelastic:$passwd -HContent-Type:application/json -XPOST https://localhost:9200/test/_refresh
echo
curl -sk -uelastic:$passwd https://localhost:9200/_cat/indices?v
text_eq() {
echo -n " text eq "
curl -sk -uelastic:$passwd -HContent-Type:application/json -XPOST 'https://localhost:9200/_query?pretty' -d'{
"query": "FROM test | WHERE text == \"text 1\" | STATS COUNT(*)",
"pragma": {
"data_partitioning": "shard"
}
}' | jq -c '{took, documents_found}'
}
not_text_eq() {
echo -n "not text eq "
curl -sk -uelastic:$passwd -HContent-Type:application/json -XPOST 'https://localhost:9200/_query?pretty' -d'{
"query": "FROM test | WHERE NOT text == \"text 1\" | STATS COUNT(*)",
"pragma": {
"data_partitioning": "shard"
}
}' | jq -c '{took, documents_found}'
}
for a in {1..100}; do
text_eq
not_text_eq
done
```
nik9000
added a commit
to nik9000/elasticsearch
that referenced
this pull request
May 19, 2025
Reenables `text ==` pushdown and adds support for `text !=` pushdown.
It does so by making `TranslationAware#translatable` return something
we can turn into a tri-valued function. It has these values:
* `YES`
* `NO`
* `RECHECK`
`YES` means the `Expression` is entirely pushable into Lucene. They will
be pushed into Lucene and removed from the plan.
`NO` means the `Expression` can't be pushed to Lucene at all and will stay
in the plan.
`RECHECK` mean the `Expression` can push a query that makes *candidate*
matches but must be rechecked. Documents that don't match the query won't
match the expression, but documents that match the query might not match
the expression. These are pushed to Lucene *and* left in the plan.
This is required because `txt != "b"` can build a *candidate* query
against the `txt.keyword` subfield but it can't be sure of the match
without loading the `_source` - which we do in the compute engine.
I haven't plugged rally into this, but here's some basic
performance tests:
```
Before:
not text eq {"took":460,"documents_found":1000000}
text eq {"took":432,"documents_found":1000000}
After:
text eq {"took":5,"documents_found":1}
not text eq {"took":351,"documents_found":800000}
```
This comes from:
```
rm -f /tmp/bulk*
for a in {1..1000}; do
echo '{"index":{}}' >> /tmp/bulk
echo '{"text":"text '$(printf $(($a % 5)))'"}' >> /tmp/bulk
done
ls -l /tmp/bulk*
passwd="redacted"
curl -sk -uelastic:$passwd -HContent-Type:application/json -XDELETE https://localhost:9200/test
curl -sk -uelastic:$passwd -HContent-Type:application/json -XPUT https://localhost:9200/test -d'{
"settings": {
"index.codec": "best_compression",
"index.refresh_interval": -1
},
"mappings": {
"properties": {
"many": {
"enabled": false
}
}
}
}'
for a in {1..1000}; do
printf %04d: $a
curl -sk -uelastic:$passwd -HContent-Type:application/json -XPOST https://localhost:9200/test/_bulk?pretty --data-binary @/tmp/bulk | grep errors
done
curl -sk -uelastic:$passwd -HContent-Type:application/json -XPOST https://localhost:9200/test/_forcemerge?max_num_segments=1
curl -sk -uelastic:$passwd -HContent-Type:application/json -XPOST https://localhost:9200/test/_refresh
echo
curl -sk -uelastic:$passwd https://localhost:9200/_cat/indices?v
text_eq() {
echo -n " text eq "
curl -sk -uelastic:$passwd -HContent-Type:application/json -XPOST 'https://localhost:9200/_query?pretty' -d'{
"query": "FROM test | WHERE text == \"text 1\" | STATS COUNT(*)",
"pragma": {
"data_partitioning": "shard"
}
}' | jq -c '{took, documents_found}'
}
not_text_eq() {
echo -n "not text eq "
curl -sk -uelastic:$passwd -HContent-Type:application/json -XPOST 'https://localhost:9200/_query?pretty' -d'{
"query": "FROM test | WHERE NOT text == \"text 1\" | STATS COUNT(*)",
"pragma": {
"data_partitioning": "shard"
}
}' | jq -c '{took, documents_found}'
}
for a in {1..100}; do
text_eq
not_text_eq
done
```
|
Backported by #128156 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Reenables
text ==pushdown and adds support fortext !=pushdown.It does so by making
TranslationAware#translatablereturn something we can turn into a tri-valued function. It has these values:YESNORECHECKYESmeans theExpressionis entirely pushable into Lucene. They will be pushed into Lucene and removed from the plan.NOmeans theExpressioncan't be pushed to Lucene at all and will stay in the plan.RECHECKmean theExpressioncan push a query that makes candidate matches but must be rechecked. Documents that don't match the query won't match the expression, but documents that match the query might not match the expression.These are pushed to Lucene and left in the plan.
This is required because
txt != "b"can build a candidate query against thetxt.keywordsubfield but it can't be sure of the match without loading the_source- which we do in the compute engine.I haven't plugged rally into this, but here's some basic performance tests:
This comes from: