Inference integrations

Stack Serverless

Elasticsearch provides a machine learning inference API to create and manage inference endpoints that integrate with services such as Elasticsearch (for built-in NLP models like ELSER and E5), as well as popular third-party services like Amazon Bedrock, Anthropic, Azure AI Studio, Cohere, Google AI, Mistral, OpenAI, Hugging Face, and more.

You can use the default inference endpoints your deployment contains or create a new inference endpoint:

• using the Create an inference endpoint API
• through the Inference endpoints UI.

Your Elasticsearch deployment contains preconfigured inference endpoints, which makes them easier to use when defining semantic_text fields or using inference processors. These endpoints come in two forms:

• Elastic Inference Service (EIS) endpoints, which provide inference as a managed service and do not consume resources from your own nodes.

• ML node-based endpoints, which run on your dedicated machine learning nodes.

The following section lists the default inference endpoints, identified by their inference_id, grouped by whether they are EIS- or ML node–based.

• .elser-2-elastic: uses the ELSER trained model as an Elastic Inference Service for sparse_embedding tasks (recommended for English language text). The model_id is .elser_model_2. Stack Preview 9.1.0 Serverless Preview

• .elser-2-elasticsearch: uses the ELSER built-in trained model for sparse_embedding tasks (recommended for English language text). The model_id is .elser_model_2_linux-x86_64.
• .multilingual-e5-small-elasticsearch: uses the E5 built-in trained model for text_embedding tasks (recommended for non-English language texts). The model_id is .e5_model_2_linux-x86_64.

Use the inference_id of the endpoint in a semantic_text field definition or when creating an inference processor. The API call will automatically download and deploy the model which might take a couple of minutes. Default inference enpoints have adaptive allocations enabled. For these models, the minimum number of allocations is 0. If there is no inference activity that uses the endpoint, the number of allocations will scale down to 0 automatically after 15 minutes.

The Inference endpoints page provides an interface for managing inference endpoints.

×

Available actions:

• Add new endpoint
• View endpoint details
• Copy the inference endpoint ID
• Delete endpoints

To add a new inference endpoint using the UI:

1. Select the Add endpoint button.
2. Select a service from the drop down menu.
3. Provide the required configuration details.
4. Select Save to create the endpoint.

If your inference endpoint uses a model deployed in Elastic’s infrastructure, such as ELSER, E5, or a model uploaded through Eland, you can configure adaptive allocations to dynamically adjust resource usage based on the current demand.

Adaptive allocations allow inference services to dynamically adjust the number of model allocations based on the current load. This feature is only supported for models deployed in Elastic’s infrastructure, such as ELSER, E5, or models uploaded through Eland. It is not available for models used through the Elastic Inference Service (EIS) and third-party services (for example, Alibaba Cloud, Cohere, or OpenAI), because those models are not deployed within your Elasticsearch cluster.

When adaptive allocations are enabled:

• The number of allocations scales up automatically when the load increases.
• Allocations scale down to a minimum of 0 when the load decreases, saving resources.

The behavior of allocations depends on several factors:

• Deployment type (Elastic Cloud Hosted, Elastic Cloud Enterprise, or Serverless)
• Usage level (low, medium, or high)
• Optimization type (ingest or search)

For more information about adaptive allocations and resources, refer to the trained model autoscaling documentation.

Inference endpoints have a limit on the amount of text they can process at once, determined by the model's input capacity. Chunking is the process of splitting the input text into pieces that remain within these limits. It occurs when ingesting documents into semantic_text fields. Chunking also helps produce sections that are digestible for humans. Returning a long document in search results is less useful than providing the most relevant chunk of text.

Each chunk will include the text subpassage and the corresponding embedding generated from it.

By default, documents are split into sentences and grouped in sections up to 250 words with 1 sentence overlap so that each chunk shares a sentence with the previous chunk. Overlapping ensures continuity and prevents vital contextual information in the input text from being lost by a hard break.

Elasticsearch uses the ICU4J library to detect word and sentence boundaries for chunking. Word boundaries are identified by following a series of rules, not just the presence of a whitespace character. For written languages that do use whitespace such as Chinese or Japanese dictionary lookups are used to detect word boundaries.

Several strategies are available for chunking:

sentence

The sentence strategy splits the input text at sentence boundaries. Each chunk contains one or more complete sentences ensuring that the integrity of sentence-level context is preserved, except when a sentence causes a chunk to exceed a word count of max_chunk_size, in which case it will be split across chunks. The sentence_overlap option defines the number of sentences from the previous chunk to include in the current chunk which is either 0 or 1.

word

The word strategy splits the input text on individual words up to the max_chunk_size limit. The overlap option is the number of words from the previous chunk to include in the current chunk.

recursive Stack 9.1.0

The recursive strategy splits the input text based on a configurable list of separator patterns (for example, newlines or Markdown headers). The chunker applies these separators in order, recursively splitting any chunk that exceeds the max_chunk_size word limit. If no separator produces a small enough chunk, the strategy falls back to sentence-level splitting.

none Stack 9.1.0

The none strategy disables chunking and processes the entire input text as a single block, without any splitting or overlap. When using this strategy, you can instead pre-chunk the input by providing an array of strings, where each element acts as a separate chunk to be sent directly to the inference service without further chunking.

The default chunking strategy is sentence.

The following example creates an inference endpoint with the elasticsearch service that deploys the ELSER model by default and configures the chunking behavior.

 PUT _inference/sparse_embedding/small_chunk_size {
  "service": "elasticsearch",
  "service_settings": {
    "num_allocations": 1,
    "num_threads": 1
  },
  "chunking_settings": {
    "strategy": "sentence",
    "max_chunk_size": 100,
    "sentence_overlap": 0
  }
}