Loading

Remote Elasticsearch output

Stack ECE ECH Self-Managed

Remote Elasticsearch outputs allow you to send Elastic Agent data to a remote Elasticsearch cluster. This is especially useful for data that you want to keep separate and independent from the deployment where you use Fleet to manage the Elastic Agents.

A remote Elasticsearch cluster supports the same output settings as your management Elasticsearch cluster.

These limitations apply to remote Elasticsearch output:

  • Using a remote Elasticsearch output with a target cluster that has network security enabled is not currently supported.
  • Using Elastic Defend when a remote Elasticsearch output is configured for an Elastic Agent is not currently supported.

To configure a remote Elasticsearch cluster for your Elastic Agent data:

  1. In your management Elasticsearch cluster, open Kibana, and search for Fleet settings in the search bar. Select Fleet/Settings in the results.

  2. In the Outputs section, select Add output.

  3. In the Add new output flyout, provide a name for the output, and select Remote Elasticsearch as the output type.

  4. In the Hosts field, add the URL that Elastic Agents should use to access the remote Elasticsearch cluster.

    Find the remote host address of the remote cluster
    1. In the remote cluster, open Kibana, and search for Fleet settings in the search bar. Select Fleet/Settings in the results.
    2. In the Outputs section, copy the Hosts value of the default Elasticsearch output. If the value is not visible in full, edit the default Elasticsearch output to display the full value.
    3. In your management cluster, paste the value you copied into the Hosts field of the remote output configuration.

  5. In the Service Token field, add a service token to access the remote cluster.

    Create a service token to access the remote cluster
    1. Copy the API request located below the Service Token field.
    2. In the remote cluster, open the Kibana menu, then go to Management → Dev Tools in self-managed deployments, or to Developer tools in Elastic Cloud deployments.
    3. Paste the API request in the console, then run it.
    4. Copy the value for the generated service token.
    5. In the management cluster, paste the value you copied into the Service Token field of the remote output configuration.
    Note

    To prevent unauthorized access, the Elasticsearch Service Token is stored as a secret value. While secret storage is recommended, you can choose to override this setting, and store the password as plain text in the agent policy definition. Secret storage requires Fleet Server version 8.12 or higher. This setting can also be stored as a secret value or as plain text for preconfigured outputs. To learn more about this option, check Preconfiguration settings.

  6. Choose whether integrations should be automatically synchronized on the remote Elasticsearch cluster. To configure this feature, refer to Automatic integrations synchronization.

    Note

    Automatic integrations synchronization is only available with certain subscriptions. For more information, refer to Subscriptions.

  7. Choose whether the remote output should be the default for agent integrations or for agent monitoring data. When set as the default, Elastic Agents use this output to send data if no other output is set in the agent policy.

  8. Select the performance tuning settings to optimize Elastic Agents for throughput, scale, or latency, or leave the default balanced setting.

  9. Add any advanced YAML configuration settings that you’d like for the remote output.

  10. Click Save and apply settings.

After the output is created, you can update an Elastic Agent policy to use the new output, and send data to the remote Elasticsearch cluster:

  1. In the management cluster, go to Fleet, then open the Agent policies tab.
  2. Click the agent policy you want to update, then click Settings.
  3. To send integrations data, set the Output for integrations option to use the output that you configured in the previous steps.
  4. To send Elastic Agent monitoring data, set the Output for agent monitoring option to use the output that you configured in the previous steps.
  5. Click Save changes.

The remote Elasticsearch output is now configured for the remote cluster.

If you choose not to synchronize integrations automatically, you need to make sure that for any integrations that are added to your Elastic Agent policy, the integration assets are also installed on the remote Elasticsearch cluster. For detailed steps on this process, refer to Install and uninstall Elastic Agent integration assets.

Note

When you use a remote Elasticsearch output, Fleet Server performs a test to ensure connectivity to the remote cluster. The result of that connectivity test is used to report whether the remote output is healthy or unhealthy, and is displayed on the Fleet → Settings → Outputs page, in the Status column.

In some cases, the remote Elasticsearch output used for Elastic Agent data can be reached by the Elastic Agents but not by Fleet Server. In those cases, you can ignore the resulting unhealthy state of the output and the associated Unable to connect error on the UI.