Upgrade Fleet-managed Elastic Agents

Stack

With Fleet upgrade capabilities, you can view and select agents that are out of date, and trigger selected agents to download, install, and run the new version. You can trigger upgrades to happen immediately, or specify a maintenance window during which the upgrades will occur. If your Elastic Stack subscription level supports it, you can schedule upgrades to occur at a specific date and time.

In most failure cases the Elastic Agent may retry an upgrade after a short wait. The wait durations between retries are: 1m, 5m, 10m, 15m, 30m, and 1h. During this time, the Elastic Agent may show up as "retrying" in the Fleet UI. As well, if agent upgrades have been detected to have stalled, you can restart the upgrade process for a single agent or in bulk for multiple agents.

This approach simplifies the process of keeping your agents up to date. It also saves you time because you don’t need third-party tools or processes to manage upgrades.

By default, Elastic Agents require internet access to perform binary upgrades from Fleet. However, you can host your own artifacts repository and configure Elastic Agents to download binaries from it. For more information, refer to Air-gapped environments.

For a detailed view of the Elastic Agent upgrade process and the interactions between Fleet, Elastic Agent, and Elasticsearch, refer to the Communications amongst components diagram in the elastic-agent GitHub repository.

Note the following restrictions with upgrading an Elastic Agent:

• Elastic Agent cannot be upgraded to a version higher than the highest currently installed version of Fleet Server. When you upgrade a set of Elastic Agents that are currently at the same version, you should first upgrade any agents that are acting as Fleet Server (any agents that have a Fleet Server policy associated with them).
• To be upgradeable, Elastic Agent must not be running inside a container.
• To be upgradeable in a Linux environment, Elastic Agent must be running as a service. The Linux Tar install instructions for Elastic Agent provided in Fleet include the commands to run it as a service. Elastic Agent RPM and DEB system packages cannot be upgraded through Fleet.

These restrictions apply whether you are upgrading Elastic Agents individually or in bulk. In the event that an upgrade isn’t eligible, Fleet generates a warning message when you attempt the upgrade.

To upgrade your Elastic Agents, go to Management → Fleet → Agents in Kibana. You can perform the following upgrade-related actions:

With the right subscription level, you can also configure an automatic, gradual upgrade of a percentage of the Elastic Agents enrolled in an Elastic Agent policy. For more information, refer to Auto-upgrade agents enrolled in a policy. Stack 9.1.0

1. On the Agents tab, agents that can be upgraded are identified with an Upgrade available indicator.

   
   ×

   You can also click the Upgrade available button to filter the list agents to only those that currently can be upgraded.

2. From the Actions menu next to the agent, choose Upgrade agent.

   
   ×

3. In the Upgrade agent window, select or specify an upgrade version and click Upgrade agent.

   In certain cases the latest available Elastic Agent version may not be recognized by Kibana. For instance, this occurs when the Kibana version is lower than the Elastic Agent version. You can specify a custom version for Elastic Agent to upgrade to by entering the version into the Upgrade version text field.

   
   ×

You can do rolling upgrades to avoid exhausting network resources when updating a large number of Elastic Agents.

1. On the Agents tab, select multiple agents, and click Actions.

2. From the Actions menu, choose to upgrade the agents.

3. In the Upgrade agents window, select an upgrade version.

4. Select the amount of time available for the maintenance window. The upgrades are spread out uniformly across this maintenance window to avoid exhausting network resources.

   To force selected agents to upgrade immediately when the upgrade is triggered, select Immediately. Avoid using this setting for batches of more than 10 agents.

5. Upgrade the agents.

1. On the Agents tab, select one or more agents, and click Actions.

2. From the Actions menu, choose to schedule an upgrade.

   
   ×

   If the schedule option is grayed out, it may not be available at your subscription level. For more information, refer to Elastic Stack subscriptions.

3. In the Upgrade window, select an upgrade version.

4. Select a maintenance window. For more information, refer to Do a rolling upgrade of multiple Elastic Agents.

5. Set the date and time when you want the upgrade to begin.

6. Click Schedule.

On the Agents tab, when you trigger an upgrade, agents that are upgrading have the status Updating until the upgrade is complete, and then the status changes back to Healthy.

Agents on version 8.12 and higher that are currently upgrading additionally show a detailed upgrade status indicator.

×

The following table explains the upgrade states in the order that they can occur.

Under routine circumstances, an Elastic Agent upgrade happens quickly. You typically will not see the agent transition through each of the upgrade states. The detailed upgrade status can be a very useful tool especially if you need to diagnose the state of an agent that may have become stuck, or just appears to have become stuck, during the upgrade process.

Beside the upgrade status indicator, you can hover your cursor over the information icon to get more detail about the upgrade.

×

×

Note that when you upgrade agents from versions below 8.12, the upgrade details are not provided.

×

When upgrading many agents, you can fine tune the maintenance window by viewing stats and metrics about the upgrade:

1. On the Agents tab, click the host name to view agent details. If you don’t see the host name, try refreshing the page.
2. Click View more agent metrics to open the [Elastic Agent] Agent metrics dashboard.

If an upgrade appears to have stalled, you can restart it.

If an upgrade fails, you can view the agent logs to find the reason:

1. In Fleet, in the Host column, click the agent’s name.

2. Open the Logs tab.

3. Search for failures.

   
   ×

An Elastic Agent upgrade process may sometimes stall. This can happen for various reasons, including, for example, network connectivity issues or a delayed shutdown.

When an Elastic Agent upgrade has been detected to be stuck, a warning indicator appears on the UI. When this occurs, you can restart the upgrade from either the Agents tab on the main Fleet page or from the details page for any individual agent.

Note that there is a required 10 minute cooldown period in between restart attempts. After launching a restart action you need to wait for the cooldown to complete before initiating another restart.

Restart from main Fleet page:

1. From the Actions menu next to an agent that is stuck in an Updating state, choose Restart upgrade.
2. In the Restart upgrade window, select an upgrade version and click Upgrade agent.

Restart from an agent details page:

1. In Fleet, in the Host column, click the agent’s name. On the Agent details tab, a warning notice appears if the agent is detected to have stalled during an upgrade.
2. Click Restart upgrade.
3. In the Restart upgrade window, select an upgrade version and click Upgrade agent.

When the upgrade process for multiple agents has been detected to have stalled, you can restart the upgrade process in bulk. As with restarting an upgrade for a single agent, a 10 minute cooldown period is enforced between restarts.

1. On the Agents tab, select any set of the agents that are indicated to be stuck, and click Actions.

2. From the Actions menu, select Restart upgrade <number> agents.

3. In the Restart upgrade… window, select an upgrade version.

4. Select the amount of time available for the maintenance window. The upgrades are spread out uniformly across this maintenance window to avoid exhausting network resources.

   To force selected agents to upgrade immediately when the upgrade is triggered, select Immediately. Avoid using this setting for batches of more than 10 agents.

5. Restart the upgrades.

Stack 9.1.0

To configure an automatic rollout of a new minor or patch version to a percentage of the agents enrolled in your Elastic Agent policy. follow these steps:

1. In Kibana, go to Management → Fleet → Agent policies.

2. Select the agent policy for which you want to configure an automatic agent upgrade.

3. On the agent policy's details page, find Auto-upgrade agents, and select Manage next to it.

4. In the Manage auto-upgrade agents window, click Add target version.

5. From the Target agent version dropdown, select the minor or patch version to which you want to upgrade a percentage of your agents.

6. In the % of agents to upgrade field, enter the percentage of active agents you want to upgrade to this target version.

   Note that:

   • Unenrolling, unenrolled, inactive, and uninstalled agents are not included in the count. For example, if you set the target upgrade percentage to 50% for a policy with 10 active agents and 10 inactive agents, the target is met when 5 active agents are upgraded.
   • Rounding is applied, and the actual percentage of the upgraded agents may vary slightly. For example, if you set the target upgrade percentage to 30% for a policy with 25 active agents, the target is met when 8 active agents are upgraded (32%).

7. You can then add a different target version, and specify the percentage of agents you want to be upgraded to that version. The total percentage of agents to be upgraded cannot exceed 100%.

8. Click Save.

Once the configuration is saved, an asynchronous task runs every 30 minutes, gradually upgrading the agents in the policy to the specified target version.

In case of any failed upgrades, the upgrades are retried with exponential backoff mechanism until the upgrade is successful, or the maximum number of retries is reached. Note that the maximum number of retries is the number of configured retry delays.

On self-managed and cloud deployments of Elastic Stack, you can configure the default task interval and the retry delays of the automatic upgrade in the Kibana Fleet settings. For example:

xpack.fleet.autoUpgrades.taskInterval: 15m
xpack.fleet.autoUpgrades.retryDelays: ['5m', '10m', '20m']

1. The time interval at which the auto-upgrade task should run. Defaults to 30m.
2. Array indicating how much time should pass before a failed auto-upgrade is retried. The array's length indicates the maximum number of retries. Defaults to ['30m', '1h', '2h', '4h', '8h', '16h', '24h'].

For more information, refer to the Kibana configuration reference.

You can view the status of the automatic upgrade in the following ways:

• On the agent policy's details page, find Auto-upgrade agents, and select Manage to open the Manage auto-upgrade agents window.

  The status of the upgrade is displayed next to the specified target version and percentage, and includes the percentage of agents that have already been upgraded.

  To view any failed upgrades, hover over the Upgrade failed status, then click Go to upgrade.

• On the Fleet → Agents page, click Agent activity to open a flyout showing logs of the Elastic Agent activity and the progress of the automatic agent upgrade.

If you have installed and enrolled Elastic Agent using either a DEB (for a Debian-based Linux distribution) or RPM (for a RedHat-based Linux distribution) install package, the upgrade cannot be managed by Fleet. Instead, you can perform the upgrade using these steps.

For installation steps refer to Install Fleet-managed Elastic Agents.

1. Download the Elastic Agent Debian install package for the release that you want to upgrade to:

   curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-9.1.0-amd64.deb
   

2. Upgrade Elastic Agent to the target release:

   sudo dpkg -i elastic-agent-9.1.0-amd64.deb
   

3. Confirm in Fleet that the agent has been upgraded to the target version. Note that the Upgrade agent option in the Actions menu next to the agent will be disabled since [fleet]-managed upgrades are not supported for this package type.

1. Download the Elastic Agent RPM install package for the release that you want to upgrade to:

   curl -L -O https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-9.1.0-x86_64.rpm
   

2. Upgrade Elastic Agent to the target release:

   sudo rpm -U elastic-agent-9.1.0-x86_64.rpm
   

3. Confirm in Fleet that the agent has been upgraded to the target version. Note that the Upgrade agent option in the Actions menu next to the agent will be disabled since Fleet-managed upgrades are not supported for this package type.