Private connectivity with AWS PrivateLink

ECH

You can use AWS PrivateLink to establish a secure connection for your Elastic Cloud deployments to communicate with other AWS services. AWS routes the PrivateLink traffic within the AWS data center and never exposes it to the public internet.

AWS PrivateLink connects your Virtual Private Cloud (VPC) to the AWS-hosted services that you use, treating them as if they were in your VPC. You can create and use VPC endpoints to securely access AWS-hosted services.

You can also optionally filter traffic to your deployments by creating virtual private connection (VPC) filters as part of your private connection policy in Elastic Cloud. This limits traffic to your deployment to the VPC specified in the policy, as well as any other policies applied to the deployment.

To learn how private connection policies impact your deployment, refer to Network security policies in Elastic Cloud.

Before you begin, review the following considerations:

Private connectivity with AWS PrivateLink is supported only in AWS regions. Elastic does not yet support cross-region AWS PrivateLink connections. Your PrivateLink endpoint needs to be in the same region as your target deployments. Additional details can be found in the AWS VPCE Documentation.

AWS interface virtual private connection (VPC) endpoints are configured for one or more availability zones (AZ). In some regions, our VPC endpoint service is not present in all the possible AZs that a region offers. You can only choose AZs that are common on both sides. As the names of AZs (for example us-east-1a) differ between AWS accounts, the following list of AWS regions shows the ID (e.g. use1-az4) of each available AZ for the service.

Refer to interface endpoint availability zone considerations for more details.

Elastic charges for inter-node traffic regardless of whether nodes are in the same or different availability zones (AZ). As a result, placing the deployment nodes within a single AZ, instead of two or three, does not reduce inter-node costs.

On the customer VPC side, the inter-availability zone data transfer, within the same AWS region, towards AWS PrivateLink endpoints, is free of charge. As a result, you do not incur charges for cross-AZ data transfer within your VPC when the target is the AWS Privatelink Elastic Cloud service endpoint. We recommend you set up the VPC endpoints in all supported Elastic Cloud AZs for a particular region for maximum traffic throughput and resiliency.

If Elastic and your VPC overlap in two AZs or less, you can create subnets and VPC PrivateLink endpoints in your VPC within the same availability zones where the Elastic PrivateLink service is present.

Transport client is not supported over PrivateLink connections.

PrivateLink Service is set up by Elastic in all supported AWS regions under the following service names:

The process of setting up a private connection with AWS PrivateLink is split between the AWS console and the Elastic Cloud UI. These are the high-level steps:

After you create your private connection policy, you can edit, disassociate, or delete it.

Before you begin, you should ensure your VPC endpoint is in all availability zones supported by Elastic Cloud on the region for the VPC service.

Ensuring that your VPC is in all supported Elastic Cloud availability zones for a particular region avoids potential for a traffic imbalance. That imbalance may saturate some coordinating nodes and underutilize others in the deployment, eventually impacting performance. Enabling all supported Elastic Cloud zones ensures that traffic is balanced optimally.

You can find the zone name to zone ID mapping with AWS CLI:

$ aws ec2 describe-availability-zones --region us-east-1 | jq -c '.AvailabilityZones[] | { id: .ZoneId, name: .ZoneName } ' | sort
{"id":"use1-az1","name":"us-east-1c"}
{"id":"use1-az2","name":"us-east-1e"}
{"id":"use1-az3","name":"us-east-1d"}
{"id":"use1-az4","name":"us-east-1a"}
{"id":"use1-az5","name":"us-east-1f"}
{"id":"use1-az6","name":"us-east-1b"}

The mapping will be different for your region. Our production VPC Service for us-east-1 is located in use1-az2, use1-az4, use1-az6. We need to create the VPC Endpoint for the preceding mapping in at least one of us-east-1e, us-east-1a, us-east-1b.

1. Create a VPC endpoint in your VPC using the service name for your region.

   Refer to the AWS documentation for details on creating a VPC interface endpoint to an endpoint service.

   Use the service name for your region.

   
   ×

   The security group for the endpoint should, at minimum, allow for inbound connectivity from your instances' CIDR range on ports 443 and 9243. Security groups for the instances should allow for outbound connectivity to the endpoint on ports 443 and 9243.

2. Create a DNS record.

   1. Create a Private hosted zone.

      Refer to the Private hosted zone domain name column in the PrivateLink service names and aliases table for the name of the zone. For example, in us-east-1, use vpce.us-east-1.aws.elastic-cloud.com as the zone domain name.

      Don’t forget to associate the zone with your VPC.

      
      ×

   2. Create a DNS CNAME alias pointing to the PrivateLink endpoint. Add the record to a private DNS zone in your VPC. Use * as the record name, and the VPC endpoint DNS name as a value.

      Refer to the AWS documentation for details on creating a CNAME record which points to your VPC endpoint DNS name.

      
      ×

After you create your VPC endpoint and DNS entries, check that you are able to reach your cluster over PrivateLink.

Use the following URL structure. This URL is built from endpoint information retrieved from your Elastic deployment and the private hosted zone domain name that you registered.

https://{{alias}}.{{product}}.{{private_hosted_zone_domain_name}}

For example:

https://my-deployment-d53192.es.vpce.us-east-1.aws.elastic-cloud.com

To test the connection:

1. If needed, find the endpoint of an application in your deployment:

   1. Log in to the Elastic Cloud Console.

   2. Under Hosted deployments, find your deployment.

   Tip

   If you have many deployments, you can instead go to the Hosted deployments (Elastic Cloud Hosted) page. On that page, you can narrow your deployments by name, ID, or choose from several other filters.

   3. Select Manage.
   4. In the deployment overview, under Applications, find the application that you want to test.
   5. Click Copy endpoint. The value looks something like the following:

   https://my-deployment-d53192.es.us-east-1.aws.elastic-cloud.com
   

   In this endpoint, my-deployment-d53192 is an alias, and es is the product you want to access within your deployment.

2. Test the setup using the following cURL command. Pass the username and password for a user that has access to the cluster. Make sure to replace the URL with your deployment's endpoint information and the private hosted zone domain name that you registered.

   Request

   $ curl -v https://my-deployment-d53192.es.vpce.us-east-1.aws.elastic-cloud.com -u {username}:{password}
   

   Response

   * Server certificate:
   *  subject: CN=*.us-east-1.aws.elastic-cloud.com
   *  SSL certificate verify ok.
   ..
       < HTTP/1.1 200 OK
   ..
   {
       "name" : "instance-0000000009",
       "cluster_name" : "fb7e805e5cfb4931bdccc4f3cb591f5f",
       "cluster_uuid" : "2cTHeCQYS2a0iH7YnQHrIQ",
       "version" : { ... },
       "tagline" : "You Know, for Search"
   }
   

The connection is established, and a valid certificate is presented to the client. Elastic responds, in the case of the Elasticsearch endpoint, with basic information about the cluster.

After you test your PrivateLink connection, you can create a private connection policy in Elastic Cloud.

Private connection policies are optional for AWS PrivateLink. After the VPC endpoint and DNS record are created, private connectivity is established.

Creating a private connection policy and associating it with your deployments allows you to do the following:

• Record that you've established private connectivity between AWS and Elastic in the applicable region.
• Filter traffic to your deployment using VPC filters.

Follow these high-level steps to add a private connection policy that can be associated with your deployments.

1. Optional: Find your VPC endpoint ID.
2. Create a private connection policy using the VPC endpoint.
3. Associate the VPC endpoint with your deployment.

The VPC endpoint ID is only required if you want to filter traffic to your deployment using VPC filters.

You can find your VPC endpoint ID in the AWS console:

×

Create a new private connection policy.

1. Log in to the Elastic Cloud Console.
2. From any deployment or project on the home page, select Manage.
3. From the left navigation menu, select Access and security > Network security.

4. Select Private connection.

5. Select the resource type that the private connection will be applied to. Currently, only hosted deployments are supported.

6. Select the cloud provider and region for the private connection.

   Tip

   Private connection policies are bound to a single region, and can be assigned only to deployments in the same region. If you want to associate a policy with resources in multiple regions, then you have to create the same policy in all the regions you want to apply it to.

7. Under Connectivity, select Privatelink.

8. Optional: Under VPC filter, enter your VPC endpoint ID. You should only specify a VPC filter if you want to filter traffic to your deployment.

   If you don't specify a VPC filter, then the private connection policy acts only as a record that you've established private connectivity between AWS and Elastic in the applicable region.

   Tip

   You can apply multiple policies to a single deployment. The policies can be of different types. In case of multiple policies, traffic can match any associated policy to be forwarded to the resource. If none of the policies match, the request is rejected with 403 Forbidden.

   Learn more about how network security policies affect your deployment.

9. Optional: Under Apply to resources, associate the new private connection policy with one or more deployments. If you specified a VPC filter, then after you associate the filter with a deployment, it starts filtering traffic.

10. To automatically attach this private connection policy to new deployments, select Apply by default.

11. Click Create.

12. (Optional) You can claim your VPC endpoint ID, so that no other organization is able to use it in a private connection policy.

The next step is to associate the policy with your deployment.

You can associate a private connection policy with your deployment from the policy's settings, or from your deployment's settings.

If the policy contains a VPC filter, then after you associate the policy with a deployment, it starts filtering traffic.

If the policy doesn't contain a VPC filter, then the association can serve as a reminder that a VPC endpoint exists for the deployment's region.

1. Find your deployment on the home page or on the Hosted deployments page, then select Manage to access its settings menus.

   On the Hosted deployments page, you can narrow your deployments by name, ID, or choose from several other filters. To customize your view, use a combination of filters, or change the format from a grid to a list.

2. On the Security page, under Network security, select Apply policies > Private connection.

3. Choose the policy you want to apply and select Apply.

1. Log in to the Elastic Cloud Console.
2. From any deployment or project on the home page, select Manage.
3. From the left navigation menu, select Access and security > Network security.

5. Find the policy you want to edit.
6. Under Apply to resources, associate the policy with one or more deployments.
7. Click Update to save your changes.

For traffic to connect with the deployment over Azure PrivateLink, the client making the request needs to be located within the VPC where you’ve created the VPC endpoint. You can also set up network traffic to flow through the originating VPC from somewhere else, such as another VPC or VPN from your corporate network. This assumes that the VPC endpoint and the DNS record are also available within that context. Check your service provider documentation for setup instructions.

Use the following URL structure. This URL is built from endpoint information retrieved from your Elastic deployment and the private hosted zone domain name that you registered.

https://{{alias}}.{{product}}.{{private_hosted_zone_domain_name}}

For example:

https://my-deployment-d53192.es.vpce.us-east-1.aws.elastic-cloud.com

To access the deployment:

1. If needed, find the endpoint of an application in your deployment:

   1. Log in to the Elastic Cloud Console.

   2. Under Hosted deployments, find your deployment.

   Tip

   If you have many deployments, you can instead go to the Hosted deployments (Elastic Cloud Hosted) page. On that page, you can narrow your deployments by name, ID, or choose from several other filters.

   3. Select Manage.
   4. In the deployment overview, under Applications, find the application that you want to test.
   5. Click Copy endpoint. The value looks something like the following:

   https://my-deployment-d53192.es.us-east-1.aws.elastic-cloud.com
   

   In this endpoint, my-deployment-d53192 is an alias, and es is the product you want to access within your deployment.

2. Send a request:

   Request

   $ curl -v https://my-deployment-d53192.es.vpce.us-east-1.aws.elastic-cloud.com -u {username}:{password}
   

   Response

   * Server certificate:
   *  subject: CN=*.us-east-1.aws.elastic-cloud.com
   *  SSL certificate verify ok.
   ..
       < HTTP/1.1 200 OK
   ..
   {
       "name" : "instance-0000000009",
       "cluster_name" : "fb7e805e5cfb4931bdccc4f3cb591f5f",
       "cluster_uuid" : "2cTHeCQYS2a0iH7YnQHrIQ",
       "version" : { ... },
       "tagline" : "You Know, for Search"
   }
   

If you are using AWS PrivateLink together with Fleet, and enrolling the Elastic Agent with a private connection URL, you need to configure Fleet Server to use and propagate the AWS PrivateLink URL by updating the Fleet Server hosts field in the Fleet settings section of Kibana. Otherwise, Elastic Agent will reset to use a default address instead of the AWS PrivateLink URL.

The URL needs to follow this pattern:

https://{{fleet_component_ID_or_deployment_alias}}.fleet.{{private_hosted_zone_domain_name}}:443`

Similarly, the Elasticsearch host needs to be updated to propagate the private connection URL. The Elasticsearch URL needs to follow this pattern:

https://{{elasticsearch_cluster_ID_or_deployment_alias}}.es.{{private_hosted_zone_domain_name}}:443

The settings xpack.fleet.agents.fleet_server.hosts and xpack.fleet.outputs that are needed to enable this configuration in Kibana are not available in the Kibana settings in Elastic Cloud.

After you create your private connection policy, you can edit it, remove it from your deployment, or delete it.

You can edit a policy's name, description, VPC endpoint ID, and more.

1. Log in to the Elastic Cloud Console.
2. From any deployment or project on the home page, select Manage.
3. From the left navigation menu, select Access and security > Network security.

1. Find the policy you want to edit, then click the Edit button.
2. Click Update to save your changes.

If you want to a specific policy from a deployment, or delete the policy, then you need to disconnect it from any associated deployments first. You can do this from the policy's settings, or from your deployment's settings. To remove an association through the UI:

1. Find your deployment on the home page or on the Hosted deployments page, then select Manage to access its settings menus.

   On the Hosted deployments page you can narrow your deployments by name, ID, or choose from several other filters. To customize your view, use a combination of filters, or change the format from a grid to a list.

2. On the Security page, under Network security, find the policy that you want to disconnect.

3. Under Actions, click the Delete icon.

1. Log in to the Elastic Cloud Console.
2. From any deployment or project on the home page, select Manage.
3. From the left navigation menu, select Access and security > Network security.

5. Find the policy you want to edit, then click the Edit button.
6. Under Apply to resources, click the x beside the resource that you want to disconnect.
7. Click Update to save your changes.

If you need to remove a policy, you must first remove any associations with deployments.

To delete a policy:

1. Log in to the Elastic Cloud Console.
2. From any deployment or project on the home page, select Manage.
3. From the left navigation menu, select Access and security > Network security.

4. Find the policy you want to edit, then click the Delete icon. The icon is inactive if there are deployments associated with the policy.