EDOT Cloud Forwarder for AWS

Serverless Observability Preview EDOT CF AWS Preview

EDOT Cloud Forwarder for AWS provides the EDOT Collector as a Lambda function that collects and forwards logs to Elastic Observability on Elastic Cloud Serverless. EDOT Cloud Forwarder for AWS supports the following log sources:

Log source	Description
VPC Flow	Logs generated by a Virtual Private Cloud (VPC)
ELB Access	Logs generated by an Elastic Load Balancer (ELB)

Read on to learn how to set up EDOT Cloud Forwarder for AWS.

Note

We are working to support other popular log types and sources. Get in touch to let us know of any specific requirements that could influence our plans.

Prerequisites

Important

EDOT Cloud Forwarder for AWS is not currently supported on Elastic Cloud Hosted or self-managed Elastic Stack.

To collect logs using EDOT Cloud Forwarder for AWS, you need:

VPC Flow

To collect VPC Flow logs, you need:

A Virtual Private Cloud (VPC)
An S3 bucket for storing flow logs
A flow log configured with the S3 bucket as the destination

ELB Access

To collect Elastic Load Balancer (ELB) Access logs, you need:

An ELB of any type (ALB, NLB, CLB)
An S3 bucket to store the access logs
Access logging enabled, with the bucket as the destination

In addition, you need to know the URL of the managed OTLP endpoint and the API key for authentication.

Deployment considerations

Before deploying EDOT Cloud Forwarder for AWS, keep these points in mind:

Deploy a separate CloudFormation stack for each log type, for example VPC Flow Logs or ELB Logs. Each CloudFormation stack can only process one log source and format at a time.
Logs stored in S3 must be placed in separate buckets. Each log type should reside in its own dedicated bucket.

Download the template

Download the CloudFormation template to deploy the appropriate stack based on your log source:

Log Source	CloudFormation template
S3 logs	`https://edot-cloud-forwarder.s3.amazonaws.com/v0/latest/cloudformation/s3_logs-cloudformation.yaml`

For specific versions, edit latest in the URL to the required version in the format vX.Y.Z.

Configure the template

EDOT Cloud Forwarder for AWS uses a CloudFormation template to deploy the EDOT Collector as a Lambda function.

Required settings

These are the required settings you need:

Setting	Description
`region`	AWS region where the CloudFormation stack is to be deployed
`stack-name`	Name of the CloudFormation stack, for example, `vpc-edot-cf` Do not use the same name for different sources.
`OTLPEndpoint`	The OTLP endpoint URL used for data ingestion
`ElasticApiKey`	API key for authentication with Elastic

Log source settings

Set the following settings based on the log source:

For S3 logs, use the following settings:

Setting	Description
`EdotCloudForwarderS3LogsType`	The encoding format for logs in the S3 bucket. Supported options: - `vpc_flow_log`: VPC Flow logs - `elb_access_log`: Elastic Load Balancer (ELB) Access logs - `s3_access_log`: S3 Access logs - `json`: JSON-formatted logs
`S3LogsJsonEncodingMode`	(Required if `EdotCloudForwarderS3LogsType` is `json`) Defines how JSON logs are structured: - `body` (default): Stores logs in the request body - `body_with_inline_attributes`: Logs include inline attributes
`SourceS3BucketARN`	Amazon Resource Name (ARN) of the S3 bucket where logs are stored. This bucket will trigger the `edot-cloud-forwarder` Lambda function automatically.

Optional settings

These are optional settings you can set in the CloudFormation template:

Setting	Description
`EdotCloudForwarderTimeout`	Maximum execution time for the Lambda function, measured in seconds. Default value is `300` seconds. Minimum value is `1` second. Maximum value is `900` seconds.
`EdotCloudForwarderVersion`	Version of the EDOT Cloud Forwarder. Expected format is semantic versioning, for example `0.1.5`. Defaults to the latest available patch version. Don't change this value unless advised by Elastic Support.
`EdotCloudForwarderMemorySize`	Set the allocated memory for the Lambda function, measured in megabytes. Default value is `1024` MB. Minimum value is `128` MB. Maximum value is `10240` MB.
`EdotCloudForwarderConcurrentExecutions`	Set the maximum number of reserved concurrent executions for the Lambda function. Default value is `50`. Make sure this value doesn't exceed your AWS account's concurrency limit.

Deployment examples

The following examples use the CloudFormation template files hosted in the public S3 bucket.

Use the --template-url flag to reference a template hosted on S3.
To always use the most recent stable templates, use the latest path. For example, v0/latest.
To pin a specific version, replace latest with the desired version tag. For example, v0/v0.1.5.

Alternatively, if you have downloaded the template file, you can use the --template-body file://<path> option with a local template file.

VPC Flow logs

This example deploys a CloudFormation stack to collect VPC Flow logs stored in an S3 bucket.

		aws cloudformation create-stack \
  --stack-name edot-cloud-forwarder-vpc \
  --template-url https://edot-cloud-forwarder.s3.amazonaws.com/v0/latest/cloudformation/s3_logs-cloudformation.yaml \
  --capabilities CAPABILITY_NAMED_IAM \
  --region eu-central-1 \
  --parameters \
    ParameterKey=SourceS3BucketARN,ParameterValue=your-s3-vpc-bucket-arn \
    ParameterKey=OTLPEndpoint,ParameterValue="<placeholder>" \
    ParameterKey=ElasticAPIKey,ParameterValue="<placeholder>" \
    ParameterKey=EdotCloudForwarderS3LogsType,ParameterValue="vpc_flow_log"

	

ELB Access logs

This example deploys a CloudFormation stack to collect ALB Access logs stored in an S3 bucket.

		aws cloudformation create-stack \
  --stack-name edot-cloud-forwarder-alb \
  --template-url https://edot-cloud-forwarder.s3.amazonaws.com/v0/latest/cloudformation/s3_logs-cloudformation.yaml \
  --capabilities CAPABILITY_NAMED_IAM \
  --region eu-central-1 \
  --parameters \
    ParameterKey=SourceS3BucketARN,ParameterValue=your-s3-alb-bucket-arn \
    ParameterKey=OTLPEndpoint,ParameterValue="<placeholder>" \
    ParameterKey=ElasticAPIKey,ParameterValue="<placeholder>" \
    ParameterKey=EdotCloudForwarderS3LogsType,ParameterValue="elb_access_log"

	

To update an existing CloudFormation stack while preserving some parameter values, follow these steps:

Identify the stack to update

Determine the name of the CloudFormation stack you want to modify.
Prepare the update command

Use the following structure for your update command:
- Include all required parameters.
- Use UsePreviousValue=true for parameters that should remain unchanged.
- Specify ParameterValue=<new-value> for parameters that need to be updated.

Run the `update-stack` command

Run the command with the following parameters:

		aws cloudformation update-stack \
  --template-url https://edot-cloud-forwarder.s3.amazonaws.com/v0/latest/cloudformation/<template-file-name>.yaml \
  --stack-name <stack-name> \
  --capabilities CAPABILITY_NAMED_IAM \
  --region eu-central-1 \
  --parameters \
      ParameterKey=Param1,UsePreviousValue=true \
      ParameterKey=Param2,UsePreviousValue=true \
      ParameterKey=Param3,UsePreviousValue=true \
      ParameterKey=Param4,ParameterValue=<new-value>

	

For example, to modify the S3 bucket ARN for the edot-cloud-forwarder-vpc stack while keeping other parameter values unchanged:

		aws cloudformation update-stack \
  --template-url https://edot-cloud-forwarder.s3.amazonaws.com/v0/latest/cloudformation/s3_logs-cloudformation.yaml \
  --stack-name edot-cloud-forwarder-vpc \
  --capabilities CAPABILITY_NAMED_IAM \
  --region eu-central-1 \
  --parameters \
      ParameterKey=OTLPEndpoint,UsePreviousValue=true \
      ParameterKey=ElasticAPIKey,UsePreviousValue=true \
      ParameterKey=EdotCloudForwarderS3LogsType,UsePreviousValue=true \
      ParameterKey=SourceS3BucketARN,ParameterValue=your-new-s3-vpc-bucket-arn

	

Verify the update

After running the command, check the stack status in the AWS Management Console under CloudFormation → Stacks. Then, run this command to confirm the updated parameter values:
```
aws cloudformation describe-stacks --stack-name <stack-name>
```

Manual deployment using the CloudFormation Console

You can deploy the stack manually using the AWS Management Console by following these steps:

Navigate to CloudFormation in the AWS Console.
Select Create Stack and choose With new resources (standard) to start a fresh deployment.
Select one of the following options under Specify template:
- Amazon S3 URL (Recommended): Use the pre-hosted CloudFormation template stored in the public S3 bucket.
- Upload a template file: Download the template from the S3 bucket and upload it manually.
Select Next and configure all required parameters.
Select Next again and check Acknowledge IAM capabilities. This is required because the template creates named IAM roles with permissions to access the required resources.
Review your configuration and select Submit to deploy the stack.
Monitor the progress until the stack reaches the CREATE_COMPLETE state.

CloudFormation stack resources

The CloudFormation templates create a number of resources to process logs from a specific log source.

Resources for S3 logs

This is a list of resources created by the stack when processing S3 logs.

Resource name	Type	Description
`CustomNotificationUpdater`	`AWS::CloudFormation::CustomResource`	Custom resource used to manage S3 event notifications dynamically.
`LambdaExecutionRole`	`AWS::IAM::Role`	IAM role granting permissions needed for the Lambda function to interact with S3 and other AWS services.
`LambdaFunction`	`AWS::Lambda::Function`	Core Lambda function responsible for processing incoming logs from S3. This is a key resource in the stack.
`LambdaInvokeConfig`	`AWS::Lambda::EventInvokeConfig`	Configures error handling and invocation settings for the Lambda function.
`LambdaLogGroup`	`AWS::Logs::LogGroup`	CloudWatch log group storing logs for the main Lambda function. Useful for debugging and monitoring.
`LambdaPermissionS3Bucket`	`AWS::Lambda::Permission`	Grants permission for S3 to invoke the Lambda function when new logs arrive.
`LambdaS3TriggerPolicy`	`AWS::IAM::Policy`	IAM policy allowing the Lambda function to process events triggered by S3.
`NotificationUpdaterLambda`	`AWS::Lambda::Function`	Utility Lambda function handling S3 event notification updates dynamically.
`NotificationUpdaterLambdaLogGroup`	`AWS::Logs::LogGroup`	CloudWatch log group storing logs for the `NotificationUpdaterLambda` function.
`S3FailureBucketARN`	`AWS::S3::Bucket`	ARN of the bucket for storing failed invocations from the `edot-cloud-forwarder` Lambda function, preventing data loss, in the format `arn:aws:s3:::your-bucket-name`. If not defined, the template creates a dedicated failure bucket automatically.

The main Lambda function, LambdaFunction, is the core component for processing S3 logs. S3 event notifications are handled dynamically using CustomNotificationUpdater and NotificationUpdaterLambda.

CloudWatch logs ensure detailed monitoring of Lambda executions. IAM roles and permissions control access between S3 and Lambda functions, while S3FailureBucketARN prevents data loss by capturing unprocessed logs.

Delete a CloudFormation stack

If you no longer need a deployed stack and want to clean up all associated resources, you can delete it using the following command:

		aws cloudformation delete-stack \
  --stack-name <stack-name> \
  --region <stack-region>

	

Deleting a stack will remove all AWS resources created by that stack. However, if you allowed the stack to automatically create a dedicated S3 bucket for failed Lambda invocations, that bucket will not be deleted if it contains objects, because CloudFormation doesn't force-delete non-empty buckets. To remove the bucket entirely, you must empty it manually before deleting it.

If you specified an existing bucket through the S3FailureBucketARN parameter instead, that bucket will not be deleted because it is not managed by the stack.

The following command deletes the edot-cloud-forwarder-vpc stack:

		aws cloudformation delete-stack \
  --stack-name edot-cloud-forwarder-vpc \
  --region eu-central-1

	

You can monitor the deletion progress in the AWS Console under CloudFormation → Stacks, or through this command:

		aws cloudformation describe-stacks \
  --stack-name <stack-name> \
  --region <stack-region>

	

If the stack deletion fails and remains in a DELETE_FAILED state, you can retry the deletion with force mode:

		aws cloudformation delete-stack \
  --stack-name edot-cloud-forwarder-vpc \
  --region eu-central-1 \
  --deletion-mode FORCE_DELETE_STACK

	

This forcibly deletes the stack's resources, except any that cannot be deleted, like the failure S3 bucket if it still contains objects. For a complete cleanup, empty the bucket manually before retrying deletion.