Deliver Logs of ECS(Fargate) containers in Private Subnets to CloudWatch Logs
One way to collect logs of containers running in an ECS task is to use CloudWatch Logs.
In this article, we will check how to distribute logs to CloudWatch Logs.
Environment
Create a Fargate type ECS on a private subnet.
Create a VPC endpoint for Logs on the container subnet to distribute logs to CloudWatch Logs.
Place a NAT gateway on the public subnet to retrieve the official Nginx image from DockerHub.
Create an EC2 instance.
Use it as a client to access the container.
CloudFormation template files
Build the above configuration with CloudFormation.
The CloudFormation templates are located at the following URL
https://github.com/awstut-an-r/awstut-fa/tree/main/065
Explanation of key points of template files
This page focuses on how to distribute logs to CloudWatch Logs.
For basic information on ECS (Fargate), please refer to the following page
For information on how to deploy Fargate on a private subnet, please see the following page
VPC Endpoints for CloudWatch Logs
Resources:
CloudWatchLogsEndpoint:
Type: AWS::EC2::VPCEndpoint
Properties:
PrivateDnsEnabled: true
SecurityGroupIds:
- !Ref EndpointSecurityGroup2
ServiceName: !Sub "com.amazonaws.${AWS::Region}.logs"
SubnetIds:
- !Ref ContainerSubnet
VpcEndpointType: Interface
VpcId: !Ref VPC
Code language: YAML (yaml)In order to deliver Fargate logs located on a private subnet to CloudWatch Logs, create a VPC endpoint for CloudWatch Logs.
Below is the security group to be applied to this endpoint.
Resources:
EndpointSecurityGroup2:
Type: AWS::EC2::SecurityGroup
Properties:
GroupName: !Sub "${Prefix}-EndpointSecurityGroup2"
GroupDescription: Allow HTTPS from ContainerSecurityGroup.
VpcId: !Ref VPC
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: !Ref HTTPSPort
ToPort: !Ref HTTPSPort
SourceSecurityGroupId: !Ref ContainerSecurityGroup
Code language: YAML (yaml)The contents allow inbound communication of 443/tcp with the security group applied to Fargate as the source.
CloudWatch Logs Log Group
Resources:
LogGroup:
Type: AWS::Logs::LogGroup
Properties:
LogGroupName: !Sub "${Prefix}-LogGroup"
Code language: YAML (yaml)Create a simple log group.
No special configuration is required to distribute logs for the Fargate container.
Log settings for ECS task
Resources:
TaskDefinition:
Type: AWS::ECS::TaskDefinition
Properties:
ContainerDefinitions:
- Name: !Sub "${Prefix}-container"
Image: nginx:latest
LogConfiguration:
LogDriver: awslogs
Options:
awslogs-group: !Ref LogGroup
awslogs-region: !Ref AWS::Region
awslogs-stream-prefix: !Sub "${Prefix}-container"
Cpu: !Ref TaskCpu
ExecutionRoleArn: !Ref FargateTaskExecutionRole
Memory: !Ref TaskMemory
NetworkMode: awsvpc
RequiresCompatibilities:
- FARGATE
TaskRoleArn: !Ref TaskRole
Code language: YAML (yaml)The key point is the LogsConfiguration property.
Specify the log driver with the LogDriver property.
To distribute logs to CloudWatch Logs, specify “awslogs”.
Set the parameters regarding the destination CloudWatch Logs in the Options property.
Set at least three parameters.
The first is awslogs-group, which specifies the destination log group.
The second, awslogs-region, specifies the region where the destination log group is created.
The third is awslogs-stream-prefix, which specifies the prefix of the stream name to be created in the log group.
Architecting
Use CloudFormation to build this environment and check the actual behavior.
Create CloudFormation stacks and check resources in stacks
Create a CloudFormation stacks.
For information on how to create stacks and check each stack, please refer to the following page
After checking the resources in each stack, information on the main resources created this time is as follows
- ECS cluster: fa-065-cluster
- ECS service: fa-065-service
- CloudWatch Logs log group: fa-065-LogGroup
- EC2 instance: i-0be518a618c126bbf
Check the created resource from the AWS Management Console.
First, check the ECS cluster.
We can see that the cluster has been successfully created and the ECS service is in active.
Check the details of the task created in the service.
We can see that a container has been created in the task, and that it is working properly.
You can also see that a private address has been assigned to the task.
The address assigned this time is “10.0.3.77”.
Checking Action
Now that everything is ready, access the EC2 instance.
% aws ssm start-session --target i-0be518a618c126bbf
Starting session with SessionId: root-0ba9574fb6aac6aaf
sh-4.2$
Code language: Bash (bash)For more information on SSM Session Manager, please refer to the following page
Access the container in the task using the curl command.
sh-4.2$ curl http://10.0.3.77
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
Code language: Bash (bash)We were able to access it successfully.
Then check CloudWatch Logs.
We can see that the log group has been successfully created and a log stream has been created inside.
Check the log stream.
You can see that the log from the EC2 instance accessed earlier has been written.
From the above, we can see that by using the VPC endpoint for CloudWatch Logs, we can distribute the logs of the Fargate container on the private subnet to CloudWatch Logs.
Summary
We have seen how to distribute logs of Fargate containers on private subnets to CloudWatch Logs.