Create Web app using DynamoDB with ECS (Fargate)
The following page shows how to attach an ECS (Fargate) in a private subnet to an ALB.
In the above page, we introduced a container that starts a web server with Nginx and returns static HTML files.
On this page, we will containerize a simple web app in Python that accesses DynamoDB and associate this with ALB.
The goal is to access DynamoDB and return all data.
Environment
The basic structure is the same as the above page.
There are three changes.
The first is a container running on Fargate.
Using the Python web framework bottle, the container runs as a web server and returns data stored in DynamoDB.
The second point is how to push the image to the ECR repository.
In this case, we will use CodeBuild to build the image for the container described above.
CodeBuild is triggered by a Lambda function.
To associate this function with a CloudFormation custom resource and set it to run automatically when the CloudFormation stack is created.
The third point is DynamoDB.
This one is also configured to automatically store items in a DyanamoDB table when the stack is created by the Lambda function associated with the CloudFormation custom resource.
In addition, the item information to be stored is referenced from the JSON format stored in the S3 bucket.
CloudFormation template files for building the environment
The above configuration is built with CloudFormation.
Place the CloudFormation templates at the following URL
https://github.com/awstut-an-r/awstut-dva/tree/main/02/004
Explanation of key points of template files
This page focuses on how to access DynamoDB in an ECS (Fargate) container.
For information on how to attach resources in a private subnet to an ALB, please see the following page.
For more information on the basics of Fargate, please see the following pages.
DynamoDB
VPC Endpoint
Resources:
PrivateSubnet1:
Type: AWS::EC2::Subnet
Properties:
CidrBlock: !Ref CidrIp3
VpcId: !Ref VPC
AvailabilityZone: !Sub "${AWS::Region}${AvailabilityZone1}"
PrivateSubnet2:
Type: AWS::EC2::Subnet
Properties:
CidrBlock: !Ref CidrIp4
VpcId: !Ref VPC
AvailabilityZone: !Sub "${AWS::Region}${AvailabilityZone2}"
PrivateRouteTable:
Type: AWS::EC2::RouteTable
Properties:
VpcId: !Ref VPC
PrivateSubnetRouteTableAssociation1:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
SubnetId: !Ref PrivateSubnet1
RouteTableId: !Ref PrivateRouteTable
PrivateSubnetRouteTableAssociation2:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
SubnetId: !Ref PrivateSubnet2
RouteTableId: !Ref PrivateRouteTable
DynamoDBEndpoint:
Type: AWS::EC2::VPCEndpoint
Properties:
RouteTableIds:
- !Ref PrivateRouteTable
ServiceName: !Sub "com.amazonaws.${AWS::Region}.dynamodb"
VpcId: !Ref VPC
Code language: YAML (yaml)Since the ECS container accessing DynamoDB is located on private subnets, a route must be provided to connect to this resource.
This time we will create a VPC endpoint for DynamoDB.
For information on how to connect to DynamoDB tables from private subnets, please see the following page.
DynamoDB table
Resources:
Table:
Type: AWS::DynamoDB::Table
Properties:
AttributeDefinitions:
- AttributeName: Artist
AttributeType: S
- AttributeName: SongTitle
AttributeType: S
BillingMode: PAY_PER_REQUEST
KeySchema:
- AttributeName: Artist
KeyType: HASH
- AttributeName: SongTitle
KeyType: RANGE
TableClass: STANDARD
TableName: !Sub "${Prefix}-Music"
Code language: YAML (yaml)No special settings are made.
The configuration shown in the following official AWS page is exported to a CloudFormation template.
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/getting-started-step-1.html
For basic information on DynamoDB, please refer to the following pages.
ECS
Resources:
Cluster:
Type: AWS::ECS::Cluster
Properties:
ClusterName: !Sub "${Prefix}-cluster"
FargateTaskExecutionRole:
Type: AWS::IAM::Role
DeletionPolicy: Delete
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- ecs-tasks.amazonaws.com
Action:
- sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
TaskRole:
Type: AWS::IAM::Role
DeletionPolicy: Delete
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- ecs-tasks.amazonaws.com
Action:
- sts:AssumeRole
Policies:
- PolicyName: AllowSQSSendMessage
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- dynamodb:Scan
Resource:
- !Sub "arn:aws:dynamodb:${AWS::Region}:${AWS::AccountId}:table/${Table}"
TaskDefinition:
Type: AWS::ECS::TaskDefinition
Properties:
ContainerDefinitions:
- Image: !Sub "${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com/${ECRRepositoryName}:latest"
LogConfiguration:
LogDriver: awslogs
Options:
awslogs-group: !Ref LogGroup
awslogs-region: !Ref AWS::Region
awslogs-stream-prefix: !Ref Prefix
Name: !Sub "${Prefix}-bottle"
PortMappings:
- ContainerPort: !Ref BottlePort
HostPort: !Ref BottlePort
Environment:
- Name: TABLE_NAME
Value: !Ref Table
Cpu: !Ref TaskCpu
ExecutionRoleArn: !Ref FargateTaskExecutionRole
Memory: !Ref TaskMemory
NetworkMode: awsvpc
RequiresCompatibilities:
- FARGATE
RuntimePlatform:
CpuArchitecture: ARM64
OperatingSystemFamily: LINUX
TaskRoleArn: !Ref TaskRole
Service:
Type: AWS::ECS::Service
Properties:
Cluster: !Ref Cluster
DesiredCount: 1
LaunchType: FARGATE
LoadBalancers:
- ContainerName: !Sub "${Prefix}-bottle"
ContainerPort: !Ref BottlePort
TargetGroupArn: !Ref ALBTargetGroup
NetworkConfiguration:
AwsvpcConfiguration:
SecurityGroups:
- !Ref ContainerSecurityGroup
Subnets:
- !Ref PrivateSubnet1
- !Ref PrivateSubnet2
ServiceName: !Sub "${Prefix}-service"
TaskDefinition: !Ref TaskDefinition
Code language: YAML (yaml)There are two points.
The first point is the role for the task.
Inline policy to authorize scanning to DynamoDB.
The second point is the task definition.
The Environment property allows you to set environment variables that can be accessed in the container, set the name of the DynamoDB table.
(Reference) Using CloudFormation custom resources and CodeBuild to automatically push test images to the ECR repository when creating CloudFormation stacks
Resources:
CodeBuildProject:
Type: AWS::CodeBuild::Project
Properties:
Artifacts:
Type: NO_ARTIFACTS
Cache:
Type: NO_CACHE
Environment:
ComputeType: !Ref ProjectEnvironmentComputeType
EnvironmentVariables:
- Name: BOTTLE_PORT
Type: PLAINTEXT
Value: !Ref BottlePort
- Name: DOCKERHUB_PASSWORD
Type: SECRETS_MANAGER
Value: !Sub "${Secret}:password"
- Name: DOCKERHUB_USERNAME
Type: SECRETS_MANAGER
Value: !Sub "${Secret}:username"
Image: !Ref ProjectEnvironmentImage
ImagePullCredentialsType: CODEBUILD
Type: !Ref ProjectEnvironmentType
PrivilegedMode: true
LogsConfig:
CloudWatchLogs:
Status: DISABLED
S3Logs:
Status: DISABLED
Name: !Ref Prefix
ServiceRole: !GetAtt CodeBuildRole.Arn
Source:
Type: NO_SOURCE
BuildSpec: !Sub |
version: 0.2
phases:
pre_build:
commands:
- echo Logging in to Amazon ECR...
- aws --version
- aws ecr get-login-password --region ${AWS::Region} | docker login --username AWS --password-stdin ${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com
- REPOSITORY_URI=${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com/${ECRRepositoryName}
- COMMIT_HASH=$(echo $CODEBUILD_RESOLVED_SOURCE_VERSION | cut -c 1-7)
- IMAGE_TAG=${!COMMIT_HASH:=latest}
- echo Logging in to Docker Hub...
- echo $DOCKERHUB_PASSWORD | docker login -u $DOCKERHUB_USERNAME --password-stdin
- |
cat << EOF > Dockerfile
FROM amazonlinux
RUN yum update -y && yum install python3 python3-pip -y
RUN pip3 install boto3 bottle
COPY main.py ./
CMD ["python3", "main.py"]
EXPOSE $BOTTLE_PORT
EOF
- |
cat << EOF > main.py
from bottle import Bottle, route, run
import boto3
import json
import os
TABLE_NAME = os.environ['TABLE_NAME']
app = Bottle()
dynamodb_client = boto3.client('dynamodb')
@app.route('/')
def scan():
response = dynamodb_client.scan(
TableName=TABLE_NAME
)
return json.dumps(response['Items'], indent=2)
if __name__ == '__main__':
run(app=app, host='0.0.0.0', port=$BOTTLE_PORT)
EOF
build:
commands:
- echo Build started on `date`
- echo Building the Docker image...
- docker build -t $REPOSITORY_URI:latest .
- docker tag $REPOSITORY_URI:latest $REPOSITORY_URI:$IMAGE_TAG
post_build:
commands:
- echo Build completed on `date`
- echo Pushing the Docker images...
- docker push $REPOSITORY_URI:latest
- docker push $REPOSITORY_URI:$IMAGE_TAG
Visibility: PRIVATE
Code language: YAML (yaml)Images for containers to run on ECS can be manually pushed to the ECR repository.
However, this page uses CodeBuild to automatically build the image and push it to the ECR repository.
The trigger to start the build is a Lambda function tied to a CloudFormation custom resource.
For more information, please see the following page.
The key points are the two commands in buildspec.yml that are executed during the pre_build phase.
The first point is where the cat command creates a Dockerfile.
The content is based on the latest version of the Amazon Linux image, and after installing Python and packages, the script described below is executed.
The environment variables specified in the EnvironmentVariables property are used to specify the ports to be published.
The second point is where the Python script main.py is created, also with the cat command.
This script uses the web framework bottle to act as a simple web server.
When the root page is accessed, it scans the DynamoDB table, retrieves all items, and returns them as a response.
The name of the table to be accessed is referenced in an environment variable.
(Reference) Initial setup of DynamoDB with CloudFormation custom resources
Resources:
Function2:
Type: AWS::Lambda::Function
Properties:
Environment:
Variables:
JSON_S3_BUCKET: !Ref JsonS3Bucket
JSON_S3_KEY: !Ref JsonS3Key
TABLE_NAME: !Ref Table
Code:
ZipFile: |
import boto3
import cfnresponse
import json
import os
JSON_S3_BUCKET = os.environ['JSON_S3_BUCKET']
JSON_S3_KEY = os.environ['JSON_S3_KEY']
TABLE_NAME = os.environ['TABLE_NAME']
CREATE = 'Create'
response_data = {}
s3_client = boto3.client('s3')
dynamodb_client = boto3.client('dynamodb')
def lambda_handler(event, context):
try:
if event['RequestType'] == CREATE:
s3_response = s3_client.get_object(
Bucket=JSON_S3_BUCKET,
Key=JSON_S3_KEY)
body = s3_response['Body'].read()
print(body)
json_data = json.loads(body.decode('utf-8'))
print(json_data)
for item in json_data:
dynamodb_response = dynamodb_client.put_item(
TableName=TABLE_NAME,
Item=item
)
print(dynamodb_response)
cfnresponse.send(event, context, cfnresponse.SUCCESS, response_data)
except Exception as e:
print(e)
cfnresponse.send(event, context, cfnresponse.FAILED, response_data)
FunctionName: !Sub "${Prefix}-function2"
Handler: !Ref Handler
Runtime: !Ref Runtime
Role: !GetAtt FunctionRole2.Arn
Timeout: !Ref Timeout
Code language: YAML (yaml)Automatically store test data in DynamoDB when creating CloudFormation stacks.
This is accomplished by associating the above Lambda function with a CloudFormation custom resource.
For details, please refer to the following page.
Summary of ALB and ECS port numbers
The communication in this configuration is organized as shown in the figure below.
Communication from the user to the ALB is directed to port 80, but communication from the ALB to the ECS (Fargate) is directed to the port used by the bottle application (8080).
To achieve the above configuration, set each property of each resource as follows
- AWS::ElasticLoadBalancingV2::Listener Port：80
- AWS::ElasticLoadBalancingV2::TargetGroup Port：8080
- AWS::ECS::Service LoadBalancers ContainerPort：8080
- AWS::ECS::TaskDefinition PortMappings ContainerPort：8080
- AWS::ECS::TaskDefinition PortMappings HostPort：8080
The security groups attached to both resources are as follows
Resources:
ALBSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupName: !Sub ${Prefix}-ALBSecurityGroup
GroupDescription: Allow HTTP Only.
VpcId: !Ref VPC
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: !Ref HTTPPort
ToPort: !Ref HTTPPort
CidrIp: 0.0.0.0/0
ContainerSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupName: !Sub "${Prefix}-ContainerSecurityGroup"
GroupDescription: Allow HTTP from ALBSecurityGroup.
VpcId: !Ref VPC
SecurityGroupIngress:
- IpProtocol: tcp
FromPort: !Ref BottlePort
ToPort: !Ref BottlePort
SourceSecurityGroupId: !Ref ALBSecurityGroup
Code language: YAML (yaml)Architecting
We will use CloudFormation to build this environment and check its actual behavior.
Create CloudFormation stacks and check the resources in the stacks
Create CloudFormation stacks.
For information on how to create stacks and check each stack, please see the following page.
After reviewing the resources in each stack, information on the main resources created in this case is as follows
- ECR Repository: dva-02-004
- DynamoDB table: dva-02-004-Music
- ECS cluster: dva-02-004-cluster
- ALB’s domain: dva-02-004-alb-909547272.ap-northeast-1.elb.amazonaws.com
Check each resource from the AWS Management Console.
Check the ECR repository.
The repository is successfully created and the image is pushed.
This means that the Lambda function associated with the CloudFormation custom resource has been executed and CodeBuild has started building the image and pushed it to the repository.
Check the DynamoDB table.
The data is stored in a table.
This means that the Lambda function associated with the CloudFormation custom resource has been executed, the JSON data stored in the S3 bucket has been read, and the item has been saved to the DynamoDB table.
Check the ECS.
Check the clusters first.
One ECS service is created in the cluster.
Check the tasks in the service.
It can be seen that they are related to the ALB.
You can also see one task running in the service.
Within the task, an image is pulled from the ECR repository mentioned earlier and a container is created.
Operation Check
Now that you are ready, access the ALB.
% curl http://dva-02-004-alb-909547272.ap-northeast-1.elb.amazonaws.com/
[ { "AlbumTitle": { "S": "Somewhat Famous" }, "Awards": { "N": "1" }, "Artist": { "S": "No One You Know" }, "SongTitle": { "S": "Call Me Today" } }, { "AlbumTitle": { "S": "Somewhat Famous" }, "Awards": { "N": "2" }, "Artist": { "S": "No One You Know" }, "SongTitle": { "S": "Howdy" } }, { "AlbumTitle": { "S": "Songs About Life" }, "Awards": { "N": "10" }, "Artist": { "S": "Acme Band" }, "SongTitle": { "S": "Happy Day" } }, { "AlbumTitle": { "S": "Another Album Title" }, "Awards": { "N": "8" }, "Artist": { "S": "Acme Band" }, "SongTitle": { "S": "PartiQL Rocks" } } ]Code language: Bash (bash)Accessing the ALB, a response was successfully received.
As a response, the item information stored in DynamoDB was returned.
I was able to access DynamoDB from ECS (Fargate).
Summary
We have created an ECS (Fargate) container application that accesses DynamoDB.