DVA_EN

Create Web app using DynamoDB with ECS (Fargate)

スポンサーリンク
Create Web app using DynamoDB with ECS(Fargate) DVA_EN
スポンサーリンク
スポンサーリンク

Create Web app using DynamoDB with ECS (Fargate)

The following page shows how to attach an ECS (Fargate) in a private subnet to an ALB.

In the above page, we introduced a container that starts a web server with Nginx and returns static HTML files.

On this page, we will containerize a simple web app in Python that accesses DynamoDB and associate this with ALB.
The goal is to access DynamoDB and return all data.

Environment

Diagram of creating Web app using DynamoDB with ECS(Fargate) 1

The basic structure is the same as the above page.

There are three changes.

The first is a container running on Fargate.
Using the Python web framework bottle, the container runs as a web server and returns data stored in DynamoDB.

The second point is how to push the image to the ECR repository.
In this case, we will use CodeBuild to build the image for the container described above.
CodeBuild is triggered by a Lambda function.
To associate this function with a CloudFormation custom resource and set it to run automatically when the CloudFormation stack is created.

The third point is DynamoDB.
This one is also configured to automatically store items in a DyanamoDB table when the stack is created by the Lambda function associated with the CloudFormation custom resource.
In addition, the item information to be stored is referenced from the JSON format stored in the S3 bucket.

CloudFormation template files for building the environment

The above configuration is built with CloudFormation.
Place the CloudFormation templates at the following URL

awstut-dva/02/004 at main · awstut-an-r/awstut-dva
Contribute to awstut-an-r/awstut-dva development by creating an account on GitHub.

Explanation of key points of template files

This page focuses on how to access DynamoDB in an ECS (Fargate) container.

For information on how to attach resources in a private subnet to an ALB, please see the following page.

For more information on the basics of Fargate, please see the following pages.

DynamoDB

VPC Endpoint

Resources:
  PrivateSubnet1:
    Type: AWS::EC2::Subnet
    Properties:
      CidrBlock: !Ref CidrIp3
      VpcId: !Ref VPC
      AvailabilityZone: !Sub "${AWS::Region}${AvailabilityZone1}"
      
  PrivateSubnet2:
    Type: AWS::EC2::Subnet
    Properties:
      CidrBlock: !Ref CidrIp4
      VpcId: !Ref VPC
      AvailabilityZone: !Sub "${AWS::Region}${AvailabilityZone2}"
      
  PrivateRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
      VpcId: !Ref VPC
      
  PrivateSubnetRouteTableAssociation1:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref PrivateSubnet1
      RouteTableId: !Ref PrivateRouteTable
      
  PrivateSubnetRouteTableAssociation2:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
      SubnetId: !Ref PrivateSubnet2
      RouteTableId: !Ref PrivateRouteTable

  DynamoDBEndpoint:
    Type: AWS::EC2::VPCEndpoint
    Properties:
      RouteTableIds:
        - !Ref PrivateRouteTable
      ServiceName: !Sub "com.amazonaws.${AWS::Region}.dynamodb"
      VpcId: !Ref VPC
Code language: YAML (yaml)

Since the ECS container accessing DynamoDB is located on private subnets, a route must be provided to connect to this resource.

This time we will create a VPC endpoint for DynamoDB.

For information on how to connect to DynamoDB tables from private subnets, please see the following page.

DynamoDB table

Resources:
  Table:
    Type: AWS::DynamoDB::Table
    Properties:
      AttributeDefinitions:
        - AttributeName: Artist
          AttributeType: S
        - AttributeName: SongTitle
          AttributeType: S
      BillingMode: PAY_PER_REQUEST
      KeySchema:
        - AttributeName: Artist
          KeyType: HASH
        - AttributeName: SongTitle
          KeyType: RANGE
      TableClass: STANDARD
      TableName: !Sub "${Prefix}-Music"
Code language: YAML (yaml)

No special settings are made.

The configuration shown in the following official AWS page is exported to a CloudFormation template.

Step 1: Create a table - Amazon DynamoDB
Get started with Amazon DynamoDB by creating a new table using the console or the AWS CLI.

For basic information on DynamoDB, please refer to the following pages.

ECS

Resources:
  Cluster:
    Type: AWS::ECS::Cluster
    Properties:
      ClusterName: !Sub "${Prefix}-cluster"
      
  FargateTaskExecutionRole:
    Type: AWS::IAM::Role
    DeletionPolicy: Delete
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - ecs-tasks.amazonaws.com
            Action:
              - sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy
        
  TaskRole:
    Type: AWS::IAM::Role
    DeletionPolicy: Delete
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - ecs-tasks.amazonaws.com
            Action:
              - sts:AssumeRole
      Policies:
        - PolicyName: AllowSQSSendMessage
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - dynamodb:Scan
                Resource:
                  - !Sub "arn:aws:dynamodb:${AWS::Region}:${AWS::AccountId}:table/${Table}"

  TaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      ContainerDefinitions:
        - Image: !Sub "${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com/${ECRRepositoryName}:latest"
          LogConfiguration:
            LogDriver: awslogs
            Options:
              awslogs-group: !Ref LogGroup
              awslogs-region: !Ref AWS::Region
              awslogs-stream-prefix: !Ref Prefix
          Name: !Sub "${Prefix}-bottle"
          PortMappings:
            - ContainerPort: !Ref BottlePort
              HostPort: !Ref BottlePort
          Environment:
            - Name: TABLE_NAME
              Value: !Ref Table
      Cpu: !Ref TaskCpu
      ExecutionRoleArn: !Ref FargateTaskExecutionRole
      Memory: !Ref TaskMemory
      NetworkMode: awsvpc
      RequiresCompatibilities:
        - FARGATE
      RuntimePlatform:
        CpuArchitecture: ARM64
        OperatingSystemFamily: LINUX
      TaskRoleArn: !Ref TaskRole
                
  Service:
    Type: AWS::ECS::Service
    Properties:
      Cluster: !Ref Cluster
      DesiredCount: 1
      LaunchType: FARGATE
      LoadBalancers:
        - ContainerName: !Sub "${Prefix}-bottle"
          ContainerPort: !Ref BottlePort
          TargetGroupArn: !Ref ALBTargetGroup
      NetworkConfiguration:
        AwsvpcConfiguration:
          SecurityGroups:
            - !Ref ContainerSecurityGroup
          Subnets:
            - !Ref PrivateSubnet1
            - !Ref PrivateSubnet2
      ServiceName: !Sub "${Prefix}-service"
      TaskDefinition: !Ref TaskDefinition
Code language: YAML (yaml)

There are two points.

The first point is the role for the task.
Inline policy to authorize scanning to DynamoDB.

The second point is the task definition.
The Environment property allows you to set environment variables that can be accessed in the container, set the name of the DynamoDB table.

(Reference) Using CloudFormation custom resources and CodeBuild to automatically push test images to the ECR repository when creating CloudFormation stacks

Resources:
  CodeBuildProject:
    Type: AWS::CodeBuild::Project
    Properties: 
      Artifacts:
        Type: NO_ARTIFACTS
      Cache: 
        Type: NO_CACHE
      Environment: 
        ComputeType: !Ref ProjectEnvironmentComputeType
        EnvironmentVariables:
          - Name: BOTTLE_PORT
            Type: PLAINTEXT
            Value: !Ref BottlePort
          - Name: DOCKERHUB_PASSWORD
            Type: SECRETS_MANAGER
            Value: !Sub "${Secret}:password"
          - Name: DOCKERHUB_USERNAME
            Type: SECRETS_MANAGER
            Value: !Sub "${Secret}:username"
        Image: !Ref ProjectEnvironmentImage
        ImagePullCredentialsType: CODEBUILD
        Type: !Ref ProjectEnvironmentType
        PrivilegedMode: true
      LogsConfig: 
        CloudWatchLogs:
          Status: DISABLED
        S3Logs:
          Status: DISABLED
      Name: !Ref Prefix
      ServiceRole: !GetAtt CodeBuildRole.Arn
      Source: 
        Type: NO_SOURCE
        BuildSpec: !Sub |
          version: 0.2
          
          phases:
            pre_build:
              commands:
                - echo Logging in to Amazon ECR...
                - aws --version
                - aws ecr get-login-password --region ${AWS::Region} | docker login --username AWS --password-stdin ${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com
                - REPOSITORY_URI=${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com/${ECRRepositoryName}
                - COMMIT_HASH=$(echo $CODEBUILD_RESOLVED_SOURCE_VERSION | cut -c 1-7)
                - IMAGE_TAG=${!COMMIT_HASH:=latest}
                - echo Logging in to Docker Hub...
                - echo $DOCKERHUB_PASSWORD | docker login -u $DOCKERHUB_USERNAME --password-stdin
                - |
                  cat << EOF > Dockerfile
                  FROM amazonlinux
                  RUN yum update -y && yum install python3 python3-pip -y
                  RUN pip3 install boto3 bottle
                  COPY main.py ./
                  CMD ["python3", "main.py"]
                  EXPOSE $BOTTLE_PORT
                  EOF
                - |
                  cat << EOF > main.py
                  from bottle import Bottle, route, run
                  import boto3
                  import json
                  import os
                  
                  TABLE_NAME = os.environ['TABLE_NAME']
                  
                  app = Bottle()
                  dynamodb_client = boto3.client('dynamodb')
                  
                  @app.route('/')
                  def scan():
                    response = dynamodb_client.scan(
                      TableName=TABLE_NAME
                    )
                    return json.dumps(response['Items'], indent=2)
    
                  if __name__ == '__main__':
                    run(app=app, host='0.0.0.0', port=$BOTTLE_PORT)
                  EOF
            build:
              commands:
                - echo Build started on `date`
                - echo Building the Docker image...
                - docker build -t $REPOSITORY_URI:latest .
                - docker tag $REPOSITORY_URI:latest $REPOSITORY_URI:$IMAGE_TAG
            post_build:
              commands:
                - echo Build completed on `date`
                - echo Pushing the Docker images...
                - docker push $REPOSITORY_URI:latest
                - docker push $REPOSITORY_URI:$IMAGE_TAG
      Visibility: PRIVATE
Code language: YAML (yaml)

Images for containers to run on ECS can be manually pushed to the ECR repository.
However, this page uses CodeBuild to automatically build the image and push it to the ECR repository.
The trigger to start the build is a Lambda function tied to a CloudFormation custom resource.
For more information, please see the following page.

The key points are the two commands in buildspec.yml that are executed during the pre_build phase.

The first point is where the cat command creates a Dockerfile.
The content is based on the latest version of the Amazon Linux image, and after installing Python and packages, the script described below is executed.
The environment variables specified in the EnvironmentVariables property are used to specify the ports to be published.

The second point is where the Python script main.py is created, also with the cat command.
This script uses the web framework bottle to act as a simple web server.
When the root page is accessed, it scans the DynamoDB table, retrieves all items, and returns them as a response.
The name of the table to be accessed is referenced in an environment variable.

(Reference) Initial setup of DynamoDB with CloudFormation custom resources

Resources:
  Function2:
    Type: AWS::Lambda::Function
    Properties:
      Environment:
        Variables:
          JSON_S3_BUCKET: !Ref JsonS3Bucket
          JSON_S3_KEY: !Ref JsonS3Key
          TABLE_NAME: !Ref Table
      Code:
        ZipFile: |
          import boto3
          import cfnresponse
          import json
          import os
          
          JSON_S3_BUCKET = os.environ['JSON_S3_BUCKET']
          JSON_S3_KEY = os.environ['JSON_S3_KEY']
          TABLE_NAME = os.environ['TABLE_NAME']
          
          CREATE = 'Create'
          response_data = {}
          
          s3_client = boto3.client('s3')
          dynamodb_client = boto3.client('dynamodb')
          
          def lambda_handler(event, context):
            try:
              if event['RequestType'] == CREATE:
                s3_response = s3_client.get_object(
                  Bucket=JSON_S3_BUCKET,
                  Key=JSON_S3_KEY)
                  
                body = s3_response['Body'].read()
                print(body)
                
                json_data = json.loads(body.decode('utf-8'))
                print(json_data)
                
                for item in json_data:
                  dynamodb_response = dynamodb_client.put_item(
                    TableName=TABLE_NAME,
                    Item=item
                  )
                  print(dynamodb_response)
                
              cfnresponse.send(event, context, cfnresponse.SUCCESS, response_data)
                  
            except Exception as e:
              print(e)
              cfnresponse.send(event, context, cfnresponse.FAILED, response_data)
      FunctionName: !Sub "${Prefix}-function2"
      Handler: !Ref Handler
      Runtime: !Ref Runtime
      Role: !GetAtt FunctionRole2.Arn
      Timeout: !Ref Timeout
Code language: YAML (yaml)

Automatically store test data in DynamoDB when creating CloudFormation stacks.
This is accomplished by associating the above Lambda function with a CloudFormation custom resource.

For details, please refer to the following page.

Summary of ALB and ECS port numbers

The communication in this configuration is organized as shown in the figure below.

Diagram of creating Web app using DynamoDB with ECS(Fargate) 2

Communication from the user to the ALB is directed to port 80, but communication from the ALB to the ECS (Fargate) is directed to the port used by the bottle application (8080).

To achieve the above configuration, set each property of each resource as follows

  • AWS::ElasticLoadBalancingV2::Listener Port:80
  • AWS::ElasticLoadBalancingV2::TargetGroup Port:8080
  • AWS::ECS::Service LoadBalancers ContainerPort:8080
  • AWS::ECS::TaskDefinition PortMappings ContainerPort:8080
  • AWS::ECS::TaskDefinition PortMappings HostPort:8080

The security groups attached to both resources are as follows

Resources:
  ALBSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupName: !Sub ${Prefix}-ALBSecurityGroup
      GroupDescription: Allow HTTP Only.
      VpcId: !Ref VPC
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: !Ref HTTPPort
          ToPort: !Ref HTTPPort
          CidrIp: 0.0.0.0/0
          
  ContainerSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupName: !Sub "${Prefix}-ContainerSecurityGroup"
      GroupDescription: Allow HTTP from ALBSecurityGroup.
      VpcId: !Ref VPC
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: !Ref BottlePort
          ToPort: !Ref BottlePort
          SourceSecurityGroupId: !Ref ALBSecurityGroup
Code language: YAML (yaml)

Architecting

We will use CloudFormation to build this environment and check its actual behavior.

Create CloudFormation stacks and check the resources in the stacks

Create CloudFormation stacks.
For information on how to create stacks and check each stack, please see the following page.

After reviewing the resources in each stack, information on the main resources created in this case is as follows

  • ECR Repository: dva-02-004
  • DynamoDB table: dva-02-004-Music
  • ECS cluster: dva-02-004-cluster
  • ALB’s domain: dva-02-004-alb-909547272.ap-northeast-1.elb.amazonaws.com

Check each resource from the AWS Management Console.

Check the ECR repository.

Detail of ECR 1

The repository is successfully created and the image is pushed.
This means that the Lambda function associated with the CloudFormation custom resource has been executed and CodeBuild has started building the image and pushed it to the repository.

Check the DynamoDB table.

Detail of DynamoDB 1

The data is stored in a table.
This means that the Lambda function associated with the CloudFormation custom resource has been executed, the JSON data stored in the S3 bucket has been read, and the item has been saved to the DynamoDB table.

Check the ECS.

Check the clusters first.

Detail of Fargate 1

One ECS service is created in the cluster.

Check the tasks in the service.

Detail of Fargate 2

It can be seen that they are related to the ALB.

You can also see one task running in the service.

Detail of Fargate 3

Within the task, an image is pulled from the ECR repository mentioned earlier and a container is created.

Operation Check

Now that you are ready, access the ALB.

% curl http://dva-02-004-alb-909547272.ap-northeast-1.elb.amazonaws.com/
[ { "AlbumTitle": { "S": "Somewhat Famous" }, "Awards": { "N": "1" }, "Artist": { "S": "No One You Know" }, "SongTitle": { "S": "Call Me Today" } }, { "AlbumTitle": { "S": "Somewhat Famous" }, "Awards": { "N": "2" }, "Artist": { "S": "No One You Know" }, "SongTitle": { "S": "Howdy" } }, { "AlbumTitle": { "S": "Songs About Life" }, "Awards": { "N": "10" }, "Artist": { "S": "Acme Band" }, "SongTitle": { "S": "Happy Day" } }, { "AlbumTitle": { "S": "Another Album Title" }, "Awards": { "N": "8" }, "Artist": { "S": "Acme Band" }, "SongTitle": { "S": "PartiQL Rocks" } } ]Code language: Bash (bash)

Accessing the ALB, a response was successfully received.
As a response, the item information stored in DynamoDB was returned.
I was able to access DynamoDB from ECS (Fargate).

Summary

We have created an ECS (Fargate) container application that accesses DynamoDB.

タイトルとURLをコピーしました