AWS_EN

Define action in CodePipeline calls Lambda function to change desired number of Fargate tasks

スポンサーリンク
Define action in CodePipeline calls Lambda function to change desired number of Fargate tasks. AWS_EN
スポンサーリンク
スポンサーリンク

Define action in CodePipeline calls Lambda function to change desired number of Fargate tasks

CodePipeline can be configured with a variety of actions, but this time we will consider calling a Lambda function within the pipeline.

This time, we will call a Lambda function in the pipeline to change the desired number of ECS tasks in the ECS (Fargate) service.

Environment

Diagram of define action in CodePipeline calls Lambda function to change desired number of Fargate tasks.

We will configure CodePipeline to link three resources.

The first is CodeCommit.
CodeCommit is responsible for the source stage of CodePipeline.
It is used as a Git repository.

The second is CodeBuild.
CodeBuild is in charge of the build stage of CodePipeline.
It builds a Docker image from code pushed to CodeCommit.
The built image is pushed to ECR.

The third is the Lambda function.
The function’s job is to change the desired number of tasks for the ECS (Fargate) service.
Specifically, it changes the desired number from 0 to 1.

Create a deploy stage in CodePipeline.
Configure it to deploy to Fargate as described below.

Save your DockerHub account information in the SSM parameter store.
These will be used to pull the base image when generating images with DockerBuild, after signing in to DockerHub.

The trigger for CodePipeline to be started is conditional on a push to CodeCommit.
Specifically, we will have a rule in EventBridge that satisfies the above.

Create a Fargate type ECS on a private subnet.

Create an EC2 instance.
Use it as a client to access containers created on Fargate.

CloudFormation template files

Build the above configuration with CloudFormation.
The CloudFormation templates are located at the following URL

awstut-fa/079 at main · awstut-an-r/awstut-fa
Contribute to awstut-an-r/awstut-fa development by creating an account on GitHub.

Explanation of key points of the template files

This page focuses on how to call Lambda functions within CodePipeline.

For basic information about CodePipeline, please refer to the following page

For information on how to create a deployment stage in CodePipeline and deploy to ECS (Fargate), please refer to the following page

For information on how to build Fargate on a private subnet, please refer to the following page

Use CloudFormation custom resources to automatically delete objects in S3 buckets and images in ECR repositories when deleting the CloudFormation stack.
For more information, please see the following page

Explanation of key points of template files

CodePipeline

Resources: Pipeline: Type: AWS::CodePipeline::Pipeline Properties: ArtifactStore: Location: !Ref BucketName Type: S3 Name: !Ref Prefix RoleArn: !GetAtt CodePipelineRole.Arn Stages: - Actions: - ActionTypeId: Category: Source Owner: AWS Provider: CodeCommit Version: 1 Configuration: BranchName: !Ref BranchName OutputArtifactFormat: CODE_ZIP PollForSourceChanges: false RepositoryName: !GetAtt CodeCommitRepository.Name Name: SourceAction OutputArtifacts: - Name: !Ref PipelineSourceArtifact Region: !Ref AWS::Region RunOrder: 1 Name: Source - Actions: - ActionTypeId: Category: Build Owner: AWS Provider: CodeBuild Version: 1 Configuration: ProjectName: !Ref CodeBuildProject InputArtifacts: - Name: !Ref PipelineSourceArtifact Name: Build OutputArtifacts: - Name: !Ref PipelineBuildArtifact Region: !Ref AWS::Region RunOrder: 1 Name: Build - Actions: - ActionTypeId: Category: Deploy Owner: AWS Provider: ECS Version: 1 Configuration: ClusterName: !Ref ECSClusterName FileName: !Ref ImageDefinitionFileName ServiceName: !Ref ECSServiceName InputArtifacts: - Name: !Ref PipelineBuildArtifact Name: Deploy Region: !Ref AWS::Region RunOrder: 1 Name: Deploy - Actions: - ActionTypeId: Category: Invoke Owner: AWS Provider: Lambda Version: 1 Configuration: FunctionName: !Ref ECSFunctionName InputArtifacts: [] Name: Invoke OutputArtifacts: [] Region: !Ref AWS::Region RunOrder: 1 Name: Invoke
Code language: YAML (yaml)

Define the stages for invoking Lambda functions in the Stages property.
Set the function to be invoked in the Configuration property. Specifically, specify the name of the function to be invoked in the FunctionName property.
Do not set valid artifacts in the InputArtifacts and OutputArtifacts properties. This is because the function to be called this time is to adjust the desired number of ECS tasks, so no reading or writing of artifacts will occur.

Here are the IAM roles for CodePipeline

Resources: CodePipelineRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Principal: Service: - codepipeline.amazonaws.com Action: - sts:AssumeRole Policies: - PolicyName: PipelinePolicy PolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: - lambda:invokeFunction Resource: - !Sub "arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${ECSFunctionName}" - Effect: Allow Action: - codecommit:CancelUploadArchive - codecommit:GetBranch - codecommit:GetCommit - codecommit:GetRepository - codecommit:GetUploadArchiveStatus - codecommit:UploadArchive Resource: - !GetAtt CodeCommitRepository.Arn - Effect: Allow Action: - codebuild:BatchGetBuilds - codebuild:StartBuild - codebuild:BatchGetBuildBatches - codebuild:StartBuildBatch Resource: - !GetAtt CodeBuildProject.Arn - Effect: Allow Action: - s3:PutObject - s3:GetObject - s3:GetObjectVersion - s3:GetBucketAcl - s3:GetBucketLocation Resource: - !Sub "arn:aws:s3:::${BucketName}" - !Sub "arn:aws:s3:::${BucketName}/*" - Effect: Allow Action: - ecs:* Resource: "*" - Effect: Allow Action: - iam:PassRole Resource: "*" Condition: StringLike: iam:PassedToService: - ecs-tasks.amazonaws.com
Code language: YAML (yaml)

Set permissions to call Lambda functions.

Lambda Function

Resources: ECSFunction: Type: AWS::Lambda::Function Properties: Code: ZipFile: | import boto3 import os cluster_name = os.environ['CLUSTER_NAME'] count = int(os.environ['COUNT']) service_name = os.environ['SERVICE_NAME'] ecs_client = boto3.client('ecs') codepipeline_client = boto3.client('codepipeline') def lambda_handler(event, context): job_id = event['CodePipeline.job']['id'] try: describe_services_response = ecs_client.describe_services( cluster=cluster_name, services=[ service_name ] ) print(describe_services_response) if describe_services_response['services'][0]['desiredCount'] > 0: codepipeline_client.put_job_success_result( jobId=job_id ) return update_service_response = ecs_client.update_service( cluster=cluster_name, service=service_name, desiredCount=count ) print(update_service_response) codepipeline_client.put_job_success_result( jobId=job_id ) except Exception as e: print(e) codepipeline_client.put_job_failure_result( jobId=job_id, failureDetails={ 'type': 'JobFailed', 'message': 'Something happened.' } ) Environment: Variables: CLUSTER_NAME: !Ref ECSClusterName COUNT: 1 SERVICE_NAME: !Ref ECSServiceName FunctionName: !Sub "${Prefix}-function-ecs" Handler: !Ref Handler Runtime: !Ref Runtime Role: !GetAtt ECSFunctionRole.Arn
Code language: YAML (yaml)

Define the code to be executed by the Lambda function in inline notation.
For more information, please refer to the following page

The Environment property allows you to define environment variables that can be passed to the function.
Specifically, the ECS cluster and service for which the number of ECS tasks will be adjusted, and the adjusted value.

The code is as follows

  • Use the describe_services method to get the status of the ECS services and check the desired number.
  • Check the number of tasks, and if the desired number is 0, continue the process.
  • Use the update_service method to change the desired number of ECS services.
  • Call the API for CodePipeline.

The last API is the key point.
The official AWS page mentions the following

As part of the implementation of the Lambda function, there must be a call to either the PutJobSuccessResult API or PutJobFailureResult API. Otherwise, the execution of this action hangs until the action times out.

AWS Lambda

This means that after changing the number of requests, one of the above two APIs must be executed.
In this case, we will execute the PutJobSuccessResult API (put_job_success_result method) if the desired number of jobs was successfully changed, and the PutJobFailureResult API (put_job_failure_ result) if any error occurs during processing.

(Reference) Application Container

Dockerfile

FROM amazonlinux RUN yum update -y && yum install python3 python3-pip -y RUN pip3 install bottle COPY main.py ./ CMD ["python3", "main.py"] EXPOSE 8080
Code language: Dockerfile (dockerfile)

The image for the app container will be based on Amazon Linux 2.

We will use Bottle, a Python web framework.
So after installing Python and pip, install this.

Copy the Python script (main.py) describing the app logic and set this to run.

As mentioned earlier, the app listens for HTTP requests on 8080/tcp, so expose this port.

main.py

from bottle import route, run @route('/') def hello(): return 'Hello CodePipeline.' if __name__ == '__main__': run(host='0.0.0.0', port=8080)
Code language: Python (python)

We will use Bottle to build a simple web server.
The simple configuration is to listen for HTTP requests at 8080/tcp and return “Hello CodePipeline”.

Architecting

Use CloudFormation to build this environment and check the actual behavior.

Create CloudFormation stacks and check resources in stacks

Create a CloudFormation stack.
For information on how to create stacks and check each stack, please refer to the following page

After checking the resources in each stack, information on the main resources created this time is as follows

  • ECR: fa-079
  • CodeCommit: fa-079
  • CodeBuild: fa-079
  • CodePipeline: fa-079
  • Lambda function: fa-079-function-ecs

Check the created resources from the AWS Management Console.
Check the ECS (Fargate) cluster service.

Detail of ECS 1.
Detail of ECS 2.

The cluster service has been successfully created.
The point is that the desired number of tasks is 0.
The point is that the number of desired tasks is zero, which means that during the initial Fargate build, not a single task is launched.

Check CodePipeline.

Detail of ECS 1.
Detail of ECS 2.

正常にクラスター・サービスが作成されています。
ポイントは希望数が0となっている点です。
つまりFargateの初期構築時は、1つもタスクが起動しないという点です。

CodePipelineを確認します。

Detail of CodePipeline 1.

The pipeline is failing to execute.
This is because the pipeline was triggered by the creation of CodeCommit when the CloudFormation stack was created.
Since we are not pushing code to CodeCommit at this time, an error occurred during the pipeline execution process.

Note the stage that is being created.
The stage named Invoke, which calls the Lambda function, is prepared at the end of the pipeline.
This means that the code pushed by CodeCommit will be used to build a Docker image, deploy the image to Fargate, and then change the desired number of Fargate tasks.

Check Action

Now that we are ready, we push the code to CodeCommit.

First, pull CodeCommit.

$ git clone https://git-codecommit.ap-northeast-1.amazonaws.com/v1/repos/fa-079 Cloning into 'fa-079'... warning: You appear to have cloned an empty repository.
Code language: Bash (bash)

An empty repository has been pulled.

Add the Dockerfile and main.py to the repository.

$ ls -al total 8 drwxrwxr-x 3 ec2-user ec2-user 51 Aug 20 08:31 . drwxrwxr-x 3 ec2-user ec2-user 20 Aug 20 08:31 .. -rw-rw-r-- 1 ec2-user ec2-user 187 Aug 12 11:01 Dockerfile drwxrwxr-x 7 ec2-user ec2-user 119 Aug 20 08:31 .git -rw-rw-r-- 1 ec2-user ec2-user 681 Aug 20 02:57 main.py
Code language: Bash (bash)

Push the two files to CodeCommit.

$ git add . $ git commit -m "first commit" [master (root-commit) 7e41437] first commit ... 2 files changed, 39 insertions(+) create mode 100644 Dockerfile create mode 100644 main.py $ git push ... * [new branch] master -> master
Code language: Bash (bash)

The push was successful.

After waiting for a while, check CodePipeline again.

Detail of CodePipeline 2.

The pipeline has completed successfully.

Check Fargate.

Detail of ECS 3.
Detail of ECS 4.

The desired number is now 1.
This means that the Lambda function has been invoked and the desired number has been changed.
Since the desired number is now 1, an ECS task is automatically generated.
Looking at the details of this task, we can see that the assigned private address is “10.0.2.234”.

It accesses the EC2 instance in order to make an HTTP request to the container.
Use SSM Session Manager to access the instance.

% aws ssm start-session --target i-0c41c5926230b480c Starting session with SessionId: root-0c76e08548a26fec6 sh-4.2$
Code language: Bash (bash)

For more information on SSM Session Manager, please refer to the following page

Use the curl command to access the container in the task.

sh-4.2$ curl http://10.0.2.224:8080/ Hello CodePipeline.
Code language: Bash (bash)

The container responded.
It is indeed the string we set up in the Bottle app.
This shows that the number of ECS tasks can be changed by calling the Lambda function within CodePipeline.

Summary

We were able to change the desired number of ECS tasks in the ECS (Fargate) service by invoking a Lambda function in the pipeline.

タイトルとURLをコピーしました