Use CodePipeline to trigger CodeCommit pushes to push images to ECR
Using CodePipeline, you can build a CI/CD configuration.
In this example, we will configure a pipeline to build a Docker image with CodeBuild and push it to the ECR when code is pushed to CodeCommit.
Environment
We will configure CodePipeline to link the two resources.
We will define two stages: source and build.
The first is CodeCommit.
This is used as a Git repository.
The second is CodeBuild.
Build a Docker image from code pushed to CodeCommit.
Push the built image to ECR.
Create an S3 bucket.
Use it to store artifacts generated by CodePipeline.
Store DockerHub account information in the SSM parameter store.
Use these to sign in to DockerHub and pull base images when generating images with DockerBuild.
The trigger for CodePipeline to be started is conditional on a push to CodeCommit.
Specifically, we will have a rule in EventBridge that satisfies the above.
CloudFormation template files
The above configuration is built using CloudFormation.
The following URL contains CloudFormation templates, etc.
https://github.com/awstut-an-r/awstut-fa/tree/main/075
Explanation of key points of the template files
S3 Bucket
Resources:
Bucket:
Type: AWS::S3::Bucket
Properties:
BucketName: !Ref Prefix
AccessControl: Private
Code language: YAML (yaml)Bucket to store artifacts generated by CodePipeline.
No special configuration is required.
SSM Parameter Store
Resources:
SSMParameterDockerHubPassword:
Type: AWS::SSM::Parameter
Properties:
Name: !Sub "${Prefix}-DockerHubPassword"
Type: String
Value: !Ref DockerHubPassword
SSMParameterDockerHubUsername:
Type: AWS::SSM::Parameter
Properties:
Name: !Sub "${Prefix}-DockerHubUsername"
Type: String
Value: !Ref DockerHubUsername
Code language: YAML (yaml)Account information for signing in to DockerHub.
Register username and password as parameters.
ECR
Resources:
ECRRepository:
Type: AWS::ECR::Repository
Properties:
RepositoryName: !Ref Prefix
Code language: YAML (yaml)Repository to store home-made repository images.
No special configuration is required.
CodeCommit
Resources:
CodeCommitRepository:
Type: AWS::CodeCommit::Repository
Properties:
RepositoryName: !Ref Prefix
Code language: YAML (yaml)Code repository.
No special configuration is required.
CodeBuild
Resources:
CodeBuildProject:
Type: AWS::CodeBuild::Project
Properties:
Artifacts:
Type: CODEPIPELINE
Cache:
Type: NO_CACHE
Environment:
ComputeType: !Ref ProjectEnvironmentComputeType
EnvironmentVariables:
- Name: DOCKERHUB_PASSWORD
Type: PARAMETER_STORE
Value: !Ref SSMParameterDockerHubPassword
- Name: DOCKERHUB_USERNAME
Type: PARAMETER_STORE
Value: !Ref SSMParameterDockerHubUsername
Image: !Ref ProjectEnvironmentImage
ImagePullCredentialsType: CODEBUILD
Type: !Ref ProjectEnvironmentType
PrivilegedMode: true
LogsConfig:
CloudWatchLogs:
Status: DISABLED
S3Logs:
Status: DISABLED
Name: !Ref Prefix
ServiceRole: !GetAtt CodeBuildRole.Arn
Source:
Type: CODEPIPELINE
BuildSpec: !Sub |
version: 0.2
phases:
pre_build:
commands:
- echo Logging in to Amazon ECR...
- aws --version
- aws ecr get-login-password --region ${AWS::Region} | docker login --username AWS --password-stdin ${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com
- REPOSITORY_URI=${AWS::AccountId}.dkr.ecr.${AWS::Region}.amazonaws.com/${ECRRepositoryName}
- COMMIT_HASH=$(echo $CODEBUILD_RESOLVED_SOURCE_VERSION | cut -c 1-7)
- IMAGE_TAG=${!COMMIT_HASH:=latest}
- echo Logging in to Docker Hub...
- echo $DOCKERHUB_PASSWORD | docker login -u $DOCKERHUB_USERNAME --password-stdin
build:
commands:
- echo Build started on `date`
- echo Building the Docker image...
- docker build -t $REPOSITORY_URI:latest .
- docker tag $REPOSITORY_URI:latest $REPOSITORY_URI:$IMAGE_TAG
post_build:
commands:
- echo Build completed on `date`
- echo Pushing the Docker images...
- docker push $REPOSITORY_URI:latest
- docker push $REPOSITORY_URI:$IMAGE_TAG
Visibility: PRIVATE
Code language: YAML (yaml)We will cover the key parameters.
The Artifacts property sets the settings related to the artifacts generated by CodeBuild.
The internal Type property sets the type of artifact.
In this case, we will specify “CODEPIPELINE” because CodeBuild will be executed within CodePipeline, which will be described later.
Define the build environment in the Environment property.
Set the resources to be allocated to the build environment in the ComputeType property. In this case, specify “BUILD_GENERAL1_SMALL,” which means the minimum environment, and allocate 3 GB of memory and 2 vCPU resources.
The EnvironmentVariables property allows you to define environment variables for the build environment. In this case, the values of the two SSM parameter stores mentioned above will be set as environment variables. Specify the parameter name of SSM parameter store.
In the Image property, set the Docker image to be used as the build environment. In this case, specify “aws/codebuild/amazonlinux2-aarch64-standard:2.0” to use the ARM version of Amazon Linux 2 image.
In the ImagePullCredentialsType property, set the credential type to be used to pull the image for the build environment. To use the aforementioned image for CodeBuild, specify “CODEBUILD”.
The PrivilegedMode property should be set to true to allow the Docker daemon to run inside the Docker container. This is required to build the Docker image.
Set the type of build environment in the Type property. In this case, specify “ARM_CONTAINER” to use the ARM version container.
Set the source code to be used in the build and the build specification file (buildspec.yml) in the Source property.
Set the type of source code repository in the Type property. In this case, we will use CodeBuild in CodePipeline, so we will specify “CODEPIPELINE”.
The BuildSpec property is a parameter related to buildspec.yml. In this case, the content of buildspec.yml is directly described in this property. The contents are described with reference to the official AWS page.
https://docs.aws.amazon.com/codepipeline/latest/userguide/ecs-cd-pipeline.html#code-build-perms
There is one change. When building an image, the base image is pulled from DockerHub, but we added a process to sign in to DockerHub. If you try to pull an image from DockerHub without signing in, the pull may fail. The following page is a reference for the description.
https://dev.classmethod.jp/articles/codebuild-has-to-use-dockerhub-login-to-avoid-ip-gacha/
Check the IAM role for CodeBuild.
Resources:
CodeBuildRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- codebuild.amazonaws.com
Action:
- sts:AssumeRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryPowerUser
Policies:
- PolicyName: PipelineExecutionPolicy
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- ssm:GetParameters
Resource:
- !Sub "arn:aws:ssm:${AWS::Region}:${AWS::AccountId}:parameter/${SSMParameterDockerHubPassword}"
- !Sub "arn:aws:ssm:${AWS::Region}:${AWS::AccountId}:parameter/${SSMParameterDockerHubUsername}"
- Effect: Allow
Action:
- s3:PutObject
- s3:GetObject
- s3:GetObjectVersion
- s3:GetBucketAcl
- s3:GetBucketLocation
Resource:
- !Sub "arn:aws:s3:::${BucketName}"
- !Sub "arn:aws:s3:::${BucketName}/*"
Code language: YAML (yaml)Attach the AWS admin policy AmazonEC2ContainerRegistryPowerUser to push the image to ECR.
When setting environment variables, allow them to retrieve parameters from the SSM parameter store.
Allow to retrieve CodePipeline artifacts stored in S3 buckets when building images.
CodePipeline
Resources:
Pipeline:
Type: AWS::CodePipeline::Pipeline
Properties:
ArtifactStore:
Location: !Ref BucketName
Type: S3
Name: !Ref Prefix
RoleArn: !GetAtt CodePipelineRole.Arn
Stages:
- Actions:
- ActionTypeId:
Category: Source
Owner: AWS
Provider: CodeCommit
Version: 1
Configuration:
BranchName: !Ref BranchName
OutputArtifactFormat: CODE_ZIP
PollForSourceChanges: false
RepositoryName: !GetAtt CodeCommitRepository.Name
Name: SourceAction
OutputArtifacts:
- Name: !Ref PipelineSourceArtifact
Region: !Ref AWS::Region
RunOrder: 1
Name: Source
- Actions:
- ActionTypeId:
Category: Build
Owner: AWS
Provider: CodeBuild
Version: 1
Configuration:
ProjectName: !Ref CodeBuildProject
InputArtifacts:
- Name: !Ref PipelineSourceArtifact
Name: Build
OutputArtifacts:
- Name: !Ref PipelineBuildArtifact
Region: !Ref AWS::Region
RunOrder: 1
Name: Build
Code language: YAML (yaml)We will cover the key parameters.
In the ArtifactStore property, specify the S3 bucket that will store the artifacts generated while the pipeline is running.
In the Stages property, define the stages that make up the pipeline.
In this case, we will define two stages.
The first stage is the source stage.
Refer to the following official AWS page to set it up.
https://docs.aws.amazon.com/codepipeline/latest/userguide/action-reference-CodeCommit.html
In the ActionTypeId property, set CodeCommit to be the source of the pipeline.
In the Configuration property, set the parameters to treat CodeCommit as the source, and in the RepositoryName and BranchName properties, specify the CodeCommit repository name and branch name to be referenced. The OutputArtifactFormat property specifies the default “CODE_ZIP” The PollForSourceChanges property is the parameter that triggers the start of the pipeline. If it is set to true, CodePipeline will poll CodeCommit to detect activities that should start the pipeline. In this case, this parameter is set to false because the activity will be detected using EventBridge, which will be described later.
The OutputArtifacts property is a parameter related to the artifact names generated in the source stage.
The second stage is the build stage.
Refer to the following official AWS page to set it up.
https://docs.aws.amazon.com/codepipeline/latest/userguide/action-reference-CodeBuild.html
The ActionTypeId property sets CodeBuild to perform the build in the pipeline.
Configuration property is a parameter to build using CodeBuild; specify the CodeBuild to be associated with the pipeline in the ProjectName property.
In the InputArtifacts property, specify the names of the artifacts to be used in the build. The point is that the value must be the same as the value of the OutputArtifacts property specified in the source stage. This is because the artifacts generated in the source stage will be used for the build.
The OutputArtifacts property is a parameter related to the artifact names generated in the build stage.
Check the IAM role for CodePipeline.
Resources:
CodePipelineRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- codepipeline.amazonaws.com
Action:
- sts:AssumeRole
Policies:
- PolicyName: PipelinePolicy
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- codecommit:CancelUploadArchive
- codecommit:GetBranch
- codecommit:GetCommit
- codecommit:GetRepository
- codecommit:GetUploadArchiveStatus
- codecommit:UploadArchive
Resource:
- !GetAtt CodeCommitRepository.Arn
- Effect: Allow
Action:
- codebuild:BatchGetBuilds
- codebuild:StartBuild
- codebuild:BatchGetBuildBatches
- codebuild:StartBuildBatch
Resource:
- !GetAtt CodeBuildProject.Arn
- Effect: Allow
Action:
- s3:PutObject
- s3:GetObject
- s3:GetObjectVersion
- s3:GetBucketAcl
- s3:GetBucketLocation
Resource:
- !Sub "arn:aws:s3:::${BucketName}"
- !Sub "arn:aws:s3:::${BucketName}/*"
Code language: YAML (yaml)To pull source code from CodeCommit in the source stage, grant permissions to allow this.
To build using CodeBuild in the build stage, grant permissions to execute the build.
Grant permission to retrieve and save artifacts on the S3 bucket during pipeline execution.\
EventBridge
Resources:
EventsRule:
Type: AWS::Events::Rule
Properties:
EventPattern:
source:
- aws.codecommit
detail-type:
- CodeCommit Repository State Change
resources:
- !GetAtt CodeCommitRepository.Arn
detail:
event:
- referenceCreated
- referenceUpdated
referenceType:
- branch
referenceName:
- !Ref BranchName
Name: !Ref Prefix
Targets:
- Arn: !Sub "arn:aws:codepipeline:${AWS::Region}:${AWS::AccountId}:${Pipeline}"
Id: !Sub "${Prefix}-CodePipeline-CodeCommit"
RoleArn: !GetAtt EventsRuleRole.Arn
Code language: YAML (yaml)Use EventBridge as the trigger for CodePipeline to start.
Refer to the following official AWS page to set it up.
Specify the event to be detected with the EventPattern property. In this case, we will set it to detect when a branch is created or updated in CodeCommit.
Specify the resource to be notified with the Targets property. In this case, we will specify CodePipeline to start the pipeline.
Confirm the IAM role for EventBridge.
Resources:
EventsRuleRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- events.amazonaws.com
Action:
- sts:AssumeRole
Policies:
- PolicyName: PipelineExecutionPolicy
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- codepipeline:StartPipelineExecution
Resource:
- !Sub "arn:aws:codepipeline:${AWS::Region}:${AWS::AccountId}:${Pipeline}"
Code language: YAML (yaml)Grant settings to allow CodePipeline pipeline execution.
App Container
Dockerfile
FROM amazonlinux
RUN yum update -y && yum install python3 python3-pip -y
RUN pip3 install bottle
COPY main.py ./
CMD ["python3", "main.py"]
EXPOSE 8080
Code language: Dockerfile (dockerfile)The image for the app container will be based on Amazon Linux 2.
We will use Bottle, a Python web framework.
So after installing Python and pip, install this.
Copy the Python script (main.py) describing the app logic and set this to run.
As mentioned earlier, the app listens for HTTP requests on tcp/8080, so expose this port.
main.py
from bottle import route, run
@route('/')
def hello():
return 'Hello CodePipeline!'
if __name__ == '__main__':
run(host='0.0.0.0', port=8080)
Code language: Python (python)Use Bottle to build a simple web server.
The simple configuration is to listen for HTTP requests at 8080/tcp and return “Hello CodePipeline!”.
Architecting
We will use CloudFormation to build this environment and check the actual behavior.
Create CloudFormation stacks and check resources in stacks
Create a CloudFormation stacks.
For information on how to create stacks and check each stack, please refer to the following page
After checking the resources in each stack, information on the main resources created this time is as follows
- ECR: fa-075
- CodeCommit: fa-075
- CodeBuild project: fa-075
- CodePipeline: fa-075
- S3 bucket: fa-075
- SSM parameter parameter name 1: fa-075-DcokerHubUsername
- SSM parameter parameter name2: fa-075-DcokerHubPassword
Confirm the created resource from the AWS Management Console.
Check the ECR.
Empty.
The image will be pushed here when the pipeline is executed.
Check CodeCommit.
This is also empty.
By pushing code here, the pipeline will be executed.
Check CodeBuild.
CodeBuild has been created as specified in the CloudFormation template.
Check CodePipeline.
The pipeline has failed to execute.
This is because the pipeline was triggered by the creation of CodeCommit and its internal branch when the CloudFormation stack was created.
Since we are not pushing any code to CodeCommit at this time, an error occurred during the pipeline execution process.
Checking Action
Now that we are ready, we push the code to CodeCommit.
First, pull CodeCommit.
$ git clone https://git-codecommit.ap-northeast-1.amazonaws.com/v1/repos/fa-075
Cloning into 'fa-075'...
warning: You appear to have cloned an empty repository.
Code language: Bash (bash)An empty repository has been pulled.
Add a Dockerfile and main.py to the repository.
$ ls -al fa-075
total 8
drwxrwxr-x 3 ec2-user ec2-user 51 Aug 11 23:53 .
drwxrwxr-x 3 ec2-user ec2-user 20 Aug 11 23:53 ..
-rw-rw-r-- 1 ec2-user ec2-user 186 Aug 11 07:02 Dockerfile
drwxrwxr-x 7 ec2-user ec2-user 119 Aug 11 23:52 .git
-rw-rw-r-- 1 ec2-user ec2-user 630 Aug 11 07:42 main.py
Code language: Bash (bash)Push the two files to CodeCommit.
$ git add .
$ git commit -m "first commit"
[master (root-commit) 2596718] first commit
...
2 files changed, 37 insertions(+)
create mode 100644 Dockerfile
create mode 100644 main.py
$ git push
...
* [new branch] master -> masterCode language: JavaScript (javascript)The push was successful.
Check CodeCommit again.
Two files have indeed been pushed.
CodePipeline has started running.
Wait for a while.
The pipeline has successfully completed execution.
Check the execution history of CodeBuild.
The build has indeed been successfully executed.
Check the ECR.
The image has been pushed.
This means that the image built by CodeBuild has been pushed to the ECR.
In this way, by configuring a pipeline with CodePipeline, you can use the code pushed to CodeCommit as the source, build the Docker image with CodeBuild, and push the image to the ECR.
Summary
We have seen how CodePipeline can be used to configure a pipeline to build a Docker image in CodeBuild and push it to ECR when code is pushed to CodeCommit.