Setting up Test Units in CodePipeline

Setting up Test Units in CodePipeline AWS_EN

Setting up Test Units in CodePipeline

Test units can be added to CodePipeline.
Test units can be created with CodeBuild.

In this case, we will use CodePipeline to build a pipeline to create a Docker image and push it to an ECR repository.
Our goal is to add test units to the process.


Diagram of setting up Test Units in CodePipeline

We will configure CodePipeline to link three resources.

The first is CodeCommit.
CodeCommit is responsible for the source stage of CodePipeline.
It is used as a Git repository.

The second is CodeBuild.
CodeBuild is in charge of the test stage of CodePipeline.
This time, the test target will be a Python script, and the tool used for testing will be pytest.

The third is also a CodeBuild.
This CodeBuild is in charge of the build stage of CodePipeline.
It builds a Docker image from the code pushed to CodeCommit.
Push the built image to ECR.

Stores DockerHub account information in the SSM parameter store.
These will be used to pull the base image when generating the image with DockerBuild, after signing in to DockerHub.

The trigger for CodePipeline to be started is conditional on a push to CodeCommit.
Specifically, we will have a rule in EventBridge that satisfies the above.

CloudFormation templates files

The above configuration is built using CloudFormation.
The CloudFormation templates are located at the following URL

Explanation of key points of the template files

The purpose of this page is to add test units in the pipeline created by CodePipeline.

For basic information about CodePipeline, please refer to the following page.

Use CloudFormation custom resources to automatically delete objects in S3 buckets and images in ECR repositories when deleting CloudFormation stacks.
For more information, please refer to the following page


    Type: AWS::CodeBuild::ReportGroup
      DeleteReports: true
        ExportConfigType: NO_EXPORT
      Name: !Sub "${CodeBuildProject1}-${ReportName}"
      Type: TEST
Code language: YAML (yaml)

To run test units in CodeBuild, create a report group.

Set the type of report group in the Type property.
If “CODE_COVERAGE” is specified, a report on code coverage rate will be created.
When “TEST” is specified, a normal test report is created.
The latter is selected this time.

The DeleteReports property is a parameter for deleting reports.
Enabling this will allow you to delete the group and the report together, even if the report remains inside the group.

ExportConfig is a parameter related to exporting report groups.
It allows exporting the raw report data to an S3 bucket.
We will not export it this time.


    Type: AWS::CodeBuild::Project
        Type: CODEPIPELINE
        Type: NO_CACHE
        ComputeType: !Ref ProjectEnvironmentComputeType
          - Name: REPORT_NAME
            Type: PLAINTEXT
            Value: !Ref ReportName
        Image: !Ref ProjectEnvironmentImage
        ImagePullCredentialsType: CODEBUILD
        Type: !Ref ProjectEnvironmentType
        PrivilegedMode: true
          GroupName: !Ref LogGroup
          Status: ENABLED
          Status: DISABLED
      Name: !Sub "${Prefix}-project-01"
      ServiceRole: !GetAtt CodeBuildRole.Arn
        Type: CODEPIPELINE
        BuildSpec: !Sub |
          version: 0.2
                python: 3.7
                - pip3 install pytest
                - pip3 install bottle
                - python -m pytest --junitxml=reports/pytest_reports.xml
                - pytest_reports.xml
              base-directory: reports
              file-format: JUNITXML
      Visibility: PRIVATE
Code language: YAML (yaml)

Set the environment in which to run the test in the Environment property.

Specify the specifications of the build environment in the ComputeType property.
In this case, specify “BUILD_GENERAL1_SMALL” for the minimum configuration (4GB memory, 2vCPU, 50GB storage).

Build environment compute types - AWS CodeBuild
AWS CodeBuild provides build environments with the following available memory, vCPUs, and disk space:

The EnvironmentVariables property allows you to set environment variables that can be used in the build environment.
In this case, we will set the name of the aforementioned report group.

The Image property specifies the Docker image for creating the build environment.
This time, specify “aws/codebuild/amazonlinux2-aarch64-standard:2.0”, which is the ARM version of Amazon Linux 2.

Docker images provided by CodeBuild - AWS CodeBuild
A supported image is the latest major version of an image available in CodeBuild and is updated with minor and patch version updates. CodeBuild optimizes the pr...

The ImagePullCredentialsType property sets the credentials for pulling images.
Setting “CODEBUILD” will use your own credentials.

The Type property sets the type of build environment.
In this case, set “ARM_CONTAINER” which means ARM environment.

The PrivilegedMode property is a parameter related to the execution of the Docker daemon.
Enable it this time as shown in the following quotation.

Enables running the Docker daemon inside a Docker container. Set to true only if the build project is used to build Docker images. Otherwise, a build that attempts to interact with the Docker daemon fails.

AWS::CodeBuild::Project Environment

Define the contents of buildspec.yml in the Source property.
Use pytest to test your Python scripts.
In this case, we used the following page as a reference for the configuration.

Set up test reporting with pytest - AWS CodeBuild
Set up test reporting in CodeBuild with the pytest testing framework.

The following are the IAM roles for CodeBuild.

    Type: AWS::IAM::Role
        Version: 2012-10-17
          - Effect: Allow
              - sts:AssumeRole
        - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryPowerUser
        - PolicyName: PipelineExecutionPolicy
            Version: 2012-10-17
              - Effect: Allow
                  - ssm:GetParameters
                  - !Sub "arn:aws:ssm:${AWS::Region}:${AWS::AccountId}:parameter/${SSMParameterDockerHubPassword}"
                  - !Sub "arn:aws:ssm:${AWS::Region}:${AWS::AccountId}:parameter/${SSMParameterDockerHubUsername}"
              - Effect: Allow
                  - s3:PutObject
                  - s3:GetObject
                  - s3:GetObjectVersion
                  - s3:GetBucketAcl
                  - s3:GetBucketLocation
                  - !Sub "arn:aws:s3:::${BucketName}"
                  - !Sub "arn:aws:s3:::${BucketName}/*"
              - Effect: Allow
                  - codebuild:CreateReportGroup
                  - codebuild:CreateReport
                  - codebuild:UpdateReport
                  - codebuild:BatchPutTestCases
                  - codebuild:BatchPutCodeCoverages
                  - !Sub "arn:aws:codebuild:${AWS::Region}:${AWS::AccountId}:report-group/*"
              - Effect: Allow
                  - logs:CreateLogGroup
                  - logs:CreateLogStream
                  - logs:PutLogEvents
                  - !GetAtt LogGroup.Arn
                  - !Sub
                    - "${LogGroupArn}:log-stream:*"
                    - LogGroupArn: !GetAtt LogGroup.Arn
Code language: YAML (yaml)

Grant permissions on report groups.


    Type: AWS::CodePipeline::Pipeline
        Location: !Ref BucketName
        Type: S3
      Name: !Ref Prefix
      RoleArn: !GetAtt CodePipelineRole.Arn
        - Actions:
            - ActionTypeId: 
                Category: Source
                Owner: AWS
                Provider: CodeCommit
                Version: 1
                BranchName: !Ref BranchName
                OutputArtifactFormat: CODE_ZIP
                PollForSourceChanges: false
                RepositoryName: !GetAtt CodeCommitRepository.Name
              Name: Source
                - Name: !Ref PipelineSourceArtifact
              Region: !Ref AWS::Region
              RunOrder: 1
          Name: Source
        - Actions:
            - ActionTypeId:
                Category: Test
                Owner: AWS
                Provider: CodeBuild
                Version: 1
                ProjectName: !Ref CodeBuildProject1
                - Name: !Ref PipelineSourceArtifact
              Name: Test
              OutputArtifacts: []
              Region: !Ref AWS::Region
              RunOrder: 1
          Name: Test
        - Actions:
            - ActionTypeId:
                Category: Build
                Owner: AWS
                Provider: CodeBuild
                Version: 1
                ProjectName: !Ref CodeBuildProject2
                - Name: !Ref PipelineSourceArtifact
              Name: Build
                - Name: !Ref PipelineBuildArtifact
              Region: !Ref AWS::Region
              RunOrder: 1
          Name: Build
Code language: YAML (yaml)

Define test stages in the Stages property.
Within the Configuration property, configure the detailed settings for the test stage.
In the ProjectName property, specify the aforementioned CodeBuild.
In the InputArtifacts property, specify the scripts, etc. to be tested. In this case, specify the artifacts of the Source stage.

(Reference) File for application container


FROM amazonlinux

RUN yum update -y && yum install python3 python3-pip -y

RUN pip3 install bottle


CMD ["python3", ""]

Code language: Dockerfile (dockerfile)

The image for the app container will be based on Amazon Linux 2.

We will use Bottle, a Python web framework.
So after installing Python and pip, install this.

Copy the Python script ( describing the app logic and set this to run.

As mentioned earlier, the app listens for HTTP requests on 8080/tcp, so expose this port.

from bottle import route, run

def hello():
  return 'Hello CodePipeline.'
if __name__ == '__main__':
  run(host='', port=8080)
Code language: Python (python)

We will use Bottle to build a simple web server.
The simple configuration is to listen for HTTP requests at 8080/tcp and return “Hello CodePipeline.

import pytest
from main import hello
@pytest.mark.parametrize(('expected',), [
  ('Hello CodePipeline.',),
def test_hello(expected):
  assert hello() == expected
Code language: Python (python)

This is a script to test the hello function defined in in pytest.
The hello function is executed and if it returns the string “Hello CodePipeline.


Using CloudFormation, we will build this environment and check the actual behavior.

Create CloudFormation stacks and check resources in stacks

Create a CloudFormation stacks.
Please refer to the following pages for information on how to create stacks and check each stack.

After reviewing the resources in each stack, the following is the information on the main resources created in this case.

  • ECR: fa-081
  • CodeCommit: fa-081
  • CodeBuild1: fa-081-project-01
  • CodeBuild2: fa-081-project-02
  • CodePipeline: fa-081

Check each resource from the AWS Management Console.
Check CodePipeline.

Detail of CodePipeline 1.

The pipeline has failed to execute.
This is because CodeCommit was created when the CloudFormation stack was created, which triggered the execution of the pipeline.
Since we are not pushing code to CodeCommit at this time, an error occurred during the pipeline execution process.

Note the stage that is being created.
A stage has been created for the test unit, named Test.
When the code is pushed to CodeCommit, the test unit is executed, and if the Operation is checked, the Docker image is built and pushed to the ECR repository.

Check Actions

In case of test succeed

Now that we are ready, we push the code to CodeCommit.

First, pull CodeCommit.

$ git clone
Cloning into 'fa-081'...
warning: You appear to have cloned an empty repository.
Code language: Bash (bash)

An empty repository has been pulled.

Add three files (Dockerfile,, and to the repository.

$ ls -al
total 12
drwxrwxr-x 3 ec2-user ec2-user  71 Aug 21 11:24 .
drwxrwxr-x 6 ec2-user ec2-user 125 Aug 21 11:24 ..
-rw-rw-r-- 1 ec2-user ec2-user 187 Aug 12 11:01 Dockerfile
drwxrwxr-x 7 ec2-user ec2-user 119 Aug 21 11:24 .git
-rw-rw-r-- 1 ec2-user ec2-user 681 Aug 21 08:07
-rw-rw-r-- 1 ec2-user ec2-user 233 Aug 21 08:34
Code language: Bash (bash)

Push the 3 files to CodeCommit.

$ git add .

$ git commit -m 'first commit'
 3 files changed, 51 insertions(+)
 create mode 100644 Dockerfile
 create mode 100644
 create mode 100644

$ git push
 * [new branch]      master -> master
Code language: Bash (bash)

The push was successful.

After waiting for a while, check CodePipeline again.

Detail of CodePipeline 2.

Pipeline has been started.
The Source stage has been successfully completed and the Test stage has been reached.
It now says “In progress,” indicating that the test unit is being executed.
Wait for a while.

Detail of CodePipeline 3.

After the test stage completes successfully, the build stage also completes successfully.

Check the report group of the test.

Detail of CodeBuild 1.

A report group is created.
A summary of the results of the test unit execution is available.

Check the details of the test executed this time.

Detail of CodeBuild 2.

Test results can be displayed in graphs and tables.

Detail of CodeBuild 3.

You can also check the phase transition process during testing.
This time everything was completed successfully.

Detail of CodeBuild 4.

You can also check the detailed logs of the test execution.

Check the ECR repository.

Detail of ECR Repository 1.

The image has been pushed. The test has completed successfully, which means that we have moved to the build stage, the Docker image has been generated and pushed to the repository.

In case of test failure

Check the behavior when the test fails for reference.
Modify the string output by the function to make the test fail on a dare.

Detail of CodePipeline 4.

On CodePipeline, you can see that the test failed.

Detail of CodeBuild 5.

Check the report group for a summary of failed tests.

Then check the details.

Detail of CodeBuild 6.

We can see that it failed in the build phase.

Detail of CodeBuild 7.

The log confirms the cause of the failure.
The log shows that the returned string was not what was expected.

Detail of CodeBuild 8.

You can see the same details from the test case details page.


We have shown you how to add a test stage on CodePipeline.