Execute CodeBuild within Step Functions
Step Functions are integrated with a variety of services.
With AWS service integrations, you can call API actions and coordinate executions directly from your workflow.
Call other AWS services
In this article, we will show you how to call CodeBuild from Step Functions.
Environment
Create a Step Functions state machine.
This state machine downloads videos from YouTube and stores them in an S3 bucket.
Create three states in this state machine.
The first state calls the Lambda function.
Returns the ID of the video to download.
The second state calls CodeBuild.
Select a Lambda type (Python version) build environment.
Download the video from YouTube using yt-dlp.
The video to download is the one with the ID passed from the previous state.
The downloaded video will be stored in an S3 bucket.
The third state calls the Lambda function.
Get a list of objects placed in the S3 bucket.
The runtime environment for the Lambda function we will create is Python 3.12.
CloudFormation template files
The above configuration is built with CloudFormation.
The CloudFormation template is placed at the following URL
Explanation of key points of template files
Step Functions
Resources:
StateMachine:
Type: AWS::StepFunctions::StateMachine
Properties:
Definition:
Comment: !Sub "${Prefix}-statemachine"
StartAt: GetVideoIdState
States:
GetVideoIdState:
Type: Task
Resource: !Ref FunctionArn1
ResultPath: $.video_id
Next: DownloadVideoState
DownloadVideoState:
Type: Task
Resource: arn:aws:states:::codebuild:startBuild.sync
Parameters:
EnvironmentVariablesOverride:
- Name: VIDEO_ID
Type: PLAINTEXT
Value.$: $.video_id
ProjectName: !Ref CodeBuildProjectName
ResultPath: $.codebuild_response
Next: CheckVideoNameState
ListBucketState:
Type: Task
Resource: !Ref FunctionArn2
End: true
LoggingConfiguration:
Destinations:
- CloudWatchLogsLogGroup:
LogGroupArn: !GetAtt StateMachineLogGroup.Arn
IncludeExecutionData: true
Level: ALL
RoleArn: !GetAtt StateMachineRole.Arn
StateMachineName: !Sub "${Prefix}-statemachine"
StateMachineType: STANDARD
Code language: YAML (yaml)Create a Step Functions state machine.
For basic information on state machines, please refer to the following pages.
The key point is the second state (DownloadVideoState).
The Resource property describes the API to be executed.
Please refer to the following page for information on APIs that can be executed in CodeBuild.
This time we will run the StartBuild API to execute the CodeBuild project.
This API also supports the job execution (.sync) integration pattern.
For integrated services such as AWS Batch and Amazon ECS, Step Functions can wait for a request to complete before progressing to the next state. To have Step Functions wait, specify the “Resource” field in your task state definition with the .sync suffix appended after the resource URI.
Run a Job (.sync)
In summary, the value specified for the Resource property is “arn:aws:states:::codebuild:startBuild.sync”.
The Parameters property allows you to set various parameters for executing the API.
Please see the following page for the parameters that can be set to run the StartBuild API.
This time, two parameters are set.
The first is environment variables.
EnvironmentVariablesOverride can be specified to set environment variables.
In this case, we will set the ID of the video to download as an environment variable.
Setting the ID as an environment variable in this way will be the argument for running CodeBuild.
EnvironmentVariablesOverride to specify the arguments for running the CodeBuild project.
This time pass the ID of the video to download.
The second is the name of the CodeBuild project to run.
Specify the name of the project to run in ProjectName.
The following are the IAM roles for this state machine.
Resources:
StateMachineRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action: sts:AssumeRole
Principal:
Service:
- states.amazonaws.com
Policies:
- PolicyName: StateMachinePolicy
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- codebuild:StartBuild
- codebuild:StopBuild
- codebuild:BatchGetBuilds
Resource:
- !Sub "arn:aws:codebuild:${AWS::Region}:${AWS::AccountId}:project/${CodeBuildProjectName}"
- Effect: Allow
Action:
- events:PutTargets
- events:PutRule
- events:DescribeRule
Resource:
- !Sub "arn:aws:events:${AWS::Region}:${AWS::AccountId}:rule/StepFunctionsGetEventForCodeBuildStartBuildRule"
- Effect: Allow
Action:
- lambda:InvokeFunction
Resource:
- !Ref FunctionArn1
- !Ref FunctionArn2
- Effect: Allow
Action:
- logs:CreateLogDelivery
- logs:GetLogDelivery
- logs:UpdateLogDelivery
- logs:DeleteLogDelivery
- logs:ListLogDeliveries
- logs:PutLogEvents
- logs:PutResourcePolicy
- logs:DescribeResourcePolicies
- logs:DescribeLogGroups
Resource: "*"
Code language: YAML (yaml)The points are the first and second policies.
We refer to the following pages
The first is related to CodeBuild.
Give permission to start, stop, etc. CodeBuild.
The second is related to EventBridge.
As for this one, the AWS official explanation is as follows.
Events sent from AWS services to Amazon EventBridge are directed to Step Functions using a managed rule, and require permissions for events:PutTargets, events:PutRule, and events:DescribeRule.
Additional permissions for tasks using the Run a Job pattern
(Reference) Lambda function 1
Resources:
Function1:
Type: AWS::Lambda::Function
Properties:
Architectures:
- !Ref Architecture
Code:
ZipFile: |
def lambda_handler(event, context):
return 'TqqaSD2qTdY'
FunctionName: !Sub "${Prefix}-function-01"
Handler: !Ref Handler
Runtime: !Ref Runtime
Role: !GetAtt FunctionRole1.Arn
FunctionRole1:
Type: AWS::IAM::Role
DeletionPolicy: Delete
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action: sts:AssumeRole
Principal:
Service:
- lambda.amazonaws.com
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
Code language: YAML (yaml)A simple Lambda function that returns a string.
This string is the ID of the YouTube video to download with CodeBuild.
This video is from the official AWS channel.
(Reference) CodeBuild
Resources:
CodeBuildProject:
Type: AWS::CodeBuild::Project
Properties:
Artifacts:
Type: NO_ARTIFACTS
Cache:
Type: NO_CACHE
Environment:
ComputeType: !Ref ProjectEnvironmentComputeType
EnvironmentVariables:
- Name: BUCKET_NAME
Type: PLAINTEXT
Value: !Ref BucketName
- Name: VIDEO_ID
Type: PLAINTEXT
Value: ""
Image: !Ref ProjectEnvironmentImage
ImagePullCredentialsType: CODEBUILD
Type: !Ref ProjectEnvironmentType
LogsConfig:
CloudWatchLogs:
GroupName: !Ref CodeBuildLogGroup
Status: ENABLED
Name: !Sub "${Prefix}-project"
ServiceRole: !GetAtt CodeBuildRole.Arn
Source:
Type: NO_SOURCE
BuildSpec: !Sub |
version: 0.2
phases:
install:
commands:
- pip3 install yt-dlp
build:
commands:
- echo $VIDEO_ID
- yt-dlp -o /tmp/$VIDEO_ID.mp4 $VIDEO_ID
- ls -al /tmp
- aws s3 cp /tmp/$VIDEO_ID.mp4 s3://$BUCKET_NAME/
Visibility: PRIVATE
CodeBuildRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- codebuild.amazonaws.com
Action:
- sts:AssumeRole
Policies:
- PolicyName: CodeBuildPolicy
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- s3:PutObject
Resource:
- !Sub "arn:aws:s3:::${BucketName}/*"
- Effect: Allow
Action:
- logs:CreateLogGroup
- logs:CreateLogStream
- logs:PutLogEvents
Resource:
- !Sub "arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:${CodeBuildLogGroup}"
- !Sub "arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:${CodeBuildLogGroup}:log-stream:*"
Code language: YAML (yaml)Create a Lambda type (Python version) CodeBuild project.
Install yt-dlp and use the package to download videos from YouTube.
The ID of the video to download is received in the form of an environment variable.
The downloaded videos will be stored in an S3 bucket.
For Lambda type CodeBuild, please refer to the following page.
(Reference) Lambda function 2
Resources:
Function2:
Type: AWS::Lambda::Function
Properties:
Architectures:
- !Ref Architecture
Environment:
Variables:
REGION: !Ref AWS::Region
BUCKET_NAME: !Ref BucketName
Code:
ZipFile: |
import boto3
import json
import os
from datetime import date, datetime
region = os.environ['REGION']
bucket_name = os.environ['BUCKET_NAME']
s3_client = boto3.client('s3', region_name=region)
def lambda_handler(event, context):
response = s3_client.list_objects_v2(
Bucket=bucket_name
)
return [obj['Key'] for obj in response['Contents']]
FunctionName: !Sub "${Prefix}-function-02"
Handler: !Ref Handler
Runtime: !Ref Runtime
Role: !GetAtt FunctionRole2.Arn
FunctionRole2:
Type: AWS::IAM::Role
DeletionPolicy: Delete
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action: sts:AssumeRole
Principal:
Service:
- lambda.amazonaws.com
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
Policies:
- PolicyName: FunctionPolicy
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- s3:ListBucket
Resource:
- !Sub "arn:aws:s3:::${BucketName}"
Code language: YAML (yaml)Obtains a list of objects stored in an S3 bucket and returns their object names.
Architecting
Use CloudFormation to build this environment and check its actual behavior.
Create CloudFormation stacks and check the resources in the stacks
Create CloudFormation stacks.
For information on how to create stacks and check each stack, please see the following page.
Check the creation status of each resource from the AWS Management Console.
Check Step Functions.
The state machine has been successfully created.
The chart shows that there are indeed three states available.
Check CodeBuild.
Indeed, a Lambda type CodeBuild project is created.
Check the two Lambda functions.
Both functions are created successfully.
Operation Check
Now that you are ready, run the state machine.
After a short wait, the process was successfully completed.
It passed through three states and completed successfully.
Check the results of each state execution from the log.
Check the first log.
The first Lambda function has been executed and the ID has been returned.
Check the second log.
Above is part of the log.
The BuildStatus value shows “SUCCEEDED”.
You can see that the build is indeed complete.
Incidentally, the result of executing the CodeBuild StartBuild API is in the following format.
Check the third log.
The filename of the video downloaded and stored in the S3 bucket was returned.
Finally, check the S3 bucket.
The video was indeed saved in the bucket.
Summary
We have shown you how to call CodeBuild from Step Functions.