Introduction to EC2 Image Builder using CloudFormation

2023-10-282024-04-27

Introduction to EC2 Image Builder using CloudFormation

This page covers the EC2 Image Builder.

Image Builder significantly reduces the effort of keeping images up-to-date and secure by providing a simple graphical interface, built-in automation, and AWS-provided security settings. With Image Builder, there are no manual steps for updating an image nor do you have to build your own automation pipeline.
EC2 Image Builder

Since this is an introduction to Image Builder, we will create an AMI with Apache installed and activated. Then use this AMI to automatically create an EC2 instance.

Environment

Diagram of introduction to EC2 Image Builder using CloudFormation.

Create an AMI for Linux using Image Builder. This AMI has Apache installed and activated.

Use this AMI to create an EC2 instance. The OS for the instance will be Amazon Linux 2. Locate this instance on a public subnet and attach a public address.

The goal of this page is to automatically run the Image Builder pipeline and, in response, automatically build an EC2 instance. To do so, we will use several Lambda functions.

The purpose of the first function is to register the ID of the created AMI in the SSM Parameter Store. The purpose of the second function is to create a custom event for EventBridge. These two functions are invoked through SNS upon successful completion of the Image Builder pipeline.

The purpose of the third function is to start the Image Builder pipeline. This function is associated with a CloudFormation custom resource and is automatically invoked when the CloudFormation stack is created.

The action of the fourth function is to send a success signal to the CloudFormation WaitCondition. This function is associated with the EventBridge rule and is triggered when an event is created by function 2.

The runtime environment for all functions is Python 3.8.

CloudFormation template files

The above configuration is built with CloudFormation. The CloudFormation templates are placed at the following URL

https://github.com/awstut-an-r/awstut-fa/tree/main/145

Explanation of key points of template files

Image Builder

In building the Image Builder, the following five resources must be created

Component
Recipe
Infrastructure Configuration and IAM Roles
Distribution setup
Pipeline

Component

AWS officially describes the components as follows

Image Builder uses the AWS Task Orchestrator and Executor (AWSTOE) component management application to orchestrate complex workflows. Build and test components that work with the AWSTOE application are based on YAML documents that define the scripts to customize or test your image.
Manage components with Image Builder

In other words, components define the packages to be installed at AMI build time and the contents of the tests.

Resources:
  Component:
    Type: AWS::ImageBuilder::Component
    Properties:
      Data: |
        schemaVersion: 1.0
        phases:
          - name: build
            steps:
              - name: InstallAndEnableApache
                action: ExecuteBash
                inputs:
                  commands:
                    - yum update -y
                    - yum install -y httpd
                    - systemctl start httpd
                    - systemctl enable httpd
                    - ec2-metadata -i > /var/www/html/index.html
      Name: !Sub "${Prefix}-Component"
      Platform: !Ref ImageBuilderPlatform
      SupportedOsVersions:
        - !Ref ImageBuilderSupportedOsVersion
      Version: !Ref ImageBuilderVersion
Code language: YAML (yaml)

For more information on the components, please see the following pages

https://docs.aws.amazon.com/imagebuilder/latest/userguide/manage-components.html

Only key points will be covered on this page.

The Data property allows you to set the YAML document for the component. In this case, we specify ExecuteBash in action to execute the Bash command. Specify the commands for installing and activating Apache in inputs. Write the ID of the EC2 instance for the AMI build in index.html and place it in the root directory.

The Platform property sets the OS of the component. In this case, “Linux” is specified.

In the SupportedOsVersions property, specify the operating systems and versions supported by this component. In this case, specify “Amazon Linux 2”.

Version specifies the version of this component. In this case, “1.0.0” is used.

Recipe

AWS officially describes the image recipe as follows

An EC2 Image Builder recipe defines the base image to use as your starting point to create a new image, along with the set of components that you add to customize your image and verify that everything works as expected.
Manage recipes

In other words, a recipe specifies the base image of the AMI and the components to be applied.

Resources:
  ImageRecipe:
    Type: AWS::ImageBuilder::ImageRecipe
    Properties:
      Components:
        - ComponentArn: !Ref Component
      Name: !Sub "${Prefix}-ImageRecipe"
      ParentImage: !Ref ImageBuilderParentImage
      Version: !Ref ImageBuilderVersion
Code language: YAML (yaml)

In the Components property, set the component to be applied to the recipe. Specify the component you just checked here.

In the ParentImage property, set the base AMI. In this case, we will specify the AMI for the ARM version of Amazon Linux 2 (arn:aws:imagebuilder:ap-northeast-1:aws:image/amazon-linux-2-arm64/x. x.x).

Version specifies the version of this recipe. This time set to “1.0.0”.

Infrastructure Configuration

The AWS official description of the infrastructure configuration is as follows

You can use infrastructure configurations to specify the Amazon EC2 infrastructure that Image Builder uses to build and test your EC2 Image Builder image.
Manage EC2 Image Builder infrastructure configuration

In other words, infrastructure settings are those that are configured with respect to the EC2 instances for AMI builds.

Resources:
  InfrastructureConfiguration:
    Type: AWS::ImageBuilder::InfrastructureConfiguration
    Properties:
      InstanceProfileName: !Ref ImageBuilderRoleProfile
      InstanceTypes:
        - !Ref InstanceType
      Name: !Sub "${Prefix}-InfrastructureConfiguration"
      SnsTopicArn: !Ref Topic
Code language: YAML (yaml)

The InstanceTypes property allows you to set the type of EC2 instance for the AMI build. In this case, we specify the ARM type t4g.nano.

The SnsTopicArn property allows you to specify the destination of SNS notifications.

When the image status reaches one of the following states, Image Builder publishes a message:

AVAILABLE

FAILED
Amazon SNS integration in Image Builder

This time, by specifying an SNS topic for this property, the Lambda function described below will be invoked when the Image Builder pipeline is completed and the AMI is available.

Specify the SLR for Image Builder in the InstanceProfileName property.

Resources:
  ImageBuilderRoleProfile:
    Type: AWS::IAM::InstanceProfile
    Properties:
      Path: /
      Roles:
        - !Ref ImageBuilderRole

  ImageBuilderRole:
    Type: AWS::IAM::Role
    DeletionPolicy: Delete
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action: sts:AssumeRole
            Principal:
              Service:
                - ec2.amazonaws.com
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
        - arn:aws:iam::aws:policy/EC2InstanceProfileForImageBuilder
Code language: YAML (yaml)

For details, please refer to the following page.

https://docs.aws.amazon.com/imagebuilder/latest/userguide/image-builder-setting-up.html

Distribution setup

AWS officially describes the distribution setup as follows

Specify the name and description of your output AMI.

Authorize other AWS accounts, organizations, and OUs to launch the AMI from the owner’s account. The owner account is billed for charges that are associated with the AMI.
Manage EC2 Image Builder distribution settings

In other words, distribution settings are the settings for the distribution region, etc., of the built image.

Resources:
  DistributionConfiguration:
    Type: AWS::ImageBuilder::DistributionConfiguration
    Properties:
      Distributions:
        - Region: !Ref AWS::Region
          AmiDistributionConfiguration: {}
      Name: !Sub "${Prefix}-DistributionConfiguration"
Code language: YAML (yaml)

You can configure distribution settings in the Distributions property. Specify the ap-northeast-1 region where this stack will be created in the Region property.

In this case, the AmiDistributionConfiguration property is not set to anything, but if you do not specify this item, this resource creation will fail. So specify “{}” for this property.

Pipeline

AWS officially describes the pipeline as follows

Image Builder image pipelines provide an automation framework for creating and maintaining custom AMIs and container images.
Manage EC2 Image Builder pipelines using the console

In other words, the pipeline automatically performs a series of package installation, testing, etc. to build the AMI.

Resources:
  ImagePipeline:
    Type: AWS::ImageBuilder::ImagePipeline
    Properties:
      DistributionConfigurationArn: !Ref DistributionConfiguration
      ImageRecipeArn: !Ref ImageRecipe
      InfrastructureConfigurationArn: !Ref InfrastructureConfiguration
      Name: !Sub "${Prefix}-ImagePipeline"
      Status: ENABLED
Code language: YAML (yaml)

Specify each resource you have identified so far. Specifically, the recipe, infrastructure settings, and distribution settings.

(Reference) SNS

Resources:
  Topic:
    Type: AWS::SNS::Topic
    Properties:
      Subscription:
        - Endpoint: !Ref FunctionArn1
          Protocol: lambda
        - Endpoint: !Ref FunctionArn2
          Protocol: lambda
      TopicName: !Ref Prefix

  SNSPermission1:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName: !Ref Function1
      Action: lambda:InvokeFunction
      Principal: sns.amazonaws.com
      SourceArn: !Ref Topic

  SNSPermission2:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName: !Ref Function2
      Action: lambda:InvokeFunction
      Principal: sns.amazonaws.com
      SourceArn: !Ref Topic
Code language: YAML (yaml)

After the Image Builder pipeline is complete, each resource can be notified of this through SNS. In this case, two Lambda functions will be invoked.

For more information on how to invoke Lambda functions through SNS, please refer to the following page.

(Reference) Lambda

Function 1

Resources:
  ParameterAmi:
    Type: AWS::SSM::Parameter
    Properties:
      Name: !Sub "${Prefix}-Ami"
      Type: String
      Value: " "

  Function1:
    Type: AWS::Lambda::Function
    Properties:
      Architectures:
        - !Ref Architecture
      Environment:
        Variables:
          PARAMETER: !Ref ParameterAmi
      Code:
        ZipFile: |
          import boto3
          import json
          import os

          parameter = os.environ['PARAMETER']
          client = boto3.client('ssm')

          def lambda_handler(event, context):
            message = json.loads(event["Records"][0]["Sns"]["Message"])
            ami = message['outputResources']['amis'][0]['image']

            response = client.put_parameter(
              Name=parameter,
              Value=ami,
              Type='String',
              Overwrite=True
            )
            print(response)
      FunctionName: !Sub "${Prefix}-function-01"
      Handler: !Ref Handler
      Runtime: !Ref Runtime
      Role: !GetAtt FunctionRole1.Arn

  FunctionRole1:
    Type: AWS::IAM::Role
    DeletionPolicy: Delete
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action: sts:AssumeRole
            Principal:
              Service:
                - lambda.amazonaws.com
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      Policies:
        - PolicyName: UpdateSSMParameter
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - ssm:PutParameter
                Resource:
                  - !Sub "arn:aws:ssm:${AWS::Region}:${AWS::AccountId}:parameter/${ParameterAmi}"
Code language: YAML (yaml)

This is one of the Lambda functions called by SNS. The purpose of this function is to store the ID of the built AMI in the SSM Parameter Store.

Function 2

Resources:
  Function2:
    Type: AWS::Lambda::Function
    Properties:
      Code:
        ZipFile: |
          import boto3
          import datetime
          import json
          import os

          event_bus_name = os.environ['EVENT_BUS_NAME']
          detail_type = os.environ['DETAIL_TYPE']
          source = os.environ['SOURCE']

          client = boto3.client('events')

          def lambda_handler(event, context):
            detail = json.dumps({})

            entry = {
              'Time': datetime.datetime.now(),
              'Source': source,
              'Resources': [],
              'DetailType': detail_type,
              'Detail': detail,
              'EventBusName': event_bus_name
            }
            print(entry)

            response = client.put_events(
              Entries=[entry,]
            )
            print(response)
      Environment:
        Variables:
          EVENT_BUS_NAME: !Ref EventBusName
          DETAIL_TYPE: image-builder-finish-event
          SOURCE: !Ref Prefix
      FunctionName: !Sub "${Prefix}-function-02"
      Handler: !Ref Handler
      Runtime: !Ref Runtime
      Role: !GetAtt FunctionRole2.Arn

  FunctionRole2:
    Type: AWS::IAM::Role
    DeletionPolicy: Delete
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action: sts:AssumeRole
            Principal:
              Service:
                - lambda.amazonaws.com
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      Policies:
        - PolicyName: !Sub "${Prefix}-PutEventsPolicy"
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - events:PutEvents
                Resource: !Sub "arn:aws:events:${AWS::Region}:${AWS::AccountId}:event-bus/${EventBusName}"
Code language: YAML (yaml)

It is one of the Lambda functions called by SNS. The purpose of this function is to generate custom events for EventBridge.

For more information, please refer to the following pages

Function 3

Resources:
  CustomResource:
    Type: Custom::CustomResource
    Properties:
      ServiceToken: !GetAtt Function3.Arn

  Function3:
    Type: AWS::Lambda::Function
    Properties:
      Architectures:
        - !Ref Architecture
      Environment:
        Variables:
          IMAGE_PIPELINE_ARN: !Ref ImagePipelineArn
      Code:
        ZipFile: |
          import boto3
          import cfnresponse
          import os

          image_pipeline_arn = os.environ['IMAGE_PIPELINE_ARN']
          client = boto3.client('imagebuilder')

          CREATE = 'Create'
          response_data = {}

          def lambda_handler(event, context):
            try:
              if event['RequestType'] == CREATE:
                response = client.start_image_pipeline_execution(
                  imagePipelineArn=image_pipeline_arn
                )
                print(response)

              cfnresponse.send(event, context, cfnresponse.SUCCESS, response_data)

            except Exception as e:
              print(e)
              cfnresponse.send(event, context, cfnresponse.FAILED, response_data)
      FunctionName: !Sub "${Prefix}-function-03"
      Handler: !Ref Handler
      Runtime: !Ref Runtime
      Role: !GetAtt FunctionRole3.Arn

  FunctionRole3:
    Type: AWS::IAM::Role
    DeletionPolicy: Delete
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action: sts:AssumeRole
            Principal:
              Service:
                - lambda.amazonaws.com
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      Policies:
        - PolicyName: UpdateSSMParameter
          PolicyDocument:
            Version: 2012-10-17
            Statement:
              - Effect: Allow
                Action:
                  - imagebuilder:StartImagePipelineExecution
                Resource:
                  - !Ref ImagePipelineArn
Code language: YAML (yaml)

This function associates with a CloudFormation custom resource.

For more information on custom resources, please see the following pages

The purpose of this function is to start the Image Builder pipeline when creating the CloudFormation stack.

Function 4

Resources:
  WaitConditionHandle:
    Type: AWS::CloudFormation::WaitConditionHandle

  WaitCondition:
    Type: AWS::CloudFormation::WaitCondition
    Properties:
      Handle: !Ref WaitConditionHandle
      Timeout: !Ref WaitConditionTimeout

  EventsRule:
    Type: AWS::Events::Rule
    Properties:
      EventBusName: !Ref EventBusName
      EventPattern:
        source:
          - !Ref Prefix
      Name: !Sub "${Prefix}-EventsRule"
      State: ENABLED
      Targets:
        - Arn: !GetAtt Function4.Arn
          Id: !Ref Function4

  EventsRulePermission:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName: !Ref Function4
      Action: lambda:InvokeFunction
      Principal: events.amazonaws.com
      SourceArn: !GetAtt EventsRule.Arn

  Function4:
    Type: AWS::Lambda::Function
    Properties:
      Architectures:
        - !Ref Architecture
      Environment:
        Variables:
          SIGNAL_URL: !Ref WaitConditionHandle
      Code:
        ZipFile: |
          import boto3
          import json
          import os
          import urllib3
          import uuid

          signal_url = os.environ['SIGNAL_URL']

          def lambda_handler(event, context):
            body = json.dumps({
                "Status": "SUCCESS",
                "Reason": "AMI Building Successed",
                "UniqueId": str(uuid.uuid4()),
                "Data": "Lambda Deploy Package Setup Successed"
            })
            http = urllib3.PoolManager()
            http.request('PUT', signal_url, body=body)
      FunctionName: !Sub "${Prefix}-function-04"
      Handler: !Ref Handler
      Runtime: !Ref Runtime
      Role: !GetAtt FunctionRole4.Arn

  FunctionRole4:
    Type: AWS::IAM::Role
    DeletionPolicy: Delete
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action: sts:AssumeRole
            Principal:
              Service:
                - lambda.amazonaws.com
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
Code language: YAML (yaml)

When using CloudFormation to create an AMI with Image Builder and then use this AMI to create an EC2 instance, the key point is the order of creation. First create the Image Builder resource, then run the pipeline to build the AMI, and finally create the EC2 instance.

To do so, use CloudFormation WaitCondition to control the timing of resource creation. Use WaitCondition to wait for the creation of an EC2 instance.

As mentioned earlier, through the CloudFormation custom resource, Lambda function 3 is invoked to start the Image Builder pipeline. Through the SNS, Lambda function 2 is invoked to generate the EventBridge custom event. Because this event satisfies the EventBridge rule, the Lambda function 4 associated with that rule is triggered to invoke. This function will signal success to WaitCondition and release the wait. In this way, WaitCondition can be used to control when an EC2 instance is created, after the AMI has been created.

For details, please refer to the following page.

The above page covers how to control the order in which the function itself and the deployment package are created when creating a Lambda function. There is a difference between a deployment package for a Lambda function and a source bundle for Elastic Beanstalk, but the idea is exactly the same.

EC2 Instance

Resources:
  Instance:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: !Sub "{{resolve:ssm:${ParameterAmi}}}"
      InstanceType: !Ref InstanceType
      NetworkInterfaces:
        - AssociatePublicIpAddress: true
          DeviceIndex: 0
          GroupSet:
            - !Ref InstanceSecurityGroup
          SubnetId: !Ref PublicSubnet1
Code language: YAML (yaml)

Specify the AMI with the ImageId property. This time, dynamically reference the AMI ID stored in the aforementioned SSM Parameter Store.

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/dynamic-references.html

As we saw earlier, the timing of the EC2 instance creation is adjusted by the CloudFormation WaitCondition. This means that it is always guaranteed that the AMI ID is stored in the SSM Parameter Store.

Architecting

Use CloudFormation to build this environment and check its actual behavior.

Create CloudFormation stacks and check the resources in the stacks

Create CloudFormation stacks. For information on how to create stacks and check each stack, please see the following page.

After reviewing the resources in each stack, information on the main resources created in this case is as follows

Component: fa-145-Component
Recipe: fa-145-ImageRecipe
Infrastructure Configuration: fa-145-InfrastructureConfiguration
Distribution Configuration: fa-145-DistributionConfiguration
Pipeline: fa-145-ImagePipeline
EC2 instance: i-0547b207bb21f2ee0

Check the created resource from the AWS Management Console.

components.

The component has been successfully created. Content to see the contents of the YAML file.

Check the recipe.

The recipe is successfully created. You can see that the component you just checked is specified.

Check infrastructure settings.

Instance types for AMI builds and SNS topics are specified. Two Lambda functions are invoked through this SNS notification.

Check distribution settings.

You can see that the regions are mainly set up.

Check the pipeline.

You can see that the pipeline has been created successfully.

Output Images shows that this pipeline has been executed and an AMI has been created. This means that this pipeline was automatically executed by the Lambda function3 associated with the CloudFormation custom resource.

Find out more about the pipeline execution.

The status of the entire workflow is Completed. There were a total of 7 steps, all with the status Skipped/Completed.

As a sample, we will review the details of the first step.

You can see the ID of the EC2 instance that was temporarily created for this AMI build. The ID of this instance was “i-009be6f511f325923”.

Check the built AMI.

The ID of the AMI created this time is “ami-08855558857f153fa9”.

Check SSM Parameter Store.

Indeed, the AMI ID is stored in the Parameter Store. This means that Lambda function 1 was invoked via SNS.

Check the EC2 instance.

Looking at the AMI ID value, it is an AMI built with Image Builder. This means that Lambda functions 2 and 4 adjusted the timing of the EC2 instance creation, and the instance was created using the AMI that was built.

Action Check

Finally, access this instance.

$ curl http://3.112.218.190
instance-id: i-009be6f511f325923
Code language: Bash (bash)

Instance responded. This ID is for the instance used for the AMI build. This indicates that it is indeed an AMI built by Image Builder.

Summary

Using the Image Builder, we created an AMI. Then we used this AMI to automatically create an EC2 instance.