AWS_EN

Preparing Lambda Layer Package with CFN Custom Resources – Python Version

スポンサーリンク
Preparing Lambda Layer Package with CloudFormation Custom Resource - Python Version AWS_EN
スポンサーリンク
スポンサーリンク

Automatically create and deploy Lambda layer package for Python using CloudFormation custom resources

The following page covers how to create a Lambda layer.

In the above page, the Lambda layer itself is created by CloudFormation, but the packages for the Lambda layer were created manually in advance.

In this case, we will use CloudFormation custom resources to automate the preparation of this package.
Specifically, the custom resource will automate the process of creating the package and placing it in the S3 bucket.

The target runtime environment is Python 3.8 and pip is used.

Environment

Diagram of preparing Lambda Layer Package with CloudFormation Custom Resource - Python Version

Create two Lambda functions.

In the first function, create a Lambda layer package and place it in an S3 bucket.
Associate the function with a CloudFormation custom resource so that it will automatically run when the CloudFormation stack is created.
Use pip to prepare the library to be included in the layer, but install it by referencing the values registered in the SSM parameter store.

The second function will be used to check the Lambda layer.
Activate the Function URL.

CloudFormation template files

Build the above configuration with CloudFormation.
The CloudFormation templates are located at the following URL

awstut-fa/064 at main · awstut-an-r/awstut-fa
Contribute to awstut-an-r/awstut-fa development by creating an account on GitHub.

Explanation of key points of the template files

SSM Parameter Store

Resources: RequirementsParameter: Type: AWS::SSM::Parameter Properties: Name: !Ref Prefix Type: String Value: | requests beautifulsoup4
Code language: YAML (yaml)

Register the libraries to be installed with pip in the SSM parameter store.
As described later, create requirements.txt from this string and install the libraries in bulk.

CloudFormation Custom Resource

First, check the Lambda functions to be executed in the custom resource.

Resources: Function1: Type: AWS::Lambda::Function Properties: Architectures: - !Ref Architecture Environment: Variables: LAYER_PACKAGE: !Ref LayerPackage REGION: !Ref AWS::Region REQUIREMENTS_PARAMETER: !Ref RequirementsParameter S3_BUCKET: !Ref CodeS3Bucket S3_BUCKET_FOLDER: !Ref Prefix Code: ZipFile: | ... EphemeralStorage: Size: !Ref EphemeralStorageSize FunctionName: !Sub "${Prefix}-function1" Handler: !Ref Handler Runtime: !Ref Runtime Role: !GetAtt FunctionRole1.Arn Timeout: !Ref Timeout
Code language: YAML (yaml)

The Architecture and Runtime properties are the key points.
These properties must match the values specified in the Lambda layer resource described below.
In this case, we set them as follows.

  • Architecture: arm64
  • Runtime: python3.8

EphemeralStorage and Timeout properties are also parameters that need to be adjusted.
Depending on the capacity and number of libraries to be included in the Lambda layer, these values may need to be set higher.
In this case, we set them as follows

  • EphemeralStorage: 512
  • Timeout: 300

The Environment property allows you to define environment variables that can be passed to the function.
You can pass the file name of the package to be created, the parameter name of the SSM parameter store, and information about the S3 bucket where the package will be placed.

Define the code to be executed by the Lambda function in inline notation.
For more information, please refer to the following page

Here is the code to execute

import boto3 import cfnresponse import os import pip import shutil import subprocess layer_package = os.environ['LAYER_PACKAGE'] region = os.environ['REGION'] requirements_parameter = os.environ['REQUIREMENTS_PARAMETER'] s3_bucket = os.environ['S3_BUCKET'] s3_bucket_folder = os.environ['S3_BUCKET_FOLDER'] CREATE = 'Create' response_data = {} work_dir = '/tmp' requirements_file = 'requirements.txt' package_dir = 'python' requirements_path = os.path.join(work_dir, requirements_file) package_dir_path = os.path.join(work_dir, package_dir) layer_package_path = os.path.join(work_dir, layer_package) def lambda_handler(event, context): try: if event['RequestType'] == CREATE: ssm_client = boto3.client('ssm', region_name=region) ssm_response = ssm_client.get_parameter(Name=requirements_parameter) requirements = ssm_response['Parameter']['Value'] with open(requirements_path, 'w') as file_data: print(requirements, file=file_data) pip.main(['install', '-t', package_dir_path, '-r', requirements_path]) shutil.make_archive( os.path.splitext(layer_package_path)[0], format='zip', root_dir=work_dir, base_dir=package_dir ) s3_resource = boto3.resource('s3') bucket = s3_resource.Bucket(s3_bucket) bucket.upload_file( layer_package_path, '/'.join([s3_bucket_folder, layer_package]) ) cfnresponse.send(event, context, cfnresponse.SUCCESS, response_data) except Exception as e: print(e) cfnresponse.send(event, context, cfnresponse.FAILED, response_data)
Code language: Python (python)

Use the cfnresponse module to implement functions as Lambda-backed custom resources.
For more information, please see the following page

The code to be executed is as follows.

  1. retrieve the environment variables defined in the CloudFormation template by accessing os.environ.
  2. retrieve the SSM parameter store values in Boto3 and create requirements.txt file.
  3. specify requirements.txt in pip and install all libraries at once.
  4. make a ZIP file of the installed libraries with shutil.make_archive.
  5. Upload the ZIP file to the S3 bucket with Boto3.

By the way, the IAM role for the function is as follows

Resources: FunctionRole1: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: sts:AssumeRole Principal: Service: - lambda.amazonaws.com ManagedPolicyArns: - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole Policies: - PolicyName: CreateLambdaLayerPackagePolicy PolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: - ssm:GetParameter Resource: - !Sub "arn:aws:ssm:${AWS::Region}:${AWS::AccountId}:parameter/${RequirementsParameter}" - Effect: Allow Action: - s3:PutObject Resource: - !Sub "arn:aws:s3:::${CodeS3Bucket}/*"
Code language: YAML (yaml)

In addition to the AWS management policy AWSLambdaVPCAccessExecutionRole, we grant permission to retrieve parameters from the SSM parameter store and upload objects to the S3 bucket.

Then check the CloudFormation custom resource body.

Resources: CustomResource: Type: Custom::CustomResource Properties: ServiceToken: !GetAtt Function1.Arn
Code language: YAML (yaml)

Specify the Lambda function described above.

Lambda Layer

Resources: LambdaLayer: Type: AWS::Lambda::LayerVersion DependsOn: - CustomResource Properties: CompatibleArchitectures: - !Ref Architecture CompatibleRuntimes: - !Ref Runtime Content: S3Bucket: !Ref CodeS3Bucket S3Key: !Ref LayerS3Key Description: !Ref Prefix LayerName: !Ref Prefix
Code language: YAML (yaml)

There are three key points.

The first is when this resource is created.
This resource must be created after the aforementioned Lambda function is executed.
Therefore, we specify a custom resource in DependsOn.

The second is the CompatibleArchitectures and CompatibleRuntimes properties.
These should be set to the same values as those specified in the Lambda function described above.

The third is the Content property.
Specify the ZIP file uploaded by the aforementioned Lambda function.

(Reference) Lambda function for confirmation

Resources: Function2: Type: AWS::Lambda::Function Properties: Architectures: - !Ref Architecture Code: ZipFile: | import json import requests from bs4 import BeautifulSoup url = 'https://news.google.com/rss' def lambda_handler(event, context): r = requests.get(url) soup = BeautifulSoup(r.text, 'lxml') titles = [item.find('title').getText() for item in soup.find_all('item')] return { 'statusCode': 200, 'body': json.dumps(titles, indent=2) } FunctionName: !Sub "${Prefix}-function2" Handler: !Ref Handler Layers: - !Ref LambdaLayer Runtime: !Ref Runtime Role: !GetAtt FunctionRole2.Arn
Code language: YAML (yaml)

Since this is a confirmation function, no special settings are required.

Specify the aforementioned Lambda layer in the Layers property.
The import statement imports the libraries included in the layer.
Checking the Operation, you can confirm that the layer package creation and Lambda layer resource creation have been successfully completed.

The content of the code is to use requests and BeautifulSoup included in the layer to retrieve RSS of Google News and return the title of each item.

Architecting

Using CloudFormation, we will build this environment and check the actual behavior.

Create CloudFormation stacks and check resources in stacks

Create a CloudFormation stacks.
For information on how to create stacks and check each stack, please refer to the following page

After checking the resources in each stack, information on the main resources created this time is as follows

  • Lambda function 1: fa-064-function1
  • Lambda function 2: fa-064-function2
  • Function URL for Lambda function 2: https://xljafu2utaoovuvrgvqcxtl2540uafya.lambda-url.ap-northeast-1.on.aws/
  • Parameters in SSM parameter store: fa-064
  • S3 bucket and folder to store packages for Lambda layer: awstut-bucket/fa-064

Check each resource from the AWS Management Console.
First, check the CloudFormation custom resource.

CloudFormation Stack.

You can see that the Lambda function and the custom resource itself have been successfully created.

Next, check the values stored in the SSM parameter store.

SSM Parameter Store.

You can see that the contents of requirements.txt are stored.

Checking Action

Now that everything is ready, let’s check the Operation.

Execution result of Lambda function

First, check the execution result of the Lambda function associated with the CloudFormation custom resource in the CloudWatch Logs log group.

Result of CloudFormation Custom Resource 1.

From the log, we can see that the library was installed according to the contents of requirements.txt stored in the SSM parameter store.
This means that the Lambda layer package was successfully created.

We can also see that the function returns “SUCCESS” as a CloudFormation custom resource.
This means that the function has successfully acted as a custom resource.

S3 Bucket

Next, we access the S3 bucket to check the installation status of the Lambda layer package.

Result of CloudFormation Custom Resource 2.

Indeed, the Lambda layer package (layer.zip) has been installed in the S3 bucket.

Lambda Layer

Check the creation status of the Lambda layer.

Lambda Layer.

The architecture and runtime are configured as specified in the CloudFormation template.
This layer should have been created by referencing the zip file on the S3 bucket mentioned above.

Confirmation Lambda Function

Check the creation status of the confirmation Lambda function.

Lambda Function 1.

You can see that there is a Lambda layer associated with it.

Below are the details of the layers.

Lambda Function 2.

You can see that the aforementioned Lambda layer is associated.

Now that we are ready, we access the Function URL of this function.
For more information on Function URL, please refer to the following page.

Result of Lambda Function with Lambda Layer.

You have successfully obtained RSS information from Google News.
We can see that we can use requests and BeautifulSoup, which are included in the Lambda layer associated with this function.

This confirms that we were able to automatically prepare a Lambda layer package for Python by using CloudFormation custom resources.

Summary

We have confirmed how to automate the process of creating a Lambda layer package for Python and placing it in an S3 bucket using a custom resource.

タイトルとURLをコピーしました