Preparing Lambda Layer Package with CFN Custom Resources – General File Version
In the following pages, we have shown you how to automatically create a Lambda layer package for Python.
In this case, we will consider layering a general file that is language-independent and can be referenced from a function.
For example, we assume a certificate file.
Environment
Basically, the structure is the same as the page introduced at the beginning of this document.
There are two points of difference.
The first point is the string to be stored in the SSM Parameter Store.
In the opening page, we registered a list of names of Python packages to install.
In this configuration, we will register a list of URLs of files to include in the Lambda layer.
The second point is the Lambda function to be associated with the custom resource.
In the opening page, we downloaded the package with pip, a package management tool for Python.
In this configuration, we use urllib.request.urlopen to download the file.
Note that the runtime environment for the function is Python 3.8.
CloudFormation template files
The above configuration is built with CloudFormation.
The CloudFormation templates are placed at the following URL
https://github.com/awstut-an-r/awstut-fa/tree/main/130
Explanation of key points of template files
SSM Parameter Store
Resources:
UrlsParameter:
Type: AWS::SSM::Parameter
Properties:
Name: !Ref Prefix
Type: String
Value: |
https://data.nasa.gov/resource/gvk9-iz74.json
Code language: YAML (yaml)Register a list of URLs of files to download.
In this case, we will target a JSON file published by NASA as a sample file.
CloudFormation Custom Resources
Lambda Function
First, identify the Lambda function to be executed on the custom resource.
Resources:
Function1:
Type: AWS::Lambda::Function
Properties:
Architectures:
- !Ref Architecture
Environment:
Variables:
LAYER_PACKAGE: !Ref LayerPackage
REGION: !Ref AWS::Region
URLS_PARAMETER: !Ref UrlsParameter
S3_BUCKET: !Ref CodeS3Bucket
S3_BUCKET_FOLDER: !Ref Prefix
Code:
ZipFile: |
import boto3
import cfnresponse
import os
import shutil
import subprocess
import urllib
layer_package = os.environ['LAYER_PACKAGE']
region = os.environ['REGION']
urls_parameter = os.environ['URLS_PARAMETER']
s3_bucket = os.environ['S3_BUCKET']
s3_bucket_folder = os.environ['S3_BUCKET_FOLDER']
CREATE = 'Create'
response_data = {}
work_dir = '/tmp'
package_dir = 'python'
package_dir_path = os.path.join(work_dir, package_dir)
layer_package_path = os.path.join(
work_dir,
layer_package
)
ssm_client = boto3.client('ssm', region_name=region)
s3_client = boto3.client('s3', region_name=region)
def lambda_handler(event, context):
try:
if event['RequestType'] == CREATE:
ssm_response = ssm_client.get_parameter(Name=urls_parameter)
urls = ssm_response['Parameter']['Value']
result = subprocess.run(
['mkdir', package_dir_path],
stdout=subprocess.PIPE,
stderr=subprocess.PIPE
)
for url in urls.splitlines():
print(url)
file_name = os.path.basename(url)
download_path = os.path.join(package_dir_path, file_name)
data = urllib.request.urlopen(url).read()
with open(download_path, mode='wb') as f:
f.write(data)
shutil.make_archive(
os.path.splitext(layer_package_path)[0],
format='zip',
root_dir=work_dir,
base_dir=package_dir
)
s3_client.upload_file(
layer_package_path,
s3_bucket,
os.path.join(s3_bucket_folder, layer_package)
)
cfnresponse.send(event, context, cfnresponse.SUCCESS, response_data)
except Exception as e:
print(e)
cfnresponse.send(event, context, cfnresponse.FAILED, response_data)
EphemeralStorage:
Size: !Ref EphemeralStorageSize
FunctionName: !Sub "${Prefix}-function1"
Handler: !Ref Handler
Runtime: !Ref Runtime
Role: !GetAtt FunctionRole1.Arn
Timeout: !Ref Timeout
Code language: YAML (yaml)The Architecture and Runtime properties are key.
It must match the value specified in the Lambda layer resource described below.
In this case, we set the following.
- Architecture：arm64
- Runtime：python3.8
EphemeralStorage and Timeout properties are also parameters that need to be adjusted.
Depending on the volume and number of libraries to be included in the Lambda layer, these values may need to be set larger.
In this case, we set the following.
- EphemeralStorage：512
- Timeout：300
The Environment property allows you to define environment variables to be passed to the function.
Pass the file name of the package to be created, the parameter name of the SSM parameter store, and information about the S3 bucket where the package will be placed.
Define the code to be executed by the Lambda function in inline notation.
For more information, please see the following page.
https://awstut.com/en/2022/02/02/3-parterns-to-create-lambda-with-cloudformation-s3-inline-container
Use the cfnresponse module to implement the function as a Lambda-backed custom resource.
For more information, please see the following page.
https://awstut.com/en/2022/05/04/introduction-to-cloudformation-custom-resources-en
The code to be executed is as follows.
- Access os.environ to obtain the environment variables defined in the CloudFormation template.
- Access the SSM parameter store in Boto3 to obtain a list of URLs.
- After downloading the file with urllib.request.urlopen, write to the file.
- ZIP the installed library with shutil.make_archive.
- Upload the ZIP file to the S3 bucket with Boto3.
IAM roles for functions are as follows
Resources:
FunctionRole1:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action: sts:AssumeRole
Principal:
Service:
- lambda.amazonaws.com
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
Policies:
- PolicyName: CreateLambdaLayerPackagePolicy
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- ssm:GetParameter
Resource:
- !Sub "arn:aws:ssm:${AWS::Region}:${AWS::AccountId}:parameter/${UrlsParameter}"
- Effect: Allow
Action:
- s3:PutObject
Resource:
- !Sub "arn:aws:s3:::${CodeS3Bucket}/*"
Code language: YAML (yaml)In addition to the AWS management policy AWSLambdaBasicExecutionRole, grant permission to retrieve parameters from the SSM parameter store and upload objects to the S3 bucket.
Custom Resources
Then check the CloudFormation custom resource.
Resources:
CustomResource:
Type: Custom::CustomResource
Properties:
ServiceToken: !GetAtt Function1.Arn
Code language: YAML (yaml)Specify the aforementioned Lambda function.
Lambda Layer
Resources:
LambdaLayer:
Type: AWS::Lambda::LayerVersion
DependsOn:
- CustomResource
Properties:
CompatibleArchitectures:
- !Ref Architecture
CompatibleRuntimes:
- !Ref Runtime
Content:
S3Bucket: !Ref CodeS3Bucket
S3Key: !Ref LayerS3Key
Description: !Ref Prefix
LayerName: !Ref Prefix
Code language: YAML (yaml)There are three points.
The first is when this resource is created.
This resource must be created after the aforementioned Lambda function is executed.
So specify a custom resource in DependsOn.
The second is the CompatibleArchitectures and CompatibleRuntimes properties.
Set the same value as specified in the Lambda function described above.
The third is the Content property.
Specify the ZIP file uploaded by the aforementioned Lambda function.
(Reference) Lambda function for confirmation
Resources:
Function2:
Type: AWS::Lambda::Function
Properties:
Architectures:
- !Ref Architecture
Environment:
Variables:
DIR_PATH: /opt/python
FILE_NAME: gvk9-iz74.json
Code:
ZipFile: |
import json
import os
dir_path = os.environ['DIR_PATH']
file_name = os.environ['FILE_NAME']
def lambda_handler(event, context):
file_path = os.path.join(dir_path, file_name)
with open(file_path, 'r') as f:
json_data = json.load(f)
print(json_data)
return {
'statusCode': 200,
'body': json.dumps(json_data, indent=2)
}
FunctionName: !Sub "${Prefix}-function2"
Handler: !Ref Handler
Layers:
- !Ref LambdaLayer
Runtime: !Ref Runtime
Role: !GetAtt FunctionRole2.Arn
Timeout: !Ref Timeout
FunctionUrl:
Type: AWS::Lambda::Url
Properties:
AuthType: NONE
TargetFunctionArn: !GetAtt Function2.Arn
FunctionUrlPermission:
Type: AWS::Lambda::Permission
Properties:
Action: lambda:InvokeFunctionUrl
FunctionName: !GetAtt Function2.Arn
FunctionUrlAuthType: NONE
Principal: "*"
FunctionRole2:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action: sts:AssumeRole
Principal:
Service:
- lambda.amazonaws.com
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
Code language: YAML (yaml)Since this is a confirmation function, no special settings are required.
In the Layers property, specify the Lambda layer described above.
Open the JSON file contained in the layer and return its contents.
The key point is the location of the Lambda layer content.
If your Lambda function includes layers, Lambda extracts the layer contents into the /opt directory in the function execution environment.
Accessing layer content from your function
To invoke this function, create a Function URL.
For details on Function URLs, please see the following page.
https://awstut.com/en/2022/04/24/lambda-function-url-by-cloudformation-auth-type-none-en
Architecting
Use CloudFormation to build this environment and check its actual behavior.
Create CloudFormation stacks and check the resources in the stacks
Create CloudFormation stacks.
For information on how to create stacks and check each stack, please see the following page.
https://awstut.com/en/2021/12/11/cloudformations-nested-stack
After reviewing the resources in each stack, information on the main resources created in this case is as follows
- Lambda function 1: fa-130-function1
- Lambda function 2: fa-130-function2
- Function URL for Lambda Function 2: https://24ugu7yzkz6jl3lqrskebwwpoa0gnymj.lambda-url.ap-northeast-1.on.aws
- Parameters in SSM parameter store: fa-130
- S3 bucket and folder to store packages for Lambda layer: awstut-bucket/fa-130
Check each resource from the AWS Management Console.
First check the CloudFormation custom resource.
You can see that the Lambda function and the custom resource body have been successfully created.
Next, check the values stored in the SSM parameter store.
The URL of the file to download and include in the Lambda layer is registered.
Operation Check
Now that we are ready, we will check the actual operation.
Result of Lambda function execution
First, check the execution results of the Lambda function associated with the CloudFormation custom resource in the CloudWatch Logs log group.
You can see that this function has been executed by a CloudFormation custom resource.
And we can also see that the file was downloaded.
This means that the Lambda layer package was successfully created.
S3 bucket
Next, access the S3 bucket to check the installation status of the Lambda layer package.
Indeed, the Lambda layer package (layer.zip) is installed in the S3 bucket.
Lambda Layer
Check the status of Lambda layer creation.
The architecture and runtime are configured as specified in the CloudFormation template.
This layer should have been created by referencing the ZIP file on the S3 bucket mentioned earlier.
Lambda function for test
Check the status of Lambda function creation for test.
You can see that the aforementioned Lambda layers are related.
Now that you are ready, access the Function URL for this function.
$ curl https://24ugu7yzkz6jl3lqrskebwwpoa0gnymj.lambda-url.ap-northeast-1.on.aws/
[
{
"center": "Kennedy Space Center",
"center_search_status": "Public",
"facility": "Control Room 2/1726/HGR-S ",
"occupied": "1957-01-01T00:00:00.000",
"record_date": "1996-03-01T00:00:00.000",
"last_update": "2015-06-22T00:00:00.000",
"country": "US",
"contact": "Sheryl Chaffee",
"phone": "321-867-8047",
"location": {
"latitude": "28.538331",
"longitude": "-81.378879",
"human_address": "{\"address\": \"\", \"city\": \"\", \"state\": \"\", \"zip\": \"32899\"}"
},
"city": "Kennedy Space Center",
"state": "FL",
"zipcode": "32899",
":@computed_region_bigw_e76g": "173",
":@computed_region_cbhk_fwbd": "30",
":@computed_region_nnqa_25f4": "1078"
},
...
]Code language: Bash (bash)The contents of the JSON file are returned.
We can see that the Lambda layer contains this file and that we can access it from our function.
Summary
By using CloudFormation custom resources, we were able to automate the creation of a Lambda layer containing general files.