How to create/delete S3 objects during stack creation/deletion with CloudFormation custom resources
CloudFormation custom resources can perform any action during stack operations (create, update, delete).
In this case, we will use a custom resource to achieve the following actions
- Automatically create S3 object when creating S3 buckets in CloudFormation.
- When deleting the CloudFormation stack containing the S3 bucket, automatically delete all objects in the bucket.
For more information on the basics of custom resources, please refer to the following page
Environment
Create a CloudFormation stack and define two resources inside it.
The first is an S3 bucket.
You will perform object operations on this bucket.
The second is a Lambda function.
This function is set up as a custom resource.
It should be configured to automatically perform object creation and deletion on the bucket when creating and deleting stacks.
The object to be created is index.html for static website hosting.
The runtime for the function is Python 3.8.
CloudFormation template files
The above configuration is built with CloudFormation.
The CloudFormation template is located at the following URL
https://github.com/awstut-an-r/awstut-fa/tree/main/047
Explanation of key points of template files
This page focuses on how to manipulate S3 objects using custom resources.
For more information on S3 static website hosting, please refer to the following page
Lambda functions that work as custom resources
Resources:
Function:
Type: AWS::Lambda::Function
Properties:
Code:
ZipFile: |
import boto3
import cfnresponse
import os
bucket_name = os.environ['BUCKET_NAME']
object_name = 'index.html'
object_body = """<html>
<head></head>
<body>
<h1>index.html</h1>
<p>{bucket_name}</p>
</body>
</html>""".format(bucket_name=bucket_name)
content_type = 'text/html'
char_code= 'utf-8'
s3_client = boto3.client('s3')
CREATE = 'Create'
DELETE = 'Delete'
response_data = {}
def lambda_handler(event, context):
try:
if event['RequestType'] == CREATE:
put_response = s3_client.put_object(
Bucket=bucket_name,
Key=object_name,
Body=object_body.encode(char_code),
ContentEncoding=char_code,
ContentType=content_type)
print(put_response)
elif event['RequestType'] == DELETE:
list_response = s3_client.list_objects_v2(
Bucket=bucket_name)
for obj in list_response['Contents']:
delete_response = s3_client.delete_object(
Bucket=bucket_name,
Key=obj['Key'])
print(delete_response)
cfnresponse.send(event, context, cfnresponse.SUCCESS, response_data)
except Exception as e:
print(e)
cfnresponse.send(event, context, cfnresponse.FAILED, response_data)
Environment:
Variables:
BUCKET_NAME: !Ref BucketName
FunctionName: !Sub "${Prefix}-function"
Handler: !Ref Handler
Runtime: !Ref Runtime
Role: !GetAtt FunctionRole.Arn
Code language: YAML (yaml)There are no special items in the configuration of the function itself.
One thing to mention is the environment variable.
The name of the bucket in which the object is to be created or deleted is passed to the function as an environment variable.
Refer to the value of event[‘RequestType’] to implement the processing according to the stack operation.
When creating a stack, this value is “Create”, and when deleting a stack, it is “Delete”.
When the stack is created, that is, “Create”, an HTML file (index.html) is created.
Execute “put_object” to save the file in the bucket.
When deleting the stack, i.e., “Delete”, all objects in the bucket are deleted.
After retrieving all objects in the bucket with list_objects_v2, delete them one by one with delete_object.
A function execution completion message must be returned to the CloudFormation stack.
In this case, cfnresponse.send is used to implement this function.
There are two issues with the object deletion we have implemented this time.
The first is that we do not expect to delete more than 1000 objects.
The boto3 list_objects_v2 is designed to return a token to retrieve the remaining data when trying to retrieve more than 1000 objects.
In this code, that part of the process has not yet been implemented.
Second, versioning is not taken into account.
If versioning of S3 buckets is enabled, the bucket cannot be deleted without also deleting the object version information.
In this code, that part of the process is not yet implemented.
The following page introduces a code that takes the two points into account.
https://dev.classmethod.jp/articles/custom-resource-empty-s3-objects/
(Reference) IAM role for Lambda functions to create and delete S3 objects
Resources:
FunctionRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action: sts:AssumeRole
Principal:
Service:
- lambda.amazonaws.com
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
Policies:
- PolicyName: !Sub "${Prefix}-S3Access"
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- s3:ListBucket
- s3:GetObject
- s3:PutObject
- s3:DeleteObject
Resource:
- !Ref BucketArn
- !Sub "${BucketArn}/*"
Code language: YAML (yaml)IAM role for Lambda functions.
The content of this role grants permission to create and delete internal objects for the target bucket.
CloudFormation Custom Resource
Resources:
CustomResource:
Type: Custom::CustomResource
Properties:
ServiceToken: !Ref FunctionArn
Code language: YAML (yaml)Specify the ARN of the aforementioned Lambda function in the ServiceToken property.
This setting will cause the function to be executed each time the CloudFormation stack is operated.
Architecting
Using CloudFormation, we will build our environment and check the actual behavior.
Create CloudFormation stacks and check resources in stacks
Create a CloudFormation stack using AWS CLI.
This configuration consists of four separate template files, which are placed in an arbitrary bucket.
The following is an example of creating a stack by referencing template files placed in an arbitrary S3 bucket.
The stack name is “fa-047”, the bucket name is “awstut-bucket”, and the folder name where the files are placed is “fa-047”.
$ aws cloudformation create-stack \
--stack-name fa-047 \
--template-url https://awstut-bucket.s3.ap-northeast-1.amazonaws.com/fa-047/fa-047.yaml \
--capabilities CAPABILITY_IAM
Code language: YAML (yaml)For information on how to create stacks and check each stack, please refer to the following page
Check the stack creation status from the AWS Management Console.
You will see that the stack created by the command and three stacks nested in this stack have been created.
Check the resources created from the stack for S3 among the nested stacks.
Indeed, an S3 bucket has been created.
Then check the custom resource and the Lambda function.
Both are created.
If they work properly, the files should have been created in the bucket when the stack was created.
Test
Now that everything is ready, check the contents of the bucket.
Indeed, index.html has been created.
Next, access the static website hosting endpoint.
It was successfully accessed.
When the CloudFormation stack was created, the Lambda function associated with the custom resource was executed and the index.html file was placed in the S3 bucket.
Next, let’s try deleting the entire stack.
Essentially, if any objects remain in the S3 bucket, the stack deletion will fail.
After waiting for a while, the deletion completes successfully.
This indicates that when deleting the CloudFormation stack, the Lambda function associated with the custom resource is executed and all objects stored in the S3 bucket are deleted.
Summary
We have seen how to use CloudFormation custom resources to create/delete S3 objects during stack creation/deletion.