AWS_EN

Create and Delete S3 Object by CFN Custom Resource

スポンサーリンク
Create and Delete S3 Object by CFN Custom Resource AWS_EN
スポンサーリンク
スポンサーリンク

How to create/delete S3 objects during stack creation/deletion with CloudFormation custom resources

CloudFormation custom resources can perform any action during stack operations (create, update, delete).

In this case, we will use a custom resource to achieve the following actions

  • Automatically create S3 object when creating S3 buckets in CloudFormation.
  • When deleting the CloudFormation stack containing the S3 bucket, automatically delete all objects in the bucket.

For more information on the basics of custom resources, please refer to the following page

Environment

Diagram of create and delete S3 object by CFN Custom Resource.

Create a CloudFormation stack and define two resources inside it.

The first is an S3 bucket.
You will perform object operations on this bucket.

The second is a Lambda function.
This function is set up as a custom resource.
It should be configured to automatically perform object creation and deletion on the bucket when creating and deleting stacks.
The object to be created is index.html for static website hosting.
The runtime for the function is Python 3.8.

CloudFormation template files

The above configuration is built with CloudFormation.
The CloudFormation template is located at the following URL

awstut-fa/047 at main · awstut-an-r/awstut-fa
Contribute to awstut-an-r/awstut-fa development by creating an account on GitHub.

Explanation of key points of template files

This page focuses on how to manipulate S3 objects using custom resources.

For more information on S3 static website hosting, please refer to the following page

Lambda functions that work as custom resources

Resources: Function: Type: AWS::Lambda::Function Properties: Code: ZipFile: | import boto3 import cfnresponse import os bucket_name = os.environ['BUCKET_NAME'] object_name = 'index.html' object_body = """<html> <head></head> <body> <h1>index.html</h1> <p>{bucket_name}</p> </body> </html>""".format(bucket_name=bucket_name) content_type = 'text/html' char_code= 'utf-8' s3_client = boto3.client('s3') CREATE = 'Create' DELETE = 'Delete' response_data = {} def lambda_handler(event, context): try: if event['RequestType'] == CREATE: put_response = s3_client.put_object( Bucket=bucket_name, Key=object_name, Body=object_body.encode(char_code), ContentEncoding=char_code, ContentType=content_type) print(put_response) elif event['RequestType'] == DELETE: list_response = s3_client.list_objects_v2( Bucket=bucket_name) for obj in list_response['Contents']: delete_response = s3_client.delete_object( Bucket=bucket_name, Key=obj['Key']) print(delete_response) cfnresponse.send(event, context, cfnresponse.SUCCESS, response_data) except Exception as e: print(e) cfnresponse.send(event, context, cfnresponse.FAILED, response_data) Environment: Variables: BUCKET_NAME: !Ref BucketName FunctionName: !Sub "${Prefix}-function" Handler: !Ref Handler Runtime: !Ref Runtime Role: !GetAtt FunctionRole.Arn
Code language: YAML (yaml)

There are no special items in the configuration of the function itself.
One thing to mention is the environment variable.
The name of the bucket in which the object is to be created or deleted is passed to the function as an environment variable.

Refer to the value of event[‘RequestType’] to implement the processing according to the stack operation.
When creating a stack, this value is “Create”, and when deleting a stack, it is “Delete”.

When the stack is created, that is, “Create”, an HTML file (index.html) is created.
Execute “put_object” to save the file in the bucket.

When deleting the stack, i.e., “Delete”, all objects in the bucket are deleted.
After retrieving all objects in the bucket with list_objects_v2, delete them one by one with delete_object.

A function execution completion message must be returned to the CloudFormation stack.
In this case, cfnresponse.send is used to implement this function.

There are two issues with the object deletion we have implemented this time.

The first is that we do not expect to delete more than 1000 objects.
The boto3 list_objects_v2 is designed to return a token to retrieve the remaining data when trying to retrieve more than 1000 objects.
In this code, that part of the process has not yet been implemented.

Second, versioning is not taken into account.
If versioning of S3 buckets is enabled, the bucket cannot be deleted without also deleting the object version information.
In this code, that part of the process is not yet implemented.

The following page introduces a code that takes the two points into account.

カスタムリソースを使用して CloudFormation スタックの削除時に S3 バケットと一緒にオブジェクトを削除する方法 | DevelopersIO
CloudFormation スタックを削除する際にカスタムリソースを使用して S3 バケットとオブジェクトを同時に削除する方法について紹介します。

(Reference) IAM role for Lambda functions to create and delete S3 objects

Resources: FunctionRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: sts:AssumeRole Principal: Service: - lambda.amazonaws.com ManagedPolicyArns: - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole Policies: - PolicyName: !Sub "${Prefix}-S3Access" PolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: - s3:ListBucket - s3:GetObject - s3:PutObject - s3:DeleteObject Resource: - !Ref BucketArn - !Sub "${BucketArn}/*"
Code language: YAML (yaml)

IAM role for Lambda functions.
The content of this role grants permission to create and delete internal objects for the target bucket.

CloudFormation Custom Resource

Resources: CustomResource: Type: Custom::CustomResource Properties: ServiceToken: !Ref FunctionArn
Code language: YAML (yaml)

Specify the ARN of the aforementioned Lambda function in the ServiceToken property.
This setting will cause the function to be executed each time the CloudFormation stack is operated.

Architecting

Using CloudFormation, we will build our environment and check the actual behavior.

Create CloudFormation stacks and check resources in stacks

Create a CloudFormation stack using AWS CLI.
This configuration consists of four separate template files, which are placed in an arbitrary bucket.

The following is an example of creating a stack by referencing template files placed in an arbitrary S3 bucket.
The stack name is “fa-047”, the bucket name is “awstut-bucket”, and the folder name where the files are placed is “fa-047”.

$ aws cloudformation create-stack \ --stack-name fa-047 \ --template-url https://awstut-bucket.s3.ap-northeast-1.amazonaws.com/fa-047/fa-047.yaml \ --capabilities CAPABILITY_IAM
Code language: YAML (yaml)

For information on how to create stacks and check each stack, please refer to the following page

Check the stack creation status from the AWS Management Console.

CloudFormation Stack 1

You will see that the stack created by the command and three stacks nested in this stack have been created.

Check the resources created from the stack for S3 among the nested stacks.

CloudFormation Stack 2

Indeed, an S3 bucket has been created.

Then check the custom resource and the Lambda function.

CloudFormation Stack 3
CloudFormation Stack 4

Both are created.
If they work properly, the files should have been created in the bucket when the stack was created.

Test

Now that everything is ready, check the contents of the bucket.

HTML file was generated by CFN custom resource.

Indeed, index.html has been created.

Next, access the static website hosting endpoint.

HTML file was generated by CFN custom resource.

It was successfully accessed.
When the CloudFormation stack was created, the Lambda function associated with the custom resource was executed and the index.html file was placed in the S3 bucket.

Next, let’s try deleting the entire stack.
Essentially, if any objects remain in the S3 bucket, the stack deletion will fail.

Delete the CloudFormation stack with the object still in the S3 bucket.

After waiting for a while, the deletion completes successfully.

The CloudFormation custom resource deleted the objects in the bucket and then deleted the stack.

This indicates that when deleting the CloudFormation stack, the Lambda function associated with the custom resource is executed and all objects stored in the S3 bucket are deleted.

Summary

We have seen how to use CloudFormation custom resources to create/delete S3 objects during stack creation/deletion.

タイトルとURLをコピーしました