Perform initial setup of DynamoDB with CloudFormation custom resources
When creating DynamoDB with CloudFormation, we will consider adding test records as part of the DB initialization.
In this case, we will use a CloudFormation custom resource to perform the initial setup of the DynamoDB table.
Environment
Create a DynamoDB table using CloudFormation.
Create a Lambda function.
Associate the function with a CloudFormation custom resource so that it will be automatically executed when the CloudFormation stack is created.
The function’s action is to initialize the DynamoDB table.
The initialization process reads JSON data placed in an S3 bucket and registers it as an item in the DynamoDB table.
The runtime environment for the function is Python 3.8.
CloudFormation template files
The above configuration is built using CloudFormation.
The CloudFormation templates are located at the following URL
https://github.com/awstut-an-r/awstut-fa/tree/main/105
Explanation of key points of the template files
DynamoDB table
Resources:
Table:
Type: AWS::DynamoDB::Table
Properties:
AttributeDefinitions:
- AttributeName: Artist
AttributeType: S
- AttributeName: SongTitle
AttributeType: S
BillingMode: PROVISIONED
KeySchema:
- AttributeName: Artist
KeyType: HASH
- AttributeName: SongTitle
KeyType: RANGE
ProvisionedThroughput:
ReadCapacityUnits: !Ref ReadCapacityUnits
WriteCapacityUnits: !Ref WriteCapacityUnits
TableClass: STANDARD
TableName: !Sub "${Prefix}-Music"
Code language: YAML (yaml)No special configuration is required.
The configuration on the following official AWS page is exported to a CloudFormation template.
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/getting-started-step-1.html
For basic information about DynamoDB, please also check the following page
Lambda Function
Resources:
Function:
Type: AWS::Lambda::Function
Properties:
Environment:
Variables:
JSON_S3_BUCKET: !Ref JsonS3Bucket
JSON_S3_KEY: !Ref JsonS3Key
TABLE_NAME: !Ref Table
Code:
ZipFile: |
import boto3
import cfnresponse
import json
import os
JSON_S3_BUCKET = os.environ['JSON_S3_BUCKET']
JSON_S3_KEY = os.environ['JSON_S3_KEY']
TABLE_NAME = os.environ['TABLE_NAME']
CREATE = 'Create'
response_data = {}
s3_client = boto3.client('s3')
dynamodb_client = boto3.client('dynamodb')
def lambda_handler(event, context):
try:
if event['RequestType'] == CREATE:
s3_response = s3_client.get_object(
Bucket=JSON_S3_BUCKET,
Key=JSON_S3_KEY)
body = s3_response['Body'].read()
json_data = json.loads(body.decode('utf-8'))
print(json_data)
for item in json_data:
dynamodb_response = dynamodb_client.put_item(
TableName=TABLE_NAME,
Item=item
)
print(dynamodb_response)
cfnresponse.send(event, context, cfnresponse.SUCCESS, response_data)
except Exception as e:
print(e)
cfnresponse.send(event, context, cfnresponse.FAILED, response_data)
FunctionName: !Sub "${Prefix}-function"
Handler: !Ref Handler
Runtime: !Ref Runtime
Role: !GetAtt FunctionRole.Arn
Timeout: !Ref Timeout
Code language: YAML (yaml)Define the code to be executed by the Lambda function in inline notation.
For more information, please refer to the following page
Use the cfnresponse module to implement functions as Lambda-backed custom resources.
For more information, please see the following page
Environment variables can be defined with the Variables property.
Environment variables can be referenced from within functions.
In this case, we will pass the name of the S3 bucket that contains the data for the initialization process and the file name as environment variables.
The code to be executed is as follows
- Access os.environ to obtain the environment variables defined in the CloudFormation template.
- Create a client object for S3 in Boto3 and retrieve the JSON file for initialization from the S3 bucket.
- Create a DynamoDB client object in Boto3 and store the sample data in a table.
- Notify CloudFormation of the successful completion of the custom resource using the send function of the cfnresponse module.
The JSON file to be read from the S3 bucket is as follows
[
{"Artist": {"S": "No One You Know"}, "SongTitle": {"S": "Call Me Today"}, "AlbumTitle": {"S": "Somewhat Famous"}, "Awards": {"N": "1"}},
{"Artist": {"S": "No One You Know"}, "SongTitle": {"S": "Howdy"}, "AlbumTitle": {"S": "Somewhat Famous"}, "Awards": {"N": "2"}},
{"Artist": {"S": "Acme Band"}, "SongTitle": {"S": "Happy Day"}, "AlbumTitle": {"S": "Songs About Life"}, "Awards": {"N": "10"}},
{"Artist": {"S": "Acme Band"}, "SongTitle": {"S": "PartiQL Rocks"}, "AlbumTitle": {"S": "Another Album Title"}, "Awards": {"N": "8"}}
]
Code language: JSON / JSON with Comments (json)This data is also presented on the official AWS page.
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/getting-started-step-2.html
The IAM roles for the functions are as follows
Resources:
FunctionRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action: sts:AssumeRole
Principal:
Service:
- lambda.amazonaws.com
ManagedPolicyArns:
- arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
Policies:
- PolicyName: !Sub "${Prefix}-S3GetObjectPolicy"
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- s3:GetObject
Resource:
- !Sub "arn:aws:s3:::${JsonS3Bucket}/*"
- PolicyName: !Sub "${Prefix}-DynamodbPutItemPolicy"
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- dynamodb:PutItem
Resource:
- !Sub "arn:aws:dynamodb:${AWS::Region}:${AWS::AccountId}:table/${Table}"
Code language: YAML (yaml)In addition to the AWS management policy AWSLambdaBasicExecutionRole, an inline policy grants two permissions.
The first is the authority to retrieve objects from the S3 bucket.
This is the permission needed to retrieve the JSON file that is the sample data.
The second is the authority to store items in DynamoDB.
This is the authorization needed to write the data in the JSON file to a DynamoDB table.
CloudFormation Custom Resources
Next, we will check the CloudFormation custom resource itself.
Resources:
CustomResource:
Type: Custom::CustomResource
Properties:
ServiceToken: !Ref FunctionArn
Code language: YAML (yaml)Associate the custom resource with the Lambda function described above.
Architecting
Use CloudFormation to build this environment and check the actual behavior.
Create CloudFormation stacks and check resources in stacks
Create a CloudFormation stack.
For information on how to create a stack and check each stack, please refer to the following page
After checking the resources in each stack, information on the main resources created this time is as follows
- DynamoDB table: fa-105-Music
- Lambda function: fa-105-function
Check each resource from the AWS Management Console.
First, check the CloudFormation custom resources and Lambda functions.
We can see that the Lambda function and the custom resource itself have indeed been created successfully.
This means that the Lambda function was automatically created and invoked when the CloudFormation stack was created.
Next, check the DynamoDB.
The DynamoDB table is created as specified in the CloudFormation template.
Checking Action
Checking the Operation of the CloudFormation custom resource.
Check the execution log of the Lambda function associated with the CloudFormation custom resource.
The JSON data in the S3 bucket is read and saved one row at a time in the DynamoDB table.
Finally, a message is notified to CloudFormation.
Check the contents of the DynamoDB table.
The items are indeed stored in the table.
From the above, we can confirm that when the CloudFormation stack is created, the Lambda function associated with it is automatically invoked by the CloudFormation custom resource, and the item is saved in the DynamoDB table.
Summary
We have seen how to perform the initial setup of a DynamoDB table using a CloudFormation custom resource.